Fetch `t.bilibili.com/<id>`, `www.bilibili.com/opus/<id>`,
`t.bilibili.com/h5/dynamic/detail/<id>` and `m.bilibili.com/dynamic/<id>`
through the anonymous `/x/polymer/web-dynamic/v1/detail` endpoint (no
cookie, no WBI signature; the site adds the device cookies
`/x/frontend/finger/spi` hands out, which is what lifts bilibili's
`-352` risk control).
Media: the `major.draw` grid (`.gif` sources become animations, the rest
photos with a downscaled `@518w.jpg` thumbnail used both as preview and
as the oversized fallback), an attached video's cover, and the quoted
dynamic's media for forwards. The video stream itself is not resolved;
`b23.tv` short links stay unmatched (they mostly point at videos, so
matching them would turn a silently ignored link into a failure reply).
`-352`/`-412` map to a retryable error so the queue backs off instead of
dropping the post; a removed dynamic (`500`) is permanent.
Registry-driven, so no bot-side code changes beyond the site lists in the
command replies; found while researching nazurin and
telegram-bili-feed-helper (see BILIBILI_PLAN.md).
Resolve the manifest and lockfile overlap with the rusqlite 0.40 and zip 8.6
bumps that landed on master after this branch was opened:
- crates/xmedia-bot/Cargo.toml: keep rusqlite 0.40 and rand 0.10
- Cargo.lock: re-point x-media/xmedia-bot at rand 0.10.2; teloxide keeps its
own rand 0.8.8 (it requires ^0.8.5), and cargo pruned the now-unused
version_check entry
Verified on the merged tree: cargo metadata --locked, cargo fmt --check,
cargo clippy --workspace --all-targets --locked -- -D warnings,
cargo test --workspace --locked (69 + 73 tests).
rand 0.10 renamed the entry points dependabot's version bump alone cannot
follow: `thread_rng()` -> `rng()`, `gen_range()` -> `random_range()` and the
`Rng` trait -> `RngExt`.
The jitter only needs one float, so use the free function
`rand::random_range(0.2..0.8)` and drop the now-unused trait import instead
of importing `RngExt`.
Verified: cargo fmt --check, cargo clippy --workspace --all-targets -- -D
warnings, cargo test --workspace --locked (retry_delay_seconds_bounds keeps
the 0.2..0.8 jitter window).
Bump both crates (x-media, xmedia-bot) and refresh the lockfile to the
latest semver-compatible releases. No direct dependency or manifest
requirement changed.
- `/test <url>` now runs the ordinary link pipeline and actually sends the
media, but with the chat's post-send actions suppressed: no channel forward,
no edit-before-forward prompt. It is the same code path as a normal link
(same caption/format handling, link cache, retries, dead-letter
notification), so "does this link work?" is answered by the send itself.
- `/debug <url>` keeps what `/test` used to do: fetch and reply with the HTML
parse report, sending/caching/forwarding nothing.
- `urls::url_media` takes a `PostSend` mode (`FromChat` for the URL workers,
`Suppressed` for `/test`); `build_send_task` maps it to the task's
`edit_before_forward`/`forward_channel_id`. Notification ids stay set in both
modes, so a queued retry still reports a dead-letter to the chat.
- `/test` rejects an unsupported URL with the same message the old parse-only
command used (the URL flow would otherwise ignore it silently).
- Report builder renamed `test_parse_report` -> `debug_report` (with the cap
constant), `parse_test_arg` -> `parse_arg_remainder` (now shared by both
commands). README/README.en command tables and AGENTS.md updated; `/help`
descriptions come from the enum.
Tests: +3 (normal flow still honours the chat's settings, `/test` sends with
them suppressed and keeps the cache entry, `build_send_task` mode mapping). The
suppression test was verified to fail when the mode is ignored.
fmt/clippy clean, 73 + 69 tests pass.
- `--locked` on every cargo invocation (ci.yml clippy/test/build, both
Dockerfile builds). The version bump edits Cargo.lock by hand, so a stale
lock must fail loudly instead of being silently re-resolved: CI would
otherwise test a different dependency set than the one committed — and than
the one the released image is built from.
- docker.yml: build the image (no push, no registry login, read-only build
cache) on pull requests touching the build inputs. The Dockerfile's
stub-source machinery, the ffmpeg download and the entrypoint previously
only ran at release time. Also: a release tag must equal both crate versions
before anything is built (the binary carries no version, so `v1.5.1` with
manifests at 1.5.0 used to publish silently wrong tags), `FFMPEG_URL`/
`FFMPEG_SHA256` are taken from repository variables when set, and the
unused `setup-qemu-action` step is gone (single-arch build; the comment says
what arm64 would need).
- ci.yml: `concurrency` cancels superseded runs, `permissions: contents: read`,
`RUST_BACKTRACE=1`, job timeouts, and a release-profile build of the same
package the Dockerfile builds (the profile was otherwise never compiled
before a merge). The `live` job narrows to `-p x-media`: every network- or
secret-gated test lives there, and the bot crate's offline suite already ran
in the `test` job. Timeout is 45 min because the release build is cold on
the first run — a timeout there would kill the job before rust-cache could
save its cache, leaving every later run cold too.
- Actions pinned to commit SHAs (Dependabot keeps them current);
`dtolnay/rust-toolchain` stays on its channel ref by design.
- .github/dependabot.yml: crates (patch bumps grouped), action pins, Docker
base images — the audit gate reports advisories, this is what moves them.
- tokio's `sync` feature is now declared instead of arriving transitively via
teloxide; `.dockerignore` drops docs and markdown.
Verified locally: `cargo fmt --check`, `cargo clippy --workspace
--all-targets --locked`, `cargo test --workspace --locked` (70 + 69 pass),
`cargo build --release --locked` (6m03s cold, the 15.9 MB stripped binary
starts and registers 10 commands), the tag/version gate against both a
matching and a mismatching tag, and YAML parsing of all three workflow files.
send.rs had grown back into the shape handlers.rs was split out of: payload
types, error classification, the download-and-reupload fallback, the senders
and the whole post-send/queue shell in one file. Split by concern, leaving
call sites (`crate::send::x`) unchanged:
- `send/input_media.rs`: payload → `InputFile`/`InputMedia` selection and
`build_media_group` (with its caption-on-first-item rule).
- `send/upload.rs`: the fallback pipeline (download with the upload cap,
photo downscale handoff, smaller-URL fallback, multipart upload).
- `send/post_send.rs`: link-cache write, the `KEEP_ALIVE` registry for locally
produced media, `settle_task`, the post-send actions and the queue entry
points; the parts other modules call are re-exported.
- `send/mod.rs`: payloads, error classification, classification helpers and
the senders themselves, plus the test module.
No behaviour change: 128 + 1326 + 366 + 345 lines, 70 tests still pass.
AGENTS.md updated for the new layout and for `ctx.rs`.
docs/architecture-refactor.md §3 sketched the trait with "按需扩展：
edit_message_caption / delete_message / answer_callback_query …", but only the
five send methods landed, so `callback.rs` and the edit-before-forward caption
swap were stuck on the concrete `Bot` and remained untested (AGENTS.md still
lists callback.rs as untestable).
- `MediaSender` gains `answer_callback_query`, `edit_message_caption` (HTML
parse mode baked in, every caller uses it) and `delete_message`; the mock
records call order plus the texts, captions and answer toasts, so tests can
assert what the user saw.
- `send_message` now returns the sent message id instead of the whole
`Message`: the only consumer of the value is the edit-before-forward prompt
(which keys its record by it), and returning a `Message` forced every mock
to build a teloxide type. `reply`/`reply_html` follow.
- `callback.rs`: the dptree entry only unpacks the update; `handle_callback`
takes plain values + `&AppContext`. `handlers/mod.rs::edit_message_handler`
likewise takes the values the reply carries. Admin/setup APIs
(`get_chat`, `get_chat_administrators`, `get_me`, `set_my_commands`) stay on
the concrete `Bot`: they are not user flows worth a trait.
- The scripted mock moves to `parking_lot::Mutex` (no poisoning unwraps).
Tests: +11 (template button, forward ok/no-channel/retryable, expired+unknown
prompt, caption swap via template, escaping of user text into the caption,
failed swap still consuming the reply, prompt record written by post_send).
fmt/clippy clean, 70 + 69 tests pass.
docs/architecture-refactor.md §3 stopped half-done: `url_media` got an injected
`AppContext`, but `send.rs`'s post-send half kept reaching for the process-wide
`CHAT_STORE`/`TASK_QUEUE`/`LINK_CACHE` statics, so the whole shell after a
successful send (edit-before-forward prompt, channel forward, retry enqueue,
cache write) had no test and no way to get one.
- `ctx.rs` now owns `AppContext` (sender + the three stores + config) with
`from_statics` for production and a `CONTEXT` static for the spawned worker
closures; `handlers/urls.rs` drops its private copy and the duplicated
assembler, and the queue handler/dead-letter callbacks take the context
(main wires them with `CONTEXT`).
- `send_media_sequence`/`send_animation`/`forward_messages`/`post_send_actions`
take `&AppContext`; the cache write goes through the injected cache.
- New `settle_task(ctx, task, Sent|Failed)` is the single place that ends a
task: release its keep-alive temp media, and drop the link-cache entry only
on failure. All five former call sites funnel through it — the earlier
keep-alive leak existed precisely because one of them had to remember.
`invalidate_cache`/`invalidate_cache_with` (static + injected pair, the
latter only existing because of the former) collapse into one private fn.
- `ctx::test_support::TestStores` gives tests a tempdir store set + context;
`handlers/urls.rs` tests use it instead of hand-rolled setup.
Tests: +5 (post-send forward ok / queued / notified, settle Sent/Failed); the
post-send and settle paths were previously untested. fmt/clippy clean,
60 + 69 tests pass.
AGENTS.md:
- db.rs row claimed a per-store connection pool; there is one shared pool for
all three tables (statics.rs builds it once).
- retry enqueue moved to send.rs, noted in both handlers rows.
- queue row now names both notifies (workers' + the sweep's).
- Retries bullet documents fetch vs fetch_once.
- test count ~125 -> ~135, the untested-files list no longer claims state.rs
and handlers.rs are untested, and the live-test inventory mentions the
token-gated, not-#[ignore]d pixiv download test that makes a local
`cargo test --workspace` hit the network.
- /bot_dict is admin-only now.
Code docs:
- site/mod.rs: the module doc pointed new sites at `fetch_once` (a name that
did not exist then and now means a single-attempt fetch) -> `SITES`; the
cache_key/SITES/Site docs still said "twitter -> bsky -> pixiv" (misskey
is registered third); RenderData now documents which fields are escaped
and why url/author_url are not.
- state.rs, callback.rs: drop the pre-misskey site list and the `<name>`
that rustdoc read as an HTML tag.
- Fixed the remaining rustdoc links/warnings: `cargo doc --workspace
--no-deps` is now warning-free (was 8).
- docs/site-registry-refactor.md: §1 describes the pre-refactor state; said so.
No behavior change. fmt/clippy clean, 55 + 68 tests pass (the live pixiv
download test flaked on a CDN body timeout, as before).
- queue: the lease-expiry sweep waited on the workers' `Notify`. `notify_one`
stores a permit, so a sweep wakeup could consume the one meant for a worker,
which then blocked on `notified()` (it only waits when the table looked
empty, i.e. indefinitely) with a due row sitting there. The sweep now has
its own notify, woken only by stop.
- x-media: split fetch's retry loop into `fetch` (3 attempts, unchanged) and
`fetch_once` (1 attempt); inline queries use the latter — the 800ms debounce
plus 1s/2s backoffs were outlasting the answer window of the query.
- send: chunk_media_items now moves items out of the input Vec instead of
requiring `T: Clone` and copying every payload.
fmt/clippy clean, 55 + 69 tests pass.
- commands: /bot_dict dumped the whole chat state to any member of the chat
and could exceed Telegram's 4096-char message limit (the send then failed
and bubbled up as a handler error). It is now admin-only and capped at
MAX_DEBUG_DUMP_CHARS; README, README.en and the /help description updated.
- send: the edit-before-forward template buttons were built from a HashMap
walk, so their order changed between prompts. Now sorted by name.
- link_cache: an unparseable payload (older schema) was reported as a miss
but left in place, re-failing the parse on every later hit; the row is
dropped on read.
- handlers: a link handed to the URL workers after the channel closed
(shutdown) was discarded silently; it is now logged.
- rate_limit: LIMITERS kept one bucket per chat that ever sent media,
forever. The periodic sweep now drops buckets that are idle (refilled to
capacity) and not held by an in-flight sender; acquire's refill was
factored into a shared helper used by the idle check.
Tests: +3 (corrupted row dropped, sorted markup, idle-bucket pruning); the
cache one was verified to fail before the fix. fmt/clippy clean, 55 + 69.
- send: the PHOTO_INVALID_DIMENSIONS marker never matched (the description is
lower-cased, the marker was not), so oversized photos sent by URL were
classified Permanent instead of taking the download-and-downscale fallback.
- inline: the debounce state was one global slot, so a second user's query
cancelled the first user's pending answer entirely; it is now per user.
- state: prune_expired wrote back a stale snapshot without the per-chat lock,
clobbering a concurrent update() (lost edit-message record -> "Expired");
it now re-reads and prunes under the same lock update() uses.
- send/queue: a task dead-lettered on retry exhaustion kept its keep-alive
temp media (ugoira MP4) alive until process exit; dead_letter_notify now
releases it, and enqueue_retry releases when the enqueue itself fails.
Also folds the duplicated retry enqueue in post_send_actions into
send::enqueue_retry (single clock source, single place that releases).
Tests: +6 (marker, per-user debounce x3, keep-alive release, prune contract
x2); the marker and keep-alive cases were verified to fail before the fix.
cargo fmt/clippy clean, 52 + 69 tests pass.
Fourth site adapter: POST /api/notes/show, renote-aware caption and
media normalization, DriveFile type → Illustration/Animated/Video.
Empty thumbnailUrl strings filtered out in thumbnail_for.
- updated_sequence_task: clone the Task and mutate the two fields
instead of rebuilding all 12 by hand (-22 lines; new fields no
longer need a sync here)
- unify unix_now with db::now_f64 (unix_now() = now_f64() as i64),
moved to db.rs next to its clock source
- classify_to_send_error takes the MediaFetchFailure label, folding
the duplicated inline match in send_batch_via_upload (-8 lines)
- photo.rs: chunks_exact(4)/(2) -> as_chunks::<N>().0
(chunks_exact_to_as_chunks, the new lint prefers the
compile-time-checked slice split)
- send.rs: box the Task inside SendError so the error fits the
result_large_err limit (Task is ~400 bytes; the error now moves
through Result as a pointer); unbox with *task at the two
enqueue_retry call sites (handlers/urls.rs, handlers/callback.rs)
cargo clippy --workspace --all-targets is now warning-free; the
remaining proc-macro-error2 future-incompat note is upstream
(teloxide -> aquamarine) and unfixable locally. Full test suite passes.
1.3.0 → 1.4.0: new features (DATA_DIR config, /test blockquote HTML
report) plus the twitter entity-decode, post-send-actions and queue
lease-heartbeat fixes.
Replaces the strip-tags plain-text rendering: the /test reply is now an
HTML message (reply_html helper with ParseMode::Html). Raw fields (url,
source_url, title, author_url, media urls) are escaped, the pre-escaped
render fields are embedded as-is, and the caption is wrapped in
<blockquote>...</blockquote> so the report shows it exactly as it will
render in the sent media caption — escaped text and clickable links
included, no literal &/</> and no raw markup.
The DB file was hardcoded to CWD-relative data/task_queue.db — a footgun
for systemd/cron deployments and a confusing startup failure when the
data/ dir did not exist (SQLite never creates parent dirs).
db_path() now reads DATA_DIR (default data, CWD-relative, unchanged for
local runs and the docker-compose ./data mount) and creates the
directory automatically. README/README.en.md env tables and AGENTS.md
document the new variable.
Runs actions-rust-lang/audit after the offline tests in the test job: a
crate in Cargo.lock with an unfixed security advisory fails the build.
Verified locally against the current lockfile (0 vulnerabilities; the 3
warnings — unmaintained dotenv/proc-macro-error2 and transitive anyhow
unsoundness — do not fail by default).
AGENTS.md claimed user-facing bot strings are Chinese, but every
reply/send_message string in the code is English (Hello!, Send failed,
No media found, Reply to edit message, ...). README stays Chinese;
update both the overview line and the convention line to state the
actual split.
The report's caption line still showed the raw HTML markup
(<a href="...">...</a>). strip_html_tags now drops the tags (keeping
the visible text; the links are already reported via source_url /
author_url) and the remaining entity-encoded text is decoded — the
strip runs on the escaped caption so a tweet text like >^ω^< survives
instead of being eaten as markup. Custom-format captions contain no
tags and pass through unchanged.
The lease was set once to now + LOCK_TTL_SECONDS (120 s) with no
renewal. Tasks that legitimately take longer — slow CDN downloads,
ugoira encodes, rate-limited batch forwards (a 100-message channel copy
waits ~4 min on the per-chat token bucket) — had their lease expire
mid-run; the 30 s expiry sweep flipped the row back to pending and
another worker processed it again, double-sending.
run_with_lease now drives the handler through tokio::select! and
refreshes locked_until every 30 s while it runs. The heartbeat lives in
the same future as the handler, so a panicking worker still lets the
sweep recover the row (no leaked task keeping the lease fresh forever).
A task only reaches the queue after a failed send, so the fresh attempt
never ran post_send_actions (edit-before-forward prompt / channel
forward) — it failed before that point. The old guard skipped
post_send_actions for resumed tasks (batch_index > 0 or sent ids
present), which meant any send that needed a retry after partial
progress silently lost its forward and edit prompt.
post_send_actions is now run unconditionally on a successful queue send;
it executes exactly once, after the whole sequence completed.
AGENTS.md: document the twitter API entity decode and the /test report
HTML-decoded display; add the missing db.rs / media_sender.rs /
rate_limit.rs module rows; fix the statics location (handlers/statics.rs);
refresh test counts (~115, twitter live 5, pixiv api.rs 1, photo heavy
test); versioning convention now includes README.en.md.
README.md / README.en.md: add TELOXIDE_PROXY to the env variable list.
Twitter's syndication and GraphQL APIs return tweet text and display
names pre-escaped for HTML (> < & '); the caption builder
escaped the text again, so sent messages showed literal entities (e.g.
>^ω^< came back as >^ω^<). from_syndication_json now decodes the
API text before storing it — both the syndication path and the
TWITTER_AUTH_TOKEN GraphQL fallback route through it — so the caption
escapes exactly once and renders correctly.
The /test report is a plain-text message but printed the pre-escaped
caption and render fields; it now HTML-decodes them for display so the
report shows the rendered text.
Deleted tweets answer the syndication endpoint with HTTP 200 and a
TweetTombstone (no `errors`, no `id_str`). The body classifier only
knew the `errors` shape, so tombstones fell through to the
`no id_str -> Sensitive` branch and degraded to an empty result,
making the bot reply "No media found" for a deleted tweet.
- extract parse_syndication_body(); tombstone shape -> NotFound
- propagate NotFound from the TWITTER_AUTH_TOKEN GraphQL fallback
instead of swallowing it into empty_fetched
- unit tests for all body classes + live regression test on a real
tombstoned tweet
P0 — level rework + redaction:
- info now carries only lifecycle, per-post business results (sent /
forwarded / copied / template applied), admin actions and anomalies
(upload fallback, retry enqueue; dead-letter stays error).
- Per-request detail moved to debug: message/command logging, URL
extraction, fetching/fetched, link-cache hits, media-group batch sends,
queue processing (enqueue/processing/completed/rescheduled), photo
processing (downscale/transcode), inline queries, sensitive-tweet note.
- Full user-submitted URLs and message text now appear only at debug; at
info and above links are printed via the normalized cache key.
P1 — request correlation:
- handlers::log_key() maps a URL to its normalized post key
(twitter:<id> / pixiv:<id> / bsky:<handle>/<rkey>). The whole lifecycle
of one link (fetch -> send -> cache -> fallback) now logs [key=...], so
multi-worker logs can be correlated by grepping the key.
Convention documented in AGENTS.md.
site::fetch retried every PixivError, so a bad/expired token (403) or a
deleted artwork (404) burned all 3 attempts with backoff against pixiv's
API for nothing. Add PixivError::Status(u16) — the app-API calls now
surface the HTTP status — and retry only the transient classes: network
errors, 429 and 5xx. 4xx / Api (token errors) / Json / NoAuth are
returned immediately. The classification is a pure helper
(fetch_error_is_retryable) with unit tests.
The old stop only set an atomic flag checked between jobs: a worker
blocked in recv() never woke (the channel was never closed), and queued
jobs were neither drained nor abandoned in a defined way despite the
"drains up to 256 jobs" comment. Now stop_url_workers sets the flag,
drops the sender so blocked recv() calls wake with None, and awaits the
worker JoinHandles (each finishes its in-flight job first). main awaits
it inside the existing 30s shutdown timeout.
Smaller/faster production binary (verified: cargo build --release -p
xmedia-bot builds clean with the new profile). panic=abort is
intentionally not set — queue workers and db closures rely on JoinHandle
catching panics, which abort would defeat.
Full English translation of README.md — features, quick start, webhook
deployment (domain + IP-only with acme.sh), env table, command table.
The Chinese README stays the primary one.
A typo in EDIT_MESSAGE_TTL_SECONDS / WEBHOOK_PORT etc. used to silently
fall back to a default, so the bot ran with different behavior than the
operator intended (or failed much later on a bare .expect). Unparseable
values now log a loud warning naming the variable; invalid webhook
settings still surface as a hard .expect in webhook mode.
The stop sequence awaited the queue workers, which can be mid-download
(30s client timeout) or mid-ugoira encode (minutes). A stuck worker would
hold shutdown forever; now the process logs and exits after 30s.
download_to_temp buffered the full bytes, wrote them to a temp file, and
prepare_photo then read the whole file back from disk. The bytes are
already in memory — pass them through (download_to_temp now returns
(file, bytes)) so photo processing never touches the disk for input.
Adds the bytes dependency to xmedia-bot (already in the lock via x-media).
download_media_limited buffered the whole frame zip (cap 512 MB) in
memory before extraction, spiking RAM for large ugoira. New
site::download_media_to_file streams chunks straight to a temp file with
the same Content-Length / stream cap checks, and ugoira_video now opens
the zip from disk inside spawn_blocking. Adds FetchError::Io for local
write failures (hand-rolled error pattern preserved).
The download-and-reupload fallback downloaded each batch item serially,
so a 9-item batch took 9× the slowest download. Items are now prepared
concurrently (bounded to 3 in-flight downloads + photo processing) via a
JoinSet, then the group is uploaded in its original order; the per-item
logic moved into prepare_upload_item. Temp files stay alive until the
group request completes. A failing item still aborts the batch (the
JoinSet drop cancels the remaining prep tasks, as before).
A long tweet text or a pixiv artwork with many tags can exceed Telegram's
1024-char caption cap for HTML parse mode, turning an otherwise fine send
into a permanent 400. truncate_caption() cuts at a char boundary (never
splitting a multi-byte rune or an HTML entity like &) and appends an
ellipsis. Applied in caption_with / caption_from_fields, the link-cache
re-send path and the inline-query captions.
Telegram's sendMediaGroup requires the first item to be a photo when a
group mixes photos and videos; the previous code kept the source-site
order, so a mixed post with a video first (twitter media order is not
guaranteed) would 400 permanently. photos_first() stable-sorts photos
ahead of videos/animations before chunking; within-kind order is kept.
Every DB operation (queue lease/enqueue, chat_state get/set, link_cache
read/write) used to open a fresh connection — including the busy timeout
and WAL pragma — then close it, on every message, URL job and callback.
Replace with DbPool: a tiny pool (4 connections max, semaphore-bounded
concurrency for backpressure) whose with_conn() method runs the closure on
a pooled connection inside spawn_blocking. Steady-state cost of an
operation is a list pop + semaphore acquire instead of a connection open.
The docker workflow only builds/pushes; tests were a local responsibility.
Add .github/workflows/ci.yml with two layers:
- test: cargo fmt --check + cargo clippy --workspace --all-targets -D
warnings + cargo test --workspace (fully offline, no secrets) on every
push/PR, including forks.
- live: the #[ignore]d live-network tests plus the pixiv token-gated
tests, run on schedule / manual dispatch / tag pushes only (fork PRs
cannot read repository secrets), with PIXIV_REFRESH_TOKEN /
TWITTER_AUTH_TOKEN injected and continue-on-error for flaky sites.
Test gating (documented in AGENTS.md):
- live-network tests now carry #[ignore = "live network: ..."] (twitter 3,
bsky 2, pixiv bogus-token 1) and run via -- --ignored live.
- pixiv token tests early-return when PIXIV_REFRESH_TOKEN is absent or
empty (an unset GitHub secret arrives as ""); test_fetch previously
failed without a token.
Also fixes the three clippy assertions_on_constants warnings in send.rs
(required for -D warnings).
A task whose media is a locally produced file (pixiv ugoira MP4, bsky HLS
remux MP4) references a path inside a tempfile TempDir owned by Fetched.
The retry ran after that TempDir was dropped, so the file was already gone
and the retry always failed permanently ("local media file missing") — and
the upload fallback even tried to GET the local path as a URL.
Keep the temp dirs in a process-wide registry (Fetched::take_keep_alive ->
send::KEEP_ALIVE) that is only released when the task settles (sent or
permanently failed); the upload fallback now uploads local files directly
instead of attempting to download them.
Restart-mid-queue still loses the files (documented behavior in
input_file_for) — only the in-process retry path is fixed here.
Telegram fires an inline query on every keystroke and every prefix of a
pasted URL (status/12, status/123, ...) matches the site patterns, so
typing used to trigger a full 3-attempt fetch per keystroke. Answer only
after the query has been stable for 800ms, dedupe repeats through
Telegram's inline cache (explicit cache_time 300), and let a repeat of a
query that produced no answer retry the fetch.
OpenSSL came from two removable defaults: teloxide's `default` feature
(native-tls) and x-media's reqwest default features (default-tls). Switch
both to rustls (webpki-roots) so the binary links no system TLS libs:
- teloxide: default-features=false + rustls + ctrlc_handler (was part of
the removed default)
- reqwest (x-media): default-features=false + rustls-tls
Verified: openssl-sys/native-tls gone from the tree, cargo check clean,
all 5 live twitter/bsky fetches pass over rustls, full container startup
works. The runtime image now needs no libssl.so.3/libcrypto.so.3 or CA
bundle (ffmpeg only processes local files; all downloads go through
reqwest), saving ~8MB.
c40b074 broke the image two ways:
- `cargo clean -p` removes 0 files, so the real sources (host mtimes
older than the step-1 stub build) were never recompiled and the image
shipped the 337KB fn-main stub, exiting 0 on start. Restore the
touch-based rebuild, which forces cargo to see every .rs as newer.
- bookworm-slim does not ship libssl3 despite the old comment; the bot
links OpenSSL via teloxide/reqwest native-tls, so restore the
libssl/libcrypto copies from the builder (same Debian release).
- Replace the mtime-touch stub-rebuild hack with cargo clean -p (a
future-dated host file could silently ship the stub binary)
- Optional FFMPEG_SHA256 build arg verified before extraction
- Drop the redundant libssl3/libcrypto copies and the root
supplementary group; strip the release binary; add a webhook-mode
healthcheck to the compose example
x-media's reqwest 0.13 dragged in quinn/rustls/aws-lc-rs (cmake C
build) alongside teloxide's 0.12; unifying on 0.12 removes the whole
second TLS/QUIC stack from the build and image. The unused regex dep in
xmedia-bot is gone; x-media keeps url (the bsky HLS remux uses it).
The lease/earliest_run_after queries full-scanned tasks, and the
rollback journal blocked readers behind worker writes. journal_mode=WAL
(persistent, idempotent) plus idx_tasks_pending(status, run_after)
covers both without a schema migration.
handle_task and dead_letter_notify built a fresh Bot (env parse + HTTP
client) per queue item. A single LazyLock<Bot> is forced at startup so
a missing TELOXIDE_TOKEN fails fast instead of on the first task.
Post URLs and author URLs were interpolated raw into <a href> attributes
(and the raw user URL from empty_fetched into caption text), so crafted
links could break the HTML parse and fail the send with a 400. All
attribute interpolations now use encode_double_quoted_attribute; text
stays encode_text.
Inline queries fire per keystroke and each fetch runs the 3-attempt
retry loop; a user typing any text was pumping requests into
X/Pixiv/BSky and risking rate-limit bans. Queries now pass only if
cache_key recognizes them as a supported post URL.
media_from_file/media_from_url never attached the thumbnail, so any
video that tripped the fallback lost its cover frame. Both now take
item.thumbnail_url() and apply it, matching build_media_group.
Exact-string dedup let https://x.com/u/status/1 and
https://x.com/u/status/1/photo/1 (or the same link in text and caption)
through twice, causing duplicate fetches and sends. Dedup now uses
cache_key, falling back to the raw URL for unsupported links.
prune_expired only shrank edit_message maps, so the cache kept one
ChatData per chat forever (a leak proportional to chat count). Chats
without live edit records are now dropped from the cache and their
per-chat lock (DB row persists; get() reloads). Lock order kept safe:
prune never holds the cache lock while taking the per-chat locks.
The 8-permit semaphore was acquired inside the spawned task, so a burst
queued unlimited tasks (each cloning Bot+Message) and nothing tracked
them at shutdown — in-flight sends fired after the stop notice. URL work
now flows through a 256-slot mpsc drained by 8 workers started from
main; a full channel backpressures the per-chat handler, and shutdown
sets URL_STOP so workers stop pulling.
Concurrent handler tasks (the batch-forward design spawns several per
chat) each snapshotted the same ChatData and last-writer-wins silently
dropped mutations — e.g. a second edit_message record, leaving one
prompt's Forward button dead. All write cycles now run under a per-chat
async lock; read-only callers keep get().
A retried ugoira/bsky temp MP4 was already deleted with its TempDir, so
the retry failed at multipart-build time with a confusing Io error and
wasted all three attempts. input_file_for now rejects a missing local
path up front as a clean permanent error.
A resumed SendMediaSequence (batch_index>0 or already-sent ids) ran
post_send_actions again, opening a second edit prompt and inserting a
second edit_message record for the same messages — both Forward buttons
worked, enabling double forwards. Resumes now skip it.
A lease error (e.g. persistent SQLITE_BUSY) while rows are due made the
worker spin with sleep(0), hammering SQLite and flooding the log.
lease_next now returns the error and the loop sleeps 1s before retrying.
notify_waiters drops the notification when every worker is between its
DB reads and registering notified(); a task enqueued in that window sat
until a stale timer fired. notify_one stores a permit, so the next
worker to wait wakes immediately and re-leases. stop() still wakes all
workers with notify_waiters.
Every network retry hard-coded retry_delay_seconds(0), so backoff was
flat at 1.2-1.8s regardless of attempt; a multi-minute outage dead-
lettered after three rapid tries. The queue now scales the handler's
delay by 2^attempts (cap 300s) before rescheduling.
The shared HTTP client ignored the proxy the Bot API uses, so on
proxy-required networks (e.g. behind the GFW) every site fetch failed
while the bot itself worked. Explicit proxy overrides reqwest's system
detection; unset keeps direct connections.
YoursFunny