Compare commits

...
12 Commits
Author SHA1 Message Date
YoursFunny fca3f4ae96 ci: gate the docker test run on should-build again
The previous change ran the gate on every non-PR push; but a release
pushes branch and tag together, so docker.yml fires two runs for the one
commit and both gated the suite — the duplication that change set out to
remove. Require should-build: on a release the master run is build=false
(the commit is already tagged), so only the tag run tests.
2026-10-14 23:42:43 +08:00
YoursFunny 0d06d4d31b fix(deps): bump rustls to 0.23.46 for RUSTSEC-2026-0346..0349
Four advisories dated 2026-10-14 hit rustls 0.23.45 (a brotli
certificate-decompression allocation, a ServerHello key-exchange
downgrade, and two more), all fixed in 0.23.46, and cargo audit fails the
CI gate on them. Only rustls changes in the lock (version + checksum).
The remaining audit findings (dotenv/proc-macro-error2 unmaintained,
yoke-derive yanked) are warnings the action leaves non-fatal.
2026-10-14 23:42:32 +08:00
YoursFunny dfb2ccae3e ci: run the offline suite once per commit
ci.yml also triggered on push to master/tags, so every such push ran the
suite twice: once standalone, once as docker.yml's publish gate. Drop the
push trigger; the gate becomes the only master/tag run, and docker waits
on it — one suite run per commit, and a red suite still blocks the image.

The gate runs on every non-PR push (its own `changes` job skips a
docs-only diff), so a push touching no build input is still tested. A
release tag runs the live-network suite through the gate, via an explicit
`live` input computed by the caller (the called workflow's github.ref is
not a dependable tag check).
2026-10-14 23:17:16 +08:00
YoursFunny 3b955cf321 fix(queue): gate post-send side effects on the row's lease
A handler whose row was re-leased while it sat between heartbeats (a
>=120 s stall) still ran the channel forward, opened the edit prompt and
released the keep-alive temp media the new holder was using; only the
terminal write was token-guarded, which left the side effects to run
twice. The lease is now exposed to the handler on a task-local
(Lease/lease_still_held) and the completion path re-reads it, since the
loss may have gone unnoticed by the heartbeat. The forward retry is gated
the same way.
2026-10-14 22:10:36 +08:00
YoursFunny 632cbb7c89 fix(callback): settle the prompt when a forward fails partway
A Permanent forward failure kept the prompt live, so the next Confirm
rebuilt the task with forward_offset 0 and copied the batches that had
already reached the channel a second time. A failure whose task offset is
non-zero now settles the prompt (as the retryable path does); a failure
that copied nothing still leaves it live for a retry.
2026-10-14 22:10:36 +08:00
YoursFunny 2ffe647939 fix(settings): drop a link send whose chat settings could not be read
The pipeline read settings through get(), which degrades a failed read to
defaults. A send then went out with the chat's channel forward and
edit-before-forward silently ignored, and the task it built carried those
defaults into the retry payload. try_get() is the strict read; an
unreadable row now drops the send and tells the user.
2026-10-14 22:10:36 +08:00
YoursFunny 3a944c1361 fix(queue): keep a running attempt through a failed lease heartbeat
A heartbeat write error abandoned the attempt at once. The lease is still
valid until locked_until, so the attempt was dropped for nothing: the row
was recovered later and the task re-ran from the start, although its side
effects (a sent album, a copied post) may already have happened. A failed
beat now retries at the next tick; the attempt is abandoned only once the
next beat could land at or past the last confirmed expiry.
2026-10-14 22:10:35 +08:00
YoursFunny 7fbe144b57 fix(ci): gate docker publish on the offline test suite
docker.yml pushed the image for tags and master without waiting for the
test job, so a red commit could still be published. ci.yml is now also a
workflow_call target; docker depends on its result. The gate run has its
own concurrency group and never cancels a superseded run, so a docs-only
follow-up cannot drop the image of the code commit before it. The live
job is skipped for the gate call.
2026-10-14 22:10:35 +08:00
YoursFunny a7d4d1af8c fix: derive the photo decode cap so one photo's peak fits the memory budget 2026-10-14 22:10:35 +08:00
YoursFunny 51f451d78f fix: default compose WEBHOOK to false and leave RUST_LOG unset 2026-10-14 22:10:35 +08:00
YoursFunny c937a66e5f fix: admit adapter temp media in the local-path guard 2026-10-14 22:10:34 +08:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> b42d41eaa4 build(deps): bump the cargo-patch group across 1 directory with 2 updates (#5)
Bumps the cargo-patch group with 2 updates in the / directory: [thiserror](https://github.com/dtolnay/thiserror) and [rand](https://github.com/rust-random/rand).


Updates `thiserror` from 2.0.20 to 2.0.21
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.20...2.0.21)

Updates `rand` from 0.10.2 to 0.10.3
- [Release notes](https://github.com/rust-random/rand/releases)
- [Changelog](https://github.com/rust-random/rand/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-random/rand/compare/0.10.2...0.10.3)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-version: 2.0.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: cargo-patch
- dependency-name: rand
  dependency-version: 0.10.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: cargo-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-05 18:57:46 +08:00
14 changed files with 601 additions and 93 deletions
+1 -1
View File
@@ -25,7 +25,7 @@ BILIBILI_COOKIE=
# Admin chat IDs, comma-separated: start/stop notices, admin-only commands.
BOT_ADMIN=
# Log level. Leave the line commented out for the default
# (`info,hyper_util=warn,reqwest=warn`); do not set it to an empty value.
# (`info,hyper_util=warn,reqwest=warn`); an empty value behaves the same.
# RUST_LOG=info,xmedia_bot=debug,x_media=debug
# Edit-before-forward record TTL (seconds).
EDIT_MESSAGE_TTL_SECONDS=86400
+32 -15
View File
@@ -1,19 +1,22 @@
name: CI
# Test/lint gate (offline, no secrets) on every push/PR, plus a live-network
# job that exercises the real source sites and the token-gated pixiv tests.
# Test/lint gate (offline, no secrets). Standalone on pull requests, schedule
# and manual dispatch; a master/tag push does not trigger it directly — that
# push is tested once, by docker.yml's `ci` gate, which calls this workflow
# with `gate: true` and blocks the image on the result.
#
# Layering:
# changes — decides whether anything but documentation changed; a docs-only
# push/PR skips `test` (which then reports as skipped, not missing).
# diff skips `test` (which then reports as skipped, not missing).
# test — fmt + clippy + the full offline unit suite + a release-profile
# build + cargo-audit dependency gate. Runs on every push and PR,
# including forks (it needs no secrets).
# build + cargo-audit dependency gate. Runs on PRs, and for a
# master/tag push under the docker gate; no secrets needed.
# live — the #[ignore]d live-network tests plus the pixiv tests that are
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch
# / tag pushes only, because pull requests from forks cannot read
# repository secrets. continue-on-error keeps a flaky external site
# from blocking, while the run still records the outcome.
# gated on PIXIV_REFRESH_TOKEN. Runs on schedule / manual dispatch,
# and on a release tag (through the docker gate), because pull
# requests from forks cannot read repository secrets.
# continue-on-error keeps a flaky external site from blocking,
# while the run still records the outcome.
#
# Every action is pinned to a commit SHA (Dependabot keeps the pins current);
# `dtolnay/rust-toolchain` deliberately stays on its channel ref, because the
@@ -26,14 +29,28 @@ name: CI
# PIXIV_REFRESH_TOKEN is absent or empty (empty = unset CI secret).
on:
push:
branches: [master]
tags: ['v*']
pull_request:
schedule:
# Weekly probe of the live endpoints, so external API changes surface.
- cron: '0 3 * * 1'
workflow_dispatch:
# Called by docker.yml as the publish gate. `gate` marks that call: its
# concurrency group is separate from the standalone run on the same ref, and
# it never cancels a superseded run, because each gate run carries its own
# commit's image (a docs-only follow-up must not drop the code commit's).
workflow_call:
inputs:
gate:
type: boolean
default: false
# Passed by docker.yml for a release tag: the caller computes it from its
# own `github.ref`, so the called workflow needs no ref check of its own.
# A string (not boolean) on purpose: a boolean input fed by an expression
# arrives as the string "true"/"false", and "false" is truthy in `if`.
live:
description: Run the live-network suite too (docker passes "true" on a release tag).
type: string
default: 'false'
permissions:
contents: read
@@ -41,8 +58,8 @@ permissions:
# A newer push to the same ref supersedes the older run; without this every
# intermediate commit of a PR branch keeps a runner busy to completion.
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
group: ci-${{ inputs.gate && 'gate-' || '' }}${{ github.ref }}
cancel-in-progress: ${{ !inputs.gate }}
env:
# Panicking tests print their backtrace; free when nothing fails.
@@ -134,7 +151,7 @@ jobs:
live:
needs: test
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || startsWith(github.ref, 'refs/tags/v')
if: ${{ inputs.live == 'true' || (!inputs.gate && (github.event_name == 'schedule' || github.event_name == 'workflow_dispatch')) }}
runs-on: ubuntu-latest
timeout-minutes: 30
continue-on-error: true
+22 -2
View File
@@ -111,14 +111,34 @@ jobs:
fi
echo "build=true" >> "$GITHUB_OUTPUT"
# The publish gate: the offline suite CI runs (fmt, clippy, tests, release
# build, audit), reused rather than copied, so no image is pushed for a
# commit whose tests are red. Pull requests skip it: their own CI run covers
# the code, and the docker job there only proves the image still builds.
ci:
needs: should-build
# The suite's only master/tag run (`ci.yml` has no push trigger). Gated on
# `should-build` so a release push — which fires a master *and* a tag run for
# one commit — tests once: the master run's `should-build` is false (already
# tagged), so only the tag run gates.
if: needs.should-build.outputs.build == 'true' && github.event_name != 'pull_request'
uses: ./.github/workflows/ci.yml
with:
gate: true
# A release tag also runs the live-network suite: a tag push is trusted,
# so the fork-PR reason to skip it (no secrets) does not apply.
live: ${{ startsWith(github.ref, 'refs/tags/v') && 'true' || 'false' }}
# No `actions/checkout` here on purpose: `docker/build-push-action` defaults
# to the Git context (`https://github.com/<owner>/<repo>.git#<ref>`), so
# BuildKit clones the repo itself and authenticates with the automatic
# github.token. Adding `context: .` below without a checkout step would hand
# BuildKit an empty workspace.
docker:
needs: should-build
if: needs.should-build.outputs.build == 'true'
needs: [should-build, ci]
# `always()`: a skipped `ci` (pull requests) must not skip this job; only a
# failed or cancelled gate does.
if: always() && needs.should-build.outputs.build == 'true' && (github.event_name == 'pull_request' || needs.ci.result == 'success')
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
+6 -6
View File
@@ -41,10 +41,10 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, command registration (`register_commands` — `setMyCommands` plus the profile description texts), shared `send::BOT` force-init, startup sweep of this project's leftover temp files (`x_media::TEMP_FILE_PREFIX` + an age gate, since a killed process runs no destructors), startup repair of queued retries whose local media did not survive a restart (`handlers::repair_lost_local_media`, before any worker can lease: those rows are re-fetched from their `source_url`), queue worker start, site login validation (`site::validate_all`), `periodic_sweep` (`SWEEP_INTERVAL` 300 s): expired prompts are rewritten in place to `EDIT_PROMPT_EXPIRED_TEXT` with an empty keyboard — an edit, never a new message, so a background timer cannot wake a chat — plus the link-cache prune, the idle rate-limit buckets and the idle inline-query entries, and the queue backlog line (only when non-empty). Takes its collaborators rather than the statics so its loop is testable with a paused clock, dptree handler tree, webhook vs polling dispatch |
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
| `crates/xmedia-bot/src/db.rs` | `DbPool`: one shared SQLite connection pool (`POOL_SIZE = 16`, WAL, busy_timeout) for all three tables over `$DATA_DIR/task_queue.db` (default `data/`) — the three stores share it; `open_store` creates file + schema and then applies the `PRAGMA user_version` migration chain (`MIGRATIONS` + `migrate` — append-only; `schema_init` is the version-0 baseline and must not gain columns an existing database would never receive — `db.rs`'s tests pin a pre-migration database upgrading intact, the shipped migration text frozen (appending is the only allowed change) and a fresh database landing at the latest version), `with_conn` runs all rusqlite I/O in `spawn_blocking` |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `url_workers.rs`/`repair.rs` (worker pool; startup repair), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + the per-URL pipeline; one *shared* in-flight fetch per cache key (`shared_fetch`+`IN_FLIGHT_FETCHES`: a second chat, a batch forward or a retry asking for the same post meanwhile waits for the first caller's result, the entry is dropped the moment the fetch settles so nothing is ever answered from an old fetch, and a waiter whose sharer was cancelled fetches for itself); plus the startup repair `repair_lost_local_media`, whose decision (`needs_refetch`) and rewrite (`apply_refresh`) are pure and tested while the fetch itself is a live test), `url_workers.rs` (the bounded job channel (256) and its `URL_WORKERS = 8` supervised workers, `start_url_workers`/`stop_url_workers` — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential, so batch-forwards need this concurrency), `repair.rs` (startup `repair_lost_local_media` with its `needs_refetch`/`apply_refresh`/`refetch`), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table); the 300 s sweep's `prune_expired` evicts any chat with no live edit-before-forward prompt, so the cache (and the per-chat lock map) stays bounded to active prompts — durable settings reload from the DB on next use |
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `url_workers.rs`/`repair.rs` (worker pool; startup repair), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + the per-URL pipeline; one *shared* in-flight fetch per cache key (`shared_fetch`+`IN_FLIGHT_FETCHES`: a second chat, a batch forward or a retry asking for the same post meanwhile waits for the first caller's result, the entry is dropped the moment the fetch settles so nothing is ever answered from an old fetch, and a waiter whose sharer was cancelled fetches for itself); plus the startup repair `repair_lost_local_media`, whose decision (`needs_refetch`) and rewrite (`apply_refresh`) are pure and tested while the fetch itself is a live test), `url_workers.rs` (the bounded job channel (256) and its `URL_WORKERS = 8` supervised workers, `start_url_workers`/`stop_url_workers` — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential, so batch-forwards need this concurrency), `repair.rs` (startup `repair_lost_local_media` with its `needs_refetch`/`apply_refresh`/`refetch`), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered; a Permanent failure that already copied a batch settles the prompt too, since a second Confirm would rebuild the task from offset 0 and re-copy those messages, while one that copied nothing keeps it live for a retry), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table); the 300 s sweep's `prune_expired` evicts any chat with no live edit-before-forward prompt, so the cache (and the per-chat lock map) stays bounded to active prompts — durable settings reload from the DB on next use; `get` degrades a failed read to defaults for display/control flow, while `try_get` is the strict read the send pipeline uses — an unreadable row drops the send (and tells the user) instead of silently ignoring the chat's forward settings and persisting those defaults into the retry payload |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + the source media URLs + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune; a permanent send failure *degrades* the entry instead of dropping it (the file ids go, the URLs stay, so the next request re-sends from those without a fetch), and a degraded entry that fails again is removed |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, a `lease_token` fence: `lease_next` stamps a random token and every write-back (heartbeat, `delete`, `reschedule`, `mark_done`) is guarded by it, so a lease that expired and was re-leased cannot be written by its former holder — a lost lease stops the attempt instead; a finished row's `DELETE`/reschedule retried and a failed delete falling back to a `done` tombstone (the lease query and the sweep only look at `pending`/`in_progress`, so a task that already ran cannot be resurrected and re-run), `runnable_rows`/`replace_payload` (the startup repair's read/rewrite path: it runs before the workers exist, which is why it needs no lease token), `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit; the sweep does notify the workers after it actually recovered a row, since a recovered task is due immediately while every worker may be parked on `notify` with no pending row to sleep on), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, a `lease_token` fence: `lease_next` stamps a random token and every write-back (heartbeat, `delete`, `reschedule`, `mark_done`) is guarded by it, so a lease that expired and was re-leased cannot be written by its former holder — a lost lease stops the attempt instead (and the handler verifies ownership with a fresh read — `Lease`/`lease_still_held`, carried on a task-local — before running side effects a re-lease would duplicate: a channel forward, an edit prompt, the keep-alive release); a finished row's `DELETE`/reschedule retried and a failed delete falling back to a `done` tombstone (the lease query and the sweep only look at `pending`/`in_progress`, so a task that already ran cannot be resurrected and re-run), `runnable_rows`/`replace_payload` (the startup repair's read/rewrite path: it runs before the workers exist, which is why it needs no lease token), `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit; the sweep does notify the workers after it actually recovered a row, since a recovered task is due immediately while every worker may be parked on `notify` with no pending row to sleep on), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/ctx.rs` | `AppContext`: the injected collaborators (`sender` + `ChatStore`/`PersistentTaskQueue`/`LinkCache`/`Config`), `from_statics` for production and the `CONTEXT` static the worker closures hold. `test_support::TestStores` backs handler tests with a tempdir store set, and the module also carries the fixtures those tests share — the canonical cached post (`cached_photo`), the edit-before-forward prompt (`seed_prompt` with its `PROMPT_ID`/`FORWARDED_ID`) and a scripted API error (`api_error`) — so no two test modules keep their own copies |
| `crates/xmedia-bot/src/send/` | `send/mod.rs`: `Task`/`MediaItemPayload` payloads (`media: MediaRef`, i.e. `Source` URL-or-path vs `FileId` — one field used to carry both with a flag), `send_media_sequence`/`send_animation`/`forward_messages`; `send/error.rs`: the Bot API error policy (`SendError`/`Classification`, `classify_request_error`, the media-fetch/size markers); `send/input_media.rs`: payload → `InputFile`/`InputMedia` + `build_media_group` (caption on the first item only); `send/upload.rs`: the download-and-reupload fallback (`prepare_upload_item`/`send_batch_via_upload`, photo downscale handoff); `send/post_send.rs`: link-cache write, `KEEP_ALIVE` registry, `settle_task`, `post_send_actions`, `handle_task`/`dead_letter_notify` |
| `crates/xmedia-bot/src/media_sender/{mod.rs,test_support.rs}` | `MediaSender` trait: the user-flow surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`answer_callback_query`/`edit_message_text`/`edit_message_caption`/`delete_message`/`send_chat_action`/`answer_inline_query`) implemented by teloxide `Bot` (per-chat rate-limited) and by a recording `MockSender` in tests. Admin/setup APIs (`get_chat`, `set_my_commands`, …) stay on the concrete `Bot`. `test_support.rs` (cfg(test)-only) holds the scripted `MockSender` and `fake_api` (the stand-in API the real-`Bot` tests drive) |
@@ -83,13 +83,13 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
| `crates/xmedia-bot/src/handlers/` | `statics.rs` = `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `$DATA_DIR/task_queue.db`, default `data/` **relative to CWD**, dir auto-created); `mod.rs` also holds `apply_caption_edit`, the one place a caption edit is applied and its failure classified: a short retryable delay is retried once, anything else is reported to the user instead of being swallowed (`callback.rs`'s template button answers its toast with the failure and leaves the record alone); `commands.rs` = command dispatch (incl. `/test <url>` send-only, `/debug <url>` parse-only, the read-only `/settings` every chat member can read — unlike the admin-only `/bot_dict` raw dump — and template removal; `/start`/`/help` carry the guidance teloxide's `descriptions()` cannot render, and `/set_format` rejects unknown `{…}` placeholders, resetting with `-`; `/set_template` enforces 50 templates per chat, 55 UTF-8-byte names, and 1024-character escaped bodies; `/settings` output is capped at 4000 characters); `urls.rs` = URL extraction + the per-URL pipeline (`url_media` takes a `PostSend` mode: chat settings vs `/test`'s suppressed actions); `inline.rs` = debounced inline queries (hotlink-protected and local media skipped); `callback.rs` = edit-before-forward buttons (dptree entry + testable `handle_callback` core, incl. `skip`) |
| `crates/xmedia-bot/src/send/` | `mod.rs`: constants `MAX_MEDIA_GROUP = 10` and the senders; `error.rs`: `classify_request_error` (5xx/non-JSON bodies retry, see the Retries bullet) and the media-fetch markers that route a URL send into the reupload fallback — including `failed to get HTTP url content`, the description single-media URL sends answer with. `upload.rs`: download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`), the per-kind upload cap (`MAX_MEDIA_UPLOAD_BYTES` = 50 MB for video/animation/other, Telegram's multipart limit; `photo::MAX_UPLOAD_BYTES` stays the 10 MiB photo one) with a download's class from `classify_download_error` (transport/429/5xx retry; 4xx is permanent — the media itself is gone or refused — and a temp-file *write* failure retries, being resource exhaustion far more often than a broken temp dir). Item preparation is bounded **process-wide** (`PREP_SLOTS` in `upload.rs`: URL workers and queue workers can each be inside a batch, so a per-batch bound is not a memory bound), and the check that routes an oversized item to `fallback_url` is the download's own declared-Content-Length abort (`FetchError::TooLarge` → `MediaTooLarge`) — there is no separate size probe, which used to cost a second request per item. `post_send.rs`: settlement (`settle_task`), cache write, post-send actions (dead-letter text via `failure_text`: post key + cause, since the raw error alone does not say which link died), queue handlers. `input_media.rs`: payload → `InputMedia` |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL. Two budgets, not one: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) caps the *download* in the send fallback — the whole body is buffered, once per prep slot — while `MAX_DECODE_BYTES` (512 MiB) stays the pre-allocation guard that decides whether a decoded photo can be processed at all; over either one the item degrades to its smaller URL |
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL. Two budgets, not one: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) caps the *download* in the send fallback — the whole body is buffered, once per prep slot — while `MAX_DECODE_BYTES` (188 MiB, derived so one photo's worst-case peak of download + 2.5 × decode + upload cap fits the 512 MiB memory budget) stays the pre-allocation guard that decides whether a decoded photo can be processed at all; over either one the item degrades to its smaller URL |
| `crates/x-media/src/site/{mod,download}.rs` | `mod.rs`: dispatcher, `Fetched`/`FetchError`, `needs_media_headers` (the per-site rule, asked by the inline path to skip what Telegram cannot fetch). `download.rs`: the media-download stack — the metadata vs. media HTTP clients, the host-network guard (applied to the start URL and every redirect hop) and `download_media_limited`/`download_media_to_file` (which add the site's headers, e.g. `Referer: https://www.pixiv.net/` for `pximg.net`) |
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) |
|`docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim); rejects a non-numeric or `0` `LOCAL_USER_ID`, which would otherwise survive the drop and run the bot root |
| `docker-compose.yml` | The deployment composition, committed as-is: every instance value (token, admins, site credentials, domain) is a `${VAR}` substitution read from the gitignored `.env` beside it, so the file needs no per-deployment edit — and a variable not listed in a service's `environment:` never reaches that container. Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master, plus a build-only check on PRs touching the build inputs; **no test step**; verifies a release tag matches both crate versions; buildx gha cache (`cache-from` always, `cache-to` except on PRs, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs; `FFMPEG_URL`/`FFMPEG_SHA256` come from repo variables when set |
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master, plus a build-only check on PRs touching the build inputs; the push is gated on `ci.yml`'s offline suite (reused via `workflow_call` with `gate: true`; `ci.yml` has no `push` trigger, so this gate is where a master/tag push is tested — and it is itself gated on `should-build`, so a release (whose branch and tag push fire two docker runs) still tests once); verifies a release tag matches both crate versions; buildx gha cache (`cache-from` always, `cache-to` except on PRs, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs; `FFMPEG_URL`/`FFMPEG_SHA256` come from repo variables when set |
| `README.md` | Feature docs + command table (Chinese) |
## Runtime/Tooling Preferences
@@ -109,6 +109,6 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
- No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via `tempfile` (`queue.rs::new_queue()` helper), live network fetches. Tests that must go through a **real `Bot`** (its URL/multipart building, the per-chat limiter and the bot-wide budget) talk to a stand-in API instead (`media_sender::test_support::fake_api::FakeApi`, a `tokio` TCP listener that records every call and answers the smallest result each method needs — teloxide keys methods by payload type, so the recorded name is `SendMediaGroup`, not `sendMediaGroup`): a media group, the edit-before-forward prompt through the real callback path, and `handlers::handle_message` (the context-taking body of `message_handler`, split out for exactly this).
- Live-network tests exist in `site/twitter/interface.rs` (5), `site/bsky/interface.rs` (1), `site/misskey/interface.rs` (1), `site/bilibili/interface.rs` (5), `site/pixiv/api.rs` (1) and `site/download.rs` (1: the pixiv download below); the bot crate adds one live test each in `handlers/repair.rs` and `handlers/urls.rs`; `photo.rs` adds one `#[ignore = "heavy: …"]` test. `site/download.rs`'s pixiv download test (`live_download_media_pixiv_original_with_referer`) is gated **both ways** — `#[ignore = "live network: …"]` *and* an early return without `PIXIV_REFRESH_TOKEN` — so a local `cargo test --workspace` stays fully offline and the pixiv CDN flake surfaces only in the live job. `disabled_site_is_reported_not_ignored` (same file) is gated the other way round: it asserts `fetch` answers `FetchError::Disabled { site: "pixiv" }` for a pixiv link and early-returns when `PIXIV_REFRESH_TOKEN` **is** set (the site is then enabled). Test gating convention (enforced by `.github/workflows/ci.yml`): pure unit tests always run; live-network tests carry `#[ignore = "live network: ..."]` (run via `cargo test --workspace -- --ignored live`); token-gated pixiv tests early-return when `PIXIV_REFRESH_TOKEN` is absent **or empty** (an unset GitHub secret arrives as `""` — `is_err()` alone would run them tokenless and fail), and the bilibili live tests early-return when the API answers risk control (`-352`, which bilibili applies per IP by request volume). Run the full offline suite with `cargo test --workspace`.
- Fixtures are inline `serde_json::json!` builder fns (`fixture()`, `thread_json()`, `illust_json()`), not files. The shared `CLIENT` sets `pool_max_idle_per_host(0)` under `#[cfg(test)]` to avoid cross-runtime `DispatchGone`.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels superseded runs, `RUST_BACKTRACE=1`) runs (behind a `changes` gate job, so a push/PR whose entire diff is markdown skips it instead of burning four minutes on nothing) `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets, on every push/PR) and a `live` job (schedule/manual/tag only, `continue-on-error`) that runs the `#[ignore]`d `live` tests in **both** crates — `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN`/`BILIBILI_COOKIE` pass through as secrets, and a summary step lists every `SKIP` a test printed so a green live run cannot mean zero coverage. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs**; its `should-build` gate skips a branch push that is already tagged (`git tag --points-at` — the tag run builds it, so push both refs together) or that touched no build input at all, while a tag push always builds (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- **CI** — `.github/workflows/ci.yml` (actions pinned to commit SHAs, `--locked` on every cargo invocation, `concurrency` cancels a superseded standalone run while a gate run never cancels itself, `RUST_BACKTRACE=1`) runs (behind a `changes` gate job, so a PR — or a gated push — whose entire diff is markdown skips it instead of burning four minutes on nothing) `cargo fmt --check` + `cargo clippy --workspace --all-targets --locked -- -D warnings` + `cargo test --workspace --locked` + a release-profile `cargo build --release --locked` + an `actions-rust-lang/audit` dependency-vulnerability gate (offline, no secrets) and a `live` job (schedule/manual, plus a release tag through the docker gate; `continue-on-error`) that runs the `#[ignore]`d `live` tests in **both** crates — `PIXIV_REFRESH_TOKEN`/`TWITTER_AUTH_TOKEN`/`BILIBILI_COOKIE` pass through as secrets, and a summary step lists every `SKIP` a test printed so a green live run cannot mean zero coverage. `ci.yml` has **no `push` trigger**: a master/tag push is tested once, by `docker.yml`'s `ci` job (`workflow_call`, `gate: true`), which the image build then waits on — one suite run per commit, and a red suite blocks the push. `.github/workflows/docker.yml` builds and pushes the image on master/tag and runs a **build-only check on pull requests touching the build inputs**; its `should-build` gate skips a branch push that is already tagged (`git tag --points-at` — the tag run builds it, so push both refs together) or that touched no build input at all, while a tag push always builds (`Dockerfile`, entrypoint, manifests, `.dockerignore`); a release tag must match both crate versions or the build stops, and `FFMPEG_URL`/`FFMPEG_SHA256` are taken from repository variables when set (a release can pin an exact ffmpeg build). `.github/dependabot.yml` keeps crates, the pinned actions and the Docker base images current.
- Untested and hard to test without a mock seam: `handlers/statics.rs`; `config.rs` only partly (the webhook truth table, TTL fallback and blank-secret parsing are pinned, the rest of the env parsing is not); `db.rs` is covered for the migration chain but not for pool behaviour under contention; `main.rs` is covered where it was split out (`periodic_sweep`, `sweep_temp_dir`) but not for startup/shutdown or its `dptree` branch tree (the handlers themselves are, through the stand-in API); in `x-media`: `media.rs`, `lib.rs`, all `model.rs` (their serde shapes are exercised indirectly by the adapter fixtures that deserialize into them). Everything else — `handlers/{mod,callback,inline,urls}.rs`, `send/*`, `ctx.rs`, `state.rs`, `queue.rs`, `link_cache.rs`, `rate_limit.rs`, `commands.rs` (its executor, through scripted outcomes) — is driven through `TestStores`/`ctx::test_support` and the scripted `MockSender`.
- No coverage tracking.
Generated
+18 -18
View File
@@ -275,7 +275,7 @@ version = "3.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34"
dependencies = [
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -553,7 +553,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
dependencies = [
"libc",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -1153,7 +1153,7 @@ checksum = "3640c1c38b8e4e43584d8df18be5fc6b0aa314ce6ebf51b53313d4306cca8e46"
dependencies = [
"hermit-abi",
"libc",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -1620,7 +1620,7 @@ dependencies = [
"bytes",
"getrandom 0.4.3",
"lru-slab",
"rand 0.10.2",
"rand 0.10.3",
"rand_pcg",
"ring",
"rustc-hash",
@@ -1644,7 +1644,7 @@ dependencies = [
"once_cell",
"socket2",
"tracing",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -1675,9 +1675,9 @@ dependencies = [
[[package]]
name = "rand"
version = "0.10.2"
version = "0.10.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80"
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
dependencies = [
"chacha20",
"getrandom 0.4.3",
@@ -1892,14 +1892,14 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
name = "rustls"
version = "0.23.45"
version = "0.23.46"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0d41d731c7d2f962d1ccc364cec258de3c0e93b38c2fb3ba97ac74513048d634"
checksum = "48e13bd8c0e9365c43cfa5c9e8f9ad49d3c8444926c9aac819e0e4dc503c8fdf"
dependencies = [
"once_cell",
"ring",
@@ -2318,7 +2318,7 @@ dependencies = [
"getrandom 0.4.3",
"once_cell",
"rustix",
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -2332,18 +2332,18 @@ dependencies = [
[[package]]
name = "thiserror"
version = "2.0.20"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ec86235f5fcc2a73650310756d2ac5b138a5780bbbdfae3eeccec992c435ba4f"
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
dependencies = [
"thiserror-impl",
]
[[package]]
name = "thiserror-impl"
version = "2.0.20"
version = "2.0.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524"
dependencies = [
"proc-macro2",
"quote",
@@ -2727,7 +2727,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.52.0",
"windows-sys 0.61.2",
]
[[package]]
@@ -2885,7 +2885,7 @@ dependencies = [
"dotenv",
"html-escape",
"log",
"rand 0.10.2",
"rand 0.10.3",
"regex",
"reqwest",
"serde",
@@ -2910,7 +2910,7 @@ dependencies = [
"parking_lot",
"png",
"pretty_env_logger",
"rand 0.10.2",
"rand 0.10.3",
"rusqlite",
"serde",
"serde_json",
+20
View File
@@ -7,3 +7,23 @@ pub mod site;
/// destructors), and without a marker the only safe assumption about the OS
/// temp directory is "not mine".
pub const TEMP_FILE_PREFIX: &str = "tgxmb-";
/// True when `path` is a file inside a temp directory this project created: the
/// directory's name carries [`TEMP_FILE_PREFIX`] and it sits directly under the
/// OS temp directory (`tempfile`'s default location). The prefix lives on the
/// directory, not the file, so checking the file name is not enough; requiring
/// the parent to be the temp root stops an upstream-supplied bare path that
/// merely contains a `tgxmb-` component from passing.
pub fn is_own_temp_file(path: &std::path::Path) -> bool {
let Some(dir) = path.parent().and_then(|dir| dir.canonicalize().ok()) else {
return false;
};
let Ok(root) = std::env::temp_dir().canonicalize() else {
return false;
};
dir.parent() == Some(root.as_path())
&& dir
.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| name.starts_with(TEMP_FILE_PREFIX))
}
+99 -3
View File
@@ -142,9 +142,23 @@ async fn handle_callback(
)
}
}
Err(send::SendError::Permanent { message, .. }) => {
Err(send::SendError::Permanent { message, task }) => {
log::error!("forward failed permanently: {message}");
(format!("Forward failed: {message}"), false)
// `forward_offset` is where the failed batch began, so a
// non-zero offset means earlier messages already reached
// the channel. Keeping the prompt live there let a second
// Confirm rebuild the task from offset 0 and copy those
// messages a second time.
let partial = matches!(
*task,
Task::ForwardMessages { forward_offset, .. } if forward_offset > 0
);
let answer = if partial {
format!("Forward failed partway: {message}")
} else {
format!("Forward failed: {message}")
};
(answer, partial)
}
};
if settled {
@@ -231,7 +245,9 @@ async fn handle_callback(
#[cfg(test)]
mod tests {
use super::*;
use crate::ctx::test_support::{FORWARDED_ID, PROMPT_ID, TestStores, api_error, seed_prompt};
use crate::ctx::test_support::{
FORWARDED_ID, PROMPT_ID, TestStores, api_error, permanent_error, seed_prompt,
};
use crate::media_sender::test_support::{MockSender, Outcome};
/// The Telegram wording the mocks answer with: a chat the bot cannot reach.
@@ -430,6 +446,86 @@ mod tests {
assert_eq!(stores.queued_tasks().await, 1, "no second forward row");
}
/// A Permanent forward failure that already copied some batches must settle
/// the prompt: leaving it live rebuilt the task from offset 0 on a second
/// Confirm and copied those messages to the channel a second time.
#[tokio::test]
async fn a_partial_permanent_forward_settles_the_prompt() {
let sender = MockSender::scripted(vec![Outcome::CopyOk, Outcome::CopyErr], permanent_error);
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
// Two copy batches: the first (100) succeeds, the second fails, so the
// failed task's offset is non-zero — a partial copy reached the channel.
ctx.chat_store
.update(1, |data| {
data.edit_message
.get_mut(&PROMPT_ID)
.unwrap()
.forward_message_ids = (1..=150).collect();
})
.await
.unwrap();
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.calls(),
vec![
"copy_messages",
"copy_messages",
"delete_message",
"answer_callback_query"
]
);
let answer = sender.answers()[0].clone().unwrap();
assert!(answer.starts_with("Forward failed partway"), "{answer}");
assert!(
!ctx.chat_store
.get(1)
.await
.edit_message
.contains_key(&PROMPT_ID),
"a partial copy must close the prompt so it cannot re-copy from 0"
);
// A second Confirm finds no record: it cannot copy the first batch again.
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
let copies = sender
.calls()
.iter()
.filter(|c| **c == "copy_messages")
.count();
assert_eq!(copies, 2, "the second Confirm must not copy again");
}
/// A Permanent failure on the first batch copied nothing, so the prompt
/// stays live for the user to retry.
#[tokio::test]
async fn a_permanent_forward_that_copied_nothing_keeps_the_prompt() {
let sender = MockSender::scripted(vec![Outcome::CopyErr], permanent_error);
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
seed_prompt(&ctx, "", crate::db::unix_now()).await;
handle_callback(&ctx, callback_id(), 1, PROMPT_ID, "forward").await;
assert_eq!(
sender.calls(),
vec!["copy_messages", "answer_callback_query"]
);
let answer = sender.answers()[0].clone().unwrap();
assert!(answer.starts_with("Forward failed: "), "{answer}");
assert!(
ctx.chat_store
.get(1)
.await
.edit_message
.contains_key(&PROMPT_ID),
"nothing was copied, so the prompt may be retried"
);
}
#[tokio::test]
async fn unknown_and_expired_prompts_answer_expired() {
let sender = MockSender::scripted(vec![], || api_error(API_ERROR));
+94 -6
View File
@@ -186,10 +186,7 @@ pub(super) fn media_to_payload(media: &Media, sensitive: bool) -> Option<MediaIt
let url = media.url();
// Only adapter-produced temp files may be non-HTTP. A bare path from an
// upstream JSON field must never reach InputFile::file().
let is_local_temp = std::path::Path::new(url)
.file_name()
.and_then(|name| name.to_str())
.is_some_and(|name| name.starts_with(x_media::TEMP_FILE_PREFIX));
let is_local_temp = x_media::is_own_temp_file(std::path::Path::new(url));
if !url.starts_with("http://") && !url.starts_with("https://") && !is_local_temp {
log::warn!("dropping media with a non-URL upstream path");
return None;
@@ -546,6 +543,31 @@ fn site_title(site: &str) -> String {
}
}
/// The chat's settings, or `None` after telling the user the send was dropped
/// because they could not be read. Falling back to defaults silently dropped
/// the chat's channel forward / edit-before-forward, and wrote those defaults
/// into the retry payload the task carries.
async fn settings_or_report(
ctx: &AppContext<'_>,
chat_id: i64,
reply_to: MessageId,
) -> Option<ChatData> {
match ctx.chat_store.try_get(chat_id).await {
Ok(data) => Some(data),
Err(()) => {
log::error!("chat settings read failed for chat={chat_id}; not sending");
let _ = reply(
ctx.sender,
chat_id,
reply_to,
"Couldn't read this chat's settings. Please try again.",
)
.await;
None
}
}
}
#[allow(clippy::too_many_arguments)]
async fn url_media_inner(
ctx: &AppContext<'_>,
@@ -567,7 +589,9 @@ async fn url_media_inner(
&& let Some(cached) = ctx.link_cache.get(&key, ctx.config.link_cache_ttl).await
{
log::debug!("link cache hit for {key}");
let chat_data = ctx.chat_store.get(chat_id).await;
let Some(chat_data) = settings_or_report(ctx, chat_id, reply_to).await else {
return;
};
// Cache keys are prefixed with the site id ("twitter:…"), matching
// the value a fresh fetch would read from Fetched::site_id. The key
// came out of cache_key, so its prefix is a registered id by
@@ -636,7 +660,9 @@ async fn url_media_inner(
let _ = reply(ctx.sender, chat_id, reply_to, fetch_error_message(e)).await;
}
Ok(Some(fetched)) => {
let chat_data = ctx.chat_store.get(chat_id).await;
let Some(chat_data) = settings_or_report(ctx, chat_id, reply_to).await else {
return;
};
// Per-site caption format override (empty -> built-in caption).
let format = chat_data.format_for(fetched.site_id);
if fetched.media.is_empty() {
@@ -755,6 +781,28 @@ mod tests {
assert!(entry.media[0].file_id.is_empty());
}
/// A settings read failure must drop the send, not fall back to defaults:
/// sending anyway silently ignored the chat's channel forward and
/// edit-before-forward, and wrote those defaults into the retry payload.
#[tokio::test]
async fn an_unreadable_settings_row_drops_the_send_and_tells_the_user() {
let stores = TestStores::new();
let sender = MockSender::scripted(vec![Outcome::MessageOk], permanent_error);
let ctx = stores.ctx(&sender);
stores.link_cache().put("twitter:1", &cached_photo()).await;
// Force the read to fail (the missing-table shape `state.rs` uses).
let raw = rusqlite::Connection::open(stores.db_path()).unwrap();
raw.execute_batch("DROP TABLE chat_state").unwrap();
url_media(&ctx, 1, 2, "https://x.com/u/status/1", PostSend::FromChat).await;
assert_eq!(
sender.calls(),
vec!["send_chat_action", "send_message"],
"no media may go out without the chat's settings; the user is told instead"
);
}
/// A repeat request for a single-gif post re-sends the cached file id
/// instead of failing: the animation path used to hand the id to
/// `input_file_for`, which read it as a local path and answered "local
@@ -832,6 +880,46 @@ mod tests {
assert!(media_to_payload(&media, false).is_none());
}
/// Adapter output (bsky remux, pixiv ugoira) lives at `tgxmb-XXXX/video.mp4`
/// under the OS temp dir: the prefix is on the directory, so it must pass.
#[test]
fn media_to_payload_accepts_an_adapter_temp_video() {
let dir = tempfile::Builder::new()
.prefix(x_media::TEMP_FILE_PREFIX)
.tempdir()
.unwrap();
let path = dir.path().join("video.mp4");
std::fs::write(&path, b"mp4").unwrap();
let media = Media::Video {
url: path.to_string_lossy().into_owned(),
thumbnail_url: String::new(),
};
assert!(matches!(
media_to_payload(&media, false),
Some(MediaItemPayload::Video {
media: MediaRef::Source(_),
..
})
));
}
/// A `tgxmb-` directory outside the OS temp dir is an upstream-chosen path,
/// not one of ours, and must still be dropped.
#[test]
fn media_to_payload_rejects_a_tgxmb_dir_outside_the_temp_root() {
let dir = tempfile::Builder::new()
.prefix(x_media::TEMP_FILE_PREFIX)
.tempdir_in(std::env::current_dir().unwrap())
.unwrap();
let path = dir.path().join("video.mp4");
std::fs::write(&path, b"mp4").unwrap();
let media = Media::Video {
url: path.to_string_lossy().into_owned(),
thumbnail_url: String::new(),
};
assert!(media_to_payload(&media, false).is_none());
}
/// The caption-quote threshold matches the post's text inside the caption,
/// so a long-text cache hit is quoted and a short-text one is not.
#[tokio::test]
+17 -15
View File
@@ -28,9 +28,13 @@ pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
/// own, larger cap — `send::upload::MAX_MEDIA_UPLOAD_BYTES` — and never become
/// photos.
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
/// memory; the photo degrades to the smaller URL instead.
pub(crate) const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
/// Decode budget (bytes), derived from the memory budget below. A photo's peak
/// is `download + 2.5 × decode + upload cap` (see `processing_peak_bytes`), so
/// the decode buffer is what remains after the download and the upload
/// allowance, divided by 2.5: (512 − 32 − 10) MiB × 2/5 = 188 MiB. A larger
/// photo degrades to the smaller URL instead.
pub(crate) const MAX_DECODE_BYTES: u64 =
(MEMORY_UNITS as u64 * MEMORY_UNIT_BYTES - MAX_PHOTO_DOWNLOAD_BYTES - MAX_UPLOAD_BYTES) * 2 / 5;
/// Cap for *downloading* a photo in the send fallback, kept separate from the
/// decode budget above: the whole body is buffered before it is processed, once
/// per download slot in flight, while the decode budget is about a single
@@ -47,10 +51,9 @@ const MEMORY_UNIT_BYTES: u64 = 64 * 1024 * 1024;
/// Process-wide memory budget for photo preparation, in [`MEMORY_UNIT_BYTES`]
/// units: 512 MiB. `PREP_SLOTS` bounds how many items are prepared at once but
/// not how much memory they hold — one photo's decode buffer can be up to
/// [`MAX_DECODE_BYTES`] (512 MiB), and the guard that refuses a bigger one is
/// per photo, so six concurrent photos could peak near 3 GiB on a host sized
/// for a fraction of that. Each item charges what it actually holds (its
/// not how much memory they hold, so [`MAX_DECODE_BYTES`] is derived to keep one
/// photo's worst-case peak inside this budget, not just its decode buffer.
/// Each item charges what it actually holds (its
/// downloaded bytes plus the decode buffer its header predicts), so a 10-image
/// album of ordinary photos still runs several at a time while huge ones
/// serialize.
@@ -466,13 +469,13 @@ mod tests {
/// The budget is a *process-wide* memory bound: `PREP_SLOTS` (6) caps how
/// many photos are prepared at once, but six max-size photos would still
/// hold six decode buffers of up to 512 MiB each.
/// hold six peaks of the whole budget each.
#[tokio::test]
async fn huge_decodes_cannot_overlap_but_do_run_alone() {
let budget = std::sync::Arc::new(tokio::sync::Semaphore::new(MEMORY_UNITS as usize));
let max_photo = processing_peak_bytes(MAX_PHOTO_DOWNLOAD_BYTES, MAX_DECODE_BYTES);
// One max-size photo fits (clamped to the whole budget), so it can
// One max-size photo fits exactly in the whole budget, so it can
// never wait for budget that cannot exist.
let first = tokio::time::timeout(
Duration::from_millis(50),
@@ -534,12 +537,11 @@ mod tests {
assert_eq!(memory_units(MEMORY_UNIT_BYTES + 1), 2);
// Never more than exists, or the item waits for itself forever.
assert_eq!(memory_units(u64::MAX), MEMORY_UNITS);
// One item's worst case (a max download plus a max decode) takes the
// whole budget by itself.
assert_eq!(
memory_units(MAX_DECODE_BYTES + MAX_PHOTO_DOWNLOAD_BYTES),
MEMORY_UNITS
);
// One item's worst case (a max download plus a max decode) fits the
// whole budget exactly: it never waits, and never exceeds the budget.
let worst = processing_peak_bytes(MAX_PHOTO_DOWNLOAD_BYTES, MAX_DECODE_BYTES);
assert!(worst <= MEMORY_UNIT_BYTES * MEMORY_UNITS as u64);
assert_eq!(memory_units(worst), MEMORY_UNITS);
}
#[test]
+163 -6
View File
@@ -30,6 +30,84 @@ fn terminal_write_backoff(attempt: u32) -> Duration {
}
pub const LOCK_TTL_SECONDS: f64 = 120.0;
/// Heartbeat cadence: a running handler's lease is re-extended this often.
const HEARTBEAT_SECONDS: f64 = 30.0;
/// Whether a failed heartbeat must abandon the attempt. The lease was last
/// confirmed `secs_since_confirmed` ago and expires `LOCK_TTL_SECONDS` after
/// that; if the next beat would land at or past expiry, another worker may
/// already re-lease the row, so the attempt has to stop before then.
fn heartbeat_failure_is_fatal(secs_since_confirmed: f64) -> bool {
secs_since_confirmed + HEARTBEAT_SECONDS >= LOCK_TTL_SECONDS
}
tokio::task_local! {
/// The lease of the attempt running on this worker task. A handler reads it
/// (via [`lease_still_held`]) to verify with a fresh DB read that it still
/// owns the row before running side effects a re-lease would duplicate.
static CURRENT_LEASE: Lease;
}
/// Handle to the row lease one queue attempt runs under.
#[derive(Clone)]
pub struct Lease {
pool: Arc<crate::db::DbPool>,
id: String,
token: String,
}
impl Lease {
/// Whether this attempt still owns the row. A fresh read on purpose: while
/// a stalled attempt sits between heartbeats its lease can expire and be
/// re-leased by the sweep, and only the DB knows.
pub async fn still_held(&self) -> bool {
let id = self.id.clone();
let token = self.token.clone();
self.pool
.with_conn_or(
log::Level::Warn,
"queue lease check failed",
false,
move |conn| {
conn.query_row(
"SELECT EXISTS(SELECT 1 FROM tasks \
WHERE id=?1 AND status='in_progress' AND lease_token=?2)",
params![id, token],
|row| row.get::<_, bool>(0),
)
},
)
.await
}
}
/// Whether the attempt running on this worker task still owns its row. Answers
/// `true` outside a queue worker (a direct handler call, as in tests), so a
/// caller never skips its effects for want of a lease.
pub async fn lease_still_held() -> bool {
match CURRENT_LEASE.try_with(|lease| lease.clone()) {
Ok(lease) => lease.still_held().await,
Err(_) => true,
}
}
/// Runs `f` with a lease installed as the current worker lease. Test-only seam
/// for driving a handler's lease check without a live worker loop.
#[cfg(test)]
pub(crate) async fn scope_lease_for_test<F: Future>(
pool: Arc<crate::db::DbPool>,
id: &str,
token: &str,
f: F,
) -> F::Output {
let lease = Lease {
pool,
id: id.to_string(),
token: token.to_string(),
};
CURRENT_LEASE.scope(lease, f).await
}
/// Number of concurrent worker loops. Tasks are independent (retries and
/// forward resumes); leases serialize row claims via SQLite transactions, so
/// extra workers drain backlogs faster. Each worker can be mid-send to
@@ -524,8 +602,19 @@ impl QueueWorker {
row.attempts + 1
);
let attempt_started = std::time::Instant::now();
let outcome = match self
.run_with_lease(&row.id, &row.lease_token, payload)
// The attempt runs under its own lease handle, exposed to the handler
// through a task-local: a handler about to run duplicate-able side
// effects verifies it still owns the row first (see `lease_still_held`).
let lease = Lease {
pool: Arc::clone(&self.pool),
id: row.id.clone(),
token: row.lease_token.clone(),
};
let outcome = match CURRENT_LEASE
.scope(
lease,
self.run_with_lease(&row.id, &row.lease_token, payload),
)
.await
{
Ok(outcome) => outcome,
@@ -598,10 +687,13 @@ impl QueueWorker {
) -> Result<Result<(), QueueError>, LeaseLost> {
let fut = (self.handler)(payload);
tokio::pin!(fut);
let mut interval = tokio::time::interval(Duration::from_secs(30));
let mut interval = tokio::time::interval(Duration::from_secs_f64(HEARTBEAT_SECONDS));
// The first interval tick fires immediately; skip it (the lease was
// just set by lease_next).
interval.tick().await;
// The lease was set by lease_next just before this, so the first
// confirmation point is now.
let mut confirmed = std::time::Instant::now();
let id_owned = id.to_string();
let token_owned = lease_token.to_string();
loop {
@@ -609,6 +701,7 @@ impl QueueWorker {
result = &mut fut => return Ok(result),
_ = interval.tick() => {
let now = now_f64();
let sent = std::time::Instant::now();
let id = id_owned.clone();
let token = token_owned.clone();
let result = self
@@ -622,17 +715,28 @@ impl QueueWorker {
})
.await;
match result {
// Still ours: the lease is extended.
Ok(1) => {}
// Still ours: the lease is extended, and this beat is
// the new confirmation point.
Ok(1) => confirmed = sent,
// The row is no longer leased to us (another worker
// re-leased it, or it is gone): dropping the handler
// future here stops this attempt instead of racing the
// new holder through the same send.
Ok(_) => return Err(LeaseLost),
// A failed write says nothing about the lease itself,
// which stays valid until `locked_until`. Abandoning
// the attempt here re-ran tasks whose side effects were
// half done, so keep going and beat again; stop only
// when the next beat could land past expiry.
Err(e) => {
log::error!("queue lease heartbeat failed: {e}; abandoning attempt");
if heartbeat_failure_is_fatal(confirmed.elapsed().as_secs_f64()) {
log::error!(
"queue lease heartbeat failed: {e}; lease can no longer be confirmed, abandoning attempt"
);
return Err(LeaseLost);
}
log::warn!("queue lease heartbeat failed: {e}; retrying at the next beat");
}
}
}
}
@@ -798,6 +902,59 @@ mod tests {
.unwrap();
}
#[test]
fn failed_heartbeat_is_fatal_only_when_the_next_beat_would_miss_expiry() {
// Confirmed at t=0, the lease expires at t=120 and beats land every 30s.
assert!(!heartbeat_failure_is_fatal(0.0));
assert!(!heartbeat_failure_is_fatal(89.0));
assert!(heartbeat_failure_is_fatal(90.0));
}
/// The lease handle a handler sees is a live view of the row: once the row
/// is re-leased — what a stalled attempt hits — `lease_still_held` says so,
/// which is what gates the duplicate-able side effects (forward, prompt).
#[tokio::test]
async fn lease_still_held_is_false_once_the_row_is_re_leased() {
let (queue, _dir) = new_queue().await;
let pool = Arc::clone(&queue.pool);
let seen = Arc::new(Mutex::new(Vec::new()));
let seen_handler = Arc::clone(&seen);
let pool_handler = Arc::clone(&pool);
queue
.start(
move |_payload| {
let seen = Arc::clone(&seen_handler);
let pool = Arc::clone(&pool_handler);
async move {
let held_before = lease_still_held().await;
seen.lock().push(held_before);
// Steal the lease, as the sweep does once a stalled
// attempt let it expire.
pool.with_conn(|conn| {
conn.execute(
"UPDATE tasks SET lease_token='other' WHERE status='in_progress'",
[],
)
})
.await
.unwrap();
let held_after = lease_still_held().await;
seen.lock().push(held_after);
Ok(())
}
},
|_payload, _message| async {},
)
.await;
queue
.enqueue(serde_json::json!({"n": 1}), now_f64())
.await
.unwrap();
tokio::time::sleep(Duration::from_millis(300)).await;
assert_eq!(&*seen.lock(), &[true, false]);
queue.stop().await;
}
async fn new_queue() -> (PersistentTaskQueue, tempfile::TempDir) {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("queue.db");
+79
View File
@@ -1681,6 +1681,85 @@ mod tests {
assert_eq!(payload["message_ids"], serde_json::json!([10, 11]));
}
/// A re-leased row's handler must not run the forward: the new holder owns
/// that copy, so running it here too would deliver the messages twice.
#[tokio::test]
async fn a_forward_skips_the_copy_once_the_lease_is_gone() {
// The empty script panics on any call, so a copy that sneaks through
// fails loudly rather than hiding behind an assertion.
let sender = MockSender::scripted(vec![], || api_error("unused"));
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let task = Task::ForwardMessages {
from_chat_id: 1,
to_chat_id: 2,
message_ids: vec![10, 11],
forward_offset: 0,
notify_chat_id: None,
notify_message_id: None,
};
let payload = serde_json::to_value(&task).unwrap();
// A lease the tasks table does not hold: `still_held` reads false.
let pool = crate::db::open_store(stores.db_path()).unwrap();
let outcome =
crate::queue::scope_lease_for_test(pool, "gone", "tok", handle_task(&ctx, payload))
.await;
assert!(outcome.is_ok());
assert!(sender.calls().is_empty(), "no copy may run");
}
/// The completion gate: a media send whose row was re-leased must not open
/// the edit-before-forward prompt (nor forward) — the new holder does, so
/// doing it here too would prompt and forward the post a second time.
#[tokio::test]
async fn a_media_send_skips_post_send_actions_once_the_lease_is_gone() {
let sender = MockSender::scripted(vec![Outcome::GroupOk, Outcome::MessageOk], || {
api_error("unused")
});
let stores = TestStores::new();
let ctx = stores.ctx(&sender);
let mut task = sent_task(Some(2), false);
if let Task::SendMediaSequence {
media_batches,
edit_before_forward,
..
} = &mut task
{
// Two items so the send is one `send_media_group`; the prompt on so
// the completion path has an observable call — a group send alone
// reports no message ids, so a forward would copy nothing.
*media_batches = vec![vec![
photo_item("https://p/1.jpg", false, false),
photo_item("https://p/2.jpg", false, false),
]];
*edit_before_forward = true;
}
let payload = serde_json::to_value(&task).unwrap();
let pool = crate::db::open_store(stores.db_path()).unwrap();
let outcome =
crate::queue::scope_lease_for_test(pool, "gone", "tok", handle_task(&ctx, payload))
.await;
assert!(outcome.is_ok());
assert_eq!(
sender.calls(),
vec!["send_media_group"],
"the edit prompt must not open"
);
// Control: the same task under a live lease does open the prompt — so
// the assertion above cannot pass because the prompt was never
// observable in the first place.
let live = MockSender::scripted(vec![Outcome::GroupOk, Outcome::MessageOk], || {
api_error("unused")
});
let live_ctx = stores.ctx(&live);
let outcome = handle_task(&live_ctx, serde_json::to_value(&task).unwrap()).await;
assert!(outcome.is_ok());
assert_eq!(live.calls(), vec!["send_media_group", "send_message"]);
}
#[tokio::test]
async fn post_send_notifies_a_permanent_forward_failure() {
let sender = MockSender::scripted(vec![Outcome::CopyErr, Outcome::MessageErr], || {
+25 -4
View File
@@ -480,11 +480,31 @@ pub(crate) async fn handle_task(
// whole sequence (every batch) completed, so the channel forward
// and the edit-before-forward prompt must not be lost just
// because the send needed a retry.
//
// Unless the row was re-leased while this attempt sat between
// heartbeats (a ≥120 s stall): the new holder runs the same
// effects, so running them here too would copy to the channel a
// second time and open a second prompt. The keep-alive release is
// skipped with them — the payload (and so its temp media) is the
// same one the new holder is using.
if !crate::queue::lease_still_held().await {
log::warn!(
"queue: row re-leased before post-send actions; leaving the forward and the edit prompt to the new holder"
);
return Ok(());
}
post_send_actions(ctx, &task, message_ids).await;
settle_task(ctx, &task, Settled::Sent).await;
Ok(())
}
Task::ForwardMessages { .. } => match forward_messages(ctx, &task).await {
Task::ForwardMessages { .. } => {
// The copy *is* the side effect here, so it is the thing to gate:
// a re-leased row is already being copied by its new holder.
if !crate::queue::lease_still_held().await {
log::warn!("queue: row re-leased before the forward; leaving it to the new holder");
return Ok(());
}
match forward_messages(ctx, &task).await {
Ok(()) => Ok(()),
Err(SendError::Retryable {
delay_seconds,
@@ -494,14 +514,15 @@ pub(crate) async fn handle_task(
payload: serde_json::to_value(task).expect("task serializes"),
}),
Err(SendError::Permanent { message, task }) => {
// Settled by `dead_letter_notify`, which the queue invokes for
// this payload.
// Settled by `dead_letter_notify`, which the queue invokes
// for this payload.
Err(QueueError::Permanent {
message,
payload: serde_json::to_value(task).expect("task serializes"),
})
}
},
}
}
}
}
+8
View File
@@ -112,6 +112,14 @@ impl ChatStore {
self.load(chat_id).await.unwrap_or_default()
}
/// Strict read for callers whose *behavior* depends on the settings. A
/// send that reads them via [`Self::get`] would silently use the defaults
/// on a failed read — no channel forward, no edit prompt — and persist
/// those defaults into the retry payload it builds.
pub async fn try_get(&self, chat_id: i64) -> Result<ChatData, ()> {
self.load(chat_id).await
}
/// Write-through: update the cache and the DB. Returns whether the DB
/// write landed: the cache is updated either way, so `false` means the
/// change lives only until the next restart and the caller has to say so
+3 -3
View File
@@ -65,12 +65,12 @@ services:
WEBHOOK_SECRET_TOKEN: '${WEBHOOK_SECRET_TOKEN:-}'
# Defaults, listed so they are discoverable; override in .env when needed.
LOCAL_USER_ID: '${LOCAL_USER_ID:-9001}'
RUST_LOG: '${RUST_LOG:-info}'
RUST_LOG: '${RUST_LOG:-}'
EDIT_MESSAGE_TTL_SECONDS: '${EDIT_MESSAGE_TTL_SECONDS:-86400}'
LINK_CACHE_TTL_SECONDS: '${LINK_CACHE_TTL_SECONDS:-604800}'
CAPTION_QUOTE_TEXT_CHARS: '${CAPTION_QUOTE_TEXT_CHARS:-200}'
VIRTUAL_PORT: '${VIRTUAL_PORT:-8443}'
WEBHOOK: '${WEBHOOK:-true}'
WEBHOOK: '${WEBHOOK:-false}'
WEBHOOK_LISTEN: '${WEBHOOK_LISTEN:-0.0.0.0}'
WEBHOOK_PORT: '${WEBHOOK_PORT:-8443}'
# For a certificate on a bare IP: uncomment and set ACME_HOST in .env.
@@ -93,7 +93,7 @@ services:
# reported unhealthy. nginx-proxy shows 502s while webhook mode is down,
# so surface that to the orchestrator.
healthcheck:
test: ["CMD-SHELL", "test '${WEBHOOK:-true}' != true || bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"]
test: ["CMD-SHELL", "test '${WEBHOOK:-false}' != true || bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"]
interval: 30s
timeout: 5s
retries: 3