fix(retry): let slow downloads finish, and never re-run a finished task

P1 of the retry audit, from the report's "reliability and diagnosis" batch.

- Media downloads no longer share the 30s *total* timeout of metadata
  fetches. The size caps allowed 10 MiB (reupload fallback) and 512 MiB
  (ugoira frame zip) while the clock allowed 30s, so a slow link made those
  posts impossible: `MEDIA_CLIENT` has no total timeout and instead bounds
  the response head and every chunk with a 30s *idle* window, which keeps the
  stalled-connection protection. Verified against a local probe: the old
  policy aborts a 40s download at 30.0s, the new one completes it (2 MiB,
  40.1s), and a body that stops delivering still fails after exactly 30s.
- A finished row's write-back is no longer best-effort. `delete_row` failing
  left the row `in_progress` with a live lease, so the next sweep flipped it
  back to `pending` and re-ran a completed task — a second album, a second
  prompt, a second channel copy. Both terminal writes are now retried, and a
  delete that still fails falls back to a `done` tombstone that neither the
  lease query nor the sweep looks at; reschedule (no safe tombstone: marking
  it done would drop the retry silently) logs what the sweep will do.
- bsky and pixiv no longer present a *failed* video conversion as a post with
  no media: the remux/ugoira error propagates (pixiv keeps its retry class,
  bsky reports Transient), so the user sees the real cause and `fetch` gets
  its retries. bsky's "no ffmpeg" case stays a degradation — retrying a
  deployment gap cannot help.
- pixiv's token exchange checks the HTTP status before parsing the body, so a
  429/5xx from the OAuth endpoint stays retryable instead of becoming a
  permanent Api/Json error (via the shared `pixiv_error_is_retryable`), and
  startup validation only disables pixiv for a rejected credential — one 503
  while the container came up used to turn every later pixiv link into
  "pixiv support is disabled".
This commit is contained in:
2026-09-20 21:02:46 +08:00
parent 4cf793cd7e
commit 0a82ca5a42
6 changed files with 269 additions and 53 deletions
+1 -1
View File
@@ -44,7 +44,7 @@ The `x-media` library: `site::fetch(url)` dispatches through the `SITES` registr
| `crates/xmedia-bot/src/handlers/` | Handler modules: `mod.rs` (message entry point, `reply`, `log_key`, the group-only `GROUP_LINK_HINT` for a supported link posted outside a private chat), `commands.rs` (teloxide `BotCommands` enum + command executor, incl. `/test <url>` (send-only) / `/debug <url>` (parse-only) and the admin-only `/bot_dict` state dump; `/set_format` rejects unknown `{…}` placeholders and resets with `-`), `urls.rs` (URL extraction + bounded job channel (256) drained by `URL_WORKERS = 8` workers (`start_url_workers`) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), `inline.rs`/`callback.rs` (inline queries / edit-before-forward buttons, incl. `skip`; a forward that fails retryably is both queued *and* settles the prompt — the queued row carries the message ids itself, and a prompt left live let a second Confirm copy the same messages twice and let Skip answer "nothing was forwarded" while the row still delivered), `statics.rs` (global statics) |
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, a finished row's `DELETE`/reschedule retried and a failed delete falling back to a `done` tombstone (the lease query and the sweep only look at `pending`/`in_progress`, so a task that already ran cannot be resurrected and re-run), `notify_one` worker wakeup plus a separate `Notify` for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit), `busy_timeout` on all connections |
| `crates/xmedia-bot/src/ctx.rs` | `AppContext`: the injected collaborators (`sender` + `ChatStore`/`PersistentTaskQueue`/`LinkCache`/`Config`), `from_statics` for production and the `CONTEXT` static the worker closures hold. `test_support::TestStores` backs handler tests with a tempdir store set |
| `crates/xmedia-bot/src/send/` | `send/mod.rs`: `Task`/`MediaItemPayload` payloads, `SendError`/`Classification`, `send_media_sequence`/`send_animation`/`forward_messages`; `send/input_media.rs`: payload → `InputFile`/`InputMedia` + `build_media_group` (caption on the first item only); `send/upload.rs`: the download-and-reupload fallback (`prepare_upload_item`/`send_batch_via_upload`, photo downscale handoff); `send/post_send.rs`: link-cache write, `KEEP_ALIVE` registry, `settle_task`, `post_send_actions`, `handle_task`/`dead_letter_notify` |
| `crates/xmedia-bot/src/media_sender.rs` | `MediaSender` trait: the user-flow surface (`send_media_group`/`send_animation`/`copy_messages`/`send_message`/`answer_callback_query`/`edit_message_caption`/`delete_message`/`send_chat_action`) implemented by teloxide `Bot` (per-chat rate-limited) and by a recording `MockSender` in tests. Admin/setup APIs (`get_chat`, `set_my_commands`, …) stay on the concrete `Bot` |