download_to_temp guarded against a file id, but prepare_upload_item answered a non-http media_url first — and item_url maps FileId to the id itself, so the id was presented as a local *path* and died later with a confusing file-open error instead of a classification. The guard now sits at the entry, download_to_temp takes the already-narrowed URL (the inner guard deleted as dead), and the test pins the refusal as Permanent before any I/O. The file also carries item10's download-classification arm (RateLimited is retryable there too) — same file, landed here to keep each tree compiling.
A non-photo body charged the process-wide budget for as long as download_to_temp held it; a photo charged nothing until its prepare step, so the download itself — up to MAX_PHOTO_DOWNLOAD_BYTES per item, six items per batch, eight prep slots process-wide — was memory the MEMORY_BUDGET comment promised was bounded but was not (~6x32 MiB on top of the accounted512 MiB). Photos now reserve their own download cap for that window; the prepare step still charges header probe + decode buffer, the only overlap that is actually held in memory together. Net accounting stays within the declared budget instead of exceeding it whenever a batch of large photos downloads at once (audit: upload.rs photo window uncharged).
All three callers passed item.thumbnail_url() as the fourth argument — a parameter that could never vary without defeating its own purpose, and only the video arm ever reads it. The function takes the item already; it asks the item. Three call sites lose an argument.
download_media_limited had one hard-coded total (600s) for every caller, and its heaviest caller — the bot's upload fallback — holds a PREP slot (and its memory reservation) for the whole transfer: six slow-but-alive downloads (a byte every 29s satisfies the idle window) could stall the fallback chain for ten minutes, queue retries included. The budget is a parameter now: the fallback passes 300s of its own (50 MiB in 300s ≈ 1.4 Mbit/s; a slower link is better served by retrying toward the item's smaller URL than by pinning a slot), while bsky's in-fetch HLS segments keep the generous 600s DOWNLOAD_TOTAL_TIMEOUT, now pub(crate) and re-exported for them. download_too_slow reports whichever budget it got.
download_to_temp wrote the whole buffered body — up to MAX_MEDIA_UPLOAD_BYTES, 50 MiB — with a sync write_all on whatever executor thread ran the prep task; six PREP slots could stall six runtime threads at once on a slow volume (Docker volume, SD card), pausing unrelated handlers and chat-action refreshes with them. The write now runs in spawn_blocking, the pattern the photo decode in the same file already uses; the failure classes are unchanged (write error = retryable resource exhaustion, panic = permanent, worker bugs must be loud).
One MAX_UPLOAD_BYTES (10 MiB) bounded every upload, but that is the *photo*
limit: Telegram's own docs say sendVideo/sendAnimation/sendDocument take up
to 50 MB, and RequestEntityTooLarge is "larger than 50 MB". So a 10-50 MB
video that Telegram refused to fetch by URL was refused a download too, and
a video has no smaller variant — the post was lost. The non-photo cap is now
MAX_MEDIA_UPLOAD_BYTES, and such a body charges the process-wide budget for
the length of the preparation (one 64 MiB unit covers the cap), since
PREP_SLOTS alone no longer bounds their added RAM. A const test pins both
caps against Telegram's numbers.
MediaItemPayload carried one String field with two meanings and a file_id
bool beside it to say which, in all three variants; every reader re-checked
the flag (input_file had a three-arm pattern just to find the file-id case).
MediaRef::Source/FileId says it once, and the readers now match on it —
local_media_paths, item_url, input_file and the download path each shrank to
the one branch they care about.
Fixes a real failure that the flag was hiding: send_animation built its
InputFile with input_file_for(media_url), which read a cached file id as a
local path and answered "local media file missing" — permanent. So the
second request for a single-gif post always failed (the third worked, from
the degraded entry). It now uses the payload's own input_file, and
a_cached_animation_sends_by_file_id fails without that line.
Wire shape: a queued row from before this change no longer parses, and the
queue already handles that shape (handle_task dead-letters it as an invalid
payload, and dead_letter_notify still names the post and drops the stale
cache entry).
pending_backlog, earliest_run_after, ChatStore::get and LinkCache::get each
hand-rolled prepare + query + rows.next() for what is a single-row read.
query_row + OptionalExtension::optional is the same statement and the same
error mapping with less scaffolding; the backlog's NULL-on-empty MIN still
goes through the count check, so a pending row with a NULL run_after is not
misread. Also fixes the rustfmt drift from the previous commit.
media_from_file, media_from_url and build_media_group's closure each wrote
the same per-kind match plus the same video-thumbnail attach. media_from
takes the already-selected InputFile; the two builders that differ only in
how that file is chosen are now two-line calls to it. The local-file branch
keeps InputFile::file (no existence probe) and every caller still passes the
item's own has_spoiler / thumbnail_url, so what reaches Telegram is the same.
`PREP_SLOTS` caps how many items are prepared at once (6) but says nothing
about what they hold: one photo's decode buffer can be up to
`MAX_DECODE_BYTES` (512 MiB) and that guard is *per photo*, so six of them —
an album of large scans, two chats at once — could peak near 3 GiB on a host
sized for a fraction of it. The upload fallback is the only path that
allocates like this; nothing downstream notices until the kernel does.
Photo preparation now charges a process-wide memory budget
(`MEMORY_UNITS` × 64 MiB = 512 MiB) for what it actually holds: the
downloaded bytes plus the decode buffer the *header* predicts — the same
prediction the per-photo guards apply, now shared (`decode_bytes`,
`decode_budget_bytes`) so the reservation and the guard cannot drift. A photo
that is already within Telegram's limits is billed only its download, so an
ordinary 10-image album still runs several at a time; two photos near the
per-photo cap serialize (each takes the whole budget). The request is clamped
to the budget so a single huge photo runs alone instead of waiting for
permits that cannot exist.
Verified with a throwaway harness against a locally served 9999x9999 PNG
(126 KB on the wire, ~100 MB decoded) driven through the real
`prepare_upload_item`: with the budget held the preparation waits — "after
1516ms the prep is still waiting on the budget" — and finishes in 11.8s the
moment it is released, so the accounting binds in the real path and not just
in the semaphore.
Kept as permanent tests instead: the unit math (rounding, clamp, and that a
max-size photo still gets the whole budget rather than waiting forever), the
budget sharing (huge decodes cannot overlap, ordinary ones do not queue), and
the prediction agreeing with the processing decision (over-sized PNG/JPEG
charged, within-limits and unknown formats free).
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean (122 bot + 91 x-media).
Each batch's items were prepared under their own `Semaphore::new(3)`, which
is not a memory bound: 8 URL workers and 4 queue workers can each be inside
a batch, so a burst could have two dozen downloads in flight at once, each
buffering a whole photo before it is processed. Nothing else on the media
path bounds them — the send itself is paced by the rate limiter, but the
download and the decode happen before it is charged.
One process-wide `PREP_SLOTS` (6) replaces the per-batch semaphore, and the
photo download gets its own cap: `MAX_PHOTO_DOWNLOAD_BYTES` (32 MiB) for the
transfer, with `MAX_DECODE_BYTES` (512 MiB) left as the pre-allocation guard
on a single decoded buffer. A photo over the download cap degrades to its
smaller URL exactly as one over the decode budget does
(`FallbackError::MediaTooLarge` → `fallback_url`) — never an error.
Verified with the same throwaway proxy harness: 4 concurrent 10-item batches
against a server that holds every response 150 ms peak at exactly 6
concurrent downloads (the per-batch three allowed 12) with all 40 items
prepared.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
The upload fallback asked `x_media::site::media_size` for every remote item
before downloading it. That call is a real GET (not a HEAD) on the *un-
guarded* `CLIENT` — so every fallback item cost two requests where one would
do, the response body was never consumed (the connection cannot return to
the pool), and for photos the answer was discarded outright
(`too_large && !matches!(item, Photo { .. })` still fired the request). It
bypassed `media_request`'s private-network guard as well, the one choke
point every other egress goes through.
For videos the probe was redundant twice over: `download_media_limited`
reads the declared Content-Length before any body byte and aborts with
`FetchError::TooLarge`, which the call site already turns into the item's
smaller URL (`FallbackError::MediaTooLarge` → `fallback_url`).
`media_size` is deleted (no other caller) and the download's own cap is the
only size gate. The video cap is now exactly `MAX_UPLOAD_BYTES` instead of
`MAX_UPLOAD_BYTES + 1`, so the boundary the probe drew survives byte for
byte: a file of exactly the cap is admitted (`len > max_bytes` is false),
one byte over degrades to the smaller URL.
Verified with a throwaway harness (a local HTTP server reached through
`TELOXIDE_PROXY`, the one LAN egress the guard allows): a small video, a
photo and an oversized video each cost 1 request where the probe made it 2,
and the oversized one still lands on `/fallback.mp4` without fetching it.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
P2 (hardening) of the retry audit, closing the report's remaining findings.
- Lease fencing. `lease_next` now stamps a random `lease_token`, and every
write-back a worker makes (the 30s heartbeat, `delete_row`, `reschedule`,
`mark_done`) is guarded by it. A lease that expired while its holder was
stalled and was then re-leased used to let *both* holders write the same row:
one duplicated the send, the other silently discarded the new holder's retry
(a 0-row update was not even logged). Now a worker that no longer holds the
lease drops its attempt at the next heartbeat and writes nothing. Reaching
existing databases needed a migration chain, which `db.rs` had been
pre-committed to: `MIGRATIONS` + `migrate` track `PRAGMA user_version`, with
`schema_init` as the version-0 baseline. Verified on a database created
before this change: user_version 0 -> 1, column added, rows intact.
- Dead-letter notifications no longer mislabel an unparsable payload. A row
whose payload no longer deserializes as a `Task` (an older version's shape,
corruption) used to skip the cache invalidation *and* report "Forward failed
permanently" for a send task, because both were derived from the parsed
value. The identity now comes off the raw JSON, so the stale link-cache entry
is dropped and the message names the post.
- Temp files are marked and swept. Every temp file/dir the project creates now
carries `x_media::TEMP_FILE_PREFIX`, and startup removes entries with that
prefix older than an hour — a killed process leaves its downloads (up to
hundreds of MB) behind because no destructor runs, and the age gate keeps the
sweep away from a second instance's in-flight files. Verified live: the log
reports the sweep, an aged leftover goes, a fresh prefixed file and an
unrelated file stay.
P0 of the retry audit. The main finding: a Telegram 5xx was classified
Permanent, so one Telegram-side blip dead-lettered the post.
- `classify_request_error`: a server error is retryable again. teloxide sleeps
10s on a 5xx and then parses the body, so the HTTP status is gone by the
time the error arrives; it is recognised by shape instead — a JSON
server-error description, or an `InvalidJson` whose raw body is not JSON
(a proxy/error page). A JSON body of the wrong shape stays permanent, since
retrying a type mismatch cannot help. Reproduced end to end: with the old
classification a fake 502 (HTML body) logged "failed permanently" and
dead-lettered; now it logs "queued for retry" and the retry delivers.
- The same class of mistake elsewhere: `is_media_fetch_failure` was missing
`failed to get HTTP url content`, the description single-media URL sends
answer with, so hotlink-rejected media failed permanently instead of going
through the reupload fallback.
- `enqueue_retry` now reports whether the row was written, and the callers
only promise a retry when it was — a failed enqueue (DB write) used to tell
the user "retrying in Ns" and then deliver nothing, ever.
- A forward that fails retryably now settles the prompt instead of leaving it
live: the queued row carries the message ids itself, and a live prompt let
a second Confirm copy the same messages to the channel twice and let Skip
answer "nothing was forwarded" while the row still delivered.
- A prompt that could not be sent no longer swallows the gated forward
silently: the chat is told, since nothing would ever forward.
- `scaled_retry_delay` only scales up, so a server-asked `retry_after` above
the 300s cap is honoured instead of retried early (which earned another 429
and then dead-lettered the post).
- Download classification: a 4xx media download is permanent (the media is
gone or refused) while transport errors and 429/5xx retry — previously every
download error counted as retryable and burned the whole budget. A temp-file
*write* failure retries too (resource exhaustion clears; a temp dir that
cannot be created stays permanent).
- Site status mapping: 401/403 are `Blocked` (permanent) rather than
`Transient`, so a refusal is reported at once instead of after three
wasted attempts; and a twitter 200 that is not a tweet is no longer
reported as withheld content (the empty `{}` withheld shape keeps
`Sensitive`, which is what triggers the auth fallback).
send.rs had grown back into the shape handlers.rs was split out of: payload
types, error classification, the download-and-reupload fallback, the senders
and the whole post-send/queue shell in one file. Split by concern, leaving
call sites (`crate::send::x`) unchanged:
- `send/input_media.rs`: payload → `InputFile`/`InputMedia` selection and
`build_media_group` (with its caption-on-first-item rule).
- `send/upload.rs`: the fallback pipeline (download with the upload cap,
photo downscale handoff, smaller-URL fallback, multipart upload).
- `send/post_send.rs`: link-cache write, the `KEEP_ALIVE` registry for locally
produced media, `settle_task`, the post-send actions and the queue entry
points; the parts other modules call are re-exported.
- `send/mod.rs`: payloads, error classification, classification helpers and
the senders themselves, plus the test module.
No behaviour change: 128 + 1326 + 366 + 345 lines, 70 tests still pass.
AGENTS.md updated for the new layout and for `ctx.rs`.
YoursFunny