x-media's reqwest had exactly `["json", "rustls-tls"]` — no decompression, no
HTTP/2 — so every adapter fetched its JSON as identity over HTTP/1.1. The
site APIs compress: twitter's syndication body measures 4469 bytes identity
against 1066 gzipped (4.2x) for a single-tweet response, and the fetch is on
every twitter link. The CDNs all negotiate h2 (the shared client reports
HTTP/2.0 against cdn.syndication.twimg.com, api.bilibili.com and
public.api.bsky.app), which also multiplexes the concurrent media downloads
that used to open a connection each.
Both features are one crate-level switch, so the pages' compression is what
the APIs answer with and requires no adapter change: reqwest adds
`accept-encoding: gzip` and decompresses transparently. `download_media_limited`
keeps a correct size cap either way — it checks the accumulated *body* while
streaming, not the declared Content-Length, which for a compressed response
is the compressed size.
Verified: a local echo server recorded `accept-encoding: gzip` on a request
from the shared client (both clients come from `build_client`), and a probe
over the shared `CLIENT` reported `HTTP/2.0` for the three site hosts above.
`cargo test -p x-media -- --ignored live` (14 passed) covers the real
endpoints with the new transport.
Cargo.lock gains async-compression (+codecs/core), fnv and h2.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
A long bluesky video can be 500 segments, and the remux downloaded them
strictly one at a time: the user waited for every round trip in turn, which
is the dominant cost of the whole remux (the ffmpeg concat itself is local).
Each segment's multi-megabyte body also went to disk through a blocking
`std::fs::write` on an executor thread.
Segments now download and write under a small bound (`SEGMENT_CONCURRENCY`,
4 — a segment can be 20 MiB and the playlist is capped at 256 MiB, so this is
also what bounds the remux's peak memory) and the write goes through
`tokio::fs`. Concurrent downloads complete in completion order, and ffmpeg
concatenates the list in whatever order it holds — an out-of-order list is a
*silently* scrambled video, not an error — so `concat_list` sorts by segment
index and carries its own test.
x-media's own tokio features gain `rt` (JoinSet) and `fs`: the library
already used `spawn_blocking` on the strength of the bot crate's features.
Verified against a local HLS fixture — 12 one-second segments of solid
red/green/blue, each served with a 150 ms delay, reached through a name that
resolves to loopback (the guard refuses a literal 127.0.0.1) — with a
temporary in-module test: all 12 sampled frames come back in the right colour
order, and the server recorded a peak of 4 requests in flight, where the
serial version showed 1.
`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
Resolve the manifest and lockfile overlap with the rusqlite 0.40 and zip 8.6
bumps that landed on master after this branch was opened:
- crates/xmedia-bot/Cargo.toml: keep rusqlite 0.40 and rand 0.10
- Cargo.lock: re-point x-media/xmedia-bot at rand 0.10.2; teloxide keeps its
own rand 0.8.8 (it requires ^0.8.5), and cargo pruned the now-unused
version_check entry
Verified on the merged tree: cargo metadata --locked, cargo fmt --check,
cargo clippy --workspace --all-targets --locked -- -D warnings,
cargo test --workspace --locked (69 + 73 tests).
Bump both crates (x-media, xmedia-bot) and refresh the lockfile to the
latest semver-compatible releases. No direct dependency or manifest
requirement changed.
1.3.0 → 1.4.0: new features (DATA_DIR config, /test blockquote HTML
report) plus the twitter entity-decode, post-send-actions and queue
lease-heartbeat fixes.
OpenSSL came from two removable defaults: teloxide's `default` feature
(native-tls) and x-media's reqwest default features (default-tls). Switch
both to rustls (webpki-roots) so the binary links no system TLS libs:
- teloxide: default-features=false + rustls + ctrlc_handler (was part of
the removed default)
- reqwest (x-media): default-features=false + rustls-tls
Verified: openssl-sys/native-tls gone from the tree, cargo check clean,
all 5 live twitter/bsky fetches pass over rustls, full container startup
works. The runtime image now needs no libssl.so.3/libcrypto.so.3 or CA
bundle (ffmpeg only processes local files; all downloads go through
reqwest), saving ~8MB.
x-media's reqwest 0.13 dragged in quinn/rustls/aws-lc-rs (cmake C
build) alongside teloxide's 0.12; unifying on 0.12 removes the whole
second TLS/QUIC stack from the build and image. The unused regex dep in
xmedia-bot is gone; x-media keeps url (the bsky HLS remux uses it).
YoursFunny
dependabot[bot]