perf: gzip and HTTP/2 for the site clients

x-media's reqwest had exactly `["json", "rustls-tls"]` — no decompression, no
HTTP/2 — so every adapter fetched its JSON as identity over HTTP/1.1. The
site APIs compress: twitter's syndication body measures 4469 bytes identity
against 1066 gzipped (4.2x) for a single-tweet response, and the fetch is on
every twitter link. The CDNs all negotiate h2 (the shared client reports
HTTP/2.0 against cdn.syndication.twimg.com, api.bilibili.com and
public.api.bsky.app), which also multiplexes the concurrent media downloads
that used to open a connection each.

Both features are one crate-level switch, so the pages' compression is what
the APIs answer with and requires no adapter change: reqwest adds
`accept-encoding: gzip` and decompresses transparently. `download_media_limited`
keeps a correct size cap either way — it checks the accumulated *body* while
streaming, not the declared Content-Length, which for a compressed response
is the compressed size.

Verified: a local echo server recorded `accept-encoding: gzip` on a request
from the shared client (both clients come from `build_client`), and a probe
over the shared `CLIENT` reported `HTTP/2.0` for the three site hosts above.
`cargo test -p x-media -- --ignored live` (14 passed) covers the real
endpoints with the new transport.

Cargo.lock gains async-compression (+codecs/core), fnv and h2.

`cargo fmt --check`, `cargo clippy --workspace --all-targets --locked -- -D
warnings` and `cargo test --workspace --locked` clean.
This commit is contained in:
2026-09-21 13:09:32 +08:00
parent a501a17519
commit 994734b001
3 changed files with 63 additions and 2 deletions
Generated
+61
View File
@@ -60,6 +60,18 @@ dependencies = [
"object",
]
[[package]]
name = "async-compression"
version = "0.4.43"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3976abdc8fe7d1133d43d304afd42abdf5bc3e1319d263d223bde07b5efc4be8"
dependencies = [
"compression-codecs",
"compression-core",
"pin-project-lite",
"tokio",
]
[[package]]
name = "atomic-waker"
version = "1.1.2"
@@ -266,6 +278,23 @@ dependencies = [
"windows-sys 0.61.2",
]
[[package]]
name = "compression-codecs"
version = "0.4.38"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf"
dependencies = [
"compression-core",
"flate2",
"memchr",
]
[[package]]
name = "compression-core"
version = "0.4.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789"
[[package]]
name = "const-oid"
version = "0.10.2"
@@ -583,6 +612,12 @@ dependencies = [
"zlib-rs",
]
[[package]]
name = "fnv"
version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
[[package]]
name = "foldhash"
version = "0.2.0"
@@ -713,6 +748,25 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "h2"
version = "0.4.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
dependencies = [
"atomic-waker",
"bytes",
"fnv",
"futures-core",
"futures-sink",
"http",
"indexmap 2.14.2",
"slab",
"tokio",
"tokio-util",
"tracing",
]
[[package]]
name = "hashbrown"
version = "0.12.3"
@@ -849,6 +903,7 @@ dependencies = [
"bytes",
"futures-channel",
"futures-core",
"h2",
"http",
"http-body",
"httparse",
@@ -1740,6 +1795,7 @@ dependencies = [
"bytes",
"futures-core",
"futures-util",
"h2",
"http",
"http-body",
"http-body-util",
@@ -2425,12 +2481,17 @@ version = "0.6.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840"
dependencies = [
"async-compression",
"bitflags 2.13.2",
"bytes",
"futures-core",
"futures-util",
"http",
"http-body",
"http-body-util",
"pin-project-lite",
"tokio",
"tokio-util",
"tower",
"tower-layer",
"tower-service",