mirror of
https://github.com/TheFunny/TelegramTwitterMediaBot.git
synced 2026-09-23 23:32:05 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
020e2d01a3
|
||
|
|
3d6f8548c3
|
||
|
|
063e910473
|
||
|
|
b0ced34b4c
|
||
|
|
4060a88031
|
||
|
|
fb43441c56
|
||
|
|
bf628dc999
|
||
|
|
de22aa9b4d
|
||
|
|
65a9554173
|
||
|
|
e755785147
|
||
|
|
5d5b6d56e7
|
||
|
|
d0fdf1c5da
|
||
|
|
1db4ecfafa
|
||
|
|
47039bbe2d
|
||
|
|
bc954e6e0b
|
||
|
|
f7cb809e5a
|
||
|
|
51b40cdb42
|
||
|
|
ab2306002a
|
||
|
|
a92b12f633
|
||
|
|
74e3b7593c
|
||
|
|
2faccaac42
|
||
|
|
f4e60d8946
|
||
|
|
3006dcd98c
|
||
|
|
8e8acdd859
|
@@ -27,12 +27,13 @@ The `x-media` library: `site::fetch(url)` dispatches (in order) twitter → bsky
|
||||
| Path | Purpose |
|
||||
|---|---|
|
||||
| `crates/x-media/src/` | Fetch library. `site/mod.rs` = dispatcher + `Fetched`/`FetchError`/`download_media`/`media_size`; `media.rs` = `Media` enum; `examples/fetch.rs` = end-to-end usage sample |
|
||||
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, site struct, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport) |
|
||||
| `crates/x-media/src/site/<twitter\|pixiv\|bsky>/` | One directory per site: `mod.rs` (re-exports), `interface.rs` (PATTERN, `enabled()`, `fetch_from_url()`, site struct, `From<SiteStruct> for Fetched`), `model.rs` (serde DTOs). Pixiv adds `api.rs` (auth + transport); twitter adds `auth.rs` (logged-in GraphQL `TweetDetail` fallback for NSFW tweets, gated on `TWITTER_AUTH_TOKEN`) |
|
||||
| `crates/xmedia-bot/src/main.rs` | Entry point: env/log init, queue worker start, pixiv validation, 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch |
|
||||
| `crates/xmedia-bot/src/config.rs` | Manual env parsing into `Config` |
|
||||
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics |
|
||||
| `crates/xmedia-bot/src/handlers.rs` | `Command` enum (teloxide `BotCommands`), message/inline/callback handlers, URL extraction, global statics; per-URL work spawned with a `Semaphore(8)` cap (teloxide's per-chat workers are sequential — batch-forwards need concurrency) |
|
||||
| `crates/xmedia-bot/src/state.rs` | `ChatStore`: parking_lot `Mutex<HashMap>` cache + SQLite write-through (`chat_state` table) |
|
||||
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed single-worker queue (`tasks` table) |
|
||||
| `crates/xmedia-bot/src/link_cache.rs` | `LinkCache`: SQLite-backed cache (`link_cache` table) of successfully sent posts — raw caption fields + Telegram `file_id`s; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure |
|
||||
| `crates/xmedia-bot/src/queue.rs` | `PersistentTaskQueue`: SQLite-backed queue (`tasks` table), `QUEUE_WORKERS = 4` concurrent workers (lease via `BEGIN IMMEDIATE` + `locked_until` TTL), retry→dead-letter, `Notify::notify_waiters` wakeup, `busy_timeout` on all connections |
|
||||
| `crates/xmedia-bot/src/send.rs` | Media senders, upload fallback, error classification, queue task handlers |
|
||||
|
||||
## Development Commands
|
||||
@@ -67,12 +68,13 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
||||
|---|---|
|
||||
| `crates/xmedia-bot/src/main.rs` | Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via `stop_token` for docker, → sweep stop → admin msg → queue stop) |
|
||||
| `crates/xmedia-bot/src/handlers.rs` | `CHAT_STORE`/`TASK_QUEUE`/`CONFIG` singletons (open `data/task_queue.db` **relative to CWD**); command dispatch; URL extraction; retry enqueue |
|
||||
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`, `MAX_UPLOAD_BYTES = 10 MiB`; fallback chain; `classify_request_error` |
|
||||
| `crates/xmedia-bot/src/send.rs` | Constants `MAX_MEDIA_GROUP = 9`; fallback chain; `classify_request_error`; download-and-reupload fallback triggered only by Telegram API errors (`is_media_fetch_failure` / `is_size_error`) |
|
||||
| `crates/xmedia-bot/src/photo.rs` | Pure-Rust photo processing (no ffmpeg): `png` (image-png) decode/encode + `zune-jpeg` decode + `fast_image_resize` Lanczos3 downscale + `jpeg-encoder`. Photos over Telegram's limits (width + height > 10000 px → `PHOTO_INVALID_DIMENSIONS`; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL |
|
||||
| `crates/x-media/src/site/mod.rs` | Dispatcher, `Fetched`/`FetchError`, shared `CLIENT`, `download_media` (adds `Referer: https://www.pixiv.net/` for `pximg.net` hotlink protection) |
|
||||
| `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` |
|
||||
| `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime hack, static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint |
|
||||
| `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) |
|
||||
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port |
|
||||
| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) |
|
||||
| `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master; **no test step**; buildx gha cache (`cache-from`/`cache-to`, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs |
|
||||
| `README.md` | Feature docs + command table (Chinese) |
|
||||
|
||||
@@ -81,7 +83,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi
|
||||
- **Rust, stable, edition 2024**, workspace resolver 3. No `rust-version`/MSRV pin, no `rust-toolchain.toml` — recent stable is assumed. No nightly features.
|
||||
- Package manager: **Cargo** (workspace with path dep `x-media` ← `xmedia-bot`). No `[workspace.package]`/shared deps — each crate lists deps independently.
|
||||
- **Two reqwest versions coexist in the lock** (0.12.28 via teloxide, 0.13.3 in x-media) — don't unify casually.
|
||||
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
||||
- Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only).
|
||||
- SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `data/task_queue.db` is CWD-relative — run from the workspace root, or `/app` in Docker. Mount `./data` and `./cert` volumes.
|
||||
- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`.
|
||||
- Docs are in Chinese; user-facing bot strings too. Keep that convention when editing captions/templates/docs.
|
||||
|
||||
Generated
+77
-2
@@ -505,6 +505,15 @@ dependencies = [
|
||||
"syn",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "document-features"
|
||||
version = "0.2.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d4b8a88685455ed29a21542a33abd9cb6510b6b129abadabdcef0f4c55bc8f61"
|
||||
dependencies = [
|
||||
"litrs",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "dotenv"
|
||||
version = "0.15.0"
|
||||
@@ -599,12 +608,33 @@ version = "0.1.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "7360491ce676a36bf9bb3c56c1aa791658183a54d2744120f27285738d90465a"
|
||||
|
||||
[[package]]
|
||||
name = "fast_image_resize"
|
||||
version = "6.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e9c50201dc184ba6553da1695aac20a042efffbe2d84542cee31917c86c3ab1e"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"document-features",
|
||||
"num-traits",
|
||||
"thiserror",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "fastrand"
|
||||
version = "2.4.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6"
|
||||
|
||||
[[package]]
|
||||
name = "fdeflate"
|
||||
version = "0.3.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1e6853b52649d4ac5c0bd02320cddc5ba956bdb407c4b75a2c6b75bf51500f8c"
|
||||
dependencies = [
|
||||
"simd-adler32",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.9"
|
||||
@@ -1305,6 +1335,12 @@ dependencies = [
|
||||
"libc",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jpeg-encoder"
|
||||
version = "0.7.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a0370574b86f7eca156b9f298392b5e69a23f8c86f3f865add60bbc2e79467a6"
|
||||
|
||||
[[package]]
|
||||
name = "js-sys"
|
||||
version = "0.3.98"
|
||||
@@ -1352,6 +1388,12 @@ version = "0.8.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"
|
||||
|
||||
[[package]]
|
||||
name = "litrs"
|
||||
version = "1.0.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "11d3d7f243d5c5a8b9bb5d6dd2b1602c0cb0b9db1621bafc7ed66e35ff9fe092"
|
||||
|
||||
[[package]]
|
||||
name = "lock_api"
|
||||
version = "0.4.14"
|
||||
@@ -1604,6 +1646,19 @@ version = "0.3.33"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
|
||||
|
||||
[[package]]
|
||||
name = "png"
|
||||
version = "0.18.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "60769b8b31b2a9f263dae2776c37b1b28ae246943cf719eb6946a1db05128a61"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"crc32fast",
|
||||
"fdeflate",
|
||||
"flate2",
|
||||
"miniz_oxide",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "potential_utf"
|
||||
version = "0.1.5"
|
||||
@@ -3296,12 +3351,13 @@ checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
|
||||
|
||||
[[package]]
|
||||
name = "x-media"
|
||||
version = "1.0.4"
|
||||
version = "1.0.8"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"dotenv",
|
||||
"html-escape",
|
||||
"log",
|
||||
"rand 0.8.6",
|
||||
"regex",
|
||||
"reqwest 0.13.3",
|
||||
"serde",
|
||||
@@ -3314,12 +3370,15 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "xmedia-bot"
|
||||
version = "1.0.4"
|
||||
version = "1.0.8"
|
||||
dependencies = [
|
||||
"dotenv",
|
||||
"fast_image_resize",
|
||||
"html-escape",
|
||||
"jpeg-encoder",
|
||||
"log",
|
||||
"parking_lot",
|
||||
"png",
|
||||
"pretty_env_logger",
|
||||
"rand 0.8.6",
|
||||
"regex",
|
||||
@@ -3331,6 +3390,7 @@ dependencies = [
|
||||
"tokio",
|
||||
"url",
|
||||
"x-media",
|
||||
"zune-jpeg",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3534,3 +3594,18 @@ dependencies = [
|
||||
"cc",
|
||||
"pkg-config",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zune-core"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "cb8a0807f7c01457d0379ba880ba6322660448ddebc890ce29bb64da71fb40f9"
|
||||
|
||||
[[package]]
|
||||
name = "zune-jpeg"
|
||||
version = "0.5.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "27bc9d5b815bc103f142aa054f561d9187d191692ec7c2d1e2b4737f8dbd7296"
|
||||
dependencies = [
|
||||
"zune-core",
|
||||
]
|
||||
|
||||
@@ -10,6 +10,7 @@ Telegram 机器人,将 X / Twitter、Pixiv、Bluesky 的帖子链接转换为
|
||||
- 可绑定转发频道自动转发;支持转发前编辑 caption 与自定义模板
|
||||
- 发送失败自动重试并持久化,重试耗尽后通知用户
|
||||
- Pixiv ugoira 动图自动转码为 MP4
|
||||
- 链接结果本地缓存:成功发送后缓存 Telegram file id 与 caption 等,再次收到相同链接直接本地重发,不再请求源站、不保存媒体文件(`LINK_CACHE_TTL_SECONDS` 控制过期,默认 7 天)
|
||||
|
||||
## 快速开始
|
||||
|
||||
@@ -28,7 +29,9 @@ docker build -t tgxmb .
|
||||
docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||
```
|
||||
|
||||
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`RUST_LOG`、`WEBHOOK*`。
|
||||
环境变量:`TELOXIDE_TOKEN`(必填)、`PIXIV_REFRESH_TOKEN`、`BOT_ADMIN`、`EDIT_MESSAGE_TTL_SECONDS`、`LINK_CACHE_TTL_SECONDS`、`RUST_LOG`、`WEBHOOK*`、`TWITTER_AUTH_TOKEN`(可选)。
|
||||
|
||||
NSFW 推文:公开的 syndication 接口不返回敏感内容。设置 `TWITTER_AUTH_TOKEN`(登录 x.com 后浏览器 Cookie 里的 `auth_token` 值)后,bot 会仅在遇到 NSFW 推文时以登录态获取媒体;未设置则提示无媒体。
|
||||
|
||||
### Webhook 部署(需要反向代理)
|
||||
|
||||
@@ -36,16 +39,37 @@ docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb
|
||||
|
||||
**有域名**
|
||||
1. DNS A 记录指向服务器
|
||||
2. compose 里设 `VIRTUAL_HOST`、`LETSENCRYPT_HOST` 为域名,`WEBHOOK_URL` 设为 `https://域名/`
|
||||
3. 证书自动签发与续期,无需手动处理
|
||||
2. compose 里设 `VIRTUAL_HOST`、`WEBHOOK_URL` 为域名,并取消注释 `ACME_HOST`(设为域名)
|
||||
3. acme-companion 自动签发与续期证书,无需手动处理
|
||||
|
||||
**只有 IP**
|
||||
1. 生成自签证书(PEM 格式,见第 3 步):
|
||||
`openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout nginx-certs/default.key -out nginx-certs/default.crt`
|
||||
2. compose 里 nginx-proxy 设 `DEFAULT_HOST`,bot 设 `WEBHOOK_CERT: './cert/cert.pem'`(须与代理所服务的为同一张证书)
|
||||
3. 证书必须是 PEM 编码(ASCII BASE64,以 `-----BEGIN CERTIFICATE-----` 开头)—— Telegram 只接受该格式;若现有证书是 DER 二进制,转换:
|
||||
`openssl x509 -in cert.der -inform DER -out cert.pem -outform PEM`
|
||||
(私钥同理:`openssl rsa -in key.der -inform DER -out key.pem -outform PEM`)
|
||||
Let's Encrypt 支持为公网 IP 签发证书(2026 年起可用,有效期约 7 天,须 `shortlived` profile)。用 [acme.sh](https://github.com/acmesh-official/acme.sh) 自动签发与续期,无需手动证书:
|
||||
|
||||
1. compose 里增加 acme-ip 服务(签发 + 每日检查自动续期):
|
||||
```yaml
|
||||
acme-ip:
|
||||
image: neilpang/acme.sh
|
||||
container_name: acme-ip
|
||||
command: daemon
|
||||
restart: always
|
||||
volumes:
|
||||
- certs:/acme.sh
|
||||
- html:/usr/share/nginx/html
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
networks: [proxy]
|
||||
```
|
||||
2. 首次签发(把 `<SERVER_IP>` 换成服务器公网 IP,IPv6 同样支持,多个 `-d` 可并列):
|
||||
```bash
|
||||
docker compose exec acme-ip acme.sh --issue --server letsencrypt \
|
||||
-d <SERVER_IP> --cert-profile shortlived --days 3 \
|
||||
--webroot /usr/share/nginx/html \
|
||||
--install-cert --cert-file /acme.sh/<SERVER_IP>.crt \
|
||||
--key-file /acme.sh/<SERVER_IP>.key \
|
||||
--reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP"
|
||||
```
|
||||
3. compose 里设 `VIRTUAL_HOST: '<SERVER_IP>'`、`WEBHOOK_URL: 'https://<SERVER_IP>/'`,无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成(`--days 3` = 每 3 天续一次,证书 7 天有效有缓冲),续期成功后自动 HUP 通知 nginx-proxy 加载新证书。
|
||||
|
||||
限制:证书约 7 天有效;验证仅支持 http-01/tls-alpn-01(80 端口必须公网可达);不支持 DNS-01、私有 IP 与 IP 段;同一 IP 集合每 168 小时限签发 5 张。建议先用 `--server letsencrypt_test` 试签,成功后再切正式服务器。
|
||||
|
||||
Telegram 只接受 443/80/88/8443 端口。
|
||||
|
||||
@@ -58,17 +82,17 @@ Telegram 只接受 443/80/88/8443 端口。
|
||||
| `PIXIV_REFRESH_TOKEN` | Pixiv 刷新令牌;未设置则禁用 Pixiv |
|
||||
| `BOT_ADMIN` | 管理员聊天 ID,逗号分隔;接收启动/停止通知 |
|
||||
| `EDIT_MESSAGE_TTL_SECONDS` | 转发前编辑记录过期秒数,默认 86400 |
|
||||
| `LINK_CACHE_TTL_SECONDS` | 链接结果缓存过期秒数,默认 604800(7 天) |
|
||||
| `RUST_LOG` | 日志级别 |
|
||||
| `LOCAL_USER_ID` | 容器内运行用户 UID,默认 9001 |
|
||||
| `VIRTUAL_HOST` | 对外域名或 IP,nginx-proxy 按此路由 |
|
||||
| `VIRTUAL_PORT` | bot 容器内监听端口,nginx-proxy 的转发目标 |
|
||||
| `LETSENCRYPT_HOST` | 设为域名时由 acme-companion 自动签发/续期证书 |
|
||||
| `ACME_HOST` | 域名部署:设为域名时由 acme-companion 自动签发/续期证书 |
|
||||
| `DEFAULT_HOST` | nginx-proxy 将未知 Host 的请求路由到该 vhost(IP 访问时需要) |
|
||||
| `DEFAULT_EMAIL` | acme-companion 证书通知邮箱 |
|
||||
| `WEBHOOK` | `true` 启用 webhook 模式(默认轮询) |
|
||||
| `WEBHOOK_LISTEN` / `WEBHOOK_PORT` | bot 容器内监听地址/端口 |
|
||||
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/`) |
|
||||
| `WEBHOOK_CERT` | 自签证书路径(仅 IP 路径需要,须为 PEM 且与代理所服务的一致) |
|
||||
| `WEBHOOK_URL` | 对外公网 HTTPS 地址(`https://域名/` 或 `https://IP/`) |
|
||||
| `WEBHOOK_SECRET_TOKEN` | 更新校验令牌(`X-Telegram-Bot-Api-Secret-Token`) |
|
||||
|
||||
</details>
|
||||
@@ -77,17 +101,19 @@ Telegram 只接受 443/80/88/8443 端口。
|
||||
|
||||
| 命令 | 说明 |
|
||||
|---|---|
|
||||
| `/set_forward_channel <频道>` | 设置转发频道 |
|
||||
| `/start` | 欢迎语 |
|
||||
| `/help` | 查看全部命令及用法(即本文档的命令表) |
|
||||
| `/set_forward_channel <频道>` | 设置转发频道,参数为 `@频道名` 或频道 ID;设置后发送的媒体消息会自动转发到该频道 |
|
||||
| `/remove_forward_channel` | 取消转发频道 |
|
||||
| `/edit_before_forward` | 开关转发前编辑 |
|
||||
| `/set_template <名称>` | 将回复的消息(含 `[]`)保存为模板 |
|
||||
| `/set_format <站点> <格式>` | 自定义 caption 格式(占位符 `{url}` `{title}` `{tags}` 等) |
|
||||
| `/bot_dict` | 查看聊天状态 |
|
||||
| `/edit_before_forward` | 开关「转发前编辑」:开启后,转发成功后 bot 会发一条提示消息,回复它可修改第一条转发消息的 caption(或点击模板按钮套用模板) |
|
||||
| `/set_template <名称>` | 回复一条含 `[]` 的消息,将其保存为命名模板;转发时 `[]` 会被替换为原帖链接(配合「转发前编辑」使用) |
|
||||
| `/set_format <站点> <格式>` | 自定义某站点的 caption 格式。站点:`twitter` / `bsky` / `pixiv`。占位符:`{url}` `{author}` `{author_url}` `{title}` `{tags}` |
|
||||
| `/bot_dict` | 查看当前聊天状态(调试用) |
|
||||
|
||||
链接处理仅限私聊;命令在任意聊天可用。
|
||||
|
||||
## 备注
|
||||
|
||||
- 数据持久化于 `data/task_queue.db`,容器部署需挂载该目录
|
||||
- 数据持久化于 `data/task_queue.db`,compose 部署使用 bind mount `./data`(保持目录形式便于备份)
|
||||
- 运行环境需安装 ffmpeg(Docker 镜像已内置)
|
||||
- 测试:`cargo test --workspace`
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "x-media"
|
||||
version = "1.0.4"
|
||||
version = "1.0.8"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
@@ -13,6 +13,7 @@ url = "2.5.2"
|
||||
bytes = "1"
|
||||
zip = "2"
|
||||
tempfile = "3"
|
||||
rand = "0.8"
|
||||
log = "0.4"
|
||||
tokio = { version = "1.40", features = ["time"] }
|
||||
|
||||
|
||||
@@ -5,9 +5,8 @@ use html_escape::encode_text;
|
||||
use regex::Regex;
|
||||
use std::sync::LazyLock;
|
||||
|
||||
pub static PATTERN: LazyLock<Regex> = LazyLock::new(|| {
|
||||
Regex::new(r"bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap()
|
||||
});
|
||||
pub static PATTERN: LazyLock<Regex> =
|
||||
LazyLock::new(|| Regex::new(r"bsky\.app/profile/([\w.\-:]+)/post/([\w.\-~]+)").unwrap());
|
||||
|
||||
pub fn enabled() -> bool {
|
||||
true
|
||||
@@ -15,8 +14,14 @@ pub fn enabled() -> bool {
|
||||
|
||||
pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
let caps = PATTERN.captures(url).ok_or(FetchError::NotFound)?;
|
||||
let handle = caps.get(1).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
|
||||
let rkey = caps.get(2).map(|m| m.as_str()).ok_or(FetchError::NotFound)?;
|
||||
let handle = caps
|
||||
.get(1)
|
||||
.map(|m| m.as_str())
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
let rkey = caps
|
||||
.get(2)
|
||||
.map(|m| m.as_str())
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
Ok(fetch(handle, rkey).await?.into())
|
||||
}
|
||||
|
||||
@@ -197,7 +202,10 @@ mod tests {
|
||||
}));
|
||||
let post = Post::from_json(&raw.to_string(), "3xxxx".into()).unwrap();
|
||||
let fetched: Fetched = post.into();
|
||||
assert_eq!(fetched.source_url, "https://bsky.app/profile/user.bsky.social/post/3xxxx");
|
||||
assert_eq!(
|
||||
fetched.source_url,
|
||||
"https://bsky.app/profile/user.bsky.social/post/3xxxx"
|
||||
);
|
||||
assert_eq!(fetched.title, "hello <world>");
|
||||
assert_eq!(fetched.media.len(), 1);
|
||||
assert!(!fetched.sensitive);
|
||||
@@ -246,11 +254,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn live_fetch_with_photos() {
|
||||
let fetched = fetch_from_url(
|
||||
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m",
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
let fetched =
|
||||
fetch_from_url("https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
fetched.source_url,
|
||||
"https://bsky.app/profile/asagi0398.bsky.social/post/3mqkhrq5w6k2m"
|
||||
@@ -260,9 +267,10 @@ mod tests {
|
||||
|
||||
#[tokio::test]
|
||||
async fn live_fetch_smoke() {
|
||||
let fetched = fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
|
||||
.await
|
||||
.unwrap();
|
||||
let fetched =
|
||||
fetch_from_url("https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224")
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
fetched.source_url,
|
||||
"https://bsky.app/profile/fu-futa.bsky.social/post/3laoveufjv224"
|
||||
|
||||
+118
-11
@@ -68,18 +68,73 @@ impl Fetched {
|
||||
/// caption.
|
||||
pub fn caption_with(&self, format: &str) -> String {
|
||||
match (&self.render_data, format.is_empty()) {
|
||||
(Some(data), false) => {
|
||||
let escaped = html_escape::encode_text(format).into_owned();
|
||||
escaped
|
||||
.replace("{url}", &data.url)
|
||||
.replace("{author}", &data.author)
|
||||
.replace("{author_url}", &data.author_url)
|
||||
.replace("{title}", &data.title)
|
||||
.replace("{tags}", &data.tags)
|
||||
}
|
||||
(Some(data), false) => caption_from_fields(
|
||||
format,
|
||||
"",
|
||||
&data.url,
|
||||
&data.author,
|
||||
&data.author_url,
|
||||
&data.title,
|
||||
&data.tags,
|
||||
),
|
||||
_ => self.caption.clone(),
|
||||
}
|
||||
}
|
||||
|
||||
/// The pre-escaped placeholder values (author, author_url, title, tags)
|
||||
/// a caller needs to rebuild a caption later, e.g. for a cached post
|
||||
/// where the [`Fetched`] is no longer available.
|
||||
pub fn render_fields(&self) -> Option<(&str, &str, &str, &str)> {
|
||||
self.render_data.as_ref().map(|d| {
|
||||
(
|
||||
d.author.as_str(),
|
||||
d.author_url.as_str(),
|
||||
d.title.as_str(),
|
||||
d.tags.as_str(),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Renders a user-supplied caption format from raw (already-escaped) field
|
||||
/// values with the same escaping/substitution rules as
|
||||
/// [`Fetched::caption_with`]. An empty format returns `built_in` unchanged.
|
||||
pub fn caption_from_fields(
|
||||
format: &str,
|
||||
built_in: &str,
|
||||
url: &str,
|
||||
author: &str,
|
||||
author_url: &str,
|
||||
title: &str,
|
||||
tags: &str,
|
||||
) -> String {
|
||||
if format.is_empty() {
|
||||
return built_in.to_string();
|
||||
}
|
||||
let escaped = html_escape::encode_text(format).into_owned();
|
||||
escaped
|
||||
.replace("{url}", url)
|
||||
.replace("{author}", author)
|
||||
.replace("{author_url}", author_url)
|
||||
.replace("{title}", title)
|
||||
.replace("{tags}", tags)
|
||||
}
|
||||
|
||||
/// Stable per-post cache key derived from any supported URL, so variant
|
||||
/// domains (x.com / twitter.com / fxtwitter.com, mobile, `/photo/N`
|
||||
/// suffixes) map to the same post. Returns `"twitter:<id>"`,
|
||||
/// `"pixiv:<id>"` or `"bsky:<handle>/<rkey>"`.
|
||||
pub fn cache_key(url: &str) -> Option<String> {
|
||||
if let Some(caps) = twitter::PATTERN.captures(url) {
|
||||
return Some(format!("twitter:{}", &caps[1]));
|
||||
}
|
||||
if let Some(caps) = pixiv::PATTERN.captures(url) {
|
||||
return Some(format!("pixiv:{}", &caps[1]));
|
||||
}
|
||||
if let Some(caps) = bsky::PATTERN.captures(url) {
|
||||
return Some(format!("bsky:{}/{}", &caps[1], &caps[2]));
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -89,6 +144,9 @@ pub enum FetchError {
|
||||
Pixiv(PixivError),
|
||||
NotFound,
|
||||
Blocked,
|
||||
/// The post exists but its content is withheld (twitter NSFW /
|
||||
/// age-restricted tweets come back as an empty `{}` from syndication).
|
||||
Sensitive,
|
||||
}
|
||||
|
||||
impl fmt::Display for FetchError {
|
||||
@@ -99,6 +157,7 @@ impl fmt::Display for FetchError {
|
||||
FetchError::Pixiv(e) => write!(f, "pixiv error: {e}"),
|
||||
FetchError::NotFound => write!(f, "not found"),
|
||||
FetchError::Blocked => write!(f, "blocked"),
|
||||
FetchError::Sensitive => write!(f, "content withheld (sensitive)"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -109,7 +168,7 @@ impl std::error::Error for FetchError {
|
||||
FetchError::Http(e) => Some(e),
|
||||
FetchError::Json(e) => Some(e),
|
||||
FetchError::Pixiv(e) => Some(e),
|
||||
FetchError::NotFound | FetchError::Blocked => None,
|
||||
FetchError::NotFound | FetchError::Blocked | FetchError::Sensitive => None,
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -120,7 +179,6 @@ impl From<reqwest::Error> for FetchError {
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
impl From<serde_json::Error> for FetchError {
|
||||
fn from(e: serde_json::Error) -> Self {
|
||||
FetchError::Json(e)
|
||||
@@ -222,6 +280,55 @@ pub async fn download_media(url: &str) -> Result<bytes::Bytes, FetchError> {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn cache_key_normalizes_domain_variants() {
|
||||
assert_eq!(
|
||||
cache_key("https://x.com/user/status/1234567890/photo/1"),
|
||||
Some("twitter:1234567890".into())
|
||||
);
|
||||
assert_eq!(
|
||||
cache_key("https://mobile.twitter.com/user/status/1234567890"),
|
||||
Some("twitter:1234567890".into())
|
||||
);
|
||||
assert_eq!(
|
||||
cache_key("https://fxtwitter.com/user/status/1234567890"),
|
||||
Some("twitter:1234567890".into())
|
||||
);
|
||||
assert_eq!(
|
||||
cache_key("https://www.pixiv.net/artworks/123456"),
|
||||
Some("pixiv:123456".into())
|
||||
);
|
||||
assert_eq!(
|
||||
cache_key("https://bsky.app/profile/handle.example/post/3lorem"),
|
||||
Some("bsky:handle.example/3lorem".into())
|
||||
);
|
||||
assert_eq!(cache_key("https://example.com/not-a-post"), None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn caption_from_fields_substitutes_and_escapes() {
|
||||
// The format string is escaped, the field values are substituted
|
||||
// verbatim (callers pass the already-escaped render data).
|
||||
let out = caption_from_fields(
|
||||
"see {author} at {url} — {title}",
|
||||
"",
|
||||
"https://x.com/u/status/1",
|
||||
"A & B",
|
||||
"https://x.com/u",
|
||||
"hello <world>",
|
||||
"",
|
||||
);
|
||||
assert_eq!(
|
||||
out,
|
||||
"see A & B at https://x.com/u/status/1 — hello <world>"
|
||||
);
|
||||
// Empty format keeps the built-in caption untouched.
|
||||
assert_eq!(
|
||||
caption_from_fields("", "built-in", "u", "a", "au", "t", "g"),
|
||||
"built-in"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unsupported_url_returns_none() {
|
||||
let result = fetch("https://example.com/some/article").await;
|
||||
|
||||
@@ -10,8 +10,8 @@ use crate::site::FetchError;
|
||||
use std::env;
|
||||
use std::fmt;
|
||||
use std::io::{Cursor, Read};
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::LazyLock;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, SystemTime};
|
||||
|
||||
const AUTH_TOKEN_URL: &str = "https://oauth.secure.pixiv.net/auth/token";
|
||||
@@ -127,7 +127,9 @@ impl PixivAPI {
|
||||
pub async fn illust_detail(&self, illust_id: u64) -> Result<IllustrationModel, PixivError> {
|
||||
let access_token = self.get_access_token().await?;
|
||||
let response = crate::site::CLIENT
|
||||
.get(format!("{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"))
|
||||
.get(format!(
|
||||
"{APP_API_URL}/v1/illust/detail?illust_id={illust_id}"
|
||||
))
|
||||
.header("app-os", "ios")
|
||||
.header("app-os-version", "14.6")
|
||||
.header("User-Agent", APP_USER_AGENT)
|
||||
@@ -175,7 +177,9 @@ impl PixivAPI {
|
||||
pub async fn ugoira_metadata(&self, illust_id: u64) -> Result<UgoiraMetadataModel, PixivError> {
|
||||
let access_token = self.get_access_token().await?;
|
||||
let response = crate::site::CLIENT
|
||||
.get(format!("{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"))
|
||||
.get(format!(
|
||||
"{APP_API_URL}/v1/ugoira/metadata?illust_id={illust_id}"
|
||||
))
|
||||
.header("app-os", "ios")
|
||||
.header("app-os-version", "14.6")
|
||||
.header("User-Agent", APP_USER_AGENT)
|
||||
@@ -218,87 +222,89 @@ impl PixivAPI {
|
||||
let Some(zip_url) = zip_url else {
|
||||
return Ok(None);
|
||||
};
|
||||
let zip_bytes = crate::site::download_media(&zip_url).await.map_err(|e| match e {
|
||||
FetchError::Http(e) => PixivError::Http(e),
|
||||
other => PixivError::Api(format!("frame zip download failed: {other}")),
|
||||
})?;
|
||||
let zip_bytes = crate::site::download_media(&zip_url)
|
||||
.await
|
||||
.map_err(|e| match e {
|
||||
FetchError::Http(e) => PixivError::Http(e),
|
||||
other => PixivError::Api(format!("frame zip download failed: {other}")),
|
||||
})?;
|
||||
let frame_delays = metadata.frames.iter().map(|f| f.delay).collect::<Vec<_>>();
|
||||
let result = tokio::task::spawn_blocking(
|
||||
move || -> Result<(String, tempfile::TempDir), String> {
|
||||
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
|
||||
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
|
||||
let result =
|
||||
tokio::task::spawn_blocking(move || -> Result<(String, tempfile::TempDir), String> {
|
||||
let frames_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
|
||||
let out_dir = tempfile::tempdir().map_err(|e| e.to_string())?;
|
||||
|
||||
// Extract frames to canonical zero-padded names; pixiv ugoira
|
||||
// frames are uniformly jpg or png per artwork.
|
||||
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
|
||||
.map_err(|e| format!("unzip: {e}"))?;
|
||||
// pixiv ugoira frames are uniformly jpg or png per artwork; take
|
||||
// the extension from the first entry.
|
||||
let extension = if archive.len() > 0 {
|
||||
let first_name = archive
|
||||
.by_index(0)
|
||||
.map_err(|e| e.to_string())?
|
||||
.name()
|
||||
.to_string();
|
||||
first_name
|
||||
.rsplit('.')
|
||||
.next()
|
||||
.unwrap_or("jpg")
|
||||
.to_string()
|
||||
} else {
|
||||
"jpg".to_string()
|
||||
};
|
||||
let mut count = 0usize;
|
||||
for i in 0..archive.len() {
|
||||
let mut entry = archive.by_index(i).map_err(|e| e.to_string())?;
|
||||
let mut bytes = Vec::new();
|
||||
entry.read_to_end(&mut bytes).map_err(|e| e.to_string())?;
|
||||
let path = frames_dir.path().join(format!("img_{count:05}.{extension}"));
|
||||
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
|
||||
count += 1;
|
||||
}
|
||||
if count == 0 {
|
||||
return Err("empty frame zip".to_string());
|
||||
}
|
||||
// Extract frames to canonical zero-padded names; pixiv ugoira
|
||||
// frames are uniformly jpg or png per artwork.
|
||||
let mut archive = zip::ZipArchive::new(Cursor::new(zip_bytes))
|
||||
.map_err(|e| format!("unzip: {e}"))?;
|
||||
// pixiv ugoira frames are uniformly jpg or png per artwork; take
|
||||
// the extension from the first entry.
|
||||
let extension = if archive.len() > 0 {
|
||||
let first_name = archive
|
||||
.by_index(0)
|
||||
.map_err(|e| e.to_string())?
|
||||
.name()
|
||||
.to_string();
|
||||
first_name.rsplit('.').next().unwrap_or("jpg").to_string()
|
||||
} else {
|
||||
"jpg".to_string()
|
||||
};
|
||||
let mut count = 0usize;
|
||||
for i in 0..archive.len() {
|
||||
let mut entry = archive.by_index(i).map_err(|e| e.to_string())?;
|
||||
let mut bytes = Vec::new();
|
||||
entry.read_to_end(&mut bytes).map_err(|e| e.to_string())?;
|
||||
let path = frames_dir
|
||||
.path()
|
||||
.join(format!("img_{count:05}.{extension}"));
|
||||
std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
|
||||
count += 1;
|
||||
}
|
||||
if count == 0 {
|
||||
return Err("empty frame zip".to_string());
|
||||
}
|
||||
|
||||
// Constant rate from the median frame delay (ms).
|
||||
let mut delays = frame_delays;
|
||||
delays.sort_unstable();
|
||||
let median = delays[delays.len() / 2].max(1);
|
||||
let framerate = 1000.0 / median as f64;
|
||||
// Constant rate from the median frame delay (ms).
|
||||
let mut delays = frame_delays;
|
||||
delays.sort_unstable();
|
||||
let median = delays[delays.len() / 2].max(1);
|
||||
let framerate = 1000.0 / median as f64;
|
||||
|
||||
let output = out_dir.path().join("ugoira.mp4");
|
||||
let status = std::process::Command::new("ffmpeg")
|
||||
.args([
|
||||
"-y",
|
||||
"-framerate",
|
||||
&framerate.to_string(),
|
||||
"-i",
|
||||
&frames_dir.path().join(format!("img_%05d.{extension}")).to_string_lossy(),
|
||||
// libx264 needs even dimensions; pixiv ugoira frames can
|
||||
// be odd-sized (e.g. 277x405).
|
||||
"-vf",
|
||||
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
&output.to_string_lossy(),
|
||||
])
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.status()
|
||||
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
|
||||
if !status.success() {
|
||||
return Err(format!("ffmpeg exited with {status}"));
|
||||
}
|
||||
Ok((output.to_string_lossy().into_owned(), out_dir))
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("ugoira encode worker panicked");
|
||||
let output = out_dir.path().join("ugoira.mp4");
|
||||
let status = std::process::Command::new("ffmpeg")
|
||||
.args([
|
||||
"-y",
|
||||
"-framerate",
|
||||
&framerate.to_string(),
|
||||
"-i",
|
||||
&frames_dir
|
||||
.path()
|
||||
.join(format!("img_%05d.{extension}"))
|
||||
.to_string_lossy(),
|
||||
// libx264 needs even dimensions; pixiv ugoira frames can
|
||||
// be odd-sized (e.g. 277x405).
|
||||
"-vf",
|
||||
"scale=trunc(iw/2)*2:trunc(ih/2)*2",
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-movflags",
|
||||
"+faststart",
|
||||
&output.to_string_lossy(),
|
||||
])
|
||||
.stdout(std::process::Stdio::null())
|
||||
.stderr(std::process::Stdio::null())
|
||||
.status()
|
||||
.map_err(|e| format!("ffmpeg spawn failed: {e}"))?;
|
||||
if !status.success() {
|
||||
return Err(format!("ffmpeg exited with {status}"));
|
||||
}
|
||||
Ok((output.to_string_lossy().into_owned(), out_dir))
|
||||
})
|
||||
.await
|
||||
.expect("ugoira encode worker panicked");
|
||||
match result {
|
||||
Ok(pair) => Ok(Some(pair)),
|
||||
Err(message) => {
|
||||
@@ -332,9 +338,8 @@ fn log_once_ffmpeg_missing() {
|
||||
}
|
||||
|
||||
/// pixiv3-rs replacement: `None` when `PIXIV_REFRESH_TOKEN` is unset.
|
||||
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> = LazyLock::new(|| {
|
||||
env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new)
|
||||
});
|
||||
static PIXIV_CLIENT: LazyLock<Option<PixivAPI>> =
|
||||
LazyLock::new(|| env::var("PIXIV_REFRESH_TOKEN").ok().map(PixivAPI::new));
|
||||
|
||||
/// Set at startup when the login validation fails; pixiv stays disabled until
|
||||
/// the next process start.
|
||||
|
||||
@@ -82,12 +82,15 @@ impl Illustration {
|
||||
// keeps media empty when encoding fails or ffmpeg is missing.
|
||||
} else if model.page_count > 1 {
|
||||
media.extend(model.meta_pages.iter().filter_map(|page| {
|
||||
page.image_urls.original.clone().map(|original| Media::Illustration {
|
||||
title: None,
|
||||
url: original,
|
||||
thumbnail_url: Some(page.image_urls.medium.clone()),
|
||||
fallback_url: Some(page.image_urls.large.clone()),
|
||||
})
|
||||
page.image_urls
|
||||
.original
|
||||
.clone()
|
||||
.map(|original| Media::Illustration {
|
||||
title: None,
|
||||
url: original,
|
||||
thumbnail_url: Some(page.image_urls.medium.clone()),
|
||||
fallback_url: Some(page.image_urls.large.clone()),
|
||||
})
|
||||
}));
|
||||
} else if let Some(original) = model
|
||||
.meta_single_page
|
||||
@@ -147,8 +150,8 @@ impl From<Illustration> for Fetched {
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use super::super::model::IllustrationModel;
|
||||
use super::*;
|
||||
|
||||
fn illust_json(
|
||||
type_: &str,
|
||||
@@ -203,8 +206,14 @@ mod tests {
|
||||
("https://pixiv.net/artworks/123456", "123456"),
|
||||
("https://www.pixiv.net/en/artworks/123456", "123456"),
|
||||
("https://www.pixiv.net/i/123456", "123456"),
|
||||
("https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456", "123456"),
|
||||
("https://www.pixiv.net/en/member_illust.php?illust_id=123456", "123456"),
|
||||
(
|
||||
"https://www.pixiv.net/member_illust.php?mode=medium&illust_id=123456",
|
||||
"123456",
|
||||
),
|
||||
(
|
||||
"https://www.pixiv.net/en/member_illust.php?illust_id=123456",
|
||||
"123456",
|
||||
),
|
||||
];
|
||||
for (url, id) in cases {
|
||||
let caps = PATTERN.captures(url).unwrap_or_else(|| panic!("{url}"));
|
||||
@@ -225,7 +234,14 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn ugoira_yields_empty_media() {
|
||||
let v = illust_json("ugoira", 1, Some("https://i.pximg.net/orig.jpg"), None, vec![], 0);
|
||||
let v = illust_json(
|
||||
"ugoira",
|
||||
1,
|
||||
Some("https://i.pximg.net/orig.jpg"),
|
||||
None,
|
||||
vec![],
|
||||
0,
|
||||
);
|
||||
let illustration = parse(v);
|
||||
let fetched: Fetched = illustration.into();
|
||||
assert!(fetched.media.is_empty());
|
||||
@@ -296,7 +312,12 @@ mod tests {
|
||||
let fetched: Fetched = parse(v).into();
|
||||
assert_eq!(fetched.media.len(), 1);
|
||||
match &fetched.media[0] {
|
||||
Media::Illustration { url, thumbnail_url, fallback_url, .. } => {
|
||||
Media::Illustration {
|
||||
url,
|
||||
thumbnail_url,
|
||||
fallback_url,
|
||||
..
|
||||
} => {
|
||||
assert_eq!(url, "https://i.pximg.net/p2.jpg");
|
||||
assert_eq!(thumbnail_url.as_deref(), Some("m2.jpg"));
|
||||
assert_eq!(fallback_url.as_deref(), Some("l2.jpg"));
|
||||
@@ -307,7 +328,14 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn caption_with_escapes_format_and_substitutes() {
|
||||
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
|
||||
let v = illust_json(
|
||||
"illust",
|
||||
1,
|
||||
Some("https://i.pximg.net/o.jpg"),
|
||||
None,
|
||||
vec![],
|
||||
0,
|
||||
);
|
||||
let fetched: Fetched = parse(v).into();
|
||||
// Format string is escaped in full, then placeholders substituted.
|
||||
let out = fetched.caption_with("{title} by {author} <script> {tags}");
|
||||
@@ -330,7 +358,14 @@ mod tests {
|
||||
#[test]
|
||||
fn ai_work_gets_leading_ai_tag() {
|
||||
// illust_ai_type == 2 is the only AI marker.
|
||||
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 2);
|
||||
let v = illust_json(
|
||||
"illust",
|
||||
1,
|
||||
Some("https://i.pximg.net/o.jpg"),
|
||||
None,
|
||||
vec![],
|
||||
2,
|
||||
);
|
||||
let fetched: Fetched = parse(v).into();
|
||||
assert!(
|
||||
fetched.caption.contains("#AI #tag1 #tag2"),
|
||||
@@ -338,14 +373,25 @@ mod tests {
|
||||
fetched.caption
|
||||
);
|
||||
// The {tags} placeholder reflects the tag array too.
|
||||
assert!(fetched.caption_with("{tags}").starts_with("#AI "), "got: {}", fetched.caption_with("{tags}"));
|
||||
assert!(
|
||||
fetched.caption_with("{tags}").starts_with("#AI "),
|
||||
"got: {}",
|
||||
fetched.caption_with("{tags}")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn non_ai_work_has_no_ai_tag() {
|
||||
// 1 = explicitly not AI, 0 = undefined: neither gets the #AI tag.
|
||||
for ai_type in [0, 1] {
|
||||
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], ai_type);
|
||||
let v = illust_json(
|
||||
"illust",
|
||||
1,
|
||||
Some("https://i.pximg.net/o.jpg"),
|
||||
None,
|
||||
vec![],
|
||||
ai_type,
|
||||
);
|
||||
let fetched: Fetched = parse(v).into();
|
||||
assert!(
|
||||
!fetched.caption.contains("#AI"),
|
||||
@@ -357,7 +403,14 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn caption_escapes_and_links() {
|
||||
let v = illust_json("illust", 1, Some("https://i.pximg.net/o.jpg"), None, vec![], 0);
|
||||
let v = illust_json(
|
||||
"illust",
|
||||
1,
|
||||
Some("https://i.pximg.net/o.jpg"),
|
||||
None,
|
||||
vec![],
|
||||
0,
|
||||
);
|
||||
let fetched: Fetched = parse(v).into();
|
||||
assert!(
|
||||
fetched
|
||||
@@ -367,9 +420,6 @@ mod tests {
|
||||
fetched.caption
|
||||
);
|
||||
assert!(fetched.caption.contains("#tag1 #tag2"));
|
||||
assert_eq!(
|
||||
fetched.source_url,
|
||||
"https://www.pixiv.net/artworks/123"
|
||||
);
|
||||
assert_eq!(fetched.source_url, "https://www.pixiv.net/artworks/123");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3,4 +3,4 @@ mod interface;
|
||||
mod model;
|
||||
|
||||
pub use api::{PixivAPI, PixivError, disable, fetch, validate};
|
||||
pub use interface::{PATTERN, Illustration, enabled, fetch_from_url};
|
||||
pub use interface::{Illustration, PATTERN, enabled, fetch_from_url};
|
||||
|
||||
@@ -0,0 +1,379 @@
|
||||
//! Authenticated fallback for tweets the public syndication endpoint refuses
|
||||
//! to serve (NSFW / age-restricted tweets come back as an empty `{}`).
|
||||
//!
|
||||
//! Mirrors nazurin's web API client ([`web.py`]) and is used *only* when
|
||||
//! syndication reports [`FetchError::Sensitive`]: the private GraphQL
|
||||
//! `TweetDetail` endpoint, authenticated with a browser session cookie from
|
||||
//! `TWITTER_AUTH_TOKEN` (the `auth_token` cookie value of a logged-in x.com
|
||||
//! session). A fresh random `ct0` is generated per call; X checks that the
|
||||
//! `x-csrf-token` header matches the cookie, not that it issued the value.
|
||||
//!
|
||||
//! [`web.py`]: https://github.com/y-young/nazurin/blob/master/nazurin/sites/twitter/api/web.py
|
||||
//!
|
||||
//! # Caveats
|
||||
//! - X rotates the GraphQL query id when it rolls the web app; if requests
|
||||
//! start failing, update [`TWEET_DETAIL_QUERY_ID`]. Fresh references from
|
||||
//! the actively maintained FxEmbed/FxEmbed: TweetDetail
|
||||
//! `R9IzzyzQBV87-DOWpcvDmw`, TweetResultByRestId `f2sagi1jweVHFkTUIHzmMQ`
|
||||
//! (the latter is anonymous and surfaces NSFW tweets as
|
||||
//! `reason: NsfwLoggedOut`).
|
||||
//! - `x-client-transaction-id` is only required for `SearchTimeline`
|
||||
//! (verified against FxEmbed's `proxy/allowlist.ts`) — TweetDetail works
|
||||
//! without it; no need for the nazurin home-page/JS-bundle derivation.
|
||||
|
||||
use std::sync::LazyLock;
|
||||
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::site::FetchError;
|
||||
|
||||
use super::interface::Tweet;
|
||||
|
||||
/// `auth_token` cookie of a logged-in x.com session; enables the fallback.
|
||||
/// Trimmed: a CRLF `.env` (Windows) leaves a trailing `\r` on the value,
|
||||
/// which would make the Cookie header invalid.
|
||||
static AUTH_TOKEN: LazyLock<Option<String>> = LazyLock::new(|| {
|
||||
std::env::var("TWITTER_AUTH_TOKEN")
|
||||
.ok()
|
||||
.map(|s| s.trim().to_string())
|
||||
.filter(|s| !s.is_empty())
|
||||
});
|
||||
|
||||
/// Public "logged in" client token used by the x.com web app.
|
||||
const LOGGED_IN_BEARER: &str = "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA";
|
||||
|
||||
/// `TweetDetail` query id (from nazurin; still valid as of 2026-08,
|
||||
/// corroborated by the current FxEmbed build — see module caveats).
|
||||
const TWEET_DETAIL_QUERY_ID: &str = "_8aYOgEDz35BrBcBal1-_w";
|
||||
|
||||
fn variables(id: &str) -> Value {
|
||||
json!({
|
||||
"focalTweetId": id,
|
||||
"with_rux_injections": false,
|
||||
"includePromotedContent": false,
|
||||
"withCommunity": true,
|
||||
"withQuickPromoteEligibilityTweetFields": false,
|
||||
"withBirdwatchNotes": false,
|
||||
"withVoice": true,
|
||||
})
|
||||
}
|
||||
|
||||
fn features() -> Value {
|
||||
json!({
|
||||
"rweb_video_screen_enabled": false,
|
||||
"profile_label_improvements_pcf_label_in_post_enabled": true,
|
||||
"rweb_tipjar_consumption_enabled": true,
|
||||
"verified_phone_label_enabled": false,
|
||||
"creator_subscriptions_tweet_preview_api_enabled": true,
|
||||
"responsive_web_graphql_timeline_navigation_enabled": true,
|
||||
"responsive_web_graphql_skip_user_profile_image_extensions_enabled": false,
|
||||
"premium_content_api_read_enabled": false,
|
||||
"communities_web_enable_tweet_community_results_fetch": true,
|
||||
"c9s_tweet_anatomy_moderator_badge_enabled": true,
|
||||
"responsive_web_grok_analyze_button_fetch_trends_enabled": false,
|
||||
"responsive_web_grok_analyze_post_followups_enabled": true,
|
||||
"responsive_web_jetfuel_frame": false,
|
||||
"responsive_web_grok_share_attachment_enabled": true,
|
||||
"articles_preview_enabled": true,
|
||||
"responsive_web_edit_tweet_api_enabled": true,
|
||||
"graphql_is_translatable_rweb_tweet_is_translatable_enabled": true,
|
||||
"view_counts_everywhere_api_enabled": true,
|
||||
"longform_notetweets_consumption_enabled": true,
|
||||
"responsive_web_twitter_article_tweet_consumption_enabled": true,
|
||||
"tweet_awards_web_tipping_enabled": false,
|
||||
"responsive_web_grok_show_grok_translated_post": false,
|
||||
"responsive_web_grok_analysis_button_from_backend": true,
|
||||
"creator_subscriptions_quote_tweet_preview_enabled": false,
|
||||
"freedom_of_speech_not_reach_fetch_enabled": true,
|
||||
"standardized_nudges_misinfo": true,
|
||||
"tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled": true,
|
||||
"longform_notetweets_rich_text_read_enabled": true,
|
||||
"longform_notetweets_inline_media_enabled": true,
|
||||
"responsive_web_grok_image_annotation_enabled": true,
|
||||
"responsive_web_enhance_cards_enabled": false,
|
||||
})
|
||||
}
|
||||
|
||||
/// Whether the authenticated fallback is available.
|
||||
pub fn enabled() -> bool {
|
||||
AUTH_TOKEN.is_some()
|
||||
}
|
||||
|
||||
/// Fetches a tweet as the logged-in user via the private GraphQL API.
|
||||
/// Returns the syndication-shaped [`Tweet`] (media included for NSFW posts).
|
||||
pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
|
||||
let token = AUTH_TOKEN.as_deref().ok_or(FetchError::Sensitive)?;
|
||||
// 16 random bytes as 32 hex chars: X rejects ct0 values of any other
|
||||
// length with 403 code 353 ("matching csrf cookie and header").
|
||||
let ct0: String = (0..16)
|
||||
.map(|_| format!("{:02x}", rand::random::<u8>()))
|
||||
.collect();
|
||||
|
||||
let response = crate::site::CLIENT
|
||||
.get(format!(
|
||||
"https://x.com/i/api/graphql/{TWEET_DETAIL_QUERY_ID}/TweetDetail"
|
||||
))
|
||||
.query(&[
|
||||
("variables", variables(id).to_string()),
|
||||
("features", features().to_string()),
|
||||
])
|
||||
.header("authorization", LOGGED_IN_BEARER)
|
||||
.header("x-csrf-token", &ct0)
|
||||
.header("x-twitter-auth-type", "OAuth2Session")
|
||||
.header("cookie", format!("auth_token={token}; ct0={ct0}"))
|
||||
.header("x-twitter-client-language", "en")
|
||||
.header("x-twitter-active-user", "yes")
|
||||
.header("referer", "https://x.com/")
|
||||
.send()
|
||||
.await?;
|
||||
if !response.status().is_success() {
|
||||
log::warn!("twitter auth fetch {id}: HTTP {}", response.status());
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
let text = response.text().await?;
|
||||
let json: Value = serde_json::from_str(&text)?;
|
||||
let result = parse_tweet_result(&json, id)?;
|
||||
let syndication_shape = to_syndication_shape(&result).ok_or_else(|| {
|
||||
FetchError::Json(serde_json::Error::io(std::io::Error::new(
|
||||
std::io::ErrorKind::InvalidData,
|
||||
"missing tweet fields in GraphQL response",
|
||||
)))
|
||||
})?;
|
||||
Tweet::from_syndication_json(&syndication_shape.to_string()).map_err(FetchError::Json)
|
||||
}
|
||||
|
||||
/// Locates the tweet for `id` in a `TweetDetail` response and unwraps
|
||||
/// visibility wrappers / retweets, mirroring nazurin's `_process_response`.
|
||||
fn parse_tweet_result(json: &Value, id: &str) -> Result<Value, FetchError> {
|
||||
if let Some(errors) = json.get("errors").and_then(|e| e.as_array()) {
|
||||
let messages: Vec<&str> = errors
|
||||
.iter()
|
||||
.filter_map(|e| e.get("message").and_then(|m| m.as_str()))
|
||||
.collect();
|
||||
log::warn!("twitter auth fetch {id} failed: {}", messages.join("; "));
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
|
||||
let instructions = json
|
||||
.pointer("/data/threaded_conversation_with_injections_v2/instructions")
|
||||
.and_then(|v| v.as_array())
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
for instruction in instructions {
|
||||
if instruction.get("type").and_then(|t| t.as_str()) != Some("TimelineAddEntries") {
|
||||
continue;
|
||||
}
|
||||
let entries = instruction
|
||||
.get("entries")
|
||||
.and_then(|e| e.as_array())
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
let wanted = format!("tweet-{id}");
|
||||
for entry in entries {
|
||||
if entry.get("entryId").and_then(|i| i.as_str()) == Some(wanted.as_str()) {
|
||||
let result = entry
|
||||
.pointer("/content/itemContent/tweet_results/result")
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
return normalize_tweet_result(result);
|
||||
}
|
||||
}
|
||||
}
|
||||
Err(FetchError::NotFound)
|
||||
}
|
||||
|
||||
/// Unwraps TweetTombstone/TweetUnavailable errors, the
|
||||
/// TweetWithVisibilityResults wrapper and retweets, returning the
|
||||
/// `{core, legacy, ...}` tweet object.
|
||||
fn normalize_tweet_result(result: &Value) -> Result<Value, FetchError> {
|
||||
match result.get("__typename").and_then(|t| t.as_str()) {
|
||||
Some("TweetTombstone") => {
|
||||
let text = result
|
||||
.pointer("/tombstone/text/text")
|
||||
.and_then(|t| t.as_str())
|
||||
.unwrap_or("tweet is unavailable");
|
||||
log::warn!("twitter auth fetch: tombstone: {text}");
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
Some("TweetUnavailable") => {
|
||||
let reason = result
|
||||
.get("reason")
|
||||
.and_then(|r| r.as_str())
|
||||
.unwrap_or("unknown");
|
||||
log::warn!("twitter auth fetch: tweet unavailable: {reason}");
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
|
||||
// TweetWithVisibilityResults (e.g. limited replies) nests the real tweet.
|
||||
let tweet = result.get("tweet").unwrap_or(result);
|
||||
// A retweet's media lives on the original tweet.
|
||||
if let Some(original) = tweet.pointer("/legacy/retweeted_status_result/result") {
|
||||
return Ok(original.clone());
|
||||
}
|
||||
Ok(tweet.clone())
|
||||
}
|
||||
|
||||
/// Maps a GraphQL `{core, legacy, ...}` tweet onto the syndication JSON
|
||||
/// shape [`Tweet::from_syndication_json`] parses, so the existing text /
|
||||
/// media handling (t.co expansion, `name=orig`, mp4 variant) is reused.
|
||||
fn to_syndication_shape(tweet: &Value) -> Option<Value> {
|
||||
let legacy = tweet.get("legacy")?;
|
||||
let user = tweet.pointer("/core/user_results/result/legacy")?;
|
||||
Some(json!({
|
||||
"id_str": legacy.get("id_str"),
|
||||
"text": legacy.get("full_text"),
|
||||
"user": {
|
||||
"name": user.get("name"),
|
||||
"screen_name": user.get("screen_name"),
|
||||
},
|
||||
"possibly_sensitive": legacy.get("possibly_sensitive"),
|
||||
"entities": legacy.get("entities"),
|
||||
"mediaDetails": legacy.pointer("/extended_entities/media"),
|
||||
}))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn tweet_result() -> Value {
|
||||
json!({
|
||||
"__typename": "Tweet",
|
||||
"core": {
|
||||
"user_results": {
|
||||
"result": {
|
||||
"legacy": { "name": "Display Name", "screen_name": "nsfw_author" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"legacy": {
|
||||
"id_str": "2083868672721039569",
|
||||
"full_text": "nsfw content https://t.co/abc123",
|
||||
"possibly_sensitive": true,
|
||||
"entities": {
|
||||
// The appended media link lives in extended_entities.media,
|
||||
// not entities.urls, so it has no expansion mapping and the
|
||||
// content-based strip removes it.
|
||||
"urls": []
|
||||
},
|
||||
"extended_entities": {
|
||||
"media": [
|
||||
{
|
||||
"type": "photo",
|
||||
"media_url_https": "https://pbs.twimg.com/media/nsfw.jpg",
|
||||
"original_info": { "width": 1200, "height": 800 }
|
||||
},
|
||||
{
|
||||
"type": "video",
|
||||
"media_url_https": "https://pbs.twimg.com/thumb.jpg",
|
||||
"video_info": {
|
||||
"variants": [
|
||||
{ "content_type": "application/x-mpegURL", "url": "https://x.com/pl.m3u8" },
|
||||
{ "content_type": "video/mp4", "url": "https://video.twimg.com/nsfw.mp4" }
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn conversation(tweet: Value) -> Value {
|
||||
json!({
|
||||
"data": {
|
||||
"threaded_conversation_with_injections_v2": {
|
||||
"instructions": [
|
||||
{ "type": "TimelineAddEntries", "entries": [
|
||||
{ "entryId": "tweet-2083868672721039569",
|
||||
"content": { "itemContent": { "tweet_results": { "result": tweet } } } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_graphql_tweet_into_fetched() {
|
||||
let json = conversation(tweet_result());
|
||||
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
|
||||
let shape = to_syndication_shape(&result).unwrap();
|
||||
let tweet = Tweet::from_syndication_json(&shape.to_string()).unwrap();
|
||||
let fetched: crate::site::Fetched = tweet.into();
|
||||
|
||||
assert!(fetched.sensitive);
|
||||
assert_eq!(fetched.media.len(), 2);
|
||||
match &fetched.media[0] {
|
||||
crate::media::Media::Illustration { url, .. } => {
|
||||
assert_eq!(url, "https://pbs.twimg.com/media/nsfw.jpg?name=orig");
|
||||
}
|
||||
other => panic!("expected illustration, got {other:?}"),
|
||||
}
|
||||
match &fetched.media[1] {
|
||||
crate::media::Media::Video { url, .. } => {
|
||||
assert_eq!(url, "https://video.twimg.com/nsfw.mp4");
|
||||
}
|
||||
other => panic!("expected video, got {other:?}"),
|
||||
}
|
||||
assert_eq!(
|
||||
fetched.source_url,
|
||||
"https://x.com/nsfw_author/status/2083868672721039569"
|
||||
);
|
||||
// The appended media short link (no URL-entity mapping) is stripped.
|
||||
assert_eq!(fetched.title, "nsfw content");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unwraps_retweet_to_original() {
|
||||
let original = tweet_result();
|
||||
let mut rt = tweet_result();
|
||||
rt["legacy"]["retweeted_status_result"] = json!({ "result": original });
|
||||
let json = conversation(rt);
|
||||
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
|
||||
assert!(result.pointer("/legacy/retweeted_status_result").is_none());
|
||||
assert_eq!(
|
||||
result.pointer("/legacy/id_str").unwrap(),
|
||||
"2083868672721039569"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn error_response_maps_to_not_found() {
|
||||
let json = json!({ "errors": [{ "message": "NsfwLoggedOut" }] });
|
||||
assert!(matches!(
|
||||
parse_tweet_result(&json, "1"),
|
||||
Err(FetchError::NotFound)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_entry_maps_to_not_found() {
|
||||
let json = conversation(json!({ "__typename": "Tweet" }));
|
||||
assert!(matches!(
|
||||
parse_tweet_result(&json, "999"),
|
||||
Err(FetchError::NotFound)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tombstone_maps_to_not_found() {
|
||||
let tombstone = json!({
|
||||
"__typename": "TweetTombstone",
|
||||
"tombstone": { "text": { "text": "Age-restricted adult content" } }
|
||||
});
|
||||
let json = conversation(tombstone);
|
||||
assert!(matches!(
|
||||
parse_tweet_result(&json, "2083868672721039569"),
|
||||
Err(FetchError::NotFound)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn visibility_wrapper_unwraps() {
|
||||
let inner = tweet_result();
|
||||
let wrapped = json!({ "__typename": "TweetWithVisibilityResults", "tweet": inner });
|
||||
let json = conversation(wrapped);
|
||||
let result = parse_tweet_result(&json, "2083868672721039569").unwrap();
|
||||
assert_eq!(result.get("__typename").unwrap(), "Tweet");
|
||||
}
|
||||
}
|
||||
@@ -19,7 +19,42 @@ pub async fn fetch_from_url(url: &str) -> Result<Fetched, FetchError> {
|
||||
.and_then(|caps| caps.get(1))
|
||||
.map(|m| m.as_str())
|
||||
.ok_or(FetchError::NotFound)?;
|
||||
Ok(fetch(id).await?.into())
|
||||
match fetch(id).await {
|
||||
Ok(tweet) => Ok(tweet.into()),
|
||||
// Syndication withholds NSFW/age-restricted tweets (empty `{}`).
|
||||
// Retry as the logged-in user when TWITTER_AUTH_TOKEN is set;
|
||||
// otherwise degrade to an empty result (the bot replies
|
||||
// "No media found").
|
||||
Err(FetchError::Sensitive) => {
|
||||
if super::auth::enabled() {
|
||||
match super::auth::fetch(id).await {
|
||||
Ok(tweet) => Ok(tweet.into()),
|
||||
Err(e) => {
|
||||
log::warn!("twitter auth fallback failed for {id}: {e}");
|
||||
Ok(empty_fetched(url))
|
||||
}
|
||||
}
|
||||
} else {
|
||||
log::info!("tweet {id} is sensitive; set TWITTER_AUTH_TOKEN to fetch NSFW media");
|
||||
Ok(empty_fetched(url))
|
||||
}
|
||||
}
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
/// A Fetched with no media for withheld tweets: the bot replies
|
||||
/// "No media found" and moves on instead of erroring.
|
||||
fn empty_fetched(url: &str) -> Fetched {
|
||||
Fetched {
|
||||
source_url: url.to_string(),
|
||||
caption: url.to_string(),
|
||||
title: String::new(),
|
||||
media: vec![],
|
||||
sensitive: true,
|
||||
render_data: None,
|
||||
_keep_alive: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Fetches a tweet from the syndication endpoint. Deleted/blocked tweets
|
||||
@@ -44,7 +79,16 @@ pub async fn fetch(id: &str) -> Result<Tweet, FetchError> {
|
||||
{
|
||||
return Err(FetchError::NotFound);
|
||||
}
|
||||
Ok(Tweet::from_syndication_json(&text).map_err(FetchError::Json)?)
|
||||
// NSFW / age-restricted tweets exist but are served as an empty `{}` —
|
||||
// they surface as FetchError::Sensitive so the caller can retry as a
|
||||
// logged-in user.
|
||||
if serde_json::from_str::<serde_json::Value>(&text)
|
||||
.map(|v| v.get("id_str").is_none())
|
||||
.unwrap_or(false)
|
||||
{
|
||||
return Err(FetchError::Sensitive);
|
||||
}
|
||||
Tweet::from_syndication_json(&text).map_err(FetchError::Json)
|
||||
}
|
||||
|
||||
/// The syndication token: JS `((id / 1e15) * PI).toString(36)` (the
|
||||
@@ -112,12 +156,10 @@ impl Tweet {
|
||||
pub fn from_syndication_json(raw_json: &str) -> Result<Self, serde_json::Error> {
|
||||
let json: model::SyndicationTweet = serde_json::from_str(raw_json)?;
|
||||
let id = json.id_str;
|
||||
// Strip the appended media short link first, then expand the remaining
|
||||
// t.co short links (the user's own URLs) to their real destinations.
|
||||
let text = expand_links(
|
||||
&strip_trailing_short_links(&json.text, json.display_text_range),
|
||||
&json.entities.urls,
|
||||
);
|
||||
// Expand the user's t.co short links to their real destinations and
|
||||
// strip the appended media short link, mirroring FxEmbed's linkFixer
|
||||
// (no display_text_range arithmetic — see expand_links).
|
||||
let text = expand_links(&json.text, &json.entities.urls);
|
||||
// `name` is the display name, `screen_name` the handle (Python's
|
||||
// vxtwitter mapping: author = display name, author_id = handle).
|
||||
let author = json.user.name;
|
||||
@@ -158,55 +200,48 @@ impl Tweet {
|
||||
}
|
||||
}
|
||||
|
||||
/// The raw syndication `text` ends with the appended media short link
|
||||
/// (" https://t.co/wmI8McgXul"). `display_text_range` (UTF-16 indices) marks
|
||||
/// the visible text; a regex strips any remaining trailing t.co link when the
|
||||
/// range is absent or a tweet ends in a URL short link.
|
||||
fn strip_trailing_short_links(text: &str, display_text_range: Option<[usize; 2]>) -> String {
|
||||
let mut out = match display_text_range {
|
||||
Some([start, end]) if start < end => {
|
||||
let units: Vec<u16> = text
|
||||
.encode_utf16()
|
||||
.skip(start)
|
||||
.take(end - start)
|
||||
.collect();
|
||||
// Drop the replacement char that a surrogate cut at the boundary
|
||||
// would produce (the range end is a valid UTF-16 boundary in
|
||||
// practice, so this is just a safety net).
|
||||
String::from_utf16_lossy(&units).replace('\u{FFFD}', "")
|
||||
}
|
||||
_ => text.to_string(),
|
||||
};
|
||||
while TRAILING_TCO.is_match(&out) {
|
||||
out = TRAILING_TCO.replace(&out, "").into_owned();
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Trailing Twitter short link, optionally preceded by whitespace.
|
||||
static TRAILING_TCO: LazyLock<Regex> = LazyLock::new(|| {
|
||||
Regex::new(r"\s*https?://t\.co/[A-Za-z0-9]+$").unwrap()
|
||||
});
|
||||
|
||||
/// Replaces every t.co short link that has an entity mapping with its
|
||||
/// expanded URL. Short links without a mapping stay untouched.
|
||||
/// Mirrors FxEmbed's `linkFixer` (link-fixer.ts): expand every t.co short
|
||||
/// link that has an entity mapping to its real destination, drop internal
|
||||
/// `x.com/i/web/status/…` plumbing links, then strip any remaining t.co
|
||||
/// short link (the appended media link and other unmapped short links).
|
||||
/// Pure content matching — no `display_text_range` arithmetic, so the
|
||||
/// endpoint's inconsistent index units (UTF-16 vs code points, see the
|
||||
/// deleted `strip_trailing_short_links`) never matter.
|
||||
fn expand_links(text: &str, urls: &[model::SyndicationEntityUrl]) -> String {
|
||||
let mut out = text.to_string();
|
||||
for entity in urls {
|
||||
if let Some(expanded) = &entity.expanded_url {
|
||||
out = out.replace(&entity.url, expanded);
|
||||
}
|
||||
let Some(expanded) = &entity.expanded_url else {
|
||||
continue;
|
||||
};
|
||||
let replacement = if WEB_STATUS_URL.is_match(expanded) {
|
||||
""
|
||||
} else {
|
||||
expanded
|
||||
};
|
||||
out = out.replace(&entity.url, replacement);
|
||||
}
|
||||
out
|
||||
TCO_LINK.replace_all(&out, "").into_owned()
|
||||
}
|
||||
|
||||
/// Internal x.com page links (reply / quote plumbing) expand to
|
||||
/// `x.com/i/web/status/<id>`; FxEmbed drops them — the tweet's own content
|
||||
/// already carries the information.
|
||||
static WEB_STATUS_URL: LazyLock<Regex> =
|
||||
LazyLock::new(|| Regex::new(r"^https://(?:x\.com|twitter\.com)/i/web/status/\w+").unwrap());
|
||||
|
||||
/// A t.co short link, optionally preceded by a space. Any leftover
|
||||
/// occurrence (unmapped — e.g. the appended media link) is removed,
|
||||
/// mirroring FxEmbed. Real short-link codes are 10 alphanumerics; the
|
||||
/// length-agnostic class keeps fixtures and hypothetical odd lengths safe.
|
||||
static TCO_LINK: LazyLock<Regex> =
|
||||
LazyLock::new(|| Regex::new(r" ?https?://t\.co/[A-Za-z0-9]+").unwrap());
|
||||
|
||||
/// pbs.twimg.com serves a reduced default size without size params; `name=orig`
|
||||
/// returns the original file (fxtwitter used to hand out the original
|
||||
/// directly, the syndication API does not). Non-twimg URLs pass through
|
||||
/// unchanged.
|
||||
fn original_twimg_url(url: &str) -> String {
|
||||
if url.starts_with("https://pbs.twimg.com/")
|
||||
&& (url.ends_with(".jpg") || url.ends_with(".png"))
|
||||
if url.starts_with("https://pbs.twimg.com/") && (url.ends_with(".jpg") || url.ends_with(".png"))
|
||||
{
|
||||
format!("{url}?name=orig")
|
||||
} else {
|
||||
@@ -321,24 +356,23 @@ mod tests {
|
||||
match &fetched.media[0] {
|
||||
Media::Illustration { url, .. } => {
|
||||
// Photo URL is rewritten to request the original file.
|
||||
assert_eq!(
|
||||
url,
|
||||
"https://pbs.twimg.com/media/photo.jpg?name=orig"
|
||||
);
|
||||
assert_eq!(url, "https://pbs.twimg.com/media/photo.jpg?name=orig");
|
||||
}
|
||||
other => panic!("expected illustration, got {other:?}"),
|
||||
}
|
||||
match &fetched.media[1] {
|
||||
Media::Video { url, thumbnail_url, .. } => {
|
||||
Media::Video {
|
||||
url, thumbnail_url, ..
|
||||
} => {
|
||||
assert_eq!(url, "https://video.twimg.com/v.mp4");
|
||||
assert_eq!(thumbnail_url, "https://pbs.twimg.com/thumb.jpg");
|
||||
}
|
||||
other => panic!("expected video, got {other:?}"),
|
||||
}
|
||||
assert!(
|
||||
fetched
|
||||
.caption
|
||||
.contains("<a href=\"https://x.com/author_handle\">Display Name</a>: a & b <c>"),
|
||||
fetched.caption.contains(
|
||||
"<a href=\"https://x.com/author_handle\">Display Name</a>: a & b <c>"
|
||||
),
|
||||
"caption: {}",
|
||||
fetched.caption
|
||||
);
|
||||
@@ -369,13 +403,12 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn syndication_text_strips_trailing_media_short_link() {
|
||||
// Real syndication shape: the media short link sits after the visible
|
||||
// text, and display_text_range marks where it begins.
|
||||
// Real syndication shape: the appended media short link sits after the
|
||||
// visible text; the unmapped t.co link is stripped by content.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": "hello world https://t.co/abc123",
|
||||
"display_text_range": [0, 11],
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"mediaDetails": []
|
||||
});
|
||||
@@ -385,8 +418,31 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_strips_trailing_short_link_without_range() {
|
||||
// No display_text_range: the regex fallback removes the trailing link.
|
||||
fn syndication_text_strips_trailing_link_regardless_of_index_units() {
|
||||
// Real tweet 2084567054481571919: the visible text is 30 code points
|
||||
// but 41 UTF-16 units, and the two endpoints historically reported
|
||||
// display_text_range in different units (UTF-16 on syndication, code
|
||||
// points on GraphQL). The FxEmbed-style content-based strip ignores
|
||||
// the range entirely, so the appended media link is removed for any
|
||||
// response shape.
|
||||
let text = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero https://t.co/XnIi83EkEB";
|
||||
let visible = "妄想𝑨𝒅𝒅𝒊𝒄𝒕𝒊𝒐𝒏…🩷💚❤️\n#ゼンゼロ #zzzero";
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "2084567054481571919",
|
||||
"text": text,
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"mediaDetails": []
|
||||
});
|
||||
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
|
||||
assert_eq!(tweet.text, visible, "left a partial link");
|
||||
assert!(!tweet.caption().contains("t.co"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_strips_trailing_short_link_without_entities() {
|
||||
// No URL entities at all: the leftover t.co link is stripped by the
|
||||
// content regex.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
@@ -407,7 +463,6 @@ mod tests {
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": "Test Tweet with @mentionThis $twtr https://t.co/RzmrQ6wAzD #hashtag https://t.co/9r69akA484",
|
||||
"display_text_range": [0, 67],
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"entities": {
|
||||
"urls": [{
|
||||
@@ -427,25 +482,47 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_keeps_unmapped_short_links() {
|
||||
// No entity mapping for the embedded link: it stays as-is. Only the
|
||||
// trailing media link is stripped.
|
||||
fn syndication_text_strips_unmapped_short_links() {
|
||||
// FxEmbed parity: short links without an entity mapping (appended
|
||||
// media link, embedded unmapped links) are stripped, not kept.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": "check https://t.co/abc123 #tag https://t.co/def456",
|
||||
"display_text_range": [0, 30],
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"mediaDetails": []
|
||||
});
|
||||
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
|
||||
assert_eq!(tweet.text, "check https://t.co/abc123 #tag");
|
||||
assert_eq!(tweet.text, "check #tag");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_utf16_display_range_keeps_multibyte() {
|
||||
// display_text_range is in UTF-16 units; a Japanese text must not be
|
||||
// sliced by UTF-8 bytes.
|
||||
fn syndication_text_drops_internal_web_status_links() {
|
||||
// FxEmbed parity: a mapped link expanding to an internal
|
||||
// x.com/i/web/status/... page (reply/quote plumbing) is removed
|
||||
// instead of being shown.
|
||||
let raw = serde_json::json!({
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": "see https://t.co/xyz1234567 for context",
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"entities": {
|
||||
"urls": [{
|
||||
"url": "https://t.co/xyz1234567",
|
||||
"expanded_url": "https://x.com/i/web/status/9876543210",
|
||||
"display_url": "x.com/i/web/status/9876543210"
|
||||
}]
|
||||
},
|
||||
"mediaDetails": []
|
||||
});
|
||||
let tweet = Tweet::from_syndication_json(&raw.to_string()).unwrap();
|
||||
assert_eq!(tweet.text, "see for context");
|
||||
assert!(!tweet.caption().contains("t.co"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn syndication_text_keeps_multibyte_text() {
|
||||
// Text-only tweet: no short links, the multibyte text is untouched.
|
||||
let text = "コミティア落ちたので、明日は行きません。🙏ごめんなさい";
|
||||
let units: Vec<u16> = text.encode_utf16().collect();
|
||||
assert_eq!(units.len(), 28);
|
||||
@@ -453,7 +530,6 @@ mod tests {
|
||||
"__typename": "Tweet",
|
||||
"id_str": "1",
|
||||
"text": text,
|
||||
"display_text_range": [0, 28],
|
||||
"user": { "name": "N", "screen_name": "h" },
|
||||
"mediaDetails": []
|
||||
});
|
||||
@@ -505,6 +581,9 @@ mod tests {
|
||||
async fn live_fetch_deleted_tweet_is_not_found() {
|
||||
// Deleted tweet: the syndication endpoint answers with errors.
|
||||
let result = fetch("0").await;
|
||||
assert!(matches!(result, Err(FetchError::NotFound)), "got {result:?}");
|
||||
assert!(
|
||||
matches!(result, Err(FetchError::NotFound)),
|
||||
"got {result:?}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
mod auth;
|
||||
mod interface;
|
||||
mod model;
|
||||
|
||||
|
||||
@@ -9,10 +9,6 @@ pub struct SyndicationTweet {
|
||||
pub user: SyndicationUser,
|
||||
#[serde(default)]
|
||||
pub possibly_sensitive: Option<bool>,
|
||||
/// Visible-text span; the raw `text` field has the appended media short
|
||||
/// link after it. Indices are UTF-16 code units.
|
||||
#[serde(default, rename = "display_text_range")]
|
||||
pub display_text_range: Option<[usize; 2]>,
|
||||
#[serde(default)]
|
||||
pub entities: SyndicationEntities,
|
||||
#[serde(default, rename = "mediaDetails")]
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "xmedia-bot"
|
||||
version = "1.0.4"
|
||||
version = "1.0.8"
|
||||
edition = "2024"
|
||||
|
||||
[dependencies]
|
||||
@@ -18,4 +18,8 @@ rusqlite = { version = "0.32", features = ["bundled"] }
|
||||
rand = "0.8"
|
||||
tempfile = "3"
|
||||
parking_lot = "0.12"
|
||||
png = "0.18"
|
||||
zune-jpeg = "0.5"
|
||||
fast_image_resize = "6"
|
||||
jpeg-encoder = "0.7"
|
||||
x-media = { path = "../x-media" }
|
||||
|
||||
@@ -10,6 +10,8 @@ pub struct Config {
|
||||
pub admin_ids: Vec<i64>,
|
||||
/// EDIT_MESSAGE_TTL_SECONDS, default 86400 (24h).
|
||||
pub edit_message_ttl: Duration,
|
||||
/// LINK_CACHE_TTL_SECONDS, default 604800 (7 days).
|
||||
pub link_cache_ttl: Duration,
|
||||
// Webhook settings (moved out of main; names/defaults unchanged).
|
||||
pub webhook_enabled: bool,
|
||||
pub webhook_url: Option<url::Url>,
|
||||
@@ -22,37 +24,42 @@ pub struct Config {
|
||||
impl Config {
|
||||
pub fn load() -> Config {
|
||||
let admin_ids = env::var("BOT_ADMIN")
|
||||
.ok()
|
||||
.map(|s| {
|
||||
s.split(',')
|
||||
.filter_map(|part| part.trim().parse::<i64>().ok())
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
.ok()
|
||||
.map(|s| {
|
||||
s.split(',')
|
||||
.filter_map(|part| part.trim().parse::<i64>().ok())
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
|
||||
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.map(Duration::from_secs)
|
||||
.unwrap_or(Duration::from_secs(86400));
|
||||
let edit_message_ttl = env::var("EDIT_MESSAGE_TTL_SECONDS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.map(Duration::from_secs)
|
||||
.unwrap_or(Duration::from_secs(86400));
|
||||
|
||||
let webhook_enabled = env::var("WEBHOOK")
|
||||
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
|
||||
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
|
||||
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
|
||||
// value that would otherwise come from `.env`).
|
||||
let webhook_cert = env::var("WEBHOOK_CERT")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty());
|
||||
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty());
|
||||
let link_cache_ttl = env::var("LINK_CACHE_TTL_SECONDS")
|
||||
.ok()
|
||||
.and_then(|s| s.parse::<u64>().ok())
|
||||
.map(Duration::from_secs)
|
||||
.unwrap_or(Duration::from_secs(7 * 24 * 3600));
|
||||
|
||||
let webhook_enabled = env::var("WEBHOOK")
|
||||
.is_ok_and(|v| matches!(v.to_lowercase().as_str(), "true" | "yes" | "1"));
|
||||
let webhook_url = env::var("WEBHOOK_URL").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_listen = env::var("WEBHOOK_LISTEN").ok().and_then(|s| s.parse().ok());
|
||||
let webhook_port = env::var("WEBHOOK_PORT").ok().and_then(|s| s.parse().ok());
|
||||
// Empty strings count as unset (e.g. `-e WEBHOOK_CERT=` to disable a
|
||||
// value that would otherwise come from `.env`).
|
||||
let webhook_cert = env::var("WEBHOOK_CERT").ok().filter(|s| !s.is_empty());
|
||||
let webhook_secret_token = env::var("WEBHOOK_SECRET_TOKEN")
|
||||
.ok()
|
||||
.filter(|s| !s.is_empty());
|
||||
|
||||
Config {
|
||||
admin_ids,
|
||||
edit_message_ttl,
|
||||
link_cache_ttl,
|
||||
webhook_enabled,
|
||||
webhook_url,
|
||||
webhook_listen,
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
//! Shared SQLite plumbing for the three tables in `data/task_queue.db`
|
||||
//! (`tasks` in queue.rs, `chat_state` in state.rs, `link_cache` in
|
||||
//! link_cache.rs).
|
||||
//!
|
||||
//! Every operation opens its own short-lived connection with a busy timeout:
|
||||
//! handler tasks enqueue while workers lease/update rows concurrently, and
|
||||
//! without the timeout a concurrent write fails immediately with SQLITE_BUSY
|
||||
//! and the operation is lost. All I/O runs inside `spawn_blocking` via
|
||||
//! [`with_conn`] — rusqlite connections are not Send-friendly to hold across
|
||||
//! an await point, and blocking the async executor stalls every handler.
|
||||
|
||||
use rusqlite::Connection;
|
||||
use std::time::Duration;
|
||||
|
||||
/// Opens the shared DB with a busy timeout.
|
||||
pub fn open_db(path: &str) -> rusqlite::Result<Connection> {
|
||||
let conn = Connection::open(path)?;
|
||||
conn.busy_timeout(Duration::from_secs(5))?;
|
||||
Ok(conn)
|
||||
}
|
||||
|
||||
/// Runs `f` against a fresh connection on a blocking thread, returning the
|
||||
/// closure's result. Owns the `spawn_blocking` + `expect` ceremony shared by
|
||||
/// every table access; the caller maps errors to its own log line.
|
||||
pub async fn with_conn<T, F>(path: &str, f: F) -> rusqlite::Result<T>
|
||||
where
|
||||
T: Send + 'static,
|
||||
F: FnOnce(&mut Connection) -> rusqlite::Result<T> + Send + 'static,
|
||||
{
|
||||
let path = path.to_string();
|
||||
tokio::task::spawn_blocking(move || {
|
||||
let mut conn = open_db(&path)?;
|
||||
f(&mut conn)
|
||||
})
|
||||
.await
|
||||
.expect("db worker panicked")
|
||||
}
|
||||
@@ -1,9 +1,11 @@
|
||||
use crate::config::Config;
|
||||
use crate::link_cache::{CachedMediaKind, CachedPost, LinkCache};
|
||||
use crate::queue::PersistentTaskQueue;
|
||||
use crate::send::{self, MediaItemPayload, Task};
|
||||
use crate::state::{ChatStore, unix_now};
|
||||
use crate::state::{ChatData, ChatStore, unix_now};
|
||||
use std::collections::HashSet;
|
||||
use std::sync::LazyLock;
|
||||
use teloxide::RequestError;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{
|
||||
CallbackQuery, ChatAction, ChatId, ChatKind, InlineQuery, InlineQueryResult,
|
||||
@@ -11,35 +13,58 @@ use teloxide::types::{
|
||||
MessageEntityKind, MessageId, ParseMode, Recipient, ReplyParameters,
|
||||
};
|
||||
use teloxide::utils::command::BotCommands;
|
||||
use teloxide::RequestError;
|
||||
use tokio::sync::Semaphore;
|
||||
use x_media::media::Media;
|
||||
|
||||
pub static CHAT_STORE: LazyLock<ChatStore> = LazyLock::new(|| {
|
||||
ChatStore::open("data/task_queue.db").expect("failed to open chat store")
|
||||
});
|
||||
pub static CHAT_STORE: LazyLock<ChatStore> =
|
||||
LazyLock::new(|| ChatStore::open("data/task_queue.db").expect("failed to open chat store"));
|
||||
pub static TASK_QUEUE: LazyLock<PersistentTaskQueue> =
|
||||
LazyLock::new(|| PersistentTaskQueue::new("data/task_queue.db"));
|
||||
pub static LINK_CACHE: LazyLock<LinkCache> =
|
||||
LazyLock::new(|| LinkCache::open("data/task_queue.db"));
|
||||
pub static CONFIG: LazyLock<Config> = LazyLock::new(Config::load);
|
||||
|
||||
/// Cap on concurrent per-URL processing. teloxide dispatches updates to a
|
||||
/// per-chat worker that handles them sequentially, so a batch-forward of many
|
||||
/// messages would otherwise be processed one at a time (fetch + send each,
|
||||
/// roughly a second per message). Moving the work into spawned tasks trades
|
||||
/// per-chat reply ordering for throughput; the semaphore bounds how many run
|
||||
/// at once so a big burst cannot hammer Telegram's rate limits.
|
||||
static URL_TASKS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(8));
|
||||
|
||||
#[derive(BotCommands, Clone)]
|
||||
#[command(rename_rule = "snake_case", description = "")]
|
||||
#[command(
|
||||
rename_rule = "snake_case",
|
||||
description = "Turn X/Pixiv/Bluesky links into media messages"
|
||||
)]
|
||||
enum Command {
|
||||
#[command(description = "")]
|
||||
#[command(description = "Get started")]
|
||||
Start,
|
||||
#[command(description = "")]
|
||||
#[command(description = "Show command help")]
|
||||
Help,
|
||||
#[command(description = "", parse_with = "split")]
|
||||
#[command(
|
||||
description = "Set forward channel (@channel or ID)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetForwardChannel(String),
|
||||
#[command(description = "")]
|
||||
#[command(description = "Remove forward channel")]
|
||||
RemoveForwardChannel,
|
||||
#[command(description = "")]
|
||||
#[command(description = "Toggle edit-before-forward")]
|
||||
EditBeforeForward,
|
||||
#[command(description = "", parse_with = "split")]
|
||||
#[command(
|
||||
description = "Reply with [] to save as template",
|
||||
parse_with = "split"
|
||||
)]
|
||||
SetTemplate(String),
|
||||
#[command(description = "")]
|
||||
#[command(description = "Show chat state (debug)")]
|
||||
BotDict,
|
||||
#[command(description = "", parse_with = "split")]
|
||||
#[command(description = "Set site caption format", parse_with = "split")]
|
||||
SetFormat(String),
|
||||
#[command(
|
||||
description = "Clear link cache (admin; optional URL, else all)",
|
||||
parse_with = "split"
|
||||
)]
|
||||
ClearCache(String),
|
||||
}
|
||||
|
||||
async fn reply<T>(bot: Bot, message: Message, text: T) -> Result<Message, RequestError>
|
||||
@@ -185,7 +210,11 @@ async fn set_forward_channel_handler(
|
||||
Ok(channel_id)
|
||||
}
|
||||
|
||||
async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Result<(), RequestError> {
|
||||
async fn execute_command(
|
||||
bot: &Bot,
|
||||
message: &Message,
|
||||
command: Command,
|
||||
) -> Result<(), RequestError> {
|
||||
match command {
|
||||
Command::Start => {
|
||||
bot.send_message(message.chat.id, "Hello!").await?;
|
||||
@@ -203,7 +232,8 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
|
||||
"Add successfully.".to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::EmptyParameter) => {
|
||||
"Receive empty parameter.\nYou should enter a channel id or username".to_string()
|
||||
"Receive empty parameter.\nYou should enter a channel id or username"
|
||||
.to_string()
|
||||
}
|
||||
Err(SetForwardChannelError::NotChannel) => {
|
||||
"Given id / username is not a channel".to_string()
|
||||
@@ -280,7 +310,9 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
|
||||
Command::SetFormat(arg) => {
|
||||
let chat_id = message.chat.id.0;
|
||||
let (site, format) = match arg.split_once(char::is_whitespace) {
|
||||
Some((site, format)) if !format.trim().is_empty() => (site.trim(), format.trim().to_string()),
|
||||
Some((site, format)) if !format.trim().is_empty() => {
|
||||
(site.trim(), format.trim().to_string())
|
||||
}
|
||||
_ => {
|
||||
reply(
|
||||
bot.clone(),
|
||||
@@ -305,10 +337,62 @@ async fn execute_command(bot: &Bot, message: &Message, command: Command) -> Resu
|
||||
CHAT_STORE.set(chat_id, &chat_data).await;
|
||||
reply(bot.clone(), message.clone(), "Format set.").await?;
|
||||
}
|
||||
Command::ClearCache(arg) => {
|
||||
let sender_id = message
|
||||
.from
|
||||
.as_ref()
|
||||
.map(|user| user.id.0 as i64)
|
||||
.unwrap_or(-1);
|
||||
if !CONFIG.admin_ids.contains(&sender_id) {
|
||||
reply(bot.clone(), message.clone(), "Admin only.").await?;
|
||||
return Ok(());
|
||||
}
|
||||
let arg = arg.trim();
|
||||
if arg.is_empty() {
|
||||
let removed = LINK_CACHE.clear(None).await;
|
||||
log::info!("cache cleared by {sender_id}: {removed} entries");
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
format!("Cleared {removed} cached entr{}.", plural(removed)),
|
||||
)
|
||||
.await?;
|
||||
} else {
|
||||
let key = match x_media::site::cache_key(arg) {
|
||||
Some(key) => key,
|
||||
None => {
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
"Unrecognized link. Use a twitter/x, pixiv or bsky post URL.",
|
||||
)
|
||||
.await?;
|
||||
return Ok(());
|
||||
}
|
||||
};
|
||||
let removed = LINK_CACHE.clear(Some(&key)).await;
|
||||
log::info!("cache entry cleared by {sender_id}: {key} ({removed} rows)");
|
||||
reply(
|
||||
bot.clone(),
|
||||
message.clone(),
|
||||
format!(
|
||||
"Cleared cache for {arg} ({} entr{}).",
|
||||
removed,
|
||||
plural(removed)
|
||||
),
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// `""` for one, `"ies"` for anything else — "1 entry" / "2 entries".
|
||||
fn plural(n: usize) -> &'static str {
|
||||
if n == 1 { "" } else { "ies" }
|
||||
}
|
||||
|
||||
/// For locally produced media (encoded ugoira MP4) the thumbnail URL is a
|
||||
/// hotlink-protected remote URL Telegram may not fetch; let Telegram generate
|
||||
/// its own thumbnail instead.
|
||||
@@ -330,18 +414,21 @@ fn media_to_payload(media: &Media, sensitive: bool) -> MediaItemPayload {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Video { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
Media::Animated { .. } => MediaItemPayload::Video {
|
||||
media: media.url().to_string(),
|
||||
has_spoiler: sensitive,
|
||||
thumbnail: thumbnail_for(media),
|
||||
fallback_url,
|
||||
file_id: false,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -354,11 +441,154 @@ async fn enqueue_retry(task: Task, delay_seconds: f64) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Sends a task and handles the outcome: post-send actions on success, retry
|
||||
/// enqueue on retryable failure, reply + link-cache invalidation on
|
||||
/// permanent failure (a stale cached file id must not repeat forever).
|
||||
async fn dispatch_send(bot: Bot, message: &Message, task: &Task, url: &str) {
|
||||
let result = match task {
|
||||
Task::SendAnimation { .. } => send::send_animation(&bot, task).await,
|
||||
Task::SendMediaSequence { .. } => send::send_media_sequence(&bot, task).await,
|
||||
Task::ForwardMessages { .. } => unreachable!(),
|
||||
};
|
||||
match result {
|
||||
Ok(message_ids) => {
|
||||
log::info!("sent {} message(s) for {url}", message_ids.len());
|
||||
send::post_send_actions(&bot, task, message_ids).await;
|
||||
}
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(task, delay_seconds).await;
|
||||
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
|
||||
}
|
||||
Err(send::SendError::Permanent {
|
||||
message: err_message,
|
||||
task,
|
||||
}) => {
|
||||
send::invalidate_cache(&task).await;
|
||||
log::error!("send for {url} failed permanently: {err_message}");
|
||||
let _ = reply(bot, message.clone(), format!("Send failed: {err_message}")).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Builds the send task from ready-made items, sharing the payload shape
|
||||
/// between the fresh-fetch and link-cache paths.
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn build_send_task(
|
||||
chat_data: &ChatData,
|
||||
message: &Message,
|
||||
source_url: String,
|
||||
caption: String,
|
||||
items: Vec<MediaItemPayload>,
|
||||
cache_data: Option<CachedPost>,
|
||||
) -> Task {
|
||||
let chat_id = message.chat.id.0;
|
||||
if items.len() == 1 && matches!(items[0], MediaItemPayload::Animation { .. }) {
|
||||
Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption,
|
||||
animation: items.into_iter().next().unwrap(),
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
cache_data,
|
||||
}
|
||||
} else {
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption,
|
||||
media_batches: send::chunk_media_items(items),
|
||||
batch_index: 0,
|
||||
sent_message_ids: vec![],
|
||||
source_url,
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
cache_data,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn url_media(bot: Bot, message: &Message, url: &str) {
|
||||
let chat_id = message.chat.id.0;
|
||||
if let Err(e) = bot.send_chat_action(ChatId(chat_id), ChatAction::Typing).await {
|
||||
if let Err(e) = bot
|
||||
.send_chat_action(ChatId(chat_id), ChatAction::Typing)
|
||||
.await
|
||||
{
|
||||
log::error!("send_chat_action failed: {e}");
|
||||
}
|
||||
|
||||
// Link cache: a post sent before is re-sent from Telegram file ids —
|
||||
// no source-site request, no download, no upload. Keyed by the
|
||||
// normalized post id so x.com / fxtwitter / /photo/N variants collide.
|
||||
if let Some(key) = x_media::site::cache_key(url)
|
||||
&& let Some(cached) = LINK_CACHE.get(&key, CONFIG.link_cache_ttl).await
|
||||
{
|
||||
log::info!("link cache hit for {url}");
|
||||
let chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let site = key.split(':').next().unwrap_or("unknown");
|
||||
let format = chat_data
|
||||
.message_format
|
||||
.get(site)
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = if format.is_empty() {
|
||||
cached.caption.clone()
|
||||
} else {
|
||||
x_media::site::caption_from_fields(
|
||||
&format,
|
||||
"",
|
||||
&cached.url,
|
||||
&cached.author,
|
||||
&cached.author_url,
|
||||
&cached.title,
|
||||
&cached.tags,
|
||||
)
|
||||
};
|
||||
let items: Vec<MediaItemPayload> = cached
|
||||
.media
|
||||
.iter()
|
||||
.map(|m| match m.kind {
|
||||
CachedMediaKind::Photo => MediaItemPayload::Photo {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Video => MediaItemPayload::Video {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
thumbnail: None,
|
||||
fallback_url: None,
|
||||
file_id: true,
|
||||
},
|
||||
CachedMediaKind::Animation => MediaItemPayload::Animation {
|
||||
media: m.file_id.clone(),
|
||||
has_spoiler: cached.sensitive,
|
||||
file_id: true,
|
||||
},
|
||||
})
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
message,
|
||||
cached.url.clone(),
|
||||
caption,
|
||||
items,
|
||||
Some(cached),
|
||||
);
|
||||
dispatch_send(bot, message, &task, url).await;
|
||||
return;
|
||||
}
|
||||
|
||||
log::info!("fetching {url}");
|
||||
match x_media::site::fetch(url).await {
|
||||
// Unsupported links are ignored silently (Python parity).
|
||||
@@ -368,7 +598,12 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
|
||||
// Retries exhausted: notify the user (Rust-only requirement 3).
|
||||
Err(e) => {
|
||||
log::error!("fetch {url}: {e}");
|
||||
let _ = reply(bot, message.clone(), "Failed to fetch media from this link.").await;
|
||||
let _ = reply(
|
||||
bot,
|
||||
message.clone(),
|
||||
"Failed to fetch media from this link.",
|
||||
)
|
||||
.await;
|
||||
}
|
||||
Ok(Some(fetched)) => {
|
||||
if fetched.media.is_empty() {
|
||||
@@ -388,66 +623,34 @@ async fn url_media(bot: Bot, message: &Message, url: &str) {
|
||||
.cloned()
|
||||
.unwrap_or_default();
|
||||
let caption = fetched.caption_with(&format);
|
||||
let task = if fetched.media.len() == 1
|
||||
&& matches!(fetched.media[0], Media::Animated { .. })
|
||||
{
|
||||
Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption: caption.clone(),
|
||||
animation: MediaItemPayload::Animation {
|
||||
media: fetched.media[0].url().to_string(),
|
||||
has_spoiler: fetched.sensitive,
|
||||
},
|
||||
source_url: fetched.source_url.clone(),
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
}
|
||||
} else {
|
||||
let items: Vec<MediaItemPayload> = fetched
|
||||
.media
|
||||
.iter()
|
||||
.map(|media| media_to_payload(media, fetched.sensitive))
|
||||
.collect();
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id: message.id.0 as i64,
|
||||
caption: caption.clone(),
|
||||
media_batches: send::chunk_media_items(items),
|
||||
batch_index: 0,
|
||||
sent_message_ids: vec![],
|
||||
source_url: fetched.source_url.clone(),
|
||||
edit_before_forward: chat_data.edit_before_forward,
|
||||
forward_channel_id: chat_data.forward_channel_id,
|
||||
notify_chat_id: Some(chat_id),
|
||||
notify_message_id: Some(message.id.0 as i64),
|
||||
}
|
||||
};
|
||||
let result = match &task {
|
||||
Task::SendAnimation { .. } => send::send_animation(&bot, &task).await,
|
||||
Task::SendMediaSequence { .. } => send::send_media_sequence(&bot, &task).await,
|
||||
Task::ForwardMessages { .. } => unreachable!(),
|
||||
};
|
||||
match result {
|
||||
Ok(message_ids) => {
|
||||
log::info!("sent {} message(s) for {url}", message_ids.len());
|
||||
send::post_send_actions(&bot, &task, message_ids).await;
|
||||
}
|
||||
Err(send::SendError::Retryable { delay_seconds, task }) => {
|
||||
log::info!("send for {url} failed, queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(task, delay_seconds).await;
|
||||
let _ = reply(bot, message.clone(), "Send failed. Task queued for retry.").await;
|
||||
}
|
||||
Err(send::SendError::Permanent {
|
||||
message: err_message,
|
||||
..
|
||||
}) => {
|
||||
log::error!("send for {url} failed permanently: {err_message}");
|
||||
let _ = reply(bot, message.clone(), format!("Send failed: {err_message}")).await;
|
||||
}
|
||||
}
|
||||
// Raw render data for the link cache; the send fills in the
|
||||
// Telegram file ids and persists the entry.
|
||||
let cache_data = fetched
|
||||
.render_fields()
|
||||
.map(|(author, author_url, title, tags)| CachedPost {
|
||||
url: fetched.source_url.clone(),
|
||||
caption: fetched.caption.clone(),
|
||||
title: title.to_string(),
|
||||
author: author.to_string(),
|
||||
author_url: author_url.to_string(),
|
||||
tags: tags.to_string(),
|
||||
sensitive: fetched.sensitive,
|
||||
media: vec![],
|
||||
});
|
||||
let items: Vec<MediaItemPayload> = fetched
|
||||
.media
|
||||
.iter()
|
||||
.map(|media| media_to_payload(media, fetched.sensitive))
|
||||
.collect();
|
||||
let task = build_send_task(
|
||||
&chat_data,
|
||||
message,
|
||||
fetched.source_url.clone(),
|
||||
caption,
|
||||
items,
|
||||
cache_data,
|
||||
);
|
||||
dispatch_send(bot, message, &task, url).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -463,7 +666,10 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
|
||||
.text()
|
||||
.map(|t| if t.len() > 120 { &t[..120] } else { t })
|
||||
.unwrap_or("<no text>");
|
||||
log::info!("message from {sender} in {} (private={is_private}): {text_preview}", message.chat.id);
|
||||
log::info!(
|
||||
"message from {sender} in {} (private={is_private}): {text_preview}",
|
||||
message.chat.id
|
||||
);
|
||||
// URL/edit flows only run in private chats; commands run in any chat.
|
||||
if is_private && edit_message_handler(&bot, &message).await {
|
||||
return respond(());
|
||||
@@ -481,7 +687,13 @@ pub async fn message_handler(bot: Bot, message: Message) -> Result<(), RequestEr
|
||||
log::info!("extracted {} URL(s): {urls:?}", urls.len());
|
||||
}
|
||||
for url in urls {
|
||||
url_media(bot.clone(), &message, &url).await;
|
||||
let bot = bot.clone();
|
||||
let message = message.clone();
|
||||
tokio::spawn(async move {
|
||||
// Held for the whole task; the semaphore is never closed.
|
||||
let _permit = URL_TASKS.acquire().await.expect("URL semaphore closed");
|
||||
url_media(bot, &message, &url).await;
|
||||
});
|
||||
}
|
||||
}
|
||||
respond(())
|
||||
@@ -527,8 +739,8 @@ pub async fn inline_query_handler(bot: Bot, query: InlineQuery) -> Result<(), Re
|
||||
thumbnail,
|
||||
fetched.title.clone(),
|
||||
)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html),
|
||||
),
|
||||
Media::Animated { .. } => InlineQueryResult::Mpeg4Gif(
|
||||
InlineQueryResultMpeg4Gif::new(id, url, thumbnail)
|
||||
@@ -560,7 +772,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
|
||||
let mut chat_data = CHAT_STORE.get(chat_id).await;
|
||||
let edit = chat_data.edit_message.get(&prompt_message_id).cloned();
|
||||
let Some(edit) = edit else {
|
||||
log::info!("callback from {}: no edit record for prompt {prompt_message_id}", chat_id);
|
||||
log::info!(
|
||||
"callback from {}: no edit record for prompt {prompt_message_id}",
|
||||
chat_id
|
||||
);
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
.text("Expired")
|
||||
.await?;
|
||||
@@ -579,7 +794,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
|
||||
let Some(data) = data else {
|
||||
return respond(());
|
||||
};
|
||||
log::info!("callback from {} on prompt {prompt_message_id}: {data}", chat_id);
|
||||
log::info!(
|
||||
"callback from {} on prompt {prompt_message_id}: {data}",
|
||||
chat_id
|
||||
);
|
||||
if data == "forward" {
|
||||
match chat_data.forward_channel_id {
|
||||
Some(channel_id) => {
|
||||
@@ -605,7 +823,10 @@ pub async fn callback_query_handler(bot: Bot, query: CallbackQuery) -> Result<()
|
||||
chat_data.edit_message.remove(&prompt_message_id);
|
||||
CHAT_STORE.set(chat_id, &chat_data).await;
|
||||
}
|
||||
Err(send::SendError::Retryable { delay_seconds, task }) => {
|
||||
Err(send::SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
log::info!("forward queued for retry in {delay_seconds:.1}s");
|
||||
enqueue_retry(task, delay_seconds).await;
|
||||
bot.answer_callback_query(callback_query_id)
|
||||
|
||||
@@ -0,0 +1,293 @@
|
||||
//! Persistent cache of successfully sent posts.
|
||||
//!
|
||||
//! After a media send succeeds, the raw render data plus the Telegram
|
||||
//! `file_id`s of the sent items are stored keyed by [`crate::site` cache
|
||||
//! key]. A repeated link is then answered entirely from local state — no
|
||||
//! re-fetch of the source site, no re-upload — and no media file is stored
|
||||
//! on disk (the file ids point at Telegram's servers). Entries expire after
|
||||
//! [`Config::link_cache_ttl`]; a stale entry is dropped lazily on read and
|
||||
//! by the periodic prune in `main`.
|
||||
|
||||
use rusqlite::{Connection, params};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::Duration;
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone, Debug, PartialEq)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum CachedMediaKind {
|
||||
Photo,
|
||||
Video,
|
||||
Animation,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct CachedMedia {
|
||||
pub kind: CachedMediaKind,
|
||||
pub file_id: String,
|
||||
}
|
||||
|
||||
/// Everything needed to re-send a post without touching the source site:
|
||||
/// the canonical URL, pre-escaped caption fields, and the file ids produced
|
||||
/// by the original successful send.
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
pub struct CachedPost {
|
||||
pub url: String,
|
||||
/// The site's built-in caption (used when the chat has no format
|
||||
/// override).
|
||||
pub caption: String,
|
||||
pub title: String,
|
||||
pub author: String,
|
||||
pub author_url: String,
|
||||
pub tags: String,
|
||||
pub sensitive: bool,
|
||||
pub media: Vec<CachedMedia>,
|
||||
}
|
||||
|
||||
/// SQLite-backed cache sharing `data/task_queue.db` with the queue and chat
|
||||
/// state (same `open_db` pattern: busy timeout, `spawn_blocking` I/O).
|
||||
pub struct LinkCache {
|
||||
db_path: String,
|
||||
}
|
||||
|
||||
impl LinkCache {
|
||||
pub fn open(db_path: &str) -> Self {
|
||||
if let Ok(conn) = Connection::open(db_path)
|
||||
&& let Err(e) = conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS link_cache (url TEXT PRIMARY KEY, \
|
||||
payload TEXT NOT NULL, created_at REAL NOT NULL);",
|
||||
)
|
||||
{
|
||||
log::error!("failed to initialize link cache schema: {e}");
|
||||
}
|
||||
Self {
|
||||
db_path: db_path.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the cached post if present and not expired; a stale entry is
|
||||
/// removed on the spot.
|
||||
pub async fn get(&self, key: &str, ttl: Duration) -> Option<CachedPost> {
|
||||
let key = key.to_string();
|
||||
let ttl = ttl.as_secs_f64();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT payload, created_at FROM link_cache WHERE url = ?1")?;
|
||||
let mut rows = stmt.query(params![key])?;
|
||||
let Some(row) = rows.next()? else {
|
||||
return Ok(None);
|
||||
};
|
||||
let payload: String = row.get(0)?;
|
||||
let created_at: f64 = row.get(1)?;
|
||||
if now_f64() - created_at > ttl {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
return Ok(None);
|
||||
}
|
||||
Ok(Some(serde_json::from_str::<CachedPost>(&payload).map_err(
|
||||
|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)),
|
||||
)?))
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
log::error!("link cache read failed: {e}");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn put(&self, key: &str, post: &CachedPost) {
|
||||
let key = key.to_string();
|
||||
let payload = serde_json::to_string(post).expect("cached post serializes");
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO link_cache (url, payload, created_at) VALUES (?1, ?2, ?3)",
|
||||
params![key, payload, now_f64()],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("link cache write failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Drops an entry (e.g. a cached file id that turned out invalid).
|
||||
pub async fn remove(&self, key: &str) {
|
||||
let key = key.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key])?;
|
||||
Ok(())
|
||||
})
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("link cache delete failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes expired entries; returns how many were deleted.
|
||||
pub async fn prune(&self, ttl: Duration) -> usize {
|
||||
let cutoff = now_f64() - ttl.as_secs_f64();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"DELETE FROM link_cache WHERE created_at < ?1",
|
||||
params![cutoff],
|
||||
)
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
log::error!("link cache prune failed: {e}");
|
||||
0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Deletes one entry (by normalized cache key) or the whole cache when
|
||||
/// `key` is `None`. Returns how many rows were removed.
|
||||
pub async fn clear(&self, key: Option<&str>) -> usize {
|
||||
let key = key.map(str::to_string);
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| match &key {
|
||||
Some(key) => conn.execute("DELETE FROM link_cache WHERE url = ?1", params![key]),
|
||||
None => conn.execute("DELETE FROM link_cache", []),
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(n) => n,
|
||||
Err(e) => {
|
||||
log::error!("link cache clear failed: {e}");
|
||||
0
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn now_f64() -> f64 {
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.map(|d| d.as_secs_f64())
|
||||
.unwrap_or(0.0)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn entry() -> CachedPost {
|
||||
CachedPost {
|
||||
url: "https://x.com/u/status/1".into(),
|
||||
caption: "cap".into(),
|
||||
title: "t".into(),
|
||||
author: "a".into(),
|
||||
author_url: "au".into(),
|
||||
tags: "".into(),
|
||||
sensitive: true,
|
||||
media: vec![CachedMedia {
|
||||
kind: CachedMediaKind::Photo,
|
||||
file_id: "AgAC...".into(),
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn put_get_roundtrip() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
let got = cache.get("twitter:1", Duration::from_secs(3600)).await;
|
||||
assert!(got.is_some());
|
||||
let got = got.unwrap();
|
||||
assert_eq!(got.url, "https://x.com/u/status/1");
|
||||
assert_eq!(got.media[0].file_id, "AgAC...");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn expired_entry_removed_on_read() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
// Force the row into the past so a 1s TTL expires it.
|
||||
{
|
||||
let conn = Connection::open(dir.path().join("c.db")).unwrap();
|
||||
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
|
||||
.unwrap();
|
||||
}
|
||||
assert!(
|
||||
cache
|
||||
.get("twitter:1", Duration::from_secs(1))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
cache
|
||||
.get("twitter:1", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn remove_and_prune() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
cache.put("pixiv:2", &entry()).await;
|
||||
cache.remove("twitter:1").await;
|
||||
assert!(
|
||||
cache
|
||||
.get("twitter:1", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
cache
|
||||
.get("pixiv:2", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_some()
|
||||
);
|
||||
{
|
||||
let conn = Connection::open(dir.path().join("c.db")).unwrap();
|
||||
conn.execute("UPDATE link_cache SET created_at = created_at - 100", [])
|
||||
.unwrap();
|
||||
}
|
||||
assert_eq!(cache.prune(Duration::from_secs(1)).await, 1);
|
||||
assert!(
|
||||
cache
|
||||
.get("pixiv:2", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn clear_one_entry_or_all() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let cache = LinkCache::open(dir.path().join("c.db").to_str().unwrap());
|
||||
cache.put("twitter:1", &entry()).await;
|
||||
cache.put("pixiv:2", &entry()).await;
|
||||
// By key: only the matching row is removed.
|
||||
assert_eq!(cache.clear(Some("twitter:1")).await, 1);
|
||||
assert!(
|
||||
cache
|
||||
.get("twitter:1", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
cache
|
||||
.get("pixiv:2", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_some()
|
||||
);
|
||||
// Whole cache: nothing left; removing an absent key deletes 0 rows.
|
||||
assert_eq!(cache.clear(None).await, 1);
|
||||
assert!(
|
||||
cache
|
||||
.get("pixiv:2", Duration::from_secs(3600))
|
||||
.await
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(cache.clear(None).await, 0);
|
||||
}
|
||||
}
|
||||
@@ -1,19 +1,22 @@
|
||||
use dotenv::dotenv;
|
||||
use teloxide::dptree::endpoint;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::stop::StopToken;
|
||||
use teloxide::types::{ChatId, InputFile, MessageId};
|
||||
use teloxide::update_listeners::{self, webhooks, UpdateListener};
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::update_listeners::{self, UpdateListener, webhooks};
|
||||
use tokio::sync::watch;
|
||||
use x_media::site;
|
||||
|
||||
mod config;
|
||||
mod db;
|
||||
mod handlers;
|
||||
mod link_cache;
|
||||
mod photo;
|
||||
mod queue;
|
||||
mod send;
|
||||
mod state;
|
||||
|
||||
use handlers::{CHAT_STORE, CONFIG, TASK_QUEUE};
|
||||
use handlers::{CHAT_STORE, CONFIG, LINK_CACHE, TASK_QUEUE};
|
||||
|
||||
/// Docker `stop` / `compose down` delivers SIGTERM, which teloxide's ctrlc
|
||||
/// handler (SIGINT only) never sees — without this the process would die
|
||||
@@ -72,7 +75,10 @@ async fn main() {
|
||||
}
|
||||
|
||||
// Edit-expiry sweep: clears the prompt's buttons once the record expires.
|
||||
log::info!("edit-expiry sweep: every 300s, ttl {}", CONFIG.edit_message_ttl.as_secs());
|
||||
log::info!(
|
||||
"edit-expiry sweep: every 300s, ttl {}",
|
||||
CONFIG.edit_message_ttl.as_secs()
|
||||
);
|
||||
let (stop_tx, stop_rx) = watch::channel(false);
|
||||
{
|
||||
let bot = bot.clone();
|
||||
@@ -85,11 +91,18 @@ async fn main() {
|
||||
}
|
||||
let ttl = CONFIG.edit_message_ttl;
|
||||
let removed = CHAT_STORE.prune_expired(ttl).await;
|
||||
let pruned = LINK_CACHE.prune(CONFIG.link_cache_ttl).await;
|
||||
if pruned > 0 {
|
||||
log::info!("link cache: pruned {pruned} expired entr(ies)");
|
||||
}
|
||||
for (chat_id, prompt_message_id) in removed {
|
||||
// If the prompt was already deleted, this fails with a
|
||||
// 400 "message to edit not found" — log and ignore.
|
||||
if let Err(e) = bot
|
||||
.edit_message_reply_markup(ChatId(chat_id), MessageId(prompt_message_id as i32))
|
||||
.edit_message_reply_markup(
|
||||
ChatId(chat_id),
|
||||
MessageId(prompt_message_id as i32),
|
||||
)
|
||||
.await
|
||||
{
|
||||
log::info!("edit-expiry sweep: prompt message gone: {e}");
|
||||
@@ -111,10 +124,7 @@ async fn main() {
|
||||
|
||||
if CONFIG.webhook_enabled {
|
||||
log::info!("running in webhook mode");
|
||||
let url = CONFIG
|
||||
.webhook_url
|
||||
.clone()
|
||||
.expect("WEBHOOK_URL is not set");
|
||||
let url = CONFIG.webhook_url.clone().expect("WEBHOOK_URL is not set");
|
||||
// `webhooks::axum` calls set_webhook itself (with the full options,
|
||||
// secret token included) — no explicit registration here.
|
||||
let listen = CONFIG.webhook_listen.expect("WEBHOOK_LISTEN is not set");
|
||||
|
||||
@@ -0,0 +1,527 @@
|
||||
//! Pure-Rust photo processing: brings a downloaded photo within Telegram's
|
||||
//! limits (width + height ≤ 10000 px, bytes ≤ 10 MiB) without ffmpeg.
|
||||
//!
|
||||
//! Stack: `png` (image-png) for PNG decode/encode, `zune-jpeg` for JPEG
|
||||
//! decode, `fast_image_resize` (Lanczos3) for downsampling, `jpeg-encoder`
|
||||
//! for JPEG output.
|
||||
//!
|
||||
//! Bit-depth rule: a PNG above 24 bits (32-bit RGBA or 16-bit per channel)
|
||||
//! is reduced to 24-bit RGB; 24-bit and lower depths are left untouched —
|
||||
//! gray stays gray, never upconverted. The only upconversion is palette
|
||||
//! expansion, which resampling requires. Alpha is flattened onto white (JPEG
|
||||
//! and 24-bit RGB have no alpha channel).
|
||||
|
||||
use std::io::Write;
|
||||
|
||||
use fast_image_resize as fir;
|
||||
use tempfile::NamedTempFile;
|
||||
|
||||
/// Telegram rejects photos whose width + height exceed this limit
|
||||
/// (PHOTO_INVALID_DIMENSIONS). Verified empirically: 6300x3730 (sum 10030)
|
||||
/// fails, 6100x3900 (sum 10000) passes.
|
||||
pub const PHOTO_MAX_DIMENSION_SUM: u32 = 10000;
|
||||
/// Resize target with a safety margin so rounding cannot cross the cap.
|
||||
pub const PHOTO_TARGET_DIMENSION_SUM: u32 = 9900;
|
||||
/// Upload cap (bytes): files above this are not uploaded; the bot falls back
|
||||
/// to a smaller media URL instead.
|
||||
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024;
|
||||
/// Decode budget (bytes): a larger intermediate buffer is not worth the peak
|
||||
/// memory; the photo degrades to the smaller URL instead.
|
||||
const MAX_DECODE_BYTES: u64 = 512 * 1024 * 1024;
|
||||
/// JPEG output quality (1-100).
|
||||
const JPEG_QUALITY: u8 = 90;
|
||||
|
||||
/// What to upload for a downloaded photo.
|
||||
pub enum PhotoPrep {
|
||||
/// Upload this file (the original when within limits, else the processed
|
||||
/// copy).
|
||||
Upload(NamedTempFile),
|
||||
/// The photo cannot be brought within Telegram's limits — the caller
|
||||
/// falls back to the item's smaller URL.
|
||||
UseFallback,
|
||||
}
|
||||
|
||||
/// A decoded image buffer tagged with its channel layout.
|
||||
#[derive(Debug)]
|
||||
enum PixBuf {
|
||||
Gray(Vec<u8>),
|
||||
GrayAlpha(Vec<u8>),
|
||||
Rgb(Vec<u8>),
|
||||
}
|
||||
|
||||
impl PixBuf {
|
||||
fn pixel_type(&self) -> fir::PixelType {
|
||||
match self {
|
||||
PixBuf::Gray(_) => fir::PixelType::U8,
|
||||
PixBuf::GrayAlpha(_) => fir::PixelType::U8x2,
|
||||
PixBuf::Rgb(_) => fir::PixelType::U8x3,
|
||||
}
|
||||
}
|
||||
|
||||
fn into_vec(self) -> Vec<u8> {
|
||||
match self {
|
||||
PixBuf::Gray(v) | PixBuf::GrayAlpha(v) | PixBuf::Rgb(v) => v,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Entry point: detects the format and processes the photo if needed.
|
||||
pub fn prepare_photo(file: NamedTempFile) -> Result<PhotoPrep, String> {
|
||||
let bytes = std::fs::read(file.path()).map_err(|e| format!("prepare read failed: {e}"))?;
|
||||
if bytes.starts_with(b"\x89PNG\r\n\x1a\n") {
|
||||
prepare_png(file, bytes)
|
||||
} else if bytes.starts_with(&[0xFF, 0xD8, 0xFF]) {
|
||||
prepare_jpeg(file, bytes)
|
||||
} else {
|
||||
log::warn!("photo in unsupported format; falling back to smaller media");
|
||||
Ok(PhotoPrep::UseFallback)
|
||||
}
|
||||
}
|
||||
|
||||
/// Parses the PNG IHDR (bytes 8..26: signature + length + "IHDR" + width +
|
||||
/// height + bit depth + color type).
|
||||
fn parse_png_header(bytes: &[u8]) -> Option<(u32, u32, png::BitDepth, png::ColorType)> {
|
||||
if !bytes.starts_with(b"\x89PNG\r\n\x1a\n") || bytes.len() < 26 {
|
||||
return None;
|
||||
}
|
||||
let w = u32::from_be_bytes(bytes.get(16..20)?.try_into().ok()?);
|
||||
let h = u32::from_be_bytes(bytes.get(20..24)?.try_into().ok()?);
|
||||
let depth = match *bytes.get(24)? {
|
||||
1 => png::BitDepth::One,
|
||||
2 => png::BitDepth::Two,
|
||||
4 => png::BitDepth::Four,
|
||||
8 => png::BitDepth::Eight,
|
||||
16 => png::BitDepth::Sixteen,
|
||||
_ => return None,
|
||||
};
|
||||
let color = match *bytes.get(25)? {
|
||||
0 => png::ColorType::Grayscale,
|
||||
2 => png::ColorType::Rgb,
|
||||
3 => png::ColorType::Indexed,
|
||||
4 => png::ColorType::GrayscaleAlpha,
|
||||
6 => png::ColorType::Rgba,
|
||||
_ => return None,
|
||||
};
|
||||
Some((w, h, depth, color))
|
||||
}
|
||||
|
||||
/// Output channels of a decoded frame for the given color type (post
|
||||
/// STRIP_16; palette expands to RGB).
|
||||
fn output_channels(color: png::ColorType) -> usize {
|
||||
match color {
|
||||
png::ColorType::Grayscale => 1,
|
||||
png::ColorType::GrayscaleAlpha => 2,
|
||||
png::ColorType::Rgb | png::ColorType::Indexed => 3,
|
||||
png::ColorType::Rgba => 4,
|
||||
}
|
||||
}
|
||||
|
||||
/// The 32→24 rule: RGBA (32-bit) becomes RGB with alpha composited onto
|
||||
/// white; 16-bit per channel was already stripped to 8-bit at decode.
|
||||
fn flatten_rgba_to_rgb(rgba: &[u8]) -> Vec<u8> {
|
||||
let mut rgb = Vec::with_capacity(rgba.len() / 4 * 3);
|
||||
for px in rgba.chunks_exact(4) {
|
||||
let a = px[3] as u32;
|
||||
for v in &px[..3] {
|
||||
// Over white: C = C*a/255 + 255*(1 - a/255).
|
||||
let v = (*v as u32 * a + 255 * (255 - a)) / 255;
|
||||
rgb.push(v.min(255) as u8);
|
||||
}
|
||||
}
|
||||
rgb
|
||||
}
|
||||
|
||||
/// Lanczos3 downsampling via fast_image_resize.
|
||||
fn resize_pix(pix: PixBuf, w: u32, h: u32, nw: u32, nh: u32) -> Result<PixBuf, String> {
|
||||
let pixel_type = pix.pixel_type();
|
||||
let src = fir::images::Image::from_vec_u8(w, h, pix.into_vec(), pixel_type)
|
||||
.map_err(|e| format!("resize input: {e}"))?;
|
||||
let mut dst = fir::images::Image::new(nw, nh, pixel_type);
|
||||
let mut resizer = fir::Resizer::new();
|
||||
let options = fir::ResizeOptions::default()
|
||||
.resize_alg(fir::ResizeAlg::Convolution(fir::FilterType::Lanczos3));
|
||||
resizer
|
||||
.resize(&src, &mut dst, &options)
|
||||
.map_err(|e| format!("resize: {e}"))?;
|
||||
let buf = dst.into_vec();
|
||||
Ok(match pixel_type {
|
||||
fir::PixelType::U8 => PixBuf::Gray(buf),
|
||||
fir::PixelType::U8x2 => PixBuf::GrayAlpha(buf),
|
||||
_ => PixBuf::Rgb(buf),
|
||||
})
|
||||
}
|
||||
|
||||
fn encode_png(out: &mut Vec<u8>, pix: &PixBuf, w: u32, h: u32) -> Result<(), png::EncodingError> {
|
||||
let (color, buf) = match pix {
|
||||
PixBuf::Gray(v) => (png::ColorType::Grayscale, v.as_slice()),
|
||||
PixBuf::GrayAlpha(v) => (png::ColorType::GrayscaleAlpha, v.as_slice()),
|
||||
PixBuf::Rgb(v) => (png::ColorType::Rgb, v.as_slice()),
|
||||
};
|
||||
let mut encoder = png::Encoder::new(out, w, h);
|
||||
encoder.set_color(color);
|
||||
encoder.set_depth(png::BitDepth::Eight);
|
||||
let mut writer = encoder.write_header()?;
|
||||
writer.write_image_data(buf)?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn encode_jpeg(pix: &PixBuf, w: u32, h: u32) -> Result<Vec<u8>, String> {
|
||||
use jpeg_encoder::{ColorType, Encoder};
|
||||
let mut out = Vec::new();
|
||||
let encoder = Encoder::new(&mut out, JPEG_QUALITY);
|
||||
match pix {
|
||||
PixBuf::Gray(v) => encoder
|
||||
.encode(v, w as u16, h as u16, ColorType::Luma)
|
||||
.map_err(|e| format!("jpeg encode: {e}"))?,
|
||||
PixBuf::GrayAlpha(v) => {
|
||||
// JPEG has no alpha: composite onto white, output as gray.
|
||||
let gray: Vec<u8> = v
|
||||
.chunks_exact(2)
|
||||
.map(|px| {
|
||||
let (g, a) = (px[0] as u32, px[1] as u32);
|
||||
((g * a + 255 * (255 - a)) / 255).min(255) as u8
|
||||
})
|
||||
.collect();
|
||||
encoder
|
||||
.encode(&gray, w as u16, h as u16, ColorType::Luma)
|
||||
.map_err(|e| format!("jpeg encode: {e}"))?;
|
||||
}
|
||||
PixBuf::Rgb(v) => encoder
|
||||
.encode(v, w as u16, h as u16, ColorType::Rgb)
|
||||
.map_err(|e| format!("jpeg encode: {e}"))?,
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn write_temp(bytes: &[u8], ext: &str) -> Result<NamedTempFile, String> {
|
||||
let mut file = tempfile::Builder::new()
|
||||
.suffix(&format!(".{ext}"))
|
||||
.tempfile()
|
||||
.map_err(|e| format!("temp file failed: {e}"))?;
|
||||
file.as_file_mut()
|
||||
.write_all(bytes)
|
||||
.map_err(|e| format!("temp file write failed: {e}"))?;
|
||||
Ok(file)
|
||||
}
|
||||
|
||||
fn target_dims(w: u32, h: u32) -> (u32, u32) {
|
||||
let scale = PHOTO_TARGET_DIMENSION_SUM as f64 / (w + h) as f64;
|
||||
(
|
||||
((w as f64 * scale).round() as u32).max(1),
|
||||
((h as f64 * scale).round() as u32).max(1),
|
||||
)
|
||||
}
|
||||
|
||||
/// PNG branch: decode (16→8, palette→RGB; gray/GA stay), flatten RGBA to
|
||||
/// RGB, Lanczos-downscale beyond the dimension cap, encode PNG — a PNG still
|
||||
/// over the upload cap afterwards becomes JPEG.
|
||||
fn prepare_png(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
|
||||
let (w, h, _bit_depth, color_type) = parse_png_header(&bytes).ok_or("invalid PNG header")?;
|
||||
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
|
||||
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
|
||||
return Ok(PhotoPrep::Upload(file));
|
||||
}
|
||||
log::info!(
|
||||
"photo {w}x{h} ({_bit_depth:?} {color_type:?}, {} bytes) needs processing",
|
||||
bytes.len()
|
||||
);
|
||||
|
||||
let channels = output_channels(color_type);
|
||||
if (w as u64) * (h as u64) * channels as u64 > MAX_DECODE_BYTES {
|
||||
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
|
||||
return Ok(PhotoPrep::UseFallback);
|
||||
}
|
||||
|
||||
// STRIP_16 drops 16-bit to 8-bit (the depth-reduction step); palette
|
||||
// expands to RGB (resampling requires it). Gray and gray-alpha are kept.
|
||||
let transforms = match color_type {
|
||||
png::ColorType::Indexed => png::Transformations::EXPAND,
|
||||
_ => png::Transformations::STRIP_16,
|
||||
};
|
||||
let mut decoder = png::Decoder::new(std::io::Cursor::new(&bytes));
|
||||
decoder.set_transformations(transforms);
|
||||
let mut reader = decoder
|
||||
.read_info()
|
||||
.map_err(|e| format!("png decode: {e}"))?;
|
||||
let out_w = reader.info().width;
|
||||
let out_h = reader.info().height;
|
||||
let mut buf = vec![
|
||||
0u8;
|
||||
reader
|
||||
.output_buffer_size()
|
||||
.ok_or("png output buffer size")?
|
||||
];
|
||||
reader
|
||||
.next_frame(&mut buf)
|
||||
.map_err(|e| format!("png frame: {e}"))?;
|
||||
|
||||
let mut pix = match color_type {
|
||||
png::ColorType::Rgba => PixBuf::Rgb(flatten_rgba_to_rgb(&buf)),
|
||||
png::ColorType::Grayscale => PixBuf::Gray(buf),
|
||||
png::ColorType::GrayscaleAlpha => PixBuf::GrayAlpha(buf),
|
||||
png::ColorType::Rgb | png::ColorType::Indexed => PixBuf::Rgb(buf),
|
||||
};
|
||||
|
||||
let (mut w, mut h) = (out_w, out_h);
|
||||
if w + h > PHOTO_MAX_DIMENSION_SUM {
|
||||
let (nw, nh) = target_dims(w, h);
|
||||
pix = resize_pix(pix, w, h, nw, nh)?;
|
||||
(w, h) = (nw, nh);
|
||||
log::info!("downscaled photo to {w}x{h} (Lanczos3)");
|
||||
}
|
||||
|
||||
let mut png_bytes = Vec::new();
|
||||
encode_png(&mut png_bytes, &pix, w, h).map_err(|e| format!("png encode: {e}"))?;
|
||||
if png_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
return Ok(PhotoPrep::Upload(write_temp(&png_bytes, "png")?));
|
||||
}
|
||||
log::info!("PNG still over the upload cap after processing; transcoding to JPEG");
|
||||
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
|
||||
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
|
||||
}
|
||||
log::warn!("processed photo still exceeds the upload cap; falling back to smaller media");
|
||||
Ok(PhotoPrep::UseFallback)
|
||||
}
|
||||
|
||||
/// JPEG branch: zune-jpeg decode → Lanczos downscale → jpeg-encoder output.
|
||||
fn prepare_jpeg(file: NamedTempFile, bytes: Vec<u8>) -> Result<PhotoPrep, String> {
|
||||
let mut decoder = zune_jpeg::JpegDecoder::new(std::io::Cursor::new(&bytes));
|
||||
// Decodes to RGB by default. Headers first so dimensions are known before
|
||||
// the (potentially huge) pixel decode.
|
||||
decoder
|
||||
.decode_headers()
|
||||
.map_err(|e| format!("jpeg headers: {e}"))?;
|
||||
let info = decoder.info().ok_or("jpeg info unavailable")?;
|
||||
let (w, h) = (info.width as u32, info.height as u32);
|
||||
let size_over = bytes.len() as u64 > MAX_UPLOAD_BYTES;
|
||||
if w + h <= PHOTO_MAX_DIMENSION_SUM && !size_over {
|
||||
return Ok(PhotoPrep::Upload(file));
|
||||
}
|
||||
if (w as u64) * (h as u64) * 3 > MAX_DECODE_BYTES {
|
||||
log::warn!("photo decode buffer exceeds the memory budget; falling back to smaller media");
|
||||
return Ok(PhotoPrep::UseFallback);
|
||||
}
|
||||
let pixels = decoder.decode().map_err(|e| format!("jpeg decode: {e}"))?;
|
||||
let mut pix = PixBuf::Rgb(pixels);
|
||||
let (mut w, mut h) = (w, h);
|
||||
if w + h > PHOTO_MAX_DIMENSION_SUM {
|
||||
let (nw, nh) = target_dims(w, h);
|
||||
pix = resize_pix(pix, w, h, nw, nh)?;
|
||||
(w, h) = (nw, nh);
|
||||
log::info!("downscaled jpeg to {w}x{h} (Lanczos3)");
|
||||
}
|
||||
let jpeg_bytes = encode_jpeg(&pix, w, h)?;
|
||||
if jpeg_bytes.len() as u64 <= MAX_UPLOAD_BYTES {
|
||||
return Ok(PhotoPrep::Upload(write_temp(&jpeg_bytes, "jpg")?));
|
||||
}
|
||||
log::warn!("processed photo still exceeds the upload cap; falling back to smaller media");
|
||||
Ok(PhotoPrep::UseFallback)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn png_header(w: u32, h: u32, depth: u8, color: u8) -> Vec<u8> {
|
||||
let mut bytes = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR".to_vec();
|
||||
bytes.extend(w.to_be_bytes());
|
||||
bytes.extend(h.to_be_bytes());
|
||||
bytes.extend([depth, color, 0, 0, 0]);
|
||||
bytes
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_png_header() {
|
||||
let bytes = png_header(8979, 5316, 16, 6); // 16-bit RGBA
|
||||
let (w, h, depth, color) = parse_png_header(&bytes).unwrap();
|
||||
assert_eq!((w, h), (8979, 5316));
|
||||
assert_eq!(depth, png::BitDepth::Sixteen);
|
||||
assert_eq!(color, png::ColorType::Rgba);
|
||||
|
||||
let (_, _, depth, color) = parse_png_header(&png_header(10, 10, 8, 0)).unwrap();
|
||||
assert_eq!(depth, png::BitDepth::Eight);
|
||||
assert_eq!(color, png::ColorType::Grayscale);
|
||||
|
||||
assert!(parse_png_header(b"not a png").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn flatten_rgba_to_rgb_composites_over_white() {
|
||||
// opaque red stays red
|
||||
assert_eq!(flatten_rgba_to_rgb(&[255, 0, 0, 255]), vec![255, 0, 0]);
|
||||
// fully transparent → white
|
||||
assert_eq!(flatten_rgba_to_rgb(&[0, 0, 0, 0]), vec![255, 255, 255]);
|
||||
// half alpha red → (255+255)/2 = 255, (0*128 + 255*127)/255 = 127
|
||||
let out = flatten_rgba_to_rgb(&[255, 0, 0, 128]);
|
||||
assert_eq!(out[0], 255);
|
||||
assert_eq!(out[1], 127);
|
||||
assert_eq!(out[2], 127);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn target_dims_stay_under_the_cap() {
|
||||
for (w, h) in [(12000u32, 7000u32), (10000, 10000), (8979, 5316)] {
|
||||
let (nw, nh) = target_dims(w, h);
|
||||
assert!(nw + nh <= PHOTO_MAX_DIMENSION_SUM, "{w}x{h} -> {nw}x{nh}");
|
||||
assert!(nw >= 1 && nh >= 1);
|
||||
}
|
||||
// already within limits: no change expected from the caller, but the
|
||||
// helper must not produce zero dimensions.
|
||||
let (nw, nh) = target_dims(500, 400);
|
||||
assert!(nw >= 1 && nh >= 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resize_pix_changes_dimensions() {
|
||||
// 300x200 RGB → 100x66
|
||||
let buf: Vec<u8> = (0..300 * 200 * 3).map(|i| (i % 251) as u8).collect();
|
||||
let resized = resize_pix(PixBuf::Rgb(buf), 300, 200, 100, 66).unwrap();
|
||||
match resized {
|
||||
PixBuf::Rgb(v) => assert_eq!(v.len(), 100 * 66 * 3),
|
||||
other => panic!("expected rgb, got {other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn png_encode_roundtrip_keeps_gray() {
|
||||
let gray = vec![128u8; 4 * 4];
|
||||
let mut out = Vec::new();
|
||||
encode_png(&mut out, &PixBuf::Gray(gray), 4, 4).unwrap();
|
||||
assert!(!out.is_empty());
|
||||
let (_, _, depth, color) = parse_png_header(&out).unwrap();
|
||||
assert_eq!(depth, png::BitDepth::Eight);
|
||||
assert_eq!(color, png::ColorType::Grayscale);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn jpeg_encode_produces_bytes() {
|
||||
let rgb = vec![128u8; 8 * 8 * 3];
|
||||
let out = encode_jpeg(&PixBuf::Rgb(rgb), 8, 8).unwrap();
|
||||
assert!(out.len() > 100);
|
||||
assert!(out.starts_with(&[0xFF, 0xD8]));
|
||||
}
|
||||
|
||||
/// Writes a small dimension-oversized PNG (9999x2 → sum 10001) to a temp
|
||||
/// file and runs the full pipeline.
|
||||
fn run_pipeline(w: u32, h: u32, color: png::ColorType, fill: u8) -> Result<PhotoPrep, String> {
|
||||
let (channels, data): (usize, Vec<u8>) = match color {
|
||||
png::ColorType::Grayscale => (1, vec![fill; (w * h) as usize]),
|
||||
png::ColorType::Rgb => (3, vec![fill; (w * h * 3) as usize]),
|
||||
_ => unreachable!(),
|
||||
};
|
||||
let mut bytes = Vec::new();
|
||||
{
|
||||
let mut encoder = png::Encoder::new(&mut bytes, w, h);
|
||||
encoder.set_color(color);
|
||||
encoder.set_depth(png::BitDepth::Eight);
|
||||
let mut writer = encoder.write_header().unwrap();
|
||||
writer.write_image_data(&data).unwrap();
|
||||
}
|
||||
assert_eq!(data.len(), channels * (w * h) as usize);
|
||||
|
||||
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
prepare_photo(file)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pipeline_downscales_oversized_png_keeping_format() {
|
||||
let prep = run_pipeline(9999, 2, png::ColorType::Rgb, 128).unwrap();
|
||||
match prep {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
let (w, h, depth, color) = parse_png_header(&out).unwrap();
|
||||
assert!(w + h <= PHOTO_MAX_DIMENSION_SUM, "{w}x{h}");
|
||||
assert_eq!(depth, png::BitDepth::Eight);
|
||||
assert_eq!(color, png::ColorType::Rgb);
|
||||
}
|
||||
PhotoPrep::UseFallback => panic!("over-dimension PNG should have been resized"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pipeline_keeps_gray_png_gray() {
|
||||
let prep = run_pipeline(9999, 2, png::ColorType::Grayscale, 200).unwrap();
|
||||
match prep {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
let (_, _, _, color) = parse_png_header(&out).unwrap();
|
||||
assert_eq!(color, png::ColorType::Grayscale, "gray must not upconvert");
|
||||
}
|
||||
PhotoPrep::UseFallback => panic!("over-dimension gray PNG should have been resized"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn pipeline_resizes_oversized_jpeg() {
|
||||
// Build a small over-dimension JPEG with jpeg-encoder.
|
||||
let (w, h) = (9999u16, 2u16);
|
||||
let rgb = vec![90u8; (w as usize) * (h as usize) * 3];
|
||||
let mut bytes = Vec::new();
|
||||
{
|
||||
let encoder = jpeg_encoder::Encoder::new(&mut bytes, 90);
|
||||
encoder
|
||||
.encode(&rgb, w, h, jpeg_encoder::ColorType::Rgb)
|
||||
.unwrap();
|
||||
}
|
||||
let mut file = tempfile::Builder::new().suffix(".jpg").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
match prepare_photo(file).unwrap() {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
assert!(out.starts_with(&[0xFF, 0xD8]), "output must stay jpeg");
|
||||
// 9999x2 downscaled: the buffer length tells the new dims.
|
||||
assert!(out.len() > 100);
|
||||
}
|
||||
PhotoPrep::UseFallback => panic!("over-dimension JPEG should have been resized"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[ignore = "heavy: generates a >10 MiB PNG (run explicitly)"]
|
||||
fn pipeline_transcodes_oversized_png_to_jpeg() {
|
||||
// 6000x4000 (sum 10000 — under the dimension cap) smooth gradient with
|
||||
// small per-pixel noise: PNG-incompressible (delta filters defeated)
|
||||
// but JPEG-friendly (DCT smooths the small noise). Verified with
|
||||
// ffmpeg: 8000x6000 amp-5 variant is a 59 MB PNG / 3.3 MB JPEG.
|
||||
let (w, h) = (6000u32, 4000u32);
|
||||
let mut rng = 0x1234_5678_9abc_def0u64;
|
||||
let mut data = Vec::with_capacity((w * h * 3) as usize);
|
||||
for y in 0..h {
|
||||
for x in 0..w {
|
||||
let base = (x + y) * 255 / (w + h);
|
||||
rng = rng
|
||||
.wrapping_mul(6364136223846793005)
|
||||
.wrapping_add(1442695040888963407);
|
||||
let n = ((rng >> 33) % 11) as i32 - 5; // noise in [-5, 5]
|
||||
let v = (base as i32 + n).clamp(0, 255) as u8;
|
||||
data.extend_from_slice(&[v, v, v]);
|
||||
}
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
{
|
||||
let mut encoder = png::Encoder::new(&mut bytes, w, h);
|
||||
encoder.set_color(png::ColorType::Rgb);
|
||||
encoder.set_depth(png::BitDepth::Eight);
|
||||
let mut writer = encoder.write_header().unwrap();
|
||||
writer.write_image_data(&data).unwrap();
|
||||
}
|
||||
assert!(
|
||||
bytes.len() as u64 > MAX_UPLOAD_BYTES,
|
||||
"test needs a >10MiB PNG, got {}",
|
||||
bytes.len()
|
||||
);
|
||||
|
||||
let mut file = tempfile::Builder::new().suffix(".png").tempfile().unwrap();
|
||||
std::io::Write::write_all(file.as_file_mut(), &bytes).unwrap();
|
||||
match prepare_photo(file).unwrap() {
|
||||
PhotoPrep::Upload(file) => {
|
||||
let out = std::fs::read(file.path()).unwrap();
|
||||
assert!(out.starts_with(&[0xFF, 0xD8]), "must transcode to JPEG");
|
||||
assert!(out.len() as u64 <= MAX_UPLOAD_BYTES);
|
||||
}
|
||||
PhotoPrep::UseFallback => panic!("PNG over the byte cap must transcode to JPEG"),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6,11 +6,11 @@
|
||||
//! replaced by dedicated columns.
|
||||
|
||||
use parking_lot::Mutex;
|
||||
use rusqlite::{params, Connection};
|
||||
use rusqlite::{Connection, TransactionBehavior, params};
|
||||
use serde_json::Value;
|
||||
use std::pin::Pin;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use tokio::sync::Notify;
|
||||
use tokio::task::JoinHandle;
|
||||
@@ -18,20 +18,20 @@ use tokio::task::JoinHandle;
|
||||
pub const MAX_RETRIES: u32 = 2;
|
||||
pub const LOCK_TTL_SECONDS: f64 = 120.0;
|
||||
|
||||
/// Number of concurrent worker loops. Tasks are independent (retries and
|
||||
/// forward resumes); leases serialize row claims via SQLite transactions, so
|
||||
/// extra workers drain backlogs faster. Each worker can be mid-send to
|
||||
/// Telegram at the same time as handler tasks, so keep this modest.
|
||||
const QUEUE_WORKERS: usize = 4;
|
||||
|
||||
/// What a handler returns instead of throwing. The payload it carries is the
|
||||
/// (possibly updated) task state to persist for the next attempt.
|
||||
pub enum QueueError {
|
||||
/// Reschedule with the given delay; after `MAX_RETRIES` attempts the task
|
||||
/// is dead-lettered instead.
|
||||
Retryable {
|
||||
delay_seconds: f64,
|
||||
payload: Value,
|
||||
},
|
||||
Retryable { delay_seconds: f64, payload: Value },
|
||||
/// Give up now.
|
||||
Permanent {
|
||||
message: String,
|
||||
payload: Value,
|
||||
},
|
||||
Permanent { message: String, payload: Value },
|
||||
}
|
||||
|
||||
type BoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + Send + 'a>>;
|
||||
@@ -42,7 +42,7 @@ pub struct PersistentTaskQueue {
|
||||
db_path: String,
|
||||
notify: Arc<Notify>,
|
||||
stop: Arc<AtomicBool>,
|
||||
worker: Mutex<Option<JoinHandle<()>>>,
|
||||
worker: Mutex<Vec<JoinHandle<()>>>,
|
||||
counter: AtomicU64,
|
||||
}
|
||||
|
||||
@@ -68,7 +68,7 @@ fn now_f64() -> f64 {
|
||||
.unwrap_or(0.0)
|
||||
}
|
||||
|
||||
fn ensure_schema(conn: &Connection) -> rusqlite::Result<()> {
|
||||
fn ensure_schema(conn: &rusqlite::Connection) -> rusqlite::Result<()> {
|
||||
conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
|
||||
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
|
||||
@@ -96,12 +96,12 @@ impl PersistentTaskQueue {
|
||||
db_path: db_path.to_string(),
|
||||
notify: Arc::new(Notify::new()),
|
||||
stop: Arc::new(AtomicBool::new(false)),
|
||||
worker: Mutex::new(None),
|
||||
worker: Mutex::new(Vec::new()),
|
||||
counter: AtomicU64::new(0),
|
||||
}
|
||||
}
|
||||
|
||||
/// Starts the worker loop. Also recovers rows left `in_progress` by a
|
||||
/// Starts the worker loops. Also recovers rows left `in_progress` by a
|
||||
/// previous process (lease expired).
|
||||
pub async fn start<H, F, D, G>(&self, handler: H, dead_letter: D)
|
||||
where
|
||||
@@ -114,21 +114,25 @@ impl PersistentTaskQueue {
|
||||
let dead_letter: Arc<DeadLetter> =
|
||||
Arc::new(move |payload, message| Box::pin(dead_letter(payload, message)));
|
||||
self.recover_stale().await;
|
||||
let worker = QueueWorker {
|
||||
db_path: self.db_path.clone(),
|
||||
notify: Arc::clone(&self.notify),
|
||||
stop: Arc::clone(&self.stop),
|
||||
handler,
|
||||
dead_letter,
|
||||
};
|
||||
let worker = tokio::spawn(worker.run_loop());
|
||||
*self.worker.lock() = Some(worker);
|
||||
let mut handles = Vec::with_capacity(QUEUE_WORKERS);
|
||||
for _ in 0..QUEUE_WORKERS {
|
||||
let worker = QueueWorker {
|
||||
db_path: self.db_path.clone(),
|
||||
notify: Arc::clone(&self.notify),
|
||||
stop: Arc::clone(&self.stop),
|
||||
handler: Arc::clone(&handler),
|
||||
dead_letter: Arc::clone(&dead_letter),
|
||||
};
|
||||
handles.push(tokio::spawn(worker.run_loop()));
|
||||
}
|
||||
*self.worker.lock() = handles;
|
||||
}
|
||||
|
||||
pub async fn stop(&self) {
|
||||
self.stop.store(true, Ordering::Relaxed);
|
||||
self.notify.notify_one();
|
||||
if let Some(handle) = self.worker.lock().take() {
|
||||
self.notify.notify_waiters();
|
||||
let handles = std::mem::take(&mut *self.worker.lock());
|
||||
for handle in handles {
|
||||
let _ = handle.await;
|
||||
}
|
||||
}
|
||||
@@ -143,10 +147,8 @@ impl PersistentTaskQueue {
|
||||
self.counter.fetch_add(1, Ordering::Relaxed)
|
||||
);
|
||||
let payload = payload.to_string();
|
||||
let db_path = self.db_path.clone();
|
||||
log::info!("enqueued {id} (run_after {run_after:.1})");
|
||||
let result = tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO tasks (id, payload, run_after, attempts, status, locked_until, created_at) \
|
||||
VALUES (?1, ?2, ?3, 0, 'pending', 0, ?4)",
|
||||
@@ -154,25 +156,25 @@ impl PersistentTaskQueue {
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.expect("queue insert worker panicked")?;
|
||||
self.notify.notify_one();
|
||||
Ok(result)
|
||||
.await?;
|
||||
// Wake every sleeping worker: with several workers the one that finds
|
||||
// nothing due must not starve the newly inserted row.
|
||||
self.notify.notify_waiters();
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn recover_stale(&self) {
|
||||
let db_path = self.db_path.clone();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"UPDATE tasks SET status='pending', locked_until=0 WHERE status='in_progress' AND locked_until < ?1",
|
||||
params![now_f64()],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.expect("queue recovery worker panicked")
|
||||
.unwrap_or_else(|e| log::error!("queue recovery failed: {e}"));
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue recovery failed: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -204,13 +206,15 @@ impl QueueWorker {
|
||||
|
||||
/// Leases the oldest due row (sets it `in_progress` with a lock TTL).
|
||||
async fn lease_next(&self) -> Option<LeasedRow> {
|
||||
let db_path = self.db_path.clone();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<LeasedRow>> {
|
||||
let mut conn = Connection::open(&db_path)?;
|
||||
let tx = conn.transaction()?;
|
||||
let result = crate::db::with_conn(&self.db_path, |conn| {
|
||||
// BEGIN IMMEDIATE: with several workers, a deferred transaction
|
||||
// that read before another worker's lease commit would fail with
|
||||
// SQLITE_BUSY_SNAPSHOT. Taking the write lock up front serializes
|
||||
// leases and re-reads the freshest committed state.
|
||||
let tx = conn.transaction_with_behavior(TransactionBehavior::Immediate)?;
|
||||
let now = now_f64();
|
||||
let row = tx.query_row(
|
||||
"SELECT id, payload, attempts FROM tasks WHERE status='pending' AND run_after <= ?1 \
|
||||
"SELECT id, payload, attempts FROM tasks WHERE status='pending' AND run_after <= ?1 AND locked_until <= ?1 \
|
||||
ORDER BY run_after LIMIT 1",
|
||||
params![now],
|
||||
|r| {
|
||||
@@ -240,31 +244,34 @@ impl QueueWorker {
|
||||
attempts,
|
||||
}))
|
||||
})
|
||||
.await
|
||||
.expect("queue lease worker panicked")
|
||||
.unwrap_or_else(|e| {
|
||||
log::error!("queue lease failed: {e}");
|
||||
None
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(row) => row,
|
||||
Err(e) => {
|
||||
log::error!("queue lease failed: {e}");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn earliest_run_after(&self) -> Option<f64> {
|
||||
let db_path = self.db_path.clone();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<f64>> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let mut stmt = conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
|
||||
let result = crate::db::with_conn(&self.db_path, |conn| {
|
||||
let mut stmt =
|
||||
conn.prepare("SELECT MIN(run_after) FROM tasks WHERE status='pending'")?;
|
||||
let mut rows = stmt.query([])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(row.get::<_, Option<f64>>(0)?),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("queue timing worker panicked")
|
||||
.unwrap_or_else(|e| {
|
||||
log::error!("queue timing query failed: {e}");
|
||||
None
|
||||
})
|
||||
.await;
|
||||
match result {
|
||||
Ok(v) => v,
|
||||
Err(e) => {
|
||||
log::error!("queue timing query failed: {e}");
|
||||
None
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn process(&self, row: LeasedRow) {
|
||||
@@ -311,34 +318,32 @@ impl QueueWorker {
|
||||
}
|
||||
|
||||
async fn delete_row(&self, id: &str) {
|
||||
let db_path = self.db_path.clone();
|
||||
let id = id.to_string();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute("DELETE FROM tasks WHERE id = ?1", params![id])?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.expect("queue delete worker panicked")
|
||||
.unwrap_or_else(|e| log::error!("queue delete failed: {e}"));
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue delete failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
async fn reschedule(&self, id: &str, payload: Value, delay_seconds: f64, attempts: i32) {
|
||||
let db_path = self.db_path.clone();
|
||||
let id = id.to_string();
|
||||
let payload = payload.to_string();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"UPDATE tasks SET payload=?1, run_after=?2, attempts=?3, status='pending', locked_until=0 WHERE id=?4",
|
||||
params![payload, now_f64() + delay_seconds, attempts, id],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.expect("queue reschedule worker panicked")
|
||||
.unwrap_or_else(|e| log::error!("queue reschedule failed: {e}"));
|
||||
self.notify.notify_one();
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("queue reschedule failed: {e}");
|
||||
}
|
||||
self.notify.notify_waiters();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+428
-120
@@ -3,20 +3,21 @@
|
||||
//! URL is blocked by hotlink protection; the bot downloads the file itself
|
||||
//! and uploads it via multipart).
|
||||
|
||||
use crate::handlers::{CHAT_STORE, TASK_QUEUE};
|
||||
use crate::handlers::{CHAT_STORE, LINK_CACHE, TASK_QUEUE};
|
||||
use crate::link_cache::{CachedMedia, CachedMediaKind, CachedPost};
|
||||
use crate::photo::{self, MAX_UPLOAD_BYTES, PhotoPrep};
|
||||
use crate::queue::QueueError;
|
||||
use crate::state::{EditMessage, unix_now};
|
||||
use rand::Rng;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use tempfile::NamedTempFile;
|
||||
use teloxide::prelude::*;
|
||||
use teloxide::types::{
|
||||
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia,
|
||||
InputMediaAnimation, InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode,
|
||||
ReplyParameters,
|
||||
ChatId, InlineKeyboardButton, InlineKeyboardMarkup, InputFile, InputMedia, InputMediaAnimation,
|
||||
InputMediaPhoto, InputMediaVideo, Message, MessageId, ParseMode, ReplyParameters,
|
||||
};
|
||||
use teloxide::{ApiError, RequestError};
|
||||
use tempfile::NamedTempFile;
|
||||
use x_media::site::FetchError;
|
||||
|
||||
#[derive(Serialize, Deserialize, Clone, Debug)]
|
||||
@@ -29,6 +30,9 @@ pub enum MediaItemPayload {
|
||||
/// size limits.
|
||||
#[serde(default)]
|
||||
fallback_url: Option<String>,
|
||||
/// `media` is a Telegram file id (link-cache hit), not a URL.
|
||||
#[serde(default)]
|
||||
file_id: bool,
|
||||
},
|
||||
Video {
|
||||
media: String,
|
||||
@@ -36,10 +40,16 @@ pub enum MediaItemPayload {
|
||||
thumbnail: Option<String>,
|
||||
#[serde(default)]
|
||||
fallback_url: Option<String>,
|
||||
/// `media` is a Telegram file id (link-cache hit), not a URL.
|
||||
#[serde(default)]
|
||||
file_id: bool,
|
||||
},
|
||||
Animation {
|
||||
media: String,
|
||||
has_spoiler: bool,
|
||||
/// `media` is a Telegram file id (link-cache hit), not a URL.
|
||||
#[serde(default)]
|
||||
file_id: bool,
|
||||
},
|
||||
}
|
||||
|
||||
@@ -68,6 +78,10 @@ pub enum Task {
|
||||
forward_channel_id: Option<i64>,
|
||||
notify_chat_id: Option<i64>,
|
||||
notify_message_id: Option<i64>,
|
||||
/// Raw render data captured on a cache miss; the send fills in the
|
||||
/// Telegram file ids and persists the entry (see `link_cache`).
|
||||
#[serde(default)]
|
||||
cache_data: Option<CachedPost>,
|
||||
},
|
||||
SendAnimation {
|
||||
chat_id: i64,
|
||||
@@ -79,6 +93,10 @@ pub enum Task {
|
||||
forward_channel_id: Option<i64>,
|
||||
notify_chat_id: Option<i64>,
|
||||
notify_message_id: Option<i64>,
|
||||
/// Raw render data captured on a cache miss; the send fills in the
|
||||
/// Telegram file id and persists the entry (see `link_cache`).
|
||||
#[serde(default)]
|
||||
cache_data: Option<CachedPost>,
|
||||
},
|
||||
ForwardMessages {
|
||||
from_chat_id: i64,
|
||||
@@ -89,14 +107,119 @@ pub enum Task {
|
||||
},
|
||||
}
|
||||
|
||||
impl Task {
|
||||
fn cache_data(&self) -> Option<&CachedPost> {
|
||||
match self {
|
||||
Task::SendMediaSequence { cache_data, .. } | Task::SendAnimation { cache_data, .. } => {
|
||||
cache_data.as_ref()
|
||||
}
|
||||
Task::ForwardMessages { .. } => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn source_url(&self) -> Option<&str> {
|
||||
match self {
|
||||
Task::SendMediaSequence { source_url, .. } | Task::SendAnimation { source_url, .. } => {
|
||||
Some(source_url)
|
||||
}
|
||||
Task::ForwardMessages { .. } => None,
|
||||
}
|
||||
}
|
||||
|
||||
/// True when the media payloads are Telegram file ids from the link cache
|
||||
/// (a cached file id that goes permanently bad should be dropped so the
|
||||
/// next request re-fetches).
|
||||
fn is_cached_send(&self) -> bool {
|
||||
self.cache_data().is_some_and(|c| !c.media.is_empty())
|
||||
}
|
||||
}
|
||||
|
||||
/// Telegram file id of the message's media, matched to the payload kind.
|
||||
fn file_id_of_message(message: &Message, item: &MediaItemPayload) -> Option<String> {
|
||||
match item {
|
||||
// `photo()` returns all sizes, smallest first — the largest carries
|
||||
// the file id of the sent media.
|
||||
MediaItemPayload::Photo { .. } => message
|
||||
.photo()
|
||||
.and_then(|sizes| sizes.last())
|
||||
.map(|p| p.file.id.to_string()),
|
||||
MediaItemPayload::Video { .. } => message.video().map(|v| v.file.id.to_string()),
|
||||
MediaItemPayload::Animation { .. } => message.animation().map(|a| a.file.id.to_string()),
|
||||
}
|
||||
}
|
||||
|
||||
fn kind_of_item(item: &MediaItemPayload) -> CachedMediaKind {
|
||||
match item {
|
||||
MediaItemPayload::Photo { .. } => CachedMediaKind::Photo,
|
||||
MediaItemPayload::Video { .. } => CachedMediaKind::Video,
|
||||
MediaItemPayload::Animation { .. } => CachedMediaKind::Animation,
|
||||
}
|
||||
}
|
||||
|
||||
/// Collects the Telegram file ids of a sent media group, aligned to the
|
||||
/// batch's items.
|
||||
fn collect_file_ids(messages: &[Message], batch: &[MediaItemPayload], out: &mut Vec<CachedMedia>) {
|
||||
for (message, item) in messages.iter().zip(batch.iter()) {
|
||||
if let Some(file_id) = file_id_of_message(message, item) {
|
||||
out.push(CachedMedia {
|
||||
kind: kind_of_item(item),
|
||||
file_id,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Persists a successful send under the post's cache key. Only runs for a
|
||||
/// fresh (non-resumed) task that carried raw cache data with no file ids yet.
|
||||
async fn cache_sent_task(task: &Task, media: Vec<CachedMedia>) {
|
||||
let Some(cache_data) = task.cache_data() else {
|
||||
return;
|
||||
};
|
||||
if !cache_data.media.is_empty() || media.is_empty() {
|
||||
return;
|
||||
}
|
||||
let mut post = cache_data.clone();
|
||||
post.media = media;
|
||||
if let Some(key) = x_media::site::cache_key(&post.url) {
|
||||
LINK_CACHE.put(&key, &post).await;
|
||||
log::info!("cached send for {}", post.url);
|
||||
}
|
||||
}
|
||||
|
||||
/// Persists a lone animation send under the post's cache key.
|
||||
async fn cache_animation_send(task: &Task, message: &Message) {
|
||||
if let Some(file_id) = message.animation().map(|a| a.file.id.to_string()) {
|
||||
cache_sent_task(
|
||||
task,
|
||||
vec![CachedMedia {
|
||||
kind: CachedMediaKind::Animation,
|
||||
file_id,
|
||||
}],
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
/// A cached Telegram file id failed permanently (stale/expired); drop the
|
||||
/// cache entry so the next request re-fetches instead of repeating it.
|
||||
pub async fn invalidate_cache(task: &Task) {
|
||||
if task.is_cached_send()
|
||||
&& let Some(url) = task.source_url()
|
||||
&& let Some(key) = x_media::site::cache_key(url)
|
||||
{
|
||||
log::info!("removing stale link cache entry for {url}");
|
||||
LINK_CACHE.remove(&key).await;
|
||||
}
|
||||
}
|
||||
|
||||
pub const MAX_MEDIA_GROUP: usize = 9;
|
||||
/// Upload cap (bytes): files above this are not uploaded; the bot falls back
|
||||
/// to a smaller media URL instead.
|
||||
pub const MAX_UPLOAD_BYTES: u64 = 10 * 1024 * 1024; // 10485760
|
||||
|
||||
/// Splits media into batches of at most [`MAX_MEDIA_GROUP`] items.
|
||||
pub fn chunk_media_items<T: Clone>(items: Vec<T>) -> Vec<Vec<T>> {
|
||||
items.chunks(MAX_MEDIA_GROUP).map(|chunk| chunk.to_vec()).collect()
|
||||
items
|
||||
.chunks(MAX_MEDIA_GROUP)
|
||||
.map(|chunk| chunk.to_vec())
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Exponential backoff with jitter, capped at 30s.
|
||||
@@ -109,12 +232,15 @@ pub fn retry_delay_seconds(attempts: u32) -> f64 {
|
||||
/// these errors are handled by the download-and-reupload fallback, NOT by a
|
||||
/// queue retry (resending the URL cannot succeed).
|
||||
pub fn is_media_fetch_failure(e: &ApiError) -> bool {
|
||||
const MARKERS: [&str; 5] = [
|
||||
const MARKERS: [&str; 6] = [
|
||||
"webpage_media_empty",
|
||||
"media_empty",
|
||||
"empty_web_media",
|
||||
"webpage_curl_failed",
|
||||
"timeout",
|
||||
// Oversized photos (width + height > 10000 px) are rejected on URL
|
||||
// sends too; route them to the download-and-resize fallback.
|
||||
"PHOTO_INVALID_DIMENSIONS",
|
||||
];
|
||||
let description = e.to_string().to_lowercase();
|
||||
MARKERS.iter().any(|marker| description.contains(marker))
|
||||
@@ -129,31 +255,41 @@ pub fn is_size_error(e: &ApiError) -> bool {
|
||||
return true;
|
||||
}
|
||||
let description = e.to_string().to_lowercase();
|
||||
["too large", "too big"].iter().any(|marker| description.contains(marker))
|
||||
["too large", "too big"]
|
||||
.iter()
|
||||
.any(|marker| description.contains(marker))
|
||||
}
|
||||
|
||||
/// Task-free classification of a Telegram request error. The callers attach
|
||||
/// the (updated) task when building a [`SendError`].
|
||||
pub enum Classification {
|
||||
Retryable { delay_seconds: f64 },
|
||||
Permanent { message: String },
|
||||
Retryable {
|
||||
delay_seconds: f64,
|
||||
},
|
||||
Permanent {
|
||||
message: String,
|
||||
},
|
||||
/// Handled by the download fallback, not a queue retry.
|
||||
MediaFetchFailure,
|
||||
}
|
||||
|
||||
pub fn classify_request_error(e: &RequestError) -> Classification {
|
||||
match e {
|
||||
RequestError::RetryAfter(seconds) => {
|
||||
Classification::Retryable { delay_seconds: seconds.seconds() as f64 }
|
||||
}
|
||||
RequestError::RetryAfter(seconds) => Classification::Retryable {
|
||||
delay_seconds: seconds.seconds() as f64,
|
||||
},
|
||||
RequestError::Network(_) => Classification::Retryable {
|
||||
delay_seconds: retry_delay_seconds(0),
|
||||
},
|
||||
RequestError::Api(api) if is_media_fetch_failure(api) => Classification::MediaFetchFailure,
|
||||
RequestError::Api(api) => Classification::Permanent { message: api.to_string() },
|
||||
RequestError::Api(api) => Classification::Permanent {
|
||||
message: api.to_string(),
|
||||
},
|
||||
RequestError::MigrateToChatId(_)
|
||||
| RequestError::InvalidJson { .. }
|
||||
| RequestError::Io(_) => Classification::Permanent { message: e.to_string() },
|
||||
| RequestError::Io(_) => Classification::Permanent {
|
||||
message: e.to_string(),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -198,6 +334,32 @@ fn input_file_for(media: &str) -> Result<InputFile, String> {
|
||||
}
|
||||
}
|
||||
|
||||
impl MediaItemPayload {
|
||||
/// The input for a send: a cached file id goes out as `InputFile::file_id`
|
||||
/// (no fetch, no upload), URLs go to Telegram, anything else is a local
|
||||
/// path (transient upload fallback).
|
||||
fn input_file(&self) -> Result<InputFile, String> {
|
||||
match self {
|
||||
MediaItemPayload::Photo {
|
||||
media,
|
||||
file_id: true,
|
||||
..
|
||||
}
|
||||
| MediaItemPayload::Video {
|
||||
media,
|
||||
file_id: true,
|
||||
..
|
||||
}
|
||||
| MediaItemPayload::Animation {
|
||||
media,
|
||||
file_id: true,
|
||||
..
|
||||
} => Ok(InputFile::file_id(media.clone().into())),
|
||||
_ => input_file_for(item_url(self)),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn photo_media(file: InputFile, caption: Option<&str>, spoiler: bool) -> InputMedia {
|
||||
let mut photo = InputMediaPhoto::new(file).parse_mode(ParseMode::Html);
|
||||
if let Some(caption) = caption {
|
||||
@@ -243,28 +405,23 @@ fn build_media_group(
|
||||
.map(|(i, item)| {
|
||||
let item_caption = if i == 0 { caption } else { None };
|
||||
Ok(match item {
|
||||
MediaItemPayload::Photo {
|
||||
media,
|
||||
has_spoiler,
|
||||
..
|
||||
} => photo_media(input_file_for(media)?, item_caption, *has_spoiler),
|
||||
MediaItemPayload::Photo { has_spoiler, .. } => {
|
||||
photo_media(item.input_file()?, item_caption, *has_spoiler)
|
||||
}
|
||||
MediaItemPayload::Video {
|
||||
media,
|
||||
has_spoiler,
|
||||
thumbnail,
|
||||
..
|
||||
} => {
|
||||
let mut video = video_media(input_file_for(media)?, item_caption, *has_spoiler);
|
||||
let mut video = video_media(item.input_file()?, item_caption, *has_spoiler);
|
||||
if let (Some(thumb), InputMedia::Video(v)) = (thumbnail, &mut video) {
|
||||
*v = v.clone().thumbnail(input_file_for(thumb)?);
|
||||
}
|
||||
video
|
||||
}
|
||||
MediaItemPayload::Animation {
|
||||
media,
|
||||
has_spoiler,
|
||||
..
|
||||
} => animation_media(input_file_for(media)?, item_caption, *has_spoiler),
|
||||
MediaItemPayload::Animation { has_spoiler, .. } => {
|
||||
animation_media(item.input_file()?, item_caption, *has_spoiler)
|
||||
}
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
@@ -289,13 +446,24 @@ fn sniff_ext(bytes: &[u8]) -> &'static str {
|
||||
}
|
||||
|
||||
enum FallbackError {
|
||||
Retryable { delay_seconds: f64 },
|
||||
Permanent { message: String },
|
||||
Retryable {
|
||||
delay_seconds: f64,
|
||||
},
|
||||
Permanent {
|
||||
message: String,
|
||||
},
|
||||
/// The downloaded file exceeds the upload cap; the caller falls back to
|
||||
/// the item's smaller URL.
|
||||
MediaTooLarge,
|
||||
}
|
||||
|
||||
/// Brings a downloaded photo within Telegram's limits via the pure-Rust
|
||||
/// chain in [`crate::photo`] (no ffmpeg): dimension cap / upload cap
|
||||
/// exceeded photos are decoded, downscaled with Lanczos3, PNG bit depth
|
||||
/// reduced (>24-bit → 24-bit RGB, ≤24-bit untouched) and transcoded to JPEG
|
||||
/// only if still too big. Anything that cannot be fixed falls back to the
|
||||
/// item's smaller URL.
|
||||
///
|
||||
/// Downloads one media item to a temp file (deleted on drop). Network errors
|
||||
/// are retryable; size over the upload cap and other download errors are not.
|
||||
async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, FallbackError> {
|
||||
@@ -317,7 +485,9 @@ async fn download_to_temp(item: &MediaItemPayload) -> Result<NamedTempFile, Fall
|
||||
});
|
||||
}
|
||||
};
|
||||
if bytes.len() as u64 > MAX_UPLOAD_BYTES {
|
||||
// Photos are downloaded even over the cap so `prepare_photo` can
|
||||
// downscale / transcode them; only videos/animations short-circuit.
|
||||
if !matches!(item, MediaItemPayload::Photo { .. }) && bytes.len() as u64 > MAX_UPLOAD_BYTES {
|
||||
return Err(FallbackError::MediaTooLarge);
|
||||
}
|
||||
let ext = sniff_ext(&bytes);
|
||||
@@ -390,11 +560,13 @@ async fn send_batch_via_upload(
|
||||
for (i, item) in batch.iter().enumerate() {
|
||||
let item_caption = if i == 0 { caption } else { None };
|
||||
// Size check before downloading/uploading: over the cap, use the
|
||||
// smaller URL instead of the file.
|
||||
// smaller URL instead of the file. Photos are exempt — they are
|
||||
// downloaded and processed (downscale / PNG→JPEG) before uploading.
|
||||
let too_large = match x_media::site::media_size(item_url(item)).await {
|
||||
Ok(Some(size)) => size > MAX_UPLOAD_BYTES,
|
||||
_ => false,
|
||||
};
|
||||
let too_large = too_large && !matches!(item, MediaItemPayload::Photo { .. });
|
||||
let media = if too_large {
|
||||
match item.fallback_url() {
|
||||
Some(url) => match media_from_url(item, url, item_caption) {
|
||||
@@ -412,9 +584,46 @@ async fn send_batch_via_upload(
|
||||
} else {
|
||||
match download_to_temp(item).await {
|
||||
Ok(file) => {
|
||||
let path = file.path().to_path_buf();
|
||||
files.push(file);
|
||||
media_from_file(item, path, item_caption)
|
||||
// Telegram rejects photos wider+taller than 10000 px
|
||||
// combined (PHOTO_INVALID_DIMENSIONS): downscale the
|
||||
// downloaded file before uploading; photos that cannot be
|
||||
// brought within the limits degrade to the smaller URL.
|
||||
if matches!(item, MediaItemPayload::Photo { .. }) {
|
||||
// CPU-heavy (decode/resize/encode): run off the async
|
||||
// executor thread.
|
||||
let prep = tokio::task::spawn_blocking(move || photo::prepare_photo(file))
|
||||
.await
|
||||
.map_err(|e| FallbackError::Permanent {
|
||||
message: format!("photo worker panicked: {e}"),
|
||||
})?
|
||||
.map_err(|message| FallbackError::Permanent { message })?;
|
||||
match prep {
|
||||
PhotoPrep::Upload(upload) => {
|
||||
let path = upload.path().to_path_buf();
|
||||
files.push(upload);
|
||||
media_from_file(item, path, item_caption)
|
||||
}
|
||||
PhotoPrep::UseFallback => match item.fallback_url() {
|
||||
Some(url) => match media_from_url(item, url, item_caption) {
|
||||
Ok(media) => media,
|
||||
Err(message) => {
|
||||
return Err(FallbackError::Permanent { message });
|
||||
}
|
||||
},
|
||||
None => {
|
||||
return Err(FallbackError::Permanent {
|
||||
message:
|
||||
"photo dimensions exceed Telegram limits and no smaller variant is available"
|
||||
.into(),
|
||||
});
|
||||
}
|
||||
},
|
||||
}
|
||||
} else {
|
||||
let path = file.path().to_path_buf();
|
||||
files.push(file);
|
||||
media_from_file(item, path, item_caption)
|
||||
}
|
||||
}
|
||||
Err(FallbackError::MediaTooLarge) => match item.fallback_url() {
|
||||
Some(url) => match media_from_url(item, url, item_caption) {
|
||||
@@ -436,14 +645,16 @@ async fn send_batch_via_upload(
|
||||
}
|
||||
let result = bot
|
||||
.send_media_group(ChatId(chat_id), items)
|
||||
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
|
||||
.reply_parameters(
|
||||
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
|
||||
)
|
||||
.await;
|
||||
match result {
|
||||
Ok(messages) => Ok(messages),
|
||||
Err(e) => Err(match classify_request_error(&e) {
|
||||
Classification::Retryable { delay_seconds } => FallbackError::Retryable {
|
||||
delay_seconds,
|
||||
},
|
||||
Classification::Retryable { delay_seconds } => {
|
||||
FallbackError::Retryable { delay_seconds }
|
||||
}
|
||||
Classification::Permanent { message } => FallbackError::Permanent { message },
|
||||
Classification::MediaFetchFailure => FallbackError::Permanent {
|
||||
message: "upload failed".into(),
|
||||
@@ -459,12 +670,14 @@ fn updated_sequence_task(task: &Task, batch_index: usize, sent_message_ids: Vec<
|
||||
reply_to_message_id,
|
||||
caption,
|
||||
media_batches,
|
||||
batch_index: _,
|
||||
sent_message_ids: _,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
..
|
||||
cache_data,
|
||||
} => Task::SendMediaSequence {
|
||||
chat_id: *chat_id,
|
||||
reply_to_message_id: *reply_to_message_id,
|
||||
@@ -477,6 +690,7 @@ fn updated_sequence_task(task: &Task, batch_index: usize, sent_message_ids: Vec<
|
||||
forward_channel_id: *forward_channel_id,
|
||||
notify_chat_id: *notify_chat_id,
|
||||
notify_message_id: *notify_message_id,
|
||||
cache_data: cache_data.clone(),
|
||||
},
|
||||
_ => unreachable!("updated_sequence_task requires a SendMediaSequence task"),
|
||||
}
|
||||
@@ -501,9 +715,17 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
|
||||
let chat_id = *chat_id;
|
||||
let reply_to = *reply_to_message_id;
|
||||
let mut sent = sent_message_ids.clone();
|
||||
// File ids accumulated across batches for the link cache. Only a fresh
|
||||
// (non-resumed) full send populates the cache.
|
||||
let mut cached_media: Vec<CachedMedia> = Vec::new();
|
||||
let fresh_send = *batch_index == 0 && sent.is_empty();
|
||||
for idx in *batch_index..media_batches.len() {
|
||||
let batch = &media_batches[idx];
|
||||
let caption = if idx == 0 { Some(caption.as_str()) } else { None };
|
||||
let caption = if idx == 0 {
|
||||
Some(caption.as_str())
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let items = match build_media_group(batch, caption) {
|
||||
Ok(items) => items,
|
||||
Err(message) => {
|
||||
@@ -515,7 +737,9 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
|
||||
};
|
||||
match bot
|
||||
.send_media_group(ChatId(chat_id), items)
|
||||
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply())
|
||||
.reply_parameters(
|
||||
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(messages) => {
|
||||
@@ -524,20 +748,19 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
|
||||
media_batches.len(),
|
||||
batch.len()
|
||||
);
|
||||
collect_file_ids(&messages, batch, &mut cached_media);
|
||||
sent.extend(messages.into_iter().map(|m| m.id.0 as i64));
|
||||
}
|
||||
Err(RequestError::Api(api))
|
||||
if is_media_fetch_failure(&api) || is_size_error(&api) =>
|
||||
{
|
||||
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
|
||||
log::info!(
|
||||
"Telegram could not fetch media for batch {idx} ({}), downloading and reuploading",
|
||||
batch
|
||||
.first()
|
||||
.map(|item| item_url(item))
|
||||
.unwrap_or("?")
|
||||
batch.first().map(item_url).unwrap_or("?")
|
||||
);
|
||||
match send_batch_via_upload(bot, chat_id, reply_to, batch, caption).await {
|
||||
Ok(messages) => sent.extend(messages.into_iter().map(|m| m.id.0 as i64)),
|
||||
Ok(messages) => {
|
||||
collect_file_ids(&messages, batch, &mut cached_media);
|
||||
sent.extend(messages.into_iter().map(|m| m.id.0 as i64));
|
||||
}
|
||||
Err(FallbackError::Retryable { delay_seconds }) => {
|
||||
return Err(SendError::Retryable {
|
||||
delay_seconds,
|
||||
@@ -561,6 +784,9 @@ pub async fn send_media_sequence(bot: &Bot, task: &Task) -> Result<Vec<i64>, Sen
|
||||
}
|
||||
}
|
||||
}
|
||||
if fresh_send {
|
||||
cache_sent_task(task, cached_media).await;
|
||||
}
|
||||
Ok(sent)
|
||||
}
|
||||
|
||||
@@ -576,7 +802,9 @@ async fn send_animation_inner(
|
||||
.send_animation(ChatId(chat_id), file)
|
||||
.caption(caption)
|
||||
.parse_mode(ParseMode::Html)
|
||||
.reply_parameters(ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply());
|
||||
.reply_parameters(
|
||||
ReplyParameters::new(MessageId(reply_to as i32)).allow_sending_without_reply(),
|
||||
);
|
||||
if spoiler {
|
||||
request = request.has_spoiler(true);
|
||||
}
|
||||
@@ -599,8 +827,7 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
|
||||
let reply_to = *reply_to_message_id;
|
||||
let (media_url, has_spoiler) = match animation {
|
||||
MediaItemPayload::Animation {
|
||||
media,
|
||||
has_spoiler,
|
||||
media, has_spoiler, ..
|
||||
} => (media, *has_spoiler),
|
||||
MediaItemPayload::Photo { .. } | MediaItemPayload::Video { .. } => {
|
||||
unreachable!("SendAnimation carries an Animation payload")
|
||||
@@ -608,15 +835,20 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
|
||||
};
|
||||
let url_file = match input_file_for(media_url) {
|
||||
Ok(file) => file,
|
||||
Err(message) => return Err(SendError::Permanent { message, task: task.clone() }),
|
||||
Err(message) => {
|
||||
return Err(SendError::Permanent {
|
||||
message,
|
||||
task: task.clone(),
|
||||
});
|
||||
}
|
||||
};
|
||||
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file)
|
||||
.await
|
||||
{
|
||||
Ok(message) => Ok(vec![message.id.0 as i64]),
|
||||
Err(RequestError::Api(api))
|
||||
if is_media_fetch_failure(&api) || is_size_error(&api) =>
|
||||
{
|
||||
match send_animation_inner(bot, chat_id, reply_to, caption, has_spoiler, url_file).await {
|
||||
Ok(message) => {
|
||||
let id = message.id.0 as i64;
|
||||
cache_animation_send(task, &message).await;
|
||||
Ok(vec![id])
|
||||
}
|
||||
Err(RequestError::Api(api)) if is_media_fetch_failure(&api) || is_size_error(&api) => {
|
||||
log::info!(
|
||||
"Telegram could not fetch animation URL, downloading and reuploading: {}",
|
||||
media_url
|
||||
@@ -634,7 +866,11 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(message) => Ok(vec![message.id.0 as i64]),
|
||||
Ok(message) => {
|
||||
let id = message.id.0 as i64;
|
||||
cache_animation_send(task, &message).await;
|
||||
Ok(vec![id])
|
||||
}
|
||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
||||
}
|
||||
}
|
||||
@@ -652,25 +888,32 @@ pub async fn send_animation(bot: &Bot, task: &Task) -> Result<Vec<i64>, SendErro
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(message) => Ok(vec![message.id.0 as i64]),
|
||||
Ok(message) => {
|
||||
let id = message.id.0 as i64;
|
||||
cache_animation_send(task, &message).await;
|
||||
Ok(vec![id])
|
||||
}
|
||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
||||
}
|
||||
}
|
||||
Err(message) => {
|
||||
Err(SendError::Permanent { message, task: task.clone() })
|
||||
}
|
||||
Err(message) => Err(SendError::Permanent {
|
||||
message,
|
||||
task: task.clone(),
|
||||
}),
|
||||
},
|
||||
None => Err(SendError::Permanent {
|
||||
message: "media too large".into(),
|
||||
task: task.clone(),
|
||||
}),
|
||||
},
|
||||
Err(FallbackError::Retryable { delay_seconds }) => {
|
||||
Err(SendError::Retryable { delay_seconds, task: task.clone() })
|
||||
}
|
||||
Err(FallbackError::Permanent { message }) => {
|
||||
Err(SendError::Permanent { message, task: task.clone() })
|
||||
}
|
||||
Err(FallbackError::Retryable { delay_seconds }) => Err(SendError::Retryable {
|
||||
delay_seconds,
|
||||
task: task.clone(),
|
||||
}),
|
||||
Err(FallbackError::Permanent { message }) => Err(SendError::Permanent {
|
||||
message,
|
||||
task: task.clone(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
Err(e) => Err(classify_to_send_error(&e, task.clone())),
|
||||
@@ -694,7 +937,11 @@ pub async fn forward_messages(bot: &Bot, task: &Task) -> Result<(), SendError> {
|
||||
.map(|id| MessageId(*id as i32))
|
||||
.collect::<Vec<_>>();
|
||||
match bot
|
||||
.copy_messages(ChatId(*to_chat_id), ChatId(*from_chat_id), message_ids.clone())
|
||||
.copy_messages(
|
||||
ChatId(*to_chat_id),
|
||||
ChatId(*from_chat_id),
|
||||
message_ids.clone(),
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => {
|
||||
@@ -728,12 +975,18 @@ pub fn build_edit_markup(templates: &HashMap<String, String>) -> InlineKeyboardM
|
||||
|
||||
/// Notifies a chat about a dead-lettered task (skips when `notify_chat_id` is
|
||||
/// absent).
|
||||
pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<i64>, message: &str) {
|
||||
pub async fn notify_failure(
|
||||
bot: &Bot,
|
||||
chat_id: Option<i64>,
|
||||
message_id: Option<i64>,
|
||||
message: &str,
|
||||
) {
|
||||
let Some(chat_id) = chat_id else { return };
|
||||
let mut request = bot.send_message(ChatId(chat_id), message);
|
||||
if let Some(message_id) = message_id {
|
||||
request = request
|
||||
.reply_parameters(ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply());
|
||||
request = request.reply_parameters(
|
||||
ReplyParameters::new(MessageId(message_id as i32)).allow_sending_without_reply(),
|
||||
);
|
||||
}
|
||||
if let Err(e) = request.await {
|
||||
log::error!("failed to notify about failed task: {e}");
|
||||
@@ -743,38 +996,45 @@ pub async fn notify_failure(bot: &Bot, chat_id: Option<i64>, message_id: Option<
|
||||
/// After a successful send: either open the edit-before-forward prompt or
|
||||
/// forward to the configured channel (with retry/queue handling).
|
||||
pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
|
||||
let (chat_id, reply_to, source_url, edit_before_forward, forward_channel_id, notify_chat_id, notify_message_id) =
|
||||
match task {
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
..
|
||||
}
|
||||
| Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
..
|
||||
} => (
|
||||
*chat_id,
|
||||
*reply_to_message_id,
|
||||
source_url.clone(),
|
||||
*edit_before_forward,
|
||||
*forward_channel_id,
|
||||
*notify_chat_id,
|
||||
*notify_message_id,
|
||||
),
|
||||
Task::ForwardMessages { .. } => return,
|
||||
};
|
||||
let (
|
||||
chat_id,
|
||||
reply_to,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
) = match task {
|
||||
Task::SendMediaSequence {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
..
|
||||
}
|
||||
| Task::SendAnimation {
|
||||
chat_id,
|
||||
reply_to_message_id,
|
||||
source_url,
|
||||
edit_before_forward,
|
||||
forward_channel_id,
|
||||
notify_chat_id,
|
||||
notify_message_id,
|
||||
..
|
||||
} => (
|
||||
*chat_id,
|
||||
*reply_to_message_id,
|
||||
source_url.clone(),
|
||||
*edit_before_forward,
|
||||
*forward_channel_id,
|
||||
*notify_chat_id,
|
||||
*notify_message_id,
|
||||
),
|
||||
Task::ForwardMessages { .. } => return,
|
||||
};
|
||||
|
||||
if edit_before_forward {
|
||||
let mut chat_data = CHAT_STORE.get(chat_id).await;
|
||||
@@ -811,7 +1071,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
|
||||
}
|
||||
|
||||
if let Some(channel_id) = forward_channel_id {
|
||||
log::info!("forwarding {} message(s) to channel {channel_id}", message_ids.len());
|
||||
log::info!(
|
||||
"forwarding {} message(s) to channel {channel_id}",
|
||||
message_ids.len()
|
||||
);
|
||||
let forward_task = Task::ForwardMessages {
|
||||
from_chat_id: chat_id,
|
||||
to_chat_id: channel_id,
|
||||
@@ -821,7 +1084,10 @@ pub async fn post_send_actions(bot: &Bot, task: &Task, message_ids: Vec<i64>) {
|
||||
};
|
||||
match forward_messages(bot, &forward_task).await {
|
||||
Ok(()) => {}
|
||||
Err(SendError::Retryable { delay_seconds, task }) => {
|
||||
Err(SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
let payload = serde_json::to_value(task).expect("task serializes");
|
||||
let run_after = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
@@ -861,13 +1127,17 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
|
||||
Task::SendMediaSequence { .. } | Task::SendAnimation { .. } => {
|
||||
let message_ids = match send_media_or_animation(&bot, &task).await {
|
||||
Ok(ids) => ids,
|
||||
Err(SendError::Retryable { delay_seconds, task }) => {
|
||||
Err(SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => {
|
||||
return Err(QueueError::Retryable {
|
||||
delay_seconds,
|
||||
payload: serde_json::to_value(task).expect("task serializes"),
|
||||
});
|
||||
}
|
||||
Err(SendError::Permanent { message, task }) => {
|
||||
invalidate_cache(&task).await;
|
||||
return Err(QueueError::Permanent {
|
||||
message,
|
||||
payload: serde_json::to_value(task).expect("task serializes"),
|
||||
@@ -879,7 +1149,10 @@ pub async fn handle_task(payload: serde_json::Value) -> Result<(), QueueError> {
|
||||
}
|
||||
Task::ForwardMessages { .. } => match forward_messages(&bot, &task).await {
|
||||
Ok(()) => Ok(()),
|
||||
Err(SendError::Retryable { delay_seconds, task }) => Err(QueueError::Retryable {
|
||||
Err(SendError::Retryable {
|
||||
delay_seconds,
|
||||
task,
|
||||
}) => Err(QueueError::Retryable {
|
||||
delay_seconds,
|
||||
payload: serde_json::to_value(task).expect("task serializes"),
|
||||
}),
|
||||
@@ -919,6 +1192,14 @@ pub async fn dead_letter_notify(payload: serde_json::Value, message: String) {
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn oversized_photo_boundary() {
|
||||
// The empirical Telegram limit: sum 10000 passes, 10001 fails.
|
||||
assert!(crate::photo::PHOTO_MAX_DIMENSION_SUM == 10000);
|
||||
assert!(6100 + 3900 <= crate::photo::PHOTO_MAX_DIMENSION_SUM);
|
||||
assert!(6300 + 3730 > crate::photo::PHOTO_MAX_DIMENSION_SUM);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn chunk_media_items_sizes() {
|
||||
assert_eq!(chunk_media_items::<i32>(vec![]), Vec::<Vec<i32>>::new());
|
||||
@@ -926,7 +1207,11 @@ mod tests {
|
||||
assert_eq!(chunk_media_items((0..10).collect()).len(), 2);
|
||||
assert_eq!(chunk_media_items((0..25).collect()).len(), 3);
|
||||
assert_eq!(chunk_media_items((0..25).collect())[2].len(), 7);
|
||||
assert!(chunk_media_items((0..25).collect()).iter().all(|c| c.len() <= 9));
|
||||
assert!(
|
||||
chunk_media_items((0..25).collect())
|
||||
.iter()
|
||||
.all(|c| c.len() <= 9)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -950,7 +1235,10 @@ mod tests {
|
||||
let api = ApiError::Unknown(description.to_string());
|
||||
assert!(is_media_fetch_failure(&api), "{description}");
|
||||
}
|
||||
for description in ["Bad Request: message is not modified", "Forbidden: bot was blocked by the user"] {
|
||||
for description in [
|
||||
"Bad Request: message is not modified",
|
||||
"Forbidden: bot was blocked by the user",
|
||||
] {
|
||||
let api = ApiError::Unknown(description.to_string());
|
||||
assert!(!is_media_fetch_failure(&api), "{description}");
|
||||
}
|
||||
@@ -971,7 +1259,10 @@ mod tests {
|
||||
assert!(is_size_error(&api), "{description}");
|
||||
}
|
||||
// Unrelated errors must not match.
|
||||
for description in ["Bad Request: WEBPAGE_MEDIA_EMPTY", "Bad Request: message is not modified"] {
|
||||
for description in [
|
||||
"Bad Request: WEBPAGE_MEDIA_EMPTY",
|
||||
"Bad Request: message is not modified",
|
||||
] {
|
||||
let api = ApiError::Unknown(description.to_string());
|
||||
assert!(!is_size_error(&api), "{description}");
|
||||
}
|
||||
@@ -980,9 +1271,16 @@ mod tests {
|
||||
#[test]
|
||||
fn media_item_payload_fallback_url_serde_default() {
|
||||
// Old queued payloads without the field deserialize with None.
|
||||
let json = serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
|
||||
let json =
|
||||
serde_json::json!({"kind": "photo", "media": "https://a/b.jpg", "has_spoiler": false});
|
||||
let photo: MediaItemPayload = serde_json::from_value(json).unwrap();
|
||||
assert!(matches!(photo, MediaItemPayload::Photo { fallback_url: None, .. }));
|
||||
assert!(matches!(
|
||||
photo,
|
||||
MediaItemPayload::Photo {
|
||||
fallback_url: None,
|
||||
..
|
||||
}
|
||||
));
|
||||
assert_eq!(photo.fallback_url(), None);
|
||||
}
|
||||
|
||||
@@ -1026,12 +1324,14 @@ mod tests {
|
||||
media: "https://a/b.jpg".into(),
|
||||
has_spoiler: true,
|
||||
fallback_url: Some("https://a/b_small.jpg".into()),
|
||||
file_id: false,
|
||||
}],
|
||||
vec![MediaItemPayload::Video {
|
||||
media: "https://a/v.mp4".into(),
|
||||
has_spoiler: false,
|
||||
thumbnail: Some("https://a/t.jpg".into()),
|
||||
fallback_url: None,
|
||||
file_id: false,
|
||||
}],
|
||||
],
|
||||
batch_index: 1,
|
||||
@@ -1041,6 +1341,7 @@ mod tests {
|
||||
forward_channel_id: Some(333),
|
||||
notify_chat_id: Some(111),
|
||||
notify_message_id: Some(222),
|
||||
cache_data: None,
|
||||
};
|
||||
let json = serde_json::to_value(&task).unwrap();
|
||||
assert_eq!(json["type"], "send_media_sequence");
|
||||
@@ -1058,7 +1359,13 @@ mod tests {
|
||||
assert_eq!(sent_message_ids, vec![11, 12]);
|
||||
assert_eq!(forward_channel_id, Some(333));
|
||||
assert_eq!(media_batches.len(), 2);
|
||||
assert!(matches!(media_batches[0][0], MediaItemPayload::Photo { has_spoiler: true, .. }));
|
||||
assert!(matches!(
|
||||
media_batches[0][0],
|
||||
MediaItemPayload::Photo {
|
||||
has_spoiler: true,
|
||||
..
|
||||
}
|
||||
));
|
||||
}
|
||||
other => panic!("expected SendMediaSequence, got {other:?}"),
|
||||
}
|
||||
@@ -1070,6 +1377,7 @@ mod tests {
|
||||
media: "https://a/b.jpg".into(),
|
||||
has_spoiler: false,
|
||||
fallback_url: None,
|
||||
file_id: false,
|
||||
};
|
||||
let json = serde_json::to_value(&photo).unwrap();
|
||||
assert_eq!(json["kind"], "photo");
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
//! `data/task_queue.db`, shared with the task queue).
|
||||
|
||||
use parking_lot::Mutex;
|
||||
use rusqlite::{params, Connection};
|
||||
use rusqlite::params;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::HashMap;
|
||||
use std::path::Path;
|
||||
@@ -45,7 +45,9 @@ pub fn unix_now() -> i64 {
|
||||
}
|
||||
|
||||
impl ChatStore {
|
||||
/// Creates the parent directory and both tables (idempotent).
|
||||
/// Creates the parent directory and the `chat_state` table (idempotent).
|
||||
/// The shared `tasks` / `link_cache` tables are owned by `queue.rs` and
|
||||
/// `link_cache.rs` respectively.
|
||||
pub fn open(path: &str) -> rusqlite::Result<Self> {
|
||||
if let Some(parent) = Path::new(path).parent()
|
||||
&& !parent.as_os_str().is_empty()
|
||||
@@ -53,12 +55,9 @@ impl ChatStore {
|
||||
std::fs::create_dir_all(parent)
|
||||
.map_err(|e| rusqlite::Error::ToSqlConversionFailure(Box::new(e)))?;
|
||||
}
|
||||
let conn = Connection::open(path)?;
|
||||
let conn = crate::db::open_db(path)?;
|
||||
conn.execute_batch(
|
||||
"CREATE TABLE IF NOT EXISTS tasks (id TEXT PRIMARY KEY, payload TEXT NOT NULL, \
|
||||
run_after REAL NOT NULL, attempts INTEGER NOT NULL, status TEXT NOT NULL, \
|
||||
locked_until REAL NOT NULL, created_at REAL NOT NULL); \
|
||||
CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
|
||||
"CREATE TABLE IF NOT EXISTS chat_state (chat_id TEXT PRIMARY KEY, payload TEXT NOT NULL);",
|
||||
)?;
|
||||
drop(conn);
|
||||
Ok(ChatStore {
|
||||
@@ -71,18 +70,19 @@ impl ChatStore {
|
||||
if let Some(data) = self.cache.lock().get(&chat_id) {
|
||||
return data.clone();
|
||||
}
|
||||
let db_path = self.db_path.clone();
|
||||
let payload = tokio::task::spawn_blocking(move || -> rusqlite::Result<Option<String>> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let chat_key = chat_id.to_string();
|
||||
let payload = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
// Concurrent handler tasks (batch-forwards) may write chat_state
|
||||
// while this read runs; the shared busy timeout handles the
|
||||
// write-lock collision instead of failing the query.
|
||||
let mut stmt = conn.prepare("SELECT payload FROM chat_state WHERE chat_id = ?1")?;
|
||||
let mut rows = stmt.query(params![chat_id.to_string()])?;
|
||||
let mut rows = stmt.query(params![chat_key])?;
|
||||
match rows.next()? {
|
||||
Some(row) => Ok(Some(row.get(0)?)),
|
||||
Some(row) => Ok(Some(row.get::<_, String>(0)?)),
|
||||
None => Ok(None),
|
||||
}
|
||||
})
|
||||
.await
|
||||
.expect("chat_state worker panicked")
|
||||
.unwrap_or_else(|e| {
|
||||
log::error!("chat_state read failed: {e}");
|
||||
None
|
||||
@@ -97,18 +97,18 @@ impl ChatStore {
|
||||
pub async fn set(&self, chat_id: i64, data: &ChatData) {
|
||||
self.cache.lock().insert(chat_id, data.clone());
|
||||
let payload = serde_json::to_string(data).expect("chat state serializes");
|
||||
let db_path = self.db_path.clone();
|
||||
tokio::task::spawn_blocking(move || -> rusqlite::Result<()> {
|
||||
let conn = Connection::open(&db_path)?;
|
||||
let chat_id = chat_id.to_string();
|
||||
let result = crate::db::with_conn(&self.db_path, move |conn| {
|
||||
conn.execute(
|
||||
"INSERT OR REPLACE INTO chat_state (chat_id, payload) VALUES (?1, ?2)",
|
||||
params![chat_id.to_string(), payload],
|
||||
params![chat_id, payload],
|
||||
)?;
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
.expect("chat_state worker panicked")
|
||||
.unwrap_or_else(|e| log::error!("chat_state write failed: {e}"));
|
||||
.await;
|
||||
if let Err(e) = result {
|
||||
log::error!("chat_state write failed: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes edit-before-forward records whose `created_at + ttl` is in the
|
||||
@@ -144,7 +144,10 @@ impl ChatStore {
|
||||
self.set(chat_id, &data).await;
|
||||
}
|
||||
if !removed.is_empty() {
|
||||
log::info!("pruned {} expired edit-before-forward record(s)", removed.len());
|
||||
log::info!(
|
||||
"pruned {} expired edit-before-forward record(s)",
|
||||
removed.len()
|
||||
);
|
||||
}
|
||||
removed
|
||||
}
|
||||
|
||||
+21
-16
@@ -5,30 +5,27 @@ services:
|
||||
ports:
|
||||
- '80:80'
|
||||
- '443:443'
|
||||
environment:
|
||||
# Bare-IP access only.
|
||||
# DEFAULT_HOST: 'bot.example.com'
|
||||
volumes:
|
||||
- /var/run/docker.sock:/tmp/docker.sock:ro
|
||||
- ./nginx-certs:/etc/nginx/certs:ro
|
||||
- ./nginx-vhost.d:/etc/nginx/vhost.d:ro
|
||||
- ./nginx-html:/usr/share/nginx/html:ro
|
||||
- certs:/etc/nginx/certs:ro
|
||||
- html:/usr/share/nginx/html:ro
|
||||
networks: [proxy]
|
||||
labels:
|
||||
- 'com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy=true'
|
||||
- 'com.github.nginx-proxy.nginx'
|
||||
container_name: nginx-proxy
|
||||
|
||||
acme-companion:
|
||||
image: nginxproxy/acme-companion
|
||||
restart: always
|
||||
environment:
|
||||
DEFAULT_EMAIL: 'admin@yoursfunny.top'
|
||||
DEFAULT_EMAIL: ''
|
||||
volumes:
|
||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||
- ./nginx-certs:/etc/nginx/certs:rw
|
||||
- ./nginx-vhost.d:/etc/nginx/vhost.d:rw
|
||||
- ./nginx-html:/usr/share/nginx/html:rw
|
||||
- ./nginx-acme:/etc/acme.sh
|
||||
- certs:/etc/nginx/certs:rw
|
||||
- html:/usr/share/nginx/html:rw
|
||||
- acme:/etc/acme.sh
|
||||
networks: [proxy]
|
||||
container_name: acme-companion
|
||||
depends_on:
|
||||
- nginx-proxy
|
||||
|
||||
@@ -40,22 +37,30 @@ services:
|
||||
TELOXIDE_TOKEN: ''
|
||||
BOT_ADMIN: ''
|
||||
PIXIV_REFRESH_TOKEN: ''
|
||||
TWITTER_AUTH_TOKEN: ''
|
||||
EDIT_MESSAGE_TTL_SECONDS: '86400'
|
||||
LINK_CACHE_TTL_SECONDS: '604800'
|
||||
RUST_LOG: 'info'
|
||||
VIRTUAL_HOST: 'bot.example.com'
|
||||
VIRTUAL_HOST: '<YOUR_DOMAIN>'
|
||||
VIRTUAL_PORT: '8443'
|
||||
# LETSENCRYPT_HOST: 'bot.example.com'
|
||||
# ACME_HOST: 'your.domain.com'
|
||||
WEBHOOK: 'true'
|
||||
WEBHOOK_LISTEN: '0.0.0.0'
|
||||
WEBHOOK_PORT: '8443'
|
||||
WEBHOOK_URL: 'https://bot.example.com/'
|
||||
# WEBHOOK_CERT: './cert/cert.pem'
|
||||
WEBHOOK_URL: 'https://<YOUR_DOMAIN>/'
|
||||
WEBHOOK_SECRET_TOKEN: ''
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
networks: [proxy]
|
||||
depends_on:
|
||||
- nginx-proxy
|
||||
container_name: tgxmb
|
||||
|
||||
volumes:
|
||||
certs:
|
||||
html:
|
||||
acme:
|
||||
|
||||
networks:
|
||||
proxy:
|
||||
name: proxy
|
||||
|
||||
+15
-3
@@ -5,9 +5,21 @@ if [ "$(id -u)" -eq '0' ]
|
||||
then
|
||||
USER_ID=${LOCAL_USER_ID:-9001}
|
||||
|
||||
useradd --shell /bin/bash -u ${USER_ID} -o -c "" -m user > /dev/null 2>&1
|
||||
usermod -a -G root user > /dev/null 2>&1
|
||||
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1
|
||||
# `docker compose restart` / `docker restart` reuse the same container, so
|
||||
# the overlay fs keeps the user created on first boot. A second `useradd`
|
||||
# then fails with exit code 9, which would trip `set -e` and kill the
|
||||
# container on every restart. Create only if missing; align the UID
|
||||
# otherwise so LOCAL_USER_ID changes still apply.
|
||||
if ! id user > /dev/null 2>&1
|
||||
then
|
||||
useradd --shell /bin/bash -u ${USER_ID} -o -c "" -m user > /dev/null 2>&1 || true
|
||||
else
|
||||
usermod -u ${USER_ID} -o user > /dev/null 2>&1 || true
|
||||
fi
|
||||
usermod -a -G root user > /dev/null 2>&1 || true
|
||||
# Bind-mounted volumes may not support chown; a failure here must not kill
|
||||
# the container either.
|
||||
chown -R `id -u user`:`id -u user` /app > /dev/null 2>&1 || true
|
||||
|
||||
export HOME=/home/user
|
||||
# setpriv (util-linux, present in bookworm-slim) replaces gosu: drop to the
|
||||
|
||||
Reference in New Issue
Block a user