TelegramTwitterMediaBot/AGENTS.md

Repository Guidelines

Project Overview

Telegram bot (teloxide) that turns post links from X/Twitter, Pixiv, Bluesky, and Misskey (misskey.io) into media messages (images, video, GIF) with the post's title, author, and tags. It supports batch media splitting, retry with persistence, inline queries, forward-channel rebinding with caption templates, and Pixiv ugoira→MP4 transcoding. README is in Chinese; user-facing bot strings are in English. The project is a Rust port of a Python predecessor (see queue.rs comments referencing utils/task_queue.py).

Two-crate Cargo workspace (both v1.5.0, edition 2024, resolver 3):

• crates/x-media — library that fetches and normalizes media from the four sites. Pure, no Telegram knowledge.
• crates/xmedia-bot — the bot binary: teloxide dispatcher, SQLite-backed chat state, persistent task queue.

Architecture & Data Flow

Telegram update → Dispatcher (polling or axum webhook) → dptree branches
  ├─ message    → commands (any chat) / URL links (private chat only)
  ├─ inline_query → InlineQueryResult Photo/Video/Mpeg4Gif
  └─ callback_query → "forward" (copy to channel) / "template|<name>" (apply caption template)

Message flow: message_handler extracts URLs (from url/text_link entities, text + caption, deduped) → x_media::site::fetch(url) → Fetched → builds a Task → send::send_media_sequence (media groups ≤ 9, caption on first item) or send::send_animation. On Telegram URL-fetch failure or size error (send_batch_via_upload): download via x_media::site::download_media to a temp file (≤ 10 MiB), sniff magic bytes (sniff_ext), upload via multipart; oversized items fall back to fallback_url. On failure: enqueue_retry persists resume-state Task into the SQLite queue → workers lease (120 s lock TTL) → retry with exponential backoff (≤ 30 s, MAX_RETRIES = 2) → dead-letter → notify_failure. Success → post_send_actions: edit-before-forward prompt with inline buttons, or copy_messages to the bound forward channel.

Debug command: /test <url> runs the same x_media::site::fetch and replies with test_parse_report (handlers/commands.rs) — site id, normalized cache key, source URL, title/author/tags, sensitive flag, caption and the media list — nothing is sent, cached or forwarded; the report is capped at 4000 chars and sent with HTML parse mode: raw fields are escaped, and the caption is wrapped in a <blockquote> so it renders exactly like the sent media caption (escaped text and links included). It uses a custom parse_test_arg parser (whole remainder, trimmed) because teloxide's built-in split parser takes exactly one space-separated token.

The x-media library: site::fetch(url) dispatches through the SITES registry (per-site impl Site, in order twitter → bsky → misskey → pixiv) and returns Ok(None) for unmatched URLs. Fetched { source_url, caption, title, media: Vec<Media>, sensitive, site_id, … }; caption_with(format) substitutes {url} {author} {author_url} {title} {tags}.

Key Directories

Path	Purpose
crates/x-media/src/	Fetch library. site/mod.rs = dispatcher + Fetched/FetchError/download_media/media_size; media.rs = Media enum; examples/fetch.rs = end-to-end usage sample
crates/x-media/src/site/<twitter|pixiv|bsky|misskey>/	One directory per site: mod.rs (re-exports), interface.rs (PATTERN, enabled(), fetch_from_url(), cache_key/is_retryable/media_headers, unit struct <Name>Site implementing site::Site, From<SiteStruct> for Fetched), model.rs (serde DTOs). Pixiv adds api.rs (auth + transport); twitter adds auth.rs (logged-in GraphQL TweetDetail fallback for NSFW tweets, gated on TWITTER_AUTH_TOKEN). Misskey targets misskey.io only (POST /api/notes/show, 400+NO_SUCH_NOTE → NotFound). Twitter's from_syndication_json HTML-decodes the API text — syndication and GraphQL full_text both arrive pre-escaped (&gt; &lt; &amp; &#39;) — so the stored text is raw and the caption escapes exactly once
crates/xmedia-bot/src/main.rs	Entry point: env/log init, command registration (register_commands), shared send::BOT force-init, queue worker start, site login validation (site::validate_all), 300 s edit-expiry sweep, dptree handler tree, webhook vs polling dispatch
crates/xmedia-bot/src/config.rs	Manual env parsing into Config
crates/xmedia-bot/src/db.rs	DbPool: one shared SQLite connection pool (POOL_SIZE = 4, WAL, busy_timeout) for all three tables over $DATA_DIR/task_queue.db (default data/) — the three stores share it; open_store creates file + schema, with_conn runs all rusqlite I/O in spawn_blocking
crates/xmedia-bot/src/handlers/	Handler modules: mod.rs (message entry point, reply, log_key), commands.rs (teloxide BotCommands enum + command executor, incl. the /test <url> parse-only debug command and the admin-only /bot_dict state dump), urls.rs (URL extraction + bounded job channel (256) drained by URL_WORKERS = 8 workers (start_url_workers) — backpressure instead of unbounded spawns; teloxide's per-chat workers are sequential — batch-forwards need concurrency), inline.rs/callback.rs (inline queries / edit-before-forward buttons), statics.rs (global statics)
crates/xmedia-bot/src/state.rs	ChatStore: parking_lot Mutex<HashMap> cache + SQLite write-through (chat_state table)
crates/xmedia-bot/src/link_cache.rs	LinkCache: SQLite-backed cache (link_cache table) of successfully sent posts — raw caption fields + Telegram file_ids; repeat links re-send locally (no fetch/upload), TTL + prune, invalidated on permanent send failure
crates/xmedia-bot/src/queue.rs	PersistentTaskQueue: SQLite-backed queue (tasks table), QUEUE_WORKERS = 4 concurrent workers (lease via BEGIN IMMEDIATE + locked_until TTL), retry→dead-letter, notify_one worker wakeup plus a separate Notify for the 30 s lease-expiry sweep (a shared one let the sweep steal the workers' wakeup permit), busy_timeout on all connections
crates/xmedia-bot/src/ctx.rs	AppContext: the injected collaborators (sender + ChatStore/PersistentTaskQueue/LinkCache/Config), from_statics for production and the CONTEXT static the worker closures hold. test_support::TestStores backs handler tests with a tempdir store set
crates/xmedia-bot/src/send/	send/mod.rs: Task/MediaItemPayload payloads, SendError/Classification, send_media_sequence/send_animation/forward_messages; send/input_media.rs: payload → InputFile/InputMedia + build_media_group (caption on the first item only); send/upload.rs: the download-and-reupload fallback (prepare_upload_item/send_batch_via_upload, photo downscale handoff); send/post_send.rs: link-cache write, KEEP_ALIVE registry, settle_task, post_send_actions, handle_task/dead_letter_notify
crates/xmedia-bot/src/media_sender.rs	MediaSender trait: the user-flow surface (send_media_group/send_animation/copy_messages/send_message/answer_callback_query/edit_message_caption/delete_message/send_chat_action) implemented by teloxide Bot (per-chat rate-limited) and by a recording MockSender in tests. Admin/setup APIs (get_chat, set_my_commands, …) stay on the concrete Bot
crates/xmedia-bot/src/rate_limit.rs	Per-chat token bucket (CAPACITY = 20, ~20 msg/min refill) paced before sends reach the API so batch forwards don't trip flood control

Development Commands

export TELOXIDE_TOKEN=<token>          # required; PIXIV_REFRESH_TOKEN optional (Pixiv disabled without it)
cargo run -p xmedia-bot                # run the bot (polling by default)
cargo run -p x-media --example fetch -- <url>   # test a link through the fetch library
cargo test --workspace                 # full test suite (no CI test step exists — run locally)
cargo build --release -p xmedia-bot    # release build (Dockerfile does this)
cargo clippy --workspace --all-targets # lint (Clippy is the configured IDE linter)
cargo fmt --check                      # formatting

Docker: docker build -t tgxmb . then docker run --rm -d --name tgxmb --env-file .env -v ./data:/app/data tgxmb. Runtime requires ffmpeg (built into the image). The builder fetches crates.io + ffmpeg; on restricted networks pass proxy build args, e.g. --build-arg HTTP_PROXY=http://host.docker.internal:10808 --build-arg HTTPS_PROXY=… (Docker Desktop builds can't reach the host loopback — use host.docker.internal).

Code Conventions & Common Patterns

• Errors via thiserror derive (no anyhow): the public, stringified errors — FetchError (Http/Json/Pixiv/Site/NotFound/Blocked) and PixivError — derive thiserror::Error with #[from] conversions; Display/source() come from the derive. The internal control-flow enums — QueueError (Retryable { delay_seconds, payload } / Permanent), SendError (Retryable/Permanent), Classification, FallbackError — carry no Display and are handled by direct variant matching. New errors should follow the same split: stringified/public errors derive thiserror, internal flow enums stay plain.
• Global state via std::sync::LazyLock statics, not DI: CONFIG, CHAT_STORE, TASK_QUEUE in handlers/statics.rs; shared reqwest CLIENT in x-media/src/site/mod.rs. Bot is passed/cloned into handlers; queue workers share the process-wide send::BOT (LazyLock<Bot>, force-initialized in main so a missing token fails at startup).
• Async: tokio multi-thread runtime (#[tokio::main] default). All rusqlite I/O inside tokio::task::spawn_blocking. Long loops use tokio::select! with tokio::sync::{watch, Notify} stop/wake channels. No streams.
• Blocking sync primitives: parking_lot::Mutex for hot caches, tokio::sync::Mutex for async-shared state (pixiv token cache), AtomicBool for feature gates.
• Site adapter convention: each site module exports PATTERN: LazyLock<Regex>, enabled() -> bool, fetch_from_url(url) -> Result<Fetched, FetchError>, plus cache_key/is_retryable/media_headers, and a unit struct <Name>Site implementing site::Site; the central dispatcher (site/mod.rs) only iterates the SITES registry. Adding a site = new site/<name>/{mod.rs,interface.rs,model.rs} + one Box::new(...) entry in SITES — the bot crate never lists sites (SetFormat whitelist, cache-key site lookup and startup validation all derive from the registry). Async trait methods return SiteFuture (a boxed Pin<Box<dyn Future + Send>>) because async fn in traits is not dyn-compatible.
• Serde: per-site model.rs are pure Deserialize DTOs mirroring API JSON; site structs in interface.rs have private fields, a caption() builder, and impl From<SiteStruct> for Fetched. Persisted payloads use internally-tagged enums (#[serde(tag = "kind")] / type).
• Naming: module-per-concern, snake_case files, CamelCase types, snake_case fns. //! module docs and /// docs on non-obvious logic (syndication token, ugoira encoding, display_text_range).
• Retries: only x-media::site::fetch retries (3 attempts, 1 << attempt backoff, HTTP errors only); site::fetch_once is the same code path with a single attempt, used by inline queries whose answer window is shorter than the backoff. Queue retries are explicit QueueError::Retryable with computed delay (retry_delay_seconds).
• Logging via log macros (pretty_env_logger, level from RUST_LOG). Level convention: info = lifecycle + per-post business results (sent/forwarded/copied), admin/operator actions and anomalies (fallback, retry enqueue, dead-letter is error); debug = per-request detail (message/command/URL extraction, fetching/fetched, batch sends, queue processing, photo processing, inline queries). Full user-submitted URLs and message text only appear at debug; at info and above links are printed via the normalized cache key (handlers::log_key, e.g. [key=twitter:123...]) so logs stay short and do not echo user data.

Important Files

File	Why it matters
crates/xmedia-bot/src/main.rs	Startup sequence, webhook vs polling, graceful shutdown (SIGINT via teloxide ctrlc / SIGTERM via stop_token for docker, → sweep stop → admin msg → queue stop)
crates/xmedia-bot/src/handlers/	statics.rs = CHAT_STORE/TASK_QUEUE/CONFIG singletons (open $DATA_DIR/task_queue.db, default data/ relative to CWD, dir auto-created); commands.rs = command dispatch (incl. the /test <url> parse-only debug command and the admin-only /bot_dict state dump); urls.rs = URL extraction + the per-URL pipeline (enqueue_retry lives in send/post_send.rs); inline.rs = debounced inline queries; callback.rs = edit-before-forward buttons (dptree entry + testable handle_callback core)
crates/xmedia-bot/src/send/	mod.rs: constants MAX_MEDIA_GROUP = 9; classify_request_error; the senders. upload.rs: download-and-reupload fallback triggered only by Telegram API errors (is_media_fetch_failure / is_size_error). post_send.rs: settlement (settle_task), cache write, post-send actions, queue handlers. input_media.rs: payload → InputMedia
crates/xmedia-bot/src/photo.rs	Pure-Rust photo processing (no ffmpeg): png (image-png) decode/encode + zune-jpeg decode + fast_image_resize Lanczos3 downscale + jpeg-encoder. Photos over Telegram's limits (width + height > 10000 px → PHOTO_INVALID_DIMENSIONS; bytes > 10 MiB) are decoded, downscaled keeping the format, PNG bit depth > 24 (RGBA 32-bit / 16-bit per channel) reduced to 24-bit RGB with alpha flattened white (≤24-bit untouched, never upconverted), and transcoded to JPEG only if still over the cap; memory budget guarded, otherwise the item's smaller fallback URL
crates/x-media/src/site/mod.rs	Dispatcher, Fetched/FetchError, shared CLIENT, download_media (adds Referer: https://www.pixiv.net/ for pximg.net hotlink protection)
crates/x-media/src/site/pixiv/api.rs	OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in spawn_blocking
Dockerfile	Multi-stage: cached dep layer via stub sources + touch *.rs mtime bump (cargo's freshness is mtime-based and cargo clean -p removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (FFMPEG_URL arg, optional FFMPEG_SHA256 checksum, unzip -t integrity check), debian:bookworm-slim runtime, entrypoint. Runtime ships no libssl/libcrypto/CA bundle — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest)
docker-entrypoint.sh	Privilege drop: useradd with LOCAL_USER_ID (default 9001) + setpriv (no gosu on bookworm-slim)
docker-compose.yml.example	Deployment env reference (real docker-compose.yml is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; WEBHOOK_CERT only feeds set_webhook), bot exposes VIRTUAL_HOST/VIRTUAL_PORT on the shared proxy network, no host port; container names nginx-proxy/acme-companion/tgxmb, start order via depends_on (proxy → acme → bot)
.github/workflows/docker.yml	CI: build+push to Docker Hub on tag v*/master, plus a build-only check on PRs touching the build inputs; no test step; verifies a release tag matches both crate versions; buildx gha cache (cache-from always, cache-to except on PRs, scope tgxmb-build, mode=max) so cargo deps + ffmpeg layers are restored across runs; FFMPEG_URL/FFMPEG_SHA256 come from repo variables when set
README.md	Feature docs + command table (Chinese)

Runtime/Tooling Preferences

• Rust, stable, edition 2024, workspace resolver 3. No rust-version/MSRV pin, no rust-toolchain.toml — recent stable is assumed. No nightly features.
• Package manager: Cargo (workspace with path dep x-media ← xmedia-bot). No [workspace.package]/shared deps — each crate lists deps independently.
• TLS is rustls end-to-end (no native-tls/openssl in the tree, no libssl in the Docker runtime image): teloxide is declared default-features = false with ["webhooks-axum", "macros", "rustls", "ctrlc_handler"] (the removed default also carried native-tls and ctrlc_handler — the latter must stay); x-media's reqwest is default-features = false with ["json", "rustls-tls"] (webpki-roots baked in, so the image ships no CA bundle). One reqwest 0.12.28 in the lock.
• Versioning: bump the version in all three places (crates/x-media/Cargo.toml, crates/xmedia-bot/Cargo.toml, Cargo.lock) and keep README.md, README.en.md and AGENTS.md in sync with the code on every bump, then commit (chore: bump version to X.Y.Z), create an annotated tag vX.Y.Z, and push branch + tag (the tag push triggers the Docker Hub build). The tag must equal both crate versions: .github/workflows/docker.yml verifies that before building, and --locked verifies the lock file.
• Config is environment-variable driven (dotenv loads .env, gitignored; no .env.example exists). Key vars: TELOXIDE_TOKEN (required), PIXIV_REFRESH_TOKEN, TWITTER_AUTH_TOKEN (optional; x.com auth_token cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), BOT_ADMIN (comma-separated ids), EDIT_MESSAGE_TTL_SECONDS (default 86400), LINK_CACHE_TTL_SECONDS (default 604800), DATA_DIR (default data, CWD-relative; the SQLite dir, auto-created), WEBHOOK/WEBHOOK_URL/WEBHOOK_LISTEN/WEBHOOK_PORT/WEBHOOK_CERT/WEBHOOK_SECRET_TOKEN (webhook mode requires URL/listen/port, .expected; WEBHOOK_CERT is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), RUST_LOG, TELOXIDE_PROXY, LOCAL_USER_ID (entrypoint only).
• SQLite via rusqlite with bundled feature (no system libsqlite needed). DB file $DATA_DIR/task_queue.db (default data/task_queue.db, CWD-relative — run from the workspace root, or /app in Docker; set DATA_DIR to pin state anywhere). Mount ./data and ./cert volumes.
• .gitattributes enforces LF for *.sh (CRLF breaks shebangs in containers). .gitignore: .env, data/, cert/, docker-compose.yml, /target, .idea/.
• Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs.

Testing & QA

• ~135 tests, all inline #[cfg(test)] mod tests — no tests/ integration directories. Framework: built-in Rust test + #[tokio::test] (dev-deps only in x-media: tokio macros/rt-multi-thread, dotenv).
• No mocking framework anywhere (no mockito/wiremock/mockall). Conventions: pure-function units (regex parsing, serde round-trips, chunking, retry math) tested synchronously; async tests use real dependencies — file-backed SQLite via tempfile (queue.rs::new_queue() helper), live network fetches.
• Live-network tests exist in site/twitter/interface.rs (5), site/bsky/interface.rs (2), site/misskey/interface.rs (1), site/pixiv/api.rs (1); photo.rs adds one #[ignore = "heavy: …"] test. site/mod.rs also has a token-gated but not #[ignore]d pixiv download test (download_media_pixiv_original_with_referer): it hits i.pximg.net whenever PIXIV_REFRESH_TOKEN is set, so a local cargo test --workspace is not fully offline and can flake on a pixiv CDN body timeout. Test gating convention (enforced by .github/workflows/ci.yml): pure unit tests always run; live-network tests carry #[ignore = "live network: ..."] (run via cargo test --workspace -- --ignored live); token-gated pixiv tests early-return when PIXIV_REFRESH_TOKEN is absent or empty (an unset GitHub secret arrives as "" — is_err() alone would run them tokenless and fail). Run the full offline suite with cargo test --workspace.
• Fixtures are inline serde_json::json! builder fns (fixture(), thread_json(), illust_json()), not files. The shared CLIENT sets pool_max_idle_per_host(0) under #[cfg(test)] to avoid cross-runtime DispatchGone.
• CI — .github/workflows/ci.yml (actions pinned to commit SHAs, --locked on every cargo invocation, concurrency cancels superseded runs, RUST_BACKTRACE=1) runs cargo fmt --check + cargo clippy --workspace --all-targets --locked -- -D warnings + cargo test --workspace --locked + a release-profile cargo build --release --locked + an actions-rust-lang/audit dependency-vulnerability gate (offline, no secrets, on every push/PR) and a live job (schedule/manual/tag only, -p x-media since every network/secret-gated test lives there, continue-on-error) for the #[ignore]d live + token tests. .github/workflows/docker.yml builds and pushes the image on master/tag and runs a build-only check on pull requests touching the build inputs (Dockerfile, entrypoint, manifests, .dockerignore); a release tag must match both crate versions or the build stops, and FFMPEG_URL/FFMPEG_SHA256 are taken from repository variables when set (a release can pin an exact ffmpeg build). .github/dependabot.yml keeps crates, the pinned actions and the Docker base images current.
• Untested and hard to test without a mock seam: main.rs, config.rs, db.rs, handlers/statics.rs, media_sender.rs (holds the MockSender itself); in x-media: media.rs, lib.rs, all model.rs. The commands.rs executor needs a real Bot (only its pure report builder is tested). Everything else — handlers/{mod,callback,inline,urls}.rs, send/*, ctx.rs, state.rs, queue.rs, link_cache.rs, rate_limit.rs — is driven through TestStores/ctx::test_support and the scripted MockSender.
• No coverage tracking.