diff --git a/.gitignore b/.gitignore index 1148345..933714f 100644 --- a/.gitignore +++ b/.gitignore @@ -6,8 +6,6 @@ nginx-certs/ nginx-vhost.d/ nginx-html/ nginx-acme/ -docker-compose.yml - .env diff --git a/AGENTS.md b/AGENTS.md index fdfa237..93153fd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,7 +88,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi | `crates/x-media/src/site/pixiv/api.rs` | OAuth token exchange (hardcoded app client id/secret), access-token cache, ugoira zip→MP4 via ffmpeg in `spawn_blocking` | | `Dockerfile` | Multi-stage: cached dep layer via stub sources + `touch *.rs` mtime bump (cargo's freshness is mtime-based and `cargo clean -p` removes 0 files — the touch is what forces the real sources to rebuild while deps stay cached), static ffmpeg from ffmpeg.martin-riedl.de (`FFMPEG_URL` arg, optional `FFMPEG_SHA256` checksum, `unzip -t` integrity check), `debian:bookworm-slim` runtime, entrypoint. Runtime ships **no libssl/libcrypto/CA bundle** — rustls webpki-roots handles all TLS, and the static ffmpeg only processes local files (downloads go through reqwest) | | `docker-entrypoint.sh` | Privilege drop: `useradd` with `LOCAL_USER_ID` (default 9001) + `setpriv` (no gosu on bookworm-slim) | -| `docker-compose.yml.example` | Deployment env reference (real `docker-compose.yml` is gitignored). Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) | +| `docker-compose.yml` | The deployment composition, committed as-is: every instance value (token, admins, site credentials, domain) is a `${VAR}` substitution read from the gitignored `.env` beside it, so the file needs no per-deployment edit — and a variable not listed in a service's `environment:` never reaches that container. Ships nginx-proxy + acme-companion: webhook mode needs TLS termination in front (teloxide's axum listener is HTTP-only; `WEBHOOK_CERT` only feeds `set_webhook`), bot exposes `VIRTUAL_HOST`/`VIRTUAL_PORT` on the shared `proxy` network, no host port; container names `nginx-proxy`/`acme-companion`/`tgxmb`, start order via `depends_on` (proxy → acme → bot) | | `.github/workflows/docker.yml` | CI: build+push to Docker Hub on tag `v*`/master, plus a build-only check on PRs touching the build inputs; **no test step**; verifies a release tag matches both crate versions; buildx gha cache (`cache-from` always, `cache-to` except on PRs, scope `tgxmb-build`, `mode=max`) so cargo deps + ffmpeg layers are restored across runs; `FFMPEG_URL`/`FFMPEG_SHA256` come from repo variables when set | | `README.md` | Feature docs + command table (Chinese) | @@ -100,7 +100,7 @@ Docker: `docker build -t tgxmb .` then `docker run --rm -d --name tgxmb --env-fi - **Versioning**: bump the version in all three places (`crates/x-media/Cargo.toml`, `crates/xmedia-bot/Cargo.toml`, `Cargo.lock`) and **keep `README.md`, `README.en.md` and `AGENTS.md` in sync with the code on every bump**, then commit (`chore: bump version to X.Y.Z`), create an annotated tag `vX.Y.Z`, and push branch + tag (the tag push triggers the Docker Hub build). The tag must equal both crate versions: `.github/workflows/docker.yml` verifies that before building, and `--locked` verifies the lock file. - Config is **environment-variable driven** (dotenv loads `.env`, gitignored; no `.env.example` exists). Key vars: `TELOXIDE_TOKEN` (required), `PIXIV_REFRESH_TOKEN`, `TWITTER_AUTH_TOKEN` (optional; x.com `auth_token` cookie — enables the logged-in GraphQL fallback that fetches NSFW tweets syndication withholds), `BILIBILI_COOKIE` (optional; whole bilibili cookie string — bilibili dynamics fetch anonymously and add their own device cookies, this only rescues an egress IP that bilibili has hard-flagged with `-352`/412), `BOT_ADMIN` (comma-separated ids), `EDIT_MESSAGE_TTL_SECONDS` (default 86400), `LINK_CACHE_TTL_SECONDS` (default 604800), `CAPTION_QUOTE_TEXT_CHARS` (default 200; a post whose text — the `title` plus `content` joined, see `site::compose_text` — reaches this length gets that text wrapped in an expandable blockquote inside its caption, the URL and author line staying outside; `0` disables it. Applied at the send boundary in `send::quote_long_caption`, which locates the text as what follows the author link, so a `/set_format` that moves `{title}`/`{content}` elsewhere and pixiv's title-inside-a-link layout opt out; `copy_messages` forwards and queued retries inherit the wrap, while the edit-before-forward rewrite stays unquoted by design), `DATA_DIR` (default `data`, CWD-relative; the SQLite dir, auto-created), `WEBHOOK`/`WEBHOOK_URL`/`WEBHOOK_LISTEN`/`WEBHOOK_PORT`/`WEBHOOK_CERT`/`WEBHOOK_SECRET_TOKEN` (webhook mode requires URL/listen/port, `.expect`ed; `WEBHOOK_CERT` is Telegram-facing self-signed validation only — TLS must be terminated by a reverse proxy), `RUST_LOG`, `TELOXIDE_PROXY`, `LOCAL_USER_ID` (entrypoint only). - SQLite via `rusqlite` with `bundled` feature (no system libsqlite needed). DB file `$DATA_DIR/task_queue.db` (default `data/task_queue.db`, CWD-relative — run from the workspace root, or `/app` in Docker; set `DATA_DIR` to pin state anywhere). Mount `./data` and `./cert` volumes. -- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `docker-compose.yml`, `/target`, `.idea/`. +- `.gitattributes` enforces LF for `*.sh` (CRLF breaks shebangs in containers). `.gitignore`: `.env`, `data/`, `cert/`, `nginx-*` (proxy state), `/target`, `.idea/` (the compose file is tracked; only `.env` carries the deployment's own values). - Docs are in Chinese (README, AGENTS.md); user-facing bot strings are in English. Keep that split when editing user-facing strings and docs. ## Testing & QA diff --git a/README.en.md b/README.en.md index 9e57864..33204d2 100644 --- a/README.en.md +++ b/README.en.md @@ -27,7 +27,7 @@ export PIXIV_REFRESH_TOKEN= cargo run -p xmedia-bot ``` -Docker deployment (see `docker-compose.yml.example`): +Docker deployment (`docker-compose.yml` in this repo is the orchestration; instance values live in the `.env` next to it, and compose substitutes every `${VAR}` from there): ```bash docker build -t tgxmb . @@ -42,11 +42,11 @@ Bilibili dynamics are fetched anonymously by default (no login; the bot fetches ### Webhook deployment (needs a reverse proxy) -`docker-compose.yml.example` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. Pick one deployment shape: +`docker-compose.yml` ships an [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) reverse-proxy orchestration. The committed file needs **no editing**: domain, tokens and admins are instance values and live in the `.env` beside it (compose reads and substitutes `${VAR}` at startup). Pick one deployment shape: **With a domain** 1. Point a DNS A record at the server -2. In compose set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain, and uncomment `ACME_HOST` (set it to the domain) +2. In `.env` set `VIRTUAL_HOST` and `WEBHOOK_URL` to the domain; to have acme-companion issue the certificate, also uncomment the `ACME_HOST` line in `docker-compose.yml` and set `ACME_HOST` in `.env` 3. acme-companion issues and renews certificates automatically — nothing manual **IP only** @@ -74,7 +74,7 @@ Let's Encrypt can issue certificates for public IPs (available since 2026, valid --key-file /acme.sh/.key \ --reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP" ``` -3. In compose set `VIRTUAL_HOST: ''` and `WEBHOOK_URL: 'https:///'`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate. +3. In `.env` set `VIRTUAL_HOST=` and `WEBHOOK_URL=https:///`; no `WEBHOOK_CERT` needed. Renewal is handled by the acme.sh daemon (`--days 3` = renew every 3 days, buffer against the 7-day validity), and a successful renewal HUP-notifies nginx-proxy to load the new certificate. Limitations: certificate validity ~7 days; only http-01/tls-alpn-01 validation (port 80 must be publicly reachable); no DNS-01, private IPs or IP ranges; at most 5 certificates per 168 hours for the same IP set. It is recommended to trial-issue with `--server letsencrypt_test` first, then switch to the production server. @@ -94,9 +94,9 @@ Telegram only accepts ports 443/80/88/8443. | `CAPTION_QUOTE_TEXT_CHARS` | **The text part** of the caption (the joined `{title}` + `{content}`) is wrapped in a collapsible blockquote once it reaches this many characters, default 200; `0` disables | | `DATA_DIR` | Data directory (where the SQLite `task_queue.db` lives), default `data` (relative to the working directory, created automatically) | | `RUST_LOG` | Log level, default `info,hyper_util=warn,reqwest=warn` (an unset variable no longer silences the log). Recipes: `info,xmedia_bot=debug,x_media=debug` (app detail, no dependency noise) / `debug,hyper_util=off` (everything) / `trace` (also prints full links and message text — **user data**) | -| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW) | +| `TELOXIDE_PROXY` | HTTP proxy (e.g. `http://127.0.0.1:10808`); applies to both the Telegram Bot API and site fetches — required on restricted networks (e.g. behind the GFW). **Never leave it blank** (`TELOXIDE_PROXY=`) — teloxide panics on a value it cannot parse; omit the line when unused. `docker-compose.yml` deliberately does not pass it to the container (a `127.0.0.1` proxy there is the container itself): add the line and use `host.docker.internal:` when a deployment needs one | | `LOCAL_USER_ID` | UID the container runs as, default 9001 | -| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this | +| `VIRTUAL_HOST` | Public domain or IP; nginx-proxy routes by this (set it in `.env`, which compose reads) | | `VIRTUAL_PORT` | Port the bot listens on inside the container; nginx-proxy's forwarding target | | `ACME_HOST` | Domain deployment: when set to the domain, acme-companion issues/renews certificates automatically | | `DEFAULT_HOST` | nginx-proxy routes requests with unknown Host headers to this vhost (needed for IP access) | diff --git a/README.md b/README.md index eb5af36..4b5b436 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ export PIXIV_REFRESH_TOKEN= cargo run -p xmedia-bot ``` -Docker 部署（参考 `docker-compose.yml.example`）： +Docker 部署（编排见仓库里的 `docker-compose.yml`，实例相关的值写在同目录的 `.env`，compose 会自动替换其中的 `${VAR}`）： ```bash docker build -t tgxmb . @@ -42,11 +42,11 @@ Bilibili 动态默认匿名抓取（无需登录，bot 会自动从 B 站的匿 ### Webhook 部署（需要反向代理） -`docker-compose.yml.example` 内置了 [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) 反向代理编排，按部署环境二选一： +`docker-compose.yml` 内置了 [nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) + [acme-companion](https://github.com/nginx-proxy/acme-companion) 反向代理编排，仓库里的这份文件**不需要改动**：域名、令牌、管理员等实例相关的值都写在同目录的 `.env` 里（compose 启动时自动读取并替换 `${VAR}`）。按部署环境二选一： **有域名** 1. DNS A 记录指向服务器 -2. compose 里设 `VIRTUAL_HOST`、`WEBHOOK_URL` 为域名，并取消注释 `ACME_HOST`（设为域名） +2. `.env` 里设 `VIRTUAL_HOST`、`WEBHOOK_URL` 为域名；要由 acme-companion 自动签发证书时，再取消 `docker-compose.yml` 里 `ACME_HOST` 那行的注释，并在 `.env` 里把 `ACME_HOST` 设为域名 3. acme-companion 自动签发与续期证书，无需手动处理 **只有 IP** @@ -74,7 +74,7 @@ Let's Encrypt 支持为公网 IP 签发证书（2026 年起可用，有效期约 --key-file /acme.sh/.key \ --reloadcmd "curl --unix-socket /var/run/docker.sock -X POST http://localhost/containers/nginx-proxy/kill?signal=HUP" ``` -3. compose 里设 `VIRTUAL_HOST: ''`、`WEBHOOK_URL: 'https:///'`，无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成（`--days 3` = 每 3 天续一次，证书 7 天有效有缓冲），续期成功后自动 HUP 通知 nginx-proxy 加载新证书。 +3. `.env` 里设 `VIRTUAL_HOST=`、`WEBHOOK_URL=https:///`，无需 `WEBHOOK_CERT`。续期由 acme.sh daemon 自动完成（`--days 3` = 每 3 天续一次，证书 7 天有效有缓冲），续期成功后自动 HUP 通知 nginx-proxy 加载新证书。 限制：证书约 7 天有效；验证仅支持 http-01/tls-alpn-01（80 端口必须公网可达）；不支持 DNS-01、私有 IP 与 IP 段；同一 IP 集合每 168 小时限签发 5 张。建议先用 `--server letsencrypt_test` 试签，成功后再切正式服务器。 @@ -94,9 +94,9 @@ Telegram 只接受 443/80/88/8443 端口。 | `CAPTION_QUOTE_TEXT_CHARS` | 正文（`{title}` + `{content}` 合计）达到该长度（字符）时，caption 的**正文部分**用可折叠引用块包裹，默认 200；`0` 关闭 | | `DATA_DIR` | 数据目录（SQLite 数据库 `task_queue.db` 所在目录），默认 `data`（相对工作目录，会自动创建） | | `RUST_LOG` | 日志级别，默认 `info,hyper_util=warn,reqwest=warn`（未设置也**不会**哑掉）。排障配方：`info,xmedia_bot=debug,x_media=debug`（应用细节，无依赖噪音）/ `debug,hyper_util=off`（全量）/ `trace`（额外打印完整链接与消息原文，**含用户数据**） | -| `TELOXIDE_PROXY` | HTTP 代理（如 `http://127.0.0.1:10808`）；同时作用于 Telegram Bot API 与站点抓取请求，网络受限环境（如 GFW）必需 | +| `TELOXIDE_PROXY` | HTTP 代理（如 `http://127.0.0.1:10808`）；同时作用于 Telegram Bot API 与站点抓取请求，网络受限环境（如 GFW）必需。**不要留空值**（`TELOXIDE_PROXY=`）——teloxide 对无法解析的值会直接 panic；不用代理就别写这一行。容器里要用代理时，`docker-compose.yml` 的 `environment` 里默认没有它（容器内的 `127.0.0.1` 是容器自己），需要时手动加上并把地址换成 `host.docker.internal:` | | `LOCAL_USER_ID` | 容器内运行用户 UID，默认 9001 | -| `VIRTUAL_HOST` | 对外域名或 IP，nginx-proxy 按此路由 | +| `VIRTUAL_HOST` | 对外域名或 IP，nginx-proxy 按此路由（写在 `.env`，compose 读取） | | `VIRTUAL_PORT` | bot 容器内监听端口，nginx-proxy 的转发目标 | | `ACME_HOST` | 域名部署：设为域名时由 acme-companion 自动签发/续期证书 | | `DEFAULT_HOST` | nginx-proxy 将未知 Host 的请求路由到该 vhost（IP 访问时需要） | diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..8832f74 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,108 @@ +# Deployment reference for the Docker Hub image. Instance values (token, admins, +# site credentials, domain) live in `.env` next to this file — `docker compose` +# substitutes every `${VAR}` from it automatically — so this file stays in the +# repository unmodified. A variable that is not listed here is not passed into +# the container at all. +# +# JSON-file logs grow without limit by default: a long-running bot (and the +# proxy in front of it) will fill the disk. One cap, applied to every service +# below via the anchor. +x-logging: &default-logging + driver: json-file + options: + max-size: '10m' + max-file: '3' + +services: + nginx-proxy: + image: nginxproxy/nginx-proxy:1.11.6-alpine + restart: always + environment: + # Routes requests with an unknown Host (i.e. plain IP access) here; set + # DEFAULT_HOST in .env to use it. + DEFAULT_HOST: '${DEFAULT_HOST:-}' + ports: + - '80:80' + - '443:443' + volumes: + - /var/run/docker.sock:/tmp/docker.sock:ro + - certs:/etc/nginx/certs:ro + - html:/usr/share/nginx/html:ro + networks: [proxy] + labels: + - 'com.github.nginx-proxy.nginx' + container_name: nginx-proxy + logging: *default-logging + + acme-companion: + image: nginxproxy/acme-companion + restart: always + environment: + DEFAULT_EMAIL: '${DEFAULT_EMAIL:-}' + volumes: + - /var/run/docker.sock:/var/run/docker.sock:ro + - certs:/etc/nginx/certs:rw + - html:/usr/share/nginx/html:rw + - acme:/etc/acme.sh + networks: [proxy] + container_name: acme-companion + depends_on: + - nginx-proxy + logging: *default-logging + + tgxmb: + image: yoursfunny/telegram-twitter-media-bot:latest + restart: always + environment: + # From .env (the instance's own values; see the env table in README.md). + TELOXIDE_TOKEN: '${TELOXIDE_TOKEN:-}' + BOT_ADMIN: '${BOT_ADMIN:-}' + PIXIV_REFRESH_TOKEN: '${PIXIV_REFRESH_TOKEN:-}' + TWITTER_AUTH_TOKEN: '${TWITTER_AUTH_TOKEN:-}' + BILIBILI_COOKIE: '${BILIBILI_COOKIE:-}' + VIRTUAL_HOST: '${VIRTUAL_HOST:-}' + WEBHOOK_URL: '${WEBHOOK_URL:-}' + WEBHOOK_SECRET_TOKEN: '${WEBHOOK_SECRET_TOKEN:-}' + # Defaults, listed so they are discoverable; override in .env when needed. + LOCAL_USER_ID: '${LOCAL_USER_ID:-1000}' + RUST_LOG: '${RUST_LOG:-info}' + EDIT_MESSAGE_TTL_SECONDS: '${EDIT_MESSAGE_TTL_SECONDS:-86400}' + LINK_CACHE_TTL_SECONDS: '${LINK_CACHE_TTL_SECONDS:-604800}' + CAPTION_QUOTE_TEXT_CHARS: '${CAPTION_QUOTE_TEXT_CHARS:-200}' + VIRTUAL_PORT: '${VIRTUAL_PORT:-8443}' + WEBHOOK: '${WEBHOOK:-true}' + WEBHOOK_LISTEN: '${WEBHOOK_LISTEN:-0.0.0.0}' + WEBHOOK_PORT: '${WEBHOOK_PORT:-8443}' + # For a certificate on a bare IP: uncomment and set ACME_HOST in .env. + # ACME_HOST: '${ACME_HOST:-}' + # + # Not listed on purpose: TELOXIDE_PROXY. Docker Desktop reaches a host + # proxy through host.docker.internal (a loopback address inside the + # container is the container itself), and teloxide panics on an *empty* + # value, so add the line deliberately when this deployment needs one: + # TELOXIDE_PROXY: '${TELOXIDE_PROXY}' + volumes: + - ./data:/app/data + networks: [proxy] + depends_on: + - nginx-proxy + container_name: tgxmb + logging: *default-logging + # Webhook mode only (in polling mode there is no listener, so drop this + # block or set WEBHOOK=true): the bot listens on WEBHOOK_PORT; nginx-proxy + # shows 502s while this is down, so surface it to the orchestrator. + healthcheck: + test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/${WEBHOOK_PORT:-8443}'"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + +volumes: + certs: + html: + acme: + +networks: + proxy: + name: proxy diff --git a/docker-compose.yml.example b/docker-compose.yml.example deleted file mode 100644 index 58ef2e2..0000000 --- a/docker-compose.yml.example +++ /dev/null @@ -1,86 +0,0 @@ -# JSON-file logs grow without limit by default: a long-running bot (and the -# proxy in front of it) will fill the disk. One cap, applied to every service -# below via the anchor. -x-logging: &default-logging - driver: json-file - options: - max-size: '10m' - max-file: '3' - -services: - nginx-proxy: - image: nginxproxy/nginx-proxy:1.11.6-alpine - restart: always - ports: - - '80:80' - - '443:443' - volumes: - - /var/run/docker.sock:/tmp/docker.sock:ro - - certs:/etc/nginx/certs:ro - - html:/usr/share/nginx/html:ro - networks: [proxy] - labels: - - 'com.github.nginx-proxy.nginx' - container_name: nginx-proxy - logging: *default-logging - - acme-companion: - image: nginxproxy/acme-companion - restart: always - environment: - DEFAULT_EMAIL: '' - volumes: - - /var/run/docker.sock:/var/run/docker.sock:ro - - certs:/etc/nginx/certs:rw - - html:/usr/share/nginx/html:rw - - acme:/etc/acme.sh - networks: [proxy] - container_name: acme-companion - depends_on: - - nginx-proxy - logging: *default-logging - - tgxmb: - image: yoursfunny/telegram-twitter-media-bot:latest - restart: always - environment: - LOCAL_USER_ID: '1000' - TELOXIDE_TOKEN: '' - BOT_ADMIN: '' - PIXIV_REFRESH_TOKEN: '' - TWITTER_AUTH_TOKEN: '' - EDIT_MESSAGE_TTL_SECONDS: '86400' - LINK_CACHE_TTL_SECONDS: '604800' - RUST_LOG: 'info' - VIRTUAL_HOST: '' - VIRTUAL_PORT: '8443' - # ACME_HOST: 'your.domain.com' - WEBHOOK: 'true' - WEBHOOK_LISTEN: '0.0.0.0' - WEBHOOK_PORT: '8443' - WEBHOOK_URL: 'https:///' - WEBHOOK_SECRET_TOKEN: '' - volumes: - - ./data:/app/data - networks: [proxy] - depends_on: - - nginx-proxy - container_name: tgxmb - logging: *default-logging - # Webhook mode only: the bot listens on WEBHOOK_PORT; nginx-proxy shows - # 502s while this is down, so surface it to the orchestrator. - healthcheck: - test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/8443'"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 10s - -volumes: - certs: - html: - acme: - -networks: - proxy: - name: proxy diff --git a/BILIBILI_PLAN.md b/docs/BILIBILI_PLAN.md similarity index 100% rename from BILIBILI_PLAN.md rename to docs/BILIBILI_PLAN.md