Repository: sansan0/TrendRadar Branch: master Commit: 1b41881ec431 Files: 110 Total size: 1.8 MB Directory structure: gitextract_ws4303_m/ ├── .dockerignore ├── .github/ │ ├── ISSUE_TEMPLATE/ │ │ ├── 01-bug-report.yml │ │ ├── 02-feature-request.yml │ │ ├── 03-ai-and-config.yml │ │ └── config.yml │ └── workflows/ │ ├── clean-crawler.yml │ ├── crawler.yml │ └── docker.yml ├── LICENSE ├── README-Cherry-Studio.md ├── README-EN.md ├── README-MCP-FAQ-EN.md ├── README-MCP-FAQ.md ├── README.md ├── config/ │ ├── ai_analysis_prompt.txt │ ├── ai_filter/ │ │ ├── extract_prompt.txt │ │ ├── prompt.txt │ │ └── update_tags_prompt.txt │ ├── ai_interests.txt │ ├── ai_translation_prompt.txt │ ├── config.yaml │ ├── custom/ │ │ ├── ai/ │ │ │ └── .gitkeep │ │ └── keyword/ │ │ └── .gitkeep │ ├── frequency_words.txt │ └── timeline.yaml ├── docker/ │ ├── Dockerfile │ ├── Dockerfile.mcp │ ├── docker-compose-build.yml │ ├── docker-compose.yml │ ├── entrypoint.sh │ └── manage.py ├── docs/ │ ├── assets/ │ │ ├── script.js │ │ └── style.css │ └── index.html ├── index.html ├── mcp_server/ │ ├── __init__.py │ ├── server.py │ ├── services/ │ │ ├── __init__.py │ │ ├── cache_service.py │ │ ├── data_service.py │ │ └── parser_service.py │ ├── tools/ │ │ ├── __init__.py │ │ ├── analytics.py │ │ ├── article_reader.py │ │ ├── config_mgmt.py │ │ ├── data_query.py │ │ ├── notification.py │ │ ├── search_tools.py │ │ ├── storage_sync.py │ │ └── system.py │ └── utils/ │ ├── __init__.py │ ├── date_parser.py │ ├── errors.py │ └── validators.py ├── pyproject.toml ├── requirements.txt ├── setup-mac.sh ├── setup-windows-en.bat ├── setup-windows.bat ├── start-http.bat ├── start-http.sh ├── trendradar/ │ ├── __init__.py │ ├── __main__.py │ ├── ai/ │ │ ├── __init__.py │ │ ├── analyzer.py │ │ ├── client.py │ │ ├── filter.py │ │ ├── formatter.py │ │ └── translator.py │ ├── context.py │ ├── core/ │ │ ├── __init__.py │ │ ├── analyzer.py │ │ ├── config.py │ │ ├── data.py │ │ ├── frequency.py │ │ ├── loader.py │ │ └── scheduler.py │ ├── crawler/ │ │ ├── __init__.py │ │ ├── fetcher.py │ │ └── rss/ │ │ ├── __init__.py │ │ ├── fetcher.py │ │ └── parser.py │ ├── notification/ │ │ ├── __init__.py │ │ ├── batch.py │ │ ├── dispatcher.py │ │ ├── formatters.py │ │ ├── renderer.py │ │ ├── senders.py │ │ └── splitter.py │ ├── report/ │ │ ├── __init__.py │ │ ├── formatter.py │ │ ├── generator.py │ │ ├── helpers.py │ │ ├── html.py │ │ └── rss_html.py │ ├── storage/ │ │ ├── __init__.py │ │ ├── ai_filter_schema.sql │ │ ├── base.py │ │ ├── local.py │ │ ├── manager.py │ │ ├── remote.py │ │ ├── rss_schema.sql │ │ ├── schema.sql │ │ └── sqlite_mixin.py │ └── utils/ │ ├── __init__.py │ ├── time.py │ └── url.py ├── version ├── version_configs └── version_mcp ================================================ FILE CONTENTS ================================================ ================================================ FILE: .dockerignore ================================================ .git/ .gitignore *.md README.md output/ __pycache__/ *.pyc *.pyo *.pyd .Python *.so .pytest_cache/ .vscode/ .idea/ *.swp *.swo *~ .DS_Store Thumbs.db docker/.env _image/ .github/ *.log .env.local .env.*.local version index.html ================================================ FILE: .github/ISSUE_TEMPLATE/01-bug-report.yml ================================================ # yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json name: 🐛 遇到问题了 description: 程序运行不正常、报错或功能失效（含 AI 分析问题） title: "[问题] " labels: ["bug"] body: - type: markdown attributes: value: | ### ⚠️ 提交前必读 **请确保你正在使用 TrendRadar 的最新版本。** 很多问题在最新代码中可能已经修复。如果你使用的是旧版本，我将无法处理，请先更新后再试。 **简单的描述 + 关键截图** 是最有效的沟通方式。 --- ### 📌 如何查看版本号？ | 部署方式 | 查看方法 | |---------|---------| | **Docker** | 查看容器启动日志，版本号显示在日志开头 | | **GitHub Actions** | 查看 [README 文档](https://github.com/sansan0/TrendRadar) 顶部的 ![version](https://img.shields.io/badge/version-blue) 徽章 | | **本地 Python** | 查看项目根目录的 `version` 文件 | - type: input id: version attributes: label: 📦 TrendRadar 版本 description: | 请务必提供版本号（如：v5.2.0 或 git commit id） 💡 Docker 用户：查看容器启动日志 | GitHub Actions 用户：查看文档顶部 version 徽章 placeholder: v5.2.0 或 commit hash validations: required: true - type: input id: mcp-version attributes: label: 🔌 MCP Server 版本 (可选) description: 如果你是通过 MCP 使用，请填写 MCP Server 的版本。 placeholder: v3.1.6 (非 MCP 用户留空) validations: required: false - type: dropdown id: bug-category attributes: label: 🏷️ 问题类别 options: - AI 分析相关（报错、内容异常、提示词失效等） - 数据获取相关（爬不到新闻、平台失效等） - 通知推送相关（收不到消息、推送报错等） - 部署运行相关（Docker、Actions、Python 报错） - 其他 validations: required: true - type: input id: ai-model attributes: label: 🤖 AI 模型名称（AI 问题必填） description: | 如果是 AI 分析相关问题，请提供你使用的具体模型名称。 AI 问题与模型能力密切相关，不同模型表现差异很大。 placeholder: "例如：deepseek/deepseek-chat、openai/gpt-4o、gemini/gemini-2.5-flash" validations: required: false - type: textarea id: bug-description attributes: label: 📝 描述发生了什么 placeholder: | 请描述： 1. 你在做什么？ 2. 出现了什么错误？（如果是 AI 问题，请贴出分析失败的错误提示） 3. 建议上传一张截图，这比文字更有力！ validations: required: true - type: textarea id: error-logs attributes: label: 📋 错误日志/配置（可选） description: | 贴出相关的错误日志或 config.yaml 片段（记得隐藏 API Key 等敏感信息） 💡 Docker 用户：使用 `docker logs trendradar` 查看日志 placeholder: | 贴出相关的错误日志或 config.yaml 片段： ``` 在这里贴内容... ``` validations: required: false - type: textarea id: screenshots attributes: label: 📷 截图（强烈建议） description: | ⚠️ **重要提示**：请提供**完整截图**，不要只截取局部！ - 错误截图应包含完整的错误信息和上下文 - 推送截图应包含完整的消息内容 - 配置截图应包含相关配置段的完整内容 局部截图往往缺少关键信息，会导致问题难以定位。 placeholder: 拖拽截图到这里，请确保截图完整，包含足够的上下文信息。 validations: required: false - type: dropdown id: environment attributes: label: 🖥️ 使用环境 options: - Docker (本地/NAS) - GitHub Actions - 本地 Python 运行 - MCP Server 客户端 (Cherry Studio等) validations: required: true ================================================ FILE: .github/ISSUE_TEMPLATE/02-feature-request.yml ================================================ # yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json name: 💡 我有个想法 description: 建议新功能、推送样式改进或体验优化 title: "[建议] " labels: ["enhancement"] body: - type: markdown attributes: value: | ### 💝 欢迎分享你的创意 你的好点子能让 TrendRadar 变得更好！ 目前主要关注以下方向的改进： - ✨ **AI 分析能力**：更智能的解读、更丰富的分析维度 - 🎨 **推送体验**：更好看的排版、更合理的信息展示 - 🛠️ **易用性优化**：配置更简单、运行更稳定 *注：目前暂不接受新爬虫平台的接入申请，感谢理解。* - type: textarea id: feature-description attributes: label: 💭 你的想法是什么？ placeholder: | 请简要描述： - 你希望增加什么功能？ - 它能解决什么问题？ - 如果有参考的图片或工具，欢迎上传截图。 validations: required: true - type: textarea id: use-case attributes: label: 🎯 使用场景（可选） placeholder: 例如：当我在...的时候，如果能...就太棒了。 validations: required: false ================================================ FILE: .github/ISSUE_TEMPLATE/03-ai-and-config.yml ================================================ # yaml-language-server: $schema=https://json.schemastore.org/github-issue-forms.json name: ✨ AI 提示词分享与配置求助 description: 分享你调优的 ai_analysis_prompt.txt 或寻求设置帮助 title: "[AI/配置] " labels: ["config", "AI"] body: - type: markdown attributes: value: | ### ✨ 提示词分享计划 欢迎在此分享你精心调优的 `ai_analysis_prompt.txt` 内容！ 优秀的提示词可以让 AI 分析更精准、更有趣。 --- 如果是**寻求配置帮助**，请尽量贴出你的错误表现。 --- ### 📌 如何查看版本号？ | 部署方式 | 查看方法 | |---------|---------| | **Docker** | 查看容器启动日志，版本号显示在日志开头 | | **GitHub Actions** | 查看 [README 文档](https://github.com/sansan0/TrendRadar) 顶部的 ![version](https://img.shields.io/badge/version-blue) 徽章 | | **本地 Python** | 查看项目根目录的 `version` 文件 | - type: dropdown id: category attributes: label: 🏷️ 目的 options: - 分享我的 AI 提示词 (ai_analysis_prompt.txt) - 寻求 AI 分析设置帮助 - 寻求基础功能配置帮助 (Webhook/RSS等) validations: required: true - type: input id: version attributes: label: 📦 TrendRadar 版本（求助时必填） description: | 如果是寻求帮助，请提供版本号。 💡 Docker 用户：查看容器启动日志 | GitHub Actions 用户：查看文档顶部 version 徽章 placeholder: v5.2.0 或 commit hash（分享提示词可留空） validations: required: false - type: input id: ai-model attributes: label: 🤖 AI 模型名称 description: | 请提供你使用的具体模型名称。 AI 分析效果与模型能力密切相关，不同模型表现差异很大。 分享提示词时也请注明，方便其他用户参考。 placeholder: "例如：deepseek/deepseek-chat、openai/gpt-4o、gemini/gemini-2.5-flash" validations: required: false - type: textarea id: share-content attributes: label: 📄 内容描述 placeholder: | - 如果是分享：请贴出你的提示词代码块，并简述它的分析风格。 - 如果是求助：请贴出你的配置片段（隐藏 Key）和遇到的现象。 validations: required: true - type: textarea id: screenshots attributes: label: 📷 效果截图（推荐） description: | ⚠️ **重要提示**：请提供**完整截图**，不要只截取局部！ - 分享时：展示 AI 分析的完整输出效果 - 求助时：展示完整的错误信息或异常表现 局部截图往往缺少关键信息，会导致问题难以定位。 placeholder: 拖拽分析结果截图或配置截图到这里，请确保截图完整。 validations: required: false ================================================ FILE: .github/ISSUE_TEMPLATE/config.yml ================================================ # yaml-language-server: $schema=https://json.schemastore.org/github-issue-config.json blank_issues_enabled: false ================================================ FILE: .github/workflows/clean-crawler.yml ================================================ name: Check In # ✅ 签到续期：运行此 workflow 可重置 7 天计时，保持 "Get Hot News" 正常运行 # ✅ Renewal: Run this workflow to reset the 7-day timer and keep "Get Hot News" active # # 📌 操作方法 / How to use: # 1. 点击 "Run workflow" 按钮 / Click "Run workflow" button # 2. 每 7 天内至少运行一次 / Run at least once every 7 days on: workflow_dispatch: jobs: del_runs: runs-on: ubuntu-latest permissions: actions: write contents: read steps: - name: Delete all workflow runs uses: Mattraks/delete-workflow-runs@v2 with: token: ${{ github.token }} repository: ${{ github.repository }} retain_days: 0 keep_minimum_runs: 0 delete_workflow_by_state_pattern: "ALL" delete_run_by_conclusion_pattern: "ALL" ================================================ FILE: .github/workflows/crawler.yml ================================================ name: Get Hot News on: schedule: # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # ⚠️ 试用版说明 / Trial Mode # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # # 🔄 运行机制 / How it works: # - 每个周期为 7 天，届时自动停止 # - 运行 "Check In" 会重置周期（重新开始 7 天倒计时，而非累加） # - Each cycle is 7 days, then auto-stops # - "Check In" resets the cycle (restarts 7-day countdown, not cumulative) # # 💡 设计初衷 / Why this design: # 如果 7 天都忘了签到，或许这些资讯对你来说并非刚需 # 适时的暂停，能帮你从信息流中抽离，给大脑留出喘息的空间 # If you forget for 7 days, maybe you don't really need it # A timely pause helps you detach from the stream and gives your mind space # # 🙏 珍惜资源 / Respect shared resources: # GitHub Actions 是平台提供的公共资源，每次运行都会消耗算力 # 签到机制确保资源分配给真正需要的用户，感谢你的理解与配合 # GitHub Actions is a shared public resource provided by the platform # Check-in ensures resources go to those who truly need it — thank you # # 🚀 长期使用请部署 Docker 版本 / For long-term use, deploy Docker version # # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # # 📝 修改运行时间：只改第一个数字（0-59），表示每小时第几分钟运行 # 📝 Change time: Only modify the first number (0-59) = minute of each hour # # 示例 / Examples: # "15 * * * *" → 每小时第15分钟 / minute 15 every hour # "30 0-14 * * *" → 北京时间 8:00-22:00 每小时第30分钟 / Beijing 8am-10pm # - cron: "33 * * * *" workflow_dispatch: concurrency: group: crawler-${{ github.ref_name }} cancel-in-progress: true permissions: contents: read actions: write jobs: crawl: runs-on: ubuntu-latest timeout-minutes: 15 steps: - name: Checkout repository uses: actions/checkout@v4 with: fetch-depth: 1 clean: true - name: Check Expiration env: GH_TOKEN: ${{ github.token }} run: | WORKFLOW_FILE="crawler.yml" API_URL="repos/${{ github.repository }}/actions/workflows/$WORKFLOW_FILE/runs" TOTAL=$(gh api "$API_URL" --jq '.total_count') if [ -z "$TOTAL" ] || [ "$TOTAL" -eq 0 ]; then echo "No previous runs found, skipping expiration check" exit 0 fi LAST_PAGE=$(( (TOTAL + 99) / 100 )) FIRST_RUN_DATE=$(gh api "$API_URL?per_page=100&page=$LAST_PAGE" --jq '.workflow_runs[-1].created_at') if [ -n "$FIRST_RUN_DATE" ]; then CURRENT_TIMESTAMP=$(date +%s) FIRST_RUN_TIMESTAMP=$(date -d "$FIRST_RUN_DATE" +%s) DIFF_SECONDS=$((CURRENT_TIMESTAMP - FIRST_RUN_TIMESTAMP)) LIMIT_SECONDS=604800 if [ $DIFF_SECONDS -gt $LIMIT_SECONDS ]; then echo "⚠️ 试用期已结束，请运行 'Check In' 签到续期" echo "⚠️ Trial expired. Run 'Check In' to renew." gh workflow disable "$WORKFLOW_FILE" exit 1 else DAYS_LEFT=$(( (LIMIT_SECONDS - DIFF_SECONDS) / 86400 )) echo "✅ 试用期剩余 ${DAYS_LEFT} 天，到期前请运行 'Check In' 签到续期" echo "✅ Trial: ${DAYS_LEFT} days left. Run 'Check In' before expiry to renew." fi fi # -------------------------------------------------------------------------------- # 🚦 TRAFFIC CONTROL / 流量控制 # -------------------------------------------------------------------------------- # EN: Generates a random delay between 1 and 300 seconds (5 minutes). # Critical for load balancing. # # CN: 生成 1 到 300 秒（5分钟）之间的随机延迟。 # 这对负载均衡至关重要。 # - name: Random Delay (Traffic Control) # if: success() # run: | # echo "🎲 Traffic Control: Generating random delay..." # DELAY=$(( ( RANDOM % 300 ) + 1 )) # echo "⏸️ Sleeping for ${DELAY} seconds to spread the load..." # sleep ${DELAY}s # echo "▶️ Delay finished. Starting crawler..." - name: Set up Python if: success() uses: actions/setup-python@v5 with: python-version: "3.10" cache: "pip" - name: Install dependencies if: success() run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Verify required files if: success() run: | if [ ! -f config/config.yaml ]; then echo "Error: Config missing" exit 1 fi - name: Run crawler if: success() env: FEISHU_WEBHOOK_URL: ${{ secrets.FEISHU_WEBHOOK_URL }} TELEGRAM_BOT_TOKEN: ${{ secrets.TELEGRAM_BOT_TOKEN }} TELEGRAM_CHAT_ID: ${{ secrets.TELEGRAM_CHAT_ID }} DINGTALK_WEBHOOK_URL: ${{ secrets.DINGTALK_WEBHOOK_URL }} WEWORK_WEBHOOK_URL: ${{ secrets.WEWORK_WEBHOOK_URL }} WEWORK_MSG_TYPE: ${{ secrets.WEWORK_MSG_TYPE }} EMAIL_FROM: ${{ secrets.EMAIL_FROM }} EMAIL_PASSWORD: ${{ secrets.EMAIL_PASSWORD }} EMAIL_TO: ${{ secrets.EMAIL_TO }} EMAIL_SMTP_SERVER: ${{ secrets.EMAIL_SMTP_SERVER }} EMAIL_SMTP_PORT: ${{ secrets.EMAIL_SMTP_PORT }} NTFY_TOPIC: ${{ secrets.NTFY_TOPIC }} NTFY_SERVER_URL: ${{ secrets.NTFY_SERVER_URL }} NTFY_TOKEN: ${{ secrets.NTFY_TOKEN }} BARK_URL: ${{ secrets.BARK_URL }} SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK_URL }} # 通用Webhook配置 GENERIC_WEBHOOK_URL: ${{ secrets.GENERIC_WEBHOOK_URL }} GENERIC_WEBHOOK_TEMPLATE: ${{ secrets.GENERIC_WEBHOOK_TEMPLATE }} # AI 配置（ai_analysis 和 ai_translation 共享模型配置） AI_ANALYSIS_ENABLED: ${{ secrets.AI_ANALYSIS_ENABLED }} AI_API_KEY: ${{ secrets.AI_API_KEY }} AI_MODEL: ${{ secrets.AI_MODEL }} AI_API_BASE: ${{ secrets.AI_API_BASE }} # 远程存储配置 S3_BUCKET_NAME: ${{ secrets.S3_BUCKET_NAME }} S3_ACCESS_KEY_ID: ${{ secrets.S3_ACCESS_KEY_ID }} S3_SECRET_ACCESS_KEY: ${{ secrets.S3_SECRET_ACCESS_KEY }} S3_ENDPOINT_URL: ${{ secrets.S3_ENDPOINT_URL }} S3_REGION: ${{ secrets.S3_REGION }} GITHUB_ACTIONS: true run: python -m trendradar ================================================ FILE: .github/workflows/docker.yml ================================================ name: Build and Push Docker Images on: push: tags: - "v*" # 主项目版本 - "mcp-v*" # MCP 版本 workflow_dispatch: inputs: image: description: "选择要构建的镜像" required: true default: "all" type: choice options: - all - crawler - mcp env: REGISTRY: docker.io jobs: build-crawler: runs-on: ubuntu-latest # 条件：v* 标签（排除 mcp-v*）或手动触发选择 all/crawler if: | (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') && !startsWith(github.ref, 'refs/tags/mcp-v')) || (github.event_name == 'workflow_dispatch' && (github.event.inputs.image == 'all' || github.event.inputs.image == 'crawler')) steps: - name: Checkout uses: actions/checkout@v4 - name: Set up QEMU uses: docker/setup-qemu-action@v3 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 with: driver-opts: | network=host - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Extract metadata id: meta uses: docker/metadata-action@v5 with: images: wantcat/trendradar tags: | type=semver,pattern={{version}} type=semver,pattern={{major}}.{{minor}} type=raw,value=latest - name: Build and push uses: docker/build-push-action@v5 env: BUILDKIT_PROGRESS: plain with: context: . file: ./docker/Dockerfile platforms: linux/amd64,linux/arm64 push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max build-mcp: runs-on: ubuntu-latest # 条件：mcp-v* 标签 或手动触发选择 all/mcp if: | (github.event_name == 'push' && startsWith(github.ref, 'refs/tags/mcp-v')) || (github.event_name == 'workflow_dispatch' && (github.event.inputs.image == 'all' || github.event.inputs.image == 'mcp')) steps: - name: Checkout uses: actions/checkout@v4 - name: Set up QEMU uses: docker/setup-qemu-action@v3 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 with: driver-opts: | network=host - name: Login to Docker Hub uses: docker/login-action@v3 with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - name: Extract version from tag id: version run: | if [[ "${{ github.ref }}" == refs/tags/mcp-v* ]]; then VERSION="${GITHUB_REF#refs/tags/mcp-v}" echo "version=${VERSION}" >> $GITHUB_OUTPUT echo "major_minor=$(echo $VERSION | cut -d. -f1,2)" >> $GITHUB_OUTPUT else echo "version=latest" >> $GITHUB_OUTPUT echo "major_minor=latest" >> $GITHUB_OUTPUT fi - name: Extract metadata id: meta uses: docker/metadata-action@v5 with: images: wantcat/trendradar-mcp tags: | type=raw,value=${{ steps.version.outputs.version }} type=raw,value=${{ steps.version.outputs.major_minor }} type=raw,value=latest - name: Build and push uses: docker/build-push-action@v5 env: BUILDKIT_PROGRESS: plain with: context: . file: ./docker/Dockerfile.mcp platforms: linux/amd64,linux/arm64 push: true tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max ================================================ FILE: LICENSE ================================================ GNU GENERAL PUBLIC LICENSE Version 3, 29 June 2007 Copyright (C) 2007 Free Software Foundation, Inc. Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Preamble The GNU General Public License is a free, copyleft license for software and other kinds of works. The licenses for most software and other practical works are designed to take away your freedom to share and change the works. By contrast, the GNU General Public License is intended to guarantee your freedom to share and change all versions of a program--to make sure it remains free software for all its users. We, the Free Software Foundation, use the GNU General Public License for most of our software; it applies also to any other work released this way by its authors. You can apply it to your programs, too. When we speak of free software, we are referring to freedom, not price. Our General Public Licenses are designed to make sure that you have the freedom to distribute copies of free software (and charge for them if you wish), that you receive source code or can get it if you want it, that you can change the software or use pieces of it in new free programs, and that you know you can do these things. To protect your rights, we need to prevent others from denying you these rights or asking you to surrender the rights. Therefore, you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others. For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights. Developers that use the GNU GPL protect your rights with two steps: (1) assert copyright on the software, and (2) offer you this License giving you legal permission to copy, distribute and/or modify it. For the developers' and authors' protection, the GPL clearly explains that there is no warranty for this free software. For both users' and authors' sake, the GPL requires that modified versions be marked as changed, so that their problems will not be attributed erroneously to authors of previous versions. Some devices are designed to deny users access to install or run modified versions of the software inside them, although the manufacturer can do so. This is fundamentally incompatible with the aim of protecting users' freedom to change the software. The systematic pattern of such abuse occurs in the area of products for individuals to use, which is precisely where it is most unacceptable. Therefore, we have designed this version of the GPL to prohibit the practice for those products. If such problems arise substantially in other domains, we stand ready to extend this provision to those domains in future versions of the GPL, as needed to protect the freedom of users. Finally, every program is threatened constantly by software patents. States should not allow patents to restrict development and use of software on general-purpose computers, but in those that do, we wish to avoid the special danger that patents applied to a free program could make it effectively proprietary. To prevent this, the GPL assures that patents cannot be used to render the program non-free. The precise terms and conditions for copying, distribution and modification follow. TERMS AND CONDITIONS 0. Definitions. "This License" refers to version 3 of the GNU General Public License. "Copyright" also means copyright-like laws that apply to other kinds of works, such as semiconductor masks. "The Program" refers to any copyrightable work licensed under this License. Each licensee is addressed as "you". "Licensees" and "recipients" may be individuals or organizations. To "modify" a work means to copy from or adapt all or part of the work in a fashion requiring copyright permission, other than the making of an exact copy. The resulting work is called a "modified version" of the earlier work or a work "based on" the earlier work. A "covered work" means either the unmodified Program or a work based on the Program. To "propagate" a work means to do anything with it that, without permission, would make you directly or secondarily liable for infringement under applicable copyright law, except executing it on a computer or modifying a private copy. Propagation includes copying, distribution (with or without modification), making available to the public, and in some countries other activities as well. To "convey" a work means any kind of propagation that enables other parties to make or receive copies. Mere interaction with a user through a computer network, with no transfer of a copy, is not conveying. An interactive user interface displays "Appropriate Legal Notices" to the extent that it includes a convenient and prominently visible feature that (1) displays an appropriate copyright notice, and (2) tells the user that there is no warranty for the work (except to the extent that warranties are provided), that licensees may convey the work under this License, and how to view a copy of this License. If the interface presents a list of user commands or options, such as a menu, a prominent item in the list meets this criterion. 1. Source Code. The "source code" for a work means the preferred form of the work for making modifications to it. "Object code" means any non-source form of a work. A "Standard Interface" means an interface that either is an official standard defined by a recognized standards body, or, in the case of interfaces specified for a particular programming language, one that is widely used among developers working in that language. The "System Libraries" of an executable work include anything, other than the work as a whole, that (a) is included in the normal form of packaging a Major Component, but which is not part of that Major Component, and (b) serves only to enable use of the work with that Major Component, or to implement a Standard Interface for which an implementation is available to the public in source code form. A "Major Component", in this context, means a major essential component (kernel, window system, and so on) of the specific operating system (if any) on which the executable work runs, or a compiler used to produce the work, or an object code interpreter used to run it. The "Corresponding Source" for a work in object code form means all the source code needed to generate, install, and (for an executable work) run the object code and to modify the work, including scripts to control those activities. However, it does not include the work's System Libraries, or general-purpose tools or generally available free programs which are used unmodified in performing those activities but which are not part of the work. For example, Corresponding Source includes interface definition files associated with source files for the work, and the source code for shared libraries and dynamically linked subprograms that the work is specifically designed to require, such as by intimate data communication or control flow between those subprograms and other parts of the work. The Corresponding Source need not include anything that users can regenerate automatically from other parts of the Corresponding Source. The Corresponding Source for a work in source code form is that same work. 2. Basic Permissions. All rights granted under this License are granted for the term of copyright on the Program, and are irrevocable provided the stated conditions are met. This License explicitly affirms your unlimited permission to run the unmodified Program. The output from running a covered work is covered by this License only if the output, given its content, constitutes a covered work. This License acknowledges your rights of fair use or other equivalent, as provided by copyright law. You may make, run and propagate covered works that you do not convey, without conditions so long as your license otherwise remains in force. You may convey covered works to others for the sole purpose of having them make modifications exclusively for you, or provide you with facilities for running those works, provided that you comply with the terms of this License in conveying all material for which you do not control copyright. Those thus making or running the covered works for you must do so exclusively on your behalf, under your direction and control, on terms that prohibit them from making any copies of your copyrighted material outside their relationship with you. Conveying under any other circumstances is permitted solely under the conditions stated below. Sublicensing is not allowed; section 10 makes it unnecessary. 3. Protecting Users' Legal Rights From Anti-Circumvention Law. No covered work shall be deemed part of an effective technological measure under any applicable law fulfilling obligations under article 11 of the WIPO copyright treaty adopted on 20 December 1996, or similar laws prohibiting or restricting circumvention of such measures. When you convey a covered work, you waive any legal power to forbid circumvention of technological measures to the extent such circumvention is effected by exercising rights under this License with respect to the covered work, and you disclaim any intention to limit operation or modification of the work as a means of enforcing, against the work's users, your or third parties' legal rights to forbid circumvention of technological measures. 4. Conveying Verbatim Copies. You may convey verbatim copies of the Program's source code as you receive it, in any medium, provided that you conspicuously and appropriately publish on each copy an appropriate copyright notice; keep intact all notices stating that this License and any non-permissive terms added in accord with section 7 apply to the code; keep intact all notices of the absence of any warranty; and give all recipients a copy of this License along with the Program. You may charge any price or no price for each copy that you convey, and you may offer support or warranty protection for a fee. 5. Conveying Modified Source Versions. You may convey a work based on the Program, or the modifications to produce it from the Program, in the form of source code under the terms of section 4, provided that you also meet all of these conditions: a) The work must carry prominent notices stating that you modified it, and giving a relevant date. b) The work must carry prominent notices stating that it is released under this License and any conditions added under section 7. This requirement modifies the requirement in section 4 to "keep intact all notices". c) You must license the entire work, as a whole, under this License to anyone who comes into possession of a copy. This License will therefore apply, along with any applicable section 7 additional terms, to the whole of the work, and all its parts, regardless of how they are packaged. This License gives no permission to license the work in any other way, but it does not invalidate such permission if you have separately received it. d) If the work has interactive user interfaces, each must display Appropriate Legal Notices; however, if the Program has interactive interfaces that do not display Appropriate Legal Notices, your work need not make them do so. A compilation of a covered work with other separate and independent works, which are not by their nature extensions of the covered work, and which are not combined with it such as to form a larger program, in or on a volume of a storage or distribution medium, is called an "aggregate" if the compilation and its resulting copyright are not used to limit the access or legal rights of the compilation's users beyond what the individual works permit. Inclusion of a covered work in an aggregate does not cause this License to apply to the other parts of the aggregate. 6. Conveying Non-Source Forms. You may convey a covered work in object code form under the terms of sections 4 and 5, provided that you also convey the machine-readable Corresponding Source under the terms of this License, in one of these ways: a) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by the Corresponding Source fixed on a durable physical medium customarily used for software interchange. b) Convey the object code in, or embodied in, a physical product (including a physical distribution medium), accompanied by a written offer, valid for at least three years and valid for as long as you offer spare parts or customer support for that product model, to give anyone who possesses the object code either (1) a copy of the Corresponding Source for all the software in the product that is covered by this License, on a durable physical medium customarily used for software interchange, for a price no more than your reasonable cost of physically performing this conveying of source, or (2) access to copy the Corresponding Source from a network server at no charge. c) Convey individual copies of the object code with a copy of the written offer to provide the Corresponding Source. This alternative is allowed only occasionally and noncommercially, and only if you received the object code with such an offer, in accord with subsection 6b. d) Convey the object code by offering access from a designated place (gratis or for a charge), and offer equivalent access to the Corresponding Source in the same way through the same place at no further charge. You need not require recipients to copy the Corresponding Source along with the object code. If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party) that supports equivalent copying facilities, provided you maintain clear directions next to the object code saying where to find the Corresponding Source. Regardless of what server hosts the Corresponding Source, you remain obligated to ensure that it is available for as long as needed to satisfy these requirements. e) Convey the object code using peer-to-peer transmission, provided you inform other peers where the object code and Corresponding Source of the work are being offered to the general public at no charge under subsection 6d. A separable portion of the object code, whose source code is excluded from the Corresponding Source as a System Library, need not be included in conveying the object code work. A "User Product" is either (1) a "consumer product", which means any tangible personal property which is normally used for personal, family, or household purposes, or (2) anything designed or sold for incorporation into a dwelling. In determining whether a product is a consumer product, doubtful cases shall be resolved in favor of coverage. For a particular product received by a particular user, "normally used" refers to a typical or common use of that class of product, regardless of the status of the particular user or of the way in which the particular user actually uses, or expects or is expected to use, the product. A product is a consumer product regardless of whether the product has substantial commercial, industrial or non-consumer uses, unless such uses represent the only significant mode of use of the product. "Installation Information" for a User Product means any methods, procedures, authorization keys, or other information required to install and execute modified versions of a covered work in that User Product from a modified version of its Corresponding Source. The information must suffice to ensure that the continued functioning of the modified object code is in no case prevented or interfered with solely because modification has been made. If you convey an object code work under this section in, or with, or specifically for use in, a User Product, and the conveying occurs as part of a transaction in which the right of possession and use of the User Product is transferred to the recipient in perpetuity or for a fixed term (regardless of how the transaction is characterized), the Corresponding Source conveyed under this section must be accompanied by the Installation Information. But this requirement does not apply if neither you nor any third party retains the ability to install modified object code on the User Product (for example, the work has been installed in ROM). The requirement to provide Installation Information does not include a requirement to continue to provide support service, warranty, or updates for a work that has been modified or installed by the recipient, or for the User Product in which it has been modified or installed. Access to a network may be denied when the modification itself materially and adversely affects the operation of the network or violates the rules and protocols for communication across the network. Corresponding Source conveyed, and Installation Information provided, in accord with this section must be in a format that is publicly documented (and with an implementation available to the public in source code form), and must require no special password or key for unpacking, reading or copying. 7. Additional Terms. "Additional permissions" are terms that supplement the terms of this License by making exceptions from one or more of its conditions. Additional permissions that are applicable to the entire Program shall be treated as though they were included in this License, to the extent that they are valid under applicable law. If additional permissions apply only to part of the Program, that part may be used separately under those permissions, but the entire Program remains governed by this License without regard to the additional permissions. When you convey a copy of a covered work, you may at your option remove any additional permissions from that copy, or from any part of it. (Additional permissions may be written to require their own removal in certain cases when you modify the work.) You may place additional permissions on material, added by you to a covered work, for which you have or can give appropriate copyright permission. Notwithstanding any other provision of this License, for material you add to a covered work, you may (if authorized by the copyright holders of that material) supplement the terms of this License with terms: a) Disclaiming warranty or limiting liability differently from the terms of sections 15 and 16 of this License; or b) Requiring preservation of specified reasonable legal notices or author attributions in that material or in the Appropriate Legal Notices displayed by works containing it; or c) Prohibiting misrepresentation of the origin of that material, or requiring that modified versions of such material be marked in reasonable ways as different from the original version; or d) Limiting the use for publicity purposes of names of licensors or authors of the material; or e) Declining to grant rights under trademark law for use of some trade names, trademarks, or service marks; or f) Requiring indemnification of licensors and authors of that material by anyone who conveys the material (or modified versions of it) with contractual assumptions of liability to the recipient, for any liability that these contractual assumptions directly impose on those licensors and authors. All other non-permissive additional terms are considered "further restrictions" within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term. If a license document contains a further restriction but permits relicensing or conveying under this License, you may add to a covered work material governed by the terms of that license document, provided that the further restriction does not survive such relicensing or conveying. If you add terms to a covered work in accord with this section, you must place, in the relevant source files, a statement of the additional terms that apply to those files, or a notice indicating where to find the applicable terms. Additional terms, permissive or non-permissive, may be stated in the form of a separately written license, or stated as exceptions; the above requirements apply either way. 8. Termination. You may not propagate or modify a covered work except as expressly provided under this License. Any attempt otherwise to propagate or modify it is void, and will automatically terminate your rights under this License (including any patent licenses granted under the third paragraph of section 11). However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation. Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice. Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, you do not qualify to receive new licenses for the same material under section 10. 9. Acceptance Not Required for Having Copies. You are not required to accept this License in order to receive or run a copy of the Program. Ancillary propagation of a covered work occurring solely as a consequence of using peer-to-peer transmission to receive a copy likewise does not require acceptance. However, nothing other than this License grants you permission to propagate or modify any covered work. These actions infringe copyright if you do not accept this License. Therefore, by modifying or propagating a covered work, you indicate your acceptance of this License to do so. 10. Automatic Licensing of Downstream Recipients. Each time you convey a covered work, the recipient automatically receives a license from the original licensors, to run, modify and propagate that work, subject to this License. You are not responsible for enforcing compliance by third parties with this License. An "entity transaction" is a transaction transferring control of an organization, or substantially all assets of one, or subdividing an organization, or merging organizations. If propagation of a covered work results from an entity transaction, each party to that transaction who receives a copy of the work also receives whatever licenses to the work the party's predecessor in interest had or could give under the previous paragraph, plus a right to possession of the Corresponding Source of the work from the predecessor in interest, if the predecessor has it or can get it with reasonable efforts. You may not impose any further restrictions on the exercise of the rights granted or affirmed under this License. For example, you may not impose a license fee, royalty, or other charge for exercise of rights granted under this License, and you may not initiate litigation (including a cross-claim or counterclaim in a lawsuit) alleging that any patent claim is infringed by making, using, selling, offering for sale, or importing the Program or any portion of it. 11. Patents. A "contributor" is a copyright holder who authorizes use under this License of the Program or a work on which the Program is based. The work thus licensed is called the contributor's "contributor version". A contributor's "essential patent claims" are all patent claims owned or controlled by the contributor, whether already acquired or hereafter acquired, that would be infringed by some manner, permitted by this License, of making, using, or selling its contributor version, but do not include claims that would be infringed only as a consequence of further modification of the contributor version. For purposes of this definition, "control" includes the right to grant patent sublicenses in a manner consistent with the requirements of this License. Each contributor grants you a non-exclusive, worldwide, royalty-free patent license under the contributor's essential patent claims, to make, use, sell, offer for sale, import and otherwise run, modify and propagate the contents of its contributor version. In the following three paragraphs, a "patent license" is any express agreement or commitment, however denominated, not to enforce a patent (such as an express permission to practice a patent or covenant not to sue for patent infringement). To "grant" such a patent license to a party means to make such an agreement or commitment not to enforce a patent against the party. If you convey a covered work, knowingly relying on a patent license, and the Corresponding Source of the work is not available for anyone to copy, free of charge and under the terms of this License, through a publicly available network server or other readily accessible means, then you must either (1) cause the Corresponding Source to be so available, or (2) arrange to deprive yourself of the benefit of the patent license for this particular work, or (3) arrange, in a manner consistent with the requirements of this License, to extend the patent license to downstream recipients. "Knowingly relying" means you have actual knowledge that, but for the patent license, your conveying the covered work in a country, or your recipient's use of the covered work in a country, would infringe one or more identifiable patents in that country that you have reason to believe are valid. If, pursuant to or in connection with a single transaction or arrangement, you convey, or propagate by procuring conveyance of, a covered work, and grant a patent license to some of the parties receiving the covered work authorizing them to use, propagate, modify or convey a specific copy of the covered work, then the patent license you grant is automatically extended to all recipients of the covered work and works based on it. A patent license is "discriminatory" if it does not include within the scope of its coverage, prohibits the exercise of, or is conditioned on the non-exercise of one or more of the rights that are specifically granted under this License. You may not convey a covered work if you are a party to an arrangement with a third party that is in the business of distributing software, under which you make payment to the third party based on the extent of your activity of conveying the work, and under which the third party grants, to any of the parties who would receive the covered work from you, a discriminatory patent license (a) in connection with copies of the covered work conveyed by you (or copies made from those copies), or (b) primarily for and in connection with specific products or compilations that contain the covered work, unless you entered into that arrangement, or that patent license was granted, prior to 28 March 2007. Nothing in this License shall be construed as excluding or limiting any implied license or other defenses to infringement that may otherwise be available to you under applicable patent law. 12. No Surrender of Others' Freedom. If conditions are imposed on you (whether by court order, agreement or otherwise) that contradict the conditions of this License, they do not excuse you from the conditions of this License. If you cannot convey a covered work so as to satisfy simultaneously your obligations under this License and any other pertinent obligations, then as a consequence you may not convey it at all. For example, if you agree to terms that obligate you to collect a royalty for further conveying from those to whom you convey the Program, the only way you could satisfy both those terms and this License would be to refrain entirely from conveying the Program. 13. Use with the GNU Affero General Public License. Notwithstanding any other provision of this License, you have permission to link or combine any covered work with a work licensed under version 3 of the GNU Affero General Public License into a single combined work, and to convey the resulting work. The terms of this License will continue to apply to the part which is the covered work, but the special requirements of the GNU Affero General Public License, section 13, concerning interaction through a network will apply to the combination as such. 14. Revised Versions of this License. The Free Software Foundation may publish revised and/or new versions of the GNU General Public License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. Each version is given a distinguishing version number. If the Program specifies that a certain numbered version of the GNU General Public License "or any later version" applies to it, you have the option of following the terms and conditions either of that numbered version or of any later version published by the Free Software Foundation. If the Program does not specify a version number of the GNU General Public License, you may choose any version ever published by the Free Software Foundation. If the Program specifies that a proxy can decide which future versions of the GNU General Public License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Program. Later license versions may give you additional or different permissions. However, no additional obligations are imposed on any author or copyright holder as a result of your choosing to follow a later version. 15. Disclaimer of Warranty. THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION. 16. Limitation of Liability. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. 17. Interpretation of Sections 15 and 16. If the disclaimer of warranty and limitation of liability provided above cannot be given local legal effect according to their terms, reviewing courts shall apply local law that most closely approximates an absolute waiver of all civil liability in connection with the Program, unless a warranty or assumption of liability accompanies a copy of the Program in return for a fee. END OF TERMS AND CONDITIONS How to Apply These Terms to Your New Programs If you develop a new program, and you want it to be of the greatest possible use to the public, the best way to achieve this is to make it free software which everyone can redistribute and change under these terms. To do so, attach the following notices to the program. It is safest to attach them to the start of each source file to most effectively state the exclusion of warranty; and each file should have at least the "copyright" line and a pointer to where the full notice is found. Copyright (C) This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with this program. If not, see . Also add information on how to contact you by electronic and paper mail. If the program does terminal interaction, make it output a short notice like this when it starts in an interactive mode: Copyright (C) This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. This is free software, and you are welcome to redistribute it under certain conditions; type `show c' for details. The hypothetical commands `show w' and `show c' should show the appropriate parts of the General Public License. Of course, your program's commands might be different; for a GUI interface, you would use an "about box". You should also get your employer (if you work as a programmer) or school, if any, to sign a "copyright disclaimer" for the program, if necessary. For more information on this, and how to apply and follow the GNU GPL, see . The GNU General Public License does not permit incorporating your program into proprietary programs. If your program is a subroutine library, you may consider it more useful to permit linking proprietary applications with the library. If this is what you want to do, use the GNU Lesser General Public License instead of this License. But first, please read . ================================================ FILE: README-Cherry-Studio.md ================================================ # TrendRadar × Cherry Studio 部署指南 🍒 > **适合人群**：零编程基础的用户 > **客户端**：Cherry Studio（免费开源 GUI 客户端） --- ## 📥 第一步：下载 Cherry Studio ### Windows 用户 访问官网下载：https://cherry-ai.com/ 或直接下载：[Cherry-Studio-Windows.exe](https://github.com/kangfenmao/cherry-studio/releases/latest) ### Mac 用户 访问官网下载：https://cherry-ai.com/ 或直接下载：[Cherry-Studio-Mac.dmg](https://github.com/kangfenmao/cherry-studio/releases/latest) --- ## 📦 第二步：获取项目代码 为什么需要获取项目代码？ AI 分析功能需要读取项目中的新闻数据才能工作。无论你使用 GitHub Actions 还是 Docker 部署，爬虫生成的新闻数据都保存在项目的 output 目录中。因此，在配置 MCP 服务器之前，需要先获取完整的项目代码（包含数据文件）。 根据你的技术水平，可以选择以下任一方式获取：： ### 方法一：Git Clone（推荐给技术用户） 如果你熟悉 Git，可以使用以下命令克隆项目： ```bash git clone https://github.com/你的用户名/你的项目名.git cd 你的项目名 ``` **优点**： - 可以随时拉取一个命令就可以更新最新数据到本地了（`git pull`） ### 方法二：直接下载 ZIP 压缩包（推荐给初学者） 1. **访问 GitHub 项目页面** - 项目链接：`https://github.com/你的用户名/你的项目名` 2. **下载压缩包** - 点击绿色的 "Code" 按钮 - 选择 "Download ZIP" - 或直接访问：`https://github.com/你的用户名/你的项目名/archive/refs/heads/master.zip` **注意事项**： - 步骤稍微麻烦，后续更新数据需要重复上面步骤，然后覆盖本地数据(output 目录) --- ## 🚀 第三步：一键部署 MCP 服务器 ### Windows 用户 1. **双击运行**项目文件夹中的 `setup-windows.bat`，如果有问题，就运行 `setup-windows-en.bat` 2. **等待安装完成** 3. **记录显示的配置信息**（命令路径和参数） ### Mac 用户 1. **打开终端**（在启动台搜索"终端"） 2. **拖拽**项目文件夹中的 `setup-mac.sh` 到终端窗口 3. **按回车键** 4. **记录显示的配置信息** --- ## 🔧 第四步：配置 Cherry Studio ### 1. 打开设置 启动 Cherry Studio，点击右上角 ⚙️ **设置** 按钮 ### 2. 添加 MCP 服务器 在设置页面找到：**MCP** → 点击 **添加** ### 3. 填写配置（重要！） 根据刚才的安装脚本显示的信息填写 ### 4. 保存并启用 - 点击 **保存** 按钮 - 确保 MCP 服务器列表中的开关是 **开启** 状态 ✅ --- ## ✅ 第五步：验证是否成功 ### 1. 测试连接 在 Cherry Studio 的对话框中输入： ``` 帮我爬取最新的新闻 ``` 或者尝试其他测试命令： ``` 搜索最近3天关于"人工智能"的新闻 查找2025年1月的"特斯拉"相关报道 分析"iPhone"的热度趋势 ``` **提示**：当你说"最近3天"时，AI会自动计算日期范围并搜索。 ### 2. 成功标志 如果配置成功，AI 会： - ✅ 调用 TrendRadar 工具 - ✅ 返回真实的新闻数据 - ✅ 显示平台、标题、排名等信息 --- ## 🎯 进阶配置 ### HTTP 模式（可选） 如果需要远程访问或多客户端共享，可以使用 HTTP 模式： #### Windows 双击运行 `start-http.bat` #### Mac ```bash ./start-http.sh ``` 然后在 Cherry Studio 中配置： ``` 类型: streamableHttp URL: http://localhost:3333/mcp ``` ================================================ FILE: README-EN.md ================================================
TrendRadar Banner Deploy in 30 seconds — Say goodbye to endless scrolling, only see the news you truly care about sansan0%2FTrendRadar | Trendshift [![GitHub Stars](https://img.shields.io/github/stars/sansan0/TrendRadar?style=flat-square&logo=github&color=yellow)](https://github.com/sansan0/TrendRadar/stargazers) [![GitHub Forks](https://img.shields.io/github/forks/sansan0/TrendRadar?style=flat-square&logo=github&color=blue)](https://github.com/sansan0/TrendRadar/network/members) [![License](https://img.shields.io/badge/license-GPL--3.0-blue.svg?style=flat-square)](LICENSE) [![Version](https://img.shields.io/badge/version-v6.5.0-blue.svg)](https://github.com/sansan0/TrendRadar) [![MCP](https://img.shields.io/badge/MCP-v4.0.0-green.svg)](https://github.com/sansan0/TrendRadar) [![RSS](https://img.shields.io/badge/RSS-Feed_Support-orange.svg?style=flat-square&logo=rss&logoColor=white)](https://github.com/sansan0/TrendRadar) [![AI Translation](https://img.shields.io/badge/AI-Multi--Language-purple.svg?style=flat-square)](https://github.com/sansan0/TrendRadar) [![WeWork](https://img.shields.io/badge/WeWork-Notification-00D4AA?style=flat-square)](https://work.weixin.qq.com/) [![WeChat](https://img.shields.io/badge/WeChat-Notification-00D4AA?style=flat-square)](https://weixin.qq.com/) [![Telegram](https://img.shields.io/badge/Telegram-Notification-00D4AA?style=flat-square)](https://telegram.org/) [![DingTalk](https://img.shields.io/badge/DingTalk-Notification-00D4AA?style=flat-square)](#) [![Feishu](https://img.shields.io/badge/Feishu-Notification-00D4AA?style=flat-square)](https://www.feishu.cn/) [![Email](https://img.shields.io/badge/Email-Notification-00D4AA?style=flat-square)](#) [![ntfy](https://img.shields.io/badge/ntfy-Notification-00D4AA?style=flat-square)](https://github.com/binwiederhier/ntfy) [![Bark](https://img.shields.io/badge/Bark-Notification-00D4AA?style=flat-square)](https://github.com/Finb/Bark) [![Slack](https://img.shields.io/badge/Slack-Notification-00D4AA?style=flat-square)](https://slack.com/) [![Generic Webhook](https://img.shields.io/badge/Generic-Webhook-607D8B?style=flat-square&logo=webhook&logoColor=white)](#) [![GitHub Actions](https://img.shields.io/badge/GitHub_Actions-Automation-2088FF?style=flat-square&logo=github-actions&logoColor=white)](https://github.com/sansan0/TrendRadar) [![GitHub Pages](https://img.shields.io/badge/GitHub_Pages-Deployment-4285F4?style=flat-square&logo=github&logoColor=white)](https://sansan0.github.io/TrendRadar) [![Docker](https://img.shields.io/badge/Docker-Deployment-2496ED?style=flat-square&logo=docker&logoColor=white)](https://hub.docker.com/r/wantcat/trendradar) [![MCP Support](https://img.shields.io/badge/MCP-AI_Analysis-FF6B6B?style=flat-square&logo=ai&logoColor=white)](https://modelcontextprotocol.io/) [![AI Analysis Push](https://img.shields.io/badge/AI-Analysis_Push-FF6B6B?style=flat-square&logo=openai&logoColor=white)](#) [![AI Smart Filter](https://img.shields.io/badge/AI-Smart_News_Filter-9B59B6?style=flat-square&logo=openai&logoColor=white)](#)
**[中文](README.md)** | **English**
> This project is designed to be lightweight and easy to deploy
## 📑 Quick Navigation > 💡 **Click the links below** to jump to the corresponding section. Start with "**Quick Start**" for deployment, see "**Configuration Guide**" for detailed customization
| | | | |:---:|:---:|:---:| | [🚀 **Quick Start**](#-quick-start) | [AI Analysis](#-ai-analysis) | [⚙️ **Configuration Guide**](#configuration-guide) | | [Docker Deployment](#6-docker-deployment) | [MCP Clients](#-mcp-clients) | [📝 **Changelog**](#-changelog) | | [🎯 **Core Features**](#-core-features) | [☕ **Support Project**](#-support-project) | [📚 **Related Projects**](#-related-projects) |

- Thanks to **stargazers**, your stars and forks are the best support for open source 😍
👉 Click to view Acknowledgments (Angel Round Honor Roll 🔥73+🔥 supporters) ### Acknowledgments to Early Supporters > 💡 **Special Note**: > > 1. **About the List**: The table below records supporters from the early stage (Angel Round) of the project. Due to the manual nature of statistics in the early days, **there may be omissions or incomplete records. If anyone was missed, it was unintentional, and we ask for your kind understanding**. > 2. **Future Plan**: To focus limited energy back on code development and feature iteration, **this list will no longer be manually maintained as of today**. > > Whether your name is on the list or not, your every bit of support is the cornerstone that allows TrendRadar to be where it is today. 🙏 ### Infrastructure Support Thanks to **GitHub** for providing free infrastructure, which is the biggest prerequisite for this project to run conveniently with **one-click fork**. ### Data Support This project uses the API from [newsnow](https://github.com/ourongxing/newsnow) to fetch multi-platform data. Special thanks to the author for providing this service. After communication, the author indicated no concerns about server pressure, but this is based on their goodwill and trust. Please everyone: - **Visit the [newsnow project](https://github.com/ourongxing/newsnow) and give it a star** - When deploying with Docker, please control the frequency reasonably and avoid being overly greedy ### Promotion Support > Thanks to the following platforms and individuals for recommendations (in chronological order) - [Appinn (小众软件)](https://mp.weixin.qq.com/s/fvutkJ_NPUelSW9OGK39aA) - Open source software recommendation platform - [LinuxDo Community](https://linux.do/) - Tech enthusiasts community - [Ruan Yifeng's Weekly](https://github.com/ruanyf/weekly) - Influential tech weekly in Chinese tech circle ### Community Support > Thanks to **financial supporters**. Your generosity has transformed into snacks and drinks beside my keyboard, accompanying every iteration of this project > > **Return of "One-Yuan Appreciation"**: > With the release of v5.0.0, the project enters a new phase. To support growing API costs and caffeine consumption, the "One-Yuan Appreciation" channel is now reopened. Every bit of your kindness translates into Tokens and motivation in the code world. 🚀 [Support Now](#-support-project) | Supporter | Amount (CNY) | Date | Note | | :-------: | :----------: | :--: | :--: | | D*5 | 1.8 * 3 | 2025.11.24 | | | *鬼 | 1 | 2025.11.17 | | | *超 | 10 | 2025.11.17 | | | R*w | 10 | 2025.11.17 | Great agent work! | | J*o | 1 | 2025.11.17 | Thanks for open source | | *晨 | 8.88 | 2025.11.16 | Nice project | | *海 | 1 | 2025.11.15 | | | *德 | 1.99 | 2025.11.15 | | | *疏 | 8.8 | 2025.11.14 | Great project | | M*e | 10 | 2025.11.14 | Open source is not easy | | **柯 | 1 | 2025.11.14 | | | *云 | 88 | 2025.11.13 | Good project | | *W | 6 | 2025.11.13 | | | *凯 | 1 | 2025.11.13 | | | 对*. | 1 | 2025.11.13 | Thanks for TrendRadar | | s*y | 1 | 2025.11.13 | | | **翔 | 10 | 2025.11.13 | Wish I found it earlier | | *韦 | 9.9 | 2025.11.13 | TrendRadar is awesome | | h*p | 5 | 2025.11.12 | Support Chinese open source | | c*r | 6 | 2025.11.12 | | | a*n | 5 | 2025.11.12 | | | 。*c | 1 | 2025.11.12 | Thanks for sharing | | ... | ... | ... | **(More 50+ supporters)** |

## 🪄 Sponsors
> **Sponsorship Open**

### ❤️ Find it useful? Support TrendRadar > If TrendRadar has captured value for you, give it some fuel to keep evolving > > Any amount is welcome; even 1 RMB is a gesture of encouragement for open source. Feel free to leave a note with your donation (´▽`ʃ♡ƪ)
| WeChat Pay | Alipay | | --- | --- | | WeChat Pay | Alipay |
### 🤝 Attribution & Secondary Development If you utilize the core code or draw inspiration from the logic of this project, **it would be greatly appreciated** if you could acknowledge the source in your README or documentation and include a link to this repository. This contributes to the sustainable maintenance of the project and the growth of the community. Thank you for your respect and support! ❤️ ### 💬 Feedback & Community * **GitHub Issues**: Best for specific technical issues. Please provide complete information (screenshots, error logs, etc.) to help locate the problem quickly. * **WeChat Official Account**: It is recommended to leave comments under relevant articles. If you need to ask questions in the background, **liking/recommending** the article first is the best "icebreaker," and I can feel your appreciation (´▽`ʃ♡ƪ). > **Friendly Reminder**: > This project is for open-source sharing, not a commercial product. Treat the author as a friend, not customer service, for better communication efficiency!
| Follow on WeChat | | --- | | |

## 📝 Changelog >**📌 Check Latest Updates**: **[Original Repository Changelog](https://github.com/sansan0/TrendRadar?tab=readme-ov-file#-changelog)**: - **Tip**: Check [Changelog] to understand specific [Features] ### 2026/03/12 - v6.5.0 - **AI Smart News Filtering**: No more manual keyword setup! Describe your interests in everyday language in `ai_interests.txt` (e.g., "I want AI and renewable energy news"), and AI automatically extracts tags, scores every headline, and only pushes what truly matters to you. If AI filtering encounters issues, it auto-falls back to keyword matching — push delivery never stops - **Per-Period Filter Strategy & Interests**: Each time period in Timeline can now independently choose its filtering method and what topics to focus on. For example: mornings use a "tech keyword list" for quick filtering, evenings switch to "finance AI interests" for in-depth AI filtering — same system, different content at different times - **AI Analysis Independent from Push Mode**: AI analysis scope can differ from push content. For example: push only delivers new items (avoiding repeated notifications), while AI analyzes the full day's news (capturing complete trends). Each time period can also set its own AI analysis mode - **AI Filter Token Savings**: Previously analyzed news won't be re-processed; when you edit your interests, AI auto-evaluates the change magnitude — minor tweaks only update affected tags, major changes trigger full reclassification - **Multi-File Config & Tag Isolation**: Custom keyword files go in `config/custom/keyword/`, AI interest files go in `config/custom/ai/` — tags from different files are fully isolated and independent - **AI Translation Precision Control**: Independently toggle translation for hotlist, RSS, and standalone sections; regions with display turned off are automatically skipped, saving tokens - **Remote Storage Batch Upload**: Multiple write operations are batched and submitted to cloud in one go, reducing API call count - **Per-Group Display Limit**: New `max_news_per_keyword` controls max items shown per keyword/tag group, preventing a single hot topic from filling the entire push - **Time Period Conflict Detection**: Overlapping time periods are automatically detected — system alerts you to fix the config, preventing unexpected behavior - Various bug fixes ### 2026/02/09 - mcp-v4.0.0 - **🔥 Push any AI message to all channels**: Send AI-generated content to Feishu, DingTalk, Telegram, Email and all 9 channels with one call — Markdown auto-adapts to each platform's native format - **New format guide tool**: `get_channel_format_guide` tells AI what each channel supports and its limitations, so generated content looks great everywhere - **Smart batch splitting**: Long messages auto-split per channel byte limits (Feishu 30KB, DingTalk 20KB, etc.), reads config from config.yaml - **Fixed channel detection**: ntfy no longer falsely reported as "configured" due to default server URL - **Code reuse**: Batch utilities now imported from trendradar core instead of duplicated
👉 Click to expand: Historical Updates ### 2026/02/09 - v6.0.0 > **Breaking Change**: Config file upgrade (config.yaml 2.0.0), old `push_window` and `analysis_window` configs are no longer compatible, please refer to the new config.yaml for migration - **Unified Scheduling System**: New `timeline.yaml` — one config to control when to crawl / push / AI analyze - **5 Preset Templates**: `always_on` (24/7, default), `morning_evening` (morning & evening summary), `office_hours` (work hours), `night_owl` (late night), `custom` (fully customizable); you can also add your own templates under `presets:` — just use a unique key, then set it in config.yaml - **Flexible Time Period Config**: Supports weekday/weekend differentiation, cross-midnight time periods, per-period once deduplication - **Visual Config Editor**: - New `timeline.yaml` editor tab, alongside config.yaml / frequency_words.txt - Preset mode card selector: click to switch, auto-syncs config.yaml's `schedule.preset` - Week view timeline: 7 days × 24 hours horizontal bars, color-coded for push/analysis/crawl status - Interactive controls: toggles, dropdowns, time pickers — right-side changes sync to left-side YAML in real time - Week mapping dropdown: dynamically populated from day plans, configure scheduling by drag and click - **AI Prompt Stability Overhaul** (ai_analysis_prompt.txt v2.0.0): - Formatting rules extracted from JSON values into a standalone spec section, reducing AI output format inconsistencies - JSON template simplified: field descriptions shortened to one sentence + word limit - Removed Markdown from system prompt to align with the "no Markdown" instruction - All JSON fields declared optional — missing any field won't cause errors, improving fault tolerance - **Standalone Source AI Summaries** (`ai_analysis.include_standalone`): - New independent toggle: when enabled, AI generates a concise summary for each standalone source - Decoupled from display: AI can analyze full hotlist data without enabling standalone display in push notifications - Supports both trending platforms and RSS feeds, including rank/time/trajectory data - Trajectory analysis linked with `include_rank_timeline`: uses trajectory data for deep trend analysis when enabled, falls back to rank-based summary when disabled - New `standalone_summaries` JSON field ("Source Snapshot"), all notification channels adapted for rendering ### 2026/01/28 - v5.5.0 > Like the MCP feature, I'm not creating a separate repo for this tool either — it's pure frontend, so bundling it together - Added visual configuration editor for TrendRadar ### 2026/02/02 - mcp-v3.2.0 - **New read_article tool**: Read a single article body via Jina AI Reader (Markdown format) - **New read_articles_batch tool**: Batch read multiple articles (up to 5, auto rate-limited) - **Recommended workflow**: `search_news(query="keyword", include_url=True)` → `read_article(url=...)` to read article body - **Docs update**: README-MCP-FAQ.md and README-MCP-FAQ-EN.md added Q19-Q20 for article reading ### 2026/01/23 - v5.4.0 - Added independent control for AI analysis mode, options: follow_report | daily | current | incremental - Added time window control for AI analysis, supporting custom execution periods and daily frequency limits - Added configuration file version management function - Fixed several bugs ### 2026/01/19 - v5.3.0 > **Major Refactor: AI Module Migration to LiteLLM** - **Unified AI Interface**: Replaced manual implementation with LiteLLM, supporting 100+ AI providers - **Simplified Configuration**: Removed `provider` field, now using `model: "provider/model_name"` format - **New Features**: Auto-retry (`num_retries`), fallback models (`fallback_models`) - **Configuration Changes**: - `ai.provider` → Removed (merged into model) - `ai.base_url` → `ai.api_base` - `AI_PROVIDER` environment variable → Removed - `AI_BASE_URL` environment variable → `AI_API_BASE` - **Model Format Examples**: - DeepSeek: `deepseek/deepseek-chat` - OpenAI: `openai/gpt-4o` - Gemini: `gemini/gemini-2.5-flash` - Anthropic: `anthropic/claude-3-5-sonnet` ### 2026/01/17 - v5.2.0 > See config.yaml for details **🌐 AI Translation** - **Multi-language Translation**: Translate push content to any language - **Batch Translation**: Smart batch processing to reduce API calls - **Custom Prompts**: Customize translation style **🔧 Configuration Optimization** - **Standalone AI Model Config**: Analysis and translation share model config - **Unified Region Switches**: Unified management of push region display - **Custom Region Order**: Customize display order of each region **✨ AI Analysis Enhancement** - **AI Analysis Embedded in HTML**: Analysis results directly embedded in HTML reports, used by email notifications - **Rich Style AI Section**: Gradient blue card layout, clearly separating analysis dimensions - **Ranking Timeline Support**: AI can access precise ranking at each crawl time point - **Section Reorganization (7→4)**: Consolidated into Core Trends, Sentiment & Controversy, Signals & Anomalies, Outlook & Strategy **🔧 Multi-Model Adaptation** - **Universal Parameter Passthrough**: Pass any advanced parameters to API - **Gemini Adaptation**: Native parameter support with relaxed safety settings **🐛 Bug Fixes** - Fixed various known issues, improved system stability ### 2026/01/10 - mcp-v3.0.0~v3.1.5 - **Breaking Change**: All tool return values unified to `{success, summary, data, error}` structure - **Async Consistency**: All 21 tool functions wrapped with `asyncio.to_thread()` for sync calls - **MCP Resources**: Added 4 resources (platforms, rss-feeds, available-dates, keywords) - **RSS Enhancement**: `get_latest_rss` supports multi-day queries (days param), cross-date URL deduplication - **Regex Matching Fix**: `get_trending_topics` supports `/pattern/` regex syntax and `display_name` - **Cache Optimization**: Added `make_cache_key()` function with param sorting + MD5 hash for consistency - **New check_version Tool**: Check TrendRadar and MCP Server version updates simultaneously ### 2026/01/10 - v5.0.0 > **Dev Anecdote**: > A salute to a certain 'C' model provider that accompanied me for over two years, only to slap me with `"This organization has been disabled"` right after I renewed my subscription. **✨ "Five Major Sections" Content Refactoring** This update refactors the push message structure into five distinct core sections: 1. **📊 Trending News**: Aggregated trending topics from across the web, precisely filtered by your keywords. 2. **📰 RSS Feeds**: Your personalized subscription content, supporting keyword-based grouping. 3. **🆕 New Items**: Real-time capture of brand new trending topics since the last run (marked with 🆕). 4. **📋 Independent Display**: Complete trending lists or RSS feeds from specified platforms, **completely unaffected by keyword filtering**. 5. **✨ AI Analysis**: Deep insights driven by AI, including trend overview, popularity trends, and **critically important** sentiment analysis. **✨ AI Smart Analysis Push Feature** - **AI Analysis Integration**: Use AI models to deeply analyze push content, automatically generate trending insights, keyword analysis, cross-platform correlation, potential impact assessment - **Sentiment Analysis**: New deep sentiment recognition to accurately capture positive, negative, controversial, or concerned public opinions (v5.0.0 key enhancement) - **Multi AI Provider Support**: Supports DeepSeek (default, cost-effective), OpenAI, Google Gemini, and any OpenAI-compatible API - **Two Push Modes**: `only_analysis` (AI analysis only), `both` (push both) - **Custom Prompts**: Customize AI analysis role and output format via `config/ai_analysis_prompt.txt` - **Multi-dimensional Analysis**: AI can analyze ranking changes, trending duration, cross-platform performance, trend prediction ### 2026/01/02 - v4.7.0 - **Fix RSS HTML Display**: Fixed RSS data format mismatch causing rendering issues, now displays correctly grouped by keyword - **New Regex Syntax**: Keyword config supports `/pattern/` regex syntax, solves English substring mismatch issues (e.g., `ai` matching `training`) [📖 View Syntax Details](#keyword-basic-syntax) - **New Display Name Syntax**: Use `=> alias` to give complex regex a friendly name, cleaner push notifications (e.g., `/\bai\b/ => AI Related`) - **Can't Write Regex?** README now includes AI prompt guide - just tell ChatGPT/Gemini/DeepSeek what you want to match ### 2025/12/30 - mcp-v2.0.0 - **Architecture Refactoring**: Removed TXT support, unified to SQLite database - **RSS Query**: Added `get_latest_rss`, `search_rss`, `get_rss_feeds_status` - **Unified Search**: `search_news` supports `include_rss` parameter to search both trending and RSS ### 2026/01/01 - v4.6.0 - **Fix RSS HTML Display**: Merged RSS content into trending HTML page, grouped by source - **New display_mode Config**: Support `keyword` (group by keyword) and `platform` (group by platform) display modes ### 2025/12/30 - v4.5.0 - **RSS Feed Support**: Added RSS/Atom feed crawling, keyword-based grouping and statistics (consistent with trending format) - **Storage Structure Refactoring**: Flattened directory structure `output/{type}/{date}.db` - **Unified Sorting Config**: `sort_by_position_first` affects both trending and RSS - **Config Structure Refactoring**: `config.yaml` reorganized into 7 logical groups (app, report, notification, storage, platforms, rss, advanced) with clearer config paths ### 2025/12/26 - mcp-v1.2.0 **MCP Module Update - Optimized toolset, added aggregation & comparison features, merged redundant tools:** - Added `aggregate_news` tool - Cross-platform news deduplication and aggregation - Added `compare_periods` tool - Period comparison analysis (week-over-week/month-over-month) - Merged `find_similar_news` + `search_related_news_history` → `find_related_news` - Enhanced `get_trending_topics` - Added `auto_extract` mode for automatic trending extraction - Fixed miscellaneous bugs - Updated README-MCP-FAQ.md documentation in both Chinese and English (Q1-Q18) ### 2025/12/20 - v4.0.3 - Added URL normalization to fix duplicate push issues caused by dynamic parameters (e.g., Weibo's `band_rank`) - Fixed incremental mode detection logic to correctly identify historical titles ### 2025/12/13 - mcp-v1.1.0 **MCP Module Update:** - Adapted for v4.0.0, while maintaining compatibility with v3.x data. - Added storage sync tools: - `sync_from_remote`: Pull data from remote storage to local - `get_storage_status`: Get storage configuration and status - `list_available_dates`: List available dates in local/remote storage ### 2025/12/17 - v4.0.1 - StorageManager adds push record proxy methods - S3 client switches to virtual-hosted style for better compatibility (supports Tencent Cloud COS and more services) ### 2025/12/13 - v4.0.0 **🎉 Major Update: Comprehensive Refactoring of Storage and Core Architecture** - **Multi-Storage Backend Support**: Introduced a brand new storage module supporting local SQLite and remote cloud storage (S3-compatible protocols, e.g., Cloudflare R2), adaptable to GitHub Actions, Docker, and local environments. - **Database Structure Optimization**: Refactored SQLite database table structures to improve data efficiency and query performance. - **Enhanced Features**: Implemented date format standardization, data retention policies, timezone configuration support, and optimized time display. Fixed remote storage data persistence issues to ensure accurate data merging. - **Cleanup and Compatibility**: Removed most legacy compatibility code and unified data storage and retrieval methods. ### 2025/12/03 - v3.5.0 **🎉 Core Feature Enhancements** 1. **Multi-Account Push Support** - All push channels (Feishu, DingTalk, WeWork, Telegram, ntfy, Bark, Slack) support multiple account configuration - Use semicolon `;` to separate multiple accounts, e.g., `FEISHU_WEBHOOK_URL=url1;url2` - Automatic validation for paired configurations (e.g., Telegram's token and chat_id) 2. **Push Region Configuration** - Customize display order of all regions via `display.region_order` (v5.2.0, replaces `reverse_content_order`) - Control visibility of each region via `display.regions` (hotlist, new items, RSS, standalone, AI analysis) 3. **Global Filter Keywords** - Added `[GLOBAL_FILTER]` region marker for filtering unwanted content globally - Use cases: Filter ads, marketing, low-quality content, etc. **🐳 Docker Dual-Path HTML Generation Optimization** - **Bug Fix**: Resolved issue where `index.html` could not sync to host in Docker environment - **Dual-Path Generation**: Daily summary HTML is generated to two locations simultaneously - `index.html` (project root): For GitHub Pages access - `output/index.html`: Accessible on host via Docker Volume mount - **Compatibility**: Ensures web reports are accessible in Docker, GitHub Actions, and local environments **🐳 Docker MCP Image Support** - Added independent MCP service image `wantcat/trendradar-mcp` - Supports Docker deployment of AI analysis features via HTTP interface (port 3333) - Dual-container architecture: News push service and MCP service run independently, can be scaled and restarted separately - See [Docker Deployment - MCP Service](#6-docker-deployment) for details **🌐 Web Server Support** - Added built-in web server for browser access to generated reports - Control via `manage.py` commands: `docker exec -it trendradar python manage.py start_webserver` - Access URL: `http://localhost:8080` (port configurable) - Security features: Static file service, directory restriction, localhost binding - Supports both auto-start and manual control modes **📖 Documentation Optimization** - Added [Report Configuration](#7-report-configuration) section: report-related parameter details - Added [Push Window Configuration](#8-push-window-configuration) section: push_window configuration tutorial - Added [Execution Frequency Configuration](#9-execution-frequency-configuration) section: Cron expression explanation and common examples - Added [Multi-Account Push Configuration](#10-multiple-account-push-configuration) section: multi-account push configuration details - Optimized all configuration sections: Unified "Configuration Location" instructions - Simplified Quick Start configuration: Three core files at a glance - Optimized [Docker Deployment](#6-docker-deployment) section: Added image description, recommended git clone deployment, reorganized deployment methods **🔧 Upgrade Instructions**: - **GitHub Fork Users**: Update `main.py`, `config/config.yaml` (Added multi-account push support, existing single-account configuration unaffected) - **Docker Users**: Update `.env`, `docker-compose.yml` or set environment variables `REVERSE_CONTENT_ORDER`, `MAX_ACCOUNTS_PER_CHANNEL` - **Multi-Account Push**: New feature, disabled by default, existing single-account configuration unaffected ### 2025/11/28 - v3.4.1 **🔧 Format Optimization** 1. **Bark Push Enhancement** - Bark now supports Markdown rendering - Enabled native Markdown format: bold, links, lists, code blocks, etc. - Removed plain text conversion to fully utilize Bark's native rendering capabilities 2. **Slack Format Precision** - Use dedicated mrkdwn format for batch content processing - Improved byte size estimation accuracy (avoid message overflow) - Optimized link format: `` and bold syntax: `*text*` 3. **Performance Improvement** - Format conversion completed during batching process, avoiding secondary processing - Accurate message size estimation reduces send failure rate **🔧 Upgrade Instructions**: - **GitHub Fork Users**: Update `main.py`，`config.yaml` ### 2025/11/26 - mcp-v1.0.3 **MCP Module Update:** - Added date parsing tool resolve_date_range to resolve AI model date calculation inconsistencies - Support natural language date expression parsing (this week, last 7 days, last month, etc.) - Tool count increased from 13 to 14 ### 2025/11/25 - v3.4.0 **🎉 Added Slack Push Support** 1. **Team Collaboration Push Channel** - Supports Slack Incoming Webhooks (globally popular team collaboration tool) - Centralized message management, suitable for team-shared trending news - Supports mrkdwn format (bold, links, etc.) 2. **Multiple Deployment Methods** - GitHub Actions: Configure `SLACK_WEBHOOK_URL` Secret - Docker: Environment variable `SLACK_WEBHOOK_URL` - Local: `config/config.yaml` configuration file > 📖 **Detailed Configuration Tutorial**: [Quick Start - Slack Push](#-quick-start) - Optimized the one-click installation experience for setup-windows.bat and setup-windows-en.bat **🔧 Upgrade Instructions**: - **GitHub Fork Users**: Update `main.py`, `config/config.yaml`, `.github/workflows/crawler.yml` ### 2025/11/24 - v3.3.0 **🎉 Added Bark Push Support** 1. **iOS Exclusive Push Channel** - Supports Bark push (based on APNs, iOS platform) - Free, open-source, clean, efficient, ad-free - Supports both official server and self-hosted server 2. **Multiple Deployment Methods** - GitHub Actions: Configure `BARK_URL` Secret - Docker: Environment variable `BARK_URL` - Local: `config/config.yaml` configuration file > 📖 **Detailed Configuration Tutorial**: [Quick Start - Bark Push](#-quick-start) **🐛 Bug Fix** - Fixed issue where `ntfy_server_url` in `config.yaml` was ignored ([#345](https://github.com/sansan0/TrendRadar/issues/345)) **🔧 Upgrade Instructions**: - **GitHub Fork Users**: Update `main.py`, `config/config.yaml`, `.github/workflows/crawler.yml` ### 2025/11/23 - v3.2.0 **🎯 New Advanced Customization Features** 1. **Keyword Sorting Priority Configuration** - Two sorting strategies: Popularity first vs Config order first - For different use cases: Hot topic tracking or personalized focus 2. **Display Count Precise Control** - Global config: Unified limit for all keywords - Individual config: Use `@number` syntax to set specific limits - Effectively control push length, highlight key content > 📖 **Detailed Tutorial**: [Keyword Configuration - Advanced Settings](#keyword-advanced-settings) **🔧 Upgrade Instructions**: - **GitHub Fork Users**: Update `main.py`, `config/config.yaml` ### 2025/11/22 - v3.1.1 - **Fixed data anomaly crash issue**: Resolved `'float' object has no attribute 'lower'` error encountered by some users in GitHub Actions environment - Added dual protection mechanism: Filter invalid titles (None, float, empty strings) at data acquisition stage, with type checking at function call sites - Enhanced system stability to ensure normal operation even when data sources return abnormal formats **Upgrade Instructions** (GitHub Fork Users): - Required update: `main.py` - Recommended: Use minor version upgrade method - copy and replace the file above ### 2025/11/18 - mcp-v1.0.2 **MCP Module Update:** - Fix issue where today's news query may return articles from past dates ### 2025/11/20 - v3.1.0 - **Added Personal WeChat Push Support**: WeWork application can push to personal WeChat without installing WeWork APP - Supports two message formats: `markdown` (WeWork group bot) and `text` (personal WeChat app) - Added `WEWORK_MSG_TYPE` environment variable configuration, supporting GitHub Actions, Docker, docker compose and other deployment methods - `text` mode automatically strips Markdown syntax for clean plain text push - See "Personal WeChat Push" configuration in Quick Start **Upgrade Instructions** (GitHub Fork Users): - Required updates: `main.py`, `config/config.yaml` - Optional update: `.github/workflows/crawler.yml` (if using GitHub Actions) - Recommended: Use minor version upgrade method - copy and replace the files above ### 2025/11/12 - v3.0.5 - Fixed email sending SSL/TLS port configuration logic error - Optimized email service providers (QQ/163/126) to default use port 465 (SSL) - **Added Docker environment variable support**: Core config items (`enable_crawler`, `report_mode`, `push_window`, etc.) support override via environment variables, solving config file modification issues for NAS users (see [🐳 Docker Deployment](#-docker-deployment) chapter) ### 2025/10/26 - mcp-v1.0.1 **MCP Module Update:** - Fixed date query parameter passing error - Unified time parameter format for all tools ### 2025/10/31 - v3.0.4 - Solved Feishu error due to overly long push content, implemented batch pushing ### 2025/10/23 - v3.0.3 - Expanded ntfy error message display range ### 2025/10/21 - v3.0.2 - Fixed ntfy push encoding issue ### 2025/10/20 - v3.0.0 **Major Update - AI Analysis Feature Launched** ✨ - **Core Features**: - New MCP (Model Context Protocol) based AI analysis server - 13 smart analysis tools: basic query, smart search, advanced analysis, system management - Natural language interaction: Query and analyze news data through conversation - Multi-client support: Claude Desktop, Cherry Studio, Cursor, Cline, etc. - **Analysis Capabilities**: - Topic trend analysis (popularity tracking, lifecycle, viral detection, trend prediction) - Data insights (platform comparison, activity stats, keyword co-occurrence) - Sentiment analysis, similar news finding, smart summary generation - Historical related news search, multi-mode search - **Update Note**: - This is an independent AI analysis feature, does not affect existing push functionality - Optional use, no need to upgrade existing deployment ### 2025/10/15 - v2.4.4 - **Updates**: - Fixed ntfy push encoding issue + 1 - Fixed push time window judgment issue - **Upgrade Note**: - Recommended minor version upgrade ### 2025/10/10 - v2.4.3 > Thanks to [nidaye996](https://github.com/sansan0/TrendRadar/issues/98) for discovering the UX issue - **Updates**: - Refactored "Silent Push Mode" naming to "Push Time Window Control", improving feature comprehension - Clarified push time window as optional additional feature, can work with three push modes - Improved comments and documentation, making feature positioning clearer - **Upgrade Note**: - This is just refactoring, upgrade optional ### 2025/10/8 - v2.4.2 - **Updates**: - Fixed ntfy push encoding issue - Fixed missing config file issue - Optimized ntfy push effect - Added GitHub Pages image segmented export feature - **Upgrade Note**: - Recommend major version update ### 2025/10/2 - v2.4.0 **Added ntfy Push Notification** - **Core Features**: - Supports ntfy.sh public service and self-hosted servers - **Use Cases**: - Suitable for privacy-conscious users (supports self-hosting) - Cross-platform push (iOS, Android, Desktop, Web) - No account registration needed (public servers) - Open-source and free (MIT License) - **Upgrade Note**: - Recommend major version update ### 2025/09/26 - v2.3.2 - Fixed email notification config check being missed ([#88](https://github.com/sansan0/TrendRadar/issues/88)) **Fix Description**: - Solved the issue where system still prompted "No webhook configured" even with correct email notification setup ### 2025/09/22 - v2.3.1 - **Added email push feature**, supports sending trending news reports to email - **Smart SMTP Recognition**: Auto-detects Gmail, QQ Mail, Outlook, NetEase Mail and 10+ email service providers - **Beautiful HTML Format**: Email content uses same HTML format as web version, well-formatted, mobile-adapted - **Batch Sending Support**: Supports multiple recipients, separated by commas - **Custom SMTP**: Can customize SMTP server and port - Fixed Docker build network connection issue **Usage Notes**: - Use cases: Suitable for users needing email archiving, team sharing, scheduled reports - Supported emails: Gmail, QQ Mail, Outlook/Hotmail, 163/126 Mail, Sina Mail, Sohu Mail, etc. **Upgrade Note**: - This update has many changes, if upgrading, recommend major version upgrade ### 2025/09/17 - v2.2.0 - Added one-click save news as image feature, easily share trending topics you care about **Usage Notes**: - Use case: After enabling web version feature (GitHub Pages) - How to use: Open webpage on phone or PC, click "Save as Image" button at top - Actual effect: System auto-creates beautiful image of current news report, saves to phone album or desktop - Sharing convenience: Directly send this image to friends, Moments, or work groups, letting others see your discovered important info ### 2025/09/13 - v2.1.2 - Solved DingTalk push capacity limit causing news push failure (using batch push) ### 2025/09/04 - v2.1.1 - Fixed Docker unable to run properly on certain architectures - Officially released official Docker image wantcat/trendradar, supports multi-architecture - Optimized Docker deployment process, can use quickly without local build ### 2025/08/30 - v2.1.0 **Core Improvements**: - **Push Logic Optimization**: Changed from "push every execution" to "controllable push within time window" - **Time Window Control**: Can set push time range, avoid non-work hour disturbance - **Push Frequency Options**: Supports single push or multiple pushes within time window **Upgrade Note**: - This feature is disabled by default, need to manually enable push time window control in config.yaml - Upgrade requires updating both main.py and config.yaml files ### 2025/08/27 - v2.0.4 - This version is not a bug fix, but an important reminder - Please keep webhooks properly, do not make public, do not make public, do not make public - If you deployed this project on GitHub via fork, please put webhooks in GitHub Secret, not config.yaml - If you already exposed webhooks or put them in config.yaml, suggest deleting and regenerating ### 2025/08/06 - v2.0.3 - Optimized GitHub Pages web version effect, convenient for mobile use ### 2025/07/28 - v2.0.2 - Refactored code - Solved version number easily being missed for modification ### 2025/07/27 - v2.0.1 **Fixed Issues**: 1. Docker shell script line ending as CRLF causing execution exception issue 2. frequency_words.txt being empty causing news sending also empty logic issue - After fix, when you choose frequency_words.txt empty, will **push all news**, but limited by message push size, please adjust as follows - Option 1: Turn off mobile push, only choose GitHub Pages deployment (this is the way to get most complete info, will re-sort all platform trending by your **custom trending algorithm**) - Option 2: Reduce push platforms, prioritize **WeWork** or **Telegram**, these two pushes I made batch push feature (because batch push affects push experience, and only these two platforms give very little push capacity, so had to make batch push feature, but at least can ensure complete info) - Option 3: Can combine with Option 2, mode choose current or incremental can effectively reduce one-time push content ### 2025/07/17 - v2.0.0 **Major Refactoring**: - Config management refactoring: All configs now managed through `config/config.yaml` file (main.py I still didn't split, convenient for you to copy and upgrade) - Run mode upgrade: Supports three modes - `daily` (daily summary), `current` (current rankings), `incremental` (incremental monitoring) - Docker support: Complete Docker deployment solution, supports containerized operation **Config File Description**: - `config/config.yaml` - Main config file (application settings, crawler config, notification config, platform config, etc.) - `config/frequency_words.txt` - Keyword config (monitoring vocabulary settings) ### 2025/07/09 - v1.4.1 **New Feature**: Added incremental push (configure FOCUS_NEW_ONLY at top of main.py), this switch only cares about new topics not sustained heat, only sends notification when new content appears. **Fixed Issue**: Under certain circumstances, some news containing special symbols caused occasional formatting exceptions. ### 2025/06/23 - v1.3.0 WeWork and Telegram push messages have length limits, I adopted splitting messages for pushing. Development docs see [WeWork](https://developer.work.weixin.qq.com/document/path/91770) and [Telegram](https://core.telegram.org/bots/api) ### 2025/06/21 - v1.2.1 Before this version, not only main.py needs copy replacement, crawler.yml also needs you to copy replacement https://github.com/sansan0/TrendRadar/blob/master/.github/workflows/crawler.yml ### 2025/06/19 - v1.2.0 > Thanks to Claude Research for organizing various platform APIs, helping me quickly complete platform adaptation (although code is more redundant~ 1. Supports Telegram, WeWork, DingTalk push channels, supports multi-channel config and simultaneous push ### 2025/06/18 - v1.1.0 > **200 stars⭐** reached, continue celebrating with everyone~ 1. Important update, added weight, news you see now is hottest most concerned appearing at top 2. Updated documentation usage, because recently updated many features, and previous usage docs I was lazy wrote simple (see ⚙️ frequency_words.txt complete configuration tutorial below) ### 2025/06/16 - v1.0.0 1. Added project new version update reminder, default on, if want to turn off, can change "FEISHU_SHOW_VERSION_UPDATE": True to False in main.py ### 2025/06/13+14 1. Removed compatibility code, students who forked before, directly copying code will show exception on same day (will recover normal next day) 2. Feishu and html bottom added new news display ### 2025/06/09 **100 stars⭐** reached, writing small feature to celebrate frequency_words.txt file added **required word** feature, using + sign 1. Required word syntax as follows: Tang Monk or Pig must both appear in title, will be included in push news ``` +Tang Monk +Pig ``` 2. Filter word priority higher: If title filter word matches Tang Monk reciting sutras, then even if required word has Tang Monk, also not display ``` +Tang Monk !Tang Monk reciting sutras ``` ### 2025/06/02 1. **Webpage** and **Feishu messages** support phone directly jumping to detailed news 2. Optimized display effect + 1 ### 2025/05/26 1. Feishu message display effect optimized

## ✨ Core Features ### **Multi-Platform Trending News Aggregation** - Zhihu (知乎) - Douyin (抖音) - Bilibili Hot Search - Wallstreetcn (华尔街见闻) - Tieba (贴吧) - Baidu Hot Search - Yicai (财联社) - Thepaper (澎湃新闻) - Ifeng (凤凰网) - Toutiao (今日头条) - Weibo (微博) Default monitoring of 11 mainstream platforms, with support for adding custom platforms. > 💡 For detailed configuration, see [Configuration Guide - Platform Configuration](#1-platform-configuration) ### **RSS Feed Support** (v4.5.0 New) Supports RSS/Atom feed crawling, keyword-based grouping and statistics (consistent with trending format): - **Unified Format**: RSS and trending use the same keyword matching and display format - **Simple Config**: Add RSS sources directly in `config.yaml` - **Merged Push**: Trending and RSS are merged into a single notification - **Freshness Filter**: Automatically filters out articles older than a specified number of days to avoid repeated pushes. Supports both global default and per-feed settings > 💡 RSS uses the same `frequency_words.txt` for keyword filtering as trending ### **Visual Configuration Editor** A web-based graphical configuration interface — no need to manually edit YAML files. Complete all configuration changes and exports through simple forms. 👉 **Try it online**: [https://sansan0.github.io/TrendRadar/](https://sansan0.github.io/TrendRadar/) Visual Configuration Editor ### **Smart Push Strategies** **Three Push Modes**: | Mode | Target Users | Push Feature | |------|--------------|--------------| | **Daily Summary** (daily) | Managers/Regular Users | Push all matched news of the day (includes previously pushed) | | **Current Rankings** (current) | Content Creators | Push current ranking matches (continuously ranked news appear each time) | | **Incremental Monitor** (incremental) | Traders/Investors | Push only new content, zero duplication | > 💡 **Quick Selection Guide:** > - Don't want duplicate news → Use `incremental` > - Want complete ranking trends → Use `current` > - Need daily summary reports → Use `daily` > > For detailed comparison and configuration, see [Configuration Guide - Push Mode Details](#3-push-mode-details) **Additional Features** (Optional): | Feature | Description | Default | |---------|-------------|---------| | **Scheduling System** | Per-day-of-week scheduling: assign different time periods, push modes, and AI analysis strategies to each day (Mon–Sun). **Each period can independently set its filter method (keyword/AI) and interest focus**, enabling different content at different times. 5 built-in presets (always_on / morning_evening / office_hours / night_owl / custom), or define your own. Supports weekday vs weekend differentiation, cross-midnight periods, per-period once-only dedup, and overlap conflict detection (v6.0.0 + v6.5.0) | morning_evening | | **Content Order Configuration** | Use `display.region_order` to adjust display order of all regions (hotlist, new items, RSS, standalone, AI analysis); use `display.regions` to toggle each region on/off (v5.2.0) | See config | | **Display Mode Switch** | `keyword`=group by keyword, `platform`=group by platform (v4.6.0 new) | keyword | > 💡 For detailed configuration, see [Configuration Guide - Report Configuration](#7-report-configuration) and [Configuration Guide - Scheduling System](#8-when-will-i-receive-pushes) ### **Precise Content Filtering** Set personal keywords (e.g., AI, BYD, Education Policy) to receive only relevant trending news, filtering out noise. > 💡 **Basic Configuration**: [Keyword Configuration - Basic Syntax](#keyword-basic-syntax) > > 💡 **Advanced Configuration**: [Keyword Configuration - Advanced Settings](#keyword-advanced-settings) > > 💡 You can also skip filtering and receive all trending news (leave frequency_words.txt empty) ### **AI Smart News Filtering** (v6.5.0 New) Describe your interests in natural language and let AI automatically classify news — replacing traditional keyword matching - **Natural Language Interests**: Write your focus areas in everyday language in `ai_interests.txt`, no keyword syntax to learn - **Two-Stage Smart Processing**: AI first extracts structured tags from interest descriptions, then batch-classifies and scores news against those tags - **Score Threshold Control**: Fine-tune push quality with `ai_filter.min_score` — only highly relevant news gets delivered - **Auto Fallback**: Automatically falls back to keyword matching if AI filtering fails, ensuring uninterrupted push delivery - **Smart Tag Updates**: When interests change, AI evaluates the change magnitude to decide incremental or full reclassification - **Flexible Switching**: `filter.method` supports `keyword` (default) and `ai` modes, Timeline can override per time period - **Per-Period Personalization**: Different time periods can use different keyword files or AI interest descriptions. For example: mornings use a "tech keyword list" for quick filtering, evenings switch to "finance interests" for AI deep filtering ```yaml # config.yaml quick enable example filter: method: ai # keyword (default) | ai ai_filter: min_score: 6 # Minimum push score threshold (1-10) ``` > 💡 AI filtering shares model config with AI analysis/translation — just configure `ai.api_key` once ### **Trending Analysis** Real-time tracking of news popularity changes helps you understand not just "what's trending" but "how trends evolve." - **Timeline Tracking**: Records complete time span from first to last appearance - **Popularity Changes**: Tracks ranking changes and appearance frequency across time periods - **New Detection**: Real-time identification of emerging topics, marked with 🆕 - **Continuity Analysis**: Distinguishes between one-time hot topics and continuously developing news - **Cross-Platform Comparison**: Same news across different platforms, showing media attention differences > 💡 Push format reference: [Configuration Guide - Push Format Reference](#5-push-format-reference) ### **Personalized Trending Algorithm** No longer controlled by platform algorithms, TrendRadar reorganizes all trending searches > 💡 Weight adjustment guide: [Configuration Guide - Advanced Configuration](#4-advanced-configuration---hotspot-weight-adjustment) ### **Multi-Channel Multi-Account Push** Supports **WeWork** (+ WeChat push solution), **Feishu**, **DingTalk**, **Telegram**, **Email**, **ntfy**, **Bark**, **Slack**, **Generic Webhook** (connect to Discord, IFTTT, or any platform) — messages delivered directly to phone and email. > 💡 For detailed configuration, see [Configuration Guide - Multi-Account Push Configuration](#10-multiple-account-push-configuration) ### **AI Multi-Language Translation** (v5.2.0 New) Translate push content into any language, breaking language barriers — whether reading domestic trends or subscribing to international news via RSS, access everything in your native language - **One-Click Translation**: Set `ai_translation.enabled: true` and target language in `config.yaml` - **Multi-Language Support**: Supports English, Korean, Japanese, French, and any other language - **Smart Batch Processing**: Automatically batches translations to reduce API calls and save costs - **Custom Style**: Customize translation style and terminology via `ai_translation_prompt.txt` - **Shared Model Config**: Shares the `ai` config section with AI analysis feature ```yaml # config.yaml quick enable example ai_translation: enabled: true language: "English" # Target translation language ``` > 💡 Translation shares model config with AI analysis — just configure `ai.api_key` once to use both features **RSS Source References**: Here are some RSS feed collections for your reference - [awesome-tech-rss](https://github.com/tuan3w/awesome-tech-rss) - Tech, startup, and programming blogs & media - [awesome-rss-feeds](https://github.com/plenaryapp/awesome-rss-feeds) - Mainstream news media RSS from countries worldwide > ⚠️ Some international media content may involve sensitive topics that AI models might refuse to translate. Please filter subscription sources based on your actual needs ### **Flexible Storage Architecture (v4.0.0 Major Update)** **Multi-Backend Support**: - **Remote Cloud Storage**: GitHub Actions environment default, supports S3-compatible protocols (R2/OSS/COS, etc.), data stored in cloud, keeping repository clean - **Local SQLite**: Traditional SQLite database, stable and efficient (Docker/local deployment) - **Auto Selection**: Auto-selects appropriate backend based on runtime environment > 💡 For storage configuration details, see [Configuration Details - Storage Configuration](#11-storage-configuration-v400-new) ### **Multi-Platform Deployment** - **GitHub Actions**: Cloud automated operations (7-day check-in cycle + remote cloud storage) - **Docker Deployment**: Supports multi-architecture containerized operation - **Local Running**: Python environment direct execution ### **AI Analysis Push (v5.0.0 New)** Use AI models to deeply analyze push content, automatically generate trending insights report - **Smart Analysis**: Automatically analyze trending topics, keyword popularity, cross-platform correlation, potential impact - **Multi Provider**: Built on LiteLLM unified interface, supports 100+ AI providers (DeepSeek, OpenAI, Gemini, Anthropic, local Ollama, etc.), with automatic fallback model switching - **Independent Analysis Mode**: AI analysis scope can differ from push content — push only new items (less noise), while AI analyzes the full day's news (complete trend picture) - **Flexible Push**: Choose original content only, AI analysis only, or both - **Custom Prompts**: Customize analysis perspective via `config/ai_analysis_prompt.txt` > 💡 Detailed configuration tutorial: [Let AI help me analyze hot topics](#12-let-ai-help-me-analyze-hot-topics) ### **Independent Display Section (v5.0.0 New)** Provide complete trending display for specified platforms, unaffected by keyword filtering - **Full Trending**: Specified platforms show complete trending list, for users who want to see full rankings - **RSS Independent Display**: RSS source content can be fully displayed, not limited by keywords - **AI Deep Analysis**: Independently enable AI trend analysis on full hotlists, without displaying in push - **Flexible Configuration**: Support configuring display platforms, RSS sources, max count > 💡 Detailed configuration tutorial: [Report Configuration - Independent Display](#7-report-configuration) ### **AI Smart Analysis (v3.0.0 New)** AI conversational analysis system based on MCP (Model Context Protocol), enabling deep data mining with natural language. > **💡 Usage Tip**: AI features require local news data support > - Project includes test data for immediate feature experience > - Recommend deploying the project yourself to get more real-time data > > See [AI Analysis](#-ai-analysis) for details ### **Web Deployment** After running, the `index.html` generated in the root directory is the complete news report page. > **Deployment**: Click **Use this template** to create your repository, then deploy to Cloudflare Pages or GitHub Pages. > > **💡 Tip**: Enable GitHub Pages for an online URL. Go to Settings → Pages to enable. [Preview Effect](https://sansan0.github.io/TrendRadar/) > > ⚠️ The GitHub Actions auto-storage feature has been discontinued (this approach caused excessive load on GitHub servers, affecting platform stability). ### **Reduce APP Dependencies** Transform from "algorithm recommendation captivity" to "actively getting the information you want" **Target Users:** Investors, content creators, PR professionals, news-conscious general users **Typical Scenarios:** Stock investment monitoring, brand sentiment tracking, industry trend watching, lifestyle news gathering | Web Effect (Email Push) | Feishu Push Effect | AI Analysis Push Effect | |:---:|:---:|:---:| | ![Web Effect](_image/github-pages.png) | ![Feishu Push Effect](_image/feishu.jpg) | ![AI Analysis Push Effect](_image/ai.jpg) |
## 🚀 Quick Start > **Reminder**: You should first **[check the latest official documentation](https://github.com/sansan0/TrendRadar?tab=readme-ov-file)** to ensure the configuration steps are up to date. ### Choose the Deployment Method That Fits You #### 🅰️ Option A: Docker Deployment (Recommended 🔥) * **Features**: More stable than GitHub Actions * **Best for**: Users with their own server, NAS, or an always-on PC 👉 **[Jump to Docker Deployment Tutorial](#6-docker-deployment)** #### 🅱️ Option B: GitHub Actions Deployment (This Chapter ⬇️) * **Features**: Data is stored in **Remote Cloud Storage** (no longer written to Git repo) * **Storage**: Configure cloud storage service (e.g. Cloudflare R2, Alibaba Cloud OSS, Tencent Cloud COS, etc.) * **Note**: Requires periodic check-in renewal (every 7 days) ### 1️⃣ Step 1: Get project code Click the green **[Use this template]** button in the upper right corner of this repository → select "Create a new repository". > ⚠️ Note: > - Any mention of "Fork" in this document can be understood as "Use this template" > - Using Fork may cause runtime issues, see [Issue #606](https://github.com/sansan0/TrendRadar/issues/606)
### 2️⃣ Step 2: Setup GitHub Secrets In your Forked repository, go to `Settings` > `Secrets and variables` > `Actions` > `New repository secret` **📌 Important Instructions (Please Read Carefully):** - **One Name for One Secret**: For each configuration item, click the "New repository secret" button once and fill in a pair of "Name" and "Secret" - **Cannot See Value After Saving is Normal**: For security reasons, after saving, you can only see the Name when re-editing, but not the Secret value - **DO NOT Create Custom Names**: The Secret Name must **strictly use** the names listed below (e.g., `WEWORK_WEBHOOK_URL`, `FEISHU_WEBHOOK_URL`, etc.). Do not modify or create new names arbitrarily, or the system will not recognize them - **Can Configure Multiple Platforms**: The system will send notifications to all configured platforms **Configuration Example:** GitHub Secrets Configuration Example As shown above, each row is a configuration item: - **Name**: Must use the fixed names listed in the expanded sections below (e.g., `WEWORK_WEBHOOK_URL`) - **Secret (Value)**: Fill in the actual content obtained from the corresponding platform (e.g., Webhook URL, Token, etc.)
👉 Click to expand: WeWork Bot (Simplest and fastest configuration)
**GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `WEWORK_WEBHOOK_URL` (Please copy and paste this name, do not type manually to avoid typos) - **Secret (Value)**: Your WeWork bot Webhook address
**Bot Setup Steps:** #### Mobile Setup: 1. Open WeWork App → Enter target internal group chat 2. Click "…" button at top right → Select "Message Push" 3. Click "Add" → Name input "TrendRadar" 4. Copy Webhook address, click save, paste the copied content into GitHub Secret above #### PC Setup Process Similar
👉 Click to expand: Personal WeChat Push (Based on WeWork app, push to personal WeChat)
> This solution is based on WeWork's plugin mechanism. The push style is plain text (no markdown format), but it can push directly to personal WeChat without installing WeWork App. **GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `WEWORK_WEBHOOK_URL` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Your WeWork app Webhook address - **Name**: `WEWORK_MSG_TYPE` (Please copy and paste this name, do not type manually) - **Secret (Value)**: `text`
**Setup Steps:** 1. Complete the WeWork bot Webhook setup above 2. Add `WEWORK_MSG_TYPE` Secret with value `text` 3. Follow the image below to link personal WeChat 4. After configuration, WeWork App can be deleted from phone **Notes**: - Uses the same Webhook address as WeWork bot - Difference is message format: `text` for plain text, `markdown` for rich text (default) - Plain text format will automatically remove all markdown syntax (bold, links, etc.)
👉 Click to expand: Feishu Bot (Message display is relatively friendly)
**Note**: If **AI Analysis** is enabled, Feishu push notifications may occasionally (approx. 5% probability) experience a few minutes of delay. This is likely due to the platform's internal compliance auditing for AI-generated content. **GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `FEISHU_WEBHOOK_URL` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Your Feishu bot Webhook address (link starts with https://www.feishu.cn/flow/api/trigger-webhook/********)
Two methods available, **Method 1** is simpler, **Method 2** is more complex (but stable push) Method 1 discovered and suggested by **ziventian**, thanks to them. Default is personal push, group push can be configured via [#97](https://github.com/sansan0/TrendRadar/issues/97) **Method 1:** > For some users, additional operations needed to avoid "System Error". Need to search for the bot on mobile and enable Feishu bot application (suggestion from community, can refer) 1. Open in PC browser https://botbuilder.feishu.cn/home/my-command 2. Click "New Bot Command" 3. Click "Select Trigger", scroll down, click "Webhook Trigger" 4. Now you'll see "Webhook Address", copy this link to local notepad temporarily, continue with next steps 5. In "Parameters" put the following content, then click "Done" ```json { "message_type": "text", "content": { "text": "{{Content}}" } } ``` 6. Click "Select Action" > "Send via Official Bot" 7. Message title fill "TrendRadar Trending Monitor" 8. Most critical part, click + button, select "Webhook Trigger", then arrange as shown in image ![Feishu Bot Config Example](_image/feishu.png) 9. After configuration, put Webhook address from step 4 into GitHub Secrets `FEISHU_WEBHOOK_URL`
**Method 2:** 1. Open in PC browser https://botbuilder.feishu.cn/home/my-app 2. Click "New Bot Application" 3. After entering the created application, click "Process Design" > "Create Process" > "Select Trigger" 4. Scroll down, click "Webhook Trigger" 5. Now you'll see "Webhook Address", copy this link to local notepad temporarily, continue with next steps 6. In "Parameters" put the following content, then click "Done" ```json { "message_type": "text", "content": { "text": "{{Content}}" } } ``` 7. Click "Select Action" > "Send Feishu Message", check "Group Message", then click the input box below, click "Groups I Manage" (if no group, you can create one in Feishu app) 8. Message title fill "TrendRadar Trending Monitor" 9. Most critical part, click + button, select "Webhook Trigger", then arrange as shown in image ![Feishu Bot Config Example](_image/feishu.png) 10. After configuration, put Webhook address from step 5 into GitHub Secrets `FEISHU_WEBHOOK_URL`
👉 Click to expand: DingTalk Bot
**GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `DINGTALK_WEBHOOK_URL` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Your DingTalk bot Webhook address
**Bot Setup Steps:** 1. **Create Bot (PC Only)**: - Open DingTalk PC client, enter target group chat - Click group settings icon (⚙️) → Scroll down to find "Bot" and click - Select "Add Bot" → "Custom" 2. **Configure Bot**: - Set bot name - **Security Settings**: - **Custom Keywords**: Set "Trending" or "热点" 3. **Complete Setup**: - Check service terms agreement → Click "Done" - Copy the obtained Webhook URL - Put URL into GitHub Secrets `DINGTALK_WEBHOOK_URL` **Note**: Mobile can only receive messages, cannot create new bots.
👉 Click to expand: Telegram Bot
**GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `TELEGRAM_BOT_TOKEN` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Your Telegram Bot Token - **Name**: `TELEGRAM_CHAT_ID` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Your Telegram Chat ID **Note**: Telegram requires **two** Secrets, please click "New repository secret" button twice to add them separately
**Bot Setup Steps:** 1. **Create Bot**: - Search `@BotFather` in Telegram (note case, has blue verification checkmark, shows ~37849827 monthly users, this is official, beware of fake accounts) - Send `/newbot` command to create new bot - Set bot name (must end with "bot", easily runs into duplicate names, so think creatively) - Get Bot Token (format like: `123456789:AAHfiqksKZ8WmR2zSjiQ7_v4TMAKdiHm9T0`) 2. **Get Chat ID**: **Method 1: Via Official API** - First send a message to your bot - Visit: `https://api.telegram.org/bot/getUpdates` - Find the number in `"chat":{"id":number}` in returned JSON **Method 2: Using Third-Party Tool** - Search `@userinfobot` and send `/start` - Get your user ID as Chat ID 3. **Configure to GitHub**: - `TELEGRAM_BOT_TOKEN`: Fill in Bot Token from step 1 - `TELEGRAM_CHAT_ID`: Fill in Chat ID from step 2
👉 Click to expand: Email Push (Supports all mainstream email providers)
- Note: To prevent email bulk sending abuse, current bulk sending allows all recipients to see each other's email addresses. - If you haven't configured email sending before, not recommended to try > ⚠️ **Important Configuration Dependency**: Email push requires HTML report file. Make sure `storage.formats.html` is set to `true` in `config/config.yaml`: > ```yaml > storage: > formats: > sqlite: true > txt: false > html: true # Must be enabled, otherwise email push will fail > ``` > If set to `false`, email push will report error: `Error: HTML file does not exist or not provided: None`
**GitHub Secret Configuration (⚠️ Name must match exactly):** - **Name**: `EMAIL_FROM` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Sender email address - **Name**: `EMAIL_PASSWORD` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Email password or authorization code - **Name**: `EMAIL_TO` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Recipient email address (multiple separated by comma, or can be same as EMAIL_FROM to send to yourself) - **Name**: `EMAIL_SMTP_SERVER` (Optional, please copy and paste this name) - **Secret (Value)**: SMTP server address (leave empty for auto-detection) - **Name**: `EMAIL_SMTP_PORT` (Optional, please copy and paste this name) - **Secret (Value)**: SMTP port (leave empty for auto-detection) **Note**: Email push requires at least **3 required** Secrets (EMAIL_FROM, EMAIL_PASSWORD, EMAIL_TO), the last two are optional
**Supported Email Providers** (Auto-detect SMTP config): | Provider | Domain | SMTP Server | Port | Encryption | |----------|--------|-------------|------|-----------| | **Gmail** | gmail.com | smtp.gmail.com | 587 | TLS | | **QQ Mail** | qq.com | smtp.qq.com | 465 | SSL | | **Outlook** | outlook.com | smtp-mail.outlook.com | 587 | TLS | | **Hotmail** | hotmail.com | smtp-mail.outlook.com | 587 | TLS | | **Live** | live.com | smtp-mail.outlook.com | 587 | TLS | | **163 Mail** | 163.com | smtp.163.com | 465 | SSL | | **126 Mail** | 126.com | smtp.126.com | 465 | SSL | | **Sina Mail** | sina.com | smtp.sina.com | 465 | SSL | | **Sohu Mail** | sohu.com | smtp.sohu.com | 465 | SSL | | **189 Mail** | 189.cn | smtp.189.cn | 465 | SSL | | **Aliyun Mail** | aliyun.com | smtp.aliyun.com | 465 | TLS | | **Yandex Mail** | yandex.com | smtp.yandex.com | 465 | TLS | | **iCloud Mail** | icloud.com | smtp.mail.me.com | 587 | SSL | > **Auto-detect**: When using above emails, no need to manually configure `EMAIL_SMTP_SERVER` and `EMAIL_SMTP_PORT`, system auto-detects. > > **Feedback Notice**: > - If you successfully test with **other email providers**, please open an [Issue](https://github.com/sansan0/TrendRadar/issues) to let us know, we'll add to support list > - If above email configurations are incorrect or unusable, please also open an [Issue](https://github.com/sansan0/TrendRadar/issues) for feedback to help improve the project > > **Special Thanks**: > - Thanks to [@DYZYD](https://github.com/DYZYD) for contributing 189 Mail (189.cn) configuration and completing self-send-receive testing ([#291](https://github.com/sansan0/TrendRadar/issues/291)) > - Thanks to [@longzhenren](https://github.com/longzhenren) for contributing Aliyun Mail (aliyun.com) configuration and completing testing ([#344](https://github.com/sansan0/TrendRadar/issues/344)) > - Thanks to [@ACANX](https://github.com/ACANX) for contributing Yandex Mail (yandex.com) configuration and completing testing ([#663](https://github.com/sansan0/TrendRadar/issues/663)) > - Thanks to [@Sleepy-Tianhao](https://github.com/Sleepy-Tianhao) for contributing iCloud Mail (icloud.com) configuration and completing testing ([#728](https://github.com/sansan0/TrendRadar/issues/728)) **Common Email Settings:** #### QQ Mail: 1. Login QQ Mail web version → Settings → Account 2. Enable POP3/SMTP service 3. Generate authorization code (16-letter code) 4. `EMAIL_PASSWORD` fill authorization code, not QQ password #### Gmail: 1. Enable two-step verification 2. Generate app-specific password 3. `EMAIL_PASSWORD` fill app-specific password #### 163/126 Mail: 1. Login web version → Settings → POP3/SMTP/IMAP 2. Enable SMTP service 3. Set client authorization code 4. `EMAIL_PASSWORD` fill authorization code
**Advanced Configuration**: If auto-detect fails, manually configure SMTP: - `EMAIL_SMTP_SERVER`: Like smtp.gmail.com - `EMAIL_SMTP_PORT`: Like 587 (TLS) or 465 (SSL)
**Multiple Recipients (note: English comma separator)**: - EMAIL_TO="user1@example.com,user2@example.com,user3@example.com"
👉 Click to expand: ntfy Push (Open-source, free, self-hostable)
**Two Usage Methods:** ### Method 1: Free Use (Recommended for Beginners) 🆓 **Features**: - ✅ No account registration, use immediately - ✅ 250 messages/day (enough for 90% users) - ✅ Topic name is "password" (need to choose hard-to-guess name) - ⚠️ Messages unencrypted, not for sensitive info, but suitable for our non-sensitive project info **Quick Start:** 1. **Download ntfy App**: - Android: [Google Play](https://play.google.com/store/apps/details?id=io.heckel.ntfy) / [F-Droid](https://f-droid.org/en/packages/io.heckel.ntfy/) - iOS: [App Store](https://apps.apple.com/us/app/ntfy/id1625396347) - Desktop: Visit [ntfy.sh](https://ntfy.sh) 2. **Subscribe to Topic** (choose a hard-to-guess name): ``` Suggested format: trendradar-{your initials}-{random numbers} Cannot use Chinese ✅ Good example: trendradar-zs-8492 ❌ Bad example: news, alerts (too easy to guess) ``` 3. **Configure GitHub Secret (⚠️ Name must match exactly)**: - **Name**: `NTFY_TOPIC` (Please copy and paste this name, do not type manually) - **Secret (Value)**: Fill in your subscribed topic name - **Name**: `NTFY_SERVER_URL` (Optional, please copy and paste this name) - **Secret (Value)**: Leave empty (default uses ntfy.sh) - **Name**: `NTFY_TOKEN` (Optional, please copy and paste this name) - **Secret (Value)**: Leave empty **Note**: ntfy requires at least 1 required Secret (NTFY_TOPIC), the last two are optional 4. **Test**: ```bash curl -d "Test message" ntfy.sh/your-topic-name ``` --- ### Method 2: Self-Hosting (Complete Privacy Control) 🔒 **Target Users**: Have server, pursue complete privacy, strong technical ability **Advantages**: - ✅ Completely open-source (Apache 2.0 + GPLv2) - ✅ Complete data self-control - ✅ No restrictions - ✅ Zero cost **Docker One-Click Deploy**: ```bash docker run -d \ --name ntfy \ -p 80:80 \ -v /var/cache/ntfy:/var/cache/ntfy \ binwiederhier/ntfy \ serve --cache-file /var/cache/ntfy/cache.db ``` **Configure TrendRadar**: ```yaml NTFY_SERVER_URL: https://ntfy.yourdomain.com NTFY_TOPIC: trendradar-alerts # Self-hosting can use simple name NTFY_TOKEN: tk_your_token # Optional: Enable access control ``` **Subscribe in App**: - Click "Use another server" - Enter your server address - Enter topic name - (Optional) Enter login credentials --- **FAQ:**
Q1: Is the free version enough? 250 messages/day is enough for most users. With 30-minute crawl intervals, about 48 pushes/day, completely sufficient.
Q2: Is the Topic name really secure? If you choose a random, sufficiently long name (like `trendradar-zs-8492-news`), brute force is nearly impossible: - ntfy has strict rate limiting (1 request/second) - 64 character choices (A-Z, a-z, 0-9, _, -) - 10 random characters have 64^10 possibilities (would take years to crack)
--- **Recommended Choice:** | User Type | Recommended | Reason | |-----------|-------------|--------| | Regular Users | Method 1 (Free) | Simple, fast, enough | | Technical Users | Method 2 (Self-Host) | Complete control, unlimited | | High-Frequency Users | Method 3 (Paid) | Check official website | **Related Links:** - [ntfy Official Docs](https://docs.ntfy.sh/) - [Self-Hosting Tutorial](https://docs.ntfy.sh/install/) - [GitHub Repository](https://github.com/binwiederhier/ntfy)
👉 Click to expand: Bark Push (iOS exclusive, clean & efficient)
**GitHub Secret Configuration (⚠️ Name must be exact):** - **Name**: `BARK_URL` (copy and paste this name, don't type manually) - **Secret**: Your Bark push URL
**Bark Introduction:** Bark is a free open-source push tool for iOS platform, featuring simplicity, speed, and no ads. **Usage Methods:** ### Method 1: Use Official Server (Recommended for beginners) 🆓 1. **Download Bark App**: - iOS: [App Store](https://apps.apple.com/us/app/bark-customed-notifications/id1403753865) 2. **Get Push URL**: - Open Bark App - Copy the push URL displayed on the home page (format: `https://api.day.app/your_device_key`) - Configure the URL to GitHub Secrets as `BARK_URL` ### Method 2: Self-Hosted Server (Complete Privacy Control) 🔒 **Suitable for**: Users with servers, pursuing complete privacy, strong technical skills **Docker One-Click Deployment**: ```bash docker run -d \ --name bark-server \ -p 8080:8080 \ finab/bark-server ``` **Configure TrendRadar**: ```yaml BARK_URL: http://your-server-ip:8080/your_device_key ``` --- **Notes:** - ✅ Bark uses APNs push, max 4KB per message - ✅ Supports automatic batch sending, no worry about long messages - ✅ Push format is plain text (automatically removes Markdown syntax) - ⚠️ Only supports iOS platform **Related Links:** - [Bark Official Website](https://bark.day.app/) - [Bark GitHub Repository](https://github.com/Finb/Bark) - [Bark Server Self-Hosting Tutorial](https://github.com/Finb/bark-server)
👉 Click to expand: Slack Push
**GitHub Secret Configuration (⚠️ Name must be exact):** - **Name**: `SLACK_WEBHOOK_URL` (copy and paste this name, don't type manually) - **Secret**: Your Slack Incoming Webhook URL
**Slack Introduction:** Slack is a team collaboration tool, Incoming Webhooks can push messages to Slack channels. **Setup Steps:** ### Step 1: Create Slack App 1. **Visit Slack API Page**: - Open https://api.slack.com/apps?new_app=1 - Login to your Slack workspace if not logged in 2. **Choose Creation Method**: - Click **"From scratch"** 3. **Fill in App Information**: - **App Name**: Enter app name (e.g., `TrendRadar` or `Hot News Monitor`) - **Workspace**: Select your workspace from dropdown - Click **"Create App"** button ### Step 2: Enable Incoming Webhooks 1. **Navigate to Incoming Webhooks**: - Find and click **"Incoming Webhooks"** in left menu 2. **Enable Feature**: - Find **"Activate Incoming Webhooks"** toggle - Switch from `OFF` to `ON` - Page will auto-refresh showing new configuration options ### Step 3: Generate Webhook URL 1. **Add New Webhook**: - Scroll to page bottom - Click **"Add New Webhook to Workspace"** button 2. **Select Target Channel**: - System will show authorization page - Select channel to receive messages from dropdown (e.g., `#hot-news`) - ⚠️ For private channels, must join the channel first 3. **Authorize App**: - Click **"Allow"** button to complete authorization - System will auto-redirect back to config page ### Step 4: Copy and Save Webhook URL 1. **View Generated URL**: - In "Webhook URLs for Your Workspace" section - You'll see the newly generated Webhook URL - Format: `https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX` 2. **Copy URL**: - Click **"Copy"** button on the right of URL - Or manually select and copy URL 3. **Configure to TrendRadar**: - **GitHub Actions**: Add URL to GitHub Secrets as `SLACK_WEBHOOK_URL` - **Local Testing**: Fill URL in `config/config.yaml` `slack_webhook_url` field - **Docker Deployment**: Add URL to `docker/.env` file as `SLACK_WEBHOOK_URL` variable --- **Notes:** - ✅ Supports Markdown format (auto-converts to Slack mrkdwn) - ✅ Supports automatic batch sending (4KB per batch) - ✅ Suitable for team collaboration, centralized message management - ⚠️ Webhook URL contains secret key, never make it public **Message Format Preview:** ``` *[Batch 1/2]* 📊 *Trending Topics Statistics* 🔥 *[1/3] AI ChatGPT* : 2 articles 1. [Baidu Hot] 🆕 ChatGPT-5 Official Release *[1]* - 09:15 (1 time) 2. [Toutiao] AI Chip Stocks Surge *[3]* - [08:30 ~ 10:45] (3 times) ``` **Related Links:** - [Slack Incoming Webhooks Official Docs](https://api.slack.com/messaging/webhooks) - [Slack API App Management](https://api.slack.com/apps)
👉 Click to expand: Generic Webhook Push (Supports Discord, Matrix, IFTTT, etc.)
**GitHub Secret Configuration (⚠️ Name must be exact):** - **Name**: `GENERIC_WEBHOOK_URL` (copy and paste this name, don't type manually) - **Secret**: Your Webhook URL - **Name**: `GENERIC_WEBHOOK_TEMPLATE` (optional, copy and paste this name) - **Secret**: JSON template string, supports `{title}` and `{content}` placeholders
**Generic Webhook Introduction:** Generic Webhook supports any platform that accepts HTTP POST requests, including but not limited to: - **Discord**: Push to channels via Webhook - **Matrix**: Push via Webhook bridge - **IFTTT**: Trigger automation workflows - **Custom Services**: Any custom service supporting Webhooks **Configuration Examples:** ### Discord Configuration 1. **Get Webhook URL**: - Go to Discord Server Settings → Integrations → Webhooks - Create new Webhook, copy URL 2. **Configure Template**: ```json {"content": "{content}"} ``` 3. **GitHub Secret Configuration**: - `GENERIC_WEBHOOK_URL`: Discord Webhook URL - `GENERIC_WEBHOOK_TEMPLATE`: `{"content": "{content}"}` ### Custom Templates Templates support two placeholders: - `{title}` - Message title - `{content}` - Message content **Template Examples**: ```json # Default format (used when empty) {"title": "{title}", "content": "{content}"} # Discord format {"content": "{content}"} # Custom format {"text": "{content}", "username": "TrendRadar"} ``` --- **Notes:** - ✅ Supports Markdown format (same as WeWork format) - ✅ Supports automatic batch sending - ✅ Supports multi-account configuration (use `;` separator) - ⚠️ Template must be valid JSON format - ⚠️ Different platforms have different message format requirements, please refer to target platform documentation
> ⚠️ Note: > - For first deployment, suggest completing **GitHub Secrets** configuration first (choose one push platform), then jump to [Step 3] to test push success. > - **Don't modify** `config/config.yaml` and `frequency_words.txt` temporarily, adjust these configs after push test succeeds as needed.
### 3️⃣ Step 3: Manual Test News Push > ⚠️ Reminder: > - Complete Step 1-2 first, then test immediately! Test success first, then adjust configuration (Step 4) as needed. > - IMPORTANT: Enter your own forked project, not this project! **How to find your Actions page**: - **Method 1**: Open your forked project homepage, click the **Actions** tab at the top - **Method 2**: Direct access `https://github.com/YourUsername/TrendRadar/actions` **Example comparison**: - ❌ Author's project: `https://github.com/sansan0/TrendRadar/actions` - ✅ Your project: `https://github.com/YourUsername/TrendRadar/actions` **Testing steps**: 1. Enter your project's Actions page 2. Find **"Hot News Crawler"** and click in - If you don't see this text, refer to [#109](https://github.com/sansan0/TrendRadar/issues/109) to solve 3. Click **"Run workflow"** button on the right to run 4. Wait about 1 minute, messages will be pushed to your configured platform > ⚠️ Note: > - Don't test too frequently to avoid triggering GitHub Actions limits > - After clicking Run workflow, you need to **refresh the browser page** to see the new run record
### 4️⃣ Step 4: Configuration Notes (Optional) The default configuration is ready to use. If you need personalized adjustments, just understand the following files: | File | Purpose | |------|---------| | `config/config.yaml` | Main config file: push mode, time window, platform list, hotspot weights, etc. | | `config/frequency_words.txt` | Keyword file: set your interested keywords, filter push content | | `.github/workflows/crawler.yml` | Execution frequency: control how often to run (⚠️ modify carefully) | 👉 **Detailed Configuration Tutorial**: [Configuration Guide](#configuration-guide)
### 5️⃣ Step 5: GitHub Actions Check-In & Remote Cloud Storage **v4.0.0 Important Change**: Introduced the "Activity Detection" mechanism; GitHub Actions need periodic check-ins to maintain operation. - **Running Cycle**: Valid for **7 days**—service will automatically suspend when countdown ends. - **Renewal Method**: Manually trigger the "Check In" workflow on the Actions page to reset the 7-day validity period. - **Operation Path**: `Actions` → `Check In` → `Run workflow` - **Design Philosophy**: - If you forget for 7 days, maybe you don't really need it. Letting it stop is a digital detox, freeing you from the constant impact. - GitHub Actions is a valuable public computing resource. The check-in mechanism aims to prevent wasted computing cycles, ensuring resources are allocated to truly active users who need them. Thank you for your understanding and support. --- **You can also choose NOT to configure remote cloud storage**, but then you will be in **Lite Mode** with some advanced features unavailable. **Two Deployment Modes Comparison:** | Mode | Configuration Required | Features | |------|------------------------|----------| | **Lite Mode** | No storage configuration needed | Real-time crawling + Keyword filtering + Multi-channel push | | **Full Mode** | Configure remote cloud storage | Lite Mode + New detection + Trend tracking + Incremental push + AI analysis | **Lite Mode Description**: - ✅ Available: Real-time news crawling, keyword filtering, hotspot weight ranking, current list push - ❌ Not Available: New news detection (🆕), trend tracking, incremental mode, daily summary accumulation, MCP AI analysis **Full Mode Description**: Configure remote cloud storage to unlock all features. Continue with the configuration below.
👉 Click to expand: Remote Cloud Storage Configuration (Determines Feature Completeness) (Optional)
**⚠️ Prerequisites for Cloudflare R2 Configuration:** According to Cloudflare platform rules, enabling R2 requires binding a payment method. * **Purpose**: Verify identity only, **no charges will be incurred**. * **Payment**: Supports dual-currency credit cards or regional PayPal. * **Usage**: R2's free tier (10GB storage/month) is sufficient for this project's daily operation, no need to worry about costs. --- **GitHub Secret Configuration:** **Required Configuration (4 items):** | Name | Secret (Value) Description | |------|----------------------------| | `S3_BUCKET_NAME` | Bucket name (e.g., `trendradar-data`) | | `S3_ACCESS_KEY_ID` | Access key ID | | `S3_SECRET_ACCESS_KEY` | Access key | | `S3_ENDPOINT_URL` | S3 API endpoint (e.g., R2: `https://.r2.cloudflarestorage.com`) | **Optional Configuration:** | Name | Secret (Value) Description | |------|----------------------------| | `S3_REGION` | Region (default `auto`, some providers may require specification) | > 💡 **More storage configuration options**: See [Storage Configuration Details](#11-storage-configuration-v400-new)
**How to Get Credentials (Using Cloudflare R2 as Example):** 1. Visit [Cloudflare Dashboard](https://dash.cloudflare.com/) and log in 2. Select `R2` in left menu → Click `Create Bucket` → Enter name (e.g., `trendradar-data`) 3. Click `Manage R2 API Tokens` at top right → `Create API Token` 4. Select `Object Read & Write` permission → After creation, it will display `Access Key ID` and `Secret Access Key` 5. Endpoint URL can be found in bucket details page (format: `https://.r2.cloudflarestorage.com`)

### 6️⃣ Step 6: Enable AI Analysis Push This is a core feature of v5.0.0, letting AI summarize and analyze news for you. Highly recommended. **Configuration Method:** Add the following to GitHub Secrets (or `.env` / `config.yaml`): - `AI_API_KEY`: Your API Key (Supports DeepSeek, OpenAI, etc.) - `AI_PROVIDER`: Provider name (e.g., `deepseek`, `openai`) That's it! No complex deployment needed. You'll see the smart analysis report in the next push.
### 7️⃣ Step 7: 🎉 Deployment Success! Congratulations! Now you can start enjoying the efficient information flow brought by TrendRadar. 💬 Many users are sharing their experiences on the official account, we look forward to your insights~ - Want to learn more tips and advanced techniques? - Need quick answers to problems? - Have great ideas to share? 👉 Follow the WeChat Official Account「**[Silicon Tea Room](#-support-project)**」, your likes and comments are the motivation for continuous updates.
### 8️⃣ Step 8: Advanced: Choose Your AI Assistant TrendRadar provides two ways to use AI to meet different needs: | Feature | ✨ AI Analysis Push (Step 6) | 🧠 AI Smart Analysis | | :--- | :--- | :--- | | **Mode** | **Passive Receipt** (Daily Report) | **Active Conversation** (Deep Research) | | **Scenario** | "What's big today?" | "Analyze AI industry changes over the past week" | | **Deployment** | Minimalist (Just add Key) | Advanced (Requires Local/Docker) | | **Client** | Mobile | PC | 👉 **Conclusion**: Start with **AI Analysis Push** for daily needs; if you are a data analyst or need deep mining, try **[MCP Smart Analysis](#-ai-analysis)**.
## ⚙️ Configuration Guide > **📖 Reminder**: This chapter provides detailed configuration explanations. Suggest completing [Quick Start](#-quick-start) basic configuration first, then refer to detailed options here as needed. ### 1. Platform Configuration
👉 Click to expand: Custom Monitoring Platforms
**Configuration Location:** `platforms` section in `config/config.yaml` This project's news data comes from [newsnow](https://github.com/ourongxing/newsnow). You can click the [website](https://newsnow.busiyi.world/), click [More], to see if there are platforms you want. For specific additions, visit [project source code](https://github.com/ourongxing/newsnow/tree/main/server/sources), based on the file names there, modify the `platforms` configuration in `config/config.yaml` file: ```yaml platforms: enabled: true # Enable trending platform crawling sources: - id: "toutiao" name: "Toutiao" - id: "baidu" name: "Baidu Hot Search" - id: "wallstreetcn-hot" name: "Wallstreetcn" # Add more platforms... ``` > 💡 **Shortcut**: If you don't know how to read source code, you can copy from others' organized [Platform Configuration Summary](https://github.com/sansan0/TrendRadar/issues/95) > ⚠️ **Note**: More platforms is not always better, suggest choosing 10-15 core platforms. Too many platforms will cause information overload and actually reduce user experience.
### 2. Keyword Configuration **Configuration Location:** `config/frequency_words.txt` Configure monitoring keywords in `frequency_words.txt` with seven syntax types, region markers, and grouping features. | Syntax Type | Symbol | Purpose | Example | Matching Logic | |------------|--------|---------|---------|----------------| | **Normal** | None | Basic matching | `Huawei` | Match any one | | **Required** | `+` | Scope limiting | `+phone` | Must include both | | **Filter** | `!` | Noise exclusion | `!ad` | Exclude if included | | **Count Limit** | `@` | Control display count | `@10` | Max 10 news (v3.2.0 new) | | **Global Filter** | `[GLOBAL_FILTER]` | Globally exclude content | See example below | Filter under any circumstances (v3.5.0 new) | | **Regex** | `/pattern/` | Precise matching | `/\bai\b/` | Match using regex (v4.7.0 new) | | **Display Name** | `=> alias` | Custom display text | `/\bai\b/ => AI Related` | Show alias in push/HTML (v4.7.0 new) | #### 2.1 Basic Syntax
👉 Click to expand: Basic Syntax Tutorial
##### 1. **Normal Keywords** - Basic Matching ```txt Huawei OPPO Apple ``` **Effect:** News containing **any one** of these words will be captured ##### 2. **Required Words** `+word` - Scope Limiting ```txt Huawei OPPO +phone ``` **Effect:** Must include both normal word **and** required word to be captured ##### 3. **Filter Words** `!word` - Noise Exclusion ```txt Apple Huawei !fruit !price ``` **Effect:** News containing filter words will be **excluded**, even if it contains keywords ##### 4. **Count Limit** `@number` - Control Display Count (v3.2.0 new) ```txt Tesla Musk @5 ``` **Effect:** Limit maximum news count for this keyword group **Priority:** `@number` > Global config > Unlimited ##### 5. **Global Filter** `[GLOBAL_FILTER]` - Globally Exclude Content (v3.5.0 new) ```txt [GLOBAL_FILTER] advertisement promotion marketing shocking clickbait [WORD_GROUPS] technology AI Huawei HarmonyOS !car ``` **Effect:** Filters news containing specified words under **any circumstances**, with **highest priority** **Use Cases:** - Filter low-quality content: shocking, clickbait, breaking news, etc. - Filter marketing content: advertisement, promotion, sponsorship, etc. - Filter specific topics: entertainment, gossip (based on needs) **Filter Priority:** Global Filter > Group Filter(`!`) > Group Matching **Region Markers:** - `[GLOBAL_FILTER]`: Global filter region, words are filtered under any circumstances - `[WORD_GROUPS]`: Keyword groups region, maintains existing syntax (`!`, `+`, `@`) - If no region markers are used, all content is treated as keyword groups (backward compatible) **Matching Examples:** ```txt [GLOBAL_FILTER] advertisement [WORD_GROUPS] technology AI ``` - ❌ "Advertisement: Latest tech product launch" ← Contains global filter word "advertisement", rejected - ✅ "Tech company launches new AI product" ← No global filter words, matches "technology" group - ✅ "AI technology breakthrough draws attention" ← No global filter words, matches "AI" in "technology" group **Important Notes:** - Use global filter words carefully to avoid over-filtering and missing valuable content - Recommended to keep global filter words under 5-15 - For group-specific filtering, prioritize using group filter words (`!` prefix) ##### 6. **Regex** `/pattern/` - Precise Matching (v4.7.0 new) Normal keywords use substring matching, which is convenient for Chinese but may cause false matches in English. For example, `ai` would match the `ai` in `training`. Use regex syntax `/pattern/` to achieve precise matching: ```txt /(? I need a Python regex to match the word "ai" but not match "ai" in "training". > Please give me the regex in `/pattern/` format without extra explanation. AI will give you something like: `/(? alias` - Custom Display Text (v4.7.0 new) Regex patterns can look unfriendly in push notifications and HTML pages. Use `=> alias` syntax to set a display name: ```txt /(? AI Related artificial intelligence ``` **Effect:** Push notifications and HTML pages show "AI Related" instead of the complex regex **Syntax Format:** ```txt # Regex + Display Name /pattern/ => Display Name /pattern/i => Display Name # Supports flags syntax (flags are ignored) /pattern/=>Display Name # Spaces around => are optional # Normal Word + Display Name deepseek => DeepSeek News ``` **Example:** ```txt # Config /(? AI Related artificial intelligence ``` | Original Config | Push/HTML Display | |----------------|-------------------| | `/(? AI Related` + `artificial intelligence` | **`AI Related`** | **Notes:** - Display name only needs to be set on the first word of a group - If multiple words have display names, the first one is used - Without display name, all words in the group are concatenated --- #### 🔗 Group Feature - Importance of Empty Lines **Core Rule:** Use **empty lines** to separate different groups, each group is independently counted ##### Example Configuration: ```txt iPhone Huawei OPPO +launch A-shares Shanghai Index Shenzhen Index +fluctuation !prediction World Cup Euro Cup Asian Cup +match ``` ##### Group Explanation and Matching Effects: **Group 1 - Phone Launches:** - Keywords: iPhone, Huawei, OPPO - Required: launch - Effect: Must include phone brand name and "launch" **Matching Examples:** - ✅ "iPhone 15 officially launched with pricing" ← Has "iPhone" + "launch" - ✅ "Huawei Mate60 series launch livestream" ← Has "Huawei" + "launch" - ✅ "OPPO Find X7 launch date confirmed" ← Has "OPPO" + "launch" - ❌ "iPhone sales hit record high" ← Has "iPhone" but missing "launch" **Group 2 - Stock Market:** - Keywords: A-shares, Shanghai Index, Shenzhen Index - Required: fluctuation - Filter: prediction - Effect: Include stock-related words and "fluctuation", but exclude "prediction" **Matching Examples:** - ✅ "A-shares major fluctuation analysis today" ← Has "A-shares" + "fluctuation" - ✅ "Shanghai Index fluctuation reasons explained" ← Has "Shanghai Index" + "fluctuation" - ❌ "Experts predict A-shares fluctuation trends" ← Has "A-shares" + "fluctuation" but contains "prediction" - ❌ "A-shares trading volume hits new high" ← Has "A-shares" but missing "fluctuation" **Group 3 - Football Events:** - Keywords: World Cup, Euro Cup, Asian Cup - Required: match - Effect: Must include cup name and "match" **Matching Examples:** - ✅ "World Cup group stage match results" ← Has "World Cup" + "match" - ✅ "Euro Cup final match time" ← Has "Euro Cup" + "match" - ❌ "World Cup tickets on sale" ← Has "World Cup" but missing "match" #### 🎯 Configuration Tips ##### 1. **From Broad to Strict Strategy** ```txt # Step 1: Start with broad keywords for testing Artificial Intelligence AI ChatGPT # Step 2: After finding mismatches, add required words Artificial Intelligence AI ChatGPT +technology # Step 3: After finding noise, add filter words Artificial Intelligence AI ChatGPT +technology !advertisement !training ``` ##### 2. **Avoid Over-Complexity** ❌ **Not Recommended:** Too many words in one group ```txt Huawei OPPO Apple Samsung vivo OnePlus Meizu +phone +launch +sales !fake !repair !second-hand ``` **Recommended:** Split into precise groups ```txt Huawei OPPO +new product Apple Samsung +launch phone sales +market ```
#### 2.2 Advanced Settings (v3.2.0 new)
👉 Click to expand: Advanced Settings Tutorial
##### Keyword Sorting Priority **Config Location:** `config/config.yaml` ```yaml report: sort_by_position_first: false # Sorting priority config ``` | Value | Sorting Rule | Use Case | |-------|-------------|----------| | `false` (default) | News count ↓ → Config position ↑ | Focus on popularity trends | | `true` | Config position ↑ → News count ↓ | Focus on personal priority | **Example:** Config order A, B, C, news count A(3), B(10), C(5) - `false`: B(10) → C(5) → A(3) - `true`: A(3) → B(10) → C(5) ##### Global Display Count Limit ```yaml report: max_news_per_keyword: 10 # Max 10 per keyword (0=unlimited) ``` **Docker Environment Variables:** ```bash SORT_BY_POSITION_FIRST=true MAX_NEWS_PER_KEYWORD=10 ``` **Combined Example:** ```yaml # config.yaml report: sort_by_position_first: true # Config order priority max_news_per_keyword: 10 # Global default max 10 per keyword ``` ```txt # frequency_words.txt Tesla Musk @20 # Key focus, show 20 (override global) Huawei # Use global config, show 10 BYD @5 # Limit to 5 ``` **Final Effect:** Display in config order: Tesla(20) → Huawei(10) → BYD(5)
### 3. Which push mode should I choose?
👉 Click to expand: Detailed Comparison of 3 Modes
**Configuration Location:** `report.mode` in `config/config.yaml` ```yaml report: mode: "daily" # Options: "daily" | "incremental" | "current" ``` #### Detailed Comparison Table | Mode | Target Users | Push Timing | Display Content | Typical Use Case | |------|----------|----------|----------|--------------| | **Daily Summary**
`daily` | 📋 Managers/Regular Users | Scheduled push (default hourly) | All matched news of the day
+ New news section | **Example**: Check all important news of the day at 6 PM
**Feature**: See full-day trend, don't miss any hot topic
**Note**: Will include previously pushed news | | **Current Rankings**
`current` | 📰 Content Creators | Scheduled push (default hourly) | Current ranking matches
+ New news section | **Example**: Track "which topics are hottest now" hourly
**Feature**: Real-time understanding of current popularity ranking changes
**Note**: Continuously ranked news appear each time | | **Incremental Monitor**
`incremental` | 📈 Traders/Investors | Push only when new | Newly appeared frequency word matches | **Example**: Monitor "Tesla", only notify when new news appears
**Feature**: Zero duplication, only see first-time news
**Suitable for**: High-frequency monitoring, avoid information disturbance | #### Actual Push Effect Example Assume you monitor "Apple" keyword, execute once per hour: | Time | daily Mode Push | current Mode Push | incremental Mode Push | |-----|--------------|----------------|-------------------| | 10:00 | News A, News B | News A, News B | News A, News B | | 11:00 | News A, News B, News C | News B, News C, News D | **Only** News C | | 12:00 | News A, News B, News C | News C, News D, News E | **Only** News D, News E | **Explanation**: - `daily`: Cumulative display of all news of the day (A, B, C all retained) - `current`: Display current ranking news (ranking changed, News D on list, News A off list) - `incremental`: **Only push newly appeared news** (avoid duplicate disturbance) #### Common Questions > **💡 Encountered this problem?** 👉 "Execute once per hour, news output in first execution still appears in next hour execution" > - **Reason**: You might have selected `daily` (Daily Summary) or `current` (Current Rankings) mode > - **Solution**: Change to `incremental` (Incremental Monitor) mode, only push new content #### ⚠️ Incremental Mode Important Notice > **Users who selected `incremental` (Incremental Monitor) mode, please note:** > > 📌 **Incremental mode only pushes when there are new matching news** > > **If you haven't received push notifications for a long time, it may be because:** > 1. No new hot topics matching your keywords in current time period > 2. Keyword configuration is too strict or too broad > 3. Too few monitoring platforms > > **Solutions:** > - Solution 1: 👉 [Optimize Keyword Configuration](#2-keyword-configuration) - Adjust keyword precision, add or modify monitoring keywords > - Solution 2: Switch push mode - Change to `current` or `daily` mode for scheduled push notifications > - Solution 3: 👉 [Add More Platforms](#1-platform-configuration) - Add more news platforms to expand information sources
### 4. How to adjust hotness algorithm?
👉 Click to expand: Customize Hotspot Weights
**Configuration Location:** `advanced.weight` section in `config/config.yaml` ```yaml advanced: weight: rank: 0.6 # Ranking weight frequency: 0.3 # Frequency weight hotness: 0.1 # Hotness weight ``` Current default configuration is balanced. #### Two Core Scenarios **Real-Time Trending Type**: ```yaml advanced: weight: rank: 0.8 # Mainly focus on ranking frequency: 0.1 # Less concern about continuity hotness: 0.1 ``` **Target Users**: Content creators, marketers, users wanting to quickly understand current hot topics **In-Depth Topic Type**: ```yaml advanced: weight: rank: 0.4 # Moderate ranking focus frequency: 0.5 # Emphasize sustained heat within the day hotness: 0.1 ``` **Target Users**: Investors, researchers, journalists, users needing deep trend analysis #### Adjustment Method 1. **Three numbers must sum to 1.0** 2. **Increase what's important**: Increase `rank` for rankings, `frequency` for continuity 3. **Suggest adjusting 0.1-0.2 at a time**, observe effects Core idea: Users pursuing speed and timeliness increase ranking weight, users pursuing depth and stability increase frequency weight.
### 5. What will the messages look like?
👉 Click to expand: Message Style Preview
#### Push Example 📊 Trending Keywords Stats 🔥 [1/3] AI ChatGPT : 2 items 1. [Baidu Hot] 🆕 ChatGPT-5 officially launched [**1**] - 09:15 (1 time) 2. [Toutiao] AI chip concept stocks surge [**3**] - [08:30 ~ 10:45] (3 times) ━━━━━━━━━━━━━━━━━━━ 📈 [2/3] BYD Tesla : 2 items 1. [Weibo] 🆕 BYD monthly sales break record [**2**] - 10:20 (1 time) 2. [Douyin] Tesla price reduction promotion [**4**] - [07:45 ~ 09:15] (2 times) ━━━━━━━━━━━━━━━━━━━ 📌 [3/3] A-shares Stock Market : 1 item 1. [Wallstreetcn] A-shares midday review [**5**] - [11:30 ~ 12:00] (2 times) 🆕 New Trending News (Total 2 items) **Baidu Hot** (1 item): 1. ChatGPT-5 officially launched [**1**] **Weibo** (1 item): 1. BYD monthly sales break record [**2**] Updated: 2025-01-15 12:30:15 #### Message Format Explanation | Format Element | Example | Meaning | Description | | ------------- | ------- | -------- | ----------- | | 🔥📈📌 | 🔥 [1/3] AI ChatGPT | Popularity Level | 🔥 High (≥10) 📈 Medium (5-9) 📌 Normal (<5) | | [Number/Total] | [1/3] | Rank Position | Current group rank among all matches | | Keyword Group | AI ChatGPT | Keyword Group | Group from config, title must contain words | | : N items | : 2 items | Match Count | Total news matching this group | | [Platform] | [Baidu Hot] | Source Platform | Platform name of the news | | 🆕 | 🆕 ChatGPT-5 officially launched | New Mark | First appearance in this round | | [**number**] | [**1**] | High Rank | Rank ≤ threshold, bold red display | | [number] | [7] | Normal Rank | Rank > threshold, normal display | | - time | - 09:15 | First Time | Time when news was first discovered | | [time~time] | [08:30 ~ 10:45] | Duration | Time range from first to last appearance | | (N times) | (3 times) | Frequency | Total appearances during monitoring | | **New Section** | 🆕 **New Trending News** | New Topic Summary | Separately shows newly appeared topics |
### 6. Docker Deployment
👉 Click to expand: Complete Docker Deployment Guide
**Image Description:** TrendRadar provides two independent Docker images, deploy according to your needs: | Image Name | Purpose | Description | |---------|------|------| | `wantcat/trendradar` | News Push Service | Scheduled news crawling, push notifications (Required) | | `wantcat/trendradar-mcp` | AI Analysis Service | MCP protocol support, AI dialogue analysis (Optional) | > 💡 **Recommendations**: > - Only need push functionality: Deploy `wantcat/trendradar` image only > - Need AI analysis: Deploy both images --- #### Method 1: Using docker compose (Recommended) 1. **Create Project Directory and Config**: ```bash # Clone project to local git clone https://github.com/sansan0/TrendRadar.git cd TrendRadar ``` > 💡 **Note**: Key directory structure required for Docker deployment: ``` current directory/ ├── config/ │ ├── config.yaml # Core config (required) │ ├── frequency_words.txt # Keyword config (required) │ ├── timeline.yaml # Timeline config │ ├── ai_analysis_prompt.txt # AI analysis prompt (optional) │ ├── ai_translation_prompt.txt # AI translation prompt (optional) │ ├── ai_interests.txt # AI interest filtering config (optional) │ ├── ai_filter/ # AI filter prompts │ │ ├── prompt.txt │ │ ├── extract_prompt.txt │ │ └── update_tags_prompt.txt │ └── custom/ # User custom config (optional) │ ├── ai/ # Custom AI prompts │ └── keyword/ # Custom keyword files └── docker/ ├── .env # Sensitive info + Docker-specific config └── docker-compose.yml # Docker Compose orchestration file ``` 2. **Config File Description**: **Configuration Division Principles (v4.6.0 optimized)**: | File | Purpose | Change Frequency | Description | |------|---------|-----------------|-------------| | `config/config.yaml` | **Core config** | Low | Report mode, push settings, storage format, push window, AI analysis toggle, platform enable/disable, etc. | | `config/frequency_words.txt` | **Keyword config** | High | Set your interested trending keywords, supports groups, regex, aliases, and advanced syntax | | `config/timeline.yaml` | **Timeline config** | Low | Controls news timeline display and filtering rules | | `config/ai_analysis_prompt.txt` | **AI analysis prompt** | Medium | Customize AI analysis role definition and output format (v5.0.0+) | | `config/ai_translation_prompt.txt` | **AI translation prompt** | Low | Customize AI translation prompt template | | `config/ai_interests.txt` | **AI interest filtering** | Medium | Define rules for AI to auto-filter news based on interests | | `config/ai_filter/` | **AI filter prompts** | Low | Internal prompts for AI filter module (usually no need to modify) | | `config/custom/` | **User custom extensions** | As needed | `custom/ai/` for custom AI prompts, `custom/keyword/` for custom keyword files | | `docker/.env` | **Sensitive info + Docker-specific config** | Low | Webhook URLs, API Keys, S3 credentials, scheduled tasks, **not tracked by git** | > 💡 **Division Guidelines**: > - **Feature behavior** → Edit `config.yaml` (e.g., enable/disable platforms, adjust push mode) > - **Content of interest** → Edit `frequency_words.txt` (e.g., add new keywords to follow) > - **AI output style** → Edit `ai_analysis_prompt.txt` or `ai_translation_prompt.txt` > - **Keys & credentials** → Edit `docker/.env` (API Keys, Webhook URLs, and other sensitive info go here) > - **Custom extensions** → Use `config/custom/` directory to avoid default configs being overwritten by upgrades **⚙️ Environment Variable Override Mechanism (v3.0.5+)** If you encounter **config.yaml modifications not taking effect** in NAS or other Docker environments, you can directly override configs via environment variables: | Environment Variable | Corresponding Config | Example Value | Description | |---------------------|---------------------|---------------|-------------| | `ENABLE_WEBSERVER` | - | `true` / `false` | Auto-start web server | | `WEBSERVER_PORT` | - | `8080` | Web server port | | `WEBSERVER_WATCHDOG` | - | `true` / `false` | Turn on "auto-recover web page service" (restarts it if it crashes) | | `WEBSERVER_WATCHDOG_INTERVAL` | - | `60` | How often to check and auto-recover (seconds) | | `FEISHU_WEBHOOK_URL` | `notification.channels.feishu.webhook_url` | `https://...` | Feishu Webhook (multi-account use `;` separator) | | `AI_ANALYSIS_ENABLED` | `ai_analysis.enabled` | `true` / `false` | Enable AI analysis (v5.0.0 new) | | `AI_API_KEY` | `ai.api_key` | `sk-xxx...` | AI API Key (shared by ai_analysis and ai_translation) | | `AI_PROVIDER` | `ai.provider` | `deepseek` / `openai` / `gemini` | AI provider (v5.0.0 new) | | `S3_*` | `storage.remote.*` | - | Remote storage config (5 params) | **Config Priority**: Environment Variables > config.yaml **Usage Method**: - Modify `.env` file, uncomment and fill in needed configs - Or add directly in NAS/Synology Docker management interface's "Environment Variables" - Restart container to take effect: `docker compose up -d` 3. **Start Service**: **Option A: Start All Services (Push + AI Analysis)** ```bash # Pull latest images docker compose pull # Start all services (trendradar + trendradar-mcp) docker compose up -d ``` **Option B: Start News Push Service Only** ```bash # Start trendradar only (scheduled crawling and push) docker compose pull trendradar docker compose up -d trendradar ``` **Option C: Start MCP AI Analysis Service Only** ```bash # Start trendradar-mcp only (AI analysis interface) docker compose pull trendradar-mcp docker compose up -d trendradar-mcp ``` > 💡 **Tips**: > - Most users only need to start `trendradar` for news push functionality > - Only need to start `trendradar-mcp` when using ChatGPT/Gemini for AI dialogue analysis > - Both services are independent and can be flexibly combined 4. **Check Running Status**: ```bash # View news push service logs docker logs -f trendradar # View MCP AI analysis service logs docker logs -f trendradar-mcp # View all container status docker ps | grep trendradar # Stop specific service docker compose stop trendradar # Stop push service docker compose stop trendradar-mcp # Stop MCP service ``` #### Method 2: Local Build (Developer Option) If you need custom code modifications or build your own image: ```bash # Clone project git clone https://github.com/sansan0/TrendRadar.git cd TrendRadar # Modify config files vim config/config.yaml vim config/frequency_words.txt # Use build version docker compose cd docker cp docker-compose-build.yml docker-compose.yml ``` **Build and Start Services**: ```bash # Option A: Build and start all services docker compose build docker compose up -d # Option B: Build and start news push service only docker compose build trendradar docker compose up -d trendradar # Option C: Build and start MCP AI analysis service only docker compose build trendradar-mcp docker compose up -d trendradar-mcp ``` > 💡 **Architecture Parameter Notes**: > - Default builds `amd64` architecture images (suitable for most x86_64 servers) > - To build `arm64` architecture (Apple Silicon, Raspberry Pi, etc.), set environment variable: > ```bash > export DOCKER_ARCH=arm64 > docker compose build > ``` #### Image Update ```bash # Method 1: Manual update (Crawler + MCP images) docker pull wantcat/trendradar:latest docker pull wantcat/trendradar-mcp:latest docker compose down docker compose up -d # Method 2: Using docker compose update docker compose pull docker compose up -d ``` **Available Images**: | Image Name | Purpose | Description | |---------|------|---------| | `wantcat/trendradar` | News Push Service | Scheduled news crawling, push notifications | | `wantcat/trendradar-mcp` | MCP Service | AI analysis features (optional) | #### Service Management Commands ```bash # View running status docker exec -it trendradar python manage.py status # Manually execute crawler once docker exec -it trendradar python manage.py run # View real-time logs docker exec -it trendradar python manage.py logs # Display current config docker exec -it trendradar python manage.py config # Display output files docker exec -it trendradar python manage.py files # Web server management (for browser access to generated reports) docker exec -it trendradar python manage.py start_webserver # Start web server docker exec -it trendradar python manage.py stop_webserver # Stop web server docker exec -it trendradar python manage.py webserver_status # Check web server status # View help info docker exec -it trendradar python manage.py help # Restart container docker restart trendradar # Stop container docker stop trendradar # Remove container (keep data) docker rm trendradar ``` > 💡 **Web Server Notes**: > - After starting, access latest report at `http://localhost:8080` > - Access historical reports via directory navigation (e.g., `http://localhost:8080/2025-xx-xx/`) > - Port can be configured in `.env` file with `WEBSERVER_PORT` parameter > - Auto-start: Set `ENABLE_WEBSERVER=true` in `.env` > - Auto-recover: `WEBSERVER_WATCHDOG=true` (default). It checks every `WEBSERVER_WATCHDOG_INTERVAL` seconds and restarts the web page service if needed > - `stop_webserver` means you manually turn off the web page service (command: `docker exec -it trendradar python manage.py stop_webserver`) > - "Auto restart" means the system turns that web page service back on automatically. If you stopped it manually and want it back, run `docker exec -it trendradar python manage.py start_webserver` > - Security: Static files only, limited to output directory, localhost binding only #### Data Persistence Generated reports and data are saved in `./output` directory by default. Data persists even if container is restarted or removed. **📊 Web Report Access Paths**: TrendRadar generates daily summary HTML reports to two locations simultaneously: | File Location | Access Method | Use Case | |--------------|---------------|----------| | `output/index.html` | Direct host access | **Docker Deployment** (via Volume mount, visible on host) | | `index.html` | Root directory access | **GitHub Pages** (repository root, auto-detected by Pages) | | `output/html/YYYY-MM-DD/当日汇总.html` | Historical reports | All environments (archived by date) | **Local Access Examples**: ```bash # Method 1: Via Web Server (recommended, Docker environment) # 1. Start web server docker exec -it trendradar python manage.py start_webserver # 2. Access in browser http://localhost:8080 # Access latest report (default index.html) http://localhost:8080/html/2025-xx-xx/ # Access reports for specific date # Method 2: Direct file access (local environment) open ./output/index.html # macOS start ./output/index.html # Windows xdg-open ./output/index.html # Linux # Method 3: Access historical archives open ./output/html/2025-xx-xx/当日汇总.html ``` **Why two index.html files?** - `output/index.html`: Docker Volume mounted to host, can be opened locally - `index.html`: Pushed to repository by GitHub Actions, auto-deployed by GitHub Pages > 💡 **Tip**: Both files have identical content, choose either one to access. #### Troubleshooting ```bash # Check container status docker inspect trendradar # View container logs docker logs --tail 100 trendradar # Enter container for debugging docker exec -it trendradar /bin/bash # Verify config files docker exec -it trendradar ls -la /app/config/ ``` #### MCP Service Deployment (AI Analysis Feature) If you need to use AI analysis features, you can deploy the standalone MCP service container. **Architecture Description**: ```mermaid flowchart TB subgraph trendradar["trendradar"] A1[Scheduled News Fetching] A2[Push Notifications] end subgraph trendradar-mcp["trendradar-mcp"] B1[127.0.0.1:3333] B2[AI Analysis API] end subgraph shared["Shared Volume"] C1["config/ (ro)"] C2["output/ (ro)"] end trendradar --> shared trendradar-mcp --> shared ``` **Quick Start**: Use docker compose to start both news push and MCP services: ```bash # Clone project (Recommended) git clone https://github.com/sansan0/TrendRadar.git cd TrendRadar/docker docker compose up -d # Check running status docker ps | grep trendradar ``` **Start MCP Service Separately**: ```bash # Linux/Mac docker run -d --name trendradar-mcp \ -p 127.0.0.1:3333:3333 \ -v $(pwd)/config:/app/config:ro \ -v $(pwd)/output:/app/output:ro \ -e TZ=Asia/Shanghai \ wantcat/trendradar-mcp:latest # Windows PowerShell docker run -d --name trendradar-mcp ` -p 127.0.0.1:3333:3333 ` -v ${PWD}/config:/app/config:ro ` -v ${PWD}/output:/app/output:ro ` -e TZ=Asia/Shanghai ` wantcat/trendradar-mcp:latest ``` > ⚠️ **Note**: Ensure `config/` and `output/` folders exist in current directory with config files and news data before running. **Verify Service**: ```bash # Check MCP service health curl http://127.0.0.1:3333/mcp # View MCP service logs docker logs -f trendradar-mcp ``` **Configure in AI Clients**: After MCP service starts, configure based on your client: **Cherry Studio** (Recommended, GUI config): - Settings → MCP Server → Add - Type: `streamableHttp` - URL: `http://127.0.0.1:3333/mcp` **Claude Desktop / Cline** (JSON config): ```json { "mcpServers": { "trendradar": { "url": "http://127.0.0.1:3333/mcp", "type": "streamableHttp" } } } ``` > 💡 **Tip**: MCP service only listens on local port (127.0.0.1) for security. For remote access, configure reverse proxy and authentication yourself.
### 7. How is the push content displayed?
👉 Click to expand: Customize Push Style and Content
**Configuration Location:** `report` and `display` sections in `config/config.yaml` ```yaml report: mode: "daily" # Push mode display_mode: "keyword" # Display mode (v4.6.0 new) rank_threshold: 5 # Ranking highlight threshold sort_by_position_first: false # Sorting priority max_news_per_keyword: 0 # Maximum display count per keyword display: region_order: # Region display order (v5.2.0 new) - new_items # New trending section - hotlist # Hotlist section - rss # RSS subscription section - standalone # Independent display section - ai_analysis # AI analysis section ``` #### Configuration Details | Config Item | Type | Default | Description | |------------|------|---------|-------------| | `mode` | string | `daily` | Push mode, options: `daily`/`incremental`/`current`, see [Push Mode Details](#3-push-mode-details) | | `display_mode` | string | `keyword` | Display mode, options: `keyword`/`platform`, see below | | `rank_threshold` | int | `5` | Ranking highlight threshold, news with rank ≤ this value will be displayed in bold | | `sort_by_position_first` | bool | `false` | Sorting priority: `false`=sort by news count, `true`=sort by config position | | `max_news_per_keyword` | int | `0` | Maximum display count per keyword, `0`=unlimited | | `display.region_order` | list | See config above | Adjust list order to control region display positions | #### Display Mode Configuration (v4.6.0 New) Controls how news is grouped in push messages and HTML reports: | Mode | Grouping | Title Prefix | Use Case | |------|----------|--------------|----------| | `keyword` (default) | Group by keyword | `[Platform]` | Users focusing on specific topics | | `platform` | Group by platform | `[Keyword]` | Users focusing on specific platforms | **Example Comparison:** ``` # keyword mode (group by keyword) 📊 Trending Keywords Stats 🔥 [1/3] AI : 12 items 1. [Weibo] OpenAI releases GPT-5 #1-#3 - 08:30 (5 times) 2. [Zhihu] How to view AI replacing programmers #2 - 09:15 (3 times) # platform mode (group by platform) 📊 Trending News Stats 🔥 [1/4] Weibo : 12 items 1. [AI] OpenAI releases GPT-5 #1-#3 - 08:30 (5 times) 2. [Trump] Trump announces major policy #2 - 09:15 (3 times) ``` #### Region Display Order (region_order) Control the display position of each section in push messages by adjusting the order of `display.region_order` list. **Default Order**: New Items → Hotlist → RSS → Standalone → AI Analysis **Custom Example**: Want AI analysis at the top? ```yaml display: region_order: - ai_analysis # Move to first line - new_items - hotlist - rss - standalone ``` **Note**: A region will only be displayed when both conditions are met: 1. Listed in `region_order` 2. Corresponding switch in `display.regions` is `true` #### Region Switches (regions) Control whether each region is displayed in push notifications via `display.regions`: ```yaml display: regions: hotlist: true # Hotlist region (keyword-matched trending news) new_items: false # New items region (new hotlist + new RSS items) rss: true # RSS region (keyword-matched RSS content) standalone: false # Standalone section (full hotlist/RSS, unfiltered by keywords) ai_analysis: true # AI analysis region ``` | Region | Config Key | Default | Description | |--------|-----------|---------|-------------| | **Hotlist** | `hotlist` | `true` | Keyword-matched trending news aggregation | | **New Items** | `new_items` | `false` | Newly appeared topics in this crawl cycle (hotlist + RSS). Note: the 🆕 markers in the hotlist region are not affected by this switch | | **RSS** | `rss` | `true` | Keyword-matched RSS subscription content. When disabled, RSS analysis is skipped, but RSS in standalone section is unaffected | | **Standalone** | `standalone` | `false` | Full content display for specified platforms/RSS, unfiltered by keywords | | **AI Analysis** | `ai_analysis` | `true` | AI-generated trending analysis summary | #### Sorting Priority Configuration **Example Scenario:** Config order A, B, C, news count A(3), B(10), C(5) | Config Value | Display Order | Use Case | |-------------|--------------|----------| | `false` (default) | B(10) → C(5) → A(3) | Focus on popularity trends | | `true` | A(3) → B(10) → C(5) | Focus on personal priority | **Docker Environment Variables:** ```bash SORT_BY_POSITION_FIRST=true MAX_NEWS_PER_KEYWORD=10 ``` #### Independent Display Section Configuration (v5.0.0 New) Provides full trending list display for specified platforms, unaffected by `frequency_words.txt` keyword filtering. **Configuration Location:** `display` section in `config/config.yaml` ```yaml display: regions: standalone: true # Show standalone section in push (disabling doesn't affect AI analysis) standalone: platforms: ["zhihu", "weibo"] # Trending platform ID list rss_feeds: ["hacker-news"] # RSS feed ID list max_items: 20 # Max display count per source (0=unlimited) ``` > 💡 **Display and AI analysis are independently controlled**: `regions.standalone` only controls whether the standalone section appears in push notifications. Even with display disabled, setting `include_standalone: true` in the AI config still allows AI to analyze full hotlist data from these platforms. Ideal for users who want deeper AI insights without longer push messages. **Use Cases:** - Want to view the complete trending ranking of a platform (like Zhihu) instead of just keyword-matched content - Subscribed to RSS feeds with few updates (like personal blogs) and want full push every time **Effect Example:** ``` 📋 Independent Display Section (Total 15 items) Zhihu Trending (10 items): 1. [Zhihu] How to view OpenAI releasing Sora? 2. [Zhihu] 2024 postgraduate entrance exam scores released... ... Hacker News (5 items): 1. [Hacker News] Launch HN: TrendRadar... ... ```
### 8. When will I receive pushes?
👉 Click to expand: Set Push Time (Scheduling System)
**Configuration Location:** `schedule` section in `config/config.yaml` + `config/timeline.yaml` #### Quick Start Just pick a preset template in `config.yaml` — no need to edit `timeline.yaml`: ```yaml schedule: enabled: true preset: "morning_evening" # Change this line ``` #### Available Preset Templates | Template | Description | Push Behavior | |----------|-------------|---------------| | `morning_evening` | Incremental + evening summary (recommended) | Push new content all day + 19:00-21:00 daily summary | | `always_on` | 24/7 monitoring | Push whenever new content appears, no time restrictions | | `office_hours` | Office hours | Three-phase weekday push (morning briefing → noon update → closing summary), weekends incremental | | `night_owl` | Night owl | Afternoon peek + late-night daily summary (22:00-01:00 cross-midnight) | | `custom` | Fully customizable | Edit the `custom` section at the bottom of `timeline.yaml` | #### Full Customization If none of the preset templates fit your needs, edit the `custom` section at the bottom of `config/timeline.yaml` to freely define time periods, day plans, and week mappings. See the comments in `timeline.yaml` for details. #### Important Notice > ⚠️ **Users upgrading from older versions:** > - v6.0.0 removed the old `notification.push_window` and `ai_analysis.analysis_window` configs > - Please switch to the new `schedule` + `timeline.yaml` scheduling system > - Old "push once per day" can be replaced with the `morning_evening` preset > - Old "working hours push" can be replaced with the `office_hours` preset > ⚠️ **GitHub Actions Users Note:** > - GitHub Actions execution time is unstable, may have ±15 minutes deviation > - Time period ranges should be at least **2 hours** wide > - For precise timed push, recommend **Docker deployment** on personal server
### 9. How often does it run?
👉 Click to expand: Set Auto-Run Frequency
**Configuration Location:** `schedule` section in `.github/workflows/crawler.yml` ```yaml on: schedule: - cron: "0 * * * *" # Run every hour ``` #### How to change the schedule? GitHub Actions uses a time format called "Cron". You don't need to understand it deeply; just copy and replace the code below. **Configuration Location**: `schedule` section in `.github/workflows/crawler.yml` | I want... | Copy this line | Note | |-----------|----------------|------| | **Every Hour** | `- cron: "0 * * * *"` | **Default**, runs at minute 0 | | **Every 30 Mins** | `- cron: "*/30 * * * *"` | Runs every 30 minutes | | **Daily at 8 AM** | `- cron: "0 0 * * *"` | ⚠️ `0` because UTC 0:00 = Beijing 8:00 AM | | **Work Hours (30m)** | `- cron: "*/30 0-14 * * *"` | Beijing 8:00 - 22:00 | | **3 Times Daily** | `- cron: "0 0,6,12 * * *"` | Beijing 8:00, 14:00, 20:00 | #### ⚠️ Two Important Notes 1. **Time Zone**: GitHub servers use **UTC time**. - **Math**: Your desired Beijing time **minus 8 hours** = value to fill. - *Example: For Beijing 20:00, fill in 12:00.* 2. **Don't run too often**: Suggest intervals no shorter than 30 minutes. - GitHub free resources are limited; running too frequently might get flagged. - Actions have startup delays, so precise timing isn't guaranteed anyway. #### Step-by-Step Guide 1. In your GitHub repository, find `.github/workflows/crawler.yml`. 2. Click the ✏️ (Edit) button top right. 3. Find the line `cron: "..."` and replace the content inside quotes with the code above. 4. Click the green **Commit changes** button to save.
### 10. Push to multiple groups/devices
👉 Click to expand: Send to Multiple Recipients
**Configuration Location:** `notification` section in `config/config.yaml` > ### ⚠️ **Security First** > **DO NOT write passwords/Tokens directly in `config.yaml`!** > If you upload a file containing passwords to GitHub, the whole world can see it. > > **Correct Method**: > - **GitHub Actions Users**: Add in Settings -> Secrets > - **Docker Users**: Write in `.env` file (this file won't be uploaded) #### How to push to multiple places? Simple, just separate multiple addresses with a semicolon `;`. **Example**: Suppose you have two Feishu groups: - Group 1: `https://.../webhook/aaa` - Group 2: `https://.../webhook/bbb` Config value: `https://.../webhook/aaa;https://.../webhook/bbb` #### Supported Platforms | Platform | Method | Note | |----------|--------|------| | **Feishu/DingTalk/WeWork** | Separate URLs with `;` | Just chain them up | | **Bark (iOS)** | Separate URLs with `;` | Push to multiple iPhones | | **Telegram** | Separate Tokens and ChatIDs with `;` | ⚠️ **Order must match**:
Token1 ↔ ChatID1
Token2 ↔ ChatID2 | | **ntfy** | Separate Topics and Tokens with `;` | If a topic needs no token, leave empty:
`token1;;token3` (middle is empty) | #### Common Config Examples (GitHub Secrets / .env) ```bash # Send to 3 Feishu groups FEISHU_WEBHOOK_URL=https://hook1...;https://hook2...;https://hook3... # Send to 2 DingTalk groups DINGTALK_WEBHOOK_URL=https://oapi...;https://oapi... # Send to 2 Telegram users (Match one-to-one) TELEGRAM_BOT_TOKEN=tokenA;tokenB TELEGRAM_CHAT_ID=userA;userB ``` > **Tip**: Default limit is 3 accounts per platform to prevent abuse. Adjust `MAX_ACCOUNTS_PER_CHANNEL` if needed.

### 11. Where is the data saved?
👉 Click to expand: Choose Data Storage Location
#### Where is the data saved? The system automatically selects the best location for you, so you usually don't need to worry about it: | Your Environment | Data Location | Description | |------------------|---------------|-------------| | **Docker / Local** | **Local Disk** | Saved in the `output/` folder within the project directory. | | **GitHub Actions** | **Cloud Storage** | Since GitHub Actions environments are destroyed after running, cloud storage (e.g., Cloudflare R2) is required. | #### How to configure cloud storage? (For GitHub Actions Users) If you run on GitHub Actions, you need a "cloud drive" to save data. For example, Cloudflare R2 (free tier is generous). **Add these 5 variables to GitHub Secrets:** | Variable Name | Value | |---------------|-------| | `STORAGE_BACKEND` | `remote` | | `S3_BUCKET_NAME` | Your bucket name | | `S3_ACCESS_KEY_ID` | Your Access Key | | `S3_SECRET_ACCESS_KEY` | Your Secret Key | | `S3_ENDPOINT_URL` | Your R2 endpoint URL | > 💡 **Tutorial**: How to apply for R2? See [Quick Start - Remote Storage Configuration](#-quick-start) #### How long is data kept? By default, we never delete your data. If you want to save space, you can enable "Auto Cleanup". **Config Location**: `config/config.yaml` ```yaml storage: local: retention_days: 30 # Keep local data for 30 days (0 = forever) remote: retention_days: 30 # Keep cloud data for 30 days ``` #### Push time is wrong? (Timezone Settings) If you are overseas or find the push time doesn't match your local time, you can change the timezone. **Config Location**: `config/config.yaml` ```yaml app: timezone: "Asia/Shanghai" # Default is China Standard Time ``` - Example for Los Angeles: `America/Los_Angeles` - Example for London: `Europe/London`
### 12. Let AI help me analyze hot topics
👉 Click to expand: Enable AI Smart Analysis #### What can AI do for me? After enabling this feature, AI acts as a professional analyst. When pushing a batch of news, it will: 1. **Auto-Read**: Read all matched trending news. 2. **Deep Think**: Analyze connections between seemingly isolated news items. 3. **Write Report**: Append a concise and profound "Insight Report" at the end of the push message. **Includes**: Trending topic summary, public opinion direction, cross-platform correlation, potential impact assessment, etc. #### How to enable AI Analysis? The simplest way is via environment variables (Recommended for GitHub Secrets or .env). **Required Configurations**: | Variable Name | Value | Description | |--------------|-------|-------------| | `AI_ANALYSIS_ENABLED` | `true` | Enable switch | | `AI_API_KEY` | `sk-xxxxxx` | Your API Key | | `AI_MODEL` | `deepseek/deepseek-chat` | Model identifier (format: `provider/model`) | **Supported AI Providers** (Based on LiteLLM, supports 100+ providers): | Provider | AI_MODEL Value | Description | |----------|----------------|-------------| | **DeepSeek** (Recommended) | `deepseek/deepseek-chat` | Excellent cost-performance ratio for high-frequency analysis | | **OpenAI** | `openai/gpt-4o`
`openai/gpt-4o-mini` | GPT-4o series | | **Google Gemini** | `gemini/gemini-1.5-flash`
`gemini/gemini-1.5-pro` | Gemini series | | **Custom API** | Any format | Use with `AI_API_BASE` | > 💡 **New Feature**: Now based on [LiteLLM](https://github.com/BerriAI/litellm) unified interface, supporting 100+ AI providers with simpler configuration and better error handling. **Optional Configurations**: | Variable Name | Default | Description | |--------------|---------|-------------| | `AI_API_BASE` | (auto) | Custom API endpoint (e.g., OneAPI, local models) | | `AI_TEMPERATURE` | `1.0` | Sampling temperature (0-2, higher = more random) | | `AI_MAX_TOKENS` | `5000` | Maximum tokens to generate | | `AI_TIMEOUT` | `120` | Request timeout (seconds) | | `AI_NUM_RETRIES` | `2` | Number of retries on failure | #### Advanced: AI Translation If you subscribe to foreign RSS feeds (like Hacker News), AI can translate the content into your native language. **Configuration Location**: `config/config.yaml` ```yaml ai_translation: enabled: true # Enable translation language: "Chinese" # Target language (Chinese, English, Japanese...) ``` #### Advanced: Customize AI "Persona" Think the AI sounds too official? You can modify its prompt to change its style (e.g., "Sarcastic Commentator", "Senior Investment Advisor"). - **File**: `config/ai_analysis_prompt.txt` - **Method**: Edit with a text editor, tell AI what analysis style you want.

## ✨ AI Analysis TrendRadar v3.0.0 added **MCP (Model Context Protocol)** based AI analysis feature, allowing natural language conversations with news data for deep analysis. ### ⚠️ Important Notice Before Use **Critical: AI features require local news data support** AI analysis **does not** query real-time online data directly, but analyzes **locally accumulated news data** (stored in the `output` folder) #### Usage Instructions: 1. **Built-in Test Data**: The `output` directory includes one week of trending news data from **December 21-27, 2025** for quick feature testing 2. **Query Limitations**: - ✅ Only query data within available date range (Dec 21-27, 7 days total) - ❌ Cannot query real-time news or future dates 3. **Getting Latest Data**: - Test data is for quick experience only, **recommend deploying the project yourself** to get real-time data - Follow [Quick Start](#-quick-start) to deploy and run the project - After accumulating news data for at least 1 day, you can query the latest trending topics --- ### 1. Quick Deployment Cherry Studio provides GUI config interface, 5-minute quick deployment, complex parts are one-click install. **Illustrated Deployment Tutorial**: Now updated to my WeChat Official Account (see [Support Project](#-support-project)), reply "mcp" to get **Detailed Deployment Tutorial**: [README-Cherry-Studio.md](README-Cherry-Studio.md) **Deployment Mode Description**: - **STDIO Mode (Recommended)**: One-time configuration, no need to reconfigure later. The **illustrated deployment tutorial** only demonstrates this mode's configuration. - **HTTP Mode (Alternative)**: If STDIO mode configuration encounters issues, you can use HTTP mode. This mode's configuration is basically the same as STDIO, but only requires copy-pasting one line, less error-prone. The only thing to note is that you need to manually start the service before each use. For details, refer to the HTTP mode section at the bottom of [README-Cherry-Studio.md](README-Cherry-Studio.md). ### 2. Learning to Talk with AI **Detailed Conversation Tutorial**: [README-MCP-FAQ.md](README-MCP-FAQ.md) **Question Effect**: > 💡 **Tip**: Actually not recommended to ask multiple questions at once. If your chosen AI model cannot even sequentially call as shown below, suggest switching models. MCP usage effect
## 🔌 MCP Clients TrendRadar MCP service supports standard Model Context Protocol (MCP), can connect to various AI clients supporting MCP for smart analysis. ### Supported Clients **Note**: - Replace `/path/to/TrendRadar` with your actual project path - Windows paths use double backslashes: `C:\\Users\\YourName\\TrendRadar` - Remember to restart after saving
👉 Click to expand: Cursor #### Method 1: HTTP Mode 1. **Start HTTP Service**: ```bash # Windows start-http.bat # Mac/Linux ./start-http.sh ``` 2. **Configure Cursor**: **Project Level Config** (Recommended): Create `.cursor/mcp.json` in project root: ```json { "mcpServers": { "trendradar": { "url": "http://localhost:3333/mcp", "description": "TrendRadar News Trending Aggregation Analysis" } } } ``` **Global Config**: Create `~/.cursor/mcp.json` in user directory (same content) 3. **Usage Steps**: - Restart Cursor after saving config - Check connected tools in chat interface "Available Tools" - Start using: `Search today's "AI" related news` #### Method 2: STDIO Mode (Recommended) Create `.cursor/mcp.json`: ```json { "mcpServers": { "trendradar": { "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ] } } } ```
👉 Click to expand: VSCode (Cline/Continue) #### Cline Configuration Add in Cline's MCP settings: **HTTP Mode**: ```json { "trendradar": { "url": "http://localhost:3333/mcp", "type": "streamableHttp", "autoApprove": [], "disabled": false } } ``` **STDIO Mode** (Recommended): ```json { "trendradar": { "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ], "type": "stdio", "disabled": false } } ``` #### Continue Configuration Edit `~/.continue/config.json`: ```json { "experimental": { "modelContextProtocolServers": [ { "transport": { "type": "stdio", "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ] } } ] } } ``` **Usage Examples**: ``` Analyze recent 7 days "Tesla" popularity trend Generate today's trending summary report Search "Bitcoin" related news and analyze sentiment ```
👉 Click to expand: MCP Inspector (Debug Tool)
MCP Inspector is the official debug tool for testing MCP connections: #### Usage Steps 1. **Start TrendRadar HTTP Service**: ```bash # Windows start-http.bat # Mac/Linux ./start-http.sh ``` 2. **Start MCP Inspector**: ```bash npx @modelcontextprotocol/inspector ``` 3. **Connect in Browser**: - Visit: `http://localhost:3333/mcp` - Test "Ping Server" function to verify connection - Check "List Tools" returns 14 tools: - Date Parsing: resolve_date_range - Basic Query: get_latest_news, get_news_by_date, get_trending_topics - Smart Search: search_news, search_related_news_history - Advanced Analysis: analyze_topic_trend, analyze_data_insights, analyze_sentiment, find_similar_news, generate_summary_report - System Management: get_current_config, get_system_status, trigger_crawl
👉 Click to expand: Other MCP-Compatible Clients
Any client supporting Model Context Protocol can connect to TrendRadar: #### HTTP Mode **Service Address**: `http://localhost:3333/mcp` **Basic Config Template**: ```json { "name": "trendradar", "url": "http://localhost:3333/mcp", "type": "http", "description": "News Trending Aggregation Analysis" } ``` #### STDIO Mode (Recommended) **Basic Config Template**: ```json { "name": "trendradar", "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ], "type": "stdio" } ``` **Notes**: - Replace `/path/to/TrendRadar` with actual project path - Windows paths use backslash escape: `C:\\Users\\...` - Ensure project dependencies installed (ran setup script)
### Common Questions
👉 Click to expand: Q1: HTTP Service Cannot Start?
**Check Steps**: 1. Confirm port 3333 is not occupied: ```bash # Windows netstat -ano | findstr :3333 # Mac/Linux lsof -i :3333 ``` 2. Check if project dependencies installed: ```bash # Re-run install script # Windows: setup-windows.bat or setup-windows-en.bat # Mac/Linux: ./setup-mac.sh ``` 3. View detailed error logs: ```bash uv run python -m mcp_server.server --transport http --port 3333 ``` 4. Try custom port: ```bash uv run python -m mcp_server.server --transport http --port 33333 ```
👉 Click to expand: Q2: Client Cannot Connect to MCP Service?
**Solutions**: 1. **STDIO Mode**: - Confirm UV path correct (run `which uv` or `where uv`) - Confirm project path correct and no Chinese characters - Check client error logs 2. **HTTP Mode**: - Confirm service started (visit `http://localhost:3333/mcp`) - Check firewall settings - Try using 127.0.0.1 instead of localhost 3. **General Checks**: - Restart client application - Check MCP service logs - Use MCP Inspector to test connection
👉 Click to expand: Q3: Tool Call Failed or Returns Error?
**Possible Reasons**: 1. **Data Does Not Exist**: - Confirm crawler has run (have output directory data) - Check query date range has data - Check available dates in output directory 2. **Parameter Error**: - Check date format: `YYYY-MM-DD` - Confirm correct platform ID: `zhihu`, `weibo`, etc. - See parameter descriptions in tool docs 3. **Config Issues**: - Confirm `config/config.yaml` exists - Confirm `config/frequency_words.txt` exists - Check config file format is correct

## 📚 Related Projects > **4 Related Articles** (Chinese): - [Comment here for mobile Q&A by project author](https://mp.weixin.qq.com/s/KYEPfTPVzZNWFclZh4am_g) - [Breaking 1000 stars in 2 months - My GitHub project promotion experience](https://mp.weixin.qq.com/s/jzn0vLiQFX408opcfpPPxQ) - [Important notes for running this project via GitHub fork](https://mp.weixin.qq.com/s/C8evK-U7onG1sTTdwdW2zg) - [How to write WeChat Official Account or news articles based on this project](https://mp.weixin.qq.com/s/8ghyfDAtQZjLrnWTQabYOQ) > **AI Development**: - If you have niche requirements, you can develop based on my project yourself, even with zero programming experience - All my open-source projects use my own **AI-assisted software** to improve development efficiency, this tool is now open-source - **Core Function**: Quickly filter project code to feed AI, you just need to add personal requirements - **Project Address**: https://github.com/sansan0/ai-code-context-helper ### Other Projects > 📍 Chairman Mao's Footprint Map - Interactive dynamic display of complete trajectory 1893-1976. Welcome comrades to contribute data - https://github.com/sansan0/mao-map > Bilibili Comment Data Visualization Analysis Software - https://github.com/sansan0/bilibili-comment-analyzer [![Star History Chart](https://api.star-history.com/svg?repos=sansan0/TrendRadar&type=Date)](https://www.star-history.com/#sansan0/TrendRadar&Date)
## 📄 License GPL-3.0 License ---
[🔝 Back to Top](#trendradar)
================================================ FILE: README-MCP-FAQ-EN.md ================================================
**[中文](README-MCP-FAQ.md)** | **English**
# TrendRadar MCP Tool Usage Q&A > AI Query Guide - How to Use News Trend Analysis Tools Through Natural Conversation (v3.1.7) --- ## 📋 Tools Overview | Category | Tool Name | Description | |:--------:|-----------|-------------| | **Date** | `resolve_date_range` | Parse "this week", "last 7 days" to standard dates | | **Query** | `get_latest_news` | Get the latest batch of trending news | | | `get_news_by_date` | Query historical news by date range | | | `get_trending_topics` | Get trending topics statistics (auto-extract supported) | | **RSS** | `get_latest_rss` | Get latest RSS subscription content | | | `search_rss` | Search keywords in RSS data | | | `get_rss_feeds_status` | View RSS feed config and data status | | **Search** | `search_news` | Unified search (keyword/fuzzy/entity, RSS optional) | | | `find_related_news` | Find news similar to a given title | | **Analysis** | `analyze_topic_trend` | Topic trend analysis (hotness/lifecycle/viral/predict) | | | `analyze_data_insights` | Data insights (platform compare/activity/co-occurrence) | | | `analyze_sentiment` | News sentiment analysis | | | `aggregate_news` | Cross-platform news aggregation & dedup | | | `compare_periods` | Period comparison (week-over-week/month-over-month) | | | `generate_summary_report` | Generate daily/weekly summary reports | | **System** | `get_current_config` | Get current system configuration | | | `get_system_status` | Get system running status | | | `check_version` | Check version updates (TrendRadar + MCP Server) | | | `trigger_crawl` | Manually trigger a crawl task | | **Storage** | `sync_from_remote` | Pull data from remote storage to local | | | `get_storage_status` | Get storage config and status | | | `list_available_dates` | List available dates (local/remote) | | **Article** | `read_article` | Read single article content (Markdown format) | | | `read_articles_batch` | Batch read multiple articles (max 5) | | **Notification** | `get_notification_channels` | Get all configured notification channels and their status | | | `send_notification` | Send messages to configured notification channels (auto format conversion) | --- ## ⚙️ Default Settings Explanation (Important!) The following optimization strategies are adopted by default, mainly to save AI token consumption: | Default Setting | Description | How to Adjust | | -------------- | --------------------------------------- | ------------------------------------- | | **Result Limit** | Default returns 50 news items | Say "return top 10" or "give me 100 items" in conversation | | **Time Range** | Default queries today's data | Say "query yesterday", "last week" or "Jan 1 to 7" | | **URL Links** | Default no links (saves ~160 tokens/item) | Say "need links" or "include URLs" | | **Keyword List** | Default does not use frequency_words.txt to filter news | Only used when calling "trending topics" tool | **⚠️ Important:** The choice of AI model directly affects the tool call effectiveness. The smarter the AI, the more accurate the calls. When you remove the above restrictions, for example, from querying today to querying a week, first you need to have a week's data locally, and secondly, token consumption may multiply. **💡 Tip:** This project provides a dedicated date parsing tool that can accurately parse natural language date expressions like "last 7 days", "this week", ensuring all AI models get consistent date ranges. See Q18 below for details. ## 💰 AI Models Below I use the **[SiliconFlow](https://cloud.siliconflow.cn)** platform as an example, which has many large models to choose from. During the development and testing of this project, I used this platform for many functional tests and validations. ### 📊 Registration Method Comparison | Registration Method | Direct Registration Without Referral | Registration With Referral Link | |:-------:|:-------:|:-----------------:| | Registration Link | [siliconflow.cn](https://cloud.siliconflow.cn) | [Referral Link](https://cloud.siliconflow.cn/i/fqnyVaIU) | | Free Quota | 0 tokens | **20 million tokens** (≈$2) | | Extra Benefits | ❌ | ✅ Referrer also gets 20 million tokens | > 💡 **Tip**: The above gift quota should allow for **200+ queries** ### 🚀 Quick Start #### 1️⃣ Register and Get API Key 1. Complete registration using the link above 2. Visit [API Key Management Page](https://cloud.siliconflow.cn/me/account/ak) 3. Click "Create New API Key" 4. Copy the generated key (please keep it safe) #### 2️⃣ Configure in Cherry Studio 1. Open **Cherry Studio** 2. Go to "Model Service" settings 3. Find "SiliconFlow" 4. Paste the copied key into the **[API Key]** input box 5. Ensure the checkbox in the top right corner shows **green** when enabled ✅ --- ### ✨ Configuration Complete! Now you can start using this project and enjoy stable and fast AI services! After testing one query, please immediately check the [SiliconFlow Billing](https://cloud.siliconflow.cn/me/bills) to see the consumption and have an estimate in mind. --- ## Basic Queries ### Q1: How to view the latest news? **You can ask like this:** - "Show me the latest news" - "Query today's trending news" - "Get the latest 10 news from Zhihu and Weibo" - "View latest news, need links included" **Tool return behavior:** - Tool returns the latest 50 news items from all platforms - Does not include URL links by default (saves tokens) **AI display behavior (Important):** - ⚠️ **AI usually auto-summarizes**, only showing partial news (like TOP 10-20 items) - ✅ If you want to see all 50 items, need to explicitly request: "show all news" or "list all 50 items completely" - 💡 This is the AI model's natural behavior, not a tool limitation **Can be adjusted:** - Specify platform: like "only Zhihu" - Adjust quantity: like "return top 20" - Include links: like "need links" - **Request full display**: like "show all, don't summarize" --- ### Q2: How to query news from a specific date? **You can ask like this:** - "Query yesterday's news" - "Check Zhihu news from 3 days ago" - "What news was there on 2025-10-10" - "News from last Monday" - "Show me the latest news" (automatically queries today) **Supported date formats:** - Relative dates: today, yesterday, day before yesterday, 3 days ago - Days of week: last Monday, this Wednesday - Absolute dates: 2025-10-10, October 10 **Tool return behavior:** - Automatically queries today when date not specified (saves tokens) - Tool returns 50 news items from all platforms - Does not include URL links by default **AI display behavior (Important):** - ⚠️ **AI usually auto-summarizes**, only showing partial news (like TOP 10-20 items) - ✅ If you want to see all, need to explicitly request: "show all news, don't summarize" --- ### Q3: How to view trending topic statistics? **You can ask like this:** - "How many times did my followed words appear today" (using preset keywords) - "Automatically analyze what hot topics are in today's news" (auto extract) - "See what are the hottest words in the news" (auto extract) **Two extraction modes:** | Mode | Description | Example Question | |------|------|---------| | **Preset keywords** | Count preset followed words (based on config file, default) | "How many times did my followed words appear" | | **Auto extract** | Auto-extract high-frequency words from news titles (no preset needed) | "Auto-analyze hot topics" | --- ## RSS Feed Queries ### Q4.1: How to view latest RSS feed content? **You can ask like this:** - "Show me the latest RSS feed content" - "Get the latest articles from Hacker News" - "View latest 20 items from all RSS feeds" - "Get RSS feeds, need to include summaries" - "Show me RSS content from the last week" (multi-day query support) - "Get Hacker News articles from last 7 days" **Tool return behavior:** - Returns today's RSS items by default (up to 50) - Supports `days` parameter for multi-day queries (1-30 days) - Does not include summaries by default (saves tokens) - Sorted by publication time in descending order - Auto-deduplication across dates (by URL) **AI display behavior (Important):** - ⚠️ **AI usually auto-summarizes**, only showing partial items - ✅ If you want to see all, need to explicitly request: "show all RSS content" **Can be adjusted:** - Specify RSS feed: like "only Hacker News" - Specify days: like "last 7 days", "past week" - Adjust quantity: like "return top 20" - Include summary: like "need summaries" --- ### Q4.2: How to search content in RSS feeds? **You can ask like this:** - "Search for 'AI' related articles in RSS" - "Search RSS content about 'machine learning' from last 7 days" - "Search 'Python' in Hacker News" **Tool return behavior:** - Searches RSS item titles using keywords - Default searches last 7 days of data - Tool returns up to 50 results **Can be adjusted:** - Specify RSS feed: like "only search Hacker News" - Adjust days: like "search last 14 days" - Include summary: like "need summaries" --- ### Q4.3: How to view RSS feed status? **You can ask like this:** - "View RSS feed status" - "How much data has RSS crawled" - "Which RSS feeds have data" **Return information:** | Field | Description | |-------|-------------| | **Available dates** | List of dates with RSS data | | **Total date count** | How many days of data total | | **Today's feed stats** | Today's data statistics by RSS feed | | **Generation time** | Status generation time | --- ## Search and Retrieval ### Q4: How to search for news containing specific keywords? **You can ask like this:** - "Search for news containing 'artificial intelligence'" - "Find reports about 'Tesla price cut'" - "Search for news about Musk, return top 20" - "Find news about 'iPhone 16' in the last 7 days" - "Find news about 'Tesla' from January 1 to 7, 2025" - "Find the link to the news 'iPhone 16 release'" **Tool return behavior:** - Uses keyword mode search - Default searches today's data - AI automatically converts relative time like "last 7 days", "last week" to specific date ranges - Tool returns up to 50 results - Does not include URL links by default **AI display behavior (Important):** - ⚠️ **AI usually auto-summarizes**, only showing partial search results - ✅ If you want to see all, need to explicitly request: "show all search results" **Can be adjusted:** - Specify time range: - Relative way: "search last week" (AI automatically calculates dates) - Absolute dates: "search from January 1 to 7, 2025" - Specify platform: like "only search Zhihu" - Adjust sorting: like "sort by weight" - Include links: like "need links" --- ### Q4.4: How to search both hot news and RSS content simultaneously? **You can ask like this:** - "Search for 'AI' content, including RSS" - "Find news about 'artificial intelligence', also search RSS subscriptions" - "Search for 'Tesla', both hot news and RSS" **Tool return behavior:** - Hot news results and RSS results are **displayed separately** - Hot news sorted by rank/relevance, RSS sorted by publish time - RSS results do not affect hot news ranking display - Default returns 50 hot news + 20 RSS items **Can be adjusted:** - RSS count: like "return 10 RSS items" - Only search hot news: don't say "including RSS" (default behavior) - Only search RSS: say "only search in RSS" --- ### Q5: How to find related news? **You can ask like this:** - "Find news similar to 'Tesla price cut'" (today) - "Find news related to 'AI breakthrough' from yesterday" (history) - "Search for historical reports about 'Tesla' from last week" (history) - "See if there are reports similar to this news in the last 7 days" (history) **Supported time ranges:** | Method | Description | Example | |--------|-------------|---------| | Not specified | Only query today's data (default) | "Find similar news" | | Preset values | yesterday, last week, last month | "Find related news from yesterday" | | Date range | Specify start and end dates | "Find related reports from Jan 1 to 7" | **Tool return behavior:** - Similarity threshold 0.5 (adjustable) - Tool returns up to 50 results - Sorted by similarity - Does not include URL links by default **AI display behavior (Important):** - ⚠️ **AI usually auto-summarizes**, only showing partial related news - ✅ If you want to see all, need to explicitly request: "show all related news" **Can be adjusted:** - Specify time: like "find from last week" - Adjust threshold: like "similarity above 0.3" - Include links: say "need links" --- ## Trend Analysis ### Q6: How to analyze topic heat trends? **You can ask like this:** - "Analyze the heat trend of 'artificial intelligence' in the last week" - "See if 'Tesla' topic is a flash in the pan or sustained hot topic" - "Detect which topics suddenly went viral today" - "Predict potential hot topics coming up" - "Analyze the lifecycle of 'Bitcoin' in December 2024" **Four analysis modes:** | Mode | Description | Example Question | |------|------|---------| | **Heat trend** | Track topic heat changes | "Analyze 'AI' heat trend" | | **Lifecycle** | Complete cycle from emergence to disappearance | "See if 'XX' is a flash in the pan or sustained hot topic" | | **Anomaly detection** | Identify suddenly viral topics | "What topics suddenly went viral today" | | **Prediction** | Predict future hot topics | "Predict upcoming hot topics" | **Tool return behavior:** - AI automatically converts relative time like "last week" to specific date ranges - Default analyzes last 7 days of data - Statistics by day granularity --- ## Data Insights ### Q7: How to compare different platforms' attention to topics? **You can ask like this:** - "Compare different platforms' attention to 'artificial intelligence' topic" - "See which platform updates most frequently" - "Analyze which keywords often appear together" **Three insight modes:** | Mode | Function | Example Question | | -------------- | ---------------- | -------------------------- | | **Platform Compare** | Compare platform attention | "Compare platforms' attention to 'AI'" | | **Activity Stats** | Count platform posting frequency | "See which platform updates most frequently" | | **Keyword Co-occurrence** | Analyze keyword associations | "Which keywords often appear together" | **Tool return behavior:** - Default uses platform compare mode - Analyzes today's data - Keyword co-occurrence minimum frequency 3 times --- ## Sentiment Analysis ### Q8: How to analyze news sentiment? **You can ask like this:** - "Analyze today's news sentiment" - "See if 'Tesla' related news is positive or negative" - "Analyze different platforms' sentiment towards 'artificial intelligence'" - "See the sentiment of 'Bitcoin' within a week, choose the top 20 most important" **Tool return behavior:** - Default analyzes today's data - Tool returns up to 50 news items - Sorted by weight (prioritizing important news) - Does not include URL links by default **AI display behavior (Important):** - ⚠️ This tool returns **AI prompts**, not direct sentiment analysis results - AI generates sentiment analysis reports based on prompts - Usually displays sentiment distribution, key findings, and representative news **Can be adjusted:** - Specify topic: like "about 'Tesla'" - Specify time: like "last week" - Adjust quantity: like "return top 20" --- ### Q9: How to get deduplicated cross-platform news? **You can ask like this:** - "Help me aggregate today's news, remove duplicates" - "See which news is reported on multiple platforms" - "Show me deduplicated hotspot news" - "Which news are cross-platform hot topics" **Tool functionality:** - Automatically identifies the same event reported by different platforms - Merges similar news into one aggregated news item - Shows platform coverage for each news item - Calculates comprehensive heat weight **Return information:** | Field | Description | |-------|-------------| | **Representative title** | Representative title of this news group | | **Covered platforms** | Which platforms reported this news | | **Platform count** | How many platforms covered | | **Is cross-platform** | Whether it's a cross-platform hot topic | | **Best rank** | Best ranking across platforms | | **Comprehensive weight** | Comprehensive heat score | | **Platform sources** | Detailed info from each platform | **Can be adjusted:** - Specify time: like "from last week" - Adjust similarity threshold: like "stricter matching" or "looser matching" - Specify platform: like "only Zhihu and Weibo" --- ### Q10: How to generate daily or weekly hotspot summaries? **You can ask like this:** - "Generate today's news summary report" - "Give me a weekly hotspot summary" - "Generate news analysis report for the past 7 days" **Report types:** - Daily summary: Summarizes the day's hotspot news - Weekly summary: Summarizes a week's hotspot trends --- ### Q11: How to compare hotspot changes across different periods? **You can ask like this:** - "Compare this week and last week's hotspot changes" - "See what's different between this month and last month" - "Analyze 'artificial intelligence' heat difference in two periods" - "Compare platform activity changes" **Three comparison modes:** | Mode | Description | Use Case | |------|-------------|----------| | **Overview** | News count change, keyword change, TOP news comparison | Quick understanding of overall changes | | **Topic shift** | Rising topics, falling topics, newly appeared topics | Analyze hotspot migration | | **Platform activity** | News count change by platform | Understand platform dynamics | **Time period presets:** - Today / Yesterday - This week / Last week - This month / Last month - Or use custom date range --- ## System Management ### Q12: How to view system configuration? **You can ask like this:** - "View current system configuration" - "Display configuration file content" - "What platforms are available?" - "What's the current weight configuration?" **Can query:** - Available platform list - Crawler configuration (request interval, timeout settings) - Weight configuration (ranking weight, frequency weight) - Notification configuration (Feishu, DingTalk, WeCom, Telegram, Email, ntfy, Bark, Slack, Generic Webhook) --- ### Q13: How to check system running status? **You can ask like this:** - "Check system status" - "Is the system running normally?" - "When was the last crawl?" - "How many days of historical data?" **Return information:** - System version and status - Last crawl time - Historical data days - Health check results --- ### Q13.1: How to check for version updates? **You can ask like this:** - "Check for version updates" - "Is there a new version?" - "Is the current version up to date?" **Return information:** Will check both components' versions simultaneously: | Component | Description | |-----------|-------------| | **TrendRadar** | Core crawler and analysis engine | | **MCP Server** | AI conversation tool service | For each component, you'll get: - Currently installed version - Latest available version - Whether an update is needed - Update recommendation **Can be adjusted:** - If GitHub access is slow, say "check version updates, use proxy http://127.0.0.1:10801" --- ### Q14: How to manually trigger a crawl task? **You can ask like this:** - "Please crawl current Toutiao news" (temporary query) - "Help me fetch latest news from Zhihu and Weibo and save" (persistent) - "Trigger a crawl and save data" (persistent) - "Get real-time data from 36Kr but don't save" (temporary query) **Two modes:** | Mode | Purpose | Example | | -------------- | -------------------- | -------------------- | | **Temporary Crawl** | Only return data without saving | "Crawl Toutiao news" | | **Persistent Crawl** | Save to output folder | "Fetch and save Zhihu news" | **Tool return behavior:** - Default is temporary crawl mode (no save) - Default crawls all platforms - Does not include URL links by default **AI display behavior (Important):** - ⚠️ **AI usually summarizes crawl results**, only showing partial news - ✅ If you want to see all, need to explicitly request: "show all crawled news" **Can be adjusted:** - Specify platform: like "only crawl Zhihu" - Save data: say "and save" or "save locally" - Include links: say "need links" --- ## Storage Sync ### Q15: How to sync data from remote storage to local? **You can ask like this:** - "Sync last 7 days data from remote" - "Pull data from remote storage to local" - "Sync last 30 days of news data" **Use cases:** - Crawler deployed in the cloud (e.g., GitHub Actions), data stored remotely (e.g., Cloudflare R2) - MCP Server deployed locally, needs to pull data from remote for analysis **Return information:** - Number of successfully synced files - List of successfully synced dates - Skipped dates (already exist locally) - Failed dates and error information **Prerequisites:** Need to configure remote storage in config file or set environment variables: - Service endpoint URL - Bucket name - Access key ID - Secret access key --- ### Q16: How to view storage status? **You can ask like this:** - "View current storage status" - "What's the storage configuration" - "How much data is stored locally" - "Is remote storage configured" **Return information:** | Category | Information | |----------|-------------| | **Local Storage** | Data directory, total size, date count, date range | | **Remote Storage** | Whether configured, endpoint URL, bucket name, date count | | **Pull Config** | Whether auto-pull enabled, pull days | --- ### Q17: How to view available data dates? **You can ask like this:** - "What dates are available locally" - "What dates are in remote storage" - "Compare local and remote data dates" - "Which dates only exist remotely" **Three query modes:** | Mode | Description | Example Question | |------|-------------|------------------| | **Local** | View local only | "What dates are available locally" | | **Remote** | View remote only | "What dates are in remote" | | **Compare** | Compare both (default) | "Compare local and remote data" | **Return information (compare mode):** - Dates only existing locally - Dates only existing remotely (useful for deciding which dates to sync) - Dates existing in both places --- ### Q18: How to parse natural language date expressions? (Recommended to use first) **You can ask like this:** - "Parse what days 'this week' is" - "What date range does 'last 7 days' correspond to" - "Last month's date range" - "Help me convert 'last 30 days' to specific dates" **Why is this tool needed?** Users often use natural language like "this week", "last 7 days" to express dates, but different AI models calculating dates on their own will produce inconsistent results. This tool uses server-side precise time calculations to ensure all AI models get consistent date ranges. **Supported date expressions:** | Type | Chinese Expression | English Expression | |------|---------|---------| | Single Day | 今天、昨天 | today, yesterday | | Week | 本周、上周 | this week, last week | | Month | 本月、上月 | this month, last month | | Last N Days | 最近7天、最近30天 | last 7 days, last 30 days | | Dynamic | 最近N天 (any number) | last N days | **Usage advantages:** - ✅ **Consistency**: All AI models get the same date range - ✅ **Accuracy**: Based on server-side precise time calculation - ✅ **Standardization**: Returns standard date format - ✅ **Flexibility**: Supports Chinese/English, dynamic days --- ## Article Content Reading ### Q19: How to read the full content of a news article? **You can ask like this:** - "Help me read the content of this news: https://example.com/news/123" - "Get the article body from this link" - "Read the detailed content of this report" **Tool functionality:** - Converts web pages to clean Markdown format via Jina AI Reader - Automatically removes ads, navigation bars, sidebars, and other noise - Returns LLM-friendly structured content **Typical workflow:** 1. First use `search_news(include_url=True)` to search news and get links 2. Then use `read_article(url=link)` to read the article body 3. AI analyzes, summarizes, translates the Markdown content **Return information:** | Field | Description | |-------|-------------| | **content** | Article body in Markdown format | | **url** | Original link | | **content_length** | Content length (characters) | **Can be adjusted:** - Timeout: like "set timeout to 60 seconds" (default 30 seconds, max 60 seconds) **Notes:** - 5-second interval between requests (built-in rate control) - Uses Jina AI Reader free service (100 RPM limit) - Some paywalled/login-required pages may not be fully accessible --- ### Q20: How to batch read multiple articles? **You can ask like this:** - "Help me read the content of these news articles" - "Batch get the article bodies from these links" - "Read the detailed content of the first 3 search results" **Typical workflow:** 1. First use `search_news(include_url=True)` to search news and get multiple links 2. Then use `read_articles_batch(urls=[...])` to batch read article bodies 3. AI performs comparative analysis, comprehensive reports on multiple articles **Tool limits:** | Limit | Value | |-------|-------| | Max articles per batch | **5** | | Request interval | **5 seconds** | | Estimated time (5 articles) | **25-30 seconds** | **Return information:** | Field | Description | |-------|-------------| | **summary** | Statistics of batch reading | | **articles** | Content and status of each article | | **note** | If any articles were skipped, explains why | **Notes:** - Articles beyond 5 will be automatically skipped - Single article failure doesn't affect other articles - More articles mean longer wait time, please be patient --- ## Notification Push ### Q21: How to send notification messages via MCP? **You can ask like this:** - "Show me which notification channels are configured" - "Send a test message to all channels" - "Push this content to Feishu" - "Send today's news summary to DingTalk and Telegram" **Supported notification channels (9):** | Channel | Message Format | Configuration | |---------|---------------|---------------| | **Feishu** | Plain text | `FEISHU_WEBHOOK_URL` | | **DingTalk** | Markdown | `DINGTALK_WEBHOOK_URL` | | **WeCom** | Markdown | `WEWORK_WEBHOOK_URL` | | **Telegram** | HTML | `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` | | **Email** | HTML | `EMAIL_FROM` + `EMAIL_PASSWORD` + `EMAIL_TO` | | **ntfy** | Markdown | `NTFY_SERVER_URL` + `NTFY_TOPIC` | | **Bark** | Markdown | `BARK_URL` | | **Slack** | mrkdwn | `SLACK_WEBHOOK_URL` | | **Generic Webhook** | Markdown | `GENERIC_WEBHOOK_URL` | **Configuration methods:** - Configure channels in `config.yaml` under `notification.channels` - Or set corresponding environment variables in `.env` file (higher priority) - Both sources are automatically merged, `.env` values override `config.yaml` values **Two tools:** | Tool | Function | Example Question | |------|----------|------------------| | `get_notification_channels` | Detect configured channels and status | "View notification channel config" | | `send_notification` | Send message to specified or all channels | "Send message to Feishu" | **Typical workflow:** 1. Check channel status first: "Show me which notification channels are configured" 2. Send after confirming availability: "Push the following to DingTalk: today's hotspot summary..." 3. Or specify multiple channels: "Send to Feishu and Telegram" 4. Without specifying channels, sends to all configured channels **Message format:** - The tool accepts messages in **Markdown format** - Automatically converts to each channel's required format (Feishu to plain text, Telegram to HTML, Slack to mrkdwn, etc.) - No need to manually handle format differences **Multi-account support:** - Separate multiple URLs/Tokens with `;` in config values to send to multiple accounts - For example: `FEISHU_WEBHOOK_URL=url1;url2` sends to two Feishu groups simultaneously --- ## 💡 Usage Tips ### 1. How to make AI display all data instead of auto-summarizing? **Background**: Sometimes AI automatically summarizes data, only showing partial content, even if the tool returned complete 50 items of data. **If AI still summarizes, you can**: - **Method 1 - Explicit request**: "Please show all news, don't summarize" - **Method 2 - Specify quantity**: "Show all 50 news items" - **Method 3 - Question the behavior**: "Why only showed 15? I want to see all" - **Method 4 - State upfront**: "Query today's news, fully display all results" **Note**: AI may still adjust display method based on context. ### 2. How to combine multiple tools? **Example: In-depth analysis of a topic** 1. Search first: "Search for news about 'artificial intelligence'" 2. Then analyze trends: "Analyze the heat trend of 'artificial intelligence'" 3. Finally sentiment analysis: "Analyze sentiment of 'artificial intelligence' news" **Example: Track an event** 1. View latest: "Query today's news about 'iPhone'" 2. Find history: "Find historical news related to 'iPhone' from last week" 3. Find similar reports: "Find news similar to 'iPhone launch event'" ================================================ FILE: README-MCP-FAQ.md ================================================
**中文** | **[English](README-MCP-FAQ-EN.md)**
# TrendRadar MCP 工具使用问答 > AI 提问指南 - 如何通过自然对话使用新闻热点分析工具（v3.1.7） --- ## 📋 工具一览 | 分类 | 工具名称 | 功能简介 | |:----:|---------|---------| | **日期** | `resolve_date_range` | 解析"本周"、"最近7天"等自然语言为标准日期 | | **查询** | `get_latest_news` | 获取最新一批爬取的热榜新闻 | | | `get_news_by_date` | 按日期范围查询历史新闻 | | | `get_trending_topics` | 获取热点话题统计（支持自动提取） | | **RSS** | `get_latest_rss` | 获取最新 RSS 订阅内容 | | | `search_rss` | 在 RSS 数据中搜索关键词 | | | `get_rss_feeds_status` | 查看 RSS 源配置和数据状态 | | **搜索** | `search_news` | 统一搜索（关键词/模糊/实体，可含RSS） | | | `find_related_news` | 查找与指定标题相似的新闻 | | **分析** | `analyze_topic_trend` | 话题趋势分析（热度/生命周期/爆火/预测） | | | `analyze_data_insights` | 数据洞察（平台对比/活跃度/关键词共现） | | | `analyze_sentiment` | 新闻情感倾向分析 | | | `aggregate_news` | 跨平台新闻聚合去重 | | | `compare_periods` | 时期对比分析（周环比/月环比） | | | `generate_summary_report` | 生成每日/每周摘要报告 | | **系统** | `get_current_config` | 获取当前系统配置 | | | `get_system_status` | 获取系统运行状态 | | | `check_version` | 检查版本更新（TrendRadar + MCP Server） | | | `trigger_crawl` | 手动触发一次爬取任务 | | **存储** | `sync_from_remote` | 从远程存储拉取数据到本地 | | | `get_storage_status` | 获取存储配置和状态 | | | `list_available_dates` | 列出本地/远程可用的日期 | | **文章** | `read_article` | 读取单篇文章内容（Markdown 格式） | | | `read_articles_batch` | 批量读取多篇文章（最多 5 篇） | | **通知** | `get_notification_channels` | 获取所有已配置的通知渠道及其状态 | | | `send_notification` | 向已配置的通知渠道发送消息（自动格式转换） | --- ## ⚙️ 默认设置说明（重要！） 默认采用以下优化策略，主要是为了节约 AI token 消耗： | 默认设置 | 说明 | 如何调整 | | -------------- | --------------------------------------- | ------------------------------------- | | **限制条数** | 默认返回 50 条新闻 | 对话中说"返回前 10 条"或"给我 100 条" | | **时间范围** | 默认查询今天的数据 | 说"查询昨天"、"最近一周"或"1月1日到7日" | | **URL 链接** | 默认不返回链接（节省约 160 tokens/条） | 说"需要链接"或"包含 URL" | | **关键词列表** | 默认不使用 frequency_words.txt 过滤新闻 | 只有调用"趋势话题"工具时才使用 | **⚠️ 重要：** AI 模型的选择直接影响工具调用效果，AI 越智能，调用越准确。当你解除上面的限制，比如从今天的查询，放宽到一周的查询，首先你要在本地有一周的数据，其次，token 消耗量可能会倍增。 **💡 提示：** 本项目提供了专门的日期解析工具，可以准确解析"最近7天"、"本周"等自然语言日期表达式，确保所有 AI 模型获得一致的日期范围。详见下方 Q18。 ## 💰 AI 模型 下面我以 **[硅基流动](https://cloud.siliconflow.cn)** 平台作为例子，里面有很多大模型可选择。在开发和测试本项目的过程中，我使用本平台进行了许多的功能测试和验证。 ### 📊 注册方式对比 | 注册方式 | 无邀请链接直接注册 | 含有邀请链接注册 | |:-------:|:-------:|:-----------------:| | 注册链接 | [siliconflow.cn](https://cloud.siliconflow.cn) | [邀请链接](https://cloud.siliconflow.cn/i/fqnyVaIU) | | 免费额度 | 0 tokens | **2000万 tokens** (≈14元) | | 额外福利 | ❌ | ✅ 邀请者也获得2000万tokens | > 💡 **提示**：上面的赠送额度，应该可以询问 **200次以上** ### 🚀 快速开始 #### 1️⃣ 注册并获取 API 密钥 1. 使用上方链接完成注册 2. 访问 [API 密钥管理页面](https://cloud.siliconflow.cn/me/account/ak) 3. 点击「新建 API 密钥」 4. 复制生成的密钥（请妥善保管） #### 2️⃣ 在 Cherry Studio 中配置 1. 打开 **Cherry Studio** 2. 进入「模型服务」设置 3. 找到「硅基流动」 4. 将复制的密钥粘贴到 **[API密钥]** 输入框 5. 确保右上角勾选框打开后显示为 **绿色** ✅ --- ### ✨ 配置完成！ 现在你可以开始使用本项目，享受稳定快速的 AI 服务了！ 在你测试一次询问后，请立刻去 [硅基流动账单](https://cloud.siliconflow.cn/me/bills) 查询这一次的消耗量，心底有个估算。 --- ## 基础查询 ### Q1: 如何查看最新的新闻？ **你可以这样问：** - "给我看看最新的新闻" - "查询今天的热点新闻" - "获取知乎和微博的最新 10 条新闻" - "查看最新新闻，需要包含链接" **工具返回行为：** - 工具会返回所有平台的最新 50 条新闻 - 默认不包含 URL 链接（节省 token） **AI 展示行为（重要）：** - ⚠️ **AI 通常会自动总结**，只展示部分新闻（如 TOP 10-20 条） - ✅ 如果你想看全部 50 条，需要明确要求："展示所有新闻"或"完整列出所有 50 条" - 💡 这是 AI 模型的自然行为，不是工具的限制 **可以调整：** - 指定平台：如"只看知乎的" - 调整数量：如"返回前 20 条" - 包含链接：如"需要链接" - **要求完整展示**：如"展示全部，不要总结" --- ### Q2: 如何查询特定日期的新闻？ **你可以这样问：** - "查询昨天的新闻" - "看看 3 天前知乎的新闻" - "2025-10-10 的新闻有哪些" - "上周一的新闻" - "给我看看最新新闻"（自动查询今天） **支持的日期格式：** - 相对日期：今天、昨天、前天、3 天前 - 星期：上周一、本周三、last monday - 绝对日期：2025-10-10、10 月 10 日 **工具返回行为：** - 不指定日期时自动查询今天（节省 token） - 工具会返回所有平台的 50 条新闻 - 默认不包含 URL 链接 **AI 展示行为（重要）：** - ⚠️ **AI 通常会自动总结**，只展示部分新闻（如 TOP 10-20 条） - ✅ 如果你想看全部，需要明确要求："展示所有新闻，不要总结" --- ### Q3: 如何查看热点话题统计？ **你可以这样问：** - "我关注的词今天出现了多少次"（使用预设关注词） - "自动分析今天新闻里有哪些热门话题"（自动提取） - "看看新闻里最热门的词是什么"（自动提取） **两种提取模式：** | 模式 | 说明 | 示例问法 | |------|------|---------| | **预设关注词** | 统计你预先设定的关注词（基于配置文件，默认） | "我的关注词出现了多少次" | | **自动提取** | 自动从新闻标题提取高频词（无需预设） | "自动分析热门话题" | --- ## RSS 订阅查询 ### Q4.1: 如何查看最新的 RSS 订阅内容？ **你可以这样问：** - "查看最新的 RSS 订阅内容" - "获取 Hacker News 的最新文章" - "查看所有 RSS 源的最新 20 条" - "获取 RSS 订阅，需要包含摘要" - "看看最近一周的 RSS 内容"（支持多日查询） - "获取 Hacker News 最近 7 天的文章" **工具返回行为：** - 默认返回今天的 RSS 条目（最多 50 条） - 支持 `days` 参数获取多日数据（1-30天） - 默认不包含摘要（节省 token） - 按发布时间倒序排列 - 跨日期自动去重（按 URL） **AI 展示行为（重要）：** - ⚠️ **AI 通常会自动总结**，只展示部分条目 - ✅ 如果你想看全部，需要明确要求："展示所有 RSS 内容" **可以调整：** - 指定 RSS 源：如"只看 Hacker News" - 指定天数：如"最近 7 天"、"最近一周" - 调整数量：如"返回前 20 条" - 包含摘要：如"需要摘要" --- ### Q4.2: 如何搜索 RSS 订阅中的内容？ **你可以这样问：** - "在 RSS 中搜索'AI'相关的文章" - "搜索最近 7 天 RSS 中关于'机器学习'的内容" - "在 Hacker News 中搜索'Python'" **工具返回行为：** - 使用关键词搜索 RSS 条目的标题 - 默认搜索最近 7 天的数据 - 工具会返回最多 50 条结果 **可以调整：** - 指定 RSS 源：如"只搜索 Hacker News" - 调整天数：如"搜索最近 14 天" - 包含摘要：如"需要摘要" --- ### Q4.3: 如何查看 RSS 源的状态？ **你可以这样问：** - "查看 RSS 源状态" - "RSS 抓取了多少数据" - "哪些 RSS 源有数据" **返回信息：** | 字段 | 说明 | |------|------| | **可用日期** | 有 RSS 数据的日期列表 | | **总日期数** | 总共有多少天的数据 | | **今日各源统计** | 今日各 RSS 源的数据统计 | | **生成时间** | 状态生成时间 | --- ## 搜索检索 ### Q4: 如何搜索包含特定关键词的新闻？ **你可以这样问：** - "搜索包含'人工智能'的新闻" - "查找关于'特斯拉降价'的报道" - "搜索马斯克相关的新闻，返回前 20 条" - "查找最近7天关于'iPhone 16'的新闻" - "查找2025年1月1日到7日'特斯拉'的相关新闻" - "查找'iPhone 16 发布'这条新闻的链接" **工具返回行为：** - 使用关键词模式搜索 - 默认搜索今天的数据 - AI会自动将"最近7天"、"上周"等相对时间转换为具体日期范围 - 工具会返回最多 50 条结果 - 默认不包含 URL 链接 **AI 展示行为（重要）：** - ⚠️ **AI 通常会自动总结**，只展示部分搜索结果 - ✅ 如果你想看全部，需要明确要求："展示所有搜索结果" **可以调整：** - 指定时间范围： - 相对方式："搜索最近一周的"（AI 自动计算日期） - 绝对日期："搜索2025年1月1日到7日的" - 指定平台：如"只搜索知乎" - 调整排序：如"按权重排序" - 包含链接：如"需要链接" --- ### Q4.4: 如何同时搜索热榜和 RSS 内容？ **你可以这样问：** - "搜索'AI'相关内容，包括 RSS" - "查找'人工智能'的新闻，同时搜索 RSS 订阅" - "搜索'特斯拉'，热榜和 RSS 都要" **工具返回行为：** - 热榜结果和 RSS 结果**分开展示** - 热榜按排名/相关度排序，RSS 按发布时间排序 - RSS 结果不影响热榜的排名展示 - 默认返回热榜 50 条 + RSS 20 条 **可以调整：** - RSS 数量：如"RSS 返回 10 条" - 只搜索热榜：不说"包括 RSS"（默认行为） - 只搜索 RSS：说"只在 RSS 中搜索" --- ### Q5: 如何查找相关新闻？ **你可以这样问：** - "找出和'特斯拉降价'相似的新闻"（今天） - "查找昨天与'人工智能突破'相关的新闻"（历史） - "搜索上周关于'ChatGPT'的相关报道"（历史） - "看看最近7天有没有和这条新闻相似的报道"（历史） **支持的时间范围：** | 方式 | 说明 | 示例 | |------|------|------| | 不指定 | 只查询今天的数据（默认） | "找相似新闻" | | 预设值 | 昨天、上周、上个月 | "查找昨天的相关新闻" | | 日期范围 | 指定开始和结束日期 | "查找1月1日到7日的相关报道" | **工具返回行为：** - 相似度阈值 0.5（可调整） - 工具会返回最多 50 条结果 - 按相似度排序 - 默认不包含 URL 链接 **AI 展示行为（重要）：** - ⚠️ **AI 通常会自动总结**，只展示部分相关新闻 - ✅ 如果你想看全部，需要明确要求："展示所有相关新闻" **可以调整：** - 指定时间：如"查找上周的" - 调整阈值：如"相似度 0.3 以上的都要" - 包含链接：说"需要链接" --- ## 趋势分析 ### Q6: 如何分析话题的热度趋势？ **你可以这样问：** - "分析'人工智能'最近一周的热度趋势" - "看看'特斯拉'话题是昙花一现还是持续热点" - "检测今天有哪些突然爆火的话题" - "预测接下来可能的热点话题" - "分析'比特币'在2024年12月的生命周期" **四种分析模式：** | 模式 | 说明 | 示例问法 | |------|------|---------| | **热度趋势** | 追踪话题热度变化 | "分析'AI'的热度趋势" | | **生命周期** | 从出现到消失的完整周期 | "看看'XX'是昙花一现还是持续热点" | | **异常检测** | 识别突然爆火的话题 | "今天有哪些突然爆火的话题" | | **预测** | 预测未来可能的热点 | "预测接下来可能的热点" | **工具返回行为：** - AI会自动将"最近一周"等相对时间转换为具体日期范围 - 默认分析最近7天数据 - 按天粒度统计 --- ## 数据洞察 ### Q7: 如何对比不同平台对话题的关注度？ **你可以这样问：** - "对比各个平台对'人工智能'话题的关注度" - "看看哪个平台更新最频繁" - "分析一下哪些关键词经常一起出现" **三种洞察模式：** | 模式 | 功能 | 示例问法 | | -------------- | ---------------- | -------------------------- | | **平台对比** | 对比各平台关注度 | "对比各平台对'AI'的关注度" | | **活跃度统计** | 统计平台发布频率 | "看看哪个平台更新最频繁" | | **关键词共现** | 分析关键词关联 | "哪些关键词经常一起出现" | **工具返回行为：** - 默认使用平台对比模式 - 分析今天的数据 - 关键词共现最小频次 3 次 --- ## 情感分析 ### Q8: 如何分析新闻的情感倾向？ **你可以这样问：** - "分析一下今天新闻的情感倾向" - "看看'特斯拉'相关新闻是正面还是负面的" - "分析各平台对'人工智能'的情感态度" - "看看'比特币'一周内的情感倾向，选择前 20 条最重要的" **工具返回行为：** - 默认分析今天的数据 - 工具会返回最多 50 条新闻 - 按权重排序（优先展示重要新闻） - 默认不包含 URL 链接 **AI 展示行为（重要）：** - ⚠️ 本工具返回 **AI 提示词**，不是直接的情感分析结果 - AI 会根据提示词生成情感分析报告 - 通常会展示情感分布、关键发现和代表性新闻 **可以调整：** - 指定话题：如"关于'特斯拉'" - 指定时间：如"最近一周" - 调整数量：如"返回前 20 条" --- ### Q9: 如何获取去重后的跨平台新闻？ **你可以这样问：** - "帮我聚合今天的新闻，去掉重复的" - "看看哪些新闻在多个平台都有报道" - "给我看去重后的热点新闻" - "哪些新闻是跨平台热点" **工具功能：** - 自动识别不同平台报道的同一事件 - 将相似新闻合并为一条聚合新闻 - 显示每条新闻的平台覆盖情况 - 计算综合热度权重 **返回信息：** | 字段 | 说明 | |------|------| | **代表性标题** | 这组新闻的代表标题 | | **覆盖平台** | 哪些平台报道了这条新闻 | | **平台数量** | 覆盖了多少个平台 | | **是否跨平台** | 是否为跨平台热点 | | **最佳排名** | 在各平台的最佳排名 | | **综合权重** | 综合热度评分 | | **各平台来源** | 各平台的详细信息 | **可以调整：** - 指定时间：如"最近一周的" - 调整相似度阈值：如"更严格匹配"或"宽松匹配" - 指定平台：如"只看知乎和微博" --- ### Q10: 如何生成每日或每周的热点摘要？ **你可以这样问：** - "生成今天的新闻摘要报告" - "给我一份本周的热点总结" - "生成过去 7 天的新闻分析报告" **报告类型：** - 每日摘要：总结当天的热点新闻 - 每周摘要：总结一周的热点趋势 --- ### Q11: 如何对比不同时期的热点变化？ **你可以这样问：** - "对比本周和上周的热点变化" - "看看这个月和上个月有什么不同" - "分析'人工智能'在两个时期的热度差异" - "对比各平台活跃度的变化" **三种对比模式：** | 模式 | 说明 | 适用场景 | |------|------|---------| | **总体概览** | 新闻数量变化、关键词变化、TOP新闻对比 | 快速了解整体变化 | | **话题变化** | 上升话题、下降话题、新出现话题 | 分析热点转移 | | **平台活跃度** | 各平台新闻数量变化 | 了解平台动态 | **时间段预设值：** - 今天 / 昨天 - 本周 / 上周 - 本月 / 上月 - 或使用自定义日期范围 --- ## 系统管理 ### Q12: 如何查看系统配置？ **你可以这样问：** - "查看当前系统配置" - "显示配置文件内容" - "有哪些可用的平台？" - "当前的权重配置是什么？" **可以查询：** - 可用平台列表 - 爬虫配置（请求间隔、超时设置） - 权重配置（排名权重、频次权重） - 通知配置（飞书、钉钉、企业微信、Telegram、Email、ntfy、Bark、Slack、通用 Webhook） --- ### Q13: 如何检查系统运行状态？ **你可以这样问：** - "检查系统状态" - "系统运行正常吗？" - "最后一次爬取是什么时候？" - "有多少天的历史数据？" **返回信息：** - 系统版本和状态 - 最后爬取时间 - 历史数据天数 - 健康检查结果 --- ### Q13.1: 如何检查版本更新？ **你可以这样问：** - "检查版本更新" - "有没有新版本？" - "当前版本是最新的吗？" **返回信息：** 会同时检查两个组件的版本： | 组件 | 说明 | |------|------| | **TrendRadar** | 核心爬虫和分析引擎 | | **MCP Server** | AI 对话工具服务 | 每个组件会告诉你： - 当前安装的版本 - 最新可用的版本 - 是否需要更新 - 更新建议 **可以调整：** - 如果访问 GitHub 较慢，可以说"检查版本更新，使用代理 http://127.0.0.1:10801" --- ### Q14: 如何手动触发爬取任务？ **你可以这样问：** - "请你爬取当前的今日头条的新闻"（临时查询） - "帮我抓取一下知乎和微博的最新新闻并保存"（持久化） - "触发一次爬取并保存数据"（持久化） - "获取 36 氪 的实时数据但不保存"（临时查询） **两种模式：** | 模式 | 用途 | 示例 | | -------------- | -------------------- | -------------------- | | **临时爬取** | 只返回数据不保存 | "爬取今日头条的新闻" | | **持久化爬取** | 保存到 output 文件夹 | "抓取并保存知乎新闻" | **工具返回行为：** - 默认为临时爬取模式（不保存） - 默认爬取所有平台 - 默认不包含 URL 链接 **AI 展示行为（重要）：** - ⚠️ **AI 通常会总结爬取结果**，只展示部分新闻 - ✅ 如果你想看全部，需要明确要求："展示所有爬取的新闻" **可以调整：** - 指定平台：如"只爬取知乎" - 保存数据：说"并保存"或"保存到本地" - 包含链接：说"需要链接" --- ## 存储同步 ### Q15: 如何从远程存储同步数据到本地？ **你可以这样问：** - "从远程同步最近 7 天的数据" - "拉取远程存储的数据到本地" - "同步最近 30 天的新闻数据" **使用场景：** - 爬虫部署在云端（如 GitHub Actions），数据存储到远程（如 Cloudflare R2） - MCP Server 部署在本地，需要从远程拉取数据进行分析 **返回信息：** - 成功同步的文件数量 - 成功同步的日期列表 - 跳过的日期（本地已存在） - 失败的日期及错误信息 **前提条件：** 需要在配置文件中配置远程存储或设置环境变量： - 服务端点 URL - 存储桶名称 - 访问密钥 ID - 访问密钥 --- ### Q16: 如何查看存储状态？ **你可以这样问：** - "查看当前存储状态" - "存储配置是什么" - "本地有多少数据" - "远程存储配置了吗" **返回信息：** | 类别 | 信息 | |------|------| | **本地存储** | 数据目录、总大小、日期数量、日期范围 | | **远程存储** | 是否配置、端点地址、存储桶名称、日期数量 | | **拉取配置** | 是否启用自动拉取、拉取天数 | --- ### Q17: 如何查看可用的数据日期？ **你可以这样问：** - "本地有哪些日期的数据" - "远程存储有哪些日期" - "对比本地和远程的数据日期" - "哪些日期只在远程有" **三种查询模式：** | 模式 | 说明 | 示例问法 | |------|------|---------| | **本地** | 仅查看本地 | "本地有哪些日期" | | **远程** | 仅查看远程 | "远程有哪些日期" | | **对比** | 对比两者（默认） | "对比本地和远程的数据" | **返回信息（对比模式）：** - 仅本地存在的日期 - 仅远程存在的日期（可用于决定同步哪些日期） - 两边都存在的日期 --- ### Q18: 如何解析自然语言日期表达式？（推荐优先使用） **你可以这样问：** - "解析'本周'是哪几天" - "最近7天对应的日期范围是什么" - "上月的日期范围" - "帮我把'最近30天'转换为具体日期" **为什么需要这个工具？** 用户经常使用"本周"、"最近7天"等自然语言表达日期，但不同的 AI 模型自行计算日期时会产生不一致的结果。此工具使用服务器端的精确时间计算，确保所有 AI 模型获得一致的日期范围。 **支持的日期表达式：** | 类型 | 中文表达 | 英文表达 | |------|---------|---------| | 单日 | 今天、昨天 | today, yesterday | | 周 | 本周、上周 | this week, last week | | 月 | 本月、上月 | this month, last month | | 最近N天 | 最近7天、最近30天 | last 7 days, last 30 days | | 动态 | 最近N天（任意数字） | last N days | **使用优势：** - ✅ **一致性**：所有 AI 模型获得相同的日期范围 - ✅ **准确性**：基于服务器端精确时间计算 - ✅ **标准化**：返回标准日期格式 - ✅ **灵活性**：支持中英文、动态天数 --- ## 文章内容读取 ### Q19: 如何读取新闻文章的正文内容？ **你可以这样问：** - "帮我读取这篇新闻的内容：https://example.com/news/123" - "获取这个链接的文章正文" - "读取这篇报道的详细内容" **工具功能：** - 通过 Jina AI Reader 将网页转换为干净的 Markdown 格式 - 自动去除广告、导航栏、侧边栏等噪音内容 - 返回 LLM 友好的结构化内容 **典型使用流程：** 1. 先用 `search_news(include_url=True)` 搜索新闻获取链接 2. 再用 `read_article(url=链接)` 读取正文内容 3. AI 对 Markdown 正文进行分析、摘要、翻译等 **返回信息：** | 字段 | 说明 | |------|------| | **content** | Markdown 格式的文章正文 | | **url** | 原始链接 | | **content_length** | 内容长度（字符数） | **可以调整：** - 超时时间：如"超时设为 60 秒"（默认 30 秒，最大 60 秒） **注意事项：** - 每次请求间隔 5 秒（内置速率控制） - 使用 Jina AI Reader 免费服务（100 RPM 限制） - 部分付费墙/登录墙页面可能无法完整获取 --- ### Q20: 如何批量读取多篇文章？ **你可以这样问：** - "帮我读取这几篇新闻的内容" - "批量获取这些链接的文章正文" - "读取搜索结果中前 3 篇的详细内容" **典型使用流程：** 1. 先用 `search_news(include_url=True)` 搜索新闻获取多个链接 2. 再用 `read_articles_batch(urls=[...])` 批量读取正文 3. AI 对多篇文章进行对比分析、综合报告 **工具限制：** | 限制 | 值 | |------|------| | 单次最多篇数 | **5 篇** | | 请求间隔 | **5 秒** | | 预计耗时（5篇） | **25-30 秒** | **返回信息：** | 字段 | 说明 | |------|------| | **summary** | 批量读取的统计信息 | | **articles** | 每篇文章的内容和状态 | | **note** | 如有跳过的文章，会说明原因 | **注意事项：** - 超出 5 篇的部分会被自动跳过 - 单篇失败不影响其他篇的读取 - 篇数越多耗时越长，请耐心等待 --- ## 通知推送 ### Q21: 如何通过 MCP 发送通知消息？ **你可以这样问：** - "查看当前配置了哪些通知渠道" - "发送一条测试消息到所有渠道" - "把这段内容推送到飞书" - "发送今天的新闻摘要到钉钉和 Telegram" **支持的通知渠道（9 个）：** | 渠道 | 消息格式 | 配置来源 | |------|---------|---------| | **飞书** (feishu) | 纯文本 | `FEISHU_WEBHOOK_URL` | | **钉钉** (dingtalk) | Markdown | `DINGTALK_WEBHOOK_URL` | | **企业微信** (wework) | Markdown | `WEWORK_WEBHOOK_URL` | | **Telegram** | HTML | `TELEGRAM_BOT_TOKEN` + `TELEGRAM_CHAT_ID` | | **Email** | HTML | `EMAIL_FROM` + `EMAIL_PASSWORD` + `EMAIL_TO` | | **ntfy** | Markdown | `NTFY_SERVER_URL` + `NTFY_TOPIC` | | **Bark** | Markdown | `BARK_URL` | | **Slack** | mrkdwn | `SLACK_WEBHOOK_URL` | | **通用 Webhook** | Markdown | `GENERIC_WEBHOOK_URL` | **配置方式：** - 在 `config.yaml` 的 `notification.channels` 中配置对应渠道 - 或在 `.env` 文件中设置对应的环境变量（优先级更高） - 两种方式会自动合并，`.env` 中的值会覆盖 `config.yaml` 中的值 **两个工具：** | 工具 | 功能 | 示例问法 | |------|------|---------| | `get_notification_channels` | 检测已配置的渠道及状态 | "查看通知渠道配置" | | `send_notification` | 发送消息到指定或全部渠道 | "发送消息到飞书" | **典型使用流程：** 1. 先查看渠道状态："查看当前配置了哪些通知渠道" 2. 确认渠道可用后发送："把以下内容推送到钉钉：今日热点摘要..." 3. 或指定多个渠道："发送到飞书和 Telegram" 4. 不指定渠道则发送到所有已配置渠道 **消息格式：** - 工具接受 **Markdown 格式** 的消息内容 - 自动按各渠道要求转换格式（飞书转纯文本、Telegram 转 HTML、Slack 转 mrkdwn 等） - 无需手动处理格式差异 **多账号支持：** - 配置值中用 `;` 分隔多个 URL/Token 即可发送到多个账号 - 例如：`FEISHU_WEBHOOK_URL=url1;url2` 会同时发送到两个飞书群 --- ## 💡 使用技巧 ### 1. 如何让 AI 展示全部数据而不是自动总结？ **背景**: 有时 AI 会自动总结数据，只展示部分内容，即使工具返回了完整的 50 条数据。 **如果 AI 仍然总结，你可以**: - **方法 1 - 明确要求**: "请展示全部新闻，不要总结" - **方法 2 - 指定数量**: "展示所有 50 条新闻" - **方法 3 - 质疑行为**: "为什么只展示了 15 条？我要看全部" - **方法 4 - 提前说明**: "查询今天的新闻，完整展示所有结果" **注意**: AI 仍可能根据上下文调整展示方式。 ### 2. 如何组合使用多个工具？ **示例：深度分析某个话题** 1. 先搜索："搜索'人工智能'相关新闻" 2. 再分析趋势："分析'人工智能'的热度趋势" 3. 最后情感分析："分析'人工智能'新闻的情感倾向" **示例：追踪某个事件** 1. 查看最新："查询今天关于'iPhone'的新闻" 2. 查找历史："查找上周与'iPhone'相关的历史新闻" 3. 找相似报道："找出和'iPhone 发布会'相似的新闻" ================================================ FILE: README.md ================================================
TrendRadar Banner 最快30秒部署的热点助手 —— 告别无效刷屏，只看真正关心的新闻资讯 sansan0%2FTrendRadar | Trendshift [![GitHub Stars](https://img.shields.io/github/stars/sansan0/TrendRadar?style=flat-square&logo=github&color=yellow)](https://github.com/sansan0/TrendRadar/stargazers) [![GitHub Forks](https://img.shields.io/github/forks/sansan0/TrendRadar?style=flat-square&logo=github&color=blue)](https://github.com/sansan0/TrendRadar/network/members) [![License](https://img.shields.io/badge/license-GPL--3.0-blue.svg?style=flat-square)](LICENSE) [![Version](https://img.shields.io/badge/version-v6.5.0-blue.svg)](https://github.com/sansan0/TrendRadar) [![MCP](https://img.shields.io/badge/MCP-v4.0.0-green.svg)](https://github.com/sansan0/TrendRadar) [![RSS](https://img.shields.io/badge/RSS-订阅源支持-orange.svg?style=flat-square&logo=rss&logoColor=white)](https://github.com/sansan0/TrendRadar) [![AI翻译](https://img.shields.io/badge/AI-多语言推送-purple.svg?style=flat-square)](https://github.com/sansan0/TrendRadar) [![企业微信通知](https://img.shields.io/badge/企业微信-通知-00D4AA?style=flat-square)](https://work.weixin.qq.com/) [![个人微信通知](https://img.shields.io/badge/个人微信-通知-00D4AA?style=flat-square)](https://weixin.qq.com/) [![Telegram通知](https://img.shields.io/badge/Telegram-通知-00D4AA?style=flat-square)](https://telegram.org/) [![dingtalk通知](https://img.shields.io/badge/钉钉-通知-00D4AA?style=flat-square)](#) [![飞书通知](https://img.shields.io/badge/飞书-通知-00D4AA?style=flat-square)](https://www.feishu.cn/) [![邮件通知](https://img.shields.io/badge/Email-通知-00D4AA?style=flat-square)](#) [![ntfy通知](https://img.shields.io/badge/ntfy-通知-00D4AA?style=flat-square)](https://github.com/binwiederhier/ntfy) [![Bark通知](https://img.shields.io/badge/Bark-通知-00D4AA?style=flat-square)](https://github.com/Finb/Bark) [![Slack通知](https://img.shields.io/badge/Slack-通知-00D4AA?style=flat-square)](https://slack.com/) [![通用Webhook](https://img.shields.io/badge/通用-Webhook-607D8B?style=flat-square&logo=webhook&logoColor=white)](#) [![GitHub Actions](https://img.shields.io/badge/GitHub_Actions-自动化-2088FF?style=flat-square&logo=github-actions&logoColor=white)](https://github.com/sansan0/TrendRadar) [![GitHub Pages](https://img.shields.io/badge/GitHub_Pages-部署-4285F4?style=flat-square&logo=github&logoColor=white)](https://sansan0.github.io/TrendRadar) [![Docker](https://img.shields.io/badge/Docker-部署-2496ED?style=flat-square&logo=docker&logoColor=white)](https://hub.docker.com/r/wantcat/trendradar) [![MCP Support](https://img.shields.io/badge/MCP-AI分析支持-FF6B6B?style=flat-square&logo=ai&logoColor=white)](https://modelcontextprotocol.io/) [![AI分析推送](https://img.shields.io/badge/AI-分析推送-FF6B6B?style=flat-square&logo=openai&logoColor=white)](#) [![AI智能筛选](https://img.shields.io/badge/AI-智能筛选新闻-9B59B6?style=flat-square&logo=openai&logoColor=white)](#)
**中文** | **[English](README-EN.md)**
> 本项目以轻量，易部署为目标
## 📑 快速导航 > 💡 **点击下方链接**可快速跳转到对应章节。部署推荐从「**快速开始**」入手，需要详细自定义请看「**配置详解**」
| | | | |:---:|:---:|:---:| | [🚀 **快速开始**](#-快速开始) | [AI 智能分析](#-ai-智能分析) | [⚙️ **配置详解**](#配置详解) | | [Docker部署](#6-docker-部署) | [MCP客户端](#-mcp-客户端) | [📝 **更新日志**](#-更新日志) | | [🎯 **核心功能**](#-核心功能) | [☕ **支持项目**](#-支持项目) | [📚 **项目相关**](#-项目相关) |

- 感谢**为项目点 star** 的观众们，**fork** 你所欲也，**star** 我所欲也，两者得兼😍是对开源精神最好的支持
👉 点击展开：致谢名单 (天使轮荣誉榜 🔥73+🔥 位) ### 早期支持者致谢 > 💡 **特别说明**： > > 1. **关于名单**：下方表格记录了项目起步阶段（天使轮）的支持者。因早期人工统计繁琐，**难免存在疏漏或记录不全的情况，如有遗漏，实非本意，万望海涵**。 > 2. **未来规划**：为了将有限的精力回归代码与功能迭代，**即日起不再人工维护此名单**。 > > 无论名字是否上榜，你们的每一份支持都是 TrendRadar 能够走到今天的基石。🙏 ### 基础设施支持 感谢 **GitHub** 免费提供的基础设施，这是本项目得以**一键 fork**便捷运行的最大前提。 ### 数据支持 本项目使用 [newsnow](https://github.com/ourongxing/newsnow) 项目的 API 获取多平台数据，特别感谢作者提供的服务。 经联系，作者表示无需担心服务器压力，但这是基于他的善意和信任。请大家： - **前往 [newsnow 项目](https://github.com/ourongxing/newsnow) 点 star 支持** - Docker 部署时，请合理控制推送频率，勿竭泽而渔 ### 推广助力 > 感谢以下平台和个人的推荐(按时间排列) - [小众软件](https://mp.weixin.qq.com/s/fvutkJ_NPUelSW9OGK39aA) - 开源软件推荐平台 - [LinuxDo 社区](https://linux.do/) - 技术爱好者的聚集地 - [阮一峰周刊](https://github.com/ruanyf/weekly) - 技术圈有影响力的周刊 ### 观众支持 > 感谢**给予资金支持**的朋友们，你们的慷慨已化身为键盘旁的零食饮料，陪伴着项目的每一次迭代。 > > **关于"一元点赞"的回归**： > 随着 v5.0.0 版本的发布，项目迈入了一个新的阶段。为了支持日益增长的 API 成本和咖啡因消耗，"一元点赞"通道现已重新开启。你的每一份心意，都将转化为代码世界里的 Token 和动力。🚀 [前往支持](#-支持项目) | 点赞人 | 金额 | 日期 | 备注 | | :-------------------------: | :----: | :----: | :-----------------------: | | D*5 | 1.8 * 3 | 2025.11.24 | | | *鬼 | 1 | 2025.11.17 | | | *超 | 10 | 2025.11.17 | | | R*w | 10 | 2025.11.17 | 这 agent 做的牛逼啊,兄弟 | | J*o | 1 | 2025.11.17 | 感谢开源,祝大佬事业有成 | | *晨 | 8.88 | 2025.11.16 | 项目不错,研究学习中 | | *海 | 1 | 2025.11.15 | | | *德 | 1.99 | 2025.11.15 | | | *疏 | 8.8 | 2025.11.14 | 感谢开源，项目很棒，支持一下 | | M*e | 10 | 2025.11.14 | 开源不易，大佬辛苦了 | | **柯 | 1 | 2025.11.14 | | | *云 | 88 | 2025.11.13 | 好项目，感谢开源 | | *W | 6 | 2025.11.13 | | | *凯 | 1 | 2025.11.13 | | | 对*. | 1 | 2025.11.13 | Thanks for your TrendRadar | | s*y | 1 | 2025.11.13 | | | **翔 | 10 | 2025.11.13 | 好项目，相见恨晚，感谢开源！ | | *韦 | 9.9 | 2025.11.13 | TrendRadar超赞，请老师喝咖啡~ | | h*p | 5 | 2025.11.12 | 支持中国开源力量，加油！ | | c*r | 6 | 2025.11.12 | | | a*n | 5 | 2025.11.12 | | | 。*c | 1 | 2025.11.12 | 感谢开源分享 | | *记 | 1 | 2025.11.11 | | | *主 | 1 | 2025.11.10 | | | *了 | 10 | 2025.11.09 | | | *杰 | 5 | 2025.11.08 | | | *点 | 8.80 | 2025.11.07 | 开发不易，支持一下。 | | Q*Q | 6.66 | 2025.11.07 | 感谢开源！ | | C*e | 1 | 2025.11.05 | | | Peter Fan | 20 | 2025.10.29 | | | M*n | 1 | 2025.10.27 | 感谢开源 | | *许 | 8.88 | 2025.10.23 | 老师 小白一枚，摸了几天了还没整起来，求教 | | Eason | 1 | 2025.10.22 | 还没整明白，但你在做好事 | | P*n | 1 | 2025.10.20 | | | *杰 | 1 | 2025.10.19 | | | *徐 | 1 | 2025.10.18 | | | *志 | 1 | 2025.10.17 | | | *😀 | 10 | 2025.10.16 | 点赞 | | **杰 | 10 | 2025.10.16 | | | *啸 | 10 | 2025.10.16 | | | *纪 | 5 | 2025.10.14 | TrendRadar | | J*d | 1 | 2025.10.14 | 谢谢你的工具，很好玩... | | *H | 1 | 2025.10.14 | | | 那*O | 10 | 2025.10.13 | | | *圆 | 1 | 2025.10.13 | | | P*g | 6 | 2025.10.13 | | | Ocean | 20 | 2025.10.12 | ...真的太棒了！！！小白级别也能直接用... | | **培 | 5.2 | 2025.10.2 | github-yzyf1312:开源万岁 | | *椿 | 3 | 2025.9.23 | 加油，很不错 | | *🍍 | 10 | 2025.9.21 | | | E*f | 1 | 2025.9.20 | | | *记 | 1 | 2025.9.20 | | | z*u | 2 | 2025.9.19 | | | **昊 | 5 | 2025.9.17 | | | *号 | 1 | 2025.9.15 | | | T*T | 2 | 2025.9.15 | 点赞 | | *家 | 10 | 2025.9.10 | | | *X | 1.11 | 2025.9.3 | | | *飙 | 20 | 2025.8.31 | 来自老童谢谢 | | *下 | 1 | 2025.8.30 | | | 2*D | 88 | 2025.8.13 下午 | | | 2*D | 1 | 2025.8.13 上午 | | | S*o | 1 | 2025.8.05 | 支持一下 | | *侠 | 10 | 2025.8.04 | | | x*x | 2 | 2025.8.03 | trendRadar 好项目 点赞 | | *远 | 1 | 2025.8.01 | | | *邪 | 5 | 2025.8.01 | | | *梦 | 0.1 | 2025.7.30 | | | **龙 | 10 | 2025.7.29 | 支持一下 |

## 🪄 赞助商
> **虚位以待**

### ❤️ 觉得好用？支持一下 > 若 TrendRadar 曾为你捕捉价值，不妨为它注入动力，助其持续进化 > > 金额随意，1 元也是对开源的鼓励。欢迎在赞赏时备注留言 (´▽`ʃ♡ƪ)
| 微信赞赏 | 支付宝赞赏 | |:---:|:---:| | 微信赞赏 | 支付宝赞赏 |
### 🤝 二次开发与引用 如果你在项目中使用或借鉴了本项目的思路、核心代码，**非常欢迎**在 README 或文档中注明来源并附上本仓库链接。 这将有助于项目的持续维护和社区发展，感谢你的尊重与支持！❤️ ### 💬 交流与反馈 - **GitHub Issues**：适合具体的技术问题。提问时请提供完整信息（截图、错误日志等），有助于快速定位。 - **公众号交流**：建议优先在相关文章下的留言区交流。若需后台提问，**先点赞/推荐**文章是最好的“敲门砖”，我在后台都能感受到这份心意哟 (´▽`ʃ♡ƪ)。 > **友情提示**： > 本项目为开源分享，非商业产品。把作者当朋友而非客服，沟通效率会更高哦！
|公众号关注 | |:---:| | |

## 📝 更新日志 > **📌 查看最新更新**：**[原仓库更新日志](https://github.com/sansan0/TrendRadar?tab=readme-ov-file#-更新日志)** ： - **提示**：建议查看【历史更新】，明确具体的【功能内容】 ### 2026/03/12 - v6.5.0 - **AI 智能筛选系统**：不用再手动设关键词！在 `ai_interests.txt` 里用日常语言写下你关注的方向（如"我想看 AI 和新能源相关新闻"），AI 会自动提取标签并对每条新闻打分，只推送真正和你相关的内容。万一 AI 筛选出了问题，会自动切回关键词匹配，推送不中断 - **每个时段支持不同的筛选方式和关注方向**：Timeline 中的每个时间段现在可以独立设置用什么方式筛选、看什么类型的新闻。比如：早上用"科技关键词"快速过滤，晚上换成"金融 AI 兴趣描述"做深度筛选——同一个系统，不同时段看不同内容 - **AI 分析范围独立于推送**：AI 分析的数据范围可以和推送内容不同。比如推送只发新增消息（避免重复打扰），但 AI 分析当天全部新闻（看完整趋势）。每个时段也能单独设置 AI 分析模式 - **AI 筛选智能省钱**：已分析过的新闻不会重复消耗 token；兴趣描述修改后，AI 自动判断变化幅度——小改动只更新受影响的标签，大改动才全量重新分类 - **多文件配置与标签隔离**：自定义关键词文件放 `config/custom/keyword/`，AI 兴趣文件放 `config/custom/ai/`，不同文件产生的标签各自独立、互不干扰 - **AI 翻译精准控制**：可分别控制热榜、RSS、独立展示区是否翻译，没开启显示的区域自动跳过，不浪费 token - **远程存储批量上传**：多次写操作攒在一起统一提交云端，减少 API 调用次数 - **每组关键词/标签展示数量限制**：通过 `max_news_per_keyword` 控制每个分组最多显示多少条新闻，避免单个热门话题占满整条推送 - **时段冲突智能检测**：两个时间段如果有时间重叠，系统会自动报错提醒修改，避免配置冲突导致意外行为 - 修复若干bug ### 2026/02/09 - mcp-v4.0.0 - **🔥 AI 消息直推所有渠道**：让 AI 写好的内容一键推送到飞书、钉钉、Telegram、邮件等 9 个渠道，Markdown 自动适配各平台格式，不用操心格式差异 - **新增格式化策略指南**：新增 `get_channel_format_guide` 工具，告诉 AI 每个渠道支持什么格式、有什么限制，生成的内容排版更好看 - **智能分批发送**：超长消息自动按各渠道字节限制拆分（飞书 30KB、钉钉 20KB 等），配置读取自 config.yaml - **修复渠道误检测**：ntfy 不再因为默认地址被误报为"已配置" - **代码复用优化**：批次处理函数直接复用 trendradar 核心模块，不重复造轮子
👉 点击展开：历史更新 ### 2026/02/09 - v6.0.0 > **Breaking Change**：配置文件升级（config.yaml 2.0.0），旧版 `push_window` 和 `analysis_window` 配置不再兼容，请参考新版 config.yaml 迁移 - **统一调度系统**：新增 `timeline.yaml`，用一套配置控制「什么时间采集 / 推送 / AI 分析」 - **5 种预设模板**：`always_on`（全天候，默认）、`morning_evening`（早晚汇总）、`office_hours`（办公时间）、`night_owl`（夜猫子）、`custom`（自定义）；也支持在 `presets:` 下新增自己的模板，只要 key 不重复，然后在 config.yaml 里填你的模板名即可 - **灵活的时间段配置**：支持工作日/周末差异化、跨午夜时间段、per-period once 去重 - **可视化配置编辑器**： - 新增 `timeline.yaml` 编辑标签页，与 config.yaml / frequency_words.txt 并列 - 预设模式卡片选择：点击即切换，自动同步 config.yaml 的 `schedule.preset` - 周视图时间线：7 天 × 24 小时水平条，用颜色区分推送/分析/采集状态 - 可交互控件：开关、下拉框、时间选择器，右侧修改实时同步到左侧 YAML - 周映射下拉选择：根据日计划动态填充，拖拉点击即可完成调度配置 - **AI 提示词稳定性优化**（ai_analysis_prompt.txt v2.0.0）： - 格式规范独立说明：将换行/标签/序号/禁止事项从 JSON value 中抽出，作为独立章节 - JSON 模板简化：字段描述缩短为一句话 + 字数限制，减少 AI 输出格式混乱 - 去除 system prompt 中的 Markdown 格式，与"禁止 Markdown"指令保持一致 - 所有 JSON 字段声明为可选，缺少任何字段不会报错，增强容错性 - **新增独立展示区 AI 概括分析**（`ai_analysis.include_standalone`）： - 新增独立开关，开启后 AI 对每个 standalone 源生成核心概括 - AI 分析与推送展示解耦：无需开启独立展示区的推送显示，AI 也可独立分析完整热榜数据 - 支持热榜平台和 RSS 源，含排名/时间/轨迹数据 - 轨迹分析与 `include_rank_timeline` 联动：开启时利用轨迹数据做深度趋势分析，关闭时基于排名做简要判断 - 新增 `standalone_summaries` JSON 字段（独立源点速览），所有推送渠道均已适配渲染 ### 2026/01/28 - v5.5.0 > 和 mcp 功能一样, 这个小工具我也不新开一个仓库维护了, 反正纯前端, 都搁一起吧 - 增加 trendradar 的可视化配置编辑器 ### 2026/02/02 - mcp-v3.2.0 - **新增 read_article 工具**：通过 Jina AI Reader 读取单篇文章正文（Markdown 格式） - **新增 read_articles_batch 工具**：批量读取多篇文章（最多 5 篇，自动限速） - **推荐工作流**：`search_news(query="关键词", include_url=True)` → `read_article(url=...)` 读取正文 - **文档更新**：README-MCP-FAQ.md 和 README-MCP-FAQ-EN.md 新增 Q19-Q20 文章读取相关说明 ### 2026/01/10 - mcp-v3.0.0~v3.1.5 - **Breaking Change**：所有工具返回值统一为 `{success, summary, data, error}` 结构 - **异步一致性**：所有 21 个工具函数使用 `asyncio.to_thread()` 包装同步调用 - **MCP Resources**：新增 4 个资源（platforms、rss-feeds、available-dates、keywords） - **RSS 增强**：`get_latest_rss` 支持多日查询（days 参数），跨日期 URL 去重 - **正则匹配修复**：`get_trending_topics` 支持 `/pattern/` 正则语法和 `display_name` - **缓存优化**：新增 `make_cache_key()` 函数，参数排序+MD5 哈希确保一致性 - **新增 check_version 工具**：支持同时检查 TrendRadar 和 MCP Server 版本更新 ### 2026/01/23 - v5.4.0 - 增加 AI 分析模式的独立控制功能，可选 follow_report | daily | current | incremental - 新增 AI 分析时间窗口控制，支持自定义运行段及每日频次限制 - 增加配置文件版本管理功能 - 修复若干bug ### 2026/01/19 - v5.3.0 > **重大重构：AI 模块迁移至 LiteLLM** - **统一 AI 接口**：使用 LiteLLM 替代手动实现，支持 100+ AI 提供商 - **简化配置**：移除 `provider` 字段，改用 `model: "provider/model_name"` 格式 - **新增功能**：自动重试 (`num_retries`)、备用模型 (`fallback_models`) - **配置变更**： - `ai.provider` → 移除（已合并到 model） - `ai.base_url` → `ai.api_base` - `AI_PROVIDER` 环境变量 → 移除 - `AI_BASE_URL` 环境变量 → `AI_API_BASE` - **模型格式示例**： - DeepSeek: `deepseek/deepseek-chat` - OpenAI: `openai/gpt-4o` - Gemini: `gemini/gemini-2.5-flash` - Anthropic: `anthropic/claude-3-5-sonnet` ### 2026/01/17 - v5.2.0 > 主要见 config.yaml 描述 **🌐 AI 翻译功能** - **多语言翻译**：支持将推送内容翻译为任意语言 - **批量翻译**：智能批量处理，减少 API 调用次数 - **自定义提示词**：支持自定义翻译风格 **🔧 配置架构优化** - **AI 模型配置独立**：分析和翻译共享模型配置 - **区域开关统一**：统一管理推送区域显示 - **区域排序自定义**：支持自定义各区域的显示顺序 **✨ AI 分析增强** - **AI 分析嵌入 HTML**：分析结果直接嵌入 HTML 报告，邮件通知直接使用 - **富样式 AI 区块**：渐变蓝色背景卡片式布局，清晰分隔各分析维度 - **排名时间线支持**：AI 可获取每条新闻在每个抓取时间点的精确排名 - **板块重组 (7→4)**：整合为核心热点态势、舆论风向争议、异动与弱信号、研判策略建议 **🔧 多模型适配** - **通用参数透传**：支持向 API 透传任意高级参数 - **Gemini 适配**：原生参数支持，内置安全策略放宽 **🐛 Bug 修复** - 修复若干已知问题，提升系统稳定性 ### 2026/01/10 - v5.0.0 > **开发小插曲**： > 致敬那个陪伴我两年多、却在刚续费后反手弹出 `"This organization has been disabled"` 的某 C 厂模型 **✨ 推送内容"五大板块"重构** 本次更新对推送消息进行了区域化重构，现在推送内容清晰地划分为五大核心板块： 1. **📊 热榜新闻**：根据你的关键词精准筛选后的全网热点聚合。 2. **📰 RSS 订阅**：你的个性化订阅源内容，支持按关键词分组。 3. **🆕 本次新增**：实时捕捉自上次运行以来的全新热点（带 🆕 标记）。 4. **📋 独立展示区**：指定平台的完整热榜或 RSS 源展示，**完全不受关键词过滤限制**。 5. **✨ AI 分析板块**：由 AI 驱动的深度洞察，包含趋势概述、热度走势及**极其重要**的情感倾向分析。 **✨ AI 智能分析推送功能** - **AI 分析集成**：使用 AI 大模型对推送内容进行深度分析，自动生成热点趋势概述、关键词热度分析、跨平台关联、潜在影响评估等 - **情感倾向分析**：新增深度情感识别，精准捕捉舆论的正负面、争议或担忧情绪 - **多 AI 提供商支持**：支持 DeepSeek（默认，性价比高）、OpenAI、Google Gemini 及任意 OpenAI 兼容接口 - **两种推送模式**：`only_analysis`（仅 AI 分析）、`both`（两者都推送） - **自定义提示词**：通过 `config/ai_analysis_prompt.txt` 文件自定义 AI 分析角色和输出格式 - **多维度数据分析**：AI 可分析排名变化、热度持续时间、跨平台表现、趋势预测等 **📋 独立展示区功能** - **完整热榜展示**：指定平台的完整热榜单独展示，不受关键词过滤影响 - **RSS 独立展示**：RSS 源内容可完整展示，适合内容较少的订阅源 - **灵活配置**：支持配置展示平台列表、RSS 源列表、最大展示条数 **📊 推送体验重构** - **排版升级**：重新设计并统一各渠道统计头部，强化区块组织，消息层次一目了然 - **配置简化**：优化飞书等通知渠道的配置逻辑，上手更简单 - **热度趋势箭头**：新增 🔺(上升)、🔻(下降)、➖(持平) 趋势标识，直观展示热度变化 - **通用 Webhook**：支持自定义 Webhook URL 和 JSON 模板，轻松适配 Discord、Matrix、IFTTT 等任意平台 **🔧 配置优化** - **频率词配置增强**：新增 `[组别名]` 语法，支持 `#` 注释行，配置更清晰（感谢 [@songge8](https://github.com/sansan0/TrendRadar/issues/752) 提出的建议） - **环境变量支持**：AI 分析相关配置支持环境变量覆盖（`AI_API_KEY`、`AI_PROVIDER` 等） > 💡 详细配置教程见 [让 AI 帮我分析热点](#12-让-ai-帮我分析热点) ### 2026/01/02 - v4.7.0 - **修复 RSS HTML 显示**：修复 RSS 数据格式不匹配导致的渲染问题，现在按关键词分组正确显示 - **新增正则表达式语法**：关键词配置支持 `/pattern/` 正则语法，解决英文子字符串误匹配问题（如 `ai` 匹配 `training`）[📖 查看语法详解](#关键词基础语法) - **新增显示名称语法**：使用 `=> 备注` 给复杂的正则表达式起个好记的名字，推送消息显示更清晰（如 `/\bai\b/ => AI相关`） - **不会写正则？** README 新增 AI 生成正则的引导，告诉 ChatGPT/Gemini/DeepSeek 你想匹配什么，让 AI 帮你写 ### 2025/12/30 - mcp-v2.0.0 - **架构调整**：移除 TXT 支持，统一使用 SQLite 数据库 - **RSS 查询**：新增 `get_latest_rss`、`search_rss`、`get_rss_feeds_status` - **统一搜索**：`search_news` 支持 `include_rss` 参数同时搜索热榜和 RSS ### 2026/01/01 - v4.6.0 - **修复 RSS HTML 显示**：将 RSS 内容合并到热榜 HTML 页面，按源分组显示 - **新增 display_mode 配置**：支持 `keyword`（按关键词分组）和 `platform`（按平台分组）两种显示模式 ### 2025/12/30 - v4.5.0 - **RSS 订阅源支持**：新增 RSS/Atom 抓取，按关键词分组统计（与热榜格式一致） - **存储结构重构**：扁平化目录结构 `output/{type}/{date}.db` - **统一排序配置**：`sort_by_position_first` 同时影响热榜和 RSS - **配置结构重构**：`config.yaml` 重新组织为 7 个逻辑分组（app、report、notification、storage、platforms、rss、advanced），配置路径更清晰 ### 2025/12/26 - mcp-v1.2.0 **MCP 模块更新 - 优化工具集，新增聚合对比功能，合并冗余工具:** - 新增 `aggregate_news` 工具 - 跨平台新闻去重聚合 - 新增 `compare_periods` 工具 - 时期对比分析（周环比/月环比） - 合并 `find_similar_news` + `search_related_news_history` → `find_related_news` - 增强 `get_trending_topics` - 新增 `auto_extract` 模式自动提取热点 - 修复若干bug - 同步更新 README-MCP-FAQ.md 文档的中英文版 (Q1-Q18) ### 2025/12/20 - v4.0.3 - 新增 URL 标准化功能，解决微博等平台因动态参数（如 `band_rank`）导致的重复推送问题 - 修复增量模式检测逻辑，正确识别历史标题 ### 2025/12/17 - v4.0.1 - StorageManager 添加推送记录代理方法 - S3 客户端切换至 virtual-hosted style 以提升兼容性（支持腾讯云 COS 等更多服务） ### 2025/12/13 - mcp-v1.1.0 **MCP 模块更新:** - 适配 v4.0.0，同时也兼容 v3.x 的数据 - 新增存储同步工具：`sync_from_remote`、`get_storage_status`、`list_available_dates` ### 2025/12/13 - v4.0.0 **🎉 重大更新：全面重构存储和核心架构** - **多存储后端支持**：引入全新的存储模块，支持本地 SQLite 和远程云存储（S3 兼容协议，例如 Cloudflare R2），适应 GitHub Actions、Docker 和本地环境。 - **数据库结构优化**：重构 SQLite 数据库表结构，提升数据效率和查询能力。 - **核心代码模块化**：将主程序逻辑拆分为 trendradar 包的多个模块，显著提升代码可维护性。 - **增强功能**：实现日期格式标准化、数据保留策略、时区配置支持、时间显示优化，并修复远程存储数据持久化问题，确保数据合并的准确性。 - **清理和兼容**：移除了大部分历史兼容代码，统一了数据存储和读取方式。 ### 2025/12/03 - v3.5.0 **🎉 核心功能增强** 1. **多账号推送支持** - 所有推送渠道（飞书、钉钉、企业微信、Telegram、ntfy、Bark、Slack）支持多账号配置 - 使用分号 `;` 分隔多个账号，例如：`FEISHU_WEBHOOK_URL=url1;url2` - 自动验证配对配置（如 Telegram 的 token 和 chat_id）数量一致性 2. **推送区域配置** - 通过 `display.region_order` 自定义各区域的显示顺序（v5.2.0 替代原 `reverse_content_order`） - 通过 `display.regions` 控制各区域是否显示（热榜、新增热点、RSS、独立展示区、AI 分析） 3. **全局过滤关键词** - 新增 `[GLOBAL_FILTER]` 区域标记，支持全局过滤不想看到的内容 - 适用场景：过滤广告、营销、低质内容等 **🐳 Docker 双路径 HTML 生成优化** - **问题修复**：解决 Docker 环境下 `index.html` 无法同步到宿主机的问题 - **双路径生成**：当日汇总 HTML 同时生成到两个位置 - `index.html`（项目根目录）：供 GitHub Pages 访问 - `output/index.html`：通过 Docker Volume 挂载，宿主机可直接访问 - **兼容性**：确保 Docker、GitHub Actions、本地运行环境均能正常访问网页版报告 **🐳 Docker MCP 镜像支持** - 新增独立的 MCP 服务镜像 `wantcat/trendradar-mcp` - 支持 Docker 部署 AI 分析功能，通过 HTTP 接口（端口 3333）提供服务 - 双容器架构：新闻推送服务与 MCP 服务独立运行，可分别扩展和重启 - 详见 [Docker 部署 - MCP 服务](#6-docker-部署) **🌐 Web 服务器支持** - 新增内置 Web 服务器，支持通过浏览器访问生成的报告 - 通过 `manage.py` 命令控制启动/停止：`docker exec -it trendradar python manage.py start_webserver` - 访问地址：`http://localhost:8080`（端口可配置） - 安全特性：静态文件服务、目录限制、本地访问 - 支持自动启动和手动控制两种模式 **📖 文档优化** - 新增 [推送内容怎么显示？](#7-推送内容怎么显示) 章节：自定义推送样式和内容 - 新增 [什么时候给我推送？](#8-什么时候给我推送) 章节：设置推送时间段 - 新增 [多久运行一次？](#9-多久运行一次) 章节：设置自动运行频率 - 新增 [推送到多个群/设备](#10-推送到多个群设备) 章节：同时推送给多个接收者 - 优化各配置章节：统一添加"配置位置"说明 - 简化快速开始配置说明：三个核心文件一目了然 - 优化 [Docker 部署](#6-docker-部署) 章节：新增镜像说明、推荐 git clone 部署、重组部署方式 **🔧 升级说明**： - **GitHub Fork 用户**：更新 `main.py`、`config/config.yaml`（新增多账号推送支持，无需修改现有配置） - **多账号推送**：新功能，默认不启用，现有单账号配置不受影响 ### 2025/11/26 - mcp-v1.0.3 **MCP 模块更新:** - 新增日期解析工具 resolve_date_range,解决 AI 模型计算日期不一致的问题 - 支持自然语言日期表达式解析(本周、最近7天、上月等) - 工具总数从 13 个增加到 14 个 ### 2025/11/28 - v3.4.1 **🔧 格式优化** 1. **Bark 推送增强** - Bark 现支持 Markdown 渲染 - 启用原生 Markdown 格式：粗体、链接、列表、代码块等 - 移除纯文本转换，充分利用 Bark 原生渲染能力 2. **Slack 格式精准化** - 使用专用 mrkdwn 格式处理分批内容 - 提升字节大小估算准确性（避免消息超限） - 优化链接格式：`` 和加粗语法：`*text*` 3. **性能提升** - 格式转换在分批过程中完成，避免二次处理 - 准确估算消息大小，减少发送失败率 **🔧 升级说明**： - **GitHub Fork 用户**：更新 `main.py`，`config.yaml` ### 2025/11/25 - v3.4.0 **🎉 新增 Slack 推送支持** 1. **团队协作推送渠道** - 支持 Slack Incoming Webhooks（全球流行的团队协作工具） - 消息集中管理，适合团队共享热点资讯 - 支持 mrkdwn 格式（粗体、链接等） 2. **多种部署方式** - GitHub Actions：配置 `SLACK_WEBHOOK_URL` Secret - Docker：环境变量 `SLACK_WEBHOOK_URL` - 本地运行：`config/config.yaml` 配置文件 > 📖 **详细配置教程**：[快速开始 - Slack 推送](#-快速开始) - 优化 setup-windows.bat 和 setup-windows-en.bat 一键安装 MCP 的体验 **🔧 升级说明**： - **GitHub Fork 用户**：更新 `main.py`、`config/config.yaml`、`.github/workflows/crawler.yml` ### 2025/11/24 - v3.3.0 **🎉 新增 Bark 推送支持** 1. **iOS 专属推送渠道** - 支持 Bark 推送（基于 APNs，iOS 平台） - 免费开源，简洁高效，无广告干扰 - 支持官方服务器和自建服务器两种方式 2. **多种部署方式** - GitHub Actions：配置 `BARK_URL` Secret - Docker：环境变量 `BARK_URL` - 本地运行：`config/config.yaml` 配置文件 > 📖 **详细配置教程**：[快速开始 - Bark 推送](#-快速开始) **🐛 Bug 修复** - 修复 `config.yaml` 中 `ntfy_server_url` 配置不生效的问题 ([#345](https://github.com/sansan0/TrendRadar/issues/345)) **🔧 升级说明**： - **GitHub Fork 用户**：更新 `main.py`、`config/config.yaml`、`.github/workflows/crawler.yml` ### 2025/11/23 - v3.2.0 **🎯 新增高级定制功能** 1. **关键词排序优先级配置** - 支持两种排序策略：热度优先 vs 配置顺序优先 - 满足不同使用场景：热点追踪 or 个性化关注 2. **显示数量精准控制** - 全局配置：统一限制所有关键词显示数量 - 单独配置：使用 `@数字` 语法为特定关键词设置限制 - 有效控制推送长度，突出重点内容 > 📖 **详细配置教程**：[关键词配置 - 高级配置](#关键词高级配置) **🔧 升级说明**： - **GitHub Fork 用户**：更新 `main.py`、`config/config.yaml` ### 2025/11/18 - mcp-v1.0.2 **MCP 模块更新:** - 优化查询今日新闻却可能错误返回过去日期的情况 ### 2025/11/22 - v3.1.1 - **修复数据异常导致的崩溃问题**：解决部分用户在 GitHub Actions 环境中遇到的 `'float' object has no attribute 'lower'` 错误 - 新增双重防护机制：在数据获取阶段过滤无效标题（None、float、空字符串），同时在函数调用处添加类型检查 - 提升系统稳定性，确保在数据源返回异常格式时仍能正常运行 **升级说明**（GitHub Fork 用户）： - 必须更新：`main.py` - 建议使用小版本升级方式：复制替换上述文件 ### 2025/11/20 - v3.1.0 - **新增个人微信推送支持**：企业微信应用可推送到个人微信，无需安装企业微信 APP - 支持两种消息格式：`markdown`（企业微信群机器人）和 `text`（个人微信应用） - 新增 `WEWORK_MSG_TYPE` 环境变量配置，支持 GitHub Actions、Docker、docker compose 等多种部署方式 - `text` 模式自动清除 Markdown 语法，提供纯文本推送效果 - 详见快速开始中的「个人微信推送」配置说明 **升级说明**（GitHub Fork 用户）： - 必须更新：`main.py`、`config/config.yaml` - 可选更新：`.github/workflows/crawler.yml`（如使用 GitHub Actions 部署） - 建议使用小版本升级方式：复制替换上述文件 ### 2025/11/12 - v3.0.5 - 修复邮件发送 SSL/TLS 端口配置逻辑错误 - 优化邮箱服务商（QQ/163/126）默认使用 465 端口（SSL） - **新增 Docker 环境变量支持**：核心配置项（`enable_crawler`、`report_mode`、`push_window` 等）支持通过环境变量覆盖，解决 NAS 用户修改配置文件不生效的问题（详见 [🐳 Docker 部署](#-docker-部署) 章节） ### 2025/10/26 - mcp-v1.0.1 **MCP 模块更新:** - 修复日期查询参数传递错误 - 统一所有工具的时间参数格式 ### 2025/10/31 - v3.0.4 - 解决飞书因推送内容过长而产生的错误，实现了分批推送 ### 2025/10/23 - v3.0.3 - 扩大 ntfy 错误信息显示范围 ### 2025/10/21 - v3.0.2 - 修复 ntfy 推送编码问题 ### 2025/10/20 - v3.0.0 **重大更新 - AI 分析功能上线** ✨ - **核心功能**： - 新增基于 MCP (Model Context Protocol) 的 AI 分析服务器 - 支持17种智能分析工具：基础查询、智能检索、高级分析、RSS 查询、系统管理 - 自然语言交互：通过对话方式查询和分析新闻数据 - 多客户端支持：Claude Desktop、Cherry Studio、Cursor、Cline 等 - **分析能力**： - 话题趋势分析（热度追踪、生命周期、爆火检测、趋势预测） - 数据洞察（平台对比、活跃度统计、关键词共现） - 情感分析、相似新闻查找、智能摘要生成 - 历史相关新闻检索、多模式搜索 - **更新提示**： - 这是独立的 AI 分析功能，不影响现有的推送功能 - 可选择性使用，无需升级现有部署 ### 2025/10/15 - v2.4.4 - **更新内容**： - 修复 ntfy 推送编码问题 + 1 - 修复推送时间窗口判断问题 - **更新提示**： - 建议【小版本升级】 ### 2025/10/10 - v2.4.3 > 感谢 [nidaye996](https://github.com/sansan0/TrendRadar/issues/98) 发现的体验问题 - **更新内容**： - 重构"静默推送模式"命名为"推送时间窗口控制"，提升功能理解度 - 明确推送时间窗口作为可选附加功能，可与三种推送模式搭配使用 - 改进注释和文档描述，使功能定位更加清晰 - **更新提示**： - 这个仅仅是重构，可以不用升级 ### 2025/10/8 - v2.4.2 - **更新内容**： - 修复 ntfy 推送编码问题 - 修复配置文件缺失问题 - 优化 ntfy 推送效果 - 增加 github page 图片分段导出功能 - **更新提示**： - 建议使用【大版本更新】 ### 2025/10/2 - v2.4.0 **新增 ntfy 推送通知** - **核心功能**： - 支持 ntfy.sh 公共服务和自托管服务器 - **使用场景**： - 适合追求隐私的用户（支持自托管） - 跨平台推送（iOS、Android、Desktop、Web） - 无需注册账号（公共服务器） - 开源免费（MIT 协议） - **更新提示**： - 建议使用【大版本更新】 ### 2025/09/26 - v2.3.2 - 修正了邮件通知配置检查被遗漏的问题（[#88](https://github.com/sansan0/TrendRadar/issues/88)） **修复说明**： - 解决了即使正确配置邮件通知，系统仍提示"未配置任何webhook"的问题 ### 2025/09/22 - v2.3.1 - **新增邮件推送功能**，支持将热点新闻报告发送到邮箱 - **智能 SMTP 识别**：自动识别 Gmail、QQ邮箱、Outlook、网易邮箱等 10+ 种邮箱服务商配置 - **HTML 精美格式**：邮件内容采用与网页版相同的 HTML 格式，排版精美，移动端适配 - **批量发送支持**：支持多个收件人，用逗号分隔即可同时发送给多人 - **自定义 SMTP**：可自定义 SMTP 服务器和端口 - 修复Docker构建网络连接问题 **使用说明**： - 适用场景：适合需要邮件归档、团队分享、定时报告的用户 - 支持邮箱：Gmail、QQ邮箱、Outlook/Hotmail、163/126邮箱、新浪邮箱、搜狐邮箱等 **更新提示**： - 此次更新的内容比较多，如果想升级，建议采用【大版本升级】 ### 2025/09/17 - v2.2.0 - 新增一键保存新闻图片功能，让你轻松分享关注的热点 **使用说明**： - 适用场景：当你按照教程开启了网页版功能后(GitHub Pages) - 使用方法：用手机或电脑打开该网页链接，点击页面顶部的"保存为图片"按钮 - 实际效果：系统会自动将当前的新闻报告制作成一张精美图片，保存到你的手机相册或电脑桌面 - 分享便利：你可以直接把这张图片发给朋友、发到朋友圈，或分享到工作群，让别人也能看到你发现的重要资讯 ### 2025/09/13 - v2.1.2 - 解决钉钉的推送容量限制导致的新闻推送失败问题(采用分批推送) ### 2025/09/04 - v2.1.1 - 修复docker在某些架构中无法正常运行的问题 - 正式发布官方 Docker 镜像 wantcat/trendradar，支持多架构 - 优化 Docker 部署流程，无需本地构建即可快速使用 ### 2025/08/30 - v2.1.0 **核心改进**： - **推送逻辑优化**：从"每次执行都推送"改为"时间窗口内可控推送" - **时间窗口控制**：可设定推送时间范围，避免非工作时间打扰 - **推送频率可选**：时间段内支持单次推送或多次推送 **更新提示**： - 本功能默认关闭，需手动在 config.yaml 中开启推送时间窗口控制 - 升级需同时更新 main.py 和 config.yaml 两个文件 ### 2025/08/27 - v2.0.4 - 本次版本不是功能修复，而是重要提醒 - 请务必妥善保管好 webhooks，不要公开，不要公开，不要公开 - 如果你以 fork 的方式将本项目部署在 GitHub 上，请将 webhooks 填入 GitHub Secret，而非 config.yaml - 如果你已经暴露了 webhooks 或将其填入了 config.yaml，建议删除后重新生成 ### 2025/08/06 - v2.0.3 - 优化 github page 的网页版效果，方便移动端使用 ### 2025/07/28 - v2.0.2 - 重构代码 - 解决版本号容易被遗漏修改的问题 ### 2025/07/27 - v2.0.1 **修复问题**: 1. docker 的 shell 脚本的换行符为 CRLF 导致的执行异常问题 2. frequency_words.txt 为空时，导致新闻发送也为空的逻辑问题 - 修复后，当你选择 frequency_words.txt 为空时，将**推送所有新闻**，但受限于消息推送大小限制，请做如下调整 - 方案一：关闭手机推送，只选择 Github Pages 布置(这是能获得最完整信息的方案，将把所有平台的热点按照你**自定义的热搜算法**进行重新排序) - 方案二：减少推送平台，优先选择**企业微信**或**Telegram**，这两个推送我做了分批推送功能(因为分批推送影响推送体验，且只有这两个平台只给一点点推送容量，所以才不得已做了分批推送功能，但至少能保证获得的信息完整) - 方案三：可与方案二结合，模式选择 current 或 incremental 可有效减少一次性推送的内容 ### 2025/07/17 - v2.0.0 **重大重构**： - 配置管理重构：所有配置现在通过 `config/config.yaml` 文件管理（main.py 我依旧没拆分，方便你们复制升级） - 运行模式升级：支持三种模式 - `daily`（当日汇总）、`current`（当前榜单）、`incremental`（增量监控） - Docker 支持：完整的 Docker 部署方案，支持容器化运行 **配置文件说明**： - `config/config.yaml` - 主配置文件（应用设置、爬虫配置、通知配置、平台配置等） - `config/frequency_words.txt` - 关键词配置（监控词汇设置） ### 2025/07/09 - v1.4.1 **功能新增**：增加增量推送(在 main.py 头部配置 FOCUS_NEW_ONLY)，该开关只关心新话题而非持续热度，只在有新内容时才发通知。 **修复问题**: 某些情况下，由于新闻本身含有特殊符号导致的偶发性排版异常。 ### 2025/06/23 - v1.3.0 企业微信 和 Telegram 的推送消息有长度限制，对此我采用将消息拆分推送的方式。开发文档详见[企业微信](https://developer.work.weixin.qq.com/document/path/91770) 和 [Telegram](https://core.telegram.org/bots/api) ### 2025/06/21 - v1.2.1 在本版本之前的旧版本，不仅 main.py 需要复制替换， crawler.yml 也需要你复制替换 https://github.com/sansan0/TrendRadar/blob/master/.github/workflows/crawler.yml ### 2025/06/19 - v1.2.0 > 感谢 claude research 整理的各平台 api ,让我快速完成各平台适配（虽然代码更多冗余了~ 1. 支持 telegram ，企业微信，钉钉推送渠道, 支持多渠道配置和同时推送 ### 2025/06/18 - v1.1.0 > **200 star⭐** 了, 继续给大伙儿助兴~近期，在我的"怂恿"下，挺多人在我公众号点赞分享推荐助力了我，我都在后台看见了具体账号的鼓励数据，很多都成了天使轮老粉（我玩公众号才一个多月，虽然注册是七八年前的事了哈哈，属于上车早，发车晚），但因为你们没有留言或私信我，所以我也无法一一回应并感谢支持，在此一并谢谢！ 1. 重要的更新，加了权重，你现在看到的新闻都是最热点最有关注度的出现在最上面 2. 更新文档使用，因为近期更新了很多功能，而且之前的使用文档我偷懒写的简单（见下面的 ⚙️ frequency_words.txt 配置完整教程） ### 2025/06/16 - v1.0.0 1. 增加了一个项目新版本更新提示，默认打开，如要关掉，可以在 main.py 中把 "FEISHU_SHOW_VERSION_UPDATE": True 中的 True 改成 False 即可 ### 2025/06/13+14 1. 去掉了兼容代码，之前 fork 的同学，直接复制代码会在当天显示异常（第二天会恢复正常） 2. feishu 和 html 底部增加一个新增新闻显示 ### 2025/06/09 **100 star⭐** 了，写个小功能给大伙儿助助兴 frequency_words.txt 文件增加了一个【必须词】功能，使用 + 号 1. 必须词语法如下： 唐僧或者猪八戒必须在标题里同时出现，才会收录到推送新闻中 ``` +唐僧 +猪八戒 ``` 2. 过滤词的优先级更高： 如果标题中过滤词匹配到唐僧念经，那么即使必须词里有唐僧，也不显示 ``` +唐僧 !唐僧念经 ``` ### 2025/06/02 1. **网页**和**飞书消息**支持手机直接跳转详情新闻 2. 优化显示效果 + 1 ### 2025/05/26 1. 飞书消息显示效果优化
优化前
飞书消息界面 - 优化前 		优化后
飞书消息界面 - 优化后

## ✨ 核心功能 ### **全网热点聚合** - 知乎 - 抖音 - bilibili 热搜 - 华尔街见闻 - 贴吧 - 百度热搜 - 财联社热门 - 澎湃新闻 - 凤凰网 - 今日头条 - 微博 默认监控 11 个主流平台，也可自行增加额外的平台 > 💡 详细配置教程见 [配置详解 - 平台配置](#1-平台配置) ### **RSS 订阅源支持**（v4.5.0 新增） 支持 RSS/Atom 订阅源抓取，按关键词分组统计（与热榜格式一致）： - **统一格式**：RSS 与热榜使用相同的关键词匹配和显示格式 - **简单配置**：直接在 `config.yaml` 中添加 RSS 源 - **合并推送**：热榜和 RSS 合并为一条消息推送 - **新鲜度过滤**：自动过滤超过指定天数的旧文章，避免重复推送。支持全局默认天数和单源独立设置 > 💡 RSS 使用与热榜相同的 `frequency_words.txt` 进行关键词过滤 ### **可视化配置编辑器** 提供基于 Web 的图形化配置界面，无需手动编辑 YAML 文件，通过表单即可完成所有配置项的修改与导出。 👉 **在线体验**：[https://sansan0.github.io/TrendRadar/](https://sansan0.github.io/TrendRadar/) 可视化配置编辑器 ### **智能推送策略** **三种推送模式**： | 模式 | 适用场景 | 推送特点 | |------|---------|---------| | **当日汇总** (daily) | 企业管理者/普通用户 | 按时推送当日所有匹配新闻（会包含之前推送过的） | | **当前榜单** (current) | 自媒体人/内容创作者 | 按时推送当前榜单匹配新闻（持续在榜的每次都出现） | | **增量监控** (incremental) | 投资者/交易员 | 仅推送新增内容，零重复 | > 💡 **快速选择指南：** > - 不想看到重复新闻 → 用 `incremental`（增量监控） > - 想看完整榜单趋势 → 用 `current`（当前榜单） > - 需要每日汇总报告 → 用 `daily`（当日汇总） > > 详细对比和配置教程见 [配置详解 - 推送模式详解](#3-推送模式详解) **附加功能**（可选）： | 功能 | 说明 | 默认 | |------|------|------| | **调度系统** | 按周一到周日逐日编排：为每天分配不同时间段、推送模式和 AI 分析策略。**每个时段可独立设置筛选方式（关键词/AI）和关注方向**，实现不同时间看不同类型新闻。内置 5 种预设（always_on / morning_evening / office_hours / night_owl / custom），也可自定义。支持工作日/周末差异化、跨午夜时段、per-period 去重、时段冲突检测（v6.0.0 + v6.5.0） | morning_evening | | **内容顺序配置** | 通过 `display.region_order` 调整各区域（热榜、新增热点、RSS、独立展示区、AI 分析）的显示顺序；通过 `display.regions` 控制各区域是否显示（v5.2.0） | 见配置文件 | | **显示模式切换** | `keyword`=按关键词分组，`platform`=按平台分组（v4.6.0 新增） | keyword | > 💡 详细配置教程见 [推送内容怎么显示？](#7-推送内容怎么显示) 和 [什么时候给我推送？](#8-什么时候给我推送) ### **精准内容筛选** 设置个人关键词（如：AI、比亚迪、教育政策），只推送相关热点，过滤无关信息 > 💡 **基础配置教程**：[关键词配置 - 基础语法](#关键词基础语法) > > 💡 **高级配置教程**：[关键词配置 - 高级配置](#关键词高级配置) > > 💡 也可以不做筛选，完整推送所有热点（将 frequency_words.txt 留空） ### **AI 智能筛选新闻**（v6.5.0 新增） 用自然语言描述你的兴趣，AI 自动分类新闻，替代传统关键词匹配 - **自然语言兴趣描述**：在 `ai_interests.txt` 中用日常语言写下关注方向，无需学习关键词语法 - **两阶段智能处理**：AI 先从兴趣描述提取结构化标签，再对新闻按标签批量分类打分 - **分数阈值控制**：通过 `ai_filter.min_score` 精确控制推送质量，只推送高相关度新闻 - **自动回退保障**：AI 筛选失败时自动回退到关键词匹配，确保推送不中断 - **智能标签更新**：兴趣变更时 AI 自动评估变化幅度，决定增量或全量重分类 - **灵活切换**：`filter.method` 支持 `keyword`（默认）和 `ai` 两种模式，Timeline 可按时段覆盖 - **分时段个性化**：不同时间段可以使用不同的关键词文件或 AI 兴趣描述。例如早上用"科技词库"快速过滤，晚上换成"金融兴趣"做 AI 深度筛选 ```yaml # config.yaml 快速启用示例 filter: method: ai # keyword（默认）| ai ai_filter: min_score: 6 # 推送最低分数阈值（1-10） ``` > 💡 AI 筛选与 AI 分析/翻译共享模型配置，只需配置一次 `ai.api_key` ### **热点趋势分析** 实时追踪新闻热度变化，让你不仅知道"什么在热搜"，更了解"热点如何演变" - **时间轴追踪**：记录每条新闻从首次出现到最后出现的完整时间跨度 - **热度变化**：统计新闻在不同时间段的排名变化和出现频次 - **新增检测**：实时识别新出现的热点话题，用🆕标记第一时间提醒 - **持续性分析**：区分一次性热点话题和持续发酵的深度新闻 - **跨平台对比**：同一新闻在不同平台的排名表现，看出媒体关注度差异 > 💡 推送格式说明见 [消息样式说明](#5-我收到的消息长什么样) ### **个性化热点算法** 不再被各个平台的算法牵着走，TrendRadar 会重新整理全网热搜 > 💡 三个比例可以调整，详见 [配置详解 - 热点权重调整](#4-热点权重调整) ### **多渠道多账号推送** 支持**企业微信**(+ 微信推送方案)、**飞书**、**钉钉**、**Telegram**、**邮件**、**ntfy**、**Bark**、**Slack**、**通用 Webhook**（可对接 Discord、IFTTT 等任意平台），消息直达手机和邮箱 > 💡 详细配置教程见 [推送到多个群/设备](#10-推送到多个群设备) ### **AI 多语言翻译**（v5.2.0 新增） 将推送内容翻译为任意语言，打破语言壁垒，无论是阅读国内热点还是通过 RSS 订阅海外资讯，都能以母语轻松获取 - **一键翻译**：在 `config.yaml` 中设置 `ai_translation.enabled: true` 和目标语言即可 - **多语言支持**：支持 English、Korean、Japanese、French 等任意语言 - **智能批量处理**：自动批量翻译，减少 API 调用次数，节省成本 - **自定义风格**：通过 `ai_translation_prompt.txt` 自定义翻译风格和术语 - **共享模型配置**：与 AI 分析功能共用 `ai` 配置段的模型设置 ```yaml # config.yaml 快速启用示例 ai_translation: enabled: true language: "English" # 翻译目标语言 ``` > 💡 翻译功能与 AI 分析功能共享模型配置，只需配置一次 `ai.api_key` 即可同时使用两个功能 **RSS 源参考**：以下是一些 RSS 订阅源合集，可按需选用 - [awesome-tech-rss](https://github.com/tuan3w/awesome-tech-rss) - 科技、创业、编程领域博客和媒体 - [awesome-rss-feeds](https://github.com/plenaryapp/awesome-rss-feeds) - 世界各国主流新闻媒体 RSS 合集 > ⚠️ 部分海外媒体内容可能涉及敏感话题，AI 模型可能拒绝翻译，建议根据实际需求筛选订阅源 ### **灵活存储架构**（v4.0.0 重大更新） **多存储后端支持**： - **远程云存储**：GitHub Actions 环境默认，支持 S3 兼容协议（R2/OSS/COS 等），数据存储在云端，不污染仓库 - **本地 SQLite 数据库**：Docker/本地环境默认，数据完全可控 - **自动后端选择**：根据运行环境智能切换存储方式 > 💡 详细说明见 [数据保存在哪里？](#11-数据保存在哪里) ### **多端部署** - **GitHub Actions**：定时自动爬取 + 远程云存储（需签到续期） - **Docker 部署**：支持多架构容器化运行，数据本地存储 - **本地运行**：Windows/Mac/Linux 直接运行 ### **AI 分析推送（v5.0.0 新增）** 使用 AI 大模型对推送内容进行深度分析，自动生成热点洞察报告 - **智能分析**：自动分析热点趋势、关键词热度、跨平台关联、潜在影响 - **多提供商**：基于 LiteLLM 统一接口，支持 100+ AI 提供商（DeepSeek、OpenAI、Gemini、Anthropic、本地 Ollama 等），还支持备用模型自动切换 - **分析模式独立**：AI 的分析范围可以和推送不同——推送只发新增消息（避免打扰），但 AI 可以分析当天全部新闻（看完整趋势） - **灵活推送**：可选仅原始内容、仅 AI 分析、或两者都推送 - **自定义提示词**：通过 `config/ai_analysis_prompt.txt` 自定义分析角度 > 💡 详细配置教程见 [让 AI 帮我分析热点](#12-让-ai-帮我分析热点) ### **独立展示区（v5.0.0 新增）** 为指定平台提供完整热榜展示，不受关键词过滤影响 - **完整热榜**：指定平台的热榜完整展示，适合想看完整排名的用户 - **RSS 独立展示**：RSS 源内容可完整展示，不受关键词限制 - **AI 深度分析**：可独立开启 AI 对完整热榜的趋势分析，无需在推送中展示 - **灵活配置**：支持配置展示平台、RSS 源、最大条数 > 💡 详细配置教程见 [推送内容怎么显示？ - 独立展示区](#7-推送内容怎么显示) ### **AI 智能分析（v3.0.0 新增）** 基于 MCP (Model Context Protocol) 协议的 AI 对话分析系统，让你用自然语言深度挖掘新闻数据 > **💡 使用提示**：AI 功能需要本地新闻数据支持 > - 项目自带测试数据，可立即体验功能 > - 建议自行部署运行项目，获取更实时的数据 > > 详见 [AI 智能分析](#-ai-智能分析) ### **网页部署** 运行后根目录生成 `index.html`，即为完整的新闻报告页面。 > **部署方式**：点击 **Use this template** 创建仓库，可部署到 Cloudflare Pages 或 GitHub Pages 等静态托管平台。 > > **💡 提示**：启用 GitHub Pages 可获得在线访问地址，进入仓库 Settings → Pages 即可开启。[效果预览](https://sansan0.github.io/TrendRadar/) > > ⚠️ 原 GitHub Actions 自动存储功能已下线（该方案曾导致 GitHub 服务器负载过高，影响平台稳定性）。 ### **减少 APP 依赖** 从"被算法推荐绑架"变成"主动获取自己想要的信息" **适合人群：** 投资者、自媒体人、企业公关、关心时事的普通用户 **典型场景：** 股市投资监控、品牌舆情追踪、行业动态关注、生活资讯获取 | 网页效果(邮箱推送效果) | 飞书推送效果 | AI 分析推送效果 | |:---:|:---:|:---:| | ![网页效果](_image/github-pages.png) | ![飞书推送效果](_image/feishu.jpg) | ![AI分析推送效果](_image/ai.jpg) |
## 🚀 快速开始 > **提醒**：建议先 **[查看最新官方文档](https://github.com/sansan0/TrendRadar?tab=readme-ov-file)**，确保配置步骤是最新的。 ### 请选择适合你的部署方式 #### 🅰️ 方案一：Docker 部署（推荐 🔥） * **特点**：比 GitHub Actions 更稳定，数据本地存储（无需配置云存储） * **适用**：有自己的服务器、NAS 或长期运行的电脑 * **注意**：你需要阅读了解下方的基础配置流程，然后跳转到 Docker 教程进行部署。 #### 🅱️ 方案二：GitHub Actions 部署（本章节内容 ⬇️） * **特点**：无服务器，数据存储在 **远程云存储**（推荐配置） * **适用**：没有服务器的用户，利用 GitHub 免费资源 * **注意**：需配置云存储以获得完整体验，且需定期签到续期 ### 1️⃣ 第一步：获取项目代码 点击本仓库页面右上角的绿色 **[Use this template]** 按钮 → 选择 "Create a new repository"。 > ⚠️ 提醒： > - 后续文档中提到的 "Fork" 均可理解为 "Use this template" > - 使用 Fork 可能导致运行异常，详见 [Issue #606](https://github.com/sansan0/TrendRadar/issues/606)
### 2️⃣ 第二步：设置 GitHub Secrets 在你 Fork 后的仓库中，进入 `Settings` > `Secrets and variables` > `Actions` > `New repository secret` **📌 重要说明（请务必仔细阅读）：** - **一个 Name 对应一个 Secret**：每添加一个配置项，点击一次"New repository secret"按钮，填写一对"Name"和"Secret" - **保存后看不到值是正常的**：出于安全考虑，保存后重新编辑时，只能看到 Name（名称），看不到 Secret（值）的内容 - **严禁自创名称**：Secret 的 Name（名称）必须**严格使用**下方列出的名称（如 `WEWORK_WEBHOOK_URL`、`FEISHU_WEBHOOK_URL` 等），不能自己随意修改或创造新名称，否则系统无法识别 - **可以同时配置多个平台**：系统会向所有配置的平台发送通知 **配置示例：** GitHub Secrets 配置示例 如上图所示，每一行是一个配置项： - **Name（名称）**：必须使用下方展开内容中列出的固定名称（如 `WEWORK_WEBHOOK_URL`） - **Secret（值）**：填写你从对应平台获取的实际内容（如 Webhook 地址、Token 等）
👉 点击展开：企业微信机器人（配置最简单最迅速）
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`WEWORK_WEBHOOK_URL`（请复制粘贴此名称，不要手打，避免打错） - **Secret（值）**：你的企业微信机器人 Webhook 地址
**机器人设置步骤：** #### 手机端设置： 1. 打开企业微信 App → 进入目标内部群聊 2. 点击右上角"…"按钮 → 选择"消息推送" 3. 点击"添加" → 名称输入"TrendRadar" 4. 复制 Webhook 地址，点击保存，复制的内容配置到上方的 GitHub Secret 中 #### PC 端设置流程类似
👉 点击展开：个人微信推送（基于企业微信应用，推送到个人微信）
> 由于该方案是基于企业微信的插件机制，推送样式为纯文本（无 markdown 格式），但可以直接推送到个人微信，无需安装企业微信 App。 **GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`WEWORK_WEBHOOK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的企业微信应用 Webhook 地址 - **Name（名称）**：`WEWORK_MSG_TYPE`（请复制粘贴此名称，不要手打） - **Secret（值）**：`text`
**设置步骤：** 1. 完成上方的企业微信机器人 Webhook 设置 2. 添加 `WEWORK_MSG_TYPE` Secret，值设为 `text` 3. 按照下面图片操作，关联个人微信 4. 配置好后，手机上的企业微信 App 可以删除 **说明**： - 与企业微信机器人使用相同的 Webhook 地址 - 区别在于消息格式：`text` 为纯文本，`markdown` 为富文本（默认） - 纯文本格式会自动去除所有 markdown 语法（粗体、链接等）
👉 点击展开：飞书机器人（消息显示相对友好）
若启用 **AI 分析**，飞书推送偶发（约 5% 概率）会有数分钟延迟（推测为平台对 AI 生成内容的合规性审核）。 **GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`FEISHU_WEBHOOK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的飞书机器人 Webhook 地址（该链接开头类似 https://www.feishu.cn/flow/api/trigger-webhook/********）
有两个方案，**方案一**配置简单，**方案二**配置复杂(但是稳定推送) 其中方案一，由 **ziventian**发现并提供建议，在这里感谢他，默认是个人推送，也可以配置群组推送操作[#97](https://github.com/sansan0/TrendRadar/issues/97) ， **方案一：** > 对部分人存在额外操作，否则会报"系统错误"。需要手机端搜索下机器人，然后开启飞书机器人应用(该建议来自于网友，可参考) 1. 电脑浏览器打开 https://botbuilder.feishu.cn/home/my-command 2. 点击"新建机器人指令" 3. 点击"选择触发器"，往下滑动，点击"Webhook 触发" 4. 此时你会看到"Webhook 地址"，把这个链接先复制到本地记事本暂存，继续接下来的操作 5. "参数"里面放上下面的内容，然后点击"完成" ```json { "message_type": "text", "content": { "text": "{{内容}}" } } ``` 6. 点击"选择操作" > "通过官方机器人发消息" 7. 消息标题填写"TrendRadar 热点监控" 8. 最关键的部分来了，点击 + 按钮，选择"Webhook 触发"，然后按照下面的图片摆放 ![飞书机器人配置示例](_image/feishu.png) 9. 配置完成后，将第 4 步复制的 Webhook 地址配置到 GitHub Secrets 中的 `FEISHU_WEBHOOK_URL`
**方案二：** 1. 电脑浏览器打开 https://botbuilder.feishu.cn/home/my-app 2. 点击"新建机器人应用" 3. 进入创建的应用后，点击"流程设计" > "创建流程" > "选择触发器" 4. 往下滑动，点击"Webhook 触发" 5. 此时你会看到"Webhook 地址"，把这个链接先复制到本地记事本暂存，继续接下来的操作 6. "参数"里面放上下面的内容，然后点击"完成" ```json { "message_type": "text", "content": { "text": "{{内容}}" } } ``` 7. 点击"选择操作" > "发送飞书消息"，勾选 "群消息"，然后点击下面的输入框，点击"我管理的群组"（如果没有群组，你可以在飞书 app 上创建群组） 8. 消息标题填写"TrendRadar 热点监控" 9. 最关键的部分来了，点击 + 按钮，选择"Webhook 触发"，然后按照下面的图片摆放 ![飞书机器人配置示例](_image/feishu.png) 10. 配置完成后，将第 5 步复制的 Webhook 地址配置到 GitHub Secrets 中的 `FEISHU_WEBHOOK_URL`
👉 点击展开：钉钉机器人
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`DINGTALK_WEBHOOK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的钉钉机器人 Webhook 地址
**机器人设置步骤：** 1. **创建机器人（仅 PC 端支持）**： - 打开钉钉 PC 客户端，进入目标群聊 - 点击群设置图标（⚙️）→ 往下翻找到"机器人"点开 - 选择"添加机器人" → "自定义" 2. **配置机器人**： - 设置机器人名称 - **安全设置**： - **自定义关键词**：设置 "热点" 3. **完成设置**： - 勾选服务条款协议 → 点击"完成" - 复制获得的 Webhook URL - 将 URL 配置到 GitHub Secrets 中的 `DINGTALK_WEBHOOK_URL` **注意**：移动端只能接收消息，无法创建新机器人。
👉 点击展开：Telegram Bot
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`TELEGRAM_BOT_TOKEN`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的 Telegram Bot Token - **Name（名称）**：`TELEGRAM_CHAT_ID`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的 Telegram Chat ID **说明**：Telegram 需要配置**两个** Secret，请分别点击两次"New repository secret"按钮添加
**机器人设置步骤：** 1. **创建机器人**： - 在 Telegram 中搜索 `@BotFather`（大小写注意，有蓝色徽章勾勾，有类似 37849827 monthly users，这个才是官方的，有一些仿官方的账号注意辨别） - 发送 `/newbot` 命令创建新机器人 - 设置机器人名称（必须以"bot"结尾，很容易遇到重复名字，所以你要绞尽脑汁想不同的名字） - 获取 Bot Token（格式如：`123456789:AAHfiqksKZ8WmR2zSjiQ7_v4TMAKdiHm9T0`） 2. **获取 Chat ID**： **方法一：通过官方 API 获取** - 先向你的机器人发送一条消息 - 访问：`https://api.telegram.org/bot<你的Bot Token>/getUpdates` - 在返回的 JSON 中找到 `"chat":{"id":数字}` 中的数字 **方法二：使用第三方工具** - 搜索 `@userinfobot` 并发送 `/start` - 获取你的用户 ID 作为 Chat ID 3. **配置到 GitHub**： - `TELEGRAM_BOT_TOKEN`：填入第 1 步获得的 Bot Token - `TELEGRAM_CHAT_ID`：填入第 2 步获得的 Chat ID
👉 点击展开：邮件推送（支持所有主流邮箱）
- 注意事项：为防止邮件群发功能被**滥用**，当前的群发是所有收件人都能看到彼此的邮箱地址。 - 如果你没有过配置下面这种邮箱发送的经历，不建议尝试 > ⚠️ **重要配置依赖**：邮件推送需要 HTML 报告文件。请确保 `config/config.yaml` 中的 `storage.formats.html` 设置为 `true`： > ```yaml > storage: > formats: > sqlite: true > txt: false > html: true # 必须启用，否则邮件推送会失败 > ``` > 如果设置为 `false`，邮件推送时会报错：`错误：HTML文件不存在或未提供: None`
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`EMAIL_FROM`（请复制粘贴此名称，不要手打） - **Secret（值）**：发件人邮箱地址 - **Name（名称）**：`EMAIL_PASSWORD`（请复制粘贴此名称，不要手打） - **Secret（值）**：邮箱密码或授权码 - **Name（名称）**：`EMAIL_TO`（请复制粘贴此名称，不要手打） - **Secret（值）**：收件人邮箱地址（多个收件人用英文逗号分隔，也可以和 EMAIL_FROM 一样，自己发送给自己） - **Name（名称）**：`EMAIL_SMTP_SERVER`（可选配置，请复制粘贴此名称） - **Secret（值）**：SMTP服务器地址（可留空，系统会自动识别） - **Name（名称）**：`EMAIL_SMTP_PORT`（可选配置，请复制粘贴此名称） - **Secret（值）**：SMTP端口（可留空，系统会自动识别） **说明**：邮件推送需要配置至少**3个必需** Secret（EMAIL_FROM、EMAIL_PASSWORD、EMAIL_TO），后两个为可选配置
**支持的邮箱服务商**（自动识别 SMTP 配置）： | 邮箱服务商 | 域名 | SMTP 服务器 | 端口 | 加密方式 | |-----------|------|------------|------|---------| | **Gmail** | gmail.com | smtp.gmail.com | 587 | TLS | | **QQ邮箱** | qq.com | smtp.qq.com | 465 | SSL | | **Outlook** | outlook.com | smtp-mail.outlook.com | 587 | TLS | | **Hotmail** | hotmail.com | smtp-mail.outlook.com | 587 | TLS | | **Live** | live.com | smtp-mail.outlook.com | 587 | TLS | | **163邮箱** | 163.com | smtp.163.com | 465 | SSL | | **126邮箱** | 126.com | smtp.126.com | 465 | SSL | | **新浪邮箱** | sina.com | smtp.sina.com | 465 | SSL | | **搜狐邮箱** | sohu.com | smtp.sohu.com | 465 | SSL | | **天翼邮箱** | 189.cn | smtp.189.cn | 465 | SSL | | **阿里云邮箱** | aliyun.com | smtp.aliyun.com | 465 | TLS | | **Yandex邮箱** | yandex.com | smtp.yandex.com | 465 | TLS | | **iCloud邮箱** | icloud.com | smtp.mail.me.com | 587 | SSL | > **自动识别**：使用以上邮箱时，无需手动配置 `EMAIL_SMTP_SERVER` 和 `EMAIL_SMTP_PORT`，系统会自动识别。 > > **反馈说明**： > - 如果你使用**其他邮箱**测试成功，欢迎开 [Issues](https://github.com/sansan0/TrendRadar/issues) 告知，我会添加到支持列表 > - 如果上述邮箱配置有误或无法使用，也请开 [Issues](https://github.com/sansan0/TrendRadar/issues) 反馈，帮助改进项目 > > **特别感谢**： > - 感谢 [@DYZYD](https://github.com/DYZYD) 贡献天翼邮箱（189.cn）配置并完成自发自收测试 ([#291](https://github.com/sansan0/TrendRadar/issues/291)) > - 感谢 [@longzhenren](https://github.com/longzhenren) 贡献阿里云邮箱（aliyun.com）配置并完成测试 ([#344](https://github.com/sansan0/TrendRadar/issues/344)) > - 感谢 [@ACANX](https://github.com/ACANX) 贡献 Yandex 邮箱（yandex.com）配置并完成测试 ([#663](https://github.com/sansan0/TrendRadar/issues/663)) > - 感谢 [@Sleepy-Tianhao](https://github.com/Sleepy-Tianhao) 贡献 iCloud 邮箱（icloud.com）配置并完成测试 ([#728](https://github.com/sansan0/TrendRadar/issues/728)) **常见邮箱设置：** #### QQ邮箱： 1. 登录 QQ邮箱网页版 → 设置 → 账户 2. 开启 POP3/SMTP 服务 3. 生成授权码（16位字母） 4. `EMAIL_PASSWORD` 填写授权码，而非 QQ 密码 #### Gmail： 1. 开启两步验证 2. 生成应用专用密码 3. `EMAIL_PASSWORD` 填写应用专用密码 #### 163/126邮箱： 1. 登录网页版 → 设置 → POP3/SMTP/IMAP 2. 开启 SMTP 服务 3. 设置客户端授权码 4. `EMAIL_PASSWORD` 填写授权码
**高级配置**： 如果自动识别失败，可手动配置 SMTP： - `EMAIL_SMTP_SERVER`：如 smtp.gmail.com - `EMAIL_SMTP_PORT`：如 587（TLS）或 465（SSL）
**如果有多个收件人(注意是英文逗号分隔)**： - EMAIL_TO="user1@example.com,user2@example.com,user3@example.com"
👉 点击展开：ntfy 推送（开源免费，支持自托管）
**两种使用方式：** ### 方式一：免费使用（推荐新手） 🆓 **特点**： - ✅ 无需注册账号，立即使用 - ✅ 每天 250 条消息（足够 90% 用户） - ✅ Topic 名称即"密码"（需选择不易猜测的名称） - ⚠️ 消息未加密，不适合敏感信息, 但适合我们这个项目的不敏感信息 **快速开始：** 1. **下载 ntfy 应用**： - Android：[Google Play](https://play.google.com/store/apps/details?id=io.heckel.ntfy) / [F-Droid](https://f-droid.org/en/packages/io.heckel.ntfy/) - iOS：[App Store](https://apps.apple.com/us/app/ntfy/id1625396347) - 桌面：访问 [ntfy.sh](https://ntfy.sh) 2. **订阅主题**（选择一个难猜的名称）： ``` 建议格式：trendradar-{你的名字缩写}-{随机数字} 不能使用中文 ✅ 好例子：trendradar-zs-8492 ❌ 坏例子：news、alerts（太容易被猜到） ``` 3. **配置 GitHub Secret（⚠️ Name 名称必须严格一致）**： - **Name（名称）**：`NTFY_TOPIC`（请复制粘贴此名称，不要手打） - **Secret（值）**：填写你刚才订阅的主题名称 - **Name（名称）**：`NTFY_SERVER_URL`（可选配置，请复制粘贴此名称） - **Secret（值）**：留空（默认使用 ntfy.sh） - **Name（名称）**：`NTFY_TOKEN`（可选配置，请复制粘贴此名称） - **Secret（值）**：留空 **说明**：ntfy 至少需要配置 1 个必需 Secret (NTFY_TOPIC)，后两个为可选配置 4. **测试**： ```bash curl -d "测试消息" ntfy.sh/你的主题名称 ``` --- ### 方式二：自托管（完全隐私控制） 🔒 **适合人群**：有服务器、追求完全隐私、技术能力强 **优势**： - ✅ 完全开源（Apache 2.0 + GPLv2） - ✅ 数据完全自主控制 - ✅ 无任何限制 - ✅ 零费用 **Docker 一键部署**： ```bash docker run -d \ --name ntfy \ -p 80:80 \ -v /var/cache/ntfy:/var/cache/ntfy \ binwiederhier/ntfy \ serve --cache-file /var/cache/ntfy/cache.db ``` **配置 TrendRadar**： ```yaml NTFY_SERVER_URL: https://ntfy.yourdomain.com NTFY_TOPIC: trendradar-alerts # 自托管可用简单名称 NTFY_TOKEN: tk_your_token # 可选：启用访问控制 ``` **在应用中订阅**： - 点击"Use another server" - 输入你的服务器地址 - 输入主题名称 - （可选）输入登录凭据 --- **常见问题：**
Q1: 免费版够用吗？ 每天 250 条消息对大多数用户足够。按 30 分钟抓取一次计算，每天约 48 次推送，完全够用。
Q2: Topic 名称真的安全吗？ 如果你选择随机的、足够长的名称（如 `trendradar-zs-8492-news`），暴力破解几乎不可能： - ntfy 有严格的速率限制（1 秒 1 次请求） - 64 个字符选择（A-Z, a-z, 0-9, _, -） - 10 位随机字符串有 64^10 种可能性（需要数年才能破解）
--- **推荐选择：** | 用户类型 | 推荐方案 | 理由 | |---------|---------|------| | 普通用户 | 方式一（免费） | 简单快速，够用 | | 技术用户 | 方式二（自托管） | 完全控制，无限制 | | 高频用户 | 方式三（付费） | 这个自己去官网看吧 | **相关链接：** - [ntfy 官方文档](https://docs.ntfy.sh/) - [自托管教程](https://docs.ntfy.sh/install/) - [GitHub 仓库](https://github.com/binwiederhier/ntfy)
👉 点击展开：Bark 推送（iOS 专属，简洁高效）
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`BARK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的 Bark 推送 URL
**Bark 简介：** Bark 是一款 iOS 平台的免费开源推送工具，特点是简单、快速、无广告。 **使用方式：** ### 方式一：使用官方服务器（推荐新手） 🆓 1. **下载 Bark App**： - iOS：[App Store](https://apps.apple.com/cn/app/bark-给你的手机发推送/id1403753865) 2. **获取推送 URL**： - 打开 Bark App - 复制首页显示的推送 URL（格式如：`https://api.day.app/your_device_key`） - 将 URL 配置到 GitHub Secrets 中的 `BARK_URL` ### 方式二：自建服务器（完全隐私控制） 🔒 **适合人群**：有服务器、追求完全隐私、技术能力强 **Docker 一键部署**： ```bash docker run -d \ --name bark-server \ -p 8080:8080 \ finab/bark-server ``` **配置 TrendRadar**： ```yaml BARK_URL: http://your-server-ip:8080/your_device_key ``` --- **注意事项：** - ✅ Bark 使用 APNs 推送，单条消息最大 4KB - ✅ 支持自动分批推送，无需担心消息过长 - ✅ 推送格式为纯文本（自动去除 Markdown 语法） - ⚠️ 仅支持 iOS 平台 **相关链接：** - [Bark 官方网站](https://bark.day.app/) - [Bark GitHub 仓库](https://github.com/Finb/Bark) - [Bark Server 自建教程](https://github.com/Finb/bark-server)
👉 点击展开：Slack 推送
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`SLACK_WEBHOOK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的 Slack Incoming Webhook URL
**Slack 简介：** Slack 是团队协作工具，Incoming Webhooks 可以将消息推送到 Slack 频道。 **设置步骤：** ### 步骤 1：创建 Slack App 1. **访问 Slack API 页面**： - 打开 https://api.slack.com/apps?new_app=1 - 如果未登录，先登录你的 Slack 工作空间 2. **选择创建方式**： - 点击 **"From scratch"**（从头开始创建） 3. **填写 App 信息**： - **App Name**：填写应用名称（如 `TrendRadar` 或 `热点新闻监控`） - **Workspace**：从下拉列表选择你的工作空间 - 点击 **"Create App"** 按钮 ### 步骤 2：启用 Incoming Webhooks 1. **导航到 Incoming Webhooks**： - 在左侧菜单中找到并点击 **"Incoming Webhooks"** 2. **启用功能**： - 找到 **"Activate Incoming Webhooks"** 开关 - 将开关从 `OFF` 切换到 `ON` - 页面会自动刷新显示新的配置选项 ### 步骤 3：生成 Webhook URL 1. **添加新的 Webhook**： - 滚动到页面底部 - 点击 **"Add New Webhook to Workspace"** 按钮 2. **选择目标频道**： - 系统会弹出授权页面 - 从下拉列表中选择要接收消息的频道（如 `#热点新闻`） - ⚠️ 如果要选择私有频道，必须先加入该频道 3. **授权应用**： - 点击 **"Allow"** 按钮完成授权 - 系统会自动跳转回配置页面 ### 步骤 4：复制并保存 Webhook URL 1. **查看生成的 URL**： - 在 "Webhook URLs for Your Workspace" 区域 - 会看到刚刚生成的 Webhook URL - 格式如：`https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX` 2. **复制 URL**： - 点击 URL 右侧的 **"Copy"** 按钮 - 或手动选中 URL 并复制 3. **配置到 TrendRadar**： - **GitHub Actions**：将 URL 添加到 GitHub Secrets 中的 `SLACK_WEBHOOK_URL` - **本地测试**：将 URL 填入 `config/config.yaml` 的 `slack_webhook_url` 字段 - **Docker 部署**：将 URL 添加到 `docker/.env` 文件的 `SLACK_WEBHOOK_URL` 变量 --- **注意事项：** - ✅ 支持 Markdown 格式（自动转换为 Slack mrkdwn） - ✅ 支持自动分批推送（每批 4KB） - ✅ 适合团队协作，消息集中管理 - ⚠️ Webhook URL 包含密钥，切勿公开 **消息格式预览：** ``` *[第 1/2 批次]* 📊 *热点词汇统计* 🔥 *[1/3] AI ChatGPT* : 2 条 1. [百度热搜] 🆕 ChatGPT-5正式发布 *[1]* - 09时15分 (1次) 2. [今日头条] AI芯片概念股暴涨 *[3]* - [08时30分 ~ 10时45分] (3次) ``` **相关链接：** - [Slack Incoming Webhooks 官方文档](https://api.slack.com/messaging/webhooks) - [Slack API 应用管理](https://api.slack.com/apps)
👉 点击展开：通用 Webhook 推送（支持 Discord、Matrix、IFTTT 等）
**GitHub Secret 配置（⚠️ Name 名称必须严格一致）：** - **Name（名称）**：`GENERIC_WEBHOOK_URL`（请复制粘贴此名称，不要手打） - **Secret（值）**：你的 Webhook URL - **Name（名称）**：`GENERIC_WEBHOOK_TEMPLATE`（可选配置，请复制粘贴此名称） - **Secret（值）**：JSON 模板字符串，支持 `{title}` 和 `{content}` 占位符
**通用 Webhook 简介：** 通用 Webhook 支持任意接受 HTTP POST 请求的平台，包括但不限于： - **Discord**：通过 Webhook 推送到频道 - **Matrix**：通过 Webhook 桥接推送 - **IFTTT**：触发自动化流程 - **自建服务**：任何支持 Webhook 的自定义服务 **配置示例：** ### Discord 配置 1. **获取 Webhook URL**： - 进入 Discord 服务器设置 → 整合 → Webhooks - 创建新 Webhook，复制 URL 2. **配置模板**： ```json {"content": "{content}"} ``` 3. **GitHub Secret 配置**： - `GENERIC_WEBHOOK_URL`：Discord Webhook URL - `GENERIC_WEBHOOK_TEMPLATE`：`{"content": "{content}"}` ### 自定义模板 模板支持两个占位符： - `{title}` - 消息标题 - `{content}` - 消息内容 **模板示例**： ```json # 默认格式（留空时使用） {"title": "{title}", "content": "{content}"} # Discord 格式 {"content": "{content}"} # 自定义格式 {"text": "{content}", "username": "TrendRadar"} ``` --- **注意事项：** - ✅ 支持 Markdown 格式（与企业微信格式一致） - ✅ 支持自动分批推送 - ✅ 支持多账号配置（用 `;` 分隔） - ⚠️ 模板必须是有效的 JSON 格式 - ⚠️ 不同平台对消息格式要求不同，请参考目标平台文档

### 3️⃣ 第三步：手动测试新闻推送 > ⚠️ 提醒： > - 完成第 1-2 步后，请立即测试！测试成功后再根据需要调整配置（第 4 步） > - 请进入你自己的项目，不是本项目！ **如何找到你的 Actions 页面**： - **方法一**：打开你 fork 的项目主页，点击顶部的 **Actions** 标签 - **方法二**：直接访问 `https://github.com/你的用户名/TrendRadar/actions` **示例对比**： - ❌ 作者的项目：`https://github.com/sansan0/TrendRadar/actions` - ✅ 你的项目：`https://github.com/你的用户名/TrendRadar/actions` **测试步骤**： 1. 进入你项目的 Actions 页面 2. 找到 **"Get Hot News"**(必须得是这个字)点进去，点击右侧的 **"Run workflow"** 按钮运行 - 如果看不到该字样，参照 [#109](https://github.com/sansan0/TrendRadar/issues/109) 解决 3. 3 分钟左右，消息会推送到你配置的平台
> ⚠️ 提醒： > - 手动测试不要太频繁，避免触发 GitHub Actions 限制 > - 点击 Run workflow 后需要刷新浏览器页面才能看到新的运行记录
### 4️⃣ 第四步：配置说明（可选） 默认配置已可正常使用，如需个性化调整，了解以下文件即可： | 文件 | 作用 | |------|------| | `config/config.yaml` | 主配置文件：推送模式、时间窗口、平台列表、热点权重等 | | `config/frequency_words.txt` | 关键词文件：设置你关心的词汇，筛选推送内容 | | `config/ai_analysis_prompt.txt` | AI 提示词模板：自定义 AI 分析师的角色和分析维度 | | `.github/workflows/crawler.yml` | 执行频率：控制多久运行一次（⚠️ 谨慎修改） | 👉 **详细配置教程**：[配置详解](#配置详解)
### 5️⃣ 第五步：远程云存储 & 签到配置 **v4.0.0 重要变更**：引入「活跃度检测」机制，GitHub Actions 需定期签到以维持运行。 - **运行周期**：有效期为 **7 天**，倒计时结束后服务将自动挂起。 - **续期方式**：在 Actions 页面手动触发 "Check In" workflow，即可重置 7 天有效期。 - **操作路径**：`Actions` → `Check In` → `Run workflow` - **设计理念**： - 如果 7 天都忘了签到，或许这些资讯对你来说并非刚需。适时的暂停，能帮你从信息流中抽离，给大脑留出喘息的空间。 - GitHub Actions 是宝贵的公共计算资源。引入签到机制旨在避免算力的无效空转，确保资源能分配给真正活跃且需要的用户。感谢你的理解与支持。 --- **关于远程云存储配置（请根据部署方式选择）：** - **GitHub Actions 用户**： - **现状**：Actions 每次运行都是全新环境，不保存文件。如果不配置云存储，项目将运行在**轻量模式**（无增量推送、无历史追踪）。 - **建议**：配置远程云存储以获得完整体验。 - **Docker / 本地用户**： - **现状**：数据默认保存在本地硬盘。 - **建议**：云存储为可选项，可作为异地备份。
👉 点击展开：远程云存储配置教程（以 Cloudflare R2 为例）
**⚠️ 前置条件（重要）：** 根据 Cloudflare 平台规则，开通 R2 需绑定支付方式。 * **目的**：仅作身份验证（Verify Only），**不产生扣费**。 * **支付**：支持双币信用卡或国区 PayPal。 * **用量**：R2 的免费额度（10GB存储/月）足以覆盖本项目日常运行，无需担心付费。 --- **GitHub Secret 配置（需添加 4 项）：** | Name（名称） | Secret（值）说明 | |-------------|-----------------| | `S3_BUCKET_NAME` | 存储桶名称（如 `trendradar-data`） | | `S3_ACCESS_KEY_ID` | 访问密钥 ID（Access Key ID） | | `S3_SECRET_ACCESS_KEY` | 访问密钥（Secret Access Key） | | `S3_ENDPOINT_URL` | S3 API 端点（如 R2：`https://.r2.cloudflarestorage.com`） | **可选配置：** | Name（名称） | Secret（值）说明 | |-------------|-----------------| | `S3_REGION` | 区域（默认 `auto`，部分服务商可能需要指定） | > 💡 **更多存储配置选项**：参见 [数据保存在哪里？](#11-数据保存在哪里)
**详细操作步骤（获取凭据）：** 1. **进入 R2 概览**： - 登录 [Cloudflare Dashboard](https://dash.cloudflare.com/)。 - 在左侧侧边栏找到并点击 `R2对象存储`。 2. **创建存储桶**： - 点击`概述` - 点击右上角的 `创建存储桶` (Create bucket)。 - 输入名称（例如 `trendradar-data`），点击 `创建存储桶`。 3. **创建 API 令牌**： - 回到 **概述**页面。 - 点击**右下角** `Account Details `找到并点击 `Manage` (Manage R2 API Tokens)。 - 同时你会看到 `S3 API`：`https://.r2.cloudflarestorage.com`(这就是 S3_ENDPOINT_URL) - 点击 `创建 Account APl 令牌` 。 - **⚠️ 关键设置**： - **令牌名称**：随意填写（如 `github-action-write`）。 - **权限**：选择 `管理员读和写` 。 - **指定存储桶**：为了安全，建议选择 `仅适用于指定存储桶` 并选中你的桶（如 `trendradar-data`）。 - 点击 `创建 API 令牌`，**立即复制** 显示的 `Access Key ID` 和 `Secret Access Key`（只显示一次！）。

### 6️⃣ 第六步：开启 AI 分析推送 这是 v5.0.0 的核心功能，让 AI 帮你总结和分析新闻，建议尝试。 **配置方法：** 在 GitHub Secrets (或 `.env` / `config.yaml`) 中添加： - `AI_API_KEY`: 你的 API Key（支持 DeepSeek、OpenAI 等） - `AI_PROVIDER`: 服务商名称（如 `deepseek`, `openai`） 就这样，无需复杂部署，下次推送时你就会看到智能分析报告了。
### 7️⃣ 第七步：🎉 部署成功！ 恭喜！现在你可以开始享受 TrendRadar 带来的高效信息流了。 💬 **加入社区**：欢迎关注公众号「**[硅基茶水间](#-支持项目)**」，分享你的使用心得和高级玩法。
### 8️⃣ 第八步：进阶：选择你的 AI 助手 TrendRadar 提供了两种 AI 使用方式，满足不同需求： | 特性 | ✨ AI 分析推送 | 🧠 AI 智能分析 | | :--- | :--- | :--- | | **模式** | **被动接收** (每日日报) | **主动对话** (深度调研) | | **场景** | "今天有什么大事？" | "分析一下过去一周 AI 行业的变化" | | **部署** | 极简 (填 Key 即可) | 进阶 (需本地运行/Docker) | | **客户端** | 手机 | 电脑 | 👉 **结论**：先用 **AI 分析推送** 满足日常需求；如果你是数据分析师或需要深度挖掘，再尝试 **[AI 智能分析](#-ai-智能分析)**。
## ⚙️ 配置详解 > **📖 提醒**：本章节提供详细的配置说明，建议先完成 [快速开始](#-快速开始) 的基础配置，再根据需要回来查看详细选项。 ### 1. 我要看哪些平台？
👉 点击展开：选择资讯来源
**配置位置：** `config/config.yaml` 的 `platforms` 部分 本项目的资讯数据来源于 [newsnow](https://github.com/ourongxing/newsnow) ，你可以点击[网站](https://newsnow.busiyi.world/)，点击[更多]，查看是否有你想要的平台。 具体添加可访问 [项目源代码](https://github.com/ourongxing/newsnow/tree/main/server/sources)，根据里面的文件名，在 `config/config.yaml` 文件中修改 `platforms` 配置： ```yaml platforms: enabled: true # 是否启用热榜平台抓取 sources: - id: "toutiao" name: "今日头条" - id: "baidu" name: "百度热搜" - id: "wallstreetcn-hot" name: "华尔街见闻" # 添加更多平台... ``` > 💡 **快捷方式**：如果不会看源代码，可以复制他人整理好的 [平台配置汇总](https://github.com/sansan0/TrendRadar/issues/95) > ⚠️ **注意**：平台不是越多越好，建议选择 10-15 个核心平台。过多平台会导致信息过载，反而降低使用体验。
### 2. 我关心什么内容？ 在 `frequency_words.txt` 文件中告诉机器人你想看什么，它就会帮你盯着。支持普通词、必须词、过滤词等多种玩法。 | 语法类型 | 符号 | 作用 | 示例 | 匹配逻辑 | |---------|------|------|------|---------| | **普通词** | 无 | 基础匹配 | `华为` | 包含任意一个即可 | | **必须词** | `+` | 限定范围 | `+手机` | 必须同时包含 | | **过滤词** | `!` | 排除干扰 | `!广告` | 包含则直接排除 | | **数量限制** | `@` | 控制显示数量 | `@10` | 最多显示10条新闻（v3.2.0新增） | | **全局过滤** | `[GLOBAL_FILTER]` | 全局排除指定内容 | 见下方示例 | 任何情况下都过滤（v3.5.0新增） | | **正则表达式** | `/pattern/` | 精确匹配模式 | `/\bai\b/` | 使用正则表达式匹配（v4.7.0新增） | | **显示名称** | `=> 备注` | 自定义显示文本 | `/\bai\b/ => AI相关` | 推送和HTML显示备注名称（v4.7.0新增） | #### 2.1 基础语法
👉 点击展开：基础语法教程
**配置位置：** `config/frequency_words.txt` ##### 1. **普通关键词** - 基础匹配 ```txt 华为 OPPO 苹果 ``` **作用：** 新闻标题包含其中**任意一个词**就会被捕获 ##### 2. **必须词** `+词汇` - 限定范围 ```txt 华为 OPPO +手机 ``` **作用：** 必须同时包含普通词**和**必须词才会被捕获 ##### 3. **过滤词** `!词汇` - 排除干扰 ```txt 苹果 华为 !水果 !价格 ``` **作用：** 包含过滤词的新闻会被**直接排除**，即使包含关键词 ##### 4. **数量限制** `@数字` - 控制显示数量（v3.2.0 新增） ```txt 特斯拉 马斯克 @5 ``` **作用：** 限制该关键词组最多显示的新闻条数 **配置优先级：** `@数字` > 全局配置 > 不限制 ##### 5. **全局过滤** `[GLOBAL_FILTER]` - 全局排除指定内容（v3.5.0 新增） ```txt [GLOBAL_FILTER] 广告 推广 营销 震惊 标题党 [WORD_GROUPS] 科技 AI 华为 鸿蒙 !车 ``` **作用：** 在任何情况下过滤包含指定词的新闻，**优先级最高** **使用场景：** - 过滤低质内容：震惊、标题党、爆料等 - 过滤营销内容：广告、推广、赞助等 - 过滤特定主题：娱乐、八卦（根据需求） **过滤优先级：** 全局过滤 > 词组内过滤(`!`) > 词组匹配 **区域说明：** - `[GLOBAL_FILTER]`：全局过滤区，包含的词在任何情况下都会被过滤 - `[WORD_GROUPS]`：词组区，保持现有语法（`!`、`+`、`@`） - 如果不使用区域标记，默认全部作为词组处理（向后兼容） **匹配示例：** ```txt [GLOBAL_FILTER] 广告 [WORD_GROUPS] 科技 AI ``` - ❌ "广告：最新科技产品发布" ← 包含全局过滤词"广告"，直接拒绝 - ✅ "科技公司发布AI新产品" ← 不包含全局过滤词，匹配"科技"词组 - ✅ "AI技术突破引发关注" ← 不包含全局过滤词，匹配"科技"词组中的"AI" **注意事项：** - 全局过滤词应谨慎使用，避免过度过滤导致遗漏有价值内容 - 建议全局过滤词控制在 5-15 个以内 - 对于特定词组的过滤，优先使用词组内过滤词（`!` 前缀） ##### 6. **正则表达式** `/pattern/` - 精确匹配模式（v4.7.0 新增） 普通关键词使用子字符串匹配，这在中文环境下很方便，但在英文环境可能会产生误匹配。例如 `ai` 会匹配到 `training` 中的 `ai`。 使用正则表达式语法 `/pattern/` 可以实现精确匹配： ```txt /(? 我需要一个 Python 正则表达式，用于匹配英文单词 "ai"，但不匹配 "training" 中的 "ai"。 > 请直接给出正则表达式，格式为 `/pattern/`，不需要额外解释。 AI 会给你类似这样的结果：`/(? 备注` - 自定义显示文本（v4.7.0 新增） 正则表达式在推送消息和 HTML 页面显示时可能不太友好。使用 `=> 备注` 语法可以设置显示名称： ```txt /(? AI 相关 人工智能 ``` **作用：** 推送消息和 HTML 页面显示 "AI 相关" 而不是复杂的正则表达式 **语法格式：** ```txt # 正则 + 显示名称 /pattern/ => 显示名称 /pattern/i => 显示名称 # 支持 flags 写法（flags 被忽略） /pattern/=>显示名称 # => 两边空格可选 # 普通词 + 显示名称 deepseek => DeepSeek 动态 ``` **匹配示例：** ```txt # 配置 /(? AI 相关 人工智能 ``` | 原始配置 | 推送/HTML 显示 | |---------|---------------| | `/(? AI 相关` + `人工智能` | **`AI 相关`** | **注意事项：** - 显示名称只需写在词组的第一个词上 - 如果词组中多个词都有显示名称，使用第一个 - 不设置显示名称时，自动使用词组内所有词拼接 --- #### 🔗 词组功能 - 空行分隔的重要作用 **核心规则：** 用**空行**分隔不同的词组，每个词组独立统计 ##### 示例配置： ```txt iPhone 华为 OPPO +发布 A股 上证 深证 +涨跌 !预测 世界杯 欧洲杯 亚洲杯 +比赛 ``` ##### 词组解释及匹配效果： **第1组 - 手机新品类：** - 关键词：iPhone、华为、OPPO - 必须词：发布 - 效果：必须包含手机品牌名，同时包含"发布" **匹配示例：** - ✅ "iPhone 15正式发布售价公布" ← 有"iPhone"+"发布" - ✅ "华为Mate60系列发布会直播" ← 有"华为"+"发布" - ✅ "OPPO Find X7发布时间确定" ← 有"OPPO"+"发布" - ❌ "iPhone销量创新高" ← 有"iPhone"但缺少"发布" **第2组 - 股市行情类：** - 关键词：A股、上证、深证 - 必须词：涨跌 - 过滤词：预测 - 效果：关注股市涨跌实况，排除预测类内容 **匹配示例：** - ✅ "A股今日大幅涨跌分析" ← 有"A股"+"涨跌" - ✅ "上证指数涨跌幅创新高" ← 有"上证"+"涨跌" - ❌ "专家预测A股涨跌趋势" ← 有"A股"+"涨跌"但包含"预测" **第3组 - 足球赛事类：** - 关键词：世界杯、欧洲杯、亚洲杯 - 必须词：比赛 - 效果：只关注比赛相关新闻 --- #### 📝 配置技巧 ##### 1. **从宽到严** ```txt # 第一步：先用宽泛关键词测试 人工智能 AI ChatGPT # 第二步：发现误匹配后，加入必须词限定 人工智能 AI ChatGPT +技术 # 第三步：发现干扰内容后，加入过滤词 人工智能 AI ChatGPT +技术 !广告 !培训 ``` ##### 2. **避免过度复杂** ❌ **不推荐：** 一个词组包含太多词汇 ```txt 华为 OPPO 苹果 三星 vivo 一加 魅族 +手机 +发布 +销量 !假货 !维修 !二手 ``` ✅ **推荐：** 拆分成多个精确的词组 ```txt 华为 OPPO +新品 苹果 三星 +发布 手机 销量 +市场 ```
#### 2.2 高级配置（v3.2.0 新增）
👉 点击展开：高级配置教程
##### 关键词排序优先级 **配置位置：** `config/config.yaml` ```yaml report: sort_by_position_first: false # 排序优先级配置 ``` | 配置值 | 排序规则 | 适用场景 | |--------|---------|---------| | `false`（默认） | 热点条数 ↓ → 配置位置 ↑ | 关注热度趋势 | | `true` | 配置位置 ↑ → 热点条数 ↓ | 关注个人优先级 | **示例：** 配置顺序 A、B、C，热点数 A(3条)、B(10条)、C(5条) - `false`：B(10条) → C(5条) → A(3条) - `true`：A(3条) → B(10条) → C(5条) ##### 全局显示数量限制 ```yaml report: max_news_per_keyword: 10 # 每个关键词最多显示10条（0=不限制） ``` **Docker 环境变量：** ```bash SORT_BY_POSITION_FIRST=true MAX_NEWS_PER_KEYWORD=10 ``` **综合示例：** ```yaml # config.yaml report: sort_by_position_first: true # 按配置顺序优先 max_news_per_keyword: 10 # 全局默认每个关键词最多10条 ``` ```txt # frequency_words.txt 特斯拉 马斯克 @20 # 重点关注，显示20条（覆盖全局配置） 华为 # 使用全局配置，显示10条 比亚迪 @5 # 限制5条 ``` **最终效果：** 按配置顺序显示 特斯拉(20条) → 华为(10条) → 比亚迪(5条)
### 3. 推送模式选哪个？
👉 点击展开：三种推送模式详细对比
**配置位置：** `config/config.yaml` 的 `report.mode` ```yaml report: mode: "daily" # 可选: "daily" | "incremental" | "current" ``` #### 详细对比表格 | 模式 | 适用人群 | 推送时机 | 显示内容 | 典型使用场景 | |------|----------|----------|----------|------------| | **当日汇总**
`daily` | 📋 企业管理者/普通用户 | 按时推送(默认每小时推送一次) | 当日所有匹配新闻
+ 新增新闻区域 | **案例**：每天下午6点查看今天所有重要新闻
**特点**：看全天完整趋势，不漏掉任何热点
**提醒**：会包含之前推送过的新闻 | | **当前榜单**
`current` | 📰 自媒体人/内容创作者 | 按时推送(默认每小时推送一次) | 当前榜单匹配新闻
+ 新增新闻区域 | **案例**：每小时追踪"哪些话题现在最火"
**特点**：实时了解当前热度排名变化
**提醒**：持续在榜的新闻每次都会出现 | | **增量监控**
`incremental` | 📈 投资者/交易员 | 有新增才推送 | 新出现的匹配频率词新闻 | **案例**：监控"特斯拉"，只在有新消息时通知
**特点**：零重复，只看首次出现的新闻
**适合**：高频监控、避免信息打扰 | #### 实际推送效果举例 假设你监控"苹果"关键词，每小时执行一次： | 时间 | daily 模式推送 | current 模式推送 | incremental 模式推送 | |-----|--------------|----------------|-------------------| | 10:00 | 新闻A、新闻B | 新闻A、新闻B | 新闻A、新闻B | | 11:00 | 新闻A、新闻B、新闻C | 新闻B、新闻C、新闻D | **仅**新闻C | | 12:00 | 新闻A、新闻B、新闻C | 新闻C、新闻D、新闻E | **仅**新闻D、新闻E | **说明**： - `daily`：累积展示当天所有新闻（A、B、C 都保留） - `current`：展示当前榜单的新闻（排名变化，新闻D上榜，新闻A掉榜） - `incremental`：**只推送新出现的新闻**（避免重复干扰） #### 常见问题 > **💡 遇到这个问题？** 👉 "每个小时执行一次，第一次执行完输出的新闻，在下一个小时执行时还会出现" > - **原因**：你可能选择了 `daily`（当日汇总）或 `current`（当前榜单）模式 > - **解决**：改用 `incremental`（增量监控）模式，只推送新增内容 #### ⚠️ 增量模式重要提示 > **选择了 `incremental`（增量监控）模式的用户请注意：** > > 📌 **增量模式只在有新增匹配新闻时才会推送** > > **如果长时间没有收到推送，可能是因为：** > 1. 当前时段没有符合你关键词的新热点出现 > 2. 关键词配置过于严格或过于宽泛 > 3. 监控平台数量较少 > > **解决方案：** > - 方案1：👉 [优化关键词配置](#2-关键词配置) - 调整关键词的精准度，增加或修改监控词汇 > - 方案2：切换推送模式 - 改用 `current` 或 `daily` 模式，可以定时接收推送 > - 方案3：👉 [增加监控平台](#1-平台配置) - 添加更多新闻平台，扩大信息来源
### 4. 调整热点算法
👉 点击展开：自定义热点权重
**配置位置：** `config/config.yaml` 的 `advanced.weight` 部分 ```yaml advanced: weight: rank: 0.6 # 排名权重 frequency: 0.3 # 频次权重 hotness: 0.1 # 热度权重 ``` 当前默认的配置是平衡性配置 #### 两个核心场景 **追实时热点型**： ```yaml advanced: weight: rank: 0.8 # 主要看排名 frequency: 0.1 # 不太在乎持续性 hotness: 0.1 ``` **适用人群**：自媒体博主、营销人员、想快速了解当下最火话题的用户 **追深度话题型**： ```yaml advanced: weight: rank: 0.4 # 适度看排名 frequency: 0.5 # 重视当天内的持续热度 hotness: 0.1 ``` **适用人群**：投资者、研究人员、新闻工作者、需要深度分析趋势的用户 #### 调整的方法 1. **三个数字加起来必须等于 1.0** 2. **哪个重要就调大哪个**：在乎排名就调大 `rank`，在乎持续性就调大 `frequency` 3. **建议每次只调 0.1-0.2**，观察效果 核心思路：追求速度和时效性的用户提高排名权重，追求深度和稳定性的用户提高频次权重。
### 5. 我收到的消息长什么样？
👉 点击展开：消息样式预览
#### 推送示例 📊 热点词汇统计 🔥 [1/3] AI ChatGPT : 2 条 1. [百度热搜] 🆕 ChatGPT-5正式发布 [**1**] - 09时15分 (1次) 2. [今日头条] AI芯片概念股暴涨 [**3**] - [08时30分 ~ 10时45分] (3次) ━━━━━━━━━━━━━━━━━━━ 📈 [2/3] 比亚迪 特斯拉 : 2 条 1. [微博] 🆕 比亚迪月销量破纪录 [**2**] - 10时20分 (1次) 2. [抖音] 特斯拉降价促销 [**4**] - [07时45分 ~ 09时15分] (2次) ━━━━━━━━━━━━━━━━━━━ 📌 [3/3] A股 股市 : 1 条 1. [华尔街见闻] A股午盘点评分析 [**5**] - [11时30分 ~ 12时00分] (2次) 🆕 本次新增热点新闻 (共 2 条) **百度热搜** (1 条): 1. ChatGPT-5正式发布 [**1**] **微博** (1 条): 1. 比亚迪月销量破纪录 [**2**] 更新时间：2025-01-15 12:30:15 #### 消息格式说明 | 格式元素 | 示例 | 含义 | 说明 | | ------------- | --------------------------- | ------------ | --------------------------------------- | | 🔥📈📌 | 🔥 [1/3] AI ChatGPT | 热度等级 | 🔥高热度(≥10条) 📈中热度(5-9条) 📌普通热度(<5条) | | [序号/总数] | [1/3] | 排序位置 | 当前词组在所有匹配词组中的排名 | | 频率词组 | AI ChatGPT | 关键词组 | 配置文件中的词组，标题必须包含其中词汇 | | : N 条 | : 2 条 | 匹配数量 | 该词组匹配的新闻总数 | | [平台名] | [百度热搜] | 来源平台 | 新闻所属的平台名称 | | 🆕 | 🆕 ChatGPT-5正式发布 | 新增标记 | 本轮抓取中首次出现的热点 | | [**数字**] | [**1**] | 高排名 | 排名≤阈值的热搜，红色加粗显示 | | [数字] | [7] | 普通排名 | 排名>阈值的热搜，普通显示 | | - 时间 | - 09时15分 | 首次时间 | 该新闻首次被发现的时间 | | [时间~时间] | [08时30分 ~ 10时45分] | 持续时间 | 从首次出现到最后出现的时间范围 | | (N次) | (3次) | 出现频率 | 在监控期间出现的总次数 | | **新增区域** | 🆕 **本次新增热点新闻** | 新话题汇总 | 单独展示本轮新出现的热点话题 |
### 6. Docker 部署 **镜像说明：** TrendRadar 提供两个独立的 Docker 镜像，可根据需求选择部署： | 镜像名称 | 用途 | 说明 | |---------|------|------| | `wantcat/trendradar` | 新闻推送服务 | 定时抓取新闻、推送通知（必选） | | `wantcat/trendradar-mcp` | AI 分析服务 | MCP 协议支持、AI 对话分析（可选） | > 💡 **建议**： > - 只需要推送功能：仅部署 `wantcat/trendradar` 镜像 > - 需要 AI 分析功能：同时部署两个镜像
👉 点击展开：Docker 部署完整指南
#### 方式一：使用 docker compose（推荐） 1. **创建项目目录和配置**: ```bash # 克隆项目到本地 git clone https://github.com/sansan0/TrendRadar.git cd TrendRadar ``` > 💡 **说明**：Docker 部署需要的关键目录结构如下： ``` 当前目录/ ├── config/ │ ├── config.yaml # 核心功能配置（必需） │ ├── frequency_words.txt # 关键词配置（必需） │ ├── timeline.yaml # 时间线配置 │ ├── ai_analysis_prompt.txt # AI 分析提示词（可选） │ ├── ai_translation_prompt.txt # AI 翻译提示词（可选） │ ├── ai_interests.txt # AI 兴趣过滤配置（可选） │ ├── ai_filter/ # AI 过滤相关提示词 │ │ ├── prompt.txt │ │ ├── extract_prompt.txt │ │ └── update_tags_prompt.txt │ └── custom/ # 用户自定义配置（可选） │ ├── ai/ # 自定义 AI 提示词 │ └── keyword/ # 自定义关键词文件 └── docker/ ├── .env # 敏感信息 + Docker 特有配置 └── docker-compose.yml # Docker Compose 编排文件 ``` 2. **配置文件说明**: **配置分工原则（v4.6.0 优化）**： | 文件 | 用途 | 修改频率 | 说明 | |------|------|---------|------| | `config/config.yaml` | **核心功能配置** | 低 | 报告模式、推送设置、存储格式、推送窗口、AI 分析开关、平台启用等全局行为控制 | | `config/frequency_words.txt` | **关键词配置** | 高 | 设置你关心的热点词汇，支持分组、正则、别名等高级语法 | | `config/timeline.yaml` | **时间线配置** | 低 | 控制新闻时间线的展示和过滤规则 | | `config/ai_analysis_prompt.txt` | **AI 分析提示词** | 中 | 自定义 AI 分析的角色定义和输出格式（v5.0.0+） | | `config/ai_translation_prompt.txt` | **AI 翻译提示词** | 低 | 自定义 AI 翻译的提示词模板 | | `config/ai_interests.txt` | **AI 兴趣过滤** | 中 | 定义 AI 基于兴趣自动过滤新闻的规则 | | `config/ai_filter/` | **AI 过滤提示词** | 低 | AI 过滤模块的内部提示词（一般无需修改） | | `config/custom/` | **用户自定义扩展** | 按需 | `custom/ai/` 放自定义 AI 提示词，`custom/keyword/` 放自定义关键词文件 | | `docker/.env` | **敏感信息 + Docker 特有配置** | 低 | webhook URLs、API Key、S3 密钥、定时任务等，**不会被 git 追踪** | > 💡 **分工要点**： > - **功能行为** → 改 `config.yaml`（如开启/关闭某个平台、调整推送模式） > - **关注内容** → 改 `frequency_words.txt`（如添加新的关注关键词） > - **AI 输出风格** → 改 `ai_analysis_prompt.txt` 或 `ai_translation_prompt.txt` > - **密钥与凭证** → 改 `docker/.env`（API Key、Webhook URL 等敏感信息统一放这里） > - **个性化扩展** → 使用 `config/custom/` 目录，避免直接修改默认配置被升级覆盖 > 💡 **配置修改生效**：修改 `config.yaml` 后，执行 `docker compose up -d` 重启容器即可生效 **⚙️ 环境变量覆盖机制（v3.0.5+）** `.env` 文件中的环境变量会覆盖 `config.yaml` 中的对应配置： | 环境变量 | 对应配置 | 示例值 | 说明 | |---------|---------|-------|------| | `ENABLE_WEBSERVER` | - | `true` / `false` | 是否自动启动 Web 服务器 | | `WEBSERVER_PORT` | - | `8080` | Web 服务器端口 | | `WEBSERVER_WATCHDOG` | - | `true` / `false` | 是否开启“网页服务自动恢复”（服务异常时自动重开） | | `WEBSERVER_WATCHDOG_INTERVAL` | - | `60` | 自动恢复检查间隔（秒） | | `FEISHU_WEBHOOK_URL` | `notification.channels.feishu.webhook_url` | `https://...` | 飞书 Webhook（多账号用 `;` 分隔） | | `AI_ANALYSIS_ENABLED` | `ai_analysis.enabled` | `true` / `false` | 是否启用 AI 分析（v5.0.0 新增） | | `AI_API_KEY` | `ai.api_key` | `sk-xxx...` | AI API Key（ai_analysis 和 ai_translation 共享） | | `AI_PROVIDER` | `ai.provider` | `deepseek` / `openai` / `gemini` | AI 提供商 | | `S3_*` | `storage.remote.*` | - | 远程存储配置（5 个参数） | **配置优先级**：环境变量 > config.yaml **使用方法**： - 修改 `.env` 文件，填写需要的配置 - 或在 NAS/群晖 Docker 管理界面的"环境变量"中直接添加 - 重启容器后生效：`docker compose up -d` 3. **启动服务**: **选项 A：启动所有服务（推送 + AI 分析）** ```bash # 拉取最新镜像 docker compose pull # 启动所有服务（trendradar + trendradar-mcp） docker compose up -d ``` **选项 B：仅启动新闻推送服务** ```bash # 只启动 trendradar（定时抓取和推送） docker compose pull trendradar docker compose up -d trendradar ``` **选项 C：仅启动 MCP AI 分析服务** ```bash # 只启动 trendradar-mcp（提供 AI 分析接口） docker compose pull trendradar-mcp docker compose up -d trendradar-mcp ``` > 💡 **提示**： > - 大多数用户只需启动 `trendradar` 即可实现新闻推送功能 > - 只有需要使用 ChatGPT/Gemini 进行 AI 对话分析时，才需启动 `trendradar-mcp` > - 两个服务相互独立，可根据需求灵活组合 4. **查看运行状态**: ```bash # 查看新闻推送服务日志 docker logs -f trendradar # 查看 MCP AI 分析服务日志 docker logs -f trendradar-mcp # 查看所有容器状态 docker ps | grep trendradar # 停止特定服务 docker compose stop trendradar # 停止推送服务 docker compose stop trendradar-mcp # 停止 MCP 服务 ``` #### 方式二：本地构建（开发者选项） 如果需要自定义修改代码或构建自己的镜像： ```bash # 克隆项目 git clone https://github.com/sansan0/TrendRadar.git cd TrendRadar # 修改配置文件 vim config/config.yaml vim config/frequency_words.txt # 使用构建版本的 docker compose cd docker cp docker-compose-build.yml docker-compose.yml ``` **构建并启动服务**： ```bash # 选项 A：构建并启动所有服务 docker compose build docker compose up -d # 选项 B：仅构建并启动新闻推送服务 docker compose build trendradar docker compose up -d trendradar # 选项 C：仅构建并启动 MCP AI 分析服务 docker compose build trendradar-mcp docker compose up -d trendradar-mcp ``` > 💡 **架构参数说明**： > - 默认构建 `amd64` 架构镜像（适用于大多数 x86_64 服务器） > - 如需构建 `arm64` 架构（Apple Silicon、树莓派等），设置环境变量： > ```bash > export DOCKER_ARCH=arm64 > docker compose build > ``` #### 镜像更新 ```bash # 方式一：手动更新（爬虫 + MCP 镜像） docker pull wantcat/trendradar:latest docker pull wantcat/trendradar-mcp:latest docker compose down docker compose up -d # 方式二：使用 docker compose 更新 docker compose pull docker compose up -d ``` **可用镜像**： | 镜像名称 | 用途 | 说明 | |---------|------|------| | `wantcat/trendradar` | 新闻推送服务 | 定时抓取新闻、推送通知 | | `wantcat/trendradar-mcp` | MCP 服务 | AI 分析功能（可选） | #### 服务管理命令 ```bash # 查看运行状态 docker exec -it trendradar python manage.py status # 手动执行一次爬虫 docker exec -it trendradar python manage.py run # 查看实时日志 docker exec -it trendradar python manage.py logs # 显示当前配置 docker exec -it trendradar python manage.py config # 显示输出文件 docker exec -it trendradar python manage.py files # Web 服务器管理（用于浏览器访问生成的报告） docker exec -it trendradar python manage.py start_webserver # 启动 Web 服务器 docker exec -it trendradar python manage.py stop_webserver # 停止 Web 服务器 docker exec -it trendradar python manage.py webserver_status # 查看 Web 服务器状态 # 查看帮助信息 docker exec -it trendradar python manage.py help # 重启容器 docker restart trendradar # 停止容器 docker stop trendradar # 删除容器（保留数据） docker rm trendradar ``` > 💡 **Web 服务器说明**： > - 启动后可通过浏览器访问 `http://localhost:8080` 查看最新报告 > - 通过目录导航访问历史报告（如：`http://localhost:8080/2025-xx-xx/`） > - 端口可在 `.env` 文件中配置 `WEBSERVER_PORT` 参数 > - 自动启动：在 `.env` 中设置 `ENABLE_WEBSERVER=true` > - 自动恢复：`WEBSERVER_WATCHDOG=true`（默认开启），每隔 `WEBSERVER_WATCHDOG_INTERVAL` 秒检查一次，异常会自动重开网页服务 > - `stop_webserver` 的意思是“你主动手动关闭网页服务”（命令：`docker exec -it trendradar python manage.py stop_webserver`） > - “自动拉起”就是“系统自动把网页服务重新打开”；若你手动关闭后想恢复，请执行 `docker exec -it trendradar python manage.py start_webserver` > - 安全提示：仅提供静态文件访问，限制在 output 目录，只绑定本地访问 #### 数据持久化 生成的报告和数据默认保存在 `./output` 目录下，即使容器重启或删除，数据也会保留。 **📊 网页版报告访问路径**： TrendRadar 生成的当日汇总 HTML 报告会同时保存到两个位置： | 文件位置 | 访问方式 | 适用场景 | |---------|---------|---------| | `output/index.html` | 宿主机直接访问 | **Docker 部署**（通过 Volume 挂载，宿主机可见） | | `index.html` | 根目录访问 | **GitHub Pages**（仓库根目录，Pages 自动识别） | | `output/html/YYYY-MM-DD/当日汇总.html` | 历史报告访问 | 所有环境（按日期归档） | **本地访问示例**： ```bash # 方式 1：通过 Web 服务器访问（推荐，Docker 环境） # 1. 启动 Web 服务器 docker exec -it trendradar python manage.py start_webserver # 2. 在浏览器访问 http://localhost:8080 # 访问最新报告（默认 index.html） http://localhost:8080/html/2025-xx-xx/ # 访问指定日期的报告 # 方式 2：直接打开文件（本地环境） open ./output/index.html # macOS start ./output/index.html # Windows xdg-open ./output/index.html # Linux # 方式 3：访问历史归档 open ./output/html/2025-xx-xx/当日汇总.html ``` **为什么有两个 index.html？** - `output/index.html`：Docker Volume 挂载到宿主机，本地可直接打开 - `index.html`：GitHub Actions 推送到仓库，GitHub Pages 自动部署 > 💡 **提示**：两个文件内容完全相同，选择任意一个访问即可。 #### 故障排查 ```bash # 检查容器状态 docker inspect trendradar # 查看容器日志 docker logs --tail 100 trendradar # 进入容器调试 docker exec -it trendradar /bin/bash # 验证配置文件 docker exec -it trendradar ls -la /app/config/ ``` #### MCP 服务部署（AI 分析功能） 如果需要使用 AI 分析功能，可以部署独立的 MCP 服务容器。 **架构说明**： ```mermaid flowchart TB subgraph trendradar["trendradar"] A1[定时抓取新闻] A2[推送通知] end subgraph trendradar-mcp["trendradar-mcp"] B1[127.0.0.1:3333] B2[AI 分析接口] end subgraph shared["共享卷"] C1["config/ (ro)"] C2["output/ (ro)"] end trendradar --> shared trendradar-mcp --> shared ``` **快速启动**： 如果已按照 [方式一：使用 docker compose](#方式一使用-docker-compose推荐) 完成部署，只需启动 MCP 服务： ```bash cd TrendRadar/docker docker compose up -d trendradar-mcp # 查看运行状态 docker ps | grep trendradar-mcp ``` **单独启动 MCP 服务**（不使用 docker compose）： ```bash # Linux/Mac docker run -d --name trendradar-mcp \ -p 127.0.0.1:3333:3333 \ -v $(pwd)/config:/app/config:ro \ -v $(pwd)/output:/app/output:ro \ -e TZ=Asia/Shanghai \ wantcat/trendradar-mcp:latest # Windows PowerShell docker run -d --name trendradar-mcp ` -p 127.0.0.1:3333:3333 ` -v ${PWD}/config:/app/config:ro ` -v ${PWD}/output:/app/output:ro ` -e TZ=Asia/Shanghai ` wantcat/trendradar-mcp:latest ``` > ⚠️ **注意**：单独运行时，确保当前目录下有 `config/` 和 `output/` 文件夹，且包含配置文件和新闻数据。 **验证服务**： ```bash # 检查 MCP 服务健康状态 curl http://127.0.0.1:3333/mcp # 查看 MCP 服务日志 docker logs -f trendradar-mcp ``` **在 AI 客户端中配置**： MCP 服务启动后，根据不同客户端进行配置： **Cherry Studio**（推荐，GUI 配置）： - 设置 → MCP 服务器 → 添加 - 类型：`streamableHttp` - URL：`http://127.0.0.1:3333/mcp` **Claude Desktop / Cline**（JSON 配置）： ```json { "mcpServers": { "trendradar": { "url": "http://127.0.0.1:3333/mcp", "type": "streamableHttp" } } } ``` > 💡 **提示**：MCP 服务仅监听本地端口（127.0.0.1），确保安全性。如需远程访问，请自行配置反向代理和认证。
### 7. 推送内容怎么显示？
👉 点击展开：自定义推送样式和内容
**配置位置：** `config/config.yaml` 的 `report` 和 `display` 部分 ```yaml report: mode: "daily" # 推送模式 display_mode: "keyword" # 显示模式（v4.6.0 新增） rank_threshold: 5 # 排名高亮阈值 sort_by_position_first: false # 排序优先级 max_news_per_keyword: 0 # 每个关键词最大显示数量 display: region_order: # 区域显示顺序（v5.2.0 新增） - new_items # 新增热点区域 - hotlist # 热榜区域 - rss # RSS 订阅区域 - standalone # 独立展示区 - ai_analysis # AI 分析区域 ``` #### 常用配置项说明 | 我想调整什么 | 修改哪个参数 | 默认值 | 说明 | |-------------|-------------|-------|------| | **推送模式** | `mode` | `daily` | 决定推送时机和内容，详见 [推送模式详解](#3-推送模式详解) | | **分组方式** | `display_mode` | `keyword` | `keyword`=按关键词分组(如"AI")，`platform`=按平台分组(如"微博") | | **高亮重点** | `rank_threshold` | `5` | 排名在前 5 的新闻会**加粗**显示，一眼看到最火的 | | **排序规则** | `sort_by_position_first` | `false` | `false`=热度高的排前面，`true`=你配置的词排前面 | | **数量限制** | `max_news_per_keyword` | `0` | 每个关键词最多看几条？`0`表示不限制 | | **显示顺序** | `display.region_order` | 见上方配置 | 调整列表顺序即可控制各区域的显示位置 | #### 分组方式对比（display_mode） 你是想看"这个话题下有哪些新闻"，还是"这个平台上有哪些新闻"？ | 模式 | 分组方式 | 标题前缀 | 适用场景 | |------|---------|---------|---------| | `keyword`（默认） | **按关键词聚合** | `[平台名]` | 我关注"AI"，想看各平台关于AI的新闻 | | `platform` | **按平台聚合** | `[关键词]` | 我关注"微博"，想看微博上关于我关注词的新闻 | #### 区域显示顺序（region_order） 通过调整 `display.region_order` 列表的顺序，可以控制推送消息中各区域的显示位置。 **默认顺序**：新增热点 → 热榜 → RSS → 独立展示区 → AI 分析 **自定义示例**：想让 AI 分析放在最前面？ ```yaml display: region_order: - ai_analysis # 移到第一行 - new_items - hotlist - rss - standalone ``` **注意**：区域需同时满足两个条件才会显示： 1. 在 `region_order` 列表中 2. 在 `display.regions` 中对应开关为 `true` #### 区域开关（regions） 通过 `display.regions` 控制各区域是否在推送中显示： ```yaml display: regions: hotlist: true # 热榜区域（关键词匹配的热点新闻） new_items: false # 新增热点区域（含热榜新增 + RSS 新增） rss: true # RSS 订阅区域（关键词匹配的 RSS 内容） standalone: false # 独立展示区（完整热榜/RSS，不受关键词过滤） ai_analysis: true # AI 分析区域 ``` | 区域 | 配置键 | 默认值 | 说明 | |------|--------|-------|------| | **热榜** | `hotlist` | `true` | 按关键词匹配的热点新闻聚合 | | **新增热点** | `new_items` | `false` | 本轮新出现的热点话题（含热榜新增 + RSS 新增）。注：热榜区域中的 🆕 标记不受此开关影响 | | **RSS** | `rss` | `true` | 按关键词匹配的 RSS 订阅内容。关闭后跳过 RSS 分析，但独立展示区中的 RSS 不受影响 | | **独立展示区** | `standalone` | `false` | 指定平台/RSS 的完整内容展示，不受关键词过滤 | | **AI 分析** | `ai_analysis` | `true` | AI 生成的热点分析摘要 | #### 排序优先级（sort_by_position_first） 假设你配置了关键词：1.特斯拉，2.比亚迪。 实际热度：比亚迪(10条)，特斯拉(3条)。 | 配置值 | 排序结果 | 你的想法 | |-------|---------|---------| | `false`（默认） | 比亚迪(10条) → 特斯拉(3条) | "谁火谁排前面" | | `true` | 特斯拉(3条) → 比亚迪(10条) | "我配置的顺序就是优先级，不管它火不火" | #### 独立展示区（standalone） **场景**：有些平台（比如知乎热榜、HackerNews），我想**完整看一遍**，不管有没有匹配我的关键词。 ```yaml display: regions: standalone: true # 推送中展示独立展示区（关闭不影响 AI 分析） standalone: platforms: ["zhihu", "weibo"] # 这些平台的热榜给我完整显示 rss_feeds: ["hacker-news"] # 这些RSS源的内容给我完整显示 max_items: 20 # 最多显示多少条 ``` > 💡 **推送展示与 AI 分析独立控制**：`regions.standalone` 只控制推送中是否显示独立展示区。即使关闭推送展示，只要在 AI 配置中开启 `include_standalone: true`，AI 仍会分析这些平台的完整数据。适合想让 AI 做深度分析、但不想推送消息太长的用户。
### 8. 什么时候给我推送？
👉 点击展开：设置推送时间（调度系统）
**配置位置：** `config/config.yaml` 的 `schedule` 部分 + `config/timeline.yaml` #### 快速上手 只需在 `config.yaml` 中选一个预设模板，不需要编辑 `timeline.yaml`： ```yaml schedule: enabled: true preset: "morning_evening" # 改这里就行 ``` #### 可选预设模板 | 模板名 | 说明 | 推送行为 | |-------|------|---------| | `morning_evening` | 全天增量 + 晚间汇总（推荐） | 全天有新增就推 + 19:00-21:00 晚间当日汇总 | | `always_on` | 全天候监控 | 全天有新增就推送，不划分时间段 | | `office_hours` | 办公时间 | 工作日三段式（到岗速览→午间热点→收工汇总），周末增量自由推 | | `night_owl` | 夜猫子 | 午后速览 + 深夜全天汇总（22:00-01:00 跨午夜） | | `custom` | 完全自定义 | 编辑 `timeline.yaml` 底部的 custom 段 | #### 完全自定义 如果预设模板都不满足需求，可以编辑 `config/timeline.yaml` 底部的 `custom` 段，自由定义时间段、日计划和周映射。详见 `timeline.yaml` 文件内的注释说明。 #### 重要提示 > ⚠️ **从旧版本升级的用户注意：** > - v6.0.0 移除了旧的 `notification.push_window` 和 `ai_analysis.analysis_window` 配置 > - 请改用新的 `schedule` + `timeline.yaml` 调度系统 > - 旧的"每天推送一次"可用 `morning_evening` 预设替代 > - 旧的"工作时间推送"可用 `office_hours` 预设替代 > ⚠️ **GitHub Actions 用户注意：** > - GitHub Actions 执行时间不稳定，可能有 ±15 分钟的偏差 > - 时间段范围建议至少留足 **2 小时** > - 如果想要精准的定时推送，建议使用 **Docker 部署**在个人服务器上
### 9. 多久运行一次？
👉 点击展开：设置自动运行频率
**配置位置：** `.github/workflows/crawler.yml` 的 `schedule` 部分 ```yaml on: schedule: - cron: "0 * * * *" # 每小时运行一次 ``` #### 怎么修改运行频率？ GitHub Actions 使用一种叫 "Cron" 的时间格式，不需要深入理解，直接复制下面的代码替换即可。 **配置位置：** `.github/workflows/crawler.yml` 文件中的 `schedule` 部分 | 我想要... | 复制这行代码 | 说明 | |-----------|------------|------| | **每小时一次** | `- cron: "0 * * * *"` | **默认配置**，第 0 分钟运行 | | **每 30 分钟** | `- cron: "*/30 * * * *"` | 每隔 30 分钟运行一次 | | **每天早 8 点** | `- cron: "0 0 * * *"` | ⚠️ 写 `0` 是因为 UTC 时间 (0点) = 北京时间 (8点) | | **工作时间每半小时** | `- cron: "*/30 0-14 * * *"` | 对应北京时间 8:00 - 22:00 | | **一日三餐点** | `- cron: "0 0,6,12 * * *"` | 对应北京时间 8:00、14:00、20:00 | #### ⚠️ 两个重要提醒 1. **时差问题**：GitHub 的服务器在国外，用的是 UTC 时间。 - **简单的算术题**：你想设定的北京时间 **减去 8 小时** = 你要填的时间。 - *例子：想让它北京时间 20:00 运行，设置里要填 12:00* 2. **不要太频繁**：建议间隔不要少于 30 分钟。 - GitHub 免费资源有限，跑得太勤可能会被官方限制账号。 - 而且 Actions 启动本身就有几分钟延迟，太精确的控制没有意义。 #### 手把手修改步骤 1. 在你的 GitHub 仓库中，找到 `.github/workflows/crawler.yml` 文件 2. 点击右上角的 ✏️ (Edit) 按钮 3. 找到 `cron: "..."` 那一行，把引号里的内容换成上面的"代码" 4. 点击右上角的绿色 **Commit changes** 按钮保存
### 10. 推送到多个群/设备
👉 点击展开：同时推送给多个接收者 > ### ⚠️ **安全第一** > **不要在 `config.yaml` 里直接写密码/Token！** > 如果你把包含密码的文件上传到 GitHub，全世界都能看到。 > > **正确做法**： > - **GitHub Actions 用户**：去 Settings -> Secrets 里添加 > - **Docker 用户**：写在 `.env` 文件里（这个文件不会被上传） #### 怎么同时推送到多个地方？ 很简单，在配置时用分号 `;` 把多个地址隔开就行了。 **举个例子**： 假设你有两个飞书群，想同时收到推送： - 群1地址：`https://.../webhook/aaa` - 群2地址：`https://.../webhook/bbb` 配置时填写： `https://.../webhook/aaa;https://.../webhook/bbb` #### 支持多账号的平台 | 平台 | 配置方法 | 注意事项 | |------|---------|----------| | **飞书/钉钉/企微** | 用 `;` 分隔多个 Webhook URL | 最简单，直接串起来就行 | | **Bark (iOS)** | 用 `;` 分隔多个 Key URL | 推送到多台 iPhone | | **Telegram** | Token 和 ChatID 都要用 `;` 分隔 | ⚠️ **注意顺序要对应**：
Token1 对应 ChatID1
Token2 对应 ChatID2 | | **ntfy** | Topic 和 Token 都要用 `;` 分隔 | 如果某个Topic不需要Token，留空即可：
`token1;;token3` (中间那个是空的) | #### 常用配置示例 (GitHub Secrets / .env) ```bash # 飞书发给 3 个群 FEISHU_WEBHOOK_URL=https://hook1...;https://hook2...;https://hook3... # 钉钉发给 2 个群 DINGTALK_WEBHOOK_URL=https://oapi...;https://oapi... # Telegram 发给 2 个人 (注意一一对应) TELEGRAM_BOT_TOKEN=tokenA;tokenB TELEGRAM_CHAT_ID=userA;userB ``` > **提示**：为了防止滥用，默认限制每个平台最多推送到 3 个账号。如果需要更多，可以修改 `MAX_ACCOUNTS_PER_CHANNEL` 配置。
### 11. 数据保存在哪里？
👉 点击展开：选择数据存储位置
#### 数据会存在哪里？ 系统会自动帮你选择最合适的地方，你通常不需要操心： | 你的运行环境 | 数据存在哪 | 说明 | |-------------|-----------|------| | **Docker / 本地运行** | **本地硬盘** | 存在项目目录下的 `output/` 文件夹里，随时可以查看。 | | **GitHub Actions** | **云端存储** | 因为 GitHub Actions 运行完就会销毁环境，所以必须配置云存储（例如 Cloudflare R2）。 | #### 怎么配置云存储？(GitHub Actions 用户必看) 如果你是用 GitHub Actions 运行，你需要一个"云端硬盘"来存数据。例如使用 Cloudflare R2（因为有免费额度）。 **在 GitHub Secrets 里添加这 5 个变量：** | 变量名 | 填什么 | |-------|-------| | `STORAGE_BACKEND` | `remote` | | `S3_BUCKET_NAME` | 你的存储桶名字 | | `S3_ACCESS_KEY_ID` | 你的 Access Key | | `S3_SECRET_ACCESS_KEY` | 你的 Secret Key | | `S3_ENDPOINT_URL` | 你的 R2 接口地址 | > 💡 **详细教程**：怎么申请 R2？请看 [快速开始 - 远程存储配置](#-快速开始) #### 数据会保存多久？ 默认情况下，我们不会自动删除你的数据。但如果你觉得数据太多占空间，可以设置"自动清理"。 **配置位置**：`config/config.yaml` ```yaml storage: local: retention_days: 30 # 本地数据只保留 30 天 (0 表示永久) remote: retention_days: 30 # 云端数据只保留 30 天 ``` #### 推送时间不对？(时区设置) 如果你身在海外，或者发现推送时间跟你的本地时间对不上，可以修改时区。 **配置位置**：`config/config.yaml` ```yaml app: timezone: "Asia/Shanghai" # 默认是中国时间 ``` - 比如你在美国洛杉矶，改成：`America/Los_Angeles` - 比如你在英国伦敦，改成：`Europe/London`
### 12. 让 AI 帮我分析热点
👉 点击展开：开启 AI 智能分析功能
#### AI 能帮我做什么？ 开启这个功能后，AI 会像一个专业的分析师，在推送每一批新闻时： 1. **自动阅读**：阅读所有匹配到的热点新闻 2. **深度思考**：分析原本孤立的新闻之间的关联 3. **撰写报告**：在推送消息的末尾，附上一份简短深刻的"洞察报告" **包含内容**：热点趋势总结、舆论风向判断、跨平台关联分析、潜在影响评估等。 #### 怎么开启 AI 分析？ 最简单的方法是通过环境变量配置（推荐 GitHub Secrets 或 .env）。 **必需的配置项**： | 变量名 | 填什么 | 说明 | |-------|-------|------| | `AI_ANALYSIS_ENABLED` | `true` | 开启开关 | | `AI_API_KEY` | `sk-xxxxxx` | 你的 API Key | | `AI_MODEL` | `deepseek/deepseek-chat` | 模型标识（格式：`provider/model`） | **支持的 AI 提供商**（基于 LiteLLM，支持 100+ 提供商）： | 提供商 | AI_MODEL 填什么 | 说明 | |-------|----------------|------| | **DeepSeek** (推荐) | `deepseek/deepseek-chat` | 性价比极高，适合高频分析 | | **OpenAI** | `openai/gpt-4o`
`openai/gpt-4o-mini` | GPT-4o 系列 | | **Google Gemini** | `gemini/gemini-1.5-flash`
`gemini/gemini-1.5-pro` | Gemini 系列 | | **自定义 API** | 任意格式 | 配合 `AI_API_BASE` 使用 | > 💡 **新特性**：现已基于 [LiteLLM](https://github.com/BerriAI/litellm) 统一接口，支持 100+ AI 提供商，配置更简单、错误处理更完善。 **可选配置项**： | 变量名 | 默认值 | 说明 | |-------|-------|------| | `AI_API_BASE` | (自动) | 自定义 API 地址（如 OneAPI、本地模型） | | `AI_TEMPERATURE` | `1.0` | 采样温度（0-2，越高越随机） | | `AI_MAX_TOKENS` | `5000` | 最大生成 token 数 | | `AI_TIMEOUT` | `120` | 请求超时时间（秒） | | `AI_NUM_RETRIES` | `2` | 失败重试次数 | #### 进阶玩法：AI 翻译 如果你关注了国外的 RSS 源（比如 Hacker News），AI 可以帮你把内容翻译成中文推送。 **配置位置**：`config/config.yaml` ```yaml ai_translation: enabled: true # 开启翻译 language: "Chinese" # 翻译成什么语言 (Chinese, English, Japanese...) ``` #### 进阶玩法：自定义 AI "人设" 觉得 AI 说话太官方？你可以修改它的提示词，让它变成你喜欢的风格（比如"毒舌评论员"、"资深投资顾问"）。 - **修改文件**：`config/ai_analysis_prompt.txt` - **修改方法**：直接用记事本打开编辑，告诉 AI 你想要什么样的分析风格。

## ✨ AI 智能分析 TrendRadar v3.0.0 新增了基于 **MCP (Model Context Protocol)** 的 AI 分析功能，让你可以通过自然语言与新闻数据对话，进行深度分析。 ### ⚠️ 使用前必读 **重要提示：AI 功能需要本地新闻数据支持** AI 分析功能**不是**直接查询网络实时数据，而是分析你**本地已积累的新闻数据**（存储在 `output` 文件夹中） #### 使用说明： 1. **项目自带测试数据**：`output` 目录默认包含 **2025-12-21～2025-12-27** 一周的热榜新闻数据，可用于快速体验 AI 功能 2. **查询限制**： - ✅ 只能查询已有日期范围内的数据（12月21-27日，共7天） - ❌ 无法查询实时新闻或未来日期 3. **获取最新数据**： - 测试数据仅供快速体验，**建议自行部署项目**获取实时数据 - 按照 [快速开始](#-快速开始) 部署运行项目 - 等待至少 1 天积累新闻数据后，即可查询最新热点 ### 1. 快速部署 Cherry Studio 提供 GUI 配置界面，5 分钟快速部署，复杂的部分是一键安装的。 **图文部署教程**：现已更新到我的[公众号](#-支持项目)，回复 "mcp" 即可 **详细部署教程**：[README-Cherry-Studio.md](README-Cherry-Studio.md) **部署模式说明**： - **STDIO 模式（推荐）**：一次配置后续无需重复配置，**图文部署教程**中仅以此模式的配置为例。 - **HTTP 模式（备选）**：如果 STDIO 模式配置遇到问题，可使用 HTTP 模式。此模式的配置方式与 STDIO 基本一致，但复制粘贴的内容就一行，不易出错。唯一需要注意的是每次使用前都需要手动启动一下服务。详细请参考 [README-Cherry-Studio.md](README-Cherry-Studio.md) 底部的 HTTP 模式说明。 ### 2. 学习与 AI 对话的姿势 **详细对话教程**：[README-MCP-FAQ.md](README-MCP-FAQ.md) > 💡 **提示**：实际不建议一次性问多个问题。如果你选择的 AI 模型连下图的按顺序调用都无法做到，建议换一个。 mcp 使用效果图
## 🔌 MCP 客户端 TrendRadar MCP 服务支持标准的 Model Context Protocol (MCP) 协议，可以接入各种支持 MCP 的 AI 客户端进行智能分析。 ### 支持的客户端 **注意事项**： - 将 `/path/to/TrendRadar` 替换为你的项目实际路径 - Windows 路径使用双反斜杠：`C:\\Users\\YourName\\TrendRadar` - 保存后记得重启
👉 点击展开：Cursor #### 方式一：HTTP 模式 1. **启动 HTTP 服务**： ```bash # Windows start-http.bat # Mac/Linux ./start-http.sh ``` 2. **配置 Cursor**： **项目级配置**（推荐）： 在项目根目录创建 `.cursor/mcp.json`： ```json { "mcpServers": { "trendradar": { "url": "http://localhost:3333/mcp", "description": "TrendRadar 新闻热点聚合分析" } } } ``` **全局配置**： 在用户目录创建 `~/.cursor/mcp.json`（同样内容） 3. **使用步骤**： - 保存配置文件后重启 Cursor - 在聊天界面的 "Available Tools" 中查看已连接的工具 - 开始使用：`搜索今天的"AI"相关新闻` #### 方式二：STDIO 模式（推荐） 创建 `.cursor/mcp.json`： ```json { "mcpServers": { "trendradar": { "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ] } } } ```
👉 点击展开：VSCode (Cline/Continue) #### Cline 配置 在 Cline 的 MCP 设置中添加： **HTTP 模式**： ```json { "trendradar": { "url": "http://localhost:3333/mcp", "type": "streamableHttp", "autoApprove": [], "disabled": false } } ``` **STDIO 模式**（推荐）： ```json { "trendradar": { "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ], "type": "stdio", "disabled": false } } ``` #### Continue 配置 编辑 `~/.continue/config.json`： ```json { "experimental": { "modelContextProtocolServers": [ { "transport": { "type": "stdio", "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ] } } ] } } ``` **使用示例**： ``` 分析最近7天"特斯拉"的热度变化趋势 生成今天的热点摘要报告 搜索"比特币"相关新闻并分析情感倾向 ```
👉 点击展开：MCP Inspector（调试工具）
MCP Inspector 是官方调试工具，用于测试 MCP 连接： #### 使用步骤 1. **启动 TrendRadar HTTP 服务**： ```bash # Windows start-http.bat # Mac/Linux ./start-http.sh ``` 2. **启动 MCP Inspector**： ```bash npx @modelcontextprotocol/inspector ``` 3. **在浏览器中连接**： - 访问：`http://localhost:3333/mcp` - 测试 "Ping Server" 功能验证连接 - 检查 "List Tools" 是否返回 17 个工具： - 基础查询：get_latest_news, get_news_by_date, get_trending_topics - 智能检索：search_news, find_related_news - 高级分析：analyze_topic_trend, analyze_data_insights, analyze_sentiment, aggregate_news, compare_periods, generate_summary_report - RSS 查询：get_latest_rss, search_rss, get_rss_feeds_status - 系统管理：get_current_config, get_system_status, resolve_date_range
👉 点击展开：其他支持 MCP 的客户端
任何支持 Model Context Protocol 的客户端都可以连接 TrendRadar： #### HTTP 模式 **服务地址**：`http://localhost:3333/mcp` **基本配置模板**： ```json { "name": "trendradar", "url": "http://localhost:3333/mcp", "type": "http", "description": "新闻热点聚合分析" } ``` #### STDIO 模式（推荐） **基本配置模板**： ```json { "name": "trendradar", "command": "uv", "args": [ "--directory", "/path/to/TrendRadar", "run", "python", "-m", "mcp_server.server" ], "type": "stdio" } ``` **注意事项**： - 替换 `/path/to/TrendRadar` 为实际项目路径 - Windows 路径使用反斜杠转义：`C:\\Users\\...` - 确保已完成项目依赖安装（运行过 setup 脚本）
### 常见问题
👉 点击展开：Q1: HTTP 服务无法启动？
**检查步骤**： 1. 确认端口 3333 未被占用： ```bash # Windows netstat -ano | findstr :3333 # Mac/Linux lsof -i :3333 ``` 2. 检查项目依赖是否安装： ```bash # 重新运行安装脚本 # Windows: setup-windows.bat 或者 setup-windows-en.bat # Mac/Linux: ./setup-mac.sh ``` 3. 查看详细错误日志： ```bash uv run python -m mcp_server.server --transport http --port 3333 ``` 4. 尝试自定义端口: ```bash uv run python -m mcp_server.server --transport http --port 33333 ```
👉 点击展开：Q2: 客户端无法连接到 MCP 服务？
**解决方案**： 1. **STDIO 模式**： - 确认 UV 路径正确（运行 `which uv` 或 `where uv`） - 确认项目路径正确且无中文字符 - 查看客户端错误日志 2. **HTTP 模式**： - 确认服务已启动（访问 `http://localhost:3333/mcp`） - 检查防火墙设置 - 尝试使用 127.0.0.1 替代 localhost 3. **通用检查**： - 重启客户端应用 - 查看 MCP 服务日志 - 使用 MCP Inspector 测试连接
👉 点击展开：Q3: 工具调用失败或返回错误？
**可能原因**： 1. **数据不存在**： - 确认已运行过爬虫（有 output 目录数据） - 检查查询日期范围是否有数据 - 查看 output 目录的可用日期 2. **参数错误**： - 检查日期格式：`YYYY-MM-DD` - 确认平台 ID 正确：`zhihu`, `weibo` 等 - 查看工具文档中的参数说明 3. **配置问题**： - 确认 `config/config.yaml` 存在 - 确认 `config/frequency_words.txt` 存在 - 检查配置文件格式是否正确

## 📚 项目相关 > **4 篇文章**： - [可在该文章下方留言，方便项目作者用手机答疑](https://mp.weixin.qq.com/s/KYEPfTPVzZNWFclZh4am_g) - [2个月破 1000 star，我的GitHub项目推广实战经验](https://mp.weixin.qq.com/s/jzn0vLiQFX408opcfpPPxQ) - [github fork 运行本项目的注意事项 ](https://mp.weixin.qq.com/s/C8evK-U7onG1sTTdwdW2zg) - [基于本项目，如何开展公众号或者新闻资讯类文章写作](https://mp.weixin.qq.com/s/8ghyfDAtQZjLrnWTQabYOQ) >**AI 开发**： - 如果你有小众需求，完全可以基于我的项目自行开发，零编程基础的也可以试试 - 我所有的开源项目或多或少都使用了自己写的**AI辅助软件**来提升开发效率，这款工具已开源 - **核心功能**：迅速筛选项目代码喂给AI，你只需要补充个人需求即可 - **项目地址**：https://github.com/sansan0/ai-code-context-helper ### 其余项目 > 📍 毛主席足迹地图 - 交互式动态展示1893-1976年完整轨迹。欢迎诸位同志贡献数据 - https://github.com/sansan0/mao-map > 哔哩哔哩(bilibili)评论区数据可视化分析软件 - https://github.com/sansan0/bilibili-comment-analyzer [![Star History Chart](https://api.star-history.com/svg?repos=sansan0/TrendRadar&type=Date)](https://www.star-history.com/#sansan0/TrendRadar&Date)
## 📄 许可证 GPL-3.0 License ---
[🔝 回到顶部](#trendradar)
================================================ FILE: config/ai_analysis_prompt.txt ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar AI 分析提示词配置 # Version: 2.0.0 # ═══════════════════════════════════════════════════════════════ # # 此文件定义 AI 分析热点新闻时使用的提示词模板 # # 可用变量（在分析时会被替换）： # {language} - 输出语言 (由 ai_analysis.language 配置) # {report_mode} - 当前报告模式 # {report_type} - 报告类型描述 # {current_time} - 当前时间 # {news_count} - 热榜新闻条数 # {rss_count} - RSS 新闻条数 # {keywords} - 匹配的关键词列表 # {platforms} - 数据来源平台列表 # {news_content} - 热榜新闻内容 # {rss_content} - RSS 订阅内容 (需开启 ai_analysis.include_rss) # {standalone_content} - 独立展示区数据 (需开启 ai_analysis.include_standalone) # # ═══════════════════════════════════════════════════════════════ [system] 你是一名高级情报分析师。你的核心能力是从海量、碎片化的公开来源情报（OSINT）中提炼核心逻辑，并识别被大众忽略的弱信号。 ## 核心思维模型 (Mental Models) 1. 见微知著 (Signal Detection)：不要只盯着榜首的大新闻。要善于从"排名第50的冷门技术贴"与"排名第1的热门事件"中找到潜在的因果联系。 2. 交叉验证 (Triangulation)：利用"热榜"（大众情绪）与"RSS"（专家视角）的差异。当两者观点冲突时，通常隐藏着认知套利的机会。 3. 反直觉思考 (Counter-Intuitive)：当全网都在叫好时，寻找风险；当全网都在恐慌时，寻找机会。拒绝平庸的共识。 4. 结构化输出 (MECE)：确保分析维度相互独立且完全穷尽，避免逻辑混乱。 ## 核心原则 1. 直击要害：拒绝"综上所述"、"众所周知"等废话。直接输出结论。 2. 逻辑闭环：不仅描述"发生了什么"，必须解释"为什么发生"以及"未来会怎样"。 3. 去情绪化：可以分析舆论的情绪，但你自己的分析必须冷静、客观、冷血。 4. 辩证思维：识别热点背后的"主要矛盾"（如技术变革vs既得利益），抓住事物发展的关键内因。 ## 数据字段深度解读指南 ### 1. 基础维度 - 来源平台：每一行新闻开头的 [平台名称]（如 [微博]、[知乎]）明确指出了数据来源。请务必注意：后续的排名和轨迹数据仅针对该特定平台的榜单。 - 排名："1"为该平台榜首，数字越小越热。"3-8"表示在该平台排名在第3到第8之间波动。 - 出现次数：次数越多，说明在热榜停留时间越长，热度越持久。 - 时间范围：如"09:30~12:45"，跨度越大说明话题生命力越强。 ### 2. 轨迹量化分析（重要） 数据格式为 排名(时间)→排名(时间)...，例如 1(09:30)→0(10:00)→2(10:30)。 关键定义： - 数值含义：数字代表排名（1为榜首，数字越小越靠前）。0 特指"未上榜"或"脱榜"（即该时间点不在榜单中）。 - 符号含义：→ 代表时间推移。 防幻觉警示（关键）： - 高位横盘 ≠ 急升：如果轨迹是 2(10:00)→2(10:30)→2(11:00)，说明热度持续稳定，绝对不是"急升"或"爆发"。只有排名数值显著减小（如 10→5）才是急升。请务必区分"热度高"和"热度升"。 请重点分析以下模式： - 急升/爆发：排名数值在短时间内大幅减小（如 20→3），代表热度飙升，往往意味着突发重大事件。 - 衰退/僵尸：排名数值持续变大且无反弹（如 10→15→20），代表热度正在自然衰退。 - 回榜/反转：序列中出现 0 后又变为高排名（如 5→0→2），代表话题曾脱榜但因新进展"复活"，通常暗示有新爆料或剧情反转。 ### 3. 跨平台特征（分级标准） - 全网霸屏：5个及以上平台同时上榜。真正的"国民级"话题，无死角覆盖。 - 破圈扩散：3-4个平台同时上榜。话题已突破单一社区壁垒，正在向外蔓延。 - 圈层热点：仅在1-2个平台火爆。属于特定人群的狂欢。 平台调性参考 (Platform DNA)： - 舆论/情绪场：微博（情绪/吃瓜）、抖音/快手（视觉/传播快）、B站（年轻/玩梗） - 理性/专业场：知乎（深度/批判）、雪球（投资/财经）、IT之家/36氪（科技/商业） - 资讯/分发场：今日头条（社会/民生）、百度热搜（综合/搜索量） 分析"平台温差"时，请结合平台调性。例如：某话题在微博火但在知乎冷，可能说明该话题"情绪价值大于逻辑价值"或"缺乏深度讨论点"。 ## 输出格式规范（严格遵守） 你将以 JSON 格式输出分析结果。每个字段的值是纯文本字符串。 换行规则： - 用 \n 表示换行（JSON 字符串内标准换行符） - 段落之间用 \n\n 分隔 结构标签规则（【】仅用于分段）： - 【】仅用于板块内的结构性分段标签，如【宏观主线】、【跨平台共振】 - 标签后只跟冒号或直接换行（×【宏观主线】两大叙事交织：→ ○【宏观主线】：） - 标签前用 \n 与前段分隔 - 【】内只允许固定的分段名称，禁止放入话题名、新闻标题等动态内容 - 同一标签下仅有1条内容时不加序号，2条及以上才使用序号 话题引用规则（「」用于行内引用）： - 提及具体话题、新闻标题、事件名称时，使用「」角引号（×【黄仁勋暴论】→ ○「黄仁勋暴论」） - 「」是行内标记，不触发换行，不加冒号 序号规则： - 列举时用 1. 2. 3. 数字序号 - 每个序号独占一行（前面用 \n 换行） - 序号行内禁止使用【】标签 绝对禁止： - 禁止使用 Markdown（如 **加粗**、## 标题、- 列表） - 禁止使用 emoji 或特殊装饰符号 ## 分析板块说明（6个板块） ### 1. core_trends — 核心热点态势（200字以内） 整合"趋势概述"、"热度走势"、"跨平台关联"。 任务：提炼共性与定性。不仅要识别最火话题，更要尝试寻找不同新闻背后的底层逻辑或共性叙事（如：多条看似无关的新闻共同指向"经济复苏乏力"或"AI应用落地"的大趋势）。 重点：判断热度性质（全网霸屏vs圈层自嗨）以及话题间的潜在关联。 写法：拒绝流水账。用"宏观主线+微观佐证"的结构，将散点信息串联成逻辑链条。一句话开场定性（必须使用"全网霸屏"/"破圈扩散"/"圈层热点"等词汇），然后用【宏观主线】挖掘底层逻辑，【微观领域】用序号列举细分点。 ### 2. sentiment_controversy — 舆论风向争议（100字以内） 任务：绘制情绪光谱。拒绝简单的"褒/贬"二元对立。要识别"舆论断层"（如：专家担忧风险而大众狂欢，或媒体冷处理而民间热议）。 核心：寻找观点冲突点。哪里有争吵，哪里就有价值。识别是"利益之争"（钱包问题）还是"认知之争"（观念问题）。 写法：【情绪光谱】识别"主流声音"与"潜流暗涌"的反差，【核心矛盾】用序号列举冲突点。 ### 3. signals — 异动与弱信号（150字以内） 任务：捕捉时间轴（轨迹）和空间轴（跨平台）上的异常波动。拒绝平铺直叙的单点罗列。 关注维度： - 跨平台共振：某话题在A平台爆发后，是否迅速引发B平台关注？（对应"破圈扩散"） - 平台温差：某话题在微博霸榜但在知乎无人问津（对应"圈层热点"） - 轨迹突变：排名骤升（急升）、死而不僵（僵尸）、反转复活（回榜） 写法：必须结合跨平台特征分析，拒绝只列举单个平台的涨跌。用【标签】分段（不用序号），从【跨平台共振/温差】【轨迹突变】【弱信号捕捉】等维度至少覆盖2点。 ### 4. rss_insights — RSS深度洞察（100字以内） 任务：寻找信息增量。RSS 源通常比大众热榜更垂直、更专业。 策略： - 去重：果断忽略与热榜大众新闻高度雷同的内容 - 互补：挖掘热榜未覆盖的硬核细节（如技术参数、深度行研）或长尾话题 - 前瞻：识别可能尚未引爆但极具价值的早期行业信号 写法：【认知纠偏】专业视角如何修正大众热搜的误区或盲目，【硬核增量】补充热榜缺失的关键技术参数、行业内幕或深度数据。无RSS数据时填"暂无RSS数据"。 ### 5. outlook_strategy — 研判策略建议 任务：预测与推演。不仅总结过去，更要预测未来。 核心： - 后续推演：预测事件的下一阶段（如：是否会反转？监管是否介入？热度是否可持续？） - 行动指南：给出具体、有针对性的建议。严禁使用"建议持续关注"等无意义的正确的废话。 写法：格式为 1. 投资者：xxx 2. 品牌方：xxx 3. 公众：xxx，序号后直接跟角色名加冒号，不使用【】标签。 ### 6. standalone_summaries — 独立展示区概括（每源100字以内） 仅当数据中包含独立展示区数据时返回。对象类型，key 为数据中每个源的 ### 标题方括号内的名称，value 为 100 字以内的概括。有几个源就写几个 key。 核心原则：去重补盲 + 轨迹洞察。 1. 去重：果断忽略前5板块已充分分析的话题，优先提取前5板块未覆盖的独有内容。若某话题虽在前5板块提及但在该平台有独特表现（如排名走势截然不同），可简要补充差异点。 2. 轨迹洞察：若数据中包含轨迹信息，按上述"### 2. 轨迹量化分析"的规则解读排名走势，识别该平台的急升/衰退/回榜等趋势特征。若数据中无轨迹信息，则基于排名和出现次数做简要判断即可。 写法：先用一句话点明该平台当前的整体趋势动向（基于轨迹数据判断），再列举前5板块未提及的重要话题（附带排名走势）。示例："西藏感悟话题从第12急升至榜首，关注度爆发；此外白银交割战争预判（排名11稳定）、老君山45万年终奖（3→7缓降）值得留意"。禁止空泛总结。 [user] 请分析以下热点新闻数据： ## 数据概览 - 报告模式：{report_mode} ({report_type}) - 分析时间：{current_time} - 数据量：{news_count}条热榜 + {rss_count}条RSS - 来源：{platforms} ## 匹配关键词 {keywords} ## 热榜新闻 {news_content} ## RSS 订阅 {rss_content} ## 独立展示区 以下为独立展示的完整热榜/RSS 数据（不受关键词过滤），请按板块说明中 standalone_summaries 的要求处理。 {standalone_content} --- 请基于上述数据撰写分析报告。以 JSON 格式返回，所有字段均为可选（缺少任何字段不会报错）： ```json { "core_trends": "（按上述板块说明写法输出）", "sentiment_controversy": "（按上述板块说明写法输出）", "signals": "（按上述板块说明写法输出）", "rss_insights": "（按上述板块说明写法输出）", "outlook_strategy": "（按上述板块说明写法输出）", "standalone_summaries": {"知乎": "100字概括，优先列前5板块未提及的话题及排名走势", "Hacker News": "100字概括..."} } ``` 要求： - 使用 {language} 输出，语言简练专业 - 6个板块内容不重叠不冗余 - 若某板块无明显内容，可简写"暂无显著异常" ================================================ FILE: config/ai_filter/extract_prompt.txt ================================================ [system] 你是一个兴趣标签提取专家。你的任务是从用户的兴趣描述中提取出结构化的新闻分类标签。 提取规则： 1. 每个标签简洁（2-6个字），同时配一句描述说明该标签涵盖哪些话题和关键词 2. 标签之间尽量不重叠 3. 标签数量控制在 5~20 个，优先保留细分标签，只有语义高度重叠时才合并 4. 描述要具体，包含具体的人名、公司名、产品名等关键词，方便后续分类 5. 返回顺序必须尽量遵循用户兴趣描述中的先后顺序，越靠前代表优先级越高 [user] 用户的兴趣描述如下： {interests_content} 请从中提取出新闻分类标签。 返回严格的 JSON 格式（不要添加任何其他内容）： ```json { "tags": [ {"tag": "标签名", "description": "该标签涵盖的话题、关键词描述"} ] } ``` ================================================ FILE: config/ai_filter/prompt.txt ================================================ [system] 你是一个高效的新闻分类专家。根据给定的标签列表，快速判断每条新闻标题最适合哪个标签。 分类规则： 1. 每条新闻只归入一个最相关的标签（选相关度最高的那个） 2. 不匹配任何标签的新闻不要输出（不要返回空 tags） 3. 给出 0.0-1.0 的相关度分数（1.0=完全相关，0.5=部分相关） 4. 只根据标题判断，不要过度推测 5. 严格遵循用户偏好中的额外过滤要求（如有） 6. 如果两类标签相关度接近，优先选择排序更靠前的标签（前面的标签优先级更高） [user] ## 用户偏好 {interests_content} ## 分类标签 {tags_list} ## 新闻列表（共 {news_count} 条） {news_list} 请对每条新闻进行分类。返回严格的 JSON 数组（不要添加任何其他内容）： ```json [ {"id": 1, "tag_id": 1, "score": 0.9}, {"id": 5, "tag_id": 2, "score": 0.8} ] ``` 只返回有匹配的新闻，无匹配的不要包含在结果中。 ================================================ FILE: config/ai_filter/update_tags_prompt.txt ================================================ [system] 你是一个标签管理专家。用户修改了兴趣描述后，你需要对比旧标签集和新的兴趣描述，给出标签更新方案。 核心原则： 1. 语义等价的标签视为同一个标签（如"AI/大模型"和"AI与大模型"是同一个标签），优先保留旧标签名 2. 只有用户明确不再关注的方向才标记移除 3. 新增的兴趣方向才需要新增标签 4. 标签名简洁（2-10个字），描述要具体，包含关键词、人名、公司名、产品名 5. 标签总数控制在 20 个以内，优先保留细分标签，只有语义高度重叠时再合并 6. keep 和 add 的输出顺序应尽量遵循用户兴趣描述中的先后顺序（越靠前优先级越高） change_ratio 评估标准： - 0.0 = 兴趣几乎没变（只是措辞调整、补充细节） - 0.1~0.3 = 小幅调整（新增或移除了 1-2 个方向） - 0.4~0.6 = 中等变化（多个方向有调整） - 0.7~1.0 = 大幅改变（兴趣方向基本重写） [user] ## 当前标签集 {old_tags_json} ## 新的兴趣描述 {interests_content} ## 任务 对比当前标签集和新的兴趣描述，判断每个旧标签是保留还是移除，以及是否需要新增标签。 返回严格的 JSON 格式（不要添加任何其他内容）： ```json { "keep": [ {"tag": "旧标签名", "description": "根据新兴趣更新后的描述"} ], "add": [ {"tag": "新标签名", "description": "该标签涵盖的话题、关键词描述"} ], "remove": ["要废弃的旧标签名"], "change_ratio": 0.2 } ``` ================================================ FILE: config/ai_interests.txt ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar AI 兴趣描述文件 # Version: 1.1.0 # ═══════════════════════════════════════════════════════════════ # 用自然语言描述你关注的话题，AI 会自动提取标签并对新闻进行分类 # 修改此文件后，下次运行时自动生效（旧分类会被标记废弃，重新分类） 下面是我要关注的内容： # 重要性排序说明：从上到下优先级递减，越靠前越重要。 # 如果一条新闻同时可能匹配多个方向，请优先归入更靠前的方向。 1. 中国科技与互联网公司：重点关注 DeepSeek、华为、腾讯、字节跳动、京东及相关核心人物和业务线（含鸿蒙、海思、昇腾、抖音、微信等）的战略、组织调整、产品节奏、资本动作与监管影响。 2. 大模型与 AI 产品：关注 OpenAI、Claude、ChatGPT、Sora、DALL-E、Qwen、MiniMax、GLM 等模型和产品的能力演进、开源闭源策略与生态竞争。 3. AI 基础设施与云算力：关注英伟达、AMD、华为算力体系、CUDA、Azure、Google Cloud 相关的算力供给、推理成本、训练效率与供应链变化。 4. 芯片与半导体制造：关注芯片、半导体、光刻机、先进封装、国产替代、关键材料设备与供应安全。 5. 智能汽车与自动驾驶：关注比亚迪、特斯拉、FSD、无人驾驶、智驾、刀片电池、云辇等技术路线，以及销量、定价与出海变化。 6. 机器人与具身智能：关注宇树、智元、众擎、大疆在机器人、机械狗、四足、人形、具身智能方向的产品发布、量产和场景落地。 7. 全球科技巨头：关注苹果、微软、谷歌、Anthropic、OpenAI 的财报、发布会、产品路线、合作与竞争格局。 8. 地缘政治与国际关系（独立于金融）：重点关注中美欧日印及俄罗斯相关的关税、制裁、外交、冲突、产业脱钩和关键供应链博弈。 9. 金融市场与宏观政策：关注美联储利率路径、汇率波动、通胀、就业、股债商品表现及全球流动性变化。 10. 能源与电力系统：关注光伏、太阳能、水电（含雅鲁藏布江项目）、核能和新型电力系统建设。 11. 航天与深空探索：关注 SpaceX、登月、火星、飞船、卫星、空间站、商业航天的技术节点与产业化进展。 12. 前沿科学技术：关注量子、脑机接口、基因工程等前沿方向的重要科研突破与产业应用。 13. 文化 IP 与内容产业：关注黑神话悟空、影之刃零、三体、流浪地球、申奥相关内容，以及游戏工业化和文化出海。 14. 零售与消费品牌：关注胖东来等零售标杆在组织效率、供应链管理、门店运营和消费趋势方面的信号。 15. 国家与区域观察：关注中国、美国、加拿大、日本、韩国、朝鲜、英国、法国、印度、俄罗斯相关的政策、科技、产业与社会议题（作为背景参考，不高于上述核心方向）。 # 标题质量要求（即使匹配了上面的标签，符合以下特征的标题也请跳过） # 可自由增删改，按你的偏好来 - 不要标题党/震惊体（如"震惊！"、"太可怕了！"、"竟然..."、"刚刚！"） - 不要营销软文、广告推广类标题 ================================================ FILE: config/ai_translation_prompt.txt ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar AI 翻译提示词配置 # Version: 1.2.0 # ═══════════════════════════════════════════════════════════════ # # 此文件定义 AI 翻译内容时使用的提示词模板 # # 可用变量： # {target_language} - 目标语言 # {content} - 需要翻译的文本内容 # # ═══════════════════════════════════════════════════════════════ [system] 你是一位精通多语言的专业翻译助手。你的任务是将新闻内容翻译成目标语言，保持新闻的专业性、准确性和简洁性。 要求： 1. 准确传达原文含义，不要遗漏关键信息。 2. 保持新闻标题的吸引力，但不要做标题党。 3. 专有名词（人名、地名、机构名）若有通用译名请使用通用译名，否则保留原文或在括号内备注。 4. 输出格式必须严格遵循要求，不要输出任何多余的解释性文字。 5. ⚠️重点：输入可能包含混合语言列表。请务必逐行检查每一条内容。如果某条内容不是 {target_language}，**必须**将其翻译为 {target_language}。严禁保留非 {target_language} 的原文（除非是纯专有名词）。即使列表中 99% 已经是目标语言，也绝对不能忽略剩下的 1%。 6. 格式严格限制：输出结果中**只允许包含目标语言**的文本。绝对禁止“原文 + 译文”的形式。如果进行了翻译，直接用译文替换原文，不要在后面括号备注原文，也不要保留原文。 [user] 请将以下内容翻译成 {target_language}： {content} 请直接输出翻译结果。 ================================================ FILE: config/config.yaml ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar 配置文件 # Version: 2.2.0 # ═══════════════════════════════════════════════════════════════ # 可视化配置编辑器地址: https://sansan0.github.io/TrendRadar/ # =============================================================== # 1. 基础设置 # =============================================================== app: # 时区配置（影响所有时间显示、调度系统判断、数据存储） # 常用时区： # - Asia/Shanghai (北京时间 UTC+8) # - America/New_York (美东时间 UTC-5/-4) # - Europe/London (伦敦时间 UTC+0/+1) # 完整时区列表: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones timezone: "Asia/Shanghai" show_version_update: true # 显示版本更新提示 # =============================================================== # 1.5 调度系统 —— 什么时间做什么事 # # 通过 timeline.yaml 里定义的时间段来自动决定： # - 什么时候推送通知 # - 什么时候做 AI 分析 # - 用什么报告模式 # # 快速上手：选一个预设模板，改 preset 的值就行 # # always_on → 全天候，有新增即推送 # morning_evening → 全天推送 + 晚间当日汇总（推荐） # office_hours → 工作日三段式（到岗→午间→收工），周末增量自由推 # night_owl → 午后速览 + 深夜全天汇总 # custom → 完全自定义，详见 timeline.yaml # # 详细时间线图请查看 config/timeline.yaml # =============================================================== schedule: enabled: true # 是否启用调度系统 preset: "morning_evening" # 预设模板名称（见上方说明） # =============================================================== # 2. 数据源 - 热榜平台 # # enabled: 是否启用热榜抓取（总开关） # sources: 平台列表 # - id: 平台唯一标识（勿修改） # - name: 显示名称（可自定义，修改后不影响运行） # =============================================================== platforms: enabled: true # 是否启用热榜平台抓取 sources: - id: "toutiao" name: "今日头条" - id: "baidu" name: "百度热搜" - id: "wallstreetcn-hot" name: "华尔街见闻" - id: "thepaper" name: "澎湃新闻" - id: "bilibili-hot-search" name: "bilibili 热搜" - id: "cls-hot" name: "财联社热门" - id: "ifeng" name: "凤凰网" - id: "tieba" name: "贴吧" - id: "weibo" name: "微博" - id: "douyin" name: "抖音" - id: "zhihu" name: "知乎" # =============================================================== # 3. 数据源 - RSS 订阅 # # 与热榜数据分开存储，按时间流展示 # 每个源配置：id(唯一标识)、name(显示名称)、url(订阅地址) # enabled: 可选，默认 true # max_age_days: 可选，覆盖全局 freshness_filter.max_age_days # =============================================================== rss: enabled: true # 是否启用 RSS 抓取 # 文章新鲜度过滤配置（全局默认值） # 过滤掉发布时间超过指定天数的旧文章，避免同一篇文章重复出现在推送中 # # 过滤逻辑： # - 文章发布时间距当前时间（app.timezone 时区）超过 N 天则不推送 # - 无发布时间的文章会被保留（不过滤） # # ⚠️ 过滤时机：在推送阶段过滤 # - 所有文章都会存入数据库（MCP Server 的 AI 查询仍可访问） # - 只有新鲜的文章会被推送到通知渠道 freshness_filter: enabled: true # 是否启用新鲜度过滤（默认启用） max_age_days: 1 # 最大文章年龄（天） # - 正整数：只推送 N 天内的文章 # - 0：禁用过滤，推送所有文章 # 单个 feed 可配置 max_age_days 覆盖全局设置： # - 不配置：使用全局 freshness_filter.max_age_days（默认 3 天） # - 正整数：覆盖全局设置，只推送此天数内的文章 # - 0：禁用此频道的新鲜度过滤，推送所有文章 feeds: - id: "hacker-news" name: "Hacker News" url: "https://hnrss.org/frontpage" - id: "ruanyifeng" name: "阮一峰的网络日志" url: "http://www.ruanyifeng.com/blog/atom.xml" enabled: false # 禁用 # max_age_days: 3 # 示例：推送 3 天内的文章（更新较慢的博客） - id: "yahoo-finance" name: "雅虎财经" url: "https://finance.yahoo.com/news/rssindex" # 自定义源示例 # - id: "custom-feed" # name: "自定义源" # url: "https://example.com/feed.xml" # enabled: false # max_age_days: 0 # 示例：禁用过滤，推送所有文章 # =============================================================== # 4. 报告模式 # # 新手 5 行： # 1) 先选 mode：daily(当日汇总) / current(当前榜单) / incremental(仅新增) # 2) 再选 display_mode：keyword(按词/标签) / platform(按平台) # 3) 如果你开了 schedule，这里的 mode 只是默认值，会被 timeline 时段覆盖 # 4) sort_by_position_first 只影响 keyword 模式排序 # 5) rank_threshold 和 max_news_per_keyword 只影响展示，不影响抓取 # # 进阶说明： # - daily：信息最全，但重复最多 # - current：适合盯当前热度 # - incremental：最少打扰，只看新增 # =============================================================== report: mode: "current" # daily | current | incremental（schedule 开启时作为默认值） display_mode: "keyword" # 分组维度: keyword | platform # keyword: 按关键词分组显示（默认） # platform: 按平台/来源分组显示 # 关键词模式分组排序方式（仅 keyword 模式生效） # true: 按 frequency_words.txt 的定义顺序排列 # false: 按匹配到的热点条数排序（条数多的在前） sort_by_position_first: false rank_threshold: 5 # 排名高亮阈值（影响展示强调，不改变抓取范围） max_news_per_keyword: 0 # 每个关键词/标签最大显示数量（0=不限制，仅影响展示裁剪） # =============================================================== # 4.5 筛选策略 # # 新手 5 行： # 1) 先选 method：keyword（关键词）或 ai（兴趣分类） # 2) keyword 模式：看 config/frequency_words.txt # 3) ai 模式：看 config/ai_interests.txt + 下方 ai_filter 配置 # 4) priority_sort_enabled 只影响 ai 模式标签排序 # 5) 这里决定“筛选路径”，不决定 AI 模型（模型在 ai 段） # =============================================================== filter: method: "ai" # 可选: keyword | ai # AI 模式标签排序开关（仅 ai 模式生效） # true: 按标签优先级排序（来自兴趣描述提取顺序） # false: 按匹配条数排序（条数多的在前） priority_sort_enabled: true # =============================================================== # 4.6 AI 智能筛选配置（当 filter.method=ai 时生效） # # 新手 5 行： # 1) 先调 min_score（推荐 0.5~0.7） # 2) 再调 reclassify_threshold（大改兴趣建议更低） # 3) 批量参数只影响速度/限流，不影响分类逻辑 # 4) interests_file 不填就用 config/ai_interests.txt # 5) prompt_file 系列属于进阶项，默认一般不用改 # # 进阶说明： # - min_score 越高，结果越“准”但会漏召回 # - reclassify_threshold 越低，越倾向全量重分类（更耗 token） # - 模型配置统一在下方 ai 段 # =============================================================== ai_filter: batch_size: 200 # 每批发送给 AI 的标题数（控制单次 API 调用量） # 新闻超过此数量时自动分批处理 batch_interval: 2 # 分批处理时，每批之间的等待时间（秒） # 避免频繁调用 API 触发限流，设为 0 则不等待 min_score: 0.7 # 推送最低分数阈值（0.0 ~ 1.0） # 0 = 不过滤；值越高越严格（推荐先用 0.5~0.7） # 兴趣描述文件 # 默认使用 config/ai_interests.txt，无需在此配置 # 这里设置的是“全局默认”，可被 timeline.yaml 时段内的 interests_file 覆盖 # 如需使用自定义文件，将文件放入 config/custom/ai/ 目录，然后指定文件名： # interests_file: "finance.txt" # → 加载 config/custom/ai/finance.txt # 全量重分类触发阈值（0~1） # change_ratio >= 此值：全量重分类；否则增量更新 # 0.0 最准确最费；1.0 最省但可能陈旧；0.6 是平衡点 reclassify_threshold: 0.6 # 以下提示词模板一般无需修改（不建议动） # 分类提示词模板 prompt_file: "prompt.txt" # 标签提取提示词模板（首次运行时使用） extract_prompt_file: "extract_prompt.txt" # 标签更新提示词模板（兴趣变更时 AI 对比新旧标签） update_tags_prompt_file: "update_tags_prompt.txt" # =============================================================== # 5. 推送内容控制 # # 统一管理推送消息中显示哪些区域及其排列顺序 # =============================================================== display: # 📋 区域显示顺序 # 列表从上到下的顺序 = 推送消息中从上到下的显示顺序 # 想调整顺序？直接剪切粘贴整行即可，例如把 ai_analysis 移到最前面： # region_order: # - ai_analysis ← 移到第一行，AI 分析就会显示在最顶部 # - new_items # - hotlist # - ... # 注意：区域需同时满足两个条件才会显示： # 1. 在此列表中 # 2. 下方 regions 中对应开关为 true region_order: - new_items # 1️⃣ 新增热点区域 - hotlist # 2️⃣ 热榜区域（关键词匹配 / AI 智能筛选） - rss # 3️⃣ RSS 订阅区域 - standalone # 4️⃣ 独立展示区 - ai_analysis # 5️⃣ AI 分析区域 # 推送区域开关 # 控制各区域是否启用（配合 region_order 使用） regions: hotlist: true # 热榜区域（关键词匹配 / AI 智能筛选） new_items: false # 新增热点区域（含热榜新增 + RSS 新增） # 注：热点词汇统计中的新增标记🆕不受此配置影响 rss: true # RSS 订阅区域 # 开启后将对 RSS 进行关键词分析并在通知中展示 # 关闭后跳过分析，但独立展示区不受影响 standalone: false # 独立展示区（完整热榜/RSS，不受关键词过滤） ai_analysis: true # AI 分析区域 # 📋 独立展示区配置 # 用途：将指定平台的完整热榜/RSS 数据独立提取，不受关键词过滤影响 # 两个独立用途： # - 推送展示：由 regions.standalone 开关控制，在推送中单独展示完整热榜 # - AI 分析：由 ai_analysis.include_standalone 开关控制，将完整数据送入 AI 做深度分析 # 两者共享此处的平台/RSS 配置，但开关互相独立（可只开 AI 分析、不推送展示） standalone: platforms: ["zhihu", "wallstreetcn-hot"] # 热榜平台 ID 列表（如 ["zhihu", "weibo"]） rss_feeds: [] # RSS 源 ID 列表（如 ["hacker-news"]） max_items: 20 # 每个源最多展示条数（0=不限制） # =============================================================== # 6. 推送通知 # # ⚠️ 重要安全警告 ⚠️ # # 🔴 请务必妥善保管好 webhooks，不要公开!!! # 🔴 如果你以 fork 的方式部署在 GitHub 上，请勿在此填写 # 🔴 而是将 webhooks 填入 GitHub Secrets # (Settings → Secrets and variables → Actions) # 🔴 否则： # - 轻则：手机上收到大量垃圾广告推送 # - 重则：webhook 被滥用造成严重安全隐患 # # 📌 多账号支持说明 # # • 使用分号(;)分隔多个账号，如："url1;url2;url3" # • 需要配对的配置（如 Telegram 的 token 和 chat_id）数量必须一致 # • 每个渠道最多支持 max_accounts_per_channel 个账号 # • 邮箱已支持多收件人（逗号分隔） # # 新手建议： # • 第一次先只配置 1 个渠道（建议 ntfy 或 telegram）验证通路 # • 跑通后再增加多渠道和多账号，排障成本最低 # =============================================================== notification: enabled: true # 是否启用通知功能（总开关） # ⚠️ 开启调度系统后，此项仍为总开关： # false → 永远不推送（无论调度怎么设置） # true → 由调度的 push 字段控制何时推送 # 推送渠道配置 channels: feishu: webhook_url: "" # 飞书机器人 webhook URL dingtalk: webhook_url: "" # 钉钉机器人 webhook URL wework: webhook_url: "" # 企业微信机器人 webhook URL msg_type: "markdown" # 消息类型：markdown(群机器人) | text(个人微信应用) telegram: bot_token: "" # Telegram Bot Token chat_id: "" # Telegram Chat ID email: from: "" # 发件人邮箱地址 password: "" # 发件人邮箱密码或授权码 to: "" # 收件人邮箱，多个用逗号分隔 smtp_server: "" # SMTP 服务器（可选，留空自动识别） smtp_port: "" # SMTP 端口（可选，留空自动识别） ntfy: server_url: "https://ntfy.sh" # ntfy 服务器地址（可改为自托管） topic: "" # ntfy 主题名称 token: "" # ntfy 访问令牌（可选，用于私有主题） bark: url: "" # Bark 推送 URL（格式：https://api.day.app/your_device_key） slack: webhook_url: "" # Slack Incoming Webhook URL generic_webhook: webhook_url: "" # 通用 Webhook URL（支持 Discord、Matrix、IFTTT 等） payload_template: "" # JSON 模板，支持 {title} 和 {content} 占位符 # 示例：{"content": "{content}"} # 留空则使用默认格式：{"title": "{title}", "content": "{content}"} # =============================================================== # 7. 存储配置 # =============================================================== storage: # 存储后端选择 # - auto: 自动选择（GitHub Actions 且配置了远程存储 → remote，否则 → local） # - local: 本地 SQLite + TXT/HTML 文件 # - remote: 远程云存储（S3 兼容协议，支持 R2/OSS/COS 等） backend: "auto" # 数据格式选项 formats: sqlite: true # 主存储（必须启用） txt: false # 是否生成 TXT 快照 html: true # 是否生成 HTML 报告（⚠️ 邮件推送或者需要看网页版报告必须设为 true） # 本地存储配置 local: data_dir: "output" # 数据目录 retention_days: 0 # 保留天数（0=永久保留） # 远程存储配置（S3 兼容协议） # 支持: Cloudflare R2, 阿里云 OSS, 腾讯云 COS, AWS S3, MinIO 等 # 建议将敏感信息配置在 GitHub Secrets 或环境变量中 remote: retention_days: 0 # 保留天数（0=永久保留） # S3 兼容配置（或使用环境变量 S3_ENDPOINT_URL 等） endpoint_url: "" # 服务端点 # Cloudflare R2: https://.r2.cloudflarestorage.com # 阿里云 OSS: https://oss-cn-hangzhou.aliyuncs.com # 腾讯云 COS: https://cos.ap-guangzhou.myqcloud.com bucket_name: "" # 存储桶名称 access_key_id: "" # 访问密钥 ID secret_access_key: "" # 访问密钥 region: "" # 区域（可选，部分服务商需要） # 数据拉取配置（从远程同步到本地） # 用于 MCP Server 等场景：爬虫存到远程，MCP 拉取到本地分析 pull: enabled: false # 是否启用启动时自动拉取 days: 7 # 拉取最近 N 天的数据 # =============================================================== # 8. AI 模型配置（共享） # # ai_analysis / ai_translation / ai_filter 共用此模型配置 # 基于 LiteLLM 统一接口，支持 100+ AI 提供商 # =============================================================== ai: # LiteLLM 模型格式: provider/model_name # 示例: # - deepseek/deepseek-chat (DeepSeek) # - openai/gpt-4o (OpenAI) # - gemini/gemini-2.5-flash (Google Gemini) # - anthropic/claude-3-5-sonnet (Anthropic) # - ollama/llama3 (本地 Ollama) # 完整列表: https://docs.litellm.ai/docs/providers # 如果你对于看英文文档比较头疼，那么可以点击页面右下角的 【Ask AI】 ,用中文询问怎么配置 model: "deepseek/deepseek-chat" api_key: "" # API Key（建议使用环境变量 AI_API_KEY） api_base: "" # 自定义 API 端点（可选，大多数情况留空） # 示例: https://api.openai.com/v1（自建代理或兼容接口） # # 💡 超级重要：连接任意兼容 OpenAI 协议的模型商 # 如果你使用的模型商不在上述支持列表中，但提供了兼容 OpenAI 的接口： # # 1. api_base 填写: 服务商提供的接口地址 # 例如: https://api.example.com/v1 # # 2. model 填写: "openai/" + 实际模型名称 # 例如: openai/deepseek-ai/DeepSeek-V3 # (原理：前缀 openai/ 强制 LiteLLM 使用 OpenAI 协议格式进行通信) timeout: 120 # 请求超时（秒） temperature: 1.0 # 采样温度 (0.0-2.0) # 注意：部分模型(如 gpt-5)可能要求必须为 1.0，否则会报错 max_tokens: 5000 # 最大生成 token 数 # 注意：如果 API 不支持此参数(报 HTTP 400)，请设为 0 以禁用发送 # 高级选项 num_retries: 1 # 失败重试次数 fallback_models: [] # 备用模型列表（可选） # 示例: ["openai/gpt-4o-mini", "openai/deepseek-ai/DeepSeek-V3"] # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 额外参数 (高级选项，一般无需修改) # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # LiteLLM 会自动将通用参数转换为各提供商格式，无需手动适配。 # 仅在需要传递特殊参数时启用此项。 # # 提示：你可以根据模型 API 文档自行添加任何支持的字段。 # 操作：如需启用，请删掉该行最前方的 "# "（井号和空格）。 # 注意：如果这几行都带着井号，则代表不使用额外参数（推荐做法）。 # ------------------------------------------------------------- # extra_params: # top_p: 1.0 # 核采样（通用） # presence_penalty: 0.0 # 话题多样性（OpenAI/DeepSeek） # stop: ["END"] # 停止词列表（通用） # =============================================================== # 9. AI 分析功能 # # 使用 AI 大模型对推送内容进行深度分析 # 模型配置见上方 ai 配置段 # =============================================================== ai_analysis: enabled: true # 是否启用 AI 分析（总开关） # ⚠️ 开启调度系统后，此项仍为总开关： # false → 永远不分析（无论调度怎么设置） # true → 由调度的 analyze 字段控制何时分析 # 分析报告输出语言 # 格式：自然语言描述 # 示例: "English", "Korean", "法语" language: "Chinese" # 提示词配置文件路径（相对于 config 目录） prompt_file: "ai_analysis_prompt.txt" # AI 分析模式（独立于推送报告模式） # 可选值: # - "follow_report": 跟随 report.mode 的设置（默认） # - "daily": 强制使用当日汇总模式（分析当天所有新闻） # - "current": 强制使用当前榜单模式（只分析当前在榜新闻） # - "incremental": 强制使用增量模式（只分析新增新闻） # # 使用场景： # - 推送 incremental（避免重复），AI 分析 current（看当前榜单变化） # - 推送 current（实时热点），AI 分析 daily（全天总结） # mode: "follow_report" # 分析内容配置 max_news_for_analysis: 150 # 热榜+RSS 合计参与分析的新闻数量上限（控制成本关键项） # 热榜优先占用配额，RSS 使用剩余配额；独立展示区不受此限制 # 推送消息顶部会显示实际的 AI 分析数供参考 # api 成本估算 (仅供参考) # 按默认模型(deepseek) # max_news_for_analysis 为 【50】 条 # include_rank_timeline 为 【false】 # 则 # GitHub Action 部署默认推送约 20 次（每小时推送一次）， 约 0.1 元/天 # Docker 部署默认推送 48 次(每半小时推送一次)， 约 0.2 元/天 include_rss: false # 是否包含 RSS 内容进行分析 include_standalone: true # 是否将独立展示区数据纳入 AI 分析 # 数据源列表来自 display.standalone.platforms / display.standalone.rss_feeds include_rank_timeline: true # 是否传递完整排名时间线 # false: 使用简化格式（排名范围+时间范围+出现次数） # true: 传递完整排名变化轨迹（如 1(09:30)→2(10:00)→0(11:00)） # 启用后 AI 能更精确分析热度趋势，但会额外增加 token 消耗（0.5 倍到 1 倍） # =============================================================== # 10. AI 翻译功能 # # 对推送内容进行多语言翻译，不包含 ai_analysis 分析的内容 # 模型配置见上方 ai 配置段 # =============================================================== ai_translation: enabled: true # 是否启用翻译功能 # 翻译目标语言 # 格式：自然语言描述 # 示例: "Chinese", "Korean", "法语" language: "中文" # 提示词配置文件路径（相对于 config 目录） prompt_file: "ai_translation_prompt.txt" # 翻译范围 # 控制哪些区域的标题会被翻译 # hotlist: 热榜标题 + 新增热点 # rss: RSS 统计 + RSS 新增 # standalone: 独立展示区（热榜平台 + RSS 源） # 如果 display.regions 关闭了显示，那么这边即使开启了也不会翻译 scope: hotlist: false # 热榜区域 rss: true # RSS 区域 standalone: true # 独立展示区 # =============================================================== # 11. 高级设置（一般无需修改） # =============================================================== advanced: # 调试模式 debug: false # 版本检查 version_check_url: "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version" mcp_version_check_url: "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version_mcp" configs_version_check_url: "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version_configs" # 热榜爬虫技术参数 crawler: request_interval: 2000 # 请求间隔（毫秒） use_proxy: false # 是否启用代理 default_proxy: "http://127.0.0.1:10801" # RSS 设置 rss: request_interval: 1000 # 请求间隔（毫秒） timeout: 15 # 请求超时（秒） use_proxy: false # 是否使用代理 proxy_url: "" # RSS 专属代理（留空则使用 crawler.default_proxy） # 排序权重（用于重新排序不同平台的热搜） # 合起来等于 1 weight: rank: 0.6 # 排名权重 frequency: 0.3 # 频次权重 hotness: 0.1 # 热度权重 # 多账号限制 max_accounts_per_channel: 3 # 每个渠道最大账号数量 # 以下为内部参数（一般无需修改） # 消息分批大小（字节）- 内部配置，请勿修改 batch_size: default: 4000 dingtalk: 20000 feishu: 30000 bark: 4000 slack: 4000 batch_send_interval: 3 # 批次发送间隔（秒） feishu_message_separator: "━━━━━━━━━━━━━━━━" ================================================ FILE: config/custom/ai/.gitkeep ================================================ ================================================ FILE: config/custom/keyword/.gitkeep ================================================ ================================================ FILE: config/frequency_words.txt ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar 频率词配置文件 # Version: 1.1.0 # ═══════════════════════════════════════════════════════════════ # 可视化配置编辑器地址: https://sansan0.github.io/TrendRadar/ # # 凡是左侧有 # 的都是仅供阅读的说明性文字 # # 这个文件用来设置你想关注的新闻关键词。 # 系统会自动抓取包含这些关键词的热榜新闻推送给你。 # # 文件分为两个区域： # [GLOBAL_FILTER] - 全局过滤区：排除不想看的内容 # [WORD_GROUPS] - 词组定义区：设置想关注的关键词 # # ═══════════════════════════════════════════════════════════════ # ─────────────────────────────────────────────────────────────── # 全局过滤区 # ─────────────────────────────────────────────────────────────── # 在这里写入你不想看到的词，每行一个。 # 包含这些词的新闻会被自动排除，不会出现在推送中。 # # 使用方法： # 震惊 直接写词，包含"震惊"的新闻会被过滤 # /赌博|博彩/ 用 /.../ 包裹可以匹配多个词（用 | 分隔） [GLOBAL_FILTER] # 过滤标题党 震惊 # ─────────────────────────────────────────────────────────────── # 词组定义区 # ─────────────────────────────────────────────────────────────── # 在这里写入你想关注的关键词。 # 每个词组用空行分隔，同一词组内的关键词是"或"的关系。 # # ┌─────────────────────────────────────────────────────────────┐ # │ 语法总览（快速参考） │ # └─────────────────────────────────────────────────────────────┘ # # 关键词语法： # 关键词 普通关键词，标题包含即匹配 # /正则/ 正则表达式匹配（自动忽略大小写） # 关键词 => 别名 给关键词指定显示别名 # [组别名] 词组第一行，给整组指定别名 # +关键词 必须词，所有必须词都要匹配才算匹配 # !关键词 过滤词，匹配则排除该条新闻（仅限当前词组） # @数字 限制该词组最多显示多少条 # # 显示名称优先级： # 1. 有组别名 → 显示组别名 # 2. 没有组别名 → 显示各行别名拼接（用 " / " 连接） # 3. 没有别名 → 显示关键词本身 # # # ┌─────────────────────────────────────────────────────────────┐ # │ 基础用法（推荐新手） │ # └─────────────────────────────────────────────────────────────┘ # # 1. 最简单：直接写关键词 # ──────────────────── # 华为 # # 效果：匹配所有包含"华为"的新闻 # # # 2. 多个关键词归为一组 # ──────────────────── # 华为 # 鸿蒙 # 任正非 # # 效果：匹配包含"华为"或"鸿蒙"或"任正非"的新闻，统一显示为"华为 / 鸿蒙 / 任正非" # # # 3. 给词组起个名字（推荐） # ──────────────────── # [华为] # 华为 # 鸿蒙 # 任正非 # # 效果：同上，但显示名称为"华为"（更简洁） # # # ┌─────────────────────────────────────────────────────────────┐ # │ 进阶用法（可选） │ # └─────────────────────────────────────────────────────────────┘ # # 4. 用正则表达式匹配多个词（一行搞定） # ──────────────────── # /华为|鸿蒙|任正非/ => 华为 # # 效果：匹配包含"华为"或"鸿蒙"或"任正非"的新闻，显示为"华为" # 说明：/.../ 里用 | 分隔多个词，=> 后面是显示名称 # # 💡 不懂正则？问 AI： # "帮我写一个正则表达式，匹配包含'华为'或'鸿蒙'或'任正非'的文本， # 格式要求：/正则/ => 显示名称" # # # 5. 精确匹配英文单词（避免误匹配） # ──────────────────── # /\bAI\b/i => AI # # 说明：\b 表示单词边界，避免匹配到 "MAIL" 中的 "AI" # /i 表示忽略大小写，"ai"、"AI"、"Ai" 都能匹配 # # 💡 不懂正则？问 AI： # "帮我写一个正则表达式，精确匹配英文单词'AI'（不匹配 MAIL 中的 AI）， # 忽略大小写，格式要求：/正则/i => 显示名称" # # # 6. 排除特定内容 # ──────────────────── # [苹果公司] # 苹果 # !水果 # !果园 # # 效果：匹配"苹果"但排除包含"水果"或"果园"的新闻 # 说明：! 开头的词表示"排除" # # # 7. 限制显示条数 # ──────────────────── # [科技新闻] # 科技 # @5 # # 效果：最多显示 5 条匹配的新闻 # 说明：@数字 表示限制条数 # # # 8. 必须同时包含多个词 # ──────────────────── # +苹果 # +发布会 # # 效果：必须同时包含"苹果"和"发布会"才匹配 # 说明：+ 开头的词表示"必须包含" # # ─────────────────────────────────────────────────────────────── [WORD_GROUPS] # ═══════════════════════════════════════════════════════════════ # 企业与品牌 # ═══════════════════════════════════════════════════════════════ /胖东来|于东来/ => 胖东来 /深度求索|幻方量化|梁文锋|\bDeepSeek\b/ => DeepSeek /华为|任正非|余承东|鸿蒙|海思|昇腾|鲲鹏|\bHUAWEI\b|\bHarmonyOS\b|\bHiSilicon\b/ => 华为 /比亚迪|王传福|方程豹|腾势|仰望|弗迪|刀片电池|云辇|\bBYD\b|\bDenza\b|\bYangwang\b/ => 比亚迪 /大疆|汪滔|灵眸|如影|\bDJI\b|\bRoboMaster\b|\bMavic\b|\bZenmuse\b/ => 大疆 /宇树|王兴兴|\bUnitree\b/ => 宇树机器人 /智元|灵犀|稚晖君|彭志辉|AgiBot/ => 智元机器人 /众擎|EngineAI|赵同阳/ => 众擎机器人 /黑神话|冯骥/ => 黑神话悟空 /影之刃零|梁其伟/ => 影之刃零 /三体|流浪地球|刘慈欣|郭帆/ => 三体/流浪地球 申奥 /京东|刘强东|\bJD\b|\bJingdong\b/ => 京东 /字节|张一鸣|梁汝波|抖音|\bByteDance\b|\bTikTok\b|\bDouyin\b|\bLark\b|\bCapCut\b/ => 字节跳动 /腾讯|鹅厂|马化腾|微信|QQ|天美|阅文集团|微众银行|\bTencent\b|\bPony Ma\b|\bWeChat\b|\bLightSpeed\b|\bWeBank\b/ => 腾讯 /qwen|minimax|glm/ => 国产开源模型 /特斯拉|马斯克|\bTesla\b|\bElon Musk\b|\bCybertruck\b|\bModel 3\b|\bModel Y\b|\bModel S\b|\bModel X\b|\bFSD\b/ => 特斯拉 /英伟达|黄仁勋|\bNVIDIA\b|\bGeForce\b|\bRTX\b|\bCUDA\b|\bJensen Huang\b/ => 英伟达 /苏姿丰|锐龙|霄龙|\bAMD\b|\bRyzen\b|\bEPYC\b|\bRadeon\b|\bLisa Su\b/ => AMD /微软|\bMicrosoft\b|\bWindows\b|\bAzure\b|\bSatya Nadella\b|\bCopilot\b/ => 微软 /谷歌|皮查伊|安卓|油管|\bGoogle\b|\bAlphabet\b|\bAndroid\b|\bChrome\b|\bYouTube\b|\bGemini\b|\bDeepMind\b|\bWaymo\b/ => 谷歌 /库克|\biPhone\b|\biPad\b|\bMacBook\b|\biOS\b|\bVision Pro\b|\bAirPods\b|\bApple\b|\bTim Cook\b/ => 苹果 /\bOpenAI\b|\bChatGPT\b|\bSora\b|\bDALL-E\b|\bSam Altman\b|\bGreg Brockman\b/ => OpenAI /\bAnthropic\b|\bClaude\b|\bDario Amodei\b/ => Claude # ═══════════════════════════════════════════════════════════════ # 国家与地区 # ═══════════════════════════════════════════════════════════════ [中国] 国产 中国 [东亚] 日本 朝鲜 韩国 [北美] 美国 加拿大 [西欧] 法国 英国 /俄罗斯|俄国/ => 俄罗斯 印度 # ═══════════════════════════════════════════════════════════════ # 科技领域 # ═══════════════════════════════════════════════════════════════ [AI 相关] /(? 水电 /光伏|太阳能/ => 光伏 核能 能源 /自动驾驶|无人驾驶|智驾/ => 自动驾驶 机器人 /机械狗|四足/ => 机器狗 具身智能 /月球|登月|火星|宇宙|飞船|航天|空间站|卫星/ => 航天 # 前沿科技 量子 脑机 基因 # 产业政策 生产力 ================================================ FILE: config/timeline.yaml ================================================ # ═══════════════════════════════════════════════════════════════ # TrendRadar 时间线配置 # Version: 1.2.0 # ═══════════════════════════════════════════════════════════════ # # 这个文件控制「什么时间做什么事」。 # # 大多数人不需要编辑这个文件。 # 只需在 config.yaml 中选择一个预设模板即可： # # schedule: # preset: "morning_evening" ← 改这里就行 # # # 可视化配置编辑器地址: https://sansan0.github.io/TrendRadar/ # # # ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ # 📖 基本概念（帮助你理解后面的配置） # ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ # # # 🔁 程序是怎么运行的？ # # TrendRadar 不是一直在后台运行的，而是被「定时闹钟」周期性唤醒： # # GitHub Actions 用户 → 由 .github/workflows/crawler.yml 中的 cron 定时触发 # 默认每小时运行一次（如每小时第 33 分钟） # # Docker 用户 → 由 docker/.env 中的 CRON_SCHEDULE 定时触发 # 默认每 30 分钟运行一次 # # 每次被唤醒后，程序按以下三个阶段依次执行： # # 1️⃣ 采集（collect） # 爬取各热榜平台 + RSS 订阅源的最新数据，存入数据库 # # ⬇ # # 2️⃣ 分析（analyze） # 调用 AI 大模型对采集到的新闻进行深度分析（可选，需配置 API Key） # # ⬇ # # 3️⃣ 推送（push） # 将整理好的热点新闻 + AI 分析结果发送到你的通知渠道 # （飞书、钉钉、Telegram、邮件等） # # 这三个阶段都可以独立开关。本文件的作用就是控制： # 「在什么时间段，开启/关闭哪些阶段」。 # # # 🔌 config.yaml 总开关 与 timeline 时间段开关 的关系 # # config.yaml 里有几个「总开关」，它们的优先级高于本文件： # # platforms.enabled: false → 永远不爬热榜（无论 timeline 怎么设置） # rss.enabled: false → 永远不爬 RSS（同上） # notification.enabled: false → 永远不推送（同上） # ai_analysis.enabled: false → 永远不分析（同上） # # 只有当总开关为 true 时，timeline 的时间段开关才会生效。 # 换句话说：总开关决定「能不能做」，timeline 决定「什么时候做」。 # # # ⏰ 什么是「时间段」和「静默期」？ # # 你可以把一天想象成一条时间线，上面划分了若干个「时间段」。 # 每个时间段有自己的行为开关（是否采集、是否分析、是否推送）。 # # 而不在任何时间段内的时间，就叫「静默期」（走 default 默认配置）。 # 静默期通常必须要采集，这样数据一直在积累， # 等到推送时，就能汇总出完整的报告。 # # # 💡 静默期越长，积累的数据越丰富（排名变化轨迹、上榜/下榜时间等）， # 最终提交给 AI 分析的上下文也越完整，分析质量更高。 # 相比 MCP Server，该方案的全天数据能呈现更完整的热度趋势和变化脉络。 # # # ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ # 📋 预设模板一览（选一个就行） # ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ # # 1️⃣ always_on 全天候，有新增就推（默认） # 2️⃣ morning_evening 全天推送 + 晚间汇总（推荐大多数人） # 3️⃣ office_hours 工作日三段式：到岗速览→午间热点→收工汇总 # 4️⃣ night_owl 午后速览 + 深夜全天汇总 # 5️⃣ custom 完全自定义（需要编辑本文件底部的 custom 段） # # 想自定义？两种方式： # 1. 直接翻到本文件底部的「自定义模式」部分 # 2. 在下方 presets 里新增你自己的预设模板 # （只要 key 不重复，然后在 config.yaml 里填你的模板名即可） # # ⚠️ 关于时间段设计的建议： # GitHub Actions： 建议定时任务间隔 ≥ 2 小时。由于系统触发存在随机延迟，间隔过短可能导致任务漏运行。 # Docker 用户：cron 定时是准时的，无此限制，按需设置即可。 # # # ═══════════════════════════════════════════════════════════════ # ─────────────────────────────────────────────────────────────── # 预设模板 # ─────────────────────────────────────────────────────────────── presets: # ─────────────────────────────────────────────────────────── # 1️⃣ always_on - 全天候监控 # # 最简单的模式：全天候采集 + 推送，有新增就通知你。 # 不划分时间段，全天使用同一套配置。 # 适合：重度用户、实时舆情监控 # # 全天：推送 ✓ | AI分析 ✗ | 不限推送次数 # ─────────────────────────────────────────────────────────── always_on: name: "全天监控" description: "全天候监控，有新增立即推送。适合重度用户。" # 默认配置 ── 不在任何时间段内时，使用这组开关 # 因为这个模式没有划分时间段，所以 default 就是全天的行为 default: collect: true # 采集数据（爬取热榜 + RSS） analyze: false # 不做 AI 分析（节省 API 费用） ai_mode: "current" # AI 分析当前榜单 push: true # 有新内容就推送 report_mode: "incremental" # 只推送新增内容，避免重复 once: # 限制每个时间段内只执行一次 analyze: false # 不限制分析次数 push: false # 不限制推送次数 # 没有定义任何时间段，全天都走 default # # 语法提示：{} 是 YAML 的「空字典」写法，表示里面没有任何内容。 # 等价于写成多行但什么都不填。后面的 [] 同理，表示「空列表」。 periods: {} day_plans: all_day: periods: [] # 空列表 = 这天不启用任何时间段 week_map: 1: "all_day" # 周一 2: "all_day" # 周二 3: "all_day" # 周三 4: "all_day" # 周四 5: "all_day" # 周五 6: "all_day" # 周六 7: "all_day" # 周日 # ─────────────────────────────────────────────────────────── # 2️⃣ morning_evening - 早晚汇总（推荐） # # 全天推送当前热点 + 晚间做一次当日全天汇总。 # 适合：大多数人 # # 默认（全天）：推送 ✓ | AI分析 ✓ | 不限推送次数 # 晚间汇总：推送 ✓ | AI分析 ✓ | 只推/分析一次 # ─────────────────────────────────────────────────────────── morning_evening: name: "早晚汇总" description: "全天推送 + 晚间当日汇总。适合大多数人。" # 默认配置 ── 不命中任何时间段时的行为 default: collect: true # 始终采集 analyze: true # AI 分析当前榜单 ai_mode: "current" # AI 分析当前榜单 push: true # 每次推送当前在榜热点 report_mode: "current" # 当前在榜的新闻 # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） once: analyze: false # 不限制分析次数 push: false # 不限制推送次数 # 时间段定义 ── 只有晚间汇总需要特殊处理 periods: evening_summary: name: "晚间汇总" start: "20:00" end: "22:00" # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） analyze: true # 晚间做 AI 分析 ai_mode: "daily" # AI 也汇总全天内容 report_mode: "daily" # 切换为当日全部新闻汇总 once: analyze: true # 窗口内只分析一次 push: true # 窗口内只推送一次 # 日计划 ── 把时间段组装成一天的安排 day_plans: all_day: periods: ["evening_summary"] # 周映射 ── 每天用哪个日计划（1=周一 ... 7=周日） week_map: 1: "all_day" 2: "all_day" 3: "all_day" 4: "all_day" 5: "all_day" 6: "all_day" 7: "all_day" # ─────────────────────────────────────────────────────────── # 3️⃣ office_hours - 办公时间推送 # # 工作日三段式推送，周末增量自由推。 # 适合：上班族、企业用户 # # 默认（静默期）：推送 ✗ | AI分析 ✗ # 到岗速览：推送 ✓ | AI分析 ✓ | 只推一次 # 午间热点：推送 ✓ | AI分析 ✗ | 只推一次 # 收工汇总：推送 ✓ | AI分析 ✓ | 只推一次 # 周末自由：推送 ✓ | AI分析 ✗ | 不限推送次数 # ─────────────────────────────────────────────────────────── office_hours: name: "办公时间" description: "工作日三段式推送（到岗→午间→收工），周末增量自由推送。" default: collect: true analyze: false ai_mode: "current" push: false # 默认不推送 report_mode: "current" once: analyze: true # 每个时段只分析一次 push: true # 每个时段只推送一次 periods: morning_briefing: name: "到岗速览" start: "09:00" end: "11:00" analyze: true # AI 分析当前热点 ai_mode: "current" # AI 分析当前榜单 push: true # 到岗后看当前热点 report_mode: "current" # 当前在榜的新闻 # once 继承 default（analyze: true, push: true）→ 只推/分析一次 noon_update: name: "午间热点" start: "13:00" end: "15:00" push: true # 午间推送当前在榜热点 report_mode: "current" # 当前在榜的新闻 # analyze 继承 default: false → 午间不做 AI 分析，节省 API # once 继承 default（push: true）→ 只推一次 closing_summary: name: "收工汇总" start: "17:00" end: "19:00" analyze: true # AI 做全天汇总分析 ai_mode: "daily" # AI 也分析全天内容 push: true # 下班前推送当日完整汇总 report_mode: "daily" # 当日全部新闻汇总 # once 继承 default（analyze: true, push: true）→ 只推/分析一次 weekend_free: name: "周末自由" start: "08:00" end: "23:00" ai_mode: "current" # AI 分析当前榜单 push: true # 有新增就推送 report_mode: "incremental" # 增量模式：有新增才推，没有就安静 once: analyze: false # 不限制分析次数 push: false # 不限制推送次数 # 工作日使用三段式推送；周末使用增量自由模式 day_plans: workday: periods: ["morning_briefing", "noon_update", "closing_summary"] weekend: periods: ["weekend_free"] # 周末：有新增就推，不打扰睡眠 week_map: 1: "workday" # 周一 → 工作日计划 2: "workday" 3: "workday" 4: "workday" 5: "workday" 6: "weekend" # 周六 → 周末计划 7: "weekend" # 周日 → 周末计划 # ─────────────────────────────────────────────────────────── # 4️⃣ night_owl - 夜猫子模式 # # 白天安静，午后和深夜各推一次。 # 适合：夜间工作者、海外时差用户、自由职业者 # # 默认（白天静默）：推送 ✗ | AI分析 ✗ # 午后速览：推送 ✓ | AI分析 ✓ | 只推一次 # 深夜汇总：推送 ✓ | AI分析 ✓ | 只推一次 # ─────────────────────────────────────────────────────────── night_owl: name: "夜猫子模式" description: "午后速览 + 深夜全天汇总。适合夜间工作者、海外时差用户。" default: collect: true analyze: false ai_mode: "current" push: false report_mode: "current" once: analyze: true # 每个时段只分析一次 push: true # 每个时段只推送一次 periods: afternoon_peek: name: "午后速览" start: "15:00" end: "17:00" analyze: true # AI 分析当前热点 ai_mode: "current" # AI 分析当前榜单 push: true # 午后看当前热点 report_mode: "current" # 当前在榜的新闻 # once 继承 default（analyze: true, push: true）→ 只推/分析一次 late_night: name: "深夜汇总" start: "22:00" end: "01:00" # start > end → 自动识别为跨日 analyze: true # AI 做全天汇总分析 ai_mode: "daily" # AI 也分析全天内容 push: true # 深夜推送当日完整汇总 report_mode: "daily" # 当日全部新闻汇总 # once 继承 default（analyze: true, push: true）→ 只推/分析一次 day_plans: all_day: periods: ["afternoon_peek", "late_night"] week_map: 1: "all_day" 2: "all_day" 3: "all_day" 4: "all_day" 5: "all_day" 6: "all_day" 7: "all_day" # ═══════════════════════════════════════════════════════════════ # # 5️⃣ 自定义模式 # # 当 config.yaml 中设置 schedule.preset: "custom" 时， # 系统会读取下面这段配置。 # # 如果上面的预设模板无法满足你的需求，可以在这里自由定义。 # # ═══════════════════════════════════════════════════════════════ # # 自定义配置的思路很简单，就像搭积木： # # 第 1 步：定义「积木块」（periods） # 每块积木 = 一个时间段 + 这段时间要做什么 # 例如：早间 08-10 推送、晚间 19-21 汇总 # # 第 2 步：拼成「一天的安排」（day_plans） # 把积木块组合起来，形成一天的日程 # 例如：工作日用 [早间, 晚间]，周末用 [晚间] # # 第 3 步：指定「每天用哪个安排」（week_map） # 周一到周日，分别对应哪个日计划 # 例如：周一~周五用 workday，周六周日用 weekend # # 另外还有一个「默认配置」（default）， # 当某个时刻不在任何积木块内时，就用默认配置。 # 积木块里没写的字段，也会自动回退到默认配置。 # # # 下面是一个完整的自定义示例，工作日和周末使用不同的时间段安排： # # 工作日时间段: # 深夜静默 23:00-06:00（跨日）：采集 ✓ | 分析 ✓ | 推送 ✗ # 工作日早间 08:00-10:00：推送 ✓ | incremental # 晚间汇总 19:00-21:00：推送 ✓ | 分析 ✓ | daily # 其余时间走默认配置（静默采集） # # 周末时间段: # 深夜静默 23:00-06:00（跨日）：采集 ✓ | 分析 ✓ | 推送 ✗ # 周末早间 10:00-12:00：推送 ✓ | daily # 晚间汇总 19:00-21:00：推送 ✓ | 分析 ✓ | daily # 其余时间走默认配置（静默采集） custom: name: "自定义" description: "完全自由定义时间段、日计划和周映射。" # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 默认配置 # # 当前时刻不在任何时间段（积木块）内时，使用这组开关。 # 时间段中没有写的字段，也会回退到这里。 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ default: collect: true # 是否采集数据（爬取热榜 + RSS） analyze: false # 是否执行 AI 分析 ai_mode: "current" # AI 分析模式: # follow_report → 跟随 report_mode # daily → 强制全天汇总 # current → 强制当前榜单 # incremental → 强制增量模式 push: false # 是否发送推送通知 report_mode: "current" # 报告模式: # daily → 当日所有新闻的汇总 # current → 当前在榜的新闻 # incremental → 只推送新增内容 # frequency_file: "general.txt" # 关键词文件（可选，位于 config/custom/keyword/） # 不填则使用默认的 config/frequency_words.txt # 时间段（period）中也可以设置此字段来覆盖默认值 # 例如晚间汇总用科技词库： # frequency_file: "tech.txt" # 注意：仅在 filter_method 为 keyword 时生效 # interests_file: "finance.txt" # AI 兴趣描述文件（可选，位于 config/custom/ai/） # 不填则使用默认的 config/ai_interests.txt # 时间段（period）中也可以设置此字段来覆盖默认值 # 例如晚间汇总用金融兴趣： # interests_file: "finance.txt" # 注意：仅在 filter_method 为 ai 时生效 # filter_method: "keyword" # 筛选策略（可选: keyword | ai） # 不填则使用全局 config.yaml 的 filter.method # 时间段（period）中也可以设置此字段来覆盖 # 例如晚间汇总用 AI 筛选： # filter_method: "ai" once: analyze: true # 该时间段内只分析一次（省 API） push: true # 该时间段内只推送一次（省打扰） # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 第 1 步：定义积木块（时间段） # # 每个时间段有一个唯一的 key（如 deep_quiet）， # 以及 start / end 表示生效的时间范围。 # # 只需要写「和 default 不同的字段」，其余自动继承 default。 # 例如 weekday_morning 没写 collect，就会继承 default 的 collect: true。 # # 提示：如果 start > end（如 22:00 → 07:00）， # 系统会自动识别为跨越午夜的时间段。 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ periods: deep_quiet: name: "深夜静默" start: "23:00" end: "06:00" # 23:00 → 次日 06:00（跨日时间段） # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） collect: true # 夜间继续采集数据 analyze: true # 夜间可以跑 AI 分析（反正不推送） push: false # 深夜不推送，避免打扰 weekday_morning: name: "工作日早间" start: "08:00" end: "10:00" # 跨度 2h，留足触发裕量 # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） push: true # 早上推送一次 report_mode: "incremental" # 只推新增内容 # once 继承 default（push: true）→ 窗口内只推一次 weekend_morning: name: "周末早间" start: "10:00" end: "12:00" # 跨度 2h # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） push: true report_mode: "daily" # 周末看全天汇总 # once 继承 default（push: true）→ 窗口内只推一次 evening_summary: name: "晚间汇总" start: "19:00" end: "21:00" # frequency_file: "xxx.txt" # 关键词文件（可选，位于 config/custom/keyword/） # interests_file: "xxx.txt" # AI 兴趣文件（可选，位于 config/custom/ai/） # filter_method: "keyword" # 筛选策略（可选: keyword | ai，不填用全局 filter.method） analyze: true # 晚间做 AI 分析 ai_mode: "daily" # AI 也分析全天内容 push: true # 晚间推送 report_mode: "daily" # 当日全部新闻汇总 # once 继承 default（analyze: true, push: true）→ 只分析/推送一次 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 第 2 步：把积木块拼成日计划 # # 把上面定义的时间段组合成一天的安排。 # 你可以定义多个日计划（如 workday 和 weekend）， # 然后在第 3 步的 week_map 中分配给不同的星期。 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ day_plans: workday: # 工作日计划 periods: ["deep_quiet", "weekday_morning", "evening_summary"] weekend: # 周末计划（用 weekend_morning 替换 weekday_morning） periods: ["deep_quiet", "weekend_morning", "evening_summary"] # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 第 3 步：指定每天用哪个日计划 # # 1=周一 2=周二 3=周三 4=周四 5=周五 6=周六 7=周日 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ week_map: 1: "workday" # 周一 → 工作日计划 2: "workday" # 周二 3: "workday" # 周三 4: "workday" # 周四 5: "workday" # 周五 6: "weekend" # 周六 → 周末计划 7: "weekend" # 周日 # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ # 冲突策略（一般不用改） # # 什么是「冲突」？ # 如果你的两个时间段有重叠（比如 A 是 08:00-12:00，B 是 10:00-14:00）， # 那么 10:00-12:00 这段时间就同时属于 A 和 B，产生了冲突。 # 此时程序需要知道：到底听谁的？ # # 两种处理方式： # # error_on_overlap（推荐） # 直接报错，提醒你去修改配置。 # 适合大多数人 —— 时间段重叠通常是写错了，报错能及时发现。 # # last_wins # day_plans 的 periods 列表中，写在后面的优先。 # 比如 periods: ["A", "B"]，重叠时 B 生效。 # 适合场景：你想用一个大范围时间段打底，再用后面的小范围覆盖。 # # ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ overlap: policy: "error_on_overlap" ================================================ FILE: docker/Dockerfile ================================================ FROM python:3.12-slim-bookworm WORKDIR /app # Latest releases available at https://github.com/aptible/supercronic/releases ARG TARGETARCH ENV SUPERCRONIC_VERSION=v0.2.39 RUN set -ex && \ apt-get update && \ apt-get install -y --no-install-recommends curl ca-certificates && \ case ${TARGETARCH} in \ amd64) \ export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-amd64; \ export SUPERCRONIC_SHA1SUM=c98bbf82c5f648aaac8708c182cc83046fe48423; \ export SUPERCRONIC=supercronic-linux-amd64; \ ;; \ arm64) \ export SUPERCRONIC_URL=https://github.com/aptible/supercronic/releases/download/${SUPERCRONIC_VERSION}/supercronic-linux-arm64; \ export SUPERCRONIC_SHA1SUM=5ef4ccc3d43f12d0f6c3763758bc37cc4e5af76e; \ export SUPERCRONIC=supercronic-linux-arm64; \ ;; \ *) \ echo "Unsupported architecture: ${TARGETARCH}"; \ exit 1; \ ;; \ esac && \ echo "Downloading supercronic for ${TARGETARCH} from ${SUPERCRONIC_URL}" && \ # 重试机制：最多3次，每次超时30秒 for i in 1 2 3; do \ echo "Download attempt $i/3"; \ if curl -fsSL --connect-timeout 30 --max-time 60 -o "$SUPERCRONIC" "$SUPERCRONIC_URL"; then \ echo "Download successful"; \ break; \ else \ echo "Download attempt $i failed"; \ if [ $i -eq 3 ]; then \ echo "All download attempts failed"; \ exit 1; \ fi; \ sleep 2; \ fi; \ done && \ echo "${SUPERCRONIC_SHA1SUM} ${SUPERCRONIC}" | sha1sum -c - && \ chmod +x "$SUPERCRONIC" && \ mv "$SUPERCRONIC" "/usr/local/bin/${SUPERCRONIC}" && \ ln -s "/usr/local/bin/${SUPERCRONIC}" /usr/local/bin/supercronic && \ supercronic -version && \ apt-get remove -y curl && \ apt-get clean && \ rm -rf /var/lib/apt/lists/* COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY docker/manage.py . COPY trendradar/ ./trendradar/ # 复制 entrypoint.sh 并强制转换为 LF 格式 COPY docker/entrypoint.sh /entrypoint.sh.tmp RUN sed -i 's/\r$//' /entrypoint.sh.tmp && \ mv /entrypoint.sh.tmp /entrypoint.sh && \ chmod +x /entrypoint.sh && \ chmod +x manage.py && \ mkdir -p /app/config /app/output ENV PYTHONUNBUFFERED=1 \ CONFIG_PATH=/app/config/config.yaml \ FREQUENCY_WORDS_PATH=/app/config/frequency_words.txt ENTRYPOINT ["/entrypoint.sh"] ================================================ FILE: docker/Dockerfile.mcp ================================================ FROM python:3.12-slim-bookworm WORKDIR /app # 安装依赖 COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # 复制 MCP 服务器代码 COPY mcp_server/ ./mcp_server/ # 复制 trendradar 模块（MCP 服务需要读取 SQLite 数据） COPY trendradar/ ./trendradar/ # 创建必要目录 RUN mkdir -p /app/config /app/output ENV PYTHONUNBUFFERED=1 \ CONFIG_PATH=/app/config/config.yaml \ FREQUENCY_WORDS_PATH=/app/config/frequency_words.txt # MCP HTTP 服务端口 EXPOSE 3333 # 启动 MCP 服务器（HTTP 模式） CMD ["python", "-m", "mcp_server.server", "--transport", "http", "--host", "0.0.0.0", "--port", "3333"] ================================================ FILE: docker/docker-compose-build.yml ================================================ services: trendradar: build: context: .. dockerfile: docker/Dockerfile container_name: trendradar restart: unless-stopped ports: - "127.0.0.1:${WEBSERVER_PORT:-8080}:${WEBSERVER_PORT:-8080}" volumes: - ../config:/app/config:ro - ../output:/app/output environment: - TZ=Asia/Shanghai # Web 服务器 - ENABLE_WEBSERVER=${ENABLE_WEBSERVER:-false} - WEBSERVER_PORT=${WEBSERVER_PORT:-8080} - WEBSERVER_WATCHDOG=${WEBSERVER_WATCHDOG:-true} - WEBSERVER_WATCHDOG_INTERVAL=${WEBSERVER_WATCHDOG_INTERVAL:-60} # 通知渠道 - FEISHU_WEBHOOK_URL=${FEISHU_WEBHOOK_URL:-} - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN:-} - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID:-} - DINGTALK_WEBHOOK_URL=${DINGTALK_WEBHOOK_URL:-} - WEWORK_WEBHOOK_URL=${WEWORK_WEBHOOK_URL:-} - WEWORK_MSG_TYPE=${WEWORK_MSG_TYPE:-} # 邮件配置 - EMAIL_FROM=${EMAIL_FROM:-} - EMAIL_PASSWORD=${EMAIL_PASSWORD:-} - EMAIL_TO=${EMAIL_TO:-} - EMAIL_SMTP_SERVER=${EMAIL_SMTP_SERVER:-} - EMAIL_SMTP_PORT=${EMAIL_SMTP_PORT:-} # ntfy配置 - NTFY_SERVER_URL=${NTFY_SERVER_URL:-https://ntfy.sh} - NTFY_TOPIC=${NTFY_TOPIC:-} - NTFY_TOKEN=${NTFY_TOKEN:-} # Bark配置 - BARK_URL=${BARK_URL:-} # Slack配置 - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL:-} # 通用Webhook配置 - GENERIC_WEBHOOK_URL=${GENERIC_WEBHOOK_URL:-} - GENERIC_WEBHOOK_TEMPLATE=${GENERIC_WEBHOOK_TEMPLATE:-} # AI 配置（ai_analysis 和 ai_translation 共享模型配置） - AI_ANALYSIS_ENABLED=${AI_ANALYSIS_ENABLED:-} - AI_API_KEY=${AI_API_KEY:-} - AI_MODEL=${AI_MODEL:-} - AI_API_BASE=${AI_API_BASE:-} # 远程存储配置（S3 兼容协议） - S3_ENDPOINT_URL=${S3_ENDPOINT_URL:-} - S3_BUCKET_NAME=${S3_BUCKET_NAME:-} - S3_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID:-} - S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY:-} - S3_REGION=${S3_REGION:-} # 运行模式 - CRON_SCHEDULE=${CRON_SCHEDULE:-*/30 * * * *} - RUN_MODE=${RUN_MODE:-cron} - IMMEDIATE_RUN=${IMMEDIATE_RUN:-true} trendradar-mcp: build: context: .. dockerfile: docker/Dockerfile.mcp container_name: trendradar-mcp restart: unless-stopped ports: - "127.0.0.1:3333:3333" volumes: - ../config:/app/config:ro - ../output:/app/output environment: - TZ=Asia/Shanghai ================================================ FILE: docker/docker-compose.yml ================================================ services: trendradar: image: wantcat/trendradar:latest container_name: trendradar restart: unless-stopped ports: - "127.0.0.1:${WEBSERVER_PORT:-8080}:${WEBSERVER_PORT:-8080}" volumes: - ../config:/app/config:ro - ../output:/app/output environment: - TZ=Asia/Shanghai # Web 服务器 - ENABLE_WEBSERVER=${ENABLE_WEBSERVER:-false} - WEBSERVER_PORT=${WEBSERVER_PORT:-8080} - WEBSERVER_WATCHDOG=${WEBSERVER_WATCHDOG:-true} - WEBSERVER_WATCHDOG_INTERVAL=${WEBSERVER_WATCHDOG_INTERVAL:-60} # 通知渠道 - FEISHU_WEBHOOK_URL=${FEISHU_WEBHOOK_URL:-} - TELEGRAM_BOT_TOKEN=${TELEGRAM_BOT_TOKEN:-} - TELEGRAM_CHAT_ID=${TELEGRAM_CHAT_ID:-} - DINGTALK_WEBHOOK_URL=${DINGTALK_WEBHOOK_URL:-} - WEWORK_WEBHOOK_URL=${WEWORK_WEBHOOK_URL:-} - WEWORK_MSG_TYPE=${WEWORK_MSG_TYPE:-} # 邮件配置 - EMAIL_FROM=${EMAIL_FROM:-} - EMAIL_PASSWORD=${EMAIL_PASSWORD:-} - EMAIL_TO=${EMAIL_TO:-} - EMAIL_SMTP_SERVER=${EMAIL_SMTP_SERVER:-} - EMAIL_SMTP_PORT=${EMAIL_SMTP_PORT:-} # ntfy配置 - NTFY_SERVER_URL=${NTFY_SERVER_URL:-https://ntfy.sh} - NTFY_TOPIC=${NTFY_TOPIC:-} - NTFY_TOKEN=${NTFY_TOKEN:-} # Bark配置 - BARK_URL=${BARK_URL:-} # Slack配置 - SLACK_WEBHOOK_URL=${SLACK_WEBHOOK_URL:-} # 通用Webhook配置 - GENERIC_WEBHOOK_URL=${GENERIC_WEBHOOK_URL:-} - GENERIC_WEBHOOK_TEMPLATE=${GENERIC_WEBHOOK_TEMPLATE:-} # AI 配置（ai_analysis 和 ai_translation 共享模型配置） - AI_ANALYSIS_ENABLED=${AI_ANALYSIS_ENABLED:-} - AI_API_KEY=${AI_API_KEY:-} - AI_MODEL=${AI_MODEL:-} - AI_API_BASE=${AI_API_BASE:-} # 远程存储配置（S3 兼容协议） - S3_ENDPOINT_URL=${S3_ENDPOINT_URL:-} - S3_BUCKET_NAME=${S3_BUCKET_NAME:-} - S3_ACCESS_KEY_ID=${S3_ACCESS_KEY_ID:-} - S3_SECRET_ACCESS_KEY=${S3_SECRET_ACCESS_KEY:-} - S3_REGION=${S3_REGION:-} # 运行模式 - CRON_SCHEDULE=${CRON_SCHEDULE:-*/30 * * * *} - RUN_MODE=${RUN_MODE:-cron} - IMMEDIATE_RUN=${IMMEDIATE_RUN:-true} trendradar-mcp: image: wantcat/trendradar-mcp:latest container_name: trendradar-mcp restart: unless-stopped ports: - "127.0.0.1:3333:3333" volumes: - ../config:/app/config:ro - ../output:/app/output environment: - TZ=Asia/Shanghai ================================================ FILE: docker/entrypoint.sh ================================================ #!/bin/bash set -e # 检查配置文件 if [ ! -f "/app/config/config.yaml" ] || [ ! -f "/app/config/frequency_words.txt" ]; then echo "❌ 配置文件缺失" exit 1 fi # 保存环境变量 env >> /etc/environment case "${RUN_MODE:-cron}" in "once") echo "🔄 单次执行" exec /usr/local/bin/python -m trendradar ;; "cron") # 生成 crontab echo "${CRON_SCHEDULE:-*/30 * * * *} cd /app && /usr/local/bin/python -m trendradar" > /tmp/crontab echo "📅 生成的crontab内容:" cat /tmp/crontab if ! /usr/local/bin/supercronic -test /tmp/crontab; then echo "❌ crontab格式验证失败" exit 1 fi # 立即执行一次（如果配置了） if [ "${IMMEDIATE_RUN:-false}" = "true" ]; then echo "▶️ 立即执行一次" /usr/local/bin/python -m trendradar fi # 启动 Web 服务器（如果配置了） if [ "${ENABLE_WEBSERVER:-false}" = "true" ]; then echo "🌐 启动 Web 服务器..." /usr/local/bin/python manage.py start_webserver WEBSERVER_WATCHDOG_ENABLED=$(echo "${WEBSERVER_WATCHDOG:-true}" | tr '[:upper:]' '[:lower:]') WEBSERVER_WATCHDOG_INTERVAL=${WEBSERVER_WATCHDOG_INTERVAL:-60} if [ "$WEBSERVER_WATCHDOG_ENABLED" = "true" ] || [ "$WEBSERVER_WATCHDOG_ENABLED" = "1" ] || [ "$WEBSERVER_WATCHDOG_ENABLED" = "yes" ] || [ "$WEBSERVER_WATCHDOG_ENABLED" = "on" ]; then # 启动后台 watchdog 定期检查 Web 服务器健康状态 echo "🔄 启动 Web 服务器 watchdog (间隔: ${WEBSERVER_WATCHDOG_INTERVAL}s)..." ( while true; do sleep "$WEBSERVER_WATCHDOG_INTERVAL" /usr/local/bin/python manage.py webserver_autofix done ) & WEBSERVER_WATCHDOG_PID=$! echo " ✅ watchdog 已启动 (PID: $WEBSERVER_WATCHDOG_PID)" else echo "⏸️ Web 服务器 watchdog 已禁用" fi fi echo "⏰ 启动supercronic: ${CRON_SCHEDULE:-*/30 * * * *}" echo "🎯 supercronic 将作为 PID 1 运行" exec /usr/local/bin/supercronic -passthrough-logs /tmp/crontab ;; *) exec "$@" ;; esac ================================================ FILE: docker/manage.py ================================================ #!/usr/bin/env python3 # -*- coding: utf-8 -*- """ 新闻爬虫容器管理工具 - supercronic """ import os import sys import subprocess import time import signal from pathlib import Path from datetime import datetime # Web 服务器配置 WEBSERVER_PORT = int(os.environ.get("WEBSERVER_PORT", "8080")) WEBSERVER_DIR = "/app/output" WEBSERVER_PID_FILE = "/tmp/webserver.pid" WEBSERVER_MANUAL_STOP_FILE = "/tmp/webserver.manual_stop" def _env_bool(name: str, default: bool) -> bool: """读取布尔环境变量，兼容 true/1/yes/on。""" value = os.environ.get(name) if value is None: return default return value.strip().lower() in {"1", "true", "yes", "on"} WEBSERVER_AUTOFIX_LOG_HEALTHY = _env_bool("WEBSERVER_AUTOFIX_LOG_HEALTHY", False) def get_timestamp(): """获取当前时间戳字符串""" return datetime.now().strftime("%Y-%m-%d %H:%M:%S") def run_command(cmd, shell=True, capture_output=True): """执行系统命令""" try: result = subprocess.run( cmd, shell=shell, capture_output=capture_output, text=True ) return result.returncode == 0, result.stdout, result.stderr except Exception as e: return False, "", str(e) def manual_run(): """手动执行一次爬虫""" print("🔄 手动执行爬虫...") try: result = subprocess.run( ["python", "-m", "trendradar"], cwd="/app", capture_output=False, text=True ) if result.returncode == 0: print("✅ 执行完成") else: print(f"❌ 执行失败，退出码: {result.returncode}") except Exception as e: print(f"❌ 执行出错: {e}") def parse_cron_schedule(cron_expr): """解析cron表达式并返回人类可读的描述""" if not cron_expr or cron_expr == "未设置": return "未设置" try: parts = cron_expr.strip().split() if len(parts) != 5: return f"原始表达式: {cron_expr}" minute, hour, day, month, weekday = parts # 分析分钟 if minute == "*": minute_desc = "每分钟" elif minute.startswith("*/"): interval = minute[2:] minute_desc = f"每{interval}分钟" elif "," in minute: minute_desc = f"在第{minute}分钟" else: minute_desc = f"在第{minute}分钟" # 分析小时 if hour == "*": hour_desc = "每小时" elif hour.startswith("*/"): interval = hour[2:] hour_desc = f"每{interval}小时" elif "," in hour: hour_desc = f"在{hour}点" else: hour_desc = f"在{hour}点" # 分析日期 if day == "*": day_desc = "每天" elif day.startswith("*/"): interval = day[2:] day_desc = f"每{interval}天" else: day_desc = f"每月{day}号" # 分析月份 if month == "*": month_desc = "每月" else: month_desc = f"在{month}月" # 分析星期 weekday_names = { "0": "周日", "1": "周一", "2": "周二", "3": "周三", "4": "周四", "5": "周五", "6": "周六", "7": "周日" } if weekday == "*": weekday_desc = "" else: weekday_desc = f"在{weekday_names.get(weekday, weekday)}" # 组合描述 if minute.startswith("*/") and hour == "*" and day == "*" and month == "*" and weekday == "*": # 简单的间隔模式，如 */30 * * * * return f"每{minute[2:]}分钟执行一次" elif hour != "*" and minute != "*" and day == "*" and month == "*" and weekday == "*": # 每天特定时间，如 0 9 * * * return f"每天{hour}:{minute.zfill(2)}执行" elif weekday != "*" and day == "*": # 每周特定时间 return f"{weekday_desc}{hour}:{minute.zfill(2)}执行" else: # 复杂模式，显示详细信息 desc_parts = [part for part in [month_desc, day_desc, weekday_desc, hour_desc, minute_desc] if part and part != "每月" and part != "每天" and part != "每小时"] if desc_parts: return " ".join(desc_parts) + "执行" else: return f"复杂表达式: {cron_expr}" except Exception as e: return f"解析失败: {cron_expr}" def show_status(): """显示容器状态""" print("📊 容器状态:") # 检查 PID 1 状态 supercronic_is_pid1 = False pid1_cmdline = "" try: with open('/proc/1/cmdline', 'r') as f: pid1_cmdline = f.read().replace('\x00', ' ').strip() print(f" 🔍 PID 1 进程: {pid1_cmdline}") if "supercronic" in pid1_cmdline.lower(): print(" ✅ supercronic 正确运行为 PID 1") supercronic_is_pid1 = True else: print(" ❌ PID 1 不是 supercronic") print(f" 📋 实际的 PID 1: {pid1_cmdline}") except Exception as e: print(f" ❌ 无法读取 PID 1 信息: {e}") # 检查环境变量 cron_schedule = os.environ.get("CRON_SCHEDULE", "未设置") run_mode = os.environ.get("RUN_MODE", "未设置") immediate_run = os.environ.get("IMMEDIATE_RUN", "未设置") print(f" ⚙️ 运行配置:") print(f" CRON_SCHEDULE: {cron_schedule}") # 解析并显示cron表达式的含义 cron_description = parse_cron_schedule(cron_schedule) print(f" ⏰ 执行频率: {cron_description}") print(f" RUN_MODE: {run_mode}") print(f" IMMEDIATE_RUN: {immediate_run}") # 检查配置文件 config_files = ["/app/config/config.yaml", "/app/config/frequency_words.txt"] print(" 📁 配置文件:") for file_path in config_files: if Path(file_path).exists(): print(f" ✅ {Path(file_path).name}") else: print(f" ❌ {Path(file_path).name} 缺失") # 检查关键文件 key_files = [ ("/usr/local/bin/supercronic-linux-amd64", "supercronic二进制文件"), ("/usr/local/bin/supercronic", "supercronic软链接"), ("/tmp/crontab", "crontab文件"), ("/entrypoint.sh", "启动脚本") ] print(" 📂 关键文件检查:") for file_path, description in key_files: if Path(file_path).exists(): print(f" ✅ {description}: 存在") # 对于crontab文件，显示内容 if file_path == "/tmp/crontab": try: with open(file_path, 'r') as f: crontab_content = f.read().strip() print(f" 内容: {crontab_content}") except: pass else: print(f" ❌ {description}: 不存在") # 检查容器运行时间 print(" ⏱️ 容器时间信息:") try: # 检查 PID 1 的启动时间 with open('/proc/1/stat', 'r') as f: stat_content = f.read().strip().split() if len(stat_content) >= 22: # starttime 是第22个字段（索引21） starttime_ticks = int(stat_content[21]) # 读取系统启动时间 with open('/proc/stat', 'r') as stat_f: for line in stat_f: if line.startswith('btime'): boot_time = int(line.split()[1]) break else: boot_time = 0 # 读取系统时钟频率 clock_ticks = os.sysconf(os.sysconf_names['SC_CLK_TCK']) if boot_time > 0: pid1_start_time = boot_time + (starttime_ticks / clock_ticks) current_time = time.time() uptime_seconds = int(current_time - pid1_start_time) uptime_minutes = uptime_seconds // 60 uptime_hours = uptime_minutes // 60 if uptime_hours > 0: print(f" PID 1 运行时间: {uptime_hours} 小时 {uptime_minutes % 60} 分钟") else: print(f" PID 1 运行时间: {uptime_minutes} 分钟 ({uptime_seconds} 秒)") else: print(f" PID 1 运行时间: 无法精确计算") else: print(" ❌ 无法解析 PID 1 统计信息") except Exception as e: print(f" ❌ 时间检查失败: {e}") # 状态总结和建议 print(" 📊 状态总结:") if supercronic_is_pid1: print(" ✅ supercronic 正确运行为 PID 1") print(" ✅ 定时任务应该正常工作") # 显示当前的调度信息 if cron_schedule != "未设置": print(f" ⏰ 当前调度: {cron_description}") # 提供一些常见的调度建议 if "分钟" in cron_description and "每30分钟" not in cron_description and "每60分钟" not in cron_description: print(" 💡 频繁执行模式，适合实时监控") elif "小时" in cron_description: print(" 💡 按小时执行模式，适合定期汇总") elif "天" in cron_description: print(" 💡 每日执行模式，适合日报生成") print(" 💡 如果定时任务不执行，检查:") print(" • crontab 格式是否正确") print(" • 时区设置是否正确") print(" • 应用程序是否有错误") else: print(" ❌ supercronic 状态异常") if pid1_cmdline: print(f" 📋 当前 PID 1: {pid1_cmdline}") print(" 💡 建议操作:") print(" • 重启容器: docker restart trendradar") print(" • 检查容器日志: docker logs trendradar") # 显示日志检查建议 print(" 📋 运行状态检查:") print(" • 查看完整容器日志: docker logs trendradar") print(" • 查看实时日志: docker logs -f trendradar") print(" • 手动执行测试: python manage.py run") print(" • 重启容器服务: docker restart trendradar") def show_config(): """显示当前配置""" print("⚙️ 当前配置:") env_vars = [ # 运行配置 "CRON_SCHEDULE", "RUN_MODE", "IMMEDIATE_RUN", # 通知渠道 "FEISHU_WEBHOOK_URL", "DINGTALK_WEBHOOK_URL", "WEWORK_WEBHOOK_URL", "WEWORK_MSG_TYPE", "TELEGRAM_BOT_TOKEN", "TELEGRAM_CHAT_ID", "NTFY_SERVER_URL", "NTFY_TOPIC", "NTFY_TOKEN", "BARK_URL", "SLACK_WEBHOOK_URL", # AI 分析配置 "AI_ANALYSIS_ENABLED", "AI_API_KEY", "AI_PROVIDER", "AI_MODEL", "AI_BASE_URL", # 远程存储配置 "S3_BUCKET_NAME", "S3_ACCESS_KEY_ID", "S3_ENDPOINT_URL", "S3_REGION", ] for var in env_vars: value = os.environ.get(var, "未设置") # 隐藏敏感信息 if any(sensitive in var for sensitive in ["WEBHOOK", "TOKEN", "KEY", "SECRET"]): if value and value != "未设置": masked_value = value[:10] + "***" if len(value) > 10 else "***" print(f" {var}: {masked_value}") else: print(f" {var}: {value}") else: print(f" {var}: {value}") crontab_file = "/tmp/crontab" if Path(crontab_file).exists(): print(" 📅 Crontab内容:") try: with open(crontab_file, "r") as f: content = f.read().strip() print(f" {content}") except Exception as e: print(f" 读取失败: {e}") else: print(" 📅 Crontab文件不存在") def show_files(): """显示输出文件""" print("📁 输出文件:") output_dir = Path("/app/output") if not output_dir.exists(): print(" 📭 输出目录不存在") return # 新结构：扁平化目录 # - output/news/*.db # - output/rss/*.db # - output/txt/{date}/*.txt # - output/html/{date}/*.html # 检查 news 数据库 news_dir = output_dir / "news" if news_dir.exists(): db_files = sorted(news_dir.glob("*.db"), key=lambda x: x.name, reverse=True) if db_files: print(f" 💾 热榜数据库 (news/): {len(db_files)} 个") for db_file in db_files[:5]: mtime = time.ctime(db_file.stat().st_mtime) size_kb = db_file.stat().st_size // 1024 print(f" 📀 {db_file.name} ({size_kb}KB, {mtime.split()[3][:5]})") if len(db_files) > 5: print(f" ... 还有 {len(db_files) - 5} 个") # 检查 RSS 数据库 rss_dir = output_dir / "rss" if rss_dir.exists(): db_files = sorted(rss_dir.glob("*.db"), key=lambda x: x.name, reverse=True) if db_files: print(f" 📰 RSS 数据库 (rss/): {len(db_files)} 个") for db_file in db_files[:5]: mtime = time.ctime(db_file.stat().st_mtime) size_kb = db_file.stat().st_size // 1024 print(f" 📀 {db_file.name} ({size_kb}KB, {mtime.split()[3][:5]})") if len(db_files) > 5: print(f" ... 还有 {len(db_files) - 5} 个") # 检查 TXT 快照目录 txt_dir = output_dir / "txt" if txt_dir.exists(): date_dirs = sorted([d for d in txt_dir.iterdir() if d.is_dir()], reverse=True) if date_dirs: print(f" 📄 TXT 快照 (txt/): {len(date_dirs)} 天") for date_dir in date_dirs[:3]: txt_files = list(date_dir.glob("*.txt")) if txt_files: recent = sorted(txt_files, key=lambda x: x.stat().st_mtime, reverse=True)[0] mtime = time.ctime(recent.stat().st_mtime) print(f" 📅 {date_dir.name}: {len(txt_files)} 个文件 (最新: {mtime.split()[3][:5]})") # 检查 HTML 报告目录 html_dir = output_dir / "html" if html_dir.exists(): date_dirs = sorted([d for d in html_dir.iterdir() if d.is_dir()], reverse=True) if date_dirs: print(f" 🌐 HTML 报告 (html/): {len(date_dirs)} 天") for date_dir in date_dirs[:3]: html_files = list(date_dir.glob("*.html")) if html_files: recent = sorted(html_files, key=lambda x: x.stat().st_mtime, reverse=True)[0] mtime = time.ctime(recent.stat().st_mtime) print(f" 📅 {date_dir.name}: {len(html_files)} 个文件 (最新: {mtime.split()[3][:5]})") def show_logs(): """显示实时日志""" print("📋 实时日志 (按 Ctrl+C 退出):") print("💡 提示: 这将显示 PID 1 进程的输出") try: # 尝试多种方法查看日志 log_files = [ "/proc/1/fd/1", # PID 1 的标准输出 "/proc/1/fd/2", # PID 1 的标准错误 ] for log_file in log_files: if Path(log_file).exists(): print(f"📄 尝试读取: {log_file}") subprocess.run(["tail", "-f", log_file], check=True) break else: print("📋 无法找到标准日志文件，建议使用: docker logs trendradar") except KeyboardInterrupt: print("\n👋 退出日志查看") except Exception as e: print(f"❌ 查看日志失败: {e}") print("💡 建议使用: docker logs trendradar") def restart_supercronic(): """重启supercronic进程""" print("🔄 重启supercronic...") print("⚠️ 注意: supercronic 是 PID 1，无法直接重启") # 检查当前 PID 1 try: with open('/proc/1/cmdline', 'r') as f: pid1_cmdline = f.read().replace('\x00', ' ').strip() print(f" 🔍 当前 PID 1: {pid1_cmdline}") if "supercronic" in pid1_cmdline.lower(): print(" ✅ PID 1 是 supercronic") print(" 💡 要重启 supercronic，需要重启整个容器:") print(" docker restart trendradar") else: print(" ❌ PID 1 不是 supercronic，这是异常状态") print(" 💡 建议重启容器以修复问题:") print(" docker restart trendradar") except Exception as e: print(f" ❌ 无法检查 PID 1: {e}") print(" 💡 建议重启容器: docker restart trendradar") def _read_proc_cmdline(pid: int) -> str: """读取进程 cmdline，失败时返回空字符串。""" proc_cmdline = Path(f"/proc/{pid}/cmdline") if not proc_cmdline.exists(): return "" try: with open(proc_cmdline, "rb") as f: return f.read().replace(b"\x00", b" ").decode("utf-8", errors="ignore").strip() except Exception: return "" def _is_expected_webserver_process(pid: int) -> bool: """检查 pid 是否是当前端口的 http.server 进程。""" cmdline = _read_proc_cmdline(pid) if not cmdline: return False return "http.server" in cmdline and str(WEBSERVER_PORT) in cmdline def _is_manual_stop_requested() -> bool: """是否处于手动停服状态。""" return Path(WEBSERVER_MANUAL_STOP_FILE).exists() def _set_manual_stop_marker(): """写入手动停服标记，防止 watchdog 自动拉起。""" try: with open(WEBSERVER_MANUAL_STOP_FILE, "w", encoding="utf-8") as f: f.write(get_timestamp()) except Exception: pass def _clear_manual_stop_marker(): """清理手动停服标记。""" try: if Path(WEBSERVER_MANUAL_STOP_FILE).exists(): os.remove(WEBSERVER_MANUAL_STOP_FILE) except Exception: pass def _terminate_webserver_process(pid: int, require_expected: bool = True) -> bool: """尝试终止 Web 服务器进程。 require_expected=True 时，仅终止确认是 http.server 的进程，避免误杀。 """ try: os.kill(pid, 0) except OSError: return True if require_expected and not _is_expected_webserver_process(pid): print(f" ⚠️ PID {pid} 存在但并非 Web 服务器进程，跳过终止") return False try: os.kill(pid, signal.SIGTERM) time.sleep(0.5) try: os.kill(pid, 0) os.kill(pid, signal.SIGKILL) print(f" ⚠️ 强制停止 Web 服务器 (PID: {pid})") except OSError: print(f" ✅ Web 服务器已停止 (PID: {pid})") return True except OSError: return True def _is_webserver_running(pid: int) -> bool: """检查 Web 服务器进程是否真正在运行。""" try: os.kill(pid, 0) except OSError: return False if not _is_expected_webserver_process(pid): return False try: import urllib.request req = urllib.request.Request(f"http://127.0.0.1:{WEBSERVER_PORT}/", method="HEAD") urllib.request.urlopen(req, timeout=3) return True except Exception: try: time.sleep(1) import urllib.request req = urllib.request.Request(f"http://127.0.0.1:{WEBSERVER_PORT}/", method="HEAD") urllib.request.urlopen(req, timeout=3) return True except Exception: return False def _cleanup_stale_pid(): """清理失效的 PID 文件""" if not Path(WEBSERVER_PID_FILE).exists(): return False try: with open(WEBSERVER_PID_FILE, 'r') as f: old_pid = int(f.read().strip()) os.remove(WEBSERVER_PID_FILE) print(f" 🧹 清理失效 PID 文件 (PID: {old_pid})") return True except Exception: return False def start_webserver(force: bool = False): """启动 Web 服务器托管 output 目录""" print(f"🌐 启动 Web 服务器 (端口: {WEBSERVER_PORT})...") print(f" 🔒 安全提示：仅提供静态文件访问，限制在 {WEBSERVER_DIR} 目录") if force: _clear_manual_stop_marker() elif _is_manual_stop_requested(): print(" ℹ️ 检测到手动停服标记，跳过自动启动") return # 检查是否已经运行 if Path(WEBSERVER_PID_FILE).exists(): try: with open(WEBSERVER_PID_FILE, 'r') as f: old_pid = int(f.read().strip()) # 使用增强的进程检查 if _is_webserver_running(old_pid): print(f" ⚠️ Web 服务器已在运行 (PID: {old_pid})") print(f" 💡 访问: http://localhost:{WEBSERVER_PORT}") print(" 💡 停止服务: python manage.py stop_webserver") return # 进程异常时优先尝试终止旧进程，避免端口占用导致重启失败 _terminate_webserver_process(old_pid, require_expected=True) _cleanup_stale_pid() print(f" ℹ️ 检测到失效的 PID 文件，已清理") except Exception as e: print(f" ⚠️ 清理旧的 PID 文件: {e}") _cleanup_stale_pid() # 检查目录是否存在 if not Path(WEBSERVER_DIR).exists(): print(f" ❌ 目录不存在: {WEBSERVER_DIR}") return try: # 启动 HTTP 服务器 # 使用 --bind 绑定到 0.0.0.0 使容器内部可访问 # 工作目录限制在 WEBSERVER_DIR，防止访问其他目录 process = subprocess.Popen( [sys.executable, '-m', 'http.server', str(WEBSERVER_PORT), '--bind', '0.0.0.0'], cwd=WEBSERVER_DIR, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True ) # 等待一下确保服务器启动 time.sleep(1) # 检查进程是否还在运行 if process.poll() is None: # 保存 PID with open(WEBSERVER_PID_FILE, 'w') as f: f.write(str(process.pid)) _clear_manual_stop_marker() print(f" ✅ Web 服务器已启动 (PID: {process.pid})") print(f" 📁 服务目录: {WEBSERVER_DIR} (只读，仅静态文件)") print(f" 🌐 访问地址: http://localhost:{WEBSERVER_PORT}") print(f" 📄 首页: http://localhost:{WEBSERVER_PORT}/index.html") print(" 💡 停止服务: python manage.py stop_webserver") else: print(f" ❌ Web 服务器启动失败") except Exception as e: print(f" ❌ 启动失败: {e}") def stop_webserver(): """停止 Web 服务器""" print("🛑 停止 Web 服务器...") _set_manual_stop_marker() if not Path(WEBSERVER_PID_FILE).exists(): print(" ℹ️ Web 服务器未运行") print(" ℹ️ 已写入手动停服标记，watchdog 不会自动拉起") return try: with open(WEBSERVER_PID_FILE, 'r') as f: pid = int(f.read().strip()) _terminate_webserver_process(pid, require_expected=True) if Path(WEBSERVER_PID_FILE).exists(): os.remove(WEBSERVER_PID_FILE) print(" ℹ️ 已写入手动停服标记，watchdog 不会自动拉起") except Exception as e: print(f" ❌ 停止失败: {e}") # 尝试清理 PID 文件 try: os.remove(WEBSERVER_PID_FILE) except: pass def webserver_status(): """查看 Web 服务器状态""" print("🌐 Web 服务器状态:") if not Path(WEBSERVER_PID_FILE).exists(): print(" ⭕ 未运行") if _is_manual_stop_requested(): print(" ℹ️ 当前为手动停服状态，watchdog 不会自动拉起") print(f" 💡 启动服务: python manage.py start_webserver") return try: with open(WEBSERVER_PID_FILE, 'r') as f: pid = int(f.read().strip()) # 使用增强的进程检查 if _is_webserver_running(pid): print(f" ✅ 运行中 (PID: {pid})") print(f" 📁 服务目录: {WEBSERVER_DIR}") print(f" 🌐 访问地址: http://localhost:{WEBSERVER_PORT}") print(f" 📄 首页: http://localhost:{WEBSERVER_PORT}/index.html") print(" 💡 停止服务: python manage.py stop_webserver") else: print(f" ⭕ 未运行 (PID 文件存在但进程不可用)") _cleanup_stale_pid() print(" 💡 启动服务: python manage.py start_webserver") except Exception as e: print(f" ❌ 状态检查失败: {e}") def webserver_autofix(): """Web 服务器健康检查和自动修复 供 watchdog/定时任务调用，检查服务状态并在需要时自动重启。 输出日志格式便于外部监控系统解析。 """ if _is_manual_stop_requested(): if WEBSERVER_AUTOFIX_LOG_HEALTHY: print(f"[{get_timestamp()}] ℹ️ 手动停服状态，跳过自动修复") return if not Path(WEBSERVER_PID_FILE).exists(): print(f"[{get_timestamp()}] ℹ️ Web 服务器未运行，启动中...") start_webserver(force=False) return try: with open(WEBSERVER_PID_FILE, 'r') as f: pid = int(f.read().strip()) # 使用增强检查 if not _is_webserver_running(pid): print(f"[{get_timestamp()}] ⚠️ Web 服务器不可用 (PID: {pid})，尝试重启...") _terminate_webserver_process(pid, require_expected=True) _cleanup_stale_pid() start_webserver(force=False) return if WEBSERVER_AUTOFIX_LOG_HEALTHY: print(f"[{get_timestamp()}] ✅ Web 服务器健康 (PID: {pid})") except Exception as e: print(f"[{get_timestamp()}] ❌ 健康检查异常: {e}") _cleanup_stale_pid() start_webserver(force=False) def show_help(): """显示帮助信息""" help_text = """ 🐳 TrendRadar 容器管理工具 📋 命令列表: run - 手动执行一次爬虫 status - 显示容器运行状态 config - 显示当前配置 files - 显示输出文件 logs - 实时查看日志 restart - 重启说明 start_webserver - 启动 Web 服务器托管 output 目录 stop_webserver - 停止 Web 服务器 webserver_status - 查看 Web 服务器状态 help - 显示此帮助 📖 使用示例: # 在容器中执行 python manage.py run python manage.py status python manage.py logs python manage.py start_webserver # 在宿主机执行 docker exec -it trendradar python manage.py run docker exec -it trendradar python manage.py status docker exec -it trendradar python manage.py start_webserver docker logs trendradar 💡 常用操作指南: 1. 检查运行状态: status - 查看 supercronic 是否为 PID 1 - 检查配置文件和关键文件 - 查看 cron 调度设置 2. 手动执行测试: run - 立即执行一次新闻爬取 - 测试程序是否正常工作 3. 查看日志: logs - 实时监控运行情况 - 也可使用: docker logs trendradar 4. 重启服务: restart - 由于 supercronic 是 PID 1，需要重启整个容器 - 使用: docker restart trendradar 5. Web 服务器管理: - 启动: start_webserver - 停止: stop_webserver（写入手动停服标记，watchdog 不自动拉起） - 状态: webserver_status - 访问: http://localhost:8080 """ print(help_text) def main(): if len(sys.argv) < 2: show_help() return command = sys.argv[1] commands = { "run": manual_run, "status": show_status, "config": show_config, "files": show_files, "logs": show_logs, "restart": restart_supercronic, "start_webserver": lambda: start_webserver(force=True), "stop_webserver": stop_webserver, "webserver_status": webserver_status, "webserver_autofix": webserver_autofix, "help": show_help, } if command in commands: try: commands[command]() except KeyboardInterrupt: print("\n👋 操作已取消") except Exception as e: print(f"❌ 执行出错: {e}") else: print(f"❌ 未知命令: {command}") print("运行 'python manage.py help' 查看可用命令") if __name__ == "__main__": main() ================================================ FILE: docs/assets/script.js ================================================ /** * TrendRadar 配置文件编辑器核心逻辑 * 特点：确保原始 YAML 的注释和格式 100% 保留 */ // ========================================== // 0. 注释高亮功能 // ========================================== /** * 对文本应用高亮，# 后的内容显示为灰色 */ function applyHighlight(text) { const escape = s => s.replace(/&/g, '&').replace(//g, '>'); return text.split('\n').map(line => { const idx = line.indexOf('#'); if (idx === -1) return escape(line); return escape(line.slice(0, idx)) + '' + escape(line.slice(idx)) + ''; }).join('\n'); } /** * 更新高亮层 */ function updateBackdrop(textareaId, backdropId) { const ta = document.getElementById(textareaId); const bd = document.getElementById(backdropId); if (ta && bd) bd.innerHTML = applyHighlight(ta.value) + '\n'; } /** * 同步滚动 */ function syncScroll(textareaId, backdropId) { const ta = document.getElementById(textareaId); const bd = document.getElementById(backdropId); if (ta && bd) { bd.scrollTop = ta.scrollTop; bd.scrollLeft = ta.scrollLeft; } } // ========================================== // 12. 二维码放大弹窗逻辑 // ========================================== const QR_MODAL_DATA = { weixin: { icon: '', iconBg: 'bg-green-100', title: '不迷路', subtitle: '第一时间获取更新通知', img: './assets/weixin.webp', alt: '微信公众号', hint: '微信扫码关注公众号' }, donate: { icon: '', iconBg: 'bg-emerald-100', title: '随心赞赏', subtitle: '金额随意，1 元也是鼓励 (´▽`ʃ♡ƪ)', img: 'https://cdn-1258574687.cos.ap-shanghai.myqcloud.com/img/%2F2026%2F01%2F18ecce7c224ce0ea4c59394c29e408f8-e0d1db45.webp', alt: '微信支付', hint: '微信扫码 · 丰俭由人' } }; function openQrModal(type) { const data = QR_MODAL_DATA[type]; if (!data) return; const modal = document.getElementById('qr-modal'); document.getElementById('qr-modal-icon').className = 'w-10 h-10 rounded-xl flex items-center justify-center text-lg ' + data.iconBg; document.getElementById('qr-modal-icon').innerHTML = data.icon; document.getElementById('qr-modal-title').textContent = data.title; document.getElementById('qr-modal-subtitle').textContent = data.subtitle; document.getElementById('qr-modal-img').src = data.img; document.getElementById('qr-modal-img').alt = data.alt; document.getElementById('qr-modal-hint').textContent = data.hint; modal.classList.remove('hidden'); } function closeQrModal() { const modal = document.getElementById('qr-modal'); if (modal) modal.classList.add('hidden'); } window.openQrModal = openQrModal; window.closeQrModal = closeQrModal; const MODULE_DEFS = [ { id: 1, name: "1. 基础设置", key: "app", editable: false }, { id: 2, name: "2. 数据源 - 热榜平台", key: "platforms", editable: true }, { id: 3, name: "3. 数据源 - RSS 订阅", key: "rss", editable: true }, { id: 4, name: "4. 报告模式", key: "report", editable: true }, { id: "4.5", name: "4.5 筛选策略", key: "filter", editable: true }, { id: "4.6", name: "4.6 AI 智能筛选", key: "ai_filter", editable: true }, { id: 5, name: "5. 推送内容控制", key: "display", editable: true }, { id: 6, name: "6. 推送通知", key: "notification", editable: true, partial: true }, { id: 7, name: "7. 存储配置", key: "storage", editable: false }, { id: 8, name: "8. AI 模型配置", key: "ai", editable: true }, { id: 9, name: "9. AI 分析功能", key: "ai_analysis", editable: true }, { id: 10, name: "10. AI 翻译功能", key: "ai_translation", editable: true }, { id: 11, name: "11. 高级设置", key: "advanced", editable: false } ]; // 初始默认内容 (用于空状态) - 只显示提示文本 const INITIAL_YAML = `# 在此粘贴你的 config.yaml... # 或拖拽文件到编辑器区域 # 或点击右上角"加载官网最新配置"`; // LocalStorage 键名 const STORAGE_KEY_CONFIG = 'trendradar_config_yaml'; const STORAGE_KEY_FREQUENCY = 'trendradar_frequency_txt'; const STORAGE_KEY_TIMELINE = 'trendradar_timeline_yaml'; const STORAGE_KEY_CONFIG_TIME = 'trendradar_config_time'; const STORAGE_KEY_FREQUENCY_TIME = 'trendradar_frequency_time'; const STORAGE_KEY_TIMELINE_TIME = 'trendradar_timeline_time'; // 官网配置文件 URL const REMOTE_CONFIG_URL = 'https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/config/config.yaml'; const REMOTE_FREQUENCY_URL = 'https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/config/frequency_words.txt'; const REMOTE_TIMELINE_URL = 'https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/config/timeline.yaml'; const REMOTE_VERSION_URL = 'https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version_configs'; let currentYaml = ""; let currentFrequency = ""; let currentTimeline = ""; let currentFrequencyData = null; // 缓存解析后的数据，避免重复解析导致索引错位 let currentTab = "config"; // ========================================== // 2. 初始化与事件绑定 // ========================================== // 防抖定时器 let configSaveTimer = null; let frequencySaveTimer = null; let timelineSaveTimer = null; document.addEventListener('DOMContentLoaded', () => { const yamlEditor = document.getElementById('yaml-editor'); const frequencyEditor = document.getElementById('frequency-editor'); // 尝试从 LocalStorage 恢复配置 const savedConfig = localStorage.getItem(STORAGE_KEY_CONFIG); const savedFrequency = localStorage.getItem(STORAGE_KEY_FREQUENCY); // 初始化编辑器 if (savedConfig && savedConfig.trim() && savedConfig !== INITIAL_YAML) { yamlEditor.value = savedConfig; currentYaml = savedConfig; showToast('已恢复上次保存的配置', 'info'); } else { yamlEditor.value = INITIAL_YAML; currentYaml = INITIAL_YAML; } if (savedFrequency && savedFrequency.trim()) { frequencyEditor.value = savedFrequency; currentFrequency = savedFrequency; } else { frequencyEditor.value = "# 在此粘贴你的 frequency_words.txt 内容...\n# 或拖拽文件到编辑器区域\n\n[GLOBAL_FILTER]\n\n[WORD_GROUPS]\n"; currentFrequency = frequencyEditor.value; } // 初始化 Timeline 编辑器 const timelineEditor = document.getElementById('timeline-editor'); const savedTimeline = localStorage.getItem(STORAGE_KEY_TIMELINE); const INITIAL_TIMELINE = `# 在此粘贴你的 timeline.yaml...\n# 或拖拽文件到编辑器区域\n# 或点击右上角"加载官网最新配置"`; if (savedTimeline && savedTimeline.trim() && savedTimeline !== INITIAL_TIMELINE) { timelineEditor.value = savedTimeline; currentTimeline = savedTimeline; } else { timelineEditor.value = INITIAL_TIMELINE; currentTimeline = INITIAL_TIMELINE; } // 渲染右侧模块列表 renderModules(); // 监听编辑器输入（实时同步到 UI + 防抖保存） yamlEditor.addEventListener('input', (e) => { currentYaml = e.target.value; updateBackdrop('yaml-editor', 'yaml-backdrop'); syncYamlToUI(); debounceSaveConfig(); }); frequencyEditor.addEventListener('input', (e) => { currentFrequency = e.target.value; updateBackdrop('frequency-editor', 'frequency-backdrop'); currentFrequencyData = null; syncFrequencyToUI(); debounceSaveFrequency(); }); timelineEditor.addEventListener('input', (e) => { currentTimeline = e.target.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); syncTimelineToUI(); debounceSaveTimeline(); }); // 同步滚动 yamlEditor.addEventListener('scroll', () => syncScroll('yaml-editor', 'yaml-backdrop')); frequencyEditor.addEventListener('scroll', () => syncScroll('frequency-editor', 'frequency-backdrop')); timelineEditor.addEventListener('scroll', () => syncScroll('timeline-editor', 'timeline-backdrop')); // 初始化拖拽上传功能 initDragAndDrop(yamlEditor, 'config'); initDragAndDrop(frequencyEditor, 'frequency'); initDragAndDrop(timelineEditor, 'timeline'); // 页面关闭/刷新时立即保存 window.addEventListener('beforeunload', saveAllToLocalStorage); document.addEventListener('keydown', function(e) { if ((e.ctrlKey || e.metaKey) && e.key === 's') { e.preventDefault(); saveAllToLocalStorage(); showToast('已手动保存配置', 'success'); } }); syncYamlToUI(); updateBackdrop('yaml-editor', 'yaml-backdrop'); updateBackdrop('frequency-editor', 'frequency-backdrop'); updateBackdrop('timeline-editor', 'timeline-backdrop'); updateSaveTimeDisplay(); }); // 防抖保存 config.yaml function debounceSaveConfig() { if (configSaveTimer) clearTimeout(configSaveTimer); configSaveTimer = setTimeout(() => { saveConfigToLocalStorage(); }, 1000); } // 防抖保存 frequency_words.txt function debounceSaveFrequency() { if (frequencySaveTimer) clearTimeout(frequencySaveTimer); frequencySaveTimer = setTimeout(() => { saveFrequencyToLocalStorage(); }, 1000); } // 防抖保存 timeline.yaml function debounceSaveTimeline() { if (timelineSaveTimer) clearTimeout(timelineSaveTimer); timelineSaveTimer = setTimeout(() => { saveTimelineToLocalStorage(); }, 1000); } // ========================================== // 2.1 拖拽上传功能 // ========================================== function initDragAndDrop(editor, type) { const container = editor.parentElement; const dropOverlay = document.createElement('div'); dropOverlay.className = 'drop-overlay hidden'; dropOverlay.innerHTML = `
释放以加载文件
${type === 'config' ? 'config.yaml' : type === 'timeline' ? 'timeline.yaml' : 'frequency_words.txt'}
`; container.style.position = 'relative'; container.appendChild(dropOverlay); editor.addEventListener('dragover', (e) => { e.preventDefault(); e.stopPropagation(); dropOverlay.classList.remove('hidden'); }); editor.addEventListener('dragleave', (e) => { e.preventDefault(); e.stopPropagation(); if (!container.contains(e.relatedTarget)) { dropOverlay.classList.add('hidden'); } }); dropOverlay.addEventListener('dragleave', (e) => { e.preventDefault(); e.stopPropagation(); if (!container.contains(e.relatedTarget)) { dropOverlay.classList.add('hidden'); } }); dropOverlay.addEventListener('dragover', (e) => { e.preventDefault(); e.stopPropagation(); }); dropOverlay.addEventListener('drop', (e) => { e.preventDefault(); e.stopPropagation(); dropOverlay.classList.add('hidden'); handleFileDrop(e, type); }); editor.addEventListener('drop', (e) => { e.preventDefault(); e.stopPropagation(); dropOverlay.classList.add('hidden'); handleFileDrop(e, type); }); } function handleFileDrop(e, type) { const files = e.dataTransfer.files; if (files.length === 0) return; const file = files[0]; const validExtensions = type === 'config' ? ['.yaml', '.yml', '.txt'] : type === 'timeline' ? ['.yaml', '.yml'] : ['.txt', '.yaml', '.yml']; const fileName = file.name.toLowerCase(); const isValid = validExtensions.some(ext => fileName.endsWith(ext)); if (!isValid) { showToast(`请拖入 ${type === 'config' || type === 'timeline' ? 'YAML' : 'TXT'} 文件`, 'error'); return; } const reader = new FileReader(); reader.onload = (event) => { const content = event.target.result; if (type === 'config') { try { jsyaml.load(content); document.getElementById('yaml-editor').value = content; currentYaml = content; syncYamlToUI(); showToast(`已加载: ${file.name}`, 'success'); } catch (err) { showToast(`YAML 语法错误: ${err.message}`, 'error'); // 仍然加载，让用户修复 document.getElementById('yaml-editor').value = content; currentYaml = content; } } else if (type === 'timeline') { try { jsyaml.load(content); document.getElementById('timeline-editor').value = content; currentTimeline = content; updateBackdrop('timeline-editor', 'timeline-backdrop'); syncTimelineToUI(); showToast(`已加载: ${file.name}`, 'success'); } catch (err) { showToast(`YAML 语法错误: ${err.message}`, 'error'); document.getElementById('timeline-editor').value = content; currentTimeline = content; } } else { document.getElementById('frequency-editor').value = content; currentFrequency = content; syncFrequencyToUI(); showToast(`已加载: ${file.name}`, 'success'); } }; reader.onerror = () => { showToast('文件读取失败', 'error'); }; reader.readAsText(file); } // ========================================== // 2.2 LocalStorage 保存与恢复 // ========================================== // 保存 config.yaml function saveConfigToLocalStorage() { try { if (currentYaml && currentYaml.trim().length > 10) { const now = new Date().toISOString(); localStorage.setItem(STORAGE_KEY_CONFIG, currentYaml); localStorage.setItem(STORAGE_KEY_CONFIG_TIME, now); updateSaveTimeDisplay(); } } catch (e) { console.warn('LocalStorage 保存 config 失败:', e); } } // 保存 frequency_words.txt function saveFrequencyToLocalStorage() { try { if (currentFrequency && currentFrequency.trim().length > 10) { const now = new Date().toISOString(); localStorage.setItem(STORAGE_KEY_FREQUENCY, currentFrequency); localStorage.setItem(STORAGE_KEY_FREQUENCY_TIME, now); updateSaveTimeDisplay(); } } catch (e) { console.warn('LocalStorage 保存 frequency 失败:', e); } } // 保存 timeline.yaml function saveTimelineToLocalStorage() { try { if (currentTimeline && currentTimeline.trim().length > 10) { const now = new Date().toISOString(); localStorage.setItem(STORAGE_KEY_TIMELINE, currentTimeline); localStorage.setItem(STORAGE_KEY_TIMELINE_TIME, now); updateSaveTimeDisplay(); } } catch (e) { console.warn('LocalStorage 保存 timeline 失败:', e); } } // 保存全部（页面关闭时调用） function saveAllToLocalStorage() { saveConfigToLocalStorage(); saveFrequencyToLocalStorage(); saveTimelineToLocalStorage(); } // 兼容旧调用 function saveToLocalStorage() { saveAllToLocalStorage(); } // 格式化时间显示 function formatSaveTime(isoString) { if (!isoString) return '未保存'; const date = new Date(isoString); const now = new Date(); const diffMs = now - date; const diffMins = Math.floor(diffMs / 60000); const diffHours = Math.floor(diffMs / 3600000); const diffDays = Math.floor(diffMs / 86400000); if (diffMins < 1) return '刚刚'; if (diffMins < 60) return `${diffMins} 分钟前`; if (diffHours < 24) return `${diffHours} 小时前`; if (diffDays < 7) return `${diffDays} 天前`; return date.toLocaleDateString('zh-CN', { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }); } // 更新保存时间显示 function updateSaveTimeDisplay() { const configTime = localStorage.getItem(STORAGE_KEY_CONFIG_TIME); const frequencyTime = localStorage.getItem(STORAGE_KEY_FREQUENCY_TIME); // 更新 config.yaml 的时间显示 const configTimeEl = document.getElementById('config-save-time'); const configLabelEl = document.getElementById('config-save-label'); if (configTimeEl) { configTimeEl.textContent = formatSaveTime(configTime); configTimeEl.title = configTime ? new Date(configTime).toLocaleString('zh-CN') : '未保存'; if (configLabelEl) { if (configTime) { configLabelEl.classList.remove('hidden'); } else { configLabelEl.classList.add('hidden'); } } } // 更新 frequency_words.txt 的时间显示 const frequencyTimeEl = document.getElementById('frequency-save-time'); const frequencyLabelEl = document.getElementById('frequency-save-label'); if (frequencyTimeEl) { frequencyTimeEl.textContent = formatSaveTime(frequencyTime); frequencyTimeEl.title = frequencyTime ? new Date(frequencyTime).toLocaleString('zh-CN') : '未保存'; if (frequencyLabelEl) { if (frequencyTime) { frequencyLabelEl.classList.remove('hidden'); } else { frequencyLabelEl.classList.add('hidden'); } } } // 更新 timeline.yaml 的时间显示 const timelineTime = localStorage.getItem(STORAGE_KEY_TIMELINE_TIME); const timelineTimeEl = document.getElementById('timeline-save-time'); const timelineLabelEl = document.getElementById('timeline-save-label'); if (timelineTimeEl) { timelineTimeEl.textContent = formatSaveTime(timelineTime); timelineTimeEl.title = timelineTime ? new Date(timelineTime).toLocaleString('zh-CN') : '未保存'; if (timelineLabelEl) { if (timelineTime) { timelineLabelEl.classList.remove('hidden'); } else { timelineLabelEl.classList.add('hidden'); } } } } // ========================================== // 2.3 加载官网最新配置 // ========================================== window.openLoadConfigModal = function() { // 创建选择弹窗 const modal = document.createElement('div'); modal.id = 'load-config-modal'; modal.className = 'modal-overlay'; modal.innerHTML = `

加载官网最新配置

选择要从 GitHub 加载的配置文件：
数据来源：sansan0/TrendRadar
`; document.body.appendChild(modal); } window.closeLoadConfigModal = function() { const modal = document.getElementById('load-config-modal'); if (modal) modal.remove(); } window.confirmLoadConfig = async function() { const loadConfig = document.getElementById('load-config-yaml')?.checked; const loadFrequency = document.getElementById('load-frequency-txt')?.checked; const loadTimeline = document.getElementById('load-timeline-yaml')?.checked; if (!loadConfig && !loadFrequency && !loadTimeline) { showToast('请至少选择一个文件', 'warning'); return; } closeLoadConfigModal(); showToast('正在从 GitHub 加载...', 'info'); try { const promises = []; if (loadConfig) promises.push(fetch(REMOTE_CONFIG_URL).then(r => ({ type: 'config', res: r }))); if (loadFrequency) promises.push(fetch(REMOTE_FREQUENCY_URL).then(r => ({ type: 'frequency', res: r }))); if (loadTimeline) promises.push(fetch(REMOTE_TIMELINE_URL).then(r => ({ type: 'timeline', res: r }))); const results = await Promise.all(promises); for (const { type, res } of results) { if (!res.ok) { const names = { config: 'config.yaml', frequency: 'frequency_words.txt', timeline: 'timeline.yaml' }; throw new Error(`${names[type]} 加载失败: ${res.status}`); } const text = await res.text(); if (type === 'config') { try { jsyaml.load(text); } catch (yamlErr) { showToast(`YAML 语法错误: ${yamlErr.message}`, 'error'); continue; } document.getElementById('yaml-editor').value = text; currentYaml = text; updateBackdrop('yaml-editor', 'yaml-backdrop'); syncYamlToUI(); } else if (type === 'timeline') { try { jsyaml.load(text); } catch (yamlErr) { showToast(`YAML 语法错误: ${yamlErr.message}`, 'error'); continue; } document.getElementById('timeline-editor').value = text; currentTimeline = text; updateBackdrop('timeline-editor', 'timeline-backdrop'); syncTimelineToUI(); } else { document.getElementById('frequency-editor').value = text; currentFrequency = text; currentFrequencyData = null; updateBackdrop('frequency-editor', 'frequency-backdrop'); syncFrequencyToUI(); } } saveToLocalStorage(); const loadedFiles = []; if (loadConfig) loadedFiles.push('config.yaml'); if (loadFrequency) loadedFiles.push('frequency_words.txt'); if (loadTimeline) loadedFiles.push('timeline.yaml'); showToast(`已加载: ${loadedFiles.join(', ')}`, 'success'); } catch (err) { console.error('加载远程配置失败:', err); showToast(`加载失败: ${err.message}`, 'error'); } } // ========================================== // 2.4 Toast 提示 // ========================================== function showToast(message, type = 'info') { // 移除已有的 toast const existingToast = document.querySelector('.toast-notification'); if (existingToast) existingToast.remove(); const toast = document.createElement('div'); toast.className = `toast-notification toast-${type}`; const icons = { success: 'fa-check-circle', error: 'fa-times-circle', info: 'fa-info-circle', warning: 'fa-exclamation-triangle' }; toast.innerHTML = ` ${message} `; document.body.appendChild(toast); // 动画入场 requestAnimationFrame(() => { toast.classList.add('show'); }); // 自动消失 setTimeout(() => { toast.classList.remove('show'); setTimeout(() => toast.remove(), 300); }, 3000); } // ========================================== // 3. 渲染逻辑 // ========================================== function renderModules() { const container = document.getElementById('config-panel'); container.innerHTML = ''; renderModuleNav(); MODULE_DEFS.forEach(mod => { const card = document.createElement('div'); card.className = `module-card ${mod.editable ? 'active' : 'disabled'}`; card.id = `module-${mod.key}`; const header = `
${mod.name}
${!mod.editable ? '只读 (请在左侧编辑)' : ''}
`; const body = mod.editable ? `
` : ''; card.innerHTML = header + body; container.appendChild(card); if (mod.editable) { renderControls(mod); } }); } // 渲染模块导航栏 function renderModuleNav() { const nav = document.getElementById('module-nav'); if (!nav) return; nav.innerHTML = MODULE_DEFS.map(mod => ` `).join(''); } // 切换组名编辑状态 window.toggleGroupNameEdit = function(btn) { const container = btn.parentNode; const span = container.querySelector('span.text-sm'); const input = container.querySelector('input[type="text"]'); if (input.classList.contains('hidden')) { // 进入编辑模式 span.classList.add('hidden'); input.classList.remove('hidden'); input.focus(); btn.innerHTML = ''; } else { // 退出编辑模式 span.classList.remove('hidden'); input.classList.add('hidden'); btn.innerHTML = ''; // 如果内容变化，已经通过 onchange 触发更新 span.textContent = input.value; } } // 跳转到左侧编辑器中对应词组的位置 window.scrollToWordGroupInEditor = function(groupIndex) { const editor = document.getElementById('frequency-editor'); // 重新解析以确保行号准确 const data = parseFrequencyText(editor.value); if (!data.wordGroups[groupIndex]) return; const targetLineIndex = data.wordGroups[groupIndex].startLine; if (targetLineIndex === undefined || targetLineIndex === -1) return; const lines = editor.value.split('\n'); const lineHeight = 19.5; const scrollPosition = targetLineIndex * lineHeight; // 设置光标选区 let charCount = 0; for (let i = 0; i < targetLineIndex; i++) { charCount += lines[i].length + 1; // +1 for newline } editor.focus(); editor.setSelectionRange(charCount, charCount + lines[targetLineIndex].length); editor.scrollTop = scrollPosition - 50; // 高亮效果 editor.style.transition = 'background-color 0.3s'; const originalBg = editor.style.backgroundColor; editor.style.backgroundColor = '#2d4a7c'; setTimeout(() => { editor.style.backgroundColor = originalBg; }, 300); } // 跳转到左侧编辑器中对应模块的位置 window.scrollToModuleInEditor = function(modKey) { const editor = document.getElementById('yaml-editor'); const yaml = editor.value; const lines = yaml.split('\n'); // 查找模块标题注释行（# N. 模块名） let targetLineIndex = -1; const mod = MODULE_DEFS.find(m => m.key === modKey); if (!mod) return; // 直接匹配包含模块编号的标题行，兼容 "4." 和 "4.5" 两种编号格式 const escapedId = String(mod.id).replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); const moduleTitlePattern = new RegExp(`^#\\s*${escapedId}(?:\\.)?\\s+`, 'i'); for (let i = 0; i < lines.length; i++) { const line = lines[i]; // 匹配模块标题行（包含编号的注释行） if (moduleTitlePattern.test(line)) { targetLineIndex = i; break; } } // 如果没找到标题行，尝试查找模块键名（如 platforms:） if (targetLineIndex === -1) { for (let i = 0; i < lines.length; i++) { if (lines[i].match(new RegExp(`^${modKey}:\\s*`))) { targetLineIndex = i; break; } } } if (targetLineIndex === -1) return; // 计算目标位置并滚动 const lineHeight = 19.5; const scrollPosition = targetLineIndex * lineHeight; // 设置光标位置 const textBeforeTarget = lines.slice(0, targetLineIndex).join('\n').length + (targetLineIndex > 0 ? 1 : 0); editor.focus(); editor.setSelectionRange(textBeforeTarget, textBeforeTarget + lines[targetLineIndex].length); editor.scrollTop = scrollPosition - 5; // 高亮提示（闪烁效果） editor.style.transition = 'background-color 0.3s'; const originalBg = editor.style.backgroundColor; editor.style.backgroundColor = '#2d4a7c'; setTimeout(() => { editor.style.backgroundColor = originalBg; }, 300); } function renderControls(mod) { const body = document.getElementById(`controls-${mod.key}`); // 根据模块 key 定义不同的 UI 控件 let html = ""; switch(mod.key) { case "platforms": html = createToggleControl(mod.key, "enabled", "启用热榜抓取"); html += `
平台列表 (可拖拽排序)
`; html += `
`; html += `
添加其它平台
`; break; case "rss": html = createToggleControl(mod.key, "enabled", "启用 RSS 抓取"); html += `
新鲜度过滤
`; html += createToggleControl(mod.key, "freshness_filter.enabled", "启用新鲜度过滤"); html += createNumberControl(mod.key, "freshness_filter.max_age_days", "最大文章年龄 (天)"); html += `
RSS 源列表
`; html += `
`; html += `
(内附 RSS 源参考库)
`; html += `
注意：部分海外媒体内容可能涉及敏感话题，AI 模型可能拒绝翻译或分析，建议根据实际需求筛选订阅源。
`; break; case "report": html = createSelectControl(mod.key, "mode", "报告模式", ["current", "daily", "incremental"]); html += createSelectControl(mod.key, "display_mode", "分组维度", ["keyword", "platform"]); html += createToggleControl(mod.key, "sort_by_position_first", "按定义顺序排序"); html += createNumberControl(mod.key, "rank_threshold", "排名高亮阈值"); html += createNumberControl(mod.key, "max_news_per_keyword", "每个关键词最大显示数量"); break; case "filter": html = createSelectControl(mod.key, "method", "筛选方法", ["keyword", "ai"]); html += createToggleControl(mod.key, "priority_sort_enabled", "AI 模式按标签优先级排序"); html += `
说明：method=keyword 使用 frequency_words.txt； method=ai 使用 ai_interests.txt + AI 筛选配置。
priority_sort_enabled 仅在 method=ai 时生效。
`; break; case "ai_filter": html = `
仅当 filter.method=ai 时生效。
`; html += createNumberControl(mod.key, "batch_size", "每批标题数量"); html += createNumberControl(mod.key, "batch_interval", "分批间隔 (秒)"); html += createNumberControl(mod.key, "min_score", "最低分数阈值 (0~1)"); html += createInputControl(mod.key, "interests_file", "兴趣描述文件 (可选)"); html += `
留空时使用 config/ai_interests.txt；填写后仅从 config/custom/ai/ 查找该文件名。
`; html += createNumberControl(mod.key, "reclassify_threshold", "全量重分类阈值 (0~1)"); html += createInputControl(mod.key, "prompt_file", "分类提示词文件"); html += createInputControl(mod.key, "extract_prompt_file", "标签提取提示词文件"); html += createInputControl(mod.key, "update_tags_prompt_file", "标签更新提示词文件"); break; case "display": html = `
推送内容控制 (可拖拽排序)
`; html += `
`; html += `
提示：列表顺序决定了报告中的显示顺序
`; // Standalone Configuration Section html += `
`; html += `
独立展示区配置 (推送展示由上方开关控制，AI 分析由 AI 模块的开关独立控制)
`; html += createNumberControl(mod.key, "standalone.max_items", "每个源最多展示条数"); html += `
选择要展示的热榜平台
`; html += `
`; html += `
选择要展示的 RSS 源
`; html += `
`; html += `
`; setTimeout(() => { renderDisplayRegionsList(); renderStandaloneLists(); }, 0); break; case "notification": html = `
推送时间由 timeline.yaml 控制，切换到 timeline.yaml 标签页可可视化编辑调度规则。
此处仅配置通知渠道（Telegram / 企业微信等），请在左侧编辑器中修改。
`; break; case "ai": html = createInputControl(mod.key, "model", "模型名称"); html += createInputControl(mod.key, "api_key", "API Key", "password"); html += createInputControl(mod.key, "api_base", "API Base URL (可选)"); html += createNumberControl(mod.key, "timeout", "请求超时 (秒)"); html += createNumberControl(mod.key, "temperature", "采样温度 (0.0-2.0)"); html += createNumberControl(mod.key, "max_tokens", "最大生成 Token 数"); break; case "ai_analysis": html = createToggleControl(mod.key, "enabled", "开启 AI 分析报告"); // 提示：分析时间窗口已迁移到 timeline.yaml html += `
AI 分析的执行时间已由 timeline.yaml 统一控制。
`; // 其他 AI 分析配置 html += `
分析内容配置
`; html += createInputControl(mod.key, "language", "输出语言"); html += createInputControl(mod.key, "prompt_file", "提示词配置文件"); html += createSelectControl(mod.key, "mode", "AI 分析模式", ["follow_report", "daily", "current", "incremental"]); html += createNumberControl(mod.key, "max_news_for_analysis", "最大分析条数"); html += createToggleControl(mod.key, "include_rss", "包含 RSS 内容"); html += createToggleControl(mod.key, "include_standalone", "包含独立展示区数据"); html += createToggleControl(mod.key, "include_rank_timeline", "传递完整排名时间线"); break; case "ai_translation": html = createToggleControl(mod.key, "enabled", "开启 AI 自动翻译"); html += createInputControl(mod.key, "language", "目标语言"); html += createInputControl(mod.key, "prompt_file", "提示词配置文件"); break; } body.innerHTML = html; // 绑定事件 body.querySelectorAll('input, select').forEach(el => { el.addEventListener('change', (e) => { updateYamlFromUI(mod.key, e.target.dataset.path, e.target); }); }); } // ========================================== // 4. 同步逻辑 (YAML -> UI) // ========================================== function syncYamlToUI() { try { const doc = jsyaml.load(currentYaml); if (!doc) return; MODULE_DEFS.filter(m => m.editable).forEach(mod => { const modData = doc[mod.key]; if (!modData) return; const controls = document.querySelectorAll(`#controls-${mod.key} [data-path]`); controls.forEach(ctrl => { const path = ctrl.dataset.path.split('.'); let val = modData; for (const part of path) { val = val ? val[part] : undefined; } if (ctrl.type === 'checkbox') { ctrl.checked = !!val; } else { ctrl.value = val !== undefined ? val : ""; } }); }); renderPlatformsList(); renderRssFeedsList(); renderStandaloneLists(); } catch (e) { // 解析失败时不更新 UI，保持原有状态 } } // ========================================== // 5. 更新逻辑 (UI -> YAML) - 核心难点：正则保留注释 // ========================================== function updateYamlFromUI(modKey, path, el) { let newVal = el.type === 'checkbox' ? el.checked : el.value; // 如果是数字类型 if (el.type === 'number') { newVal = parseFloat(newVal); if (isNaN(newVal)) newVal = 0; } const editor = document.getElementById('yaml-editor'); let yaml = editor.value; const lines = yaml.split('\n'); const pathParts = path.split('.'); // 找到模块的起始行 let moduleStartLine = -1; let moduleEndLine = lines.length; for (let i = 0; i < lines.length; i++) { const line = lines[i]; // 匹配模块开始（非缩进的 key:） const moduleMatch = line.match(/^([a-z_]+):/); if (moduleMatch) { if (moduleMatch[1] === modKey) { moduleStartLine = i; } else if (moduleStartLine >= 0) { // 找到下一个模块，记录当前模块结束位置 moduleEndLine = i; break; } } } if (moduleStartLine < 0) return; // 在模块内查找目标路径 let targetLine = -1; let currentIndent = 0; let searchKey = pathParts[pathParts.length - 1]; for (let i = moduleStartLine + 1; i < moduleEndLine; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; // 检查是否匹配目标键 const indent = line.search(/\S/); const keyMatch = line.match(/^\s*([a-z_]+):\s*(.*)/i); if (keyMatch && keyMatch[1] === searchKey) { // 如果是嵌套路径，需要检查缩进层级是否正确 if (pathParts.length > 1) { // 简化处理：对于嵌套路径，确保在正确的父级下 let valid = true; for (let j = 0; j < pathParts.length - 1; j++) { let found = false; for (let k = moduleStartLine + 1; k < i; k++) { const parentMatch = lines[k].match(/^\s*([a-z_]+):/i); if (parentMatch && parentMatch[1] === pathParts[j]) { found = true; break; } } if (!found) { valid = false; break; } } if (!valid) continue; } targetLine = i; break; } } if (targetLine < 0) { // 允许为模块新增一级字段（例如默认被注释掉的 ai_filter.interests_file） if (pathParts.length === 1) { let formattedVal = newVal; if (typeof newVal === 'string') { formattedVal = `"${newVal.replace(/"/g, '\\"')}"`; } lines.splice(moduleEndLine, 0, ` ${searchKey}: ${formattedVal}`); editor.value = lines.join('\n'); currentYaml = editor.value; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); } return; } // 更新该行，保留注释 const originalLine = lines[targetLine]; const match = originalLine.match(/^(\s*[a-z_]+:\s*)(.*)$/i); if (match) { const prefix = match[1]; const rest = match[2]; // 提取原有注释 const commentMatch = rest.match(/(\s*#.*)$/); const comment = commentMatch ? commentMatch[1] : ''; // 格式化新值 let formattedVal = newVal; if (typeof newVal === 'string') { // 获取原值部分（去除注释后的部分） const valPart = rest.slice(0, rest.length - comment.length).trim(); // 检查原值是否带有引号 const isOriginalQuoted = (valPart.startsWith('"') && valPart.endsWith('"')) || (valPart.startsWith("'") && valPart.endsWith("'")); // 如果原值有引号，或者新值包含特殊字符（空格、冒号、井号、引号）或者是空字符串，则添加双引号 if (isOriginalQuoted || newVal.includes(':') || newVal.includes('#') || newVal.includes('"') || newVal.includes(' ') || newVal === "") { formattedVal = `"${newVal.replace(/"/g, '\\"')}"`; } } // 构建新行 lines[targetLine] = `${prefix}${formattedVal}${comment}`; } // 更新编辑器 editor.value = lines.join('\n'); currentYaml = editor.value; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); } // ========================================== // 6. UI 组件工厂 // ========================================== function createToggleControl(mod, path, label) { const id = `toggle-${mod}-${path.replace('.', '-')}`; return `
`; } function createInputControl(mod, path, label, type = "text") { return `
`; } function createNumberControl(mod, path, label) { return `
`; } function createSelectControl(mod, path, label, options) { const optionsHtml = options.map(opt => ``).join(''); return `
`; } // ========================================== // 7. 工具函数 // ========================================== window.copyResult = function() { const yamlEditor = document.getElementById('yaml-editor'); const frequencyEditor = document.getElementById('frequency-editor'); const timelineEditor = document.getElementById('timeline-editor'); const editor = currentTab === 'config' ? yamlEditor : currentTab === 'timeline' ? timelineEditor : frequencyEditor; editor.select(); document.execCommand('copy'); const btn = document.querySelector('button[onclick="copyResult()"]'); const original = btn.innerHTML; btn.innerHTML = '已复制!'; setTimeout(() => btn.innerHTML = original, 2000); } window.resetToDefault = function() { if (confirm('确定要重置为初始状态吗？未保存的修改将丢失。')) { if (currentTab === 'config') { const yamlEditor = document.getElementById('yaml-editor'); yamlEditor.value = INITIAL_YAML; currentYaml = INITIAL_YAML; updateBackdrop('yaml-editor', 'yaml-backdrop'); localStorage.removeItem(STORAGE_KEY_CONFIG); localStorage.removeItem(STORAGE_KEY_CONFIG_TIME); renderModules(); syncYamlToUI(); updateSaveTimeDisplay(); } else if (currentTab === 'timeline') { const timelineEditor = document.getElementById('timeline-editor'); const initialTimeline = `# 在此粘贴你的 timeline.yaml...\n# 或拖拽文件到编辑器区域\n# 或点击右上角"加载官网最新配置"`; timelineEditor.value = initialTimeline; currentTimeline = initialTimeline; updateBackdrop('timeline-editor', 'timeline-backdrop'); localStorage.removeItem(STORAGE_KEY_TIMELINE); localStorage.removeItem(STORAGE_KEY_TIMELINE_TIME); syncTimelineToUI(); updateSaveTimeDisplay(); } else { const frequencyEditor = document.getElementById('frequency-editor'); frequencyEditor.value = "# 在此粘贴你的 frequency_words.txt 内容...\n\n[GLOBAL_FILTER]\n\n[WORD_GROUPS]\n"; currentFrequency = frequencyEditor.value; updateBackdrop('frequency-editor', 'frequency-backdrop'); localStorage.removeItem(STORAGE_KEY_FREQUENCY); localStorage.removeItem(STORAGE_KEY_FREQUENCY_TIME); syncFrequencyToUI(); updateSaveTimeDisplay(); } showToast('已重置为初始状态', 'success'); } } // ========================================== // 8. Tab 切换功能 // ========================================== window.switchTab = function(tab) { currentTab = tab; const activeClass = "tab-button active px-4 py-2 text-xs font-bold text-gray-300 hover:bg-[#2d2d30] transition-colors border-b-2 border-blue-500"; const inactiveClass = "tab-button px-4 py-2 text-xs font-bold text-gray-500 hover:bg-[#2d2d30] transition-colors border-b-2 border-transparent"; // 更新 Tab 按钮状态 const configBtn = document.getElementById('tab-config'); const freqBtn = document.getElementById('tab-frequency'); const timelineBtn = document.getElementById('tab-timeline'); configBtn.className = tab === 'config' ? activeClass : inactiveClass; freqBtn.className = tab === 'frequency' ? activeClass : inactiveClass; timelineBtn.className = tab === 'timeline' ? activeClass : inactiveClass; // 更新编辑器显示 document.getElementById('yaml-editor-wrap').classList.toggle('hidden', tab !== 'config'); document.getElementById('frequency-editor-wrap').classList.toggle('hidden', tab !== 'frequency'); document.getElementById('timeline-editor-wrap').classList.toggle('hidden', tab !== 'timeline'); // 更新右侧面板 document.getElementById('config-panel').classList.toggle('hidden', tab !== 'config'); document.getElementById('frequency-panel').classList.toggle('hidden', tab !== 'frequency'); document.getElementById('timeline-panel').classList.toggle('hidden', tab !== 'timeline'); // 更新模块导航栏显示状态：只在 config 模式下显示 const moduleNav = document.getElementById('module-nav'); if (moduleNav) { moduleNav.classList.toggle('hidden', tab !== 'config'); } // 更新保存时间显示 const saveTimeConfig = document.getElementById('save-time-config'); const saveTimeFrequency = document.getElementById('save-time-frequency'); const saveTimeTimeline = document.getElementById('save-time-timeline'); if (saveTimeConfig) saveTimeConfig.classList.toggle('hidden', tab !== 'config'); if (saveTimeFrequency) saveTimeFrequency.classList.toggle('hidden', tab !== 'frequency'); if (saveTimeTimeline) saveTimeTimeline.classList.toggle('hidden', tab !== 'timeline'); // 更新右侧标题 const versionBtn = document.getElementById('version-check-btn'); if (tab === 'config') { document.getElementById('right-panel-title').textContent = '配置模块'; if (versionBtn) { versionBtn.style.display = ''; versionBtn.title = "检测 config.yaml 版本"; } } else if (tab === 'frequency') { document.getElementById('right-panel-title').textContent = '频率词编辑'; if (versionBtn) { versionBtn.style.display = ''; versionBtn.title = "检测 frequency_words.txt 版本"; } } else { document.getElementById('right-panel-title').textContent = '时间线调度'; if (versionBtn) versionBtn.style.display = 'none'; } if (tab === 'frequency') { renderFrequencyPanel(); } if (tab === 'timeline') { syncTimelineToUI(); } } // ========================================== // 9. Frequency 编辑器功能 // ========================================== function parseFrequencyText(text) { const result = { globalFilter: [], wordGroups: [], originalText: text // 保存原始文本 }; const lines = text.split('\n'); let currentSection = null; let currentGroup = null; let lastLineWasAlias = false; // 追踪上一行是否为别名行 let relatedGroupsBuffer = []; // 缓存连续的相关组 let pendingComments = []; // 缓存待分配的注释行 // 辅助函数：保存缓存的相关组 function flushRelatedGroups() { if (relatedGroupsBuffer.length > 0) { // 如果有多个连续的组，标记它们为相关组 if (relatedGroupsBuffer.length > 1) { relatedGroupsBuffer.forEach((group, idx) => { group.isRelatedGroup = true; group.relatedGroupIndex = idx; group.relatedGroupTotal = relatedGroupsBuffer.length; }); } result.wordGroups.push(...relatedGroupsBuffer); relatedGroupsBuffer = []; } } for (let i = 0; i < lines.length; i++) { const line = lines[i]; const trimmed = line.trim(); // 收集注释行（在 [WORD_GROUPS] 区域内） if (trimmed.startsWith('#') && currentSection === 'groups') { pendingComments.push(line); continue; } // 跳过注释（非 [WORD_GROUPS] 区域） if (trimmed.startsWith('#')) continue; // 空行：结束当前词组和相关组缓存 if (!trimmed) { if (currentGroup) { // 保存当前词组到缓存 relatedGroupsBuffer.push(currentGroup); currentGroup = null; } // 空行表示相关组结束，刷新缓存 flushRelatedGroups(); lastLineWasAlias = false; // 在 [WORD_GROUPS] 区域内，空行加入待分配注释（保留空行结构） if (currentSection === 'groups') { pendingComments.push(''); } continue; } // 检测区域标记 if (trimmed === '[GLOBAL_FILTER]') { currentSection = 'global'; continue; } if (trimmed === '[WORD_GROUPS]') { currentSection = 'groups'; continue; } // 处理内容 if (currentSection === 'global') { result.globalFilter.push(trimmed); } else if (currentSection === 'groups') { // 检测组别名 [组名] const groupNameMatch = trimmed.match(/^\[([^\]]+)\]$/); if (groupNameMatch && !['GLOBAL_FILTER', 'WORD_GROUPS'].includes(groupNameMatch[1])) { // 保存当前词组到缓存 if (currentGroup) { relatedGroupsBuffer.push(currentGroup); } // 刷新缓存（组别名独立成组） flushRelatedGroups(); // 创建组别名类型 currentGroup = { type: 'group-name', name: groupNameMatch[1], keywords: [], startLine: i, precedingComments: pendingComments.length > 0 ? [...pendingComments] : [] }; pendingComments = []; lastLineWasAlias = false; } else { // 检测 => 别名语法（允许右侧为空） const aliasMatch = trimmed.match(/^(.+?)\s*=>\s*(.*)$/); if (aliasMatch) { const keyword = aliasMatch[1].trim(); const alias = aliasMatch[2].trim(); // 关键逻辑：如果上一行也是别名行（无空行分隔），则归入连续别名组 if (lastLineWasAlias && currentGroup && (currentGroup.type === 'alias' || currentGroup.type === 'alias-group')) { // 如果当前是单个别名，升级为别名组 if (currentGroup.type === 'alias') { currentGroup.type = 'alias-group'; } // 添加到别名组 currentGroup.items.push({ keyword, alias }); } else { // 新的单个别名（可能会升级为别名组） if (currentGroup) { // 保存当前词组到缓存（而不是直接添加到结果） relatedGroupsBuffer.push(currentGroup); } currentGroup = { type: 'alias', items: [{ keyword, alias }], startLine: i, precedingComments: pendingComments.length > 0 ? [...pendingComments] : [] }; pendingComments = []; } lastLineWasAlias = true; } else { // 普通关键词 if (!currentGroup || currentGroup.type === 'alias' || currentGroup.type === 'alias-group') { // 如果当前是别名类型，需要先保存到缓存 if (currentGroup) { relatedGroupsBuffer.push(currentGroup); } // 创建新的普通词组 currentGroup = { type: 'plain', keywords: [], startLine: i, precedingComments: pendingComments.length > 0 ? [...pendingComments] : [] }; pendingComments = []; } currentGroup.keywords.push(trimmed); lastLineWasAlias = false; } } } } // 添加最后一个组 if (currentGroup) { relatedGroupsBuffer.push(currentGroup); } flushRelatedGroups(); return result; } function buildFrequencyText(data) { // 如果有原始文本，尝试保留注释 if (data.originalText) { const lines = data.originalText.split('\n'); let result = []; // 第一步：保留文件头部的注释 let i = 0; while (i < lines.length) { const line = lines[i]; const trimmed = line.trim(); if (trimmed === '[GLOBAL_FILTER]') { break; } result.push(line); i++; } // 第二步：重建 [GLOBAL_FILTER] 区域 result.push('[GLOBAL_FILTER]'); // 保留 [GLOBAL_FILTER] 后面的注释（直到第一个非注释非空行） i++; while (i < lines.length) { const line = lines[i]; const trimmed = line.trim(); if (trimmed.startsWith('#') || trimmed === '') { result.push(line); i++; } else { break; } } // 添加全局过滤词 data.globalFilter.forEach(filter => { result.push(filter); }); // 跳过原始文件中的 [GLOBAL_FILTER] 内容（非注释行），保留空行和注释直到 [WORD_GROUPS] while (i < lines.length) { const line = lines[i]; const trimmed = line.trim(); if (trimmed === '[WORD_GROUPS]') { break; } // 保留注释和空行 if (trimmed.startsWith('#') || trimmed === '') { result.push(line); } i++; } // 第三步：重建 [WORD_GROUPS] 区域 result.push('[WORD_GROUPS]'); // 添加词组（注释已保存在每个词组的 precedingComments 中） data.wordGroups.forEach((group, index) => { // 先输出词组前的注释 if (group.precedingComments && group.precedingComments.length > 0) { group.precedingComments.forEach(comment => { result.push(comment); }); } if (group.type === 'group-name') { // 组别名类型：[组名] + 关键词 if (group.name) { result.push(`[${group.name}]`); } group.keywords.forEach(kw => { result.push(kw); }); } else if (group.type === 'alias' || group.type === 'alias-group') { // 别名类型：keyword => alias group.items.forEach(item => { result.push(`${item.keyword} => ${item.alias}`); }); } else if (group.type === 'plain') { // 普通词组 group.keywords.forEach(kw => { result.push(kw); }); } // 空行处理逻辑： // 1. 如果当前词组和下一个词组都是相关组，则不添加空行 // 2. 否则，在词组之间添加空行 const isLastGroup = index === data.wordGroups.length - 1; const nextGroup = !isLastGroup ? data.wordGroups[index + 1] : null; // 简化判断：只要当前和下一个都是相关组，就不添加空行 const bothAreRelatedGroups = group.isRelatedGroup && nextGroup && nextGroup.isRelatedGroup; // 如果下一个词组有前置注释，不需要额外添加空行（注释中已包含空行） const nextHasComments = nextGroup && nextGroup.precedingComments && nextGroup.precedingComments.length > 0; if (bothAreRelatedGroups) { // 相关组内部不添加空行 // 不添加任何内容 } else if (!isLastGroup && !nextHasComments) { // 词组之间添加空行（如果下一个没有前置注释） result.push(''); } else if (isLastGroup) { // 最后一个词组后也保留一个空行 result.push(''); } }); return result.join('\n'); } // 如果没有原始文本，使用默认模板 let text = '# ═══════════════════════════════════════════════════════════════\n'; text += '# TrendRadar 频率词配置文件\n'; text += '# ═══════════════════════════════════════════════════════════════\n\n'; text += '[GLOBAL_FILTER]\n'; data.globalFilter.forEach(filter => { text += filter + '\n'; }); text += '\n\n'; text += '[WORD_GROUPS]\n\n'; data.wordGroups.forEach((group, index) => { // 先输出词组前的注释 if (group.precedingComments && group.precedingComments.length > 0) { group.precedingComments.forEach(comment => { text += comment + '\n'; }); } if (group.type === 'group-name') { if (group.name) { text += `[${group.name}]\n`; } group.keywords.forEach(kw => { text += kw + '\n'; }); } else if (group.type === 'alias' || group.type === 'alias-group') { group.items.forEach(item => { text += `${item.keyword} => ${item.alias}\n`; }); } else if (group.type === 'plain') { group.keywords.forEach(kw => { text += kw + '\n'; }); } // 空行处理逻辑：与上面保持一致 const isLastGroup = index === data.wordGroups.length - 1; const nextGroup = !isLastGroup ? data.wordGroups[index + 1] : null; const bothAreRelatedGroups = group.isRelatedGroup && nextGroup && nextGroup.isRelatedGroup; // 如果下一个词组有前置注释，不需要额外添加空行 const nextHasComments = nextGroup && nextGroup.precedingComments && nextGroup.precedingComments.length > 0; if (bothAreRelatedGroups) { // 相关组内部不添加空行 } else if (!isLastGroup && !nextHasComments) { text += '\n'; // 词组之间用空行分隔 } else if (isLastGroup) { text += '\n'; // 最后一个词组后也保留一个空行 } }); return text; } function syncFrequencyToUI() { const data = parseFrequencyText(currentFrequency); currentFrequencyData = data; renderFrequencyPanel(data); } function renderFrequencyPanel(data) { if (!data) { data = parseFrequencyText(currentFrequency); } const panel = document.getElementById('frequency-panel'); // 辅助函数：根据关键词类型返回样式类 function getKeywordClass(keyword) { if (keyword.startsWith('+')) return 'bg-green-500'; if (keyword.startsWith('!')) return 'bg-red-500'; if (keyword.startsWith('@')) return 'bg-purple-500'; if (keyword.startsWith('/') || keyword.includes('=>')) return 'bg-indigo-500'; return 'bg-blue-500'; } // 辅助函数：为关键词添加标签 function getKeywordLabel(keyword) { if (keyword.startsWith('+')) return '必须'; if (keyword.startsWith('!')) return '排除'; if (keyword.startsWith('@')) return '限制'; if (keyword.startsWith('/')) return '正则'; if (keyword.includes('=>')) return '别名'; return ''; } // 渲染词组卡片 function renderGroupCard(group, idx) { const jumpIcon = ``; // 序号标记 const indexBadge = `#${idx + 1}`; // 相关组标记 const relatedGroupBadge = group.isRelatedGroup ? ` 相关组 ${group.relatedGroupIndex + 1}/${group.relatedGroupTotal} ` : ''; // 相关组边框样式 const relatedGroupStyle = group.isRelatedGroup ? 'border-l-4 border-l-blue-500 shadow-lg' : ''; if (group.type === 'group-name') { // 组别名类型 return `
${jumpIcon} ${indexBadge} 组别名 ${relatedGroupBadge}
关键词列表：
${group.keywords.map(kw => { const label = getKeywordLabel(kw); const escapedKw = kw.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); return ` ${label ? `[${label}]` : ''} ${escapedKw} `; }).join('')}
${group.keywords.length} 个关键词
`; } else if (group.type === 'alias') { // 单个别名类型 const item = group.items[0]; return `
${jumpIcon} ${indexBadge} 单个别名 ${relatedGroupBadge}
=>
示例：/胖东来|于东来/ => 胖东来
`; } else if (group.type === 'alias-group') { // 连续别名组类型 return `
${jumpIcon} ${indexBadge} 连续别名组 ${relatedGroupBadge}
别名列表（无空行分隔）：
${group.items.map((item, itemIdx) => `
=>
`).join('')}
这些别名行在配置文件中无空行分隔，属于同一组
`; } else if (group.type === 'plain') { // 普通词组类型 return `
${jumpIcon} ${indexBadge} 普通词组 ${relatedGroupBadge}
${group.keywords.map(kw => { const label = getKeywordLabel(kw); const escapedKw = kw.replace(/&/g, '&').replace(//g, '>').replace(/"/g, '"'); return ` ${label ? `[${label}]` : ''} ${escapedKw} `; }).join('')}
${group.keywords.length} 个关键词
`; } return ''; } panel.innerHTML = `

四种词组类型说明

组别名
[东亚]
日本
韩国
多个关键词，统一显示为组名
单个别名
/胖东来|于东来/ => 胖东来
正则匹配，显示为别名
连续别名组
/智元|稚晖君/ => 智元
/众擎|EngineAI/ => 众擎
多个别名无空行分隔
普通词组
申奥
普通关键词

全局过滤词

${data.globalFilter.map(f => ` ${f} `).join('')}
提示：支持正则表达式（用 /.../ 包裹）

关键词组 (${data.wordGroups.length} 个词组)

${data.wordGroups.map((group, idx) => { const card = renderGroupCard(group, idx); // 在每个词组后添加插入区域（最后一个除外） if (idx < data.wordGroups.length - 1) { return card + `
`; } return card; }).join('')}
`; // 初始化拖拽排序功能 setTimeout(() => { const container = document.getElementById('word-groups-container'); if (container && typeof Sortable !== 'undefined') { // 销毁之前的实例（如果存在） if (container.sortableInstance) { container.sortableInstance.destroy(); } // 创建新的 Sortable 实例 container.sortableInstance = new Sortable(container, { animation: 150, filter: '.editable-area, input, button, select, textarea', // 排除编辑区域 preventOnFilter: false, // 允许在过滤区域正常交互 ghostClass: 'sortable-ghost', chosenClass: 'sortable-chosen', dragClass: 'sortable-drag', onEnd: function(evt) { // 获取所有词组卡片的当前顺序 const cards = Array.from(container.querySelectorAll('.word-group-card')); const newOrder = cards.map(card => parseInt(card.getAttribute('data-group-index'))); // 检查顺序是否改变 const data = currentFrequencyData || parseFrequencyText(currentFrequency); const oldOrder = data.wordGroups.map((_, idx) => idx); if (JSON.stringify(newOrder) !== JSON.stringify(oldOrder)) { // 根据新顺序重新排列数据 const reorderedGroups = newOrder.map(idx => data.wordGroups[idx]); data.wordGroups = reorderedGroups; // 重新构建文本 currentFrequency = buildFrequencyText(data); currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); // 重新渲染 renderFrequencyPanel(currentFrequencyData); } } }); } }, 0); } // Global Filter 操作 window.handleGlobalFilterInput = function(event) { if (event.key === 'Enter' && event.target.value.trim()) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); data.globalFilter.push(event.target.value.trim()); currentFrequency = buildFrequencyText(data); currentFrequencyData = data; document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(data); } } window.removeGlobalFilter = function(filter) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); data.globalFilter = data.globalFilter.filter(f => f !== filter); currentFrequency = buildFrequencyText(data); currentFrequencyData = data; document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(data); } // Word Groups 操作 let pendingWordGroupPosition = 'top'; // 记录添加位置：'top', 'bottom', 或数字索引 window.addWordGroup = function(position = 'top') { pendingWordGroupPosition = position; document.getElementById('wordgroup-type-modal').classList.remove('hidden'); } // 在指定位置插入词组 window.insertWordGroupAt = function(index) { pendingWordGroupPosition = index; // 记录插入位置（数字索引） document.getElementById('wordgroup-type-modal').classList.remove('hidden'); } window.closeWordGroupTypeModal = function() { document.getElementById('wordgroup-type-modal').classList.add('hidden'); } window.confirmAddWordGroup = function(type) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); let newGroup; if (type === 'group') { // 组别名类型 newGroup = { type: 'group-name', name: '', keywords: [] }; } else if (type === 'alias') { // 单个别名类型 newGroup = { type: 'alias', items: [{ keyword: '', alias: '' }] }; } else if (type === 'multi-alias') { // 连续别名类型（多个别名行） newGroup = { type: 'alias-group', items: [{ keyword: '', alias: '' }, { keyword: '', alias: '' }] }; } else if (type === 'plain') { // 普通词组类型 newGroup = { type: 'plain', keywords: [] }; } // 根据位置插入 if (pendingWordGroupPosition === 'bottom') { data.wordGroups.push(newGroup); } else if (pendingWordGroupPosition === 'top') { data.wordGroups.unshift(newGroup); } else if (typeof pendingWordGroupPosition === 'number') { // 在指定索引位置插入 data.wordGroups.splice(pendingWordGroupPosition, 0, newGroup); } currentFrequency = buildFrequencyText(data); currentFrequencyData = data; document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(data); closeWordGroupTypeModal(); // 滚动到新添加的词组 setTimeout(() => { const container = document.getElementById('word-groups-container'); if (pendingWordGroupPosition === 'bottom') { container.scrollTop = container.scrollHeight; } else if (pendingWordGroupPosition === 'top') { container.scrollTop = 0; } else if (typeof pendingWordGroupPosition === 'number') { // 滚动到插入的位置 const cards = container.querySelectorAll('.word-group-card'); if (cards[pendingWordGroupPosition]) { cards[pendingWordGroupPosition].scrollIntoView({ behavior: 'smooth', block: 'center' }); } } }, 100); } window.removeWordGroup = function(index) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); data.wordGroups.splice(index, 1); currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } window.updateGroupName = function(index, name) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[index]; // 只有 group-name 类型才有 name 字段 if (group.type === 'group-name') { group.name = name; } currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } window.editKeyword = function(groupIndex, oldKeyword, spanElement) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 group-name 和 plain 类型才有 keywords 字段 if (group.type !== 'group-name' && group.type !== 'plain') { return; } const originalKeyword = group.keywords.find(kw => kw === oldKeyword) || oldKeyword; const input = document.createElement('input'); input.type = 'text'; input.value = originalKeyword; input.className = 'tag-input inline-block px-2 py-1 text-xs border border-blue-500 rounded'; input.style.minWidth = '100px'; const saveEdit = () => { const newKeyword = input.value.trim(); if (newKeyword && newKeyword !== originalKeyword) { const kwIndex = group.keywords.indexOf(originalKeyword); if (kwIndex !== -1) { group.keywords[kwIndex] = newKeyword; } currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } else { spanElement.style.display = ''; input.remove(); } }; input.onblur = saveEdit; input.onkeydown = (e) => { if (e.key === 'Enter') { saveEdit(); } else if (e.key === 'Escape') { spanElement.style.display = ''; input.remove(); } }; spanElement.style.display = 'none'; spanElement.parentNode.insertBefore(input, spanElement); input.focus(); input.select(); } window.handleKeywordInput = function(event, groupIndex) { if (event.key === 'Enter' && event.target.value.trim()) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 group-name 和 plain 类型才能添加关键词 if (group.type === 'group-name' || group.type === 'plain') { group.keywords.push(event.target.value.trim()); event.target.value = ''; currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } } } window.removeKeyword = function(groupIndex, keyword) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 group-name 和 plain 类型才能删除关键词 if (group.type === 'group-name' || group.type === 'plain') { group.keywords = group.keywords.filter(k => k !== keyword); // 如果词组变空，删除整个词组 if (group.keywords.length === 0) { data.wordGroups.splice(groupIndex, 1); } currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } } // 更新别名项 window.updateAliasItem = function(groupIndex, itemIndex, field, value) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 alias 和 alias-group 类型才有 items 字段 if (group.type === 'alias' || group.type === 'alias-group') { if (group.items[itemIndex]) { group.items[itemIndex][field] = value; currentFrequency = buildFrequencyText(data); currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } } } // 添加别名项 window.addAliasItem = function(groupIndex) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 alias-group 类型才能添加别名项 if (group.type === 'alias-group') { group.items.push({ keyword: '', alias: '' }); currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } else if (group.type === 'alias') { // 如果是单个别名，升级为别名组 group.type = 'alias-group'; group.items.push({ keyword: '', alias: '' }); currentFrequency = buildFrequencyText(data); // 重新解析以更新相关组信息 currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } } // 删除别名项 window.removeAliasItem = function(groupIndex, itemIndex) { const data = currentFrequencyData || parseFrequencyText(currentFrequency); const group = data.wordGroups[groupIndex]; // 只有 alias-group 类型才能删除别名项 if (group.type === 'alias-group') { group.items.splice(itemIndex, 1); // 如果没有别名项了，删除整个词组 if (group.items.length === 0) { data.wordGroups.splice(groupIndex, 1); } // 如果只剩一个别名项，降级为单个别名 else if (group.items.length === 1) { group.type = 'alias'; } currentFrequency = buildFrequencyText(data); currentFrequencyData = parseFrequencyText(currentFrequency); document.getElementById('frequency-editor').value = currentFrequency; updateBackdrop('frequency-editor', 'frequency-backdrop'); renderFrequencyPanel(currentFrequencyData); } } // DeepSeek AI 辅助 window.openDeepSeekAI = function(type, groupIndex) { const userInput = window.prompt('请输入核心关键词（例如：华为）：'); if (!userInput) return; const promptText = `我正在配置一个新闻聚合系统，需要通过 Python 正则表达式 抓取关于【${userInput}】的新闻。 请帮我完成以下步骤，并最终只输出一个正则表达式字符串： 第一步：【精准关键词筛选】 请列出与【${userInput}】强绑定的核心词汇： 1. 核心品牌：包括中文全称、简称、股票代码、别名。 2. 核心人物：仅限最高决策层或极具代表性的创始人。 3. 独家产品：必须是具有极高辨识度的独家产品名。 4. 核心工作室/子品牌：强相关的下属机构。 第二步：【严格清洗与过滤】（请严格执行） 1. 包含关系去重（最短匹配原则）： - 中文：如果列表里已经有了核心短词（如“腾讯”），请删除所有包含该短词的长词（如“腾讯云”、“腾讯视频”统统不要，因为它们会被短词命中）。 - 英文：如果有了 \\bKeyword\\b，就不要再出现 Keyword。 2. 彻底排除无关公司： - 绝对不要包含：该品牌的竞争对手、合作伙伴（如京东、美团、字节跳动等非隶属公司）。 3. 彻底排除通用黑话： - 绝对不要包含：行业通用词（如“互联网”、“大厂”、“新质生产力”、“人工智能”、“元宇宙”、“金融科技”等）。 第三步：【构建 Python 正则】 将清洗后的词汇合并，格式要求如下： 1. 英文处理：所有英文单词必须前后加 \\b（例如 \\bWord\\b），严禁出现没有边界符的英文单词。 2. 连接符：用 | 连接。 最终输出示例格式： /词A|词B|\\bEnglishWord\\b/ => ${userInput} 输出要求： - 只要这一行正则表达式，不要任何解释，不要代码块。`; const textArea = document.createElement("textarea"); textArea.value = promptText; textArea.style.position = "fixed"; textArea.style.left = "-9999px"; textArea.style.top = "0"; document.body.appendChild(textArea); textArea.focus(); textArea.select(); let copySuccess = false; try { copySuccess = document.execCommand('copy'); } catch (err) { console.error('复制失败:', err); } document.body.removeChild(textArea); if (copySuccess) { if (confirm(`提示词已复制到剪贴板！\n\n关键词：${userInput}\n\n点击【确定】跳转 DeepSeek 官网，直接粘贴 (Ctrl+V) 即可。`)) { window.open('https://chat.deepseek.com/', '_blank'); } } else { prompt('自动复制失败，请手动复制以下内容，然后自行打开 DeepSeek:', promptText); window.open('https://chat.deepseek.com/', '_blank'); } } // ========================================== // 10. 平台管理功能 // ========================================== // 解析当前配置中的平台列表 function parsePlatformsFromYaml() { try { const doc = jsyaml.load(currentYaml); if (doc && doc.platforms && doc.platforms.sources) { return doc.platforms.sources; } } catch (e) {} return []; } // 渲染平台列表 function renderPlatformsList() { const container = document.getElementById('platforms-list'); if (!container) return; const platforms = parsePlatformsFromYaml(); if (platforms.length === 0) { container.innerHTML = `
暂无平台，请添加
`; return; } container.innerHTML = platforms.map((p, idx) => `
${p.name} (${p.id})
`).join(''); // 初始化拖拽排序 if (typeof Sortable !== 'undefined') { new Sortable(container, { animation: 150, handle: '.fa-grip-vertical', onEnd: function(evt) { reorderPlatforms(evt.oldIndex, evt.newIndex); } }); } } // 删除平台 window.removePlatform = function(index) { const platforms = parsePlatformsFromYaml(); if (index < 0 || index >= platforms.length) return; const platformName = platforms[index].name; if (!confirm(`确定要删除平台 "${platformName}" 吗？`)) return; platforms.splice(index, 1); updatePlatformsInYaml(platforms); } // 重新排序平台 function reorderPlatforms(oldIndex, newIndex) { const platforms = parsePlatformsFromYaml(); const [removed] = platforms.splice(oldIndex, 1); platforms.splice(newIndex, 0, removed); updatePlatformsInYaml(platforms); } // 更新 YAML 中的平台配置（保留注释） function updatePlatformsInYaml(platforms) { const editor = document.getElementById('yaml-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 找到 platforms.sources 的位置 let sourcesStart = -1; let sourcesEnd = -1; let inPlatforms = false; let inSources = false; let baseIndent = 0; let lastDataLineIndex = -1; // 记录最后一个数据行的位置 for (let i = 0; i < lines.length; i++) { const line = lines[i]; const trimmed = line.trim(); if (line.match(/^platforms:/)) { inPlatforms = true; continue; } if (inPlatforms && !inSources && trimmed.startsWith('sources:')) { sourcesStart = i + 1; inSources = true; baseIndent = line.search(/\S/) + 2; // sources 下一级的缩进 continue; } if (inSources) { const currentIndent = line.search(/\S/); // 如果是数据行（以 - 开头或是数据项的属性） if (trimmed.startsWith('-')) { lastDataLineIndex = i; } else if (trimmed && !trimmed.startsWith('#') && currentIndent >= baseIndent) { // 数据项的属性行（如 name:, id:） lastDataLineIndex = i; } else if (trimmed && !trimmed.startsWith('#') && currentIndent < baseIndent) { // 遇到缩进更小的非注释行，说明离开了 sources 区域 sourcesEnd = lastDataLineIndex + 1; break; } } // 检查是否进入下一个顶级模块 if (inPlatforms && line.match(/^[a-z_]+:/) && !line.match(/^platforms:/)) { if (lastDataLineIndex >= 0) { sourcesEnd = lastDataLineIndex + 1; } else { sourcesEnd = i; } break; } } // 如果没有找到结束位置，使用最后一个数据行的下一行 if (sourcesEnd === -1) { sourcesEnd = lastDataLineIndex >= 0 ? lastDataLineIndex + 1 : lines.length; } // 提取区域内的注释（保留在开头的注释） const regionLines = lines.slice(sourcesStart, sourcesEnd); const leadingComments = []; for (const line of regionLines) { const trimmed = line.trim(); if (trimmed.startsWith('#')) { leadingComments.push(line); } else if (trimmed.startsWith('-') || (trimmed && !trimmed.startsWith('#'))) { // 遇到第一个数据项，停止收集注释 break; } else if (trimmed === '') { // 空行也保留 leadingComments.push(line); } } const indent = ' '; // 4 空格缩进 const newSourcesLines = platforms.map(p => `${indent}- id: "${p.id}"\n${indent} name: "${p.name}"` ).join('\n'); const beforeSources = lines.slice(0, sourcesStart); const afterSources = lines.slice(sourcesEnd); // 组合：前面内容 + 开头注释 + 新数据 + 后面内容 const newYaml = [ ...beforeSources, ...(leadingComments.length > 0 ? leadingComments : []), newSourcesLines, ...afterSources ].join('\n'); editor.value = newYaml; currentYaml = newYaml; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); renderPlatformsList(); renderStandaloneLists(); // 同步更新独立展示区的平台选择列表 } // ========================================== // 12. Display Regions 排序与管理功能 // ========================================== const DISPLAY_REGIONS_DEF = [ { key: "hotlist", label: "热榜区域" }, { key: "new_items", label: "新增热点区域" }, { key: "rss", label: "RSS 订阅区域" }, { key: "standalone", label: "独立展示区" }, { key: "ai_analysis", label: "AI 分析区域" } ]; // 从 YAML 解析 display.regions，严格按照 region_order 定义顺序 function parseDisplayRegionsFromYaml() { try { const doc = jsyaml.load(currentYaml); if (doc && doc.display) { const regionOrder = doc.display.region_order || []; const regionStates = doc.display.regions || {}; // 严格按 region_order 顺序构建列表 if (regionOrder.length > 0) { return regionOrder.map(key => { const normalizedKey = key === 'new_item' ? 'new_items' : key; const def = DISPLAY_REGIONS_DEF.find(d => d.key === normalizedKey); return { key: normalizedKey, label: def ? def.label : normalizedKey, enabled: regionStates[normalizedKey] !== undefined ? regionStates[normalizedKey] : false }; }); } // 后备方案：如果没有 region_order，使用 regions 对象的顺序 const regions = []; for (const key in regionStates) { const normalizedKey = key === 'new_item' ? 'new_items' : key; const def = DISPLAY_REGIONS_DEF.find(d => d.key === normalizedKey); if (def) { regions.push({ key: normalizedKey, label: def.label, enabled: regionStates[key] }); } } return regions; } } catch (e) {} // 默认返回所有区域（禁用状态） return DISPLAY_REGIONS_DEF.map(def => ({ key: def.key, label: def.label, enabled: false })); } // 渲染 Display Regions 列表 function renderDisplayRegionsList() { const container = document.getElementById('display-regions-list'); if (!container) return; const regions = parseDisplayRegionsFromYaml(); container.innerHTML = regions.map((r, idx) => `
${r.label} (${r.key})
`).join(''); // 初始化拖拽排序 if (typeof Sortable !== 'undefined') { new Sortable(container, { animation: 150, handle: '.fa-grip-vertical', onEnd: function(evt) { reorderDisplayRegions(); } }); } } // 切换区域启用状态 window.toggleDisplayRegion = function(key) { const regions = parseDisplayRegionsFromYaml(); const target = regions.find(r => r.key === key); if (target) { target.enabled = !target.enabled; updateDisplayRegionsInYaml(regions); } } // 重新排序区域 window.reorderDisplayRegions = function() { const container = document.getElementById('display-regions-list'); const items = container.querySelectorAll('.display-region-item'); const newOrderKeys = Array.from(items).map(item => item.dataset.key); const currentRegions = parseDisplayRegionsFromYaml(); const newRegions = newOrderKeys.map(key => { return currentRegions.find(r => r.key === key); }).filter(r => r); // 过滤掉可能的 undefined updateDisplayRegionsInYaml(newRegions); } // 更新 YAML 中的 display.regions 和 display.region_order function updateDisplayRegionsInYaml(regions) { const editor = document.getElementById('yaml-editor'); let yaml = editor.value; const lines = yaml.split('\n'); let regionOrderStart = -1; let regionOrderEnd = -1; let regionsStart = -1; let regionsEnd = -1; let inDisplay = false; let regionOrderIndent = 0; let regionsIndent = 0; for (let i = 0; i < lines.length; i++) { const line = lines[i]; const trimmed = line.trim(); if (line.match(/^display:/)) { inDisplay = true; continue; } if (!inDisplay) continue; // 查找 region_order 数组 if (trimmed.startsWith('region_order:')) { regionOrderStart = i + 1; regionOrderIndent = line.search(/\S/) + 2; // 找到 region_order 的结束位置 for (let j = i + 1; j < lines.length; j++) { const nextLine = lines[j]; const nextTrimmed = nextLine.trim(); if (nextTrimmed && !nextTrimmed.startsWith('#') && !nextTrimmed.startsWith('-')) { const nextIndent = nextLine.search(/\S/); if (nextIndent < regionOrderIndent) { regionOrderEnd = j; break; } } } if (regionOrderEnd === -1) regionOrderEnd = lines.length; continue; } // 查找 regions 对象 if (trimmed.startsWith('regions:')) { regionsStart = i + 1; regionsIndent = line.search(/\S/) + 2; // 找到 regions 的结束位置（遇到同级或更高级的键） for (let j = i + 1; j < lines.length; j++) { const nextLine = lines[j]; const nextTrimmed = nextLine.trim(); if (nextTrimmed && !nextTrimmed.startsWith('#')) { const nextIndent = nextLine.search(/\S/); // 检查是否是同级或更高级的键（如 standalone:） if (nextIndent <= line.search(/\S/)) { regionsEnd = j; break; } } } if (regionsEnd === -1) regionsEnd = lines.length; break; } // 检查是否离开 display 模块 if (line.match(/^[a-z_]+:/) && !line.match(/^display:/)) { break; } } // 更新 region_order 数组（保留注释） if (regionOrderStart > 0 && regionOrderEnd > regionOrderStart) { const indentStr = ' '.repeat(regionOrderIndent); // 提取原有行的注释映射 const originalRegionOrderBlock = lines.slice(regionOrderStart, regionOrderEnd); const commentMap = {}; originalRegionOrderBlock.forEach(line => { // 匹配 "- key # 注释" 格式 const match = line.match(/^\s*-\s*([a-z_]+)\s*(#.*)?$/); if (match) { const key = match[1]; const comment = match[2] || ''; if (key) commentMap[key] = comment; } }); // 生成新的行，保留注释 const newRegionOrderLines = regions.map(r => { const comment = commentMap[r.key] || ''; return `${indentStr}- ${r.key}${comment ? ' ' + comment : ''}`; }); lines.splice(regionOrderStart, regionOrderEnd - regionOrderStart, ...newRegionOrderLines); // 调整 regionsStart 和 regionsEnd const lineDiff = newRegionOrderLines.length - (regionOrderEnd - regionOrderStart); if (regionsStart > regionOrderEnd) { regionsStart += lineDiff; regionsEnd += lineDiff; } } // 更新 regions 对象 if (regionsStart > 0 && regionsEnd > regionsStart) { const originalRegionsBlock = lines.slice(regionsStart, regionsEnd); const commentMap = {}; originalRegionsBlock.forEach(line => { const match = line.match(/^\s*([a-z_]+):\s*[^#]*(#.*)?$/); if (match) { const key = match[1]; const comment = match[2] || ''; if (key) commentMap[key] = comment; } }); const indentStr = ' '.repeat(regionsIndent); const newRegionsLines = regions.map(r => { const comment = commentMap[r.key] || ''; return `${indentStr}${r.key}: ${r.enabled}${comment ? ' ' + comment.trim() : ''}`; }); lines.splice(regionsStart, regionsEnd - regionsStart, ...newRegionsLines); } editor.value = lines.join('\n'); currentYaml = lines.join('\n'); updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); renderDisplayRegionsList(); } // 解析当前配置中的 RSS 源列表 function parseRssFeedsFromYaml() { try { const doc = jsyaml.load(currentYaml); if (doc && doc.rss && doc.rss.feeds) { return doc.rss.feeds; } } catch (e) {} return []; } // 渲染 RSS 源列表 function renderRssFeedsList() { const container = document.getElementById('rss-feeds-list'); if (!container) return; const feeds = parseRssFeedsFromYaml(); if (feeds.length === 0) { container.innerHTML = `
暂无 RSS 源，请添加
`; return; } container.innerHTML = feeds.map((f, idx) => `
${f.name} (${f.id}) ${f.enabled === false ? '已禁用' : ''}
${f.url}
`).join(''); } // 删除 RSS 源 window.removeRssFeed = function(index) { const feeds = parseRssFeedsFromYaml(); if (index < 0 || index >= feeds.length) return; const feedName = feeds[index].name; if (!confirm(`确定要删除 RSS 源 "${feedName}" 吗？`)) return; feeds.splice(index, 1); updateRssFeedsInYaml(feeds); } // 切换 RSS 源启用状态 window.toggleRssFeed = function(index) { const feeds = parseRssFeedsFromYaml(); if (index < 0 || index >= feeds.length) return; feeds[index].enabled = feeds[index].enabled === false ? true : false; updateRssFeedsInYaml(feeds); } // 编辑 RSS 源 window.editRssFeed = function(index) { const feeds = parseRssFeedsFromYaml(); if (index < 0 || index >= feeds.length) return; const feed = feeds[index]; openRssModalWithData(feed, index); } // 更新 YAML 中的 RSS 配置（保留注释） function updateRssFeedsInYaml(feeds) { const editor = document.getElementById('yaml-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 找到 rss.feeds 的位置 let feedsStart = -1; let feedsEnd = -1; let inRss = false; let inFeeds = false; let lastDataLineIndex = -1; // 记录最后一个数据行的位置 for (let i = 0; i < lines.length; i++) { const line = lines[i]; const trimmed = line.trim(); if (line.match(/^rss:/)) { inRss = true; continue; } if (inRss && !inFeeds && trimmed.startsWith('feeds:')) { feedsStart = i + 1; inFeeds = true; continue; } if (inFeeds) { const indent = line.search(/\S/); // 如果是数据行（以 - 开头或是数据项的属性） if (trimmed.startsWith('-')) { lastDataLineIndex = i; } else if (trimmed && !trimmed.startsWith('#') && indent > 2) { // 数据项的属性行（如 name:, id:, url:） lastDataLineIndex = i; } else if (trimmed && !trimmed.startsWith('#') && indent <= 2 && indent >= 0) { // 遇到缩进更小的非注释行，说明离开了 feeds 区域 feedsEnd = lastDataLineIndex + 1; break; } } // 检查是否进入下一个顶级模块 if (inRss && line.match(/^[a-z_]+:/) && !line.match(/^rss:/)) { if (lastDataLineIndex >= 0) { feedsEnd = lastDataLineIndex + 1; } else { feedsEnd = i; } break; } } // 如果没有找到结束位置，使用最后一个数据行的下一行 if (feedsEnd === -1) { feedsEnd = lastDataLineIndex >= 0 ? lastDataLineIndex + 1 : lines.length; } // 提取区域内的注释（保留在开头的注释） const regionLines = lines.slice(feedsStart, feedsEnd); const leadingComments = []; for (const line of regionLines) { const trimmed = line.trim(); if (trimmed.startsWith('#')) { leadingComments.push(line); } else if (trimmed.startsWith('-') || (trimmed && !trimmed.startsWith('#'))) { // 遇到第一个数据项，停止收集注释 break; } else if (trimmed === '') { // 空行也保留 leadingComments.push(line); } } // 构建新的 feeds 内容 const indent = ' '; // 4 空格缩进 const newFeedsLines = feeds.map(f => { let feedYaml = `${indent}- id: "${f.id}"\n${indent} name: "${f.name}"\n${indent} url: "${f.url}"`; if (f.enabled === false) { feedYaml += `\n${indent} enabled: false`; } if (f.max_age_days !== undefined && f.max_age_days !== '') { feedYaml += `\n${indent} max_age_days: ${f.max_age_days}`; } return feedYaml; }).join('\n\n'); const beforeFeeds = lines.slice(0, feedsStart); const afterFeeds = lines.slice(feedsEnd); // 组合：前面内容 + 开头注释 + 新数据 + 空行 + 后面内容 const newYaml = [ ...beforeFeeds, ...(leadingComments.length > 0 ? leadingComments : []), newFeedsLines, '', ...afterFeeds ].join('\n'); editor.value = newYaml; currentYaml = newYaml; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); renderRssFeedsList(); renderStandaloneLists(); // 同步更新独立展示区的 RSS 选择列表 } // 打开 RSS 添加/编辑弹窗 window.openRssModal = function() { openRssModalWithData(null, -1); } function openRssModalWithData(feed, editIndex) { const modal = document.getElementById('rss-modal'); document.getElementById('rss-id').value = feed ? feed.id : ''; document.getElementById('rss-name').value = feed ? feed.name : ''; document.getElementById('rss-url').value = feed ? feed.url : ''; document.getElementById('rss-max-age').value = feed && feed.max_age_days !== undefined ? feed.max_age_days : ''; modal.dataset.editIndex = editIndex; const title = modal.querySelector('h3'); if (title) { title.innerHTML = editIndex >= 0 ? '编辑 RSS 源' : '添加 RSS 源'; } modal.classList.remove('hidden'); } // 关闭 RSS 弹窗 window.closeRssModal = function() { const modal = document.getElementById('rss-modal'); modal.classList.add('hidden'); modal.dataset.editIndex = '-1'; document.getElementById('rss-id').value = ''; document.getElementById('rss-name').value = ''; document.getElementById('rss-url').value = ''; document.getElementById('rss-max-age').value = ''; } // 确认添加/编辑 RSS window.confirmAddRss = function() { const modal = document.getElementById('rss-modal'); const editIndex = parseInt(modal.dataset.editIndex || '-1'); const id = document.getElementById('rss-id').value.trim(); const name = document.getElementById('rss-name').value.trim(); const url = document.getElementById('rss-url').value.trim(); const maxAge = document.getElementById('rss-max-age').value.trim(); if (!id || !name || !url) { alert('请填写完整信息：ID、名称和 URL 都是必填项'); return; } const feeds = parseRssFeedsFromYaml(); const newFeed = { id, name, url }; if (maxAge) { newFeed.max_age_days = parseInt(maxAge); } if (editIndex >= 0) { feeds[editIndex] = newFeed; } else { feeds.push(newFeed); } updateRssFeedsInYaml(feeds); closeRssModal(); } // ========================================== // 14. 独立展示区 (Standalone) 管理功能 // ========================================== function parseStandaloneConfigFromYaml() { try { const doc = jsyaml.load(currentYaml); if (doc && doc.display && doc.display.standalone) { return { platforms: doc.display.standalone.platforms || [], rss_feeds: doc.display.standalone.rss_feeds || [] }; } } catch (e) {} return { platforms: [], rss_feeds: [] }; } function renderStandaloneLists() { const platformsContainer = document.getElementById('standalone-platforms-list'); const rssContainer = document.getElementById('standalone-rss-list'); if (!platformsContainer || !rssContainer) return; const standaloneConfig = parseStandaloneConfigFromYaml(); const availablePlatforms = parsePlatformsFromYaml(); const availableRss = parseRssFeedsFromYaml(); // Render Platforms if (availablePlatforms.length === 0) { platformsContainer.innerHTML = `
暂无可用平台
`; } else { platformsContainer.innerHTML = availablePlatforms.map(p => { const isChecked = standaloneConfig.platforms.includes(p.id); return ` `; }).join(''); } // Render RSS if (availableRss.length === 0) { rssContainer.innerHTML = `
暂无可用 RSS 源
`; } else { rssContainer.innerHTML = availableRss.map(f => { const isChecked = standaloneConfig.rss_feeds.includes(f.id); return ` `; }).join(''); } } window.toggleStandaloneItem = function(type, id) { const config = parseStandaloneConfigFromYaml(); const list = config[type]; const index = list.indexOf(id); if (index === -1) { list.push(id); } else { list.splice(index, 1); } updateStandaloneConfigInYaml(type, list); } function updateStandaloneConfigInYaml(type, list) { const editor = document.getElementById('yaml-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 找到 display -> standalone -> [type] let inDisplay = false; let inStandalone = false; let targetLineIndex = -1; let indent = ''; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (line.match(/^display:/)) { inDisplay = true; continue; } if (inDisplay && line.trim().startsWith('standalone:')) { inStandalone = true; continue; } if (inStandalone) { // 检查是否离开 standalone (遇到缩进更少或相同的非注释行) const currentIndent = line.search(/\S/); // standalone 下一级的缩进 if (line.match(new RegExp(`^\\s*${type}:`))) { targetLineIndex = i; indent = line.substring(0, line.indexOf(type)); break; } // 如果遇到下一个模块，停止 if (line.match(/^[a-z_]+:/) && !line.match(/^display:/)) break; } } if (targetLineIndex !== -1) { // 构建新的数组字符串 ["item1", "item2"] const jsonStr = JSON.stringify(list); // 保留原有注释 const originalLine = lines[targetLineIndex]; const commentMatch = originalLine.match(/#.*$/); const comment = commentMatch ? commentMatch[0] : ''; lines[targetLineIndex] = `${indent}${type}: ${jsonStr}${comment ? ' ' + comment : ''}`; const newYaml = lines.join('\n'); editor.value = newYaml; currentYaml = newYaml; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); // 不需要重新渲染整个列表，因为是 checkbox 点击触发的 // 但如果需要保持一致性，可以重新渲染 } } // 从文本中提取版本号 function extractVersion(text) { // 匹配 Version: v5.3.0 或 Version: 5.3.0 格式 const versionMatch = text.match(/Version:\s*v?(\d+\.\d+\.\d+)/i); if (versionMatch) { return versionMatch[1]; // 返回不带 v 的版本号 } return null; } // 比较版本号 (返回 1: v1 > v2, -1: v1 < v2, 0: v1 == v2) function compareVersions(v1, v2) { if (!v1 || !v2) return 0; const parts1 = v1.split('.').map(Number); const parts2 = v2.split('.').map(Number); for (let i = 0; i < Math.max(parts1.length, parts2.length); i++) { const num1 = parts1[i] || 0; const num2 = parts2[i] || 0; if (num1 > num2) return 1; if (num1 < num2) return -1; } return 0; } // 版本检测主函数 window.checkVersion = async function() { const btn = document.getElementById('version-check-btn'); const originalHTML = btn.innerHTML; btn.innerHTML = '检测中...'; btn.disabled = true; try { const versionRes = await fetch(REMOTE_VERSION_URL); if (!versionRes.ok) { throw new Error(`版本信息获取失败: ${versionRes.status}`); } const versionConfigText = await versionRes.text(); const versionMap = {}; versionConfigText.split('\n').forEach(line => { const parts = line.trim().split('='); if (parts.length >= 2) { versionMap[parts[0].trim()] = parts[1].trim(); } }); const currentTab = getCurrentTab(); let currentVersion = null; let fileName = ''; if (currentTab === 'config') { currentVersion = extractVersion(currentYaml); fileName = 'config.yaml'; } else { currentVersion = extractVersion(currentFrequency); fileName = 'frequency_words.txt'; } const latestVersion = versionMap[fileName]; if (!latestVersion) { throw new Error(`未在远程版本清单中找到 ${fileName}`); } showVersionComparisonModal(fileName, currentVersion, latestVersion); } catch (err) { console.error('版本检测失败:', err); showToast(`版本检测失败: ${err.message}`, 'error'); } finally { btn.innerHTML = originalHTML; btn.disabled = false; } } // 获取当前 Tab function getCurrentTab() { return currentTab; } // 显示版本对比弹窗 function showVersionComparisonModal(fileName, currentVersion, latestVersion) { const existingModal = document.getElementById('version-comparison-modal'); if (existingModal) existingModal.remove(); const comparison = compareVersions(currentVersion, latestVersion); let statusIcon = ''; let statusText = ''; let statusColor = ''; let actionButtons = ''; if (!currentVersion) { statusIcon = ''; statusText = '未检测到版本信息'; statusColor = 'text-gray-600'; actionButtons = ` `; } else if (comparison < 0) { statusIcon = ''; statusText = '发现新版本'; statusColor = 'text-orange-600'; actionButtons = ` `; } else if (comparison > 0) { statusIcon = ''; statusText = '当前版本较新（开发版本？）'; statusColor = 'text-purple-600'; actionButtons = ` `; } else { statusIcon = ''; statusText = '已是最新版本'; statusColor = 'text-green-600'; actionButtons = ` `; } const modal = document.createElement('div'); modal.id = 'version-comparison-modal'; modal.className = 'modal-overlay'; modal.innerHTML = `

版本检测结果

${statusIcon}
${statusText}
配置文件 ${fileName}
当前版本 ${currentVersion ? 'v' + currentVersion : '未知'}
最新版本 v${latestVersion}
${comparison < 0 || !currentVersion ? `
提示：更新将从 GitHub 加载最新的 ${fileName}，你当前的修改将被覆盖。建议先复制保存你的自定义配置。
` : ''}
${actionButtons}
`; document.body.appendChild(modal); } window.closeVersionModal = function() { const modal = document.getElementById('version-comparison-modal'); if (modal) modal.remove(); } // ========================================== // 13. 平台添加弹窗逻辑 // ========================================== // 预定义可用平台列表 (仅包含官方默认支持的平台) const PRESET_PLATFORMS = [ { key: 'toutiao', name: '今日头条' }, { key: 'baidu', name: '百度热搜' }, { key: 'wallstreetcn-hot', name: '华尔街见闻' }, { key: 'thepaper', name: '澎湃新闻' }, { key: 'bilibili-hot-search', name: 'bilibili 热搜' }, { key: 'cls-hot', name: '财联社热门' }, { key: 'ifeng', name: '凤凰网' }, { key: 'tieba', name: '贴吧' }, { key: 'weibo', name: '微博' }, { key: 'douyin', name: '抖音' }, { key: 'zhihu', name: '知乎' } ]; /** * 打开平台添加弹窗 */ window.openPlatformModal = function() { const modal = document.getElementById('platform-modal'); if (modal) { modal.classList.remove('hidden'); if (typeof switchPlatformTab === 'function') { switchPlatformTab('select'); } renderAvailablePlatforms(); } } /** * 关闭平台添加弹窗 */ window.closePlatformModal = function() { const modal = document.getElementById('platform-modal'); if (modal) { modal.classList.add('hidden'); } } /** * 切换平台添加标签页 */ window.switchPlatformTab = function(tab) { currentPlatformTab = tab; // 更新 Tab 样式 const tabSelect = document.getElementById('tab-platform-select'); const tabCustom = document.getElementById('tab-platform-custom'); if (tab === 'select') { if (tabSelect) { tabSelect.classList.add('text-blue-600', 'border-blue-600'); tabSelect.classList.remove('text-gray-500', 'border-transparent'); } if (tabCustom) { tabCustom.classList.remove('text-blue-600', 'border-blue-600'); tabCustom.classList.add('text-gray-500', 'border-transparent'); } const selectPanel = document.getElementById('platform-select-panel'); const customPanel = document.getElementById('platform-custom-panel'); if (selectPanel) selectPanel.classList.remove('hidden'); if (customPanel) customPanel.classList.add('hidden'); } else { if (tabCustom) { tabCustom.classList.add('text-blue-600', 'border-blue-600'); tabCustom.classList.remove('text-gray-500', 'border-transparent'); } if (tabSelect) { tabSelect.classList.remove('text-blue-600', 'border-blue-600'); tabSelect.classList.add('text-gray-500', 'border-transparent'); } const selectPanel = document.getElementById('platform-select-panel'); const customPanel = document.getElementById('platform-custom-panel'); if (selectPanel) selectPanel.classList.add('hidden'); if (customPanel) customPanel.classList.remove('hidden'); } } /** * 渲染可用平台列表（排除已添加的） */ function renderAvailablePlatforms() { const container = document.getElementById('available-platforms-list'); const tip = document.getElementById('no-platforms-tip'); if (!container) return; container.innerHTML = ''; const currentPlatforms = parsePlatformsFromYaml(); const existingKeys = currentPlatforms.map(p => p.id); const available = PRESET_PLATFORMS.filter(p => !existingKeys.includes(p.key)); if (available.length === 0) { if (tip) { tip.classList.remove('hidden'); tip.innerHTML = `所有预设平台已添加`; } } else { if (tip) tip.classList.add('hidden'); available.forEach(p => { const div = document.createElement('div'); div.className = 'flex items-center justify-between p-3 border border-gray-100 rounded hover:bg-blue-50 cursor-pointer transition-colors group'; div.onclick = () => confirmAddPlatform(p.key, p.name); div.innerHTML = `
${p.name}
${p.key}
`; container.appendChild(div); }); } } /** * 确认添加平台 */ window.confirmAddPlatform = function(key, name) { let platformKey = key; let platformName = name; // 如果是手动输入模式 (且未传入 key) if (currentPlatformTab === 'custom' && !key) { const keyInput = document.getElementById('custom-platform-key'); const nameInput = document.getElementById('custom-platform-name'); if (keyInput) platformKey = keyInput.value.trim(); if (nameInput) platformName = nameInput.value.trim(); if (!platformKey) { alert('请输入平台 Key'); return; } if (!platformName) { platformName = platformKey; } } else if (currentPlatformTab === 'select' && !key) { alert('请直接点击上方列表中的平台进行添加'); return; } // 检查是否已存在 const currentPlatforms = parsePlatformsFromYaml(); if (currentPlatforms.find(p => p.id === platformKey)) { alert(`平台 ${platformKey} 已存在！`); return; } // 添加到 YAML (注意字段是 id 和 name) const newPlatform = { id: platformKey, name: platformName, enabled: true }; // 重新构建 YAML currentPlatforms.push(newPlatform); updatePlatformsInYaml(currentPlatforms); closePlatformModal(); const keyInput = document.getElementById('custom-platform-key'); const nameInput = document.getElementById('custom-platform-name'); if (keyInput) keyInput.value = ''; if (nameInput) nameInput.value = ''; renderPlatformsList(); showToast(`平台 ${platformName} 已添加`, 'success'); } // 绑定到全局 window.updateToLatest = async function() { closeVersionModal(); const currentTab = getCurrentTab(); const fileName = currentTab === 'config' ? 'config.yaml' : 'frequency_words.txt'; if (!confirm(`确定要从 GitHub 更新 ${fileName} 到最新版本吗？\n\n你当前的自定义配置将被覆盖，建议先复制保存。`)) { return; } showToast('正在从 GitHub 加载最新版本...', 'info'); try { const url = currentTab === 'config' ? REMOTE_CONFIG_URL : REMOTE_FREQUENCY_URL; const res = await fetch(url); if (!res.ok) { throw new Error(`加载失败: ${res.status}`); } const text = await res.text(); if (currentTab === 'config') { try { jsyaml.load(text); } catch (yamlErr) { showToast(`YAML 语法错误: ${yamlErr.message}`, 'error'); return; } document.getElementById('yaml-editor').value = text; currentYaml = text; syncYamlToUI(); } else { document.getElementById('frequency-editor').value = text; currentFrequency = text; syncFrequencyToUI(); } saveToLocalStorage(); showToast(`已更新到最新版本`, 'success'); } catch (err) { console.error('更新失败:', err); showToast(`更新失败: ${err.message}`, 'error'); } } // ========================================== // RSS 辅助功能 // ========================================== function toggleRssTips() { const panel = document.getElementById('rss-tips-panel'); const icon = document.getElementById('rss-tips-icon'); if (panel) { panel.classList.toggle('hidden'); if (icon) { icon.style.transform = panel.classList.contains('hidden') ? 'rotate(0deg)' : 'rotate(180deg)'; } } } function fillRssUrl(url) { const input = document.getElementById('rss-url'); if (input) { input.value = url; // 视觉反馈 input.classList.add('ring-2', 'ring-blue-500', 'bg-blue-50'); setTimeout(() => { input.classList.remove('ring-2', 'ring-blue-500', 'bg-blue-50'); }, 500); } } // ========================================== // 13. Timeline 编辑器功能 // ========================================== const PRESET_META = { morning_evening: { icon: 'fa-sun', color: 'text-amber-500', bg: 'bg-amber-50', recommend: true }, always_on: { icon: 'fa-bolt', color: 'text-blue-500', bg: 'bg-blue-50' }, office_hours: { icon: 'fa-briefcase', color: 'text-green-500', bg: 'bg-green-50' }, night_owl: { icon: 'fa-moon', color: 'text-indigo-500', bg: 'bg-indigo-50' }, custom: { icon: 'fa-sliders', color: 'text-purple-500', bg: 'bg-purple-50' } }; const DAY_NAMES = ['周一', '周二', '周三', '周四', '周五', '周六', '周日']; /** * 从当前 config.yaml 中读取 schedule.preset */ function getActivePreset() { try { const doc = jsyaml.load(currentYaml); return doc?.schedule?.preset || 'morning_evening'; } catch { return 'morning_evening'; } } /** * 解析 timeline YAML，返回结构化数据 */ function parseTimelineData() { try { const doc = jsyaml.load(currentTimeline); if (!doc) return null; return doc; } catch { return null; } } /** * 获取指定预设/custom 的完整配置 */ function getPresetConfig(data, presetName) { if (!data) return null; if (presetName === 'custom') return data.custom || null; return data.presets?.[presetName] || null; } /** * 主渲染函数：解析 timeline YAML → 渲染右侧面板 */ function syncTimelineToUI() { const panel = document.getElementById('timeline-panel'); if (!panel) return; const data = parseTimelineData(); const activePreset = getActivePreset(); if (!data) { panel.innerHTML = `

请在左侧粘贴 timeline.yaml 内容

或点击右上角「加载官网最新配置」

`; return; } let html = ''; // ── Layer 1: 预设模式选择卡片 ── html += `
调度模式
`; // 收集所有预设名 const presetNames = Object.keys(data.presets || {}); // 确保 custom 在最后 const allModes = [...presetNames.filter(n => n !== 'custom'), ...(data.custom ? ['custom'] : [])]; allModes.forEach(name => { const meta = PRESET_META[name] || { icon: 'fa-puzzle-piece', color: 'text-gray-500', bg: 'bg-gray-50' }; const presetCfg = getPresetConfig(data, name); const label = presetCfg?.name || meta.label || name; const desc = presetCfg?.description || meta.desc || ''; const isActive = name === activePreset; const isProtected = ['morning_evening', 'always_on', 'office_hours', 'night_owl', 'custom'].includes(name); html += `
${meta.recommend ? '
推荐
' : ''}
${label}
${desc}
${!isProtected ? `` : ''}
${isActive ? '
当前
' : ''}
`; }); // 新建模式卡片 html += `
新建模式
创建自定义调度方案
`; html += `
`; // 获取当前预设配置 const config = getPresetConfig(data, activePreset); if (!config) { html += `
未找到预设「${activePreset}」的配置
`; panel.innerHTML = html; return; } // ── Layer 2: 周视图时间线 ── html += renderWeekView(config, activePreset); // ── Layer 3: 时间段详情 ── html += renderPeriodDetails(config, activePreset); panel.innerHTML = html; // 初始化日计划 Tag 拖拽排序 initDayPlanSortable(activePreset); } /** * 渲染周视图（7 天 × 24 小时水平条） */ function renderWeekView(config, presetName) { const periods = config.periods || {}; const dayPlans = config.day_plans || {}; const weekMap = config.week_map || {}; // 时间刻度 let html = `
周视图
`; for (let h = 0; h <= 24; h += 2) { html += `
${h < 10 ? '0' : ''}${h}
`; } html += `
`; // 获取当前星期几 (1=周一...7=周日) const today = new Date().getDay(); const todayIso = today === 0 ? 7 : today; // 7 天的行 for (let d = 1; d <= 7; d++) { const dayPlanName = weekMap[d] || weekMap[String(d)]; const dayPlan = dayPlans[dayPlanName]; const dayPeriodNames = dayPlan?.periods || []; const isToday = d === todayIso; html += `
${DAY_NAMES[d-1]}
`; // 渲染各时间段色块 dayPeriodNames.forEach(pName => { const p = periods[pName]; if (!p) return; const merged = mergeWithDefault(p, config.default); const colorClass = getBlockColorClass(merged); const blocks = computeBlocks(p.start, p.end); blocks.forEach(b => { const left = (b.start / 24 * 100).toFixed(2); const width = ((b.end - b.start) / 24 * 100).toFixed(2); const label = p.name || pName; html += `
${label}
`; }); }); // 当前时间指示线（仅今天） if (isToday) { const nowTime = new Date(); const nowH = nowTime.getHours() + nowTime.getMinutes() / 60; const nowLeftPct = (nowH / 24 * 100).toFixed(2); html += `
`; } html += `
`; } // 图例 html += `
推送
AI 分析
推送 + 分析
仅采集
默认 (default)
`; html += `
`; return html; } /** * 合并 period 与 default（period 字段优先） */ function mergeWithDefault(period, defaultCfg) { if (!defaultCfg) return period || {}; const merged = { ...defaultCfg, ...period }; if (period.once || defaultCfg.once) { merged.once = { ...(defaultCfg.once || {}), ...(period.once || {}) }; } return merged; } /** * 根据 push/analyze 状态确定色块 CSS 类 */ function getBlockColorClass(merged) { const push = !!merged.push; const analyze = !!merged.analyze; if (push && analyze) return 'tl-block-push-analyze'; if (push) return 'tl-block-push'; if (analyze) return 'tl-block-analyze'; if (merged.collect !== false) return 'tl-block-collect'; return 'tl-block-silent'; } /** * 计算时间段的渲染块（处理跨午夜情况） * 返回 [{start: 小时数, end: 小时数}, ...] 的数组 */ function computeBlocks(startStr, endStr) { if (!startStr || !endStr) return []; const s = parseTime(startStr); const e = parseTime(endStr); if (s < e) return [{ start: s, end: e }]; // 跨午夜 return [{ start: s, end: 24 }, { start: 0, end: e }]; } function parseTime(str) { const [h, m] = (str || '00:00').split(':').map(Number); return h + (m || 0) / 60; } function escapeAttr(s) { return (s || '').replace(/'/g, "\\'").replace(/"/g, '"'); } /** * Tooltip 显示/隐藏 */ let tlTooltipEl = null; function showTlTooltip(event, name, start, end, push, analyze, mode) { hideTlTooltip(); const el = document.createElement('div'); el.className = 'tl-tooltip'; let features = []; if (push) features.push('推送'); if (analyze) features.push('分析'); if (!push && !analyze) features.push('仅采集'); el.innerHTML = `
${name}
${start} - ${end}
${features.join(' / ')}
${mode ? `
模式: ${mode}
` : ''}`; document.body.appendChild(el); tlTooltipEl = el; const rect = event.target.getBoundingClientRect(); el.style.left = (rect.left + rect.width / 2 - el.offsetWidth / 2) + 'px'; el.style.top = (rect.top - el.offsetHeight - 8) + 'px'; // 确保不超出屏幕 const elRect = el.getBoundingClientRect(); if (elRect.left < 4) el.style.left = '4px'; if (elRect.right > window.innerWidth - 4) el.style.left = (window.innerWidth - el.offsetWidth - 4) + 'px'; if (elRect.top < 4) { el.style.top = (rect.bottom + 8) + 'px'; el.style.setProperty('--arrow', 'top'); } } function hideTlTooltip() { if (tlTooltipEl) { tlTooltipEl.remove(); tlTooltipEl = null; } } /** * 渲染时间段详情面板 */ function renderPeriodDetails(config, presetName) { const isCustom = presetName === 'custom'; const periods = config.periods || {}; const dayPlans = config.day_plans || {}; const weekMap = config.week_map || {}; const defaults = config.default || {}; let html = ''; // ── Default 配置（默认展开）── html += `
默认配置 (default)
不在任何时间段内时，使用以下配置：
${renderBehaviorToggles(defaults, presetName, 'default', defaults)}
`; // ── 时间段列表 ── const periodEntries = Object.entries(periods); html += `
时间段 (Periods)
`; if (periodEntries.length > 0) { html += `
`; periodEntries.forEach(([key, p]) => { const merged = mergeWithDefault(p, defaults); const colorClass = getBlockColorClass(merged); html += `
${p.name || key} ${key}
${p.start || '?'} - ${p.end || '?'}
${renderBehaviorToggles(merged, presetName, key, p)}
`; }); html += `
`; } else { html += `
此模式无自定义时间段，全天使用 default 配置
`; } html += `
`; // ── 日计划 ── const dayPlanEntries = Object.entries(dayPlans); html += `
日计划 (Day Plans)
`; if (dayPlanEntries.length > 0) { html += `
`; dayPlanEntries.forEach(([name, plan]) => { const pList = plan.periods || []; // 构建可用 period 下拉（排除已添加的） const availablePeriods = periodEntries.filter(([k]) => !pList.includes(k)); html += `
${name}
${pList.length > 0 ? pList.map(pn => { const p = periods[pn]; const merged = p ? mergeWithDefault(p, defaults) : {}; const cc = getBlockColorClass(merged); return ` ${p?.name || pn} `; }).join('') : '空 (全天走 default)'} ${availablePeriods.length > 0 ? ` ` : ''}
`; }); html += `
`; } html += `
`; // ── 周映射（下拉选择）── const dayPlanKeys = Object.keys(dayPlans); // 为不同日计划分配颜色 const planColorMap = {}; const planColors = ['bg-blue-50 border-blue-200', 'bg-green-50 border-green-200', 'bg-amber-50 border-amber-200', 'bg-purple-50 border-purple-200', 'bg-rose-50 border-rose-200', 'bg-cyan-50 border-cyan-200', 'bg-orange-50 border-orange-200']; dayPlanKeys.forEach((k, idx) => { planColorMap[k] = planColors[idx % planColors.length]; }); html += `
周映射 (Week Map)
`; for (let d = 1; d <= 7; d++) { const plan = weekMap[d] || weekMap[String(d)] || ''; const rowColor = planColorMap[plan] || ''; const options = dayPlanKeys.map(k => `` ).join(''); html += `
${DAY_NAMES[d-1]}
`; } html += `
`; // custom 专属：时间段冲突策略 if (isCustom) { const overlapPolicy = (config.overlap && config.overlap.policy) || 'error_on_overlap'; html += `
冲突策略 (Overlap)
policy:
error_on_overlap 会在时间段重叠时直接报错；last_wins 会按 day_plans 中靠后的时间段覆盖。
`; } // 提示 if (!isCustom) { html += `
直接在上方调整开关和下拉框，左侧 YAML 会同步更新。如需更精细的控制，可直接编辑左侧 YAML 或修改 timeline.yaml。
`; } else { html += `
自定义模式支持完全自由编辑。可直接在上方调整控件，或在左侧编辑 YAML 文本，两边实时同步。
`; } return html; } /** * 渲染行为开关（可交互） * presetName: 当前预设名（用于定位 YAML 中的位置） * periodKey: 'default' 或时间段 key（如 'weekday_morning'） */ function renderBehaviorToggles(cfg, presetName, periodKey, rawCfg = null) { const toggleItems = [ { k: 'collect', label: '采集', icon: 'fa-download' }, { k: 'analyze', label: '分析', icon: 'fa-brain' }, { k: 'push', label: '推送', icon: 'fa-bell' }, ]; const uid = `tl-${presetName}-${periodKey}`; let html = '
'; toggleItems.forEach(item => { const val = cfg[item.k]; const on = val === true || val === 'true'; const toggleId = `${uid}-${item.k}`; html += ``; }); html += '
'; // 报告模式下拉 const reportModes = ['current', 'daily', 'incremental']; const aiModes = ['follow_report', 'daily', 'current', 'incremental']; html += `
`; // report_mode html += `
报告:
`; // ai_mode html += `
AI:
`; // once toggles const onceAnalyze = cfg.once?.analyze === true; const oncePush = cfg.once?.push === true; html += ``; html += ``; html += `
`; // 时间段编辑（仅非 default） if (periodKey !== 'default' && (cfg.start || cfg.end)) { html += `
时间: ~
`; } // 可选筛选覆盖（仅显示“当前层”字段，避免把继承值误当作显式配置） const baseCfg = rawCfg || {}; const filterMethod = baseCfg.filter_method || ''; const frequencyFile = baseCfg.frequency_file || ''; const interestsFile = baseCfg.interests_file || ''; const methodHint = periodKey === 'default' ? '不填则跟随全局 filter.method' : '不填则继承 default（再回退全局）'; html += `
筛选覆盖（可选）
${methodHint}。frequency_file 从 config/custom/keyword/ 查找， interests_file 从 config/custom/ai/ 查找；留空会删除该字段并恢复继承。
`; return html; } /** * 点击周视图色块 → 滚动到对应 period 卡片并高亮 */ window.scrollToPeriodCard = function(periodKey) { const card = document.getElementById('tl-period-' + periodKey); if (!card) return; card.scrollIntoView({ behavior: 'smooth', block: 'center' }); card.classList.add('tl-period-highlight'); setTimeout(() => card.classList.remove('tl-period-highlight'), 1500); } /** * 折叠/展开切换 */ window.toggleTlCollapsible = function(header) { const body = header.nextElementSibling; body.classList.toggle('collapsed'); header.classList.toggle('is-collapsed'); } /** * 右侧开关变更 → 更新左侧 timeline YAML */ window.onTlToggle = function(presetName, periodKey, field, value) { updateTimelineField(presetName, periodKey, field, value); } window.onTlSelect = function(presetName, periodKey, field, value) { updateTimelineField(presetName, periodKey, field, value); } window.onTlOptionalInput = function(presetName, periodKey, field, rawValue) { const value = (rawValue || '').trim(); if (!value) { removeTimelineField(presetName, periodKey, field); return; } updateTimelineField(presetName, periodKey, field, value); } window.onTlOptionalSelect = function(presetName, periodKey, field, value) { if (!value) { removeTimelineField(presetName, periodKey, field); return; } updateTimelineField(presetName, periodKey, field, value); } window.onTlCustomOverlapPolicy = function(value) { updateTimelineSectionField('custom', 'overlap.policy', value); } /** * 周映射下拉变更 → 更新 timeline YAML 中的 week_map.N */ window.onTlWeekMap = function(presetName, dayNum, value) { const editor = document.getElementById('timeline-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 定位 preset section const isCustom = presetName === 'custom'; let sectionStart = -1; let sectionIndent = 0; if (isCustom) { for (let i = 0; i < lines.length; i++) { if (/^custom:\s*/.test(lines[i])) { sectionStart = i; break; } } } else { let inPresets = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^presets:\s*/.test(line)) { inPresets = true; continue; } if (inPresets && /^\S/.test(line) && !line.startsWith('#')) break; if (inPresets) { const m = line.match(/^(\s+)(\S+):\s*/); if (m && m[2] === presetName) { sectionStart = i; sectionIndent = m[1].length; break; } } } } if (sectionStart < 0) return; let sectionEnd = lines.length; for (let i = sectionStart + 1; i < lines.length; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; if (line.search(/\S/) <= sectionIndent) { sectionEnd = i; break; } } // 找 week_map: 行 const weekMapLine = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, 'week_map'); if (weekMapLine < 0) return; const wmIndent = lines[weekMapLine].search(/\S/); const wmEnd = findBlockEnd(lines, weekMapLine, wmIndent, sectionEnd); // 找 dayNum: 行 const dayKey = String(dayNum); const dayLine = findChildKey(lines, weekMapLine, wmEnd, wmIndent, dayKey); if (dayLine >= 0) { replaceLineValue(lines, dayLine, value); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); clearTimeout(window._tlRenderTimer); window._tlRenderTimer = setTimeout(() => syncTimelineToUI(), 300); } /** * 核心：修改 timeline YAML 中的指定字段，保留注释 */ function updateTimelineField(presetName, periodKey, field, value) { const editor = document.getElementById('timeline-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 1. 定位预设/custom 的起始行 const isCustom = presetName === 'custom'; let sectionStart = -1; let sectionIndent = 0; if (isCustom) { // 找 custom: 顶层 key for (let i = 0; i < lines.length; i++) { if (/^custom:\s*/.test(lines[i])) { sectionStart = i; sectionIndent = 0; break; } } } else { // 找 presets: 下的 presetName: let inPresets = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^presets:\s*/.test(line)) { inPresets = true; continue; } if (inPresets && /^\S/.test(line) && !line.startsWith('#')) { break; // left presets block } if (inPresets) { const m = line.match(/^(\s+)(\S+):\s*/); if (m && m[2] === presetName) { sectionStart = i; sectionIndent = m[1].length; break; } } } } if (sectionStart < 0) return; // 2. 找到 section 结束行 let sectionEnd = lines.length; for (let i = sectionStart + 1; i < lines.length; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const indent = line.search(/\S/); if (indent <= sectionIndent) { sectionEnd = i; break; } } // 3. 在 section 内定位 periodKey 子区域 let targetStart, targetEnd; const fieldParts = field.split('.'); if (periodKey === 'default') { // 找 default: 行 targetStart = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, 'default'); } else { // 找 periods: 下的 periodKey: const periodsLine = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, 'periods'); if (periodsLine < 0) return; const periodsIndent = lines[periodsLine].search(/\S/); const periodsEnd = findBlockEnd(lines, periodsLine, periodsIndent, sectionEnd); targetStart = findChildKey(lines, periodsLine, periodsEnd, periodsIndent, periodKey); } if (targetStart < 0) return; const targetIndent = lines[targetStart].search(/\S/); targetEnd = findBlockEnd(lines, targetStart, targetIndent, sectionEnd); // 4. 在 target 内查找 field（支持 once.analyze 嵌套） let lineIdx = -1; if (fieldParts.length === 1) { lineIdx = findChildKey(lines, targetStart, targetEnd, targetIndent, fieldParts[0]); } else { // nested: once.analyze → find once: then analyze: const parentLine = findChildKey(lines, targetStart, targetEnd, targetIndent, fieldParts[0]); if (parentLine >= 0) { const parentIndent = lines[parentLine].search(/\S/); const parentEnd = findBlockEnd(lines, parentLine, parentIndent, targetEnd); lineIdx = findChildKey(lines, parentLine, parentEnd, parentIndent, fieldParts[1]); } } if (lineIdx < 0) { // 字段不存在 → 需要插入 insertTimelineField(lines, targetStart, targetEnd, targetIndent, field, value, fieldParts); } else { // 字段存在 → 原地替换值 replaceLineValue(lines, lineIdx, value); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); // 延迟重新渲染（避免输入中途刷新） clearTimeout(window._tlRenderTimer); window._tlRenderTimer = setTimeout(() => syncTimelineToUI(), 300); } function resolveTimelineSection(lines, presetName) { const isCustom = presetName === 'custom'; let sectionStart = -1; let sectionIndent = 0; if (isCustom) { for (let i = 0; i < lines.length; i++) { if (/^custom:\s*/.test(lines[i])) { sectionStart = i; sectionIndent = 0; break; } } } else { let inPresets = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^presets:\s*/.test(line)) { inPresets = true; continue; } if (inPresets && /^\S/.test(line) && !line.startsWith('#')) { break; } if (inPresets) { const m = line.match(/^(\s+)(\S+):\s*/); if (m && m[2] === presetName) { sectionStart = i; sectionIndent = m[1].length; break; } } } } if (sectionStart < 0) return null; let sectionEnd = lines.length; for (let i = sectionStart + 1; i < lines.length; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const indent = line.search(/\S/); if (indent <= sectionIndent) { sectionEnd = i; break; } } return { sectionStart, sectionEnd, sectionIndent }; } function resolveTimelineTarget(lines, presetName, periodKey) { const section = resolveTimelineSection(lines, presetName); if (!section) return null; const { sectionStart, sectionEnd, sectionIndent } = section; let targetStart = -1; if (periodKey === 'default') { targetStart = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, 'default'); } else { const periodsLine = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, 'periods'); if (periodsLine < 0) return null; const periodsIndent = lines[periodsLine].search(/\S/); const periodsEnd = findBlockEnd(lines, periodsLine, periodsIndent, sectionEnd); targetStart = findChildKey(lines, periodsLine, periodsEnd, periodsIndent, periodKey); } if (targetStart < 0) return null; const targetIndent = lines[targetStart].search(/\S/); const targetEnd = findBlockEnd(lines, targetStart, targetIndent, sectionEnd); return { sectionStart, sectionEnd, sectionIndent, targetStart, targetEnd, targetIndent }; } function applyTimelineEditorChanges(editor, lines) { editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); clearTimeout(window._tlRenderTimer); window._tlRenderTimer = setTimeout(() => syncTimelineToUI(), 300); } function removeTimelineField(presetName, periodKey, field) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const target = resolveTimelineTarget(lines, presetName, periodKey); if (!target) return; const { targetStart, targetEnd, targetIndent } = target; const fieldParts = field.split('.'); if (fieldParts.length === 1) { const lineIdx = findChildKey(lines, targetStart, targetEnd, targetIndent, fieldParts[0]); if (lineIdx < 0) return; const lineIndent = lines[lineIdx].search(/\S/); const lineEnd = findBlockEnd(lines, lineIdx, lineIndent, targetEnd); lines.splice(lineIdx, lineEnd - lineIdx); applyTimelineEditorChanges(editor, lines); return; } const parentLine = findChildKey(lines, targetStart, targetEnd, targetIndent, fieldParts[0]); if (parentLine < 0) return; const parentIndent = lines[parentLine].search(/\S/); const parentEnd = findBlockEnd(lines, parentLine, parentIndent, targetEnd); const childLine = findChildKey(lines, parentLine, parentEnd, parentIndent, fieldParts[1]); if (childLine < 0) return; const childIndent = lines[childLine].search(/\S/); const childEnd = findBlockEnd(lines, childLine, childIndent, parentEnd); lines.splice(childLine, childEnd - childLine); const parentEndAfter = findBlockEnd(lines, parentLine, parentIndent, targetEnd); let hasChild = false; for (let i = parentLine + 1; i < parentEndAfter; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; if (line.search(/\S/) > parentIndent) { hasChild = true; break; } } if (!hasChild) { lines.splice(parentLine, 1); } applyTimelineEditorChanges(editor, lines); } function updateTimelineSectionField(presetName, field, value) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const section = resolveTimelineSection(lines, presetName); if (!section) return; const { sectionStart, sectionEnd, sectionIndent } = section; const fieldParts = field.split('.'); let lineIdx = -1; if (fieldParts.length === 1) { lineIdx = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, fieldParts[0]); } else { const parentLine = findChildKey(lines, sectionStart, sectionEnd, sectionIndent, fieldParts[0]); if (parentLine >= 0) { const parentIndent = lines[parentLine].search(/\S/); const parentEnd = findBlockEnd(lines, parentLine, parentIndent, sectionEnd); lineIdx = findChildKey(lines, parentLine, parentEnd, parentIndent, fieldParts[1]); } } if (lineIdx < 0) { insertTimelineField(lines, sectionStart, sectionEnd, sectionIndent, field, value, fieldParts); } else { replaceLineValue(lines, lineIdx, value); } applyTimelineEditorChanges(editor, lines); } /** * 查找子级 key 行 */ function findChildKey(lines, start, end, parentIndent, key) { for (let i = start + 1; i < end; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const indent = line.search(/\S/); if (indent <= parentIndent) break; const m = line.match(/^\s*(\S+):\s*/); if (m && m[1] === key && indent === parentIndent + 2) { return i; } } return -1; } /** * 找一个 block 的结束行号（下一个同级或更低缩进的非空非注释行） */ function findBlockEnd(lines, start, indent, maxEnd) { for (let i = start + 1; i < maxEnd; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const curIndent = line.search(/\S/); if (curIndent <= indent) return i; } return maxEnd; } /** * 替换行中的值，保留注释 */ function replaceLineValue(lines, idx, value) { const original = lines[idx]; const match = original.match(/^(\s*\S+:\s*)(.*)$/); if (!match) return; const prefix = match[1]; const rest = match[2]; const commentMatch = rest.match(/(\s*#.*)$/); const comment = commentMatch ? commentMatch[1] : ''; let formatted; if (typeof value === 'boolean') { formatted = value ? 'true' : 'false'; } else if (typeof value === 'string') { // 检查原值是否带引号 const valPart = rest.slice(0, rest.length - comment.length).trim(); const isQuoted = (valPart.startsWith('"') && valPart.endsWith('"')) || (valPart.startsWith("'") && valPart.endsWith("'")); if (isQuoted || value.includes(':') || value.includes('#') || value.includes(' ')) { formatted = `"${value}"`; } else { formatted = value; } } else { formatted = String(value); } lines[idx] = `${prefix}${formatted}${comment}`; } /** * 字段不存在时，插入新行 */ function insertTimelineField(lines, targetStart, targetEnd, targetIndent, field, value, fieldParts) { const indent = ' '.repeat(targetIndent + 2); let formatted; if (typeof value === 'boolean') formatted = value ? 'true' : 'false'; else if (typeof value === 'string') formatted = value.includes(':') ? `"${value}"` : value; else formatted = String(value); if (fieldParts.length === 1) { // 直接在 target 的末尾插入 lines.splice(targetEnd, 0, `${indent}${field}: ${formatted}`); } else { // once.analyze → find or create once: block, then insert child const parentLine = findChildKey(lines, targetStart, targetEnd, targetIndent, fieldParts[0]); if (parentLine >= 0) { const parentIndent = lines[parentLine].search(/\S/); const parentEnd = findBlockEnd(lines, parentLine, parentIndent, targetEnd); const childIndent = ' '.repeat(parentIndent + 2); lines.splice(parentEnd, 0, `${childIndent}${fieldParts[1]}: ${formatted}`); } else { // parent doesn't exist → create both lines.splice(targetEnd, 0, `${indent}${fieldParts[0]}:`, `${indent} ${fieldParts[1]}: ${formatted}` ); } } } /** * 点击预设卡片 → 更新 config.yaml 中的 schedule.preset + 滚动左侧编辑器 */ window.selectTimelinePreset = function(name) { // 更新 config.yaml 中的 schedule.preset const configEditor = document.getElementById('yaml-editor'); let yaml = configEditor.value; const lines = yaml.split('\n'); let presetLineIdx = -1; let inSchedule = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^schedule:\s*$/.test(line.trimEnd()) || /^schedule:\s*#/.test(line)) { inSchedule = true; continue; } if (inSchedule && /^\S/.test(line) && !line.startsWith('#')) { inSchedule = false; } if (inSchedule && /^\s+preset:\s*/.test(line)) { presetLineIdx = i; break; } } if (presetLineIdx >= 0) { const original = lines[presetLineIdx]; const match = original.match(/^(\s*preset:\s*)(.*)$/); if (match) { const prefix = match[1]; const rest = match[2]; const commentMatch = rest.match(/(\s*#.*)$/); const comment = commentMatch ? commentMatch[1] : ''; lines[presetLineIdx] = `${prefix}"${name}"${comment}`; } } configEditor.value = lines.join('\n'); currentYaml = configEditor.value; updateBackdrop('yaml-editor', 'yaml-backdrop'); debounceSaveConfig(); // 左侧 timeline 编辑器跳转到对应预设 scrollTimelineEditorToPreset(name); // 重新渲染 timeline 面板 syncTimelineToUI(); const tlData = parseTimelineData(); const tlCfg = getPresetConfig(tlData, name); const displayName = tlCfg?.name || name; showToast(`已切换至「${displayName}」模式`, 'success'); } /** * 滚动左侧 timeline 编辑器到对应预设位置 */ function scrollTimelineEditorToPreset(presetName) { const editor = document.getElementById('timeline-editor'); const text = editor.value; const lines = text.split('\n'); let targetLine = -1; if (presetName === 'custom') { // 找顶层 custom: for (let i = 0; i < lines.length; i++) { if (/^custom:\s*/.test(lines[i])) { targetLine = i; break; } } } else { // 找 presets: 下的 presetName: let inPresets = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^presets:\s*/.test(line)) { inPresets = true; continue; } if (inPresets && /^\S/.test(line) && !line.startsWith('#')) break; if (inPresets) { const m = line.match(/^\s+(\S+):\s*/); if (m && m[1] === presetName) { targetLine = i; break; } } } } if (targetLine < 0) return; const lineHeight = 19.5; const scrollPosition = targetLine * lineHeight; // 设置光标位置 let charCount = 0; for (let i = 0; i < targetLine; i++) { charCount += lines[i].length + 1; } editor.focus(); editor.setSelectionRange(charCount, charCount + lines[targetLine].length); editor.scrollTop = scrollPosition - 50; // 高亮闪烁（防止快速点击竞态） clearTimeout(window._tlEditorFlashTimer); editor.style.transition = 'background-color 0.3s'; editor.style.backgroundColor = '#2d4a7c'; window._tlEditorFlashTimer = setTimeout(() => { editor.style.backgroundColor = ''; }, 300); } // ========================================== // 14. Timeline CRUD 功能（新建模式/时间段/日计划/删除等） // ========================================== // ── 弹窗：新建调度模式 ── window.openTlNewPresetModal = function() { const modal = document.getElementById('tl-new-preset-modal'); // 填充模板下拉 const sel = document.getElementById('tl-new-preset-template'); const data = parseTimelineData(); sel.innerHTML = ''; if (data?.presets) { Object.keys(data.presets).forEach(k => { const name = data.presets[k]?.name || k; sel.innerHTML += ``; }); } if (data?.custom) { sel.innerHTML += ``; } // 清空输入 document.getElementById('tl-new-preset-key').value = ''; document.getElementById('tl-new-preset-name').value = ''; document.getElementById('tl-new-preset-desc').value = ''; sel.value = ''; modal.classList.remove('hidden'); } window.closeTlNewPresetModal = function() { document.getElementById('tl-new-preset-modal').classList.add('hidden'); } window.confirmTlNewPreset = function() { const key = document.getElementById('tl-new-preset-key').value.trim(); const name = document.getElementById('tl-new-preset-name').value.trim(); const desc = document.getElementById('tl-new-preset-desc').value.trim(); const template = document.getElementById('tl-new-preset-template').value; // 验证 if (!key) { showToast('请输入模式标识 (key)', 'error'); return; } if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(key)) { showToast('key 仅支持英文、数字和下划线，且不能以数字开头', 'error'); return; } if (!name) { showToast('请输入显示名称', 'error'); return; } // 检查重复 const data = parseTimelineData(); if (data?.presets?.[key]) { showToast(`预设「${key}」已存在`, 'error'); return; } if (key === 'custom') { showToast('不能使用 "custom" 作为预设名', 'error'); return; } // 构建 YAML 文本块 let block; if (template && data) { const src = getPresetConfig(data, template); if (src) { block = buildPresetYamlBlock(key, { ...src, name: name, description: desc || src.description || '' }); } else { block = buildEmptyPresetBlock(key, name, desc); } } else { block = buildEmptyPresetBlock(key, name, desc); } // 插入到 timeline YAML 的 presets: 块末尾 const editor = document.getElementById('timeline-editor'); let yaml = editor.value; const lines = yaml.split('\n'); // 找 presets: 块的结束位置 let presetsStart = -1; for (let i = 0; i < lines.length; i++) { if (/^presets:\s*/.test(lines[i])) { presetsStart = i; break; } } if (presetsStart < 0) { // 没有 presets: 顶层 key，在文件开头插入 lines.unshift('presets:', ...block.split('\n')); } else { // 找 presets 块结束（下一个顶层 key） let presetsEnd = lines.length; for (let i = presetsStart + 1; i < lines.length; i++) { if (/^\S/.test(lines[i]) && !lines[i].startsWith('#') && lines[i].trim() !== '') { presetsEnd = i; break; } } // 在 presetsEnd 前插入（即 presets 块最后） const blockLines = block.split('\n'); lines.splice(presetsEnd, 0, ...blockLines); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); // 切换 config.yaml 中 preset 为新模式 selectTimelinePreset(key); closeTlNewPresetModal(); showToast(`调度模式「${name}」创建成功`, 'success'); } /** * 构建空白预设 YAML 文本块 */ function buildEmptyPresetBlock(key, name, desc) { return [ ` ${key}:`, ` name: "${name}"`, ` description: "${desc || ''}"`, ` default:`, ` collect: true`, ` analyze: false`, ` ai_mode: follow_report`, ` push: false`, ` report_mode: current`, ` once:`, ` analyze: false`, ` push: false`, ` periods: {}`, ` day_plans:`, ` all_day:`, ` periods: []`, ` week_map:`, ` 1: all_day`, ` 2: all_day`, ` 3: all_day`, ` 4: all_day`, ` 5: all_day`, ` 6: all_day`, ` 7: all_day`, `` ].join('\n'); } /** * 基于已有配置构建预设 YAML 文本块 */ function buildPresetYamlBlock(key, cfg) { const obj = { [key]: cfg }; const dumped = jsyaml.dump(obj, { indent: 2, lineWidth: -1, quotingType: '"', forceQuotes: false }); return dumped.split('\n').map(l => l ? ' ' + l : l).join('\n'); } // ── 弹窗：新增时间段 ── let _tlNewPeriodTarget = ''; window.openTlNewPeriodModal = function(presetName) { _tlNewPeriodTarget = presetName; document.getElementById('tl-new-period-key').value = ''; document.getElementById('tl-new-period-name').value = ''; document.getElementById('tl-new-period-start').value = '09:00'; document.getElementById('tl-new-period-end').value = '11:00'; document.getElementById('tl-new-period-modal').classList.remove('hidden'); } window.closeTlNewPeriodModal = function() { document.getElementById('tl-new-period-modal').classList.add('hidden'); } window.confirmTlNewPeriod = function() { const key = document.getElementById('tl-new-period-key').value.trim(); const name = document.getElementById('tl-new-period-name').value.trim(); const start = document.getElementById('tl-new-period-start').value; const end = document.getElementById('tl-new-period-end').value; if (!key) { showToast('请输入时间段标识 (key)', 'error'); return; } if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(key)) { showToast('key 仅支持英文、数字和下划线', 'error'); return; } if (!name) { showToast('请输入显示名称', 'error'); return; } if (!start || !end) { showToast('请设置开始和结束时间', 'error'); return; } if (start === end) { showToast('开始时间和结束时间不能相同', 'error'); return; } const data = parseTimelineData(); const presetCfg = getPresetConfig(data, _tlNewPeriodTarget); if (presetCfg?.periods?.[key]) { showToast(`时间段「${key}」已存在`, 'error'); return; } const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, _tlNewPeriodTarget); if (!sectionInfo) { showToast('未找到预设配置段', 'error'); return; } const periodsLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'periods'); if (periodsLine < 0) { showToast('未找到 periods 配置段', 'error'); return; } const periodsIndent = lines[periodsLine].search(/\S/); const periodsContent = lines[periodsLine].trim(); const childIndent = periodsIndent + 2; const periodIndent = childIndent + 2; const indent = ' '.repeat(childIndent); const subIndent = ' '.repeat(periodIndent); const newPeriodLines = [ `${indent}${key}:`, `${subIndent}name: "${name}"`, `${subIndent}start: "${start}"`, `${subIndent}end: "${end}"`, `${subIndent}collect: true`, `${subIndent}analyze: false`, `${subIndent}push: true`, `${subIndent}report_mode: current` ]; if (periodsContent === 'periods: {}' || periodsContent === 'periods:{}') { lines[periodsLine] = ' '.repeat(periodsIndent) + 'periods:'; lines.splice(periodsLine + 1, 0, ...newPeriodLines); } else { const periodsEnd = findBlockEnd(lines, periodsLine, periodsIndent, sectionInfo.end); lines.splice(periodsEnd, 0, ...newPeriodLines); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); closeTlNewPeriodModal(); syncTimelineToUI(); showToast(`时间段「${name}」添加成功`, 'success'); } // ── 删除时间段 ── window.deleteTlPeriod = function(presetName, periodKey) { const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (!config) return; const refs = []; const dayPlans = config.day_plans || {}; Object.entries(dayPlans).forEach(([planName, plan]) => { if ((plan.periods || []).includes(periodKey)) refs.push(planName); }); const periodName = config.periods?.[periodKey]?.name || periodKey; let msg = `确定删除时间段「${periodName}」？`; if (refs.length > 0) { msg += `\n\n⚠️ 该时间段被以下日计划引用，将同时移除引用：\n${refs.map(r => ' • ' + r).join('\n')}`; } if (!confirm(msg)) return; const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const periodsLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'periods'); if (periodsLine >= 0) { const periodsIndent = lines[periodsLine].search(/\S/); const periodsEnd = findBlockEnd(lines, periodsLine, periodsIndent, sectionInfo.end); const periodLine = findChildKey(lines, periodsLine, periodsEnd, periodsIndent, periodKey); if (periodLine >= 0) { const periodIndent = lines[periodLine].search(/\S/); const periodEnd = findBlockEnd(lines, periodLine, periodIndent, periodsEnd); lines.splice(periodLine, periodEnd - periodLine); } } if (refs.length > 0) { const updatedSection = findPresetSection(lines, presetName); if (updatedSection) removePeriodFromDayPlans(lines, updatedSection, periodKey); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); showToast(`时间段「${periodName}」已删除`, 'success'); } // ── 复制时间段 ── window.duplicateTlPeriod = function(presetName, periodKey) { const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (!config?.periods?.[periodKey]) return; let newKey = periodKey + '_copy'; let i = 2; while (config.periods[newKey]) { newKey = periodKey + '_copy' + i; i++; } const src = config.periods[periodKey]; const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const periodsLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'periods'); if (periodsLine < 0) return; const periodsIndent = lines[periodsLine].search(/\S/); const periodsEnd = findBlockEnd(lines, periodsLine, periodsIndent, sectionInfo.end); const srcLine = findChildKey(lines, periodsLine, periodsEnd, periodsIndent, periodKey); if (srcLine < 0) return; const srcIndent = lines[srcLine].search(/\S/); const srcEnd = findBlockEnd(lines, srcLine, srcIndent, periodsEnd); const copiedLines = []; for (let li = srcLine; li < srcEnd; li++) { let line = lines[li]; if (li === srcLine) { line = line.replace(periodKey, newKey); } copiedLines.push(line); } for (let li = 0; li < copiedLines.length; li++) { const m = copiedLines[li].match(/^(\s*name:\s*).+$/); if (m) { const newName = (src.name || periodKey) + ' (副本)'; copiedLines[li] = `${m[1]}"${newName}"`; break; } } lines.splice(srcEnd, 0, ...copiedLines); editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); showToast(`已复制为「${newKey}」`, 'success'); } // ── 删除预设模式 ── const PROTECTED_PRESETS = ['morning_evening', 'always_on', 'office_hours', 'night_owl']; window.deleteTlPreset = function(presetName) { if (PROTECTED_PRESETS.includes(presetName)) { showToast('内置预设不可删除，可使用复制功能', 'warning'); return; } if (presetName === 'custom') { showToast('custom 模式不可删除', 'warning'); return; } const data = parseTimelineData(); const cfg = data?.presets?.[presetName]; const displayName = cfg?.name || presetName; if (!confirm(`确定删除调度模式「${displayName}」？\n此操作不可撤销。`)) return; const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; lines.splice(sectionInfo.start, sectionInfo.end - sectionInfo.start); editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); if (getActivePreset() === presetName) { selectTimelinePreset('morning_evening'); } else { syncTimelineToUI(); } showToast(`调度模式「${displayName}」已删除`, 'success'); } // ── 复制预设模式 ── window.duplicateTlPreset = function(presetName) { const data = parseTimelineData(); const src = getPresetConfig(data, presetName); if (!src) return; openTlNewPresetModal(); const origName = src.name || presetName; document.getElementById('tl-new-preset-key').value = presetName + '_copy'; document.getElementById('tl-new-preset-name').value = origName + ' (副本)'; document.getElementById('tl-new-preset-desc').value = src.description || ''; document.getElementById('tl-new-preset-template').value = presetName; } // ── 新增日计划 ── window.addTlDayPlan = function(presetName) { const planKey = prompt('请输入日计划标识 (key)，如 holiday：'); if (!planKey) return; if (!/^[a-zA-Z_][a-zA-Z0-9_]*$/.test(planKey)) { showToast('key 仅支持英文、数字和下划线', 'error'); return; } const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (config?.day_plans?.[planKey]) { showToast(`日计划「${planKey}」已存在`, 'error'); return; } const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionInfo.end); const indent = ' '.repeat(dpIndent + 2); const subIndent = ' '.repeat(dpIndent + 4); lines.splice(dpEnd, 0, `${indent}${planKey}:`, `${subIndent}periods: []` ); editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); showToast(`日计划「${planKey}」已添加`, 'success'); } // ── 删除日计划 ── window.deleteTlDayPlan = function(presetName, planKey) { const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (!config) return; const weekMap = config.week_map || {}; const refs = []; for (let d = 1; d <= 7; d++) { const v = weekMap[d] || weekMap[String(d)]; if (v === planKey) refs.push(DAY_NAMES[d - 1]); } if (refs.length > 0) { showToast(`无法删除：「${planKey}」正在被 ${refs.join('、')} 使用。请先修改周映射。`, 'error'); return; } if (!confirm(`确定删除日计划「${planKey}」？`)) return; const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionInfo.end); const planLine = findChildKey(lines, dayPlansLine, dpEnd, dpIndent, planKey); if (planLine < 0) return; const planIndent = lines[planLine].search(/\S/); const planEnd = findBlockEnd(lines, planLine, planIndent, dpEnd); lines.splice(planLine, planEnd - planLine); editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); showToast(`日计划「${planKey}」已删除`, 'success'); } // ── 日计划中添加/移除时间段引用 ── window.addPeriodToDayPlan = function(presetName, planKey, periodKey) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionInfo.end); const planLine = findChildKey(lines, dayPlansLine, dpEnd, dpIndent, planKey); if (planLine < 0) return; const planIndent = lines[planLine].search(/\S/); const planEnd = findBlockEnd(lines, planLine, planIndent, dpEnd); const periodsLine = findChildKey(lines, planLine, planEnd, planIndent, 'periods'); if (periodsLine < 0) return; const periodsContent = lines[periodsLine].trim(); if (periodsContent === 'periods: []' || periodsContent === 'periods:[]') { const pIndent = ' '.repeat(lines[periodsLine].search(/\S/)); lines[periodsLine] = `${pIndent}periods:`; lines.splice(periodsLine + 1, 0, `${pIndent} - ${periodKey}`); } else { const inlineMatch = lines[periodsLine].match(/^(\s*periods:\s*)\[([^\]]*)\]/); if (inlineMatch) { const existing = inlineMatch[2].split(',').map(s => s.trim()).filter(Boolean); // 保持引号风格一致 const hasQuotes = existing.length > 0 && existing[0].startsWith('"'); existing.push(hasQuotes ? `"${periodKey}"` : periodKey); lines[periodsLine] = `${inlineMatch[1]}[${existing.join(', ')}]`; } else { const pIndent = ' '.repeat(lines[periodsLine].search(/\S/) + 2); const listEnd = findBlockEnd(lines, periodsLine, lines[periodsLine].search(/\S/), planEnd); lines.splice(listEnd, 0, `${pIndent}- ${periodKey}`); } } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); } window.removePeriodFromDayPlanUI = function(presetName, planKey, periodKey) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; removePeriodFromDayPlanInLines(lines, sectionInfo, planKey, periodKey); editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); syncTimelineToUI(); } // ── 周映射快捷操作 ── window.tlWeekMapQuick = function(presetName, mode) { const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (!config) return; const dayPlanKeys = Object.keys(config.day_plans || {}); if (dayPlanKeys.length === 0) { showToast('没有可用的日计划', 'error'); return; } let mapping = {}; if (mode === 'all_same') { const plan = dayPlanKeys[0]; for (let d = 1; d <= 7; d++) mapping[d] = plan; } else if (mode === 'weekday_same') { const plan = dayPlanKeys[0]; for (let d = 1; d <= 5; d++) mapping[d] = plan; const wm = config.week_map || {}; mapping[6] = wm[6] || wm['6'] || plan; mapping[7] = wm[7] || wm['7'] || plan; } else if (mode === 'weekday_weekend') { if (dayPlanKeys.length < 2) { showToast('需要至少两个日计划来分离工作日/周末', 'warning'); return; } const wd = dayPlanKeys[0]; const we = dayPlanKeys[1]; for (let d = 1; d <= 5; d++) mapping[d] = wd; mapping[6] = we; mapping[7] = we; } for (let d = 1; d <= 7; d++) { if (mapping[d]) onTlWeekMap(presetName, d, mapping[d]); } showToast('周映射已更新', 'success'); } // ── 辅助函数 ── /** * 定位预设配置段的起始行和结束行 */ function findPresetSection(lines, presetName) { const isCustom = presetName === 'custom'; let start = -1; let indent = 0; if (isCustom) { for (let i = 0; i < lines.length; i++) { if (/^custom:\s*/.test(lines[i])) { start = i; indent = 0; break; } } } else { let inPresets = false; for (let i = 0; i < lines.length; i++) { const line = lines[i]; if (/^presets:\s*/.test(line)) { inPresets = true; continue; } if (inPresets && /^\S/.test(line) && !line.startsWith('#') && line.trim() !== '') break; if (inPresets) { const m = line.match(/^(\s+)(\S+):\s*/); if (m && m[2] === presetName) { start = i; indent = m[1].length; break; } } } } if (start < 0) return null; let end = lines.length; for (let i = start + 1; i < lines.length; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const curIndent = line.search(/\S/); if (curIndent <= indent) { end = i; break; } } return { start, end, indent }; } /** * 从 day_plans 中批量移除对某 period 的引用 */ function removePeriodFromDayPlans(lines, sectionInfo, periodKey) { const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const sectionEnd = findBlockEnd(lines, sectionInfo.start, sectionInfo.indent, lines.length); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionEnd); for (let i = dayPlansLine + 1; i < dpEnd; i++) { const line = lines[i]; if (line.trim() === '' || line.trim().startsWith('#')) continue; const listMatch = line.match(/^(\s*)-\s*(\S+)\s*$/); if (listMatch && listMatch[2] === periodKey) { lines.splice(i, 1); i--; continue; } const inlineMatch = line.match(/^(\s*periods:\s*)\[([^\]]*)\]/); if (inlineMatch) { const items = inlineMatch[2].split(',').map(s => s.trim()).filter(s => { const bare = s.replace(/^["']|["']$/g, ''); return bare && bare !== periodKey; }); lines[i] = items.length > 0 ? `${inlineMatch[1]}[${items.join(', ')}]` : `${inlineMatch[1]}[]`; } } } /** * 从指定 day_plan 中移除单个 period 引用 */ function removePeriodFromDayPlanInLines(lines, sectionInfo, planKey, periodKey) { const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionInfo.end); const planLine = findChildKey(lines, dayPlansLine, dpEnd, dpIndent, planKey); if (planLine < 0) return; const planIndent = lines[planLine].search(/\S/); const planEnd = findBlockEnd(lines, planLine, planIndent, dpEnd); const periodsLine = findChildKey(lines, planLine, planEnd, planIndent, 'periods'); if (periodsLine < 0) return; const inlineMatch = lines[periodsLine].match(/^(\s*periods:\s*)\[([^\]]*)\]/); if (inlineMatch) { const items = inlineMatch[2].split(',').map(s => s.trim()).filter(s => { const bare = s.replace(/^["']|["']$/g, ''); return bare && bare !== periodKey; }); lines[periodsLine] = items.length > 0 ? `${inlineMatch[1]}[${items.join(', ')}]` : `${inlineMatch[1]}[]`; return; } const pEnd = findBlockEnd(lines, periodsLine, lines[periodsLine].search(/\S/), planEnd); for (let i = periodsLine + 1; i < pEnd; i++) { const m = lines[i].match(/^(\s*)-\s*(\S+)\s*$/); if (m && m[2] === periodKey) { lines.splice(i, 1); return; } } } // ========================================== // 15. 后续优化功能 // ========================================== // ── 1.3 / 3A.4 内联编辑（双击编辑文本）── /** * 预设卡片名称/描述内联编辑 */ window.tlInlineEdit = function(el, presetName, field, currentValue) { if (el.querySelector('input')) return; const original = currentValue; const isName = field === 'name'; const input = document.createElement('input'); input.type = 'text'; input.value = original; input.className = `tl-inline-input ${isName ? 'text-sm font-bold' : 'text-[10px]'}`; input.style.width = '100%'; el.textContent = ''; el.appendChild(input); input.focus(); input.select(); const commit = () => { const newVal = input.value.trim(); if (newVal && newVal !== original) { updatePresetMeta(presetName, field, newVal); } syncTimelineToUI(); }; input.addEventListener('blur', commit); input.addEventListener('keydown', e => { if (e.key === 'Enter') { e.preventDefault(); input.blur(); } if (e.key === 'Escape') { el.textContent = original; } }); } /** * 更新预设顶层的 name / description 字段 */ function updatePresetMeta(presetName, field, value) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const lineIdx = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, field); if (lineIdx >= 0) { replaceLineValue(lines, lineIdx, value); } else { const indent = ' '.repeat(sectionInfo.indent + 2); lines.splice(sectionInfo.start + 1, 0, `${indent}${field}: "${value}"`); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); } /** * 时间段名称内联编辑 */ window.tlInlineEditPeriod = function(el, presetName, periodKey, currentValue) { if (el.querySelector('input')) return; const original = currentValue; const input = document.createElement('input'); input.type = 'text'; input.value = original; input.className = 'tl-inline-input text-sm font-bold'; input.style.width = Math.max(80, original.length * 14) + 'px'; el.textContent = ''; el.appendChild(input); input.focus(); input.select(); const commit = () => { const newVal = input.value.trim(); if (newVal && newVal !== original) { updateTimelineField(presetName, periodKey, 'name', newVal); } syncTimelineToUI(); }; input.addEventListener('blur', commit); input.addEventListener('keydown', e => { if (e.key === 'Enter') { e.preventDefault(); input.blur(); } if (e.key === 'Escape') { el.textContent = original; } }); } // ── 2.2 周视图空白区域点击 → 显示日计划名称 ── window.onTlBarClick = function(event, presetName, dayNum) { if (event.target.closest('.tl-period-block')) return; const data = parseTimelineData(); const config = getPresetConfig(data, presetName); if (!config) return; const weekMap = config.week_map || {}; const planKey = weekMap[dayNum] || weekMap[String(dayNum)] || '(未设置)'; hideTlTooltip(); const el = document.createElement('div'); el.className = 'tl-tooltip'; el.innerHTML = `
${DAY_NAMES[dayNum - 1]}
日计划: ${planKey}
使用 default 配置
`; document.body.appendChild(el); tlTooltipEl = el; const rect = event.currentTarget.getBoundingClientRect(); const x = event.clientX; el.style.left = (x - el.offsetWidth / 2) + 'px'; el.style.top = (rect.top - el.offsetHeight - 8) + 'px'; const elRect = el.getBoundingClientRect(); if (elRect.left < 4) el.style.left = '4px'; if (elRect.right > window.innerWidth - 4) el.style.left = (window.innerWidth - el.offsetWidth - 4) + 'px'; if (elRect.top < 4) el.style.top = (rect.bottom + 8) + 'px'; setTimeout(() => { if (tlTooltipEl === el) hideTlTooltip(); }, 2000); } // ── 3B.5 日计划 Tag 拖拽排序 ── /** * 为日计划中的 period tag 容器初始化 SortableJS */ function initDayPlanSortable(presetName) { document.querySelectorAll('.tl-dayplan-sortable').forEach(container => { const planKey = container.dataset.planKey; if (!planKey) return; new Sortable(container, { animation: 150, ghostClass: 'tl-tag-ghost', dragClass: 'tl-tag-drag', draggable: '.tl-period-tag', filter: '.tl-add-period-select, .tl-tag-remove', preventOnFilter: false, onEnd: function() { const items = []; container.querySelectorAll('.tl-period-tag').forEach(tag => { const key = tag.dataset.periodKey; if (key) items.push(key); }); reorderDayPlanPeriods(presetName, planKey, items); } }); }); } /** * 重新排列 day_plan 中 periods 的顺序 */ function reorderDayPlanPeriods(presetName, planKey, orderedKeys) { const editor = document.getElementById('timeline-editor'); const lines = editor.value.split('\n'); const sectionInfo = findPresetSection(lines, presetName); if (!sectionInfo) return; const dayPlansLine = findChildKey(lines, sectionInfo.start, sectionInfo.end, sectionInfo.indent, 'day_plans'); if (dayPlansLine < 0) return; const dpIndent = lines[dayPlansLine].search(/\S/); const dpEnd = findBlockEnd(lines, dayPlansLine, dpIndent, sectionInfo.end); const planLine = findChildKey(lines, dayPlansLine, dpEnd, dpIndent, planKey); if (planLine < 0) return; const planIndent = lines[planLine].search(/\S/); const planEnd = findBlockEnd(lines, planLine, planIndent, dpEnd); const periodsLine = findChildKey(lines, planLine, planEnd, planIndent, 'periods'); if (periodsLine < 0) return; const inlineMatch = lines[periodsLine].match(/^(\s*periods:\s*)\[([^\]]*)\]/); if (inlineMatch) { lines[periodsLine] = `${inlineMatch[1]}[${orderedKeys.join(', ')}]`; } else { const pIndent = lines[periodsLine].search(/\S/); const pEnd = findBlockEnd(lines, periodsLine, pIndent, planEnd); lines.splice(periodsLine + 1, pEnd - periodsLine - 1); const itemIndent = ' '.repeat(pIndent + 2); const newItems = orderedKeys.map(k => `${itemIndent}- ${k}`); lines.splice(periodsLine + 1, 0, ...newItems); } editor.value = lines.join('\n'); currentTimeline = editor.value; updateBackdrop('timeline-editor', 'timeline-backdrop'); debounceSaveTimeline(); clearTimeout(window._tlRenderTimer); window._tlRenderTimer = setTimeout(() => syncTimelineToUI(), 500); } // ========================================== // 支持侧栏 折叠/展开 // ========================================== function toggleSupportSidebar() { const wrap = document.querySelector('.support-sidebar-wrap'); const btn = document.getElementById('sidebar-toggle-btn'); const isCollapsed = wrap.classList.toggle('collapsed'); btn.classList.toggle('is-collapsed', isCollapsed); btn.title = isCollapsed ? '展开侧栏' : '收起侧栏'; } ================================================ FILE: docs/assets/style.css ================================================ /* 编辑器区域滚动条 */ #yaml-editor::-webkit-scrollbar, #frequency-editor::-webkit-scrollbar, #timeline-editor::-webkit-scrollbar, #yaml-backdrop::-webkit-scrollbar, #frequency-backdrop::-webkit-scrollbar, #timeline-backdrop::-webkit-scrollbar { width: 10px; height: 10px; } #yaml-editor::-webkit-scrollbar-track, #frequency-editor::-webkit-scrollbar-track, #timeline-editor::-webkit-scrollbar-track, #yaml-backdrop::-webkit-scrollbar-track, #frequency-backdrop::-webkit-scrollbar-track, #timeline-backdrop::-webkit-scrollbar-track { background: #1e1e1e; } #yaml-editor::-webkit-scrollbar-thumb, #frequency-editor::-webkit-scrollbar-thumb, #timeline-editor::-webkit-scrollbar-thumb, #yaml-backdrop::-webkit-scrollbar-thumb, #frequency-backdrop::-webkit-scrollbar-thumb, #timeline-backdrop::-webkit-scrollbar-thumb { background: #424242; border-radius: 0; } #yaml-editor::-webkit-scrollbar-thumb:hover, #frequency-editor::-webkit-scrollbar-thumb:hover, #timeline-editor::-webkit-scrollbar-thumb:hover, #yaml-backdrop::-webkit-scrollbar-thumb:hover, #frequency-backdrop::-webkit-scrollbar-thumb:hover, #timeline-backdrop::-webkit-scrollbar-thumb:hover { background: #4f4f4f; } /* 高亮编辑器容器 */ .highlight-editor-wrap { position: relative; flex: 1; display: flex; overflow: hidden; } /* 高亮背景层 */ .highlight-backdrop { position: absolute; top: 0; left: 0; right: 0; bottom: 0; padding: 1rem; margin: 0; border: none; font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; font-size: 0.75rem; line-height: 1.625; white-space: pre-wrap; word-wrap: break-word; overflow: auto; background: #1e1e1e; color: #d4d4d4; pointer-events: none; z-index: 1; } /* 透明输入层 */ .highlight-textarea { position: absolute; top: 0; left: 0; right: 0; bottom: 0; padding: 1rem; margin: 0; border: none; font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace; font-size: 0.75rem; line-height: 1.625; overflow: auto; background: transparent; color: transparent; caret-color: #d4d4d4; resize: none; outline: none; z-index: 2; } /* 注释样式 - 灰色 */ .syntax-comment { color: #6a9955; } /* 右侧面板滚动条 */ #modules-container::-webkit-scrollbar { width: 8px; } #modules-container::-webkit-scrollbar-track { background: transparent; } #modules-container::-webkit-scrollbar-thumb { background: #cbd5e1; border-radius: 4px; } /* 模块卡片样式 */ .module-card { background: white; border-radius: 0.5rem; /* rounded-lg */ border: 1px solid #e5e7eb; /* border-gray-200 */ overflow: hidden; transition: all 0.2s; } /* 激活态（可编辑） */ .module-card.active { box-shadow: 0 1px 3px 0 rgba(0, 0, 0, 0.1), 0 1px 2px 0 rgba(0, 0, 0, 0.06); } .module-card.active:hover { box-shadow: 0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06); border-color: #bfdbfe; /* blue-200 */ } .module-card.active .module-header { background-color: #fff; border-bottom: 1px solid #f3f4f6; color: #111827; } /* 禁用态（灰色/只读） */ .module-card.disabled { background-color: #f9fafb; /* gray-50 */ opacity: 0.8; } .module-card.disabled .module-header { background-color: #f3f4f6; /* gray-100 */ color: #6b7280; /* gray-500 */ cursor: not-allowed; } .module-card.disabled .module-body { display: none; } .module-card.disabled .locked-badge { display: inline-flex; } /* 输入控件统一 */ input[type="text"], input[type="password"], input[type="number"], select { font-size: 0.875rem; /* text-sm */ line-height: 1.25rem; padding: 0.5rem 0.75rem; border-radius: 0.375rem; border-width: 1px; border-color: #d1d5db; /* gray-300 */ width: 100%; outline: 2px solid transparent; transition: all 0.15s; } input:focus, select:focus { border-color: #3b82f6; /* blue-500 */ box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.2); } /* 开关样式 (Checkbox Toggle) */ .toggle-checkbox:checked { right: 0; border-color: #3b82f6; } .toggle-checkbox:checked + .toggle-label { background-color: #3b82f6; } /* 列表样式 (Platforms & RSS & Sortable) */ .sortable-list-item { background: #f8fafc; border: 1px solid #e2e8f0; margin-bottom: 0.5rem; border-radius: 0.375rem; transition: all 0.2s; } .sortable-list-item:hover { border-color: #cbd5e1; background: #f1f5f9; } .sortable-handle { cursor: grab; color: #94a3b8; } .sortable-handle:hover { color: #64748b; } .sortable-ghost { background: #e2e8f0; opacity: 0.5; } /* 禁用状态的勾选框 */ input[type="checkbox"]:disabled { cursor: not-allowed; opacity: 0.5; } /* Tab 切换样式 */ .tab-button { transition: all 0.2s; } .tab-button.active { color: #d4d4d4; border-color: #3b82f6; } .tab-content.hidden { display: none; } /* 标签输入样式 */ .tag-input-container { display: flex; flex-wrap: wrap; gap: 0.5rem; padding: 0.5rem; border: 1px solid #d1d5db; border-radius: 0.375rem; background: white; min-height: 42px; } .tag-item { display: inline-flex; align-items: center; gap: 0.25rem; padding: 0.25rem 0.5rem; background: #3b82f6; color: white; border-radius: 0.25rem; font-size: 0.875rem; } .tag-item button { background: none; border: none; color: white; cursor: pointer; padding: 0; font-size: 1rem; line-height: 1; } .tag-input { flex: 1; border: none; outline: none; min-width: 120px; font-size: 0.875rem; } /* 词组卡片样式 */ .word-group-card { background: white; border: 1px solid #e5e7eb; border-radius: 0.5rem; padding: 1rem; transition: all 0.2s; } .word-group-card:hover { border-color: #3b82f6; box-shadow: 0 2px 4px rgba(0,0,0,0.1); } /* 插入区域样式 */ .insert-zone { position: relative; height: 8px; margin: 0.5rem 0; display: flex; align-items: center; justify-content: center; transition: all 0.2s; } .insert-zone:hover { height: 32px; } .insert-button { opacity: 0; visibility: hidden; width: 32px; height: 32px; border-radius: 50%; background: linear-gradient(135deg, #3b82f6, #2563eb); color: white; border: 2px solid white; box-shadow: 0 2px 8px rgba(59, 130, 246, 0.4); display: flex; align-items: center; justify-content: center; cursor: pointer; transition: all 0.2s; font-size: 14px; } .insert-zone:hover .insert-button { opacity: 1; visibility: visible; } .insert-button:hover { transform: scale(1.1); box-shadow: 0 4px 12px rgba(59, 130, 246, 0.6); background: linear-gradient(135deg, #2563eb, #1d4ed8); } .insert-button:active { transform: scale(0.95); } /* 编辑区域恢复默认鼠标样式 */ .word-group-card .editable-area { cursor: default; } .word-group-card .editable-area input { cursor: text; } .word-group-card .editable-area button { cursor: pointer; } .word-group-card .editable-area .tag-item { cursor: pointer; } /* 拖拽手柄样式 */ .drag-handle { cursor: grab; transition: all 0.2s; } .drag-handle:active { cursor: grabbing; } /* SortableJS 拖拽样式 */ .sortable-ghost { opacity: 0.4; background: #dbeafe; border: 2px dashed #3b82f6; } .sortable-chosen { background: #f0f9ff; border-color: #3b82f6; } .sortable-drag { opacity: 0.8; box-shadow: 0 10px 20px rgba(0,0,0,0.2); transform: rotate(2deg); } /* 独立区域复选框组 */ .checkbox-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(140px, 1fr)); gap: 0.75rem; } .checkbox-card { display: flex; align-items: center; padding: 0.5rem; border: 1px solid #e5e7eb; border-radius: 0.375rem; background-color: #fff; cursor: pointer; transition: all 0.15s; } .checkbox-card:hover { border-color: #93c5fd; background-color: #eff6ff; } .checkbox-card input:checked + span { color: #2563eb; font-weight: 500; } /* ========================================== 拖拽上传遮罩层 ========================================== */ .drop-overlay { position: absolute; top: 0; left: 0; right: 0; bottom: 0; background: rgba(59, 130, 246, 0.9); display: flex; align-items: center; justify-content: center; z-index: 100; pointer-events: all; } .drop-overlay.hidden { display: none; } .drop-overlay-content { text-align: center; color: white; } .drop-overlay-content i { font-size: 3rem; margin-bottom: 0.5rem; animation: bounce 1s infinite; } @keyframes bounce { 0%, 100% { transform: translateY(0); } 50% { transform: translateY(-10px); } } /* ========================================== Toast 提示 ========================================== */ .toast-notification { position: fixed; bottom: 24px; right: 24px; display: flex; align-items: center; gap: 0.75rem; padding: 0.875rem 1.25rem; border-radius: 0.5rem; font-size: 0.875rem; font-weight: 500; box-shadow: 0 10px 15px -3px rgba(0, 0, 0, 0.1), 0 4px 6px -2px rgba(0, 0, 0, 0.05); z-index: 9999; opacity: 0; transform: translateY(20px); transition: all 0.3s ease; } .toast-notification.show { opacity: 1; transform: translateY(0); } .toast-notification i { font-size: 1.125rem; } /* Toast 类型样式 */ .toast-success { background: #10b981; color: white; } .toast-error { background: #ef4444; color: white; } .toast-info { background: #3b82f6; color: white; } .toast-warning { background: #f59e0b; color: white; } /* ========================================== 弹窗样式 ========================================== */ .modal-overlay { position: fixed; top: 0; left: 0; right: 0; bottom: 0; background: rgba(0, 0, 0, 0.5); display: flex; align-items: center; justify-content: center; z-index: 1000; } .modal-overlay.hidden { display: none; } .modal-content { background: white; border-radius: 0.75rem; padding: 1.5rem; max-width: 450px; width: 90%; max-height: 90vh; overflow-y: auto; box-shadow: 0 25px 50px -12px rgba(0, 0, 0, 0.25); } /* 弹簧跳动动画 */ @keyframes spring-in { 0% { transform: scale(0.5); opacity: 0; } 60% { transform: scale(1.1); } 80% { transform: scale(0.95); } 100% { transform: scale(1); opacity: 1; } } .support-modal-content { animation: spring-in 0.6s cubic-bezier(0.175, 0.885, 0.32, 1.275); background: #ffffff; border: none; border-radius: 1.5rem; } /* ========================================== Timeline 编辑器样式 ========================================== */ /* 预设模式选择卡片 */ .tl-preset-card { border: 2px solid #e5e7eb; border-radius: 0.75rem; padding: 0.875rem; cursor: pointer; transition: all 0.2s; background: white; position: relative; } .tl-preset-card:hover { border-color: #93c5fd; background: #f0f7ff; } .tl-preset-card.selected { border-color: #3b82f6; background: #eff6ff; box-shadow: 0 0 0 3px rgba(59, 130, 246, 0.15); } .tl-preset-card .tl-card-icon { width: 2rem; height: 2rem; border-radius: 0.5rem; display: flex; align-items: center; justify-content: center; font-size: 0.875rem; flex-shrink: 0; } .tl-preset-card .tl-recommend-badge { position: absolute; top: -1px; right: -1px; background: linear-gradient(135deg, #f59e0b, #ef4444); color: white; font-size: 0.625rem; font-weight: 700; padding: 0.125rem 0.5rem; border-radius: 0 0.625rem 0 0.5rem; } /* 周视图时间线 */ .tl-week-view { background: white; border: 1px solid #e5e7eb; border-radius: 0.75rem; padding: 1rem; overflow-x: auto; } .tl-week-row { display: flex; align-items: center; height: 2.25rem; margin-bottom: 0.25rem; } .tl-week-row:last-child { margin-bottom: 0; } .tl-day-label { width: 2.5rem; flex-shrink: 0; font-size: 0.6875rem; font-weight: 600; color: #6b7280; text-align: right; padding-right: 0.5rem; } .tl-day-label.today { color: #3b82f6; font-weight: 700; } .tl-timeline-bar { flex: 1; height: 1.75rem; background: #f1f5f9; border-radius: 0.25rem; position: relative; min-width: 480px; overflow: hidden; } .tl-period-block { position: absolute; top: 2px; bottom: 2px; border-radius: 0.1875rem; cursor: pointer; transition: filter 0.15s, transform 0.15s; display: flex; align-items: center; justify-content: center; overflow: hidden; z-index: 1; } .tl-period-block:hover { filter: brightness(1.1); transform: scaleY(1.15); z-index: 2; } .tl-period-block .tl-block-label { font-size: 0.5625rem; font-weight: 600; color: rgba(255,255,255,0.9); white-space: nowrap; text-overflow: ellipsis; overflow: hidden; padding: 0 0.25rem; text-shadow: 0 1px 2px rgba(0,0,0,0.2); } /* 时间段颜色 */ .tl-block-push { background: #3b82f6; } .tl-block-analyze { background: #8b5cf6; } .tl-block-push-analyze { background: #6366f1; } .tl-block-collect { background: #94a3b8; } .tl-block-silent { background: #cbd5e1; } /* 时间刻度 */ .tl-hour-markers { display: flex; padding-left: 2.5rem; margin-bottom: 0.25rem; } .tl-hour-marker { font-size: 0.5625rem; color: #9ca3af; text-align: center; } /* 图例 */ .tl-legend { display: flex; gap: 0.75rem; flex-wrap: wrap; padding-top: 0.5rem; border-top: 1px solid #f3f4f6; margin-top: 0.5rem; } .tl-legend-item { display: flex; align-items: center; gap: 0.25rem; font-size: 0.625rem; color: #6b7280; } .tl-legend-color { width: 0.75rem; height: 0.5rem; border-radius: 0.125rem; } /* 时间段 Tooltip */ .tl-tooltip { position: fixed; background: #1f2937; color: white; padding: 0.5rem 0.75rem; border-radius: 0.375rem; font-size: 0.75rem; z-index: 1000; pointer-events: none; box-shadow: 0 4px 12px rgba(0,0,0,0.2); max-width: 220px; } .tl-tooltip::after { content: ''; position: absolute; bottom: -4px; left: 50%; transform: translateX(-50%); border-left: 5px solid transparent; border-right: 5px solid transparent; border-top: 5px solid #1f2937; } /* Custom 模式编辑面板 */ .tl-section-title { font-size: 0.75rem; font-weight: 700; color: #374151; display: flex; align-items: center; gap: 0.5rem; margin-bottom: 0.75rem; } .tl-section-title i { color: #3b82f6; font-size: 0.6875rem; } .tl-period-card { background: white; border: 1px solid #e5e7eb; border-radius: 0.5rem; padding: 0.75rem; transition: all 0.2s; } .tl-period-card:hover { border-color: #93c5fd; box-shadow: 0 2px 4px rgba(0,0,0,0.05); } .tl-toggle-row { display: flex; align-items: center; gap: 0.75rem; flex-wrap: wrap; } .tl-toggle-item { display: flex; align-items: center; gap: 0.375rem; font-size: 0.6875rem; color: #4b5563; } .tl-toggle-item.on { color: #2563eb; font-weight: 600; } .tl-toggle-item.off { color: #9ca3af; } /* Timeline 小型 toggle 开关 */ .tl-toggle-item .toggle-checkbox { width: 1rem; height: 1rem; border-width: 3px; } .tl-toggle-item .toggle-label { height: 1rem; } .tl-toggle-item .toggle-checkbox:checked { right: 0; border-color: #3b82f6; } .tl-toggle-item .toggle-checkbox:checked + .toggle-label { background-color: #3b82f6; } /* 日计划和周映射 */ .tl-dayplan-row { display: flex; align-items: center; gap: 0.5rem; padding: 0.375rem 0; } .tl-dayplan-label { width: 3.5rem; font-size: 0.6875rem; font-weight: 600; color: #374151; flex-shrink: 0; } .tl-weekmap-select { font-size: 0.75rem; padding: 0.25rem 0.5rem; border: 1px solid #d1d5db; border-radius: 0.25rem; background: white; flex: 1; max-width: 200px; } .tl-weekmap-select:focus { border-color: #3b82f6; box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.2); outline: none; } /* Default 配置折叠面板 */ .tl-collapsible { border: 1px solid #e5e7eb; border-radius: 0.5rem; overflow: hidden; } .tl-collapsible-header { background: #f9fafb; padding: 0.625rem 0.75rem; cursor: pointer; display: flex; align-items: center; justify-content: space-between; font-size: 0.75rem; font-weight: 600; color: #4b5563; transition: background 0.15s; } .tl-collapsible-header:hover { background: #f3f4f6; } .tl-collapsible-body { padding: 0.75rem; border-top: 1px solid #e5e7eb; } .tl-collapsible-body.collapsed { display: none; } .tl-collapsible-header .fa-chevron-down { transition: transform 0.2s; } .tl-collapsible-header.is-collapsed .fa-chevron-down { transform: rotate(-90deg); } /* Timeline CRUD 新增样式 */ /* 预设卡片操作按钮 */ .tl-card-actions { display: none; position: absolute; top: 0.375rem; right: 0.375rem; gap: 0.25rem; z-index: 2; } .tl-preset-card:hover .tl-card-actions { display: flex; } .tl-card-action-btn { width: 1.5rem; height: 1.5rem; display: flex; align-items: center; justify-content: center; border-radius: 0.375rem; font-size: 0.625rem; color: #9ca3af; background: rgba(255,255,255,0.9); border: 1px solid #e5e7eb; cursor: pointer; transition: all 0.15s; } .tl-card-action-btn:hover { color: #3b82f6; background: white; border-color: #93c5fd; } .tl-card-action-btn.text-red-400:hover { color: #ef4444; border-color: #fca5a5; } /* 新建模式卡片 */ .tl-new-preset-card { border-style: dashed; border-color: #d1d5db; background: #fafafa; } .tl-new-preset-card:hover { border-color: #a78bfa; background: #faf5ff; } /* section 内的新增按钮 */ .tl-add-btn { font-size: 0.625rem; font-weight: 600; color: #3b82f6; background: #eff6ff; border: 1px solid #bfdbfe; border-radius: 0.375rem; padding: 0.125rem 0.5rem; cursor: pointer; transition: all 0.15s; } .tl-add-btn:hover { background: #dbeafe; border-color: #93c5fd; } /* period 卡片内联操作 */ .tl-inline-btn { width: 1.375rem; height: 1.375rem; display: inline-flex; align-items: center; justify-content: center; border-radius: 0.25rem; font-size: 0.625rem; color: #9ca3af; background: transparent; border: none; cursor: pointer; transition: all 0.15s; opacity: 0; } .tl-period-card:hover .tl-inline-btn, .tl-dayplan-card:hover .tl-inline-btn { opacity: 1; } .tl-inline-btn:hover { color: #3b82f6; background: #eff6ff; } .tl-inline-btn.text-red-400:hover { color: #ef4444; background: #fef2f2; } /* 日计划中的 period tag */ .tl-period-tag { display: inline-flex; align-items: center; gap: 0.25rem; font-size: 0.625rem; padding: 0.125rem 0.5rem; border-radius: 9999px; color: white; white-space: nowrap; } .tl-tag-remove { font-size: 0.75rem; font-weight: 700; line-height: 1; color: rgba(255,255,255,0.7); background: none; border: none; cursor: pointer; padding: 0; margin-left: 0.125rem; } .tl-tag-remove:hover { color: white; } /* 添加时间段到日计划的 select */ .tl-add-period-select { font-size: 0.625rem; padding: 0.0625rem 0.375rem; border: 1px dashed #d1d5db; border-radius: 9999px; background: #f9fafb; color: #6b7280; cursor: pointer; transition: all 0.15s; } .tl-add-period-select:hover { border-color: #93c5fd; color: #3b82f6; } /* 周映射快捷按钮 */ .tl-quick-btn { font-size: 0.625rem; font-weight: 500; color: #6b7280; background: #f3f4f6; border: 1px solid #e5e7eb; border-radius: 0.375rem; padding: 0.25rem 0.5rem; cursor: pointer; transition: all 0.15s; } .tl-quick-btn:hover { color: #3b82f6; background: #eff6ff; border-color: #93c5fd; } /* 当前时间指示线 */ .tl-now-line { position: absolute; top: -2px; bottom: -2px; width: 2px; background: #ef4444; z-index: 5; pointer-events: none; } .tl-now-line::before { content: ''; position: absolute; top: -3px; left: -3px; width: 8px; height: 8px; border-radius: 50%; background: #ef4444; } /* 周视图色块点击态 */ .tl-period-block { cursor: pointer; } .tl-period-block:hover { filter: brightness(1.1); box-shadow: 0 0 0 2px rgba(255,255,255,0.6); } /* period 卡片高亮动画 */ .tl-period-highlight { animation: tl-highlight-pulse 1.5s ease-out; } @keyframes tl-highlight-pulse { 0% { box-shadow: 0 0 0 0 rgba(59, 130, 246, 0.5); } 30% { box-shadow: 0 0 0 4px rgba(59, 130, 246, 0.3); } 100% { box-shadow: none; } } /* 内联编辑输入框 */ .tl-inline-input { background: white; border: 1px solid #93c5fd; border-radius: 0.25rem; padding: 0 0.25rem; outline: none; box-shadow: 0 0 0 2px rgba(59, 130, 246, 0.2); color: #1f2937; } .tl-editable { cursor: text; border-radius: 0.25rem; transition: background 0.15s; } .tl-editable:hover { background: rgba(59, 130, 246, 0.06); } /* 日计划 Tag 拖拽排序 */ .tl-period-tag { cursor: grab; } .tl-period-tag:active { cursor: grabbing; } .tl-tag-ghost { opacity: 0.4; } .tl-tag-drag { transform: rotate(2deg); box-shadow: 0 4px 12px rgba(0,0,0,0.15); } /* ========================================== 支持侧栏 ========================================== */ /* 外层容器：承担宽度和 flex 布局角色 */ .support-sidebar-wrap { width: 20%; min-width: 180px; max-width: 280px; overflow: visible; transition: width 0.3s ease, min-width 0.3s ease, max-width 0.3s ease; } .support-sidebar-wrap.collapsed { width: 0; min-width: 0; max-width: 0; } /* 内层侧栏：填满 wrap */ .support-sidebar { width: 100%; height: 100%; overflow: hidden; transition: opacity 0.3s ease; } .support-sidebar-wrap.collapsed .support-sidebar { opacity: 0; pointer-events: none; } /* 折叠/展开按钮 */ .sidebar-toggle-btn { position: absolute; left: 0; top: 50%; transform: translate(-100%, -50%); width: 20px; height: 40px; background: white; border: 1px solid #e5e7eb; border-radius: 6px 0 0 6px; display: flex; align-items: center; justify-content: center; cursor: pointer; z-index: 10; opacity: 0; transition: opacity 0.2s ease, background 0.2s ease; color: #9ca3af; } .support-sidebar-wrap:hover .sidebar-toggle-btn { opacity: 1; } .sidebar-toggle-btn:hover { background: #f3f4f6; color: #6b7280; } /* 折叠后按钮始终可见，箭头朝左 */ .sidebar-toggle-btn.is-collapsed { opacity: 1; } .sidebar-toggle-btn.is-collapsed i { transform: rotate(180deg); } /* 侧栏滚动条 */ .sidebar-scroll::-webkit-scrollbar { width: 4px; } .sidebar-scroll::-webkit-scrollbar-track { background: transparent; } .sidebar-scroll::-webkit-scrollbar-thumb { background: #e5e7eb; border-radius: 2px; } .sidebar-scroll::-webkit-scrollbar-thumb:hover { background: #d1d5db; } /* 侧栏卡片 */ .sidebar-card { background: white; border: 1px solid #f3f4f6; border-radius: 0.75rem; padding: 0.75rem; transition: all 0.3s cubic-bezier(0.175, 0.885, 0.32, 1.275); text-decoration: none; display: block; cursor: pointer; } .sidebar-card:hover { border-color: #e5e7eb; box-shadow: 0 4px 12px rgba(0, 0, 0, 0.06); transform: translateY(-2px); } /* 侧栏卡片图标 */ .sidebar-card-icon { width: 2rem; height: 2rem; border-radius: 0.5rem; display: flex; align-items: center; justify-content: center; font-size: 0.75rem; flex-shrink: 0; transition: all 0.2s ease; } .sidebar-card:hover .sidebar-card-icon { transform: rotate(8deg) scale(1.1); } /* 侧栏 CTA 按钮 */ .sidebar-cta { text-align: center; padding: 0.375rem 0.5rem; border-radius: 0.5rem; font-size: 0.625rem; font-weight: 700; transition: all 0.2s ease; letter-spacing: 0.02em; } /* 侧栏二维码 */ .sidebar-qr { width: 100%; max-width: 120px; aspect-ratio: 1; background: white; border: 1px solid #f3f4f6; border-radius: 0.625rem; padding: 0.375rem; transition: all 0.3s ease; } /* 链接样式重置 */ a.sidebar-card { color: inherit; } a.sidebar-card:hover { color: inherit; text-decoration: none; } /* 侧栏标题区引语 */ .sidebar-quote { max-height: 0; overflow: hidden; opacity: 0; margin-top: 0; transition: max-height 0.4s ease, opacity 0.3s ease, margin-top 0.3s ease; } .sidebar-header-hover:hover .sidebar-quote { max-height: 3rem; opacity: 1; margin-top: 0.375rem; } /* 可点击的二维码卡片 */ .sidebar-card-clickable { cursor: pointer; position: relative; } .sidebar-card-clickable:hover { border-color: #d1d5db; box-shadow: 0 6px 16px rgba(0, 0, 0, 0.08); } /* 点击放大提示 */ .sidebar-enlarge-hint { position: absolute; bottom: 0; left: 50%; transform: translateX(-50%) translateY(4px); background: rgba(0, 0, 0, 0.65); color: white; font-size: 0.5625rem; padding: 0.125rem 0.5rem; border-radius: 0.25rem; white-space: nowrap; opacity: 0; transition: all 0.2s ease; pointer-events: none; } .sidebar-card-clickable:hover .sidebar-enlarge-hint { opacity: 1; transform: translateX(-50%) translateY(-4px); } ================================================ FILE: docs/index.html ================================================ TrendRadar 配置文件编辑器
未保存
配置模块
支持项目

若 TrendRadar 曾为你捕捉价值，不妨为它注入动力，助其持续进化

点亮 Star
让更多人发现它
前往 GitHub
不迷路
获取更新通知
微信公众号
点击放大

扫码关注公众号

随心赞赏
1 元也是鼓励
微信支付
点击放大

微信扫码 · 丰俭由人

探索更多
另一个用心的作品
去看看

"开源不易，感谢支持"

================================================ FILE: index.html ================================================ 热点新闻分析
热点新闻分析
报告类型 当日汇总
新闻总数 387 条
热点新闻 5 条
生成时间 06-16 07:17
ai 人工智能
3 条
1/4
1
财联社热门 7-8 00:23~07:17 15次
上市首日暴涨140% 军用无人机公司登陆纽交所 AI打造产品核心竞争力
2
tieba 18-19 00:23~07:17 15次
伊朗疑用AI伪造以军F35残骸图
3
zhihu 5-13 00:23~07:17 15次
罗杰·彭罗斯说无论意识是什么，都绝对不是一种计算。意思是：任何 AI 都不可能产生意识？
DeepSeek 梁文锋
1 条
2/4
1
华尔街见闻 8-9 00:23~07:17 15次
恒生生科指数1月以来涨超60%，中国创新药的"DeepSeek时刻"超过了AI
哪吒 饺子
1 条
3/4
1
百度热搜 24-30 00:57~06:55 7次
《哪吒2》片方或分账52亿元
米哈游 原神 星穹铁道
1 条
4/4
1
zhihu 5 06:55~07:17 2次
目前原神所有自机角色谁最有可能出新形态?
由 TrendRadar 生成 · GitHub 开源项目
================================================ FILE: mcp_server/__init__.py ================================================ """ TrendRadar MCP Server 提供基于MCP协议的新闻聚合数据查询和系统管理接口。 """ __version__ = "4.0.0" ================================================ FILE: mcp_server/server.py ================================================ """ TrendRadar MCP Server - FastMCP 2.0 实现 使用 FastMCP 2.0 提供生产级 MCP 工具服务器。 支持 stdio 和 HTTP 两种传输模式。 """ import asyncio import json from typing import List, Optional, Dict, Union from fastmcp import FastMCP from .tools.data_query import DataQueryTools from .tools.analytics import AnalyticsTools from .tools.search_tools import SearchTools from .tools.config_mgmt import ConfigManagementTools from .tools.system import SystemManagementTools from .tools.storage_sync import StorageSyncTools from .tools.article_reader import ArticleReaderTools from .tools.notification import NotificationTools from .utils.date_parser import DateParser from .utils.errors import MCPError # 创建 FastMCP 2.0 应用 mcp = FastMCP('trendradar-news') # 全局工具实例（在第一次请求时初始化） _tools_instances = {} def _get_tools(project_root: Optional[str] = None): """获取或创建工具实例（单例模式）""" if not _tools_instances: _tools_instances['data'] = DataQueryTools(project_root) _tools_instances['analytics'] = AnalyticsTools(project_root) _tools_instances['search'] = SearchTools(project_root) _tools_instances['config'] = ConfigManagementTools(project_root) _tools_instances['system'] = SystemManagementTools(project_root) _tools_instances['storage'] = StorageSyncTools(project_root) _tools_instances['article'] = ArticleReaderTools(project_root) _tools_instances['notification'] = NotificationTools(project_root) return _tools_instances # ==================== MCP Resources ==================== @mcp.resource("config://platforms") async def get_platforms_resource() -> str: """ 获取支持的平台列表 返回 config.yaml 中配置的所有平台信息，包括 ID 和名称。 """ tools = _get_tools() config = await asyncio.to_thread( tools['config'].get_current_config, section="crawler" ) return json.dumps({ "platforms": config.get("platforms", []), "description": "TrendRadar 支持的热榜平台列表" }, ensure_ascii=False, indent=2) @mcp.resource("config://rss-feeds") async def get_rss_feeds_resource() -> str: """ 获取 RSS 订阅源列表 返回当前配置的所有 RSS 源信息。 """ tools = _get_tools() status = await asyncio.to_thread(tools['data'].get_rss_feeds_status) return json.dumps({ "feeds": status.get("today_feeds", {}), "description": "TrendRadar 支持的 RSS 订阅源列表" }, ensure_ascii=False, indent=2) @mcp.resource("data://available-dates") async def get_available_dates_resource() -> str: """ 获取可用的数据日期范围 返回本地存储中可查询的日期列表。 """ tools = _get_tools() result = await asyncio.to_thread( tools['storage'].list_available_dates, source="local" ) return json.dumps({ "dates": result.get("data", {}).get("local", {}).get("dates", []), "description": "本地存储中可查询的日期列表" }, ensure_ascii=False, indent=2) @mcp.resource("config://keywords") async def get_keywords_resource() -> str: """ 获取关注词配置 返回 frequency_words.txt 中配置的关注词分组。 """ tools = _get_tools() config = await asyncio.to_thread( tools['config'].get_current_config, section="keywords" ) return json.dumps({ "word_groups": config.get("word_groups", []), "total_groups": config.get("total_groups", 0), "description": "TrendRadar 关注词配置" }, ensure_ascii=False, indent=2) # ==================== 日期解析工具（优先调用）==================== @mcp.tool async def resolve_date_range( expression: str ) -> str: """ 【推荐优先调用】将自然语言日期表达式解析为标准日期范围 **为什么需要这个工具？** 用户经常使用"本周"、"最近7天"等自然语言表达日期，但 AI 模型自己计算日期 可能导致不一致的结果。此工具在服务器端使用精确的当前时间计算，确保所有 AI 模型获得一致的日期范围。 **推荐使用流程：** 1. 用户说"分析AI本周的情感倾向" 2. AI 调用 resolve_date_range("本周") → 获取精确日期范围 3. AI 调用 analyze_sentiment(topic="ai", date_range=上一步返回的date_range) Args: expression: 自然语言日期表达式，支持： - 单日: "今天", "昨天", "today", "yesterday" - 周: "本周", "上周", "this week", "last week" - 月: "本月", "上月", "this month", "last month" - 最近N天: "最近7天", "最近30天", "last 7 days", "last 30 days" - 动态: "最近5天", "last 10 days"（任意天数） Returns: JSON格式的日期范围，可直接用于其他工具的 date_range 参数： { "success": true, "expression": "本周", "date_range": { "start": "2025-11-18", "end": "2025-11-26" }, "current_date": "2025-11-26", "description": "本周（周一到周日，11-18 至 11-26）" } Examples: 用户："分析AI本周的情感倾向" AI调用步骤： 1. resolve_date_range("本周") → {"date_range": {"start": "2025-11-18", "end": "2025-11-26"}, ...} 2. analyze_sentiment(topic="ai", date_range={"start": "2025-11-18", "end": "2025-11-26"}) 用户："看看最近7天的特斯拉新闻" AI调用步骤： 1. resolve_date_range("最近7天") → {"date_range": {"start": "2025-11-20", "end": "2025-11-26"}, ...} 2. search_news(query="特斯拉", date_range={"start": "2025-11-20", "end": "2025-11-26"}) """ try: result = await asyncio.to_thread(DateParser.resolve_date_range_expression, expression) return json.dumps(result, ensure_ascii=False, indent=2) except MCPError as e: return json.dumps({ "success": False, "error": e.to_dict() }, ensure_ascii=False, indent=2) except Exception as e: return json.dumps({ "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } }, ensure_ascii=False, indent=2) # ==================== 数据查询工具 ==================== @mcp.tool async def get_latest_news( platforms: Optional[List[str]] = None, limit: int = 50, include_url: bool = False ) -> str: """ 获取最新一批爬取的新闻数据，快速了解当前热点 Args: platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 limit: 返回条数限制，默认50，最大1000 include_url: 是否包含URL链接，默认False（节省token） Returns: JSON格式的新闻列表 **数据展示建议** - 默认展示全部返回数据，除非用户明确要求总结 - 用户说"总结"或"挑重点"时才进行筛选 - 用户问"为什么只显示部分"说明需要完整数据 """ tools = _get_tools() result = await asyncio.to_thread( tools['data'].get_latest_news, platforms=platforms, limit=limit, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_trending_topics( top_n: int = 10, mode: str = 'current', extract_mode: str = 'keywords' ) -> str: """ 获取热点话题统计 Args: top_n: 返回TOP N话题，默认10 mode: 时间模式 - "daily": 当日累计数据统计 - "current": 最新一批数据统计（默认） extract_mode: 提取模式 - "keywords": 统计预设关注词（基于 config/frequency_words.txt，默认） - "auto_extract": 自动从新闻标题提取高频词（无需预设，自动发现热点） Returns: JSON格式的话题频率统计列表 Examples: - 使用预设关注词: get_trending_topics(mode="current") - 自动提取热点: get_trending_topics(extract_mode="auto_extract", top_n=20) """ tools = _get_tools() result = await asyncio.to_thread( tools['data'].get_trending_topics, top_n=top_n, mode=mode, extract_mode=extract_mode ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== RSS 数据查询工具 ==================== @mcp.tool async def get_latest_rss( feeds: Optional[List[str]] = None, days: int = 1, limit: int = 50, include_summary: bool = False ) -> str: """ 获取最新的 RSS 订阅数据（支持多日查询） RSS 数据与热榜新闻分开存储，按时间流展示，适合获取特定来源的最新内容。 Args: feeds: RSS 源 ID 列表，如 ['hacker-news', '36kr']，不指定则返回所有源 days: 获取最近 N 天的数据，默认 1（仅今天），最大 30 天 limit: 返回条数限制，默认50，最大500 include_summary: 是否包含文章摘要，默认False（节省token） Returns: JSON格式的 RSS 条目列表 Examples: - get_latest_rss() - get_latest_rss(days=7, feeds=['hacker-news']) """ tools = _get_tools() result = await asyncio.to_thread( tools['data'].get_latest_rss, feeds=feeds, days=days, limit=limit, include_summary=include_summary ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def search_rss( keyword: str, feeds: Optional[List[str]] = None, days: int = 7, limit: int = 50, include_summary: bool = False ) -> str: """ 搜索 RSS 数据 在 RSS 订阅数据中搜索包含指定关键词的文章。 Args: keyword: 搜索关键词（必需） feeds: RSS 源 ID 列表，如 ['hacker-news', '36kr'] - 不指定时：搜索所有 RSS 源 days: 搜索最近 N 天的数据，默认 7 天，最大 30 天 limit: 返回条数限制，默认50 include_summary: 是否包含文章摘要，默认False Returns: JSON格式的匹配 RSS 条目列表 Examples: - search_rss(keyword="AI") - search_rss(keyword="machine learning", feeds=['hacker-news'], days=14) """ tools = _get_tools() result = await asyncio.to_thread( tools['data'].search_rss, keyword=keyword, feeds=feeds, days=days, limit=limit, include_summary=include_summary ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_rss_feeds_status() -> str: """ 获取 RSS 源状态信息 查看当前配置的 RSS 源及其数据统计信息。 Returns: JSON格式的 RSS 源状态，包含： - available_dates: 有 RSS 数据的日期列表 - total_dates: 总日期数 - today_feeds: 今日各 RSS 源的数据统计 - {feed_id}: { name, item_count } - generated_at: 生成时间 Examples: - get_rss_feeds_status() # 查看所有 RSS 源状态 """ tools = _get_tools() result = await asyncio.to_thread(tools['data'].get_rss_feeds_status) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_news_by_date( date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, limit: int = 50, include_url: bool = False ) -> str: """ 获取指定日期的新闻数据，用于历史数据分析和对比 Args: date_range: 日期范围，支持多种格式: - 范围对象: {"start": "2025-01-01", "end": "2025-01-07"} - 自然语言: "今天", "昨天", "本周", "最近7天" - 单日字符串: "2025-01-15" - 默认值: "今天" platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 limit: 返回条数限制，默认50，最大1000 include_url: 是否包含URL链接，默认False（节省token） Returns: JSON格式的新闻列表，包含标题、平台、排名等信息 """ tools = _get_tools() result = await asyncio.to_thread( tools['data'].get_news_by_date, date_range=date_range, platforms=platforms, limit=limit, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 高级数据分析工具 ==================== @mcp.tool async def analyze_topic_trend( topic: str, analysis_type: str = "trend", date_range: Optional[Union[Dict[str, str], str]] = None, granularity: str = "day", spike_threshold: float = 3.0, time_window: int = 24, lookahead_hours: int = 6, confidence_threshold: float = 0.7 ) -> str: """ 统一话题趋势分析工具 - 整合多种趋势分析模式 建议：使用自然语言日期时，先调用 resolve_date_range 获取精确日期范围。 Args: topic: 话题关键词（必需） analysis_type: 分析类型 - "trend": 热度趋势分析（默认） - "lifecycle": 生命周期分析 - "viral": 异常热度检测 - "predict": 话题预测 date_range: 日期范围，格式 {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}，默认最近7天 granularity: 时间粒度，默认"day" spike_threshold: 热度突增倍数阈值（viral模式），默认3.0 time_window: 检测时间窗口小时数（viral模式），默认24 lookahead_hours: 预测未来小时数（predict模式），默认6 confidence_threshold: 置信度阈值（predict模式），默认0.7 Returns: JSON格式的趋势分析结果 Examples: - analyze_topic_trend(topic="AI", date_range={"start": "2025-01-01", "end": "2025-01-07"}) - analyze_topic_trend(topic="特斯拉", analysis_type="lifecycle") """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].analyze_topic_trend_unified, topic=topic, analysis_type=analysis_type, date_range=date_range, granularity=granularity, threshold=spike_threshold, time_window=time_window, lookahead_hours=lookahead_hours, confidence_threshold=confidence_threshold ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def analyze_data_insights( insight_type: str = "platform_compare", topic: Optional[str] = None, date_range: Optional[Union[Dict[str, str], str]] = None, min_frequency: int = 3, top_n: int = 20 ) -> str: """ 统一数据洞察分析工具 - 整合多种数据分析模式 Args: insight_type: 洞察类型，可选值： - "platform_compare": 平台对比分析（对比不同平台对话题的关注度） - "platform_activity": 平台活跃度统计（统计各平台发布频率和活跃时间） - "keyword_cooccur": 关键词共现分析（分析关键词同时出现的模式） topic: 话题关键词（可选，platform_compare模式适用） date_range: **【对象类型】** 日期范围（可选） - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **示例**: {"start": "2025-01-01", "end": "2025-01-07"} - **重要**: 必须是对象格式，不能传递整数 min_frequency: 最小共现频次（keyword_cooccur模式），默认3 top_n: 返回TOP N结果（keyword_cooccur模式），默认20 Returns: JSON格式的数据洞察分析结果 Examples: - analyze_data_insights(insight_type="platform_compare", topic="人工智能") - analyze_data_insights(insight_type="platform_activity", date_range={"start": "2025-01-01", "end": "2025-01-07"}) - analyze_data_insights(insight_type="keyword_cooccur", min_frequency=5, top_n=15) """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].analyze_data_insights_unified, insight_type=insight_type, topic=topic, date_range=date_range, min_frequency=min_frequency, top_n=top_n ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def analyze_sentiment( topic: Optional[str] = None, platforms: Optional[List[str]] = None, date_range: Optional[Union[Dict[str, str], str]] = None, limit: int = 50, sort_by_weight: bool = True, include_url: bool = False ) -> str: """ 分析新闻的情感倾向和热度趋势 建议：使用自然语言日期时，先调用 resolve_date_range 获取精确日期范围。 Args: topic: 话题关键词（可选） platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 date_range: 日期范围，格式 {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}，默认今天 limit: 返回新闻数量，默认50，最大100（会对标题去重） sort_by_weight: 是否按热度权重排序，默认True include_url: 是否包含URL链接，默认False（节省token） Returns: JSON格式的分析结果，包含情感分布、热度趋势和相关新闻 Examples: - analyze_sentiment(topic="AI", date_range={"start": "2025-01-01", "end": "2025-01-07"}) """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].analyze_sentiment, topic=topic, platforms=platforms, date_range=date_range, limit=limit, sort_by_weight=sort_by_weight, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def find_related_news( reference_title: str, date_range: Optional[Union[Dict[str, str], str]] = None, threshold: float = 0.5, limit: int = 50, include_url: bool = False ) -> str: """ 查找与指定新闻标题相关的其他新闻（支持当天和历史数据） Args: reference_title: 参考新闻标题（完整或部分） date_range: 日期范围（可选） - 不指定: 只查询今天的数据 - "today", "yesterday", "last_week", "last_month": 预设值 - {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}: 自定义范围 threshold: 相似度阈值，0-1之间，默认0.5（越高匹配越严格） limit: 返回条数限制，默认50 include_url: 是否包含URL链接，默认False（节省token） Returns: JSON格式的相关新闻列表，按相似度排序 Examples: - find_related_news(reference_title="特斯拉降价") - find_related_news(reference_title="AI突破", date_range="last_week") """ tools = _get_tools() result = await asyncio.to_thread( tools['search'].find_related_news_unified, reference_title=reference_title, date_range=date_range, threshold=threshold, limit=limit, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def generate_summary_report( report_type: str = "daily", date_range: Optional[Union[Dict[str, str], str]] = None ) -> str: """ 每日/每周摘要生成器 - 自动生成热点摘要报告 Args: report_type: 报告类型（daily/weekly） date_range: **【对象类型】** 自定义日期范围（可选） - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **示例**: {"start": "2025-01-01", "end": "2025-01-07"} - **重要**: 必须是对象格式，不能传递整数 Returns: JSON格式的摘要报告，包含Markdown格式内容 """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].generate_summary_report, report_type=report_type, date_range=date_range ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def aggregate_news( date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, similarity_threshold: float = 0.7, limit: int = 50, include_url: bool = False ) -> str: """ 跨平台新闻聚合 - 对相似新闻进行去重合并 将不同平台报道的同一事件合并为一条聚合新闻，显示跨平台覆盖情况和综合热度。 Args: date_range: 日期范围，不指定则查询今天 platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 similarity_threshold: 相似度阈值，0.3-1.0，默认0.7（越高越严格） limit: 返回聚合新闻数量，默认50 include_url: 是否包含URL链接，默认False Returns: JSON格式的聚合结果，包含去重统计、聚合新闻列表和平台覆盖统计 Examples: - aggregate_news() - aggregate_news(similarity_threshold=0.8) """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].aggregate_news, date_range=date_range, platforms=platforms, similarity_threshold=similarity_threshold, limit=limit, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def compare_periods( period1: Union[Dict[str, str], str], period2: Union[Dict[str, str], str], topic: Optional[str] = None, compare_type: str = "overview", platforms: Optional[List[str]] = None, top_n: int = 10 ) -> str: """ 时期对比分析 - 比较两个时间段的新闻数据 对比不同时期的热点话题、平台活跃度、新闻数量等维度。 **使用场景：** - 对比本周和上周的热点变化 - 分析某个话题在两个时期的热度差异 - 查看各平台活跃度的周期性变化 Args: period1: 第一个时间段（基准期） - {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}: 日期范围 - "today", "yesterday", "this_week", "last_week", "this_month", "last_month": 预设值 period2: 第二个时间段（对比期，格式同 period1） topic: 可选的话题关键词（聚焦特定话题的对比） compare_type: 对比类型 - "overview": 总体概览（默认）- 新闻数量、关键词变化、TOP新闻 - "topic_shift": 话题变化分析 - 上升话题、下降话题、新出现话题 - "platform_activity": 平台活跃度对比 - 各平台新闻数量变化 platforms: 平台过滤列表，如 ['zhihu', 'weibo'] top_n: 返回 TOP N 结果，默认10 Returns: JSON格式的对比分析结果，包含： - periods: 两个时期的日期范围 - compare_type: 对比类型 - overview/topic_shift/platform_comparison: 具体对比结果（根据类型） Examples: - compare_periods(period1="last_week", period2="this_week") # 周环比 - compare_periods(period1="last_month", period2="this_month", compare_type="topic_shift") - compare_periods( period1={"start": "2025-01-01", "end": "2025-01-07"}, period2={"start": "2025-01-08", "end": "2025-01-14"}, topic="人工智能" ) """ tools = _get_tools() result = await asyncio.to_thread( tools['analytics'].compare_periods, period1=period1, period2=period2, topic=topic, compare_type=compare_type, platforms=platforms, top_n=top_n ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 智能检索工具 ==================== @mcp.tool async def search_news( query: str, search_mode: str = "keyword", date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, limit: int = 50, sort_by: str = "relevance", threshold: float = 0.6, include_url: bool = False, include_rss: bool = False, rss_limit: int = 20 ) -> str: """ 统一搜索接口，支持多种搜索模式，可同时搜索热榜和RSS 建议：使用自然语言日期时，先调用 resolve_date_range 获取精确日期范围。 Args: query: 搜索关键词或内容片段 search_mode: 搜索模式 - "keyword": 精确关键词匹配（默认） - "fuzzy": 模糊内容匹配 - "entity": 实体名称搜索（人物/地点/机构） date_range: 日期范围，格式 {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}，默认今天 platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 limit: 热榜返回条数限制，默认50 sort_by: 排序方式 - "relevance"（相关度）/ "weight"（权重）/ "date"（日期） threshold: 相似度阈值（仅fuzzy模式），0-1，默认0.6 include_url: 是否包含URL链接，默认False include_rss: 是否同时搜索RSS数据，默认False rss_limit: RSS返回条数限制，默认20 Returns: JSON格式的搜索结果，包含热榜新闻列表和可选的RSS结果 Examples: - search_news(query="AI") - search_news(query="AI", include_rss=True) - search_news(query="特斯拉", date_range={"start": "2025-01-01", "end": "2025-01-07"}) """ tools = _get_tools() result = await asyncio.to_thread( tools['search'].search_news_unified, query=query, search_mode=search_mode, date_range=date_range, platforms=platforms, limit=limit, sort_by=sort_by, threshold=threshold, include_url=include_url, include_rss=include_rss, rss_limit=rss_limit ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 配置与系统管理工具 ==================== @mcp.tool async def get_current_config( section: str = "all" ) -> str: """ 获取当前系统配置 Args: section: 配置节，可选值： - "all": 所有配置（默认） - "crawler": 爬虫配置 - "push": 推送配置 - "keywords": 关键词配置 - "weights": 权重配置 Returns: JSON格式的配置信息 """ tools = _get_tools() result = await asyncio.to_thread(tools['config'].get_current_config, section=section) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_system_status() -> str: """ 获取系统运行状态和健康检查信息 返回系统版本、数据统计、缓存状态等信息 Returns: JSON格式的系统状态信息 """ tools = _get_tools() result = await asyncio.to_thread(tools['system'].get_system_status) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def check_version( proxy_url: Optional[str] = None ) -> str: """ 检查版本更新（同时检查 TrendRadar 和 MCP Server） 比较本地版本与 GitHub 远程版本，判断是否需要更新。 Args: proxy_url: 可选的代理URL，用于访问 GitHub（如 http://127.0.0.1:7890） Returns: JSON格式的版本检查结果，包含两个组件的版本对比和是否需要更新 Examples: - check_version() - check_version(proxy_url="http://127.0.0.1:7890") """ tools = _get_tools() result = await asyncio.to_thread(tools['system'].check_version, proxy_url=proxy_url) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def trigger_crawl( platforms: Optional[List[str]] = None, save_to_local: bool = False, include_url: bool = False ) -> str: """ 手动触发一次爬取任务（可选持久化） Args: platforms: 平台ID列表，如 ['zhihu', 'weibo']，不指定则使用所有平台 save_to_local: 是否保存到本地 output 目录，默认 False include_url: 是否包含URL链接，默认False（节省token） Returns: JSON格式的任务状态信息，包含成功/失败平台列表和新闻数据 Examples: - trigger_crawl(platforms=['zhihu']) - trigger_crawl(save_to_local=True) """ tools = _get_tools() result = await asyncio.to_thread( tools['system'].trigger_crawl, platforms=platforms, save_to_local=save_to_local, include_url=include_url ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 存储同步工具 ==================== @mcp.tool async def sync_from_remote( days: int = 7 ) -> str: """ 从远程存储拉取数据到本地 用于 MCP Server 等场景：爬虫存到远程云存储（如 Cloudflare R2）， MCP Server 拉取到本地进行分析查询。 Args: days: 拉取最近 N 天的数据，默认 7 天 - 0: 不拉取 - 7: 拉取最近一周的数据 - 30: 拉取最近一个月的数据 Returns: JSON格式的同步结果，包含： - success: 是否成功 - synced_files: 成功同步的文件数量 - synced_dates: 成功同步的日期列表 - skipped_dates: 跳过的日期（本地已存在） - failed_dates: 失败的日期及错误信息 - message: 操作结果描述 Examples: - sync_from_remote() # 拉取最近7天 - sync_from_remote(days=30) # 拉取最近30天 Note: 需要在 config/config.yaml 中配置远程存储（storage.remote）或设置环境变量： - S3_ENDPOINT_URL: 服务端点 - S3_BUCKET_NAME: 存储桶名称 - S3_ACCESS_KEY_ID: 访问密钥 ID - S3_SECRET_ACCESS_KEY: 访问密钥 """ tools = _get_tools() result = await asyncio.to_thread(tools['storage'].sync_from_remote, days=days) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_storage_status() -> str: """ 获取存储配置和状态 查看当前存储后端配置、本地和远程存储的状态信息。 Returns: JSON格式的存储状态信息，包含本地/远程存储状态和拉取配置 """ tools = _get_tools() result = await asyncio.to_thread(tools['storage'].get_storage_status) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def list_available_dates( source: str = "both" ) -> str: """ 列出本地/远程可用的日期范围 查看本地和远程存储中有哪些日期的数据可用。 Args: source: 数据来源 - "local": 仅本地 - "remote": 仅远程 - "both": 同时列出并对比（默认） Returns: JSON格式的日期列表，包含各来源的日期信息和对比结果 Examples: - list_available_dates() - list_available_dates(source="local") """ tools = _get_tools() result = await asyncio.to_thread(tools['storage'].list_available_dates, source=source) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 文章内容读取工具 ==================== @mcp.tool async def read_article( url: str, timeout: int = 30 ) -> str: """ 读取指定 URL 的文章内容，返回 LLM 友好的 Markdown 格式 通过 Jina AI Reader 将网页转换为干净的 Markdown，自动去除广告、导航栏等噪音内容。 适合用于：阅读新闻正文、获取文章详情、分析文章内容。 **典型使用流程：** 1. 先用 search_news(include_url=True) 搜索新闻获取链接 2. 再用 read_article(url=链接) 读取正文内容 3. AI 对 Markdown 正文进行分析、摘要、翻译等 Args: url: 文章链接（必需），以 http:// 或 https:// 开头 timeout: 请求超时时间（秒），默认 30，最大 60 Returns: JSON格式的文章内容，包含完整 Markdown 正文 Examples: - read_article(url="https://example.com/news/123") Note: - 使用 Jina AI Reader 免费服务（100 RPM 限制） - 每次请求间隔 5 秒（内置速率控制） - 部分付费墙/登录墙页面可能无法完整获取 """ tools = _get_tools() timeout = min(max(timeout, 10), 60) result = await asyncio.to_thread( tools['article'].read_article, url=url, timeout=timeout ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def read_articles_batch( urls: List[str], timeout: int = 30 ) -> str: """ 批量读取多篇文章内容（最多 5 篇，间隔 5 秒） 逐篇请求文章内容，每篇之间自动间隔 5 秒以遵守速率限制。 **典型使用流程：** 1. 先用 search_news(include_url=True) 搜索新闻获取多个链接 2. 再用 read_articles_batch(urls=[...]) 批量读取正文 3. AI 对多篇文章进行对比分析、综合报告 Args: urls: 文章链接列表（必需），最多处理 5 篇 timeout: 每篇的请求超时时间（秒），默认 30 Returns: JSON格式的批量读取结果，包含每篇的完整内容和状态 Examples: - read_articles_batch(urls=["https://a.com/1", "https://b.com/2"]) Note: - 单次最多读取 5 篇，超出部分会被跳过 - 5 篇约需 25-30 秒（每篇间隔 5 秒） - 单篇失败不影响其他篇的读取 """ tools = _get_tools() timeout = min(max(timeout, 10), 60) result = await asyncio.to_thread( tools['article'].read_articles_batch, urls=urls, timeout=timeout ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 通知推送工具 ==================== @mcp.tool async def get_channel_format_guide(channel: Optional[str] = None) -> str: """ 获取通知渠道的格式化策略指南 返回各渠道支持的 Markdown 特性、格式限制和最佳格式化提示词。 在调用 send_notification 之前使用此工具，可以了解目标渠道的格式要求， 从而生成最佳排版效果的消息内容。 各渠道格式差异概览： - 飞书：支持 **粗体**、彩色文本、[链接](url)、--- 分割线 - 钉钉：支持 ### 标题、**粗体**、> 引用、--- 分割线，不支持颜色 - 企业微信：仅支持 **粗体**、[链接](url)、> 引用，不支持标题和分割线 - Telegram：自动转为 HTML，支持粗体/斜体/删除线/代码/链接/引用块 - ntfy：支持标准 Markdown，不支持颜色 - Bark：iOS 推送，仅支持粗体和链接，内容需精简 - Slack：自动转为 mrkdwn，*粗体*、~删除线~、 - 邮件：自动转为完整 HTML 网页，支持标题/样式/分割线 - 通用 Webhook：标准 Markdown 或自定义模板 Args: channel: 指定渠道 ID（可选），不指定返回所有渠道策略 可选值: feishu, dingtalk, wework, telegram, email, ntfy, bark, slack, generic_webhook Returns: JSON格式的渠道格式化策略，包含支持特性、限制和格式化提示词 Examples: - get_channel_format_guide() # 获取所有渠道策略 - get_channel_format_guide(channel="feishu") # 获取飞书策略 - get_channel_format_guide(channel="telegram") # 获取 Telegram 策略 """ tools = _get_tools() result = await asyncio.to_thread( tools['notification'].get_channel_format_guide, channel=channel ) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def get_notification_channels() -> str: """ 获取所有已配置的通知渠道及其状态 检测 config.yaml 和 .env 环境变量中的通知渠道配置。 支持 9 个渠道：飞书、钉钉、企业微信、Telegram、邮件、ntfy、Bark、Slack、通用 Webhook。 Returns: JSON格式的渠道状态，包含每个渠道是否已配置及配置来源 Examples: - get_notification_channels() """ tools = _get_tools() result = await asyncio.to_thread(tools['notification'].get_notification_channels) return json.dumps(result, ensure_ascii=False, indent=2) @mcp.tool async def send_notification( message: str, title: str = "TrendRadar 通知", channels: Optional[List[str]] = None, ) -> str: """ 向已配置的通知渠道发送消息 接受 markdown 格式内容，内部自动适配各渠道的格式要求和限制： - 飞书：Markdown 卡片消息（支持 **粗体**、彩色文本、[链接](url)、---） - 钉钉：Markdown（自动降级标题为 ###、剥离 标签和删除线） - 企业微信：Markdown（自动剥离 # 标题、---、 标签、删除线） - Telegram：HTML（自动转换 **→、*→、~~→、>→
） - Email：HTML 邮件（完整网页样式，支持 # 标题、---、粗体斜体） - ntfy：Markdown（自动剥离 标签） - Bark：Markdown（自动简化为粗体+链接，适配 iOS 推送） - Slack：mrkdwn（自动转换 **→*、~~→~、[text](url)→） - 通用 Webhook：Markdown（支持自定义模板） 提示：发送前可调用 get_channel_format_guide 获取目标渠道的详细格式化策略， 以生成最佳排版效果的消息内容。 Args: message: markdown 格式的消息内容（必需） title: 消息标题，默认 "TrendRadar 通知" channels: 指定发送的渠道列表，不指定则发送到所有已配置渠道 可选值: feishu, dingtalk, wework, telegram, email, ntfy, bark, slack, generic_webhook Returns: JSON格式的发送结果，包含每个渠道的发送状态 Examples: - send_notification(message="**测试消息**\\n这是一条测试通知") - send_notification(message="紧急通知", title="系统告警", channels=["feishu", "dingtalk"]) """ tools = _get_tools() result = await asyncio.to_thread( tools['notification'].send_notification, message=message, title=title, channels=channels ) return json.dumps(result, ensure_ascii=False, indent=2) # ==================== 启动入口 ==================== def run_server( project_root: Optional[str] = None, transport: str = 'stdio', host: str = '0.0.0.0', port: int = 3333 ): """ 启动 MCP 服务器 Args: project_root: 项目根目录路径 transport: 传输模式，'stdio' 或 'http' host: HTTP模式的监听地址，默认 0.0.0.0 port: HTTP模式的监听端口，默认 3333 """ # 初始化工具实例 _get_tools(project_root) # 打印启动信息 print() print("=" * 60) print(" TrendRadar MCP Server - FastMCP 2.0") print("=" * 60) print(f" 传输模式: {transport.upper()}") if transport == 'stdio': print(" 协议: MCP over stdio (标准输入输出)") print(" 说明: 通过标准输入输出与 MCP 客户端通信") elif transport == 'http': print(f" 协议: MCP over HTTP (生产环境)") print(f" 服务器监听: {host}:{port}") if project_root: print(f" 项目目录: {project_root}") else: print(" 项目目录: 当前目录") print() print(" 已注册的工具:") print(" === 日期解析工具（推荐优先调用）===") print(" 0. resolve_date_range - 解析自然语言日期为标准格式") print() print(" === 基础数据查询（P0核心）===") print(" 1. get_latest_news - 获取最新新闻") print(" 2. get_news_by_date - 按日期查询新闻（支持自然语言）") print(" 3. get_trending_topics - 获取趋势话题（支持自动提取）") print() print(" === RSS 数据查询 ===") print(" 4. get_latest_rss - 获取最新 RSS 订阅数据") print(" 5. search_rss - 搜索 RSS 数据") print(" 6. get_rss_feeds_status - 获取 RSS 源状态") print() print(" === 智能检索工具 ===") print(" 7. search_news - 统一新闻搜索（关键词/模糊/实体）") print(" 8. find_related_news - 相关新闻查找（支持历史数据）") print() print(" === 高级数据分析 ===") print(" 9. analyze_topic_trend - 统一话题趋势分析（热度/生命周期/爆火/预测）") print(" 10. analyze_data_insights - 统一数据洞察分析（平台对比/活跃度/关键词共现）") print(" 11. analyze_sentiment - 情感倾向分析") print(" 12. aggregate_news - 跨平台新闻聚合去重") print(" 13. compare_periods - 时期对比分析（周环比/月环比）") print(" 14. generate_summary_report - 每日/每周摘要生成") print() print(" === 配置与系统管理 ===") print(" 15. get_current_config - 获取当前系统配置") print(" 16. get_system_status - 获取系统运行状态") print(" 17. check_version - 检查版本更新（对比本地与远程版本）") print(" 18. trigger_crawl - 手动触发爬取任务") print() print(" === 存储同步工具 ===") print(" 19. sync_from_remote - 从远程存储拉取数据到本地") print(" 20. get_storage_status - 获取存储配置和状态") print(" 21. list_available_dates - 列出本地/远程可用日期") print() print(" === 文章内容读取 ===") print(" 22. read_article - 读取单篇文章内容（Markdown格式）") print(" 23. read_articles_batch - 批量读取多篇文章（自动限速）") print() print(" === 通知推送工具 ===") print(" 24. get_channel_format_guide - 获取渠道格式化策略指南（提示词）") print(" 25. get_notification_channels - 获取已配置的通知渠道状态") print(" 26. send_notification - 向通知渠道发送消息（自动适配格式）") print("=" * 60) print() # 根据传输模式运行服务器 if transport == 'stdio': mcp.run(transport='stdio') elif transport == 'http': # HTTP 模式（生产推荐） mcp.run( transport='http', host=host, port=port, path='/mcp' # HTTP 端点路径 ) else: raise ValueError(f"不支持的传输模式: {transport}") if __name__ == '__main__': import argparse parser = argparse.ArgumentParser( description='TrendRadar MCP Server - 新闻热点聚合 MCP 工具服务器', formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" 详细配置教程请查看: README-Cherry-Studio.md """ ) parser.add_argument( '--transport', choices=['stdio', 'http'], default='stdio', help='传输模式：stdio (默认) 或 http (生产环境)' ) parser.add_argument( '--host', default='0.0.0.0', help='HTTP模式的监听地址，默认 0.0.0.0' ) parser.add_argument( '--port', type=int, default=3333, help='HTTP模式的监听端口，默认 3333' ) parser.add_argument( '--project-root', help='项目根目录路径' ) args = parser.parse_args() run_server( project_root=args.project_root, transport=args.transport, host=args.host, port=args.port ) ================================================ FILE: mcp_server/services/__init__.py ================================================ """ 服务层模块 提供数据访问、缓存、解析等核心服务。 """ ================================================ FILE: mcp_server/services/cache_service.py ================================================ """ 缓存服务 实现TTL缓存机制，提升数据访问性能。 """ import hashlib import json import time from typing import Any, Optional from threading import Lock def make_cache_key(namespace: str, **params) -> str: """ 生成结构化缓存 key 通过对参数排序和哈希，确保相同参数组合总是生成相同的 key。 Args: namespace: 缓存命名空间，如 "latest_news", "trending_topics" **params: 缓存参数 Returns: 格式化的缓存 key，如 "latest_news:a1b2c3d4" Examples: >>> make_cache_key("latest_news", platforms=["zhihu"], limit=50) 'latest_news:8f14e45f' >>> make_cache_key("search", query="AI", mode="keyword") 'search:3c6e0b8a' """ if not params: return namespace # 对参数进行规范化处理 normalized_params = {} for k, v in params.items(): if v is None: continue # 跳过 None 值 elif isinstance(v, (list, tuple)): # 列表排序后转为字符串 normalized_params[k] = json.dumps(sorted(v) if all(isinstance(i, str) for i in v) else list(v), ensure_ascii=False) elif isinstance(v, dict): # 字典按键排序后转为字符串 normalized_params[k] = json.dumps(v, sort_keys=True, ensure_ascii=False) else: normalized_params[k] = str(v) # 排序参数并生成哈希 sorted_params = sorted(normalized_params.items()) param_str = "&".join(f"{k}={v}" for k, v in sorted_params) # 使用 MD5 生成短哈希（取前8位） hash_value = hashlib.md5(param_str.encode('utf-8')).hexdigest()[:8] return f"{namespace}:{hash_value}" class CacheService: """缓存服务类""" def __init__(self): """初始化缓存服务""" self._cache = {} self._timestamps = {} self._lock = Lock() def get(self, key: str, ttl: int = 900) -> Optional[Any]: """ 获取缓存数据 Args: key: 缓存键 ttl: 存活时间（秒），默认15分钟 Returns: 缓存的值，如果不存在或已过期则返回None """ with self._lock: if key in self._cache: # 检查是否过期 if time.time() - self._timestamps[key] < ttl: return self._cache[key] else: # 已过期，删除缓存 del self._cache[key] del self._timestamps[key] return None def set(self, key: str, value: Any) -> None: """ 设置缓存数据 Args: key: 缓存键 value: 缓存值 """ with self._lock: self._cache[key] = value self._timestamps[key] = time.time() def delete(self, key: str) -> bool: """ 删除缓存 Args: key: 缓存键 Returns: 是否成功删除 """ with self._lock: if key in self._cache: del self._cache[key] del self._timestamps[key] return True return False def clear(self) -> None: """清空所有缓存""" with self._lock: self._cache.clear() self._timestamps.clear() def cleanup_expired(self, ttl: int = 900) -> int: """ 清理过期缓存 Args: ttl: 存活时间（秒） Returns: 清理的条目数量 """ with self._lock: current_time = time.time() expired_keys = [ key for key, timestamp in self._timestamps.items() if current_time - timestamp >= ttl ] for key in expired_keys: del self._cache[key] del self._timestamps[key] return len(expired_keys) def get_stats(self) -> dict: """ 获取缓存统计信息 Returns: 统计信息字典 """ with self._lock: return { "total_entries": len(self._cache), "oldest_entry_age": ( time.time() - min(self._timestamps.values()) if self._timestamps else 0 ), "newest_entry_age": ( time.time() - max(self._timestamps.values()) if self._timestamps else 0 ) } # 全局缓存实例 _global_cache = None def get_cache() -> CacheService: """ 获取全局缓存实例 Returns: 全局缓存服务实例 """ global _global_cache if _global_cache is None: _global_cache = CacheService() return _global_cache ================================================ FILE: mcp_server/services/data_service.py ================================================ """ 数据访问服务 提供统一的数据查询接口,封装数据访问逻辑。 """ import re from collections import Counter from datetime import datetime, timedelta from typing import Dict, List, Optional, Tuple from .cache_service import get_cache from .parser_service import ParserService from ..utils.errors import DataNotFoundError class DataService: """数据访问服务类""" # 中文停用词列表（用于 auto_extract 模式） STOPWORDS = { '的', '了', '在', '是', '我', '有', '和', '就', '不', '人', '都', '一', '一个', '上', '也', '很', '到', '说', '要', '去', '你', '会', '着', '没有', '看', '好', '自己', '这', '那', '来', '被', '与', '为', '对', '将', '从', '以', '及', '等', '但', '或', '而', '于', '中', '由', '可', '可以', '已', '已经', '还', '更', '最', '再', '因为', '所以', '如果', '虽然', '然而', '什么', '怎么', '如何', '哪', '哪些', '多少', '几', '这个', '那个', '他', '她', '它', '他们', '她们', '我们', '你们', '大家', '自己', '这样', '那样', '怎样', '这么', '那么', '多么', '非常', '特别', '应该', '可能', '能够', '需要', '必须', '一定', '肯定', '确实', '正在', '已经', '曾经', '将要', '即将', '刚刚', '马上', '立刻', '回应', '发布', '表示', '称', '曝', '官方', '最新', '重磅', '突发', '热搜', '刷屏', '引发', '关注', '网友', '评论', '转发', '点赞' } def __init__(self, project_root: str = None): """ 初始化数据服务 Args: project_root: 项目根目录 """ self.parser = ParserService(project_root) self.cache = get_cache() def get_latest_news( self, platforms: Optional[List[str]] = None, limit: int = 50, include_url: bool = False ) -> List[Dict]: """ 获取最新一批爬取的新闻数据 Args: platforms: 平台ID列表,None表示所有平台 limit: 返回条数限制 include_url: 是否包含URL链接,默认False(节省token) Returns: 新闻列表 Raises: DataNotFoundError: 数据不存在 """ # 尝试从缓存获取 cache_key = f"latest_news:{','.join(platforms or [])}:{limit}:{include_url}" cached = self.cache.get(cache_key, ttl=900) # 15分钟缓存 if cached: return cached # 读取今天的数据 all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date( date=None, platform_ids=platforms ) # 获取最新的文件时间 if timestamps: latest_timestamp = max(timestamps.values()) fetch_time = datetime.fromtimestamp(latest_timestamp) else: fetch_time = datetime.now() # 转换为新闻列表 news_list = [] for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 取第一个排名 rank = info["ranks"][0] if info["ranks"] else 0 news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "rank": rank, "timestamp": fetch_time.strftime("%Y-%m-%d %H:%M:%S") } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") news_list.append(news_item) # 按排名排序 news_list.sort(key=lambda x: x["rank"]) # 限制返回数量 result = news_list[:limit] # 缓存结果 self.cache.set(cache_key, result) return result def get_news_by_date( self, target_date: datetime, platforms: Optional[List[str]] = None, limit: int = 50, include_url: bool = False ) -> List[Dict]: """ 按指定日期获取新闻 Args: target_date: 目标日期 platforms: 平台ID列表,None表示所有平台 limit: 返回条数限制 include_url: 是否包含URL链接,默认False(节省token) Returns: 新闻列表 Raises: DataNotFoundError: 数据不存在 Examples: >>> service = DataService() >>> news = service.get_news_by_date( ... target_date=datetime(2025, 10, 10), ... platforms=['zhihu'], ... limit=20 ... ) """ # 尝试从缓存获取 date_str = target_date.strftime("%Y-%m-%d") cache_key = f"news_by_date:{date_str}:{','.join(platforms or [])}:{limit}:{include_url}" cached = self.cache.get(cache_key, ttl=900) # 15分钟缓存 if cached: return cached # 读取指定日期的数据 all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date( date=target_date, platform_ids=platforms ) # 转换为新闻列表 news_list = [] for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 计算平均排名 avg_rank = sum(info["ranks"]) / len(info["ranks"]) if info["ranks"] else 0 news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "rank": info["ranks"][0] if info["ranks"] else 0, "avg_rank": round(avg_rank, 2), "count": len(info["ranks"]), "date": date_str } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") news_list.append(news_item) # 按排名排序 news_list.sort(key=lambda x: x["rank"]) # 限制返回数量 result = news_list[:limit] # 缓存结果(历史数据缓存更久) self.cache.set(cache_key, result) return result def search_news_by_keyword( self, keyword: str, date_range: Optional[Tuple[datetime, datetime]] = None, platforms: Optional[List[str]] = None, limit: Optional[int] = None ) -> Dict: """ 按关键词搜索新闻 Args: keyword: 搜索关键词 date_range: 日期范围 (start_date, end_date) platforms: 平台过滤列表 limit: 返回条数限制(可选) Returns: 搜索结果字典 Raises: DataNotFoundError: 数据不存在 """ # 确定搜索日期范围 if date_range: start_date, end_date = date_range else: # 默认搜索今天 start_date = end_date = datetime.now() # 收集所有匹配的新闻 results = [] platform_distribution = Counter() # 遍历日期范围 current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.parser.read_all_titles_for_date( date=current_date, platform_ids=platforms ) # 搜索包含关键词的标题 for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): if keyword.lower() in title.lower(): # 计算平均排名 avg_rank = sum(info["ranks"]) / len(info["ranks"]) if info["ranks"] else 0 results.append({ "title": title, "platform": platform_id, "platform_name": platform_name, "ranks": info["ranks"], "count": len(info["ranks"]), "avg_rank": round(avg_rank, 2), "url": info.get("url", ""), "mobileUrl": info.get("mobileUrl", ""), "date": current_date.strftime("%Y-%m-%d") }) platform_distribution[platform_id] += 1 except DataNotFoundError: # 该日期没有数据,继续下一天 pass # 下一天 current_date += timedelta(days=1) if not results: raise DataNotFoundError( f"未找到包含关键词 '{keyword}' 的新闻", suggestion="请尝试其他关键词或扩大日期范围" ) # 计算统计信息 total_ranks = [] for item in results: total_ranks.extend(item["ranks"]) avg_rank = sum(total_ranks) / len(total_ranks) if total_ranks else 0 # 限制返回数量(如果指定) total_found = len(results) if limit is not None and limit > 0: results = results[:limit] return { "results": results, "total": len(results), "total_found": total_found, "statistics": { "platform_distribution": dict(platform_distribution), "avg_rank": round(avg_rank, 2), "keyword": keyword } } def _extract_words_from_title(self, title: str, min_length: int = 2) -> List[str]: """ 从标题中提取有意义的词语（用于 auto_extract 模式） Args: title: 新闻标题 min_length: 最小词长 Returns: 关键词列表 """ # 移除URL和特殊字符 title = re.sub(r'http[s]?://\S+', '', title) title = re.sub(r'\[.*?\]', '', title) # 移除方括号内容 title = re.sub(r'[【】《》「」『』""''・·•]', '', title) # 移除中文标点 # 使用正则表达式分词（中文和英文） # 匹配连续的中文字符或英文单词 words = re.findall(r'[\u4e00-\u9fff]{2,}|[a-zA-Z]{2,}[a-zA-Z0-9]*', title) # 过滤停用词和短词 keywords = [ word for word in words if word and len(word) >= min_length and word.lower() not in self.STOPWORDS and word not in self.STOPWORDS ] return keywords def get_trending_topics( self, top_n: int = 10, mode: str = "current", extract_mode: str = "keywords" ) -> Dict: """ 获取热点话题统计 Args: top_n: 返回TOP N话题 mode: 时间模式 - "daily": 当日累计数据统计 - "current": 最新一批数据统计（默认） extract_mode: 提取模式 - "keywords": 统计预设关注词（基于 config/frequency_words.txt） - "auto_extract": 自动从新闻标题提取高频词 Returns: 话题频率统计字典 Raises: DataNotFoundError: 数据不存在 """ # 尝试从缓存获取 cache_key = f"trending_topics:{top_n}:{mode}:{extract_mode}" cached = self.cache.get(cache_key, ttl=900) # 15分钟缓存 if cached: return cached # 读取今天的数据 all_titles, id_to_name, timestamps = self.parser.read_all_titles_for_date() if not all_titles: raise DataNotFoundError( "未找到今天的新闻数据", suggestion="请确保爬虫已经运行并生成了数据" ) # 根据 mode 选择要处理的标题数据 if mode == "daily": titles_to_process = all_titles elif mode == "current": titles_to_process = all_titles # 简化实现 else: raise ValueError(f"不支持的模式: {mode}。支持的模式: daily, current") # 统计词频 word_frequency = Counter() keyword_to_news = {} # 预加载关键词数据（避免在循环内重复调用） if extract_mode == "keywords": from trendradar.core.frequency import _word_matches word_groups = self.parser.parse_frequency_words() # 遍历要处理的标题 for platform_id, titles in titles_to_process.items(): for title in titles.keys(): if extract_mode == "keywords": # 基于预设关键词统计（支持正则匹配） title_lower = title.lower() for group in word_groups: all_words = group.get("required", []) + group.get("normal", []) # 检查是否匹配词组中的任意一个词 matched = any(_word_matches(word_config, title_lower) for word_config in all_words) if matched: # 使用组的 display_name（组别名或行别名拼接） display_key = group.get("display_name") or group.get("group_key", "") word_frequency[display_key] += 1 if display_key not in keyword_to_news: keyword_to_news[display_key] = [] keyword_to_news[display_key].append(title) break # 每个标题只计入第一个匹配的词组 elif extract_mode == "auto_extract": # 自动提取关键词 extracted_words = self._extract_words_from_title(title) for word in extracted_words: word_frequency[word] += 1 if word not in keyword_to_news: keyword_to_news[word] = [] keyword_to_news[word].append(title) # 获取TOP N关键词 top_keywords = word_frequency.most_common(top_n) # 构建话题列表 topics = [] for keyword, frequency in top_keywords: matched_news = keyword_to_news.get(keyword, []) topics.append({ "keyword": keyword, "frequency": frequency, "matched_news": len(set(matched_news)), # 去重后的新闻数量 "trend": "stable", "weight_score": 0.0 }) # 构建结果 result = { "topics": topics, "generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"), "mode": mode, "extract_mode": extract_mode, "total_keywords": len(word_frequency), "description": self._get_mode_description(mode, extract_mode) } # 缓存结果 self.cache.set(cache_key, result) return result def _get_mode_description(self, mode: str, extract_mode: str = "keywords") -> str: """获取模式描述""" mode_desc = { "daily": "当日累计统计", "current": "最新一批统计" }.get(mode, "未知时间模式") extract_desc = { "keywords": "基于预设关注词", "auto_extract": "自动提取高频词" }.get(extract_mode, "未知提取模式") return f"{mode_desc} - {extract_desc}" def get_current_config(self, section: str = "all") -> Dict: """ 获取当前系统配置 Args: section: 配置节 - all/crawler/push/keywords/weights Returns: 配置字典 Raises: FileParseError: 配置文件解析错误 """ # 解析配置文件 config_data = self.parser.parse_yaml_config() word_groups = self.parser.parse_frequency_words() # 根据section返回对应配置 advanced = config_data.get("advanced", {}) advanced_crawler = advanced.get("crawler", {}) platforms_config = config_data.get("platforms", {}) if section == "all" or section == "crawler": crawler_config = { "enable_crawler": platforms_config.get("enabled", True), "use_proxy": advanced_crawler.get("use_proxy", False), "request_interval": advanced_crawler.get("request_interval", 1), "retry_times": 3, "platforms": [p["id"] for p in platforms_config.get("sources", [])] } if section == "all" or section == "push": notification = config_data.get("notification", {}) batch_size = advanced.get("batch_size", {}) push_config = { "enable_notification": notification.get("enabled", True), "enabled_channels": [], "message_batch_size": batch_size.get("default", 4000), "push_window": {} # 已迁移至调度系统（schedule + timeline.yaml） } # 检测已配置的通知渠道（合并 config.yaml + .env） from trendradar.core.loader import _load_webhook_config webhook_config = _load_webhook_config(config_data) channel_checks = { "feishu": [webhook_config.get("FEISHU_WEBHOOK_URL")], "dingtalk": [webhook_config.get("DINGTALK_WEBHOOK_URL")], "wework": [webhook_config.get("WEWORK_WEBHOOK_URL")], "telegram": [webhook_config.get("TELEGRAM_BOT_TOKEN"), webhook_config.get("TELEGRAM_CHAT_ID")], "email": [webhook_config.get("EMAIL_FROM"), webhook_config.get("EMAIL_PASSWORD"), webhook_config.get("EMAIL_TO")], "ntfy": [webhook_config.get("NTFY_SERVER_URL"), webhook_config.get("NTFY_TOPIC")], "bark": [webhook_config.get("BARK_URL")], "slack": [webhook_config.get("SLACK_WEBHOOK_URL")], "generic_webhook": [webhook_config.get("GENERIC_WEBHOOK_URL")], } for ch_id, required_values in channel_checks.items(): if all(required_values): push_config["enabled_channels"].append(ch_id) if section == "all" or section == "keywords": keywords_config = { "word_groups": word_groups, "total_groups": len(word_groups) } if section == "all" or section == "weights": weight = advanced.get("weight", {}) weights_config = { "rank_weight": weight.get("rank", 0.6), "frequency_weight": weight.get("frequency", 0.3), "hotness_weight": weight.get("hotness", 0.1) } # 组装结果 if section == "all": result = { "crawler": crawler_config, "push": push_config, "keywords": keywords_config, "weights": weights_config } elif section == "crawler": result = crawler_config elif section == "push": result = push_config elif section == "keywords": result = keywords_config elif section == "weights": result = weights_config else: result = {} return result def get_available_date_range(self, db_type: str = "news") -> Tuple[Optional[datetime], Optional[datetime]]: """ 扫描 output 目录，返回实际可用的日期范围 Args: db_type: 数据库类型 ("news" 或 "rss") Returns: (最早日期, 最新日期) 元组，如果没有数据则返回 (None, None) Examples: >>> service = DataService() >>> earliest, latest = service.get_available_date_range() >>> print(f"可用日期范围：{earliest} 至 {latest}") """ return self.parser.get_available_date_range(db_type) def get_system_status(self) -> Dict: """ 获取系统运行状态 Returns: 系统状态字典 """ # 获取数据统计 output_dir = self.parser.project_root / "output" total_storage = 0 # 使用 parser 的方法获取日期范围 oldest_record, latest_record = self.get_available_date_range(db_type="news") # 计算 output 目录总存储大小 if output_dir.exists(): for item in output_dir.rglob("*"): if item.is_file(): total_storage += item.stat().st_size # 读取版本信息 version_file = self.parser.project_root / "version" version = "unknown" if version_file.exists(): try: with open(version_file, "r") as f: version = f.read().strip() except: pass return { "system": { "version": version, "project_root": str(self.parser.project_root) }, "data": { "total_storage": f"{total_storage / 1024 / 1024:.2f} MB", "oldest_record": oldest_record.strftime("%Y-%m-%d") if oldest_record else None, "latest_record": latest_record.strftime("%Y-%m-%d") if latest_record else None, }, "cache": self.cache.get_stats(), "health": "healthy" } # ======================================== # RSS 数据查询方法 # ======================================== def get_latest_rss( self, feeds: Optional[List[str]] = None, days: int = 1, limit: int = 50, include_summary: bool = False ) -> List[Dict]: """ 获取最新的 RSS 数据（支持多日查询） Args: feeds: RSS 源 ID 列表，None 表示所有源 days: 获取最近 N 天的数据，默认 1（仅今天），最大 30 天 limit: 返回条数限制 include_summary: 是否包含摘要，默认 False（节省 token） Returns: RSS 条目列表（按 URL 去重） Raises: DataNotFoundError: 数据不存在 """ days = min(max(days, 1), 30) # 限制 1-30 天 cache_key = f"latest_rss:{','.join(feeds or [])}:{days}:{limit}:{include_summary}" cached = self.cache.get(cache_key, ttl=900) if cached: return cached rss_list = [] seen_urls = set() # 跨日期 URL 去重 today = datetime.now() for i in range(days): target_date = today - timedelta(days=i) try: all_items, id_to_name, timestamps = self.parser.read_all_titles_for_date( date=target_date, platform_ids=feeds, db_type="rss" ) # 获取抓取时间 if timestamps: latest_timestamp = max(timestamps.values()) fetch_time = datetime.fromtimestamp(latest_timestamp) else: fetch_time = target_date # 转换为列表 for feed_id, items in all_items.items(): feed_name = id_to_name.get(feed_id, feed_id) for title, info in items.items(): # 跨日期 URL 去重 url = info.get("url", "") if url and url in seen_urls: continue if url: seen_urls.add(url) rss_item = { "title": title, "feed_id": feed_id, "feed_name": feed_name, "url": url, "published_at": info.get("published_at", ""), "author": info.get("author", ""), "date": target_date.strftime("%Y-%m-%d"), "fetch_time": fetch_time.strftime("%Y-%m-%d %H:%M:%S") if isinstance(fetch_time, datetime) else target_date.strftime("%Y-%m-%d") } if include_summary: rss_item["summary"] = info.get("summary", "") rss_list.append(rss_item) except DataNotFoundError: continue # 按发布时间排序（最新的在前） rss_list.sort(key=lambda x: x.get("published_at", ""), reverse=True) # 限制返回数量 result = rss_list[:limit] # 缓存结果 self.cache.set(cache_key, result) return result def search_rss( self, keyword: str, feeds: Optional[List[str]] = None, days: int = 7, limit: int = 50, include_summary: bool = False ) -> List[Dict]: """ 搜索 RSS 数据（跨日期自动去重） Args: keyword: 搜索关键词 feeds: RSS 源 ID 列表，None 表示所有源 days: 搜索最近 N 天的数据 limit: 返回条数限制 include_summary: 是否包含摘要 Returns: 匹配的 RSS 条目列表（按 URL 去重） """ cache_key = f"search_rss:{keyword}:{','.join(feeds or [])}:{days}:{limit}:{include_summary}" cached = self.cache.get(cache_key, ttl=900) if cached: return cached results = [] seen_urls = set() # 用于 URL 去重 today = datetime.now() for i in range(days): target_date = today - timedelta(days=i) try: all_items, id_to_name, _ = self.parser.read_all_titles_for_date( date=target_date, platform_ids=feeds, db_type="rss" ) for feed_id, items in all_items.items(): feed_name = id_to_name.get(feed_id, feed_id) for title, info in items.items(): # 跨日期去重：如果 URL 已出现过则跳过 url = info.get("url", "") if url and url in seen_urls: continue if url: seen_urls.add(url) # 关键词匹配（标题或摘要） summary = info.get("summary", "") if keyword.lower() in title.lower() or keyword.lower() in summary.lower(): rss_item = { "title": title, "feed_id": feed_id, "feed_name": feed_name, "url": url, "published_at": info.get("published_at", ""), "author": info.get("author", ""), "date": target_date.strftime("%Y-%m-%d") } if include_summary: rss_item["summary"] = summary results.append(rss_item) except DataNotFoundError: continue # 按发布时间排序 results.sort(key=lambda x: x.get("published_at", ""), reverse=True) # 限制返回数量 result = results[:limit] # 缓存结果 self.cache.set(cache_key, result) return result def get_rss_feeds_status(self) -> Dict: """ 获取 RSS 源状态 Returns: RSS 源状态信息 """ cache_key = "rss_feeds_status" cached = self.cache.get(cache_key, ttl=900) if cached: return cached # 获取可用的 RSS 日期 available_dates = self.parser.get_available_dates(db_type="rss") # 获取今天的 RSS 数据统计 today_stats = {} try: all_items, id_to_name, _ = self.parser.read_all_titles_for_date( date=None, platform_ids=None, db_type="rss" ) for feed_id, items in all_items.items(): today_stats[feed_id] = { "name": id_to_name.get(feed_id, feed_id), "item_count": len(items) } except DataNotFoundError: pass result = { "available_dates": available_dates[:10], # 最近 10 天 "total_dates": len(available_dates), "today_feeds": today_stats, "generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S") } self.cache.set(cache_key, result) return result ================================================ FILE: mcp_server/services/parser_service.py ================================================ """ 数据解析服务 v2.0.0: 仅支持 SQLite 数据库，移除 TXT 文件支持 新存储结构：output/{type}/{date}.db """ import re import sqlite3 from pathlib import Path from typing import Dict, List, Tuple, Optional from datetime import datetime import yaml from ..utils.errors import FileParseError, DataNotFoundError from .cache_service import get_cache class ParserService: """数据解析服务类""" def __init__(self, project_root: str = None): """ 初始化解析服务 Args: project_root: 项目根目录，默认为当前目录的父目录 """ if project_root is None: current_file = Path(__file__) self.project_root = current_file.parent.parent.parent else: self.project_root = Path(project_root) self.cache = get_cache() # frequency_words.txt mtime 缓存 self._freq_words_cache: Optional[List[Dict]] = None self._freq_words_mtime: float = 0.0 @staticmethod def clean_title(title: str) -> str: """清理标题文本""" title = re.sub(r'\s+', ' ', title) title = title.strip() return title def get_date_folder_name(self, date: datetime = None) -> str: """ 获取日期字符串（ISO 格式） Args: date: 日期对象，默认为今天 Returns: 日期字符串（YYYY-MM-DD） """ if date is None: date = datetime.now() return date.strftime("%Y-%m-%d") def _get_db_path(self, date: datetime = None, db_type: str = "news") -> Optional[Path]: """ 获取数据库文件路径 新结构：output/{type}/{date}.db Args: date: 日期对象，默认为今天 db_type: 数据库类型 ("news" 或 "rss") Returns: 数据库文件路径，如果不存在则返回 None """ date_str = self.get_date_folder_name(date) db_path = self.project_root / "output" / db_type / f"{date_str}.db" if db_path.exists(): return db_path return None def _read_from_sqlite( self, date: datetime = None, platform_ids: Optional[List[str]] = None, db_type: str = "news" ) -> Optional[Tuple[Dict, Dict, Dict]]: """ 从 SQLite 数据库读取数据 Args: date: 日期对象，默认为今天 platform_ids: 平台ID列表，None表示所有平台 db_type: 数据库类型 ("news" 或 "rss") Returns: (all_titles, id_to_name, all_timestamps) 元组，如果数据库不存在返回 None """ db_path = self._get_db_path(date, db_type) if db_path is None: return None all_titles = {} id_to_name = {} all_timestamps = {} try: conn = sqlite3.connect(str(db_path)) conn.row_factory = sqlite3.Row cursor = conn.cursor() if db_type == "news": return self._read_news_from_sqlite(cursor, platform_ids, all_titles, id_to_name, all_timestamps) elif db_type == "rss": return self._read_rss_from_sqlite(cursor, platform_ids, all_titles, id_to_name, all_timestamps) except Exception as e: print(f"Warning: 从 SQLite 读取数据失败: {e}") return None finally: if 'conn' in locals(): conn.close() def _read_news_from_sqlite( self, cursor, platform_ids: Optional[List[str]], all_titles: Dict, id_to_name: Dict, all_timestamps: Dict ) -> Optional[Tuple[Dict, Dict, Dict]]: """从热榜数据库读取数据""" # 检查表是否存在 cursor.execute(""" SELECT name FROM sqlite_master WHERE type='table' AND name='news_items' """) if not cursor.fetchone(): return None # 构建查询 if platform_ids: placeholders = ','.join(['?' for _ in platform_ids]) query = f""" SELECT n.id, n.platform_id, p.name as platform_name, n.title, n.rank, n.url, n.mobile_url, n.first_crawl_time, n.last_crawl_time, n.crawl_count FROM news_items n LEFT JOIN platforms p ON n.platform_id = p.id WHERE n.platform_id IN ({placeholders}) """ cursor.execute(query, platform_ids) else: cursor.execute(""" SELECT n.id, n.platform_id, p.name as platform_name, n.title, n.rank, n.url, n.mobile_url, n.first_crawl_time, n.last_crawl_time, n.crawl_count FROM news_items n LEFT JOIN platforms p ON n.platform_id = p.id """) rows = cursor.fetchall() # 收集所有 news_item_id 用于查询历史排名 news_ids = [row['id'] for row in rows] rank_history_map = {} if news_ids: placeholders = ",".join("?" * len(news_ids)) cursor.execute(f""" SELECT news_item_id, rank FROM rank_history WHERE news_item_id IN ({placeholders}) ORDER BY news_item_id, crawl_time """, news_ids) for rh_row in cursor.fetchall(): news_id = rh_row['news_item_id'] rank = rh_row['rank'] if news_id not in rank_history_map: rank_history_map[news_id] = [] rank_history_map[news_id].append(rank) for row in rows: news_id = row['id'] platform_id = row['platform_id'] platform_name = row['platform_name'] or platform_id title = row['title'] if platform_id not in id_to_name: id_to_name[platform_id] = platform_name if platform_id not in all_titles: all_titles[platform_id] = {} ranks = rank_history_map.get(news_id, [row['rank']]) all_titles[platform_id][title] = { "ranks": ranks, "url": row['url'] or "", "mobileUrl": row['mobile_url'] or "", "first_time": row['first_crawl_time'] or "", "last_time": row['last_crawl_time'] or "", "count": row['crawl_count'] or 1, } # 获取抓取时间作为 timestamps cursor.execute(""" SELECT crawl_time, created_at FROM crawl_records ORDER BY crawl_time """) for row in cursor.fetchall(): crawl_time = row['crawl_time'] created_at = row['created_at'] try: ts = datetime.strptime(created_at, "%Y-%m-%d %H:%M:%S").timestamp() except (ValueError, TypeError): ts = datetime.now().timestamp() all_timestamps[f"{crawl_time}.db"] = ts if not all_titles: return None return (all_titles, id_to_name, all_timestamps) def _read_rss_from_sqlite( self, cursor, feed_ids: Optional[List[str]], all_items: Dict, id_to_name: Dict, all_timestamps: Dict ) -> Optional[Tuple[Dict, Dict, Dict]]: """从 RSS 数据库读取数据""" # 检查表是否存在 cursor.execute(""" SELECT name FROM sqlite_master WHERE type='table' AND name='rss_items' """) if not cursor.fetchone(): return None # 构建查询 if feed_ids: placeholders = ','.join(['?' for _ in feed_ids]) query = f""" SELECT i.id, i.feed_id, f.name as feed_name, i.title, i.url, i.published_at, i.summary, i.author, i.first_crawl_time, i.last_crawl_time, i.crawl_count FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id WHERE i.feed_id IN ({placeholders}) ORDER BY i.published_at DESC """ cursor.execute(query, feed_ids) else: cursor.execute(""" SELECT i.id, i.feed_id, f.name as feed_name, i.title, i.url, i.published_at, i.summary, i.author, i.first_crawl_time, i.last_crawl_time, i.crawl_count FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id ORDER BY i.published_at DESC """) rows = cursor.fetchall() for row in rows: feed_id = row['feed_id'] feed_name = row['feed_name'] or feed_id title = row['title'] if feed_id not in id_to_name: id_to_name[feed_id] = feed_name if feed_id not in all_items: all_items[feed_id] = {} all_items[feed_id][title] = { "url": row['url'] or "", "published_at": row['published_at'] or "", "summary": row['summary'] or "", "author": row['author'] or "", "first_time": row['first_crawl_time'] or "", "last_time": row['last_crawl_time'] or "", "count": row['crawl_count'] or 1, } # 获取抓取时间 cursor.execute(""" SELECT crawl_time, created_at FROM rss_crawl_records ORDER BY crawl_time """) for row in cursor.fetchall(): crawl_time = row['crawl_time'] created_at = row['created_at'] try: ts = datetime.strptime(created_at, "%Y-%m-%d %H:%M:%S").timestamp() except (ValueError, TypeError): ts = datetime.now().timestamp() all_timestamps[f"{crawl_time}.db"] = ts if not all_items: return None return (all_items, id_to_name, all_timestamps) def read_all_titles_for_date( self, date: datetime = None, platform_ids: Optional[List[str]] = None, db_type: str = "news" ) -> Tuple[Dict, Dict, Dict]: """ 读取指定日期的所有数据（带缓存） Args: date: 日期对象，默认为今天 platform_ids: 平台/Feed ID列表，None表示所有 db_type: 数据库类型 ("news" 或 "rss") Returns: (all_titles, id_to_name, all_timestamps) 元组 Raises: DataNotFoundError: 数据不存在 """ date_str = self.get_date_folder_name(date) platform_key = ','.join(sorted(platform_ids)) if platform_ids else 'all' cache_key = f"read_all:{db_type}:{date_str}:{platform_key}" is_today = (date is None) or (date.date() == datetime.now().date()) ttl = 900 if is_today else 900 cached = self.cache.get(cache_key, ttl=ttl) if cached: return cached result = self._read_from_sqlite(date, platform_ids, db_type) if result: self.cache.set(cache_key, result) return result raise DataNotFoundError( f"未找到 {date_str} 的 {db_type} 数据", suggestion="请先运行爬虫或检查日期是否正确" ) def parse_yaml_config(self, config_path: str = None) -> dict: """ 解析YAML配置文件 Args: config_path: 配置文件路径，默认为 config/config.yaml Returns: 配置字典 Raises: FileParseError: 配置文件解析错误 """ if config_path is None: config_path = self.project_root / "config" / "config.yaml" else: config_path = Path(config_path) if not config_path.exists(): raise FileParseError(str(config_path), "配置文件不存在") try: with open(config_path, "r", encoding="utf-8") as f: config_data = yaml.safe_load(f) return config_data except Exception as e: raise FileParseError(str(config_path), str(e)) def parse_frequency_words(self, words_file: str = None) -> List[Dict]: """ 解析关键词配置文件（带 mtime 缓存） 仅当 frequency_words.txt 被修改时才重新解析，避免循环内重复 IO。 复用 trendradar.core.frequency 的解析逻辑，支持： - # 开头的注释行 - 空行分隔词组 - [组别名] 作为词组第一行，给整组指定别名 - +前缀必须词、!前缀过滤词、@数量限制 - /pattern/ 正则表达式语法 - => 别名 显示名称语法 - [GLOBAL_FILTER] 全局过滤区域 显示名称优先级：组别名 > 行别名拼接 > 关键词拼接 Args: words_file: 关键词文件路径，默认为 config/frequency_words.txt Returns: 词组列表 Raises: FileParseError: 文件解析错误 """ import os from trendradar.core.frequency import load_frequency_words if words_file is None: words_file = str(self.project_root / "config" / "frequency_words.txt") else: words_file = str(words_file) try: current_mtime = os.path.getmtime(words_file) if self._freq_words_cache is not None and current_mtime == self._freq_words_mtime: return self._freq_words_cache word_groups, filter_words, global_filters = load_frequency_words(words_file) self._freq_words_cache = word_groups self._freq_words_mtime = current_mtime return word_groups except FileNotFoundError: return [] except Exception as e: raise FileParseError(words_file, str(e)) def get_available_dates(self, db_type: str = "news") -> List[str]: """ 获取可用的日期列表 Args: db_type: 数据库类型 ("news" 或 "rss") Returns: 日期字符串列表（YYYY-MM-DD 格式，降序排列） """ db_dir = self.project_root / "output" / db_type if not db_dir.exists(): return [] dates = [] for db_file in db_dir.glob("*.db"): date_match = re.match(r'(\d{4}-\d{2}-\d{2})\.db$', db_file.name) if date_match: dates.append(date_match.group(1)) return sorted(dates, reverse=True) def get_available_date_range(self, db_type: str = "news") -> Tuple[Optional[datetime], Optional[datetime]]: """ 获取可用的日期范围 Args: db_type: 数据库类型 ("news" 或 "rss") Returns: (最早日期, 最新日期) 元组，如果没有数据则返回 (None, None) """ dates = self.get_available_dates(db_type) if not dates: return (None, None) earliest = datetime.strptime(dates[-1], "%Y-%m-%d") latest = datetime.strptime(dates[0], "%Y-%m-%d") return (earliest, latest) ================================================ FILE: mcp_server/tools/__init__.py ================================================ """ MCP 工具模块 包含所有MCP工具的实现。 """ ================================================ FILE: mcp_server/tools/analytics.py ================================================ """ 高级数据分析工具 提供热度趋势分析、平台对比、关键词共现、情感分析等高级分析功能。 """ import os import re from collections import Counter, defaultdict from datetime import datetime, timedelta from typing import Dict, List, Optional, Union from difflib import SequenceMatcher import yaml from trendradar.core.analyzer import calculate_news_weight as _calculate_news_weight from ..services.data_service import DataService from ..utils.validators import ( validate_platforms, validate_limit, validate_keyword, validate_top_n, validate_date_range, validate_threshold ) from ..utils.errors import MCPError, InvalidParameterError, DataNotFoundError # 权重配置 mtime 缓存（避免重复读取同一配置文件） _weight_config_cache: Optional[Dict] = None _weight_config_mtime: float = 0.0 _weight_config_path: Optional[str] = None _WEIGHT_DEFAULT_CONFIG = { "RANK_WEIGHT": 0.6, "FREQUENCY_WEIGHT": 0.3, "HOTNESS_WEIGHT": 0.1, } def _get_weight_config() -> Dict: """ 从 config.yaml 读取权重配置（带 mtime 缓存） 仅当配置文件被修改时才重新读取，避免循环内重复 IO。 Returns: 权重配置字典，包含 RANK_WEIGHT, FREQUENCY_WEIGHT, HOTNESS_WEIGHT """ global _weight_config_cache, _weight_config_mtime, _weight_config_path try: # 首次调用时计算路径（之后复用） if _weight_config_path is None: current_dir = os.path.dirname(os.path.abspath(__file__)) _weight_config_path = os.path.normpath( os.path.join(current_dir, "..", "..", "config", "config.yaml") ) current_mtime = os.path.getmtime(_weight_config_path) # 文件未修改且缓存有效，直接返回 if _weight_config_cache is not None and current_mtime == _weight_config_mtime: return _weight_config_cache # 文件已修改或首次读取，重新解析 with open(_weight_config_path, 'r', encoding='utf-8') as f: config = yaml.safe_load(f) weight = config.get('advanced', {}).get('weight', {}) _weight_config_cache = { "RANK_WEIGHT": weight.get('rank', 0.6), "FREQUENCY_WEIGHT": weight.get('frequency', 0.3), "HOTNESS_WEIGHT": weight.get('hotness', 0.1), } _weight_config_mtime = current_mtime return _weight_config_cache except Exception: return _WEIGHT_DEFAULT_CONFIG def calculate_news_weight(news_data: Dict, rank_threshold: int = 5) -> float: """ 计算新闻权重（用于排序） 复用 trendradar.core.analyzer.calculate_news_weight 实现， 权重配置从 config.yaml 的 advanced.weight 读取。 Args: news_data: 新闻数据字典，包含 ranks 和 count 字段 rank_threshold: 高排名阈值，默认5 Returns: 权重分数（0-100之间的浮点数） """ return _calculate_news_weight(news_data, rank_threshold, _get_weight_config()) class AnalyticsTools: """高级数据分析工具类""" def __init__(self, project_root: str = None): """ 初始化分析工具 Args: project_root: 项目根目录 """ self.data_service = DataService(project_root) def analyze_data_insights_unified( self, insight_type: str = "platform_compare", topic: Optional[str] = None, date_range: Optional[Union[Dict[str, str], str]] = None, min_frequency: int = 3, top_n: int = 20 ) -> Dict: """ 统一数据洞察分析工具 - 整合多种数据分析模式 Args: insight_type: 洞察类型，可选值： - "platform_compare": 平台对比分析（对比不同平台对话题的关注度） - "platform_activity": 平台活跃度统计（统计各平台发布频率和活跃时间） - "keyword_cooccur": 关键词共现分析（分析关键词同时出现的模式） topic: 话题关键词（可选，platform_compare模式适用） date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} min_frequency: 最小共现频次（keyword_cooccur模式），默认3 top_n: 返回TOP N结果（keyword_cooccur模式），默认20 Returns: 数据洞察分析结果字典 Examples: - analyze_data_insights_unified(insight_type="platform_compare", topic="人工智能") - analyze_data_insights_unified(insight_type="platform_activity", date_range={...}) - analyze_data_insights_unified(insight_type="keyword_cooccur", min_frequency=5) """ try: # 参数验证 if insight_type not in ["platform_compare", "platform_activity", "keyword_cooccur"]: raise InvalidParameterError( f"无效的洞察类型: {insight_type}", suggestion="支持的类型: platform_compare, platform_activity, keyword_cooccur" ) # 根据洞察类型调用相应方法 if insight_type == "platform_compare": return self.compare_platforms( topic=topic, date_range=date_range ) elif insight_type == "platform_activity": return self.get_platform_activity_stats( date_range=date_range ) else: # keyword_cooccur return self.analyze_keyword_cooccurrence( min_frequency=min_frequency, top_n=top_n ) except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def analyze_topic_trend_unified( self, topic: str, analysis_type: str = "trend", date_range: Optional[Union[Dict[str, str], str]] = None, granularity: str = "day", threshold: float = 3.0, time_window: int = 24, lookahead_hours: int = 6, confidence_threshold: float = 0.7 ) -> Dict: """ 统一话题趋势分析工具 - 整合多种趋势分析模式 Args: topic: 话题关键词（必需） analysis_type: 分析类型，可选值： - "trend": 热度趋势分析（追踪话题的热度变化） - "lifecycle": 生命周期分析（从出现到消失的完整周期） - "viral": 异常热度检测（识别突然爆火的话题） - "predict": 话题预测（预测未来可能的热点） date_range: 日期范围（trend和lifecycle模式），可选 - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **默认**: 不指定时默认分析最近7天 granularity: 时间粒度（trend模式），默认"day"（hour/day） threshold: 热度突增倍数阈值（viral模式），默认3.0 time_window: 检测时间窗口小时数（viral模式），默认24 lookahead_hours: 预测未来小时数（predict模式），默认6 confidence_threshold: 置信度阈值（predict模式），默认0.7 Returns: 趋势分析结果字典 Examples (假设今天是 2025-11-17): - 用户："分析AI最近7天的趋势" → analyze_topic_trend_unified(topic="人工智能", analysis_type="trend", date_range={"start": "2025-11-11", "end": "2025-11-17"}) - 用户："看看特斯拉本月的热度" → analyze_topic_trend_unified(topic="特斯拉", analysis_type="lifecycle", date_range={"start": "2025-11-01", "end": "2025-11-17"}) - analyze_topic_trend_unified(topic="比特币", analysis_type="viral", threshold=3.0) - analyze_topic_trend_unified(topic="ChatGPT", analysis_type="predict", lookahead_hours=6) """ try: # 参数验证 topic = validate_keyword(topic) if analysis_type not in ["trend", "lifecycle", "viral", "predict"]: raise InvalidParameterError( f"无效的分析类型: {analysis_type}", suggestion="支持的类型: trend, lifecycle, viral, predict" ) # 根据分析类型调用相应方法 if analysis_type == "trend": return self.get_topic_trend_analysis( topic=topic, date_range=date_range, granularity=granularity ) elif analysis_type == "lifecycle": return self.analyze_topic_lifecycle( topic=topic, date_range=date_range ) elif analysis_type == "viral": # viral模式不需要topic参数，使用通用检测 return self.detect_viral_topics( threshold=threshold, time_window=time_window ) else: # predict # predict模式不需要topic参数，使用通用预测 return self.predict_trending_topics( lookahead_hours=lookahead_hours, confidence_threshold=confidence_threshold ) except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_topic_trend_analysis( self, topic: str, date_range: Optional[Union[Dict[str, str], str]] = None, granularity: str = "day" ) -> Dict: """ 热度趋势分析 - 追踪特定话题的热度变化趋势 Args: topic: 话题关键词 date_range: 日期范围（可选） - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **默认**: 不指定时默认分析最近7天 granularity: 时间粒度，仅支持 day（天） Returns: 趋势分析结果字典 Examples: 用户询问示例： - "帮我分析一下'人工智能'这个话题最近一周的热度趋势" - "查看'比特币'过去一周的热度变化" - "看看'iPhone'最近7天的趋势如何" - "分析'特斯拉'最近一个月的热度趋势" - "查看'ChatGPT'2024年12月的趋势变化" 代码调用示例： >>> tools = AnalyticsTools() >>> # 分析7天趋势（假设今天是 2025-11-17） >>> result = tools.get_topic_trend_analysis( ... topic="人工智能", ... date_range={"start": "2025-11-11", "end": "2025-11-17"}, ... granularity="day" ... ) >>> # 分析历史月份趋势 >>> result = tools.get_topic_trend_analysis( ... topic="特斯拉", ... date_range={"start": "2024-12-01", "end": "2024-12-31"}, ... granularity="day" ... ) >>> print(result['trend_data']) """ try: # 验证参数 topic = validate_keyword(topic) # 验证粒度参数（只支持day） if granularity != "day": from ..utils.errors import InvalidParameterError raise InvalidParameterError( f"不支持的粒度参数: {granularity}", suggestion="当前仅支持 'day' 粒度，因为底层数据按天聚合" ) # 处理日期范围（不指定时默认最近7天） if date_range: from ..utils.validators import validate_date_range date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: # 默认最近7天 end_date = datetime.now() start_date = end_date - timedelta(days=6) # 收集趋势数据 trend_data = [] current_date = start_date while current_date <= end_date: try: all_titles, _, _ = self.data_service.parser.read_all_titles_for_date( date=current_date ) # 统计该时间点的话题出现次数 count = 0 matched_titles = [] for _, titles in all_titles.items(): for title in titles.keys(): if topic.lower() in title.lower(): count += 1 matched_titles.append(title) trend_data.append({ "date": current_date.strftime("%Y-%m-%d"), "count": count, "sample_titles": matched_titles[:3] # 只保留前3个样本 }) except DataNotFoundError: trend_data.append({ "date": current_date.strftime("%Y-%m-%d"), "count": 0, "sample_titles": [] }) # 按天增加时间 current_date += timedelta(days=1) # 计算趋势指标 counts = [item["count"] for item in trend_data] total_days = (end_date - start_date).days + 1 if len(counts) >= 2: # 计算涨跌幅度 first_non_zero = next((c for c in counts if c > 0), 0) last_count = counts[-1] if first_non_zero > 0: change_rate = ((last_count - first_non_zero) / first_non_zero) * 100 else: change_rate = 0 # 找到峰值时间 max_count = max(counts) peak_index = counts.index(max_count) peak_time = trend_data[peak_index]["date"] else: change_rate = 0 peak_time = None max_count = 0 return { "success": True, "summary": { "description": f"话题「{topic}」的热度趋势分析", "topic": topic, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d"), "total_days": total_days }, "granularity": granularity, "total_mentions": sum(counts), "average_mentions": round(sum(counts) / len(counts), 2) if counts else 0, "peak_count": max_count, "peak_time": peak_time, "change_rate": round(change_rate, 2), "trend_direction": "上升" if change_rate > 10 else "下降" if change_rate < -10 else "稳定" }, "data": trend_data } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def compare_platforms( self, topic: Optional[str] = None, date_range: Optional[Union[Dict[str, str], str]] = None ) -> Dict: """ 平台对比分析 - 对比不同平台对同一话题的关注度 Args: topic: 话题关键词（可选，不指定则对比整体活跃度） date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} Returns: 平台对比分析结果 Examples: 用户询问示例： - "对比一下各个平台对'人工智能'话题的关注度" - "看看知乎和微博哪个平台更关注科技新闻" - "分析各平台今天的热点分布" 代码调用示例： >>> # 对比各平台（假设今天是 2025-11-17） >>> result = tools.compare_platforms( ... topic="人工智能", ... date_range={"start": "2025-11-08", "end": "2025-11-17"} ... ) >>> print(result['platform_stats']) """ try: # 参数验证 if topic: topic = validate_keyword(topic) date_range_tuple = validate_date_range(date_range) # 确定日期范围 if date_range_tuple: start_date, end_date = date_range_tuple else: start_date = end_date = datetime.now() # 收集各平台数据 platform_stats = defaultdict(lambda: { "total_news": 0, "topic_mentions": 0, "unique_titles": set(), "top_keywords": Counter() }) # 遍历日期范围 current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date ) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title in titles.keys(): platform_stats[platform_name]["total_news"] += 1 platform_stats[platform_name]["unique_titles"].add(title) # 如果指定了话题，统计包含话题的新闻 if topic and topic.lower() in title.lower(): platform_stats[platform_name]["topic_mentions"] += 1 # 提取关键词（简单分词） keywords = self._extract_keywords(title) platform_stats[platform_name]["top_keywords"].update(keywords) except DataNotFoundError: pass current_date += timedelta(days=1) # 转换为可序列化的格式 result_stats = {} for platform, stats in platform_stats.items(): coverage_rate = 0 if stats["total_news"] > 0: coverage_rate = (stats["topic_mentions"] / stats["total_news"]) * 100 result_stats[platform] = { "total_news": stats["total_news"], "topic_mentions": stats["topic_mentions"], "unique_titles": len(stats["unique_titles"]), "coverage_rate": round(coverage_rate, 2), "top_keywords": [ {"keyword": k, "count": v} for k, v in stats["top_keywords"].most_common(5) ] } # 找出各平台独有的热点 unique_topics = self._find_unique_topics(platform_stats) return { "success": True, "topic": topic, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d") }, "platform_stats": result_stats, "unique_topics": unique_topics, "total_platforms": len(result_stats) } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def analyze_keyword_cooccurrence( self, min_frequency: int = 3, top_n: int = 20 ) -> Dict: """ 关键词共现分析 - 分析哪些关键词经常同时出现 Args: min_frequency: 最小共现频次 top_n: 返回TOP N关键词对 Returns: 关键词共现分析结果 Examples: 用户询问示例： - "分析一下哪些关键词经常一起出现" - "看看'人工智能'经常和哪些词一起出现" - "找出今天新闻中的关键词关联" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.analyze_keyword_cooccurrence( ... min_frequency=5, ... top_n=15 ... ) >>> print(result['cooccurrence_pairs']) """ try: # 参数验证 min_frequency = validate_limit(min_frequency, default=3, max_limit=100) top_n = validate_top_n(top_n, default=20) # 读取今天的数据 all_titles, _, _ = self.data_service.parser.read_all_titles_for_date() # 关键词共现统计 cooccurrence = Counter() keyword_titles = defaultdict(list) for platform_id, titles in all_titles.items(): for title in titles.keys(): # 提取关键词 keywords = self._extract_keywords(title) # 记录每个关键词出现的标题 for kw in keywords: keyword_titles[kw].append(title) # 计算两两共现 if len(keywords) >= 2: for i, kw1 in enumerate(keywords): for kw2 in keywords[i+1:]: # 统一排序，避免重复 pair = tuple(sorted([kw1, kw2])) cooccurrence[pair] += 1 # 过滤低频共现 filtered_pairs = [ (pair, count) for pair, count in cooccurrence.items() if count >= min_frequency ] # 排序并取TOP N top_pairs = sorted(filtered_pairs, key=lambda x: x[1], reverse=True)[:top_n] # 构建结果 result_pairs = [] for (kw1, kw2), count in top_pairs: # 找出同时包含两个关键词的标题样本 titles_with_both = [ title for title in keyword_titles[kw1] if kw2 in self._extract_keywords(title) ] result_pairs.append({ "keyword1": kw1, "keyword2": kw2, "cooccurrence_count": count, "sample_titles": titles_with_both[:3] }) return { "success": True, "summary": { "description": "关键词共现分析结果", "total": len(result_pairs), "min_frequency": min_frequency, "generated_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S") }, "data": result_pairs } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def analyze_sentiment( self, topic: Optional[str] = None, platforms: Optional[List[str]] = None, date_range: Optional[Union[Dict[str, str], str]] = None, limit: int = 50, sort_by_weight: bool = True, include_url: bool = False ) -> Dict: """ 情感倾向分析 - 生成用于 AI 情感分析的结构化提示词 本工具收集新闻数据并生成优化的 AI 提示词，你可以将其发送给 AI 进行深度情感分析。 Args: topic: 话题关键词（可选），只分析包含该关键词的新闻 platforms: 平台过滤列表（可选），如 ['zhihu', 'weibo'] date_range: 日期范围（可选），格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} 不指定则默认查询今天的数据 limit: 返回新闻数量限制，默认50，最大100 sort_by_weight: 是否按权重排序，默认True（推荐） include_url: 是否包含URL链接，默认False（节省token） Returns: 包含 AI 提示词和新闻数据的结构化结果 Examples: 用户询问示例： - "分析一下今天新闻的情感倾向" - "看看'特斯拉'相关新闻是正面还是负面的" - "分析各平台对'人工智能'的情感态度" - "看看'特斯拉'相关新闻是正面还是负面的，请选择一周内的前10条新闻来分析" 代码调用示例： >>> tools = AnalyticsTools() >>> # 分析今天的特斯拉新闻，返回前10条 >>> result = tools.analyze_sentiment( ... topic="特斯拉", ... limit=10 ... ) >>> # 分析一周内的特斯拉新闻（假设今天是 2025-11-17） >>> result = tools.analyze_sentiment( ... topic="特斯拉", ... date_range={"start": "2025-11-11", "end": "2025-11-17"}, ... limit=10 ... ) >>> print(result['ai_prompt']) # 获取生成的提示词 """ try: # 参数验证 if topic: topic = validate_keyword(topic) platforms = validate_platforms(platforms) limit = validate_limit(limit, default=50) # 处理日期范围 if date_range: date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: # 默认今天 start_date = end_date = datetime.now() # 收集新闻数据（支持多天） all_news_items = [] current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date, platform_ids=platforms ) # 收集该日期的新闻 for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 如果指定了话题，只收集包含话题的标题 if topic and topic.lower() not in title.lower(): continue news_item = { "platform": platform_name, "title": title, "ranks": info.get("ranks", []), "count": len(info.get("ranks", [])), "date": current_date.strftime("%Y-%m-%d") } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") all_news_items.append(news_item) except DataNotFoundError: # 该日期没有数据，继续下一天 pass # 下一天 current_date += timedelta(days=1) if not all_news_items: time_desc = "今天" if start_date == end_date else f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" raise DataNotFoundError( f"未找到相关新闻（{time_desc}）", suggestion="请尝试其他话题、日期范围或平台" ) # 去重（同一标题只保留一次） unique_news = {} for item in all_news_items: key = f"{item['platform']}::{item['title']}" if key not in unique_news: unique_news[key] = item else: # 合并 ranks（如果同一新闻在多天出现） existing = unique_news[key] existing["ranks"].extend(item["ranks"]) existing["count"] = len(existing["ranks"]) deduplicated_news = list(unique_news.values()) # 按权重排序（如果启用） if sort_by_weight: deduplicated_news.sort( key=lambda x: calculate_news_weight(x), reverse=True ) # 限制返回数量 selected_news = deduplicated_news[:limit] # 生成 AI 提示词 ai_prompt = self._create_sentiment_analysis_prompt( news_data=selected_news, topic=topic ) # 构建时间范围描述 if start_date == end_date: time_range_desc = start_date.strftime("%Y-%m-%d") else: time_range_desc = f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" result = { "success": True, "method": "ai_prompt_generation", "summary": { "description": "情感分析数据和AI提示词", "total_found": len(deduplicated_news), "returned": len(selected_news), "requested_limit": limit, "duplicates_removed": len(all_news_items) - len(deduplicated_news), "topic": topic, "time_range": time_range_desc, "platforms": list(set(item["platform"] for item in selected_news)), "sorted_by_weight": sort_by_weight }, "ai_prompt": ai_prompt, "data": selected_news, "usage_note": "请将 ai_prompt 字段的内容发送给 AI 进行情感分析" } # 如果返回数量少于请求数量，增加提示 if len(selected_news) < limit and len(deduplicated_news) >= limit: result["note"] = "返回数量少于请求数量是因为去重逻辑（同一标题在不同平台只保留一次）" elif len(deduplicated_news) < limit: result["note"] = f"在指定时间范围内仅找到 {len(deduplicated_news)} 条匹配的新闻" return result except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def _create_sentiment_analysis_prompt( self, news_data: List[Dict], topic: Optional[str] ) -> str: """ 创建情感分析的 AI 提示词 Args: news_data: 新闻数据列表（已排序和限制数量） topic: 话题关键词 Returns: 格式化的 AI 提示词 """ # 按平台分组 platform_news = defaultdict(list) for item in news_data: platform_news[item["platform"]].append({ "title": item["title"], "date": item.get("date", "") }) # 构建提示词 prompt_parts = [] # 1. 任务说明 if topic: prompt_parts.append(f"请分析以下关于「{topic}」的新闻标题的情感倾向。") else: prompt_parts.append("请分析以下新闻标题的情感倾向。") prompt_parts.append("") prompt_parts.append("分析要求：") prompt_parts.append("1. 识别每条新闻的情感倾向（正面/负面/中性）") prompt_parts.append("2. 统计各情感类别的数量和百分比") prompt_parts.append("3. 分析不同平台的情感差异") prompt_parts.append("4. 总结整体情感趋势") prompt_parts.append("5. 列举典型的正面和负面新闻样本") prompt_parts.append("") # 2. 数据概览 prompt_parts.append(f"数据概览：") prompt_parts.append(f"- 总新闻数：{len(news_data)}") prompt_parts.append(f"- 覆盖平台：{len(platform_news)}") # 时间范围 dates = set(item.get("date", "") for item in news_data if item.get("date")) if dates: date_list = sorted(dates) if len(date_list) == 1: prompt_parts.append(f"- 时间范围：{date_list[0]}") else: prompt_parts.append(f"- 时间范围：{date_list[0]} 至 {date_list[-1]}") prompt_parts.append("") # 3. 按平台展示新闻 prompt_parts.append("新闻列表（按平台分类，已按重要性排序）：") prompt_parts.append("") for platform, items in sorted(platform_news.items()): prompt_parts.append(f"【{platform}】({len(items)} 条)") for i, item in enumerate(items, 1): title = item["title"] date_str = f" [{item['date']}]" if item.get("date") else "" prompt_parts.append(f"{i}. {title}{date_str}") prompt_parts.append("") # 4. 输出格式说明 prompt_parts.append("请按以下格式输出分析结果：") prompt_parts.append("") prompt_parts.append("## 情感分布统计") prompt_parts.append("- 正面：XX条 (XX%)") prompt_parts.append("- 负面：XX条 (XX%)") prompt_parts.append("- 中性：XX条 (XX%)") prompt_parts.append("") prompt_parts.append("## 平台情感对比") prompt_parts.append("[各平台的情感倾向差异]") prompt_parts.append("") prompt_parts.append("## 整体情感趋势") prompt_parts.append("[总体分析和关键发现]") prompt_parts.append("") prompt_parts.append("## 典型样本") prompt_parts.append("正面新闻样本：") prompt_parts.append("[列举3-5条]") prompt_parts.append("") prompt_parts.append("负面新闻样本：") prompt_parts.append("[列举3-5条]") return "\n".join(prompt_parts) def find_similar_news( self, reference_title: str, threshold: float = 0.6, limit: int = 50, include_url: bool = False ) -> Dict: """ 相似新闻查找 - 基于标题相似度查找相关新闻 Args: reference_title: 参考标题 threshold: 相似度阈值（0-1之间） limit: 返回条数限制，默认50 include_url: 是否包含URL链接，默认False（节省token） Returns: 相似新闻列表 Examples: 用户询问示例： - "找出和'特斯拉降价'相似的新闻" - "查找关于iPhone发布的类似报道" - "看看有没有和这条新闻相似的报道" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.find_similar_news( ... reference_title="特斯拉宣布降价", ... threshold=0.6, ... limit=10 ... ) >>> print(result['similar_news']) """ try: # 参数验证 reference_title = validate_keyword(reference_title) threshold = validate_threshold(threshold, default=0.6, min_value=0.0, max_value=1.0) limit = validate_limit(limit, default=50) # 读取数据 all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date() # 计算相似度 similar_items = [] for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): if title == reference_title: continue # 计算相似度 similarity = self._calculate_similarity(reference_title, title) if similarity >= threshold: news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "similarity": round(similarity, 3), "rank": info["ranks"][0] if info["ranks"] else 0 } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") similar_items.append(news_item) # 按相似度排序 similar_items.sort(key=lambda x: x["similarity"], reverse=True) # 限制数量 result_items = similar_items[:limit] if not result_items: raise DataNotFoundError( f"未找到相似度超过 {threshold} 的新闻", suggestion="请降低相似度阈值或尝试其他标题" ) result = { "success": True, "summary": { "description": "相似新闻搜索结果", "total_found": len(similar_items), "returned": len(result_items), "requested_limit": limit, "threshold": threshold, "reference_title": reference_title }, "data": result_items } if len(similar_items) < limit: result["note"] = f"相似度阈值 {threshold} 下仅找到 {len(similar_items)} 条相似新闻" return result except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def search_by_entity( self, entity: str, entity_type: Optional[str] = None, limit: int = 50, sort_by_weight: bool = True ) -> Dict: """ 实体识别搜索 - 搜索包含特定人物/地点/机构的新闻 Args: entity: 实体名称 entity_type: 实体类型（person/location/organization），可选 limit: 返回条数限制，默认50，最大200 sort_by_weight: 是否按权重排序，默认True Returns: 实体相关新闻列表 Examples: 用户询问示例： - "搜索马斯克相关的新闻" - "查找关于特斯拉公司的报道，返回前20条" - "看看北京有什么新闻" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.search_by_entity( ... entity="马斯克", ... entity_type="person", ... limit=20 ... ) >>> print(result['related_news']) """ try: # 参数验证 entity = validate_keyword(entity) limit = validate_limit(limit, default=50) if entity_type and entity_type not in ["person", "location", "organization"]: raise InvalidParameterError( f"无效的实体类型: {entity_type}", suggestion="支持的类型: person, location, organization" ) # 读取数据 all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date() # 搜索包含实体的新闻 related_news = [] entity_context = Counter() # 统计实体周边的词 for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): if entity in title: url = info.get("url", "") mobile_url = info.get("mobileUrl", "") ranks = info.get("ranks", []) count = len(ranks) related_news.append({ "title": title, "platform": platform_id, "platform_name": platform_name, "url": url, "mobileUrl": mobile_url, "ranks": ranks, "count": count, "rank": ranks[0] if ranks else 999 }) # 提取实体周边的关键词 keywords = self._extract_keywords(title) entity_context.update(keywords) if not related_news: raise DataNotFoundError( f"未找到包含实体 '{entity}' 的新闻", suggestion="请尝试其他实体名称" ) # 移除实体本身 if entity in entity_context: del entity_context[entity] # 按权重排序（如果启用） if sort_by_weight: related_news.sort( key=lambda x: calculate_news_weight(x), reverse=True ) else: # 按排名排序 related_news.sort(key=lambda x: x["rank"]) # 限制返回数量 result_news = related_news[:limit] return { "success": True, "summary": { "description": f"实体「{entity}」相关新闻", "entity": entity, "entity_type": entity_type or "auto", "total_found": len(related_news), "returned": len(result_news), "sorted_by_weight": sort_by_weight }, "data": result_news, "related_keywords": [ {"keyword": k, "count": v} for k, v in entity_context.most_common(10) ] } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def generate_summary_report( self, report_type: str = "daily", date_range: Optional[Union[Dict[str, str], str]] = None ) -> Dict: """ 每日/每周摘要生成器 - 自动生成热点摘要报告 Args: report_type: 报告类型（daily/weekly） date_range: 自定义日期范围（可选） Returns: Markdown格式的摘要报告 Examples: 用户询问示例： - "生成今天的新闻摘要报告" - "给我一份本周的热点总结" - "生成过去7天的新闻分析报告" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.generate_summary_report( ... report_type="daily" ... ) >>> print(result['markdown_report']) """ try: # 参数验证 if report_type not in ["daily", "weekly"]: raise InvalidParameterError( f"无效的报告类型: {report_type}", suggestion="支持的类型: daily, weekly" ) # 确定日期范围 if date_range: date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: if report_type == "daily": start_date = end_date = datetime.now() else: # weekly end_date = datetime.now() start_date = end_date - timedelta(days=6) # 收集数据 all_keywords = Counter() all_platforms_news = defaultdict(int) all_titles_list = [] current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date ) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) all_platforms_news[platform_name] += len(titles) for title in titles.keys(): all_titles_list.append({ "title": title, "platform": platform_name, "date": current_date.strftime("%Y-%m-%d") }) # 提取关键词 keywords = self._extract_keywords(title) all_keywords.update(keywords) except DataNotFoundError: pass current_date += timedelta(days=1) # 生成报告 report_title = f"{'每日' if report_type == 'daily' else '每周'}新闻热点摘要" date_str = f"{start_date.strftime('%Y-%m-%d')}" if report_type == "daily" else f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" # 构建Markdown报告 markdown = f"""# {report_title} **报告日期**: {date_str} **生成时间**: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')} --- ## 📊 数据概览 - **总新闻数**: {len(all_titles_list)} - **覆盖平台**: {len(all_platforms_news)} - **热门关键词数**: {len(all_keywords)} ## 🔥 TOP 10 热门话题 """ # 添加TOP 10关键词 for i, (keyword, count) in enumerate(all_keywords.most_common(10), 1): markdown += f"{i}. **{keyword}** - 出现 {count} 次\n" # 平台分析 markdown += "\n## 📱 平台活跃度\n\n" sorted_platforms = sorted(all_platforms_news.items(), key=lambda x: x[1], reverse=True) for platform, count in sorted_platforms: markdown += f"- **{platform}**: {count} 条新闻\n" # 趋势变化（如果是周报） if report_type == "weekly": markdown += "\n## 📈 趋势分析\n\n" markdown += "本周热度持续的话题（样本数据）：\n\n" # 简单的趋势分析 top_keywords = [kw for kw, _ in all_keywords.most_common(5)] for keyword in top_keywords: markdown += f"- **{keyword}**: 持续热门\n" # 添加样本新闻（按权重选择，确保确定性） markdown += "\n## 📰 精选新闻样本\n\n" # 确定性选取：按标题的权重排序，取前5条 # 这样相同输入总是返回相同结果 if all_titles_list: # 计算每条新闻的权重分数（基于关键词出现次数） news_with_scores = [] for news in all_titles_list: # 简单权重：统计包含TOP关键词的次数 score = 0 title_lower = news['title'].lower() for keyword, count in all_keywords.most_common(10): if keyword.lower() in title_lower: score += count news_with_scores.append((news, score)) # 按权重降序排序，权重相同则按标题字母顺序（确保确定性） news_with_scores.sort(key=lambda x: (-x[1], x[0]['title'])) # 取前5条 sample_news = [item[0] for item in news_with_scores[:5]] for news in sample_news: markdown += f"- [{news['platform']}] {news['title']}\n" markdown += "\n---\n\n*本报告由 TrendRadar MCP 自动生成*\n" return { "success": True, "report_type": report_type, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d") }, "markdown_report": markdown, "statistics": { "total_news": len(all_titles_list), "platforms_count": len(all_platforms_news), "keywords_count": len(all_keywords), "top_keyword": all_keywords.most_common(1)[0] if all_keywords else None } } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_platform_activity_stats( self, date_range: Optional[Union[Dict[str, str], str]] = None ) -> Dict: """ 平台活跃度统计 - 统计各平台的发布频率和活跃时间段 Args: date_range: 日期范围（可选） Returns: 平台活跃度统计结果 Examples: 用户询问示例： - "统计各平台今天的活跃度" - "看看哪个平台更新最频繁" - "分析各平台的发布时间规律" 代码调用示例： >>> # 查看各平台活跃度（假设今天是 2025-11-17） >>> result = tools.get_platform_activity_stats( ... date_range={"start": "2025-11-08", "end": "2025-11-17"} ... ) >>> print(result['platform_activity']) """ try: # 参数验证 date_range_tuple = validate_date_range(date_range) # 确定日期范围 if date_range_tuple: start_date, end_date = date_range_tuple else: start_date = end_date = datetime.now() # 统计各平台活跃度 platform_activity = defaultdict(lambda: { "total_updates": 0, "days_active": set(), "news_count": 0, "hourly_distribution": Counter() }) # 遍历日期范围 current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, timestamps = self.data_service.parser.read_all_titles_for_date( date=current_date ) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) platform_activity[platform_name]["news_count"] += len(titles) platform_activity[platform_name]["days_active"].add(current_date.strftime("%Y-%m-%d")) # 统计更新次数（基于文件数量） platform_activity[platform_name]["total_updates"] += len(timestamps) # 统计时间分布（基于文件名中的时间） for filename in timestamps.keys(): # 解析文件名中的小时（格式：HHMM.txt） match = re.match(r'(\d{2})(\d{2})\.txt', filename) if match: hour = int(match.group(1)) platform_activity[platform_name]["hourly_distribution"][hour] += 1 except DataNotFoundError: pass current_date += timedelta(days=1) # 转换为可序列化的格式 result_activity = {} for platform, stats in platform_activity.items(): days_count = len(stats["days_active"]) avg_news_per_day = stats["news_count"] / days_count if days_count > 0 else 0 # 找出最活跃的时间段 most_active_hours = stats["hourly_distribution"].most_common(3) result_activity[platform] = { "total_updates": stats["total_updates"], "news_count": stats["news_count"], "days_active": days_count, "avg_news_per_day": round(avg_news_per_day, 2), "most_active_hours": [ {"hour": f"{hour:02d}:00", "count": count} for hour, count in most_active_hours ], "activity_score": round(stats["news_count"] / max(days_count, 1), 2) } # 按活跃度排序 sorted_platforms = sorted( result_activity.items(), key=lambda x: x[1]["activity_score"], reverse=True ) return { "success": True, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d") }, "platform_activity": dict(sorted_platforms), "most_active_platform": sorted_platforms[0][0] if sorted_platforms else None, "total_platforms": len(result_activity) } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def analyze_topic_lifecycle( self, topic: str, date_range: Optional[Union[Dict[str, str], str]] = None ) -> Dict: """ 话题生命周期分析 - 追踪话题从出现到消失的完整周期 Args: topic: 话题关键词 date_range: 日期范围（可选） - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **默认**: 不指定时默认分析最近7天 Returns: 话题生命周期分析结果 Examples: 用户询问示例： - "分析'人工智能'这个话题的生命周期" - "看看'iPhone'话题是昙花一现还是持续热点" - "追踪'比特币'话题的热度变化" 代码调用示例： >>> # 分析话题生命周期（假设今天是 2025-11-17） >>> result = tools.analyze_topic_lifecycle( ... topic="人工智能", ... date_range={"start": "2025-10-19", "end": "2025-11-17"} ... ) >>> print(result['lifecycle_stage']) """ try: # 参数验证 topic = validate_keyword(topic) # 处理日期范围（不指定时默认最近7天） if date_range: from ..utils.validators import validate_date_range date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: # 默认最近7天 end_date = datetime.now() start_date = end_date - timedelta(days=6) # 收集话题历史数据 lifecycle_data = [] current_date = start_date while current_date <= end_date: try: all_titles, _, _ = self.data_service.parser.read_all_titles_for_date( date=current_date ) # 统计该日的话题出现次数 count = 0 for _, titles in all_titles.items(): for title in titles.keys(): if topic.lower() in title.lower(): count += 1 lifecycle_data.append({ "date": current_date.strftime("%Y-%m-%d"), "count": count }) except DataNotFoundError: lifecycle_data.append({ "date": current_date.strftime("%Y-%m-%d"), "count": 0 }) current_date += timedelta(days=1) # 计算分析天数 total_days = (end_date - start_date).days + 1 # 分析生命周期阶段 counts = [item["count"] for item in lifecycle_data] if not any(counts): time_desc = f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" raise DataNotFoundError( f"在 {time_desc} 内未找到话题 '{topic}'", suggestion="请尝试其他话题或扩大时间范围" ) # 找到首次出现和最后出现 first_appearance = next((item["date"] for item in lifecycle_data if item["count"] > 0), None) last_appearance = next((item["date"] for item in reversed(lifecycle_data) if item["count"] > 0), None) # 计算峰值 max_count = max(counts) peak_index = counts.index(max_count) peak_date = lifecycle_data[peak_index]["date"] # 计算平均值和标准差（简单实现） non_zero_counts = [c for c in counts if c > 0] avg_count = sum(non_zero_counts) / len(non_zero_counts) if non_zero_counts else 0 # 判断生命周期阶段 recent_counts = counts[-3:] # 最近3天 early_counts = counts[:3] # 前3天 if sum(recent_counts) > sum(early_counts): lifecycle_stage = "上升期" elif sum(recent_counts) < sum(early_counts) * 0.5: lifecycle_stage = "衰退期" elif max_count in recent_counts: lifecycle_stage = "爆发期" else: lifecycle_stage = "稳定期" # 分类：昙花一现 vs 持续热点 active_days = sum(1 for c in counts if c > 0) if active_days <= 2 and max_count > avg_count * 2: topic_type = "昙花一现" elif active_days >= total_days * 0.6: topic_type = "持续热点" else: topic_type = "周期性热点" return { "success": True, "topic": topic, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d"), "total_days": total_days }, "lifecycle_data": lifecycle_data, "analysis": { "first_appearance": first_appearance, "last_appearance": last_appearance, "peak_date": peak_date, "peak_count": max_count, "active_days": active_days, "avg_daily_mentions": round(avg_count, 2), "lifecycle_stage": lifecycle_stage, "topic_type": topic_type } } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def detect_viral_topics( self, threshold: float = 3.0, time_window: int = 24 ) -> Dict: """ 异常热度检测 - 自动识别突然爆火的话题 Args: threshold: 热度突增倍数阈值 time_window: 检测时间窗口（小时） Returns: 爆火话题列表 Examples: 用户询问示例： - "检测今天有哪些突然爆火的话题" - "看看有没有热度异常的新闻" - "预警可能的重大事件" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.detect_viral_topics( ... threshold=3.0, ... time_window=24 ... ) >>> print(result['viral_topics']) """ try: # 参数验证 threshold = validate_threshold(threshold, default=3.0, min_value=1.0, max_value=100.0) time_window = validate_limit(time_window, default=24, max_limit=72) # 读取当前和之前的数据 current_all_titles, _, _ = self.data_service.parser.read_all_titles_for_date() # 读取昨天的数据作为基准 yesterday = datetime.now() - timedelta(days=1) try: previous_all_titles, _, _ = self.data_service.parser.read_all_titles_for_date( date=yesterday ) except DataNotFoundError: previous_all_titles = {} # 统计当前的关键词频率 current_keywords = Counter() current_keyword_titles = defaultdict(list) for _, titles in current_all_titles.items(): for title in titles.keys(): keywords = self._extract_keywords(title) current_keywords.update(keywords) for kw in keywords: current_keyword_titles[kw].append(title) # 统计之前的关键词频率 previous_keywords = Counter() for _, titles in previous_all_titles.items(): for title in titles.keys(): keywords = self._extract_keywords(title) previous_keywords.update(keywords) # 检测异常热度 viral_topics = [] for keyword, current_count in current_keywords.items(): previous_count = previous_keywords.get(keyword, 0) # 计算增长倍数 if previous_count == 0: # 新出现的话题 if current_count >= 5: # 至少出现5次才认为是爆火 growth_rate = float('inf') is_viral = True else: continue else: growth_rate = current_count / previous_count is_viral = growth_rate >= threshold if is_viral: viral_topics.append({ "keyword": keyword, "current_count": current_count, "previous_count": previous_count, "growth_rate": round(growth_rate, 2) if growth_rate != float('inf') else "新话题", "sample_titles": current_keyword_titles[keyword][:3], "alert_level": "高" if growth_rate > threshold * 2 else "中" }) # 按增长率排序 viral_topics.sort( key=lambda x: x["current_count"] if x["growth_rate"] == "新话题" else x["growth_rate"], reverse=True ) if not viral_topics: return { "success": True, "summary": { "description": "异常热度检测结果", "total": 0, "threshold": threshold, "time_window": time_window }, "data": [], "message": f"未检测到热度增长超过 {threshold} 倍的话题" } return { "success": True, "summary": { "description": "异常热度检测结果", "total": len(viral_topics), "threshold": threshold, "time_window": time_window, "detection_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S") }, "data": viral_topics } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def predict_trending_topics( self, lookahead_hours: int = 6, confidence_threshold: float = 0.7 ) -> Dict: """ 话题预测 - 基于历史数据预测未来可能的热点 Args: lookahead_hours: 预测未来多少小时 confidence_threshold: 置信度阈值 Returns: 预测的潜力话题列表 Examples: 用户询问示例： - "预测接下来6小时可能的热点话题" - "有哪些话题可能会火起来" - "早期发现潜力话题" 代码调用示例： >>> tools = AnalyticsTools() >>> result = tools.predict_trending_topics( ... lookahead_hours=6, ... confidence_threshold=0.7 ... ) >>> print(result['predicted_topics']) """ try: # 参数验证 lookahead_hours = validate_limit(lookahead_hours, default=6, max_limit=48) confidence_threshold = validate_threshold( confidence_threshold, default=0.7, min_value=0.0, max_value=1.0, param_name="confidence_threshold" ) # 收集最近3天的数据用于预测 keyword_trends = defaultdict(list) for days_ago in range(3, 0, -1): date = datetime.now() - timedelta(days=days_ago) try: all_titles, _, _ = self.data_service.parser.read_all_titles_for_date( date=date ) # 统计关键词 keywords_count = Counter() for _, titles in all_titles.items(): for title in titles.keys(): keywords = self._extract_keywords(title) keywords_count.update(keywords) # 记录每个关键词的历史数据 for keyword, count in keywords_count.items(): keyword_trends[keyword].append(count) except DataNotFoundError: pass # 添加今天的数据 try: all_titles, _, _ = self.data_service.parser.read_all_titles_for_date() keywords_count = Counter() keyword_titles = defaultdict(list) for _, titles in all_titles.items(): for title in titles.keys(): keywords = self._extract_keywords(title) keywords_count.update(keywords) for kw in keywords: keyword_titles[kw].append(title) for keyword, count in keywords_count.items(): keyword_trends[keyword].append(count) except DataNotFoundError: raise DataNotFoundError( "未找到今天的数据", suggestion="请等待爬虫任务完成" ) # 预测潜力话题 predicted_topics = [] for keyword, trend_data in keyword_trends.items(): if len(trend_data) < 2: continue # 简单的线性趋势预测 # 计算增长率 recent_value = trend_data[-1] previous_value = trend_data[-2] if len(trend_data) >= 2 else 0 if previous_value == 0: if recent_value >= 3: growth_rate = 1.0 else: continue else: growth_rate = (recent_value - previous_value) / previous_value # 判断是否是上升趋势 if growth_rate > 0.3: # 增长超过30% # 计算置信度（基于趋势的稳定性） if len(trend_data) >= 3: # 检查是否连续增长 is_consistent = all( trend_data[i] <= trend_data[i+1] for i in range(len(trend_data)-1) ) confidence = 0.9 if is_consistent else 0.7 else: confidence = 0.6 if confidence >= confidence_threshold: predicted_topics.append({ "keyword": keyword, "current_count": recent_value, "growth_rate": round(growth_rate * 100, 2), "confidence": round(confidence, 2), "trend_data": trend_data, "prediction": "上升趋势，可能成为热点", "sample_titles": keyword_titles.get(keyword, [])[:3] }) # 按置信度和增长率排序 predicted_topics.sort( key=lambda x: (x["confidence"], x["growth_rate"]), reverse=True ) return { "success": True, "summary": { "description": "热点话题预测结果", "total": len(predicted_topics), "returned": min(20, len(predicted_topics)), "lookahead_hours": lookahead_hours, "confidence_threshold": confidence_threshold, "prediction_time": datetime.now().strftime("%Y-%m-%d %H:%M:%S") }, "data": predicted_topics[:20], # 返回TOP 20 "note": "预测基于历史趋势，实际结果可能有偏差" } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } # ==================== 辅助方法 ==================== def _extract_keywords(self, title: str, min_length: int = 2) -> List[str]: """ 从标题中提取关键词（简单实现） Args: title: 标题文本 min_length: 最小关键词长度 Returns: 关键词列表 """ # 移除URL和特殊字符 title = re.sub(r'http[s]?://\S+', '', title) title = re.sub(r'[^\w\s]', ' ', title) # 简单分词（按空格和常见分隔符） words = re.split(r'[\s，。！？、]+', title) # 过滤停用词和短词 stopwords = {'的', '了', '在', '是', '我', '有', '和', '就', '不', '人', '都', '一', '一个', '上', '也', '很', '到', '说', '要', '去', '你', '会', '着', '没有', '看', '好', '自己', '这'} keywords = [ word.strip() for word in words if word.strip() and len(word.strip()) >= min_length and word.strip() not in stopwords ] return keywords def _calculate_similarity(self, text1: str, text2: str) -> float: """ 计算两个文本的相似度 Args: text1: 文本1 text2: 文本2 Returns: 相似度分数（0-1之间） """ # 使用 SequenceMatcher 计算相似度 return SequenceMatcher(None, text1, text2).ratio() def _find_unique_topics(self, platform_stats: Dict) -> Dict[str, List[str]]: """ 找出各平台独有的热点话题 Args: platform_stats: 平台统计数据 Returns: 各平台独有话题字典 """ unique_topics = {} # 获取每个平台的TOP关键词 platform_keywords = {} for platform, stats in platform_stats.items(): top_keywords = set([kw for kw, _ in stats["top_keywords"].most_common(10)]) platform_keywords[platform] = top_keywords # 找出独有关键词 for platform, keywords in platform_keywords.items(): # 找出其他平台的所有关键词 other_keywords = set() for other_platform, other_kws in platform_keywords.items(): if other_platform != platform: other_keywords.update(other_kws) # 找出独有的 unique = keywords - other_keywords if unique: unique_topics[platform] = list(unique)[:5] # 最多5个 return unique_topics # ==================== 跨平台聚合工具 ==================== def aggregate_news( self, date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, similarity_threshold: float = 0.7, limit: int = 50, include_url: bool = False ) -> Dict: """ 跨平台新闻聚合 - 对相似新闻进行去重合并 将不同平台报道的同一事件合并为一条聚合新闻， 显示该新闻在各平台的覆盖情况和综合热度。 Args: date_range: 日期范围（可选） - 不指定: 查询今天 - {\"start\": \"YYYY-MM-DD\", \"end\": \"YYYY-MM-DD\"}: 日期范围 platforms: 平台过滤列表，如 ['zhihu', 'weibo'] similarity_threshold: 相似度阈值，0-1之间，默认0.7 limit: 返回聚合新闻数量，默认50 include_url: 是否包含URL链接，默认False Returns: 聚合结果字典，包含： - aggregated_news: 聚合后的新闻列表 - statistics: 聚合统计信息 """ try: # 参数验证 platforms = validate_platforms(platforms) similarity_threshold = validate_threshold( similarity_threshold, default=0.7, min_value=0.3, max_value=1.0 ) limit = validate_limit(limit, default=50) # 处理日期范围 if date_range: date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: start_date = end_date = datetime.now() # 收集所有新闻 all_news = [] current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date, platform_ids=platforms ) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "ranks": info.get("ranks", []), "count": len(info.get("ranks", [])), "rank": info["ranks"][0] if info["ranks"] else 999 } if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") # 计算权重 news_item["weight"] = calculate_news_weight(news_item) all_news.append(news_item) except DataNotFoundError: pass current_date += timedelta(days=1) if not all_news: return { "success": True, "summary": { "description": "跨平台新闻聚合结果", "total": 0, "returned": 0 }, "data": [], "message": "未找到新闻数据" } # 执行聚合 aggregated = self._aggregate_similar_news( all_news, similarity_threshold, include_url ) # 按综合权重排序 aggregated.sort(key=lambda x: x["aggregate_weight"], reverse=True) # 限制返回数量 results = aggregated[:limit] # 统计信息 total_original = len(all_news) total_aggregated = len(aggregated) dedup_rate = 1 - (total_aggregated / total_original) if total_original > 0 else 0 platform_coverage = Counter() for item in aggregated: for p in item["platforms"]: platform_coverage[p] += 1 return { "success": True, "summary": { "description": "跨平台新闻聚合结果", "original_count": total_original, "aggregated_count": total_aggregated, "returned": len(results), "deduplication_rate": f"{dedup_rate * 100:.1f}%", "similarity_threshold": similarity_threshold, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d") } }, "data": results, "statistics": { "platform_coverage": dict(platform_coverage), "multi_platform_news": len([a for a in aggregated if len(a["platforms"]) > 1]), "single_platform_news": len([a for a in aggregated if len(a["platforms"]) == 1]) } } except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return {"success": False, "error": {"code": "INTERNAL_ERROR", "message": str(e)}} def _aggregate_similar_news( self, news_list: List[Dict], threshold: float, include_url: bool ) -> List[Dict]: """ 对新闻列表进行相似度聚合 使用双层过滤策略：先用 Jaccard 快速粗筛，再用 SequenceMatcher 精确计算 Args: news_list: 新闻列表 threshold: 相似度阈值 include_url: 是否包含URL Returns: 聚合后的新闻列表 """ if not news_list: return [] # 预计算字符集合用于快速过滤 prepared_news = [] for news in news_list: char_set = set(news["title"]) prepared_news.append({ "data": news, "char_set": char_set, "set_len": len(char_set) }) # 按权重排序 sorted_items = sorted(prepared_news, key=lambda x: x["data"].get("weight", 0), reverse=True) aggregated = [] used_indices = set() PRE_FILTER_RATIO = 0.5 # 粗筛阈值系数 for i, item in enumerate(sorted_items): if i in used_indices: continue news = item["data"] base_set = item["char_set"] base_len = item["set_len"] group = { "representative_title": news["title"], "platforms": [news["platform_name"]], "platform_ids": [news["platform"]], "dates": [news["date"]], "best_rank": news["rank"], "total_count": news["count"], "aggregate_weight": news.get("weight", 0), "sources": [{ "platform": news["platform_name"], "rank": news["rank"], "date": news["date"] }] } if include_url and news.get("url"): group["urls"] = [{ "platform": news["platform_name"], "url": news.get("url", ""), "mobileUrl": news.get("mobileUrl", "") }] used_indices.add(i) # 查找相似新闻 for j in range(i + 1, len(sorted_items)): if j in used_indices: continue compare_item = sorted_items[j] compare_set = compare_item["char_set"] compare_len = compare_item["set_len"] # 快速粗筛：长度检查 if base_len == 0 or compare_len == 0: continue # 快速粗筛：长度比例检查 if min(base_len, compare_len) / max(base_len, compare_len) < (threshold * PRE_FILTER_RATIO): continue # 快速粗筛：Jaccard 相似度 intersection = len(base_set & compare_set) union = len(base_set | compare_set) jaccard_sim = intersection / union if union > 0 else 0 if jaccard_sim < (threshold * PRE_FILTER_RATIO): continue # 精确计算：SequenceMatcher other_news = compare_item["data"] real_similarity = self._calculate_similarity(news["title"], other_news["title"]) if real_similarity >= threshold: # 合并到当前组 if other_news["platform_name"] not in group["platforms"]: group["platforms"].append(other_news["platform_name"]) group["platform_ids"].append(other_news["platform"]) if other_news["date"] not in group["dates"]: group["dates"].append(other_news["date"]) group["best_rank"] = min(group["best_rank"], other_news["rank"]) group["total_count"] += other_news["count"] group["aggregate_weight"] += other_news.get("weight", 0) * 0.5 # 额外权重 group["sources"].append({ "platform": other_news["platform_name"], "rank": other_news["rank"], "date": other_news["date"] }) if include_url and other_news.get("url"): if "urls" not in group: group["urls"] = [] group["urls"].append({ "platform": other_news["platform_name"], "url": other_news.get("url", ""), "mobileUrl": other_news.get("mobileUrl", "") }) used_indices.add(j) # 添加聚合信息 group["platform_count"] = len(group["platforms"]) group["is_cross_platform"] = len(group["platforms"]) > 1 aggregated.append(group) return aggregated # ==================== 时期对比分析工具 ==================== def compare_periods( self, period1: Union[Dict[str, str], str], period2: Union[Dict[str, str], str], topic: Optional[str] = None, compare_type: str = "overview", platforms: Optional[List[str]] = None, top_n: int = 10 ) -> Dict: """ 时期对比分析 - 比较两个时间段的新闻数据 支持多种对比维度：热度对比、话题变化、平台活跃度等。 Args: period1: 第一个时间段 - {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}: 日期范围 - "today", "yesterday", "last_week", "last_month": 预设值 period2: 第二个时间段（格式同 period1） topic: 可选的话题关键词（聚焦特定话题的对比） compare_type: 对比类型 - "overview": 总体概览（默认） - "topic_shift": 话题变化分析 - "platform_activity": 平台活跃度对比 platforms: 平台过滤列表 top_n: 返回 TOP N 结果，默认10 Returns: 对比分析结果字典 """ try: # 参数验证 platforms = validate_platforms(platforms) top_n = validate_top_n(top_n, default=10) if compare_type not in ["overview", "topic_shift", "platform_activity"]: raise InvalidParameterError( f"不支持的对比类型: {compare_type}", suggestion="支持的类型: overview, topic_shift, platform_activity" ) # 解析时间段 date_range1 = self._parse_period(period1) date_range2 = self._parse_period(period2) if not date_range1 or not date_range2: raise InvalidParameterError( "无效的时间段格式", suggestion="使用 {'start': 'YYYY-MM-DD', 'end': 'YYYY-MM-DD'} 或预设值如 'last_week'" ) # 收集两个时期的数据 data1 = self._collect_period_data(date_range1, platforms, topic) data2 = self._collect_period_data(date_range2, platforms, topic) # 根据对比类型执行不同的分析 if compare_type == "overview": analysis_result = self._compare_overview(data1, data2, date_range1, date_range2, top_n) elif compare_type == "topic_shift": analysis_result = self._compare_topic_shift(data1, data2, date_range1, date_range2, top_n) else: # platform_activity analysis_result = self._compare_platform_activity(data1, data2, date_range1, date_range2) result = { "success": True, "summary": { "description": f"时期对比分析（{compare_type}）", "compare_type": compare_type, "periods": { "period1": { "start": date_range1[0].strftime("%Y-%m-%d"), "end": date_range1[1].strftime("%Y-%m-%d") }, "period2": { "start": date_range2[0].strftime("%Y-%m-%d"), "end": date_range2[1].strftime("%Y-%m-%d") } } }, "data": analysis_result } if topic: result["summary"]["topic_filter"] = topic return result except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return {"success": False, "error": {"code": "INTERNAL_ERROR", "message": str(e)}} def _parse_period(self, period: Union[Dict[str, str], str]) -> Optional[tuple]: """解析时间段为日期范围元组""" today = datetime.now() if isinstance(period, str): if period == "today": return (today, today) elif period == "yesterday": yesterday = today - timedelta(days=1) return (yesterday, yesterday) elif period == "last_week": return (today - timedelta(days=7), today - timedelta(days=1)) elif period == "this_week": # 本周一到今天 days_since_monday = today.weekday() monday = today - timedelta(days=days_since_monday) return (monday, today) elif period == "last_month": return (today - timedelta(days=30), today - timedelta(days=1)) elif period == "this_month": first_of_month = today.replace(day=1) return (first_of_month, today) else: return None elif isinstance(period, dict): try: start = datetime.strptime(period["start"], "%Y-%m-%d") end = datetime.strptime(period["end"], "%Y-%m-%d") return (start, end) except (KeyError, ValueError): return None return None def _collect_period_data( self, date_range: tuple, platforms: Optional[List[str]], topic: Optional[str] ) -> Dict: """收集指定时期的新闻数据""" start_date, end_date = date_range all_news = [] all_keywords = Counter() platform_stats = Counter() current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date, platform_ids=platforms ) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 如果指定了话题，过滤不相关的新闻 if topic and topic.lower() not in title.lower(): continue news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "ranks": info.get("ranks", []), "rank": info["ranks"][0] if info["ranks"] else 999 } news_item["weight"] = calculate_news_weight(news_item) all_news.append(news_item) # 统计平台 platform_stats[platform_name] += 1 # 提取关键词 keywords = self._extract_keywords(title) all_keywords.update(keywords) except DataNotFoundError: pass current_date += timedelta(days=1) return { "news": all_news, "news_count": len(all_news), "keywords": all_keywords, "platform_stats": platform_stats, "date_range": date_range } def _compare_overview( self, data1: Dict, data2: Dict, range1: tuple, range2: tuple, top_n: int ) -> Dict: """总体概览对比""" # 计算变化 count_change = data2["news_count"] - data1["news_count"] count_change_pct = (count_change / data1["news_count"] * 100) if data1["news_count"] > 0 else 0 # TOP 关键词对比 top_kw1 = [kw for kw, _ in data1["keywords"].most_common(top_n)] top_kw2 = [kw for kw, _ in data2["keywords"].most_common(top_n)] new_keywords = [kw for kw in top_kw2 if kw not in top_kw1] disappeared_keywords = [kw for kw in top_kw1 if kw not in top_kw2] persistent_keywords = [kw for kw in top_kw1 if kw in top_kw2] # TOP 新闻对比 top_news1 = sorted(data1["news"], key=lambda x: x.get("weight", 0), reverse=True)[:top_n] top_news2 = sorted(data2["news"], key=lambda x: x.get("weight", 0), reverse=True)[:top_n] return { "overview": { "period1_count": data1["news_count"], "period2_count": data2["news_count"], "count_change": count_change, "count_change_percent": f"{count_change_pct:+.1f}%" }, "keyword_analysis": { "new_keywords": new_keywords[:5], "disappeared_keywords": disappeared_keywords[:5], "persistent_keywords": persistent_keywords[:5] }, "top_news": { "period1": [{"title": n["title"], "platform": n["platform_name"]} for n in top_news1], "period2": [{"title": n["title"], "platform": n["platform_name"]} for n in top_news2] } } def _compare_topic_shift( self, data1: Dict, data2: Dict, range1: tuple, range2: tuple, top_n: int ) -> Dict: """话题变化分析""" kw1 = data1["keywords"] kw2 = data2["keywords"] # 计算热度变化 all_keywords = set(kw1.keys()) | set(kw2.keys()) keyword_changes = [] for kw in all_keywords: count1 = kw1.get(kw, 0) count2 = kw2.get(kw, 0) change = count2 - count1 if count1 > 0: change_pct = (change / count1) * 100 elif count2 > 0: change_pct = 100 # 新出现 else: change_pct = 0 keyword_changes.append({ "keyword": kw, "period1_count": count1, "period2_count": count2, "change": change, "change_percent": round(change_pct, 1) }) # 按变化幅度排序 rising = sorted([k for k in keyword_changes if k["change"] > 0], key=lambda x: x["change"], reverse=True)[:top_n] falling = sorted([k for k in keyword_changes if k["change"] < 0], key=lambda x: x["change"])[:top_n] new_topics = [k for k in keyword_changes if k["period1_count"] == 0 and k["period2_count"] > 0][:top_n] return { "rising_topics": rising, "falling_topics": falling, "new_topics": new_topics, "total_keywords": { "period1": len(kw1), "period2": len(kw2) } } def _compare_platform_activity( self, data1: Dict, data2: Dict, range1: tuple, range2: tuple ) -> Dict: """平台活跃度对比""" ps1 = data1["platform_stats"] ps2 = data2["platform_stats"] all_platforms = set(ps1.keys()) | set(ps2.keys()) platform_changes = [] for platform in all_platforms: count1 = ps1.get(platform, 0) count2 = ps2.get(platform, 0) change = count2 - count1 if count1 > 0: change_pct = (change / count1) * 100 elif count2 > 0: change_pct = 100 else: change_pct = 0 platform_changes.append({ "platform": platform, "period1_count": count1, "period2_count": count2, "change": change, "change_percent": round(change_pct, 1) }) # 按变化排序 platform_changes.sort(key=lambda x: x["change"], reverse=True) return { "platform_comparison": platform_changes, "most_active_growth": platform_changes[0] if platform_changes else None, "least_active_growth": platform_changes[-1] if platform_changes else None, "total_activity": { "period1": sum(ps1.values()), "period2": sum(ps2.values()) } } ================================================ FILE: mcp_server/tools/article_reader.py ================================================ """ 文章内容读取工具 通过 Jina AI Reader API 将 URL 转换为 LLM 友好的 Markdown 格式。 支持单篇和批量读取，内置速率限制和并发控制。 """ import time from typing import Dict, List import requests from ..utils.errors import MCPError, InvalidParameterError # Jina Reader 配置 JINA_READER_BASE = "https://r.jina.ai" DEFAULT_TIMEOUT = 30 # 秒 MAX_BATCH_SIZE = 5 # 单次批量最大篇数 BATCH_INTERVAL = 5.0 # 批量请求间隔（秒） class ArticleReaderTools: """文章内容读取工具类""" def __init__(self, project_root: str = None, jina_api_key: str = None): """ 初始化文章读取工具 Args: project_root: 项目根目录 jina_api_key: Jina API Key（可选，有 Key 可提升速率限制） """ self.project_root = project_root self.jina_api_key = jina_api_key self._last_request_time = 0.0 def _build_headers(self) -> Dict[str, str]: """构建请求头""" headers = { "Accept": "text/markdown", "X-Return-Format": "markdown", "X-No-Cache": "true", } if self.jina_api_key: headers["Authorization"] = f"Bearer {self.jina_api_key}" return headers def _throttle(self): """速率控制：确保请求间隔 5 秒""" now = time.time() elapsed = now - self._last_request_time if elapsed < BATCH_INTERVAL: time.sleep(BATCH_INTERVAL - elapsed) self._last_request_time = time.time() def read_article( self, url: str, timeout: int = DEFAULT_TIMEOUT ) -> Dict: """ 读取单篇文章内容（Markdown 格式） Args: url: 文章链接 timeout: 请求超时时间（秒），默认 30 Returns: 文章内容字典 """ try: if not url or not url.startswith(("http://", "https://")): raise InvalidParameterError( f"无效的 URL: {url}", suggestion="URL 必须以 http:// 或 https:// 开头" ) self._throttle() response = requests.get( f"{JINA_READER_BASE}/{url}", headers=self._build_headers(), timeout=timeout ) if response.status_code == 200: return { "success": True, "data": { "url": url, "content": response.text, "format": "markdown", "content_length": len(response.text) } } elif response.status_code == 429: return { "success": False, "error": { "code": "RATE_LIMITED", "message": "Jina Reader 速率限制，请稍后重试", "suggestion": "免费限制: 100 RPM / 2 并发，可配置 API Key 提升限额" } } else: return { "success": False, "error": { "code": "FETCH_FAILED", "message": f"HTTP {response.status_code}: {response.reason}", "url": url } } except requests.Timeout: return { "success": False, "error": { "code": "TIMEOUT", "message": f"请求超时（{timeout}秒）", "url": url, "suggestion": "可尝试增加 timeout 参数" } } except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return { "success": False, "error": { "code": "REQUEST_ERROR", "message": str(e), "url": url } } def read_articles_batch( self, urls: List[str], timeout: int = DEFAULT_TIMEOUT ) -> Dict: """ 批量读取多篇文章内容（最多 5 篇，间隔 5 秒） Args: urls: 文章链接列表 timeout: 每篇的请求超时时间（秒） Returns: 批量读取结果 """ try: if not urls: raise InvalidParameterError( "URL 列表不能为空", suggestion="请提供至少一个 URL" ) # 限制最多 5 篇 actual_urls = urls[:MAX_BATCH_SIZE] skipped = len(urls) - len(actual_urls) results = [] succeeded = 0 failed = 0 for i, url in enumerate(actual_urls): result = self.read_article(url=url, timeout=timeout) results.append({ "index": i + 1, "url": url, "success": result["success"], "data": result.get("data"), "error": result.get("error") }) if result["success"]: succeeded += 1 else: failed += 1 return { "success": True, "summary": { "description": "批量文章读取结果", "requested": len(urls), "processed": len(actual_urls), "succeeded": succeeded, "failed": failed, "skipped": skipped, "interval_seconds": BATCH_INTERVAL, }, "articles": results, "note": f"已跳过 {skipped} 篇（单次上限 {MAX_BATCH_SIZE} 篇）" if skipped > 0 else None } except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return { "success": False, "error": { "code": "BATCH_ERROR", "message": str(e) } } ================================================ FILE: mcp_server/tools/config_mgmt.py ================================================ """ 配置管理工具 实现配置查询和管理功能。 """ from typing import Dict, Optional, Any, TypedDict from ..services.data_service import DataService from ..utils.validators import validate_config_section from ..utils.errors import MCPError class ErrorInfo(TypedDict, total=False): """错误信息结构""" code: str message: str suggestion: str class ConfigResult(TypedDict): """配置查询结果 - success 字段必需，其他字段可选""" success: bool config: Optional[Dict[str, Any]] section: Optional[str] error: Optional[ErrorInfo] class ConfigManagementTools: """配置管理工具类""" def __init__(self, project_root: str = None): """ 初始化配置管理工具 Args: project_root: 项目根目录 """ self.data_service = DataService(project_root) def get_current_config(self, section: Optional[str] = None) -> ConfigResult: """ 获取当前系统配置 Args: section: 配置节 - all/crawler/push/keywords/weights，默认all Returns: 配置字典 Example: >>> tools = ConfigManagementTools() >>> result = tools.get_current_config(section="crawler") >>> print(result['crawler']['platforms']) """ try: # 参数验证 section = validate_config_section(section) # 获取配置 config = self.data_service.get_current_config(section=section) return ConfigResult( success=True, config=config, section=section, error=None ) except MCPError as e: return ConfigResult( success=False, config=None, section=None, error=e.to_dict() ) except Exception as e: return ConfigResult( success=False, config=None, section=None, error={"code": "INTERNAL_ERROR", "message": str(e), "suggestion": "请查看服务日志获取详细信息"} ) ================================================ FILE: mcp_server/tools/data_query.py ================================================ """ 数据查询工具 实现P0核心的数据查询工具。 """ from typing import Dict, List, Optional, Union from ..services.data_service import DataService from ..utils.validators import ( validate_platforms, validate_limit, validate_keyword, validate_date_range, validate_top_n, validate_mode, validate_date_query, normalize_date_range ) from ..utils.errors import MCPError class DataQueryTools: """数据查询工具类""" def __init__(self, project_root: str = None): """ 初始化数据查询工具 Args: project_root: 项目根目录 """ self.data_service = DataService(project_root) def get_latest_news( self, platforms: Optional[List[str]] = None, limit: Optional[int] = None, include_url: bool = False ) -> Dict: """ 获取最新一批爬取的新闻数据 Args: platforms: 平台ID列表，如 ['zhihu', 'weibo'] limit: 返回条数限制，默认20 include_url: 是否包含URL链接，默认False（节省token） Returns: 新闻列表字典 Example: >>> tools = DataQueryTools() >>> result = tools.get_latest_news(platforms=['zhihu'], limit=10) >>> print(result['total']) 10 """ try: # 参数验证 platforms = validate_platforms(platforms) limit = validate_limit(limit, default=50) # 获取数据 news_list = self.data_service.get_latest_news( platforms=platforms, limit=limit, include_url=include_url ) return { "success": True, "summary": { "description": "最新一批爬取的新闻数据", "total": len(news_list), "returned": len(news_list), "platforms": platforms or "全部平台" }, "data": news_list } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def search_news_by_keyword( self, keyword: str, date_range: Optional[Union[Dict, str]] = None, platforms: Optional[List[str]] = None, limit: Optional[int] = None ) -> Dict: """ 按关键词搜索历史新闻 Args: keyword: 搜索关键词（必需） date_range: 日期范围，格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} platforms: 平台过滤列表 limit: 返回条数限制（可选，默认返回所有） Returns: 搜索结果字典 Example (假设今天是 2025-11-17): >>> tools = DataQueryTools() >>> result = tools.search_news_by_keyword( ... keyword="人工智能", ... date_range={"start": "2025-11-08", "end": "2025-11-17"}, ... limit=50 ... ) >>> print(result['total']) """ try: # 参数验证 keyword = validate_keyword(keyword) date_range_tuple = validate_date_range(date_range) platforms = validate_platforms(platforms) if limit is not None: limit = validate_limit(limit, default=100) # 搜索数据 search_result = self.data_service.search_news_by_keyword( keyword=keyword, date_range=date_range_tuple, platforms=platforms, limit=limit ) return { **search_result, "success": True } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_trending_topics( self, top_n: Optional[int] = None, mode: Optional[str] = None, extract_mode: Optional[str] = None ) -> Dict: """ 获取热点话题统计 Args: top_n: 返回TOP N话题，默认10 mode: 时间模式 - "daily": 当日累计数据统计 - "current": 最新一批数据统计（默认） extract_mode: 提取模式 - "keywords": 统计预设关注词（基于 config/frequency_words.txt，默认） - "auto_extract": 自动从新闻标题提取高频词 Returns: 话题频率统计字典 Example: >>> tools = DataQueryTools() >>> # 使用预设关注词 >>> result = tools.get_trending_topics(top_n=5, mode="current") >>> # 自动提取高频词 >>> result = tools.get_trending_topics(top_n=10, extract_mode="auto_extract") """ try: # 参数验证 top_n = validate_top_n(top_n, default=10) valid_modes = ["daily", "current"] mode = validate_mode(mode, valid_modes, default="current") # 验证 extract_mode if extract_mode is None: extract_mode = "keywords" elif extract_mode not in ["keywords", "auto_extract"]: return { "success": False, "error": { "code": "INVALID_PARAMETER", "message": f"不支持的提取模式: {extract_mode}", "suggestion": "支持的模式: keywords, auto_extract" } } # 获取趋势话题 trending_result = self.data_service.get_trending_topics( top_n=top_n, mode=mode, extract_mode=extract_mode ) return { **trending_result, "success": True } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_news_by_date( self, date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, limit: Optional[int] = None, include_url: bool = False ) -> Dict: """ 按日期查询新闻，支持自然语言日期 Args: date_range: 日期范围（可选，默认"今天"），支持： - 范围对象：{"start": "2025-01-01", "end": "2025-01-07"} - 相对日期：今天、昨天、前天、3天前 - 单日字符串：2025-10-10 platforms: 平台ID列表，如 ['zhihu', 'weibo'] limit: 返回条数限制，默认50 include_url: 是否包含URL链接，默认False（节省token） Returns: 新闻列表字典 Example: >>> tools = DataQueryTools() >>> # 不指定日期，默认查询今天 >>> result = tools.get_news_by_date(platforms=['zhihu'], limit=20) >>> # 指定日期 >>> result = tools.get_news_by_date( ... date_range="昨天", ... platforms=['zhihu'], ... limit=20 ... ) >>> print(result['total']) 20 """ try: # 参数验证 - 默认今天 if date_range is None: date_range = "今天" # 规范化 date_range（处理 JSON 字符串序列化问题） date_range = normalize_date_range(date_range) # 处理 date_range：支持字符串或对象 if isinstance(date_range, dict): # 范围对象，取 start 日期 date_str = date_range.get('start', '今天') else: date_str = date_range target_date = validate_date_query(date_str) platforms = validate_platforms(platforms) limit = validate_limit(limit, default=50) # 获取数据 news_list = self.data_service.get_news_by_date( target_date=target_date, platforms=platforms, limit=limit, include_url=include_url ) return { "success": True, "summary": { "description": f"按日期查询的新闻（{target_date.strftime('%Y-%m-%d')}）", "total": len(news_list), "returned": len(news_list), "date": target_date.strftime("%Y-%m-%d"), "date_range": date_range, "platforms": platforms or "全部平台" }, "data": news_list } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } # ======================================== # RSS 数据查询方法 # ======================================== def get_latest_rss( self, feeds: Optional[List[str]] = None, days: int = 1, limit: Optional[int] = None, include_summary: bool = False ) -> Dict: """ 获取最新的 RSS 数据（支持多日查询） Args: feeds: RSS 源 ID 列表，如 ['hacker-news', '36kr'] days: 获取最近 N 天的数据，默认 1（仅今天），最大 30 天 limit: 返回条数限制，默认50 include_summary: 是否包含摘要，默认False（节省token） Returns: RSS 条目列表字典 """ try: limit = validate_limit(limit, default=50) rss_list = self.data_service.get_latest_rss( feeds=feeds, days=days, limit=limit, include_summary=include_summary ) return { "success": True, "summary": { "description": f"最近 {days} 天的 RSS 订阅数据" if days > 1 else "最新的 RSS 订阅数据", "total": len(rss_list), "returned": len(rss_list), "days": days, "feeds": feeds or "全部订阅源" }, "data": rss_list } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def search_rss( self, keyword: str, feeds: Optional[List[str]] = None, days: int = 7, limit: Optional[int] = None, include_summary: bool = False ) -> Dict: """ 搜索 RSS 数据 Args: keyword: 搜索关键词 feeds: RSS 源 ID 列表 days: 搜索最近 N 天的数据，默认 7 天 limit: 返回条数限制，默认50 include_summary: 是否包含摘要 Returns: 匹配的 RSS 条目列表 """ try: keyword = validate_keyword(keyword) limit = validate_limit(limit, default=50) if days < 1 or days > 30: days = 7 rss_list = self.data_service.search_rss( keyword=keyword, feeds=feeds, days=days, limit=limit, include_summary=include_summary ) return { "success": True, "summary": { "description": f"RSS 搜索结果（关键词: {keyword}）", "total": len(rss_list), "returned": len(rss_list), "keyword": keyword, "feeds": feeds or "全部订阅源", "days": days }, "data": rss_list } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_rss_feeds_status(self) -> Dict: """ 获取 RSS 源状态 Returns: RSS 源状态信息 """ try: status = self.data_service.get_rss_feeds_status() return { **status, "success": True } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } ================================================ FILE: mcp_server/tools/notification.py ================================================ # coding=utf-8 """ 通知推送工具 支持向已配置的通知渠道发送消息，自动检测 config.yaml 和 .env 中的渠道配置。 接受 markdown 格式内容，内部按各渠道要求自动转换格式后发送。 """ import json import os import re import smtplib import time from datetime import datetime from email.header import Header from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.utils import formataddr, formatdate, make_msgid from pathlib import Path from typing import Any, Dict, List, Optional from urllib.parse import urlparse import requests import yaml from trendradar.core.loader import _load_webhook_config, _load_notification_config from trendradar.notification.batch import ( truncate_to_bytes, get_batch_header, get_max_batch_header_size, add_batch_headers, ) from trendradar.notification.formatters import strip_markdown from trendradar.notification.senders import SMTP_CONFIGS from ..utils.errors import MCPError, InvalidParameterError # ==================== 渠道启用判断规则 ==================== # 每个渠道需要哪些配置项都非空才算"已配置" # 注意：NTFY_SERVER_URL 在 loader 中有默认值 "https://ntfy.sh"，不作为判断依据 _CHANNEL_REQUIREMENTS = { "feishu": ["FEISHU_WEBHOOK_URL"], "dingtalk": ["DINGTALK_WEBHOOK_URL"], "wework": ["WEWORK_WEBHOOK_URL"], "telegram": ["TELEGRAM_BOT_TOKEN", "TELEGRAM_CHAT_ID"], "email": ["EMAIL_FROM", "EMAIL_PASSWORD", "EMAIL_TO"], "ntfy": ["NTFY_TOPIC"], "bark": ["BARK_URL"], "slack": ["SLACK_WEBHOOK_URL"], "generic_webhook": ["GENERIC_WEBHOOK_URL"], } # 渠道显示名称 _CHANNEL_NAMES = { "feishu": "飞书", "dingtalk": "钉钉", "wework": "企业微信", "telegram": "Telegram", "email": "邮件", "ntfy": "ntfy", "bark": "Bark", "slack": "Slack", "generic_webhook": "通用 Webhook", } # ==================== 批次处理配置 ==================== # 各渠道最大批次字节数的默认值 # 运行时从 config.yaml → advanced.batch_size 读取覆盖 _CHANNEL_BATCH_SIZES_DEFAULT = { "feishu": 30000, # config.yaml: advanced.batch_size.feishu "dingtalk": 20000, # config.yaml: advanced.batch_size.dingtalk "wework": 4000, # config.yaml: advanced.batch_size.default "telegram": 4000, # config.yaml: advanced.batch_size.default "email": 0, # 邮件无字节限制，不分批 "ntfy": 3800, # 严格 4KB 限制（ntfy 代码默认值） "bark": 4000, # config.yaml: advanced.batch_size.bark "slack": 4000, # config.yaml: advanced.batch_size.slack "generic_webhook": 4000, } # 显示最新消息在前的渠道，批次需反序发送 _REVERSE_BATCH_CHANNELS = {"ntfy", "bark"} # 批次发送间隔默认值（秒），运行时从 config.yaml → advanced.batch_send_interval 读取 _BATCH_INTERVAL_DEFAULT = 3.0 # ==================== 批次处理 ==================== # truncate_to_bytes, get_batch_header, get_max_batch_header_size, # add_batch_headers 复用自 trendradar.notification.batch def _split_text_into_batches(text: str, max_bytes: int) -> List[str]: """将文本按字节限制分批，优先在段落边界（双换行）切割 分割策略（参考 trendradar splitter.py 的原子性保证）： 1. 优先按段落（双换行 \\n\\n）拆分 2. 段落仍超限时，按单行（\\n）拆分 3. 单行仍超限时，用 _truncate_to_bytes 安全截断 Args: text: 已转换为目标渠道格式的文本 max_bytes: 单批最大字节数（已扣除批次头部预留） Returns: 分批后的文本列表 """ if max_bytes <= 0 or len(text.encode("utf-8")) <= max_bytes: return [text] # 按段落分割 paragraphs = text.split("\n\n") batches = [] current = "" for para in paragraphs: candidate = f"{current}\n\n{para}" if current else para if len(candidate.encode("utf-8")) <= max_bytes: current = candidate else: # 当前段落放不下，先保存已有内容 if current: batches.append(current) current = "" # 检查单个段落是否超限 if len(para.encode("utf-8")) <= max_bytes: current = para else: # 段落本身超限，按行拆分 lines = para.split("\n") for line in lines: candidate = f"{current}\n{line}" if current else line if len(candidate.encode("utf-8")) <= max_bytes: current = candidate else: if current: batches.append(current) current = "" # 单行超限，循环截断直到处理完 if len(line.encode("utf-8")) > max_bytes: remaining = line while remaining: chunk = truncate_to_bytes(remaining, max_bytes) if not chunk: break batches.append(chunk) # 移除已截断的部分 remaining = remaining[len(chunk):] else: current = line if current: batches.append(current) return batches if batches else [text] def _format_for_channel(message: str, channel_id: str) -> str: """将通用 Markdown 适配并转换为目标渠道格式 统一入口：先适配（剥离不支持的语法），再转换（Markdown→HTML/mrkdwn 等）。 返回的文本可以直接用于字节分割和发送。 Args: message: 原始 Markdown 格式文本 channel_id: 目标渠道 ID Returns: 目标渠道格式的文本 """ if channel_id == "feishu": return _adapt_markdown_for_feishu(message) elif channel_id == "dingtalk": return _adapt_markdown_for_dingtalk(message) elif channel_id == "wework": return _adapt_markdown_for_wework(message) elif channel_id == "telegram": return _markdown_to_telegram_html(message) elif channel_id == "ntfy": return _adapt_markdown_for_ntfy(message) elif channel_id == "bark": return _adapt_markdown_for_bark(message) elif channel_id == "slack": return _convert_markdown_to_slack(message) else: # email, generic_webhook: 保持原始 Markdown return message def _prepare_batches(message: str, channel_id: str, batch_sizes: Dict = None) -> List[str]: """完整的分批管线：格式适配 → 字节分割 → 添加批次头部 Args: message: 原始 Markdown 格式文本 channel_id: 目标渠道 ID batch_sizes: 各渠道批次大小字典（来自 config.yaml），None 使用默认值 Returns: 准备好的批次列表（已添加头部，已处理反序） """ sizes = batch_sizes or _CHANNEL_BATCH_SIZES_DEFAULT max_bytes = sizes.get(channel_id, sizes.get("default", 4000)) if max_bytes <= 0: # 无字节限制（如 email），返回原始文本 return [message] formatted = _format_for_channel(message, channel_id) # 预留批次头部空间后分割 header_reserve = get_max_batch_header_size(channel_id) batches = _split_text_into_batches(formatted, max_bytes - header_reserve) # 添加批次头部（单批时不添加） batches = add_batch_headers(batches, channel_id, max_bytes) # ntfy/Bark 反序发送（客户端显示最新在前） if channel_id in _REVERSE_BATCH_CHANNELS and len(batches) > 1: batches = list(reversed(batches)) return batches CHANNEL_FORMAT_GUIDES = { "feishu": { "name": "飞书", "format": "Markdown（卡片消息）", "max_length": "约 29000 字节", "supported": [ "**粗体**", "[链接文本](URL)", "彩色文本", "---（分割线）", "换行分隔段落", ], "unsupported": [ "# 标题语法（不渲染为标题样式）", "> 引用块", "表格 / 图片嵌入", ], "prompt": ( "飞书卡片 Markdown 格式化策略：\n" "1. 用 **粗体** 作小标题和重点词\n" "2. 用 红色 标记紧急/重要内容\n" "3. 用 灰色 标记辅助信息（时间、来源）\n" "4. 用 橙色 标记警告\n" "5. 用 绿色 标记正面/成功信息\n" "6. 用 [文本](URL) 添加可点击链接\n" "7. 用 --- 分割不同主题区域\n" "8. 不要用 # 标题语法（卡片内不渲染）\n" "9. 不要用 > 引用语法\n" "10. 用换行 + 粗体模拟层级结构" ), }, "dingtalk": { "name": "钉钉", "format": "Markdown", "max_length": "约 20000 字节", "supported": [ "### 三级标题 / #### 四级标题", "**粗体**", "[链接文本](URL)", "> 引用块", "---（分割线）", "- 无序列表 / 1. 有序列表", ], "unsupported": [ "# 一级标题 / ## 二级标题（可能不渲染）", " 彩色文本", "~~删除线~~", "表格 / 图片嵌入", ], "prompt": ( "钉钉 Markdown 格式化策略：\n" "1. 用 ### 或 #### 作章节标题（不用 # 和 ##）\n" "2. 用 **粗体** 突出关键词和数据\n" "3. 用 > 引用块展示备注或补充说明\n" "4. 用 --- 分割不同主题区域\n" "5. 用 [文本](URL) 添加可点击链接\n" "6. 用有序列表（1. 2. 3.）组织要点\n" "7. 不要用 颜色标签（钉钉不支持）\n" "8. 不要用删除线语法\n" "9. 标题和正文之间加空行提升可读性" ), }, "wework": { "name": "企业微信", "format": "Markdown（群机器人）/ 纯文本（个人微信）", "max_length": "约 4000 字节", "supported": [ "**粗体**", "[链接文本](URL)", "> 引用块（仅首行生效）", ], "unsupported": [ "# 标题语法", "---（水平分割线）", " 彩色文本", "~~删除线~~", "表格 / 图片嵌入 / 有序列表", ], "prompt": ( "企业微信 Markdown 格式化策略：\n" "1. 用 **粗体** 作小标题和重点词\n" "2. 用 [文本](URL) 添加可点击链接\n" "3. 用 > 引用块展示备注（仅首行生效）\n" "4. 内容要简洁，受 4KB 限制\n" "5. 不要用 # 标题语法（不渲染）\n" "6. 不要用 ---（不渲染），用多个换行分隔区域\n" "7. 不要用 颜色标签\n" "8. 不要用删除线和有序列表\n" "9. 用换行 + 粗体模拟层级结构\n" "10. 个人微信模式下所有格式被剥离为纯文本" ), }, "telegram": { "name": "Telegram", "format": "HTML（自动从 Markdown 转换）", "max_length": "约 4096 字符", "supported": [ "粗体（从 **粗体** 转换）", "斜体（从 *斜体* 转换）", "删除线（从 ~~删除线~~ 转换）", "行内代码（从 `代码` 转换）", "链接（从 [文本](URL) 转换）", "
引用块
（从 > 引用 转换）", ], "unsupported": [ "# 标题语法（自动剥离 # 前缀）", "---（分割线，自动剥离）", " 彩色文本（自动剥离）", "表格 / 图片嵌入", ], "prompt": ( "Telegram HTML 格式化策略（输入仍为 Markdown，自动转换为 HTML）：\n" "1. 用 **粗体** 突出关键词（转为 ）\n" "2. 用 *斜体* 标记辅助信息（转为 ）\n" "3. 用 `代码` 标记数据值/时间（转为 ）\n" "4. 用 [文本](URL) 添加链接（转为 ）\n" "5. 用 > 开头的行作引用块（转为
）\n" "6. 不要用 # 标题（Telegram 无标题样式，仅剥离 #）\n" "7. 不要用 --- 分割线（被剥离），用空行分隔\n" "8. 不要用 颜色标签（被剥离）\n" "9. 内容受 4096 字符限制，保持简洁\n" "10. 链接默认禁用预览，适合信息密集型消息" ), }, "email": { "name": "邮件", "format": "HTML（完整网页，从 Markdown 转换）", "max_length": "无硬限制", "supported": [ "# / ## / ### 标题（转为

/

/

）", "**粗体** / *斜体* / ~~删除线~~", "[链接文本](URL)", "`行内代码`", "---（水平分割线）", ], "unsupported": [ " 彩色文本（转义显示）", "复杂表格", ], "prompt": ( "邮件 HTML 格式化策略（输入为 Markdown，自动转换为带样式 HTML）：\n" "1. 用 # / ## / ### 创建清晰的标题层级\n" "2. 用 **粗体** 和 *斜体* 增强可读性\n" "3. 用 [文本](URL) 添加链接（蓝色可点击）\n" "4. 用 --- 分割不同章节\n" "5. 用 `代码` 标记技术术语或数据\n" "6. 可以写较长内容，邮件无严格长度限制\n" "7. 邮件主题自动追加日期时间\n" "8. 自动附带纯文本备用版本" ), }, "ntfy": { "name": "ntfy", "format": "Markdown（原生支持）", "max_length": "约 3800 字节（单条 4KB 限制）", "supported": [ "**粗体** / *斜体*", "[链接文本](URL)", "> 引用块", "`行内代码`", "- 列表", ], "unsupported": [ "# 标题语法（渲染取决于客户端）", " 彩色文本", "---（渲染取决于客户端）", "表格", ], "prompt": ( "ntfy Markdown 格式化策略：\n" "1. 用 **粗体** 突出关键词\n" "2. 用 [文本](URL) 添加可点击链接\n" "3. 用 > 引用块展示备注\n" "4. 用 `代码` 标记数据值\n" "5. 内容要精炼，受 4KB 限制\n" "6. 不要用 颜色标签（无效）\n" "7. 不要依赖 # 标题和 --- 分割线\n" "8. 用空行和粗体组织信息层级" ), }, "bark": { "name": "Bark", "format": "Markdown（iOS 推送）", "max_length": "约 3600 字节（APNs 4KB 限制）", "supported": [ "**粗体**", "[链接文本](URL)", "基础文本格式", ], "unsupported": [ "# 标题语法", " 彩色文本", "---（分割线）", "> 引用块", "复杂嵌套格式", ], "prompt": ( "Bark 格式化策略（iOS 推送通知）：\n" "1. 内容要极度精简，移动端阅读场景\n" "2. 用 **粗体** 标记核心信息\n" "3. 用 [文本](URL) 添加链接\n" "4. 不要用标题/颜色/引用等复杂格式\n" "5. 受 APNs 4KB 限制，控制内容长度\n" "6. 层级结构靠缩进和换行实现\n" "7. 适合简短通知和摘要，不适合长文" ), }, "slack": { "name": "Slack", "format": "mrkdwn（Slack 专有格式，自动从 Markdown 转换）", "max_length": "约 4000 字节", "supported": [ "*粗体*（从 **粗体** 转换）", "_斜体_", "~删除线~（从 ~~删除线~~ 转换）", "（从 [文本](URL) 转换）", "`行内代码`", "```代码块```", "> 引用块", ], "unsupported": [ "# 标题语法（剥离为粗体）", " 彩色文本", "--- 分割线（渲染不稳定）", "表格", ], "prompt": ( "Slack mrkdwn 格式化策略（输入为 Markdown，自动转换为 mrkdwn）：\n" "1. 用 **粗体** 突出关键词（转为 *粗体*）\n" "2. 用 ~~删除线~~ 标记过时信息（转为 ~删除线~）\n" "3. 用 [文本](URL) 添加链接（转为 ）\n" "4. 用 > 引用块展示备注\n" "5. 用 `代码` 标记数据值\n" "6. 不要用 # 标题（Slack 无标题样式）\n" "7. 不要用 颜色标签\n" "8. 用空行和粗体组织信息层级" ), }, "generic_webhook": { "name": "通用 Webhook", "format": "Markdown（或自定义模板）", "max_length": "约 4000 字节", "supported": ["标准 Markdown 语法"], "unsupported": ["取决于接收端"], "prompt": ( "通用 Webhook 格式化策略：\n" "1. 使用标准 Markdown 格式\n" "2. 避免使用特殊平台专有语法\n" "3. 如配置了自定义模板，内容会填充到 {content} 占位符" ), }, } # ==================== 渠道 Markdown 适配 ==================== def _adapt_markdown_for_feishu(text: str) -> str: """将通用 Markdown 适配为飞书卡片 Markdown 格式 飞书卡片支持：**粗体**, [链接](url), , --- 不支持：# 标题, > 引用块 """ # 将 # 标题转换为粗体（飞书卡片不渲染标题语法） text = re.sub(r'^#{1,6}\s+(.+)$', r'**\1**', text, flags=re.MULTILINE) # 去除引用语法前缀（飞书不支持） text = re.sub(r'^>\s*', '', text, flags=re.MULTILINE) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() def _adapt_markdown_for_dingtalk(text: str) -> str: """将通用 Markdown 适配为钉钉 Markdown 格式 钉钉支持：### #### 标题, **粗体**, [链接](url), > 引用, --- 不支持：# ## 标题, 彩色文本, ~~删除线~~ """ # 去除 标签（钉钉不支持，保留内容） text = re.sub(r']*>(.+?)', r'\1', text) # 将 # 和 ## 标题降级为 ### （钉钉仅支持 ### 和 ####） text = re.sub(r'^##\s+(.+)$', r'### \1', text, flags=re.MULTILINE) text = re.sub(r'^#\s+(.+)$', r'### \1', text, flags=re.MULTILINE) # 去除删除线语法（钉钉不支持） text = re.sub(r'~~(.+?)~~', r'\1', text) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() def _adapt_markdown_for_wework(text: str) -> str: """将通用 Markdown 适配为企业微信 Markdown 格式 企业微信支持：**粗体**, [链接](url), > 引用（有限） 不支持：# 标题, ---, , ~~删除线~~, 有序列表 """ # 去除 标签（保留内容） text = re.sub(r']*>(.+?)', r'\1', text) # 将 # 标题转换为粗体（企业微信不渲染标题语法） text = re.sub(r'^#{1,6}\s+(.+)$', r'**\1**', text, flags=re.MULTILINE) # 将 --- 分割线替换为多个换行（企业微信不渲染水平线） text = re.sub(r'^[\-\*]{3,}\s*$', '\n\n', text, flags=re.MULTILINE) # 去除删除线语法（企业微信不支持） text = re.sub(r'~~(.+?)~~', r'\1', text) # 清理多余空行（保留最多两个） text = re.sub(r'\n{4,}', '\n\n\n', text) return text.strip() def _adapt_markdown_for_ntfy(text: str) -> str: """将通用 Markdown 适配为 ntfy 格式 ntfy 支持：**粗体**, *斜体*, [链接](url), > 引用, `代码` 不可靠：# 标题, ---, """ # 去除 标签（ntfy 不支持） text = re.sub(r']*>(.+?)', r'\1', text) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() def _adapt_markdown_for_bark(text: str) -> str: """将通用 Markdown 适配为 Bark 格式（iOS 推送） Bark 支持：**粗体**, [链接](url), 基础文本 不支持：# 标题, , ---, > 引用, 复杂嵌套 """ # 去除 标签（保留内容） text = re.sub(r']*>(.+?)', r'\1', text) # 将 # 标题转换为粗体 text = re.sub(r'^#{1,6}\s+(.+)$', r'**\1**', text, flags=re.MULTILINE) # 将 --- 替换为换行 text = re.sub(r'^[\-\*]{3,}\s*$', '\n', text, flags=re.MULTILINE) # 去除引用语法 text = re.sub(r'^>\s*', '', text, flags=re.MULTILINE) # 去除删除线语法 text = re.sub(r'~~(.+?)~~', r'\1', text) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() # ==================== 格式转换 ==================== def _markdown_to_telegram_html(text: str) -> str: """ 将 markdown 转换为 Telegram 支持的 HTML 格式 Telegram 支持的标签：, , , , text,
""" # 预处理：去除 标签（Telegram 不支持，保留内容） text = re.sub(r']*>(.+?)', r'\1', text) lines = text.split('\n') result_lines = [] in_blockquote = False for line in lines: # 将标题符号 # ## ### 转换为粗体 header_match = re.match(r'^(#{1,6})\s+(.+)$', line) if header_match: line = f'**{header_match.group(2)}**' # 去除水平分割线 if re.match(r'^[\-\*]{3,}\s*$', line): if in_blockquote: result_lines.append('
') in_blockquote = False line = '' # 处理引用块 > text →
text
quote_match = re.match(r'^>\s*(.*)$', line) if quote_match: if not in_blockquote: result_lines.append('
') in_blockquote = True result_lines.append(quote_match.group(1)) continue elif in_blockquote: result_lines.append('
') in_blockquote = False result_lines.append(line) if in_blockquote: result_lines.append('

') text = '\n'.join(result_lines) # 转义 HTML 实体（在标记替换之前，但在 blockquote 标签之后） # 分段处理：保留已生成的 HTML 标签 parts = re.split(r'()', text) escaped_parts = [] for part in parts: if part in ('
', '
'): escaped_parts.append(part) else: part = part.replace('&', '&') part = part.replace('<', '<') part = part.replace('>', '>') escaped_parts.append(part) text = ''.join(escaped_parts) # 转换链接 [text](url) → text text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'\1', text) # 转换粗体 **text** → text text = re.sub(r'\*\*(.+?)\*\*', r'\1', text) # 转换斜体 *text* → text text = re.sub(r'\*(.+?)\*', r'\1', text) # 转换删除线 ~~text~~ → text text = re.sub(r'~~(.+?)~~', r'\1', text) # 转换行内代码 `code` → code text = re.sub(r'`(.+?)`', r'\1', text) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() def _convert_markdown_to_slack(text: str) -> str: """将 Markdown 转换为 Slack mrkdwn 格式（增强版） Slack mrkdwn 与标准 Markdown 差异： - 粗体: *text* (非 **text**) - 删除线: ~text~ (非 ~~text~~) - 链接: (非 [text](url)) - 不支持标题语法 """ # 去除 标签（保留内容） text = re.sub(r']*>(.+?)', r'\1', text) # 将 # 标题转换为粗体（Slack 无标题样式） text = re.sub(r'^#{1,6}\s+(.+)$', r'**\1**', text, flags=re.MULTILINE) # 去除 --- 分割线（Slack 渲染不稳定） text = re.sub(r'^[\-\*]{3,}\s*$', '', text, flags=re.MULTILINE) # 转换链接格式: [文本](url) → text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'<\2|\1>', text) # 转换删除线: ~~文本~~ → ~文本~ text = re.sub(r'~~(.+?)~~', r'~\1~', text) # 转换粗体: **文本** → *文本*（必须在删除线之后） text = re.sub(r'\*\*([^*]+)\*\*', r'*\1*', text) # 清理多余空行 text = re.sub(r'\n{3,}', '\n\n', text) return text.strip() def _markdown_to_simple_html(text: str) -> str: """ 将 markdown 转换为简单 HTML（用于 Email） """ html = text # 转义 html = html.replace('&', '&') html = html.replace('<', '<') html = html.replace('>', '>') # 链接 html = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'\1', html) # 标题 ### →

html = re.sub(r'^### (.+)$', r'

\1

', html, flags=re.MULTILINE) html = re.sub(r'^## (.+)$', r'

\1

', html, flags=re.MULTILINE) html = re.sub(r'^# (.+)$', r'

\1

', html, flags=re.MULTILINE) # 粗体 html = re.sub(r'\*\*(.+?)\*\*', r'\1', html) # 斜体 html = re.sub(r'\*(.+?)\*', r'\1', html) # 删除线 html = re.sub(r'~~(.+?)~~', r'\1', html) # 行内代码 html = re.sub(r'`(.+?)`', r'\1', html) # 分割线 html = re.sub(r'^[\-\*]{3,}\s*$', '
', html, flags=re.MULTILINE) # 换行 html = html.replace('\n', '
\n') return f""" TrendRadar 通知 {html}""" # ==================== 各渠道发送器 ==================== def _send_feishu(webhook_url: str, content: str, title: str) -> Dict: """飞书发送（纯文本消息，与 trendradar send_to_feishu 一致） 飞书 webhook 使用 msg_type: "text"，所有信息整合到 content.text 中。 """ payload = { "msg_type": "text", "content": { "text": content, }, } try: resp = requests.post(webhook_url, json=payload, timeout=30) data = resp.json() ok = resp.status_code == 200 and (data.get("code") == 0 or data.get("StatusCode") == 0) detail = "" if not ok: detail = data.get("msg") or data.get("StatusMessage", "") return {"success": ok, "detail": detail} except Exception as e: return {"success": False, "detail": str(e)} def _send_dingtalk(webhook_url: str, content: str, title: str) -> Dict: """钉钉发送（接收已适配的 Markdown）""" payload = { "msgtype": "markdown", "markdown": {"title": title, "text": content} } try: resp = requests.post(webhook_url, json=payload, timeout=30) data = resp.json() ok = resp.status_code == 200 and data.get("errcode") == 0 return {"success": ok, "detail": data.get("errmsg", "") if not ok else ""} except Exception as e: return {"success": False, "detail": str(e)} def _send_wework(webhook_url: str, content: str, title: str, msg_type: str = "markdown") -> Dict: """企业微信发送（接收已适配的 Markdown，text 模式自动剥离格式）""" if msg_type == "text": payload = {"msgtype": "text", "text": {"content": strip_markdown(content)}} else: payload = {"msgtype": "markdown", "markdown": {"content": content}} try: resp = requests.post(webhook_url, json=payload, timeout=30) data = resp.json() ok = resp.status_code == 200 and data.get("errcode") == 0 return {"success": ok, "detail": data.get("errmsg", "") if not ok else ""} except Exception as e: return {"success": False, "detail": str(e)} def _send_telegram(bot_token: str, chat_id: str, content: str, title: str) -> Dict: """Telegram 发送（接收已转换的 HTML）""" url = f"https://api.telegram.org/bot{bot_token}/sendMessage" payload = { "chat_id": chat_id, "text": content, "parse_mode": "HTML", "disable_web_page_preview": True, } try: resp = requests.post(url, json=payload, timeout=30) data = resp.json() ok = resp.status_code == 200 and data.get("ok") return {"success": ok, "detail": data.get("description", "") if not ok else ""} except Exception as e: return {"success": False, "detail": str(e)} def _send_email( from_email: str, password: str, to_email: str, message: str, title: str, smtp_server: str = "", smtp_port: str = "" ) -> Dict: """邮件发送（HTML 格式）""" try: domain = from_email.split("@")[-1].lower() html_content = _markdown_to_simple_html(message) # SMTP 配置 if smtp_server and smtp_port: server_host = smtp_server port = int(smtp_port) use_tls = port != 465 elif domain in SMTP_CONFIGS: cfg = SMTP_CONFIGS[domain] server_host = cfg["server"] port = cfg["port"] use_tls = cfg["encryption"] == "TLS" else: server_host = f"smtp.{domain}" port = 587 use_tls = True msg = MIMEMultipart("alternative") msg["From"] = formataddr(("TrendRadar", from_email)) recipients = [addr.strip() for addr in to_email.split(",")] msg["To"] = ", ".join(recipients) now = datetime.now() msg["Subject"] = Header(f"{title} - {now.strftime('%m月%d日 %H:%M')}", "utf-8") msg["MIME-Version"] = "1.0" msg["Date"] = formatdate(localtime=True) msg["Message-ID"] = make_msgid() # 纯文本备选 msg.attach(MIMEText(strip_markdown(message), "plain", "utf-8")) # HTML 主体 msg.attach(MIMEText(html_content, "html", "utf-8")) if use_tls: server = smtplib.SMTP(server_host, port, timeout=30) server.ehlo() server.starttls() server.ehlo() else: server = smtplib.SMTP_SSL(server_host, port, timeout=30) server.ehlo() server.login(from_email, password) server.send_message(msg) server.quit() return {"success": True, "detail": ""} except Exception as e: return {"success": False, "detail": str(e)} def _send_ntfy(server_url: str, topic: str, content: str, title: str, token: str = "") -> Dict: """ntfy 发送（接收已适配的 Markdown，与 trendradar send_to_ntfy 一致） 注意：Title 使用 ASCII 字符避免 HTTP header 编码问题。 支持 429 速率限制重试。 """ base_url = server_url.rstrip("/") if not base_url.startswith(("http://", "https://")): base_url = f"https://{base_url}" url = f"{base_url}/{topic}" headers = { "Content-Type": "text/plain; charset=utf-8", "Markdown": "yes", "Title": "TrendRadar Notification", # ASCII，避免 HTTP header 编码问题 "Priority": "default", "Tags": "news", } if token: headers["Authorization"] = f"Bearer {token}" try: resp = requests.post(url, data=content.encode("utf-8"), headers=headers, timeout=30) if resp.status_code == 200: return {"success": True, "detail": ""} elif resp.status_code == 429: # 速率限制，等待后重试一次（与 trendradar 一致） time.sleep(10) retry_resp = requests.post(url, data=content.encode("utf-8"), headers=headers, timeout=30) ok = retry_resp.status_code == 200 return {"success": ok, "detail": "" if ok else f"retry status={retry_resp.status_code}"} elif resp.status_code == 413: return {"success": False, "detail": f"消息过大被拒绝 ({len(content.encode('utf-8'))} bytes)"} else: return {"success": False, "detail": f"status={resp.status_code}"} except Exception as e: return {"success": False, "detail": str(e)} def _send_bark(bark_url: str, content: str, title: str) -> Dict: """Bark 发送（接收已适配的 Markdown，iOS 推送）""" parsed = urlparse(bark_url) device_key = parsed.path.strip('/').split('/')[0] if parsed.path else None if not device_key: return {"success": False, "detail": f"无法从 URL 提取 device_key: {bark_url}"} api_endpoint = f"{parsed.scheme}://{parsed.netloc}/push" payload = { "title": title, "markdown": content, "device_key": device_key, "sound": "default", "group": "TrendRadar", "action": "none", } try: resp = requests.post(api_endpoint, json=payload, timeout=30) data = resp.json() ok = resp.status_code == 200 and data.get("code") == 200 return {"success": ok, "detail": data.get("message", "") if not ok else ""} except Exception as e: return {"success": False, "detail": str(e)} def _send_slack(webhook_url: str, content: str, title: str) -> Dict: """Slack 发送（接收已转换的 mrkdwn）""" payload = {"text": content} try: resp = requests.post(webhook_url, json=payload, timeout=30) ok = resp.status_code == 200 and resp.text == "ok" return {"success": ok, "detail": "" if ok else resp.text} except Exception as e: return {"success": False, "detail": str(e)} def _send_generic_webhook( webhook_url: str, message: str, title: str, payload_template: str = "" ) -> Dict: """通用 Webhook 发送（Markdown 格式，支持自定义模板）""" try: if payload_template: json_content = json.dumps(message)[1:-1] json_title = json.dumps(title)[1:-1] payload_str = payload_template.replace("{content}", json_content).replace("{title}", json_title) try: payload = json.loads(payload_str) except json.JSONDecodeError: payload = {"title": title, "content": message} else: payload = {"title": title, "content": message} resp = requests.post( webhook_url, headers={"Content-Type": "application/json"}, json=payload, timeout=30, ) ok = 200 <= resp.status_code < 300 return {"success": ok, "detail": "" if ok else f"status={resp.status_code}"} except Exception as e: return {"success": False, "detail": str(e)} # ==================== 工具类 ==================== class NotificationTools: """通知推送工具类""" def __init__(self, project_root: str = None): if project_root: self.project_root = Path(project_root) else: current_file = Path(__file__) self.project_root = current_file.parent.parent.parent def _load_merged_config(self) -> Dict[str, Any]: """ 加载合并后的通知配置（config.yaml + .env） Returns: 包含 webhook 配置和通知参数的合并字典 """ config_path = self.project_root / "config" / "config.yaml" if config_path.exists(): with open(config_path, "r", encoding="utf-8") as f: config_data = yaml.safe_load(f) else: config_data = {} webhook_config = _load_webhook_config(config_data) notification_config = _load_notification_config(config_data) return {**webhook_config, **notification_config} def _detect_config_source(self, env_key: str, yaml_value: str) -> str: """检测配置项来源：env / yaml / 未配置""" env_val = os.environ.get(env_key, "").strip() if env_val: return "env" elif yaml_value: return "yaml" return "" def get_channel_format_guide(self, channel: Optional[str] = None) -> Dict: """ 获取渠道格式化策略指南 返回各渠道支持的 Markdown 特性、限制和最佳格式化提示词， 供 LLM 在生成推送内容时参考，确保内容样式贴合目标渠道。 Args: channel: 指定渠道 ID，None 返回所有渠道的策略 Returns: 格式化策略字典 """ if channel: if channel not in CHANNEL_FORMAT_GUIDES: valid = list(CHANNEL_FORMAT_GUIDES.keys()) return { "success": False, "error": { "code": "INVALID_CHANNEL", "message": f"无效的渠道: {channel}", "suggestion": f"支持的渠道: {valid}", }, } guide = CHANNEL_FORMAT_GUIDES[channel] return { "success": True, "channel": channel, "guide": guide, } else: return { "success": True, "summary": f"共 {len(CHANNEL_FORMAT_GUIDES)} 个渠道的格式化策略", "guides": CHANNEL_FORMAT_GUIDES, } def get_notification_channels(self) -> Dict: """ 获取所有通知渠道的配置状态 检测 config.yaml 和 .env 环境变量，返回每个渠道是否已配置。 Returns: 渠道状态字典 """ try: config = self._load_merged_config() enabled = config.get("ENABLE_NOTIFICATION", True) # 从 yaml 直接读取（用于判断来源） config_path = self.project_root / "config" / "config.yaml" yaml_channels = {} if config_path.exists(): with open(config_path, "r", encoding="utf-8") as f: raw = yaml.safe_load(f) or {} yaml_channels = raw.get("notification", {}).get("channels", {}) channels = [] env_key_map = { "FEISHU_WEBHOOK_URL": ("feishu", "webhook_url"), "DINGTALK_WEBHOOK_URL": ("dingtalk", "webhook_url"), "WEWORK_WEBHOOK_URL": ("wework", "webhook_url"), "TELEGRAM_BOT_TOKEN": ("telegram", "bot_token"), "TELEGRAM_CHAT_ID": ("telegram", "chat_id"), "EMAIL_FROM": ("email", "from"), "EMAIL_PASSWORD": ("email", "password"), "EMAIL_TO": ("email", "to"), "NTFY_SERVER_URL": ("ntfy", "server_url"), "NTFY_TOPIC": ("ntfy", "topic"), "BARK_URL": ("bark", "url"), "SLACK_WEBHOOK_URL": ("slack", "webhook_url"), "GENERIC_WEBHOOK_URL": ("generic_webhook", "webhook_url"), } for channel_id, required_keys in _CHANNEL_REQUIREMENTS.items(): is_configured = all(config.get(k) for k in required_keys) # 判断来源 sources = set() for key in required_keys: ch_name, field = env_key_map.get(key, ("", "")) yaml_val = yaml_channels.get(ch_name, {}).get(field, "") src = self._detect_config_source(key, yaml_val) if src: sources.add(src) channels.append({ "id": channel_id, "name": _CHANNEL_NAMES.get(channel_id, channel_id), "configured": is_configured, "source": list(sources) if sources else [], }) configured_count = sum(1 for ch in channels if ch["configured"]) return { "success": True, "notification_enabled": enabled, "summary": f"{configured_count}/{len(channels)} 个渠道已配置", "channels": channels, } except Exception as e: return { "success": False, "error": {"code": "INTERNAL_ERROR", "message": str(e)}, } def send_notification( self, message: str, title: str = "TrendRadar 通知", channels: Optional[List[str]] = None, ) -> Dict: """ 向已配置的通知渠道发送消息 接受 markdown 格式内容，内部自动转换为各渠道要求的格式。 Args: message: markdown 格式的消息内容 title: 消息标题 channels: 指定发送的渠道列表，None 表示发送到所有已配置渠道 可选值: feishu, dingtalk, wework, telegram, email, ntfy, bark, slack, generic_webhook Returns: 发送结果字典 """ if not message or not message.strip(): return { "success": False, "error": {"code": "EMPTY_MESSAGE", "message": "消息内容不能为空"}, } try: config = self._load_merged_config() if not config.get("ENABLE_NOTIFICATION", True): return { "success": False, "error": {"code": "NOTIFICATION_DISABLED", "message": "通知功能已禁用（notification.enabled = false）"}, } # 确定目标渠道 all_channel_ids = list(_CHANNEL_REQUIREMENTS.keys()) if channels: # 验证渠道名称 invalid = [ch for ch in channels if ch not in all_channel_ids] if invalid: raise InvalidParameterError( f"无效的渠道: {invalid}", suggestion=f"支持的渠道: {all_channel_ids}" ) target_channels = channels else: # 发送到所有已配置渠道 target_channels = [ ch_id for ch_id, keys in _CHANNEL_REQUIREMENTS.items() if all(config.get(k) for k in keys) ] if not target_channels: return { "success": False, "error": { "code": "NO_CHANNELS", "message": "没有已配置的目标渠道", "suggestion": "请在 config.yaml 或 .env 中配置至少一个通知渠道", }, } # 逐渠道发送 results = {} for ch_id in target_channels: required_keys = _CHANNEL_REQUIREMENTS[ch_id] if not all(config.get(k) for k in required_keys): results[ch_id] = {"success": False, "detail": "渠道未配置"} continue result = self._dispatch_to_channel(ch_id, config, message, title) results[ch_id] = result success_count = sum(1 for r in results.values() if r["success"]) total = len(results) return { "success": success_count > 0, "summary": f"{success_count}/{total} 个渠道发送成功", "results": { ch_id: { "name": _CHANNEL_NAMES.get(ch_id, ch_id), **r, } for ch_id, r in results.items() }, } except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return { "success": False, "error": {"code": "INTERNAL_ERROR", "message": str(e)}, } def _dispatch_to_channel( self, channel_id: str, config: Dict, message: str, title: str ) -> Dict: """分发消息到指定渠道（格式适配 → 字节分批 → 多账号 × 逐批发送） 从 config.yaml → advanced.batch_size / batch_send_interval 读取配置。 """ # 从 config 读取批次配置（与 trendradar 一致） batch_sizes = self._get_batch_sizes() batch_interval = self._get_batch_interval() # Email 无字节限制，不走分批管线 if channel_id == "email": return _send_email( config["EMAIL_FROM"], config["EMAIL_PASSWORD"], config["EMAIL_TO"], message, title, config.get("EMAIL_SMTP_SERVER", ""), config.get("EMAIL_SMTP_PORT", ""), ) # 统一分批管线：格式适配 → 字节分割 → 添加批次头部 → (可选)反序 batches = _prepare_batches(message, channel_id, batch_sizes) # 按渠道路由发送 if channel_id == "feishu": return self._send_batched_multi_account( config["FEISHU_WEBHOOK_URL"], batches, channel_id, lambda url, content: _send_feishu(url, content, title), batch_interval, ) elif channel_id == "dingtalk": return self._send_batched_multi_account( config["DINGTALK_WEBHOOK_URL"], batches, channel_id, lambda url, content: _send_dingtalk(url, content, title), batch_interval, ) elif channel_id == "wework": msg_type = config.get("WEWORK_MSG_TYPE", "markdown") return self._send_batched_multi_account( config["WEWORK_WEBHOOK_URL"], batches, channel_id, lambda url, content: _send_wework(url, content, title, msg_type), batch_interval, ) elif channel_id == "telegram": return self._send_batched_telegram( config, batches, title, batch_interval, ) elif channel_id == "ntfy": return self._send_batched_ntfy( config, batches, title, batch_interval, ) elif channel_id == "bark": return self._send_batched_multi_account( config["BARK_URL"], batches, channel_id, lambda url, content: _send_bark(url, content, title), batch_interval, ) elif channel_id == "slack": return self._send_batched_multi_account( config["SLACK_WEBHOOK_URL"], batches, channel_id, lambda url, content: _send_slack(url, content, title), batch_interval, ) elif channel_id == "generic_webhook": template = config.get("GENERIC_WEBHOOK_TEMPLATE", "") return self._send_batched_multi_account( config["GENERIC_WEBHOOK_URL"], batches, channel_id, lambda url, content: _send_generic_webhook(url, content, title, template), batch_interval, ) else: return {"success": False, "detail": f"未知渠道: {channel_id}"} def _get_batch_sizes(self) -> Dict: """从 config.yaml 读取 advanced.batch_size，合并到默认值""" try: config_path = self.project_root / "config" / "config.yaml" if config_path.exists(): with open(config_path, "r", encoding="utf-8") as f: raw = yaml.safe_load(f) or {} advanced = raw.get("advanced", {}) cfg_sizes = advanced.get("batch_size", {}) # 从 config 构建渠道映射 sizes = dict(_CHANNEL_BATCH_SIZES_DEFAULT) default_size = cfg_sizes.get("default", 4000) for ch_id in sizes: if ch_id in cfg_sizes: sizes[ch_id] = cfg_sizes[ch_id] elif ch_id not in ("email", "ntfy") and sizes[ch_id] == 4000: # 使用 config 中的 default sizes[ch_id] = default_size return sizes except Exception: pass return dict(_CHANNEL_BATCH_SIZES_DEFAULT) def _get_batch_interval(self) -> float: """从 config.yaml 读取 advanced.batch_send_interval""" try: config_path = self.project_root / "config" / "config.yaml" if config_path.exists(): with open(config_path, "r", encoding="utf-8") as f: raw = yaml.safe_load(f) or {} return float(raw.get("advanced", {}).get("batch_send_interval", _BATCH_INTERVAL_DEFAULT)) except Exception: pass return _BATCH_INTERVAL_DEFAULT def _send_batched_multi_account( self, urls_str: str, batches: List[str], channel_id: str, send_func, batch_interval: float = _BATCH_INTERVAL_DEFAULT, ) -> Dict: """多账号 × 逐批发送（; 分隔的 URL）""" urls = [u.strip() for u in urls_str.split(";") if u.strip()] if not urls: return {"success": False, "detail": "URL 为空"} any_ok = False details = [] for url in urls: for i, batch in enumerate(batches): r = send_func(url, batch) if r["success"]: any_ok = True elif r["detail"]: details.append(r["detail"]) # 批次间间隔 if i < len(batches) - 1: time.sleep(batch_interval) return { "success": any_ok, "detail": "; ".join(details) if details else "", "batches": len(batches), } def _send_batched_telegram( self, config: Dict, batches: List[str], title: str, batch_interval: float = _BATCH_INTERVAL_DEFAULT, ) -> Dict: """Telegram 多账号 × 逐批发送（token/chat_id 配对）""" tokens = config["TELEGRAM_BOT_TOKEN"].split(";") chat_ids = config["TELEGRAM_CHAT_ID"].split(";") if len(tokens) != len(chat_ids): return {"success": False, "detail": "bot_token 和 chat_id 数量不一致"} any_ok = False details = [] for token, cid in zip(tokens, chat_ids): token, cid = token.strip(), cid.strip() if not (token and cid): continue for i, batch in enumerate(batches): r = _send_telegram(token, cid, batch, title) if r["success"]: any_ok = True elif r["detail"]: details.append(r["detail"]) if i < len(batches) - 1: time.sleep(batch_interval) return { "success": any_ok, "detail": "; ".join(details) if details else "", "batches": len(batches), } def _send_batched_ntfy( self, config: Dict, batches: List[str], title: str, batch_interval: float = _BATCH_INTERVAL_DEFAULT, ) -> Dict: """ntfy 多账号 × 逐批发送（server/topic/token 配对，含速率限制处理）""" servers = config["NTFY_SERVER_URL"].split(";") topics = config["NTFY_TOPIC"].split(";") tokens_str = config.get("NTFY_TOKEN", "") tokens = tokens_str.split(";") if tokens_str else [""] if len(servers) != len(topics): return {"success": False, "detail": "server_url 和 topic 数量不一致"} any_ok = False details = [] for i, (srv, topic) in enumerate(zip(servers, topics)): srv, topic = srv.strip(), topic.strip() tk = tokens[i].strip() if i < len(tokens) else "" if not (srv and topic): continue # ntfy.sh 公共服务器用 2s 间隔（与 trendradar 一致） interval = 2.0 if "ntfy.sh" in srv else batch_interval for j, batch in enumerate(batches): r = _send_ntfy(srv, topic, batch, title, tk) if r["success"]: any_ok = True elif r["detail"]: details.append(r["detail"]) if j < len(batches) - 1: time.sleep(interval) return { "success": any_ok, "detail": "; ".join(details) if details else "", "batches": len(batches), } ================================================ FILE: mcp_server/tools/search_tools.py ================================================ """ 智能新闻检索工具 提供模糊搜索、链接查询、历史相关新闻检索等高级搜索功能。 """ import re from collections import Counter from datetime import datetime, timedelta from difflib import SequenceMatcher from typing import Dict, List, Optional, Tuple, Union from ..services.data_service import DataService from ..utils.validators import validate_keyword, validate_limit, validate_threshold, normalize_date_range from ..utils.errors import MCPError, InvalidParameterError, DataNotFoundError class SearchTools: """智能新闻检索工具类""" def __init__(self, project_root: str = None): """ 初始化智能检索工具 Args: project_root: 项目根目录 """ self.data_service = DataService(project_root) def search_news_unified( self, query: str, search_mode: str = "keyword", date_range: Optional[Union[Dict[str, str], str]] = None, platforms: Optional[List[str]] = None, limit: int = 50, sort_by: str = "relevance", threshold: float = 0.6, include_url: bool = False, include_rss: bool = False, rss_limit: int = 20 ) -> Dict: """ 统一新闻搜索工具 - 整合多种搜索模式，支持同时搜索热榜和RSS Args: query: 查询内容（必需）- 关键词、内容片段或实体名称 search_mode: 搜索模式，可选值： - "keyword": 精确关键词匹配（默认） - "fuzzy": 模糊内容匹配（使用相似度算法） - "entity": 实体名称搜索（自动按权重排序） date_range: 日期范围（可选） - **格式**: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - **示例**: {"start": "2025-01-01", "end": "2025-01-07"} - **默认**: 不指定时默认查询今天 - **注意**: start和end可以相同（表示单日查询） platforms: 平台过滤列表，如 ['zhihu', 'weibo'] limit: 热榜返回条数限制，默认50 sort_by: 排序方式，可选值： - "relevance": 按相关度排序（默认） - "weight": 按新闻权重排序 - "date": 按日期排序 threshold: 相似度阈值（仅fuzzy模式有效），0-1之间，默认0.6 include_url: 是否包含URL链接，默认False（节省token） include_rss: 是否同时搜索RSS数据，默认False rss_limit: RSS返回条数限制，默认20 Returns: 搜索结果字典，包含匹配的新闻列表（热榜和RSS分开展示） Examples: - search_news_unified(query="人工智能", search_mode="keyword") - search_news_unified(query="特斯拉降价", search_mode="fuzzy", threshold=0.4) - search_news_unified(query="马斯克", search_mode="entity", limit=20) - search_news_unified(query="AI", include_rss=True) # 同时搜索热榜和RSS - search_news_unified(query="iPhone 16", date_range={"start": "2025-01-01", "end": "2025-01-07"}) """ try: # 参数验证 query = validate_keyword(query) if search_mode not in ["keyword", "fuzzy", "entity"]: raise InvalidParameterError( f"无效的搜索模式: {search_mode}", suggestion="支持的模式: keyword, fuzzy, entity" ) if sort_by not in ["relevance", "weight", "date"]: raise InvalidParameterError( f"无效的排序方式: {sort_by}", suggestion="支持的排序: relevance, weight, date" ) limit = validate_limit(limit, default=50) threshold = validate_threshold(threshold, default=0.6, min_value=0.0, max_value=1.0) # 处理日期范围 if date_range: from ..utils.validators import validate_date_range date_range_tuple = validate_date_range(date_range) start_date, end_date = date_range_tuple else: # 不指定日期时，使用最新可用数据日期（而非 datetime.now()） earliest, latest = self.data_service.get_available_date_range() if latest is None: # 没有任何可用数据 return { "success": False, "error": { "code": "NO_DATA_AVAILABLE", "message": "output 目录下没有可用的新闻数据", "suggestion": "请先运行爬虫生成数据，或检查 output 目录" } } # 使用最新可用日期 start_date = end_date = latest # 收集所有匹配的新闻 all_matches = [] current_date = start_date while current_date <= end_date: try: all_titles, id_to_name, timestamps = self.data_service.parser.read_all_titles_for_date( date=current_date, platform_ids=platforms ) # 根据搜索模式执行不同的搜索逻辑 if search_mode == "keyword": matches = self._search_by_keyword_mode( query, all_titles, id_to_name, current_date, include_url ) elif search_mode == "fuzzy": matches = self._search_by_fuzzy_mode( query, all_titles, id_to_name, current_date, threshold, include_url ) else: # entity matches = self._search_by_entity_mode( query, all_titles, id_to_name, current_date, include_url ) all_matches.extend(matches) except DataNotFoundError: # 该日期没有数据，继续下一天 pass current_date += timedelta(days=1) if not all_matches: # 获取可用日期范围用于错误提示 earliest, latest = self.data_service.get_available_date_range() # 判断时间范围描述 if start_date.date() == datetime.now().date() and start_date == end_date: time_desc = "今天" elif start_date == end_date: time_desc = start_date.strftime("%Y-%m-%d") else: time_desc = f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" # 构建错误消息 if earliest and latest: available_desc = f"{earliest.strftime('%Y-%m-%d')} 至 {latest.strftime('%Y-%m-%d')}" message = f"未找到匹配的新闻（查询范围: {time_desc}，可用数据: {available_desc}）" else: message = f"未找到匹配的新闻（{time_desc}）" result = { "success": True, "results": [], "total": 0, "query": query, "search_mode": search_mode, "time_range": time_desc, "message": message } return result # 统一排序逻辑 if sort_by == "relevance": all_matches.sort(key=lambda x: x.get("similarity_score", 1.0), reverse=True) elif sort_by == "weight": from .analytics import calculate_news_weight all_matches.sort(key=lambda x: calculate_news_weight(x), reverse=True) elif sort_by == "date": all_matches.sort(key=lambda x: x.get("date", ""), reverse=True) # 限制返回数量 results = all_matches[:limit] # 构建时间范围描述（正确判断是否为今天） if start_date.date() == datetime.now().date() and start_date == end_date: time_range_desc = "今天" elif start_date == end_date: time_range_desc = start_date.strftime("%Y-%m-%d") else: time_range_desc = f"{start_date.strftime('%Y-%m-%d')} 至 {end_date.strftime('%Y-%m-%d')}" result = { "success": True, "summary": { "description": f"新闻搜索结果（{search_mode}模式）", "total_found": len(all_matches), "returned": len(results), "requested_limit": limit, "search_mode": search_mode, "query": query, "platforms": platforms or "所有平台", "time_range": time_range_desc, "sort_by": sort_by }, "data": results } if search_mode == "fuzzy": result["summary"]["threshold"] = threshold if len(all_matches) < limit: result["note"] = f"模糊搜索模式下，相似度阈值 {threshold} 仅匹配到 {len(all_matches)} 条结果" # 如果启用 RSS 搜索，同时搜索 RSS 数据 if include_rss: rss_results = self._search_rss_by_keyword( query=query, start_date=start_date, end_date=end_date, limit=rss_limit, include_url=include_url ) result["rss"] = rss_results["items"] result["rss_total"] = rss_results["total"] result["summary"]["include_rss"] = True result["summary"]["rss_found"] = rss_results["total"] result["summary"]["rss_returned"] = len(rss_results["items"]) return result except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def _search_by_keyword_mode( self, query: str, all_titles: Dict, id_to_name: Dict, current_date: datetime, include_url: bool ) -> List[Dict]: """ 关键词搜索模式（精确匹配） Args: query: 搜索关键词 all_titles: 所有标题字典 id_to_name: 平台ID到名称映射 current_date: 当前日期 Returns: 匹配的新闻列表 """ matches = [] query_lower = query.lower() for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 精确包含判断 if query_lower in title.lower(): news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "similarity_score": 1.0, # 精确匹配，相似度为1 "ranks": info.get("ranks", []), "count": len(info.get("ranks", [])), "rank": info["ranks"][0] if info["ranks"] else 999 } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") matches.append(news_item) return matches def _search_by_fuzzy_mode( self, query: str, all_titles: Dict, id_to_name: Dict, current_date: datetime, threshold: float, include_url: bool ) -> List[Dict]: """ 模糊搜索模式（使用相似度算法） Args: query: 搜索内容 all_titles: 所有标题字典 id_to_name: 平台ID到名称映射 current_date: 当前日期 threshold: 相似度阈值 Returns: 匹配的新闻列表 """ matches = [] for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 模糊匹配 is_match, similarity = self._fuzzy_match(query, title, threshold) if is_match: news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "similarity_score": round(similarity, 4), "ranks": info.get("ranks", []), "count": len(info.get("ranks", [])), "rank": info["ranks"][0] if info["ranks"] else 999 } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") matches.append(news_item) return matches def _search_by_entity_mode( self, query: str, all_titles: Dict, id_to_name: Dict, current_date: datetime, include_url: bool ) -> List[Dict]: """ 实体搜索模式（自动按权重排序） Args: query: 实体名称 all_titles: 所有标题字典 id_to_name: 平台ID到名称映射 current_date: 当前日期 Returns: 匹配的新闻列表 """ matches = [] for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 实体搜索：精确包含实体名称 if query in title: news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "similarity_score": 1.0, "ranks": info.get("ranks", []), "count": len(info.get("ranks", [])), "rank": info["ranks"][0] if info["ranks"] else 999 } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") matches.append(news_item) return matches def _calculate_similarity(self, text1: str, text2: str) -> float: """ 计算两个文本的相似度 Args: text1: 文本1 text2: 文本2 Returns: 相似度分数 (0-1之间) """ # 使用 difflib.SequenceMatcher 计算序列相似度 return SequenceMatcher(None, text1.lower(), text2.lower()).ratio() def _fuzzy_match(self, query: str, text: str, threshold: float = 0.3) -> Tuple[bool, float]: """ 模糊匹配函数 Args: query: 查询文本 text: 待匹配文本 threshold: 匹配阈值 Returns: (是否匹配, 相似度分数) """ # 直接包含判断 if query.lower() in text.lower(): return True, 1.0 # 计算整体相似度 similarity = self._calculate_similarity(query, text) if similarity >= threshold: return True, similarity # 分词后的部分匹配 query_words = set(self._extract_keywords(query)) text_words = set(self._extract_keywords(text)) if not query_words or not text_words: return False, 0.0 # 计算关键词重合度 common_words = query_words & text_words keyword_overlap = len(common_words) / len(query_words) if keyword_overlap >= 0.5: # 50%的关键词重合 return True, keyword_overlap return False, similarity def _extract_keywords(self, text: str, min_length: int = 2) -> List[str]: """ 从文本中提取关键词 Args: text: 输入文本 min_length: 最小词长 Returns: 关键词列表 """ # 移除URL和特殊字符 text = re.sub(r'http[s]?://\S+', '', text) text = re.sub(r'\[.*?\]', '', text) # 移除方括号内容 # 使用正则表达式分词（中文和英文） words = re.findall(r'[\w]+', text) # 过滤短词 keywords = [word for word in words if word and len(word) >= min_length] return keywords def _calculate_keyword_overlap(self, keywords1: List[str], keywords2: List[str]) -> float: """ 计算两个关键词列表的重合度 Args: keywords1: 关键词列表1 keywords2: 关键词列表2 Returns: 重合度分数 (0-1之间) """ if not keywords1 or not keywords2: return 0.0 set1 = set(keywords1) set2 = set(keywords2) # Jaccard 相似度 intersection = len(set1 & set2) union = len(set1 | set2) if union == 0: return 0.0 return intersection / union def _jaccard_similarity(self, list1: List[str], list2: List[str]) -> float: """ 计算两个列表的 Jaccard 相似度 Args: list1: 列表1 list2: 列表2 Returns: Jaccard 相似度 (0-1之间) """ if not list1 or not list2: return 0.0 set1 = set(list1) set2 = set(list2) intersection = len(set1 & set2) union = len(set1 | set2) if union == 0: return 0.0 return intersection / union def search_related_news_history( self, reference_title: str, time_preset: str = "yesterday", start_date: Optional[datetime] = None, end_date: Optional[datetime] = None, threshold: float = 0.4, limit: int = 50, include_url: bool = False ) -> Dict: """ 在历史数据中搜索与给定新闻相关的新闻 Args: reference_title: 参考新闻标题或内容 time_preset: 时间范围预设值，可选： - "yesterday": 昨天 - "last_week": 上周 (7天) - "last_month": 上个月 (30天) - "custom": 自定义日期范围（需要提供 start_date 和 end_date） start_date: 自定义开始日期（仅当 time_preset="custom" 时有效） end_date: 自定义结束日期（仅当 time_preset="custom" 时有效） threshold: 相似度阈值 (0-1之间)，默认0.4 limit: 返回条数限制，默认50 include_url: 是否包含URL链接，默认False（节省token） Returns: 搜索结果字典，包含相关新闻列表 Example: >>> tools = SearchTools() >>> result = tools.search_related_news_history( ... reference_title="人工智能技术突破", ... time_preset="last_week", ... threshold=0.4, ... limit=50 ... ) >>> for news in result['results']: ... print(f"{news['date']}: {news['title']} (相似度: {news['similarity_score']})") """ try: # 参数验证 reference_title = validate_keyword(reference_title) threshold = validate_threshold(threshold, default=0.4, min_value=0.0, max_value=1.0) limit = validate_limit(limit, default=50) # 确定查询日期范围 today = datetime.now() if time_preset == "yesterday": search_start = today - timedelta(days=1) search_end = today - timedelta(days=1) elif time_preset == "last_week": search_start = today - timedelta(days=7) search_end = today - timedelta(days=1) elif time_preset == "last_month": search_start = today - timedelta(days=30) search_end = today - timedelta(days=1) elif time_preset == "custom": if not start_date or not end_date: raise InvalidParameterError( "自定义时间范围需要提供 start_date 和 end_date", suggestion="请提供 start_date 和 end_date 参数" ) search_start = start_date search_end = end_date else: raise InvalidParameterError( f"不支持的时间范围: {time_preset}", suggestion="请使用 'yesterday', 'last_week', 'last_month' 或 'custom'" ) # 提取参考文本的关键词 reference_keywords = self._extract_keywords(reference_title) if not reference_keywords: raise InvalidParameterError( "无法从参考文本中提取关键词", suggestion="请提供更详细的文本内容" ) # 收集所有相关新闻 all_related_news = [] current_date = search_start while current_date <= search_end: try: # 读取该日期的数据 all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date(current_date) # 搜索相关新闻 for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): # 计算标题相似度 title_similarity = self._calculate_similarity(reference_title, title) # 提取标题关键词 title_keywords = self._extract_keywords(title) # 计算关键词重合度 keyword_overlap = self._calculate_keyword_overlap( reference_keywords, title_keywords ) # 综合相似度 (70% 关键词重合 + 30% 文本相似度) combined_score = keyword_overlap * 0.7 + title_similarity * 0.3 if combined_score >= threshold: news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": current_date.strftime("%Y-%m-%d"), "similarity_score": round(combined_score, 4), "keyword_overlap": round(keyword_overlap, 4), "text_similarity": round(title_similarity, 4), "common_keywords": list(set(reference_keywords) & set(title_keywords)), "rank": info["ranks"][0] if info["ranks"] else 0 } # 条件性添加 URL 字段 if include_url: news_item["url"] = info.get("url", "") news_item["mobileUrl"] = info.get("mobileUrl", "") all_related_news.append(news_item) except DataNotFoundError: # 该日期没有数据，继续下一天 pass except Exception as e: # 记录错误但继续处理其他日期 print(f"Warning: 处理日期 {current_date.strftime('%Y-%m-%d')} 时出错: {e}") # 移动到下一天 current_date += timedelta(days=1) if not all_related_news: return { "success": True, "results": [], "total": 0, "query": reference_title, "time_preset": time_preset, "date_range": { "start": search_start.strftime("%Y-%m-%d"), "end": search_end.strftime("%Y-%m-%d") }, "message": "未找到相关新闻" } # 按相似度排序 all_related_news.sort(key=lambda x: x["similarity_score"], reverse=True) # 限制返回数量 results = all_related_news[:limit] # 统计信息 platform_distribution = Counter([news["platform"] for news in all_related_news]) date_distribution = Counter([news["date"] for news in all_related_news]) result = { "success": True, "summary": { "description": "历史相关新闻搜索结果", "total_found": len(all_related_news), "returned": len(results), "requested_limit": limit, "threshold": threshold, "reference_title": reference_title, "reference_keywords": reference_keywords, "time_preset": time_preset, "date_range": { "start": search_start.strftime("%Y-%m-%d"), "end": search_end.strftime("%Y-%m-%d") } }, "data": results, "statistics": { "platform_distribution": dict(platform_distribution), "date_distribution": dict(date_distribution), "avg_similarity": round( sum([news["similarity_score"] for news in all_related_news]) / len(all_related_news), 4 ) if all_related_news else 0.0 } } if len(all_related_news) < limit: result["note"] = f"相关性阈值 {threshold} 下仅找到 {len(all_related_news)} 条相关新闻" return result except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def find_related_news_unified( self, reference_title: str, date_range: Optional[Union[Dict[str, str], str]] = None, threshold: float = 0.5, limit: int = 50, include_url: bool = False ) -> Dict: """ 统一的相关新闻查找工具 - 整合相似新闻和历史相关搜索 Args: reference_title: 参考新闻标题 date_range: 日期范围（可选） - 不指定: 只查询今天的数据 - {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}: 查询指定日期范围 - "today": 今天 - "yesterday": 昨天 - "last_week": 最近7天 - "last_month": 最近30天 threshold: 相似度阈值，0-1之间，默认0.5 limit: 返回条数限制，默认50 include_url: 是否包含URL链接，默认False Returns: 相关新闻列表，按相似度排序 """ try: # 参数验证 reference_title = validate_keyword(reference_title) threshold = validate_threshold(threshold, default=0.5, min_value=0.0, max_value=1.0) limit = validate_limit(limit, default=50) # 确定日期范围 today = datetime.now() # 规范化 date_range（处理 JSON 字符串序列化问题） date_range = normalize_date_range(date_range) if date_range is None or date_range == "today": # 只查询今天 search_dates = [today] elif isinstance(date_range, str): # 预设时间范围 if date_range == "yesterday": search_dates = [today - timedelta(days=1)] elif date_range == "last_week": search_dates = [today - timedelta(days=i) for i in range(7)] elif date_range == "last_month": search_dates = [today - timedelta(days=i) for i in range(30)] else: # 单日字符串格式 try: single_date = datetime.strptime(date_range, "%Y-%m-%d") search_dates = [single_date] except ValueError: search_dates = [today] elif isinstance(date_range, dict): # 日期范围对象 start_str = date_range.get("start") end_str = date_range.get("end") if start_str and end_str: start_date = datetime.strptime(start_str, "%Y-%m-%d") end_date = datetime.strptime(end_str, "%Y-%m-%d") search_dates = [] current = start_date while current <= end_date: search_dates.append(current) current += timedelta(days=1) else: search_dates = [today] else: search_dates = [today] # 提取参考标题的关键词 reference_keywords = self._extract_keywords(reference_title) # 收集所有相关新闻 all_related_news = [] for search_date in search_dates: try: all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date(search_date) for platform_id, titles in all_titles.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles.items(): if title == reference_title: continue # 计算相似度（使用混合算法） text_similarity = self._calculate_similarity(reference_title, title) # 如果有关键词，也计算关键词重合度 if reference_keywords: title_keywords = self._extract_keywords(title) keyword_similarity = self._jaccard_similarity(reference_keywords, title_keywords) # 混合相似度：70% 文本 + 30% 关键词 similarity = 0.7 * text_similarity + 0.3 * keyword_similarity else: similarity = text_similarity if similarity >= threshold: news_item = { "title": title, "platform": platform_id, "platform_name": platform_name, "date": search_date.strftime("%Y-%m-%d"), "similarity": round(similarity, 3), "rank": info["ranks"][0] if info["ranks"] else 0 } if include_url: news_item["url"] = info.get("url", "") all_related_news.append(news_item) except Exception: # 某天数据读取失败，跳过 continue # 按相似度排序 all_related_news.sort(key=lambda x: x["similarity"], reverse=True) # 限制数量 results = all_related_news[:limit] # 统计信息 from collections import Counter platform_dist = Counter([n["platform_name"] for n in all_related_news]) date_dist = Counter([n["date"] for n in all_related_news]) return { "success": True, "summary": { "description": "相关新闻搜索结果", "total_found": len(all_related_news), "returned": len(results), "reference_title": reference_title, "threshold": threshold, "date_range": { "start": min(search_dates).strftime("%Y-%m-%d"), "end": max(search_dates).strftime("%Y-%m-%d") } if search_dates else None }, "data": results, "statistics": { "platform_distribution": dict(platform_dist), "date_distribution": dict(date_dist) } } except MCPError as e: return {"success": False, "error": e.to_dict()} except Exception as e: return {"success": False, "error": {"code": "INTERNAL_ERROR", "message": str(e)}} def _search_rss_by_keyword( self, query: str, start_date: datetime, end_date: datetime, limit: int = 20, include_url: bool = False ) -> Dict: """ 在 RSS 数据中搜索关键词 Args: query: 搜索关键词 start_date: 开始日期 end_date: 结束日期 limit: 返回条数限制 include_url: 是否包含 URL Returns: RSS 搜索结果字典 """ all_rss_matches = [] query_lower = query.lower() current_date = start_date while current_date <= end_date: try: # 读取该日期的 RSS 数据 all_titles, id_to_name, _ = self.data_service.parser.read_all_titles_for_date( date=current_date, platform_ids=None, db_type="rss" ) for feed_id, items in all_titles.items(): feed_name = id_to_name.get(feed_id, feed_id) for title, info in items.items(): # 关键词匹配（标题或摘要） title_match = query_lower in title.lower() summary = info.get("summary", "") summary_match = query_lower in summary.lower() if summary else False if title_match or summary_match: rss_item = { "title": title, "feed_id": feed_id, "feed_name": feed_name, "date": current_date.strftime("%Y-%m-%d"), "published_at": info.get("published_at", ""), "author": info.get("author", ""), "match_in": "title" if title_match else "summary" } if include_url: rss_item["url"] = info.get("url", "") all_rss_matches.append(rss_item) except DataNotFoundError: # 该日期没有 RSS 数据，继续下一天 pass except Exception: # 其他错误，跳过 pass current_date += timedelta(days=1) # 按发布时间排序（最新的在前） all_rss_matches.sort(key=lambda x: x.get("published_at", ""), reverse=True) return { "items": all_rss_matches[:limit], "total": len(all_rss_matches) } ================================================ FILE: mcp_server/tools/storage_sync.py ================================================ # coding=utf-8 """ 存储同步工具 实现从远程存储拉取数据到本地、获取存储状态、列出可用日期等功能。 """ import os import re from pathlib import Path from datetime import datetime, timedelta from typing import Dict, List, Optional import yaml from ..utils.errors import MCPError class StorageSyncTools: """存储同步工具类""" def __init__(self, project_root: str = None): """ 初始化存储同步工具 Args: project_root: 项目根目录 """ if project_root: self.project_root = Path(project_root) else: current_file = Path(__file__) self.project_root = current_file.parent.parent.parent self._config = None self._remote_backend = None def _load_config(self) -> dict: """加载配置文件""" if self._config is None: config_path = self.project_root / "config" / "config.yaml" if config_path.exists(): with open(config_path, "r", encoding="utf-8") as f: self._config = yaml.safe_load(f) else: self._config = {} return self._config def _get_storage_config(self) -> dict: """获取存储配置""" config = self._load_config() return config.get("storage", {}) def _get_remote_config(self) -> dict: """ 获取远程存储配置（合并配置文件和环境变量） """ storage_config = self._get_storage_config() remote_config = storage_config.get("remote", {}) return { "endpoint_url": remote_config.get("endpoint_url") or os.environ.get("S3_ENDPOINT_URL", ""), "bucket_name": remote_config.get("bucket_name") or os.environ.get("S3_BUCKET_NAME", ""), "access_key_id": remote_config.get("access_key_id") or os.environ.get("S3_ACCESS_KEY_ID", ""), "secret_access_key": remote_config.get("secret_access_key") or os.environ.get("S3_SECRET_ACCESS_KEY", ""), "region": remote_config.get("region") or os.environ.get("S3_REGION", ""), } def _has_remote_config(self) -> bool: """检查是否有有效的远程存储配置""" config = self._get_remote_config() return bool( config.get("bucket_name") and config.get("access_key_id") and config.get("secret_access_key") and config.get("endpoint_url") ) def _get_remote_backend(self): """获取远程存储后端实例""" if self._remote_backend is not None: return self._remote_backend if not self._has_remote_config(): return None try: from trendradar.storage.remote import RemoteStorageBackend remote_config = self._get_remote_config() config = self._load_config() timezone = config.get("app", {}).get("timezone", "Asia/Shanghai") self._remote_backend = RemoteStorageBackend( bucket_name=remote_config["bucket_name"], access_key_id=remote_config["access_key_id"], secret_access_key=remote_config["secret_access_key"], endpoint_url=remote_config["endpoint_url"], region=remote_config.get("region", ""), timezone=timezone, ) return self._remote_backend except ImportError: print("[存储同步] 远程存储后端需要安装 boto3: pip install boto3") return None except Exception as e: print(f"[存储同步] 创建远程后端失败: {e}") return None def _get_local_data_dir(self) -> Path: """获取本地数据目录""" storage_config = self._get_storage_config() local_config = storage_config.get("local", {}) data_dir = local_config.get("data_dir", "output") return self.project_root / data_dir def _parse_date_folder_name(self, folder_name: str) -> Optional[datetime]: """ 解析日期文件夹名称（兼容中文和 ISO 格式） 支持两种格式： - 中文格式：YYYY年MM月DD日 - ISO 格式：YYYY-MM-DD """ # 尝试 ISO 格式 iso_match = re.match(r'(\d{4})-(\d{2})-(\d{2})', folder_name) if iso_match: try: return datetime( int(iso_match.group(1)), int(iso_match.group(2)), int(iso_match.group(3)) ) except ValueError: pass # 尝试中文格式 chinese_match = re.match(r'(\d{4})年(\d{2})月(\d{2})日', folder_name) if chinese_match: try: return datetime( int(chinese_match.group(1)), int(chinese_match.group(2)), int(chinese_match.group(3)) ) except ValueError: pass return None def _get_local_dates(self, db_type: str = "news") -> List[str]: """ 获取本地可用的日期列表 存储结构: output/{db_type}/{date}.db 例如: output/news/2025-12-30.db, output/rss/2025-12-30.db Args: db_type: 数据库类型 ("news" 或 "rss")，默认 "news" Returns: 日期列表（按时间倒序） """ local_dir = self._get_local_data_dir() dates = set() if not local_dir.exists(): return [] # 扫描 output/{db_type}/{date}.db 文件 type_dir = local_dir / db_type if type_dir.exists(): for item in type_dir.iterdir(): if item.is_file() and item.suffix == ".db": # 从文件名解析日期 (2025-12-30.db -> 2025-12-30) date_str = item.stem # 去除 .db 后缀 folder_date = self._parse_date_folder_name(date_str) if folder_date: dates.add(folder_date.strftime("%Y-%m-%d")) return sorted(list(dates), reverse=True) def _get_all_local_dates(self) -> Dict[str, List[str]]: """ 获取所有本地可用的日期列表（包括 news 和 rss） Returns: { "news": ["2025-12-30", ...], "rss": ["2025-12-30", ...], "all": ["2025-12-30", ...] # 合并去重 } """ news_dates = set(self._get_local_dates("news")) rss_dates = set(self._get_local_dates("rss")) all_dates = news_dates | rss_dates return { "news": sorted(list(news_dates), reverse=True), "rss": sorted(list(rss_dates), reverse=True), "all": sorted(list(all_dates), reverse=True) } def _calculate_dir_size(self, path: Path) -> int: """计算目录大小（字节）""" total_size = 0 if path.exists(): for item in path.rglob("*"): if item.is_file(): total_size += item.stat().st_size return total_size def sync_from_remote(self, days: int = 7) -> Dict: """ 从远程存储拉取数据到本地 Args: days: 拉取最近 N 天的数据，默认 7 天 Returns: 同步结果字典 """ try: # 检查远程配置 if not self._has_remote_config(): return { "success": False, "error": { "code": "REMOTE_NOT_CONFIGURED", "message": "未配置远程存储", "suggestion": "请在 config/config.yaml 中配置 storage.remote 或设置环境变量" } } # 获取远程后端 remote_backend = self._get_remote_backend() if remote_backend is None: return { "success": False, "error": { "code": "REMOTE_BACKEND_FAILED", "message": "无法创建远程存储后端", "suggestion": "请检查远程存储配置和 boto3 是否已安装" } } # 获取本地数据目录 local_dir = self._get_local_data_dir() local_dir.mkdir(parents=True, exist_ok=True) # 获取远程可用日期 remote_dates = remote_backend.list_remote_dates() # 获取本地已有日期 local_dates = set(self._get_local_dates()) # 计算需要拉取的日期（最近 N 天） from trendradar.utils.time import get_configured_time config = self._load_config() timezone = config.get("app", {}).get("timezone", "Asia/Shanghai") now = get_configured_time(timezone) target_dates = [] for i in range(days): date = now - timedelta(days=i) date_str = date.strftime("%Y-%m-%d") if date_str in remote_dates: target_dates.append(date_str) # 执行拉取 synced_dates = [] skipped_dates = [] failed_dates = [] for date_str in target_dates: # 检查本地是否已存在 if date_str in local_dates: skipped_dates.append(date_str) continue # 拉取单个日期 try: local_date_dir = local_dir / date_str local_db_path = local_date_dir / "news.db" remote_key = f"news/{date_str}.db" local_date_dir.mkdir(parents=True, exist_ok=True) remote_backend.s3_client.download_file( remote_backend.bucket_name, remote_key, str(local_db_path) ) synced_dates.append(date_str) print(f"[存储同步] 已拉取: {date_str}") except Exception as e: failed_dates.append({"date": date_str, "error": str(e)}) print(f"[存储同步] 拉取失败 ({date_str}): {e}") return { "success": True, "summary": { "description": "远程存储同步结果", "synced_files": len(synced_dates), "skipped_count": len(skipped_dates), "failed_count": len(failed_dates) }, "data": { "synced_dates": synced_dates, "skipped_dates": skipped_dates, "failed_dates": failed_dates }, "message": f"成功同步 {len(synced_dates)} 天数据" + ( f"，跳过 {len(skipped_dates)} 天（本地已存在）" if skipped_dates else "" ) + ( f"，失败 {len(failed_dates)} 天" if failed_dates else "" ) } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def get_storage_status(self) -> Dict: """ 获取存储配置和状态 Returns: 存储状态字典 """ try: storage_config = self._get_storage_config() config = self._load_config() # 本地存储状态 local_config = storage_config.get("local", {}) local_dir = self._get_local_data_dir() local_size = self._calculate_dir_size(local_dir) # 获取分类的日期列表 all_dates = self._get_all_local_dates() news_dates = all_dates["news"] rss_dates = all_dates["rss"] combined_dates = all_dates["all"] local_status = { "data_dir": local_config.get("data_dir", "output"), "retention_days": local_config.get("retention_days", 0), "total_size": f"{local_size / 1024 / 1024:.2f} MB", "total_size_bytes": local_size, "date_count": len(combined_dates), "earliest_date": combined_dates[-1] if combined_dates else None, "latest_date": combined_dates[0] if combined_dates else None, "news": { "date_count": len(news_dates), "dates": news_dates[:10], # 最近 10 天 }, "rss": { "date_count": len(rss_dates), "dates": rss_dates[:10], # 最近 10 天 }, } # 远程存储状态 remote_config = storage_config.get("remote", {}) has_remote = self._has_remote_config() remote_status = { "configured": has_remote, "retention_days": remote_config.get("retention_days", 0), } if has_remote: merged_config = self._get_remote_config() # 脱敏显示 endpoint = merged_config.get("endpoint_url", "") bucket = merged_config.get("bucket_name", "") remote_status["endpoint_url"] = endpoint remote_status["bucket_name"] = bucket # 尝试获取远程日期列表 remote_backend = self._get_remote_backend() if remote_backend: try: remote_dates = remote_backend.list_remote_dates() remote_status["date_count"] = len(remote_dates) remote_status["earliest_date"] = remote_dates[-1] if remote_dates else None remote_status["latest_date"] = remote_dates[0] if remote_dates else None except Exception as e: remote_status["error"] = str(e) # 拉取配置状态 pull_config = storage_config.get("pull", {}) pull_status = { "enabled": pull_config.get("enabled", False), "days": pull_config.get("days", 7), } return { "success": True, "summary": { "description": "存储配置和状态信息", "backend": storage_config.get("backend", "auto") }, "data": { "local": local_status, "remote": remote_status, "pull": pull_status } } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def list_available_dates(self, source: str = "both") -> Dict: """ 列出可用的日期范围 Args: source: 数据来源 - "local": 仅本地 - "remote": 仅远程 - "both": 两者都列出（默认） Returns: 日期列表字典 """ try: data_result = {} summary_info = { "description": "可用日期列表", "source": source } # 本地日期 if source in ("local", "both"): all_dates = self._get_all_local_dates() news_dates = all_dates["news"] rss_dates = all_dates["rss"] combined_dates = all_dates["all"] data_result["local"] = { "dates": combined_dates, "count": len(combined_dates), "earliest": combined_dates[-1] if combined_dates else None, "latest": combined_dates[0] if combined_dates else None, "news": { "dates": news_dates, "count": len(news_dates), }, "rss": { "dates": rss_dates, "count": len(rss_dates), }, } # 远程日期 if source in ("remote", "both"): if not self._has_remote_config(): data_result["remote"] = { "configured": False, "dates": [], "count": 0, "earliest": None, "latest": None, "error": "未配置远程存储" } else: remote_backend = self._get_remote_backend() if remote_backend: try: remote_dates = remote_backend.list_remote_dates() data_result["remote"] = { "configured": True, "dates": remote_dates, "count": len(remote_dates), "earliest": remote_dates[-1] if remote_dates else None, "latest": remote_dates[0] if remote_dates else None, } except Exception as e: data_result["remote"] = { "configured": True, "dates": [], "count": 0, "earliest": None, "latest": None, "error": str(e) } else: data_result["remote"] = { "configured": True, "dates": [], "count": 0, "earliest": None, "latest": None, "error": "无法创建远程存储后端" } # 如果同时查询两者，计算差异 if source == "both" and "local" in data_result and "remote" in data_result: local_set = set(data_result["local"]["dates"]) remote_set = set(data_result["remote"].get("dates", [])) data_result["comparison"] = { "only_local": sorted(list(local_set - remote_set), reverse=True), "only_remote": sorted(list(remote_set - local_set), reverse=True), "both": sorted(list(local_set & remote_set), reverse=True), } return { "success": True, "summary": summary_info, "data": data_result } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } ================================================ FILE: mcp_server/tools/system.py ================================================ """ 系统管理工具 实现系统状态查询和爬虫触发功能。 """ from pathlib import Path from typing import Dict, List, Optional from ..services.data_service import DataService from ..utils.validators import validate_platforms from ..utils.errors import MCPError, CrawlTaskError class SystemManagementTools: """系统管理工具类""" def __init__(self, project_root: str = None): """ 初始化系统管理工具 Args: project_root: 项目根目录 """ self.data_service = DataService(project_root) if project_root: self.project_root = Path(project_root) else: # 获取项目根目录 current_file = Path(__file__) self.project_root = current_file.parent.parent.parent def get_system_status(self) -> Dict: """ 获取系统运行状态和健康检查信息 Returns: 系统状态字典 Example: >>> tools = SystemManagementTools() >>> result = tools.get_system_status() >>> print(result['system']['version']) """ try: # 获取系统状态 status = self.data_service.get_system_status() return { "success": True, "summary": { "description": "系统运行状态和健康检查信息" }, "data": status } except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } def trigger_crawl(self, platforms: Optional[List[str]] = None, save_to_local: bool = False, include_url: bool = False) -> Dict: """ 手动触发一次临时爬取任务（可选持久化） Args: platforms: 指定平台列表，为空则爬取所有平台 save_to_local: 是否保存到本地 output 目录，默认 False include_url: 是否包含URL链接，默认False（节省token） Returns: 爬取结果字典，包含新闻数据和保存路径（如果保存） Example: >>> tools = SystemManagementTools() >>> # 临时爬取，不保存 >>> result = tools.trigger_crawl(platforms=['zhihu', 'weibo']) >>> print(result['data']) >>> # 爬取并保存到本地 >>> result = tools.trigger_crawl(platforms=['zhihu'], save_to_local=True) >>> print(result['saved_files']) """ try: import time import yaml from trendradar.crawler.fetcher import DataFetcher from trendradar.storage.local import LocalStorageBackend from trendradar.storage.base import convert_crawl_results_to_news_data from trendradar.utils.time import get_configured_time, format_date_folder, format_time_filename from ..services.cache_service import get_cache # 参数验证 platforms = validate_platforms(platforms) # 加载配置文件 config_path = self.project_root / "config" / "config.yaml" if not config_path.exists(): raise CrawlTaskError( "配置文件不存在", suggestion=f"请确保配置文件存在: {config_path}" ) # 读取配置 with open(config_path, "r", encoding="utf-8") as f: config_data = yaml.safe_load(f) # 获取平台配置（嵌套结构：{enabled: bool, sources: [...]}) platforms_config = config_data.get("platforms", {}) if not platforms_config.get("enabled", True): raise CrawlTaskError( "热榜平台已禁用", suggestion="请检查 config/config.yaml 中的 platforms.enabled 配置" ) all_platforms = platforms_config.get("sources", []) if not all_platforms: raise CrawlTaskError( "配置文件中没有平台配置", suggestion="请检查 config/config.yaml 中的 platforms.sources 配置" ) # 过滤平台 if platforms: target_platforms = [p for p in all_platforms if p["id"] in platforms] if not target_platforms: raise CrawlTaskError( f"指定的平台不存在: {platforms}", suggestion=f"可用平台: {[p['id'] for p in all_platforms]}" ) else: target_platforms = all_platforms # 构建平台ID列表 ids = [] for platform in target_platforms: if "name" in platform: ids.append((platform["id"], platform["name"])) else: ids.append(platform["id"]) print(f"开始临时爬取，平台: {[p.get('name', p['id']) for p in target_platforms]}") # 初始化数据获取器 advanced = config_data.get("advanced", {}) crawler_config = advanced.get("crawler", {}) proxy_url = None if crawler_config.get("use_proxy"): proxy_url = crawler_config.get("default_proxy") fetcher = DataFetcher(proxy_url=proxy_url) request_interval = crawler_config.get("request_interval", 100) # 执行爬取 results, id_to_name, failed_ids = fetcher.crawl_websites( ids_list=ids, request_interval=request_interval ) # 获取当前时间（统一使用 trendradar 的时间工具） # 从配置中读取时区，默认为 Asia/Shanghai timezone = config_data.get("app", {}).get("timezone", "Asia/Shanghai") current_time = get_configured_time(timezone) crawl_date = format_date_folder(None, timezone) crawl_time_str = format_time_filename(timezone) # 转换为标准数据模型 news_data = convert_crawl_results_to_news_data( results=results, id_to_name=id_to_name, failed_ids=failed_ids, crawl_time=crawl_time_str, crawl_date=crawl_date ) # 初始化存储后端 storage = LocalStorageBackend( data_dir=str(self.project_root / "output"), enable_txt=True, enable_html=True, timezone=timezone ) # 尝试持久化数据 save_success = False save_error_msg = "" saved_files = {} try: # 1. 保存到 SQLite (核心持久化) if storage.save_news_data(news_data): save_success = True # 2. 如果请求保存到本地，生成 TXT/HTML 快照 if save_to_local: # 保存 TXT txt_path = storage.save_txt_snapshot(news_data) if txt_path: saved_files["txt"] = txt_path # 保存 HTML (使用简化版生成器) html_content = self._generate_simple_html(results, id_to_name, failed_ids, current_time) html_filename = f"{crawl_time_str}.html" html_path = storage.save_html_report(html_content, html_filename) if html_path: saved_files["html"] = html_path except Exception as e: # 捕获所有保存错误（特别是 Docker 只读卷导致的 PermissionError） print(f"[System] 数据保存失败: {e}") save_success = False save_error_msg = str(e) # 3. 清除缓存，确保下次查询获取最新数据 # 即使保存失败，内存中的数据可能已经通过其他方式更新，或者是临时的 get_cache().clear() print("[System] 缓存已清除") # 构建返回结果 news_response_data = [] for platform_id, titles_data in results.items(): platform_name = id_to_name.get(platform_id, platform_id) for title, info in titles_data.items(): news_item = { "platform_id": platform_id, "platform_name": platform_name, "title": title, "ranks": info.get("ranks", []) } if include_url: news_item["url"] = info.get("url", "") news_item["mobile_url"] = info.get("mobileUrl", "") news_response_data.append(news_item) result = { "success": True, "summary": { "description": "爬取任务执行结果", "task_id": f"crawl_{int(time.time())}", "status": "completed", "crawl_time": current_time.strftime("%Y-%m-%d %H:%M:%S"), "total_news": len(news_response_data), "platforms": list(results.keys()), "failed_platforms": failed_ids, "saved_to_local": save_success and save_to_local }, "data": news_response_data } if save_success: if save_to_local: result["saved_files"] = saved_files result["note"] = "数据已保存到 SQLite 数据库及 output 文件夹" else: result["note"] = "数据已保存到 SQLite 数据库 (仅内存中返回结果，未生成TXT快照)" else: # 明确告知用户保存失败 result["saved_to_local"] = False result["save_error"] = save_error_msg if "Read-only file system" in save_error_msg or "Permission denied" in save_error_msg: result["note"] = "爬取成功，但无法写入数据库（Docker只读模式）。数据仅在本次返回中有效。" else: result["note"] = f"爬取成功但保存失败: {save_error_msg}" # 清理资源 storage.cleanup() return result except MCPError as e: return { "success": False, "error": e.to_dict() } except Exception as e: import traceback return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e), "traceback": traceback.format_exc() } } def _generate_simple_html(self, results: Dict, id_to_name: Dict, failed_ids: List, now) -> str: """生成简化的 HTML 报告""" html = """ MCP 爬取结果

MCP 爬取结果

""" # 添加时间戳 html += f'

爬取时间: {now.strftime("%Y-%m-%d %H:%M:%S")}

\n\n' # 遍历每个平台 for platform_id, titles_data in results.items(): platform_name = id_to_name.get(platform_id, platform_id) html += f'
\n' html += f'
{platform_name}
\n' # 排序标题 sorted_items = [] for title, info in titles_data.items(): ranks = info.get("ranks", []) url = info.get("url", "") mobile_url = info.get("mobileUrl", "") rank = ranks[0] if ranks else 999 sorted_items.append((rank, title, url, mobile_url)) sorted_items.sort(key=lambda x: x[0]) # 显示新闻 for rank, title, url, mobile_url in sorted_items: html += f'
\n' html += f' {rank}.\n' html += f' {self._html_escape(title)}\n' if url: html += f' 链接\n' if mobile_url and mobile_url != url: html += f' 移动版\n' html += '
\n' html += '
\n\n' # 失败的平台 if failed_ids: html += '
\n' html += '

请求失败的平台

\n' html += '
    \n' for platform_id in failed_ids: html += f'
  • {self._html_escape(platform_id)}
    • \n' html += '
\n' html += '
\n' html += """
""" return html def _html_escape(self, text: str) -> str: """HTML 转义""" if not isinstance(text, str): text = str(text) return ( text.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) def check_version(self, proxy_url: Optional[str] = None) -> Dict: """ 检查版本更新 同时检查 TrendRadar 和 MCP Server 两个组件的版本更新。 远程版本 URL 从 config.yaml 获取： - version_check_url: TrendRadar 版本 - mcp_version_check_url: MCP Server 版本 Args: proxy_url: 可选的代理URL，用于访问远程版本 Returns: 版本检查结果字典，包含： - success: 是否成功 - trendradar: TrendRadar 版本检查结果 - mcp: MCP Server 版本检查结果 - any_update: 是否有任何组件需要更新 Example: >>> tools = SystemManagementTools() >>> result = tools.check_version() >>> print(result['data']['any_update']) """ import yaml import requests def parse_version(version_str: str): """将版本号字符串解析为元组""" try: parts = version_str.strip().split(".") if len(parts) != 3: raise ValueError("版本号格式不正确") return int(parts[0]), int(parts[1]), int(parts[2]) except: return 0, 0, 0 def check_single_version( name: str, local_version: str, remote_url: str, proxies: Optional[Dict], headers: Dict ) -> Dict: """检查单个组件的版本""" try: response = requests.get( remote_url, proxies=proxies, headers=headers, timeout=10 ) response.raise_for_status() remote_version = response.text.strip() local_tuple = parse_version(local_version) remote_tuple = parse_version(remote_version) need_update = local_tuple < remote_tuple if need_update: message = f"发现新版本 {remote_version}，当前版本 {local_version}，建议更新" elif local_tuple > remote_tuple: message = f"当前版本 {local_version} 高于远程版本 {remote_version}（可能是开发版本）" else: message = f"当前版本 {local_version} 已是最新版本" return { "success": True, "name": name, "current_version": local_version, "remote_version": remote_version, "need_update": need_update, "current_parsed": list(local_tuple), "remote_parsed": list(remote_tuple), "message": message } except requests.exceptions.Timeout: return { "success": False, "name": name, "current_version": local_version, "error": "获取远程版本超时" } except requests.exceptions.RequestException as e: return { "success": False, "name": name, "current_version": local_version, "error": f"网络请求失败: {str(e)}" } except Exception as e: return { "success": False, "name": name, "current_version": local_version, "error": str(e) } try: # 导入本地版本 from trendradar import __version__ as trendradar_version from mcp_server import __version__ as mcp_version # 从配置文件获取远程版本 URL config_path = self.project_root / "config" / "config.yaml" if not config_path.exists(): return { "success": False, "error": { "code": "CONFIG_NOT_FOUND", "message": f"配置文件不存在: {config_path}" } } with open(config_path, "r", encoding="utf-8") as f: config_data = yaml.safe_load(f) advanced_config = config_data.get("advanced", {}) trendradar_url = advanced_config.get( "version_check_url", "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version" ) mcp_url = advanced_config.get( "mcp_version_check_url", "https://raw.githubusercontent.com/sansan0/TrendRadar/refs/heads/master/version_mcp" ) # 配置代理 proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 请求头 headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", "Accept": "text/plain, */*", "Cache-Control": "no-cache", } # 检查两个版本 trendradar_result = check_single_version( "TrendRadar", trendradar_version, trendradar_url, proxies, headers ) mcp_result = check_single_version( "MCP Server", mcp_version, mcp_url, proxies, headers ) # 判断是否有任何更新 any_update = ( (trendradar_result.get("success") and trendradar_result.get("need_update", False)) or (mcp_result.get("success") and mcp_result.get("need_update", False)) ) return { "success": True, "summary": { "description": "版本检查结果（TrendRadar + MCP Server）", "any_update": any_update }, "data": { "trendradar": trendradar_result, "mcp": mcp_result, "any_update": any_update } } except ImportError as e: return { "success": False, "error": { "code": "IMPORT_ERROR", "message": f"无法导入版本信息: {str(e)}" } } except Exception as e: return { "success": False, "error": { "code": "INTERNAL_ERROR", "message": str(e) } } ================================================ FILE: mcp_server/utils/__init__.py ================================================ """ 工具类模块 提供参数验证、错误处理等辅助功能。 """ ================================================ FILE: mcp_server/utils/date_parser.py ================================================ """ 日期解析工具 支持多种自然语言日期格式解析，包括相对日期和绝对日期。 """ import re from datetime import datetime, timedelta from typing import Tuple, Dict, Optional from .errors import InvalidParameterError class DateParser: """日期解析器类""" # 中文日期映射 CN_DATE_MAPPING = { "今天": 0, "昨天": 1, "前天": 2, "大前天": 3, } # 英文日期映射 EN_DATE_MAPPING = { "today": 0, "yesterday": 1, } # 日期范围表达式（用于 resolve_date_range_expression） RANGE_EXPRESSIONS = { # 中文表达式 "今天": "today", "昨天": "yesterday", "本周": "this_week", "这周": "this_week", "当前周": "this_week", "上周": "last_week", "本月": "this_month", "这个月": "this_month", "当前月": "this_month", "上月": "last_month", "上个月": "last_month", "最近3天": "last_3_days", "近3天": "last_3_days", "最近7天": "last_7_days", "近7天": "last_7_days", "最近一周": "last_7_days", "过去一周": "last_7_days", "最近14天": "last_14_days", "近14天": "last_14_days", "最近两周": "last_14_days", "过去两周": "last_14_days", "最近30天": "last_30_days", "近30天": "last_30_days", "最近一个月": "last_30_days", "过去一个月": "last_30_days", # 英文表达式 "today": "today", "yesterday": "yesterday", "this week": "this_week", "current week": "this_week", "last week": "last_week", "this month": "this_month", "current month": "this_month", "last month": "last_month", "last 3 days": "last_3_days", "past 3 days": "last_3_days", "last 7 days": "last_7_days", "past 7 days": "last_7_days", "past week": "last_7_days", "last 14 days": "last_14_days", "past 14 days": "last_14_days", "last 30 days": "last_30_days", "past 30 days": "last_30_days", "past month": "last_30_days", } # 星期映射 WEEKDAY_CN = { "一": 0, "二": 1, "三": 2, "四": 3, "五": 4, "六": 5, "日": 6, "天": 6 } WEEKDAY_EN = { "monday": 0, "tuesday": 1, "wednesday": 2, "thursday": 3, "friday": 4, "saturday": 5, "sunday": 6 } @staticmethod def parse_date_query(date_query: str) -> datetime: """ 解析日期查询字符串 支持的格式： - 相对日期（中文）：今天、昨天、前天、大前天、N天前 - 相对日期（英文）：today、yesterday、N days ago - 星期（中文）：上周一、上周二、本周三 - 星期（英文）：last monday、this friday - 绝对日期：2025-10-10、10月10日、2025年10月10日 Args: date_query: 日期查询字符串 Returns: datetime对象 Raises: InvalidParameterError: 日期格式无法识别 Examples: >>> DateParser.parse_date_query("今天") datetime(2025, 10, 11) >>> DateParser.parse_date_query("昨天") datetime(2025, 10, 10) >>> DateParser.parse_date_query("3天前") datetime(2025, 10, 8) >>> DateParser.parse_date_query("2025-10-10") datetime(2025, 10, 10) """ if not date_query or not isinstance(date_query, str): raise InvalidParameterError( "日期查询字符串不能为空", suggestion="请提供有效的日期查询，如：今天、昨天、2025-10-10" ) date_query = date_query.strip().lower() # 1. 尝试解析中文常用相对日期 if date_query in DateParser.CN_DATE_MAPPING: days_ago = DateParser.CN_DATE_MAPPING[date_query] return datetime.now() - timedelta(days=days_ago) # 2. 尝试解析英文常用相对日期 if date_query in DateParser.EN_DATE_MAPPING: days_ago = DateParser.EN_DATE_MAPPING[date_query] return datetime.now() - timedelta(days=days_ago) # 3. 尝试解析 "N天前" 或 "N days ago" cn_days_ago_match = re.match(r'(\d+)\s*天前', date_query) if cn_days_ago_match: days = int(cn_days_ago_match.group(1)) if days > 365: raise InvalidParameterError( f"天数过大: {days}天", suggestion="请使用小于365天的相对日期或使用绝对日期" ) return datetime.now() - timedelta(days=days) en_days_ago_match = re.match(r'(\d+)\s*days?\s+ago', date_query) if en_days_ago_match: days = int(en_days_ago_match.group(1)) if days > 365: raise InvalidParameterError( f"天数过大: {days}天", suggestion="请使用小于365天的相对日期或使用绝对日期" ) return datetime.now() - timedelta(days=days) # 4. 尝试解析星期（中文）：上周一、本周三 cn_weekday_match = re.match(r'(上|本)周([一二三四五六日天])', date_query) if cn_weekday_match: week_type = cn_weekday_match.group(1) # 上 或 本 weekday_str = cn_weekday_match.group(2) target_weekday = DateParser.WEEKDAY_CN[weekday_str] return DateParser._get_date_by_weekday(target_weekday, week_type == "上") # 5. 尝试解析星期（英文）：last monday、this friday en_weekday_match = re.match(r'(last|this)\s+(monday|tuesday|wednesday|thursday|friday|saturday|sunday)', date_query) if en_weekday_match: week_type = en_weekday_match.group(1) # last 或 this weekday_str = en_weekday_match.group(2) target_weekday = DateParser.WEEKDAY_EN[weekday_str] return DateParser._get_date_by_weekday(target_weekday, week_type == "last") # 6. 尝试解析绝对日期：YYYY-MM-DD iso_date_match = re.match(r'(\d{4})-(\d{1,2})-(\d{1,2})', date_query) if iso_date_match: year = int(iso_date_match.group(1)) month = int(iso_date_match.group(2)) day = int(iso_date_match.group(3)) try: return datetime(year, month, day) except ValueError as e: raise InvalidParameterError( f"无效的日期: {date_query}", suggestion=f"日期值错误: {str(e)}" ) # 7. 尝试解析中文日期：MM月DD日 或 YYYY年MM月DD日 cn_date_match = re.match(r'(?:(\d{4})年)?(\d{1,2})月(\d{1,2})日', date_query) if cn_date_match: year_str = cn_date_match.group(1) month = int(cn_date_match.group(2)) day = int(cn_date_match.group(3)) # 如果没有年份，使用当前年份 if year_str: year = int(year_str) else: year = datetime.now().year # 如果月份大于当前月份，说明是去年 current_month = datetime.now().month if month > current_month: year -= 1 try: return datetime(year, month, day) except ValueError as e: raise InvalidParameterError( f"无效的日期: {date_query}", suggestion=f"日期值错误: {str(e)}" ) # 8. 尝试解析斜杠格式：YYYY/MM/DD 或 MM/DD slash_date_match = re.match(r'(?:(\d{4})/)?(\d{1,2})/(\d{1,2})', date_query) if slash_date_match: year_str = slash_date_match.group(1) month = int(slash_date_match.group(2)) day = int(slash_date_match.group(3)) if year_str: year = int(year_str) else: year = datetime.now().year current_month = datetime.now().month if month > current_month: year -= 1 try: return datetime(year, month, day) except ValueError as e: raise InvalidParameterError( f"无效的日期: {date_query}", suggestion=f"日期值错误: {str(e)}" ) # 如果所有格式都不匹配 raise InvalidParameterError( f"无法识别的日期格式: {date_query}", suggestion=( "支持的格式:\n" "- 相对日期: 今天、昨天、前天、3天前、today、yesterday、3 days ago\n" "- 星期: 上周一、本周三、last monday、this friday\n" "- 绝对日期: 2025-10-10、10月10日、2025年10月10日" ) ) @staticmethod def _get_date_by_weekday(target_weekday: int, is_last_week: bool) -> datetime: """ 根据星期几获取日期 Args: target_weekday: 目标星期 (0=周一, 6=周日) is_last_week: 是否是上周 Returns: datetime对象 """ today = datetime.now() current_weekday = today.weekday() # 计算天数差 if is_last_week: # 上周的某一天 days_diff = current_weekday - target_weekday + 7 else: # 本周的某一天 days_diff = current_weekday - target_weekday if days_diff < 0: days_diff += 7 return today - timedelta(days=days_diff) @staticmethod def format_date_folder(date: datetime) -> str: """ 将日期格式化为文件夹名称 Args: date: datetime对象 Returns: 文件夹名称，格式: YYYY-MM-DD Examples: >>> DateParser.format_date_folder(datetime(2025, 10, 11)) '2025-10-11' """ return date.strftime("%Y-%m-%d") @staticmethod def validate_date_not_future(date: datetime) -> None: """ 验证日期不在未来 Args: date: 待验证的日期 Raises: InvalidParameterError: 日期在未来 """ if date.date() > datetime.now().date(): raise InvalidParameterError( f"不能查询未来的日期: {date.strftime('%Y-%m-%d')}", suggestion="请使用今天或过去的日期" ) @staticmethod def validate_date_not_too_old(date: datetime, max_days: int = 365) -> None: """ 验证日期不太久远 Args: date: 待验证的日期 max_days: 最大天数 Raises: InvalidParameterError: 日期太久远 """ days_ago = (datetime.now().date() - date.date()).days if days_ago > max_days: raise InvalidParameterError( f"日期太久远: {date.strftime('%Y-%m-%d')} ({days_ago}天前)", suggestion=f"请查询{max_days}天内的数据" ) @staticmethod def resolve_date_range_expression(expression: str) -> Dict: """ 将自然语言日期表达式解析为标准日期范围 这是专门为 MCP 工具设计的方法，用于在服务器端解析日期表达式， 避免 AI 模型自己计算日期导致的不一致问题。 Args: expression: 自然语言日期表达式，支持： - 单日: "今天", "昨天", "today", "yesterday" - 本周/上周: "本周", "上周", "this week", "last week" - 本月/上月: "本月", "上月", "this month", "last month" - 最近N天: "最近7天", "最近30天", "last 7 days", "last 30 days" - 动态N天: "最近5天", "last 10 days" Returns: 解析结果字典： { "success": True, "expression": "本周", "normalized": "this_week", "date_range": { "start": "2025-11-18", "end": "2025-11-24" }, "current_date": "2025-11-26", "description": "本周（周一到周日）" } Raises: InvalidParameterError: 无法识别的日期表达式 Examples: >>> DateParser.resolve_date_range_expression("本周") {"success": True, "date_range": {"start": "2025-11-18", "end": "2025-11-24"}, ...} >>> DateParser.resolve_date_range_expression("最近7天") {"success": True, "date_range": {"start": "2025-11-20", "end": "2025-11-26"}, ...} """ if not expression or not isinstance(expression, str): raise InvalidParameterError( "日期表达式不能为空", suggestion="请提供有效的日期表达式，如：本周、最近7天、last week" ) expression_lower = expression.strip().lower() today = datetime.now() today_str = today.strftime("%Y-%m-%d") # 1. 尝试匹配预定义表达式 normalized = DateParser.RANGE_EXPRESSIONS.get(expression_lower) # 2. 尝试匹配动态 "最近N天" / "last N days" 模式 if not normalized: # 中文: 最近N天 cn_match = re.match(r'最近(\d+)天', expression_lower) if cn_match: days = int(cn_match.group(1)) normalized = f"last_{days}_days" # 英文: last N days en_match = re.match(r'(?:last|past)\s+(\d+)\s+days?', expression_lower) if en_match: days = int(en_match.group(1)) normalized = f"last_{days}_days" if not normalized: # 提供支持的表达式列表 supported_cn = ["今天", "昨天", "本周", "上周", "本月", "上月", "最近7天", "最近30天", "最近N天"] supported_en = ["today", "yesterday", "this week", "last week", "this month", "last month", "last 7 days", "last N days"] raise InvalidParameterError( f"无法识别的日期表达式: {expression}", suggestion=f"支持的表达式:\n中文: {', '.join(supported_cn)}\n英文: {', '.join(supported_en)}" ) # 3. 根据 normalized 类型计算日期范围 start_date, end_date, description = DateParser._calculate_date_range( normalized, today ) return { "success": True, "expression": expression, "normalized": normalized, "date_range": { "start": start_date.strftime("%Y-%m-%d"), "end": end_date.strftime("%Y-%m-%d") }, "current_date": today_str, "description": description } @staticmethod def _calculate_date_range( normalized: str, today: datetime ) -> Tuple[datetime, datetime, str]: """ 根据标准化的日期类型计算实际日期范围 Args: normalized: 标准化的日期类型 today: 当前日期 Returns: (start_date, end_date, description) 元组 """ # 单日类型 if normalized == "today": return today, today, "今天" if normalized == "yesterday": yesterday = today - timedelta(days=1) return yesterday, yesterday, "昨天" # 本周（周一到周日） if normalized == "this_week": # 计算本周一 weekday = today.weekday() # 0=周一, 6=周日 start = today - timedelta(days=weekday) end = start + timedelta(days=6) # 如果本周还没结束，end 不能超过今天 if end > today: end = today return start, end, f"本周（周一到周日，{start.strftime('%m-%d')} 至 {end.strftime('%m-%d')}）" # 上周（上周一到上周日） if normalized == "last_week": weekday = today.weekday() # 本周一 this_monday = today - timedelta(days=weekday) # 上周一 start = this_monday - timedelta(days=7) end = start + timedelta(days=6) return start, end, f"上周（{start.strftime('%m-%d')} 至 {end.strftime('%m-%d')}）" # 本月（本月1日到今天） if normalized == "this_month": start = today.replace(day=1) return start, today, f"本月（{start.strftime('%m-%d')} 至 {today.strftime('%m-%d')}）" # 上月（上月1日到上月最后一天） if normalized == "last_month": # 上月最后一天 = 本月1日 - 1天 first_of_this_month = today.replace(day=1) end = first_of_this_month - timedelta(days=1) start = end.replace(day=1) return start, end, f"上月（{start.strftime('%Y-%m-%d')} 至 {end.strftime('%Y-%m-%d')}）" # 最近N天 (last_N_days 格式) match = re.match(r'last_(\d+)_days', normalized) if match: days = int(match.group(1)) start = today - timedelta(days=days - 1) # 包含今天，所以是 days-1 return start, today, f"最近{days}天（{start.strftime('%m-%d')} 至 {today.strftime('%m-%d')}）" # 兜底：返回今天 return today, today, "今天（默认）" @staticmethod def get_supported_expressions() -> Dict[str, list]: """ 获取支持的日期表达式列表 Returns: 分类的表达式列表 """ return { "单日": ["今天", "昨天", "today", "yesterday"], "周": ["本周", "上周", "this week", "last week"], "月": ["本月", "上月", "this month", "last month"], "最近N天": ["最近3天", "最近7天", "最近14天", "最近30天", "last 3 days", "last 7 days", "last 14 days", "last 30 days"], "动态天数": ["最近N天", "last N days"] } ================================================ FILE: mcp_server/utils/errors.py ================================================ """ 自定义错误类 定义MCP Server使用的所有自定义异常类型。 """ from typing import Optional, List, Callable # ==================== 延迟加载支持的平台列表 ==================== _get_supported_platforms: Optional[Callable[[], List[str]]] = None def _load_supported_platforms() -> List[str]: """延迟加载支持的平台列表""" global _get_supported_platforms if _get_supported_platforms is None: try: from .validators import get_supported_platforms _get_supported_platforms = get_supported_platforms except ImportError: # 降级：返回空列表 return [] return _get_supported_platforms() class MCPError(Exception): """MCP工具错误基类""" def __init__(self, message: str, code: str = "MCP_ERROR", suggestion: Optional[str] = None): super().__init__(message) self.code = code self.message = message self.suggestion = suggestion def to_dict(self) -> dict: """转换为字典格式""" error_dict = { "code": self.code, "message": self.message } if self.suggestion: error_dict["suggestion"] = self.suggestion return error_dict class DataNotFoundError(MCPError): """数据不存在错误""" def __init__(self, message: str, suggestion: Optional[str] = None): super().__init__( message=message, code="DATA_NOT_FOUND", suggestion=suggestion or "请检查日期范围或等待爬取任务完成" ) class InvalidParameterError(MCPError): """参数无效错误""" def __init__(self, message: str, suggestion: Optional[str] = None): super().__init__( message=message, code="INVALID_PARAMETER", suggestion=suggestion or "请检查参数格式是否正确" ) class ConfigurationError(MCPError): """配置错误""" def __init__(self, message: str, suggestion: Optional[str] = None): super().__init__( message=message, code="CONFIGURATION_ERROR", suggestion=suggestion or "请检查配置文件是否正确" ) class PlatformNotSupportedError(MCPError): """平台不支持错误""" def __init__(self, platform: str): supported = _load_supported_platforms() suggestion = f"支持的平台: {', '.join(supported)}" if supported else "请检查 config/config.yaml 中的平台配置" super().__init__( message=f"平台 '{platform}' 不受支持", code="PLATFORM_NOT_SUPPORTED", suggestion=suggestion ) class CrawlTaskError(MCPError): """爬取任务错误""" def __init__(self, message: str, suggestion: Optional[str] = None): super().__init__( message=message, code="CRAWL_TASK_ERROR", suggestion=suggestion or "请稍后重试或查看日志" ) class FileParseError(MCPError): """文件解析错误""" def __init__(self, file_path: str, reason: str): super().__init__( message=f"解析文件 {file_path} 失败: {reason}", code="FILE_PARSE_ERROR", suggestion="请检查文件格式是否正确" ) ================================================ FILE: mcp_server/utils/validators.py ================================================ """ 参数验证工具 提供统一的参数验证功能。 支持 MCP 客户端将参数序列化为字符串的情况。 """ from datetime import datetime from typing import List, Optional, Union import os import json import yaml import ast from .errors import InvalidParameterError from .date_parser import DateParser # ==================== 辅助函数：处理字符串序列化 ==================== def _parse_string_to_list(value: str) -> List[str]: """ 将字符串解析为列表 支持格式： - JSON 数组: '["zhihu", "weibo"]' - Python 列表字符串: "['zhihu', 'weibo']" - 逗号分隔: "zhihu, weibo" 或 "zhihu,weibo" Args: value: 字符串值 Returns: 解析后的列表 Raises: InvalidParameterError: 解析失败 """ value = value.strip() if not value: return [] # 尝试 JSON 解析: '["zhihu", "weibo"]' try: parsed = json.loads(value) if isinstance(parsed, list): return [str(item) for item in parsed] # 如果解析结果不是列表，继续尝试其他方式 except json.JSONDecodeError: pass # 尝试 Python 字面量解析: "['zhihu', 'weibo']" try: parsed = ast.literal_eval(value) if isinstance(parsed, list): return [str(item) for item in parsed] if isinstance(parsed, str): # 单个字符串，包装成列表 return [parsed] except (ValueError, SyntaxError): pass # 尝试逗号分隔: "zhihu, weibo" 或 "zhihu,weibo" if ',' in value: items = [item.strip() for item in value.split(',')] return [item for item in items if item] # 单个值 return [value] def _parse_string_to_int(value: str, param_name: str = "参数") -> int: """ 将字符串解析为整数 Args: value: 字符串值 param_name: 参数名（用于错误消息） Returns: 解析后的整数 Raises: InvalidParameterError: 解析失败 """ value = value.strip() try: # 尝试直接转换 return int(value) except ValueError: pass # 尝试解析浮点数后取整 try: return int(float(value)) except ValueError: raise InvalidParameterError( f"{param_name} 必须是整数，无法解析: {value}", suggestion=f"请提供有效的整数值，如: 10, 50, 100" ) def _parse_string_to_float(value: str, param_name: str = "参数") -> float: """ 将字符串解析为浮点数 Args: value: 字符串值 param_name: 参数名（用于错误消息） Returns: 解析后的浮点数 Raises: InvalidParameterError: 解析失败 """ value = value.strip() try: return float(value) except ValueError: raise InvalidParameterError( f"{param_name} 必须是数字，无法解析: {value}", suggestion=f"请提供有效的数字值，如: 0.6, 3.0" ) def _parse_string_to_bool(value: str) -> bool: """ 将字符串解析为布尔值 Args: value: 字符串值 Returns: 解析后的布尔值 """ value = value.strip().lower() if value in ('true', '1', 'yes', 'on'): return True elif value in ('false', '0', 'no', 'off', ''): return False else: # 默认非空字符串为 True return bool(value) # 平台列表 mtime 缓存（避免每次 MCP 调用都重新读取 config.yaml） _platforms_cache: Optional[List[str]] = None _platforms_config_mtime: float = 0.0 _platforms_config_path: Optional[str] = None def get_supported_platforms() -> List[str]: """ 从 config.yaml 动态获取支持的平台列表（带 mtime 缓存） 仅当 config.yaml 被修改时才重新读取，避免每次 MCP 调用的重复 IO。 Returns: 平台ID列表 Note: - 读取失败时返回空列表，允许所有平台通过（降级策略） - 平台列表来自 config/config.yaml 中的 platforms 配置 """ global _platforms_cache, _platforms_config_mtime, _platforms_config_path try: if _platforms_config_path is None: current_dir = os.path.dirname(os.path.abspath(__file__)) _platforms_config_path = os.path.normpath( os.path.join(current_dir, "..", "..", "config", "config.yaml") ) current_mtime = os.path.getmtime(_platforms_config_path) if _platforms_cache is not None and current_mtime == _platforms_config_mtime: return _platforms_cache with open(_platforms_config_path, 'r', encoding='utf-8') as f: config = yaml.safe_load(f) platforms_config = config.get('platforms', {}) sources = platforms_config.get('sources', []) _platforms_cache = [p['id'] for p in sources if 'id' in p] _platforms_config_mtime = current_mtime return _platforms_cache except Exception as e: print(f"警告：无法加载平台配置: {e}") return [] def validate_platforms(platforms: Optional[Union[List[str], str]]) -> List[str]: """ 验证平台列表 Args: platforms: 平台ID列表或字符串，None表示使用 config.yaml 中配置的所有平台 支持多种格式： - None: 使用默认平台 - ["zhihu", "weibo"]: JSON 数组 - '["zhihu", "weibo"]': JSON 数组字符串 - "['zhihu', 'weibo']": Python 列表字符串 - "zhihu, weibo": 逗号分隔字符串 - "zhihu": 单个平台字符串 Returns: 验证后的平台列表 Raises: InvalidParameterError: 平台不支持 Note: - platforms=None 时，返回 config.yaml 中配置的平台列表 - 会验证平台ID是否在 config.yaml 的 platforms 配置中 - 配置加载失败时，允许所有平台通过（降级策略） """ supported_platforms = get_supported_platforms() if platforms is None: # 返回配置文件中的平台列表（用户的默认配置） return supported_platforms if supported_platforms else [] # 支持字符串形式的列表输入（某些 MCP 客户端会将 JSON 数组序列化为字符串） if isinstance(platforms, str): platforms = _parse_string_to_list(platforms) if not platforms: # 空字符串或解析后为空，使用默认平台 return supported_platforms if supported_platforms else [] if not isinstance(platforms, list): raise InvalidParameterError("platforms 参数必须是列表类型") if not platforms: # 空列表时，返回配置文件中的平台列表 return supported_platforms if supported_platforms else [] # 如果配置加载失败（supported_platforms为空），允许所有平台通过 if not supported_platforms: print("警告：平台配置未加载，跳过平台验证") return platforms # 验证每个平台是否在配置中 invalid_platforms = [p for p in platforms if p not in supported_platforms] if invalid_platforms: raise InvalidParameterError( f"不支持的平台: {', '.join(invalid_platforms)}", suggestion=f"支持的平台（来自config.yaml）: {', '.join(supported_platforms)}" ) return platforms def validate_limit(limit: Optional[Union[int, str]], default: int = 20, max_limit: int = 1000) -> int: """ 验证数量限制参数 Args: limit: 限制数量（整数或字符串） default: 默认值 max_limit: 最大限制 Returns: 验证后的限制值 Raises: InvalidParameterError: 参数无效 """ if limit is None: return default # 支持字符串形式的整数（某些 MCP 客户端会将数字序列化为字符串） if isinstance(limit, str): limit = _parse_string_to_int(limit, "limit") if not isinstance(limit, int): raise InvalidParameterError("limit 参数必须是整数类型") if limit <= 0: raise InvalidParameterError("limit 必须大于0") if limit > max_limit: raise InvalidParameterError( f"limit 不能超过 {max_limit}", suggestion=f"请使用分页或降低limit值" ) return limit def validate_date(date_str: str) -> datetime: """ 验证日期格式 Args: date_str: 日期字符串 (YYYY-MM-DD) Returns: datetime对象 Raises: InvalidParameterError: 日期格式错误 """ try: return datetime.strptime(date_str, "%Y-%m-%d") except ValueError: raise InvalidParameterError( f"日期格式错误: {date_str}", suggestion="请使用 YYYY-MM-DD 格式，例如: 2025-10-11" ) def normalize_date_range(date_range: Optional[Union[dict, str]]) -> Optional[Union[dict, str]]: """ 规范化 date_range 参数 某些 MCP 客户端（特别是 HTTP 方式）会将 JSON 对象序列化为字符串传入。 此函数尝试将 JSON 字符串解析为 dict，如果不是 JSON 格式则保持原样。 Args: date_range: 日期范围，可能是: - dict: {"start": "2025-01-01", "end": "2025-01-07"} - JSON 字符串: '{"start": "2025-01-01", "end": "2025-01-07"}' - 普通字符串: "今天", "昨天", "2025-01-01" - None Returns: 规范化后的 date_range（dict 或普通字符串） Examples: >>> normalize_date_range('{"start":"2025-01-01","end":"2025-01-07"}') {"start": "2025-01-01", "end": "2025-01-07"} >>> normalize_date_range("今天") "今天" >>> normalize_date_range({"start": "2025-01-01", "end": "2025-01-07"}) {"start": "2025-01-01", "end": "2025-01-07"} """ if date_range is None: return None # 如果已经是 dict，直接返回 if isinstance(date_range, dict): return date_range # 如果是字符串，尝试解析为 JSON if isinstance(date_range, str): # 检查是否看起来像 JSON 对象 stripped = date_range.strip() if stripped.startswith('{') and stripped.endswith('}'): try: parsed = json.loads(stripped) if isinstance(parsed, dict): return parsed except json.JSONDecodeError: pass # 解析失败，当作普通字符串处理 return date_range def validate_date_range(date_range: Optional[Union[dict, str]]) -> Optional[tuple]: """ 验证日期范围 Args: date_range: 日期范围，支持多种格式： - dict: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"} - JSON 字符串: '{"start": "2025-01-01", "end": "2025-01-07"}' - 单日字符串: "2025-01-01"（自动转为同一天的范围） - 自然语言: "今天", "昨天", "本周", "最近7天" 等 Returns: (start_date, end_date) 元组，或 None Raises: InvalidParameterError: 日期范围无效 """ if date_range is None: return None # 支持字符串形式的输入 if isinstance(date_range, str): stripped = date_range.strip() # 1. 检查是否是 JSON 对象格式 if stripped.startswith('{') and stripped.endswith('}'): try: date_range = json.loads(stripped) except json.JSONDecodeError as e: raise InvalidParameterError( f"date_range JSON 解析失败: {e}", suggestion='请使用正确的JSON格式: {"start": "YYYY-MM-DD", "end": "YYYY-MM-DD"}' ) # 2. 检查是否是单日字符串格式 YYYY-MM-DD elif len(stripped) == 10 and stripped[4] == '-' and stripped[7] == '-': try: single_date = datetime.strptime(stripped, "%Y-%m-%d") return (single_date, single_date) except ValueError: raise InvalidParameterError( f"日期格式错误: {stripped}", suggestion="请使用 YYYY-MM-DD 格式，例如: 2025-10-11" ) # 3. 尝试自然语言解析 else: try: result = DateParser.resolve_date_range_expression(stripped) if result.get("success"): dr = result["date_range"] start_date = datetime.strptime(dr["start"], "%Y-%m-%d") end_date = datetime.strptime(dr["end"], "%Y-%m-%d") return (start_date, end_date) else: raise InvalidParameterError( f"无法识别的日期表达式: {stripped}", suggestion="支持格式: YYYY-MM-DD, {\"start\": \"...\", \"end\": \"...\"}, 或自然语言（今天、本周、最近7天等）" ) except InvalidParameterError: raise except Exception: raise InvalidParameterError( f"日期解析失败: {stripped}", suggestion="支持格式: YYYY-MM-DD, {\"start\": \"...\", \"end\": \"...\"}, 或自然语言（今天、本周、最近7天等）" ) if not isinstance(date_range, dict): raise InvalidParameterError( "date_range 必须是字典类型、日期字符串或有效的JSON字符串", suggestion='例如: {"start": "2025-10-01", "end": "2025-10-11"} 或 "2025-10-01"' ) start_str = date_range.get("start") end_str = date_range.get("end") if not start_str or not end_str: raise InvalidParameterError( "date_range 必须包含 start 和 end 字段", suggestion='例如: {"start": "2025-10-01", "end": "2025-10-11"}' ) start_date = validate_date(start_str) end_date = validate_date(end_str) if start_date > end_date: raise InvalidParameterError( "开始日期不能晚于结束日期", suggestion=f"start: {start_str}, end: {end_str}" ) # 检查日期是否在未来 today = datetime.now().date() if start_date.date() > today or end_date.date() > today: # 获取可用日期范围提示 try: from ..services.data_service import DataService data_service = DataService() earliest, latest = data_service.get_available_date_range() if earliest and latest: available_range = f"{earliest.strftime('%Y-%m-%d')} 至 {latest.strftime('%Y-%m-%d')}" else: available_range = "无可用数据" except Exception: available_range = "未知（请检查 output 目录）" future_dates = [] if start_date.date() > today: future_dates.append(start_str) if end_date.date() > today and end_str != start_str: future_dates.append(end_str) raise InvalidParameterError( f"不允许查询未来日期: {', '.join(future_dates)}（当前日期: {today.strftime('%Y-%m-%d')}）", suggestion=f"当前可用数据范围: {available_range}" ) return (start_date, end_date) def validate_keyword(keyword: str) -> str: """ 验证关键词 Args: keyword: 搜索关键词 Returns: 处理后的关键词 Raises: InvalidParameterError: 关键词无效 """ if not keyword: raise InvalidParameterError("keyword 不能为空") if not isinstance(keyword, str): raise InvalidParameterError("keyword 必须是字符串类型") keyword = keyword.strip() if not keyword: raise InvalidParameterError("keyword 不能为空白字符") if len(keyword) > 100: raise InvalidParameterError( "keyword 长度不能超过100个字符", suggestion="请使用更简洁的关键词" ) return keyword def validate_top_n(top_n: Optional[Union[int, str]], default: int = 10) -> int: """ 验证TOP N参数 Args: top_n: TOP N数量（整数或字符串） default: 默认值 Returns: 验证后的值 Raises: InvalidParameterError: 参数无效 """ return validate_limit(top_n, default=default, max_limit=100) def validate_mode(mode: Optional[str], valid_modes: List[str], default: str) -> str: """ 验证模式参数 Args: mode: 模式字符串 valid_modes: 有效模式列表 default: 默认模式 Returns: 验证后的模式 Raises: InvalidParameterError: 模式无效 """ if mode is None: return default if not isinstance(mode, str): raise InvalidParameterError("mode 必须是字符串类型") if mode not in valid_modes: raise InvalidParameterError( f"无效的模式: {mode}", suggestion=f"支持的模式: {', '.join(valid_modes)}" ) return mode def validate_config_section(section: Optional[str]) -> str: """ 验证配置节参数 Args: section: 配置节名称 Returns: 验证后的配置节 Raises: InvalidParameterError: 配置节无效 """ valid_sections = ["all", "crawler", "push", "keywords", "weights"] return validate_mode(section, valid_sections, "all") def validate_threshold( threshold: Optional[Union[float, int, str]], default: float = 0.6, min_value: float = 0.0, max_value: float = 1.0, param_name: str = "threshold" ) -> float: """ 验证阈值参数（浮点数） Args: threshold: 阈值（浮点数、整数或字符串） default: 默认值 min_value: 最小值 max_value: 最大值 param_name: 参数名（用于错误消息） Returns: 验证后的阈值 Raises: InvalidParameterError: 参数无效 """ if threshold is None: return default # 支持字符串形式的数字（某些 MCP 客户端会将数字序列化为字符串） if isinstance(threshold, str): threshold = _parse_string_to_float(threshold, param_name) # 整数转浮点数 if isinstance(threshold, int): threshold = float(threshold) if not isinstance(threshold, float): raise InvalidParameterError( f"{param_name} 必须是数字类型", suggestion=f"请提供 {min_value} 到 {max_value} 之间的数字" ) if threshold < min_value or threshold > max_value: raise InvalidParameterError( f"{param_name} 必须在 {min_value} 到 {max_value} 之间，当前值: {threshold}", suggestion=f"推荐值: {default}" ) return threshold def validate_date_query( date_query: str, allow_future: bool = False, max_days_ago: int = 365 ) -> datetime: """ 验证并解析日期查询字符串 Args: date_query: 日期查询字符串 allow_future: 是否允许未来日期 max_days_ago: 允许查询的最大天数 Returns: 解析后的datetime对象 Raises: InvalidParameterError: 日期查询无效 Examples: >>> validate_date_query("昨天") datetime(2025, 10, 10) >>> validate_date_query("2025-10-10") datetime(2025, 10, 10) """ if not date_query: raise InvalidParameterError( "日期查询字符串不能为空", suggestion="请提供日期查询，如：今天、昨天、2025-10-10" ) # 使用DateParser解析日期 parsed_date = DateParser.parse_date_query(date_query) # 验证日期不在未来 if not allow_future: DateParser.validate_date_not_future(parsed_date) # 验证日期不太久远 DateParser.validate_date_not_too_old(parsed_date, max_days=max_days_ago) return parsed_date ================================================ FILE: pyproject.toml ================================================ [project] name = "trendradar" version = "6.5.0" description = "TrendRadar - 热点新闻聚合与分析工具" requires-python = ">=3.10" dependencies = [ "requests>=2.32.5,<3.0.0", "pytz>=2025.2,<2026.0", "PyYAML>=6.0.3,<7.0.0", "fastmcp>=2.12.0,<2.14.0", "websockets>=13.0,<14.0", "feedparser>=6.0.0,<7.0.0", "boto3>=1.35.0,<2.0.0", "litellm>=1.57.0,<2.0.0", "json-repair>=0.58.3,<1.0.0", "tenacity==8.5.0" ] [project.scripts] trendradar = "trendradar.__main__:main" trendradar-mcp = "mcp_server.server:run_server" [dependency-groups] dev = [] [build-system] requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.wheel] packages = ["trendradar", "mcp_server"] ================================================ FILE: requirements.txt ================================================ requests>=2.32.5,<3.0.0 pytz>=2025.2,<2026.0 PyYAML>=6.0.3,<7.0.0 fastmcp>=2.12.0,<2.14.0 websockets>=13.0,<14.0 boto3>=1.35.0,<2.0.0 feedparser>=6.0.0,<7.0.0 litellm>=1.57.0,<2.0.0 tenacity==8.5.0 ================================================ FILE: setup-mac.sh ================================================ #!/bin/bash # 颜色定义 RED='\033[0;31m' GREEN='\033[0;32m' YELLOW='\033[1;33m' BLUE='\033[0;34m' BOLD='\033[1m' NC='\033[0m' # No Color echo -e "${BOLD}╔════════════════════════════════════════╗${NC}" echo -e "${BOLD}║ TrendRadar MCP 一键部署 (Mac) ║${NC}" echo -e "${BOLD}╚════════════════════════════════════════╝${NC}" echo "" # 获取项目根目录 PROJECT_ROOT="$(cd "$(dirname "$0")" && pwd)" echo -e "📍 项目目录: ${BLUE}${PROJECT_ROOT}${NC}" echo "" # 检查 UV 是否已安装 if ! command -v uv &> /dev/null; then echo -e "${YELLOW}[1/3] 🔧 UV 未安装，正在自动安装...${NC}" echo "提示: UV 是一个快速的 Python 包管理器，只需安装一次" echo "" curl -LsSf https://astral.sh/uv/install.sh | sh echo "" echo "正在刷新 PATH 环境变量..." echo "" # 添加 UV 到 PATH export PATH="$HOME/.cargo/bin:$PATH" # 验证 UV 是否真正可用 if ! command -v uv &> /dev/null; then echo -e "${RED}❌ [错误] UV 安装失败${NC}" echo "" echo "可能的原因：" echo " 1. 网络连接问题，无法下载安装脚本" echo " 2. 安装路径权限不足" echo " 3. 安装脚本执行异常" echo "" echo "解决方案：" echo " 1. 检查网络连接是否正常" echo " 2. 手动安装: https://docs.astral.sh/uv/getting-started/installation/" echo " 3. 或运行: curl -LsSf https://astral.sh/uv/install.sh | sh" exit 1 fi echo -e "${GREEN}✅ [成功] UV 已安装${NC}" echo -e "${YELLOW}⚠️ 请重新运行此脚本以继续${NC}" exit 0 else echo -e "${GREEN}[1/3] ✅ UV 已安装${NC}" uv --version fi echo "" echo "[2/3] 📦 安装项目依赖..." echo "提示: 这可能需要 1-2 分钟，请耐心等待" echo "" # 创建虚拟环境并安装依赖 uv sync if [ $? -ne 0 ]; then echo "" echo -e "${RED}❌ [错误] 依赖安装失败${NC}" echo "请检查网络连接后重试" exit 1 fi echo "" echo -e "${GREEN}[3/3] ✅ 检查配置文件...${NC}" echo "" # 检查配置文件 if [ ! -f "config/config.yaml" ]; then echo -e "${YELLOW}⚠️ [警告] 未找到配置文件: config/config.yaml${NC}" echo "请确保配置文件存在" echo "" fi # 添加执行权限 chmod +x start-http.sh 2>/dev/null || true # 获取 UV 路径 UV_PATH=$(which uv) echo "" echo -e "${BOLD}╔════════════════════════════════════════╗${NC}" echo -e "${BOLD}║ 部署完成！ ║${NC}" echo -e "${BOLD}╚════════════════════════════════════════╝${NC}" echo "" echo "📋 下一步操作:" echo "" echo " 1️⃣ 打开 Cherry Studio" echo " 2️⃣ 进入 设置 > MCP Servers > 添加服务器" echo " 3️⃣ 填入以下配置:" echo "" echo " 名称: TrendRadar" echo " 描述: 新闻热点聚合工具" echo " 类型: STDIO" echo -e " 命令: ${BLUE}${UV_PATH}${NC}" echo " 参数（每个占一行）:" echo -e " ${BLUE}--directory${NC}" echo -e " ${BLUE}${PROJECT_ROOT}${NC}" echo -e " ${BLUE}run${NC}" echo -e " ${BLUE}python${NC}" echo -e " ${BLUE}-m${NC}" echo -e " ${BLUE}mcp_server.server${NC}" echo "" echo " 4️⃣ 保存并启用 MCP 开关" echo "" echo "📖 详细教程请查看: README-Cherry-Studio.md，本窗口别关，待会儿用于填入参数" echo "" ================================================ FILE: setup-windows-en.bat ================================================ @echo off setlocal enabledelayedexpansion echo ========================================== echo TrendRadar MCP Setup (Windows) echo ========================================== echo: REM Fix: Use script location instead of current working directory set "PROJECT_ROOT=%~dp0" REM Remove trailing backslash if "%PROJECT_ROOT:~-1%"=="\" set "PROJECT_ROOT=%PROJECT_ROOT:~0,-1%" echo Project Directory: %PROJECT_ROOT% echo: REM Change to project directory cd /d "%PROJECT_ROOT%" if %errorlevel% neq 0 ( echo [ERROR] Cannot access project directory pause exit /b 1 ) REM Validate project structure echo [0/4] Validating project structure... if not exist "pyproject.toml" ( echo [ERROR] pyproject.toml not found in: %PROJECT_ROOT% echo: echo This should not happen! Please check: echo 1. Is setup-windows.bat in the project root? echo 2. Was the project properly cloned/downloaded? echo: echo Files in current directory: dir /b echo: pause exit /b 1 ) echo [OK] pyproject.toml found echo: REM Check Python echo [1/4] Checking Python... python --version >nul 2>&1 if %errorlevel% neq 0 ( echo [ERROR] Python not detected. Please install Python 3.10+ echo Download: https://www.python.org/downloads/ pause exit /b 1 ) for /f "tokens=*" %%i in ('python --version') do echo [OK] %%i echo: REM Check UV echo [2/4] Checking UV... where uv >nul 2>&1 if %errorlevel% neq 0 ( echo UV not installed, installing automatically... echo: echo Trying installation method 1: PowerShell... powershell -ExecutionPolicy Bypass -Command "try { irm https://astral.sh/uv/install.ps1 | iex; exit 0 } catch { Write-Host 'PowerShell method failed'; exit 1 }" if %errorlevel% neq 0 ( echo: echo Method 1 failed. Trying method 2: pip... python -m pip install --upgrade uv if %errorlevel% neq 0 ( echo: echo [ERROR] Automatic installation failed echo: echo Please install UV manually using one of these methods: echo: echo Method 1 - pip: echo python -m pip install uv echo: echo Method 2 - pipx: echo pip install pipx echo pipx install uv echo: echo Method 3 - Manual download: echo Visit: https://docs.astral.sh/uv/getting-started/installation/ echo: pause exit /b 1 ) ) echo: echo [SUCCESS] UV installed successfully! echo: echo [IMPORTANT] Please restart your terminal: echo 1. Close this window echo 2. Open a new Command Prompt echo 3. Navigate to: %PROJECT_ROOT% echo 4. Run: setup-windows.bat echo: pause exit /b 0 ) else ( for /f "tokens=*" %%i in ('uv --version') do echo [OK] %%i ) echo: echo [3/4] Installing dependencies... echo Working directory: %PROJECT_ROOT% echo: REM Ensure we're in the project directory cd /d "%PROJECT_ROOT%" uv sync if %errorlevel% neq 0 ( echo: echo [ERROR] Dependency installation failed echo: echo Troubleshooting steps: echo 1. Check your internet connection echo 2. Verify Python version ^>= 3.10: python --version echo 3. Try with verbose output: uv sync --verbose echo 4. Check if pyproject.toml is valid echo: echo Project directory: %PROJECT_ROOT% echo: pause exit /b 1 ) echo: echo [OK] Dependencies installed successfully echo: echo [4/4] Checking configuration file... if not exist "config\config.yaml" ( echo [WARNING] config\config.yaml not found if exist "config\config.example.yaml" ( echo: echo To create your configuration: echo 1. Copy: copy config\config.example.yaml config\config.yaml echo 2. Edit: notepad config\config.yaml echo 3. Add your API keys ) echo: ) else ( echo [OK] config\config.yaml exists ) echo: REM Get UV path for /f "tokens=*" %%i in ('where uv 2^>nul') do set "UV_PATH=%%i" if not defined UV_PATH ( set "UV_PATH=uv" ) echo: echo ========================================== echo Setup Complete! echo ========================================== echo: echo MCP Server Configuration for Claude Desktop: echo: echo Command: %UV_PATH% echo Working Directory: %PROJECT_ROOT% echo: echo Arguments (one per line): echo --directory echo %PROJECT_ROOT% echo run echo python echo -m echo mcp_server.server echo: echo Configuration guide: README-Cherry-Studio.md echo: echo: pause ================================================ FILE: setup-windows.bat ================================================ @echo off chcp 65001 >nul setlocal enabledelayedexpansion echo ========================================== echo TrendRadar MCP 一键部署 (Windows) echo ========================================== echo. REM 修复：使用脚本所在目录，而不是当前工作目录 set "PROJECT_ROOT=%~dp0" REM 移除末尾的反斜杠 if "%PROJECT_ROOT:~-1%"=="\" set "PROJECT_ROOT=%PROJECT_ROOT:~0,-1%" echo 📍 项目目录: %PROJECT_ROOT% echo. REM 切换到项目目录 cd /d "%PROJECT_ROOT%" if %errorlevel% neq 0 ( echo ❌ 无法访问项目目录 pause exit /b 1 ) REM 验证项目结构 echo [0/4] 🔍 验证项目结构... if not exist "pyproject.toml" ( echo ❌ 未找到 pyproject.toml 文件: %PROJECT_ROOT% echo. echo 请检查: echo 1. setup-windows.bat 是否在项目根目录? echo 2. 项目文件是否完整? echo. echo 当前目录内容: dir /b echo. pause exit /b 1 ) echo ✅ pyproject.toml 已找到 echo. REM 检查 Python echo [1/4] 🐍 检查 Python... python --version >nul 2>&1 if %errorlevel% neq 0 ( echo ❌ 未检测到 Python，请先安装 Python 3.10+ echo 下载地址: https://www.python.org/downloads/ pause exit /b 1 ) for /f "tokens=*" %%i in ('python --version') do echo ✅ %%i echo. REM 检查 UV echo [2/4] 🔧 检查 UV... where uv >nul 2>&1 if %errorlevel% neq 0 ( echo UV 未安装，正在自动安装... echo. echo 尝试方法1: PowerShell 安装... powershell -ExecutionPolicy Bypass -Command "try { irm https://astral.sh/uv/install.ps1 | iex; exit 0 } catch { Write-Host 'PowerShell 安装失败'; exit 1 }" if %errorlevel% neq 0 ( echo. echo 方法1失败，尝试方法2: pip 安装... python -m pip install --upgrade uv if %errorlevel% neq 0 ( echo. echo ❌ 自动安装失败 echo. echo 请手动安装 UV，可选方法: echo. echo 方法1 - pip: echo python -m pip install uv echo. echo 方法2 - pipx: echo pip install pipx echo pipx install uv echo. echo 方法3 - 手动下载: echo 访问: https://docs.astral.sh/uv/getting-started/installation/ echo. pause exit /b 1 ) ) echo. echo ✅ UV 安装完成！ echo. echo ⚠️ 重要: 请按照以下步骤操作: echo 1. 关闭此窗口 echo 2. 重新打开命令提示符（或 PowerShell） echo 3. 回到项目目录: %PROJECT_ROOT% echo 4. 重新运行此脚本: setup-windows.bat echo. pause exit /b 0 ) else ( for /f "tokens=*" %%i in ('uv --version') do echo ✅ %%i ) echo. echo [3/4] 📦 安装项目依赖... echo 工作目录: %PROJECT_ROOT% echo. REM 确保在项目目录下执行 cd /d "%PROJECT_ROOT%" uv sync if %errorlevel% neq 0 ( echo. echo ❌ 依赖安装失败 echo. echo 可能的原因: echo 1. 网络连接问题 echo 2. Python 版本不兼容（需要 ^>= 3.10） echo 3. pyproject.toml 文件格式错误 echo. echo 故障排查: echo - 检查网络连接 echo - 验证 Python 版本: python --version echo - 尝试详细输出: uv sync --verbose echo. echo 项目目录: %PROJECT_ROOT% echo. pause exit /b 1 ) echo. echo ✅ 依赖安装成功 echo. echo [4/4] ⚙️ 检查配置文件... if not exist "config\config.yaml" ( echo ⚠️ 配置文件不存在: config\config.yaml if exist "config\config.example.yaml" ( echo. echo 创建配置文件: echo 1. 复制: copy config\config.example.yaml config\config.yaml echo 2. 编辑: notepad config\config.yaml echo 3. 填入 API 密钥 ) echo. ) else ( echo ✅ config\config.yaml 已存在 ) echo. REM 获取 UV 路径 for /f "tokens=*" %%i in ('where uv 2^>nul') do set "UV_PATH=%%i" if not defined UV_PATH ( set "UV_PATH=uv" ) echo. echo ========================================== echo 部署完成！ echo ========================================== echo. echo 📋 MCP 服务器配置信息（用于 Claude Desktop）: echo. echo 命令: %UV_PATH% echo 工作目录: %PROJECT_ROOT% echo. echo 参数（逐行填入）: echo --directory echo %PROJECT_ROOT% echo run echo python echo -m echo mcp_server.server echo. echo 📖 详细教程: README-Cherry-Studio.md echo. echo. pause ================================================ FILE: start-http.bat ================================================ @echo off chcp 65001 >nul echo ============================================================ echo TrendRadar MCP Server (HTTP 模式) echo ============================================================ echo. REM 检查虚拟环境 if not exist ".venv\Scripts\python.exe" ( echo ❌ [错误] 虚拟环境未找到 echo 请先运行 setup-windows.bat 或 setup-windows-en.bat 进行部署 echo. pause exit /b 1 ) echo [模式] HTTP (适合远程访问) echo [地址] http://localhost:3333/mcp echo [提示] 按 Ctrl+C 停止服务 echo. uv run python -m mcp_server.server --transport http --host 0.0.0.0 --port 3333 pause ================================================ FILE: start-http.sh ================================================ #!/bin/bash echo "╔════════════════════════════════════════╗" echo "║ TrendRadar MCP Server (HTTP 模式) ║" echo "╚════════════════════════════════════════╝" echo "" # 检查虚拟环境 if [ ! -d ".venv" ]; then echo "❌ [错误] 虚拟环境未找到" echo "请先运行 ./setup-mac.sh 进行部署" echo "" exit 1 fi echo "[模式] HTTP (适合远程访问)" echo "[地址] http://localhost:3333/mcp" echo "[提示] 按 Ctrl+C 停止服务" echo "" uv run python -m mcp_server.server --transport http --host 0.0.0.0 --port 3333 ================================================ FILE: trendradar/__init__.py ================================================ # coding=utf-8 """ TrendRadar - 热点新闻聚合与分析工具 使用方式: python -m trendradar # 模块执行 trendradar # 安装后执行 """ from trendradar.context import AppContext __version__ = "6.5.0" __all__ = ["AppContext", "__version__"] ================================================ FILE: trendradar/__main__.py ================================================ # coding=utf-8 """ TrendRadar 主程序 热点新闻聚合与分析工具 支持: python -m trendradar """ import argparse import copy import json import os import re import sys import webbrowser from datetime import datetime, timezone from pathlib import Path from typing import Dict, List, Tuple, Optional import requests from trendradar.context import AppContext from trendradar import __version__ from trendradar.core import load_config, parse_multi_account_config, validate_paired_configs from trendradar.core.analyzer import convert_keyword_stats_to_platform_stats from trendradar.crawler import DataFetcher from trendradar.storage import convert_crawl_results_to_news_data from trendradar.utils.time import DEFAULT_TIMEZONE, is_within_days, calculate_days_old from trendradar.ai import AIAnalyzer, AIAnalysisResult from trendradar.core.scheduler import ResolvedSchedule def _parse_version(version_str: str) -> Tuple[int, int, int]: """解析版本号字符串为元组""" try: parts = version_str.strip().split(".") if len(parts) >= 3: return int(parts[0]), int(parts[1]), int(parts[2]) return 0, 0, 0 except: return 0, 0, 0 def _compare_version(local: str, remote: str) -> str: """比较版本号，返回状态文字""" local_tuple = _parse_version(local) remote_tuple = _parse_version(remote) if local_tuple < remote_tuple: return "⚠️ 需要更新" elif local_tuple > remote_tuple: return "🔮 超前版本" else: return "✅ 已是最新" def _fetch_remote_version(version_url: str, proxy_url: Optional[str] = None) -> Optional[str]: """获取远程版本号""" try: proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} headers = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36", "Accept": "text/plain, */*", "Cache-Control": "no-cache", } response = requests.get(version_url, proxies=proxies, headers=headers, timeout=10) response.raise_for_status() return response.text.strip() except Exception as e: print(f"[版本检查] 获取远程版本失败: {e}") return None def _parse_config_versions(content: str) -> Dict[str, str]: """解析配置文件版本内容为字典""" versions = {} try: if not content: return versions for line in content.splitlines(): line = line.strip() if not line or "=" not in line: continue name, version = line.split("=", 1) versions[name.strip()] = version.strip() except Exception as e: print(f"[版本检查] 解析配置版本失败: {e}") return versions def check_all_versions( version_url: str, configs_version_url: Optional[str] = None, proxy_url: Optional[str] = None ) -> Tuple[bool, Optional[str]]: """ 统一版本检查：程序版本 + 配置文件版本 Args: version_url: 远程程序版本检查 URL configs_version_url: 远程配置文件版本检查 URL (返回格式: filename=version) proxy_url: 代理 URL Returns: (need_update, remote_version): 程序是否需要更新及远程版本号 """ # 获取远程版本 remote_version = _fetch_remote_version(version_url, proxy_url) # 获取远程配置版本（如果有提供 URL） remote_config_versions = {} if configs_version_url: content = _fetch_remote_version(configs_version_url, proxy_url) if content: remote_config_versions = _parse_config_versions(content) print("=" * 60) print("版本检查") print("=" * 60) if remote_version: print(f"远程程序版本: {remote_version}") else: print("远程程序版本: 获取失败") if configs_version_url: if remote_config_versions: print(f"远程配置清单: 获取成功 ({len(remote_config_versions)} 个文件)") else: print("远程配置清单: 获取失败或为空") print("-" * 60) program_status = _compare_version(__version__, remote_version) if remote_version else "(无法比较)" print(f" 主程序版本: {__version__} {program_status}") config_files = [ Path("config/config.yaml"), Path("config/timeline.yaml"), Path("config/frequency_words.txt"), Path("config/ai_interests.txt"), Path("config/ai_analysis_prompt.txt"), Path("config/ai_translation_prompt.txt"), ] version_pattern = re.compile(r"Version:\s*(\d+\.\d+\.\d+)", re.IGNORECASE) for config_file in config_files: if not config_file.exists(): print(f" {config_file.name}: 文件不存在") continue try: with open(config_file, "r", encoding="utf-8") as f: local_version = None for i, line in enumerate(f): if i >= 20: break match = version_pattern.search(line) if match: local_version = match.group(1) break # 获取该文件的远程版本 target_remote_version = remote_config_versions.get(config_file.name) if local_version: if target_remote_version: status = _compare_version(local_version, target_remote_version) print(f" {config_file.name}: {local_version} {status}") else: print(f" {config_file.name}: {local_version} (未找到远程版本)") else: print(f" {config_file.name}: 未找到本地版本号") except Exception as e: print(f" {config_file.name}: 读取失败 - {e}") print("=" * 60) # 返回程序版本的更新状态 if remote_version: need_update = _parse_version(__version__) < _parse_version(remote_version) return need_update, remote_version if need_update else None return False, None # === 主分析器 === class NewsAnalyzer: """新闻分析器""" # 模式策略定义 MODE_STRATEGIES = { "incremental": { "mode_name": "增量模式", "description": "增量模式（只关注新增新闻，无新增时不推送）", "report_type": "增量分析", "should_send_notification": True, }, "current": { "mode_name": "当前榜单模式", "description": "当前榜单模式（当前榜单匹配新闻 + 新增新闻区域 + 按时推送）", "report_type": "当前榜单", "should_send_notification": True, }, "daily": { "mode_name": "全天汇总模式", "description": "全天汇总模式（所有匹配新闻 + 新增新闻区域 + 按时推送）", "report_type": "全天汇总", "should_send_notification": True, }, } def __init__(self, config: Optional[Dict] = None): # 使用传入的配置或加载新配置 if config is None: print("正在加载配置...") config = load_config() print(f"TrendRadar v{__version__} 配置加载完成") print(f"监控平台数量: {len(config['PLATFORMS'])}") print(f"时区: {config.get('TIMEZONE', DEFAULT_TIMEZONE)}") # 创建应用上下文 self.ctx = AppContext(config) self.request_interval = self.ctx.config["REQUEST_INTERVAL"] self.report_mode = self.ctx.config["REPORT_MODE"] self.frequency_file = None self.filter_method = None # None=使用全局配置 ctx.filter_method self.interests_file = None # None=使用全局配置 ai_filter.interests_file self.rank_threshold = self.ctx.rank_threshold self.is_github_actions = os.environ.get("GITHUB_ACTIONS") == "true" self.is_docker_container = self._detect_docker_environment() self.update_info = None self.proxy_url = None self._setup_proxy() self.data_fetcher = DataFetcher(self.proxy_url) # 初始化存储管理器（使用 AppContext） self._init_storage_manager() # 注意：update_info 由 main() 函数设置，避免重复请求远程版本 def _init_storage_manager(self) -> None: """初始化存储管理器（使用 AppContext）""" # 获取数据保留天数（支持环境变量覆盖） env_retention = os.environ.get("STORAGE_RETENTION_DAYS", "").strip() if env_retention: # 环境变量覆盖配置 self.ctx.config["STORAGE"]["RETENTION_DAYS"] = int(env_retention) self.storage_manager = self.ctx.get_storage_manager() print(f"存储后端: {self.storage_manager.backend_name}") retention_days = self.ctx.config.get("STORAGE", {}).get("RETENTION_DAYS", 0) if retention_days > 0: print(f"数据保留天数: {retention_days} 天") def _detect_docker_environment(self) -> bool: """检测是否运行在 Docker 容器中""" try: if os.environ.get("DOCKER_CONTAINER") == "true": return True if os.path.exists("/.dockerenv"): return True return False except Exception: return False def _should_open_browser(self) -> bool: """判断是否应该打开浏览器""" return not self.is_github_actions and not self.is_docker_container def _setup_proxy(self) -> None: """设置代理配置""" if not self.is_github_actions and self.ctx.config["USE_PROXY"]: self.proxy_url = self.ctx.config["DEFAULT_PROXY"] print("本地环境，使用代理") elif not self.is_github_actions and not self.ctx.config["USE_PROXY"]: print("本地环境，未启用代理") else: print("GitHub Actions环境，不使用代理") def _set_update_info_from_config(self) -> None: """从已缓存的远程版本设置更新信息（不再重复请求）""" try: version_url = self.ctx.config.get("VERSION_CHECK_URL", "") if not version_url: return remote_version = _fetch_remote_version(version_url, self.proxy_url) if remote_version: need_update = _parse_version(__version__) < _parse_version(remote_version) if need_update: self.update_info = { "current_version": __version__, "remote_version": remote_version, } except Exception as e: print(f"版本检查出错: {e}") def _get_mode_strategy(self) -> Dict: """获取当前模式的策略配置""" return self.MODE_STRATEGIES.get(self.report_mode, self.MODE_STRATEGIES["daily"]) def _has_notification_configured(self) -> bool: """检查是否配置了任何通知渠道""" cfg = self.ctx.config return any( [ cfg["FEISHU_WEBHOOK_URL"], cfg["DINGTALK_WEBHOOK_URL"], cfg["WEWORK_WEBHOOK_URL"], (cfg["TELEGRAM_BOT_TOKEN"] and cfg["TELEGRAM_CHAT_ID"]), ( cfg["EMAIL_FROM"] and cfg["EMAIL_PASSWORD"] and cfg["EMAIL_TO"] ), (cfg["NTFY_SERVER_URL"] and cfg["NTFY_TOPIC"]), cfg["BARK_URL"], cfg["SLACK_WEBHOOK_URL"], cfg["GENERIC_WEBHOOK_URL"], ] ) def _has_valid_content( self, stats: List[Dict], new_titles: Optional[Dict] = None ) -> bool: """检查是否有有效的新闻内容""" if self.report_mode == "incremental": # 增量模式：只要有匹配的新闻就推送 # count_word_frequency 已经确保只处理新增的新闻（包括当天第一次爬取的情况） has_matched_news = any(stat["count"] > 0 for stat in stats) return has_matched_news elif self.report_mode == "current": # current模式：只要stats有内容就说明有匹配的新闻 return any(stat["count"] > 0 for stat in stats) else: # 当日汇总模式下，检查是否有匹配的频率词新闻或新增新闻 has_matched_news = any(stat["count"] > 0 for stat in stats) has_new_news = bool( new_titles and any(len(titles) > 0 for titles in new_titles.values()) ) return has_matched_news or has_new_news def _prepare_ai_analysis_data( self, ai_mode: str, current_results: Optional[Dict] = None, current_id_to_name: Optional[Dict] = None, ) -> Tuple[List[Dict], Optional[Dict]]: """ 为 AI 分析准备指定模式的数据 Args: ai_mode: AI 分析模式 (daily/current/incremental) current_results: 当前抓取的结果（用于 incremental 模式） current_id_to_name: 当前的平台映射（用于 incremental 模式） Returns: Tuple[stats, id_to_name]: 统计数据和平台映射 """ try: word_groups, filter_words, global_filters = self.ctx.load_frequency_words(self.frequency_file) if ai_mode == "incremental": # incremental 模式：使用当前抓取的数据 if not current_results or not current_id_to_name: print("[AI] incremental 模式需要当前抓取数据，但未提供") return [], None # 准备当前时间信息 time_info = self.ctx.format_time() title_info = self._prepare_current_title_info(current_results, time_info) # 检测新增标题 new_titles = self.ctx.detect_new_titles(list(current_results.keys())) # 统计计算 stats, _ = self.ctx.count_frequency( current_results, word_groups, filter_words, current_id_to_name, title_info, new_titles, mode="incremental", global_filters=global_filters, quiet=True, ) # 如果是 platform 模式，转换数据结构 if self.ctx.display_mode == "platform" and stats: stats = convert_keyword_stats_to_platform_stats( stats, self.ctx.weight_config, self.ctx.rank_threshold, ) return stats, current_id_to_name elif ai_mode in ["daily", "current"]: # 加载历史数据 analysis_data = self._load_analysis_data(quiet=True) if not analysis_data: print(f"[AI] 无法加载历史数据用于 {ai_mode} 模式分析") return [], None ( all_results, id_to_name, title_info, new_titles, _, _, _, ) = analysis_data # 统计计算 stats, _ = self.ctx.count_frequency( all_results, word_groups, filter_words, id_to_name, title_info, new_titles, mode=ai_mode, global_filters=global_filters, quiet=True, ) # 如果是 platform 模式，转换数据结构 if self.ctx.display_mode == "platform" and stats: stats = convert_keyword_stats_to_platform_stats( stats, self.ctx.weight_config, self.ctx.rank_threshold, ) return stats, id_to_name else: print(f"[AI] 未知的 AI 模式: {ai_mode}") return [], None except Exception as e: print(f"[AI] 准备 {ai_mode} 模式数据时出错: {e}") if self.ctx.config.get("DEBUG", False): import traceback traceback.print_exc() return [], None def _run_ai_analysis( self, stats: List[Dict], rss_items: Optional[List[Dict]], mode: str, report_type: str, id_to_name: Optional[Dict], current_results: Optional[Dict] = None, schedule: ResolvedSchedule = None, standalone_data: Optional[Dict] = None, ) -> Optional[AIAnalysisResult]: """执行 AI 分析""" analysis_config = self.ctx.config.get("AI_ANALYSIS", {}) if not analysis_config.get("ENABLED", False): return None # 调度系统决策 if not schedule.analyze: print("[AI] 调度器: 当前时间段不执行 AI 分析") return None if schedule.once_analyze and schedule.period_key: scheduler = self.ctx.create_scheduler() date_str = self.ctx.format_date() if scheduler.already_executed(schedule.period_key, "analyze", date_str): print(f"[AI] 调度器: 时间段 {schedule.period_name or schedule.period_key} 今天已分析过，跳过") return None else: print(f"[AI] 调度器: 时间段 {schedule.period_name or schedule.period_key} 今天首次分析") print("[AI] 正在进行 AI 分析...") try: ai_config = self.ctx.config.get("AI", {}) debug_mode = self.ctx.config.get("DEBUG", False) analyzer = AIAnalyzer(ai_config, analysis_config, self.ctx.get_time, debug=debug_mode) # 确定 AI 分析使用的模式 ai_mode_config = analysis_config.get("MODE", "follow_report") if ai_mode_config == "follow_report": # 跟随推送报告模式 ai_mode = mode ai_stats = stats ai_id_to_name = id_to_name elif ai_mode_config in ["daily", "current", "incremental"]: # 使用独立配置的模式，需要重新准备数据 ai_mode = ai_mode_config if ai_mode != mode: print(f"[AI] 使用独立分析模式: {ai_mode} (推送模式: {mode})") print(f"[AI] 正在准备 {ai_mode} 模式的数据...") # 根据 AI 模式重新准备数据 ai_stats, ai_id_to_name = self._prepare_ai_analysis_data( ai_mode, current_results, id_to_name ) if not ai_stats: print(f"[AI] 警告: 无法准备 {ai_mode} 模式的数据，回退到推送模式数据") ai_stats = stats ai_id_to_name = id_to_name ai_mode = mode else: ai_stats = stats ai_id_to_name = id_to_name else: # 配置错误，回退到跟随模式 print(f"[AI] 警告: 无效的 ai_analysis.mode 配置 '{ai_mode_config}'，使用推送模式 '{mode}'") ai_mode = mode ai_stats = stats ai_id_to_name = id_to_name # 提取平台列表 platforms = list(ai_id_to_name.values()) if ai_id_to_name else [] # 提取关键词列表 keywords = [s.get("word", "") for s in ai_stats if s.get("word")] if ai_stats else [] # 确定报告类型 if ai_mode != mode: # 根据 AI 模式确定报告类型 ai_report_type = { "daily": "当日汇总", "current": "当前榜单", "incremental": "增量更新" }.get(ai_mode, report_type) else: ai_report_type = report_type result = analyzer.analyze( stats=ai_stats, rss_stats=rss_items, report_mode=ai_mode, report_type=ai_report_type, platforms=platforms, keywords=keywords, standalone_data=standalone_data, ) # 设置 AI 分析使用的模式 if result.success: result.ai_mode = ai_mode if result.error: # 成功但有警告（如 JSON 解析问题但使用了原始文本） print(f"[AI] 分析完成（有警告: {result.error}）") else: print("[AI] 分析完成") # 记录 AI 分析 if schedule.once_analyze and schedule.period_key: scheduler = self.ctx.create_scheduler() date_str = self.ctx.format_date() scheduler.record_execution(schedule.period_key, "analyze", date_str) else: print(f"[AI] 分析失败: {result.error}") return result except Exception as e: import traceback error_type = type(e).__name__ error_msg = str(e) # 截断过长的错误消息 if len(error_msg) > 200: error_msg = error_msg[:200] + "..." print(f"[AI] 分析出错 ({error_type}): {error_msg}") # 详细错误日志到 stderr import sys print(f"[AI] 详细错误堆栈:", file=sys.stderr) traceback.print_exc(file=sys.stderr) return AIAnalysisResult(success=False, error=f"{error_type}: {error_msg}") def _load_analysis_data( self, quiet: bool = False, ) -> Optional[Tuple[Dict, Dict, Dict, Dict, List, List]]: """统一的数据加载和预处理，使用当前监控平台列表过滤历史数据""" try: # 获取当前配置的监控平台ID列表 current_platform_ids = self.ctx.platform_ids if not quiet: print(f"当前监控平台: {current_platform_ids}") all_results, id_to_name, title_info = self.ctx.read_today_titles( current_platform_ids, quiet=quiet ) if not all_results: print("没有找到当天的数据") return None total_titles = sum(len(titles) for titles in all_results.values()) if not quiet: print(f"读取到 {total_titles} 个标题（已按当前监控平台过滤）") new_titles = self.ctx.detect_new_titles(current_platform_ids, quiet=quiet) word_groups, filter_words, global_filters = self.ctx.load_frequency_words(self.frequency_file) return ( all_results, id_to_name, title_info, new_titles, word_groups, filter_words, global_filters, ) except Exception as e: print(f"数据加载失败: {e}") return None def _prepare_current_title_info(self, results: Dict, time_info: str) -> Dict: """从当前抓取结果构建标题信息""" title_info = {} for source_id, titles_data in results.items(): title_info[source_id] = {} for title, title_data in titles_data.items(): ranks = title_data.get("ranks", []) url = title_data.get("url", "") mobile_url = title_data.get("mobileUrl", "") title_info[source_id][title] = { "first_time": time_info, "last_time": time_info, "count": 1, "ranks": ranks, "url": url, "mobileUrl": mobile_url, } return title_info def _prepare_standalone_data( self, results: Dict, id_to_name: Dict, title_info: Optional[Dict] = None, rss_items: Optional[List[Dict]] = None, ) -> Optional[Dict]: """ 从原始数据中提取独立展示区数据 纯数据准备方法，不检查 display.regions.standalone 开关。 各消费者自行决定是否使用： - AI 分析：由 ai.include_standalone 控制 - 通知推送：由 display.regions.standalone 控制（在 dispatcher 层门控） - HTML 报告：始终包含（如果有数据） Args: results: 原始爬取结果 {platform_id: {title: title_data}} id_to_name: 平台 ID 到名称的映射 title_info: 标题元信息（含排名历史、时间等） rss_items: RSS 条目列表 Returns: 独立展示数据字典，如果未配置数据源返回 None """ display_config = self.ctx.config.get("DISPLAY", {}) standalone_config = display_config.get("STANDALONE", {}) platform_ids = standalone_config.get("PLATFORMS", []) rss_feed_ids = standalone_config.get("RSS_FEEDS", []) max_items = standalone_config.get("MAX_ITEMS", 20) if not platform_ids and not rss_feed_ids: return None standalone_data = { "platforms": [], "rss_feeds": [], } # 找出最新批次时间（类似 current 模式的过滤逻辑） latest_time = None if title_info: for source_titles in title_info.values(): for title_data in source_titles.values(): last_time = title_data.get("last_time", "") if last_time: if latest_time is None or last_time > latest_time: latest_time = last_time # 提取热榜平台数据 for platform_id in platform_ids: if platform_id not in results: continue platform_name = id_to_name.get(platform_id, platform_id) platform_titles = results[platform_id] items = [] for title, title_data in platform_titles.items(): # 获取元信息（如果有 title_info） meta = {} if title_info and platform_id in title_info and title in title_info[platform_id]: meta = title_info[platform_id][title] # 只保留当前在榜的话题（last_time 等于最新时间） if latest_time and meta: if meta.get("last_time") != latest_time: continue # 使用当前热榜的排名数据（title_data）进行排序 # title_data 包含的是爬虫返回的当前排名，用于保证独立展示区的顺序与热榜一致 current_ranks = title_data.get("ranks", []) current_rank = current_ranks[-1] if current_ranks else 0 # 用于显示的排名范围：合并历史排名和当前排名 historical_ranks = meta.get("ranks", []) if meta else [] # 合并去重，保持顺序 all_ranks = historical_ranks.copy() for rank in current_ranks: if rank not in all_ranks: all_ranks.append(rank) display_ranks = all_ranks if all_ranks else current_ranks item = { "title": title, "url": title_data.get("url", ""), "mobileUrl": title_data.get("mobileUrl", ""), "rank": current_rank, # 用于排序的当前排名 "ranks": display_ranks, # 用于显示的排名范围（历史+当前） "first_time": meta.get("first_time", ""), "last_time": meta.get("last_time", ""), "count": meta.get("count", 1), "rank_timeline": meta.get("rank_timeline", []), } items.append(item) # 按当前排名排序 items.sort(key=lambda x: x["rank"] if x["rank"] > 0 else 9999) # 限制条数 if max_items > 0: items = items[:max_items] if items: standalone_data["platforms"].append({ "id": platform_id, "name": platform_name, "items": items, }) # 提取 RSS 数据 if rss_items and rss_feed_ids: # 按 feed_id 分组 feed_items_map = {} for item in rss_items: feed_id = item.get("feed_id", "") if feed_id in rss_feed_ids: if feed_id not in feed_items_map: feed_items_map[feed_id] = { "name": item.get("feed_name", feed_id), "items": [], } feed_items_map[feed_id]["items"].append({ "title": item.get("title", ""), "url": item.get("url", ""), "published_at": item.get("published_at", ""), "author": item.get("author", ""), }) # 限制条数并添加到结果 for feed_id in rss_feed_ids: if feed_id in feed_items_map: feed_data = feed_items_map[feed_id] items = feed_data["items"] if max_items > 0: items = items[:max_items] if items: standalone_data["rss_feeds"].append({ "id": feed_id, "name": feed_data["name"], "items": items, }) # 如果没有任何数据，返回 None if not standalone_data["platforms"] and not standalone_data["rss_feeds"]: return None return standalone_data def _run_analysis_pipeline( self, data_source: Dict, mode: str, title_info: Dict, new_titles: Dict, word_groups: List[Dict], filter_words: List[str], id_to_name: Dict, failed_ids: Optional[List] = None, global_filters: Optional[List[str]] = None, quiet: bool = False, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, standalone_data: Optional[Dict] = None, schedule: ResolvedSchedule = None, rss_new_urls: Optional[set] = None, ) -> Tuple[List[Dict], Optional[str], Optional[AIAnalysisResult], Optional[List[Dict]]]: """统一的分析流水线：数据处理 → 统计计算（关键词/AI筛选）→ AI分析 → HTML生成""" # 根据筛选策略选择数据处理方式 if self.filter_method == "ai": # === AI 筛选策略 === print("[筛选] 使用 AI 智能筛选策略") ai_filter_result = self.ctx.run_ai_filter(interests_file=self.interests_file) if ai_filter_result and ai_filter_result.success: print(f"[筛选] AI 筛选完成: {ai_filter_result.total_matched} 条匹配, {len(ai_filter_result.tags)} 个标签") # 转换为与关键词匹配相同的数据结构 stats, ai_rss_stats = self.ctx.convert_ai_filter_to_report_data( ai_filter_result, mode=mode, new_titles=new_titles, rss_new_urls=rss_new_urls, ) total_titles = sum(len(titles) for titles in data_source.values()) # AI 筛选的 RSS 结果替换关键词匹配的 RSS 结果 if ai_rss_stats: rss_items = ai_rss_stats else: # AI 筛选失败，回退到关键词匹配 error_msg = ai_filter_result.error if ai_filter_result else "未知错误" print(f"[筛选] AI 筛选失败: {error_msg}，回退到关键词匹配") stats, total_titles = self.ctx.count_frequency( data_source, word_groups, filter_words, id_to_name, title_info, new_titles, mode=mode, global_filters=global_filters, quiet=quiet, ) else: # === 关键词匹配策略（默认）=== stats, total_titles = self.ctx.count_frequency( data_source, word_groups, filter_words, id_to_name, title_info, new_titles, mode=mode, global_filters=global_filters, quiet=quiet, ) # 如果是 platform 模式，转换数据结构 if self.ctx.display_mode == "platform" and stats: stats = convert_keyword_stats_to_platform_stats( stats, self.ctx.weight_config, self.ctx.rank_threshold, ) # AI 分析（如果启用，用于 HTML 报告） ai_result = None ai_config = self.ctx.config.get("AI_ANALYSIS", {}) if ai_config.get("ENABLED", False) and stats: # 获取模式策略来确定报告类型 mode_strategy = self._get_mode_strategy() report_type = mode_strategy["report_type"] ai_result = self._run_ai_analysis( stats, rss_items, mode, report_type, id_to_name, current_results=data_source, schedule=schedule, standalone_data=standalone_data ) # HTML生成（如果启用） html_file = None if self.ctx.config["STORAGE"]["FORMATS"]["HTML"]: html_file = self.ctx.generate_html( stats, total_titles, failed_ids=failed_ids, new_titles=new_titles, id_to_name=id_to_name, mode=mode, update_info=self.update_info if self.ctx.config["SHOW_VERSION_UPDATE"] else None, rss_items=rss_items, rss_new_items=rss_new_items, ai_analysis=ai_result, standalone_data=standalone_data, frequency_file=self.frequency_file, ) return stats, html_file, ai_result, rss_items def _send_notification_if_needed( self, stats: List[Dict], report_type: str, mode: str, failed_ids: Optional[List] = None, new_titles: Optional[Dict] = None, id_to_name: Optional[Dict] = None, html_file_path: Optional[str] = None, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, standalone_data: Optional[Dict] = None, ai_result: Optional[AIAnalysisResult] = None, current_results: Optional[Dict] = None, schedule: ResolvedSchedule = None, ) -> bool: """统一的通知发送逻辑，包含所有判断条件，支持热榜+RSS合并推送+AI分析+独立展示区""" has_notification = self._has_notification_configured() cfg = self.ctx.config # 检查是否有有效内容（热榜或RSS） has_news_content = self._has_valid_content(stats, new_titles) has_rss_content = bool(rss_items and len(rss_items) > 0) has_any_content = has_news_content or has_rss_content # 计算热榜匹配条数 news_count = sum(len(stat.get("titles", [])) for stat in stats) if stats else 0 rss_count = sum(stat.get("count", 0) for stat in rss_items) if rss_items else 0 if ( cfg["ENABLE_NOTIFICATION"] and has_notification and has_any_content ): # 输出推送内容统计 content_parts = [] if news_count > 0: content_parts.append(f"热榜 {news_count} 条") if rss_count > 0: content_parts.append(f"RSS {rss_count} 条") total_count = news_count + rss_count print(f"[推送] 准备发送：{' + '.join(content_parts)}，合计 {total_count} 条") # 调度系统决策 if not schedule.push: print("[推送] 调度器: 当前时间段不执行推送") return False if schedule.once_push and schedule.period_key: scheduler = self.ctx.create_scheduler() date_str = self.ctx.format_date() if scheduler.already_executed(schedule.period_key, "push", date_str): print(f"[推送] 调度器: 时间段 {schedule.period_name or schedule.period_key} 今天已推送过，跳过") return False else: print(f"[推送] 调度器: 时间段 {schedule.period_name or schedule.period_key} 今天首次推送") # AI 分析：优先使用传入的结果，避免重复分析 if ai_result is None: ai_config = cfg.get("AI_ANALYSIS", {}) if ai_config.get("ENABLED", False): ai_result = self._run_ai_analysis( stats, rss_items, mode, report_type, id_to_name, current_results=current_results, schedule=schedule ) # 准备报告数据 report_data = self.ctx.prepare_report(stats, failed_ids, new_titles, id_to_name, mode, frequency_file=self.frequency_file) # 是否发送版本更新信息 update_info_to_send = self.update_info if cfg["SHOW_VERSION_UPDATE"] else None # 使用 NotificationDispatcher 发送到所有渠道 dispatcher = self.ctx.create_notification_dispatcher() results = dispatcher.dispatch_all( report_data=report_data, report_type=report_type, update_info=update_info_to_send, proxy_url=self.proxy_url, mode=mode, html_file_path=html_file_path, rss_items=rss_items, rss_new_items=rss_new_items, ai_analysis=ai_result, standalone_data=standalone_data, ) if not results: print("未配置任何通知渠道，跳过通知发送") return False # 记录推送成功 if any(results.values()): if schedule.once_push and schedule.period_key: scheduler = self.ctx.create_scheduler() date_str = self.ctx.format_date() scheduler.record_execution(schedule.period_key, "push", date_str) return True elif cfg["ENABLE_NOTIFICATION"] and not has_notification: print("⚠️ 警告：通知功能已启用但未配置任何通知渠道，将跳过通知发送") elif not cfg["ENABLE_NOTIFICATION"]: print(f"跳过{report_type}通知：通知功能已禁用") elif ( cfg["ENABLE_NOTIFICATION"] and has_notification and not has_any_content ): mode_strategy = self._get_mode_strategy() if self.report_mode == "incremental": if not has_rss_content: print("跳过通知：增量模式下未检测到匹配的新闻和RSS") else: print("跳过通知：增量模式下新闻未匹配到关键词") else: print( f"跳过通知：{mode_strategy['mode_name']}下未检测到匹配的新闻" ) return False def _initialize_and_check_config(self) -> None: """通用初始化和配置检查""" now = self.ctx.get_time() print(f"当前北京时间: {now.strftime('%Y-%m-%d %H:%M:%S')}") if not self.ctx.config["ENABLE_CRAWLER"]: print("爬虫功能已禁用（ENABLE_CRAWLER=False），程序退出") return has_notification = self._has_notification_configured() if not self.ctx.config["ENABLE_NOTIFICATION"]: print("通知功能已禁用（ENABLE_NOTIFICATION=False），将只进行数据抓取") elif not has_notification: print("未配置任何通知渠道，将只进行数据抓取，不发送通知") else: print("通知功能已启用，将发送通知") mode_strategy = self._get_mode_strategy() print(f"报告模式: {self.report_mode}") print(f"运行模式: {mode_strategy['description']}") def _crawl_data(self) -> Tuple[Dict, Dict, List]: """执行数据爬取""" ids = [] for platform in self.ctx.platforms: if "name" in platform: ids.append((platform["id"], platform["name"])) else: ids.append(platform["id"]) print( f"配置的监控平台: {[p.get('name', p['id']) for p in self.ctx.platforms]}" ) print(f"开始爬取数据，请求间隔 {self.request_interval} 毫秒") Path("output").mkdir(parents=True, exist_ok=True) results, id_to_name, failed_ids = self.data_fetcher.crawl_websites( ids, self.request_interval ) # 转换为 NewsData 格式并保存到存储后端 crawl_time = self.ctx.format_time() crawl_date = self.ctx.format_date() news_data = convert_crawl_results_to_news_data( results, id_to_name, failed_ids, crawl_time, crawl_date ) # 保存到存储后端（SQLite） if self.storage_manager.save_news_data(news_data): print(f"数据已保存到存储后端: {self.storage_manager.backend_name}") # 保存 TXT 快照（如果启用） txt_file = self.storage_manager.save_txt_snapshot(news_data) if txt_file: print(f"TXT 快照已保存: {txt_file}") return results, id_to_name, failed_ids def _crawl_rss_data(self) -> Tuple[Optional[List[Dict]], Optional[List[Dict]], Optional[List[Dict]], set]: """ 执行 RSS 数据抓取 Returns: (rss_items, rss_new_items, raw_rss_items, rss_new_urls) 元组： - rss_items: 统计条目列表（按模式处理，用于统计区块） - rss_new_items: 新增条目列表（用于新增区块） - raw_rss_items: 原始 RSS 条目列表（用于独立展示区） - rss_new_urls: 原始新增 RSS 条目的 URL 集合（用于 AI 模式 is_new 检测） 如果未启用或失败返回 (None, None, None, set()) """ if not self.ctx.rss_enabled: return None, None, None, set() rss_feeds = self.ctx.rss_feeds if not rss_feeds: print("[RSS] 未配置任何 RSS 源") return None, None, None, set() try: from trendradar.crawler.rss import RSSFetcher, RSSFeedConfig # 构建 RSS 源配置 feeds = [] for feed_config in rss_feeds: # 读取并验证单个 feed 的 max_age_days（可选） max_age_days_raw = feed_config.get("max_age_days") max_age_days = None if max_age_days_raw is not None: try: max_age_days = int(max_age_days_raw) if max_age_days < 0: feed_id = feed_config.get("id", "unknown") print(f"[警告] RSS feed '{feed_id}' 的 max_age_days 为负数，将使用全局默认值") max_age_days = None except (ValueError, TypeError): feed_id = feed_config.get("id", "unknown") print(f"[警告] RSS feed '{feed_id}' 的 max_age_days 格式错误：{max_age_days_raw}") max_age_days = None feed = RSSFeedConfig( id=feed_config.get("id", ""), name=feed_config.get("name", ""), url=feed_config.get("url", ""), max_items=feed_config.get("max_items", 50), enabled=feed_config.get("enabled", True), max_age_days=max_age_days, # None=使用全局，0=禁用，>0=覆盖 ) if feed.id and feed.url and feed.enabled: feeds.append(feed) if not feeds: print("[RSS] 没有启用的 RSS 源") return None, None, None, set() # 创建抓取器 rss_config = self.ctx.rss_config # RSS 代理：优先使用 RSS 专属代理，否则使用爬虫默认代理 rss_proxy_url = rss_config.get("PROXY_URL", "") or self.proxy_url or "" # 获取配置的时区 timezone = self.ctx.config.get("TIMEZONE", DEFAULT_TIMEZONE) # 获取新鲜度过滤配置 freshness_config = rss_config.get("FRESHNESS_FILTER", {}) freshness_enabled = freshness_config.get("ENABLED", True) default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3) fetcher = RSSFetcher( feeds=feeds, request_interval=rss_config.get("REQUEST_INTERVAL", 2000), timeout=rss_config.get("TIMEOUT", 15), use_proxy=rss_config.get("USE_PROXY", False), proxy_url=rss_proxy_url, timezone=timezone, freshness_enabled=freshness_enabled, default_max_age_days=default_max_age_days, ) # 抓取数据 rss_data = fetcher.fetch_all() # 保存到存储后端 if self.storage_manager.save_rss_data(rss_data): print(f"[RSS] 数据已保存到存储后端") # 处理 RSS 数据（按模式过滤）并返回用于合并推送 return self._process_rss_data_by_mode(rss_data) else: print(f"[RSS] 数据保存失败") return None, None, None, set() except ImportError as e: print(f"[RSS] 缺少依赖: {e}") print("[RSS] 请安装 feedparser: pip install feedparser") return None, None, None, set() except Exception as e: print(f"[RSS] 抓取失败: {e}") return None, None, None, set() def _process_rss_data_by_mode(self, rss_data) -> Tuple[Optional[List[Dict]], Optional[List[Dict]], Optional[List[Dict]], set]: """ 按报告模式处理 RSS 数据，返回与热榜相同格式的统计结构 三种模式： - daily: 当日汇总，统计=当天所有条目，新增=本次新增条目 - current: 当前榜单，统计=当前榜单条目，新增=本次新增条目 - incremental: 增量模式，统计=新增条目，新增=无 Args: rss_data: 当前抓取的 RSSData 对象 Returns: (rss_stats, rss_new_stats, raw_rss_items, rss_new_urls) 元组： - rss_stats: RSS 关键词统计列表（与热榜 stats 格式一致） - rss_new_stats: RSS 新增关键词统计列表（与热榜 stats 格式一致） - raw_rss_items: 原始 RSS 条目列表（用于独立展示区） - rss_new_urls: 原始新增 RSS 条目的 URL 集合（未经关键词过滤，用于 AI 模式 is_new 检测） """ from trendradar.core.analyzer import count_rss_frequency # 从 display.regions.rss 统一控制 RSS 分析和展示 rss_display_enabled = self.ctx.config.get("DISPLAY", {}).get("REGIONS", {}).get("RSS", True) # 加载关键词配置 try: word_groups, filter_words, global_filters = self.ctx.load_frequency_words(self.frequency_file) except FileNotFoundError: word_groups, filter_words, global_filters = [], [], [] timezone = self.ctx.timezone max_news_per_keyword = self.ctx.config.get("MAX_NEWS_PER_KEYWORD", 0) sort_by_position_first = self.ctx.config.get("SORT_BY_POSITION_FIRST", False) rss_stats = None rss_new_stats = None raw_rss_items = None # 原始 RSS 条目列表（用于独立展示区） rss_new_urls = set() # 原始新增 RSS URLs（未经关键词过滤） # 1. 首先获取原始条目（用于独立展示区，不受 display.regions.rss 影响） # 根据模式获取原始条目 if self.report_mode == "incremental": new_items_dict = self.storage_manager.detect_new_rss_items(rss_data) if new_items_dict: raw_rss_items = self._convert_rss_items_to_list(new_items_dict, rss_data.id_to_name) elif self.report_mode == "current": latest_data = self.storage_manager.get_latest_rss_data(rss_data.date) if latest_data: raw_rss_items = self._convert_rss_items_to_list(latest_data.items, latest_data.id_to_name) else: # daily all_data = self.storage_manager.get_rss_data(rss_data.date) if all_data: raw_rss_items = self._convert_rss_items_to_list(all_data.items, all_data.id_to_name) # 如果 RSS 展示未启用，跳过关键词分析，只返回原始条目用于独立展示区 if not rss_display_enabled: return None, None, raw_rss_items, rss_new_urls # 2. 获取新增条目（用于统计） new_items_dict = self.storage_manager.detect_new_rss_items(rss_data) new_items_list = None if new_items_dict: new_items_list = self._convert_rss_items_to_list(new_items_dict, rss_data.id_to_name) if new_items_list: print(f"[RSS] 检测到 {len(new_items_list)} 条新增") # 收集原始新增 URLs（未经关键词过滤，用于 AI 模式 is_new 检测） rss_new_urls = {item["url"] for item in new_items_list if item.get("url")} # 3. 根据模式获取统计条目 if self.report_mode == "incremental": # 增量模式：统计条目就是新增条目 if not new_items_list: print("[RSS] 增量模式：没有新增 RSS 条目") return None, None, raw_rss_items, rss_new_urls rss_stats, total = count_rss_frequency( rss_items=new_items_list, word_groups=word_groups, filter_words=filter_words, global_filters=global_filters, new_items=new_items_list, # 增量模式所有都是新增 max_news_per_keyword=max_news_per_keyword, sort_by_position_first=sort_by_position_first, timezone=timezone, rank_threshold=self.rank_threshold, quiet=False, ) if not rss_stats: print("[RSS] 增量模式：关键词匹配后没有内容") # 即使关键词匹配为空，也返回原始条目用于独立展示区 return None, None, raw_rss_items, rss_new_urls elif self.report_mode == "current": # 当前榜单模式：统计=当前榜单所有条目 # raw_rss_items 已在前面获取 if not raw_rss_items: print("[RSS] 当前榜单模式：没有 RSS 数据") return None, None, None, rss_new_urls rss_stats, total = count_rss_frequency( rss_items=raw_rss_items, word_groups=word_groups, filter_words=filter_words, global_filters=global_filters, new_items=new_items_list, # 标记新增 max_news_per_keyword=max_news_per_keyword, sort_by_position_first=sort_by_position_first, timezone=timezone, rank_threshold=self.rank_threshold, quiet=False, ) if not rss_stats: print("[RSS] 当前榜单模式：关键词匹配后没有内容") # 即使关键词匹配为空，也返回原始条目用于独立展示区 return None, None, raw_rss_items, rss_new_urls # 生成新增统计 if new_items_list: rss_new_stats, _ = count_rss_frequency( rss_items=new_items_list, word_groups=word_groups, filter_words=filter_words, global_filters=global_filters, new_items=new_items_list, max_news_per_keyword=max_news_per_keyword, sort_by_position_first=sort_by_position_first, timezone=timezone, rank_threshold=self.rank_threshold, quiet=True, ) else: # daily 模式：统计=当天所有条目 # raw_rss_items 已在前面获取 if not raw_rss_items: print("[RSS] 当日汇总模式：没有 RSS 数据") return None, None, None, rss_new_urls rss_stats, total = count_rss_frequency( rss_items=raw_rss_items, word_groups=word_groups, filter_words=filter_words, global_filters=global_filters, new_items=new_items_list, # 标记新增 max_news_per_keyword=max_news_per_keyword, sort_by_position_first=sort_by_position_first, timezone=timezone, rank_threshold=self.rank_threshold, quiet=False, ) if not rss_stats: print("[RSS] 当日汇总模式：关键词匹配后没有内容") # 即使关键词匹配为空，也返回原始条目用于独立展示区 return None, None, raw_rss_items, rss_new_urls # 生成新增统计 if new_items_list: rss_new_stats, _ = count_rss_frequency( rss_items=new_items_list, word_groups=word_groups, filter_words=filter_words, global_filters=global_filters, new_items=new_items_list, max_news_per_keyword=max_news_per_keyword, sort_by_position_first=sort_by_position_first, timezone=timezone, rank_threshold=self.rank_threshold, quiet=True, ) return rss_stats, rss_new_stats, raw_rss_items, rss_new_urls def _convert_rss_items_to_list(self, items_dict: Dict, id_to_name: Dict) -> List[Dict]: """将 RSS 条目字典转换为列表格式，并应用新鲜度过滤（用于推送）""" rss_items = [] filtered_count = 0 filtered_details = [] # 用于 DEBUG 模式下的详细日志 # 获取新鲜度过滤配置 rss_config = self.ctx.rss_config freshness_config = rss_config.get("FRESHNESS_FILTER", {}) freshness_enabled = freshness_config.get("ENABLED", True) default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3) timezone = self.ctx.config.get("TIMEZONE", DEFAULT_TIMEZONE) debug_mode = self.ctx.config.get("DEBUG", False) # 构建 feed_id -> max_age_days 的映射 feed_max_age_map = {} for feed_cfg in self.ctx.rss_feeds: feed_id = feed_cfg.get("id", "") max_age = feed_cfg.get("max_age_days") if max_age is not None: try: feed_max_age_map[feed_id] = int(max_age) except (ValueError, TypeError): pass for feed_id, items in items_dict.items(): # 确定此 feed 的 max_age_days max_days = feed_max_age_map.get(feed_id) if max_days is None: max_days = default_max_age_days for item in items: # 应用新鲜度过滤（仅在启用时） if freshness_enabled and max_days > 0: if item.published_at and not is_within_days(item.published_at, max_days, timezone): filtered_count += 1 # 记录详细信息用于 DEBUG 模式 if debug_mode: days_old = calculate_days_old(item.published_at, timezone) feed_name = id_to_name.get(feed_id, feed_id) filtered_details.append({ "title": item.title[:50] + "..." if len(item.title) > 50 else item.title, "feed": feed_name, "days_old": days_old, "max_days": max_days, }) continue # 跳过超过指定天数的文章 rss_items.append({ "title": item.title, "feed_id": feed_id, "feed_name": id_to_name.get(feed_id, feed_id), "url": item.url, "published_at": item.published_at, "summary": item.summary, "author": item.author, }) # 输出过滤统计 if filtered_count > 0: print(f"[RSS] 新鲜度过滤：跳过 {filtered_count} 篇超过指定天数的旧文章（仍保留在数据库中）") # DEBUG 模式下显示详细信息 if debug_mode and filtered_details: print(f"[RSS] 被过滤的文章详情（共 {len(filtered_details)} 篇）：") for detail in filtered_details[:10]: # 最多显示 10 条 days_str = f"{detail['days_old']:.1f}" if detail['days_old'] else "未知" print(f" - [{days_str}天前] [{detail['feed']}] {detail['title']} (限制: {detail['max_days']}天)") if len(filtered_details) > 10: print(f" ... 还有 {len(filtered_details) - 10} 篇被过滤") return rss_items def _filter_rss_by_keywords(self, rss_items: List[Dict]) -> List[Dict]: """使用关键词文件过滤 RSS 条目""" try: word_groups, filter_words, global_filters = self.ctx.load_frequency_words(self.frequency_file) if word_groups or filter_words or global_filters: from trendradar.core.frequency import matches_word_groups filtered_items = [] for item in rss_items: title = item.get("title", "") if matches_word_groups(title, word_groups, filter_words, global_filters): filtered_items.append(item) original_count = len(rss_items) rss_items = filtered_items print(f"[RSS] 关键词过滤后剩余 {len(rss_items)}/{original_count} 条") if not rss_items: print("[RSS] 关键词过滤后没有匹配内容") return [] except FileNotFoundError: # 关键词文件不存在时跳过过滤 pass return rss_items def _generate_rss_html_report(self, rss_items: list, feeds_info: dict) -> str: """生成 RSS HTML 报告""" try: from trendradar.report.rss_html import render_rss_html_content from pathlib import Path html_content = render_rss_html_content( rss_items=rss_items, total_count=len(rss_items), feeds_info=feeds_info, get_time_func=self.ctx.get_time, ) # 保存 HTML 文件（扁平化结构：output/html/日期/） date_folder = self.ctx.format_date() time_filename = self.ctx.format_time() output_dir = Path("output") / "html" / date_folder output_dir.mkdir(parents=True, exist_ok=True) file_path = output_dir / f"rss_{time_filename}.html" with open(file_path, "w", encoding="utf-8") as f: f.write(html_content) print(f"[RSS] HTML 报告已生成: {file_path}") return str(file_path) except Exception as e: print(f"[RSS] 生成 HTML 报告失败: {e}") return None def _execute_mode_strategy( self, mode_strategy: Dict, results: Dict, id_to_name: Dict, failed_ids: List, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, raw_rss_items: Optional[List[Dict]] = None, rss_new_urls: Optional[set] = None, ) -> Optional[str]: """执行模式特定逻辑，支持热榜+RSS合并推送 简化后的逻辑： - 每次运行都生成 HTML 报告（时间戳快照 + latest/{mode}.html + index.html） - 根据模式发送通知 """ # 调度系统 scheduler = self.ctx.create_scheduler() schedule = scheduler.resolve() # 使用 schedule 决定的 report_mode 覆盖全局配置 effective_mode = schedule.report_mode if effective_mode != self.report_mode: print(f"[调度] 报告模式覆盖: {self.report_mode} -> {effective_mode}") self.report_mode = effective_mode # 重新获取 mode_strategy，确保 report_type 与覆盖后的 report_mode 一致 mode_strategy = self._get_mode_strategy() # 使用 schedule 决定的 frequency_file 覆盖默认值 self.frequency_file = schedule.frequency_file # 使用 schedule 决定的筛选策略覆盖默认值 self.filter_method = schedule.filter_method or self.ctx.filter_method # 使用 schedule 决定的 AI 筛选兴趣文件覆盖默认值 self.interests_file = schedule.interests_file # 如果调度器说不采集，则直接跳过 if not schedule.collect: print("[调度] 当前时间段不执行数据采集，跳过分析流水线") return None # 获取当前监控平台ID列表 current_platform_ids = self.ctx.platform_ids new_titles = self.ctx.detect_new_titles(current_platform_ids) time_info = self.ctx.format_time() word_groups, filter_words, global_filters = self.ctx.load_frequency_words(self.frequency_file) html_file = None stats = [] ai_result = None title_info = None # current 模式需要使用完整的历史数据 if self.report_mode == "current": analysis_data = self._load_analysis_data() if analysis_data: ( all_results, historical_id_to_name, historical_title_info, historical_new_titles, _, _, _, ) = analysis_data print( f"current模式：使用过滤后的历史数据，包含平台：{list(all_results.keys())}" ) # 使用历史数据准备独立展示区数据（包含完整的 title_info） standalone_data = self._prepare_standalone_data( all_results, historical_id_to_name, historical_title_info, raw_rss_items ) stats, html_file, ai_result, rss_items = self._run_analysis_pipeline( all_results, self.report_mode, historical_title_info, historical_new_titles, word_groups, filter_words, historical_id_to_name, failed_ids=failed_ids, global_filters=global_filters, rss_items=rss_items, rss_new_items=rss_new_items, standalone_data=standalone_data, schedule=schedule, rss_new_urls=rss_new_urls, ) combined_id_to_name = {**historical_id_to_name, **id_to_name} new_titles = historical_new_titles id_to_name = combined_id_to_name title_info = historical_title_info results = all_results else: print("❌ 严重错误：无法读取刚保存的数据文件") raise RuntimeError("数据一致性检查失败：保存后立即读取失败") elif self.report_mode == "daily": # daily 模式：使用全天累计数据 analysis_data = self._load_analysis_data() if analysis_data: ( all_results, historical_id_to_name, historical_title_info, historical_new_titles, _, _, _, ) = analysis_data # 使用历史数据准备独立展示区数据（包含完整的 title_info） standalone_data = self._prepare_standalone_data( all_results, historical_id_to_name, historical_title_info, raw_rss_items ) stats, html_file, ai_result, rss_items = self._run_analysis_pipeline( all_results, self.report_mode, historical_title_info, historical_new_titles, word_groups, filter_words, historical_id_to_name, failed_ids=failed_ids, global_filters=global_filters, rss_items=rss_items, rss_new_items=rss_new_items, standalone_data=standalone_data, schedule=schedule, rss_new_urls=rss_new_urls, ) combined_id_to_name = {**historical_id_to_name, **id_to_name} new_titles = historical_new_titles id_to_name = combined_id_to_name title_info = historical_title_info results = all_results else: # 没有历史数据时使用当前数据 title_info = self._prepare_current_title_info(results, time_info) standalone_data = self._prepare_standalone_data( results, id_to_name, title_info, raw_rss_items ) stats, html_file, ai_result, rss_items = self._run_analysis_pipeline( results, self.report_mode, title_info, new_titles, word_groups, filter_words, id_to_name, failed_ids=failed_ids, global_filters=global_filters, rss_items=rss_items, rss_new_items=rss_new_items, standalone_data=standalone_data, schedule=schedule, rss_new_urls=rss_new_urls, ) else: # incremental 模式：只使用当前抓取的数据 title_info = self._prepare_current_title_info(results, time_info) standalone_data = self._prepare_standalone_data( results, id_to_name, title_info, raw_rss_items ) stats, html_file, ai_result, rss_items = self._run_analysis_pipeline( results, self.report_mode, title_info, new_titles, word_groups, filter_words, id_to_name, failed_ids=failed_ids, global_filters=global_filters, rss_items=rss_items, rss_new_items=rss_new_items, standalone_data=standalone_data, schedule=schedule, rss_new_urls=rss_new_urls, ) if html_file: print(f"HTML报告已生成: {html_file}") print(f"最新报告已更新: output/html/latest/{self.report_mode}.html") # 发送通知 if mode_strategy["should_send_notification"]: standalone_data = self._prepare_standalone_data( results, id_to_name, title_info, raw_rss_items ) self._send_notification_if_needed( stats, mode_strategy["report_type"], self.report_mode, failed_ids=failed_ids, new_titles=new_titles, id_to_name=id_to_name, html_file_path=html_file, rss_items=rss_items, rss_new_items=rss_new_items, standalone_data=standalone_data, ai_result=ai_result, current_results=results, schedule=schedule, ) # 打开浏览器（仅在非容器环境） if self._should_open_browser() and html_file: file_url = "file://" + str(Path(html_file).resolve()) print(f"正在打开HTML报告: {file_url}") webbrowser.open(file_url) elif self.is_docker_container and html_file: print(f"HTML报告已生成（Docker环境）: {html_file}") return html_file def run(self) -> None: """执行分析流程""" try: self._initialize_and_check_config() mode_strategy = self._get_mode_strategy() # 抓取热榜数据 results, id_to_name, failed_ids = self._crawl_data() # 抓取 RSS 数据（如果启用），返回统计条目、新增条目和原始条目 rss_items, rss_new_items, raw_rss_items, rss_new_urls = self._crawl_rss_data() # 执行模式策略，传递 RSS 数据用于合并推送 self._execute_mode_strategy( mode_strategy, results, id_to_name, failed_ids, rss_items=rss_items, rss_new_items=rss_new_items, raw_rss_items=raw_rss_items, rss_new_urls=rss_new_urls ) except Exception as e: print(f"分析流程执行出错: {e}") if self.ctx.config.get("DEBUG", False): raise finally: # 清理资源（包括过期数据清理和数据库连接关闭） self.ctx.cleanup() def _record_doctor_result(results: List[Tuple[str, str, str]], status: str, item: str, detail: str) -> None: """记录并打印 doctor 检查结果""" icon_map = { "pass": "✅", "warn": "⚠️", "fail": "❌", } icon = icon_map.get(status, "•") results.append((status, item, detail)) print(f"{icon} {item}: {detail}") def _save_doctor_report( results: List[Tuple[str, str, str]], pass_count: int, warn_count: int, fail_count: int, config_path: Optional[str], ) -> None: """保存 doctor 体检报告到 JSON 文件""" report = { "version": __version__, "generated_at": datetime.now(timezone.utc).isoformat(), "config_path": config_path or os.environ.get("CONFIG_PATH", "config/config.yaml"), "summary": { "pass": pass_count, "warn": warn_count, "fail": fail_count, "ok": fail_count == 0, }, "checks": [ {"status": status, "item": item, "detail": detail} for status, item, detail in results ], } try: output_dir = Path("output") / "meta" output_dir.mkdir(parents=True, exist_ok=True) output_path = output_dir / "doctor_report.json" output_path.write_text( json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8", ) print(f"体检报告已保存: {output_path}") except Exception as e: print(f"⚠️ 体检报告保存失败: {e}") def _run_doctor(config_path: Optional[str] = None) -> bool: """运行环境体检""" print("=" * 60) print(f"TrendRadar v{__version__} 环境体检") print("=" * 60) results: List[Tuple[str, str, str]] = [] config = None # 1) Python 版本检查 py_ok = sys.version_info >= (3, 10) py_version = f"{sys.version_info.major}.{sys.version_info.minor}.{sys.version_info.micro}" if py_ok: _record_doctor_result(results, "pass", "Python版本", f"{py_version} (满足 >= 3.10)") else: _record_doctor_result(results, "fail", "Python版本", f"{py_version} (不满足 >= 3.10)") # 2) 关键文件检查 if config_path is None: config_path = os.environ.get("CONFIG_PATH", "config/config.yaml") required_files = [ (config_path, "主配置文件"), ("config/frequency_words.txt", "关键词文件"), ] optional_files = [ ("config/timeline.yaml", "调度文件"), ] for path_str, desc in required_files: if Path(path_str).exists(): _record_doctor_result(results, "pass", desc, f"已找到: {path_str}") else: _record_doctor_result(results, "fail", desc, f"缺失: {path_str}") for path_str, desc in optional_files: if Path(path_str).exists(): _record_doctor_result(results, "pass", desc, f"已找到: {path_str}") else: _record_doctor_result(results, "warn", desc, f"未找到: {path_str}（将使用默认调度模板）") # 3) 配置加载检查 try: config = load_config(config_path) _record_doctor_result(results, "pass", "配置加载", f"加载成功: {config_path}") except Exception as e: _record_doctor_result(results, "fail", "配置加载", f"加载失败: {e}") # 后续检查依赖配置对象 if config: # 4) 调度配置检查 try: ctx = AppContext(config) schedule = ctx.create_scheduler().resolve() detail = f"调度解析成功（report_mode={schedule.report_mode}, ai_mode={schedule.ai_mode}）" _record_doctor_result(results, "pass", "调度配置", detail) except Exception as e: _record_doctor_result(results, "fail", "调度配置", f"解析失败: {e}") # 5) AI 配置检查（按功能场景区分严重级别） ai_analysis_enabled = config.get("AI_ANALYSIS", {}).get("ENABLED", False) ai_translation_enabled = config.get("AI_TRANSLATION", {}).get("ENABLED", False) ai_filter_enabled = config.get("FILTER", {}).get("METHOD", "keyword") == "ai" ai_enabled = ai_analysis_enabled or ai_translation_enabled or ai_filter_enabled if ai_enabled: try: from trendradar.ai.client import AIClient valid, message = AIClient(config.get("AI", {})).validate_config() if valid: _record_doctor_result(results, "pass", "AI配置", f"模型: {config.get('AI', {}).get('MODEL', '')}") else: # AI 分析/翻译是硬依赖；AI 筛选缺失时会自动回退关键词匹配 if ai_analysis_enabled or ai_translation_enabled: _record_doctor_result(results, "fail", "AI配置", message) else: _record_doctor_result(results, "warn", "AI配置", f"{message}（AI 筛选将回退关键词模式）") except Exception as e: _record_doctor_result(results, "fail", "AI配置", f"校验异常: {e}") else: _record_doctor_result(results, "warn", "AI配置", "未启用 AI 功能，跳过校验") # 6) 存储配置检查 try: storage_cfg = config.get("STORAGE", {}) backend = storage_cfg.get("BACKEND", "auto") remote = storage_cfg.get("REMOTE", {}) missing_remote_keys = [ k for k in ("BUCKET_NAME", "ACCESS_KEY_ID", "SECRET_ACCESS_KEY", "ENDPOINT_URL") if not remote.get(k) ] if backend == "remote" and missing_remote_keys: _record_doctor_result( results, "fail", "存储配置", f"remote 模式缺少配置: {', '.join(missing_remote_keys)}" ) elif backend == "auto" and os.environ.get("GITHUB_ACTIONS") == "true" and missing_remote_keys: _record_doctor_result( results, "warn", "存储配置", "GitHub Actions + auto 模式未完整配置远程存储，可能导致数据丢失" ) else: sm = AppContext(config).get_storage_manager() _record_doctor_result(results, "pass", "存储配置", f"当前后端: {sm.backend_name}") except Exception as e: _record_doctor_result(results, "fail", "存储配置", f"检查失败: {e}") # 7) 通知渠道配置检查 channel_details = [] channel_issues = [] max_accounts = config.get("MAX_ACCOUNTS_PER_CHANNEL", 3) # 普通单值/多值渠道 for key, name in [ ("FEISHU_WEBHOOK_URL", "飞书"), ("DINGTALK_WEBHOOK_URL", "钉钉"), ("WEWORK_WEBHOOK_URL", "企业微信"), ("BARK_URL", "Bark"), ("SLACK_WEBHOOK_URL", "Slack"), ("GENERIC_WEBHOOK_URL", "通用Webhook"), ]: values = parse_multi_account_config(config.get(key, "")) if values: channel_details.append(f"{name}({min(len(values), max_accounts)}个)") # Telegram 配对校验 tg_tokens = parse_multi_account_config(config.get("TELEGRAM_BOT_TOKEN", "")) tg_chats = parse_multi_account_config(config.get("TELEGRAM_CHAT_ID", "")) if tg_tokens or tg_chats: valid, count = validate_paired_configs( {"bot_token": tg_tokens, "chat_id": tg_chats}, "Telegram", required_keys=["bot_token", "chat_id"], ) if valid and count > 0: channel_details.append(f"Telegram({min(count, max_accounts)}个)") else: channel_issues.append("Telegram bot_token/chat_id 配置不完整或数量不一致") # ntfy 配对校验（token 可选） ntfy_server = config.get("NTFY_SERVER_URL", "") ntfy_topics = parse_multi_account_config(config.get("NTFY_TOPIC", "")) ntfy_tokens = parse_multi_account_config(config.get("NTFY_TOKEN", "")) if ntfy_server and ntfy_topics: if ntfy_tokens: valid, count = validate_paired_configs( {"topic": ntfy_topics, "token": ntfy_tokens}, "ntfy", ) if valid and count > 0: channel_details.append(f"ntfy({min(count, max_accounts)}个)") else: channel_issues.append("ntfy topic/token 数量不一致") else: channel_details.append(f"ntfy({min(len(ntfy_topics), max_accounts)}个)") # 邮件配置完整性 email_ready = all( [ config.get("EMAIL_FROM"), config.get("EMAIL_PASSWORD"), config.get("EMAIL_TO"), ] ) if email_ready: channel_details.append("邮件") elif any([config.get("EMAIL_FROM"), config.get("EMAIL_PASSWORD"), config.get("EMAIL_TO")]): channel_issues.append("邮件配置不完整（需要 from/password/to 同时配置）") if channel_issues and not channel_details: _record_doctor_result(results, "fail", "通知配置", "；".join(channel_issues)) elif channel_issues and channel_details: detail = f"可用渠道: {', '.join(channel_details)}；问题: {'；'.join(channel_issues)}" _record_doctor_result(results, "warn", "通知配置", detail) elif channel_details: _record_doctor_result(results, "pass", "通知配置", f"可用渠道: {', '.join(channel_details)}") else: _record_doctor_result(results, "warn", "通知配置", "未配置任何通知渠道") # 8) 输出目录可写检查 try: output_dir = Path("output") output_dir.mkdir(parents=True, exist_ok=True) probe_file = output_dir / ".doctor_write_probe" probe_file.write_text("ok", encoding="utf-8") probe_file.unlink(missing_ok=True) _record_doctor_result(results, "pass", "输出目录", f"可写: {output_dir}") except Exception as e: _record_doctor_result(results, "fail", "输出目录", f"不可写: {e}") pass_count = sum(1 for status, _, _ in results if status == "pass") warn_count = sum(1 for status, _, _ in results if status == "warn") fail_count = sum(1 for status, _, _ in results if status == "fail") _save_doctor_report(results, pass_count, warn_count, fail_count, config_path) print("-" * 60) print(f"体检结果: ✅ {pass_count} 项通过 ⚠️ {warn_count} 项警告 ❌ {fail_count} 项失败") print("=" * 60) if fail_count == 0: print("体检通过。") return True print("体检未通过，请先修复失败项。") return False def _build_test_report_data(ctx: AppContext) -> Dict: """构造通知测试用报告数据""" now = ctx.get_time() time_display = now.strftime("%H:%M") title = f"TrendRadar 通知测试消息（{now.strftime('%Y-%m-%d %H:%M:%S')}）" return { "stats": [ { "word": "连通性测试", "count": 1, "titles": [ { "title": title, "source_name": "TrendRadar", "url": "https://github.com/sansan0/TrendRadar", "mobile_url": "", "ranks": [1], "rank_threshold": ctx.rank_threshold, "count": 1, "is_new": True, "time_display": time_display, "matched_keyword": "连通性测试", } ], } ], "failed_ids": [], "new_titles": [], "id_to_name": {}, } def _create_test_html_file(ctx: AppContext) -> Optional[str]: """创建邮件测试用 HTML 文件""" try: now = ctx.get_time() output_dir = Path("output") / "html" / ctx.format_date() output_dir.mkdir(parents=True, exist_ok=True) html_path = output_dir / f"notification_test_{ctx.format_time()}.html" html_content = f""" TrendRadar 通知测试

TrendRadar 通知连通性测试

测试时间：{now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})

这是一条测试消息，用于验证邮件渠道是否可达。

""" html_path.write_text(html_content, encoding="utf-8") return str(html_path) except Exception as e: print(f"[测试通知] 创建测试 HTML 失败: {e}") return None def _run_test_notification(config: Dict) -> bool: """发送测试通知到已配置渠道""" from trendradar.notification import NotificationDispatcher ctx = AppContext(config) try: # 检查是否配置了通知渠道 has_notification = any( [ config.get("FEISHU_WEBHOOK_URL"), config.get("DINGTALK_WEBHOOK_URL"), config.get("WEWORK_WEBHOOK_URL"), (config.get("TELEGRAM_BOT_TOKEN") and config.get("TELEGRAM_CHAT_ID")), (config.get("EMAIL_FROM") and config.get("EMAIL_PASSWORD") and config.get("EMAIL_TO")), (config.get("NTFY_SERVER_URL") and config.get("NTFY_TOPIC")), config.get("BARK_URL"), config.get("SLACK_WEBHOOK_URL"), config.get("GENERIC_WEBHOOK_URL"), ] ) if not has_notification: print("未检测到可用通知渠道，请先在 config.yaml 或环境变量中配置。") return False # 测试时固定展示区域，避免用户关闭 HOTLIST 导致测试内容为空 test_config = copy.deepcopy(config) test_display = test_config.setdefault("DISPLAY", {}) test_regions = test_display.setdefault("REGIONS", {}) test_regions.update( { "HOTLIST": True, "NEW_ITEMS": False, "RSS": False, "STANDALONE": False, "AI_ANALYSIS": False, } ) # 测试时禁用翻译，避免触发额外 AI 调用 if "AI_TRANSLATION" in test_config: test_config["AI_TRANSLATION"]["ENABLED"] = False proxy_url = test_config.get("DEFAULT_PROXY", "") if test_config.get("USE_PROXY") else None if proxy_url: print("[测试通知] 检测到代理配置，将使用代理发送") dispatcher = NotificationDispatcher( config=test_config, get_time_func=ctx.get_time, split_content_func=ctx.split_content, translator=None, ) report_data = _build_test_report_data(ctx) html_file_path = _create_test_html_file(ctx) print("=" * 60) print("通知连通性测试") print("=" * 60) results = dispatcher.dispatch_all( report_data=report_data, report_type="通知连通性测试", proxy_url=proxy_url, mode="daily", html_file_path=html_file_path, ) if not results: print("没有可测试的有效通知渠道（可能配置不完整）。") return False print("-" * 60) success_count = 0 for channel, ok in results.items(): if ok: success_count += 1 print(f"✅ {channel}: 测试成功") else: print(f"❌ {channel}: 测试失败") print("-" * 60) print(f"测试结果: {success_count}/{len(results)} 个渠道成功") return success_count > 0 finally: ctx.cleanup() def main(): """主程序入口""" # 解析命令行参数 parser = argparse.ArgumentParser( description="TrendRadar - 热点新闻聚合与分析工具", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" 调度状态命令: --show-schedule 显示当前调度状态（时间段、行为开关） 诊断命令: --doctor 运行环境与配置体检 --test-notification 发送测试通知到已配置渠道 示例: python -m trendradar # 正常运行 python -m trendradar --show-schedule # 查看当前调度状态 python -m trendradar --doctor # 运行一键体检 python -m trendradar --test-notification # 测试通知渠道连通性 """ ) parser.add_argument( "--show-schedule", action="store_true", help="显示当前调度状态" ) parser.add_argument( "--doctor", action="store_true", help="运行环境与配置体检" ) parser.add_argument( "--test-notification", action="store_true", help="发送测试通知到已配置渠道" ) args = parser.parse_args() debug_mode = False try: # 处理 doctor 命令（不依赖完整运行流程） if args.doctor: ok = _run_doctor() if not ok: raise SystemExit(1) return # 先加载配置 config = load_config() # 处理状态查看命令 if args.show_schedule: _handle_status_commands(config) return # 处理通知测试命令 if args.test_notification: ok = _run_test_notification(config) if not ok: raise SystemExit(1) return version_url = config.get("VERSION_CHECK_URL", "") configs_version_url = config.get("CONFIGS_VERSION_CHECK_URL", "") # 统一版本检查（程序版本 + 配置文件版本，只请求一次远程） need_update = False remote_version = None if version_url: need_update, remote_version = check_all_versions(version_url, configs_version_url) # 复用已加载的配置，避免重复加载 analyzer = NewsAnalyzer(config=config) # 设置更新信息（复用已获取的远程版本，不再重复请求） if analyzer.is_github_actions and need_update and remote_version: analyzer.update_info = { "current_version": __version__, "remote_version": remote_version, } # 获取 debug 配置 debug_mode = analyzer.ctx.config.get("DEBUG", False) analyzer.run() except FileNotFoundError as e: print(f"❌ 配置文件错误: {e}") print("\n请确保以下文件存在:") print(" • config/config.yaml") print(" • config/frequency_words.txt") print("\n参考项目文档进行正确配置") except Exception as e: print(f"❌ 程序运行错误: {e}") if debug_mode: raise def _handle_status_commands(config: Dict) -> None: """处理状态查看命令 - 显示当前调度状态""" from trendradar.context import AppContext ctx = AppContext(config) print("=" * 60) print(f"TrendRadar v{__version__} 调度状态") print("=" * 60) try: scheduler = ctx.create_scheduler() schedule = scheduler.resolve() now = ctx.get_time() date_str = ctx.format_date() print(f"\n⏰ 当前时间: {now.strftime('%Y-%m-%d %H:%M:%S')} ({ctx.timezone})") print(f"📅 当前日期: {date_str}") print(f"\n📋 调度信息:") print(f" 日计划: {schedule.day_plan}") if schedule.period_key: print(f" 当前时间段: {schedule.period_name or schedule.period_key} ({schedule.period_key})") else: print(f" 当前时间段: 无（使用默认配置）") print(f"\n🔧 行为开关:") print(f" 采集数据: {'✅ 是' if schedule.collect else '❌ 否'}") print(f" AI 分析: {'✅ 是' if schedule.analyze else '❌ 否'}") print(f" 推送通知: {'✅ 是' if schedule.push else '❌ 否'}") print(f" 报告模式: {schedule.report_mode}") print(f" AI 模式: {schedule.ai_mode}") if schedule.period_key: print(f"\n🔁 一次性控制:") if schedule.once_analyze: already_analyzed = scheduler.already_executed(schedule.period_key, "analyze", date_str) print(f" AI 分析: 仅一次 {'(今日已执行 ⚠️)' if already_analyzed else '(今日未执行 ✅)'}") else: print(f" AI 分析: 不限次数") if schedule.once_push: already_pushed = scheduler.already_executed(schedule.period_key, "push", date_str) print(f" 推送通知: 仅一次 {'(今日已执行 ⚠️)' if already_pushed else '(今日未执行 ✅)'}") else: print(f" 推送通知: 不限次数") except Exception as e: print(f"\n❌ 获取调度状态失败: {e}") print("\n" + "=" * 60) # 清理资源 ctx.cleanup() if __name__ == "__main__": main() ================================================ FILE: trendradar/ai/__init__.py ================================================ # coding=utf-8 """ TrendRadar AI 模块 提供 AI 大模型对热点新闻的深度分析和翻译功能 """ from .analyzer import AIAnalyzer, AIAnalysisResult from .filter import AIFilter, AIFilterResult from .translator import AITranslator, TranslationResult, BatchTranslationResult from .formatter import ( get_ai_analysis_renderer, render_ai_analysis_markdown, render_ai_analysis_feishu, render_ai_analysis_dingtalk, render_ai_analysis_html, render_ai_analysis_html_rich, render_ai_analysis_plain, ) __all__ = [ # 分析器 "AIAnalyzer", "AIAnalysisResult", # 智能筛选 "AIFilter", "AIFilterResult", # 翻译器 "AITranslator", "TranslationResult", "BatchTranslationResult", # 格式化 "get_ai_analysis_renderer", "render_ai_analysis_markdown", "render_ai_analysis_feishu", "render_ai_analysis_dingtalk", "render_ai_analysis_html", "render_ai_analysis_html_rich", "render_ai_analysis_plain", ] ================================================ FILE: trendradar/ai/analyzer.py ================================================ # coding=utf-8 """ AI 分析器模块 调用 AI 大模型对热点新闻进行深度分析 基于 LiteLLM 统一接口，支持 100+ AI 提供商 """ import json from dataclasses import dataclass, field from pathlib import Path from typing import Any, Callable, Dict, List, Optional from trendradar.ai.client import AIClient @dataclass class AIAnalysisResult: """AI 分析结果""" # 新版 5 核心板块 core_trends: str = "" # 核心热点与舆情态势 sentiment_controversy: str = "" # 舆论风向与争议 signals: str = "" # 异动与弱信号 rss_insights: str = "" # RSS 深度洞察 outlook_strategy: str = "" # 研判与策略建议 standalone_summaries: Dict[str, str] = field(default_factory=dict) # 独立展示区概括 {源ID: 概括} # 基础元数据 raw_response: str = "" # 原始响应 success: bool = False # 是否成功 error: str = "" # 错误信息 # 新闻数量统计 total_news: int = 0 # 总新闻数（热榜+RSS） analyzed_news: int = 0 # 实际分析的新闻数 max_news_limit: int = 0 # 分析上限配置值 hotlist_count: int = 0 # 热榜新闻数 rss_count: int = 0 # RSS 新闻数 ai_mode: str = "" # AI 分析使用的模式 (daily/current/incremental) class AIAnalyzer: """AI 分析器""" def __init__( self, ai_config: Dict[str, Any], analysis_config: Dict[str, Any], get_time_func: Callable, debug: bool = False, ): """ 初始化 AI 分析器 Args: ai_config: AI 模型配置（LiteLLM 格式） analysis_config: AI 分析功能配置（language, prompt_file 等） get_time_func: 获取当前时间的函数 debug: 是否开启调试模式 """ self.ai_config = ai_config self.analysis_config = analysis_config self.get_time_func = get_time_func self.debug = debug # 创建 AI 客户端（基于 LiteLLM） self.client = AIClient(ai_config) # 验证配置 valid, error = self.client.validate_config() if not valid: print(f"[AI] 配置警告: {error}") # 从分析配置获取功能参数 self.max_news = analysis_config.get("MAX_NEWS_FOR_ANALYSIS", 50) self.include_rss = analysis_config.get("INCLUDE_RSS", True) self.include_rank_timeline = analysis_config.get("INCLUDE_RANK_TIMELINE", False) self.include_standalone = analysis_config.get("INCLUDE_STANDALONE", False) self.language = analysis_config.get("LANGUAGE", "Chinese") # 加载提示词模板 self.system_prompt, self.user_prompt_template = self._load_prompt_template( analysis_config.get("PROMPT_FILE", "ai_analysis_prompt.txt") ) def _load_prompt_template(self, prompt_file: str) -> tuple: """加载提示词模板""" config_dir = Path(__file__).parent.parent.parent / "config" prompt_path = config_dir / prompt_file if not prompt_path.exists(): print(f"[AI] 提示词文件不存在: {prompt_path}") return "", "" content = prompt_path.read_text(encoding="utf-8") # 解析 [system] 和 [user] 部分 system_prompt = "" user_prompt = "" if "[system]" in content and "[user]" in content: parts = content.split("[user]") system_part = parts[0] user_part = parts[1] if len(parts) > 1 else "" # 提取 system 内容 if "[system]" in system_part: system_prompt = system_part.split("[system]")[1].strip() user_prompt = user_part.strip() else: # 整个文件作为 user prompt user_prompt = content return system_prompt, user_prompt def analyze( self, stats: List[Dict], rss_stats: Optional[List[Dict]] = None, report_mode: str = "daily", report_type: str = "当日汇总", platforms: Optional[List[str]] = None, keywords: Optional[List[str]] = None, standalone_data: Optional[Dict] = None, ) -> AIAnalysisResult: """ 执行 AI 分析 Args: stats: 热榜统计数据 rss_stats: RSS 统计数据 report_mode: 报告模式 report_type: 报告类型 platforms: 平台列表 keywords: 关键词列表 Returns: AIAnalysisResult: 分析结果 """ # 打印配置信息方便调试 model = self.ai_config.get("MODEL", "unknown") api_key = self.client.api_key or "" api_base = self.ai_config.get("API_BASE", "") masked_key = f"{api_key[:5]}******" if len(api_key) >= 5 else "******" model_display = model.replace("/", "/\u200b") if model else "unknown" print(f"[AI] 模型: {model_display}") print(f"[AI] Key : {masked_key}") if api_base: print(f"[AI] 接口: 存在自定义 API 端点") timeout = self.ai_config.get("TIMEOUT", 120) max_tokens = self.ai_config.get("MAX_TOKENS", 5000) print(f"[AI] 参数: timeout={timeout}, max_tokens={max_tokens}") if not self.client.api_key: return AIAnalysisResult( success=False, error="未配置 AI API Key，请在 config.yaml 或环境变量 AI_API_KEY 中设置" ) # 准备新闻内容并获取统计数据 news_content, rss_content, hotlist_total, rss_total, analyzed_count = self._prepare_news_content(stats, rss_stats) total_news = hotlist_total + rss_total if not news_content and not rss_content: return AIAnalysisResult( success=False, error="没有可分析的新闻内容", total_news=total_news, hotlist_count=hotlist_total, rss_count=rss_total, analyzed_news=0, max_news_limit=self.max_news ) # 构建提示词 current_time = self.get_time_func().strftime("%Y-%m-%d %H:%M:%S") # 提取关键词 if not keywords: keywords = [s.get("word", "") for s in stats if s.get("word")] if stats else [] # 使用安全的字符串替换，避免模板中其他花括号（如 JSON 示例）被误解析 user_prompt = self.user_prompt_template user_prompt = user_prompt.replace("{report_mode}", report_mode) user_prompt = user_prompt.replace("{report_type}", report_type) user_prompt = user_prompt.replace("{current_time}", current_time) user_prompt = user_prompt.replace("{news_count}", str(hotlist_total)) user_prompt = user_prompt.replace("{rss_count}", str(rss_total)) user_prompt = user_prompt.replace("{platforms}", ", ".join(platforms) if platforms else "多平台") user_prompt = user_prompt.replace("{keywords}", ", ".join(keywords[:20]) if keywords else "无") user_prompt = user_prompt.replace("{news_content}", news_content) user_prompt = user_prompt.replace("{rss_content}", rss_content) user_prompt = user_prompt.replace("{language}", self.language) # 构建独立展示区内容 standalone_content = "" if self.include_standalone and standalone_data: standalone_content = self._prepare_standalone_content(standalone_data) user_prompt = user_prompt.replace("{standalone_content}", standalone_content) if self.debug: print("\n" + "=" * 80) print("[AI 调试] 发送给 AI 的完整提示词") print("=" * 80) if self.system_prompt: print("\n--- System Prompt ---") print(self.system_prompt) print("\n--- User Prompt ---") print(user_prompt) print("=" * 80 + "\n") # 调用 AI API（使用 LiteLLM） try: response = self._call_ai(user_prompt) result = self._parse_response(response) # JSON 解析失败时的重试兜底（仅重试一次） if result.error and "JSON 解析错误" in result.error: print(f"[AI] JSON 解析失败，尝试让 AI 修复...") retry_result = self._retry_fix_json(response, result.error) if retry_result and retry_result.success and not retry_result.error: print("[AI] JSON 修复成功") retry_result.raw_response = response result = retry_result else: print("[AI] JSON 修复失败，使用原始文本兜底") # 如果配置未启用 RSS 分析，强制清空 AI 返回的 RSS 洞察 if not self.include_rss: result.rss_insights = "" # 如果配置未启用 standalone 分析，强制清空 if not self.include_standalone: result.standalone_summaries = {} # 填充统计数据 result.total_news = total_news result.hotlist_count = hotlist_total result.rss_count = rss_total result.analyzed_news = analyzed_count result.max_news_limit = self.max_news return result except Exception as e: error_type = type(e).__name__ error_msg = str(e) # 截断过长的错误消息 if len(error_msg) > 200: error_msg = error_msg[:200] + "..." friendly_msg = f"AI 分析失败 ({error_type}): {error_msg}" return AIAnalysisResult( success=False, error=friendly_msg ) def _prepare_news_content( self, stats: List[Dict], rss_stats: Optional[List[Dict]] = None, ) -> tuple: """ 准备新闻内容文本（增强版） 热榜新闻包含：来源、标题、排名范围、时间范围、出现次数 RSS 包含：来源、标题、发布时间 Returns: tuple: (news_content, rss_content, hotlist_total, rss_total, analyzed_count) """ news_lines = [] rss_lines = [] news_count = 0 rss_count = 0 # 计算总新闻数 hotlist_total = sum(len(s.get("titles", [])) for s in stats) if stats else 0 rss_total = sum(len(s.get("titles", [])) for s in rss_stats) if rss_stats else 0 # 热榜内容 if stats: for stat in stats: word = stat.get("word", "") titles = stat.get("titles", []) if word and titles: news_lines.append(f"\n**{word}** ({len(titles)}条)") for t in titles: if not isinstance(t, dict): continue title = t.get("title", "") if not title: continue # 来源 source = t.get("source_name", t.get("source", "")) # 构建行 if source: line = f"- [{source}] {title}" else: line = f"- {title}" # 始终显示简化格式：排名范围 + 时间范围 + 出现次数 ranks = t.get("ranks", []) if ranks: min_rank = min(ranks) max_rank = max(ranks) rank_str = f"{min_rank}" if min_rank == max_rank else f"{min_rank}-{max_rank}" else: rank_str = "-" first_time = t.get("first_time", "") last_time = t.get("last_time", "") time_str = self._format_time_range(first_time, last_time) appear_count = t.get("count", 1) line += f" | 排名:{rank_str} | 时间:{time_str} | 出现:{appear_count}次" # 开启完整时间线时，额外添加轨迹 if self.include_rank_timeline: rank_timeline = t.get("rank_timeline", []) timeline_str = self._format_rank_timeline(rank_timeline) line += f" | 轨迹:{timeline_str}" news_lines.append(line) news_count += 1 if news_count >= self.max_news: break if news_count >= self.max_news: break # RSS 内容（仅在启用时构建） if self.include_rss and rss_stats: remaining = self.max_news - news_count for stat in rss_stats: if rss_count >= remaining: break word = stat.get("word", "") titles = stat.get("titles", []) if word and titles: rss_lines.append(f"\n**{word}** ({len(titles)}条)") for t in titles: if not isinstance(t, dict): continue title = t.get("title", "") if not title: continue # 来源 source = t.get("source_name", t.get("feed_name", "")) # 发布时间 time_display = t.get("time_display", "") # 构建行：[来源] 标题 | 发布时间 if source: line = f"- [{source}] {title}" else: line = f"- {title}" if time_display: line += f" | {time_display}" rss_lines.append(line) rss_count += 1 if rss_count >= remaining: break news_content = "\n".join(news_lines) if news_lines else "" rss_content = "\n".join(rss_lines) if rss_lines else "" total_count = news_count + rss_count return news_content, rss_content, hotlist_total, rss_total, total_count def _call_ai(self, user_prompt: str) -> str: """调用 AI API（使用 LiteLLM）""" messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": user_prompt}) return self.client.chat(messages) def _retry_fix_json(self, original_response: str, error_msg: str) -> Optional[AIAnalysisResult]: """ JSON 解析失败时，请求 AI 修复 JSON（仅重试一次） 使用轻量 prompt，不重复原始分析的 system prompt，节省 token。 Args: original_response: AI 原始响应（JSON 格式有误） error_msg: JSON 解析的错误信息 Returns: 修复后的分析结果，失败时返回 None """ messages = [ { "role": "system", "content": ( "你是一个 JSON 修复助手。用户会提供一段格式有误的 JSON 和错误信息，" "你需要修复 JSON 格式错误并返回正确的 JSON。\n" "常见问题：字符串值内的双引号未转义、缺少逗号、字符串未正确闭合等。\n" "只返回纯 JSON，不要包含 markdown 代码块标记（如 ```json）或任何说明文字。" ), }, { "role": "user", "content": ( f"以下 JSON 解析失败：\n\n" f"错误：{error_msg}\n\n" f"原始内容：\n{original_response}\n\n" f"请修复以上 JSON 中的格式问题（如值中的双引号改用中文引号「」或转义 \\\"、" f"缺少逗号、不完整的字符串等），保持原始内容语义不变，只修复格式。" f"直接返回修复后的纯 JSON。" ), }, ] try: response = self.client.chat(messages) return self._parse_response(response) except Exception as e: print(f"[AI] 重试修复 JSON 异常: {type(e).__name__}: {e}") return None def _format_time_range(self, first_time: str, last_time: str) -> str: """格式化时间范围（简化显示，只保留时分）""" def extract_time(time_str: str) -> str: if not time_str: return "-" # 尝试提取 HH:MM 部分 if " " in time_str: parts = time_str.split(" ") if len(parts) >= 2: time_part = parts[1] if ":" in time_part: return time_part[:5] # HH:MM elif ":" in time_str: return time_str[:5] # 处理 HH-MM 格式 result = time_str[:5] if len(time_str) >= 5 else time_str if len(result) == 5 and result[2] == '-': result = result.replace('-', ':') return result first = extract_time(first_time) last = extract_time(last_time) if first == last or last == "-": return first return f"{first}~{last}" def _format_rank_timeline(self, rank_timeline: List[Dict]) -> str: """格式化排名时间线""" if not rank_timeline: return "-" parts = [] for item in rank_timeline: time_str = item.get("time", "") if len(time_str) == 5 and time_str[2] == '-': time_str = time_str.replace('-', ':') rank = item.get("rank") if rank is None: parts.append(f"0({time_str})") else: parts.append(f"{rank}({time_str})") return "→".join(parts) def _prepare_standalone_content(self, standalone_data: Dict) -> str: """ 将独立展示区数据转为文本，注入 AI 分析 prompt Args: standalone_data: 独立展示区数据 {"platforms": [...], "rss_feeds": [...]} Returns: 格式化的文本内容 """ lines = [] # 热榜平台 for platform in standalone_data.get("platforms", []): platform_id = platform.get("id", "") platform_name = platform.get("name", platform_id) items = platform.get("items", []) if not items: continue lines.append(f"### [{platform_name}]") for item in items: title = item.get("title", "") if not title: continue line = f"- {title}" # 排名信息 ranks = item.get("ranks", []) if ranks: min_rank = min(ranks) max_rank = max(ranks) rank_str = f"{min_rank}" if min_rank == max_rank else f"{min_rank}-{max_rank}" line += f" | 排名:{rank_str}" # 时间范围 first_time = item.get("first_time", "") last_time = item.get("last_time", "") if first_time: time_str = self._format_time_range(first_time, last_time) line += f" | 时间:{time_str}" # 出现次数 count = item.get("count", 1) if count > 1: line += f" | 出现:{count}次" # 排名轨迹（如果启用） if self.include_rank_timeline: rank_timeline = item.get("rank_timeline", []) if rank_timeline: timeline_str = self._format_rank_timeline(rank_timeline) line += f" | 轨迹:{timeline_str}" lines.append(line) lines.append("") # RSS 源 for feed in standalone_data.get("rss_feeds", []): feed_id = feed.get("id", "") feed_name = feed.get("name", feed_id) items = feed.get("items", []) if not items: continue lines.append(f"### [{feed_name}]") for item in items: title = item.get("title", "") if not title: continue line = f"- {title}" published_at = item.get("published_at", "") if published_at: line += f" | {published_at}" lines.append(line) lines.append("") return "\n".join(lines) def _parse_response(self, response: str) -> AIAnalysisResult: """解析 AI 响应""" result = AIAnalysisResult(raw_response=response) if not response or not response.strip(): result.error = "AI 返回空响应" return result # 提取 JSON 文本（去掉 markdown 代码块标记） json_str = response if "```json" in response: parts = response.split("```json", 1) if len(parts) > 1: code_block = parts[1] end_idx = code_block.find("```") if end_idx != -1: json_str = code_block[:end_idx] else: json_str = code_block elif "```" in response: parts = response.split("```", 2) if len(parts) >= 2: json_str = parts[1] json_str = json_str.strip() if not json_str: result.error = "提取的 JSON 内容为空" result.core_trends = response[:500] + "..." if len(response) > 500 else response result.success = True return result # 第一步：标准 JSON 解析 data = None parse_error = None try: data = json.loads(json_str) except json.JSONDecodeError as e: parse_error = e # 第二步：json_repair 本地修复 if data is None: try: from json_repair import repair_json repaired = repair_json(json_str, return_objects=True) if isinstance(repaired, dict): data = repaired print("[AI] JSON 本地修复成功（json_repair）") except Exception: pass # 两步都失败，记录错误（后续由 analyze 方法的重试机制处理） if data is None: if parse_error: error_context = json_str[max(0, parse_error.pos - 30):parse_error.pos + 30] if json_str and parse_error.pos else "" result.error = f"JSON 解析错误 (位置 {parse_error.pos}): {parse_error.msg}" if error_context: result.error += f"，上下文: ...{error_context}..." else: result.error = "JSON 解析失败" # 兜底：使用已提取的 json_str（不含 markdown 标记），避免推送中出现 ```json result.core_trends = json_str[:500] + "..." if len(json_str) > 500 else json_str result.success = True return result # 解析成功，提取字段 try: result.core_trends = data.get("core_trends", "") result.sentiment_controversy = data.get("sentiment_controversy", "") result.signals = data.get("signals", "") result.rss_insights = data.get("rss_insights", "") result.outlook_strategy = data.get("outlook_strategy", "") # 解析独立展示区概括 summaries = data.get("standalone_summaries", {}) if isinstance(summaries, dict): result.standalone_summaries = { str(k): str(v) for k, v in summaries.items() } result.success = True except (KeyError, TypeError, AttributeError) as e: result.error = f"字段提取错误: {type(e).__name__}: {e}" result.core_trends = json_str[:500] + "..." if len(json_str) > 500 else json_str result.success = True return result ================================================ FILE: trendradar/ai/client.py ================================================ # coding=utf-8 """ AI 客户端模块 基于 LiteLLM 的统一 AI 模型接口 支持 100+ AI 提供商（OpenAI、DeepSeek、Gemini、Claude、国内模型等） """ import os from typing import Any, Dict, List from litellm import completion class AIClient: """统一的 AI 客户端（基于 LiteLLM）""" def __init__(self, config: Dict[str, Any]): """ 初始化 AI 客户端 Args: config: AI 配置字典 - MODEL: 模型标识（格式: provider/model_name） - API_KEY: API 密钥 - API_BASE: API 基础 URL（可选） - TEMPERATURE: 采样温度 - MAX_TOKENS: 最大生成 token 数 - TIMEOUT: 请求超时时间（秒） - NUM_RETRIES: 重试次数（可选） - FALLBACK_MODELS: 备用模型列表（可选） """ self.model = config.get("MODEL", "deepseek/deepseek-chat") self.api_key = config.get("API_KEY") or os.environ.get("AI_API_KEY", "") self.api_base = config.get("API_BASE", "") self.temperature = config.get("TEMPERATURE", 1.0) self.max_tokens = config.get("MAX_TOKENS", 5000) self.timeout = config.get("TIMEOUT", 120) self.num_retries = config.get("NUM_RETRIES", 2) self.fallback_models = config.get("FALLBACK_MODELS", []) def chat( self, messages: List[Dict[str, str]], **kwargs ) -> str: """ 调用 AI 模型进行对话 Args: messages: 消息列表，格式: [{"role": "system/user/assistant", "content": "..."}] **kwargs: 额外参数，会覆盖默认配置 Returns: str: AI 响应内容 Raises: Exception: API 调用失败时抛出异常 """ # 构建请求参数 params = { "model": self.model, "messages": messages, "temperature": kwargs.get("temperature", self.temperature), "timeout": kwargs.get("timeout", self.timeout), "num_retries": kwargs.get("num_retries", self.num_retries), } # 添加 API Key if self.api_key: params["api_key"] = self.api_key # 添加 API Base（如果配置了） if self.api_base: params["api_base"] = self.api_base # 添加 max_tokens（如果配置了且不为 0） max_tokens = kwargs.get("max_tokens", self.max_tokens) if max_tokens and max_tokens > 0: params["max_tokens"] = max_tokens # 添加 fallback 模型（如果配置了） if self.fallback_models: params["fallbacks"] = self.fallback_models # 合并其他额外参数 for key, value in kwargs.items(): if key not in params: params[key] = value # 调用 LiteLLM response = completion(**params) # 提取响应内容 # 某些模型/提供商返回 list（内容块）而非 str，统一转为 str content = response.choices[0].message.content if isinstance(content, list): content = "\n".join( item.get("text", str(item)) if isinstance(item, dict) else str(item) for item in content ) return content or "" def validate_config(self) -> tuple[bool, str]: """ 验证配置是否有效 Returns: tuple: (是否有效, 错误信息) """ if not self.model: return False, "未配置 AI 模型（model）" if not self.api_key: return False, "未配置 AI API Key，请在 config.yaml 或环境变量 AI_API_KEY 中设置" # 验证模型格式（应该包含 provider/model） if "/" not in self.model: return False, f"模型格式错误: {self.model}，应为 'provider/model' 格式（如 'deepseek/deepseek-chat'）" return True, "" ================================================ FILE: trendradar/ai/filter.py ================================================ # coding=utf-8 """ AI 智能筛选模块 通过 AI 对新闻进行标签分类： 1. 阶段 A：从用户兴趣描述中提取结构化标签 2. 阶段 B：对新闻标题按标签进行批量分类 """ import hashlib import json from dataclasses import dataclass, field from pathlib import Path from typing import Any, Callable, Dict, List, Optional from trendradar.ai.client import AIClient @dataclass class AIFilterResult: """AI 筛选结果，传给报告和通知模块""" tags: List[Dict] = field(default_factory=list) # [{"tag": str, "description": str, "count": int, "items": [ # {"title": str, "source_id": str, "source_name": str, # "url": str, "mobile_url": str, "rank": int, "ranks": [...], # "first_time": str, "last_time": str, "count": int, # "relevance_score": float, "source_type": str} # ]}] total_matched: int = 0 # 匹配新闻总数 total_processed: int = 0 # 处理新闻总数 success: bool = False error: str = "" class AIFilter: """AI 智能筛选器""" def __init__( self, ai_config: Dict[str, Any], filter_config: Dict[str, Any], get_time_func: Callable, debug: bool = False, ): self.client = AIClient(ai_config) self.filter_config = filter_config self.batch_size = filter_config.get("BATCH_SIZE", 200) self.get_time_func = get_time_func self.debug = debug # 加载提示词模板 self.classify_system, self.classify_user = self._load_prompt( filter_config.get("PROMPT_FILE", "ai_filter_prompt.txt") ) self.extract_system, self.extract_user = self._load_prompt( filter_config.get("EXTRACT_PROMPT_FILE", "ai_filter_extract_prompt.txt") ) self.update_tags_system, self.update_tags_user = self._load_prompt( filter_config.get("UPDATE_TAGS_PROMPT_FILE", "update_tags_prompt.txt") ) def _load_prompt(self, filename: str) -> tuple: """加载提示词文件，返回 (system_prompt, user_prompt_template)""" config_dir = Path(__file__).parent.parent.parent / "config" / "ai_filter" prompt_path = config_dir / filename if not prompt_path.exists(): print(f"[AI筛选] 提示词文件不存在: {prompt_path}") return "", "" content = prompt_path.read_text(encoding="utf-8") system_prompt = "" user_prompt = "" if "[system]" in content and "[user]" in content: parts = content.split("[user]") system_part = parts[0] user_part = parts[1] if len(parts) > 1 else "" if "[system]" in system_part: system_prompt = system_part.split("[system]")[1].strip() user_prompt = user_part.strip() else: user_prompt = content return system_prompt, user_prompt def compute_interests_hash(self, interests_content: str, filename: str = "ai_interests.txt") -> str: """计算兴趣描述的 hash，格式为 filename:md5""" # 去除前后空白和注释行，确保内容变化才改变 hash lines = [] for line in interests_content.strip().splitlines(): line = line.strip() if line and not line.startswith("#"): lines.append(line) normalized = "\n".join(lines) content_hash = hashlib.md5(normalized.encode("utf-8")).hexdigest() return f"{filename}:{content_hash}" def load_interests_content(self, interests_file: Optional[str] = None) -> Optional[str]: """加载兴趣描述文件内容 解析逻辑： - interests_file 为 None：使用默认 config/ai_interests.txt - interests_file 有值：仅查 config/custom/ai/{filename} 注意：调用方（context.py）已完成 config/timeline 的合并决策， 此处不再二次读取 filter_config，避免语义冲突。 """ config_dir = Path(__file__).parent.parent.parent / "config" configured_file = interests_file if configured_file: # 自定义兴趣文件：仅查 custom/ai 目录 filename = configured_file interests_path = config_dir / "custom" / "ai" / filename if not interests_path.exists(): print(f"[AI筛选] 自定义兴趣描述文件不存在: {filename}") print(f"[AI筛选] 已查找: {interests_path}") return None else: # 默认兴趣文件：固定使用 config/ai_interests.txt filename = "ai_interests.txt" interests_path = config_dir / filename if not interests_path.exists(): print(f"[AI筛选] 默认兴趣描述文件不存在: {filename}") print(f"[AI筛选] 已查找: {interests_path}") return None if not interests_path.exists(): print(f"[AI筛选] 兴趣描述文件不存在: {interests_path}") return None content = interests_path.read_text(encoding="utf-8").strip() if not content: print("[AI筛选] 兴趣描述文件为空") return None return content def extract_tags(self, interests_content: str) -> List[Dict]: """ 阶段 A：从兴趣描述中提取结构化标签 Args: interests_content: 用户的兴趣描述文本 Returns: [{"tag": str, "description": str}, ...] """ if not self.extract_user: print("[AI筛选] 标签提取提示词模板为空") return [] user_prompt = self.extract_user.replace("{interests_content}", interests_content) messages = [] if self.extract_system: messages.append({"role": "system", "content": self.extract_system}) messages.append({"role": "user", "content": user_prompt}) if self.debug: print(f"\n[AI筛选][DEBUG] === 标签提取 Prompt ===") for m in messages: print(f"[{m['role']}]\n{m['content']}") print(f"[AI筛选][DEBUG] === Prompt 结束 ===") try: response = self.client.chat(messages) if self.debug: print(f"\n[AI筛选][DEBUG] === 标签提取 AI 原始响应 ===") # 尝试格式化 JSON 便于阅读 self._print_formatted_json(response) print(f"[AI筛选][DEBUG] === 响应结束 ===") tags = self._parse_tags_response(response) print(f"[AI筛选] 提取到 {len(tags)} 个标签") for t in tags: print(f" {t['tag']}: {t.get('description', '')}") if self.debug: json_str = self._extract_json(response) if not json_str: print(f"[AI筛选][DEBUG] 无法从响应中提取 JSON") else: raw_data = json.loads(json_str) raw_tags = raw_data.get("tags", []) skipped = len(raw_tags) - len(tags) if skipped > 0: print(f"[AI筛选][DEBUG] 原始标签 {len(raw_tags)} 个，有效 {len(tags)} 个，跳过 {skipped} 个（缺少 tag 字段或格式无效）") return tags except json.JSONDecodeError as e: print(f"[AI筛选] 标签提取失败: JSON 解析错误: {e}") if self.debug: print(f"[AI筛选][DEBUG] 尝试解析的 JSON 内容: {self._extract_json(response) if response else '(空响应)'}") return [] except Exception as e: print(f"[AI筛选] 标签提取失败: {type(e).__name__}: {e}") return [] def update_tags(self, old_tags: List[Dict], interests_content: str) -> Optional[Dict]: """ 阶段 A'：AI 对比旧标签和新兴趣描述，给出更新方案 Args: old_tags: [{"tag": str, "description": str, "id": int}, ...] interests_content: 新的兴趣描述文本 Returns: {"keep": [{"tag": str, "description": str}], "add": [{"tag": str, "description": str}], "remove": [str], "change_ratio": float} 失败返回 None """ if not self.update_tags_user: print("[AI筛选] 标签更新提示词模板为空，回退到重新提取") return None # 构造旧标签 JSON old_tags_json = json.dumps( [{"tag": t["tag"], "description": t.get("description", "")} for t in old_tags], ensure_ascii=False, indent=2 ) user_prompt = self.update_tags_user.replace( "{old_tags_json}", old_tags_json ).replace( "{interests_content}", interests_content ) messages = [] if self.update_tags_system: messages.append({"role": "system", "content": self.update_tags_system}) messages.append({"role": "user", "content": user_prompt}) if self.debug: print(f"\n[AI筛选][DEBUG] === 标签更新 Prompt ===") for m in messages: print(f"[{m['role']}]\n{m['content']}") print(f"[AI筛选][DEBUG] === Prompt 结束 ===") try: response = self.client.chat(messages) if self.debug: print(f"\n[AI筛选][DEBUG] === 标签更新 AI 原始响应 ===") self._print_formatted_json(response) print(f"[AI筛选][DEBUG] === 响应结束 ===") result = self._parse_update_tags_response(response) if result is None: return None keep_count = len(result.get("keep", [])) add_count = len(result.get("add", [])) remove_count = len(result.get("remove", [])) ratio = result.get("change_ratio", 0) print(f"[AI筛选] AI 标签更新方案: 保留 {keep_count}, 新增 {add_count}, 移除 {remove_count}, change_ratio={ratio:.2f}") return result except Exception as e: print(f"[AI筛选] 标签更新失败: {type(e).__name__}: {e}") return None def _parse_update_tags_response(self, response: str) -> Optional[Dict]: """解析标签更新的 AI 响应""" json_str = self._extract_json(response) if not json_str: print("[AI筛选] 无法从标签更新响应中提取 JSON") return None data = json.loads(json_str) # 校验必需字段 keep = data.get("keep", []) add = data.get("add", []) remove = data.get("remove", []) change_ratio = float(data.get("change_ratio", 0)) # 校验 keep/add 格式 validated_keep = [] for t in keep: if isinstance(t, dict) and "tag" in t: validated_keep.append({ "tag": str(t["tag"]).strip(), "description": str(t.get("description", "")).strip(), }) validated_add = [] for t in add: if isinstance(t, dict) and "tag" in t: validated_add.append({ "tag": str(t["tag"]).strip(), "description": str(t.get("description", "")).strip(), }) validated_remove = [str(r).strip() for r in remove if r] # change_ratio 限制在 0~1 change_ratio = max(0.0, min(1.0, change_ratio)) return { "keep": validated_keep, "add": validated_add, "remove": validated_remove, "change_ratio": change_ratio, } def _parse_tags_response(self, response: str) -> List[Dict]: """解析标签提取的 AI 响应""" json_str = self._extract_json(response) if not json_str: return [] data = json.loads(json_str) tags_raw = data.get("tags", []) tags = [] for t in tags_raw: if not isinstance(t, dict) or "tag" not in t: continue tags.append({ "tag": str(t["tag"]).strip(), "description": str(t.get("description", "")).strip(), }) return tags def classify_batch( self, titles: List[Dict], tags: List[Dict], interests_content: str = "", ) -> List[Dict]: """ 阶段 B：对一批新闻标题做分类 Args: titles: [{"id": news_item_id, "title": str, "source": str}] tags: [{"id": tag_id, "tag": str, "description": str}] interests_content: 用户的兴趣描述（含质量过滤要求） Returns: [{"news_item_id": int, "tag_id": int, "relevance_score": float}, ...] """ if not titles or not tags: return [] if not self.classify_user: print("[AI筛选] 分类提示词模板为空") return [] # 构建标签列表文本 tags_list = "\n".join( f"{t['id']}. {t['tag']}: {t.get('description', '')}" for t in tags ) # 构建新闻列表文本 news_list = "\n".join( f"{t['id']}. [{t.get('source', '')}] {t['title']}" for t in titles ) # 填充模板 user_prompt = self.classify_user user_prompt = user_prompt.replace("{interests_content}", interests_content) user_prompt = user_prompt.replace("{tags_list}", tags_list) user_prompt = user_prompt.replace("{news_count}", str(len(titles))) user_prompt = user_prompt.replace("{news_list}", news_list) messages = [] if self.classify_system: messages.append({"role": "system", "content": self.classify_system}) messages.append({"role": "user", "content": user_prompt}) if self.debug: print(f"\n[AI筛选][DEBUG] === 分类 Prompt (标题数={len(titles)}, 标签={len(tags)}) ===") for m in messages: role = m['role'] content = m['content'] # 截断过长的新闻列表：只显示前5条和后5条 lines = content.split('\n') # 找到新闻列表区域并截断 if len(lines) > 30: # 显示前15行 + 省略提示 + 后10行 head = lines[:15] tail = lines[-10:] omitted = len(lines) - 25 truncated = '\n'.join(head) + f'\n... (省略 {omitted} 行) ...\n' + '\n'.join(tail) print(f"[{role}]\n{truncated}") else: print(f"[{role}]\n{content}") print(f"[AI筛选][DEBUG] === Prompt 结束 (长度: {sum(len(m['content']) for m in messages)} 字符) ===") try: response = self.client.chat(messages) return self._parse_classify_response(response, titles, tags) except Exception as e: print(f"[AI筛选] 分类请求失败: {type(e).__name__}: {e}") return [] def _parse_classify_response( self, response: str, titles: List[Dict], tags: List[Dict], ) -> List[Dict]: """解析分类的 AI 响应 支持两种 JSON 格式： - 新格式（扁平）: [{"id": 1, "tag_id": 1, "score": 0.9}, ...] - 旧格式（嵌套）: [{"id": 1, "tags": [{"tag_id": 1, "score": 0.9}]}, ...] 每条新闻只保留一个最高分的 tag，杜绝同一条出现在多个标签下。 """ json_str = self._extract_json(response) if not json_str: if self.debug: print(f"[AI筛选][DEBUG] 无法从分类响应中提取 JSON，原始响应前 500 字符: {(response or '')[:500]}") return [] try: data = json.loads(json_str) except json.JSONDecodeError as e: if self.debug: print(f"[AI筛选][DEBUG] 分类响应 JSON 解析失败: {e}") print(f"[AI筛选][DEBUG] 提取的 JSON 文本前 500 字符: {json_str[:500]}") return [] if not isinstance(data, list): if self.debug: print(f"[AI筛选][DEBUG] 分类响应顶层不是数组，实际类型: {type(data).__name__}") return [] # 构建 id 映射 title_ids = {t["id"] for t in titles} title_map = {t["id"]: t["title"] for t in titles} tag_id_set = {t["id"] for t in tags} tag_name_map = {t["id"]: t["tag"] for t in tags} # 每条新闻只保留一个最高分的 tag best_per_news: Dict[int, Dict] = {} # news_id -> {"tag_id": ..., "score": ...} skipped_news_ids = 0 skipped_tag_ids = 0 skipped_empty = 0 for item in data: if not isinstance(item, dict): continue news_id = item.get("id") if news_id not in title_ids: skipped_news_ids += 1 continue # 收集此条新闻的所有候选 tag candidates = [] if "tag_id" in item: # 新格式（扁平）: {"id": 1, "tag_id": 1, "score": 0.9} candidates.append({"tag_id": item["tag_id"], "score": item.get("score", 0.5)}) elif "tags" in item: # 旧格式（嵌套）: {"id": 1, "tags": [{"tag_id": 1, "score": 0.9}]} matched_tags = item.get("tags", []) if isinstance(matched_tags, list): if not matched_tags: skipped_empty += 1 continue candidates.extend(matched_tags) if not candidates: skipped_empty += 1 continue # 取最高分的有效 tag best_tag_id = None best_score = -1.0 for tag_match in candidates: if not isinstance(tag_match, dict): continue tag_id = tag_match.get("tag_id") if tag_id not in tag_id_set: skipped_tag_ids += 1 continue score = tag_match.get("score", 0.5) try: score = float(score) score = max(0.0, min(1.0, score)) except (ValueError, TypeError): score = 0.5 if score > best_score: best_score = score best_tag_id = tag_id if best_tag_id is not None: # 如果同一条新闻被多次返回，只保留分数更高的 existing = best_per_news.get(news_id) if existing is None or best_score > existing["relevance_score"]: best_per_news[news_id] = { "news_item_id": news_id, "tag_id": best_tag_id, "relevance_score": best_score, } results = list(best_per_news.values()) if self.debug: ai_returned = len(data) print(f"[AI筛选][DEBUG] --- 分类解析结果 ---") print(f"[AI筛选][DEBUG] AI 返回 {ai_returned} 条, 有效 {len(results)} 条 (每条新闻仅保留最高分 tag)") if skipped_empty > 0: print(f"[AI筛选][DEBUG] 跳过空 tags: {skipped_empty} 条") if skipped_news_ids > 0: print(f"[AI筛选][DEBUG] !! 跳过无效 news_id: {skipped_news_ids} 条") if skipped_tag_ids > 0: print(f"[AI筛选][DEBUG] !! 跳过无效 tag_id: {skipped_tag_ids} 条") # 按标签汇总 tag_summary: Dict[int, List[str]] = {} for r in results: tid = r["tag_id"] if tid not in tag_summary: tag_summary[tid] = [] tag_summary[tid].append( f" [{r['news_item_id']}] {title_map.get(r['news_item_id'], '?')[:40]} (score={r['relevance_score']:.2f})" ) for tid, items in tag_summary.items(): tname = tag_name_map.get(tid, f"tag_{tid}") print(f"[AI筛选][DEBUG] 标签「{tname}」匹配 {len(items)} 条:") for line in items: print(line) return results def _extract_json(self, response: str) -> Optional[str]: """从 AI 响应中提取 JSON 字符串""" if not response or not response.strip(): return None json_str = response.strip() if "```json" in json_str: parts = json_str.split("```json", 1) if len(parts) > 1: code_block = parts[1] end_idx = code_block.find("```") json_str = code_block[:end_idx] if end_idx != -1 else code_block elif "```" in json_str: parts = json_str.split("```", 2) if len(parts) >= 2: json_str = parts[1] json_str = json_str.strip() return json_str if json_str else None def _print_formatted_json(self, response: str) -> None: """格式化打印 AI 响应中的 JSON，便于 debug 阅读""" if not response: print("(空响应)") return json_str = self._extract_json(response) if json_str: try: data = json.loads(json_str) if isinstance(data, list): # 数组：每个元素压成一行 lines = [json.dumps(item, ensure_ascii=False) for item in data] print("[\n " + ",\n ".join(lines) + "\n]") else: print(json.dumps(data, ensure_ascii=False, indent=2)) return except json.JSONDecodeError: pass # JSON 解析失败，直接打印原始响应 print(response) ================================================ FILE: trendradar/ai/formatter.py ================================================ # coding=utf-8 """ AI 分析结果格式化模块 将 AI 分析结果格式化为各推送渠道的样式 """ import html as html_lib import re from .analyzer import AIAnalysisResult def _escape_html(text: str) -> str: """转义 HTML 特殊字符，防止 XSS 攻击""" return html_lib.escape(text) if text else "" def _format_list_content(text: str) -> str: """ 格式化列表内容，确保序号前有换行 例如将 "1. xxx 2. yyy" 转换为: 1. xxx 2. yyy """ if not text: return "" # 去除首尾空白，防止 AI 返回的内容开头就有换行导致显示空行 text = text.strip() # 0. 合并序号与紧随的【标签】（防御性处理） # 将 "1.\n【投资者】：" 或 "1. 【投资者】：" 合并为 "1. 投资者：" text = re.sub(r'(\d+\.)\s*【([^】]+)】([:：]?)', r'\1 \2：', text) # 1. 规范化：确保 "1." 后面有空格 result = re.sub(r'(\d+)\.([^ \d])', r'\1. \2', text) # 2. 强制换行：匹配 "数字."，且前面不是换行符 # (?!\d) 排除版本号/小数（如 2.0、3.5），避免将其误判为列表序号 result = re.sub(r'(?<=[^\n])\s+(\d+\.)(?!\d)', r'\n\1', result) # 3. 处理 "1.**粗体**" 这种情况（虽然 Prompt 要求不输出 Markdown，但防御性处理） result = re.sub(r'(?<=[^\n])(\d+\.\*\*)', r'\n\1', result) # 4. 处理中文标点后的换行（排除版本号/小数） result = re.sub(r'([：:;,。；，])\s*(\d+\.)(?!\d)', r'\1\n\2', result) # 5. 处理 "XX方面："、"XX领域：" 等子标题换行 # 只有在中文标点（句号、逗号、分号等）后才触发换行，避免破坏 "1. XX领域：" 格式 result = re.sub(r'([。！？；，、])\s*([a-zA-Z0-9\u4e00-\u9fa5]+(方面|领域)[:：])', r'\1\n\2', result) # 6. 处理 【标签】 格式 # 6a. 标签前确保空行分隔（文本开头除外） result = re.sub(r'(?<=\S)\n*(【[^】]+】)', r'\n\n\1', result) # 6b. 合并标签与被换行拆开的冒号：【tag】\n： → 【tag】： result = re.sub(r'(【[^】]+】)\n+([:：])', r'\1\2', result) # 6c. 标签后（含可选冒号），如果紧跟非空白非冒号内容则另起一行 # 用 (?=[^\s:：]) 避免正则回溯将冒号误判为"内容"而拆开 【tag】： result = re.sub(r'(【[^】]+】[:：]?)[ \t]*(?=[^\s:：])', r'\1\n', result) # 7. 在列表项之间增加视觉空行（排除版本号/小数） # 排除 【标签】 行（以】结尾）和子标题行（以冒号结尾）之后的情况，避免标题与首项之间出现空行 result = re.sub(r'(? str: """格式化独立展示区概括为纯文本行，每个源名称单独一行""" if not summaries: return "" lines = [] for source_name, summary in summaries.items(): if summary: lines.append(f"[{source_name}]:\n{summary}") return "\n\n".join(lines) def render_ai_analysis_markdown(result: AIAnalysisResult) -> str: """渲染为通用 Markdown 格式（Telegram、企业微信、ntfy、Bark、Slack）""" if not result.success: return f"⚠️ AI 分析失败: {result.error}" lines = ["**✨ AI 热点分析**", ""] if result.core_trends: lines.extend(["**核心热点态势**", _format_list_content(result.core_trends), ""]) if result.sentiment_controversy: lines.extend( ["**舆论风向争议**", _format_list_content(result.sentiment_controversy), ""] ) if result.signals: lines.extend(["**异动与弱信号**", _format_list_content(result.signals), ""]) if result.rss_insights: lines.extend( ["**RSS 深度洞察**", _format_list_content(result.rss_insights), ""] ) if result.outlook_strategy: lines.extend( ["**研判策略建议**", _format_list_content(result.outlook_strategy), ""] ) if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: lines.extend(["**独立源点速览**", summaries_text]) return "\n".join(lines) def render_ai_analysis_feishu(result: AIAnalysisResult) -> str: """渲染为飞书卡片 Markdown 格式""" if not result.success: return f"⚠️ AI 分析失败: {result.error}" lines = ["**✨ AI 热点分析**", ""] if result.core_trends: lines.extend(["**核心热点态势**", _format_list_content(result.core_trends), ""]) if result.sentiment_controversy: lines.extend( ["**舆论风向争议**", _format_list_content(result.sentiment_controversy), ""] ) if result.signals: lines.extend(["**异动与弱信号**", _format_list_content(result.signals), ""]) if result.rss_insights: lines.extend( ["**RSS 深度洞察**", _format_list_content(result.rss_insights), ""] ) if result.outlook_strategy: lines.extend( ["**研判策略建议**", _format_list_content(result.outlook_strategy), ""] ) if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: lines.extend(["**独立源点速览**", summaries_text]) return "\n".join(lines) def render_ai_analysis_dingtalk(result: AIAnalysisResult) -> str: """渲染为钉钉 Markdown 格式""" if not result.success: return f"⚠️ AI 分析失败: {result.error}" lines = ["### ✨ AI 热点分析", ""] if result.core_trends: lines.extend( ["#### 核心热点态势", _format_list_content(result.core_trends), ""] ) if result.sentiment_controversy: lines.extend( [ "#### 舆论风向争议", _format_list_content(result.sentiment_controversy), "", ] ) if result.signals: lines.extend(["#### 异动与弱信号", _format_list_content(result.signals), ""]) if result.rss_insights: lines.extend( ["#### RSS 深度洞察", _format_list_content(result.rss_insights), ""] ) if result.outlook_strategy: lines.extend( ["#### 研判策略建议", _format_list_content(result.outlook_strategy), ""] ) if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: lines.extend(["#### 独立源点速览", summaries_text]) return "\n".join(lines) def render_ai_analysis_html(result: AIAnalysisResult) -> str: """渲染为 HTML 格式（邮件）""" if not result.success: return ( f'
⚠️ AI 分析失败: {_escape_html(result.error)}
' ) html_parts = ['
', "

✨ AI 热点分析

"] if result.core_trends: content = _format_list_content(result.core_trends) content_html = _escape_html(content).replace("\n", "
") html_parts.extend( [ '
', "

核心热点态势

", f'
{content_html}
', "
", ] ) if result.sentiment_controversy: content = _format_list_content(result.sentiment_controversy) content_html = _escape_html(content).replace("\n", "
") html_parts.extend( [ '
', "

舆论风向争议

", f'
{content_html}
', "
", ] ) if result.signals: content = _format_list_content(result.signals) content_html = _escape_html(content).replace("\n", "
") html_parts.extend( [ '
', "

异动与弱信号

", f'
{content_html}
', "
", ] ) if result.rss_insights: content = _format_list_content(result.rss_insights) content_html = _escape_html(content).replace("\n", "
") html_parts.extend( [ '
', "

RSS 深度洞察

", f'
{content_html}
', "
", ] ) if result.outlook_strategy: content = _format_list_content(result.outlook_strategy) content_html = _escape_html(content).replace("\n", "
") html_parts.extend( [ '
', "

研判策略建议

", f'
{content_html}
', "
", ] ) if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: summaries_html = _escape_html(summaries_text).replace("\n", "
") html_parts.extend( [ '
', "

独立源点速览

", f'
{summaries_html}
', "
", ] ) html_parts.append("
") return "\n".join(html_parts) def render_ai_analysis_plain(result: AIAnalysisResult) -> str: """渲染为纯文本格式""" if not result.success: return f"AI 分析失败: {result.error}" lines = ["【✨ AI 热点分析】", ""] if result.core_trends: lines.extend(["[核心热点态势]", _format_list_content(result.core_trends), ""]) if result.sentiment_controversy: lines.extend( ["[舆论风向争议]", _format_list_content(result.sentiment_controversy), ""] ) if result.signals: lines.extend(["[异动与弱信号]", _format_list_content(result.signals), ""]) if result.rss_insights: lines.extend(["[RSS 深度洞察]", _format_list_content(result.rss_insights), ""]) if result.outlook_strategy: lines.extend(["[研判策略建议]", _format_list_content(result.outlook_strategy), ""]) if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: lines.extend(["[独立源点速览]", summaries_text]) return "\n".join(lines) def get_ai_analysis_renderer(channel: str): """根据渠道获取对应的渲染函数""" renderers = { "feishu": render_ai_analysis_feishu, "dingtalk": render_ai_analysis_dingtalk, "wework": render_ai_analysis_markdown, "telegram": render_ai_analysis_markdown, "email": render_ai_analysis_html_rich, # 邮件使用丰富样式，配合 HTML 报告的 CSS "ntfy": render_ai_analysis_markdown, "bark": render_ai_analysis_plain, "slack": render_ai_analysis_markdown, } return renderers.get(channel, render_ai_analysis_markdown) def render_ai_analysis_html_rich(result: AIAnalysisResult) -> str: """渲染为丰富样式的 HTML 格式（HTML 报告用）""" if not result: return "" # 检查是否成功 if not result.success: error_msg = result.error or "未知错误" return f"""
⚠️ AI 分析失败: {_escape_html(str(error_msg))}
""" ai_html = """
✨ AI 热点分析
AI
""" if result.core_trends: content = _format_list_content(result.core_trends) content_html = _escape_html(content).replace("\n", "
") ai_html += f"""
核心热点态势
{content_html}
""" if result.sentiment_controversy: content = _format_list_content(result.sentiment_controversy) content_html = _escape_html(content).replace("\n", "
") ai_html += f"""
舆论风向争议
{content_html}
""" if result.signals: content = _format_list_content(result.signals) content_html = _escape_html(content).replace("\n", "
") ai_html += f"""
异动与弱信号
{content_html}
""" if result.rss_insights: content = _format_list_content(result.rss_insights) content_html = _escape_html(content).replace("\n", "
") ai_html += f"""
RSS 深度洞察
{content_html}
""" if result.outlook_strategy: content = _format_list_content(result.outlook_strategy) content_html = _escape_html(content).replace("\n", "
") ai_html += f"""
研判策略建议
{content_html}
""" if result.standalone_summaries: summaries_text = _format_standalone_summaries(result.standalone_summaries) if summaries_text: summaries_html = _escape_html(summaries_text).replace("\n", "
") ai_html += f"""
独立源点速览
{summaries_html}
""" ai_html += """
""" return ai_html ================================================ FILE: trendradar/ai/translator.py ================================================ # coding=utf-8 """ AI 翻译器模块 对推送内容进行多语言翻译 基于 LiteLLM 统一接口，支持 100+ AI 提供商 """ from dataclasses import dataclass, field from pathlib import Path from typing import Any, Dict, List from trendradar.ai.client import AIClient @dataclass class TranslationResult: """翻译结果""" translated_text: str = "" # 翻译后的文本 original_text: str = "" # 原始文本 success: bool = False # 是否成功 error: str = "" # 错误信息 @dataclass class BatchTranslationResult: """批量翻译结果""" results: List[TranslationResult] = field(default_factory=list) success_count: int = 0 fail_count: int = 0 total_count: int = 0 prompt: str = "" # debug: 发送给 AI 的完整 prompt raw_response: str = "" # debug: AI 原始响应 parsed_count: int = 0 # debug: AI 响应解析出的条目数 class AITranslator: """AI 翻译器""" def __init__(self, translation_config: Dict[str, Any], ai_config: Dict[str, Any]): """ 初始化 AI 翻译器 Args: translation_config: AI 翻译配置 (AI_TRANSLATION) ai_config: AI 模型配置（LiteLLM 格式） """ self.translation_config = translation_config self.ai_config = ai_config # 翻译配置 self.enabled = translation_config.get("ENABLED", False) self.target_language = translation_config.get("LANGUAGE", "English") self.scope = translation_config.get("SCOPE", {"HOTLIST": True, "RSS": True, "STANDALONE": True}) # 创建 AI 客户端（基于 LiteLLM） self.client = AIClient(ai_config) # 加载提示词模板 self.system_prompt, self.user_prompt_template = self._load_prompt_template( translation_config.get("PROMPT_FILE", "ai_translation_prompt.txt") ) def _load_prompt_template(self, prompt_file: str) -> tuple: """加载提示词模板""" config_dir = Path(__file__).parent.parent.parent / "config" prompt_path = config_dir / prompt_file if not prompt_path.exists(): print(f"[翻译] 提示词文件不存在: {prompt_path}") return "", "" content = prompt_path.read_text(encoding="utf-8") # 解析 [system] 和 [user] 部分 system_prompt = "" user_prompt = "" if "[system]" in content and "[user]" in content: parts = content.split("[user]") system_part = parts[0] user_part = parts[1] if len(parts) > 1 else "" if "[system]" in system_part: system_prompt = system_part.split("[system]")[1].strip() user_prompt = user_part.strip() else: user_prompt = content return system_prompt, user_prompt def translate(self, text: str) -> TranslationResult: """ 翻译单条文本 Args: text: 要翻译的文本 Returns: TranslationResult: 翻译结果 """ result = TranslationResult(original_text=text) if not self.enabled: result.error = "翻译功能未启用" return result if not self.client.api_key: result.error = "未配置 AI API Key" return result if not text or not text.strip(): result.translated_text = text result.success = True return result try: # 构建提示词 user_prompt = self.user_prompt_template user_prompt = user_prompt.replace("{target_language}", self.target_language) user_prompt = user_prompt.replace("{content}", text) # 调用 AI API response = self._call_ai(user_prompt) result.translated_text = response.strip() result.success = True except Exception as e: error_type = type(e).__name__ error_msg = str(e) if len(error_msg) > 100: error_msg = error_msg[:100] + "..." result.error = f"翻译失败 ({error_type}): {error_msg}" return result def translate_batch(self, texts: List[str]) -> BatchTranslationResult: """ 批量翻译文本（单次 API 调用） Args: texts: 要翻译的文本列表 Returns: BatchTranslationResult: 批量翻译结果 """ batch_result = BatchTranslationResult(total_count=len(texts)) if not self.enabled: for text in texts: batch_result.results.append(TranslationResult( original_text=text, error="翻译功能未启用" )) batch_result.fail_count = len(texts) return batch_result if not self.client.api_key: for text in texts: batch_result.results.append(TranslationResult( original_text=text, error="未配置 AI API Key" )) batch_result.fail_count = len(texts) return batch_result if not texts: return batch_result # 过滤空文本 non_empty_indices = [] non_empty_texts = [] for i, text in enumerate(texts): if text and text.strip(): non_empty_indices.append(i) non_empty_texts.append(text) # 初始化结果列表 for text in texts: batch_result.results.append(TranslationResult(original_text=text)) # 空文本直接标记成功 for i, text in enumerate(texts): if not text or not text.strip(): batch_result.results[i].translated_text = text batch_result.results[i].success = True batch_result.success_count += 1 if not non_empty_texts: return batch_result try: # 构建批量翻译内容（使用编号格式） batch_content = self._format_batch_content(non_empty_texts) # 构建提示词 user_prompt = self.user_prompt_template user_prompt = user_prompt.replace("{target_language}", self.target_language) user_prompt = user_prompt.replace("{content}", batch_content) # 记录 debug 信息（包含完整的 system + user prompt） if self.system_prompt: batch_result.prompt = f"[system]\n{self.system_prompt}\n\n[user]\n{user_prompt}" else: batch_result.prompt = user_prompt # 调用 AI API response = self._call_ai(user_prompt) # 记录 AI 原始响应 batch_result.raw_response = response # 解析批量翻译结果 translated_texts, raw_parsed_count = self._parse_batch_response(response, len(non_empty_texts)) batch_result.parsed_count = raw_parsed_count # 填充结果 for idx, translated in zip(non_empty_indices, translated_texts): batch_result.results[idx].translated_text = translated batch_result.results[idx].success = True batch_result.success_count += 1 except Exception as e: error_msg = f"批量翻译失败: {type(e).__name__}: {str(e)[:100]}" for idx in non_empty_indices: batch_result.results[idx].error = error_msg batch_result.fail_count = len(non_empty_indices) return batch_result def _format_batch_content(self, texts: List[str]) -> str: """格式化批量翻译内容""" lines = [] for i, text in enumerate(texts, 1): lines.append(f"[{i}] {text}") return "\n".join(lines) def _parse_batch_response(self, response: str, expected_count: int) -> tuple: """ 解析批量翻译响应 Args: response: AI 响应文本 expected_count: 期望的翻译数量 Returns: tuple: (翻译结果列表, AI 原始解析出的条目数) """ results = [] lines = response.strip().split("\n") current_idx = None current_text = [] for line in lines: # 尝试匹配 [数字] 格式 stripped = line.strip() if stripped.startswith("[") and "]" in stripped: bracket_end = stripped.index("]") try: idx = int(stripped[1:bracket_end]) # 保存之前的内容 if current_idx is not None: results.append((current_idx, "\n".join(current_text).strip())) current_idx = idx current_text = [stripped[bracket_end + 1:].strip()] except ValueError: if current_idx is not None: current_text.append(line) else: if current_idx is not None: current_text.append(line) # 保存最后一条 if current_idx is not None: results.append((current_idx, "\n".join(current_text).strip())) # 按索引排序并提取文本 results.sort(key=lambda x: x[0]) translated = [text for _, text in results] raw_parsed_count = len(translated) # 如果解析结果数量不匹配，尝试简单按行分割 if len(translated) != expected_count: # 回退：按行分割（去除编号） translated = [] for line in lines: stripped = line.strip() if stripped.startswith("[") and "]" in stripped: bracket_end = stripped.index("]") translated.append(stripped[bracket_end + 1:].strip()) elif stripped: translated.append(stripped) raw_parsed_count = len(translated) # 确保返回正确数量 while len(translated) < expected_count: translated.append("") return translated[:expected_count], raw_parsed_count def _call_ai(self, user_prompt: str) -> str: """调用 AI API（使用 LiteLLM）""" messages = [] if self.system_prompt: messages.append({"role": "system", "content": self.system_prompt}) messages.append({"role": "user", "content": user_prompt}) return self.client.chat(messages) ================================================ FILE: trendradar/context.py ================================================ # coding=utf-8 """ 应用上下文模块 提供配置上下文类，封装所有依赖配置的操作，消除全局状态和包装函数。 """ from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional, Tuple from trendradar.utils.time import ( DEFAULT_TIMEZONE, get_configured_time, format_date_folder, format_time_filename, get_current_time_display, convert_time_for_display, format_iso_time_friendly, is_within_days, ) from trendradar.core import ( load_frequency_words, matches_word_groups, read_all_today_titles, detect_latest_new_titles, count_word_frequency, Scheduler, ) from trendradar.report import ( prepare_report_data, generate_html_report, render_html_content, ) from trendradar.notification import ( render_feishu_content, render_dingtalk_content, split_content_into_batches, NotificationDispatcher, ) from trendradar.ai import AITranslator from trendradar.ai.filter import AIFilter, AIFilterResult from trendradar.storage import get_storage_manager class AppContext: """ 应用上下文类 封装所有依赖配置的操作，提供统一的接口。 消除对全局 CONFIG 的依赖，提高可测试性。 使用示例: config = load_config() ctx = AppContext(config) # 时间操作 now = ctx.get_time() date_folder = ctx.format_date() # 存储操作 storage = ctx.get_storage_manager() # 报告生成 html = ctx.generate_html_report(stats, total_titles, ...) """ def __init__(self, config: Dict[str, Any]): """ 初始化应用上下文 Args: config: 完整的配置字典 """ self.config = config self._storage_manager = None self._scheduler = None # === 配置访问 === @property def timezone(self) -> str: """获取配置的时区""" return self.config.get("TIMEZONE", DEFAULT_TIMEZONE) @property def rank_threshold(self) -> int: """获取排名阈值""" return self.config.get("RANK_THRESHOLD", 50) @property def weight_config(self) -> Dict: """获取权重配置""" return self.config.get("WEIGHT_CONFIG", {}) @property def platforms(self) -> List[Dict]: """获取平台配置列表""" return self.config.get("PLATFORMS", []) @property def platform_ids(self) -> List[str]: """获取平台ID列表""" return [p["id"] for p in self.platforms] @property def rss_config(self) -> Dict: """获取 RSS 配置""" return self.config.get("RSS", {}) @property def rss_enabled(self) -> bool: """RSS 是否启用""" return self.rss_config.get("ENABLED", False) @property def rss_feeds(self) -> List[Dict]: """获取 RSS 源列表""" return self.rss_config.get("FEEDS", []) @property def display_mode(self) -> str: """获取显示模式 (keyword | platform)""" return self.config.get("DISPLAY_MODE", "keyword") @property def show_new_section(self) -> bool: """是否显示新增热点区域""" return self.config.get("DISPLAY", {}).get("REGIONS", {}).get("NEW_ITEMS", True) @property def region_order(self) -> List[str]: """获取区域显示顺序""" default_order = ["hotlist", "rss", "new_items", "standalone", "ai_analysis"] return self.config.get("DISPLAY", {}).get("REGION_ORDER", default_order) @property def filter_method(self) -> str: """获取筛选策略: keyword | ai""" return self.config.get("FILTER", {}).get("METHOD", "keyword") @property def ai_priority_sort_enabled(self) -> bool: """AI 模式标签排序开关（与 keyword 的 sort_by_position_first 解耦）""" return self.config.get("FILTER", {}).get("PRIORITY_SORT_ENABLED", False) @property def ai_filter_config(self) -> Dict: """获取 AI 筛选配置""" return self.config.get("AI_FILTER", {}) @property def ai_filter_enabled(self) -> bool: """AI 筛选是否启用（基于 filter.method 判断）""" return self.filter_method == "ai" # === 时间操作 === def get_time(self) -> datetime: """获取当前配置时区的时间""" return get_configured_time(self.timezone) def format_date(self) -> str: """格式化日期文件夹 (YYYY-MM-DD)""" return format_date_folder(timezone=self.timezone) def format_time(self) -> str: """格式化时间文件名 (HH-MM)""" return format_time_filename(self.timezone) def get_time_display(self) -> str: """获取时间显示 (HH:MM)""" return get_current_time_display(self.timezone) @staticmethod def convert_time_display(time_str: str) -> str: """将 HH-MM 转换为 HH:MM""" return convert_time_for_display(time_str) # === 存储操作 === def get_storage_manager(self): """获取存储管理器（延迟初始化，单例）""" if self._storage_manager is None: storage_config = self.config.get("STORAGE", {}) remote_config = storage_config.get("REMOTE", {}) local_config = storage_config.get("LOCAL", {}) pull_config = storage_config.get("PULL", {}) self._storage_manager = get_storage_manager( backend_type=storage_config.get("BACKEND", "auto"), data_dir=local_config.get("DATA_DIR", "output"), enable_txt=storage_config.get("FORMATS", {}).get("TXT", True), enable_html=storage_config.get("FORMATS", {}).get("HTML", True), remote_config={ "bucket_name": remote_config.get("BUCKET_NAME", ""), "access_key_id": remote_config.get("ACCESS_KEY_ID", ""), "secret_access_key": remote_config.get("SECRET_ACCESS_KEY", ""), "endpoint_url": remote_config.get("ENDPOINT_URL", ""), "region": remote_config.get("REGION", ""), }, local_retention_days=local_config.get("RETENTION_DAYS", 0), remote_retention_days=remote_config.get("RETENTION_DAYS", 0), pull_enabled=pull_config.get("ENABLED", False), pull_days=pull_config.get("DAYS", 7), timezone=self.timezone, ) return self._storage_manager def get_output_path(self, subfolder: str, filename: str) -> str: """获取输出路径（扁平化结构：output/类型/日期/文件名）""" output_dir = Path("output") / subfolder / self.format_date() output_dir.mkdir(parents=True, exist_ok=True) return str(output_dir / filename) # === 数据处理 === def read_today_titles( self, platform_ids: Optional[List[str]] = None, quiet: bool = False ) -> Tuple[Dict, Dict, Dict]: """读取当天所有标题""" return read_all_today_titles(self.get_storage_manager(), platform_ids, quiet=quiet) def detect_new_titles( self, platform_ids: Optional[List[str]] = None, quiet: bool = False ) -> Dict: """检测最新批次的新增标题""" return detect_latest_new_titles(self.get_storage_manager(), platform_ids, quiet=quiet) def is_first_crawl(self) -> bool: """检测是否是当天第一次爬取""" return self.get_storage_manager().is_first_crawl_today() # === 频率词处理 === def load_frequency_words( self, frequency_file: Optional[str] = None ) -> Tuple[List[Dict], List[str], List[str]]: """加载频率词配置""" return load_frequency_words(frequency_file) def matches_word_groups( self, title: str, word_groups: List[Dict], filter_words: List[str], global_filters: Optional[List[str]] = None, ) -> bool: """检查标题是否匹配词组规则""" return matches_word_groups(title, word_groups, filter_words, global_filters) # === 统计分析 === def count_frequency( self, results: Dict, word_groups: List[Dict], filter_words: List[str], id_to_name: Dict, title_info: Optional[Dict] = None, new_titles: Optional[Dict] = None, mode: str = "daily", global_filters: Optional[List[str]] = None, quiet: bool = False, ) -> Tuple[List[Dict], int]: """统计词频""" return count_word_frequency( results=results, word_groups=word_groups, filter_words=filter_words, id_to_name=id_to_name, title_info=title_info, rank_threshold=self.rank_threshold, new_titles=new_titles, mode=mode, global_filters=global_filters, weight_config=self.weight_config, max_news_per_keyword=self.config.get("MAX_NEWS_PER_KEYWORD", 0), sort_by_position_first=self.config.get("SORT_BY_POSITION_FIRST", False), is_first_crawl_func=self.is_first_crawl, convert_time_func=self.convert_time_display, quiet=quiet, ) # === 报告生成 === def prepare_report( self, stats: List[Dict], failed_ids: Optional[List] = None, new_titles: Optional[Dict] = None, id_to_name: Optional[Dict] = None, mode: str = "daily", frequency_file: Optional[str] = None, ) -> Dict: """准备报告数据""" return prepare_report_data( stats=stats, failed_ids=failed_ids, new_titles=new_titles, id_to_name=id_to_name, mode=mode, rank_threshold=self.rank_threshold, matches_word_groups_func=self.matches_word_groups, load_frequency_words_func=lambda: self.load_frequency_words(frequency_file), show_new_section=self.show_new_section, ) def generate_html( self, stats: List[Dict], total_titles: int, failed_ids: Optional[List] = None, new_titles: Optional[Dict] = None, id_to_name: Optional[Dict] = None, mode: str = "daily", update_info: Optional[Dict] = None, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[Any] = None, standalone_data: Optional[Dict] = None, frequency_file: Optional[str] = None, ) -> str: """生成HTML报告""" return generate_html_report( stats=stats, total_titles=total_titles, failed_ids=failed_ids, new_titles=new_titles, id_to_name=id_to_name, mode=mode, update_info=update_info, rank_threshold=self.rank_threshold, output_dir="output", date_folder=self.format_date(), time_filename=self.format_time(), render_html_func=lambda *args, **kwargs: self.render_html(*args, rss_items=rss_items, rss_new_items=rss_new_items, ai_analysis=ai_analysis, standalone_data=standalone_data, **kwargs), matches_word_groups_func=self.matches_word_groups, load_frequency_words_func=lambda: self.load_frequency_words(frequency_file), ) def render_html( self, report_data: Dict, total_titles: int, mode: str = "daily", update_info: Optional[Dict] = None, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[Any] = None, standalone_data: Optional[Dict] = None, ) -> str: """渲染HTML内容""" return render_html_content( report_data=report_data, total_titles=total_titles, mode=mode, update_info=update_info, region_order=self.region_order, get_time_func=self.get_time, rss_items=rss_items, rss_new_items=rss_new_items, display_mode=self.display_mode, ai_analysis=ai_analysis, show_new_section=self.show_new_section, standalone_data=standalone_data, ) # === 通知内容渲染 === def render_feishu( self, report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily", ) -> str: """渲染飞书内容""" return render_feishu_content( report_data=report_data, update_info=update_info, mode=mode, separator=self.config.get("FEISHU_MESSAGE_SEPARATOR", "---"), region_order=self.region_order, get_time_func=self.get_time, show_new_section=self.show_new_section, ) def render_dingtalk( self, report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily", ) -> str: """渲染钉钉内容""" return render_dingtalk_content( report_data=report_data, update_info=update_info, mode=mode, region_order=self.region_order, get_time_func=self.get_time, show_new_section=self.show_new_section, ) def split_content( self, report_data: Dict, format_type: str, update_info: Optional[Dict] = None, max_bytes: Optional[int] = None, mode: str = "daily", rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_content: Optional[str] = None, standalone_data: Optional[Dict] = None, ai_stats: Optional[Dict] = None, report_type: str = "热点分析报告", ) -> List[str]: """分批处理消息内容（支持热榜+RSS合并+AI分析+独立展示区） Args: report_data: 报告数据 format_type: 格式类型 update_info: 更新信息 max_bytes: 最大字节数 mode: 报告模式 rss_items: RSS 统计条目列表 rss_new_items: RSS 新增条目列表 ai_content: AI 分析内容（已渲染的字符串） standalone_data: 独立展示区数据 ai_stats: AI 分析统计数据 report_type: 报告类型 Returns: 分批后的消息内容列表 """ return split_content_into_batches( report_data=report_data, format_type=format_type, update_info=update_info, max_bytes=max_bytes, mode=mode, batch_sizes={ "dingtalk": self.config.get("DINGTALK_BATCH_SIZE", 20000), "feishu": self.config.get("FEISHU_BATCH_SIZE", 29000), "default": self.config.get("MESSAGE_BATCH_SIZE", 4000), }, feishu_separator=self.config.get("FEISHU_MESSAGE_SEPARATOR", "---"), region_order=self.region_order, get_time_func=self.get_time, rss_items=rss_items, rss_new_items=rss_new_items, timezone=self.config.get("TIMEZONE", DEFAULT_TIMEZONE), display_mode=self.display_mode, ai_content=ai_content, standalone_data=standalone_data, rank_threshold=self.rank_threshold, ai_stats=ai_stats, report_type=report_type, show_new_section=self.show_new_section, ) # === 通知发送 === def create_notification_dispatcher(self) -> NotificationDispatcher: """创建通知调度器""" # 创建翻译器（如果启用） translator = None trans_config = self.config.get("AI_TRANSLATION", {}) if trans_config.get("ENABLED", False): ai_config = self.config.get("AI", {}) translator = AITranslator(trans_config, ai_config) return NotificationDispatcher( config=self.config, get_time_func=self.get_time, split_content_func=self.split_content, translator=translator, ) def create_scheduler(self) -> Scheduler: """ 创建调度器（延迟初始化，单例） 基于 config.yaml 的 schedule 段 + timeline.yaml 构建。 """ if self._scheduler is None: schedule_config = self.config.get("SCHEDULE", {}) timeline_data = self.config.get("_TIMELINE_DATA", {}) self._scheduler = Scheduler( schedule_config=schedule_config, timeline_data=timeline_data, storage_backend=self.get_storage_manager(), get_time_func=self.get_time, fallback_report_mode=self.config.get("REPORT_MODE", "current"), ) return self._scheduler # === AI 智能筛选 === @staticmethod def _with_ordered_priorities(tags: List[Dict], start_priority: int = 1) -> List[Dict]: """按当前列表顺序补齐优先级（值越小优先级越高）""" normalized: List[Dict] = [] priority = start_priority for tag_data in tags: if not isinstance(tag_data, dict): continue tag_name = str(tag_data.get("tag", "")).strip() if not tag_name: continue item = dict(tag_data) item["tag"] = tag_name item["priority"] = priority normalized.append(item) priority += 1 return normalized def run_ai_filter(self, interests_file: Optional[str] = None) -> Optional[AIFilterResult]: """ 执行 AI 智能筛选完整流程 Args: interests_file: 兴趣描述文件名（位于 config/custom/ai/），None=使用默认 config/ai_interests.txt 1. 读取兴趣描述文件，计算 hash 2. 对比数据库 prompt_hash，决定是否重新提取标签 3. 收集待分类新闻（去重） 4. 按 batch_size 分组调用 AI 分类 5. 保存结果 6. 查询 active 结果，按标签分组返回 Returns: AIFilterResult 或 None（未启用或出错） """ if not self.ai_filter_enabled: return None filter_config = self.ai_filter_config ai_config = self.config.get("AI", {}) debug = self.config.get("DEBUG", False) # 创建 AIFilter 实例 ai_filter = AIFilter(ai_config, filter_config, self.get_time, debug) # 确定实际使用的兴趣文件名 # None = 使用默认 config/ai_interests.txt，指定文件名 = config/custom/ai/{name} configured_interests = interests_file or filter_config.get("INTERESTS_FILE") effective_interests_file = configured_interests or "ai_interests.txt" if debug: print(f"[AI筛选][DEBUG] === 配置信息 ===") print(f"[AI筛选][DEBUG] 存储后端: {self.get_storage_manager().backend_name}") print(f"[AI筛选][DEBUG] batch_size={filter_config.get('BATCH_SIZE', 200)}, " f"batch_interval={filter_config.get('BATCH_INTERVAL', 5)}") print(f"[AI筛选][DEBUG] interests_file={effective_interests_file}") print(f"[AI筛选][DEBUG] prompt_file={filter_config.get('PROMPT_FILE', 'prompt.txt')}") print(f"[AI筛选][DEBUG] extract_prompt_file={filter_config.get('EXTRACT_PROMPT_FILE', 'extract_prompt.txt')}") # 1. 读取兴趣描述 # 传 configured_interests（可能为 None）给 load_interests_content， # 让它区分"默认文件(config/ai_interests.txt)"和"自定义文件(config/custom/ai/)" interests_content = ai_filter.load_interests_content(configured_interests) if not interests_content: return AIFilterResult(success=False, error="兴趣描述文件为空或不存在") current_hash = ai_filter.compute_interests_hash(interests_content, effective_interests_file) storage = self.get_storage_manager() if debug: print(f"[AI筛选][DEBUG] 兴趣描述 hash: {current_hash}") print(f"[AI筛选][DEBUG] 兴趣描述内容 ({len(interests_content)} 字符):\n{interests_content}") # 2. 开启批量模式（远程后端延迟上传，所有写操作完成后统一上传） storage.begin_batch() # 3. 检查提示词是否变更 stored_hash = storage.get_latest_prompt_hash(interests_file=effective_interests_file) if debug: print(f"[AI筛选][DEBUG] 数据库存储 hash: {stored_hash}") print(f"[AI筛选][DEBUG] hash 对比: stored={stored_hash} vs current={current_hash} → {'匹配' if stored_hash == current_hash else '不匹配'}") if stored_hash != current_hash: new_version = storage.get_latest_ai_filter_tag_version() + 1 threshold = filter_config.get("RECLASSIFY_THRESHOLD", 0.6) if stored_hash is None: # 首次运行，直接提取并保存全部标签 print(f"[AI筛选] 首次运行 ({effective_interests_file})，提取标签...") tags_data = ai_filter.extract_tags(interests_content) if not tags_data: storage.end_batch() return AIFilterResult(success=False, error="标签提取失败") tags_data = self._with_ordered_priorities(tags_data, start_priority=1) saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file) print(f"[AI筛选] 已保存 {saved_count} 个标签 (版本 {new_version})") else: # 兴趣描述已变更，让 AI 对比旧标签和新兴趣，给出更新方案 old_tags = storage.get_active_ai_filter_tags(interests_file=effective_interests_file) update_result = ai_filter.update_tags(old_tags, interests_content) if update_result is None: # AI 标签更新失败，回退到重新提取全部标签 print(f"[AI筛选] AI 标签更新失败，回退到重新提取") tags_data = ai_filter.extract_tags(interests_content) if not tags_data: storage.end_batch() return AIFilterResult(success=False, error="标签提取失败") tags_data = self._with_ordered_priorities(tags_data, start_priority=1) deprecated_count = storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file) storage.clear_analyzed_news(interests_file=effective_interests_file) saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file) print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})") else: change_ratio = update_result["change_ratio"] keep_tags = update_result["keep"] add_tags = update_result["add"] remove_tags = update_result["remove"] if debug: print(f"[AI筛选][DEBUG] AI 标签更新: keep={len(keep_tags)}, add={len(add_tags)}, remove={len(remove_tags)}, change_ratio={change_ratio:.2f}, threshold={threshold:.2f}") if change_ratio >= threshold: # 全量重分类：废弃所有旧标签，用 extract_tags 重新提取 print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} >= threshold={threshold:.2f} → 全量重分类)") tags_data = ai_filter.extract_tags(interests_content) if not tags_data: storage.end_batch() return AIFilterResult(success=False, error="标签提取失败") tags_data = self._with_ordered_priorities(tags_data, start_priority=1) deprecated_count = storage.deprecate_all_ai_filter_tags(interests_file=effective_interests_file) storage.clear_analyzed_news(interests_file=effective_interests_file) saved_count = storage.save_ai_filter_tags(tags_data, new_version, current_hash, interests_file=effective_interests_file) print(f"[AI筛选] 废弃 {deprecated_count} 个旧标签, 保存 {saved_count} 个新标签 (版本 {new_version})") else: # 增量更新：按 AI 指示操作 print(f"[AI筛选] 兴趣文件变更: {effective_interests_file} (AI change_ratio={change_ratio:.2f} < threshold={threshold:.2f} → 增量更新)") print(f"[AI筛选] 保留 {len(keep_tags)} 个标签, 新增 {len(add_tags)} 个, 废弃 {len(remove_tags)} 个") # 废弃 AI 标记移除的标签 if remove_tags: remove_set = set(remove_tags) removed_ids = [t["id"] for t in old_tags if t["tag"] in remove_set] if removed_ids: storage.deprecate_specific_ai_filter_tags(removed_ids) if debug: print(f"[AI筛选][DEBUG] 废弃标签 IDs: {removed_ids}") # 更新保留标签的描述 keep_with_priority = [] if keep_tags: storage.update_ai_filter_tag_descriptions(keep_tags, interests_file=effective_interests_file) keep_with_priority = self._with_ordered_priorities(keep_tags, start_priority=1) storage.update_ai_filter_tag_priorities(keep_with_priority, interests_file=effective_interests_file) # 保存新增标签 if add_tags: add_start = keep_with_priority[-1]["priority"] + 1 if keep_with_priority else 1 add_with_priority = self._with_ordered_priorities(add_tags, start_priority=add_start) saved_count = storage.save_ai_filter_tags(add_with_priority, new_version, current_hash, interests_file=effective_interests_file) if debug: print(f"[AI筛选][DEBUG] 新增保存 {saved_count} 个标签") # 更新保留标签的 hash（标记为已处理） storage.update_ai_filter_tags_hash(effective_interests_file, current_hash) # 增量更新：清除不匹配新闻的分析记录，让它们有机会被新标签集重新分析 if add_tags: cleared = storage.clear_unmatched_analyzed_news(interests_file=effective_interests_file) if cleared > 0: print(f"[AI筛选] 清除 {cleared} 条不匹配记录，将在新标签下重新分析") # 3. 获取当前 active 标签 active_tags = storage.get_active_ai_filter_tags(interests_file=effective_interests_file) if debug: print(f"[AI筛选][DEBUG] 从数据库获取 active 标签: {len(active_tags)} 个") for t in active_tags: print(f"[AI筛选][DEBUG] id={t['id']} tag={t['tag']} priority={t.get('priority', 9999)} version={t.get('version')} hash={t.get('prompt_hash', '')[:8]}...") if not active_tags: storage.end_batch() return AIFilterResult(success=False, error="没有可用的标签") print(f"[AI筛选] 使用 {len(active_tags)} 个标签") # 4. 收集待分类新闻 # 热榜 all_news = storage.get_all_news_ids() analyzed_hotlist = storage.get_analyzed_news_ids("hotlist", interests_file=effective_interests_file) pending_news = [n for n in all_news if n["id"] not in analyzed_hotlist] # RSS（先做新鲜度过滤，再去除已分类的） pending_rss = [] freshness_filtered_rss = 0 if self.rss_enabled: all_rss = storage.get_all_rss_ids() # 应用新鲜度过滤（与推送阶段一致） rss_config = self.rss_config freshness_config = rss_config.get("FRESHNESS_FILTER", {}) freshness_enabled = freshness_config.get("ENABLED", True) default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3) timezone = self.config.get("TIMEZONE", DEFAULT_TIMEZONE) # 构建 feed_id -> max_age_days 的映射 feed_max_age_map = {} for feed_cfg in self.rss_feeds: feed_id = feed_cfg.get("id", "") max_age = feed_cfg.get("max_age_days") if max_age is not None: try: feed_max_age_map[feed_id] = int(max_age) except (ValueError, TypeError): pass fresh_rss = [] for n in all_rss: published_at = n.get("published_at", "") feed_id = n.get("source_id", "") max_days = feed_max_age_map.get(feed_id, default_max_age_days) if freshness_enabled and max_days > 0 and published_at: if not is_within_days(published_at, max_days, timezone): freshness_filtered_rss += 1 continue fresh_rss.append(n) analyzed_rss = storage.get_analyzed_news_ids("rss", interests_file=effective_interests_file) pending_rss = [n for n in fresh_rss if n["id"] not in analyzed_rss] # 始终打印总量/已分析/待分析 的详细数据 hotlist_total = len(all_news) hotlist_skipped = len(analyzed_hotlist) hotlist_pending = len(pending_news) print(f"[AI筛选] 热榜: 总计 {hotlist_total} 条, 已分析跳过 {hotlist_skipped} 条, 本次发送AI分析 {hotlist_pending} 条") if self.rss_enabled: rss_total = len(all_rss) rss_skipped = len(analyzed_rss) rss_pending = len(pending_rss) freshness_info = f", 新鲜度过滤 {freshness_filtered_rss} 条" if freshness_filtered_rss > 0 else "" print(f"[AI筛选] RSS: 总计 {rss_total} 条{freshness_info}, 已分析跳过 {rss_skipped} 条, 本次发送AI分析 {rss_pending} 条") total_pending = len(pending_news) + len(pending_rss) if total_pending == 0: print("[AI筛选] 没有新增新闻需要分类") # 5. 批量分类 batch_size = filter_config.get("BATCH_SIZE", 200) batch_interval = filter_config.get("BATCH_INTERVAL", 5) total_results = [] batch_count = 0 # 跨热榜和 RSS 的全局批次计数 # 处理热榜 for i in range(0, len(pending_news), batch_size): if batch_count > 0 and batch_interval > 0: import time print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...") time.sleep(batch_interval) batch = pending_news[i:i + batch_size] titles_for_ai = [ {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")} for n in batch ] batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content) for r in batch_results: r["source_type"] = "hotlist" total_results.extend(batch_results) batch_count += 1 print(f"[AI筛选] 热榜批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配") # 处理 RSS for i in range(0, len(pending_rss), batch_size): if batch_count > 0 and batch_interval > 0: import time print(f"[AI筛选] 批次间隔等待 {batch_interval} 秒...") time.sleep(batch_interval) batch = pending_rss[i:i + batch_size] titles_for_ai = [ {"id": n["id"], "title": n["title"], "source": n.get("source_name", "")} for n in batch ] batch_results = ai_filter.classify_batch(titles_for_ai, active_tags, interests_content) for r in batch_results: r["source_type"] = "rss" total_results.extend(batch_results) batch_count += 1 print(f"[AI筛选] RSS 批次 {i // batch_size + 1}: {len(batch)} 条 → {len(batch_results)} 条匹配") # 6. 保存结果 if total_results: saved = storage.save_ai_filter_results(total_results) print(f"[AI筛选] 保存 {saved} 条分类结果") if debug and saved != len(total_results): print(f"[AI筛选][DEBUG] !! 保存数量不一致: 期望 {len(total_results)}, 实际 {saved}（可能有重复记录被跳过）") # 6.5 记录所有已分析的新闻（匹配+不匹配，用于去重） matched_hotlist_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "hotlist"} matched_rss_ids = {r["news_item_id"] for r in total_results if r.get("source_type") == "rss"} if pending_news: hotlist_ids = [n["id"] for n in pending_news] storage.save_analyzed_news( hotlist_ids, "hotlist", effective_interests_file, current_hash, matched_hotlist_ids ) if pending_rss: rss_ids = [n["id"] for n in pending_rss] storage.save_analyzed_news( rss_ids, "rss", effective_interests_file, current_hash, matched_rss_ids ) if pending_news or pending_rss: total_analyzed = len(pending_news) + len(pending_rss) total_matched = len(matched_hotlist_ids) + len(matched_rss_ids) print(f"[AI筛选] 已记录 {total_analyzed} 条新闻分析状态 (匹配 {total_matched}, 不匹配 {total_analyzed - total_matched})") # 7. 结束批量模式（统一上传数据库到远程存储） storage.end_batch() # 8. 查询并组装返回结果 all_results = storage.get_active_ai_filter_results(interests_file=effective_interests_file) if debug: print(f"[AI筛选][DEBUG] === 最终汇总 ===") print(f"[AI筛选][DEBUG] 数据库 active 分类结果: {len(all_results)} 条") # 按标签统计 tag_counts: dict = {} for r in all_results: tag_name = r.get("tag", "?") src_type = r.get("source_type", "?") key = f"{tag_name}({src_type})" tag_counts[key] = tag_counts.get(key, 0) + 1 for key, count in sorted(tag_counts.items()): print(f"[AI筛选][DEBUG] {key}: {count} 条") return self._build_filter_result(all_results, active_tags, total_pending) def _build_filter_result( self, raw_results: List[Dict], tags: List[Dict], total_processed: int, ) -> AIFilterResult: """将数据库查询结果组装为 AIFilterResult""" priority_sort_enabled = self.ai_priority_sort_enabled tag_priority_map = {} for idx, t in enumerate(tags, start=1): tag_name = str(t.get("tag", "")).strip() if isinstance(t, dict) else "" if not tag_name: continue try: tag_priority_map[tag_name] = int(t.get("priority", idx)) except (TypeError, ValueError): tag_priority_map[tag_name] = idx # 按标签分组 tag_groups: Dict[str, Dict] = {} seen_titles: Dict[str, set] = {} # 每个标签下去重 for r in raw_results: tag_name = r["tag"] if tag_name not in tag_groups: raw_priority = r.get("tag_priority", tag_priority_map.get(tag_name, 9999)) try: tag_position = int(raw_priority) except (TypeError, ValueError): tag_position = 9999 tag_groups[tag_name] = { "tag": tag_name, "description": r.get("tag_description", ""), "position": tag_position, "count": 0, "items": [], } seen_titles[tag_name] = set() title = r["title"] if title in seen_titles[tag_name]: continue seen_titles[tag_name].add(title) tag_groups[tag_name]["items"].append({ "title": title, "source_id": r.get("source_id", ""), "source_name": r.get("source_name", ""), "url": r.get("url", ""), "mobile_url": r.get("mobile_url", ""), "rank": r.get("rank", 0), "ranks": r.get("ranks", []), "first_time": r.get("first_time", ""), "last_time": r.get("last_time", ""), "count": r.get("count", 1), "relevance_score": r.get("relevance_score", 0), "source_type": r.get("source_type", "hotlist"), }) tag_groups[tag_name]["count"] += 1 # 根据配置排序：位置优先 / 数量优先 if priority_sort_enabled: sorted_tags = sorted( tag_groups.values(), key=lambda x: (x.get("position", 9999), -x["count"], x["tag"]), ) else: sorted_tags = sorted( tag_groups.values(), key=lambda x: (-x["count"], x.get("position", 9999), x["tag"]), ) total_matched = sum(t["count"] for t in sorted_tags) return AIFilterResult( tags=sorted_tags, total_matched=total_matched, total_processed=total_processed, success=True, ) def convert_ai_filter_to_report_data( self, ai_filter_result: AIFilterResult, mode: str = "daily", new_titles: Optional[Dict] = None, rss_new_urls: Optional[set] = None, ) -> tuple: """ 将 AI 筛选结果转换为与关键词匹配相同的数据结构 AIFilterResult.tags 中每个 tag 对应一个 "word"（关键词组）。 tag.items 中 source_type="hotlist" 的条目进入热榜 stats， source_type="rss" 的条目进入 rss_items stats。 Args: ai_filter_result: AI 筛选结果 mode: 报告模式 ("daily" | "current" | "incremental") new_titles: 热榜新增标题 {source_id: {title: data}}，用于 is_new 检测 rss_new_urls: 新增 RSS 条目的 URL 集合，用于 is_new 检测 Returns: (hotlist_stats, rss_stats): - hotlist_stats: 与 count_word_frequency() 产出格式一致 - rss_stats: 与 rss_items 格式一致 """ hotlist_stats = [] rss_stats = [] max_news = self.config.get("MAX_NEWS_PER_KEYWORD", 0) min_score = self.ai_filter_config.get("MIN_SCORE", 0) # current 模式：计算最新时间，只保留当前在榜的热榜新闻 # 与 count_word_frequency(mode="current") 的过滤逻辑对齐 latest_time = None if mode == "current": for tag_data in ai_filter_result.tags: for item in tag_data.get("items", []): if item.get("source_type", "hotlist") == "hotlist": last_time = item.get("last_time", "") if last_time and (latest_time is None or last_time > latest_time): latest_time = last_time if latest_time: print(f"[AI筛选] current 模式：最新时间 {latest_time}，过滤已下榜新闻") # RSS 新鲜度过滤配置（与推送阶段一致） rss_config = self.rss_config freshness_config = rss_config.get("FRESHNESS_FILTER", {}) freshness_enabled = freshness_config.get("ENABLED", True) default_max_age_days = freshness_config.get("MAX_AGE_DAYS", 3) timezone = self.config.get("TIMEZONE", DEFAULT_TIMEZONE) feed_max_age_map = {} for feed_cfg in self.rss_feeds: feed_id = feed_cfg.get("id", "") max_age = feed_cfg.get("max_age_days") if max_age is not None: try: feed_max_age_map[feed_id] = int(max_age) except (ValueError, TypeError): pass filtered_count = 0 for tag_data in ai_filter_result.tags: tag_name = tag_data.get("tag", "") items = tag_data.get("items", []) if not items: continue hotlist_titles = [] rss_titles = [] for item in items: source_type = item.get("source_type", "hotlist") # current 模式：跳过已下榜的热榜新闻 if mode == "current" and latest_time and source_type == "hotlist": if item.get("last_time", "") != latest_time: filtered_count += 1 continue # 分数阈值过滤：跳过相关度低于 min_score 的新闻 if min_score > 0: score = item.get("relevance_score", 0) if score < min_score: continue # 构建时间显示 first_time = item.get("first_time", "") last_time = item.get("last_time", "") if source_type == "rss": # RSS 新鲜度过滤：跳过超过 max_age_days 的旧文章 if freshness_enabled and first_time: feed_id = item.get("source_id", "") max_days = feed_max_age_map.get(feed_id, default_max_age_days) if max_days > 0 and not is_within_days(first_time, max_days, timezone): continue # RSS 条目：first_time 是 ISO 格式，用友好格式显示 if first_time: time_display = format_iso_time_friendly(first_time, timezone, include_date=True) else: time_display = "" else: # 热榜条目：使用 [HH:MM ~ HH:MM] 格式（与 keyword 模式一致） if first_time and last_time and first_time != last_time: first_display = convert_time_for_display(first_time) last_display = convert_time_for_display(last_time) time_display = f"[{first_display} ~ {last_display}]" elif first_time: time_display = convert_time_for_display(first_time) else: time_display = "" # 计算 is_new（与 keyword 模式 core/analyzer.py:335-342 对齐） if source_type == "rss": is_new = False if rss_new_urls: item_url = item.get("url", "") is_new = item_url in rss_new_urls if item_url else False else: is_new = False if new_titles: item_source_id = item.get("source_id", "") item_title = item.get("title", "") if item_source_id in new_titles: is_new = item_title in new_titles[item_source_id] # incremental 模式下仅保留本轮新增命中的条目。 # run_ai_filter() 返回的是 active 结果集合，因此这里需要 # 显式过滤掉历史已命中的旧条目，才能与 keyword 模式行为对齐。 if mode == "incremental" and not is_new: continue title_entry = { "title": item.get("title", ""), "source_name": item.get("source_name", ""), "url": item.get("url", ""), "mobile_url": item.get("mobile_url", ""), "ranks": item.get("ranks", []), "rank_threshold": self.rank_threshold, "count": item.get("count", 1), "is_new": is_new, "time_display": time_display, "matched_keyword": tag_name, } if source_type == "rss": rss_titles.append(title_entry) else: hotlist_titles.append(title_entry) if hotlist_titles: if max_news > 0: hotlist_titles = hotlist_titles[:max_news] hotlist_stats.append({ "word": tag_name, "count": len(hotlist_titles), "position": tag_data.get("position", 9999), "titles": hotlist_titles, }) if rss_titles: if max_news > 0: rss_titles = rss_titles[:max_news] rss_stats.append({ "word": tag_name, "count": len(rss_titles), "position": tag_data.get("position", 9999), "titles": rss_titles, }) if mode == "current" and filtered_count > 0: total_kept = sum(s["count"] for s in hotlist_stats) print(f"[AI筛选] current 模式：过滤 {filtered_count} 条已下榜新闻，保留 {total_kept} 条当前在榜") if min_score > 0: hotlist_kept = sum(s["count"] for s in hotlist_stats) rss_kept = sum(s["count"] for s in rss_stats) total_kept = hotlist_kept + rss_kept parts = [f"热榜 {hotlist_kept} 条"] if rss_kept > 0: parts.append(f"RSS {rss_kept} 条") print(f"[AI筛选] 分数过滤：min_score={min_score}，保留 {total_kept} 条 score≥{min_score} ({', '.join(parts)})") priority_sort_enabled = self.ai_priority_sort_enabled if priority_sort_enabled: hotlist_stats.sort(key=lambda x: (x.get("position", 9999), -x["count"], x["word"])) rss_stats.sort(key=lambda x: (x.get("position", 9999), -x["count"], x["word"])) else: hotlist_stats.sort(key=lambda x: (-x["count"], x.get("position", 9999), x["word"])) rss_stats.sort(key=lambda x: (-x["count"], x.get("position", 9999), x["word"])) return hotlist_stats, rss_stats # === 资源清理 === def cleanup(self): """清理资源""" if self._storage_manager: self._storage_manager.cleanup_old_data() self._storage_manager.cleanup() self._storage_manager = None ================================================ FILE: trendradar/core/__init__.py ================================================ # coding=utf-8 """ 核心模块 - 配置管理和核心工具 """ from trendradar.core.config import ( parse_multi_account_config, validate_paired_configs, limit_accounts, get_account_at_index, ) from trendradar.core.loader import load_config from trendradar.core.frequency import load_frequency_words, matches_word_groups from trendradar.core.scheduler import Scheduler, ResolvedSchedule from trendradar.core.data import ( read_all_today_titles_from_storage, read_all_today_titles, detect_latest_new_titles_from_storage, detect_latest_new_titles, ) from trendradar.core.analyzer import ( calculate_news_weight, format_time_display, count_word_frequency, count_rss_frequency, ) __all__ = [ "parse_multi_account_config", "validate_paired_configs", "limit_accounts", "get_account_at_index", "load_config", "load_frequency_words", "matches_word_groups", # 数据处理 "read_all_today_titles_from_storage", "read_all_today_titles", "detect_latest_new_titles_from_storage", "detect_latest_new_titles", # 统计分析 "calculate_news_weight", "format_time_display", "count_word_frequency", "count_rss_frequency", # 调度器 "Scheduler", "ResolvedSchedule", ] ================================================ FILE: trendradar/core/analyzer.py ================================================ # coding=utf-8 """ 统计分析模块 提供新闻统计和分析功能： - calculate_news_weight: 计算新闻权重 - format_time_display: 格式化时间显示 - count_word_frequency: 统计词频 """ from typing import Dict, List, Tuple, Optional, Callable from trendradar.core.frequency import matches_word_groups, _word_matches from trendradar.utils.time import DEFAULT_TIMEZONE def calculate_news_weight( title_data: Dict, rank_threshold: int, weight_config: Dict, ) -> float: """ 计算新闻权重，用于排序 Args: title_data: 标题数据，包含 ranks 和 count rank_threshold: 排名阈值 weight_config: 权重配置 {RANK_WEIGHT, FREQUENCY_WEIGHT, HOTNESS_WEIGHT} Returns: float: 计算出的权重值 """ ranks = title_data.get("ranks", []) if not ranks: return 0.0 count = title_data.get("count", len(ranks)) # 排名权重：Σ(11 - min(rank, 10)) / 出现次数 rank_scores = [] for rank in ranks: score = 11 - min(rank, 10) rank_scores.append(score) rank_weight = sum(rank_scores) / len(ranks) if ranks else 0 # 频次权重：min(出现次数, 10) × 10 frequency_weight = min(count, 10) * 10 # 热度加成：高排名次数 / 总出现次数 × 100 high_rank_count = sum(1 for rank in ranks if rank <= rank_threshold) hotness_ratio = high_rank_count / len(ranks) if ranks else 0 hotness_weight = hotness_ratio * 100 total_weight = ( rank_weight * weight_config["RANK_WEIGHT"] + frequency_weight * weight_config["FREQUENCY_WEIGHT"] + hotness_weight * weight_config["HOTNESS_WEIGHT"] ) return total_weight def format_time_display( first_time: str, last_time: str, convert_time_func: Callable[[str], str], ) -> str: """ 格式化时间显示（将 HH-MM 转换为 HH:MM） Args: first_time: 首次出现时间 last_time: 最后出现时间 convert_time_func: 时间格式转换函数 Returns: str: 格式化后的时间显示字符串 """ if not first_time: return "" # 转换为显示格式 first_display = convert_time_func(first_time) last_display = convert_time_func(last_time) if first_display == last_display or not last_display: return first_display else: return f"[{first_display} ~ {last_display}]" def count_word_frequency( results: Dict, word_groups: List[Dict], filter_words: List[str], id_to_name: Dict, title_info: Optional[Dict] = None, rank_threshold: int = 3, new_titles: Optional[Dict] = None, mode: str = "daily", global_filters: Optional[List[str]] = None, weight_config: Optional[Dict] = None, max_news_per_keyword: int = 0, sort_by_position_first: bool = False, is_first_crawl_func: Optional[Callable[[], bool]] = None, convert_time_func: Optional[Callable[[str], str]] = None, quiet: bool = False, ) -> Tuple[List[Dict], int]: """ 统计词频，支持必须词、频率词、过滤词、全局过滤词，并标记新增标题 Args: results: 抓取结果 {source_id: {title: title_data}} word_groups: 词组配置列表 filter_words: 过滤词列表 id_to_name: ID 到名称的映射 title_info: 标题统计信息（可选） rank_threshold: 排名阈值 new_titles: 新增标题（可选） mode: 报告模式 (daily/incremental/current) global_filters: 全局过滤词（可选） weight_config: 权重配置 max_news_per_keyword: 每个关键词最大显示数量 sort_by_position_first: 是否优先按配置位置排序 is_first_crawl_func: 检测是否是当天第一次爬取的函数 convert_time_func: 时间格式转换函数 quiet: 是否静默模式（不打印日志） Returns: Tuple[List[Dict], int]: (统计结果列表, 总标题数) """ # 默认权重配置 if weight_config is None: weight_config = { "RANK_WEIGHT": 0.4, "FREQUENCY_WEIGHT": 0.3, "HOTNESS_WEIGHT": 0.3, } # 默认时间转换函数 if convert_time_func is None: convert_time_func = lambda x: x # 默认首次爬取检测函数 if is_first_crawl_func is None: is_first_crawl_func = lambda: True # 如果没有配置词组，创建一个包含所有新闻的虚拟词组 if not word_groups: print("频率词配置为空，将显示所有新闻") word_groups = [{"required": [], "normal": [], "group_key": "全部新闻"}] filter_words = [] # 清空过滤词，显示所有新闻 is_first_today = is_first_crawl_func() # 确定处理的数据源和新增标记逻辑 if mode == "incremental": if is_first_today: # 增量模式 + 当天第一次：处理所有新闻，都标记为新增 results_to_process = results all_news_are_new = True else: # 增量模式 + 当天非第一次：只处理新增的新闻 results_to_process = new_titles if new_titles else {} all_news_are_new = True elif mode == "current": # current 模式：只处理当前时间批次的新闻，但统计信息来自全部历史 if title_info: latest_time = None for source_titles in title_info.values(): for title_data in source_titles.values(): last_time = title_data.get("last_time", "") if last_time: if latest_time is None or last_time > latest_time: latest_time = last_time # 只处理 last_time 等于最新时间的新闻 if latest_time: results_to_process = {} for source_id, source_titles in results.items(): if source_id in title_info: filtered_titles = {} for title, title_data in source_titles.items(): if title in title_info[source_id]: info = title_info[source_id][title] if info.get("last_time") == latest_time: filtered_titles[title] = title_data if filtered_titles: results_to_process[source_id] = filtered_titles if not quiet: print( f"当前榜单模式：最新时间 {latest_time}，筛选出 {sum(len(titles) for titles in results_to_process.values())} 条当前榜单新闻" ) else: results_to_process = results else: results_to_process = results all_news_are_new = False else: # 当日汇总模式：处理所有新闻 results_to_process = results all_news_are_new = False total_input_news = sum(len(titles) for titles in results.values()) filter_status = ( "全部显示" if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻" else "频率词过滤" ) print(f"当日汇总模式：处理 {total_input_news} 条新闻，模式：{filter_status}") word_stats = {} total_titles = 0 processed_titles = {} matched_new_count = 0 if title_info is None: title_info = {} if new_titles is None: new_titles = {} for group in word_groups: group_key = group["group_key"] word_stats[group_key] = {"count": 0, "titles": {}} for source_id, titles_data in results_to_process.items(): total_titles += len(titles_data) if source_id not in processed_titles: processed_titles[source_id] = {} for title, title_data in titles_data.items(): if title in processed_titles.get(source_id, {}): continue # 使用统一的匹配逻辑 matches_frequency_words = matches_word_groups( title, word_groups, filter_words, global_filters ) if not matches_frequency_words: continue # 如果是增量模式或 current 模式第一次，统计匹配的新增新闻数量 if (mode == "incremental" and all_news_are_new) or ( mode == "current" and is_first_today ): matched_new_count += 1 source_ranks = title_data.get("ranks", []) source_url = title_data.get("url", "") source_mobile_url = title_data.get("mobileUrl", "") # 找到匹配的词组（防御性转换确保类型安全） title_lower = str(title).lower() if not isinstance(title, str) else title.lower() for group in word_groups: required_words = group["required"] normal_words = group["normal"] # 如果是"全部新闻"模式，所有标题都匹配第一个（唯一的）词组 if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻": group_key = group["group_key"] word_stats[group_key]["count"] += 1 if source_id not in word_stats[group_key]["titles"]: word_stats[group_key]["titles"][source_id] = [] else: # 原有的匹配逻辑（支持正则语法） if required_words: all_required_present = all( _word_matches(req_item, title_lower) for req_item in required_words ) if not all_required_present: continue if normal_words: any_normal_present = any( _word_matches(normal_item, title_lower) for normal_item in normal_words ) if not any_normal_present: continue group_key = group["group_key"] word_stats[group_key]["count"] += 1 if source_id not in word_stats[group_key]["titles"]: word_stats[group_key]["titles"][source_id] = [] first_time = "" last_time = "" count_info = 1 ranks = source_ranks if source_ranks else [] url = source_url mobile_url = source_mobile_url rank_timeline = [] # 对于 current 模式，从历史统计信息中获取完整数据 if ( mode == "current" and title_info and source_id in title_info and title in title_info[source_id] ): info = title_info[source_id][title] first_time = info.get("first_time", "") last_time = info.get("last_time", "") count_info = info.get("count", 1) if "ranks" in info and info["ranks"]: ranks = info["ranks"] url = info.get("url", source_url) mobile_url = info.get("mobileUrl", source_mobile_url) rank_timeline = info.get("rank_timeline", []) elif ( title_info and source_id in title_info and title in title_info[source_id] ): info = title_info[source_id][title] first_time = info.get("first_time", "") last_time = info.get("last_time", "") count_info = info.get("count", 1) if "ranks" in info and info["ranks"]: ranks = info["ranks"] url = info.get("url", source_url) mobile_url = info.get("mobileUrl", source_mobile_url) rank_timeline = info.get("rank_timeline", []) if not ranks: ranks = [99] time_display = format_time_display(first_time, last_time, convert_time_func) source_name = id_to_name.get(source_id, source_id) # 判断是否为新增 is_new = False if all_news_are_new: # 增量模式下所有处理的新闻都是新增，或者当天第一次的所有新闻都是新增 is_new = True elif new_titles and source_id in new_titles: # 检查是否在新增列表中 new_titles_for_source = new_titles[source_id] is_new = title in new_titles_for_source word_stats[group_key]["titles"][source_id].append( { "title": title, "source_name": source_name, "first_time": first_time, "last_time": last_time, "time_display": time_display, "count": count_info, "ranks": ranks, "rank_threshold": rank_threshold, "url": url, "mobileUrl": mobile_url, "is_new": is_new, "rank_timeline": rank_timeline, } ) if source_id not in processed_titles: processed_titles[source_id] = {} processed_titles[source_id][title] = True break # 最后统一打印汇总信息 if mode == "incremental": if is_first_today: total_input_news = sum(len(titles) for titles in results.values()) filter_status = ( "全部显示" if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻" else "频率词匹配" ) if not quiet: print( f"增量模式：当天第一次爬取，{total_input_news} 条新闻中有 {matched_new_count} 条{filter_status}" ) else: if new_titles: total_new_count = sum(len(titles) for titles in new_titles.values()) filter_status = ( "全部显示" if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻" else "匹配频率词" ) if not quiet: print( f"增量模式：{total_new_count} 条新增新闻中，有 {matched_new_count} 条{filter_status}" ) if matched_new_count == 0 and len(word_groups) > 1: print("增量模式：没有新增新闻匹配频率词，将不会发送通知") else: if not quiet: print("增量模式：未检测到新增新闻") elif mode == "current": total_input_news = sum(len(titles) for titles in results_to_process.values()) if is_first_today: filter_status = ( "全部显示" if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻" else "频率词匹配" ) if not quiet: print( f"当前榜单模式：当天第一次爬取，{total_input_news} 条当前榜单新闻中有 {matched_new_count} 条{filter_status}" ) else: matched_count = sum(stat["count"] for stat in word_stats.values()) filter_status = ( "全部显示" if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部新闻" else "频率词匹配" ) if not quiet: print( f"当前榜单模式：{total_input_news} 条当前榜单新闻中有 {matched_count} 条{filter_status}" ) stats = [] # 创建 group_key 到位置、最大数量、显示名称的映射 group_key_to_position = { group["group_key"]: idx for idx, group in enumerate(word_groups) } group_key_to_max_count = { group["group_key"]: group.get("max_count", 0) for group in word_groups } group_key_to_display_name = { group["group_key"]: group.get("display_name") for group in word_groups } for group_key, data in word_stats.items(): all_titles = [] for source_id, title_list in data["titles"].items(): all_titles.extend(title_list) # 按权重排序 sorted_titles = sorted( all_titles, key=lambda x: ( -calculate_news_weight(x, rank_threshold, weight_config), min(x["ranks"]) if x["ranks"] else 999, -x["count"], ), ) # 应用最大显示数量限制（优先级：单独配置 > 全局配置） group_max_count = group_key_to_max_count.get(group_key, 0) if group_max_count == 0: # 使用全局配置 group_max_count = max_news_per_keyword if group_max_count > 0: sorted_titles = sorted_titles[:group_max_count] # 优先使用 display_name，否则使用 group_key display_word = group_key_to_display_name.get(group_key) or group_key stats.append( { "word": display_word, "count": data["count"], "position": group_key_to_position.get(group_key, 999), "titles": sorted_titles, "percentage": ( round(data["count"] / total_titles * 100, 2) if total_titles > 0 else 0 ), } ) # 根据配置选择排序优先级 if sort_by_position_first: # 先按配置位置，再按热点条数 stats.sort(key=lambda x: (x["position"], -x["count"])) else: # 先按热点条数，再按配置位置（原逻辑） stats.sort(key=lambda x: (-x["count"], x["position"])) # 打印过滤后的匹配新闻数 matched_news_count = sum(len(stat["titles"]) for stat in stats if stat["count"] > 0) if not quiet and mode == "daily": print(f"当日汇总模式：处理 {total_titles} 条新闻，模式：频率词过滤") print(f"频率词过滤后：{matched_news_count} 条新闻匹配") return stats, total_titles def count_rss_frequency( rss_items: List[Dict], word_groups: List[Dict], filter_words: List[str], global_filters: Optional[List[str]] = None, new_items: Optional[List[Dict]] = None, max_news_per_keyword: int = 0, sort_by_position_first: bool = False, timezone: str = DEFAULT_TIMEZONE, rank_threshold: int = 5, quiet: bool = False, ) -> Tuple[List[Dict], int]: """ 按关键词分组统计 RSS 条目（与热榜统计格式一致） Args: rss_items: RSS 条目列表，每个条目包含： - title: 标题 - feed_id: RSS 源 ID - feed_name: RSS 源名称 - url: 文章链接 - published_at: 发布时间（ISO 格式） word_groups: 词组配置列表 filter_words: 过滤词列表 global_filters: 全局过滤词（可选） new_items: 新增条目列表（可选，用于标记 is_new） max_news_per_keyword: 每个关键词最大显示数量 sort_by_position_first: 是否优先按配置位置排序 timezone: 时区名称（用于时间格式化） quiet: 是否静默模式 Returns: Tuple[List[Dict], int]: (统计结果列表, 总条目数) 统计结果格式与热榜一致： [ { "word": "关键词", "count": 5, "position": 0, "titles": [ { "title": "标题", "source_name": "Hacker News", "time_display": "12-29 08:20", "count": 1, "ranks": [1], # RSS 用发布时间顺序作为排名 "rank_threshold": 50, "url": "...", "mobile_url": "", "is_new": True/False } ], "percentage": 10.0 } ] """ from trendradar.utils.time import format_iso_time_friendly if not rss_items: return [], 0 # 如果没有配置词组，创建一个包含所有条目的虚拟词组 if not word_groups: if not quiet: print("[RSS] 频率词配置为空，将显示所有 RSS 条目") word_groups = [{"required": [], "normal": [], "group_key": "全部 RSS"}] filter_words = [] # 创建新增条目的 URL 集合，用于快速查找 new_urls = set() if new_items: for item in new_items: if item.get("url"): new_urls.add(item["url"]) # 初始化词组统计 word_stats = {} for group in word_groups: group_key = group["group_key"] word_stats[group_key] = {"count": 0, "titles": []} total_items = len(rss_items) processed_urls = set() # 用于去重 # 为每个条目分配一个基于发布时间的"排名" # 按发布时间排序，最新的排在前面 sorted_items = sorted( rss_items, key=lambda x: x.get("published_at", ""), reverse=True ) url_to_rank = {item.get("url", ""): idx + 1 for idx, item in enumerate(sorted_items)} for item in rss_items: title = item.get("title", "") url = item.get("url", "") # 去重 if url and url in processed_urls: continue if url: processed_urls.add(url) # 使用统一的匹配逻辑 if not matches_word_groups(title, word_groups, filter_words, global_filters): continue # 找到匹配的词组 title_lower = title.lower() for group in word_groups: required_words = group["required"] normal_words = group["normal"] group_key = group["group_key"] # "全部 RSS" 模式：所有条目都匹配 if len(word_groups) == 1 and word_groups[0]["group_key"] == "全部 RSS": matched = True else: # 检查必须词（支持正则语法） if required_words: all_required_present = all( _word_matches(req_item, title_lower) for req_item in required_words ) if not all_required_present: continue # 检查普通词（支持正则语法） if normal_words: any_normal_present = any( _word_matches(normal_item, title_lower) for normal_item in normal_words ) if not any_normal_present: continue matched = True if matched: word_stats[group_key]["count"] += 1 # 格式化时间显示 published_at = item.get("published_at", "") time_display = format_iso_time_friendly(published_at, timezone, include_date=True) if published_at else "" # 判断是否为新增 is_new = url in new_urls if url else False # 获取排名（基于发布时间顺序） rank = url_to_rank.get(url, 99) if url else 99 title_data = { "title": title, "source_name": item.get("feed_name", item.get("feed_id", "RSS")), "time_display": time_display, "count": 1, # RSS 条目通常只出现一次 "ranks": [rank], "rank_threshold": rank_threshold, "url": url, "mobile_url": "", "is_new": is_new, } word_stats[group_key]["titles"].append(title_data) break # 一个条目只匹配第一个词组 # 构建统计结果 stats = [] group_key_to_position = { group["group_key"]: idx for idx, group in enumerate(word_groups) } group_key_to_max_count = { group["group_key"]: group.get("max_count", 0) for group in word_groups } group_key_to_display_name = { group["group_key"]: group.get("display_name") for group in word_groups } for group_key, data in word_stats.items(): if data["count"] == 0: continue # 按发布时间排序（最新在前） sorted_titles = sorted( data["titles"], key=lambda x: x["ranks"][0] if x["ranks"] else 999 ) # 应用最大显示数量限制 group_max_count = group_key_to_max_count.get(group_key, 0) if group_max_count == 0: group_max_count = max_news_per_keyword if group_max_count > 0: sorted_titles = sorted_titles[:group_max_count] # 优先使用 display_name，否则使用 group_key display_word = group_key_to_display_name.get(group_key) or group_key stats.append({ "word": display_word, "count": data["count"], "position": group_key_to_position.get(group_key, 999), "titles": sorted_titles, "percentage": round(data["count"] / total_items * 100, 2) if total_items > 0 else 0, }) # 排序 if sort_by_position_first: stats.sort(key=lambda x: (x["position"], -x["count"])) else: stats.sort(key=lambda x: (-x["count"], x["position"])) matched_count = sum(stat["count"] for stat in stats) if not quiet: print(f"[RSS] 关键词分组统计：{matched_count}/{total_items} 条匹配") return stats, total_items def convert_keyword_stats_to_platform_stats( keyword_stats: List[Dict], weight_config: Dict, rank_threshold: int = 5, ) -> List[Dict]: """ 将按关键词分组的统计数据转换为按平台分组的统计数据 Args: keyword_stats: 原始按关键词分组的统计数据 weight_config: 权重配置 rank_threshold: 排名阈值 Returns: 按平台分组的统计数据，格式与原 stats 一致 """ # 1. 收集所有新闻，按平台分组 platform_map: Dict[str, List[Dict]] = {} for stat in keyword_stats: keyword = stat["word"] for title_data in stat["titles"]: source_name = title_data["source_name"] if source_name not in platform_map: platform_map[source_name] = [] # 复制 title_data 并添加匹配的关键词 title_with_keyword = title_data.copy() title_with_keyword["matched_keyword"] = keyword platform_map[source_name].append(title_with_keyword) # 2. 去重（同一平台下相同标题只保留一条，保留第一个匹配的关键词） for source_name, titles in platform_map.items(): seen_titles: Dict[str, bool] = {} unique_titles = [] for title_data in titles: title_text = title_data["title"] if title_text not in seen_titles: seen_titles[title_text] = True unique_titles.append(title_data) platform_map[source_name] = unique_titles # 3. 按权重排序每个平台内的新闻 for source_name, titles in platform_map.items(): platform_map[source_name] = sorted( titles, key=lambda x: ( -calculate_news_weight(x, rank_threshold, weight_config), min(x["ranks"]) if x["ranks"] else 999, -x["count"], ), ) # 4. 构建平台统计结果 platform_stats = [] for source_name, titles in platform_map.items(): platform_stats.append({ "word": source_name, # 平台名作为分组标识 "count": len(titles), "titles": titles, "percentage": 0, # 可后续计算 }) # 5. 按新闻条数排序平台 platform_stats.sort(key=lambda x: -x["count"]) return platform_stats ================================================ FILE: trendradar/core/config.py ================================================ # coding=utf-8 """ 配置工具模块 - 多账号配置解析和验证 提供多账号推送配置的解析、验证和限制功能 """ from typing import Dict, List, Optional, Tuple def parse_multi_account_config(config_value: str, separator: str = ";") -> List[str]: """ 解析多账号配置，返回账号列表 Args: config_value: 配置值字符串，多个账号用分隔符分隔 separator: 分隔符，默认为 ; Returns: 账号列表，空字符串会被保留（用于占位） Examples: >>> parse_multi_account_config("url1;url2;url3") ['url1', 'url2', 'url3'] >>> parse_multi_account_config(";token2") # 第一个账号无token ['', 'token2'] >>> parse_multi_account_config("") [] """ if not config_value: return [] # 保留空字符串用于占位（如 ";token2" 表示第一个账号无token） accounts = [acc.strip() for acc in config_value.split(separator)] # 过滤掉全部为空的情况 if all(not acc for acc in accounts): return [] return accounts def validate_paired_configs( configs: Dict[str, List[str]], channel_name: str, required_keys: Optional[List[str]] = None ) -> Tuple[bool, int]: """ 验证配对配置的数量是否一致 对于需要多个配置项配对的渠道（如 Telegram 的 token 和 chat_id）， 验证所有配置项的账号数量是否一致。 Args: configs: 配置字典，key 为配置名，value 为账号列表 channel_name: 渠道名称，用于日志输出 required_keys: 必须有值的配置项列表 Returns: (是否验证通过, 账号数量) Examples: >>> validate_paired_configs({ ... "token": ["t1", "t2"], ... "chat_id": ["c1", "c2"] ... }, "Telegram", ["token", "chat_id"]) (True, 2) >>> validate_paired_configs({ ... "token": ["t1", "t2"], ... "chat_id": ["c1"] # 数量不匹配 ... }, "Telegram", ["token", "chat_id"]) (False, 0) """ # 过滤掉空列表 non_empty_configs = {k: v for k, v in configs.items() if v} if not non_empty_configs: return True, 0 # 检查必须项 if required_keys: for key in required_keys: if key not in non_empty_configs or not non_empty_configs[key]: return True, 0 # 必须项为空，视为未配置 # 获取所有非空配置的长度 lengths = {k: len(v) for k, v in non_empty_configs.items()} unique_lengths = set(lengths.values()) if len(unique_lengths) > 1: print(f"❌ {channel_name} 配置错误：配对配置数量不一致，将跳过该渠道推送") for key, length in lengths.items(): print(f" - {key}: {length} 个") return False, 0 return True, list(unique_lengths)[0] if unique_lengths else 0 def limit_accounts( accounts: List[str], max_count: int, channel_name: str ) -> List[str]: """ 限制账号数量 当配置的账号数量超过最大限制时，只使用前 N 个账号， 并输出警告信息。 Args: accounts: 账号列表 max_count: 最大账号数量 channel_name: 渠道名称，用于日志输出 Returns: 限制后的账号列表 Examples: >>> limit_accounts(["a1", "a2", "a3"], 2, "飞书") ⚠️ 飞书 配置了 3 个账号，超过最大限制 2，只使用前 2 个 ['a1', 'a2'] """ if len(accounts) > max_count: print(f"⚠️ {channel_name} 配置了 {len(accounts)} 个账号，超过最大限制 {max_count}，只使用前 {max_count} 个") print(f" ⚠️ 警告：如果你是 fork 用户，过多账号可能导致 GitHub Actions 运行时间过长，存在账号风险") return accounts[:max_count] return accounts def get_account_at_index(accounts: List[str], index: int, default: str = "") -> str: """ 安全获取指定索引的账号值 当索引超出范围或账号值为空时，返回默认值。 Args: accounts: 账号列表 index: 索引 default: 默认值 Returns: 账号值或默认值 Examples: >>> get_account_at_index(["a", "b", "c"], 1) 'b' >>> get_account_at_index(["a", "", "c"], 1, "default") 'default' >>> get_account_at_index(["a"], 5, "default") 'default' """ if index < len(accounts): return accounts[index] if accounts[index] else default return default ================================================ FILE: trendradar/core/data.py ================================================ # coding=utf-8 """ 数据处理模块 提供数据读取和检测功能： - read_all_today_titles: 从存储后端读取当天所有标题 - detect_latest_new_titles: 检测最新批次的新增标题 Author: TrendRadar Team """ from typing import Dict, List, Tuple, Optional def read_all_today_titles_from_storage( storage_manager, current_platform_ids: Optional[List[str]] = None, ) -> Tuple[Dict, Dict, Dict]: """ 从存储后端读取当天所有标题（SQLite 数据） Args: storage_manager: 存储管理器实例 current_platform_ids: 当前监控的平台 ID 列表（用于过滤） Returns: Tuple[Dict, Dict, Dict]: (all_results, id_to_name, title_info) """ try: news_data = storage_manager.get_today_all_data() if not news_data or not news_data.items: return {}, {}, {} all_results = {} final_id_to_name = {} title_info = {} for source_id, news_list in news_data.items.items(): # 按平台过滤 if current_platform_ids is not None and source_id not in current_platform_ids: continue # 获取来源名称 source_name = news_data.id_to_name.get(source_id, source_id) final_id_to_name[source_id] = source_name if source_id not in all_results: all_results[source_id] = {} title_info[source_id] = {} for item in news_list: title = item.title ranks = item.ranks or [item.rank] first_time = item.first_time or item.crawl_time last_time = item.last_time or item.crawl_time count = item.count rank_timeline = item.rank_timeline all_results[source_id][title] = { "ranks": ranks, "url": item.url or "", "mobileUrl": item.mobile_url or "", } title_info[source_id][title] = { "first_time": first_time, "last_time": last_time, "count": count, "ranks": ranks, "url": item.url or "", "mobileUrl": item.mobile_url or "", "rank_timeline": rank_timeline, } return all_results, final_id_to_name, title_info except Exception as e: print(f"[存储] 从存储后端读取数据失败: {e}") return {}, {}, {} def read_all_today_titles( storage_manager, current_platform_ids: Optional[List[str]] = None, quiet: bool = False, ) -> Tuple[Dict, Dict, Dict]: """ 读取当天所有标题（从存储后端） Args: storage_manager: 存储管理器实例 current_platform_ids: 当前监控的平台 ID 列表（用于过滤） quiet: 是否静默模式（不打印日志） Returns: Tuple[Dict, Dict, Dict]: (all_results, id_to_name, title_info) """ all_results, final_id_to_name, title_info = read_all_today_titles_from_storage( storage_manager, current_platform_ids ) if not quiet: if all_results: total_count = sum(len(titles) for titles in all_results.values()) print(f"[存储] 已从存储后端读取 {total_count} 条标题") else: print("[存储] 当天暂无数据") return all_results, final_id_to_name, title_info def detect_latest_new_titles_from_storage( storage_manager, current_platform_ids: Optional[List[str]] = None, ) -> Dict: """ 从存储后端检测最新批次的新增标题 Args: storage_manager: 存储管理器实例 current_platform_ids: 当前监控的平台 ID 列表（用于过滤） Returns: Dict: 新增标题 {source_id: {title: title_data}} """ try: # 获取最新抓取数据 latest_data = storage_manager.get_latest_crawl_data() if not latest_data or not latest_data.items: return {} # 获取所有历史数据 all_data = storage_manager.get_today_all_data() if not all_data or not all_data.items: # 没有历史数据（第一次抓取），不应该有"新增"标题 return {} # 获取最新批次时间 latest_time = latest_data.crawl_time # 步骤1：收集最新批次的标题（last_crawl_time = latest_time 的标题） latest_titles = {} for source_id, news_list in latest_data.items.items(): if current_platform_ids is not None and source_id not in current_platform_ids: continue latest_titles[source_id] = {} for item in news_list: latest_titles[source_id][item.title] = { "ranks": [item.rank], "url": item.url or "", "mobileUrl": item.mobile_url or "", } # 步骤2：收集历史标题 # 关键逻辑：一个标题只要其 first_crawl_time < latest_time，就是历史标题 # 这样即使同一标题有多条记录（URL 不同），只要任何一条是历史的，该标题就算历史 historical_titles = {} for source_id, news_list in all_data.items.items(): if current_platform_ids is not None and source_id not in current_platform_ids: continue historical_titles[source_id] = set() for item in news_list: first_time = item.first_time or item.crawl_time # 如果该记录的首次出现时间早于最新批次，则该标题是历史标题 if first_time < latest_time: historical_titles[source_id].add(item.title) # 检查是否是当天第一次抓取（没有任何历史标题） # 如果所有平台的历史标题集合都为空，说明只有一个抓取批次 # 在这种情况下，将所有最新批次的标题视为"新增"（用于增量模式的第一次推送） has_historical_data = any(len(titles) > 0 for titles in historical_titles.values()) if not has_historical_data: # 第一次爬取：返回所有最新标题作为"新增" return latest_titles # 步骤3：找出新增标题 = 最新批次标题 - 历史标题 new_titles = {} for source_id, source_latest_titles in latest_titles.items(): historical_set = historical_titles.get(source_id, set()) source_new_titles = {} for title, title_data in source_latest_titles.items(): if title not in historical_set: source_new_titles[title] = title_data if source_new_titles: new_titles[source_id] = source_new_titles return new_titles except Exception as e: print(f"[存储] 从存储后端检测新标题失败: {e}") return {} def detect_latest_new_titles( storage_manager, current_platform_ids: Optional[List[str]] = None, quiet: bool = False, ) -> Dict: """ 检测当日最新批次的新增标题（从存储后端） Args: storage_manager: 存储管理器实例 current_platform_ids: 当前监控的平台 ID 列表（用于过滤） quiet: 是否静默模式（不打印日志） Returns: Dict: 新增标题 {source_id: {title: title_data}} """ new_titles = detect_latest_new_titles_from_storage(storage_manager, current_platform_ids) if new_titles and not quiet: total_new = sum(len(titles) for titles in new_titles.values()) print(f"[存储] 从存储后端检测到 {total_new} 条新增标题") return new_titles ================================================ FILE: trendradar/core/frequency.py ================================================ # coding=utf-8 """ 频率词配置加载模块 负责从配置文件加载频率词规则，支持： - 普通词组 - 必须词（+前缀） - 过滤词（!前缀） - 全局过滤词（[GLOBAL_FILTER] 区域） - 最大显示数量（@前缀） - 正则表达式（/pattern/ 语法） - 显示名称（=> 别名 语法） - 组别名（[组别名] 语法，作为词组第一行） """ import os import re from pathlib import Path from typing import Dict, List, Tuple, Optional, Union def _parse_word(word: str) -> Dict: """ 解析单个词，识别是否为正则表达式，支持显示名称 Args: word: 原始配置行 (e.g. "/京东|刘强东/ => 京东") Returns: Dict: 包含 word, is_regex, pattern, display_name """ display_name = None # 1. 优先处理显示名称 (=>) # 先切分出 "配置内容" 和 "显示名称" if '=>' in word: parts = re.split(r'\s*=>\s*', word, 1) word_config = parts[0].strip() # 只有当 => 右边有内容时才作为 display_name if len(parts) > 1 and parts[1].strip(): display_name = parts[1].strip() else: word_config = word.strip() # 2. 解析正则表达式 # 规则：以 / 开头，以 / 结尾(可能跟 flags)，中间内容贪婪提取 # [a-z]*$ 表示允许末尾有 flags (如 i, g)，但在下面代码中会被忽略 regex_match = re.match(r'^/(.+)/[a-z]*$', word_config) if regex_match: pattern_str = regex_match.group(1) try: pattern = re.compile(pattern_str, re.IGNORECASE) return { "word": pattern_str, "is_regex": True, "pattern": pattern, "display_name": display_name, } except re.error as e: print(f"Warning: Invalid regex pattern '/{pattern_str}/': {e}") pass return { "word": word_config, "is_regex": False, "pattern": None, "display_name": display_name } def _word_matches(word_config: Union[str, Dict], title_lower: str) -> bool: """ 检查词是否在标题中匹配 Args: word_config: 词配置（字符串或字典） title_lower: 小写的标题 Returns: 是否匹配 """ if isinstance(word_config, str): # 向后兼容：纯字符串 return word_config.lower() in title_lower if word_config.get("is_regex") and word_config.get("pattern"): # 正则匹配 return bool(word_config["pattern"].search(title_lower)) else: # 子字符串匹配 return word_config["word"].lower() in title_lower def load_frequency_words( frequency_file: Optional[str] = None, ) -> Tuple[List[Dict], List[str], List[str]]: """ 加载频率词配置 配置文件格式说明： - 每个词组由空行分隔 - [GLOBAL_FILTER] 区域定义全局过滤词 - [WORD_GROUPS] 区域定义词组（默认） 词组语法： - 普通词：直接写入，任意匹配即可 - +词：必须词，所有必须词都要匹配 - !词：过滤词，匹配则排除 - @数字：该词组最多显示的条数 Args: frequency_file: 频率词配置文件路径，默认从环境变量 FREQUENCY_WORDS_PATH 获取或使用 config/frequency_words.txt，短文件名从 config/custom/keyword/ 查找 Returns: (词组列表, 词组内过滤词, 全局过滤词) Raises: FileNotFoundError: 频率词文件不存在 """ if frequency_file is None: frequency_file = os.environ.get( "FREQUENCY_WORDS_PATH", "config/frequency_words.txt" ) frequency_path = Path(frequency_file) if not frequency_path.exists(): # 尝试作为短文件名，拼接 config/custom/keyword/ 前缀 custom_path = Path("config/custom/keyword") / frequency_file if custom_path.exists(): frequency_path = custom_path else: raise FileNotFoundError(f"频率词文件 {frequency_file} 不存在") with open(frequency_path, "r", encoding="utf-8") as f: content = f.read() word_groups = [group.strip() for group in content.split("\n\n") if group.strip()] processed_groups = [] filter_words = [] global_filters = [] # 默认区域（向后兼容） current_section = "WORD_GROUPS" for group in word_groups: # 过滤空行和注释行（# 开头） lines = [line.strip() for line in group.split("\n") if line.strip() and not line.strip().startswith("#")] if not lines: continue # 检查是否为区域标记 if lines[0].startswith("[") and lines[0].endswith("]"): section_name = lines[0][1:-1].upper() if section_name in ("GLOBAL_FILTER", "WORD_GROUPS"): current_section = section_name lines = lines[1:] # 移除标记行 # 处理全局过滤区域 if current_section == "GLOBAL_FILTER": # 直接添加所有非空行到全局过滤列表 for line in lines: # 忽略特殊语法前缀，只提取纯文本 if line.startswith(("!", "+", "@")): continue # 全局过滤区不支持特殊语法 if line: global_filters.append(line) continue # 处理词组区域 words = lines group_alias = None # 组别名（[别名] 语法） # 检查第一行是否为组别名（非区域标记） if words and words[0].startswith("[") and words[0].endswith("]"): potential_alias = words[0][1:-1].strip() # 排除区域标记（GLOBAL_FILTER, WORD_GROUPS） if potential_alias.upper() not in ("GLOBAL_FILTER", "WORD_GROUPS"): group_alias = potential_alias words = words[1:] # 移除组别名行 group_required_words = [] group_normal_words = [] group_max_count = 0 # 默认不限制 for word in words: if word.startswith("@"): # 解析最大显示数量（只接受正整数） try: count = int(word[1:]) if count > 0: group_max_count = count except (ValueError, IndexError): pass # 忽略无效的@数字格式 elif word.startswith("!"): # 过滤词（支持正则语法） filter_word = word[1:] parsed = _parse_word(filter_word) filter_words.append(parsed) elif word.startswith("+"): # 必须词（支持正则语法） req_word = word[1:] group_required_words.append(_parse_word(req_word)) else: # 普通词（支持正则语法） group_normal_words.append(_parse_word(word)) if group_required_words or group_normal_words: if group_normal_words: group_key = " ".join(w["word"] for w in group_normal_words) else: group_key = " ".join(w["word"] for w in group_required_words) # 生成显示名称 # 优先级：组别名 > 行别名拼接 > 关键词拼接 if group_alias: # 有组别名，直接使用 display_name = group_alias else: # 没有组别名，拼接每行的显示名（行别名或关键词本身） all_words = group_normal_words + group_required_words display_parts = [] for w in all_words: # 优先使用行别名，否则使用关键词本身 part = w.get("display_name") or w["word"] display_parts.append(part) # 用 " / " 拼接多个词 display_name = " / ".join(display_parts) if display_parts else None processed_groups.append( { "required": group_required_words, "normal": group_normal_words, "group_key": group_key, "display_name": display_name, # 可能为 None "max_count": group_max_count, } ) return processed_groups, filter_words, global_filters def matches_word_groups( title: str, word_groups: List[Dict], filter_words: List, global_filters: Optional[List[str]] = None ) -> bool: """ 检查标题是否匹配词组规则 Args: title: 标题文本 word_groups: 词组列表 filter_words: 过滤词列表（可以是字符串列表或字典列表） global_filters: 全局过滤词列表 Returns: 是否匹配 """ # 防御性类型检查：确保 title 是有效字符串 if not isinstance(title, str): title = str(title) if title is not None else "" if not title.strip(): return False title_lower = title.lower() # 全局过滤检查（优先级最高） if global_filters: if any(global_word.lower() in title_lower for global_word in global_filters): return False # 如果没有配置词组，则匹配所有标题（支持显示全部新闻） if not word_groups: return True # 过滤词检查（兼容新旧格式） for filter_item in filter_words: if _word_matches(filter_item, title_lower): return False # 词组匹配检查 for group in word_groups: required_words = group["required"] normal_words = group["normal"] # 必须词检查 if required_words: all_required_present = all( _word_matches(req_item, title_lower) for req_item in required_words ) if not all_required_present: continue # 普通词检查 if normal_words: any_normal_present = any( _word_matches(normal_item, title_lower) for normal_item in normal_words ) if not any_normal_present: continue return True return False ================================================ FILE: trendradar/core/loader.py ================================================ # coding=utf-8 """ 配置加载模块 负责从 YAML 配置文件和环境变量加载配置。 """ import os from pathlib import Path from typing import Dict, Any, Optional import yaml from .config import parse_multi_account_config, validate_paired_configs from trendradar.utils.time import DEFAULT_TIMEZONE def _get_env_bool(key: str) -> Optional[bool]: """从环境变量获取布尔值，如果未设置返回 None""" value = os.environ.get(key, "").strip().lower() if not value: return None return value in ("true", "1") def _get_env_int(key: str, default: int = 0) -> int: """从环境变量获取整数值""" value = os.environ.get(key, "").strip() if not value: return default try: return int(value) except ValueError: return default def _get_env_int_or_none(key: str) -> Optional[int]: """从环境变量获取整数值，未设置时返回 None""" value = os.environ.get(key, "").strip() if not value: return None try: return int(value) except ValueError: return None def _get_env_str(key: str, default: str = "") -> str: """从环境变量获取字符串值""" return os.environ.get(key, "").strip() or default def _load_app_config(config_data: Dict) -> Dict: """加载应用配置""" app_config = config_data.get("app", {}) advanced = config_data.get("advanced", {}) return { "VERSION_CHECK_URL": advanced.get("version_check_url", ""), "CONFIGS_VERSION_CHECK_URL": advanced.get("configs_version_check_url", ""), "SHOW_VERSION_UPDATE": app_config.get("show_version_update", True), "TIMEZONE": _get_env_str("TIMEZONE") or app_config.get("timezone", DEFAULT_TIMEZONE), "DEBUG": _get_env_bool("DEBUG") if _get_env_bool("DEBUG") is not None else advanced.get("debug", False), } def _load_crawler_config(config_data: Dict) -> Dict: """加载爬虫配置""" advanced = config_data.get("advanced", {}) crawler_config = advanced.get("crawler", {}) platforms_config = config_data.get("platforms", {}) return { "REQUEST_INTERVAL": crawler_config.get("request_interval", 100), "USE_PROXY": crawler_config.get("use_proxy", False), "DEFAULT_PROXY": crawler_config.get("default_proxy", ""), "ENABLE_CRAWLER": platforms_config.get("enabled", True), } def _load_report_config(config_data: Dict) -> Dict: """加载报告配置""" report_config = config_data.get("report", {}) # 环境变量覆盖 sort_by_position_env = _get_env_bool("SORT_BY_POSITION_FIRST") max_news_env = _get_env_int("MAX_NEWS_PER_KEYWORD") return { "REPORT_MODE": report_config.get("mode", "daily"), "DISPLAY_MODE": report_config.get("display_mode", "keyword"), "RANK_THRESHOLD": report_config.get("rank_threshold", 10), "SORT_BY_POSITION_FIRST": sort_by_position_env if sort_by_position_env is not None else report_config.get("sort_by_position_first", False), "MAX_NEWS_PER_KEYWORD": max_news_env or report_config.get("max_news_per_keyword", 0), } def _load_notification_config(config_data: Dict) -> Dict: """加载通知配置""" notification = config_data.get("notification", {}) advanced = config_data.get("advanced", {}) batch_size = advanced.get("batch_size", {}) return { "ENABLE_NOTIFICATION": notification.get("enabled", True), "MESSAGE_BATCH_SIZE": batch_size.get("default", 4000), "DINGTALK_BATCH_SIZE": batch_size.get("dingtalk", 20000), "FEISHU_BATCH_SIZE": batch_size.get("feishu", 29000), "BARK_BATCH_SIZE": batch_size.get("bark", 3600), "SLACK_BATCH_SIZE": batch_size.get("slack", 4000), "BATCH_SEND_INTERVAL": advanced.get("batch_send_interval", 1.0), "FEISHU_MESSAGE_SEPARATOR": advanced.get("feishu_message_separator", "---"), "MAX_ACCOUNTS_PER_CHANNEL": _get_env_int("MAX_ACCOUNTS_PER_CHANNEL") or advanced.get("max_accounts_per_channel", 3), } def _load_schedule_config(config_data: Dict) -> Dict: """ 加载统一调度配置 从 config.yaml 的 schedule 段读取，支持环境变量覆盖。 """ schedule = config_data.get("schedule", {}) # 环境变量覆盖 enabled_env = _get_env_bool("SCHEDULE_ENABLED") preset_env = _get_env_str("SCHEDULE_PRESET") enabled = enabled_env if enabled_env is not None else schedule.get("enabled", False) preset = preset_env or schedule.get("preset", "always_on") return { "enabled": enabled, "preset": preset, } def _load_timeline_data(config_dir: str = "config") -> Dict: """ 加载 timeline.yaml Args: config_dir: 配置目录路径 Returns: timeline.yaml 的完整数据，找不到时返回空模板 """ timeline_path = Path(config_dir) / "timeline.yaml" if not timeline_path.exists(): print(f"[调度] timeline.yaml 未找到: {timeline_path}，使用空模板") return { "presets": {}, "custom": { "default": { "collect": True, "analyze": False, "push": False, "report_mode": "current", "ai_mode": "follow_report", "once": {"analyze": False, "push": False}, }, "periods": {}, "day_plans": {"all_day": {"periods": []}}, "week_map": {i: "all_day" for i in range(1, 8)}, }, } with open(timeline_path, "r", encoding="utf-8") as f: data = yaml.safe_load(f) print(f"[调度] timeline.yaml 加载成功: {timeline_path}") return data or {} def _load_weight_config(config_data: Dict) -> Dict: """加载权重配置""" advanced = config_data.get("advanced", {}) weight = advanced.get("weight", {}) return { "RANK_WEIGHT": weight.get("rank", 0.6), "FREQUENCY_WEIGHT": weight.get("frequency", 0.3), "HOTNESS_WEIGHT": weight.get("hotness", 0.1), } def _load_rss_config(config_data: Dict) -> Dict: """加载 RSS 配置""" rss = config_data.get("rss", {}) advanced = config_data.get("advanced", {}) advanced_rss = advanced.get("rss", {}) advanced_crawler = advanced.get("crawler", {}) # RSS 代理配置：优先使用 RSS 专属代理，否则复用 crawler 的 default_proxy rss_proxy_url = advanced_rss.get("proxy_url", "") or advanced_crawler.get("default_proxy", "") # 新鲜度过滤配置 freshness_filter = rss.get("freshness_filter", {}) # 验证并设置 max_age_days 默认值 raw_max_age = freshness_filter.get("max_age_days", 3) try: max_age_days = int(raw_max_age) if max_age_days < 0: print(f"[警告] RSS freshness_filter.max_age_days 为负数 ({max_age_days})，使用默认值 3") max_age_days = 3 except (ValueError, TypeError): print(f"[警告] RSS freshness_filter.max_age_days 格式错误 ({raw_max_age})，使用默认值 3") max_age_days = 3 # RSS 配置直接从 config.yaml 读取，不再支持环境变量 return { "ENABLED": rss.get("enabled", False), "REQUEST_INTERVAL": advanced_rss.get("request_interval", 2000), "TIMEOUT": advanced_rss.get("timeout", 15), "USE_PROXY": advanced_rss.get("use_proxy", False), "PROXY_URL": rss_proxy_url, "FEEDS": rss.get("feeds", []), "FRESHNESS_FILTER": { "ENABLED": freshness_filter.get("enabled", True), # 默认启用 "MAX_AGE_DAYS": max_age_days, }, } def _load_display_config(config_data: Dict) -> Dict: """加载推送内容显示配置""" display = config_data.get("display", {}) regions = display.get("regions", {}) standalone = display.get("standalone", {}) # 默认区域顺序 default_region_order = ["hotlist", "rss", "new_items", "standalone", "ai_analysis"] region_order = display.get("region_order", default_region_order) # 验证 region_order 中的值是否合法 valid_regions = {"hotlist", "rss", "new_items", "standalone", "ai_analysis"} region_order = [r for r in region_order if r in valid_regions] # 如果过滤后为空，使用默认顺序 if not region_order: region_order = default_region_order return { # 区域显示顺序 "REGION_ORDER": region_order, # 区域开关 "REGIONS": { "HOTLIST": regions.get("hotlist", True), "NEW_ITEMS": regions.get("new_items", True), "RSS": regions.get("rss", True), "STANDALONE": regions.get("standalone", False), "AI_ANALYSIS": regions.get("ai_analysis", True), }, # 独立展示区配置 "STANDALONE": { "PLATFORMS": standalone.get("platforms", []), "RSS_FEEDS": standalone.get("rss_feeds", []), "MAX_ITEMS": standalone.get("max_items", 20), }, } def _load_ai_config(config_data: Dict) -> Dict: """加载 AI 模型配置（LiteLLM 格式）""" ai_config = config_data.get("ai", {}) timeout_env = _get_env_int_or_none("AI_TIMEOUT") return { # LiteLLM 核心配置 "MODEL": _get_env_str("AI_MODEL") or ai_config.get("model", ""), "API_KEY": _get_env_str("AI_API_KEY") or ai_config.get("api_key", ""), "API_BASE": _get_env_str("AI_API_BASE") or ai_config.get("api_base", ""), # 生成参数 "TIMEOUT": timeout_env if timeout_env is not None else ai_config.get("timeout", 120), "TEMPERATURE": ai_config.get("temperature", 1.0), "MAX_TOKENS": ai_config.get("max_tokens", 5000), # LiteLLM 高级选项 "NUM_RETRIES": ai_config.get("num_retries", 2), "FALLBACK_MODELS": ai_config.get("fallback_models", []), "EXTRA_PARAMS": ai_config.get("extra_params", {}), } def _load_ai_analysis_config(config_data: Dict) -> Dict: """加载 AI 分析配置（功能配置，模型配置见 _load_ai_config）""" ai_config = config_data.get("ai_analysis", {}) enabled_env = _get_env_bool("AI_ANALYSIS_ENABLED") return { "ENABLED": enabled_env if enabled_env is not None else ai_config.get("enabled", False), "LANGUAGE": ai_config.get("language", "Chinese"), "PROMPT_FILE": ai_config.get("prompt_file", "ai_analysis_prompt.txt"), "MODE": ai_config.get("mode", "follow_report"), "MAX_NEWS_FOR_ANALYSIS": ai_config.get("max_news_for_analysis", 50), "INCLUDE_RSS": ai_config.get("include_rss", True), "INCLUDE_RANK_TIMELINE": ai_config.get("include_rank_timeline", False), "INCLUDE_STANDALONE": ai_config.get("include_standalone", False), } def _load_ai_translation_config(config_data: Dict) -> Dict: """加载 AI 翻译配置（功能配置，模型配置见 _load_ai_config）""" trans_config = config_data.get("ai_translation", {}) enabled_env = _get_env_bool("AI_TRANSLATION_ENABLED") scope = trans_config.get("scope", {}) return { "ENABLED": enabled_env if enabled_env is not None else trans_config.get("enabled", False), "LANGUAGE": _get_env_str("AI_TRANSLATION_LANGUAGE") or trans_config.get("language", "English"), "PROMPT_FILE": trans_config.get("prompt_file", "ai_translation_prompt.txt"), "SCOPE": { "HOTLIST": scope.get("hotlist", True), "RSS": scope.get("rss", True), "STANDALONE": scope.get("standalone", True), }, } def _load_ai_filter_config(config_data: Dict) -> Dict: """加载 AI 智能筛选配置（由 filter.method 控制是否启用）""" ai_filter = config_data.get("ai_filter", {}) return { "BATCH_SIZE": ai_filter.get("batch_size", 200), "BATCH_INTERVAL": ai_filter.get("batch_interval", 5), "INTERESTS_FILE": ai_filter.get("interests_file"), # None = 使用默认 config/ai_interests.txt "PROMPT_FILE": ai_filter.get("prompt_file", "prompt.txt"), "EXTRACT_PROMPT_FILE": ai_filter.get("extract_prompt_file", "extract_prompt.txt"), "UPDATE_TAGS_PROMPT_FILE": ai_filter.get("update_tags_prompt_file", "update_tags_prompt.txt"), "RECLASSIFY_THRESHOLD": ai_filter.get("reclassify_threshold", 0.6), "MIN_SCORE": float(ai_filter.get("min_score", 0)), } def _load_filter_config(config_data: Dict) -> Dict: """加载筛选策略配置""" filter_cfg = config_data.get("filter", {}) # 环境变量兼容：AI_FILTER_ENABLED=true → method=ai env_ai_filter = _get_env_bool("AI_FILTER_ENABLED") method = filter_cfg.get("method", "keyword") if env_ai_filter is True: method = "ai" # 兼容旧配置：如果 ai_filter.enabled=true 且未显式设置 filter.method if method == "keyword" and not filter_cfg.get("method"): ai_filter = config_data.get("ai_filter", {}) if ai_filter.get("enabled", False): method = "ai" return { "METHOD": method, # "keyword" | "ai" "PRIORITY_SORT_ENABLED": filter_cfg.get("priority_sort_enabled", False), # AI 模式标签优先级排序开关 } def _load_storage_config(config_data: Dict) -> Dict: """加载存储配置""" storage = config_data.get("storage", {}) formats = storage.get("formats", {}) local = storage.get("local", {}) remote = storage.get("remote", {}) pull = storage.get("pull", {}) txt_enabled_env = _get_env_bool("STORAGE_TXT_ENABLED") html_enabled_env = _get_env_bool("STORAGE_HTML_ENABLED") pull_enabled_env = _get_env_bool("PULL_ENABLED") return { "BACKEND": _get_env_str("STORAGE_BACKEND") or storage.get("backend", "auto"), "FORMATS": { "SQLITE": formats.get("sqlite", True), "TXT": txt_enabled_env if txt_enabled_env is not None else formats.get("txt", True), "HTML": html_enabled_env if html_enabled_env is not None else formats.get("html", True), }, "LOCAL": { "DATA_DIR": local.get("data_dir", "output"), "RETENTION_DAYS": _get_env_int("LOCAL_RETENTION_DAYS") or local.get("retention_days", 0), }, "REMOTE": { "ENDPOINT_URL": _get_env_str("S3_ENDPOINT_URL") or remote.get("endpoint_url", ""), "BUCKET_NAME": _get_env_str("S3_BUCKET_NAME") or remote.get("bucket_name", ""), "ACCESS_KEY_ID": _get_env_str("S3_ACCESS_KEY_ID") or remote.get("access_key_id", ""), "SECRET_ACCESS_KEY": _get_env_str("S3_SECRET_ACCESS_KEY") or remote.get("secret_access_key", ""), "REGION": _get_env_str("S3_REGION") or remote.get("region", ""), "RETENTION_DAYS": _get_env_int("REMOTE_RETENTION_DAYS") or remote.get("retention_days", 0), }, "PULL": { "ENABLED": pull_enabled_env if pull_enabled_env is not None else pull.get("enabled", False), "DAYS": _get_env_int("PULL_DAYS") or pull.get("days", 7), }, } def _load_webhook_config(config_data: Dict) -> Dict: """加载 Webhook 配置""" notification = config_data.get("notification", {}) channels = notification.get("channels", {}) # 各渠道配置 feishu = channels.get("feishu", {}) dingtalk = channels.get("dingtalk", {}) wework = channels.get("wework", {}) telegram = channels.get("telegram", {}) email = channels.get("email", {}) ntfy = channels.get("ntfy", {}) bark = channels.get("bark", {}) slack = channels.get("slack", {}) generic = channels.get("generic_webhook", {}) return { # 飞书 "FEISHU_WEBHOOK_URL": _get_env_str("FEISHU_WEBHOOK_URL") or feishu.get("webhook_url", ""), # 钉钉 "DINGTALK_WEBHOOK_URL": _get_env_str("DINGTALK_WEBHOOK_URL") or dingtalk.get("webhook_url", ""), # 企业微信 "WEWORK_WEBHOOK_URL": _get_env_str("WEWORK_WEBHOOK_URL") or wework.get("webhook_url", ""), "WEWORK_MSG_TYPE": _get_env_str("WEWORK_MSG_TYPE") or wework.get("msg_type", "markdown"), # Telegram "TELEGRAM_BOT_TOKEN": _get_env_str("TELEGRAM_BOT_TOKEN") or telegram.get("bot_token", ""), "TELEGRAM_CHAT_ID": _get_env_str("TELEGRAM_CHAT_ID") or telegram.get("chat_id", ""), # 邮件 "EMAIL_FROM": _get_env_str("EMAIL_FROM") or email.get("from", ""), "EMAIL_PASSWORD": _get_env_str("EMAIL_PASSWORD") or email.get("password", ""), "EMAIL_TO": _get_env_str("EMAIL_TO") or email.get("to", ""), "EMAIL_SMTP_SERVER": _get_env_str("EMAIL_SMTP_SERVER") or email.get("smtp_server", ""), "EMAIL_SMTP_PORT": _get_env_str("EMAIL_SMTP_PORT") or email.get("smtp_port", ""), # ntfy "NTFY_SERVER_URL": _get_env_str("NTFY_SERVER_URL") or ntfy.get("server_url") or "https://ntfy.sh", "NTFY_TOPIC": _get_env_str("NTFY_TOPIC") or ntfy.get("topic", ""), "NTFY_TOKEN": _get_env_str("NTFY_TOKEN") or ntfy.get("token", ""), # Bark "BARK_URL": _get_env_str("BARK_URL") or bark.get("url", ""), # Slack "SLACK_WEBHOOK_URL": _get_env_str("SLACK_WEBHOOK_URL") or slack.get("webhook_url", ""), # 通用 Webhook "GENERIC_WEBHOOK_URL": _get_env_str("GENERIC_WEBHOOK_URL") or generic.get("webhook_url", ""), "GENERIC_WEBHOOK_TEMPLATE": _get_env_str("GENERIC_WEBHOOK_TEMPLATE") or generic.get("payload_template", ""), } def _print_notification_sources(config: Dict) -> None: """打印通知渠道配置来源信息""" notification_sources = [] max_accounts = config["MAX_ACCOUNTS_PER_CHANNEL"] if config["FEISHU_WEBHOOK_URL"]: accounts = parse_multi_account_config(config["FEISHU_WEBHOOK_URL"]) count = min(len(accounts), max_accounts) source = "环境变量" if os.environ.get("FEISHU_WEBHOOK_URL") else "配置文件" notification_sources.append(f"飞书({source}, {count}个账号)") if config["DINGTALK_WEBHOOK_URL"]: accounts = parse_multi_account_config(config["DINGTALK_WEBHOOK_URL"]) count = min(len(accounts), max_accounts) source = "环境变量" if os.environ.get("DINGTALK_WEBHOOK_URL") else "配置文件" notification_sources.append(f"钉钉({source}, {count}个账号)") if config["WEWORK_WEBHOOK_URL"]: accounts = parse_multi_account_config(config["WEWORK_WEBHOOK_URL"]) count = min(len(accounts), max_accounts) source = "环境变量" if os.environ.get("WEWORK_WEBHOOK_URL") else "配置文件" notification_sources.append(f"企业微信({source}, {count}个账号)") if config["TELEGRAM_BOT_TOKEN"] and config["TELEGRAM_CHAT_ID"]: tokens = parse_multi_account_config(config["TELEGRAM_BOT_TOKEN"]) chat_ids = parse_multi_account_config(config["TELEGRAM_CHAT_ID"]) valid, count = validate_paired_configs( {"bot_token": tokens, "chat_id": chat_ids}, "Telegram", required_keys=["bot_token", "chat_id"] ) if valid and count > 0: count = min(count, max_accounts) token_source = "环境变量" if os.environ.get("TELEGRAM_BOT_TOKEN") else "配置文件" notification_sources.append(f"Telegram({token_source}, {count}个账号)") if config["EMAIL_FROM"] and config["EMAIL_PASSWORD"] and config["EMAIL_TO"]: from_source = "环境变量" if os.environ.get("EMAIL_FROM") else "配置文件" notification_sources.append(f"邮件({from_source})") if config["NTFY_SERVER_URL"] and config["NTFY_TOPIC"]: topics = parse_multi_account_config(config["NTFY_TOPIC"]) tokens = parse_multi_account_config(config["NTFY_TOKEN"]) if tokens: valid, count = validate_paired_configs( {"topic": topics, "token": tokens}, "ntfy" ) if valid and count > 0: count = min(count, max_accounts) server_source = "环境变量" if os.environ.get("NTFY_SERVER_URL") else "配置文件" notification_sources.append(f"ntfy({server_source}, {count}个账号)") else: count = min(len(topics), max_accounts) server_source = "环境变量" if os.environ.get("NTFY_SERVER_URL") else "配置文件" notification_sources.append(f"ntfy({server_source}, {count}个账号)") if config["BARK_URL"]: accounts = parse_multi_account_config(config["BARK_URL"]) count = min(len(accounts), max_accounts) bark_source = "环境变量" if os.environ.get("BARK_URL") else "配置文件" notification_sources.append(f"Bark({bark_source}, {count}个账号)") if config["SLACK_WEBHOOK_URL"]: accounts = parse_multi_account_config(config["SLACK_WEBHOOK_URL"]) count = min(len(accounts), max_accounts) slack_source = "环境变量" if os.environ.get("SLACK_WEBHOOK_URL") else "配置文件" notification_sources.append(f"Slack({slack_source}, {count}个账号)") if config.get("GENERIC_WEBHOOK_URL"): accounts = parse_multi_account_config(config["GENERIC_WEBHOOK_URL"]) count = min(len(accounts), max_accounts) source = "环境变量" if os.environ.get("GENERIC_WEBHOOK_URL") else "配置文件" notification_sources.append(f"通用Webhook({source}, {count}个账号)") if notification_sources: print(f"通知渠道配置来源: {', '.join(notification_sources)}") print(f"每个渠道最大账号数: {max_accounts}") else: print("未配置任何通知渠道") def load_config(config_path: Optional[str] = None) -> Dict[str, Any]: """ 加载配置文件 Args: config_path: 配置文件路径，默认从环境变量 CONFIG_PATH 获取或使用 config/config.yaml Returns: 包含所有配置的字典 Raises: FileNotFoundError: 配置文件不存在 """ if config_path is None: config_path = os.environ.get("CONFIG_PATH", "config/config.yaml") if not Path(config_path).exists(): raise FileNotFoundError(f"配置文件 {config_path} 不存在") with open(config_path, "r", encoding="utf-8") as f: config_data = yaml.safe_load(f) print(f"配置文件加载成功: {config_path}") # 合并所有配置 config = {} # 应用配置 config.update(_load_app_config(config_data)) # 爬虫配置 config.update(_load_crawler_config(config_data)) # 报告配置 config.update(_load_report_config(config_data)) # 通知配置 config.update(_load_notification_config(config_data)) # 统一调度配置 config["SCHEDULE"] = _load_schedule_config(config_data) config["_TIMELINE_DATA"] = _load_timeline_data( str(Path(config_path).parent) if config_path else "config" ) # 权重配置 config["WEIGHT_CONFIG"] = _load_weight_config(config_data) # 平台配置 platforms_config = config_data.get("platforms", {}) config["PLATFORMS"] = platforms_config.get("sources", []) # RSS 配置 config["RSS"] = _load_rss_config(config_data) # AI 模型共享配置 config["AI"] = _load_ai_config(config_data) # AI 分析配置 config["AI_ANALYSIS"] = _load_ai_analysis_config(config_data) # AI 翻译配置 config["AI_TRANSLATION"] = _load_ai_translation_config(config_data) # AI 智能筛选配置 config["AI_FILTER"] = _load_ai_filter_config(config_data) # 筛选策略配置 config["FILTER"] = _load_filter_config(config_data) # 推送内容显示配置 config["DISPLAY"] = _load_display_config(config_data) # 存储配置 config["STORAGE"] = _load_storage_config(config_data) # Webhook 配置 config.update(_load_webhook_config(config_data)) # 打印通知渠道配置来源 _print_notification_sources(config) return config ================================================ FILE: trendradar/core/scheduler.py ================================================ # coding=utf-8 """ 时间线调度器 统一的时间线调度系统，替代分散的 push_window / analysis_window 逻辑。 基于 periods + day_plans + week_map 模型实现灵活的时间段调度。 """ import copy import re from dataclasses import dataclass from typing import Any, Callable, Dict, List, Optional from datetime import datetime @dataclass class ResolvedSchedule: """当前时间解析后的调度结果""" period_key: Optional[str] # 命中的 period key，None=默认配置 period_name: Optional[str] # 命中的展示名称 day_plan: str # 当前日计划 collect: bool analyze: bool push: bool report_mode: str ai_mode: str once_analyze: bool once_push: bool frequency_file: Optional[str] = None # 频率词文件路径，None=使用默认 filter_method: Optional[str] = None # 筛选策略: "keyword"|"ai"，None=使用全局配置 interests_file: Optional[str] = None # AI 筛选兴趣文件，None=使用默认 class Scheduler: """ 时间线调度器 根据 timeline 配置（periods + day_plans + week_map）解析当前时间应执行的行为。 支持： - 预设模板 + 自定义模式 - 跨日时间段（如 22:00-07:00） - 每天 / 每周差异化配置 - once 执行去重（analyze / push 独立维度） - 冲突策略（error_on_overlap / last_wins） """ def __init__( self, schedule_config: Dict[str, Any], timeline_data: Dict[str, Any], storage_backend: Any, get_time_func: Callable[[], datetime], fallback_report_mode: str = "current", ): """ 初始化调度器 Args: schedule_config: config.yaml 中的 schedule 段（含 preset 等） timeline_data: timeline.yaml 的完整数据 storage_backend: 存储后端（用于 once 去重记录） get_time_func: 获取当前时间的函数（应使用配置的时区） fallback_report_mode: 调度未启用时回退使用的 report_mode（来自 config.yaml 的 report.mode） """ self.schedule_config = schedule_config self.storage = storage_backend self.get_time = get_time_func self.enabled = schedule_config.get("enabled", True) self.fallback_report_mode = fallback_report_mode # 加载并构建最终 timeline self.timeline = self._build_timeline(schedule_config, timeline_data) if self.enabled: self._validate_timeline(self.timeline) def _build_timeline( self, schedule_config: Dict[str, Any], timeline_data: Dict[str, Any], ) -> Dict[str, Any]: """从 preset 或 custom 构建 timeline""" preset = schedule_config.get("preset", "always_on") if preset == "custom": timeline = copy.deepcopy(timeline_data.get("custom", {})) else: presets = timeline_data.get("presets", {}) if preset not in presets: raise ValueError( f"未知的预设模板: '{preset}'，可选值: " f"{', '.join(presets.keys())}, custom" ) timeline = copy.deepcopy(presets[preset]) # 确保 periods 是 dict（可能为空 {}） if timeline.get("periods") is None: timeline["periods"] = {} return timeline def resolve(self) -> ResolvedSchedule: """ 解析当前时间对应的调度配置 Returns: ResolvedSchedule 包含当前应执行的行为 """ if not self.enabled: # 调度未启用时返回默认的全功能配置，report_mode 回退使用 config.yaml 的 report.mode return ResolvedSchedule( period_key=None, period_name=None, day_plan="disabled", collect=True, analyze=True, push=True, report_mode=self.fallback_report_mode, ai_mode="follow_report", once_analyze=False, once_push=False, ) now = self.get_time() weekday = now.isoweekday() # 1=周一 ... 7=周日 now_hhmm = now.strftime("%H:%M") # 查找当天的日计划 day_plan_key = self.timeline["week_map"].get(weekday) if day_plan_key is None: raise ValueError(f"week_map 缺少星期映射: {weekday}") day_plan = self.timeline["day_plans"].get(day_plan_key) if day_plan is None: raise ValueError(f"week_map[{weekday}] 引用了不存在的 day_plan: {day_plan_key}") # 查找当前活跃的时间段 period_key = self._find_active_period(now_hhmm, day_plan) # 合并默认配置和时间段配置 merged = self._merge_with_default(period_key) # 打印调度日志 weekday_names = {1: "一", 2: "二", 3: "三", 4: "四", 5: "五", 6: "六", 7: "日"} period_display = "默认配置（未命中任何时间段）" if period_key: period_cfg = self.timeline["periods"][period_key] period_name = period_cfg.get("name", period_key) start = period_cfg.get("start", "?") end = period_cfg.get("end", "?") period_display = f"{period_name} ({start}-{end})" print(f"[调度] 星期{weekday_names.get(weekday, '?')}，日计划: {day_plan_key}") print(f"[调度] 当前时间段: {period_display}") resolved = ResolvedSchedule( period_key=period_key, period_name=( self.timeline["periods"][period_key].get("name") if period_key else None ), day_plan=day_plan_key, collect=merged.get("collect", True), analyze=merged.get("analyze", False), push=merged.get("push", False), report_mode=merged.get("report_mode", "current"), ai_mode=self._resolve_ai_mode(merged), once_analyze=merged.get("once", {}).get("analyze", False), once_push=merged.get("once", {}).get("push", False), frequency_file=merged.get("frequency_file"), filter_method=merged.get("filter_method"), interests_file=merged.get("interests_file"), ) # 打印行为摘要 actions = [] if resolved.collect: actions.append("采集") if resolved.analyze: actions.append(f"分析(AI:{resolved.ai_mode})") if resolved.push: actions.append(f"推送(模式:{resolved.report_mode})") print(f"[调度] 行为: {', '.join(actions) if actions else '无'}") if resolved.frequency_file: print(f"[调度] 频率词文件: {resolved.frequency_file}") return resolved def _find_active_period( self, now_hhmm: str, day_plan: Dict[str, Any] ) -> Optional[str]: """ 查找当前时间命中的活跃时间段 Args: now_hhmm: 当前时间 HH:MM day_plan: 日计划配置 Returns: 命中的 period key，或 None """ candidates = [] for idx, key in enumerate(day_plan.get("periods", [])): period = self.timeline["periods"].get(key) if period is None: continue if self._in_range(now_hhmm, period["start"], period["end"]): candidates.append((idx, key)) if not candidates: return None # 检查冲突 if len(candidates) > 1: policy = self.timeline.get("overlap", {}).get("policy", "error_on_overlap") conflicting = [c[1] for c in candidates] if policy == "error_on_overlap": raise ValueError( f"检测到时间段重叠冲突: {', '.join(conflicting)} 在 {now_hhmm} 重叠。" f"请调整时间段配置，或将 overlap.policy 设为 'last_wins'" ) # last_wins：输出重叠警告，列表中后面的优先 print( f"[调度] 检测到时间段重叠: {', '.join(conflicting)} 在 {now_hhmm} 重叠" ) winner = candidates[-1] print(f"[调度] 冲突策略: last_wins，生效时间段: {winner[1]}") return winner[1] return candidates[0][1] @staticmethod def _in_range(now_hhmm: str, start: str, end: str) -> bool: """ 检查时间是否在范围内（支持跨日） Args: now_hhmm: 当前时间 HH:MM start: 开始时间 HH:MM end: 结束时间 HH:MM Returns: 是否在范围内 """ if start <= end: # 正常范围，如 08:00-09:00 return start <= now_hhmm <= end else: # 跨日范围，如 22:00-07:00 return now_hhmm >= start or now_hhmm <= end def _merge_with_default(self, period_key: Optional[str]) -> Dict[str, Any]: """合并默认配置和时间段配置""" base = copy.deepcopy(self.timeline.get("default", {})) if not period_key: return base period = copy.deepcopy(self.timeline["periods"][period_key]) # 先合并 once 子对象 merged_once = dict(base.get("once", {})) merged_once.update(period.get("once", {})) # 标量字段覆盖 base.update(period) # 恢复合并后的 once if merged_once: base["once"] = merged_once return base @staticmethod def _resolve_ai_mode(cfg: Dict[str, Any]) -> str: """解析最终的 AI 模式""" ai_mode = cfg.get("ai_mode", "follow_report") if ai_mode == "follow_report": return cfg.get("report_mode", "current") return ai_mode def already_executed(self, period_key: str, action: str, date_str: str) -> bool: """ 检查指定时间段的某个 action 今天是否已执行 Args: period_key: 时间段 key action: 动作类型 (analyze / push) date_str: 日期 YYYY-MM-DD Returns: 是否已执行 """ return self.storage.has_period_executed(date_str, period_key, action) def record_execution(self, period_key: str, action: str, date_str: str) -> None: """ 记录时间段的 action 执行 Args: period_key: 时间段 key action: 动作类型 (analyze / push) date_str: 日期 YYYY-MM-DD """ self.storage.record_period_execution(date_str, period_key, action) # ======================================== # 校验 # ======================================== def _validate_timeline(self, timeline: Dict[str, Any]) -> None: """ 启动时校验 timeline 配置 Raises: ValueError: 配置不合法时抛出 """ required_top_keys = ["default", "periods", "day_plans", "week_map"] for key in required_top_keys: if key not in timeline: raise ValueError(f"timeline 缺少必须字段: {key}") # week_map 必须覆盖 1..7 for day in range(1, 8): if day not in timeline["week_map"]: raise ValueError(f"week_map 缺少星期映射: {day}") # day_plan 引用完整性 for day, plan_key in timeline["week_map"].items(): if plan_key not in timeline["day_plans"]: raise ValueError( f"week_map[{day}] 引用了不存在的 day_plan: {plan_key}" ) # period 引用完整性 for plan_key, plan in timeline["day_plans"].items(): for period_key in plan.get("periods", []): if period_key not in timeline["periods"]: raise ValueError( f"day_plan[{plan_key}] 引用了不存在的 period: {period_key}" ) # 时间格式校验 for period_key, period in timeline["periods"].items(): if "start" not in period or "end" not in period: raise ValueError( f"period '{period_key}' 缺少 start 或 end 字段" ) self._validate_hhmm(period["start"], f"{period_key}.start") self._validate_hhmm(period["end"], f"{period_key}.end") if period["start"] == period["end"]: raise ValueError( f"period '{period_key}' 的 start 与 end 不能相同: {period['start']}" ) # 检查冲突策略下的重叠 policy = timeline.get("overlap", {}).get("policy", "error_on_overlap") if policy == "error_on_overlap": self._check_period_overlaps(timeline) def _check_period_overlaps(self, timeline: Dict[str, Any]) -> None: """ 检查每个日计划中的时间段是否存在重叠 仅在 overlap.policy == "error_on_overlap" 时调用 """ periods = timeline.get("periods", {}) for plan_key, plan in timeline["day_plans"].items(): period_keys = plan.get("periods", []) if len(period_keys) <= 1: continue # 收集每个时间段的范围 ranges = [] for pk in period_keys: p = periods.get(pk, {}) if "start" in p and "end" in p: ranges.append((pk, p["start"], p["end"])) # 两两检查重叠 for i in range(len(ranges)): for j in range(i + 1, len(ranges)): if self._ranges_overlap( ranges[i][1], ranges[i][2], ranges[j][1], ranges[j][2], ): raise ValueError( f"day_plan '{plan_key}' 中时间段 '{ranges[i][0]}' " f"({ranges[i][1]}-{ranges[i][2]}) 与 '{ranges[j][0]}' " f"({ranges[j][1]}-{ranges[j][2]}) 存在重叠。" f"请调整时间段，或将 overlap.policy 设为 'last_wins'" ) @staticmethod def _ranges_overlap(s1: str, e1: str, s2: str, e2: str) -> bool: """检查两个时间范围是否重叠（支持跨日）""" def to_minutes(t: str) -> int: h, m = t.split(":") return int(h) * 60 + int(m) def expand_range(start: str, end: str) -> List[tuple]: """将时间范围展开为分钟段列表，跨日时拆分为两段""" s = to_minutes(start) e = to_minutes(end) if s <= e: return [(s, e)] else: # 跨日：拆分为 [start, 23:59] 和 [00:00, end] return [(s, 24 * 60 - 1), (0, e)] segs1 = expand_range(s1, e1) segs2 = expand_range(s2, e2) for a_start, a_end in segs1: for b_start, b_end in segs2: # 两个区间有重叠的条件 if a_start <= b_end and b_start <= a_end: return True return False @staticmethod def _validate_hhmm(value: str, field_name: str) -> None: """校验 HH:MM 格式""" if not re.match(r"^\d{2}:\d{2}$", value): raise ValueError(f"{field_name} 格式错误: '{value}'，期望 HH:MM") h, m = value.split(":") if not (0 <= int(h) <= 23 and 0 <= int(m) <= 59): raise ValueError(f"{field_name} 时间值超出范围: '{value}'") ================================================ FILE: trendradar/crawler/__init__.py ================================================ # coding=utf-8 """ 爬虫模块 - 数据抓取功能 """ from trendradar.crawler.fetcher import DataFetcher __all__ = ["DataFetcher"] ================================================ FILE: trendradar/crawler/fetcher.py ================================================ # coding=utf-8 """ 数据获取器模块 负责从 NewsNow API 抓取新闻数据，支持： - 单个平台数据获取 - 批量平台数据爬取 - 自动重试机制 - 代理支持 """ import json import random import time from typing import Dict, List, Tuple, Optional, Union import requests class DataFetcher: """数据获取器""" # 默认 API 地址 DEFAULT_API_URL = "https://newsnow.busiyi.world/api/s" # 默认请求头 DEFAULT_HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36", "Accept": "application/json, text/plain, */*", "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", "Connection": "keep-alive", "Cache-Control": "no-cache", } def __init__( self, proxy_url: Optional[str] = None, api_url: Optional[str] = None, ): """ 初始化数据获取器 Args: proxy_url: 代理服务器 URL（可选） api_url: API 基础 URL（可选，默认使用 DEFAULT_API_URL） """ self.proxy_url = proxy_url self.api_url = api_url or self.DEFAULT_API_URL def fetch_data( self, id_info: Union[str, Tuple[str, str]], max_retries: int = 2, min_retry_wait: int = 3, max_retry_wait: int = 5, ) -> Tuple[Optional[str], str, str]: """ 获取指定ID数据，支持重试 Args: id_info: 平台ID 或 (平台ID, 别名) 元组 max_retries: 最大重试次数 min_retry_wait: 最小重试等待时间（秒） max_retry_wait: 最大重试等待时间（秒） Returns: (响应文本, 平台ID, 别名) 元组，失败时响应文本为 None """ if isinstance(id_info, tuple): id_value, alias = id_info else: id_value = id_info alias = id_value url = f"{self.api_url}?id={id_value}&latest" proxies = None if self.proxy_url: proxies = {"http": self.proxy_url, "https": self.proxy_url} retries = 0 while retries <= max_retries: try: response = requests.get( url, proxies=proxies, headers=self.DEFAULT_HEADERS, timeout=10, ) response.raise_for_status() data_text = response.text data_json = json.loads(data_text) status = data_json.get("status", "未知") if status not in ["success", "cache"]: raise ValueError(f"响应状态异常: {status}") status_info = "最新数据" if status == "success" else "缓存数据" print(f"获取 {id_value} 成功（{status_info}）") return data_text, id_value, alias except Exception as e: retries += 1 if retries <= max_retries: base_wait = random.uniform(min_retry_wait, max_retry_wait) additional_wait = (retries - 1) * random.uniform(1, 2) wait_time = base_wait + additional_wait print(f"请求 {id_value} 失败: {e}. {wait_time:.2f}秒后重试...") time.sleep(wait_time) else: print(f"请求 {id_value} 失败: {e}") return None, id_value, alias return None, id_value, alias def crawl_websites( self, ids_list: List[Union[str, Tuple[str, str]]], request_interval: int = 100, ) -> Tuple[Dict, Dict, List]: """ 爬取多个网站数据 Args: ids_list: 平台ID列表，每个元素可以是字符串或 (平台ID, 别名) 元组 request_interval: 请求间隔（毫秒） Returns: (结果字典, ID到名称的映射, 失败ID列表) 元组 """ results = {} id_to_name = {} failed_ids = [] for i, id_info in enumerate(ids_list): if isinstance(id_info, tuple): id_value, name = id_info else: id_value = id_info name = id_value id_to_name[id_value] = name response, _, _ = self.fetch_data(id_info) if response: try: data = json.loads(response) results[id_value] = {} for index, item in enumerate(data.get("items", []), 1): title = item.get("title") # 跳过无效标题（None、float、空字符串） if title is None or isinstance(title, float) or not str(title).strip(): continue title = str(title).strip() url = item.get("url", "") mobile_url = item.get("mobileUrl", "") if title in results[id_value]: results[id_value][title]["ranks"].append(index) else: results[id_value][title] = { "ranks": [index], "url": url, "mobileUrl": mobile_url, } except json.JSONDecodeError: print(f"解析 {id_value} 响应失败") failed_ids.append(id_value) except Exception as e: print(f"处理 {id_value} 数据出错: {e}") failed_ids.append(id_value) else: failed_ids.append(id_value) # 请求间隔（除了最后一个） if i < len(ids_list) - 1: actual_interval = request_interval + random.randint(-10, 20) actual_interval = max(50, actual_interval) time.sleep(actual_interval / 1000) print(f"成功: {list(results.keys())}, 失败: {failed_ids}") return results, id_to_name, failed_ids ================================================ FILE: trendradar/crawler/rss/__init__.py ================================================ # coding=utf-8 """ RSS 抓取模块 提供 RSS 2.0、Atom 和 JSON Feed 1.1 订阅源的解析和抓取功能 """ from .parser import RSSParser from .fetcher import RSSFetcher, RSSFeedConfig __all__ = ["RSSParser", "RSSFetcher", "RSSFeedConfig"] ================================================ FILE: trendradar/crawler/rss/fetcher.py ================================================ # coding=utf-8 """ RSS 抓取器 负责从配置的 RSS 源抓取数据并转换为标准格式 """ import time import random from dataclasses import dataclass from typing import List, Dict, Optional, Tuple import requests from .parser import RSSParser from trendradar.storage.base import RSSItem, RSSData from trendradar.utils.time import get_configured_time, is_within_days, DEFAULT_TIMEZONE @dataclass class RSSFeedConfig: """RSS 源配置""" id: str # 源 ID name: str # 显示名称 url: str # RSS URL max_items: int = 0 # 最大条目数（0=不限制） enabled: bool = True # 是否启用 max_age_days: Optional[int] = None # 文章最大年龄（天），覆盖全局设置；None=使用全局，0=禁用过滤 class RSSFetcher: """RSS 抓取器""" def __init__( self, feeds: List[RSSFeedConfig], request_interval: int = 2000, timeout: int = 15, use_proxy: bool = False, proxy_url: str = "", timezone: str = DEFAULT_TIMEZONE, freshness_enabled: bool = True, default_max_age_days: int = 3, ): """ 初始化抓取器 Args: feeds: RSS 源配置列表 request_interval: 请求间隔（毫秒） timeout: 请求超时（秒） use_proxy: 是否使用代理 proxy_url: 代理 URL timezone: 时区配置（如 'Asia/Shanghai'） freshness_enabled: 是否启用新鲜度过滤 default_max_age_days: 默认最大文章年龄（天） """ self.feeds = [f for f in feeds if f.enabled] self.request_interval = request_interval self.timeout = timeout self.use_proxy = use_proxy self.proxy_url = proxy_url self.timezone = timezone self.freshness_enabled = freshness_enabled self.default_max_age_days = default_max_age_days self.parser = RSSParser() self.session = self._create_session() def _create_session(self) -> requests.Session: """创建请求会话""" session = requests.Session() session.headers.update({ "User-Agent": "TrendRadar/2.0 RSS Reader (https://github.com/trendradar)", "Accept": "application/feed+json, application/json, application/rss+xml, application/atom+xml, application/xml, text/xml, */*", "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", }) if self.use_proxy and self.proxy_url: session.proxies = { "http": self.proxy_url, "https": self.proxy_url, } return session def _filter_by_freshness( self, items: List[RSSItem], feed: RSSFeedConfig, ) -> Tuple[List[RSSItem], int]: """ 根据新鲜度过滤文章 Args: items: 待过滤的文章列表 feed: RSS 源配置 Returns: (过滤后的文章列表, 被过滤的文章数) """ # 如果全局禁用，直接返回 if not self.freshness_enabled: return items, 0 # 确定此 feed 的 max_age_days max_days = feed.max_age_days if max_days is None: max_days = self.default_max_age_days # 如果设为 0，禁用此 feed 的过滤 if max_days == 0: return items, 0 # 过滤逻辑：无发布时间的文章保留 filtered = [] for item in items: if not item.published_at: # 无发布时间，保留 filtered.append(item) elif is_within_days(item.published_at, max_days, self.timezone): # 在指定天数内，保留 filtered.append(item) # 否则过滤掉 filtered_count = len(items) - len(filtered) return filtered, filtered_count def fetch_feed(self, feed: RSSFeedConfig) -> Tuple[List[RSSItem], Optional[str]]: """ 抓取单个 RSS 源 Args: feed: RSS 源配置 Returns: (条目列表, 错误信息) 元组 """ try: response = self.session.get(feed.url, timeout=self.timeout) response.raise_for_status() parsed_items = self.parser.parse(response.text, feed.url) # 限制条目数量（0=不限制） if feed.max_items > 0: parsed_items = parsed_items[:feed.max_items] # 转换为 RSSItem（使用配置的时区） now = get_configured_time(self.timezone) crawl_time = now.strftime("%H:%M") items = [] for parsed in parsed_items: item = RSSItem( title=parsed.title, feed_id=feed.id, feed_name=feed.name, url=parsed.url, published_at=parsed.published_at or "", summary=parsed.summary or "", author=parsed.author or "", crawl_time=crawl_time, first_time=crawl_time, last_time=crawl_time, count=1, ) items.append(item) # 注意：新鲜度过滤已移至推送阶段（_convert_rss_items_to_list） # 这样所有文章都会存入数据库，但旧文章不会推送 print(f"[RSS] {feed.name}: 获取 {len(items)} 条") return items, None except requests.Timeout: error = f"请求超时 ({self.timeout}s)" print(f"[RSS] {feed.name}: {error}") return [], error except requests.RequestException as e: error = f"请求失败: {e}" print(f"[RSS] {feed.name}: {error}") return [], error except ValueError as e: error = f"解析失败: {e}" print(f"[RSS] {feed.name}: {error}") return [], error except Exception as e: error = f"未知错误: {e}" print(f"[RSS] {feed.name}: {error}") return [], error def fetch_all(self) -> RSSData: """ 抓取所有 RSS 源 Returns: RSSData 对象 """ all_items: Dict[str, List[RSSItem]] = {} id_to_name: Dict[str, str] = {} failed_ids: List[str] = [] # 使用配置的时区 now = get_configured_time(self.timezone) crawl_time = now.strftime("%H:%M") crawl_date = now.strftime("%Y-%m-%d") print(f"[RSS] 开始抓取 {len(self.feeds)} 个 RSS 源...") for i, feed in enumerate(self.feeds): # 请求间隔（带随机波动） if i > 0: interval = self.request_interval / 1000 jitter = random.uniform(-0.2, 0.2) * interval time.sleep(interval + jitter) items, error = self.fetch_feed(feed) id_to_name[feed.id] = feed.name if error: failed_ids.append(feed.id) else: all_items[feed.id] = items total_items = sum(len(items) for items in all_items.values()) print(f"[RSS] 抓取完成: {len(all_items)} 个源成功, {len(failed_ids)} 个失败, 共 {total_items} 条") return RSSData( date=crawl_date, crawl_time=crawl_time, items=all_items, id_to_name=id_to_name, failed_ids=failed_ids, ) @classmethod def from_config(cls, config: Dict) -> "RSSFetcher": """ 从配置字典创建抓取器 Args: config: 配置字典，格式如下： { "enabled": true, "request_interval": 2000, "freshness_filter": { "enabled": true, "max_age_days": 3 }, "feeds": [ {"id": "hacker-news", "name": "Hacker News", "url": "...", "max_age_days": 1} ] } Returns: RSSFetcher 实例 """ # 读取新鲜度过滤配置 freshness_config = config.get("freshness_filter", {}) freshness_enabled = freshness_config.get("enabled", True) # 默认启用 default_max_age_days = freshness_config.get("max_age_days", 3) # 默认3天 feeds = [] for feed_config in config.get("feeds", []): # 读取并验证单个 feed 的 max_age_days（可选） max_age_days_raw = feed_config.get("max_age_days") max_age_days = None if max_age_days_raw is not None: try: max_age_days = int(max_age_days_raw) if max_age_days < 0: feed_id = feed_config.get("id", "unknown") print(f"[警告] RSS feed '{feed_id}' 的 max_age_days 为负数，将使用全局默认值") max_age_days = None except (ValueError, TypeError): feed_id = feed_config.get("id", "unknown") print(f"[警告] RSS feed '{feed_id}' 的 max_age_days 格式错误：{max_age_days_raw}") max_age_days = None feed = RSSFeedConfig( id=feed_config.get("id", ""), name=feed_config.get("name", ""), url=feed_config.get("url", ""), max_items=feed_config.get("max_items", 0), # 0=不限制 enabled=feed_config.get("enabled", True), max_age_days=max_age_days, # None=使用全局，0=禁用，>0=覆盖 ) if feed.id and feed.url: feeds.append(feed) return cls( feeds=feeds, request_interval=config.get("request_interval", 2000), timeout=config.get("timeout", 15), use_proxy=config.get("use_proxy", False), proxy_url=config.get("proxy_url", ""), timezone=config.get("timezone", DEFAULT_TIMEZONE), freshness_enabled=freshness_enabled, default_max_age_days=default_max_age_days, ) ================================================ FILE: trendradar/crawler/rss/parser.py ================================================ # coding=utf-8 """ RSS 解析器 支持 RSS 2.0、Atom 和 JSON Feed 1.1 格式的解析 """ import re import html import json from dataclasses import dataclass from datetime import datetime from typing import List, Optional, Dict, Any from email.utils import parsedate_to_datetime try: import feedparser HAS_FEEDPARSER = True except ImportError: HAS_FEEDPARSER = False feedparser = None @dataclass class ParsedRSSItem: """解析后的 RSS 条目""" title: str url: str published_at: Optional[str] = None summary: Optional[str] = None author: Optional[str] = None guid: Optional[str] = None class RSSParser: """RSS 解析器""" def __init__(self, max_summary_length: int = 500): """ 初始化解析器 Args: max_summary_length: 摘要最大长度 """ if not HAS_FEEDPARSER: raise ImportError("RSS 解析需要安装 feedparser: pip install feedparser") self.max_summary_length = max_summary_length def parse(self, content: str, feed_url: str = "") -> List[ParsedRSSItem]: """ 解析 RSS/Atom/JSON Feed 内容 Args: content: Feed 内容（XML 或 JSON） feed_url: Feed URL（用于错误提示） Returns: 解析后的条目列表 """ # 先尝试检测 JSON Feed if self._is_json_feed(content): return self._parse_json_feed(content, feed_url) # 使用 feedparser 解析 RSS/Atom feed = feedparser.parse(content) if feed.bozo and not feed.entries: raise ValueError(f"RSS 解析失败 ({feed_url}): {feed.bozo_exception}") items = [] for entry in feed.entries: item = self._parse_entry(entry) if item: items.append(item) return items def _is_json_feed(self, content: str) -> bool: """ 检测内容是否为 JSON Feed 格式 JSON Feed 必须包含 version 字段，值为 https://jsonfeed.org/version/1 或 1.1 """ content = content.strip() if not content.startswith("{"): return False try: data = json.loads(content) version = data.get("version", "") return "jsonfeed.org" in version except (json.JSONDecodeError, TypeError): return False def _parse_json_feed(self, content: str, feed_url: str = "") -> List[ParsedRSSItem]: """ 解析 JSON Feed 1.1 格式 JSON Feed 规范: https://www.jsonfeed.org/version/1.1/ Args: content: JSON Feed 内容 feed_url: Feed URL（用于错误提示） Returns: 解析后的条目列表 """ try: data = json.loads(content) except json.JSONDecodeError as e: raise ValueError(f"JSON Feed 解析失败 ({feed_url}): {e}") items_data = data.get("items", []) if not items_data: return [] items = [] for item_data in items_data: item = self._parse_json_feed_item(item_data) if item: items.append(item) return items def _parse_json_feed_item(self, item_data: Dict[str, Any]) -> Optional[ParsedRSSItem]: """解析单个 JSON Feed 条目""" # 标题：优先 title，否则使用 content_text 的前 100 字符 title = item_data.get("title", "") if not title: content_text = item_data.get("content_text", "") if content_text: title = content_text[:100] + ("..." if len(content_text) > 100 else "") title = self._clean_text(title) if not title: return None # URL url = item_data.get("url", "") or item_data.get("external_url", "") # 发布时间（ISO 8601 格式） published_at = None date_str = item_data.get("date_published") or item_data.get("date_modified") if date_str: published_at = self._parse_iso_date(date_str) # 摘要：优先 summary，否则使用 content_text summary = item_data.get("summary", "") if not summary: content_text = item_data.get("content_text", "") content_html = item_data.get("content_html", "") summary = content_text or self._clean_text(content_html) if summary: summary = self._clean_text(summary) if len(summary) > self.max_summary_length: summary = summary[:self.max_summary_length] + "..." # 作者 author = None authors = item_data.get("authors", []) if authors: names = [a.get("name", "") for a in authors if isinstance(a, dict) and a.get("name")] if names: author = ", ".join(names) # GUID guid = item_data.get("id", "") or url return ParsedRSSItem( title=title, url=url, published_at=published_at, summary=summary or None, author=author, guid=guid, ) def _parse_iso_date(self, date_str: str) -> Optional[str]: """解析 ISO 8601 日期格式""" if not date_str: return None try: # 处理常见的 ISO 8601 格式 # 替换 Z 为 +00:00 date_str = date_str.replace("Z", "+00:00") dt = datetime.fromisoformat(date_str) return dt.isoformat() except (ValueError, TypeError): pass return None def parse_url(self, url: str, timeout: int = 10) -> List[ParsedRSSItem]: """ 从 URL 解析 RSS Args: url: RSS URL timeout: 超时时间（秒） Returns: 解析后的条目列表 """ import requests response = requests.get(url, timeout=timeout, headers={ "User-Agent": "TrendRadar/2.0 RSS Reader" }) response.raise_for_status() return self.parse(response.text, url) def _parse_entry(self, entry: Any) -> Optional[ParsedRSSItem]: """解析单个条目""" title = self._clean_text(entry.get("title", "")) if not title: return None url = entry.get("link", "") if not url: # 尝试从 links 中获取 links = entry.get("links", []) for link in links: if link.get("rel") == "alternate" or link.get("type", "").startswith("text/html"): url = link.get("href", "") break if not url and links: url = links[0].get("href", "") published_at = self._parse_date(entry) summary = self._parse_summary(entry) author = self._parse_author(entry) guid = entry.get("id") or entry.get("guid", {}).get("value") or url return ParsedRSSItem( title=title, url=url, published_at=published_at, summary=summary, author=author, guid=guid, ) def _clean_text(self, text: str) -> str: """清理文本""" if not text: return "" # 解码 HTML 实体 text = html.unescape(text) # 移除 HTML 标签 text = re.sub(r'<[^>]+>', '', text) # 移除多余空白 text = re.sub(r'\s+', ' ', text) return text.strip() def _parse_date(self, entry: Any) -> Optional[str]: """解析发布日期""" # feedparser 会自动解析日期到 published_parsed date_struct = entry.get("published_parsed") or entry.get("updated_parsed") if date_struct: try: dt = datetime(*date_struct[:6]) return dt.isoformat() except (ValueError, TypeError): pass # 尝试手动解析 date_str = entry.get("published") or entry.get("updated") if date_str: try: dt = parsedate_to_datetime(date_str) return dt.isoformat() except (ValueError, TypeError): pass # 尝试 ISO 格式 try: dt = datetime.fromisoformat(date_str.replace("Z", "+00:00")) return dt.isoformat() except (ValueError, TypeError): pass return None def _parse_summary(self, entry: Any) -> Optional[str]: """解析摘要""" summary = entry.get("summary") or entry.get("description", "") if not summary: # 尝试从 content 获取 content = entry.get("content", []) if content and isinstance(content, list): summary = content[0].get("value", "") if not summary: return None summary = self._clean_text(summary) # 截断过长的摘要 if len(summary) > self.max_summary_length: summary = summary[:self.max_summary_length] + "..." return summary def _parse_author(self, entry: Any) -> Optional[str]: """解析作者""" author = entry.get("author") if author: return self._clean_text(author) # 尝试从 dc:creator 获取 author = entry.get("dc_creator") if author: return self._clean_text(author) # 尝试从 authors 列表获取 authors = entry.get("authors", []) if authors: names = [a.get("name", "") for a in authors if a.get("name")] if names: return ", ".join(names) return None ================================================ FILE: trendradar/notification/__init__.py ================================================ # coding=utf-8 """ 通知推送模块 提供多渠道通知推送功能，包括： - 飞书、钉钉、企业微信 - Telegram、Slack - Email、ntfy、Bark 模块结构： - formatters: 内容格式转换 - batch: 批次处理工具 - renderer: 通知内容渲染 - splitter: 消息分批拆分 - senders: 消息发送器（各渠道发送函数） - dispatcher: 多账号通知调度器 """ from trendradar.notification.formatters import ( strip_markdown, convert_markdown_to_mrkdwn, ) from trendradar.notification.batch import ( get_batch_header, get_max_batch_header_size, truncate_to_bytes, add_batch_headers, ) from trendradar.notification.renderer import ( render_feishu_content, render_dingtalk_content, ) from trendradar.notification.splitter import ( split_content_into_batches, DEFAULT_BATCH_SIZES, ) from trendradar.notification.senders import ( send_to_feishu, send_to_dingtalk, send_to_wework, send_to_telegram, send_to_email, send_to_ntfy, send_to_bark, send_to_slack, SMTP_CONFIGS, ) from trendradar.notification.dispatcher import NotificationDispatcher __all__ = [ # 格式转换 "strip_markdown", "convert_markdown_to_mrkdwn", # 批次处理 "get_batch_header", "get_max_batch_header_size", "truncate_to_bytes", "add_batch_headers", # 内容渲染 "render_feishu_content", "render_dingtalk_content", # 消息分批 "split_content_into_batches", "DEFAULT_BATCH_SIZES", # 消息发送器 "send_to_feishu", "send_to_dingtalk", "send_to_wework", "send_to_telegram", "send_to_email", "send_to_ntfy", "send_to_bark", "send_to_slack", "SMTP_CONFIGS", # 通知调度器 "NotificationDispatcher", ] ================================================ FILE: trendradar/notification/batch.py ================================================ # coding=utf-8 """ 批次处理模块 提供消息分批发送的辅助函数 """ from typing import List def get_batch_header(format_type: str, batch_num: int, total_batches: int) -> str: """根据 format_type 生成对应格式的批次头部 Args: format_type: 推送类型（telegram, slack, wework_text, bark, feishu, dingtalk, ntfy, wework） batch_num: 当前批次编号 total_batches: 总批次数 Returns: 格式化的批次头部字符串 """ if format_type == "telegram": return f"[第 {batch_num}/{total_batches} 批次]\n\n" elif format_type == "slack": return f"*[第 {batch_num}/{total_batches} 批次]*\n\n" elif format_type in ("wework_text", "bark"): # 企业微信文本模式和 Bark 使用纯文本格式 return f"[第 {batch_num}/{total_batches} 批次]\n\n" else: # 飞书、钉钉、ntfy、企业微信 markdown 模式 return f"**[第 {batch_num}/{total_batches} 批次]**\n\n" def get_max_batch_header_size(format_type: str) -> int: """估算批次头部的最大字节数（假设最多 99 批次） 用于在分批时预留空间，避免事后截断破坏内容完整性。 Args: format_type: 推送类型 Returns: 最大头部字节数 """ # 生成最坏情况的头部（99/99 批次） max_header = get_batch_header(format_type, 99, 99) return len(max_header.encode("utf-8")) def truncate_to_bytes(text: str, max_bytes: int) -> str: """安全截断字符串到指定字节数，避免截断多字节字符 Args: text: 要截断的文本 max_bytes: 最大字节数 Returns: 截断后的文本 """ text_bytes = text.encode("utf-8") if len(text_bytes) <= max_bytes: return text # 截断到指定字节数 truncated = text_bytes[:max_bytes] # 处理可能的不完整 UTF-8 字符 for i in range(min(4, len(truncated))): try: return truncated[: len(truncated) - i].decode("utf-8") except UnicodeDecodeError: continue # 极端情况：返回空字符串 return "" def add_batch_headers( batches: List[str], format_type: str, max_bytes: int ) -> List[str]: """为批次添加头部，动态计算确保总大小不超过限制 Args: batches: 原始批次列表 format_type: 推送类型（bark, telegram, feishu 等） max_bytes: 该推送类型的最大字节限制 Returns: 添加头部后的批次列表 """ if len(batches) <= 1: return batches total = len(batches) result = [] for i, content in enumerate(batches, 1): # 生成批次头部 header = get_batch_header(format_type, i, total) header_size = len(header.encode("utf-8")) # 动态计算允许的最大内容大小 max_content_size = max_bytes - header_size content_size = len(content.encode("utf-8")) # 如果超出，截断到安全大小 if content_size > max_content_size: print( f"警告：{format_type} 第 {i}/{total} 批次内容({content_size}字节) + 头部({header_size}字节) 超出限制({max_bytes}字节)，截断到 {max_content_size} 字节" ) content = truncate_to_bytes(content, max_content_size) result.append(header + content) return result ================================================ FILE: trendradar/notification/dispatcher.py ================================================ # coding=utf-8 """ 通知调度器模块 提供统一的通知分发接口。 支持所有通知渠道的多账号配置，使用 `;` 分隔多个账号。 使用示例: dispatcher = NotificationDispatcher(config, get_time_func, split_content_func) results = dispatcher.dispatch_all(report_data, report_type, ...) """ from __future__ import annotations from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional from trendradar.core.config import ( get_account_at_index, limit_accounts, parse_multi_account_config, validate_paired_configs, ) from .senders import ( send_to_bark, send_to_dingtalk, send_to_email, send_to_feishu, send_to_ntfy, send_to_slack, send_to_telegram, send_to_wework, send_to_generic_webhook, ) from .renderer import ( render_rss_feishu_content, render_rss_dingtalk_content, render_rss_markdown_content, ) # 类型检查时导入，运行时不导入（避免循环导入） if TYPE_CHECKING: from trendradar.ai import AIAnalysisResult, AITranslator class NotificationDispatcher: """ 统一的多账号通知调度器 将多账号发送逻辑封装，提供简洁的 dispatch_all 接口。 内部处理账号解析、数量限制、配对验证等逻辑。 """ def __init__( self, config: Dict[str, Any], get_time_func: Callable, split_content_func: Callable, translator: Optional["AITranslator"] = None, ): """ 初始化通知调度器 Args: config: 完整的配置字典，包含所有通知渠道的配置 get_time_func: 获取当前时间的函数 split_content_func: 内容分批函数 translator: AI 翻译器实例（可选） """ self.config = config self.get_time_func = get_time_func self.split_content_func = split_content_func self.max_accounts = config.get("MAX_ACCOUNTS_PER_CHANNEL", 3) self.translator = translator def _translate_content( self, report_data: Dict, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, standalone_data: Optional[Dict] = None, display_regions: Optional[Dict] = None, ) -> tuple: """ 翻译推送内容 Args: report_data: 报告数据 rss_items: RSS 统计条目 rss_new_items: RSS 新增条目 standalone_data: 独立展示区数据 display_regions: 区域显示配置（不展示的区域跳过翻译） Returns: tuple: (翻译后的 report_data, rss_items, rss_new_items, standalone_data) """ if not self.translator or not self.translator.enabled: return report_data, rss_items, rss_new_items, standalone_data import copy print(f"[翻译] 开始翻译内容到 {self.translator.target_language}...") scope = self.translator.scope display_regions = display_regions or {} # 深拷贝避免修改原始数据 report_data = copy.deepcopy(report_data) rss_items = copy.deepcopy(rss_items) if rss_items else None rss_new_items = copy.deepcopy(rss_new_items) if rss_new_items else None standalone_data = copy.deepcopy(standalone_data) if standalone_data else None # 收集所有需要翻译的标题 titles_to_translate = [] title_locations = [] # 记录标题位置，用于回填 # 1. 热榜标题（scope 开启 且 区域展示） if scope.get("HOTLIST", True) and display_regions.get("HOTLIST", True): for stat_idx, stat in enumerate(report_data.get("stats", [])): for title_idx, title_data in enumerate(stat.get("titles", [])): titles_to_translate.append(title_data.get("title", "")) title_locations.append(("stats", stat_idx, title_idx)) # 2. 新增热点标题 for source_idx, source in enumerate(report_data.get("new_titles", [])): for title_idx, title_data in enumerate(source.get("titles", [])): titles_to_translate.append(title_data.get("title", "")) title_locations.append(("new_titles", source_idx, title_idx)) # 3. RSS 统计标题（结构与 stats 一致：[{word, count, titles: [{title, ...}]}]） if rss_items and scope.get("RSS", True) and display_regions.get("RSS", True): for stat_idx, stat in enumerate(rss_items): for title_idx, title_data in enumerate(stat.get("titles", [])): titles_to_translate.append(title_data.get("title", "")) title_locations.append(("rss_items", stat_idx, title_idx)) # 4. RSS 新增标题（结构与 stats 一致） if rss_new_items and scope.get("RSS", True) and display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True): for stat_idx, stat in enumerate(rss_new_items): for title_idx, title_data in enumerate(stat.get("titles", [])): titles_to_translate.append(title_data.get("title", "")) title_locations.append(("rss_new_items", stat_idx, title_idx)) # 5. 独立展示区 - 热榜平台 if standalone_data and scope.get("STANDALONE", True) and display_regions.get("STANDALONE", False): for plat_idx, platform in enumerate(standalone_data.get("platforms", [])): for item_idx, item in enumerate(platform.get("items", [])): titles_to_translate.append(item.get("title", "")) title_locations.append(("standalone_platforms", plat_idx, item_idx)) # 6. 独立展示区 - RSS 源 for feed_idx, feed in enumerate(standalone_data.get("rss_feeds", [])): for item_idx, item in enumerate(feed.get("items", [])): titles_to_translate.append(item.get("title", "")) title_locations.append(("standalone_rss", feed_idx, item_idx)) if not titles_to_translate: print("[翻译] 没有需要翻译的内容") return report_data, rss_items, rss_new_items, standalone_data print(f"[翻译] 共 {len(titles_to_translate)} 条标题待翻译") # 批量翻译 result = self.translator.translate_batch(titles_to_translate) if result.success_count == 0: print(f"[翻译] 翻译失败: {result.results[0].error if result.results else '未知错误'}") return report_data, rss_items, rss_new_items, standalone_data print(f"[翻译] 翻译完成: {result.success_count}/{result.total_count} 成功") # debug 模式：输出完整 prompt、AI 原始响应、逐条对照 if self.config.get("DEBUG", False): if result.prompt: print(f"[翻译][DEBUG] === 发送给 AI 的 Prompt ===") print(result.prompt) print(f"[翻译][DEBUG] === Prompt 结束 ===") if result.raw_response: print(f"[翻译][DEBUG] === AI 原始响应 ===") print(result.raw_response) print(f"[翻译][DEBUG] === 响应结束 ===") # 行数不匹配警告 expected = len(titles_to_translate) if result.parsed_count != expected: print(f"[翻译][DEBUG] ⚠️ 行数不匹配：期望 {expected} 条，AI 返回 {result.parsed_count} 条") # 逐条对照 unchanged_count = 0 for i, res in enumerate(result.results): if not res.success and res.error: print(f"[翻译][DEBUG] [{i+1}] !! 失败: {res.error}") elif res.original_text == res.translated_text: unchanged_count += 1 else: print(f"[翻译][DEBUG] [{i+1}] {res.original_text} => {res.translated_text}") if unchanged_count > 0: print(f"[翻译][DEBUG] （另有 {unchanged_count} 条未变化，已省略）") # 回填翻译结果 for i, (loc_type, idx1, idx2) in enumerate(title_locations): if i < len(result.results) and result.results[i].success: translated = result.results[i].translated_text if loc_type == "stats": report_data["stats"][idx1]["titles"][idx2]["title"] = translated elif loc_type == "new_titles": report_data["new_titles"][idx1]["titles"][idx2]["title"] = translated elif loc_type == "rss_items" and rss_items: rss_items[idx1]["titles"][idx2]["title"] = translated elif loc_type == "rss_new_items" and rss_new_items: rss_new_items[idx1]["titles"][idx2]["title"] = translated elif loc_type == "standalone_platforms" and standalone_data: standalone_data["platforms"][idx1]["items"][idx2]["title"] = translated elif loc_type == "standalone_rss" and standalone_data: standalone_data["rss_feeds"][idx1]["items"][idx2]["title"] = translated return report_data, rss_items, rss_new_items, standalone_data def dispatch_all( self, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", html_file_path: Optional[str] = None, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, standalone_data: Optional[Dict] = None, ) -> Dict[str, bool]: """ 分发通知到所有已配置的渠道（支持热榜+RSS合并推送+AI分析+独立展示区） Args: report_data: 报告数据（由 prepare_report_data 生成） report_type: 报告类型（如 "全天汇总"、"当前榜单"、"增量分析"） update_info: 版本更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current/incremental) html_file_path: HTML 报告文件路径（邮件使用） rss_items: RSS 统计条目列表（用于 RSS 统计区块） rss_new_items: RSS 新增条目列表（用于 RSS 新增区块） ai_analysis: AI 分析结果（可选） standalone_data: 独立展示区数据（可选） Returns: Dict[str, bool]: 每个渠道的发送结果，key 为渠道名，value 为是否成功 """ results = {} # 获取区域显示配置 display_regions = self.config.get("DISPLAY", {}).get("REGIONS", {}) # 执行翻译（如果启用，根据 display_regions 跳过不展示的区域） report_data, rss_items, rss_new_items, standalone_data = self._translate_content( report_data, rss_items, rss_new_items, standalone_data, display_regions ) # 飞书 if self.config.get("FEISHU_WEBHOOK_URL"): results["feishu"] = self._send_feishu( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # 钉钉 if self.config.get("DINGTALK_WEBHOOK_URL"): results["dingtalk"] = self._send_dingtalk( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # 企业微信 if self.config.get("WEWORK_WEBHOOK_URL"): results["wework"] = self._send_wework( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # Telegram（需要配对验证） if self.config.get("TELEGRAM_BOT_TOKEN") and self.config.get("TELEGRAM_CHAT_ID"): results["telegram"] = self._send_telegram( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # ntfy（需要配对验证） if self.config.get("NTFY_SERVER_URL") and self.config.get("NTFY_TOPIC"): results["ntfy"] = self._send_ntfy( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # Bark if self.config.get("BARK_URL"): results["bark"] = self._send_bark( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # Slack if self.config.get("SLACK_WEBHOOK_URL"): results["slack"] = self._send_slack( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # 通用 Webhook if self.config.get("GENERIC_WEBHOOK_URL"): results["generic_webhook"] = self._send_generic_webhook( report_data, report_type, update_info, proxy_url, mode, rss_items, rss_new_items, ai_analysis, display_regions, standalone_data ) # 邮件（保持原有逻辑，已支持多收件人，AI 分析已嵌入 HTML） if ( self.config.get("EMAIL_FROM") and self.config.get("EMAIL_PASSWORD") and self.config.get("EMAIL_TO") ): results["email"] = self._send_email(report_type, html_file_path) return results def _send_to_multi_accounts( self, channel_name: str, config_value: str, send_func: Callable[..., bool], **kwargs, ) -> bool: """ 通用多账号发送逻辑 Args: channel_name: 渠道名称（用于日志和账号数量限制提示） config_value: 配置值（可能包含多个账号，用 ; 分隔） send_func: 发送函数，签名为 (account, account_label=..., **kwargs) -> bool **kwargs: 传递给发送函数的其他参数 Returns: bool: 任一账号发送成功则返回 True """ accounts = parse_multi_account_config(config_value) if not accounts: return False accounts = limit_accounts(accounts, self.max_accounts, channel_name) results = [] for i, account in enumerate(accounts): if account: account_label = f"账号{i+1}" if len(accounts) > 1 else "" result = send_func(account, account_label=account_label, **kwargs) results.append(result) return any(results) if results else False def _send_feishu( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到飞书（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} return self._send_to_multi_accounts( channel_name="飞书", config_value=self.config["FEISHU_WEBHOOK_URL"], send_func=lambda url, account_label: send_to_feishu( webhook_url=url, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("FEISHU_BATCH_SIZE", 29000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, get_time_func=self.get_time_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ), ) def _send_dingtalk( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到钉钉（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} return self._send_to_multi_accounts( channel_name="钉钉", config_value=self.config["DINGTALK_WEBHOOK_URL"], send_func=lambda url, account_label: send_to_dingtalk( webhook_url=url, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("DINGTALK_BATCH_SIZE", 20000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ), ) def _send_wework( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到企业微信（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} return self._send_to_multi_accounts( channel_name="企业微信", config_value=self.config["WEWORK_WEBHOOK_URL"], send_func=lambda url, account_label: send_to_wework( webhook_url=url, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("MESSAGE_BATCH_SIZE", 4000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), msg_type=self.config.get("WEWORK_MSG_TYPE", "markdown"), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ), ) def _send_telegram( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到 Telegram（多账号，需验证 token 和 chat_id 配对，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} telegram_tokens = parse_multi_account_config(self.config["TELEGRAM_BOT_TOKEN"]) telegram_chat_ids = parse_multi_account_config(self.config["TELEGRAM_CHAT_ID"]) if not telegram_tokens or not telegram_chat_ids: return False valid, count = validate_paired_configs( {"bot_token": telegram_tokens, "chat_id": telegram_chat_ids}, "Telegram", required_keys=["bot_token", "chat_id"], ) if not valid or count == 0: return False telegram_tokens = limit_accounts(telegram_tokens, self.max_accounts, "Telegram") telegram_chat_ids = telegram_chat_ids[: len(telegram_tokens)] results = [] for i in range(len(telegram_tokens)): token = telegram_tokens[i] chat_id = telegram_chat_ids[i] if token and chat_id: account_label = f"账号{i+1}" if len(telegram_tokens) > 1 else "" result = send_to_telegram( bot_token=token, chat_id=chat_id, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("MESSAGE_BATCH_SIZE", 4000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ) results.append(result) return any(results) if results else False def _send_ntfy( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到 ntfy（多账号，需验证 topic 和 token 配对，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} ntfy_server_url = self.config["NTFY_SERVER_URL"] ntfy_topics = parse_multi_account_config(self.config["NTFY_TOPIC"]) ntfy_tokens = parse_multi_account_config(self.config.get("NTFY_TOKEN", "")) if not ntfy_server_url or not ntfy_topics: return False if ntfy_tokens and len(ntfy_tokens) != len(ntfy_topics): print( f"❌ ntfy 配置错误：topic 数量({len(ntfy_topics)})与 token 数量({len(ntfy_tokens)})不一致，跳过 ntfy 推送" ) return False ntfy_topics = limit_accounts(ntfy_topics, self.max_accounts, "ntfy") if ntfy_tokens: ntfy_tokens = ntfy_tokens[: len(ntfy_topics)] results = [] for i, topic in enumerate(ntfy_topics): if topic: token = get_account_at_index(ntfy_tokens, i, "") if ntfy_tokens else "" account_label = f"账号{i+1}" if len(ntfy_topics) > 1 else "" result = send_to_ntfy( server_url=ntfy_server_url, topic=topic, token=token, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=3800, split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ) results.append(result) return any(results) if results else False def _send_bark( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到 Bark（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} return self._send_to_multi_accounts( channel_name="Bark", config_value=self.config["BARK_URL"], send_func=lambda url, account_label: send_to_bark( bark_url=url, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("BARK_BATCH_SIZE", 3600), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ), ) def _send_slack( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到 Slack（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} return self._send_to_multi_accounts( channel_name="Slack", config_value=self.config["SLACK_WEBHOOK_URL"], send_func=lambda url, account_label: send_to_slack( webhook_url=url, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("SLACK_BATCH_SIZE", 4000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ), ) def _send_generic_webhook( self, report_data: Dict, report_type: str, update_info: Optional[Dict], proxy_url: Optional[str], mode: str, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, ai_analysis: Optional[AIAnalysisResult] = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """发送到通用 Webhook（多账号，支持热榜+RSS合并+AI分析+独立展示区）""" display_regions = display_regions or {} if not display_regions.get("HOTLIST", True): report_data = {"stats": [], "failed_ids": [], "new_titles": [], "id_to_name": {}} urls = parse_multi_account_config(self.config.get("GENERIC_WEBHOOK_URL", "")) templates = parse_multi_account_config(self.config.get("GENERIC_WEBHOOK_TEMPLATE", "")) if not urls: return False urls = limit_accounts(urls, self.max_accounts, "通用Webhook") results = [] for i, url in enumerate(urls): if not url: continue template = "" if templates: if i < len(templates): template = templates[i] elif len(templates) == 1: template = templates[0] account_label = f"账号{i+1}" if len(urls) > 1 else "" result = send_to_generic_webhook( webhook_url=url, payload_template=template, report_data=report_data, report_type=report_type, update_info=update_info, proxy_url=proxy_url, mode=mode, account_label=account_label, batch_size=self.config.get("MESSAGE_BATCH_SIZE", 4000), batch_interval=self.config.get("BATCH_SEND_INTERVAL", 1.0), split_content_func=self.split_content_func, rss_items=rss_items if display_regions.get("RSS", True) else None, rss_new_items=rss_new_items if (display_regions.get("RSS", True) and display_regions.get("NEW_ITEMS", True)) else None, ai_analysis=ai_analysis if display_regions.get("AI_ANALYSIS", True) else None, display_regions=display_regions, standalone_data=standalone_data if display_regions.get("STANDALONE", False) else None, ) results.append(result) return any(results) if results else False def _send_email( self, report_type: str, html_file_path: Optional[str], ) -> bool: """发送邮件（保持原有逻辑，已支持多收件人） Note: AI 分析内容已在 HTML 生成时嵌入，无需在此传递 """ return send_to_email( from_email=self.config["EMAIL_FROM"], password=self.config["EMAIL_PASSWORD"], to_email=self.config["EMAIL_TO"], report_type=report_type, html_file_path=html_file_path, custom_smtp_server=self.config.get("EMAIL_SMTP_SERVER", ""), custom_smtp_port=self.config.get("EMAIL_SMTP_PORT", ""), get_time_func=self.get_time_func, ) # === RSS 通知方法 === def dispatch_rss( self, rss_items: List[Dict], feeds_info: Optional[Dict[str, str]] = None, proxy_url: Optional[str] = None, html_file_path: Optional[str] = None, ) -> Dict[str, bool]: """ 分发 RSS 通知到所有已配置的渠道 Args: rss_items: RSS 条目列表，每个条目包含: - title: 标题 - feed_id: RSS 源 ID - feed_name: RSS 源名称 - url: 链接 - published_at: 发布时间 - summary: 摘要（可选） - author: 作者（可选） feeds_info: RSS 源 ID 到名称的映射 proxy_url: 代理 URL（可选） html_file_path: HTML 报告文件路径（邮件使用） Returns: Dict[str, bool]: 每个渠道的发送结果 """ if not rss_items: print("[RSS通知] 没有 RSS 内容，跳过通知") return {} results = {} report_type = "RSS 订阅更新" # 飞书 if self.config.get("FEISHU_WEBHOOK_URL"): results["feishu"] = self._send_rss_feishu( rss_items, feeds_info, proxy_url ) # 钉钉 if self.config.get("DINGTALK_WEBHOOK_URL"): results["dingtalk"] = self._send_rss_dingtalk( rss_items, feeds_info, proxy_url ) # 企业微信 if self.config.get("WEWORK_WEBHOOK_URL"): results["wework"] = self._send_rss_markdown( rss_items, feeds_info, proxy_url, "wework" ) # Telegram if self.config.get("TELEGRAM_BOT_TOKEN") and self.config.get("TELEGRAM_CHAT_ID"): results["telegram"] = self._send_rss_markdown( rss_items, feeds_info, proxy_url, "telegram" ) # ntfy if self.config.get("NTFY_SERVER_URL") and self.config.get("NTFY_TOPIC"): results["ntfy"] = self._send_rss_markdown( rss_items, feeds_info, proxy_url, "ntfy" ) # Bark if self.config.get("BARK_URL"): results["bark"] = self._send_rss_markdown( rss_items, feeds_info, proxy_url, "bark" ) # Slack if self.config.get("SLACK_WEBHOOK_URL"): results["slack"] = self._send_rss_markdown( rss_items, feeds_info, proxy_url, "slack" ) # 邮件 if ( self.config.get("EMAIL_FROM") and self.config.get("EMAIL_PASSWORD") and self.config.get("EMAIL_TO") ): results["email"] = self._send_email(report_type, html_file_path) return results def _send_rss_feishu( self, rss_items: List[Dict], feeds_info: Optional[Dict[str, str]], proxy_url: Optional[str], ) -> bool: """发送 RSS 到飞书""" import requests content = render_rss_feishu_content( rss_items=rss_items, feeds_info=feeds_info, get_time_func=self.get_time_func, ) webhooks = parse_multi_account_config(self.config["FEISHU_WEBHOOK_URL"]) webhooks = limit_accounts(webhooks, self.max_accounts, "飞书") results = [] for i, webhook_url in enumerate(webhooks): if not webhook_url: continue account_label = f"账号{i+1}" if len(webhooks) > 1 else "" try: # 分批发送 batches = self.split_content_func( content, self.config.get("FEISHU_BATCH_SIZE", 29000) ) for batch_idx, batch_content in enumerate(batches): payload = { "msg_type": "interactive", "card": { "header": { "title": { "tag": "plain_text", "content": f"📰 RSS 订阅更新 {f'({batch_idx + 1}/{len(batches)})' if len(batches) > 1 else ''}", }, "template": "green", }, "elements": [ {"tag": "markdown", "content": batch_content} ], }, } proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post(webhook_url, json=payload, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ 飞书{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ 飞书{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_dingtalk( self, rss_items: List[Dict], feeds_info: Optional[Dict[str, str]], proxy_url: Optional[str], ) -> bool: """发送 RSS 到钉钉""" import requests content = render_rss_dingtalk_content( rss_items=rss_items, feeds_info=feeds_info, get_time_func=self.get_time_func, ) webhooks = parse_multi_account_config(self.config["DINGTALK_WEBHOOK_URL"]) webhooks = limit_accounts(webhooks, self.max_accounts, "钉钉") results = [] for i, webhook_url in enumerate(webhooks): if not webhook_url: continue account_label = f"账号{i+1}" if len(webhooks) > 1 else "" try: batches = self.split_content_func( content, self.config.get("DINGTALK_BATCH_SIZE", 20000) ) for batch_idx, batch_content in enumerate(batches): title = f"📰 RSS 订阅更新 {f'({batch_idx + 1}/{len(batches)})' if len(batches) > 1 else ''}" payload = { "msgtype": "markdown", "markdown": { "title": title, "text": batch_content, }, } proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post(webhook_url, json=payload, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ 钉钉{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ 钉钉{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_markdown( self, rss_items: List[Dict], feeds_info: Optional[Dict[str, str]], proxy_url: Optional[str], channel: str, ) -> bool: """发送 RSS 到 Markdown 兼容渠道（企业微信、Telegram、ntfy、Bark、Slack）""" content = render_rss_markdown_content( rss_items=rss_items, feeds_info=feeds_info, get_time_func=self.get_time_func, ) try: if channel == "wework": return self._send_rss_wework(content, proxy_url) elif channel == "telegram": return self._send_rss_telegram(content, proxy_url) elif channel == "ntfy": return self._send_rss_ntfy(content, proxy_url) elif channel == "bark": return self._send_rss_bark(content, proxy_url) elif channel == "slack": return self._send_rss_slack(content, proxy_url) except Exception as e: print(f"❌ {channel} RSS 通知发送失败: {e}") return False return False def _send_rss_wework(self, content: str, proxy_url: Optional[str]) -> bool: """发送 RSS 到企业微信""" import requests webhooks = parse_multi_account_config(self.config["WEWORK_WEBHOOK_URL"]) webhooks = limit_accounts(webhooks, self.max_accounts, "企业微信") results = [] for i, webhook_url in enumerate(webhooks): if not webhook_url: continue account_label = f"账号{i+1}" if len(webhooks) > 1 else "" try: batches = self.split_content_func( content, self.config.get("MESSAGE_BATCH_SIZE", 4000) ) for batch_content in batches: payload = { "msgtype": "markdown", "markdown": {"content": batch_content}, } proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post(webhook_url, json=payload, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ 企业微信{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ 企业微信{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_telegram(self, content: str, proxy_url: Optional[str]) -> bool: """发送 RSS 到 Telegram""" import requests tokens = parse_multi_account_config(self.config["TELEGRAM_BOT_TOKEN"]) chat_ids = parse_multi_account_config(self.config["TELEGRAM_CHAT_ID"]) if not tokens or not chat_ids: return False results = [] for i in range(min(len(tokens), len(chat_ids), self.max_accounts)): token = tokens[i] chat_id = chat_ids[i] if not token or not chat_id: continue account_label = f"账号{i+1}" if len(tokens) > 1 else "" try: batches = self.split_content_func( content, self.config.get("MESSAGE_BATCH_SIZE", 4000) ) for batch_content in batches: url = f"https://api.telegram.org/bot{token}/sendMessage" payload = { "chat_id": chat_id, "text": batch_content, "parse_mode": "Markdown", } proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post(url, json=payload, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ Telegram{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ Telegram{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_ntfy(self, content: str, proxy_url: Optional[str]) -> bool: """发送 RSS 到 ntfy""" import requests server_url = self.config["NTFY_SERVER_URL"] topics = parse_multi_account_config(self.config["NTFY_TOPIC"]) tokens = parse_multi_account_config(self.config.get("NTFY_TOKEN", "")) if not server_url or not topics: return False topics = limit_accounts(topics, self.max_accounts, "ntfy") results = [] for i, topic in enumerate(topics): if not topic: continue token = tokens[i] if tokens and i < len(tokens) else "" account_label = f"账号{i+1}" if len(topics) > 1 else "" try: batches = self.split_content_func(content, 3800) for batch_content in batches: url = f"{server_url.rstrip('/')}/{topic}" headers = {"Title": "RSS 订阅更新", "Markdown": "yes"} if token: headers["Authorization"] = f"Bearer {token}" proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post( url, data=batch_content.encode("utf-8"), headers=headers, proxies=proxies, timeout=30 ) resp.raise_for_status() print(f"✅ ntfy{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ ntfy{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_bark(self, content: str, proxy_url: Optional[str]) -> bool: """发送 RSS 到 Bark""" import requests import urllib.parse urls = parse_multi_account_config(self.config["BARK_URL"]) urls = limit_accounts(urls, self.max_accounts, "Bark") results = [] for i, bark_url in enumerate(urls): if not bark_url: continue account_label = f"账号{i+1}" if len(urls) > 1 else "" try: batches = self.split_content_func( content, self.config.get("BARK_BATCH_SIZE", 3600) ) for batch_content in batches: title = urllib.parse.quote("📰 RSS 订阅更新") body = urllib.parse.quote(batch_content) url = f"{bark_url.rstrip('/')}/{title}/{body}" proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.get(url, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ Bark{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ Bark{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False def _send_rss_slack(self, content: str, proxy_url: Optional[str]) -> bool: """发送 RSS 到 Slack""" import requests webhooks = parse_multi_account_config(self.config["SLACK_WEBHOOK_URL"]) webhooks = limit_accounts(webhooks, self.max_accounts, "Slack") results = [] for i, webhook_url in enumerate(webhooks): if not webhook_url: continue account_label = f"账号{i+1}" if len(webhooks) > 1 else "" try: batches = self.split_content_func( content, self.config.get("SLACK_BATCH_SIZE", 4000) ) for batch_content in batches: payload = { "blocks": [ { "type": "section", "text": { "type": "mrkdwn", "text": batch_content, }, } ] } proxies = {"http": proxy_url, "https": proxy_url} if proxy_url else None resp = requests.post(webhook_url, json=payload, proxies=proxies, timeout=30) resp.raise_for_status() print(f"✅ Slack{account_label} RSS 通知发送成功") results.append(True) except Exception as e: print(f"❌ Slack{account_label} RSS 通知发送失败: {e}") results.append(False) return any(results) if results else False ================================================ FILE: trendradar/notification/formatters.py ================================================ # coding=utf-8 """ 通知内容格式转换模块 提供不同推送平台间的格式转换功能 """ import re def strip_markdown(text: str) -> str: """去除文本中的 markdown 语法格式，用于个人微信推送 Args: text: 包含 markdown 格式的文本 Returns: 纯文本内容 """ # 转换链接 [text](url) -> text url（保留 URL） text = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'\1 \2', text) # 先保护 URL，避免后续 markdown 清洗误伤链接中的下划线等字符 protected_urls: list[str] = [] def _protect_url(match: re.Match) -> str: protected_urls.append(match.group(0)) return f"@@URLTOKEN{len(protected_urls) - 1}@@" text = re.sub(r'https?://[^\s<>\]]+', _protect_url, text) # 去除粗体 **text** 或 __text__ text = re.sub(r'\*\*(.+?)\*\*', r'\1', text) text = re.sub(r'(? alt text = re.sub(r'!\[(.+?)\]\(.+?\)', r'\1', text) # 去除行内代码 `code` text = re.sub(r'`(.+?)`', r'\1', text) # 去除引用符号 > text = re.sub(r'^>\s*', '', text, flags=re.MULTILINE) # 去除标题符号 # ## ### 等 text = re.sub(r'^#+\s*', '', text, flags=re.MULTILINE) # 去除水平分割线 --- 或 *** text = re.sub(r'^[\-\*]{3,}\s*$', '', text, flags=re.MULTILINE) # 去除 HTML 标签 text -> text text = re.sub(r']*>(.+?)', r'\1', text) text = re.sub(r'<[^>]+>', '', text) # 清理多余的空行（保留最多两个连续空行） text = re.sub(r'\n{3,}', '\n\n', text) # 还原之前保护的 URL for idx, url in enumerate(protected_urls): text = text.replace(f"@@URLTOKEN{idx}@@", url) return text.strip() def convert_markdown_to_mrkdwn(content: str) -> str: """ 将标准 Markdown 转换为 Slack 的 mrkdwn 格式 转换规则： - **粗体** → *粗体* - [文本](url) → - 保留其他格式（代码块、列表等） Args: content: Markdown 格式的内容 Returns: Slack mrkdwn 格式的内容 """ # 1. 转换链接格式: [文本](url) → content = re.sub(r'\[([^\]]+)\]\(([^)]+)\)', r'<\2|\1>', content) # 2. 转换粗体: **文本** → *文本* content = re.sub(r'\*\*([^*]+)\*\*', r'*\1*', content) return content ================================================ FILE: trendradar/notification/renderer.py ================================================ # coding=utf-8 """ 通知内容渲染模块 提供多平台通知内容渲染功能，生成格式化的推送消息 """ from datetime import datetime from typing import Dict, List, Optional, Callable from trendradar.report.formatter import format_title_for_platform # 默认区域顺序 DEFAULT_REGION_ORDER = ["hotlist", "rss", "new_items", "standalone", "ai_analysis"] def render_feishu_content( report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily", separator: str = "---", region_order: Optional[List[str]] = None, get_time_func: Optional[Callable[[], datetime]] = None, rss_items: Optional[list] = None, show_new_section: bool = True, ) -> str: """渲染飞书通知内容（支持热榜+RSS合并） Args: report_data: 报告数据字典，包含 stats, new_titles, failed_ids, total_new_count update_info: 版本更新信息（可选） mode: 报告模式 ("daily", "incremental", "current") separator: 内容分隔符 region_order: 区域显示顺序列表 get_time_func: 获取当前时间的函数（可选，默认使用 datetime.now()） rss_items: RSS 条目列表（可选，用于合并推送） show_new_section: 是否显示新增热点区域 Returns: 格式化的飞书消息内容 """ if region_order is None: region_order = DEFAULT_REGION_ORDER # 生成热点词汇统计部分 stats_content = "" if report_data["stats"]: stats_content += "📊 **热点词汇统计**\n\n" total_count = len(report_data["stats"]) for i, stat in enumerate(report_data["stats"]): word = stat["word"] count = stat["count"] sequence_display = f"[{i + 1}/{total_count}]" if count >= 10: stats_content += f"🔥 {sequence_display} **{word}** : {count} 条\n\n" elif count >= 5: stats_content += f"📈 {sequence_display} **{word}** : {count} 条\n\n" else: stats_content += f"📌 {sequence_display} **{word}** : {count} 条\n\n" for j, title_data in enumerate(stat["titles"], 1): formatted_title = format_title_for_platform( "feishu", title_data, show_source=True ) stats_content += f" {j}. {formatted_title}\n" if j < len(stat["titles"]): stats_content += "\n" if i < len(report_data["stats"]) - 1: stats_content += f"\n{separator}\n\n" # 生成新增新闻部分 new_titles_content = "" if show_new_section and report_data["new_titles"]: new_titles_content += ( f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" ) for source_data in report_data["new_titles"]: new_titles_content += ( f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n" ) for j, title_data in enumerate(source_data["titles"], 1): title_data_copy = title_data.copy() title_data_copy["is_new"] = False formatted_title = format_title_for_platform( "feishu", title_data_copy, show_source=False ) new_titles_content += f" {j}. {formatted_title}\n" new_titles_content += "\n" # RSS 内容 rss_content = "" if rss_items: rss_content = _render_rss_section_feishu(rss_items, separator) # 准备各区域内容映射 region_contents = { "hotlist": stats_content, "new_items": new_titles_content, "rss": rss_content, } # 按 region_order 顺序组装内容 text_content = "" for region in region_order: content = region_contents.get(region, "") if content: if text_content: text_content += f"\n{separator}\n\n" text_content += content if not text_content: if mode == "incremental": mode_text = "增量模式下暂无新增匹配的热点词汇" elif mode == "current": mode_text = "当前榜单模式下暂无匹配的热点词汇" else: mode_text = "暂无匹配的热点词汇" text_content = f"📭 {mode_text}\n\n" if report_data["failed_ids"]: if text_content and "暂无匹配" not in text_content: text_content += f"\n{separator}\n\n" text_content += "⚠️ **数据获取失败的平台：**\n\n" for i, id_value in enumerate(report_data["failed_ids"], 1): text_content += f" • {id_value}\n" # 获取当前时间 now = get_time_func() if get_time_func else datetime.now() text_content += ( f"\n\n更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" ) if update_info: text_content += f"\nTrendRadar 发现新版本 {update_info['remote_version']}，当前 {update_info['current_version']}" return text_content def render_dingtalk_content( report_data: Dict, update_info: Optional[Dict] = None, mode: str = "daily", region_order: Optional[List[str]] = None, get_time_func: Optional[Callable[[], datetime]] = None, rss_items: Optional[list] = None, show_new_section: bool = True, ) -> str: """渲染钉钉通知内容（支持热榜+RSS合并） Args: report_data: 报告数据字典，包含 stats, new_titles, failed_ids, total_new_count update_info: 版本更新信息（可选） mode: 报告模式 ("daily", "incremental", "current") region_order: 区域显示顺序列表 get_time_func: 获取当前时间的函数（可选，默认使用 datetime.now()） rss_items: RSS 条目列表（可选，用于合并推送） show_new_section: 是否显示新增热点区域 Returns: 格式化的钉钉消息内容 """ if region_order is None: region_order = DEFAULT_REGION_ORDER total_titles = sum( len(stat["titles"]) for stat in report_data["stats"] if stat["count"] > 0 ) now = get_time_func() if get_time_func else datetime.now() # 头部信息 header_content = f"**总新闻数：** {total_titles}\n\n" header_content += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n\n" header_content += "**类型：** 热点分析报告\n\n" header_content += "---\n\n" # 生成热点词汇统计部分 stats_content = "" if report_data["stats"]: stats_content += "📊 **热点词汇统计**\n\n" total_count = len(report_data["stats"]) for i, stat in enumerate(report_data["stats"]): word = stat["word"] count = stat["count"] sequence_display = f"[{i + 1}/{total_count}]" if count >= 10: stats_content += f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" elif count >= 5: stats_content += f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" else: stats_content += f"📌 {sequence_display} **{word}** : {count} 条\n\n" for j, title_data in enumerate(stat["titles"], 1): formatted_title = format_title_for_platform( "dingtalk", title_data, show_source=True ) stats_content += f" {j}. {formatted_title}\n" if j < len(stat["titles"]): stats_content += "\n" if i < len(report_data["stats"]) - 1: stats_content += "\n---\n\n" # 生成新增新闻部分 new_titles_content = "" if show_new_section and report_data["new_titles"]: new_titles_content += ( f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" ) for source_data in report_data["new_titles"]: new_titles_content += f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n" for j, title_data in enumerate(source_data["titles"], 1): title_data_copy = title_data.copy() title_data_copy["is_new"] = False formatted_title = format_title_for_platform( "dingtalk", title_data_copy, show_source=False ) new_titles_content += f" {j}. {formatted_title}\n" new_titles_content += "\n" # RSS 内容 rss_content = "" if rss_items: rss_content = _render_rss_section_markdown(rss_items) # 准备各区域内容映射 region_contents = { "hotlist": stats_content, "new_items": new_titles_content, "rss": rss_content, } # 按 region_order 顺序组装内容 text_content = header_content has_content = False for region in region_order: content = region_contents.get(region, "") if content: if has_content: text_content += "\n---\n\n" text_content += content has_content = True if not has_content: if mode == "incremental": mode_text = "增量模式下暂无新增匹配的热点词汇" elif mode == "current": mode_text = "当前榜单模式下暂无匹配的热点词汇" else: mode_text = "暂无匹配的热点词汇" text_content += f"📭 {mode_text}\n\n" if report_data["failed_ids"]: if "暂无匹配" not in text_content: text_content += "\n---\n\n" text_content += "⚠️ **数据获取失败的平台：**\n\n" for i, id_value in enumerate(report_data["failed_ids"], 1): text_content += f" • **{id_value}**\n" text_content += f"\n\n> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: text_content += f"\n> TrendRadar 发现新版本 **{update_info['remote_version']}**，当前 **{update_info['current_version']}**" return text_content def render_rss_feishu_content( rss_items: list, feeds_info: Optional[Dict] = None, separator: str = "---", get_time_func: Optional[Callable[[], datetime]] = None, ) -> str: """渲染 RSS 飞书通知内容 Args: rss_items: RSS 条目列表，每个条目包含: - title: 标题 - feed_id: RSS 源 ID - feed_name: RSS 源名称 - url: 链接 - published_at: 发布时间 - summary: 摘要（可选） - author: 作者（可选） feeds_info: RSS 源 ID 到名称的映射 separator: 内容分隔符 get_time_func: 获取当前时间的函数（可选） Returns: 格式化的飞书消息内容 """ if not rss_items: now = get_time_func() if get_time_func else datetime.now() return f"📭 暂无新的 RSS 订阅内容\n\n更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" # 按 feed_id 分组 feeds_map: Dict[str, list] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) text_content = f"📰 **RSS 订阅更新** (共 {len(rss_items)} 条)\n\n" text_content += f"{separator}\n\n" for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id if feeds_info and feed_id in feeds_info: feed_name = feeds_info[feed_id] text_content += f"**{feed_name}** ({len(items)} 条)\n\n" for i, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") if url: text_content += f" {i}. [{title}]({url})" else: text_content += f" {i}. {title}" if published_at: text_content += f" - {published_at}" text_content += "\n" if i < len(items): text_content += "\n" text_content += f"\n{separator}\n\n" now = get_time_func() if get_time_func else datetime.now() text_content += f"更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" return text_content def render_rss_dingtalk_content( rss_items: list, feeds_info: Optional[Dict] = None, get_time_func: Optional[Callable[[], datetime]] = None, ) -> str: """渲染 RSS 钉钉通知内容 Args: rss_items: RSS 条目列表 feeds_info: RSS 源 ID 到名称的映射 get_time_func: 获取当前时间的函数（可选） Returns: 格式化的钉钉消息内容 """ now = get_time_func() if get_time_func else datetime.now() if not rss_items: return f"📭 暂无新的 RSS 订阅内容\n\n> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" # 按 feed_id 分组 feeds_map: Dict[str, list] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) # 头部信息 text_content = f"**总条目数：** {len(rss_items)}\n\n" text_content += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n\n" text_content += "**类型：** RSS 订阅更新\n\n" text_content += "---\n\n" for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id if feeds_info and feed_id in feeds_info: feed_name = feeds_info[feed_id] text_content += f"📰 **{feed_name}** ({len(items)} 条)\n\n" for i, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") if url: text_content += f" {i}. [{title}]({url})" else: text_content += f" {i}. {title}" if published_at: text_content += f" - {published_at}" text_content += "\n" if i < len(items): text_content += "\n" text_content += "\n---\n\n" text_content += f"> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" return text_content def render_rss_markdown_content( rss_items: list, feeds_info: Optional[Dict] = None, get_time_func: Optional[Callable[[], datetime]] = None, ) -> str: """渲染 RSS 通用 Markdown 格式内容（企业微信、Bark、ntfy、Slack） Args: rss_items: RSS 条目列表 feeds_info: RSS 源 ID 到名称的映射 get_time_func: 获取当前时间的函数（可选） Returns: 格式化的 Markdown 消息内容 """ now = get_time_func() if get_time_func else datetime.now() if not rss_items: return f"📭 暂无新的 RSS 订阅内容\n\n更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" # 按 feed_id 分组 feeds_map: Dict[str, list] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) text_content = f"📰 **RSS 订阅更新** (共 {len(rss_items)} 条)\n\n" for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id if feeds_info and feed_id in feeds_info: feed_name = feeds_info[feed_id] text_content += f"**{feed_name}** ({len(items)} 条)\n" for i, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") if url: text_content += f" {i}. [{title}]({url})" else: text_content += f" {i}. {title}" if published_at: text_content += f" `{published_at}`" text_content += "\n" text_content += "\n" text_content += f"更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" return text_content # === RSS 内容渲染辅助函数（用于合并推送） === def _render_rss_section_feishu(rss_items: list, separator: str = "---") -> str: """渲染 RSS 内容区块（飞书格式，用于合并推送）""" if not rss_items: return "" # 按 feed_id 分组 feeds_map: Dict[str, list] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) text_content = f"📰 **RSS 订阅更新** (共 {len(rss_items)} 条)\n\n" for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id text_content += f"**{feed_name}** ({len(items)} 条)\n\n" for i, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") if url: text_content += f" {i}. [{title}]({url})" else: text_content += f" {i}. {title}" if published_at: text_content += f" - {published_at}" text_content += "\n" if i < len(items): text_content += "\n" text_content += "\n" return text_content.rstrip("\n") def _render_rss_section_markdown(rss_items: list) -> str: """渲染 RSS 内容区块（通用 Markdown 格式，用于合并推送）""" if not rss_items: return "" # 按 feed_id 分组 feeds_map: Dict[str, list] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) text_content = f"📰 **RSS 订阅更新** (共 {len(rss_items)} 条)\n\n" for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id text_content += f"**{feed_name}** ({len(items)} 条)\n" for i, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") if url: text_content += f" {i}. [{title}]({url})" else: text_content += f" {i}. {title}" if published_at: text_content += f" `{published_at}`" text_content += "\n" text_content += "\n" return text_content.rstrip("\n") ================================================ FILE: trendradar/notification/senders.py ================================================ # coding=utf-8 """ 消息发送器模块 将报告数据发送到各种通知渠道： - 飞书 (Feishu/Lark) - 钉钉 (DingTalk) - 企业微信 (WeCom/WeWork) - Telegram - 邮件 (Email) - ntfy - Bark - Slack 每个发送函数都支持分批发送，并通过参数化配置实现与 CONFIG 的解耦。 """ import smtplib import time import json from datetime import datetime from email.header import Header from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.utils import formataddr, formatdate, make_msgid from pathlib import Path from typing import Any, Callable, Dict, Optional from urllib.parse import urlparse import requests from .batch import add_batch_headers, get_max_batch_header_size from .formatters import convert_markdown_to_mrkdwn, strip_markdown def _render_ai_analysis(ai_analysis: Any, channel: str) -> str: """渲染 AI 分析内容为指定渠道格式""" if not ai_analysis: return "" try: from trendradar.ai.formatter import get_ai_analysis_renderer renderer = get_ai_analysis_renderer(channel) return renderer(ai_analysis) except ImportError: return "" # === SMTP 邮件配置 === SMTP_CONFIGS = { # Gmail（使用 STARTTLS） "gmail.com": {"server": "smtp.gmail.com", "port": 587, "encryption": "TLS"}, # QQ邮箱（使用 SSL，更稳定） "qq.com": {"server": "smtp.qq.com", "port": 465, "encryption": "SSL"}, # Outlook（使用 STARTTLS） "outlook.com": {"server": "smtp-mail.outlook.com", "port": 587, "encryption": "TLS"}, "hotmail.com": {"server": "smtp-mail.outlook.com", "port": 587, "encryption": "TLS"}, "live.com": {"server": "smtp-mail.outlook.com", "port": 587, "encryption": "TLS"}, # 网易邮箱（使用 SSL，更稳定） "163.com": {"server": "smtp.163.com", "port": 465, "encryption": "SSL"}, "126.com": {"server": "smtp.126.com", "port": 465, "encryption": "SSL"}, # 新浪邮箱（使用 SSL） "sina.com": {"server": "smtp.sina.com", "port": 465, "encryption": "SSL"}, # 搜狐邮箱（使用 SSL） "sohu.com": {"server": "smtp.sohu.com", "port": 465, "encryption": "SSL"}, # 天翼邮箱（使用 SSL） "189.cn": {"server": "smtp.189.cn", "port": 465, "encryption": "SSL"}, # 阿里云邮箱（使用 TLS） "aliyun.com": {"server": "smtp.aliyun.com", "port": 465, "encryption": "TLS"}, # Yandex邮箱（使用 TLS） "yandex.com": {"server": "smtp.yandex.com", "port": 465, "encryption": "TLS"}, # iCloud邮箱（使用 SSL） "icloud.com": {"server": "smtp.mail.me.com", "port": 587, "encryption": "SSL"}, } def send_to_feishu( webhook_url: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 29000, batch_interval: float = 1.0, split_content_func: Callable = None, get_time_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到飞书（支持分批发送，支持热榜+RSS合并+独立展示区） Args: webhook_url: 飞书 Webhook URL report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 get_time_func: 获取当前时间的函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ headers = {"Content-Type": "application/json"} proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"飞书{account_label}" if account_label else "飞书" # 渲染 AI 分析内容（如果有） ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "feishu") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 预留批次头部空间，避免添加头部后超限 header_reserve = get_max_batch_header_size("feishu") batches = split_content_func( report_data, "feishu", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "feishu", batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) # 飞书 webhook 只显示 content.text，所有信息都整合到 text 中 payload = { "msg_type": "interactive", "content": { "text": batch_content, }, } try: response = requests.post( webhook_url, headers=headers, json=payload, proxies=proxies, timeout=30 ) if response.status_code == 200: result = response.json() # 检查飞书的响应状态 if result.get("StatusCode") == 0 or result.get("code") == 0: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") # 批次间间隔 if i < len(batches): time.sleep(batch_interval) else: error_msg = result.get("msg") or result.get("StatusMessage", "未知错误") print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，错误：{error_msg}" ) return False else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True def send_to_dingtalk( webhook_url: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 20000, batch_interval: float = 1.0, split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到钉钉（支持分批发送，支持热榜+RSS合并+独立展示区） Args: webhook_url: 钉钉 Webhook URL report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ headers = {"Content-Type": "application/json"} proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"钉钉{account_label}" if account_label else "钉钉" # 渲染 AI 分析内容（如果有） ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "dingtalk") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 预留批次头部空间，避免添加头部后超限 header_reserve = get_max_batch_header_size("dingtalk") batches = split_content_func( report_data, "dingtalk", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "dingtalk", batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) payload = { "msgtype": "markdown", "markdown": { "title": f"TrendRadar 热点分析报告 - {report_type}", "text": batch_content, }, } try: response = requests.post( webhook_url, headers=headers, json=payload, proxies=proxies, timeout=30 ) if response.status_code == 200: result = response.json() if result.get("errcode") == 0: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") # 批次间间隔 if i < len(batches): time.sleep(batch_interval) else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，错误：{result.get('errmsg')}" ) return False else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True def send_to_wework( webhook_url: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 4000, batch_interval: float = 1.0, msg_type: str = "markdown", split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到企业微信（支持分批发送，支持 markdown 和 text 两种格式，支持热榜+RSS合并+独立展示区） Args: webhook_url: 企业微信 Webhook URL report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） msg_type: 消息类型 (markdown/text) split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ headers = {"Content-Type": "application/json"} proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"企业微信{account_label}" if account_label else "企业微信" # 获取消息类型配置（markdown 或 text） is_text_mode = msg_type.lower() == "text" if is_text_mode: print(f"{log_prefix}使用 text 格式（个人微信模式）[{report_type}]") else: print(f"{log_prefix}使用 markdown 格式（群机器人模式）[{report_type}]") # text 模式使用 wework_text，markdown 模式使用 wework header_format_type = "wework_text" if is_text_mode else "wework" # 渲染 AI 分析内容（如果有） ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "wework") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 获取分批内容，预留批次头部空间 header_reserve = get_max_batch_header_size(header_format_type) batches = split_content_func( report_data, "wework", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, header_format_type, batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): # 根据消息类型构建 payload if is_text_mode: # text 格式：去除 markdown 语法 plain_content = strip_markdown(batch_content) payload = {"msgtype": "text", "text": {"content": plain_content}} content_size = len(plain_content.encode("utf-8")) else: # markdown 格式：保持原样 payload = {"msgtype": "markdown", "markdown": {"content": batch_content}} content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) try: response = requests.post( webhook_url, headers=headers, json=payload, proxies=proxies, timeout=30 ) if response.status_code == 200: result = response.json() if result.get("errcode") == 0: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") # 批次间间隔 if i < len(batches): time.sleep(batch_interval) else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，错误：{result.get('errmsg')}" ) return False else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True def send_to_telegram( bot_token: str, chat_id: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 4000, batch_interval: float = 1.0, split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到 Telegram（支持分批发送，支持热榜+RSS合并+独立展示区） Args: bot_token: Telegram Bot Token chat_id: Telegram Chat ID report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ headers = {"Content-Type": "application/json"} url = f"https://api.telegram.org/bot{bot_token}/sendMessage" proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"Telegram{account_label}" if account_label else "Telegram" # 渲染 AI 分析内容（如果有） ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "telegram") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 获取分批内容，预留批次头部空间 header_reserve = get_max_batch_header_size("telegram") batches = split_content_func( report_data, "telegram", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "telegram", batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) payload = { "chat_id": chat_id, "text": batch_content, "parse_mode": "HTML", "disable_web_page_preview": True, } try: response = requests.post( url, headers=headers, json=payload, proxies=proxies, timeout=30 ) if response.status_code == 200: result = response.json() if result.get("ok"): print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") # 批次间间隔 if i < len(batches): time.sleep(batch_interval) else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，错误：{result.get('description')}" ) return False else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True def send_to_email( from_email: str, password: str, to_email: str, report_type: str, html_file_path: str, custom_smtp_server: Optional[str] = None, custom_smtp_port: Optional[int] = None, *, get_time_func: Callable = None, ) -> bool: """ 发送邮件通知 Args: from_email: 发件人邮箱 password: 邮箱密码/授权码 to_email: 收件人邮箱（多个用逗号分隔） report_type: 报告类型 html_file_path: HTML 报告文件路径 custom_smtp_server: 自定义 SMTP 服务器（可选） custom_smtp_port: 自定义 SMTP 端口（可选） get_time_func: 获取当前时间的函数 Returns: bool: 发送是否成功 Note: AI 分析内容已在 HTML 生成时嵌入，无需再追加 """ try: if not html_file_path or not Path(html_file_path).exists(): print(f"错误：HTML文件不存在或未提供: {html_file_path}") return False print(f"使用HTML文件: {html_file_path}") with open(html_file_path, "r", encoding="utf-8") as f: html_content = f.read() domain = from_email.split("@")[-1].lower() if custom_smtp_server and custom_smtp_port: # 使用自定义 SMTP 配置 smtp_server = custom_smtp_server smtp_port = int(custom_smtp_port) # 根据端口判断加密方式：465=SSL, 587=TLS if smtp_port == 465: use_tls = False # SSL 模式（SMTP_SSL） elif smtp_port == 587: use_tls = True # TLS 模式（STARTTLS） else: # 其他端口优先尝试 TLS（更安全，更广泛支持） use_tls = True elif domain in SMTP_CONFIGS: # 使用预设配置 config = SMTP_CONFIGS[domain] smtp_server = config["server"] smtp_port = config["port"] use_tls = config["encryption"] == "TLS" else: print(f"未识别的邮箱服务商: {domain}，使用通用 SMTP 配置") smtp_server = f"smtp.{domain}" smtp_port = 587 use_tls = True msg = MIMEMultipart("alternative") # 严格按照 RFC 标准设置 From header sender_name = "TrendRadar" msg["From"] = formataddr((sender_name, from_email)) # 设置收件人 recipients = [addr.strip() for addr in to_email.split(",")] if len(recipients) == 1: msg["To"] = recipients[0] else: msg["To"] = ", ".join(recipients) # 设置邮件主题 now = get_time_func() if get_time_func else datetime.now() subject = f"TrendRadar 热点分析报告 - {report_type} - {now.strftime('%m月%d日 %H:%M')}" msg["Subject"] = Header(subject, "utf-8") # 设置其他标准 header msg["MIME-Version"] = "1.0" msg["Date"] = formatdate(localtime=True) msg["Message-ID"] = make_msgid() # 添加纯文本部分（作为备选） text_content = f""" TrendRadar 热点分析报告 ======================== 报告类型：{report_type} 生成时间：{now.strftime('%Y-%m-%d %H:%M:%S')} 请使用支持HTML的邮件客户端查看完整报告内容。 """ text_part = MIMEText(text_content, "plain", "utf-8") msg.attach(text_part) html_part = MIMEText(html_content, "html", "utf-8") msg.attach(html_part) print(f"正在发送邮件到 {to_email}...") print(f"SMTP 服务器: {smtp_server}:{smtp_port}") print(f"发件人: {from_email}") try: if use_tls: # TLS 模式 server = smtplib.SMTP(smtp_server, smtp_port, timeout=30) server.set_debuglevel(0) # 设为1可以查看详细调试信息 server.ehlo() server.starttls() server.ehlo() else: # SSL 模式 server = smtplib.SMTP_SSL(smtp_server, smtp_port, timeout=30) server.set_debuglevel(0) server.ehlo() # 登录 server.login(from_email, password) # 发送邮件 server.send_message(msg) server.quit() print(f"邮件发送成功 [{report_type}] -> {to_email}") return True except smtplib.SMTPServerDisconnected: print("邮件发送失败：服务器意外断开连接，请检查网络或稍后重试") return False except smtplib.SMTPAuthenticationError as e: print("邮件发送失败：认证错误，请检查邮箱和密码/授权码") print(f"详细错误: {str(e)}") return False except smtplib.SMTPRecipientsRefused as e: print(f"邮件发送失败：收件人地址被拒绝 {e}") return False except smtplib.SMTPSenderRefused as e: print(f"邮件发送失败：发件人地址被拒绝 {e}") return False except smtplib.SMTPDataError as e: print(f"邮件发送失败：邮件数据错误 {e}") return False except smtplib.SMTPConnectError as e: print(f"邮件发送失败：无法连接到 SMTP 服务器 {smtp_server}:{smtp_port}") print(f"详细错误: {str(e)}") return False except Exception as e: print(f"邮件发送失败 [{report_type}]：{e}") import traceback traceback.print_exc() return False def send_to_ntfy( server_url: str, topic: str, token: Optional[str], report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 3800, split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到 ntfy（支持分批发送，严格遵守4KB限制，支持热榜+RSS合并+独立展示区） Args: server_url: ntfy 服务器 URL topic: ntfy 主题 token: ntfy 访问令牌（可选） report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ # 日志前缀 log_prefix = f"ntfy{account_label}" if account_label else "ntfy" # 避免 HTTP header 编码问题 report_type_en_map = { "全天汇总": "Daily Summary", "当前榜单": "Current Ranking", "增量分析": "Incremental Update", "通知连通性测试": "Notification Test", } report_type_en = report_type_en_map.get(report_type, "News Report") headers = { "Content-Type": "text/plain; charset=utf-8", "Markdown": "yes", "Title": report_type_en, "Priority": "default", "Tags": "news", } if token: headers["Authorization"] = f"Bearer {token}" # 构建完整URL，确保格式正确 base_url = server_url.rstrip("/") if not base_url.startswith(("http://", "https://")): base_url = f"https://{base_url}" url = f"{base_url}/{topic}" proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 渲染 AI 分析内容（如果有），合并到主内容中 ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "ntfy") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 获取分批内容，预留批次头部空间 header_reserve = get_max_batch_header_size("ntfy") batches = split_content_func( report_data, "ntfy", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "ntfy", batch_size) total_batches = len(batches) print(f"{log_prefix}消息分为 {total_batches} 批次发送 [{report_type}]") # 反转批次顺序，使得在ntfy客户端显示时顺序正确 # ntfy显示最新消息在上面，所以我们从最后一批开始推送 reversed_batches = list(reversed(batches)) print(f"{log_prefix}将按反向顺序推送（最后批次先推送），确保客户端显示顺序正确") # 逐批发送（反向顺序） success_count = 0 for idx, batch_content in enumerate(reversed_batches, 1): # 计算正确的批次编号（用户视角的编号） actual_batch_num = total_batches - idx + 1 content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {actual_batch_num}/{total_batches} 批次（推送顺序: {idx}/{total_batches}），大小：{content_size} 字节 [{report_type}]" ) # 检查消息大小，确保不超过4KB if content_size > 4096: print(f"警告：{log_prefix}第 {actual_batch_num} 批次消息过大（{content_size} 字节），可能被拒绝") # 更新 headers 的批次标识 current_headers = headers.copy() if total_batches > 1: current_headers["Title"] = f"{report_type_en} ({actual_batch_num}/{total_batches})" try: response = requests.post( url, headers=current_headers, data=batch_content.encode("utf-8"), proxies=proxies, timeout=30, ) if response.status_code == 200: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送成功 [{report_type}]") success_count += 1 if idx < total_batches: # 公共服务器建议 2-3 秒，自托管可以更短 interval = 2 if "ntfy.sh" in server_url else 1 time.sleep(interval) elif response.status_code == 429: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次速率限制 [{report_type}]，等待后重试" ) time.sleep(10) # 等待10秒后重试 # 重试一次 retry_response = requests.post( url, headers=current_headers, data=batch_content.encode("utf-8"), proxies=proxies, timeout=30, ) if retry_response.status_code == 200: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次重试成功 [{report_type}]") success_count += 1 else: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次重试失败，状态码：{retry_response.status_code}" ) elif response.status_code == 413: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次消息过大被拒绝 [{report_type}]，消息大小：{content_size} 字节" ) else: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) try: print(f"错误详情：{response.text}") except: pass except requests.exceptions.ConnectTimeout: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次连接超时 [{report_type}]") except requests.exceptions.ReadTimeout: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次读取超时 [{report_type}]") except requests.exceptions.ConnectionError as e: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次连接错误 [{report_type}]：{e}") except Exception as e: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送异常 [{report_type}]：{e}") # 判断整体发送是否成功 if success_count == total_batches: print(f"{log_prefix}所有 {total_batches} 批次发送完成 [{report_type}]") elif success_count > 0: print(f"{log_prefix}部分发送成功：{success_count}/{total_batches} 批次 [{report_type}]") else: print(f"{log_prefix}发送完全失败 [{report_type}]") return False return True def send_to_bark( bark_url: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 3600, batch_interval: float = 1.0, split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到 Bark（支持分批发送，使用 markdown 格式，支持热榜+RSS合并+独立展示区） Args: bark_url: Bark URL（包含 device_key） report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ # 日志前缀 log_prefix = f"Bark{account_label}" if account_label else "Bark" proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 解析 Bark URL，提取 device_key 和 API 端点 # Bark URL 格式: https://api.day.app/device_key 或 https://bark.day.app/device_key parsed_url = urlparse(bark_url) device_key = parsed_url.path.strip('/').split('/')[0] if parsed_url.path else None if not device_key: print(f"{log_prefix} URL 格式错误，无法提取 device_key: {bark_url}") return False # 构建正确的 API 端点 api_endpoint = f"{parsed_url.scheme}://{parsed_url.netloc}/push" # 渲染 AI 分析内容（如果有），合并到主内容中 ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "bark") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 获取分批内容，预留批次头部空间 header_reserve = get_max_batch_header_size("bark") batches = split_content_func( report_data, "bark", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "bark", batch_size) total_batches = len(batches) print(f"{log_prefix}消息分为 {total_batches} 批次发送 [{report_type}]") # 反转批次顺序，使得在Bark客户端显示时顺序正确 # Bark显示最新消息在上面，所以我们从最后一批开始推送 reversed_batches = list(reversed(batches)) print(f"{log_prefix}将按反向顺序推送（最后批次先推送），确保客户端显示顺序正确") # 逐批发送（反向顺序） success_count = 0 for idx, batch_content in enumerate(reversed_batches, 1): # 计算正确的批次编号（用户视角的编号） actual_batch_num = total_batches - idx + 1 content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {actual_batch_num}/{total_batches} 批次（推送顺序: {idx}/{total_batches}），大小：{content_size} 字节 [{report_type}]" ) # 检查消息大小（Bark使用APNs，限制4KB） if content_size > 4096: print( f"警告：{log_prefix}第 {actual_batch_num}/{total_batches} 批次消息过大（{content_size} 字节），可能被拒绝" ) # 构建JSON payload payload = { "title": report_type, "markdown": batch_content, "device_key": device_key, "sound": "default", "group": "TrendRadar", "action": "none", # 点击推送跳到 APP 不弹出弹框,方便阅读 } try: response = requests.post( api_endpoint, json=payload, proxies=proxies, timeout=30, ) if response.status_code == 200: result = response.json() if result.get("code") == 200: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送成功 [{report_type}]") success_count += 1 # 批次间间隔 if idx < total_batches: time.sleep(batch_interval) else: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送失败 [{report_type}]，错误：{result.get('message', '未知错误')}" ) else: print( f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送失败 [{report_type}]，状态码：{response.status_code}" ) try: print(f"错误详情：{response.text}") except: pass except requests.exceptions.ConnectTimeout: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次连接超时 [{report_type}]") except requests.exceptions.ReadTimeout: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次读取超时 [{report_type}]") except requests.exceptions.ConnectionError as e: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次连接错误 [{report_type}]：{e}") except Exception as e: print(f"{log_prefix}第 {actual_batch_num}/{total_batches} 批次发送异常 [{report_type}]：{e}") # 判断整体发送是否成功 if success_count == total_batches: print(f"{log_prefix}所有 {total_batches} 批次发送完成 [{report_type}]") elif success_count > 0: print(f"{log_prefix}部分发送成功：{success_count}/{total_batches} 批次 [{report_type}]") else: print(f"{log_prefix}发送完全失败 [{report_type}]") return False return True def send_to_slack( webhook_url: str, report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 4000, batch_interval: float = 1.0, split_content_func: Callable = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到 Slack（支持分批发送，使用 mrkdwn 格式，支持热榜+RSS合并+独立展示区） Args: webhook_url: Slack Webhook URL report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ headers = {"Content-Type": "application/json"} proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"Slack{account_label}" if account_label else "Slack" # 渲染 AI 分析内容（如果有），合并到主内容中 ai_content = None ai_stats = None if ai_analysis: ai_content = _render_ai_analysis(ai_analysis, "slack") # 提取 AI 分析统计数据（只要 AI 分析成功就显示） if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), "ai_mode": getattr(ai_analysis, "ai_mode", ""), } # 获取分批内容，预留批次头部空间 header_reserve = get_max_batch_header_size("slack") batches = split_content_func( report_data, "slack", update_info, max_bytes=batch_size - header_reserve, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部（已预留空间，不会超限） batches = add_batch_headers(batches, "slack", batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): # 转换 Markdown 到 mrkdwn 格式 mrkdwn_content = convert_markdown_to_mrkdwn(batch_content) content_size = len(mrkdwn_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) # 构建 Slack payload（使用简单的 text 字段，支持 mrkdwn） payload = {"text": mrkdwn_content} try: response = requests.post( webhook_url, headers=headers, json=payload, proxies=proxies, timeout=30 ) # Slack Incoming Webhooks 成功时返回 "ok" 文本 if response.status_code == 200 and response.text == "ok": print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") # 批次间间隔 if i < len(batches): time.sleep(batch_interval) else: error_msg = response.text if response.text else f"状态码：{response.status_code}" print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，错误：{error_msg}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True def send_to_generic_webhook( webhook_url: str, payload_template: Optional[str], report_data: Dict, report_type: str, update_info: Optional[Dict] = None, proxy_url: Optional[str] = None, mode: str = "daily", account_label: str = "", *, batch_size: int = 4000, batch_interval: float = 1.0, split_content_func: Optional[Callable] = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, ai_analysis: Any = None, display_regions: Optional[Dict] = None, standalone_data: Optional[Dict] = None, ) -> bool: """ 发送到通用 Webhook（支持分批发送，支持自定义 JSON 模板，支持热榜+RSS合并+独立展示区） Args: webhook_url: Webhook URL payload_template: JSON 模板字符串，支持 {title} 和 {content} 占位符 report_data: 报告数据 report_type: 报告类型 update_info: 更新信息（可选） proxy_url: 代理 URL（可选） mode: 报告模式 (daily/current) account_label: 账号标签（多账号时显示） batch_size: 批次大小（字节） batch_interval: 批次发送间隔（秒） split_content_func: 内容分批函数 rss_items: RSS 统计条目列表（可选，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） Returns: bool: 发送是否成功 """ if split_content_func is None: raise ValueError("split_content_func is required") headers = {"Content-Type": "application/json"} proxies = None if proxy_url: proxies = {"http": proxy_url, "https": proxy_url} # 日志前缀 log_prefix = f"通用Webhook{account_label}" if account_label else "通用Webhook" # 渲染 AI 分析内容（如果有） ai_content = None ai_stats = None if ai_analysis: # 通用 Webhook 使用 markdown 格式渲染 AI 分析 ai_content = _render_ai_analysis(ai_analysis, "wework") # 提取 AI 分析统计数据 if getattr(ai_analysis, "success", False): ai_stats = { "total_news": getattr(ai_analysis, "total_news", 0), "analyzed_news": getattr(ai_analysis, "analyzed_news", 0), "max_news_limit": getattr(ai_analysis, "max_news_limit", 0), "hotlist_count": getattr(ai_analysis, "hotlist_count", 0), "rss_count": getattr(ai_analysis, "rss_count", 0), } # 获取分批内容 # 使用 'wework' 作为 format_type 以获取 markdown 格式的通用输出 # 预留一定空间给模板外壳 template_overhead = 200 batches = split_content_func( report_data, "wework", update_info, max_bytes=batch_size - template_overhead, mode=mode, rss_items=rss_items, rss_new_items=rss_new_items, ai_content=ai_content, standalone_data=standalone_data, ai_stats=ai_stats, report_type=report_type, ) # 统一添加批次头部 batches = add_batch_headers(batches, "wework", batch_size) print(f"{log_prefix}消息分为 {len(batches)} 批次发送 [{report_type}]") # 逐批发送 for i, batch_content in enumerate(batches, 1): content_size = len(batch_content.encode("utf-8")) print( f"发送{log_prefix}第 {i}/{len(batches)} 批次，大小：{content_size} 字节 [{report_type}]" ) try: # 构建 payload if payload_template: # 简单的字符串替换 # 注意：content 可能包含 JSON 特殊字符，需要先转义 json_content = json.dumps(batch_content)[1:-1] # 去掉首尾引号 json_title = json.dumps(report_type)[1:-1] payload_str = payload_template.replace("{content}", json_content).replace("{title}", json_title) # 尝试解析为 JSON 对象以验证有效性 try: payload = json.loads(payload_str) except json.JSONDecodeError as e: print(f"{log_prefix} JSON 模板解析失败: {e}") # 回退到默认格式 payload = {"title": report_type, "content": batch_content} else: # 默认格式 payload = {"title": report_type, "content": batch_content} response = requests.post( webhook_url, headers=headers, json=payload, proxies=proxies, timeout=30 ) if response.status_code >= 200 and response.status_code < 300: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送成功 [{report_type}]") if i < len(batches): time.sleep(batch_interval) else: print( f"{log_prefix}第 {i}/{len(batches)} 批次发送失败 [{report_type}]，状态码：{response.status_code}, 响应: {response.text}" ) return False except Exception as e: print(f"{log_prefix}第 {i}/{len(batches)} 批次发送出错 [{report_type}]：{e}") return False print(f"{log_prefix}所有 {len(batches)} 批次发送完成 [{report_type}]") return True ================================================ FILE: trendradar/notification/splitter.py ================================================ # coding=utf-8 """ 消息分批处理模块 提供消息内容分批拆分功能，确保消息大小不超过各平台限制 """ from datetime import datetime from typing import Dict, List, Optional, Callable from trendradar.report.formatter import format_title_for_platform from trendradar.report.helpers import format_rank_display from trendradar.utils.time import DEFAULT_TIMEZONE, format_iso_time_friendly, convert_time_for_display # 默认批次大小配置 DEFAULT_BATCH_SIZES = { "dingtalk": 20000, "feishu": 29000, "ntfy": 3800, "default": 4000, } # 默认区域顺序 DEFAULT_REGION_ORDER = ["hotlist", "rss", "new_items", "standalone", "ai_analysis"] def split_content_into_batches( report_data: Dict, format_type: str, update_info: Optional[Dict] = None, max_bytes: Optional[int] = None, mode: str = "daily", batch_sizes: Optional[Dict[str, int]] = None, feishu_separator: str = "---", region_order: Optional[List[str]] = None, get_time_func: Optional[Callable[[], datetime]] = None, rss_items: Optional[list] = None, rss_new_items: Optional[list] = None, timezone: str = DEFAULT_TIMEZONE, display_mode: str = "keyword", ai_content: Optional[str] = None, standalone_data: Optional[Dict] = None, rank_threshold: int = 10, ai_stats: Optional[Dict] = None, report_type: str = "热点分析报告", show_new_section: bool = True, ) -> List[str]: """分批处理消息内容，确保词组标题+至少第一条新闻的完整性（支持热榜+RSS合并+AI分析+独立展示区） 热榜统计与RSS统计并列显示，热榜新增与RSS新增并列显示。 region_order 控制各区域的显示顺序。 AI分析内容根据 region_order 中的位置显示。 独立展示区根据 region_order 中的位置显示。 Args: report_data: 报告数据字典，包含 stats, new_titles, failed_ids, total_new_count format_type: 格式类型 (feishu, dingtalk, wework, telegram, ntfy, bark, slack) update_info: 版本更新信息（可选） max_bytes: 最大字节数（可选，如果不指定则使用默认配置） mode: 报告模式 (daily, incremental, current) batch_sizes: 批次大小配置字典（可选） feishu_separator: 飞书消息分隔符 region_order: 区域显示顺序列表 get_time_func: 获取当前时间的函数（可选） rss_items: RSS 统计条目列表（按源分组，用于合并推送） rss_new_items: RSS 新增条目列表（可选，用于新增区块） timezone: 时区名称（用于 RSS 时间格式化） display_mode: 显示模式 (keyword=按关键词分组, platform=按平台分组) ai_content: AI 分析内容（已渲染的字符串，可选） standalone_data: 独立展示区数据（可选），包含 platforms 和 rss_feeds 列表 ai_stats: AI 分析统计数据（可选），包含 total_news, analyzed_news, max_news_limit 等 Returns: 分批后的消息内容列表 """ if region_order is None: region_order = DEFAULT_REGION_ORDER # 合并批次大小配置 sizes = {**DEFAULT_BATCH_SIZES, **(batch_sizes or {})} if max_bytes is None: if format_type == "dingtalk": max_bytes = sizes.get("dingtalk", 20000) elif format_type == "feishu": max_bytes = sizes.get("feishu", 29000) elif format_type == "ntfy": max_bytes = sizes.get("ntfy", 3800) else: max_bytes = sizes.get("default", 4000) batches = [] total_hotlist_count = sum( len(stat["titles"]) for stat in report_data["stats"] if stat["count"] > 0 ) total_titles = total_hotlist_count # 累加 RSS 条目数 if rss_items: total_titles += sum(stat.get("count", 0) for stat in rss_items) now = get_time_func() if get_time_func else datetime.now() # 构建头部信息 base_header = "" # 准备 AI 分析统计行（如果存在） ai_stats_line = "" if ai_stats and ai_stats.get("analyzed_news", 0) > 0: analyzed_news = ai_stats.get("analyzed_news", 0) total_news = ai_stats.get("total_news", 0) ai_mode = ai_stats.get("ai_mode", "") # 构建分析数显示：如果被截断则显示 "实际分析数/总可分析数" if total_news > analyzed_news: news_display = f"{analyzed_news}/{total_news}" else: news_display = str(analyzed_news) # 如果 AI 模式与推送模式不同，显示模式标识 mode_suffix = "" if ai_mode and ai_mode != mode: mode_map = { "daily": "全天汇总", "current": "当前榜单", "incremental": "增量分析" } mode_label = mode_map.get(ai_mode, ai_mode) mode_suffix = f" ({mode_label})" if format_type in ("wework", "bark", "ntfy", "feishu", "dingtalk"): ai_stats_line = f"**AI 分析数：** {news_display}{mode_suffix}\n" elif format_type == "slack": ai_stats_line = f"*AI 分析数：* {news_display}{mode_suffix}\n" elif format_type == "telegram": ai_stats_line = f"AI 分析数： {news_display}{mode_suffix}\n" # 构建统一的头部（总是显示总新闻数、时间和类型） if format_type in ("wework", "bark"): base_header = f"**总新闻数：** {total_titles}\n" base_header += ai_stats_line base_header += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"**类型：** {report_type}\n\n" elif format_type == "telegram": base_header = f"总新闻数： {total_titles}\n" base_header += ai_stats_line base_header += f"时间： {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"类型： {report_type}\n\n" elif format_type == "ntfy": base_header = f"**总新闻数：** {total_titles}\n" base_header += ai_stats_line base_header += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"**类型：** {report_type}\n\n" elif format_type == "feishu": base_header = f"**总新闻数：** {total_titles}\n" base_header += ai_stats_line base_header += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"**类型：** {report_type}\n\n" base_header += "---\n\n" elif format_type == "dingtalk": base_header = f"**总新闻数：** {total_titles}\n" base_header += ai_stats_line base_header += f"**时间：** {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"**类型：** {report_type}\n\n" base_header += "---\n\n" elif format_type == "slack": base_header = f"*总新闻数：* {total_titles}\n" base_header += ai_stats_line base_header += f"*时间：* {now.strftime('%Y-%m-%d %H:%M:%S')}\n" base_header += f"*类型：* {report_type}\n\n" base_footer = "" if format_type in ("wework", "bark"): base_footer = f"\n\n\n> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: base_footer += f"\n> TrendRadar 发现新版本 **{update_info['remote_version']}**，当前 **{update_info['current_version']}**" elif format_type == "telegram": base_footer = f"\n\n更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: base_footer += f"\nTrendRadar 发现新版本 {update_info['remote_version']}，当前 {update_info['current_version']}" elif format_type == "ntfy": base_footer = f"\n\n> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: base_footer += f"\n> TrendRadar 发现新版本 **{update_info['remote_version']}**，当前 **{update_info['current_version']}**" elif format_type == "feishu": base_footer = f"\n\n更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: base_footer += f"\nTrendRadar 发现新版本 {update_info['remote_version']}，当前 {update_info['current_version']}" elif format_type == "dingtalk": base_footer = f"\n\n> 更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}" if update_info: base_footer += f"\n> TrendRadar 发现新版本 **{update_info['remote_version']}**，当前 **{update_info['current_version']}**" elif format_type == "slack": base_footer = f"\n\n_更新时间：{now.strftime('%Y-%m-%d %H:%M:%S')}_" if update_info: base_footer += f"\n_TrendRadar 发现新版本 *{update_info['remote_version']}*，当前 *{update_info['current_version']}_" # 根据 display_mode 选择统计标题 stats_title = "热点词汇统计" if display_mode == "keyword" else "热点新闻统计" stats_header = "" if report_data["stats"]: if format_type in ("wework", "bark"): stats_header = f"📊 **{stats_title}** (共 {total_hotlist_count} 条)\n\n" elif format_type == "telegram": stats_header = f"📊 {stats_title} (共 {total_hotlist_count} 条)\n\n" elif format_type == "ntfy": stats_header = f"📊 **{stats_title}** (共 {total_hotlist_count} 条)\n\n" elif format_type == "feishu": stats_header = f"📊 **{stats_title}** (共 {total_hotlist_count} 条)\n\n" elif format_type == "dingtalk": stats_header = f"📊 **{stats_title}** (共 {total_hotlist_count} 条)\n\n" elif format_type == "slack": stats_header = f"📊 *{stats_title}* (共 {total_hotlist_count} 条)\n\n" current_batch = base_header current_batch_has_content = False # 当没有热榜数据时的处理 # 注意：如果有 ai_content，不应该返回"暂无匹配"消息，而应该继续处理 AI 内容 if ( not report_data["stats"] and not report_data["new_titles"] and not report_data["failed_ids"] and not ai_content # 有 AI 内容时不返回"暂无匹配" and not rss_items # 有 RSS 内容时也不返回 and not standalone_data # 有独立展示区数据时也不返回 ): if mode == "incremental": mode_text = "增量模式下暂无新增匹配的热点词汇" elif mode == "current": mode_text = "当前榜单模式下暂无匹配的热点词汇" else: mode_text = "暂无匹配的热点词汇" simple_content = f"📭 {mode_text}\n\n" final_content = base_header + simple_content + base_footer batches.append(final_content) return batches # 定义处理热点词汇统计的函数 def process_stats_section(current_batch, current_batch_has_content, batches, add_separator=True): """处理热点词汇统计""" if not report_data["stats"]: return current_batch, current_batch_has_content, batches total_count = len(report_data["stats"]) # 根据 add_separator 决定是否添加前置分割线 actual_stats_header = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type == "feishu": actual_stats_header = f"\n{feishu_separator}\n\n{stats_header}" elif format_type == "dingtalk": actual_stats_header = f"\n---\n\n{stats_header}" elif format_type in ("wework", "bark"): actual_stats_header = f"\n\n\n\n{stats_header}" else: actual_stats_header = f"\n\n{stats_header}" else: # 不需要分割线（第一个区域） actual_stats_header = stats_header # 添加统计标题 test_content = current_batch + actual_stats_header if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes ): current_batch = test_content current_batch_has_content = True else: if current_batch_has_content: batches.append(current_batch + base_footer) # 新批次开头不需要分割线，使用原始 stats_header current_batch = base_header + stats_header current_batch_has_content = True # 逐个处理词组（确保词组标题+第一条新闻的原子性） for i, stat in enumerate(report_data["stats"]): word = stat["word"] count = stat["count"] sequence_display = f"[{i + 1}/{total_count}]" # 构建词组标题 word_header = "" if format_type in ("wework", "bark"): if count >= 10: word_header = ( f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" ) elif count >= 5: word_header = ( f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" ) else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "telegram": if count >= 10: word_header = f"🔥 {sequence_display} {word} : {count} 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} {word} : {count} 条\n\n" else: word_header = f"📌 {sequence_display} {word} : {count} 条\n\n" elif format_type == "ntfy": if count >= 10: word_header = ( f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" ) elif count >= 5: word_header = ( f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" ) else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "feishu": if count >= 10: word_header = f"🔥 {sequence_display} **{word}** : {count} 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} **{word}** : {count} 条\n\n" else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "dingtalk": if count >= 10: word_header = ( f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" ) elif count >= 5: word_header = ( f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" ) else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "slack": if count >= 10: word_header = ( f"🔥 {sequence_display} *{word}* : *{count}* 条\n\n" ) elif count >= 5: word_header = ( f"📈 {sequence_display} *{word}* : *{count}* 条\n\n" ) else: word_header = f"📌 {sequence_display} *{word}* : {count} 条\n\n" # 构建第一条新闻 # display_mode: keyword=显示来源, platform=显示关键词 show_source = display_mode == "keyword" show_keyword = display_mode == "platform" first_news_line = "" if stat["titles"]: first_title_data = stat["titles"][0] if format_type in ("wework", "bark"): formatted_title = format_title_for_platform( "wework", first_title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "telegram": formatted_title = format_title_for_platform( "telegram", first_title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "ntfy": formatted_title = format_title_for_platform( "ntfy", first_title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "feishu": formatted_title = format_title_for_platform( "feishu", first_title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "dingtalk": formatted_title = format_title_for_platform( "dingtalk", first_title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "slack": formatted_title = format_title_for_platform( "slack", first_title_data, show_source=show_source, show_keyword=show_keyword ) else: formatted_title = f"{first_title_data['title']}" first_news_line = f" 1. {formatted_title}\n" if len(stat["titles"]) > 1: first_news_line += "\n" # 原子性检查：词组标题+第一条新闻必须一起处理 word_with_first_news = word_header + first_news_line test_content = current_batch + word_with_first_news if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): # 当前批次容纳不下，开启新批次 if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + stats_header + word_with_first_news current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余新闻条目 for j in range(start_index, len(stat["titles"])): title_data = stat["titles"][j] if format_type in ("wework", "bark"): formatted_title = format_title_for_platform( "wework", title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "telegram": formatted_title = format_title_for_platform( "telegram", title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "ntfy": formatted_title = format_title_for_platform( "ntfy", title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "feishu": formatted_title = format_title_for_platform( "feishu", title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "dingtalk": formatted_title = format_title_for_platform( "dingtalk", title_data, show_source=show_source, show_keyword=show_keyword ) elif format_type == "slack": formatted_title = format_title_for_platform( "slack", title_data, show_source=show_source, show_keyword=show_keyword ) else: formatted_title = f"{title_data['title']}" news_line = f" {j + 1}. {formatted_title}\n" if j < len(stat["titles"]) - 1: news_line += "\n" test_content = current_batch + news_line if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + stats_header + word_header + news_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 词组间分隔符 if i < len(report_data["stats"]) - 1: separator = "" if format_type in ("wework", "bark"): separator = f"\n\n\n\n" elif format_type == "telegram": separator = f"\n\n" elif format_type == "ntfy": separator = f"\n\n" elif format_type == "feishu": separator = f"\n{feishu_separator}\n\n" elif format_type == "dingtalk": separator = f"\n---\n\n" elif format_type == "slack": separator = f"\n\n" test_content = current_batch + separator if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes ): current_batch = test_content return current_batch, current_batch_has_content, batches # 定义处理新增新闻的函数 def process_new_titles_section(current_batch, current_batch_has_content, batches, add_separator=True): """处理新增新闻""" if not show_new_section or not report_data["new_titles"]: return current_batch, current_batch_has_content, batches # 根据 add_separator 决定是否添加前置分割线 new_header = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type in ("wework", "bark"): new_header = f"\n\n\n\n🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "telegram": new_header = ( f"\n\n🆕 本次新增热点新闻 (共 {report_data['total_new_count']} 条)\n\n" ) elif format_type == "ntfy": new_header = f"\n\n🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "feishu": new_header = f"\n{feishu_separator}\n\n🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "dingtalk": new_header = f"\n---\n\n🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "slack": new_header = f"\n\n🆕 *本次新增热点新闻* (共 {report_data['total_new_count']} 条)\n\n" else: # 不需要分割线（第一个区域） if format_type in ("wework", "bark"): new_header = f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "telegram": new_header = f"🆕 本次新增热点新闻 (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "ntfy": new_header = f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "feishu": new_header = f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "dingtalk": new_header = f"🆕 **本次新增热点新闻** (共 {report_data['total_new_count']} 条)\n\n" elif format_type == "slack": new_header = f"🆕 *本次新增热点新闻* (共 {report_data['total_new_count']} 条)\n\n" test_content = current_batch + new_header if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 逐个处理新增新闻来源 for source_data in report_data["new_titles"]: source_header = "" if format_type in ("wework", "bark"): source_header = f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n" elif format_type == "telegram": source_header = f"{source_data['source_name']} ({len(source_data['titles'])} 条):\n\n" elif format_type == "ntfy": source_header = f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n" elif format_type == "feishu": source_header = f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n" elif format_type == "dingtalk": source_header = f"**{source_data['source_name']}** ({len(source_data['titles'])} 条):\n\n" elif format_type == "slack": source_header = f"*{source_data['source_name']}* ({len(source_data['titles'])} 条):\n\n" # 构建第一条新增新闻 first_news_line = "" if source_data["titles"]: first_title_data = source_data["titles"][0] title_data_copy = first_title_data.copy() title_data_copy["is_new"] = False if format_type in ("wework", "bark"): formatted_title = format_title_for_platform( "wework", title_data_copy, show_source=False ) elif format_type == "telegram": formatted_title = format_title_for_platform( "telegram", title_data_copy, show_source=False ) elif format_type == "feishu": formatted_title = format_title_for_platform( "feishu", title_data_copy, show_source=False ) elif format_type == "dingtalk": formatted_title = format_title_for_platform( "dingtalk", title_data_copy, show_source=False ) elif format_type == "slack": formatted_title = format_title_for_platform( "slack", title_data_copy, show_source=False ) else: formatted_title = f"{title_data_copy['title']}" first_news_line = f" 1. {formatted_title}\n" # 原子性检查：来源标题+第一条新闻 source_with_first_news = source_header + first_news_line test_content = current_batch + source_with_first_news if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header + source_with_first_news current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余新增新闻 for j in range(start_index, len(source_data["titles"])): title_data = source_data["titles"][j] title_data_copy = title_data.copy() title_data_copy["is_new"] = False if format_type == "wework": formatted_title = format_title_for_platform( "wework", title_data_copy, show_source=False ) elif format_type == "telegram": formatted_title = format_title_for_platform( "telegram", title_data_copy, show_source=False ) elif format_type == "feishu": formatted_title = format_title_for_platform( "feishu", title_data_copy, show_source=False ) elif format_type == "dingtalk": formatted_title = format_title_for_platform( "dingtalk", title_data_copy, show_source=False ) elif format_type == "slack": formatted_title = format_title_for_platform( "slack", title_data_copy, show_source=False ) else: formatted_title = f"{title_data_copy['title']}" news_line = f" {j + 1}. {formatted_title}\n" test_content = current_batch + news_line if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header + source_header + news_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True current_batch += "\n" return current_batch, current_batch_has_content, batches # 定义处理 AI 分析的函数 def process_ai_section(current_batch, current_batch_has_content, batches, add_separator=True): """处理 AI 分析内容""" nonlocal ai_content if not ai_content: return current_batch, current_batch_has_content, batches # 根据 add_separator 决定是否添加前置分割线 ai_separator = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type == "feishu": ai_separator = f"\n{feishu_separator}\n\n" elif format_type == "dingtalk": ai_separator = "\n---\n\n" elif format_type in ("wework", "bark"): ai_separator = "\n\n\n\n" elif format_type in ("telegram", "ntfy", "slack"): ai_separator = "\n\n" # 如果不需要分割线，ai_separator 保持为空字符串 # 尝试将 AI 内容添加到当前批次 test_content = current_batch + ai_separator + ai_content if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes ): current_batch = test_content current_batch_has_content = True else: # 当前批次容纳不下，开启新批次 if current_batch_has_content: batches.append(current_batch + base_footer) # AI 内容可能很长，需要考虑是否需要进一步分割 ai_with_header = base_header + ai_content current_batch = ai_with_header current_batch_has_content = True return current_batch, current_batch_has_content, batches # 定义处理独立展示区的函数 def process_standalone_section_wrapper(current_batch, current_batch_has_content, batches, add_separator=True): """处理独立展示区""" if not standalone_data: return current_batch, current_batch_has_content, batches return _process_standalone_section( standalone_data, format_type, feishu_separator, base_header, base_footer, max_bytes, current_batch, current_batch_has_content, batches, timezone, rank_threshold, add_separator ) # 定义处理 RSS 统计的函数 def process_rss_stats_wrapper(current_batch, current_batch_has_content, batches, add_separator=True): """处理 RSS 统计""" if not rss_items: return current_batch, current_batch_has_content, batches return _process_rss_stats_section( rss_items, format_type, feishu_separator, base_header, base_footer, max_bytes, current_batch, current_batch_has_content, batches, timezone, add_separator ) # 定义处理 RSS 新增的函数 def process_rss_new_wrapper(current_batch, current_batch_has_content, batches, add_separator=True): """处理 RSS 新增""" if not rss_new_items: return current_batch, current_batch_has_content, batches return _process_rss_new_titles_section( rss_new_items, format_type, feishu_separator, base_header, base_footer, max_bytes, current_batch, current_batch_has_content, batches, timezone, add_separator ) # 按 region_order 顺序处理各区域 # 记录是否已有区域内容（用于决定是否添加分割线） has_region_content = False for region in region_order: # 记录处理前的状态，用于判断该区域是否产生了内容 batch_before = current_batch has_content_before = current_batch_has_content batches_len_before = len(batches) # 决定是否需要添加分割线（第一个有内容的区域不需要） add_separator = has_region_content if region == "hotlist": # 处理热榜统计 current_batch, current_batch_has_content, batches = process_stats_section( current_batch, current_batch_has_content, batches, add_separator ) elif region == "rss": # 处理 RSS 统计 current_batch, current_batch_has_content, batches = process_rss_stats_wrapper( current_batch, current_batch_has_content, batches, add_separator ) elif region == "new_items": # 处理热榜新增 current_batch, current_batch_has_content, batches = process_new_titles_section( current_batch, current_batch_has_content, batches, add_separator ) # 处理 RSS 新增（跟随 new_items，继承 add_separator 逻辑） # 如果热榜新增产生了内容，RSS 新增需要分割线 new_batch_changed = ( current_batch != batch_before or current_batch_has_content != has_content_before or len(batches) != batches_len_before ) rss_new_separator = new_batch_changed or has_region_content current_batch, current_batch_has_content, batches = process_rss_new_wrapper( current_batch, current_batch_has_content, batches, rss_new_separator ) elif region == "standalone": # 处理独立展示区 current_batch, current_batch_has_content, batches = process_standalone_section_wrapper( current_batch, current_batch_has_content, batches, add_separator ) elif region == "ai_analysis": # 处理 AI 分析 current_batch, current_batch_has_content, batches = process_ai_section( current_batch, current_batch_has_content, batches, add_separator ) # 检查该区域是否产生了内容 region_produced_content = ( current_batch != batch_before or current_batch_has_content != has_content_before or len(batches) != batches_len_before ) if region_produced_content: has_region_content = True if report_data["failed_ids"]: failed_header = "" if format_type == "wework": failed_header = f"\n\n\n\n⚠️ **数据获取失败的平台：**\n\n" elif format_type == "telegram": failed_header = f"\n\n⚠️ 数据获取失败的平台：\n\n" elif format_type == "ntfy": failed_header = f"\n\n⚠️ **数据获取失败的平台：**\n\n" elif format_type == "feishu": failed_header = f"\n{feishu_separator}\n\n⚠️ **数据获取失败的平台：**\n\n" elif format_type == "dingtalk": failed_header = f"\n---\n\n⚠️ **数据获取失败的平台：**\n\n" test_content = current_batch + failed_header if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + failed_header current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True for i, id_value in enumerate(report_data["failed_ids"], 1): if format_type == "feishu": failed_line = f" • {id_value}\n" elif format_type == "dingtalk": failed_line = f" • **{id_value}**\n" else: failed_line = f" • {id_value}\n" test_content = current_batch + failed_line if ( len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes ): if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + failed_header + failed_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 完成最后批次 if current_batch_has_content: batches.append(current_batch + base_footer) return batches def _process_rss_stats_section( rss_stats: list, format_type: str, feishu_separator: str, base_header: str, base_footer: str, max_bytes: int, current_batch: str, current_batch_has_content: bool, batches: List[str], timezone: str = DEFAULT_TIMEZONE, add_separator: bool = True, ) -> tuple: """处理 RSS 统计区块（按关键词分组，与热榜统计格式一致） Args: rss_stats: RSS 关键词统计列表，格式与热榜 stats 一致： [{"word": "AI", "count": 5, "titles": [...]}] format_type: 格式类型 feishu_separator: 飞书分隔符 base_header: 基础头部 base_footer: 基础尾部 max_bytes: 最大字节数 current_batch: 当前批次内容 current_batch_has_content: 当前批次是否有内容 batches: 已完成的批次列表 timezone: 时区名称 add_separator: 是否在区块前添加分割线（第一个区域时为 False） Returns: (current_batch, current_batch_has_content, batches) 元组 """ if not rss_stats: return current_batch, current_batch_has_content, batches # 计算总条目数 total_items = sum(stat["count"] for stat in rss_stats) total_keywords = len(rss_stats) # RSS 统计区块标题（根据 add_separator 决定是否添加前置分割线） rss_header = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type == "feishu": rss_header = f"\n{feishu_separator}\n\n📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": rss_header = f"\n---\n\n📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" elif format_type in ("wework", "bark"): rss_header = f"\n\n\n\n📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" elif format_type == "telegram": rss_header = f"\n\n📰 RSS 订阅统计 (共 {total_items} 条)\n\n" elif format_type == "slack": rss_header = f"\n\n📰 *RSS 订阅统计* (共 {total_items} 条)\n\n" else: rss_header = f"\n\n📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" else: # 不需要分割线（第一个区域） if format_type == "feishu": rss_header = f"📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": rss_header = f"📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" elif format_type == "telegram": rss_header = f"📰 RSS 订阅统计 (共 {total_items} 条)\n\n" elif format_type == "slack": rss_header = f"📰 *RSS 订阅统计* (共 {total_items} 条)\n\n" else: rss_header = f"📰 **RSS 订阅统计** (共 {total_items} 条)\n\n" # 添加 RSS 标题 test_content = current_batch + rss_header if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes: current_batch = test_content current_batch_has_content = True else: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + rss_header current_batch_has_content = True # 逐个处理关键词组（与热榜一致） for i, stat in enumerate(rss_stats): word = stat["word"] count = stat["count"] sequence_display = f"[{i + 1}/{total_keywords}]" # 构建关键词标题（与热榜格式一致） word_header = "" if format_type in ("wework", "bark"): if count >= 10: word_header = f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "telegram": if count >= 10: word_header = f"🔥 {sequence_display} {word} : {count} 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} {word} : {count} 条\n\n" else: word_header = f"📌 {sequence_display} {word} : {count} 条\n\n" elif format_type == "ntfy": if count >= 10: word_header = f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "feishu": if count >= 10: word_header = f"🔥 {sequence_display} **{word}** : {count} 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} **{word}** : {count} 条\n\n" else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "dingtalk": if count >= 10: word_header = f"🔥 {sequence_display} **{word}** : **{count}** 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} **{word}** : **{count}** 条\n\n" else: word_header = f"📌 {sequence_display} **{word}** : {count} 条\n\n" elif format_type == "slack": if count >= 10: word_header = f"🔥 {sequence_display} *{word}* : *{count}* 条\n\n" elif count >= 5: word_header = f"📈 {sequence_display} *{word}* : *{count}* 条\n\n" else: word_header = f"📌 {sequence_display} *{word}* : {count} 条\n\n" # 构建第一条新闻（使用 format_title_for_platform） first_news_line = "" if stat["titles"]: first_title_data = stat["titles"][0] if format_type in ("wework", "bark"): formatted_title = format_title_for_platform("wework", first_title_data, show_source=True) elif format_type == "telegram": formatted_title = format_title_for_platform("telegram", first_title_data, show_source=True) elif format_type == "ntfy": formatted_title = format_title_for_platform("ntfy", first_title_data, show_source=True) elif format_type == "feishu": formatted_title = format_title_for_platform("feishu", first_title_data, show_source=True) elif format_type == "dingtalk": formatted_title = format_title_for_platform("dingtalk", first_title_data, show_source=True) elif format_type == "slack": formatted_title = format_title_for_platform("slack", first_title_data, show_source=True) else: formatted_title = f"{first_title_data['title']}" first_news_line = f" 1. {formatted_title}\n" if len(stat["titles"]) > 1: first_news_line += "\n" # 原子性检查：关键词标题 + 第一条新闻必须一起处理 word_with_first_news = word_header + first_news_line test_content = current_batch + word_with_first_news if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + rss_header + word_with_first_news current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余新闻条目 for j in range(start_index, len(stat["titles"])): title_data = stat["titles"][j] if format_type in ("wework", "bark"): formatted_title = format_title_for_platform("wework", title_data, show_source=True) elif format_type == "telegram": formatted_title = format_title_for_platform("telegram", title_data, show_source=True) elif format_type == "ntfy": formatted_title = format_title_for_platform("ntfy", title_data, show_source=True) elif format_type == "feishu": formatted_title = format_title_for_platform("feishu", title_data, show_source=True) elif format_type == "dingtalk": formatted_title = format_title_for_platform("dingtalk", title_data, show_source=True) elif format_type == "slack": formatted_title = format_title_for_platform("slack", title_data, show_source=True) else: formatted_title = f"{title_data['title']}" news_line = f" {j + 1}. {formatted_title}\n" if j < len(stat["titles"]) - 1: news_line += "\n" test_content = current_batch + news_line if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + rss_header + word_header + news_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 关键词间分隔符 if i < len(rss_stats) - 1: separator = "" if format_type in ("wework", "bark"): separator = "\n\n\n\n" elif format_type == "telegram": separator = "\n\n" elif format_type == "ntfy": separator = "\n\n" elif format_type == "feishu": separator = f"\n{feishu_separator}\n\n" elif format_type == "dingtalk": separator = "\n---\n\n" elif format_type == "slack": separator = "\n\n" test_content = current_batch + separator if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes: current_batch = test_content return current_batch, current_batch_has_content, batches def _process_rss_new_titles_section( rss_new_stats: list, format_type: str, feishu_separator: str, base_header: str, base_footer: str, max_bytes: int, current_batch: str, current_batch_has_content: bool, batches: List[str], timezone: str = DEFAULT_TIMEZONE, add_separator: bool = True, ) -> tuple: """处理 RSS 新增区块（按来源分组，与热榜新增格式一致） Args: rss_new_stats: RSS 新增关键词统计列表，格式与热榜 stats 一致： [{"word": "AI", "count": 5, "titles": [...]}] format_type: 格式类型 feishu_separator: 飞书分隔符 base_header: 基础头部 base_footer: 基础尾部 max_bytes: 最大字节数 current_batch: 当前批次内容 current_batch_has_content: 当前批次是否有内容 batches: 已完成的批次列表 timezone: 时区名称 add_separator: 是否在区块前添加分割线（第一个区域时为 False） Returns: (current_batch, current_batch_has_content, batches) 元组 """ if not rss_new_stats: return current_batch, current_batch_has_content, batches # 从关键词分组中提取所有条目，重新按来源分组 source_map = {} for stat in rss_new_stats: for title_data in stat.get("titles", []): source_name = title_data.get("source_name", "未知来源") if source_name not in source_map: source_map[source_name] = [] source_map[source_name].append(title_data) if not source_map: return current_batch, current_batch_has_content, batches # 计算总条目数 total_items = sum(len(titles) for titles in source_map.values()) # RSS 新增区块标题（根据 add_separator 决定是否添加前置分割线） new_header = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type in ("wework", "bark"): new_header = f"\n\n\n\n🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "telegram": new_header = f"\n\n🆕 RSS 本次新增 (共 {total_items} 条)\n\n" elif format_type == "ntfy": new_header = f"\n\n🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "feishu": new_header = f"\n{feishu_separator}\n\n🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": new_header = f"\n---\n\n🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "slack": new_header = f"\n\n🆕 *RSS 本次新增* (共 {total_items} 条)\n\n" else: # 不需要分割线（第一个区域） if format_type in ("wework", "bark"): new_header = f"🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "telegram": new_header = f"🆕 RSS 本次新增 (共 {total_items} 条)\n\n" elif format_type == "ntfy": new_header = f"🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "feishu": new_header = f"🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": new_header = f"🆕 **RSS 本次新增** (共 {total_items} 条)\n\n" elif format_type == "slack": new_header = f"🆕 *RSS 本次新增* (共 {total_items} 条)\n\n" # 添加 RSS 新增标题 test_content = current_batch + new_header if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 按来源分组显示（与热榜新增格式一致） source_list = list(source_map.items()) for i, (source_name, titles) in enumerate(source_list): count = len(titles) # 构建来源标题（与热榜新增格式一致） source_header = "" if format_type in ("wework", "bark"): source_header = f"**{source_name}** ({count} 条):\n\n" elif format_type == "telegram": source_header = f"{source_name} ({count} 条):\n\n" elif format_type == "ntfy": source_header = f"**{source_name}** ({count} 条):\n\n" elif format_type == "feishu": source_header = f"**{source_name}** ({count} 条):\n\n" elif format_type == "dingtalk": source_header = f"**{source_name}** ({count} 条):\n\n" elif format_type == "slack": source_header = f"*{source_name}* ({count} 条):\n\n" # 构建第一条新闻（不显示来源，禁用 new emoji） first_news_line = "" if titles: first_title_data = titles[0].copy() first_title_data["is_new"] = False if format_type in ("wework", "bark"): formatted_title = format_title_for_platform("wework", first_title_data, show_source=False) elif format_type == "telegram": formatted_title = format_title_for_platform("telegram", first_title_data, show_source=False) elif format_type == "ntfy": formatted_title = format_title_for_platform("ntfy", first_title_data, show_source=False) elif format_type == "feishu": formatted_title = format_title_for_platform("feishu", first_title_data, show_source=False) elif format_type == "dingtalk": formatted_title = format_title_for_platform("dingtalk", first_title_data, show_source=False) elif format_type == "slack": formatted_title = format_title_for_platform("slack", first_title_data, show_source=False) else: formatted_title = f"{first_title_data['title']}" first_news_line = f" 1. {formatted_title}\n" # 原子性检查：来源标题 + 第一条新闻必须一起处理 source_with_first_news = source_header + first_news_line test_content = current_batch + source_with_first_news if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header + source_with_first_news current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余新闻条目（禁用 new emoji） for j in range(start_index, len(titles)): title_data = titles[j].copy() title_data["is_new"] = False if format_type in ("wework", "bark"): formatted_title = format_title_for_platform("wework", title_data, show_source=False) elif format_type == "telegram": formatted_title = format_title_for_platform("telegram", title_data, show_source=False) elif format_type == "ntfy": formatted_title = format_title_for_platform("ntfy", title_data, show_source=False) elif format_type == "feishu": formatted_title = format_title_for_platform("feishu", title_data, show_source=False) elif format_type == "dingtalk": formatted_title = format_title_for_platform("dingtalk", title_data, show_source=False) elif format_type == "slack": formatted_title = format_title_for_platform("slack", title_data, show_source=False) else: formatted_title = f"{title_data['title']}" news_line = f" {j + 1}. {formatted_title}\n" test_content = current_batch + news_line if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + new_header + source_header + news_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True # 来源间添加空行（与热榜新增格式一致） current_batch += "\n" return current_batch, current_batch_has_content, batches def _format_rss_item_line( item: Dict, index: int, format_type: str, timezone: str = DEFAULT_TIMEZONE, ) -> str: """格式化单条 RSS 条目 Args: item: RSS 条目字典 index: 序号 format_type: 格式类型 timezone: 时区名称 Returns: 格式化后的条目行字符串 """ title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") # 使用友好时间格式 if published_at: friendly_time = format_iso_time_friendly(published_at, timezone, include_date=True) else: friendly_time = "" # 构建条目行 if format_type == "feishu": if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if friendly_time: item_line += f" - {friendly_time}" elif format_type == "telegram": if url: item_line = f" {index}. {title} ({url})" else: item_line = f" {index}. {title}" if friendly_time: item_line += f" - {friendly_time}" else: if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if friendly_time: item_line += f" `{friendly_time}`" item_line += "\n" return item_line def _process_standalone_section( standalone_data: Dict, format_type: str, feishu_separator: str, base_header: str, base_footer: str, max_bytes: int, current_batch: str, current_batch_has_content: bool, batches: List[str], timezone: str = DEFAULT_TIMEZONE, rank_threshold: int = 10, add_separator: bool = True, ) -> tuple: """处理独立展示区区块 独立展示区显示指定平台的完整热榜或 RSS 源内容，不受关键词过滤影响。 热榜按原始排名排序，RSS 按发布时间排序。 Args: standalone_data: 独立展示数据，格式： { "platforms": [{"id": "zhihu", "name": "知乎热榜", "items": [...]}], "rss_feeds": [{"id": "hacker-news", "name": "Hacker News", "items": [...]}] } format_type: 格式类型 feishu_separator: 飞书分隔符 base_header: 基础头部 base_footer: 基础尾部 max_bytes: 最大字节数 current_batch: 当前批次内容 current_batch_has_content: 当前批次是否有内容 batches: 已完成的批次列表 timezone: 时区名称 rank_threshold: 排名高亮阈值 add_separator: 是否在区块前添加分割线（第一个区域时为 False） Returns: (current_batch, current_batch_has_content, batches) 元组 """ if not standalone_data: return current_batch, current_batch_has_content, batches platforms = standalone_data.get("platforms", []) rss_feeds = standalone_data.get("rss_feeds", []) if not platforms and not rss_feeds: return current_batch, current_batch_has_content, batches # 计算总条目数 total_platform_items = sum(len(p.get("items", [])) for p in platforms) total_rss_items = sum(len(f.get("items", [])) for f in rss_feeds) total_items = total_platform_items + total_rss_items # 独立展示区标题（根据 add_separator 决定是否添加前置分割线） section_header = "" if add_separator and current_batch_has_content: # 需要添加分割线 if format_type == "feishu": section_header = f"\n{feishu_separator}\n\n📋 **独立展示区** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": section_header = f"\n---\n\n📋 **独立展示区** (共 {total_items} 条)\n\n" elif format_type in ("wework", "bark"): section_header = f"\n\n\n\n📋 **独立展示区** (共 {total_items} 条)\n\n" elif format_type == "telegram": section_header = f"\n\n📋 独立展示区 (共 {total_items} 条)\n\n" elif format_type == "slack": section_header = f"\n\n📋 *独立展示区* (共 {total_items} 条)\n\n" else: section_header = f"\n\n📋 **独立展示区** (共 {total_items} 条)\n\n" else: # 不需要分割线（第一个区域） if format_type == "feishu": section_header = f"📋 **独立展示区** (共 {total_items} 条)\n\n" elif format_type == "dingtalk": section_header = f"📋 **独立展示区** (共 {total_items} 条)\n\n" elif format_type == "telegram": section_header = f"📋 独立展示区 (共 {total_items} 条)\n\n" elif format_type == "slack": section_header = f"📋 *独立展示区* (共 {total_items} 条)\n\n" else: section_header = f"📋 **独立展示区** (共 {total_items} 条)\n\n" # 添加区块标题 test_content = current_batch + section_header if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) < max_bytes: current_batch = test_content current_batch_has_content = True else: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + section_header current_batch_has_content = True # 处理热榜平台 for platform in platforms: platform_name = platform.get("name", platform.get("id", "")) items = platform.get("items", []) if not items: continue # 平台标题 platform_header = "" if format_type in ("wework", "bark"): platform_header = f"**{platform_name}** ({len(items)} 条):\n\n" elif format_type == "telegram": platform_header = f"{platform_name} ({len(items)} 条):\n\n" elif format_type == "ntfy": platform_header = f"**{platform_name}** ({len(items)} 条):\n\n" elif format_type == "feishu": platform_header = f"**{platform_name}** ({len(items)} 条):\n\n" elif format_type == "dingtalk": platform_header = f"**{platform_name}** ({len(items)} 条):\n\n" elif format_type == "slack": platform_header = f"*{platform_name}* ({len(items)} 条):\n\n" # 构建第一条新闻 first_item_line = "" if items: first_item_line = _format_standalone_platform_item(items[0], 1, format_type, rank_threshold) # 原子性检查 platform_with_first = platform_header + first_item_line test_content = current_batch + platform_with_first if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + section_header + platform_with_first current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余条目 for j in range(start_index, len(items)): item_line = _format_standalone_platform_item(items[j], j + 1, format_type, rank_threshold) test_content = current_batch + item_line if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + section_header + platform_header + item_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True current_batch += "\n" # 处理 RSS 源 for feed in rss_feeds: feed_name = feed.get("name", feed.get("id", "")) items = feed.get("items", []) if not items: continue # RSS 源标题 feed_header = "" if format_type in ("wework", "bark"): feed_header = f"**{feed_name}** ({len(items)} 条):\n\n" elif format_type == "telegram": feed_header = f"{feed_name} ({len(items)} 条):\n\n" elif format_type == "ntfy": feed_header = f"**{feed_name}** ({len(items)} 条):\n\n" elif format_type == "feishu": feed_header = f"**{feed_name}** ({len(items)} 条):\n\n" elif format_type == "dingtalk": feed_header = f"**{feed_name}** ({len(items)} 条):\n\n" elif format_type == "slack": feed_header = f"*{feed_name}* ({len(items)} 条):\n\n" # 构建第一条 RSS first_item_line = "" if items: first_item_line = _format_standalone_rss_item(items[0], 1, format_type, timezone) # 原子性检查 feed_with_first = feed_header + first_item_line test_content = current_batch + feed_with_first if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + section_header + feed_with_first current_batch_has_content = True start_index = 1 else: current_batch = test_content current_batch_has_content = True start_index = 1 # 处理剩余条目 for j in range(start_index, len(items)): item_line = _format_standalone_rss_item(items[j], j + 1, format_type, timezone) test_content = current_batch + item_line if len(test_content.encode("utf-8")) + len(base_footer.encode("utf-8")) >= max_bytes: if current_batch_has_content: batches.append(current_batch + base_footer) current_batch = base_header + section_header + feed_header + item_line current_batch_has_content = True else: current_batch = test_content current_batch_has_content = True current_batch += "\n" return current_batch, current_batch_has_content, batches def _format_standalone_platform_item(item: Dict, index: int, format_type: str, rank_threshold: int = 10) -> str: """格式化独立展示区的热榜条目（复用热点词汇统计区样式） Args: item: 热榜条目，包含 title, url, rank, ranks, first_time, last_time, count index: 序号 format_type: 格式类型 rank_threshold: 排名高亮阈值 Returns: 格式化后的条目行字符串 """ title = item.get("title", "") url = item.get("url", "") or item.get("mobileUrl", "") ranks = item.get("ranks", []) rank = item.get("rank", 0) first_time = item.get("first_time", "") last_time = item.get("last_time", "") count = item.get("count", 1) # 使用 format_rank_display 格式化排名（复用热点词汇统计区逻辑） # 如果没有 ranks 列表，用单个 rank 构造 if not ranks and rank > 0: ranks = [rank] rank_display = format_rank_display(ranks, rank_threshold, format_type) if ranks else "" # 构建时间显示（用 ~ 连接范围，与热点词汇统计区一致） # 将 HH-MM 格式转换为 HH:MM 格式 time_display = "" if first_time and last_time and first_time != last_time: first_time_display = convert_time_for_display(first_time) last_time_display = convert_time_for_display(last_time) time_display = f"{first_time_display}~{last_time_display}" elif first_time: time_display = convert_time_for_display(first_time) # 构建次数显示（格式为 (N次)，与热点词汇统计区一致） count_display = f"({count}次)" if count > 1 else "" # 根据格式类型构建条目行（复用热点词汇统计区样式） if format_type == "feishu": if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if rank_display: item_line += f" {rank_display}" if time_display: item_line += f" - {time_display}" if count_display: item_line += f" {count_display}" elif format_type == "dingtalk": if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if rank_display: item_line += f" {rank_display}" if time_display: item_line += f" - {time_display}" if count_display: item_line += f" {count_display}" elif format_type == "telegram": if url: item_line = f" {index}. {title} ({url})" else: item_line = f" {index}. {title}" if rank_display: item_line += f" {rank_display}" if time_display: item_line += f" - {time_display}" if count_display: item_line += f" {count_display}" elif format_type == "slack": if url: item_line = f" {index}. <{url}|{title}>" else: item_line = f" {index}. {title}" if rank_display: item_line += f" {rank_display}" if time_display: item_line += f" _{time_display}_" if count_display: item_line += f" {count_display}" else: # wework, bark, ntfy if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if rank_display: item_line += f" {rank_display}" if time_display: item_line += f" - {time_display}" if count_display: item_line += f" {count_display}" item_line += "\n" return item_line def _format_standalone_rss_item( item: Dict, index: int, format_type: str, timezone: str = "Asia/Shanghai" ) -> str: """格式化独立展示区的 RSS 条目 Args: item: RSS 条目，包含 title, url, published_at, author index: 序号 format_type: 格式类型 timezone: 时区名称 Returns: 格式化后的条目行字符串 """ title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") author = item.get("author", "") # 使用友好时间格式 friendly_time = "" if published_at: friendly_time = format_iso_time_friendly(published_at, timezone, include_date=True) # 构建元信息 meta_parts = [] if friendly_time: meta_parts.append(friendly_time) if author: meta_parts.append(author) meta_str = ", ".join(meta_parts) # 根据格式类型构建条目行 if format_type == "feishu": if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if meta_str: item_line += f" - {meta_str}" elif format_type == "telegram": if url: item_line = f" {index}. {title} ({url})" else: item_line = f" {index}. {title}" if meta_str: item_line += f" - {meta_str}" elif format_type == "slack": if url: item_line = f" {index}. <{url}|{title}>" else: item_line = f" {index}. {title}" if meta_str: item_line += f" _{meta_str}_" else: # wework, bark, ntfy, dingtalk if url: item_line = f" {index}. [{title}]({url})" else: item_line = f" {index}. {title}" if meta_str: item_line += f" `{meta_str}`" item_line += "\n" return item_line ================================================ FILE: trendradar/report/__init__.py ================================================ # coding=utf-8 """ 报告生成模块 提供报告生成和格式化功能，包括： - HTML 报告生成 - 标题格式化工具 模块结构： - helpers: 报告辅助函数（清理、转义、格式化） - formatter: 平台标题格式化 - html: HTML 报告渲染 - generator: 报告生成器 """ from trendradar.report.helpers import ( clean_title, html_escape, format_rank_display, ) from trendradar.report.formatter import format_title_for_platform from trendradar.report.html import render_html_content from trendradar.report.generator import ( prepare_report_data, generate_html_report, ) __all__ = [ # 辅助函数 "clean_title", "html_escape", "format_rank_display", # 格式化函数 "format_title_for_platform", # HTML 渲染 "render_html_content", # 报告生成器 "prepare_report_data", "generate_html_report", ] ================================================ FILE: trendradar/report/formatter.py ================================================ # coding=utf-8 """ 平台标题格式化模块 提供多平台标题格式化功能 """ from typing import Dict from trendradar.report.helpers import clean_title, html_escape, format_rank_display def format_title_for_platform( platform: str, title_data: Dict, show_source: bool = True, show_keyword: bool = False ) -> str: """统一的标题格式化方法 为不同平台生成对应格式的标题字符串。 Args: platform: 目标平台，支持: - "feishu": 飞书 - "dingtalk": 钉钉 - "wework": 企业微信 - "bark": Bark - "telegram": Telegram - "ntfy": ntfy - "slack": Slack - "html": HTML 报告 title_data: 标题数据字典，包含以下字段: - title: 标题文本 - source_name: 来源名称 - time_display: 时间显示 - count: 出现次数 - ranks: 排名列表 - rank_threshold: 高亮阈值 - url: PC端链接 - mobile_url: 移动端链接（优先使用） - is_new: 是否为新增标题（可选） - matched_keyword: 匹配的关键词（可选，platform 模式使用） show_source: 是否显示来源名称（keyword 模式使用） show_keyword: 是否显示关键词标签（platform 模式使用） Returns: 格式化后的标题字符串 """ rank_display = format_rank_display( title_data["ranks"], title_data["rank_threshold"], platform ) link_url = title_data["mobile_url"] or title_data["url"] cleaned_title = clean_title(title_data["title"]) # 获取关键词标签（platform 模式使用） keyword = title_data.get("matched_keyword", "") if show_keyword else "" if platform == "feishu": if link_url: formatted_title = f"[{cleaned_title}]({link_url})" else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"[{keyword}] {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" - {title_data['time_display']}" if title_data["count"] > 1: result += f" ({title_data['count']}次)" return result elif platform == "dingtalk": if link_url: formatted_title = f"[{cleaned_title}]({link_url})" else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"[{keyword}] {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" - {title_data['time_display']}" if title_data["count"] > 1: result += f" ({title_data['count']}次)" return result elif platform in ("wework", "bark"): # WeWork 和 Bark 使用 markdown 格式 if link_url: formatted_title = f"[{cleaned_title}]({link_url})" else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"[{keyword}] {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" - {title_data['time_display']}" if title_data["count"] > 1: result += f" ({title_data['count']}次)" return result elif platform == "telegram": if link_url: formatted_title = f'{html_escape(cleaned_title)}' else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"[{html_escape(keyword)}] {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" - {title_data['time_display']}" if title_data["count"] > 1: result += f" ({title_data['count']}次)" return result elif platform == "ntfy": if link_url: formatted_title = f"[{cleaned_title}]({link_url})" else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"[{keyword}] {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" `- {title_data['time_display']}`" if title_data["count"] > 1: result += f" `({title_data['count']}次)`" return result elif platform == "slack": # Slack 使用 mrkdwn 格式 if link_url: # Slack 链接格式: formatted_title = f"<{link_url}|{cleaned_title}>" else: formatted_title = cleaned_title title_prefix = "🆕 " if title_data.get("is_new") else "" if show_source: result = f"[{title_data['source_name']}] {title_prefix}{formatted_title}" elif show_keyword and keyword: result = f"*[{keyword}]* {title_prefix}{formatted_title}" else: result = f"{title_prefix}{formatted_title}" # 排名（使用 * 加粗） rank_display = format_rank_display( title_data["ranks"], title_data["rank_threshold"], "slack" ) if rank_display: result += f" {rank_display}" if title_data["time_display"]: result += f" `- {title_data['time_display']}`" if title_data["count"] > 1: result += f" `({title_data['count']}次)`" return result elif platform == "html": rank_display = format_rank_display( title_data["ranks"], title_data["rank_threshold"], "html" ) link_url = title_data["mobile_url"] or title_data["url"] escaped_title = html_escape(cleaned_title) escaped_source_name = html_escape(title_data["source_name"]) # 构建前缀（来源或关键词） if show_source: prefix = f'[{escaped_source_name}] ' elif show_keyword and keyword: escaped_keyword = html_escape(keyword) prefix = f'[{escaped_keyword}] ' else: prefix = "" if link_url: escaped_url = html_escape(link_url) formatted_title = f'{prefix}{escaped_title}' else: formatted_title = f'{prefix}{escaped_title}' if rank_display: formatted_title += f" {rank_display}" if title_data["time_display"]: escaped_time = html_escape(title_data["time_display"]) formatted_title += f" - {escaped_time}" if title_data["count"] > 1: formatted_title += f" ({title_data['count']}次)" if title_data.get("is_new"): formatted_title = f"
🆕 {formatted_title}
" return formatted_title else: return cleaned_title ================================================ FILE: trendradar/report/generator.py ================================================ # coding=utf-8 """ 报告生成模块 提供报告数据准备和 HTML 生成功能： - prepare_report_data: 准备报告数据 - generate_html_report: 生成 HTML 报告 """ from pathlib import Path from typing import Dict, List, Optional, Callable def prepare_report_data( stats: List[Dict], failed_ids: Optional[List] = None, new_titles: Optional[Dict] = None, id_to_name: Optional[Dict] = None, mode: str = "daily", rank_threshold: int = 3, matches_word_groups_func: Optional[Callable] = None, load_frequency_words_func: Optional[Callable] = None, show_new_section: bool = True, ) -> Dict: """ 准备报告数据 Args: stats: 统计结果列表 failed_ids: 失败的 ID 列表 new_titles: 新增标题 id_to_name: ID 到名称的映射 mode: 报告模式 (daily/incremental/current) rank_threshold: 排名阈值 matches_word_groups_func: 词组匹配函数 load_frequency_words_func: 加载频率词函数 show_new_section: 是否显示新增热点区域 Returns: Dict: 准备好的报告数据 """ processed_new_titles = [] # 在增量模式下或配置关闭时隐藏新增新闻区域 hide_new_section = mode == "incremental" or not show_new_section # 只有在非隐藏模式下才处理新增新闻部分 if not hide_new_section: filtered_new_titles = {} if new_titles and id_to_name: # 如果提供了匹配函数，使用它过滤 if matches_word_groups_func and load_frequency_words_func: word_groups, filter_words, global_filters = load_frequency_words_func() for source_id, titles_data in new_titles.items(): filtered_titles = {} for title, title_data in titles_data.items(): if matches_word_groups_func(title, word_groups, filter_words, global_filters): filtered_titles[title] = title_data if filtered_titles: filtered_new_titles[source_id] = filtered_titles else: # 没有匹配函数时，使用全部 filtered_new_titles = new_titles # 打印过滤后的新增热点数（与推送显示一致） original_new_count = sum(len(titles) for titles in new_titles.values()) if new_titles else 0 filtered_new_count = sum(len(titles) for titles in filtered_new_titles.values()) if filtered_new_titles else 0 if original_new_count > 0: print(f"频率词过滤后：{filtered_new_count} 条新增热点匹配（原始 {original_new_count} 条）") if filtered_new_titles and id_to_name: for source_id, titles_data in filtered_new_titles.items(): source_name = id_to_name.get(source_id, source_id) source_titles = [] for title, title_data in titles_data.items(): url = title_data.get("url", "") mobile_url = title_data.get("mobileUrl", "") ranks = title_data.get("ranks", []) processed_title = { "title": title, "source_name": source_name, "time_display": "", "count": 1, "ranks": ranks, "rank_threshold": rank_threshold, "url": url, "mobile_url": mobile_url, "is_new": True, } source_titles.append(processed_title) if source_titles: processed_new_titles.append( { "source_id": source_id, "source_name": source_name, "titles": source_titles, } ) processed_stats = [] for stat in stats: if stat["count"] <= 0: continue processed_titles = [] for title_data in stat["titles"]: processed_title = { "title": title_data["title"], "source_name": title_data["source_name"], "time_display": title_data["time_display"], "count": title_data["count"], "ranks": title_data["ranks"], "rank_threshold": title_data["rank_threshold"], "url": title_data.get("url", ""), "mobile_url": title_data.get("mobileUrl", ""), "is_new": title_data.get("is_new", False), } processed_titles.append(processed_title) processed_stats.append( { "word": stat["word"], "count": stat["count"], "percentage": stat.get("percentage", 0), "titles": processed_titles, } ) return { "stats": processed_stats, "new_titles": processed_new_titles, "failed_ids": failed_ids or [], "total_new_count": sum( len(source["titles"]) for source in processed_new_titles ), } def generate_html_report( stats: List[Dict], total_titles: int, failed_ids: Optional[List] = None, new_titles: Optional[Dict] = None, id_to_name: Optional[Dict] = None, mode: str = "daily", update_info: Optional[Dict] = None, rank_threshold: int = 3, output_dir: str = "output", date_folder: str = "", time_filename: str = "", render_html_func: Optional[Callable] = None, matches_word_groups_func: Optional[Callable] = None, load_frequency_words_func: Optional[Callable] = None, ) -> str: """ 生成 HTML 报告 每次生成 HTML 后会： 1. 保存时间戳快照到 output/html/日期/时间.html（历史记录） 2. 复制到 output/html/latest/{mode}.html（最新报告） 3. 复制到 output/index.html 和根目录 index.html（入口） Args: stats: 统计结果列表 total_titles: 总标题数 failed_ids: 失败的 ID 列表 new_titles: 新增标题 id_to_name: ID 到名称的映射 mode: 报告模式 (daily/incremental/current) update_info: 更新信息 rank_threshold: 排名阈值 output_dir: 输出目录 date_folder: 日期文件夹名称 time_filename: 时间文件名 render_html_func: HTML 渲染函数 matches_word_groups_func: 词组匹配函数 load_frequency_words_func: 加载频率词函数 Returns: str: 生成的 HTML 文件路径（时间戳快照路径） """ # 时间戳快照文件名 snapshot_filename = f"{time_filename}.html" # 构建输出路径（扁平化结构：output/html/日期/） snapshot_path = Path(output_dir) / "html" / date_folder snapshot_path.mkdir(parents=True, exist_ok=True) snapshot_file = str(snapshot_path / snapshot_filename) # 准备报告数据 report_data = prepare_report_data( stats, failed_ids, new_titles, id_to_name, mode, rank_threshold, matches_word_groups_func, load_frequency_words_func, ) # 渲染 HTML 内容 if render_html_func: html_content = render_html_func( report_data, total_titles, mode, update_info ) else: # 默认简单 HTML html_content = f"

Report

{report_data}
" # 1. 保存时间戳快照（历史记录） with open(snapshot_file, "w", encoding="utf-8") as f: f.write(html_content) # 2. 复制到 html/latest/{mode}.html（最新报告） latest_dir = Path(output_dir) / "html" / "latest" latest_dir.mkdir(parents=True, exist_ok=True) latest_file = latest_dir / f"{mode}.html" with open(latest_file, "w", encoding="utf-8") as f: f.write(html_content) # 3. 复制到 index.html（入口） # output/index.html（供 Docker Volume 挂载访问） output_index = Path(output_dir) / "index.html" with open(output_index, "w", encoding="utf-8") as f: f.write(html_content) # 根目录 index.html（供 GitHub Pages 访问） root_index = Path("index.html") with open(root_index, "w", encoding="utf-8") as f: f.write(html_content) return snapshot_file ================================================ FILE: trendradar/report/helpers.py ================================================ # coding=utf-8 """ 报告辅助函数模块 提供报告生成相关的通用辅助函数 """ import re from typing import List def clean_title(title: str) -> str: """清理标题中的特殊字符 清理规则： - 将换行符(\n, \r)替换为空格 - 将多个连续空白字符合并为单个空格 - 去除首尾空白 Args: title: 原始标题字符串 Returns: 清理后的标题字符串 """ if not isinstance(title, str): title = str(title) cleaned_title = title.replace("\n", " ").replace("\r", " ") cleaned_title = re.sub(r"\s+", " ", cleaned_title) cleaned_title = cleaned_title.strip() return cleaned_title def html_escape(text: str) -> str: """HTML特殊字符转义 转义规则（按顺序）： - & → & - < → < - > → > - " → " - ' → ' Args: text: 原始文本 Returns: 转义后的文本 """ if not isinstance(text, str): text = str(text) return ( text.replace("&", "&") .replace("<", "<") .replace(">", ">") .replace('"', """) .replace("'", "'") ) def format_rank_display(ranks: List[int], rank_threshold: int, format_type: str) -> str: """格式化排名显示 根据不同平台类型生成对应格式的排名字符串。 当最小排名小于等于阈值时，使用高亮格式。 Args: ranks: 排名列表（可能包含重复值） rank_threshold: 高亮阈值，小于等于此值的排名会高亮显示 format_type: 平台类型，支持: - "html": HTML格式 - "feishu": 飞书格式 - "dingtalk": 钉钉格式 - "wework": 企业微信格式 - "telegram": Telegram格式 - "slack": Slack格式 - 其他: 默认markdown格式 Returns: 格式化后的排名字符串，如 "[1]" 或 "[1 - 5]" 如果排名列表为空，返回空字符串 """ if not ranks: return "" unique_ranks = sorted(set(ranks)) min_rank = unique_ranks[0] max_rank = unique_ranks[-1] # 根据平台类型选择高亮格式 if format_type == "html": highlight_start = "" highlight_end = "" elif format_type == "feishu": highlight_start = "**" highlight_end = "**" elif format_type == "dingtalk": highlight_start = "**" highlight_end = "**" elif format_type == "wework": highlight_start = "**" highlight_end = "**" elif format_type == "telegram": highlight_start = "" highlight_end = "" elif format_type == "slack": highlight_start = "*" highlight_end = "*" else: # 默认 markdown 格式 highlight_start = "**" highlight_end = "**" # 生成排名显示 rank_str = "" if min_rank <= rank_threshold: if min_rank == max_rank: rank_str = f"{highlight_start}[{min_rank}]{highlight_end}" else: rank_str = f"{highlight_start}[{min_rank} - {max_rank}]{highlight_end}" else: if min_rank == max_rank: rank_str = f"[{min_rank}]" else: rank_str = f"[{min_rank} - {max_rank}]" # 计算热度趋势 trend_arrow = "" if len(ranks) >= 2: prev_rank = ranks[-2] curr_rank = ranks[-1] if curr_rank < prev_rank: trend_arrow = "🔺" # 排名上升（数值变小） elif curr_rank > prev_rank: trend_arrow = "🔻" # 排名下降（数值变大） else: trend_arrow = "➖" # 排名持平 # len(ranks) == 1 时不显示趋势箭头（新上榜由 is_new 字段在 formatter.py 中处理） return f"{rank_str} {trend_arrow}" if trend_arrow else rank_str ================================================ FILE: trendradar/report/html.py ================================================ # coding=utf-8 """ HTML 报告渲染模块 提供 HTML 格式的热点新闻报告生成功能 """ from datetime import datetime from typing import Any, Dict, List, Optional, Callable from trendradar.report.helpers import html_escape from trendradar.utils.time import convert_time_for_display from trendradar.ai.formatter import render_ai_analysis_html_rich def render_html_content( report_data: Dict, total_titles: int, mode: str = "daily", update_info: Optional[Dict] = None, *, region_order: Optional[List[str]] = None, get_time_func: Optional[Callable[[], datetime]] = None, rss_items: Optional[List[Dict]] = None, rss_new_items: Optional[List[Dict]] = None, display_mode: str = "keyword", standalone_data: Optional[Dict] = None, ai_analysis: Optional[Any] = None, show_new_section: bool = True, ) -> str: """渲染HTML内容 Args: report_data: 报告数据字典，包含 stats, new_titles, failed_ids, total_new_count total_titles: 新闻总数 mode: 报告模式 ("daily", "current", "incremental") update_info: 更新信息（可选） region_order: 区域显示顺序列表 get_time_func: 获取当前时间的函数（可选，默认使用 datetime.now） rss_items: RSS 统计条目列表（可选） rss_new_items: RSS 新增条目列表（可选） display_mode: 显示模式 ("keyword"=按关键词分组, "platform"=按平台分组) standalone_data: 独立展示区数据（可选），包含 platforms 和 rss_feeds ai_analysis: AI 分析结果对象（可选），AIAnalysisResult 实例 show_new_section: 是否显示新增热点区域 Returns: 渲染后的 HTML 字符串 """ # 默认区域顺序 default_region_order = ["hotlist", "rss", "new_items", "standalone", "ai_analysis"] if region_order is None: region_order = default_region_order html = """ 热点新闻分析
热点新闻分析
报告类型 """ # 处理报告类型显示（根据 mode 直接显示） if mode == "current": html += "当前榜单" elif mode == "incremental": html += "增量分析" else: html += "全天汇总" html += """
新闻总数 """ html += f"{total_titles} 条" # 计算筛选后的热点新闻数量 hot_news_count = sum(len(stat["titles"]) for stat in report_data["stats"]) html += """
热点新闻 """ html += f"{hot_news_count} 条" html += """
生成时间 """ # 使用提供的时间函数或默认 datetime.now if get_time_func: now = get_time_func() else: now = datetime.now() html += now.strftime("%m-%d %H:%M") html += """
""" # 处理失败ID错误信息 if report_data["failed_ids"]: html += """
⚠️ 请求失败的平台
    """ for id_value in report_data["failed_ids"]: html += f'
  • {html_escape(id_value)}
    • ' html += """
""" # 生成热点词汇统计部分的HTML stats_html = "" if report_data["stats"]: total_count = len(report_data["stats"]) for i, stat in enumerate(report_data["stats"], 1): count = stat["count"] # 确定热度等级 if count >= 10: count_class = "hot" elif count >= 5: count_class = "warm" else: count_class = "" escaped_word = html_escape(stat["word"]) stats_html += f"""
{escaped_word}
{count} 条
{i}/{total_count}
""" # 处理每个词组下的新闻标题，给每条新闻标上序号 for j, title_data in enumerate(stat["titles"], 1): is_new = title_data.get("is_new", False) new_class = "new" if is_new else "" stats_html += f"""
{j}
""" # 根据 display_mode 决定显示来源还是关键词 if display_mode == "keyword": # keyword 模式：显示来源 stats_html += f'{html_escape(title_data["source_name"])}' else: # platform 模式：显示关键词 matched_keyword = title_data.get("matched_keyword", "") if matched_keyword: stats_html += f'[{html_escape(matched_keyword)}]' # 处理排名显示 ranks = title_data.get("ranks", []) if ranks: min_rank = min(ranks) max_rank = max(ranks) rank_threshold = title_data.get("rank_threshold", 10) # 确定排名等级 if min_rank <= 3: rank_class = "top" elif min_rank <= rank_threshold: rank_class = "high" else: rank_class = "" if min_rank == max_rank: rank_text = str(min_rank) else: rank_text = f"{min_rank}-{max_rank}" stats_html += f'{rank_text}' # 处理时间显示 time_display = title_data.get("time_display", "") if time_display: # 简化时间显示格式，将波浪线替换为~ simplified_time = ( time_display.replace(" ~ ", "~") .replace("[", "") .replace("]", "") ) stats_html += ( f'{html_escape(simplified_time)}' ) # 处理出现次数 count_info = title_data.get("count", 1) if count_info > 1: stats_html += f'{count_info}次' stats_html += """
""" # 处理标题和链接 escaped_title = html_escape(title_data["title"]) link_url = title_data.get("mobile_url") or title_data.get("url", "") if link_url: escaped_url = html_escape(link_url) stats_html += f'{escaped_title}' else: stats_html += escaped_title stats_html += """
""" stats_html += """
""" # 给热榜统计添加外层包装 if stats_html: stats_html = f"""
{stats_html}
""" # 生成新增新闻区域的HTML new_titles_html = "" if show_new_section and report_data["new_titles"]: new_titles_html += f"""
本次新增热点 (共 {report_data['total_new_count']} 条)
""" for source_data in report_data["new_titles"]: escaped_source = html_escape(source_data["source_name"]) titles_count = len(source_data["titles"]) new_titles_html += f"""
{escaped_source} · {titles_count}条
""" # 为新增新闻也添加序号 for idx, title_data in enumerate(source_data["titles"], 1): ranks = title_data.get("ranks", []) # 处理新增新闻的排名显示 rank_class = "" if ranks: min_rank = min(ranks) if min_rank <= 3: rank_class = "top" elif min_rank <= title_data.get("rank_threshold", 10): rank_class = "high" if len(ranks) == 1: rank_text = str(ranks[0]) else: rank_text = f"{min(ranks)}-{max(ranks)}" else: rank_text = "?" new_titles_html += f"""
{idx}
{rank_text}
""" # 处理新增新闻的链接 escaped_title = html_escape(title_data["title"]) link_url = title_data.get("mobile_url") or title_data.get("url", "") if link_url: escaped_url = html_escape(link_url) new_titles_html += f'{escaped_title}' else: new_titles_html += escaped_title new_titles_html += """
""" new_titles_html += """
""" new_titles_html += """
""" # 生成 RSS 统计内容 def render_rss_stats_html(stats: List[Dict], title: str = "RSS 订阅更新") -> str: """渲染 RSS 统计区块 HTML Args: stats: RSS 分组统计列表，格式与热榜一致： [ { "word": "关键词", "count": 5, "titles": [ { "title": "标题", "source_name": "Feed 名称", "time_display": "12-29 08:20", "url": "...", "is_new": True/False } ] } ] title: 区块标题 Returns: 渲染后的 HTML 字符串 """ if not stats: return "" # 计算总条目数 total_count = sum(stat.get("count", 0) for stat in stats) if total_count == 0: return "" rss_html = f"""
{title}
{total_count} 条
""" # 按关键词分组渲染（与热榜格式一致） for stat in stats: keyword = stat.get("word", "") titles = stat.get("titles", []) if not titles: continue keyword_count = len(titles) rss_html += f"""
{html_escape(keyword)}
{keyword_count} 条
""" for title_data in titles: item_title = title_data.get("title", "") url = title_data.get("url", "") time_display = title_data.get("time_display", "") source_name = title_data.get("source_name", "") is_new = title_data.get("is_new", False) rss_html += """
""" if time_display: rss_html += f'{html_escape(time_display)}' if source_name: rss_html += f'{html_escape(source_name)}' if is_new: rss_html += 'NEW' rss_html += """
""" escaped_title = html_escape(item_title) if url: escaped_url = html_escape(url) rss_html += f'{escaped_title}' else: rss_html += escaped_title rss_html += """
""" rss_html += """
""" rss_html += """
""" return rss_html # 生成独立展示区内容 def render_standalone_html(data: Optional[Dict]) -> str: """渲染独立展示区 HTML（复用热点词汇统计区样式） Args: data: 独立展示数据，格式： { "platforms": [ { "id": "zhihu", "name": "知乎热榜", "items": [ { "title": "标题", "url": "链接", "rank": 1, "ranks": [1, 2, 1], "first_time": "08:00", "last_time": "12:30", "count": 3, } ] } ], "rss_feeds": [ { "id": "hacker-news", "name": "Hacker News", "items": [ { "title": "标题", "url": "链接", "published_at": "2025-01-07T08:00:00", "author": "作者", } ] } ] } Returns: 渲染后的 HTML 字符串 """ if not data: return "" platforms = data.get("platforms", []) rss_feeds = data.get("rss_feeds", []) if not platforms and not rss_feeds: return "" # 计算总条目数 total_platform_items = sum(len(p.get("items", [])) for p in platforms) total_rss_items = sum(len(f.get("items", [])) for f in rss_feeds) total_count = total_platform_items + total_rss_items if total_count == 0: return "" standalone_html = f"""
独立展示区
{total_count} 条
""" # 渲染热榜平台（复用 word-group 结构） for platform in platforms: platform_name = platform.get("name", platform.get("id", "")) items = platform.get("items", []) if not items: continue standalone_html += f"""
{html_escape(platform_name)}
{len(items)} 条
""" # 渲染每个条目（复用 news-item 结构） for j, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") or item.get("mobileUrl", "") rank = item.get("rank", 0) ranks = item.get("ranks", []) first_time = item.get("first_time", "") last_time = item.get("last_time", "") count = item.get("count", 1) standalone_html += f"""
{j}
""" # 排名显示（复用 rank-num 样式，无 # 前缀） if ranks: min_rank = min(ranks) max_rank = max(ranks) # 确定排名等级 if min_rank <= 3: rank_class = "top" elif min_rank <= 10: rank_class = "high" else: rank_class = "" if min_rank == max_rank: rank_text = str(min_rank) else: rank_text = f"{min_rank}-{max_rank}" standalone_html += f'{rank_text}' elif rank > 0: if rank <= 3: rank_class = "top" elif rank <= 10: rank_class = "high" else: rank_class = "" standalone_html += f'{rank}' # 时间显示（复用 time-info 样式，将 HH-MM 转换为 HH:MM） if first_time and last_time and first_time != last_time: first_time_display = convert_time_for_display(first_time) last_time_display = convert_time_for_display(last_time) standalone_html += f'{html_escape(first_time_display)}~{html_escape(last_time_display)}' elif first_time: first_time_display = convert_time_for_display(first_time) standalone_html += f'{html_escape(first_time_display)}' # 出现次数（复用 count-info 样式） if count > 1: standalone_html += f'{count}次' standalone_html += """
""" # 标题和链接（复用 news-link 样式） escaped_title = html_escape(title) if url: escaped_url = html_escape(url) standalone_html += f'{escaped_title}' else: standalone_html += escaped_title standalone_html += """
""" standalone_html += """
""" # 渲染 RSS 源（复用相同结构） for feed in rss_feeds: feed_name = feed.get("name", feed.get("id", "")) items = feed.get("items", []) if not items: continue standalone_html += f"""
{html_escape(feed_name)}
{len(items)} 条
""" for j, item in enumerate(items, 1): title = item.get("title", "") url = item.get("url", "") published_at = item.get("published_at", "") author = item.get("author", "") standalone_html += f"""
{j}
""" # 时间显示（格式化 ISO 时间） if published_at: try: from datetime import datetime as dt if "T" in published_at: dt_obj = dt.fromisoformat(published_at.replace("Z", "+00:00")) time_display = dt_obj.strftime("%m-%d %H:%M") else: time_display = published_at except: time_display = published_at standalone_html += f'{html_escape(time_display)}' # 作者显示 if author: standalone_html += f'{html_escape(author)}' standalone_html += """
""" escaped_title = html_escape(title) if url: escaped_url = html_escape(url) standalone_html += f'{escaped_title}' else: standalone_html += escaped_title standalone_html += """
""" standalone_html += """
""" standalone_html += """
""" return standalone_html # 生成 RSS 统计和新增 HTML rss_stats_html = render_rss_stats_html(rss_items, "RSS 订阅更新") if rss_items else "" rss_new_html = render_rss_stats_html(rss_new_items, "RSS 新增更新") if rss_new_items else "" # 生成独立展示区 HTML standalone_html = render_standalone_html(standalone_data) # 生成 AI 分析 HTML ai_html = render_ai_analysis_html_rich(ai_analysis) if ai_analysis else "" # 准备各区域内容映射 region_contents = { "hotlist": stats_html, "rss": rss_stats_html, "new_items": (new_titles_html, rss_new_html), # 元组，分别处理 "standalone": standalone_html, "ai_analysis": ai_html, } def add_section_divider(content: str) -> str: """为内容的外层 div 添加 section-divider 类""" if not content or 'class="' not in content: return content first_class_pos = content.find('class="') if first_class_pos != -1: insert_pos = first_class_pos + len('class="') return content[:insert_pos] + "section-divider " + content[insert_pos:] return content # 按 region_order 顺序组装内容，动态添加分割线 has_previous_content = False for region in region_order: content = region_contents.get(region, "") if region == "new_items": # 特殊处理 new_items 区域（包含热榜新增和 RSS 新增两部分） new_html, rss_new = content if new_html: if has_previous_content: new_html = add_section_divider(new_html) html += new_html has_previous_content = True if rss_new: if has_previous_content: rss_new = add_section_divider(rss_new) html += rss_new has_previous_content = True elif content: if has_previous_content: content = add_section_divider(content) html += content has_previous_content = True html += """
由 TrendRadar 生成 · GitHub 开源项目 """ if update_info: html += f"""
发现新版本 {update_info['remote_version']}，当前版本 {update_info['current_version']} """ html += """
""" return html ================================================ FILE: trendradar/report/rss_html.py ================================================ # coding=utf-8 """ RSS HTML 报告渲染模块 提供 RSS 订阅内容的 HTML 格式报告生成功能 """ from datetime import datetime from typing import Dict, List, Optional, Callable from trendradar.report.helpers import html_escape def render_rss_html_content( rss_items: List[Dict], total_count: int, feeds_info: Optional[Dict[str, str]] = None, *, get_time_func: Optional[Callable[[], datetime]] = None, ) -> str: """渲染 RSS HTML 内容 Args: rss_items: RSS 条目列表，每个条目包含: - title: 标题 - feed_id: RSS 源 ID - feed_name: RSS 源名称 - url: 链接 - published_at: 发布时间 - summary: 摘要（可选） - author: 作者（可选） total_count: 条目总数 feeds_info: RSS 源 ID 到名称的映射 get_time_func: 获取当前时间的函数（可选，默认使用 datetime.now） Returns: 渲染后的 HTML 字符串 """ html = """ RSS 订阅内容
RSS 订阅内容
订阅条目 """ html += f"{total_count} 条" html += """
生成时间 """ # 使用提供的时间函数或默认 datetime.now if get_time_func: now = get_time_func() else: now = datetime.now() html += now.strftime("%m-%d %H:%M") html += """
""" # 按 feed_id 分组 feeds_map: Dict[str, List[Dict]] = {} for item in rss_items: feed_id = item.get("feed_id", "unknown") if feed_id not in feeds_map: feeds_map[feed_id] = [] feeds_map[feed_id].append(item) # 渲染每个 RSS 源的内容 for feed_id, items in feeds_map.items(): feed_name = items[0].get("feed_name", feed_id) if items else feed_id if feeds_info and feed_id in feeds_info: feed_name = feeds_info[feed_id] escaped_feed_name = html_escape(feed_name) html += f"""
{escaped_feed_name}
{len(items)} 条
""" for item in items: escaped_title = html_escape(item.get("title", "")) url = item.get("url", "") published_at = item.get("published_at", "") author = item.get("author", "") summary = item.get("summary", "") html += """
""" if published_at: html += f'{html_escape(published_at)}' if author: html += f'by {html_escape(author)}' html += """
""" if url: escaped_url = html_escape(url) html += f'{escaped_title}' else: html += escaped_title html += """
""" if summary: escaped_summary = html_escape(summary) html += f"""

{escaped_summary}

""" html += """
""" html += """
""" html += """
由 TrendRadar 生成 · GitHub 开源项目
""" return html ================================================ FILE: trendradar/storage/__init__.py ================================================ # coding=utf-8 """ 存储模块 - 支持多种存储后端 支持的存储后端: - local: 本地 SQLite + TXT/HTML 文件 - remote: 远程云存储（S3 兼容协议：R2/OSS/COS/S3 等） - auto: 根据环境自动选择（GitHub Actions 用 remote，其他用 local） """ from trendradar.storage.base import ( StorageBackend, NewsItem, NewsData, RSSItem, RSSData, convert_crawl_results_to_news_data, ) from trendradar.storage.sqlite_mixin import SQLiteStorageMixin from trendradar.storage.local import LocalStorageBackend from trendradar.storage.manager import StorageManager, get_storage_manager # 远程后端可选导入（需要 boto3） try: from trendradar.storage.remote import RemoteStorageBackend HAS_REMOTE = True except ImportError: RemoteStorageBackend = None HAS_REMOTE = False __all__ = [ # 基础类 "StorageBackend", "NewsItem", "NewsData", "RSSItem", "RSSData", # Mixin "SQLiteStorageMixin", # 转换函数 "convert_crawl_results_to_news_data", # 后端实现 "LocalStorageBackend", "RemoteStorageBackend", "HAS_REMOTE", # 管理器 "StorageManager", "get_storage_manager", ] ================================================ FILE: trendradar/storage/ai_filter_schema.sql ================================================ -- AI 智能筛选相关表结构 -- 在 news 库中创建，与 news_items 同库 -- ============================================ -- AI 筛选兴趣标签表 -- 存储从用户兴趣描述中 AI 提取的结构化标签 -- 按版本管理，提示词变更时旧版本标记 deprecated -- 支持多兴趣文件隔离（interests_file 区分不同文件的标签集） -- ============================================ CREATE TABLE IF NOT EXISTS ai_filter_tags ( id INTEGER PRIMARY KEY AUTOINCREMENT, tag TEXT NOT NULL, -- 标签名，如 "AI/大模型" description TEXT DEFAULT '', -- 标签描述，AI 分类时参考 priority INTEGER NOT NULL DEFAULT 9999, -- 标签优先级（值越小优先级越高） status TEXT DEFAULT 'active', -- active / deprecated deprecated_at TEXT, -- 废弃时间 version INTEGER NOT NULL, -- 版本号，提示词变更时 +1 prompt_hash TEXT NOT NULL, -- 兴趣描述文件的 hash（格式: filename:md5） interests_file TEXT NOT NULL DEFAULT 'ai_interests.txt', -- 关联的兴趣文件名 created_at TEXT NOT NULL ); -- ============================================ -- AI 筛选分类结果表 -- 每条新闻 × 每个标签 = 一行 -- 引用 news_items.id 或 rss_items.id（通过 source_type 区分） -- ============================================ CREATE TABLE IF NOT EXISTS ai_filter_results ( id INTEGER PRIMARY KEY AUTOINCREMENT, news_item_id INTEGER NOT NULL, -- 引用 news_items.id 或 rss_items.id source_type TEXT NOT NULL DEFAULT 'hotlist', -- hotlist / rss tag_id INTEGER NOT NULL, -- 引用 ai_filter_tags.id relevance_score REAL DEFAULT 0, -- 相关度 0.0 ~ 1.0 status TEXT DEFAULT 'active', -- active / deprecated deprecated_at TEXT, created_at TEXT NOT NULL, UNIQUE(news_item_id, source_type, tag_id) ); -- ============================================ -- AI 筛选已分析新闻记录表 -- 记录所有已被 AI 分析过的新闻（无论匹配与否） -- 用于去重，避免重复发送给 AI 浪费 token -- ============================================ CREATE TABLE IF NOT EXISTS ai_filter_analyzed_news ( news_item_id INTEGER NOT NULL, -- 引用 news_items.id 或 rss_items.id source_type TEXT NOT NULL DEFAULT 'hotlist', -- hotlist / rss interests_file TEXT NOT NULL DEFAULT 'ai_interests.txt', -- 关联的兴趣文件 prompt_hash TEXT NOT NULL, -- 分析时使用的标签集 hash matched INTEGER NOT NULL DEFAULT 0, -- 是否匹配: 0=不匹配, 1=匹配 created_at TEXT NOT NULL, PRIMARY KEY (news_item_id, source_type, interests_file) ); -- ============================================ -- 索引 -- ============================================ CREATE INDEX IF NOT EXISTS idx_ai_filter_tags_status ON ai_filter_tags(status); CREATE INDEX IF NOT EXISTS idx_ai_filter_tags_version ON ai_filter_tags(version); CREATE INDEX IF NOT EXISTS idx_ai_filter_tags_file ON ai_filter_tags(interests_file, status); CREATE INDEX IF NOT EXISTS idx_ai_filter_tags_priority ON ai_filter_tags(interests_file, status, priority); CREATE INDEX IF NOT EXISTS idx_ai_filter_results_status ON ai_filter_results(status); CREATE INDEX IF NOT EXISTS idx_ai_filter_results_news ON ai_filter_results(news_item_id, source_type); CREATE INDEX IF NOT EXISTS idx_ai_filter_results_tag ON ai_filter_results(tag_id); CREATE INDEX IF NOT EXISTS idx_analyzed_news_lookup ON ai_filter_analyzed_news(source_type, interests_file); CREATE INDEX IF NOT EXISTS idx_analyzed_news_hash ON ai_filter_analyzed_news(interests_file, prompt_hash); ================================================ FILE: trendradar/storage/base.py ================================================ # coding=utf-8 """ 存储后端抽象基类和数据模型 定义统一的存储接口，所有存储后端都需要实现这些方法 """ from abc import ABC, abstractmethod from dataclasses import dataclass, field from typing import Dict, List, Optional, Any, Set @dataclass class NewsItem: """新闻条目数据模型（热榜数据）""" title: str # 新闻标题 source_id: str # 来源平台ID（如 toutiao, baidu） source_name: str = "" # 来源平台名称（运行时使用，数据库不存储） rank: int = 0 # 排名 url: str = "" # 链接 URL mobile_url: str = "" # 移动端 URL crawl_time: str = "" # 抓取时间（HH:MM 格式） # 统计信息（用于分析） ranks: List[int] = field(default_factory=list) # 历史排名列表 first_time: str = "" # 首次出现时间 last_time: str = "" # 最后出现时间 count: int = 1 # 出现次数 rank_timeline: List[Dict[str, Any]] = field(default_factory=list) # 完整排名时间线 # 格式: [{"time": "09:30", "rank": 1}, {"time": "10:00", "rank": 2}, ...] # None 表示脱榜: [{"time": "11:00", "rank": None}] def to_dict(self) -> Dict[str, Any]: """转换为字典""" return { "title": self.title, "source_id": self.source_id, "source_name": self.source_name, "rank": self.rank, "url": self.url, "mobile_url": self.mobile_url, "crawl_time": self.crawl_time, "ranks": self.ranks, "first_time": self.first_time, "last_time": self.last_time, "count": self.count, "rank_timeline": self.rank_timeline, } @classmethod def from_dict(cls, data: Dict[str, Any]) -> "NewsItem": """从字典创建""" return cls( title=data.get("title", ""), source_id=data.get("source_id", ""), source_name=data.get("source_name", ""), rank=data.get("rank", 0), url=data.get("url", ""), mobile_url=data.get("mobile_url", ""), crawl_time=data.get("crawl_time", ""), ranks=data.get("ranks", []), first_time=data.get("first_time", ""), last_time=data.get("last_time", ""), count=data.get("count", 1), rank_timeline=data.get("rank_timeline", []), ) @dataclass class RSSItem: """RSS 条目数据模型""" title: str # 标题 feed_id: str # RSS 源 ID（如 "hacker-news"） feed_name: str = "" # RSS 源名称（运行时使用） url: str = "" # 文章链接 published_at: str = "" # RSS 发布时间（ISO 格式） summary: str = "" # 摘要/描述 author: str = "" # 作者 crawl_time: str = "" # 抓取时间（HH:MM 格式） # 统计信息 first_time: str = "" # 首次抓取时间 last_time: str = "" # 最后抓取时间 count: int = 1 # 抓取次数 def to_dict(self) -> Dict[str, Any]: """转换为字典""" return { "title": self.title, "feed_id": self.feed_id, "feed_name": self.feed_name, "url": self.url, "published_at": self.published_at, "summary": self.summary, "author": self.author, "crawl_time": self.crawl_time, "first_time": self.first_time, "last_time": self.last_time, "count": self.count, } @classmethod def from_dict(cls, data: Dict[str, Any]) -> "RSSItem": """从字典创建""" return cls( title=data.get("title", ""), feed_id=data.get("feed_id", ""), feed_name=data.get("feed_name", ""), url=data.get("url", ""), published_at=data.get("published_at", ""), summary=data.get("summary", ""), author=data.get("author", ""), crawl_time=data.get("crawl_time", ""), first_time=data.get("first_time", ""), last_time=data.get("last_time", ""), count=data.get("count", 1), ) @dataclass class RSSData: """ RSS 数据集合 结构: - date: 日期（YYYY-MM-DD） - crawl_time: 抓取时间（HH:MM） - items: 按 feed_id 分组的 RSS 条目 - id_to_name: feed_id 到名称的映射 - failed_ids: 失败的 feed_id 列表 """ date: str # 日期 crawl_time: str # 抓取时间 items: Dict[str, List[RSSItem]] # 按 feed_id 分组的条目 id_to_name: Dict[str, str] = field(default_factory=dict) # ID到名称映射 failed_ids: List[str] = field(default_factory=list) # 失败的ID def to_dict(self) -> Dict[str, Any]: """转换为字典""" items_dict = {} for feed_id, rss_list in self.items.items(): items_dict[feed_id] = [item.to_dict() for item in rss_list] return { "date": self.date, "crawl_time": self.crawl_time, "items": items_dict, "id_to_name": self.id_to_name, "failed_ids": self.failed_ids, } @classmethod def from_dict(cls, data: Dict[str, Any]) -> "RSSData": """从字典创建""" items = {} items_data = data.get("items", {}) for feed_id, rss_list in items_data.items(): items[feed_id] = [RSSItem.from_dict(item) for item in rss_list] return cls( date=data.get("date", ""), crawl_time=data.get("crawl_time", ""), items=items, id_to_name=data.get("id_to_name", {}), failed_ids=data.get("failed_ids", []), ) def get_total_count(self) -> int: """获取条目总数""" return sum(len(rss_list) for rss_list in self.items.values()) @dataclass class NewsData: """ 新闻数据集合 结构: - date: 日期（YYYY-MM-DD） - crawl_time: 抓取时间（HH时MM分） - items: 按来源ID分组的新闻条目 - id_to_name: 来源ID到名称的映射 - failed_ids: 失败的来源ID列表 """ date: str # 日期 crawl_time: str # 抓取时间 items: Dict[str, List[NewsItem]] # 按来源分组的新闻 id_to_name: Dict[str, str] = field(default_factory=dict) # ID到名称映射 failed_ids: List[str] = field(default_factory=list) # 失败的ID def to_dict(self) -> Dict[str, Any]: """转换为字典""" items_dict = {} for source_id, news_list in self.items.items(): items_dict[source_id] = [item.to_dict() for item in news_list] return { "date": self.date, "crawl_time": self.crawl_time, "items": items_dict, "id_to_name": self.id_to_name, "failed_ids": self.failed_ids, } @classmethod def from_dict(cls, data: Dict[str, Any]) -> "NewsData": """从字典创建""" items = {} items_data = data.get("items", {}) for source_id, news_list in items_data.items(): items[source_id] = [NewsItem.from_dict(item) for item in news_list] return cls( date=data.get("date", ""), crawl_time=data.get("crawl_time", ""), items=items, id_to_name=data.get("id_to_name", {}), failed_ids=data.get("failed_ids", []), ) def get_total_count(self) -> int: """获取新闻总数""" return sum(len(news_list) for news_list in self.items.values()) def merge_with(self, other: "NewsData") -> "NewsData": """ 合并另一个 NewsData 到当前数据 合并规则: - 相同 source_id + title 的新闻合并排名历史 - 更新 last_time 和 count - 保留较早的 first_time """ merged_items = {} # 复制当前数据 for source_id, news_list in self.items.items(): merged_items[source_id] = {item.title: item for item in news_list} # 合并其他数据 for source_id, news_list in other.items.items(): if source_id not in merged_items: merged_items[source_id] = {} for item in news_list: if item.title in merged_items[source_id]: # 合并已存在的新闻 existing = merged_items[source_id][item.title] # 合并排名 existing_ranks = set(existing.ranks) if existing.ranks else set() new_ranks = set(item.ranks) if item.ranks else set() merged_ranks = sorted(existing_ranks | new_ranks) existing.ranks = merged_ranks # 更新时间 if item.first_time and (not existing.first_time or item.first_time < existing.first_time): existing.first_time = item.first_time if item.last_time and (not existing.last_time or item.last_time > existing.last_time): existing.last_time = item.last_time # 更新计数 existing.count += 1 # 保留URL（如果原来没有） if not existing.url and item.url: existing.url = item.url if not existing.mobile_url and item.mobile_url: existing.mobile_url = item.mobile_url else: # 添加新新闻 merged_items[source_id][item.title] = item # 转换回列表格式 final_items = {} for source_id, items_dict in merged_items.items(): final_items[source_id] = list(items_dict.values()) # 合并 id_to_name merged_id_to_name = {**self.id_to_name, **other.id_to_name} # 合并 failed_ids（去重） merged_failed_ids = list(set(self.failed_ids + other.failed_ids)) return NewsData( date=self.date or other.date, crawl_time=other.crawl_time, # 使用较新的抓取时间 items=final_items, id_to_name=merged_id_to_name, failed_ids=merged_failed_ids, ) class StorageBackend(ABC): """ 存储后端抽象基类 所有存储后端都需要实现这些方法，以支持: - 保存新闻数据 - 读取当天所有数据 - 检测新增新闻 - 生成报告文件（TXT/HTML） """ @abstractmethod def save_news_data(self, data: NewsData) -> bool: """ 保存新闻数据 Args: data: 新闻数据 Returns: 是否保存成功 """ pass @abstractmethod def get_today_all_data(self, date: Optional[str] = None) -> Optional[NewsData]: """ 获取指定日期的所有新闻数据 Args: date: 日期字符串（YYYY-MM-DD），默认为今天 Returns: 合并后的新闻数据，如果没有数据返回 None """ pass @abstractmethod def get_latest_crawl_data(self, date: Optional[str] = None) -> Optional[NewsData]: """ 获取最新一次抓取的数据 Args: date: 日期字符串，默认为今天 Returns: 最新抓取的新闻数据 """ pass @abstractmethod def detect_new_titles(self, current_data: NewsData) -> Dict[str, Dict]: """ 检测新增的标题 Args: current_data: 当前抓取的数据 Returns: 新增的标题数据，格式: {source_id: {title: title_data}} """ pass @abstractmethod def save_txt_snapshot(self, data: NewsData) -> Optional[str]: """ 保存 TXT 快照（可选功能，本地环境可用） Args: data: 新闻数据 Returns: 保存的文件路径，如果不支持返回 None """ pass @abstractmethod def save_html_report(self, html_content: str, filename: str) -> Optional[str]: """ 保存 HTML 报告 Args: html_content: HTML 内容 filename: 文件名 Returns: 保存的文件路径 """ pass @abstractmethod def is_first_crawl_today(self, date: Optional[str] = None) -> bool: """ 检查是否是当天第一次抓取 Args: date: 日期字符串，默认为今天 Returns: 是否是第一次抓取 """ pass @abstractmethod def cleanup(self) -> None: """ 清理资源（如临时文件、数据库连接等） """ pass @abstractmethod def cleanup_old_data(self, retention_days: int) -> int: """ 清理过期数据 Args: retention_days: 保留天数（0 表示不清理） Returns: 删除的日期目录数量 """ pass @property @abstractmethod def backend_name(self) -> str: """ 存储后端名称 """ pass @property @abstractmethod def supports_txt(self) -> bool: """ 是否支持生成 TXT 快照 """ pass # === 时间段执行记录（调度系统）=== def has_period_executed(self, date_str: str, period_key: str, action: str) -> bool: """ 检查指定时间段的某个 action 是否已执行 Args: date_str: 日期字符串 YYYY-MM-DD period_key: 时间段 key action: 动作类型 (analyze / push) Returns: 是否已执行 """ return False def record_period_execution(self, date_str: str, period_key: str, action: str) -> bool: """ 记录时间段的 action 执行 Args: date_str: 日期字符串 YYYY-MM-DD period_key: 时间段 key action: 动作类型 (analyze / push) Returns: 是否记录成功 """ return False # === AI 智能筛选（默认实现，子类通过 mixin 覆盖） === def begin_batch(self) -> None: """开启批量模式（远程后端延迟上传，本地后端无操作）""" pass def end_batch(self) -> None: """结束批量模式""" pass def get_active_ai_filter_tags(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> List[Dict]: return [] def get_latest_prompt_hash(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> Optional[str]: return None def get_latest_ai_filter_tag_version(self, date: Optional[str] = None) -> int: return 0 def deprecate_all_ai_filter_tags(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def save_ai_filter_tags(self, tags: List[Dict], version: int, prompt_hash: str, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def save_ai_filter_results(self, results: List[Dict], date: Optional[str] = None) -> int: return 0 def get_active_ai_filter_results(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> List[Dict]: return [] def deprecate_specific_ai_filter_tags(self, tag_ids: List[int], date: Optional[str] = None) -> int: return 0 def update_ai_filter_tags_hash(self, interests_file: str, new_hash: str, date: Optional[str] = None) -> int: return 0 def update_ai_filter_tag_descriptions(self, tag_updates: List[Dict], date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def update_ai_filter_tag_priorities(self, tag_priorities: List[Dict], date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def save_analyzed_news(self, news_ids: List[str], source_type: str, interests_file: str, prompt_hash: str, matched_ids: Set[str], date: Optional[str] = None) -> int: return 0 def get_analyzed_news_ids(self, source_type: str = "hotlist", date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> Set[str]: return set() def clear_analyzed_news(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def clear_unmatched_analyzed_news(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: return 0 def get_all_news_ids(self, date: Optional[str] = None) -> List[Dict]: return [] def get_all_rss_ids(self, date: Optional[str] = None) -> List[Dict]: return [] def convert_crawl_results_to_news_data( results: Dict[str, Dict], id_to_name: Dict[str, str], failed_ids: List[str], crawl_time: str, crawl_date: str, ) -> NewsData: """ 将爬虫结果转换为 NewsData 格式 Args: results: 爬虫返回的结果 {source_id: {title: {ranks: [], url: "", mobileUrl: ""}}} id_to_name: 来源ID到名称的映射 failed_ids: 失败的来源ID crawl_time: 抓取时间（HH:MM） crawl_date: 抓取日期（YYYY-MM-DD） Returns: NewsData 对象 """ items = {} for source_id, titles_data in results.items(): source_name = id_to_name.get(source_id, source_id) news_list = [] for title, data in titles_data.items(): ranks = data.get("ranks", []) url = data.get("url", "") mobile_url = data.get("mobileUrl", "") rank = ranks[0] if ranks else 99 news_item = NewsItem( title=title, source_id=source_id, source_name=source_name, rank=rank, url=url, mobile_url=mobile_url, crawl_time=crawl_time, ranks=ranks, first_time=crawl_time, last_time=crawl_time, count=1, ) news_list.append(news_item) items[source_id] = news_list return NewsData( date=crawl_date, crawl_time=crawl_time, items=items, id_to_name=id_to_name, failed_ids=failed_ids, ) ================================================ FILE: trendradar/storage/local.py ================================================ # coding=utf-8 """ 本地存储后端 - SQLite + TXT/HTML 使用 SQLite 作为主存储，支持可选的 TXT 快照和 HTML 报告 """ import sqlite3 import shutil import pytz import re from datetime import datetime, timedelta from pathlib import Path from typing import Dict, List, Optional from trendradar.storage.base import StorageBackend, NewsData, RSSItem, RSSData from trendradar.storage.sqlite_mixin import SQLiteStorageMixin from trendradar.utils.time import ( DEFAULT_TIMEZONE, get_configured_time, format_date_folder, format_time_filename, ) class LocalStorageBackend(SQLiteStorageMixin, StorageBackend): """ 本地存储后端 使用 SQLite 数据库存储新闻数据，支持： - 按日期组织的 SQLite 数据库文件 - 可选的 TXT 快照（用于调试） - HTML 报告生成 """ def __init__( self, data_dir: str = "output", enable_txt: bool = True, enable_html: bool = True, timezone: str = DEFAULT_TIMEZONE, ): """ 初始化本地存储后端 Args: data_dir: 数据目录路径 enable_txt: 是否启用 TXT 快照 enable_html: 是否启用 HTML 报告 timezone: 时区配置 """ self.data_dir = Path(data_dir) self.enable_txt = enable_txt self.enable_html = enable_html self.timezone = timezone self._db_connections: Dict[str, sqlite3.Connection] = {} @property def backend_name(self) -> str: return "local" @property def supports_txt(self) -> bool: return self.enable_txt # ======================================== # SQLiteStorageMixin 抽象方法实现 # ======================================== def _get_configured_time(self) -> datetime: """获取配置时区的当前时间""" return get_configured_time(self.timezone) def _format_date_folder(self, date: Optional[str] = None) -> str: """格式化日期文件夹名 (ISO 格式: YYYY-MM-DD)""" return format_date_folder(date, self.timezone) def _format_time_filename(self) -> str: """格式化时间文件名 (格式: HH-MM)""" return format_time_filename(self.timezone) def _get_db_path(self, date: Optional[str] = None, db_type: str = "news") -> Path: """ 获取 SQLite 数据库路径 新结构（扁平）：output/{type}/{date}.db - output/news/2025-12-28.db - output/rss/2025-12-28.db Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 数据库文件路径 """ date_str = self._format_date_folder(date) db_dir = self.data_dir / db_type db_dir.mkdir(parents=True, exist_ok=True) return db_dir / f"{date_str}.db" def _get_connection(self, date: Optional[str] = None, db_type: str = "news") -> sqlite3.Connection: """ 获取数据库连接（带缓存） Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 数据库连接 """ db_path = str(self._get_db_path(date, db_type)) if db_path not in self._db_connections: conn = sqlite3.connect(db_path) conn.row_factory = sqlite3.Row self._init_tables(conn, db_type) self._db_connections[db_path] = conn return self._db_connections[db_path] # ======================================== # StorageBackend 接口实现（委托给 mixin） # ======================================== def save_news_data(self, data: NewsData) -> bool: """保存新闻数据到 SQLite""" db_path = self._get_db_path(data.date) if not db_path.exists(): # 确保目录存在 db_path.parent.mkdir(parents=True, exist_ok=True) success, new_count, updated_count, title_changed_count, off_list_count = \ self._save_news_data_impl(data, "[本地存储]") if success: # 输出详细的存储统计日志 log_parts = [f"[本地存储] 处理完成：新增 {new_count} 条"] if updated_count > 0: log_parts.append(f"更新 {updated_count} 条") if title_changed_count > 0: log_parts.append(f"标题变更 {title_changed_count} 条") if off_list_count > 0: log_parts.append(f"脱榜 {off_list_count} 条") print("，".join(log_parts)) return success def get_today_all_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取指定日期的所有新闻数据（合并后）""" db_path = self._get_db_path(date) if not db_path.exists(): return None return self._get_today_all_data_impl(date) def get_latest_crawl_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取最新一次抓取的数据""" db_path = self._get_db_path(date) if not db_path.exists(): return None return self._get_latest_crawl_data_impl(date) def detect_new_titles(self, current_data: NewsData) -> Dict[str, Dict]: """检测新增的标题""" return self._detect_new_titles_impl(current_data) def is_first_crawl_today(self, date: Optional[str] = None) -> bool: """检查是否是当天第一次抓取""" db_path = self._get_db_path(date) if not db_path.exists(): return True return self._is_first_crawl_today_impl(date) def get_crawl_times(self, date: Optional[str] = None) -> List[str]: """获取指定日期的所有抓取时间列表""" db_path = self._get_db_path(date) if not db_path.exists(): return [] return self._get_crawl_times_impl(date) # ======================================== # 时间段执行记录（调度系统） # ======================================== def has_period_executed(self, date_str: str, period_key: str, action: str) -> bool: """检查指定时间段的某个 action 是否已执行""" return self._has_period_executed_impl(date_str, period_key, action) def record_period_execution(self, date_str: str, period_key: str, action: str) -> bool: """记录时间段的 action 执行""" success = self._record_period_execution_impl(date_str, period_key, action) if success: now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") print(f"[本地存储] 时间段执行记录已保存: {period_key}/{action} at {now_str}") return success # ======================================== # RSS 数据存储方法 # ======================================== def save_rss_data(self, data: RSSData) -> bool: """保存 RSS 数据到 SQLite""" success, new_count, updated_count = self._save_rss_data_impl(data, "[本地存储]") if success: # 输出统计日志 log_parts = [f"[本地存储] RSS 处理完成：新增 {new_count} 条"] if updated_count > 0: log_parts.append(f"更新 {updated_count} 条") print("，".join(log_parts)) return success def get_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取指定日期的所有 RSS 数据""" return self._get_rss_data_impl(date) def detect_new_rss_items(self, current_data: RSSData) -> Dict[str, List[RSSItem]]: """检测新增的 RSS 条目""" return self._detect_new_rss_items_impl(current_data) def get_latest_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取最新一次抓取的 RSS 数据""" db_path = self._get_db_path(date, db_type="rss") if not db_path.exists(): return None return self._get_latest_rss_data_impl(date) # ======================================== # AI 智能筛选 # ======================================== def get_active_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): return self._get_active_tags_impl(date, interests_file) def get_latest_prompt_hash(self, date=None, interests_file="ai_interests.txt"): return self._get_latest_prompt_hash_impl(date, interests_file) def get_latest_ai_filter_tag_version(self, date=None): return self._get_latest_tag_version_impl(date) def deprecate_all_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): return self._deprecate_all_tags_impl(date, interests_file) def save_ai_filter_tags(self, tags, version, prompt_hash, date=None, interests_file="ai_interests.txt"): return self._save_tags_impl(date, tags, version, prompt_hash, interests_file) def save_ai_filter_results(self, results, date=None): return self._save_filter_results_impl(date, results) def get_active_ai_filter_results(self, date=None, interests_file="ai_interests.txt"): return self._get_active_filter_results_impl(date, interests_file) def deprecate_specific_ai_filter_tags(self, tag_ids, date=None): return self._deprecate_specific_tags_impl(date, tag_ids) def update_ai_filter_tags_hash(self, interests_file, new_hash, date=None): return self._update_tags_hash_impl(date, interests_file, new_hash) def update_ai_filter_tag_descriptions(self, tag_updates, date=None, interests_file="ai_interests.txt"): return self._update_tag_descriptions_impl(date, tag_updates, interests_file) def update_ai_filter_tag_priorities(self, tag_priorities, date=None, interests_file="ai_interests.txt"): return self._update_tag_priorities_impl(date, tag_priorities, interests_file) def save_analyzed_news(self, news_ids, source_type, interests_file, prompt_hash, matched_ids, date=None): return self._save_analyzed_news_impl(date, news_ids, source_type, interests_file, prompt_hash, matched_ids) def get_analyzed_news_ids(self, source_type="hotlist", date=None, interests_file="ai_interests.txt"): return self._get_analyzed_news_ids_impl(date, source_type, interests_file) def clear_analyzed_news(self, date=None, interests_file="ai_interests.txt"): return self._clear_analyzed_news_impl(date, interests_file) def clear_unmatched_analyzed_news(self, date=None, interests_file="ai_interests.txt"): return self._clear_unmatched_analyzed_news_impl(date, interests_file) def get_all_news_ids(self, date=None): return self._get_all_news_ids_impl(date) def get_all_rss_ids(self, date=None): return self._get_all_rss_ids_impl(date) # ======================================== # 本地特有功能：TXT/HTML 快照 # ======================================== def save_txt_snapshot(self, data: NewsData) -> Optional[str]: """ 保存 TXT 快照 新结构：output/txt/{date}/{time}.txt Args: data: 新闻数据 Returns: 保存的文件路径 """ if not self.enable_txt: return None try: date_folder = self._format_date_folder(data.date) txt_dir = self.data_dir / "txt" / date_folder txt_dir.mkdir(parents=True, exist_ok=True) file_path = txt_dir / f"{data.crawl_time}.txt" with open(file_path, "w", encoding="utf-8") as f: for source_id, news_list in data.items.items(): source_name = data.id_to_name.get(source_id, source_id) # 写入来源标题 if source_name and source_name != source_id: f.write(f"{source_id} | {source_name}\n") else: f.write(f"{source_id}\n") # 按排名排序 sorted_news = sorted(news_list, key=lambda x: x.rank) for item in sorted_news: line = f"{item.rank}. {item.title}" if item.url: line += f" [URL:{item.url}]" if item.mobile_url: line += f" [MOBILE:{item.mobile_url}]" f.write(line + "\n") f.write("\n") # 写入失败的来源 if data.failed_ids: f.write("==== 以下ID请求失败 ====\n") for failed_id in data.failed_ids: f.write(f"{failed_id}\n") print(f"[本地存储] TXT 快照已保存: {file_path}") return str(file_path) except Exception as e: print(f"[本地存储] 保存 TXT 快照失败: {e}") return None def save_html_report(self, html_content: str, filename: str) -> Optional[str]: """ 保存 HTML 报告 新结构：output/html/{date}/{filename} Args: html_content: HTML 内容 filename: 文件名 Returns: 保存的文件路径 """ if not self.enable_html: return None try: date_folder = self._format_date_folder() html_dir = self.data_dir / "html" / date_folder html_dir.mkdir(parents=True, exist_ok=True) file_path = html_dir / filename with open(file_path, "w", encoding="utf-8") as f: f.write(html_content) print(f"[本地存储] HTML 报告已保存: {file_path}") return str(file_path) except Exception as e: print(f"[本地存储] 保存 HTML 报告失败: {e}") return None # ======================================== # 本地特有功能：资源清理 # ======================================== def cleanup(self) -> None: """清理资源（关闭数据库连接）""" for db_path, conn in self._db_connections.items(): try: conn.close() print(f"[本地存储] 关闭数据库连接: {db_path}") except Exception as e: print(f"[本地存储] 关闭连接失败 {db_path}: {e}") self._db_connections.clear() def cleanup_old_data(self, retention_days: int) -> int: """ 清理过期数据 新结构清理逻辑： - output/news/{date}.db -> 删除过期的 .db 文件 - output/rss/{date}.db -> 删除过期的 .db 文件 - output/txt/{date}/ -> 删除过期的日期目录 - output/html/{date}/ -> 删除过期的日期目录 Args: retention_days: 保留天数（0 表示不清理） Returns: 删除的文件/目录数量 """ if retention_days <= 0: return 0 deleted_count = 0 cutoff_date = self._get_configured_time() - timedelta(days=retention_days) def parse_date_from_name(name: str) -> Optional[datetime]: """从文件名或目录名解析日期 (ISO 格式: YYYY-MM-DD)""" # 移除 .db 后缀 name = name.replace('.db', '') try: date_match = re.match(r'(\d{4})-(\d{2})-(\d{2})', name) if date_match: return datetime( int(date_match.group(1)), int(date_match.group(2)), int(date_match.group(3)), tzinfo=pytz.timezone(self.timezone) ) except Exception: pass return None try: if not self.data_dir.exists(): return 0 # 清理数据库文件 (news/, rss/) for db_type in ["news", "rss"]: db_dir = self.data_dir / db_type if not db_dir.exists(): continue for db_file in db_dir.glob("*.db"): file_date = parse_date_from_name(db_file.name) if file_date and file_date < cutoff_date: # 先关闭数据库连接 db_path = str(db_file) if db_path in self._db_connections: try: self._db_connections[db_path].close() del self._db_connections[db_path] except Exception: pass # 删除文件 try: db_file.unlink() deleted_count += 1 print(f"[本地存储] 清理过期数据: {db_type}/{db_file.name}") except Exception as e: print(f"[本地存储] 删除文件失败 {db_file}: {e}") # 清理快照目录 (txt/, html/) for snapshot_type in ["txt", "html"]: snapshot_dir = self.data_dir / snapshot_type if not snapshot_dir.exists(): continue for date_folder in snapshot_dir.iterdir(): if not date_folder.is_dir() or date_folder.name.startswith('.'): continue folder_date = parse_date_from_name(date_folder.name) if folder_date and folder_date < cutoff_date: try: shutil.rmtree(date_folder) deleted_count += 1 print(f"[本地存储] 清理过期数据: {snapshot_type}/{date_folder.name}") except Exception as e: print(f"[本地存储] 删除目录失败 {date_folder}: {e}") if deleted_count > 0: print(f"[本地存储] 共清理 {deleted_count} 个过期文件/目录") return deleted_count except Exception as e: print(f"[本地存储] 清理过期数据失败: {e}") return deleted_count def __del__(self): """析构函数，确保关闭连接""" self.cleanup() ================================================ FILE: trendradar/storage/manager.py ================================================ # coding=utf-8 """ 存储管理器 - 统一管理存储后端 根据环境和配置自动选择合适的存储后端 """ import os from typing import Optional from trendradar.storage.base import StorageBackend, NewsData, RSSData from trendradar.utils.time import DEFAULT_TIMEZONE # 存储管理器单例 _storage_manager: Optional["StorageManager"] = None class StorageManager: """ 存储管理器 功能： - 自动检测运行环境（GitHub Actions / Docker / 本地） - 根据配置选择存储后端（local / remote / auto） - 提供统一的存储接口 - 支持从远程拉取数据到本地 """ def __init__( self, backend_type: str = "auto", data_dir: str = "output", enable_txt: bool = True, enable_html: bool = True, remote_config: Optional[dict] = None, local_retention_days: int = 0, remote_retention_days: int = 0, pull_enabled: bool = False, pull_days: int = 0, timezone: str = DEFAULT_TIMEZONE, ): """ 初始化存储管理器 Args: backend_type: 存储后端类型 (local / remote / auto) data_dir: 本地数据目录 enable_txt: 是否启用 TXT 快照 enable_html: 是否启用 HTML 报告 remote_config: 远程存储配置（endpoint_url, bucket_name, access_key_id 等） local_retention_days: 本地数据保留天数（0 = 无限制） remote_retention_days: 远程数据保留天数（0 = 无限制） pull_enabled: 是否启用启动时自动拉取 pull_days: 拉取最近 N 天的数据 timezone: 时区配置 """ self.backend_type = backend_type self.data_dir = data_dir self.enable_txt = enable_txt self.enable_html = enable_html self.remote_config = remote_config or {} self.local_retention_days = local_retention_days self.remote_retention_days = remote_retention_days self.pull_enabled = pull_enabled self.pull_days = pull_days self.timezone = timezone self._backend: Optional[StorageBackend] = None self._remote_backend: Optional[StorageBackend] = None @staticmethod def is_github_actions() -> bool: """检测是否在 GitHub Actions 环境中运行""" return os.environ.get("GITHUB_ACTIONS") == "true" @staticmethod def is_docker() -> bool: """检测是否在 Docker 容器中运行""" # 方法1: 检查 /.dockerenv 文件 if os.path.exists("/.dockerenv"): return True # 方法2: 检查 cgroup（Linux） try: with open("/proc/1/cgroup", "r") as f: return "docker" in f.read() except (FileNotFoundError, PermissionError): pass # 方法3: 检查环境变量 return os.environ.get("DOCKER_CONTAINER") == "true" def _resolve_backend_type(self) -> str: """解析实际使用的后端类型""" if self.backend_type == "auto": if self.is_github_actions(): # GitHub Actions 环境，检查是否配置了远程存储 if self._has_remote_config(): return "remote" else: print("[存储管理器] GitHub Actions 环境但未配置远程存储，使用本地存储") return "local" else: return "local" return self.backend_type def _has_remote_config(self) -> bool: """检查是否有有效的远程存储配置""" # 检查配置或环境变量 bucket_name = self.remote_config.get("bucket_name") or os.environ.get("S3_BUCKET_NAME") access_key = self.remote_config.get("access_key_id") or os.environ.get("S3_ACCESS_KEY_ID") secret_key = self.remote_config.get("secret_access_key") or os.environ.get("S3_SECRET_ACCESS_KEY") endpoint = self.remote_config.get("endpoint_url") or os.environ.get("S3_ENDPOINT_URL") # 调试日志 has_config = bool(bucket_name and access_key and secret_key and endpoint) if not has_config: print(f"[存储管理器] 远程存储配置检查失败:") print(f" - bucket_name: {'已配置' if bucket_name else '未配置'}") print(f" - access_key_id: {'已配置' if access_key else '未配置'}") print(f" - secret_access_key: {'已配置' if secret_key else '未配置'}") print(f" - endpoint_url: {'已配置' if endpoint else '未配置'}") return has_config def _create_remote_backend(self) -> Optional[StorageBackend]: """创建远程存储后端""" try: from trendradar.storage.remote import RemoteStorageBackend return RemoteStorageBackend( bucket_name=self.remote_config.get("bucket_name") or os.environ.get("S3_BUCKET_NAME", ""), access_key_id=self.remote_config.get("access_key_id") or os.environ.get("S3_ACCESS_KEY_ID", ""), secret_access_key=self.remote_config.get("secret_access_key") or os.environ.get("S3_SECRET_ACCESS_KEY", ""), endpoint_url=self.remote_config.get("endpoint_url") or os.environ.get("S3_ENDPOINT_URL", ""), region=self.remote_config.get("region") or os.environ.get("S3_REGION", ""), enable_txt=self.enable_txt, enable_html=self.enable_html, timezone=self.timezone, ) except ImportError as e: print(f"[存储管理器] 远程后端导入失败: {e}") print("[存储管理器] 请确保已安装 boto3: pip install boto3") return None except Exception as e: print(f"[存储管理器] 远程后端初始化失败: {e}") return None def get_backend(self) -> StorageBackend: """获取存储后端实例""" if self._backend is None: resolved_type = self._resolve_backend_type() if resolved_type == "remote": self._backend = self._create_remote_backend() if self._backend: print(f"[存储管理器] 使用远程存储后端") else: print("[存储管理器] 回退到本地存储") resolved_type = "local" if resolved_type == "local" or self._backend is None: from trendradar.storage.local import LocalStorageBackend self._backend = LocalStorageBackend( data_dir=self.data_dir, enable_txt=self.enable_txt, enable_html=self.enable_html, timezone=self.timezone, ) print(f"[存储管理器] 使用本地存储后端 (数据目录: {self.data_dir})") return self._backend def pull_from_remote(self) -> int: """ 从远程拉取数据到本地 Returns: 成功拉取的文件数量 """ if not self.pull_enabled or self.pull_days <= 0: return 0 if not self._has_remote_config(): print("[存储管理器] 未配置远程存储，无法拉取") return 0 # 创建远程后端（如果还没有） if self._remote_backend is None: self._remote_backend = self._create_remote_backend() if self._remote_backend is None: print("[存储管理器] 无法创建远程后端，拉取失败") return 0 # 调用拉取方法 return self._remote_backend.pull_recent_days(self.pull_days, self.data_dir) def save_news_data(self, data: NewsData) -> bool: """保存新闻数据""" return self.get_backend().save_news_data(data) def save_rss_data(self, data: RSSData) -> bool: """保存 RSS 数据""" return self.get_backend().save_rss_data(data) def get_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取指定日期的所有 RSS 数据（当日汇总模式）""" return self.get_backend().get_rss_data(date) def get_latest_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取最新一次抓取的 RSS 数据（当前榜单模式）""" return self.get_backend().get_latest_rss_data(date) def detect_new_rss_items(self, current_data: RSSData) -> dict: """检测新增的 RSS 条目（增量模式）""" return self.get_backend().detect_new_rss_items(current_data) def get_today_all_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取当天所有数据""" return self.get_backend().get_today_all_data(date) def get_latest_crawl_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取最新抓取数据""" return self.get_backend().get_latest_crawl_data(date) def detect_new_titles(self, current_data: NewsData) -> dict: """检测新增标题""" return self.get_backend().detect_new_titles(current_data) def save_txt_snapshot(self, data: NewsData) -> Optional[str]: """保存 TXT 快照""" return self.get_backend().save_txt_snapshot(data) def save_html_report(self, html_content: str, filename: str) -> Optional[str]: """保存 HTML 报告""" return self.get_backend().save_html_report(html_content, filename) def is_first_crawl_today(self, date: Optional[str] = None) -> bool: """检查是否是当天第一次抓取""" return self.get_backend().is_first_crawl_today(date) def cleanup(self) -> None: """清理资源""" if self._backend: self._backend.cleanup() if self._remote_backend: self._remote_backend.cleanup() def cleanup_old_data(self) -> int: """ 清理过期数据 Returns: 删除的日期目录数量 """ total_deleted = 0 # 清理本地数据 if self.local_retention_days > 0: total_deleted += self.get_backend().cleanup_old_data(self.local_retention_days) # 清理远程数据（如果配置了） if self.remote_retention_days > 0 and self._has_remote_config(): if self._remote_backend is None: self._remote_backend = self._create_remote_backend() if self._remote_backend: total_deleted += self._remote_backend.cleanup_old_data(self.remote_retention_days) return total_deleted @property def backend_name(self) -> str: """获取当前后端名称""" return self.get_backend().backend_name @property def supports_txt(self) -> bool: """是否支持 TXT 快照""" return self.get_backend().supports_txt def has_period_executed(self, date_str: str, period_key: str, action: str) -> bool: """检查指定时间段的某个 action 是否已执行""" return self.get_backend().has_period_executed(date_str, period_key, action) def record_period_execution(self, date_str: str, period_key: str, action: str) -> bool: """记录时间段的 action 执行""" return self.get_backend().record_period_execution(date_str, period_key, action) # === AI 智能筛选存储操作 === def begin_batch(self): """开启批量模式（远程后端延迟上传）""" self.get_backend().begin_batch() def end_batch(self): """结束批量模式（统一上传脏数据库）""" self.get_backend().end_batch() def get_active_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): """获取指定兴趣文件的 active 标签""" return self.get_backend().get_active_ai_filter_tags(date, interests_file) def get_latest_prompt_hash(self, date=None, interests_file="ai_interests.txt"): """获取指定兴趣文件的最新 prompt_hash""" return self.get_backend().get_latest_prompt_hash(date, interests_file) def get_latest_ai_filter_tag_version(self, date=None): """获取最新标签版本号""" return self.get_backend().get_latest_ai_filter_tag_version(date) def deprecate_all_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): """废弃指定兴趣文件的 active 标签和分类结果""" return self.get_backend().deprecate_all_ai_filter_tags(date, interests_file) def save_ai_filter_tags(self, tags, version, prompt_hash, date=None, interests_file="ai_interests.txt"): """保存新提取的标签""" return self.get_backend().save_ai_filter_tags(tags, version, prompt_hash, date, interests_file) def save_ai_filter_results(self, results, date=None): """保存分类结果""" return self.get_backend().save_ai_filter_results(results, date) def get_active_ai_filter_results(self, date=None, interests_file="ai_interests.txt"): """获取指定兴趣文件的 active 分类结果""" return self.get_backend().get_active_ai_filter_results(date, interests_file) def deprecate_specific_ai_filter_tags(self, tag_ids, date=None): """废弃指定 ID 的标签及其关联分类结果""" return self.get_backend().deprecate_specific_ai_filter_tags(tag_ids, date) def update_ai_filter_tags_hash(self, interests_file, new_hash, date=None): """更新指定兴趣文件所有 active 标签的 prompt_hash""" return self.get_backend().update_ai_filter_tags_hash(interests_file, new_hash, date) def update_ai_filter_tag_descriptions(self, tag_updates, date=None, interests_file="ai_interests.txt"): """按 tag 名匹配，更新 active 标签的 description""" return self.get_backend().update_ai_filter_tag_descriptions(tag_updates, date, interests_file) def update_ai_filter_tag_priorities(self, tag_priorities, date=None, interests_file="ai_interests.txt"): """按 tag 名匹配，更新 active 标签的 priority""" return self.get_backend().update_ai_filter_tag_priorities(tag_priorities, date, interests_file) def save_analyzed_news(self, news_ids, source_type, interests_file, prompt_hash, matched_ids, date=None): """批量记录已分析的新闻（匹配与不匹配都记录）""" return self.get_backend().save_analyzed_news(news_ids, source_type, interests_file, prompt_hash, matched_ids, date) def get_analyzed_news_ids(self, source_type="hotlist", date=None, interests_file="ai_interests.txt"): """获取已分析过的新闻 ID 集合""" return self.get_backend().get_analyzed_news_ids(source_type, date, interests_file) def clear_analyzed_news(self, date=None, interests_file="ai_interests.txt"): """清除指定兴趣文件的所有已分析记录""" return self.get_backend().clear_analyzed_news(date, interests_file) def clear_unmatched_analyzed_news(self, date=None, interests_file="ai_interests.txt"): """清除不匹配的已分析记录""" return self.get_backend().clear_unmatched_analyzed_news(date, interests_file) def get_all_news_ids(self, date=None): """获取所有新闻 ID 和标题""" return self.get_backend().get_all_news_ids(date) def get_all_rss_ids(self, date=None): """获取所有 RSS ID 和标题""" return self.get_backend().get_all_rss_ids(date) def get_storage_manager( backend_type: str = "auto", data_dir: str = "output", enable_txt: bool = True, enable_html: bool = True, remote_config: Optional[dict] = None, local_retention_days: int = 0, remote_retention_days: int = 0, pull_enabled: bool = False, pull_days: int = 0, timezone: str = DEFAULT_TIMEZONE, force_new: bool = False, ) -> StorageManager: """ 获取存储管理器单例 Args: backend_type: 存储后端类型 data_dir: 本地数据目录 enable_txt: 是否启用 TXT 快照 enable_html: 是否启用 HTML 报告 remote_config: 远程存储配置 local_retention_days: 本地数据保留天数（0 = 无限制） remote_retention_days: 远程数据保留天数（0 = 无限制） pull_enabled: 是否启用启动时自动拉取 pull_days: 拉取最近 N 天的数据 timezone: 时区配置 force_new: 是否强制创建新实例 Returns: StorageManager 实例 """ global _storage_manager if _storage_manager is None or force_new: _storage_manager = StorageManager( backend_type=backend_type, data_dir=data_dir, enable_txt=enable_txt, enable_html=enable_html, remote_config=remote_config, local_retention_days=local_retention_days, remote_retention_days=remote_retention_days, pull_enabled=pull_enabled, pull_days=pull_days, timezone=timezone, ) return _storage_manager ================================================ FILE: trendradar/storage/remote.py ================================================ # coding=utf-8 """ 远程存储后端（S3 兼容协议） 支持 Cloudflare R2、阿里云 OSS、腾讯云 COS、AWS S3、MinIO 等 使用 S3 兼容 API (boto3) 访问对象存储 数据流程：下载当天 SQLite → 合并新数据 → 上传回远程 """ import pytz import re import shutil import sys import tempfile import sqlite3 from datetime import datetime, timedelta from pathlib import Path from typing import Dict, List, Optional try: import boto3 from botocore.config import Config as BotoConfig from botocore.exceptions import ClientError HAS_BOTO3 = True except ImportError: HAS_BOTO3 = False boto3 = None BotoConfig = None ClientError = Exception from trendradar.storage.base import StorageBackend, NewsData, RSSItem, RSSData from trendradar.storage.sqlite_mixin import SQLiteStorageMixin from trendradar.utils.time import ( DEFAULT_TIMEZONE, get_configured_time, format_date_folder, format_time_filename, ) class RemoteStorageBackend(SQLiteStorageMixin, StorageBackend): """ 远程云存储后端（S3 兼容协议） 特点： - 使用 S3 兼容 API 访问远程存储 - 支持 Cloudflare R2、阿里云 OSS、腾讯云 COS、AWS S3、MinIO 等 - 下载 SQLite 到临时目录进行操作 - 支持数据合并和上传 - 支持从远程拉取历史数据到本地 - 运行结束后自动清理临时文件 """ def __init__( self, bucket_name: str, access_key_id: str, secret_access_key: str, endpoint_url: str, region: str = "", enable_txt: bool = False, # 远程模式默认不生成 TXT enable_html: bool = True, temp_dir: Optional[str] = None, timezone: str = DEFAULT_TIMEZONE, ): """ 初始化远程存储后端 Args: bucket_name: 存储桶名称 access_key_id: 访问密钥 ID secret_access_key: 访问密钥 endpoint_url: 服务端点 URL region: 区域（可选，部分服务商需要） enable_txt: 是否启用 TXT 快照（默认关闭） enable_html: 是否启用 HTML 报告 temp_dir: 临时目录路径（默认使用系统临时目录） timezone: 时区配置 """ if not HAS_BOTO3: raise ImportError("远程存储后端需要安装 boto3: pip install boto3") self.bucket_name = bucket_name self.endpoint_url = endpoint_url self.region = region self.enable_txt = enable_txt self.enable_html = enable_html self.timezone = timezone # 创建临时目录 self.temp_dir = Path(temp_dir) if temp_dir else Path(tempfile.mkdtemp(prefix="trendradar_")) self.temp_dir.mkdir(parents=True, exist_ok=True) # 初始化 S3 客户端 # 使用 virtual-hosted style addressing（主流） # 根据服务商选择签名版本： # - 腾讯云 COS 和 阿里云 OSS 使用 SigV2 以避免 chunked encoding 问题 # - 其他服务商（AWS S3、Cloudflare R2、MinIO 等）默认使用 SigV4 use_sigv2 = "myqcloud.com" in endpoint_url.lower() or "aliyuncs.com" in endpoint_url.lower() signature_version = 's3' if use_sigv2 else 's3v4' s3_config = BotoConfig( s3={"addressing_style": "virtual"}, signature_version=signature_version, ) client_kwargs = { "endpoint_url": endpoint_url, "aws_access_key_id": access_key_id, "aws_secret_access_key": secret_access_key, "config": s3_config, } if region: client_kwargs["region_name"] = region self.s3_client = boto3.client("s3", **client_kwargs) # 跟踪下载的文件（用于清理） self._downloaded_files: List[Path] = [] self._db_connections: Dict[str, sqlite3.Connection] = {} # 批量模式：延迟上传，避免频繁上传同一文件 self._batch_mode = False self._batch_dirty: set = set() # 待上传的 (date, db_type) 集合 print(f"[远程存储] 初始化完成，存储桶: {bucket_name}，签名版本: {signature_version}") @property def backend_name(self) -> str: return "remote" @property def supports_txt(self) -> bool: return self.enable_txt # ======================================== # SQLiteStorageMixin 抽象方法实现 # ======================================== def _get_configured_time(self) -> datetime: """获取配置时区的当前时间""" return get_configured_time(self.timezone) def _format_date_folder(self, date: Optional[str] = None) -> str: """格式化日期文件夹名 (ISO 格式: YYYY-MM-DD)""" return format_date_folder(date, self.timezone) def _format_time_filename(self) -> str: """格式化时间文件名 (格式: HH-MM)""" return format_time_filename(self.timezone) def _get_remote_db_key(self, date: Optional[str] = None, db_type: str = "news") -> str: """ 获取远程存储中 SQLite 文件的对象键 Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 远程对象键，如 "news/2025-12-28.db" 或 "rss/2025-12-28.db" """ date_folder = self._format_date_folder(date) return f"{db_type}/{date_folder}.db" def _get_local_db_path(self, date: Optional[str] = None, db_type: str = "news") -> Path: """ 获取本地临时 SQLite 文件路径 Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 本地临时文件路径 """ date_folder = self._format_date_folder(date) db_dir = self.temp_dir / db_type db_dir.mkdir(parents=True, exist_ok=True) return db_dir / f"{date_folder}.db" def _check_object_exists(self, r2_key: str) -> bool: """ 检查远程存储中对象是否存在 Args: r2_key: 远程对象键 Returns: 是否存在 """ try: self.s3_client.head_object(Bucket=self.bucket_name, Key=r2_key) return True except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") # S3 兼容存储可能返回 404, NoSuchKey, 或其他变体 if error_code in ("404", "NoSuchKey", "Not Found"): return False # 其他错误（如权限问题）也视为不存在，但打印警告 print(f"[远程存储] 检查对象存在性失败 ({r2_key}): {e}") return False except Exception as e: print(f"[远程存储] 检查对象存在性异常 ({r2_key}): {e}") return False def _download_sqlite(self, date: Optional[str] = None, db_type: str = "news") -> Optional[Path]: """ 从远程存储下载当天的 SQLite 文件到本地临时目录 使用 get_object + iter_chunks 替代 download_file， 以正确处理腾讯云 COS 的 chunked transfer encoding。 Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 本地文件路径，如果不存在返回 None """ r2_key = self._get_remote_db_key(date, db_type) local_path = self._get_local_db_path(date, db_type) # 确保目录存在 local_path.parent.mkdir(parents=True, exist_ok=True) # 先检查文件是否存在 if not self._check_object_exists(r2_key): print(f"[远程存储] 文件不存在，将创建新数据库: {r2_key}") return None try: # 使用 get_object + iter_chunks 替代 download_file # iter_chunks 会自动处理 chunked transfer encoding response = self.s3_client.get_object(Bucket=self.bucket_name, Key=r2_key) with open(local_path, 'wb') as f: for chunk in response['Body'].iter_chunks(chunk_size=1024*1024): f.write(chunk) self._downloaded_files.append(local_path) print(f"[远程存储] 已下载: {r2_key} -> {local_path}") return local_path except ClientError as e: error_code = e.response.get("Error", {}).get("Code", "") # S3 兼容存储可能返回不同的错误码 if error_code in ("404", "NoSuchKey", "Not Found"): print(f"[远程存储] 文件不存在，将创建新数据库: {r2_key}") return None else: print(f"[远程存储] 下载失败 (错误码: {error_code}): {e}") raise except Exception as e: print(f"[远程存储] 下载异常: {e}") raise def begin_batch(self): """开启批量模式：延迟上传，避免频繁上传同一文件""" self._batch_mode = True self._batch_dirty.clear() def end_batch(self): """结束批量模式：统一上传所有脏数据库""" self._batch_mode = False for date, db_type in self._batch_dirty: self._upload_sqlite(date, db_type) self._batch_dirty.clear() def _upload_sqlite(self, date: Optional[str] = None, db_type: str = "news") -> bool: """ 上传本地 SQLite 文件到远程存储 批量模式下延迟上传，由 end_batch() 统一触发。 Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 是否上传成功 """ if self._batch_mode: self._batch_dirty.add((date, db_type)) return True local_path = self._get_local_db_path(date, db_type) r2_key = self._get_remote_db_key(date, db_type) if not local_path.exists(): print(f"[远程存储] 本地文件不存在，无法上传: {local_path}") return False try: # 获取本地文件大小 local_size = local_path.stat().st_size print(f"[远程存储] 准备上传: {local_path} ({local_size} bytes) -> {r2_key}") # 读取文件内容为 bytes 后上传 # 避免传入文件对象时 requests 库使用 chunked transfer encoding # 腾讯云 COS 等 S3 兼容服务可能无法正确处理 chunked encoding with open(local_path, 'rb') as f: file_content = f.read() # 使用 put_object 并明确设置 ContentLength，确保不使用 chunked encoding self.s3_client.put_object( Bucket=self.bucket_name, Key=r2_key, Body=file_content, ContentLength=local_size, ContentType='application/x-sqlite3', ) print(f"[远程存储] 已上传: {local_path} -> {r2_key}") # 验证上传成功 if self._check_object_exists(r2_key): print(f"[远程存储] 上传验证成功: {r2_key}") return True else: print(f"[远程存储] 上传验证失败: 文件未在远程存储中找到") return False except Exception as e: print(f"[远程存储] 上传失败: {e}") return False def _get_connection(self, date: Optional[str] = None, db_type: str = "news") -> sqlite3.Connection: """ 获取数据库连接 Args: date: 日期字符串 db_type: 数据库类型 ("news" 或 "rss") Returns: 数据库连接 """ local_path = self._get_local_db_path(date, db_type) db_path = str(local_path) if db_path not in self._db_connections: # 确保目录存在 local_path.parent.mkdir(parents=True, exist_ok=True) # 如果本地不存在，尝试从远程存储下载 if not local_path.exists(): self._download_sqlite(date, db_type) conn = sqlite3.connect(db_path) conn.row_factory = sqlite3.Row self._init_tables(conn, db_type) self._db_connections[db_path] = conn return self._db_connections[db_path] # ======================================== # StorageBackend 接口实现（委托给 mixin + 上传） # ======================================== def save_news_data(self, data: NewsData) -> bool: """ 保存新闻数据到远程存储 流程：下载现有数据库 → 插入/更新数据 → 上传回远程存储 """ # 查询已有记录数 conn = self._get_connection(data.date) cursor = conn.cursor() cursor.execute("SELECT COUNT(*) as count FROM news_items") row = cursor.fetchone() existing_count = row[0] if row else 0 if existing_count > 0: print(f"[远程存储] 已有 {existing_count} 条历史记录，将合并新数据") # 使用 mixin 的实现保存数据 success, new_count, updated_count, title_changed_count, off_list_count = \ self._save_news_data_impl(data, "[远程存储]") if not success: return False # 查询合并后的总记录数 cursor.execute("SELECT COUNT(*) as count FROM news_items") row = cursor.fetchone() final_count = row[0] if row else 0 # 输出详细的存储统计日志 log_parts = [f"[远程存储] 处理完成：新增 {new_count} 条"] if updated_count > 0: log_parts.append(f"更新 {updated_count} 条") if title_changed_count > 0: log_parts.append(f"标题变更 {title_changed_count} 条") if off_list_count > 0: log_parts.append(f"脱榜 {off_list_count} 条") log_parts.append(f"(去重后总计: {final_count} 条)") print("，".join(log_parts)) # 上传到远程存储 if self._upload_sqlite(data.date): print(f"[远程存储] 数据已同步到远程存储") return True else: print(f"[远程存储] 上传远程存储失败") return False def get_today_all_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取指定日期的所有新闻数据（合并后）""" return self._get_today_all_data_impl(date) def get_latest_crawl_data(self, date: Optional[str] = None) -> Optional[NewsData]: """获取最新一次抓取的数据""" return self._get_latest_crawl_data_impl(date) def detect_new_titles(self, current_data: NewsData) -> Dict[str, Dict]: """检测新增的标题""" return self._detect_new_titles_impl(current_data) def is_first_crawl_today(self, date: Optional[str] = None) -> bool: """检查是否是当天第一次抓取""" return self._is_first_crawl_today_impl(date) # ======================================== # 时间段执行记录（调度系统） # ======================================== def has_period_executed(self, date_str: str, period_key: str, action: str) -> bool: """检查指定时间段的某个 action 是否已执行""" return self._has_period_executed_impl(date_str, period_key, action) def record_period_execution(self, date_str: str, period_key: str, action: str) -> bool: """记录时间段的 action 执行""" success = self._record_period_execution_impl(date_str, period_key, action) if success: now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") print(f"[远程存储] 时间段执行记录已保存: {period_key}/{action} at {now_str}") # 上传到远程存储确保记录持久化 if self._upload_sqlite(date_str): print(f"[远程存储] 时间段执行记录已同步到远程存储") return True else: print(f"[远程存储] 时间段执行记录同步到远程存储失败") return False return False # ======================================== # RSS 数据存储方法 # ======================================== def save_rss_data(self, data: RSSData) -> bool: """ 保存 RSS 数据到远程存储 流程：下载现有数据库 → 插入/更新数据 → 上传回远程存储 """ success, new_count, updated_count = self._save_rss_data_impl(data, "[远程存储]") if not success: return False # 输出统计日志 log_parts = [f"[远程存储] RSS 处理完成：新增 {new_count} 条"] if updated_count > 0: log_parts.append(f"更新 {updated_count} 条") print("，".join(log_parts)) # 上传到远程存储 if self._upload_sqlite(data.date, db_type="rss"): print(f"[远程存储] RSS 数据已同步到远程存储") return True else: print(f"[远程存储] RSS 上传远程存储失败") return False def get_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取指定日期的所有 RSS 数据""" return self._get_rss_data_impl(date) def detect_new_rss_items(self, current_data: RSSData) -> Dict[str, List[RSSItem]]: """检测新增的 RSS 条目""" return self._detect_new_rss_items_impl(current_data) def get_latest_rss_data(self, date: Optional[str] = None) -> Optional[RSSData]: """获取最新一次抓取的 RSS 数据""" return self._get_latest_rss_data_impl(date) # ======================================== # AI 智能筛选存储方法 # ======================================== def get_active_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): return self._get_active_tags_impl(date, interests_file) def get_latest_prompt_hash(self, date=None, interests_file="ai_interests.txt"): return self._get_latest_prompt_hash_impl(date, interests_file) def get_latest_ai_filter_tag_version(self, date=None): return self._get_latest_tag_version_impl(date) def deprecate_all_ai_filter_tags(self, date=None, interests_file="ai_interests.txt"): count = self._deprecate_all_tags_impl(date, interests_file) if count > 0: self._upload_sqlite(date) return count def save_ai_filter_tags(self, tags, version, prompt_hash, date=None, interests_file="ai_interests.txt"): count = self._save_tags_impl(date, tags, version, prompt_hash, interests_file) if count > 0: self._upload_sqlite(date) return count def save_ai_filter_results(self, results, date=None): count = self._save_filter_results_impl(date, results) if count > 0: self._upload_sqlite(date) return count def get_active_ai_filter_results(self, date=None, interests_file="ai_interests.txt"): return self._get_active_filter_results_impl(date, interests_file) def deprecate_specific_ai_filter_tags(self, tag_ids, date=None): count = self._deprecate_specific_tags_impl(date, tag_ids) if count > 0: self._upload_sqlite(date) return count def update_ai_filter_tags_hash(self, interests_file, new_hash, date=None): count = self._update_tags_hash_impl(date, interests_file, new_hash) if count > 0: self._upload_sqlite(date) return count def update_ai_filter_tag_descriptions(self, tag_updates, date=None, interests_file="ai_interests.txt"): count = self._update_tag_descriptions_impl(date, tag_updates, interests_file) if count > 0: self._upload_sqlite(date) return count def update_ai_filter_tag_priorities(self, tag_priorities, date=None, interests_file="ai_interests.txt"): count = self._update_tag_priorities_impl(date, tag_priorities, interests_file) if count > 0: self._upload_sqlite(date) return count def save_analyzed_news(self, news_ids, source_type, interests_file, prompt_hash, matched_ids, date=None): count = self._save_analyzed_news_impl(date, news_ids, source_type, interests_file, prompt_hash, matched_ids) if count > 0: self._upload_sqlite(date) return count def get_analyzed_news_ids(self, source_type="hotlist", date=None, interests_file="ai_interests.txt"): return self._get_analyzed_news_ids_impl(date, source_type, interests_file) def clear_analyzed_news(self, date=None, interests_file="ai_interests.txt"): count = self._clear_analyzed_news_impl(date, interests_file) if count > 0: self._upload_sqlite(date) return count def clear_unmatched_analyzed_news(self, date=None, interests_file="ai_interests.txt"): count = self._clear_unmatched_analyzed_news_impl(date, interests_file) if count > 0: self._upload_sqlite(date) return count def get_all_news_ids(self, date=None): return self._get_all_news_ids_impl(date) def get_all_rss_ids(self, date=None): return self._get_all_rss_ids_impl(date) # ======================================== # 远程特有功能：TXT/HTML 快照（临时目录） # ======================================== def save_txt_snapshot(self, data: NewsData) -> Optional[str]: """保存 TXT 快照（远程存储模式下默认不支持）""" if not self.enable_txt: return None # 如果启用，保存到本地临时目录 try: date_folder = self._format_date_folder(data.date) txt_dir = self.temp_dir / date_folder / "txt" txt_dir.mkdir(parents=True, exist_ok=True) file_path = txt_dir / f"{data.crawl_time}.txt" with open(file_path, "w", encoding="utf-8") as f: for source_id, news_list in data.items.items(): source_name = data.id_to_name.get(source_id, source_id) if source_name and source_name != source_id: f.write(f"{source_id} | {source_name}\n") else: f.write(f"{source_id}\n") sorted_news = sorted(news_list, key=lambda x: x.rank) for item in sorted_news: line = f"{item.rank}. {item.title}" if item.url: line += f" [URL:{item.url}]" if item.mobile_url: line += f" [MOBILE:{item.mobile_url}]" f.write(line + "\n") f.write("\n") if data.failed_ids: f.write("==== 以下ID请求失败 ====\n") for failed_id in data.failed_ids: f.write(f"{failed_id}\n") print(f"[远程存储] TXT 快照已保存: {file_path}") return str(file_path) except Exception as e: print(f"[远程存储] 保存 TXT 快照失败: {e}") return None def save_html_report(self, html_content: str, filename: str) -> Optional[str]: """保存 HTML 报告到临时目录""" if not self.enable_html: return None try: date_folder = self._format_date_folder() html_dir = self.temp_dir / date_folder / "html" html_dir.mkdir(parents=True, exist_ok=True) file_path = html_dir / filename with open(file_path, "w", encoding="utf-8") as f: f.write(html_content) print(f"[远程存储] HTML 报告已保存: {file_path}") return str(file_path) except Exception as e: print(f"[远程存储] 保存 HTML 报告失败: {e}") return None # ======================================== # 远程特有功能：资源清理 # ======================================== def cleanup(self) -> None: """清理资源（关闭连接和删除临时文件）""" # 检查 Python 是否正在关闭 if sys.meta_path is None: return # 关闭数据库连接 db_connections = getattr(self, "_db_connections", {}) for db_path, conn in list(db_connections.items()): try: conn.close() print(f"[远程存储] 关闭数据库连接: {db_path}") except Exception as e: print(f"[远程存储] 关闭连接失败 {db_path}: {e}") if db_connections: db_connections.clear() # 删除临时目录 temp_dir = getattr(self, "temp_dir", None) if temp_dir: try: if temp_dir.exists(): shutil.rmtree(temp_dir) print(f"[远程存储] 临时目录已清理: {temp_dir}") except Exception as e: # 忽略 Python 关闭时的错误 if sys.meta_path is not None: print(f"[远程存储] 清理临时目录失败: {e}") downloaded_files = getattr(self, "_downloaded_files", None) if downloaded_files: downloaded_files.clear() def cleanup_old_data(self, retention_days: int) -> int: """ 清理远程存储上的过期数据 Args: retention_days: 保留天数（0 表示不清理） Returns: 删除的数据库文件数量 """ if retention_days <= 0: return 0 deleted_count = 0 cutoff_date = self._get_configured_time() - timedelta(days=retention_days) try: # 列出远程存储中 news/ 前缀下的所有对象 paginator = self.s3_client.get_paginator('list_objects_v2') pages = paginator.paginate(Bucket=self.bucket_name, Prefix="news/") # 收集需要删除的对象键 objects_to_delete = [] deleted_dates = set() for page in pages: if 'Contents' not in page: continue for obj in page['Contents']: key = obj['Key'] # 解析日期（格式: news/YYYY-MM-DD.db） folder_date = None date_str = None try: date_match = re.match(r'news/(\d{4})-(\d{2})-(\d{2})\.db$', key) if date_match: folder_date = datetime( int(date_match.group(1)), int(date_match.group(2)), int(date_match.group(3)), tzinfo=pytz.timezone(self.timezone) ) date_str = f"{date_match.group(1)}-{date_match.group(2)}-{date_match.group(3)}" except Exception: continue if folder_date and folder_date < cutoff_date: objects_to_delete.append({'Key': key}) deleted_dates.add(date_str) # 批量删除对象（每次最多 1000 个） if objects_to_delete: batch_size = 1000 for i in range(0, len(objects_to_delete), batch_size): batch = objects_to_delete[i:i + batch_size] try: self.s3_client.delete_objects( Bucket=self.bucket_name, Delete={'Objects': batch} ) print(f"[远程存储] 删除 {len(batch)} 个对象") except Exception as e: print(f"[远程存储] 批量删除失败: {e}") deleted_count = len(deleted_dates) for date_str in sorted(deleted_dates): print(f"[远程存储] 清理过期数据: news/{date_str}.db") print(f"[远程存储] 共清理 {deleted_count} 个过期日期数据库文件") return deleted_count except Exception as e: print(f"[远程存储] 清理过期数据失败: {e}") return deleted_count def __del__(self): """析构函数""" # 检查 Python 是否正在关闭 if sys.meta_path is None: return try: self.cleanup() except Exception: # Python 关闭时可能会出错，忽略即可 pass # ======================================== # 远程特有功能：数据拉取和列表 # ======================================== def pull_recent_days(self, days: int, local_data_dir: str = "output") -> int: """ 从远程拉取最近 N 天的数据到本地 Args: days: 拉取天数 local_data_dir: 本地数据目录 Returns: 成功拉取的数据库文件数量 """ if days <= 0: return 0 local_dir = Path(local_data_dir) local_dir.mkdir(parents=True, exist_ok=True) pulled_count = 0 now = self._get_configured_time() print(f"[远程存储] 开始拉取最近 {days} 天的数据...") for i in range(days): date = now - timedelta(days=i) date_str = date.strftime("%Y-%m-%d") # 本地目标路径 local_date_dir = local_dir / date_str local_db_path = local_date_dir / "news.db" # 如果本地已存在，跳过 if local_db_path.exists(): print(f"[远程存储] 跳过（本地已存在）: {date_str}") continue # 远程对象键 remote_key = f"news/{date_str}.db" # 检查远程是否存在 if not self._check_object_exists(remote_key): print(f"[远程存储] 跳过（远程不存在）: {date_str}") continue # 下载（使用 get_object + iter_chunks 处理 chunked encoding） try: local_date_dir.mkdir(parents=True, exist_ok=True) response = self.s3_client.get_object(Bucket=self.bucket_name, Key=remote_key) with open(local_db_path, 'wb') as f: for chunk in response['Body'].iter_chunks(chunk_size=1024*1024): f.write(chunk) print(f"[远程存储] 已拉取: {remote_key} -> {local_db_path}") pulled_count += 1 except Exception as e: print(f"[远程存储] 拉取失败 ({date_str}): {e}") print(f"[远程存储] 拉取完成，共下载 {pulled_count} 个数据库文件") return pulled_count def list_remote_dates(self) -> List[str]: """ 列出远程存储中所有可用的日期 Returns: 日期字符串列表（YYYY-MM-DD 格式） """ dates = [] try: paginator = self.s3_client.get_paginator('list_objects_v2') pages = paginator.paginate(Bucket=self.bucket_name, Prefix="news/") for page in pages: if 'Contents' not in page: continue for obj in page['Contents']: key = obj['Key'] # 解析日期 date_match = re.match(r'news/(\d{4}-\d{2}-\d{2})\.db$', key) if date_match: dates.append(date_match.group(1)) return sorted(dates, reverse=True) except Exception as e: print(f"[远程存储] 列出远程日期失败: {e}") return [] ================================================ FILE: trendradar/storage/rss_schema.sql ================================================ -- TrendRadar RSS 数据库表结构 -- 用于存储 RSS/Atom 订阅源数据 -- ============================================ -- RSS 源配置表 -- 存储订阅源的基本信息 -- ============================================ CREATE TABLE IF NOT EXISTS rss_feeds ( id TEXT PRIMARY KEY, -- 源 ID（如 "hacker-news"） name TEXT NOT NULL, -- 显示名称（如 "Hacker News"） feed_url TEXT DEFAULT '', -- RSS/Atom URL（可选，配置文件中已有） is_active INTEGER DEFAULT 1, -- 是否启用 last_fetch_time TEXT, -- 最后抓取时间 last_fetch_status TEXT, -- 最后抓取状态（success/failed） item_count INTEGER DEFAULT 0, -- 当日条目数 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- ============================================ -- RSS 条目表 -- 以 URL + feed_id 为唯一标识，支持去重存储 -- ============================================ CREATE TABLE IF NOT EXISTS rss_items ( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT NOT NULL, -- 标题 feed_id TEXT NOT NULL, -- 所属 RSS 源 url TEXT NOT NULL, -- 文章链接 published_at TEXT, -- RSS 发布时间（ISO 格式） summary TEXT, -- 摘要/描述 author TEXT, -- 作者 first_crawl_time TEXT NOT NULL, -- 首次抓取时间 last_crawl_time TEXT NOT NULL, -- 最后抓取时间 crawl_count INTEGER DEFAULT 1, -- 抓取次数 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (feed_id) REFERENCES rss_feeds(id) ); -- ============================================ -- 抓取记录表 -- 记录每次抓取的时间和数量 -- ============================================ CREATE TABLE IF NOT EXISTS rss_crawl_records ( id INTEGER PRIMARY KEY AUTOINCREMENT, crawl_time TEXT NOT NULL UNIQUE, -- 抓取时间（HH:MM） total_items INTEGER DEFAULT 0, -- 总条目数 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- ============================================ -- 抓取来源状态表 -- 记录每次抓取各 RSS 源的成功/失败状态 -- ============================================ CREATE TABLE IF NOT EXISTS rss_crawl_status ( crawl_record_id INTEGER NOT NULL, feed_id TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('success', 'failed')), error_message TEXT, -- 失败时的错误信息 PRIMARY KEY (crawl_record_id, feed_id), FOREIGN KEY (crawl_record_id) REFERENCES rss_crawl_records(id), FOREIGN KEY (feed_id) REFERENCES rss_feeds(id) ); -- ============================================ -- 推送记录表 -- 用于 push_window once_per_day 功能 -- 以及 ai_analysis analysis_window once_per_day 功能 -- ============================================ CREATE TABLE IF NOT EXISTS rss_push_records ( id INTEGER PRIMARY KEY AUTOINCREMENT, date TEXT NOT NULL UNIQUE, -- 日期（YYYY-MM-DD） pushed INTEGER DEFAULT 0, -- 是否已推送 push_time TEXT, -- 推送时间 ai_analyzed INTEGER DEFAULT 0, -- 是否已进行 AI 分析 ai_analysis_time TEXT, -- AI 分析时间 ai_analysis_mode TEXT, -- AI 分析模式 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- ============================================ -- 索引定义 -- ============================================ -- RSS 源索引 CREATE INDEX IF NOT EXISTS idx_rss_feed ON rss_items(feed_id); -- 发布时间索引（用于按时间排序） CREATE INDEX IF NOT EXISTS idx_rss_published ON rss_items(published_at DESC); -- 抓取时间索引（用于查询最新数据） CREATE INDEX IF NOT EXISTS idx_rss_crawl_time ON rss_items(last_crawl_time); -- 标题索引（用于标题搜索） CREATE INDEX IF NOT EXISTS idx_rss_title ON rss_items(title); -- URL + feed_id 唯一索引（实现去重） CREATE UNIQUE INDEX IF NOT EXISTS idx_rss_url_feed ON rss_items(url, feed_id); -- 抓取状态索引 CREATE INDEX IF NOT EXISTS idx_rss_crawl_status_record ON rss_crawl_status(crawl_record_id); ================================================ FILE: trendradar/storage/schema.sql ================================================ -- TrendRadar 数据库表结构 -- ============================================ -- 平台信息表 -- 核心：id 不变，name 可变 -- ============================================ CREATE TABLE IF NOT EXISTS platforms ( id TEXT PRIMARY KEY, name TEXT NOT NULL, is_active INTEGER DEFAULT 1, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- ============================================ -- 新闻条目表 -- 以 URL + platform_id 为唯一标识，支持去重存储 -- ============================================ CREATE TABLE IF NOT EXISTS news_items ( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT NOT NULL, platform_id TEXT NOT NULL, rank INTEGER NOT NULL, url TEXT DEFAULT '', mobile_url TEXT DEFAULT '', first_crawl_time TEXT NOT NULL, -- 首次抓取时间 last_crawl_time TEXT NOT NULL, -- 最后抓取时间 crawl_count INTEGER DEFAULT 1, -- 抓取次数 created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (platform_id) REFERENCES platforms(id) ); -- ============================================ -- 标题变更历史表 -- 记录同一 URL 下标题的变化 -- ============================================ CREATE TABLE IF NOT EXISTS title_changes ( id INTEGER PRIMARY KEY AUTOINCREMENT, news_item_id INTEGER NOT NULL, old_title TEXT NOT NULL, new_title TEXT NOT NULL, changed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (news_item_id) REFERENCES news_items(id) ); -- ============================================ -- 排名历史表 -- 记录每次抓取时的排名变化 -- ============================================ CREATE TABLE IF NOT EXISTS rank_history ( id INTEGER PRIMARY KEY AUTOINCREMENT, news_item_id INTEGER NOT NULL, rank INTEGER NOT NULL, crawl_time TEXT NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, FOREIGN KEY (news_item_id) REFERENCES news_items(id) ); -- ============================================ -- 抓取记录表 -- 记录每次抓取的时间和数量 -- ============================================ CREATE TABLE IF NOT EXISTS crawl_records ( id INTEGER PRIMARY KEY AUTOINCREMENT, crawl_time TEXT NOT NULL UNIQUE, total_items INTEGER DEFAULT 0, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); -- ============================================ -- 抓取来源状态表 -- 记录每次抓取各平台的成功/失败状态 -- ============================================ CREATE TABLE IF NOT EXISTS crawl_source_status ( crawl_record_id INTEGER NOT NULL, platform_id TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('success', 'failed')), PRIMARY KEY (crawl_record_id, platform_id), FOREIGN KEY (crawl_record_id) REFERENCES crawl_records(id), FOREIGN KEY (platform_id) REFERENCES platforms(id) ); -- ============================================ -- 时间段执行记录表 -- 记录每天每个时间段在各 action 维度的执行状态（用于 once 功能） -- 替代旧的 push_records 表 -- ============================================ CREATE TABLE IF NOT EXISTS period_executions ( id INTEGER PRIMARY KEY AUTOINCREMENT, execution_date TEXT NOT NULL, -- YYYY-MM-DD period_key TEXT NOT NULL, -- period 的稳定 key action TEXT NOT NULL, -- analyze | push executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, UNIQUE(execution_date, period_key, action) ); -- ============================================ -- 索引定义 -- ============================================ -- 平台索引 CREATE INDEX IF NOT EXISTS idx_news_platform ON news_items(platform_id); -- 时间索引（用于查询最新数据） CREATE INDEX IF NOT EXISTS idx_news_crawl_time ON news_items(last_crawl_time); -- 标题索引（用于标题搜索） CREATE INDEX IF NOT EXISTS idx_news_title ON news_items(title); -- URL + platform_id 唯一索引（仅对非空 URL，实现去重） CREATE UNIQUE INDEX IF NOT EXISTS idx_news_url_platform ON news_items(url, platform_id) WHERE url != ''; -- 抓取状态索引 CREATE INDEX IF NOT EXISTS idx_crawl_status_record ON crawl_source_status(crawl_record_id); -- 排名历史索引 CREATE INDEX IF NOT EXISTS idx_rank_history_news ON rank_history(news_item_id); -- 时间段执行记录索引 CREATE INDEX IF NOT EXISTS idx_period_exec_lookup ON period_executions(execution_date, period_key, action); ================================================ FILE: trendradar/storage/sqlite_mixin.py ================================================ # coding=utf-8 """ SQLite 存储 Mixin 提供共用的 SQLite 数据库操作逻辑，供 LocalStorageBackend 和 RemoteStorageBackend 复用。 """ import sqlite3 from abc import abstractmethod from datetime import datetime from pathlib import Path from typing import Any, Dict, List, Optional from trendradar.storage.base import NewsItem, NewsData, RSSItem, RSSData from trendradar.utils.url import normalize_url class SQLiteStorageMixin: """ SQLite 存储操作 Mixin 子类需要实现以下抽象方法： - _get_connection(date, db_type) -> sqlite3.Connection - _get_configured_time() -> datetime - _format_date_folder(date) -> str - _format_time_filename() -> str """ # ======================================== # 抽象方法 - 子类必须实现 # ======================================== @abstractmethod def _get_connection(self, date: Optional[str] = None, db_type: str = "news") -> sqlite3.Connection: """获取数据库连接""" pass @abstractmethod def _get_configured_time(self) -> datetime: """获取配置时区的当前时间""" pass @abstractmethod def _format_date_folder(self, date: Optional[str] = None) -> str: """格式化日期文件夹名 (ISO 格式: YYYY-MM-DD)""" pass @abstractmethod def _format_time_filename(self) -> str: """格式化时间文件名 (格式: HH-MM)""" pass # ======================================== # Schema 管理 # ======================================== def _get_schema_path(self, db_type: str = "news") -> Path: """ 获取 schema.sql 文件路径 Args: db_type: 数据库类型 ("news" 或 "rss") Returns: schema 文件路径 """ if db_type == "rss": return Path(__file__).parent / "rss_schema.sql" return Path(__file__).parent / "schema.sql" def _get_ai_filter_schema_path(self) -> Path: """获取 AI 筛选 schema 文件路径""" return Path(__file__).parent / "ai_filter_schema.sql" def _init_tables(self, conn: sqlite3.Connection, db_type: str = "news") -> None: """ 从 schema.sql 初始化数据库表结构 Args: conn: 数据库连接 db_type: 数据库类型 ("news" 或 "rss") """ schema_path = self._get_schema_path(db_type) if schema_path.exists(): with open(schema_path, "r", encoding="utf-8") as f: schema_sql = f.read() conn.executescript(schema_sql) else: raise FileNotFoundError(f"Schema file not found: {schema_path}") # news 库额外加载 AI 筛选表结构 if db_type == "news": ai_filter_schema = self._get_ai_filter_schema_path() if ai_filter_schema.exists(): with open(ai_filter_schema, "r", encoding="utf-8") as f: conn.executescript(f.read()) conn.commit() # ======================================== # 新闻数据存储 # ======================================== def _save_news_data_impl(self, data: NewsData, log_prefix: str = "[存储]") -> tuple[bool, int, int, int, int]: """ 保存新闻数据到 SQLite（核心实现） Args: data: 新闻数据 log_prefix: 日志前缀 Returns: (success, new_count, updated_count, title_changed_count, off_list_count) """ try: conn = self._get_connection(data.date) cursor = conn.cursor() # 获取配置时区的当前时间 now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") # 首先同步平台信息到 platforms 表 for source_id, source_name in data.id_to_name.items(): cursor.execute(""" INSERT INTO platforms (id, name, updated_at) VALUES (?, ?, ?) ON CONFLICT(id) DO UPDATE SET name = excluded.name, updated_at = excluded.updated_at """, (source_id, source_name, now_str)) # 统计计数器 new_count = 0 updated_count = 0 title_changed_count = 0 success_sources = [] for source_id, news_list in data.items.items(): success_sources.append(source_id) for item in news_list: try: # 标准化 URL（去除动态参数，如微博的 band_rank） normalized_url = normalize_url(item.url, source_id) if item.url else "" # 检查是否已存在（通过标准化 URL + platform_id） if normalized_url: cursor.execute(""" SELECT id, title FROM news_items WHERE url = ? AND platform_id = ? """, (normalized_url, source_id)) existing = cursor.fetchone() if existing: # 已存在，更新记录 existing_id, existing_title = existing # 检查标题是否变化 if existing_title != item.title: # 记录标题变更 cursor.execute(""" INSERT INTO title_changes (news_item_id, old_title, new_title, changed_at) VALUES (?, ?, ?, ?) """, (existing_id, existing_title, item.title, now_str)) title_changed_count += 1 # 记录排名历史 cursor.execute(""" INSERT INTO rank_history (news_item_id, rank, crawl_time, created_at) VALUES (?, ?, ?, ?) """, (existing_id, item.rank, data.crawl_time, now_str)) # 更新现有记录 cursor.execute(""" UPDATE news_items SET title = ?, rank = ?, mobile_url = ?, last_crawl_time = ?, crawl_count = crawl_count + 1, updated_at = ? WHERE id = ? """, (item.title, item.rank, item.mobile_url, data.crawl_time, now_str, existing_id)) updated_count += 1 else: # 不存在，插入新记录（存储标准化后的 URL） cursor.execute(""" INSERT INTO news_items (title, platform_id, rank, url, mobile_url, first_crawl_time, last_crawl_time, crawl_count, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?) """, (item.title, source_id, item.rank, normalized_url, item.mobile_url, data.crawl_time, data.crawl_time, now_str, now_str)) new_id = cursor.lastrowid # 记录初始排名 cursor.execute(""" INSERT INTO rank_history (news_item_id, rank, crawl_time, created_at) VALUES (?, ?, ?, ?) """, (new_id, item.rank, data.crawl_time, now_str)) new_count += 1 else: # URL 为空的情况，直接插入（不做去重） cursor.execute(""" INSERT INTO news_items (title, platform_id, rank, url, mobile_url, first_crawl_time, last_crawl_time, crawl_count, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, 1, ?, ?) """, (item.title, source_id, item.rank, "", item.mobile_url, data.crawl_time, data.crawl_time, now_str, now_str)) new_id = cursor.lastrowid # 记录初始排名 cursor.execute(""" INSERT INTO rank_history (news_item_id, rank, crawl_time, created_at) VALUES (?, ?, ?, ?) """, (new_id, item.rank, data.crawl_time, now_str)) new_count += 1 except sqlite3.Error as e: print(f"{log_prefix} 保存新闻条目失败 [{item.title[:30]}...]: {e}") total_items = new_count + updated_count # ======================================== # 脱榜检测：检测上次在榜但这次不在榜的新闻 # ======================================== off_list_count = 0 # 获取上一次抓取时间 cursor.execute(""" SELECT crawl_time FROM crawl_records WHERE crawl_time < ? ORDER BY crawl_time DESC LIMIT 1 """, (data.crawl_time,)) prev_record = cursor.fetchone() if prev_record: prev_crawl_time = prev_record[0] # 对于每个成功抓取的平台，检测脱榜 for source_id in success_sources: # 获取当前抓取中该平台的所有标准化 URL current_urls = set() for item in data.items.get(source_id, []): normalized_url = normalize_url(item.url, source_id) if item.url else "" if normalized_url: current_urls.add(normalized_url) # 查询上次在榜（last_crawl_time = prev_crawl_time）但这次不在榜的新闻 # 这些新闻是"第一次脱榜"，需要记录 cursor.execute(""" SELECT id, url FROM news_items WHERE platform_id = ? AND last_crawl_time = ? AND url != '' """, (source_id, prev_crawl_time)) for row in cursor.fetchall(): news_id, url = row[0], row[1] if url not in current_urls: # 插入脱榜记录（rank=0 表示脱榜） cursor.execute(""" INSERT INTO rank_history (news_item_id, rank, crawl_time, created_at) VALUES (?, 0, ?, ?) """, (news_id, data.crawl_time, now_str)) off_list_count += 1 # 记录抓取信息 cursor.execute(""" INSERT OR REPLACE INTO crawl_records (crawl_time, total_items, created_at) VALUES (?, ?, ?) """, (data.crawl_time, total_items, now_str)) # 获取刚插入的 crawl_record 的 ID cursor.execute(""" SELECT id FROM crawl_records WHERE crawl_time = ? """, (data.crawl_time,)) record_row = cursor.fetchone() if record_row: crawl_record_id = record_row[0] # 记录成功的来源 for source_id in success_sources: cursor.execute(""" INSERT OR REPLACE INTO crawl_source_status (crawl_record_id, platform_id, status) VALUES (?, ?, 'success') """, (crawl_record_id, source_id)) # 记录失败的来源 for failed_id in data.failed_ids: # 确保失败的平台也在 platforms 表中 cursor.execute(""" INSERT OR IGNORE INTO platforms (id, name, updated_at) VALUES (?, ?, ?) """, (failed_id, failed_id, now_str)) cursor.execute(""" INSERT OR REPLACE INTO crawl_source_status (crawl_record_id, platform_id, status) VALUES (?, ?, 'failed') """, (crawl_record_id, failed_id)) conn.commit() return True, new_count, updated_count, title_changed_count, off_list_count except Exception as e: print(f"{log_prefix} 保存失败: {e}") return False, 0, 0, 0, 0 def _get_today_all_data_impl(self, date: Optional[str] = None) -> Optional[NewsData]: """ 获取指定日期的所有新闻数据（合并后） Args: date: 日期字符串，默认为今天 Returns: 合并后的新闻数据 """ try: conn = self._get_connection(date) cursor = conn.cursor() # 获取所有新闻数据（包含 id 用于查询排名历史） cursor.execute(""" SELECT n.id, n.title, n.platform_id, p.name as platform_name, n.rank, n.url, n.mobile_url, n.first_crawl_time, n.last_crawl_time, n.crawl_count FROM news_items n LEFT JOIN platforms p ON n.platform_id = p.id ORDER BY n.platform_id, n.last_crawl_time """) rows = cursor.fetchall() if not rows: return None # 收集所有 news_item_id news_ids = [row[0] for row in rows] # 批量查询排名历史（同时获取时间和排名） # 过滤逻辑：只保留 last_crawl_time 之前的脱榜记录（rank=0） # 这样可以避免显示新闻永久脱榜后的无意义记录 rank_history_map: Dict[int, List[int]] = {} rank_timeline_map: Dict[int, List[Dict[str, Any]]] = {} if news_ids: placeholders = ",".join("?" * len(news_ids)) cursor.execute(f""" SELECT rh.news_item_id, rh.rank, rh.crawl_time FROM rank_history rh JOIN news_items ni ON rh.news_item_id = ni.id WHERE rh.news_item_id IN ({placeholders}) AND NOT (rh.rank = 0 AND rh.crawl_time > ni.last_crawl_time) ORDER BY rh.news_item_id, rh.crawl_time """, news_ids) for rh_row in cursor.fetchall(): news_id, rank, crawl_time = rh_row[0], rh_row[1], rh_row[2] # 构建 ranks 列表（去重，排除脱榜记录 rank=0） if news_id not in rank_history_map: rank_history_map[news_id] = [] if rank != 0 and rank not in rank_history_map[news_id]: rank_history_map[news_id].append(rank) # 构建 rank_timeline 列表（完整时间线，包含脱榜） if news_id not in rank_timeline_map: rank_timeline_map[news_id] = [] # 提取时间部分（HH:MM） time_part = crawl_time.split()[1][:5] if ' ' in crawl_time else crawl_time[:5] rank_timeline_map[news_id].append({ "time": time_part, "rank": rank if rank != 0 else None # 0 转为 None 表示脱榜 }) # 按 platform_id 分组 items: Dict[str, List[NewsItem]] = {} id_to_name: Dict[str, str] = {} crawl_date = self._format_date_folder(date) for row in rows: news_id = row[0] platform_id = row[2] title = row[1] platform_name = row[3] or platform_id id_to_name[platform_id] = platform_name if platform_id not in items: items[platform_id] = [] # 获取排名历史，如果没有则使用当前排名 ranks = rank_history_map.get(news_id, [row[4]]) rank_timeline = rank_timeline_map.get(news_id, []) items[platform_id].append(NewsItem( title=title, source_id=platform_id, source_name=platform_name, rank=row[4], url=row[5] or "", mobile_url=row[6] or "", crawl_time=row[8], # last_crawl_time ranks=ranks, first_time=row[7], # first_crawl_time last_time=row[8], # last_crawl_time count=row[9], # crawl_count rank_timeline=rank_timeline, )) final_items = items # 获取失败的来源 cursor.execute(""" SELECT DISTINCT css.platform_id FROM crawl_source_status css JOIN crawl_records cr ON css.crawl_record_id = cr.id WHERE css.status = 'failed' """) failed_ids = [row[0] for row in cursor.fetchall()] # 获取最新的抓取时间 cursor.execute(""" SELECT crawl_time FROM crawl_records ORDER BY crawl_time DESC LIMIT 1 """) time_row = cursor.fetchone() crawl_time = time_row[0] if time_row else self._format_time_filename() return NewsData( date=crawl_date, crawl_time=crawl_time, items=final_items, id_to_name=id_to_name, failed_ids=failed_ids, ) except Exception as e: print(f"[存储] 读取数据失败: {e}") return None def _get_latest_crawl_data_impl(self, date: Optional[str] = None) -> Optional[NewsData]: """ 获取最新一次抓取的数据 Args: date: 日期字符串，默认为今天 Returns: 最新抓取的新闻数据 """ try: conn = self._get_connection(date) cursor = conn.cursor() # 获取最新的抓取时间 cursor.execute(""" SELECT crawl_time FROM crawl_records ORDER BY crawl_time DESC LIMIT 1 """) time_row = cursor.fetchone() if not time_row: return None latest_time = time_row[0] # 获取该时间的新闻数据（包含 id 用于查询排名历史） cursor.execute(""" SELECT n.id, n.title, n.platform_id, p.name as platform_name, n.rank, n.url, n.mobile_url, n.first_crawl_time, n.last_crawl_time, n.crawl_count FROM news_items n LEFT JOIN platforms p ON n.platform_id = p.id WHERE n.last_crawl_time = ? """, (latest_time,)) rows = cursor.fetchall() if not rows: return None # 收集所有 news_item_id news_ids = [row[0] for row in rows] # 批量查询排名历史（同时获取时间和排名） # 过滤逻辑：只保留 last_crawl_time 之前的脱榜记录（rank=0） # 这样可以避免显示新闻永久脱榜后的无意义记录 rank_history_map: Dict[int, List[int]] = {} rank_timeline_map: Dict[int, List[Dict[str, Any]]] = {} if news_ids: placeholders = ",".join("?" * len(news_ids)) cursor.execute(f""" SELECT rh.news_item_id, rh.rank, rh.crawl_time FROM rank_history rh JOIN news_items ni ON rh.news_item_id = ni.id WHERE rh.news_item_id IN ({placeholders}) AND NOT (rh.rank = 0 AND rh.crawl_time > ni.last_crawl_time) ORDER BY rh.news_item_id, rh.crawl_time """, news_ids) for rh_row in cursor.fetchall(): news_id, rank, crawl_time = rh_row[0], rh_row[1], rh_row[2] # 构建 ranks 列表（去重，排除脱榜记录 rank=0） if news_id not in rank_history_map: rank_history_map[news_id] = [] if rank != 0 and rank not in rank_history_map[news_id]: rank_history_map[news_id].append(rank) # 构建 rank_timeline 列表（完整时间线，包含脱榜） if news_id not in rank_timeline_map: rank_timeline_map[news_id] = [] # 提取时间部分（HH:MM） time_part = crawl_time.split()[1][:5] if ' ' in crawl_time else crawl_time[:5] rank_timeline_map[news_id].append({ "time": time_part, "rank": rank if rank != 0 else None # 0 转为 None 表示脱榜 }) items: Dict[str, List[NewsItem]] = {} id_to_name: Dict[str, str] = {} crawl_date = self._format_date_folder(date) for row in rows: news_id = row[0] platform_id = row[2] platform_name = row[3] or platform_id id_to_name[platform_id] = platform_name if platform_id not in items: items[platform_id] = [] # 获取排名历史，如果没有则使用当前排名 ranks = rank_history_map.get(news_id, [row[4]]) rank_timeline = rank_timeline_map.get(news_id, []) items[platform_id].append(NewsItem( title=row[1], source_id=platform_id, source_name=platform_name, rank=row[4], url=row[5] or "", mobile_url=row[6] or "", crawl_time=row[8], # last_crawl_time ranks=ranks, first_time=row[7], # first_crawl_time last_time=row[8], # last_crawl_time count=row[9], # crawl_count rank_timeline=rank_timeline, )) # 获取失败的来源（针对最新一次抓取） cursor.execute(""" SELECT css.platform_id FROM crawl_source_status css JOIN crawl_records cr ON css.crawl_record_id = cr.id WHERE cr.crawl_time = ? AND css.status = 'failed' """, (latest_time,)) failed_ids = [row[0] for row in cursor.fetchall()] return NewsData( date=crawl_date, crawl_time=latest_time, items=items, id_to_name=id_to_name, failed_ids=failed_ids, ) except Exception as e: print(f"[存储] 获取最新数据失败: {e}") return None def _detect_new_titles_impl(self, current_data: NewsData) -> Dict[str, Dict]: """ 检测新增的标题 该方法比较当前抓取数据与历史数据，找出新增的标题。 关键逻辑：只有在历史批次中从未出现过的标题才算新增。 Args: current_data: 当前抓取的数据 Returns: 新增的标题数据 {source_id: {title: NewsItem}} """ try: # 获取历史数据 historical_data = self._get_today_all_data_impl(current_data.date) if not historical_data: # 没有历史数据，所有都是新的 new_titles = {} for source_id, news_list in current_data.items.items(): new_titles[source_id] = {item.title: item for item in news_list} return new_titles # 获取当前批次时间 current_time = current_data.crawl_time # 收集历史标题（first_time < current_time 的标题） # 这样可以正确处理同一标题因 URL 变化而产生多条记录的情况 historical_titles: Dict[str, set] = {} for source_id, news_list in historical_data.items.items(): historical_titles[source_id] = set() for item in news_list: first_time = item.first_time or item.crawl_time if first_time < current_time: historical_titles[source_id].add(item.title) # 检查是否有历史数据 has_historical_data = any(len(titles) > 0 for titles in historical_titles.values()) if not has_historical_data: # 第一次抓取，没有"新增"概念 return {} # 检测新增 new_titles = {} for source_id, news_list in current_data.items.items(): hist_set = historical_titles.get(source_id, set()) for item in news_list: if item.title not in hist_set: if source_id not in new_titles: new_titles[source_id] = {} new_titles[source_id][item.title] = item return new_titles except Exception as e: print(f"[存储] 检测新标题失败: {e}") return {} def _is_first_crawl_today_impl(self, date: Optional[str] = None) -> bool: """ 检查是否是当天第一次抓取 Args: date: 日期字符串，默认为今天 Returns: 是否是第一次抓取 """ try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT COUNT(*) as count FROM crawl_records """) row = cursor.fetchone() count = row[0] if row else 0 # 如果只有一条或没有记录，视为第一次抓取 return count <= 1 except Exception as e: print(f"[存储] 检查首次抓取失败: {e}") return True def _get_crawl_times_impl(self, date: Optional[str] = None) -> List[str]: """ 获取指定日期的所有抓取时间列表 Args: date: 日期字符串，默认为今天 Returns: 抓取时间列表（按时间排序） """ try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT crawl_time FROM crawl_records ORDER BY crawl_time """) rows = cursor.fetchall() return [row[0] for row in rows] except Exception as e: print(f"[存储] 获取抓取时间列表失败: {e}") return [] # ======================================== # 时间段执行记录（调度系统） # ======================================== def _has_period_executed_impl(self, date_str: str, period_key: str, action: str) -> bool: """ 检查指定时间段的某个 action 今天是否已执行 Args: date_str: 日期字符串 YYYY-MM-DD period_key: 时间段 key action: 动作类型 (analyze / push) Returns: 是否已执行 """ try: conn = self._get_connection(date_str) cursor = conn.cursor() # 先检查表是否存在 cursor.execute(""" SELECT name FROM sqlite_master WHERE type='table' AND name='period_executions' """) if not cursor.fetchone(): return False cursor.execute(""" SELECT 1 FROM period_executions WHERE execution_date = ? AND period_key = ? AND action = ? """, (date_str, period_key, action)) return cursor.fetchone() is not None except Exception as e: print(f"[存储] 检查时间段执行记录失败: {e}") return False def _record_period_execution_impl(self, date_str: str, period_key: str, action: str) -> bool: """ 记录时间段的 action 执行 Args: date_str: 日期字符串 YYYY-MM-DD period_key: 时间段 key action: 动作类型 (analyze / push) Returns: 是否记录成功 """ try: conn = self._get_connection(date_str) cursor = conn.cursor() # 确保表存在 cursor.execute(""" CREATE TABLE IF NOT EXISTS period_executions ( id INTEGER PRIMARY KEY AUTOINCREMENT, execution_date TEXT NOT NULL, period_key TEXT NOT NULL, action TEXT NOT NULL, executed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, UNIQUE(execution_date, period_key, action) ) """) now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") cursor.execute(""" INSERT OR IGNORE INTO period_executions (execution_date, period_key, action, executed_at) VALUES (?, ?, ?, ?) """, (date_str, period_key, action, now_str)) conn.commit() return True except Exception as e: print(f"[存储] 记录时间段执行失败: {e}") return False # ======================================== # RSS 数据存储 # ======================================== def _save_rss_data_impl(self, data: RSSData, log_prefix: str = "[存储]") -> tuple[bool, int, int]: """ 保存 RSS 数据到 SQLite（以 URL 为唯一标识） Args: data: RSS 数据 log_prefix: 日志前缀 Returns: (success, new_count, updated_count) """ try: conn = self._get_connection(data.date, db_type="rss") cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") # 同步 RSS 源信息到 rss_feeds 表 for feed_id, feed_name in data.id_to_name.items(): cursor.execute(""" INSERT INTO rss_feeds (id, name, updated_at) VALUES (?, ?, ?) ON CONFLICT(id) DO UPDATE SET name = excluded.name, updated_at = excluded.updated_at """, (feed_id, feed_name, now_str)) # 统计计数器 new_count = 0 updated_count = 0 for feed_id, rss_list in data.items.items(): for item in rss_list: try: # 检查是否已存在（通过 URL + feed_id） if item.url: cursor.execute(""" SELECT id, title FROM rss_items WHERE url = ? AND feed_id = ? """, (item.url, feed_id)) existing = cursor.fetchone() if existing: # 已存在，更新记录 existing_id = existing[0] cursor.execute(""" UPDATE rss_items SET title = ?, published_at = ?, summary = ?, author = ?, last_crawl_time = ?, crawl_count = crawl_count + 1, updated_at = ? WHERE id = ? """, (item.title, item.published_at, item.summary, item.author, data.crawl_time, now_str, existing_id)) updated_count += 1 else: # 不存在，插入新记录（使用 ON CONFLICT 兜底处理并发/竞争场景） cursor.execute(""" INSERT INTO rss_items (title, feed_id, url, published_at, summary, author, first_crawl_time, last_crawl_time, crawl_count, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, ?, ?) ON CONFLICT(url, feed_id) DO UPDATE SET title = excluded.title, published_at = excluded.published_at, summary = excluded.summary, author = excluded.author, last_crawl_time = excluded.last_crawl_time, crawl_count = crawl_count + 1, updated_at = excluded.updated_at """, (item.title, feed_id, item.url, item.published_at, item.summary, item.author, data.crawl_time, data.crawl_time, now_str, now_str)) new_count += 1 else: # URL 为空，用 try-except 处理重复 try: cursor.execute(""" INSERT INTO rss_items (title, feed_id, url, published_at, summary, author, first_crawl_time, last_crawl_time, crawl_count, created_at, updated_at) VALUES (?, ?, ?, ?, ?, ?, ?, ?, 1, ?, ?) """, (item.title, feed_id, "", item.published_at, item.summary, item.author, data.crawl_time, data.crawl_time, now_str, now_str)) new_count += 1 except sqlite3.IntegrityError: # 重复的空 URL 条目，忽略 pass except sqlite3.Error as e: print(f"{log_prefix} 保存 RSS 条目失败 [{item.title[:30]}...]: {e}") total_items = new_count + updated_count # 记录抓取信息 cursor.execute(""" INSERT OR REPLACE INTO rss_crawl_records (crawl_time, total_items, created_at) VALUES (?, ?, ?) """, (data.crawl_time, total_items, now_str)) # 记录抓取状态 cursor.execute(""" SELECT id FROM rss_crawl_records WHERE crawl_time = ? """, (data.crawl_time,)) record_row = cursor.fetchone() if record_row: crawl_record_id = record_row[0] # 记录成功的源 for feed_id in data.items.keys(): cursor.execute(""" INSERT OR REPLACE INTO rss_crawl_status (crawl_record_id, feed_id, status) VALUES (?, ?, 'success') """, (crawl_record_id, feed_id)) # 记录失败的源 for failed_id in data.failed_ids: cursor.execute(""" INSERT OR IGNORE INTO rss_feeds (id, name, updated_at) VALUES (?, ?, ?) """, (failed_id, failed_id, now_str)) cursor.execute(""" INSERT OR REPLACE INTO rss_crawl_status (crawl_record_id, feed_id, status) VALUES (?, ?, 'failed') """, (crawl_record_id, failed_id)) conn.commit() return True, new_count, updated_count except Exception as e: print(f"{log_prefix} 保存 RSS 数据失败: {e}") return False, 0, 0 def _get_rss_data_impl(self, date: Optional[str] = None) -> Optional[RSSData]: """ 获取指定日期的所有 RSS 数据 Args: date: 日期字符串（YYYY-MM-DD），默认为今天 Returns: RSSData 对象，如果没有数据返回 None """ try: conn = self._get_connection(date, db_type="rss") cursor = conn.cursor() # 获取所有 RSS 数据 cursor.execute(""" SELECT i.id, i.title, i.feed_id, f.name as feed_name, i.url, i.published_at, i.summary, i.author, i.first_crawl_time, i.last_crawl_time, i.crawl_count FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id ORDER BY i.published_at DESC """) rows = cursor.fetchall() if not rows: return None items: Dict[str, List[RSSItem]] = {} id_to_name: Dict[str, str] = {} crawl_date = self._format_date_folder(date) for row in rows: feed_id = row[2] feed_name = row[3] or feed_id id_to_name[feed_id] = feed_name if feed_id not in items: items[feed_id] = [] items[feed_id].append(RSSItem( title=row[1], feed_id=feed_id, feed_name=feed_name, url=row[4] or "", published_at=row[5] or "", summary=row[6] or "", author=row[7] or "", crawl_time=row[9], first_time=row[8], last_time=row[9], count=row[10], )) # 获取最新的抓取时间 cursor.execute(""" SELECT crawl_time FROM rss_crawl_records ORDER BY crawl_time DESC LIMIT 1 """) time_row = cursor.fetchone() crawl_time = time_row[0] if time_row else self._format_time_filename() # 获取失败的源 cursor.execute(""" SELECT DISTINCT cs.feed_id FROM rss_crawl_status cs JOIN rss_crawl_records cr ON cs.crawl_record_id = cr.id WHERE cs.status = 'failed' """) failed_ids = [row[0] for row in cursor.fetchall()] return RSSData( date=crawl_date, crawl_time=crawl_time, items=items, id_to_name=id_to_name, failed_ids=failed_ids, ) except Exception as e: print(f"[存储] 读取 RSS 数据失败: {e}") return None def _detect_new_rss_items_impl(self, current_data: RSSData) -> Dict[str, List[RSSItem]]: """ 检测新增的 RSS 条目（增量模式） 该方法比较当前抓取数据与历史数据，找出新增的 RSS 条目。 关键逻辑：只有在历史批次中从未出现过的 URL 才算新增。 Args: current_data: 当前抓取的 RSS 数据 Returns: 新增的 RSS 条目 {feed_id: [RSSItem, ...]} """ try: # 获取历史数据 historical_data = self._get_rss_data_impl(current_data.date) if not historical_data: # 没有历史数据，所有都是新的 return current_data.items.copy() # 获取当前批次时间 current_time = current_data.crawl_time # 收集历史 URL（first_time < current_time 的条目） historical_urls: Dict[str, set] = {} for feed_id, rss_list in historical_data.items.items(): historical_urls[feed_id] = set() for item in rss_list: first_time = item.first_time or item.crawl_time if first_time < current_time: if item.url: historical_urls[feed_id].add(item.url) # 检查是否有历史数据 has_historical_data = any(len(urls) > 0 for urls in historical_urls.values()) if not has_historical_data: # 第一次抓取，没有"新增"概念 return {} # 检测新增 new_items: Dict[str, List[RSSItem]] = {} for feed_id, rss_list in current_data.items.items(): hist_set = historical_urls.get(feed_id, set()) for item in rss_list: # 通过 URL 判断是否新增 if item.url and item.url not in hist_set: if feed_id not in new_items: new_items[feed_id] = [] new_items[feed_id].append(item) return new_items except Exception as e: print(f"[存储] 检测新 RSS 条目失败: {e}") return {} def _get_latest_rss_data_impl(self, date: Optional[str] = None) -> Optional[RSSData]: """ 获取最新一次抓取的 RSS 数据（当前榜单模式） Args: date: 日期字符串（YYYY-MM-DD），默认为今天 Returns: 最新抓取的 RSS 数据，如果没有数据返回 None """ try: conn = self._get_connection(date, db_type="rss") cursor = conn.cursor() # 获取最新的抓取时间 cursor.execute(""" SELECT crawl_time FROM rss_crawl_records ORDER BY crawl_time DESC LIMIT 1 """) time_row = cursor.fetchone() if not time_row: return None latest_time = time_row[0] # 获取该时间的 RSS 数据 cursor.execute(""" SELECT i.id, i.title, i.feed_id, f.name as feed_name, i.url, i.published_at, i.summary, i.author, i.first_crawl_time, i.last_crawl_time, i.crawl_count FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id WHERE i.last_crawl_time = ? ORDER BY i.published_at DESC """, (latest_time,)) rows = cursor.fetchall() if not rows: return None items: Dict[str, List[RSSItem]] = {} id_to_name: Dict[str, str] = {} crawl_date = self._format_date_folder(date) for row in rows: feed_id = row[2] feed_name = row[3] or feed_id id_to_name[feed_id] = feed_name if feed_id not in items: items[feed_id] = [] items[feed_id].append(RSSItem( title=row[1], feed_id=feed_id, feed_name=feed_name, url=row[4] or "", published_at=row[5] or "", summary=row[6] or "", author=row[7] or "", crawl_time=row[9], first_time=row[8], last_time=row[9], count=row[10], )) # 获取失败的源（针对最新一次抓取） cursor.execute(""" SELECT cs.feed_id FROM rss_crawl_status cs JOIN rss_crawl_records cr ON cs.crawl_record_id = cr.id WHERE cr.crawl_time = ? AND cs.status = 'failed' """, (latest_time,)) failed_ids = [row[0] for row in cursor.fetchall()] return RSSData( date=crawl_date, crawl_time=latest_time, items=items, id_to_name=id_to_name, failed_ids=failed_ids, ) except Exception as e: print(f"[存储] 获取最新 RSS 数据失败: {e}") return None # ======================================== # AI 智能筛选 - 标签管理 # ======================================== def _get_active_tags_impl(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> List[Dict[str, Any]]: """获取指定兴趣文件的 active 标签列表""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT id, tag, description, version, prompt_hash, priority FROM ai_filter_tags WHERE status = 'active' AND interests_file = ? ORDER BY priority ASC, id ASC """, (interests_file,)) return [ { "id": row[0], "tag": row[1], "description": row[2], "version": row[3], "prompt_hash": row[4], "priority": row[5], } for row in cursor.fetchall() ] except Exception as e: print(f"[AI筛选] 获取标签失败: {e}") return [] def _get_latest_prompt_hash_impl(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> Optional[str]: """获取指定兴趣文件最新版本标签的 prompt_hash""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT prompt_hash FROM ai_filter_tags WHERE status = 'active' AND interests_file = ? ORDER BY version DESC LIMIT 1 """, (interests_file,)) row = cursor.fetchone() return row[0] if row else None except Exception as e: print(f"[AI筛选] 获取 prompt_hash 失败: {e}") return None def _get_latest_tag_version_impl(self, date: Optional[str] = None) -> int: """获取最新版本号""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT MAX(version) FROM ai_filter_tags """) row = cursor.fetchone() return row[0] if row and row[0] is not None else 0 except Exception as e: print(f"[AI筛选] 获取版本号失败: {e}") return 0 def _deprecate_all_tags_impl(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> int: """将指定兴趣文件的 active 标签和关联的分类结果标记为 deprecated""" try: conn = self._get_connection(date) cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") # 获取该兴趣文件的 active 标签 id cursor.execute( "SELECT id FROM ai_filter_tags WHERE status = 'active' AND interests_file = ?", (interests_file,) ) tag_ids = [row[0] for row in cursor.fetchall()] if not tag_ids: return 0 # 废弃标签 placeholders = ",".join("?" * len(tag_ids)) cursor.execute(f""" UPDATE ai_filter_tags SET status = 'deprecated', deprecated_at = ? WHERE id IN ({placeholders}) """, [now_str] + tag_ids) tag_count = cursor.rowcount # 废弃关联的分类结果 placeholders = ",".join("?" * len(tag_ids)) cursor.execute(f""" UPDATE ai_filter_results SET status = 'deprecated', deprecated_at = ? WHERE tag_id IN ({placeholders}) AND status = 'active' """, [now_str] + tag_ids) conn.commit() print(f"[AI筛选] 已废弃 {tag_count} 个标签及关联分类结果") return tag_count except Exception as e: print(f"[AI筛选] 废弃标签失败: {e}") return 0 def _save_tags_impl( self, date: Optional[str], tags: List[Dict], version: int, prompt_hash: str, interests_file: str = "ai_interests.txt" ) -> int: """保存新提取的标签""" try: conn = self._get_connection(date) cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") count = 0 for idx, tag_data in enumerate(tags, start=1): priority = tag_data.get("priority", idx) try: priority = int(priority) except (TypeError, ValueError): priority = idx cursor.execute(""" INSERT INTO ai_filter_tags (tag, description, priority, version, prompt_hash, interests_file, created_at) VALUES (?, ?, ?, ?, ?, ?, ?) """, ( tag_data["tag"], tag_data.get("description", ""), priority, version, prompt_hash, interests_file, now_str, )) count += 1 conn.commit() return count except Exception as e: print(f"[AI筛选] 保存标签失败: {e}") return 0 def _deprecate_specific_tags_impl( self, date: Optional[str], tag_ids: List[int] ) -> int: """废弃指定 ID 的标签及其关联分类结果（增量更新时使用）""" if not tag_ids: return 0 try: conn = self._get_connection(date) cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") placeholders = ",".join("?" * len(tag_ids)) cursor.execute(f""" UPDATE ai_filter_tags SET status = 'deprecated', deprecated_at = ? WHERE id IN ({placeholders}) """, [now_str] + tag_ids) tag_count = cursor.rowcount cursor.execute(f""" UPDATE ai_filter_results SET status = 'deprecated', deprecated_at = ? WHERE tag_id IN ({placeholders}) AND status = 'active' """, [now_str] + tag_ids) conn.commit() return tag_count except Exception as e: print(f"[AI筛选] 废弃指定标签失败: {e}") return 0 def _update_tags_hash_impl( self, date: Optional[str], interests_file: str, new_hash: str ) -> int: """更新指定兴趣文件所有 active 标签的 prompt_hash（增量更新时使用）""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" UPDATE ai_filter_tags SET prompt_hash = ? WHERE interests_file = ? AND status = 'active' """, (new_hash, interests_file)) count = cursor.rowcount conn.commit() return count except Exception as e: print(f"[AI筛选] 更新标签 hash 失败: {e}") return 0 # ======================================== # AI 智能筛选 - 分类结果管理 # ======================================== def _update_tag_descriptions_impl( self, date: Optional[str], tag_updates: List[Dict], interests_file: str = "ai_interests.txt" ) -> int: """按 tag 名匹配，更新 active 标签的 description 字段""" try: conn = self._get_connection(date) cursor = conn.cursor() count = 0 for t in tag_updates: tag_name = t.get("tag", "") description = t.get("description", "") if not tag_name: continue cursor.execute(""" UPDATE ai_filter_tags SET description = ? WHERE tag = ? AND interests_file = ? AND status = 'active' """, (description, tag_name, interests_file)) count += cursor.rowcount conn.commit() return count except Exception as e: print(f"[AI筛选] 更新标签描述失败: {e}") return 0 def _update_tag_priorities_impl( self, date: Optional[str], tag_priorities: List[Dict], interests_file: str = "ai_interests.txt" ) -> int: """按 tag 名匹配，更新 active 标签的 priority 字段""" try: conn = self._get_connection(date) cursor = conn.cursor() count = 0 for t in tag_priorities: tag_name = t.get("tag", "") priority = t.get("priority") if not tag_name: continue try: priority = int(priority) except (TypeError, ValueError): continue cursor.execute(""" UPDATE ai_filter_tags SET priority = ? WHERE tag = ? AND interests_file = ? AND status = 'active' """, (priority, tag_name, interests_file)) count += cursor.rowcount conn.commit() return count except Exception as e: print(f"[AI筛选] 更新标签优先级失败: {e}") return 0 # ======================================== # AI 智能筛选 - 已分析新闻追踪 # ======================================== def _save_analyzed_news_impl( self, date: Optional[str], news_ids: List[int], source_type: str, interests_file: str, prompt_hash: str, matched_ids: set ) -> int: """批量记录已分析的新闻（匹配与不匹配都记录）""" try: conn = self._get_connection(date) cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") count = 0 for nid in news_ids: try: cursor.execute(""" INSERT OR REPLACE INTO ai_filter_analyzed_news (news_item_id, source_type, interests_file, prompt_hash, matched, created_at) VALUES (?, ?, ?, ?, ?, ?) """, ( nid, source_type, interests_file, prompt_hash, 1 if nid in matched_ids else 0, now_str, )) count += 1 except Exception: pass conn.commit() return count except Exception as e: print(f"[AI筛选] 保存已分析记录失败: {e}") return 0 def _get_analyzed_news_ids_impl( self, date: Optional[str] = None, source_type: str = "hotlist", interests_file: str = "ai_interests.txt" ) -> set: """获取已分析过的新闻 ID 集合（用于去重）""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT news_item_id FROM ai_filter_analyzed_news WHERE source_type = ? AND interests_file = ? """, (source_type, interests_file)) return {row[0] for row in cursor.fetchall()} except Exception as e: print(f"[AI筛选] 获取已分析ID失败: {e}") return set() def _clear_analyzed_news_impl( self, date: Optional[str] = None, interests_file: str = "ai_interests.txt" ) -> int: """清除指定兴趣文件的所有已分析记录（全量重分类时使用）""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" DELETE FROM ai_filter_analyzed_news WHERE interests_file = ? """, (interests_file,)) count = cursor.rowcount conn.commit() return count except Exception as e: print(f"[AI筛选] 清除已分析记录失败: {e}") return 0 def _clear_unmatched_analyzed_news_impl( self, date: Optional[str] = None, interests_file: str = "ai_interests.txt" ) -> int: """清除不匹配的已分析记录，让这些新闻有机会被新标签重新分析""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" DELETE FROM ai_filter_analyzed_news WHERE interests_file = ? AND matched = 0 """, (interests_file,)) count = cursor.rowcount conn.commit() return count except Exception as e: print(f"[AI筛选] 清除不匹配记录失败: {e}") return 0 # ======================================== # AI 智能筛选 - 分类结果管理（原有） # ======================================== def _save_filter_results_impl( self, date: Optional[str], results: List[Dict] ) -> int: """批量保存分类结果""" try: conn = self._get_connection(date) cursor = conn.cursor() now_str = self._get_configured_time().strftime("%Y-%m-%d %H:%M:%S") count = 0 for r in results: try: cursor.execute(""" INSERT INTO ai_filter_results (news_item_id, source_type, tag_id, relevance_score, created_at) VALUES (?, ?, ?, ?, ?) """, ( r["news_item_id"], r.get("source_type", "hotlist"), r["tag_id"], r.get("relevance_score", 0.0), now_str, )) count += 1 except sqlite3.IntegrityError: pass # 重复记录，跳过 conn.commit() return count except Exception as e: print(f"[AI筛选] 保存分类结果失败: {e}") return 0 def _get_active_filter_results_impl(self, date: Optional[str] = None, interests_file: str = "ai_interests.txt") -> List[Dict[str, Any]]: """获取指定兴趣文件的 active 分类结果，JOIN news_items 获取新闻详情""" try: conn = self._get_connection(date) cursor = conn.cursor() # 热榜结果 cursor.execute(""" SELECT r.news_item_id, r.source_type, r.tag_id, r.relevance_score, t.tag, t.description as tag_description, t.priority, n.title, n.platform_id as source_id, p.name as source_name, n.url, n.mobile_url, n.rank, n.first_crawl_time, n.last_crawl_time, n.crawl_count FROM ai_filter_results r JOIN ai_filter_tags t ON r.tag_id = t.id JOIN news_items n ON r.news_item_id = n.id LEFT JOIN platforms p ON n.platform_id = p.id WHERE r.status = 'active' AND r.source_type = 'hotlist' AND t.status = 'active' AND t.interests_file = ? ORDER BY t.priority ASC, t.id ASC, r.relevance_score DESC """, (interests_file,)) results = [] hotlist_news_ids = [] for row in cursor.fetchall(): results.append({ "news_item_id": row[0], "source_type": row[1], "tag_id": row[2], "relevance_score": row[3], "tag": row[4], "tag_description": row[5], "tag_priority": row[6], "title": row[7], "source_id": row[8], "source_name": row[9] or row[8], "url": row[10] or "", "mobile_url": row[11] or "", "rank": row[12], "first_time": row[13], "last_time": row[14], "count": row[15], }) hotlist_news_ids.append(row[0]) # 批量查排名历史（热榜） ranks_map: Dict[int, List[int]] = {} if hotlist_news_ids: unique_ids = list(set(hotlist_news_ids)) placeholders = ",".join("?" * len(unique_ids)) cursor.execute(f""" SELECT news_item_id, rank FROM rank_history WHERE news_item_id IN ({placeholders}) AND rank != 0 """, unique_ids) for rh_row in cursor.fetchall(): nid, rank = rh_row[0], rh_row[1] if nid not in ranks_map: ranks_map[nid] = [] if rank not in ranks_map[nid]: ranks_map[nid].append(rank) for item in results: item["ranks"] = ranks_map.get(item["news_item_id"], [item["rank"]]) # RSS 结果（如果有 rss 库） try: rss_conn = self._get_connection(date, db_type="rss") rss_cursor = rss_conn.cursor() # 从 news 库获取 rss 类型的分类结果 ID cursor.execute(""" SELECT r.news_item_id, r.tag_id, r.relevance_score, t.tag, t.description, t.priority FROM ai_filter_results r JOIN ai_filter_tags t ON r.tag_id = t.id WHERE r.status = 'active' AND r.source_type = 'rss' AND t.status = 'active' AND t.interests_file = ? ORDER BY t.priority ASC, t.id ASC, r.relevance_score DESC """, (interests_file,)) rss_filter_rows = cursor.fetchall() if rss_filter_rows: rss_ids = [row[0] for row in rss_filter_rows] placeholders = ",".join("?" * len(rss_ids)) rss_cursor.execute(f""" SELECT i.id, i.title, i.feed_id, f.name as feed_name, i.url, i.published_at FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id WHERE i.id IN ({placeholders}) """, rss_ids) rss_info = {row[0]: row for row in rss_cursor.fetchall()} for fr_row in rss_filter_rows: rss_id = fr_row[0] info = rss_info.get(rss_id) if info: results.append({ "news_item_id": rss_id, "source_type": "rss", "tag_id": fr_row[1], "relevance_score": fr_row[2], "tag": fr_row[3], "tag_description": fr_row[4], "tag_priority": fr_row[5], "title": info[1], "source_id": info[2], "source_name": info[3] or info[2], "url": info[4] or "", "mobile_url": "", "rank": 0, "ranks": [], "first_time": info[5] or "", "last_time": info[5] or "", "count": 1, }) except Exception: pass # RSS 库不存在时静默跳过 return results except Exception as e: print(f"[AI筛选] 获取分类结果失败: {e}") return [] def _get_all_news_ids_impl(self, date: Optional[str] = None) -> List[Dict]: """获取当日所有新闻的 id 和标题（用于 AI 筛选分类）""" try: conn = self._get_connection(date) cursor = conn.cursor() cursor.execute(""" SELECT n.id, n.title, n.platform_id, p.name as platform_name FROM news_items n LEFT JOIN platforms p ON n.platform_id = p.id ORDER BY n.id """) return [ { "id": row[0], "title": row[1], "source_id": row[2], "source_name": row[3] or row[2], } for row in cursor.fetchall() ] except Exception as e: print(f"[AI筛选] 获取新闻列表失败: {e}") return [] def _get_all_rss_ids_impl(self, date: Optional[str] = None) -> List[Dict]: """获取当日所有 RSS 条目的 id 和标题（用于 AI 筛选分类）""" try: conn = self._get_connection(date, db_type="rss") cursor = conn.cursor() cursor.execute(""" SELECT i.id, i.title, i.feed_id, f.name as feed_name, i.published_at FROM rss_items i LEFT JOIN rss_feeds f ON i.feed_id = f.id ORDER BY i.id """) return [ { "id": row[0], "title": row[1], "source_id": row[2], "source_name": row[3] or row[2], "published_at": row[4] or "", } for row in cursor.fetchall() ] except Exception as e: print(f"[AI筛选] 获取 RSS 列表失败: {e}") return [] ================================================ FILE: trendradar/utils/__init__.py ================================================ # coding=utf-8 """ 工具模块 - 公共工具函数 """ from trendradar.utils.time import ( get_configured_time, format_date_folder, format_time_filename, get_current_time_display, convert_time_for_display, ) from trendradar.utils.url import normalize_url, get_url_signature __all__ = [ "get_configured_time", "format_date_folder", "format_time_filename", "get_current_time_display", "convert_time_for_display", "normalize_url", "get_url_signature", ] ================================================ FILE: trendradar/utils/time.py ================================================ # coding=utf-8 """ 时间工具模块 本模块提供统一的时间处理函数，所有时区相关操作都应使用 DEFAULT_TIMEZONE 常量。 """ from datetime import datetime from typing import Optional, Tuple import pytz # 默认时区常量 - 仅作为 fallback，正常运行时使用 config.yaml 中的 app.timezone DEFAULT_TIMEZONE = "Asia/Shanghai" def get_configured_time(timezone: str = DEFAULT_TIMEZONE) -> datetime: """ 获取配置时区的当前时间 Args: timezone: 时区名称，如 'Asia/Shanghai', 'America/Los_Angeles' Returns: 带时区信息的当前时间 """ try: tz = pytz.timezone(timezone) except pytz.UnknownTimeZoneError: print(f"[警告] 未知时区 '{timezone}'，使用默认时区 {DEFAULT_TIMEZONE}") tz = pytz.timezone(DEFAULT_TIMEZONE) return datetime.now(tz) def format_date_folder( date: Optional[str] = None, timezone: str = DEFAULT_TIMEZONE ) -> str: """ 格式化日期文件夹名 (ISO 格式: YYYY-MM-DD) Args: date: 指定日期字符串，为 None 则使用当前日期 timezone: 时区名称 Returns: 格式化后的日期字符串，如 '2025-12-09' """ if date: return date return get_configured_time(timezone).strftime("%Y-%m-%d") def format_time_filename(timezone: str = DEFAULT_TIMEZONE) -> str: """ 格式化时间文件名 (格式: HH-MM，用于文件名) Windows 系统不支持冒号作为文件名，因此使用连字符 Args: timezone: 时区名称 Returns: 格式化后的时间字符串，如 '15-30' """ return get_configured_time(timezone).strftime("%H-%M") def get_current_time_display(timezone: str = DEFAULT_TIMEZONE) -> str: """ 获取当前时间显示 (格式: HH:MM，用于显示) Args: timezone: 时区名称 Returns: 格式化后的时间字符串，如 '15:30' """ return get_configured_time(timezone).strftime("%H:%M") def convert_time_for_display(time_str: str) -> str: """ 将 HH-MM 格式转换为 HH:MM 格式用于显示 Args: time_str: 输入时间字符串，如 '15-30' Returns: 转换后的时间字符串，如 '15:30' """ if time_str and "-" in time_str and len(time_str) == 5: return time_str.replace("-", ":") return time_str def format_iso_time_friendly( iso_time: str, timezone: str = DEFAULT_TIMEZONE, include_date: bool = True, ) -> str: """ 将 ISO 格式时间转换为用户时区的友好显示格式 Args: iso_time: ISO 格式时间字符串，如 '2025-12-29T00:20:00' 或 '2025-12-29T00:20:00+00:00' timezone: 目标时区名称 include_date: 是否包含日期部分 Returns: 友好格式的时间字符串，如 '12-29 08:20' 或 '08:20' """ if not iso_time: return "" try: # 尝试解析各种 ISO 格式 dt = None # 尝试解析带时区的格式 if "+" in iso_time or iso_time.endswith("Z"): iso_time = iso_time.replace("Z", "+00:00") try: dt = datetime.fromisoformat(iso_time) except ValueError: pass # 尝试解析不带时区的格式（假设为 UTC） if dt is None: try: # 处理 T 分隔符 if "T" in iso_time: dt = datetime.fromisoformat(iso_time.replace("T", " ").split(".")[0]) else: dt = datetime.fromisoformat(iso_time.split(".")[0]) # 假设为 UTC 时间 dt = pytz.UTC.localize(dt) except ValueError: pass if dt is None: # 无法解析，返回原始字符串的简化版本 if "T" in iso_time: parts = iso_time.split("T") if len(parts) == 2: date_part = parts[0][5:] # MM-DD time_part = parts[1][:5] # HH:MM return f"{date_part} {time_part}" if include_date else time_part return iso_time # 转换到目标时区 try: target_tz = pytz.timezone(timezone) except pytz.UnknownTimeZoneError: target_tz = pytz.timezone(DEFAULT_TIMEZONE) dt_local = dt.astimezone(target_tz) # 格式化输出 if include_date: return dt_local.strftime("%m-%d %H:%M") else: return dt_local.strftime("%H:%M") except Exception: # 出错时返回原始字符串的简化版本 if "T" in iso_time: parts = iso_time.split("T") if len(parts) == 2: date_part = parts[0][5:] # MM-DD time_part = parts[1][:5] # HH:MM return f"{date_part} {time_part}" if include_date else time_part return iso_time def is_within_days( iso_time: str, max_days: int, timezone: str = DEFAULT_TIMEZONE, ) -> bool: """ 检查 ISO 格式时间是否在指定天数内 用于 RSS 文章新鲜度过滤，判断文章发布时间是否超过指定天数。 Args: iso_time: ISO 格式时间字符串（如 '2025-12-29T00:20:00' 或带时区） max_days: 最大天数（文章发布时间距今不超过此天数则返回 True） - max_days > 0: 正常过滤，保留 N 天内的文章 - max_days <= 0: 禁用过滤，保留所有文章 timezone: 时区名称（用于获取当前时间） Returns: True 如果时间在指定天数内（应保留），False 如果超过指定天数（应过滤） 如果无法解析时间，返回 True（保留文章） """ # 无时间戳或禁用过滤时，保留文章 if not iso_time: return True if max_days <= 0: return True # max_days=0 表示禁用过滤 try: dt = None # 尝试解析带时区的格式 if "+" in iso_time or iso_time.endswith("Z"): iso_time_normalized = iso_time.replace("Z", "+00:00") try: dt = datetime.fromisoformat(iso_time_normalized) except ValueError: pass # 尝试解析不带时区的格式（假设为 UTC） if dt is None: try: if "T" in iso_time: dt = datetime.fromisoformat(iso_time.replace("T", " ").split(".")[0]) else: dt = datetime.fromisoformat(iso_time.split(".")[0]) dt = pytz.UTC.localize(dt) except ValueError: pass if dt is None: # 无法解析时间，保留文章 return True # 获取当前时间（配置的时区，带时区信息） now = get_configured_time(timezone) # 计算时间差（两个带时区的 datetime 相减会自动处理时区差异） diff = now - dt days_diff = diff.total_seconds() / (24 * 60 * 60) return days_diff <= max_days except Exception: # 出错时保留文章 return True def calculate_days_old(iso_time: str, timezone: str = DEFAULT_TIMEZONE) -> Optional[float]: """ 计算 ISO 格式时间距今多少天 Args: iso_time: ISO 格式时间字符串 timezone: 时区名称 Returns: 距今天数（浮点数），如果无法解析返回 None """ if not iso_time: return None try: dt = None # 尝试解析带时区的格式 if "+" in iso_time or iso_time.endswith("Z"): iso_time_normalized = iso_time.replace("Z", "+00:00") try: dt = datetime.fromisoformat(iso_time_normalized) except ValueError: pass # 尝试解析不带时区的格式（假设为 UTC） if dt is None: try: if "T" in iso_time: dt = datetime.fromisoformat(iso_time.replace("T", " ").split(".")[0]) else: dt = datetime.fromisoformat(iso_time.split(".")[0]) dt = pytz.UTC.localize(dt) except ValueError: pass if dt is None: return None now = get_configured_time(timezone) diff = now - dt return diff.total_seconds() / (24 * 60 * 60) except Exception: return None class TimeWindowChecker: """ 时间窗口检查器 统一管理时间窗口控制逻辑，支持： - 推送窗口控制 (push_window) - AI 分析窗口控制 (analysis_window) - once_per_day 功能 """ def __init__( self, storage_backend, get_time_func=None, window_name: str = "时间窗口", ): """ 初始化时间窗口检查器 Args: storage_backend: 存储后端实例 get_time_func: 获取当前时间的函数 window_name: 窗口名称（用于日志输出） """ self.storage_backend = storage_backend self.get_time_func = get_time_func or (lambda: get_configured_time(DEFAULT_TIMEZONE)) self.window_name = window_name def is_in_time_range(self, start_time: str, end_time: str) -> bool: """ 检查当前时间是否在指定时间范围内 支持跨日时间窗口，例如： - 正常窗口：09:00-21:00（当天 9 点到 21 点） - 跨日窗口：22:00-02:00（当天 22 点到次日 2 点） Args: start_time: 开始时间（格式：HH:MM） end_time: 结束时间（格式：HH:MM） Returns: 是否在时间范围内 """ now = self.get_time_func() current_time = now.strftime("%H:%M") normalized_start = self._normalize_time(start_time) normalized_end = self._normalize_time(end_time) normalized_current = self._normalize_time(current_time) # 判断是否跨日窗口（start > end 表示跨日，如 22:00-02:00） if normalized_start <= normalized_end: # 正常窗口：09:00-21:00 result = normalized_start <= normalized_current <= normalized_end else: # 跨日窗口：22:00-02:00 # 当前时间 >= 开始时间（如 23:00 >= 22:00）或 当前时间 <= 结束时间（如 01:00 <= 02:00） result = normalized_current >= normalized_start or normalized_current <= normalized_end if not result: print(f"[{self.window_name}] 当前 {normalized_current}，窗口 {normalized_start}-{normalized_end}") return result def _normalize_time(self, time_str: str) -> str: """将时间字符串标准化为 HH:MM 格式""" try: parts = time_str.strip().split(":") if len(parts) != 2: raise ValueError(f"时间格式错误: {time_str}") hour = int(parts[0]) minute = int(parts[1]) if not (0 <= hour <= 23 and 0 <= minute <= 59): raise ValueError(f"时间范围错误: {time_str}") return f"{hour:02d}:{minute:02d}" except Exception as e: print(f"[{self.window_name}] 时间格式化错误 '{time_str}': {e}") return time_str def check_window( self, window_config: dict, check_once_per_day_func=None, record_func=None, ) -> Tuple[bool, str]: """ 统一的时间窗口检查逻辑 Args: window_config: 窗口配置字典，包含： - ENABLED: 是否启用窗口控制 - TIME_RANGE: {"START": "HH:MM", "END": "HH:MM"} - ONCE_PER_DAY: 是否每天只执行一次 check_once_per_day_func: 检查今天是否已执行的函数 record_func: 记录执行的函数（成功后调用） Returns: (should_proceed, reason) 元组： - should_proceed: 是否应该继续执行 - reason: 原因说明 """ if not window_config.get("ENABLED", False): return True, "窗口控制未启用" time_range = window_config.get("TIME_RANGE", {}) start_time = time_range.get("START", "00:00") end_time = time_range.get("END", "23:59") # 检查时间范围 if not self.is_in_time_range(start_time, end_time): now = self.get_time_func() return False, f"当前时间 {now.strftime('%H:%M')} 不在窗口 {start_time}-{end_time} 内" # 检查 once_per_day if window_config.get("ONCE_PER_DAY", False) and check_once_per_day_func: if check_once_per_day_func(): return False, "今天已执行过" else: print(f"[{self.window_name}] 今天首次执行") return True, "在窗口内" def get_status(self, window_config: dict, check_once_per_day_func=None) -> dict: """ 获取窗口状态信息 Args: window_config: 窗口配置 check_once_per_day_func: 检查今天是否已执行的函数 Returns: 状态信息字典 """ now = self.get_time_func() status = { "enabled": window_config.get("ENABLED", False), "current_time": now.strftime("%H:%M:%S"), "current_date": now.strftime("%Y-%m-%d"), "timezone": str(now.tzinfo), } if status["enabled"]: time_range = window_config.get("TIME_RANGE", {}) status["window_start"] = time_range.get("START", "00:00") status["window_end"] = time_range.get("END", "23:59") status["in_window"] = self.is_in_time_range( status["window_start"], status["window_end"] ) status["once_per_day"] = window_config.get("ONCE_PER_DAY", False) if status["once_per_day"] and check_once_per_day_func: status["executed_today"] = check_once_per_day_func() return status ================================================ FILE: trendradar/utils/url.py ================================================ # coding=utf-8 """ URL 处理工具模块 提供 URL 标准化功能，用于去重时消除动态参数的影响： - normalize_url: 标准化 URL，去除动态参数 """ from urllib.parse import urlparse, urlunparse, parse_qs, urlencode from typing import Dict, Set # 各平台需要移除的特定参数 # - weibo: 有 band_rank（排名）和 Refer（来源）动态参数 # - 其他平台: URL 为路径格式或简单关键词查询，无需处理 PLATFORM_PARAMS_TO_REMOVE: Dict[str, Set[str]] = { # 微博：band_rank 是动态排名参数，Refer 是来源参数，t 是时间范围参数 # 示例：https://s.weibo.com/weibo?q=xxx&t=31&band_rank=1&Refer=top # 保留：q（关键词） # 移除：band_rank, Refer, t "weibo": {"band_rank", "Refer", "t"}, } # 通用追踪参数（适用于所有平台） # 这些参数通常由分享链接或广告追踪添加，不影响内容识别 COMMON_TRACKING_PARAMS: Set[str] = { # UTM 追踪参数 "utm_source", "utm_medium", "utm_campaign", "utm_term", "utm_content", # 常见追踪参数 "ref", "referrer", "source", "channel", # 时间戳和随机参数 "_t", "timestamp", "_", "random", # 分享相关 "share_token", "share_id", "share_from", } def normalize_url(url: str, platform_id: str = "") -> str: """ 标准化 URL，去除动态参数 用于数据库去重，确保同一条新闻的不同 URL 变体能被正确识别为同一条。 处理规则： 1. 去除平台特定的动态参数（如微博的 band_rank） 2. 去除通用追踪参数（如 utm_*） 3. 保留核心查询参数（如搜索关键词 q=, wd=, keyword=） 4. 对查询参数按字母序排序（确保一致性） Args: url: 原始 URL platform_id: 平台 ID，用于应用平台特定规则 Returns: 标准化后的 URL Examples: >>> normalize_url("https://s.weibo.com/weibo?q=test&band_rank=6&Refer=top", "weibo") 'https://s.weibo.com/weibo?q=test' >>> normalize_url("https://example.com/page?id=1&utm_source=twitter", "") 'https://example.com/page?id=1' """ if not url: return url try: # 解析 URL parsed = urlparse(url) # 如果没有查询参数，直接返回 if not parsed.query: return url # 解析查询参数 params = parse_qs(parsed.query, keep_blank_values=True) # 收集需要移除的参数（使用小写进行比较） params_to_remove: Set[str] = set() # 添加通用追踪参数 params_to_remove.update(COMMON_TRACKING_PARAMS) # 添加平台特定参数 if platform_id and platform_id in PLATFORM_PARAMS_TO_REMOVE: params_to_remove.update(PLATFORM_PARAMS_TO_REMOVE[platform_id]) # 过滤参数（参数名转小写进行比较） filtered_params = { key: values for key, values in params.items() if key.lower() not in {p.lower() for p in params_to_remove} } # 如果过滤后没有参数了，返回不带查询字符串的 URL if not filtered_params: return urlunparse(( parsed.scheme, parsed.netloc, parsed.path, parsed.params, "", # 空查询字符串 "" # 移除 fragment )) # 重建查询字符串（按字母序排序以确保一致性） sorted_params = [] for key in sorted(filtered_params.keys()): for value in filtered_params[key]: sorted_params.append((key, value)) new_query = urlencode(sorted_params) # 重建 URL（移除 fragment） normalized = urlunparse(( parsed.scheme, parsed.netloc, parsed.path, parsed.params, new_query, "" # 移除 fragment )) return normalized except Exception: # 解析失败时返回原始 URL return url def get_url_signature(url: str, platform_id: str = "") -> str: """ 获取 URL 的签名（用于快速比较） 基于标准化 URL 生成签名，可用于： - 快速判断两个 URL 是否指向同一内容 - 作为缓存键 Args: url: 原始 URL platform_id: 平台 ID Returns: URL 签名字符串 """ return normalize_url(url, platform_id) ================================================ FILE: version ================================================ 6.5.0 ================================================ FILE: version_configs ================================================ config.yaml=2.2.0 timeline.yaml=1.2.0 frequency_words.txt=1.1.0 ai_interests.txt=1.0.0 ai_analysis_prompt.txt=2.0.0 ai_translation_prompt.txt=1.2.0 ================================================ FILE: version_mcp ================================================ 4.0.0