[ { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nenv/\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\n*.egg-info/\n.installed.cfg\n*.egg\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*,cover\n.hypothesis/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# IPython Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# dotenv\n.env\n\n# virtualenv\nvenv/\nENV/\n\n# Spyder project settings\n.spyderproject\n\n# Rope project settings\n.ropeproject\n\n# 开发者自定义\n.DS_Store\n.idea/\nhack/_dev\n" }, { "path": ".travis.yml", "content": "language: python\n\npython:\n - 3.6\n\ninstall:\n - pip install -e .\nscript:\n - python -m unittest discover -p \"*_test.py\"\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2017 Cheng YuMeng\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "MANIFEST.in", "content": "graft spider163/www/templates\ngraft spider163/www/static\ngraft spider163/template\n" }, { "path": "Makefile", "content": ".PHONY: docker-build docker-run build\n\nUSER := $(shell id -u)\nBRANCH := $(shell git rev-parse --abbrev-ref HEAD)\nVERSION := $(shell git describe --always --tags | grep -Eo \"[0-9]+\\.[0-9]+\\.[0-9]+\")\n\ndocker-build:\n\tdocker build -t chengtian/spider163:$(VERSION) -f hack/spider/Dockerfile .\n\ndocker-run:\n\tcd hack && docker-compose up\n\nbuild:\n\tpip install -e .\n\n\n" }, { "path": "README.md", "content": " ![spider163 logo](https://github.com/Chengyumeng/spider163/blob/master/logo.jpeg)\n# spider163\n[![MIT License](http://img.shields.io/badge/license-MIT-blue.svg?style=flat-square)][license] [![pyversions](https://img.shields.io/github/tag/Chengyumeng/Spider163.svg)][releases] [![pyversions](https://img.shields.io/pypi/pyversions/spider163.svg)][pyversions] [![Build Status](https://travis-ci.org/chengyumeng/spider163.svg?branch=master)][buildstatus]\n\n[license]: https://github.com/Chengyumeng/spider163/blob/master/LICENSE\n\n[releases]:https://github.com/Chengyumeng/spider163/releases\n\n[pyversions]:https://pypi.python.org/pypi/spider163\n\n[buildstatus]:https://travis-ci.org/chengyumeng/spider163\n\n###### GitHub上最易用的网易云音乐爬虫系统\n\n\n## 安装模块\n- 第一步：指定SPIDER163_PATH环境变量，缺省情况下为$HOME/spider163\n- 第二步：把默认配置文件spider163.conf拷贝到SPIDER163_PATH下，并配置数据库\n- 第三步：pip install spider163\n- spider163 --help\n\n## 历史文档\n- [不重要，2018年spider163发布的5个版本](https://mp.weixin.qq.com/s/pim5tYPHd0zBTKYQaijkbQ)\n- [Spider163同时支持python2.x和python3.x的演进之路](https://mp.weixin.qq.com/s/FFoD3gKM5touGVbvKebRlA)\n- [Spider163支持下载网易付费歌曲了](https://mp.weixin.qq.com/s/L8uvPV_CiAS6vcnaOaifJw)\n- [非重磅 | 网易云音乐爬虫Spider163更新v2.0](https://mp.weixin.qq.com/s?__biz=MzI2NTMxMDYxMg==&mid=2247483955&idx=1&sn=c1d8a38b4929cb298fc6172cf894e641&chksm=ea9e1ac8dde993de1d6095d000f289389ee92609bccebda3d2ebc88bfa1939eceb6b94cc3fce&scene=38#wechat_redirect)\n- ...\n\n\n## 使用指南\n\n```console\n$ spider163 initdb\n$ # 根据配置文件的数据库信息自动创建数据库表，删除全部数据通过resetdb实现\n```\n```console\n$ spider163 resetdb\n$ # 重建相关数据库\n```\n```console\n$ spider163 updatedb\n$ # 根据时间重置过期数据重新抓取\n```\n```console\n$ spider163 classify\n$ # 获取已知曲风列表\n```\n```console\n$ spider163 playlist\n$ # 默认下载全部推荐歌单（1000+），也可以通过指定页码去下载（-p=1）,以及歌曲风格（--classify=小语种，默认为全部）\n```\n```console\n$ spider163 mp3 --playlist=2033391777\n$ # 默认下载指定歌单列表内的全部包含版权的歌曲\n```\n```console\n$ spider163 music\n$ # 默认下载10个歌单的歌曲数据，也可以通过指定循环大小（-c=2）来下载10 * c 个歌单内歌曲\n```\n```console\n$ spider163 comment\n$ # 默认根据数据库存储的未下载歌曲随机下载一首单曲的评论，也可以通过-c指定需要下载的单曲数量和-s强制指定歌曲id\n$ # spider163 comment -c 10 | spider163 comment -s 209115\n```\n```console\n$ spider163 lyric --count=10\n$ # 抓取10首音乐的歌词，可以通过制定歌曲ID抓取特定一首音乐（--song）\n```\n```console\n$ spider163 search -q=\"林依晨\"\n$ # 搜索功能（待完善，暂支持歌曲搜索）\n```\n```console\n$ spider163 get -s 209115\n$ # 阅读歌曲基本信息、歌词、热评\n```\n```console\n$ spider163 get --playlist 922064582\n$ # 获取歌单的基本信息、歌曲等\n```\n```console\n$ spider163 doc --playlist 922064582\n$ # 歌单/歌曲信息汇总成word文档\n```\n```console\n$ spider163 top50 --playlist 922064582 --username=xxx --password=xxx\n$ # 创建TOP 50 歌单\n```\n\n\n## TODO\n- [2018 Q2](https://github.com/Chengyumeng/spider163/blob/master/doc/2018.Q2.TODO.md)\n- [2018 Q1](https://github.com/Chengyumeng/spider163/blob/master/doc/2018.Q1.TODO.md)\n- [2017 Q4](https://github.com/Chengyumeng/spider163/blob/master/doc/2017.Q4.TODO.md)\n- ...\n\n# 欢迎关注微信公众账号：程天写代码\n![guojingcoooool](https://github.com/Chengyumeng/spider163/blob/master/wechat.jpeg)\n" }, { "path": "doc/2017.Q4.TODO.md", "content": "#### 小版本更新\n* 完善歌词下载流程，支持批量下载 √\n* 完善抓取过程的进度可视化 √\n* 错误流程的管理控制 √\n* 输出改为多彩的形式 √\n* 搜索功能完善（广度更大）√\n* 增加一些自动执行的脚本 √\n* 增加pip支持 √\n* 增加Docker镜像 √\n* 抓取歌单增加多样性 √\n\n#### WEB UI\n* WEB UI支持实时检索\n* WEB UI支持推荐 √\n* WEB UI支持在线抓取 √\n* ...\n\n#### 微信接口\n* ...\n\n#### 跨平台支持\n* 跨windows平台\n* 不同版本的MySQL支持\n* 考虑支持sqlite\n" }, { "path": "doc/2018.Q1.TODO.md", "content": "#### 主要方向\n* 文件类下载（mp3等）[完成下载mp3]\n* 搜索框架（基于ES，可能衍生子项目）[TODO]\n* 最小安装的探索（MySQL依赖太复杂）[生成word]\n* 迁移到python3 [验证性完成]\n* k8s下部署（提供官方镜像）\n* 支持生成PDF[完成支持word，pdf可手动生成]\n\n#### 优化\n* web ui\n* web ui 使用流行的前端框架（angular or react）" }, { "path": "doc/2018.Q2.TODO.md", "content": "#### 主要方向\n* 探索搜索框架【可能通过衍生子项目实现】\n* 优化集成测试\n* 开发邮箱订阅功能[dev]\n* 集成telegraf/influxdb/grafana[dev]\n* 赚钱！赚钱！！赚钱！！！" }, { "path": "hack/docker-compose.yaml", "content": "version: '2'\nservices:\n mysql163:\n image: mysql:5.6.36\n container_name: mysql163\n networks:\n - default\n environment:\n MYSQL_DATABASE: \"spider163\"\n MYSQL_ROOT_PASSWORD: \"a1b2c3d4e\"\n volumes:\n - ./_dev/mysql:/var/lib/mysql:z\n ports:\n - \"3336:3306\"\n expose:\n - \"3306\"\n spider163:\n image: chengtian/spider163:2.7.5\n container_name: spider163\n volumes:\n - ../:/root/code\n ports:\n # 将容器端口与宿主机绑定, 以便外部访问\n - \"1630:1630\"\n expose:\n - \"1630\"\n links:\n - mysql163:mysql163.localhost\n depends_on:\n - mysql163\n networks:\n - default\nnetworks:\n default:\n external:\n name: spider163\n" }, { "path": "hack/spider/Dockerfile", "content": "FROM python:3.6\nRUN mkdir /root/code & mkdir /root/spider163\nWORKDIR /root/code\nADD ./ /root/code/\nADD hack/spider/spider163.conf /root/spider163/spider163.conf\nRUN pip install -e . -i http://mirrors.aliyun.com/pypi/simple --trusted-host=mirrors.aliyun.com && spider163 --help\nENTRYPOINT spider163 webserver\n\n\n" }, { "path": "hack/spider/spider163.conf", "content": "[core]\ndb=mysql://root:a1b2c3d4e@mysql163.localhost/spider163?charset=utf8mb4\nport=1630" }, { "path": "pypi.sh", "content": "python setup.py clean\npython setup.py sdist\ntwine upload dist/*\npython setup.py clean\n" }, { "path": "setup.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*\n\nfrom setuptools import setup, find_packages, Command\nimport os\nimport imp\n\n\nclass CleanCommand(Command):\n \"\"\"Custom clean command to tidy up the project root.\"\"\"\n user_options = []\n\n def initialize_options(self):\n pass\n\n def finalize_options(self):\n pass\n\n def run(self):\n os.system('rm -vrf ./build ./dist ./*.pyc ./*.tgz ./*.egg-info')\n\n\npy3 = imp.load_source(\n 'spider163.version', os.path.join('spider163', 'version.py')).PYTHON3\n\nversion = imp.load_source(\n 'spider163.version', os.path.join('spider163', 'version.py')).VERSION\ndesc = imp.load_source(\n 'spider163.version', os.path.join('spider163', 'version.py')).DESCRIPTION\n\ninstall_requires = [\n \"beautifulsoup4==4.6.0\",\n \"bs4==0.0.1\",\n \"cement==2.10.2\",\n \"certifi==2017.7.27.1\",\n \"chardet==3.0.4\",\n \"idna==2.6\",\n \"Naked==0.1.31\",\n \"pprint==0.1\",\n \"cryptography==2.3\",\n \"PyYAML==3.12\",\n \"requests==2.18.4\",\n \"shellescape==3.4.1\",\n \"SQLAlchemy==1.1.15\",\n \"SQLAlchemy-Utils==0.32.18\",\n \"terminaltables==3.1.0\",\n \"urllib3==1.24.2\",\n \"Logbook==1.1.0\",\n \"colorama==0.3.9\",\n \"flask==1.0\",\n \"python-docx==0.8.6\",\n \"xlwt==1.3.0\"\n]\n\nif py3 is True:\n install_requires.append(\"mysqlclient==1.3.12\")\nelse:\n install_requires.append(\"MySQL-python==1.2.5\")\n\n\nsetup(\n version=version,\n name='spider163',\n author='ChengTian',\n description='简单易用、功能强大的网易云音乐爬虫',\n long_description=desc,\n entry_points={\n \"console_scripts\": [\"spider163=spider163.bin.cli:main\"]\n },\n url='https://github.com/Chengyumeng/spider163',\n author_email='792400644@qq.com',\n packages=find_packages(),\n include_package_data=True,\n license='MIT License',\n zip_safe=False,\n install_requires=install_requires,\n classifiers=[\n 'Development Status :: 5 - Production/Stable',\n 'Environment :: Console',\n 'Environment :: Web Environment',\n 'Programming Language :: Python :: 2.7',\n 'Programming Language :: Python :: 3.4',\n ],\n cmdclass={\n 'clean': CleanCommand,\n },\n\n)\n" }, { "path": "spider163/__init__.py", "content": "\n\n" }, { "path": "spider163/bin/__init__.py", "content": "" }, { "path": "spider163/bin/cli.py", "content": "# -*- coding: utf-8 -*-\n\nimport os, datetime\n\nfrom cement.core.foundation import CementApp\nfrom cement.core.controller import CementBaseController, expose\nfrom cement.core.exc import CaughtSignal\nfrom colorama import Fore\nfrom colorama import init\n\nfrom spider163.utils import pysql\nfrom spider163.spider import playlist\nfrom spider163.spider import mp3\nfrom spider163.spider import music\nfrom spider163.spider import comment\nfrom spider163.spider import lyric\nfrom spider163.spider import search\nfrom spider163.spider import read\nfrom spider163.spider import authorize\nfrom spider163.spider import public as uapi\nfrom spider163.mail import mail\nfrom spider163 import version\nfrom spider163.www import web\nfrom spider163.utils import config\nfrom spider163.utils import pylog\nfrom spider163.utils import healthz\n\nBANNER = \"\"\"\nSpider163 Application v{}\nCopyright (c) {} Cheng Tian Enterprises\nWelcome to Follow My 【微信公众账号】\"程天写代码\"\n\"\"\".format( version.VERSION , datetime.datetime.now().year)\n\ninit(autoreset=True)\n\n\nclass VersionController(CementBaseController):\n class Meta:\n label = 'base'\n description = 'Spider163是Github上最流行的网易云音乐爬虫系统'\n arguments = [\n (['-v', '--version'], dict(action='version', version=BANNER)),\n ]\n\n @expose(help=\"运行前健康检查\")\n def healthz(self):\n healthz.is_correct_config()\n healthz.is_correct_db()\n healthz.can_spider()\n\n @expose(help=\"提供生态信息数据【for telegraf】\")\n def expose(self):\n healthz.expose_data()\n\n\nclass DatabaseController(CementBaseController):\n class Meta:\n label = \"database\"\n description = \"数据库相关操作\"\n arguments = [\n (['-d', '--date'], dict(help=\"时间限制（example：-1，-2）\")),\n (['-t', '--table'], dict(help=\"修改的表（example：playlist，music）\")),\n ]\n\n @expose(help=\"自动生成数据库相关依赖\")\n def initdb(self):\n print(\"正在生成全部数据库表结构……\")\n pysql.initdb()\n\n @expose(help=\"重置数据库配置\")\n def resetdb(self):\n print(\"正在删除全部已下载数据……\")\n pysql.dropdb()\n pysql.initdb()\n\n @expose(help=\"重置过期的playlist/music数据，便于重新抓取\")\n def updatedb(self):\n if self.app.pargs.date is None:\n print(Fore.RED + '没有指定参数 date 无法进行操作')\n return\n if self.app.pargs.table == \"music\":\n m = music.Music()\n m.create_update_strategy(date=int(self.app.pargs.date))\n elif self.app.pargs.table == \"playlist\":\n p = playlist.Playlist()\n p.create_update_strategy(date=int(self.app.pargs.date))\n else:\n print(Fore.RED + '指定参数 table 不正确！')\n\n\nclass SpiderController(CementBaseController):\n class Meta:\n label = \"spider\"\n description = \"爬虫-蜘蛛等相关操作\"\n arguments = [\n (['-p', '--page'],\n dict(help=\"抓取的页码\")),\n (['-c', '--count'],\n dict(help=\"\")),\n (['-s', '--song'],\n dict(help=\"歌曲ID\")),\n (['--classify'],\n dict(help=\"歌曲风格\")),\n (['--path'],\n dict(help=\"存储路径\"))\n ]\n\n @expose(help=\"获取全部歌曲风格列表(作为抓取歌单的参照)\")\n def classify(self):\n playlist.Playlist().get_classify()\n\n @expose(help=\"根据推荐歌单抓取网易云音乐歌单数据(-p --page | --classify)\")\n def playlist(self):\n pg = self.app.pargs.page\n cf = \"全部\"\n pl = playlist.Playlist()\n if self.app.pargs.classify is not None:\n cf = self.app.pargs.classify\n if pg is not None:\n print(Fore.GREEN + '正在抓取 曲风为 {} 的第 {} 页歌单……'.format(cf, pg))\n pl.view_capture(int(pg), cf)\n else:\n for i in range(36):\n print(Fore.GREEN + '正在抓取 曲风为 {} 的第 {} 页歌单……'.format(cf ,i + 1))\n pl.view_capture(i + 1, cf)\n\n @expose(help=\"根据指定的歌单下载歌单歌曲MP3（--playlist | --path）\")\n def mp3(self):\n path = \".\"\n if self.app.pargs.path is not None:\n path = self.app.pargs.path\n if not os.path.exists(path):\n os.makedirs(path)\n if self.app.pargs.playlist is not None:\n m = mp3.MP3()\n m.view_down(self.app.pargs.playlist, path)\n\n\n @expose(help=\"通过歌单抓取网易云音乐歌曲，单次抓取歌单10个(-c --count)\")\n def music(self):\n msc = music.Music()\n if self.app.pargs.count is None:\n msc.views_capture()\n return\n cnt = int(self.app.pargs.count)\n if cnt <= 0:\n print(Fore.RED + \"不合法的--count -c 变量（ > 0 ）\")\n else:\n for i in range(cnt):\n print(Fore.GREEN + '正在执行第 {} 批抓取计划，本次抓取歌单歌曲 10 个\\r\\n'.format(i + 1))\n msc.views_capture()\n\n @expose(help=\"抓取网易云音乐官方排行榜歌单（-c --count）\")\n def toplist(self):\n msc = music.Music()\n cmt = comment.Comment(comment.Comment.Official)\n for id in uapi.top:\n pylog.print_info('正在抓取官方排行榜 歌单ID：{} 歌单名字：{}'.format(id, uapi.top[id]))\n msc.view_capture(id)\n cnt = int(self.app.pargs.count)\n if cnt <= 0:\n print(Fore.RED + \"不合法的--count -c 变量（ > 0 ）\")\n else:\n cmt.auto_view(cnt)\n\n\n @expose(help=\"通过音乐列表抓取网易云音乐热评，单次抓取音乐1首(-c --count),也可以指定歌曲ID(-s --song)\")\n def comment(self):\n cmt = comment.Comment()\n if self.app.pargs.song is not None:\n print(Fore.BLUE + '正在执行抓取歌曲 {} 热门评论计划'.format(self.app.pargs.song))\n cmt.view_capture(int(self.app.pargs.song), 1)\n print(Fore.GREEN + '抓取完成\\r\\n')\n return\n if self.app.pargs.count is not None:\n print(Fore.GREEN + '正在执行批量抓取热门评论计划，本次计划抓取歌曲 {} 首\\r\\n'.format(self.app.pargs.count))\n cmt.auto_view(int(self.app.pargs.count))\n else:\n cmt.auto_view(1)\n\n @expose(help=\"通过音乐列表抓取网易云音乐歌词,可以指定抓取歌曲数量（-c --count），也可以指定歌曲ID（-s --song）\")\n def lyric(self):\n lrc = lyric.Lyric()\n if self.app.pargs.song is not None:\n print(Fore.BLUE + '正在执行抓取歌曲 {} 歌词的计划'.format(self.app.pargs.song))\n lrc.view_lyric(self.app.pargs.song)\n print(Fore.GREEN + '抓取完成\\r\\n')\n elif self.app.pargs.count is not None:\n print(Fore.GREEN + '正在执行批量抓取歌词计划，本次计划抓取歌曲 {} 首\\r\\n'.format(self.app.pargs.count))\n lrc.view_lyrics(int(self.app.pargs.count))\n else:\n print(\"您至少指定--song或者--count一个参数\")\n\n\nclass QueryController(CementBaseController):\n class Meta:\n label = \"query\"\n stacked_on = 'base'\n description = \"爬虫-蜘蛛等相关操作\"\n arguments = [\n (['--playlist'],\n dict(help=\"\")),\n (['-q', '--query'],\n dict(help=\"\")),\n ]\n\n @expose(help=\"通过歌单ID和歌曲ID获取歌单、歌曲相关信息（--song --playlist）\")\n def get(self):\n if self.app.pargs.song is not None:\n comment.Comment().get_music(self.app.pargs.song)\n lyric.Lyric().get_lyric(self.app.pargs.song)\n if self.app.pargs.playlist is not None:\n music.Music().get_playlist(self.app.pargs.playlist)\n\n @expose(help=\"搜索功能(-q --query)\")\n def search(self):\n if self.app.pargs.query is not None:\n search.searchSong(self.app.pargs.query)\n search.searchAlbum(self.app.pargs.query)\n search.searchSinger(self.app.pargs.query)\n search.searchPlaylist(self.app.pargs.query)\n\n @expose(help=\"生成文档（word）(--playlist | --count)\")\n def doc(self):\n if self.app.pargs.playlist is not None:\n read.print_pdf(self.app.pargs.playlist)\n\n if self.app.pargs.count is not None:\n read.print_comment(int(self.app.pargs.count))\n\n @expose(help=\"Spider163邮件系统(--playlist)\")\n def mail(self):\n try:\n if self.app.pargs.playlist is not None:\n playlist_id = int(self.app.pargs.playlist)\n mail.music(playlist_id)\n except Exception as e:\n print(\"{} 发送邮件发生意外 {}\".format(Fore.RED, e))\n\n\nclass WebController(CementBaseController):\n class Meta:\n label = \"web\"\n stacked_on = 'base'\n description = \"网络平台\"\n arguments = [\n ]\n\n @expose(help=\"Spider163管理Web平台\")\n def webserver(self):\n try:\n webport = config.get_port()\n web.app.run(host=\"0.0.0.0\", port=webport, debug=True)\n except Exception as e:\n print(\"{} 退出web服务：{}\".format(Fore.RED, e))\n\n\nclass AuthController(CementBaseController):\n class Meta:\n label = \"auth\"\n stacked_on = 'base'\n description = \"登录授权操作\"\n arguments = [\n (['--username'],\n dict(help=\"登录账号（必须为中国大陆手机号）\")),\n (['--password'],\n dict(help=\"登录密码\")),\n ]\n\n @expose(help=\"维护评论Top 50 歌单\")\n def top50(self):\n if self.app.pargs.username is None:\n pylog.print_warn(\"没有指定用户名（--username）参数，无法执行任务！\")\n return\n if self.app.pargs.password is None:\n pylog.print_warn(\"没有指定密码（--password）参数，无法执行任务！\")\n return\n if self.app.pargs.playlist is None:\n pylog.print_warn(\"没有指定目标歌单ID（--playlist），无法执行任务！\")\n return\n username = self.app.pargs.username\n password = self.app.pargs.password\n playlist_id = self.app.pargs.playlist\n\n cmd = authorize.Command()\n cmd.do_login(username, password)\n cmd.clear_playlist(playlist_id)\n cmd.create_playlist_comment_top100(playlist_id)\n\n\n\nclass App(CementApp):\n class Meta:\n label = \"Spider163\"\n base_controller = \"base\"\n handlers = [VersionController, DatabaseController, SpiderController, QueryController, WebController, AuthController]\n\n\ndef main():\n with App() as app:\n try:\n app.run()\n except CaughtSignal as e:\n pylog.print_warn(\"控制台异常：{}\".format(e))\n except Exception as e:\n pylog.print_err(\"执行抓取任务遭遇配置异常： {}\".format(e))\n\n\n\n\n" }, { "path": "spider163/bin/cli_test.py", "content": "import unittest\n\nfrom spider163.spider import playlist\nfrom spider163.spider import mp3\nfrom spider163.spider import search\nfrom spider163.spider import read\nfrom spider163.utils import healthz\n\n\nclass TestStringMethods(unittest.TestCase):\n\n def test_config(self):\n healthz.is_correct_config()\n healthz.is_correct_db()\n healthz.can_spider()\n\n def test_classify(self):\n playlist.Playlist().get_classify()\n\n def test_mp3(self):\n m = mp3.MP3()\n # m.view_down(2127220577, \".\")\n\n def test_search(self):\n search.searchSong(\"李荣浩\")\n search.searchAlbum(\"韩寒\")\n search.searchSinger(\"林依晨\")\n search.searchPlaylist(\"SHE\")\n\n def test_doc(self):\n read.print_pdf(2127220577)\n\n\nif __name__ == '__main__':\n unittest.main()" }, { "path": "spider163/mail/__init__.py", "content": "" }, { "path": "spider163/mail/mail.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nfrom spider163 import settings\nfrom spider163 import version\nfrom spider163.spider import public as uapi\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import mail\nfrom spider163.utils import config\n\n\ndef music(playlist_id):\n if playlist_id not in uapi.top.keys():\n pylog.print_info(\"歌单 {} 不在合法排行榜序列,合法歌单如下\".format(playlist_id))\n tb = [['歌单ID','歌单名字']]\n for k,v in uapi.top.items():\n tb.append([k, v])\n pylog.Table(tb)\n return\n data = settings.Session.query(\n pysql.Toplist163.song_name,\n pysql.Toplist163.song_id,\n pysql.Toplist163.author,\n pysql.Toplist163.comment.label(\"count\")\n ).filter(pysql.Toplist163.playlist_id == playlist_id, pysql.Toplist163.mailed == \"N\").order_by(pysql.Toplist163.id.asc()).slice(1,5).all()\n\n page = []\n body = version.MAILBODY\n title = version.MAILMUSIC\n comments = version.MAILCOMMENT\n for m in data:\n settings.Session.query(pysql.Toplist163).filter(pysql.Toplist163.song_id == m[1]).update({'mailed': 'Y'})\n settings.Session.commit()\n detail = \"\"\n cms = settings.Session.query(pysql.Comment163).filter(pysql.Comment163.song_id == m[1]).order_by(pysql.Comment163.id).all()\n for c in cms:\n detail = detail + comments.format(c.author,c.liked,c.txt)\n head = title.format(m[1], m[0], m[2], m[3], detail)\n page.append(head + detail)\n body = body.format(uapi.top[playlist_id],\"
\".join(page))\n\n host,port,users = config.get_mail()\n for user in users.split(\",\"):\n mail.send_email(host,port,\"spider163每日网易云音乐分享\", user, body)" }, { "path": "spider163/settings.py", "content": "# -*- coding: utf-8 -*-\n\nfrom sqlalchemy import create_engine\nfrom sqlalchemy.orm import scoped_session, sessionmaker\nfrom spider163.utils import config\nfrom sqlalchemy_utils.functions import database_exists\n\nfrom spider163.utils import pylog\n\n\ndef configure_orm():\n global engine\n global Session\n engine_args = {}\n Session = scoped_session(\n sessionmaker(autocommit=False, autoflush=False))\n try:\n if database_exists(config.get_db()) is False:\n create_engine(config.get_mysql()['uri'], echo=False).execute(\"create database IF NOT EXISTS {} DEFAULT CHARACTER SET utf8mb4\".format(config.get_mysql()['db']))\n engine = create_engine(config.get_db(), **engine_args)\n Session = scoped_session(\n sessionmaker(autocommit=False, autoflush=False, bind=engine))\n except Exception as e:\n pylog.print_err(\"初始化数据库出现问题： {}\".format(e))\n\n\n\n\n\n" }, { "path": "spider163/spider/__init__.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nfrom spider163 import settings\nfrom spider163.utils import pylog\ntry:\n settings.configure_orm()\nexcept Exception as e:\n pylog.print_info(\"无法执行数据库相关的任务： {}\".format(e))" }, { "path": "spider163/spider/authorize.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport os\nimport requests\nimport time\n\nfrom spider163.utils import encrypt\nfrom spider163.spider import public as uapi\nfrom spider163.spider import music\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\nfrom spider163 import settings\n\nclass Command():\n\n def __init__(self):\n modulus = uapi.comment_module\n pubKey = uapi.pubKey\n secKey = uapi.secKey\n self.__encSecKey = self.rsaEncrypt(secKey, pubKey, modulus)\n self.session = requests.session()\n self.session.headers = uapi.header\n\n def createPlaylistParams(self,ids,playlist_id,cmd,csrf_token):\n text = '{\"trackIds\": ['+\",\".join(ids) + '],\"pid\": \"{}\",\"op\": \"{}\",\"csrf_token\": \"{}\"'.format(playlist_id,cmd,csrf_token) + '}'\n nonce = '0CoJUm6Qyw8W8jud'\n nonce2 = 16 * 'F'\n encText = encrypt.aes(\n encrypt.aes(text, nonce).decode(\"utf-8\"), nonce2\n )\n return encText\n\n def createPlaylistRemoveParams(self):\n pass\n\n def createLoginParams(self,username,password):\n psw = tools.md5(password)\n text = '{' + '\"phone\": \"{}\",\"password\": \"{}\",\"rememberLogin\": \"true\"'.format(username,psw)+'}'\n nonce = '0CoJUm6Qyw8W8jud'\n nonce2 = 16 * 'F'\n encText = encrypt.aes(\n encrypt.aes(text, nonce).decode(\"utf-8\"), nonce2\n )\n return encText\n\n def rsaEncrypt(self, text, pubKey, modulus):\n text = text[::-1]\n rs = int(tools.hex(text), 16)**int(pubKey, 16) % int(modulus, 16)\n return format(rs, 'x').zfill(256)\n\n def createSecretKey(self, size):\n return (\n ''.join(map(lambda xx: (hex(ord(xx))[2:]), os.urandom(size)))\n )[0:16]\n\n def post_playlist_add(self,ids, playlist_id=2098905487, csrf_token=\"da2216e4b4ca4efcfab94d8d4920ef9\"):\n data = {\n 'params': self.createPlaylistParams(ids,playlist_id,\"add\",csrf_token),\n 'encSecKey': self.__encSecKey\n }\n url = uapi.playlist_add_api.format(csrf_token)\n req = self.session.post(\n url, data=data, timeout=100\n )\n return req.json()\n\n def post_playlist_delete(self, ids, playlist_id=2098905487, csrf_token=\"da2216e4b4ca4efcfab94d8d4920ef9\"):\n data = {\n 'params': self.createPlaylistParams(ids, playlist_id, \"delete\", csrf_token),\n 'encSecKey': self.__encSecKey\n }\n url = uapi.playlist_add_api.format(csrf_token)\n req = self.session.post(\n url, data=data, timeout=10\n )\n return req.json()\n\n def do_login(self,username,password):\n data = {\n 'params': self.createLoginParams(username,password),\n 'encSecKey': self.__encSecKey\n }\n url = uapi.login_api\n\n res = self.session.post(url, data=data, timeout=10).json()\n # TODO 处理rep信息\n if res[\"code\"] != 200:\n if res[\"code\"] == 400:\n raise Exception(\"用户名不合法！\")\n raise Exception(res[\"msg\"])\n return res\n\n def clear_playlist(self,playlist_id=2098905487):\n m = music.Music()\n data = m.curl_playlist(playlist_id)\n for d in data[\"tracks\"]:\n res = self.post_playlist_delete([str(d[\"id\"]),],playlist_id)\n if res[\"code\"] == 200:\n pylog.print_info(\"成功删除《{}》到指定歌单,歌单目前包含歌曲 {} 首\".format(d[\"name\"],res[\"count\"]))\n else:\n time.sleep(5)\n pylog.print_warn(\"歌曲《{}》不存在于歌单中！\".format(d[\"name\"]))\n pylog.print_warn(\"删除歌单歌曲任务完成，请检查！\")\n\n def create_playlist_comment_top100(self,playlist_id=2098905487):\n data = settings.Session.query(pysql.Music163.song_name, pysql.Music163.song_id,pysql.Music163.comment.label(\"count\")).order_by(\n pysql.Music163.comment.label(\"count\").desc()).limit(200).all()\n for d in data:\n res = self.post_playlist_add([str(d[1]),],playlist_id)\n if res[\"code\"] == 502:\n pylog.print_warn(\"歌曲《{}》已经存在于歌单中！\".format(d[0]))\n elif res[\"code\"] == 200:\n pylog.print_info(\"成功添加《{}》到指定歌单,歌单目前包含歌曲 {} 首\".format(d[0],res[\"count\"]))\n else:\n time.sleep(5)\n pylog.print_warn(\"歌曲《{}》没有添加成功！\".format(d[0]))\n pylog.print_warn(\"增加歌单歌曲任务完成，请检查！\")" }, { "path": "spider163/spider/comment.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport os\nimport requests\nimport datetime\n\nfrom bs4 import BeautifulSoup\nfrom terminaltables import AsciiTable\n\nfrom spider163 import settings\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\nfrom spider163.utils import encrypt\nfrom spider163.spider import public as uapi\n\n\nclass Comment:\n Common = 'common music'\n Official = 'official music'\n\n def __init__(self, music_type=Common):\n self.__headers = uapi.header\n self.music_type = music_type\n self.session = settings.Session()\n modulus = uapi.comment_module\n pubKey = uapi.pubKey\n secKey = uapi.secKey\n self.__encSecKey = self.rsaEncrypt(secKey, pubKey, modulus)\n\n def createParams(self, page=1):\n if page == 1:\n text = (\n '{rid:\"\", offset:\"0\", total:\"true\", limit:\"20\", csrf_token:\"\"}'\n )\n else:\n offset = str((page-1)*20)\n text = (\n '{rid:\"\", offset:\"{}\", total:\"{}\", limit:\"20\", '\n 'csrf_token:\"\"}'.format(offset, 'false')\n )\n nonce = '0CoJUm6Qyw8W8jud'\n nonce2 = 16 * 'F'\n encText = encrypt.aes(\n encrypt.aes(text, nonce).decode(\"utf-8\"), nonce2\n )\n return encText\n\n def rsaEncrypt(self, text, pubKey, modulus):\n text = text[::-1]\n rs = int(tools.hex(text), 16)**int(pubKey, 16) % int(modulus, 16)\n return format(rs, 'x').zfill(256)\n\n def createSecretKey(self, size):\n return (\n ''.join(map(lambda xx: (hex(ord(xx))[2:]), os.urandom(size)))\n )[0:16]\n\n def post(self,song_id, page):\n data = {\n 'params': self.createParams(page),\n 'encSecKey': self.__encSecKey\n }\n url = uapi.comment_url.format(song_id)\n req = requests.post(\n url, headers=self.__headers, data=data, timeout=10\n )\n return req.json()\n\n def views_capture(self, song_id, page=1, pages=1024):\n if pages > 1:\n while page < pages:\n pages = self.view_capture(song_id, page)\n page = page + 1\n else:\n self.view_capture(song_id, 1)\n self.view_links(song_id)\n\n def view_capture(self, song_id, page=1):\n if page == 1:\n self.session.query(pysql.Comment163).filter(\n pysql.Comment163.song_id == song_id\n ).delete()\n self.session.commit()\n try:\n data = self.post(song_id,page)\n for comment in data['comments']:\n if comment['likedCount'] > 30:\n txt = tools.encode(comment['content'])\n author = tools.encode(comment['user']['nickname'])\n liked = comment['likedCount']\n self.session.add(pysql.Comment163(\n song_id=song_id, txt=txt, author=author, liked=liked\n ))\n self.session.flush()\n if page == 1:\n for comment in data['hotComments']:\n txt = tools.encode(comment['content'])\n author = tools.encode(comment['user']['nickname'])\n liked = comment['likedCount']\n self.session.add(pysql.Comment163(\n song_id=song_id, txt=txt, author=author, liked=liked\n ))\n self.session.flush()\n cnt = int(data['total'])\n self.session.query(pysql.Music163).filter(\n pysql.Music163.song_id == song_id\n ).update({'done': 'Y', 'comment': cnt, 'update_time': datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n if self.music_type == self.Official:\n self.session.query(pysql.Toplist163).filter(\n pysql.Toplist163.song_id == song_id\n ).update(\n {'done': 'Y', 'comment': cnt})\n self.session.commit()\n return cnt / 20\n except Exception as e:\n self.session.rollback()\n self.session.query(pysql.Music163).filter(\n pysql.Music163.song_id == song_id\n ).update({'done': 'E', 'comment': -2})\n self.session.commit()\n pylog.log.error(\n \"解析歌曲评论的时候出现问题:{} 歌曲ID：{} 页码：{}\".format(\n e, song_id, page\n )\n )\n raise\n\n def view_links(self, song_id):\n url = \"http://music.163.com/song?id=\" + str(song_id)\n data = {'id': str(song_id)}\n headers = {\n 'Cookie': 'MUSIC_U=e45797021db3403ab9fffb11c0f70a7994f71177b26efb5169b46948f2f9a60073d23a2665346106c9295f8f6dbb6c7731b299d667364ed3;' # noqa\n }\n try:\n req = requests.get(url, headers=headers, data=data, timeout=100)\n sup = BeautifulSoup(req.content, \"html.parser\")\n for link in sup.find_all('li', class_=\"f-cb\"):\n html = link.find('a', 's-fc1')\n if html is not None:\n title = tools.encode(html.get('title'))\n song_id = html.get('href')[9:]\n author = tools.encode(link.find(\n 'div', 'f-thide s-fc4'\n ).find('span').get('title'))\n if pysql.single(\"music163\", \"song_id\", song_id) is True:\n self.session.add(pysql.Music163(\n song_id=song_id, song_name=title, author=author\n ))\n self.session.flush()\n for link in sup.find_all('a', 'sname f-fs1 s-fc0'):\n play_link = link.get(\"href\").replace(\"/playlist?id=\", \"\")\n play_name = tools.encode(link.get(\"title\"))\n if pysql.single(\"playlist163\", \"link\", play_link) is True:\n self.session.add(pysql.Playlist163(\n title=play_name, link=play_link, cnt=-1,\n dsc=\"来源：热评\"\n ))\n self.session.flush()\n except Exception as e:\n pylog.log.error(\"解析页面推荐时出现问题：{} 歌曲ID：{}\".format(e, song_id))\n\n def auto_view(self, count=1):\n song = []\n if self.music_type == self.Common:\n msc = self.session.query(pysql.Music163).filter(pysql.Music163.done == \"N\").order_by(pysql.Music163.id).limit(count)\n for m in msc:\n try:\n print(\"抓取热评 ID {} 歌曲 {}\".format(m.song_id, pylog.Blue(tools.encode(m.song_name))))\n self.views_capture(m.song_id, 1, 1)\n song.append({\"name\": m.song_name, \"author\": m.author,\"song_id\": m.song_id})\n except Exception as e:\n pylog.log.error(\"自动抓取热评出现异常：{} 歌曲ID：{}\".format(e, m.song_id))\n elif self.music_type == self.Official:\n msc = self.session.query(pysql.Toplist163).filter(pysql.Toplist163.done == \"N\").order_by(pysql.Toplist163.id).limit(count)\n for m in msc:\n try:\n print(\"抓取官方榜单歌曲热评 ID {} 歌曲 {}\".format(m.song_id, pylog.Blue(tools.encode(m.song_name))))\n self.views_capture(m.song_id, 1, 2) # 意味着每一页的评论都抓取\n song.append({\"name\": m.song_name, \"author\": m.author,\"song_id\": m.song_id})\n except Exception as e:\n pylog.log.error(\"自动抓取官方榜单热评出现异常：{} 歌曲ID：{}\".format(e, m.song_id))\n\n return song\n\n def get_music(self, music_id):\n self.view_capture(int(music_id), 1)\n url = uapi.music_api.format(music_id, music_id)\n data = tools.curl(url,self.__headers)\n music = data['songs']\n print(\"《\" + tools.encode(music[0]['name']) + \"》\")\n author = []\n for a in music[0]['artists']:\n author.append(tools.encode(a['name']))\n album = str(tools.encode(music[0]['album']['name']))\n print(\"演唱：{} 专辑：{}\".format(\"，\".join(author), album))\n comments = self.session.query(pysql.Comment163).filter(\n pysql.Comment163.song_id == int(music_id)\n )\n tb = AsciiTable([[\"序号\", \"作者\", \"评论\", \"点赞\"]])\n max_width = tb.column_max_width(2) - tb.column_max_width(2) % 3\n cnt = 0\n try:\n for cmt in comments:\n cnt = cnt + 1\n au = tools.encode(cmt.author)\n txt = \"\"\n length = 0\n for u in cmt.txt:\n txt = txt + u\n if ord(u) < 128:\n length = length + 3\n else:\n length = length + 1\n if length == max_width:\n txt = txt + \"\\n\"\n length = 0\n liked = str(cmt.liked)\n tb.table_data.append([str(cnt), str(au), str(txt), liked])\n print(tb.table)\n except UnicodeEncodeError:\n pylog.log.info(\"获取歌曲详情编码存在问题，转为非表格形式，歌曲ID：{}\".format(music_id))\n for cmt in comments:\n print(\"评论： {}\".format(tools.encode(cmt.txt)))\n print(\n \"作者： {} 点赞： {}\".format(\n tools.encode(cmt.author), str(cmt.liked)\n )\n )\n print(\"\")\n except Exception as e:\n pylog.print_warn(\"获取歌曲时出现异常： {} 歌曲ID：{}\".format(e, music_id))\n\n\n\"\"\"\ncurl 'http://music.163.com/eapi/v1/resource/hotcomments/R_SO_4_439915614?limit=30&offset=30' -H 'MUSIC_U=b14e134d57809f6f2cad59071320962f70351b98b979328186bab129f64585d877f086e4dccc2d68d4631490c2eade1fcb19b68a33677785; versioncode=114; mobilename=SM901; buildver=1517983086; resolution=1920x1080; __csrf=33a68d2b8c79270a8b770ef851ca322b; channel=chuizi; os=android' -H 'Connection: keep-alive' --data 'params=E8C4EA3B185998031030633EE8255315B179427FC8206489FBB24BB0592665FDDD3729945E06958F8E1D7E9D3B8336C82A051CA692ED4EAD270699F0CCFA87BE252577E9DBA7D4ACE1ECAFAB78C190513D439E46D2E62F125C771C5A05EBF5B7F8A9783A2721EE3894DFFE3AAF6751B7A7C412947A0C49CC73F7DBE0D285B45F97A16013F7B4576F2CD2D611150B0ABFF40C8FCE075ED7ED25BE61CCA9154A4F1CB23BF9C720A7BE0A952F25EC77E746B1688AE3FCDE73BC19600468DB7D9175013144D6D759C1660A471A66B8C42B171A2BB3AA48BA8638978B7299A10F08A472D1E13D071136C670A3E748E7DFD5F0E6819E725D793FB2D2BB6852002D1E30A850F90D7F6556C50394E83D4F3FCF79C9721E766D8758399F17538CA1DF87E32DF3468FC6EB592EF5AE7F0E5D295184AEC16C1019FD6F54B41AE835D1967CA7F7E892A6059B95EBACF785D1512402C13A3C8A491970030A1F8E97B35DEDEECFF34BA27F5869047DB5FAABBFEFDE833E3B7E8C7B15C6B1F0764A1CD298039BF6BC7C38832C5B8B4644714C25F4CE1F256AC2456B9D315941CF3CBF69224CF3F0DB7D4BE81486C72562C024C6EB3897D0DED5740A345CAE3592482BD36208DA99F197119A497DC736E58ABF7C80A338EA64059455FD065C61D46499586DDD6A4BEBAF431C2839D49EE192CAA3165B3B6B116FF45760DD0C94FCC5ED5E6E0B990662EC900671ED89AEEB6B7A2F73B7008FC711CB44F9EE23F53415A6C39DF781D13A11B9BEBC87156F67DEC8E6D023394953735006FD471F3A7885B57C0F826CBB3CD4F286BC407FDBA5B4D83ED8CAE4BF17E7F07C2DC3FD072A21727B2FDECB551EB05364287AB201904518E10007EED6' --compressed\n\"\"\"" }, { "path": "spider163/spider/lyric.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nfrom spider163.spider import public as uapi\nfrom spider163 import settings\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\n\n\nclass Lyric:\n\n def __init__(self):\n self.__headers = uapi.header\n self.session = settings.Session()\n\n def view_lyric(self, song_id):\n url = uapi.lyric_url.format(str(song_id))\n try:\n data = tools.curl(url,self.__headers)\n lrc = data['lrc']['lyric']\n if pysql.single(\"lyric163\", \"song_id\", song_id):\n self.session.add(pysql.Lyric163(song_id=song_id, txt=lrc))\n self.session.query(pysql.Music163).filter(pysql.Music163.song_id == song_id).update({\"has_lyric\": \"Y\"})\n self.session.commit()\n except Exception as e:\n self.session.query(pysql.Music163).filter(pysql.Music163.song_id == song_id).update({\"has_lyric\": \"E\"})\n self.session.commit()\n pylog.log.error(\"抓取歌词出现问题：{} 歌曲ID：{}\".format(e, song_id))\n # raise\n\n def get_lyric(self, song_id):\n self.view_lyric(song_id)\n lrc = self.session.query(pysql.Lyric163).filter(pysql.Lyric163.song_id == song_id)\n print(lrc[0].txt)\n\n def view_lyrics(self, count):\n song = []\n for i in range(int(count/10)):\n ms = self.session.query(pysql.Music163).filter(pysql.Music163.has_lyric == \"N\").order_by(pysql.Music163.id).limit(10)\n for m in ms:\n print(\"正在抓取歌词 ID {} 歌曲 {}\".format(m.song_id, pylog.Blue(tools.encode(m.song_name))))\n self.view_lyric(m.song_id)\n song.append({\"name\": m.song_name,\"author\": m.author,\"comment\": m.comment})\n ms = self.session.query(pysql.Music163).filter(pysql.Music163.has_lyric == \"N\").order_by(pysql.Music163.id).limit(count%10)\n for m in ms:\n print(\"正在抓取歌词 ID {} 歌曲 {}\".format(m.song_id, pylog.Blue(tools.encode(m.song_name))))\n self.view_lyric(m.song_id)\n song.append({\"name\": m.song_name, \"author\": m.author, \"comment\": m.comment})\n return song\n" }, { "path": "spider163/spider/mp3.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport os\nimport requests\n\nfrom terminaltables import AsciiTable\n\nfrom spider163 import settings\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\nfrom spider163.utils import encrypt\nfrom spider163.spider import public as uapi\n\n\nclass MP3:\n\n def __init__(self):\n self.__headers = uapi.header\n self.session = settings.Session()\n modulus = uapi.comment_module\n pubKey = uapi.pubKey\n secKey = uapi.secKey\n self.__encSecKey = self.rsa_encrypt(secKey, pubKey, modulus)\n\n def create_params(self, song_id):\n text = '{\"ids\":[' + str(song_id) + '], br:\"320000\",csrf_token:\"csrf\"}'\n nonce = '0CoJUm6Qyw8W8jud'\n nonce2 = 16 * 'F'\n encText = encrypt.aes(\n encrypt.aes(text, nonce).decode(\"utf-8\"), nonce2\n )\n return encText\n\n def rsa_encrypt(self, text, pubKey, modulus):\n text = text[::-1]\n rs = int(tools.hex(text), 16)**int(pubKey, 16) % int(modulus, 16)\n return format(rs, 'x').zfill(256)\n\n def create_secretKey(self, size):\n return (\n ''.join(map(lambda xx: (hex(ord(xx))[2:]), os.urandom(size)))\n )[0:16]\n\n def view_down(self, playlist_id, path=\".\"):\n list = self.get_playlist(str(playlist_id))\n msg = {\"success\": 0, \"failed\": 0, \"failed_list\": []}\n for music in list['tracks']:\n pylog.print_info(\n \"正在下载歌曲 {}-{}.mp3\".format(\n tools.encode(music['name']),\n tools.encode(music['artists'][0]['name'])\n )\n )\n link = self.get_mp3_link(music[\"id\"])\n if link is None:\n msg[\"failed\"] = msg[\"failed\"] + 1\n msg[\"failed_list\"].append(music)\n continue\n r = requests.get(link)\n with open(\"{}/{}-{}{}\".format(\n path,\n tools.encode(music['name']).replace(\"/\", \"-\"),\n tools.encode(music['artists'][0]['name']).replace(\"/\", \"-\"),\n \".mp3\"\n ), \"wb\") as code:\n code.write(r.content)\n msg[\"success\"] = msg[\"success\"] + 1\n pylog.print_warn(\n \"下载成功：{} 首，下载失败：{}首\".format(msg[\"success\"], msg[\"failed\"])\n )\n tb = [[\"歌曲名字\", \"艺术家\", \"ID\"]]\n for music in msg[\"failed_list\"]:\n n = music['name'].encode(\"utf-8\")\n a = music['artists'][0]['name'].encode(\"utf-8\")\n i = music['id']\n tb.append([n, a, i])\n print(AsciiTable(tb).table)\n\n def get_playlist(self, playlist_id):\n url = uapi.playlist_api.format(playlist_id)\n try:\n data = tools.curl(url,self.__headers)\n playlist = data['result']\n return playlist\n except Exception as e:\n raise\n\n def get_mp3_link(self, song_id):\n data = {\n 'params': self.create_params(song_id),\n 'encSecKey': self.__encSecKey\n }\n url = uapi.mp3_url\n try:\n req = requests.post(\n url, headers=self.__headers, data=data, timeout=10\n ).json()\n if req['code'] == 200:\n return req['data'][0]['url']\n except Exception as e:\n raise\n" }, { "path": "spider163/spider/music.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport datetime\n\nfrom spider163.spider import public as uapi\nfrom spider163 import settings\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\nfrom terminaltables import AsciiTable\n\n\nclass Music:\n \n def __init__(self):\n self.__headers = uapi.header\n self.__url = uapi.music_url\n self.session = settings.Session()\n\n def views_capture(self,source=None):\n playlist = {}\n if source is None:\n urls = self.session.query(pysql.Playlist163).filter(pysql.Playlist163.done == 'N').order_by(pysql.Playlist163.id).limit(10)\n else:\n if source.startswith(\"曲风：\") is False:\n source = \"曲风：\" + source\n urls = self.session.query(pysql.Playlist163).filter(pysql.Playlist163.done == 'N',pysql.Playlist163.dsc==source).order_by(pysql.Playlist163.id).limit(1)\n for url in urls:\n print(\"正在抓取歌单《{}》的歌曲……\".format(tools.encode(url.title)))\n songs = self.view_capture(url.link)\n playlist[tools.encode(url.title)] = songs\n return playlist\n\n def view_capture(self, link):\n url = self.__url + str(link)\n songs = []\n try:\n data = self.curl_playlist(link)\n musics = data['tracks']\n exist = 0\n for music in musics:\n name = tools.encode(music['name'])\n authors = []\n for art in music['artists']:\n authors.append(tools.encode(art['name']))\n if music[\"bMusic\"] is None:\n play_time = 0\n else:\n play_time = music[\"bMusic\"][\"playTime\"]\n if pysql.single(\"music163\", \"song_id\", (music['id'])) is True:\n self.session.add(pysql.Music163(song_id=music['id'],song_name=name,author=\",\".join(authors),playTime=play_time))\n self.session.commit()\n exist = exist + 1\n songs.append({\"name\": name,\"author\": \",\".join(authors)})\n else:\n pylog.log.info('{} : {} {}'.format(\"重复抓取歌曲\", name, \"取消持久化\"))\n # 处理官方榜单\n if int(link) in uapi.top.keys():\n updateTime = datetime.datetime.fromtimestamp(data['updateTime'] / 1000).strftime(\n \"%Y-%m-%d %H:%M:%S\")\n createTime = datetime.datetime.fromtimestamp(data['createTime'] / 1000).strftime(\n \"%Y-%m-%d %H:%M:%S\")\n position = music['position']\n lastrank = 100000000\n with tools.ignored(Exception):\n lastrank = music['lastRank']\n cnt = self.session.query(pysql.Toplist163).filter(pysql.Toplist163.update_time == updateTime,\n pysql.Toplist163.song_id == music['id'],\n pysql.Toplist163.playlist_id == link).count()\n mcnt = self.session.query(pysql.Toplist163).filter(pysql.Toplist163.mailed == \"Y\",\n pysql.Toplist163.song_id == music['id'],\n pysql.Toplist163.playlist_id == link).count()\n if cnt == 0:\n mailed = \"N\"\n if mcnt > 0:\n mailed = \"Y\"\n self.session.add(pysql.Toplist163(song_id=music['id'],song_name=name,author=\",\".join(authors),\n playTime=play_time,position=position,playlist_id=link,\n lastRank=lastrank,\n mailed = mailed,\n create_time=createTime,\n update_time=updateTime))\n self.session.commit()\n\n print(\"歌单包含歌曲 {} 首,数据库 merge 歌曲 {} 首 \\r\\n\".format(len(musics), exist))\n self.session.query(pysql.Playlist163).filter(pysql.Playlist163.link == link).update({'done': 'Y','update_time': datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n self.session.commit()\n return songs\n except Exception as e:\n pylog.log.error(\"抓取歌单页面存在问题：{} 歌单ID：{}\".format(e, url))\n self.session.query(pysql.Playlist163).filter(pysql.Playlist163.link == url).update({'done': 'E', 'update_time': datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n self.session.commit()\n\n def curl_playlist(self,playlist_id):\n url = uapi.playlist_api.format(playlist_id)\n try:\n data = tools.curl(url, self.__headers)\n playlist = data['result']\n self.session.query(pysql.Playlist163).\\\n filter(pysql.Playlist163.link == playlist_id).\\\n update({\"playCount\": playlist[\"playCount\"],\n \"shareCount\": playlist[\"shareCount\"],\n \"commentCount\": playlist[\"commentCount\"],\n \"description\": playlist[\"description\"],\n \"tags\":\",\".join(playlist[\"tags\"])})\n return playlist\n except Exception as e:\n pylog.Log(\"抓取歌单页面存在问题：{} 歌单ID：{}\".format(e, playlist_id))\n # pylog.print_warn(\"抓取歌单页面存在问题：{} 歌单ID：{}\".format(e, playlist_id))\n self.session.query(pysql.Playlist163).filter(pysql.Playlist163.link == playlist_id).update({'done': 'E', 'update_time': datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n self.session.commit()\n\n def get_playlist(self, playlist_id):\n self.view_capture(int(playlist_id))\n playlist = self.curl_playlist(playlist_id)\n\n print(\"《\" + tools.encode(playlist['name']) + \"》\")\n author = tools.encode(playlist['creator']['nickname'])\n pc = str(playlist['playCount'])\n sc = str(playlist['subscribedCount'])\n rc = str(playlist['shareCount'])\n cc = str(playlist['commentCount'])\n with tools.ignored(Exception):\n print(\"维护者：{} 播放：{} 关注：{} 分享：{} 评论：{}\".format(author, pc, sc, rc, cc))\n print(\"描述：{}\".format(tools.encode(playlist['description'])))\n print(\"标签：{}\".format(\",\".join(tools.encode(playlist['tags']))))\n\n tb = [[\"ID\", \"歌曲名字\", \"艺术家\", \"唱片\"]]\n for music in playlist['tracks']:\n artists = []\n for s in music['artists']:\n artists.append(s['name'])\n ms = tools.encode(music['name'])\n ar = tools.encode(\",\".join(artists))\n ab = tools.encode(music['album']['name'])\n id = music['id']\n tb.append([id, ms, ar, ab])\n print(AsciiTable(tb).table)\n\n # date\n def create_update_strategy(self, **kwargs):\n date = (datetime.datetime.now() + datetime.timedelta(days=kwargs[\"date\"])).strftime(\"%Y-%m-%d %H:%S:%M\")\n self.session.query(pysql.Music163).filter(pysql.Music163.done==\"Y\",pysql.Music163.update_time > date).update({ \"done\": \"N\",\"update_time\":datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n self.session.commit()\n pylog.print_info(\"完成 重置时间 {} 之后的歌曲，可重新抓取评论\".format(date))\n\n" }, { "path": "spider163/spider/playlist.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport datetime\n\nfrom terminaltables import AsciiTable\n\nfrom spider163.utils import pysql\nfrom spider163.utils import pylog\nfrom spider163.utils import tools,const\n\nfrom spider163.spider import public as uapi\nfrom spider163 import settings\n\n\nclass Playlist:\n __play_url = None\n __headers = None\n\n def __init__(self):\n self.__headers = uapi.header\n self.__play_url = uapi.play_url\n self.session = settings.Session()\n\n def get_classify(self):\n table = [[\"类别\", \"风格列表\"]]\n for k, v in uapi.classify.items():\n c = 0\n lst = \"\"\n for v in v:\n c = c + 1\n if c % 5 == 0:\n lst = lst + v + \"\\n\"\n else:\n lst = lst + v + \",\"\n table.append([k, lst])\n print(AsciiTable(table).table)\n\n def view_capture(self, page, type=\"全部\"):\n play_url = self.__play_url.format(type, page * 35)\n titles = []\n try:\n acmsk = {'class': 'msk'}\n scnb = {'class': 'nb'}\n dcu = {'class': 'u-cover u-cover-1'}\n ucm = {'class': 'm-cvrlst f-cb'}\n data = tools.curl(play_url,self.__headers,type=const.RETURE_HTML)\n lst = data.find('ul', ucm)\n for play in lst.find_all('div', dcu):\n title = tools.encode(play.find('a', acmsk)['title'])\n link = tools.encode(play.find('a', acmsk)['href']).replace(\"/playlist?id=\", \"\")\n cnt = tools.encode(play.find('span', scnb).text).replace('万', '0000')\n if pysql.single(\"playlist163\",\"link\",link) is True:\n pl = pysql.Playlist163(title=title, link=link, cnt=int(cnt), dsc=\"曲风：{}\".format(type))\n self.session.add(pl)\n self.session.commit()\n titles.append(title)\n return titles\n except Exception as e:\n pylog.log.error(\"抓取歌单出现问题：{} 歌单类型：{} 页码：{}\".format(e, type, page))\n raise\n\n # date\n def create_update_strategy(self, **kwargs):\n date = (datetime.datetime.now() + datetime.timedelta(days=kwargs[\"date\"])).strftime(\"%Y-%m-%d %H:%S:%M\")\n self.session.query(pysql.Playlist163).filter(pysql.Playlist163.done == \"Y\",\n pysql.Playlist163.update_time > date).update(\n {\"done\": \"N\", \"update_time\": datetime.datetime.now().strftime(\"%Y-%m-%d %H:%S:%M\")})\n self.session.commit()\n pylog.print_info(\"完成 重置时间 {} 之后的歌单，可重新抓取歌曲\".format(date))\n" }, { "path": "spider163/spider/public.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nheader = {\n 'Referer': 'http://music.163.com/',\n 'Host': 'music.163.com',\n 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36',\n 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',\n }\ncomment_text = {\n 'username': '13393376853',\n 'password': 'wangyidafahao',\n 'rememberLogin': 'true'\n }\ncomment_module = '00e0b509f6259df8642dbc35662901477df22677ec152b5ff68ace615bb7b725152b3ab17a876aea8a5aa76d2e417629ec4ee341f56135fccf695280104e0312ecbda92557c93870114af6c9d05c4f7f0c3685b7a46bee255932575cce10b424d813cfe4875d3e82047b97ddef52741d546b8e289dc6935b3ece0462db0a22b8e7'\npubKey = '010001'\nsecKey = 16 * 'F'\ncomment_url = \"http://music.163.com/weapi/v1/resource/comments/R_SO_4_{}/?csrf_token=\"\nlyric_url = \"http://music.163.com/api/song/lyric?os=pc&id={}&lv=-1&kv=-1&tv=-1\"\nplay_url = \"http://music.163.com/discover/playlist/?order=hot&cat={}&limit=35&offset={}\"\nmusic_url = \"http://music.163.com/api/playlist/detail?id=\"\nmp3_url = \"http://music.163.com/weapi/song/enhance/player/url?csrf_token=\"\n\nplaylist_api = \"http://music.163.com/api/playlist/detail?id={}&upd\"\n\nmusic_api = \"http://music.163.com/api/song/detail/?id={}&ids=[{}]\"\n\nsearch_api = \"http://music.163.com/api/search/pc\"\n\nplaylist_add_api = \"http://music.163.com/weapi/playlist/manipulate/tracks?csrf_token={}\"\n\nlogin_api = \"http://music.163.com/weapi/login/cellphone\"\n\nclassify = {\n \"语种\":[\"华语\", \"欧美\", \"日语\",\"韩语\", \"粤语\", \"小语种\", ],\n \"风格\":[\"流行\", \"摇滚\", \"民谣\", \"电子\", \"舞曲\", \"说唱\", \"轻音乐\", \"爵士\", \"乡村\", \"R&B/Soul\", \"古典\", \"民族\", \"英伦\", \"金属\", \"朋克\", \"蓝调\", \"雷鬼\", \"世界音乐\", \"拉丁\", \"另类/独立\", \"New Age\", \"古风\", \"后摇\", \"Bossa Nova\"],\n \"场景\":[\"清晨\", \"夜晚\", \"学习\", \"工作\", \"午休\", \"下午茶\", \"地铁\", \"驾车\", \"运动\", \"旅行\", \"散步\", \"酒吧\"],\n \"情感\":[\"怀旧\", \"清新\", \"浪漫\", \"性感\", \"伤感\", \"治愈\", \"放松\", \"孤独\", \"感动\", \"兴奋\", \"快乐\", \"安静\", \"思念\"],\n \"主题\":[\"影视原声\", \"ACG\", \"校园\", \"游戏\", \"70后\", \"80后\", \"90后\", \"网络歌曲\", \"KTV\", \"经典\", \"翻唱\", \"吉他\", \"钢琴\", \"器乐\", \"儿童\", \"榜单\", \"00后\"]\n}\n\ntop = {\n 19723756: '云音乐飙升榜',\n 3779629: '云音乐新歌榜',\n 2884035: '网易原创歌曲榜',\n 3778678: '云音乐热歌榜',\n 1978921795: '云音乐电音榜',\n 991319590: '云音乐嘻哈榜',\n 71385702: '云音乐ACG音乐榜',\n 10520166: '云音乐新电力榜',\n}\n" }, { "path": "spider163/spider/read.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nfrom docx import Document\nfrom xlwt import Workbook\nfrom docx.enum.dml import MSO_THEME_COLOR_INDEX\nfrom sqlalchemy import desc\n\nfrom spider163 import settings\nfrom spider163.spider import public as uapi\nfrom spider163.utils import tools\nfrom spider163.utils import pylog\nfrom spider163.utils import pysql\nfrom spider163.spider import comment\n\n\ndef read_playlist_json(id):\n url = uapi.playlist_api.format(id)\n data = tools.curl(url,uapi.header)\n return data\n\n\ndef read_music_data(id):\n url = uapi.mp3_url.format(id)\n\n\ndef read_comment_data(id):\n cmt = comment.Comment()\n return cmt.post(id,1)\n\n\ndef read_lyric_data(id):\n url = uapi.lyric_url.format(id)\n data = tools.curl(url,uapi.header)\n return data\n\n\ndef print_pdf(id):\n data = read_playlist_json(id)\n if data[\"code\"] != 200:\n pylog.print_warn(\"歌单信息拉取失败！\")\n return\n\n document = Document()\n try:\n document.add_heading(data[\"result\"][\"name\"], 0)\n tags = document.add_paragraph(\" \".join(data[\"result\"][\"tags\"]))\n desc = document.add_paragraph(data[\"result\"][\"description\"])\n for m in data[\"result\"][\"tracks\"]:\n document.add_paragraph().add_run(m[\"name\"]).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_2\n\n lyric = read_lyric_data(m[\"id\"])\n document.add_paragraph().add_run(lyric[\"lrc\"][\"lyric\"]).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_3\n\n comments = read_comment_data(m[\"id\"])\n for c in comments[\"hotComments\"]:\n author = document.add_paragraph().add_run(c[\"user\"][\"nickname\"]).style = 'Emphasis'\n content = document.add_paragraph(c[\"content\"])\n except Exception as e:\n pylog.print_warn(e)\n\n document.save(\"{}.docx\".format(data[\"result\"][\"name\"]))\n pylog.print_info(\"文档 {}.docx 已经生成！\".format(data[\"result\"][\"name\"]))\n\n\ndef print_comment(count):\n session = settings.Session()\n comments = session.query(pysql.Comment163).order_by(\n desc(pysql.Comment163.liked)).limit(count)\n document = Document()\n workbook = Workbook()\n try:\n document.add_heading(\"TOP {} 评论\".format(count), 0)\n sheet = workbook.add_sheet(\"TOP {} 评论\".format(count))\n i = 0\n sheet.write(i, 0, \"歌曲名字\")\n sheet.write(i, 1, \"评论作者\")\n sheet.write(i, 2, \"评论内容\")\n sheet.write(i, 3, \"点赞数量\")\n sheet.write(i, 4, \"歌曲链接\")\n for c in comments:\n i = i + 1\n song = session.query(pysql.Music163).filter(pysql.Music163.song_id == c.song_id)\n pylog.print_info(\"正在填充第 {} 条评论,歌曲：{}\".format(i, song[0].song_name))\n document.add_paragraph().add_run(\n \"作者：{}\".format(c.author)).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_2\n document.add_paragraph().add_run(\n \"内容：{}\".format(c.txt)).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_2\n document.add_paragraph().add_run(\n \"歌曲：《{}》 链接：http://music.163.com/#/song?id={}\".format(song[0].song_name, c.song_id)).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_2\n document.add_paragraph().add_run(\n \"赞同：{}\".format(c.liked)).font.color.theme_color = MSO_THEME_COLOR_INDEX.ACCENT_2\n document.add_paragraph(\"\")\n\n sheet.write(i, 0, song[0].song_name)\n sheet.write(i, 1, c.author)\n sheet.write(i, 2, c.txt)\n sheet.write(i, 3, c.liked)\n sheet.write(i, 4, \"http://music.163.com/#/song?id={}\".format(c.song_id))\n\n except Exception as e:\n pylog.print_warn(e)\n document.save(\"TOP {} 评论.docx\".format(count))\n pylog.print_warn(\"\\n完成文档 TOP {} 评论.docx 的生成！\\n\".format(count))\n\n workbook.save(\"TOP {} 评论.xls\".format(count))\n pylog.print_warn(\"\\n完成文档 TOP {} 评论.xls 的生成！\\n\".format(count))" }, { "path": "spider163/spider/search.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport requests\nfrom terminaltables import AsciiTable\n\nfrom spider163.utils import pylog\nfrom spider163.utils import tools\nfrom spider163.spider import public as uapi\n\noffset = 0\nlimit = 20\ntype = {1: \"歌曲\", 10: \"专辑\", 100: \"歌手\", 1000: \"歌单\"}\n\n\ndef searchSong(key):\n url = uapi.search_api\n data = {'s': key, 'offset': 0, 'limit': 20, 'type': \"1\"}\n req = requests.post(url, headers=uapi.header, data=data, timeout=10)\n if req.json()[\"result\"]['songCount'] == 0:\n pylog.log.warn(\"关键词 {} 没有可搜索歌曲\".format(key))\n return\n songs = req.json()[\"result\"]['songs']\n song_table = AsciiTable([[\"ID\", \"歌曲\", \"专辑\", \"演唱\"]])\n for item in songs:\n id = item['id']\n name = tools.encode(item['name'])\n album = tools.encode(item['album']['name'])\n artist = []\n for a in item['artists']:\n artist.append(tools.encode(a['name']))\n song_table.table_data.append([str(id), name, album, \",\".join(artist)])\n print(pylog.Blue(\"与 \\\"{}\\\" 有关的歌曲\".format(key)))\n print(song_table.table)\n\n\ndef searchAlbum(key):\n url = uapi.search_api\n data = {'s': key, 'offset': 0, 'limit': 20, 'type': \"10\"}\n req = requests.post(url, headers=uapi.header, data=data, timeout=10)\n if req.json()[\"result\"]['albumCount'] == 0:\n pylog.log.warn(\"关键词 {} 没有可搜索专辑\".format(key))\n return\n albums = req.json()[\"result\"]['albums']\n song_table = AsciiTable([[\"ID\", \"专辑\", \"演唱\",\"发行方\"]])\n for item in albums:\n id = item['id']\n name = tools.encode(item['name'])\n company = \"\"\n if item['company'] != None:\n company = tools.encode(item['company'])\n artist = []\n for a in item['artists']:\n artist.append(tools.encode(a['name']))\n song_table.table_data.append([str(id), name, \",\".join(artist), company])\n print(pylog.Blue(\"与 \\\"{}\\\" 有关的专辑\".format(key)))\n print(song_table.table)\n\n\ndef searchSinger(key):\n url = uapi.search_api\n data = {'s': key, 'offset': 0, 'limit': 10, 'type': \"100\"}\n req = requests.post(url, headers=uapi.header, data=data, timeout=10)\n if req.json()[\"result\"]['artistCount'] == 0:\n pylog.log.warn(\"关键词 {} 没有可搜索艺术家\".format(key))\n return\n artists = req.json()[\"result\"]['artists']\n song_table = AsciiTable([[\"ID\", \"姓名\", \"专辑数量\", \"MV数量\"]])\n for item in artists:\n id = str(item['id'])\n name = tools.encode(item['name'])\n acount = str(item['albumSize'])\n mcount = str(item['mvSize'])\n song_table.table_data.append([id, name, acount, mcount])\n print(pylog.Blue(\"与 \\\"{}\\\" 有关的歌手\".format(key)))\n print(song_table.table)\n\n\ndef searchPlaylist(key):\n url = uapi.search_api\n data = {'s': key, 'offset': 0, 'limit': 5, 'type': \"1000\"}\n req = requests.post(url, headers=uapi.header, data=data, timeout=10)\n if req.json()[\"result\"]['playlistCount'] == 0:\n pylog.log.warn(\"关键词 {} 没有可搜索歌单\".format(key))\n return\n playlists = req.json()[\"result\"]['playlists']\n song_table = AsciiTable([[\"ID\", \"歌单\", \"维护者\", \"播放数量\", \"收藏数量\"]])\n for item in playlists:\n id = str(item['id'])\n name = tools.encode(item['name'])\n creator = tools.encode(item['creator']['nickname'])\n pcount = str(item['playCount'])\n bcount = str(item['bookCount'])\n song_table.table_data.append([id, name, creator, pcount, bcount])\n print(pylog.Blue(\"与 \\\"{}\\\" 有关的歌单\".format(key)))\n print(song_table.table)" }, { "path": "spider163/template/spider163.conf", "content": "# 请把本配置文件名字中删掉.default保存到工作目录中，默认为~/spider163/\n# 请修改相关配置为本地可用配置\n[core]\ndb=mysql://root:password@127.0.0.1/database?charset=utf8mb4\nport=1630\n\n[mail]\nhost=localhost\nport=25\nusers=792400644@qq.com,\n" }, { "path": "spider163/utils/__init__.py", "content": "# --* coding: utf-8 -*-" }, { "path": "spider163/utils/config.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport os\nimport re\n\nfrom spider163 import version\nif version.PYTHON3 is True:\n import configparser as ConfigParser\nelse:\n import ConfigParser\n\n\n\nPATH = os.environ.get(\"HOME\") + \"/spider163\"\nif os.environ.get(\"SPIDER163_PATH\") is not None:\n PATH = os.environ.get(\"SPIDER163_PATH\")\n\nif not os.path.exists(PATH):\n os.makedirs(PATH)\ncf = ConfigParser.ConfigParser()\nif not os.path.exists(PATH + \"/spider163.conf\"):\n print(\"请在默认路径 \" + PATH + \" 下增加配置文件 spider163.conf 格式参照官方\")\n cf.read(\"{}/template/spider163.conf\".format(version.root_path))\nelse:\n cf.read(PATH + \"/spider163.conf\")\n\n\ndef get_path():\n return PATH\n\n\ndef get_db():\n try:\n return cf.get(\"core\", \"db\")\n except Exception as e:\n print(\"配置文件存在问题，请在 {}/spider163.conf 中配置db=xxx选项\".format(PATH))\n print(\"错误详情： {}\".format(e))\n raise e\n\n\ndef get_mail():\n try:\n return cf.get(\"mail\", \"host\"),cf.get(\"mail\", \"port\"),cf.get(\"mail\", \"users\"),\n except Exception as e:\n print(\"配置文件存在问题，请在 {}/spider163.conf 中配置mail选项\".format(PATH))\n print(\"错误详情： {}\".format(e))\n raise e\n\ndef format_db():\n \"\"\"db=mysql://root:password@127.0.0.1/spider?charset=utf8mb4\"\"\"\n link = get_db()\n r = re.search(\"mysql:\\/\\/([^:]+):([^@]+)@((?:[0-9]{1,3}\\.){3}[0-9]{1,3})/([^\\?]+)\\?charset=utf8mb4\", link)\n if r is None:\n return r\n else:\n return {\n \"link\": r.group(0),\n \"user\": r.group(1),\n \"password\": r.group(2),\n \"ip\": r.group(3),\n \"database\": r.group(4)\n }\n\n\ndef get_mysql():\n link = get_db()\n db = re.search('(?<=/)[^/]+(?=\\?)', link).group(0)\n uri = re.search('.*(?=/)', link).group(0)\n return {\"db\": db, \"uri\": uri}\n\n\ndef get_port():\n try:\n return int(cf.get(\"core\", \"port\"))\n except Exception as e:\n print(\"配置文件存在问题，请在 {}/spider163.conf 中配置port=xxx选项\".format(PATH))\n print(\"错误详情： {}\".format(e))\n raise e\n\n\n\n\n" }, { "path": "spider163/utils/const.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nRETURN_JSON = \"return json data\"\nRETURE_HTML = \"return html data\"" }, { "path": "spider163/utils/encrypt.py", "content": "# coding=utf-8\n\nimport base64\n\nfrom cryptography.hazmat.primitives.ciphers import (\n Cipher, algorithms, modes\n)\nfrom cryptography.hazmat.backends import default_backend\n\n\ndef aes(text, sec_key):\n backend = default_backend()\n pad = 16 - len(text) % 16\n text = text + pad * chr(pad)\n cipher = Cipher(\n algorithms.AES(sec_key.encode('utf-8')),\n modes.CBC(b'0102030405060708'),\n backend=backend\n )\n encryptor = cipher.encryptor()\n ciphertext = encryptor.update(text.encode('utf-8')) + encryptor.finalize()\n ciphertext = base64.b64encode(ciphertext)\n return ciphertext\n" }, { "path": "spider163/utils/healthz.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport os\nimport json\nfrom sqlalchemy import func\n\nfrom spider163.utils import config\nfrom spider163.utils import pylog\nfrom spider163.utils import pysql\nfrom spider163 import settings\n\n\ndef is_correct_config():\n pylog.print_info(\"正在检查配置路径：{}\".format(config.PATH))\n if not os.path.exists(config.PATH + \"/spider163.conf\"):\n print(\" - 配置路径下 spider163.conf {}\".format(pylog.red(\"不存在\")))\n else:\n print(\" - 配置路径下 spider163.conf {}\".format(pylog.green(\"存在\")))\n pylog.print_info(\"正在检查配置文件 {}/spider163.conf 内容是否完整\".format(config.PATH))\n try:\n config.cf.get(\"core\", \"db\")\n print(\" - 配置文件中 db 选项 {}\".format(pylog.green(\"存在\")))\n except Exception:\n print(\" - 配置文件中 db 选项 {}\".format(pylog.red(\"不存在\")))\n try:\n config.cf.get(\"core\", \"port\")\n print(\" - 配置文件中 port 选项 {}\".format(pylog.green(\"存在\")))\n except Exception:\n print(\" - 配置文件中 port 选项 {}\".format(pylog.red(\"不存在\")))\n\n\ndef is_correct_db():\n db = config.format_db()\n pylog.print_info(\"正在检查配置的数据库格式和可用性\")\n if db is None:\n print(\" - 配置文件中 db 选项 {}\".format(pylog.red(\"不正确\")))\n else:\n print(\" - 账号：{} 密码：{} IP：{} 数据库：{}\".format(db[\"user\"], db[\"password\"], db[\"ip\"], db[\"database\"]))\n try:\n from sqlalchemy import create_engine\n create_engine(db[\"link\"], echo=False).execute(\"show databases\")\n print(\"数据库连接验证 {}\".format(pylog.green(\"成功\")))\n except Exception as e:\n pylog.print_err(\"数据库连接失败，上述配置信息有问题: {}\".format(e))\n\n\ndef can_spider():\n print(\"抓取验证未完成\")\n\n\ndef expose_data():\n playlist = settings.Session.query(pysql.Playlist163).count()\n playlist_type = settings.Session.query(pysql.Playlist163, func.count(pysql.Playlist163.id)).group_by(pysql.Playlist163.id).all()\n music = settings.Session.query(pysql.Music163).count()\n comment = settings.Session.query(pysql.Comment163).count()\n lyric = settings.Session.query(pysql.Lyric163).count()\n top = settings.Session.query(pysql.Toplist163).count()\n data = {'playlist': {'count': playlist},\n 'music': {'count': music,},\n 'comment': {'count': comment},\n 'lyric': {'count': lyric},\n 'top':{'count': top}\n }\n js = json.dumps(data, ensure_ascii=False, indent=2)\n print(js)" }, { "path": "spider163/utils/mail.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport smtplib\nimport datetime\n\n\ndef send_email(host,port, subject, user, content):\n\n illegal = [\"\\n\", \"\\r\"]\n for ill in illegal:\n subject = subject.replace(ill, ' ')\n\n headers = {\n 'Content-Type': 'text/html; charset=utf-8',\n 'Content-Disposition': 'inline',\n 'Content-Transfer-Encoding': '8bit',\n 'Subject': subject,\n 'From': \"chengyumeng@github.com\",\n 'To': user,\n 'Date': datetime.datetime.now().strftime('%a, %d %b %Y %H:%M:%S %Z'),\n 'X-Mailer': 'ChengYumeng',\n }\n\n msg = ''\n for key, value in headers.items():\n msg += \"%s: %s\\n\" % (key, value)\n\n # add contents\n msg += \"\\n%s\\n\" % content\n\n s = smtplib.SMTP(host, port)\n\n print (\"sending %s to %s\" % (subject,headers['To']))\n s.sendmail( headers['From'], headers['To'], msg.encode(\"utf8\"))\n s.quit()" }, { "path": "spider163/utils/pylog.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport logging\nfrom spider163.utils import config\nfrom logbook import FileHandler, Logger\nfrom terminaltables import AsciiTable\nfrom colorama import Fore\nfrom colorama import init\n\npath = config.get_path()\nlog_handler = FileHandler(filename=path + '/spider163.log')\nlog_handler.push_application()\nlog = Logger(\"\")\n\ninit(autoreset=True)\n\n\ndef Log(msg):\n print_warn(msg)\n log.warn(msg)\n\n\ndef Table(tb):\n print(AsciiTable(tb).table)\n\n\ndef Blue(msg):\n return Fore.BLUE + msg\n\n\ndef green(msg):\n return Fore.GREEN + msg\n\n\ndef red(msg):\n return Fore.RED + msg\n\n\ndef print_err(msg):\n print(Fore.RED + msg)\n\n\ndef print_warn(msg):\n print(Fore.YELLOW + msg)\n\n\ndef print_info(msg):\n print(Fore.BLUE + msg)\n" }, { "path": "spider163/utils/pysql.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nimport random\n\nfrom sqlalchemy import Column, Integer, String, TIMESTAMP, Index\nfrom sqlalchemy.ext.declarative import declarative_base\nfrom sqlalchemy.sql import func\nfrom sqlalchemy.dialects.mysql import MEDIUMTEXT\n\nfrom spider163 import settings\nfrom spider163.utils import pylog\n\nBase = declarative_base()\n\n\nclass Playlist163(Base):\n __tablename__ = \"playlist163\"\n\n id = Column(Integer(), primary_key=True, autoincrement=True) # 歌曲ID\n title = Column(String(5000), server_default=\"System Title\") # 歌单名字\n link = Column(String(255), server_default=\"No Link\") # 歌曲链接\n cnt = Column(Integer(), server_default=\"-1\") # 歌曲数量\n playCount = Column(Integer(), server_default=\"-1\") # 播放次数\n shareCount = Column(Integer(), server_default=\"-1\") # 分享次数\n commentCount = Column(Integer(), server_default=\"-1\") # 评论数量\n description = Column(MEDIUMTEXT)\n tags = Column(String(255), server_default=\"\")\n dsc = Column(String(255), server_default=\"No Description\")\n create_time = Column(TIMESTAMP, server_default=func.now())\n update_time = Column(TIMESTAMP, server_default=func.now())\n done = Column(String(255), server_default=\"N\")\n done_link = Index(\"done_link\", done, link)\n\n\nclass Music163(Base):\n __tablename__ = \"music163\"\n id = Column(Integer(), primary_key=True, autoincrement=True)\n song_id = Column(Integer())\n song_name = Column(String(5000), server_default=\"No Name\")\n author = Column(String(5000), server_default=\"No Author\")\n playTime = Column(Integer(), server_default=\"-1\") # 歌曲播放次数\n done = Column(String(255), server_default=\"N\")\n has_lyric = Column(String(255), server_default=\"N\")\n create_time = Column(TIMESTAMP, server_default=func.now())\n update_time = Column(TIMESTAMP, server_default=func.now())\n comment = Column(Integer(), server_default=\"-1\")\n done_id = Index(\"done_id\", done,id)\n song_id_comment = Index(\"song_id_comment\", song_id, comment)\n\n\nclass Toplist163(Base):\n __tablename__ = \"top163\"\n id = Column(Integer(), primary_key=True, autoincrement=True)\n song_id = Column(Integer())\n song_name = Column(String(5000), server_default=\"No Name\")\n author = Column(String(5000), server_default=\"No Author\")\n playTime = Column(Integer(), server_default=\"-1\") # 歌曲播放次数\n done = Column(String(255), server_default=\"N\")\n mailed = Column(String(255), server_default=\"N\")\n has_lyric = Column(String(255), server_default=\"N\")\n create_time = Column(TIMESTAMP, server_default=func.now())\n update_time = Column(TIMESTAMP, server_default=func.now())\n comment = Column(Integer(), server_default=\"-1\")\n lastRank = Column(Integer(), server_default=\"100000000\") # 上次排名字段\n playlist_id = Column(Integer(), server_default=\"-1\") # 排行榜歌单ID\n position = Column(Integer(), server_default=\"0\")\n done_id = Index(\"done_id\", done, id)\n song_id_comment = Index(\"song_id_comment\", song_id, comment)\n\n\nclass Comment163(Base):\n __tablename__ = \"comment163\"\n id = Column(Integer(), primary_key=True, autoincrement=True)\n song_id = Column(Integer())\n txt = Column(MEDIUMTEXT)\n author = Column(String(5000), server_default=\"No Author\")\n liked = Column(Integer(), server_default=\"0\")\n create_time = Column(TIMESTAMP, server_default=func.now())\n Index(\"liked_song_id\", liked, song_id)\n Index(\"song_id_liked\", song_id, liked)\n\n\nclass Lyric163(Base):\n __tablename__ = \"lyric163\"\n id = Column(Integer(), primary_key=True, autoincrement=True)\n song_id = Column(Integer())\n txt = Column(MEDIUMTEXT)\n create_time = Column(TIMESTAMP, server_default=func.now())\n key_song_id = Index(\"song_id\", song_id)\n\n\ndef single(table, k, v):\n cnt = settings.engine.execute('select count(*) from ' + table + ' where ' + k + '=\\'' + str(v) + '\\'').fetchone()\n if cnt[0] == 0:\n return True\n else:\n return False\n\n\ndef stat_playlist():\n data = {}\n data[\"gdType\"] = settings.Session.query(func.substring(Playlist163.dsc, 4, 2).label('type'), func.count('*').label('count')).group_by(\"type\").all()\n data[\"gdOver\"] = settings.Session.query(Playlist163.done.label('over'), func.count('*').label('count')).group_by(\"over\").all()\n return data\n\n\ndef stat_music():\n data = {\"author-comment-count\": []}\n cd = settings.Session.query(Music163.author.label('author'), func.sum(Music163.comment).label('count')).group_by(\"author\").order_by(func.sum(Music163.comment).label('count').label('count').desc()).limit(30).all()\n for m in cd:\n data[\"author-comment-count\"].append([m[0], int(m[1])])\n data[\"music-comment-count\"] = settings.Session.query(Music163.song_name, Music163.comment.label(\"count\")).order_by(Music163.comment.label(\"count\").desc()).limit(30).all()\n return data\n\n\ndef stat_data():\n data = {}\n data[\"countPlaylist\"] = int(settings.engine.execute(\"select(select count(*) from playlist163 where done = 'Y')*100 / count(*) from playlist163\").fetchone()[0]);\n data[\"countComment\"] = int(settings.engine.execute(\"select(select count(*) from music163 where done = 'Y')*100 / count(*) from music163\").fetchone()[0]);\n data[\"countLyric\"] = int(settings.engine.execute(\"select(select count(*) from music163 where has_lyric = 'Y')*100 / count(*) from music163\").fetchone()[0]);\n return data\n\n\ndef random_data():\n rng = settings.Session.query(func.min(Comment163.id), func.max(Comment163.id)).all()[0]\n data = []\n for i in range(12):\n v = random.uniform(rng[0], rng[1])\n d = settings.engine.execute(\"select txt,liked,a.author,song_name,a.song_id,b.author from comment163 a inner join music163 b on a.song_id= b.song_id where a.id>\" +str(v) + \" limit 1\").fetchone()\n data.append({\"txt\": d[0],\"like\": d[1] ,\"author\": d[2], \"song\" :{\"name\":d[3], \"author\": d[5], \"id\": d[4]}})\n return data\n\n\ndef initdb():\n try:\n Base.metadata.create_all(settings.engine)\n except Exception as e:\n pylog.print_warn(\"自动生成数据库表出现问题: {}\".format(e))\n\n\ndef dropdb():\n try:\n Base.metadata.drop_all(settings.engine)\n except Exception as e:\n pylog.print_warn(\"自动删除数据库表出现问题: {}\".format(e))\n\n" }, { "path": "spider163/utils/tools.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\nimport contextlib\nimport codecs\nimport requests\nimport json\nimport hashlib\nfrom bs4 import BeautifulSoup\n\nfrom spider163 import version\nfrom spider163.utils import const\n\n\n@contextlib.contextmanager\ndef ignored(*exceptions):\n try:\n yield\n except exceptions:\n pass\n\n\ndef encode(s):\n if version.PYTHON3 is True:\n return codecs.encode(s,\"utf-8\").decode(\"utf-8\")\n else:\n return s.encode(\"utf-8\")\n\n\ndef hex(s):\n if version.PYTHON3 is True:\n return codecs.encode(bytes(s, encoding = \"utf8\"), 'hex')\n else:\n return s.encode(\"hex\")\n\n\ndef md5(s):\n m = hashlib.md5()\n m.update(s.encode(\"utf-8\"))\n return m.hexdigest()\n\n\ndef curl(url, headers, type = const.RETURN_JSON):\n try:\n s = requests.session()\n bs = BeautifulSoup(s.get(url, headers=headers).content, \"html.parser\")\n if type == const.RETURN_JSON:\n return json.loads(bs.text)\n elif type == const.RETURE_HTML:\n return bs\n else:\n return bs.text\n except Exception:\n raise\n\n" }, { "path": "spider163/version.py", "content": "# -*- coding: utf-8 -*-\nimport os\nimport sys\n\nPYTHON3 = False\nif sys.version > \"3\":\n PYTHON3 = True\n\nVERSION = \"2.7.8\"\nDESCRIPTION = \"\"\"\nSpider163的数据基础，来源于网易公司的网易云音乐产品。其授权协议，包含《网易云音乐服务条款》但不包含其霸王条款。\n\n该项目遵循MIT开源协议。我们认为知识属于全人类，网易云音乐的评论区属于用户，不属于网易云音乐。\n\n而广大网民有权利根据自己的喜好阅读、整理、分析和总结开放的、非私密信息。\n\n\n您可以选择四种方式支持本项目的开发：\n\nNo.1 在Github上star本项目，或者在其它任何场合宣传本项目。\n\n附：Spider163 GitHub 地址 https://github.com/Chengyumeng/spider163\n\nNo.2 关注本项目作者的唯一个人微信公众账号。\n\n公众号名字：程天写代码\n\nNo.3 通过支付宝向作者转账赞助。\n\n支付宝二维码：https://github.com/Chengyumeng/spider163/blob/master/spider163/www/static/img/zhifubao.jpeg\n\nNo.4 通过微信向作者转账赞助。\n\n微信二维码：https://github.com/Chengyumeng/spider163/blob/master/spider163/www/static/img/weixin.jpeg\n\n\"\"\"\nroot_path = os.path.dirname(os.path.abspath(__file__))\n\nMAILBODY = \"\"\"\n

Spider163 云音乐今日精彩推荐(微信公众号：pod1024)

\n

\n

今日分享 {} 摘编歌曲：

\n\n
\n

欢迎关注程天写代码微信公众号：pod1024

\n
\n\"\"\"\n\nMAILMUSIC = \"\"\"\n
  • {}\n{} \n评论数：{}
    • \n{}\n\"\"\"\n\nMAILCOMMENT = \"\"\"\n

    {}\n{} :

    \n

    {}

    \n\"\"\"\n" }, { "path": "spider163/www/__init__.py", "content": "" }, { "path": "spider163/www/static/css/spider163.css", "content": ".brand-intro {\n text-align: center;\n background-color: #a811b5;\n color: #fff;\n padding: 120px 0;\n background-position: center center;\n background-repeat: no-repeat;\n}\n\n.sui-btn {\n display: inline-block;\n padding: 2px 14px;\n box-sizing: border-box;\n margin-bottom: 0;\n font-size: 12px;\n line-height: 18px;\n text-align: center;\n vertical-align: middle;\n cursor: pointer;\n color: #333333;\n background-color: #eeeeee;\n filter: progid:DXImageTransform.Microsoft.gradient(enabled = false);\n border: 1px solid #e1e1e1;\n -webkit-border-radius: 2px;\n -moz-border-radius: 2px;\n border-radius: 2px;\n -webkit-user-select: none;\n -moz-user-select: none;\n -ms-user-select: none;\n -o-user-select: none;\n user-select: none;\n}\n\n.btn-lead {\n padding: 1em 4em;\n border: 2px solid #fff;\n opacity: .8;\n background: transparent;\n color: #fff;\n font-size: 24px;\n height: auto;\n}\n\n.footer {\n border-top: 1px solid #aaa;\n padding: 30px;\n text-align: center;\n color: #666;\n font-size: 13px;\n margin-top: 50px;\n}\n\n\n\n.index-body {\n margin-top: 22px;\n margin-bottom: 22px;\n}\n\n#index-group .i-c-p1 {\n background: #A020F0 no-repeat 35px -47px;\n}\n\n#index-group .i-c-p2 {\n background: #FF1493 no-repeat 35px -47px;\n}\n\n#index-group .i-c-p3 {\n background: #e8b875 no-repeat 35px -47px;\n}\n\n#index-group .i-c-p4 {\n background: #baa1e2 no-repeat 25px -260px;\n}\n\n\n#index-group .i-c-p {\n height: 42px;\n position: relative;\n text-align: center;\n overflow: hidden;\n}\n\n#index-group .i-c-ph, #index-group .i-c-ph1 {\n color: #fff;\n font-size: 1.3em;\n margin: 0;\n text-align: center;\n position: absolute;\n z-index: 10;\n font-weight: 400;\n left: 0;\n width: 100%;\n}\n\n\n.i-c-ph, .i-c-ph1 {\n color: #fff;\n font-size: 1.5em;\n text-align: center;\n position: absolute;\n z-index: 10;\n bottom: .7em;\n font-weight: 400;\n left: 0;\n width: 100%;\n}\n\n.i-c-list li {\n height: 2.5em;\n line-height: 2.5em;\n overflow: hidden;\n border-bottom: 1px dashed #eaeaea;\n list-style: none;\n padding-left: 10px;\n font-size: 1.1em;\n}\n\n.modals {\nmargin-top:200px;\n}\n\n.gdspider, .gqspider, .gcspider, .rpspider {\nmargin:20px auto auto auto;\nwidth:50%;\nmin-width:400px;\n}" }, { "path": "spider163/www/static/js/macarons.js", "content": "(function (root, factory) {\n if (typeof define === 'function' && define.amd) {\n // AMD. Register as an anonymous module.\n define(['exports', 'echarts'], factory);\n } else if (typeof exports === 'object' && typeof exports.nodeName !== 'string') {\n // CommonJS\n factory(exports, require('echarts'));\n } else {\n // Browser globals\n factory({}, root.echarts);\n }\n}(this, function (exports, echarts) {\n var log = function (msg) {\n if (typeof console !== 'undefined') {\n console && console.error && console.error(msg);\n }\n };\n if (!echarts) {\n log('ECharts is not Loaded');\n return;\n }\n\n var colorPalette = [\n '#2ec7c9','#b6a2de','#5ab1ef','#ffb980','#d87a80',\n '#8d98b3','#e5cf0d','#97b552','#95706d','#dc69aa',\n '#07a2a4','#9a7fd1','#588dd5','#f5994e','#c05050',\n '#59678c','#c9ab00','#7eb00a','#6f5553','#c14089'\n ];\n\n\n var theme = {\n color: colorPalette,\n\n title: {\n textStyle: {\n fontWeight: 'normal',\n color: '#008acd'\n }\n },\n\n visualMap: {\n itemWidth: 15,\n color: ['#5ab1ef','#e0ffff']\n },\n\n toolbox: {\n iconStyle: {\n normal: {\n borderColor: colorPalette[0]\n }\n }\n },\n\n tooltip: {\n backgroundColor: 'rgba(50,50,50,0.5)',\n axisPointer : {\n type : 'line',\n lineStyle : {\n color: '#008acd'\n },\n crossStyle: {\n color: '#008acd'\n },\n shadowStyle : {\n color: 'rgba(200,200,200,0.2)'\n }\n }\n },\n\n dataZoom: {\n dataBackgroundColor: '#efefff',\n fillerColor: 'rgba(182,162,222,0.2)',\n handleColor: '#008acd'\n },\n\n grid: {\n borderColor: '#eee'\n },\n\n categoryAxis: {\n axisLine: {\n lineStyle: {\n color: '#008acd'\n }\n },\n splitLine: {\n lineStyle: {\n color: ['#eee']\n }\n }\n },\n\n valueAxis: {\n axisLine: {\n lineStyle: {\n color: '#008acd'\n }\n },\n splitArea : {\n show : true,\n areaStyle : {\n color: ['rgba(250,250,250,0.1)','rgba(200,200,200,0.1)']\n }\n },\n splitLine: {\n lineStyle: {\n color: ['#eee']\n }\n }\n },\n\n timeline : {\n lineStyle : {\n color : '#008acd'\n },\n controlStyle : {\n normal : { color : '#008acd'},\n emphasis : { color : '#008acd'}\n },\n symbol : 'emptyCircle',\n symbolSize : 3\n },\n\n line: {\n smooth : true,\n symbol: 'emptyCircle',\n symbolSize: 3\n },\n\n candlestick: {\n itemStyle: {\n normal: {\n color: '#d87a80',\n color0: '#2ec7c9',\n lineStyle: {\n color: '#d87a80',\n color0: '#2ec7c9'\n }\n }\n }\n },\n\n scatter: {\n symbol: 'circle',\n symbolSize: 4\n },\n\n map: {\n label: {\n normal: {\n textStyle: {\n color: '#d87a80'\n }\n }\n },\n itemStyle: {\n normal: {\n borderColor: '#eee',\n areaColor: '#ddd'\n },\n emphasis: {\n areaColor: '#fe994e'\n }\n }\n },\n\n graph: {\n color: colorPalette\n },\n\n gauge : {\n axisLine: {\n lineStyle: {\n color: [[0.2, '#2ec7c9'],[0.8, '#5ab1ef'],[1, '#d87a80']],\n width: 10\n }\n },\n axisTick: {\n splitNumber: 10,\n length :15,\n lineStyle: {\n color: 'auto'\n }\n },\n splitLine: {\n length :22,\n lineStyle: {\n color: 'auto'\n }\n },\n pointer : {\n width : 5\n }\n }\n };\n\n echarts.registerTheme('macarons', theme);\n}));" }, { "path": "spider163/www/static/js/scan.js", "content": "$(function () {\n this.createDom = function(){\n }\n this.documentEvent = function(){\n\n }\n this.init = function() {\n commentlist()\n }\n this.init()\n});\nfunction commentlist() {\n $.ajax({\n url : \"/scan/data\",\n type:\"get\",\n dataType : \"json\",\n success : function (data) {\n code = \"\";\n for ( i in data['comment']) {\n c = data['comment'][i];\n code = code + \"
    \"\n + \"

    \"\n + c[\"song\"][\"name\"] + \" - \" + c[\"song\"][\"author\"] + \"

    \"\n + \"

    \" + c[\"txt\"] + \"

    \"\n + \"
    \" + c[\"author\"] + \" \" + c[\"like\"] + \"\" + \"
    \"\n }\n $(\"#index-group\").html(code);\n\n }\n });\n}\n" }, { "path": "spider163/www/static/js/spider163.js", "content": "$(function () {\n this.createDom = function () {\n this.spiderPlaylistObj = $(\"#spiderPlaylist\");\n this.SpiderMusicObj = $(\"#spiderMusic\");\n this.SpiderLyricObj = $(\"#spiderLyric\");\n this.SpiderCommentObj = $(\"#spiderComment\");\n }\n this.documentEvent = function () {\n\t\tvar self = this;\n\t\t this.hideBox = function() {\n\t\t $(\".gdspider\").css({\"display\":\"none\"});\n\t\t $(\".gqspider\").css({\"display\":\"none\"});\n\t\t $(\".gcspider\").css({\"display\":\"none\"});\n\t\t $(\".rpspider\").css({\"display\":\"none\"});\n\t\t}\n\t\t$(\"#gd\").click(function(){\n\t\t self.hideBox();\n\t\t $(\".gdspider\").css({\"display\":\"block\"});\n\t\t});\n\t\t$(\"#gq\").click(function(){\n\t\t self.hideBox();\n\t\t $(\".gqspider\").css({\"display\":\"block\"});\n\t\t});\n\t\t$(\"#gc\").click(function(){\n\t\t self.hideBox();\n\t\t $(\".gcspider\").css({\"display\":\"block\"});\n\t\t});\n\n\t\t$(\"#rp\").click(function(){\n\t\t self.hideBox();\n\t\t $(\".rpspider\").css({\"display\":\"block\"});\n\t\t});\n\n\t\tthis.spiderPlaylistObj.click(function() {\n\t\t var gdType = $(\"#gdType\").val();\n\t\t var gdPage = $(\"#gdPage\").val();\n\t\t $.ajax({\n url : \"/spider/getPlaylist\",\n data:\"gdType=\"+gdType+\"&gdPage=\"+gdPage,\n type:\"post\",\n dataType : \"json\",\n success : function (data) {\n var thead = \" #歌单名字\";\n var tbody = \"\";\n for (t in data['title']) {\n tbody = tbody + \"\"+ t +\"\"+ data['title'][t] +\"\";\n }\n $(\"#printTable\").html(thead + \"\" + tbody + \"\");\n },\n });\n\t\t});\n\n\t\tthis.SpiderMusicObj.click(function() {\n\t\t var gdSource = $(\"#gdSource\").val();\n\t\t var gdCount = $(\"#gdCount\").val();\n\t\t for (i=0;i< gdCount; i++ ){\n\t\t $.ajax({\n url : \"/spider/getMusic\",\n data:\"gdSource=\"+gdSource,\n type:\"post\",\n dataType : \"json\",\n success : function (data) {\n var thead = \" #歌单名字歌曲名字作者\";\n var tbody = \"\";\n for (playlist in data['data']) {\n for ( m in data['data'][playlist]) {\n tbody = tbody + \"\"\n + \"\"+ playlist +\"\"\n +\"\"+ data['data'][playlist][m][\"name\"] + \"\"\n + \"\"+ data['data'][playlist][m][\"author\"] + \"\"\n +\"\";\n }\n }\n if((data['data'][playlist]).length > 0) {\n $(\"#printTable\").html(thead + \"\" + tbody + \"\");\n }\n },\n });\n\t\t }\n\t\t});\n\n\t\tthis.SpiderLyricObj.click(function() {\n\t\t var gqCount = $(\"#gqCount\").val();\n\t\t $.ajax({\n url : \"/spider/getLyric\",\n data:\"gqCount=\"+gqCount,\n type:\"post\",\n dataType : \"json\",\n success : function (data) {\n var thead = \" #歌曲名字作者评论数量\";\n var tbody = \"\";\n for (cnt in data['data']) {\n tbody = tbody + \"\" + cnt\n + \"\"\n +\"\"+ data['data'][cnt][\"name\"] + \"\"\n + \"\"+ data['data'][cnt][\"author\"] + \"\"\n +\"\"+ data['data'][cnt][\"comment\"] + \"\"\n +\"\";\n }\n $(\"#printTable\").html(thead + \"\" + tbody + \"\");\n },\n });\n\t\t});\n\n\t\tthis.SpiderCommentObj.click(function() {\n\t\t var gqCount = $(\"#gqCount-1\").val();\n\t\t $.ajax({\n url : \"/spider/getComment\",\n data:\"gqCount=\"+gqCount,\n type:\"post\",\n dataType : \"json\",\n success : function (data) {\n var thead = \" #歌曲名字作者ID\";\n var tbody = \"\";\n for (cnt in data['data']) {\n tbody = tbody + \"\" + cnt\n + \"\"\n +\"\"+ data['data'][cnt][\"name\"] + \"\"\n + \"\"+ data['data'][cnt][\"author\"] + \"\"\n +\"\"+ data['data'][cnt][\"song_id\"] + \"\"\n +\"\";\n }\n $(\"#printTable\").html(thead + \"\" + tbody + \"\");\n },\n });\n\t\t});\n\n\n\t};\n\tthis.init = function() {\n\t this.createDom();\n\t this.documentEvent();\n\t}\n\tthis.init()\n\n});\n\n\n" }, { "path": "spider163/www/static/js/stat.js", "content": "$(function () {\n this.createDom = function () {\n this.spiderPlaylistObj = $(\"#spiderPlaylist\");\n }\n this.documentEvent = function () {\n\t\tvar self = this;\n\t\tthis.spiderPlaylistObj.click(function() {\n\t\t var gdType = $(\"#gdType\").val();\n\t\t var gdCount = $(\"#gdCount\").val();\n\t\t $('#gdModal').modal('hide')\n\t\t $.ajax({\n url : \"/spider/getPlaylist\",\n data:\"gdType=\"+gdType+\"&gdCount=\"+gdCount,\n type:\"post\",\n dataType : \"json\",\n success : function (data) {\n alert(data[\"test\"]);\n },\n });\n\t\t});\n\t};\n\n\tthis.createCharts = function() {\n\t dataCount()\n\t playlist()\n\t music()\n setInterval(dataCount,10000);\n setInterval(playlist,1000000);\n setInterval(music,1000000);\n\t}\n\tthis.init = function() {\n\t this.createDom();\n\t this.documentEvent();\n\t this.createCharts();\n\t}\n\tthis.init()\n\n});\n\nfunction dataCount() {\n $.ajax({\n url : \"/stat/dataCount\",\n type:\"get\",\n dataType : \"json\",\n success : function (data) {\n var name = {\"countPlaylist\":\"歌单抓取\",\"countLyric\":\"歌词抓取\",\"countComment\":\"评论抓取\"};\n for (k in data){\n var chart = echarts.init(document.getElementById(k), 'macarons');\n var option = {\n tooltip : {formatter: \"{a}
    {b} : {c}%\"},\n// toolbox: {feature: {restore: {},saveAsImage: {}}},\n series: [\n {\n name: k,\n type: 'gauge',\n detail: {formatter:'{value}%'},\n data: [{value: data[k], name: name[k]}]\n }\n ]};\n chart.setOption(option);\n }\n },\n });\n}\n\nfunction playlist() {\n $.ajax({\n url : \"/stat/playlist\",\n type:\"get\",\n dataType : \"json\",\n success : function (data) {\n for (k in data){\n var chart = echarts.init(document.getElementById(k), 'macarons');\n var t = [];\n var v = [];\n for (d in data[k]) {\n t.push(data[k][d][0]);\n v.push(data[k][d][1]);\n }\n var option = {\n title: {text: k},\n tooltip: {},\n legend: {data:['数量']},\n xAxis: {data: t},\n yAxis: {},\n series: [{name: '数量',type: 'bar',data: v }]\n };\n chart.setOption(option);\n }\n },\n });\n\n\n\n}\n\nfunction music() {\n $.ajax({\n url : \"/stat/music\",\n type:\"get\",\n dataType : \"json\",\n success : function (data) {\n for (k in data){\n var chart = echarts.init(document.getElementById(k), 'macarons');\n var t = [];\n var v = [];\n for (d in data[k]) {\n t.push(data[k][d][0]);\n v.push(data[k][d][1]);\n }\n var option = {\n title: {text: k},\n tooltip: {},\n legend: {data:['评论数量']},\n xAxis: {data: t,\n axisLabel:{\n interval:0,\n rotate:45,//倾斜度 -90 至 90 默认为0\n margin:4,\n },\n },\n yAxis: {},\n series: [{name: '评论数量',type: 'bar',data: v }]\n };\n chart.setOption(option);\n }\n },\n });\n}" }, { "path": "spider163/www/templates/bussiness.html", "content": "\n\n\n \n Welcome To Spider163\n \n \n\n \n\n\n\n\n
    \n\t\t\t\t\t\t

    Spider163 用户许可 和 我们的期待

    \n
    \n
    \n

    \n Spider163的数据基础，来源于网易公司的网易云音乐产品。其授权协议，包含《网易云音乐服务条款》但不包含其霸王条款。
    \n 该项目遵循MIT开源协议。我们认为知识属于全人类，网易云音乐的评论区属于用户，不属于网易云音乐。
    \n 而广大网民有权利根据自己的喜好阅读、整理、分析和总结开放的、非私密信息。
    \n 附：《网易云音乐服务条款》
    \n 您可以选择四种方式支持本项目的开发：
    \n No.1 在Github上star本项目，或者在其它任何场合宣传本项目。
    \n 附：Spider163 GitHub 地址
    \n No.2 关注本项目作者的唯一个人微信公众账号。
    \n 公众号名字：程天写代码
    \n No.3 通过支付宝向作者转账赞助。
    \n
    \n No.4 通过微信向作者转账赞助。
    \n
    \n 自2018年起，所有的现金转账赞助的朋友都可以在通过转账留言在本页面留言（50字以内）
    \n

    \n
    \n\n" }, { "path": "spider163/www/templates/index.html", "content": "\n\n\n \n Welcome To Spider163\n \n \n\n\n\n\n
    \n
    \n

    简单易用、功能强大的网易云音乐爬虫

    \n

    Spider163 是程天（GitHub:Chengyumeng,微信公众账号：程天写代码）开发的网易云音乐爬虫总库。
    \n 项目遵循 MIT 开源协议

    \n

    \n 抓取\n 统计\n 浏览\n 商业\n

    \n
    \n
    \n\n
    \n \n
    \n\n\n\n\n\n\n" }, { "path": "spider163/www/templates/scan.html", "content": "\n\n\n \n Welcome To Spider163\n \n \n \n\n\n\n\n\n\n\n
    \n

    随机展示 本页面未来将增加搜索功能

    \n
    \n
    \n
    \n\n\n
    \n
    \n
    \n \n
    \n\n\n\n \n \n" }, { "path": "spider163/www/templates/spider.html", "content": "\n\n\n \n Welcome To Spider163\n \n \n \n\n\n\n\n
    \n

    智能抓取 用户需求 · 智能定制

    \n
    \n
    \n
    \n
    \n

    歌单

    \n
    \n
    \n
    \n
    \n

    歌曲

    \n
    \n
    \n
    \n
    \n

    歌词

    \n
    \n
    \n
    \n
    \n

    热评

    \n
    \n
    \n
    \n
    \n\n
    \n
    \n \n
    \n \n
    \n
    \n
    \n \n
    \n \n
    \n
    \n \n
    \n\n
    \n
    \n \n
    \n \n
    \n
    \n
    \n \n
    \n \n
    \n
    \n \n
    \n\n
    \n
    \n \n
    \n \n
    \n
    \n \n
    \n\n
    \n
    \n \n
    \n \n
    \n
    \n \n
    \n\n\n
    \n \n
    \n
    \n
    \n\n\n\n\n
    \n \n
    \n\n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "spider163/www/templates/stat.html", "content": "\n\n\n \n Welcome To Spider163\n \n \n \n \n \n\n\n\n\n\n
    \n

    Spider163 统计 · 分析

    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n \n \n \n \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n
    \n\n\n
    \n
    \n\n\n\n\n
    \n \n
    \n\n\n\n\n\n\n\n\n\n\n\n" }, { "path": "spider163/www/web.py", "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\nfrom flask import Flask, request, json, jsonify\nfrom flask import render_template, make_response\n\nfrom spider163.spider import playlist\nfrom spider163.spider import music\nfrom spider163.spider import lyric\nfrom spider163.spider import comment\nfrom spider163.utils import pysql\nfrom spider163.utils import tools\n\napp = Flask(__name__, static_path='/static')\n\n\n@app.route(\"/\")\ndef index():\n return render_template('index.html')\n\n\n@app.route(\"/spider\")\ndef spider(type=None):\n return render_template('spider.html')\n\n\n@app.route(\"/spider/getPlaylist\", methods=['POST'])\ndef get_playlist():\n pl = playlist.Playlist()\n title = pl.view_capture(int(request.form['gdPage']),tools.encode(request.form[\"gdType\"]))\n return jsonify({\"type\": request.form[\"gdType\"],\"title\": title})\n\n\n@app.route(\"/spider/getMusic\", methods=['POST'])\ndef get_music():\n mu = music.Music()\n data = mu.views_capture(tools.encode(request.form[\"gdSource\"]))\n return jsonify({\"type\": request.form[\"gdSource\"],\"data\": data})\n\n\n@app.route(\"/spider/getLyric\", methods=[\"POST\"])\ndef get_lyric():\n ly = lyric.Lyric()\n data = ly.view_lyrics(int(request.form[\"gqCount\"]))\n return jsonify({\"count\": request.form[\"gqCount\"],\"data\": data})\n\n\n@app.route(\"/spider/getComment\", methods=[\"POST\"])\ndef get_comment():\n cm = comment.Comment()\n data = cm.auto_view(int(request.form[\"gqCount\"]))\n return jsonify({\"count\": request.form[\"gqCount\"],\"data\": data})\n\n\n@app.route(\"/stat\")\ndef statistics():\n return render_template('stat.html')\n\n\n@app.route(\"/stat/playlist\")\ndef stat_playlist():\n return jsonify(pysql.stat_playlist())\n\n\n@app.route(\"/stat/music\")\ndef stat_music():\n return jsonify(pysql.stat_music())\n\n\n@app.route(\"/stat/dataCount\")\ndef stat_data():\n return jsonify(pysql.stat_data())\n\n\n@app.route(\"/scan\")\ndef scan():\n return render_template('scan.html')\n\n\n@app.route(\"/scan/data\")\ndef scan_data():\n comment = pysql.random_data()\n if len(comment) > 0:\n res = {\"msg\":\"ok\",\"num\":len(comment),\"comment\":comment}\n return jsonify(res)\n else:\n return jsonify({\"msg\":\"ok\",\"num\":len(comment),\"comment\":[]})\n\n\n@app.route(\"/bussiness\")\ndef bussiness():\n return render_template('bussiness.html')\n\n\n\n\n" } ]