[
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\ntemp/\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n\n# Custom\nconfig.ini"
  },
  {
    "path": ".vscode/launch.json",
    "content": "{\n    // 使用 IntelliSense 了解相关属性。\n    // 悬停以查看现有属性的描述。\n    // 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387\n    \"version\": \"0.2.0\",\n    \"configurations\": [\n        {\n            \"name\": \"Python: ArtStationDownloader\",\n            \"type\": \"python\",\n            \"request\": \"launch\",\n            \"program\": \"${workspaceFolder}/src/ArtStationDownloader.py\",\n            \"console\": \"integratedTerminal\",\n            \"justMyCode\": true\n        },\n        {\n            \"name\": \"Python: ArtStationDownload in Terminal\",\n            \"type\": \"python\",\n            \"request\": \"launch\",\n            \"program\": \"${workspaceFolder}/src/ArtStationDownloader.py\",\n            \"args\": [\n                \"-u\",\n                \"ase\",\n                \"ikaired\",\n                \"-d\",\n                \"${env:UserProfile}\\\\Desktop\\\\ArtStation\",\n                \"-t\",\n                \"all\"\n            ],\n            \"console\": \"integratedTerminal\",\n            \"justMyCode\": true\n        },\n        {\n            \"name\": \"Python: Current File (Integrated Terminal)\",\n            \"type\": \"python\",\n            \"request\": \"launch\",\n            \"program\": \"${file}\",\n            \"console\": \"integratedTerminal\",\n            \"justMyCode\": true\n        },\n        {\n            \"name\": \"Python: Current File (External Terminal)\",\n            \"type\": \"python\",\n            \"request\": \"launch\",\n            \"program\": \"${file}\",\n            \"console\": \"externalTerminal\",\n            \"justMyCode\": true\n        }\n    ]\n}"
  },
  {
    "path": ".vscode/settings.json",
    "content": "{\n    \"python.linting.pylintEnabled\": true,\n    \"python.formatting.provider\": \"black\"\n}"
  },
  {
    "path": "CHANGELOG",
    "content": "20180611 0.1.0-alpha1\n允许在txt中使用Python风格的注释，即以#开头的内容会被忽略\n对txt中的空白符进行处理\n保存路径不再强制包含 ArtStation\n默认保存路径现在为用户根路径\n\n20190131 0.1.1-alpha\n重构代码结构\n加入命令行模式\nGUI和命令行均加入允许选择下载文件类型（所有、图片、视频）功能\n\n20190422 0.1.1-alpha1\n对不同报错情况进行提示\n增加独立的中文README"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2018 Sean Feng\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
  },
  {
    "path": "README-zh.md",
    "content": "# ArtStation Downloader\n\nArtStation Downloader 是一个帮助你批量从[ArtStation](https://www.artstation.com/)网站下载图片和视频的小工具\n\n## 用法\n\n### 从这里开始\n\n[在此下载](https://github.com/findix/ArtStationDownloader/releases)\n\n### 如何下载\n\n输入你希望下载的作者的主页地址，或者其用户名\n\n如\n\n`https://www.artstation.com/xrnothing` 或者 `xrnothing`\n\n然后点击 Download 按钮\n\n你可以一次下载多位作者的作品\n\n只需要在输入他们的 URL 或用户名时，在中间加入英文的\",\"即可\n\n或者，你也可以新建一个文本文件（.txt），每一行输入一位作者的信息。\n（允许使用 Python 风格的注释，即 # 后的内容会被忽略，空白符与空行同样也会被忽略）\n\n然后点击 Download txt 按钮选择文件即可。\n\nType 下拉选单可以设置下载资源的类型, 你可以选择“只下载图片”、“只下载视频”以及“全部下载”\n\nPath 输入框内是下载文件夹位置，你可以按需求设置。\n\n### Proxy\n\n在 `config.ini` 配置文件中像这样设置代理信息(需要重启程序):\n\n```ini\n[Proxy]\nhttp = http://127.0.0.1:7890\nhttps = http://127.0.0.1:7890\n```\n\n## 缺陷与反馈\n\n如果在使用过程中发现任何错误、问题或是希望讨论，请使用 [Github Issues](https://github.com/findix/ArtStationDownloader/issues).\n\n非常欢迎提交 Pull requests！\n\n## 打包\n\n需要 Pyinstaller.\n\n运行 `build.bat`\n\n## For macOS/Linux/Shell\n\n首先执行 `pip install -r requirements.txt` 安装依赖\n\n在 shell 中执行 `python ./src/ArtStationDownloader.py` 开启图形界面\n\n或者\n\n直接运行类似下面的命令:\n\n`python ./src/ArtStationDownloader.py -u username_of_artist other_username or_more -d where/you/what`\n\n您可以尝试输入 `python ./src/ArtStationDownloader.py --help` 查看更多用法\n\n## FAQ\n\n> ~~**为什么我在点击下载后报错 `[Error] [403 Forbidden] You are blocked by artstation`？**~~\n\n~~ArtStation 有一个 [验证码](https://zh.wikipedia.org/zh-hans/%E9%AA%8C%E8%AF%81%E7%A0%81) 系统 (由 Cloudflare 提供)。如果你无法证明你不是爬虫，就会被禁止访问，这个问题目前我还没有找到解决办法，如果您有方式解决，请告诉我。~~\n\n使用 [samarthshrivas](https://github.com/findix/ArtStationDownloader/issues/24#issuecomment-1124734529) 推荐的方式，这个问题已经解决了。\n\n## LICENSE\n\nMIT License\n\nCopyright (c) 2018 Sean Feng"
  },
  {
    "path": "README.md",
    "content": "# ArtStation Downloader\n\n[中文说明](./README-zh.md)\n\nArtStation Downloader is a lightweight tool to help you download images and videos from the [ArtStation](https://www.artstation.com/)\n\n## Usage\n\n### Getting started\n\n[Download here](https://github.com/findix/ArtStationDownloader/releases)\n\n### Download from ArtStation\n\nInput the URL of the artist or just the username\n\njust like\n\n`https://www.artstation.com/xrnothing` or `xrnothing`\n\nand click the Download button.\n\nYou can download more then one artists' whole works at one time.\n\nJust input all the URL or usernames split with ','.\n\nor you can create a txt file with These, one artist one line.\n(you can use python style comment, any text after # will be ignored, space charactor and empty line are also ignored)\n\nand click the Download txt button to select file.\n\nThe combobox named Type means you can choose what resources are required. image only, video only or both\n\nThe Path means the path of the download folder，just set it.\n\n### Proxy\n\nSet proxy config in config.ini like this(need restart app):\n\n```ini\n[Proxy]\nhttp = http://127.0.0.1:7890\nhttps = http://127.0.0.1:7890\n```\n\n## Bugs and Feedback\n\nFor bugs, questions and discussions please use the [Github Issues](https://github.com/findix/ArtStationDownloader/issues).\n\nPull requests are all welcome!\n\n## Package\n\nRequire Pyinstaller.\n\nJust execute `build.bat`\n\n## For macOS/Linux/Shell\n\ninstall dependencies py run `pip install -r requirements.txt` first\n\njust run `python ./src/ArtStationDownloader.py` in shell to run in GUI mode\n\nor\n\nrun like this:\n\n`python ./src/ArtStationDownloader.py -u username_of_artist other_username or_more -d where/you/what` in shell\n\ntry `python ./src/ArtStationDownloader.py --help` to get more usage\n\n## FAQ\n\n> ~~**Why I get a message says `[Error] [403 Forbidden] You are blocked by artstation`?**~~\n\n~~The ArtStation has a [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) system (by Cloudflare) which need you proof you are human otherwise forbid you. I haven't found a way solve this for now. If you have idea, please tell me.~~\n\nIt seems solved by inspire of [samarthshrivas](https://github.com/findix/ArtStationDownloader/issues/24#issuecomment-1124734529)\n\n## LICENSE\n\nMIT License\n\nCopyright (c) 2018 Sean Feng"
  },
  {
    "path": "build.bat",
    "content": "pyinstaller -y --windowed .\\src\\ArtStationDownloader.py\ndel dist\\ArtStationDownloader.zip\n7z a -tzip dist\\ArtStationDownloader.zip dist\\ArtStationDownloader\npyinstaller -y --windowed -F .\\src\\ArtStationDownloader.py\n@pause"
  },
  {
    "path": "requirements_dev.txt",
    "content": "black==23.7.0\npylint==2.17.5"
  },
  {
    "path": "src/ArtStationDownloader.py",
    "content": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n\n\"\"\"批量下载ArtStation图片\n\nCopyright 2018 Sean Feng(sean@fantablade.com)\n\"\"\"\n\n__version__ = \"0.3.3\"\n# $Source$\n\nimport argparse\n\nfrom app import App\nfrom core import DownloadSorting\nfrom console import Console\n\n\ndef main():\n    parser = argparse.ArgumentParser(\n        prog=\"ArtStationDownloader\",\n        description=\"ArtStation Downloader is a lightweight tool to help you download images and videos from the ArtStation\",\n    )\n    parser.add_argument(\n        \"--version\", action=\"version\", version=\"%(prog)s \" + __version__\n    )\n    parser.add_argument(\n        \"-u\",\n        \"--username\",\n        help=\"choose who's project you want to download, one or more\",\n        nargs=\"*\",\n    )\n    parser.add_argument(\"-d\", \"--directory\", help=\"output directory\")\n    parser.add_argument(\n        \"-t\",\n        \"--type\",\n        choices=[\"all\", \"image\", \"video\"],\n        default=\"all\",\n        help=\"what do you what to download, default is all\",\n    )\n    parser.add_argument(\n        \"-s\",\n        \"--sorting\",\n        choices=[sorting.name for sorting in DownloadSorting],\n        default=DownloadSorting.TITLE_BASED.name,\n        help=\"download sorting\",\n    )\n    parser.add_argument(\n        \"-v\", \"--verbosity\", action=\"count\", help=\"increase output verbosity\"\n    )\n    args = parser.parse_args()\n\n    if args.username:\n        if args.directory:\n            console = Console()\n            console.download_by_usernames(args.username, args.directory, args.type, DownloadSorting[args.sorting])\n        else:\n            print(\"no output directory, please use -d or --directory option to set\")\n    else:\n        app = App(version=__version__)\n        app.run()  # 进入主循环，程序运行\n\n\nif __name__ == \"__main__\":\n    main()\n"
  },
  {
    "path": "src/app.py",
    "content": "# -*- coding: utf-8 -*-\n\nimport os\nimport PySimpleGUI as sg\n\nimport config\nfrom core import Core, DownloadSorting\n\n\nclass App:\n    def __init__(self, version):\n        self.core = Core(self.log)\n        self.user_config = sg.UserSettings(\n            os.path.join(os.path.abspath(\".\"), \"config.ini\"),\n            use_config_file=True,\n            convert_bools_and_none=True,\n        )\n        self.user_settings = self.user_config[\"Settings\"]\n\n        # 兼容模式\n        root_path = self.user_config[\"Paths\"][\"root_path\"]\n        if root_path:\n            self.user_settings[\"root_path\"] = root_path\n            self.user_config[\"Paths\"].delete_section()\n\n        self.root_path = self.user_settings.get(\n            \"root_path\", os.path.join(os.path.expanduser(\"~\"), \"ArtStation\")\n        )\n        self.download_sorting: DownloadSorting = DownloadSorting[\n            self.user_settings.get(\"download_sorting\", DownloadSorting.TITLE_BASED.name)\n        ]\n        self.window = sg.Window(\n            \"ArtStation Downloader \" + version,\n            layout=self.create_layout(),\n            finalize=True,\n        )\n\n        self.window[\"-DOWNLOAD-SORTING-\"].Update(self.download_sorting)\n\n        self.event_callbacks = {\n            \"-DOWNLOAD-\": lambda: self.window.perform_long_operation(self.download, \"\"),\n            \"-DOWNLOAD_TXT-\": self.get_download_txt_file,\n            \"continue_download_txt\": lambda args: self.window.perform_long_operation(\n                lambda: self.download_txt(args), \"\"\n            ),\n            \"-DOWNLOAD-SORTING-\": self._set_download_sorting,\n            \"-BROWSE-\": self.browse_directory,\n            \"log\": self._log,\n            \"popup\": self._popup,\n            \"set_download_buttons\": self._set_download_buttons,\n        }\n\n    def _set_download_sorting(self, value: DownloadSorting):\n        self.download_sorting = value\n        self.user_settings.set(\"download_sorting\", value.name)\n\n    def _log(self, value):\n        current_text = self.window[\"-LOG-\"].get()\n        self.window[\"-LOG-\"].update(f\"{current_text}\\n{value}\\n\")\n        self.window[\"-LOG-\"].Widget.see(\"end\")\n\n    def log(self, value):\n        self.window.write_event_value(\"log\", value)\n\n    def _set_download_buttons(self, state):\n        self.window[\"-DOWNLOAD-\"].update(disabled=not state)\n        self.window[\"-DOWNLOAD_TXT-\"].update(disabled=not state)\n\n    def _popup(self, args):\n        message, title = args\n        sg.popup_ok(\n            message,\n            title=title,\n            modal=True,\n        )\n\n    def download(self):\n        username_text = self.window[\"-USERNAME-\"].get()\n        if not username_text:\n            self.window.write_event_value(\n                \"popup\", (\"Please input usernames\", \"Warning\")\n            )\n            return\n        self.window.write_event_value(\"set_download_buttons\", False)\n        usernames = username_text.split(\",\")\n        self.core.root_path = self.root_path\n        self.core.download_by_usernames(\n            usernames, self.window[\"-TYPE-\"].get(), self.download_sorting\n        )\n        self.window.write_event_value(\"set_download_buttons\", True)\n        self.user_settings.set(\"default_username\", username_text)\n\n    def get_download_txt_file(self):\n        self.window.write_event_value(\"set_download_buttons\", False)\n        filename = sg.popup_get_file(\n            \"Select a file\", file_types=((\"Text Files\", \"*.txt\"), (\"All Files\", \"*.*\"))\n        )\n        self.window.write_event_value(\"continue_download_txt\", filename)\n\n    def download_txt(self, filename):\n        if filename and filename != \".\":\n            with open(filename, \"r\", encoding=\"utf-8\") as f:\n                usernames = []\n                for username in f.readlines():\n                    username = username.strip()\n                    if not username:\n                        continue\n                    sharp_at = username.find(\"#\")\n                    if sharp_at == 0:\n                        continue\n                    if sharp_at != -1:\n                        username = username[:sharp_at]\n                    usernames.append(username.strip())\n                self.core.root_path = self.root_path\n                self.core.download_by_usernames(\n                    usernames, self.window[\"-TYPE-\"].get(), self.download_sorting\n                )\n        self.window.write_event_value(\"set_download_buttons\", True)\n\n    def browse_directory(self):\n        root_path = sg.popup_get_folder(\"Select a folder\", default_path=self.root_path)\n        if root_path:\n            self.root_path = root_path\n            self.window[\"-PATH-\"].update(root_path)\n            self.user_settings.set(\"root_path\", root_path)\n\n    def create_layout(self):\n        sg.theme(\"Dark Blue 3\")\n        layout = [\n            [\n                sg.Text('Usernames (split by \",\"):'),\n                sg.InputText(\n                    self.user_settings.get(\"default_username\", \"\"), key=\"-USERNAME-\"\n                ),\n            ],\n            [\n                sg.Text(\"Type:\"),\n                sg.Combo(\n                    values=(\"all\", \"image\", \"video\"),\n                    key=\"-TYPE-\",\n                    default_value=\"all\",\n                    readonly=True,\n                    enable_events=True,\n                ),\n                sg.Text(\"File Download Sorting\"),\n                sg.Combo(\n                    tuple(DownloadSorting.__members__.values()),\n                    key=\"-DOWNLOAD-SORTING-\",\n                    default_value=DownloadSorting.TITLE_BASED,\n                    readonly=True,\n                    enable_events=True,\n                ),\n            ],\n            [\n                sg.Text(\"Path:\"),\n                sg.InputText(key=\"-PATH-\", default_text=self.root_path, disabled=True),\n                sg.Button(\"Browse\", key=\"-BROWSE-\", bind_return_key=True),\n            ],\n            [\n                sg.Button(\"Download\", key=\"-DOWNLOAD-\", bind_return_key=True),\n                sg.Button(\"Download txt\", key=\"-DOWNLOAD_TXT-\"),\n            ],\n            [sg.Multiline(size=(80, 20), key=\"-LOG-\", disabled=True)],\n            [sg.StatusBar(\"Feel free to use! Support: Sean Feng(sean@fantablade.com)\")],\n        ]\n        return layout\n\n    def run(self):\n        while True:\n            event, values = self.window.read()\n            if event == sg.WINDOW_CLOSED:\n                break\n            elif event in self.event_callbacks:\n                if event in values:\n                    self.event_callbacks[event](values[event])\n                else:\n                    self.event_callbacks[event]()\n\n        self.window.close()\n"
  },
  {
    "path": "src/config.py",
    "content": "#!/usr/bin/python\n# -*- coding:utf-8 -*-\n# author: lingyue.wkl\n# desc: use to read ini\n# ---------------------\n# 2012-02-18 created\n# 2012-09-02 changed for class support\n\n# ---------------------\n\nimport sys\nimport configparser\n\n\nclass Config:\n    def __init__(self, path):\n        self.path = path\n        self.cf = configparser.ConfigParser()\n        self.cf.read(self.path)\n\n    def get(self, field, key):\n        result = \"\"\n        try:\n            result = self.cf.get(field, key)\n        except:\n            result = \"\"\n        return result\n\n    def set(self, filed, key, value):\n        try:\n            self.cf.set(field, key, value)\n            self.cf.write(open(self.path, \"w\", encoding=\"utf-8\"))\n        except:\n            return False\n        return True\n\n\ndef read_config(config_file_path, field, key):\n    cf = configparser.ConfigParser()\n    try:\n        cf.read(config_file_path)\n        if field in cf:\n            result = cf[field][key]\n        else:\n            return \"\"\n    except configparser.Error as e:\n        print(e)\n        return \"\"\n    return result\n\n\ndef write_config(config_file_path, field, key, value):\n    cf = configparser.ConfigParser()\n    try:\n        cf.read(config_file_path)\n        if field not in cf:\n            cf.add_section(field)\n        cf[field][key] = value\n        cf.write(open(config_file_path, \"w\", encoding=\"utf-8\"))\n    except configparser.Error as e:\n        print(e)\n        return False\n    return True\n\n\nif __name__ == \"__main__\":\n    if len(sys.argv) < 4:\n        sys.exit(1)\n\n    config_file_path = sys.argv[1]\n    field = sys.argv[2]\n    key = sys.argv[3]\n    if len(sys.argv) == 4:\n        print(read_config(config_file_path, field, key))\n    else:\n        value = sys.argv[4]\n        write_config(config_file_path, field, key, value)\n"
  },
  {
    "path": "src/console.py",
    "content": "# -*- coding: utf-8 -*-\n\nfrom core import Core\n\n\nclass Console:\n    def __init__(self):\n        self.core = Core()\n\n    def download_by_usernames(self, usernames, directory, download_type, download_sorting):\n        self.core.root_path = directory\n        self.core.download_by_usernames(usernames, download_type, download_sorting)\n"
  },
  {
    "path": "src/core.py",
    "content": "# -*- coding: utf-8 -*-\n\n\"\"\"内核方法\nCopyright 2018-2019 Sean Feng(sean@FantaBlade.com)\n\"\"\"\n\nimport os\nfrom concurrent import futures\nfrom enum import Enum\nfrom multiprocessing import cpu_count\n\nfrom bs4 import BeautifulSoup, element\nfrom pytube import YouTube\n\nfrom http_client import HttpClient\n\n\nclass DownloadSorting(Enum):\n    TITLE_BASED = \"Title-based\"\n    USERNAME_BASED = \"Username-based\"\n    ALL_IN_ONE = \"All-in-one\"\n\n    def __str__(self) -> str:\n        return self.value\n\n\nclass Core:\n    def log(self, message):\n        if self._log_print:\n            self._log_print(message)\n        else:\n            print(message)\n\n    def __init__(self, log_print=None):\n        self._log_print = log_print\n        max_workers = cpu_count() * 4\n        self.executor = futures.ThreadPoolExecutor(max_workers)\n        self.executor_video = futures.ThreadPoolExecutor(1)\n        self.invoke = self._get_invoke()\n        self.invoke_video = self._get_invoke(\"video\")\n        self.root_path: str = None\n        self.download_sorting: DownloadSorting = None\n        self.futures = []\n        self.http_client = HttpClient(log_print=log_print)\n\n    def download_file(self, url, file_path, file_name):\n        url = url.replace(\"/large/\", \"/4k/\")\n        file_full_path = os.path.join(file_path, file_name)\n        if os.path.exists(file_full_path):\n            self.log(\"[Exist][image][{}]\".format(file_full_path))\n        else:\n            resp = self.http_client.http_get(url)\n            os.makedirs(file_path, exist_ok=True)\n            with open(file_full_path, \"wb\") as code:\n                code.write(resp.content)\n            self.log(\"[Finish][image][{}]\".format(file_full_path))\n\n    def download_video(self, youtube_id, file_path):\n        file_full_path = os.path.join(file_path, \"{}.{}\".format(youtube_id, \"mp4\"))\n        if os.path.exists(file_full_path):\n            self.log(\"[Exist][video][{}]\".format(file_full_path))\n        else:\n            try:\n                yt = YouTube(f'https://www.youtube.com/watch?v={youtube_id}')\n                stream = yt.streams.filter(file_extension='mp4').first()\n                stream.download(output_path=file_path)\n                self.log(\"[Finish][video][{}]\".format(file_full_path))\n            except Exception as e:\n                self.log(\"[Error][video][{}]\".format(e))\n\n    def download_project(self, hash_id):\n        url = \"https://www.artstation.com/projects/{}.json\".format(hash_id)\n        resp = self.http_client.http_client_get_json(url)\n        j = resp\n        assets = j[\"assets\"]\n        title = j[\"slug\"].strip()\n        # self.log('=========={}=========='.format(title))\n        username = j[\"user\"][\"username\"]\n        for asset in assets:\n            assert self.root_path\n            if self.download_sorting != DownloadSorting.ALL_IN_ONE:\n                user_path = os.path.join(self.root_path, username)\n            else:\n                user_path = self.root_path\n            os.makedirs(user_path, exist_ok=True)\n            if self.download_sorting == DownloadSorting.TITLE_BASED:\n                file_path = os.path.join(user_path, title)\n            else:\n                file_path = user_path\n            if not self.no_image and asset[\"has_image\"]:  # 包含图片\n                url = asset[\"image_url\"]\n                file_name = HttpClient.urlparse(url).path.split(\"/\")[-1]\n                try:\n                    self.futures.append(\n                        self.invoke(self.download_file, url, file_path, file_name)\n                    )\n                except Exception as e:\n                    self.log(e)\n            if not self.no_video and asset[\"has_embedded_player\"]:  # 包含视频\n                player_embedded = BeautifulSoup(asset[\"player_embedded\"], \"html.parser\")\n                src = player_embedded.find(\"iframe\").get(\"src\")\n                if \"youtube\" in src:\n                    youtube_id = self.http_client.urlparse(src).path[-11:]\n                    try:\n                        self.futures.append(\n                            self.invoke_video(self.download_video, youtube_id, file_path)\n                        )\n                    except Exception as e:\n                        self.log(e)\n\n    def get_projects(self, username) -> element.ResultSet[element.Tag]:\n        data = []\n        if username != \"\":\n            page = 0\n            while True:\n                page += 1\n                url = \"https://{}.artstation.com/rss?page={}\".format(username, page)\n                resp = self.http_client.http_client_get(url)\n                if resp.status != 200:\n                    err = \"[Error] [{} {}] \".format(resp.status, resp.reason)\n                    if resp.status == 403:\n                        self.log(err + \"You are blocked by artstation\")\n                    elif resp.status == 404:\n                        self.log(err + \"Username not found\")\n                    else:\n                        self.log(err + \"Unknown error\")\n                    break\n                channel = BeautifulSoup(\n                    resp.read().decode(\"utf-8\"), \"lxml-xml\"\n                ).rss.channel\n                links = channel.select(\"item > link\")\n                if len(links) == 0:\n                    break\n                if page == 1:\n                    self.log(\"\\n==========[{}] BEGIN==========\".format(username))\n                data += links\n                self.log(\"\\n==========Get page {}==========\".format(page))\n        return data\n\n    def download_by_username(self, username):\n        data = self.get_projects(username)\n        if len(data) != 0:\n            future_list = []\n            for project in data:\n                if project.string.startswith(\"https://www.artstation.com/artwork/\"):\n                    future = self.invoke(\n                        self.download_project, project.string.split(\"/\")[-1]\n                    )\n                    future_list.append(future)\n            futures.wait(future_list)\n\n    def download_by_usernames(\n        self, usernames, download_type, download_sorting: DownloadSorting\n    ):\n        self.http_client.proxy_setup()\n        self.no_image = download_type == \"video\"\n        self.no_video = download_type == \"image\"\n        self.download_sorting = download_sorting\n        # 去重与处理网址\n        username_set = set()\n        for username in usernames:\n            username = username.strip().split(\"/\")[-1]\n            if username not in username_set:\n                username_set.add(username)\n                self.download_by_username(username)\n        futures.wait(self.futures)\n        self.log(\"\\n========ALL DONE========\")\n\n    def _get_invoke(self, executor=None):\n        def invoke(func, *args, **kwargs):\n            def done_callback(worker):\n                worker_exception = worker.exception()\n                if worker_exception:\n                    self.log(str(worker_exception))\n                    raise (worker_exception)\n\n            if executor == \"video\":\n                futurn = self.executor_video.submit(func, *args, **kwargs)\n                futurn.add_done_callback(done_callback)\n            else:\n                futurn = self.executor.submit(func, *args, **kwargs)\n                futurn.add_done_callback(done_callback)\n            return futurn\n\n        return invoke\n"
  },
  {
    "path": "src/http_client.py",
    "content": "import http.client as http_client\nimport ssl\nimport json\nimport os\nimport ssl\nfrom urllib.parse import urlparse, ParseResult\n\nimport requests\n\nfrom config import Config\n\n\nclass HttpClient:\n    def log(self, message):\n        if self._log_print:\n            self._log_print(message)\n        else:\n            print(message)\n\n    def __init__(self, log_print=None):\n        self._log_print = log_print\n        self._session = requests.session()\n        self.proxy_setup(self._session)\n\n    def http_client_get(self, url, ignoreCertificateError=None):\n        try:\n            headers = {\n                \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/113.0\",\n                \"Accept\": \"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\",\n                \"Accept-Language\": \"en-US,en;q=0.5\",\n            }\n            parsed_url = urlparse(url)\n            if ignoreCertificateError:\n                context = ssl._create_unverified_context()\n            else:\n                context = None\n            conn = http_client.HTTPSConnection(parsed_url.netloc, context=context)\n            conn.request(\n                \"GET\", parsed_url.path + \"?\" + parsed_url.query, headers=headers\n            )\n\n            resp = conn.getresponse()\n        except ssl.SSLCertVerificationError:\n            return self.http_client_get(url, ignoreCertificateError=True)\n        except Exception as e:\n            self.log(f\"Connect error [{e}]\")\n            return\n\n        return resp\n\n    def http_client_get_json(self, url):\n        resp = self.http_client_get(url)\n        try:\n            resp_str = resp.read().decode()\n            json_result = json.loads(resp_str)\n        except json.decoder.JSONDecodeError:\n            self.log(f\"json decode error\\nurl:{url}\\n{resp_str}\")\n            return\n        return json_result\n\n    def http_get(self, url):\n        try:\n            resp = self._session.get(url, timeout=10)\n        except requests.exceptions.InvalidURL:\n            self.log(f'\"{url}\" is not valid url')\n            return\n        return resp\n\n    def proxy_setup(self, session=None):\n        if not session:\n            session = self._session\n        # 设置 User Agent\n        session.headers.update(\n            {\n                \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36\"\n            }\n        )\n        # 设置代理\n        config = Config(\"config.ini\")\n        http = config.get(\"Proxy\", \"http\")\n        https = config.get(\"Proxy\", \"https\")\n        if http or https:\n            proxys = {}\n            if http:\n                proxys[\"http\"] = http\n                os.environ[\"HTTP_PROXY\"] = http\n            if https:\n                proxys[\"https\"] = https\n                os.environ[\"HTTPS_PROXY\"] = https\n            session.proxies.update(proxys)\n\n    @staticmethod\n    def urlparse(url: str) -> ParseResult:\n        return urlparse(url)\n"
  }
]