Full Code of findix/ArtStationDownloader for AI

Repository: findix/ArtStationDownloader
Branch: master
Commit: 6c659c5b7231
Files: 16
Total size: 28.4 KB

Directory structure:
gitextract_ejxwnxrw/

├── .gitignore
├── .vscode/
│   ├── launch.json
│   └── settings.json
├── CHANGELOG
├── LICENSE
├── README-zh.md
├── README.md
├── build.bat
├── requirements.txt
├── requirements_dev.txt
└── src/
    ├── ArtStationDownloader.py
    ├── app.py
    ├── config.py
    ├── console.py
    ├── core.py
    └── http_client.py

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so
temp/
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/

# Translations
*.mo
*.pot

# Django stuff:
*.log
local_settings.py
db.sqlite3

# Flask stuff:
instance/
.webassets-cache

# Scrapy stuff:
.scrapy

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Jupyter Notebook
.ipynb_checkpoints

# pyenv
.python-version

# celery beat schedule file
celerybeat-schedule

# SageMath parsed files
*.sage.py

# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/

# Spyder project settings
.spyderproject
.spyproject

# Rope project settings
.ropeproject

# mkdocs documentation
/site

# mypy
.mypy_cache/

# Custom
config.ini

================================================
FILE: .vscode/launch.json
================================================
{
    // 使用 IntelliSense 了解相关属性。
    // 悬停以查看现有属性的描述。
    // 欲了解更多信息，请访问: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: ArtStationDownloader",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/src/ArtStationDownloader.py",
            "console": "integratedTerminal",
            "justMyCode": true
        },
        {
            "name": "Python: ArtStationDownload in Terminal",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/src/ArtStationDownloader.py",
            "args": [
                "-u",
                "ase",
                "ikaired",
                "-d",
                "${env:UserProfile}\\Desktop\\ArtStation",
                "-t",
                "all"
            ],
            "console": "integratedTerminal",
            "justMyCode": true
        },
        {
            "name": "Python: Current File (Integrated Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "integratedTerminal",
            "justMyCode": true
        },
        {
            "name": "Python: Current File (External Terminal)",
            "type": "python",
            "request": "launch",
            "program": "${file}",
            "console": "externalTerminal",
            "justMyCode": true
        }
    ]
}

================================================
FILE: .vscode/settings.json
================================================
{
    "python.linting.pylintEnabled": true,
    "python.formatting.provider": "black"
}

================================================
FILE: CHANGELOG
================================================
20180611 0.1.0-alpha1
允许在txt中使用Python风格的注释，即以#开头的内容会被忽略
对txt中的空白符进行处理
保存路径不再强制包含 ArtStation
默认保存路径现在为用户根路径

20190131 0.1.1-alpha
重构代码结构
加入命令行模式
GUI和命令行均加入允许选择下载文件类型（所有、图片、视频）功能

20190422 0.1.1-alpha1
对不同报错情况进行提示
增加独立的中文README

================================================
FILE: LICENSE
================================================
MIT License

Copyright (c) 2018 Sean Feng

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

================================================
FILE: README-zh.md
================================================
# ArtStation Downloader

ArtStation Downloader 是一个帮助你批量从[ArtStation](https://www.artstation.com/)网站下载图片和视频的小工具

## 用法

### 从这里开始

[在此下载](https://github.com/findix/ArtStationDownloader/releases)

### 如何下载

输入你希望下载的作者的主页地址，或者其用户名

如

`https://www.artstation.com/xrnothing` 或者 `xrnothing`

然后点击 Download 按钮

你可以一次下载多位作者的作品

只需要在输入他们的 URL 或用户名时，在中间加入英文的","即可

或者，你也可以新建一个文本文件（.txt），每一行输入一位作者的信息。
（允许使用 Python 风格的注释，即 # 后的内容会被忽略，空白符与空行同样也会被忽略）

然后点击 Download txt 按钮选择文件即可。

Type 下拉选单可以设置下载资源的类型, 你可以选择“只下载图片”、“只下载视频”以及“全部下载”

Path 输入框内是下载文件夹位置，你可以按需求设置。

### Proxy

在 `config.ini` 配置文件中像这样设置代理信息(需要重启程序):

```ini
[Proxy]
http = http://127.0.0.1:7890
https = http://127.0.0.1:7890
```

## 缺陷与反馈

如果在使用过程中发现任何错误、问题或是希望讨论，请使用 [Github Issues](https://github.com/findix/ArtStationDownloader/issues).

非常欢迎提交 Pull requests！

## 打包

需要 Pyinstaller.

运行 `build.bat`

## For macOS/Linux/Shell

首先执行 `pip install -r requirements.txt` 安装依赖

在 shell 中执行 `python ./src/ArtStationDownloader.py` 开启图形界面

或者

直接运行类似下面的命令:

`python ./src/ArtStationDownloader.py -u username_of_artist other_username or_more -d where/you/what`

您可以尝试输入 `python ./src/ArtStationDownloader.py --help` 查看更多用法

## FAQ

> ~~**为什么我在点击下载后报错 `[Error] [403 Forbidden] You are blocked by artstation`？**~~

~~ArtStation 有一个 [验证码](https://zh.wikipedia.org/zh-hans/%E9%AA%8C%E8%AF%81%E7%A0%81) 系统 (由 Cloudflare 提供)。如果你无法证明你不是爬虫，就会被禁止访问，这个问题目前我还没有找到解决办法，如果您有方式解决，请告诉我。~~

使用 [samarthshrivas](https://github.com/findix/ArtStationDownloader/issues/24#issuecomment-1124734529) 推荐的方式，这个问题已经解决了。

## LICENSE

MIT License

Copyright (c) 2018 Sean Feng

================================================
FILE: README.md
================================================
# ArtStation Downloader

[中文说明](./README-zh.md)

ArtStation Downloader is a lightweight tool to help you download images and videos from the [ArtStation](https://www.artstation.com/)

## Usage

### Getting started

[Download here](https://github.com/findix/ArtStationDownloader/releases)

### Download from ArtStation

Input the URL of the artist or just the username

just like

`https://www.artstation.com/xrnothing` or `xrnothing`

and click the Download button.

You can download more then one artists' whole works at one time.

Just input all the URL or usernames split with ','.

or you can create a txt file with These, one artist one line.
(you can use python style comment, any text after # will be ignored, space charactor and empty line are also ignored)

and click the Download txt button to select file.

The combobox named Type means you can choose what resources are required. image only, video only or both

The Path means the path of the download folder，just set it.

### Proxy

Set proxy config in config.ini like this(need restart app):

```ini
[Proxy]
http = http://127.0.0.1:7890
https = http://127.0.0.1:7890
```

## Bugs and Feedback

For bugs, questions and discussions please use the [Github Issues](https://github.com/findix/ArtStationDownloader/issues).

Pull requests are all welcome!

## Package

Require Pyinstaller.

Just execute `build.bat`

## For macOS/Linux/Shell

install dependencies py run `pip install -r requirements.txt` first

just run `python ./src/ArtStationDownloader.py` in shell to run in GUI mode

or

run like this:

`python ./src/ArtStationDownloader.py -u username_of_artist other_username or_more -d where/you/what` in shell

try `python ./src/ArtStationDownloader.py --help` to get more usage

## FAQ

> ~~**Why I get a message says `[Error] [403 Forbidden] You are blocked by artstation`?**~~

~~The ArtStation has a [CAPTCHA](https://en.wikipedia.org/wiki/CAPTCHA) system (by Cloudflare) which need you proof you are human otherwise forbid you. I haven't found a way solve this for now. If you have idea, please tell me.~~

It seems solved by inspire of [samarthshrivas](https://github.com/findix/ArtStationDownloader/issues/24#issuecomment-1124734529)

## LICENSE

MIT License

Copyright (c) 2018 Sean Feng

================================================
FILE: build.bat
================================================
pyinstaller -y --windowed .\src\ArtStationDownloader.py
del dist\ArtStationDownloader.zip
7z a -tzip dist\ArtStationDownloader.zip dist\ArtStationDownloader
pyinstaller -y --windowed -F .\src\ArtStationDownloader.py
@pause

================================================
FILE: requirements_dev.txt
================================================
black==23.7.0
pylint==2.17.5

================================================
FILE: src/ArtStationDownloader.py
================================================
#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""批量下载ArtStation图片

Copyright 2018 Sean Feng(sean@fantablade.com)
"""

__version__ = "0.3.3"
# $Source$

import argparse

from app import App
from core import DownloadSorting
from console import Console


def main():
    parser = argparse.ArgumentParser(
        prog="ArtStationDownloader",
        description="ArtStation Downloader is a lightweight tool to help you download images and videos from the ArtStation",
    )
    parser.add_argument(
        "--version", action="version", version="%(prog)s " + __version__
    )
    parser.add_argument(
        "-u",
        "--username",
        help="choose who's project you want to download, one or more",
        nargs="*",
    )
    parser.add_argument("-d", "--directory", help="output directory")
    parser.add_argument(
        "-t",
        "--type",
        choices=["all", "image", "video"],
        default="all",
        help="what do you what to download, default is all",
    )
    parser.add_argument(
        "-s",
        "--sorting",
        choices=[sorting.name for sorting in DownloadSorting],
        default=DownloadSorting.TITLE_BASED.name,
        help="download sorting",
    )
    parser.add_argument(
        "-v", "--verbosity", action="count", help="increase output verbosity"
    )
    args = parser.parse_args()

    if args.username:
        if args.directory:
            console = Console()
            console.download_by_usernames(args.username, args.directory, args.type, DownloadSorting[args.sorting])
        else:
            print("no output directory, please use -d or --directory option to set")
    else:
        app = App(version=__version__)
        app.run()  # 进入主循环，程序运行


if __name__ == "__main__":
    main()


================================================
FILE: src/app.py
================================================
# -*- coding: utf-8 -*-

import os
import PySimpleGUI as sg

import config
from core import Core, DownloadSorting


class App:
    def __init__(self, version):
        self.core = Core(self.log)
        self.user_config = sg.UserSettings(
            os.path.join(os.path.abspath("."), "config.ini"),
            use_config_file=True,
            convert_bools_and_none=True,
        )
        self.user_settings = self.user_config["Settings"]

        # 兼容模式
        root_path = self.user_config["Paths"]["root_path"]
        if root_path:
            self.user_settings["root_path"] = root_path
            self.user_config["Paths"].delete_section()

        self.root_path = self.user_settings.get(
            "root_path", os.path.join(os.path.expanduser("~"), "ArtStation")
        )
        self.download_sorting: DownloadSorting = DownloadSorting[
            self.user_settings.get("download_sorting", DownloadSorting.TITLE_BASED.name)
        ]
        self.window = sg.Window(
            "ArtStation Downloader " + version,
            layout=self.create_layout(),
            finalize=True,
        )

        self.window["-DOWNLOAD-SORTING-"].Update(self.download_sorting)

        self.event_callbacks = {
            "-DOWNLOAD-": lambda: self.window.perform_long_operation(self.download, ""),
            "-DOWNLOAD_TXT-": self.get_download_txt_file,
            "continue_download_txt": lambda args: self.window.perform_long_operation(
                lambda: self.download_txt(args), ""
            ),
            "-DOWNLOAD-SORTING-": self._set_download_sorting,
            "-BROWSE-": self.browse_directory,
            "log": self._log,
            "popup": self._popup,
            "set_download_buttons": self._set_download_buttons,
        }

    def _set_download_sorting(self, value: DownloadSorting):
        self.download_sorting = value
        self.user_settings.set("download_sorting", value.name)

    def _log(self, value):
        current_text = self.window["-LOG-"].get()
        self.window["-LOG-"].update(f"{current_text}\n{value}\n")
        self.window["-LOG-"].Widget.see("end")

    def log(self, value):
        self.window.write_event_value("log", value)

    def _set_download_buttons(self, state):
        self.window["-DOWNLOAD-"].update(disabled=not state)
        self.window["-DOWNLOAD_TXT-"].update(disabled=not state)

    def _popup(self, args):
        message, title = args
        sg.popup_ok(
            message,
            title=title,
            modal=True,
        )

    def download(self):
        username_text = self.window["-USERNAME-"].get()
        if not username_text:
            self.window.write_event_value(
                "popup", ("Please input usernames", "Warning")
            )
            return
        self.window.write_event_value("set_download_buttons", False)
        usernames = username_text.split(",")
        self.core.root_path = self.root_path
        self.core.download_by_usernames(
            usernames, self.window["-TYPE-"].get(), self.download_sorting
        )
        self.window.write_event_value("set_download_buttons", True)
        self.user_settings.set("default_username", username_text)

    def get_download_txt_file(self):
        self.window.write_event_value("set_download_buttons", False)
        filename = sg.popup_get_file(
            "Select a file", file_types=(("Text Files", "*.txt"), ("All Files", "*.*"))
        )
        self.window.write_event_value("continue_download_txt", filename)

    def download_txt(self, filename):
        if filename and filename != ".":
            with open(filename, "r", encoding="utf-8") as f:
                usernames = []
                for username in f.readlines():
                    username = username.strip()
                    if not username:
                        continue
                    sharp_at = username.find("#")
                    if sharp_at == 0:
                        continue
                    if sharp_at != -1:
                        username = username[:sharp_at]
                    usernames.append(username.strip())
                self.core.root_path = self.root_path
                self.core.download_by_usernames(
                    usernames, self.window["-TYPE-"].get(), self.download_sorting
                )
        self.window.write_event_value("set_download_buttons", True)

    def browse_directory(self):
        root_path = sg.popup_get_folder("Select a folder", default_path=self.root_path)
        if root_path:
            self.root_path = root_path
            self.window["-PATH-"].update(root_path)
            self.user_settings.set("root_path", root_path)

    def create_layout(self):
        sg.theme("Dark Blue 3")
        layout = [
            [
                sg.Text('Usernames (split by ","):'),
                sg.InputText(
                    self.user_settings.get("default_username", ""), key="-USERNAME-"
                ),
            ],
            [
                sg.Text("Type:"),
                sg.Combo(
                    values=("all", "image", "video"),
                    key="-TYPE-",
                    default_value="all",
                    readonly=True,
                    enable_events=True,
                ),
                sg.Text("File Download Sorting"),
                sg.Combo(
                    tuple(DownloadSorting.__members__.values()),
                    key="-DOWNLOAD-SORTING-",
                    default_value=DownloadSorting.TITLE_BASED,
                    readonly=True,
                    enable_events=True,
                ),
            ],
            [
                sg.Text("Path:"),
                sg.InputText(key="-PATH-", default_text=self.root_path, disabled=True),
                sg.Button("Browse", key="-BROWSE-", bind_return_key=True),
            ],
            [
                sg.Button("Download", key="-DOWNLOAD-", bind_return_key=True),
                sg.Button("Download txt", key="-DOWNLOAD_TXT-"),
            ],
            [sg.Multiline(size=(80, 20), key="-LOG-", disabled=True)],
            [sg.StatusBar("Feel free to use! Support: Sean Feng(sean@fantablade.com)")],
        ]
        return layout

    def run(self):
        while True:
            event, values = self.window.read()
            if event == sg.WINDOW_CLOSED:
                break
            elif event in self.event_callbacks:
                if event in values:
                    self.event_callbacks[event](values[event])
                else:
                    self.event_callbacks[event]()

        self.window.close()


================================================
FILE: src/config.py
================================================
#!/usr/bin/python
# -*- coding:utf-8 -*-
# author: lingyue.wkl
# desc: use to read ini
# ---------------------
# 2012-02-18 created
# 2012-09-02 changed for class support

# ---------------------

import sys
import configparser


class Config:
    def __init__(self, path):
        self.path = path
        self.cf = configparser.ConfigParser()
        self.cf.read(self.path)

    def get(self, field, key):
        result = ""
        try:
            result = self.cf.get(field, key)
        except:
            result = ""
        return result

    def set(self, filed, key, value):
        try:
            self.cf.set(field, key, value)
            self.cf.write(open(self.path, "w", encoding="utf-8"))
        except:
            return False
        return True


def read_config(config_file_path, field, key):
    cf = configparser.ConfigParser()
    try:
        cf.read(config_file_path)
        if field in cf:
            result = cf[field][key]
        else:
            return ""
    except configparser.Error as e:
        print(e)
        return ""
    return result


def write_config(config_file_path, field, key, value):
    cf = configparser.ConfigParser()
    try:
        cf.read(config_file_path)
        if field not in cf:
            cf.add_section(field)
        cf[field][key] = value
        cf.write(open(config_file_path, "w", encoding="utf-8"))
    except configparser.Error as e:
        print(e)
        return False
    return True


if __name__ == "__main__":
    if len(sys.argv) < 4:
        sys.exit(1)

    config_file_path = sys.argv[1]
    field = sys.argv[2]
    key = sys.argv[3]
    if len(sys.argv) == 4:
        print(read_config(config_file_path, field, key))
    else:
        value = sys.argv[4]
        write_config(config_file_path, field, key, value)


================================================
FILE: src/console.py
================================================
# -*- coding: utf-8 -*-

from core import Core


class Console:
    def __init__(self):
        self.core = Core()

    def download_by_usernames(self, usernames, directory, download_type, download_sorting):
        self.core.root_path = directory
        self.core.download_by_usernames(usernames, download_type, download_sorting)


================================================
FILE: src/core.py
================================================
# -*- coding: utf-8 -*-

"""内核方法
Copyright 2018-2019 Sean Feng(sean@FantaBlade.com)
"""

import os
from concurrent import futures
from enum import Enum
from multiprocessing import cpu_count

from bs4 import BeautifulSoup, element
from pytube import YouTube

from http_client import HttpClient


class DownloadSorting(Enum):
    TITLE_BASED = "Title-based"
    USERNAME_BASED = "Username-based"
    ALL_IN_ONE = "All-in-one"

    def __str__(self) -> str:
        return self.value


class Core:
    def log(self, message):
        if self._log_print:
            self._log_print(message)
        else:
            print(message)

    def __init__(self, log_print=None):
        self._log_print = log_print
        max_workers = cpu_count() * 4
        self.executor = futures.ThreadPoolExecutor(max_workers)
        self.executor_video = futures.ThreadPoolExecutor(1)
        self.invoke = self._get_invoke()
        self.invoke_video = self._get_invoke("video")
        self.root_path: str = None
        self.download_sorting: DownloadSorting = None
        self.futures = []
        self.http_client = HttpClient(log_print=log_print)

    def download_file(self, url, file_path, file_name):
        url = url.replace("/large/", "/4k/")
        file_full_path = os.path.join(file_path, file_name)
        if os.path.exists(file_full_path):
            self.log("[Exist][image][{}]".format(file_full_path))
        else:
            resp = self.http_client.http_get(url)
            os.makedirs(file_path, exist_ok=True)
            with open(file_full_path, "wb") as code:
                code.write(resp.content)
            self.log("[Finish][image][{}]".format(file_full_path))

    def download_video(self, youtube_id, file_path):
        file_full_path = os.path.join(file_path, "{}.{}".format(youtube_id, "mp4"))
        if os.path.exists(file_full_path):
            self.log("[Exist][video][{}]".format(file_full_path))
        else:
            try:
                yt = YouTube(f'https://www.youtube.com/watch?v={youtube_id}')
                stream = yt.streams.filter(file_extension='mp4').first()
                stream.download(output_path=file_path)
                self.log("[Finish][video][{}]".format(file_full_path))
            except Exception as e:
                self.log("[Error][video][{}]".format(e))

    def download_project(self, hash_id):
        url = "https://www.artstation.com/projects/{}.json".format(hash_id)
        resp = self.http_client.http_client_get_json(url)
        j = resp
        assets = j["assets"]
        title = j["slug"].strip()
        # self.log('=========={}=========='.format(title))
        username = j["user"]["username"]
        for asset in assets:
            assert self.root_path
            if self.download_sorting != DownloadSorting.ALL_IN_ONE:
                user_path = os.path.join(self.root_path, username)
            else:
                user_path = self.root_path
            os.makedirs(user_path, exist_ok=True)
            if self.download_sorting == DownloadSorting.TITLE_BASED:
                file_path = os.path.join(user_path, title)
            else:
                file_path = user_path
            if not self.no_image and asset["has_image"]:  # 包含图片
                url = asset["image_url"]
                file_name = HttpClient.urlparse(url).path.split("/")[-1]
                try:
                    self.futures.append(
                        self.invoke(self.download_file, url, file_path, file_name)
                    )
                except Exception as e:
                    self.log(e)
            if not self.no_video and asset["has_embedded_player"]:  # 包含视频
                player_embedded = BeautifulSoup(asset["player_embedded"], "html.parser")
                src = player_embedded.find("iframe").get("src")
                if "youtube" in src:
                    youtube_id = self.http_client.urlparse(src).path[-11:]
                    try:
                        self.futures.append(
                            self.invoke_video(self.download_video, youtube_id, file_path)
                        )
                    except Exception as e:
                        self.log(e)

    def get_projects(self, username) -> element.ResultSet[element.Tag]:
        data = []
        if username != "":
            page = 0
            while True:
                page += 1
                url = "https://{}.artstation.com/rss?page={}".format(username, page)
                resp = self.http_client.http_client_get(url)
                if resp.status != 200:
                    err = "[Error] [{} {}] ".format(resp.status, resp.reason)
                    if resp.status == 403:
                        self.log(err + "You are blocked by artstation")
                    elif resp.status == 404:
                        self.log(err + "Username not found")
                    else:
                        self.log(err + "Unknown error")
                    break
                channel = BeautifulSoup(
                    resp.read().decode("utf-8"), "lxml-xml"
                ).rss.channel
                links = channel.select("item > link")
                if len(links) == 0:
                    break
                if page == 1:
                    self.log("\n==========[{}] BEGIN==========".format(username))
                data += links
                self.log("\n==========Get page {}==========".format(page))
        return data

    def download_by_username(self, username):
        data = self.get_projects(username)
        if len(data) != 0:
            future_list = []
            for project in data:
                if project.string.startswith("https://www.artstation.com/artwork/"):
                    future = self.invoke(
                        self.download_project, project.string.split("/")[-1]
                    )
                    future_list.append(future)
            futures.wait(future_list)

    def download_by_usernames(
        self, usernames, download_type, download_sorting: DownloadSorting
    ):
        self.http_client.proxy_setup()
        self.no_image = download_type == "video"
        self.no_video = download_type == "image"
        self.download_sorting = download_sorting
        # 去重与处理网址
        username_set = set()
        for username in usernames:
            username = username.strip().split("/")[-1]
            if username not in username_set:
                username_set.add(username)
                self.download_by_username(username)
        futures.wait(self.futures)
        self.log("\n========ALL DONE========")

    def _get_invoke(self, executor=None):
        def invoke(func, *args, **kwargs):
            def done_callback(worker):
                worker_exception = worker.exception()
                if worker_exception:
                    self.log(str(worker_exception))
                    raise (worker_exception)

            if executor == "video":
                futurn = self.executor_video.submit(func, *args, **kwargs)
                futurn.add_done_callback(done_callback)
            else:
                futurn = self.executor.submit(func, *args, **kwargs)
                futurn.add_done_callback(done_callback)
            return futurn

        return invoke


================================================
FILE: src/http_client.py
================================================
import http.client as http_client
import ssl
import json
import os
import ssl
from urllib.parse import urlparse, ParseResult

import requests

from config import Config


class HttpClient:
    def log(self, message):
        if self._log_print:
            self._log_print(message)
        else:
            print(message)

    def __init__(self, log_print=None):
        self._log_print = log_print
        self._session = requests.session()
        self.proxy_setup(self._session)

    def http_client_get(self, url, ignoreCertificateError=None):
        try:
            headers = {
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/113.0",
                "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
                "Accept-Language": "en-US,en;q=0.5",
            }
            parsed_url = urlparse(url)
            if ignoreCertificateError:
                context = ssl._create_unverified_context()
            else:
                context = None
            conn = http_client.HTTPSConnection(parsed_url.netloc, context=context)
            conn.request(
                "GET", parsed_url.path + "?" + parsed_url.query, headers=headers
            )

            resp = conn.getresponse()
        except ssl.SSLCertVerificationError:
            return self.http_client_get(url, ignoreCertificateError=True)
        except Exception as e:
            self.log(f"Connect error [{e}]")
            return

        return resp

    def http_client_get_json(self, url):
        resp = self.http_client_get(url)
        try:
            resp_str = resp.read().decode()
            json_result = json.loads(resp_str)
        except json.decoder.JSONDecodeError:
            self.log(f"json decode error\nurl:{url}\n{resp_str}")
            return
        return json_result

    def http_get(self, url):
        try:
            resp = self._session.get(url, timeout=10)
        except requests.exceptions.InvalidURL:
            self.log(f'"{url}" is not valid url')
            return
        return resp

    def proxy_setup(self, session=None):
        if not session:
            session = self._session
        # 设置 User Agent
        session.headers.update(
            {
                "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.54 Safari/537.36"
            }
        )
        # 设置代理
        config = Config("config.ini")
        http = config.get("Proxy", "http")
        https = config.get("Proxy", "https")
        if http or https:
            proxys = {}
            if http:
                proxys["http"] = http
                os.environ["HTTP_PROXY"] = http
            if https:
                proxys["https"] = https
                os.environ["HTTPS_PROXY"] = https
            session.proxies.update(proxys)

    @staticmethod
    def urlparse(url: str) -> ParseResult:
        return urlparse(url)