Showing preview only (1,282K chars total). Download the full file or copy to clipboard to get everything.
Repository: KurtBestor/Hitomi-Downloader
Branch: master
Commit: d46e53ea3bde
Files: 113
Total size: 1.2 MB
Directory structure:
gitextract_gsyu7ov9/
├── .github/
│ └── stale.yml
├── .gitignore
├── FUNDING.yml
├── README.md
├── push_^q^.bat
├── src/
│ └── extractor/
│ ├── _4chan_downloader.py
│ ├── afreeca_downloader.py
│ ├── artstation_downloader.py
│ ├── asmhentai_downloader.py
│ ├── avgle_downloader.py
│ ├── baraag_downloader.py
│ ├── bcy_downloader.py
│ ├── bdsmlr_downloader.py
│ ├── bili_downloader.py
│ ├── coub_downloader.py
│ ├── danbooru_downloader.py
│ ├── discord_emoji_downloader.py
│ ├── etc_downloader.py
│ ├── fc2_downloader.py
│ ├── file_downloader.py
│ ├── flickr_downloader.py
│ ├── gelbooru_downloader.py
│ ├── hameln_downloader.py
│ ├── hanime_downloader.py
│ ├── hentaicosplay_downloader.py
│ ├── hf_downloader.py
│ ├── imgur_downloader.py
│ ├── iwara_downloader.py
│ ├── jmana_downloader.py
│ ├── kakaotv_downloader.py
│ ├── kakuyomu_downloader.py
│ ├── kissjav_downloader.py
│ ├── lhscan_downloader.py
│ ├── luscious_downloader.py
│ ├── m3u8_downloader.py
│ ├── manatoki_downloader.py
│ ├── mastodon_downloader.py
│ ├── misskey_downloader.py
│ ├── mrm_downloader.py
│ ├── naver_downloader.py
│ ├── navercafe_downloader.py
│ ├── naverpost_downloader.py
│ ├── navertoon_downloader.py
│ ├── navertv_downloader.py
│ ├── newgrounds_downloader.py
│ ├── nhentai_com_downloader.py
│ ├── nhentai_downloader.py
│ ├── nico_downloader.py
│ ├── nijie_downloader.py
│ ├── nozomi_downloader.py
│ ├── pawoo_downloader.py
│ ├── pinter_downloader.py
│ ├── pixiv_downloader.py
│ ├── pornhub_downloader.py
│ ├── rule34_xxx_downloader.py
│ ├── sankaku_downloader.py
│ ├── soundcloud_downloader.py
│ ├── syosetu_downloader.py
│ ├── talk_op_gg_downloader.py
│ ├── tiktok_downloader.py
│ ├── tokyomotion_downloader.py
│ ├── torrent_downloader.py
│ ├── tumblr_downloader.py
│ ├── twitch_downloader.py
│ ├── v2ph_downloader.py
│ ├── vimeo_downloader.py
│ ├── wayback_machine_downloader.py
│ ├── webtoon_downloader.py
│ ├── weibo_downloader.py
│ ├── wikiart_downloader.py
│ ├── xhamster_downloader.py
│ ├── xnxx_downloader.py
│ ├── xvideo_downloader.py
│ ├── yandere_downloader.py
│ ├── youku_downloader.py
│ ├── youporn_downloader.py
│ └── youtube_downloader.py
└── translation/
├── changelog_en.txt
├── changelog_ko.txt
├── help_en.html
├── help_ja.html
├── help_ko.html
├── help_pl.html
├── help_ru.html
├── help_si.html
├── help_zh.html
├── qt_ar.json
├── qt_es.json
├── qt_fr.json
├── qt_ja.json
├── qt_ko.json
├── qt_pl.json
├── qt_pt.json
├── qt_ru.json
├── qt_si.json
├── qt_tr.json
├── qt_vi.json
├── qt_zh-TW.json
├── qt_zh.json
├── tr_ar.hdl
├── tr_en.hdl
├── tr_es.hdl
├── tr_fr.hdl
├── tr_ja.hdl
├── tr_ko.hdl
├── tr_pl.hdl
├── tr_pt.hdl
├── tr_ru.hdl
├── tr_si.hdl
├── tr_tr.hdl
├── tr_vi.hdl
├── tr_zh-TW.hdl
└── tr_zh.hdl
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/stale.yml
================================================
# Configuration for probot-stale - https://github.com/probot/stale
# Number of days of inactivity before an Issue or Pull Request becomes stale
daysUntilStale: 90
# Number of days of inactivity before a stale Issue or Pull Request is closed
daysUntilClose: 30
# Issues or Pull Requests with these labels will never be considered stale
exemptLabels:
- help wanted
- notice
# Label to use when marking as stale
staleLabel: stale
# Comment to post when marking as stale. Set to `false` to disable
markComment: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed after 30 days if no further activity
occurs, but feel free to re-open a closed issue if needed.
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# Jupyter Notebook
.ipynb_checkpoints
# pyenv
.python-version
# celery beat schedule file
celerybeat-schedule
# SageMath parsed files
*.sage.py
# dotenv
.env
# virtualenv
.venv
venv/
ENV/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
# etc
#*.bat
call cmd.bat
================================================
FILE: FUNDING.yml
================================================
patreon: KurtBestor
================================================
FILE: README.md
================================================
<p align="center">
<img src="imgs/card_crop.png" width="50%"/>
<br>
</p>
[](https://github.com/KurtBestor/Hitomi-Downloader/releases/latest)
[](https://github.com/KurtBestor/Hitomi-Downloader/releases/latest)
[](https://github.com/KurtBestor/Hitomi-Downloader/releases)
## Links
- [Download](https://github.com/KurtBestor/Hitomi-Downloader/releases/latest)
- [Issues](https://github.com/KurtBestor/Hitomi-Downloader/issues)
- [Scripts & Plugins](https://github.com/KurtBestor/Hitomi-Downloader/wiki/Scripts-&-Plugins)
- [Chrome Extension](https://github.com/KurtBestor/Hitomi-Downloader/wiki/Chrome-Extension)
## Demo
<img src="imgs/how_to_download.gif">
## Features
- 🍰 Simple and clear user interface
- 🚀 Download acceleration
- 💻 Supports 24 threads in a single task
- 🚥 Supports speed limit
- 📜 Supports user scripts
- 🧲 Supports BitTorrent & Magnet
- 🎞️ Supports M3U8 & MPD format videos
- 🌙 Dark mode
- 🧳 Portable
- 📋 Clipboard monitor
- 🗃️ Easy to organize tasks
## Supported Sites
| Site | URL |
| :--: | -- |
| **4chan** | <https://4chan.org> |
| **AfreecaTV** | <https://afreecatv.com> |
| **ArtStation** | <https://artstation.com> |
| **baraag.net** | <https://baraag.net> |
| **bilibili** | <https://bilibili.com> |
| **ComicWalker** | <https://comic-walker.com> |
| **Coub** | <https://coub.com> |
| **DeviantArt** | <https://deviantart.com> |
| **Facebook** | <https://facebook.com> |
| **FC2 Video** | <https://video.fc2.com> |
| **Flickr** | <https://flickr.com> |
| **Hameln** | <https://syosetu.org> |
| **Imgur** | <https://imgur.com> |
| **Instagram** | <https://instagram.com> |
| **カクヨム** | <https://kakuyomu.jp> |
| **Mastodon** | <https://mastodon.social> |
| **Misskey** | <https://misskey.io> |
| **Naver Blog** | <https://blog.naver.com> |
| **Naver Cafe** | <https://cafe.naver.com> |
| **Naver Post** | <https://post.naver.com> |
| **Naver Webtoon** | <https://comic.naver.com> |
| **Naver TV** | <https://tv.naver.com> |
| **Niconico** | <http://nicovideo.jp> |
| **Pawoo** | <https://pawoo.net> |
| **Pinterest** | <https://pinterest.com> |
| **Pixiv** | <https://pixiv.net> |
| **pixivコミック** | <https://comic.pixiv.net> |
| **Soundcloud** | <https://soundcloud.com> |
| **小説家になろう** | <https://syosetu.com> |
| **TikTok** | <https://tiktok.com><br><https://douyin.com>|
| **Tumblr** | <https://tumblr.com> |
| **Twitch** | <https://twitch.tv> |
| **Twitter** | <https://twitter.com> |
| **Vimeo** | <https://vimeo.com> |
| **Wayback Machine** | <https://archive.org> |
| **Weibo** | <https://weibo.com> |
| **WikiArt** | <https://www.wikiart.org> |
| **Youku** | <https://youku.com> |
| **YouTube** | <https://youtube.com> |
| **and more...** | [Supported sites by yt-dlp](https://github.com/yt-dlp/yt-dlp/blob/master/supportedsites.md) |
================================================
FILE: push_^q^.bat
================================================
@echo off
git add .
git commit -m "^q^"
git push
echo Done!
pause>nul
================================================
FILE: src/extractor/_4chan_downloader.py
================================================
import downloader
from utils import Downloader, File, clean_title, urljoin, get_ext, limits
import utils
class File_4chan(File):
type = '4chan'
format = 'page:04;'
@limits(.5)
def get(self):
return {}
class Downloader_4chan(Downloader):
type = '4chan'
URLS = [r'regex:boards.(4chan|4channel).org']
MAX_CORE = 4
display_name = '4chan'
ACCEPT_COOKIES = [r'(.*\.)?(4chan|4channel)\.org']
@classmethod
def fix_url(cls, url):
return url.split('#')[0]
def read(self):
soup = downloader.read_soup(self.url)
for div in soup.findAll('div', class_='fileText'):
href = urljoin(self.url, div.a['href'])
d = {
'page': len(self.urls),
}
file = File_4chan({'url': href, 'referer': self.url, 'name': utils.format('4chan', d, get_ext(href))})
self.urls.append(file)
board = self.url.split('/')[3]
title = soup.find('span', class_='subject').text
id_ = int(self.url.split('/thread/')[1].split('/')[0])
self.title = clean_title(f'[{board}] {title} ({id_})')
================================================
FILE: src/extractor/afreeca_downloader.py
================================================
import downloader
from utils import Soup, Downloader, Session, try_n, format_filename, cut_pair, File, get_print, print_error, json
import ree as re
from io import BytesIO
from m3u8_tools import playlist2stream, M3u8_stream
import errors
import utils
import os
import dateutil.parser
class LoginRequired(errors.LoginRequired):
def __init__(self, *args):
super().__init__(*args, method='browser', url='https://login.sooplive.co.kr/afreeca/login.php')
class Downloader_afreeca(Downloader):
type = 'afreeca'
URLS = ['afreecatv.com', 'sooplive.co.kr']
single = True
display_name = 'SOOP'
ACCEPT_COOKIES = [r'(.*\.)?afreecatv\.com', r'(.*\.)?sooplive\.co\.kr']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
if Live_afreeca.is_live(url):
url = Live_afreeca.fix_url(url)
return url.rstrip(' /')
def read(self):
video = Video({'referer': self.url})
video.ready(self.cw)
self.urls.append(video)
thumb = BytesIO()
downloader.download(video['url_thumb'], buffer=thumb)
self.setIcon(thumb)
self.title = os.path.splitext(video['name'])[0].replace('：', ':')
self.artist = video['artist']
if video['live']:
d = {}
d['url'] = self.url
d['title'] = self.artist
d['thumb'] = thumb.getvalue()
utils.update_live(d, self.cw)
@try_n(4)
def _get_stream(url_m3u8, session, referer, cw=None):
print_ = get_print(cw)
print_(f'_get_stream: {url_m3u8}')
try:
stream = playlist2stream(url_m3u8, referer=referer, session=session)
except Exception as e:
print_(print_error(e))
stream = M3u8_stream(url_m3u8, referer=referer, session=session)
return stream
class Video(File):
type = 'afreeca'
_live_info = None
def get(self):
print_ = get_print(self.cw)
url, session = self['referer'], self.session
if session is None:
session = Session()
session.purge('afreeca')
html = downloader.read_html(url, session=session)
if "document.location.href='https://login." in html:
raise LoginRequired()
if len(html) < 2000:
alert = re.find(r'''alert\(['"](.+?)['"]\)''', html)
if alert:
raise LoginRequired(alert)
soup = Soup(html)
date = None
url_thumb = soup.find('meta', {'property': 'og:image'}).attrs['content']
print_('url_thumb: {}'.format(url_thumb))
vid = re.find('/player/([0-9]+)', url)
if vid is None: # live
bid = re.find('sooplive.co.kr/([^/]+)', url, err='no bid')
url_api = f'https://st.sooplive.co.kr/api/get_station_status.php?szBjId={bid}'
r = session.post(url_api, headers={'Referer': url})
d = json.loads(r.text)
artist = d['DATA']['user_nick']
if self._live_info is not None:
self._live_info['title'] = artist
url_api = f'https://live.sooplive.co.kr/afreeca/player_live_api.php?bjid={bid}'
#bno = re.find('sooplive.co.kr/[^/]+/([0-9]+)', url, err='no bno')
bno = re.find(r'nBroadNo\s=\s([0-9]+)', html, err='no bno') #6915
r = session.post(url_api, data={'bid': bid, 'bno': bno, 'type': 'aid', 'pwd': '', 'player_type': 'html5', 'stream_type': 'common', 'quality': 'master', 'mode': 'landing', 'from_api': '0'}, headers={'Referer': url})
d = json.loads(r.text)
res = d['CHANNEL'].get('RESULT')
print_(f'result: {res}')
if res == -6:
raise LoginRequired()
aid = d['CHANNEL']['AID']
data = {}
data['title'] = soup.find('meta', {'property': 'og:title'})['content'].strip()
data['files'] = [{'file': f'https://pc-web.stream.sooplive.co.kr/live-stm-16/auth_master_playlist.m3u8?aid={aid}'}]
data['writer_nick'] = artist
data['live'] = True
elif f'{vid}/catch' in url: #6215
url_api = 'https://api.m.sooplive.co.kr/station/video/a/catchview'
r = session.post(url_api, data={'nPageNo': '1', 'nLimit': '10', 'nTitleNo': vid}, headers={'Referer': url})
try:
s = cut_pair(r.text)
d = json.loads(s)
except Exception as e:
print_(r.text)
raise e
data = d['data'][0]
date = dateutil.parser.parse(data['reg_date']) #7054
else:
url_api = 'https://api.m.sooplive.co.kr/station/video/a/view'
r = session.post(url_api, data={'nTitleNo': vid, 'nApiLevel': '10', 'nPlaylistIdx': '0'}, headers={'Referer': url})
try:
s = cut_pair(r.text)
d = json.loads(s)
except Exception as e:
print_(r.text)
raise e
data = d['data']
date = dateutil.parser.parse(data.get('broad_start') or data['write_tm']) #7054, #7093
title = data.get('full_title') or data['title']
artist = data.get('copyright_nickname') or data.get('original_user_nick') or data['writer_nick']
if data.get('adult_status') == 'notLogin':
raise LoginRequired(title)
urls_m3u8 = []
for file in data['files']:
if file.get('quality_info'):
file = file['quality_info'][0]['file']
else:
file = file['file']
urls_m3u8.append(file)
print_(f'urls_m3u8: {len(urls_m3u8)}')
if data.get('live'):
stream = playlist2stream(urls_m3u8[0], url, session=session) #6934
if stream.ms:
stream = stream.live
stream._cw = self.cw
if not stream.live:
stream.live = True#
else:
streams = []
for url_m3u8 in urls_m3u8:
try:
stream = _get_stream(url_m3u8, session, url, cw=self.cw)
except Exception as e:
print_(print_error(e))
continue #2193
streams.append(stream)
for stream in streams[1:]:
streams[0] += stream
stream = streams[0]
stream.live = None#
live = data.get('live') or False
return {'url': stream, 'title': title, 'name': format_filename(title, vid, '.mp4', artist=artist, live=live, date=date), 'url_thumb': url_thumb, 'artist': artist, 'live': live}
class Live_afreeca(utils.Live):
type = 'afreeca'
@classmethod
def is_live(cls, url):
return bool(re.match(r'https?://(play|bj|ch).(afreecatv.com|sooplive.co.kr)/([^/?#]+)', url)) and url.strip('/').count('/') <= 4
@classmethod
def fix_url(cls, url):
bj = re.find(r'https?://(play|bj|ch).(afreecatv.com|sooplive.co.kr)/([^/?#]+)', url)[2]
return f'https://play.sooplive.co.kr/{bj}'
@classmethod
def check_live(cls, url, info=None):
try:
video = Video({'referer': url})
video._live_info = info
video.ready(None)
return True
except Exception as e:
print(e)
return False
================================================
FILE: src/extractor/artstation_downloader.py
================================================
#coding:utf8
import downloader
from error_printer import print_error
from translator import tr_
from utils import Downloader, Soup, get_print, lazy, Session, try_n, File, clean_title, check_alive, get_ext, get_max_range
import dateutil.parser
import utils
class File_artstation(File):
type = 'artstation'
format = '[date] name_ppage'
c_alter = 0
def alter(self): #6401
self.c_alter += 1
if self.c_alter % 2 == 0:
url = self['url']
else:
url = self['url'].replace('/4k/', '/large/')
return url
class Downloader_artstation(Downloader):
type = 'artstation'
URLS = ['artstation.com']
display_name = 'ArtStation'
ACCEPT_COOKIES = [r'(.*\.)?artstation\.(com|co)']
url_main = None
@try_n(8)
def init(self):
# 3849
self.session = Session()
import clf2
clf2.solve(self.url, self.session, self.cw)
_ = self._id.replace('artstation_', '', 1)
self.url_main = f'https://www.artstation.com/{_}'
if '/artwork/' in self.url or '/projects/' in self.url:
pass
else:
self.url = self.url_main
self.print_(self.url)
@classmethod
def fix_url(cls, url): #6516
if '.artstation.com' in url:
sub = url.split('.artstation.com')[0].split('/')[-1]
if sub != 'www':
url = f'https://www.artstation.com/{sub}'
return url
@lazy
def _id(self):
_id = get_id(self.url, self.cw)
return f'artstation_{_id}'
@lazy
@try_n(2)
def name(self):
soup = downloader.read_soup(self.url_main, session=self.session)
name = soup.find('meta', {'property': 'og:title'}).attrs['content']
return clean_title(f'{name} ({self._id})')
def read(self):
self.title = self.name
id_ = self._id.replace('artstation_', '', 1)
if '/' in id_:
id_ = id_.split('/')[0]
if '/artwork/' in self.url or '/projects/' in self.url:
id_art = get_id_art(self.url)
imgs = get_imgs_page(id_art, self.session, cw=self.cw)
else:
imgs = get_imgs(id_, self.title, self.session, cw=self.cw)
for img in imgs:
self.urls.append(img)
self.title = self.name
@try_n(2)
def get_imgs(id_, title, session, cw=None):
referer = f'https://www.artstation.com/{id_}'
downloader.read_html(referer, session=session)
#print(session.cookies.keys())
url = f'https://www.artstation.com/users/{id_}/quick.json'
j = downloader.read_json(url, referer, session=session)
uid = j['id']
datas = []
ids = set()
for p in range(1, 1000):
check_alive(cw)
url = f'https://www.artstation.com/users/{id_}/projects.json??user_id={uid}&page={p}' #6516
j = try_n(4)(downloader.read_json)(url, referer, session=session)
data = j['data']
if not data:
break
for d in data:
if d['id'] not in ids:
ids.add(d['id'])
datas.append(d)
if cw:
cw.setTitle(f'{tr_("페이지 읽는 중...")} {title} - {len(datas)}')
else:
print(len(datas))
datas = sorted(datas, key=lambda data: int(data['id']), reverse=True)
imgs = []
i = 0
names = set()
while i < len(datas):
check_alive(cw)
data = datas[i]
date = data['created_at']
post_url = data['permalink']
#print('post_url', post_url)
id_art = get_id_art(post_url)
imgs += get_imgs_page(id_art, session, date=date, cw=cw, names=names)
if len(imgs) >= get_max_range(cw):
break
if cw:
cw.setTitle(f'{tr_("이미지 읽는 중...")} {title} - {i+1} / {len(datas)} ({len(imgs)})')
else:
print(len(imgs))
i += 1
return imgs
def get_id_art(post_url):
return post_url.split('/artwork/')[-1].split('/projects/')[-1].split('/')[0].split('?')[0].split('#')[0]
def get_id(url, cw=None):
print_ = get_print(cw)
url = url.split('?')[0].split('#')[0]
if '/artwork/' in url:
id_art = get_id_art(url)
imgs = get_imgs_page(id_art, session=Session(), cw=cw)
return imgs[0].data['user']['username']
if '.artstation.' in url and 'www.artstation.' not in url:
id_ = url.split('.artstation')[0].split('//')[-1]
type_ = None
elif 'artstation.com' in url:
paths = url.split('artstation.com/')[1].split('/')
id_ = paths[0]
type_ = paths[1] if len(paths) > 1 else None
else:
id_ = url.replace('artstation_', '').replace('／', '/')
type_ = None
if type_ not in [None, 'likes']:
type_ = None
print_(f'type: {type_}, id: {id_}')
if type_:
return f'{id_}/{type_}'
return id_
def get_imgs_page(id_art, session, date=None, cw=None, names=None):
print_ = get_print(cw)
url_json = f'https://www.artstation.com/projects/{id_art}.json'
post_url = f'https://www.artstation.com/artwork/{id_art}'
name = post_url.strip('/').split('/')[-1]
if names is not None:
while name.lower() in names:
name += '_'
names.add(name.lower())
try:
data = downloader.read_json(url_json, session=session, referer=post_url)
imgs_ = data['assets']
except Exception as e:
print_(print_error(e))
return []
if date is None:
date = data['created_at']
date = dateutil.parser.parse(date)
imgs = []
for page, img in enumerate(imgs_):
if not img['has_image']:
print('no img')
continue
url = None
embed = img.get('player_embedded')
if embed:
soup = Soup(embed)
url_embed = soup.find('iframe').attrs['src']
print_(f'embed: {url_embed}')
try:
soup = downloader.read_soup(url_embed, post_url, session=session)
v = soup.find('video')
if v:
url = v.find('source').attrs['src']
except Exception as e:
print_(print_error(e))
if not url:
try:
url = soup.find('link', {'rel': 'canonical'}).attrs['href']
print_(f'YouTube: {url}')
raise Exception('YouTube')
except Exception as e:
print(e)
url = None
if not url:
url = img['image_url']
d = {
'date': date,
'name': clean_title(name),
'page': page,
}
filename = utils.format('artstation', d, get_ext(url))
img = File_artstation({'referer':post_url, 'url':url.replace('/large/', '/4k/'), 'name': filename})
img.data = data
imgs.append(img)
return imgs
================================================
FILE: src/extractor/asmhentai_downloader.py
================================================
#coding: utf8
import downloader
import ree as re
from utils import Soup, urljoin, Downloader, join, Session, File, clean_title, limits
import os
import utils
def get_id(url):
try:
return int(url)
except:
return int(re.find('/(g|gallery)/([0-9]+)', url)[1])
class File_asmhentai(File):
type = 'asmhentai'
format = 'name'
@limits(.25)
def get(self):
soup = downloader.read_soup(self['referer'], self['rereferer'], session=self.session)
img = soup.find('img', id='fimg')
url = img['data-src']
name, ext = os.path.splitext(os.path.basename(url).split('?')[0])
d = {
'name': clean_title(name),
}
return {'url': url, 'name': utils.format('asmhentai', d, ext)}
class Downloader_asmhentai(Downloader):
type = 'asmhentai'
URLS = ['asmhentai.com']
MAX_CORE = 8
display_name = 'AsmHentai'
ACCEPT_COOKIES = [r'(.*\.)?asmhentai\.com']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
id_ = get_id(url)
return f'https://asmhentai.com/g/{id_}/'
def read(self):
info = get_info(self.url, self.session, self.cw)
self.print_(info)
# 1225
artist = join(info['artist'])
self.artist = artist
group = join(info['group']) if info['group'] else 'N／A'
lang = info['language'][0] if info['language'] else 'N／A'
series = info['parody'][0] if info['parody'] else 'N／A'
title = self.format_title(info['category'][0], info['id'], info['title'], artist, group, series, lang)
for i in range(info['n']):
url = f'https://asmhentai.com/gallery/{info["id"]}/{i+1}/'
file = File_asmhentai({'referer':url, 'rereferer': self.url})
self.urls.append(file)
self.title = title
def get_info(url, session, cw):
html = downloader.read_html(url, session=session)
soup = Soup(html)
info = {}
info['id'] = get_id(url)
title = soup.find('h1').text.strip()
info['title'] = title
for tag in soup.findAll('span', class_='tag'):
href = tag.parent.attrs['href']
href = urljoin(url, href).strip('/')
key = href.split('/')[3]
value = href.split('/')[-1]
if key == 'language' and value == 'translated':
continue
if key in info:
info[key].append(value)
else:
info[key] = [value]
for key in ['artist', 'group', 'parody', 'tag', 'character']:
if key not in info:
info[key] = []
info['n'] = int(soup.find('input', id='t_pages')['value'])
return info
================================================
FILE: src/extractor/avgle_downloader.py
================================================
#coding: utf8
import downloader
from m3u8_tools import M3u8_stream
from utils import Soup, Downloader, LazyUrl, get_print, try_n, check_alive, format_filename, json
from io import BytesIO
import base64
import webbrowser
import errors
class Downloader_avgle(Downloader):
type = 'avgle'
single = True
URLS = ['avgle.com']
ACCEPT_COOKIES = [r'(.*\.)?avgle\.com']
def init(self):
if not self.cw.data_:
link = 'https://github.com/KurtBestor/Hitomi-Downloader/wiki/Chrome-Extension'
webbrowser.open(link)
raise errors.Invalid('No data; See: {}'.format(link))
def read(self):
video = get_video(self.url, cw=self.cw)
self.urls.append(video.url)
self.setIcon(video.thumb)
self.title = video.title
@try_n(2)
def get_video(url, cw=None):
print_ = get_print(cw)
check_alive(cw)
data = cw.data_
version = data['version']
print_('version: {}'.format(version))
if version == '0.1':
raise errors.OutdatedExtension()
data = data['data']
if not isinstance(data, bytes):
data = data.encode('utf8')
s = base64.b64decode(data).decode('utf8')
urls = json.loads(s)
print_('\n'.join(urls[:4]))
referer_seg = 'auto' if 'referer=force' in urls[0] else None # 1718
stream = M3u8_stream(url, urls=urls, n_thread=4, referer_seg=referer_seg)
html = downloader.read_html(url)
soup = Soup(html)
url_thumb = soup.find('meta', {'property': 'og:image'}).attrs['content']
title = soup.find('meta', {'property': 'og:title'}).attrs['content'].strip()
video = Video(stream, url_thumb, url, title)
return video
class Video:
def __init__(self, url, url_thumb, referer, title):
self.url = LazyUrl(referer, lambda x: url, self)
self.url_thumb = url_thumb
self.thumb = BytesIO()
downloader.download(url_thumb, referer=referer, buffer=self.thumb)
self.title = title
self.filename = format_filename(title, '', '.mp4')
================================================
FILE: src/extractor/baraag_downloader.py
================================================
#coding:utf8
from utils import Downloader, clean_title, Session
from mastodon import get_info
import ree as re
def get_id(url):
return re.find('baraag.net/([^/]+)', url.lower())
class Downloader_baraag(Downloader):
type = 'baraag'
URLS = ['baraag.net']
display_name = 'baraag.net'
ACCEPT_COOKIES = [r'(.*\.)?baraag\.net']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
id_ = get_id(url) or url
return f'https://baraag.net/{id_}'
def read(self):
id_ = get_id(self.url)
info = get_info('baraag.net', id_, f'baraag_{id_}', self.session, self.cw)
self.urls += info['files']
self.title = clean_title('{} (baraag_{})'.format(info['title'], id_))
================================================
FILE: src/extractor/bcy_downloader.py
================================================
#coding:utf8
import downloader
from utils import Soup, cut_pair, LazyUrl, Downloader, get_print, get_max_range, try_n, clean_title, check_alive, json
import os
from translator import tr_
class Downloader_bcy(Downloader):
type = 'bcy'
URLS = ['bcy.net/item/detail/', 'bcy.net/u/']
MAX_CORE = 8
display_name = '半次元'
ACCEPT_COOKIES = [r'(.*\.)?bcy\.net']
def init(self):
self.html = downloader.read_html(self.url)
self.info = get_info(self.url, self.html)
@property
def name(self):
info = self.info
if '/detail/' in self.url:
title = '{} (bcy_{}) - {}'.format(clean_title(info['artist']), info['uid'], info['id'])
else:
title = '{} (bcy_{})'.format(clean_title(info['artist']), info['uid'])
return title
def read(self):
imgs = get_imgs(self.url, self.html, cw=self.cw)
for img in imgs:
self.urls.append(img.url)
self.title = self.name
self.artist = self.info['artist']
def get_ssr_data(html):
s = html.split('window.__ssr_data = JSON.parse("')[1].replace('\\"', '"')
s = cut_pair(s).replace('"', '\\"')
data = json.loads(json.loads('"{}"'.format(s)))
return data
@try_n(2)
def get_imgs(url, html=None, cw=None):
if '/detail/' not in url:
return get_imgs_channel(url, html, cw)
if html is None:
html = downloader.read_html(url)
data = get_ssr_data(html)
multi = data['detail']['post_data']['multi']
imgs = []
for m in multi:
path = m['original_path']
img = json.loads('"{}"'.format(path))
img = Image_single(img, url, len(imgs))
imgs.append(img)
return imgs
class Image_single:
def __init__(self, url ,referer, p):
self._url = url
self.p = p
self.url = LazyUrl(referer, self.get, self)
def get(self, referer):
ext = get_ext(self._url, referer)
self.filename = '{:04}{}'.format(self.p, ext)
return self._url
class Image:
def __init__(self, url, referer, id, p):
self.id = id
self.p = p
self._url = url
self.url = LazyUrl(referer, self.get, self)
def get(self, referer):
ext = get_ext(self._url, referer)
self.filename = '{}_p{}{}'.format(self.id, self.p, ext)
return self._url
def get_ext(url, referer=None):
ext = os.path.splitext(url.split('?')[0].replace('~noop.image', ''))[1]
if ext in ['.image', '']:
ext = downloader.get_ext(url, referer=referer)
return ext
def get_info(url, html):
soup = Soup(html)
info = {}
uname = soup.find('div', class_='user-name') or soup.find('p', class_='uname') or soup.find('div', class_='user-info-name')
info['artist'] = uname.text.strip()
j = get_ssr_data(html)
if '/detail/' in url:
info['uid'] = j['detail']['detail_user']['uid']
info['id'] = j['detail']['post_data']['item_id']
else:
info['uid'] = j['homeInfo']['uid']
return info
def get_imgs_channel(url, html=None, cw=None):
print_ = get_print(cw)
if html is None:
html = downloader.read_html(url)
info = get_info(url, html)
# Range
max_pid = get_max_range(cw)
ids = set()
imgs = []
for p in range(1000):
url_api = 'https://bcy.net/apiv3/user/selfPosts?uid={}'.format(info['uid'])
if imgs:
url_api += '&since={}'.format(imgs[-1].id)
data_raw = downloader.read_html(url_api, url)
data = json.loads(data_raw)['data']
items = data['items']
if not items:
print('no items')
break
c = 0
for item in items:
check_alive(cw)
id = item['item_detail']['item_id']
if id in ids:
print('duplicate')
continue
c += 1
ids.add(id)
url_single = 'https://bcy.net/item/detail/{}'.format(id)
imgs_single = get_imgs(url_single, cw=cw)
print_(str(id))
for p, img in enumerate(imgs_single):
img = Image(img._url, url_single, id, p)
imgs.append(img)
s = '{} {} - {}'.format(tr_('읽는 중...'), info['artist'], min(len(imgs), max_pid))
if cw:
cw.setTitle(s)
else:
print(s)
if len(imgs) >= max_pid:
break
if not c:
print('not c')
break
if len(imgs) >= max_pid:
print('over max_pid:', max_pid)
break
return imgs[:max_pid]
================================================
FILE: src/extractor/bdsmlr_downloader.py
================================================
#coding:utf8
import downloader
from utils import Session, Soup, LazyUrl, Downloader, get_max_range, try_n, get_print, clean_title, check_alive
from datetime import datetime
import ree as re
import os
from translator import tr_
from error_printer import print_error
import clf2
import errors
class Downloader_bdsmlr(Downloader):
type = 'bdsmlr'
URLS = ['bdsmlr.com']
display_name = 'BDSMlr'
ACCEPT_COOKIES = [r'(.*\.)?bdsmlr\.com']
def init(self):
if 'bdsmlr.com/post/' in self.url:
raise errors.Invalid(tr_('개별 다운로드는 지원하지 않습니다: {}').format(self.url))
self.url = 'https://{}.bdsmlr.com'.format(self.id_)
self.session = Session()
clf2.solve(self.url, session=self.session, cw=self.cw)
@property
def id_(self):
url = self.url
if 'bdsmlr.com' in url:
if 'www.bdsmlr.com' in url:
raise Exception('www.bdsmlr.com')
gal_num = url.split('.bdsmlr.com')[0].split('/')[-1]
else:
gal_num = url
return gal_num
def read(self):
info = get_imgs(self.id_, session=self.session, cw=self.cw)
for post in info['posts']:
self.urls.append(post.url)
self.title = '{} (bdsmlr_{})'.format(clean_title(info['username']), self.id_)
class Post:
def __init__(self, url, referer, id, p):
self.id = id
self.url = LazyUrl(referer, lambda x: url, self)
ext = os.path.splitext(url)[1]
self.filename = '{}_p{}{}'.format(id, p, ext)
def foo(url, soup, info, reblog=False):
#print('foo', info['c'], len(info['ids']))
for post in soup.findAll('div', class_='wrap-post'):
try:
id = int(re.find('[0-9]+', post.attrs['class'][1]))
except Exception as e:
print(print_error(e))
continue
if id in info['ids']:
continue
info['ids'].add(id)
info['last'] = id
if not reblog and post.find('div', class_='ogname'):
continue
for p, mag in enumerate(post.findAll(['a', 'div'], class_='magnify')):
post = Post(mag.attrs['href'], url, id, p)
info['posts'].append(post)
info['c'] += 20 if info['c'] else 5
@try_n(2)
def get_imgs(user_id, session, cw=None):
print_ = get_print(cw)
url = 'https://{}.bdsmlr.com/'.format(user_id)
info = {'c': 0, 'posts': [], 'ids': set()}
html = downloader.read_html(url, session=session)
soup = Soup(html)
sorry = soup.find('div', class_='sorry')
if sorry:
raise Exception(sorry.text.strip())
username = soup.find('title').text.strip()###
print('username:', username)
info['username'] = username
token = soup.find('meta', {'name': 'csrf-token'}).attrs['content']
print_('token: {}'.format(token))
max_pid = get_max_range(cw)
n = len(info['ids'])
for p in range(1000):
check_alive(cw)
if p == 0:
url_api = 'https://{}.bdsmlr.com/loadfirst'.format(user_id)
else:
url_api = 'https://{}.bdsmlr.com/infinitepb2/{}'.format(user_id, user_id)
data = {
'scroll': str(info['c']),
'timenow': datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
}
if 'last' in info:
data['last'] = str(info['last'])
print_('n:{}, scroll:{}, last:{}'.format(len(info['posts']), data['scroll'], data.get('last')))
headers = {
'Referer': url,
'X-CSRF-TOKEN': token,
}
_e = None
for try_ in range(4):
try:
r = session.post(url_api, data=data, headers=headers)
if p == 0:
r.raise_for_status()
break
except Exception as e:
_e = e
print(e)
else:
if _e is not None:
raise _e
soup = Soup(r.text)
foo(url, soup, info)
if len(info['ids']) == n:
print('same; break')
break
n = len(info['ids'])
s = '{} {} (tumblr_{}) - {}'.format(tr_('읽는 중...'), username, user_id, len(info['posts']))
if cw is not None:
cw.setTitle(s)
else:
print(s)
if len(info['posts']) > max_pid:
break
return info
================================================
FILE: src/extractor/bili_downloader.py
================================================
import downloader
import downloader_v3
from utils import Downloader, get_print, format_filename, clean_title, get_resolution, try_n, Session, uuid, File, get_max_range, query_url
import os
from io import BytesIO
import ffmpeg
import math
import ree as re
import ytdl
import constants
import putils
import threading
import errors
_VALID_URL = r'''(?x)
https?://
(?:(?:www|bangumi)\.)?
bilibili\.(?:tv|com)/
(?:
(?:
video/[aA][vV]|
anime/(?P<anime_id>\d+)/play\#
)(?P<id_bv>\d+)|
video/[bB][vV](?P<id>[^/?#&]+)
)
'''
class File_bili(File):
type = 'bili'
thread_audio = None
@try_n(4)
def get(self):
session = self.session
cw = self.cw
print_ = get_print(cw)
options = {
#'noplaylist': True, #5562
#'extract_flat': True,
'playlistend': 1,
}
ydl = ytdl.YoutubeDL(options, cw=cw)
info = ydl.extract_info(self['referer'])
#5562
entries = info.get('entries')
if entries:
info.update(entries[0])
fs = info['formats']
res = max(get_resolution(), min(f.get('height', 0) for f in fs))
print_(f'res: {res}')
fs = [f for f in fs if f.get('height', 0) <= res]
for f in fs:
print_(f"{f['format']} - {f['url']}")
f_video = sorted(fs, key=lambda f:(f.get('height', 0), f.get('vbr', 0)))[-1]
print_('video: {}'.format(f_video['format']))
if f_video.get('abr'):
f_audio = None
else:
fs_audio = sorted([f_audio for f_audio in fs if f_audio.get('abr')], key=lambda f:f['abr'])
if fs_audio:
f_audio = fs_audio[-1]
else:
raise Exception('no audio')
print_('audio: {}'.format(f_audio['format']))
title = info['title']
url_thumb = info['thumbnail']
ext = info['ext']
session.headers.update(info.get('http_headers', {}))
mobj = re.match(_VALID_URL, self['referer'])
video_id = mobj.group('id')
artist = info.get('uploader') or None
info = {
'url': f_video['url'],
'url_thumb': url_thumb,
'artist': artist,
'name': format_filename(title, video_id, ext, artist=artist), #7127
}
if f_audio:
def f():
audio = f_audio['url']
path = os.path.join(putils.DIRf, f'{uuid()}_a.tmp')
if cw is not None:
cw.trash_can.append(path)
if constants.FAST:
downloader_v3.download(audio, session=self.session, chunk=1024*1024, n_threads=2, outdir=os.path.dirname(path), fileName=os.path.basename(path), customWidget=cw, overwrite=True)
else:
downloader.download(audio, session=self.session, outdir=os.path.dirname(path), fileName=os.path.basename(path), customWidget=cw, overwrite=True)
self.audio_path = path
print_('audio done')
self.thread_audio = threading.Thread(target=f, daemon=True)
self.thread_audio.start()
return info
def pp(self, filename):
if self.thread_audio:
self.thread_audio.join()
ffmpeg.merge(filename, self.audio_path, cw=self.cw)
return filename
# 1804
@try_n(2)
def fix_url(url, cw=None):
print_ = get_print(cw)
if '?' in url:
tail = url.split('?')[1]
else:
tail = None
soup = downloader.read_soup(url, methods={'requests'})
err = soup.find('div', class_='error-text')
if err:
raise errors.Invalid('{}: {}'.format(err.text.strip(), url))
meta = soup.find('meta', {'itemprop': 'url'})
if meta:
url_new = meta.attrs['content']
if tail:
url_new = '{}?{}'.format(url_new, tail)
print_('redirect: {} -> {}'.format(url, url_new))
else:
url_new = url
print_('no redirect')
return url_new
class Downloader_bili(Downloader):
type = 'bili'
URLS = [r'regex:'+_VALID_URL, 'space.bilibili.com/']
lock = True
detect_removed = False
detect_local_lazy = False
display_name = 'bilibili'
single = True
ACCEPT_COOKIES = [r'(.*\.)?bilibili\.com']
def init(self):
self.url = fix_url(self.url, self.cw)
if 'bilibili.com' not in self.url.lower():
self.url = 'https://www.bilibili.com/video/{}'.format(self.url)
self.url = self.url.replace('m.bilibili', 'bilibili')
self.session = Session()
@classmethod
def key_id(cls, url):
mobj = re.match(_VALID_URL, url)
video_id = mobj.group('id')
qs = query_url(url)
p = qs.get('p', ['1'])[0] #6580
return f'{video_id or url} {p}'
@property
def id_(self):
mobj = re.match(_VALID_URL, self.url)
video_id = mobj.group('id')
#anime_id = mobj.group('anime_id')
return video_id
def read(self):
sd = self.session.cookies.get('SESSDATA', domain='.bilibili.com')
self.print_('sd: {}'.format(sd))
if not sd: #5647
self.cw.showCookie()
self.cw.showLogin('https://passport.bilibili.com/login', 1030, None)
sid = re.find(r'/channel/collectiondetail?sid=([0-9]+)', self.url)
mid = re.find(r'space.bilibili.com/([0-9]+)', self.url)
if sid or mid:
if not sd:
raise errors.LoginRequired()
if sid:
url_api = f'https://api.bilibili.com/x/polymer/web-space/seasons_archives_list?mid={mid}&season_id={sid}'
j = downloader.read_json(url_api, self.url)
title = clean_title(j['data']['meta']['name'])
elif mid:
url_api = f'https://api.bilibili.com/x/space/wbi/acc/info?mid={mid}'
j = downloader.read_json(url_api, self.url)
title = clean_title(j['data']['name'])
else:
raise NotImplementedError()
self.single = False
options = {
'extract_flat': True,
'playlistend': get_max_range(self.cw),
}
ydl = ytdl.YoutubeDL(options, cw=self.cw)
info = ydl.extract_info(self.url)
files = []
for e in info['entries']:
files.append(File_bili({'referer': e['url']}))
self.print_(f'urls: {len(files)}')
file = self.process_playlist(title, files)
self.title = title
else:
file = File_bili({'referer': self.url})
file.ready(self.cw)
self.urls.append(file)
self.title = os.path.splitext(file['name'])[0]
thumb = BytesIO()
downloader.download(file['url_thumb'], buffer=thumb)
self.setIcon(thumb)
n = int(math.ceil(8.0 / len(self.urls)))
self.print_(f'n_threads: {n}')
self.enableSegment(n_threads=n, overwrite=True)
self.artist = file['artist']
================================================
FILE: src/extractor/coub_downloader.py
================================================
from utils import Downloader, LazyUrl, try_n, format_filename, get_ext
import ytdl
from io import BytesIO as IO
import downloader
import ree as re
PATTEN_IMAGIZER = r'coub-com-.+\.imagizer\.com'
def get_id(url):
return re.find(r'/view/([0-9a-z]+)', url, err='no id')
class Downloader_coub(Downloader):
type = 'coub'
URLS = ['coub.com', r'regex:'+PATTEN_IMAGIZER]
single = True
ACCEPT_COOKIES = [r'(.*\.)?coub\.com']
@classmethod
def fix_url(cls, url):
return re.sub(PATTEN_IMAGIZER, 'coub.com', url)
@classmethod
def key_id(cls, url):
return get_id(url)
def read(self):
video = Video(self.url, cw=self.cw)
video.url()#
self.urls.append(video.url)
self.setIcon(video.thumb)
self.enableSegment()
self.title = video.title
class Video:
_url = None
def __init__(self, url, cw=None):
self.url = LazyUrl(url, self.get, self, pp=self.pp)
self.cw = cw
@try_n(2)
def get(self, url):
if self._url:
return self._url
ydl = ytdl.YoutubeDL(cw=self.cw)
info = ydl.extract_info(url)
fs = [f for f in info['formats'] if f['ext'] == 'mp4']
f = sorted(fs, key=lambda f: int(f.get('filesize', 0)))[-1]
self._url = f['url']
## fs = [f for f in info['formats'] if f['ext'] == 'mp3']
## self.f_audio = sorted(fs, key=lambda f: int(f.get('filesize', 0)))[-1]
self.thumb_url = info['thumbnails'][0]['url']
self.thumb = IO()
downloader.download(self.thumb_url, buffer=self.thumb)
self.title = info['title']
ext = get_ext(self._url)
self.filename = format_filename(self.title, info['id'], ext)
return self._url
def pp(self, filename):
## import ffmpeg
## f = IO()
## downloader.download(self.f_audio['url'], buffer=f)
## ffmpeg.merge(filename, f)
return filename
================================================
FILE: src/extractor/danbooru_downloader.py
================================================
#coding: utf-8
import downloader
import ree as re
from utils import Downloader, Session, get_max_range, clean_title, get_print, try_n, urljoin, check_alive, LazyUrl, get_ext, limits
from translator import tr_
from urllib.parse import urlparse, parse_qs, quote
import clf2
class Downloader_danbooru(Downloader):
type = 'danbooru'
URLS = ['danbooru.donmai.us']
MAX_CORE = 4
_name = None
ACCEPT_COOKIES = [r'(.*\.)?donmai\.us']
def init(self):
self.session = Session('chrome')
clf2.solve(self.url, session=self.session, cw=self.cw) #5336
self.session.headers['User-Agent'] = 'Mozilla/5.' #7034
@classmethod
def fix_url(cls, url):
if 'donmai.us' in url:
url = url.replace('http://', 'https://')
else:
url = url.replace(' ', '+')
while '++' in url:
url = url.replace('++', '+')
url = f'https://danbooru.donmai.us/posts?tags={quote(url)}'
if 'donmai.us/posts/' in url:
url = url.split('?')[0]
return url.strip('+')
@property
def name(self):
if self._name is None:
parsed_url = urlparse(self.url)
qs = parse_qs(parsed_url.query)
if 'donmai.us/favorites' in self.url:
id = qs.get('user_id', [''])[0]
print('len(id) =', len(id), f'"{id}"')
if not id:
raise AssertionError('[Fav] User id is not specified')
id = f'fav_{id}'
elif 'donmai.us/explore/posts/popular' in self.url: #4160
soup = read_soup(self.url, self.session, self.cw)
id = soup.find('h1').text
elif 'donmai.us/posts/' in self.url:
id = re.find(r'donmai\.us/posts/([0-9]+)', self.url, err='no id')
else:
tags = qs.get('tags', [])
tags.sort()
id = ' '.join(tags)
if not id:
id = 'N/A'
self._name = id
return clean_title(self._name)
def read(self):
self.title = self.name
if 'donmai.us/posts/' in self.url:
self.single = True
imgs = get_imgs(self.url, self.session, self.name, cw=self.cw)
for img in imgs:
self.urls.append(img.url)
self.title = self.name
class Image:
def __init__(self, id, url, session, cw):
self._cw = cw
self.id = id
self._session = session
self.url = LazyUrl(url, self.get, self)
def get(self, url):
soup = read_soup(url, self._session, self._cw)
ori = soup.find('li', id='post-option-view-original')
if ori:
img = ori.find('a')['href']
else:
img = soup.find('li', id='post-info-size').find('a')['href']
if get_ext(img) == '.zip': #4635
img = soup.find('section', id='content').find('video')['src']
img = urljoin(url, img)
ext = get_ext(img)
self.filename = f'{self.id}{ext}'
return img, None
@limits(1)
def wait(cw):
check_alive(cw)
def setPage(url, page):
# Main page
if re.findall(r'https://[\w]*[.]?donmai.us/?$', url):
url = f"https://{'danbooru.' if 'danbooru.' in url else ''}donmai.us/posts?page=1"
# Change the page
if 'page=' in url:
url = re.sub('page=[0-9]*', f'page={page}', url)
else:
url += f'&page={page}'
return url
@try_n(12) #4103
def read_soup(url, session, cw, try_=1):
check_alive(cw)
wait(cw)
if try_ > 1:
session.headers['User-Agent'] = downloader.ua.random #5730
return downloader.read_soup(url, session=session)
def get_imgs(url, session, title=None, range_=None, cw=None):
if 'donmai.us/artists' in url:
raise NotImplementedError()
if 'donmai.us/posts/' in url:
id = re.find(r'donmai\.us/posts/([0-9]+)', url, err='no id')
img = Image(id, url, session, cw)
return [img]
print_ = get_print(cw)
# Range
max_pid = get_max_range(cw)
if range_ is None:
range_ = range(1, 1001)
print(range_)
imgs = []
i = 0
empty_count = 0
empty_count_global = 0
url_imgs = set()
while i < len(range_):
check_alive(cw)
p = range_[i]
url = setPage(url, p)
print_(url)
soup = read_soup(url, session, cw)
articles = soup.findAll('article')
if articles:
empty_count_global = 0
else:
empty_count += 1
if empty_count < 4:
s = f'empty page; retry... {p}'
print_(s)
continue
else:
empty_count = 0
empty_count_global += 1
if empty_count_global >= 6:
break
for article in articles:
id = article.attrs['data-id']
#url_img = article.attrs['data-file-url'].strip()
url_img = urljoin(url, article.find('a', class_='post-preview-link')['href']) #4160
#print(url_img)
if url_img not in url_imgs:
url_imgs.add(url_img)
img = Image(id, url_img, session, cw)
imgs.append(img)
if len(imgs) >= max_pid:
break
if cw is not None:
cw.setTitle(f'{tr_("읽는 중...")} {title} - {len(imgs)}')
i += 1
return imgs[:max_pid]
================================================
FILE: src/extractor/discord_emoji_downloader.py
================================================
# coding: UTF-8
# title: Discord 서버 커스텀 이모지 다운로드
# author: SaidBySolo
"""
MIT License
Copyright (c) 2020 SaidBySolo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
from utils import Downloader, clean_title
import requests
import errors
class DownloaderDiscordEmoji(Downloader):
type = "discord"
def init(self):
pass
def read(self):
token_guild_id_list = self.url.split(
"/"
) # 값을 어떻게 받을지 몰라서 일단 나눴어요. discord_이메일/비밀번호/서버아이디 또는 discord_토큰/서버아이디 이런식으로 받게 해놨어요.
if len(token_guild_id_list) == 2:
token = token_guild_id_list[0]
guild_id = token_guild_id_list[1]
elif len(token_guild_id_list) == 3:
email = token_guild_id_list[0]
password = token_guild_id_list[1]
guild_id = token_guild_id_list[2]
response = self.post_account_info(email, password)
account_info = response.json()
if response.status_code == 400:
if account_info.get("captcha_key"):
raise errors.Invalid(
"먼저 웹 또는 디스코드 앱에서 로그인하신후 캡차를 인증해주세요."
) # 메세지 박스 return하니까 멈춰서 raise로 해놨어요
else:
raise errors.Invalid("이메일 또는 비밀번호가 잘못되었습니다. 확인후 다시 시도해주세요.")
else:
if not account_info["token"]:
raise errors.Invalid("토큰을 받아오지 못했어요. 2단계인증을 사용중이신경우 토큰을 이용해 요청해주세요.")
else:
token = account_info["token"]
else:
raise errors.Invalid("인자값이 더 많이왔어요.")
guild_info_response = self.get_emoji_list(token, int(guild_id)) # 토큰과 함께 get요청함
if guild_info_response.status_code != 200:
raise errors.Invalid("정상적인 토큰이 아니거나 서버를 찾을수없어요. 맞는 토큰인지, 해당 서버에 접속해있는지 확인해주세요.")
else:
guild_info = guild_info_response.json()
if guild_info["emojis"]:
base_url = "https://cdn.discordapp.com/emojis/"
for emoji in guild_info["emojis"]: # 이모지 리스트로 가져옴
if emoji["animated"] is True: # 만약 gif면 gif 다운로드
param = emoji["id"] + ".gif"
else: # 아닐경우 png로
param = emoji["id"] + ".png"
self.title = clean_title(
f'{guild_info["name"]}({guild_info["id"]})' # 폴더 이름은 서버 이름, id
)
self.urls.append(base_url + param + "?v=1") # 인자 합치기
else:
raise errors.Invalid("해당 서버에는 이모지가 없어요")
def get_emoji_list(self, token: str, guild_id: int) -> dict:
response = requests.get(
f"https://discordapp.com/api/v6/guilds/{guild_id}",
headers={"Authorization": token},
)
if response.status_code == 401:
response = requests.get(
f"https://discordapp.com/api/v6/guilds/{guild_id}",
headers={"Authorization": f"Bot {token}"},
)
return response
def post_account_info(self, email: str, password: str) -> dict:
response = requests.post(
"https://discordapp.com/api/v8/auth/login",
json={
"email": email,
"password": password,
"undelete": False,
"captcha_key": None,
"login_source": None,
"gift_code_sku_id": None,
},
)
return response
================================================
FILE: src/extractor/etc_downloader.py
================================================
import downloader
import ytdl
from utils import Downloader, Session, try_n, LazyUrl, get_ext, format_filename, get_print, get_resolution, print_error, cut_pair, json
from io import BytesIO
import ree as re
from m3u8_tools import playlist2stream, M3u8_stream
import utils
import ffmpeg
import clf2
import os
class Downloader_etc(Downloader):
type = 'etc'
URLS = ['thisvid.com'] #5153
single = True
MAX_PARALLEL = 8
display_name = 'Etc'
PRIORITY = 10
@try_n(2)
def read(self):
self.session = Session()
name = ytdl.get_extractor_name(self.url)
self.print_('extractor: {}'.format(name))
if name == 'ixigua': #6290
clf2.solve(self.url, session=self.session)
#if name == 'generic':
# raise NotImplementedError()
video = get_video(self.url, self.session, self.cw)
if video.artist:
self.artist = video.artist
self.print_('url_thumb: {}'.format(video.url_thumb))
self.setIcon(video.thumb)
if video.header.lower() not in ['yourporn']:
self.enableSegment()#
if isinstance(video.url(), M3u8_stream):
self.disableSegment()
self.urls.append(video.url)
self.title = os.path.splitext(video.filename)[0].replace('：', ':')
def int_or_none(s):
try:
return int(s)
except:
return None
def format_(f):
if f is None:
return 'None'
return 'format:{} - resolution:{} - vbr:{} - audio:{} - url:{}'.format(f['format'], f['_resolution'], f['_vbr'], f['_audio'], f['url'])
class UnSupportedError(Exception):pass
def get_video(url, session, cw, ie_key=None):
print_ = get_print(cw)
try:
video = _get_video(url, session, cw, ie_key, allow_m3u8=True)
if isinstance(video, Exception):
raise video
if isinstance(video.url(), M3u8_stream):
c = video.url().urls[0][1].download(cw)
if not c:
raise Exception('invalid m3u8')
return video
except Exception as e:
if isinstance(e, UnSupportedError):
raise e
print_(print_error(e))
return _get_video(url, session, cw, ie_key, allow_m3u8=False)
def extract_info_spankbang(url, session, cw): # temp fix
print_ = get_print(cw)
soup = downloader.read_soup(url, session=session)
for script in soup.findAll('script'):
script = script.string
if script and 'var stream_data'in script:
s = cut_pair(script.split('var stream_data')[-1].strip(' \t=').replace("'", '"'))
break
else:
raise Exception('no stream_data')
info = {}
info['url'] = url
info['title'] = soup.find('h1').text.strip()
info['id'] = re.find(r'spankbang\.com/([^/]+)', soup.find('meta', {'property': 'og:url'})['content'], err='no id')
info['thumbnail'] = soup.find('meta', {'property': 'og:image'})['content']
info['formats'] = []
data = json.loads(s)
for res, item in data.items():
if res == 'main':
continue
if item and isinstance(item, list):
item = item[0]
else:
continue
ext = get_ext_(item, session, url)
res = {'4k': '2160p', '8k': '4320p', '16k': '8640p'}.get(res, res)
f = {'url': item, 'format': res}
info['formats'].append(f)
return info
@try_n(2)
def _get_video(url, session, cw, ie_key=None, allow_m3u8=True):
print_ = get_print(cw)
print_('get_video: {}, {}'.format(allow_m3u8, url))
options = {
'noplaylist': True,
#'extract_flat': True,
'playlistend': 1,
'writesubtitles': True,
}
if ytdl.get_extractor_name(url) == 'spankbang':
info = extract_info_spankbang(url, session, cw)
else:
ydl = ytdl.YoutubeDL(options, cw=cw)
try:
info = ydl.extract_info(url)
except Exception as e:
if 'ERROR: Unsupported URL' in str(e):
return UnSupportedError(str(e))
raise e
if not ie_key:
ie_key = ytdl.get_extractor_name(url)
info['ie_key'] = ie_key
url_new = info.get('url')
formats = info.get('formats', [])
if not formats and (info.get('entries') or 'title' not in info):
if 'entries' in info:
entry = info['entries'][0]
url_new = entry.get('url') or entry['webpage_url']
if url_new != url:
return get_video(url_new, session, cw, ie_key=get_ie_key(info))
session.headers.update(info.get('http_headers', {}))
#session.cookies.update(ydl.cookiejar)
if not formats:
if url_new:
f = {'url': url_new, 'format': ''}
formats.append(f)
fs = []
for i, f in enumerate(formats):
f['_index'] = i
f['_resolution'] = int_or_none(re.find(r'([0-9]+)p', f['format'], re.I)) or f.get('height') or f.get('width') or int_or_none(f.get('quality')) or int(f.get('vcodec', 'none') != 'none') #5995
f['_vbr'] = f.get('vbr') or 0
f['_audio'] = f.get('abr') or f.get('asr') or int(f.get('acodec', 'none') != 'none')
print_(format_(f))
fs.append(f)
#4773
res = max(get_resolution(), min(f['_resolution'] for f in fs))
print_(f'res: {res}')
fs = [f for f in fs if f['_resolution'] <= res]
if not fs:
raise Exception('No videos')
def filter_f(fs):
for f in fs:
if allow_m3u8:
return f
ext = get_ext_(f['url'], session, url)
if ext.lower() != '.m3u8':
return f
print_('invalid url: {}'.format(f['url']))
return list(fs)[0]#
f_video = filter_f(sorted(fs, key=lambda f:(f['_resolution'], int(bool(f['_audio'])), f['_vbr'], f['_index']), reverse=True)) #6072, #6118
print_('video0: {}'.format(format_(f_video)))
if f_video['_audio']:
f_audio = None
else:
fs_audio = sorted([f_audio for f_audio in fs if (not f_audio['_resolution'] and f_audio['_audio'])], key=lambda f:(f['_audio'], f['_vbr'], f['_index']))
if fs_audio:
f_audio = fs_audio[-1]
else:
try:
print_('trying to get f_video with audio')
f_video = filter_f(reversed(sorted([f for f in fs if f['_audio']], key=lambda f:(f['_resolution'], f['_index']))))
except Exception as e:
print_('failed to get f_video with audio: {}'.format(e))
f_audio = None
print_('video: {}'.format(format_(f_video)))
print_('audio: {}'.format(format_(f_audio)))
video = Video(f_video, f_audio, info, session, url, cw=cw)
return video
def get_ie_key(info):
ie_key = info.get('ie_key') or info['extractor']
ie_key = ie_key.split(':')[0]
if ie_key.lower().endswith('playlist'):
ie_key = ie_key[:-len('playlist')]
return ie_key
def get_ext_(url, session, referer):
try:
ext = downloader.get_ext(url, session, referer)
except Exception as e:
ext = get_ext(url)
return ext
class Video:
live = False
def __init__(self, f, f_audio, info, session, referer, cw=None):
self.f_audio = f_audio
self.cw = cw
print_ = get_print(cw)
self.title = title = info['title']
self.id = info['id']
self.url = f['url']
self.artist = info.get('uploader')
self.header = utils.capitalize(get_ie_key(info))
self.session = session
self.referer = referer
self.subs = ytdl.get_subtitles(info)
self.url_thumb = info.get('thumbnail')
self.thumb = BytesIO()
if self.url_thumb:
downloader.download(self.url_thumb, referer=referer, buffer=self.thumb, session=session)
ext = get_ext_(self.url, session, referer)
def foo():
hdr = session.headers.copy()
if referer:
hdr['Referer'] = referer
self.live = True
return utils.LiveStream(self.url, headers=hdr, fragments=f.get('fragments') if ytdl.LIVE_FROM_START.get('etc') else None)
if not ext:
if f['_resolution']:
ext = '.mp4'
else:
ext = '.mp3'
if ext.lower() == '.m3u8':
res = get_resolution() #4773
ls = info.get('live_status')
print_(f'live_status: {ls}')
if ls == 'is_live':
url = foo()
else:
try:
url = playlist2stream(self.url, referer, session=session)
except:
url = M3u8_stream(self.url, referer=referer, session=session)
print_(f'mpegts: {url.mpegts}')
if url.ms or url.mpegts == False: #5110
url = url.live
url._cw = cw
ext = '.mp4'
elif ext.lower() == '.mpd': # TVer
url = foo()
ext = '.mp4'
else:
url = self.url
self.url = LazyUrl(referer, lambda x: url, self, pp=self.pp)
info_ext = info.get('ext')
if info_ext == 'unknown_video': #vk
info_ext = None
self.filename = format_filename(title, self.id, info_ext or ext, header=self.header, live=self.live)
def pp(self, filename):
if self.f_audio:
f = BytesIO()
downloader.download(self.f_audio['url'], buffer=f, referer=self.referer, session=self.session)
ffmpeg.merge(filename, f, cw=self.cw)
utils.pp_subtitle(self, filename, self.cw)
return filename
================================================
FILE: src/extractor/fc2_downloader.py
================================================
import downloader
import ree as re
from utils import urljoin, Downloader, format_filename, Soup, LazyUrl, get_print, Session
from m3u8_tools import M3u8_stream
from io import BytesIO
PATTERN_ID = r'/content/([^/]+)'
class Downloader_fc2(Downloader):
type = 'fc2'
single = True
URLS = ['video.fc2.com']
ACCEPT_COOKIES = [r'(.*\.)?fc2\.com']
@classmethod
def fix_url(cls, url):
if not re.match(r'https?://.+', url, re.I):
url = f'https://video.fc2.com/content/{url}'
return url
@classmethod
def key_id(cls, url):
return re.find(PATTERN_ID, url) or url
def read(self):
self.session = Session()
self.session.cookies.set('_ac', '1', domain='.video.fc2.com')
info = get_info(self.url, self.session, self.cw)
video = info['videos'][0]
self.urls.append(video.url)
f = BytesIO()
downloader.download(video.url_thumb, referer=self.url, buffer=f)
self.setIcon(f)
self.title = info['title']
class Video:
def __init__(self, url, url_thumb, referer, title, id_, session):
self._url = url
self.url = LazyUrl(referer, self.get, self)
self.filename = format_filename(title, id_, '.mp4')
self.url_thumb = url_thumb
self.session = session
def get(self, referer):
ext = downloader.get_ext(self._url, session=self.session, referer=referer)
if ext == '.m3u8':
video = M3u8_stream(self._url, referer=referer, session=self.session, n_thread=4)
else:
video = self._url
return video
def get_info(url, session, cw=None):
print_ = get_print(cw)
info = {'videos': []}
html = downloader.read_html(url, session=session)
soup = Soup(html)
info['title'] = soup.find('h2', class_='videoCnt_title').text.strip()
id_ = re.find(PATTERN_ID, url, err='no id')
print_('id: {}'.format(id_))
token = re.find(r'''window.FC2VideoObject.push\(\[['"]ae['"], *['"](.+?)['"]''', html, err='no token')
print_('token: {}'.format(token))
url_api = 'https://video.fc2.com/api/v3/videoplaylist/{}?sh=1&fs=0'.format(id_)
hdr = {
'X-FC2-Video-Access-Token': token,
}
data = downloader.read_json(url_api, url, session=session, headers=hdr)
pl = data['playlist']
url_video = urljoin(url, pl.get('hq') or pl.get('nq') or pl['sample']) #3784
url_thumb = soup.find('meta', {'property':'og:image'})['content']
video = Video(url_video, url_thumb, url, info['title'], id_, session)
info['videos'].append(video)
return info
================================================
FILE: src/extractor/file_downloader.py
================================================
import downloader, os
from utils import Downloader, query_url, clean_title, get_ext, Session, Soup, File, urljoin, fix_dup, try_n
from hashlib import md5
import clf2
import os
class Downloader_file(Downloader):
type = 'file'
single = True
URLS = []
ACC_MTIME = True
@classmethod
def fix_url(cls, url):
if url and '://' not in url:
url = 'https://' + url.lstrip('/')
return url
@try_n(4)
def read(self):
if not self.url.strip():
raise Exception('empty url')
self.session = Session() #6525
qs = query_url(self.url)
for key in qs:
if key.lower() in ('file', 'filename'):
name = qs[key][-1]
break
else:
name = self.url
for esc in ['?', '#']:
name = name.split(esc)[0]
name = os.path.basename(name.strip('/'))
try:
ext = downloader.get_ext(self.url)
except:
ext = ''
if not ext:
ext = get_ext(name)
name = os.path.splitext(name)[0]
id_ = md5(self.url.encode('utf8')).hexdigest()[:8]
if ext.lower()[1:] in ['htm', 'html']:
self.single = False
res = clf2.solve(self.url, session=self.session, cw=self.cw)
soup = Soup(res['html'])
ext = ''
title = soup.find('meta', {'property': 'og:title'})
title = title['content']
names = {}
srcs = set()
for img in soup.findAll('img'):
src = img.get('src')
if not src:
continue
src = urljoin(self.url, src)
if src in srcs:
continue
srcs.add(src)
name = os.path.basename(src.split('?')[0].split('#')[0])
ext = get_ext(name)
if not ext:
try:
ext = downloader.get_ext(src)
except Exception as e:
print(e)
name = clean_title(os.path.splitext(name)[0], n=-len(ext)) + ext
name = fix_dup(name, names)
file = File({'referer': self.url, 'url': src, 'name': name})
self.urls.append(file)
else:
title = name
file = File({'url': self.url, 'name': name})
self.urls.append(file)
tail = f' ({id_})'
if self.single:
tail += f'{ext}'
self.title = clean_title(title, n=-len(tail)) + tail
def parse(s):
_ = {'none': None, 'true': True, 'false': False}.get(s.lower(), ' ')
return int(s) if _ == ' ' else _
kwargs = {}
c = self.cw.comment()
if c.startswith('segment:') and (s := c[len('segment:'):].strip()):
if s.count('=') != 1:
raise ValueError('not one "="')
key, value = s.split('=')
kwargs[key] = parse(value)
if self.single or kwargs:
self.enableSegment(**kwargs)
================================================
FILE: src/extractor/flickr_downloader.py
================================================
from utils import Downloader, File, Session, urljoin, get_ext, clean_title, Soup, limits
import utils
import ree as re
import downloader
import clf2
from timee import time
TIMEOUT = 10
class File_flickr(File):
type = 'flickr'
format = '[date] id'
@limits(1)
def get(self):
url = self['referer']
soup = downloader.read_soup(url, session=self.session)
img = soup.find('meta', {'property': 'og:image'})['content']
date = re.find(r'"dateCreated":{"data":"([0-9]+)"', soup.html, err='no date')
ext = get_ext(img)
d = {
'date': int(date),
'id': re.find(r'/photos/[^/]+/([0-9]+)', url, err='no id'),
}
return {'url': img, 'name': utils.format('flickr', d, ext)}
class Downloader_flickr(Downloader):
type = 'flickr'
URLS = ['flickr.com']
MAX_CORE = 4
ACCEPT_COOKIES = [r'(.*\.)?flickr\.com']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
url = url.replace('flickr.com/people/', 'flickr.com/photos/')
uid = re.find(r'flickr.com/photos/([^/]+)', url)
if uid:
url = f'https://www.flickr.com/photos/{uid}'
return url
def read(self):
tab = ''.join(self.url.split('/')[3:4])
if tab == 'photos':
uid = self.url.split('/')[4]
title = None
ids = set()
c = 0
ct = None
p_max = 1
def f(html, browser=None):
nonlocal title, c, ct, p_max
soup = Soup(html)
browser.runJavaScript('window.scrollTo(0,document.body.scrollHeight);')
for a in soup.findAll('a'):
href = a.get('href') or ''
href = urljoin(self.url, href)
p_max = max(p_max, int(re.find(rf'flickr.com/photos/{uid}/page([0-9]+)', href) or 0))
id_ = re.find(rf'/photos/{uid}/([0-9]+)', href)
if not id_:
continue
if id_ in ids:
continue
ids.add(id_)
file = File_flickr({'referer': href})
self.urls.append(file)
if ids:
uname = soup.h1.text.strip()
title = f'{clean_title(uname)} (flickr_{uid})'
self.cw.setTitle(f'{title} - {len(ids)}')
if c == len(ids):
if not ct:
ct = time()
dt = time() - ct
if dt > TIMEOUT:
return True
else:
ct = None
c = len(ids)
p = 1
while p <= p_max:
url = f'https://www.flickr.com/photos/{uid}/page{p}'
self.print_(url)
clf2.solve(url, session=self.session, f=f)
p += 1
self.title = title
else:
raise NotImplementedError(tab)
================================================
FILE: src/extractor/gelbooru_downloader.py
================================================
#coding: utf-8
import downloader
import ree as re
from utils import Downloader, urljoin, query_url, get_max_range, get_print, get_ext, clean_title, Session, check_alive, File, clean_url
from translator import tr_
from urllib.parse import quote
import utils
def get_tags(url):
url = clean_url(url)
qs = query_url(url)
if 'page=favorites' in url:
id = qs.get('id', ['N/A'])[0]
id = 'fav_{}'.format(id)
else:
tags = qs.get('tags', [])
tags.sort()
id = ' '.join(tags)
if not id:
id = 'N/A'
return id
class Downloader_gelbooru(Downloader):
type = 'gelbooru'
URLS = ['gelbooru.com']
MAX_CORE = 8
_name = None
ACCEPT_COOKIES = [r'(.*\.)?gelbooru\.com']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
if 'gelbooru.com' in url.lower():
url = url.replace('http://', 'https://')
else:
url = url.replace(' ', '+')
while '++' in url:
url = url.replace('++', '+')
url = quote(url)
url = url.replace('%2B', '+')
url = 'https://gelbooru.com/index.php?page=post&s=list&tags={}'.format(url)
return url
@property
def name(self):
if self._name is None:
tags = get_tags(self.url)
self._name = tags
return clean_title(self._name)
def read(self):
self.title = self.name
self.urls += get_imgs(self.url, self.session, self.name, cw=self.cw)
self.title = self.name
class File_gelbooru(File):
type = 'gelbooru'
format = 'id'
def get(self):
soup = downloader.read_soup(self['referer'], session=self.session)
for li in soup.findAll('li'):
if li.text.strip() == 'Original image':
break
else:
raise Exception('no Original image')
url = li.find('a')['href']
d = {
'id': self['id'],
}
return {'url': url, 'name': utils.format('gelbooru', d, get_ext(url))}
def alter(self):
return self.get()['url']
def setPage(url, page):
if 'pid=' in url:
url = re.sub('pid=[0-9]*', f'pid={page}', url)
else:
url += f'&pid={page}'
if page == 0:
url = url.replace('&pid=0', '')
return url
def get_imgs(url, session, title=None, cw=None):
print_ = get_print(cw)
url = clean_url(url)
if 's=view' in url and 'page=favorites' not in url:
raise NotImplementedError('Not Implemented')
tags = get_tags(url)
tags = quote(tags, safe='/')
tags = tags.replace('%20', '+')
url = f'https://gelbooru.com/index.php?page=post&s=list&tags={tags}'
# 2566
user_id = session.cookies.get('user_id', domain='gelbooru.com')
if not user_id:
cookies = {'fringeBenefits': 'yup'}
session.cookies.update(cookies)
print_('user_id: {}'.format(user_id))
# Range
max_pid = get_max_range(cw)
imgs = []
ids = set()
count_no_imgs = 0
for p in range(500): #1017
check_alive(cw)
url = setPage(url, len(ids))
print_(url)
soup = downloader.read_soup(url, session=session)
posts = soup.findAll(class_='thumbnail-preview')
imgs_new = []
for post in posts:
id_ = int(re.find('[0-9]+', post.find('a')['id'], err='no id'))
if id_ in ids:
print('duplicate:', id_)
continue
ids.add(id_)
url_img = urljoin(url, post.find('a')['href'])
img = File_gelbooru({'id': id_, 'referer': url_img, 'name_hint': f'{id_}{{ext}}'})
imgs_new.append(img)
if imgs_new:
imgs += imgs_new
count_no_imgs = 0
else:
print('no imgs')
count_no_imgs += 1
if count_no_imgs > 1:
print('break')
break
if len(imgs) >= max_pid:
break
if cw is not None:
cw.setTitle('{} {} - {}'.format(tr_('읽는 중...'), title, len(imgs)))
return imgs[:max_pid]
================================================
FILE: src/extractor/hameln_downloader.py
================================================
#coding: utf8
import downloader
import os
import utils
from utils import Soup, urljoin, get_text, LazyUrl, try_n, Downloader, lazy, clean_title
import ree as re
from io import BytesIO
from translator import tr_
class Downloader_hameln(Downloader):
type = 'hameln'
URLS = ['syosetu.org']
MAX_CORE = 2
detect_removed = False
ACCEPT_COOKIES = [r'(.*\.)?syosetu\.org']
def init(self):
id_ = re.find('/novel/([^/]+)', self.url)
if id_ is not None:
self.url = 'https://syosetu.org/novel/{}/'.format(id_)
@lazy
def soup(self):
html = read_html(self.url)
soup = Soup(html)
return soup
@lazy
def info(self):
return get_info(self.url, self.soup)
def read(self):
for page in get_pages(self.url, self.soup):
text = Text(page, len(self.urls)+1)
self.urls.append(text.url)
self.artist = self.info['artist']
self.title = clean_title('[{}] {}'.format(self.artist, self.info['title']), n=-len('[merged] .txt'))
def post_processing(self):
names = self.cw.names
filename = os.path.join(self.dir, '[merged] {}.txt'.format(self.title))
try:
with utils.open(filename, 'wb') as f:
f.write(' {}\n\n 作者：{}\n\n\n'.format(self.info['title'], self.artist).encode('utf8'))
if self.info['novel_ex']:
f.write(self.info['novel_ex'].encode('utf8'))
for i, file in enumerate(names):
self.cw.pbar.setFormat('[%v/%m] {} [{}/{}]'.format(tr_('병합...'), i, len(names)))
with open(file, 'rb') as f_:
text = f_.read()
f.write(b'\n\n\n\n')
f.write(text)
finally:
self.cw.pbar.setFormat('[%v/%m]')
class Text:
def __init__(self, page, p):
self.page = page
self.url = LazyUrl(page.url, self.get, self)
self.filename = clean_title('[{:04}] {}'.format(p, page.title), n=-4) + '.txt'
def get(self, url):
text = read_page(self.page)
f = BytesIO()
f.write(text.encode('utf8'))
f.seek(0)
return f
class Page:
def __init__(self, title, url):
self.title = clean_title(title)
self.url = url
def read_html(url):
return downloader.read_html(url, cookies={'over18': 'off'}, headers={'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36'})
def get_sss(soup):
sss = [ss for ss in soup.findAll('div', class_='ss') if ss.attrs.get('id')!='fmenu']
return sss
def get_pages(url, soup=None):
if soup is None:
html = read_html(url)
soup = Soup(html)
sss = get_sss(soup)
list = sss[-1]
pages = []
for tr in list.findAll('tr'):
a = tr.find('a')
if a is None:
continue
text =a.text.strip()
href = urljoin(url, a.attrs['href'])
page = Page(text, href)
pages.append(page)
return pages
@try_n(22, sleep=30)
def read_page(page):
html = read_html(page.url)
soup = Soup(html)
text_top = get_text(soup.find('div', id='maegaki'))
print(text_top.count('\n'))
text_mid = get_text(soup.find('div', id='honbun'))
text_bot = get_text(soup.find('div', id='atogaki'))
texts = [text for text in (text_top, text_mid, text_bot) if text]
story = '''
────────────────────────────────
'''.join(texts)
text = '''────────────────────────────────
◆ {}
────────────────────────────────
{}'''.format(page.title, story)
return text
def get_info(url, soup=None):
if soup is None:
html = read_html(url)
soup = Soup(html)
info = {}
info['artist'] = soup.find('span', {'itemprop':'author'}).text.strip()
info['title'] = soup.find('span', {'itemprop':'name'}).text.strip()
sss = get_sss(soup)
info['novel_ex'] = get_text(sss[-2])
return info
================================================
FILE: src/extractor/hanime_downloader.py
================================================
import downloader
from utils import Session, Downloader, try_n, Soup, format_filename, get_print, get_resolution, json
import ree as re
from io import BytesIO
import os
from timee import time
from m3u8_tools import M3u8_stream
from random import randrange
class Video:
def __init__(self, info, stream):
self.info = info
self.id = info['id']
self.title = info['name']
self.brand = info['brand']
self.url = stream['url']
self.url_thumb = info['poster_url']
self.thumb = BytesIO()
downloader.download(self.url_thumb, buffer=self.thumb)
ext = os.path.splitext(self.url.split('?')[0].split('#')[0])[1]
if ext.lower() == '.m3u8':
ext = '.mp4'
self.url = M3u8_stream(self.url, n_thread=4)
for i, seg in self.url.urls[-20:]:
seg._ignore_err = True #5272
else:
size = downloader.get_size(self.url)
if size <= 0:
raise Exception('Size is 0')
self.filename = format_filename('[{}] {}'.format(self.brand, self.title), self.id, ext)
def __repr__(self):
return f'Video({self.id})'
class Downloader_hanime(Downloader):
type = 'hanime'
URLS = ['hanime.tv/hentai-videos/', 'hanime.tv/videos/']
single = True
display_name = 'hanime.tv'
ACCEPT_COOKIES = [r'(.*\.)?hanime\.tv']
def init(self):
self.session = Session('chrome')
def read(self):
video = get_video(self.url, self.session, cw=self.cw)
self.video = video
self.urls.append(video.url)
self.filenames[video.url] = video.filename
self.setIcon(video.thumb)
self.title = '[{}] {}'.format(video.brand, video.title)
@try_n(8)
def get_video(url, session, cw=None):
print_ = get_print(cw)
session.headers['X-Directive'] = 'api'
html = downloader.read_html(url, session=session)
soup = Soup(html)
for script in soup.findAll('script'):
script = script.text or script.string or ''
data = re.find('window.__NUXT__=(.+)', script)
if data is not None:
data = data.strip()
if data.endswith(';'):
data = data[:-1]
data = json.loads(data)
break
else:
raise Exception('No __NUXT__')
info = data['state']['data']['video']['hentai_video']
query = info['slug']
#url_api = 'https://members.hanime.tv/api/v3/videos_manifests/{}?'.format(query) # old
url_api = 'https://hanime.tv/rapi/v7/videos_manifests/{}?'.format(query) # new
hdr = {
'x-signature': ''.join('{:x}'.format(randrange(16)) for i in range(32)),
'x-signature-version': 'web2',
'x-time': str(int(time())),
}
r = session.get(url_api, headers=hdr)
data = json.loads(r.text)
streams = []
for server in data['videos_manifest']['servers']:
streams += server['streams']
streams_good = []
for stream in streams:
url_video = stream['url']
if not url_video or 'deprecated.' in url_video:
continue
stream['height'] = int(stream['height'])
streams_good.append(stream)
if not streams_good:
raise Exception('No video available')
res = get_resolution()
def print_stream(stream):
print_([stream['extension'], stream['height'], stream['filesize_mbs'], stream['url']])
steams_filtered = []
for stream in streams_good:
print_stream(stream)
if stream['height'] <= res: #3712
steams_filtered.append(stream)
if steams_filtered:
stream = sorted(steams_filtered, key=lambda _: _['height'])[-1]
else:
stream = sorted(streams_good, key=lambda _: _['height'])[0]
print_('Final stream:')
print_stream(stream)
return Video(info, stream)
================================================
FILE: src/extractor/hentaicosplay_downloader.py
================================================
#coding: utf8
import downloader
from utils import Downloader, Session, Soup, LazyUrl, urljoin, get_ext, clean_title, try_n, limits
import utils
import ree as re
from translator import tr_
import clf2
from m3u8_tools import M3u8_stream
from timee import sleep
import os
class Image:
def __init__(self, url, referer, p, session):
self._url = url
self._referer = referer
self._p = p
self.url = LazyUrl(url, self.get, self)
self.session = session
@try_n(3, 5)
@limits(1)
def get(self, _=None):
soup = downloader.read_soup(self._url, self._referer, session=self.session)
div = soup.find('div', id='display_image_detail') or soup.find('ul', id='detail_list')
parent = div.find('img').parent
while not parent.get('href'):
parent = parent.parent
url = urljoin(self._url, parent['href'])
ext = get_ext(url)
self.filename = '{:04}{}'.format(self._p, ext)
return url, self._url
class Video:
def __init__(self, src, referer, title, session):
ext = get_ext(src)
if ext == '.m3u8':
_src = src
src = M3u8_stream(_src, referer=referer, session=session)
ext = '.mp4'
self.url = LazyUrl(referer, lambda _: src, self)
self.filename = '{}{}'.format(clean_title(title), ext)
class Downloader_hentaicosplay(Downloader):
type = 'hentaicosplay'
URLS = ['hentai-cosplays.com', 'porn-images-xxx.com', 'hentai-img.com', 'porn-video-xxx.com']
icon = None
display_name = 'Hentai Cosplay'
MAX_PARALLEL = 1 # must be 1
MAX_CORE = 4
ACCEPT_COOKIES = [rf'(.*\.)?{domain}' for domain in URLS]
@classmethod
def fix_url(cls, url):
url = re.sub(r'/page/[0-9]+', '', url)
url = re.sub(r'/attachment/[0-9]+', '', url)
url = re.sub(r'([a-zA-Z]+\.)hentai-cosplays\.com', 'hentai-cosplays.com', url)
url = re.sub(r'.com/story/', '.com/image/', url)
return url
def init(self):
self.session = Session()
@try_n(2)
def read(self):
#4961
ua = downloader.ua.random
self.print_(f'read start ua: {ua}')
downloader.REPLACE_UA[r'hentai-cosplays\.com'] = ua
downloader.REPLACE_UA[r'porn-images-xxx\.com'] = ua
if '/video/' in self.url:
res = clf2.solve(self.url, session=self.session, cw=self.cw)
soup = Soup(res['html'])
title = (soup.find('h1', id='post_title') or soup.find('div', id='page').find('h2')).text.strip()
self.title = title
view = soup.find('div', id='post') or soup.find('div', class_='video-container')
video = view.find('video')
src = video.find('source')['src']
src = urljoin(self.url, src)
video = Video(src, self.url, title, self.session)
self.urls.append(video.url)
self.single = True
return
if '/image/' not in self.url:
raise NotImplementedError('Not a post')
res = clf2.solve(self.url, session=self.session, cw=self.cw)
soup = Soup(res['html'])
title = (soup.find('h2') or soup.find('h3')).text
paginator = soup.find('div', id='paginator') or soup.find('div', class_='paginator_area')
pages = [self.url]
for a in paginator.findAll('a'):
href = a.get('href')
if not href:
continue
href = urljoin(self.url, href)
if href not in pages:
pages.append(href)
self.print_(f'pages: {len(pages)}')
imgs = []
for i, page in enumerate(pages):
sleep(2, self.cw)
if page == self.url:
soup_page = soup
else:
soup_page = try_n(3, 5)(downloader.read_soup)(page, session=self.session)
view = soup_page.find('div', id='post') or soup_page.find('ul', id='detail_list')
for img in view.findAll('img'):
href = img.parent.get('href') or img.parent.parent.get('href')
if not href:
continue
href = urljoin(page, href)
img = Image(href, page, len(imgs), self.session)
imgs.append(img)
self.print_(f'imgs: {len(imgs)}')
self.cw.setTitle('{} {} ({} / {})'.format(tr_('읽는 중...'), title, i+1, len(pages)))
names = {}
dirname = utils.dir(self.type, clean_title(title), self.cw)
try:
files = os.listdir(dirname)
except:
files = []
for file in files:
name, ext = os.path.splitext(file)
names[name] = ext
for p, img in enumerate(imgs):
name = '{:04}'.format(p)
ext = names.get(name)
if ext:
self.urls.append(os.path.join(dirname, '{}{}'.format(name, ext)))
else:
self.urls.append(img.url)
self.title = clean_title(title)
================================================
FILE: src/extractor/hf_downloader.py
================================================
#coding:utf8
import downloader
from utils import Soup, urljoin, Session, LazyUrl, Downloader, try_n, clean_title, check_alive
import ree as re
import os
from translator import tr_
URL_ENTER = 'https://www.hentai-foundry.com/site/index?enterAgree=1&size=1550'
URL_FILTER = 'https://www.hentai-foundry.com/site/filters'
class Image:
def __init__(self, url, session):
@try_n(4)
def f(_):
html = downloader.read_html(url, session=session)
soup = Soup(html)
box = soup.find('section', id='picBox')
img = box.find('img')
if img is None:
raise Exception('No img')
onclick = img.attrs.get('onclick', '')
if onclick and '.src' in onclick:
print('onclick', onclick)
img = re.find('''.src *= *['"](.+?)['"]''', onclick)
else:
img = img.attrs['src']
img = urljoin(url, img)
filename = clean_title(os.path.basename(img.split('?')[0]))
name, ext = os.path.splitext(filename)
# https://www.hentai-foundry.com/pictures/user/DrGraevling/74069/Eversong-Interrogation-pg.-13
if ext.lower() not in ['.bmp', '.png', '.gif', '.jpg', '.jpeg', '.webp', '.webm', '.avi', '.mp4', '.mkv', '.wmv']:
filename = '{}.jpg'.format(name)
self.filename = filename
return img
self.url = LazyUrl(url, f, self)
def get_username(url):
if 'user/' in url:
username = url.split('user/')[1].split('?')[0].split('/')[0]
return username
class Downloader_hf(Downloader):
type = 'hf'
URLS = ['hentai-foundry.com']
MAX_CORE = 16
display_name = 'Hentai Foundry'
ACCEPT_COOKIES = [r'(.*\.)?hentai-foundry\.com']
def init(self):
self.session = enter()
@classmethod
def fix_url(cls, url):
username = get_username(url)
return 'https://www.hentai-foundry.com/user/{}'.format(username)
def read(self):
username = get_username(self.url)
self.title = username
imgs = get_imgs(username, self.title, self.session, cw=self.cw)
for img in imgs:
self.urls.append(img.url)
self.title = username
@try_n(2)
def enter():
print('enter')
session = Session()
r = session.get(URL_ENTER)
# 862
html = r.text
soup = Soup(html)
box = soup.find('aside', id='FilterBox')
data = {}
for select in box.findAll('select'):
name = select.attrs['name']
value = select.findAll('option')[-1].attrs['value']
print(name, value)
data[name] = value
for input in box.findAll('input'):
name = input.attrs['name']
value = input.attrs['value']
if name.startswith('rating_') or 'CSRF_TOKEN' in name:
print(name, value)
data[name] = value
data.update({
'filter_media': 'A',
'filter_order': 'date_new',
'filter_type': '0',
})
r = session.post(URL_FILTER, data=data, headers={'Referer': r.url})
print(r)
return session
def get_imgs(username, title, session, cw=None):
url = 'https://www.hentai-foundry.com/pictures/user/{}'.format(username)
#downloader.read_html(url_enter, session=session)
hrefs = []
for p in range(100):
check_alive(cw)
print(url)
html = downloader.read_html(url, session=session)
soup = Soup(html)
if soup.find('div', id='entryButtonContainer'):
session = enter()
continue
tab = soup.find('a', class_='active')
n = re.find(r'\(([0-9]+)', tab.text)
view = soup.find('div', class_='galleryViewTable')
for a in view.findAll('a', class_='thumbLink'):
href = urljoin(url, a.attrs['href'])
if href in hrefs:
print('dup')
continue
hrefs.append(href)
next = soup.find(lambda tag: tag.name == 'li' and tag.get('class') == ['next'])
if next is None:
break
url = urljoin(url, next.a.attrs['href'])
s = '{} {} ({} / {})'.format(tr_('읽는 중...'), title, len(hrefs), n)
if cw:
cw.setTitle(s)
else:
print(s)
imgs = []
for href in hrefs:
img = Image(href, session)
imgs.append(img)
return imgs
================================================
FILE: src/extractor/imgur_downloader.py
================================================
import downloader
from utils import Downloader, Soup, try_n, urljoin, get_max_range, clean_title, cut_pair, check_alive, json
import ree as re
import os
from translator import tr_
class Downloader_imgur(Downloader):
type = 'imgur'
URLS = ['imgur.com']
MAX_CORE = 16
ACCEPT_COOKIES = [r'(.*\.)?imgur\.com']
def init(self):
self.info = get_info(self.url)
@property
def id_(self):
return re.find('imgur.com/.+?/([0-9a-zA-Z]+)', self.url)
@property
def name(self):
title = self.info['title'] or 'N/A'
return clean_title(title, n=100)
def read(self):
imgs = get_imgs(self.url, self.info, self.cw)
for img in imgs:
ext = os.path.splitext(img.split('?')[0])[1]
if len(imgs) > 1:
self.filenames[img] = '{:04}{}'.format(len(self.urls), ext)
else:
self.filenames[img] = clean_title(self.name, n=-len(ext)) + ext
self.urls.append(img)
self.single = len(imgs) == 1
self.referer = self.url
self.title = '{} (imgur_{})'.format(self.name, self.id_)
@try_n(4)
def get_info(url):
url = url.replace('/gallery/', '/a/')
if '/r/' in url and url.split('/r/')[1].strip('/').count('/') == 0:
title = re.find(r'/r/([^/]+)', url)
info = {}
info['title'] = title
info['type'] = 'r'
else:
try: # legacy
html = downloader.read_html(url, cookies={'over18':'1'})
s = re.find('image *: *({.+)', html)
info_raw = cut_pair(s)
except Exception as e: # new
print(e)
id_ = re.find(r'/a/([0-9a-zA-Z_]+)', url) or re.find(r'/r/[0-9a-zA-Z_]+/([0-9a-zA-Z_]+)', url, err='no id')
url_api = 'https://api.imgur.com/post/v1/albums/{}?client_id=546c25a59c58ad7&include=media%2Cadconfig%2Caccount'.format(id_)
info_raw = downloader.read_html(url_api, cookies={'over18':'1'})
info = json.loads(info_raw)
info['type'] = 'a'
return info
def get_imgs(url, info=None, cw=None):
print('get_imgs', url)
if info is None:
info = get_info(url)
imgs = []
# Range
max_pid = get_max_range(cw)
if info['type'] == 'a':
if 'album_images' in info: # legacy
imgs_ = info['album_images']['images']
elif 'media' in info: # new
imgs_ = info['media']
else: # legacy
imgs_ = [info]
for img in imgs_:
img_url = img.get('url') # new
if not img_url: # legacy
hash = img['hash']
ext = img['ext']
img_url = 'https://i.imgur.com/{}{}'.format(hash, ext)
if img_url in imgs:
continue
imgs.append(img_url)
elif info['type'] == 'r':
urls = set()
for p in range(100):
url_api = 'https://imgur.com/r/{}/new/page/{}/hit?scrolled'.format(info['title'], p)
print(url_api)
html = downloader.read_html(url_api, referer=url)
soup = Soup(html)
c = 0
for post in soup.findAll('div', class_='post'):
check_alive(cw)
a = post.find('a', class_='image-list-link')
url_post = urljoin(url, a.attrs['href'])
if url_post in urls:
continue
urls.add(url_post)
c += 1
try: # for r18 images
imgs += get_imgs(url_post)
except Exception as e:
print(e)
s = '{} {} ({})'.format(tr_('읽는 중...'), info['title'], len(imgs))
if cw is not None:
cw.setTitle(s)
else:
print(s)
if c == 0:
print('same; break')
break
if len(imgs) >= max_pid:
break
return imgs
================================================
FILE: src/extractor/iwara_downloader.py
================================================
import downloader
from utils import Soup, urljoin, Downloader, LazyUrl, get_print, clean_url, clean_title, check_alive, Session, try_n, format_filename, tr_, get_ext, print_error, get_max_range
import ree as re
import errors
import clf2
import hashlib
import urllib
from io import BytesIO
from timee import time
TIMEOUT = 300
PATTERN_ID = r'(image|video)/([0-9a-zA-Z_-]+)'
class Downloader_iwara(Downloader):
type = 'iwara'
URLS = ['iwara.tv']
MAX_CORE = 16#
single = True
display_name = 'Iwara'
ACCEPT_COOKIES = [r'(.*\.)?iwara\.tv']
@classmethod
def fix_url(cls, url):
url = clean_url(url)
return url.split('?')[0]
def init(self):
self.session = Session()
self.setTimeout(TIMEOUT)
def read(self):
info = get_info(self.url, self.session, self.cw)
if info is None:
return # embeded
self.title = clean_title(info['title'])
videos = info['files']
self.single = len(videos) < 2
# first video must be valid
while videos:
video = videos[0]
try:
video.url()
break
except Exception as e:
e_ = e
self.print_(print_error(e))
videos.remove(video)
else:
raise e_
if info.get('playlist', False):
video = self.process_playlist(info['title'], videos)
else: #6031
self.urls += [file.url for file in videos]
self.enableSegment(n_threads=8)
url_thumb = video.url_thumb
self.print_(f'url_thumb: {url_thumb}')
if url_thumb:
f = BytesIO()
downloader.download(url_thumb, buffer=f, session=self.session, customWidget=self.cw)
f.seek(0)
self.setIcon(f)
username = info.get('username')
if username:
self.artist = username
class File:
def __init__(self, type, url, referer, info, session, multi_post=False):
title = info['title']
p = len(info['files'])
self.url = LazyUrl(referer, lambda _: url, self)
ext = get_ext(url) or downloader.get_ext(url, session=session)
if type == 'video':
id_ = re.find(PATTERN_ID, referer, err='no video id')[1]
self.filename = format_filename(title, id_, ext, artist=info.get('username')) #4287, #7127
else:
name = '{}_p{}'.format(clean_title(title), p) if multi_post else p
self.filename = '{}{}'.format(name, ext)
self.url_thumb = info.get('url_thumb')
class LazyFile:
def __init__(self, url, session, cw):
self.session = session
self.cw = cw
self.url = LazyUrl(url, self.get, self)
def get(self, url):
info = get_info(url, self.session, self.cw)
file = info['files'][0]
self.filename = file.filename
self.url_thumb = file.url_thumb
return file.url()
def get_token(session, cw=None): #5794, #6031, #7030
token = None
def f(html, browser=None):
def callback(r):
nonlocal token
token = r
browser.runJavaScript('window.localStorage.getItem("token")', callback=callback)
return bool(token)
clf2.solve('https://iwara.tv', session=session, cw=cw, f=f, timeout=15)
#print_(f'token: {token}')
r = session.post('https://api.iwara.tv/user/token', headers={'Authorization': f'Bearer {token}'})
d = r.json()
token = d['accessToken']
#print_(f'token2: {token}')
return token
@try_n(2)
def get_info(url, session, cw, multi_post=False):
print_ = get_print(cw)
t0 = None
def f(html, browser=None):
nonlocal t0
soup = Soup(html)
if t0 is None:
t0 = time()
if time() - t0 > 10 or '/profile/' in url.lower():
for a in soup.findAll('a'):
if urljoin(url, a.get('href', '')) == urljoin(url, '/login'):
raise errors.LoginRequired(method='browser', url='https://www.iwara.tv/login', cookie=False, w=1460) #5794
buttons = soup.findAll(class_='button--primary')
if buttons:
for i, button in enumerate(buttons):
button_text = button.text
if not button_text:
continue
print_(f'button: {button_text}')
if button_text.lower() in ['i am over 18', 'continue', '私は18歳以上です', '続ける', '继续', '我已满年满 18 岁']: #7534
browser.runJavaScript(f'btns=document.getElementsByClassName("button--primary");btns[{i}].click();') #5794#issuecomment-1517879513
if '/profile/' in url.lower():
return soup.find('div', class_='page-profile__header') is not None
else:
details = soup.find('div', class_='page-video__details')
if details and not soup.find('div', class_='vjs-poster') and not soup.find(class_='embedPlayer__youtube'): #6737, #6836
print_('no poster')
return False
details = details or soup.find('div', class_='page-image__details')
return details is not None and details.find('div', class_='text--h1') is not None
html = clf2.solve(url, session=session, f=f, cw=cw, timeout=30)['html'] #5794
soup = Soup(html)
try:
token = get_token(session, cw=cw)
except Exception as e:
print_(print_error(e))
token = None
hdr = {}
if token:
session.headers['Authorization'] = f'Bearer {token}'
info = {}
info['files'] = []
type = url.split('/')[3]
if type == 'profile':
max_pid = get_max_range(cw)
ids = set()
sub = (url+'/').split('/')[5]
if not sub:
sub = 'videos'
uid = url.split('/')[4]
url_api = f'https://api.iwara.tv/profile/{uid}'
j = downloader.read_json(url_api, session=session)
info['username'] = username = j['user']['name']
info['id'] = id = j['user']['username']
info['title'] = f'[Channel] [{sub.capitalize()}] {username} ({id})'
id = j['user']['id']
if sub == 'videos':
info['playlist'] = True
for p in range(100):
url_api = f'https://api.iwara.tv/videos?sort=date&page={p}&user={id}'
j = downloader.read_json(url_api, session=session)
for post in j['results']:
id_ = post['id']
if id_ in ids:
continue
ids.add(id_)
slug = post['slug']
url_post = f'https://www.iwara.tv/video/{id_}/{slug}'
file = LazyFile(url_post, session, cw)
info['files'].append(file)
if cw: cw.setTitle(tr_('읽는 중... {} ({} / {})').format(info['title'], len(ids), j['count']))
if len(info['files']) >= max_pid:
break
if j['limit']*(p+1) >= j['count']: break
elif sub == 'images':
for p in range(100):
url_api = f'https://api.iwara.tv/images?page={p}&sort=date&user={id}'
j = downloader.read_json(url_api, session=session)
for post in j['results']:
check_alive(cw)
id_ = post['id']
if id_ in ids:
continue
ids.add(id_)
slug = post['slug']
url_post = f'https://www.iwara.tv/image/{id_}/{slug}'
info_post = get_info(url_post, session, cw, True)
info['files'] += info_post['files']
print_(f'imgs: {len(info["files"])}')
if cw: cw.setTitle(tr_('읽는 중... {} ({} / {})').format(info['title'], len(ids), j['count']))
if len(info['files']) >= max_pid:
break
if len(info['files']) >= max_pid:
break
if j['limit']*(p+1) >= j['count']: break
else:
raise NotImplementedError(f'profile: {sub}')
return info
details = soup.find('div', class_='page-video__details') or soup.find('div', class_='page-image__details')
info['title'] = details.find('div', class_='text--h1').text.strip()
info['username'] = soup.find('a', class_='username')['title']
soup.find('div', class_='videoPlayer') or soup.find('div', class_='page-image__slideshow')
id = re.find(PATTERN_ID, url, err='no id')[1]
url_api = f'https://api.iwara.tv/{type}/{id}'
data = downloader.read_json(url_api, url, session=session)
if data.get('embedUrl'):
if cw and not cw.downloader.single:
raise errors.Invalid('[iwara] Embeded: {}'.format(data['embedUrl']))
#5869
cw.downloader.pass_()
cw.gal_num = cw.url = data['embedUrl']
d = Downloader.get('youtube')(data['embedUrl'], cw, cw.downloader.thread, 1)
d.start()
return
if not data.get('files'):
data['files'] = [data['file']]
for file in data['files']:
id_ = file['id']
if type == 'video':
fileurl = data['fileUrl']
up = urllib.parse.urlparse(fileurl)
q = urllib.parse.parse_qs(up.query)
paths = up.path.rstrip('/').split('/')
x_version = hashlib.sha1('_'.join((paths[-1], q['expires'][0], '5nFp9kmbNnHdAFhaqMvt')).encode()).hexdigest() # https://github.com/yt-dlp/yt-dlp/issues/6549#issuecomment-1473771047
j = downloader.read_json(fileurl, url, session=session, headers={'X-Version': x_version})
def key(x):
if x['name'].lower() == 'source':
return float('inf')
try:
return float(x['name'])
except:
return -1
x = sorted(j, key=key)[-1]
print_(f'name: {x["name"]}')
url_file = urljoin(url, x['src']['view'])
poster = soup.find('div', class_='vjs-poster')['style']
info['url_thumb'] = urljoin(url, re.find(r'url\("(.+?)"', poster, err='no poster'))
else:
name = file['name']
url_file = f'https://i.iwara.tv/image/original/{id_}/{name}'
if len(data['files']) == 1:
multi_post = True#
file = File(type, url_file, url, info, session, multi_post)
info['files'].append(file)
return info
================================================
FILE: src/extractor/jmana_downloader.py
================================================
import downloader
from utils import Soup, urljoin, Downloader, fix_title, Session, get_print, LazyUrl, clean_title, get_imgs_already, check_alive, try_n, clean_url
import ree as re
from timee import sleep
from translator import tr_
import page_selector
import bs4
import clf2
PATTERN = r'jmana[0-9]*.*/(comic_list_title|book)\?book'
PATTERN_ALL = r'jmana[0-9]*.*/((comic_list_title|book|bookdetail)\?book|book_by_title\?title)' #6157
PATTERN_ID = '[?&]bookdetailid=([0-9]+)'
class Image:
def __init__(self, url, page, p):
self.url = LazyUrl(page.url, lambda _: url, self)
ext = '.jpg'
name = '{:04}{}'.format(p, ext)
self.filename = '{}/{}'.format(page.title, name)
class Page:
def __init__(self, title, url):
self.title = clean_title(title)
self.url = url
self.id = int(re.find(PATTERN_ID, url))
class Downloader_jmana(Downloader):
type = 'jmana'
URLS = ['regex:'+PATTERN_ALL]
MAX_CORE = 8
_soup = None
def init(self):
self.url = clean_url(self.url)
self.session = Session()
if re.search(PATTERN_ID, self.url): #1799
select = self.soup.find('select', class_='bookselect')
for i, op in enumerate(select.findAll('option')[::-1]):
if 'selected' in op.attrs:
break
else:
raise Exception('no selected option')
for a in self.soup.findAll('a'):
url = urljoin(self.url, a.get('href') or '')
if re.search(PATTERN, url):
break
else:
raise Exception('list not found')
self.url = self.fix_url(url)
self._soup = None
for i, page in enumerate(get_pages(self.url, self.soup, self.session)):
if page.id == int(op['value']):
break
else:
raise Exception('can not find page')
self.cw.range_p = [i]
@classmethod
def fix_url(cls, url):
return url
@property
def soup(self):
if self._soup is None:
res = clf2.solve(self.url, session=self.session) #4070
html = res['html']
soup = Soup(html)
self._soup = soup
return self._soup
@property
def name(self):
title = get_title(self.soup)
artist = get_artist(self.soup)
title = fix_title(self, title, artist)
return title
def read(self):
title = self.name
artist = get_artist(self.soup)
self.artist = artist
for img in get_imgs(self.url, title, self.session, soup=self.soup, cw=self.cw):
if isinstance(img, Image):
self.urls.append(img.url)
else:
self.urls.append(img)
self.title = self.name
def get_title(soup):
a = soup.find('a', class_='tit')
if a:
return a.text.strip()
return re.find(r'제목\s*:\s*(.+)', soup.find('a', class_='tit').text, err='no title')
def get_artist(soup):
return re.find(r'작가\s*:\s*(.+)', soup.text, default='').strip() or 'N/A'
@try_n(4, sleep=60)
def get_imgs_page(page, referer, session, cw=None):
print_ = get_print(cw)
sleep(5, cw) #2017
html = downloader.read_html(page.url, referer, session=session)
inserted = re.find(r'''var\s*inserted\s*=\s*['"](.*?)['"]''', html)
print_('inserted: {}'.format(inserted))
inserted = set(int(i) for i in inserted.split(',')) if inserted else set()
soup = Soup(html)
view = soup.find(class_='pdf-wrap')
imgs = []
for i, img in enumerate(child for child in view.children if isinstance(child, bs4.element.Tag)):
src = img.get('data-src') or img.get('src') or ''
if i in inserted:
print_('remove: {}'.format(src))
continue
if not src:
continue
src = urljoin(page.url, src.strip())
if '/adimg/' in src:
print('adimg:', src)
continue
if '/notice' in src:
print('notice:', src)
continue
img = Image(src, page, len(imgs))
imgs.append(img)
return imgs
def get_pages(url, soup, session):
pages = []
for inner in soup.findAll('div', class_='inner'):
a = inner.find('a')
if not a:
continue
href = a.attrs.get('href', '')
if not re.search(PATTERN_ID, href):
continue
if a.find('img'):
print('skip img', a.attrs.get('href'))
continue
href = urljoin(url, href)
title_page = a.text
page = Page(title_page, href)
pages.append(page)
pages = list(reversed(pages))
return pages
@page_selector.register('jmana')
def f(url, win):
if re.search(PATTERN_ID, url):
raise Exception(tr_('목록 주소를 입력해주세요'))
session = Session()
res = clf2.solve(url, session=session, win=win) #4070
soup = Soup(res['html'])
pages = get_pages(url, soup, session)
return pages
def get_imgs(url, title, session, soup=None, cw=None):
print_ = get_print(cw)
if soup is None:
html = downloader.read_html(url, session=session)
soup = Soup(html)
pages = get_pages(url, soup, session)
print_('pages: {}'.format(len(pages)))
pages = page_selector.filter(pages, cw)
imgs = []
for i, page in enumerate(pages):
check_alive(cw)
imgs_already = get_imgs_already('jmana', title, page, cw)
if imgs_already:
imgs += imgs_already
continue
imgs += get_imgs_page(page, url, session, cw)
if cw is not None:
cw.setTitle('{} {} / {} ({} / {})'.format(tr_('읽는 중...'), title, page.title, i + 1, len(pages)))
if not imgs:
raise Exception('no imgs')
return imgs
================================================
FILE: src/extractor/kakaotv_downloader.py
================================================
import downloader
import ytdl
from utils import Downloader, try_n, LazyUrl, get_ext, format_filename
from io import BytesIO as IO
class Downloader_kakaotv(Downloader):
type = 'kakaotv'
URLS = ['tv.kakao']
single = True
display_name = 'KakaoTV'
ACCEPT_COOKIES = [r'(.*\.)?kakao\.com']
@classmethod
def fix_url(cls, url):
url = url.replace('.kakao.com/m/', '.kakao.com/')
return url.split('?')[0].strip('/')
def read(self):
video = Video(self.url, cw=self.cw)
video.url()#
self.urls.append(video.url)
self.setIcon(video.thumb)
self.enableSegment()
self.title = video.title
class Video:
_url = None
def __init__(self, url, cw=None):
self.url = LazyUrl(url, self.get, self)
self.cw = cw
@try_n(2)
def get(self, url):
if self._url:
return self._url
ydl = ytdl.YoutubeDL(cw=self.cw)
info = ydl.extract_info(url)
fs = [f for f in info['formats'] if f['ext'] == 'mp4']
f = sorted(fs, key=lambda f: f['height'])[-1]
self._url = f['url']
self.thumb_url = info['thumbnails'][0]['url']
self.thumb = IO()
downloader.download(self.thumb_url, buffer=self.thumb)
self.title = info['title']
ext = get_ext(self._url)
self.filename = format_filename(self.title, info['id'], ext)
return self._url
================================================
FILE: src/extractor/kakuyomu_downloader.py
================================================
#coding:utf8
import downloader
import utils
from utils import Soup, urljoin, Downloader, LazyUrl, try_n, clean_title, get_print, json, File
import os
from io import BytesIO
from translator import tr_
class Page(File):
type = 'kakuyomu'
format = 'title'
def __init__(self, info):
info['title_all'] = clean_title('[{:04}] {}'.format(info['p'], info['title']))
d = {
'title': info['title_all'],
}
info['name'] = utils.format(self.type, d, '.txt')
super().__init__(info)
def get(self):
text = get_text(self)
f = BytesIO()
f.write(text.encode('utf8'))
f.seek(0)
return {'url': f}
class Downloader_kakuyomu(Downloader):
type = 'kakuyomu'
URLS = ['kakuyomu.jp']
MAX_CORE = 2
detect_removed = False
display_name = 'カクヨム'
ACCEPT_COOKIES = [r'(.*\.)?kakuyomu\.jp']
atts = ['info_title', 'info_description']
def read(self):
self.info = get_info(self.url, cw=self.cw)
self.artist = self.info['artist']
title_dir = clean_title('[{}] {}'.format(self.artist, self.info['title']))
outdir = utils.dir(self.type, title_dir, self.cw)
self.urls += self.info['pages']
self.title = title_dir
self.info_title = self.info['title']
self.info_description = self.info['description']
def post_processing(self):
names = self.cw.names
filename = clean_title('[merged] [{}] {}'.format(self.artist, self.info_title), n=-4) + '.txt'
filename = os.path.join(self.dir, filename)
try:
with utils.open(filename, 'wb') as f:
f.write(' {}\n\n 作者：{}\n\n\n'.format(self.info_title, self.artist).encode('utf8'))
f.write(self.info_description.encode('utf8'))
for i, file in enumerate(names):
self.cw.pbar.setFormat('[%v/%m] {} [{}/{}]'.format(tr_('병합...'), i, len(names)))
with open(file, 'rb') as f_:
text = f_.read()
f.write(b'\n\n\n\n')
f.write(text)
finally:
self.cw.pbar.setFormat('[%v/%m]')
@try_n(4, sleep=30)
def get_text(page):
html = downloader.read_html(page['referer'])
soup = Soup(html)
view = soup.find('div', class_='widget-episodeBody')
story = utils.get_text(view, '')
text = '''────────────────────────────────
◆ {} {}
────────────────────────────────
{}'''.format(page['title_all'], page['date'], story)
return text
def get_info(url, soup=None, cw=None):
print_ = get_print(cw)
if soup is None:
html = downloader.read_html(url)
soup = Soup(html)
info = {}
rdata = soup.find('script', id='__NEXT_DATA__').string #6620
data = json.loads(rdata)
wid = data['query']['workId']
info['title'] = data['props']['pageProps']['__APOLLO_STATE__'][f'Work:{wid}']['title']
aid = data['props']['pageProps']['__APOLLO_STATE__'][f'Work:{wid}']['author']['__ref']
info['artist'] = data['props']['pageProps']['__APOLLO_STATE__'][f'{aid}']['activityName']
catch = data['props']['pageProps']['__APOLLO_STATE__'][f'Work:{wid}'].get('catchphrase') or ''
intro = data['props']['pageProps']['__APOLLO_STATE__'][f'Work:{wid}'].get('introduction') or ''
desc = ' {}{}'.format(catch, ('\n\n\n'+intro) if intro else '')
info['description'] = desc
eps = []
for tc in data['props']['pageProps']['__APOLLO_STATE__'][f'Work:{wid}']['tableOfContents']:
_ = data['props']['pageProps']['__APOLLO_STATE__'][tc['__ref']].get('episodes')
if _:
eps += _
else: #6708
eps += data['props']['pageProps']['__APOLLO_STATE__'][tc['__ref']]['episodeUnions']
pages = []
for ep in eps:
eid = ep['__ref'].split('Episode:')[1]
href = urljoin(url, f'/works/{wid}/episodes/{eid}')
subtitle = data['props']['pageProps']['__APOLLO_STATE__'][ep['__ref']]['title']
date = data['props']['pageProps']['__APOLLO_STATE__'][ep['__ref']]['publishedAt']
page = Page({'referer': href, 'title': subtitle, 'date': date, 'p': len(pages)+1})
pages.append(page)
info['pages'] = pages
return info
================================================
FILE: src/extractor/kissjav_downloader.py
================================================
import downloader
from utils import urljoin, Downloader, LazyUrl, Session, try_n, format_filename, get_resolution, get_print
import ree as re
from io import BytesIO
import clf2
class Downloader_kissjav(Downloader):
type = 'kissjav'
URLS = ['kissjav.com', 'kissjav.li', 'mrjav.net'] #4835
single = True
display_name = 'KissJAV'
ACCEPT_COOKIES = [r'(.*\.)?(kissjav|mrjav)\.(com|li|net)']
def read(self):
self.session = Session()#get_session(self.url, cw=self.cw)
video = get_video(self.url, self.session, self.cw)
self.urls.append(video.url)
self.setIcon(video.thumb)
self.enableSegment(1024*1024//2)
self.title = video.title
@try_n(2)
def get_video(url, session, cw):
print_ = get_print(cw)
soup = downloader.read_soup(url, session=session)
view = soup.find('div', id='player-container-fluid')
fs = []
for source in view.findAll('source'):
src = urljoin(url, source.attrs['src'])
res = re.find('([0-9]+)p', source.attrs['title'])
res = int(res) if res else 0
f = {'res': res, 'src': src}
fs.append(f)
print_(f)
if not fs:
raise Exception('No source')
#4773
res = max(get_resolution(), min(f['res'] for f in fs))
print_(f'res: {res}')
fs = sorted([f for f in fs if f['res'] <= res], key=lambda f: f['res'])
f = fs[-1]
print_(f'best: {f}')
src_best = f['src']
title = soup.find('h1').text.strip()
id = soup.find('div', id='video').attrs['data-id']
url_thumb = soup.find('meta', {'property': 'og:image'}).attrs['content']
#src_best = downloader.real_url(src_best)
video = Video(src_best, url_thumb, url, title, id, session)
return video
class Video:
def __init__(self, url, url_thumb, referer, title, id, session):
self.title = title
self.filename = format_filename(title, id, '.mp4')
self.url = LazyUrl(referer, lambda x: url, self)
self.thumb = BytesIO()
self.url_thumb = url_thumb
downloader.download(url_thumb, buffer=self.thumb, session=session)
@try_n(2)
def get_session(url, cw=None):
session = Session()
clf2.solve(url, session=session, cw=cw)
return session
================================================
FILE: src/extractor/lhscan_downloader.py
================================================
#coding:utf8
import downloader
from utils import Soup, urljoin, LazyUrl, Downloader, try_n, Session, clean_title, get_print, check_alive
import os
from translator import tr_
import page_selector
import clf2
import utils
import base64
import ree as re
import errors
##from image_reader import QPixmap
class Image:
def __init__(self, url, page, p):
self._url = url
self.url = LazyUrl(page.url, self.get, self)#, pp=self.pp)
ext = os.path.splitext(url)[1]
if ext.lower()[1:] not in ['jpg', 'jpeg', 'bmp', 'png', 'gif', 'webm', 'webp']:
ext = '.jpg'
self.filename = '{}/{:04}{}'.format(page.title, p, ext)
def get(self, _):
return self._url
## def pp(self, filename):
## pixmap = QPixmap(filename)
## pixmap.save(filename)
## return filename
class Page:
def __init__(self, title, url):
self.title = clean_title(title)
self.url = url
def get_soup_session(url, cw=None, win=None):
print_ = get_print(cw)
session = Session()
res = clf2.solve(url, session=session, cw=cw, win=win)
print_('{} -> {}'.format(url, res['url']))
if res['url'].rstrip('/') == 'https://welovemanga.one':
raise errors.LoginRequired()
return Soup(res['html']), session
class Downloader_lhscan(Downloader):
type = 'lhscan'
URLS = [
#'lhscan.net', 'loveheaven.net',
'lovehug.net', 'welovemanga.', 'nicomanga.com',
]
MAX_CORE = 16
display_name = 'LHScan'
ACCEPT_COOKIES = [rf'(.*\.)?{domain}' for domain in URLS]
def init(self):
self.soup, self.session = get_soup_session(self.url, self.cw)
try:
self.name
except:
raise errors.Invalid('{}: {}'.format(tr_('목록 주소를 입력해주세요'), self.url))
@classmethod
def fix_url(cls, url):
url = url.replace('lovehug.net', 'welovemanga.one')
url = url.replace('welovemanga.net', 'welovemanga.one') #4298
return url
@property
def name(self):
title = self.soup.find('ul', class_='manga-info').find('h3').text
return clean_title(title)
def read(self):
self.title = tr_('읽는 중... {}').format(self.name)
imgs = get_imgs(self.url, self.name, self.session, self.soup, self.cw)
for img in imgs:
self.urls.append(img.url)
self.title = self.name
@try_n(8)
def get_imgs_page(page, referer, session, cw=None):
print_ = get_print(cw)
print_(page.title)
html = downloader.read_html(page.url, referer, session=session)
if clf2._is_captcha(Soup(html)): #4124
html = clf2.solve(page.url, session, cw)['html']
if not html:
raise Exception('empty html')
try:
html = html.replace('{}='.format(re.find(r"\$\(this\)\.attr\('(.+?)'", html, err='no cn')), 'data-src=')
except: #5351
pass
soup = Soup(html)
m = re.find(r'''(load_image|imgsListchap)\(([0-9]+)''', html)
if m: #6186
cid = m[1]
if utils.domain(page.url, 2).lower() == 'nicomanga.com':
url_api = urljoin(page.url, f'/app/manga/controllers/cont.imgsList.php?cid={cid}')
else:
url_api = urljoin(page.url, f'/app/manga/controllers/cont.listImg.php?cid={cid}')
soup = downloader.read_soup(url_api, page.url, session=session)
imgs = []
for img in soup.findAll('img', class_='chapter-img'):
src = img.get('data-pagespeed-lazy-src') or img.get('data-src') or img.get('data-srcset') or img.get('data-aload') or img.get('data-original') or img['src']
try:
src = base64.b64decode(src).strip().decode('utf8')
except:
pass
src0 = src
src = src.replace('welovemanga.one', '1')#
src = urljoin(page.url, src).strip()
if 'Credit_LHScan_' in src or '5e1ad960d67b2_5e1ad962338c7' in src:
continue
if 'fe132b3d32acc39f5adcea9075bedad4LoveHeaven' in src:
continue
if 'LoveHug_600cfd96e98ff.jpg' in src:
continue
if 'image_5f0ecf23aed2e.png' in src:
continue
if '/uploads/lazy_loading.gif' in src:
continue
if '/xstaff.jpg.pagespeed.ic.gPQ2SGcYaN.webp' in src:
continue
if '/uploads/loading-mm.gif' in src:
continue
src = src.replace('\n', '').replace('\r', '') #5238
#6105
## if 'proxy.php?link=' not in src: #5351
## src = 'https://welovekai.com/proxy.php?link=' + src #5238
if not imgs:
print_(src0)
print_(src)
img = Image(src, page, len(imgs))
imgs.append(img)
return imgs
def get_pages(url, session, soup=None, cw=None):
if soup is None:
html = downloader.read_html(url, session=session)
soup = Soup(html)
tab = soup.find('ul', class_='list-chapters')
pages = []
for li in tab.findAll('li'):
text = li.find('div', class_='chapter-name').text.strip()
href = li.parent['href']
href = urljoin(url, href)
page = Page(text, href)
pages.append(page)
if not pages:
raise Exception('no pages')
return pages[::-1]
@page_selector.register('lhscan')
def f(url, win):
soup, session = get_soup_session(url, win=win)
pages = get_pages(url, session, soup=soup)
return pages
@try_n(2)
def get_imgs(url, title, session, soup=None, cw=None):
if soup is None:
html = downloader.read_html(url, session=session)
soup = Soup(html)
pages = get_pages(url, session, soup, cw)
pages = page_selector.filter(pages, cw)
imgs = []
for i, page in enumerate(pages):
check_alive(cw)
imgs += get_imgs_page(page, url, session, cw)
s = '{} {} / {} ({} / {})'.format(tr_('읽는 중...'), title, page.title, i+1, len(pages))
if cw is not None:
cw.setTitle(s)
else:
print(s)
return imgs
================================================
FILE: src/extractor/luscious_downloader.py
================================================
#coding:utf8
import downloader
import utils
from utils import Soup, Downloader, LazyUrl, urljoin, try_n, clean_title, get_max_range, json, print_error
import ree as re
import os
from translator import tr_
from io import BytesIO
import clf2
import errors
downloader.REPLACE_UA[r'\.luscious\.net'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36'
class LoginRequired(errors.LoginRequired):
def __init__(self, *args):
super().__init__(*args, method='browser', url='https://members.luscious.net/login/')
class Image:
def __init__(self, item, referer):
self.item = item
self.id = str(item['id'])
self.referer = referer
self.url = LazyUrl(referer, self.get, self)
def get(self, url):
img = urljoin(url, self.item['url_to_original'])
ext = os.path.splitext(img.split('?')[0])[1]
self.filename = '{}{}'.format(self.id, ext)
return img
class Video:
id = None
def __init__(self, url, title, url_thumb):
self.url = url
self.title = title
ext = os.path.splitext(url.split('?')[0])[1]
self.filename = '{}{}'.format(title, ext)
self.url_thumb = url_thumb
self.thumb = BytesIO()
downloader.download(self.url_thumb, buffer=self.thumb)
class Downloader_luscious(Downloader):
type = 'luscious'
URLS = ['luscious.net']
MAX_CORE = 4
ACCEPT_COOKIES = [r'(.*\.)?luscious\.net']
@classmethod
def fix_url(cls, url):
url = url.replace('members.luscious.', 'www.luscious.')
return url
@classmethod
def key_id(cls, url):
return '/'.join(url.split('/')[3:])
def read(self):
shown = True
def f(html, browser=None):
nonlocal shown
soup = Soup(html)
if soup.find(class_='http-error-404-page-container'):
raise LoginRequired(get_title(soup, False)) #6912
if soup.find('input', type='password') or soup.find('img', class_='warning-page-image'):
if not shown:
shown = True
browser.show()
return False
if shown:
shown = False
browser.hide()
try:
get_title(soup)
except Exception as e:
self.print_(print_error(e))
return False
return True
res = clf2.solve(self.url, f=f, cw=self.cw, show=True)
self.url = res['url']
soup = Soup(res['html'])
title = get_title(soup)
self.title = tr_('읽는 중... {}').format(title)
if '/videos/' in self.url:
video = get_video(self.url, soup)
imgs = [video]
self.setIcon(video.thumb)
else:
imgs = get_imgs(self.url, soup, self.cw)
dir = utils.dir(self.type, title, self.cw)
names = {}
try:
for name in os.listdir(dir):
id = os.path.splitext(name)[0]
names[id] = name
except:
pass
for img in imgs:
if img.id in names:
url = os.path.join(dir, names[img.id])
else:
url = img.url
self.urls.append(url)
self.title = title#
def update(cw, title, imgs):
s = '{} {} - {}'.format(tr_('읽는 중...'), title, len(imgs))
if cw is not None:
cw.setTitle(s)
else:
print(s)
def get_imgs(url, soup=None, cw=None):
if soup is None:
html = downloader.read_html(url)
soup = Soup(html)
title = get_title(soup)
n = get_max_range(cw)
imgs = []
p = 1
while True:
imgs_new, has_next_page = get_imgs_p(url, p)
if not imgs_new:
break
imgs += imgs_new
update(cw, title, imgs)
p += 1
if len(imgs) >= n or not has_next_page:
break
return imgs[:n]
@try_n(4, sleep=30)
def get_imgs_p(url, p=1):
id = re.find('/albums/[^/]+?([0-9]+)/', url+'/')
print(url, id)
#5699
#url_api = 'https://api.luscious.net/graphql/nobatch/?operationName=AlbumListOwnPictures&query=+query+AlbumListOwnPictures%28%24input%3A+PictureListInput%21%29+%7B+picture+%7B+list%28input%3A+%24input%29+%7B+info+%7B+...FacetCollectionInfo+%7D+items+%7B+...PictureStandardWithoutAlbum+%7D+%7D+%7D+%7D+fragment+FacetCollectionInfo+on+FacetCollectionInfo+%7B+page+has_next_page+has_previous_page+total_items+total_pages+items_per_page+url_complete+%7D+fragment+PictureStandardWithoutAlbum+on+Picture+%7B+__typename+id+title+created+like_status+number_of_comments+number_of_favorites+status+width+height+resolution+aspect_ratio+url_to_original+url_to_video+is_animated+position+tags+%7B+category+text+url+%7D+permissions+url+thumbnails+%7B+width+height+size+url+%7D+%7D+&variables=%7B%22input%22%3A%7B%22filters%22%3A%5B%7B%22name%22%3A%22album_id%22%2C%22value%22%3A%22{}%22%7D%5D%2C%22display%22%3A%22position%22%2C%22page%22%3A{}%7D%7D'.format(id, p)
url_api = f'https://apicdn.luscious.net/graphql/nobatch/?operationName=AlbumListOwnPictures&query=%2520query%2520AlbumListOwnPictures%28%2524input%253A%2520PictureListInput%21%29%2520%257B%2520picture%2520%257B%2520list%28input%253A%2520%2524input%29%2520%257B%2520info%2520%257B%2520...FacetCollectionInfo%2520%257D%2520items%2520%257B%2520__typename%2520id%2520title%2520description%2520created%2520like_status%2520number_of_comments%2520number_of_favorites%2520moderation_status%2520width%2520height%2520resolution%2520aspect_ratio%2520url_to_original%2520url_to_video%2520is_animated%2520position%2520tags%2520%257B%2520category%2520text%2520url%2520%257D%2520permissions%2520url%2520thumbnails%2520%257B%2520width%2520height%2520size%2520url%2520%257D%2520%257D%2520%257D%2520%257D%2520%257D%2520fragment%2520FacetCollectionInfo%2520on%2520FacetCollectionInfo%2520%257B%2520page%2520has_next_page%2520has_previous_page%2520total_items%2520total_pages%2520items_per_page%2520url_complete%2520%257D%2520&variables=%7B%22input%22%3A%7B%22filters%22%3A%5B%7B%22name%22%3A%22album_id%22%2C%22value%22%3A%22{id}%22%7D%5D%2C%22display%22%3A%22rating_all_time%22%2C%22items_per_page%22%3A50%2C%22page%22%3A{p}%7D%7D'
data_raw = downloader.read_html(url_api, referer=url)
data = json.loads(data_raw)
imgs = []
for item in data['data']['picture']['list']['items']:
img = Image(item, url)
imgs.append(img)
return imgs, data['data']['picture']['list']['info']['has_next_page']
def get_video(url, soup):
title = re.find('videos/([^/]+)', url)
video = soup.find('video')
url = urljoin(url, video.source.attrs['src'])
url_thumb = urljoin(url, video['poster'])
video = Video(url, title, url_thumb)
return video
def get_title(soup, id=True):
title = soup.find('h1').text.strip()
if not id:
return title
id = re.find(r'/moderation/flag/album/[0-9]+/.+id=([0-9]+)', soup.html, err='no id') #7026
title = clean_title(title, n=-len(f' ({id})'))
return f'{title} ({id})'
================================================
FILE: src/extractor/m3u8_downloader.py
================================================
from utils import Downloader, LazyUrl, clean_title, Session, get_ext
import utils
from m3u8_tools import playlist2stream, M3u8_stream
import os
from hashlib import md5
from translator import tr_
DEFAULT_N_THREAD = 2
def suitable(url):
return get_ext(url).lower() in ('.m3u8', '.mpd')
class Downloader_m3u8(Downloader):
type = 'm3u8'
URLS = [suitable]
single = True
display_name = 'M3U8'
@classmethod
def fix_url(cls, url):
if '://' not in url:
url = 'http://' + url
return url
def read(self):
fmt = self.cw.format
referer = self.url
if isinstance(fmt, str) and fmt.startswith('referer:'):
referer = fmt[len('referer:'):]
self.print_('referer: {}'.format(referer))
n_thread = DEFAULT_N_THREAD
if isinstance(fmt, int) and fmt > 0:
n_thread = fmt
self.print_('n_thread: {}'.format(n_thread))
video = Video(self.url, n_thread, referer)
self.urls.append(video.url)
self.title = os.path.splitext(os.path.basename(video.filename))[0].replace(b'\xef\xbc\x9a'.decode('utf8'), ':')
class Video:
def __init__(self, url, n_thread, referer):
session = Session()
session.purge([rf'(.*\.)?{utils.domain(url)}'])
if get_ext(url).lower() == '.mpd':
def m():
hdr = session.headers.copy()
if referer:
hdr['Referer'] = referer
return utils.LiveStream(url, headers=hdr)
ms = [m]
else:
ms = [
lambda: playlist2stream(url, n_thread=n_thread, session=session),
lambda: M3u8_stream(url, n_thread=n_thread, session=session),
]
for m in ms:
try:
m = m()
break
except Exception as e:
e_ = e
else:
raise e_
if getattr(m, 'live', None) is not None: #5110
#m = m.live
hdr = session.headers.copy()
if referer:
hdr['Referer'] = referer
m = utils.LiveStream(url, headers=hdr)
live = True
else:
live = False
self.url = LazyUrl(url, lambda _: m, self)
self.title = os.path.splitext(os.path.basename(url).split('?')[0])[0][:50]
self.id_ = md5(url.encode('utf8')).hexdigest()[:8]
tail = f' ({self.id_}).mp4'
if live: #5110
from datetime import datetime
tail = ' ' + clean_title(datetime.now().strftime('%Y-%m-%d %H:%M')) + tail
self.filename = clean_title(self.title, n=-len(tail)) + tail
import selector
@selector.options('m3u8')
def options(urls):
def f(urls):
n_thread, ok = utils.QInputDialog.getInt(Downloader.mainWindow, tr_('Set number of threads'), tr_('Number of threads?'), value=DEFAULT_N_THREAD, min=1, max=4, step=1)
if not ok:
return
return n_thread
def f2(urls):
referer, ok = utils.QInputDialog.getText(Downloader.mainWindow, tr_('Set a referer'), tr_('Referer?'))
if not ok:
return
return 'referer:'+referer
return [
{'text': 'Set the number of threads...', 'format': f},
{'text': 'Set the referer...', 'format': f2},
]
================================================
FILE: src/extractor/manatoki_downloader.py
================================================
from utils import Soup, try_n, Downloader, urljoin, get_print, Session, clean_title, get_ext, fix_title, lazy, get_imgs_already, check_alive, File, limits
from translator import tr_
import page_selector
import utils
import clf2
import ree as re
from PIL import Image
class File_manatoki(File):
type = 'manatoki'
format = 'title/page:04;'
show_pp = False
def __init__(self, info):
ext = get_ext(info['url'])
if ext.lower()[1:] not in ['jpg', 'jpeg', 'bmp', 'png', 'gif', 'webm', 'webp']:
ext = '.jpg'
d = {
'title': info['title'],
'page': info['page'],
'chapterid': re.find(r'/comic/([0-9]+)', info['referer']), #6380
}
info['name'] = utils.format('manatoki', d, ext)
super().__init__(info)
@limits(.5)
def get(self):
return {}
def pp(self, filename): #5233
img = Image.open(filename)
nf = getattr(img, 'n_frames', 1)
loop = img.info.get('loop')
if nf > 1 and loop:
img.seek(nf-1)
img.save(filename)
img.close()
return filename
class Page:
def __init__(self, title, url):
self.title = clean_title(title)
self.url = url
self.id = int(re.find(r'/(comic|webtoon)/([0-9]+)', url, err='no id')[1])
class Downloader_manatoki(Downloader):
type = 'manatoki'
URLS = [r'regex:(mana|new)toki[0-9]*\.(com|net)']
MAX_CORE = 4
ACCEPT_COOKIES = [r'(.*\.)?(mana|new)toki[0-9]*\.(com|net)']
@try_n(2)
def init(self):
self.session, self.soup, url = get_soup(self.url, cw=self.cw)
self.url = self.fix_url(url)
# 2377
list = self.soup.find(attrs={'data-original-title': '목록'})
if list:
url = urljoin(self.url, list.parent['href'])
nav = self.soup.find('div', class_='toon-nav')
select = nav.find('select', {'name': 'wr_id'})
for i, op in enumerate(select.findAll('option')[::-1]):
if 'selected' in op.attrs:
break
else:
raise Exception('no selected option')
self.session, self.soup, url = get_soup(url, cw=self.cw)
url_page = self.fix_url(url)
for i, page in enumerate(get_pages(url_page, self.soup)):
if page.id == int(op['value']):
break
else:
raise Exception('can not find page')
self.cw.range_p = [i]
self.url = url_page
self.name
@classmethod
def fix_url(cls, url):
# 2377
m = re.find(r'/board.php\?bo_table=([0-9a-zA-Z_]+)&wr_id=([0-9]+)', url)
if m:
return urljoin(url, '/{}/{}'.format(*m))
return url.split('?')[0]
@classmethod
def key_id(cls, url):
return '/'.join(url.split('/')[3:5])
@lazy
def name(self):
artist = get_artist(self.soup)
title = self.soup.find('meta', {'name':'subject'})['content'].strip()
return fix_title(self, title, artist)
def read(self):
self.title = tr_('읽는 중... {}').format(self.name)
self.artist = get_artist(self.soup)
imgs = get_imgs(self.url, self.name, self.soup, self.session, self.cw)
for img in imgs:
self.urls.append(img)
self.title = self.name
def get_artist(soup):
view = soup.find('div', class_='view-title', err='no title')
text = view.text.replace('\n', '#')
artist = re.find(r'작가[ #]*:[ #]*(.+?)#', text, default='N/A').strip()
return artist
@limits(10)
def get_soup(url, session=None, cw=None, win=None):
if session is None:
session = Session()
virgin = True
def f(html, browser=None):
nonlocal virgin
soup = Soup(html)
if soup.find('form', {'name':'fcaptcha'}): #4660
browser.show()
if virgin:
virgin = False
browser.runJavaScript('window.scrollTo({top: document.getElementsByClassName("form-box")[0].getBoundingClientRect().top-150})') #5504
return False
if not virgin: #7158
browser.hide()
return True
res = clf2.solve(url, session=session, f=f, cw=cw, win=win)
soup = Soup(res['html'], apply_css=True)
return session, soup, res['url']
def get_pages(url, soup, sub=False):
list = soup.find('ul', class_='list-body')
pages = []
for item in list.findAll('div', 'wr-subject'):
for span in item.a.findAll('span'):
span.decompose()
title = item.a.text.strip()
href = item.a['href']
href = urljoin(url, href)
pages.append((title, href))
if not pages:
raise Exception('no pages')
## if sub: #4909
## return pages
## else:
## pg = soup.find('ul', class_='pagination')
## as_ = pg.findAll('a')
## for a in as_:
## href = a.get('href')
## if not href:
## continue
## href = urljoin(url, href)
## for try_ in range(2):
## try:
## session, soup2, href = get_soup(href)
## pages += get_pages(href, soup2, sub=True)
## break
## except Exception as e:
## e_ = e
## print(e)
## else:
## raise e_
titles = {}
pages_ = []
for title, href in pages[::-1]:
title = utils.fix_dup(title, titles) #4161
page = Page(title, href)
pages_.append(page)
return pages_
@page_selector.register('manatoki')
def f(url, win):
session, soup, url = get_soup(url, win=win)
list = soup.find('ul', class_='list-body')
if list is None:
raise Exception(tr_('목록 주소를 입력해주세요'))
pages = get_pages(url, soup)
return pages
def get_imgs(url, title, soup=None, session=None, cw=None):
print_ = get_print(cw)
if soup is None or session is None:
session, soup, url = get_soup(url, session, cw)
pages = get_pages(url, soup)
pages = page_selector.filter(pages, cw)
imgs = []
for i, page in enumerate(pages):
check_alive(cw)
imgs_already = get_imgs_already('manatoki', title, page, cw)
if imgs_already:
imgs += imgs_already
continue
imgs_ = get_imgs_page(page, title, url, session, cw)
imgs += imgs_
s = '{} {} / {} ({} / {})'.format(tr_('읽는 중...'), title, page.title, i+1, len(pages))
print_('{} {}'.format(page.title, len(imgs_)))
if cw is not None:
cw.setTitle(s)
else:
print('read page... {} ({})'.format(page.url, len(imgs)))
return imgs
@try_n(4)
def get_imgs_page(page, title, referer, session, cw):
print_ = get_print(cw)
# 2183
session, soup, page.url = get_soup(page.url, session, cw)
title_page = page.title#clean_title(soup.find('span', class_='page-desc').text.strip())
if page.title != title_page:
print_('{} -> {}'.format(page.title, title_page))
page.title = title_page
views = soup.findAll('div', class_='view-content')\
+ soup.findAll('div', class_='view-padding')
if not views:
raise Exception('no views')
hash = re.find(r'''data_attribute\s*:\s*['"](.+?)['"]''', soup.html)
print_('hash: {}'.format(hash))
if hash is None:
raise Exception('no hash')
imgs = []
for view in views:
if view is None:
continue
for img in view.findAll('img'):
if not isVisible(img):
continue
src = img.get('data-{}'.format(hash))
src = src or img.get('content') # https://manatoki77.net/comic/5266935
if not src:
continue
img = urljoin(page.url, src)
if '/img/cang' in img:
continue
if '/img/blank.gif' in img:
continue
img = File_manatoki({'referer': page.url, 'url': img, 'title': page.title, 'page': len(imgs)})
imgs.append(img)
## if not imgs:
## raise Exception('no imgs')
return imgs
def isVisible(tag):
while tag:
if re.search(r'display:\s*none', tag.get('style', ''), re.I):
return False
tag = tag.parent
return True
================================================
FILE: src/extractor/mastodon_downloader.py
================================================
#coding:utf8
from utils import Downloader, clean_title, Session
from mastodon import get_info
import ree as re
def get_id(url):
return re.find('mastodon.social/([^/]+)', url.lower())
class Downloader_mastodon(Downloader):
type = 'mastodon'
URLS = ['mastodon.social']
ACCEPT_COOKIES = [r'(.*\.)?mastodon\.social']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
id_ = get_id(url) or url
return f'https://mastodon.social/{id_}'
def read(self):
id_ = get_id(self.url)
info = get_info('mastodon.social', id_, f'mastodon_{id_}', self.session, self.cw)
self.urls += info['files']
self.title = clean_title('{} (mastodon_{})'.format(info['title'], id_))
================================================
FILE: src/extractor/misskey_downloader.py
================================================
from utils import Downloader, Session, clean_title, get_ext, check_alive, tr_, try_n, File, get_max_range, limits
import downloader
import ree as re
from datetime import datetime
import utils
import errors
DOMAIN = 'misskey.io'
SUBFOLDER = True
class File_misskey(File):
type = 'misskey'
format = '[date] id_ppage'
def get_file(nid, url, referer, session, p, time):
ext = get_ext(url) or downloader.get_ext(url, session, referer)
d = {
'date': time,
'id': nid,
'page': p,
}
filename = utils.format('misskey', d, ext)
info = {'name': filename, 'url': url, 'referer': referer}
return File_misskey(info)
def get_time(note):
ds = note['createdAt']
time = datetime.strptime(ds.split('.')[0], '%Y-%m-%dT%H:%M:%S')
time = (time-datetime(1970,1,1)).total_seconds()
return time
class Downloader_misskey(Downloader):
type = 'misskey'
URLS = [f'{DOMAIN}/notes/', f'{DOMAIN}/@']
display_name = 'Misskey'
ACCEPT_COOKIES = [rf'(.*\.)?{DOMAIN}']
MAX_CORE = 8
@classmethod
def fix_url(cls, url):
if DOMAIN.lower() in url.lower() and '://' not in url:
url = 'https://' + url
if url.startswith('@'):
url = f'https://{DOMAIN}/{url}'
return url
def init(self):
self.session = Session()
if f'{DOMAIN}/notes/' in self.url:
raise errors.Invalid(tr_('개별 다운로드는 지원하지 않습니다: {}').format(self.url))
@try_n(4, sleep=5)
@limits(2)
def call(self, path, payload):
token = self.session.cookies.get('token', domain=DOMAIN)
url_api = f'https://{DOMAIN}/api/{path}'
if token:
payload['i'] = token
r = self.session.post(url_api, json=payload)
d = r.json()
if isinstance(d, dict):
err = d.get('error')
if err:
raise errors.Invalid(err['message'])
return d
def read(self):
nid = re.find(rf'{DOMAIN}/notes/([^/]+)', self.url)
if nid:
self.single = True
data = {'noteId':nid,
}
note = self.call('notes/show', data)
username = note['user']['username']
self.artist = note['user']['name'] or username
host = note['user']['host']
if host:
username += f'@{host}'
self.title = f'{clean_title(self.artist)} (misskey_@{username})'
time = get_time(note)
for file in note['files']:
file = get_file(note['id'], file['url'], self.url, self.session, len(self.urls), time)
if SUBFOLDER:
file['name'] = self.title + '/' + file['name']
self.urls.append(file)
else:
username = re.find(rf'{DOMAIN}/@([a-zA-Z0-9_@\.]+)', self.url, err='no username')
if '@' in username:
username, host = username.split('@')
else:
host = None
data = {"username":username,
"host":host,
}
d = self.call('users/show', data)
username = d['username']
self.artist = d['name'] or username
host = d['host'] or None
if host:
username += f'@{host}'
uid = d['id']
self.title = title = f'{clean_title(self.artist)} (misskey_@{username})'
untilId = None
nids = set()
n = get_max_range(self.cw)
while check_alive(self.cw):
data = {"userId":uid,
"limit":30,
}
if untilId:
data["untilId"] = untilId
d = self.call('users/notes', data)
if not d:
break
for note in d:
nid = note['id']
if nid in nids:
continue
nids.add(nid)
time = get_time(note)
url_note = f'https://{DOMAIN}/notes/{nid}'
for p, file in enumerate(note['files']):
file = get_file(note['id'], file['url'], url_note, self.session, p, time)
self.urls.append(file)
untilId = nid
self.cw.setTitle(f'{tr_("읽는 중...")} {title} - {len(self.urls)}')
if len(self.urls) >= n:
break
self.title = title
================================================
FILE: src/extractor/mrm_downloader.py
================================================
#coding:utf8
from utils import Soup, urljoin, LazyUrl, Downloader, try_n, get_print, clean_title, get_ext, check_alive
from translator import tr_
import ree as re
import clf2#
class Image:
def __init__(self, url, p, page, cw):
self.cw = cw
ext = get_ext(url)
self.filename = '{:04}{}'.format(p, ext)
if page.title is not None:
self.filename = '{}/{}'.format(page.title, self.filename)
self._url = url
self.url = LazyUrl(page.url, self.get, self)
def get(self, _):
return self._url#'tmp://' + clf2.download(self._url, cw=self.cw)
class Page:
def __init__(self, title, url, soup=None):
self.title = clean_title(title)
self.url = url
self.soup = soup
class Downloader_mrm(Downloader):
type = 'mrm'
URLS = ['myreadingmanga.info']
_soup = None
MAX_CORE = 4
display_name = 'MyReadingManga'
ACCEPT_COOKIES = [r'(.*\.)?myreadingmanga\.info']
def init(self):
self.session = get_session(self.url, self.cw)
@classmethod
def fix_url(cls, url):
return re.find('https?://myreadingmanga.info/[^/]+', url, err='err')
@property
def soup(self):
if self._soup is None:
for try_ in range(8):
try:
html = read_html(self.url, session=self.session, cw=self.cw)
break
except Exception as e:
e_ = e
self.print_(e)
else:
raise e_
self._soup = Soup(html)
return self._soup
@property
def name(self):
title = get_title(self.soup)
return title
def read(self):
self.title = '읽는 중... {}'.format(self.name)
imgs = get_imgs(self.url, self.soup, self.session, self.cw)
for img in imgs:
self.urls.append(img.url)
self.title = self.name
def get_title(soup):
title = soup.find('h1', class_='entry-title').text.strip()
title = fix_title(title)
title = clean_title(title)
return title
def get_imgs(url, soup=None, session=None, cw=None):
if soup is None:
html = read_html(url, session=session, cw=cw)
soup = Soup(html)
title = get_title(soup)
pagination = soup.find('div', class_='pagination')
if pagination is None:
page = Page(None, url, soup)
imgs = get_imgs_page(page, session=session, cw=cw)
else:
pages = get_pages(url, soup, session=session)
imgs = []
for i, page in enumerate(pages):
check_alive(cw)
s = '{} {} / {} ({} / {})'.format(tr_('읽는 중...'), title, page.title, i+1, len(pages))
if cw:
cw.setTitle(s)
else:
print(s)
imgs += get_imgs_page(page, session=session, cw=cw)
if not imgs:
raise Exception('no imgs')
return imgs
def get_pages(url, soup=None, session=None):
if soup is None:
html = read_html(url, session=session, cw=None)
soup = Soup(html)
pagination = soup.find('div', class_='pagination')
pages = []
hrefs = set()
for a in pagination.findAll('a'):
href = a.attrs.get('href', '')
href = urljoin(url, href)
if not href.startswith(url):
print('not match', href)
continue
while href.endswith('/'):
href = href[:-1]
if href in hrefs:
print('duplicate', href)
continue
hrefs.add(href)
text = a.text.strip()
page = Page(text, href)
pages.append(page)
if url not in hrefs:
page = Page('1', url, soup)
pages.insert(0, page)
return pages
@try_n(4)
def get_imgs_page(page, session=None, cw=None):
url = page.url
soup = page.soup
if soup is None:
html = read_html(url, session=session, cw=None)
soup = Soup(html)
page.soup = soup
view = soup.find('div', class_='entry-content')
imgs = []
for img in view.findAll('img'):
img = img.get('data-lazy-src') or img.get('data-src') or img['src'] #7125
img = urljoin(url, img)
img = Image(img, len(imgs), page, cw)
imgs.append(img)
print(page.title, len(imgs), page.url)
return imgs
def fix_title(title):
title = re.sub(r'\(?[^()]*?c\.[^() ]+\)?', '', title)
while ' ' in title:
title = title.replace(' ', ' ')
return title
def read_html(url, session, cw):
## html = downloader.read_html(url, session=session)
## soup = Soup(html)
##
## cf = soup.find('div', class_='cf-browser-verification')
## if cf is None:
## return html
r = clf2.solve(url, cw=cw, session=session)
return r['html']
@try_n(4)
def get_session(url, cw=None):
print_ = get_print(cw)
## html = downloader.read_html(url)
## soup = Soup(html)
##
## cf = soup.find('div', class_='cf-browser-verification')
## if cf is None:
## print_('no cf protection')
## return None
print_('cf protection')
r = clf2.solve(url, cw=cw)
session = r['session']
return session
================================================
FILE: src/extractor/naver_downloader.py
================================================
#coding:utf-8
import downloader
import ree as re
from utils import urljoin, Downloader, Soup, LazyUrl, clean_title, get_ext, get_print, Session, json
import errors
PATTERNS = ['.*blog.naver.com/(?P<username>.+)/(?P<pid>[0-9]+)',
'.*blog.naver.com/.+?blogId=(?P<username>[^&]+).+?logNo=(?P<pid>[0-9]+)',
'.*?(?P<username>[0-9a-zA-Z_-]+)\.blog\.me/(?P<pid>[0-9]+)']
def get_id(url):
for pattern in PATTERNS:
m = re.match(pattern, url)
if m is None:
continue
username = m.group('username')
pid = m.group('pid')
break
else:
username, pid = None, None
return username, pid
class Downloader_naver(Downloader):
type = 'naver'
URLS = ['blog.naver.', '.blog.me']
display_name = 'Naver Blog'
ACCEPT_COOKIES = [r'(.*\.)?naver\.com', r'(.*\.)?blog\.me']
def init(self):
self.session = Session()
username, pid = get_id(self.url)
if username is None:
raise errors.Invalid(f'Invalid format: {self.url}')
self.url = f'https://blog.naver.com/{username}/{pid}'
def read(self):
info = get_imgs(self.url, self.session, self.cw)
for img in info['imgs']:
self.urls.append(img.url)
username, pid = get_id(self.url)
self.title = clean_title(f'[{username}] {info["title"]} ({pid})')
class Image:
def __init__(self, url, referer, p):
self.url = LazyUrl(referer, lambda _: url, self)
#3788, #3817
ext = get_ext(url)
self.filename = f'{p:04}{ext}'
class Video:
def __init__(self, url, referer, p):
self.url = LazyUrl(referer, lambda _: url, self)
self.filename = f'video_{p}.mp4'
def read_page(url, session, depth=0):
print('read_page', url, depth)
if depth > 10:
raise Exception('Too deep')
html = downloader.read_html(url, session=session)
if len(html) < 5000:
id = re.find('logNo=([0-9]+)', html, err='no id')
username = re.find('blog.naver.com/([0-9a-zA-Z]+)', url) or re.find('blogId=([0-9a-zA-Z]+)', url, err='no username')
url = f'https://m.blog.naver.com/PostView.nhn?blogId={username}&logNo={id}&proxyReferer='
soup = Soup(html)
if soup.find('div', {'id': 'viewTypeSelector'}):
return url, soup
frame = soup.find('frame')
if frame is None:
print('frame is None')
return read_page(url, session, depth+1)
return read_page(urljoin('https://blog.naver.com', frame['src']), session, depth+1)
def get_imgs(url, session, cw):
print_ = get_print(cw)
info = {}
url = url.replace('blog.naver', 'm.blog.naver')
referer = url
url_frame, soup = read_page(url, session)
info['title'] = soup.find('meta', {'property': 'og:title'})['content'].strip()
imgs = []
urls = set()
view = soup.find('div', {'id': 'viewTypeSelector'})
imgs_ = view.findAll('span', class_='_img') + view.findAll(['img', 'video']) #7062
for img in imgs_:
url = img.get('data-gif-url') or img.get('src')
if not url:
url = img.get('thumburl')
if not url:
continue
if 'ssl.pstatic.net' in url: #
continue
if 'blogpfthumb-phinf.pstatic.net' in url: # profile
continue
if 'dthumb-phinf.pstatic.net' in url: # link
continue
if 'storep-phinf.pstatic.net' in url: # emoticon
continue
url = url.replace('mblogthumb-phinf', 'blogfiles')
#url = re.sub('\?type=[a-zA-Z0-9]*', '?type=w1@2x', url)
#url = re.sub('\?type=[a-zA-Z0-9]*', '', url)
url = url.split('?')[0]
if url in urls:
print('### Duplicate:', url)
continue
urls.add(url)
#url = url.split('?type=')[0]
img = Image(url, referer, len(imgs))
imgs.append(img)
pairs = []
for video in soup.findAll(class_='_naverVideo'):
vid = video['vid']
key = video['key']
pairs.append((vid, key))
print_(f'pairs: {pairs}')
for script in soup.findAll('script', class_='__se_module_data'):
data_raw = script.get('data-module') or script.get('data-module-v2')
data = json.loads(data_raw)['data']
vid = data.get('vid')
if not vid:
continue
key = data['inkey']
pairs.append((vid, key))
videos = []
for vid, key in pairs:
url_api = f'https://apis.naver.com/rmcnmv/rmcnmv/vod/play/v2.0/{vid}?key={key}'
data_raw = downloader.read_html(url_api, session=session)
data = json.loads(data_raw)
fs = data['videos']['list']
fs = sorted(fs, key=lambda f: f['size'], reverse=True)
video = Video(fs[0]['source'], url_frame, len(videos))
videos.append(video)
info['imgs'] = imgs + videos
return info
================================================
FILE: src/extractor/navercafe_downloader.py
================================================
# coding:utf8
from utils import Downloader, get_print, urljoin, Soup, get_ext, File, clean_title, downloader, re, try_n, errors, json, Session
import utils
class LoginRequired(errors.LoginRequired):
def __init__(self, *args):
super().__init__(*args, method='browser', url='https://nid.naver.com/nidlogin.login')
class Downloader_navercafe(Downloader):
type = 'navercafe'
URLS = ['cafe.naver.com']
display_name = 'Naver Cafes'
ACCEPT_COOKIES = [r'(.*\.)?naver\.com']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
print('origin_url', url)
# 신형 우선 처리 (성능 최적화)
patterns = [
# REST API 스타일
(r'cafe\.naver\.com/[^/]+/cafes/([0-9]+)/articles/([0-9]+)',
lambda m: f'https://cafe.naver.com/ArticleRead.nhn?articleid={m[1]}&clubid={m[0]}'),
# 구형 스타일
(r'cafe\.naver\.com/([^/?#]+).+?articleid%3D([0-9]+)',
lambda m: f'https://cafe.naver.com/{m[0]}/{m[1]}'),
]
for pattern, formatter in patterns:
m = re.search(pattern, url)
if m:
fixed_url = formatter(m.groups())
print('fixed_url', fixed_url)
return fixed_url
print('no_fix_needed', url)
return url
def read(self):
info = get_info(self.url, self.session, self.cw)
for img in info['imgs']:
self.urls.append(img)
tail = f' ({info["cafename"]}_{info["id"]})'
self.title = clean_title(info['title'], n=-len(tail)) + tail
@try_n(4)
def get_info(url, session, cw=None):
print_ = get_print(cw)
info = {}
html = downloader.read_html(
url, 'http://search.naver.com', session=session)
soup = Soup(html)
if '"cafe_cautionpage"' in html:
raise LoginRequired()
PATTERN = r"//cafe\.naver\.com/ArticleRead\.nhn\?[^'\"]*articleid=[0-9]+[^'\"]*"
matches = [match.group()
for src in [html, url]
for match in [re.search(PATTERN, src)]
if match]
url_article = matches[0] if matches else "no articleid"
url_article = urljoin(url, url_article)
print_(url_article)
articleid = re.find(r'articleid=([0-9]+)', url_article)
clubid = re.find(r'clubid(=|%3D)([0-9]+)', url_article)[1]
art = re.find(r'art=(.+?)&', url_article)
if art:
url_api = f'https://apis.naver.com/cafe-web/cafe-articleapi/v2.1/cafes/{clubid}/articles/{articleid}?art={art}&useCafeId=true&requestFrom=A'
else:
url_api = f'https://apis.naver.com/cafe-web/cafe-articleapi/v2.1/cafes/{clubid}/articles/{articleid}?query=&useCafeId=true&requestFrom=A'
j = downloader.read_json(url_api, url_article, session=session)
if j['result'].get('errorCode'): # 6358
raise LoginRequired(j['result'].get('reason'))
info['title'] = j['result']['article']['subject']
info['cafename'] = j['result']['cafe']['url']
info['cafeid'] = clubid
info['id'] = articleid
html_content = j['result']['article']['contentHtml']
soup = Soup(html_content)
imgs = []
pairs = []
for video in soup.findAll('span', class_='_naverVideo'):
vid = video.attrs['vid']
key = video.attrs['key']
pairs.append((vid, key))
for script in soup.findAll('script', class_='__se_module_data'):
data_raw = script.get('data-module') or script.get('data-module-v2')
data = json.loads(data_raw)['data']
vid = data.get('vid')
if not vid:
continue
key = data['inkey']
pairs.append((vid, key))
for vid, key in pairs:
url_api = f'https://apis.naver.com/rmcnmv/rmcnmv/vod/play/v2.0/{vid}?key={key}'
data_raw = downloader.read_html(url_api)
data = json.loads(data_raw)
fs = data['videos']['list']
fs = sorted(fs, key=lambda f: f['size'], reverse=True)
video = Image(
{'url': fs[0]['source'], 'referer': url_article, 'p': len(imgs)})
imgs.append(video)
for img in soup.findAll('img'):
img = Image(
{'url': urljoin(url_article, img['src']), 'referer': url, 'p': len(imgs)})
imgs.append(img)
info['imgs'] = imgs
return info
class Image(File):
type = 'navercafe'
format = 'page:04;'
def __init__(self, info):
self._url = info['url']
info['url'] = re.sub(r'[?&]type=[wh0-9]+', '', self._url) # 6460
ext = get_ext(info['url'])
d = {
'page': info['p'],
}
info['name'] = utils.format('navercafe', d, ext)
super().__init__(info)
def alter(self):
return self._url
================================================
FILE: src/extractor/naverpost_downloader.py
================================================
# coding: UTF-8
# title: Download naver post image
# author: SaidBySolo
# comment: 네이버 포스트의 이미지를 다운로드합니다
"""
MIT License
Copyright (c) 2020 SaidBySolo
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
"""
import codecs
import json
import re
from distutils.util import strtobool
from typing import Any, Iterator, List
from urllib.parse import ParseResult, urlparse, parse_qs
import requests
from bs4 import BeautifulSoup
import clf2
import page_selector
from utils import Downloader, Soup, clean_title
class Page:
def __init__(self, title, url) -> None:
self.title = clean_title(title)
self.url = url
class DownloaderNaverPost(Downloader):
type = "naverpost" # 타입
URLS = ["m.post.naver.com", "post.naver.com"]
def init(self) -> None:
self.parsed_url = urlparse(self.url) # url 나눔
self.soup = get_soup(self.url)
@property
def client(self):
return Client(self.parsed_url, self.soup)
def read(self):
if self.client.single:
self.title = self.client.title
posts = self.client.posts
else:
raise NotImplementedError
for img_link in img_src_generator(posts):
self.urls.append(img_link)
# https://github.com/KurtBestor/Hitomi-Downloader/blob/master/src/extractor/manatoki_downloader.py#L106 참고
@page_selector.register("naverpost")
def f(url, win):
client = Client(urlparse(url), get_soup(url, win=win))
return [
page for page_list in client.posts for page in page_list
] # 2차원 리스트 -> 1차원 리스트
# https://github.com/KurtBestor/Hitomi-Downloader/blob/master/src/extractor/manatoki_downloader.py#L84 참고
def get_soup(url: str, win=None) -> BeautifulSoup:
res = clf2.solve(url, win=win)
return Soup(res["html"])
# 페이지 파싱에서 사용되는 파서
def page_soup(url: str) -> BeautifulSoup:
get_html_regex = re.compile(r"\"html\"\:(.+)(\n|\s)\}")
response = requests.get(url)
like_html = get_html_regex.search(response.text)[1]
html = decode_escapes(like_html).replace(r"\/", "/")
return Soup(html)
# HTML5 data-* 속성이 사용됨.
def get_img_data_linkdatas(soup: Any) -> Iterator[str]:
a_elements = soup.find_all("a", {"data-linktype": "img"}) # 링크 타입이 img인것만 전부 찾음
for a_element in a_elements:
yield a_element["data-linkdata"]
def img_src_generator(linkdatas: Iterator[str]) -> Iterator[str]:
for linkdata in linkdatas:
data = json.loads(linkdata)
if data.get("linkUse") is None:
yield data["src"] # 제네레이터
else:
if not strtobool(data["linkUse"]):
yield data["src"]
# https://stackoverflow.com/a/24519338 참고
def decode_escapes(like_html: str) -> str:
escape_sequence_regex = re.compile(
r"""
( \\U........ # 8-digit hex escapes
| \\u.... # 4-digit hex escapes
| \\x.. # 2-digit hex escapes
| \\[0-7]{1,3} # Octal escapes
| \\N\{[^}]+\} # Unicode characters by name
| \\[\\'"abfnrtv] # Single-character escapes
)""",
re.UNICODE | re.VERBOSE,
)
return escape_sequence_regex.sub(
lambda match: codecs.decode(match.group(0)), like_html
)
# 제목
class Title:
def __init__(self, soup: Any):
self.soup = soup
def get_profile_title(self) -> str:
profile_name = self.soup.find("p", class_="nick_name").find(
"span", class_="name"
) # 프로필 닉네임
return clean_title(profile_name.text) # 닉네임으로만
def get_series_title(self) -> str:
series_name = self.soup.find("h2", class_="tit_series").find(
"span", class_="ell"
) # 시리즈 제목
author = self.soup.find("div", class_="series_author_wrap").find(
"strong", class_="ell1"
) # 작성자
return clean_title(f"{series_name.text} ({author.text})") # 무난하게 붙임
def get_title(self) -> str:
title = self.soup.find("h3", class_="se_textarea") # 포스트 제목
author = self.soup.find("span", class_="se_author") # 작성자
return clean_title(f"{title.text.replace(' ', '')} ({author.text})") # 무난하게 붙임
# 총 포스트 수
class Total:
def __init__(self, soup: Any) -> None:
self.soup = soup
# 0: 팔로워 1: 팔로잉 2: 포스트 3: 좋아요한글
def get_total_post(self) -> int:
profile_info = self.soup.find("div", class_="expert_num_info") # 프로필 정보
total_post_element = profile_info.find_all("li", class_="inner")[2]
return int(total_post_element.find("span", class_="num").text) # 총몇개인지만 리턴
# 0: 포스트 1: 팔로워
def get_series_total_post(self) -> int:
series_info = self.soup.find("div", class_="series_follow_area") # 시리즈 정보
total_post_element = series_info.find_all("a")[0]
return int(total_post_element.find("em").text) # 총몇개인지만 리턴
class UrlGenerator:
def __init__(self, parsed_url: ParseResult, total_count: int) -> None:
self.parsed_url = parsed_url
self.count = (
round(total_count / 20) + 1
if not (total_count / 20).is_integer()
else round(total_count / 20)
)
def all_post_url_generator(self) -> Iterator[str]:
query = parse_qs(self.parsed_url.query)
for i in range(self.count):
new_url_query = f"?memberNo={query['memberNo'][0]}&fromNo={i + 1}"
url = f"https://{self.parsed_url.netloc}/async{self.parsed_url.path}{new_url_query}"
yield url
def all_series_url_generator(self) -> Iterator[str]:
query = parse_qs(self.parsed_url.query)
for i in range(self.count):
new_url_query = f"?memberNo={query['memberNo'][0]}&seriesNo={query['seriesNo'][0]}&fromNo={i + 1}"
url = f"https://{self.parsed_url.netloc}/my/series/detail/more.nhn{new_url_query}"
yield url
# 여기서 페이지 리스트 만듬
class PostPage:
def __init__(self, soup: Any):
self.soup = soup
def all_post_page_generator(self) -> Iterator[List[Page]]:
titles = self.soup.find_all("strong", class_="tit_feed ell")
link_elements = self.soup.find_all("a", class_="link_end", href=True)
page = [
Page(title.text.replace(" ", ""), link_element["href"])
for link_element, title in zip(link_elements, titles)
]
yield page[::-1]
def all_series_page_generator(self) -> Iterator[List[Page]]:
titles = [
element.find("span")
for element in self.soup.find_all("div", class_="spot_post_name")
]
link_elements = self.soup.find_all("a", class_="spot_post_area", href=True)
page = [
Page(title.text.replace(" ", ""), link_element["href"])
for link_element, title in zip(link_elements, titles)
]
yield page[::-1]
# 필요한 클래스 전부 상속후 편하게 쓸수있게 만듬
class Client(Title, Total, UrlGenerator):
def __init__(self, parsed_url: ParseResult, soup: BeautifulSoup):
Title.__init__(self, soup)
Total.__init__(self, soup)
if parsed_url.path.startswith("/viewer"):
self.title = self.get_title()
self.posts = get_img_data_linkdatas(self.soup)
self.single = True
elif parsed_url.path.startswith("/my.nhn"):
UrlGenerator.__init__(self, parsed_url, self.get_total_post())
self.title = self.get_profile_title()
self.posts = self.all_post_url_generator()
self.single = False
elif parsed_url.path.startswith("/my/series"):
UrlGenerator.__init__(self, parsed_url, self.get_series_total_post())
self.title = self.get_series_title()
self.posts = self.all_series_url_generator()
self.single = False
else:
raise Exception("유효하지 않습니다.")
================================================
FILE: src/extractor/navertoon_downloader.py
================================================
import downloader
from utils import Soup, urljoin, Downloader, get_imgs_already, clean_title, get_ext, get_print, errors, check_alive, File, try_n, json
import ree as re
import page_selector
from translator import tr_
import utils
class Page:
def __init__(self, url, title, p):
self.url = url
self.title = title
self.p = p
class File_navertoon(File):
type = 'navertoon'
format = 'title/page:04;'
def __init__(self, info):
ext = get_ext(info['url'])
d = {
'title': clean_title(info['title']),
'page': info['page'],
'chapterid': re.find(r'[?&]no=([0-9]+)', info['referer']), #6380
}
info['name'] = utils.format('navertoon', d, ext)
super().__init__(info)
class Info:
def __init__(self, id, title, artist):
self.id = id
self.title = title
self.artist = artist
class Downloader_navertoon(Downloader):
type = 'navertoon'
URLS = ['comic.naver.com']
MAX_CORE = 8
MAX_SPEED = 4.0
display_name = 'Naver Webtoon'
ACCEPT_COOKIES = [r'(.*\.)?naver\.com']
def init(self):
self.__info, _ = get_pages(self.url, self.cw)
@classmethod
def fix_url(cls, url):
url = re.sub(r'[?&]page=[0-9]+', '', re.sub(r'[?&]no=[0-9]+', '', url)).replace('m.comic.naver.', 'comic.naver.')
url = url.replace('detail.nhn', 'list.nhn').replace('/detail?', '/list?')
return url.rstrip('#')
@property
def name(self):
id = self.__info.id
title = self.__info.title
artist = self.__info.artist
title = self.format_title('N/A', id, title, artist, 'N/A', 'N/A', 'Korean', prefix='navertoon_')
return clean_title(title)
def read(self):
self.title = tr_('읽는 중... {}').format(self.name)
imgs = get_imgs_all(self.url, self.name, cw=self.cw)
for img in imgs:
self.urls.append(img)
self.title = self.name
def set_no(url, p):
if '&no=' not in url:
url = url + f'&no={p}'
return url
url = re.sub('&no=[0-9]+', f'&no={p}', url)
return url
def get_id(url):
return int(url.lower().split('titleid=')[1].split('&')[0])
def set_page(url, p):
if '&page=' in url:
url = re.sub('&page=[0-9]+', f'&page={p}', url)
else:
url += f'&page={p}'
return url
@try_n(4)
def get_pages(url, cw=None):
print_ = get_print(cw)
url = Downloader_navertoon.fix_url(url).replace('comic.naver.', 'm.comic.naver.')
id = get_id(url)
print('id:', id)
print(url)
html = downloader.read_html(url)
soup = Soup(html)
if soup.find('button', class_='btn_check'):
raise errors.LoginRequired()
try:
info = soup.find('div', class_='area_info')
artist = info.find('span', class_='author').text.strip()
except Exception as e:
print(e)
try:
title = ('\n').join(soup.find('div', class_='title').text.strip().split('\n')[:-1]).strip()
except:
title = 'artist not found'
raise Exception(title)
print_('artist: {}'.format(artist))
title = soup.find('meta', {'property': 'og:title'}).attrs['content']
pages = []
nos = set()
for p in range(1, 100):
if p == 1:
url_page = url
else:
url_page = set_page(url, p)
html = downloader.read_html(url_page)
print('read page:', url_page)
soup = Soup(html)
view = soup.findAll('ul', class_='section_episode_list')[-1]
for lst in view.findAll('li'):
url_page = urljoin(url, lst.find('a').attrs['href'])
if 'detail.nhn' not in url_page.lower() and 'detail?' not in url_page.lower(): #3540
continue
print_('url_page: {}'.format(url_page))
text = lst.find('strong', class_='title').find('span', class_='name').text.strip()
no = int(re.findall('[?&]no=([0-9]+)', url_page)[0])
if no in nos:
print('duplicate no: {}'.format(no))
continue
nos.add(no)
text = '{:04} - {}'.format(no, text)
page = Page(url_page, text, p)
pages.append(page)
btn_next = soup.find('a', class_='btn_next')
if btn_next is None or btn_next.attrs['href'] == '#':
print('end of page')
break
info = Info(id, title, artist)
return (
info, pages)
@page_selector.register('navertoon')
@try_n(4)
def f(url):
url = Downloader_navertoon.fix_url(url)
info, pages = get_pages(url)
return pages
@try_n(6)
def get_imgs(page, cw=None):
print_ = get_print(cw)
html = downloader.read_html(page.url)
soup = Soup(html)
type_ = re.find('''webtoonType *: *['"](.+?)['"]''', html)
print_('type: {}'.format(type_))
imgs = []
if type_ == 'DEFAULT': # https://m.comic.naver.com/webtoon/detail.nhn?titleId=715772
view = soup.find('div', class_='toon_view_lst')
for img in view.findAll('img'):
img = img.attrs.get('data-src')
if not img:
continue
img = urljoin(page.url, img)
img = File_navertoon({'referer': page.url, 'url':img, 'title': page.title, 'page': len(imgs)})
imgs.append(img)
elif type_ == 'CUTTOON': # https://m.comic.naver.com/webtoon/detail.nhn?titleId=752803
view = soup.find('div', class_='swiper-wrapper')
for div in view.findAll('div', class_='swiper-slide'):
if div.parent != view:
continue
if div.find('div', class_='cut_viewer_last'):
print('cut_viewer_last')
continue
if div.find('div', class_='cut_viewer_recomm'):
print('cut_viewer_recomm')
continue
img = div.find('img')
img = img.attrs['data-src']
img = urljoin(page.url, img)
img = File_navertoon({'referer': page.url, 'url':img, 'title': page.title, 'page': len(imgs)})
imgs.append(img)
elif type_ == 'EFFECTTOON': #2313; https://m.comic.naver.com/webtoon/detail.nhn?titleId=670144
img_base = re.find('''imageUrl *: *['"](.+?)['"]''', html) + '/'
print('img_base:', img_base)
url_api = re.find('''documentUrl *: *['"](.+?)['"]''', html)
data_raw = downloader.read_html(url_api, page.url)
data = json.loads(data_raw)
for img in data['assets']['stillcut'].values(): # ordered in python3.7+
img = urljoin(img_base, img)
img = File_navertoon({'referer': page.url, 'url':img, 'title': page.title, 'page': len(imgs)})
imgs.append(img)
else:
_imgs = re.findall('sImageUrl *: *[\'"](.+?)[\'"]', html)
if not _imgs:
raise Exception('no imgs')
for img in _imgs:
img = urljoin(page.url, img)
img = File_navertoon({'referer': page.url, 'url':img, 'title': page.title, 'page': len(imgs)})
imgs.append(img)
return imgs
def get_imgs_all(url, title, cw=None):
print_ = get_print(cw)
info, pages = get_pages(url, cw)
pages = page_selector.filter(pages, cw)
imgs = []
for p, page in enumerate(pages):
check_alive(cw)
imgs_already = get_imgs_already('navertoon', title, page, cw)
if imgs_already:
imgs += imgs_already
continue
imgs_new = get_imgs(page, cw)
print_('{}: {}'.format(page.title, len(imgs_new)))
imgs += imgs_new
if cw is not None:
cw.setTitle(tr_('읽는 중... {} / {} ({}/{})').format(title, page.title, p + 1, len(pages)))
return imgs
================================================
FILE: src/extractor/navertv_downloader.py
================================================
import downloader
import ree as re
from io import BytesIO as IO
from utils import Downloader, LazyUrl, get_ext, format_filename, try_n
import ytdl
class Downloader_navertv(Downloader):
type = 'navertv'
single = True
URLS = ['tv.naver.com']
display_name = 'Naver TV'
ACCEPT_COOKIES = [r'(.*\.)?naver\.com']
@classmethod
def fix_url(cls, url):
if not re.match(r'https?://.+', url, re.I):
url = f'https://tv.naver.com/v/{url}'
return url
def read(self):
video = Video(self.url, cw=self.cw)
video.url()#
self.urls.append(video.url)
self.setIcon(video.thumb)
self.enableSegment()
self.title = video.title
class Video:
_url = None
def __init__(self, url, cw=None):
self.url = LazyUrl(url, self.get, self)
self.cw = cw
@try_n(4)
def get(self, url):
if self._url:
return self._url
ydl = ytdl.YoutubeDL(cw=self.cw)
info = ydl.extract_info(url)
fs = [f for f in info['formats'] if f['protocol'] in ['http', 'https']]
fs = sorted(fs, key=lambda f: int(f.get('width', 0)), reverse=True)
if not fs:
raise Exception('No MP4 videos')
f = fs[0]
self._url = f['url']
self.thumb_url = info['thumbnails'][0]['url']
self.thumb = IO()
downloader.download(self.thumb_url, buffer=self.thumb)
self.title = info['title']
id = info['id']
ext = get_ext(self._url)
self.filename = format_filename(self.title, id, ext)
return self._url
================================================
FILE: src/extractor/newgrounds_downloader.py
================================================
# coding:utf8
from datetime import datetime
import downloader
import errors
from functools import reduce
import os
import ree as re
from timee import sleep
from translator import tr_
from utils import Downloader, clean_title, Session, get_print, try_n, check_alive
class Downloader_newgrounds(Downloader):
type = 'newgrounds'
URLS = ['newgrounds.com']
ACCEPT_COOKIES = [r'(.*\.)?newgrounds\.com']
def init(self):
self.session = Session()
@classmethod
def fix_url(cls, url):
user = re.find(r'(?:http(?:s)?://)?([^\.]+).newgrounds.com', url.lower())
if not user or user == 'www':
user = re.find(r'newgrounds.com/art/view/([^/?#]+)', url, err='no user id')
return 'https://{}.newgrounds.com/art'.format(user)
def read(self):
user = re.find('(?:http(?:s)?://)?([^\.]+).newgrounds.com', self.url.lower())
title = clean_title(user)
for img in get_imgs(user, title, self.session, self.cw):
self.urls.append(img.url)
self.filenames[img.url] = img.filename
self.title = title
class Image:
def __init__(self, url, filename=None):
self.url = url
if filename is None:
filename = os.path.basename(url)
self.filename = filename
@try_n(10, sleep=20)
def get_posts(url, params, session, print_):
posts, data, items = [], None, None
try:
data = session.get(url, params=params).json()
items = data.get('items')
if items:
for item in reduce(lambda x, y: x + y, items.values()):
posts.append(re.find('(?<=href=")([^"]+)', item))
except Exception as e:
print_('failed to get posts')
print_('no. posts: {}'.format(len(posts)))
print_('data: {}'.format(data))
print_('items: {}'.format(items))
raise e
return posts
@try_n(10, sleep=20)
def get_html(post, session):
return downloader.read_html(post, session=session)
def get_img(post, session, print_):
html, url, name, ext, _datetime = None, None, None, None, None
try:
html = get_html(post, session)
if 'You must be logged in, and at least 18 years of age to view this content!' in html:
raise errors.LoginRequired()
url = re.find('(?<="full_image_text":"<img src=\\\\")([^"]+)', html).replace('\\', '')
name = re.find('(?<=alt=\\\\")([^\\\\]+)', html)
ext = os.path.splitext(url)[1].split('?')[0]
_datetime = datetime.strptime(re.find('(?<="datePublished" content=")([^"]+)', html), '%Y-%m-%dT%H:%M:%S%z')
except Exception as e:
print_('failed to get images')
print_('post: {}'.format(post))
print_('url: {}'.format(url))
print_('name: {}'.format(name))
print_('ext: {}'.format(ext))
print_('_datetime: {}'.format(_datetime))
raise e
return Image(url=url, filename='[{}] {}{}'.format(_datetime.strftime("%Y-%m-%d"), name, ext))
def get_imgs(user, title, session, cw=None):
print_ = get_print(cw)
imgs = []
url = 'https://{}.newgrounds.com/art'.format(user)
params = {'page': 1, 'isAjaxRequest': 1}
while check_alive(cw):
posts = get_posts(url, params, session, print_)
if not posts:
break
for post in posts:
sleep(0.75)
imgs.append(get_img(post, session, print_))
s = '{} {} - {}'.format(tr_('읽는 중...'), title, len(imgs))
if cw:
cw.setTitle(s)
print_('processed: {}'.format(len(imgs)))
print_('page: {}'.format(params['page']))
params['page'] += 1
return imgs
================================================
FILE: src/extractor/nhentai_com_downloader.py
================================================
#coding:utf8
import downloader
import ree as re
from utils import urljoin, LazyUrl, Downloader, try_n, join, json
import os
class Downloader_nhentai_com(Downloader):
type = 'nhentai_com'
URLS = [r'regex:https?://nhentai.com']
MAX_CORE = 16
display_name = 'nhentai.com'
ACCEPT_COOKIES = [r'(.*\.)?nhentai\.com']
def init(self):
self.info = get_info(self.url)
self.url = self.info['url']
@classmethod
def key_id(cls, url):
url = url.lower()
return re.find(r'/comic/([^/?]+)', url) or url
def read(self):
info = self.info
artist = join(info['artists'])
self.artist = artist if info['artists'] else None
group = join(info['groups'])
lang = info['lang'] or 'N／A'
series = info['seriess'][0] if info['seriess'] else 'N／A'
title = self.format_title(info['type'], info['id'], info['title'], artist, group, series, lang)
for img in info['imgs']:
self.urls.append(img.url)
self.title = title
@LazyUrl.register
class LazyUrl_nhentai_com(LazyUrl):
type = 'nhentai_com'
def dump(self):
referer = self._url
url = self.image.url_img
return {
'referer': referer,
'url': url,
'p': self.image.p,
}
@classmethod
def load(cls, data):
referer = data['referer']
url = data['url']
img = Image(referer, url, data['p'])
return img.url
class Image:
def __init__(self, url_page, url_img, p):
self.p = p
self.referer = url_page
self.filename = os.path.basename(url_img)
self.url_img = url_img
self.url = LazyUrl_nhentai_com(url_page, lambda _: self.url_img, self)
@try_n(4)
def get_info(url):
url = downloader.real_url(url)
q = re.find(r'/comic/([^/?]+)', url)
url_api = 'https://nhentai.com/api/comics/{}'.format(q)
data_raw = downloader.read_html(url_api, url)
data = json.loads(data_raw)
url_api = 'https://nhentai.com/api/comics/{}/images'.format(q)
data_raw = downloader.read_html(url_api, url)
data_images = json.loads(data_raw)
info = {}
info['url'] = url
info['id'] = int(data['id'])
info['type'] = data['category']['name']
info['title'] = data['title']
info['artists'] = [x['name'] for x in data['artists']]
info['groups'] = [x['name'] for x in data['groups']]
info['seriess'] = [x['name'] for x in data['parodies']]
info['lang'] = data['language']['name']
imgs = []
for img in data_images['images']:
img = urljoin(url, img['source_url'])
img = Image(url, img, len(imgs))
imgs.append(img)
info['imgs'] = imgs
return info
================================================
FILE: src/extractor/nhentai_downloader.py
================================================
#coding:utf8
import downloader
import ree as re
from utils import urljoin, File, Downloader, try_n, join, get_ext, json
import utils
import clf2
def get_id(url):
try:
return int(url)
except:
return int(re.find('/g/([0-9]+)', url))
class File_nhentai(File):
type = 'nhentai'
format = 'page:04;'
class Downloader_nhentai(Downloader):
type = 'nhentai'
URLS = ['nhentai.net']
MAX_CORE = 16
display_name = 'nhentai'
ACCEPT_COOKIES = [r'(.*\.)?nhentai\.net']
def init(self):
self.session = clf2.solve(self.url, cw=self.cw)['session'] #4541
@classmethod
def fix_url(cls, url):
return f'https://nhentai.net/g/{get_id(url)}/'
def read(self):
info, imgs = get_imgs(get_id(self.url), self.session)
# 1225
artist = join(info.artists)
self.artist = artist if info.artists else None
group = join(info.groups)
lang = info.lang or 'N／A'
series = info.seriess[0] if info.seriess else 'N／A'
title = self.format_title(info.type, info.id, info.title, artist, group, series, lang)
self.urls += imgs
self.title = title
class Info:
def __init__(self, host, id, id_media, title, p, artists, groups, seriess, lang, type, formats):
self.host = host
self.id = id
self.id_media = id_media
self.ti
gitextract_gsyu7ov9/
├── .github/
│ └── stale.yml
├── .gitignore
├── FUNDING.yml
├── README.md
├── push_^q^.bat
├── src/
│ └── extractor/
│ ├── _4chan_downloader.py
│ ├── afreeca_downloader.py
│ ├── artstation_downloader.py
│ ├── asmhentai_downloader.py
│ ├── avgle_downloader.py
│ ├── baraag_downloader.py
│ ├── bcy_downloader.py
│ ├── bdsmlr_downloader.py
│ ├── bili_downloader.py
│ ├── coub_downloader.py
│ ├── danbooru_downloader.py
│ ├── discord_emoji_downloader.py
│ ├── etc_downloader.py
│ ├── fc2_downloader.py
│ ├── file_downloader.py
│ ├── flickr_downloader.py
│ ├── gelbooru_downloader.py
│ ├── hameln_downloader.py
│ ├── hanime_downloader.py
│ ├── hentaicosplay_downloader.py
│ ├── hf_downloader.py
│ ├── imgur_downloader.py
│ ├── iwara_downloader.py
│ ├── jmana_downloader.py
│ ├── kakaotv_downloader.py
│ ├── kakuyomu_downloader.py
│ ├── kissjav_downloader.py
│ ├── lhscan_downloader.py
│ ├── luscious_downloader.py
│ ├── m3u8_downloader.py
│ ├── manatoki_downloader.py
│ ├── mastodon_downloader.py
│ ├── misskey_downloader.py
│ ├── mrm_downloader.py
│ ├── naver_downloader.py
│ ├── navercafe_downloader.py
│ ├── naverpost_downloader.py
│ ├── navertoon_downloader.py
│ ├── navertv_downloader.py
│ ├── newgrounds_downloader.py
│ ├── nhentai_com_downloader.py
│ ├── nhentai_downloader.py
│ ├── nico_downloader.py
│ ├── nijie_downloader.py
│ ├── nozomi_downloader.py
│ ├── pawoo_downloader.py
│ ├── pinter_downloader.py
│ ├── pixiv_downloader.py
│ ├── pornhub_downloader.py
│ ├── rule34_xxx_downloader.py
│ ├── sankaku_downloader.py
│ ├── soundcloud_downloader.py
│ ├── syosetu_downloader.py
│ ├── talk_op_gg_downloader.py
│ ├── tiktok_downloader.py
│ ├── tokyomotion_downloader.py
│ ├── torrent_downloader.py
│ ├── tumblr_downloader.py
│ ├── twitch_downloader.py
│ ├── v2ph_downloader.py
│ ├── vimeo_downloader.py
│ ├── wayback_machine_downloader.py
│ ├── webtoon_downloader.py
│ ├── weibo_downloader.py
│ ├── wikiart_downloader.py
│ ├── xhamster_downloader.py
│ ├── xnxx_downloader.py
│ ├── xvideo_downloader.py
│ ├── yandere_downloader.py
│ ├── youku_downloader.py
│ ├── youporn_downloader.py
│ └── youtube_downloader.py
└── translation/
├── changelog_en.txt
├── changelog_ko.txt
├── help_en.html
├── help_ja.html
├── help_ko.html
├── help_pl.html
├── help_ru.html
├── help_si.html
├── help_zh.html
├── qt_ar.json
├── qt_es.json
├── qt_fr.json
├── qt_ja.json
├── qt_ko.json
├── qt_pl.json
├── qt_pt.json
├── qt_ru.json
├── qt_si.json
├── qt_tr.json
├── qt_vi.json
├── qt_zh-TW.json
├── qt_zh.json
├── tr_ar.hdl
├── tr_en.hdl
├── tr_es.hdl
├── tr_fr.hdl
├── tr_ja.hdl
├── tr_ko.hdl
├── tr_pl.hdl
├── tr_pt.hdl
├── tr_ru.hdl
├── tr_si.hdl
├── tr_tr.hdl
├── tr_vi.hdl
├── tr_zh-TW.hdl
└── tr_zh.hdl
SYMBOL INDEX (841 symbols across 72 files)
FILE: src/extractor/_4chan_downloader.py
class File_4chan (line 7) | class File_4chan(File):
method get (line 12) | def get(self):
class Downloader_4chan (line 17) | class Downloader_4chan(Downloader):
method fix_url (line 25) | def fix_url(cls, url):
method read (line 28) | def read(self):
FILE: src/extractor/afreeca_downloader.py
class LoginRequired (line 12) | class LoginRequired(errors.LoginRequired):
method __init__ (line 13) | def __init__(self, *args):
class Downloader_afreeca (line 18) | class Downloader_afreeca(Downloader):
method init (line 25) | def init(self):
method fix_url (line 29) | def fix_url(cls, url):
method read (line 34) | def read(self):
function _get_stream (line 55) | def _get_stream(url_m3u8, session, referer, cw=None):
class Video (line 67) | class Video(File):
method get (line 71) | def get(self):
class Live_afreeca (line 182) | class Live_afreeca(utils.Live):
method is_live (line 186) | def is_live(cls, url):
method fix_url (line 190) | def fix_url(cls, url):
method check_live (line 195) | def check_live(cls, url, info=None):
FILE: src/extractor/artstation_downloader.py
class File_artstation (line 10) | class File_artstation(File):
method alter (line 15) | def alter(self): #6401
class Downloader_artstation (line 25) | class Downloader_artstation(Downloader):
method init (line 33) | def init(self):
method fix_url (line 50) | def fix_url(cls, url): #6516
method _id (line 58) | def _id(self):
method name (line 64) | def name(self):
method read (line 69) | def read(self):
function get_imgs (line 88) | def get_imgs(id_, title, session, cw=None):
function get_id_art (line 140) | def get_id_art(post_url):
function get_id (line 144) | def get_id(url, cw=None):
function get_imgs_page (line 175) | def get_imgs_page(id_art, session, date=None, cw=None, names=None):
FILE: src/extractor/asmhentai_downloader.py
function get_id (line 10) | def get_id(url):
class File_asmhentai (line 17) | class File_asmhentai(File):
method get (line 22) | def get(self):
class Downloader_asmhentai (line 34) | class Downloader_asmhentai(Downloader):
method init (line 41) | def init(self):
method fix_url (line 45) | def fix_url(cls, url):
method read (line 49) | def read(self):
function get_info (line 69) | def get_info(url, session, cw):
FILE: src/extractor/avgle_downloader.py
class Downloader_avgle (line 12) | class Downloader_avgle(Downloader):
method init (line 18) | def init(self):
method read (line 24) | def read(self):
function get_video (line 34) | def get_video(url, cw=None):
class Video (line 67) | class Video:
method __init__ (line 68) | def __init__(self, url, url_thumb, referer, title):
FILE: src/extractor/baraag_downloader.py
function get_id (line 8) | def get_id(url):
class Downloader_baraag (line 13) | class Downloader_baraag(Downloader):
method init (line 19) | def init(self):
method fix_url (line 23) | def fix_url(cls, url):
method read (line 27) | def read(self):
FILE: src/extractor/bcy_downloader.py
class Downloader_bcy (line 9) | class Downloader_bcy(Downloader):
method init (line 16) | def init(self):
method name (line 21) | def name(self):
method read (line 29) | def read(self):
function get_ssr_data (line 39) | def get_ssr_data(html):
function get_imgs (line 47) | def get_imgs(url, html=None, cw=None):
class Image_single (line 69) | class Image_single:
method __init__ (line 70) | def __init__(self, url ,referer, p):
method get (line 75) | def get(self, referer):
class Image (line 81) | class Image:
method __init__ (line 82) | def __init__(self, url, referer, id, p):
method get (line 88) | def get(self, referer):
function get_ext (line 94) | def get_ext(url, referer=None):
function get_info (line 101) | def get_info(url, html):
function get_imgs_channel (line 120) | def get_imgs_channel(url, html=None, cw=None):
FILE: src/extractor/bdsmlr_downloader.py
class Downloader_bdsmlr (line 14) | class Downloader_bdsmlr(Downloader):
method init (line 20) | def init(self):
method id_ (line 29) | def id_(self):
method read (line 39) | def read(self):
class Post (line 48) | class Post:
method __init__ (line 49) | def __init__(self, url, referer, id, p):
function foo (line 56) | def foo(url, soup, info, reblog=False):
function get_imgs (line 77) | def get_imgs(user_id, session, cw=None):
FILE: src/extractor/bili_downloader.py
class File_bili (line 28) | class File_bili(File):
method get (line 33) | def get(self):
method pp (line 105) | def pp(self, filename):
function fix_url (line 114) | def fix_url(url, cw=None):
class Downloader_bili (line 137) | class Downloader_bili(Downloader):
method init (line 147) | def init(self):
method key_id (line 155) | def key_id(cls, url):
method id_ (line 163) | def id_(self):
method read (line 169) | def read(self):
FILE: src/extractor/coub_downloader.py
function get_id (line 9) | def get_id(url):
class Downloader_coub (line 14) | class Downloader_coub(Downloader):
method fix_url (line 21) | def fix_url(cls, url):
method key_id (line 25) | def key_id(cls, url):
method read (line 28) | def read(self):
class Video (line 41) | class Video:
method __init__ (line 44) | def __init__(self, url, cw=None):
method get (line 49) | def get(self, url):
method pp (line 69) | def pp(self, filename):
FILE: src/extractor/danbooru_downloader.py
class Downloader_danbooru (line 10) | class Downloader_danbooru(Downloader):
method init (line 17) | def init(self):
method fix_url (line 23) | def fix_url(cls, url):
method name (line 36) | def name(self):
method read (line 60) | def read(self):
class Image (line 74) | class Image:
method __init__ (line 75) | def __init__(self, id, url, session, cw):
method get (line 81) | def get(self, url):
function wait (line 100) | def wait(cw):
function setPage (line 104) | def setPage(url, page):
function read_soup (line 119) | def read_soup(url, session, cw, try_=1):
function get_imgs (line 127) | def get_imgs(url, session, title=None, range_=None, cw=None):
FILE: src/extractor/discord_emoji_downloader.py
class DownloaderDiscordEmoji (line 34) | class DownloaderDiscordEmoji(Downloader):
method init (line 37) | def init(self):
method read (line 40) | def read(self):
method get_emoji_list (line 92) | def get_emoji_list(self, token: str, guild_id: int) -> dict:
method post_account_info (line 105) | def post_account_info(self, email: str, password: str) -> dict:
FILE: src/extractor/etc_downloader.py
class Downloader_etc (line 14) | class Downloader_etc(Downloader):
method read (line 23) | def read(self):
function int_or_none (line 49) | def int_or_none(s):
function format_ (line 56) | def format_(f):
class UnSupportedError (line 62) | class UnSupportedError(Exception):pass
function get_video (line 65) | def get_video(url, session, cw, ie_key=None):
function extract_info_spankbang (line 83) | def extract_info_spankbang(url, session, cw): # temp fix
function _get_video (line 118) | def _get_video(url, session, cw, ie_key=None, allow_m3u8=True):
function get_ie_key (line 208) | def get_ie_key(info):
function get_ext_ (line 216) | def get_ext_(url, session, referer):
class Video (line 224) | class Video:
method __init__ (line 227) | def __init__(self, f, f_audio, info, session, referer, cw=None):
method pp (line 287) | def pp(self, filename):
FILE: src/extractor/fc2_downloader.py
class Downloader_fc2 (line 10) | class Downloader_fc2(Downloader):
method fix_url (line 17) | def fix_url(cls, url):
method key_id (line 23) | def key_id(cls, url):
method read (line 26) | def read(self):
class Video (line 42) | class Video:
method __init__ (line 44) | def __init__(self, url, url_thumb, referer, title, id_, session):
method get (line 51) | def get(self, referer):
function get_info (line 60) | def get_info(url, session, cw=None):
FILE: src/extractor/file_downloader.py
class Downloader_file (line 9) | class Downloader_file(Downloader):
method fix_url (line 16) | def fix_url(cls, url):
method read (line 22) | def read(self):
FILE: src/extractor/flickr_downloader.py
class File_flickr (line 10) | class File_flickr(File):
method get (line 15) | def get(self):
class Downloader_flickr (line 28) | class Downloader_flickr(Downloader):
method init (line 34) | def init(self):
method fix_url (line 38) | def fix_url(cls, url):
method read (line 45) | def read(self):
FILE: src/extractor/gelbooru_downloader.py
function get_tags (line 10) | def get_tags(url):
class Downloader_gelbooru (line 26) | class Downloader_gelbooru(Downloader):
method init (line 33) | def init(self):
method fix_url (line 37) | def fix_url(cls, url):
method name (line 50) | def name(self):
method read (line 56) | def read(self):
class File_gelbooru (line 64) | class File_gelbooru(File):
method get (line 68) | def get(self):
method alter (line 81) | def alter(self):
function setPage (line 85) | def setPage(url, page):
function get_imgs (line 97) | def get_imgs(url, session, title=None, cw=None):
FILE: src/extractor/hameln_downloader.py
class Downloader_hameln (line 12) | class Downloader_hameln(Downloader):
method init (line 19) | def init(self):
method soup (line 25) | def soup(self):
method info (line 31) | def info(self):
method read (line 34) | def read(self):
method post_processing (line 42) | def post_processing(self):
class Text (line 60) | class Text:
method __init__ (line 61) | def __init__(self, page, p):
method get (line 66) | def get(self, url):
class Page (line 74) | class Page:
method __init__ (line 75) | def __init__(self, title, url):
function read_html (line 81) | def read_html(url):
function get_sss (line 85) | def get_sss(soup):
function get_pages (line 90) | def get_pages(url, soup=None):
function read_page (line 112) | def read_page(page):
function get_info (line 141) | def get_info(url, soup=None):
FILE: src/extractor/hanime_downloader.py
class Video (line 11) | class Video:
method __init__ (line 13) | def __init__(self, info, stream):
method __repr__ (line 34) | def __repr__(self):
class Downloader_hanime (line 39) | class Downloader_hanime(Downloader):
method init (line 46) | def init(self):
method read (line 49) | def read(self):
function get_video (line 61) | def get_video(url, session, cw=None):
FILE: src/extractor/hentaicosplay_downloader.py
class Image (line 14) | class Image:
method __init__ (line 15) | def __init__(self, url, referer, p, session):
method get (line 24) | def get(self, _=None):
class Video (line 36) | class Video:
method __init__ (line 38) | def __init__(self, src, referer, title, session):
class Downloader_hentaicosplay (line 49) | class Downloader_hentaicosplay(Downloader):
method fix_url (line 59) | def fix_url(cls, url):
method init (line 66) | def init(self):
method read (line 70) | def read(self):
FILE: src/extractor/hf_downloader.py
class Image (line 11) | class Image:
method __init__ (line 12) | def __init__(self, url, session):
function get_username (line 43) | def get_username(url):
class Downloader_hf (line 50) | class Downloader_hf(Downloader):
method init (line 57) | def init(self):
method fix_url (line 61) | def fix_url(cls, url):
method read (line 65) | def read(self):
function enter (line 78) | def enter():
function get_imgs (line 111) | def get_imgs(username, title, session, cw=None):
FILE: src/extractor/imgur_downloader.py
class Downloader_imgur (line 9) | class Downloader_imgur(Downloader):
method init (line 15) | def init(self):
method id_ (line 19) | def id_(self):
method name (line 23) | def name(self):
method read (line 27) | def read(self):
function get_info (line 43) | def get_info(url):
function get_imgs (line 65) | def get_imgs(url, info=None, cw=None):
FILE: src/extractor/iwara_downloader.py
class Downloader_iwara (line 15) | class Downloader_iwara(Downloader):
method fix_url (line 24) | def fix_url(cls, url):
method init (line 28) | def init(self):
method read (line 32) | def read(self):
class File (line 75) | class File:
method __init__ (line 76) | def __init__(self, type, url, referer, info, session, multi_post=False):
class LazyFile (line 90) | class LazyFile:
method __init__ (line 91) | def __init__(self, url, session, cw):
method get (line 96) | def get(self, url):
function get_token (line 104) | def get_token(session, cw=None): #5794, #6031, #7030
function get_info (line 122) | def get_info(url, session, cw, multi_post=False):
FILE: src/extractor/jmana_downloader.py
class Image (line 14) | class Image:
method __init__ (line 16) | def __init__(self, url, page, p):
class Page (line 23) | class Page:
method __init__ (line 25) | def __init__(self, title, url):
class Downloader_jmana (line 32) | class Downloader_jmana(Downloader):
method init (line 38) | def init(self):
method fix_url (line 65) | def fix_url(cls, url):
method soup (line 69) | def soup(self):
method name (line 78) | def name(self):
method read (line 84) | def read(self):
function get_title (line 98) | def get_title(soup):
function get_artist (line 105) | def get_artist(soup):
function get_imgs_page (line 110) | def get_imgs_page(page, referer, session, cw=None):
function get_pages (line 148) | def get_pages(url, soup, session):
function f (line 170) | def f(url, win):
function get_imgs (line 180) | def get_imgs(url, title, session, soup=None, cw=None):
FILE: src/extractor/kakaotv_downloader.py
class Downloader_kakaotv (line 8) | class Downloader_kakaotv(Downloader):
method fix_url (line 16) | def fix_url(cls, url):
method read (line 20) | def read(self):
class Video (line 33) | class Video:
method __init__ (line 36) | def __init__(self, url, cw=None):
method get (line 41) | def get(self, url):
FILE: src/extractor/kakuyomu_downloader.py
class Page (line 11) | class Page(File):
method __init__ (line 15) | def __init__(self, info):
method get (line 23) | def get(self):
class Downloader_kakuyomu (line 32) | class Downloader_kakuyomu(Downloader):
method read (line 41) | def read(self):
method post_processing (line 54) | def post_processing(self):
function get_text (line 73) | def get_text(page):
function get_info (line 89) | def get_info(url, soup=None, cw=None):
FILE: src/extractor/kissjav_downloader.py
class Downloader_kissjav (line 9) | class Downloader_kissjav(Downloader):
method read (line 16) | def read(self):
function get_video (line 28) | def get_video(url, session, cw):
class Video (line 64) | class Video:
method __init__ (line 65) | def __init__(self, url, url_thumb, referer, title, id, session):
function get_session (line 76) | def get_session(url, cw=None):
FILE: src/extractor/lhscan_downloader.py
class Image (line 15) | class Image:
method __init__ (line 16) | def __init__(self, url, page, p):
method get (line 24) | def get(self, _):
class Page (line 33) | class Page:
method __init__ (line 34) | def __init__(self, title, url):
function get_soup_session (line 39) | def get_soup_session(url, cw=None, win=None):
class Downloader_lhscan (line 50) | class Downloader_lhscan(Downloader):
method init (line 60) | def init(self):
method fix_url (line 68) | def fix_url(cls, url):
method name (line 74) | def name(self):
method read (line 78) | def read(self):
function get_imgs_page (line 90) | def get_imgs_page(page, referer, session, cw=None):
function get_pages (line 151) | def get_pages(url, session, soup=None, cw=None):
function f (line 173) | def f(url, win):
function get_imgs (line 180) | def get_imgs(url, title, session, soup=None, cw=None):
FILE: src/extractor/luscious_downloader.py
class LoginRequired (line 14) | class LoginRequired(errors.LoginRequired):
method __init__ (line 15) | def __init__(self, *args):
class Image (line 19) | class Image:
method __init__ (line 20) | def __init__(self, item, referer):
method get (line 26) | def get(self, url):
class Video (line 33) | class Video:
method __init__ (line 35) | def __init__(self, url, title, url_thumb):
class Downloader_luscious (line 46) | class Downloader_luscious(Downloader):
method fix_url (line 53) | def fix_url(cls, url):
method key_id (line 58) | def key_id(cls, url):
method read (line 61) | def read(self):
function update (line 116) | def update(cw, title, imgs):
function get_imgs (line 123) | def get_imgs(url, soup=None, cw=None):
function get_imgs_p (line 146) | def get_imgs_p(url, p=1):
function get_video (line 162) | def get_video(url, soup):
function get_title (line 172) | def get_title(soup, id=True):
FILE: src/extractor/m3u8_downloader.py
function suitable (line 10) | def suitable(url):
class Downloader_m3u8 (line 14) | class Downloader_m3u8(Downloader):
method fix_url (line 21) | def fix_url(cls, url):
method read (line 26) | def read(self):
class Video (line 41) | class Video:
method __init__ (line 42) | def __init__(self, url, n_thread, referer):
function options (line 86) | def options(urls):
FILE: src/extractor/manatoki_downloader.py
class File_manatoki (line 10) | class File_manatoki(File):
method __init__ (line 15) | def __init__(self, info):
method get (line 29) | def get(self):
method pp (line 32) | def pp(self, filename): #5233
class Page (line 44) | class Page:
method __init__ (line 45) | def __init__(self, title, url):
class Downloader_manatoki (line 52) | class Downloader_manatoki(Downloader):
method init (line 59) | def init(self):
method fix_url (line 88) | def fix_url(cls, url):
method key_id (line 96) | def key_id(cls, url):
method name (line 100) | def name(self):
method read (line 105) | def read(self):
function get_artist (line 118) | def get_artist(soup):
function get_soup (line 126) | def get_soup(url, session=None, cw=None, win=None):
function get_pages (line 148) | def get_pages(url, soup, sub=False):
function f (line 194) | def f(url, win):
function get_imgs (line 203) | def get_imgs(url, title, soup=None, session=None, cw=None):
function get_imgs_page (line 234) | def get_imgs_page(page, title, referer, session, cw):
function isVisible (line 280) | def isVisible(tag):
FILE: src/extractor/mastodon_downloader.py
function get_id (line 8) | def get_id(url):
class Downloader_mastodon (line 13) | class Downloader_mastodon(Downloader):
method init (line 18) | def init(self):
method fix_url (line 22) | def fix_url(cls, url):
method read (line 26) | def read(self):
FILE: src/extractor/misskey_downloader.py
class File_misskey (line 11) | class File_misskey(File):
function get_file (line 16) | def get_file(nid, url, referer, session, p, time):
function get_time (line 28) | def get_time(note):
class Downloader_misskey (line 35) | class Downloader_misskey(Downloader):
method fix_url (line 43) | def fix_url(cls, url):
method init (line 50) | def init(self):
method call (line 57) | def call(self, path, payload):
method read (line 70) | def read(self):
FILE: src/extractor/mrm_downloader.py
class Image (line 8) | class Image:
method __init__ (line 9) | def __init__(self, url, p, page, cw):
method get (line 18) | def get(self, _):
class Page (line 22) | class Page:
method __init__ (line 23) | def __init__(self, title, url, soup=None):
class Downloader_mrm (line 30) | class Downloader_mrm(Downloader):
method init (line 38) | def init(self):
method fix_url (line 42) | def fix_url(cls, url):
method soup (line 46) | def soup(self):
method name (line 61) | def name(self):
method read (line 65) | def read(self):
function get_title (line 76) | def get_title(soup):
function get_imgs (line 83) | def get_imgs(url, soup=None, session=None, cw=None):
function get_pages (line 114) | def get_pages(url, soup=None, session=None):
function get_imgs_page (line 146) | def get_imgs_page(page, session=None, cw=None):
function fix_title (line 167) | def fix_title(title):
function read_html (line 174) | def read_html(url, session, cw):
function get_session (line 188) | def get_session(url, cw=None):
FILE: src/extractor/naver_downloader.py
function get_id (line 10) | def get_id(url):
class Downloader_naver (line 24) | class Downloader_naver(Downloader):
method init (line 30) | def init(self):
method read (line 37) | def read(self):
class Image (line 47) | class Image:
method __init__ (line 48) | def __init__(self, url, referer, p):
class Video (line 55) | class Video:
method __init__ (line 56) | def __init__(self, url, referer, p):
function read_page (line 61) | def read_page(url, session, depth=0):
function get_imgs (line 83) | def get_imgs(url, session, cw):
FILE: src/extractor/navercafe_downloader.py
class LoginRequired (line 6) | class LoginRequired(errors.LoginRequired):
method __init__ (line 7) | def __init__(self, *args):
class Downloader_navercafe (line 11) | class Downloader_navercafe(Downloader):
method init (line 17) | def init(self):
method fix_url (line 21) | def fix_url(cls, url):
method read (line 45) | def read(self):
function get_info (line 54) | def get_info(url, session, cw=None):
class Image (line 133) | class Image(File):
method __init__ (line 137) | def __init__(self, info):
method alter (line 147) | def alter(self):
FILE: src/extractor/naverpost_downloader.py
class Page (line 45) | class Page:
method __init__ (line 46) | def __init__(self, title, url) -> None:
class DownloaderNaverPost (line 52) | class DownloaderNaverPost(Downloader):
method init (line 56) | def init(self) -> None:
method client (line 61) | def client(self):
method read (line 64) | def read(self):
function f (line 77) | def f(url, win):
function get_soup (line 85) | def get_soup(url: str, win=None) -> BeautifulSoup:
function page_soup (line 91) | def page_soup(url: str) -> BeautifulSoup:
function get_img_data_linkdatas (line 100) | def get_img_data_linkdatas(soup: Any) -> Iterator[str]:
function img_src_generator (line 106) | def img_src_generator(linkdatas: Iterator[str]) -> Iterator[str]:
function decode_escapes (line 117) | def decode_escapes(like_html: str) -> str:
class Title (line 136) | class Title:
method __init__ (line 137) | def __init__(self, soup: Any):
method get_profile_title (line 140) | def get_profile_title(self) -> str:
method get_series_title (line 146) | def get_series_title(self) -> str:
method get_title (line 155) | def get_title(self) -> str:
class Total (line 162) | class Total:
method __init__ (line 163) | def __init__(self, soup: Any) -> None:
method get_total_post (line 167) | def get_total_post(self) -> int:
method get_series_total_post (line 173) | def get_series_total_post(self) -> int:
class UrlGenerator (line 179) | class UrlGenerator:
method __init__ (line 180) | def __init__(self, parsed_url: ParseResult, total_count: int) -> None:
method all_post_url_generator (line 188) | def all_post_url_generator(self) -> Iterator[str]:
method all_series_url_generator (line 195) | def all_series_url_generator(self) -> Iterator[str]:
class PostPage (line 204) | class PostPage:
method __init__ (line 205) | def __init__(self, soup: Any):
method all_post_page_generator (line 208) | def all_post_page_generator(self) -> Iterator[List[Page]]:
method all_series_page_generator (line 219) | def all_series_page_generator(self) -> Iterator[List[Page]]:
class Client (line 235) | class Client(Title, Total, UrlGenerator):
method __init__ (line 236) | def __init__(self, parsed_url: ParseResult, soup: BeautifulSoup):
FILE: src/extractor/navertoon_downloader.py
class Page (line 9) | class Page:
method __init__ (line 11) | def __init__(self, url, title, p):
class File_navertoon (line 17) | class File_navertoon(File):
method __init__ (line 21) | def __init__(self, info):
class Info (line 33) | class Info:
method __init__ (line 35) | def __init__(self, id, title, artist):
class Downloader_navertoon (line 42) | class Downloader_navertoon(Downloader):
method init (line 50) | def init(self):
method fix_url (line 54) | def fix_url(cls, url):
method name (line 60) | def name(self):
method read (line 67) | def read(self):
function set_no (line 76) | def set_no(url, p):
function get_id (line 84) | def get_id(url):
function set_page (line 88) | def set_page(url, p):
function get_pages (line 97) | def get_pages(url, cw=None):
function f (line 159) | def f(url):
function get_imgs (line 166) | def get_imgs(page, cw=None):
function get_imgs_all (line 222) | def get_imgs_all(url, title, cw=None):
FILE: src/extractor/navertv_downloader.py
class Downloader_navertv (line 9) | class Downloader_navertv(Downloader):
method fix_url (line 17) | def fix_url(cls, url):
method read (line 22) | def read(self):
class Video (line 35) | class Video:
method __init__ (line 38) | def __init__(self, url, cw=None):
method get (line 43) | def get(self, url):
FILE: src/extractor/newgrounds_downloader.py
class Downloader_newgrounds (line 13) | class Downloader_newgrounds(Downloader):
method init (line 18) | def init(self):
method fix_url (line 22) | def fix_url(cls, url):
method read (line 28) | def read(self):
class Image (line 39) | class Image:
method __init__ (line 40) | def __init__(self, url, filename=None):
function get_posts (line 48) | def get_posts(url, params, session, print_):
function get_html (line 67) | def get_html(post, session):
function get_img (line 71) | def get_img(post, session, print_):
function get_imgs (line 93) | def get_imgs(user, title, session, cw=None):
FILE: src/extractor/nhentai_com_downloader.py
class Downloader_nhentai_com (line 9) | class Downloader_nhentai_com(Downloader):
method init (line 16) | def init(self):
method key_id (line 21) | def key_id(cls, url):
method read (line 25) | def read(self):
class LazyUrl_nhentai_com (line 42) | class LazyUrl_nhentai_com(LazyUrl):
method dump (line 44) | def dump(self):
method load (line 53) | def load(cls, data):
class Image (line 60) | class Image:
method __init__ (line 61) | def __init__(self, url_page, url_img, p):
function get_info (line 70) | def get_info(url):
FILE: src/extractor/nhentai_downloader.py
function get_id (line 9) | def get_id(url):
class File_nhentai (line 16) | class File_nhentai(File):
class Downloader_nhentai (line 21) | class Downloader_nhentai(Downloader):
method init (line 28) | def init(self):
method fix_url (line 32) | def fix_url(cls, url):
method read (line 35) | def read(self):
class Info (line 51) | class Info:
method __init__ (line 52) | def __init__(self, host, id, id_media, title, p, artists, groups, seri...
function get_info (line 67) | def get_info(id, session):
function get_imgs (line 104) | def get_imgs(id, session):
FILE: src/extractor/nico_downloader.py
function get_id (line 17) | def get_id(url):
class LoginRequired (line 22) | class LoginRequired(errors.LoginRequired):
method __init__ (line 23) | def __init__(self, *args):
class Video (line 27) | class Video:
method __init__ (line 28) | def __init__(self, session, info, format, cw, hb=None, d=None, live=Fa...
method get (line 50) | def get(self, _):
method pp (line 82) | def pp(self, filename):
method __repr__ (line 99) | def __repr__(self):
function suitable (line 103) | def suitable(url):
class Downloader_nico (line 112) | class Downloader_nico(Downloader):
method fix_url (line 122) | def fix_url(cls, url):
method init (line 134) | def init(self):
method read (line 140) | def read(self):
function get_video (line 166) | def get_video(session, url, format, cw=None, d=None):
function options (line 251) | def options(urls):
function get_live_from_user (line 258) | def get_live_from_user(url, session):
class Live_nico (line 278) | class Live_nico(utils.Live):
method is_live (line 282) | def is_live(cls, url):
method fix_url (line 289) | def fix_url(cls, url):
method check_live (line 295) | def check_live(cls, url, info=None):
FILE: src/extractor/nijie_downloader.py
function get_id (line 10) | def get_id(url):
function isLogin (line 14) | def isLogin(soup):
class Downloader_nijie (line 21) | class Downloader_nijie(Downloader):
method init (line 28) | def init(self):
method fix_url (line 34) | def fix_url(cls, url):
method name (line 40) | def name(self):
method read (line 45) | def read(self):
class Image (line 60) | class Image(File):
method get (line 63) | def get(self):
function read_soup (line 87) | def read_soup(*args, **kwargs):
function setPage (line 91) | def setPage(url, page):
function get_imgs (line 100) | def get_imgs(url, title=None, session=None, cw=None):
FILE: src/extractor/nozomi_downloader.py
class File_nozomi (line 12) | class File_nozomi(File):
method get (line 16) | def get(self):
function read_post (line 31) | def read_post(id, referer, session, cw):
class Downloader_nozomi (line 57) | class Downloader_nozomi(Downloader):
method init (line 65) | def init(self):
method fix_url (line 69) | def fix_url(cls, url):
method name (line 73) | def name(self):
method read (line 80) | def read(self):
function get_ids (line 117) | def get_ids(q, popular, session, cw):
function get_ids_multi (line 142) | def get_ids_multi(q, popular, session, cw=None):
FILE: src/extractor/pawoo_downloader.py
function get_id (line 9) | def get_id(url):
class Downloader_pawoo (line 14) | class Downloader_pawoo(Downloader):
method init (line 19) | def init(self):
method fix_url (line 28) | def fix_url(cls, url):
method read (line 36) | def read(self):
FILE: src/extractor/pinter_downloader.py
class Downloader_pinter (line 10) | class Downloader_pinter(Downloader):
method init (line 18) | def init(self):
method fix_url (line 39) | def fix_url(cls, url):
method name (line 45) | def name(self):
method read (line 58) | def read(self):
function get_info (line 71) | def get_info(username, board, api):
class PinterestAPI (line 94) | class PinterestAPI:
method __init__ (line 105) | def __init__(self, session):
method pin (line 109) | def pin(self, pin_id):
method pin_related (line 113) | def pin_related(self, pin_id):
method board (line 117) | def board(self, user, board):
method board_pins (line 121) | def board_pins(self, board_id):
method board_related (line 125) | def board_related(self, board_id):
method board_sections (line 129) | def board_sections(self, board_id):
method board_section_pins (line 133) | def board_section_pins(self, section_id):
method board_created (line 137) | def board_created(self, user):
method board_created_pins (line 141) | def board_created_pins(self, user):
method _call (line 147) | def _call(self, resource, options):
method _pagination (line 167) | def _pagination(self, resource, options):
class Image (line 184) | class Image:
method __init__ (line 187) | def __init__(self, img):
function get_imgs (line 211) | def get_imgs(id, api, cw=None, title=None, type='board'):
function get_username_board (line 252) | def get_username_board(url):
FILE: src/extractor/pixiv_downloader.py
class LoginRequired (line 29) | class LoginRequired(errors.LoginRequired):
method __init__ (line 30) | def __init__(self, *args):
class Downloader_pixiv (line 35) | class Downloader_pixiv(Downloader):
method init (line 44) | def init(self):
method fix_url (line 72) | def fix_url(cls, url):
method key_id (line 103) | def key_id(cls, url):
method read (line 106) | def read(self):
class PixivAPIError (line 123) | class PixivAPIError(LoginRequired): pass
class HTTPError (line 124) | class HTTPError(Exception): pass
class PixivAPI (line 127) | class PixivAPI:
method __init__ (line 129) | def __init__(self, session, cw):
method illust_id (line 142) | def illust_id(self, url):
method user_id (line 145) | def user_id(self, url):
method call (line 150) | def call(self, url):
method illust (line 169) | def illust(self, id_):
method pages (line 172) | def pages(self, id_):
method ugoira_meta (line 175) | def ugoira_meta(self, id_):
method profile (line 178) | def profile(self, id_):
method top (line 181) | def top(self, id_):
method bookmarks (line 184) | def bookmarks(self, id_, offset=0, limit=None, rest='show'):
method search (line 189) | def search(self, q, order='date_d', mode='all', p=1, s_mode='s_tag_ful...
method following (line 213) | def following(self, p, r18=False): #4077
class Image (line 220) | class Image:
method __init__ (line 222) | def __init__(self, url, referer, id_, p, info, cw, ugoira=None):
method get (line 234) | def get(self, referer):
method pp (line 252) | def pp(self, filename):
function pretty_tag (line 272) | def pretty_tag(tag):
function tags_matched (line 277) | def tags_matched(tags_illust, tags_add, cw=None):
function get_info (line 318) | def get_info(url, session, cw=None, depth=0, tags_add=None):
function parse_time (line 493) | def parse_time(ds):
function my_id (line 502) | def my_id(session, cw):
function process_user (line 518) | def process_user(id_, info, api):
function process_ids (line 524) | def process_ids(ids, info, imgs, session, cw, depth=0, tags_add=None):
FILE: src/extractor/pornhub_downloader.py
class File (line 20) | class File:
method __init__ (line 26) | def __init__(self, id_, title, url, url_thumb, artist=''):
method thumb (line 44) | def thumb(self):
class Video (line 55) | class Video:
method __init__ (line 64) | def __init__(self, url, cw, session):
method get (line 71) | def get(self, url):
function is_login (line 181) | def is_login(session, cw=None, n=2):
class Downloader_pornhub (line 198) | class Downloader_pornhub(Downloader):
method fix_url (line 209) | def fix_url(cls, url):
method key_id (line 226) | def key_id(cls, url):
method read (line 236) | def read(self):
function fix_soup (line 296) | def fix_soup(soup, url, session=None, cw=None):
class Photo_lazy (line 312) | class Photo_lazy:
method __init__ (line 317) | def __init__(self, url, session):
method get (line 321) | def get(self, url):
class Photo (line 330) | class Photo:
method __init__ (line 335) | def __init__(self, url, referer, id_, session):
function read_album (line 343) | def read_album(url, session=None):
function read_photo (line 364) | def read_photo(url, session=None):
function get_videos (line 381) | def get_videos(url, session, cw=None):
FILE: src/extractor/rule34_xxx_downloader.py
function get_tags (line 10) | def get_tags(url):
class Downloader_rule34_xxx (line 26) | class Downloader_rule34_xxx(Downloader):
method fix_url (line 35) | def fix_url(cls, url):
method name (line 46) | def name(self):
method read (line 52) | def read(self):
class Image (line 64) | class Image:
method __init__ (line 65) | def __init__(self, id_, url):
function setPage (line 71) | def setPage(url, page):
function get_imgs (line 84) | def get_imgs(url, title=None, cw=None):
FILE: src/extractor/sankaku_downloader.py
class LoginRequired (line 20) | class LoginRequired(errors.LoginRequired):
method __init__ (line 21) | def __init__(self, *args):
class File_sankaku (line 26) | class File_sankaku(File):
method get (line 30) | def get(self):
class Downloader_sankaku (line 67) | class Downloader_sankaku(Downloader):
method init (line 75) | def init(self):
method soup (line 87) | def soup(self):
method fix_url (line 91) | def fix_url(cls, url):
method id (line 114) | def id(self):
method name (line 131) | def name(self):
method read (line 134) | def read(self):
function get_imgs_www (line 167) | def get_imgs_www(url, soup):
function setPage (line 182) | def setPage(url, page):
function wait (line 196) | def wait(cw):
function get_imgs (line 200) | def get_imgs(url, title=None, cw=None, types=['img', 'gif', 'video'], se...
function get_id (line 342) | def get_id(url, soup=None):
FILE: src/extractor/soundcloud_downloader.py
function get_cid (line 12) | def get_cid(force=False):
class Audio (line 23) | class Audio:
method __init__ (line 26) | def __init__(self, url, album_art, cw=None):
method get (line 33) | def get(self, url):
method pp (line 91) | def pp(self, filename):
class Downloader_soundcloud (line 97) | class Downloader_soundcloud(Downloader):
method init (line 106) | def init(self):
method fix_url (line 113) | def fix_url(cls, url):
method read (line 116) | def read(self):
function get_audios (line 148) | def get_audios(url, cw, album_art):
FILE: src/extractor/syosetu_downloader.py
class Text (line 12) | class Text(File):
method __init__ (line 16) | def __init__(self, info):
method get (line 28) | def get(self):
function get_id (line 37) | def get_id(url):
class Downloader_syosetu (line 42) | class Downloader_syosetu(Downloader):
method fix_url (line 52) | def fix_url(cls, url):
method read (line 55) | def read(self):
method post_processing (line 121) | def post_processing(self):
function get_title_artist (line 141) | def get_title_artist(soup):
function get_text (line 150) | def get_text(url, subtitle, update, session, cw):
function get_session (line 184) | def get_session():
FILE: src/extractor/talk_op_gg_downloader.py
class DownloaderTalkOPGG (line 35) | class DownloaderTalkOPGG(Downloader):
method init (line 39) | def init(self) -> None:
method read (line 42) | def read(self) -> None:
FILE: src/extractor/tiktok_downloader.py
function is_captcha (line 11) | def is_captcha(soup, cw=None):
class Downloader_tiktok (line 19) | class Downloader_tiktok(Downloader):
method init (line 26) | def init(self):
method fix_url (line 38) | def fix_url(cls, url):
method read (line 44) | def read(self):
class Video (line 78) | class Video:
method __init__ (line 81) | def __init__(self, url, session, format, cw):
method get (line 88) | def get(self, url):
function read_channel (line 106) | def read_channel(url, session, cw=None, title=None):
FILE: src/extractor/tokyomotion_downloader.py
class Downloader_tokyomotion (line 9) | class Downloader_tokyomotion(Downloader):
method init (line 17) | def init(self):
method name (line 26) | def name(self):
method read (line 30) | def read(self):
class Video (line 46) | class Video:
method __init__ (line 47) | def __init__(self, url, url_thumb, referer, filename):
function get_title (line 55) | def get_title(soup):
function get_video (line 64) | def get_video(url, soup=None):
class Image (line 77) | class Image:
method __init__ (line 78) | def __init__(self, url, referer):
function get_imgs (line 83) | def get_imgs(url):
FILE: src/extractor/torrent_downloader.py
function isInfoHash (line 18) | def isInfoHash(s):
class Downloader_torrent (line 29) | class Downloader_torrent(Downloader):
method fix_url (line 54) | def fix_url(cls, url):
method set_max_speed (line 60) | def set_max_speed(cls, speed):
method set_anon (line 65) | def set_anon(cls, flag):
method set_proxy (line 70) | def set_proxy(cls, protocol, host, port, username, password):
method updateSettings (line 76) | def updateSettings(cls):
method _import_torrent (line 85) | def _import_torrent(cls):
method __init (line 91) | def __init(self):
method key_id (line 96) | def key_id(cls, url):
method name (line 106) | def name(self):
method get_dn (line 112) | def get_dn(cls, url):
method read (line 120) | def read(self):
method update_files (line 174) | def update_files(self):
method update_pause (line 190) | def update_pause(self):
method start_ (line 204) | def start_(self):
method _updateIcon (line 291) | def _updateIcon(self):
method update_progress (line 299) | def update_progress(self, h):
method callback (line 310) | def callback(self, h, s, alerts):
method _callback (line 317) | def _callback(self, h, s, alerts):
function actions (line 426) | def actions(cw):
FILE: src/extractor/tumblr_downloader.py
class Image (line 10) | class Image:
method __init__ (line 12) | def __init__(self, url, id, referer, p, cw=None):
method get (line 20) | def get(self, _):
class Downloader_tumblr (line 35) | class Downloader_tumblr(Downloader):
method init (line 41) | def init(self):
method fix_url (line 47) | def fix_url(cls, url):
method read (line 55) | def read(self):
class TumblrAPI (line 66) | class TumblrAPI:
method __init__ (line 80) | def __init__(self, session, cw=None):
method print_ (line 84) | def print_(self, s):
method call (line 88) | def call(self, path, qs, default_qs=True):
method name (line 107) | def name(self, username):
method posts (line 112) | def posts(self, username):
class Post (line 140) | class Post:
method __init__ (line 142) | def __init__(self, data, url, cw=None):
function get_name (line 166) | def get_name(username, session):
function get_imgs (line 170) | def get_imgs(username, session, cw=None):
function get_id (line 191) | def get_id(url):
FILE: src/extractor/twitch_downloader.py
class Downloader_twitch (line 14) | class Downloader_twitch(Downloader):
method init (line 20) | def init(self):
method fix_url (line 32) | def fix_url(cls, url):
method get_filter (line 43) | def get_filter(cls, url):
method read (line 59) | def read(self):
function get_videos (line 96) | def get_videos(url, cw=None):
function alter (line 119) | def alter(seg, cw):
function extract_info (line 132) | def extract_info(url, cw=None):
class Video (line 150) | class Video:
method __init__ (line 153) | def __init__(self, url, session, cw, live=False):
method get (line 160) | def get(self, url):
function get_streamer_name (line 210) | def get_streamer_name(url):
class Live_twitch (line 237) | class Live_twitch(utils.Live):
method is_live (line 241) | def is_live(cls, url):
method check_live (line 245) | def check_live(cls, url, info=None):
FILE: src/extractor/v2ph_downloader.py
function setPage (line 10) | def setPage(url, p):
function getPage (line 17) | def getPage(url):
class Image (line 22) | class Image:
method __init__ (line 23) | def __init__(self, url, referer, p):
method get (line 30) | def get(self, _):
class Downloader_v2ph (line 35) | class Downloader_v2ph(Downloader):
method init (line 43) | def init(self):
method fix_url (line 47) | def fix_url(cls, url):
method read (line 50) | def read(self):
function get_info (line 61) | def get_info(url, session):
function read_soup (line 70) | def read_soup(url, session):
function get_imgs (line 74) | def get_imgs(url, session, title, cw=None):
FILE: src/extractor/vimeo_downloader.py
class Downloader_vimeo (line 8) | class Downloader_vimeo(Downloader):
method init (line 14) | def init(self):
method read (line 18) | def read(self):
function format_ (line 31) | def format_(f):
class Video (line 37) | class Video:
method __init__ (line 40) | def __init__(self, url, cw=None):
method get (line 45) | def get(self, url):
FILE: src/extractor/wayback_machine_downloader.py
class Downloader_wayback_machine (line 14) | class Downloader_wayback_machine(Downloader):
method read (line 20) | def read(self):
class WaybackMachineAPI (line 28) | class WaybackMachineAPI:
method __init__ (line 29) | def __init__(self, session, cw=None):
method call (line 41) | def call(self, url):
method snapshots (line 46) | def snapshots(self, url):
class Filter (line 51) | class Filter:
method __init__ (line 56) | def __init__(self, url, cw=None):
method __get_mode (line 64) | def __get_mode(self):
method __get_title (line 69) | def __get_title(self):
class Bitmap (line 83) | class Bitmap:
method __init__ (line 86) | def __init__(self, size=0, cw=None):
method set (line 90) | def set(self, index):
method unset (line 93) | def unset(self, index):
method get (line 96) | def get(self, index):
method save (line 99) | def save(self, path):
method load (line 104) | def load(self, size, path):
method update (line 109) | def update(self, id_, path):
function get_imgs (line 114) | def get_imgs(url, filter_, directory, session=Session(), cw=None):
FILE: src/extractor/webtoon_downloader.py
class Downloader_webtoon (line 10) | class Downloader_webtoon(Downloader):
method init (line 18) | def init(self):
method fix_url (line 25) | def fix_url(cls, url):
method read (line 28) | def read(self):
class Page (line 41) | class Page:
method __init__ (line 43) | def __init__(self, url, title):
class Image (line 48) | class Image:
method __init__ (line 50) | def __init__(self, url, session, page, p):
function get_imgs (line 58) | def get_imgs(page, session):
function get_main (line 72) | def get_main(url, session):
function set_page (line 79) | def set_page(url, p):
function get_pages (line 89) | def get_pages(url, session=None):
function f (line 126) | def f(url):
function get_imgs_all (line 131) | def get_imgs_all(url, session, title, cw=None):
FILE: src/extractor/weibo_downloader.py
function suitable (line 12) | def suitable(url):
class LoginRequired (line 20) | class LoginRequired(errors.LoginRequired):
method __init__ (line 21) | def __init__(self, *args):
class Downloader_weibo (line 26) | class Downloader_weibo(Downloader):
method init (line 32) | def init(self):
method fix_url (line 36) | def fix_url(cls, url):
method read (line 52) | def read(self):
function checkLogin (line 63) | def checkLogin(session):
class Album (line 69) | class Album:
method __init__ (line 71) | def __init__(self, id, type):
function wait (line 77) | def wait():
class Image (line 81) | class Image(File):
function _get_page_id (line 87) | def _get_page_id(html):
function get_id (line 91) | def get_id(url, cw=None):
function extract_video (line 118) | def extract_video(d):
function get_imgs (line 122) | def get_imgs(uid, title, session, cw=None): #6739
FILE: src/extractor/wikiart_downloader.py
class Image (line 9) | class Image:
method __init__ (line 10) | def __init__(self, url, referer, title, id):
class Downloader_wikiart (line 19) | class Downloader_wikiart(Downloader):
method init (line 25) | def init(self):
method fix_url (line 29) | def fix_url(cls, url):
method read (line 33) | def read(self):
function get_id (line 44) | def get_id(url):
function get_imgs (line 49) | def get_imgs(url, artist, session, cw=None):
function get_artist (line 95) | def get_artist(userid, session):
FILE: src/extractor/xhamster_downloader.py
class Downloader_xhamster (line 9) | class Downloader_xhamster(Downloader):
method init (line 17) | def init(self):
method fix_url (line 25) | def fix_url(cls, url):
method key_id (line 30) | def key_id(cls, url):
method read (line 33) | def read(self):
class Video (line 62) | class Video:
method __init__ (line 65) | def __init__(self, url):
method get (line 70) | def get(self, url):
function get_info (line 94) | def get_info(url): #6318
function read_page (line 112) | def read_page(type_, username, p, session, cw):
function read_channel (line 138) | def read_channel(url, session, cw=None):
class Image (line 176) | class Image:
method __init__ (line 177) | def __init__(self, url, id, referer):
method get (line 182) | def get(self, referer):
function setPage (line 189) | def setPage(url, p):
function read_gallery (line 199) | def read_gallery(url, session, cw=None):
FILE: src/extractor/xnxx_downloader.py
class Video (line 9) | class Video:
method __init__ (line 11) | def __init__(self, url, url_page, title, url_thumb):
method get (line 22) | def get(self, _):
function get_id (line 26) | def get_id(url):
class Downloader_xnxx (line 31) | class Downloader_xnxx(Downloader):
method fix_url (line 39) | def fix_url(cls, url):
method read (line 42) | def read(self):
function get_video (line 49) | def get_video(url):
function get_title (line 71) | def get_title(soup):
FILE: src/extractor/xvideo_downloader.py
function get_id (line 12) | def get_id(url):
class Video (line 19) | class Video:
method __init__ (line 22) | def __init__(self, url_page):
method get (line 26) | def get(self, url_page):
method _get (line 33) | def _get(self, url_page):
method thumb (line 49) | def thumb(self):
class Downloader_xvideo (line 57) | class Downloader_xvideo(Downloader):
method init (line 64) | def init(self):
method fix_url (line 71) | def fix_url(cls, url):
method key_id (line 77) | def key_id(cls, url):
method read (line 83) | def read(self):
function read_channel (line 99) | def read_channel(url_page, cw=None):
FILE: src/extractor/yandere_downloader.py
function read_soup (line 10) | def read_soup(url):
class Downloader_yandere (line 15) | class Downloader_yandere(Downloader):
method fix_url (line 22) | def fix_url(cls, url):
method read (line 29) | def read(self):
method get_id (line 60) | def get_id(self, url:str) -> str:
method get_title (line 64) | def get_title(self, url:str) -> str:
class Image (line 73) | class Image:
method __init__ (line 75) | def __init__(self, url, id_):
method get (line 79) | def get(self, url):
FILE: src/extractor/youku_downloader.py
class Downloader_youku (line 9) | class Downloader_youku(Downloader):
method read (line 15) | def read(self):
class Video (line 25) | class Video:
method __init__ (line 28) | def __init__(self, url, cw=None):
method get (line 32) | def get(self, url):
FILE: src/extractor/youporn_downloader.py
class Downloader_youporn (line 9) | class Downloader_youporn(Downloader):
method fix_url (line 17) | def fix_url(cls, url):
method read (line 22) | def read(self):
class Video (line 33) | class Video:
method __init__ (line 35) | def __init__(self, url, cw=None):
FILE: src/extractor/youtube_downloader.py
function print_streams (line 22) | def print_streams(streams, cw):
class Video (line 30) | class Video(File):
method yt (line 39) | def yt(self):
method thumb (line 57) | def thumb(self):
method get (line 63) | def get(self):
method pp (line 330) | def pp(self, filename, i=0):
method pp_always (line 395) | def pp_always(self, filename):
function get_id (line 416) | def get_id(url):
class Downloader_youtube (line 423) | class Downloader_youtube(Downloader):
method init (line 435) | def init(self):
method fix_url (line 458) | def fix_url(cls, url): #2033
method key_id (line 481) | def key_id(cls, url):
method is_channel_url (line 485) | def is_channel_url(cls, url):
method read (line 492) | def read(self):
function int_ (line 539) | def int_(x):
function get_videos (line 547) | def get_videos(url, session, type='video', only_mp4=False, audio_include...
function read_channel (line 582) | def read_channel(url, n, cw=None, reverse=False):
function read_playlist (line 587) | def read_playlist(url, n, cw=None, reverse=False):
function select (line 619) | def select():
function options (line 698) | def options(urls):
function default_option (line 705) | def default_option():
function get_streamer_name (line 709) | def get_streamer_name(url):
class Live_youtube (line 717) | class Live_youtube(utils.Live):
method is_live (line 721) | def is_live(cls, url):
method fix_url (line 725) | def fix_url(cls, url):
method check_live (line 730) | def check_live(cls, url, info=None):
Condensed preview — 113 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,382K chars).
[
{
"path": ".github/stale.yml",
"chars": 724,
"preview": "# Configuration for probot-stale - https://github.com/probot/stale\n\n# Number of days of inactivity before an Issue or Pu"
},
{
"path": ".gitignore",
"chars": 1184,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "FUNDING.yml",
"chars": 19,
"preview": "patreon: KurtBestor"
},
{
"path": "README.md",
"chars": 3109,
"preview": "<p align=\"center\">\n <img src=\"imgs/card_crop.png\" width=\"50%\"/>\n <br>\n</p>\n\n[![GitHub release](https://img.shields.io/"
},
{
"path": "push_^q^.bat",
"chars": 71,
"preview": "@echo off\n\ngit add .\ngit commit -m \"^q^\"\ngit push\n\necho Done!\npause>nul"
},
{
"path": "src/extractor/_4chan_downloader.py",
"chars": 1141,
"preview": "import downloader\nfrom utils import Downloader, File, clean_title, urljoin, get_ext, limits\nimport utils\n\n\n\nclass File_4"
},
{
"path": "src/extractor/afreeca_downloader.py",
"chars": 7352,
"preview": "import downloader\nfrom utils import Soup, Downloader, Session, try_n, format_filename, cut_pair, File, get_print, print_"
},
{
"path": "src/extractor/artstation_downloader.py",
"chars": 6928,
"preview": "#coding:utf8\nimport downloader\nfrom error_printer import print_error\nfrom translator import tr_\nfrom utils import Downlo"
},
{
"path": "src/extractor/asmhentai_downloader.py",
"chars": 2691,
"preview": "#coding: utf8\nimport downloader\nimport ree as re\nfrom utils import Soup, urljoin, Downloader, join, Session, File, clean"
},
{
"path": "src/extractor/avgle_downloader.py",
"chars": 2036,
"preview": "#coding: utf8\nimport downloader\nfrom m3u8_tools import M3u8_stream\nfrom utils import Soup, Downloader, LazyUrl, get_prin"
},
{
"path": "src/extractor/baraag_downloader.py",
"chars": 773,
"preview": "#coding:utf8\nfrom utils import Downloader, clean_title, Session\nfrom mastodon import get_info\nimport ree as re\n\n\n\ndef ge"
},
{
"path": "src/extractor/bcy_downloader.py",
"chars": 4633,
"preview": "#coding:utf8\nimport downloader\nfrom utils import Soup, cut_pair, LazyUrl, Downloader, get_print, get_max_range, try_n, c"
},
{
"path": "src/extractor/bdsmlr_downloader.py",
"chars": 4362,
"preview": "#coding:utf8\nimport downloader\nfrom utils import Session, Soup, LazyUrl, Downloader, get_max_range, try_n, get_print, cl"
},
{
"path": "src/extractor/bili_downloader.py",
"chars": 7351,
"preview": "import downloader\nimport downloader_v3\nfrom utils import Downloader, get_print, format_filename, clean_title, get_resolu"
},
{
"path": "src/extractor/coub_downloader.py",
"chars": 1961,
"preview": "from utils import Downloader, LazyUrl, try_n, format_filename, get_ext\nimport ytdl\nfrom io import BytesIO as IO\nimport d"
},
{
"path": "src/extractor/danbooru_downloader.py",
"chars": 5449,
"preview": "#coding: utf-8\nimport downloader\nimport ree as re\nfrom utils import Downloader, Session, get_max_range, clean_title, get"
},
{
"path": "src/extractor/discord_emoji_downloader.py",
"chars": 4416,
"preview": "# coding: UTF-8\n# title: Discord 서버 커스텀 이모지 다운로드\n# author: SaidBySolo\n\n\"\"\"\nMIT License\n\nCopyright (c) 2020 SaidBySolo\n\nP"
},
{
"path": "src/extractor/etc_downloader.py",
"chars": 9639,
"preview": "import downloader\nimport ytdl\nfrom utils import Downloader, Session, try_n, LazyUrl, get_ext, format_filename, get_print"
},
{
"path": "src/extractor/fc2_downloader.py",
"chars": 2615,
"preview": "import downloader\nimport ree as re\nfrom utils import urljoin, Downloader, format_filename, Soup, LazyUrl, get_print, Ses"
},
{
"path": "src/extractor/file_downloader.py",
"chars": 3119,
"preview": "import downloader, os\nfrom utils import Downloader, query_url, clean_title, get_ext, Session, Soup, File, urljoin, fix_d"
},
{
"path": "src/extractor/flickr_downloader.py",
"chars": 3122,
"preview": "from utils import Downloader, File, Session, urljoin, get_ext, clean_title, Soup, limits\nimport utils\nimport ree as re\ni"
},
{
"path": "src/extractor/gelbooru_downloader.py",
"chars": 4153,
"preview": "#coding: utf-8\nimport downloader\nimport ree as re\nfrom utils import Downloader, urljoin, query_url, get_max_range, get_p"
},
{
"path": "src/extractor/hameln_downloader.py",
"chars": 4042,
"preview": "#coding: utf8\nimport downloader\nimport os\nimport utils\nfrom utils import Soup, urljoin, get_text, LazyUrl, try_n, Downlo"
},
{
"path": "src/extractor/hanime_downloader.py",
"chars": 3844,
"preview": "import downloader\nfrom utils import Session, Downloader, try_n, Soup, format_filename, get_print, get_resolution, json\ni"
},
{
"path": "src/extractor/hentaicosplay_downloader.py",
"chars": 5044,
"preview": "#coding: utf8\nimport downloader\nfrom utils import Downloader, Session, Soup, LazyUrl, urljoin, get_ext, clean_title, try"
},
{
"path": "src/extractor/hf_downloader.py",
"chars": 4412,
"preview": "#coding:utf8\nimport downloader\nfrom utils import Soup, urljoin, Session, LazyUrl, Downloader, try_n, clean_title, check_"
},
{
"path": "src/extractor/imgur_downloader.py",
"chars": 3980,
"preview": "import downloader\nfrom utils import Downloader, Soup, try_n, urljoin, get_max_range, clean_title, cut_pair, check_alive,"
},
{
"path": "src/extractor/iwara_downloader.py",
"chars": 10539,
"preview": "import downloader\nfrom utils import Soup, urljoin, Downloader, LazyUrl, get_print, clean_url, clean_title, check_alive, "
},
{
"path": "src/extractor/jmana_downloader.py",
"chars": 5862,
"preview": "import downloader\nfrom utils import Soup, urljoin, Downloader, fix_title, Session, get_print, LazyUrl, clean_title, get_"
},
{
"path": "src/extractor/kakaotv_downloader.py",
"chars": 1437,
"preview": "import downloader\nimport ytdl\nfrom utils import Downloader, try_n, LazyUrl, get_ext, format_filename\nfrom io import Byte"
},
{
"path": "src/extractor/kakuyomu_downloader.py",
"chars": 4289,
"preview": "#coding:utf8\nimport downloader\nimport utils\nfrom utils import Soup, urljoin, Downloader, LazyUrl, try_n, clean_title, ge"
},
{
"path": "src/extractor/kissjav_downloader.py",
"chars": 2251,
"preview": "import downloader\nfrom utils import urljoin, Downloader, LazyUrl, Session, try_n, format_filename, get_resolution, get_p"
},
{
"path": "src/extractor/lhscan_downloader.py",
"chars": 5978,
"preview": "#coding:utf8\nimport downloader\nfrom utils import Soup, urljoin, LazyUrl, Downloader, try_n, Session, clean_title, get_pr"
},
{
"path": "src/extractor/luscious_downloader.py",
"chars": 7146,
"preview": "#coding:utf8\nimport downloader\nimport utils\nfrom utils import Soup, Downloader, LazyUrl, urljoin, try_n, clean_title, ge"
},
{
"path": "src/extractor/m3u8_downloader.py",
"chars": 3345,
"preview": "from utils import Downloader, LazyUrl, clean_title, Session, get_ext\nimport utils\nfrom m3u8_tools import playlist2stream"
},
{
"path": "src/extractor/manatoki_downloader.py",
"chars": 8415,
"preview": "from utils import Soup, try_n, Downloader, urljoin, get_print, Session, clean_title, get_ext, fix_title, lazy, get_imgs_"
},
{
"path": "src/extractor/mastodon_downloader.py",
"chars": 774,
"preview": "#coding:utf8\nfrom utils import Downloader, clean_title, Session\nfrom mastodon import get_info\nimport ree as re\n\n\n\ndef ge"
},
{
"path": "src/extractor/misskey_downloader.py",
"chars": 4538,
"preview": "from utils import Downloader, Session, clean_title, get_ext, check_alive, tr_, try_n, File, get_max_range, limits\nimport"
},
{
"path": "src/extractor/mrm_downloader.py",
"chars": 5174,
"preview": "#coding:utf8\nfrom utils import Soup, urljoin, LazyUrl, Downloader, try_n, get_print, clean_title, get_ext, check_alive\nf"
},
{
"path": "src/extractor/naver_downloader.py",
"chars": 4884,
"preview": "#coding:utf-8\nimport downloader\nimport ree as re\nfrom utils import urljoin, Downloader, Soup, LazyUrl, clean_title, get_"
},
{
"path": "src/extractor/navercafe_downloader.py",
"chars": 4718,
"preview": "# coding:utf8\nfrom utils import Downloader, get_print, urljoin, Soup, get_ext, File, clean_title, downloader, re, try_n,"
},
{
"path": "src/extractor/naverpost_downloader.py",
"chars": 8780,
"preview": "# coding: UTF-8\n# title: Download naver post image\n# author: SaidBySolo\n# comment: 네이버 포스트의 이미지를 다운로드합니다\n\n\"\"\"\nMIT Licens"
},
{
"path": "src/extractor/navertoon_downloader.py",
"chars": 7753,
"preview": "import downloader\nfrom utils import Soup, urljoin, Downloader, get_imgs_already, clean_title, get_ext, get_print, errors"
},
{
"path": "src/extractor/navertv_downloader.py",
"chars": 1614,
"preview": "import downloader\nimport ree as re\nfrom io import BytesIO as IO\nfrom utils import Downloader, LazyUrl, get_ext, format_f"
},
{
"path": "src/extractor/newgrounds_downloader.py",
"chars": 3662,
"preview": "# coding:utf8\nfrom datetime import datetime\nimport downloader\nimport errors\nfrom functools import reduce\nimport os\nimpor"
},
{
"path": "src/extractor/nhentai_com_downloader.py",
"chars": 2747,
"preview": "#coding:utf8\nimport downloader\nimport ree as re\nfrom utils import urljoin, LazyUrl, Downloader, try_n, join, json\nimport"
},
{
"path": "src/extractor/nhentai_downloader.py",
"chars": 3363,
"preview": "#coding:utf8\nimport downloader\nimport ree as re\nfrom utils import urljoin, File, Downloader, try_n, join, get_ext, json\n"
},
{
"path": "src/extractor/nico_downloader.py",
"chars": 10042,
"preview": "#coding:utf8\nimport downloader\nfrom io import BytesIO\nimport ree as re\nfrom utils import Downloader, get_print, format_f"
},
{
"path": "src/extractor/nijie_downloader.py",
"chars": 3929,
"preview": "#coding: utf-8\nimport downloader\nfrom utils import Downloader, Session, urljoin, get_max_range, get_print, clean_title, "
},
{
"path": "src/extractor/nozomi_downloader.py",
"chars": 5430,
"preview": "import downloader\nfrom urllib.parse import quote\nfrom io import BytesIO\nfrom utils import Downloader, query_url, get_ext"
},
{
"path": "src/extractor/pawoo_downloader.py",
"chars": 1155,
"preview": "#coding:utf8\nfrom utils import Downloader, clean_title, Session, Soup, urljoin\nimport clf2\nfrom mastodon import get_info"
},
{
"path": "src/extractor/pinter_downloader.py",
"chars": 8523,
"preview": "from utils import Session, Downloader, LazyUrl, clean_url, try_n, clean_title, get_ext, get_max_range, check_alive, limi"
},
{
"path": "src/extractor/pixiv_downloader.py",
"chars": 20999,
"preview": "import downloader\nfrom utils import Downloader, urljoin, clean_title, LazyUrl, get_ext, get_print, try_n, compatstr, get"
},
{
"path": "src/extractor/pornhub_downloader.py",
"chars": 17022,
"preview": "#coding:utf8\n'''\nPornhub Downloader\n'''\nfrom io import BytesIO\nimport downloader\nimport ree as re\nfrom utils import (Dow"
},
{
"path": "src/extractor/rule34_xxx_downloader.py",
"chars": 3302,
"preview": "import downloader\nimport ree as re\nimport os\nfrom utils import Downloader, query_url, Soup, get_max_range, get_print, cl"
},
{
"path": "src/extractor/sankaku_downloader.py",
"chars": 11439,
"preview": "#coding: utf-8\n#https://chan.sankakucomplex.com/\n#https://idol.sankakucomplex.com/\n#https://beta.sankakucomplex.com/\n#ht"
},
{
"path": "src/extractor/soundcloud_downloader.py",
"chars": 5229,
"preview": "#coding: utf8\nimport downloader\nfrom io import BytesIO\nfrom utils import Downloader, LazyUrl, get_print, try_n, lock, ge"
},
{
"path": "src/extractor/syosetu_downloader.py",
"chars": 6279,
"preview": "#coding:utf8\nimport downloader\nimport utils\nfrom utils import urljoin, try_n, Downloader, clean_title, Session, File, ch"
},
{
"path": "src/extractor/talk_op_gg_downloader.py",
"chars": 1708,
"preview": "# coding: UTF-8\n# title: Download talk op.gg image\n# author: SaidBySolo\n# comment: op.gg 커뮤니티의 이미지를 다운로드합니다\n\n\"\"\"\nMIT Lic"
},
{
"path": "src/extractor/tiktok_downloader.py",
"chars": 4981,
"preview": "import downloader\nimport ree as re\nfrom utils import Soup, LazyUrl, Downloader, try_n, compatstr, get_print, Session, ge"
},
{
"path": "src/extractor/tokyomotion_downloader.py",
"chars": 2748,
"preview": "#coding:utf8\nimport downloader\nfrom utils import Soup, Downloader, LazyUrl, clean_title, format_filename\nfrom io import "
},
{
"path": "src/extractor/torrent_downloader.py",
"chars": 15921,
"preview": "from utils import Downloader, clean_title, lock, json\nimport constants, os, downloader\nfrom size import Size\nfrom timee "
},
{
"path": "src/extractor/tumblr_downloader.py",
"chars": 6490,
"preview": "#coding:utf8\nimport downloader\nfrom translator import tr_\nfrom utils import Session, query_url, get_max_range, Downloade"
},
{
"path": "src/extractor/twitch_downloader.py",
"chars": 8743,
"preview": "#coding: utf8\nimport downloader\nimport ytdl\nfrom utils import Downloader, LazyUrl, try_n, format_filename, get_ext, Sess"
},
{
"path": "src/extractor/v2ph_downloader.py",
"chars": 2744,
"preview": "#coding:utf8\nimport downloader\nfrom utils import get_ext, LazyUrl, Downloader, try_n, clean_title, get_print, print_erro"
},
{
"path": "src/extractor/vimeo_downloader.py",
"chars": 1949,
"preview": "import downloader\nfrom io import BytesIO as IO\nfrom utils import Downloader, LazyUrl, get_ext, format_filename, try_n, g"
},
{
"path": "src/extractor/wayback_machine_downloader.py",
"chars": 5934,
"preview": "# coding: utf8\n# title: Wayback Machine Downloader\n# author: bog_4t\nimport downloader\nimport concurrent.futures\nimport o"
},
{
"path": "src/extractor/webtoon_downloader.py",
"chars": 4281,
"preview": "import downloader\nfrom utils import Soup, Session, LazyUrl, clean_title, get_ext, get_imgs_already, urljoin, try_n, Down"
},
{
"path": "src/extractor/weibo_downloader.py",
"chars": 5968,
"preview": "#coding:utf8\nimport downloader\nimport ree as re\nfrom utils import Downloader, Session, get_print, clean_title, Soup, fix"
},
{
"path": "src/extractor/wikiart_downloader.py",
"chars": 2587,
"preview": "#coding:utf8\nimport downloader\nfrom utils import LazyUrl, Downloader, Session, get_print, clean_title, check_alive, json"
},
{
"path": "src/extractor/xhamster_downloader.py",
"chars": 7011,
"preview": "import downloader, ree as re\nfrom utils import Downloader, Session, LazyUrl, get_print, get_ext, try_n, format_filename,"
},
{
"path": "src/extractor/xnxx_downloader.py",
"chars": 1804,
"preview": "import downloader\nfrom utils import Soup, Downloader, LazyUrl, format_filename\nimport ree as re\nfrom m3u8_tools import p"
},
{
"path": "src/extractor/xvideo_downloader.py",
"chars": 4601,
"preview": "import downloader\nfrom utils import Downloader, Soup, LazyUrl, urljoin, format_filename, Session, get_ext, get_print, ge"
},
{
"path": "src/extractor/yandere_downloader.py",
"chars": 2519,
"preview": "from utils import Downloader, urljoin, clean_title, try_n, check_alive, LazyUrl, get_ext, get_max_range, limits\nfrom tra"
},
{
"path": "src/extractor/youku_downloader.py",
"chars": 1454,
"preview": "import downloader\nimport ytdl\nfrom m3u8_tools import M3u8_stream\nfrom utils import LazyUrl, Downloader, format_filename\n"
},
{
"path": "src/extractor/youporn_downloader.py",
"chars": 1390,
"preview": "import downloader\nfrom io import BytesIO\nfrom utils import Downloader, LazyUrl, get_ext, format_filename, try_n\nimport y"
},
{
"path": "src/extractor/youtube_downloader.py",
"chars": 28365,
"preview": "#coding: utf-8\nimport ytdl\nimport downloader\nimport downloader_v3\nfrom error_printer import print_error\nfrom timee impor"
},
{
"path": "translation/changelog_en.txt",
"chars": 88469,
"preview": "4.2 【Oct 27, 2024】\n\n[버그 해결 / 사이트 변경에 의한 수정]\n\n- 스크립트 호환성 문제 해결 (#6943)\n\n- TVer 문제 해결 (#6956)\n\n- 필터 북마크 내보내기 / 가져오기 형식 "
},
{
"path": "translation/changelog_ko.txt",
"chars": 82393,
"preview": "4.2 【Oct 27, 2024】\n\n[버그 해결 / 사이트 변경에 의한 수정]\n\n- 스크립트 호환성 문제 해결 (#6943)\n\n- TVer 문제 해결 (#6956)\n\n- 필터 북마크 내보내기 / 가져오기 형식 "
},
{
"path": "translation/help_en.html",
"chars": 8651,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">How to use</h1>\n<p align=\"right\">{dat"
},
{
"path": "translation/help_ja.html",
"chars": 7010,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">使い方</h1>\n<p align=\"right\">{date}</p>\n"
},
{
"path": "translation/help_ko.html",
"chars": 6905,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">사용법</h1>\n<p align=\"right\">{date}</p>\n"
},
{
"path": "translation/help_pl.html",
"chars": 8414,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">Jak urzywać</h1>\n<p align=\"right\">{da"
},
{
"path": "translation/help_ru.html",
"chars": 9255,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">Как пользоваться</h1>\n<p align=\"right"
},
{
"path": "translation/help_si.html",
"chars": 12103,
"preview": "<html>\n <head>\n {head}\n </head>\n\n <body>\n <p align=\"right\"><br /></p>\n <h1 align=\"center\">භාවිතා කරන ආකාරය</"
},
{
"path": "translation/help_zh.html",
"chars": 5668,
"preview": "<html>\n\n<head>\n{head}\n</head>\n\n<body>\n<p align=\"right\"><br></p>\n<h1 align=\"center\">使用方法</h1>\n<p align=\"right\">{date}</p>"
},
{
"path": "translation/qt_ar.json",
"chars": 10728,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"أسطوانة\",\n \"Go\": \"انطلاق\",\n \"No\": \"لا\",\n \"Up\": \"الأعلى\",\n \"Alt\":"
},
{
"path": "translation/qt_es.json",
"chars": 8029,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"No\": \"No\",\n \"Up\": \"Arriba\",\n \"Alt\": \"Alt\",\n \"F%1\": \"F%1\",\n \"Del\": \"Sup"
},
{
"path": "translation/qt_fr.json",
"chars": 11377,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"Aller\",\n \"No\": \"Non\",\n \"Up\": \"Haut\",\n \"Alt\": \"Alt\","
},
{
"path": "translation/qt_ja.json",
"chars": 9517,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"確定\",\n \"No\": \"いいえ\",\n \"Up\": \"↑\",\n \"Alt\": \"Alt\",\n \""
},
{
"path": "translation/qt_ko.json",
"chars": 10320,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"이동\",\n \"No\": \"아니오\",\n \"Up\": \"위\",\n \"Alt\": \"Alt\",\n \""
},
{
"path": "translation/qt_pl.json",
"chars": 11027,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"Przejdź\",\n \"No\": \"Nie\",\n \"Up\": \"Góra\",\n \"Alt\": \"Alt"
},
{
"path": "translation/qt_pt.json",
"chars": 6882,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"No\": \"Não\",\n \"Up\": \"Cima\",\n \"Alt\": \"Alt\",\n \"F%1\": \"F%1\",\n \"Del\": \"Dele"
},
{
"path": "translation/qt_ru.json",
"chars": 11043,
"preview": "{\n \"QShortcut\": {\n \"Space\": \"Пробел\"\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"Перейти\",\n \"No\": \"Нет\",\n \"Up\": \""
},
{
"path": "translation/qt_si.json",
"chars": 12215,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"යන්න\",\n \"No\": \"නෑ\",\n \"Up\": \"ඉහළට\",\n \"Alt\": \"Alt\",\n "
},
{
"path": "translation/qt_tr.json",
"chars": 10479,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD\",\n \"Go\": \"Git\",\n \"No\": \"Hayır\",\n \"Up\": \"Yukarı\",\n \"Alt\": \"Alt"
},
{
"path": "translation/qt_vi.json",
"chars": 6851,
"preview": "{\n \"QMessageBox\": {\n \"OK\": \"Đồng ý\",\n \"Help\": \"Trợ giúp\",\n \"Show Details...\": \"Hiện chi tiết...\",\n \"Hide De"
},
{
"path": "translation/qt_zh-TW.json",
"chars": 8046,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"CD\": \"CD 光碟\",\n \"Go\": \"前往\",\n \"No\": \"否\",\n \"Up\": \"上鍵\",\n \"Alt\": \"Alt\",\n "
},
{
"path": "translation/qt_zh.json",
"chars": 7940,
"preview": "{\n \"QShortcut\": {\n \"+\": \"+\",\n \"No\": \"否\",\n \"Up\": \"Up\",\n \"Alt\": \"Alt\",\n \"F%1\": \"F%1\",\n \"Del\": \"Del\",\n "
},
{
"path": "translation/tr_ar.hdl",
"chars": 38783,
"preview": "{\n \"lang\": \"ar\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_en.hdl",
"chars": 38763,
"preview": "{\n \"lang\": \"en\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_es.hdl",
"chars": 40193,
"preview": "{\n \"lang\": \"es\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_fr.hdl",
"chars": 41154,
"preview": "{\n \"lang\": \"fr\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_ja.hdl",
"chars": 32314,
"preview": "{\n \"lang\": \"ja\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"「一部の提供元を除外」が有効なら、フォルダは複数必要です\",\n \"#Cancel"
},
{
"path": "translation/tr_ko.hdl",
"chars": 13910,
"preview": "{\n \"lang\": \"ko\",\n \"items\": {\n \"#Cancel#\": \"취소\",\n \"#EB#\": \"{} EB\",\n \"#EMPTY#\": \"Empty (´･ω･`)\",\n \"#GB#\": "
},
{
"path": "translation/tr_pl.hdl",
"chars": 40459,
"preview": "{\n \"lang\": \"pl\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_pt.hdl",
"chars": 40946,
"preview": "{\n \"lang\": \"pt\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_ru.hdl",
"chars": 41292,
"preview": "{\n \"lang\": \"ru\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Папок должно быть больше, чем одна, если «И"
},
{
"path": "translation/tr_si.hdl",
"chars": 40256,
"preview": "{\n \"lang\": \"si\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_tr.hdl",
"chars": 38781,
"preview": "{\n \"lang\": \"tr\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_vi.hdl",
"chars": 39681,
"preview": "{\n \"lang\": \"vi\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclude "
},
{
"path": "translation/tr_zh-TW.hdl",
"chars": 32159,
"preview": "{\n \"lang\": \"zh-tw\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"Folders must be more than one if \\\"Exclu"
},
{
"path": "translation/tr_zh.hdl",
"chars": 31180,
"preview": "{\n \"lang\": \"zh\",\n \"items\": {\n \"\\\"같은 소스 제외\\\" 를 켠 경우 폴더는 두 개 이상이어야 합니다\": \"如果开启了“排除相同来源”选项，文件夹的数量必须要超过一个。\",\n \"#Canc"
}
]
About this extraction
This page contains the full source code of the KurtBestor/Hitomi-Downloader GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 113 files (1.2 MB), approximately 435.0k tokens, and a symbol index with 841 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.