[ { "path": ".gitignore", "content": "HELP.md\ntarget/\n!.mvn/wrapper/maven-wrapper.jar\n!**/src/main/**\n!**/src/test/**\n\n### STS ###\n.apt_generated\n.classpath\n.factorypath\n.project\n.settings\n.springBeans\n.sts4-cache\n\n### IntelliJ IDEA ###\n.idea\n*.iws\n*.iml\n*.ipr\n\n### NetBeans ###\n/nbproject/private/\n/nbbuild/\n/dist/\n/nbdist/\n/.nb-gradle/\nbuild/\n\n### VS Code ###\n.vscode/\n" }, { "path": "README-JAV.md", "content": "- IPX=265 5T2\n- SSNI=256 停更 5T2\n- SSIS=162 5T4\n- MIAA=135 5T5\n- STARS=132 5T3\n- MIDE=128 5T3\n- FSDSS=128 5T4\n- JUL=103 5T4\n- JUFE=85 5T5\n- HND=85 5T1\n- /\n- ABP=84 5T1\n- CAWD=71 5T4\n- FCH=66 停更 5T5\n- DASD=58 5T3\n- ABW=54 5T4\n- CJOD=53 5T5\n- PPPD=52 5T3\n- RKI=46 5T5\n- PRED=45 5T1\n- MGMQ=45 5T5\n- /\n- DPMI=45 5T3\n- DNJR=44 5T5\n- MSFH=42 5T5 停更\n- MKMP=41 5T3\n- MOPG=39 停更 5T5\n- DMOW=37 停更 5T5\n- GVH=37 5T3\n- OTIM=34 停更 5T3\n- GENM=33 5T5\n- MEYD=32 5T1\n- /\n- OKP=32 5T2\n- MVSD=31 5T3\n- WANZ=30 停更 5T1\n- MGMJ=30 停更 5T5\n- BLK=30 5T3\n- APNS=29 5T3\n- MGMP=27 5T5\n- EKDV=27 5T1\n- EBOD=26 5T2\n- OFJE=26 5T2\n- /\n- ARM=25 5T5\n- MDTM=23 5T1\n- AARM=22 5T5\n- IPVR=21 5T4\n- CLOT=18 5T5\n- TPPN=18 5T3\n- ECB=18 5T1\n- MMUS=16\n- DVAJ=16\n- SHKD=16\n\n\n---\n\nOVG-149\nPFES-003\nPFES-012\nRCTD-108\nSNIS-792\nSW-690\nXMOM-016\n\n---\n\nIPX-992\nIPX-838\nIPX-819\n\nIPX-916\nIPX-851\nIPX-832\nIPX-812\n\nJUFE-400\nJUFE-385\nJUFE-375\n\nCJOD-340\n\nRKI-623\n\nMGMQ\n\nMOPT-019\nMOPT-021\n\nAARM-109\nAARM-110\n\n松本いちか\n\nVRKM-497\nEBVR-035" }, { "path": "README.md", "content": "# OpenYspider 4.x\n\n千万级图片、视频爬虫 [开源版本]\n\n![](swagger3.png)\n\n## 简介\n\nOpenYspider 是一个使用 Java 编写的简单爬虫。主要用到的技术栈有：\n\n1. spring-boot-starter-web\n2. spring-boot-starter-test\n3. mybatis-plus-boot-starter\n4. springfox-boot-starter\n5. lombok\n6. jsoup\n7. mockito + jacoco\n\n当前 LTS 的网站有：\n\n1. `tujidao.com`\n\nDeprecated 的网站（请于历史提交中查看）：\n\n1. `tangyun365.com`\n2. `yalayi.com`\n3. `rosmm88.com`\n4. `mzsock.com`\n5. `meinvla.net`\n6. `leetcode-cn.com`\n\n## 开发环境\n\n`Windows 11` + `JDK 17` + `Mysql 8.x`\n\n```sh\n$ java --version\nopenjdk 17.0.1 2021-10-19\nOpenJDK Runtime Environment (build 17.0.1+12-39)\nOpenJDK 64-Bit Server VM (build 17.0.1+12-39, mixed mode, sharing)\n```\n\n运行启动类 `OpenYspiderApplication` 后，浏览器访问 [http://localhost:23333/swagger-ui/index.html#/](http://localhost:23333/swagger-ui/index.html#/)\n\n数据库脚本: [sql_scripts](./sql_scripts/oys3_all_scripts_mysql.sql)\n\n## 爬取网站\n\n数据统计截止 2022-02-12\n\n### 1 图集岛（原美图日） [ 2,647,717P / 905G ]\n\n- 目标网站：[https://www.tujidao.com/](https://www.tujidao.com/)\n- 特点：图片路径可遍历\n\n```sql\nselect count(*) from oys_tujidao_album_t where album_id > 0 and album_id <= 10000; -- 9995 ok\nselect count(*) from oys_tujidao_album_t where album_id > 10000 and album_id <= 20000; -- 10000\nselect count(*) from oys_tujidao_album_t where album_id > 20000 and album_id <= 30000; -- 9999 [23001]\nselect count(*) from oys_tujidao_album_t where album_id > 30000 and album_id <= 40000; -- 10000\nselect count(*) from oys_tujidao_album_t where album_id > 40000 and album_id <= 50000; -- 8925 [46018]\n```\n\n## 部分成果展示\n\n![](result1.png)\n\n![](result2.png)\n" }, { "path": "oys-boot-starter/pom.xml", "content": "\n\n \n openyspider\n com.devyy\n ${revision}\n \n 4.0.0\n\n oys-boot-starter\n\n \n \n com.devyy\n oys-tujidao\n ${revision}\n \n \n com.devyy\n oys-codeforces\n ${revision}\n \n \n\n \n \n \n org.springframework.boot\n spring-boot-maven-plugin\n ${org.springframework.boot.spring-boot-dependencies.version}\n \n \n \n org.jacoco\n jacoco-maven-plugin\n ${jacoco-maven-plugin.version}\n \n \n report\n verify\n \n report-aggregate\n \n \n \n \n \n \n" }, { "path": "oys-boot-starter/src/main/java/com/devyy/oys/srarter/OpenYspiderApplication.java", "content": "package com.devyy.oys.srarter;\n\nimport org.springframework.boot.SpringApplication;\nimport org.springframework.boot.autoconfigure.SpringBootApplication;\nimport org.springframework.context.annotation.ComponentScan;\n\n/**\n * 启动类\n *\n * @since 2019-12-01\n */\n@SpringBootApplication\n@ComponentScan(\"com.devyy.oys\")\npublic class OpenYspiderApplication {\n public static void main(String[] args) {\n SpringApplication.run(OpenYspiderApplication.class, args);\n }\n}\n\n" }, { "path": "oys-boot-starter/src/main/java/com/devyy/oys/srarter/config/MybatisPlusConfig.java", "content": "package com.devyy.oys.srarter.config;\n\nimport org.mybatis.spring.annotation.MapperScan;\nimport org.springframework.context.annotation.Configuration;\n\n/**\n * Mybatis-Plus 配置类\n *\n * @since 2020-03-22\n */\n@Configuration\n@MapperScan(\"com.devyy.oys.**.dao\")\npublic class MybatisPlusConfig {\n}\n" }, { "path": "oys-boot-starter/src/main/java/com/devyy/oys/srarter/config/Swagger3Config.java", "content": "package com.devyy.oys.srarter.config;\n\nimport io.swagger.annotations.ApiOperation;\n\nimport org.springframework.context.annotation.Bean;\nimport org.springframework.context.annotation.Configuration;\n\nimport springfox.documentation.builders.ApiInfoBuilder;\nimport springfox.documentation.builders.PathSelectors;\nimport springfox.documentation.builders.RequestHandlerSelectors;\nimport springfox.documentation.oas.annotations.EnableOpenApi;\nimport springfox.documentation.service.ApiInfo;\nimport springfox.documentation.spi.DocumentationType;\nimport springfox.documentation.spring.web.plugins.Docket;\n\n/**\n * Swagger3 配置类\n *\n * http://localhost:23333/swagger-ui/index.html\n */\n@EnableOpenApi\n@Configuration\npublic class Swagger3Config {\n @Bean\n public Docket createRestApi() {\n return new Docket(DocumentationType.OAS_30)\n .apiInfo(apiInfo())\n .select()\n .apis(RequestHandlerSelectors.withMethodAnnotation(ApiOperation.class))\n .paths(PathSelectors.any())\n .build();\n }\n\n private ApiInfo apiInfo() {\n return new ApiInfoBuilder()\n .title(\"OpenYspider 千万级图片爬虫、视频爬虫 [开源版本]\")\n .description(\"Github: https://github.com/RxGirlz/OpenYspider\")\n .version(\"v4.x\")\n .build();\n }\n\n}\n" }, { "path": "oys-boot-starter/src/main/java/com/devyy/oys/srarter/main/JavMain.java", "content": "package com.devyy.oys.srarter.main;\n\nimport com.kennycason.kumo.CollisionMode;\nimport com.kennycason.kumo.WordCloud;\nimport com.kennycason.kumo.WordFrequency;\nimport com.kennycason.kumo.bg.CircleBackground;\nimport com.kennycason.kumo.font.scale.LinearFontScalar;\nimport com.kennycason.kumo.nlp.FrequencyAnalyzer;\nimport com.kennycason.kumo.nlp.normalize.UpperCaseNormalizer;\nimport com.kennycason.kumo.palette.ColorPalette;\nimport org.apache.commons.io.FileUtils;\n\nimport java.awt.*;\nimport java.io.File;\nimport java.io.IOException;\nimport java.nio.charset.StandardCharsets;\nimport java.util.Collection;\nimport java.util.Collections;\nimport java.util.List;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.function.Function;\nimport java.util.stream.Collectors;\nimport java.util.stream.Stream;\n\n/**\n * @since 2021-05-01\n */\npublic class JavMain {\n /**\n * Jav 文件夹目录\n */\n private static final String JAV_BASE_DIR = \"D:\\\\GITHUB\\\\Jav\";\n /**\n * CMD 输出命令\n * dir /b /s > D:\\GITHUB\\Jav\\jav20220824.txt\n */\n private static final String JAV_FILE_NAME = \"jav20220824\";\n private static final String JAV_INPUT_FILE = String.format(Locale.ENGLISH, \"%s/%s.txt\", JAV_BASE_DIR, JAV_FILE_NAME);\n private static final String JAV_OUTPUT_FILE = String.format(Locale.ENGLISH, \"%s/%s.sh\", JAV_BASE_DIR, JAV_FILE_NAME);\n\n public static void main(String[] args) {\n // 打印命令行\n doPrintCommand();\n // 分析\n// doAnalyse();\n // 查重\n// doFindRepeat();\n // 词云\n// doWordCloud();\n }\n\n /**\n * 打印 echo 番号 shell 脚本\n */\n private static void doPrintCommand() {\n try {\n File inputFile = new File(JAV_INPUT_FILE);\n List fanHao = FileUtils.readLines(inputFile, StandardCharsets.UTF_8.name());\n List cmdList = fanHao.stream()\n // 过滤掉 jav202xxxxx.txt\n .filter(name -> !name.endsWith(\".txt\") && name.contains(\".\"))\n .map(name -> name.substring(name.lastIndexOf(\"\\\\\") + 1))\n .map(name -> String.format(Locale.ENGLISH, \"echo > %s.txt\", name))\n .collect(Collectors.toList());\n File outputFile = new File(JAV_OUTPUT_FILE);\n FileUtils.writeLines(outputFile, cmdList);\n } catch (IOException e) {\n e.printStackTrace();\n }\n }\n\n private static Stream getJavStream() {\n File directory = new File(JAV_BASE_DIR);\n // 递归遍历 Jav txt 文件\n Collection avFiles = FileUtils.listFiles(directory, new String[]{\"txt\"}, true);\n return avFiles.stream().map(File::getName);\n }\n\n /**\n * 统计分析 Jav 目录索引\n */\n private static void doAnalyse() {\n long totals = getJavStream().map(name -> name.split(\"-\")[0]).count();\n System.out.printf(Locale.ENGLISH, \"Jav totals: %s, Ranks:%n\", totals);\n getJavStream()\n .map(name -> name.split(\"-\")[0])\n .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()))\n .entrySet().stream()\n// .filter(entry -> entry.getValue() >= 5)\n .collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue))\n .entrySet().stream()\n .sorted(Collections.reverseOrder(Map.Entry.comparingByValue()))\n .forEach(System.out::println);\n }\n\n /**\n * 查重\n */\n private static void doFindRepeat() {\n getJavStream()\n .map(name -> name.split(\"\\\\.\")[0])\n .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()))\n .entrySet().stream()\n .filter(entry -> entry.getValue() > 1)\n .forEach(System.out::println);\n }\n\n /**\n * 输出词云\n */\n private static void doWordCloud() {\n List avPrefix = getJavStream()\n .map(name -> name.split(\"-\")[0])\n// .filter(str -> !str.equals(\"SSNI\") && !str.equals(\"IPX\"))\n .collect(Collectors.toList());\n\n final FrequencyAnalyzer frequencyAnalyzer = new FrequencyAnalyzer();\n frequencyAnalyzer.setNormalizer(new UpperCaseNormalizer());\n final List wordFrequencies = frequencyAnalyzer.load(avPrefix);\n final Dimension dimension = new Dimension(600, 600);\n final WordCloud wordCloud = new WordCloud(dimension, CollisionMode.PIXEL_PERFECT);\n wordCloud.setPadding(0);\n wordCloud.setBackground(new CircleBackground(300));\n wordCloud.setColorPalette(new ColorPalette(new Color(0x4055F1), new Color(0x408DF1), new Color(0x40AAF1), new Color(0x40C5F1), new Color(0x40D3F1), new Color(0xFFFFFF)));\n wordCloud.setFontScalar(new LinearFontScalar(20, 160));\n wordCloud.build(wordFrequencies);\n wordCloud.writeToFile(JAV_BASE_DIR + \"/jav-wordCloud.png\");\n }\n}\n" }, { "path": "oys-boot-starter/src/main/resources/META-INF/additional-spring-configuration-metadata.json", "content": "{\n \"properties\": [\n {\n \"name\": \"oys.config.webdriver.chrome.driver.path\",\n \"type\": \"java.lang.String\",\n \"description\": \"webdriver.chrome.driver path（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.url.prefix\",\n \"type\": \"java.lang.String\",\n \"description\": \"图集岛-相册信息目录路径前缀.\"\n },\n {\n \"name\": \"oys.tujidao.img.url.prefix\",\n \"type\": \"java.lang.String\",\n \"description\": \"图集岛-图片资源目录路径前缀.\"\n },\n {\n \"name\": \"oys.tujidao.local.folder.prefix\",\n \"type\": \"java.lang.String\",\n \"description\": \"图集岛-本地存储路径前缀（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.preview.prefix\",\n \"type\": \"java.lang.String\",\n \"description\": \"图集岛-本地预览存储路径前缀（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.cover.prefix\",\n \"type\": \"java.lang.String\",\n \"description\": \"图集岛-本地封面存储路径前缀（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.cover.num.start\",\n \"type\": \"java.lang.Integer\",\n \"description\": \"图集岛-本地封面开始下标（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.cover.num.end\",\n \"type\": \"java.lang.Integer\",\n \"description\": \"图集岛-本地封面结束下标（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.preview.num.start\",\n \"type\": \"java.lang.Integer\",\n \"description\": \"图集岛-预下载开始下标（根据情况自定义）.\"\n },\n {\n \"name\": \"oys.tujidao.local.preview.num.end\",\n \"type\": \"java.lang.Integer\",\n \"description\": \"图集岛-预下载结束下标（根据情况自定义）.\"\n }\n ]\n}" }, { "path": "oys-boot-starter/src/main/resources/application-mysql.properties", "content": "datasource.url.1=jdbc:mysql://127.0.0.1:3306/dev?useUnicode=true&characterEncoding=UTF-8&serverTimezone=UTC\ndatasource.user.1=root\ndatasource.password.1=123456\ndatasource.jdbcDriverClass.1=com.mysql.cj.jdbc.Driver" }, { "path": "oys-boot-starter/src/main/resources/application.properties", "content": "spring.profiles.active=mysql\n# springboot 2.6.x\nspring.mvc.pathmatch.matching-strategy=ant_path_matcher\n# port\nserver.port=23333\n# spring datasource\nspring.datasource.url=${datasource.url.1}\nspring.datasource.username=${datasource.user.1}\nspring.datasource.password=${datasource.password.1}\nspring.datasource.driver-class-name=${datasource.jdbcDriverClass.1}\nspring.datasource.type=com.zaxxer.hikari.HikariDataSource\n# mybatis-plus\nmybatis-plus.type-aliases-package=com.devyy.*\n# self config\nserver.servlet.encoding.charset=utf-8\nserver.servlet.encoding.force=true\nserver.servlet.encoding.enabled=true\n# tujidao\noys.tujidao.url.prefix=https://www.tujidao03.com/u/?action=gengxin&page=\noys.tujidao.img.url.prefix=https://tjg.gzhuibei.com/a/1/\noys.tujidao.local.folder.prefix=D:/\\u56FE\\u96C6\\u5C9B\\u722C\\u866B\\uFF0850001-60000\\uFF09/\noys.tujidao.local.preview.prefix=D:/\\u56FE\\u96C6\\u5C9B\\u722C\\u866BPreview/\noys.tujidao.local.cover.prefix=D:/\\u56FE\\u96C6\\u5C9B\\u722C\\u866B\\u5C01\\u9762/\noys.tujidao.local.cover.num.start=58540\noys.tujidao.local.cover.num.end=59448\noys.tujidao.local.preview.num.start=58540\noys.tujidao.local.preview.num.end=59448\n# 56425-57050 625\n# 57051-57791 740\n# 57792-58539 747\n# 58540-59448 908\n# kafka\nspring.kafka.bootstrap-servers=localhost.ubuntu0.com:9092,localhost.ubuntu1.com:9092,localhost.ubuntu2.com:9092\nspring.kafka.consumer.auto-offset-reset=none\nspring.kafka.consumer.enable-auto-commit=false\n# redis\nspring.redis.cluster.nodes=localhost.ubuntu0.com:6379,localhost.ubuntu0.com:6380,localhost.ubuntu1.com:6379,localhost.ubuntu1.com:6380,localhost.ubuntu2.com:6379,localhost.ubuntu2.com:6380\nspring.redis.password=\n#spring.redis.host=local.ubuntu.com\n# mongodb\nspring.data.mongodb.uri=" }, { "path": "oys-codeforces/pom.xml", "content": "\n\n \n openyspider\n com.devyy\n ${revision}\n \n 4.0.0\n\n oys-codeforces\n\n \n 17\n 17\n \n\n" }, { "path": "oys-codeforces/src/main/java/com/devyy/oys/codeforces/CfController.java", "content": "package com.devyy.oys.codeforces;\n\nimport com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;\n\nimport com.devyy.oys.codeforces.dao.CfMapper;\n\nimport io.swagger.annotations.Api;\nimport io.swagger.annotations.ApiOperation;\n\nimport lombok.extern.slf4j.Slf4j;\n\nimport org.openqa.selenium.By;\nimport org.openqa.selenium.WebDriver;\nimport org.openqa.selenium.WebElement;\nimport org.openqa.selenium.chrome.ChromeDriver;\n\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.beans.factory.annotation.Value;\nimport org.springframework.data.redis.core.RedisTemplate;\nimport org.springframework.data.redis.core.StringRedisTemplate;\nimport org.springframework.kafka.core.KafkaTemplate;\nimport org.springframework.web.bind.annotation.PostMapping;\nimport org.springframework.web.bind.annotation.RequestMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\nimport java.time.Duration;\nimport java.util.List;\n\n/**\n * Controller + ServiceImpl\n *\n * @since 2021-01-24\n */\n@Slf4j\n@RestController\n@RequestMapping(\"/codeforces\")\n@Api(tags = \"Codeforces 爬虫\")\npublic class CfController {\n @Value(\"${oys.codeforces.handleOrEmail:123}\")\n private String handleOrEmail;\n\n @Value(\"${oys.codeforces.password:123}\")\n private String password;\n\n @Autowired\n private CfMapper cfMapper;\n\n @ApiOperation(value = \"获取片段\")\n @PostMapping(\"/step1\")\n public String doGetFragment() throws InterruptedException {\n System.setProperty(\"webdriver.chrome.driver\", \"D:\\\\GITHUB\\\\LTS\\\\codeforces-spider\\\\chromedriver.exe\");\n WebDriver webDriver = new ChromeDriver();\n webDriver.manage().timeouts().implicitlyWait(Duration.ofSeconds(30));\n\n // login\n webDriver.get(\"https://codeforces.com/enter?back=%2F\");\n webDriver.findElement(By.id(\"handleOrEmail\")).sendKeys(handleOrEmail);\n webDriver.findElement(By.id(\"password\")).sendKeys(password);\n webDriver.findElement(By.id(\"remember\")).click();\n webDriver.findElement(By.className(\"submit\")).click();\n Thread.sleep(5000);\n\n List cfDOList = cfMapper.selectList(new QueryWrapper().select().isNull(\"FRAGMENT\"));\n log.info(\"==>cfDOList size={}\", cfDOList.size());\n for (CfDO cfDO : cfDOList) {\n String submissionId = cfDO.getSubmissionId();\n\n String url = \"https://codeforces.com/contest/1593/submission/\" + submissionId;\n log.info(\"==>url={}\", url);\n try {\n webDriver.get(url);\n webDriver.findElement(By.className(\"click-to-view-tests\")).click();\n Thread.sleep(3000);\n\n List outputs = webDriver.findElements(By.className(\"output\"));\n int size = outputs.size();\n if (outputs.size() < 7) {\n Thread.sleep(3000);\n outputs = webDriver.findElements(By.className(\"output\"));\n size = outputs.size();\n }\n\n WebElement outputs8 = outputs.get(size - 1);\n String fragment = outputs8.getText().substring(0, 500);\n cfDO.setFragment(fragment);\n log.info(fragment);\n\n cfMapper.updateById(cfDO);\n log.info(\"<==success submissionId={}\", submissionId);\n } catch (Exception e) {\n log.warn(\"<==failed submissionId={}\", submissionId);\n }\n }\n webDriver.close();\n\n return \"success\";\n }\n\n @ApiOperation(value = \"合并片段\")\n @PostMapping(\"/step2\")\n public String doMergeFragment() {\n List cfDOList = cfMapper.selectList(new QueryWrapper().select().orderByAsc(\"NO\"));\n log.info(\"==>cfDOList size={}\", cfDOList.size());\n StringBuilder stringBuilder = new StringBuilder();\n for (CfDO cfDO : cfDOList) {\n String fragment = cfDO.getFragment();\n stringBuilder.append(fragment);\n }\n log.info(\"==>doMergeFragment={}\", stringBuilder);\n\n return \"success\";\n }\n\n @Autowired\n private KafkaTemplate kafkaTemplate;\n\n\n @ApiOperation(value = \"Kafka\")\n @PostMapping(\"/step3\")\n public String testKafka() {\n kafkaTemplate.send(\"T_ubuntu\", \"test\");\n log.info(\"==>kafkaTemplate.send success\");\n return \"success\";\n }\n\n @Autowired\n private StringRedisTemplate stringRedisTemplate;\n\n @ApiOperation(value = \"redis\")\n @PostMapping(\"/step4\")\n public String testRedis() {\n stringRedisTemplate.boundValueOps(\"ubuntu:redis:test\").set(\"hello\");\n log.info(\"==>redis set success\");\n\n String value = stringRedisTemplate.boundValueOps(\"ubuntu:redis:test\").get();\n log.info(\"==>redis get value={}\", value);\n return \"success\";\n }\n}\n" }, { "path": "oys-codeforces/src/main/java/com/devyy/oys/codeforces/CfDO.java", "content": "package com.devyy.oys.codeforces;\n\nimport com.baomidou.mybatisplus.annotation.TableField;\nimport com.baomidou.mybatisplus.annotation.TableId;\nimport com.baomidou.mybatisplus.annotation.TableName;\n\nimport lombok.Data;\n\n/**\n * Entity 实体类\n *\n * @since 2021-01-24\n */\n@Data\n@TableName(\"oys_codeforces\")\npublic class CfDO {\n @TableId(\"SUBMISSION_ID\")\n private String submissionId;\n\n @TableField(\"NO\")\n private Long no;\n\n @TableField(\"FRAGMENT\")\n private String fragment;\n}\n" }, { "path": "oys-codeforces/src/main/java/com/devyy/oys/codeforces/dao/CfMapper.java", "content": "package com.devyy.oys.codeforces.dao;\n\nimport com.baomidou.mybatisplus.core.mapper.BaseMapper;\n\nimport com.devyy.oys.codeforces.CfDO;\n\nimport org.springframework.stereotype.Repository;\n\n/**\n * Mybatis-Plus Mapper\n *\n * @since 2021-01-24\n */\n@Repository\npublic interface CfMapper extends BaseMapper {\n}\n" }, { "path": "oys-core/pom.xml", "content": "\n\n \n openyspider\n com.devyy\n ${revision}\n \n 4.0.0\n\n oys-core\n\n\n" }, { "path": "oys-core/src/main/java/com/devyy/oys/srarter/core/enums/StateTypeEnum.java", "content": "package com.devyy.oys.srarter.core.enums;\n\nimport lombok.Getter;\n\n/**\n * 资源状态\n *\n * @since 2019-12-01\n */\n@Getter\npublic enum StateTypeEnum {\n /**\n * 黑名单\n */\n BLACKLIST(-1, \"黑名单\"),\n /**\n * 白名单\n */\n WHITELIST(0, \"白名单\"),\n\n //\n\n /**\n * 下载完成\n */\n DONE(102, \"下载完成\"),\n /**\n * 下载中\n */\n DOWNLOADING(101, \"下载中\"),\n /**\n * 未开始/待重试\n */\n STARTED(100, \"未开始/待重试\"),\n\n //\n\n /**\n * 已解析--album专用\n */\n ANALYSIS(200, \"已解析\"),\n /**\n * 未达预期\n */\n EXCEPTION(500, \"未达预期\"),\n /**\n * 找不到资源\n */\n NOTFOUND(404, \"找不到资源\"),\n\n ;\n\n private final int seq;\n private final String desc;\n\n StateTypeEnum(int seq, String desc) {\n this.seq = seq;\n this.desc = desc;\n }\n}\n" }, { "path": "oys-core/src/main/java/com/devyy/oys/srarter/core/util/SpiderUtil.java", "content": "package com.devyy.oys.srarter.core.util;\n\nimport lombok.extern.slf4j.Slf4j;\n\nimport org.apache.commons.io.FileUtils;\n\nimport java.io.File;\nimport java.io.FileNotFoundException;\nimport java.io.IOException;\nimport java.io.InputStream;\nimport java.net.URL;\nimport java.net.URLConnection;\n\n/**\n * 爬虫工具类\n *\n * @since 2019-12-01\n */\n@Slf4j\npublic class SpiderUtil {\n private static boolean ioDownload2(String onlineUrl, String localUrl, int timeout) {\n try {\n URL source = new URL(onlineUrl);\n File destination = new File(localUrl);\n// FileUtils.copyURLToFile(new URL(onlineUrl), new File(localUrl), timeout, timeout);\n\n final URLConnection connection = source.openConnection();\n connection.setRequestProperty(\"Referer\", \"https://www.tujidao.com/\");\n connection.setConnectTimeout(timeout);\n connection.setReadTimeout(timeout);\n try (final InputStream stream = connection.getInputStream()) {\n FileUtils.copyInputStreamToFile(stream, destination);\n }\n\n log.info(\"==>io下载成功 localUrl={}\", localUrl);\n return true;\n } catch (Exception e) {\n if (!(e instanceof FileNotFoundException)) {\n log.warn(\"FileUtils.copyURLToFile failed={} e.message={}\", onlineUrl, e.getMessage());\n }\n return false;\n }\n }\n\n /**\n * 带重试次数\n *\n * @param onlineUrl onlineUrl\n * @param localUrl localUrl\n * @param times 重试次数\n * @return success\n */\n public static boolean ioDownload2Times(String onlineUrl, String localUrl, int times) {\n if (times < 0) {\n return false;\n } else {\n // timeout 递增\n int timeout = 10000;\n if (times < 1) {\n timeout = 30000;\n } else if (times < 2) {\n timeout = 20000;\n }\n boolean success = ioDownload2(onlineUrl, localUrl, timeout);\n if (!success) {\n return ioDownload2Times(onlineUrl, localUrl, times - 1);\n }\n }\n return true;\n }\n\n\n /**\n * 图片移动\n *\n * @param oldPath 原始路径\n * @param newPath 目标路径\n */\n public static void fileMove(String oldPath, String newPath) {\n File oldName = new File(oldPath);\n File newName = new File(newPath);\n oldName.renameTo(newName);\n }\n\n /**\n * 图片移动\n *\n * @param oldPath 原始路径\n * @param newPath 目标路径\n */\n public static void fileCopy(String oldPath, String newPath) {\n File oldName = new File(oldPath);\n File newName = new File(newPath);\n try {\n FileUtils.copyFile(oldName, newName);\n log.info(\"==>fileCopy success oldPath={} newPath={}\", oldPath, newPath);\n } catch (IOException e) {\n log.warn(\"==>fileCopy failed oldPath={} newPath={}\", oldPath, newPath);\n }\n }\n}" }, { "path": "oys-tujidao/pom.xml", "content": "\n\n \n openyspider\n com.devyy\n ${revision}\n \n 4.0.0\n\n oys-tujidao\n\n \n \n com.devyy\n oys-core\n ${revision}\n \n \n" }, { "path": "oys-tujidao/src/main/java/com/devyy/oys/tujidao/TuJiDaoAlbumDO.java", "content": "package com.devyy.oys.tujidao;\n\nimport com.baomidou.mybatisplus.annotation.IdType;\nimport com.baomidou.mybatisplus.annotation.TableField;\nimport com.baomidou.mybatisplus.annotation.TableId;\nimport com.baomidou.mybatisplus.annotation.TableName;\n\nimport lombok.Data;\n\n/**\n * 实体类\n *\n * @since 2019-12-01\n */\n@Data\n@TableName(\"oys_tujidao_album_t\")\npublic class TuJiDaoAlbumDO {\n /**\n * 自增 ID\n */\n @TableId(type = IdType.AUTO)\n private Long id;\n\n /**\n * 状态\n *\n * @see com.devyy.oys.srarter.core.enums.StateTypeEnum\n */\n private Integer state;\n\n /**\n * 总数\n */\n private Integer total;\n\n /**\n * 相册名\n */\n @TableField(\"album_name\")\n private String albumName;\n\n /**\n * 相册 ID\n */\n @TableField(\"album_id\")\n private Integer albumId;\n}\n" }, { "path": "oys-tujidao/src/main/java/com/devyy/oys/tujidao/TuJiDaoController.java", "content": "package com.devyy.oys.tujidao;\n\nimport io.swagger.annotations.Api;\nimport io.swagger.annotations.ApiOperation;\n\nimport lombok.extern.slf4j.Slf4j;\n\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.web.bind.annotation.PostMapping;\nimport org.springframework.web.bind.annotation.RequestMapping;\nimport org.springframework.web.bind.annotation.RestController;\n\n/**\n * REST 接口层\n *\n * @since 2019-12-01\n */\n@Slf4j\n@RestController\n@RequestMapping(\"/tujidao\")\n@Api(tags = \"图集岛爬虫\")\npublic class TuJiDaoController {\n @Autowired\n private TuJiDaoService tujidaoService;\n\n /**\n * 预下载\n *\n * @return \"success\"\n */\n @ApiOperation(value = \"预下载\")\n @PostMapping(\"/step1\")\n public String step1() {\n return tujidaoService.doPreDownload();\n }\n\n /**\n * 生成封面\n *\n * @return \"success\"\n */\n @ApiOperation(value = \"生成封面\")\n @PostMapping(\"/step2\")\n public String step2() {\n return tujidaoService.doGenerateCover();\n }\n\n /**\n * 同步更新记录\n *\n * @return \"success\"\n */\n @ApiOperation(value = \"同步更新记录\")\n @PostMapping(\"/step3\")\n public String step3() {\n return tujidaoService.doSyncRecords();\n }\n\n /**\n * 本地迁移\n *\n * @return \"success\"\n */\n @ApiOperation(value = \"本地迁移\")\n @PostMapping(\"/step4\")\n public String step4() {\n return tujidaoService.doLocalMigration();\n }\n}\n" }, { "path": "oys-tujidao/src/main/java/com/devyy/oys/tujidao/TuJiDaoService.java", "content": "package com.devyy.oys.tujidao;\n\n/**\n * Service 层\n *\n * @since 2019-12-01\n */\npublic interface TuJiDaoService {\n /**\n * 预下载\n *\n * @return success\n * @since 2019-12-08\n */\n String doPreDownload();\n\n /**\n * 生成封面\n *\n * @return success\n * @since 2019-12-08\n */\n String doGenerateCover();\n\n /**\n * 同步更新记录\n *\n * @return success\n */\n String doSyncRecords();\n\n /**\n * 本地迁移\n *\n * @return success\n * @since 2019-12-08\n */\n String doLocalMigration();\n}\n" }, { "path": "oys-tujidao/src/main/java/com/devyy/oys/tujidao/TuJiDaoServiceImpl.java", "content": "package com.devyy.oys.tujidao;\n\nimport com.devyy.oys.srarter.core.enums.StateTypeEnum;\nimport com.devyy.oys.srarter.core.util.SpiderUtil;\nimport com.devyy.oys.tujidao.dao.TuJiDaoAlbumMapper;\n\nimport lombok.extern.slf4j.Slf4j;\n\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.io.filefilter.DirectoryFileFilter;\n\nimport org.jsoup.Jsoup;\nimport org.jsoup.nodes.Document;\n\nimport org.jsoup.nodes.Element;\nimport org.springframework.beans.factory.annotation.Autowired;\nimport org.springframework.beans.factory.annotation.Value;\nimport org.springframework.stereotype.Service;\nimport org.springframework.util.CollectionUtils;\n\nimport javax.net.ssl.HostnameVerifier;\nimport javax.net.ssl.HttpsURLConnection;\n\nimport java.io.File;\nimport java.io.FilenameFilter;\nimport java.io.IOException;\nimport java.text.MessageFormat;\nimport java.util.Collection;\nimport java.util.HashMap;\nimport java.util.HashSet;\nimport java.util.Locale;\nimport java.util.Map;\nimport java.util.Objects;\nimport java.util.Set;\nimport java.util.concurrent.ExecutorService;\nimport java.util.concurrent.Executors;\nimport java.util.regex.Pattern;\n\n/**\n * Service 层实现\n *\n * @since 2019-12-01\n */\n@Slf4j\n@Service\npublic class TuJiDaoServiceImpl implements TuJiDaoService {\n /**\n * 源站网页 url\n */\n @Value(\"${oys.tujidao.url.prefix}\")\n private String tjdWebUrlPrefix;\n\n /**\n * 源站图片 cdn\n */\n @Value(\"${oys.tujidao.img.url.prefix}\")\n private String tjdImgUrlPrefix;\n\n /**\n * 本地归档目录\n */\n @Value(\"${oys.tujidao.local.folder.prefix}\")\n private String tjdLocalFolderPrefix;\n\n /**\n * 本地 Preview 目录\n */\n @Value(\"${oys.tujidao.local.preview.prefix}\")\n private String tjdLocalPreviewPrefix;\n\n /**\n * 本地封面目录\n */\n @Value(\"${oys.tujidao.local.cover.prefix}\")\n private String tjdLocalCoverPrefix;\n\n /**\n * 封面范围 [start, end]\n */\n @Value(\"${oys.tujidao.local.cover.num.start}\")\n private Integer tjdCoverStart;\n @Value(\"${oys.tujidao.local.cover.num.end}\")\n private Integer tjdLocalCoverEnd;\n\n /**\n * Preview 范围 [start, end]\n */\n @Value(\"${oys.tujidao.local.preview.num.start}\")\n private Integer tjdPreviewStart;\n @Value(\"${oys.tujidao.local.preview.num.end}\")\n private Integer tjdPreviewEnd;\n\n @Autowired\n private TuJiDaoAlbumMapper tuJiDaoAlbumMapper;\n\n @Override\n public String doPreDownload() {\n ExecutorService executors = Executors.newFixedThreadPool(8);\n for (int i = tjdPreviewStart; i <= tjdPreviewEnd; i++) {\n final int finalI = i;\n executors.submit(() -> downloadByAlbumId(String.valueOf(finalI)));\n }\n return \"success\";\n }\n\n static Map map = new HashMap<>();\n\n static {\n map.put(119, 50);\n map.put(1562, 86);\n map.put(293, 129);\n map.put(313, 100);\n map.put(331, 116);\n map.put(338, 80);\n map.put(37, 80);\n map.put(3849, 100);\n map.put(543, 16);\n map.put(714, 61);\n map.put(716, 68);\n map.put(727, 69);\n\n map.put(16471, 24);\n map.put(16681, 46);\n map.put(16968, 75);\n map.put(17058, 100);\n\n map.put(32793, 158);\n map.put(34456, 44);\n map.put(35381, 33);\n map.put(35527, 150);\n map.put(35534, 58);\n map.put(35576, 21);\n map.put(35697, 39);\n map.put(35854, 25);\n map.put(36655, 60);\n map.put(36656, 52);\n map.put(36658, 68);\n map.put(36659, 73);\n map.put(36697, 76);\n map.put(36698, 52);\n map.put(36700, 80);\n map.put(36701, 92);\n map.put(37180, 45);\n map.put(38013, 41);\n map.put(38313, 42);\n map.put(38432, 35);\n map.put(39740, 48);\n\n map.put(44264, 17);\n map.put(45681, 42);\n map.put(45697, 19);\n map.put(46207, 16);\n map.put(48080, 62);\n }\n\n private void downloadByAlbumId(String albumId) {\n String localFolder = tjdLocalPreviewPrefix + albumId + \"/\";\n // 若文件夹路径不存在，则新建\n File file = new File(localFolder);\n if (!file.exists()) {\n if (!file.mkdirs()) {\n log.error(\"==>localFolder={} 创建文件路径失败\", localFolder);\n return;\n }\n }\n // 顺序扫描直到 404\n for (int i = 0; ; i++) {\n String onlinePath = String.format(Locale.CHINESE, \"%s%s/%d.jpg\", tjdImgUrlPrefix, albumId, i);\n String localPath = String.format(Locale.CHINESE, \"%s%d.jpg\", localFolder, i);\n\n // 幂等，若当前文件未下载，则进行下载\n File file2 = new File(localPath);\n if (!file2.exists()) {\n if (!SpiderUtil.ioDownload2Times(onlinePath, localPath, 3)) {\n break;\n }\n }\n }\n }\n\n @Override\n public String doGenerateCover() {\n // 若文件夹路径不存在，则新建\n File file = new File(tjdLocalCoverPrefix);\n if (!file.exists()) {\n if (!file.mkdirs()) {\n log.error(\"==>localFolder={} 创建文件路径失败\", tjdLocalCoverPrefix);\n return \"error\";\n }\n }\n\n // 选取 1.jpg 作为封面\n for (int i = tjdCoverStart; i <= tjdLocalCoverEnd; i++) {\n String onlinePath = String.format(Locale.CHINESE, \"%s%d/1.jpg\", tjdLocalPreviewPrefix, i);\n String localPath = String.format(Locale.CHINESE, \"%s%d-1.jpg\", tjdLocalCoverPrefix, i);\n\n // 幂等，若当前文件未下载，则进行下载\n File file2 = new File(localPath);\n if (!file2.exists()) {\n SpiderUtil.fileCopy(onlinePath, localPath);\n }\n }\n return \"success\";\n }\n\n @Override\n public String doSyncRecords() {\n // 解决 cookie 和 https 问题\n Map cookiesMap = buildCookies();\n try {\n trustAllHttpsCertificates();\n } catch (Exception e) {\n log.error(\"==>TujidaoService#doScanAlbums failed e=\", e);\n }\n HttpsURLConnection.setDefaultHostnameVerifier(hv);\n\n Document document = null;\n // 55625-56283 659\n // 56284-56939 656\n // 56940-57639 700\n // 57640-58330 690\n for (int i = 1; i <= 50; i++) {\n try {\n document = Jsoup.connect(tjdWebUrlPrefix + i).cookies(cookiesMap).get();\n } catch (IOException e) {\n log.error(\"==>url={} e={}\", tjdWebUrlPrefix + i, e.getMessage());\n }\n if (Objects.isNull(document)) {\n continue;\n }\n Element heziElement = document.getElementsByClass(\"hezi\").first();\n if (Objects.isNull(heziElement)) {\n continue;\n }\n\n heziElement.getElementsByTag(\"li\").forEach(liElement -> {\n Element biaotiElement = liElement.getElementsByClass(\"biaoti\").first();\n Element shuliangElement = liElement.getElementsByClass(\"shuliang\").first();\n\n if (biaotiElement == null || shuliangElement == null) {\n return;\n }\n\n int albumId = Integer.parseInt(liElement.attr(\"id\"));\n int total = Integer.parseInt(shuliangElement.text().replace(\"P\", \"\"));\n String albumName = MessageFormat.format(\"{0}-[{1}P] {2}\",\n String.valueOf(albumId), total, rmIllegalName(biaotiElement.text()));\n\n TuJiDaoAlbumDO tjdDO = new TuJiDaoAlbumDO();\n tjdDO.setAlbumId(albumId);\n tjdDO.setAlbumName(albumName);\n tjdDO.setTotal(total);\n tjdDO.setState(StateTypeEnum.STARTED.getSeq());\n\n // 幂等，保证记录数唯一\n Map queryMap = new HashMap<>(1);\n queryMap.put(\"album_id\", tjdDO.getAlbumId());\n if (CollectionUtils.isEmpty(tuJiDaoAlbumMapper.selectByMap(queryMap))) {\n tuJiDaoAlbumMapper.insert(tjdDO);\n log.info(\"album_id={} 同步成功,total={},title={}\",\n tjdDO.getAlbumId(), tjdDO.getTotal(), tjdDO.getAlbumName());\n } else {\n log.warn(\"album_id={} 已存在\", tjdDO.getAlbumId());\n }\n });\n }\n return \"success\";\n }\n\n @Override\n public String doLocalMigration() {\n Map queryMap = new HashMap<>(1);\n queryMap.put(\"state\", StateTypeEnum.STARTED.getSeq());\n tuJiDaoAlbumMapper.selectByMap(queryMap).forEach(albumDO -> {\n int total = albumDO.getTotal();\n int albumId = albumDO.getAlbumId();\n String albumName = albumDO.getAlbumName();\n String localFolder = tjdLocalFolderPrefix + albumName;\n\n // 若文件夹路径不存在，则新建\n File file = new File(localFolder);\n if (!file.exists()) {\n if (!file.mkdirs()) {\n log.error(\"==>localFolder={} 创建文件路径失败\", localFolder);\n return;\n }\n }\n\n // 2022-01-16 针对源站已经隐去 图片数 信息 fix\n if (total == -1) {\n String previewDir = tjdLocalPreviewPrefix + albumId;\n total = FileUtils.listFiles(new File(previewDir), new String[]{\"jpg\"}, false).size() - 1;\n }\n\n for (int i = 0; i <= total; i++) {\n String onlinePath = String.format(Locale.CHINESE, \"%s%s/%d.jpg\", tjdLocalPreviewPrefix, albumId, i);\n String localPath = String.format(Locale.CHINESE, \"%s/%d.jpg\", localFolder, i);\n\n // 幂等，若当前文件未下载，则进行下载\n File file2 = new File(localPath);\n if (!file2.exists()) {\n SpiderUtil.fileMove(onlinePath, localPath);\n }\n }\n albumDO.setState(StateTypeEnum.ANALYSIS.getSeq());\n tuJiDaoAlbumMapper.updateById(albumDO);\n log.info(\"==>albumId={} 相册已本地迁移完成\", albumId);\n });\n return \"success\";\n }\n\n private Map buildCookies() {\n Map cookiesMap = new HashMap<>();\n cookiesMap.put(\"TujidaoService%5Fid\", \"411999177-1588406642-%7C1600614708\");\n cookiesMap.put(\"PHPSESSID\", \"uulfluqmbunauqqjnj4mf8i8dr\");\n cookiesMap.put(\"UM_distinctid\", \"171d493d83c1c-074736cab8c8d9-670103b-1fa400-171d493d83d151\");\n cookiesMap.put(\"atpsida\", \"7a72b6965366155670da7487_1600614730_4\");\n cookiesMap.put(\"cna\", \"MkUPF0HprU0CAXWIT4ixum+C\");\n cookiesMap.put(\"leixing\", \"0\");\n cookiesMap.put(\"name\", \"rxgirlz\");\n cookiesMap.put(\"sca\", \"5cafd77a\");\n cookiesMap.put(\"uid\", \"157108\");\n return cookiesMap;\n }\n\n private final HostnameVerifier hv = (s, sslSession) -> true;\n\n private void trustAllHttpsCertificates() throws Exception {\n javax.net.ssl.TrustManager[] trustAllCerts = new javax.net.ssl.TrustManager[1];\n javax.net.ssl.TrustManager tm = new miTM();\n trustAllCerts[0] = tm;\n javax.net.ssl.SSLContext sc = javax.net.ssl.SSLContext.getInstance(\"SSL\");\n sc.init(null, trustAllCerts, null);\n javax.net.ssl.HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());\n }\n\n private static class miTM implements javax.net.ssl.TrustManager, javax.net.ssl.X509TrustManager {\n public java.security.cert.X509Certificate[] getAcceptedIssuers() {\n return null;\n }\n\n public boolean isServerTrusted(java.security.cert.X509Certificate[] certs) {\n return true;\n }\n\n public boolean isClientTrusted(java.security.cert.X509Certificate[] certs) {\n return true;\n }\n\n public void checkServerTrusted(java.security.cert.X509Certificate[] certs, String authType)\n throws java.security.cert.CertificateException {\n return;\n }\n\n public void checkClientTrusted(java.security.cert.X509Certificate[] certs, String authType)\n throws java.security.cert.CertificateException {\n return;\n }\n\n }\n\n /**\n * 文件不合法名正则\n */\n private static final Pattern FILE_PATTERN = Pattern.compile(\"[\\\\\\\\/:*?\\\"<>|]\");\n\n /**\n * 去除不合法文件名\n */\n private String rmIllegalName(String s) {\n return FILE_PATTERN.matcher(s).replaceAll(\"\");\n }\n\n static String[] jpgs = {\n \"D:/图集岛爬虫Preview/49322/20.jpg\",\n \"D:/图集岛爬虫Preview/49323/13.jpg\",\n \"D:/图集岛爬虫Preview/49324/34.jpg\",\n \"D:/图集岛爬虫Preview/49331/49.jpg\",\n \"D:/图集岛爬虫Preview/49332/22.jpg\",\n \"D:/图集岛爬虫Preview/49340/19.jpg\",\n \"D:/图集岛爬虫Preview/49340/83.jpg\",\n \"D:/图集岛爬虫Preview/49349/38.jpg\",\n \"D:/图集岛爬虫Preview/49355/56.jpg\",\n \"D:/图集岛爬虫Preview/49357/83.jpg\",\n \"D:/图集岛爬虫Preview/49360/42.jpg\",\n \"D:/图集岛爬虫Preview/49366/1.jpg\",\n \"D:/图集岛爬虫Preview/49367/13.jpg\",\n \"D:/图集岛爬虫Preview/49369/20.jpg\",\n \"D:/图集岛爬虫Preview/49372/1.jpg\",\n \"D:/图集岛爬虫Preview/49374/61.jpg\",\n \"D:/图集岛爬虫Preview/49380/27.jpg\",\n \"D:/图集岛爬虫Preview/49382/23.jpg\",\n \"D:/图集岛爬虫Preview/49386/28.jpg\",\n \"D:/图集岛爬虫Preview/49387/20.jpg\",\n \"D:/图集岛爬虫Preview/49387/21.jpg\",\n \"D:/图集岛爬虫Preview/49388/88.jpg\",\n \"D:/图集岛爬虫Preview/49390/8.jpg\",\n \"D:/图集岛爬虫Preview/49392/23.jpg\",\n \"D:/图集岛爬虫Preview/49393/26.jpg\",\n \"D:/图集岛爬虫Preview/49397/82.jpg\",\n \"D:/图集岛爬虫Preview/49408/8.jpg\",\n \"D:/图集岛爬虫Preview/49410/91.jpg\",\n \"D:/图集岛爬虫Preview/49412/30.jpg\",\n \"D:/图集岛爬虫Preview/49414/21.jpg\",\n \"D:/图集岛爬虫Preview/49418/5.jpg\",\n \"D:/图集岛爬虫Preview/49420/8.jpg\",\n \"D:/图集岛爬虫Preview/49421/22.jpg\",\n \"D:/图集岛爬虫Preview/49421/42.jpg\",\n \"D:/图集岛爬虫Preview/49424/9.jpg\",\n \"D:/图集岛爬虫Preview/49425/19.jpg\",\n \"D:/图集岛爬虫Preview/49426/68.jpg\",\n \"D:/图集岛爬虫Preview/49426/82.jpg\",\n \"D:/图集岛爬虫Preview/49432/28.jpg\",\n \"D:/图集岛爬虫Preview/49501/34.jpg\",\n \"D:/图集岛爬虫Preview/49501/52.jpg\",\n \"D:/图集岛爬虫Preview/49503/19.jpg\",\n \"D:/图集岛爬虫Preview/49503/40.jpg\",\n \"D:/图集岛爬虫Preview/49508/14.jpg\",\n \"D:/图集岛爬虫Preview/49512/33.jpg\",\n \"D:/图集岛爬虫Preview/49513/1.jpg\",\n \"D:/图集岛爬虫Preview/49521/34.jpg\",\n \"D:/图集岛爬虫Preview/49522/31.jpg\",\n \"D:/图集岛爬虫Preview/49537/26.jpg\",\n \"D:/图集岛爬虫Preview/49544/1.jpg\",\n \"D:/图集岛爬虫Preview/49544/11.jpg\",\n \"D:/图集岛爬虫Preview/49544/4.jpg\",\n \"D:/图集岛爬虫Preview/49547/43.jpg\",\n \"D:/图集岛爬虫Preview/49550/5.jpg\",\n \"D:/图集岛爬虫Preview/49551/16.jpg\",\n \"D:/图集岛爬虫Preview/49551/65.jpg\",\n \"D:/图集岛爬虫Preview/49559/6.jpg\",\n \"D:/图集岛爬虫Preview/49562/75.jpg\",\n \"D:/图集岛爬虫Preview/49563/42.jpg\",\n \"D:/图集岛爬虫Preview/49563/55.jpg\",\n \"D:/图集岛爬虫Preview/49563/9.jpg\",\n \"D:/图集岛爬虫Preview/49565/17.jpg\",\n \"D:/图集岛爬虫Preview/49574/1.jpg\",\n \"D:/图集岛爬虫Preview/49574/4.jpg\",\n \"D:/图集岛爬虫Preview/49578/12.jpg\",\n \"D:/图集岛爬虫Preview/49578/15.jpg\",\n \"D:/图集岛爬虫Preview/49578/27.jpg\",\n \"D:/图集岛爬虫Preview/49583/28.jpg\",\n \"D:/图集岛爬虫Preview/49591/64.jpg\",\n \"D:/图集岛爬虫Preview/49598/10.jpg\",\n \"D:/图集岛爬虫Preview/49599/38.jpg\",\n \"D:/图集岛爬虫Preview/49601/42.jpg\",\n \"D:/图集岛爬虫Preview/49602/64.jpg\",\n \"D:/图集岛爬虫Preview/49602/75.jpg\",\n \"D:/图集岛爬虫Preview/49605/31.jpg\",\n \"D:/图集岛爬虫Preview/49606/40.jpg\",\n \"D:/图集岛爬虫Preview/49608/15.jpg\",\n \"D:/图集岛爬虫Preview/49611/27.jpg\",\n \"D:/图集岛爬虫Preview/49611/45.jpg\",\n \"D:/图集岛爬虫Preview/49614/36.jpg\",\n \"D:/图集岛爬虫Preview/49627/28.jpg\",\n \"D:/图集岛爬虫Preview/49627/7.jpg\",\n \"D:/图集岛爬虫Preview/49628/51.jpg\",\n \"D:/图集岛爬虫Preview/49628/76.jpg\",\n \"D:/图集岛爬虫Preview/49629/47.jpg\",\n \"D:/图集岛爬虫Preview/49633/11.jpg\",\n \"D:/图集岛爬虫Preview/49637/5.jpg\",\n \"D:/图集岛爬虫Preview/49639/24.jpg\",\n \"D:/图集岛爬虫Preview/49640/12.jpg\",\n \"D:/图集岛爬虫Preview/49645/21.jpg\",\n \"D:/图集岛爬虫Preview/49646/41.jpg\",\n \"D:/图集岛爬虫Preview/49653/12.jpg\",\n \"D:/图集岛爬虫Preview/49658/11.jpg\",\n \"D:/图集岛爬虫Preview/49665/2.jpg\",\n \"D:/图集岛爬虫Preview/49666/72.jpg\",\n \"D:/图集岛爬虫Preview/49671/4.jpg\",\n \"D:/图集岛爬虫Preview/49672/22.jpg\",\n \"D:/图集岛爬虫Preview/49677/25.jpg\",\n \"D:/图集岛爬虫Preview/49677/27.jpg\",\n \"D:/图集岛爬虫Preview/49691/23.jpg\",\n \"D:/图集岛爬虫Preview/49702/1.jpg\",\n \"D:/图集岛爬虫Preview/49704/9.jpg\",\n \"D:/图集岛爬虫Preview/49706/48.jpg\",\n \"D:/图集岛爬虫Preview/49713/5.jpg\",\n \"D:/图集岛爬虫Preview/49716/10.jpg\",\n \"D:/图集岛爬虫Preview/49721/18.jpg\",\n \"D:/图集岛爬虫Preview/49722/0.jpg\",\n \"D:/图集岛爬虫Preview/49722/28.jpg\",\n \"D:/图集岛爬虫Preview/49722/3.jpg\",\n \"D:/图集岛爬虫Preview/49722/31.jpg\",\n \"D:/图集岛爬虫Preview/49724/8.jpg\",\n \"D:/图集岛爬虫Preview/49725/2.jpg\",\n \"D:/图集岛爬虫Preview/49742/27.jpg\",\n \"D:/图集岛爬虫Preview/49750/5.jpg\",\n \"D:/图集岛爬虫Preview/49755/1.jpg\",\n \"D:/图集岛爬虫Preview/49763/52.jpg\",\n \"D:/图集岛爬虫Preview/49763/63.jpg\",\n \"D:/图集岛爬虫Preview/49770/38.jpg\",\n \"D:/图集岛爬虫Preview/49771/49.jpg\",\n \"D:/图集岛爬虫Preview/49774/38.jpg\",\n \"D:/图集岛爬虫Preview/49778/41.jpg\",\n \"D:/图集岛爬虫Preview/49779/11.jpg\",\n \"D:/图集岛爬虫Preview/49779/3.jpg\",\n \"D:/图集岛爬虫Preview/49780/21.jpg\",\n \"D:/图集岛爬虫Preview/49784/46.jpg\",\n \"D:/图集岛爬虫Preview/49789/25.jpg\",\n \"D:/图集岛爬虫Preview/49789/36.jpg\",\n \"D:/图集岛爬虫Preview/49791/47.jpg\",\n \"D:/图集岛爬虫Preview/49792/13.jpg\",\n \"D:/图集岛爬虫Preview/49804/15.jpg\",\n \"D:/图集岛爬虫Preview/49805/14.jpg\",\n \"D:/图集岛爬虫Preview/49805/24.jpg\",\n \"D:/图集岛爬虫Preview/49805/4.jpg\",\n \"D:/图集岛爬虫Preview/49805/7.jpg\",\n \"D:/图集岛爬虫Preview/49815/49.jpg\",\n \"D:/图集岛爬虫Preview/49816/75.jpg\",\n \"D:/图集岛爬虫Preview/49818/6.jpg\",\n \"D:/图集岛爬虫Preview/49819/26.jpg\",\n \"D:/图集岛爬虫Preview/49822/13.jpg\",\n \"D:/图集岛爬虫Preview/49822/18.jpg\",\n \"D:/图集岛爬虫Preview/49823/11.jpg\",\n \"D:/图集岛爬虫Preview/49826/15.jpg\",\n \"D:/图集岛爬虫Preview/49830/62.jpg\",\n \"D:/图集岛爬虫Preview/49836/2.jpg\",\n \"D:/图集岛爬虫Preview/49842/20.jpg\",\n \"D:/图集岛爬虫Preview/49842/35.jpg\",\n \"D:/图集岛爬虫Preview/49858/45.jpg\",\n \"D:/图集岛爬虫Preview/49859/1.jpg\",\n \"D:/图集岛爬虫Preview/49863/46.jpg\",\n \"D:/图集岛爬虫Preview/49863/5.jpg\",\n \"D:/图集岛爬虫Preview/49870/17.jpg\",\n \"D:/图集岛爬虫Preview/49874/6.jpg\",\n \"D:/图集岛爬虫Preview/49878/46.jpg\",\n \"D:/图集岛爬虫Preview/49879/15.jpg\",\n \"D:/图集岛爬虫Preview/49885/20.jpg\",\n \"D:/图集岛爬虫Preview/49891/13.jpg\",\n \"D:/图集岛爬虫Preview/49891/68.jpg\",\n \"D:/图集岛爬虫Preview/49895/69.jpg\",\n \"D:/图集岛爬虫Preview/49900/18.jpg\",\n \"D:/图集岛爬虫Preview/49900/33.jpg\",\n \"D:/图集岛爬虫Preview/49901/14.jpg\",\n \"D:/图集岛爬虫Preview/49905/71.jpg\",\n \"D:/图集岛爬虫Preview/49909/5.jpg\",\n \"D:/图集岛爬虫Preview/49910/32.jpg\",\n \"D:/图集岛爬虫Preview/49916/39.jpg\",\n \"D:/图集岛爬虫Preview/49933/29.jpg\",\n \"D:/图集岛爬虫Preview/49933/41.jpg\",\n \"D:/图集岛爬虫Preview/49936/32.jpg\",\n \"D:/图集岛爬虫Preview/49942/46.jpg\",\n \"D:/图集岛爬虫Preview/49954/79.jpg\",\n \"D:/图集岛爬虫Preview/49960/54.jpg\",\n \"D:/图集岛爬虫Preview/49962/21.jpg\",\n \"D:/图集岛爬虫Preview/49962/34.jpg\",\n \"D:/图集岛爬虫Preview/49964/7.jpg\",\n \"D:/图集岛爬虫Preview/49967/9.jpg\",\n \"D:/图集岛爬虫Preview/49969/1.jpg\",\n \"D:/图集岛爬虫Preview/49969/55.jpg\",\n \"D:/图集岛爬虫Preview/49969/85.jpg\",\n \"D:/图集岛爬虫Preview/49975/40.jpg\",\n \"D:/图集岛爬虫Preview/49976/3.jpg\",\n \"D:/图集岛爬虫Preview/49977/19.jpg\",\n \"D:/图集岛爬虫Preview/49978/25.jpg\",\n \"D:/图集岛爬虫Preview/49978/38.jpg\",\n \"D:/图集岛爬虫Preview/49978/51.jpg\",\n \"D:/图集岛爬虫Preview/49979/18.jpg\",\n \"D:/图集岛爬虫Preview/49980/4.jpg\",\n \"D:/图集岛爬虫Preview/49984/16.jpg\",\n \"D:/图集岛爬虫Preview/49984/7.jpg\",\n \"D:/图集岛爬虫Preview/49986/48.jpg\",\n \"D:/图集岛爬虫Preview/49988/24.jpg\",\n \"D:/图集岛爬虫Preview/49999/48.jpg\",\n \"D:/图集岛爬虫Preview/49999/53.jpg\",\n \"D:/图集岛爬虫Preview/49999/57.jpg\",\n \"D:/图集岛爬虫Preview/49999/64.jpg\",\n \"D:/图集岛爬虫Preview/49999/8.jpg\",\n \"D:/图集岛爬虫Preview/50000/16.jpg\",\n \"D:/图集岛爬虫Preview/50000/17.jpg\",\n \"D:/图集岛爬虫Preview/50002/38.jpg\",\n \"D:/图集岛爬虫Preview/50003/20.jpg\",\n \"D:/图集岛爬虫Preview/50004/20.jpg\",\n \"D:/图集岛爬虫Preview/50004/7.jpg\",\n \"D:/图集岛爬虫Preview/50006/42.jpg\",\n \"D:/图集岛爬虫Preview/50007/9.jpg\",\n \"D:/图集岛爬虫Preview/50012/75.jpg\",\n \"D:/图集岛爬虫Preview/50021/25.jpg\",\n \"D:/图集岛爬虫Preview/50022/46.jpg\",\n \"D:/图集岛爬虫Preview/50026/4.jpg\",\n \"D:/图集岛爬虫Preview/50028/1.jpg\",\n \"D:/图集岛爬虫Preview/50028/4.jpg\",\n \"D:/图集岛爬虫Preview/50030/19.jpg\",\n \"D:/图集岛爬虫Preview/50031/6.jpg\",\n \"D:/图集岛爬虫Preview/50032/1.jpg\",\n \"D:/图集岛爬虫Preview/50035/38.jpg\",\n \"D:/图集岛爬虫Preview/50038/18.jpg\",\n \"D:/图集岛爬虫Preview/50038/2.jpg\",\n \"D:/图集岛爬虫Preview/50038/34.jpg\",\n \"D:/图集岛爬虫Preview/50042/28.jpg\",\n \"D:/图集岛爬虫Preview/50042/38.jpg\",\n \"D:/图集岛爬虫Preview/50043/6.jpg\",\n \"D:/图集岛爬虫Preview/50048/10.jpg\",\n \"D:/图集岛爬虫Preview/50048/13.jpg\",\n \"D:/图集岛爬虫Preview/50048/18.jpg\",\n \"D:/图集岛爬虫Preview/50048/3.jpg\",\n \"D:/图集岛爬虫Preview/50048/35.jpg\",\n \"D:/图集岛爬虫Preview/50048/4.jpg\",\n \"D:/图集岛爬虫Preview/50053/10.jpg\",\n \"D:/图集岛爬虫Preview/50053/19.jpg\",\n \"D:/图集岛爬虫Preview/50053/39.jpg\",\n \"D:/图集岛爬虫Preview/50053/8.jpg\",\n \"D:/图集岛爬虫Preview/50053/9.jpg\",\n \"D:/图集岛爬虫Preview/50066/39.jpg\",\n \"D:/图集岛爬虫Preview/50068/2.jpg\",\n \"D:/图集岛爬虫Preview/50069/63.jpg\",\n \"D:/图集岛爬虫Preview/50069/64.jpg\",\n \"D:/图集岛爬虫Preview/50071/43.jpg\",\n \"D:/图集岛爬虫Preview/50077/41.jpg\",\n \"D:/图集岛爬虫Preview/50083/3.jpg\",\n \"D:/图集岛爬虫Preview/50088/19.jpg\",\n \"D:/图集岛爬虫Preview/50089/1.jpg\",\n \"D:/图集岛爬虫Preview/50091/15.jpg\",\n \"D:/图集岛爬虫Preview/50091/19.jpg\",\n \"D:/图集岛爬虫Preview/50100/36.jpg\",\n \"D:/图集岛爬虫Preview/50101/18.jpg\",\n \"D:/图集岛爬虫Preview/50101/2.jpg\",\n \"D:/图集岛爬虫Preview/50102/8.jpg\",\n \"D:/图集岛爬虫Preview/50103/24.jpg\",\n \"D:/图集岛爬虫Preview/50106/61.jpg\",\n \"D:/图集岛爬虫Preview/50109/22.jpg\",\n \"D:/图集岛爬虫Preview/50110/11.jpg\",\n \"D:/图集岛爬虫Preview/50113/53.jpg\",\n \"D:/图集岛爬虫Preview/50114/23.jpg\",\n \"D:/图集岛爬虫Preview/50114/4.jpg\",\n \"D:/图集岛爬虫Preview/50117/1.jpg\",\n \"D:/图集岛爬虫Preview/50120/13.jpg\",\n \"D:/图集岛爬虫Preview/50121/4.jpg\",\n \"D:/图集岛爬虫Preview/50121/6.jpg\",\n \"D:/图集岛爬虫Preview/50125/0.jpg\",\n \"D:/图集岛爬虫Preview/50126/39.jpg\",\n \"D:/图集岛爬虫Preview/50131/30.jpg\",\n \"D:/图集岛爬虫Preview/50140/12.jpg\",\n \"D:/图集岛爬虫Preview/50142/9.jpg\",\n \"D:/图集岛爬虫Preview/50149/12.jpg\",\n \"D:/图集岛爬虫Preview/50151/57.jpg\",\n \"D:/图集岛爬虫Preview/50154/54.jpg\",\n \"D:/图集岛爬虫Preview/50160/19.jpg\",\n \"D:/图集岛爬虫Preview/50160/21.jpg\",\n \"D:/图集岛爬虫Preview/50167/33.jpg\",\n \"D:/图集岛爬虫Preview/50170/16.jpg\",\n \"D:/图集岛爬虫Preview/50170/2.jpg\",\n \"D:/图集岛爬虫Preview/50174/22.jpg\",\n \"D:/图集岛爬虫Preview/50176/4.jpg\",\n \"D:/图集岛爬虫Preview/50177/19.jpg\",\n \"D:/图集岛爬虫Preview/50179/7.jpg\",\n \"D:/图集岛爬虫Preview/50181/1.jpg\",\n \"D:/图集岛爬虫Preview/50181/8.jpg\",\n \"D:/图集岛爬虫Preview/50183/4.jpg\",\n \"D:/图集岛爬虫Preview/50189/16.jpg\",\n \"D:/图集岛爬虫Preview/50192/17.jpg\",\n \"D:/图集岛爬虫Preview/50192/24.jpg\",\n \"D:/图集岛爬虫Preview/50192/36.jpg\",\n \"D:/图集岛爬虫Preview/50193/26.jpg\",\n \"D:/图集岛爬虫Preview/50194/8.jpg\",\n \"D:/图集岛爬虫Preview/50319/22.jpg\"\n };\n\n private static void reDownload() {\n String directoryStr = \"F:/图集岛爬虫Preview/\";\n for (String jpg : jpgs) {\n String tmp = jpg.replace(directoryStr, \"\");\n\n String albumId = tmp.split(\"-\")[0];\n String fileName = tmp.split(\"/\")[1];\n\n String onlinePath = \"https://tjg.gzhuibei.com/a/1/\" + albumId + \"/\" + fileName;\n\n SpiderUtil.ioDownload2Times(onlinePath, jpg, 3);\n }\n }\n\n private static void delSize0() {\n String part = \"图集岛爬虫Preview\";\n String directoryStr = \"F:/\" + part + \"/\";\n File directory = new File(directoryStr);\n\n File[] folders = directory.listFiles((FilenameFilter) DirectoryFileFilter.DIRECTORY);\n if (folders == null) {\n return;\n }\n for (File folder : folders) {\n String folderName = folder.getName();\n String subDirectoryStr = directoryStr + folderName;\n\n File subDirectory = new File(subDirectoryStr);\n Collection jpgFiles = FileUtils.listFiles(subDirectory, new String[]{\"jpg\"}, false);\n for (File jpg : jpgFiles) {\n if (jpg.length() == 0) {\n\n// jpg.delete();\n System.out.println(\"size0 jpg=\" + subDirectoryStr + \"/\" + jpg.getName());\n }\n }\n }\n }\n\n public static void main(String[] args) {\n// reDownload();\n\n// delSize0();\n\n Set zipFileNames = new HashSet<>();\n Set fileNames = new HashSet<>();\n File zipDirectory = new File(\"F:/zip图集岛爬虫（40001-50000）/\");\n File directory = new File(\"F:/图集岛爬虫（40001-50000）/\");\n\n Collection zipFiles = FileUtils.listFiles(zipDirectory, new String[]{\"zip\"}, false);\n for (File folder : zipFiles) {\n String folderName = folder.getName();\n zipFileNames.add(folderName.replace(\".zip\", \"\"));\n }\n\n File[] folders = directory.listFiles((FilenameFilter) DirectoryFileFilter.DIRECTORY);\n if (folders == null) {\n return;\n }\n for (File folder : folders) {\n String folderName = folder.getName();\n fileNames.add(folderName);\n }\n\n fileNames.removeAll(zipFileNames);\n System.out.println(fileNames);\n }\n}\n" }, { "path": "oys-tujidao/src/main/java/com/devyy/oys/tujidao/dao/TuJiDaoAlbumMapper.java", "content": "package com.devyy.oys.tujidao.dao;\n\nimport com.baomidou.mybatisplus.core.mapper.BaseMapper;\n\nimport com.devyy.oys.tujidao.TuJiDaoAlbumDO;\n\nimport org.springframework.stereotype.Repository;\n\n/**\n * Mybatis-Plus Mapper\n *\n * @since 2019-12-01\n */\n@Repository\npublic interface TuJiDaoAlbumMapper extends BaseMapper {\n\n}\n" }, { "path": "oys-tujidao/src/test/java/com/devyy/oys/tujidao/TuJiDaoServiceImplTest.java", "content": "package com.devyy.oys.tujidao;\n\nimport com.devyy.oys.tujidao.dao.TuJiDaoAlbumMapper;\n\nimport org.jsoup.Jsoup;\nimport org.jsoup.nodes.Document;\n\nimport org.junit.jupiter.api.Assertions;\nimport org.junit.jupiter.api.Test;\nimport org.junit.jupiter.api.extension.ExtendWith;\n\nimport org.mockito.InjectMocks;\nimport org.mockito.Mock;\nimport org.mockito.MockedStatic;\nimport org.mockito.Mockito;\nimport org.mockito.junit.jupiter.MockitoExtension;\n\nimport java.io.File;\nimport java.io.IOException;\nimport java.net.URL;\nimport java.nio.charset.StandardCharsets;\nimport java.util.ArrayList;\n\n/**\n * TuJiDaoServiceImpl UT\n *\n * @since 2022-01-18\n */\n@ExtendWith(MockitoExtension.class)\npublic class TuJiDaoServiceImplTest {\n @InjectMocks\n private final TuJiDaoServiceImpl tuJiDaoService = new TuJiDaoServiceImpl();\n\n @Mock\n private TuJiDaoAlbumMapper tuJiDaoAlbumMapper;\n\n @Test\n public void doSyncRecords() throws IOException {\n // mock 网页\n URL url = getClass().getResource(\"/gengxin2.html\");\n Assertions.assertNotNull(url);\n File file = new File(url.getPath());\n Document document = Jsoup.parse(file, StandardCharsets.UTF_8.name());\n\n // mock Jsoup.connect().cookies().get()\n MockedStatic jsoupMockedStatic = Mockito.mockStatic(Jsoup.class, Mockito.RETURNS_DEEP_STUBS);\n jsoupMockedStatic.when(() -> Jsoup.connect(Mockito.anyString()).cookies(Mockito.anyMap()).get())\n .thenReturn(document);\n\n // mock tuJiDaoAlbumMapper.selectByMap()\n Mockito.when(tuJiDaoAlbumMapper.selectByMap(Mockito.anyMap())).thenReturn(new ArrayList<>());\n\n // mock tuJiDaoAlbumMapper.insert()\n Mockito.when(tuJiDaoAlbumMapper.insert(Mockito.any())).thenReturn(1);\n\n // doTest\n tuJiDaoService.doSyncRecords();\n // verify\n jsoupMockedStatic.verify(() -> Jsoup.connect(Mockito.anyString()), Mockito.atLeastOnce());\n Mockito.verify(tuJiDaoAlbumMapper, Mockito.atLeastOnce()).insert(Mockito.any());\n }\n}" }, { "path": "oys-tujidao/src/test/resources/gengxin.html", "content": "\n\n\n\n \n \n \n \n 更新记录_图集岛\n \n \n \n \n\n \n \n \n\n\n
\n \n
\n\n
\n\n\n\n
\n\n
\n
\n\n
\n
    \n
  • 标签类型
    • \n
  • 机构厂商
    • \n
  • 地区
    • \n\n
\n
\n
\n\n
\n\n\n 极品\n 美少女\n 御姐\n 女神\n 日本少女\n 嫩模\n 妹子\n 尤物\n 正妹\n 熟女\n 乙女\n 女星\n 阳光\n 网红\n 少FU\n 日本少FU\n 泳装少女\n 软妹\n 日本妹子\n 萌女\n 车模\n 女优\n 少女\n 日本嫩模\n 清纯少女\n 性感少女\n 混血\n 黑长直\n 明星\n 杂志\n 欧美熟女\n 金发\n 黑人\n 女郎\n 轻熟女\n 巨RU女优\n 女友\n 白领\n 萝莉\n\n
风格
\n\n 可爱\n 甜美\n 清纯\n 性感\n 气质\n 妩媚\n 清新\n 唯美\n 美胸\n 诱惑\n 私房\n 翘T\n 大胆\n 福利\n 大胸\n 高贵\n 粉嫩\n 惊艳\n 白嫩\n 养眼\n 巨RU\n 情趣\n 短发\n 丰T\n 丰满\n 巨RU萝莉\n 勾魂\n 半裸\n 欧美大胸\n 蕾丝诱惑\n 制服诱惑\n 丰满少F\n 风骚少F\n 风骚\n 顶级少F\n 美T少F\n 肥TUN\n 欧美巨RU\n 学生\n 肉感\n 双马尾\n 大尺度\n 骨感\n 长发\n 冷艳\n 包T\n 美T\n 爆RU\n 魅惑\n 优雅\n 美RU\n\n
丝 · 腿 · 足
\n 丝袜女郎\n 黑丝萝莉\n 白丝萝莉\n 长筒袜\n 丝袜\n 黑丝\n 丝袜美腿\n 灰丝\n 美腿\n 高跟\n 腿模\n 丝袜诱惑\n 网袜\n 丝足\n 白丝\n 肉丝\n 红丝\n 吊带丝袜\n 美脚\n 玉足\n 情趣丝袜\n 美足\n 网衣\n 长腿\n 凉高跟\n 高跟美腿\n 欧美制服\n 欧美丝袜\n 街拍美T\n 丝袜美T\n 美丝\n 丝足高跟\n 丝足诱惑\n 街拍丝袜\n 丝袜人体\n 丝袜短裙\n 街拍短裙\n 丝袜制服\n 街拍美腿\n 街拍高跟\n 丝袜肥T\n 街拍黑丝\n 黑丝制服\n 黑丝美腿\n 丝袜熟女\n 丝袜少F\n 肉丝美腿\n 黑丝诱惑\n 情趣制服\n\n\n
情趣
\n 薄纱\n 剧情\n 肚兜\n 旗袍\n 空姐\n 赛车女郎\n 护士\n 运动\n 女仆\n 兔女郎\n 制服\n COS\n 猫女\n 美束\n 古典\n SM\n 圣诞\n 湿身\n 人体\n 厨娘\n JK\n 女警\n 老师\n 医生\n 新年\n 捆绑\n 绳艺\n 眼镜\n 不知火舞\n COSER\n 清纯护士\n 纹身\n OL\n 秘书\n 调教\n 镣铐\n 姐妹花\n 透视\n\n
场景
\n 桌球\n 浴室\n 篮球\n 拳击\n 足球\n 居家\n 办公室\n 街拍\n 外拍\n 海边\n 展台\n 钢管\n 床上\n 酒店\n 健身\n 户外\n 香车美人\n 沙滩\n\n\n
服装
\n 连衣裙\n 泳装\n 校服\n 长裙\n 热裤\n 睡衣\n 蕾丝\n 超短裙\n 和服\n 牛仔\n 水手服\n 比基尼\n 丁字裤\n 内衣\n 高叉\n 职业装\n 死库水\n 婚纱\n 豹纹\n 衬衫\n 紧身裤\n 体操服\n 皮衣\n 西装\n 学生制服\n 日本制服\n 迷你裙\n 毛衣\n 长靴\n 吊带\n 抹胸\n 背心\n\n\n
\n\n
\n
\n\n
\n\n Graphis\n 秀人网\n Beautyleg\n 克拉女神\n 绅士约拍\n DJAWA\n 壹吻映画\n LD零度\n 宅急约\n 尤蜜\n ARTGRAVIA\n 喵糖映画\n 语画界\n 尤美\n 日本CosPlay\n 森萝财团\n 蜜丝\n Cosdoki\n Girlz-High\n 台湾正妹\n 美媛馆\n 尤物馆\n 魅妍社\n 蜜桃社\n 模范学院\n 爱蜜社\n 嗲囡囡\n 尤果圈爱尤物\n 尤蜜荟\n YS Web\n Hello! Project Digital Books\n 丽柜\n ISHOW爱秀\n 丝享家\n 丝足便当\n 异思趣向\n 网络美女\n Minisuka.tv\n Digi-Gra\n LovePop\n\n\n
停更 · 倒闭
\n 推女郎\n 头条女神\n Young Magazine\n Young Gangan\n Young Champion\n Weekly Big Comic Spirits\n FRIDAY杂志写真\n FLASH杂志写真\n 阳光宝贝\n 丝意SIEE\n 风之领域\n 猎女神\n OnlyTease\n 星乐园\n 尤果网\n 波萝社\n @misty\n Wanibooks\n 影私荟\n 顽味生活\n 优星馆\n 御女郎\n NS Eyes\n Image.tv\n Bomb.TV\n PB写真集\n 4K-STAR\n 动感之星\n Bejean On Line\n 51MODO\n Imuto.tv\n 推女神\n DDY Pantyhose\n 爱丝\n VYJ\n 网红馆\n WPB写真\n 美腿宝贝\n 瑞丝馆\n 薄荷叶\n Sabra\n 果团网\n 青豆客\n 花の颜\n 模特联盟\n 花漾\n 兔几盟\n Juicy Honey\n X-City\n Princess Collection\n 熊川纪信\n 星颜社\n The Black Alley\n 激萌文化\n Young Animal Arashi\n DGC\n RQ-STAR\n Young Animal\n For-side\n Weekly Playboy\n Weekly Young Jump\n BWH\n 山茶摄影\n 奈丝\n 大生模拍\n 佳爷SJA\n 一千零一夜\n 丽丝映像\n 斯文传媒SIW\n 轻兰映画\n 纳丝摄影\n 袜涩\n 思话\n 丝慕\n\n\n
\n\n
\n\n
\n\n
国内日本中国台湾欧美韩国泰国
\n\n
\n\n
\n
\n\n
\n\n
\n\n
\n
\n 主页\n 更新记录\n 我的收藏\n 帐户升级\n 修改\n 退出\n
\n
\n
\n

公告:春节放假停更，2021.02.10 - 02.19 节后恢复正常更新，祝所有用户新年快乐，感谢你们的支持 。

\n
\n 更新记录\n
\n\n
\n \n
\n\n\n
\n
\n
\n \n
到底了~ APP下载 | 网址发布\n

\n\n" }, { "path": "oys-tujidao/src/test/resources/gengxin2.html", "content": "\n\n\n\n \n \n \n \n 更新记录_图集岛\n \n \n \n \n\n \n \n \n\n\n
\n \n
\n\n
\n\n\n\n
\n\n
\n
\n\n
\n \n
\n
\n
\n\n
\n\n\n 极品\n 美少女\n 御姐\n 女神\n 日本少女\n 嫩模\n 妹子\n 尤物\n 正妹\n 熟女\n 乙女\n 女星\n 阳光\n 网红\n 少FU\n 日本少FU\n 泳装少女\n 软妹\n 日本妹子\n 萌女\n 车模\n 女优\n 少女\n 日本嫩模\n 清纯少女\n 性感少女\n 混血\n 黑长直\n 明星\n 杂志\n 欧美熟女\n 金发\n 黑人\n 女郎\n 轻熟女\n 巨RU女优\n 女友\n 白领\n 萝莉\n\n
风格
\n\n 可爱\n 甜美\n 清纯\n 性感\n 气质\n 妩媚\n 清新\n 唯美\n 美胸\n 诱惑\n 私房\n 翘T\n 大胆\n 福利\n 大胸\n 高贵\n 粉嫩\n 惊艳\n 白嫩\n 养眼\n 巨RU\n 情趣\n 短发\n 丰T\n 丰满\n 巨RU萝莉\n 勾魂\n 半裸\n 欧美大胸\n 蕾丝诱惑\n 制服诱惑\n 丰满少F\n 风骚少F\n 风骚\n 顶级少F\n 美T少F\n 肥TUN\n 欧美巨RU\n 学生\n 肉感\n 双马尾\n 大尺度\n 骨感\n 长发\n 冷艳\n 包T\n 美T\n 爆RU\n 魅惑\n 优雅\n 美RU\n\n
丝 · 腿 · 足
\n 丝袜女郎\n 黑丝萝莉\n 白丝萝莉\n 长筒袜\n 丝袜\n 黑丝\n 丝袜美腿\n 灰丝\n 美腿\n 高跟\n 腿模\n 丝袜诱惑\n 网袜\n 丝足\n 白丝\n 肉丝\n 红丝\n 吊带丝袜\n 美脚\n 玉足\n 情趣丝袜\n 美足\n 网衣\n 长腿\n 凉高跟\n 高跟美腿\n 欧美制服\n 欧美丝袜\n 街拍美T\n 丝袜美T\n 美丝\n 丝足高跟\n 丝足诱惑\n 街拍丝袜\n 丝袜人体\n 丝袜短裙\n 街拍短裙\n 丝袜制服\n 街拍美腿\n 街拍高跟\n 丝袜肥T\n 街拍黑丝\n 黑丝制服\n 黑丝美腿\n 丝袜熟女\n 丝袜少F\n 肉丝美腿\n 黑丝诱惑\n 情趣制服\n\n\n
情趣
\n 薄纱\n 剧情\n 肚兜\n 旗袍\n 空姐\n 赛车女郎\n 护士\n 运动\n 女仆\n 兔女郎\n 制服\n COS\n 猫女\n 美束\n 古典\n SM\n 圣诞\n 湿身\n 人体\n 厨娘\n JK\n 女警\n 老师\n 医生\n 新年\n 捆绑\n 绳艺\n 眼镜\n 不知火舞\n COSER\n 清纯护士\n 纹身\n OL\n 秘书\n 调教\n 镣铐\n 姐妹花\n 透视\n\n
场景
\n 桌球\n 浴室\n 篮球\n 拳击\n 足球\n 居家\n 办公室\n 街拍\n 外拍\n 海边\n 展台\n 钢管\n 床上\n 酒店\n 健身\n 户外\n 香车美人\n 沙滩\n\n\n
服装
\n 连衣裙\n 泳装\n 校服\n 长裙\n 热裤\n 睡衣\n 蕾丝\n 超短裙\n 和服\n 牛仔\n 水手服\n 比基尼\n 丁字裤\n 内衣\n 高叉\n 职业装\n 死库水\n 婚纱\n 豹纹\n 衬衫\n 紧身裤\n 体操服\n 皮衣\n 西装\n 学生制服\n 日本制服\n 迷你裙\n 毛衣\n 长靴\n 吊带\n 抹胸\n 背心\n\n\n
\n\n
\n
\n\n
\n\n Graphis\n 秀人网\n Beautyleg\n 克拉女神\n 绅士约拍\n DJAWA\n 壹吻映画\n LD零度\n 宅急约\n 尤蜜\n ARTGRAVIA\n 喵糖映画\n 语画界\n 尤美\n 日本CosPlay\n 森萝财团\n 蜜丝\n Cosdoki\n Girlz-High\n 台湾正妹\n 美媛馆\n 尤物馆\n 魅妍社\n 蜜桃社\n 模范学院\n 爱蜜社\n 嗲囡囡\n 尤果圈爱尤物\n 尤蜜荟\n YS Web\n Hello! Project Digital Books\n 丽柜\n ISHOW爱秀\n 丝享家\n 丝足便当\n 异思趣向\n 网络美女\n Minisuka.tv\n Digi-Gra\n LovePop\n\n\n
停更 · 倒闭
\n 推女郎\n 头条女神\n Young Magazine\n Young Gangan\n Young Champion\n Weekly Big Comic Spirits\n FRIDAY杂志写真\n FLASH杂志写真\n 阳光宝贝\n 丝意SIEE\n 风之领域\n 猎女神\n OnlyTease\n 星乐园\n 尤果网\n 波萝社\n @misty\n Wanibooks\n 影私荟\n 顽味生活\n 优星馆\n 御女郎\n NS Eyes\n Image.tv\n Bomb.TV\n PB写真集\n 4K-STAR\n 动感之星\n Bejean On Line\n 51MODO\n Imuto.tv\n 推女神\n DDY Pantyhose\n 爱丝\n VYJ\n 网红馆\n WPB写真\n 美腿宝贝\n 瑞丝馆\n 薄荷叶\n Sabra\n 果团网\n 青豆客\n 花の颜\n 模特联盟\n 花漾\n 兔几盟\n Juicy Honey\n X-City\n Princess Collection\n 熊川纪信\n 星颜社\n The Black Alley\n 激萌文化\n Young Animal Arashi\n DGC\n RQ-STAR\n Young Animal\n For-side\n Weekly Playboy\n Weekly Young Jump\n BWH\n 山茶摄影\n 奈丝\n 大生模拍\n 佳爷SJA\n 一千零一夜\n 丽丝映像\n 斯文传媒SIW\n 轻兰映画\n 纳丝摄影\n 袜涩\n 思话\n 丝慕\n\n\n
\n\n
\n\n
\n\n
国内日本中国台湾欧美韩国泰国
\n\n
\n\n\n
\n
\n\n
\n\n
\n\n
\n
\n 全部图集 / 更新记录\n
\n
\n\n
\n \n
\n\n
\n
\n
\n
跳转至 页
\n
\n\n\n\n\n\n
到底了~ APP下载 | 网址发布\n

\n\n" }, { "path": "pom.xml", "content": "\n\n 4.0.0\n\n com.devyy\n openyspider\n ${revision}\n pom\n\n \n oys-core\n oys-tujidao\n oys-boot-starter\n oys-codeforces\n \n\n \n \n 2.7.5\n \n 1.18.24\n 3.5.2\n 3.0.0\n 2.11.0\n 1.15.3\n 4.1.4\n\n 1.3.0\n 2.22.2\n 2.22.2\n 0.8.8\n\n \n 4.0-SNAPSHOT\n\n 17\n 17\n 17\n UTF-8\n \n\n \n \n \n org.springframework.boot\n spring-boot-dependencies\n ${org.springframework.boot.spring-boot-dependencies.version}\n pom\n import\n \n \n org.seleniumhq.selenium\n selenium-api\n ${org.seleniumhq.selenium.selenium-chrome-driver.version}\n \n \n org.seleniumhq.selenium\n selenium-remote-driver\n ${org.seleniumhq.selenium.selenium-chrome-driver.version}\n \n \n \n\n \n \n org.springframework.boot\n spring-boot-starter-web\n \n \n \n org.springframework.boot\n spring-boot-starter-test\n \n \n org.mockito\n mockito-inline\n \n\n \n \n org.jsoup\n jsoup\n ${org.jsoup.jsoup.version}\n \n \n \n org.projectlombok\n lombok\n ${org.projectlombok.lombok.version}\n \n \n \n mysql\n mysql-connector-java\n \n \n \n com.baomidou\n mybatis-plus-boot-starter\n ${com.baomidou.mybatis-plus-boot-starter.version}\n \n \n \n io.springfox\n springfox-boot-starter\n ${io.springfox.springfox-boot-starter.version}\n \n \n \n commons-io\n commons-io\n ${commons-io.commons-io.version}\n \n\n \n \n org.seleniumhq.selenium\n selenium-chrome-driver\n ${org.seleniumhq.selenium.selenium-chrome-driver.version}\n \n\n \n org.springframework.kafka\n spring-kafka\n \n \n org.springframework.boot\n spring-boot-starter-data-redis\n \n\n \n \n com.kennycason\n kumo-core\n 1.28\n \n \n com.kennycason\n kumo-api\n 1.28\n \n \n\n \n \n \n org.codehaus.mojo\n flatten-maven-plugin\n ${flatten-maven-plugin.version}\n \n true\n resolveCiFriendliesOnly\n \n expand\n \n \n \n \n flatten\n process-resources\n \n flatten\n \n \n \n flatten.clean\n clean\n \n clean\n \n \n \n \n\n \n org.apache.maven.plugins\n maven-surefire-plugin\n ${maven-surefire-plugin.version}\n \n \n org.apache.maven.plugins\n maven-failsafe-plugin\n ${maven-failsafe-plugin.version}\n \n\n \n \n org.jacoco\n jacoco-maven-plugin\n ${jacoco-maven-plugin.version}\n \n \n \n prepare-agent\n \n \n \n \n \n \n" }, { "path": "sql_scripts/oys3_all_scripts_mysql.sql", "content": "/*==============================================================*/\n/* Table: OYS_TUJIDAO_ALBUM */\n/*==============================================================*/\nCREATE TABLE OYS_TUJIDAO_ALBUM_T\n(\n ID BIGINT NOT NULL AUTO_INCREMENT COMMENT '主键 ID' ,\n STATE INT COMMENT '状态',\n TOTAL INT COMMENT '图片总数',\n ALBUM_NAME VARCHAR(255) COMMENT '相册名',\n ALBUM_ID INT COMMENT '相册id',\n CREATION_DATE DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间',\n LAST_UPDATED_DATE DATETIME DEFAULT CURRENT_TIMESTAMP COMMENT '最后修改时间',\n PRIMARY KEY (ID),\n KEY ALBUM_ID_UNIQUE (ALBUM_ID)\n);\n\n/*==============================================================*/\n/* Table: OYS_CODEFORCES */\n/*==============================================================*/\nCREATE TABLE OYS_CODEFORCES\n(\n SUBMISSION_ID VARCHAR(32) NOT NULL COMMENT '提交 ID',\n NO NUMERIC(11,0) COMMENT '序号',\n FRAGMENT VARCHAR(1024) COMMENT '片段值',\n PRIMARY KEY (SUBMISSION_ID)\n);" } ]