[ { "path": "README.md", "content": "## 百度指数抓取，再用图像识别得到指数\n\n## 前言：\n土福曾说，百度指数很难抓，在淘宝上面是20块1个关键字：\n\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110162432795-1923984431.png)\n\n哥那么叼的人怎么会被他吓到，于是乎花了零零碎碎加起来大约2天半搞定，在此鄙视一下土福\n\n### 安装的库很多：\n>谷歌图像识别tesseract-ocr\n\n>pip3 install pillow\n\n>pip3 install pyocr\n\n>selenium2.45\n\n>Chrome47.0.2526.106 m or Firebox32.0.1\n\n>chromedriver.exe\n\n### 图像识别验证码请参考我的博客：\n[python图像识别--验证码](http://www.cnblogs.com/TTyb/p/5996847.html)\n\n### selenium用法请参考我的博客：\n[python之selenium](http://www.cnblogs.com/TTyb/p/5842015.html)\n\n### 进入百度指数需要登陆，登陆的账号密码写在文本account里面：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110153714827-1835068903.png)\n\n### 万能登陆代码如下：\n```\n# 打开浏览器\ndef openbrowser():\n global browser\n\n # https://passport.baidu.com/v2/?login\n url = \"https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F\"\n # 打开谷歌浏览器\n # Firefox()\n # Chrome()\n browser = webdriver.Chrome()\n # 输入网址\n browser.get(url)\n # 打开浏览器时间\n # print(\"等待10秒打开浏览器...\")\n # time.sleep(10)\n\n # 找到id=\"TANGRAM__PSP_3__userName\"的对话框\n # 清空输入框\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").clear()\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").clear()\n\n # 输入账号密码\n # 输入账号密码\n account = []\n try:\n fileaccount = open(\"../baidu/account.txt\")\n accounts = fileaccount.readlines()\n for acc in accounts:\n account.append(acc.strip())\n fileaccount.close()\n except Exception as err:\n print(err)\n input(\"请正确在account.txt里面写入账号密码\")\n exit()\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").send_keys(account[0])\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").send_keys(account[1])\n\n # 点击登陆登陆\n # id=\"TANGRAM__PSP_3__submit\"\n browser.find_element_by_id(\"TANGRAM__PSP_3__submit\").click()\n\n # 等待登陆10秒\n # print('等待登陆10秒...')\n # time.sleep(10)\n print(\"等待网址加载完毕...\")\n\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n while 1:\n if select == \"y\" or select == \"Y\":\n print(\"登陆成功！\")\n print(\"准备打开新的窗口...\")\n # time.sleep(1)\n # browser.quit()\n break\n\n elif select == \"n\" or select == \"N\":\n selectno = input(\"账号密码错误请按0，验证码出现请按1...\")\n # 账号密码错误则重新输入\n if selectno == \"0\":\n\n # 找到id=\"TANGRAM__PSP_3__userName\"的对话框\n # 清空输入框\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").clear()\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").clear()\n\n # 输入账号密码\n account = []\n try:\n fileaccount = open(\"../baidu/account.txt\")\n accounts = fileaccount.readlines()\n for acc in accounts:\n account.append(acc.strip())\n fileaccount.close()\n except Exception as err:\n print(err)\n input(\"请正确在account.txt里面写入账号密码\")\n exit()\n\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").send_keys(account[0])\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").send_keys(account[1])\n # 点击登陆sign in\n # id=\"TANGRAM__PSP_3__submit\"\n browser.find_element_by_id(\"TANGRAM__PSP_3__submit\").click()\n\n elif selectno == \"1\":\n # 验证码的id为id=\"ap_captcha_guess\"的对话框\n input(\"请在浏览器中输入验证码并登陆...\")\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n\n else:\n print(\"请输入“y”或者“n”！\")\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n```\n\n### 登陆的页面：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110154107624-1393804790.png)\n\n### 登陆过后需要打开新的窗口，也就是打开百度指数，并且切换窗口，在selenium用：\n```\n# 新开一个窗口，通过执行js来新开一个窗口\njs = 'window.open(\"http://index.baidu.com\");'\nbrowser.execute_script(js)\n# 新窗口句柄切换，进入百度指数\n# 获得当前打开所有窗口的句柄handles\n# handles为一个数组\nhandles = browser.window_handles\n# print(handles)\n# 切换到当前最新打开的窗口\nbrowser.switch_to_window(handles[-1])\n \n```\n\n### 清空输入框，构造点击天数：\n```\n# 清空输入框\nbrowser.find_element_by_id(\"schword\").clear()\n# 写入需要搜索的百度指数\nbrowser.find_element_by_id(\"schword\").send_keys(keyword)\n# 点击搜索\n# \nbrowser.find_element_by_id(\"searchWords\").click()\ntime.sleep(2)\n# 最大化窗口\nbrowser.maximize_window()\n# 构造天数\nsel = int(input(\"查询7天请按0，30天请按1，90天请按2，半年请按3：\"))\nday = 0\nif sel == 0:\n day = 7\nelif sel == 1:\n day = 30\nelif sel == 2:\n day = 90\nelif sel == 3:\n day = 180\nsel = '//a[@rel=\"' + str(day) + '\"]'\nbrowser.find_element_by_xpath(sel).click()\n# 太快了\ntime.sleep(2)\n```\n\n### 天数也就是这里：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110154603561-1092775179.png)\n\n### 找到图形框：\n```\nxoyelement = browser.find_elements_by_css_selector(\"#trend rect\")[2]\n```\n\n### 图形框就是：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110154817608-982142410.png)\n\n### 根据坐标点的不同构造偏移量：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110155142530-319352053.png)\n\n### 选取7天的坐标来观察：\n>第一个点的横坐标为1031.66666\n\n>第二个点的横坐标为1234\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110155720764-1186100464.png)\n\n所以7天两个坐标之间的差为：202.33，其他的天数类似\n\n### 用selenium库来模拟鼠标滑动悬浮：\n```\nfrom selenium.webdriver.common.action_chains import ActionChains\nActionChains(browser).move_to_element_with_offset(xoyelement,x_0,y_0).perform()\n```\n\n### 但是这样子确定的点指出是在这个位置：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110155752202-120333333.png)\n\n也就是矩形的左上角，这里是不会加载js显示弹出框的，所以要给横坐标+1：\n```\nx_0 = 1\ny_0 = 0\n```\n\n### 写个按照天数的循环，让横坐标累加：\n```\n# 按照选择的天数循环\nfor i in range(day):\n # 构造规则\n if day == 7:\n x_0 = x_0 + 202.33\n elif day == 30:\n x_0 = x_0 + 41.68\n elif day == 90:\n x_0 = x_0 + 13.64\n elif day == 180:\n x_0 = x_0 + 6.78\n```\n\n### 鼠标横移时会弹出框，在网址里面找到这个框：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110160257592-723215476.png)\n\n### selenium自动识别之...：\n```\n#
\nimgelement = browser.find_element_by_xpath('//div[@id=\"viewbox\"]')\n```\n\n### 并且确定这个框的大小位置：\n```\n# 找到图片坐标\nlocations = imgelement.location\nprint(locations)\n# 找到图片大小\nsizes = imgelement.size\nprint(sizes)\n# 构造指数的位置\nrangle = (int(locations['x']), int(locations['y']), int(locations['x'] + sizes['width']),\n int(locations['y'] + sizes['height']))\n```\n\n截取的图形为：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110160502389-924750650.png)\n\n### 下面的思路就是：\n>1. 将整个屏幕截图下来\n>2. 打开截图用上面得到的这个坐标rangle进行裁剪\n\n### 但是最后裁剪出来的是上面的那个黑框，我想要的效果是：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110160724577-1831216031.jpg)\n\n### 本次更新加入了对于关键词长度的判断，能够自动识别关键词长度而进行截取：\n```\nadd_length = (len(keyword) - 2) * sizes['width'] / 15\n```\n\n### 找到位置：\n```\n# 跨浏览器兼容\nscroll = browser.execute_script(\"return window.scrollY;\")\ntop = locations['y'] - scroll\n# 构造指数的位置\nrangle = (\n int(locations['x'] + sizes['width'] / 4 + add_length), int(top + sizes['height'] / 2),\n int(locations['x'] + sizes['width'] * 2 / 3), int(top + sizes['height']))\n```\n\n### 后面的完整代码是：\n```\n#
\nimgelement = browser.find_element_by_xpath('//div[@id=\"viewbox\"]')\n# 找到图片坐标\nlocations = imgelement.location\nprint(locations)\n# 找到图片大小\nsizes = imgelement.size\nprint(sizes)\n# 构造指数的位置\nrangle = (int(locations['x'] + sizes['width']/3), int(locations['y'] + sizes['height']/2), int(locations['x'] + sizes['width']*2/3),\n int(locations['y'] + sizes['height']))\n# 截取当前浏览器\npath = \"../baidu/\" + str(num)\nbrowser.save_screenshot(str(path) + \".png\")\n# 打开截图切割\nimg = Image.open(str(path) + \".png\")\njpg = img.crop(rangle)\njpg.save(str(path) + \".jpg\")\n```\n\n### 但是后面发现裁剪的图片太小，识别精度太低，所以需要对图片进行扩大：\n```\n# 将图片放大一倍\n# 原图大小73.29\njpgzoom = Image.open(str(path) + \".jpg\")\n(x, y) = jpgzoom.size\nx_s = 146\ny_s = 58\nout = jpgzoom.resize((x_s, y_s), Image.ANTIALIAS)\nout.save(path + 'zoom.jpg', 'png', quality=95)\n```\n\n原图大小请 **右键->属性->详细信息** 查看，我的是长73像素，宽29像素\n\n### 最后就是图像识别\n```\n# 图像识别\nindex = []\nimage = Image.open(str(path) + \"zoom.jpg\")\ncode = pytesseract.image_to_string(image)\nif code:\n index.append(code)\n```\n\n### 最后效果图：\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110161512889-300916957.png)\n\n![](http://images2015.cnblogs.com/blog/996148/201611/996148-20161110161525874-60911542.png)\n\n\n## 详细解说请观看我的博客：\n[TTyb](http://www.cnblogs.com/TTyb)\n\n# 更新日志：\n\n> 2017-10-23修复截图位置不对的bug，优化关键词自动识别长度的漏洞\n" }, { "path": "baidu/account.txt", "content": "你的账号\n你的密码" }, { "path": "baidu/index.txt", "content": "64.261\n71.179\n63. 644\n78. 694\n105. 963\n60.110\n1.092.397\n" }, { "path": "baidu/viewbox.md", "content": "
\n
\n
2016-10-10 һ
\n
\n
\n
\n
\n \n \n \n \n \n \n \n \n
\n \n ϣ\n \n
\n \n \n
\n \n \n
\n \n \n
\n \n \n
\n \n \n
\n \n
\n
\n
\n
\n
\n
" }, { "path": "py/Baiduindex.py", "content": "# !/usr/bin/python3.4\n# -*- coding: utf-8 -*-\n\n\n# 百度指数的抓取\n# 截图教程：http://www.myexception.cn/web/2040513.html\n#\n# 登陆百度地址：https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F\n# 百度指数地址：http://index.baidu.com\n\nimport time\nfrom selenium import webdriver\nfrom selenium.webdriver.common.action_chains import ActionChains\nfrom PIL import Image\nimport pytesseract\n\n\n# 打开浏览器\ndef openbrowser():\n global browser\n\n # https://passport.baidu.com/v2/?login\n url = \"https://passport.baidu.com/v2/?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2F\"\n # 打开谷歌浏览器\n # Firefox()\n # Chrome()\n browser = webdriver.Chrome()\n # 输入网址\n browser.get(url)\n # 打开浏览器时间\n # print(\"等待10秒打开浏览器...\")\n # time.sleep(10)\n\n # 找到id=\"TANGRAM__PSP_3__userName\"的对话框\n # 清空输入框\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").clear()\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").clear()\n\n # 输入账号密码\n # 输入账号密码\n account = []\n try:\n fileaccount = open(\"../baidu/account.txt\", encoding='UTF-8')\n accounts = fileaccount.readlines()\n for acc in accounts:\n account.append(acc.strip())\n fileaccount.close()\n except Exception as err:\n print(err)\n input(\"请正确在account.txt里面写入账号密码\")\n exit()\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").send_keys(account[0])\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").send_keys(account[1])\n\n # 点击登陆登陆\n # id=\"TANGRAM__PSP_3__submit\"\n browser.find_element_by_id(\"TANGRAM__PSP_3__submit\").click()\n\n # 等待登陆10秒\n # print('等待登陆10秒...')\n # time.sleep(10)\n print(\"等待网址加载完毕...\")\n\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n while 1:\n if select == \"y\" or select == \"Y\":\n print(\"登陆成功！\")\n print(\"准备打开新的窗口...\")\n # time.sleep(1)\n # browser.quit()\n break\n\n elif select == \"n\" or select == \"N\":\n selectno = input(\"账号密码错误请按0，验证码出现请按1...\")\n # 账号密码错误则重新输入\n if selectno == \"0\":\n\n # 找到id=\"TANGRAM__PSP_3__userName\"的对话框\n # 清空输入框\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").clear()\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").clear()\n\n # 输入账号密码\n account = []\n try:\n fileaccount = open(\"../baidu/account.txt\", encoding='UTF-8')\n accounts = fileaccount.readlines()\n for acc in accounts:\n account.append(acc.strip())\n fileaccount.close()\n except Exception as err:\n print(err)\n input(\"请正确在account.txt里面写入账号密码\")\n exit()\n\n browser.find_element_by_id(\"TANGRAM__PSP_3__userName\").send_keys(account[0])\n browser.find_element_by_id(\"TANGRAM__PSP_3__password\").send_keys(account[1])\n # 点击登陆sign in\n # id=\"TANGRAM__PSP_3__submit\"\n browser.find_element_by_id(\"TANGRAM__PSP_3__submit\").click()\n\n elif selectno == \"1\":\n # 验证码的id为id=\"ap_captcha_guess\"的对话框\n input(\"请在浏览器中输入验证码并登陆...\")\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n\n else:\n print(\"请输入“y”或者“n”！\")\n select = input(\"请观察浏览器网站是否已经登陆(y/n)：\")\n\n\ndef getindex(keyword, day):\n openbrowser()\n time.sleep(2)\n\n # 这里开始进入百度指数\n # 要不这里就不要关闭了，新打开一个窗口\n # http://blog.csdn.net/DongGeGe214/article/details/52169761\n # 新开一个窗口，通过执行js来新开一个窗口\n js = 'window.open(\"http://index.baidu.com\");'\n browser.execute_script(js)\n # 新窗口句柄切换，进入百度指数\n # 获得当前打开所有窗口的句柄handles\n # handles为一个数组\n handles = browser.window_handles\n # print(handles)\n # 切换到当前最新打开的窗口\n browser.switch_to_window(handles[-1])\n # 在新窗口里面输入网址百度指数\n # 清空输入框\n time.sleep(5)\n browser.find_element_by_id(\"schword\").clear()\n # 写入需要搜索的百度指数\n browser.find_element_by_id(\"schword\").send_keys(keyword)\n # 点击搜索\n # \n browser.find_element_by_id(\"searchWords\").click()\n time.sleep(5)\n # 最大化窗口\n browser.maximize_window()\n time.sleep(2)\n # 构造天数\n sel = '//a[@rel=\"' + str(day) + '\"]'\n browser.find_element_by_xpath(sel).click()\n # 太快了\n time.sleep(2)\n # 滑动思路：http://blog.sina.com.cn/s/blog_620987bf0102v2r8.html\n # 滑动思路：http://blog.csdn.net/zhouxuan623/article/details/39338511\n # 向上移动鼠标80个像素，水平方向不同\n # ActionChains(browser).move_by_offset(0,-80).perform()\n #
\n # \n # xoyelement = browser.find_element_by_xpath('//rect[@stroke=\"none\"]')\n xoyelement = browser.find_elements_by_css_selector(\"#trend rect\")[2]\n num = 0\n # 获得坐标长宽\n # x = xoyelement.location['x']\n # y = xoyelement.location['y']\n # width = xoyelement.size['width']\n # height = xoyelement.size['height']\n # print(x,y,width,height)\n # 常用js:http://www.cnblogs.com/hjhsysu/p/5735339.html\n # 搜索词：selenium JavaScript模拟鼠标悬浮\n x_0 = 1\n y_0 = 0\n\n if day == \"all\":\n day = 1000000\n\n # 储存数字的数组\n index = []\n try:\n # webdriver.ActionChains(driver).move_to_element().click().perform()\n # 只有移动位置xoyelement[2]是准确的\n for i in range(day):\n # 坐标偏移量???\n ActionChains(browser).move_to_element_with_offset(xoyelement, x_0, y_0).perform()\n\n # 构造规则\n if day == 7:\n x_0 = x_0 + 202.33\n elif day == 30:\n x_0 = x_0 + 41.68\n elif day == 90:\n x_0 = x_0 + 13.64\n elif day == 180:\n x_0 = x_0 + 6.78\n elif day == 1000000:\n x_0 = x_0 + 3.37222222\n time.sleep(2)\n #
\n imgelement = browser.find_element_by_xpath('//div[@id=\"viewbox\"]')\n # 找到图片坐标\n locations = imgelement.location\n # 跨浏览器兼容\n scroll = browser.execute_script(\"return window.scrollY;\")\n top = locations['y'] - scroll\n # 找到图片大小\n sizes = imgelement.size\n # 构造关键词长度\n add_length = (len(keyword) - 2) * sizes['width'] / 15\n # 构造指数的位置\n rangle = (\n int(locations['x'] + sizes['width'] / 4 + add_length), int(top + sizes['height'] / 2),\n int(locations['x'] + sizes['width'] * 2 / 3), int(top + sizes['height']))\n # 截取当前浏览器\n path = \"../baidu/\" + str(num)\n browser.save_screenshot(str(path) + \".png\")\n # 打开截图切割\n img = Image.open(str(path) + \".png\")\n jpg = img.crop(rangle)\n jpg.save(str(path) + \".jpg\")\n\n # 将图片放大一倍\n # 原图大小73.29\n jpgzoom = Image.open(str(path) + \".jpg\")\n (x, y) = jpgzoom.size\n x_s = 146\n y_s = 58\n out = jpgzoom.resize((x_s, y_s), Image.ANTIALIAS)\n out.save(path + 'zoom.jpg', 'png', quality=95)\n\n # 图像识别\n try:\n image = Image.open(str(path) + \"zoom.jpg\")\n code = pytesseract.image_to_string(image)\n if code:\n index.append(code)\n else:\n index.append(\"\")\n except:\n index.append(\"\")\n num = num + 1\n\n except Exception as err:\n print(err)\n print(num)\n\n print(index)\n # 日期也是可以图像识别下来的\n # 只是要构造rangle就行，但是我就是懒\n file = open(\"../baidu/index.txt\", \"w\")\n for item in index:\n file.write(str(item) + \"\\n\")\n file.close()\n\n\nif __name__ == \"__main__\":\n # 每个字大约占横坐标12.5这样\n # 按照字节可自行更改切割横坐标的大小rangle\n keyword = input(\"请输入查询关键字：\")\n sel = int(input(\"查询7天请按0，30天请按1，90天请按2，半年请按3，全部请按4：\"))\n day = 0\n if sel == 0:\n day = 7\n elif sel == 1:\n day = 30\n elif sel == 2:\n day = 90\n elif sel == 3:\n day = 180\n elif sel == 4:\n day = \"all\"\n getindex(keyword, day)\n" }, { "path": "update_log.md", "content": "# 更新日志：\n\n> 2017-12-07修复截图位置不对的bug，自适应不同分辨率的浏览器\n\n> 2017-11-27增加全部选项抓取\n\n> 2017-10-23修复截图位置不对的bug，优化关键词自动识别长度的漏洞" } ]