Repository: hardikvasa/google-images-download
Branch: master
Commit: 0d2bf8f17b5a
Files: 24
Total size: 115.5 KB
Directory structure:
gitextract_7o9o5ztx/
├── .gitignore
├── Licence.txt
├── MANIFEST.in
├── README.rst
├── docs/
│ ├── Makefile
│ ├── arguments.rst
│ ├── conf.py
│ ├── examples.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── make.bat
│ ├── structure.rst
│ ├── troubleshooting.rst
│ └── usage.rst
├── google_images_download/
│ ├── __init__.py
│ ├── __main__.py
│ ├── google_images_download.py
│ └── sample_config.json
├── requirements.txt
├── setup.cfg
├── setup.py
└── tests/
├── __init__.py
├── test_google_images_download.py
└── test_sample.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
*.py[cod]
# C extensions
*.so
#env
env/
env
# pycharm
.idea/
.idea
# Packages
*.egg
*.egg-info
build
dist
dist/*
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage
.tox
nosetests.xml
# Complexity
output/*.html
output/*/index.html
# Sphinx
docs/_build
docs/.DS_Store
docs/_static/*
# Cookiecutter
output/
# Downloads
downloads/
# Logs
logs/
================================================
FILE: Licence.txt
================================================
The MIT License (MIT)
Copyright (c) 2015-2019 Hardik Vasa
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: MANIFEST.in
================================================
include README.md
include requirements.txt
================================================
FILE: README.rst
================================================
Google Images Download
######################
Python Script for 'searching' and 'downloading' hundreds of Google images to the local hard disk!
Documentation
=============
* `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
* `Installation <https://google-images-download.readthedocs.io/en/latest/installation.html>`__
* `Input arguments <https://google-images-download.readthedocs.io/en/latest/arguments.html>`__
* `Examples and Code Samples <https://google-images-download.readthedocs.io/en/latest/examples.html#>`__
Disclaimer
==========
This program lets you download tons of images from Google.
Please do not download or use any image that violates its copyright terms.
Google Images is a search engine that merely indexes images and allows you to find them.
It does NOT produce its own images and, as such, it doesn't own copyright on any of them.
The original creators of the images own the copyrights.
Images published in the United States are automatically copyrighted by their owners,
even if they do not explicitly carry a copyright warning.
You may not reproduce copyright images without their owner's permission,
except in "fair use" cases,
or you could risk running into lawyer's warnings, cease-and-desist letters, and copyright suits.
Please be very careful before its usage! Use this script/code only for educational purposes.
================================================
FILE: docs/Makefile
================================================
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SOURCEDIR = .
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
================================================
FILE: docs/arguments.rst
================================================
===============
Input Arguments
===============
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| Argument | Short hand | Description |
+===================+=============+===============================================================================================================================+
| config_file | cf | You can pass the arguments inside a config file. This is an alternative to passing arguments on the command line directly. |
| | | |
| | | Please refer to the |
| | | `config file format <https://github.com/hardikvasa/google-images-download/blob/master/README.rst#config-file-format>`__ below |
| | | |
| | | * If 'config_file' argument is present, the program will use the config file and command line arguments will be discarded |
| | | * Config file can only be in **JSON** format |
| | | * Please refrain from passing invalid arguments from config file. Refer to the below arguments list |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| keywords | k | Denotes the keywords/key phrases you want to search for. For more than one keywords, wrap it in single quotes. |
| | | |
| | | Tips: |
| | | |
| | | * If you simply type the keyword, Google will best try to match it |
| | | * If you want to search for exact phrase, you can wrap the keywords in double quotes ("") |
| | | * If you want to search to contain either of the words provided, use **OR** between the words. |
| | | * If you want to explicitly not want a specific word use a minus sign before the word (-) |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| keywords_from_file| kf | Denotes the file name from where you would want to import the keywords. |
| | | |
| | | Add one keyword per line. Blank/Empty lines are truncated automatically. |
| | | |
| | | Only file types '.txt' or '.csv' are allowed. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| prefix_keywords | pk | Denotes additional words added before main keyword while making the search query. |
| | | |
| | | The final search query would be: <prefix keyword> <keyword> |
| | | |
| | | So, for example, if the keyword is 'car' and prefix_keyword is 'red,yellow,blue', it will search and download images for |
| | | 'red car', 'yellow car' and 'blue car' individually |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| suffix_keywords | sk | Denotes additional words added after main keyword while making the search query. |
| | | |
| | | The final search query would be: <keyword> <suffix keyword> |
| | | |
| | | So, for example, if the keyword is 'car' and suffix_keyword is 'red,yellow,blue', it will search and download images for |
| | | 'car red', 'car yellow' and 'car blue' individually |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| limit | l | Denotes number of images that you want to download. |
| | | |
| | | You can specify any integer value here. It will try and get all the images that it finds in the google image search page. |
| | | |
| | | If this value is not specified, it defaults to 100. |
| | | |
| | | **Note**: In case of occasional errors while downloading images, you could get less than 100 (if the limit is set to 100) |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| related_images | ri | This argument downloads a ton of images related to the keyword you provided. |
| | | |
| | | Google Images page returns list of related keywords to the keyword you have mentioned in the query. This tool downloads |
| | | images from each of those related keywords based on the limit you have mentioned in your query |
| | | |
| | | This argument does not take any value. Just add '--related_images' or '-ri' in your query. |
| | | |
| | | **Note:** This argument can download hundreds or thousands of additional images so please use this carefully. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| format | f | Denotes the format/extension of the image that you want to download. |
| | | |
| | | `Possible values: jpg, gif, png, bmp, svg, webp, ico, raw` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| color | co | Denotes the color filter that you want to apply to the images. |
| | | |
| | | `Possible values: red, orange, yellow, green, teal, blue, purple, pink, white, gray, black, brown` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| color_type | ct | Denotes the color type you want to apply to the images. |
| | | |
| | | `Possible values: full-color, black-and-white, transparent` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| usage_rights | r | Denotes the usage rights/licence under which the image is classified. |
| | | |
| | | `Possible values:` |
| | | |
| | | * `labeled-for-reuse-with-modifications`, |
| | | * `labeled-for-reuse`, |
| | | * `labeled-for-noncommercial-reuse-with-modification`, |
| | | * `labeled-for-nocommercial-reuse` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| size | s | Denotes the relative size of the image to be downloaded. |
| | | |
| | | `Possible values: large, medium, icon, >400*300, >640*480, >800*600, >1024*768, >2MP, >4MP, >6MP, >8MP, >10MP, |
| | | >12MP, >15MP, >20MP, >40MP, >70MP` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| exact_size | es | You can specify the exact size/resolution of the images |
| | | |
| | | This value of this argument can be specified as ``<integer,integer>`` where the fist integer stands for width of the image |
| | | and the second integer stands for the height of the image. For example, ``-es 1024,786`` |
| | | |
| | | **Note**: You cannot specify both 'size' and 'exact_size' arguments in the same query. You can only give one of them. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| aspect_ratio | a | Denotes the aspect ratio of images to download. |
| | | |
| | | `Possible values: tall, square, wide, panoramic` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| type | t | Denotes the type of image to be downloaded. |
| | | |
| | | `Possible values: face, photo, clip-art, line-drawing, animated` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| time | w | Denotes the time the image was uploaded/indexed. |
| | | |
| | | `Possible values: past-24-hours, past-7-days, past-month, past-year` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| time_range | wr | Denotes the time range for which you want to search the images |
| | | |
| | | The value of this parameter should be in the following format '{"time_min":"MM/DD/YYYY","time_max":"MM/DD/YYYY"}' |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| delay | d | Time to wait between downloading two images |
| | | |
| | | Time is to be specified in seconds. But you can have sub-second times by using decimal points. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| url | u | Allows you search by image when you have the URL from the Google Images page. |
| | | It downloads images from the google images link provided |
| | | |
| | | If you are searching an image on the browser google images page, simply grab the browser URL and paste it in this parameter |
| | | It will download all the images seen on that page. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| single_image | x | Allows you to download one image if the complete (absolute) URL of the image is provided |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| output_directory | o | Allows you specify the main directory name in which the images are downloaded. |
| | | |
| | | If not specified, it will default to 'downloads' directory. This directory is located in the path from where you run this code|
| | | |
| | | The directory structure would look like: ``<output_directory><image_directory><images>`` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| image_directory | i | This lets you specify a directory inside of the main directory (output_directory) in which the images will be saved |
| | | |
| | | If not specified, it will default to the name of the keyword. |
| | | |
| | | The directory structure would look like: ``<output_directory><image_directory><images>`` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| no_directory | n | This option allows you download images directly in the main directory (output_directory) without an image_directory |
| | | |
| | | The directory structure would look like: ``<output_directory><images>`` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| proxy | px | Allows you to specify proxy server setting for all your requests |
| | | |
| | | You can specify the proxy settings in 'IP:Port' format |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| similar_images | si | Reverse Image Search or 'Search by Image' as it is referred to on Google. |
| | | |
| | | Searches and downloads images that are similar to the absolute image link/url you provide. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| specific_site | ss | Allows you to download images with keywords only from a specific website/domain name you mention. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| print_urls | p | Print the URLs of the images on the console. These image URLs can be used for debugging purposes |
| | | |
| | | This argument does not take any value. Just add '--print_urls' or '-p' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| print_size | ps | Prints the size of the images on the console |
| | | |
| | | The size denoted the actual size of the image and not the size of the image on disk |
| | | |
| | | This argument does not take any value. Just add '--print_size' or '-ps' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| print_paths | pp | Prints the list of all the absolute paths of the downloaded images |
| | | |
| | | When calling the script from another python file, this list will be saved in a variable (as shown in the example below) |
| | | |
| | | This argument also allows you to print the list on the console |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| metadata | m | Prints the metada of the image on the console. |
| | | |
| | | This includes image size, origin, image attributes, description, image URL, etc. |
| | | |
| | | This argument does not take any value. Just add '--metadata' or '-m' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| extract_metadata | e | This option allows you to save metadata of all the downloaded images in a JSON file. |
| | | |
| | | This file can be found in the ``logs/`` directory. The name of the file would be same as the keyword name |
| | | |
| | | This argument does not take any value. Just add '--extract_metadata' or '-e' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| socket_timeout | st | Allows you to specify the time to wait for socket connection. |
| | | |
| | | You could specify a higher timeout time for slow internet connection. The default value is 10 seconds. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| thumbnail | th | Downloads image thumbnails corresponding to each image downloaded. |
| | | |
| | | Thumbnails are saved in their own sub-directories inside of the main directory. |
| | | |
| | | This argument does not take any value. Just add '--thumbnail' or '-th' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| thumbnail_only | tho | Downloads only thumbnails without downloading actual size images |
| | | |
| | | Thumbnails are saved in their own sub-directories inside of the main directory. |
| | | |
| | | This argument does not take any value. Just add '--thumbnail_only' or '-tho' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| language | la | Defines the language filter. The search results are automatically returned in that language |
| | | |
| | | `Possible Values: Arabic, Chinese (Simplified), Chinese (Traditional), Czech, Danish, Dutch, English, Estonian. Finnish, |
| | | French, German, Greek, Hebrew, Hungarian, Icelandic, Italian, Japanese, Korean, Latvianm, Lithuanian, Norwegian, Portuguese, |
| | | Polish, Romanian, Russian, Spanish, Swedish, Turkish` |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| prefix | pr | A word that you would want to prefix in front of actual image name. |
| | | |
| | | This feature can be used to rename files for image identification purpose. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| chromedriver | cd | With this argument you can pass the path to the 'chromedriver'. |
| | | |
| | | The path looks like this: "path/to/chromedriver". In windows it will be "C:\\path\\to\\chromedriver.exe" |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| safe_search | sa | Searches for images with the Safe Search filter On |
| | | |
| | | And this filter will be Off by default if you do not specify the safe_search argument |
| | | |
| | | This argument does not take any value. Just add '--safe_search' or '-sa' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| no_numbering | nn | When you specify this argument, the script does not add ordered numbering as prefix to the images it downloads |
| | | |
| | | If this argument is not specified, the images are numbered in order in which they are downloaded |
| | | |
| | | This argument does not take any value. Just add '--no_numbering' or '-nn' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| offset | of | When you specify this argument, it will skip the offset number of links before it starts downloading images |
| | | |
| | | If this argument is not specified, the script will start downloading form the first link until the limit is reached |
| | | |
| | | This argument takes integer. Make sure the value of this argument is less than the value of limit |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| save_source | is | Creates a text file with list of downloaded images along with their source page paths. |
| | | |
| | | This argument takes a string, name of the text file. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| no_download | nd | Print the URLs on the console without downloading images or thumbnails. These image URLs can be used for other purposes |
| | | |
| | | This argument does not take any value. Just add '--no-download' or '-nd' in your query. |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| silent_mode | sil | Remains silent. Does not print notification messages on the terminal/command prompt. |
| | | |
| | | This argument will override all the other print arguments (like print_urls, print_size, etc.) |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| ignore_urls | iu | Skip downloading of images whose urls contain certain strings such as wikipedia.org |
| | | |
| | | This argument takes a delimited set of values e.g. wikipedia.org,wikimedia.org |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
| help | h | show the help message regarding the usage of the above arguments |
+-------------------+-------------+-------------------------------------------------------------------------------------------------------------------------------+
**Note:** If ``single_image`` or ``url`` parameter is not present, then keywords is a mandatory parameter. No other parameters are mandatory.
================================================
FILE: docs/conf.py
================================================
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# http://www.sphinx-doc.org/en/master/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
version = '1.0.1'
source_suffix = '.rst'
master_doc = 'index'
html_static_path = ['_static']
def setup(app):
app.add_stylesheet('overrides.css') # may also be an URL
html_context = {
"display_github": False, # Add 'Edit on Github' link instead of 'View page source'
"last_updated": True,
"commit": False,
}
# -- Project information -----------------------------------------------------
project = 'Google Images Download'
copyright = '2019, Hardik Vasa'
author = 'Hardik Vasa'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'bizstyle'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
html_sidebars = { '**': ['globaltoc.html', 'relations.html', 'searchbox.html'] }
================================================
FILE: docs/examples.rst
================================================
========
Examples
========
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
Link to `Input arguments or parameters <https://google-images-download.readthedocs.io/en/latest/arguments.html>`__
Config File Format
==================
You can either pass the arguments directly from the command as in the examples below or you can pass it through a config file. Below is a sample of how a config
file looks.
You can pass more than one record through a config file. The below sample consist of two set of records. The code will iterate through each of the record and
download images based on arguments passed.
.. code:: json
{
"Records": [
{
"keywords": "apple",
"limit": 5,
"color": "green",
"print_urls": true
},
{
"keywords": "universe",
"limit": 15,
"size": "large",
"print_urls": true
}
]
}
Code sample - Importing the library
===================================
- If you are calling this library from another python file, below is the sample code
.. code-block:: python
from google_images_download import google_images_download #importing the library
response = google_images_download.googleimagesdownload() #class instantiation
arguments = {"keywords":"Polar bears,baloons,Beaches","limit":20,"print_urls":True} #creating list of arguments
paths = response.download(arguments) #passing the arguments to the function
print(paths) #printing absolute paths of the downloaded images
Command line examples
=====================
- If you are passing arguments from a config file, simply pass the config_file argument with name of your JSON file
.. code-block:: bash
$ googleimagesdownload -cf example.json
- Simple example of using keywords and limit arguments
.. code-block:: bash
$ googleimagesdownload --keywords "Polar bears, baloons, Beaches" --limit 20
- Using Suffix Keywords allows you to specify words after the main
keywords. For example if the ``keyword = car`` and
``suffix keyword = 'red,blue'`` then it will first search for
``car red`` and then ``car blue``
.. code-block:: bash
$ googleimagesdownload --k "car" -sk 'red,blue,white' -l 10
- To use the short hand command
.. code-block:: bash
$ googleimagesdownload -k "Polar bears, baloons, Beaches" -l 20
- To download images with specific image extension/format
.. code-block:: bash
$ googleimagesdownload --keywords "logo" --format svg
- To use color filters for the images
.. code-block:: bash
$ googleimagesdownload -k "playground" -l 20 -co red
- To use non-English keywords for image search
.. code-block:: bash
$ googleimagesdownload -k "北极熊" -l 5
- To download images from the google images link
.. code-block:: bash
$ googleimagesdownload -k "sample" -u <google images page URL>
- To save images in specific main directory (instead of in 'downloads')
.. code-block:: bash
$ googleimagesdownload -k "boat" -o "boat_new"
- To download one single image with the image URL
.. code-block:: bash
$ googleimagesdownload --keywords "baloons" --single_image <URL of the images>
- To download images with size and type constrains
.. code-block:: bash
$ googleimagesdownload --keywords "baloons" --size medium --type animated
- To download images with specific usage rights
.. code-block:: bash
$ googleimagesdownload --keywords "universe" --usage_rights labeled-for-reuse
- To download images with specific color type
.. code-block:: bash
$ googleimagesdownload --keywords "flowers" --color_type black-and-white
- To download images with specific aspect ratio
.. code-block:: bash
$ googleimagesdownload --keywords "universe" --aspect_ratio panoramic
- To download images which are similar to the image in the image URL that you provided (Reverse Image search).
.. code-block:: bash
$ googleimagesdownload -si <image url> -l 10
- To download images from specific website or domain name for a given keyword
.. code-block:: bash
$ googleimagesdownload --keywords "universe" --specific_site example.com
===> The images would be downloaded in their own sub-directories inside the main directory
(either the one you provided or in 'downloads') in the same folder you are in.
Library extensions
==================
The downloading algorithm does a good job of keeping out corrupt images. However it is not ideal. There are still some chances of getting one-off corrupt image that cannot be used for processing. Below script will help clean those corrupt image files. This script was ideated by @devajith in `Issue 81 <https://github.com/hardikvasa/google-images-download/issues/81>`__.
.. code:: python
import os
from PIL import Image
img_dir = r"path/to/downloads/directory"
for filename in os.listdir(img_dir):
try :
with Image.open(img_dir + "/" + filename) as im:
print('ok')
except :
print(img_dir + "/" + filename)
os.remove(img_dir + "/" + filename)
================================================
FILE: docs/index.rst
================================================
======================
Google Images Download
======================
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
.. index:: Summary
Summary
=======
This is a command line python program to search keywords/key-phrases on Google Images
and optionally download images to your computer. You can also invoke this script from
another python file.
This is a small and ready-to-run program. No dependencies are required to be installed
if you would only want to download up to 100 images per keyword. If you would want **more than 100
images** per keyword, then you would need to install ``Selenium`` library along with ``chromedriver``.
Detailed instructions in the troubleshooting section.
.. index:: Compatability
Compatibility
=============
This program is compatible with both the versions of python - 2.x and 3.x (recommended).
It is a download-and-run program with no changes to the file.
You will just have to specify parameters through the command line.
.. index:: Installation
Installation
============
The guide provides detailed instructions on how to install the library.
.. toctree::
:maxdepth: 3
installation
.. index:: Usage
Usage
=====
The following section provides details on using the library - from CLI or by standard imports.
.. toctree::
:maxdepth: 3
usage
.. index:: Arguments
Arguments
=========
This section provides all the arguments/parameters/options you can provide to this library.
.. toctree::
:maxdepth: 3
arguments
.. index:: Examples
Examples
========
Many examples have been provided to help new users quickly ramp up the the usage.
.. toctree::
:maxdepth: 3
examples
.. index:: Troubleshooting
Troubleshooting
===============
This section proviedes troubleshooting guide for commonly seen issues.
.. toctree::
:maxdepth: 2
troubleshooting
.. index:: Workflow
Workflow
========
Workflow showcases the algorithm used within this module to download the images.
.. toctree::
:maxdepth: 2
structure
.. index:: Contribute
Contribute
==========
Anyone is welcomed to contribute to this script.
If you would like to make a change, open a pull request.
For issues and discussion visit the
`Issue Tracker <https://github.com/hardikvasa/google-images-download/issues>`__.
The aim of this repo is to keep it simple, stand-alone, backward compatible and 3rd party dependency proof.
.. index:: Disclaimer
Disclaimer
==========
.. warning::
This program lets you download tons of images from Google.
Please do not download or use any image that violates its copyright terms.
Google Images is a search engine that merely indexes images and allows you to find them.
It does NOT produce its own images and, as such, it doesn't own copyright on any of them.
The original creators of the images own the copyrights.
Images published in the United States are automatically copyrighted by their owners,
even if they do not explicitly carry a copyright warning.
You may not reproduce copyright images without their owner's permission,
except in "fair use" cases,
or you could risk running into lawyer's warnings, cease-and-desist letters, and copyright suits.
Please be very careful before its usage!
================================================
FILE: docs/installation.rst
================================================
============
Installation
============
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
You can use **one of the below methods** to download and use this repository.
Install using pip
-----------------
.. code-block:: bash
$ pip install google_images_download
Manually install using CLI
--------------------------
.. code-block:: bash
$ git clone https://github.com/hardikvasa/google-images-download.git
$ cd google-images-download && sudo python setup.py install
Manually install using UI
-------------------------
Go to the `repo on github <https://github.com/hardikvasa/google-images-download>`__ ==> Click on 'Clone or Download' ==> Click on 'Download ZIP' and save it on your local disk.
================================================
FILE: docs/make.bat
================================================
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=_build
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
================================================
FILE: docs/structure.rst
================================================
========
Workflow
========
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
Below diagram represents the algorithm logic to download images.
.. figure:: http://www.zseries.in/flow-chart.png
:alt:
================================================
FILE: docs/troubleshooting.rst
================================================
=============================
Troubleshooting Errors/Issues
=============================
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
SSL Errors
==========
If you do see SSL errors on Mac for Python 3,
please go to Finder —> Applications —> Python 3 —> Click on the ‘Install Certificates.command’
and run the file.
googleimagesdownload: command not found
=======================================
While using the above commands, if you get ``Error: -bash: googleimagesdownload: command not found`` then you have to set the correct path variable.
To get the details of the repo, run the following command:
.. code-block:: bash
$ pip show -f google_images_download
you will get the result like this:
.. code-block:: bash
Location: /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages
Files:
../../../bin/googleimagesdownload
together they make: ``/Library/Frameworks/Python.framework/Versions/2.7/bin`` which you need add it to the path:
.. code-block:: bash
$ export PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin"
[Errno 13] Permission denied creating directory 'downloads'
===========================================================
When you run the command, it downloads the images in the current directory (the directory from where you are running the command). If you get permission denied error for creating the `downloads directory`, then move to a directory in which you have the write permission and then run the command again.
Permission denied while installing the library
==============================================
On MAC and Linux, when you get permission denied when installing the library using pip, try doing a user install.
.. code-block:: bash
$ pip install google_images_download --user
You can also run pip install as a superuser with ``sudo pip install google_images_download`` but it is not generally a good idea because it can cause issues with your system-level packages.
Installing the chromedriver (with Selenium)
===========================================
If you would want to download more than 100 images per keyword, then you will need to install 'selenium' library along with 'chromedriver' extension.
If you have pip-installed the library or had run the setup.py file, Selenium would have automatically installed on your machine. You will also need Chrome browser on your machine. For chromedriver:
`Download the correct chromedriver <https://sites.google.com/a/chromium.org/chromedriver/downloads>`__ based on your operating system.
On **Windows** or **MAC** if for some reason the chromedriver gives you trouble, download it under the current directory and run the command.
On windows however, the path to chromedriver has to be given in the following format:
``C:\\complete\\path\\to\\chromedriver.exe``
On **Linux** if you are having issues installing google chrome browser, refer to this `CentOS or Amazon Linux Guide <https://intoli.com/blog/installing-google-chrome-on-centos/>`__
or `Ubuntu Guide <https://askubuntu.com/questions/510056/how-to-install-google-chrome in documentation>`__
For **All the operating systems** you will have to use '--chromedriver' or '-cd' argument to specify the path of
chromedriver that you have downloaded in your machine.
If on any rare occasion the chromedriver does not work for you, try downgrading it to a lower version.
urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
==============================================
`Reference to this issue <https://github.com/hardikvasa/google-images-download/issues/140>`__
Use the below command to install the SSL certificate on your machine.
.. code-block:: bash
cd /Applications/Python\ 3.7/
./Install\ Certificates.command
================================================
FILE: docs/usage.rst
================================================
=====
Usage
=====
Link to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__
Link to `Documentation Homepage <https://google-images-download.readthedocs.io/en/latest/index.html>`__
Using the library from Command Line Interface
=============================================
If installed via pip or using CLI, use the following command:
.. code-block:: bash
$ googleimagesdownload [Arguments...]
If downloaded via the UI, unzip the file downloaded, go to the 'google_images_download' directory and use one of the below commands:
.. code-block:: bash
$ python3 google_images_download.py [Arguments...]
OR
$ python google_images_download.py [Arguments...]
Using the library from another python file
==========================================
If you would want to use this library from another python file, you could use it as shown below:
.. code-block:: python
from google_images_download import google_images_download
response = google_images_download.googleimagesdownload()
absolute_image_paths = response.download({<Arguments...>})
================================================
FILE: google_images_download/__init__.py
================================================
#!/usr/bin/env python
from __future__ import absolute_import
def main():
import google_images_download.google_images_download
if __name__ == '__main__':
main()
================================================
FILE: google_images_download/__main__.py
================================================
#!/usr/bin/env python
from __future__ import absolute_import
from .__init__ import main
if __name__ == '__main__':
main()
================================================
FILE: google_images_download/google_images_download.py
================================================
#!/usr/bin/env python
# In[ ]:
# coding: utf-8
###### Searching and Downloading Google Images to the local disk ######
# Import Libraries
import sys
version = (3, 0)
cur_version = sys.version_info
if cur_version >= version: # If the Current Version of Python is 3.0 or above
import urllib.request
from urllib.request import Request, urlopen
from urllib.request import URLError, HTTPError
from urllib.parse import quote
import http.client
from http.client import IncompleteRead, BadStatusLine
http.client._MAXHEADERS = 1000
else: # If the Current Version of Python is 2.x
import urllib2
from urllib2 import Request, urlopen
from urllib2 import URLError, HTTPError
from urllib import quote
import httplib
from httplib import IncompleteRead, BadStatusLine
httplib._MAXHEADERS = 1000
import time # Importing the time library to check the time of code execution
import os
import argparse
import ssl
import datetime
import json
import re
import codecs
import socket
args_list = ["keywords", "keywords_from_file", "prefix_keywords", "suffix_keywords",
"limit", "format", "color", "color_type", "usage_rights", "size",
"exact_size", "aspect_ratio", "type", "time", "time_range", "delay", "url", "single_image",
"output_directory", "image_directory", "no_directory", "proxy", "similar_images", "specific_site",
"print_urls", "print_size", "print_paths", "metadata", "extract_metadata", "socket_timeout",
"thumbnail", "thumbnail_only", "language", "prefix", "chromedriver", "related_images", "safe_search", "no_numbering",
"offset", "no_download","save_source","silent_mode","ignore_urls"]
def user_input():
config = argparse.ArgumentParser()
config.add_argument('-cf', '--config_file', help='config file name', default='', type=str, required=False)
config_file_check = config.parse_known_args()
object_check = vars(config_file_check[0])
if object_check['config_file'] != '':
records = []
json_file = json.load(open(config_file_check[0].config_file))
for record in range(0,len(json_file['Records'])):
arguments = {}
for i in args_list:
arguments[i] = None
for key, value in json_file['Records'][record].items():
arguments[key] = value
records.append(arguments)
records_count = len(records)
else:
# Taking command line arguments from users
parser = argparse.ArgumentParser()
parser.add_argument('-k', '--keywords', help='delimited list input', type=str, required=False)
parser.add_argument('-kf', '--keywords_from_file', help='extract list of keywords from a text file', type=str, required=False)
parser.add_argument('-sk', '--suffix_keywords', help='comma separated additional words added after to main keyword', type=str, required=False)
parser.add_argument('-pk', '--prefix_keywords', help='comma separated additional words added before main keyword', type=str, required=False)
parser.add_argument('-l', '--limit', help='delimited list input', type=str, required=False)
parser.add_argument('-f', '--format', help='download images with specific format', type=str, required=False,
choices=['jpg', 'gif', 'png', 'bmp', 'svg', 'webp', 'ico'])
parser.add_argument('-u', '--url', help='search with google image URL', type=str, required=False)
parser.add_argument('-x', '--single_image', help='downloading a single image from URL', type=str, required=False)
parser.add_argument('-o', '--output_directory', help='download images in a specific main directory', type=str, required=False)
parser.add_argument('-i', '--image_directory', help='download images in a specific sub-directory', type=str, required=False)
parser.add_argument('-n', '--no_directory', default=False, help='download images in the main directory but no sub-directory', action="store_true")
parser.add_argument('-d', '--delay', help='delay in seconds to wait between downloading two images', type=int, required=False)
parser.add_argument('-co', '--color', help='filter on color', type=str, required=False,
choices=['red', 'orange', 'yellow', 'green', 'teal', 'blue', 'purple', 'pink', 'white', 'gray', 'black', 'brown'])
parser.add_argument('-ct', '--color_type', help='filter on color', type=str, required=False,
choices=['full-color', 'black-and-white', 'transparent'])
parser.add_argument('-r', '--usage_rights', help='usage rights', type=str, required=False,
choices=['labeled-for-reuse-with-modifications','labeled-for-reuse','labeled-for-noncommercial-reuse-with-modification','labeled-for-nocommercial-reuse'])
parser.add_argument('-s', '--size', help='image size', type=str, required=False,
choices=['large','medium','icon','>400*300','>640*480','>800*600','>1024*768','>2MP','>4MP','>6MP','>8MP','>10MP','>12MP','>15MP','>20MP','>40MP','>70MP'])
parser.add_argument('-es', '--exact_size', help='exact image resolution "WIDTH,HEIGHT"', type=str, required=False)
parser.add_argument('-t', '--type', help='image type', type=str, required=False,
choices=['face','photo','clipart','line-drawing','animated'])
parser.add_argument('-w', '--time', help='image age', type=str, required=False,
choices=['past-24-hours','past-7-days','past-month','past-year'])
parser.add_argument('-wr', '--time_range', help='time range for the age of the image. should be in the format {"time_min":"MM/DD/YYYY","time_max":"MM/DD/YYYY"}', type=str, required=False)
parser.add_argument('-a', '--aspect_ratio', help='comma separated additional words added to keywords', type=str, required=False,
choices=['tall', 'square', 'wide', 'panoramic'])
parser.add_argument('-si', '--similar_images', help='downloads images very similar to the image URL you provide', type=str, required=False)
parser.add_argument('-ss', '--specific_site', help='downloads images that are indexed from a specific website', type=str, required=False)
parser.add_argument('-p', '--print_urls', default=False, help="Print the URLs of the images", action="store_true")
parser.add_argument('-ps', '--print_size', default=False, help="Print the size of the images on disk", action="store_true")
parser.add_argument('-pp', '--print_paths', default=False, help="Prints the list of absolute paths of the images",action="store_true")
parser.add_argument('-m', '--metadata', default=False, help="Print the metadata of the image", action="store_true")
parser.add_argument('-e', '--extract_metadata', default=False, help="Dumps all the logs into a text file", action="store_true")
parser.add_argument('-st', '--socket_timeout', default=False, help="Connection timeout waiting for the image to download", type=float)
parser.add_argument('-th', '--thumbnail', default=False, help="Downloads image thumbnail along with the actual image", action="store_true")
parser.add_argument('-tho', '--thumbnail_only', default=False, help="Downloads only thumbnail without downloading actual images", action="store_true")
parser.add_argument('-la', '--language', default=False, help="Defines the language filter. The search results are authomatically returned in that language", type=str, required=False,
choices=['Arabic','Chinese (Simplified)','Chinese (Traditional)','Czech','Danish','Dutch','English','Estonian','Finnish','French','German','Greek','Hebrew','Hungarian','Icelandic','Italian','Japanese','Korean','Latvian','Lithuanian','Norwegian','Portuguese','Polish','Romanian','Russian','Spanish','Swedish','Turkish'])
parser.add_argument('-pr', '--prefix', default=False, help="A word that you would want to prefix in front of each image name", type=str, required=False)
parser.add_argument('-px', '--proxy', help='specify a proxy address and port', type=str, required=False)
parser.add_argument('-cd', '--chromedriver', help='specify the path to chromedriver executable in your local machine', type=str, required=False)
parser.add_argument('-ri', '--related_images', default=False, help="Downloads images that are similar to the keyword provided", action="store_true")
parser.add_argument('-sa', '--safe_search', default=False, help="Turns on the safe search filter while searching for images", action="store_true")
parser.add_argument('-nn', '--no_numbering', default=False, help="Allows you to exclude the default numbering of images", action="store_true")
parser.add_argument('-of', '--offset', help="Where to start in the fetched links", type=str, required=False)
parser.add_argument('-nd', '--no_download', default=False, help="Prints the URLs of the images and/or thumbnails without downloading them", action="store_true")
parser.add_argument('-iu', '--ignore_urls', default=False, help="delimited list input of image urls/keywords to ignore", type=str)
parser.add_argument('-sil', '--silent_mode', default=False, help="Remains silent. Does not print notification messages on the terminal", action="store_true")
parser.add_argument('-is', '--save_source', help="creates a text file containing a list of downloaded images along with source page url", type=str, required=False)
args = parser.parse_args()
arguments = vars(args)
records = []
records.append(arguments)
return records
class googleimagesdownload:
def __init__(self):
pass
# Downloading entire Web Document (Raw Page Content)
def download_page(self,url):
version = (3, 0)
cur_version = sys.version_info
if cur_version >= version: # If the Current Version of Python is 3.0 or above
try:
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
req = urllib.request.Request(url, headers=headers)
resp = urllib.request.urlopen(req)
respData = str(resp.read())
return respData
except Exception as e:
print("Could not open URL. Please check your internet connection and/or ssl settings \n"
"If you are using proxy, make sure your proxy settings is configured correctly")
sys.exit()
else: # If the Current Version of Python is 2.x
try:
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
req = urllib2.Request(url, headers=headers)
try:
response = urllib2.urlopen(req)
except URLError: # Handling SSL certificate failed
context = ssl._create_unverified_context()
response = urlopen(req, context=context)
page = response.read()
return page
except:
print("Could not open URL. Please check your internet connection and/or ssl settings \n"
"If you are using proxy, make sure your proxy settings is configured correctly")
sys.exit()
return "Page Not found"
# Download Page for more than 100 images
def download_extended_page(self,url,chromedriver):
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
if sys.version_info[0] < 3:
reload(sys)
sys.setdefaultencoding('utf8')
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument("--headless")
try:
browser = webdriver.Chrome(chromedriver, chrome_options=options)
except Exception as e:
print("Looks like we cannot locate the path the 'chromedriver' (use the '--chromedriver' "
"argument to specify the path to the executable.) or google chrome browser is not "
"installed on your machine (exception: %s)" % e)
sys.exit()
browser.set_window_size(1024, 768)
# Open the link
browser.get(url)
time.sleep(1)
print("Getting you a lot of images. This may take a few moments...")
element = browser.find_element_by_tag_name("body")
# Scroll down
for i in range(30):
element.send_keys(Keys.PAGE_DOWN)
time.sleep(0.3)
try:
browser.find_element_by_id("smb").click()
for i in range(50):
element.send_keys(Keys.PAGE_DOWN)
time.sleep(0.3) # bot id protection
except:
for i in range(10):
element.send_keys(Keys.PAGE_DOWN)
time.sleep(0.3) # bot id protection
print("Reached end of Page.")
time.sleep(0.5)
source = browser.page_source #page source
#close the browser
browser.close()
return source
#Correcting the escape characters for python2
def replace_with_byte(self,match):
return chr(int(match.group(0)[1:], 8))
def repair(self,brokenjson):
invalid_escape = re.compile(r'\\[0-7]{1,3}') # up to 3 digits for byte values up to FF
return invalid_escape.sub(self.replace_with_byte, brokenjson)
# Finding 'Next Image' from the given raw page
def get_next_tab(self,s):
start_line = s.find('class="dtviD"')
if start_line == -1: # If no links are found then give an error!
end_quote = 0
link = "no_tabs"
return link,'',end_quote
else:
start_line = s.find('class="dtviD"')
start_content = s.find('href="', start_line + 1)
end_content = s.find('">', start_content + 1)
url_item = "https://www.google.com" + str(s[start_content + 6:end_content])
url_item = url_item.replace('&', '&')
start_line_2 = s.find('class="dtviD"')
s = s.replace('&', '&')
start_content_2 = s.find(':', start_line_2 + 1)
end_content_2 = s.find('&usg=', start_content_2 + 1)
url_item_name = str(s[start_content_2 + 1:end_content_2])
chars = url_item_name.find(',g_1:')
chars_end = url_item_name.find(":", chars + 6)
if chars_end == -1:
updated_item_name = (url_item_name[chars + 5:]).replace("+", " ")
else:
updated_item_name = (url_item_name[chars+5:chars_end]).replace("+", " ")
return url_item, updated_item_name, end_content
# Getting all links with the help of '_images_get_next_image'
def get_all_tabs(self,page):
tabs = {}
while True:
item,item_name,end_content = self.get_next_tab(page)
if item == "no_tabs":
break
else:
if len(item_name) > 100 or item_name == "background-color":
break
else:
tabs[item_name] = item # Append all the links in the list named 'Links'
time.sleep(0.1) # Timer could be used to slow down the request for image downloads
page = page[end_content:]
return tabs
#Format the object in readable format
def format_object(self,object):
formatted_object = {}
formatted_object['image_format'] = object['ity']
formatted_object['image_height'] = object['oh']
formatted_object['image_width'] = object['ow']
formatted_object['image_link'] = object['ou']
formatted_object['image_description'] = object['pt']
formatted_object['image_host'] = object['rh']
formatted_object['image_source'] = object['ru']
formatted_object['image_thumbnail_url'] = object['tu']
return formatted_object
#function to download single image
def single_image(self,image_url):
main_directory = "downloads"
extensions = (".jpg", ".gif", ".png", ".bmp", ".svg", ".webp", ".ico")
url = image_url
try:
os.makedirs(main_directory)
except OSError as e:
if e.errno != 17:
raise
pass
req = Request(url, headers={
"User-Agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"})
response = urlopen(req, None, 10)
data = response.read()
response.close()
image_name = str(url[(url.rfind('/')) + 1:])
if '?' in image_name:
image_name = image_name[:image_name.find('?')]
# if ".jpg" in image_name or ".gif" in image_name or ".png" in image_name or ".bmp" in image_name or ".svg" in image_name or ".webp" in image_name or ".ico" in image_name:
if any(map(lambda extension: extension in image_name, extensions)):
file_name = main_directory + "/" + image_name
else:
file_name = main_directory + "/" + image_name + ".jpg"
image_name = image_name + ".jpg"
try:
output_file = open(file_name, 'wb')
output_file.write(data)
output_file.close()
except IOError as e:
raise e
except OSError as e:
raise e
print("completed ====> " + image_name.encode('raw_unicode_escape').decode('utf-8'))
return
def similar_images(self,similar_images):
version = (3, 0)
cur_version = sys.version_info
if cur_version >= version: # If the Current Version of Python is 3.0 or above
try:
searchUrl = 'https://www.google.com/searchbyimage?site=search&sa=X&image_url=' + similar_images
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36"
req1 = urllib.request.Request(searchUrl, headers=headers)
resp1 = urllib.request.urlopen(req1)
content = str(resp1.read())
l1 = content.find('AMhZZ')
l2 = content.find('&', l1)
urll = content[l1:l2]
newurl = "https://www.google.com/search?tbs=sbi:" + urll + "&site=search&sa=X"
req2 = urllib.request.Request(newurl, headers=headers)
resp2 = urllib.request.urlopen(req2)
l3 = content.find('/search?sa=X&q=')
l4 = content.find(';', l3 + 19)
urll2 = content[l3 + 19:l4]
return urll2
except:
return "Cloud not connect to Google Images endpoint"
else: # If the Current Version of Python is 2.x
try:
searchUrl = 'https://www.google.com/searchbyimage?site=search&sa=X&image_url=' + similar_images
headers = {}
headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"
req1 = urllib2.Request(searchUrl, headers=headers)
resp1 = urllib2.urlopen(req1)
content = str(resp1.read())
l1 = content.find('AMhZZ')
l2 = content.find('&', l1)
urll = content[l1:l2]
newurl = "https://www.google.com/search?tbs=sbi:" + urll + "&site=search&sa=X"
req2 = urllib2.Request(newurl, headers=headers)
resp2 = urllib2.urlopen(req2)
l3 = content.find('/search?sa=X&q=')
l4 = content.find(';', l3 + 19)
urll2 = content[l3 + 19:l4]
return(urll2)
except:
return "Cloud not connect to Google Images endpoint"
#Building URL parameters
def build_url_parameters(self,arguments):
if arguments['language']:
lang = "&lr="
lang_param = {"Arabic":"lang_ar","Chinese (Simplified)":"lang_zh-CN","Chinese (Traditional)":"lang_zh-TW","Czech":"lang_cs","Danish":"lang_da","Dutch":"lang_nl","English":"lang_en","Estonian":"lang_et","Finnish":"lang_fi","French":"lang_fr","German":"lang_de","Greek":"lang_el","Hebrew":"lang_iw ","Hungarian":"lang_hu","Icelandic":"lang_is","Italian":"lang_it","Japanese":"lang_ja","Korean":"lang_ko","Latvian":"lang_lv","Lithuanian":"lang_lt","Norwegian":"lang_no","Portuguese":"lang_pt","Polish":"lang_pl","Romanian":"lang_ro","Russian":"lang_ru","Spanish":"lang_es","Swedish":"lang_sv","Turkish":"lang_tr"}
lang_url = lang+lang_param[arguments['language']]
else:
lang_url = ''
if arguments['time_range']:
json_acceptable_string = arguments['time_range'].replace("'", "\"")
d = json.loads(json_acceptable_string)
time_range = ',cdr:1,cd_min:' + d['time_min'] + ',cd_max:' + d['time_max']
else:
time_range = ''
if arguments['exact_size']:
size_array = [x.strip() for x in arguments['exact_size'].split(',')]
exact_size = ",isz:ex,iszw:" + str(size_array[0]) + ",iszh:" + str(size_array[1])
else:
exact_size = ''
built_url = "&tbs="
counter = 0
params = {'color':[arguments['color'],{'red':'ic:specific,isc:red', 'orange':'ic:specific,isc:orange', 'yellow':'ic:specific,isc:yellow', 'green':'ic:specific,isc:green', 'teal':'ic:specific,isc:teel', 'blue':'ic:specific,isc:blue', 'purple':'ic:specific,isc:purple', 'pink':'ic:specific,isc:pink', 'white':'ic:specific,isc:white', 'gray':'ic:specific,isc:gray', 'black':'ic:specific,isc:black', 'brown':'ic:specific,isc:brown'}],
'color_type':[arguments['color_type'],{'full-color':'ic:color', 'black-and-white':'ic:gray','transparent':'ic:trans'}],
'usage_rights':[arguments['usage_rights'],{'labeled-for-reuse-with-modifications':'sur:fmc','labeled-for-reuse':'sur:fc','labeled-for-noncommercial-reuse-with-modification':'sur:fm','labeled-for-nocommercial-reuse':'sur:f'}],
'size':[arguments['size'],{'large':'isz:l','medium':'isz:m','icon':'isz:i','>400*300':'isz:lt,islt:qsvga','>640*480':'isz:lt,islt:vga','>800*600':'isz:lt,islt:svga','>1024*768':'visz:lt,islt:xga','>2MP':'isz:lt,islt:2mp','>4MP':'isz:lt,islt:4mp','>6MP':'isz:lt,islt:6mp','>8MP':'isz:lt,islt:8mp','>10MP':'isz:lt,islt:10mp','>12MP':'isz:lt,islt:12mp','>15MP':'isz:lt,islt:15mp','>20MP':'isz:lt,islt:20mp','>40MP':'isz:lt,islt:40mp','>70MP':'isz:lt,islt:70mp'}],
'type':[arguments['type'],{'face':'itp:face','photo':'itp:photo','clipart':'itp:clipart','line-drawing':'itp:lineart','animated':'itp:animated'}],
'time':[arguments['time'],{'past-24-hours':'qdr:d','past-7-days':'qdr:w','past-month':'qdr:m','past-year':'qdr:y'}],
'aspect_ratio':[arguments['aspect_ratio'],{'tall':'iar:t','square':'iar:s','wide':'iar:w','panoramic':'iar:xw'}],
'format':[arguments['format'],{'jpg':'ift:jpg','gif':'ift:gif','png':'ift:png','bmp':'ift:bmp','svg':'ift:svg','webp':'webp','ico':'ift:ico','raw':'ift:craw'}]}
for key, value in params.items():
if value[0] is not None:
ext_param = value[1][value[0]]
# counter will tell if it is first param added or not
if counter == 0:
# add it to the built url
built_url = built_url + ext_param
counter += 1
else:
built_url = built_url + ',' + ext_param
counter += 1
built_url = lang_url+built_url+exact_size+time_range
return built_url
#building main search URL
def build_search_url(self,search_term,params,url,similar_images,specific_site,safe_search):
#check safe_search
safe_search_string = "&safe=active"
# check the args and choose the URL
if url:
url = url
elif similar_images:
print(similar_images)
keywordem = self.similar_images(similar_images)
url = 'https://www.google.com/search?q=' + keywordem + '&espv=2&biw=1366&bih=667&site=webhp&source=lnms&tbm=isch&sa=X&ei=XosDVaCXD8TasATItgE&ved=0CAcQ_AUoAg'
elif specific_site:
url = 'https://www.google.com/search?q=' + quote(
search_term.encode('utf-8')) + '&as_sitesearch=' + specific_site + '&espv=2&biw=1366&bih=667&site=webhp&source=lnms&tbm=isch' + params + '&sa=X&ei=XosDVaCXD8TasATItgE&ved=0CAcQ_AUoAg'
else:
url = 'https://www.google.com/search?q=' + quote(
search_term.encode('utf-8')) + '&espv=2&biw=1366&bih=667&site=webhp&source=lnms&tbm=isch' + params + '&sa=X&ei=XosDVaCXD8TasATItgE&ved=0CAcQ_AUoAg'
#safe search check
if safe_search:
url = url + safe_search_string
return url
#measures the file size
def file_size(self,file_path):
if os.path.isfile(file_path):
file_info = os.stat(file_path)
size = file_info.st_size
for x in ['bytes', 'KB', 'MB', 'GB', 'TB']:
if size < 1024.0:
return "%3.1f %s" % (size, x)
size /= 1024.0
return size
#keywords from file
def keywords_from_file(self,file_name):
search_keyword = []
with codecs.open(file_name, 'r', encoding='utf-8-sig') as f:
if '.csv' in file_name:
for line in f:
if line in ['\n', '\r\n']:
pass
else:
search_keyword.append(line.replace('\n', '').replace('\r', ''))
elif '.txt' in file_name:
for line in f:
if line in ['\n', '\r\n']:
pass
else:
search_keyword.append(line.replace('\n', '').replace('\r', ''))
else:
print("Invalid file type: Valid file types are either .txt or .csv \n"
"exiting...")
sys.exit()
return search_keyword
# make directories
def create_directories(self,main_directory, dir_name,thumbnail,thumbnail_only):
dir_name_thumbnail = dir_name + " - thumbnail"
# make a search keyword directory
try:
if not os.path.exists(main_directory):
os.makedirs(main_directory)
time.sleep(0.2)
path = (dir_name)
sub_directory = os.path.join(main_directory, path)
if not os.path.exists(sub_directory):
os.makedirs(sub_directory)
if thumbnail or thumbnail_only:
sub_directory_thumbnail = os.path.join(main_directory, dir_name_thumbnail)
if not os.path.exists(sub_directory_thumbnail):
os.makedirs(sub_directory_thumbnail)
else:
path = (dir_name)
sub_directory = os.path.join(main_directory, path)
if not os.path.exists(sub_directory):
os.makedirs(sub_directory)
if thumbnail or thumbnail_only:
sub_directory_thumbnail = os.path.join(main_directory, dir_name_thumbnail)
if not os.path.exists(sub_directory_thumbnail):
os.makedirs(sub_directory_thumbnail)
except OSError as e:
if e.errno != 17:
raise
pass
return
# Download Image thumbnails
def download_image_thumbnail(self,image_url,main_directory,dir_name,return_image_name,print_urls,socket_timeout,print_size,no_download,save_source,img_src,ignore_urls):
if print_urls or no_download:
print("Image URL: " + image_url)
if no_download:
return "success","Printed url without downloading"
try:
req = Request(image_url, headers={
"User-Agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"})
try:
# timeout time to download an image
if socket_timeout:
timeout = float(socket_timeout)
else:
timeout = 10
response = urlopen(req, None, timeout)
data = response.read()
response.close()
path = main_directory + "/" + dir_name + " - thumbnail" + "/" + return_image_name
try:
output_file = open(path, 'wb')
output_file.write(data)
output_file.close()
if save_source:
list_path = main_directory + "/" + save_source + ".txt"
list_file = open(list_path,'a')
list_file.write(path + '\t' + img_src + '\n')
list_file.close()
except OSError as e:
download_status = 'fail'
download_message = "OSError on an image...trying next one..." + " Error: " + str(e)
except IOError as e:
download_status = 'fail'
download_message = "IOError on an image...trying next one..." + " Error: " + str(e)
download_status = 'success'
download_message = "Completed Image Thumbnail ====> " + return_image_name
# image size parameter
if print_size:
print("Image Size: " + str(self.file_size(path)))
except UnicodeEncodeError as e:
download_status = 'fail'
download_message = "UnicodeEncodeError on an image...trying next one..." + " Error: " + str(e)
except HTTPError as e: # If there is any HTTPError
download_status = 'fail'
download_message = "HTTPError on an image...trying next one..." + " Error: " + str(e)
except URLError as e:
download_status = 'fail'
download_message = "URLError on an image...trying next one..." + " Error: " + str(e)
except ssl.CertificateError as e:
download_status = 'fail'
download_message = "CertificateError on an image...trying next one..." + " Error: " + str(e)
except IOError as e: # If there is any IOError
download_status = 'fail'
download_message = "IOError on an image...trying next one..." + " Error: " + str(e)
return download_status, download_message
# Download Images
def download_image(self,image_url,image_format,main_directory,dir_name,count,print_urls,socket_timeout,prefix,print_size,no_numbering,no_download,save_source,img_src,silent_mode,thumbnail_only,format,ignore_urls):
if not silent_mode:
if print_urls or no_download:
print("Image URL: " + image_url)
if ignore_urls:
if any(url in image_url for url in ignore_urls.split(',')):
return "fail", "Image ignored due to 'ignore url' parameter", None, image_url
if thumbnail_only:
return "success", "Skipping image download...", str(image_url[(image_url.rfind('/')) + 1:]), image_url
if no_download:
return "success","Printed url without downloading",None,image_url
try:
req = Request(image_url, headers={
"User-Agent": "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17"})
try:
# timeout time to download an image
if socket_timeout:
timeout = float(socket_timeout)
else:
timeout = 10
response = urlopen(req, None, timeout)
data = response.read()
response.close()
extensions = [".jpg", ".jpeg", ".gif", ".png", ".bmp", ".svg", ".webp", ".ico"]
# keep everything after the last '/'
image_name = str(image_url[(image_url.rfind('/')) + 1:])
if format:
if not image_format or image_format != format:
download_status = 'fail'
download_message = "Wrong image format returned. Skipping..."
return_image_name = ''
absolute_path = ''
return download_status, download_message, return_image_name, absolute_path
if image_format == "" or not image_format or "." + image_format not in extensions:
download_status = 'fail'
download_message = "Invalid or missing image format. Skipping..."
return_image_name = ''
absolute_path = ''
return download_status, download_message, return_image_name, absolute_path
elif image_name.lower().find("." + image_format) < 0:
image_name = image_name + "." + image_format
else:
image_name = image_name[:image_name.lower().find("." + image_format) + (len(image_format) + 1)]
# prefix name in image
if prefix:
prefix = prefix + " "
else:
prefix = ''
if no_numbering:
path = main_directory + "/" + dir_name + "/" + prefix + image_name
else:
path = main_directory + "/" + dir_name + "/" + prefix + str(count) + "." + image_name
try:
output_file = open(path, 'wb')
output_file.write(data)
output_file.close()
if save_source:
list_path = main_directory + "/" + save_source + ".txt"
list_file = open(list_path,'a')
list_file.write(path + '\t' + img_src + '\n')
list_file.close()
absolute_path = os.path.abspath(path)
except OSError as e:
download_status = 'fail'
download_message = "OSError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
#return image name back to calling method to use it for thumbnail downloads
download_status = 'success'
download_message = "Completed Image ====> " + prefix + str(count) + "." + image_name
return_image_name = prefix + str(count) + "." + image_name
# image size parameter
if not silent_mode:
if print_size:
print("Image Size: " + str(self.file_size(path)))
except UnicodeEncodeError as e:
download_status = 'fail'
download_message = "UnicodeEncodeError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except URLError as e:
download_status = 'fail'
download_message = "URLError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except BadStatusLine as e:
download_status = 'fail'
download_message = "BadStatusLine on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except HTTPError as e: # If there is any HTTPError
download_status = 'fail'
download_message = "HTTPError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except URLError as e:
download_status = 'fail'
download_message = "URLError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except ssl.CertificateError as e:
download_status = 'fail'
download_message = "CertificateError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except IOError as e: # If there is any IOError
download_status = 'fail'
download_message = "IOError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
except IncompleteRead as e:
download_status = 'fail'
download_message = "IncompleteReadError on an image...trying next one..." + " Error: " + str(e)
return_image_name = ''
absolute_path = ''
return download_status,download_message,return_image_name,absolute_path
# Finding 'Next Image' from the given raw page
def _get_next_item(self,s):
start_line = s.find('rg_meta notranslate')
if start_line == -1: # If no links are found then give an error!
end_quote = 0
link = "no_links"
return link, end_quote
else:
start_line = s.find('class="rg_meta notranslate">')
start_object = s.find('{', start_line + 1)
end_object = s.find('</div>', start_object + 1)
object_raw = str(s[start_object:end_object])
#remove escape characters based on python version
version = (3, 0)
cur_version = sys.version_info
if cur_version >= version: #python3
try:
object_decode = bytes(object_raw, "utf-8").decode("unicode_escape")
final_object = json.loads(object_decode)
except:
final_object = ""
else: #python2
try:
final_object = (json.loads(self.repair(object_raw)))
except:
final_object = ""
return final_object, end_object
# Getting all links with the help of '_images_get_next_image'
def _get_all_items(self,page,main_directory,dir_name,limit,arguments):
items = []
abs_path = []
errorCount = 0
i = 0
count = 1
while count < limit+1:
object, end_content = self._get_next_item(page)
if object == "no_links":
break
elif object == "":
page = page[end_content:]
elif arguments['offset'] and count < int(arguments['offset']):
count += 1
page = page[end_content:]
else:
#format the item for readability
object = self.format_object(object)
if arguments['metadata']:
if not arguments["silent_mode"]:
print("\nImage Metadata: " + str(object))
#download the images
download_status,download_message,return_image_name,absolute_path = self.download_image(object['image_link'],object['image_format'],main_directory,dir_name,count,arguments['print_urls'],arguments['socket_timeout'],arguments['prefix'],arguments['print_size'],arguments['no_numbering'],arguments['no_download'],arguments['save_source'],object['image_source'],arguments["silent_mode"],arguments["thumbnail_only"],arguments['format'],arguments['ignore_urls'])
if not arguments["silent_mode"]:
print(download_message)
if download_status == "success":
# download image_thumbnails
if arguments['thumbnail'] or arguments["thumbnail_only"]:
download_status, download_message_thumbnail = self.download_image_thumbnail(object['image_thumbnail_url'],main_directory,dir_name,return_image_name,arguments['print_urls'],arguments['socket_timeout'],arguments['print_size'],arguments['no_download'],arguments['save_source'],object['image_source'],arguments['ignore_urls'])
if not arguments["silent_mode"]:
print(download_message_thumbnail)
count += 1
object['image_filename'] = return_image_name
items.append(object) # Append all the links in the list named 'Links'
abs_path.append(absolute_path)
else:
errorCount += 1
#delay param
if arguments['delay']:
time.sleep(int(arguments['delay']))
page = page[end_content:]
i += 1
if count < limit:
print("\n\nUnfortunately all " + str(
limit) + " could not be downloaded because some images were not downloadable. " + str(
count-1) + " is all we got for this search filter!")
return items,errorCount,abs_path
# Bulk Download
def download(self,arguments):
paths_agg = {}
# for input coming from other python files
if __name__ != "__main__":
# if the calling file contains config_file param
if 'config_file' in arguments:
records = []
json_file = json.load(open(arguments['config_file']))
for record in range(0, len(json_file['Records'])):
arguments = {}
for i in args_list:
arguments[i] = None
for key, value in json_file['Records'][record].items():
arguments[key] = value
records.append(arguments)
total_errors = 0
for rec in records:
paths, errors = self.download_executor(rec)
for i in paths:
paths_agg[i] = paths[i]
if not arguments["silent_mode"]:
if arguments['print_paths']:
print(paths.encode('raw_unicode_escape').decode('utf-8'))
total_errors = total_errors + errors
return paths_agg,total_errors
# if the calling file contains params directly
else:
paths, errors = self.download_executor(arguments)
for i in paths:
paths_agg[i] = paths[i]
if not arguments["silent_mode"]:
if arguments['print_paths']:
print(paths.encode('raw_unicode_escape').decode('utf-8'))
return paths_agg, errors
# for input coming from CLI
else:
paths, errors = self.download_executor(arguments)
for i in paths:
paths_agg[i] = paths[i]
if not arguments["silent_mode"]:
if arguments['print_paths']:
print(paths.encode('raw_unicode_escape').decode('utf-8'))
return paths_agg, errors
def download_executor(self,arguments):
paths = {}
errorCount = None
for arg in args_list:
if arg not in arguments:
arguments[arg] = None
######Initialization and Validation of user arguments
if arguments['keywords']:
search_keyword = [str(item) for item in arguments['keywords'].split(',')]
if arguments['keywords_from_file']:
search_keyword = self.keywords_from_file(arguments['keywords_from_file'])
# both time and time range should not be allowed in the same query
if arguments['time'] and arguments['time_range']:
raise ValueError('Either time or time range should be used in a query. Both cannot be used at the same time.')
# both time and time range should not be allowed in the same query
if arguments['size'] and arguments['exact_size']:
raise ValueError('Either "size" or "exact_size" should be used in a query. Both cannot be used at the same time.')
# both image directory and no image directory should not be allowed in the same query
if arguments['image_directory'] and arguments['no_directory']:
raise ValueError('You can either specify image directory or specify no image directory, not both!')
# Additional words added to keywords
if arguments['suffix_keywords']:
suffix_keywords = [" " + str(sk) for sk in arguments['suffix_keywords'].split(',')]
else:
suffix_keywords = ['']
# Additional words added to keywords
if arguments['prefix_keywords']:
prefix_keywords = [str(sk) + " " for sk in arguments['prefix_keywords'].split(',')]
else:
prefix_keywords = ['']
# Setting limit on number of images to be downloaded
if arguments['limit']:
limit = int(arguments['limit'])
else:
limit = 100
if arguments['url']:
current_time = str(datetime.datetime.now()).split('.')[0]
search_keyword = [current_time.replace(":", "_")]
if arguments['similar_images']:
current_time = str(datetime.datetime.now()).split('.')[0]
search_keyword = [current_time.replace(":", "_")]
# If single_image or url argument not present then keywords is mandatory argument
if arguments['single_image'] is None and arguments['url'] is None and arguments['similar_images'] is None and \
arguments['keywords'] is None and arguments['keywords_from_file'] is None:
print('-------------------------------\n'
'Uh oh! Keywords is a required argument \n\n'
'Please refer to the documentation on guide to writing queries \n'
'https://github.com/hardikvasa/google-images-download#examples'
'\n\nexiting!\n'
'-------------------------------')
sys.exit()
# If this argument is present, set the custom output directory
if arguments['output_directory']:
main_directory = arguments['output_directory']
else:
main_directory = "downloads"
# Proxy settings
if arguments['proxy']:
os.environ["http_proxy"] = arguments['proxy']
os.environ["https_proxy"] = arguments['proxy']
######Initialization Complete
total_errors = 0
for pky in prefix_keywords: # 1.for every prefix keywords
for sky in suffix_keywords: # 2.for every suffix keywords
i = 0
while i < len(search_keyword): # 3.for every main keyword
iteration = "\n" + "Item no.: " + str(i + 1) + " -->" + " Item name = " + (pky) + (search_keyword[i]) + (sky)
if not arguments["silent_mode"]:
print(iteration.encode('raw_unicode_escape').decode('utf-8'))
print("Evaluating...")
else:
print("Downloading images for: " + (pky) + (search_keyword[i]) + (sky) + " ...")
search_term = pky + search_keyword[i] + sky
if arguments['image_directory']:
dir_name = arguments['image_directory']
elif arguments['no_directory']:
dir_name = ''
else:
dir_name = search_term + ('-' + arguments['color'] if arguments['color'] else '') #sub-directory
if not arguments["no_download"]:
self.create_directories(main_directory,dir_name,arguments['thumbnail'],arguments['thumbnail_only']) #create directories in OS
params = self.build_url_parameters(arguments) #building URL with params
url = self.build_search_url(search_term,params,arguments['url'],arguments['similar_images'],arguments['specific_site'],arguments['safe_search']) #building main search url
if limit < 101:
raw_html = self.download_page(url) # download page
else:
raw_html = self.download_extended_page(url,arguments['chromedriver'])
if not arguments["silent_mode"]:
if arguments['no_download']:
print("Getting URLs without downloading images...")
else:
print("Starting Download...")
items,errorCount,abs_path = self._get_all_items(raw_html,main_directory,dir_name,limit,arguments) #get all image items and download images
paths[pky + search_keyword[i] + sky] = abs_path
#dumps into a json file
if arguments['extract_metadata']:
try:
if not os.path.exists("logs"):
os.makedirs("logs")
except OSError as e:
print(e)
json_file = open("logs/"+search_keyword[i]+".json", "w")
json.dump(items, json_file, indent=4, sort_keys=True)
json_file.close()
#Related images
if arguments['related_images']:
print("\nGetting list of related keywords...this may take a few moments")
tabs = self.get_all_tabs(raw_html)
for key, value in tabs.items():
final_search_term = (search_term + " - " + key)
print("\nNow Downloading - " + final_search_term)
if limit < 101:
new_raw_html = self.download_page(value) # download page
else:
new_raw_html = self.download_extended_page(value,arguments['chromedriver'])
self.create_directories(main_directory, final_search_term,arguments['thumbnail'],arguments['thumbnail_only'])
self._get_all_items(new_raw_html, main_directory, search_term + " - " + key, limit,arguments)
i += 1
total_errors = total_errors + errorCount
if not arguments["silent_mode"]:
print("\nErrors: " + str(errorCount) + "\n")
return paths, total_errors
#------------- Main Program -------------#
def main():
records = user_input()
total_errors = 0
t0 = time.time() # start the timer
for arguments in records:
if arguments['single_image']: # Download Single Image using a URL
response = googleimagesdownload()
response.single_image(arguments['single_image'])
else: # or download multiple images based on keywords/keyphrase search
response = googleimagesdownload()
paths,errors = response.download(arguments) #wrapping response in a variable just for consistency
total_errors = total_errors + errors
t1 = time.time() # stop the timer
total_time = t1 - t0 # Calculating the total time required to crawl, find and download all the links of 60,000 images
if not arguments["silent_mode"]:
print("\nEverything downloaded!")
print("Total errors: " + str(total_errors))
print("Total time taken: " + str(total_time) + " Seconds")
if __name__ == "__main__":
main()
# In[ ]:
================================================
FILE: google_images_download/sample_config.json
================================================
{
"Records": [
{
"keywords": "apple",
"limit": 5,
"color": "green",
"print_urls": true
},
{
"keywords": "universe",
"limit": 15,
"size": "large",
"print_urls": true
}
]
}
================================================
FILE: requirements.txt
================================================
selenium
================================================
FILE: setup.cfg
================================================
[bdist_wheel]
universal=1
[metadata]
description-file=README.rst
================================================
FILE: setup.py
================================================
from setuptools import setup, find_packages
from codecs import open
from os import path
__version__ = '2.8.0'
here = path.abspath(path.dirname(__file__))
# Get the long description from the README file
with open('README.rst', encoding='utf-8') as f:
long_description = f.read()
# get the dependencies and installs
with open(path.join(here, 'requirements.txt'), encoding='utf-8') as f:
all_reqs = f.read().split('\n')
install_requires = [x.strip() for x in all_reqs if 'git+' not in x]
dependency_links = [x.strip().replace('git+', '') for x in all_reqs if x.startswith('git+')]
setup(
name='google_images_download',
version=__version__,
description="Python Script to download hundreds of images from 'Google Images'. It is a ready-to-run code! ",
long_description=long_description,
url='https://github.com/hardikvasa/google-images-download',
download_url='https://github.com/hardikvasa/google-images-download/tarball/' + __version__,
license='MIT',
classifiers=[
'Development Status :: 4 - Beta',
'Intended Audience :: Developers',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.3',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
keywords='google images download save filter color image-search image-dataset image-scrapper image-gallery terminal command-line',
packages=find_packages(exclude=['docs', 'tests*']),
include_package_data=True,
author='Hardik Vasa',
install_requires=install_requires,
dependency_links=dependency_links,
author_email='hnvasa@gmail.com',
entry_points={
'console_scripts': [
'googleimagesdownload = google_images_download.google_images_download:main'
]},
)
================================================
FILE: tests/__init__.py
================================================
================================================
FILE: tests/test_google_images_download.py
================================================
from google_images_download import google_images_download
import os, errno
import time
def silent_remove_of_file(file):
try:
os.remove(file)
except OSError as e:
if e.errno != errno.ENOENT:
raise e
return False
return True
def test_download_images_to_default_location():
start_time = time.time()
argumnets = {
"keywords": "Polar bears",
"limit": 5,
"print_urls": False
}
try:
temp = argumnets['output_folder']
except KeyError:
pass
else:
assert False, "This test checks download to default location yet an output folder was provided"
output_folder_path = os.path.join(os.path.realpath('.'), 'downloads', '{}'.format(argumnets['keywords']))
if os.path.exists(output_folder_path):
start_amount_of_files_in_output_folder = len([name for name in os.listdir(output_folder_path) if os.path.isfile(os.path.join(output_folder_path, name)) and os.path.getctime(os.path.join(output_folder_path, name)) < start_time])
else:
start_amount_of_files_in_output_folder = 0
response = google_images_download.googleimagesdownload()
response.download(argumnets)
files_modified_after_test_started = [name for name in os.listdir(output_folder_path) if os.path.isfile(os.path.join(output_folder_path, name)) and os.path.getmtime(os.path.join(output_folder_path, name)) > start_time]
end_amount_of_files_in_output_folder = len(files_modified_after_test_started)
print(f"Files downloaded by test {__name__}:")
for file in files_modified_after_test_started:
print(os.path.join(output_folder_path, file))
# assert end_amount_of_files_in_output_folder - start_amount_of_files_in_output_folder == argumnets['limit']
assert end_amount_of_files_in_output_folder == argumnets['limit']
print(f"Cleaning up all files downloaded by test {__name__}...")
for file in files_modified_after_test_started:
if silent_remove_of_file(os.path.join(output_folder_path, file)):
print(f"Deleted {os.path.join(output_folder_path, file)}")
else:
print(f"Failed to delete {os.path.join(output_folder_path, file)}")
================================================
FILE: tests/test_sample.py
================================================
# Sample Test passing with nose and pytest
def test_pass():
assert True, "dummy sample test"
gitextract_7o9o5ztx/
├── .gitignore
├── Licence.txt
├── MANIFEST.in
├── README.rst
├── docs/
│ ├── Makefile
│ ├── arguments.rst
│ ├── conf.py
│ ├── examples.rst
│ ├── index.rst
│ ├── installation.rst
│ ├── make.bat
│ ├── structure.rst
│ ├── troubleshooting.rst
│ └── usage.rst
├── google_images_download/
│ ├── __init__.py
│ ├── __main__.py
│ ├── google_images_download.py
│ └── sample_config.json
├── requirements.txt
├── setup.cfg
├── setup.py
└── tests/
├── __init__.py
├── test_google_images_download.py
└── test_sample.py
SYMBOL INDEX (29 symbols across 5 files)
FILE: docs/conf.py
function setup (line 23) | def setup(app):
FILE: google_images_download/__init__.py
function main (line 5) | def main():
FILE: google_images_download/google_images_download.py
function user_input (line 46) | def user_input():
class googleimagesdownload (line 126) | class googleimagesdownload:
method __init__ (line 127) | def __init__(self):
method download_page (line 131) | def download_page(self,url):
method download_extended_page (line 166) | def download_extended_page(self,url,chromedriver):
method replace_with_byte (line 217) | def replace_with_byte(self,match):
method repair (line 220) | def repair(self,brokenjson):
method get_next_tab (line 226) | def get_next_tab(self,s):
method get_all_tabs (line 256) | def get_all_tabs(self,page):
method format_object (line 273) | def format_object(self,object):
method single_image (line 287) | def single_image(self,image_url):
method similar_images (line 325) | def similar_images(self,similar_images):
method build_url_parameters (line 374) | def build_url_parameters(self,arguments):
method build_search_url (line 421) | def build_search_url(self,search_term,params,url,similar_images,specif...
method file_size (line 446) | def file_size(self,file_path):
method keywords_from_file (line 457) | def keywords_from_file(self,file_name):
method create_directories (line 479) | def create_directories(self,main_directory, dir_name,thumbnail,thumbna...
method download_image_thumbnail (line 511) | def download_image_thumbnail(self,image_url,main_directory,dir_name,re...
method download_image (line 578) | def download_image(self,image_url,image_format,main_directory,dir_name...
method _get_next_item (line 714) | def _get_next_item(self,s):
method _get_all_items (line 743) | def _get_all_items(self,page,main_directory,dir_name,limit,arguments):
method download (line 798) | def download(self,arguments):
method download_executor (line 842) | def download_executor(self,arguments):
function main (line 987) | def main():
FILE: tests/test_google_images_download.py
function silent_remove_of_file (line 6) | def silent_remove_of_file(file):
function test_download_images_to_default_location (line 16) | def test_download_images_to_default_location():
FILE: tests/test_sample.py
function test_pass (line 3) | def test_pass():
Condensed preview — 24 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (123K chars).
[
{
"path": ".gitignore",
"chars": 433,
"preview": "*.py[cod]\n\n# C extensions\n*.so\n\n#env\nenv/\nenv\n# pycharm\n.idea/\n.idea\n\n# Packages\n*.egg\n*.egg-info\nbuild\ndist\ndist/*\neggs"
},
{
"path": "Licence.txt",
"chars": 1083,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2015-2019 Hardik Vasa\n\nPermission is hereby granted, free of charge, to any person "
},
{
"path": "MANIFEST.in",
"chars": 42,
"preview": "include README.md\ninclude requirements.txt"
},
{
"path": "README.rst",
"chars": 1397,
"preview": "Google Images Download\n######################\n\nPython Script for 'searching' and 'downloading' hundreds of Google images"
},
{
"path": "docs/Makefile",
"chars": 580,
"preview": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHI"
},
{
"path": "docs/arguments.rst",
"chars": 39573,
"preview": "===============\nInput Arguments\n===============\n\nLink to `GitHub repo <https://github.com/hardikvasa/google-images-downl"
},
{
"path": "docs/conf.py",
"chars": 2256,
"preview": "# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common op"
},
{
"path": "docs/examples.rst",
"chars": 5357,
"preview": "========\nExamples\n========\n\nLink to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__\n\nLink to `Doc"
},
{
"path": "docs/index.rst",
"chars": 3267,
"preview": "======================\nGoogle Images Download\n======================\n\nLink to `GitHub repo <https://github.com/hardikvas"
},
{
"path": "docs/installation.rst",
"chars": 771,
"preview": "============\nInstallation\n============\n\nLink to `Documentation Homepage <https://google-images-download.readthedocs.io/e"
},
{
"path": "docs/make.bat",
"chars": 787,
"preview": "@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sp"
},
{
"path": "docs/structure.rst",
"chars": 336,
"preview": "========\nWorkflow\n========\n\nLink to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__\n\nLink to `Doc"
},
{
"path": "docs/troubleshooting.rst",
"chars": 3882,
"preview": "=============================\nTroubleshooting Errors/Issues\n=============================\n\nLink to `GitHub repo <https:/"
},
{
"path": "docs/usage.rst",
"chars": 1094,
"preview": "=====\nUsage\n=====\n\nLink to `GitHub repo <https://github.com/hardikvasa/google-images-download>`__\n\nLink to `Documentatio"
},
{
"path": "google_images_download/__init__.py",
"chars": 171,
"preview": "#!/usr/bin/env python\nfrom __future__ import absolute_import\n\n\ndef main():\n import google_images_download.google_imag"
},
{
"path": "google_images_download/__main__.py",
"chars": 128,
"preview": "#!/usr/bin/env python\nfrom __future__ import absolute_import\n\nfrom .__init__ import main\n\nif __name__ == '__main__':\n "
},
{
"path": "google_images_download/google_images_download.py",
"chars": 52513,
"preview": "#!/usr/bin/env python\n# In[ ]:\n# coding: utf-8\n\n###### Searching and Downloading Google Images to the local disk ######"
},
{
"path": "google_images_download/sample_config.json",
"chars": 323,
"preview": "{\r\n \"Records\": [\r\n {\r\n \"keywords\": \"apple\",\r\n \"limit\": 5,\r\n \"color\": \"green\","
},
{
"path": "requirements.txt",
"chars": 8,
"preview": "selenium"
},
{
"path": "setup.cfg",
"chars": 65,
"preview": "[bdist_wheel]\nuniversal=1\n\n[metadata]\ndescription-file=README.rst"
},
{
"path": "setup.py",
"chars": 1910,
"preview": "from setuptools import setup, find_packages\nfrom codecs import open\nfrom os import path\n\n__version__ = '2.8.0'\n\nhere = p"
},
{
"path": "tests/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "tests/test_google_images_download.py",
"chars": 2208,
"preview": "from google_images_download import google_images_download\nimport os, errno\nimport time\n\n\ndef silent_remove_of_file(file)"
},
{
"path": "tests/test_sample.py",
"chars": 102,
"preview": "# Sample Test passing with nose and pytest\n\ndef test_pass():\n assert True, \"dummy sample test\"\n"
}
]
About this extraction
This page contains the full source code of the hardikvasa/google-images-download GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 24 files (115.5 KB), approximately 22.0k tokens, and a symbol index with 29 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.