Repository: harismuneer/Android-Apps-Downloader
Branch: master
Commit: 4c9fb911a1b2
Files: 10
Total size: 17.0 KB
Directory structure:
gitextract_2kw0kymx/
├── .gitattributes
├── .github/
│ └── FUNDING.yml
├── LICENSE
├── README.md
└── code/
├── google_apps/
│ └── com.codebear.igame.apk
├── gplaycli.conf
├── ids_done.txt
├── scraper.py
├── tmp/
│ └── Note.txt
└── xiomi_apps/
└── com.codebear.igame.apk
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitattributes
================================================
# Auto detect text files and perform LF normalization
* text=auto
================================================
FILE: .github/FUNDING.yml
================================================
# These are supported funding model platforms
github: # Replace with up to 4 GitHub Sponsors-enabled usernames e.g., [user1, user2]
patreon: # Replace with a single Patreon username
open_collective: # Replace with a single Open Collective username
ko_fi: # Replace with a single Ko-fi username
tidelift: # Replace with a single Tidelift platform-name/package-name e.g., npm/babel
community_bridge: # Replace with a single Community Bridge project-name e.g., cloud-foundry
liberapay: # Replace with a single Liberapay username
issuehunt: # Replace with a single IssueHunt username
otechie: # Replace with a single Otechie username
custom: ['https://www.cyfylabs.com']
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2018 harismuneer, hussamh10
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# 📱 Android Apps Scraper/Downloader
[](https://doi.org/10.5281/zenodo.2537131)
[](#)
[](https://www.github.com/harismuneer/Android-Apps-Downloader/fork)
[](https://www.github.com/harismuneer/Android-Apps-Downloader/issues)
[](#)
We did a research project on comparison of Official Play Store Apps and their 3rd Party App Stores counterparts to analyze what modifications are done to the 3rd Party versions of an app. For this purpose, we wrote this script to download pairs of an app from Google Play Store and Xiaomi App Store (a famous 3rd Party Chinese App Store). It downloads an app from Xiaomi and Google Play store only when that app is available on both stores. This way it creates a dataset of pairs of an app.
We are open-sourcing this tool so that it can be utilized by the research community for research in Android Security.
Moroever, to compare two Android Apps we wrote another tool named [AndroCompare](https://github.com/harismuneer/AndroCompare). We have open-sourced it as well.
For details regarding **citing/referencing** this tool for your research, check the 'Citation' section below.
## Approach
The download URL of an app on Xiaomi App Store is like http://app.mi.com/download/23
The number at the end of the URL can be incremented to download as many apps as you want. So, theoretically you can download each and every app on the Xiaomi App Store. Hence there's a variable named 'target' in the tool. If target = 1000 then the tool will scan first 1000 urls for the apps. You can change the target to any number you want.
## Features
* download all apps from the famous Xiaomi App Store
* download pairs of an app from Play Store and Xiaomi App Store
* the record of all downloaded apps is maintained in a SQLite database
* if the code is interrupted using CTRL + Z then the current progress is saved so that next time the code resumes downloading from where it left previously
* incase the script is running and there occurs some internet connectivity issue then all current progress is saved and the script waits until the internet is connected again and resumes from where it left
## How to Run Code
The code is in ready to run condition. It can be run on both Windows/Ubuntu Linux.
Its written in Python 3. Moreover it uses [gplaycli](https://github.com/matlink/gplaycli), so install it using pip.
You can use [DB Browser for SQLite](http://sqlitebrowser.org/) to view the database.
----------------------------------------------------------------------------------------------------------------------------------------
## Note
This script can be easily modified to meet your specific needs e.g currently it first checks whether an app is present on both stores and if yes then it downloads it from each store. You can remove this constraint to download every possible app from Xiaomi App Store.
This code is for research purposes only.
----------------------------------------------------------------------------------------------------------------------------------------
## Citation
[](https://doi.org/10.5281/zenodo.2537131)
If you use this tool for your research, then kindly cite it. Click the above badge for more information regarding the complete citation for this tool and diffferent citation formats like IEEE, APA etc.
---------------------------------------------------------------------------------------------------------------------------------------
## Authors
You can get in touch with us on our LinkedIn profiles:
#### Haris Muneer
[](https://www.linkedin.com/in/harismuneer)
To stay updated about my latest projects: [](https://github.com/harismuneer)
#### Hussam Habib
[](https://www.linkedin.com/in/hussam-habib-0bb098104/)
To stay updated about my latest projects: [](https://github.com/hussamh10)
---
If you liked the repo then kindly support it by giving it a star ⭐ and share in your circles so more people can benefit from the effort.
## Contributions Welcome
[](https://www.github.com/harismuneer/Android-Apps-Downloader/issues)
If you find any bugs, have suggestions, or face issues:
- Open an Issue in the Issues Tab to discuss them.
- Submit a Pull Request to propose fixes or improvements.
- Review Pull Requests from other contributors to help maintain the project's quality and progress.
This project thrives on community collaboration! Members are encouraged to take the initiative, support one another, and actively engage in all aspects of the project. Whether it’s debugging, fixing issues, or brainstorming new ideas, your contributions are what keep this project moving forward.
With modern AI tools like ChatGPT, solving challenges and contributing effectively is easier than ever. Let’s work together to make this project the best it can be! 🚀
## License
[](../master/LICENSE)
Copyright (c) 2018-present, harismuneer, hussamh10
- 🛠️ Product Builder: Agile Product Manager with 5+ years of hands-on experience delivering SaaS solutions across sales, recruiting, AI, social media, and public sector domains. Background in Computer Science, with a proven track record of scaling products from inception to $XXM+ ARR, launching 3 top-ranking tools on Product Hunt, and developing solutions adopted by 250+ B2B clients in 40+ countries.
- 🌟 Open Source Advocate: Passionate about making technology accessible, I’ve developed and open-sourced several software projects for web, mobile, desktop, and AI on my GitHub profile. These projects have been used by thousands of learners worldwide to enhance their skills and knowledge.
- 📫 How to Reach Me: To learn more about my skills and work, visit my LinkedIn profile. For collaboration or inquiries, feel free to reach out via email.
🤝 Follow my journey
================================================
FILE: code/gplaycli.conf
================================================
[Credentials]
gmail_address=
gmail_password=
#keyring_service=gplaycli
token=True
token_url=https://matlink.fr/token/email/gsfid
[Cache]
token=~/.cache/gplaycli/token
[Locale]
locale=en_GB
timezone=CEST
================================================
FILE: code/ids_done.txt
================================================
1
================================================
FILE: code/scraper.py
================================================
import os
import shutil
import sqlite3
import ssl
import time
import platform
from urllib.request import urlopen
try:
import httplib
except:
import http.client as httplib
# check whether the internet is working or not
def have_internet():
conn = httplib.HTTPConnection("www.google.com", timeout=5)
try:
conn.request("HEAD", "/")
conn.close()
return True
except:
conn.close()
return False
# --------------------------------------------------------------------
def exit_gracefully():
f.close()
conn.commit()
print("Exiting....")
exit()
# --------------------------------------------------------------------
# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
if __name__ == '__main__':
target = 1000000
if platform.system()=="Linux":
download_dir = os.getcwd() + "/tmp/"
else:
download_dir = os.getcwd() + "\\tmp\\"
database = "g_x_apps.sqlite"
conn = sqlite3.connect(database)
cur = conn.cursor()
base_site = "http://app.mi.com/download/"
# load the last id number from which to continue downloading
f = open("ids_done.txt", "r")
numbers = f.readlines()
numbers = [a.rstrip() for a in numbers]
curr_num = int(numbers[-1])
f.close()
f = open("ids_done.txt", "a")
c = 1
while curr_num != target:
try:
curr_num += 1
c += 1
# commit after every 500 iterations
if c == 500:
conn.commit()
c = 0
# check net connectivity
if not have_internet():
# save progress
conn.commit()
f.close()
print("Internet disconnected.. waiting")
while not have_internet():
time.sleep(5)
f = open("ids_done.txt", "a")
print("Connected!")
print("------------------------------------------")
print("Processing:", curr_num)
# empty left over downloads
if platform.system() == 'Linux':
os.system("rm -f '" + download_dir + "{*,.*}'")
else:
os.system('del "' + download_dir + '*" /Q')
# ------------------------------------------------------------------
# check whether app on xiomi exists or not
try:
url = base_site + str(curr_num)
html = urlopen(url, context=ctx)
# check apk exists or not
if html.url[-4:] != ".apk":
f.write(str(curr_num) + "\n")
print("No app against this ID on Xiaomi Store")
continue
# if it exists
package = html.url.split('/')[-1]
package = package.split(".apk")[0]
print("Found on Xiaomi:", package)
except:
f.write(str(curr_num) + "\n")
print("No app against this ID on Xiaomi Store")
continue
# ------------------------------------------------------------------
# ------------------------------------------------------------------
# check if the same app exists on google
try:
url = "https://play.google.com/store/apps/details?id=" + package
html = urlopen(url, context=ctx)
if str(html.getcode()) != '200':
f.write(str(curr_num) + "\n")
print(package, " doesn't exist on Play Store")
continue
# if it exists
print("Found on PlayStore:", package)
except:
f.write(str(curr_num) + "\n")
print(package, " doesn't exist on Play Store")
continue
# ------------------------------------------------------------------
# ------------------------------------------------------------------
# download google
print("Downloading from Playstore")
os.system('gplaycli -d ' + package + ' -f "' + 'google_apps"' + ' -p')
# check if a file of that apk is created
if platform.system() == "Linux":
dir = os.getcwd() + "/google_apps/" + package + ".apk"
else:
dir = os.getcwd() + "\\google_apps\\" + package + ".apk"
time.sleep(1)
save = False
# check if that directory exists
if os.path.exists(dir):
save = True
if not save:
f.write(str(curr_num) + "\n")
print("App from playstore not downloaded (might be paid app or not available in your country).")
continue
print(package, ": Google Download Successful")
# ------------------------------------------------------------------
# ------------------------------------------------------------------
# download xiomi
print("Downloading from Xiaomi")
os.system("wget -P tmp/ --content-disposition -q " + base_site + str(curr_num))
# rename the file and move it to its folder
for file in os.listdir(download_dir):
if file.endswith(".apk"):
print('Moving Xiomi File...')
shutil.move(download_dir + file, "xiomi_apps/" + package + ".apk")
cur.execute("INSERT OR IGNORE INTO APPS VALUES (?,1,1,0)",
(package,))
print(package, ": Xiomi Download Successful")
else:
error = 'Error in downloading'
print(error, package)
# ------------------------------------------------------------------
f.write(str(curr_num) + "\n")
except KeyboardInterrupt:
exit_gracefully()
except:
pass
print("-----------------------------")
print("Downloads Complete!!!")
print("-----------------------------")
print('\n\nProcess Successfully finished!!!!!\n\n')
f.close()
conn.commit()
================================================
FILE: code/tmp/Note.txt
================================================
This folder is the temporary download directory for an app. After the app is downloaded, it is moved
to the relavant download folder.
Hey there, I'm Haris Muneer 👨🏻💻