Repository: cooperhammond/irs
Branch: master
Commit: c99e8257e928
Files: 26
Total size: 67.0 KB
Directory structure:
gitextract_npef2uk3/
├── .editorconfig
├── .gitignore
├── .travis.yml
├── LICENSE
├── README.md
├── shard.yml
├── spec/
│ ├── irs_spec.cr
│ └── spec_helper.cr
└── src/
├── bottle/
│ ├── cli.cr
│ ├── config.cr
│ ├── pattern.cr
│ ├── styles.cr
│ └── version.cr
├── glue/
│ ├── album.cr
│ ├── list.cr
│ ├── mapper.cr
│ ├── playlist.cr
│ └── song.cr
├── interact/
│ ├── future.cr
│ ├── logger.cr
│ ├── ripper.cr
│ └── tagger.cr
├── irs.cr
└── search/
├── ranking.cr
├── spotify.cr
└── youtube.cr
================================================
FILE CONTENTS
================================================
================================================
FILE: .editorconfig
================================================
root = true
[*.cr]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true
================================================
FILE: .gitignore
================================================
/docs/
/lib/
/bin/
/logs/
/.shards/
/Music/
*.dwarf
*.mp3
*.webm*
.ripper.log
ffmpeg
ffprobe
youtube-dl
*.temp
================================================
FILE: .travis.yml
================================================
language: crystal
# Uncomment the following if you'd like Travis to run specs and check code formatting
# script:
# - crystal spec
# - crystal tool format --check
================================================
FILE: LICENSE
================================================
The MIT License (MIT)
Copyright (c) 2020 Cooper Hammond
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
================================================
FILE: README.md
================================================
# irs: The Ironic Repositioning System
[](https://crystal-lang.org/)
[](https://github.com/cooperhammond/irs/blob/master/LICENSE)
[](https://saythanks.io/to/kepoorh%40gmail.com)
> A music scraper that understands your metadata needs.
`irs` is a command-line application that downloads audio and metadata in order
to package an mp3 with both. Extensible, the user can download individual
songs, entire albums, or playlists from Spotify.
---
## Table of Contents
- [Usage](#usage)
- [Demo](#demo)
- [Installation](#installation)
- [Pre-built](#pre-built)
- [From source](#from-source)
- [Set up](#setup)
- [Config](#config)
- [How it works](#how-it-works)
- [Contributing](#contributing)
## Usage
```
~ $ irs -h
Usage: irs [--help] [--version] [--install]
[-s -a ]
[-A -a ]
[-p -a ]
Arguments:
-h, --help Show this help message and exit
-v, --version Show the program version and exit
-i, --install Download binaries to config location
-c, --config Show config file location
-a, --artist Specify artist name for downloading
-s, --song Specify song name to download
-A, --album Specify the album name to download
-p, --playlist Specify the playlist name to download
-u, --url Specify the youtube url to download from (for single songs only)
-g, --give-url Specify the youtube url sources while downloading (for albums or playlists only only)
Examples:
$ irs --song "Bohemian Rhapsody" --artist "Queen"
# => downloads the song "Bohemian Rhapsody" by "Queen"
$ irs --album "Demon Days" --artist "Gorillaz"
# => downloads the album "Demon Days" by "Gorillaz"
$ irs --playlist "a different drummer" --artist "prakkillian"
# => downloads the playlist "a different drummer" by the user prakkillian
```
### Demo
[](https://asciinema.org/a/332793)
## Installation
### Pre-built
Just download the latest release for your platform
[here](https://github.com/cooperhammond/irs/releases).
Note that the binaries right now have only been tested on WSL. They *should* run on most linux distros, and OS X, but if they don't please make an issue above.
### From Source
If you're one of those cool people who compiles from source
1. Install crystal-lang
([`https://crystal-lang.org/install/`](https://crystal-lang.org/install/))
1. Clone it (`git clone https://github.com/cooperhammond/irs`)
1. CD it (`cd irs`)
1. Build it (`shards build`)
### Setup
1. Create a `.yaml` config file somewhere on your system (usually `~/.irs/`)
1. Copy the following into it
```yaml
binary_directory: ~/.irs/bin
music_directory: ~/Music
filename_pattern: "{track_number} - {title}"
directory_pattern: "{artist}/{album}"
client_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
client_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
single_folder_playlist:
enabled: true
retain_playlist_order: true
unify_into_album: false
```
1. Set the environment variable `IRS_CONFIG_LOCATION` pointing to that file
1. Go to [`https://developer.spotify.com/dashboard/`](https://developer.spotify.com/dashboard/)
1. Log in or create an account
1. Click `CREATE A CLIENT ID`
1. Enter all necessary info, true or false, continue
1. Find your client key and client secret
1. Copy each respectively into the X's in your config file
1. Run `irs --install` and answer the prompts!
You should be good to go! Run the file from your command line to get more help on
usage or keep reading!
# Config
You may have noticed that there's a config file with more than a few options.
Here's what they do:
```yaml
binary_directory: ~/.irs/bin
music_directory: ~/Music
search_terms: "lyrics"
filename_pattern: "{track_number} - {title}"
directory_pattern: "{artist}/{album}"
client_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
client_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
single_folder_playlist:
enabled: true
retain_playlist_order: true
unify_into_album: false
```
- `binary_directory`: a path specifying where the downloaded binaries should
be placed
- `music_directory`: a path specifying where downloaded mp3s should be placed.
- `search_terms`: additional search terms to plug into youtube, which can be
potentially useful for not grabbing erroneous audio.
- `filename_pattern`: a pattern for the output filename of the mp3
- `directory_pattern`: a pattern for the folder structure your mp3s are saved in
- `client_key`: a client key from your spotify API application
- `client_secret`: a client secret key from your spotify API application
- `single_folder_playlist/enabled`: if set to true, all mp3s from a downloaded
playlist will be placed in the same folder.
- `single_folder_playlist/retain_playlist_order`: if set to true, the track
numbers of the mp3s of the playlist will be overwritten to correspond to
their place in the playlist
- `single_folder_playlist/unify_into_album`: if set to true, will overwrite
the album name and album image of the mp3 with the title of your playlist
and the image for your playlist respectively
In a pattern following keywords will be replaced:
| Keyword | Replacement | Example |
| :----: | :----: | :----: |
| `{artist}` | Artist Name | Queen |
| `{title}` | Track Title | Bohemian Rhapsody |
| `{album}` | Album Name | Stone Cold Classics |
| `{track_number}` | Track Number | 9 |
| `{total_tracks}` | Total Tracks in Album | 14 |
| `{disc_number}` | Disc Number | 1 |
| `{day}` | Release Day | 01 |
| `{month}` | Release Month | 01 |
| `{year}` | Release Year | 2006 |
| `{id}` | Spotify ID | 6l8GvAyoUZwWDgF1e4822w |
Beware OS-restrictions when naming your mp3s.
Pattern Examples:
```yaml
music_directory: ~/Music
filename_pattern: "{track_number} - {title}"
directory_pattern: "{artist}/{album}"
```
Outputs: `~/Music/Queen/Stone Cold Classics/9 - Bohemian Rhapsody.mp3`
```yaml
music_directory: ~/Music
filename_pattern: "{artist} - {title}"
directory_pattern: ""
```
Outputs: `~/Music/Queen - Bohemian Rhapsody.mp3`
```yaml
music_directory: ~/Music
filename_pattern: "{track_number} of {total_tracks} - {title}"
directory_pattern: "{year}/{artist}/{album}"
```
Outputs: `~/Music/2006/Queen/Stone Cold Classics/9 of 14 - Bohemian Rhapsody.mp3`
```yaml
music_directory: ~/Music
filename_pattern: "{track_number}. {title}"
directory_pattern: "irs/{artist} - {album}"
```
Outputs: `~/Music/irs/Queen - Stone Cold Classics/9. Bohemian Rhapsody.mp3`
## How it works
**At it's core** `irs` downloads individual songs. It does this by interfacing
with the Spotify API, grabbing metadata, and then searching Youtube for a video
containing the song's audio. It will download the video using
[`youtube-dl`](https://github.com/ytdl-org/youtube-dl), extract the audio using
[`ffmpeg`](https://ffmpeg.org/), and then pack the audio and metadata together
into an MP3.
From the core, it has been extended to download the index of albums and
playlists through the spotify API, and then iteratively use the method above
for downloading each song.
It used to be in python, but
1. I wasn't a fan of python's limited ability to distribute standalone binaries
1. It was a charlie foxtrot of code that I made when I was little and I wanted
to refine it
1. `crystal-lang` made some promises and I was interested in seeing how well it
did (verdict: if you're building high-level tools you want to run quickly
and distribute, it's perfect)
## Contributing
Any and all contributions are welcome. If you think of a cool feature, send a
PR or shoot me an [email](mailto:kepoorh@gmail.com). If you think something
could be implemented better, _please_ shoot me an email. If you like what I'm
doing here, _pretty please_ shoot me an email.
1. Fork it ()
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Add some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create a new Pull Request
================================================
FILE: shard.yml
================================================
name: irs
version: 1.4.0
authors:
- Cooper Hammond
targets:
irs:
main: src/irs.cr
license: MIT
dependencies:
ydl_binaries:
github: cooperhammond/ydl-binaries
json_mapping:
github: crystal-lang/json_mapping.cr
================================================
FILE: spec/irs_spec.cr
================================================
require "./spec_helper"
describe CLI do
# TODO: Write tests
it "can show help" do
run_CLI_with_args(["--help"])
end
it "can show version" do
run_CLI_with_args(["--version"])
end
# !!TODO: make a long and short version of the test suite
# TODO: makes so this doesn't need user input
it "can install ytdl and ffmpeg binaries" do
# run_CLI_with_args(["--install"])
end
it "can show config file loc" do
run_CLI_with_args(["--config"])
end
it "can download a single song" do
run_CLI_with_args(["--song", "Bohemian Rhapsody", "--artist", "Queen"])
end
it "can download an album" do
run_CLI_with_args(["--artist", "Arctic Monkeys", "--album", "Da Frame 2R / Matador"])
end
it "can download a playlist" do
run_CLI_with_args(["--artist", "prakkillian", "--playlist", "IRS Testing"])
end
end
================================================
FILE: spec/spec_helper.cr
================================================
require "spec"
# https://github.com/mosop/stdio
require "../src/bottle/cli"
def run_CLI_with_args(argv : Array(String))
cli = CLI.new(argv)
cli.act_on_args
end
================================================
FILE: src/bottle/cli.cr
================================================
require "ydl_binaries"
require "./config"
require "./styles"
require "./version"
require "../glue/song"
require "../glue/album"
require "../glue/playlist"
class CLI
# layout:
# [[shortflag, longflag], key, type]
@options = [
[["-h", "--help"], "help", "bool"],
[["-v", "--version"], "version", "bool"],
[["-i", "--install"], "install", "bool"],
[["-c", "--config"], "config", "bool"],
[["-a", "--artist"], "artist", "string"],
[["-s", "--song"], "song", "string"],
[["-A", "--album"], "album", "string"],
[["-p", "--playlist"], "playlist", "string"],
[["-u", "--url"], "url", "string"],
[["-S", "--select"], "select", "bool"],
[["--ask-skip"], "ask_skip", "bool"],
[["--apply"], "apply_file", "string"]
]
@args : Hash(String, String)
def initialize(argv : Array(String))
@args = parse_args(argv)
end
def version
puts "irs v#{IRS::VERSION}"
end
def help
msg = <<-EOP
#{Style.bold "Usage: irs [--help] [--version] [--install]"}
#{Style.bold " [-s -a ]"}
#{Style.bold " [-A -a ]"}
#{Style.bold " [-p -a ]"}
#{Style.bold "Arguments:"}
#{Style.blue "-h, --help"} Show this help message and exit
#{Style.blue "-v, --version"} Show the program version and exit
#{Style.blue "-i, --install"} Download binaries to config location
#{Style.blue "-c, --config"} Show config file location
#{Style.blue "-a, --artist "} Specify artist name for downloading
#{Style.blue "-s, --song "} Specify song name to download
#{Style.blue "-A, --album "} Specify the album name to download
#{Style.blue "-p, --playlist "} Specify the playlist name to download
#{Style.blue "-u, --url "} Specify the youtube url to download from
#{Style.blue " "} (for albums and playlists, the command-line
#{Style.blue " "} argument is ignored, and it should be '')
#{Style.blue "-S, --select"} Use a menu to choose each song's video source
#{Style.blue "--ask-skip"} Before every playlist/album song, ask to skip
#{Style.blue "--apply "} Apply metadata to a existing file
#{Style.bold "Examples:"}
$ #{Style.green %(irs --song "Bohemian Rhapsody" --artist "Queen")}
#{Style.dim %(# => downloads the song "Bohemian Rhapsody" by "Queen")}
$ #{Style.green %(irs --album "Demon Days" --artist "Gorillaz")}
#{Style.dim %(# => downloads the album "Demon Days" by "Gorillaz")}
$ #{Style.green %(irs --playlist "a different drummer" --artist "prakkillian")}
#{Style.dim %(# => downloads the playlist "a different drummer" by the user prakkillian)}
#{Style.bold "This project is licensed under the MIT license."}
#{Style.bold "Project page: "}
EOP
puts msg
end
def act_on_args
Config.check_necessities
if @args["help"]? || @args.keys.size == 0
help
elsif @args["version"]?
version
elsif @args["install"]?
YdlBinaries.get_both(Config.binary_location)
elsif @args["config"]?
puts ENV["IRS_CONFIG_LOCATION"]?
elsif @args["song"]? && @args["artist"]?
s = Song.new(@args["song"], @args["artist"])
s.provide_client_keys(Config.client_key, Config.client_secret)
s.grab_it(flags: @args)
s.organize_it()
elsif @args["album"]? && @args["artist"]?
a = Album.new(@args["album"], @args["artist"])
a.provide_client_keys(Config.client_key, Config.client_secret)
a.grab_it(flags: @args)
elsif @args["playlist"]? && @args["artist"]?
p = Playlist.new(@args["playlist"], @args["artist"])
p.provide_client_keys(Config.client_key, Config.client_secret)
p.grab_it(flags: @args)
else
puts Style.red("Those arguments don't do anything when used that way.")
puts "Type `irs -h` to see usage."
end
end
private def parse_args(argv : Array(String)) : Hash(String, String)
arguments = {} of String => String
i = 0
current_key = ""
pass_next_arg = false
argv.each do |arg|
# If the previous arg was an arg flag, this is an arg, so pass it
if pass_next_arg
pass_next_arg = false
i += 1
next
end
flag = [] of Array(String) | String
valid_flag = false
@options.each do |option|
if option[0].includes?(arg)
flag = option
valid_flag = true
break
end
end
# ensure the flag is actually defined
if !valid_flag
arg_error argv, i, %("#{arg}" is an invalid flag or argument.)
end
# ensure there's an argument if the program needs one
if flag[2] == "string" && i + 1 >= argv.size
arg_error argv, i, %("#{arg}" needs an argument.)
end
key = flag[1].as(String)
if flag[2] == "string"
arguments[key] = argv[i + 1]
pass_next_arg = true
elsif flag[2] == "bool"
arguments[key] = "true"
end
i += 1
end
return arguments
end
private def arg_error(argv : Array(String), arg : Int32, msg : String) : Nil
precursor = "irs"
precursor += " " if arg != 0
if arg == 0
start = [] of String
else
start = argv[..arg - 1]
end
last = argv[arg + 1..]
distance = (precursor + start.join(" ")).size
print Style.dim(precursor + start.join(" "))
print Style.bold(Style.red(" " + argv[arg]).to_s)
puts Style.dim (" " + last.join(" "))
(0..distance).each do |i|
print " "
end
puts "^"
puts Style.red(Style.bold(msg).to_s)
puts "Type `irs -h` to see usage."
exit 1
end
end
================================================
FILE: src/bottle/config.cr
================================================
require "yaml"
require "./styles"
require "../search/spotify"
EXAMPLE_CONFIG = <<-EOP
#{Style.dim "exampleconfig.yml"}
#{Style.dim "===="}
#{Style.blue "search_terms"}: #{Style.green "\"lyrics\""}
#{Style.blue "binary_directory"}: #{Style.green "~/.irs/bin"}
#{Style.blue "music_directory"}: #{Style.green "~/Music"}
#{Style.blue "filename_pattern"}: #{Style.green "\"{track_number} - {title}\""}
#{Style.blue "directory_pattern"}: #{Style.green "\"{artist}/{album}\""}
#{Style.blue "client_key"}: #{Style.green "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
#{Style.blue "client_secret"}: #{Style.green "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
#{Style.blue "single_folder_playlist"}:
#{Style.blue "enabled"}: #{Style.green "true"}
#{Style.blue "retain_playlist_order"}: #{Style.green "true"}
#{Style.blue "unify_into_album"}: #{Style.green "false"}
#{Style.dim "===="}
EOP
module Config
extend self
@@arguments = [
"search_terms",
"binary_directory",
"music_directory",
"filename_pattern",
"directory_pattern",
"client_key",
"client_secret",
"single_folder_playlist: enabled",
"single_folder_playlist: retain_playlist_order",
"single_folder_playlist: unify_into_album",
]
@@conf = YAML.parse("")
begin
@@conf = YAML.parse(File.read(ENV["IRS_CONFIG_LOCATION"]))
rescue
puts Style.red "Before anything else, define the environment variable IRS_CONFIG_LOCATION pointing to a .yml file like this one."
puts EXAMPLE_CONFIG
puts Style.bold "See https://github.com/cooperhammond/irs for more information on the config file"
exit 1
end
def search_terms : String
return @@conf["search_terms"].to_s
end
def binary_location : String
path = @@conf["binary_directory"].to_s
return Path[path].expand(home: true).to_s
end
def music_directory : String
path = @@conf["music_directory"].to_s
return Path[path].expand(home: true).to_s
end
def filename_pattern : String
return @@conf["filename_pattern"].to_s
end
def directory_pattern : String
return @@conf["directory_pattern"].to_s
end
def client_key : String
return @@conf["client_key"].to_s
end
def client_secret : String
return @@conf["client_secret"].to_s
end
def single_folder_playlist? : Bool
return @@conf["single_folder_playlist"]["enabled"].as_bool
end
def retain_playlist_order? : Bool
return @@conf["single_folder_playlist"]["retain_playlist_order"].as_bool
end
def unify_into_album? : Bool
return @@conf["single_folder_playlist"]["unify_into_album"].as_bool
end
def check_necessities
missing_configs = [] of String
@@arguments.each do |argument|
if !check_conf(argument)
missing_configs.push(argument)
end
end
if missing_configs.size > 0
puts Style.red("You are missing the following key(s) in your YAML config file:")
missing_configs.each do |config|
puts " " + config
end
puts "\nHere's an example of what your config should look like:"
puts EXAMPLE_CONFIG
puts Style.bold "See https://github.com/cooperhammond/irs for more information on the config file"
exit 1
end
spotify = SpotifySearcher.new
spotify.authorize(self.client_key, self.client_secret)
if !spotify.authorized?
puts Style.red("There's something wrong with your client key and/or client secret")
puts "Get your keys from https://developer.spotify.com/dashboard, and enter them in your config file"
puts "Your config file is at #{ENV["IRS_CONFIG_LOCATION"]}"
puts EXAMPLE_CONFIG
puts Style.bold "See https://github.com/cooperhammond/irs for more information on the config file"
exit 1
end
end
private def check_conf(key : String) : YAML::Any?
if key.includes?(": ")
args = key.split(": ")
if @@conf[args[0]]?
return @@conf[args[0]][args[1]]?
else
return @@conf[args[0]]?
end
else
return @@conf[key]?
end
end
end
================================================
FILE: src/bottle/pattern.cr
================================================
module Pattern
extend self
def parse(formatString : String, metadata : JSON::Any)
formatted : String = formatString
date : Array(String) = (metadata["album"]? || JSON.parse("{}"))["release_date"]?.to_s.split('-')
keys : Hash(String, String) = {
"artist" => ((metadata.dig?("artists") || JSON.parse("{}"))[0]? || JSON.parse("{}"))["name"]?.to_s,
"title" => metadata["name"]?.to_s,
"album" => (metadata["album"]? || JSON.parse("{}"))["name"]?.to_s,
"track_number" => metadata["track_number"]?.to_s,
"disc_number" => metadata["disc_number"]?.to_s,
"total_tracks" => (metadata["album"]? || JSON.parse("{}"))["total_tracks"]?.to_s,
"year" => date[0]?.to_s,
"month" => date[1]?.to_s,
"day" => date[2]?.to_s,
"id" => metadata["id"]?.to_s
}
keys.each do |pair|
formatted = formatted.gsub("{#{pair[0]}}", pair[1] || "")
end
return formatted
end
end
================================================
FILE: src/bottle/styles.cr
================================================
require "colorize"
class Style
def self.bold(txt)
txt.colorize.mode(:bold).to_s
end
def self.dim(txt)
txt.colorize.mode(:dim).to_s
end
def self.blue(txt)
txt.colorize(:light_blue).to_s
end
def self.green(txt)
txt.colorize(:light_green).to_s
end
def self.red(txt)
txt.colorize(:light_red).to_s
end
end
================================================
FILE: src/bottle/version.cr
================================================
module IRS
VERSION = "1.4.0"
end
================================================
FILE: src/glue/album.cr
================================================
require "../bottle/config"
require "./mapper"
require "./song"
require "./list"
class Album < SpotifyList
@home_music_directory = Config.music_directory
# Uses the `spotify_searcher` defined in parent `SpotifyList` to find the
# correct metadata of the list
def find_it : JSON::Any
album = @spotify_searcher.find_item("album", {
"name" => @list_name.as(String),
"artist" => @list_author.as(String),
})
if album
return album.as(JSON::Any)
else
puts "No album was found by that name and artist."
exit 1
end
end
# Will define specific metadata that may not be included in the raw return
# of spotify's album json. Moves the title of the album and the album art
# to the json of the single song
def organize_song_metadata(list : JSON::Any, datum : JSON::Any) : JSON::Any
album_metadata = parse_to_json(%(
{
"name": "#{list["name"]}",
"images": [{"url": "#{list["images"][0]["url"]}"}]
}
))
prepped_data = TrackMapper.from_json(datum.to_json)
prepped_data.album = album_metadata
data = parse_to_json(prepped_data.to_json)
return data
end
private def organize(song : Song)
song.organize_it()
end
end
================================================
FILE: src/glue/list.cr
================================================
require "json"
require "../search/spotify"
require "../search/youtube"
require "../interact/ripper"
require "../interact/tagger"
require "./song"
# A parent class for downloading albums and playlists from spotify
abstract class SpotifyList
@spotify_searcher = SpotifySearcher.new
@file_names = [] of String
@outputs : Hash(String, Array(String)) = {
"searching" => [
Style.bold("Searching for %l by %a ... \r"),
Style.green("+ ") + Style.bold("%l by %a \n")
],
"url" => [
Style.bold("When prompted for a URL, provide a youtube URL or press enter to scrape for one\n")
]
}
def initialize(@list_name : String, @list_author : String?)
end
# Finds the list, and downloads all of the songs using the `Song` class
def grab_it(flags = {} of String => String)
ask_url = flags["url"]?
ask_skip = flags["ask_skip"]?
is_playlist = flags["playlist"]?
if !@spotify_searcher.authorized?
raise("Need to call provide_client_keys on Album or Playlist class.")
end
if ask_url
outputter("url", 0)
end
outputter("searching", 0)
list = find_it()
outputter("searching", 1)
contents = list["tracks"]["items"].as_a
i = 0
contents.each do |datum|
i += 1
if datum["track"]?
datum = datum["track"]
end
data = organize_song_metadata(list, datum)
s_name = data["name"].to_s
s_artist = data["artists"][0]["name"].to_s
song = Song.new(s_name, s_artist)
song.provide_spotify(@spotify_searcher)
song.provide_metadata(data)
puts Style.bold("[#{i}/#{contents.size}]")
unless ask_skip && skip?(s_name, s_artist, is_playlist)
song.grab_it(flags: flags)
organize(song)
else
puts "Skipping..."
end
end
end
# Will authorize the class associated `SpotifySearcher`
def provide_client_keys(client_key : String, client_secret : String)
@spotify_searcher.authorize(client_key, client_secret)
end
private def skip?(name, artist, is_playlist)
print "Skip #{Style.blue name}" +
(is_playlist ? " (by #{Style.green artist})": "") + "? "
response = gets
return response && response.lstrip.downcase.starts_with? "y"
end
private def outputter(key : String, index : Int32)
text = @outputs[key][index]
.gsub("%l", @list_name)
.gsub("%a", @list_author)
print text
end
# Defined in subclasses, will return the appropriate information or call an
# error if the info is not found and exit
abstract def find_it : JSON::Any
# If there's a need to organize the individual song data so that the `Song`
# class can better handle it, this function will be defined in the subclass
private abstract def organize_song_metadata(list : JSON::Any,
datum : JSON::Any) : JSON::Any
# Will define the specific type of organization for a list of songs.
# Needed because most people want albums sorted by artist, but playlists all
# in one folder
private abstract def organize(song : Song)
end
================================================
FILE: src/glue/mapper.cr
================================================
require "json"
require "json_mapping"
class PlaylistExtensionMapper
JSON.mapping(
tracks: {
type: PlaylistTracksMapper,
setter: true,
},
id: String,
images: JSON::Any,
name: String,
owner: JSON::Any,
type: String
)
end
class PlaylistTracksMapper
JSON.mapping(
items: {
type: Array(JSON::Any),
setter: true,
},
total: Int32
)
end
class TrackMapper
JSON.mapping(
album: {
type: JSON::Any,
nilable: true,
setter: true,
},
artists: {
type: Array(JSON::Any),
setter: true
},
disc_number: {
type: Int32,
setter: true
},
id: String,
name: String,
track_number: {
type: Int32,
setter: true
},
duration_ms: Int32,
type: String,
uri: String
)
end
def parse_to_json(string_json : String) : JSON::Any
return JSON.parse(string_json)
end
================================================
FILE: src/glue/playlist.cr
================================================
require "json"
require "../bottle/config"
require "./song"
require "./list"
require "./mapper"
class Playlist < SpotifyList
@song_index = 1
@home_music_directory = Config.music_directory
@playlist : JSON::Any?
# Uses the `spotify_searcher` defined in parent `SpotifyList` to find the
# correct metadata of the list
def find_it : JSON::Any
@playlist = @spotify_searcher.find_item("playlist", {
"name" => @list_name.as(String),
"username" => @list_author.as(String),
})
if @playlist
return @playlist.as(JSON::Any)
else
puts "No playlists were found by that name and user."
exit 1
end
end
# Will define specific metadata that may not be included in the raw return
# of spotify's album json. Moves the title of the album and the album art
# to the json of the single song
def organize_song_metadata(list : JSON::Any, datum : JSON::Any) : JSON::Any
data = datum
if Config.retain_playlist_order?
track = TrackMapper.from_json(data.to_json)
track.track_number = @song_index
track.disc_number = 1
data = JSON.parse(track.to_json)
end
if Config.unify_into_album?
track = TrackMapper.from_json(data.to_json)
track.album = JSON.parse(%({
"name": "#{list["name"]}",
"images": [{"url": "#{list["images"][0]["url"]}"}]
}))
track.artists.push(JSON.parse(%({
"name": "#{list["owner"]["display_name"]}",
"owner": true
})))
data = JSON.parse(track.to_json)
end
@song_index += 1
return data
end
private def organize(song : Song)
if Config.single_folder_playlist?
path = Path[@home_music_directory].expand(home: true)
path = path / @playlist.as(JSON::Any)["name"].to_s
.gsub(/[\/]/, "").gsub(" ", " ")
strpath = path.to_s
if !File.directory?(strpath)
FileUtils.mkdir_p(strpath)
end
safe_filename = song.filename.gsub(/[\/]/, "").gsub(" ", " ")
FileUtils.cp("./" + song.filename, (path / safe_filename).to_s)
FileUtils.rm("./" + song.filename)
else
song.organize_it()
end
end
end
================================================
FILE: src/glue/song.cr
================================================
require "../search/spotify"
require "../search/youtube"
require "../interact/ripper"
require "../interact/tagger"
require "../bottle/config"
require "../bottle/pattern"
require "../bottle/styles"
class Song
@spotify_searcher = SpotifySearcher.new
@client_id = ""
@client_secret = ""
@metadata : JSON::Any?
getter filename = ""
@artist = ""
@album = ""
@outputs : Hash(String, Array(String)) = {
"intro" => [Style.bold("[%s by %a]\n")],
"metadata" => [
" Searching for metadata ...\r",
Style.green(" + ") + Style.dim("Metadata found \n")
],
"url" => [
" Searching for URL ...\r",
Style.green(" + ") + Style.dim("URL found \n"),
" Validating URL ...\r",
Style.green(" + ") + Style.dim("URL validated \n"),
" URL?: "
],
"download" => [
" Downloading video:\n",
Style.green("\r + ") + Style.dim("Converted to mp3 \n")
],
"albumart" => [
" Downloading album art ...\r",
Style.green(" + ") + Style.dim("Album art downloaded \n")
],
"tagging" => [
" Attaching metadata ...\r",
Style.green(" + ") + Style.dim("Metadata attached \n")
],
"finished" => [
Style.green(" + ") + "Finished!\n"
]
}
def initialize(@song_name : String, @artist_name : String)
end
# Find, downloads, and tags the mp3 song that this class represents.
# Optionally takes a youtube URL to download from
#
# ```
# Song.new("Bohemian Rhapsody", "Queen").grab_it
# ```
def grab_it(url : (String | Nil) = nil, flags = {} of String => String)
passed_url : (String | Nil) = flags["url"]?
passed_file : (String | Nil) = flags["apply_file"]?
select_link = flags["select"]?
outputter("intro", 0)
if !@spotify_searcher.authorized? && !@metadata
if @client_id != "" && @client_secret != ""
@spotify_searcher.authorize(@client_id, @client_secret)
else
raise("Need to call either `provide_metadata`, `provide_spotify`, " +
"or `provide_client_keys` so that Spotify can be interfaced with.")
end
end
if !@metadata
outputter("metadata", 0)
@metadata = @spotify_searcher.find_item("track", {
"name" => @song_name,
"artist" => @artist_name,
})
if !@metadata
raise("There was no metadata found on Spotify for " +
%("#{@song_name}" by "#{@artist_name}". ) +
"Check your input and try again.")
end
outputter("metadata", 1)
end
data = @metadata.as(JSON::Any)
@song_name = data["name"].as_s
@artist_name = data["artists"][0]["name"].as_s
@filename = "#{Pattern.parse(Config.filename_pattern, data)}.mp3"
if passed_file
puts Style.green(" +") + Style.dim(" Moving file: ") + passed_file
File.rename(passed_file, @filename)
else
if passed_url
if passed_url.strip != ""
url = passed_url
else
outputter("url", 4)
url = gets
if !url.nil? && url.strip == ""
url = nil
end
end
end
if !url
outputter("url", 0)
url = Youtube.find_url(data, flags: flags)
if !url
raise("There was no url found on youtube for " +
%("#{@song_name}" by "#{@artist_name}. ) +
"Check your input and try again.")
end
outputter("url", 1)
else
outputter("url", 2)
url = Youtube.validate_url(url)
if !url
raise("The url is an invalid youtube URL " +
"Check the URL and try again")
end
outputter("url", 3)
end
outputter("download", 0)
Ripper.download_mp3(url.as(String), @filename)
outputter("download", 1)
end
outputter("albumart", 0)
temp_albumart_filename = ".tempalbumart.jpg"
HTTP::Client.get(data["album"]["images"][0]["url"].as_s) do |response|
File.write(temp_albumart_filename, response.body_io)
end
outputter("albumart", 0)
# check if song's metadata has been modded in playlist, update artist accordingly
if data["artists"][-1]["owner"]?
@artist = data["artists"][-1]["name"].as_s
else
@artist = data["artists"][0]["name"].as_s
end
@album = data["album"]["name"].as_s
tagger = Tags.new(@filename)
tagger.add_album_art(temp_albumart_filename)
tagger.add_text_tag("title", data["name"].as_s)
tagger.add_text_tag("artist", @artist)
if !@album.empty?
tagger.add_text_tag("album", @album)
end
if genre = @spotify_searcher.find_genre(data["artists"][0]["id"].as_s)
tagger.add_text_tag("genre", genre)
end
tagger.add_text_tag("track", data["track_number"].to_s)
tagger.add_text_tag("disc", data["disc_number"].to_s)
outputter("tagging", 0)
tagger.save
File.delete(temp_albumart_filename)
outputter("tagging", 1)
outputter("finished", 0)
end
# Will organize the song into the user's provided music directory
# in the user's provided structure
# Must be called AFTER the song has been downloaded.
#
# ```
# s = Song.new("Bohemian Rhapsody", "Queen").grab_it
# s.organize_it()
# # With
# # directory_pattern = "{artist}/{album}"
# # filename_pattern = "{track_number} - {title}"
# # Mp3 will be moved to
# # /home/cooper/Music/Queen/A Night At The Opera/1 - Bohemian Rhapsody.mp3
# ```
def organize_it()
path = Path[Config.music_directory].expand(home: true)
Pattern.parse(Config.directory_pattern, @metadata.as(JSON::Any)).split('/').each do |dir|
path = path / dir.gsub(/[\/]/, "").gsub(" ", " ")
end
strpath = path.to_s
if !File.directory?(strpath)
FileUtils.mkdir_p(strpath)
end
safe_filename = @filename.gsub(/[\/]/, "").gsub(" ", " ")
FileUtils.cp("./" + @filename, (path / safe_filename).to_s)
FileUtils.rm("./" + @filename)
end
# Provide metadata so that it doesn't have to find it. Useful for overwriting
# metadata. Must be called if provide_client_keys and provide_spotify are not
# called.
#
# ```
# Song.new(...).provide_metadata(...).grab_it
# ```
def provide_metadata(metadata : JSON::Any) : self
@metadata = metadata
return self
end
# Provide an already authenticated `SpotifySearcher` class. Useful to avoid
# authenticating over and over again. Must be called if provide_metadata and
# provide_client_keys are not called.
#
# ```
# Song.new(...).provide_spotify(SpotifySearcher.new
# .authenticate("XXXXXXXXXX", "XXXXXXXXXXX")).grab_it
# ```
def provide_spotify(spotify : SpotifySearcher) : self
@spotify_searcher = spotify
return self
end
# Provide spotify client keys. Must be called if provide_metadata and
# provide_spotify are not called.
#
# ```
# Song.new(...).provide_client_keys("XXXXXXXXXX", "XXXXXXXXX").grab_it
# ```
def provide_client_keys(client_id : String, client_secret : String) : self
@client_id = client_id
@client_secret = client_secret
return self
end
private def outputter(key : String, index : Int32)
text = @outputs[key][index]
.gsub("%s", @song_name)
.gsub("%a", @artist_name)
print text
end
end
================================================
FILE: src/interact/future.cr
================================================
# copy and pasted from crystal 0.33.1
# https://github.com/crystal-lang/crystal/blob/18e76172444c7bd07f58bf360bc21981b667668d/src/concurrent/future.cr#L138
# :nodoc:
class Concurrent::Future(R)
enum State
Idle
Delayed
Running
Completed
Canceled
end
@value : R?
@error : Exception?
@delay : Float64
def initialize(run_immediately = true, delay = 0.0, &@block : -> R)
@state = State::Idle
@value = nil
@error = nil
@channel = Channel(Nil).new
@delay = delay.to_f
@cancel_msg = nil
spawn_compute if run_immediately
end
def get
wait
value_or_raise
end
def success?
completed? && !@error
end
def failure?
completed? && @error
end
def canceled?
@state == State::Canceled
end
def completed?
@state == State::Completed
end
def running?
@state == State::Running
end
def delayed?
@state == State::Delayed
end
def idle?
@state == State::Idle
end
def cancel(msg = "Future canceled, you reached the [End of Time]")
return if @state >= State::Completed
@state = State::Canceled
@cancel_msg = msg
@channel.close
nil
end
private def compute
return if @state >= State::Delayed
run_compute
end
private def spawn_compute
return if @state >= State::Delayed
@state = @delay > 0 ? State::Delayed : State::Running
spawn { run_compute }
end
private def run_compute
delay = @delay
if delay > 0
sleep delay
return if @state >= State::Canceled
@state = State::Running
end
begin
@value = @block.call
rescue ex
@error = ex
ensure
@channel.close
@state = State::Completed
end
end
private def wait
return if @state >= State::Completed
compute
@channel.receive?
end
private def value_or_raise
raise Exception.new(@cancel_msg) if @state == State::Canceled
value = @value
if value.is_a?(R)
value
elsif error = @error
raise error
else
raise "compiler bug"
end
end
end
# Spawns a `Fiber` to compute *&block* in the background after *delay* has elapsed.
# Access to get is synchronized between fibers. *&block* is only called once.
# May be canceled before *&block* is called by calling `cancel`.
# ```
# d = delay(1) { Process.kill(Process.pid) }
# long_operation
# d.cancel
# ```
def delay(delay, &block : -> _)
Concurrent::Future.new delay: delay, &block
end
# Spawns a `Fiber` to compute *&block* in the background.
# Access to get is synchronized between fibers. *&block* is only called once.
# ```
# f = future { http_request }
# ... other actions ...
# f.get #=> String
# ```
def future(&exp : -> _)
Concurrent::Future.new &exp
end
# Conditionally spawns a `Fiber` to run *&block* in the background.
# Access to get is synchronized between fibers. *&block* is only called once.
# *&block* doesn't run by default, only when `get` is called.
# ```
# l = lazy { expensive_computation }
# spawn { maybe_use_computation(l) }
# spawn { maybe_use_computation(l) }
# ```
def lazy(&block : -> _)
Concurrent::Future.new run_immediately: false, &block
end
================================================
FILE: src/interact/logger.cr
================================================
require "./future"
class Logger
@done_signal = "---DONE---"
@command : String
# *command* is the bash command that you want to run and capture the output
# of. *@log_name* is the name of the log file you want to temporarily create.
# *@sleept* is the time you want to wait before rechecking if the command has
# started yet, probably something you don't want to worry about
def initialize(command : String, @log_name : String, @sleept = 0.01)
# Have the command output its information to a log and after the command is
# finished, append an end signal to the document
@command = "#{command} > #{@log_name} " # standard output to log
@command += "2> #{@log_name} && " # errors to log
@command += "echo #{@done_signal} >> #{@log_name}" #
end
# Run @command in the background and pipe its output to the log file, with
# something constantly monitoring the log file and yielding each new line to
# the block call. Useful for changing the output of binaries you don't have
# much control over.
# Note that the created temp log will be deleted unless the command fails
# its exit or .start is called with delete_file: false
#
# ```
# l = Logger.new(".temp.log", %(echo "CIA spying" && sleep 2 && echo "new veggie tales season"))
# l.start do |output, index|
# case output
# when "CIA spying"
# puts "i sleep"
# when .includes?("veggie tales")
# puts "real shit"
# end
# end
# ```
def start(delete_file = true, &block) : Bool
# Delete the log if it already exists
File.delete(@log_name) if File.exists?(@log_name)
# Run the command in the background
called = future {
system(@command)
}
# Wait for the log file to be written to
while !File.exists?(@log_name)
sleep @sleept
end
log = File.open(@log_name)
log_content = read_file(log)
index = 0
while true
temp_content = read_file(log)
# make sure that there is new data
if temp_content.size > 0 && log_content != temp_content
log_content = temp_content
# break the loop if the command has completed
break if log_content[0] == @done_signal
# give the line and index to the block
yield log_content[0], index
index += 1
end
end
status = called.get
if status == true && delete_file == true
log.delete
end
return called.get
end
# Reads each line of the file into an Array of Strings
private def read_file(file : IO) : Array(String)
content = [] of String
file.each_line do |line|
content.push(line)
end
return content
end
end
================================================
FILE: src/interact/ripper.cr
================================================
require "./logger"
require "../bottle/config"
require "../bottle/styles"
module Ripper
extend self
BIN_LOC = Path[Config.binary_location]
# Downloads the video from the given *video_url* using the youtube-dl binary
# Will create any directories that don't exist specified in *output_filename*
#
# ```
# Ripper.download_mp3("https://youtube.com/watch?v=0xnciFWAqa0",
# "Queen/A Night At The Opera/Bohemian Rhapsody.mp3")
# ```
def download_mp3(video_url : String, output_filename : String)
ydl_loc = BIN_LOC.join("youtube-dl")
# remove the extension that will be added on by ydl
output_filename = output_filename.split(".")[..-2].join(".")
options = {
"--output" => %("#{output_filename}.%(ext)s"), # auto-add correct ext
# "--quiet" => "",
"--verbose" => "",
"--ffmpeg-location" => BIN_LOC,
"--extract-audio" => "",
"--audio-format" => "mp3",
"--audio-quality" => "0",
}
command = ydl_loc.to_s + " " + video_url
options.keys.each do |option|
command += " #{option} #{options[option]}"
end
l = Logger.new(command, ".ripper.log")
o = RipperOutputCensor.new
return l.start do |line, index|
o.censor_output(line, index)
end
end
# An internal class that will keep track of what to output to the user or
# what should be hidden.
private class RipperOutputCensor
@dl_status_index = 0
def censor_output(line : String, index : Int32)
case line
when .includes? "[download]"
if @dl_status_index != 0
print "\e[1A"
print "\e[0K\r"
end
puts line.sub("[download]", " ")
@dl_status_index += 1
if line.includes? "100%"
print " Converting to mp3 ..."
end
end
end
end
end
================================================
FILE: src/interact/tagger.cr
================================================
require "../bottle/config"
# Uses FFMPEG binary to add metadata to mp3 files
# ```
# t = Tags.new("bohem rap.mp3")
# t.add_album_art("a night at the opera album cover.jpg")
# t.add_text_tag("title", "Bohemian Rhapsody")
# t.save
# ```
class Tags
# TODO: export this path to a config file
@BIN_LOC = Config.binary_location
@query_args = [] of String
# initialize the class with an already created MP3
def initialize(@filename : String)
if !File.exists?(@filename)
raise "MP3 not found at location: #{@filename}"
end
@query_args.push(%(-i "#{@filename}"))
end
# Add album art to the mp3. Album art must be added BEFORE text tags are.
# Check the usage above to see a working example.
def add_album_art(image_location : String) : Nil
if !File.exists?(image_location)
raise "Image file not found at location: #{image_location}"
end
@query_args.push(%(-i "#{image_location}"))
@query_args.push("-map 0:0 -map 1:0")
@query_args.push("-c copy")
@query_args.push("-id3v2_version 3")
@query_args.push(%(-metadata:s:v title="Album cover"))
@query_args.push(%(-metadata:s:v comment="Cover (front)"))
@query_args.push(%(-metadata:s:v title="Album cover"))
end
# Add a text tag to the mp3. If you want to see what text tags are supported,
# check out: https://wiki.multimedia.cx/index.php?title=FFmpeg_Metadata
def add_text_tag(key : String, value : String) : Nil
@query_args.push(%(-metadata #{key}="#{value}"))
end
# Run the necessary commands to attach album art to the mp3
def save : Nil
@query_args.push(%("_#{@filename}"))
command = @BIN_LOC + "/ffmpeg " + @query_args.join(" ")
l = Logger.new(command, ".tagger.log")
l.start { |line, start| }
File.delete(@filename)
File.rename("_" + @filename, @filename)
end
end
# a = Tags.new("test.mp3")
# a.add_text_tag("title", "Warwick Avenue")
# a.add_album_art("file.png")
# a.save()
================================================
FILE: src/irs.cr
================================================
require "./bottle/cli"
def main
cli = CLI.new(ARGV)
cli.act_on_args
end
main()
================================================
FILE: src/search/ranking.cr
================================================
alias VID_VALUE_CLASS = String
alias VID_METADATA_CLASS = Hash(String, VID_VALUE_CLASS)
alias YT_METADATA_CLASS = Array(VID_METADATA_CLASS)
module Ranker
extend self
GARBAGE_PHRASES = [
"cover", "album", "live", "clean", "version", "full", "full album", "row",
"at", "@", "session", "how to", "npr music", "reimagined", "version",
"trailer"
]
GOLDEN_PHRASES = [
"official video", "official music video",
]
# Will rank videos according to their title and the user input, returns a sorted array of hashes
# of the points a song was assigned and its original index
# *spotify_metadata* is the metadate (from spotify) of the song that you want
# *yt_metadata* is an array of hashes with metadata scraped from the youtube search result page
# *query* is the query that you submitted to youtube for the results you now have
# ```
# Ranker.rank_videos(spotify_metadata, yt_metadata, query)
# => [
# {"points" => x, "index" => x},
# ...
# ]
# ```
# "index" corresponds to the original index of the song in yt_metadata
def rank_videos(spotify_metadata : JSON::Any, yt_metadata : YT_METADATA_CLASS,
query : String) : Array(Hash(String, Int32))
points = [] of Hash(String, Int32)
index = 0
actual_song_name = spotify_metadata["name"].as_s
actual_artist_name = spotify_metadata["artists"][0]["name"].as_s
yt_metadata.each do |vid|
pts = 0
pts += points_string_compare(actual_song_name, vid["title"])
pts += points_string_compare(actual_artist_name, vid["title"])
pts += count_buzzphrases(query, vid["title"])
pts += compare_timestamps(spotify_metadata, vid)
points.push({
"points" => pts,
"index" => index,
})
index += 1
end
# Sort first by points and then by original index of the song
points.sort! { |a, b|
if b["points"] == a["points"]
a["index"] <=> b["index"]
else
b["points"] <=> a["points"]
end
}
return points
end
# SINGULAR COMPONENT OF RANKING ALGORITHM
private def compare_timestamps(spotify_metadata : JSON::Any, node : VID_METADATA_CLASS) : Int32
# puts spotify_metadata.to_pretty_json()
actual_time = spotify_metadata["duration_ms"].as_i
vid_time = node["duration_ms"].to_i
difference = (actual_time - vid_time).abs
# puts "actual: #{actual_time}, vid: #{vid_time}"
# puts "\tdiff: #{difference}"
# puts "\ttitle: #{node["title"]}"
if difference <= 1000
return 3
elsif difference <= 2000
return 2
elsif difference <= 5000
return 1
else
return 0
end
end
# SINGULAR COMPONENT OF RANKING ALGORITHM
# Returns an `Int` based off the number of points worth assigning to the
# matchiness of the string. First the strings are downcased and then all
# nonalphanumeric characters are stripped.
# If *item1* includes *item2*, return 3 pts.
# If after the items have been blanked, *item1* includes *item2*,
# return 1 pts.
# Else, return 0 pts.
private def points_string_compare(item1 : String, item2 : String) : Int32
if item2.includes?(item1)
return 3
end
item1 = item1.downcase.gsub(/[^a-z0-9]/, "")
item2 = item2.downcase.gsub(/[^a-z0-9]/, "")
if item2.includes?(item1)
return 1
else
return 0
end
end
# SINGULAR COMPONENT OF RANKING ALGORITHM
# Checks if there are any phrases in the title of the video that would
# indicate audio having what we want.
# *video_name* is the title of the video, and *query* is what the user the
# program searched for. *query* is needed in order to make sure we're not
# subtracting points from something that's naturally in the title
private def count_buzzphrases(query : String, video_name : String) : Int32
good_phrases = 0
bad_phrases = 0
GOLDEN_PHRASES.each do |gold_phrase|
gold_phrase = gold_phrase.downcase.gsub(/[^a-z0-9]/, "")
if query.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
next
elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
good_phrases += 1
end
end
GARBAGE_PHRASES.each do |garbage_phrase|
garbage_phrase = garbage_phrase.downcase.gsub(/[^a-z0-9]/, "")
if query.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
next
elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
bad_phrases += 1
end
end
return good_phrases - bad_phrases
end
end
================================================
FILE: src/search/spotify.cr
================================================
require "http"
require "json"
require "base64"
require "../glue/mapper"
class SpotifySearcher
@root_url = Path["https://api.spotify.com/v1/"]
@access_header : (HTTP::Headers | Nil) = nil
@authorized = false
# Saves an access token for future program use with spotify using client IDs.
# Specs defined on spotify's developer api:
# https://developer.spotify.com/documentation/general/guides/authorization-guide/#client-credentials-flow
#
# ```
# SpotifySearcher.new.authorize("XXXXXXXXXX", "XXXXXXXXXX")
# ```
def authorize(client_id : String, client_secret : String) : self
auth_url = "https://accounts.spotify.com/api/token"
headers = HTTP::Headers{
"Authorization" => "Basic " +
Base64.strict_encode "#{client_id}:#{client_secret}",
}
payload = "grant_type=client_credentials"
response = HTTP::Client.post(auth_url, headers: headers, form: payload)
if response.status_code != 200
@authorized = false
return self
end
access_token = JSON.parse(response.body)["access_token"]
@access_header = HTTP::Headers{
"Authorization" => "Bearer #{access_token}",
}
@authorized = true
return self
end
# Check if the class is authorized or not
def authorized? : Bool
return @authorized
end
# Searches spotify with the specified parameters for the specified items
#
# ```
# spotify_searcher.find_item("track", {
# "artist" => "Queen",
# "track" => "Bohemian Rhapsody"
# })
# => {track metadata}
# ```
def find_item(item_type : String, item_parameters : Hash, offset = 0,
limit = 20) : JSON::Any?
query = generate_query(item_type, item_parameters)
url = "search?q=#{query}&type=#{item_type}&limit=#{limit}&offset=#{offset}"
url = @root_url.join(url).to_s
response = HTTP::Client.get(url, headers: @access_header)
error_check(response)
items = JSON.parse(response.body)[item_type + "s"]["items"].as_a
points = rank_items(items, item_parameters)
to_return = nil
begin
# this means no points were assigned so don't return the "best guess"
if points[0][0] <= 0
to_return = nil
else
to_return = get_item(item_type, items[points[0][1]]["id"].to_s)
end
rescue IndexError
to_return = nil
end
# if this triggers, it means that a playlist has failed to be found, so
# the search will be bootstrapped into find_user_playlist
if to_return == nil && item_type == "playlist"
return find_user_playlist(
item_parameters["username"],
item_parameters["name"]
)
end
return to_return
end
# Grabs a users playlists and searches through it for the specified playlist
#
# ```
# spotify_searcher.find_user_playlist("prakkillian", "the little man")
# => {playlist metadata}
# ```
def find_user_playlist(username : String, name : String, offset = 0,
limit = 20) : JSON::Any?
url = "users/#{username}/playlists?limit=#{limit}&offset=#{offset}"
url = @root_url.join(url).to_s
response = HTTP::Client.get(url, headers: @access_header)
error_check(response)
body = JSON.parse(response.body)
items = body["items"]
points = [] of Array(Int32)
items.as_a.each_index do |i|
points.push([points_compare(items[i]["name"].to_s, name), i])
end
points.sort! { |a, b| b[0] <=> a[0] }
begin
if points[0][0] < 3
return find_user_playlist(username, name, offset + limit, limit)
else
return get_item("playlist", items[points[0][1]]["id"].to_s)
end
rescue IndexError
return nil
end
end
# Get the complete metadata of an item based off of its id
#
# ```
# SpotifySearcher.new.authorize(...).get_item("artist", "1dfeR4HaWDbWqFHLkxsg1d")
# ```
def get_item(item_type : String, id : String, offset = 0,
limit = 100) : JSON::Any
if item_type == "playlist"
return get_playlist(id, offset, limit)
end
url = "#{item_type}s/#{id}?limit=#{limit}&offset=#{offset}"
url = @root_url.join(url).to_s
response = HTTP::Client.get(url, headers: @access_header)
error_check(response)
body = JSON.parse(response.body)
return body
end
# The only way this method differs from `get_item` is that it makes sure to
# insert ALL tracks from the playlist into the `JSON::Any`
#
# ```
# SpotifySearcher.new.authorize(...).get_playlist("122Fc9gVuSZoksEjKEx7L0")
# ```
def get_playlist(id, offset = 0, limit = 100) : JSON::Any
url = "playlists/#{id}?limit=#{limit}&offset=#{offset}"
url = @root_url.join(url).to_s
response = HTTP::Client.get(url, headers: @access_header)
error_check(response)
body = JSON.parse(response.body)
parent = PlaylistExtensionMapper.from_json(response.body)
more_tracks = body["tracks"]["total"].as_i > offset + limit
if more_tracks
return playlist_extension(parent, id, offset = offset + limit)
end
return body
end
# This method exists to loop through spotify API requests and combine all
# tracks that may not be captured by the limit of 100.
private def playlist_extension(parent : PlaylistExtensionMapper,
id : String, offset = 0, limit = 100) : JSON::Any
url = "playlists/#{id}/tracks?limit=#{limit}&offset=#{offset}"
url = @root_url.join(url).to_s
response = HTTP::Client.get(url, headers: @access_header)
error_check(response)
body = JSON.parse(response.body)
new_tracks = PlaylistTracksMapper.from_json(response.body)
new_tracks.items.each do |track|
parent.tracks.items.push(track)
end
more_tracks = body["total"].as_i > offset + limit
if more_tracks
return playlist_extension(parent, id, offset = offset + limit)
end
return JSON.parse(parent.to_json)
end
# Find the genre of an artist based off of their id
#
# ```
# SpotifySearcher.new.authorize(...).find_genre("1dfeR4HaWDbWqFHLkxsg1d")
# ```
def find_genre(id : String) : String | Nil
genre = get_item("artist", id)["genres"]
if genre.as_a.empty?
return nil
end
genre = genre[0].to_s
genre = genre.split(" ").map { |x| x.capitalize }.join(" ")
return genre
end
# Checks for errors in HTTP requests and raises one if found
private def error_check(response : HTTP::Client::Response) : Nil
if response.status_code != 200
raise("There was an error with your request.\n" +
"Status code: #{response.status_code}\n" +
"Response: \n#{response.body}")
end
end
# Generates url to run a GET request against to the Spotify open API
# Returns a `String.`
private def generate_query(item_type : String, item_parameters : Hash) : String
query = ""
# parameter keys to exclude in the api request. These values will be put
# in, just not their keys.
query_exclude = ["username"]
item_parameters.keys.each do |k|
# This will map album and track names from the name key to the query
if k == "name"
# will remove the "name:" param from the query
if item_type == "playlist"
query += item_parameters[k] + "+"
else
query += as_field(item_type, item_parameters[k])
end
# check if the key is to be excluded
elsif query_exclude.includes?(k)
next
# if it's none of the above, treat it normally
# NOTE: playlist names will be inserted into the query normally, without
# a parameter.
else
query += as_field(k, item_parameters[k])
end
end
return URI.encode(query.rchop("+"))
end
# Returns a `String` encoded for the spotify api
#
# ```
# query_encode("album", "A Night At The Opera")
# => "album:A Night At The Opera+"
# ```
private def as_field(key, value) : String
return "#{key}:#{value}+"
end
# Ranks the given items based off of the info from parameters.
# Meant to find the item that the user desires.
# Returns an `Array` of `Array(Int32)` or [[3, 1], [...], ...]
private def rank_items(items : Array,
parameters : Hash) : Array(Array(Int32))
points = [] of Array(Int32)
index = 0
items.each do |item|
pts = 0
# Think about whether this following logic is worth having in one method.
# Is it nice to have a single method that handles it all or having a few
# methods for each of the item types? (track, album, playlist)
parameters.keys.each do |k|
val = parameters[k]
# The key to compare to for artist
if k == "artist"
pts += points_compare(item["artists"][0]["name"].to_s, val)
end
# The key to compare to for playlists
if k == "username"
pts_to_add = points_compare(item["owner"]["display_name"].to_s, val)
pts += pts_to_add
pts += -10 if pts_to_add == 0
end
# The key regardless of whether item is track, album,or playlist
if k == "name"
pts += points_compare(item["name"].to_s, val)
end
end
points.push([pts, index])
index += 1
end
points.sort! { |a, b| b[0] <=> a[0] }
return points
end
# Returns an `Int` based off the number of points worth assigning to the
# matchiness of the string. First the strings are downcased and then all
# nonalphanumeric characters are stripped.
# If the strings are the exact same, return 3 pts.
# If *item1* includes *item2*, return 1 pt.
# Else, return 0 pts.
private def points_compare(item1 : String, item2 : String) : Int32
item1 = item1.downcase.gsub(/[^a-z0-9]/, "")
item2 = item2.downcase.gsub(/[^a-z0-9]/, "")
if item1 == item2
return 3
elsif item1.includes?(item2)
return 1
else
return 0
end
end
end
# puts SpotifySearcher.new()
# .authorize("XXXXXXXXXXXXXXX",
# "XXXXXXXXXXXXXXX")
# .find_item("playlist", {
# "name" => "Brain Food",
# "username" => "spotify"
# # "name " => "A Night At The Opera",
# # "artist" => "Queen"
# # "track" => "Bohemian Rhapsody",
# # "artist" => "Queen"
# })
================================================
FILE: src/search/youtube.cr
================================================
require "http"
require "xml"
require "json"
require "uri"
require "./ranking"
require "../bottle/config"
require "../bottle/styles"
module Youtube
extend self
VALID_LINK_CLASSES = [
"yt-simple-endpoint style-scope ytd-video-renderer",
"yt-uix-tile-link yt-ui-ellipsis yt-ui-ellipsis-2 yt-uix-sessionlink spf-link ",
]
# Note that VID_VALUE_CLASS, VID_METADATA_CLASS, and YT_METADATA_CLASS are found in ranking.cr
# Finds a youtube url based off of the given information.
# The query to youtube is constructed like this:
# "<song_name> <artist_name> <search terms>"
# If *download_first* is provided, the first link found will be downloaded.
# If *select_link* is provided, a menu of options will be shown for the user to choose their poison
#
# ```
# Youtube.find_url("Bohemian Rhapsody", "Queen")
# => "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
# ```
def find_url(spotify_metadata : JSON::Any,
flags = {} of String => String) : String?
search_terms = Config.search_terms
select_link = flags["select"]?
song_name = spotify_metadata["name"].as_s
artist_name = spotify_metadata["artists"][0]["name"].as_s
human_query = "#{song_name} #{artist_name} #{search_terms.strip}"
params = HTTP::Params.encode({"search_query" => human_query})
response = HTTP::Client.get("https://www.youtube.com/results?#{params}")
yt_metadata = get_yt_search_metadata(response.body)
if yt_metadata.size == 0
puts "There were no results for this query on youtube: \"#{human_query}\""
return nil
end
root = "https://youtube.com"
ranked = Ranker.rank_videos(spotify_metadata, yt_metadata, human_query)
if select_link
return root + select_link_menu(spotify_metadata, yt_metadata)
end
begin
puts Style.dim(" Video: ") + yt_metadata[ranked[0]["index"]]["title"]
return root + yt_metadata[ranked[0]["index"]]["href"]
rescue IndexError
return nil
end
exit 1
end
# Presents a menu with song info for the user to choose which url they want to download
private def select_link_menu(spotify_metadata : JSON::Any,
yt_metadata : YT_METADATA_CLASS) : String
puts Style.dim(" Spotify info: ") +
Style.bold("\"" + spotify_metadata["name"].to_s) + "\" by \"" +
Style.bold(spotify_metadata["artists"][0]["name"].to_s + "\"") +
" @ " + Style.blue((spotify_metadata["duration_ms"].as_i / 1000).to_i.to_s) + "s"
puts " Choose video to download:"
index = 1
yt_metadata.each do |vid|
print " " + Style.bold(index.to_s + " ")
puts "\"" + vid["title"] + "\" @ " + Style.blue((vid["duration_ms"].to_i / 1000).to_i.to_s) + "s"
index += 1
if index > 5
break
end
end
input = 0
while true # not between 1 and 5
begin
print Style.bold(" > ")
input = gets.not_nil!.chomp.to_i
if input < 6 && input > 0
break
end
rescue
puts Style.red(" Invalid input, try again.")
end
end
return yt_metadata[input-1]["href"]
end
# Finds valid video links from a `HTTP::Client.get` request
# Returns an `Array` of `NODES_CLASS` containing additional metadata from Youtube
private def get_yt_search_metadata(response_body : String) : YT_METADATA_CLASS
yt_initial_data : JSON::Any = JSON.parse("{}")
response_body.each_line do |line|
# timestamp 11/8/2020:
# youtube's html page has a line previous to this literally with 'scraper_data_begin' as a comment
if line.includes?("var ytInitialData")
# Extract JSON data from line
data = line.split(" = ")[2].delete(';')
dataEnd = (data.index("</script>") || 0) - 1
begin
yt_initial_data = JSON.parse(data[0..dataEnd])
rescue
break
end
end
end
if yt_initial_data == JSON.parse("{}")
puts "Youtube has changed the way it organizes its webpage, submit a bug"
puts "saying it has done so on https://github.com/cooperhammond/irs"
exit(1)
end
# where the vid metadata lives
yt_initial_data = yt_initial_data["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"]
video_metadata = [] of VID_METADATA_CLASS
i = 0
while true
begin
# video title
raw_metadata = yt_initial_data[0]["itemSectionRenderer"]["contents"][i]["videoRenderer"]
metadata = {} of String => VID_VALUE_CLASS
metadata["title"] = raw_metadata["title"]["runs"][0]["text"].as_s
metadata["href"] = raw_metadata["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"].as_s
timestamp = raw_metadata["lengthText"]["simpleText"].as_s
metadata["timestamp"] = timestamp
metadata["duration_ms"] = ((timestamp.split(":")[0].to_i * 60 +
timestamp.split(":")[1].to_i) * 1000).to_s
video_metadata.push(metadata)
rescue IndexError
break
rescue Exception
end
i += 1
end
return video_metadata
end
# Returns as a valid URL if possible
#
# ```
# Youtube.validate_url("https://www.youtube.com/watch?v=NOTANACTUALVIDEOID")
# => nil
# ```
def validate_url(url : String) : String | Nil
uri = URI.parse url
return nil if !uri
query = uri.query
return nil if !query
# find the video ID
vID = nil
query.split('&').each do |q|
if q.starts_with?("v=")
vID = q[2..-1]
end
end
return nil if !vID
url = "https://www.youtube.com/watch?v=#{vID}"
# this is an internal endpoint to validate the video ID
params = HTTP::Params.encode({"format" => "json", "url" => url})
response = HTTP::Client.get "https://www.youtube.com/oembed?#{params}"
return nil unless response.success?
res_json = JSON.parse(response.body)
title = res_json["title"].as_s
puts Style.dim(" Video: ") + title
return url
end
end