[
  {
    "path": ".appveyor.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-24 16:19:35 +0000 (Mon, 24 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             A p p V e y o r   C I\n# ============================================================================ #\n\n# https://www.appveyor.com/docs/appveyor-yml/\n\nimage: Ubuntu\n\n# workaround for default JDK9 have old CA certs:\n#\n#   https://github.com/appveyor/ci/issues/3833\n#\n#   https://www.appveyor.com/docs/getting-started-with-appveyor-for-linux/#configuring-language-stack\n#\nstack: jdk 15\n\nskip_commits:\n  files:\n    - docs/*\n    - '**/*.md'\n\n# https://www.appveyor.com/docs/how-to/ssh-to-build-worker/\nenvironment:\n  APPVEYOR_SSH_KEY: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvihSRU+YjBKvKiacDfUoZ7ghoVMcwNh4cWIYUNFGZosXOzNtyOcBpIb71TCgLFhOd+aMWKXCEC67BpNSIjt+a/FLD27AwmgVHv6cPlE3G0JJ9zmIrNmx9511dshTsxUW2O0SbYG+3InuO7FUkSrld+kA1OucyjgmZU7/+Cs9shpAEOaIVYmGlpDGRucAHpwtckvdgRTtnA3WNZ/Qg1vU6Ik4Xm03vjrW6lSiuTffYO1kbdcMQ4IZBlzfmovOtXQ0PomvN5NMCpgOyQuoNlvyS11tOXoqNiWOkiLE15XEzAQth9hHbNiH8jHJbAtkHqWWh0KK4IUyNGvoL6QfNxsTlw== hari@anotherdimension\n\n# enable SSH session accessible via my public key\n#init:\n#  - sh: curl -sflL 'https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-ssh.sh' | bash -e -\n\n# more useful at end to leverage .appveyor.yml tweaks like disabling broken mssql repo/dependencies, checking out project and building the core stuff happen first so we don't have to do all that manually in SSH session\non_finish:\n  # set this in Settings -> Environment dynamically instead of here\n  #- sh: export APPVEYOR_SSH_BLOCK=true\n  #\n  # workaround for https://github.com/appveyor/ci/issues/3373\n  #            and https://github.com/appveyor/ci/issues/3384\n  #\n  # has since been added to AppVeyor's own scripts:\n  #\n  #     https://github.com/appveyor/ci/pull/3385\n  #\n  #- sh: curl -sflL 'https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/install/install_openssh.sh' | bash -e -\n  #\n  # https://www.appveyor.com/docs/how-to/ssh-to-build-worker/\n  - sh: if [ \"$APPVEYOR_SSH_BLOCK\" = true ]; then curl -sflL 'https://raw.githubusercontent.com/appveyor/ci/master/scripts/enable-ssh.sh' | bash -e -; fi\n\ninstall:\n  # workaround for:\n  # Some packages could not be installed. This may mean that you have\n  # requested an impossible situation or if you are using the unstable\n  # distribution that some required packages have not yet been created\n  # or been moved out of Incoming.\n  # The following information may help to resolve the situation:\n  #\n  # The following packages have unmet dependencies:\n  #  mssql-server : Depends: libsasl2-modules-gssapi-mit but it is not going to be installed\n  #  E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.\n  #  bash-tools/Makefile.in:272: recipe for target 'apt-packages' failed\n  #  make[2]: *** [apt-packages] Error 123\n  #  make[2]: Leaving directory '/home/appveyor/projects/pylib'\n  #  bash-tools/Makefile.in:212: recipe for target 'system-packages' failed\n  #\n  #  adding \"|| :\" to the end of these commands causes them to be silently ignored!\n  - sudo sed -i '/https:\\/\\/packages.microsoft.com\\/ubuntu\\/.*\\/mssql-server/d' /etc/apt/sources.list\n  - sudo apt purge -yq --allow-change-held-packages mssql-server\n  # this prevents conflicts installing default-jdk - see https://github.com/appveyor/ci/issues/3411\n  #- dpkg -l | awk '/openjdk/{print $2}' | DEBIAN_FRONTEND=noninteractive xargs sudo apt-get remove -y --allow-change-held-packages\n  - setup/ci_bootstrap.sh\n  - make\n\ntest_script:\n  - make test\n\nbuild: off\n"
  },
  {
    "path": ".bash.d/Makefile",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:56:53 +0000 (Sun, 17 Jan 2016)\n#\n#  vim:ts=4:sts=4:sw=4:noet\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\ninclude ../Makefile.in\n\nREPO := HariSekhon/DevOps-Bash-tools\n\nBASH_TOOLS := ..\n\n.PHONY: readme\nreadme:\n\t@source $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://github.com/$(REPO)/blob/master/.bash.d/README.md\"\n"
  },
  {
    "path": ".bash.d/README.md",
    "content": "Advanced Bashrc Code - Interactive Functions, Aliases and Shell Customizations\n==============================================================\n\nAdvanced bashrc code I've been using for ~15 years, I've ported nearly 5000 lines to this public repo so far.\n\nAll `*.sh` files in this directory are automatically sourced by `.bashrc` at the top level which is itself designed to be sourced in your `$HOME/.bashrc`.\n\nTo disable any these source files, simply rename them to not match the `*.sh` glob, eg. => `*.sh.disabled`.\n\n- `aliases.sh` - general aliases\n- `functions.sh` - general functions eg. `pass` which prompts for a password which is saved to an environment variable and auto-populated in various top-level API querying scripts built on `curl_auth.sh`\n- `env.sh` - general environment variables and var/unvar functions for setting environment variables for the current and all new shell sessions\n- `paths.sh` - deduplicated adding to `$PATH` for lots of common places (eg. /usr/sbin, /usr/local/bin, ~/bin) and commands to clearly print one path per line for Bash `$PATH`, Perl `@INC` and Python `sys.path`. Also contains technology specific paths when there is no `<technology>.sh` file. All other includes use `add_PATH()` function defined here.\n- `<technology>.sh` - aliases, functions and environment variables to make interactive day-to-day use of a specific technologies easier\n  - Cloud / Containerization / Virtualization:\n    - `aws.sh` - [AWS](https://aws.amazon.com) functions:\n      - `aws_env` - populates credentials from `~/.aws/credentials` / `~/.boto` section given as an argument to `$AWS_ACCESS_KEY` and `$AWS_SECRET_KEY` environment variables and sets `$AWS_PROFILE` to the profile name (defaults to the 'default' profile and creds if no argument is specified)\n      - `aws_envs` - prints the available envs configured in the aws credentials file and stars the one currently in use\n      - `awk_token` - generates a 24-hour MFA session token, exports it as `$AWS_SESSION_TOKEN` for use with [AWS CLI](https://aws.amazon.com/cli/), and saves it to `~/.aws/token` for loading to other shells that call `aws_env`\n    - `docker.sh` - [Docker](https://www.docker.com/) convenient aliases and functions like clearing old containers and dangling image layers to clean up space\n    - `kubernetes.sh` - [Kubernetes](https://kubernetes.io/) aliases and functions, managing contexts and namespaces even for periodically regenerated `.kube/config` with refreshed embedded certificates, switching between open source [Kubernetes](https://kubernetes.io/) and Redhat [OpenShift](https://www.openshift.com/) `kubectl` and `oc` commands, automating getting authentication token and Kubernetes API endpoints\n    - `vagrant.sh` - [Vagrant](https://www.vagrantup.com/) aliases and functions\n  - Automation / Distributed Systems:\n    - `ansible.sh` - [Ansible](https://www.ansible.com) aliases and environment variables\n    - `kafka.sh` - [Kafka](http://kafka.apache.org/) environment variables for Kerberos security and CLI appropriate heap size (avoids heap allocation failures on VMs that otherwise default to using larger server configured heap size), avoiding need for common broker and zookeeper arguments when using `kafka_wrappers/` scripts by setting your Kafka broker and zookeeper addresses once instead of in every command\n  - Coding:\n    - `git.sh` - [Git](https://git-scm.com/) aliases and functions\n    - `mercurial.sh` - [Mercurial](https://www.mercurial-scm.org/) aliases and functions\n    - `svn.sh` - [Svn](https://subversion.apache.org) aliases and functions\n    - `java.sh` - [Java](https://www.java.com/en/) detection and setting of `$JAVA_HOME` for Linux and Mac environments\n  - OS:\n    - `apple.sh` - [Apple Mac OS X / macOS](https://en.wikipedia.org/wiki/MacOS) specific tricks\n    - `linux.sh` - [Linux](https://en.wikipedia.org/wiki/Linux) specific miscellaneous bits like X.org\n    - `network.sh` - network aliases and functions\n    - `ssh.sh` - SSH convenience functions and key management\n    - `ssh-agent.sh` / `gpg-agent.sh` - auto-starts SSH and GPG agents if not already running, stores and auto-sources their details for new shells to automatically use them\n    - `title.sh` - auto-title tricks for Screen and Terminals\n\nMore script related functions can be found in the [lib/](https://github.com/HariSekhon/DevOps-Bash-tools/tree/master/lib) directory at the top level.\n"
  },
  {
    "path": ".bash.d/aliases.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230,SC2139\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         G e n e r a l   A l i a s e s\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/paths.sh\"\n\n# manual local aliases\n# shellcheck disable=SC1090,SC1091\n[ -f ~/.aliases ] && . ~/.aliases\n\n# bash_tools defined in .bashrc\n# shellcheck disable=SC2154\nexport bashrc=~/.bashrc\nexport bashrc2=\"$bash_tools/.bashrc\"\nalias reload='. $bashrc'\nalias r='reload'\nalias rq='set +x; . $bashrc; set -x'\nalias bashrc='$EDITOR $bashrc && reload'\nalias bashrc2='$EDITOR $bashrc2 && reload'\nalias bashrclocal='$EDITOR $bashrc.local; reload'\nalias bashrc3=bashrclocal\nalias v='vim'\nalias vimrc='$EDITOR ~/.vimrc'\nalias screenrc='$EDITOR ~/.screenrc'\nalias aliases='$EDITOR $bashd/aliases.sh'\nalias ae=aliases\nalias be=bashrc\nalias be2=bashrc2\nalias be3=bashrc3\nalias se=screenrc\nalias ve=vimrc\nalias creds='$EDITOR ~/.env/creds'\nalias pbc=pbcopy\nalias pbp=pbpaste\n# keep emacs with no window, use terminal, not X, otherwise I'd run xemacs...\n#alias emacs=\"emacs -nw\"\n#em(){ emacs \"$@\" ; }\n#alias em=emacs\n#alias e=em\n#xe(){ xemacs $@ & }\n#alias x=xe\n\n# from DevOps-Perl-tools repo which must be in $PATH\n# done via functions.sh now\n#alias new=new.pl\n\n# not present on Mac\n#type tailf &>/dev/null || alias tailf=\"tail -f\"\nalias tailf=\"tail -f\"  # tail -f is better than tailf anyway\nalias mv='mv -i'\nalias cp='cp -i'\n#alias rm='rm -i'\n# allows to re-use custommized less behaviour throughout profile without duplicating options\n#less='less -RFXig'\n#alias less='$less'\nexport LESS=\"-RFXig --tabs=4\"\n# will require LESS=\"-R\"\nif type -P pygmentize &>/dev/null; then\n    # shellcheck disable=SC2016\n    export LESSOPEN='| \"$bash_tools/python/pygmentize.sh\" \"%s\"'\nfi\nalias l='less'\nalias m='more'\nalias vi='vim'\n# used by vagrant now\n#alias v='vim'\nalias grep='grep --color=auto'\n# in functions.sh for multiple args now\n#alias envg=\"env | grep -i\"\nalias dec=\"decomment.sh\"\n\nalias dns='cat /etc/resolv.conf'\nalias hosts='sudo $EDITOR /etc/hosts'\nalias repos='$EDITOR $bash_tools/setup/repos.txt'\n\nalias path=\"echo \\$PATH | tr ':' '\\\\n' | less\"\nalias paths=path\n\nalias tmp=\"cd /tmp\"\n\nalias mksupportdir=\"mkdir -v support-bundle-$(date '+%F_%H%M')\"\n\n# not as compatible, better to call pypy explicitly or in #! line\n#if type -P pypy &>/dev/null; then\n#    alias python=pypy\n#fi\n\n# shellcheck disable=SC2139\nbt=\"$(dirname \"${BASH_SOURCE[0]}\")/..\"\nexport bt\nalias bt='sti bt; cd $bt'\n\n# shellcheck disable=SC2154\nexport bashd=\"$bash_tools/.bash.d\"\nalias bashd='sti bashd; cd $bashd'\n\n#alias cleanshell='env - bash --rcfile /dev/null'\nalias cleanshell='env - bash --norc --noprofile'\nalias newshell='exec bash'\nalias rr='newshell'\n\nalias record=script\n\nalias dl=\"BACKGROUND_VIDEO=1 youtube_download_video.sh\"\n#alias vidopen=\"vidopen.sh\"\n#alias vidopenplay=\"PLAY_VIDEO=1 vidopen.sh\"\n\nalias l33tmode='welcome; retmode=on; echo l33tm0de on'\nalias leetmode=l33tmode\n\nalias hist=history\nalias clhist='HISTSIZE=0; HISTSIZE=5000'\nalias nohist='unset HISTFILE'\nalias histgrep='history | grep'\n\nexport LS_OPTIONS='-F'\nif is_mac; then\n    export CLICOLOR=1 # equiv to using -G switch when calling\nelse\n    export LS_OPTIONS=\"$LS_OPTIONS --color=auto\"\n    export PS_OPTIONS=\"$LS_OPTIONS -L\"\nfi\n\nalias ls='ls $LS_OPTIONS'\n# omit . and .. in listall with -A instead of -a\nalias lA='ls -lA $LS_OPTIONS'\nalias la='ls -la $LS_OPTIONS'\nalias ll='ls -l $LS_OPTIONS'\nalias lh='ls -lh $LS_OPTIONS'\nalias lr='ls -ltrh $LS_OPTIONS'\nalias ltr='lr'\nalias lR='ls -lRh $LS_OPTIONS'\n\n# shellcheck disable=SC2086\nlw(){ ls -lh $LS_OPTIONS \"$(type -P \"$@\")\"; }\n\n# shellcheck disable=SC2086,SC2012\nlll(){ ls -l \"$(readlink -f \"${@:-.}\")\" | less -R; }\n\nalias cd..='cd ..'\nalias ..='cd ..'\nalias ...='cd ../..'\nalias ....='cd ../../..'\n#up(){\n#    local times=\"${1:-1}\"\n#    if ! [[ \"$times\" =~ ^[[:digit:]]$ ]]; then\n#        echo \"How many directories to go up\"\n#        echo\n#        echo \"usage: up <num>\"\n#        return 1\n#    fi\n#    while [ \"$times\" -gt 0 ]; do\n#        cd ..\n#        times=$((times - 1))\n#    done\n#}\n# use bare 'cd' instead, it's more standard\n#alias ~='cd ~'\n\nalias screen='screen -T $TERM'\n#alias mt=multitail\n#alias halt='shutdown -h now -P'\n# my pytools github repo\nalias ht='headtail.py'\n\nalias run='run.sh'\n\n# ============================================================================ #\n#           GitHub / GitLab / BitBucket / Azure DevOps repo checkouts\n# ============================================================================ #\n\nexport github=~/github\nexport gitlab=~/gitlab\nexport azure_devops=~/azure-devops\nalias github=\"sti github; cd '$github'\";\nexport work=\"$github/work\"\nalias work=\"sti work; cd '$work'\"\n\nalias btup=\"bt; u; cd -\"\n\nexport bitbucket=~/bitbucket\nalias bitb='cd $bitbucket'\n# clashes with bitbucket-cli\n#alias bitbucket='cd $bitbucket'\n# used to gitbrowse to bitbucket now in git.sh\n#alias bb=bitbucket\n\nalias diag=diagrams\n\naliasdir(){\n    local directory=\"$1\"\n    local suffix=\"${2:-}\"\n    [ -d \"$directory\" ] || return 0\n    name=\"${directory##*/}\"\n    name=\"${name//-/_}\"\n    name=\"${name//./_}\"\n    name=\"${name// /}\"\n    # alias terraform /tf -> terra\n    if [[ \"$name\" =~ ^(terraform|tf)$ ]]; then\n        name=\"terra\"\n    fi\n    if [ -z \"${!name:-}\" ]; then\n        export \"$name\"=\"$directory\"\n    fi\n    # don't clash with any binaries\n    #if ! type -P \"${name}${suffix}\" &>/dev/null; then\n    # don't clash with binaries or any previous defined aliases or functions\n    if ! type \"${name}${suffix}\" &>/dev/null; then\n        # shellcheck disable=SC2139,SC2140\n        alias \"${name}${suffix}\"=\"sti $name; cd $directory\"\n    elif ! type \"g${name}${suffix}\" &>/dev/null; then\n        # shellcheck disable=SC2139,SC2140\n        alias \"g${name}${suffix}\"=\"sti $name; cd $directory\"\n    fi\n}\n\nfor basedir in \"$github\" \"$gitlab\" \"$bitbucket\" \"$azure_devops\"; do\n    if [ -d \"$basedir\" ]; then\n        for directory in \"$basedir/\"*; do\n            aliasdir \"$directory\"\n            if [[ \"$directory\" =~ /work$ ]]; then\n                for workdir in \"$directory/\"*; do\n                    aliasdir \"$workdir\" \"w\"  # work dirs should have a w suffix\n                done\n            fi\n        done\n    fi\ndone\n\n\ndoc_alias(){\n    local docpath=\"$1\"\n    local prefix=\"${2:-d}\"\n    [ -f \"$docpath\" ] || return 1\n    docfile=\"${docpath##*/}\"\n    if ! [[ \"$docfile\" =~ \\.(txt|md)$ ]]; then\n        if [[ \"$docfile\" =~ \\. ]]; then\n            return 1\n        fi\n    fi\n    # slows down shell creation, will drain battery\n#    if [ -L \"$docpath\" ]; then\n#        # brew install coreutils to get greadlink since Mac doesn't have readlink -f\n#        if type -P greadlink &>/dev/null; then\n#            docfile=\"$(greadlink -f \"$docpath\")\"\n#        else\n#            docfile=\"$(readlink -f \"$docpath\")\"\n#        fi\n#    fi\n    #local count=0\n    #[ -f ~/docs/$docfile ] && ((count+=1))\n    #[ -f \"$github/docs/$docfile\" ] && ((count+=1))\n    #[ -f \"$bitbucket/docs/$docfile\" ] && ((count+=1))\n    #if [ $count -gt 1 ]; then\n    #    echo \"WARNING: $docfile conflicts with existing alias, duplicate doc '$docfile' among ~/docs, ~/github/docs, ~/bitbucket/docs?\"\n    #    return\n    #fi\n    local shortname=\"${docfile%.md}\"\n    local shortname=\"${shortname%.txt}\"\n    # shellcheck disable=SC2139,SC2140\n    alias \"${prefix}${shortname}\"=\"ti ${docpath##*/}; \\$EDITOR $docpath\"\n}\n\nfor x in ~/docs/* \"$github\"/docs/* \"$bitbucket\"/docs/*; do\n    doc_alias \"$x\" || :\ndone\n\nalias know=\"knowledge\"\nfor x in ~/knowledge/* \"$github\"/knowledge/* \"$bitbucket\"/knowledge/*; do\n    doc_alias \"$x\" k || :\ndone\n\n# ============================================================================ #\n\n# set in ansible.sh\n#alias a='ansible -Db'\nalias anonymize='anonymize.py'\nalias an='anonymize -a'\nalias bc='bc -l'\nalias chromekill='pkill -f \"Google Chrome Helper\"'\nalias eclipse='~/eclipse/Eclipse.app/Contents/MacOS/eclipse';\nalias visualvm='~/visualvm/bin/visualvm'\n\nalias tmpl=templates\n\n# using brew version on Mac\npmd_opts=\"-R rulesets/java/quickstart.xml -f text\"\nif is_mac; then\n    # yes evaluate $pmd_opts here\n    # shellcheck disable=SC2139\n    pmd=\"pmd $pmd_opts\"\nelse\n    for x in ~/pmd-bin-*; do\n        if [ -f \"$x/bin/run.sh\" ]; then\n            # yes evaluate $x here, and don't export it's lazy evaluated in alias below\n            # shellcheck disable=SC2139,SC2034\n            pmd=\"$x/bin/run.sh pmd $pmd_opts\"\n        fi\n    done\nfi\nalias pmd='$pmd'\n\n# from DevOps Perl tools repo - like uniq but doesn't require pre-sorting keeps the original ordering\n# Devops Golang tools uniq2 should be on path instead now\n#alias uniq2=uniq_order_preserved.pl\n\n# for piping from grep\nalias uniqfiles=\"sed 's/:.*//;/^[[:space:]]*$/d' | sort -u\"\n\nexport etc=~/etc\nalias etc='cd $etc'\n\n\nalias distro='cat /etc/*release /etc/*version 2>/dev/null'\nalias trace=traceroute\nalias t='$EDITOR ~/temp-notes.txt'\n# causes more problems than it solves on a slow machine missing the prompt\n#alias y=yes\nalias t2='$EDITOR ~/tmp2'\nalias t3='$EDITOR ~/tmp3'\n#alias tg='traceroute www.google.com'\n#alias sec='ps -ef| grep -e arpwatc[h] -e swatc[h] -e scanlog[d]'\n\n\nexport lab=~/lab\nalias lab='cd $lab'\n\n# Auto-alias uppercase directories in ~ like Desktop and Downloads\n#for dir in $(find ~ -maxdepth 1 -name '[A-Z]*' -type d); do [ -d \"$dir\" ] && alias ${dir##*/}=\"cd '$dir'\"; done\n\nexport Downloads=~/Downloads\nexport Documents=~/Documents\nalias Downloads='cd \"$Downloads\"'\nalias Documents='cd \"$Documents\"'\nexport down=\"$Downloads\"\nexport docu=\"$Documents\"\nalias down='cd \"$Downloads\"'\nalias docu='cd \"$Documents\"'\nalias doc='cd ~/docs'\n\nexport desktop=~/Desktop\nexport desk=\"$desktop\"\nalias desktop='cd \"$desktop\"'\nalias desk=desktop\n\nexport screenshots=~/Desktops/Screenshots\nalias screenshots='cd \"$screenshots\"'\n\nexport bin=~/bin\nalias bin=\"cd $bin\"\n\nalias todo='ti T; $EDITOR ~/TODO'\nalias TODO=\"todo\"\nalias don='ti D; $EDITOR ~/DONE'\nalias DON=don\n\n# drive => Google Drive\nexport google_drive=~/drive\nexport drive=\"$google_drive\"\nalias drive='cd \"$drive\"'\n\nfor v in ~/github/pytools/validate_*.py; do\n    z=\"${v##*/}\"\n    z=\"${z#validate_}\"\n    z=\"${z%.py}\"\n    # needs to expand now for dynamic alias creation\n    # shellcheck disable=SC2139,SC2140\n    alias \"v$z\"=\"$v\"\ndone\n\n# in some environments I do ldap with Kerberos auth - see ldapsearch.sh script at top level which is more flexible with pre-tuned environment variables\n#alias ldapsearch=\"ldapsearch -xW\"\n#alias ldapadd=\"ldapadd -xW\"\n#alias ldapmodify=\"ldapmodify -xW\"\n#alias ldapdelete=\"ldapdelete -xW\"\n#alias ldappasswd=\"ldappasswd -xW\"\n#alias ldapwhoami=\"ldapwhoami -xW\"\n#alias ldapvi=\"ldapvi -b dc=domain,dc=local -D cn=admin,dc=domain,dc=local\"\n\nalias fluxkeys='$EDITOR ~/.fluxbox/keys'\nalias fke=fluxkeys\nalias fluxmenu='$EDITOR ~/.fluxbox/mymenu'\nalias fme=fluxmenu\nalias mymenu=fluxmenu\nalias menu=mymenu\n\n# trigger script in ~/.config/mpv/scripts/delete-on-eof.lua to delete a video once it has been completely watched\nalias mpvd=\"MPV_DELETE_ON_EOF=1 mpv --speed=2\"\n"
  },
  {
    "path": ".bash.d/android.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-27 02:08:34 +0700 (Thu, 27 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nexport ANDROID_HOME=\"$HOME/Android/Sdk\"\n\n# Doesn't work\n#export ANDROID_SDK_ROOT=\"$ANDROID_HOME\"\n\nadd_PATH \"$ANDROID_HOME/platform-tools\"\n\nadd_PATH \"$ANDROID_HOME/cmdline-tools/latest/bin\"\nadd_PATH \"$ANDROID_HOME/cmdline-tools/bin\"\n\nfor x in \"$ANDROID_HOME/build-tools/\"*; do\n    if [ -d \"$x\" ]; then\n        add_PATH \"$x\"\n    fi\ndone\n"
  },
  {
    "path": ".bash.d/ansible.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2139\n#\n#  Author: Hari Sekhon\n#  Date: 2014-07-13 16:56:14 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 A n s i b l e\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# order of precedence:\n#\n#   $ANSIBLE_CONFIG\n#   $PWD/ansible.cfg\n#   $HOME/.ansible.cfg\n#   /etc/ansible/ansible.cfg\n#\n# so don't set ANSIBLE_CONFIG because it'll cause issues in work repos\n# which would otherwise correctly default to $PWD/ansible.cfg\n#\n#export ANSIBLE_CONFIG=~/.ansible.cfg  # symlinked to $bash_tools/configs/.ansible.cfg\n\nif [ -n \"${ANSIBLE_HOME:-}\" ]; then\n    add_PATH PYTHONPATH \"$ANSIBLE_HOME/lib\"\n    add_PATH ANSIBLE_LIBRARY \"$ANSIBLE_HOME/library\"\n    # resets man search path, breaking man lookups\n    #add_PATH MANPATH \"$ANSIBLE_HOME/docs/man\"\nfi\n\n# don't set this in case it causes issues in work repos\n#if [ -f ~/etc/ansible/hosts ]; then\n#    export ANSIBLE_HOSTS=~/etc/ansible/hosts\n#fi\n\n# set in ~/.ansible.cfg now\n#export ANSIBLE_HOST_KEY_CHECKING=False\n\n# -D diff switch requires newish ansible, doesn't work on 1.7\n# -b - matter of preference between using lots of sudo in manifests or not, better to remove it for tighter authz & logging purposes in governed environments\nansible_opts=\"-D -b\"\n\nalias a=ansible\n# expand now, no dynamic surprises\nalias ansible=\"ansible $ansible_opts\"\nalias ansible_playbook=\"ansible-playbook $ansible_opts\"\n#alias ansible_playbook_vault=\"ansible-playbook $ansible_opts --ask-vault-pass\"\nalias ansible_playbook_vault=\"ansible-playbook $ansible_opts --vault-id '$bash_tools/bin/vault_pass.sh'\"\n"
  },
  {
    "path": ".bash.d/argocd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-07 16:03:39 +0000 (Fri, 07 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  A r g o C D\n# ============================================================================ #\n\n# gets default admin pw and copies it to clipboard for quick pasting to UI\nalias argopass=\"argocd_password.sh | copy_to_clipboard.sh\"\n\n# XXX: set the following in your local environment:\n#\n# ARGOCD_SERVER=localhost:8080  # without the http:// or https:// prefix\n# ARGOCD_AUTH_TOKEN='<token>'\n\nif ! [[ \"${ARGOCD_OPTS:-}\" =~ --grpc-web ]]; then\n    export ARGOCD_OPTS=\"$ARGOCD_OPTS --grpc-web\"\nfi\n#export ARGOCD_OPTS=\"--grpc-web --insecure\"  # only in local dev\n\nargosync(){\n    local seconds=\"${1:-60}\"\n    shift || :\n    if [ -z \"${ARGOCD_APP:-}\" ]; then\n        namespace=\"${K8S_NAMESPACE:-$(kubectl_namespace)}\"\n        if argocd app list -o name | grep -Fxq \"$namespace\"; then\n            ARGOCD_APP=\"$namespace\"\n        fi\n    fi\n    if [ -n \"${ARGOCD_APP:-}\" ]; then\n        timestamp \"ArgoCD Syncing App '$ARGOCD_APP'\"\n        echo\n        argocd app sync \"$ARGOCD_APP\" --force \"$@\"\n        argocd app wait \"$ARGOCD_APP\" --timeout \"$seconds\" \"$@\"\n        echo\n        if [ \"$ARGOCD_APP\" = argocd ]; then\n            for x in projects apps; do\n                if argocd app list -o name | grep -Fxq \"$x\"; then\n                    ARGOCD_APP=\"$x\" argosync \"$@\"\n                fi\n            done\n        fi\n    else\n        echo \"\\$ARGOCD_APP is not set\" >&2\n        return 1\n    fi\n}\n"
  },
  {
    "path": ".bash.d/aws-cloudshell.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 18:54:23 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# This is designed to mimick the standard GCP CloudShell behaviour\n# of re-customizing a new CloudShell\n\nif ! [ \"${AWS_EXECUTION_ENV:-}\" = \"CloudShell\" ]; then\n    return\nfi\n\ncustomize_script=~/.aws_customize_environment\n\nlockdir=/tmp/aws_customize_environment.lock\n\ncompletion_semaphore=\"/.aws_customize_environment_completed\"\n\nif ! [ -f \"$customize_script\" ]; then\n    return\nfi\n\nif [ -f \"$completion_semaphore\" ]; then\n    return\nfi\n\n# XXX: if the lockdir is more than 2 hours old and the completion semaphore wasn't found, remove the lockdir to try again\nif [ -d \"$lockdir\" ] && ! [ -f \"$completion_semaphore\" ]; then\n    lockdir_epoch=\"$(stat -c %Y \"$lockdir\")\"\n    current_epoch=\"$(date +%s)\"\n    if [ $((current_epoch - lockdir_epoch)) -gt 7200 ]; then\n        rmdir \"$lockdir\"\n    fi\nfi\n\n# used as a mutex lock\nmkdir \"$lockdir\" 2>/dev/null || return\n\nsudo bash <<EOF\n    {\n\n        date\n        echo\n\n        bash \"$customize_script\" &&\n\n        rmdir \"$lockdir\" &&\n\n        sudo UMASK=0044 touch \"$completion_semaphore\" &\n\n    } > /var/log/customize_environment 2>&1\nEOF\n"
  },
  {
    "path": ".bash.d/aws.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-09-01 13:01:11 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                A W S  -  A m a z o n   W e b   S e r v i c e s\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\ntype add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n# shellcheck disable=SC1090,SC1091\n#type autocomplete &>/dev/null || . \"$bash_tools/.bash.d/functions.sh\"\n\n# ==================\n# AWS CLI completion\n\naws_completer=\"$(type -P aws_completer 2>/dev/null)\"\n\nif [ -n \"$aws_completer\" ]; then\n    complete -C \"$aws_completer\" aws\nfi\n\n#autocomplete eksctl\n\n# =====================\n# Elastic Beanstalk CLI (easier to use than AWS CLI)\n\nif [ -d ~/.ebcli-virtual-env/executables/ ]; then\n    add_PATH ~/.ebcli-virtual-env/executables/\nfi\n\n# ============================================================================ #\n#                   A l i a s e s   a n d   F u n c t i o n s\n# ============================================================================ #\n\nalias awsl='aws sso login'\n\n#alias s3='s3cmd'\nalias s3='aws s3'\nalias dockerecr='aws ecr get-login-password | docker login -u AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com'\n\nalias awscon='aws_consoler -o'\nalias awsc='awscon'\n\nalias aws_whoami=\"aws sts get-caller-identity\"\nalias awhoami=aws_whoami\n\n# loads creds from a CLI cache file (eg. for AWS SSO) into environment variables\n# better done via direnv\nawscreds(){\n    # should be something like ~/.aws/cli/cached/[hash].json\n    local cred_cache_file=\"$1\"\n    AWS_ACCESS_KEY_ID=\"$(jq -r .Credentials.AccessKeyId < \"$cred_cache_file\")\"\n    AWS_SECRET_ACCESS_KEY=\"$(jq -r .Credentials.SecretAccessKey < \"$cred_cache_file\")\"\n    AWS_SESSION_TOKEN=\"$(jq -r .Credentials.SessionToken < \"$cred_cache_file\")\"\n    export AWS_ACCESS_KEY_ID\n    export AWS_SECRET_ACCESS_KEY\n    export AWS_SESSION_TOKEN\n}\n\n# ==================\n# AWLess completion\n\nalias awl=awless\nalias assh=\"awless ssh\"\n\n#autocomplete awless\n# make completion work with awl alias above\n#if ! [ -f ~/.bash.autocomplete.d/awl.sh ]; then\n#    sed 's/awless/awl/g' ~/.bash.autocomplete.d/awless.sh > ~/.bash.autocomplete.d/awl.sh\n#fi\n#autocomplete awl\n\n# ==================\n\n# JAVA_HOME needs to be set to use EC2 api tools\n#[ -x /usr/bin/java ] && export JAVA_HOME=/usr  # errors but still works\n\n# Shouldn't be needed any more, all these sorts of tools were unified on awscli\n#\n# link_latest '/usr/local/ec2-api-tools-*'\n#if [ -d /usr/local/ec2-api-tools/bin ]; then\n#    export EC2_HOME=/usr/local/ec2-api-tools   # this should be a link to the unzipped ec2-api-tools-1.6.1.4/\n#    add_PATH \"$EC2_HOME/bin\"\n#fi\n\n# ============================================================================ #\n\n# Old: new direnv now\n#\n# ec2dre - ec2-describe-regions - list regions you have access to and put them here\n# TODO: pull a more recent list and have aliases/functions auto-generated from that to export\n#aws_eu(){\n#    export EC2_URL=ec2.eu-west-1.amazonaws.com\n#}\n#aws_useast(){\n#    export EC2_URL=ec2.us-east-1.amazonaws.com\n#}\n#aws_eu\n\n# ============================================================================ #\n\n# https://github.com/remind101/assume-role\nassume-role(){\n    #eval \"$(command assume-role \"$@\")\"\n    local output\n    output=\"$(command assume-role \"$@\")\"\n    # shellcheck disable=SC2181\n    if [ $? -eq 0 ]; then\n        eval \"$output\"\n    fi\n}\n\n# ============================================================================ #\n\naws_get_cred_path(){\n    # unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n    [ -n \"${HOME:-}\" ] || HOME=~\n    local aws_credentials=\"${AWS_SHARED_CREDENTIALS_FILE:-$HOME/.aws/credentials}\"\n    local aws_config=\"${AWS_CONFIG_FILE:-$HOME/.aws/config}\"\n    local boto=\"${BOTO_CONFIG:-$HOME/.boto}\"\n    local credentials_file\n    if [ -f \"$aws_credentials\" ]; then\n        credentials_file=\"$aws_credentials\"\n    # older boto creds\n    elif [ -f \"$boto\" ]; then\n        credentials_file=\"$boto\"\n    elif [ -f \"$aws_config\" ]; then\n        credentials_file=\"$aws_config\"\n    else\n        echo \"no credentials found - didn't find $aws_credentials or $boto or $aws_config\" 2>/dev/null\n        return 1\n    fi\n    echo \"$credentials_file\"\n}\n\n# this rarely changes so just set it once as initialization instead of passing lots of params\n# or re-executing aws_get_cred_path() multiple times in different functions\naws_credentials_file=\"$(aws_get_cred_path)\"\n\naws_clean_env(){\n    echo \"clearing AWS_* environment variables\"\n    while read -r envvar; do\n        unset \"$envvar\"\n    done < <(env | sed -n '/^AWS_/ s/=.*// p')\n}\n\n# easily set a profile env var\n#aws_profile(){\n#    # false positive\n#    # shellcheck disable=SC2317\n#    export AWS_PROFILE=\"$*\"\n#}\n\nalias awsprofile=aws_profile.sh\nalias awsp=aws_profile.sh\n\naws_get_profile_data(){\n    local profile=\"$1\"\n    local filename=\"${2:-$aws_credentials_file}\"\n    sed -n \"/[[:space:]]*\\\\[\\\\(profile[[:space:]]*\\\\)*$profile\\\\]/,/^[[:space:]]*\\\\[/p\" \"$filename\"\n}\n\n# Storing creds in one place in Boto creds file, pull them straight from there\n# if only using new creds, might want to just export AWS_PROFILE instead using aws_profile which provides validation\naws_env(){\n    local profile=\"${1:-default}\"\n    # export AWS_ACCESS_KEY\n    # export AWS_SECRET_KEY\n    # export AWS_SESSION_TOKEN - for multi-factor authentication\n    local aws_token=~/.aws/token\n    aws_profile \"$profile\" || return 1\n    # section is checked for existence as part of aws_profile(), will return before here if not valid\n    local profile_data\n    profile_data=\"$(aws_get_profile_data \"$profile\")\"\n    echo \"loading [$profile] creds from $aws_credentials_file\"\n    eval \"$(\n    for key in aws_access_key_id aws_secret_access_key aws_session_token; do\n        awk -F= \"/^[[:space:]]*$key/\"'{gsub(/[[:space:]]+/, \"\", $0); gsub(/_id/, \"\", $1); gsub(/_secret_access/, \"_secret\", $1); print \"export \"toupper($1)\"=\"$2}' <<< \"$profile_data\"\n    done\n    )\"\n    if [ -f \"$aws_token\" ]; then\n        echo \"sourcing $aws_token\"\n        # shellcheck disable=SC1090,SC1091\n        source \"$aws_token\"\n    fi\n}\nalias awsenv=aws_env\n\naws_envs(){\n    awk '/^[[:space:]]*\\[.+\\]/{print $1}' < \"$aws_credentials_file\" |\n    sed 's/\\[//;s/\\]//' |\n    while read -r profile; do\n        default=0\n        if [ \"$profile\" = \"$AWS_PROFILE\" ]; then\n            local default=1\n        elif [ -z \"$AWS_PROFILE\" ] &&\n             [ \"$profile\" = \"default\" ]; then\n            local default=1\n        fi\n        if [ \"$default\" = 1 ]; then\n            echo -n \"* \"\n        else\n            echo -n \"  \"\n        fi\n        echo -n \"$profile\"\n        if [ \"$default\" = 1 ] &&\n           ! env | grep -q '^AWS_SECRET_KEY='; then\n            echo -n \" (keys not loaded to env)\"\n        fi\n        echo\n    done\n}\nalias awsenvs=aws_envs\n\naws_unenv(){\n    unset AWS_ACCESS_KEY\n    unset AWS_SECRET_KEY\n    unset AWS_SESSION_TOKEN\n}\nalias awsunenv=aws_unenv\n\naws_token(){\n    local output\n    local token\n    if [ $# -eq 0 ]; then\n        echo \"usage: aws_token <token_from_mfa_device> [<other_options>]\"\n        return 1\n    fi\n    if [ -z \"${AWS_MFA_ARN:-}\" ]; then\n        echo \"environment variable \\$AWS_MFA_ARN not set - you need to\"\n        echo\n        echo \"export AWS_MFA_ARN=arn:aws:iam::<123456789012>:mfa/<user>\"\n        echo\n        echo \"(you might want to put that in your ~/.bashrc.local or similar)\"\n        return 1\n    fi\n    #aws sts get-session-token --serial-number arn-of-the-mfa-device --token-code code-from-token\n    set -x\n    output=\"$(aws sts get-session-token --serial-number \"$AWS_MFA_ARN\" --duration-seconds \"${AWS_STS_DURATION_SECS:-129600}\" --token-code \"$@\")\"\n    result=$?\n    set +x\n    echo \"$output\"\n    if [ $result -ne 0 ]; then\n        return $result\n    fi\n    if type -P jq &>/dev/null; then\n        token=\"$(jq -r '.Credentials.SessionToken' <<< \"$output\")\"\n    else\n        token-\"$(awk -F: '/SessionToken/{print $2}' | sed 's/\"//')\"\n    fi\n    export AWS_SESSION_TOKEN=\"$token\"\n    echo \"exported AWS_SESSION_TOKEN\"\n    echo\n    echo \"export AWS_SESSION_TOKEN=$token\" > ~/.aws/token\n    echo \"saved to ~/.aws/token for other shells to source via aws_env()\"\n    echo\n    echo \"you can now use AWS CLI normally\"\n}\nalias awstoken=aws_token\n"
  },
  {
    "path": ".bash.d/azure.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC1090,SC1091\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 16:36:42 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   A z u r e\n# ============================================================================ #\n\nsrcdir=\"${srcdir:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$srcdir/.bash.d/paths.sh\"\n\n# Azure CLI from script install, installs to $HOME/lib and $HOME/bin\nif [ -f ~/lib/azure-cli/az.completion ]; then\n    source ~/lib/azure-cli/az.completion\nfi\n\n# assh is an alias to awless ssh\nazssh(){\n    local ip\n    ip=\"$(az vm show --name \"$1\" -d --query \"[publicIps]\" -o tsv)\"\n    ssh azureuser@\"$ip\"\n}\n"
  },
  {
    "path": ".bash.d/bash_it.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-26 15:21:46 +0000 (Wed, 26 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 B a s h   I T\n# ============================================================================ #\n\nreturn\n\nif ! [ -d ~/.bash_it ]; then\n    git clone --depth=1 https://github.com/Bash-it/bash-it.git ~/.bash_it\nfi\n\nexport BASH_IT=~/.bash_it\n\nexport BASH_IT_THEME='bobby'\n"
  },
  {
    "path": ".bash.d/circleci.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-02 12:21:34 +0100 (Tue, 02 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                C i r c l e C I\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\nif ! type github_owner_repo &>/dev/null; then\n    # shellcheck disable=SC1090,SC1091\n    . \"$bash_tools/.bash.d/git.sh\"\nfi\n\ncircleci_debug(){\n    circleci_project_set_env_vars.sh github/$(github_owner_repo) DEBUG=1\n}\n\ncircleci_undebug(){\n    circleci_project_delete_env_vars.sh github/$(github_owner_repo) DEBUG\n}\n"
  },
  {
    "path": ".bash.d/colors.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2034\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-06-25 15:20:39 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        Terminal ANSI Escape Color Codes\n# ============================================================================ #\n\n# Show color codes - adapted from http://codesnippets.joyent.com/posts/show/1517 example 1\n#\n# see also colors.pl from DevOps-Perl-tools repo which is slightly better\ncolors(){\n    local text=\"hari\"\n\n    echo -e '\\n                  40m      41m      42m      43m      44m      45m      46m      47m';\n\n    for FGs in '    m' '   1m' '  30m' '1;30m' '  31m' '1;31m' '  32m' \\\n               '1;32m' '  33m' '1;33m' '  34m' '1;34m' '  35m' '1;35m' \\\n               '  36m' '1;36m' '  37m' '1;37m'; do\n        FG=${FGs// /}\n        # shellcheck disable=SC1117\n        echo -en \" $FGs \\033[$FG  $text  \"\n        for BG in 40m 41m 42m 43m 44m 45m 46m 47m; do\n            # shellcheck disable=SC1117\n            echo -en \"$EINS \\033[$FG\\033[$BG  $text  \\033[0m\";\n        done\n        echo\n    done\n    echo\n}\n\n# ============================================================================ #\n\n# For Gentoo stylish prompts\n#\n# Find or write a full colour output table like seen here:\n# Daniel Robbins prompt magic tip on ibm developerworks\n#\n# from http://wiki.archlinux.org/index.php/Color_Bash_Prompt\n#\n# would set 'readonly' but causes reloads to output readonly variable errors\n# replaced \\e with \\033 as it is more portable on Mac script includes for lib/utils.sh tick_msg()\ntxtblk='\\033[0;30m'  # Black - Regular\ntxtred='\\033[0;31m'  # Red\ntxtgrn='\\033[0;32m'  # Green\ntxtylw='\\033[0;33m'  # Yellow\ntxtblu='\\033[0;34m'  # Blue\ntxtpur='\\033[0;35m'  # Purple\ntxtcyn='\\033[0;36m'  # Cyan\ntxtwht='\\033[0;37m'  # White\nbldblk='\\033[1;30m'  # Black - Bold\nbldred='\\033[1;31m'  # Red\nbldgrn='\\033[1;32m'  # Green\nbldylw='\\033[1;33m'  # Yellow\nbldblu='\\033[1;34m'  # Blue\nbldpur='\\033[1;35m'  # Purple\nbldcyn='\\033[1;36m'  # Cyan\nbldwht='\\033[1;37m'  # White\nunkblk='\\033[4;30m'  # Black - Underline\nundred='\\033[4;31m'  # Red\nundgrn='\\033[4;32m'  # Green\nundylw='\\033[4;33m'  # Yellow\nundblu='\\033[4;34m'  # Blue\nundpur='\\033[4;35m'  # Purple\nundcyn='\\033[4;36m'  # Cyan\nundwht='\\033[4;37m'  # White\nbakblk='\\033[40m'    # Black - Background\nbakred='\\033[41m'    # Red\nbakgrn='\\033[42m'    # Green\nbakylw='\\033[43m'    # Yellow\nbakblu='\\033[44m'    # Blue\nbakpur='\\033[45m'    # Purple\nbakcyn='\\033[46m'    # Cyan\nbakwht='\\033[47m'    # White\ntxtrst='\\033[0m'     # Text Reset\n"
  },
  {
    "path": ".bash.d/custom.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  C u s t o m\n# ============================================================================ #\n\n# Stuff that's overly custom and only sourced for my own user\n#\n# eg. $USER specific env vars and too short less generic aliases\n\nif ! [[ $USER =~ hari|sekhon ]]; then\n    return 0\nfi\n\n# put secret tokens in vars() or ~/.bashrc.local instead\nexport GITHUB_USER=HariSekhon\nexport TRAVIS_USER=\"HariSekhon\"\nexport BUILDKITE_ORGANIZATION=hari-sekhon\nexport SEMAPHORE_CI_ORGANIZATION=harisekhon\n\nalias tll=\"travis_last_log.py\"\n\n# can't set this to just the shorter 'go' or 'perl' because it'll clash with the actual commands\nalias goto=go_tools\nalias pyt=pytools\nalias to=perl_tools\n\n# shellcheck disable=SC2154\nexport plugins=\"$github/nagios-plugins\"\nexport pl=\"$plugins\"\nalias plugins='sti pl; cd $pl'\nalias pl=plugins\n\n# travis_last_log.py should be in $PATH from DevOps-Python-tools repo\nalias pll=\"travis_last_log.py HariSekhon/nagios-plugins\"\nexport pl2=\"${plugins}2\"\nalias pl2='sti pl2; cd $pl2'\n\nalias pytl=\"tll /pytools\"\nalias pyt2=\"pytools2\"\nalias pyl=\"pylib\"\nalias pyll=\"tll /pylib\"\n\nalias tol=\"tll /tools\"\nalias to2=\"tool2\"\n\n# clashes with the D2 diagramming language\n#alias d2=\"Dockerfiles2\"\nalias Dockerfilesl=\"tll /Dockerfiles\"\n"
  },
  {
    "path": ".bash.d/direnv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-10 13:02:46 +0100 (Fri, 10 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nif type -P direnv &>/dev/null; then\n    if ! type _direnv_hook &>/dev/null ||\n       ! [[ \"${PROMPT_COMMAND:-}\" =~ _direnv_hook ]]; then\n        eval \"$(direnv hook bash)\"\n    fi\nfi\n\n# direnv seems to inserts a double semi-colon which breaks PROMPT_COMMAND\n#export PROMPT_COMMAND=\"${PROMPT_COMMAND%%;;*}\"\nexport PROMPT_COMMAND=\"${PROMPT_COMMAND//;;/;}\"\n\n#alias envrc='$EDITOR .envrc && direnv allow .'\n# same effect as above\nalias envrc='direnv edit'\n\n# allow all .envrc under your current root - use only inside trusted repos\nalias direnvallowall='find . -name .envrc -exec direnv allow {} \\;'\nalias da='direnv allow'\nalias daa='direnvallowall'\n"
  },
  {
    "path": ".bash.d/docker.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-11-05 20:53:32 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  D o c k e r\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# shellcheck disable=SC1090,SC1091\n[ -f ~/.docker_vars ] && . ~/.docker_vars\n\n#if is_linux && type -P podman &>/dev/null; then\n#    alias docker=\"podman\"\n#fi\n\nexport DOCKER_BUILDKIT=1\n\n# for new M1 Macs which otherwise fail to build with errors like this:\n#\n#   AWS CLI version: qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory\n#\nexport DOCKER_DEFAULT_PLATFORM=linux/amd64\n\nalias dh=hub-tool\nalias dc=docker-compose\nalias dps='docker ps'\nalias dpsa='docker ps -a'\nalias dst=\"dockerhub_show_tags.py\"\n# -l shows latest container, -q shows only ID\n#alias dl='docker ps -lq'\nalias dockerimg='$EDITOR \"$bash_tools/setup/docker-images.txt\"'\n\n# wipe out exited containers\nalias dockerrm='docker rm -- $(docker ps -qf status=exited)'\n\nalias dockerr=dockerrunrm\nalias dock=dockerr\nalias dockere=dockerexec\nalias de=dockere\n\nalias dockerrma=dockerrmall\n\n# wipe out dangling image layers\n#alias dockerrmi='docker rmi $(docker images -q --filter dangling=true)'\ndockerrmi(){\n    # want word splitting here\n    # shellcheck disable=SC2046\n    docker rmi $(docker images -q --filter dangling=true)\n}\n\n# docker-compose -f ...\ndcf(){\n    local docker_compose_yaml=\"$1\"\n    if ! [ -f \"$docker_compose_yaml\" ] &&\n         [ -f \"docker-compose.yaml\" ]; then\n        docker_compose_yaml=docker-compose.yaml\n    fi\n    shift\n    docker-compose -f \"$docker_compose_yaml\" up \"$@\"\n    docker-compose -f \"$docker_compose_yaml\" logs -f\n}\n\n# starts the docker VM, shows ASCII whale, but slow\n#alias dockershell=\"/Applications/Docker/Docker\\ Quickstart\\ Terminal.app/Contents/Resources/Scripts/start.sh\"\n# better\n#alias dockervm=\"VBoxManage controlvm startvm default\"\n#alias dockervm=\"docker-machine start default\"\n\n#alias dm=\"docker-machine\"\n#alias dockerrr=\"docker-machine restart default\"\n#alias dockerreload=\"docker-machine env default > '$bash_tools/.docker_vars'; . '$bash_tools/.docker_vars'\"\n\n#dockerstart(){\n#    if ! docker-machine status default | grep -q Running; then\n#        docker-machine start default\n#        sleep 20\n#    fi\n#    docker start $(cat \"$bash_tools/docker-start.txt\")\n#}\n\n# avoid external commands per shell, slows down new shells and wastes battery\n# switched to using ~/.docker_vars file which is cheaper due to less forks and picked up in each new shell\n#if type -P docker-machine &>/dev/null; then\n#    if docker-machine status default | grep -q -e Started -e Running; then\n#        eval $(docker-machine env default)\n#    fi\n#fi\n\n#alias dockerr=\"docker run --rm -ti\"\nfunction dockerrunrm(){\n    local args=()\n    local passed_first_non_switch_arg=0  # when this latch gets to level 3 we stop doing prefix processing to not adulterate ls -l / type args\n    for x in \"$@\"; do\n        if [ $passed_first_non_switch_arg -lt 3 ]; then\n            if [ \"${x:0:1}\" = \"-\" ]; then\n                passed_first_non_switch_arg=1\n            elif [ $passed_first_non_switch_arg -eq 1 ]; then\n                passed_first_non_switch_arg=2\n            elif [ $passed_first_non_switch_arg -lt 3 ]; then\n                if [ \"${x:0:1}\" = \"/\" ]; then\n                    if [[ \"$x\" != */Users/* && \"$x\" != */home/* ]] &&\n                       [ \"$(strLastIndexOf \"$x\" / )\" -eq 1 ]; then\n                        x=\"harisekhon$x\"\n                    fi\n                fi\n                passed_first_non_switch_arg=3\n            else\n                ((passed_first_non_switch_arg+=1))\n            fi\n        fi\n        args+=(\"$x\")\n    done\n    # Alpine 2 is dead in the water since the package list repos don't even load any more:\n    #\n    # # apk update\n    # fetch http://dl-4.alpinelinux.org/alpine/v2.7/main/x86_64/APKINDEX.tar.gz\n    # wget: server returned error: HTTP/1.1 404 Not Found\n    # ERROR: http://dl-4.alpinelinux.org/alpine/v2.7/main: Bad address\n    # WARNING: Ignoring APKINDEX.0f59c441.tar.gz: No such file or directory\n    #\n    #if [[ \"$args\" =~ alpine:2 ]] && ! [[ \"$args\" =~ [[:space:]] ]]; then\n    #    echo \"warning: using alpine:2.* with args but alpine:2.* doesn't have a default CMD so adding 'sh' arg\" >&2\n    #    args=\"$args sh\"\n    #fi\n    local basedir=\"${PWD##*/}\"\n    docker run --rm -ti -v \"$PWD\":\"/$basedir\" -w \"/$basedir\" \"${args[@]}\"\n}\nalias drun='docker run --rm -ti -v \"${PWD}\":/app'\n\ndocker_get_container_ids(){\n    local exclude_file=~/docker-perm.txt\n    local args=()\n    # if exclude file doesn't exist, grep fails entirely and we get no IDs returned, even pre-emptively replacing with /dev/null doesn't work, so omit the option entirely\n    if [ -f \"$exclude_file\" ]; then\n        args=(-f \"$exclude_file\")\n    fi\n    docker ps -a --format \"{{.ID}} {{.Names}}\" |\n    if [ ${#args} -gt 0 ]; then\n        grep -vi \"${args[@]}\" 2>/dev/null\n    else\n        cat\n    fi |\n    awk '{print $1}'\n}\n\ndockerrmall(){\n    # would use xargs -r / --no-run-if-empty but that is GNU only, doesn't work on Mac\n    local ids=()\n    read -r -a ids <<< \"$(docker_get_container_ids)\"\n    if [ ${#ids} -gt 0 ]; then\n        docker rm -f -- \"${ids[@]}\"\n    fi\n}\n\ndockerrmigrep(){\n    for x in \"$@\"; do\n        docker images |\n        awk \"/$x/{print \\$1\\\":\\\"\\$2}\" |\n        sed '/<none>/d' |\n        xargs -r docker rmi --\n    done\n}\n\ndockerrmgrep(){\n    for x in \"$@\"; do\n        docker ps -a |\n        grep \"$x\" |\n        awk '{print $NF}' |\n        xargs -r docker rm -f --\n    done\n}\n\ndockerip(){\n    docker inspect --format '{{ .NetworkSettings.IPAddress }}' \"$@\"\n}\n\n# this goes to the last created and sometimes exited container\n#alias dockere='docker exec -ti $(docker ps -lq) /bin/bash'\ndockerexec(){\n    if [ $# -gt 0 ]; then\n        container=\"$(docker ps | grep -i \"$1\" | awk '{print $1}' | head -n1)\"\n    else\n        container=\"$(docker ps -q | head -n1)\"\n    fi\n    docker exec -ti \"$container\" /bin/sh\n}\n\ndocker_get_images(){\n    # uniq_order_preserved.pl is in the DevOps-Perl-tools repo on github and should be in the $PATH\n    # too many images on dockerhub to pull, fills up filesystem\n    #echo \"$(dockerhub_search.py harisekhon -n 1000 | tail -n +2 | awk '{print $1}' | sort) $(sed 's/#.*//;/^[[:space:]]*$/d' \"$bash_tools/setup/docker-images.txt\" | uniq_order_preserved.pl)\"\n    sed 's/#.*//;/^[[:space:]]*$/d' \"$bash_tools/setup/docker-images.txt\" |\n    uniq_order_preserved.pl\n}\n\ndockerpull1(){\n    # pull only latest tag, mine first, then official\n    local images=\"${*:-}\"\n    [ -z \"$images\" ] && images=\"$(docker_get_images)\"\n    images=\"$(grep -v \":\" <<< \"$images\")\"\n    whendone \"docker pull\" # must be first arg so quoted, [l] trick not needed as grep -v grep's\n    for image in $images; do\n        #whendone \"docker pull\" # must be first arg so quoted, [l] trick not needed as grep -v grep's\n        timestamp \"docker pull $image\"\n        #docker pull \"$image\" | cat &\n        docker pull \"$image\"\n        # wipe out dangling image layers\n        dockerrmi\n        echo\n    done\n}\ndockerpullgithub(){\n    dockerpull1 harisekhon/{nagios-plugins,pytools,tools,centos-github,debian-github,ubuntu-github,alpine-github}\n}\n\ndockerpull(){\n    local images=\"${*:-}\"\n    [ -z \"$images\" ] && images=\"$(docker_get_images)\"\n    dockerpull1 \"$images\"\n    images=\"$(grep -i -e harisekhon -e \":\" <<< \"$images\")\"\n    #local images=\"$(grep -i -e \":\" <<< \"$images\")\"\n    # now pull all tags, mine first, then official\n    whendone \"docker pull\" # must be first arg so quoted, [l] trick not needed as grep -v grep's\n    for image in $images; do\n        #whendone \"docker pull\" # must be first arg so quoted, [l] trick not needed as grep -v grep's\n        if [[ \"$image\" = harisekhon/* && ! \"$image\" =~ \":\" ]]; then\n            [[ \"$image\" =~ presto.*-dev ]] && continue\n            for tag in $(dockerhub_show_tags.py -q \"$image\" | grep -v '^latest$'); do\n                timestamp \"docker pull $image:$tag\"\n                #docker pull \"$image\":\"$tag\" | cat &\n                docker pull \"$image\":\"$tag\"\n                echo\n            done\n        else\n            timestamp docker pull \"$image\"\n            #docker pull \"$image\" | cat &\n            docker pull \"$image\"\n            echo\n        fi\n        # wipe out dangling image layers\n        dockerrmi\n    done\n}\n\ndockerpull1r(){\n    while true; do\n        dockerpull1 \"$@\"\n        wait\n        echo -e '\\n\\nsleeping for 1 hour\\n\\n'\n        sleep 3600\n    done\n}\n\ndockerpullr(){\n    while true; do\n        dockerpull \"$@\"\n        wait\n        echo -e '\\n\\nsleeping for 1 hour\\n\\n'\n        sleep 3600\n    done\n}\n\n# quick, only pull things for which we don't already have local images\ndockerpullq(){\n    for x in $(docker_get_images); do\n        docker images | grep -q \"^${x}[[:space:]]\" && continue\n        whendone \"docker pull\" # must be first arg so quoted, [l] trick not needed as grep -v grep's\n        timestamp docker pull \"$x\"\n        docker pull \"$x\"\n    done\n    # wipe out dangling image layers\n    dockerrmi\n}\n"
  },
  {
    "path": ".bash.d/env.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                   E n v i r o n m e n t   V a r i a b l e s\n# ============================================================================ #\n\n# more environment variables defined next to the their corresponding aliases in aliases.sh\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# similar to what zsh does by default\nif [ -f ~/.bashenv ]; then\n    # shellcheck disable=SC1090,SC1091\n    . ~/.bashenv\nfi\n\n#export DISPLAY=:0.0\n\n#export TERM=xterm\n\nexport EDITOR=vim\n\nexport INPUTRC=~/.inputrc\n\n# allow programs to use $LINES and $COLUMNS\nexport LINES\nexport COLUMNS\n\n# sets directories to cyan on default bg so they stand out more in dark terminal - see 'man ls' for more details\n# works on Mac - you may need to see 'man 5 dir_colors' on Linux\nexport LSCOLORS=\"gx\"\n\n# ENV refers to the file that sh attempts to read as a startup file (done on my Mac OSX Snow Leopard)\n# Needs the following line added to sudoers for ENV to be passed through on sudo su\n#Defaults\tenv_keep += \"ENV\"\nexport ENV=~/.bashrc\n\n# ============================================================================ #\n\ncpenv(){\n    local env_var=\"$1\"\n    if [[ -z \"${!env_var}\" ]]; then\n        echo \"Error: Environment variable '$env_var' is not set\"\n        return 1\n    fi\n    copy_to_clipboard.sh <<< \"${!env_var}\"\n    echo \"Value of '$env_var' has been copied to the clipboard\"\n}\n\n# Autocomplete function for environment variables\n_cpenv_autocomplete() {\n    # 'compgen -v' lists all environment variables\n    # COMPREPLY is set to the autocomplete options\n    local cur_word=\"${COMP_WORDS[COMP_CWORD]}\"\n    COMPREPLY=($(compgen -v -- \"$cur_word\"))\n}\n\n# Register autocomplete function for `cpenv`\ncomplete -F _cpenv_autocomplete cpenv\n\n\n# ============================================================================ #\n#             L o c a l e   I n t e r n a t i o n a l i z a t i o n\n# ============================================================================ #\n\n# Run this to see available locales:\n#\n#   locale -a\n#\n# See details of a specific locale variable eg. time formats:\n#\n#   LC_ALL=C locale -ck LC_TIME\n\n# aterm doesn't support UTF-8 and you get horrible chars here and there\n# so don't use utf and aterm together. xterm works ok with utf8 though\n#export LANG=en_GB\n#\n# LANG becomes default value for any LC_xxx variables not set\n#export LANG=C\n#\n# overrides all other LC_xxx variables\n#export LC_ALL=C\n#\nexport LANG=en_US.UTF-8\nexport LC_ALL=en_US.UTF-8\nexport LANGUAGE=en_US.UTF-8\n#export LC_ALL=en_GB\n# didn't seem to work\n#export LANG=\"en_GB.UTF-8\"\n#export LC_ALL=\"en_GB.UTF-8\"\n\n# ============================================================================ #\n\n# Clever dynamic environment variables, set using var() function sourced between shells\nexport varfile=~/.bash_vars\n# shellcheck disable=SC1090,SC1091\n[ -f \"$varfile\" ] && . \"$varfile\"\n\n# Secret Credentials\n#\n#   separate cred files so if you accidentally expose it on a screen\n#   to colleagues or on a presentation or screen share\n#   you don't have to change all of your passwords\n#   which you would have to if using the above ~/.bash_vars file\nif [ -d ~/.env/creds ]; then\n    for credfile in ~/.env/creds/*; do\n        if [ -f \"$credfile\" ]; then\n            # shellcheck disable=SC1090,SC1091\n            . \"$credfile\"\n        fi\n    done\nfi\n\n#export DISTCC_DIR=\"/var/tmp/portage/.distcc/\"\n\n# ============================================================================ #\n\nif is_mac; then\n    #BROWSER=open\n    unset BROWSER\nelif type -P google-chrome &>/dev/null; then\n    BROWSER=google-chrome\nelif type -P firefox &>/dev/null; then\n    BROWSER=firefox\nelif type -P konqueror &>/dev/null; then\n    BROWSER=konqueror\nelif [ -n \"${GOOGLE_CLOUD_SHELL:-}\" ]; then\n    :\nelse\n    :\n    #BROWSER=UNKNOWN\n    #echo \"COULD NOT FIND ANY BROWSER IN PATH\"\nfi\n\n# don't export BROWSER on Mac, trigger python bug:\n# AttributeError: 'MacOSXOSAScript' object has no attribute 'basename'\n# from python's webbrowser library\nif ! is_mac; then\n    export BROWSER\nfi\n\nvar(){\n    local var=\"${*%%=*}\"\n    local val=\"${*#*=}\"\n    if grep -i \"export $var\" \"$varfile\" &>/dev/null; then\n        perl -pi -e 's/^export '\"$var\"'=.*$/export '\"$var\"'='\"$val\"'/' \"$varfile\"\n    else\n        echo \"export $var=$val\" >> \"$varfile\"\n    fi\n    export \"$var\"=\"$val\"\n}\nvars(){\n    \"$EDITOR\" \"$varfile\"\n    chmod 0600 \"$varfile\"\n    # shellcheck disable=SC1090,SC1091\n    . \"$varfile\"\n}\n\nunvar(){\n    local var=\"${*%%=*}\"\n    [ -f \"$varfile\" ] || { echo \"$varfile not found\" ; return 1; }\n    perl -pi -e 's/^export '\"$var\"'=.*\\n$//' \"$varfile\"\n    unset \"$var\"\n}\n\n# ============================================================================ #\n\nunsetall(){\n    local match=\"${1:-.*}\"\n    while read -r env_var; do\n        if [ \"$env_var\" = PATH ]; then\n            continue\n        fi\n        unset \"$env_var\"\n    done < <( env |\n        grep -i \"$match\" |\n        sed 's/=.*//' )\n}\n"
  },
  {
    "path": ".bash.d/functions.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                  B a s h   G e n e r a l   F u n c t i o n s\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# Enables colourized return codes in prompt_func\n# better leave it as the same as already set. This way a reload of bashrc doesn't change the mode\n# could do retmode=${retmode:-off} but this is unnecessary overhead\n#retmode=off\n# safer to set it to off because otherwise it's possible to get in to a loop with set -u\nretmode=${retmode:-off}\nretmode(){\n    if [ \"$retmode\" = \"on\" ]; then\n        retmode=off\n        echo \"retmode off\"\n    else\n        retmode=on\n        echo \"retmode on\"\n    fi\n}\n\ncddir(){\n    cd \"$(dirname \"$1\")\" || return 1\n}\n\njq(){\n    command jq -CS \"$@\"\n}\n\nenvg(){\n    env |\n    eval grep -i \"$(for arg; do echo -n \" -e '$arg'\"; done)\"\n}\n\nnew(){\n    if [ $# -eq 2 ]; then\n        title \"${2#modules/}\"\n    else\n        title \"$1\"\n    fi\n    command new.pl \"$@\"\n    title \"$LAST_TITLE\"\n}\n\n# generates bash autocompletion if not available\n# sources bash autocompletion from local standardized path\nautocomplete(){\n    local name=\"$1\"\n    shift || :\n    if [ -f ~/.bash.autocomplete.d/\"$name.sh\" ]; then\n        # shellcheck disable=SC1090,SC1091\n        . ~/.bash.autocomplete.d/\"$name.sh\"\n    elif type -P \"$name\" &>/dev/null; then\n        # doesn't work\n        # shellcheck disable=SC1090,SC1091\n        #source <(command \"$name\" completion bash)\n        mkdir -pv ~/.bash.autocomplete.d\n        command \"$name\" completion \"$@\" bash > ~/.bash.autocomplete.d/\"$name.sh\"\n        # shellcheck disable=SC1090,SC1091\n        . ~/.bash.autocomplete.d/\"$name.sh\"\n    fi\n}\n\npg(){\n    # don't want pgrep, want color coding\n    # shellcheck disable=SC2009\n    ps -ef |\n    grep -i --color=yes \"$@\" |\n    grep -v grep\n}\n\npstg(){\n    # want splitting of options\n    # shellcheck disable=SC2086\n    pstree |\n    grep -5 -i --color=always \"$@\" |\n    less $LESS\n}\n\n# externalized to copy_to_clipboard.sh script\n#copy_to_clipboard(){\n#    if is_mac; then\n#        cat | pbcopy\n#    elif is_linux; then\n#        cat | xclip\n#    else\n#        echo \"ERROR: OS is not Darwin/Linux\"\n#        return 1\n#    fi\n#}\n\nunalias clip &>/dev/null || :\n# args are optional\n# shellcheck disable=SC2120\nclip(){\n    if [ $# -gt 0 ]; then\n        copy_to_clipboard.sh < \"$1\"\n    else\n        copy_to_clipboard.sh\n    fi\n}\n\ndle(){\n    if [[ \"$PWD\" =~ $HOME(/Downloads(/Transmission)?)?$ ]]; then\n        echo \"Switching to $HOME/Downloads/YouTube\"\n        mkdir -p -v ~/Downloads/YouTube\n        cd ~/Downloads/YouTube || return 1\n        if [ -f .envrc ]; then\n            eval \"$(direnv export bash)\"\n        fi\n    fi\n    while true; do\n        if BACKGROUND_VIDEO=1 youtube_download_video.sh \"$@\"; then\n            # doesn't persist past a pause/unpause,\n            # and this starts playing which we don't want which is why it's backgrounded\n            #osascript -e 'tell application \"QuickTime Player\" to set rate of document 1 to 2' &&\n            break\n        fi\n        local sleep_secs=\"$((RANDOM % 300))\"\n        echo \"Sleeping for $sleep_secs secs before retrying...\"\n        sleep \"$sleep_secs\"\n    done\n    exit\n}\nalias ytp=\"cd ~/Downloads/YouTube && ./play.sh\"\n\ndeccp(){\n    # shellcheck disable=SC2119\n    decomment.sh \"$@\" |\n    clip\n}\n\ndecdiff(){\n    diff <(decomment.sh \"$1\" | sed 's/[[:space:]]*$//') <(decomment.sh \"$2\" | sed 's/[[:space:]]*$//') \"${@:3}\"\n}\n\nrmdirempty(){\n    find \"${1:-.}\" -type d -empty -exec rmdir \"{}\" \\;\n}\n\ncheckprog(){\n    if type -P \"$1\" &>/dev/null; then\n        return 0\n    else\n        echo \"$1 could not be found in path\"\n        return 1\n    fi\n}\n\nfunction count() {\n    local total=\"$1\"\n    for ((i = total; i > 0; i--)); do\n        sleep 1\n        printf \"Time remaining %d secs\\r\" \"$i\"\n    done\n    bell\n}\n\ndum(){\n    du -max \"${@:-.}\" |\n    sort -k1n |\n    tail -n 10000\n}\n\ntyper(){\n    local alias_target\n    local type_output\n    for x in \"$@\"; do\n        type_output=\"$(type \"$x\")\"\n        # shellcheck disable=SC2119\n        alias_target=\"$(\n            awk '/aliased to/{print $5}' <<< \"$type_output\" |\n            unquote\n        )\"\n        if [ -n \"$alias_target\" ]; then\n            echo \"$type_output\"\n            typer \"$alias_target\"\n        else\n            type \"$x\"\n        fi\n    done\n}\n\nfindup(){\n    local arg=\"$1\"\n    current_dir=\"${PWD:-$(pwd)}\"\n    while [ \"$current_dir\" != \"\" ]; do\n        if [ -e \"$current_dir/$arg\" ]; then\n            echo \"$current_dir/$arg\"\n            return 0\n        fi\n        current_dir=\"${current_dir%/*}\"\n    done\n    echo \"Not found in above path: $arg\" >&2\n    return 1\n}\n\ncdup(){\n    local arg=\"$1\"\n    cd \"$(findup \"$arg\")\"\n}\n\nlld(){\n    {\n        local target=\"$1\"\n        ls -ld \"$target\"\n        [ \"$target\" = \"/\" ] && return\n        lld \"$(dirname \"$target\")\"\n    } | column -t\n}\n\n# shellcheck disable=SC2120\nunquote(){\n    sed '\n        s/^[[:space:]]*[`'\"'\"'\"]//;\n        s/[`'\"'\"'\"][[:space:]]*$//;\n    ' \"$@\"\n}\n\nbell(){\n    echo -e '\\a'\n}\n\nresolve_symlinks(){\n    local readlink=readlink\n    if is_mac; then\n        if type -P greadlink &>/dev/null; then\n            readlink=greadlink\n        else\n            readlink=\"\"\n        fi\n    fi\n    if [ -z \"$readlink\" ]; then\n        echo \"$*\"\n        return\n    fi\n    for x in \"$@\"; do\n        \"$readlink\" -m \"$x\"\n    done\n}\n\n# for all files listed, return the highest directory - useful for pushd to the right git root following symlinks before doing git diff and commmits, used by gitu() in git.sh which is called in inline vimrc 'nmap ;;'\nbasedir(){\n    local dir_list=\"\"\n    for x in \"$@\"; do\n        dir_list=\"$dir_list $(dirname \"$x\")\"\n    done\n    # assumes they share the same base and that the shortest one will be right - could put more comparison here and return error if not\n    local output\n    output=\"$(tr ' ' '\\n'  <<< \"$dir_list\" | grep -v '^[[:space:]]*$' | sort | head -n 1)\"\n    if [ -z \"$output\" ]; then\n        echo \"ERROR: empty basedir\"\n        return 1\n    fi\n    echo \"$output\"\n}\n\ntoLower(){\n    tr '[:upper:]' '[:lower:]'\n}\n\ntoUpper(){\n    tr '[:lower:]' '[:upper:]'\n}\n\ntrim(){\n    sed 's/^[[:space:]]*//; s/[[:space:]]*$//' \"$@\"\n}\n\nnormalize_spaces(){\n    # not variant of + \\+ works on Mac\n    #sed 's/[[:space:]]\\+/ /'\n    # flattens out newlines, which changes behaviour in scripts like check_git_commit_authors.sh\n    #perl -pe 's/\\s+/ /g'\n    # horizontal newlines, don't match \\n\n    perl -pe 's/\\h+/ /g'\n}\n\nremove_last_column(){\n    awk '{$NF=\"\"; print $0}'\n}\n\nstrip_basedirs(){\n    local basedir=\"$1\"\n    shift\n    while read -r filename; do\n        filename=\"${filename#\"${basedir%%/}\"/}\"\n        filename=\"${filename##/}\"\n        echo \"$filename\"\n    done <<< \"$@\"\n}\n\nuser(){\n    read -r -p 'user: ' USERNAME\n    export USERNAME\n    if [ -z \"${PASSWORD:-}\" ]; then\n        pass PASSWORD\n    fi\n}\n\npass(){\n    # doesn't echo, we can do better by making it star for each char\n    #read -r -s -p 'password: ' PASSWORD\n    # don't local PASSWORD or default case will not export PASSWORD, changing case to work around\n    local password=\"\"\n    prompt=\"Enter password: \"\n    while IFS= read -p \"$prompt\" -r -s -n 1 char; do\n        if [[ \"$char\" == $'\\0' ]]; then\n            break\n        fi\n        prompt='*'\n        password=\"${password}${char}\"\n    done\n    #passvar=\"${1:-PASSWORD}\"\n    for passvar in \"${@:-PASSWORD}\"; do\n        export \"$passvar\"=\"$password\"\n    done\n    echo\n}\n\nunpass(){\n    unset PASSWORD\n}\n\nhr(){\n    echo \"# ============================================================================ #\"\n}\n\nrepeat(){\n    local i n\n    n=\"$1\"\n    shift\n    if [ -z \"$n\" ]; then\n        echo \"usage: repeat N command args\"\n        return 1\n    fi\n    for ((i=1; i <= n; i++)); do\n        \"$@\"\n    done\n}\n\nloop(){\n    while true; do\n        eval \"${*//\\$/\\\\$}\"\n        sleep 1\n    done\n}\n\nptop(){\n    if [ -z \"$1\" ]; then\n        echo \"usage: ptop program1 program2 ...\"\n        return 1\n    fi\n    local pids\n    #pids=\"$(pgrep -f \"$(sed 's/ /|/g' <<< \"$*\")\")\"\n    pids=\"$(pgrep -f \"${*// /|}\")\"\n    local pid_args=()\n    if is_mac; then\n        # shellcheck disable=SC2001\n        read -r -a pid_args <<< \"$(sed 's/^/-pid /' <<< \"$pids\")\"\n    else\n        # shellcheck disable=SC2001\n        read -r -a pid_args <<< \"$(sed 's/^/-p /' <<< \"$pids\")\"\n    fi\n    if [ -z \"${pids[*]}\" ]; then\n        echo \"No matching programs found\"\n        return 1\n    fi\n    top \"${pid_args[@]}\"\n}\n\ntopcommands(){\n    # first awk print $2 but my advanced history records `date '+%F %T'` in between number and command for $2 and $3, making command $4\n    history |\n    awk '{print $4}' |\n    awk 'BEGIN {FS=\"|\"} {print $1}' |\n    sort |\n    uniq -c |\n    sort -n |\n    tail -n \"${1:-10}\" |\n    sort -nr\n}\nalias topcmds=topcommands\n\n# easy quick find recursing down current directory tree\n#\n#f(){\n#    [ -n \"$*\" ] || { echo \"usage: f <partial_pattern>\"; return 1; }\n#    pattern=\"\"\n#    for x in $*; do\n#        pattern+=\"*$x\"\n#    done\n#    pattern+=\"*\"\n#    find -L . -iname \"$pattern\"\n#}\n#\n# shellcheck disable=SC2032\nf(){\n    local grep=\"\"\n    # shellcheck disable=SC2013\n    for x in \"${@//[^A-Za-z0-9_-]/.}\"; do\n        if [[ \"$x\" =~ [a-zA-Z0-9._-] ]]; then\n            grep=\"$grep | grep -i --color=auto $x\"\n        fi\n    done\n    # times about the same\n    #eval find -L . -type f -iname \"\\*$1\\*\" $grep\n    eval find -L . -type f \"$grep\"\n}\n\ndgrep(){\n    local pattern=\"$*\"\n    # auto-exported in aliases.sh when iterating git repos\n    # shellcheck disable=SC2154\n    ls \"$docs/\"*\"${pattern// /}\"* 2>/dev/null\n    # shellcheck disable=SC2046,SC2033\n    grep -iER \"$pattern\" $(find ~/docs \"$docs\" -type f -maxdepth 1 2>/dev/null | grep -v '/\\.')\n}\n\ndiffl(){\n    diff \"$@\" | less\n}\n\nforeachfile(){\n    # not passing function f()\n    # shellcheck disable=SC2033\n    find . -type f -maxdepth 1 |\n    while read -r file; do\n        [ ! -f \"$file\" ] && continue\n        [ -b \"$file\"   ] && continue\n        [ -c \"$file\"   ] && continue\n        [ -d \"$file\"   ] && continue\n        [ -p \"$file\"   ] && continue\n        [ -S \"$file\"   ] && continue\n        [ -L \"$file\"   ] && continue\n        \"$@\"\n    done\n}\n\n# vim which\n# vw() moved to vim.sh\n\n# file which\nfw(){\n    local path\n    for x in \"$@\"; do\n        path=\"$(which \"$x\")\"\n        if [ -z \"$path\" ]; then\n            return 1\n        fi\n        file \"$path\"\n        echo\n        # shellcheck disable=SC2086\n        ls -l $LS_OPTIONS \"$path\"\n    done\n}\n\ncdwhich(){\n    local path\n    local directory\n    if [ $# -ne 1 ]; then\n        echo \"usage: cdwhich programname\"\n        return 1\n    fi\n    path=\"$(which \"$1\")\"\n    if [ -z \"$path\" ]; then\n        echo\n        echo \"$1 could not be found in \\$PATH\"\n        return 1\n    fi\n    directory=\"$(dirname \"$path\")\"\n    if [ -z \"$directory\" ]; then\n        echo \"cannot find directory for $path\"\n        return 2\n    fi\n    echo \"$directory\"\n    cd \"$directory\" || return 1\n}\n\nwhichall(){\n    local bin=\"$1\"\n    shift || :\n    which -a \"$bin\" |\n    while read -r bin; do\n        echo -n \"$bin: \"\n        \"$bin\" \"$@\"\n    done\n}\n\nadd_etc_host(){\n    local host_line=\"$*\"\n    # $sudo is set in .bashrc if needed\n    # shellcheck disable=SC2154\n    $sudo grep -q \"^$host_line\" /etc/hosts ||\n    $sudo echo \"$host_line\" >> /etc/hosts\n}\n\n# vihosts() moved to vim.sh\n\nproxy(){\n    export proxy_host=\"${1:-${proxy_host:-localhost}}\"\n    export proxy_port=\"${2:-${proxy_port:-8080}}\"\n    export proxy_port_ssl=\"${3:-${proxy_port_ssl:-8443}}\"\n    export proxy_user=\"${4:-${proxy_user:-$USER}}\"\n    if [ -z \"$proxy_password\" ]; then\n        read -r -s -p 'proxy password: ' proxy_password\n    fi\n    export http_proxy=\"http://$proxy_user:$proxy_password@$proxy_host:$proxy_port\"\n    export https_proxy=\"https://$proxy_user:$proxy_password@$proxy_host:$proxy_port_ssl\"\n    # MiniShift respects these next three\n    export HTTP_PROXY=\"$http_proxy\"\n    export HTTPS_PROXY=\"$https_proxy\"\n    export NO_PROXY=\".local,.localdomain,.intra,169.254.169.254\" # works only on suffixes or IP addresses - ignore the EC2 Metadata API address\n    export ftp_proxy=\"$http_proxy\" # might need to replace protocol prefix here, would check, but who even uses ftp any more\n    JAVA_NO_PROXY=\"$(sed 's/^/*/;s/,/|*/g' <<< \"$NO_PROXY\")\"\n    # strip the additions we just added off the end so that we don't end up with dups if running proxy more than once\n    JAVA_OPTS=\"${JAVA_OPTS%%-Dhttp.proxyHost*}\"\n    export JAVA_OPTS=\"$JAVA_OPTS -Dhttp.proxyHost=$proxy_host -Dhttp.proxyPort=$proxy_port -Dhttp.proxyUser=$proxy_user -Dhttp.proxyPassword=$proxy_password -Dhttps.proxyHost=$proxy_host -Dhttps.proxyPort=$proxy_port_ssl -DnonProxyHosts='$JAVA_NO_PROXY'\"\n    export SBT_OPTS=\"$JAVA_OPTS\"\n}\n\nreadlink(){\n    if is_mac; then\n        greadlink \"$@\"\n    else\n        command readlink \"$@\"\n    fi\n}\n\nabspath(){\n    readlink --canonicalize-missing \"$1\"\n}\n#abspath(){\n#    if [ -z \"$1\" ]; then\n#        echo \"NO PATH GIVEN!\"\n#        return 1\n#    fi\n#    # shellcheck disable=SC2001\n#    sed 's@^\\./@'\"$PWD\"'/@;\n#         s@^\\([^\\./]\\)@'\"$PWD\"'/\\1@;\n#         s@^\\.\\./@'\"${PWD%/*}\"'/@;\n#         s@/../@/@g;\n#         s@/\\./@/@g;\n#         s@\\(.*\\/?\\)\\.\\./?$@\\1/@;\n#         s@//@/@g;\n#         s@/$@@;' <<< \"$1\"\n#}\n\nwcbash(){\n    # $github defined in aliases.sh\n    # shellcheck disable=SC2154\n    wc ~/.bashrc \\\n       ~/.bash_profile \\\n       ~/.bash_logout \\\n       ~/.alias* \\\n       ~/.aliases* \\\n       ~/.bashrc_dynamichosts \\\n       \"$github/bash-tools/.bashrc\" \\\n       \"$github/bash-tools/.bash_profile\" \\\n       \"$github/bash-tools/.bash.d/\"*.sh 2>/dev/null\n}\n\nepoch2date(){\n    if is_mac; then\n        date -r \"$1\"\n    else\n        date -d \"@$1\"\n    fi\n}\n\npdf(){\n    if ! [[ \"$1\" =~ .*.pdf$ ]]; then\n        echo \"'$1' does not end in .pdf!\"\n        return 1\n    fi\n    if ! [ -f \"$1\" ]; then\n        echo \"file not found: $1\"\n        return 1\n    fi\n    if is_mac; then\n        open \"$1\"\n        return $?\n    fi\n    for x in acroread evince xpdf; do\n        if type -P \"$x\" &>/dev/null; then\n            echo \"opening with $x...\"\n            \"$x\" \"$1\" &\n            return $?\n        fi\n    done\n    echo \"Error cannot find acroread, evince or xpdf in PATH.\"\n    return 1\n}\n\ncurrentScreenResolution(){\n    #xrandr | awk '/\\*/ {print $1}'\n    xdpyinfo  | awk '/dimensions/ {print $2}'\n}\n\nyy(){\n    cal \"$(date '+%Y')\"\n}\n\n# ============================================================================ #\n\ntimestamp(){\n    printf \"%s\" \"$(date '+%F %T')  $*\"\n    [ $# -gt 0 ] && printf '\\n'\n}\nalias tstamp=timestamp\n\ntimestampcmd(){\n    local output\n    output=\"$(\"$@\" 2>&1)\"\n    timestamp \"$output\"\n}\nalias tstampcmd=timestampcmd\n\n# ============================================================================ #\n\nbak(){\n    # TODO: switch this to a .backupstore folder for keeping this stuff instead\n    for filename in \"$@\"; do\n        [ -n \"$filename\" ] || { echo \"usage: bak filename\"; return 1; }\n        [ -f \"$filename\" ] || { echo \"file '$filename' does not exist\"; return 1; }\n        [[ $filename =~ .*\\.bak\\..* ]] && continue\n        local bakfile\n        bakfile=\"$filename.bak.$(date '+%F_%T' | sed 's/:/-/g')\"\n        until ! [ -f \"$bakfile\" ]; do\n            echo \"WARNING: bakfile '$bakfile' already exists, retrying with a new timestamp\"\n            sleep 1\n            bakfile=\"$filename.bak.$(date '+%F_%T' | sed 's/:/-/g')\"\n        done\n        cp -av -- \"$filename\" \"$bakfile\"\n    done\n}\n\nunbak(){\n    # restores the most recent backup of a file\n    for filename in \"$@\"; do\n        #[ -n \"$filename\" -o \"${filename: -4}\" != \".bak\" ] || { echo \"usage: unbak filename.bak\"; return 1; }\n        [ -n \"$filename\" ] || { echo \"usage: unbak filename\"; return 1; }\n        #[ -f \"$filename\" ] || { echo \"file '$filename' does not exist\"; return 1; }\n        local bakfile\n        local dirname\n        dirname=\"$(dirname \"$filename\")\"\n        filename=\"${filename##*/}\"\n        # don't use -t switch, we want the newest by name, not one that got touched recently\n        bakfile=\"$(find \"$dirname\" -path \"*/$filename.bak.*\" -o -path \"*/$filename.*.bak\" 2>/dev/null | sort | tail -n 1)\"\n        echo \"restoring $bakfile\"\n        cp -av -- \"$bakfile\" \"$dirname/$filename\"\n    done\n}\n\norig(){\n    if [ $# -lt 1 ]; then\n        echo \"usage: orig file1 file2 file3 ...\"\n        return 1\n    fi\n    for filename in \"$@\"; do\n        [ -f \"$filename\" ] || { echo \"file '$filename' does not exist\"; return 1; }\n        [ -f \"$filename.org\" ] && { echo \"$filename.orig already exists, aborting...\"; return 1; }\n    done\n    for filename in \"$@\"; do\n        cp -av -- \"$filename\" \"$filename.orig\"\n    done\n}\n\nunorig(){\n    if [ $# -lt 1 ]; then\n        echo \"usage: unorig file1.orig file2.orig file3.orig ...\"\n        return 1\n    fi\n    for filename in \"$@\"; do\n        if [ -z \"$filename\" ] || [ \"${filename: -5}\" != \".orig\" ]; then\n            echo \"usage: unorig filename.orig\"\n            return 1\n        fi\n        if ! [ -f \"$filename\" ]; then\n            echo \"file '$filename' does not exist\"\n            return 1\n        fi\n    done\n    for filename in \"$@\"; do\n        cp -av -- \"$filename\" \"${filename%.orig}\"\n    done\n}\n\n# ============================================================================ #\n\nstrLastIndexOf(){\n    local str=\"$1\"\n    local substr=\"$2\"\n    local remainder=\"${str##*\"$substr\"}\"\n    local lastIndex=$((${#str} - ${#remainder}))\n    echo $lastIndex\n}\n\n# ============================================================================ #\n\nprogs(){\n    # not passing function f()\n    # shellcheck disable=SC2033\n    find \"${@:-.}\" -type f |\n    grep -Evf ~/code_regex_exclude.txt |\n    grep -v -e '/lib/' -e '.*-env.sh' -e '/tests/'\n}\n\nprogs2(){\n    # not passing function f()\n    # shellcheck disable=SC2033\n    find \"${@:-.}\" -type f -o -type l |\n    grep -Evf ~/code_regex_exclude.txt\n}\n\nfindpy(){\n    # not passing function f()\n    # shellcheck disable=SC2033\n    find \"${@:-.}\" -type f -iname '*.py' -o -type f -iname '*.jy' |\n    grep -vf ~/code_regex_exclude.txt\n}\n\n# ============================================================================ #\n\nldapmaxuid(){\n    ldapsearch -x -W \"uidNumber=*\" uidNumber |\n    sed 's/#.*//' |\n    grep -v \"^[[:space:]]*$\" |\n    grep uidNumber |\n    sort -k2n |\n    tail -n1\n}\n\nldapmaxuidgid(){\n    ldapsearch -xW -x -W \"(|(objectClass=posixAccount)(objectClass=posixGroup))\" uidNumber gidNumber |\n    sed 's/#.*//' |\n    grep --color=auto -v \"^[[:space:]]*$\" |\n    grep -R --color=auto \"(uidNumber|gidNumber)\" |\n    sort -k2n |\n    tail -n1\n}\n"
  },
  {
    "path": ".bash.d/gcp.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC1090,SC1091\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-14 22:22:35 +0000 (Thu, 14 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#              G C P  -  G o o g l e   C l o u d   P l a t f o r m\n# ============================================================================ #\n\nsrcdir=\"${srcdir:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$srcdir/.bash.d/paths.sh\"\n\n# adds GCloud CLI tools to $PATH\nif [ -f ~/google-cloud-sdk/path.bash.inc ]; then\n    #source ~/google-cloud-sdk/path.bash.inc\n    add_PATH ~/google-cloud-sdk/bin  # appends rather than above which prepends to \\$PATH, messing up kubectl version requirement for fluxcd\nfi\n\n# Bash completion for GCloud CLI tools\nif [ -f ~/google-cloud-sdk/completion.bash.inc ]; then\n    source ~/google-cloud-sdk/completion.bash.inc\nfi\n\nexport USE_GKE_GCLOUD_AUTH_PLUGIN=True\n\n# having to retype this way too much\nalias gal=\"gcloud auth login\"\n# often bugs me to do this\nalias gcu=\"gcloud components update\"\n\nalias gce=\"gcloud compute\"\nalias gke=\"gcloud container clusters\"\nalias gc=\"gcloud container\"\nalias gbs=\"gcloud builds submit --tag\"\nalias bqq=\"bq query\"\nalias gssh=\"gcloud compute ssh\"\n\n# open GCP Console in the current project and preferably on a relevant page if we can detect one\ngcpcon(){\n    local project\n    local path\n    # open in the compute instances page if we don't know where else to go\n    path=\"compute/instances\"\n    if [[ \"$PWD\" =~ kubernetes|k8 ]]; then\n        path=\"kubernetes/list/overview\"\n    elif [[ \"$PWD\" =~ iam ]]; then\n        path=\"iam-admin/iam\"\n    fi\n    project=\"${CLOUDSDK_CORE_PROJECT:-$(gcloud config get core.project 2>/dev/null)}\"\n    open \"https://console.cloud.google.com/$path?project=$project\"\n}\n\n# when switching an alias to a function during re-source without un-aliasing, declare function explicitly to avoid errors\nfunction gcloudconfig(){\n    # configurations are usually called the same as the project name so export GOOGLE_PROJECT_ID for convenience too\n    gcloud config configurations activate \"$1\" || return 1\n    export GOOGLE_PROJECT_ID=\"$1\"\n}\n\ngsopen(){\n    local gspath=\"$1\"\n    gspath=\"${gspath#gs:\\/\\/}\"\n    browser \"https://console.cloud.google.com/storage/browser/$gspath\"\n}\n\ngcropen(){\n    local image=\"$1\"\n    if ! [[ \"$image\" =~ gcr\\.io/ ]]; then\n        echo \"'$image' is not a GCR image name (requires gcr.io to know where to open)\"\n        return 1\n    fi\n    image=\"${image%:*}\"\n    browser \"https://$image\"\n}\n"
  },
  {
    "path": ".bash.d/git.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                   R e v i s i o n   C o n t r o l  -  G i t\n# ============================================================================ #\n\n# Primary revision control system\n#\n# if svn.sh and hg.sh functions are enabled, detects and calls svn and mercurial commands if inside those repos so some of the same commands work dynamically\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\nif [ -f ~/.github_token  ]; then\n    GITHUB_TOKEN=\"$(cat ~/.github_token)\"\n    export GITHUB_TOKEN\nfi\n\nif [ -f ~/.gitlab_token  ]; then\n    GITLAB_API_PRIVATE_TOKEN=\"$(cat ~/.gitlab_token)\"\n    export GITLAB_API_PRIVATE_TOKEN\nfi\n#if [ -z \"${GITLAB_API_ENDPOINT:-}\" ]; then\n#    export GITLAB_API_ENDPOINT=\"https://gitlab.com/api/v4\"\n#fi\n\nif ! type basedir &>/dev/null; then\n    # shellcheck disable=SC1090,SC1091\n    . \"$bash_tools/.bash.d/functions.sh\"\nfi\n\nif type -P gh &>/dev/null; then\n    autocomplete gh -s\nfi\n\n# shellcheck disable=SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n#add_PATH ~/bin/codeql\n\n# find out who your 'gh' CLI is authenticating as - useful if you have multiple Personal Access Tokens for different environments\nalias githubwhoami='github_api.sh /user | jq -r .login'\nalias ghwhoami='github_api.sh /user | jq -r .login'\n\n# set location where you check out all the github repos\nexport github=~/github\n\nexport GIT_PAGER=\"less ${LESS:-}\"\n# shellcheck disable=SC2230\n#if [ -z \"${GIT_PAGER:-}\" ] && \\\nif type -P diff-so-fancy &>/dev/null; then\n    # pre-loading a pattern to 'n' / 'N' / '?' / '/' search through will force you in to pager and disregard -F / --quit-if-one-screen\n    #export GIT_PAGER=\"diff-so-fancy --color=yes | less -RFX --tabs=4 --pattern '^(Date|added|deleted|modified): '\"\n    export GIT_PAGER=\"diff-so-fancy --color=yes | $GIT_PAGER\"\nfi\n\nalias gitconfig=\"\\$EDITOR ~/.gitconfig\"\nalias gitignore=\"\\$EDITOR ~/.gitignore_global\"\nalias gitrc=gitconfig\n# false positive, not calling this from xargs\n# shellcheck disable=SC2032\n#alias add=gitadd\nadd(){ gitadd \"$@\"; }\naddf(){\n    git add -f \"$@\"\n    git ci -m \"added $*\"\n}\nalias gadd='git add'\n# beware covers up ImageMagick 'import' screenshot command (see HariSekhon/Knowledge-Base mac.md page)\nalias import=gitimport\nalias co=checkout\nalias commit=\"git commit\"\nalias clone=\"git clone\"\nalias cherry-pick=\"git cherry-pick\"\nalias gitci=commit\nalias ci=commit\nalias gitco=checkout\nalias up=pull\nalias u=up\nalias uu=\"GIT_PULL_IN_BACKGROUND=1 u\"\nalias pu=push\nalias gitp=\"git push\"\nalias gdiff=\"git diff\"\n# bypasses diff-so-fancy, could also just pipe through | cat to disable pager and color effects\nalias gdiff2=\"git --no-pager diff\"\nalias gdiffc=\"git diff --cached\"\nalias gdiffm=\"gdiff origin/master..\"\nalias gd=gdiff\nalias gdc=gdiffc\nalias gdo=gdiffm\nalias branch=\"githg branch\"\nalias br=branch\nalias fetch='git fetch'\nalias stash=\"git stash\"\nalias tag=\"githg tag\"\nalias tags='git tag'\nalias tagr='git_tag_release.sh'\nalias gitlogwc='git log --oneline | wc -l'\nalias um=git_submodules_update.sh\n#type browse &>/dev/null || alias browse=gbrowse\nalias gbrowse=gitbrowse\nalias gb='gitbrowse'\nalias prbrowse='gh pr view --web'  # uses GitHub CLI\nalias prb=prbrowse\n#alias gh='gitbrowse github'  # clashes with GitHub CLI\nalias gl='gitbrowse gitlab'\nalias bb='gitbrowse bitbucket'\nalias azd='gitbrowse azure'\nalias gha='gitbrowse github actions'\nalias ghw='github_workflows'\nalias wf='cd $(git_root)/.github/workflows/'\nalias ggrep=\"git grep\"\nalias gfr='git_foreach_repo.sh'\nalias gfrur='git_foreach_repo_update_readme.sh'\nalias gfrrra='git_foreach_repo_replace_readme_actions.sh'\nalias remotes='git remote -v'\nalias remote='remotes'\n# much quicker to just 'cd $github; f <pattern>'\n#githubls(){\n#    # GitHub is svn compatible, use this to list files remotely\n#    svn ls \"https://github.com/$1.git/branches/master/\"\n#}\n#githubgrep(){\n#    for repo in $(sed 's/#.*//;s/:.*//;/^[[:space:]]*$/d' \"$srcdir/setup/repos.txt\"); do\n#        githubls \"HariSekhon/$repo\"\n#    done |\n#    grep \"$@\"\n#}\n\n# git fetch -p or git remote prune origin\n#alias prune=\"co master; git pull --no-edit; git remote prune origin; git branch --merged | grep -v -e '^\\\\*' -e 'master' | xargs git branch -d\"\nremoved_branches(){\n    git remote prune origin >&2\n    git branch -vv |\n    cut -c 3- |\n    awk '$4 ~ /gone\\]/ {print $1}'\n}\nalias prune=\"removed_branches | xargs -r git branch -d\"\n\n# don't use this unless you are a git pro and understand unwinding history and merge conflicts\nalias GRH=\"git reset HEAD^\"\n\nalias gitnored='git status --ignored'\n\nalias master=\"switchbranch master\"\nalias main=\"switchbranch main\"\nalias prod=\"switchbranch prod\"\nalias production=\"switchbranch production\"\nalias staging=\"switchbranch staging\"\nalias stage=staging\nalias dev=\"switchbranch dev\"\nalias develop=\"switchbranch develop\"\nalias grp=\"git_review_push.sh\"\n\n# edit all GitHub READMEs\nalias readmes='$EDITOR $(git_foreach_repo.sh '\"'\"'echo $PWD/README.md'\"')\"\nalias readmesi='idea $(git_foreach_repo.sh '\"'\"'echo $PWD/README.md'\"')\"\nalias ureadmes='git_foreach_repo.sh '\"'\"'gitu README.md || :'\"'\"\n\n# equivalent of hg root\n# shellcheck disable=SC2120\ngit_root(){\n    local path=\"${1:-.}\"\n    local dir\n    if [ -f \"$path\" ]; then\n        dir=\"$(dirname \"$path\")\"\n    elif [ -d \"$path\" ]; then\n        dir=\"$path\"\n    else\n        echo \"ERROR: arg passed is not a regular file or directory: $path\" >&2\n        return 1\n    fi\n    pushd \"$dir\" &>/dev/null || return 1\n    git rev-parse --show-toplevel\n    popd &>/dev/null || return 1\n}\nalias gitroot=git_root\nalias cdgitroot='cd \"$(git_root)\"'\n\ngitgc(){\n    cd \"$(git_root)\" || :\n    if ! [ -d .git ]; then\n        echo \"not in a git repo, no .git/ directory and git root dir not found\"\n        return 1\n    fi\n    du -sh .git\n    git gc --aggressive\n    du -sh .git\n}\n\ngit_default_branch(){\n    git remote show origin |\n    #awk '/^[[:space:]]*HEAD branch:[[:space:]]/{print $3}'\n    sed -n '/HEAD branch/s/.*: //p'\n}\n\ngit_url_base(){\n    local filter=\"${1:-.*}\"\n    git remote -v |\n    { grep \"$filter\" || : ; }|\n    awk '/git@|https:\\/\\/|ssh:\\/\\//{print $2}' |\n    head -n1 |\n    sed 's|^ssh://||;\n         s|^https://.*@||;\n         s|^https://||;\n         s|:[[:digit:]][[:digit:]]*||;\n         s/^git@ssh.dev.azure.com:v3/dev.azure.com/;\n         s|^git@||;\n         s|^|https://|;\n         s/\\.git$//;\n         # Azure DevOps only puts this in https urls, not ssh, so strip for standardizing output\n         # Update: actually dont do this because we cannot differentiate internal Azure DevOps\n         # by internal fqdn urls so it is better to leave this in\n         #s|/_git/|/|;\n         ' |\n    perl -pe 's/:(?!\\/\\/)/\\//'\n}\n\ngitbrowse(){\n    local filter=\"${1:-origin}\"\n    local path=\"${2:-}\"\n    local url_base\n    url_base=\"$(git_url_base \"$filter\")\"\n    if [ -z \"$url_base\" ] && [ \"$filter\" != origin ]; then\n        url_base=\"$(git_url_base \"origin\")\"\n    fi\n    if [ -z \"$url_base\" ]; then\n        echo -n \"git remote url not found for filter '$filter'\"\n        if [ \"$filter\" != \"origin\" ]; then\n            echo \" or 'origin'\"\n        else\n            echo\n        fi\n        return 1\n    fi\n    if [ -n \"$path\" ]; then\n        path=\"$(git ls-files --full-name \"$path\")\"\n    fi\n    if [[ \"$url_base\" =~ github.com ]]; then\n        if [ -z \"$path\" ]; then\n            path+=\"#readme\"\n        fi\n    else\n        if [ -z \"$path\" ]; then\n            local default_branch\n            default_branch=\"$(git_default_branch)\"\n        fi\n        if [[ \"$url_base\" =~ gitlab.com ]]; then\n            if [ -z \"$path\" ]; then\n                url_base+=\"/-/blob/$default_branch/README.md\"\n            fi\n        elif [[ \"$url_base\" =~ dev.azure.com ]]; then\n            # don't re-add this as it's no longer stripped out in git_url_base\n            # because we cannot differentiate internal Azure Devops by fqdn so need to leave it in\n            #url_base=\"${url_base%/*}/_git/${url_base##*/}\"\n            if [ -z \"$path\" ]; then\n                url_base+=\"?path=/README.md&_a=preview\"\n            fi\n        elif [[ \"$url_base\" =~ bitbucket.org ]]; then\n            if [ -z \"$path\" ]; then\n                url_base+=\"/src/$default_branch/README.md\"\n            fi\n        fi\n    fi\n    url=\"$url_base\"\n    if [ -n \"$path\" ]; then\n        if [[ \"$url_base\" =~ github.com ]]; then\n            local default_branch\n            default_branch=\"$(git_default_branch)\"\n            url+=\"/blob/$default_branch\"\n        fi\n        url+=\"/$path\"\n    fi\n    browser \"$url\"\n}\n\ninstall_git_completion(){\n    if ! [ -f ~/.git-completion.bash ]; then\n        wget -O ~/.git-completion.bash https://raw.githubusercontent.com/git/git/master/contrib/completion/git-completion.bash\n    fi\n}\n\n# shellcheck disable=SC1090,SC1091\n[ -f ~/.git-completion.bash ] && . ~/.git-completion.bash\n\n# usage: gi python,perl,go\n#        gi list\ngitignore_api(){\n    local url\n    local langs\n    local options=()\n    local args=()\n    # noop - set to use 'tr' to separate items to newlines when given the 'list' arg\n    local commas_to_newlines=\"cat\"\n    for arg; do\n        if [ \"$arg\" = -- ]; then\n            options+=(\"$arg\")\n        else\n            args+=(\"$arg\")\n        fi\n    done\n    # take args 'python perl', store as 'python,perl' for the API call\n    langs=\"$(IFS=, ; echo \"${args[*]}\")\"\n    url=\"https://www.gitignore.io/api/$langs\"\n    if [ \"$langs\" = \"list\" ]; then\n        commas_to_newlines=\"tr ',' '\\\\n'\"\n    fi\n    {\n    if hash curl 2>/dev/null; then\n        curl -sSL \"${options[*]}\" \"$url\"\n    elif hash wget 2>/dev/null; then\n        wget -O - \"${options[*]}\" \"$url\"\n    fi\n    } | eval \"$commas_to_newlines\"\n    echo\n}\nalias gi=gitignore_api\n\ngit_user_repo(){\n    git remote -v | awk '{print $2}' | head -n1 | git_repo_strip\n}\n\ngit_repo(){\n    git_user_repo | sed 's|.*/||'\n}\n\ngithub_owner_repo(){\n    git remote -v | awk '/github.com/{print $2}' | head -n1 | git_repo_strip\n}\ngithub_user_repo(){\n    github_owner_repo\n}\n\ngithub_repo(){\n    github_user_repo | sed 's|.*/||'\n}\n\ngit_repo_strip(){\n    git_repo_strip_auth | sed 's|.*\\.[^:/]*[:/]||; s/\\.git[[:space:]]*$//'\n}\n\ngit_repo_strip_auth(){\n    sed 's/[[:alnum:]]*@//'\n}\n\nisGit(){\n    local target=${1:-.}\n    # There aren't local .hg dirs everywhere only at top level so this is difficult in bash\n    if [ -d \"$target/.git\" ]; then\n        return 0\n    elif [ -f \"$target\" ] &&\n         [ -d \"${target%/*}/.git\" ]; then\n        #-o \"$target/../.git\" -o \"${target%/*}/../.git\" ]; then\n        return 0\n    else\n        # This is because git command doesn't return correctly when running from outside git root, complains there is not .git\n        if [ -d \"$target\" ]; then\n            pushd \"$target\" >/dev/null || return 1\n            #if [ -n \"$(git log -1 . 2>/dev/null)\" ]; then\n            # better because it will succeed in subdirectories of git repos which are not checked in yet\n            if git status &>/dev/null; then\n                # shellcheck disable=SC2164\n                popd &>/dev/null\n                return 0\n            fi\n        else\n            pushd \"$(dirname \"$target\")\" >/dev/null || return 1\n            #if git log -1 \"$target\" 2>/dev/null | grep -q '.*'; then\n            #if [ -n \"$(git log -1 \"$(basename \"$target\")\" 2>/dev/null)\" ]; then\n            # better because it will succeed in subdirectories of git repos which are not checked in yet\n            if git status &>/dev/null; then\n                # shellcheck disable=SC2164\n                popd &>/dev/null\n                return 0\n            fi\n        fi\n        # shellcheck disable=SC2164\n        popd &>/dev/null\n        return 2\n    fi\n}\n\ngit_revision(){\n    echo \"Revision: $(git rev-parse HEAD)\"\n}\n\nst(){\n  # shellcheck disable=SC2086\n  {\n    local target=\"${1:-.}\"\n    shift\n    if ! [ -e \"$target\" ]; then\n        echo \"$target does not exist\"\n        return 1\n    fi\n    local target_basename\n    local target_dirname\n    target_basename=\"$(basename \"$target\")\"\n    target_dirname=\"$(dirname \"$target\")\"\n    #if [ -f \"Vagrantfile\" ]; then\n    #    echo \"> vagrant status\"\n    #    vagrant status\n    # shellcheck disable=SC2166\n    if [ \"$target\" = \".\" ] &&\n       [ \"$PWD\" = \"$HOME/github\" ]; then\n        hr\n        for x in \"$target\"/*; do\n            [ -d \"$x\" ] || continue\n            pushd \"$x\" >/dev/null || { echo \"failed to pushd to '$x'\"; return 1; }\n            if git remote -v | grep -qi harisekhon; then\n                echo \"> GitHub: git status $x $*\"\n                git status . \"$@\"\n                echo\n                hr\n                echo\n            fi\n            # shellcheck disable=SC2164\n            popd &>/dev/null\n        done\n    elif { [ \"$target\" = \".\" ] &&\n         [ \"${PWD##*/}\" = work ] ; } ||\n         grep -Fxq \"$PWD\" <<< \"${GIT_BASEDIRS:-}\" ||\n         [ -f .iterate ]; then\n         #ls ./*/.git &>/dev/null; then  # matches inside repos with submodules unfortunately\n        hr\n        for x in \"$target\"/*; do\n            [ -d \"$x\" ] || continue\n            pushd \"$x\" >/dev/null || { echo \"failed to pushd to '$x'\"; return 1; }\n            echo \"> Work: git status $x $*\"\n            git status . \"$@\"\n            echo\n            hr\n            echo\n            # shellcheck disable=SC2164\n            popd &>/dev/null\n        done\n    elif isGit \"$target\"; then\n        if [ -d \"$target\" ]; then\n            pushd \"$target\" >/dev/null || { echo \"Error: failed to pushd to $target\"; return 1; }\n            echo \"> git stash list\" >&2\n            git stash list && echo\n            #\"$bash_tools/git/git_summary_line.sh\"\n            echo \"> git status $target $*\" >&2\n            #git -c color.status=always status -sb . \"$@\"\n            git -c color.status=always status . \"$@\"\n            echo\n            git_revision\n            echo\n        else\n            pushd \"$target_dirname\" >/dev/null || { echo \"Error: failed to pushed to '$target_dirname'\"; return 1; }\n            echo \"> git status $target $*\" >&2\n            #\"$bash_tools/git/git_summary_line.sh\"\n            git -c color.status=always status \"$target_basename\" \"$@\"\n        fi\n        #git status \"$target\" \"${*:2}\"\n        # shellcheck disable=SC2164\n        popd &>/dev/null\n    elif type isHg &>/dev/null && isHg \"$target\"; then\n        echo \"> hg status $target $*\" >&2\n        hg status \"$target\" \"$@\" | grep -v \"^?\"\n        # to see relative paths instead of the default absolute paths\n        #hg status \"$(hg root)\"\n    elif type isSvn &>/dev/null && isSvn \"$target\"; then\n        echo \"> svn st $*\" >&2\n        svn st --ignore-externals \"$target\" \"$@\" | grep -v -e \"^?\" -e \"^x\";\n    else\n        echo \"not a revision controlled resource as far as bashrc can tell\"\n    fi\n  } |\n  # more calls less on Mac, and gets stuck in interactive mode ignoring the less alias switches\n  #more -R -n \"$((LINES - 3))\"\n  #less -RFX\n  eval ${GIT_PAGER:-cat}\n}\n\nstq(){\n    st \"$@\" | grep --color=no -e \"=======\" -e branch -e GitHub | eval \"${GIT_PAGER:-cat}\"\n}\n\n# disabling this as I don't use Mercurial or Svn any more,\n# replacing with simpler function below that will pass through more things like --rebase\n#pull(){\n#    local target=\"${1:-.}\"\n#    if ! [ -e \"$target\" ]; then\n#        echo \"$target does not exist\"\n#        return 1\n#    fi\n#    local target_basename\n#    target_basename=\"$(basename \"$target\")\"\n#    # shellcheck disable=SC2166\n#    if [ \"$target_basename\" = \"github\" ] || [ \"$target\" = \".\" -a \"$(pwd)\" = \"$github\" ]; then\n#        for x in \"$target\"/*; do\n#            [ -d \"$x\" ] || continue\n#            # get last character of string\n#            [ \"${x: -1}\" = 2 ] && continue\n#            pushd \"$x\" >/dev/null || { echo \"failed to pushd to '$x'\"; return 1; }\n#            if git remote -v | grep -qi harisekhon; then\n#                echo \"> GitHub: git pull $x ${*:2}\"\n#                git pull \"${@:2}\"\n#                echo\n#                echo \"> GitHub: git submodule update --init --recursive\"\n#                git submodule update --init --recursive\n#                echo\n#            fi\n#            # shellcheck disable=SC2164\n#            popd &>/dev/null\n#        done\n#        return\n#    elif isGit \"$target\"; then\n#        pushd \"$target\" >/dev/null &&\n#        echo \"> git pull -v ${*:2}\" >&2\n#        git pull -v \"${@:2}\"\n#        echo \"> git submodule update --init --recursive\"\n#        git submodule update --init --recursive\n#        #local orig_branch=$(git branch | awk '/^\\*/ {print $2}')\n#        #for branch in $(git branch | cut -c 3- ); do\n#        #    git checkout -q \"$branch\" &&\n#        #    echo -n \"$branch => \" &&\n#        #    git pull -v\n#        #    echo\n#        #    echo\n#        #done\n#        #git checkout -q \"$orig_branch\"\n#        # shellcheck disable=SC2164\n#        popd &>/dev/null\n#    elif type isHg &>/dev/null && isHg \"$target\"; then\n#        pushd \"$target\" >/dev/null &&\n#        echo \"> hg pull && hg up\" >&2  &&\n#        hg pull && hg up\n#        # shellcheck disable=SC2164\n#        popd &>/dev/null\n#    elif type isSvn &>/dev/null && isSvn \"$target\"; then\n#        echo \"> svn up $target\" >&2\n#        svn up \"$target\"\n#    else\n#        echo \"not a revision controlled resource as far as bashrc can tell\"\n#        return 1\n#    fi\n#}\n\n# simpler replacement function to above\n# shellcheck disable=SC2120\npull(){\n    # shellcheck disable=SC2166\n    if [ \"$PWD\" = \"$HOME/github\" ]; then\n        for x in *; do\n            [ -d \"$x/.git\" ] || continue\n            # get last character of string - don't pull blah2, as I use them as clean checkouts\n            [ \"${x: -1}\" = 2 ] && continue\n            pushd \"$x\" >/dev/null || { echo \"failed to pushd to '$x'\"; return 1; }\n            if git remote -v | grep -qi \"${GITHUB_USER:-${GIT_USER:-${USER:-}}}\"; then\n                hr\n                echo \"> GitHub $x: git pull --no-edit $*\"\n                #echo \"> GitHub $x: git submodule update --init --recursive\"\n                if [ -n \"${GIT_PULL_IN_BACKGROUND:-}\" ]; then\n                    git_pull \"$@\" &\n                else\n                    git_pull \"$@\"\n                fi\n            fi\n            # shellcheck disable=SC2164\n            popd &>/dev/null\n        done\n    elif [ \"${PWD##*/}\" = work ] ||\n         grep -Fxq \"$PWD\" <<< \"${GIT_BASEDIRS:-}\" ||\n         [ -f .iterate ]; then\n         #ls ./*/.git &>/dev/null; then  # matches inside repos with submodules unfortunately\n        for x in *; do\n            [ -d \"$x/.git\" ] || continue\n            hr\n            pushd \"$x\" >/dev/null || { echo \"failed to pushd to '$x'\"; return 1; }\n            echo \"> Work $x: git pull --no-edit $*\"\n            #echo \"> work $x: git submodule update --init --recursive\"\n            if [ -n \"${GIT_PULL_IN_BACKGROUND:-}\" ]; then\n                git_pull \"$@\" &\n            else\n                git_pull \"$@\"\n            fi\n            # shellcheck disable=SC2164\n            popd &>/dev/null\n        done\n    else\n        echo \"> git pull --no-edit $*\"\n        echo \"> git submodule update --init --recursive\"\n        git_pull \"$@\"\n    fi\n}\n\ngit_pull(){\n    echo\n    git pull --no-edit \"$@\"\n    echo\n    git submodule update --init --recursive\n    echo\n}\n\nalias coj=\"git_branch_jira_ticket\"\ngit_branch_jira_ticket(){\n    local ticket=\"$1\"\n    local branch=\"${ticket##*/}\"\n    if git branch | sed 's/^..//' | grep -Fxq \"$branch\"; then\n        git checkout \"$branch\"\n    else\n        git checkout -b \"$branch\"\n    fi\n}\n\ncheckout(){\n    if isGit \".\"; then\n        git checkout \"$@\";\n    else\n        echo \"not a Git checkout, cannot switch to branch $*\"\n        return 1\n    fi\n}\n\n_gitaddimport() {\n    local action=\"$1\"\n    shift;\n    [ -z \"$*\" ] && return 1\n    local basedir\n    local trap_codes=\"INT ERR\"\n    for filename in \"$@\"; do\n        basedir=\"$(basedir \"$filename\")\"\n        # shellcheck disable=SC2064,SC2086\n        trap \"popd &>/dev/null; trap - $trap_codes; return 1 2>/dev/null\" $trap_codes;\n        pushd \"$basedir\" > /dev/null || return 1;\n        # shellcheck disable=SC2086\n        filename=\"$(strip_basedirs \"$basedir\" \"$filename\")\";\n        if ! [ -e \"$filename\" ]; then\n            echo \"ERROR: $filename does not exist\" >&2\n            #return 1\n        elif git status -s \"$filename\" | grep -q '^[?A]'; then\n            git add \"$filename\" &&\n            git commit -m \"$action $filename\" \"$filename\"\n        elif git status -s \"$filename\" | grep -q '^.M'; then\n            echo \"ERROR: '$filename' already in git, but has changes, commit as an update instead\" >&2\n            #return 1\n        elif git status --ignored -s \"$filename\" | grep -q '^!!'; then\n            echo \"ERROR: '$filename' is ignored!!! =>  $(git check-ignore -v \"$filename\")\" >&2\n            #return 1\n        else\n            echo \"ERROR: '$filename' already in git\" >&2\n            #return 1\n        fi\n        popd > /dev/null || return 1\n    done\n}\n\ngitadd(){\n    _gitaddimport added \"$@\"\n}\n\ngitimport(){\n    _gitaddimport imported \"$@\"\n}\n\n\ngitu(){\n    git_diff_commit.sh \"$@\"\n}\ngituu(){\n    # avoiding xargs due to function reference:\n    # gxargs: gitu: No such file or directory\n    eval gitu \"$(\n        git status --porcelain -s . |\n        grep -e '^M' -e '^.M' |\n        sed 's/^...//' |\n        while read -r filename; do\n            echo \"\\\"$filename\\\"\"\n        done\n    )\"\n}\n\n#githgu(){\n#    target=\"${1:-.}\"\n#    #count=0\n#    while [ -L \"$target\" ]; do\n#        #target=\"$(readlink \"$target\")\"\n#        #let count+=1\n#        #if [ $count -gt 10 ]; then\n#        #    echo \"looping over links more than 10 times in hggitu! \"\n#        #    exit 2\n#        #fi\n#        echo \"$target is a symlink! \"\n#        return 1\n#    done\n#    if ! [ -e \"$target\" ]; then\n#        echo \"$target does not exist\"\n#        return 1\n#    fi\n#    if isGit \"$target\"; then\n#        echo \"> git\" >&2\n#        #if [ -d \"$target\" ]; then\n#        #    pushd \"$target\" >/dev/null\n#        #else\n#        #    pushd \"$(dirname \"$target\")\" >/dev/null\n#        #fi\n#        #\"$srcdir2/gitu\" \"${target##*/}\" &&\n#        gitu \"$target\"\n#        #popd &>/dev/null\n#    elif type isHg &>/dev/null && isHg \"$target\"; then\n#        echo \"> hg\" >&2\n#        #if [ -d \"$target\" ]; then\n#        #    pushd \"$target\" >/dev/null\n#        #else\n#        #    pushd \"$(dirname \"$target\")\" >/dev/null\n#        #fi\n#        #\"$srcdir2/hgu\" \"${target##*/}\" &&\n#        hgu \"$target\"\n#        #popd &>/dev/null\n#    # Not supporting SVN any more\n#    #elif type isSvn &>/dev/null && isSvn \"$target\"; then\n#    #    echo \"> svn\" >&2\n#    #    svnu \"$target\"\n#    else\n#        echo \"not a revision controlled resource as far as bashrc can tell\"\n#        return 1\n#    fi\n#}\n\npush(){\n    # shellcheck disable=SC2119\n    pull || return 1\n    if isGit .; then\n        echo \"> git push $*\"\n        #for remote in $(git remote); do\n        #    git push -v $remote $@\n        #done\n        # can't be sure where we're pushing without parsing the command args, so omit for now\n        if [ $# -eq 0 ]; then\n            echo \"pushing to:\"\n            # uniq_ordered.pl from my DevOps-Perl-tools repo or\n            # uniq2 from my DevOps-Golang-tools repo would be better here\n            # but not sure I want to create a dependency on that\n            # unix's standard uniq unfortnately will only deduplicate adjacent lines but should be good enough in most cases\n            git remote -v | awk '/^origin/{print $1\"\\t\"$2}' | sed 's,://.*@,://,' | uniq\n            echo\n        fi\n        # exposes your Github / GitLab / Bitbucket tokens on the screen, not secure, use printing above instead\n        #git push -v \"$@\"\n        git push \"$@\"\n        echo\n        st\n    elif type isHg &>/dev/null && isHg .; then\n        echo \"> hg push $*\"\n        hg push \"$@\"\n    else\n        echo \"not in a Git or Mercurial controlled directory\"\n        return 1\n    fi\n}\nunalias pushu 2>/dev/null || :\npushu(){\n    if git remote -v | grep -qi '^origin[[:space:]].*gitlab\\.'; then\n        \"$bash_tools/gitlab/gitlab_push_mr_preview.sh\"\n    else\n        \"$bash_tools/github/github_push_pr_preview.sh\"\n    fi\n}\nunalias pushup 2>/dev/null || :\npushup(){\n    if git remote -v | grep -qi '^origin[[:space:]].*gitlab\\.'; then\n        \"$bash_tools/gitlab/gitlab_push_mr.sh\"\n    else\n        \"$bash_tools/github/github_push_pr.sh\"\n    fi\n}\nunalias pushupmerge 2>/dev/null || :\npushupmerge(){\n    if git remote -v | grep -qi '^origin[[:space:]].*gitlab\\.'; then\n        GITLAB_MERGE_PULL_REQUEST=true \\\n        \"$bash_tools/gitlab/gitlab_push_mr.sh\"\n    else\n        GITHUB_MERGE_PULL_REQUEST=true \\\n        \"$bash_tools/github/github_push_pr.sh\"\n    fi\n}\nalias pushupm=pushupmerge\n\npushr(){\n    for remote in $(git remote); do\n        echo \"> git push \\\"$remote\\\"\"\n        git push \"$remote\"\n        echo\n    done\n}\n\nalias pr=github_pull_request_create.sh\n\nalias mup=masterupdateprune\nmasterupdateprune(){\n    #local master_branch=\"master\"\n    #if git branch | sed 's/^..//' | grep -Fx main; then\n    #    master_branch=\"main\"\n    #fi\n    git checkout \"$(git_default_branch)\"\n    pull\n    prune\n    git_revision\n    echo\n}\n\ncurrent_branch(){\n    git rev-parse --abbrev-ref HEAD\n}\nalias currentbranch=current_branch\n\nswitchbranch(){\n    if isGit \".\"; then\n        git checkout \"$1\";\n    elif type isHg &>/dev/null && isHg \".\"; then\n        hg update \"$1\"\n    else\n        echo \"not a Git / Mercurial checkout, cannot switch to branch $1\"\n        return 1\n    fi\n}\n\ngitrm(){\n    git rm -- \"$@\" &&\n    git commit -m \"removed $*\" -- \"$@\"\n}\n\ngitrename(){\n    if [ $# -ne 2 ]; then\n        echo \"usage: gitrename <original_filename> <new_filename>\"\n        return 1\n    fi\n    if [ -f \"$2\" ]; then\n        local file_already_exists=1\n        mv -iv -- \"$2\" \"$2.tmp\"\n    fi\n    git mv -- \"$1\" \"$2\" &&\n    git commit -m \"renamed $1 to $2\" \"$1\" \"$2\"\n    if [ \"${file_already_exists:-}\" = 1 ]; then\n        mv -fv -- \"$2.tmp\" \"$2\"\n    fi\n}\n\ngitmv(){\n    if [ $# -ne 2 ]; then\n        echo \"usage: gitmv <original_filename> <new_filename>\"\n        return 1\n    fi\n    if [ -f \"$2\" ]; then\n        local file_already_exists=1\n        mv -iv -- \"$2\" \"$2.tmp\"\n    fi\n    git mv -- \"$1\" \"$2\" &&\n    git commit -m \"moved $1 to $2\" \"$1\" \"$2\"\n    if [ \"${file_already_exists:-}\" = 1 ]; then\n        mv -fv -- \"$2.tmp\" \"$2\"\n    fi\n}\n\ngitd(){\n    git diff \"${@:-.}\"\n}\n\ngitadded(){\n    git log --name-status \"$@\" |\n    grep -e '^A[^u]' -e '^Date' |\n    grep -B 1 '^A' |\n    less\n}\n\n# doesn't need pipe | less, git drops you in to less anyway\ngitl(){\n    git log --all --graph --decorate --name-status \"$@\"\n}\n\ngitlp(){\n    git log -p \"$@\"\n}\nalias gitlp1=\"gitlp -1\"\n\ngitl2(){\n    git log --all --graph --decorate --stat \"$@\"\n}\n\ngitl3(){\n    git log --pretty=format:\"%n%an => %ar%n%s\" --name-status \"$@\"\n}\n\ngithg(){\n    if isGit .; then\n        git \"$@\"\n    elif type isHg &>/dev/null && isHg .; then\n        hg \"$@\"\n    else\n        echo \"not a Git/Mercurial checkout\"\n        return 1\n    fi\n}\n\nretag(){\n    local tag1=\"$1\"\n    local checksum=\"$2\"\n    local additional_tags=\"${*:2}\"\n    for tag in $tag1 $additional_tags; do\n        git tag -d \"$tag\" || :\n        echo \"Creating git tag '$tag'\"\n        # quoting checksum causes failure with unrecognized checksum ''\n        git tag \"$tag\" \"$checksum\"\n        git tag |\n        grep -qF \"$tag\" ||\n            echo \"FAILED\"\n    done\n}\n\ngitfind(){\n    local refids\n    refids=\"$(git log --all --oneline | grep \"$@\" | awk '{print $1}')\"\n    printf 'Branches:\\n\\n'\n    for refid in $refids; do\n        git branch --contains \"$refid\"\n    done | sort -u\n    printf '\\nTags:\\n\\n'\n    for refid in $refids; do\n        git tag --contains \"$refid\"\n    done | sort -u\n}\n\n#stagemerge(){\n#    if isGit \".\"; then\n#        git checkout prod    && git pull &&\n#        git checkout staging && git pull &&\n#        git merge prod\n#        git checkout prod\n#    else\n#        echo \"Not a Git working copy\";\n#    fi\n#}\n\ngitdiff(){\n    local filename=\"${1:-}\"\n    [ -n \"$filename\" ] || { echo \"usage: gitdiff filename\"; return 1; }\n    git diff \"$filename\" > \"/tmp/gitdiff.tmp\"\n    diffnet.pl \"/tmp/hgdiff.tmp\"\n}\n\ngit_author_names(){\n    git log --all --pretty=format:\"%an\" | sort | uniq -c | sort -k1nr | less\n}\n\ngit_author_emails(){\n    git log --all --pretty=format:\"%ae\" | sort | uniq -c | sort -k1nr | less\n}\n\ngit_author_names_emails(){\n    git log --all --pretty=format:\"%an %ae\" | sort | uniq -c | sort -k1nr | less\n}\n\ngit_authors(){\n    git_author_emails\n}\n\ngit_commit_count(){\n    # interestingly, even on 10,000 commit repos, there are no duplicate short hashes shown from:\n    # git log --all --pretty=format:\"%h\" | sort | uniq -d\n    git log --all --pretty=format:\"%h\" | wc -l\n}\n\ngit_revert_typechange(){\n    # want splitting to separate filenames\n    # shellcheck disable=SC2046\n    co $(git status --porcelain -s \"${1:-.}\" | awk '/^.T/{print $2}')\n}\n\ngit_rm_untracked(){\n    if [ $# -lt 1 ]; then\n        echo \"usage: rm_untracked <target_dir_or_files_or_glob>\"\n        return 1\n    fi\n    # iterate on explicit targets only\n    # intentionally not including current directory to avoid accidentally wiping out untracked files - you must specify \"rm_untracked .\" if you really intend this\n    for x in \"${@:-}\"; do\n        git status --porcelain -s --untracked-files=all \"$x\" |\n        # this breaks the correct spacings for Spotify playlist filenames\n        #awk '/^\\?\\?/{$1=\"\"; print}' |\n        grep '^??' |\n        sed 's/^?? //' |\n        while read -r filename; do\n            # git status --porcelain double quotes file paths when containing unicode chars which are representated in \\xxx format\n            # you must set 'git config --global core.quotePath false' for this to work properly\n            #\n            # this doesn't help because you are still stuck with \\xxx chars throughout\n            filename=\"${filename#\\\"}\"\n            filename=\"${filename%\\\"}\"\n            rm -v -- \"$filename\" || break\n        done\n    done\n}\n\n# example of usage of this in the function below - make sure to put '$repo' or \"\\$repo\" somewhere in the argument body to make use of the iteration variable\nforeachrepo(){\n    local repolist=\"${REPOLIST:-$bash_tools/setup/repos.txt}\"\n    while read -r repo; do\n        \"$@\"\n    done < <(sed 's/#.*$//; s/.*://; /^[[:space:]]*$/d' \"$repolist\")\n}\n\ngithub_authors(){\n    # deferring expansion into loop\n    # shellcheck disable=SC2016\n    foreachrepo 'echo \"repo: $repo\"; pushd \"$github/$repo\" >/dev/null || return 1; git_authors; popd >/dev/null || return 1; echo' | ${less:-less}\n}\n\nmerge_conflicting_files(){\n    # merge conflicts:\n    #\n    # UU = both updated\n    # AA = both added\n    #\n    git status --porcelain | awk '/^UU|^AA/{$1=\"\"; print}'\n}\n\nmerge_deleted_files(){\n    git status --porcelain | awk '/^DU/{$1=\"\"; print}'\n}\n\n# useful for Dockerfiles merging lots of branches\n#\n# while ! make mergemasterpull; do fixmerge \"merged master\"; done\n#\nfixmerge(){\n    local msg=\"${*:-merged}\"\n    local merge_conflicted_files\n    local merge_deleted_files\n    merge_deleted_files=\"$(merge_deleted_files)\"\n    if [ -n \"$merge_deleted_files\" ]; then\n        # false positive, not passing add function/alias add to git\n        # shellcheck disable=SC2033\n        xargs git add <<< \"$merge_deleted_files\"\n    fi\n    merge_conflicted_files=\"$(merge_conflicting_files)\"\n    if [ -n \"$merge_conflicted_files\" ]; then\n        # shellcheck disable=SC2086\n        \"$EDITOR\" $merge_conflicted_files &&\n        git add $merge_conflicted_files\n    fi\n    git ci -m \"$msg\"\n}\n\nbuildkite_browse(){\n    if [ -z \"${BUILDKITE_ORGANIZATION:-}\" ]; then\n        echo \"\\$BUILDKITE_ORGANIZATION not set\"\n        return 1\n    fi\n    local repo\n    repo=\"$(git_repo | tr '[:upper:]' '[:lower:]')\"\n    browser \"https://buildkite.com/$BUILDKITE_ORGANIZATION/$repo\"\n}\n# bk is used by buildkite cli now\nalias bkb=buildkite_browse\n"
  },
  {
    "path": ".bash.d/golang.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2015 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  G o l a n g\n# ============================================================================ #\n# Golang\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\ngithub=\"${github:-$HOME/github}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/os_detection.sh\"\n\n#export GOPATH=\"$github/go-tools\"\nexport GOPATH=\"$HOME/go\"\nalias gopath='cd \"$GOPATH\"'\nalias gogo='gopath'\nalias cdgo='gopath'\nalias gosrc='cd \"$GOPATH/src\"'\nalias gobin='cd \"$GOPATH/bin\"'\n\nalias go-tools='cd \"$github/go-tools\"; export GOPATH=\"$github/go-tools\"'\nalias gtools=go-tools\nalias gt=gtools\n\n# already added in paths.sh GitHub section\n#add_PATH \"$github/go-tools\"\n\nadd_PATH \"$github/go-tools/bin\"\n\nif [ -d ~/go/bin ]; then\n    add_PATH ~/go/bin\nfi\n\n# manual installation of 1.5 mismatches with HomeBrew 1.6 installed to $PATH and\n#export GOROOT=\"/usr/local/go\"\n# causes:\n# imports runtime/internal/sys: cannot find package \"runtime/internal/sys\" in any of:\n# /usr/local/go/src/runtime/internal/sys (from $GOROOT)\n# /Users/hari/github/go-tools/src/runtime/internal/sys (from $GOPATH)\nif type -P go &>/dev/null; then\n    if is_mac; then\n        GOROOT=\"$(dirname \"$(dirname \"$(greadlink -f \"$(type -P go)\")\")\")\"\n    else\n        GOROOT=\"$(dirname \"$(dirname \"$(readlink -f \"$(type -P go)\")\")\")\"\n    fi\n    export GOROOT\n    add_PATH \"$GOROOT/bin\"\n    add_PATH \"$GOROOT/libexec/bin\"\n    add_PATH \"$GOPATH/bin\"\nfi\n\nif type -P colorgo &>/dev/null; then\n    alias go=colorgo\nfi\n\nalias lsgobin='ls -d ~/go/bin/* \"$GOROOT\"/{bin,libexec/bin}/* \"$GOPATH/bin/\"* 2>/dev/null'\nalias llgobin='ls -ld ~/go/bin/* \"$GOROOT\"/{bin,libexec/bin}/* \"$GOPATH/bin/\"* 2>/dev/null'\n"
  },
  {
    "path": ".bash.d/gpg-agent.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               G P G   A g e n t\n# ============================================================================ #\n\n# Pinentry is important, gpg-agent won't work without it.\n# pinentry intercepts and stores passphrase.\n\ngpg_agent(){\n    if [ $UID != 0 ]; then\n        if type -P gpg-agent &>/dev/null; then\n            # looks like gpg-agent not longer outputs the pid to stdout to capture\n            #GPG_ENV_FILE=~/.gpg-agent.env\n            #if [ -f \"$GPG_ENV_FILE\" ]; then\n                # shellcheck disable=SC1090,SC1091\n                #. \"$GPG_ENV_FILE\" > /dev/null\n\n                #GPG_AGENT_PID=\"${GPG_AGENT_INFO#*:}\"\n                #GPG_AGENT_PID=\"${GPG_AGENT_PID%:*}\"\n                #if ! kill -0 \"$GPG_AGENT_PID\" > /dev/null 2>&1; then\n                #    echo \"Stale gpg-agent found. Spawning new agent...\"\n                #    killall -9 gpg-agent\n                #    eval \"$(gpg-agent --daemon | tee \"$GPG_ENV_FILE\")\"\n                #elif [ \"$(ps -p \"$GPG_AGENT_PID\" -o comm=)\" != \"gpg-agent\" ]; then\n                #    echo \"gpg-agent PID does not belong to gpg-agent, spawning new agent...\"\n                #    eval \"$(gpg-agent --daemon | tee \"$GPG_ENV_FILE\")\"\n                #fi\n            if type -P pgrep &>/dev/null; then\n                if ! pgrep -qf gpg-agent.*--daemon; then\n                    echo \"Starting gpg-agent...\"\n                    killall -9 gpg-agent\n                    #eval \"$(gpg-agent --daemon | tee \"$GPG_ENV_FILE\")\"\n                    gpg-agent --daemon\n                fi\n            fi\n            #clear\n        fi\n    fi\n}\n# don't really use this any more anyway so don't bother starting it\n#gpg_agent\n"
  },
  {
    "path": ".bash.d/grype.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:58:03 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   G r y p e\n# ============================================================================ #\n\n#set -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n#eval \"$(grype completion bash)\"\n\n# generates auto-completion file to avoid repeating running auto-completion command, and sources from there\n#autocomplete grype\n"
  },
  {
    "path": ".bash.d/hadoop.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2009+ (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        H a d o o p   E c o s y s t e m\n# ============================================================================ #\n\n#srcdir=\"${srcdir:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$srcdir/.bash.d/paths.sh\"\n\n# ============================================================================ #\n#                                    E n v s\n# ============================================================================ #\n\n## ln -s -- /usr/local/hadoop-x.y.z /usr/local/hadoop\n## ln -s -- /usr/local/hbase-x.y.z /usr/local/hadoop\n## ln -s -- /usr/local/zookeeper-x.y.z /usr/local/zookeeper\n#\n# #find /usr/local -type d -name 'hadoop-*' -o -type d -name 'hbase-*' -o -type d -name 'zookeeper-*' -maxdepth 1 | while read path; do sudo ln -vfsh \"$path\" \"${path%%-*}\"; done\n# link_latest '/usr/local/hadoop-*' '/usr/local/hbase-*' '/usr/local/pig-*' '/usr/local/zookeeper-*'\n# chown -R hari -- /usr/local/{hadoop,hbase,zookeeper}\n# re-enabled HADOOP_HOME for Kite SDK\n\n#export HADOOP_HOME=\"/usr/local/hadoop\"    # Deprecated. Annoying error msgs\n#export HADOOP_PREFIX=\"/usr/local/hadoop\"  # Hate this\n## For OSX\n#export HADOOP_OPTS=\"$HADOOP_OPTS -Djava.security.krb5.realm= -Djava.security.krb5.kdc=\"\n#export HBASE_OPTS=\"  $HBASE_OPTS -Djava.security.krb5.realm= -Djava.security.krb5.kdc=\"\n#export HBASE_HOME=/usr/local/hbase\n#export PIG_HOME=/usr/local/pig\n#export ZOOKEEPER_HOME=/usr/local/zookeeper\n#add_PATH \"$HADOOP_PREFIX/bin\"\n#add_PATH \"$HBASE_HOME/bin\"\n#add_PATH \"$PIG_HOME/bin\"\n#add_PATH \"$ZOOKEEPER_HOME/bin\"\n\n#export MAHOUT_HOME=/usr/local/mahout\n## indicates to run locally instead of on Hadoop\n#export MAHOUT_LOCAL=true\n#add_PATH \"$MAHOUT_HOME/bin\"\n\n# ============================================================================ #\n#                                     C L I\n# ============================================================================ #\n\n# Hadoop CLI usability is weak so some conveniences for day to day\n\nalias dfs='hdfs dfs'\nalias dfsls='hdfs dfs -ls'\n\nalias yarnapp='yarn application'\n\nalias impala='impala_shell.sh'\n\n# nobody should use hive 1 cli any more, remap it to HS2 beeline\nalias hive='beeline.sh'\nalias hivezk='beeline_zk.sh'\n"
  },
  {
    "path": ".bash.d/intellij.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: 2023-07-26 00:10:06 +0100\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                I n t e l l i J\n# ============================================================================ #\n\n# requires git.sh for git_root() function\n\n# so that you can open files in IntelliJ from the command line on Mac like so:\n#\n#   idea <filename>\n#\nif [ -d \"/Applications/IntelliJ IDEA CE.app/Contents/MacOS\" ]; then\n    add_PATH \"/Applications/IntelliJ IDEA CE.app/Contents/MacOS\"\nfi\n\nalias i='idea'\n\n# wrote find_lock.sh to try to find if IntelliJ uses a lock file in the git project\n# whether it's open or not but it found not file existence changes at all\n#\n# after various experimentation, cannot find a reliable indicator via either process or lockfile\n# or any .idea/ file change if IntelliJ has a project dir open or not, surprising\n#\n#is_intellij_project_open(){\n#    local path=\"$1\"\n#    if [ -f \"$path\" ]; then\n#        path=\"$(dirname \"$path\")\"\n#    fi\n#    local git_root\n#    git_root=\"$(git_root \"$path\")\"\n#    if [ -z \"$git_root\" ]; then\n#        echo \"Given path is not in a git checkout!!\" >&2\n#        return 1\n#    fi\n#    local pid\n#    pid=\"$(lsof -n -c java | awk \"/${git_root//\\//\\\\/}/ {print \\$2}\" | sort -u)\"\n#    if [ -z \"$pid\" ]; then\n#        return 1\n#    fi\n#    if ps -ef | awk \"\\$2 == $pid { print }\" | grep -q IntelliJ; then\n#        return 0\n#    fi\n#    return 1\n#}\n#\n#open_intellij_project_if_not_already(){\n#    local path=\"$1\"\n#    local dir\n#    if [ -e \"$path\" ]; then\n#        if ! is_intellij_project_open \"$path\"; then\n#            if [ -f \"$path\" ]; then\n#                dir=\"$(dirname \"$path\")\"\n#            else\n#                dir=\"$path\"\n#            fi\n#            idea_bg_disown \"$dir\"\n#            sleep 1\n#        fi\n#    fi\n#}\n\nidea_bg_disown(){\n    nohup command idea \"$@\" &\n    # disowns the first backgrounded command instead of the latest command,\n    # so use $! to specify the pid of the latest command in this shell\n    disown $!\n}\n\n# so that you can quickly open files without them holding your terminal open and spewing Java logs all over your screen\nidea(){\n    local dir\n    for arg in \"$@\"; do\n        # because otherwise README Markdown Preview will not render images with relative paths to images inside project:\n        #\n        #   https://github.com/HariSekhon/Knowledge-Base/blob/69bb8d4220596e90e6c0e61c48dd8e1b9ffdf720/intellij.md#markdown-images-with-relative-paths-not-displaying-in-preview\n        #\n        # can't find any reliable method for this function, see comment just above function itself for more details\n        #open_intellij_project_if_not_already \"$arg\"\n        # XXX: caveat here - in order to not eat CLI args, we open all args after this loop,\n        #      which means multiple markdown files in one command will open in the last project\n        #      This will still result in broken markdown preview for any markdown files that are outside\n        #      the last project directory which will be ithe foreground window\n        if [[ \"$arg\" =~ \\.md$ ]]; then\n            dir=\"$(git_root \"$arg\" || :)\"\n            if [ -n \"$dir\" ]; then\n                idea_bg_disown \"$dir\"\n                # give time to settle otherwise race condition of immediate idea_bg_own() call will open the file in the other existing project\n                sleep 1\n            fi\n        fi\n    done\n    idea_bg_disown \"$@\"\n}\n\nidearoot(){\n    idea \"$(git_root)\"\n}\nalias idear=idearoot\n\n# if a file does not already exist then IntelliJ opens it in a new light IDE instead of in the current project\ntouch_idea(){\n    touch \"$@\"\n    idea \"$@\"\n}\nalias tidea=\"touch_idea\"\n"
  },
  {
    "path": ".bash.d/java.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Sun Sep 9 21:20:49 2012 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    J a v a\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\nadd_PATH CLASSPATH ~/bin/java\n\n# turn off those annoying Java 11 warnings when using Groovy scripting\nexport GROOVY_TURN_OFF_JAVA_WARNINGS=true\n\n#alias rmclass='rm -fv *.class'\nalias rmclass='find . -type f -name \"*.class\" -exec rm -fv {} \\;'\n\nif is_mac; then\n    mac_export_java_home(){\n        local version=\"$1\"\n        local args=()\n        local java_home\n        local java_library_base=\"/Library/Java/JavaVirtualMachines\"\n        local java_home_variable=\"JAVA_HOME\"\n        # for cross compiling to be found by gradle build\n        if [ -n \"$version\" ]; then\n            args+=(-v \"1.$version\")\n            java_home_variable=\"JAVA${version}_HOME\"\n        fi\n        if [ -x /usr/libexec/java_home ]; then\n            # want arg splutting\n            java_home=\"$(/usr/libexec/java_home \"${args[@]}\" 2>/dev/null)\"\n            # $? is fine here thanks shellcheck\n            # shellcheck disable=SC2181\n            if [ $? -eq 0 ] && [ -d \"$java_home\" ]; then\n                export \"$java_home_variable\"=\"$java_home\"\n                if [ -n \"$DEBUG\" ]; then\n                    echo \"Determined $java_home_variable from /usr/libexec/java_home to be '$java_home', update ~/.bashrc to optimize by setting this explicitly\" >&2\n                fi\n            fi\n        else\n            ## java_home=/Library/Java/JavaVirtualMachines/jdk1.7.0_45.jdk/Contents/Home\n            ## JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home\n            ## JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/Current/Home\n            ## JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home\n            java_home=\"$(find \"$java_library_base/\"*1.\"$version\"* -type d -name 'Home*' 2>/dev/null | tail -n1)\"\n            if [ -d \"$java_home\" ]; then\n                export \"$java_home_variable\"=\"$java_home\"\n                if [ -n \"$DEBUG\" ]; then\n                    echo \"Determined $java_home_variable from searching $java_library_base to be '$java_home', update ~/.bashrc to optimize by setting this explicitly\" >&2\n                fi\n            fi\n        fi\n    }\n    if [ -z \"$JAVA_HOME\" ]; then\n        mac_export_java_home\n        mac_export_java_home 7\n    fi\nelif is_linux; then\n    if [ -z \"$JAVA_HOME\" ]; then\n        # RHEL / CentOS\n        if type -P alternatives &>/dev/null; then\n            java_home=\"$(alternatives --list | awk '/^java[[:space:]]/{print $3; exit}' | sed 's,\\(/jre\\)\\?/bin/java$,,')\"\n            if [ -n \"$java_home\" ]; then\n                export JAVA_HOME=\"$java_home\"\n            fi\n        # Debian / Ubuntu\n        elif type -P update-alternatives &>/dev/null; then\n            java_home=\"$(update-alternatives --list java 2>/dev/null | sed 's,\\(/jre\\)\\?/bin/java$,,' | head -n1)\"\n            if [ -n \"$java_home\" ]; then\n                export JAVA_HOME=\"$java_home\"\n            fi\n        # Alpine / Other / or if all else fails\n        else\n            # prefers Sun's JDK to OpenJDK, put it higher in the testing list\n            # readlink -f => /etc/alternatives/java_sdk => /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el7_6.x86_64\n            for java_home in \\\n                    /usr/java/latest    \\\n                    /usr/java/latest/jre \\\n                    /usr/lib/jvm/java \\\n                    /usr/lib/jvm/java-openjdk \\\n                    /usr/lib/jvm/jre-openjdk \\\n                    /usr/lib/jvm/jre \\\n                    /usr/lib/jvm/default-jvm \\\n                    ; do       # default-jvm is on Alpine\n                if   [ -x \"$java_home/bin/java\" ]; then\n                    export JAVA_HOME=\"$java_home\"\n                    break\n                fi\n            done\n            if [ -z \"$JAVA_HOME\" ]; then\n                if [ -n \"$DEBUG\" ]; then\n                    echo \"WARNING: failed to find JAVA_HOME\" >&2\n                fi\n                # last ditch effort, this will work with warnings\n                if [ -x /usr/bin/java ]; then\n                    export JAVA_HOME=/usr\n                fi\n            fi\n        fi\n    fi\nfi\n\n# haven't used this in many years\n#j(){\n#    for x in \"$@\"; do\n#        echo \"javac $x\" &&\n#        javac \"$x\"      &&\n#        echo \"java ${x%.java} $x\"  &&\n#        java \"${x%.java}\" \"$x\"\n#    done\n#}\n\nif ! type sdk &>/dev/null && [ -s ~/.sdkman/bin/sdkman-init.sh ]; then\n    # shellcheck disable=SC1090,SC1091\n    source ~/.sdkman/bin/sdkman-init.sh\nfi\n"
  },
  {
    "path": ".bash.d/jenkins.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-22 17:18:09 +0000 (Mon, 22 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 J e n k i n s\n# ============================================================================ #\n\n#alias jenkins_cli='java -jar ~/jenkins-cli.jar -s http://jenkins:8080'\nalias jenkins-cli='jenkins_cli.sh'\n\n#alias backup_jenkins=\"rsync -av root@jenkins:/jenkins_backup/*.zip '~/jenkins_backup/'\"\n\n# sets Jenkins URL to the local docker and finds and loads the current container's superuser token to the environment for immediate use with jenkins_api.sh\njenkins_local(){\n    JENKINS_SUPERUSER_PASSWORD=\"$(\n        docker-compose -p bash-tools -f \"$(dirname \"${BASH_SOURCE[0]}\")/../docker-compose/jenkins.yml\" \\\n            exec -T jenkins-server cat /var/jenkins_home/secrets/initialAdminPassword </dev/null\n            # use terminal or gets carriage return and would have to tr -d '\\r'\n    )\"\n\n    export JENKINS_USER=admin\n    export JENKINS_PASSWORD=\"$JENKINS_SUPERUSER_PASSWORD\"\n    export JENKINS_TOKEN=\"$JENKINS_PASSWORD\"\n    export JENKINS_API_TOKEN=\"$JENKINS_PASSWORD\"\n    export JENKINS_USER_ID=\"$JENKINS_USER\"\n    export JENKINS_URL=\"http://localhost:8080\"\n}\n\nalias jenkinspass=\"jenkins_password.sh | copy_to_clipboard.sh\"\n"
  },
  {
    "path": ".bash.d/k3d.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-03 20:30:17 +0100 (Mon, 03 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     K 3 d\n# ============================================================================ #\n\n#bash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\nautocomplete k3d\n"
  },
  {
    "path": ".bash.d/kafka.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-28 14:46:37 +0100 (Sun, 28 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   K a f k a\n# ============================================================================ #\n\nsrcdir=\"${srcdir:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$srcdir/.bash.d/paths.sh\"\n\nfor directory in \\\n    /usr/hdp/current/kafka-broker \\\n    /usr/local/kafka \\\n    ; do\n    if [ -d \"$directory\" ]; then\n        #export KAFKA_HOME=/usr/local/kafka\n        export KAFKA_HOME=/usr/hdp/current/kafka-broker\n        add_PATH \"$KAFKA_HOME/bin\"\n        break\n    fi\ndone\n\n# kafka wrapper scripts use underscores instead of dashes so do not conflict with above kafka scripts which appear first in $PATH\n# moved to top level now\n#kafka_wrappers=\"$(dirname \"${BASH_SOURCE[0]}\")/../kafka_wrappers\"\n#add_PATH \"$kafka_wrappers\"\n\n# HDP defaults to 8GB, on VMs that often breaks cli commands which try to claim too much ram and fail\nexport KAFKA_OPTS=\"${KAFKA_OPTS:-} -Xms1G -Xmx1G\"\n\n# there was another setting like KAFKA_KERBEROS_CLIENT I've used before but can't remember, this should work too\n#kafka_cli_jaas_conf=\"$(dirname \"${BASH_SOURCE[0]}\")/../kafka_wrappers/kafka_cli_jaas.conf\"\nkafka_cli_jaas_conf=\"$(dirname \"${BASH_SOURCE[0]}\")/../kafka_cli_jaas.conf\"\nexport KAFKA_OPTS=\"${KAFKA_OPTS:-} -Djava.security.auth.login.config=$kafka_cli_jaas_conf\"\n\n# ============================================================================ #\n\n# XXX: Enable KAFKA_BROKERS and KAFKA_ZOOKEEPER for convenience\n#      of not having to specify them each time when using kafka_wrapper/ commands\n\n# XXX: Must use FQDNs to match Kerberos service principals\n\n# Apache / Cloudera\n#export KAFKA_BROKERS=\"$(hostname -f):9092\"\n\n# Hortonworks\n#export KAFKA_BROKERS=\"$(hostname -f):6667\"\n\n#export KAFKA_ZOOKEEPERS=\"$(hostname -f):2181\"\n\n# optional - use if chrooting in zookeeper\n#export KAFKA_ZOOKEEPER_ROOT=/kafka\n\n# ============================================================================ #\n\nbootstrap_server=\"\"\n\nif [ -n \"${KAFKA_BROKERS:-}\" ] &&\n   ! [[ \"$*\" =~ --bootstrap-server ]]; then\n    # shellcheck disable=SC2034\n    bootstrap_server=\"--bootstrap-server $KAFKA_BROKERS\"\nfi\n\nbroker_list=\"\"\nif [ -n \"${KAFKA_BROKERS:-}\" ] &&\n   ! [[ \"$*\" =~ --broker-list ]]; then\n    # shellcheck disable=SC2034\n    broker_list=\"--broker-list $KAFKA_BROKERS\"\nfi\n\nkafka_zookeeper=\"\"\nif [ -n \"${KAFKA_ZOOKEEPERS:-}\" ] &&\n   ! [[ \"$*\" =~ --zookeeper ]]; then\n    # shellcheck disable=SC2034\n    kafka_zookeeper=\"--zookeeper $KAFKA_ZOOKEEPERS\"\nfi\n"
  },
  {
    "path": ".bash.d/kubernetes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-28 14:56:41 +0100 (Sun, 28 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                  K u b e r n e t e s   /   O p e n S h i f t\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\nfor x in kubectl oc helm flux; do\n    autocomplete \"$x\"\ndone\n\n# minishift oc-env > ~/.minishift.env\nif [ -f ~/.minishift.env ]; then\n    # remove .minishift.env if it causes errors, which can happen if it was generated when there was no MiniShift VM running\n    # shellcheck disable=SC1090,SC1091\n    . ~/.minishift.env || rm -f -- ~/.minishift.env\nfi\n\n#if [ -f \"/usr/local/opt/kube-ps1/share/kube-ps1.sh\" ]; then\n#    . \"/usr/local/opt/kube-ps1/share/kube-ps1.sh\"\n#    # overriden in prompt.sh which is evaluated later so this is sourced there\n#    #PS1='$(kube_ps1)'\" $PS1\"\n#fi\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nadd_PATH \"${KREW_ROOT:-$HOME/.krew/bin}\"\n\nfor x in \"$bash_tools\"/kubernetes*.sh; do\n    x=\"${x##*/}\"\n    name=\"${x#kubernetes_}\"\n    eval \"k8s_${name}(){\n        '$x' \\\"\\$@\\\"\n    }\"\ndone\n\n# ============================================================================ #\n\n# replaced by function further down\n#alias k=kubectl\n# still need this autocomplete\ncomplete -F __start_kubectl k\n\n# 'k8s-app' label is set by dashboard creation but who uses that\n# false positive, the comma doesn't separate args\n# shellcheck disable=SC2054\nk8s_get_pod_opts=(-o wide -L app,env --show-labels)\n\n#alias po='k get po \"${k8s_get_pod_opts[@]}\"'\nalias po='k get po'\nalias pow='po -o wide'\nalias powc='poc -o wide'\nalias pocw='poc -o wide'\nalias kapply='k apply -f'\nalias kapp=kapply\nalias kget='k get'\nalias kedit='k edit'\nalias kdel='k delete'\nalias kdelf='kdel -f'\nalias wp=watchpods\nalias kd=kdesc\nalias ke=kubectl_exec.sh\nalias kexec=kubectl_exec.sh\nalias ke2=kubectl_exec2.sh\nalias keg=kubectl_exec_grep.sh\nalias kg='k get'\nalias ka='k apply'\nalias kaf='ka -f'\nalias kl='k logs'\nalias ktp='k top po'\nalias kshell='kube-shell'\nalias kubesh='kube-shell'\nalias kubeconfig='$EDITOR \"${KUBECONFIG:-~/.kube/config}\"'\nalias kubeconf=kubeconfig\n\n#alias use=\"k config use-context\"\nalias contexts=\"k config get-contexts\"\n#alias context=\"k config current-context\"\ncontext(){ k config current-context; }\n#alias con=context\nalias cons=contexts\n# contexts has this info and is more useful\n#alias clusters=\"k config get-clusters\"\n\n# scripts in kubernetes/ directory that should be added to \\$PATH (done automatically by sourcing this repo's .bashrc)\nalias kbusybox=\"kubectl_busybox.sh\"\nalias kalpine=\"kubectl_alpine.sh\"\nalias kcurl=\"kubectl_curl.sh\"\nalias kdns=\"kubectl_dnsutils.sh\"\n\nkube_config_isolate(){\n    local tmpdir=\"/tmp/.kube\"\n\n    mkdir -pv \"$tmpdir\"\n\n    local default_kubeconfig=\"${HOME:-$(cd ~ && pwd)}/.kube/config\"\n    local original_kubeconfig=\"${KUBECONFIG:-$default_kubeconfig}\"\n\n    # reload safety - do not source from new tmpdir - not necessary for direnv but useful for local sourcing tests\n    #if [[ \"$original_kubeconfig\" =~ $tmpdir ]]; then\n    #    echo \"ignoring \\$KUBECONFIG=$original_kubeconfig, using default home location $default_kubeconfig\"\n    #    original_kubeconfig=\"$default_kubeconfig\"\n    #fi\n\n    # isolate the kubernetes context to avoid a race condition affecting any other shells or scripts\n    # epoch is added because $$ and $PPID are direnv sub-processes and may be reused later, so using epoch to add uniqueness\n    local epoch\n    epoch=\"$(date +%s)\"\n    export KUBECONFIG=\"$tmpdir/config.${EUID:-${UID:-$(id -u)}}.$$.$epoch\"\n\n    # load your real kube config to isolated staging area to source the context info\n    if [ -f \"$original_kubeconfig\" ]; then\n        cp -v -- \"$original_kubeconfig\" \"$KUBECONFIG\"\n    elif [ -f \"$default_kubeconfig\" ]; then\n        cp -v -- \"$default_kubeconfig\" \"$KUBECONFIG\"\n    elif [ -f \"$PWD/.kube/config\" ]; then\n        cp -v -- \"$PWD/.kube/config\" \"$KUBECONFIG\"\n    elif [ -f \"/etc/rancher/k3s/k3s.yaml\" ]; then\n        cp -v -- \"/etc/rancher/k3s/k3s.yaml\" \"$KUBECONFIG\"\n    else\n        echo \"WARNING: failed to find one of:\n\n        $original_kubeconfig\n        $default_kubeconfig\n        $PWD/.kube/config\n        /etc/rancher/k3s/k3s.yaml\n    \" >&2\n    fi\n}\n\n# false positive, not using positional parameters\n# shellcheck disable=SC2142\nalias namespace='k config get-contexts | grep -F \"$(kubectl config current-context)\" | awk \"{print \\$5}\"'\nalias kwhere=\"{ echo -n 'context: '; context; echo -n 'namespace: '; namespace; }\"\nalias con='kwhere'\n\n#alias kcd='k config set-context \"$(kubectl config current-context)\" --namespace'\n\nalias menv='eval $(minikube docker-env)'\n\n# scripts at top level, automatically included in $PATH\nalias labels=\"kubectl_node_labels.sh\"\nalias taints=\"kubectl_node_taints.sh\"\n\nunalias kcd 2>/dev/null\nkcd(){\n    if [ $# -lt 1 ] || [ $# -gt 2 ]; then\n        echo \"usage: kcd <namespace>\" >&2\n        return 1\n    fi\n    local namespace=\"$1\"\n    echo \"Switching to namespace '$namespace'\"\n    k config set-context \"$(kubectl config current-context)\" --namespace \"$namespace\"\n}\n\nunalias use 2>/dev/null\nuse(){\n    if [ $# -lt 1 ] || [ $# -gt 2 ]; then\n        echo \"usage: use <context> [<namespace>]\" >&2\n        return 1\n    fi\n    local context=\"$1\"\n    local namespace=\"${2:-}\"\n    local contexts\n    contexts=\"$(k config get-contexts -o name)\"\n    if ! grep -Fxq \"$context\" <<< \"$contexts\"; then\n        #echo \"No matching contexts, inferring first partial match\"\n        context=\"$(grep -Em1 \"$context\" <<< \"$contexts\" || :)\"\n        if [ -z \"$context\" ]; then\n            echo \"Couldn't find any matching context name\" >&2\n            return 1\n        fi\n        #echo \"Inferred context to be '$context'\"\n    fi\n    #local args=()\n    #if [ -n \"$namespace\" ]; then\n    #    args+=(--namespace \"$namespace\")\n    #fi\n    #k config use-context \"$context\" \"${args[@]}\"\n    # less efficient, but more verbose\n    k config use-context \"$context\"\n    if [ -n \"$namespace\" ]; then\n        kcd \"$namespace\"\n    fi\n}\n\nkubectl_namespace(){\n    kubectl config get-contexts | awk '/^\\*/{print $5}'\n}\n\n#alias poc='po | grep -v Completed'\nunalias poc &>/dev/null\npoc(){\n    po \"$@\" | grep -v Completed\n}\n\n#alias dat='datree test --only-k8s-files --ignore-missing-schemas'\ndat(){\n    if [ $# -eq 0 ]; then\n        find . -type f -iname '*.y*ml' |\n        # datree doesn't handle patches well\n        grep -v patch |\n        tr '\\n' '\\0' |\n        xargs -0 datree test --only-k8s-files --ignore-missing-schemas\n    else\n        datree test --only-k8s-files --ignore-missing-schemas \"$@\"\n    fi\n}\ndatkust(){\n    datree_kustomize_all.sh . -- --enable-helm\n}\n\n# kustomize\nalias kbuild='kustomize build --enable-helm'\nalias kustomizebuilddiff='kbuild | kubectl_create_namespaces.sh; kbuild | kubectl diff -f -'\nalias kbuilddiff=kustomizebuilddiff\nalias kbuildd=kbuilddiff\nalias kbd=kbuildd\nalias kda=kustomize_diff_apply.sh\n\n# workaround for the fact that kustomize doesn't accept other filenames\nkustomize_build_file(){\n    local kustomization=\"$1\"\n    if [ -z \"$kustomization\" ]; then\n        echo \"usage: kustomize_build_file <something>-kustomization.yaml\" >&2\n        return 1\n    fi\n    # because shell completion will stop at the prefix, so allow us to just enter and have it figure out what we're doing\n    if ! [ -f \"$kustomization\" ];then\n        if [ -f \"${kustomization}kustomization.yaml\" ]; then\n            kustomization+=\"kustomization.yaml\"\n        elif [ -f \"${kustomization}-kustomization.yaml\" ]; then\n            kustomization+=\"-kustomization.yaml\"\n        elif [ -f \"${kustomization}kustomization.yml\" ]; then\n            kustomization+=\"kustomization.yml\"\n        elif [ -f \"${kustomization}-kustomization.yml\" ]; then\n            kustomization+=\"-kustomization.yml\"\n        else\n            echo \"File not found: $kustomization\" >&2\n            return 1\n        fi\n    fi\n    local prefix=\"${kustomization%kustomization.y*ml}\"\n    prefix=\"${prefix%-}\"\n    prefix=\"${prefix%_}\"\n    command cp -v -- \"$prefix\"*.yaml /tmp/ >&2\n    cd /tmp >&2 || return 1\n    echo >&2\n    command mv -v -- \"${kustomization##*/}\" kustomization.yaml >&2\n    echo >&2\n    kbuild\n    local result=$?\n    echo >&2\n    cd - >&2 || return 1\n    return $result\n}\nalias kbuildf=kustomize_build_file\nalias kbf=kbuildf\nkbfa(){\n    kbuildf \"$@\" >/dev/null || return 1\n    cd /tmp >&2 || return 1\n    kustomize_diff_apply.sh\n    cd - >&2 || return 1\n}\n\n# copies kustomization and values files while stripping their comments and filename prefixes\nkustcp(){\n    local name=\"$1\"\n    local dir=\"$2\"\n    echo \"Copying $name-kustomization.yaml to $dir/kustomization.yaml\" >&2\n    decomment \"$name-kustomization.yaml\" > \"$dir/kustomization.yaml\"\n    echo \"Copying $name-values.yaml to $dir/values.yaml\" >&2\n    decomment \"$name-values.yaml\" > \"$dir/values.yaml\"\n    echo \"Replacing values filename reference in kustomization.yaml\" >&2\n    perl -pi -e \"s/$name-values\\\\.yaml/values.yaml/\" \"$dir/kustomization.yaml\"\n    echo \"Done\" >&2\n}\n\n# ============================================================================ #\n\n# results in a blank arg which breaks kubectl command\n#kubectl_opts=(\"${KUBECTL_OPTS:-}\")\n# split KUBECTL_OPTS to array properly\nread -r -a kubectl_opts <<< \"${KUBECTL_OPTS:-}\"\n# set K8S_NAMESPACE in local .bashrc or similar files for environments where your ~/.kube/config\n# gets regenerated daily with certification authentication from a kerberos login script, which\n# resets the 'kcd bigdata' namespace change. This way you automatically send the right namespace every time\nif [ \"${K8S_NAMESPACE:-}\" ]; then\n    kubectl_opts+=(-n \"$K8S_NAMESPACE\")\nfi\n# TODO: might split this later\noc_opts=(\"${kubectl_opts[@]:-}\")\n\n# ============================================================================ #\n\n# oc() and kubectl() fix future invocations of k() to the each command if you want to explicitly switch between them\noc(){\n    export KUBERNETES_CLI=oc\n    command oc \"${oc_opts[@]}\" \"$@\"\n}\n\nkubectl(){\n    export KUBERNETES_CLI=kubectl\n    # if empty causes 'bash: kubectl_opts[@]: unbound variable', and can't use \"${kubectl_opts[@]:-}\" default because this results in a blank arg which ruins commands\n    if [ -n \"${kubectl_opts[*]:-}\" ]; then\n        command kubectl \"${kubectl_opts[@]}\" \"$@\"\n    else\n        command kubectl \"$@\"\n    fi\n}\n\nk(){\n    local opts=()\n    # more efficient than forking to check history every time\n    if [ -n \"$KUBERNETES_CLI\" ]; then\n        case \"$KUBERNETES_CLI\" in\n            kubectl)    opts+=(\"${kubectl_opts[@]}\")\n                        ;;\n                 oc)    opts+=(\"${oc_opts[@]:-}\")\n                        ;;\n                  *)    echo \"invalid command '$KUBERNETES_CLI' listed in \\$KUBERNETES_CLI (must be either 'kubectl' or 'oc' depending on whether you are using straight Kubernetes or OpenShift). Fix the variable or unset it to auto-detect when calling the k() function\"\n                        return\n                        ;;\n        esac\n        command \"$KUBERNETES_CLI\" \"${opts[@]}\" \"$@\"\n    else\n        # shellcheck disable=SC2086\n        case \"$(k8s_or_openshift)\" in\n                openshift)   command oc \"${oc_opts[@]}\" \"$@\"\n                             export KUBERNETES_CLI=oc\n                             ;;\n                    k8s|*)   command kubectl \"${kubectl_opts[@]}\" \"$@\"\n                             export KUBERNETES_CLI=kubectl\n                             ;;\n        esac\n    fi\n}\n\nkrun(){\n    local image=\"$1\"\n    local name=\"${image//\\//-}\"\n    shift\n    # sleep infinity only works on some distros\n    k run --generator=run-pod/v1 \"$name\" --image \"$image\" -ti -- /bin/sh\n}\n\n# use ../kubernetes/kubectl_exec.sh via alias instead\n#kexec(){\n#    local lines\n#    local name=\"${1//\\//-}\"\n#    if [ -z \"$name\" ]; then\n#        echo \"usage: kexec <name>\"\n#        return 1\n#    fi\n#    for ((i=0;i<100;i++)); do\n#        lines=\"$(k get po | grep -F \"$name\")\"\n#        if [ -z \"$lines\" ]; then\n#            echo \"No pods matching name $name found!\"\n#            return 1\n#        fi\n#        name=\"$(awk '$3 ~ /Running/{print $1; exit}' <<< \"$lines\")\"\n#        if [ -n \"$name\" ]; then\n#            break\n#        fi\n#        echo \"waiting for pod to start running...\"\n#        sleep 1\n#    done\n#    local cmd=(kubectl exec -ti \"$name\" \"$@\" -- /bin/sh -c 'if type bash >/dev/null 2>&1; then exec bash; else exec sh; fi')\n#    echo \"${cmd[*]}\"\n#    \"${cmd[@]}\"\n#}\n\nklog(){\n    local name=\"$1\"\n    k logs -f -n \"$name\" \"deploy/$name\"\n}\nklogs(){\n    local lines\n    local name=\"${1//\\//-}\"\n    shift || :\n    if [ -z \"$name\" ]; then\n        echo \"usage: klogs <name>\"\n        return 1\n    fi\n    for ((i=0;i<100;i++)); do\n        lines=\"$(k get po | grep -F \"$name\")\"\n        if [ -z \"$lines\" ]; then\n            echo \"No pods matching name $name found!\"\n            return 1\n        fi\n        # often want to see the logs of the last pod restart in 'Crashing' status\n        #name=\"$(awk '$3 ~ /Running/{print $1; exit}' <<< \"$lines\")\"\n        name=\"$(awk '{print $1; exit}' <<< \"$lines\")\"\n        if [ -n \"$name\" ]; then\n            break\n        fi\n        echo \"waiting for pod to start running...\"\n        sleep 1\n    done\n    echo kubectl logs \"$@\" \"\\\"$name\\\"\"\n    k logs \"$@\" \"$name\"\n}\n\nkfwd(){\n    local filter=\"$1\"\n    local port=\"$2\"\n    local hostport=\"$3\"\n    shift\n    shift\n    shift\n    # mind need splitting if it's a filter\n    # shellcheck disable=SC2086\n    kubectl port-forward $filter \"$port\" \"$hostport\" &\n    open \"http://localhost:$hostport\"\n}\n\n# looks like both of these work on OpenShift context\n#\n# 'kubectl get pods'\n#\n# 'oc get pods'\n\n# figure out if we're using k8s or openshift via most recent commands - return either 'k8s' or 'openshift'\nk8s_or_openshift(){\n    local last_k8s_cmd\n    last_k8s_cmd=\"$(\n        history |\n        grep -v history |\n        grep -Eo -e '\\<oc\\>' \\\n                 -e '\\<kubect[l]\\>' \\\n                 -e '\\<minikub[e]\\>' \\\n                 -e '\\<minishif[t]\\>' |\n        tail -n 1\n    )\"\n    case \"$last_k8s_cmd\" in\n            oc|minishift)   echo openshift\n                            # these end up in a subshell so aren't really useful, set in k() instead\n                            #export KUBERNETES_CLI=oc\n                            ;;\n        kubectl|minikube)   echo k8s\n                            #export KUBERNETES_CLI=kubectl\n                            ;;\n                       *)   echo unknown\n                            ;;\n    esac\n}\n\noc_get_pods(){\n    # shellcheck disable=SC2086\n    oc get pods \"${k8s_get_pod_opts[@]}\"\n}\n\nk8s_get_pods(){\n    # shellcheck disable=SC2086\n    k get pods \"${k8s_get_pod_opts[@]}\"\n}\n\nget_pods(){\n    #case \"$(k8s_or_openshift)\" in\n    #        openshift)   oc_get_pods\n    #                     ;;\n    #              k8s)   k8s_get_pods\n    #                     ;;\n    #                *)   k8s_get_pods\n    #                     ;;\n    #esac\n    #\n    # k8s functions now include k8s vs oc detection, no need for above or would end up double calling k8s_or_openshift\n    k8s_get_pods\n}\nexport -f get_pods\n\nget_pod(){\n    local filter=\"${1:-.*}\"\n    get_pods |\n    grep -v '^NAME[[:space:]]' |\n    grep Running |\n    awk \"/$filter/{print \\$1; exit}\"\n}\n\nwatchpods(){\n    # watch on Mac (brew installed) doesn't have -x switch and doesn't work on even 'export -f function'\n    # leave using kubectl call for now as that works on openshift too\n    watch \"\n        echo 'Context: '\n        echo\n        kubectl config current-context\n        echo\n        echo\n        echo 'Pods:'\n        echo\n        kubectl \" \"${kubectl_opts[@]}\" \" get pods \" \"${k8s_get_pod_opts[@]:-}\" \" 2>&1\n        echo\n    \"\n}\n\nkdesc(){\n    k describe \"$@\" | less\n}\n\n# kdesc pod with grep filter on name for fast describing a pod in the current or given namespace\nkdp(){\n    local filter=\"${1:-.*}\"\n    shift || :\n    pod=\"$(k get po -o name \"$@\" | grep -Em 1 \"$filter\")\" || return\n    kdesc \"$pod\" \"$@\"\n}\n\nkdelp(){\n    k delete pod \"$@\"\n}\n\n# Getting token works on stock Kubernetes but not OpenShift due to stricter defaults\n#\n# Error from server (Forbidden): secrets is forbidden: User \"developer\" cannot list secrets in the namespace \"kube-system\": no RBAC policy matched\n# error: resource name may not be empty\n#\n## even after 'oc login' as system/admin\n#\n# Error from server (Forbidden): secrets is forbidden: User \"system\" cannot list secrets in the namespace \"kube-system\": no RBAC policy matched\n# error: resource name may not be empty\n#\nk8s_get_token(){\n    kubectl describe secret -n kube-system \"$(kubectl get secrets -n kube-system | awk '/^default-token/ {print $1}')\" |\n    awk '/^token/ {print $2}'\n}\n\n# better than: kubectl config view | grep server\nk8s_get_api(){\n    local context\n    local cluster\n    context=\"$(context)\"\n    cluster=\"$(k config view -o jsonpath=\"{.contexts[?(@.name == \\\"$context\\\")].context.cluster}\")\"\n    k config view -o jsonpath=\"{.clusters[?(@.name == \\\"$cluster\\\")].cluster.server}\"\n    # or if you have jq installed:\n    # k get --raw=/api | jq -r '.serverAddressByClientCIDRs[0].serverAddress'\n    echo\n}\n\n# TODO: path like above to get the current context's cluster\nk8s_get_client_cert(){\n    awk '/^[[:space:]]*client-cert/{print $2}' ~/.kube/config | head -n 1\n}\n\nk8s_get_client_key(){\n    awk '/^[[:space:]]*client-key-data/{print $2}' ~/.kube/config | head -n 1\n}\n\nk8s_get_ca_cert(){\n    awk '/^[[:space:]]*certificate-authority-data/{print $2}' ~/.kube/config | head -n 1\n}\n\n# generates files for authenticating to kube-apiserver via curl:\n#\n# curl --cert client_cert.pem --key client_key.pem --cacert ca_cert.pem https://k8smaster:6443/api/v1/pods\n# curl --cert client_cert.pem --key client_key.pem --cacert ca_cert.pem https://k8smaster:6443/api/v1/pods/namespaces/default/pods -XPOST -H 'Content-Type: application/json' -d @pod_defintion.json\nk8s_get_keys(){\n    # use --decode not -d / -D which varies between Linux and Mac\n    k8s_get_client_cert | base64 --decode - > client_cert.pem\n    echo \"generated client_cert.pem\"\n    k8s_get_client_key | base64 --decode - > client_key.pem\n    echo \"generated client_key.pem\"\n    k8s_get_ca_cert | base64 --decode - > ca_cert.pem\n    echo \"generated ca_cert.pem\"\n}\n\n# run kubectl commands against multiple clusters\nkclusters(){\n    for context in $(kubectl config get-contexts -o=name --kubeconfig clusters.yaml); do\n        kubectl \"$@\" --kubeconfig clusters.yaml --context=\"$context\"\n    done\n}\n\n# to kubectl apply manifests to both clusters for multi-cluster deployments\nkclustersapply(){\n    kclusters apply -f \"$@\"  # eg. manifests\n}\n\n# inspired by my class 'when' functions in when.sh\nwhenpodup(){\n    local name=\"${1:-}\"\n    shift || :\n    if [ -z \"$name\" ]; then\n        echo \"usage: whenpodup <name>\"\n        return 1\n    fi\n    local count=0\n    while ! kubectl get pods \"$name\" -o 'jsonpath={.status.phase}' | grep -q 'Running'; do\n        ((count+=1))\n        timestamp \"waiting for pod '$name' to come up...\"\n        if [ $count -gt 22 ]; then\n            sleep 10\n        else\n            sleep 5\n        fi\n    done\n    timestamp \"pod '$name' is up\"\n    \"$@\"\n}\n"
  },
  {
    "path": ".bash.d/linux.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   L i n u x\n# ============================================================================ #\n\n# Linux specific bits to not include on Mac\n\n# most of the regular stuff is in the other bash.d/*.sh files\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\nis_linux || return\n\nopen(){\n    if type -P xdg-open &>/dev/null; then\n        xdg-open \"$@\"\n    elif sensible-browser &>/dev/null; then\n        sensible-browser \"$@\"\n    elif x-www-browser &>/dev/null; then\n        x-www-browser \"$@\"\n    elif gnome-open &>/dev/null; then\n        gnome-open \"$@\"\n    else\n        echo \"Neither 'xdg-open' nor 'sensible-browser' were found in \\$PATH - install one of them to automatically open this URL:\"\n        echo\n        echo \"$*\"\n        echo\n    fi\n}\n\nalias reloadXdefaults=\"xrdb ~/.Xdefaults\"\n\n#setxkbmap us\n\n# Assign middle mouse to my Alt Gr key\n# TODO:  change this to keysym as keycodes can change between keyboards, to find keymaps do\n# xmodmap -pkie\n\nif [ -n \"$DISPLAY\" ] && ! is_mac; then\n    # This caused the left to be remapped, must test and handle better\n    #xmodmap -e 'keycode 113 = Pointer_Button2'\n    #xmodmap -e 'keycode 113 = Left NoSymbol Left'\n    xkbset m\n    # Ubuntu 12.04.1 LTS had a bug where it turned off repeat keys on my down arrow, this fixed it\n    xkbset repeatkeys 116\nfi\n\nrpmqf(){\n    rpm -qf \"$(readlink -m \"$1\")\"\n}\n\nfixtime(){\n    # $sudo defined in .bashrc\n    # shellcheck disable=SC2154\n    $sudo /etc/init.d/ntp stop\n    $sudo ntpdate pool.ntp.org\n    $sudo /etc/init.d/ntp start\n}\n\ngetmounts(){\n    #grep -e \"ext\" -e \"reiser\" -e \"fat\" -e \"ntfs\" < /proc/mounts |\n    #awk '{ print $2 }'\n    awk '/ext|reiser|fat|ntfs|btrfs|xfs/{print $2}'\n}\n\nfindsuid(){\n    for x in $(getmounts); do\n        echo \"Searching $x for suid programs:\"\n        # $sudo defined in .bashrc if not root\n        # shellcheck disable=SC2154\n        $sudo find \"$x\" -xdev -type f -perm -u+s -exec ls -l {} \\;\n    done\n}\n\nfindguid(){\n    for x in $(getmounts); do\n        echo \"Searching $x for guid programs:\"\n        $sudo find \"$x\" -xdev -type f -perm -g+s -exec ls -l {} \\;\n    done\n}\n\nfindsguid(){\n    for x in $(getmounts); do\n        echo \"Searching $x for suid and guid programs:\"\n        $sudo find \"$x\" -xdev -type f \\( -perm -u+s -o -perm -g+s \\) -exec ls -l {} \\;\n    done\n}\n\nfindwritable(){\n    for x in $(getmounts); do\n        echo \"Searching $x for world writeable files:\"\n        $sudo find \"$x\" -xdev -type f -perm -o+w -exec ls -l {} \\;\n    done\n}\n\n# ==========================\n# When using Samba WinPopups on Linux in Windows workgroups - convenient back in the day but shouldn't be needed today with the plethora of better chat tools\n#\n#netsend(){ smbclient -M \"$1\" <<< \"${*:2}\"; }\n#alias ns=netsend\n#\n# clear pop-ups and alerts if sending instant security alerts\n#clearnetsend(){\n#    sudo pkill -f sambapopup\n#}\n#alias cns=clearnetsend\n#\n#clearxmessage(){\n#    while pkill xmessage; do\n#        sleep 0.1\n#    done\n#    while pkill gmessage; do\n#        sleep 0.1\n#    done\n#}\n# =========================\n"
  },
  {
    "path": ".bash.d/lolcat.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-03 11:31:02 +0000 (Sun, 03 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nlolz(){\n    exec 1> >(lolcat >&2)\n}\n"
  },
  {
    "path": ".bash.d/mac.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2011 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           A p p l e   M a c   O S X\n# ============================================================================ #\n\n# More Mac specific stuff in adjacent *.sh files, especially network.sh\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\nis_mac || return\n\nexport HOMEBREW_DISPLAY_INSTALL_TIMES=1\n#export HOMEBREW_DEBUG=1\n#export HOMEBREW_CLEANUP_MAX_AGE_DAYS=30  # default: 120\n\n# Stops Mac calling update_terminal_cwd() which causes a tonne of noise during set -x tracing\nexport INSIDE_EMACS=1\n\nexport icloud=\"$HOME/Library/Mobile Documents/com~apple~CloudDocs\"\nalias icloud=\"cd '$icloud'\"\n\nalias osash=\"osascript -i\"\nalias osashell=osash\n\nif [ -x /opt/homebrew/bin/brew ]; then\n    # shellcheck disable=SC2046\n    eval $(/opt/homebrew/bin/brew shellenv)\nfi\n\ndate(){\n    gdate \"$@\"\n}\n\nxargs(){\n    # because --no-run-if-empty is useful\n    command gxargs \"$@\"\n}\n\nif ! type tac &>/dev/null; then\n    tac(){\n        gtac \"$@\"\n    }\nfi\n\n# used for Shazaming while on headphones - see:\n#\n#   https://github.com/HariSekhon/Knowledge-Base/blob/master/audio.md#shazam-songs-while-using-headphones-on-mac\n#\n# Switches to Multi-Output Device which should already be configured as above and contain your headphones and BlackHole 2ch\n#\n#alias mshazam='SwitchAudioSource -s \"Multi-Output Device\"; open -a Shazam'\n#alias msound='SwitchAudioSource -s \"Multi-Output Device\"'\nunalias msound 2>/dev/null || :\nmsound(){\n    # because if you have 2 multi-output devices eg.\n    #\n    # - one using AirPods + Blackhole\n    # - another using Speakers + Blackhole\n    #\n    # then Mac automatically renames \"Multi-Output Device\" to \"Multi-Output Device 1\",\n    # even if the other multi-output device has already been differentiated, it refuses to let you set\n    # it back to the default name, so just determine what the first one is called and use that\n    #\n    local multi_output_device\n    multi_output_device=\"$(SwitchAudioSource -a | grep -m1 '^Multi-Output Device')\"\n    #echo \"Found first multi-output device: $multi_output_device\"\n    echo \"Using first found multi-output device\"\n    SwitchAudioSource -s \"$multi_output_device\"\n}\n\nalias restartsound='sudo killall coreaudiod'\n\nalias mshazam='msound; open -a Shazam'\n\nvol(){\n    if [ $# -ne 1 ]; then\n        echo \"usage: vol <num>\"\n        return 1\n    fi\n    osascript -e \"set volume output volume $1\"\n}\n\n# put in inputrc for readline\n#set completion-ignore-case on\n\n# Apple default in Terminal is xterm\n#export TERM=xterm\n# not sure why I set it to linux\n#export TERM=linux\n#ulimit -u 512\n\ndhcprenew(){\n    local interface=\"${1:-en0}\"\n    watch -q 1 ifconfig \"$interface\"\n    sudo scutil <<< \"add State:/Network/Interface/$interface/RefreshConfiguration temporary\"\n    watch ifconfig \"$interface\"\n}\n\ndhcpdiscover(){\n    local interface=\"${1:-en0}\"\n    watch -q 1 ifconfig \"$interface\"\n    sudo ipconfig set \"$interface\" BOOTP\n    sudo ipconfig set \"$interface\" DHCP\n    watch ifconfig \"$interface\"\n}\n\nmacsleep(){\n    sudo pmset sleepnow\n}\n\nnosleep(){\n    echo \"Running: caffeinate -s $*\"\n    echo \"(works even if you close the Macbook lid but will still sleep on battery power)\"\n    caffeinate -s \"$@\"\n}\n\nsilence_startup(){\n    sudo nvram SystemAudioVolume=%80\n}\n\ntop(){\n    local opts=(-F -R -o)\n    if [ $# -eq 1 ]; then\n        command top \"${opts[@]}\" \"$1\"\n    elif [ $# -gt 1 ]; then\n        command top \"$@\"\n    else\n        command top \"${opts[@]}\" cpu\n    fi\n}\n\nfixvbox(){\n    sudo /Library/StartupItems/VirtualBox/VirtualBox restart\n}\n\nfixaudio(){\n    sudo kextunload /System/Library/Extensions/AppleHDA.kext\n    sudo kextload   /System/Library/Extensions/AppleHDA.kext\n}\n\nshowhiddenfiles(){\n    defaults write com.apple.finder AppleShowAllFiles YES\n    # must killall Finder after this\n}\n\nalias reloadprefs='killall -u $USER cfprefsd'\nalias strace=\"dtruss -f\"\nalias usbinfo='system_profiler SPUSBDataType'\nalias vlc=\"/Applications/VLC.app/Contents/MacOS/VLC\"\n\n\n# clear paste buffer\nclpb(){\n    copy_to_clipboard.sh < /dev/null\n}\n\nmacmac(){\n    ifconfig |\n    awk '\n        /^en[[:digit:]]+:/{gsub(\":\", \"\", $1); printf \"%s:\\t\", $1}\n        /^[[:space:]]ether[[:space:]]/{print $2}\n    ' |\n    # filters to only the lines with prefixed interfaces from first match\n    grep '\\t'\n}\n\nduall(){\n    # bash_tools defined in .bashrc\n    # shellcheck disable=SC2154\n    du -ax \"$bash_tools\" | sort -k1n | tail -n 2000\n    sudo du -ax / | sort -k1n | tail -n 50\n}\nalias dua=duall\nif type -P brew &>/dev/null; then\n    brew_prefix=\"$(brew --prefix)\"\n    if [ -f \"$brew_prefix/etc/bash_completion\" ]; then\n        # shellcheck disable=SC1090,SC1091\n        . \"$brew_prefix/etc/bash_completion\"\n    fi\nfi\n\nbrewupdate(){\n    if ! brew update; then\n        echo \"remove the following to brew update\"\n        brew update 2>&1 | tee /dev/stderr | grep '^[[:space:]]*Library/Formula/' |\n        while read -r formula; do\n            echo rm -fv -- \"/usr/local/$formula\"\n        done\n        return 1\n    fi\n}\n\nbrewinstall(){\n    brewupdate &&\n    sed 's/#.*// ; /^[[:space:]]*$/d' < ~/mac-list.txt |\n    while read -r pkg; do\n        brew install \"$pkg\" #||\n            #{ echo \"FAILED\"; break; }\n    done\n}\n\nbrew_find_unlinked_bins(){\n     for x in /usr/local/Cellar/*/*/bin/*; do\n         if ! [ -f \"/usr/local/bin/${x##*/}\" ]; then\n             echo \"$x\"\n        fi\n    done\n}\n\n# don't export BROWSER on Mac, trigger python bug:\n# AttributeError: 'MacOSXOSAScript' object has no attribute 'basename'\n# from python's webbrowser library\n#export BROWSER=\"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome\"\n#export BROWSER=\"/Applications/Firefox.app/Contents/MacOS/firefox\"\n\n# MacPorts - using HomeBrew instead in recent years\n#if [ -e \"/sw/etc/bash_completion\" ]; then\n#    . /sw/etc/bash_completion\n#fi\n\n# seems Mac OS X has a native pkill now\n#pkill(){\n#    local args=\"\"\n#    local regex=\"\"\n#    local grep_args=\"\"\n#    while [ -n \"$1\" ]; do\n#        case \"$1\" in\n#            -i) grep_args=\"$grep_args -i\"\n#                shift\n#                ;;\n#            -*) args=\"$args $1\"\n#                shift\n#                ;;\n#             *) regex=\"$1\"\n#                shift\n#                ;;\n#        esac\n#    done\n#    # TODO: check this a few times and then remove the echo\n#    local proclist=$(ps -e | awk '{printf $1 OFS;for(i=4;i<=NF;i++)printf $i OFS;print\"\"}' | grep $grep_args \"$regex\")\n#    if [ -n \"$proclist\" ]; then\n#        echo \"$proclist\"\n#        awk '{print $1}' <<< \"$proclist\" | xargs echo kill $args\n#        read -r -p \"Kill all these processes? [y/N] \" answer\n#        if [ \"$answer\" = \"y\" ]; then\n#            awk '{print $1}' <<< \"$proclist\" | xargs kill $args\n#        fi\n#    fi\n#}\n"
  },
  {
    "path": ".bash.d/mercurial.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#             R e v i s i o n   C o n t r o l  -  M e r c u r i a l\n# ============================================================================ #\n\ntype -P hg &>/dev/null || return 0\n\nalias hgi=hgignore\nalias hgrc='$EDITOR ~/.hgrc'\n\n# HG doesn't record dirs and there is no .hg per subdir, rather than traverse upwards checking filesystem boundaries use hg tools themselves\nisHg(){\n    local target=\"${1:-.}\"\n    # There aren't local .hg dirs everywhere only at top level so this is difficult in bash\n    #if [ -d \"$target/.hg\" -o -d \"$(dirname \"$target\")/.hg\" ]; then\n    # shortcut for efficiency\n    if [ -d \"$target/.hg\" ] ||\n       [ -d \"${target%/*}/.hg\" ]; then # || hg parents \"$target\" &>/dev/null; then # can only call this on files not dirs anyway since Hg doesn't track dirs\n        return 0\n    #elif [ -f \"$target\" ] && hg parents \"$target\"; then\n    #    return 0\n    #elif hg status \"$target\" &>/dev/null; then # Doesn't work at all always returns 0 if a subdir of a repo and just blank even if in ignores\n        # Unfortunately the -P switch only supports a single pattern so we can't use --file\n        #grep -qP -f \"$(hg root)/.hgignore\"\n        # algorithm horribly inefficient, was going to rewrite in perl isHg.pl but see futher down\n        #while read regex; do\n        #    grep \"^[[:space:]]*$\" <<< \"$regex\" && continue\n        #    abs_path=\"$(abs_path \"$target\")\"\n        #    abs_path=\"${abs_path/$(hg root)\\/}\"\n        #    grep -qP \"$regex\" <<< \"$abs_path\" && return 1\n        #done < \"$(hg root)/.hgignore\"\n    #    return 0\n    #elif [ -d \"$target\" ]; then\n    #    echo \"WARNING: cannot call hg on a dir, $target is a dir so this returns false and will fall through\"\n    #    return 1\n    # finally found a reasonably efficient way to handle all cases\n    # trick to return False on subdirs which are not handled by Mercurial\nelif [ -n \"$(hg log --limit 1 \"$target\" 2>/dev/null)\" ]; then\n        return 0\n    else\n        return 1\n    fi\n}\n\nhgignore(){\n    #pushd \"$srcdir\" &>/dev/null\n    local hgroot\n    hgroot=\"$(hg root)\"\n    [ -n \"$hgroot\" ] || return 1\n    \"$EDITOR\" \"$hgroot/.hgignore\"\n    #popd &>/dev/null\n}\n\nhgci(){\n    local hgcimsg=\"\"\n    for x in \"$@\"; do\n        if hg st \"$x\" | grep -q \"^[?A]\"; then\n            hgcimsg+=\"$x, \"\n        fi\n    done\n    [ -z \"$hgcimsg\" ] && return 1\n    hgcimsg=\"${hgcimsg%, }\"\n    hgcimsg=\"added $hgcimsg\"\n    hg add -- \"$@\" &&\n    echo \"committing $*\"\n    hg ci -m \"$hgcimsg\" -- \"$@\"\n}\n\nhgrm(){\n    hg rm -- \"$@\" &&\n    hg ci -m \"removed $*\" -- \"$@\"\n}\n\nhgrevertrm(){\n    hg revert \"$@\"\n    rm -v -- \"$@\"\n}\n\nhgrename(){\n    hg mv -- \"$1\" \"$2\" &&\n    hg ci -m \"renamed $1 to $2\" -- \"$1\" \"$2\"\n}\n\nhgmv(){\n    hg mv -- \"$1\" \"$2\" &&\n    hg ci -m \"moved $1 to $2\" -- \"$1\" \"$2\"\n}\n\nhgl(){\n    hg log \"$@\" | less\n}\n\nhgu(){\n    [ -n \"$1\" ] || { echo \"ERROR: must supply arg\"; return 1; }\n    [ \"$(hg diff \"$@\" | wc -l)\" -gt 0 ] || return\n    hg diff -- \"$@\" | more &&\n    read -r &&\n    echo \"committing $*\" &&\n    hg ci -m \"updated $*\" -- \"$@\"\n}\n\n#hhgu(){\n#    # all playlists end in \\n from now on via paste_playlists.sh fix\n#    [ -n \"$1\" ] || { echo \"ERROR: must supply arg\"; return 1; }\n#    pushd \"$music\" >/dev/null\n#    spotify/validate_playlists.sh \"$1\" || { echo \"Playlist validation failed\"; return 1; }\n#    spotify/validate_playlist_lengths.sh \"$1\" || { echo \"Playlist dump length validation failed\"; return 1; }\n#    [ `hg st \"$1\" \"spotify/$1\" | wc -l` -gt 0 ] || { echo \"No changes in either uri or track lists\"; return 0; }\n#    local target=\"${1##*/}\"\n#    local target_tip=\"$(dirname \"$target\")/.$(basename \"$target\").tip\"\n#    hg cat \"$target\" | spotify/normalize_tracknames.pl > \"$target_tip\"\n#    cat \"$target\" | spotify/normalize_tracknames.pl > \".$target\"\n#    if [ -z \"$(diff -iwu \"$target_tip\" \".$target\")\" ]; then\n#        echo \"Noop changes only, committing...\"\n#        hg mydiff \"$target\" |\n#        #egrep '^\\+' | tee /dev/stderr |\n#        grep -v '^[+-][+-][+-]' # | sl --no-locking\n#        hg ci -m \"updated $target\" \"$target\" \"spotify/$target\"\n#        return $?\n#    elif diff -iwu \"$target_tip\" \".$target\" | grep -q '^-[^-]'; then\n#        local diffs=\"$(\n#        { hg mydiff \"$target\"\n#          hg mydiff \"spotify/$target\"\n#        })\"\n#        local removals=\"$(grep -c \"^-[^-]\" <<< \"$diffs\")\"\n#        local additions=\"$(grep -c \"^+[^+]\" <<< \"$diffs\")\"\n#        diffs=\"$(echo \"$diffs\" |\n#        egrep \"^[+-]\" |\n#        spotify/normalize_tracknames.pl |\n#        diffnet.pl -iw\n#        )\"\n#        if [ -z \"$diffs\" ]; then\n#            echo \"Noop changes to tracks, committing...\"\n#        elif ! echo \"$diffs\" | grep -q '^-[^-]'; then\n#            echo \"Net diff shows only playlist additions, committing...\"\n#            echo \"$diffs\" |\n#            more\n#        else\n#            {\n#            echo \"$additions additions $removals removals\"\n#            echo \"$diffs\"\n#            } |\n#            more &&\n#            read || return\n#        fi\n#        hg ci -m \"updated $target\" \"$target\" \"spotify/$target\"\n#    else\n#        echo \"Only playlist additions detected, committing...\"\n#        hg mydiff \"$target\" |\n#        #grep -v '^[+-][+-][+-]'\n#        egrep \"^[+-]\"\n#        hg ci -m \"updated $target\" \"$target\" \"spotify/$target\"\n#        return $?\n#    #    echo \"No additions or removals detected, playlist dump must currently be in progress\"\n#    #    return 1\n#    fi\n#    popd &>/dev/null\n#}\n\n# equiv to using the 3rd party shelve extension since HG doesn't have this Git Stash functionality\nhgshelve(){\n    local hgroot\n    hgroot=\"$(hg root)\"\n    [ -f \"$hgroot/shelve.diff\" ] &&\n        { echo \"$hgroot/shelve.diff already exists, aborting for safety to not lose changes\"; return 1; }\n    hg diff > \"$hgroot/shelve.diff\"\n    hg revert -a\n}\n\n# Then merge, hg up etc, then unshelve\n\nhgunshelve(){\n    hg import --no-commit \"$hgroot/shelve.diff\" # && rm -v \"$srcdir/shelve.diff\"\n}\n\nhgdiff(){\n    local filename=\"${1:-}\"\n    [ -n \"$filename\" ] || { echo \"usage: hgdiff filename\"; return 1; }\n    hg diff -- \"$filename\" > \"/tmp/hgdiff.tmp\"\n    diffnet.pl \"/tmp/hgdiff.tmp\"\n}\n"
  },
  {
    "path": ".bash.d/mp3.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-21 11:36:49 +0100 (Tue, 21 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# set the Track number metadata on mp3 files in the order that they are given\n# see much better mp3_set_track_order.sh at top level of this repo now\n#mp3_renumber(){\n#    local i=0\n#    for x in \"$@\"; do\n#        ((i+=1))\n#        id3v2 --track \"$i\" \"$x\"\n#    done\n#}\n\nmp3info(){\n    find \"${@:-.}\" -type f -iname '*.mp3' |\n    head -n 1 |\n    while read -r filename; do\n        mediainfo \"$filename\"\n    done\n}\n\nmp3infotail(){\n    find \"${@:-.}\" -type f -iname '*.mp3' |\n    tail -n 1 |\n    while read -r filename; do\n        mediainfo \"$filename\"\n    done\n}\n\nmp3infoheadtail(){\n    find \"${@:-.}\" -type f -iname '*.mp3' |\n    sed -n '1p;$p' |\n    while read -r filename; do\n        mediainfo \"$filename\"\n    done\n}\n\nmp3set(){\n    if [ $# != 2 ]; then\n        echo \"usage: mp3set <artist> <album>\"\n        return 1\n    fi\n    local artist=\"$1\"\n    local album=\"$2\"\n    mp3_set_artist.sh \"$artist\"\n    mp3_set_album.sh  \"$album\"\n    mp3_set_track_order.sh\n    mp3_set_track_name.sh\n}\n"
  },
  {
    "path": ".bash.d/mysql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-16 13:26:50 +0000 (Mon, 16 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   M y S Q L\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# shellcheck disable=SC1090,SC1091\ntype pass &>/dev/null ||\n. \"$bash_tools/.bash.d/functions.sh\"\n\n# highest priority env var first, common one second - export as both\nalias mysqlpass='pass MYSQL_PWD MYSQL_PASSWORD'\n"
  },
  {
    "path": ".bash.d/network.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 N e t w o r k\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\nalias 4=\"ping 4.2.2.1\"\nalias 8=\"ping 8.8.8.8\"\n\nalias ping=\"ping -n\"\nalias p=\"ping\"\n\npingwait=\"-w\"\nis_mac && pingwait=\"-W\"\n\nalias ping_google=\"while true; do ping www.google.com && sleep 1 || break; done\"\nalias g=ping_google\n\n# watch_url.pl is in DevOps-Perl-tools repo which should be in $PATH\nalias watchu=\"watch_url.pl\"\n# watch google\n# https because http often gets intercepted by routers + proxies giving false 200 OKs where there is an internet issue\nalias wg=\"watch_url.pl https://google.com\"\n\nalias speedtesturl=\"browser https://speedtest.net\"\n\n# ============================================================================ #\n#                         Y o u r   I P   A d d r e s s\n# ============================================================================ #\n\n# local/internal IP address\nmyip(){\n    ifconfig | grep 'inet[[:space:]]' | grep -v 127.0.0.1 | awk '{print $2}'\n}\n\n# public IP address\nifconfigco(){\n    curl ifconfig.co\n    # something else to consider with jq for lat/long coordinates, ASN, country etc\n    #curl ifconfig.co/json\n}\n\nipinfo(){\n    # returns json without /ip with region, reverse dns hostname, city, region, country, lat/long coordinates, org, postcode, timezone\n    curl ipinfo.io/ip\n}\n\nipify(){\n    curl http://api.ipify.org/\n    echo\n}\n\n# doesn't welcome automation / curl - requires captchas now so obsolete\n#whatismyip(){\n#    #lynx -dump $(lynx -dump www.whatismyip.com | tail -n 1)\n#    lynx -useragent=\"Mozilla\" -dump www.whatismyip.com 2>/dev/null | awk '/Your Public IPv[46] is:/ {print $6}'\n#}\n\n# ============================================================================ #\n#                               F u n c t i o n s\n# ============================================================================ #\n\ncheckhost(){\n    if [ -z \"$1\" ]; then\n        echo \"usage: checkhost hostname/ip\"\n        return 1\n    fi\n    if grep -qi \"unknown host\" <<< \"$(ping -c 1 \"$pingwait\" 1 \"$1\" 2>&1)\"; then\n        echo \"Unknown host\"\n        return 1\n    fi\n}\n\nn(){\n    if type -P host &>/dev/null; then\n        host \"$@\"\n    elif type -P nslookup &>/dev/null; then\n        nslookup \"$@\"\n    else\n        echo \"neither host nor nslookup were found in the path\"\n        return 1\n    fi\n}\nalias h=n\n\ngetip(){\n    host \"$@\" | grep \"has address\" | awk '{print $4; exit}'\n}\n\ntping(){\n    while true; do\n        #echo -n \"`date '+%F %T'`   \"\n        local output\n        output=\"$(ping -c 1 \"$pingwait\" 2 \"$@\" |\n                  grep -v -e statistics \\\n                          -e \"transmitted\" \\\n                          -e \"rtt min/avg/max/mdev\" \\\n                          -e \"bytes of data\" \\\n                          -e \"^[[:space:]]*$\" \\\n                          -e \"^PING \" \\\n                          -e \"round-trip\"\n                 )\"\n        echo \"$(date '+%F %T')   ${output:-no response from $1}\"\n        sleep 1\n    done\n}\ntpinggw(){\n    tping \"$(get_gw)\" \"$@\"\n}\n\n# for trying to find those damn wifi capture portals that disappear but block your internet http proxying\nopengw(){\n    local gateway\n    gateway=\"$(get_gw)\"\n    open \"http://$gateway\"\n    open \"https://$gateway\"\n}\n\nport(){\n    if [ -z \"$2\" ]; then\n        echo \"You must supply a hostname/ip address to test followed by a port number\"\n        return 1\n    fi\n    #sudo nmap $1 -p $2 ${@:3} -P0 | grep tcp\n    nc -zv \"$1\" \"$2\" 2>&1 | grep -v \"DNS fwd/rev misma\" | sed 's/[^]]*\\] //'\n}\n\ntestport(){\n    if [ -z \"$2\" ]; then\n        echo \"You must supply a hostname/ip address to test followed by a port number\"\n        return 1\n    fi\n    while true; do\n        timestampcmd port \"$1\" \"$2\"\n        sleep 1\n    done\n}\n\nhammerport(){\n    for i in {1..500}; do\n        printf \"%-3s: \" \"$i\"\n        nc -z -v \"$1\" \"$2\"\n    done\n}\n\nhalfopen(){\n    while true; do\n        echo -n \"half-open connections: \"\n        netstat -ant | grep c SYN_RECV\n        sleep 1\n    done\n}\n\nget_gw(){\n    local gw\n    #gw=\"$(netstat -rn | awk '/^default.*\\./ {print $2;exit}')\"\n    gw=\"$(\n        netstat -rn |\n        awk '\n            /^default.*\\./ { print $2; exit }\n            $1 == \"Internet:\" { inet = 1; next }\n            $1 == \"Internet6:\" { inet = 0 }\n            inet && ($1 == \"default\" || $1 == \"0.0.0.0\") && $2 ~ /^[0-9.]+$/ {\n                print $2\n                exit\n            }\n        '\n    )\"\n    if [ -z \"$gw\" ]; then\n        echo \"Could not find gateway, no default route! \" >&2\n        return 1\n    fi\n    echo \"$gw\"\n}\n\ngw(){\n    local gw\n    gw=\"$(get_gw)\"\n    [ -n \"$gw\" ] || return 1\n    ping \"$gw\"\n}\n\n\nz(){\n    local gw\n    gw=\"$(get_gw)\"\n    if [ -n \"$gw\" ]; then\n        whenup \"$gw\" &&\n        whenup 4.2.2.1 &&\n        whenup www.google.com echo \"INTERNET OK\"\n    else\n        echo \"Couldn't find gateway, cannot test upstream connectivity!\"\n        return 1\n    fi\n}\n\nbrowser(){\n    if [ -n \"${BROWSER:-}\" ]; then\n        \"$BROWSER\" \"$@\"\n    elif is_mac; then\n        open \"${*:-http://google.com}\"\n    else\n        echo \"\\$BROWSER environment variable not set and not on Mac OSX, not sure which browser to use, aborting...\"\n        return 1\n    fi\n}\n\nbrowse(){\n    if isGit . &>/dev/null && git remote -v | grep -qi http; then\n        gitbrowse \"$@\"\n    else\n        browser \"$@\"\n    fi\n}\n\ndownorjustme(){\n    browser \"http://www.downforeveryoneorjustme.com/$1\"\n}\n\n# directs to the same as downorjustme\nisupme(){\n    browser \"http://www.isup.me/$1\"\n}\n\nchrome(){\n    if is_mac; then\n        # opens in most recent Chrome window\n        # could use one of these: --new --args --incognito --new-window\n        open -a 'Google Chrome' \"${*:-http://www.google.com}\"\n    else\n        checkprog google-chrome || return 1\n        google-chrome \"${*:-http://www.google.com}\" &\n    fi\n\n}\n\nff(){\n    if is_mac; then\n        open -a 'Firefox' \"http://${*:-www.google.com}\"\n    else\n        checkprog firefox || return 1\n        firefox \"${*:-http://www.google.com}\" &\n    fi\n}\n\ngg(){\n    if [ -z \"$*\" ]; then\n        browser &\n    else\n        searchterm=\"${*// /%20}\"\n        browser \"http://www.google.com/search?q=$searchterm\" &\n    fi\n}\n\nnetcraft(){\n    checkprog firefox || return 1\n    browser \"http://uptime.netcraft.com/up/graph?site=$*\" &\n}\n\nwikipedia(){\n    checkprog \"firefox\" || return 1\n    local searchterm\n    searchterm=\"${*// /%20}\"\n    browser \"http://en.wikipedia.org?search=$searchterm&go=Go\" &\n}\nalias wiki=wikipedia\n\ndefinition(){\n    checkprog \"firefox\" || return 1\n    local searchterm\n    searchterm=\"${*// /%20}\"\n    # hl=en&q=test&btnI=I%27m+Feeling+Lucky&meta=&aq=f\n    browser \"http://www.google.co.uk/search?hl=en&q=definition+$searchterm&btnI=I%27m+Feeling+Lucky\" &\n}\n# alias def=definition\n\n# gh(){\n#     url=\"http://www.google.com/search?q=\"\n#     browser \"${url}site%3A$*\" &\n#     browser \"${url}site%3A$* login\" &\n#     browser \"${url}link%3A$*\" &\n#     browser \"${url}related%3A$*\" &\n# }\n\n\nretry(){\n    local cmd=\"$1\"\n    local host=\"${2##*@}\"\n    #local user=\"${2%%@*}\"\n    local args=(\"${@:3}\")\n    if [ -z \"$host\" ]; then\n        echo \"You must supply a hostname or ip address to connect to\"\n        return 2\n    fi\n    if [ \"$cmd\" = \"ssh\" ] || [ \"$cmd\" = \"rdp\" ]; then\n        whenup \"$host\" || return 1\n    fi\n    #[ \"$cmd\" = \"ssh\" ] && host=\"root@$host\"\n    if [ \"$cmd\" = \"ssh\" ]; then\n        until port \"${host##*@}\" 22 >/dev/null; do\n            tstamp \"trying $host port 22\"\n            sleep 1\n        done\n    elif [ \"$cmd\" = \"rdp\" ]; then\n        until port \"$host\" 3389 >/dev/null; do\n            tstamp \"trying $host port 3389\"\n            sleep 1\n        done\n    fi\n    [ \"$cmd\" = \"ssh\" ] && printargs=\"\" || printargs=\"${args[*]}\"\n    until \"$cmd\" \"$host\" \"${args[@]}\"; do\n        sleep 1\n        tstamp \"trying $cmd $host $printargs\"\n    done\n    echo >/dev/null\n}\n\n\nrdp(){\n    if is_mac; then\n        \"/Applications/Remote Desktop Connection.app/Contents/MacOS/Remote Desktop Connection\" \"$@\" &\n    else\n        [ -n \"$1\" ] || return 1\n        local resolution=\"800x600\"\n        if [ \"$(xdpyinfo | awk '/dimensions/ {print $2}' | sed 's/x.*//')\" -gt 1024 ]; then\n            resolution=\"1024x768\"\n        fi\n        if type -P krdc &>/dev/null; then\n            krdc \"rdp:/$WINDOWSDOMAIN\\\\$WINDOWSUSER@$*\" &\n            exit 0\n        elif type -P rdesktop &>/dev/null; then\n            rdesktop -u \"$WINDOWSUSER\" -d \"$WINDOWSDOMAIN\" \"$@\" -g \"$resolution\" &\n            exit 0\n        else\n            echo  \"Could not find krdc or rdesktop in path\"\n            return 1\n        fi\n    fi\n}\n\nrerdp(){\n    retry \"whenport $1 3389; rdp\" \"$1\"\n}\n\n\n# ============================================================================ #\n#                                   L i n u x\n# ============================================================================ #\n\nif is_linux; then\n\n    ipl(){\n        iptables -L | nl\n    }\n\nfi\n\n\n# ============================================================================ #\n#                                 M a c   O S X\n# ============================================================================ #\n\nif ! is_mac; then\n    return\nfi\n\ndnsservers(){\n    scutil --dns | grep 'nameserver\\[[0-9]*\\]' | sort -u\n}\n\nflushdns(){\n    dscacheutil -flushcache\n    sudo killall -HUP mDNSResponder\n}\nalias flushcache=flushdns\n\n#APPLE_INTERFACES=\"Ethernet Airport\"\n#APPLE_INTERFACES=\"$(networksetup -listallnetworkservices | grep -E 'Ethernet|Wi-Fi')\"\nunset APPLE_INTERFACES\n\nget_apple_interfaces(){\n    networksetup -listallnetworkservices | grep -E 'Ethernet|Wi-Fi'\n}\n\n# Cisco AnyConnect set these rules which mess up my ability to connect directly to VirtualBox VMs on HostOnly Networking\ncleardeny(){\n    sudo ipfw delete \"$(sudo ipfw list | grep deny | awk '{print $1}')\"\n}\nipfwqflush(){\n    sudo ipfw -q flush\n}\n\nisMacNetworkService(){\n    local interface=\"$1\"\n    if [ \"$interface\" != \"Thunderbolt Ethernet\" ] &&\n       [ \"$interface\" != \"Wi-Fi\" ]; then\n        echo \"interface must be one of Thunderbolt Ethernet or Wi-Fi\"\n        return 1\n    fi\n}\n\nset_dns(){\n    get_apple_interfaces |\n    while read -r interface; do\n        sudo networksetup -setdnsservers \"$interface\" \"$@\"\n    done\n}\n\nset_dns_search(){\n    get_apple_interfaces |\n    while read -r interface; do\n        sudo networksetup -setsearchdomains \"$interface\" \"$@\"\n    done\n}\n\nset_dns_search_empty(){\n        set_dns_search \"Empty\"\n}\n# this wasn't found as an alias from another function\nclear_dns_search(){\n    set_dns_search_empty\n}\n\nfunction publicdns(){\n    set_dns 4.2.2.1 4.2.2.2 4.2.2.3 4.2.2.4 4.2.2.5 4.2.2.6\n    set_dns_search_empty\n}\n\nfunction dhcpdns(){\n    clear_dns_search # hangs without this as I think it tries to query DNS for all the suffixes in the list\n    set_dns \"Empty\"\n    #networksetup -setsearchdomains <networkservice> <domain1> [domain2]\n}\n\nget_wifi_interface(){\n    networksetup -listnetworkserviceorder |\n    grep \"Hardware.*Wi-Fi\" |\n    sed 's/.*: //;s/)$//'\n}\n\nget_wifi_network(){\n    networksetup -getairportnetwork \"$(get_wifi_interface)\" | sed 's/^Current Wi-Fi Network: //'\n}\n\nset_wifi_network(){\n    networksetup -setairportnetwork \"$(get_wifi_interface)\" \"$*\"\n}\n\nwifi(){\n    if [ $# -eq 1 ]; then\n        airport on\n        set_wifi_network \"$1\"\n    elif [ $# -eq 0 ]; then\n        get_wifi_network\n    else\n        echo \"usage: wifi <network>\"\n        return 1\n    fi\n}\n\nwifi_networks_preferred(){\n    networksetup -listpreferredwirelessnetworks \"$(get_wifi_interface)\"\n}\n\nairport(){\n    networksetup -setairportpower \"$(get_wifi_interface)\" \"$1\"\n}\nalias air=airport\n\nairportr(){\n    airport off\n    airport on\n}\nalias airr=airportr\nalias ag=\"airr; g\"\n\nwatchwifi(){\n    scnum 39\n    while true; do\n        checkwifi\n        sleep 30 || break\n    done\n}\n\ncheckwifi(){\n    # needs to be global otherwise will be forgotten between runs of this program\n    [ -z \"$wifi_failures\" ] && wifi_failures=0\n    for((i=1;i<=3;i++)); do\n        if ping -c1 -W1 4.2.2.1 >/dev/null; then\n            if [ \"$wifi_failures\" -gt 0 ]; then\n                tstamp \"wifi recovered from $wifi_failures failures\"\n            fi\n            wifi_failures=0\n            return\n        else\n            ((wifi_failures+=1))\n            tstamp \"$wifi_failures wifi failures\"\n        fi\n    done\n    timestamp \"RESTARTING WIFI\"\n    airportr\n}\n\nsetdhcp(){\n    isMacNetworkService \"$1\" || return 1;\n    sudo networksetup -setdhcp \"$1\"\n}\n\nrenewdhcp(){\n    sudo ipconfig set \"$1\" DHCP\n}\n\n#sethomenet(){\n#    isMacNetworkService \"$1\" || return 1;\n#    sudo networksetup -setmanual \"$1\" x.x.x.x 255.255.255.0 x.x.x.1\n#    sudo route delete 0.0.0.0\n#    sudo route add 0.0.0.0 x.x.x.1\n#    publicdns\n#}\n"
  },
  {
    "path": ".bash.d/nodejs.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2019 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    N o d e\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/os_detection.sh\"\n\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# output from 'npm bin'\nif [ -d ~/node_modules/.bin ]; then\n    add_PATH ~/node_modules/.bin\nfi\n\nif [ -d \"$bash_tools/node_modules/.bin\" ]; then\n    add_PATH \"$bash_tools/node_modules/.bin\"\nfi\n\nalias lsnodebin='ls -d ~/node_modules/.bin/* 2>/dev/null'\nalias llnodebin='ls -ld ~/node_modules/.bin/* 2>/dev/null'\n"
  },
  {
    "path": ".bash.d/os_detection.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2010 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\ntype is_linux &>/dev/null &&\ntype isMac &>/dev/null &&\ntype isGoogleCloudShell &>/dev/null &&\ntype isAzureCloudShell &>/dev/null &&\nreturn\n\nget_os(){\n    if [ -z \"${operating_system:-}\" ] ||\n       ! [[ \"$operating_system\" =~ ^(Linux|Darwin)$ ]]; then\n        operating_system=\"$(uname -s)\"\n        export operating_system\n    fi\n}\n\nisLinux(){\n    [ -n \"${LINUX:-}\" ] && return 0\n    get_os\n    if [ \"$operating_system\" = Linux ]; then\n        export LINUX=1\n        return 0\n    fi\n    return 1\n}\n\nisMac(){\n    [ -n \"${OSX:-}\" ] && return 0\n    get_os\n    if [ \"$operating_system\" = Darwin ]; then\n        export APPLE=1\n        export OSX=1\n        return 0\n    fi\n    return 1\n}\n\nisGoogleCloudShell(){\n    [ -n \"${GOOGLE_CLOUD_SHELL:-}\" ] && return 0\n    get_os\n    [ \"$operating_system\" = Linux ] || return 1\n    # DEVSHELL_PROJECT_ID is more likely to be unique to GCP Cloud Shell environment\n    #if [ -n \"${GOOGLE_CLOUD_PROJECT:-}\" ]; then\n    if [ -n \"${DEVSHELL_PROJECT_ID:-}\" ]; then\n        export GOOGLE_CLOUD_SHELL=1\n        return 0\n    fi\n    return 1\n}\n\nisAzureCloudShell(){\n    [ -n \"${AZURE_CLOUD_SHELL:-}\"  ] && return 0\n    get_os\n    [ \"$operating_system\" = Linux ] || return 1\n    if [ -n \"${ACC_TERM_ID:-}\" ]; then\n        export AZURE_CLOUD_SHELL=1\n        return 0\n    fi\n    return 1\n}\n\n# for compatibility to use the same names as non-interactive lib/\nis_linux(){\n    isLinux\n}\n\nis_mac(){\n    isMac\n}\n\nis_google_cloud_shell(){\n    isGoogleCloudShell \"$@\"\n}\n\nis_azure_cloud_shell(){\n    isAzureCloudShell \"$@\"\n}\n\n# make this safe to import in set -e scripts\nis_linux || :\nisMac || :\nisGoogleCloudShell || :\nisAzureCloudShell || :\n"
  },
  {
    "path": ".bash.d/paths.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   $ P A T H\n# ============================================================================ #\n\n# general path additions that aren't big enough to have their own <technology>.sh file\n\n# this is sourced in .bashrc before .bash.d/*.sh because add_PATH() is used extensively everywhere to deduplicate $PATHs across disparate code and also reloads before it gets to this point in the .bash.d/*.sh lexically ordered list\n\nif type add_PATH &>/dev/null && [ -n \"${PATHS_SET:-}\" ]; then\n    return\nfi\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\ngithub=\"${github:-$HOME/github}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# ============================================================================ #\n\nrepaths(){\n    unset PATHS_SET\n    # shellcheck disable=SC1091\n    source \"$bash_tools/.bash.d/paths.sh\"\n}\n\n#export PATH=\"${PATH%%:~/github*}\"\nadd_PATH(){\n    export PATH=\"$PATH:$1\"\n    # this clever stuff kills performance and I want my shell to open faster\n    # it's not worth saving a few duplicates in $PATH\n    # was used by dedupe paths at the end of this file\n    #local env_var\n    #local path\n    #if [ $# -gt 1 ]; then\n    #    env_var=\"$1\"\n    #    path=\"$2\"\n    #else\n    #    env_var=PATH\n    #    path=\"${1:-}\"\n    #fi\n    #path=\"${path%/}\"\n    #path=\"${path//[[:space:]]/}\"\n    #if [[ \"$path\" =~ \\$ ]]; then\n    #    echo \"WARNING: skipping add path '$path' for safety\"\n    #    return\n    #fi\n    #if ! [[ \"${!env_var}\" =~ (^|:)$path(:|$) ]]; then\n    #    # shellcheck disable=SC2140\n    #    eval \"$env_var\"=\"${!env_var}:$path\"\n    #fi\n    ## to prevent Empty compile time value given to use lib at /Users/hari/perl5/lib/perl5/perl5lib.pm line 17.\n    ##PERL5LIB=\"${PERL5LIB##:}\"\n    ## fix for Codeship having a space after one of the items in their $PATH, causing the second half of the $PATH to error out as a command\n    #eval \"$env_var\"=\"${!env_var//[[:space:]]/}\"\n    #eval \"$env_var\"=\"${!env_var##:}\"\n    #export \"${env_var?env_var not defined in add_PATH}\"\n}\n\n# use 'which -a'\n#\n#binpaths(){\n#    if [ $# != 1 ]; then\n#        echo \"usage: binpaths <binary>\"\n#        return 1\n#    fi\n#    local bin=\"$1\"\n#    tr ':' '\\n' <<< \"$PATH\" |\n#    while read -r path; do\n#        if [ -x \"$path/$bin\" ]; then\n#            echo \"$path/$bin\"\n#        fi\n#    done\n#}\n\n# need newer Homebrew curl for --cookies-from-browser functionality\n# Homebrew build wasn't compiled with it anyway - see curl_with_cookies.sh script for a more generic workaround\n# that works with any version of curl\n#if [ -d /opt/homebrew/opt/curl/bin ]; then\n#    PATH=\"/opt/homebrew/opt/curl/bin:$PATH\"\n#fi\n\nadd_PATH \"/bin\"\nadd_PATH \"/usr/bin\"\nadd_PATH \"/sbin\"\nadd_PATH \"/usr/sbin\"\nadd_PATH \"/usr/local/sbin\"\nadd_PATH \"/usr/local/bin\"\nadd_PATH \"/usr/local/opt/python/libexec/bin\"  # Mac brew installed Python, must be ahead of ~/anaconda/bin below\nadd_PATH \"/opt/homebrew/bin/\"  # on new M1 Macs\nadd_PATH \"$bash_tools\"\nadd_PATH ~/bin\nadd_PATH ~/.local/bin\nadd_PATH ~/venv/bin\nwhile read -r x; do\n    # much less noisy to just just find the right dirs instead of testing lots of files\n    #[ -d \"$x\" ] || continue\n    #if [ -d \"$x/bin\" ]; then\n    #    add_PATH \"$x/bin\"\n    #else\n        add_PATH \"$x\"\n    #fi\ndone < <(for x in \"$bash_tools\" ~/bin; do find \"$x\" -maxdepth 2 -type d -name bin; done)\n\n# Serverless.com framework\nif [ -d ~/.serverless/bin ]; then\n    add_PATH ~/.serverless/bin\nfi\n\n# HomeBrew on Linux\nif [ -d /opt/homebrew/bin ]; then\n    add_PATH /opt/homebrew/bin\nfi\n\n# HomeBrew on Linux\nif [ -d ~/.linuxbrew/bin ]; then\n    add_PATH ~/.linuxbrew/bin\nfi\n\n# AWS CLI Linux install location\nif [ -d ~/.local/bin ]; then\n    add_PATH ~/.local/bin\nfi\n\n# AWS SAM CLI Linux install location\nif [ -d \"/home/linuxbrew/.linuxbrew/bin\" ]; then\n    add_PATH \"/home/linuxbrew/.linuxbrew/bin\"\nfi\n\n# Rancher Desktop\nif [ -d ~/.rd/bin ]; then\n    add_PATH ~/.rd/bin\nfi\n\nif [ -d ~/.pulumi/bin ]; then\n    add_PATH ~/.pulumi/bin\nfi\n\n#add_PATH \"${JX_HOME:-$HOME/.jx}/bin\"\nadd_PATH ~/.jx/bin\n\n# do the same with MANPATH\nif [ -d ~/man ]; then\n    MANPATH=~/man:\"${MANPATH:-}\"\n    export MANPATH\nfi\n\n# added to .bash_profile by SnowSQL installer\n#if [ -d /Applications/SnowSQL.app/Contents/MacOS ]; then\n#    add_PATH /Applications/SnowSQL.app/Contents/MacOS\n#fi\n\n# so that you can open files in IntelliJ from the command line: idea <filename>\nif [ -d \"/Applications/IntelliJ IDEA CE.app/Contents/MacOS\" ]; then\n    add_PATH \"/Applications/IntelliJ IDEA CE.app/Contents/MacOS\"\nfi\n\nif [ -d \"/Applications/Visual Studio Code.app\" ]; then\n    # don't need this one as you can just 'code /path/to/filename' to open the file in VS Code\n    #add_PATH \"/Applications/Visual Studio Code.app/Contents/MacOS\"  # Electron IDE is here\n    add_PATH \"/Applications/Visual Studio Code.app/Contents/Resources/app/bin\"  # code CLI is here\nfi\n\n# ============================================================================ #\n#                                A n a c o n d a\n# ============================================================================ #\n\n# Make sure to customize Anaconda installation and de-select Modify Path otherwise it'll change the bash profile\n\n# XXX: WARNING - this will appear earlier in the $PATH than the python bin paths, so if you have it installed, you should use it\n#                otherwise pylint for example may be called from anaconda/bin but not have the pip modules necessary to check files, leading to CI breakages\n\n# for the 'conda' command\nadd_PATH ~/anaconda/bin\n\n# version installed by HomeBrew\nadd_PATH /usr/local/anaconda3/bin\n\n\n# ============================================================================ #\n#                           P a r q u e t   T o o l s\n# ============================================================================ #\n\nfor x in ~/bin/parquet-tools-*; do\n    if [ -d \"$x\" ]; then\n        add_PATH \"$x\"\n    fi\ndone\n\nif [ -d /usr/local/parquet-tools ]; then\n    add_PATH \"/usr/local/parquet-tools\"\nfi\n\n\n# ============================================================================ #\n#                         M y   G i t H u b   r e p o s\n# ============================================================================ #\n\n# $github defined in aliases.sh\n# shellcheck disable=SC2154\nadd_PATH \"$bash_tools\"\nwhile read -r x; do\n    add_PATH \"$x\"\ndone < <(find \"$bash_tools\" -maxdepth 1 -type d)\nadd_PATH \"$github/go-tools\"\nadd_PATH \"$github/go\"\nadd_PATH \"$github/go-tools/bin\"\nadd_PATH \"$github/go/bin\"\nadd_PATH \"$github/perl-tools\"\nadd_PATH \"$github/perl\"\nadd_PATH \"$github/pytools\"\nadd_PATH \"$github/tools\"\n#add_PATH \"$github/tool\"\nadd_PATH \"$github/nagios-plugins\"\nadd_PATH \"$github/nagios-plugin-kafka\"\nadd_PATH \"$github/spotify\"\nadd_PATH \"$github/spotify-tools\"\n\nif is_linux; then\n    add_PATH ~/.buildkite-agent/bin\nfi\n\n# ============================================================================ #\n\nlink_latest(){\n    # -p suffixes / on dirs, which we grep filter on to make sure we only link dirs\n    # shellcheck disable=SC2010\n    ls -d -p \"$@\" |\n    grep \"/$\"  |\n    tail -n 1  |\n    while read -r path; do\n        [ -d \"$path\" ] || continue\n        #local path_noversion=\"$( echo \"$path\" | perl -pn -e 's/-\\d+(\\.v?\\d+)*(-\\d+|-[a-z]+)?\\/?$//' )\"\n        local path_noversion\n        path_noversion=\"$(perl -pn -e 's/-\\d+[\\.\\w\\d-]+\\/?$//' <<< \"$path\")\"\n        if [ \"$path_noversion\" = \"$path\" ]; then\n            echo \"FAILED to strip version, linking back on itself will create a link in subdir\"\n            return 1\n        fi\n        [ -e \"$path_noversion\" ] && [ ! -L \"$path_noversion\" ] && continue\n        if is_mac; then\n            local ln_opts=\"-h\"\n        else\n            local ln_opts=\"-T\"\n        fi\n        # if you're in 'admin' group on Mac you don't really need to sudo here\n        # shellcheck disable=SC2154\n        $sudo ln -vfs $ln_opts -- \"$path\" \"$path_noversion\"\n    done\n}\n\n\n# ============================================================================ #\n# ============================================================================ #\n#                               O l d   S t u f f\n# ============================================================================ #\n\n# Most of the stuff below has been migrated to Docker rather than /usr/local installs\n\n# ============================================================================ #\n#                            A p a c h e   D r i l l\n# ============================================================================ #\n\n#link_latest /usr/local/apache-drill-*\n#export DRILL_HOME=/usr/local/apache-drill\n#add_PATH \"$DRILL_HOME/bin\"\n\n\n# ============================================================================ #\n#                                    M i s c\n# ============================================================================ #\n\n#add_PATH \"/usr/local/etcd\"\n#add_PATH \"/usr/local/artifactory-oss/bin\"\n#add_PATH \"/usr/local/jmeter/bin\"\n#add_PATH \"/usr/local/jruby/bin\"\n#add_PATH \"/usr/local/jython/bin\"\n#add_PATH \"/usr/local/mysql/bin\"\n\n#add_PATH ~/bin/expect\n#add_PATH \"$RANCID_HOME/bin\"\n#add_PATH /usr/lib/bin/distcc\n#add_PATH \"/usr/lib/nagios/plugins\"\n#add_PATH \"/usr/nagios/libexec\"\n#add_PATH \"/usr/nagios/libexec/contrib\"\n\n#if is_mac; then\n#    # MacPort and Octave installation\n#    add_PATH /opt/local/bin\n#\n#    if [ -d \"/Applications/VMware Fusion.app/Contents/Library\" ]; then\n#        add_PATH \"/Applications/VMware Fusion.app/Contents/Library\"\n#    fi\n#fi\n\n# ============================================================================ #\n#                               C a s s a n d r a\n# ============================================================================ #\n\n#export CASSANDRA_HOME=/usr/local/cassandra\n#export CCM_HOME=/usr/local/ccm\n#add_PATH \"$CASSANDRA_HOME/bin\"\n#add_PATH \"$CASSANDRA_HOME/tools/bin\"\n#add_PATH \"$CCM_HOME/bin\"\n\n\n# ============================================================================ #\n#                           E l a s t i c s e a r c h\n# ============================================================================ #\n\n#export ELASTICSEARCH_HOME=/usr/local/elasticsearch\n#add_PATH \"$ELASTICSEARCH_HOME/bin\"\n\n\n# ============================================================================ #\n#                               C o u c h b a s e\n# ============================================================================ #\n\n#export COUCHBASE_HOME=\"/Applications/Couchbase Server.app/Contents/Resources/couchbase-core\"\n#alias cbq=\"$COUCHBASE_HOME/bin/cbq\"\n#add_PATH \"$COUCHBASE_HOME/bin\"\n\n\n# ============================================================================ #\n#                                  G r o o v y\n# ============================================================================ #\n\n# brew install groovy\n#export GROOVY_HOME=/usr/local/opt/groovy/libexec\n# brew uninstall groovy\n# brew install groovysdk\n#export GROOVY_HOME=/usr/local/opt/groovysdk/libexec\n\n# using SDK Man now, sourced at end of private .bashrc\n\n\n# ============================================================================ #\n#                              0 x d a t a   H 2 O\n# ============================================================================ #\n\n#export H2O_HOME=/usr/local/h2o\n#alias h2o=\"cd $H2O_HOME && java -jar h2o.jar -Xmx1g\"\n\n\n# ============================================================================ #\n#                                   J e t t y\n# ============================================================================ #\n\n#export JETTY_HOME=\"/usr/local/jetty-hightide\"\n#alias jetty=\"cd $JETTY_HOME/ && java -jar start.jar\"\n\n\n# ============================================================================ #\n#                                   M e s o s\n# ============================================================================ #\n\n# this breaks parsing if supplying without port and causes duplicate --master switch if supplying the switch manually to mesos-slave or mesos-slave.sh\n#export MESOS_MASTER=$HOST:5050\n\n# link_latest /usr/local/mesos\n#export MESOS_HOME=/usr/local/mesos\n#add_PATH \"$MESOS_HOME/bin\"\n\n#if is_mac; then\n#    export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/src/.libs/libmesos.dylib\n#else\n#    # check this path\n#    export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/mesos/lib/libmesos.so\n#fi\n# deprecated old var\n#export MESOS_NATIVE_LIBRARY=\"$MESOS_NATIVE_JAVA_LIBRARY\"\n\n\n# ============================================================================ #\n#                                 M o n g o D B\n# ============================================================================ #\n\n#export MONGO_HOME=/usr/local/mongo\n#add_PATH \"$MONGO_HOME/bin\"\n#add_PATH \"$github/mtools\"\n\n\n# ============================================================================ #\n#                                   N e o 4 J\n# ============================================================================ #\n\n#export NEO4J_HOME=\"/usr/local/neo4j\"\n#add_PATH \"$NEO4J_HOME/bin\"\n\n\n# ============================================================================ #\n#                                    S o l r\n# ============================================================================ #\n\n# find /usr/local -type d -name 'apache-solr-*' -maxdepth 1 | while read path; do sudo ln -vfsh \"$path\" \"${path%%-*}\"; done\n# link_latest '/usr/local/apache-solr-*'\n# ln -vsf /usr/local/apache-solr /usr/local/solr\n# 3.x\n#export SOLR_HOME=/usr/local/apache-solr\n# 4.x\n#export SOLR_HOME=/usr/local/solr\n#export APACHE_SOLR_HOME=\"$SOLR_HOME\"\n#add_PATH \"$SOLR_HOME/bin\"\n#add_PATH \"$SOLR_HOME/example/scripts/cloud-scripts\"\n\n\n# ============================================================================ #\n#                                   S t o r m\n# ============================================================================ #\n\n#export STORM_HOME=/usr/local/storm\n#add_PATH \"$STORM_HOME/bin\"\n\n\n# ============================================================================ #\n#                                 T a c h y o n\n# ============================================================================ #\n\n#export TACHYON_HOME=/usr/local/tachyon\n#add_PATH \"$TACHYON_HOME/bin\"\n\n# ============================================================================ #\n#                              B a s h o   R i a k\n# ============================================================================ #\n\n#export RIAK_HOME=/usr/local/riak\n#add_PATH \"$RIAK_HOME/bin\"\n\n\n# ============================================================================ #\n#                                   S c a l a\n# ============================================================================ #\n\n#add_PATH \"/usr/local/scala/bin\"\n\n\n# ============================================================================ #\n#                                   S p a r k\n# ============================================================================ #\n\n#export SPARK_HOME=/usr/local/spark\n#add_PATH \"$SPARK_HOME/bin\"\n\n\n# ============================================================================ #\n#                               S o n a r Q u b e\n# ============================================================================ #\n\n#export SONAR_SCANNER_HOME=/usr/local/sonar-scanner\n#add_PATH \"$SONAR_SCANNER_HOME/bin\"\n\n\n# ============================================================================ #\n#                       TypeSafe Activator - Akka, Play\n# ============================================================================ #\n\n# link_latest /usr/local/activator-dist-*\n#export ACTIVATOR_HOME=/usr/local/activator-dist\n#add_PATH \"$ACTIVATOR_HOME\"\n\n# slows down new shells\n#dedupe_paths(){\n#    local var=\"${1:-PATH}\"\n#    local path_tmp=\"\"\n#    # <( ) only works in Bash, but breaks when sourced from sh\n#    # <( ) also ignores errors which don't get passed through the /dev/fd\n#    # while read -r path; do\n#    #done < <(tr ':' '\\n' <<< \"$PATH\")\n#    local IFS=':'\n#    for path in ${!var}; do\n#        if [[ \"$path\" =~ ^[[:space:]]*$ ]]; then\n#            continue\n#        fi\n#        if ! [[ \"$path_tmp\" =~ :$path(:|$) ]]; then\n#            path_tmp=\"$path_tmp:$path\"\n#        fi\n#    done\n#    eval export \"$var\"=\"\\\"$path_tmp\\\"\"\n#}\n\n# call in z_final.sh\n#dedupe_paths\n\nexport PATHS_SET=1\n"
  },
  {
    "path": ".bash.d/perl.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    P e r l\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/os_detection.sh\"\n\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\nif [ -d ~/perl5/bin ]; then\n    add_PATH ~/perl5/bin\nfi\n\n# see the effect of inserting a path like so\n# PERL5LIB=/path/to/blah perlpath\nperlpath(){\n    perl -e 'print join(\"\\n\", @INC) . \"\\n\";'\n}\n\n# XXX: Perl Taint mode resets and restricts the Perl Path to not use PERL5LIB for security\n#\n#   scripts like those in the Advanced Nagios Plugins Collection and DevOps-Perl-tools will need to add lib\n#\n#if [ -d /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Perl/ ]; then\n#    add_PATH PERL5LIB /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Perl\n#fi\nif [ -d ~/perl5/lib/perl5 ]; then\n    #add_PATH PERL5LIB ~/perl5/lib/perl5\n    #export PERL5LIB=\"$PERL5LIB:$HOME/perl5/lib/perl5\"\n    if ! [[ \"${PERL5LIB:-}\" == *\"$HOME/perl5/lib/perl5\"* ]]; then\n        export PERL5LIB=~/perl5/lib/perl5\"${PERL5LIB+:$PERL5LIB}\"\n    fi\n\nfi\nif ! [[ \"${PERL_LOCAL_LIB_ROOT:-}\" == *\"$HOME/perl5\"* ]]; then\n    export PERL_LOCAL_LIB_ROOT=\"$HOME/perl5${PERL_LOCAL_LIB_ROOT+:$PERL_LOCAL_LIB_ROOT}\"\nfi\nexport PERL_MB_OPT=\"--install_base '$HOME/perl5'\"\nexport PERL_MM_OPT=\"INSTALL_BASE=$HOME/perl5\"\n\nalias lsperlbin='ls -d ~/perl5/bin/* 2>/dev/null'\nalias llperlbin='ls -ld ~/perl5/bin/* 2>/dev/null'\n\n# cpanm --local-lib=~/perl5 local::lib\n# populates a bunch of Perl env vars pointing to ~/perl5/...\n# eval \"$(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)\"\n"
  },
  {
    "path": ".bash.d/postgres.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-16 13:20:17 +0000 (Mon, 16 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              P o s t g r e S Q L\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/functions.sh\"\n\n# highest priority env var\nalias pgpass='pass PGPASSWORD'\n# mac\nalias postgresd='postgres -D /usr/local/var/postgres'\n"
  },
  {
    "path": ".bash.d/prompt.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            P r o m p t   M a g i c\n# ============================================================================ #\n\n# XXX: Warning: This must be perfect - edit at your own peril as imperfect PS1 prompt codes cause terminal wrap around to the same line\n\n[ -n \"${GOOGLE_CLOUD_SHELL:-}\" ] && return\n#[ \"${USER:-}\" = vagrant ] && return\n\n# \\[\\033k\\033\\\\\\] is required for Screen auto title feature to detect prompt\n# replace \\033 with \\e as it's directly supported in PS1\n\n# need 'Defaults env_keep=STY' for this to not trigger on sudo su\n# shellcheck disable=SC1117\nSCREEN_ESCAPE=\"\\[\\ek\\e\\\\\\\\\\]\"\n\nPS1=\"\"\n\n# if inside Screen, set the screen escape inside PS1\n[ -n \"$STY\" ] && PS1=\"$SCREEN_ESCAPE\"\n\n# defined in adjacent colors.sh\n# shellcheck disable=SC2154\nPS1_COLOUR=\"$txtgrn\"\n# shellcheck disable=SC2154\nPS1_USER_COLOUR=\"$txtcyn\"\n\nif [ $EUID -eq 0 ]; then\n    # shellcheck disable=SC2154\n    PS1_COLOUR=\"$bldred\"\n    PS1_USER_COLOUR=\"$bldred\"\nfi\n\n# XXX: important that every single escape sequence is enclosed in \\[ \\] to make sure it isn't included in the line wrapping calculcation otherwise the lines wrap back on to themselves\n#   \\W      basename of cwd\n#   \\w      full path of cwd\n#   \\h      host\n# shellcheck disable=SC2154\n# export PS1+=\"\\[$PS1_COLOUR\\]\\t \\[$bldblu\\]\\w \\[$PS1_COLOUR\\]> \\[$txtrst\\]\"\n# shellcheck disable=SC1117\nexport PS1+=\"\\[$PS1_COLOUR\\]\\t \\[$PS1_USER_COLOUR\\]\\u\\[$txtwht\\]@\\[$bldcyn\\]\\h:\\[$bldpur\\]\\$(git branch 2>/dev/null | grep '^*' | sed 's/^*//') \\[$bldgrn\\]\\w \\[$PS1_COLOUR\\]> \\[$txtrst\\]\"\n\n#if type kube_ps1 &>/dev/null; then\n#    PS1='$(kube_ps1)'\" $PS1\"\n#fi\n\n\n# Screen relies on prompt having a dollar sign to detect the next word to set the screen title dynamically - .screenrc needs the following setting\n#\n# shelltitle ' > |'\n\n# TODO: make screen auto title detect # for root\n\n\n# For passing PS1 around:\n#\n#   base64 <<< \"$PS1\"\n#\n# and then pass the result through base64 --decode\n#\n#   base64 <<< \"$PS1\" | base64 --decode\n#\n# is a noop to demonstrate\n\nif [ -z \"$BASH_TIMING\" ]; then\n    export PS4=\"--> \"\nfi\n\n# Bash performance profiling, can be heavy on performance, &>/tmp/bash_perf.out then use a profiling script\n# combine with DEBUG=1 or set -x\nbash_timing(){\n    export BASH_TIMING=1\n    export PS4='$(date \"+%s.%N ($LINENO) + \") --> '\n}\n\ndebug_bashrc(){\n    PS4='+ $BASH_SOURCE:$LINENO:' bash -xic '' 2>&1 | less\n}\n\ndebug_bash_profile(){\n    PS4='+ $BASH_SOURCE:$LINENO:' bash -xlic '' 2>&1 | less\n}\n\nif is_mac; then\n    # turn this off on mac thing - it heavily pollutes $DEBUG output and does nothing because we have a custom prompt\n    if [[ \"${PROMPT_COMMAND:-}\" =~ update_terminal_cwd ]]; then\n        # this unsets direnv's hook\n        #unset PROMPT_COMMAND\n        PROMPT_COMMAND=\"${PROMPT_COMMAND//update_terminal_cwd;/}\"\n    fi\n    # stripping update_terminal_cwd can leave some weird broken PROMPT_COMMAND start due to interaction with direnv, so strip it without forking to sed/perl\n    while [[ \"${PROMPT_COMMAND:-}\" =~ ^[[:space:]]|^\\; ]]; do\n        PROMPT_COMMAND=\"${PROMPT_COMMAND##[[:space:]]}\"\n        PROMPT_COMMAND=\"${PROMPT_COMMAND##;}\"\n    done\n    export PROMPT_COMMAND\n\n    export SHELL_SESSION_HISTORY=0\nfi\n"
  },
  {
    "path": ".bash.d/python.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  P y t h o n\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# silence those annoying Python 2 cryptography warnings that mess up our programs outputs\nexport PYTHONWARNINGS=ignore::UserWarning\n\n# shellcheck disable=SC1090,SC1091\ntype add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# see the effect of inserting a path like so\n# PYTHONPATH=/path/to/blah pythonpath\npythonpath(){\n    python -c 'from __future__ import print_function; import sys; [print(_) for _ in sys.path if _]'\n}\n# enable this to avoid creating .pyc files (sometimes they trip you up executing outdated python code)\n# export PYTHONDONTWRITEBYTECODE=1\n\nif is_mac; then\n    # try to find pip in brew installed Python versions since it is\n    # not in /System/Library/Frameworks/Python.framework/Versions/2.7/bin\n    for dir in /usr/local/Cellar/python*; do\n        if [ -d \"$dir\" ]; then\n            add_PATH \"$dir/bin\"\n        fi\n    done\nfi\n\nif [ -d ~/Library/Python ]; then\n    for x in ~/Library/Python/*/bin; do\n        [ -d \"$x\" ] || continue\n        add_PATH \"$x\"\n    done\nfi\n\nalias lspythonbin='ls -d ~/Library/Python/*/bin/* 2>/dev/null'\nalias llpythonbin='ls -ld ~/Library/Python/*/bin/* 2>/dev/null'\nalias lspybin=lspythonbin\nalias llpybin=llpythonbin\n\n# RHEL8 has split python2 / python3 and removed default 'python' :-(\nif ! type -P python &>/dev/null; then\n    if type -P python2 &>/dev/null; then\n        python(){ python2 \"$@\"; }\n    elif type -P python3 &>/dev/null; then\n        python(){ python3 \"$@\"; }\n    fi\nfi\n"
  },
  {
    "path": ".bash.d/rancid.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2010 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  R a n c i d\n# ============================================================================ #\n\nexport RANCID_HOME=~/rancid\n\n#flogin(){\n#    title \"$1\"\n#    command flogin \"${@:1}\"\n#    title \" \"\n#}\n\n# l33ter way of generating functions for all of the rancid programs\n#for x in $(ls \"$RANCID_HOME/bin/\"*login | sed 's/.*\\///;s/*$//'); do\n#    eval \"$x\"'(){ title \"$1\"\n#    command '\"$x\"' ${@:1}\n#    title \" \"\n#    }'\n#done\n\n# More prim and proper abstracted function with minimal function code\nrancidlogin_func(){\n    local prog=\"$1\"\n    local host=\"$2\"\n    shift\n    title \"$host\"\n    command \"$prog\" \"$@\"\n    title \"$LAST_TITLE\"\n}\n\n#for x in \"$RANCID_HOME/bin/\"*login; do y=${x##*/}; alias \"$y\"=\"rancidlogin_func $y\"; done\nfor x in \"$RANCID_HOME/bin/\"*login; do\n    y=\"${x##*/}\"\n    # needs to be evaluated here to build dynamic aliases\n    # shellcheck disable=SC2139,SC2140\n    alias \"$y\"=\"rancidlogin_func $y\"\ndone\n\n# for x in \"$RANCID_HOME/bin/\"*login; do y=\"${x##*/}\"; which \"${y%ogin}\" &>/dev/null || alias \"${y%ogin}\"=\"$y\"; done\n# This is slow to do every time so just past the echo output from:\n# for x in \"$RANCID_HOME/bin/\"*login; do y=\"${x##*/}\"; which \"${y%ogin}\" &>/dev/null || echo alias \"${y%ogin}\"=\"$y\"; done\n#alias al=alogin\n#alias avol=avologin\n#alias bl=blogin\n#alias cl=clogin\n#alias el=elogin\n#alias fl=flogin\n#alias hl=hlogin\n#alias htl=htlogin\n#alias jl=jlogin\n#alias mrvl=mrvlogin\n# nl is a real program so skipped nlogin\n#alias nsl=nslogin\n#alias rivl=rivlogin\n#alias tl=tlogin\n#alias tntl=tntlogin\n"
  },
  {
    "path": ".bash.d/ruby.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              R u b y   /   G e m\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/os_detection.sh\"\n\ntype add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# gems will be installed to ~/.gem/ruby/x.y.z/bin\n\n\n# ============================================================================ #\n#                                     R V M\n# ============================================================================ #\n# RVM\n\n# usually /usr/local/rvm/bin\nif [ -n \"${rvm_bin_path:-}\" ];then\n    add_PATH \"$rvm_bin_path\"\nfi\n\nif [ -d /usr/local/rvm/bin ]; then\n    add_PATH /usr/local/rvm/bin\nfi\n\nif [ -d ~/.rvm/bin ]; then\n    add_PATH ~/.rvm/bin\nfi\n\n# ============================================================================ #\n#                                   R b E n v\n# ============================================================================ #\n# RbEnv\n\nif [ -f ~/.rbenv/bin ]; then\n    add_PATH ~/.rbenv/bin\nfi\n\n#eval \"$(rbenv init - --no-rehash bash)\"\n\n# ============================================================================ #\n#                          R u b y   U s e r   D i r s\n# ============================================================================ #\n# Ruby User Dirs\n\n# add newest ruby to path first\nruby_bins=\"$(\n    find ~/.gem/ruby -maxdepth 2 -name bin -type d 2>/dev/null\n    find ~/.local/share/gem/ruby -maxdepth 2 -name bin -type d 2>/dev/null\n)\"\n#if is_mac; then\n#    ruby_bins_newest=\"$(tail -r <<< \"$ruby_bins\")\"\n#else\n#    ruby_bins_newest=\"$(tac <<< \"$ruby_bins\")\"\n#fi\nruby_bins_newest=\"$(sort -Vr <<< \"$ruby_bins\")\"\nfor ruby_bin in $ruby_bins_newest; do\n    add_PATH \"$ruby_bin\"\ndone\nunset ruby_bins\nunset ruby_bins_newest\n\nalias lsrubybin='ls -d ~/.gem/ruby/*/bin/* 2>/dev/null'\nalias llrubybin='ls -ld ~/.gem/ruby/*/bin/* 2>/dev/null'\n\n# ============================================================================ #\n\n# HomeBrew install on Linux (for AWS SAM CLI)\nif [ -d ~/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/current/bin ]; then\n    add_PATH ~/.linuxbrew/Homebrew/Library/Homebrew/vendor/portable-ruby/current/bin\nfi\n"
  },
  {
    "path": ".bash.d/screen.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  S c r e e n\n# ============================================================================ #\n\n# quickly open new screen terminal in the same $PWD\nalias scbash=\"screen bash\"\n\nsc(){\n    checkprog screen || return 1\n    isscreen && { echo \"I am already in a screen, aborting\"; return 1; }\n    screen -wipe\n    local session=main\n    local detached_screens\n    detached_screens=\"$(screen -ls | grep Detached)\"\n    if [ -n \"$detached_screens\" ] &&\n       [ \"$(wc -l <<< \"$detached_screens\" | awk '{print $1}')\" = 1 ]; then\n        session=\"$(awk '{print $1;exit}' <<< \"$detached_screens\")\"\n    fi\n    screen -aARRD -S \"$session\" \"$@\"\n}\n\nscreencmd(){\n    screen -X \"$@\"\n}\n\nscreensleep(){\n    screen \"$@\"\n    sleep 0.1\n}\n\nalias scnum=\"screen -X number\"\n\n#screen_get_pid(){\n#    # Mac ps doesn't have --noheaders\n#    ps -p \"${PPID}\" -o ppid | tail -n +2 | sed 's/[[:space:]]//g'\n#}\n#\n#screen_get_session_name(){\n#    local ppid\n#    ppid=\"$(screen_get_pid)\"\n#    screen -ls | awk \"/^[[:blank:]]$ppid/{print \\$1}\" | cut -d. -f2\n#}\n\n# needs modern version of screen for -Q switch - on Mac you must brew install screen to get recent version, then start new screen\n# when installing GNU screen you will lose Mac's screen since /usr/bin/screen uses a different /var/folders/...../.screen directory for screen sessions\nscreen_renumber_windows(){\n    local windowlist\n    windowlist=\"$(screen -Q windows | grep -Eo '(^|[[:blank:]])[[:digit:]]+')\"\n    i=0\n    for windownum in $windowlist; do\n        screen -p \"$windownum\" -X number \"$i\"\n        ((i+=1))\n    done\n}\nalias screnum=screen_renumber_windows\n\nscreenbuf(){\n    local tmp\n    tmp=\"$(mktemp /tmp/screen-exchange.XXXXXX)\"\n    cat > \"$tmp\"\n    screen -X readbuf \"$tmp\"\n    rm -- \"$tmp\";\n}\nalias sb=screenbuf\n\nsh_server_real(){\n    for x in \"$@\"; do\n        echo \"sh server real $x | i $x|Weight|Total\"\n    done |\n    tee /dev/stderr |\n    screenbuf\n    echo\n}\nalias fsr=sh_server_real\nalias ssr=sh_server_real\n\n# this idea's is a bust so far...\n#function c(){\n#    screen -t \"$@\" bash -c \". ~/.bashrc && eval $@\"\n#}\n"
  },
  {
    "path": ".bash.d/skaffold.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-05 16:31:14 +0100 (Mon, 05 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                S k a f f o l d\n# ============================================================================ #\n\n#bash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n#autocomplete skaffold\n"
  },
  {
    "path": ".bash.d/spinner.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-06-25 15:20:39\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 S p i n n e r\n# ============================================================================ #\n\nspinner(){\n    local msg=\"$* \"\n    #local num=${2:-100}\n    local num=1000\n    #local delay=${3:-0.00001}\n    local delay=0.00001\n    spin='-\\|/'\n    #printf \"${msg//?/ }\"\n    printf \"%s\" \"$msg \"\n    for ((i=0; i < num; i++)); do\n        sleep $delay\n        # This way results in more flashing\n        #printf \"\\r${msg}${spin:$((${i}%${#spin})):1}\"\n        # TODO: naughty allowing variables in printf format string but fiddly with msg var replaced backspace otherwise, clean up later...\n        # shellcheck disable=SC2059\n        printf \"\\\\b${msg//?/\\\\b}${msg}${spin:$((i % ${#spin})):1}\"\n    done\n    printf '\\b '\n    echo\n    echo \"done\"\n}\n"
  },
  {
    "path": ".bash.d/spotify.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-18 19:43:00 +0100 (Sat, 18 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# exports SPOTIFY_ACCESS_TOKEN so all the user private data spotify tools (../spotify_*.sh) can use it easily without re-prompting ia browser\n\nspotifysession(){\n    local SPOTIFY_ACCESS_TOKEN\n    # this would prevent it from being exported to the shell as we want to make it easier to use full spotify tools\n    #local SPOTIFY_ACCESS_TOKEN\n    # SECONDS cannot be reset in the background in spotify_token_expire_timer() function\n    SECONDS=0\n    spotify_token_expire_timer &\n    # defined in ../.bashrc\n    # shellcheck disable=SC2154\n    SPOTIFY_ACCESS_TOKEN=\"$(SPOTIFY_PRIVATE=1 \"$bash_tools/spotify/spotify_api_token.sh\")\"\n    export SPOTIFY_ACCESS_TOKEN\n    timestamp \"starting spotify session shell\"\n    \"$SHELL\"\n    timestamp \"exiting spotify session shell\"\n    unset SPOTIFY_ACCESS_TOKEN\n}\n\nspotify_token_expire_timer(){\n    # we have the same $$ as our foreground shell\n    local ppid=$$\n    while true; do\n        if [ \"$SECONDS\" -ge 3600 ]; then\n            # XXX: this would never work as it'd only affect this background thread\n            #unset SPOTIFY_ACCESS_TOKEN\n            # instead kill the shell session and handle the unset in the spotifysession() function\n            timestamp \"Spotify Token expired - killing spotify shell\"\n            pgrep -P \"$ppid\" |\n            # protect own shell so we can finish this code pipe\n            grep -v $$ |\n            # bash needs a -HUP signal, ignores TERM\n            xargs kill -HUP\n            break\n        fi\n        sleep 1\n    done\n}\n"
  },
  {
    "path": ".bash.d/ssh-agent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               S S H   A g e n t\n# ============================================================================ #\n\n# keychain id_rsa\n# . .keychain/$HOSTNAME-sh\n\nssh_agent(){\n    #if [ $UID != 0 ]; then\n        local SSH_ENV_FILE=~/.ssh-agent.env\n        if [ -f \"${SSH_ENV_FILE:-}\" ]; then\n            # shellcheck source=~/.agent.env\n            # shellcheck disable=SC1090,SC1091\n            . \"$SSH_ENV_FILE\" > /dev/null\n\n            if ! kill -0 \"$SSH_AGENT_PID\" >/dev/null 2>&1; then\n                echo \"Stale ssh-agent found. Spawning new agent...\"\n                killall -9 ssh-agent\n                eval \"$(ssh-agent | tee \"$SSH_ENV_FILE\")\" #| grep -v \"^Agent pid [[:digit:]]\\+$\"\n                # lazy evaluated ssh func now so it's not prompted until used\n                #ssh-add\n            elif [ \"$(ps -p \"$SSH_AGENT_PID\" -o comm=)\" != \"ssh-agent\" ]; then\n                echo \"ssh-agent PID does not belong to ssh-agent, spawning new agent...\"\n                eval \"$(ssh-agent | tee \"$SSH_ENV_FILE\")\" #| grep -v \"^Agent pid [[:digit:]]\\+$\"\n                # lazy evaluated ssh func now so it's not prompted until used\n                #ssh-add\n            fi\n        else\n            echo \"Starting ssh-agent...\"\n            killall -9 ssh-agent\n            eval \"$(ssh-agent | tee \"$SSH_ENV_FILE\")\"\n            # lazy evaluated ssh func now so it's not prompted until used\n            #ssh-add\n        fi\n        #clear\n    #fi\n}\n\n[ -n \"${GOOGLE_CLOUD_SHELL:-}\" ] && return\n\n# do not launch SSH Agent if we are inheriting an SSH_AUTH_SOCK from an 'ssh -A' agent forwarding connection\n[ -n \"${SSH_AUTH_SOCK:-}\" ] && return\n\nssh_agent\n"
  },
  {
    "path": ".bash.d/ssh.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     S S H\n# ============================================================================ #\n\n#ssh() { set -o xtrace ; command ssh \"$@\" <<< \"$(cat .bashrc_remote)\" ; }\n\nalias sshconfig='$EDITOR ~/.ssh/config'\nalias sshcfg=sshconfig\n\n# ssh-add\nssha(){\n    ssh_agent\n    #num_keys=\"$(ssh-add -l | grep -Ec \"(rsa|dsa)\")\"\n    #if [ \"$num_keys\" -lt 1 ]; then\n    #    ssh-add ~/.ssh/id_[rd]sa\n    #else\n    #    return 0\n    #fi\n    for key in ~/.ssh/id_[rd]sa; do\n        key_fingerprint=\"$(ssh-keygen -lf \"$key\" | awk '{print $2}')\"\n        if ! ssh-add -l | grep -Fq \"$key_fingerprint\"; then\n            ssh-add \"$key\"\n        fi\n    done\n}\n\nssh_func(){\n    ssha\n    if [ \"$1\" = \"ssh\" ] || [ \"$1\" = \"vagrant\" ]; then\n        local n=2\n        [ \"$1\" = \"vagrant\" ] && ((n+=1))\n        until [ \"$n\" -gt \"$#\" ]; do\n            case ${!n} in\n                -*) :\n                    ;;\n                 *) grep -Eq \"^[0-9]+$\" <<< \"${!n}\" || break\n                    ;;\n            esac\n            ((n+=1))\n        done\n    fi\n    local t=\"${!n}\" # indirect reference to variable number by evaluating\n    t=\"${t##*@}\"\n    t=\"${t%%.*}\"\n    title \"$t\"\n    command \"$1\" \"${@:2}\"\n    [ \"$1\" != \"scp\" ] && title \"$LAST_TITLE\"\n}\nalias ssh=\"ssh_func ssh\"\nalias sshl=\"ssh-add -l\"\nalias sshni=\"ssh_func ssh -oPreferredAuthentications=publickey -oStrictHostKeyChecking=no\"\nalias scp=\"ssh_func scp\"\nalias sftp=\"ssh_func sftp\"\n#alias sshc=\"ssh_custom\"\n\nsafe_ssh(){\n    if [ $# -lt 1 ]; then\n        echo \"usage: safe_ssh [user@]hostname\"\n        return 1\n    fi\n    host=\"${1##*@}\"\n    if grep -q '@' <<< \"$1\"; then\n        user=\"${1%%@*}\"\n    else\n        user=root\n    fi\n    #keys=`ssh -oStrictHostKeyChecking=no $@ 'for x in rsa dsa; do ssh-keygen -l -f /etc/ssh/ssh_host_${x}_key ; done' 2>&1`\n    #keys=`ssh -oStrictHostKeyChecking=no $@ 'for x in rsa dsa; do cat /etc/ssh/ssh_host_${x}_key ; done' 2>&1`\n    keys=\"$(ssh \"$user@$host\" 'for x in rsa dsa; do cat /etc/ssh/ssh_host_${x}_key.pub ; done')\"\n    rsa_key=\"$(echo \"$keys\" | awk '/ssh-rsa/ {print $1\" \"$2}')\"\n    dsa_key=\"$(echo \"$keys\" | awk '/ssh-dsa/ {print $1\" \"$2}')\"\n    known_key=\"$(grep \"${@##*@}\" ~/.ssh/known_hosts | awk '{print $2\" \"$3}' | sort -u)\"\n    echo\n    if [ \"$rsa_key\" = \"$known_key\" ]; then\n        echo \"OK: Host rsa key matches known key\"\n    elif [ \"$dsa_key\" = \"$known_key\" ]; then\n        echo \"OK: Host dsa key matches known key\"\n    else\n        echo 'WARNING: known ssh key for '\"$1\"' does not match either rsa or dsa keys obtained from server!!!'\n        echo \"Known Key: $known_key\"\n        echo \"RSA   Key: $rsa_key\"\n        echo \"DSA   Key: $dsa_key\"\n    fi\n}\nalias sssh=safe_ssh\n\ncheck_sshkey(){\n    for x in \"$@\"; do\n        grep \"$x\" ~/.ssh/known_hosts |\n        sort |\n        while read -r host id known_key; do\n            scanned_key=\"$(ssh-keyscan \"$host\" | awk \"/^$host $id / {print \\$3}\")\"\n            if [ \"$scanned_key\" != \"$known_key\" ]; then\n                echo -e \"\\\\nMISMATCH: $host\\\\nknown key: $known_key\\\\nscanned_key:$scanned_key\\\\n\\\\n\"\n            fi\n        done\n    done\n}\n\n#update_sshkey(){\n#    host=\"$1\"\n#    id=\"$2\"\n#    key=\"$3\"\n#    perl -pi -e \"s/^$host .*/$host $id $key/\" ~/.ssh/known_hosts\n#}\n\nressh(){\n    # FIXME: a cheat, it should be set properly but ssh in retry doesn't look like it's triggers ssh_func properly\n    title \"$1\"\n    retry ssh \"$@\"\n    title \"$LAST_TITLE\"\n}\n\nrissh(){\n    if [ $# -lt 1 ]; then\n        echo \"rissh <hostname>\"\n        return 1\n    fi\n    cleankey \"$@\"\n    ressh \"$@\" -oStrictHostKeyChecking=no\n}\n\nissh(){\n    if [ $# -lt 1 ]; then\n        echo \"issh <hostname>\"\n        return 1\n    fi\n    cleankey \"$@\"\n    ssh -oStrictHostKeyChecking=no \"$@\"\n}\n\nbouncessh(){\n    checkhost \"$1\" || return 1\n    title \"$1\"\n    whendown \"$1\"\n    ressh \"$@\"\n}\nalias bssh=bouncessh\n\nbouncerissh(){\n    checkhost \"$1\" || return 1\n    title \"$1\"\n    whendown \"$1\"\n    rissh \"$@\"\n}\nalias brissh=bouncerissh\nalias bissh=bouncerissh\n\nrekey(){\n    [ -n \"$1\" ] || { echo \"usage: rekey host\"; return 1; }\n    cleankey \"$1\"\n    ssh-keyscan -t rsa \"$1\" | grep \"^$1 ssh-rsa\" >> ~/.ssh/known_hosts\n    ssh-keyscan -t dsa \"$1\" | grep \"^$1 ssh-dss\" >> ~/.ssh/known_hosts\n}\n\nsshkey(){\n    local key=~/.ssh/id_rsa.pub\n    # now available on Mac, but my tried and tested function of years gone by dedupes the keys\n#    if type -P ssh-copy-id; then\n#        ssh-copy-id -i \"$key\" \"$@\"\n#    else\n        ssh \"$@\" '\n            umask 077;\n            [ -d ~/.ssh ] || mkdir -p ~/.ssh;\n            key=`cat`;\n            # my version is better than what ssh-copy-id did it would add duplicate keys\n            if ! grep \"$key\" ~/.ssh/authorized_keys >/dev/null 2>&1; then\n                echo $key >> ~/.ssh/authorized_keys;\n            fi;\n            chmod 0600 ~/.ssh/authorized_keys\n            # this line was the only advantage ssh-copy-id script had\n            test -x /sbin/restorecon && /sbin/restorecon ~/.ssh ~/.ssh/authorized_keys >/dev/null 2>&1 || true\n        ' < \"$key\"\n#    fi\n}\n\nsshkeygo(){\n    sshkey \"$@\"\n    ssh \"$@\"\n}\n\nsshkey2(){\n    sshkeygo \"$@\"\n}\n\ncleankey(){\n    if [ $# -lt 1 ]; then\n        echo \"usage: cleankey regex\"\n        return 1\n    fi\n    for x in \"$@\"; do\n        ssh-keygen -R \"$x\"\n        local aliasname\n        aliasname=\"$(host \"$x\" | awk '/is an alias for/ {print $6}')\"\n        if [ -n \"$aliasname\" ]; then\n            ssh-keygen -R \"$aliasname\"\n            ssh-keygen -R \"${aliasname%%.*}\"\n        fi\n        continue\n#        local ip\n#        ip=\"$(host -W 1 \"$x\" | grep address)\"\n#        if [ $? -eq 0 ]; then\n#            ip=\"$(cut -d\" \" -f 4 <<< \"$ip\")\"\n#            perl -pi -e 's/^\\[?[^,]+\\]?(:\\d+)?,\\[?'\"$ip\"'\\]?(:\\d)? .*$//;s/^'\"$ip\"' .*$//' ~/.ssh/known_hosts\n#        fi\n#        # need to leave second deletion just in case as you may want to specify just the ip address\n#        #perl -pi -e 's/^\\[?'\"$x\"'\\]?(:\\d+)?\\[,\\s.*$//;s/^.*[^,]+,'\"$x\"' .*$//' ~/.ssh/known_hosts\n#        perl -pi -e 's/^'\"$x\"'\\s.*$//;s/^.*[^,]+,'\"$x\"' .*$//' ~/.ssh/known_hosts\n    done\n}\n\nkeyremove(){\n    for x in \"$@\"; do\n        # shellcheck disable=SC1117\n        ssh -o \"PasswordAuthentication no\" \"$x\" '\n            for y in ~/.ssh/authorized_keys*; do\n                if [ -f \"$y\" ]; then\n                    perl -pi -e '\"'s/ssh-rsa .*= hari@.*\\n//'\"' \"$y\"\n                fi\n            done\n        '\n    done\n}\n\n#keyremoveall(){\n#    for x in \"$@\"; do\n#        for y in root hari oracle; do\n#            keyremove \"$y@$x\"\n#        done\n#    done\n#}\n"
  },
  {
    "path": ".bash.d/svn.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#            R e v i s i o n   C o n t r o l  -  S u b v e r s i o n\n# ============================================================================ #\n\n# I don't use SVN any more so a lot of the convenient aliases for daily use are commented out\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# superceded by hg and then git pull\n#alias u=\"svn up\"\n#alias sd=\"svn diff\"\n#alias svnl=svnlog\n#alias svnr=\"svn revert\"\n\n# remapped to screen\n#s(){ svn st \"$@\" | more; }\n#stx(){ svn st \"$@\" | grep -v \"^?\"; }\n\nisSvn(){\n    local target=${1:-.}\n    if [ -d \"$target/.svn\" ]; then\n        return 0\n    elif [ -f \"$target\" ] &&\n         [ -d \"$(dirname \"$target\")/.svn\" ]; then\n        return 0\n    else\n        return 1\n    fi\n}\n\nsvn(){\n    if is_mac; then\n        export stat_formatopt=f\n    else\n        export stat_formatopt=c\n    fi\n    local svn_owner\n    for x in \"$@\"; do\n        if [ -d \"$x/.svn\" ]; then\n            local dir_tested=true\n            svn_owner=\"$(stat -$stat_formatopt \"%u\" \"$x/.svn\")\"\n            if [ \"$EUID\" != \"$svn_owner\" ]; then\n                echo \"YOU ARE RUNNING SVN AS THE WRONG USER ON $x\"\n                return 1\n            fi\n        fi\n    done\n    if [ \"$dir_tested\" != \"true\" ]; then\n        if [ -d \"./.svn\" ]; then\n            svn_owner=\"$(stat -$stat_formatopt \"%u\" ./.svn)\"\n            if [ \"$EUID\" != \"$svn_owner\" ]; then\n                echo \"YOU ARE RUNNING SVN AS THE WRONG USER HERE\"\n                return 1\n            fi\n        fi\n    fi\n    command svn \"$@\"\n}\n\nsvnst(){\n    svn st \"$@\"\n}\n\nsvnkw(){\n    svn ps svn:keywords \"LastChangedBy LastChangedDate Revision URL Id\" \"$@\"\n}\n\nsvnadd(){\n    svn add -- \"$@\" &&\n    svnkw \"$@\"\n}\n\nsvni(){\n    svn pe svn:ignore -- \"${@:-.}\"\n}\n\nsvnaddci(){\n    svnadd \"$@\" &&\n    svn ci -m \"added $*\" -- \"$@\"\n}\n\nsvnci() {\n    local svncimsg=\"\"\n    for x in \"$@\"; do\n        if svn st \"$x\" | grep -q \"^[?A]\"; then\n            svncimsg+=\"$x, \"\n        fi\n    done\n    [ -z \"$svncimsg\" ] && return 1\n    svncimsg=\"${svncimsg%, }\"\n    svncimsg=\"added $svncimsg\"\n    svn add -- \"$@\" &&\n    svn ci -m \"$svncimsg\" \"$@\"\n}\n\nsvnrm(){\n    svn rm -- \"$@\" &&\n    svn ci -m \"removed $*\" -- \"$@\"\n}\n\nsvnrmf(){\n    svn rm --force -- \"$@\" &&\n    svn ci -m \"removed $*\" -- \"$@\"\n}\n\nsvnrename(){\n    svn up \"$(dirname \"$1\")\" \"$(dirname \"$2\")\"\n    svn mv -- \"$1\" \"$2\" &&\n    svn ci -m \"renamed $1 to $2\" -- \"$1\" \"$2\"\n}\n\nsvnrename2(){\n    local svn_url\n    svn_url=$(svn info \"$1\" | grep \"^URL: \" | sed 's/^URL: //')\n    [ -n \"$svn_url\" ] || return\n    svn rename -m \"renamed $1 to $2\" -- \"$svn_url\" \"$(dirname \"$svn_url\")/$2\"\n    svn up -- \"$1\" \"$2\"\n}\n\nsvnmkdir(){\n    svn mkdir --parents -- \"$@\" &&\n    svn ci -m \"created directory $*\" -- \"$@\"\n}\n\nsvnmv(){\n    svn up \"$(dirname \"$1\")\" \"$(dirname \"$2\")\"\n    svn mv -- \"$1\" \"$2\" &&\n    svn ci -m \"moved $1 to $2\" -- \"$1\" \"$2\"\n}\n\nsvnrevert(){\n    svn revert -- \"$@\"\n}\n\nsvnlog(){\n    local args=()\n    local args2=()\n    until [ $# -lt 1 ]; do\n        case \"$1\" in\n            -*) args+=(\"$1\")\n                ;;\n             *) args2+=(\"$1\")\n                ;;\n        esac\n        shift\n    done\n    svn up \"${args2[@]}\" &&\n    read -r -p \"press enter to see log\" &&\n    svn log \"${args[@]}\" \"${args2[@]}\" | less\n}\n\nsvnu(){\n    [ -n \"$1\" ] || { echo \"ERROR: must supply arg\"; return 1; }\n    [ \"$(svn diff -- \"$@\" | wc -l)\" -gt 0 ] || return\n    svn diff -- \"$@\" | more &&\n    read -r &&\n    svn ci -m \"updated $*\" -- \"$@\"\n}\n\nsvne(){\n    svn ps svn:executable on \"$@\"\n}\n\nsvnec(){\n    svne \"$@\";\n    for x in \"$@\"; do\n        svn ci -m \"set executable on $x\"\n    done\n}\n\nsvncommitauthors(){\n    svn log |\n    awk '/^r[[:digit:]]+[[:space:]]/ {print $3}' |\n    sort |\n    uniq -c |\n    sort -k1nr\n}\n\nrmnonsvn(){\n    svn st |\n    grep \"^?\" |\n    awk '{print $2}' |\n    xargs rm -f --\n}\n\nsvndifflast(){\n    local rev=HEAD\n    local rev_last=PREV\n    if grep -q '^[[:digit:]]\\+' <<< \"$1\"; then\n        rev=\"$1\"\n        shift;\n    fi\n    if [ \"$rev\" != \"HEAD\" ]; then\n        #let rev_last=$rev-1\n        (( rev_last = rev - 1 ))\n    fi\n    svn diff -r \"$rev_last:$rev\" -- \"$@\" |\n    more\n}\n#alias sdl=svndifflast\n\nsvndiff(){\n    local filename=\"${1:-}\"\n    [ -n \"$filename\" ] || { echo \"usage: svndiff filename\"; return 1; }\n    svn diff \"$filename\" > \"/tmp/svndiff.tmp\"\n    diffnet.pl \"/tmp/svndiff.tmp\"\n}\n\nsvndiffcumulative(){\n    local url\n    svn up\n    url=\"$(svn info | awk '/^URL/ {print $2}')\"\n    HEAD=\"$(svn info | awk '/Revision/ {print $2}')\"\n    for x in $(eval echo \"{25470..$HEAD}\"); do\n        ((y=x+1))\n        echo -n \"svn $x => $y: \"\n        svn diff -r -- \"$x:$y\" \"$url\"\n    done\n}\nalias svndiffcum=\"svndiffcumulative\"\n\nsvncommmitcount(){\n    svn up\n    svn log -r 25470:HEAD |\n    grep -E \"^r[[:digit:]]+ |\" |\n    wc\n}\n"
  },
  {
    "path": ".bash.d/teamcity.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-24 17:09:11 +0000 (Tue, 24 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                T e a m C i t y\n# ============================================================================ #\n\n# sets TeamCity URL to the local docker and finds and loads the current container's superuser token to the environment for immediate use with teamcity_api.sh\nteamcity_local(){\n    TEAMCITY_SUPERUSER_TOKEN=\"$(\n        # project name must match COMPOSE_PROJECT_NAME from teamcit.sh otherwise will fail to find token\n        docker-compose -p bash-tools -f \"$(dirname \"${BASH_SOURCE[0]}\")/../docker-compose/teamcity.yml\" \\\n            logs teamcity-server | \\\n        grep -E -o 'Super user authentication token: [[:alnum:]]+' | \\\n        tail -n1 | \\\n        awk '{print $5}' || :\n    )\"\n\n    export TEAMCITY_SUPERUSER_TOKEN\n    export TEAMCITY_URL=\"http://localhost:8111\"\n}\n"
  },
  {
    "path": ".bash.d/terraform.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-24 18:02:04 +0100 (Thu, 24 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               T e r r a f o r m\n# ============================================================================ #\n\nif ! [ -e ~/.tfenv/bin ] && is_mac; then\n    mkdir -p -v ~/.tfenv\n    tfenv_bin=\"$(find /usr/local/Cellar/tfenv -type d -name bin 2>/dev/null | head -n1)\"\n    if [ -d \"$tfenv_bin\" ]; then\n        ln -sfv -- \"$tfenv_bin\" ~/.tfenv/bin\n    fi\nfi\n\nadd_PATH ~/.tfenv/bin\n\n# don't get this wrong or you'll get this error:\n#\n#   ERRO[0000] fork/exec /Users/hari/.tfenv/bin: no such file or directory\n#   ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1\n#\nif [ -x ~/.tfenv/bin/terraform ]; then\n    #export TERRAGRUNT_TFPATH=~/.tfenv/bin/  # it's full path to binary executable not a search $PATH!\n    # neweer versions of Mac seems to not create ~/.tfenv/bin/ terraform link and instead use /opt/homebrew/bin/terraform link to the tfenv Cellar path\n    export TERRAGRUNT_TFPATH=~/.tfenv/bin/terraform\nelif ! [ -x \"$TERRAGRUNT_TFPATH\" ]; then\n    unset TERRAGRUNT_TFPATH\nfi\n\nexport TG_PROVIDER_CACHE=1\nexport TG_PROVIDER_CACHE_HOST=172.0.0.1\n\nalias tf=terraform\nalias tfp='tf plan'\nalias tfa='tf apply'\nalias tfip='tf init && tfp'\nalias tfia='tf init && tfa'\nalias tfaa='tfa -auto-approve'\nalias tfiaa='tfia -auto-approve'\n#complete -C /Users/hari/bin/terraform terraform\n\nunalias tffu &>/dev/null || :\ntffu(){\n    local lock_id=\"${1:-}\"\n    # self-determine the lock if not provided\n    if [ -z \"$lock_id\" ]; then\n        lock_id=\"$(terraform plan -input=false -no-color 2>&1 | grep -A 1 'Lock Info:' | awk '/ID:/{print $NF; exit}')\"\n    fi\n    if [ -z \"$lock_id\" ]; then\n        echo \"Failed to determine lock ID\" >&2\n        return 1\n    fi\n    terraform force-unlock -force \"$lock_id\"\n}\n\nalias tg=terragrunt\nalias tgp='tg plan'\nalias tga='tg apply'\nalias tgaa='tga -auto-approve'\nalias tgip='tg init && tgp'\nalias tgia='tg init && tga'\n\n# the fix for .terraform.lock.hcl:\n#\n#   the cached package for registry.terraform.io/hashicorp/aws 5.80.0 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file\n#\nalias tfprov='terraform  providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64'\nalias tgprov='terragrunt providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64'\n\nunalias tgfu &>/dev/null || :\ntgfu(){\n    local lock_id=\"${1:-}\"\n    # self-determine the lock if not provided\n    if [ -z \"$lock_id\" ]; then\n        lock_id=\"$(terragrunt plan -input=false -no-color 2>&1 | grep -A 1 'Lock Info:' | awk '/ID:/{print $NF; exit}')\"\n    fi\n    if [ -z \"$lock_id\" ]; then\n        echo \"Failed to determine lock ID\" >&2\n        return 1\n    fi\n    terragrunt force-unlock -force \"$lock_id\"\n}\n\nif [ -n \"${github:-}\" ]; then\n    for x in terraform-templates terraform tf; do\n        if [ -d \"$github/$x\" ]; then\n            # shellcheck disable=SC2139\n            alias tft=\"cd '$github/$x'\"\n            break\n        fi\n    done\nfi\n\n#generate_terraform_autocomplete(){\n#    local terraform_bin\n#    local terraform_version_number\n#\n#    for terraform_bin in ~/bin/terraform[[:digit:]]*; do\n#        [ -x \"$terraform_bin\" ] || continue\n#        terraform_version_number=\"${terraform_bin##*/terraform}\"\n#        # expand now\n#        # shellcheck disable=SC2139,SC2140\n#        alias \"tf${terraform_version_number}\"=\"$terraform_bin\"\n#        complete -C \"$terraform_bin\" terraform\n#        complete -C \"$terraform_bin\" tf\n#    done\n#\n#    terraform_bin=\"$(type -P terraform)\"\n#    complete -C \"$terraform_bin\" terraform\n#    complete -C \"$terraform_bin\" tf\n#}\n#\n#generate_terraform_autocomplete\n"
  },
  {
    "path": ".bash.d/title.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             T i t l e   M a g i c\n# ============================================================================ #\n\n# Sets the Screen and Terminal titles\n\nalias ti=\"title\"\n# static title - turn dynamic prompt escape codes off, while optionally setting new title\nalias sti=\"dpoff >/dev/null; ti\"\n\ntermtitle(){\n    # in tmux sets the secondary title at bottom right, duplicating info\n    if ! istmux; then\n        printf '\\033]0;%s\\007' \"${*:- }\"\n    fi\n}\n\nisscreen(){\n    # $STY is only set in screen it seems so this determines if we're in screen\n    [ -n \"${STY:-}\" ]\n}\n\nscreentitle(){\n    if isscreen; then\n        # shellcheck disable=SC1003\n        printf '\\033k%s\\033\\\\' \"${*:- }\"\n        # or\n        # screen -X title \"$*\"\n    fi\n}\n\nistmux(){\n    [ -n \"${TMUX:-}\" ]\n}\n\ntmuxtitle(){\n    if istmux; then\n        # window name appears in bottom left as a secondary name\n        #printf \"\\033]2;%s\\033\\\\\" \"${*:-}\"\n        # this is actually what we want to act like screen\n        tmux rename-window \"${*:-}\"\n    fi\n}\n\ntitle(){\n    export LAST_TITLE=\"$TITLE\"\n    #if [ $# -eq 0 ]; then\n    #    return\n    # various commands will reset the title after their commands so skip those calls if NO_SCREEN_UPDATES is set\n    if [ -z \"$*\" ] &&\n         [ \"$NO_SCREEN_UPDATES\" = \"1\" ]; then\n        return\n    fi\n    export TITLE=\"$*\"\n    termtitle \"$TITLE\"\n    screentitle \"$TITLE\"\n    tmuxtitle \"$TITLE\"\n}\n\n# .bashrc reload causes title loss if enabling this\n#title\n# so instead just reset the termtitle which we don't see anyway and get rid of that annoying Unnamed in the title bar\ntermtitle \" \"\n\n# ============================================================================ #\n\n# toggle dynamic prompt on / off\ndpstatus(){\n    if grep -q '^\\\\\\[\\\\ek\\\\e\\\\\\\\\\\\]' <<< \"$PS1\"; then\n        echo \"enabled\"\n        return 0\n    else\n        echo \"disabled\"\n        return 1\n    fi\n}\n\n# dynamic prompt escape codes off\ndpoff(){\n    if dpstatus >/dev/null; then\n        title \" \"\n        PS1=\"${PS1#\\\\[\\\\ek\\\\e\\\\\\\\\\\\\\]}\"\n        export PS1\n        export NO_SCREEN_UPDATES=1\n        echo \"disabled\"\n    fi\n}\n\n# dynamic prompt escape codes on\ndpon(){\n    if ! dpstatus >/dev/null; then\n        PS1=\"${SCREEN_ESCAPE}${PS1}\"\n        export PS1\n        export NO_SCREEN_UPDATES=0\n        echo \"enabled\"\n    fi\n}\n\n# toggle dynamic prompt on/off\ndp(){\n    if dpstatus; then\n        printf '\\b\\renabled => '\n        dpoff\n    else\n        printf '\\b\\rdisabled => '\n        dpon\n    fi\n    title\n}\n\n# ============================================================================ #\n\nman(){\n    title \"man $1\"\n    command man \"$@\"\n    title \"$LAST_TITLE\"\n}\n\nsudo(){\n    title \"sudo $1\"\n    command sudo \"$@\"\n    title \"$LAST_TITLE\"\n}\n\nvim(){\n    local title=\"\"\n    #until [ -z \"$1\" ]; do\n    for x in \"$@\"; do\n        case \"$x\" in\n            -*) :\n                ;;\n            +*) :\n                ;;\n             *) #title=\"$title$x \"\n                title=\"$x\"\n                break\n                ;;\n        esac\n        #shift\n    done\n    local num=10\n    if [[ \"${TITLE_SHORT:-}\" =~ ^[0-9]+$ ]]; then\n        num=$TITLE_SHORT\n    fi\n    if [ \"$num\" -lt 3 ]; then\n        num=3\n    fi\n    title=\"${title//.txt/}\"\n    if dpstatus >/dev/null; then\n        if echo \"$title\" | grep -q docs/; then\n            title=\"$(basename \"$title\")\"\n            title \"d${title:0:$num}\"\n        else\n            title=\"$(basename \"$title\")\"\n            title \"${title:0:$num}\"\n        fi\n    fi\n    command vim \"$@\"\n    #if dpstatus >/dev/null; then title; fi\n}\n"
  },
  {
    "path": ".bash.d/travis_ci.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Wed Jan 20 15:28:12 2016 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               T r a v i s   C I\n# ============================================================================ #\n\n# Travis bash autocomplete\n# adapted from Travis ruby gem auto-added to end of ~/.bashrc\n# shellcheck disable=SC1090,SC1091\n#[ -f ~/.travis/travis.sh ] && source ~/.travis/travis.sh\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\ntype git_repo &>/dev/null || . \"$bash_tools/.bash.d/git.sh\"\ntype browse &>/dev/null || . \"$bash_tools/.bash.d/network.sh\"\n\nalias trav='travis_browse'\n\ntravis_browse(){\n    local repo\n    repo=\"$(github_repo)\"\n    url=\"https://travis-ci.org/${TRAVIS_USER:-${USER:-$(whoami)}}/$repo\"\n    browser \"$url\"\n}\n\n# for auto authentication using Travis CI tools like travis_last_log.py and travis_debug_session.py\n#export TRAVIS_TOKEN=...\n\ntravis_debug(){\n    # code has better automatic handling, doesn't need this now\n    #if grep '/' <<< \"$1\" &>/dev/null; then\n    #    travis_debug_session.py -r \"$1\" ${@:2}\n    #else\n    #    travis_debug_session.py -J \"$1\" ${@:2}\n    #fi\n    opts=()\n    if [[ \"$PWD\" =~ /github/ ]]; then\n        local repo\n        repo=\"$(git_repo)\"\n        if [ -n \"$repo\" ]; then\n            opts+=(--repo \"$repo\")\n        fi\n    fi\n    travis_debug_session.py \"${opts[@]}\" \"$@\"\n}\n\ntravis_log(){\n    local repo\n    repo=\"$(git_repo)\"\n    travis_last_log.py --failed \"$repo\" \"$@\"\n}\n"
  },
  {
    "path": ".bash.d/trivy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-12 06:50:49 +0100 (Fri, 12 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   T r i v y\n# ============================================================================ #\n\n#bash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#type add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n#autocomplete trivy\n"
  },
  {
    "path": ".bash.d/vagrant.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Thu Mar 14 12:42:17 2013 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 V a g r a n t\n# ============================================================================ #\n\n#export VAGRANT_HOME=~/vagrant\n#export VAGRANT_BOXES=~/boxes\n\n# $bash_tools defined in adjacent shell modules\n# shellcheck disable=SC2154\nexport vagrant=\"$bash_tools/vagrant\"\n\n#alias cd_vagrant='[ \"$PWD\" = \"$VAGRANT_HOME\" ] || cd \"$VAGRANT_HOME\"'\n#alias cd_vagrant='cd \"$VAGRANT_HOME\"'\nalias cd_vagrant='cd \"$vagrant\"'\nalias cdv=cd_vagrant\nalias vhome=cd_vagrant\n#alias v='cd_vagrant; vagrant'\n# 'v' is aliased to vim now as it's used much more often\n#alias v='vagrant'\nalias vag='vagrant'\n#alias vf='cd_vagrant; vim Vagrantfile; vagrant_gen_etc_hosts; eval \"$(vagrant_gen_aliases)\"'\nalias vst='vagrant status'\nalias vrun='vst | grep running'\n# vr is aliased to vbox_running in virtualbox.sh\nalias vrr='vrun'\n#alias vssh='cd_vagrant; ssh_func vagrant ssh'\n#alias vssh='cd_vagrant; vagrant ssh'\nalias vssh='vagrant ssh'\nalias boxes='cd $VAGRANT_BOXES'\n\nwhile read -r directory; do\n    # shellcheck disable=SC2139,SC2140\n    alias \"v${directory##*/}\"=\"cd $directory\"\ndone < <(find \"$vagrant\" -maxdepth 1 -type d)\n\n# see ../vagrant_hosts.sh for similar parsing\n#\n#vagrant_parse_hosts(){\n#    if ! [ -f \"$VAGRANT_HOME/Vagrantfile\" ]; then\n#        return\n#    fi\n#    #grep '[^#]*config.vm.define' \"$VAGRANT_HOME/Vagrantfile\" | awk -F'\"' '{print $2}'\n#    sed 's/#.*//;/^[[:space:]]*$/d' \"$VAGRANT_HOME/Vagrantfile\" |\n#    grep -e host_name -e network |\n#    grep -B1 -e network |\n#    grep -v -e \"^--\" -e \"default_hostname\" |\n#    sed 's/^.*[[:space:]]\"//;s/\"//' |\n#    tr '\\n' ' ' |\n#    perl -pn -e 's/(\\.\\d+)\\s/$1\\n/g'\n#}\n#\n#vagrant_gen_aliases(){\n#    vagrant_parse_hosts |\n#    while read -r host ip rest; do\n#        if ! type \"$host\" &>/dev/null; then\n#            #echo \"alias $host='ssh root@$host'\"\n#            #echo \"alias $host='vups $host'\"\n#            echo \"alias $host='vssh $host'\"\n#        fi\n#    done\n#}\n#\n# don't really use these aliases to vagrant boxes any more\n#eval \"$(vagrant_gen_aliases)\"\n\n#vagrant_gen_etc_hosts(){\n#    vagrant_parse_hosts |\n#    while read -r host ip rest; do\n#        if [ -n \"${rest:-}\" ]; then\n#            echo \"error third token '$rest' detected from pipe line '$ip $host $rest'\"\n#            return 1\n#        fi\n#        [ \"${host}\" = \"localhost\" ] && continue\n#        host_record=\"$ip $host.local $host\"\n#        # sudo auto-defined in .bashrc\n#        # shellcheck disable=SC2154\n#        $sudo perl -pi -e \"s/^$ip\\\\s+.*/$host_record/\" /etc/hosts\n#        if [ -n \"$ip\" ]; then\n#            grep -q \"^$host_record\" /etc/hosts ||\n#            $sudo sh -c \"echo '$host_record' >> /etc/hosts\"\n#        else\n#            echo \"no ip passed in pipe for host '$host'! \"\n#            return 1\n#        fi\n#    done\n#}\n\nvagrant_usage(){\n    if [ -z \"$1\" ]; then\n        echo \"usage: ${FUNCNAME[1]} <vm_name>\"\n        return 1\n    fi\n}\n\nvup(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant up \"$@\"\n}\n\nvre(){\n    vagrant_usage \"$1\" || return 1\n    vagrant reload \"$@\"\n}\n\nvressh(){\n    vagrant_usage \"$1\" || return 1\n    vagrant reload \"$1\"\n    vagrant ssh \"$1\"\n}\n\nvsus(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant suspend \"$@\"\n}\n\nvres(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant resume \"$@\"\n}\n\n#alias vsusall=\"vsus $(vst | grep running | awk '{print $1}')\"\nvsusall(){\n    local running_vms=()\n    read -r -a running_vms <<< \"$(vst | grep running | awk '{print $1}')\"\n    [ ${#running_vms} -gt 0 ] || return 0\n    vsus \"${running_vms[@]}\"\n}\nalias vsusa=vsusall\n\nvupssh(){\n    vagrant_usage \"$1\" || return 1\n    local status\n    status=\"$(vst \"$1\")\"\n    if grep -Eq \"^$1[[:space:]]\" <<< \"$status\"; then\n        grep -Eq \"^$1[[:space:]]+running\" <<< \"$status\" || vup \"$1\"\n    else\n        echo \"VM not found: $1\"\n        return 1\n    fi\n    #vup $1\n    vssh \"$1\"\n}\nalias vups=\"vupssh\"\n\nvhalt(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant halt \"$@\"\n}\n\nvrhalt(){\n    # want splitting\n    # shellcheck disable=SC2046\n    vhalt \"$@\" $(vagrant status | awk '/running/ {print $1}')\n}\n\nvrsus(){\n    # want splitting\n    # shellcheck disable=SC2046\n    vsus $(vagrant status | awk '/running/ {print $1}')\n}\n\nvdestroy(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant destroy --force \"$@\"\n}\n\nvdestroyup(){\n    vdestroy \"$@\" || :\n    vup \"$@\"\n}\n\nvdestroyups(){\n    vdestroy \"$@\" || :\n    vups \"$@\"\n}\n\nvprovision(){\n    #vagrant_usage \"$1\" || return 1\n    vagrant provision \"$@\"\n}\n\nvprov(){\n    vprovision \"$@\"\n}\n\nwhenvdown(){\n    vst |\n    while grep \"$1.*running\"; do\n        sleep 0.1\n    done\n}\n"
  },
  {
    "path": ".bash.d/vim.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc and later functions.sh)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     V I M\n# ============================================================================ #\n\nbash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# vim() function is in title.sh as it is tightly integrated with the functions in there and not necessary otherwise\n\nvimhome(){\n    # must expand in vim, not in shell\n    # shellcheck disable=SC2016\n    vim -Nesc '!echo $VIMRUNTIME' -c qa |\n    #tr -dc '[:alnum:]/\\r\\n'\n    sed 's,[^/]*/,/,'\n}\n\ncdvimhome(){\n    # shellcheck disable=SC2164\n    cd \"$(vimhome)\"\n}\n\nvimfiletypes(){\n    cdvimhome || return 1\n    find syntax ftplugin -iname '*.vim' -exec basename -s .vim {} + | sort -u\n}\n\ngitgrepvim(){\n    if [ $# -lt 2 ]; then\n        echo \"usage: gitgrepvim <pattern>\"\n        return 3\n    fi\n    # want splitting\n    # shellcheck disable=SC2046\n    vim $(git grep -i \"$*\" | sed 's/:.*//' | sort -u)\n}\nalias ggrepv=gitgrepvim\n\ngrepvim(){\n    if [ $# -lt 2 ]; then\n        echo \"usage: grepvim <pattern> <files>\"\n        return 3\n    fi\n    # want splitting\n    # shellcheck disable=SC2046\n    vim $(grep -l \"$1\" \"$@\" | sort -u)\n}\nalias grepv=grepvim\nalias vimgrep=grepvim\n\nvimchanged(){\n    local git_root\n    git_root=\"$(git_root)\"\n    # want splitting\n    # shellcheck disable=SC2046\n    vim \"$@\" $(git status --porcelain | awk '/^.M/ {$1=\"\"; print}' | sed \"s|^[[:space:]]|$git_root/|\")\n}\n\nfilesvim(){\n    local files=()\n    # mapfile not available on Mac and read -a only takes first result\n    # shellcheck disable=SC2207\n    IFS=$'\\n' files=($(find . -iname \"$@\" | sort -u))\n    if [ -n \"${files[*]}\" ]; then\n        vim \"${files[@]}\"\n    fi\n}\nalias fvim=filesvim\nalias vimf=fvim\n\n# vim which\nvw(){\n    local path\n    if [ -z \"$1\" ]; then\n        echo \"usage: vw <filename>\"\n        return 1\n    fi\n    path=\"$(which \"$1\")\"\n    if [ -z \"$path\" ]; then\n        echo \"File not found in \\$PATH: $1\"\n        return 1\n    fi\n    \"$EDITOR\" \"$path\"\n}\n\nvihosts(){\n    [ $EUID -eq 0 ] && sudo=\"\" || sudo=sudo\n    $sudo vim /etc/hosts\n    $sudo pkill -1 dnsmasq\n}\n\nvimup(){\n    local arg=\"$1\"\n    up_target=\"$(findup \"$arg\")\"\n    [ -n \"$up_target\" ] || return 1\n    vim \"$(findup \"$1\")\"\n}\n"
  },
  {
    "path": ".bash.d/virtualbox.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-01-14 22:13:51 +0000\n#  Original: circa 2011 - 2012\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              V i r t u a l B o x\n# ============================================================================ #\n\nvboxmanage_list_vms=\"VBoxManage list runningvms\"\n# shellcheck disable=SC2139\nalias vbox_running=\"echo $vboxmanage_list_vms; $vboxmanage_list_vms\"\nalias vr=vbox_running\nunset vboxmanage_list_vms\n\nalias startvm=\"VBoxManage startvm\"\nalias stopvm=\"VBoxManage controlvm acpipowerbutton\"\nalias poweroff=\"VBoxManage controlvm poweroff\"\nalias savestate=\"VBoxManage savestate\"\nalias controlvm=\"VBoxManage controlvm\"\n\n#docker(){\n#    local vm='boot2docker-vm'\n#    VBoxManage list runningvms | grep -q \"$vm\" || VBoxManage startvm \"$vm\"\n#    command docker \"$@\"\n#}\n\n# fixvbox() in .bash.d/apple.sh restarts VirtualBox on Mac OSX only\n\nfixvboxnet(){\n    sudo ifconfig vboxnet0 down\n    sudo ifconfig vboxnet0 up\n}\n"
  },
  {
    "path": ".bash.d/vnc.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     V N C\n# ============================================================================ #\n\n#bash_tools=\"${bash_tools:-$(dirname \"${BASH_SOURCE[0]}\")/..}\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$bash_tools/.bash.d/os_detection.sh\"\n\nif [ -d \"/Applications/VNC Viewer.app/Contents/MacOS\" ]; then\n    export PATH+=\":/Applications/VNC Viewer.app/Contents/MacOS\"\nfi\n\nvncwho() {\n    netstat -tW |\n    grep \".*:5900 .*:.*\" |\n    awk '{a=$5; split(a,b,\":\"); print b[1]}'\n}\n\nvnc(){\n    local host_port=\"$1\"\n    # if just a port number is given, then it's shorthand for localhost:<port_number> eg. for copying and pasting Qemu's randomly generated VNC port\n    if [[ \"$host_port\" =~ ^[[:digit:]]+$ ]]; then\n        host_port=\"localhost:$1\"\n    fi\n    if test -x \"/Applications/VNC Viewer.app/Contents/MacOS/vncviewer\"; then\n        \"/Applications/VNC Viewer.app/Contents/MacOS/vncviewer\" \"$host_port\" &\n    elif type -P krdc &>/dev/null; then\n        krdc \"vnc:/$host_port\" &\n    elif type -P vncviewer &>/dev/null; then\n        vncviewer \"$host_port\" &\n    else\n        echo \"could not find krdc or vncviewer in \\$PATH\"\n        return 1\n    fi\n}\n\nrevnc(){\n    local host_port=\"$1\"\n    local host=\"$host_port\"\n    if [[ \"$host\" =~ : ]]; then\n        host=\"${host%%:*}\"\n    fi\n    if [ -z \"$1\" ]; then\n        echo \"You must supply a hostname or ip address to connect to\"\n        return 1\n    fi\n    # $pingwait is defined in network.sh\n    # shellcheck disable=SC2154\n    while ! ping -c 1 \"$pingwait\" 1 \"$host\" &>/dev/null; do\n        sleep 1\n    done\n    timestamp \"machine is up\"\n    until vnc \"$host_port\"; do\n        sleep 1\n        timestamp \"retrying $host_port\"\n    done\n}\n"
  },
  {
    "path": ".bash.d/welcome.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-06-25 15:20:39 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 W e l c o m e\n# ============================================================================ #\n\n# Original version was in Perl many years ago, but defaulting to Python version now\n\n# Bash version further down is for interest of if you don't have the other repos\n\n# welcome should be found in $PATH from DevOps-Golang-Tools repo\n# welcome.py should be found in $PATH from DevOps-Python-Tools repo\nwelcome(){\n    if type -P welcome &>/dev/null; then\n        command welcome\n    elif type -P welcome.py &>/dev/null; then\n        welcome.py\n    fi\n}\n\n# set this instead to use bash only version if you don't have the other repos\n#alias welcome=bash_welcome\n\nbash_welcome(){\n    local msg\n    msg=\"Welcome Hari - your last access was $(last|head -n2|tail -n1|sed 's/[^ ]\\+ \\+[^ ]\\+ \\+[^ ]\\+ \\+//;s/ *$//')\"\n    #local msg=\"Welcome Hari\"\n    # generated by for x in {A..z}; do printf \"%s\" $x; done\n    #charmap=\"ABCDEFGHIJKLMNOPQRSTUVWXYZ[]^_\\`abcdefghijklmnopqrstuvwxyz\"\n    # generated by: for x in {1..128}; do printf \\\\$(printf '%03o' $x); done\n    # shellcheck disable=SC1117\n    charmap=\"!\\\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_\\`abcdefghijklmnopqrstuvwxyz{|}~ \\t\\n\"\n    for ((i=0; i<\"${#msg}\"; i++)); do\n        local x=\"${msg:i:1}\"\n        #echo \"x == $x\"\n        printf \" \"\n        local j=0\n        while true; do\n        #for ((j=0; j<\"${#charmap}\"; j++)); do\n        #while true; do\n            #set -x\n            if [ $j -gt 2 ]; then\n                local y=$x\n            else\n                local y=${charmap:$((RANDOM%${#charmap})):1}\n            fi\n            #local y=\"${charmap:j:1}\"\n            printf \"\\\\b%s\" \"$y\"\n            # This does not have enough precision, re-implement in Perl\n            # This is because it's an external being called, otherwise pure bash\n            # is so fast that you don't see any effect...\n            sleep 0.000000000001\n            #perl -e 'sleep 0.0000000000000000000000000001'\n            [ \"$y\" = \"$x\" ] && break\n            ((j+=1))\n            #set +x\n        done\n    done\n    #printf \"\\\\n\"\n    printf \"\\\\n\"\n}\n"
  },
  {
    "path": ".bash.d/when.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2012 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          W h e n   F u n c t i o n s\n# ============================================================================ #\n\n# interactive time latch alternative to 'at' command\nwhen(){\n    # should be in the format HH:MM:SS\n    local time=\"$1\"\n    shift || :\n    if ! grep -Eq \"^[012]?[0-9]:[0-5]?[0-9]:[0-5]?[0-9]$\" <<< \"$time\"; then\n        echo \"invalid time format - must be in format HH:MM:SS\"\n        return 1\n    fi\n    while true; do\n        if [ \"$(date '+%T')\" = \"$time\" ]; then\n            break\n        fi\n        sleep 1\n    done\n    \"$@\"\n}\n\nwhenup(){\n    local host=\"$1\"\n    shift || :\n    checkhost \"$host\" || return 1\n    local count=0\n    # defined in network.sh\n    # shellcheck disable=SC2154\n    while ! ping -c 1 \"$pingwait\" 1 \"$host\" &>/dev/null; do\n        ((count+=1))\n        timestamp \"waiting for $host to come up...\"\n        if [ $count -gt 22 ]; then\n            sleep 10\n        else\n            sleep 5\n        fi\n    done\n    timestamp \"$host is up\"\n    \"$@\"\n}\n\n# HTTP(s) version of whenup because corporate firewalls block ping\nwhenurl(){\n    local url=\"$1\"\n    shift || :\n    local count=0\n    while ! curl -s --connect-timeout 2 \"$url\" &>/dev/null; do\n        ((count+=1))\n        timestamp \"waiting for $url to come up...\"\n        if [ $count -gt 22 ]; then\n            sleep 10\n        else\n            sleep 5\n        fi\n    done\n    timestamp \"$url is up\"\n    \"$@\"\n}\n\nwhendown(){\n    local host=\"$1\"\n    shift || :\n    checkhost \"$host\" || return 1\n    local count=0\n    while ping -c 1 \"$pingwait\" 1 \"$host\" &>/dev/null; do\n        ((count+=1))\n        timestamp \"waiting for machine to go down...\"\n        if [ $count -gt 22 ]; then\n            sleep 10\n        else\n            sleep 5\n        fi\n    done\n    timestamp \"machine is down\"\n    \"$@\"\n}\n\nwhenport(){\n    local host=\"$1\"\n    local port=\"$2\"\n    shift || :\n    shift || :\n    checkhost \"$host\" || return 1\n    local count=0\n    timestamp \"checking port $port open...\"\n    checkprog nc\n    while ! nc -z \"${host#*@}\" \"$port\" &>/dev/null; do\n        ((count+=1))\n        timestamp \"waiting for port $port to open...\"\n        if [ $count -gt 22 ]; then\n            sleep 10\n        else\n            sleep 5\n        fi\n    done\n    timestamp \"port $port is open\"\n    \"$@\"\n}\n\nwhendone(){\n    local search=\"$1\"\n    shift || :\n    if [ -z \"$search\" ]; then\n        echo \"usage: when <search_for_prog_disappearing> <cmd>\"\n        return 1\n    fi\n    while true; do\n        if ! pgrep -qf \"$search\"; then\n            echo\n            break\n        else\n            echo -n .\n            sleep 1\n        fi\n    done\n    \"$@\"\n}\n"
  },
  {
    "path": ".bash.d/z_final.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            F i n a l i z a t i o n\n# ============================================================================ #\n\n# place to add last minute clean ups like path deduping\n\n# slows down new shells\n#dedupe_paths\n"
  },
  {
    "path": ".bash_logout",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             B a s h   L o g o u t\n# ============================================================================ #\n\n# read all history lines not already read from the history file and append them to the history list\nhistory -n\n\n# destroy kerberos tickets in all caches (-A), quietly don't beep (-q)\nkdestroy -A -q\n\nclear\n\n# From Ubuntu\n## when leaving the console clear the screen to increase privacy\n#\n#if [ \"$SHLVL\" = 1 ]; then\n#    [ -x /usr/bin/clear_console ] && /usr/bin/clear_console -q\n#fi\n"
  },
  {
    "path": ".bash_profile",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#\n\n# ~/.bash_profile: executed by bash(1) for login shells.\n# see /usr/share/doc/bash/examples/startup-files for examples.\n# the files are located in the bash-doc package.\n\ntrap clear EXIT\n\n# the default umask is set in /etc/login.defs\n#umask 022\n\nif [ -f ~/.bashrc ]; then\n    . ~/.bashrc\nfi\n\n# not supported in the tmux terminal in GCP Cloud Shell\n#if ! is_mac &&\n#   ! isGoogleCloudShell &&\n#   [ \"${TERM:-}\" != \"xterm-256color\" ]; then\n#    setterm -blank 0\n#fi\n\n# prints a cool spinning welcome message which shows the time of last login\n# this is available in the Python / Perl and Golang Devops Tools repos,\n# as well as a function in .bash.d/welcome.sh in the DevOps Bash Tools repo\n#if type welcome &>/dev/null; then\n#    welcome\n#fi\n\n#eval \"$(rbenv init -)\"\n\n# from brew install bash-completion\n[[ -r \"/usr/local/etc/profile.d/bash_completion.sh\" ]] && . \"/usr/local/etc/profile.d/bash_completion.sh\"\n\n#sudo setmixer -V pcm 100\n\ncomplete -C /usr/local/bin/terragrunt terragrunt\n\n# ============================================================================ #\n# This should be automatically added to ~/.bash_profile when you install SDKman (install/install_sdkman.sh):\n#\n#THIS MUST BE AT THE END OF THE FILE FOR SDKMAN TO WORK!!!\nexport SDKMAN_DIR=\"$HOME/.sdkman\"\n[[ -s \"/Users/<user>/.sdkman/bin/sdkman-init.sh\" ]] && source \"/Users/<user>/.sdkman/bin/sdkman-init.sh\"\n"
  },
  {
    "path": ".bashrc",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC1091\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                     BASH - Heavily Customized Environment\n# ============================================================================ #\n\n# Sources thousands of lines of Bash code written over the course of ~15+ years\n# some of which is now found in this GitHub repo's .bash.d/*.sh\n\n# ============================================================================ #\n#\n# put this at the top of your ~/.bashrc to inherit the goodness here (assuming you've checked out this repo to ~/github/bash-tools):\n#\n#   if [ -f ~/github/bash-tools/.bashrc ]; then\n#       . ~/github/bash-tools/.bashrc\n#   fi\n#\n# ============================================================================ #\n\n\n# Use with PS4 further down + profile-bash.pl (still in private repos) for performance profiling this bashrc\n#set -x\n\n# If not running interactively, don't do anything:\n[ -z \"${PS1:-}\" ] && return\n\n[ -n \"${PERLBREW_PERL:-}\" ] && return\n\n# Another alternative\n#case $- in\n#   *i*) ;;\n#     *) return 0;;\n#esac\n\n# Another variation\n#if [[ $- != *i* ]] ; then\n#    # Shell is non-interactive.  Be done now!\n#    return\n#fi\n\n# ============================================================================ #\n\n# after cleanshell, not even $HOME is set, this messes up things that base off $HOME, like SDKman\nif [ -z \"${HOME:-}\" ]; then\n    export HOME=~\nfi\n\nbash_tools=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n# needed to inherit by things like vim script execution from within files using libraries rooted at this location\nexport bash_tools\n\n# shellcheck disable=SC1090,SC1091\n. \"$bash_tools/.bash.d/os_detection.sh\"\n\n# enable color support for ls\nif [ \"$TERM\" != \"dumb\" ] && \\\n   ! is_mac; then\n    eval \"$(dircolors -b)\"\nfi\n\n# shut up Mac, Bash still rocks\nexport BASH_SILENCE_DEPRECATION_WARNING=1\n\n# ============================================================================ #\n\n# technically should get called only for new login shells\n#[ -f /etc/profile     ] && . /etc/profile\n[ -f /etc/bash/bashrc ] && . /etc/bash/bashrc\n[ -f /etc/bashrc      ] && . /etc/bashrc\n\n[ -f /etc/bash_completion ] && . /etc/bash_completion\n\n[ -x /usr/bin/lesspipe ] && eval \"$(SHELL=/bin/sh lesspipe)\"\n\n# shellcheck disable=SC1090,SC1091\n[ -f \"$HOME/.aliases\" ] && source \"$HOME/.aliases\"\n\n# ============================================================================ #\n\n# SECURITY TO STOP STUFF BEING WRITTEN TO DISK\n#unset HISTFILE\n#unset HISTFILESIZE\nexport HISTSIZE=50000\nexport HISTFILESIZE=50000\n\nrmhist(){ history -d \"$1\"; }\nhistrm(){ rmhist \"$1\"; }\nhistrmlast(){ history -d \"$(history | tail -n 2 | head -n 1 | awk '{print $1}')\"; }\n\n# This adds a time format of \"YYYY-mm-dd hh:mm:ss  command\" to the bash history\nexport HISTTIMEFORMAT=\"%F %T  \"\n\n# stop logging duplicate successive commands to history\n#HISTCONTROL=ignoredups:ignorespace\nHISTCONTROL=ignoredups\n\n# Neat trick \"[ \\t]*\" to exclude any command by just prefixing it with a space. Fast way of going stealth for pw entering on cli\n# & here means any duplicate patterns, others are simple things like built-ins and ls and stuff you don't need history for\n#export HISTIGNORE=\"[ \\t]*:&:ls:[bf]g:exit\"\n\n# append rather than overwrite history\nshopt -s histappend\n\n# check window size and update $LINES and $COLUMNS after each command\nshopt -s checkwinsize\n\nshopt -s cdspell\n\n# prevent core dumps which can leak sensitive information\nulimit -c 0\n\n# tighten permissions except for root where library installations become inaccessible to my user account\nif [ $EUID = 0 ]; then\n    umask 0022\nelse\n    # caused no end of problems when doing sudo command which retained 0077 and broke library access for user accounts\n    #umask 0077\n    umask 0022\nfi\n\n# make less more friendly for non-text input files, see lesspipe(1)\n[ -x /usr/bin/lesspipe ] && eval \"$(SHELL=/bin/sh lesspipe)\"\n\n# ============================================================================ #\n\nsudo=sudo\nif [ $EUID -eq 0 ]; then\n    # used throughout .bash.d/*.sh\n    # shellcheck disable=SC2034\n    sudo=\"\"\nfi\n\n# shellcheck disable=SC1090,SC1091\ntype add_PATH &>/dev/null || . \"$bash_tools/.bash.d/paths.sh\"\n\n# ============================================================================ #\n\n# want this to fail is there is no match because we should always have local .bash.d/*.sh in this repo\n# shopt -s nullglob\nfor src in \"$bash_tools/.bash.d/\"*.sh; do\n    # shellcheck disable=SC1090,SC1091\n    . \"$src\"\ndone\n# shopt -u nullglob\n\n# added by travis gem - should be in ~/.bashrc so not needed to duplicate here\n#[ -f /Users/hari.sekhon/.travis/travis.sh ] && source /Users/hari.sekhon/.travis/travis.sh\n\n# shellcheck disable=SC1090,SC1091\n[ -f \"$HOME/.bashrc.local\" ] && . \"$HOME/.bashrc.local\"\nif [ -d \"$HOME/.bash.d\" ]; then\n    shopt -s nullglob\n    for src in \"$HOME/.bash.d/\"*.sh; do\n        # shellcheck disable=SC1090,SC1091\n        . \"$src\"\n    done\n    shopt -u nullglob\nfi\nif [ -d \"$HOME/.bash.autocomplete.d\" ]; then\n    shopt -s nullglob\n    for src in \"$HOME/.bash.autocomplete.d/\"*.sh; do\n        # shellcheck disable=SC1090,SC1091\n        . \"$src\"\n    done\n    shopt -u nullglob\nfi\n"
  },
  {
    "path": ".buildkite/pipeline.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-13 21:10:39 +0000 (Fri, 13 Mar 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            B u i l d K i t e   C I\n# ============================================================================ #\n\n# BuildKite Pipeline\n#\n# add this command to the UI and it will read the rest of the steps from here:\n#\n# - command: buildkite-agent pipeline upload\n\n# Yaml Anchors workaround to BuildKite's lack of global retries configuration - credit to Jason @ BuildKite for this workaround:\n #\n# https://forum.buildkite.community/t/reschedule-builds-on-other-agents-rather-than-fail-builds-when-agents-time-out-or-are-killed-machine-shut-down-or-put-to-sleep/1388/5\n#\nanchors:\n  std_retries: &std_retries\n    retry:\n      automatic:\n        - exit_status: -1  # Agent was lost\n          limit: 2\n        - exit_status: 255 # Forced agent shutdown\n          limit: 2\n\nsteps:\n  - command: setup/ci_bootstrap.sh\n    label: ci bootstrap\n    timeout: 30  # brew can take 10 mins just to do a brew update\n    branches: master\n    <<: [*std_retries]\n  - wait\n  - command: make init\n    label: init\n    timeout: 2\n    branches: master\n    <<: [*std_retries]\n  - wait\n  - command: make ci\n    label: build\n    timeout: 60\n    branches: master\n    <<: [*std_retries]\n  - wait\n  - command: make test\n    label: test\n    timeout: 120\n    branches: master\n    <<: [*std_retries]\n"
  },
  {
    "path": ".circleci/config.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-23 23:30:14 +0000 (Sun, 23 Feb 2020)\n#  Original: H1 2016 (Circle CI 1.x)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                C i r c l e C I\n# ============================================================================ #\n\n# Master Template with more advanced config:\n#\n#   https://github.com/HariSekhon/Templates/blob/master/circleci_config.yml\n\n# Reference:\n#\n#   https://circleci.com/docs/2.0/configuration-reference\n\nversion: 2.1\n\nworkflows:\n  version: 2\n  workflow:\n    jobs:\n      - build\n\njobs:\n  build:\n    docker:\n      - image: cimg/base:2024.12\n    resource_class: small\n    steps:\n      # CLI is too old - config validate breaks in test - install new version to fix\n      # doesn't work - existing version is too old to update\n      #- run: circleci update\n      - run: |\n          curl -sSLf https://raw.githubusercontent.com/CircleCI-Public/circleci-cli/main/install.sh | sudo bash\n      - checkout\n      #- setup_remote_docker:\n      #    version: 20.10.11\n      - run: setup/ci_bootstrap.sh\n      - run: make init\n      - run: make\n      - run: make test\n"
  },
  {
    "path": ".cirrus.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-24 16:55:36 +0000 (Mon, 24 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               C i r r u s   C I\n# ============================================================================ #\n\n# https://cirrus-ci.org/guide/writing-tasks/\n\ncontainer:\n  image: ubuntu:18.04\n\ntask:\n  env:\n    TMPDIR: /var/tmp\n  script:\n    - setup/ci_bootstrap.sh\n    - make init\n    - make ci test\n"
  },
  {
    "path": ".dockerignore",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Sep 12 13:06:25 2017 +0200\n#\n#  vim: filetype=conf\n#\n\n# ============================================================================ #\n#                           . d o c k e r i g n o r e\n# ============================================================================ #\n\n# https://docs.docker.com/engine/reference/builder/#dockerignore-file\n\n# =================================================\n# Don't send things you don't need to Docker server and avoid uploading your secrets / keys!!\n#\n# XXX: RULES:\n#      - .dockerignore must be at top-level root context from which you invoke 'docker build'\n#      - unfortunately .dockerignore doesn't match basenames like .gitignore so you need to prefix **/ for recursive matching to be safe\n#      - Last Match Wins - you must put more specific whitelisting matches after the general exclusion pattern\n\n# ====================================================================\n# Smaller more concise .dockerignore files are found in the builds at:\n#\n#   https://github.com/HariSekhon/Dockerfiles\n\n# =========================================================\n# This is a huge list of exclusions which covers most cases\n#\n# XXX: Best Practice - enable '*' ignore and create short concise whitelist of inclusions\n#\n#      *\n#\n#      Second option is to rely on this pretty extensive blacklist, which excludes most known credentials files, all hidden dot files and major RCS repos like Git / Mercurial / Subversion\n#\n# Sloppy Docker builds which do 'COPY/ADD .' are a ** Security Risk ** - see Dockerfiles repo's tests/ which checks for their existence in all Dockerfile's\n#\n#   https://github.com/HariSekhon/Dockerfiles\n\n# *\n\n# =========================================================================================\n# Whitelist - must come after the more general blacklist pattern * above as Last Match Wins\n#\n# always include package.json and requirements.txt for standard dependency installations on NodeJS and Python\n!package.json\n!requirements.txt\n!LICENSE\n\n# ===============\n# Docker specific\n\n# still sent to the daemon as needed to do the job but aren't included in the image by ADD/COPY . commands\n**/Dockerfile\n**/.dockerignore\n\n**/docker-compose.yml\n**/.gitignore\n**/.gcloudignore\n**/.ssh/\n# don't accidentally publish your whole code base via Docker like Twitter Vine did!!\n**/.git/\n**/.svn/\n**/.hg/\n**/git/\n**/github/\n**/gitolite*/\n**/gitroot/\n**/mercurial/\n**/hg/\n**/hgroot/\n**/svn/\n**/svnroot/\n\n**/node_modules/\n**/dist/\n**/fatpacks/\n**/logs/\n**/vendor/\n**/vagrant/\n**/venv/\n**/wordlists/\n**/pytools_checks/\n**/debs/\n**/rpms/\n**/drive/\n**/Google Drive/\n**/Dropbox/\n\n# XXX: don't include any hidden files unless we explicitly override and include them with a !.filename\n**/.*\n\n# don't include CI configs not covered by ignoring dotfiles\n**/Jenkinsfile\n**/azure-pipelines.y*ml\n**/bitbucket-pipelines.y*ml\n**/buddy.y*ml\n**/codefresh.y*ml\n**/shippable.y*ml\n**/wercker.y*ml\n**/gocd_config_repo.json\n**/jenkins-job.xml\n**/hadolint.y*ml\n**/scalastyle_config.xml\n**/yamllint/\n# contains webhook URL which should not be committed publicly\n**/buildkite-pipeline*.json\n\n# leave our README.md in case we want to include it in the image but filter out other .md files\n*.md\n!README.md\n\n# ========================================\n# Based on the massive adjacent .gitignore, modified for Docker ignore's Go filepath.Match() function\n*#*#\n**/*.a\n**/*.avi\n**/*.bak\n**/*.bak.*\n**/*.bin\n**/*.bkp\n**/*.class\n**/*.dump\n**/*.flv\n**/*.kdb\n**/*.lock\n**/*.log\n**/*.macports-saved_*\n**/*.mp3\n**/*.mp4\n**/*.mpeg\n**/*.mpg\n**/*.o\n**/*.orig\n**/*.out\n**/*.part\n**/*.pyc\n**/*.pyo\n**/*.stderr\n**/*.stdout\n**/*.swo\n**/*.swp\n**/*.tmp\n**/*.wmv\n**/*~\n**/tmp.*\n**/~*\n\n**/*.doc\n**/*.docx\n**/*.msg\n**/*.pages\n**/*.ppt\n**/*.pptx\n**/*.rtf\n**/*.wpd\n**/*.wps\n**/*.xls\n**/*.xlsx\n\n# ============================================================================ #\n#\n# regenerate all sections below in to a single arg for API call via:\n#\n#       grep '[C]reated by https://' .dockerignore | sed 's,.*/,,' | tr ',' '\\n' | sort -u | tr '\\n' ',' | sed 's/,$//' | xargs echo gitignore.io_api.sh\n#\n# then pipe through perl to add recursive prefix '**/':\n#\n#       gitignore.io_api.sh ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh | perl -p -e 's/^([^#\\s\\/])/**\\/$1/; s/^\\//**\\//' >> .dockerignore\n#\n# Find new or missing tags you aren't using yet:\n#\n#       grep '[C]reated by https://' .dockerignore | sed 's,.*/,,' | tr ',' '\\n' | sort -u | tr '\\n' ',' | sed 's/,$//' | gitignore.io_api.sh missing\n#\n# ============================================================================ #\n\n\n# Created by https://www.toptal.com/developers/gitignore/api/ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n# Edit at https://www.toptal.com/developers/gitignore?templates=ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n\n### Ansible ###\n**/*.retry\n\n### ApacheHadoop ###\n**/*.iml\n**/*.ipr\n**/*.iws\n**/*.orig\n**/*.rej\n**/.idea\n**/.svn\n**/.classpath\n**/.project\n**/.settings\n**/target\n**/hadoop-common-project/hadoop-kms/downloads/\n**/hadoop-hdfs-project/hadoop-hdfs/downloads\n**/hadoop-hdfs-project/hadoop-hdfs-httpfs/downloads\n**/hadoop-common-project/hadoop-common/src/test/resources/contract-test-options.xml\n**/hadoop-tools/hadoop-openstack/src/test/resources/contract-test-options.xml\n\n### AppCode ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n**/.idea/**/workspace.xml\n**/.idea/**/tasks.xml\n**/.idea/**/usage.statistics.xml\n**/.idea/**/dictionaries\n**/.idea/**/shelf\n\n# Generated files\n**/.idea/**/contentModel.xml\n\n# Sensitive or high-churn files\n**/.idea/**/dataSources/\n**/.idea/**/dataSources.ids\n**/.idea/**/dataSources.local.xml\n**/.idea/**/sqlDataSources.xml\n**/.idea/**/dynamic.xml\n**/.idea/**/uiDesigner.xml\n**/.idea/**/dbnavigator.xml\n\n# Gradle\n**/.idea/**/gradle.xml\n**/.idea/**/libraries\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n**/cmake-build-*/\n\n# Mongo Explorer plugin\n**/.idea/**/mongoSettings.xml\n\n# File-based project format\n\n# IntelliJ\n**/out/\n\n# mpeltonen/sbt-idea plugin\n**/.idea_modules/\n\n# JIRA plugin\n**/atlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n**/.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n**/com_crashlytics_export_strings.xml\n**/crashlytics.properties\n**/crashlytics-build.properties\n**/fabric.properties\n\n# Editor-based Rest Client\n**/.idea/httpRequests\n\n# Android studio 3.1+ serialized cache file\n**/.idea/caches/build_file_checksums.ser\n\n### AppCode Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n**/.idea/**/sonarlint/\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n**/.idea/**/sonarIssues.xml\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n**/.idea/**/markdown-navigator.xml\n**/.idea/**/markdown-navigator-enh.xml\n**/.idea/**/markdown-navigator/\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n**/.idea/$CACHE_FILE$\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n**/.idea/codestream.xml\n\n### AppEngine ###\n# Google App Engine generated folder\n**/appengine-generated/\n\n### Archive ###\n\n### Mostly from https://en.wikipedia.org/wiki/List_of_archive_formats\n\n## Archiving only\n# The traditional archive format on Unix-like systems, now used mainly for the creation of static libraries.\n**/*.a\n**/*.ar\n# RPM files consist of metadata concatenated with (usually) a cpio archive. Newer RPM systems also support other archives, as cpio is becoming obsolete. cpio is also used with initramfs.\n**/*.cpio\n\n# A self-extracting archive that uses the Bourne shell (sh).\n**/*.shar\n# A system for storing multiple files. LBR archives typically contained files processed by SQ, or the archive itself was compressed with SQ. LBR archives that were compressed with SQ ended with the extension .LQR\n**/*.LBR\n# An archive format originally used mainly for archiving and distribution of the exact, nearly-exact, or custom-modified contents of an optical storage medium such as a CD-ROM or DVD-ROM. However, it can be used to archive the contents of other storage media, selected partitions, folders, and/or files. The resulting archive is typically optimized for convenient rendering to (re-)writable CD or DVD media.\n**/*.iso\n# A library format used primarily on the Commodore 64 and 128 lines of computers. This bears no resemblance to the DOS LBR format. While library files were quick to implement (a number of programs exist to work with them) they are crippled in that they cannot grow with use: once a file has been created it cannot be amended (files added, changed or deleted) without recreating the entire file.\n**/*.lbr\n# An archive format used by Mozilla for storing binary diffs. Used in conjunction with bzip2.\n**/*.mar\n# A common archive format used on Unix-like systems. Generally used in conjunction with compressors such as gzip, bzip2, compress or xz to create .tar.gz, .tar.bz2, .tar.Z or tar.xz files.\n**/*.tar\n\n# Package managers\n# Red Hat Package Manager\n**/*.rpm\n# Debian package\n**/*.deb\n# MicroSoft Installer\n**/*.msi\n**/*.msm\n**/*.msp\n# Mozilla package installer\n**/*.xpi\n# Ruby Package\n**/*.gem\n\n\n### Archives ###\n# It's better to unpack these files and commit the raw source because\n# git has its own built in compression methods.\n**/*.7z\n**/*.jar\n**/*.rar\n**/*.zip\n**/*.gz\n**/*.gzip\n**/*.tgz\n**/*.bzip\n**/*.bzip2\n**/*.bz2\n**/*.xz\n**/*.lzma\n**/*.cab\n**/*.xar\n\n# Packing-only formats\n\n# Package management formats\n**/*.dmg\n**/*.egg\n**/*.txz\n\n### ArchLinuxPackages ###\n**/*.tar.*\n**/*.exe\n**/*.log\n**/*.log.*\n**/*.sig\n\n**/pkg/\n**/src/\n\n### Audio ###\n**/*.aif\n**/*.iff\n**/*.m3u\n**/*.m4a\n**/*.mid\n**/*.mp3\n**/*.mpa\n**/*.ra\n**/*.wav\n**/*.wma\n**/*.ogg\n**/*.flac\n\n### Autotools ###\n# http://www.gnu.org/software/automake\n\n**/Makefile.in\n**/ar-lib\n**/mdate-sh\n**/py-compile\n**/test-driver\n**/ylwrap\n**/.deps/\n\n# http://www.gnu.org/software/autoconf\n\n**/autom4te.cache\n**/autoscan.log\n**/autoscan-*.log\n**/aclocal.m4\n**/compile\n**/config.guess\n**/config.h.in\n**/config.log\n**/config.status\n**/config.sub\n**/configure\n**/configure.scan\n**/depcomp\n**/install-sh\n**/missing\n**/stamp-h1\n\n# https://www.gnu.org/software/libtool/\n\n**/ltmain.sh\n\n# http://www.gnu.org/software/texinfo\n\n**/texinfo.tex\n\n# http://www.gnu.org/software/m4/\n\n**/m4/libtool.m4\n**/m4/ltoptions.m4\n**/m4/ltsugar.m4\n**/m4/ltversion.m4\n**/m4/lt~obsolete.m4\n\n# Generated Makefile\n# (meta build system like autotools,\n# can automatically generate from config.status script\n# (which is called by configure script))\n**/Makefile\n\n### Autotools Patch ###\n\n### Backup ###\n**/*.bak\n**/*.gho\n**/*.ori\n**/*.tmp\n\n### Basic ###\n# Apples Build\n**/*.build\n**/*.apples\n\n# Initialized files\n**/*.ini\n**/*.basic\n\n### BitTorrent ###\n**/*.torrent\n\n### C ###\n# Prerequisites\n**/*.d\n\n# Object files\n**/*.o\n**/*.ko\n**/*.obj\n**/*.elf\n\n# Linker output\n**/*.ilk\n**/*.map\n**/*.exp\n\n# Precompiled Headers\n**/*.gch\n**/*.pch\n\n# Libraries\n**/*.lib\n**/*.la\n**/*.lo\n\n# Shared objects (inc. Windows DLLs)\n**/*.dll\n**/*.so\n**/*.so.*\n**/*.dylib\n\n# Executables\n**/*.out\n**/*.app\n**/*.i*86\n**/*.x86_64\n**/*.hex\n\n# Debug files\n**/*.dSYM/\n**/*.su\n**/*.idb\n**/*.pdb\n\n# Kernel Module Compile Results\n**/*.mod*\n**/*.cmd\n**/.tmp_versions/\n**/modules.order\n**/Module.symvers\n**/Mkfile.old\n**/dkms.conf\n\n### C++ ###\n# Prerequisites\n\n# Compiled Object files\n**/*.slo\n\n# Precompiled Headers\n\n# Compiled Dynamic libraries\n\n# Fortran module files\n**/*.mod\n**/*.smod\n\n# Compiled Static libraries\n**/*.lai\n\n# Executables\n\n### Zsh ###\n# Zsh compiled script + zrecompile backup\n**/*.zwc\n**/*.zwc.old\n\n# Zsh completion-optimization dumpfile\n**/*zcompdump*\n\n# Zsh zcalc history\n**/.zcalc_history\n\n# A popular plugin manager's files\n**/._zplugin\n**/.zplugin_lstupd\n\n# zdharma/zshelldoc tool's files\n**/zsdoc/data\n\n# robbyrussell/oh-my-zsh/plugins/per-directory-history plugin's files\n# (when set-up to store the history in the local directory)\n**/.directory_history\n\n# MichaelAquilina/zsh-autoswitch-virtualenv plugin's files\n# (for Zsh plugins using Python)\n**/.venv\n\n# Zunit tests' output\n**/tests/_output/*\n**/!/tests/_output/.gitkeep\n\n### certificates ###\n**/*.pem\n**/*.key\n**/*.crt\n**/*.cer\n**/*.priv\n\n### ChefCookbook ###\n**/.vagrant\n**/cookbooks\n\n# Bundler\n**/bin/*\n**/.bundle/*\n\n**/.kitchen/\n**/.kitchen.local.yml\n**/.kitchen.*.local.yml\n**/kitchen.local.yml\n**/kitchen.*.local.yml\n\n### CMake ###\n**/CMakeLists.txt.user\n**/CMakeCache.txt\n**/CMakeFiles\n**/CMakeScripts\n**/Testing\n**/cmake_install.cmake\n**/install_manifest.txt\n**/compile_commands.json\n**/CTestTestfile.cmake\n**/_deps\n\n### CMake Patch ###\n# External projects\n**/*-prefix/\n\n### Clojure ###\n**/pom.xml\n**/pom.xml.asc\n**/*.class\n**/lib/\n**/classes/\n**/target/\n**/checkouts/\n**/.lein-deps-sum\n**/.lein-repl-history\n**/.lein-plugins/\n**/.lein-failures\n**/.nrepl-port\n**/.cpcache/\n\n### Code-Java ###\n# Language Support for Java(TM) by Red Hat extension for Visual Studio Code - https://marketplace.visualstudio.com/items?itemName=redhat.java\n\n**/factoryConfiguration.json\n\n### Cloud9 ###\n# Cloud9 IDE - http://c9.io\n**/.c9revisions\n**/.c9\n\n### Compressed ###\n**/*.pkg\n**/*.sit\n**/*.sitx\n**/*.zipx\n\n### CompressedArchive ###\n\n\n## Archiving and compression\n# Open source file format. Used by 7-Zip.\n# Mac OS X, restoration on different platforms is possible although not immediate \tYes \tBased on 7z. Preserves Spotlight metadata, resource forks, owner/group information, dates and other data which would be otherwise lost with compression.\n**/*.s7z\n# Old archive versions only \tProprietary format\n**/*.ace\n# A format that compresses and doubly encrypt the data (AES256 and CAS256) avoiding brute force attacks, also hide files in an AFA file. It has two ways to safeguard data integrity and subsequent repair of the file if has an error (repair with AstroA2P (online) or Astrotite (offline)).\n**/*.afa\n# A mainly Korean format designed for very large archives.\n**/*.alz\n# Android application package (variant of JAR file format).\n**/*.apk\n# ??\n**/*.arc\n# Originally DOS, now multiple\n**/*.arj\n# Open archive format, used by B1 Free Archiver (http://dev.b1.org/standard/archive-format.html)\n**/*.b1\n# Binary Archive with external header\n**/*.ba\n# Proprietary format from the ZipTV Compression Components\n**/*.bh\n# The Microsoft Windows native archive format, which is also used by many commercial installers such as InstallShield and WISE.\n# Originally DOS, now DOS and Windows \tCreated by Yaakov Gringeler; released last in 2003 (Compressia 1.0.0.1 beta), now apparently defunct. Free trial of 30 days lets user create and extract archives; after that it is possible to extract, but not to create.\n**/*.car\n# Open source file format.\n**/*.cfs\n# Compact Pro archive, a common archiver used on Mac platforms until about Mac OS 7.5.x. Competed with StuffIt; now obsolete.\n**/*.cpt\n# Windows, Unix-like, Mac OS X Open source file format. Files are compressed individually with either gzip, bzip2 or lzo.\n**/*.dar\n# DiskDoubler \tMac OS \t\t\tobsolete\n**/*.dd\n# ??\n**/*.dgc\n# Apple Disk Image upports \"Internet-enabled\" disk images, which, once downloaded, are automatically decompressed, mounted, have the contents extracted, and thrown away. Currently, Safari is the only browser that supports this form of extraction; however, the images can be manually extracted as well. This format can also be password-protected or encrypted with 128-bit or 256-bit AES encryption.\n# Enterprise Java Archive archive\n**/*.ear\n# ETSoft compressed archive\n# The predecessor of DGCA.\n**/*.gca\n# Originally DOS \tYes, but may be covered by patents \tDOS era format; uses arithmetic/Markov coding\n**/*.ha\n# MS Windows \tHKI\n**/*.hki\n# Produced by ICEOWS program. Excels at text file compression.\n**/*.ice\n# Java archive, compatible with ZIP files\n# Open sourced archiver with compression using the PAQ family of algorithms and optional encryption.\n**/*.kgb\n# Originally DOS, now multiple \tMultiple \tYes \tThe standard format on Amiga.\n**/*.lzh\n**/*.lha\n# Archiver originally used on The Amiga. Now copied by Microsoft to use in their .cab and .chm files.\n**/*.lzx\n# file format from NoGate Consultings, a rival from ARC-Compressor.\n**/*.pak\n# A disk image archive format that supports several compression methods as well as splitting the archive into smaller pieces.\n**/*.partimg\n# An experimental open source packager (http://mattmahoney.net/dc)\n**/*.paq*\n# Open source archiver supporting authenticated encryption, volume spanning, customizable object level and volume level integrity checks (form CRCs to SHA-512 and Whirlpool hashes), fast deflate based compression\n**/*.pea\n# The format from the PIM - a freeware compression tool by Ilia Muraviev. It uses an LZP-based compression algorithm with set of filters for executable, image and audio files.\n**/*.pim\n# PackIt \tMac OS \t\t\tobsolete\n**/*.pit\n# Used for data in games written using the Quadruple D library for Delphi. Uses byte pair compression.\n**/*.qda\n# A proprietary archive format, second in popularity to .zip files.\n# The format from a commercial archiving package. Odd among commercial packages in that they focus on incorporating experimental algorithms with the highest possible compression (at the expense of speed and memory), such as PAQ, PPMD and PPMZ (PPMD with unlimited-length strings), as well as a proprietary algorithms.\n**/*.rk\n# Self Dissolving ARChive \tCommodore 64, Commodore 128 \tCommodore 64, Commodore 128 \tYes \tSDAs refer to Self Dissolving ARC files, and are based on the Commodore 64 and Commodore 128 versions of ARC, originally written by Chris Smeets. While the files share the same extension, they are not compatible between platforms. That is, an SDA created on a Commodore 64 but run on a Commodore 128 in Commodore 128 mode will crash the machine, and vice versa. The intended successor to SDA is SFX.\n**/*.sda\n# A pre-Mac OS X Self-Extracting Archive format. StuffIt, Compact Pro, Disk Doubler and others could create .sea files, though the StuffIt versions were the most common.\n**/*.sea\n# Scifer Archive with internal header\n**/*.sen\n# Commodore 64, Commodore 128 \tSFX is a Self Extracting Archive which uses the LHArc compression algorithm. It was originally developed by Chris Smeets on the Commodore platform, and runs primarily using the CS-DOS extension for the Commodore 128. Unlike its predecessor SDA, SFX files will run on both the Commodore 64 and Commodore 128 regardless of which machine they were created on.\n**/*.sfx\n# An archive format designed for the Apple II series of computers. The canonical implementation is ShrinkIt, which can operate on disk images as well as files. Preferred compression algorithm is a combination of RLE and 12-bit LZW. Archives can be manipulated with the command-line NuLib tool, or the Windows-based CiderPress.\n**/*.shk\n# A compression format common on Apple Macintosh computers. The free StuffIt Expander is available for Windows and OS X.\n# The replacement for the .sit format that supports more compression methods, UNIX file permissions, long file names, very large files, more encryption options, data specific compressors (JPEG, Zip, PDF, 24-bit image, MP3). The free StuffIt Expander is available for Windows and OS X.\n# A royalty-free compressing format\n**/*.sqx\n# The \"tarball\" format combines tar archives with a file-based compression scheme (usually gzip). Commonly used for source and binary distribution on Unix-like platforms, widely available elsewhere.\n**/*.tar.gz\n**/*.tar.Z\n**/*.tar.bz2\n**/*.tbz2\n**/*.tar.lzma\n**/*.tlz\n# UltraCompressor 2.3 was developed to act as an alternative to the then popular PKZIP application. The main feature of the application is its ability to create large archives. This means that compressed archives with the UC2 file extension can hold almost 1 million files.\n**/*.uc\n**/*.uc0\n**/*.uc2\n**/*.ucn\n**/*.ur2\n**/*.ue2\n# Based on PAQ, RZM, CSC, CCM, and 7zip. The format consists of a PAQ, RZM, CSC, or CCM compressed file and a manifest with compression settings stored in a 7z archive.\n**/*.uca\n# A high compression rate archive format originally for DOS.\n**/*.uha\n# Web Application archive (Java-based web app)\n**/*.war\n# File-based disk image format developed to deploy Microsoft Windows.\n**/*.wim\n# XAR\n# Native format of the Open Source KiriKiri Visual Novel engine. Uses combination of block splitting and zlib compression. The filenames and pathes are stored in UTF-16 format. For integrity check, the Adler-32 hashsum is used. For many commercial games, the files are encrypted (and decoded on runtime) via so-called \"cxdec\" module, which implements xor-based encryption.\n**/*.xp3\n# Yamazaki zipper archive. Compression format used in DeepFreezer archiver utility created by Yamazaki Satoshi. Read and write support exists in TUGZip, IZArc and ZipZag\n**/*.yz1\n# The most widely used compression format on Microsoft Windows. Commonly used on Macintosh and Unix systems as well.\n# application/x-zoo \tzoo \tMultiple \tMultiple \tYes\n**/*.zoo\n# Journaling (append-only) archive format with rollback capability. Supports deduplication and incremental update based on last-modified dates. Multi-threaded. Compresses in LZ77, BWT, and context mixing formats. Open source.\n**/*.zpaq\n# Archiver with a compression algorithm based on the Burrows-Wheeler transform method.\n**/*.zz\n\n\n### Compression ###\n\n### From https://en.wikipedia.org/wiki/List_of_archive_formats\n\n## Compression only\n# An open source, patent- and royalty-free compression format. The compression algorithm is a Burrows-Wheeler transform followed by a move-to-front transform and finally Huffman coding\n# Old compressor for QNX4 OS. The compression algorithm is a modified LZSS, with an adaptive Huffman coding.\n**/*.F\n# GNU Zip, the primary compression format used by Unix-like systems. The compression algorithm is DEFLATE.\n# An alternate LZMA algorithm implementation, with support for checksums and ident bytes.\n**/*.lz\n# The LZMA compression algorithm as used by 7-Zip\n# An implementation of the LZO data compression algorithm\n**/*.lzo\n# A compression program designed to do particularly well on very large files containing long distance redundancy.\n**/*.rz\n# Windows compress/decompress- Linux and Mac OS X decompress only \tA compression program designed to do high compression on SF2 files (SoundFont)\n**/*.sfark\n# A compression format invented by Google and open-sourced in 2011. Snappy aims for very high speeds, reasonable compression, and maximum stability rather than maximum compression or compatibility with any other compression library.\n**/*.sz\n# Squeeze: A program which compressed files. A file which was \"squeezed\" had the middle initial of the name changed to \"Q\", so that a squeezed text file would end with .TQT, a squeezed executable would end with .CQM or .EQE. Typically used with .LBR archives, either by storing the squeezed files in the archive, or by storing the files decompressed and then compressing the archive, which would have a name ending in \".LQR\".\n**/*.?Q?\n# A compression program written by Steven Greenberg implementing the LZW algorithm. For several years in the CP/M world when no implementation was available of ARC, CRUNCHed files stored in .LBR archives were very popular. CRUNCH's implementation of LZW had a somewhat unique feature of modifying and occasionally clearing the code table in memory when it became full, resulting in a few percent better compression on many files.\n**/*.?Z?\n# A compression format using LZMA2 to yield very high compression ratios.\n# The traditional Huffman coding compression format.\n**/*.z\n# The traditional LZW compression format.\n**/*.Z\n# Joke compression program, actually increasing file size\n**/*.infl\n# Compression format(s) used by some DOS and Windows install programs. MS-DOS includes expand.exe to decompress its install files. The compressed files are created with a matching compress.exe command. The compression algorithm is LZSS.\n**/*.??_\n\n\n### Data ###\n**/*.csv\n**/*.dat\n**/*.efx\n**/*.gbr\n**/*.pps\n**/*.ppt\n**/*.pptx\n**/*.sdf\n**/*.tax2010\n**/*.vcf\n**/*.xml\n\n### Code ###\n**/.vscode/*\n**/!.vscode/settings.json\n**/!.vscode/tasks.json\n**/!.vscode/launch.json\n**/!.vscode/extensions.json\n**/*.code-workspace\n\n### DataRecovery ###\n\n\n## Data recovery\n# File format used by dvdisaster to be used for data recovery when discs become damaged or partially unreadable.\n**/*.ecc\n# File format used in conjunction with any archive format to provide redundancy and data recovery, most often in newsgroup distribution of binary files.\n**/*.par\n**/*.par2\n\n\n### Diff ###\n**/*.patch\n**/*.diff\n\n### direnv ###\n**/.direnv\n**/.envrc\n\n### CodeBlocks ###\n# specific to CodeBlocks IDE\n**/*.layout\n**/*.depend\n# generated directories\n**/bin/\n**/obj/\n\n### DocFx ###\n**/.cache\n**/**/_site/\n\n### Docpress ###\n# docpress documentation generator: https://docpress.github.io/index.html\n\n**/_docpress/\n\n### Docz ###\n**/.docz\n\n\n### dotenv ###\n**/.env\n\n### DotfilesSh ###\n**/local-patch\n**/patched-src\n\n### DotSettings ###\n**/*.DotSettings\n\n### Dropbox ###\n# Dropbox settings and caches\n**/.dropbox\n**/.dropbox.attr\n**/.dropbox.cache\n\n### Eclipse ###\n**/.metadata\n**/tmp/\n**/*.swp\n**/*~.nib\n**/local.properties\n**/.settings/\n**/.loadpath\n**/.recommenders\n\n# External tool builders\n**/.externalToolBuilders/\n\n# Locally stored \"Eclipse launch configurations\"\n**/*.launch\n\n# PyDev specific (Python IDE for Eclipse)\n**/*.pydevproject\n\n# CDT-specific (C/C++ Development Tooling)\n**/.cproject\n\n# CDT- autotools\n**/.autotools\n\n# Java annotation processor (APT)\n**/.factorypath\n\n# PDT-specific (PHP Development Tools)\n**/.buildpath\n\n# sbteclipse plugin\n**/.target\n\n# Tern plugin\n**/.tern-project\n\n# TeXlipse plugin\n**/.texlipse\n\n# STS (Spring Tool Suite)\n**/.springBeans\n\n# Code Recommenders\n**/.recommenders/\n\n# Annotation Processing\n**/.apt_generated/\n**/.apt_generated_test/\n\n# Scala IDE specific (Scala & Java development for Eclipse)\n**/.cache-main\n**/.scala_dependencies\n**/.worksheet\n\n# Uncomment this line if you wish to ignore the project description file.\n# Typically, this file would be tracked if it contains build/dependency configurations:\n#.project\n\n### Eclipse Patch ###\n# Spring Boot Tooling\n**/.sts4-cache/\n\n### Emacs ###\n# -*- mode: gitignore; -*-\n**/*~\n**/\\#*\\#\n**/.emacs.desktop\n**/.emacs.desktop.lock\n**/*.elc\n**/auto-save-list\n**/tramp\n**/.\\#*\n\n# Org-mode\n**/.org-id-locations\n**/*_archive\n\n# flymake-mode\n**/*_flymake.*\n\n# eshell files\n**/eshell/history\n**/eshell/lastdir\n\n# elpa packages\n**/elpa/\n\n# reftex files\n**/*.rel\n\n# AUCTeX auto folder\n**/auto/\n\n# cask packages\n**/.cask/\n**/dist/\n\n# Flycheck\n**/flycheck_*.el\n\n# server auth directory\n**/server/\n\n# projectiles files\n**/.projectile\n\n# directory configuration\n**/.dir-locals.el\n\n# network security\n**/network-security.data\n\n\n### Database ###\n**/*.accdb\n**/*.db\n**/*.dbf\n**/*.mdb\n**/*.sqlite3\n\n### Executable ###\n**/*.bat\n**/*.cgi\n**/*.com\n**/*.gadget\n**/*.pif\n**/*.vb\n**/*.wsf\n\n### Firebase ###\n**/**/node_modules/*\n**/**/.firebaserc\n\n### Firebase Patch ###\n**/.runtimeconfig.json\n**/.firebase/\n\n### Flask ###\n**/instance/*\n**/!instance/.gitignore\n**/.webassets-cache\n\n### Flask.Python Stack ###\n# Byte-compiled / optimized / DLL files\n**/__pycache__/\n**/*.py[cod]\n**/*$py.class\n\n# C extensions\n\n# Distribution / packaging\n**/.Python\n**/build/\n**/develop-eggs/\n**/downloads/\n**/eggs/\n**/.eggs/\n**/lib/\n**/lib64/\n**/parts/\n**/sdist/\n**/var/\n**/wheels/\n**/pip-wheel-metadata/\n**/share/python-wheels/\n**/*.egg-info/\n**/.installed.cfg\n**/MANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n**/*.manifest\n**/*.spec\n\n# Installer logs\n**/pip-log.txt\n**/pip-delete-this-directory.txt\n\n# Unit test / coverage reports\n**/htmlcov/\n**/.tox/\n**/.nox/\n**/.coverage\n**/.coverage.*\n**/nosetests.xml\n**/coverage.xml\n**/*.cover\n**/*.py,cover\n**/.hypothesis/\n**/.pytest_cache/\n**/pytestdebug.log\n\n# Translations\n**/*.mo\n**/*.pot\n\n# Django stuff:\n**/local_settings.py\n**/db.sqlite3\n**/db.sqlite3-journal\n\n# Flask stuff:\n**/instance/\n\n# Scrapy stuff:\n**/.scrapy\n\n# Sphinx documentation\n**/docs/_build/\n**/doc/_build/\n\n# PyBuilder\n**/target/\n\n# Jupyter Notebook\n**/.ipynb_checkpoints\n\n# IPython\n**/profile_default/\n**/ipython_config.py\n\n# pyenv\n**/.python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n**/__pypackages__/\n\n# Celery stuff\n**/celerybeat-schedule\n**/celerybeat.pid\n\n# SageMath parsed files\n**/*.sage.py\n\n# Environments\n**/env/\n**/venv/\n**/ENV/\n**/env.bak/\n**/venv.bak/\n**/pythonenv*\n\n# Spyder project settings\n**/.spyderproject\n**/.spyproject\n\n# Rope project settings\n**/.ropeproject\n\n# mkdocs documentation\n**/site\n\n# mypy\n**/.mypy_cache/\n**/.dmypy.json\n**/dmypy.json\n\n# Pyre type checker\n**/.pyre/\n\n# pytype static type analyzer\n**/.pytype/\n\n# profiling data\n**/.prof\n\n### Git ###\n# Created by git for backups. To disable backups in Git:\n# $ git config --global mergetool.keepBackup false\n\n# Created by git when using merge tools for conflicts\n**/*.BACKUP.*\n**/*.BASE.*\n**/*.LOCAL.*\n**/*.REMOTE.*\n**/*_BACKUP_*.txt\n**/*_BASE_*.txt\n**/*_LOCAL_*.txt\n**/*_REMOTE_*.txt\n\n### GitBook ###\n# Node rules:\n## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)\n**/.grunt\n\n## Dependency directory\n## Commenting this out is preferred by some people, see\n## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git\n**/node_modules\n\n# Book build output\n**/_book\n\n# eBook build output\n**/*.epub\n**/*.mobi\n**/*.pdf\n\n### Go ###\n# Binaries for programs and plugins\n**/*.exe~\n\n# Test binary, built with `go test -c`\n**/*.test\n\n# Output of the go coverage tool, specifically when used with LiteIDE\n\n# Dependency directories (remove the comment below to include it)\n# vendor/\n\n### Go Patch ###\n**/vendor/\n**/Godeps/\n\n### GPG ###\n**/secring.*\n\n\n### DiskImage ###\n**/*.toast\n**/*.vcd\n\n### Grails ###\n# .gitignore for Grails 1.2 and 1.3\n# Although this should work for most versions of grails, it is\n# suggested that you use the \"grails integrate-with --git\" command\n# to generate your .gitignore file.\n\n# web application files\n**/web-app/WEB-INF/classes\n\n# default HSQL database files for production mode\n**/prodDb.*\n\n# general HSQL database files\n**/*Db.properties\n**/*Db.script\n\n# logs\n**/stacktrace.log\n**/test/reports\n**/logs\n\n# project release file\n**/*.war\n\n# plugin release files\n**/*.zip\n**/plugin.xml\n\n# older plugin install locations\n**/plugins\n**/web-app/plugins\n\n# \"temporary\" build files\n**/target\n\n### Groovy ###\n# .gitignore created from Groovy contributors in https://github.com/apache/groovy/blob/master/.gitignore\n\n**/user.gradle\n**/.gradle/\n\n**/*.DS_Store\n\n**/.shelf\n\n\n\n### grunt ###\n# Grunt usually compiles files inside this directory\n\n# Grunt usually preprocesses files such as coffeescript, compass... inside the .tmp directory\n**/.tmp/\n\n### Haskell ###\n**/dist\n**/dist-*\n**/cabal-dev\n**/*.hi\n**/*.hie\n**/*.chi\n**/*.chs.h\n**/*.dyn_o\n**/*.dyn_hi\n**/.hpc\n**/.hsenv\n**/.cabal-sandbox/\n**/cabal.sandbox.config\n**/*.prof\n**/*.aux\n**/*.hp\n**/*.eventlog\n**/.stack-work/\n**/cabal.project.local\n**/cabal.project.local~\n**/.HTF/\n**/.ghc.environment.*\n\n### Helm ###\n# Chart dependencies\n**/**/charts/*.tgz\n\n### Homebrew ###\n**/Brewfile.lock.json\n\n### Hugo ###\n# Generated files by hugo\n**/public/\n**/resources/_gen/\n**/hugo_stats.json\n\n# Executable may be added to repository\n**/hugo.exe\n**/hugo.darwin\n**/hugo.linux\n\n### Images ###\n# JPEG\n**/*.jpg\n**/*.jpeg\n**/*.jpe\n**/*.jif\n**/*.jfif\n**/*.jfi\n\n# JPEG 2000\n**/*.jp2\n**/*.j2k\n**/*.jpf\n**/*.jpx\n**/*.jpm\n**/*.mj2\n\n# JPEG XR\n**/*.jxr\n**/*.hdp\n**/*.wdp\n\n# Graphics Interchange Format\n**/*.gif\n\n# RAW\n**/*.raw\n\n# Web P\n**/*.webp\n\n# Portable Network Graphics\n**/*.png\n\n# Animated Portable Network Graphics\n**/*.apng\n\n# Multiple-image Network Graphics\n**/*.mng\n\n# Tagged Image File Format\n**/*.tiff\n**/*.tif\n\n# Scalable Vector Graphics\n**/*.svg\n**/*.svgz\n\n# Portable Document Format\n\n# X BitMap\n**/*.xbm\n\n# BMP\n**/*.bmp\n**/*.dib\n\n# ICO\n**/*.ico\n\n# 3D Images\n**/*.3dm\n**/*.max\n\n### Intellij ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### Intellij+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n**/.idea/\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n**/modules.xml\n**/.idea/misc.xml\n\n# Sonarlint plugin\n**/.idea/sonarlint\n\n### Intellij+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### Java ###\n# Compiled class file\n\n# Log file\n\n# BlueJ files\n**/*.ctxt\n\n# Mobile Tools for Java (J2ME)\n**/.mtj.tmp/\n\n# Package Files #\n**/*.nar\n\n# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml\n**/hs_err_pid*\n\n### Java-Web ###\n## ignoring target file\n\n### JEnv ###\n# JEnv local Java version configuration file\n**/.java-version\n\n# Used by previous versions of JEnv\n**/.jenv-version\n\n### JetBrains ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### JetBrains+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n# Sonarlint plugin\n\n### JetBrains+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### JMeter ###\n# JMeter common ignore files\n# http://jmeter.apache.org/\n\n# Ignore Summary/Aggregrate reports\n**/*.jtl\n\n# Ignore log files\n\n# Ignore customized user.properties\n**/user.properties\n\n### Erlang ###\n**/.eunit\n**/*.beam\n**/*.plt\n**/erl_crash.dump\n**/.concrete/DEV_MODE\n\n# rebar 2.x\n**/.rebar\n**/rel/example_project\n**/ebin/*.beam\n**/deps\n\n# rebar 3\n**/.rebar3\n**/_build/\n**/_checkouts/\n\n### JupyterNotebooks ###\n# gitignore template for Jupyter Notebooks\n# website: http://jupyter.org/\n\n**/*/.ipynb_checkpoints/*\n\n# IPython\n\n# Remove previous ipynb_checkpoints\n#   git rm -r .ipynb_checkpoints/\n\n### Kotlin ###\n# Compiled class file\n\n# Log file\n\n# BlueJ files\n\n# Mobile Tools for Java (J2ME)\n\n# Package Files #\n\n# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml\n\n### LAMP ###\n# LAMP Stack Base\n\n### LAMP.Linux Stack ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n**/.fuse_hidden*\n\n# KDE directory preferences\n**/.directory\n\n# Linux trash folder which might appear on any partition or disk\n**/.Trash-*\n\n# .nfs files are created when an open file is removed but is still being accessed\n**/.nfs*\n\n### LAMP.PHP Stack ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### LaTeX ###\n## Core latex/pdflatex auxiliary files:\n**/*.lof\n**/*.lot\n**/*.fls\n**/*.toc\n**/*.fmt\n**/*.fot\n**/*.cb\n**/*.cb2\n**/.*.lb\n\n## Intermediate documents:\n**/*.dvi\n**/*.xdv\n**/*-converted-to.*\n# these rules might exclude image files for figures etc.\n# *.ps\n# *.eps\n# *.pdf\n\n## Generated if empty string is given at \"Please type another file name for output:\"\n**/.pdf\n\n## Bibliography auxiliary files (bibtex/biblatex/biber):\n**/*.bbl\n**/*.bcf\n**/*.blg\n**/*-blx.aux\n**/*-blx.bib\n**/*.run.xml\n\n## Build tool auxiliary files:\n**/*.fdb_latexmk\n**/*.synctex\n**/*.synctex(busy)\n**/*.synctex.gz\n**/*.synctex.gz(busy)\n**/*.pdfsync\n\n## Build tool directories for auxiliary files\n# latexrun\n**/latex.out/\n\n## Auxiliary and intermediate files from other packages:\n# algorithms\n**/*.alg\n**/*.loa\n\n# achemso\n**/acs-*.bib\n\n# amsthm\n**/*.thm\n\n# beamer\n**/*.nav\n**/*.pre\n**/*.snm\n**/*.vrb\n\n# changes\n**/*.soc\n\n# comment\n**/*.cut\n\n# cprotect\n\n# elsarticle (documentclass of Elsevier journals)\n**/*.spl\n\n# endnotes\n**/*.ent\n\n# fixme\n**/*.lox\n\n# feynmf/feynmp\n**/*.mf\n**/*.mp\n**/*.t[1-9]\n**/*.t[1-9][0-9]\n**/*.tfm\n\n#(r)(e)ledmac/(r)(e)ledpar\n**/*.end\n**/*.?end\n**/*.[1-9]\n**/*.[1-9][0-9]\n**/*.[1-9][0-9][0-9]\n**/*.[1-9]R\n**/*.[1-9][0-9]R\n**/*.[1-9][0-9][0-9]R\n**/*.eledsec[1-9]\n**/*.eledsec[1-9]R\n**/*.eledsec[1-9][0-9]\n**/*.eledsec[1-9][0-9]R\n**/*.eledsec[1-9][0-9][0-9]\n**/*.eledsec[1-9][0-9][0-9]R\n\n# glossaries\n**/*.acn\n**/*.acr\n**/*.glg\n**/*.glo\n**/*.gls\n**/*.glsdefs\n**/*.lzs\n\n# uncomment this for glossaries-extra (will ignore makeindex's style files!)\n# *.ist\n\n# gnuplottex\n**/*-gnuplottex-*\n\n# gregoriotex\n**/*.gaux\n**/*.gtex\n\n# htlatex\n**/*.4ct\n**/*.4tc\n**/*.idv\n**/*.lg\n**/*.trc\n**/*.xref\n\n# hyperref\n**/*.brf\n\n# knitr\n**/*-concordance.tex\n# TODO Comment the next line if you want to keep your tikz graphics files\n**/*.tikz\n**/*-tikzDictionary\n\n# listings\n**/*.lol\n\n# luatexja-ruby\n**/*.ltjruby\n\n# makeidx\n**/*.idx\n**/*.ilg\n**/*.ind\n\n# minitoc\n**/*.maf\n**/*.mlf\n**/*.mlt\n**/*.mtc\n**/*.mtc[0-9]*\n**/*.slf[0-9]*\n**/*.slt[0-9]*\n**/*.stc[0-9]*\n\n# minted\n**/_minted*\n**/*.pyg\n\n# morewrites\n**/*.mw\n\n# nomencl\n**/*.nlg\n**/*.nlo\n**/*.nls\n\n# pax\n**/*.pax\n\n# pdfpcnotes\n**/*.pdfpc\n\n# sagetex\n**/*.sagetex.sage\n**/*.sagetex.py\n**/*.sagetex.scmd\n\n# scrwfile\n**/*.wrt\n\n# sympy\n**/*.sout\n**/*.sympy\n**/sympy-plots-for-*.tex/\n\n# pdfcomment\n**/*.upa\n**/*.upb\n\n# pythontex\n**/*.pytxcode\n**/pythontex-files-*/\n\n# tcolorbox\n**/*.listing\n\n# thmtools\n**/*.loe\n\n# TikZ & PGF\n**/*.dpth\n**/*.md5\n**/*.auxlock\n\n# todonotes\n**/*.tdo\n\n# vhistory\n**/*.hst\n**/*.ver\n\n# easy-todo\n**/*.lod\n\n# xcolor\n**/*.xcp\n\n# xmpincl\n**/*.xmpi\n\n# xindy\n**/*.xdy\n\n# xypic precompiled matrices and outlines\n**/*.xyc\n**/*.xyd\n\n# endfloat\n**/*.ttt\n**/*.fff\n\n# Latexian\n**/TSWLatexianTemp*\n\n## Editors:\n# WinEdt\n**/*.sav\n\n# Texpad\n**/.texpadtmp\n\n# LyX\n**/*.lyx~\n\n# Kile\n**/*.backup\n\n# gummi\n**/.*.swp\n\n# KBibTeX\n**/*~[0-9]*\n\n# TeXnicCenter\n**/*.tps\n\n# auto folder when using emacs and auctex\n**/./auto/*\n**/*.el\n\n# expex forward references with \\gathertags\n**/*-tags.tex\n\n# standalone packages\n**/*.sta\n\n# Makeindex log files\n**/*.lpz\n\n# REVTeX puts footnotes in the bibliography by default, unless the nofootinbib\n# option is specified. Footnotes are the stored in a file with suffix Notes.bib.\n# Uncomment the next line to have this generated file ignored.\n#*Notes.bib\n\n### LaTeX Patch ###\n# LIPIcs / OASIcs\n**/*.vtc\n\n# glossaries\n**/*.glstex\n\n### Less ###\n**/*.less\n\n### Linux ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n\n# KDE directory preferences\n\n# Linux trash folder which might appear on any partition or disk\n\n# .nfs files are created when an open file is removed but is still being accessed\n\n### Lua ###\n# Compiled Lua sources\n**/luac.out\n\n# luarocks build files\n**/*.src.rock\n\n# Object files\n**/*.os\n\n# Precompiled Headers\n\n# Libraries\n**/*.def\n\n# Shared objects (inc. Windows DLLs)\n\n# Executables\n\n\n### macOS ###\n# General\n**/.DS_Store\n**/.AppleDouble\n**/.LSOverride\n\n# Icon must end with two \\r\n**/Icon\r\n\n# Thumbnails\n**/._*\n\n# Files that might appear in the root of a volume\n**/.DocumentRevisions-V100\n**/.fseventsd\n**/.Spotlight-V100\n**/.TemporaryItems\n**/.Trashes\n**/.VolumeIcon.icns\n**/.com.apple.timemachine.donotpresent\n\n# Directories potentially created on remote AFP share\n**/.AppleDB\n**/.AppleDesktop\n**/Network Trash Folder\n**/Temporary Items\n**/.apdisk\n\n### MATLAB ###\n# Windows default autosave extension\n**/*.asv\n\n# OSX / *nix default autosave extension\n**/*.m~\n\n# Compiled MEX binaries (all platforms)\n**/*.mex*\n\n# Packaged app and toolbox files\n**/*.mlappinstall\n**/*.mltbx\n\n# Generated helpsearch folders\n**/helpsearch*/\n\n# Simulink code generation folders\n**/slprj/\n**/sccprj/\n\n# Matlab code generation folders\n**/codegen/\n\n# Simulink autosave extension\n**/*.autosave\n\n# Simulink cache files\n**/*.slxc\n\n# Octave session info\n**/octave-workspace\n\n### Maven ###\n**/pom.xml.tag\n**/pom.xml.releaseBackup\n**/pom.xml.versionsBackup\n**/pom.xml.next\n**/release.properties\n**/dependency-reduced-pom.xml\n**/buildNumber.properties\n**/.mvn/timing.properties\n# https://github.com/takari/maven-wrapper#usage-without-binary-jar\n**/.mvn/wrapper/maven-wrapper.jar\n\n### Mercurial ###\n**/.hg/\n**/.hgignore\n**/.hgsigs\n**/.hgsub\n**/.hgsubstate\n**/.hgtags\n\n### MicrosoftOffice ###\n\n# Word temporary\n**/~$*.doc*\n\n# Word Auto Backup File\n**/Backup of *.doc*\n\n# Excel temporary\n**/~$*.xls*\n\n# Excel Backup File\n**/*.xlk\n\n# PowerPoint temporary\n**/~$*.ppt*\n\n# Visio autosave temporary files\n**/*.~vsd*\n\n### Node ###\n# Logs\n**/logs\n**/npm-debug.log*\n**/yarn-debug.log*\n**/yarn-error.log*\n**/lerna-debug.log*\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\n**/report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json\n\n# Runtime data\n**/pids\n**/*.pid\n**/*.seed\n**/*.pid.lock\n\n# Directory for instrumented libs generated by jscoverage/JSCover\n**/lib-cov\n\n# Coverage directory used by tools like istanbul\n**/coverage\n**/*.lcov\n\n# nyc test coverage\n**/.nyc_output\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n\n# Bower dependency directory (https://bower.io/)\n**/bower_components\n\n# node-waf configuration\n**/.lock-wscript\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\n**/build/Release\n\n# Dependency directories\n**/node_modules/\n**/jspm_packages/\n\n# TypeScript v1 declaration files\n**/typings/\n\n# TypeScript cache\n**/*.tsbuildinfo\n\n# Optional npm cache directory\n**/.npm\n\n# Optional eslint cache\n**/.eslintcache\n\n# Microbundle cache\n**/.rpt2_cache/\n**/.rts2_cache_cjs/\n**/.rts2_cache_es/\n**/.rts2_cache_umd/\n\n# Optional REPL history\n**/.node_repl_history\n\n# Output of 'npm pack'\n\n# Yarn Integrity file\n**/.yarn-integrity\n\n# dotenv environment variables file\n**/.env.test\n**/.env*.local\n\n# parcel-bundler cache (https://parceljs.org/)\n**/.parcel-cache\n\n# Next.js build output\n**/.next\n\n# Nuxt.js build / generate output\n**/.nuxt\n\n# Gatsby files\n**/.cache/\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n**/.vuepress/dist\n\n# Serverless directories\n**/.serverless/\n\n# FuseBox cache\n**/.fusebox/\n\n# DynamoDB Local files\n**/.dynamodb/\n\n# TernJS port file\n**/.tern-port\n\n# Stores VSCode versions used for testing VSCode extensions\n**/.vscode-test\n\n### Octave ###\n# Windows default autosave extension\n\n# OSX / *nix default autosave extension\n\n# Compiled MEX binaries (all platforms)\n\n# Packaged app and toolbox files\n\n# Generated helpsearch folders\n\n# Simulink code generation folders\n\n# Matlab code generation folders\n\n# Simulink autosave extension\n\n# Simulink cache files\n\n# Octave session info\n\n### OSX ###\n# General\n\n# Icon must end with two \\r\n\n# Thumbnails\n\n# Files that might appear in the root of a volume\n\n# Directories potentially created on remote AFP share\n\n### Packer ###\n# Cache objects\n**/packer_cache/\n\n# Crash log\n**/crash.log\n\n# For built boxes\n**/*.box\n\n### Patch ###\n\n### Perl ###\n**/!Build/\n**/.last_cover_stats\n**/META.yml\n**/META.json\n**/MYMETA.*\n**/*.pm.tdy\n**/*.bs\n\n# Devel::Cover\n**/cover_db/\n\n# Devel::NYTProf\n**/nytprof.out\n\n# Dizt::Zilla\n**/.build/\n\n# Module::Build\n**/Build\n**/Build.bat\n\n# Module::Install\n**/inc/\n\n# ExtUtils::MakeMaker\n**/blib/\n**/_eumm/\n**/*.gz\n**/Makefile\n**/Makefile.old\n**/MANIFEST.bak\n**/pm_to_blib\n\n### Perl6 ###\n# Gitignore for Perl 6 (http://www.perl6.org)\n# As part of https://github.com/github/gitignore\n\n# precompiled files\n**/.precomp\n**/lib/.precomp\n\n\n### PHPUnit ###\n# Covers PHPUnit\n# Reference: https://phpunit.de/\n\n# Generated files\n**/.phpunit.result.cache\n\n# PHPUnit\n**/app/phpunit.xml\n**/phpunit.xml\n\n# Build data\n**/build/\n\n### PowerShell ###\n# Exclude packaged modules\n\n# Exclude .NET assemblies from source\n\n### Puppet ###\n# gitignore template for Puppet modules\n# website: https://forge.puppet.com/\n\n# Built packages\n**/pkg/*\n\n# Should run on multiple platforms so don't check in\n**/Gemfile.lock\n\n# Tests\n**/spec/fixtures/*\n**/coverage/*\n\n# Third-party\n**/vendor/*\n\n### PuTTY ###\n# Private key\n**/*.ppk\n\n### PyCharm ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### PyCharm+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n# Sonarlint plugin\n\n### PyCharm+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### pydev ###\n**/.pydevproject\n\n### Python ###\n# Byte-compiled / optimized / DLL files\n\n# C extensions\n\n# Distribution / packaging\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n\n# Installer logs\n\n# Unit test / coverage reports\n\n# Translations\n\n# Django stuff:\n\n# Flask stuff:\n\n# Scrapy stuff:\n\n# Sphinx documentation\n\n# PyBuilder\n\n# Jupyter Notebook\n\n# IPython\n\n# pyenv\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n\n# Celery stuff\n\n# SageMath parsed files\n\n# Environments\n\n# Spyder project settings\n\n# Rope project settings\n\n# mkdocs documentation\n\n# mypy\n\n# Pyre type checker\n\n# pytype static type analyzer\n\n# profiling data\n\n### R ###\n# History files\n**/.Rhistory\n**/.Rapp.history\n\n# Session Data files\n**/.RData\n\n# User-specific files\n**/.Ruserdata\n\n# Example code in package build process\n**/*-Ex.R\n\n# Output files from R CMD build\n**/*.tar.gz\n\n# Output files from R CMD check\n**/*.Rcheck/\n\n# RStudio files\n**/.Rproj.user/\n\n# produced vignettes\n**/vignettes/*.html\n**/vignettes/*.pdf\n\n# OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3\n**/.httr-oauth\n\n# knitr and R markdown default cache directories\n**/*_cache/\n**/cache/\n\n# Temporary files created by R markdown\n**/*.utf8.md\n**/*.knit.md\n\n# R Environment Variables\n**/.Renviron\n\n### R.Bookdown Stack ###\n# R package: bookdown caching files\n**/*_files/\n\n### Rails ###\n**/*.rbc\n**/capybara-*.html\n**/.rspec\n**/db/*.sqlite3\n**/db/*.sqlite3-journal\n**/db/*.sqlite3-[0-9]*\n**/public/system\n**/coverage/\n**/spec/tmp\n**/rerun.txt\n**/pickle-email-*.html\n\n# Ignore all logfiles and tempfiles.\n**/log/*\n**/tmp/*\n**/!/log/.keep\n**/!/tmp/.keep\n\n# TODO Comment out this rule if you are OK with secrets being uploaded to the repo\n**/config/initializers/secret_token.rb\n**/config/master.key\n\n# Only include if you have production secrets in this file, which is no longer a Rails default\n# config/secrets.yml\n\n# dotenv, dotenv-rails\n# TODO Comment out these rules if environment variables can be committed\n**/.env.*\n\n## Environment normalization:\n**/.bundle\n**/vendor/bundle\n\n# these should all be checked in to normalize the environment:\n# Gemfile.lock, .ruby-version, .ruby-gemset\n\n# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:\n**/.rvmrc\n\n# if using bower-rails ignore default bower_components path bower.json files\n**/vendor/assets/bower_components\n**/*.bowerrc\n**/bower.json\n\n# Ignore pow environment settings\n**/.powenv\n\n# Ignore Byebug command history file.\n**/.byebug_history\n\n# Ignore node_modules\n\n# Ignore precompiled javascript packs\n**/public/packs\n**/public/packs-test\n**/public/assets\n\n# Ignore yarn files\n**/yarn-error.log\n\n# Ignore uploaded files in development\n**/storage/*\n**/!/storage/.keep\n\n### react ###\n**/.DS_*\n**/**/*.backup.*\n**/**/*.back.*\n\n\n**/*.sublime*\n\n**/psd\n**/thumb\n**/sketch\n\n### ReactNative ###\n# React Native Stack Base\n\n**/.expo\n**/__generated__\n\n### ReactNative.Android Stack ###\n# Built application files\n**/*.aar\n**/*.ap_\n**/*.aab\n\n# Files for the ART/Dalvik VM\n**/*.dex\n\n# Java class files\n\n# Generated files\n**/gen/\n#  Uncomment the following line in case you need and you don't have the release build type files in your app\n# release/\n\n# Gradle files\n\n# Local configuration file (sdk path, etc)\n\n# Proguard folder generated by Eclipse\n**/proguard/\n\n# Log Files\n\n# Android Studio Navigation editor temp files\n**/.navigation/\n\n# Android Studio captures folder\n**/captures/\n\n# IntelliJ\n**/.idea/workspace.xml\n**/.idea/tasks.xml\n**/.idea/gradle.xml\n**/.idea/assetWizardSettings.xml\n**/.idea/dictionaries\n**/.idea/libraries\n# Android Studio 3 in .gitignore file.\n**/.idea/caches\n**/.idea/modules.xml\n# Comment next line if keeping position of elements in Navigation Editor is relevant for you\n**/.idea/navEditor.xml\n\n# Keystore files\n# Uncomment the following lines if you do not want to check your keystore files in.\n#*.jks\n#*.keystore\n\n# External native build folder generated in Android Studio 2.2 and later\n**/.externalNativeBuild\n**/.cxx/\n\n# Google Services (e.g. APIs or Firebase)\n# google-services.json\n\n# Freeline\n**/freeline.py\n**/freeline/\n**/freeline_project_description.json\n\n# fastlane\n**/fastlane/report.xml\n**/fastlane/Preview.html\n**/fastlane/screenshots\n**/fastlane/test_output\n**/fastlane/readme.md\n\n# Version control\n**/vcs.xml\n\n# lint\n**/lint/intermediates/\n**/lint/generated/\n**/lint/outputs/\n**/lint/tmp/\n# lint/reports/\n\n### ReactNative.Buck Stack ###\n**/buck-out/\n**/.buckconfig.local\n**/.buckd/\n**/.buckversion\n**/.fakebuckversion\n\n### ReactNative.Gradle Stack ###\n**/.gradle\n\n# Ignore Gradle GUI config\n**/gradle-app.setting\n\n# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)\n**/!gradle-wrapper.jar\n\n# Cache of project\n**/.gradletasknamecache\n\n# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898\n# gradle/wrapper/gradle-wrapper.properties\n\n### ReactNative.Linux Stack ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n\n# KDE directory preferences\n\n# Linux trash folder which might appear on any partition or disk\n\n# .nfs files are created when an open file is removed but is still being accessed\n\n### ReactNative.Node Stack ###\n# Logs\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\n\n# Runtime data\n\n# Directory for instrumented libs generated by jscoverage/JSCover\n\n# Coverage directory used by tools like istanbul\n\n# nyc test coverage\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n\n# Bower dependency directory (https://bower.io/)\n\n# node-waf configuration\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\n\n# Dependency directories\n\n# TypeScript v1 declaration files\n\n# TypeScript cache\n\n# Optional npm cache directory\n\n# Optional eslint cache\n\n# Microbundle cache\n\n# Optional REPL history\n\n# Output of 'npm pack'\n\n# Yarn Integrity file\n\n# dotenv environment variables file\n\n# parcel-bundler cache (https://parceljs.org/)\n\n# Next.js build output\n\n# Nuxt.js build / generate output\n\n# Gatsby files\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n\n# Serverless directories\n\n# FuseBox cache\n\n# DynamoDB Local files\n\n# TernJS port file\n\n# Stores VSCode versions used for testing VSCode extensions\n\n### ReactNative.Xcode Stack ###\n# Xcode\n#\n# gitignore contributors: remember to update Global/Xcode.gitignore, Objective-C.gitignore & Swift.gitignore\n\n## User settings\n**/xcuserdata/\n\n## compatibility with Xcode 8 and earlier (ignoring not required starting Xcode 9)\n**/*.xcscmblueprint\n**/*.xccheckout\n\n## compatibility with Xcode 3 and earlier (ignoring not required starting Xcode 4)\n**/DerivedData/\n**/*.moved-aside\n**/*.pbxuser\n**/!default.pbxuser\n**/*.mode1v3\n**/!default.mode1v3\n**/*.mode2v3\n**/!default.mode2v3\n**/*.perspectivev3\n**/!default.perspectivev3\n\n## Gcc Patch\n**/*.gcno\n\n### ReactNative.macOS Stack ###\n# General\n\n# Icon must end with two \\r\n**/Icon\r\r\n\n# Thumbnails\n\n# Files that might appear in the root of a volume\n\n# Directories potentially created on remote AFP share\n\n### Redis ###\n# Ignore redis binary dump (dump.rdb) files\n\n**/*.rdb\n\n### ROOT ###\n# ROOT Home Page : https://root.cern.ch/\n# ROOT Used by Experimental Physicists, not necessarily HEP\n# ROOT based on C++\n\n# Files generated by ROOT, observed with v6.xy\n\n**/*.pcm\n\n\n\n### Ruby ###\n**/.config\n**/InstalledFiles\n**/pkg/\n**/spec/reports/\n**/spec/examples.txt\n**/test/tmp/\n**/test/version_tmp/\n**/tmp/\n\n# Used by dotenv library to load environment variables.\n# .env\n\n# Ignore Byebug command history file.\n\n## Specific to RubyMotion:\n**/.dat*\n**/.repl_history\n**/*.bridgesupport\n**/build-iPhoneOS/\n**/build-iPhoneSimulator/\n\n## Specific to RubyMotion (use of CocoaPods):\n# We recommend against adding the Pods directory to your .gitignore. However\n# you should judge for yourself, the pros and cons are mentioned at:\n# https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control\n# vendor/Pods/\n\n## Documentation cache and generated files:\n**/.yardoc/\n**/_yardoc/\n**/doc/\n**/rdoc/\n\n**/.bundle/\n**/lib/bundler/man/\n\n# for a library or gem, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# Gemfile.lock\n# .ruby-version\n# .ruby-gemset\n\n# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:\n\n# Used by RuboCop. Remote config files pulled in from inherit_from directive.\n# .rubocop-https?--*\n\n### Ruby Patch ###\n# Used by RuboCop. Remote config files pulled in from inherit_from directive.\n# .rubocop-https?--*\n\n### Rust ###\n# Generated by Cargo\n# will have compiled files and executables\n\n# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries\n# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html\n**/Cargo.lock\n\n### SBT ###\n# Simple Build Tool\n# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control\n\n**/dist/*\n**/lib_managed/\n**/src_managed/\n**/project/boot/\n**/project/plugins/project/\n**/.history\n**/.lib/\n\n### Scala ###\n\n### Serverless ###\n# Ignore build directory\n**/.serverless\n\n### Sonar ###\n#Sonar generated dir\n**/.sonar/\n\n### SonarQube ###\n# SonarQube ignore files.\n# https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner\n# Sonar Scanner working directories\n**/.sonar/\n**/.scannerwork/\n\n# http://www.sonarlint.org/commandline/\n# SonarLint working directories, configuration files (including credentials)\n**/.sonarlint/\n\n### Spark ###\n**/*#*#\n**/*.#*\n**/*.pyc\n**/*.pyo\n**/.ensime\n**/.ensime_cache/\n**/.ensime_lucene\n**/.generated-mima*\n**/R-unit-tests.log\n**/R/unit-tests.out\n**/R/cran-check.out\n**/R/pkg/vignettes/sparkr-vignettes.html\n**/R/pkg/tests/fulltests/Rplots.pdf\n**/build/*.jar\n**/build/apache-maven*\n**/build/scala*\n**/build/zinc*\n**/cache\n**/checkpoint\n**/conf/*.cmd\n**/conf/*.conf\n**/conf/*.properties\n**/conf/*.sh\n**/conf/*.xml\n**/conf/java-opts\n**/conf/slaves\n**/derby.log\n**/dev/create-release/*final\n**/dev/create-release/*txt\n**/dev/pr-deps/\n**/docs/_site\n**/docs/api\n**/sql/docs\n**/sql/site\n**/lint-r-report.log\n**/log/\n**/logs/\n**/project/build/target/\n**/project/plugins/lib_managed/\n**/project/plugins/project/build.properties\n**/project/plugins/src_managed/\n**/project/plugins/target/\n**/python/lib/pyspark.zip\n**/python/deps\n**/python/test_coverage/coverage_data\n**/python/test_coverage/htmlcov\n**/python/pyspark/python\n**/reports/\n**/scalastyle-on-compile.generated.xml\n**/scalastyle-output.xml\n**/scalastyle.txt\n**/spark-*-bin-*.tgz\n**/spark-tests.log\n**/streaming-tests.log\n**/unit-tests.log\n**/work/\n**/docs/.jekyll-metadata\n\n# For Hive\n**/TempStatsStore/\n**/metastore/\n**/metastore_db/\n**/sql/hive-thriftserver/test_warehouses\n**/warehouse/\n**/spark-warehouse/\n\n# For R session data\n**/.RHistory\n**/*.Rproj\n**/*.Rproj.*\n\n**/.Rproj.user\n\n# For SBT\n**/.jvmopts\n\n\n### Splunk ###\n# gitignore template for Splunk apps\n# documentation: http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Defaultmetaconf\n\n# Splunk local meta file\n**/local.meta\n\n# Splunk local folder\n**/local\n\n### Spreadsheet ###\n**/*.xlr\n**/*.xls\n**/*.xlsx\n\n### SSH ###\n**/**/.ssh/id_*\n**/**/.ssh/*_id_*\n**/**/.ssh/known_hosts\n\n### SublimeText ###\n# Cache files for Sublime Text\n**/*.tmlanguage.cache\n**/*.tmPreferences.cache\n**/*.stTheme.cache\n\n# Workspace files are user-specific\n**/*.sublime-workspace\n\n# Project files should be checked into the repository, unless a significant\n# proportion of contributors will probably not be using Sublime Text\n# *.sublime-project\n\n# SFTP configuration file\n**/sftp-config.json\n\n# Package control specific files\n**/Package Control.last-run\n**/Package Control.ca-list\n**/Package Control.ca-bundle\n**/Package Control.system-ca-bundle\n**/Package Control.cache/\n**/Package Control.ca-certs/\n**/Package Control.merged-ca-bundle\n**/Package Control.user-ca-bundle\n**/oscrypto-ca-bundle.crt\n**/bh_unicode_properties.cache\n\n# Sublime-github package stores a github token in this file\n# https://packagecontrol.io/packages/sublime-github\n**/GitHub.sublime-settings\n\n### SVN ###\n**/.svn/\n\n### Terraform ###\n# Local .terraform directories\n**/**/.terraform/*\n\n# .tfstate files\n**/*.tfstate\n**/*.tfstate.*\n\n# Crash log files\n\n# Ignore any .tfvars files that are generated automatically for each Terraform run. Most\n# .tfvars files are managed as part of configuration and so should be included in\n# version control.\n# example.tfvars\n\n# Ignore override files as they are usually used to override resources locally and so\n# are not checked in\n**/override.tf\n**/override.tf.json\n**/*_override.tf\n**/*_override.tf.json\n\n# Include override files you do wish to add to version control using negated pattern\n# !example_override.tf\n\n# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan\n# example: *tfplan*\n\n### Terragrunt ###\n# terragrunt cache directories\n**/**/.terragrunt-cache/*\n\n### TortoiseGit ###\n# Project-level settings\n**/.tgitconfig\n\n### Vagrant ###\n# General\n**/.vagrant/\n\n# Log files (if you are creating logs in debug mode, uncomment this)\n# *.log\n\n### Vagrant Patch ###\n\n### venv ###\n# Virtualenv\n# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/\n**/[Bb]in\n**/[Ii]nclude\n**/[Ll]ib\n**/[Ll]ib64\n**/[Ll]ocal\n**/[Ss]cripts\n**/pyvenv.cfg\n**/pip-selfcheck.json\n\n### VirtualEnv ###\n# Virtualenv\n# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/\n\n### Julia ###\n# Files generated by invoking Julia with --code-coverage\n**/*.jl.cov\n**/*.jl.*.cov\n\n# Files generated by invoking Julia with --track-allocation\n**/*.jl.mem\n\n# System-specific files and directories generated by the BinaryProvider and BinDeps packages\n# They contain absolute paths specific to the host computer, and so should not be committed\n**/deps/deps.jl\n**/deps/build.log\n**/deps/downloads/\n**/deps/usr/\n**/deps/src/\n\n# Build artifacts for creating documentation generated by the Documenter package\n**/docs/build/\n**/docs/site/\n\n# File generated by Pkg, the package manager, based on a corresponding Project.toml\n# It records a fixed state of all packages used by the project. As such, it should not be\n# committed for packages, but should be committed for applications that require a static\n# environment.\n**/Manifest.toml\n\n### VisualStudioCode ###\n\n### VisualStudioCode Patch ###\n# Ignore all local history of files\n**/.ionide\n\n### vs ###\n## Ignore Visual Studio temporary files, build results, and\n## files generated by popular Visual Studio add-ons.\n##\n## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore\n\n# User-specific files\n**/*.rsuser\n**/*.suo\n**/*.user\n**/*.userosscache\n**/*.sln.docstates\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n**/*.userprefs\n\n# Mono auto generated files\n**/mono_crash.*\n\n# Build results\n**/[Dd]ebug/\n**/[Dd]ebugPublic/\n**/[Rr]elease/\n**/[Rr]eleases/\n**/x64/\n**/x86/\n**/[Aa][Rr][Mm]/\n**/[Aa][Rr][Mm]64/\n**/bld/\n**/[Bb]in/\n**/[Oo]bj/\n**/[Ll]og/\n**/[Ll]ogs/\n\n# Visual Studio 2015/2017 cache/options directory\n**/.vs/\n# Uncomment if you have tasks that create the project's static files in wwwroot\n#wwwroot/\n\n# Visual Studio 2017 auto generated files\n**/Generated\\ Files/\n\n# MSTest test Results\n**/[Tt]est[Rr]esult*/\n**/[Bb]uild[Ll]og.*\n\n# NUnit\n**/*.VisualState.xml\n**/TestResult.xml\n**/nunit-*.xml\n\n# Build Results of an ATL Project\n**/[Dd]ebugPS/\n**/[Rr]eleasePS/\n**/dlldata.c\n\n# Benchmark Results\n**/BenchmarkDotNet.Artifacts/\n\n# .NET Core\n**/project.lock.json\n**/project.fragment.lock.json\n**/artifacts/\n\n# StyleCop\n**/StyleCopReport.xml\n\n# Files built by Visual Studio\n**/*_i.c\n**/*_p.c\n**/*_h.h\n**/*.meta\n**/*.iobj\n**/*.ipdb\n**/*.pgc\n**/*.pgd\n**/*.rsp\n**/*.sbr\n**/*.tlb\n**/*.tli\n**/*.tlh\n**/*.tmp_proj\n**/*_wpftmp.csproj\n**/*.vspscc\n**/*.vssscc\n**/.builds\n**/*.pidb\n**/*.svclog\n**/*.scc\n\n# Chutzpah Test files\n**/_Chutzpah*\n\n# Visual C++ cache files\n**/ipch/\n**/*.aps\n**/*.ncb\n**/*.opendb\n**/*.opensdf\n**/*.cachefile\n**/*.VC.db\n**/*.VC.VC.opendb\n\n# Visual Studio profiler\n**/*.psess\n**/*.vsp\n**/*.vspx\n**/*.sap\n\n# Visual Studio Trace Files\n**/*.e2e\n\n# TFS 2012 Local Workspace\n**/$tf/\n\n# Guidance Automation Toolkit\n**/*.gpState\n\n# ReSharper is a .NET coding add-in\n**/_ReSharper*/\n**/*.[Rr]e[Ss]harper\n**/*.DotSettings.user\n\n# TeamCity is a build add-in\n**/_TeamCity*\n\n# DotCover is a Code Coverage Tool\n**/*.dotCover\n\n# AxoCover is a Code Coverage Tool\n**/.axoCover/*\n**/!.axoCover/settings.json\n\n# Coverlet is a free, cross platform Code Coverage Tool\n**/coverage*[.json, .xml, .info]\n\n# Visual Studio code coverage results\n**/*.coverage\n**/*.coveragexml\n\n# NCrunch\n**/_NCrunch_*\n**/.*crunch*.local.xml\n**/nCrunchTemp_*\n\n# MightyMoose\n**/*.mm.*\n**/AutoTest.Net/\n\n# Web workbench (sass)\n**/.sass-cache/\n\n# Installshield output folder\n**/[Ee]xpress/\n\n# DocProject is a documentation generator add-in\n**/DocProject/buildhelp/\n**/DocProject/Help/*.HxT\n**/DocProject/Help/*.HxC\n**/DocProject/Help/*.hhc\n**/DocProject/Help/*.hhk\n**/DocProject/Help/*.hhp\n**/DocProject/Help/Html2\n**/DocProject/Help/html\n\n# Click-Once directory\n**/publish/\n\n# Publish Web Output\n**/*.[Pp]ublish.xml\n**/*.azurePubxml\n# Note: Comment the next line if you want to checkin your web deploy settings,\n# but database connection strings (with potential passwords) will be unencrypted\n**/*.pubxml\n**/*.publishproj\n\n# Microsoft Azure Web App publish settings. Comment the next line if you want to\n# checkin your Azure Web App publish settings, but sensitive information contained\n# in these scripts will be unencrypted\n**/PublishScripts/\n\n# NuGet Packages\n**/*.nupkg\n# NuGet Symbol Packages\n**/*.snupkg\n# The packages folder can be ignored because of Package Restore\n**/**/[Pp]ackages/*\n# except build/, which is used as an MSBuild target.\n**/!**/[Pp]ackages/build/\n# Uncomment if necessary however generally it will be regenerated when needed\n#!**/[Pp]ackages/repositories.config\n# NuGet v3's project.json files produces more ignorable files\n**/*.nuget.props\n**/*.nuget.targets\n\n# Microsoft Azure Build Output\n**/csx/\n**/*.build.csdef\n\n# Microsoft Azure Emulator\n**/ecf/\n**/rcf/\n\n# Windows Store app package directories and files\n**/AppPackages/\n**/BundleArtifacts/\n**/Package.StoreAssociation.xml\n**/_pkginfo.txt\n**/*.appx\n**/*.appxbundle\n**/*.appxupload\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n**/*.[Cc]ache\n# but keep track of directories ending in .cache\n**/!?*.[Cc]ache/\n\n# Others\n**/ClientBin/\n**/~$*\n**/*.dbmdl\n**/*.dbproj.schemaview\n**/*.jfm\n**/*.pfx\n**/*.publishsettings\n**/orleans.codegen.cs\n\n# Including strong name files can present a security risk\n# (https://github.com/github/gitignore/pull/2483#issue-259490424)\n#*.snk\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components\n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n#bower_components/\n\n# RIA/Silverlight projects\n**/Generated_Code/\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n**/_UpgradeReport_Files/\n**/Backup*/\n**/UpgradeLog*.XML\n**/UpgradeLog*.htm\n**/ServiceFabricBackup/\n**/*.rptproj.bak\n\n# SQL Server files\n**/*.mdf\n**/*.ldf\n**/*.ndf\n\n# Business Intelligence projects\n**/*.rdl.data\n**/*.bim.layout\n**/*.bim_*.settings\n**/*.rptproj.rsuser\n**/*- [Bb]ackup.rdl\n**/*- [Bb]ackup ([0-9]).rdl\n**/*- [Bb]ackup ([0-9][0-9]).rdl\n\n# Microsoft Fakes\n**/FakesAssemblies/\n\n# GhostDoc plugin setting file\n**/*.GhostDoc.xml\n\n# Node.js Tools for Visual Studio\n**/.ntvs_analysis.dat\n\n# Visual Studio 6 build log\n**/*.plg\n\n# Visual Studio 6 workspace options file\n**/*.opt\n\n# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)\n**/*.vbw\n\n# Visual Studio LightSwitch build output\n**/**/*.HTMLClient/GeneratedArtifacts\n**/**/*.DesktopClient/GeneratedArtifacts\n**/**/*.DesktopClient/ModelManifest.xml\n**/**/*.Server/GeneratedArtifacts\n**/**/*.Server/ModelManifest.xml\n**/_Pvt_Extensions\n\n# Paket dependency manager\n**/.paket/paket.exe\n**/paket-files/\n\n# FAKE - F# Make\n**/.fake/\n\n# CodeRush personal settings\n**/.cr/personal\n\n# Python Tools for Visual Studio (PTVS)\n\n# Cake - Uncomment if you are using it\n# tools/**\n# !tools/packages.config\n\n# Tabs Studio\n**/*.tss\n\n# Telerik's JustMock configuration file\n**/*.jmconfig\n\n# BizTalk build output\n**/*.btp.cs\n**/*.btm.cs\n**/*.odx.cs\n**/*.xsd.cs\n\n# OpenCover UI analysis results\n**/OpenCover/\n\n# Azure Stream Analytics local run output\n**/ASALocalRun/\n\n# MSBuild Binary and Structured Log\n**/*.binlog\n\n# NVidia Nsight GPU debugger configuration file\n**/*.nvuser\n\n# MFractors (Xamarin productivity tool) working folder\n**/.mfractor/\n\n# Local History for Visual Studio\n**/.localhistory/\n\n# BeatPulse healthcheck temp database\n**/healthchecksdb\n\n# Backup folder for Package Reference Convert tool in Visual Studio 2017\n**/MigrationBackup/\n\n# Ionide (cross platform F# VS Code tools) working folder\n**/.ionide/\n\n### vscode ###\n\n### Vue ###\n# gitignore template for Vue.js projects\n# Recommended template: Node.gitignore\n\n# TODO: where does this rule come from?\n**/docs/_book\n\n# TODO: where does this rule come from?\n**/test/\n\n### Vuejs ###\n# Recommended template: Node.gitignore\n\n**/npm-debug.log\n**/yarn-error.log\n\n### Waf ###\n# For projects that use the Waf build system: https://waf.io/\n# Dot-hidden on Unix-like systems\n**/.waf-*-*/\n**/.waf3-*-*/\n# Hidden directory on Windows (no dot)\n**/waf-*-*/\n**/waf3-*-*/\n# Lockfile\n**/.lock-waf_*_build\n\n### Windows ###\n# Windows thumbnail cache files\n**/Thumbs.db\n**/Thumbs.db:encryptable\n**/ehthumbs.db\n**/ehthumbs_vista.db\n\n# Dump file\n**/*.stackdump\n\n# Folder config file\n**/[Dd]esktop.ini\n\n# Recycle Bin used on file shares\n**/$RECYCLE.BIN/\n\n# Windows Installer files\n**/*.msix\n\n# Windows shortcuts\n**/*.lnk\n\n### Xcode ###\n# Xcode\n# gitignore contributors: remember to update Global/Xcode.gitignore, Objective-C.gitignore & Swift.gitignore\n\n\n\n\n\n### Xcode Patch ###\n**/*.xcodeproj/*\n**/!*.xcodeproj/project.pbxproj\n**/!*.xcodeproj/xcshareddata/\n**/!*.xcworkspace/contents.xcworkspacedata\n**/**/xcshareddata/WorkspaceSettings.xcsettings\n\n### XcodeInjection ###\n# Code Injection\n# After new code Injection tools there's a generated folder /iOSInjectionProject\n# https://github.com/johnno1962/injectionforxcode\n\n**/iOSInjectionProject/\n\n### Gradle ###\n\n# Ignore Gradle GUI config\n\n# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)\n\n# Cache of project\n\n# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898\n# gradle/wrapper/gradle-wrapper.properties\n\n### Gradle Patch ###\n**/**/build/\n\n### VisualStudio ###\n\n# User-specific files\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n\n# Mono auto generated files\n\n# Build results\n\n# Visual Studio 2015/2017 cache/options directory\n# Uncomment if you have tasks that create the project's static files in wwwroot\n\n# Visual Studio 2017 auto generated files\n\n# MSTest test Results\n\n# NUnit\n\n# Build Results of an ATL Project\n\n# Benchmark Results\n\n# .NET Core\n\n# StyleCop\n\n# Files built by Visual Studio\n\n# Chutzpah Test files\n\n# Visual C++ cache files\n\n# Visual Studio profiler\n\n# Visual Studio Trace Files\n\n# TFS 2012 Local Workspace\n\n# Guidance Automation Toolkit\n\n# ReSharper is a .NET coding add-in\n\n# TeamCity is a build add-in\n\n# DotCover is a Code Coverage Tool\n\n# AxoCover is a Code Coverage Tool\n\n# Coverlet is a free, cross platform Code Coverage Tool\n\n# Visual Studio code coverage results\n\n# NCrunch\n\n# MightyMoose\n\n# Web workbench (sass)\n\n# Installshield output folder\n\n# DocProject is a documentation generator add-in\n\n# Click-Once directory\n\n# Publish Web Output\n# Note: Comment the next line if you want to checkin your web deploy settings,\n# but database connection strings (with potential passwords) will be unencrypted\n\n# Microsoft Azure Web App publish settings. Comment the next line if you want to\n# checkin your Azure Web App publish settings, but sensitive information contained\n# in these scripts will be unencrypted\n\n# NuGet Packages\n# NuGet Symbol Packages\n# The packages folder can be ignored because of Package Restore\n# except build/, which is used as an MSBuild target.\n# Uncomment if necessary however generally it will be regenerated when needed\n# NuGet v3's project.json files produces more ignorable files\n\n# Microsoft Azure Build Output\n\n# Microsoft Azure Emulator\n\n# Windows Store app package directories and files\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n# but keep track of directories ending in .cache\n\n# Others\n\n# Including strong name files can present a security risk\n# (https://github.com/github/gitignore/pull/2483#issue-259490424)\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components\n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n\n# RIA/Silverlight projects\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n\n# SQL Server files\n\n# Business Intelligence projects\n\n# Microsoft Fakes\n\n# GhostDoc plugin setting file\n\n# Node.js Tools for Visual Studio\n\n# Visual Studio 6 build log\n\n# Visual Studio 6 workspace options file\n\n# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)\n\n# Visual Studio LightSwitch build output\n\n# Paket dependency manager\n\n# FAKE - F# Make\n\n# CodeRush personal settings\n\n# Python Tools for Visual Studio (PTVS)\n\n# Cake - Uncomment if you are using it\n# tools/**\n# !tools/packages.config\n\n# Tabs Studio\n\n# Telerik's JustMock configuration file\n\n# BizTalk build output\n\n# OpenCover UI analysis results\n\n# Azure Stream Analytics local run output\n\n# MSBuild Binary and Structured Log\n\n# NVidia Nsight GPU debugger configuration file\n\n# MFractors (Xamarin productivity tool) working folder\n\n# Local History for Visual Studio\n\n# BeatPulse healthcheck temp database\n\n# Backup folder for Package Reference Convert tool in Visual Studio 2017\n\n# Ionide (cross platform F# VS Code tools) working folder\n\n# End of https://www.toptal.com/developers/gitignore/api/ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n"
  },
  {
    "path": ".drone.yml",
    "content": "---\n# XXX: putting this separator further down with code causes a parsing bug in drone lint\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-29 12:05:52 +0000 (Sat, 29 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                D r o n e   C I\n# ============================================================================ #\n\n# https://docs.drone.io/quickstart/cli/\n#\n# https://docs.drone.io/cli/install/\n#\n# brew install drone-cli\n#\n# cd to this directory\n#\n# drone exec [--pipeline default] [--include=thisstep] [--exclude=thatstep]\n\nkind: pipeline\ntype: docker\nname: default\n\nsteps:\n  - name: build\n    image: ubuntu:18.04\n    #environment:\n    #  DEBUG: 1\n    commands:\n      - setup/ci_bootstrap.sh\n      - make init\n      - make ci\n      - make test\n\ntrigger:\n  branch:\n    - master\n"
  },
  {
    "path": ".editorconfig",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-10-31 19:04:34 +0000 (Sat, 31 Oct 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#  to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# http://EditorConfig.org\n\n# stop recursing upwards for other .editorconfig files\nroot = true\n\n# Unix-style newlines with a newline ending every file\n[*]\nindent_size = 4\nindent_style = space\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n[*.go]\nindent_size = 4\nindent_style = tab\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n[Makefile]\nindent_size = 4\nindent_style = tab\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n[{*.md,*.hcl,*.tf,*.tfvars}]\nindent_size = 2\nindent_style = space\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n[*.yml,*.yaml]\nindent_size = 2\nindent_style = space\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n[.*]\nindent_size = 4\nindent_style = space\nend_of_line = lf\ntrim_trailing_whitespace = true\ninsert_final_newline = true\n\n# ============================================================================ #\n#                  Older Stuff, don't think I use this any more\n# ============================================================================ #\n\n# Matches multiple files with brace expansion notation\n# Set default charset\n#[*.{js,py}]\n#charset = utf-8\n\n# Indentation override for all JS under lib directory\n#[lib/**.js]\n#indent_style = space\n#indent_size = 2\n\n# Matches the exact files either package.json or .travis.yml\n#[{package.json,.travis.yml}]\n#indent_style = space\n#indent_size = 2\n\n#[*.xml]\n#indent_style = space\n#indent_size = 2\n"
  },
  {
    "path": ".envrc",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Feb 22 17:42:01 2021 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                  D i r E n v\n# ============================================================================ #\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\n# See Also:\n#\n#   .envrc-aws\n#   .envrc-gcp\n#   .envrc-kubernetes\n\n# direnv stdlib - loads .envrc from parent dir up to /\n#\n# useful to accumulate parent and child directory .envrc settings eg. adding Kubernetes namespace, ArgoCD app etc.\n#\n# bypasses security authorization though - use with care\n#source_up\n#\n# source_up must be loaded before set -u otherwise gets this error:\n#\n#   direnv: loading .envrc\n#   /bin/bash: line 226: $1: unbound variable\n#\n# source_up causes this error is up .envrc is found in parent directories:\n#\n#   direnv: No ancestor .envrc found\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrc=\"$(readlink -f \"${BASH_SOURCE[0]}\")\"\nsrcdir=\"$(cd \"$(dirname \"$src\")\" && pwd)\"\n\n# ============================================================================ #\n#                              P r e - C o m m i t\n# ============================================================================ #\n\n# Automatically install Pre-Commit Git hooks if not already present\n\nif ! type -P pre-commit &>/dev/null; then\n    if uname -s | grep -q Darwin &&\n       type -P brew &>/dev/null; then\n        echo\n        echo \"Pre-commit is not installed - installing now using Homebrew...\"\n        echo\n        brew install pre-commit\n        echo\n    elif type -P pip &>/dev/null; then\n        echo\n        echo \"Pre-commit is not installed - installing now using Pip...\"\n        echo\n        pip install pre-commit\n    fi\nfi\n\nif [ -f .pre-commit-config.yaml ] &&\n   type -P pre-commit &>/dev/null &&\n   git rev-parse --is-inside-work-tree &>/dev/null; then\n    hook=\"$(git rev-parse --show-toplevel)/.git/hooks/pre-commit\"\n    if [ -L \"$hook\" ]; then\n        echo \"Detected symlink hook: \"\n        echo\n        ls -l \"$hook\"\n        echo\n        echo \"Removing\"\n        rm -f \"$hook\"\n    fi\n    if ! [ -f \"$hook\" ]; then\n        echo\n        echo \"Pre-commit hook is not installed in local Git repo checkout - installing now...\"\n        echo\n        pre-commit install\n    fi\nfi\n\n# ============================================================================ #\n#                          D o c k e r   C o m p o s e\n# ============================================================================ #\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\n\n# ============================================================================ #\n#                                  G i t H u b\n# ============================================================================ #\n\n#export GITHUB_ORGANIZATION=HariSekhon\n\n# ============================================================================ #\n#                                 A n s i b l e\n# ============================================================================ #\n\n# use the local repo's ansible.cfg rather than:\n#\n#   $PWD/ansible.cfg\n#   ~/.ansible.cfg\n#   /etc/ansible/ansible.cfg\n#\n# set this in project repos to ensure user environment ANSIBLE_CONFIG doesn't get used\n#export ANSIBLE_CONFIG=\"/path/to/ansible.cfg\"\n\n# ============================================================================ #\n#                              C l o u d f l a r e\n# ============================================================================ #\n\n#export CLOUDFLARE_EMAIL=hari@...\n#export CLOUDFLARE_API_KEY=...  # generate here: https://dash.cloudflare.com/profile/api-tokens\n#export CLOUDFLARE_TOKEN=...    # used by cloudflare_api.sh but not by terraform module\n\n# export the variables for terraform\n#export TF_VAR_cloudflare_email=\"$CLOUDFLARE_EMAIL\"\n#export TF_VAR_cloudflare_api_key=\"$CLOUDFLARE_API_KEY\"  # must be a key, not a token using the link above\n\n# ============================================================================ #\n#                   Load External Envrc Files If Present\n# ============================================================================ #\n\n# XXX: safer to bring all these external .envrc inline if you're worried about changes\n#      to it bypassing 'direnv allow' authorization\nload_if_exists(){\n    # first arg is a path to a .envrc\n    # all other args are passed to the sourcing of .envrc - used by .envrc-kubernetes\n    # to pass the context name 'docker-desktop' to switch to\n    local envrc=\"$1\"\n    shift\n    if ! [[ \"$envrc\" =~ ^/ ]]; then\n        envrc=\"$srcdir/$envrc\"\n    fi\n    if [ -f \"$envrc\" ]; then\n        # prevent looping on symlinks to this .envrc if given\n        if [ \"$(readlink \"$envrc\")\" = \"$src\" ]; then\n            return\n        fi\n        echo\n        echo \"Loading $envrc\"\n        # shellcheck disable=SC1090,SC1091\n        . \"$envrc\" \"$@\"\n    fi\n}\n\n# don't do this it may lead to an infinite loop if 'make link' symlinking ~/.envrc to this repo's .envrc\n# (which I do to keep Python virtual automatically loaded at all times because recent pip on Python refuses\n# to install to system Python)\n#load_if_exists ~/.envrc\n\n# ============================================================================ #\n#                                  P y t h o n\n# ============================================================================ #\n\n#for envrc in \\\n#    .envrc-aws \\\n#    .envrc-gcp \\\n#    .envrc-terraform \\\n#    .envrc-python \\\n#    ; do\n#    load_if_exists \"$envrc\"\n#done\n\nload_if_exists .envrc-python\n\n# ============================================================================ #\n#                                    J a v a\n# ============================================================================ #\n\nload_if_exists .envrc-java\n\n# ============================================================================ #\n#                                     A W S\n# ============================================================================ #\n\n#if [[ \"$PWD\" =~ /aws/ ]]; then\n    load_if_exists .envrc-aws\n#fi\n\n# ============================================================================ #\n#                                     G C P\n# ============================================================================ #\n\n#if [[ \"$PWD\" =~ /gcp/ ]]; then\n    load_if_exists .envrc-gcp\n#fi\n\n# ============================================================================ #\n#                               T e r r a f o r m\n# ============================================================================ #\n\n#if [[ \"$PWD\" =~ /(terra(form)?|tf)(/|$) ]]; then\n    load_if_exists .envrc-terraform\n#fi\n\n# ============================================================================ #\n#                              K u b e r n e t e s\n# ============================================================================ #\n\n#if [ -f \"$srcdir/.envrc-kubernetes\" ]; then\n    load_if_exists .envrc-kubernetes docker-desktop\n#fi\n\n# ============================================================================ #\n#                                    . E n v\n# ============================================================================ #\n\necho\n# read .env too\n#dotenv\n\nload_if_exists .envrc.local\n"
  },
  {
    "path": ".envrc-aws",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-27 12:42:32 +0100 (Tue, 27 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              A W S   D i r E n v\n# ============================================================================ #\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\n# See Also:\n#\n#   .envrc\n#   .envrc-gcp\n#   .envrc-kubernetes\n\n# direnv stdlib - loads .envrc from parent dir up to /\n#\n# useful to accumulate parent and child directory .envrc settings eg. adding Kubernetes namespace, ArgoCD app etc.\n#\n# bypasses security authorization though - use with care\n#source_up\n#\n# source_up must be loaded before set -u otherwise gets this error:\n#\n#   direnv: loading .envrc\n#   /bin/bash: line 226: $1: unbound variable\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -n \"${CI:-}\" ]; then\n    exit 0\nfi\n\n# XXX: Edit - crucial to set to the right environment, the rest of the inferred settings below depend on this\nif [ -z \"${AWS_PROFILE:-}\" ]; then\n    exit 0\nfi\n\naws configure list 2>/dev/null || :\necho\n\n# If not logged in:\n#\n# - and we know the AWS_PROFILE\n# - and AWS_NO_AUTOLOGIN is not set\n# - check for SSO key in config section for this profile\n# - if found then do an automatic 'aws sso login'\n#\nif ! aws sts get-caller-identity --output table; then\n    if [ -n \"${AWS_PROFILE:-}\" ] &&\n       [ -z \"${AWS_NO_AUTOLOGIN:-}\" ]; then\n        # assumes you're not putting a blank line until the next section block\n        #if sed -n \"/profile.*$AWS_PROFILE/,/^[[:space:]]*$/p\" ~/.aws/config | grep -q sso_start_url; then\n        # goes until the next [profile ...] section instead, should be more reliable\n        if sed -n \"/profile.*$AWS_PROFILE/,/^[[:space:]]*\\[.+\\]/p\" ~/.aws/config | grep -q sso_start_url; then\n            echo\n            aws sso login\n        fi\n    fi\nfi\necho\n\n# 'aws sts get-caller-identity --query Account' succeeds in returning the account id\n# from the ~/.aws/config even if 'aws sso login' has expired\nAWS_ACCOUNT_ID=\"$(\n    aws sts get-caller-identity --query Account --output text ||\n    aws configure get sso_account_id ||\n    :\n)\"\necho \"AWS Account ID: $AWS_ACCOUNT_ID\"\nexport AWS_ACCOUNT_ID\necho\n\n# might not have permissions to the Organizations in which case this will error instead of return\nAWS_ACCOUNT=\"$(aws organizations describe-account --account-id \"$AWS_ACCOUNT_ID\" 2>/dev/null)\"\nif [ -n \"$AWS_ACCOUNT\" ]; then\n    echo \"AWS Account: $AWS_ACCOUNT\"\n    export AWS_ACCOUNT\n    echo\nfi\n\nAWS_DEFAULT_REGION=\"$(aws configure get region || :)\"  # use region configured in profile by default\nAWS_DEFAULT_REGION=\"${AWS_DEFAULT_REGION:-eu-west-1}\"  # XXX: Edit default fallback region\nexport AWS_DEFAULT_REGION\necho \"AWS Region: $AWS_DEFAULT_REGION\"\necho\n\nexport AWS_DEFAULT_OUTPUT=json\n\n# XXX: Edit, or remove if only have 1 cluster in account, will auto-determine below\nexport EKS_CLUSTER=\"mycluster\"\n\n# safer but slower\n#eks_clusters=()\n#while IFS='' read -r line; do\n#    eks_clusters+=(\"$line\")\n##done < <(aws eks list-clusters --output=json | jq -r '.clusters[]')\n#done < <(aws eks list-clusters --query 'clusters[]' --output text)\n#if [ \"${#eks_clusters[@]}\" -eq 1 ]; then\n#    export EKS_CLUSTER=\"${eks_clusters[*]}\"\n#fi\n\neks_clusters=\"$(\n    aws eks list-clusters --query 'clusters' --output text |\n    tr '[:space:]' '\\n' |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nif [ -n \"$eks_clusters\" ]; then\n    num_eks_clusters=\"$(grep -c . <<< \"$eks_clusters\")\"\n    echo \"EKS Clusters ($num_eks_clusters):\"\n    echo\n    echo \"$eks_clusters\"\n    echo\n    # If EKS_CLUSTER isn't set and there is only one EKS cluster in this account and region, then use it\n    if [ -z \"${EKS_CLUSTER:-}\" ]; then\n        if [ \"$num_eks_clusters\" = 1 ]; then\n            EKS_CLUSTER=\"$eks_clusters\"\n        fi\n    fi\nelse\n    num_eks_clusters=0\nfi\n\nif [ -n \"${EKS_CLUSTER:-}\" ]; then\n    # kubectl context is easily created by running adjacent aws_kube_creds.sh script first\n    export EKS_CONTEXT=\"arn:aws:eks:$AWS_DEFAULT_REGION:$AWS_ACCOUNT_ID:cluster/$EKS_CLUSTER\"\n\n    if command -v kubectl &>/dev/null; then\n        if ! kubectl config get-clusters | grep -Fxq \"$EKS_CONTEXT\"; then\n            echo \"EKS Cluster '$EKS_CLUSTER' not configured, configuring now\"\n            aws eks update-kubeconfig --name \"$EKS_CLUSTER\"\n            echo\n        fi\n    fi\n\n    # XXX: safer to inline .envrc-kubernetes if you're worried about changes to it bypassing 'direnv allow' authorization\n    # shellcheck disable=SC1090,SC1091\n    . \"$srcdir/.envrc-kubernetes\" \"$EKS_CONTEXT\" ${EKS_NAMESPACE:+\"$EKS_NAMESPACE\"}\nfi\n\nif [ \"$num_eks_clusters\" = 1 ]; then\n    if grep -q '^[[:space:]]*export[[:space:]]*EKS_CLUSTER' .envrc &&\n     ! grep -q \"^export EKS_CLUSTER=$eks_clusters$\" .envrc; then\n        echo\n        echo \"Updating EKS_CLUSTER in .envrc from:\"\n        echo\n        grep '^[[:space:]]*export[[:space:]]*EKS_CLUSTER' .envrc\n        echo\n        echo \"to\"\n        echo\n        echo \"export EKS_CLUSTER=$eks_clusters\"\n        echo\n        perl -pi -e \"s/^\\\\s*export\\s+EKS_CLUSTER=.*/export EKS_CLUSTER=$eks_clusters/\" .envrc\n        echo\n    fi\nfi\n\n# better to load this dynamically from credentials, using functions in .bash.d/aws.sh\n#export AWS_ACCESS_KEY_ID=...\n#export AWS_SECRET_ACCESS_KEY=...\n#export AWS_SESSION_TOKEN=...\n\n#export AWS_CONFIG_FILE=~/.aws/config\n#export AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials\n#export AWS_MAX_ATTEMPTS=3\n\n# to quickly export prefixed AWS environment keys if they exist for simple overrides, see examples below\naws_access_key_env(){\n    env=\"$1\"\n    for key in AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY; do\n        varname=\"${env}_${key}\"\n        if [ -n \"${!varname:-}\" ]; then\n            export \"$key\"=\"${!varname}\"\n        fi\n    done\n}\n\n#aws_access_key_env \"DEV\"\n#aws_access_key_env \"STAGING\"\n#aws_access_key_env \"PROD\"\n#aws_access_key_env \"MGMT\"\n\n# pull the secret using this command whenever you need it:\n#\n#   aws_secret_get.sh \"$JENKINS_ADMIN_PASSWORD_AWS_SECRET\" | copy_to_clipboard.sh\n#\nexport JENKINS_ADMIN_PASSWORD_AWS_SECRET=\"jenkins-admin-password\"\n"
  },
  {
    "path": ".envrc-gcp",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Feb 22 17:42:01 2021 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              G C P   D i r E n v\n# ============================================================================ #\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\n# See Also:\n#\n#   .envrc\n#   .envrc-aws\n#   .envrc-kubernetes\n\n# direnv stdlib - loads .envrc from parent dir up to /\n#\n# useful to accumulate parent and child directory .envrc settings eg. adding Kubernetes namespace, ArgoCD app etc.\n#\n# bypasses security authorization though - use with care\n#source_up\n#\n# source_up must be loaded before set -u otherwise gets this error:\n#\n#   direnv: loading .envrc\n#   /bin/bash: line 226: $1: unbound variable\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -n \"${CI:-}\" ]; then\n    exit 0\nfi\n\n# https://cloud.google.com/sdk/gcloud/reference/config\n\n# If using other services, infer the environment variables to put below by reading:\n#\n#   gcloud topic configurations\n#       or\n#   gcloud config set --help\n\ngcloud_config(){\n    local config=\"${1:-}\"\n    if [ -z \"$config\" ]; then\n        echo \"no config passed to gcloud_config() function\" >&2\n        return 1\n    fi\n    if [ -z \"${CI:-}\" ]; then\n        return\n    fi\n    # don't waste time if not using GCloud SDK, ie. not found in $PATH\n    if type -P gcloud; then\n        # protect from setting this if the config does exist as this can cause auth problems by unsetting the core.account\n        if gcloud config configurations list --format='get(name)' | grep -q \"^$config$\"; then\n            export CLOUDSDK_ACTIVE_CONFIG_NAME=\"$config\"\n        fi\n    fi\n}\n\n#gcloud_config dev\n#gcloud_config staging\n#gcloud_config production\n\n# XXX: Edit\nexport CLOUDSDK_CORE_PROJECT=myproject\n\necho \"CLOUDSDK_CORE_PROJECT=$CLOUDSDK_CORE_PROJECT\"\necho\n\n# XXX: Edit\nexport CLOUDSDK_COMPUTE_REGION=\"${CLOUDSDK_COMPUTE_REGION:-eu-west-2}\"\n\necho \"CLOUDSDK_COMPUTE_REGION=$CLOUDSDK_COMPUTE_REGION\"\necho\n\nREGION=\"$CLOUDSDK_COMPUTE_REGION\"\n\n# you should probably not set CLOUDSDK_COMPUTE_ZONE\n#\n# 'gcloud compute ssh' will auto-determine the zone\n#\n# setting CLOUDSDK_COMPUTE_ZONE explicitly breaks the above command in 2/3 cases due to a VM being in a different zone:\n#\n#    ERROR: (gcloud.compute.ssh) Could not fetch resource:\n#     - The resource 'projects/<MY_PROJECT>/zones/<ZONE>/instances/<VM_NAME>' was not found\n#\n# gcp/gce_ssh.sh script in this repo can work around that if you do set this\n#\n#export CLOUDSDK_COMPUTE_ZONE=\"${REGION}-a\" # or b or c\n\nexport CLOUDSDK_AI_REGION=\"$REGION\"\nexport CLOUDSDK_AI_PLATFORM_REGION=\"$REGION\"\nexport CLOUDSDK_DATAPROC_REGION=\"$REGION\"\nexport CLOUDSDK_DEPLOY_REGION=\"$REGION\"\nexport CLOUDSDK_FILESTORE_REGION=\"$REGION\"\nexport CLOUDSDK_FUNCTIONS_REGION=\"$REGION\"\nexport CLOUDSDK_MEMCACHE_REGION=\"$REGION\"\nexport CLOUDSDK_REDIS_REGION=\"$REGION\"\nexport CLOUDSDK_RUN_REGION=\"$REGION\"\nexport CLOUDSDK_RUN_CLUSTER_LOCATION=\"$REGION\"\nexport CLOUDSDK_VMWARE_REGION=\"$REGION\"\n\n# XXX: Edit\nexport CLOUDSDK_RUN_PLATFORM=managed\n#export CLOUDSDK_RUN_PLATFORM=gke\n#export CLOUDSDK_RUN_PLATFORM=kubernetes\n#export CLOUDSDK_RUN_CLUSTER=mycluster\n\nexport CLOUDSDK_GCLOUDIGNORE_ENABLED=True\n#export CLOUDSDK_BUILDS_USE_KANIKO=True\n\n# XXX: Edit, or remove if only have 1 cluster in project, will auto-determine below\nexport CLOUDSDK_CONTAINER_CLUSTER=mycluster  # GKE cluster name\n\n# safer but slower\n#gke_clusters=()\n#while IFS='' read -r line; do\n#    gke_clusters+=(\"$line\")\n#done < <(gcloud container clusters list --format='get(name)')\n#if [ \"${#gke_clusters[@]}\" -eq 1 ]; then\n#    export CLOUDSDK_CONTAINER_CLUSTER=\"${gke_clusters[*]}\"\n#fi\n\ngke_clusters=\"$(\n    gcloud container clusters list --format='get(name)' |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nif [ -n \"$gke_clusters\" ]; then\n    num_gke_clusters=\"$(grep -c . <<< \"$gke_clusters\")\"\n    echo \"GKE Clusters ($num_gke_clusters):\"\n    echo\n    echo \"$gke_clusters\"\n    echo\n    # If GKE_CLUSTER isn't set and there is only one GKE cluster in this account and region, then use it\n    if [ -z \"${GKE_CLUSTER:-}\" ]; then\n        if [ \"$num_gke_clusters\" = 1 ]; then\n            GKE_CLUSTER=\"$gke_clusters\"\n        fi\n    fi\nelse\n    num_gke_clusters=0\nfi\n\n# alternatively call gke_kube_context() function in .envrc-kubernetes which will do this\n# and comment out auto-running kube_context() on sourcing .envrc-kubernetes\nif [ -n \"${CLOUDSDK_CONTAINER_CLUSTER:-}\" ]; then\n    echo \"CLOUDSDK_CONTAINER_CLUSTER=$CLOUDSDK_CONTAINER_CLUSTER\"\n    echo\n    # kubectl context is easily created by running adjacent aws_kube_creds.sh script first\n\n    export GKE_CONTEXT=\"gke_${CLOUDSDK_CORE_PROJECT}_${CLOUDSDK_COMPUTE_REGION}_${CLOUDSDK_CONTAINER_CLUSTER}\"\n\n    # XXX: safer to inline .envrc-kubernetes if you're worried about changes to it bypassing 'direnv allow' authorization\n    # shellcheck disable=SC1090,SC1091\n    . \"$srcdir/.envrc-kubernetes\" \"$GKE_CONTEXT\" ${GKE_NAMESPACE:+\"$GKE_NAMESPACE\"}\nfi\n\n# pull the secret using this command whenever you need it:\n#\n#   gcp_secret_get.sh \"$JENKINS_ADMIN_PASSWORD_GCP_SECRET\" | copy_to_clipboard.sh\n#\nexport JENKINS_ADMIN_PASSWORD_GCP_SECRET=\"jenkins-admin-password\"\n"
  },
  {
    "path": ".envrc-java",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-20 01:55:20 +0800 (Thu, 20 Mar 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             J a v a   D i r E n v\n# ============================================================================ #\n\n# .envrc to auto-load the virtualenv inside the 'venv' directory if present\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -f .sdkmanrc ]; then\n    if ! [ -f ~/.sdkman/etc/config ] ||\n       ! grep -q '^[[:space:]]*sdkman_auto_env[[:space:]]*=[[:space:]]*true' ~/.sdkman/etc/config; then\n        if [ -f ~/.sdkman/bin/sdkman-init.sh ]; then\n            # shellcheck disable=SC1090\n            . ~/.sdkman/bin/sdkman-init.sh\n        fi\n        # it's not a binary but a function, so no type -P\n        if type sdk &>/dev/null; then\n            sdk env install\n        fi\n    fi\nfi\n\n# read .env too\n#dotenv\n"
  },
  {
    "path": ".envrc-kubernetes",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-25 10:10:53 +0000 (Thu, 25 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                       K u b e r n e t e s   D i r E n v\n# ============================================================================ #\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\n# See Also:\n#\n#   .envrc\n#   .envrc-aws\n#   .envrc-gcp\n\n# direnv stdlib - loads .envrc from parent dir up to /\n#\n# useful to accumulate parent and child directory .envrc settings eg. adding Kubernetes namespace, ArgoCD app etc.\n#\n# bypasses security authorization though - use with care\n#source_up\n#\n# source_up must be loaded before set -u otherwise gets this error:\n#\n#   direnv: loading .envrc\n#   /bin/bash: line 226: $1: unbound variable\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\narg=\"${1:-}\"\nif [ \"${arg##*/}\" = \"${BASH_SOURCE[0]##*/}\" ]; then\n    shift\nfi\n\n# XXX: Edit this - hardcode for localized convenience\nCONTEXT=\"${1:-docker-desktop}\"\n# if set will also set the namespace for extra convenience\nNAMESPACE=\"${2:-}\"\n#NAMESPACE=\"jenkins\"\n\n# function so can place in topdir .envrc and have subdirs 'source_up' or simply . ../.envrc to reuse this code among many .envrc environments\nkube_context(){\n    local context=\"$1\"\n    local namespace=\"${2:-}\"\n    if command -v kubectl &>/dev/null; then\n        local tmpdir=\"/tmp/.kube\"\n\n        mkdir -pv \"$tmpdir\"\n\n        local default_kubeconfig=\"${HOME:-$(cd ~ && pwd)}/.kube/config\"\n        local original_kubeconfig=\"${KUBECONFIG:-$default_kubeconfig}\"\n\n        # reload safety - do not source from new tmpdir - not necessary for direnv but useful for local sourcing tests\n        #if [[ \"$original_kubeconfig\" =~ $tmpdir ]]; then\n        #    echo \"ignoring \\$KUBECONFIG=$original_kubeconfig, using default home location $default_kubeconfig\"\n        #    original_kubeconfig=\"$default_kubeconfig\"\n        #fi\n\n        # isolate the kubernetes context to avoid a race condition affecting any other shells or scripts\n        # epoch is added because $$ and $PPID are direnv sub-processes and may be reused later, so using epoch to add uniqueness\n        local epoch\n        epoch=\"$(date +%s)\"\n        export KUBECONFIG=\"$tmpdir/config.${EUID:-${UID:-$(id -u)}}.$$.$epoch\"\n\n        # load your real kube config to isolated staging area to source the context info\n        local src_kubeconfig=\"\"\n        local kubeconfig_source_locations=\"\n            $original_kubeconfig\n            $default_kubeconfig\n            $PWD/.kube/config\n            /etc/rancher/k3s/k3s.yaml\"\n        for kubeconfig in $kubeconfig_source_locations; do\n            if [ -f \"$kubeconfig\" ]; then\n                src_kubeconfig=\"$kubeconfig\"\n                break\n            fi\n        done\n        if [ -n \"$src_kubeconfig\" ]; then\n            if [ \"$src_kubeconfig\" != \"$KUBECONFIG\" ]; then\n                cp -f -- \"$src_kubeconfig\" \"$KUBECONFIG\"\n            fi\n        else\n            if [[ \"$PWD\" =~ k8|kube ]]; then\n                echo \"WARNING: failed to find one of:\" >&2\n                echo \"$kubeconfig_source_locations\" | sort -u >&2\n                echo >&2\n            fi\n        fi\n\n        # race condition - 'kubectl config get-contexts' fails to find the context and switch in many runs without this sleep\n        context_found=0\n        local i\n        for ((i=0; i < 5; i++)); do\n            # surprisingly unreliable - if kubectl config get-contexts -o name | grep -Fxq \"$context\" can miss even after these succeed\n            #if [ -s \"$KUBECONFIG\" ]; then\n            #if cmp --quiet \"$from_kubeconfig\" \"$KUBECONFIG\"; then\n            if kubectl config get-contexts -o name | grep -Fxq \"$context\"; then\n                context_found=1\n                break\n            fi\n            sleep 0.1\n        done\n\n        # this randomly misses the context, and not even 'sync; sync; sleep 1' is reliable to stop that happening in testing\n        #if kubectl config get-contexts -o name 2>/dev/null | grep -Fxq \"$context\"; then\n        if [ \"$context_found\" = 1 ]; then\n            kubectl config use-context \"$CONTEXT\"\n            echo\n\n            if [ -n \"${namespace:-}\" ]; then\n                kubectl config set-context \"$context\" --namespace \"$namespace\"\n                echo\n            fi\n        fi\n    fi\n}\n\ngke_kube_context(){\n    local CONTEXT\n    for _ in CLOUDSDK_CORE_PROJECT CLOUDSDK_COMPUTE_REGION CLOUDSDK_CONTAINER_CLUSTER; do\n        if [ -z \"${!_}\" ]; then\n            echo \"WARNING: \\$$_ is not set\" >&2\n        fi\n    done\n    # if CLOUDSDK_CONTAINER_CLUSTER and it's generated as a naming convention such as \"${CLOUDSDK_CORE_PROJECT}-${CLOUDSDK_COMPUTE_REGION}\"\n    #export CLOUDSDK_CONTAINER_CLUSTER=\"${CLOUDSDK_CONTAINER_CLUSTER:-${CLOUDSDK_CORE_PROJECT}-${CLOUDSDK_COMPUTE_REGION}}\"\n\n    # the context naming convention for GKE clusters imported via:\n    #\n    #   gcloud container clusters get-credentials \"$cluster\" --zone \"$zone\"\n    #\n    # use gke_kube_creds.sh to auto-populate this for all GKE clusters in the current project\n    # and gcp_foreach_project.sh to do this for all GCP projects. Both scripts are found here:\n    #\n    #   https://github.com/HariSekhon/DevOps-Bash-tools\n    #\n    # should be using a regional cluster\n    CONTEXT=\"gke_${CLOUDSDK_CORE_PROJECT}_${CLOUDSDK_COMPUTE_REGION}_${CLOUDSDK_CONTAINER_CLUSTER}\"\n    # not a zonal cluster\n    #CONTEXT=\"gke_${CLOUDSDK_CORE_PROJECT}_${CLOUDSDK_COMPUTE_ZONE}_${CLOUDSDK_CONTAINER_CLUSTER}\"\n    kube_context \"$CONTEXT\" \"${NAMESPACE:-}\"\n}\n\nkube_context \"$CONTEXT\" \"$NAMESPACE\"\n\n#export ARGOCD_SERVER=\"argocd.mycompany.com\"\n#export ARGOCD_OPTS=\"${ARGOCD_OPTS:-} --grpc-web\"\n#if [ -n \"${ARGOCD_AUTH_TOKEN_MYCOMPANY_OR_ENV:-}\" ]; then\n#    export ARGOCD_AUTH_TOKEN=\"$ARGOCD_AUTH_TOKEN_MYCOMPANY_OR_ENV\"\n#fi\n#export ARGOCD_APP=\"myapp\"\n"
  },
  {
    "path": ".envrc-python",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Feb 22 17:42:01 2021 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           P y t h o n   D i r E n v\n# ============================================================================ #\n\n# .envrc to auto-load the virtualenv inside the 'venv' directory if present\n\n# https://direnv.net/man/direnv-stdlib.1.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# this is necessary because newer versions of pip no longer allow you to install PyPI packages in system-packages by default\nfor venv in \"$PWD/venv\" \"$HOME/venv\"; do\n    if [ -f \"$venv/bin/activate\" ]; then\n        echo\n        echo \"Virtualenv directory found in: $venv\"\n        echo\n        echo \"Activating Virtualenv inside the directory: $venv\"\n\n        # shellcheck disable=SC1091\n        source \"$venv/bin/activate\"\n        break\n    fi\ndone\n\n# read .env too\n#dotenv\n"
  },
  {
    "path": ".envrc-terraform",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Feb 22 17:42:01 2021 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        T e r r a f o r m   D i r E n v\n# ============================================================================ #\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# set this to the local repo's config instead of $HOME/.terraformrc\n#export TF_CLI_CONFIG_FILE=\"$PWD/configs/.terraformrc\"\n\n# if not already set in $HOME/.terraformrc\n#export TF_PLUGIN_CACHE_DIR=\"$HOME/.terraform.d/plugin-cache\"\n\n# XXX: beware that tfenv and tgswitch replace the terraform / terragrunt in the path and this is a race condition between different shells\n#\t   it is not as good as the KUBECONFIG trick done in the .envrc-kubernetes\n\n# would auto-determine the version from the state file, but this commands seems to always return the version of your local binary pulling the state file, not the version from the actual terraform_version field in state file if you see when opening it up in the cloud bucket\n# terraform state pull | jq -r .terraform_version\nexport TERRAFORM_VERSION=1.1.9\n\n# overrides .terraform-version file to make this single source of truth\nexport TFENV_TERRAFORM_VERSION=\"$TERRAFORM_VERSION\"\nexport TFENV_AUTO_INSTALL=true\n\n# Terragrunt\nexport TG_VERSION=0.39.2\n\n# Terragrunt Provide Cache prevents wasting tonnes of space &time re-downloading duplicate 600MB\n# Provider plugins for every Terragrunt module resulting in massive duplication see:\n#\n#   https://terragrunt.gruntwork.io/docs/features/provider-cache-server/\n#\n# Stores plugins in:\n#\n#   $HOME/.cache/terragrunt/providers\n#       or on Mac\n#   $HOME/Library/Caches/terragrunt/providers\n#\nexport TG_PROVIDER_CACHE=1\nexport TG_PROVIDER_CACHE_HOST=127.0.0.1\n#\n#export TG_PROVIDER_CACHE_DIR=\"/new/path/to/cache/dir\"\n#\n# to cache from registries other than registry.terraform.io, registry.opentofu.org\n# eg. if you have your own private registry\n#export TG_PROVIDER_CACHE_REGISTRY_NAMES=\"example1.com,example2.com\"\n\n# if tgswitch is installed, trigger it to use the above TF_VERSION environment variable and switch to the correct version of Terragrunt\n# better than adding the ugly shell hook from the docs - https://github.com/warrensbox/tgswitch?tab=readme-ov-file#get-the-version-from-a-subdirectory\nif type -P tgswitch &>/dev/null; then\n    tgswitch\nfi\n\n# XXX: set these or other variables for Terraform code to find\nexport CLOUDFLARE_EMAIL=hari@...\nexport CLOUDFLARE_API_KEY=...  # generate here: https://dash.cloudflare.com/profile/api-tokens\n#export CLOUDFLARE_TOKEN=...   # used by cloudflare_api.sh but not by terraform module\n\n# export the variables for terraform\nexport TF_VAR_cloudflare_email=\"$CLOUDFLARE_EMAIL\"\nexport TF_VAR_cloudflare_api_key=\"$CLOUDFLARE_API_KEY\"  # must be a key, not a token using the link above\n\n# GITHUB_* environment variables may interfere with GitHub provider, so unset them\nfor env_var in $(env | awk -F= '$1 ~ /GITHUB/ {print $1}'); do\n    unset \"$env_var\"\ndone\n"
  },
  {
    "path": ".git-templates/git-secrets/hooks/commit-msg",
    "content": "#!/usr/bin/env bash\ngit secrets --commit_msg_hook -- \"$@\"\n"
  },
  {
    "path": ".git-templates/git-secrets/hooks/pre-commit",
    "content": "#!/usr/bin/env bash\ngit secrets --pre_commit_hook -- \"$@\"\n"
  },
  {
    "path": ".git-templates/git-secrets/hooks/prepare-commit-msg",
    "content": "#!/usr/bin/env bash\ngit secrets --prepare_commit_msg_hook -- \"$@\"\n"
  },
  {
    "path": ".gitconfig",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2012-01-31 14:08:42 +0000 (Tue, 31 Jan 2012)\n#\n#  vim:ts=4:sts=4:sw=4:et\n\n# configure your user name and email in ~/.gitconfig.local\n#[user]\n#    name = Hari Sekhon\n#    email = harisekhon@gmail.com\n\n[include]\n  # XXX: put your [user] section in $HOME/.gitconfig.local eg.\n  #\n  #  [user]\n  #    name = Hari Sekhon\n  #    email = harisekhon@gmail.com\n  #\n  path = .gitconfig.local\n\n[core]\n    # detects filemode changes\n    filemode = true\n    # defaults to $VISUAL or $EDITOR which is set in .bash.d/env.sh\n    #editor = vim\n    # global .gitignore\n    excludesfile = ~/.gitignore\n    # stops unicode chars coming out as \\xxx and double quoted filenames in git status (used in .bash.d/git.sh git_rm_untracked function)\n    quotePath = false\n\n    #whitespace = trailing-space,space-before-tab\n\n[pull]\n    rebase = false\n\n[push]\n#   default = current\n    default = simple\n\n[alias]\n    name = config --get user.name\n    email = config --get user.email\n    who = !git config -l | grep -E '^user\\\\.(name|email)'\n    whoami = who\n    co  = checkout\n    ci  = commit\n    p   = push\n    st  = status\n    stq = !git_foreach_repo.sh git status | grep --color=no -e \"=======\" -e branch -e GitHub\n    br  = branch\n    ba  = branch -a\n    bav = branch -a -vvv\n    cp  = cherry-pick\n    ls  = ls-files\n    root = rev-parse --show-toplevel\n\n    rem      = remote -v\n    remotes  = remote -v\n    tags     = tag -l\n    branches = branch -a\n    prune-branches = ! git remote prune origin && git branch -vv | cut -c 3- | awk '$4 ~ /gone\\\\]/ {print $1}' | xargs git branch -d\n\n    # the results with multi-origin remotes are not reliable without a git pull first, even when only pulling from the primary GitHub origin\n    in      = ! git pull && git log HEAD..FETCH_HEAD\n    out     = ! git pull && git log FETCH_HEAD..HEAD\n    inp     = ! git pull && git log -p HEAD..FETCH_HEAD\n    outp    = ! git pull && git log -p FETCH_HEAD..HEAD\n    ind     = ! git pull && git diff HEAD..FETCH_HEAD\n    outd    = ! git pull && git diff FETCH_HEAD..HEAD\n    age     = for-each-ref --format '%(authordate:iso) %(refname:short)' --sort=-authordate refs/remotes refs/heads\n\n    unstage = reset HEAD --\n    last = log -1 HEAD\n\n    # Show files ignored by git:\n    ign = ls-files -o -i --exclude-standard\n    ignored = !git clean -ndX | sed -e 's/^Would remove //' | sed 's/^Would skip repository //'\n    untracked = ls-files --others --exclude-standard\n\n    # how to use commands inside git aliases\n    visual = !gitk\n\n    df = diff\n    dc = diff --cached\n    lg = log -p\n    lol = log --graph --decorate --pretty=oneline --abbrev-commit\n    lola = log --graph --decorate --pretty=oneline --abbrev-commit --all\n\n    # avoid diff-so-fancy so we can create patches\n    patch = !git --no-pager diff --no-color\n\n    #ffm     = merge --ff-only\n    #ffp     = pull --ff-only\n    #fp      = fetch --prune\n\n    #mp      = merge --no-commit --no-ff\n    #ma      = merge --abort\n    #dno     = diff --name-only\n    #gone    = !git branch -vv | grep ': gone'\n    #gd      = !git branch -vv | awk '/: gone/ {print $1}' | xargs --no-run-if-empty -n1 git branch -D\n\n[help]\n    # autocorrects git commands and executes the inferred command\n    # dangerous this just autocorrected my git rename to git rebase, lucky it errored out...\n    autocorrect = 0\n\n[homebrew]\n    donationmessage = false\n\n# ============================================================================ #\n#                               G i t   C o l o r\n# ============================================================================ #\n\n# colors: normal, black, red, green, yellow, blue, magenta, cyan, white\n# effects: bold, dim, ul, blink and reverse\n\n# if 2 colours given - 1st is foreground, 2nd is background\n\n[color]\n    ui     = auto\n    diff   = auto\n    grep   = auto\n    # do not set always here, use:\n    # -c color.status=always\n    # for specific overrides, otherwise may break gitci and related functions\n    status = auto\n    branch = auto\n\n[color \"branch\"]\n    current     = green ul\n    local       = yellow\n    remote      = red        # default\n    plain       = white\n\n[color \"diff\"]\n    new         = green\n    old         = red        # default\n    plain       = white\n    whitespace  = yellow reverse\n    func        = yellow\n    #frag        = cyan       # default\n    #meta        = green bold # default\n    # from diff-so-fancy\n    meta = 11\n    frag = magenta bold\n\n[color \"grep\"]\n    context     = white\n    filename    = cyan\n    function    = yellow\n    linenumber  = white\n    match       = white magenta\n    selected    = green\n    separator   = white\n\n[color \"status\"]\n    added       = white blue\n    changed     = magenta\n    untracked   = cyan\n    branch      = magenta blink\n    nobranch    = red blink\n\n# ============================================================================ #\n#                           D i f f - s o - f a n c y\n# ============================================================================ #\n\n# detected if installed and set via $GIT_PAGER in .bash.d/git.sh\n#[core]\n#   pager = diff-so-fancy | less --tabs=4 -RFX\n#[pager]\n    # don't set --pattern, overrides -F and doesn't quit less automatically for short diffs\n    #diff = diff-so-fancy | less --tabs=4 -RFX --pattern '^(Date|added|deleted|modified): '\n    #diff = diff-so-fancy | less --tabs=4 -RFX\n\n    # truncate lines in less, only for 'git blame'\n    #blame = less -S\n\n\n[color \"diff-highlight\"]\n  #oldNormal = red bold\n  oldHighlight = white red  # black doesn't contrast well with red bg, use white\n  #newNormal = green bold\n  newHighlight = white magenta\n\n[diff-so-fancy]\n  stripLeadingSymbols = false\n\n# diff-so-fancy but I prefer most of my old preferences\n#[color \"diff\"]\n#  meta = 11\n#  frag = magenta bold\n#  commit = yellow bold\n#  old = red bold\n#  new = green bold\n#  whitespace = red reverse\n\n# ============================================================================ #\n\n#[difftool \"sourcetree\"]\n#    cmd = opendiff \\\"$LOCAL\\\" \\\"$REMOTE\\\"\n#    path =\n#\n#[mergetool \"sourcetree\"]\n#    cmd = /Applications/SourceTree.app/Contents/Resources/opendiff-w.sh \\\"$LOCAL\\\" \\\"$REMOTE\\\" -ancestor \\\"$BASE\\\" -merge \\\"$MERGED\\\"\n#    trustExitCode = true\n\n#[filter \"media\"]\n#   clean = git media clean %f\n#   smudge = git media smudge %f\n#   required = true\n\n# ============================================================================ #\n#                   C r e d e n t i a l s   H e l p e r s\n# ============================================================================ #\n\n# more specific credential addresses below take priority, so if you want to paste in a GitHub token\n# you will need to comment out the GitHub credential help sections below\n#[credential]\n#    helper = store\n\n# When prompted, enter your username and PAT token, not password, otherwise you'll get this error:\n#\n# remote: Support for password authentication was removed on August 13, 2021.\n# remote: Please see https://docs.github.com/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.\n\n# See credential helpers:\n#\n#   git config --list --show-origin | grep credential\n\n# generated by git/git_remotes_set_https_creds_helpers.sh\n\n[credential \"https://github.com\"]\n    # without the \"sleep 1\" the Git command may miss catching the output and hang instead\n    helper = \"!f() { sleep 1; echo \\\"username=${GITHUB_USER}\\\"; echo \\\"password=${GH_TOKEN:-${GITHUB_TOKEN}}\\\"; }; f\"\n\n[credential \"https://gist.github.com\"]\n    helper = \"!f() { sleep 1; echo \\\"username=${GITHUB_USER}\\\"; echo \\\"password=${GH_TOKEN:-${GITHUB_TOKEN}}\\\"; }; f\"\n\n[credential \"https://gitlab.com\"]\n    helper = \"!f() { sleep 1; echo \\\"password=${GITLAB_TOKEN}\\\"; }; f\"\n\n# XXX: Bitbucket actually needs https://<username>@ regardless otherwise results in 403 errors\n[credential \"https://bitbucket.org\"]\n    #helper = \"!f() { sleep 1; echo \\\"password=${BITBUCKET_TOKEN}\\\"; }; f\"\n    helper = \"!f() { sleep 1; echo \\\"username=${BITBUCKET_USER}\\\"; echo \\\"password=${BITBUCKET_APP_PASSWORD:-${BITBUCKET_TOKEN}}\\\"; }; f\"\n\n[credential \"https://dev.azure.com\"]\n    helper = \"!f() { sleep 1; echo \\\"username=${AZURE_DEVOPS_USER}\\\"; echo \\\"password=${AZURE_DEVOPS_TOKEN}\\\"; }; f\"\n\n# ============================================================================ #\n#                             A W S   S e c r e t s\n# ============================================================================ #\n\n# AWS Secrets prevents committing secrets in to Git\n\n# install git-secrets hooks in any repo initialized or cloned to prevent credential leak\n[init]\n    templateDir = ~/.git-templates/git-secrets\n\n[secrets]\n    providers = git secrets --aws-provider\n    patterns = (A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}\n    patterns = (\\\"|')?(AWS|aws|Aws)?_?(SECRET|secret|Secret)?_?(ACCESS|access|Access)?_?(KEY|key|Key)(\\\"|')?\\\\s*(:|=>|=)\\\\s*(\\\"|')?[A-Za-z0-9/\\\\+=]{40}(\\\"|')?\n    patterns = (\\\"|')?(AWS|aws|Aws)?_?(ACCOUNT|account|Account)_?(ID|id|Id)?(\\\"|')?\\\\s*(:|=>|=)\\\\s*(\\\"|')?[0-9]{4}\\\\-?[0-9]{4}\\\\-?[0-9]{4}(\\\"|')?\n    # doesn't work, doesn't support (?! ) negative lookahead regex\n    #patterns = (\\bhari|sekhon\\b)(!.*@gmail.com)\n    # only applies to contents, not metadata to prevent wrong author commits\n    #patterns = hari|sekhon\n    # These are sample keys so ignore false positives from scanning tools\n    # trivy:ignore:aws-access-key-id\n    allowed = AKIAIOSFODNN7EXAMPLE\n    # trivy:ignore:aws-access-key-id\n    allowed = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY\n    # better to add to private repos .git/config only\n    #allowed = AWS_ACCOUNT_ID\n\n# ============================================================================ #\n#                 G C P   S o u r c e   R e p o s i t o r i e s\n# ============================================================================ #\n\n# GCloud SDK command clone and sets up the repo auth:\n#\n#   gcloud source repos clone \"$repo\" --project=\"$project\"\n#\n# Remote origin:\n#\n#   https://source.developers.google.com/p/$project/r/$repo\n#\n# GCloud SDK adds this to .git/config in a repo cloned via:\n#\n# # having a blank helper before the real help prevents this error when pushing:\n# # bad input: ..........\n#[credential \"https://source.developers.google.com/\"]\n#    helper =\n#    helper = !gcloud auth git-helper --account=hari@<project>.iam.gserviceaccount.com --ignore-unknown $@\n[filter \"lfs\"]\n    clean = git-lfs clean -- %f\n    smudge = git-lfs smudge -- %f\n    process = git-lfs filter-process\n    required = true\n"
  },
  {
    "path": ".gitconfig.local",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2012-01-31 14:08:42 +0000 (Tue, 31 Jan 2012)\n#\n\n[user]\n    name = Hari Sekhon\n    email = harisekhon@gmail.com\n"
  },
  {
    "path": ".github/CODEOWNERS",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2021-11-09 15:14:59 +0000 (Tue, 09 Nov 2021)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners\n\n# Good in theory, to alert on PR changes to these code paths, but for public repos which may be forked and run .github/workflows/fork-update.yaml, this will result in a lot of spam\n\n# Tips:\n#\n# * includes changes under .github/\n# dir/* only matches first level file changes but doesn't recurse\n# dir/ recurses\n#\n# - CODEOWNERS in base branch of PR determines review request\n# - paths are case sensitive\n# - last match wins, use * at top for overall owner then override with more specific teams\n\n#*          @harisekhon                  # username or email address\n#*          @myorg/platform-engineering  # team based is the way to go - team must have Write access to the repo regardless of if individuals have access\n#*          @myorg/devops\n#k8s        @myorg/devops @myorg/sre-team\n#apps/      @myorg/developers\n#apps/dir2  # ignores dir2 as no owner/team specified on this line\n#src/       @myorg/developers\n#docs/      docs@example.com\n#.github/workflows  @ci-cd-team\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE.md",
    "content": "Please be specific about your issue and include debug output from running after setting `export DEBUG=1` in your shell.\n\nYou can anonymize hostnames / FQDNs, IP / MAC addresses, Kerberos principals, email addresses and almost anything else using `anonymize.pl` or the newer `anonymize.py` available in the [DevOps Perl Tools](https://github.com/HariSekhon/DevOps-Perl-tools) and [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) respectively.\n"
  },
  {
    "path": ".github/workflows/actions-allowed.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:04:03 +0000 (Wed, 26 Jan 2022)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                      3rd Party GitHub Actions Allow List\n# ============================================================================ #\n\n# technically should fix to a SHA, but I trust these authors, if I didn't, I wouldn't run their actions at all\n\ncheckmarx/kics-action@*\nfairwindsops/pluto/github-action@*\nmegalinter/megalinter@v5\nreturntocorp/semgrep-action@v1\n"
  },
  {
    "path": ".github/workflows/alpine.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Alpine\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/alpine.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: alpine:latest\n      caches: apk pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/alpine_3.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Alpine 3\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/alpine_3.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: alpine:3\n      caches: apk pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/centos.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: CentOS\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/centos.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: centos:latest\n      caches: yum pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/centos7.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: CentOS 7\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/centos7.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: centos:7\n      caches: yum pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/centos8.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: CentOS 8\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/centos8.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: centos:8\n      caches: yum pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/checkov.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                 C h e c k o v   G i t H u b   W o r k f l o w\n# ============================================================================ #\n\n# Static analysis of Terraform code - publishes report to GitHub Security tab\n\n# https://github.com/bridgecrewio/checkov-action\n\n---\nname: Checkov\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\njobs:\n  checkov:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Checkov\n    uses: HariSekhon/GitHub-Actions/.github/workflows/checkov.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/codeowners.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              C o d e O w n e r s\n# ============================================================================ #\n\n---\nname: CodeOwners\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - CODEOWNERS\n      - .github/CODEOWNERS\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - CODEOWNERS\n      - .github/CODEOWNERS\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  validate:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Validate CODEOWNERS\n    uses: HariSekhon/GitHub-Actions/.github/workflows/codeowners.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/commit_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n\ngit_foreach_repo.sh 'gitu .github/workflows ||:'\n"
  },
  {
    "path": ".github/workflows/debian.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:latest\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_10.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 10\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_10.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:10\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_11.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 11\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_11.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:11\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_12.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 12\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_12.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:12\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_6.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 6\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_6.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:6\n      # causes nodejs errors\n      #caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_7.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 7\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_7.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:7\n      # causes nodejs errors\n      #caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_8.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 8\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_8.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:8\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/debian_9.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Debian 9\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/debian_9.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: debian:9\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/docker_bash_alpine.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-27 18:55:16 +0000 (Thu, 27 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Docker Build (Alpine)\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    # github.event.repository context not available in scheduled workflows\n    if: |\n      github.repository_owner == 'HariSekhon' &&\n      github.ref_type == 'branch' &&\n      ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build.yaml@master\n    with:\n      # GitHub Actions doesn't yet support referencing env context in uses\n      repo_tags: |\n        harisekhon/bash-tools:alpine\n        ghcr.io/harisekhon/bash-tools:alpine\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-alpine\n      debug: ${{ github.event.inputs.debug }}\n    secrets:\n      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}\n      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/docker_bash_centos.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-27 18:55:16 +0000 (Thu, 27 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Docker Build (CentOS)\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    # github.event.repository context not available in scheduled workflows\n    if: |\n      github.repository_owner == 'HariSekhon' &&\n      github.ref_type == 'branch' &&\n      ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build.yaml@master\n    with:\n      # GitHub Actions doesn't yet support referencing env context in uses\n      repo_tags: |\n        harisekhon/bash-tools:centos\n        ghcr.io/harisekhon/bash-tools:centos\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-centos\n      debug: ${{ github.event.inputs.debug }}\n    secrets:\n      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}\n      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/docker_bash_debian.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-27 18:55:16 +0000 (Thu, 27 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Docker Build (Debian)\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    # github.event.repository context not available in scheduled workflows\n    if: |\n      github.repository_owner == 'HariSekhon' &&\n      github.ref_type == 'branch' &&\n      ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build.yaml@master\n    with:\n      # GitHub Actions doesn't yet support referencing env context in uses\n      repo_tags: |\n        harisekhon/bash-tools:debian\n        ghcr.io/harisekhon/bash-tools:debian\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-debian\n      debug: ${{ github.event.inputs.debug }}\n    secrets:\n      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}\n      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/docker_bash_fedora.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-27 18:55:16 +0000 (Thu, 27 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Docker Build (Fedora)\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    # github.event.repository context not available in scheduled workflows\n    if: |\n      github.repository_owner == 'HariSekhon' &&\n      github.ref_type == 'branch' &&\n      ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build.yaml@master\n    with:\n      # GitHub Actions doesn't yet support referencing env context in uses\n      repo_tags: |\n        harisekhon/bash-tools:fedora\n        ghcr.io/harisekhon/bash-tools:fedora\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-fedora\n      debug: ${{ github.event.inputs.debug }}\n    secrets:\n      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}\n      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/docker_bash_ubuntu.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-27 18:55:16 +0000 (Thu, 27 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Docker Build (Ubuntu)\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n      - .github/\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    # github.event.repository context not available in scheduled workflows\n    if: |\n      github.repository_owner == 'HariSekhon' &&\n      github.ref_type == 'branch' &&\n      ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build.yaml@master\n    with:\n      # GitHub Actions doesn't yet support referencing env context in uses\n      repo_tags: |\n        harisekhon/bash-tools:latest\n        harisekhon/bash-tools:ubuntu\n        ghcr.io/harisekhon/bash-tools:latest\n        ghcr.io/harisekhon/bash-tools:ubuntu\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-ubuntu\n      debug: ${{ github.event.inputs.debug }}\n    secrets:\n      DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}\n      DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/dockerhub_status_alpine.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: DockerHub Alpine\n\n#env:\n#  DEBUG: 1\n\non:\n  push:\n    branches:\n      - master\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\njobs:\n  bash_tools_alpine:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/bash-tools --tag alpine\n  tools_alpine:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/tools --tag alpine\n  alpine-github:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/alpine-github\n  github_alpine:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/github --tag alpine\n"
  },
  {
    "path": ".github/workflows/dockerhub_status_centos.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: DockerHub CentOS\n\n#env:\n#  DEBUG: 1\n\non:\n  push:\n    branches:\n      - master\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\njobs:\n  bash_tools_centos:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/bash-tools --tag centos\n  tools_centos:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/tools --tag centos\n  centos-github:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/centos-github\n  github_centos:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/github --tag centos\n"
  },
  {
    "path": ".github/workflows/dockerhub_status_debian.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: DockerHub Debian\n\n#env:\n#  DEBUG: 1\n\non:\n  push:\n    branches:\n      - master\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\njobs:\n  bash_tools_debian:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/bash-tools --tag debian\n  tools_debian:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/tools --tag debian\n  debian-github:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/debian-github\n  github_debian:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/github --tag debian\n"
  },
  {
    "path": ".github/workflows/dockerhub_status_fedora.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: DockerHub Fedora\n\n#env:\n#  DEBUG: 1\n\non:\n  push:\n    branches:\n      - master\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\njobs:\n  bash_tools_fedora:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/bash-tools --tag fedora\n  tools_fedora:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/tools --tag fedora\n  fedora-github:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/fedora-github\n  github_fedora:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/github --tag fedora\n"
  },
  {
    "path": ".github/workflows/dockerhub_status_ubuntu.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: DockerHub Ubuntu\n\n#env:\n#  DEBUG: 1\n\non:\n  push:\n    branches:\n      - master\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\njobs:\n  bash_tools_ubuntu:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/bash-tools --tag ubuntu\n  tools_ubuntu:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/tools --tag ubuntu\n  ubuntu-github:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/ubuntu-github\n  github_ubuntu:\n    name: check\n    timeout-minutes: 1\n    runs-on: ubuntu-latest\n    container: harisekhon/nagios-plugins\n    steps:\n    - name: check\n      run: check_dockerhub_repo_build_status.py --repo harisekhon/github --tag ubuntu\n"
  },
  {
    "path": ".github/workflows/fedora.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Fedora\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/fedora.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: fedora\n      caches: yum pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/fork-sync.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               F o r k   S y n c\n# ============================================================================ #\n\n# For a fork of the original repo, activate to keep it up to date via straight GitHub sync to the default branch\n\n---\nname: Fork Sync\n\non:  # yamllint disable-line rule:truthy\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 */3 * * *'\n\npermissions:\n  contents: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: false\n\njobs:\n  fork_sync:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == true\n    if: github.repository_owner != 'HariSekhon'\n    name: Fork Sync\n    uses: HariSekhon/GitHub-Actions/.github/workflows/fork-sync.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/fork-update-pr.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          F o r k   U p d a t e   P R\n# ============================================================================ #\n\n# For a fork of the original repo, activate to keep its branches up to date via Pull Requests\n#\n# To be used in conjunction with the adjacent fork-sync.yaml which keeps the default branch up to date\n\n---\nname: Fork Update PR\n\non:  # yamllint disable-line rule:truthy\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 10 * * 1'\n\npermissions:\n  contents: write\n  pull-requests: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: false\n\njobs:\n  fork_update_pr:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == true\n    if: github.repository_owner != 'HariSekhon'\n    name: Fork Update PR\n    uses: HariSekhon/GitHub-Actions/.github/workflows/fork-update-pr.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ghcr_bash_ubuntu.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-02-09 18:07:10 +0000 (Wed, 09 Feb 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: GHCR Build (Ubuntu)\n\non:\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - **/*.md\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n  packages: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  docker_build:\n    if: github.event.repository.fork == false && github.ref_type == 'branch' && ( github.ref_name == github.event.repository.default_branch || github.ref_name == 'docker' )\n    name: Docker Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/docker_build_ghcr.yaml@master\n    with:\n      image: bash-tools\n      tags: ubuntu latest\n      dockerfile-repo: HariSekhon/Dockerfiles\n      context: Dockerfiles/devops-bash-tools-ubuntu\n"
  },
  {
    "path": ".github/workflows/grype.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2023-05-13 01:07:56 +0100 (Sat, 13 May 2023)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   G r y p e\n# ============================================================================ #\n\n---\nname: Grype\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  Grype:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Grype\n    uses: HariSekhon/GitHub-Actions/.github/workflows/grype.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/json.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    J S O N\n# ============================================================================ #\n\n# Validate any JSON files found in the repo\n\n---\nname: JSON\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.json'\n      - .github/workflows/json.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.json'\n      - .github/workflows/json.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\njobs:\n  check_json:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Check JSON\n    uses: HariSekhon/GitHub-Actions/.github/workflows/json.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/kics.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-02-01 19:36:08 +0000 (Tue, 01 Feb 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    K i c s\n# ============================================================================ #\n\n---\nname: Kics\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  kics:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Kics\n    uses: HariSekhon/GitHub-Actions/.github/workflows/kics.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/mac.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Mac\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/alpine.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      runs-on: macos-latest\n      caches: brew pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/mac_11.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Mac 11\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/alpine.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      runs-on: macos-11\n      caches: brew pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/mac_12.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Mac 12\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/alpine.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      runs-on: macos-12\n      caches: brew pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/markdown.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2023-04-14 23:53:43 +0100 (Fri, 14 Apr 2023)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                M a r k D o w n\n# ============================================================================ #\n\n---\nname: Markdown\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.md'\n      - .mdlrc\n      - .mdl.rb\n      - .markdownlint.rb\n      - .github/workflows/markdown.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.md'\n      - .mdlrc\n      - .mdl.rb\n      - .markdownlint.rb\n      - .github/workflows/markdown.yaml\n  workflow_dispatch:\n\npermissions:\n  contents: read\n  pull-requests: read\n\njobs:\n  Markdown:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Markdown\n    uses: HariSekhon/GitHub-Actions/.github/workflows/markdown.yaml@master\n"
  },
  {
    "path": ".github/workflows/push_all_repos.sh",
    "content": "#!/usr/bin/env bash\n\ngit_foreach_repo.sh 'push'\n"
  },
  {
    "path": ".github/workflows/pypy2.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: PyPy 2\n\non:\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: PyPy2\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: pypy2\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/pypy3.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: PyPy 3\n\non:\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: PyPy3\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: pypy3\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python2.7.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 2.7\n\non:\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python2.7.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python2.7.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 2.7\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: 2.7\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.10.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.10\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.10.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.10.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.10\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: \"3.10\"\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.11.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.11\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.11.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.11.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.11\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: \"3.11\"\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.6.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.6\n\non:\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.6.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.6.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.6\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: 3.6\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.7.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.7\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.7.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.7.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.7\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: 3.7\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.8.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.8\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.8.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.8.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.8\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: 3.8\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/python3.9.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-04 21:40:04 +0000 (Tue, 04 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Python 3.9\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.9.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.py'\n      - pylib\n      - requirements.txt\n      - .github/workflows/python3.9.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Python 3.9\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      python-version: 3.9\n      caches: apt pip\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/self_hosted.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-04-10 13:31:46 +0100 (Fri, 10 Apr 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nname: Self Hosted\n\n#env:\n#  DEBUG: 1\n\non: # [push]\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - *.md\n  schedule:\n    # * is a special character in YAML so you have to quote this string\n    - cron:  '0 7 * * *'\n\npermissions:\n  contents: read\n\njobs:\n  build:\n    #name: build\n    timeout-minutes: 60\n    runs-on: self-hosted\n    steps:\n    - uses: actions/checkout@v2\n    - uses: actions/cache@v4\n      with:\n        path: ~/.cache/pip\n        key: pip-${{ runner.os }}-${{ runner.arch }}  # -${{ hashFiles('**/requirements.txt') }}\n        restore-keys: |\n          pip-${{ runner.os }}-${{ runner.arch }}\n    - uses: actions/cache@v4\n      with:\n        path: ~/.cpanm\n        key: cpanm-${{ runner.os }}-${{ runner.arch }}  # -${{ hashFiles('**/cpan-requirements.txt') }}\n        restore-keys: |\n          cpanm-${{ runner.os }}-${{ runner.arch }}\n    - name: init\n      run: make init\n    - name: build\n      run: make ci\n    - name: test\n      run: make test\n"
  },
  {
    "path": ".github/workflows/semgrep-cloud.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                  S e m g r e p   C l o u d   W o r k f l o w\n# ============================================================================ #\n\n# Logs results to https://semgrep.dev/\n\n---\nname: Semgrep Cloud\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  semgrep:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Semgrep Cloud\n    uses: HariSekhon/GitHub-Actions/.github/workflows/semgrep-cloud.yaml@master\n    secrets:\n      SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/semgrep.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                 S e m g r e p   G i t H u b   W o r k f l o w\n# ============================================================================ #\n\n# Generates code scanning alerts in GitHub's Security tab -> Code scanning alerts\n\n# https://semgrep.dev/docs/semgrep-ci/sample-ci-configs/#github-actions\n\n---\nname: Semgrep\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  semgrep:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Semgrep GitHub Security Tab\n    uses: HariSekhon/GitHub-Actions/.github/workflows/semgrep.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/shellcheck.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              S h e l l C h e c k\n# ============================================================================ #\n\n# Validate any shell scripts found in the repo\n\n---\nname: ShellCheck\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.sh'\n      - .github/workflows/shellcheck.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.sh'\n      - .github/workflows/shellcheck.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  shellcheck:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: ShellCheck\n    uses: HariSekhon/GitHub-Actions/.github/workflows/shellcheck.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/sonarcloud.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2023-04-14 23:53:43 +0100 (Fri, 14 Apr 2023)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              S o n a r C l o u d\n# ============================================================================ #\n\n---\nname: SonarCloud\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n\npermissions:\n  contents: read\n  pull-requests: read\n\njobs:\n  SonarCloud:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: SonarCloud\n    uses: HariSekhon/GitHub-Actions/.github/workflows/sonarcloud.yaml@master\n    secrets:\n      SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}\n"
  },
  {
    "path": ".github/workflows/sync_to_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncd \"$srcdir\"\n\nsync_file(){\n    local filename=\"$1\"\n    local repo=\"$2\"\n    local dir=\"${3:-}\"\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    dir=\"$(tr '[:upper:]' '[:lower:]' <<< \"$dir\")\"\n    if ! [ -d \"../../../$dir\" ]; then\n        echo \"WARNING: repo dir $dir not found, skipping...\"\n        return 0\n    fi\n    target=\"../../../$dir/.github/workflows/$filename\"\n    if [ -f \"$target.disabled\" ]; then\n        target=\"$target.disabled\"\n    fi\n    if [ -f \"$target\" ] || [ -n \"${NEW:-}\" ]; then\n        targetdir=\"${target%/*}\"\n        mkdir -p -v \"$targetdir\"\n        echo \"syncing $filename -> $target\"\n        perl -p -e \"s/(DevOps-)?Bash-tools/$repo/i\" \"$filename\" > \"$target\"\n        #if [[ \"$repo\" =~ nagios-plugins ]]; then\n        #    timeout=240\n        #    perl -pi -e \"s/(^\\\\s*timeout-minutes:).*/\\\\1 $timeout/\" \"$target\"\n        #    perl -pi -e 's/(^[[:space:]]+make$)/\\1 build zookeeper/' \"$target\"\n        #fi\n    fi\n}\n\nsed 's/#.*//; s/:/ /' ../../setup/repos.txt |\ngrep -v -e bash-tools \\\n        -e github-actions \\\n        -e actions \\\n        -e '^[[:space:]]*$' |\nwhile read -r repo dir; do\n    if [ $# -gt 0 ]; then\n        for filename in \"$@\"; do\n            sync_file \"$filename\" \"$repo\" \"$dir\"\n        done\n    else\n        for filename in *.yaml; do\n            sync_file \"$filename\" \"$repo\" \"$dir\"\n        done\n    fi\ndone\n"
  },
  {
    "path": ".github/workflows/trivy.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-02-02 11:27:37 +0000 (Wed, 02 Feb 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   T r i v y\n# ============================================================================ #\n\n# Scan files in the local repo\n\n---\nname: Trivy\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  trivy:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Trivy\n    uses: HariSekhon/GitHub-Actions/.github/workflows/trivy.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/trivy_image.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-02-02 11:27:37 +0000 (Wed, 02 Feb 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                 T r i v y   D o c k e r   I m a g e   S c a n\n# ============================================================================ #\n\n# Scan the docker image for this repo\n\n---\nname: Trivy Docker Image Scan\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  actions: read\n  contents: read\n  security-events: write\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  trivy:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Trivy Docker Image Scan\n    uses: HariSekhon/GitHub-Actions/.github/workflows/trivy_image.yaml@master\n    with:\n      image: harisekhon/bash-tools\n      severity: ''\n"
  },
  {
    "path": ".github/workflows/ubuntu.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:latest\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_14.04.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu 14.04\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_14.04.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:14.04\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_16.04.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu 16.04\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_16.04.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:16.04\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_18.04.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu 18.04\n\non:\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_18.04.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:18.04\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_20.04.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu 20.04\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_20.04.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:20.04\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_22.04.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: Ubuntu 22.04\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_22.04.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      container: ubuntu:22.04\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/ubuntu_github.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: GitHub Actions Ubuntu\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n    paths-ignore:\n      - '**/*.md'\n      - '.github/workflows/*'\n      - '!.github/workflows/ubuntu_github.yaml'\n      - '**/Dockerfile'\n      - '**/Jenkinsfile'\n      - '**/.envrc*'\n      - .checkov.yaml\n      - .dockerignore\n      - .gcloudignore\n      - .editorconfig\n      - '.gitconfig*'\n      - .gitignore\n      - .grype.yaml\n      - .hound.yml\n      - .terraformignore\n      - Jenkinsfile\n      - .appveyor.yml\n      - .buildkite/pipeline.yml\n      - .circleci/config.yml\n      - .cirrus.yml\n      - .concourse.yml\n      - .drone.yml\n      - .gitlab-ci.yml\n      - .gocd.yml\n      - .scrutinizer.yml\n      - .semaphore/semaphore.yml\n      - .travis.yml\n      - .werckerignore\n      - azure-pipelines.yml\n      - bitbucket-pipelines.yml\n      - buddy.yml\n      - buildspec.yml\n      - cloudbuild.yaml\n      - codefresh.yml\n      - codeship.yml\n      - shippable.yml\n      - wercker.yml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 7 * * *'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  build:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Build\n    uses: HariSekhon/GitHub-Actions/.github/workflows/make.yaml@master\n    with:\n      caches: apt pip cpanm\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/url_links.yaml.disabled",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 14:10:12 +0000 (Wed, 26 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nname: URL Links\n\non:\n  push:\n    branches:\n      - master\n      - main\n  pull_request:\n    branches:\n      - master\n      - main\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  url_links:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: URL Links\n    uses: HariSekhon/GitHub-Actions/.github/workflows/url_links.yaml@master\n    with:\n      ignore_urls_without_dots: 'true'  # any value enables this\n      url_links_ignored: |\n        https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2Fv\n        somerepository.com\n        mycompany\n        https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK\n"
  },
  {
    "path": ".github/workflows/validate.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              V a l i d a t i o n\n# ============================================================================ #\n\n# Run all custom validations against files in the repo\n\n---\nname: Validation\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  pull_request:\n    branches:\n      - master\n      - main\n    paths-ignore:\n      - '**/*.md'\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  #schedule:\n  #  - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  validate:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Validate\n    uses: HariSekhon/GitHub-Actions/.github/workflows/validate.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".github/workflows/xml.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     X M L\n# ============================================================================ #\n\n# Validate any XML files found in the repo\n\n---\nname: XML\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.xml'\n      - .github/workflows/xml.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.xml'\n      - .github/workflows/xml.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\njobs:\n  check_xml:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Check XML\n    uses: HariSekhon/GitHub-Actions/.github/workflows/xml.yaml@master\n"
  },
  {
    "path": ".github/workflows/yaml.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Tue Feb 4 09:53:28 2020 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    Y A M L\n# ============================================================================ #\n\n# Validate any YAML files found in the repo\n\n---\nname: YAML\n\non:  # yamllint disable-line rule:truthy\n  push:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.yml'\n      - '**/*.yaml'\n      - .github/workflows/yaml.yaml\n  pull_request:\n    branches:\n      - master\n      - main\n    paths:\n      - '**/*.yml'\n      - '**/*.yaml'\n      - .github/workflows/yaml.yaml\n  workflow_dispatch:\n    inputs:\n      debug:\n        type: boolean\n        required: false\n        default: false\n  schedule:\n    - cron: '0 0 * * 1'\n\npermissions:\n  contents: read\n\nconcurrency:\n  group: ${{ github.workflow }}-${{ github.ref }}\n  cancel-in-progress: true\n\njobs:\n  check_yaml:\n    # github.event.repository context not available in scheduled workflows\n    #if: github.event.repository.fork == false\n    if: github.repository_owner == 'HariSekhon'\n    name: Check YAML\n    uses: HariSekhon/GitHub-Actions/.github/workflows/yaml.yaml@master\n    with:\n      debug: ${{ github.event.inputs.debug }}\n"
  },
  {
    "path": ".gitignore",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2012-06-24 22:03:34 +0000 (Sun, 24 Jun 2012)\n#\n#  vim: filetype=conf\n#\n\n# ============================================================================ #\n#                              . g i t i g n o r e\n# ============================================================================ #\n\n# https://git-scm.com/docs/gitignore\n\n# ===================================================================\n# Ignores any file in any subdirectory with matching basename or path\n#\n# (can use **/filename but not necessary)\n\n# ===================================================================\n# Once in a while you should check which files have been ignored via:\n#\n#       git status --ignored\n#\n# to check that there aren't some legit files you need to commit\n\n# =================================================================\n# To find out which line is causing a given file to be ignored, do:\n#\n#       git check-ignore -v $filename\n\n*#*#\n*.a\n*.avi\n*.bak\n*.bak.*\n*.bin\n*.bkp\n*.bz2\n*.class\n*.dump\n*.flv\n*.gif\n*.gpg\n*.gz\n*.img\n*.jar\n*.jks\n*.jpeg\n*.jpg\n*.kdb\n*.lock\n*.log\n*.lzo\n*.macports-saved_*\n*.mp3\n*.mp4\n*.mpeg\n*.mpg\n*.o\n*.orig\n*.out\n*.p12\n*.part\n*.pyc\n*.pyo\n*.stderr\n*.stdout\n*.swo\n*.swp\n*.tar\n*.tbz2\n*.tgz\n*.tmp\n*.wmv\n*.zip\n*~\n~*\n\n.a\n.activator/\n.adobe\n.agent.env\n.aliaslists\n.android/\n.anyconnect\n.ApacheDirectoryStudio/\n.atftp_history\n.atom/\n.audacious/.thumbs\n.audacious/accels\n.audacious/log\n.audacious/playlist.xspf\n.awless/\n# AWS Access + Secret keys are stored in here\n.aws/credentials\n.aws/config\n.aws/token\n# SSO Access + Secret Keys & STS are stored in .json files in here\n.aws/cli/cache/\n.aws/shell/\n.bashrc_dynamichosts\n.bashrc_dynamichosts.src\n.bash_history\n.bash_sessions\n.bash_vars\n.boot2docker/\n.cache\n.cassandra/\n.cassandra/cqlshrc\n.cassandra/cqlsh_history\n.cassandra/nodetool_history\n.cbq_history\n.ccm/\n.CFUserTextEncoding\n# Codefresh config contains API Key\n.cfconfig\n.circleci/build_agent_settings.json\n# Circle CI API Token stored in here\n.circleci/cli.yml\n.circleci/update_check.yml\n.Codefresh/agent/\n.compiz\n.conda/\n#.config\n# contains repo_token / COVERALLS_REPO_TOKEN\n.coveralls.yml\n.cpan/build/\n.cpan/FTPstats.yml\n.cpan/histfile\n.cpan/Metadata\n.cpan/sources/\n.cpanm/\n.cups/\n.data\n.dbshell\n.dbus\n.dcos/\n.DCOPserver_*\n.devcenter/.metadata/\n.devcenter/DevCenter/.default/\n.devcenter/DevCenter/.metadata/\n.devcenter/logging/\n.docker_vars\n.docker/machine/\n.dropbox/\n.DS_Store\n.dvdcss/\n# contains things like auth tokens\n#.envrc*\n.erlang.cookie\n.evolution\n.fluxbox/backgrounds\n.fluxbox/BEST\n.fluxbox/best-styles/styles-backup\n.fluxbox/DIVISION2\n.fluxbox/fbrun_history\n.fluxbox/lastwallpaper\n.fluxbox/menu\n.fluxbox/NOBG\n.fluxbox/OTHERTHEMES\n.fluxbox/startup.log\n# contains Concourse bearer auth token\n.flyrc\n# contains GCP credentials for Ruby Fog library (like Boto)\n.fog\n.fontconfig\n.fseventsd\n.gaim/accels\n.gaim/icons\n.gaim/status.xml\n.gconf\n.gconfd\n.gem\n.gimp-*\n.github_actions_runner/\n.gitk\n.gmvault/\n.gnome\n.gnome2\n.gnome2_private\n.gnupg/gpg-agent-info-*\n.gnupg/private-keys-*\n.gnuplot_history\n.gpg-agent.env\n.gradle\n.groovy\n.gstreamer-*\n.gvfs\n.hg\n#.htoprc\n.ICEauthority\n.IdeaIC*\n.idea\n.inkscape-etc/\n.ion3\n.ipython/\n.irb-save-history\n.irb_history\n# contains creds\n.iredisrc\n.ivy2/\n.jline-jython.history\n.kde\n.kodos\n.kube/\n.ldapvi_history\n.lesshst\n.links2\n.local\n.m2/\n.macports/\n.macromedia\n.matplotlib/\n.mcop\n.mcoprc\n.minikube/\n.minishift/\n.minishift.env\n.mozilla\n.mtools/\n.mysql_history\n.nbprofiler/\n.neo4j_shell_history\n.npm/\n.octave_hist\n.openoffice.org\n.openoffice.org2\n.oracle_jre_usage/\n.ovftool.ssldb\n.Qsync/\n.parallel/\n# PostgreSQL password file\n.pgpass\n# stores credentials\n.pig_history\n.pki\n.pentaho/\n.psql_history\n.pulse\n.puppet/\n.pwm3\n.PyCharm*/\n.pylint.d\n.python-eggs/\n.python_history\n.qt\n.qicon\n.qnicon\n# contains usernames and passwords\n.rabbitmqadmin.conf\n.RData\n.recently-used\n.recently-used.xbel\n.rediscli_history\n.Rhistory\n.rnd\n.rstudio-desktop/\n.rbenv/\n# 'rbenv local' $PWD version file\n#.ruby-version\n.sbt/[[:digit:]].[[:digit:]]*/\n.sbt/boot/\n.sbt/preloaded/\n.sbt/repositories\n.sbt/*/plugins/project/target/\n.sbt/*/plugins/target/\n.scala_history\n.sdkman/archives/\n.sdkman/bin/\n.sdkman/candidates/\n.sdkman/etc/config\n.sdkman/ext/\n.sdkman/src/\n.sdkman/tmp/\n.sdkman/var/\n# Semaphore CI - contains auth token\n.sem.yaml\n.serverauth.*\n# $PWD/.serverless local service artifacts\n.serverless/\n# $HOME/.serverless installation\n.serverless/bin/\n.sh_history\n# contains client id and secret for Shpotify\n.shpotify.cfg\n.Skype\n.snowsql\n# Snowflake password stored in plaintext in here\n.snowsql/config\n.spark_history\n.Spotlight-*\n.spumux/\n.sqlite_history\n.sqlline/history\n.ssh/known_hosts\n.ssh-agent.env\n.subversion/auth\n.svn\n.swatch_script.*\n.TemporaryItems\n.terraform.d/\n.terragrunt\n.themes\n.thumbnails\n.tilda/locks/*\n.tmux/\n.tomboy\n.tomboy.log\n.Trash\n.Trashes\n.travis/\n.vagrant\n.vagrant.d/\n.vboxclient-autoresize.pid\n.vboxclient-clipboard.pid\n.vboxclient-seamless.pid\n.vim/\n.viminfo\n.vnc/\n.wapi\n.wget-hsts\n.wine\n.wireshark-etc/\n.wireshark/\n.wmii-*\n.Xauthority\n.xine\n.xmms\n.xsession-errors\n.zenmap-etc/pango/pangorc\n\n# GCP credentials\napplication_default_credentials.json\n*keyfile.json\n*credentials.json\n\nabs-guide.pdf\nandroid-sdks/\nApplications/\nAT.postflight.*\n# contains OAuth token\nauth.json\nbin/altfirewalls\nbin/altnames\nbin/contrib/*\nbin/desktops\nbin/firewalls\nbin/servers\n#bitbucket/\n#bitbucket/*\nboxes/\nBox Documents/\n# contains webhook URL which should not be committed publicly\nbuildkite-pipeline*.json\n.buildkite-pipelines/\nc\nchinook.*sql*\nchinook.psql\nCalibre Library/\ncli_junkie.gif\ndebs/\nDesktop/\nDocuments/\nDownloads/\ndrive/\nDropbox/\neclipse/\nenterprise\nfatpacks/\nfatlib/\nfluxbox-themes\nfluxbox/debian007.jpg\n# internal repos\n#git/\n#git/*\n# public repos - clashes with TeamCity-CI repo's .teamcity/GitHub\n#github/\n#github/*\ngitolite-admin/\ngitroot/\nGNUstep\nGoogle*Drive\ngo/bin/\ngo/src/github.com/\ngoogle-cloud-sdk/\nhadoop-sources/\nhgroot/\nIdeaProjects/\nImages\ninfrastructure/puppetinfrastructure/\n#jython*\nLibrary\nmbox\nmcollective-plugins/\nMovies\nMusic\nnorev\nNS-GUISettings\n*OneDrive*/\nops/\noriginal-tars\noriginals_tars\notherpics\nQsync/\nperl5/\nPictures\nPublic\npuppet.git/\nPycharmProjects/\npytools_checks/\nrpms/\nsetup/mac_settings/\nshm/\nsiege.log\nSites\n# golang libraries\nsrc/github.com\nsrc/golang.org\nsubversion/\nsvnroot/\ntesting/\ntmp.*\n# common vagrant stuff\nvagrant/boxes/\nvagrant/data/\nvagrant/gems/\nvagrant/insecure_private_key\nvagrant/rgloader/\nvagrant/setup_version\nvagrant/tmp/\n# custom stuff found in vagrant\nvagrant/basho_bench/\nvagrant/id_rsa.pub\nvagrant/jce_policy-6\nvagrant/jython\nvagrant/kibana/\nvagrant/lib\nvagrant/mrepo\nvagrant/mx4j-*\nvagrant/mysql-connector-*\nvagrant/thrift-*\nvenv/\nVirtualBox VMs/\nVirtualBoxShared\nvisualvm*\nwindows/bin2\nwordlists/*.lower\nwordlists/hosts.large\nwordlists/hosts.medium\nwordlists/hosts.small\nwordlists/pw.medium\nwordlists/pw.small\nwordlists/pw.tiny\nwordlists/snmp\nwordlists/ultimate.*\nwordlists/users.large\nwordlists/users.medium\nwordlists/users.small\nwordlists/users.tiny\nwordlists/users.weighted\n\n*.doc\n*.docx\n*.xls\n*.xlsx\n*.log\n*.msg\n*.pages\n*.rtf\n*.wpd\n*.wps\n\n# ============================================================================ #\n#                           M o b i l e   B u i l d s\n# ============================================================================ #\n\n# SwiftPackageManager\nPackages\n\n# Obj-C/Swift specific\n*.hmap\n\n# App packaging\n*.ipa\n*.dSYM.zip\n*.dSYM\n\n# Playgrounds\ntimeline.xctimeline\nplayground.xcworkspace\n\n.build/\n\n# Fastlane\nfastlane/report.xml\nfastlane/Preview.html\nfastlane/screenshots/**/*.png\nfastlane/screenshots/**/*.jp*g\nfastlane/test_output\nfastlane/.env.default\n\niOSInjectionProject/\n\n# ixGuard\nixguard-license.txt\nmapping.yml\nprotectionreport.html\nstatistics.yml\ntelemetry_dump.json\n\n# ============================================================================ #\n#\n# Regenerate all sections below in to a single arg for API call via:\n#\n#       grep '[C]reated by https://' .gitignore | sed 's,.*/,,' | tr ',' '\\n' | sort -u | tr '\\n' ',' | sed 's/,$//' | xargs echo gitignore.io_api.sh\n#\n#   eg.\n#\n#       gitignore.io_api.sh ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh >> .gitignore\n#\n# Find new or missing tags you aren't using yet:\n#\n#       grep '[C]reated by https://' .gitignore | sed 's,.*/,,' | tr ',' '\\n' | sort -u | tr '\\n' ',' | sed 's/,$//' | gitignore.io_api.sh missing\n#\n# ============================================================================ #\n\n\n\n# Created by https://www.toptal.com/developers/gitignore/api/ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n# Edit at https://www.toptal.com/developers/gitignore?templates=ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n\n### Ansible ###\n*.retry\n\n### ApacheHadoop ###\n*.iml\n*.ipr\n*.iws\n*.orig\n*.rej\n.idea\n.svn\n.classpath\n.project\n.settings\ntarget\nhadoop-common-project/hadoop-kms/downloads/\nhadoop-hdfs-project/hadoop-hdfs/downloads\nhadoop-hdfs-project/hadoop-hdfs-httpfs/downloads\nhadoop-common-project/hadoop-common/src/test/resources/contract-test-options.xml\nhadoop-tools/hadoop-openstack/src/test/resources/contract-test-options.xml\n\n### AppCode ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n.idea/**/workspace.xml\n.idea/**/tasks.xml\n.idea/**/usage.statistics.xml\n.idea/**/dictionaries\n.idea/**/shelf\n\n# Generated files\n.idea/**/contentModel.xml\n\n# Sensitive or high-churn files\n.idea/**/dataSources/\n.idea/**/dataSources.ids\n.idea/**/dataSources.local.xml\n.idea/**/sqlDataSources.xml\n.idea/**/dynamic.xml\n.idea/**/uiDesigner.xml\n.idea/**/dbnavigator.xml\n\n# Gradle\n.idea/**/gradle.xml\n.idea/**/libraries\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\ncmake-build-*/\n\n# Mongo Explorer plugin\n.idea/**/mongoSettings.xml\n\n# File-based project format\n\n# IntelliJ\nout/\n\n# mpeltonen/sbt-idea plugin\n.idea_modules/\n\n# JIRA plugin\natlassian-ide-plugin.xml\n\n# Cursive Clojure plugin\n.idea/replstate.xml\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\ncom_crashlytics_export_strings.xml\ncrashlytics.properties\ncrashlytics-build.properties\nfabric.properties\n\n# Editor-based Rest Client\n.idea/httpRequests\n\n# Android studio 3.1+ serialized cache file\n.idea/caches/build_file_checksums.ser\n\n### AppCode Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n.idea/**/sonarlint/\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n.idea/**/sonarIssues.xml\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n.idea/**/markdown-navigator.xml\n.idea/**/markdown-navigator-enh.xml\n.idea/**/markdown-navigator/\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n.idea/$CACHE_FILE$\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n.idea/codestream.xml\n\n### AppEngine ###\n# Google App Engine generated folder\nappengine-generated/\n\n### Archive ###\n\n### Mostly from https://en.wikipedia.org/wiki/List_of_archive_formats\n\n## Archiving only\n# The traditional archive format on Unix-like systems, now used mainly for the creation of static libraries.\n*.a\n*.ar\n# RPM files consist of metadata concatenated with (usually) a cpio archive. Newer RPM systems also support other archives, as cpio is becoming obsolete. cpio is also used with initramfs.\n*.cpio\n\n# A self-extracting archive that uses the Bourne shell (sh).\n*.shar\n# A system for storing multiple files. LBR archives typically contained files processed by SQ, or the archive itself was compressed with SQ. LBR archives that were compressed with SQ ended with the extension .LQR\n*.LBR\n# An archive format originally used mainly for archiving and distribution of the exact, nearly-exact, or custom-modified contents of an optical storage medium such as a CD-ROM or DVD-ROM. However, it can be used to archive the contents of other storage media, selected partitions, folders, and/or files. The resulting archive is typically optimized for convenient rendering to (re-)writable CD or DVD media.\n*.iso\n# A library format used primarily on the Commodore 64 and 128 lines of computers. This bears no resemblance to the DOS LBR format. While library files were quick to implement (a number of programs exist to work with them) they are crippled in that they cannot grow with use: once a file has been created it cannot be amended (files added, changed or deleted) without recreating the entire file.\n*.lbr\n# An archive format used by Mozilla for storing binary diffs. Used in conjunction with bzip2.\n*.mar\n# A common archive format used on Unix-like systems. Generally used in conjunction with compressors such as gzip, bzip2, compress or xz to create .tar.gz, .tar.bz2, .tar.Z or tar.xz files.\n*.tar\n\n# Package managers\n# Red Hat Package Manager\n*.rpm\n# Debian package\n*.deb\n# MicroSoft Installer\n*.msi\n*.msm\n*.msp\n# Mozilla package installer\n*.xpi\n# Ruby Package\n*.gem\n\n\n### Archives ###\n# It's better to unpack these files and commit the raw source because\n# git has its own built in compression methods.\n*.7z\n*.jar\n*.rar\n*.zip\n*.gz\n*.gzip\n*.tgz\n*.bzip\n*.bzip2\n*.bz2\n*.xz\n*.lzma\n*.cab\n*.xar\n\n# Packing-only formats\n\n# Package management formats\n*.dmg\n*.egg\n*.txz\n\n### ArchLinuxPackages ###\n*.tar.*\n*.exe\n*.log\n*.log.*\n*.sig\n\npkg/\n# XXX: might conflict with standard java golang src/ directory structure\n#src/\n\n### Audio ###\n*.aif\n*.iff\n*.m3u\n*.m4a\n*.mid\n*.mp3\n*.mpa\n*.ra\n*.wav\n*.wma\n*.ogg\n*.flac\n\n### Autotools ###\n# http://www.gnu.org/software/automake\n\n# XXX: I track a master Makefile.in in DevOps-Bash-tools to inherit in other projects\n#Makefile.in\n/ar-lib\n/mdate-sh\n/py-compile\n/test-driver\n/ylwrap\n.deps/\n\n# http://www.gnu.org/software/autoconf\n\nautom4te.cache\n/autoscan.log\n/autoscan-*.log\n/aclocal.m4\n/compile\n/config.guess\n/config.h.in\n/config.log\n/config.status\n/config.sub\n/configure\n/configure.scan\n/depcomp\n/install-sh\n/missing\n/stamp-h1\n\n# https://www.gnu.org/software/libtool/\n\n/ltmain.sh\n\n# http://www.gnu.org/software/texinfo\n\n/texinfo.tex\n\n# http://www.gnu.org/software/m4/\n\nm4/libtool.m4\nm4/ltoptions.m4\nm4/ltsugar.m4\nm4/ltversion.m4\nm4/lt~obsolete.m4\n\n# Generated Makefile\n# (meta build system like autotools,\n# can automatically generate from config.status script\n# (which is called by configure script))\n#\n# XXX: always want to commit Makefiles\n#Makefile\n\n### Autotools Patch ###\n\n### Backup ###\n*.bak\n*.gho\n*.ori\n*.tmp\n\n### Basic ###\n# Apples Build\n*.build\n*.apples\n\n# Initialized files\n#*.ini  # used by Ansible for inventory.ini\n*.basic\n\n### BitTorrent ###\n*.torrent\n\n### C ###\n# Prerequisites\n# XXX: clashes with .bash.d/* and .conf.d/* type stuff\n#*.d\n\n# Object files\n*.o\n*.ko\n*.obj\n*.elf\n\n# Linker output\n*.ilk\n*.map\n*.exp\n\n# Precompiled Headers\n*.gch\n*.pch\n\n# Libraries\n*.lib\n*.la\n*.lo\n\n# Shared objects (inc. Windows DLLs)\n*.dll\n*.so\n*.so.*\n*.dylib\n\n# Executables\n*.out\n*.app\n*.i*86\n*.x86_64\n*.hex\n\n# Debug files\n*.dSYM/\n*.su\n*.idb\n*.pdb\n\n# Kernel Module Compile Results\n# XXX: *.mod* changed to avoid clashing with Golang's new module system go.mod\n*.mod?*\n.tmp_versions/\nmodules.order\nModule.symvers\nMkfile.old\ndkms.conf\n\n### Zsh ###\n# Zsh compiled script + zrecompile backup\n*.zwc\n*.zwc.old\n\n# Zsh completion-optimization dumpfile\n*zcompdump*\n\n# Zsh zcalc history\n.zcalc_history\n\n# A popular plugin manager's files\n._zplugin\n.zplugin_lstupd\n\n# zdharma/zshelldoc tool's files\nzsdoc/data\n\n# robbyrussell/oh-my-zsh/plugins/per-directory-history plugin's files\n# (when set-up to store the history in the local directory)\n.directory_history\n\n# MichaelAquilina/zsh-autoswitch-virtualenv plugin's files\n# (for Zsh plugins using Python)\n.venv\n\n# Zunit tests' output\n/tests/_output/*\n!/tests/_output/.gitkeep\n\n### C++ ###\n# Prerequisites\n\n# Compiled Object files\n*.slo\n\n# Precompiled Headers\n\n# Compiled Dynamic libraries\n\n# Fortran module files\n# XXX: would conflict with Golang's go.mod\n#*.mod\n*.smod\n\n# Compiled Static libraries\n*.lai\n\n# Executables\n\n### certificates ###\n*.pem\n*.key\n*.crt\n*.cer\n*.priv\n\n### Cloud9 ###\n# Cloud9 IDE - http://c9.io\n.c9revisions\n.c9\n\n### ChefCookbook ###\n.vagrant\n/cookbooks\n\n# Bundler\n# XXX: would prevent tracking ~/bin scripts\n#bin/*\n.bundle/*\n\n.kitchen/\n.kitchen.local.yml\n.kitchen.*.local.yml\nkitchen.local.yml\nkitchen.*.local.yml\n\n### Code ###\n.vscode/*\n!.vscode/settings.json\n!.vscode/tasks.json\n!.vscode/launch.json\n!.vscode/extensions.json\n*.code-workspace\n\n### Code-Java ###\n# Language Support for Java(TM) by Red Hat extension for Visual Studio Code - https://marketplace.visualstudio.com/items?itemName=redhat.java\n\nfactoryConfiguration.json\n\n### Clojure ###\n# XXX: always want to commit pom.xml\n#pom.xml\npom.xml.asc\n*.class\n# XXX: would break lib submodule updates\n#/lib/\n/classes/\n/target/\n/checkouts/\n.lein-deps-sum\n.lein-repl-history\n.lein-plugins/\n.lein-failures\n.nrepl-port\n.cpcache/\n\n### Compressed ###\n*.pkg\n*.sit\n*.sitx\n*.zipx\n\n### CompressedArchive ###\n\n\n## Archiving and compression\n# Open source file format. Used by 7-Zip.\n# Mac OS X, restoration on different platforms is possible although not immediate \tYes \tBased on 7z. Preserves Spotlight metadata, resource forks, owner/group information, dates and other data which would be otherwise lost with compression.\n*.s7z\n# Old archive versions only \tProprietary format\n*.ace\n# A format that compresses and doubly encrypt the data (AES256 and CAS256) avoiding brute force attacks, also hide files in an AFA file. It has two ways to safeguard data integrity and subsequent repair of the file if has an error (repair with AstroA2P (online) or Astrotite (offline)).\n*.afa\n# A mainly Korean format designed for very large archives.\n*.alz\n# Android application package (variant of JAR file format).\n*.apk\n# ??\n*.arc\n# Originally DOS, now multiple\n*.arj\n# Open archive format, used by B1 Free Archiver (http://dev.b1.org/standard/archive-format.html)\n*.b1\n# Binary Archive with external header\n*.ba\n# Proprietary format from the ZipTV Compression Components\n*.bh\n# The Microsoft Windows native archive format, which is also used by many commercial installers such as InstallShield and WISE.\n# Originally DOS, now DOS and Windows \tCreated by Yaakov Gringeler; released last in 2003 (Compressia 1.0.0.1 beta), now apparently defunct. Free trial of 30 days lets user create and extract archives; after that it is possible to extract, but not to create.\n*.car\n# Open source file format.\n*.cfs\n# Compact Pro archive, a common archiver used on Mac platforms until about Mac OS 7.5.x. Competed with StuffIt; now obsolete.\n*.cpt\n# Windows, Unix-like, Mac OS X Open source file format. Files are compressed individually with either gzip, bzip2 or lzo.\n*.dar\n# DiskDoubler \tMac OS \t\t\tobsolete\n*.dd\n# ??\n*.dgc\n# Apple Disk Image upports \"Internet-enabled\" disk images, which, once downloaded, are automatically decompressed, mounted, have the contents extracted, and thrown away. Currently, Safari is the only browser that supports this form of extraction; however, the images can be manually extracted as well. This format can also be password-protected or encrypted with 128-bit or 256-bit AES encryption.\n# Enterprise Java Archive archive\n*.ear\n# ETSoft compressed archive\n# The predecessor of DGCA.\n*.gca\n# Originally DOS \tYes, but may be covered by patents \tDOS era format; uses arithmetic/Markov coding\n*.ha\n# MS Windows \tHKI\n*.hki\n# Produced by ICEOWS program. Excels at text file compression.\n*.ice\n# Java archive, compatible with ZIP files\n# Open sourced archiver with compression using the PAQ family of algorithms and optional encryption.\n*.kgb\n# Originally DOS, now multiple \tMultiple \tYes \tThe standard format on Amiga.\n*.lzh\n*.lha\n# Archiver originally used on The Amiga. Now copied by Microsoft to use in their .cab and .chm files.\n*.lzx\n# file format from NoGate Consultings, a rival from ARC-Compressor.\n*.pak\n# A disk image archive format that supports several compression methods as well as splitting the archive into smaller pieces.\n*.partimg\n# An experimental open source packager (http://mattmahoney.net/dc)\n*.paq*\n# Open source archiver supporting authenticated encryption, volume spanning, customizable object level and volume level integrity checks (form CRCs to SHA-512 and Whirlpool hashes), fast deflate based compression\n*.pea\n# The format from the PIM - a freeware compression tool by Ilia Muraviev. It uses an LZP-based compression algorithm with set of filters for executable, image and audio files.\n*.pim\n# PackIt \tMac OS \t\t\tobsolete\n*.pit\n# Used for data in games written using the Quadruple D library for Delphi. Uses byte pair compression.\n*.qda\n# A proprietary archive format, second in popularity to .zip files.\n# The format from a commercial archiving package. Odd among commercial packages in that they focus on incorporating experimental algorithms with the highest possible compression (at the expense of speed and memory), such as PAQ, PPMD and PPMZ (PPMD with unlimited-length strings), as well as a proprietary algorithms.\n*.rk\n# Self Dissolving ARChive \tCommodore 64, Commodore 128 \tCommodore 64, Commodore 128 \tYes \tSDAs refer to Self Dissolving ARC files, and are based on the Commodore 64 and Commodore 128 versions of ARC, originally written by Chris Smeets. While the files share the same extension, they are not compatible between platforms. That is, an SDA created on a Commodore 64 but run on a Commodore 128 in Commodore 128 mode will crash the machine, and vice versa. The intended successor to SDA is SFX.\n*.sda\n# A pre-Mac OS X Self-Extracting Archive format. StuffIt, Compact Pro, Disk Doubler and others could create .sea files, though the StuffIt versions were the most common.\n*.sea\n# Scifer Archive with internal header\n*.sen\n# Commodore 64, Commodore 128 \tSFX is a Self Extracting Archive which uses the LHArc compression algorithm. It was originally developed by Chris Smeets on the Commodore platform, and runs primarily using the CS-DOS extension for the Commodore 128. Unlike its predecessor SDA, SFX files will run on both the Commodore 64 and Commodore 128 regardless of which machine they were created on.\n*.sfx\n# An archive format designed for the Apple II series of computers. The canonical implementation is ShrinkIt, which can operate on disk images as well as files. Preferred compression algorithm is a combination of RLE and 12-bit LZW. Archives can be manipulated with the command-line NuLib tool, or the Windows-based CiderPress.\n*.shk\n# A compression format common on Apple Macintosh computers. The free StuffIt Expander is available for Windows and OS X.\n# The replacement for the .sit format that supports more compression methods, UNIX file permissions, long file names, very large files, more encryption options, data specific compressors (JPEG, Zip, PDF, 24-bit image, MP3). The free StuffIt Expander is available for Windows and OS X.\n# A royalty-free compressing format\n*.sqx\n# The \"tarball\" format combines tar archives with a file-based compression scheme (usually gzip). Commonly used for source and binary distribution on Unix-like platforms, widely available elsewhere.\n*.tar.gz\n*.tar.Z\n*.tar.bz2\n*.tbz2\n*.tar.lzma\n*.tlz\n# UltraCompressor 2.3 was developed to act as an alternative to the then popular PKZIP application. The main feature of the application is its ability to create large archives. This means that compressed archives with the UC2 file extension can hold almost 1 million files.\n*.uc\n*.uc0\n*.uc2\n*.ucn\n*.ur2\n*.ue2\n# Based on PAQ, RZM, CSC, CCM, and 7zip. The format consists of a PAQ, RZM, CSC, or CCM compressed file and a manifest with compression settings stored in a 7z archive.\n*.uca\n# A high compression rate archive format originally for DOS.\n*.uha\n# Web Application archive (Java-based web app)\n*.war\n# File-based disk image format developed to deploy Microsoft Windows.\n*.wim\n# XAR\n# Native format of the Open Source KiriKiri Visual Novel engine. Uses combination of block splitting and zlib compression. The filenames and pathes are stored in UTF-16 format. For integrity check, the Adler-32 hashsum is used. For many commercial games, the files are encrypted (and decoded on runtime) via so-called \"cxdec\" module, which implements xor-based encryption.\n*.xp3\n# Yamazaki zipper archive. Compression format used in DeepFreezer archiver utility created by Yamazaki Satoshi. Read and write support exists in TUGZip, IZArc and ZipZag\n*.yz1\n# The most widely used compression format on Microsoft Windows. Commonly used on Macintosh and Unix systems as well.\n# application/x-zoo \tzoo \tMultiple \tMultiple \tYes\n*.zoo\n# Journaling (append-only) archive format with rollback capability. Supports deduplication and incremental update based on last-modified dates. Multi-threaded. Compresses in LZ77, BWT, and context mixing formats. Open source.\n*.zpaq\n# Archiver with a compression algorithm based on the Burrows-Wheeler transform method.\n*.zz\n\n\n### Compression ###\n\n### From https://en.wikipedia.org/wiki/List_of_archive_formats\n\n## Compression only\n# An open source, patent- and royalty-free compression format. The compression algorithm is a Burrows-Wheeler transform followed by a move-to-front transform and finally Huffman coding\n# Old compressor for QNX4 OS. The compression algorithm is a modified LZSS, with an adaptive Huffman coding.\n*.F\n# GNU Zip, the primary compression format used by Unix-like systems. The compression algorithm is DEFLATE.\n# An alternate LZMA algorithm implementation, with support for checksums and ident bytes.\n*.lz\n# The LZMA compression algorithm as used by 7-Zip\n# An implementation of the LZO data compression algorithm\n*.lzo\n# A compression program designed to do particularly well on very large files containing long distance redundancy.\n*.rz\n# Windows compress/decompress- Linux and Mac OS X decompress only \tA compression program designed to do high compression on SF2 files (SoundFont)\n*.sfark\n# A compression format invented by Google and open-sourced in 2011. Snappy aims for very high speeds, reasonable compression, and maximum stability rather than maximum compression or compatibility with any other compression library.\n*.sz\n# Squeeze: A program which compressed files. A file which was \"squeezed\" had the middle initial of the name changed to \"Q\", so that a squeezed text file would end with .TQT, a squeezed executable would end with .CQM or .EQE. Typically used with .LBR archives, either by storing the squeezed files in the archive, or by storing the files decompressed and then compressing the archive, which would have a name ending in \".LQR\".\n# XXX: this prevents .sql files from being tracked\n#*.?Q?\n# A compression program written by Steven Greenberg implementing the LZW algorithm. For several years in the CP/M world when no implementation was available of ARC, CRUNCHed files stored in .LBR archives were very popular. CRUNCH's implementation of LZW had a somewhat unique feature of modifying and occasionally clearing the code table in memory when it became full, resulting in a few percent better compression on many files.\n*.?Z?\n# A compression format using LZMA2 to yield very high compression ratios.\n# The traditional Huffman coding compression format.\n*.z\n# The traditional LZW compression format.\n*.Z\n# Joke compression program, actually increasing file size\n*.infl\n# Compression format(s) used by some DOS and Windows install programs. MS-DOS includes expand.exe to decompress its install files. The compressed files are created with a matching compress.exe command. The compression algorithm is LZSS.\n*.??_\n\n\n### CMake ###\nCMakeLists.txt.user\nCMakeCache.txt\nCMakeFiles\nCMakeScripts\nTesting\ncmake_install.cmake\ninstall_manifest.txt\ncompile_commands.json\nCTestTestfile.cmake\n_deps\n\n### CMake Patch ###\n# External projects\n*-prefix/\n\n### Database ###\n*.accdb\n*.db\n*.dbf\n*.mdb\n*.sqlite3\n\n### DataRecovery ###\n\n\n## Data recovery\n# File format used by dvdisaster to be used for data recovery when discs become damaged or partially unreadable.\n*.ecc\n# File format used in conjunction with any archive format to provide redundancy and data recovery, most often in newsgroup distribution of binary files.\n*.par\n*.par2\n\n\n### Diff ###\n*.patch\n*.diff\n\n### direnv ###\n.direnv\n#.envrc\n\n### CodeBlocks ###\n# specific to CodeBlocks IDE\n*.layout\n*.depend\n# generated directories\n# XXX: would prevent tracking ~/bin scripts\n#bin/\nobj/\n\n### DocFx ###\n.cache\n/**/_site/\n\n### Docpress ###\n# docpress documentation generator: https://docpress.github.io/index.html\n\n_docpress/\n\n### Docz ###\n.docz\n\n\n### dotenv ###\n.env\n\n### DotfilesSh ###\nlocal-patch\npatched-src\n\n### DotSettings ###\n*.DotSettings\n\n### Dropbox ###\n# Dropbox settings and caches\n.dropbox\n.dropbox.attr\n.dropbox.cache\n\n### Eclipse ###\n.metadata\ntmp/\n*.swp\n*~.nib\nlocal.properties\n.settings/\n.loadpath\n.recommenders\n\n# External tool builders\n.externalToolBuilders/\n\n# Locally stored \"Eclipse launch configurations\"\n*.launch\n\n# PyDev specific (Python IDE for Eclipse)\n*.pydevproject\n\n# CDT-specific (C/C++ Development Tooling)\n.cproject\n\n# CDT- autotools\n.autotools\n\n# Java annotation processor (APT)\n.factorypath\n\n# PDT-specific (PHP Development Tools)\n.buildpath\n\n# sbteclipse plugin\n.target\n\n# Tern plugin\n.tern-project\n\n# TeXlipse plugin\n.texlipse\n\n# STS (Spring Tool Suite)\n.springBeans\n\n# Code Recommenders\n.recommenders/\n\n# Annotation Processing\n.apt_generated/\n.apt_generated_test/\n\n# Scala IDE specific (Scala & Java development for Eclipse)\n.cache-main\n.scala_dependencies\n.worksheet\n\n# Uncomment this line if you wish to ignore the project description file.\n# Typically, this file would be tracked if it contains build/dependency configurations:\n#.project\n\n### Eclipse Patch ###\n# Spring Boot Tooling\n.sts4-cache/\n\n### Data ###\n# XXX: would prevent tracking sample test csv data\n#*.csv\n*.dat\n*.efx\n*.gbr\n*.pps\n*.ppt\n*.pptx\n*.sdf\n*.tax2010\n*.vcf\n# XXX: would prevent tracking pom.xml\n#*.xml\n\n### Erlang ###\n.eunit\n*.beam\n*.plt\nerl_crash.dump\n.concrete/DEV_MODE\n\n# rebar 2.x\n.rebar\nrel/example_project\nebin/*.beam\ndeps\n\n# rebar 3\n.rebar3\n_build/\n_checkouts/\n\n### Executable ###\n# XXX: .cgi scripts are old but we might want to track them\n#*.cgi\n*.com\n*.gadget\n*.pif\n# XXX: would prevent tracking Windows Scripting files\n#*.vb\n#*.wsf\n\n### Firebase ###\n**/node_modules/*\n**/.firebaserc\n\n### Firebase Patch ###\n.runtimeconfig.json\n.firebase/\n\n### Flask ###\ninstance/*\n!instance/.gitignore\n.webassets-cache\n\n### Flask.Python Stack ###\n# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\n# XXX: breaks lib submodule updates\n#lib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\npip-wheel-metadata/\nshare/python-wheels/\n*.egg-info/\n.installed.cfg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\nnosetests.xml\ncoverage.xml\n*.cover\n*.py,cover\n.hypothesis/\n.pytest_cache/\npytestdebug.log\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\nlocal_settings.py\ndb.sqlite3\ndb.sqlite3-journal\n\n# Flask stuff:\ninstance/\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\ndoc/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n#Pipfile.lock\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n__pypackages__/\n\n# Celery stuff\ncelerybeat-schedule\ncelerybeat.pid\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\npythonenv*\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\n# pytype static type analyzer\n.pytype/\n\n# profiling data\n.prof\n\n### Git ###\n# Created by git for backups. To disable backups in Git:\n# $ git config --global mergetool.keepBackup false\n\n# Created by git when using merge tools for conflicts\n*.BACKUP.*\n*.BASE.*\n*.LOCAL.*\n*.REMOTE.*\n*_BACKUP_*.txt\n*_BASE_*.txt\n*_LOCAL_*.txt\n*_REMOTE_*.txt\n\n### GitBook ###\n# Node rules:\n## Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)\n.grunt\n\n## Dependency directory\n## Commenting this out is preferred by some people, see\n## https://docs.npmjs.com/misc/faq#should-i-check-my-node_modules-folder-into-git\nnode_modules\n\n# Book build output\n_book\n\n# eBook build output\n*.epub\n*.mobi\n*.pdf\n\n### Go ###\n# Binaries for programs and plugins\n*.exe~\n\n# Test binary, built with `go test -c`\n*.test\n\n# Output of the go coverage tool, specifically when used with LiteIDE\n\n# Dependency directories (remove the comment below to include it)\n# vendor/\n\n### Go Patch ###\n/vendor/\n/Godeps/\n\n### GPG ###\nsecring.*\n\n\n### DiskImage ###\n*.toast\n*.vcd\n\n### Grails ###\n# .gitignore for Grails 1.2 and 1.3\n# Although this should work for most versions of grails, it is\n# suggested that you use the \"grails integrate-with --git\" command\n# to generate your .gitignore file.\n\n# web application files\n/web-app/WEB-INF/classes\n\n# default HSQL database files for production mode\n/prodDb.*\n\n# general HSQL database files\n*Db.properties\n*Db.script\n\n# logs\n/stacktrace.log\n/test/reports\n/logs\n\n# project release file\n/*.war\n\n# plugin release files\n/*.zip\n/plugin.xml\n\n# older plugin install locations\n/plugins\n/web-app/plugins\n\n# \"temporary\" build files\n/target\n\n### Groovy ###\n# .gitignore created from Groovy contributors in https://github.com/apache/groovy/blob/master/.gitignore\n\nuser.gradle\n.gradle/\n\n*.DS_Store\n*~\n\n.shelf\n\n\n\n### grunt ###\n# Grunt usually compiles files inside this directory\n\n# Grunt usually preprocesses files such as coffeescript, compass... inside the .tmp directory\n.tmp/\n\n### Haskell ###\ndist\ndist-*\ncabal-dev\n*.hi\n*.hie\n*.chi\n*.chs.h\n*.dyn_o\n*.dyn_hi\n.hpc\n.hsenv\n.cabal-sandbox/\ncabal.sandbox.config\n*.prof\n*.aux\n*.hp\n*.eventlog\n.stack-work/\ncabal.project.local\ncabal.project.local~\n.HTF/\n.ghc.environment.*\n\n### Helm ###\n# Chart dependencies\n**/charts/*.tgz\n\n### Homebrew ###\nBrewfile.lock.json\n\n### Hugo ###\n# Generated files by hugo\n# XXX: might interfere with projects code structure\n#/public/\n/resources/_gen/\nhugo_stats.json\n\n# Executable may be added to repository\nhugo.exe\nhugo.darwin\nhugo.linux\n\n### Images ###\n# JPEG\n*.jpg\n*.jpeg\n*.jpe\n*.jif\n*.jfif\n*.jfi\n\n# JPEG 2000\n*.jp2\n*.j2k\n*.jpf\n*.jpx\n*.jpm\n*.mj2\n\n# JPEG XR\n*.jxr\n*.hdp\n*.wdp\n\n# Graphics Interchange Format\n*.gif\n\n# RAW\n*.raw\n\n# Web P\n*.webp\n\n# Portable Network Graphics\n*.png\n\n# Animated Portable Network Graphics\n*.apng\n\n# Multiple-image Network Graphics\n*.mng\n\n# Tagged Image File Format\n*.tiff\n*.tif\n\n# Scalable Vector Graphics\n*.svg\n*.svgz\n\n# Portable Document Format\n\n# X BitMap\n*.xbm\n\n# BMP\n*.bmp\n*.dib\n\n# ICO\n*.ico\n\n# 3D Images\n*.3dm\n*.max\n\n### Intellij ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### Intellij+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n.idea/\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\nmodules.xml\n.idea/misc.xml\n\n# Sonarlint plugin\n.idea/sonarlint\n\n### Intellij+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### Intellij+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### Java ###\n# Compiled class file\n\n# Log file\n\n# BlueJ files\n*.ctxt\n\n# Mobile Tools for Java (J2ME)\n.mtj.tmp/\n\n# Package Files #\n*.nar\n\n# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml\nhs_err_pid*\n\n### Java-Web ###\n## ignoring target file\n\n### JEnv ###\n# JEnv local Java version configuration file\n.java-version\n\n# Used by previous versions of JEnv\n.jenv-version\n\n### JetBrains ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### JetBrains+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n# Sonarlint plugin\n\n### JetBrains+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### JetBrains+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### JMeter ###\n# JMeter common ignore files\n# http://jmeter.apache.org/\n\n# Ignore Summary/Aggregrate reports\n*.jtl\n\n# Ignore log files\n\n# Ignore customized user.properties\nuser.properties\n\n### Emacs ###\n# -*- mode: gitignore; -*-\n\\#*\\#\n/.emacs.desktop\n/.emacs.desktop.lock\n*.elc\nauto-save-list\ntramp\n.\\#*\n\n# Org-mode\n.org-id-locations\n*_archive\n\n# flymake-mode\n*_flymake.*\n\n# eshell files\n/eshell/history\n/eshell/lastdir\n\n# elpa packages\n/elpa/\n\n# reftex files\n*.rel\n\n# AUCTeX auto folder\n/auto/\n\n# cask packages\n.cask/\n\n# Flycheck\nflycheck_*.el\n\n# server auth directory\n/server/\n\n# projectiles files\n.projectile\n\n# directory configuration\n.dir-locals.el\n\n# network security\n/network-security.data\n\n\n### JupyterNotebooks ###\n# gitignore template for Jupyter Notebooks\n# website: http://jupyter.org/\n\n*/.ipynb_checkpoints/*\n\n# IPython\n\n# Remove previous ipynb_checkpoints\n#   git rm -r .ipynb_checkpoints/\n\n\n### Kotlin ###\n# Compiled class file\n\n# Log file\n\n# BlueJ files\n\n# Mobile Tools for Java (J2ME)\n\n# Package Files #\n\n# virtual machine crash logs, see http://www.java.com/en/download/help/error_hotspot.xml\n\n### LAMP ###\n# LAMP Stack Base\n\n### LAMP.Linux Stack ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n.fuse_hidden*\n\n# KDE directory preferences\n.directory\n\n# Linux trash folder which might appear on any partition or disk\n.Trash-*\n\n# .nfs files are created when an open file is removed but is still being accessed\n.nfs*\n\n### LAMP.PHP Stack ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### LaTeX ###\n## Core latex/pdflatex auxiliary files:\n*.lof\n*.lot\n*.fls\n*.toc\n*.fmt\n*.fot\n*.cb\n*.cb2\n.*.lb\n\n## Intermediate documents:\n*.dvi\n*.xdv\n*-converted-to.*\n# these rules might exclude image files for figures etc.\n# *.ps\n# *.eps\n# *.pdf\n\n## Generated if empty string is given at \"Please type another file name for output:\"\n.pdf\n\n## Bibliography auxiliary files (bibtex/biblatex/biber):\n*.bbl\n*.bcf\n*.blg\n*-blx.aux\n*-blx.bib\n*.run.xml\n\n## Build tool auxiliary files:\n*.fdb_latexmk\n*.synctex\n*.synctex(busy)\n*.synctex.gz\n*.synctex.gz(busy)\n*.pdfsync\n\n## Build tool directories for auxiliary files\n# latexrun\nlatex.out/\n\n## Auxiliary and intermediate files from other packages:\n# algorithms\n*.alg\n*.loa\n\n# achemso\nacs-*.bib\n\n# amsthm\n*.thm\n\n# beamer\n*.nav\n*.pre\n*.snm\n*.vrb\n\n# changes\n*.soc\n\n# comment\n*.cut\n\n# cprotect\n\n# elsarticle (documentclass of Elsevier journals)\n*.spl\n\n# endnotes\n*.ent\n\n# fixme\n*.lox\n\n# feynmf/feynmp\n*.mf\n*.mp\n*.t[1-9]\n*.t[1-9][0-9]\n*.tfm\n\n#(r)(e)ledmac/(r)(e)ledpar\n*.end\n*.?end\n*.[1-9]\n*.[1-9][0-9]\n*.[1-9][0-9][0-9]\n*.[1-9]R\n*.[1-9][0-9]R\n*.[1-9][0-9][0-9]R\n*.eledsec[1-9]\n*.eledsec[1-9]R\n*.eledsec[1-9][0-9]\n*.eledsec[1-9][0-9]R\n*.eledsec[1-9][0-9][0-9]\n*.eledsec[1-9][0-9][0-9]R\n\n# glossaries\n*.acn\n*.acr\n*.glg\n*.glo\n*.gls\n*.glsdefs\n*.lzs\n\n# uncomment this for glossaries-extra (will ignore makeindex's style files!)\n# *.ist\n\n# gnuplottex\n*-gnuplottex-*\n\n# gregoriotex\n*.gaux\n*.gtex\n\n# htlatex\n*.4ct\n*.4tc\n*.idv\n*.lg\n*.trc\n*.xref\n\n# hyperref\n*.brf\n\n# knitr\n*-concordance.tex\n# TODO Comment the next line if you want to keep your tikz graphics files\n*.tikz\n*-tikzDictionary\n\n# listings\n*.lol\n\n# luatexja-ruby\n*.ltjruby\n\n# makeidx\n*.idx\n*.ilg\n*.ind\n\n# minitoc\n*.maf\n*.mlf\n*.mlt\n*.mtc\n*.mtc[0-9]*\n*.slf[0-9]*\n*.slt[0-9]*\n*.stc[0-9]*\n\n# minted\n_minted*\n*.pyg\n\n# morewrites\n*.mw\n\n# nomencl\n*.nlg\n*.nlo\n*.nls\n\n# pax\n*.pax\n\n# pdfpcnotes\n*.pdfpc\n\n# sagetex\n*.sagetex.sage\n*.sagetex.py\n*.sagetex.scmd\n\n# scrwfile\n*.wrt\n\n# sympy\n*.sout\n*.sympy\nsympy-plots-for-*.tex/\n\n# pdfcomment\n*.upa\n*.upb\n\n# pythontex\n*.pytxcode\npythontex-files-*/\n\n# tcolorbox\n*.listing\n\n# thmtools\n*.loe\n\n# TikZ & PGF\n*.dpth\n*.md5\n*.auxlock\n\n# todonotes\n*.tdo\n\n# vhistory\n*.hst\n*.ver\n\n# easy-todo\n*.lod\n\n# xcolor\n*.xcp\n\n# xmpincl\n*.xmpi\n\n# xindy\n*.xdy\n\n# xypic precompiled matrices and outlines\n*.xyc\n*.xyd\n\n# endfloat\n*.ttt\n*.fff\n\n# Latexian\nTSWLatexianTemp*\n\n## Editors:\n# WinEdt\n*.sav\n\n# Texpad\n.texpadtmp\n\n# LyX\n*.lyx~\n\n# Kile\n*.backup\n\n# gummi\n.*.swp\n\n# KBibTeX\n*~[0-9]*\n\n# TeXnicCenter\n*.tps\n\n# auto folder when using emacs and auctex\n./auto/*\n*.el\n\n# expex forward references with \\gathertags\n*-tags.tex\n\n# standalone packages\n*.sta\n\n# Makeindex log files\n*.lpz\n\n# REVTeX puts footnotes in the bibliography by default, unless the nofootinbib\n# option is specified. Footnotes are the stored in a file with suffix Notes.bib.\n# Uncomment the next line to have this generated file ignored.\n#*Notes.bib\n\n### LaTeX Patch ###\n# LIPIcs / OASIcs\n*.vtc\n\n# glossaries\n*.glstex\n\n### Less ###\n*.less\n\n### Linux ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n\n# KDE directory preferences\n\n# Linux trash folder which might appear on any partition or disk\n\n# .nfs files are created when an open file is removed but is still being accessed\n\n### Lua ###\n# Compiled Lua sources\nluac.out\n\n# luarocks build files\n*.src.rock\n\n# Object files\n*.os\n\n# Precompiled Headers\n\n# Libraries\n*.def\n\n# Shared objects (inc. Windows DLLs)\n\n# Executables\n\n\n### macOS ###\n# General\n.DS_Store\n.AppleDouble\n.LSOverride\n\n# Icon must end with two \\r\nIcon\n\n# Thumbnails\n._*\n\n# Files that might appear in the root of a volume\n.DocumentRevisions-V100\n.fseventsd\n.Spotlight-V100\n.TemporaryItems\n.Trashes\n.VolumeIcon.icns\n.com.apple.timemachine.donotpresent\n\n# Directories potentially created on remote AFP share\n.AppleDB\n.AppleDesktop\nNetwork Trash Folder\nTemporary Items\n.apdisk\n\n### MATLAB ###\n# Windows default autosave extension\n*.asv\n\n# OSX / *nix default autosave extension\n*.m~\n\n# Compiled MEX binaries (all platforms)\n*.mex*\n\n# Packaged app and toolbox files\n*.mlappinstall\n*.mltbx\n\n# Generated helpsearch folders\nhelpsearch*/\n\n# Simulink code generation folders\nslprj/\nsccprj/\n\n# Matlab code generation folders\ncodegen/\n\n# Simulink autosave extension\n*.autosave\n\n# Simulink cache files\n*.slxc\n\n# Octave session info\noctave-workspace\n\n### Maven ###\npom.xml.tag\npom.xml.releaseBackup\npom.xml.versionsBackup\npom.xml.next\nrelease.properties\ndependency-reduced-pom.xml\nbuildNumber.properties\n.mvn/timing.properties\n# https://github.com/takari/maven-wrapper#usage-without-binary-jar\n.mvn/wrapper/maven-wrapper.jar\n\n### Mercurial ###\n.hg/\n.hgignore\n.hgsigs\n.hgsub\n.hgsubstate\n.hgtags\n\n### MicrosoftOffice ###\n\n# Word temporary\n~$*.doc*\n\n# Word Auto Backup File\nBackup of *.doc*\n\n# Excel temporary\n~$*.xls*\n\n# Excel Backup File\n*.xlk\n\n# PowerPoint temporary\n~$*.ppt*\n\n# Visio autosave temporary files\n*.~vsd*\n\n### Node ###\n# Logs\nlogs\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\nlerna-debug.log*\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\nreport.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json\n\n# Runtime data\npids\n*.pid\n*.seed\n*.pid.lock\n\n# Directory for instrumented libs generated by jscoverage/JSCover\nlib-cov\n\n# Coverage directory used by tools like istanbul\ncoverage\n*.lcov\n\n# nyc test coverage\n.nyc_output\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n\n# Bower dependency directory (https://bower.io/)\nbower_components\n\n# node-waf configuration\n.lock-wscript\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\nbuild/Release\n\n# Dependency directories\nnode_modules/\njspm_packages/\n\n# TypeScript v1 declaration files\ntypings/\n\n# TypeScript cache\n*.tsbuildinfo\n\n# Optional npm cache directory\n.npm\n\n# Optional eslint cache\n.eslintcache\n\n# Microbundle cache\n.rpt2_cache/\n.rts2_cache_cjs/\n.rts2_cache_es/\n.rts2_cache_umd/\n\n# Optional REPL history\n.node_repl_history\n\n# Output of 'npm pack'\n\n# Yarn Integrity file\n.yarn-integrity\n\n# dotenv environment variables file\n.env.test\n.env*.local\n\n# parcel-bundler cache (https://parceljs.org/)\n.parcel-cache\n\n# Next.js build output\n.next\n\n# Nuxt.js build / generate output\n.nuxt\n\n# Gatsby files\n.cache/\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n.vuepress/dist\n\n# Serverless directories\n.serverless/\n\n# FuseBox cache\n.fusebox/\n\n# DynamoDB Local files\n.dynamodb/\n\n# TernJS port file\n.tern-port\n\n# Stores VSCode versions used for testing VSCode extensions\n.vscode-test\n\n### Octave ###\n# Windows default autosave extension\n\n# OSX / *nix default autosave extension\n\n# Compiled MEX binaries (all platforms)\n\n# Packaged app and toolbox files\n\n# Generated helpsearch folders\n\n# Simulink code generation folders\n\n# Matlab code generation folders\n\n# Simulink autosave extension\n\n# Simulink cache files\n\n# Octave session info\n\n### OSX ###\n# General\n\n# Icon must end with two \\r\n\n# Thumbnails\n\n# Files that might appear in the root of a volume\n\n# Directories potentially created on remote AFP share\n\n### Packer ###\n# Cache objects\npacker_cache/\n\n# Crash log\ncrash.log\n\n# For built boxes\n*.box\n\n### Patch ###\n\n### Perl ###\n!Build/\n.last_cover_stats\n/META.yml\n/META.json\n/MYMETA.*\n*.pm.tdy\n*.bs\n\n# Devel::Cover\ncover_db/\n\n# Devel::NYTProf\nnytprof.out\n\n# Dizt::Zilla\n/.build/\n\n# Module::Build\nBuild\nBuild.bat\n\n# Module::Install\ninc/\n\n# ExtUtils::MakeMaker\n/blib/\n/_eumm/\n/*.gz\n# XXX: always want to commit Makefiles\n#/Makefile\n/Makefile.old\n/MANIFEST.bak\n/pm_to_blib\n\n### Perl6 ###\n# Gitignore for Perl 6 (http://www.perl6.org)\n# As part of https://github.com/github/gitignore\n\n# precompiled files\n.precomp\nlib/.precomp\n\n\n### PHPUnit ###\n# Covers PHPUnit\n# Reference: https://phpunit.de/\n\n# Generated files\n.phpunit.result.cache\n\n# PHPUnit\n/app/phpunit.xml\n/phpunit.xml\n\n# Build data\n/build/\n\n### PowerShell ###\n# Exclude packaged modules\n\n# Exclude .NET assemblies from source\n\n### Puppet ###\n# gitignore template for Puppet modules\n# website: https://forge.puppet.com/\n\n# Built packages\npkg/*\n\n# Should run on multiple platforms so don't check in\nGemfile.lock\n\n# Tests\nspec/fixtures/*\ncoverage/*\n\n# Third-party\nvendor/*\n\n### PuTTY ###\n# Private key\n*.ppk\n\n### PyCharm ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm Patch ###\n# Comment Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-215987721\n\n# *.iml\n# modules.xml\n# .idea/misc.xml\n# *.ipr\n\n# Sonarlint plugin\n# https://plugins.jetbrains.com/plugin/7973-sonarlint\n\n# SonarQube Plugin\n# https://plugins.jetbrains.com/plugin/7238-sonarqube-community-plugin\n\n# Markdown Navigator plugin\n# https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced\n\n# Cache file creation bug\n# See https://youtrack.jetbrains.com/issue/JBR-2257\n\n# CodeStream plugin\n# https://plugins.jetbrains.com/plugin/12206-codestream\n\n### PyCharm+all ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm+all Patch ###\n# Ignores the whole .idea folder and all .iml files\n# See https://github.com/joeblau/gitignore.io/issues/186 and https://github.com/joeblau/gitignore.io/issues/360\n\n\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n# Sonarlint plugin\n\n### PyCharm+iml ###\n# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider\n# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839\n\n# User-specific stuff\n\n# Generated files\n\n# Sensitive or high-churn files\n\n# Gradle\n\n# Gradle and Maven with auto-import\n# When using Gradle or Maven with auto-import, you should exclude module files,\n# since they will be recreated, and may cause churn.  Uncomment if using\n# auto-import.\n# .idea/artifacts\n# .idea/compiler.xml\n# .idea/jarRepositories.xml\n# .idea/modules.xml\n# .idea/*.iml\n# .idea/modules\n# *.iml\n# *.ipr\n\n# CMake\n\n# Mongo Explorer plugin\n\n# File-based project format\n\n# IntelliJ\n\n# mpeltonen/sbt-idea plugin\n\n# JIRA plugin\n\n# Cursive Clojure plugin\n\n# Crashlytics plugin (for Android Studio and IntelliJ)\n\n# Editor-based Rest Client\n\n# Android studio 3.1+ serialized cache file\n\n### PyCharm+iml Patch ###\n# Reason: https://github.com/joeblau/gitignore.io/issues/186#issuecomment-249601023\n\n\n### pydev ###\n.pydevproject\n\n### Python ###\n# Byte-compiled / optimized / DLL files\n\n# C extensions\n\n# Distribution / packaging\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n\n# Installer logs\n\n# Unit test / coverage reports\n\n# Translations\n\n# Django stuff:\n\n# Flask stuff:\n\n# Scrapy stuff:\n\n# Sphinx documentation\n\n# PyBuilder\n\n# Jupyter Notebook\n\n# IPython\n\n# pyenv\n\n# pipenv\n#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.\n#   However, in case of collaboration, if having platform-specific dependencies or dependencies\n#   having no cross-platform support, pipenv may install dependencies that don't work, or not\n#   install all needed dependencies.\n\n# PEP 582; used by e.g. github.com/David-OConnor/pyflow\n\n# Celery stuff\n\n# SageMath parsed files\n\n# Environments\n\n# Spyder project settings\n\n# Rope project settings\n\n# mkdocs documentation\n\n# mypy\n\n# Pyre type checker\n\n# pytype static type analyzer\n\n# profiling data\n\n### R ###\n# History files\n.Rhistory\n.Rapp.history\n\n# Session Data files\n.RData\n\n# User-specific files\n.Ruserdata\n\n# Example code in package build process\n*-Ex.R\n\n# Output files from R CMD build\n/*.tar.gz\n\n# Output files from R CMD check\n/*.Rcheck/\n\n# RStudio files\n.Rproj.user/\n\n# produced vignettes\nvignettes/*.html\nvignettes/*.pdf\n\n# OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3\n.httr-oauth\n\n# knitr and R markdown default cache directories\n*_cache/\n/cache/\n\n# Temporary files created by R markdown\n*.utf8.md\n*.knit.md\n\n# R Environment Variables\n.Renviron\n\n### R.Bookdown Stack ###\n# R package: bookdown caching files\n/*_files/\n\n### Rails ###\n*.rbc\ncapybara-*.html\n.rspec\n/db/*.sqlite3\n/db/*.sqlite3-journal\n/db/*.sqlite3-[0-9]*\n/public/system\n/coverage/\n/spec/tmp\nrerun.txt\npickle-email-*.html\n\n# Ignore all logfiles and tempfiles.\n/log/*\n/tmp/*\n!/log/.keep\n!/tmp/.keep\n\n# TODO Comment out this rule if you are OK with secrets being uploaded to the repo\nconfig/initializers/secret_token.rb\nconfig/master.key\n\n# Only include if you have production secrets in this file, which is no longer a Rails default\n# config/secrets.yml\n\n# dotenv, dotenv-rails\n# TODO Comment out these rules if environment variables can be committed\n.env.*\n\n## Environment normalization:\n/.bundle\n/vendor/bundle\n\n# these should all be checked in to normalize the environment:\n# Gemfile.lock, .ruby-version, .ruby-gemset\n\n# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:\n.rvmrc\n\n# if using bower-rails ignore default bower_components path bower.json files\n/vendor/assets/bower_components\n*.bowerrc\nbower.json\n\n# Ignore pow environment settings\n.powenv\n\n# Ignore Byebug command history file.\n.byebug_history\n\n# Ignore node_modules\n\n# Ignore precompiled javascript packs\n/public/packs\n/public/packs-test\n/public/assets\n\n# Ignore yarn files\n/yarn-error.log\n\n# Ignore uploaded files in development\n/storage/*\n!/storage/.keep\n\n### react ###\n.DS_*\n**/*.backup.*\n**/*.back.*\n\n\n*.sublime*\n\npsd\nthumb\nsketch\n\n### ReactNative ###\n# React Native Stack Base\n\n.expo\n__generated__\n\n### ReactNative.Android Stack ###\n# Built application files\n*.aar\n*.ap_\n*.aab\n\n# Files for the ART/Dalvik VM\n*.dex\n\n# Java class files\n\n# Generated files\ngen/\n#  Uncomment the following line in case you need and you don't have the release build type files in your app\n# release/\n\n# Gradle files\n\n# Local configuration file (sdk path, etc)\n\n# Proguard folder generated by Eclipse\nproguard/\n\n# Log Files\n\n# Android Studio Navigation editor temp files\n.navigation/\n\n# Android Studio captures folder\ncaptures/\n\n# IntelliJ\n.idea/workspace.xml\n.idea/tasks.xml\n.idea/gradle.xml\n.idea/assetWizardSettings.xml\n.idea/dictionaries\n.idea/libraries\n# Android Studio 3 in .gitignore file.\n.idea/caches\n.idea/modules.xml\n# Comment next line if keeping position of elements in Navigation Editor is relevant for you\n.idea/navEditor.xml\n\n# Keystore files\n# Uncomment the following lines if you do not want to check your keystore files in.\n#*.jks\n#*.keystore\n\n# External native build folder generated in Android Studio 2.2 and later\n.externalNativeBuild\n.cxx/\n\n# Google Services (e.g. APIs or Firebase)\n# google-services.json\n\n# Freeline\nfreeline.py\nfreeline/\nfreeline_project_description.json\n\n# fastlane\nfastlane/report.xml\nfastlane/Preview.html\nfastlane/screenshots\nfastlane/test_output\n#fastlane/readme.md\n\n# Version control\nvcs.xml\n\n# lint\nlint/intermediates/\nlint/generated/\nlint/outputs/\nlint/tmp/\n# lint/reports/\n\n### ReactNative.Buck Stack ###\nbuck-out/\n.buckconfig.local\n.buckd/\n.buckversion\n.fakebuckversion\n\n### ReactNative.Gradle Stack ###\n.gradle\n\n# Ignore Gradle GUI config\ngradle-app.setting\n\n# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)\n!gradle-wrapper.jar\n\n# Cache of project\n.gradletasknamecache\n\n# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898\n# gradle/wrapper/gradle-wrapper.properties\n\n### ReactNative.Linux Stack ###\n\n# temporary files which can be created if a process still has a handle open of a deleted file\n\n# KDE directory preferences\n\n# Linux trash folder which might appear on any partition or disk\n\n# .nfs files are created when an open file is removed but is still being accessed\n\n### ReactNative.Node Stack ###\n# Logs\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\n\n# Runtime data\n\n# Directory for instrumented libs generated by jscoverage/JSCover\n\n# Coverage directory used by tools like istanbul\n\n# nyc test coverage\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n\n# Bower dependency directory (https://bower.io/)\n\n# node-waf configuration\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\n\n# Dependency directories\n\n# TypeScript v1 declaration files\n\n# TypeScript cache\n\n# Optional npm cache directory\n\n# Optional eslint cache\n\n# Microbundle cache\n\n# Optional REPL history\n\n# Output of 'npm pack'\n\n# Yarn Integrity file\n\n# dotenv environment variables file\n\n# parcel-bundler cache (https://parceljs.org/)\n\n# Next.js build output\n\n# Nuxt.js build / generate output\n\n# Gatsby files\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n\n# Serverless directories\n\n# FuseBox cache\n\n# DynamoDB Local files\n\n# TernJS port file\n\n# Stores VSCode versions used for testing VSCode extensions\n\n### ReactNative.Xcode Stack ###\n# Xcode\n#\n# gitignore contributors: remember to update Global/Xcode.gitignore, Objective-C.gitignore & Swift.gitignore\n\n## User settings\nxcuserdata/\n\n## compatibility with Xcode 8 and earlier (ignoring not required starting Xcode 9)\n*.xcscmblueprint\n*.xccheckout\n\n## compatibility with Xcode 3 and earlier (ignoring not required starting Xcode 4)\nDerivedData/\n*.moved-aside\n*.pbxuser\n!default.pbxuser\n*.mode1v3\n!default.mode1v3\n*.mode2v3\n!default.mode2v3\n*.perspectivev3\n!default.perspectivev3\n\n## Gcc Patch\n/*.gcno\n\n### ReactNative.macOS Stack ###\n# General\n\n# Icon must end with two \\r\nIcon\n\n# Thumbnails\n\n# Files that might appear in the root of a volume\n\n# Directories potentially created on remote AFP share\n\n### Redis ###\n# Ignore redis binary dump (dump.rdb) files\n\n*.rdb\n\n### ROOT ###\n# ROOT Home Page : https://root.cern.ch/\n# ROOT Used by Experimental Physicists, not necessarily HEP\n# ROOT based on C++\n\n# Files generated by ROOT, observed with v6.xy\n\n*.pcm\n\n\n\n### Ruby ###\n/.config\n/InstalledFiles\n/pkg/\n/spec/reports/\n/spec/examples.txt\n/test/tmp/\n/test/version_tmp/\n/tmp/\n\n# Used by dotenv library to load environment variables.\n# .env\n\n# Ignore Byebug command history file.\n\n## Specific to RubyMotion:\n.dat*\n.repl_history\n*.bridgesupport\nbuild-iPhoneOS/\nbuild-iPhoneSimulator/\n\n## Specific to RubyMotion (use of CocoaPods):\n# We recommend against adding the Pods directory to your .gitignore. However\n# you should judge for yourself, the pros and cons are mentioned at:\n# https://guides.cocoapods.org/using/using-cocoapods.html#should-i-check-the-pods-directory-into-source-control\n# vendor/Pods/\n\n## Documentation cache and generated files:\n/.yardoc/\n/_yardoc/\n/doc/\n/rdoc/\n\n/.bundle/\n/lib/bundler/man/\n\n# for a library or gem, you might want to ignore these files since the code is\n# intended to run in multiple environments; otherwise, check them in:\n# Gemfile.lock\n# .ruby-version\n# .ruby-gemset\n\n# unless supporting rvm < 1.11.0 or doing something fancy, ignore this:\n\n# Used by RuboCop. Remote config files pulled in from inherit_from directive.\n# .rubocop-https?--*\n\n### Ruby Patch ###\n# Used by RuboCop. Remote config files pulled in from inherit_from directive.\n# .rubocop-https?--*\n\n### Rust ###\n# Generated by Cargo\n# will have compiled files and executables\n\n# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries\n# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html\nCargo.lock\n\n### SBT ###\n# Simple Build Tool\n# http://www.scala-sbt.org/release/docs/Getting-Started/Directories.html#configuring-version-control\n\ndist/*\nlib_managed/\nsrc_managed/\nproject/boot/\nproject/plugins/project/\n.history\n.lib/\n\n### Scala ###\n\n### Serverless ###\n# Ignore build directory\n.serverless\n\n### Sonar ###\n#Sonar generated dir\n/.sonar/\n\n### SonarQube ###\n# SonarQube ignore files.\n# https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner\n# Sonar Scanner working directories\n.sonar/\n.scannerwork/\n\n# http://www.sonarlint.org/commandline/\n# SonarLint working directories, configuration files (including credentials)\n#.sonarlint/\n\n### Spark ###\n*#*#\n*.#*\n*.pyc\n*.pyo\n.ensime\n.ensime_cache/\n.ensime_lucene\n.generated-mima*\nR-unit-tests.log\nR/unit-tests.out\nR/cran-check.out\nR/pkg/vignettes/sparkr-vignettes.html\nR/pkg/tests/fulltests/Rplots.pdf\nbuild/*.jar\nbuild/apache-maven*\nbuild/scala*\nbuild/zinc*\ncache\ncheckpoint\nconf/*.cmd\nconf/*.conf\nconf/*.properties\nconf/*.sh\nconf/*.xml\nconf/java-opts\nconf/slaves\nderby.log\ndev/create-release/*final\ndev/create-release/*txt\ndev/pr-deps/\ndocs/_site\ndocs/api\nsql/docs\nsql/site\nlint-r-report.log\nlog/\nlogs/\nproject/build/target/\nproject/plugins/lib_managed/\nproject/plugins/project/build.properties\nproject/plugins/src_managed/\nproject/plugins/target/\npython/lib/pyspark.zip\npython/deps\npython/test_coverage/coverage_data\npython/test_coverage/htmlcov\npython/pyspark/python\nreports/\nscalastyle-on-compile.generated.xml\nscalastyle-output.xml\nscalastyle.txt\nspark-*-bin-*.tgz\nspark-tests.log\nstreaming-tests.log\nunit-tests.log\nwork/\ndocs/.jekyll-metadata\n\n# For Hive\nTempStatsStore/\nmetastore/\nmetastore_db/\nsql/hive-thriftserver/test_warehouses\nwarehouse/\nspark-warehouse/\n\n# For R session data\n.RHistory\n*.Rproj\n*.Rproj.*\n\n.Rproj.user\n\n# For SBT\n.jvmopts\n\n\n### Splunk ###\n# gitignore template for Splunk apps\n# documentation: http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Defaultmetaconf\n\n# Splunk local meta file\nlocal.meta\n\n# Splunk local folder\nlocal\n\n### Spreadsheet ###\n*.xlr\n*.xls\n*.xlsx\n\n### SSH ###\n**/.ssh/id_*\n**/.ssh/*_id_*\n**/.ssh/known_hosts\n\n### SublimeText ###\n# Cache files for Sublime Text\n*.tmlanguage.cache\n*.tmPreferences.cache\n*.stTheme.cache\n\n# Workspace files are user-specific\n*.sublime-workspace\n\n# Project files should be checked into the repository, unless a significant\n# proportion of contributors will probably not be using Sublime Text\n# *.sublime-project\n\n# SFTP configuration file\nsftp-config.json\n\n# Package control specific files\nPackage Control.last-run\nPackage Control.ca-list\nPackage Control.ca-bundle\nPackage Control.system-ca-bundle\nPackage Control.cache/\nPackage Control.ca-certs/\nPackage Control.merged-ca-bundle\nPackage Control.user-ca-bundle\noscrypto-ca-bundle.crt\nbh_unicode_properties.cache\n\n# Sublime-github package stores a github token in this file\n# https://packagecontrol.io/packages/sublime-github\nGitHub.sublime-settings\n\n### SVN ###\n.svn/\n\n### Terraform ###\n# Local .terraform directories\n**/.terraform/*\n\n# .tfstate files\n*.tfstate\n*.tfstate.*\n\n# Crash log files\n\n# Ignore any .tfvars files that are generated automatically for each Terraform run. Most\n# .tfvars files are managed as part of configuration and so should be included in\n# version control.\n# example.tfvars\n\n# Ignore override files as they are usually used to override resources locally and so\n# are not checked in\noverride.tf\noverride.tf.json\n*_override.tf\n*_override.tf.json\n\n# Include override files you do wish to add to version control using negated pattern\n# !example_override.tf\n\n# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan\n# example: *tfplan*\n\n### Terragrunt ###\n# terragrunt cache directories\n**/.terragrunt-cache/*\n\n### TortoiseGit ###\n# Project-level settings\n/.tgitconfig\n\n### Vagrant ###\n# General\n.vagrant/\n\n# Log files (if you are creating logs in debug mode, uncomment this)\n# *.log\n\n### Vagrant Patch ###\n\n### venv ###\n# Virtualenv\n# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/\n# XXX: prevents committing scripts at ~/bin\n#[Bb]in\n[Ii]nclude\n# XXX: prevents committing submodule lib\n#[Ll]ib\n[Ll]ib64\n[Ll]ocal\n#[Ss]cripts\npyvenv.cfg\npip-selfcheck.json\n\n### VirtualEnv ###\n# Virtualenv\n# http://iamzed.com/2009/05/07/a-primer-on-virtualenv/\n\n### Julia ###\n# Files generated by invoking Julia with --code-coverage\n*.jl.cov\n*.jl.*.cov\n\n# Files generated by invoking Julia with --track-allocation\n*.jl.mem\n\n# System-specific files and directories generated by the BinaryProvider and BinDeps packages\n# They contain absolute paths specific to the host computer, and so should not be committed\ndeps/deps.jl\ndeps/build.log\ndeps/downloads/\ndeps/usr/\ndeps/src/\n\n# Build artifacts for creating documentation generated by the Documenter package\ndocs/build/\ndocs/site/\n\n# File generated by Pkg, the package manager, based on a corresponding Project.toml\n# It records a fixed state of all packages used by the project. As such, it should not be\n# committed for packages, but should be committed for applications that require a static\n# environment.\nManifest.toml\n\n### VisualStudioCode ###\n\n### VisualStudioCode Patch ###\n# Ignore all local history of files\n.ionide\n\n### vs ###\n## Ignore Visual Studio temporary files, build results, and\n## files generated by popular Visual Studio add-ons.\n##\n## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore\n\n# User-specific files\n*.rsuser\n*.suo\n*.user\n*.userosscache\n*.sln.docstates\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n*.userprefs\n\n# Mono auto generated files\nmono_crash.*\n\n# Build results\n[Dd]ebug/\n[Dd]ebugPublic/\n[Rr]elease/\n[Rr]eleases/\nx64/\nx86/\n[Aa][Rr][Mm]/\n[Aa][Rr][Mm]64/\nbld/\n# XXX: prevents committing scripts at ~/bin\n#[Bb]in/\n[Oo]bj/\n[Ll]og/\n[Ll]ogs/\n\n# Visual Studio 2015/2017 cache/options directory\n.vs/\n# Uncomment if you have tasks that create the project's static files in wwwroot\n#wwwroot/\n\n# Visual Studio 2017 auto generated files\nGenerated\\ Files/\n\n# MSTest test Results\n[Tt]est[Rr]esult*/\n[Bb]uild[Ll]og.*\n\n# NUnit\n*.VisualState.xml\nTestResult.xml\nnunit-*.xml\n\n# Build Results of an ATL Project\n[Dd]ebugPS/\n[Rr]eleasePS/\ndlldata.c\n\n# Benchmark Results\nBenchmarkDotNet.Artifacts/\n\n# .NET Core\nproject.lock.json\nproject.fragment.lock.json\nartifacts/\n\n# StyleCop\nStyleCopReport.xml\n\n# Files built by Visual Studio\n*_i.c\n*_p.c\n*_h.h\n*.meta\n*.iobj\n*.ipdb\n*.pgc\n*.pgd\n*.rsp\n*.sbr\n*.tlb\n*.tli\n*.tlh\n*.tmp_proj\n*_wpftmp.csproj\n*.vspscc\n*.vssscc\n.builds\n*.pidb\n*.svclog\n*.scc\n\n# Chutzpah Test files\n_Chutzpah*\n\n# Visual C++ cache files\nipch/\n*.aps\n*.ncb\n*.opendb\n*.opensdf\n*.cachefile\n*.VC.db\n*.VC.VC.opendb\n\n# Visual Studio profiler\n*.psess\n*.vsp\n*.vspx\n*.sap\n\n# Visual Studio Trace Files\n*.e2e\n\n# TFS 2012 Local Workspace\n$tf/\n\n# Guidance Automation Toolkit\n*.gpState\n\n# ReSharper is a .NET coding add-in\n_ReSharper*/\n*.[Rr]e[Ss]harper\n*.DotSettings.user\n\n# TeamCity is a build add-in\n_TeamCity*\n\n# DotCover is a Code Coverage Tool\n*.dotCover\n\n# AxoCover is a Code Coverage Tool\n.axoCover/*\n!.axoCover/settings.json\n\n# Coverlet is a free, cross platform Code Coverage Tool\ncoverage*[.json, .xml, .info]\n\n# Visual Studio code coverage results\n*.coverage\n*.coveragexml\n\n# NCrunch\n_NCrunch_*\n.*crunch*.local.xml\nnCrunchTemp_*\n\n# MightyMoose\n*.mm.*\nAutoTest.Net/\n\n# Web workbench (sass)\n.sass-cache/\n\n# Installshield output folder\n[Ee]xpress/\n\n# DocProject is a documentation generator add-in\nDocProject/buildhelp/\nDocProject/Help/*.HxT\nDocProject/Help/*.HxC\nDocProject/Help/*.hhc\nDocProject/Help/*.hhk\nDocProject/Help/*.hhp\nDocProject/Help/Html2\nDocProject/Help/html\n\n# Click-Once directory\npublish/\n\n# Publish Web Output\n*.[Pp]ublish.xml\n*.azurePubxml\n# Note: Comment the next line if you want to checkin your web deploy settings,\n# but database connection strings (with potential passwords) will be unencrypted\n*.pubxml\n*.publishproj\n\n# Microsoft Azure Web App publish settings. Comment the next line if you want to\n# checkin your Azure Web App publish settings, but sensitive information contained\n# in these scripts will be unencrypted\nPublishScripts/\n\n# NuGet Packages\n*.nupkg\n# NuGet Symbol Packages\n*.snupkg\n# The packages folder can be ignored because of Package Restore\n**/[Pp]ackages/*\n# except build/, which is used as an MSBuild target.\n!**/[Pp]ackages/build/\n# Uncomment if necessary however generally it will be regenerated when needed\n#!**/[Pp]ackages/repositories.config\n# NuGet v3's project.json files produces more ignorable files\n*.nuget.props\n*.nuget.targets\n\n# Microsoft Azure Build Output\ncsx/\n*.build.csdef\n\n# Microsoft Azure Emulator\necf/\nrcf/\n\n# Windows Store app package directories and files\nAppPackages/\nBundleArtifacts/\nPackage.StoreAssociation.xml\n_pkginfo.txt\n*.appx\n*.appxbundle\n*.appxupload\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n*.[Cc]ache\n# but keep track of directories ending in .cache\n!?*.[Cc]ache/\n\n# Others\nClientBin/\n~$*\n*.dbmdl\n*.dbproj.schemaview\n*.jfm\n*.pfx\n*.publishsettings\norleans.codegen.cs\n\n# Including strong name files can present a security risk\n# (https://github.com/github/gitignore/pull/2483#issue-259490424)\n#*.snk\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components\n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n#bower_components/\n\n# RIA/Silverlight projects\nGenerated_Code/\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n_UpgradeReport_Files/\nBackup*/\nUpgradeLog*.XML\nUpgradeLog*.htm\nServiceFabricBackup/\n*.rptproj.bak\n\n# SQL Server files\n*.mdf\n*.ldf\n*.ndf\n\n# Business Intelligence projects\n*.rdl.data\n*.bim.layout\n*.bim_*.settings\n*.rptproj.rsuser\n*- [Bb]ackup.rdl\n*- [Bb]ackup ([0-9]).rdl\n*- [Bb]ackup ([0-9][0-9]).rdl\n\n# Microsoft Fakes\nFakesAssemblies/\n\n# GhostDoc plugin setting file\n*.GhostDoc.xml\n\n# Node.js Tools for Visual Studio\n.ntvs_analysis.dat\n\n# Visual Studio 6 build log\n*.plg\n\n# Visual Studio 6 workspace options file\n*.opt\n\n# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)\n*.vbw\n\n# Visual Studio LightSwitch build output\n**/*.HTMLClient/GeneratedArtifacts\n**/*.DesktopClient/GeneratedArtifacts\n**/*.DesktopClient/ModelManifest.xml\n**/*.Server/GeneratedArtifacts\n**/*.Server/ModelManifest.xml\n_Pvt_Extensions\n\n# Paket dependency manager\n.paket/paket.exe\npaket-files/\n\n# FAKE - F# Make\n.fake/\n\n# CodeRush personal settings\n.cr/personal\n\n# Python Tools for Visual Studio (PTVS)\n\n# Cake - Uncomment if you are using it\n# tools/**\n# !tools/packages.config\n\n# Tabs Studio\n*.tss\n\n# Telerik's JustMock configuration file\n*.jmconfig\n\n# BizTalk build output\n*.btp.cs\n*.btm.cs\n*.odx.cs\n*.xsd.cs\n\n# OpenCover UI analysis results\nOpenCover/\n\n# Azure Stream Analytics local run output\nASALocalRun/\n\n# MSBuild Binary and Structured Log\n*.binlog\n\n# NVidia Nsight GPU debugger configuration file\n*.nvuser\n\n# MFractors (Xamarin productivity tool) working folder\n.mfractor/\n\n# Local History for Visual Studio\n.localhistory/\n\n# BeatPulse healthcheck temp database\nhealthchecksdb\n\n# Backup folder for Package Reference Convert tool in Visual Studio 2017\nMigrationBackup/\n\n# Ionide (cross platform F# VS Code tools) working folder\n.ionide/\n\n### vscode ###\n\n### Vue ###\n# gitignore template for Vue.js projects\n# Recommended template: Node.gitignore\n\n# TODO: where does this rule come from?\ndocs/_book\n\n# TODO: where does this rule come from?\n# XXX: covers up standard Python unit test path\n# test/\n\n### Vuejs ###\n# Recommended template: Node.gitignore\n\nnpm-debug.log\nyarn-error.log\n\n### Waf ###\n# For projects that use the Waf build system: https://waf.io/\n# Dot-hidden on Unix-like systems\n.waf-*-*/\n.waf3-*-*/\n# Hidden directory on Windows (no dot)\nwaf-*-*/\nwaf3-*-*/\n# Lockfile\n.lock-waf_*_build\n\n### Windows ###\n# Windows thumbnail cache files\nThumbs.db\nThumbs.db:encryptable\nehthumbs.db\nehthumbs_vista.db\n\n# Dump file\n*.stackdump\n\n# Folder config file\n[Dd]esktop.ini\n\n# Recycle Bin used on file shares\n$RECYCLE.BIN/\n\n# Windows Installer files\n*.msix\n\n# Windows shortcuts\n*.lnk\n\n### Xcode ###\n# Xcode\n# gitignore contributors: remember to update Global/Xcode.gitignore, Objective-C.gitignore & Swift.gitignore\n\n\n\n\n\n### Xcode Patch ###\n*.xcodeproj/*\n!*.xcodeproj/project.pbxproj\n!*.xcodeproj/xcshareddata/\n!*.xcworkspace/contents.xcworkspacedata\n**/xcshareddata/WorkspaceSettings.xcsettings\n\n### XcodeInjection ###\n# Code Injection\n# After new code Injection tools there's a generated folder /iOSInjectionProject\n# https://github.com/johnno1962/injectionforxcode\n\niOSInjectionProject/\n\n### Gradle ###\n\n# Ignore Gradle GUI config\n\n# Avoid ignoring Gradle wrapper jar file (.jar files are usually ignored)\n\n# Cache of project\n\n# # Work around https://youtrack.jetbrains.com/issue/IDEA-116898\n# gradle/wrapper/gradle-wrapper.properties\n\n### Gradle Patch ###\n**/build/\n\n### VisualStudio ###\n\n# User-specific files\n\n# User-specific files (MonoDevelop/Xamarin Studio)\n\n# Mono auto generated files\n\n# Build results\n\n# Visual Studio 2015/2017 cache/options directory\n# Uncomment if you have tasks that create the project's static files in wwwroot\n\n# Visual Studio 2017 auto generated files\n\n# MSTest test Results\n\n# NUnit\n\n# Build Results of an ATL Project\n\n# Benchmark Results\n\n# .NET Core\n\n# StyleCop\n\n# Files built by Visual Studio\n\n# Chutzpah Test files\n\n# Visual C++ cache files\n\n# Visual Studio profiler\n\n# Visual Studio Trace Files\n\n# TFS 2012 Local Workspace\n\n# Guidance Automation Toolkit\n\n# ReSharper is a .NET coding add-in\n\n# TeamCity is a build add-in\n\n# DotCover is a Code Coverage Tool\n\n# AxoCover is a Code Coverage Tool\n\n# Coverlet is a free, cross platform Code Coverage Tool\n\n# Visual Studio code coverage results\n\n# NCrunch\n\n# MightyMoose\n\n# Web workbench (sass)\n\n# Installshield output folder\n\n# DocProject is a documentation generator add-in\n\n# Click-Once directory\n\n# Publish Web Output\n# Note: Comment the next line if you want to checkin your web deploy settings,\n# but database connection strings (with potential passwords) will be unencrypted\n\n# Microsoft Azure Web App publish settings. Comment the next line if you want to\n# checkin your Azure Web App publish settings, but sensitive information contained\n# in these scripts will be unencrypted\n\n# NuGet Packages\n# NuGet Symbol Packages\n# The packages folder can be ignored because of Package Restore\n# except build/, which is used as an MSBuild target.\n# Uncomment if necessary however generally it will be regenerated when needed\n# NuGet v3's project.json files produces more ignorable files\n\n# Microsoft Azure Build Output\n\n# Microsoft Azure Emulator\n\n# Windows Store app package directories and files\n\n# Visual Studio cache files\n# files ending in .cache can be ignored\n# but keep track of directories ending in .cache\n\n# Others\n\n# Including strong name files can present a security risk\n# (https://github.com/github/gitignore/pull/2483#issue-259490424)\n\n# Since there are multiple workflows, uncomment next line to ignore bower_components\n# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)\n\n# RIA/Silverlight projects\n\n# Backup & report files from converting an old project file\n# to a newer Visual Studio version. Backup files are not needed,\n# because we have git ;-)\n\n# SQL Server files\n\n# Business Intelligence projects\n\n# Microsoft Fakes\n\n# GhostDoc plugin setting file\n\n# Node.js Tools for Visual Studio\n\n# Visual Studio 6 build log\n\n# Visual Studio 6 workspace options file\n\n# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)\n\n# Visual Studio LightSwitch build output\n\n# Paket dependency manager\n\n# FAKE - F# Make\n\n# CodeRush personal settings\n\n# Python Tools for Visual Studio (PTVS)\n\n# Cake - Uncomment if you are using it\n# tools/**\n# !tools/packages.config\n\n# Tabs Studio\n\n# Telerik's JustMock configuration file\n\n# BizTalk build output\n\n# OpenCover UI analysis results\n\n# Azure Stream Analytics local run output\n\n# MSBuild Binary and Structured Log\n\n# NVidia Nsight GPU debugger configuration file\n\n# MFractors (Xamarin productivity tool) working folder\n\n# Local History for Visual Studio\n\n# BeatPulse healthcheck temp database\n\n# Backup folder for Package Reference Convert tool in Visual Studio 2017\n\n# Ionide (cross platform F# VS Code tools) working folder\n\n# End of https://www.toptal.com/developers/gitignore/api/ansible,apachehadoop,appcode,appengine,archive,archives,archlinuxpackages,audio,autotools,backup,basic,batch,bittorrent,c,c++,certificates,chefcookbook,clojure,cloud9,cmake,code,code-java,codeblocks,compressed,compressedarchive,compression,data,database,datarecovery,diff,direnv,diskimage,docfx,docpress,docz,dotenv,dotfilessh,dotsettings,dropbox,eclipse,emacs,erlang,executable,firebase,flask,git,gitbook,go,gpg,gradle,grails,groovy,grunt,haskell,helm,homebrew,hugo,images,intellij,intellij+all,intellij+iml,java,java-web,jenv,jetbrains,jetbrains+all,jetbrains+iml,jmeter,julia,jupyternotebooks,kotlin,lamp,latex,less,linux,lua,macos,matlab,maven,mercurial,microsoftoffice,node,octave,osx,packer,patch,perl,perl6,phpunit,powershell,puppet,putty,pycharm,pycharm+all,pycharm+iml,pydev,python,r,rails,react,reactnative,redis,root,ruby,rust,sbt,scala,serverless,sonar,sonarqube,spark,splunk,spreadsheet,ssh,sublimetext,svn,terraform,terragrunt,tortoisegit,vagrant,venv,virtualenv,visualstudio,visualstudiocode,vs,vscode,vue,vuejs,waf,windows,xcode,xcodeinjection,zsh\n\n# for DevOps-Bash-tools packages/*.sh scripts\n## needed to unmask directory ignore above otherwise !packages/*packages*.sh isn't respected\n!packages/\n!packages/*.sh\n"
  },
  {
    "path": ".gitlab-ci.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: Sun Feb 23 19:02:10 2020 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#  to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               G i t L a b   C I\n# ============================================================================ #\n\n# https://docs.gitlab.com/ee/ci/yaml/README.html\n\n#include: '.gitlab/*.y*ml'\n\nimage: ubuntu:18.04\n\njob:\n  before_script:\n    - setup/ci_bootstrap.sh\n  script:\n    - make init && make ci test\n"
  },
  {
    "path": ".gitmodules",
    "content": ""
  },
  {
    "path": ".hound.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-09-23 10:28:21 +0100 (Wed, 23 Sep 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\n# https://intercom.help/hound/en/articles/2138537-flake8\nflake8:\n  enabled: true\n  #config_file: .flake8\n\n# http://help.houndci.com/en/articles/2138564-shellcheck\nshellcheck:\n  enabled: true\n  #config_file: .shellcheck.yml\n\n# http://help.houndci.com/en/articles/2138524-golint\ngolint:\n  enabled: false\n\n#fail_on_violations: true\n"
  },
  {
    "path": ".mdl.rb",
    "content": "#!/usr/bin/env ruby\n#  vim:ts=4:sts=4:sw=4:et:filetype=ruby\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-22 01:58:12 +0200 (Thu, 22 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nall\n#exclude_rule 'MD001'\n#exclude_rule 'MD003'\n#exclude_rule 'MD005'\nexclude_rule 'MD007'  # leave 2 space indentation for lists, 3 space is ugly af\n#exclude_rule 'MD012'\nexclude_rule 'MD013'  # long lines cannot be split if they are URLs\n#exclude_rule 'MD022'\nexclude_rule 'MD024'  # Multiple headers with the same content\n#exclude_rule 'MD025'\nexclude_rule 'MD026'  # Trailing punctuation in header - sometimes I want to do etc. or ... at the end of a heading\n#exclude_rule 'MD031'\n#exclude_rule 'MD032'\nexclude_rule 'MD033'  # inline HTML is important for formatting\nexclude_rule 'MD036'  # emphasis used instead of header for footer Ported from lines\n#exclude_rule 'MD039'\n#exclude_rule 'MD056'\n"
  },
  {
    "path": ".mdlrc",
    "content": "mdlrc_dir = File.expand_path('..', __FILE__)\n\nstyle_file = File.join(mdlrc_dir, '.mdl.rb')\n\nstyle style_file\n"
  },
  {
    "path": ".pre-commit-config.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2024-08-08 17:34:56 +0300 (Thu, 08 Aug 2024)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              P r e - C o m m i t\n# ============================================================================ #\n\n---\nfail_fast: false\n#exclude: *.tmp$\n\nrepos:\n\n    # will accept anything that 'git clone' understands\n    # this means you can set this to a local git repo to develop your own hook repos interactively\n  - repo: https://github.com/pre-commit/pre-commit-hooks\n    rev: v4.6.0\n    hooks:\n      - id: check-yaml\n      # Common errors\n      #- id: end-of-file-fixer  # ruins .gitignore Icon\\r\n      - id: trailing-whitespace\n        args: [--markdown-linebreak-ext=md]\n      # Git style\n      - id: check-added-large-files\n      - id: check-merge-conflict\n      - id: check-vcs-permalinks\n      #- id: forbid-new-submodules\n      # Cross platform\n      - id: check-case-conflict\n      - id: mixed-line-ending\n        args: [--fix=lf]\n      # Security\n      - id: detect-aws-credentials\n        args: ['--allow-missing-credentials']\n\n  # rewrites python files with useless changes like changing single quotes to double quotes\n  #- repo: https://github.com/psf/black\n  #  rev: 24.8.0\n  #  hooks:\n  #    - id: black\n\n  # Git secrets Leaks\n  - repo: https://github.com/awslabs/git-secrets.git\n    # the release tags for 1.2.0, 1.2.1 and 1.3.0 are broken with this error:\n    #\n    #   /Users/hari/.cache/pre-commit/repo......./.pre-commit-hooks.yaml is not a file\n    #\n    rev: 5357e18\n    hooks:\n      - id: git-secrets\n\n  - repo: https://github.com/markdownlint/markdownlint\n    rev: v0.12.0\n    hooks:\n      - id: markdownlint\n        name: Markdownlint\n        description: Run markdownlint on your Markdown files\n        entry: mdl\n        args: [-s, .mdl.rb]\n        language: ruby\n        files: \\.(md|mdown|markdown)$\n"
  },
  {
    "path": ".pylintrc",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           P y L i n t   C o n f i g\n# ============================================================================ #\n\n# pylint --generate-rcfile >> .pylintrc\n\n[MAIN]\n\n# Analyse import fallback blocks. This can be used to support both Python 2 and\n# 3 compatible code, which means that the block might have code that exists\n# only in one or another interpreter, leading to false positives when analysed.\nanalyse-fallback-blocks=no\n\n# Clear in-memory caches upon conclusion of linting. Useful if running pylint\n# in a server-like mode.\nclear-cache-post-run=no\n\n# Load and enable all available extensions. Use --list-extensions to see a list\n# all available extensions.\n#enable-all-extensions=\n\n# In error mode, messages with a category besides ERROR or FATAL are\n# suppressed, and no reports are done by default. Error mode is compatible with\n# disabling specific errors.\n#errors-only=\n\n# Always return a 0 (non-error) status code, even if lint errors are found.\n# This is primarily useful in continuous integration scripts.\n#exit-zero=\n\n# A comma-separated list of package or module names from where C extensions may\n# be loaded. Extensions are loading into the active Python interpreter and may\n# run arbitrary code.\nextension-pkg-allow-list=\n\n# A comma-separated list of package or module names from where C extensions may\n# be loaded. Extensions are loading into the active Python interpreter and may\n# run arbitrary code. (This is an alternative name to extension-pkg-allow-list\n# for backward compatibility.)\nextension-pkg-whitelist=\n\n# Return non-zero exit code if any of these messages/categories are detected,\n# even if score is above --fail-under value. Syntax same as enable. Messages\n# specified are enabled, while categories only check already-enabled messages.\nfail-on=\n\n# Specify a score threshold under which the program will exit with error.\nfail-under=10\n\n# Interpret the stdin as a python script, whose filename needs to be passed as\n# the module_or_package argument.\n#from-stdin=\n\n# Files or directories to be skipped. They should be base names, not paths.\nignore=CVS\n\n# Add files or directories matching the regular expressions patterns to the\n# ignore-list. The regex matches against paths and can be in Posix or Windows\n# format. Because '\\\\' represents the directory delimiter on Windows systems,\n# it can't be used as an escape character.\nignore-paths=\n\n# Files or directories matching the regular expression patterns are skipped.\n# The regex matches against base names, not paths. The default value ignores\n# Emacs file locks\nignore-patterns=^\\.#\n\n# List of module names for which member attributes should not be checked\n# (useful for modules/projects where namespaces are manipulated during runtime\n# and thus existing member attributes cannot be deduced by static analysis). It\n# supports qualified module names, as well as Unix pattern matching.\nignored-modules=\n\n# Python code to execute, usually for sys.path manipulation such as\n# pygtk.require().\n#init-hook=\n\n# Use multiple processes to speed up Pylint. Specifying 0 will auto-detect the\n# number of processors available to use, and will cap the count on Windows to\n# avoid hangs.\njobs=1\n\n# Control the amount of potential inferred values when inferring a single\n# object. This can help the performance when dealing with large functions or\n# complex, nested conditions.\nlimit-inference-results=100\n\n# List of plugins (as comma separated values of python module names) to load,\n# usually to register additional checkers.\nload-plugins=\n\n# Pickle collected data for later comparisons.\npersistent=yes\n\n# Minimum Python version to use for version dependent checks. Will default to\n# the version used to run pylint.\npy-version=3.11\n\n# Discover python modules and packages in the file system subtree.\nrecursive=no\n\n# Add paths to the list of the source roots. Supports globbing patterns. The\n# source root is an absolute path or a path relative to the current working\n# directory used to determine a package namespace for modules located under the\n# source root.\nsource-roots=\n\n# When enabled, pylint would attempt to guess common misconfiguration and emit\n# user-friendly hints instead of false-positive error messages.\nsuggestion-mode=yes\n\n# Allow loading of arbitrary C extensions. Extensions are imported into the\n# active Python interpreter and may run arbitrary code.\nunsafe-load-any-extension=no\n\n# In verbose mode, extra non-checker-related info will be displayed.\n#verbose=\n\n\n[BASIC]\n\n# Naming style matching correct argument names.\nargument-naming-style=snake_case\n\n# Regular expression matching correct argument names. Overrides argument-\n# naming-style. If left empty, argument names will be checked with the set\n# naming style.\n#argument-rgx=\n\n# Naming style matching correct attribute names.\nattr-naming-style=snake_case\n\n# Regular expression matching correct attribute names. Overrides attr-naming-\n# style. If left empty, attribute names will be checked with the set naming\n# style.\n#attr-rgx=\n\n# Bad variable names which should always be refused, separated by a comma.\nbad-names=foo,\n          bar,\n          baz,\n          toto,\n          tutu,\n          tata\n\n# Bad variable names regexes, separated by a comma. If names match any regex,\n# they will always be refused\nbad-names-rgxs=\n\n# Naming style matching correct class attribute names.\nclass-attribute-naming-style=any\n\n# Regular expression matching correct class attribute names. Overrides class-\n# attribute-naming-style. If left empty, class attribute names will be checked\n# with the set naming style.\n#class-attribute-rgx=\n\n# Naming style matching correct class constant names.\nclass-const-naming-style=UPPER_CASE\n\n# Regular expression matching correct class constant names. Overrides class-\n# const-naming-style. If left empty, class constant names will be checked with\n# the set naming style.\n#class-const-rgx=\n\n# Naming style matching correct class names.\nclass-naming-style=PascalCase\n\n# Regular expression matching correct class names. Overrides class-naming-\n# style. If left empty, class names will be checked with the set naming style.\n#class-rgx=\n\n# Naming style matching correct constant names.\nconst-naming-style=UPPER_CASE\n\n# Regular expression matching correct constant names. Overrides const-naming-\n# style. If left empty, constant names will be checked with the set naming\n# style.\n#const-rgx=\n\n# Minimum line length for functions/classes that require docstrings, shorter\n# ones are exempt.\ndocstring-min-length=-1\n\n# Naming style matching correct function names.\nfunction-naming-style=snake_case\n\n# Regular expression matching correct function names. Overrides function-\n# naming-style. If left empty, function names will be checked with the set\n# naming style.\n#function-rgx=\n\n# Good variable names which should always be accepted, separated by a comma.\ngood-names=i,\n           j,\n           k,\n           ex,\n           Run,\n           _\n\n# Good variable names regexes, separated by a comma. If names match any regex,\n# they will always be accepted\ngood-names-rgxs=\n\n# Include a hint for the correct naming format with invalid-name.\ninclude-naming-hint=no\n\n# Naming style matching correct inline iteration names.\ninlinevar-naming-style=any\n\n# Regular expression matching correct inline iteration names. Overrides\n# inlinevar-naming-style. If left empty, inline iteration names will be checked\n# with the set naming style.\n#inlinevar-rgx=\n\n# Naming style matching correct method names.\nmethod-naming-style=snake_case\n\n# Regular expression matching correct method names. Overrides method-naming-\n# style. If left empty, method names will be checked with the set naming style.\n#method-rgx=\n\n# Naming style matching correct module names.\nmodule-naming-style=snake_case\n\n# Regular expression matching correct module names. Overrides module-naming-\n# style. If left empty, module names will be checked with the set naming style.\n#module-rgx=\n\n# Colon-delimited sets of names that determine each other's naming style when\n# the name regexes allow several styles.\nname-group=\n\n# Regular expression which should only match function or class names that do\n# not require a docstring.\nno-docstring-rgx=^_\n\n# List of decorators that produce properties, such as abc.abstractproperty. Add\n# to this list to register other decorators that produce valid properties.\n# These decorators are taken in consideration only for invalid-name.\nproperty-classes=abc.abstractproperty\n\n# Regular expression matching correct type alias names. If left empty, type\n# alias names will be checked with the set naming style.\n#typealias-rgx=\n\n# Regular expression matching correct type variable names. If left empty, type\n# variable names will be checked with the set naming style.\n#typevar-rgx=\n\n# Naming style matching correct variable names.\nvariable-naming-style=snake_case\n\n# Regular expression matching correct variable names. Overrides variable-\n# naming-style. If left empty, variable names will be checked with the set\n# naming style.\n#variable-rgx=\n\n\n[CLASSES]\n\n# Warn about protected attribute access inside special methods\ncheck-protected-access-in-special-methods=no\n\n# List of method names used to declare (i.e. assign) instance attributes.\ndefining-attr-methods=__init__,\n                      __new__,\n                      setUp,\n                      asyncSetUp,\n                      __post_init__\n\n# List of member names, which should be excluded from the protected access\n# warning.\nexclude-protected=_asdict,_fields,_replace,_source,_make,os._exit\n\n# List of valid names for the first argument in a class method.\nvalid-classmethod-first-arg=cls\n\n# List of valid names for the first argument in a metaclass class method.\nvalid-metaclass-classmethod-first-arg=mcs\n\n\n[DESIGN]\n\n# List of regular expressions of class ancestor names to ignore when counting\n# public methods (see R0903)\nexclude-too-few-public-methods=\n\n# List of qualified class names to ignore when counting class parents (see\n# R0901)\nignored-parents=\n\n# Maximum number of arguments for function / method.\nmax-args=5\n\n# Maximum number of attributes for a class (see R0902).\nmax-attributes=7\n\n# Maximum number of boolean expressions in an if statement (see R0916).\nmax-bool-expr=5\n\n# Maximum number of branch for function / method body.\nmax-branches=12\n\n# Maximum number of locals for function / method body.\nmax-locals=15\n\n# Maximum number of parents for a class (see R0901).\nmax-parents=7\n\n# Maximum number of public methods for a class (see R0904).\nmax-public-methods=20\n\n# Maximum number of return / yield for function / method body.\nmax-returns=6\n\n# Maximum number of statements in function / method body.\nmax-statements=50\n\n# Minimum number of public methods for a class (see R0903).\nmin-public-methods=2\n\n\n[EXCEPTIONS]\n\n# Exceptions that will emit a warning when caught.\novergeneral-exceptions=builtins.BaseException,builtins.Exception\n\n\n[FORMAT]\n\n# Expected format of line ending, e.g. empty (any line ending), LF or CRLF.\nexpected-line-ending-format=\n\n# Regexp for a line that is allowed to be longer than the limit.\nignore-long-lines=^\\s*(# )?<?https?://\\S+>?$\n\n# Number of spaces of indent required inside a hanging or continued line.\nindent-after-paren=4\n\n# String used as indentation unit. This is usually \"    \" (4 spaces) or \"\\t\" (1\n# tab).\nindent-string='    '\n\n# Maximum number of characters on a single line.\nmax-line-length=120\n\n# Maximum number of lines in a module.\nmax-module-lines=1000\n\n# Allow the body of a class to be on the same line as the declaration if body\n# contains single statement.\nsingle-line-class-stmt=no\n\n# Allow the body of an if to be on the same line as the test if there is no\n# else.\nsingle-line-if-stmt=no\n\n\n[IMPORTS]\n\n# List of modules that can be imported at any level, not just the top level\n# one.\nallow-any-import-level=\n\n# Allow explicit reexports by alias from a package __init__.\nallow-reexport-from-package=no\n\n# Allow wildcard imports from modules that define __all__.\nallow-wildcard-with-all=no\n\n# Deprecated modules which should not be used, separated by a comma.\ndeprecated-modules=\n\n# Output a graph (.gv or any supported image format) of external dependencies\n# to the given file (report RP0402 must not be disabled).\next-import-graph=\n\n# Output a graph (.gv or any supported image format) of all (i.e. internal and\n# external) dependencies to the given file (report RP0402 must not be\n# disabled).\nimport-graph=\n\n# Output a graph (.gv or any supported image format) of internal dependencies\n# to the given file (report RP0402 must not be disabled).\nint-import-graph=\n\n# Force import order to recognize a module as part of the standard\n# compatibility libraries.\nknown-standard-library=\n\n# Force import order to recognize a module as part of a third party library.\nknown-third-party=enchant\n\n# Couples of modules and preferred modules, separated by a comma.\npreferred-modules=\n\n\n[LOGGING]\n\n# The type of string formatting that logging methods do. `old` means using %\n# formatting, `new` is for `{}` formatting.\nlogging-format-style=old\n\n# Logging modules to check that the string format arguments are in logging\n# function parameter format.\nlogging-modules=logging\n\n\n[MESSAGES CONTROL]\n\n# Only show warnings with the listed confidence levels. Leave empty to show\n# all. Valid levels: HIGH, CONTROL_FLOW, INFERENCE, INFERENCE_FAILURE,\n# UNDEFINED.\nconfidence=HIGH,\n           CONTROL_FLOW,\n           INFERENCE,\n           INFERENCE_FAILURE,\n           UNDEFINED\n\n# Disable the message, report, category or checker with the given id(s). You\n# can either give multiple identifiers separated by comma (,) or put this\n# option multiple times (only on the command line, not in the configuration\n# file where it should appear only once). You can also use \"--disable=all\" to\n# disable everything first and then re-enable specific checks. For example, if\n# you want to run only the similarities checker, you can use \"--disable=all\n# --enable=similarities\". If you want to run only the classes checker, but have\n# no Warning level messages displayed, use \"--disable=all --enable=classes\n# --disable=W\".\ndisable=raw-checker-failed,\n        bad-inline-option,\n        locally-disabled,\n        file-ignored,\n        suppressed-message,\n        useless-suppression,\n        deprecated-pragma,\n        use-symbolic-message-instead,\n        missing-class-docstring,\n        missing-function-docstring,\n        super-with-arguments,\n        consider-using-f-string\n\n# Enable the message, report, category or checker with the given id(s). You can\n# either give multiple identifier separated by comma (,) or put this option\n# multiple time (only on the command line, not in the configuration file where\n# it should appear only once). See also the \"--disable\" option for examples.\nenable=c-extension-no-member\n\n\n[METHOD_ARGS]\n\n# List of qualified names (i.e., library.method) which require a timeout\n# parameter e.g. 'requests.api.get,requests.api.post'\ntimeout-methods=requests.api.delete,requests.api.get,requests.api.head,requests.api.options,requests.api.patch,requests.api.post,requests.api.put,requests.api.request\n\n\n[MISCELLANEOUS]\n\n# List of note tags to take in consideration, separated by a comma.\nnotes=FIXME,\n      XXX,\n      TODO\n\n# Regular expression of note tags to take in consideration.\nnotes-rgx=\n\n\n[REFACTORING]\n\n# Maximum number of nested blocks for function / method body\nmax-nested-blocks=5\n\n# Complete name of functions that never returns. When checking for\n# inconsistent-return-statements if a never returning function is called then\n# it will be considered as an explicit return statement and no message will be\n# printed.\nnever-returning-functions=sys.exit,argparse.parse_error\n\n\n[REPORTS]\n\n# Python expression which should return a score less than or equal to 10. You\n# have access to the variables 'fatal', 'error', 'warning', 'refactor',\n# 'convention', and 'info' which contain the number of messages in each\n# category, as well as 'statement' which is the total number of statements\n# analyzed. This score is used by the global evaluation report (RP0004).\nevaluation=max(0, 0 if fatal else 10.0 - ((float(5 * error + warning + refactor + convention) / statement) * 10))\n\n# Template used to display messages. This is a python new-style format string\n# used to format the message information. See doc for all details.\nmsg-template=\n\n# Set the output format. Available formats are text, parseable, colorized, json\n# and msvs (visual studio). You can also give a reporter class, e.g.\n# mypackage.mymodule.MyReporterClass.\n#output-format=\n\n# Tells whether to display a full report or only the messages.\nreports=no\n\n# Activate the evaluation score.\nscore=yes\n\n\n[SIMILARITIES]\n\n# Comments are removed from the similarity computation\nignore-comments=yes\n\n# Docstrings are removed from the similarity computation\nignore-docstrings=yes\n\n# Imports are removed from the similarity computation\nignore-imports=yes\n\n# Signatures are removed from the similarity computation\nignore-signatures=yes\n\n# Minimum lines number of a similarity.\nmin-similarity-lines=4\n\n\n[SPELLING]\n\n# Limits count of emitted suggestions for spelling mistakes.\nmax-spelling-suggestions=4\n\n# Spelling dictionary name. No available dictionaries : You need to install\n# both the python package and the system dependency for enchant to work..\nspelling-dict=\n\n# List of comma separated words that should be considered directives if they\n# appear at the beginning of a comment and should not be checked.\nspelling-ignore-comment-directives=fmt: on,fmt: off,noqa:,noqa,nosec,isort:skip,mypy:\n\n# List of comma separated words that should not be checked.\nspelling-ignore-words=\n\n# A path to a file that contains the private dictionary; one word per line.\nspelling-private-dict-file=\n\n# Tells whether to store unknown words to the private dictionary (see the\n# --spelling-private-dict-file option) instead of raising a message.\nspelling-store-unknown-words=no\n\n\n[STRING]\n\n# This flag controls whether inconsistent-quotes generates a warning when the\n# character used as a quote delimiter is used inconsistently within a module.\ncheck-quote-consistency=no\n\n# This flag controls whether the implicit-str-concat should generate a warning\n# on implicit string concatenation in sequences defined over several lines.\ncheck-str-concat-over-line-jumps=no\n\n\n[TYPECHECK]\n\n# List of decorators that produce context managers, such as\n# contextlib.contextmanager. Add to this list to register other decorators that\n# produce valid context managers.\ncontextmanager-decorators=contextlib.contextmanager\n\n# List of members which are set dynamically and missed by pylint inference\n# system, and so shouldn't trigger E1101 when accessed. Python regular\n# expressions are accepted.\ngenerated-members=\n\n# Tells whether to warn about missing members when the owner of the attribute\n# is inferred to be None.\nignore-none=yes\n\n# This flag controls whether pylint should warn about no-member and similar\n# checks whenever an opaque object is returned when inferring. The inference\n# can return multiple potential results while evaluating a Python object, but\n# some branches might not be evaluated, which results in partial inference. In\n# that case, it might be useful to still emit no-member and other checks for\n# the rest of the inferred objects.\nignore-on-opaque-inference=yes\n\n# List of symbolic message names to ignore for Mixin members.\nignored-checks-for-mixins=no-member,\n                          not-async-context-manager,\n                          not-context-manager,\n                          attribute-defined-outside-init\n\n# List of class names for which member attributes should not be checked (useful\n# for classes with dynamically set attributes). This supports the use of\n# qualified names.\nignored-classes=optparse.Values,thread._local,_thread._local,argparse.Namespace\n\n# Show a hint with possible names when a member name was not found. The aspect\n# of finding the hint is based on edit distance.\nmissing-member-hint=yes\n\n# The minimum edit distance a name should have in order to be considered a\n# similar match for a missing member name.\nmissing-member-hint-distance=1\n\n# The total number of similar names that should be taken in consideration when\n# showing a hint for a missing member.\nmissing-member-max-choices=1\n\n# Regex pattern to define which classes are considered mixins.\nmixin-class-rgx=.*[Mm]ixin\n\n# List of decorators that change the signature of a decorated function.\nsignature-mutators=\n\n\n[VARIABLES]\n\n# List of additional names supposed to be defined in builtins. Remember that\n# you should avoid defining new builtins when possible.\nadditional-builtins=\n\n# Tells whether unused global variables should be treated as a violation.\nallow-global-unused-variables=yes\n\n# List of names allowed to shadow builtins\nallowed-redefined-builtins=\n\n# List of strings which can identify a callback function by name. A callback\n# name must start or end with one of those strings.\ncallbacks=cb_,\n          _cb\n\n# A regular expression matching the name of dummy variables (i.e. expected to\n# not be used).\ndummy-variables-rgx=_+$|(_[a-zA-Z0-9_]*[a-zA-Z0-9]+?$)|dummy|^ignored_|^unused_\n\n# Argument names that match this expression will be ignored.\nignored-argument-names=_.*|^ignored_|^unused_\n\n# Tells whether we should check for unused import in __init__ files.\ninit-import=no\n\n# List of qualified module names which can have objects that can redefine\n# builtins.\nredefining-builtins-modules=six.moves,past.builtins,future.builtins,builtins,io\n"
  },
  {
    "path": ".scrutinizer.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-17 11:41:13 +0000 (Tue, 17 Mar 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nbuild:\n  image: default-bionic\n  nodes:\n    auto:\n      commands:\n        - repo=\"${SCRUTINIZER_PROJECT#*/}\"; git clone \"https://github.com/$repo\" build\n        - cd ~/build\n        - pwd\n        - ls -l\n        - make init\n        - make ci test\n"
  },
  {
    "path": ".semaphore/semaphore.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-16 14:02:53 +0000 (Mon, 16 Mar 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            S e m a p h o r e   C I\n# ============================================================================ #\n\n# https://docs.semaphoreci.com/reference/pipeline-yaml-reference/\n\nversion: v1.0\nname: DevOps-Bash-tools\nagent:\n  # https://docs.semaphoreci.com/reference/machine-types#linux\n  machine:\n    type: e1-standard-2\n    os_image: ubuntu2004\nexecution_time_limit:\n  hours: 3\nblocks:\n  - name: Linux build\n    run:\n      when: \"branch = 'master' AND change_in('/', {exclude: ['**/*.md']})\"\n    #execution_time_limit:\n    #  hours: 2\n    task:\n      #env_vars:\n        # $PATH selects /usr/bin/python and /usr/local/bin/pip which are mismatched versions of Python\n        #- name: PYTHON\n        #  value: python3\n        #- name: PIP\n        #  value: pip3\n      prologue:\n        commands:\n          - cache restore\n          # prevents it getting stuck on config merge prompt on installing openssh-client pulling in openssh-server\n          #\n          # causes error:\n          #\n          #   Not replacing deleted config file /etc/ssh/sshd_config\n          #\n          #- sudo rm -f /etc/ssh/sshd_config\n          - export DEBIAN_FRONTEND=noninteractive\n          - sudo -E apt-get update\n          - sudo -E apt-get upgrade -y -o Dpkg::Options::=\"--force-confmiss\" -o Dpkg::Options::=\"--force-confnew\"\n          - sudo dpkg --configure -a --force-confmiss --force-confnew\n          #- echo \"openssh-server openssh-server/conffile-diff select keep\" | sudo debconf-set-selections\n          #- sudo dpkg --configure -a --force-confdef --force-confold\n          #- sudo apt-get upgrade -y -o Dpkg::Options::=\"--force-confdef\" -o Dpkg::Options::=\"--force-confold\"\n          - sudo apt-get install -y openssh-server\n      # each job is separate and could be run on a separate machine so all steps must be together\n      jobs:\n        - name: build\n          commands:\n            - checkout\n            - setup/ci_bootstrap.sh\n            - make init\n            - make ci\n            - make test\n      epilogue:\n        commands:\n          - cache store\n  - name: Mac build\n    run:\n      when: \"branch = 'master'\"\n    task:\n      # because otherwise on Mac it uses /usr/bin/python (2.7) but /usr/local/bin/pip (python 3.8)\n      #env_vars:\n        # to match /usr/local/bin/pip version from $PATH\n        #- name: PYTHON\n        #  value: python3\n        # must be quoted to force string, otherwise pipeline fails to run with this parsing error:\n        # Error: [{\"Type mismatch. Expected String but got Integer.\", \"#/blocks/1/task/env_vars/1/value\"}]\n        #- name: DEBUG\n        #  value: \"1\"\n      agent:\n        # https://docs.semaphoreci.com/reference/machine-types#macos\n        machine:\n          type: a1-standard-4\n          os_image: macos-xcode15\n      prologue:\n        commands:\n          - cache restore\n          # fix for:\n          # pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.\n          - brew install openssl\n          - brew reinstall python\n          - brew reinstall wget\n          # avoid Mac SSL errors:\n          #\n          # ERROR:  Loading command: install (LoadError)\n          #   dlopen(/Users/semaphore/.rbenv/versions/2.5.1/lib/ruby/2.5.0/x86_64-darwin18/openssl.bundle, 9): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib\n          #   Referenced from: /Users/semaphore/.rbenv/versions/2.5.1/lib/ruby/2.5.0/x86_64-darwin18/openssl.bundle\n          #   Reason: image not found - /Users/semaphore/.rbenv/versions/2.5.1/lib/ruby/2.5.0/x86_64-darwin18/openssl.bundle\n          # ERROR:  While executing gem ... (NoMethodError)\n          #     undefined method `invoke_with_build_args' for nil:NilClass#\n          #\n          - rbenv global system\n          # also considered this:\n          # - for version in $(rbenv versions | grep -v system | sed 's/^\\*//'); do yes | rbenv uninstall \"$version\"; rbenv install \"$version\"; done\n          #\n          # fix for python vs pip version mismatch\n          - ln -svf -- /usr/local/bin/python3 /usr/local/bin/python\n      jobs:\n        - name: build\n          commands:\n            - checkout\n            - make init\n            - make ci\n            - make test\n      epilogue:\n        commands:\n          - cache store\n"
  },
  {
    "path": ".sonarcloud.properties",
    "content": "sonar.host.url=https://sonarcloud.io\n"
  },
  {
    "path": ".sonarlint/connectedMode.json",
    "content": "{\n    \"sonarCloudOrganization\": \"harisekhon\",\n    \"projectKey\": \"HariSekhon_DevOps-Bash-tools\"\n}\n"
  },
  {
    "path": ".terraformignore",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-09-24 17:08:01 +0100 (Thu, 24 Sep 2020)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Requires Terraform 0.12.11+\n#\n# Prevents upload of paths to Terraform Cloud\n#\n# Same format as .gitignore\n# - directories must end with forward slash /\n# - negate matches using !\n#\n# Only works at the root of the config directory\n\n# https://www.terraform.io/docs/backends/types/remote.html#excluding-files-from-upload-with-terraformignore\n\n# defaults\n.git/\n.terraform/\n\n# custom\n.hg/\n.svn/\n.ssh/\ngithub/\ngitroot/\nmercurial/\nhg/\nhgroot/\nsvn/\nsvnroot/\n\n# exclude all hidden dot files\n.*\n"
  },
  {
    "path": ".trivyignore",
    "content": "#aws-access-key-id\n#aws-account-id\ngcp-service-account\n"
  },
  {
    "path": ".zlogin",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-13 18:58:03 +0000 (Fri, 13 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               Z S H   L o g i n\n# ============================================================================ #\n"
  },
  {
    "path": ".zlogout",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-13 18:58:03 +0000 (Fri, 13 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              Z S H   L o g o u t\n# ============================================================================ #\n"
  },
  {
    "path": ".zprofile",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#  (forked from .bash_profile)\n\n# ============================================================================ #\n#                             Z S H   P r o f i l e\n# ============================================================================ #\n\n# goes horribly wrong - too much advanced bash\n#if [[ -e ~/.profile  ]]; then\n#    emulate sh -c 'source ~/.profile'\n#fi\n"
  },
  {
    "path": ".zshenv",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-13 18:58:03 +0000 (Fri, 13 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 Z S H   E n v\n# ============================================================================ #\n\n# sourced by both interactive shells and scripts\n#\n# be careful with you put in here\n"
  },
  {
    "path": ".zshrc",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC1091\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2006-06-28 23:25:09 +0100 (Wed, 28 Jun 2006)\n#  (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                     Z S H\n# ============================================================================ #\n\n# https://wiki.archlinux.org/index.php/Zsh\n\n# goes horribly wrong - too much advanced bash\n#if [[ -e ~/.bashrc  ]]; then\n#    emulate sh -c 'source ~/.bashrc'\n#fi\n\nautoload -Uz compinit promptinit\ncompinit  # completes ssh/scp/sftp hostnames as long as HashKnownHosts not set in ~/.ssh/config\npromptinit\n\n# prompt -l - list themes\n# prompt -p - preview themes\n#prompt suse\n\n# install Oh-My-ZSH\n# sh -c \"$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)\"\n\n# custom themes:\n# mkdir ~/.zprompts\n# fpath=(\"$HOME/.zprompts\" \"$fpath[@]\")\n\n# uniq all items in $PATH and $path array\ntypeset -U PATH path\npath=(\"$HOME/.local/bin\" \"$HOME/bin\" \"$path[@]\")\nexport PATH\n\n# autocompletion with an arrow-key driven interface - tab twice to enable\nzstyle ':completion:*' menu select\n\n# autocompletions with sudo\n# allows zsh completion scripts run commands with sudo privileges - do not enable if using untrusted autocompletion scripts!!\n#zstyle ':completion::complete:*' gain-privileges 1\n\n# ============================================================================ #\n#                                S e t t i n g s\n# ============================================================================ #\n\n# compatible style with other shells\n#set -o AUTO_CD\n# casse insensitive, underscores stripped\nsetopt AUTO_CD\n\nsetopt COMPLETE_ALIASES\n\nsetopt CORRECT\nexport SPROMPT=\"Correct %R to %r? [Yes, No, Abort, Edit] \"\n\nautoload U colors && colors\n\n# expand wilcard expansion on unquoted variables like Bash\nsetopt GLOB_SUBST\n\nexport PATH=\"$PATH:/opt/homebrew/bin/\"\n\n# ============================================================================ #\n#                                   Oh-My-ZSH\n# ============================================================================ #\n\n# If you come from bash you might have to change your $PATH.\n# export PATH=$HOME/bin:/usr/local/bin:$PATH\n\n# Path to your oh-my-zsh installation.\nexport ZSH=\"/Users/hari.sekhon/.oh-my-zsh\"\n\n# Set name of the theme to load --- if set to \"random\", it will\n# load a random theme each time oh-my-zsh is loaded, in which case,\n# to know which specific one was loaded, run: echo $RANDOM_THEME\n# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes\nZSH_THEME=\"robbyrussell\"\n\n# also messed up\n#ZSH_THEME=\"agnoster\"\n\n# messes up both Terminal and iTerm2 from both brew and git cloned installations\n#if [ -f /usr/local/opt/powerlevel9k/powerlevel9k.zsh-theme ]; then\n#    source /usr/local/opt/powerlevel9k/powerlevel9k.zsh-theme\n#fi\n#\n# Oh-My-ZSH ~/.oh-my-zsh/custom/themes/powerlevel9k\n#ZSH_THEME=\"powerlevel9k/powerlevel9k\"\n\n# Set list of themes to pick from when loading at random\n# Setting this variable when ZSH_THEME=random will cause zsh to load\n# a theme from this variable instead of looking in ~/.oh-my-zsh/themes/\n# If set to an empty array, this variable will have no effect.\n# ZSH_THEME_RANDOM_CANDIDATES=( \"robbyrussell\" \"agnoster\" )\n\n# Uncomment the following line to use case-sensitive completion.\n# CASE_SENSITIVE=\"true\"\n\n# Uncomment the following line to use hyphen-insensitive completion.\n# Case-sensitive completion must be off. _ and - will be interchangeable.\n# HYPHEN_INSENSITIVE=\"true\"\n\n# Uncomment the following line to disable bi-weekly auto-update checks.\n# DISABLE_AUTO_UPDATE=\"true\"\n\n# Uncomment the following line to automatically update without prompting.\n# DISABLE_UPDATE_PROMPT=\"true\"\n\n# Uncomment the following line to change how often to auto-update (in days).\n# export UPDATE_ZSH_DAYS=13\n\n# Uncomment the following line if pasting URLs and other text is messed up.\n# DISABLE_MAGIC_FUNCTIONS=true\n\n# Uncomment the following line to disable colors in ls.\n# DISABLE_LS_COLORS=\"true\"\n\n# Uncomment the following line to disable auto-setting terminal title.\n# DISABLE_AUTO_TITLE=\"true\"\n\n# Uncomment the following line to enable command auto-correction.\n# ENABLE_CORRECTION=\"true\"\n\n# Uncomment the following line to display red dots whilst waiting for completion.\n# COMPLETION_WAITING_DOTS=\"true\"\n\n# Uncomment the following line if you want to disable marking untracked files\n# under VCS as dirty. This makes repository status check for large repositories\n# much, much faster.\n# DISABLE_UNTRACKED_FILES_DIRTY=\"true\"\n\n# Uncomment the following line if you want to change the command execution time\n# stamp shown in the history command output.\n# You can set one of the optional three formats:\n# \"mm/dd/yyyy\"|\"dd.mm.yyyy\"|\"yyyy-mm-dd\"\n# or set a custom format using the strftime function format specifications,\n# see 'man strftime' for details.\n# HIST_STAMPS=\"mm/dd/yyyy\"\n\n# Would you like to use another custom folder than $ZSH/custom?\n# ZSH_CUSTOM=/path/to/new-custom-folder\n\n# Which plugins would you like to load?\n# Standard plugins can be found in ~/.oh-my-zsh/plugins/*\n# Custom plugins may be added to ~/.oh-my-zsh/custom/plugins/\n# Example format: plugins=(rails git textmate ruby lighthouse)\n# Add wisely, as too many plugins slow down shell startup.\nplugins=(git)\n\nsource $ZSH/oh-my-zsh.sh\n\n# User configuration\n\n# export MANPATH=\"/usr/local/man:$MANPATH\"\n\n# You may need to manually set your language environment\n# export LANG=en_US.UTF-8\n\n# Preferred editor for local and remote sessions\n# if [[ -n $SSH_CONNECTION ]]; then\n#   export EDITOR='vim'\n# else\n#   export EDITOR='mvim'\n# fi\n\n# Compilation flags\n# export ARCHFLAGS=\"-arch x86_64\"\n\n# Set personal aliases, overriding those provided by oh-my-zsh libs,\n# plugins, and themes. Aliases can be placed here, though oh-my-zsh\n# users are encouraged to define aliases within the ZSH_CUSTOM folder.\n# For a full list of active aliases, run `alias`.\n#\n# Example aliases\n# alias zshconfig=\"mate ~/.zshrc\"\n# alias ohmyzsh=\"mate ~/.oh-my-zsh\"\n\n# ============================================================================ #\n\n# Lines configured by zsh-newuser-install\nHISTFILE=~/.histfile\nHISTSIZE=1000\nSAVEHIST=1000\nsetopt appendhistory autocd extendedglob nomatch notify\nunsetopt beep\nbindkey -e\n# End of lines configured by zsh-newuser-install\n# The following lines were added by compinstall\nzstyle :compinstall filename '/home/hari/.zshrc'\nautoload -Uz compinit\ncompinit\n# End of lines added by compinstall\n\n# added by travis gem - Travis is legacy now, don't bother with this\n#[ -f /Users/hari/.travis/travis.sh ] && source /Users/hari/.travis/travis.sh\n\nif type -P direnv &>/dev/null; then\n    eval \"$(direnv hook zsh)\"\nfi\n\nautoload -U +X bashcompinit && bashcompinit\ncomplete -o nospace -C /Users/hari/bin/terraform terraform\ncomplete -o nospace -C /Users/hari/bin/terraform tf\n\ncomplete -o nospace -C /usr/local/bin/terragrunt terragrunt\n\n#THIS MUST BE AT THE END OF THE FILE FOR SDKMAN TO WORK!!!\nexport SDKMAN_DIR=\"/Users/hari/.sdkman\"\n[[ -s \"/Users/hari/.sdkman/bin/sdkman-init.sh\" ]] && source \"/Users/hari/.sdkman/bin/sdkman-init.sh\"\n"
  },
  {
    "path": "DOCKER_STATUS.md",
    "content": "# Docker Status Page\n\ngenerated by `docker_generate_status_page.sh` in [HariSekhon/DevOps-Bash-tools](https://github.com/HariSekhon/DevOps-Bash-tools)\n\nThis page relies on shields.io which is slow so a lot of it may not load properly the first time so you may need to do one or more page reloads to get all the badges to load.\n\n50 docker repos - `:latest` tag build status:\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/alluxio.svg)](https://hub.docker.com/r/harisekhon/alluxio/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/alluxio.svg)](https://hub.docker.com/r/harisekhon/alluxio) -\n[harisekhon/alluxio](https://hub.docker.com/r/harisekhon/alluxio)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/alpine-dev.svg)](https://hub.docker.com/r/harisekhon/alpine-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/alpine-dev.svg)](https://hub.docker.com/r/harisekhon/alpine-dev) -\n[harisekhon/alpine-dev](https://hub.docker.com/r/harisekhon/alpine-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/alpine-github.svg)](https://hub.docker.com/r/harisekhon/alpine-github/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/alpine-github.svg)](https://hub.docker.com/r/harisekhon/alpine-github) -\n[harisekhon/alpine-github](https://hub.docker.com/r/harisekhon/alpine-github)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/apache-drill.svg)](https://hub.docker.com/r/harisekhon/apache-drill/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/apache-drill.svg)](https://hub.docker.com/r/harisekhon/apache-drill) -\n[harisekhon/apache-drill](https://hub.docker.com/r/harisekhon/apache-drill)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/cassandra-dev.svg)](https://hub.docker.com/r/harisekhon/cassandra-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/cassandra-dev.svg)](https://hub.docker.com/r/harisekhon/cassandra-dev) -\n[harisekhon/cassandra-dev](https://hub.docker.com/r/harisekhon/cassandra-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/centos-dev.svg)](https://hub.docker.com/r/harisekhon/centos-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/centos-dev.svg)](https://hub.docker.com/r/harisekhon/centos-dev) -\n[harisekhon/centos-dev](https://hub.docker.com/r/harisekhon/centos-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/centos-github.svg)](https://hub.docker.com/r/harisekhon/centos-github/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/centos-github.svg)](https://hub.docker.com/r/harisekhon/centos-github) -\n[harisekhon/centos-github](https://hub.docker.com/r/harisekhon/centos-github)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/centos-java.svg)](https://hub.docker.com/r/harisekhon/centos-java/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/centos-java.svg)](https://hub.docker.com/r/harisekhon/centos-java) -\n[harisekhon/centos-java](https://hub.docker.com/r/harisekhon/centos-java)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/centos-scala.svg)](https://hub.docker.com/r/harisekhon/centos-scala/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/centos-scala.svg)](https://hub.docker.com/r/harisekhon/centos-scala) -\n[harisekhon/centos-scala](https://hub.docker.com/r/harisekhon/centos-scala)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/ci_intentionally_broken_test_do_not_use.svg)](https://hub.docker.com/r/harisekhon/ci_intentionally_broken_test_do_not_use/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/ci_intentionally_broken_test_do_not_use.svg)](https://hub.docker.com/r/harisekhon/ci_intentionally_broken_test_do_not_use) -\n[harisekhon/ci_intentionally_broken_test_do_not_use](https://hub.docker.com/r/harisekhon/ci_intentionally_broken_test_do_not_use)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/collectd.svg)](https://hub.docker.com/r/harisekhon/collectd/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/collectd.svg)](https://hub.docker.com/r/harisekhon/collectd) -\n[harisekhon/collectd](https://hub.docker.com/r/harisekhon/collectd)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/consul.svg)](https://hub.docker.com/r/harisekhon/consul/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/consul.svg)](https://hub.docker.com/r/harisekhon/consul) -\n[harisekhon/consul](https://hub.docker.com/r/harisekhon/consul)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/consul-dev.svg)](https://hub.docker.com/r/harisekhon/consul-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/consul-dev.svg)](https://hub.docker.com/r/harisekhon/consul-dev) -\n[harisekhon/consul-dev](https://hub.docker.com/r/harisekhon/consul-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/debian-dev.svg)](https://hub.docker.com/r/harisekhon/debian-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/debian-dev.svg)](https://hub.docker.com/r/harisekhon/debian-dev) -\n[harisekhon/debian-dev](https://hub.docker.com/r/harisekhon/debian-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/debian-github.svg)](https://hub.docker.com/r/harisekhon/debian-github/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/debian-github.svg)](https://hub.docker.com/r/harisekhon/debian-github) -\n[harisekhon/debian-github](https://hub.docker.com/r/harisekhon/debian-github)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/debian-java.svg)](https://hub.docker.com/r/harisekhon/debian-java/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/debian-java.svg)](https://hub.docker.com/r/harisekhon/debian-java) -\n[harisekhon/debian-java](https://hub.docker.com/r/harisekhon/debian-java)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/h2o.svg)](https://hub.docker.com/r/harisekhon/h2o/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/h2o.svg)](https://hub.docker.com/r/harisekhon/h2o) -\n[harisekhon/h2o](https://hub.docker.com/r/harisekhon/h2o)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/hadoop.svg)](https://hub.docker.com/r/harisekhon/hadoop/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/hadoop.svg)](https://hub.docker.com/r/harisekhon/hadoop) -\n[harisekhon/hadoop](https://hub.docker.com/r/harisekhon/hadoop)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/hadoop-dev.svg)](https://hub.docker.com/r/harisekhon/hadoop-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/hadoop-dev.svg)](https://hub.docker.com/r/harisekhon/hadoop-dev) -\n[harisekhon/hadoop-dev](https://hub.docker.com/r/harisekhon/hadoop-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/hbase.svg)](https://hub.docker.com/r/harisekhon/hbase/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/hbase.svg)](https://hub.docker.com/r/harisekhon/hbase) -\n[harisekhon/hbase](https://hub.docker.com/r/harisekhon/hbase)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/hbase-dev.svg)](https://hub.docker.com/r/harisekhon/hbase-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/hbase-dev.svg)](https://hub.docker.com/r/harisekhon/hbase-dev) -\n[harisekhon/hbase-dev](https://hub.docker.com/r/harisekhon/hbase-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/jython.svg)](https://hub.docker.com/r/harisekhon/jython/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/jython.svg)](https://hub.docker.com/r/harisekhon/jython) -\n[harisekhon/jython](https://hub.docker.com/r/harisekhon/jython)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/kafka.svg)](https://hub.docker.com/r/harisekhon/kafka/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/kafka.svg)](https://hub.docker.com/r/harisekhon/kafka) -\n[harisekhon/kafka](https://hub.docker.com/r/harisekhon/kafka)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/mesos.svg)](https://hub.docker.com/r/harisekhon/mesos/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/mesos.svg)](https://hub.docker.com/r/harisekhon/mesos) -\n[harisekhon/mesos](https://hub.docker.com/r/harisekhon/mesos)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/nagios-plugin-kafka.svg)](https://hub.docker.com/r/harisekhon/nagios-plugin-kafka/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/nagios-plugin-kafka.svg)](https://hub.docker.com/r/harisekhon/nagios-plugin-kafka) -\n[harisekhon/nagios-plugin-kafka](https://hub.docker.com/r/harisekhon/nagios-plugin-kafka)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/nagios-plugins.svg)](https://hub.docker.com/r/harisekhon/nagios-plugins/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/nagios-plugins.svg)](https://hub.docker.com/r/harisekhon/nagios-plugins) -\n[harisekhon/nagios-plugins](https://hub.docker.com/r/harisekhon/nagios-plugins)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/nifi.svg)](https://hub.docker.com/r/harisekhon/nifi/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/nifi.svg)](https://hub.docker.com/r/harisekhon/nifi) -\n[harisekhon/nifi](https://hub.docker.com/r/harisekhon/nifi)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/presto.svg)](https://hub.docker.com/r/harisekhon/presto/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/presto.svg)](https://hub.docker.com/r/harisekhon/presto) -\n[harisekhon/presto](https://hub.docker.com/r/harisekhon/presto)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/presto-cli.svg)](https://hub.docker.com/r/harisekhon/presto-cli/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/presto-cli.svg)](https://hub.docker.com/r/harisekhon/presto-cli) -\n[harisekhon/presto-cli](https://hub.docker.com/r/harisekhon/presto-cli)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/presto-cli-dev.svg)](https://hub.docker.com/r/harisekhon/presto-cli-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/presto-cli-dev.svg)](https://hub.docker.com/r/harisekhon/presto-cli-dev) -\n[harisekhon/presto-cli-dev](https://hub.docker.com/r/harisekhon/presto-cli-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/presto-dev.svg)](https://hub.docker.com/r/harisekhon/presto-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/presto-dev.svg)](https://hub.docker.com/r/harisekhon/presto-dev) -\n[harisekhon/presto-dev](https://hub.docker.com/r/harisekhon/presto-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/pytools.svg)](https://hub.docker.com/r/harisekhon/pytools/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/pytools.svg)](https://hub.docker.com/r/harisekhon/pytools) -\n[harisekhon/pytools](https://hub.docker.com/r/harisekhon/pytools)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/rabbitmq-cluster.svg)](https://hub.docker.com/r/harisekhon/rabbitmq-cluster/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/rabbitmq-cluster.svg)](https://hub.docker.com/r/harisekhon/rabbitmq-cluster) -\n[harisekhon/rabbitmq-cluster](https://hub.docker.com/r/harisekhon/rabbitmq-cluster)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/riak.svg)](https://hub.docker.com/r/harisekhon/riak/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/riak.svg)](https://hub.docker.com/r/harisekhon/riak) -\n[harisekhon/riak](https://hub.docker.com/r/harisekhon/riak)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/riak-dev.svg)](https://hub.docker.com/r/harisekhon/riak-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/riak-dev.svg)](https://hub.docker.com/r/harisekhon/riak-dev) -\n[harisekhon/riak-dev](https://hub.docker.com/r/harisekhon/riak-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/serf.svg)](https://hub.docker.com/r/harisekhon/serf/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/serf.svg)](https://hub.docker.com/r/harisekhon/serf) -\n[harisekhon/serf](https://hub.docker.com/r/harisekhon/serf)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/solr.svg)](https://hub.docker.com/r/harisekhon/solr/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/solr.svg)](https://hub.docker.com/r/harisekhon/solr) -\n[harisekhon/solr](https://hub.docker.com/r/harisekhon/solr)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/solrcloud.svg)](https://hub.docker.com/r/harisekhon/solrcloud/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/solrcloud.svg)](https://hub.docker.com/r/harisekhon/solrcloud) -\n[harisekhon/solrcloud](https://hub.docker.com/r/harisekhon/solrcloud)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/solrcloud-dev.svg)](https://hub.docker.com/r/harisekhon/solrcloud-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/solrcloud-dev.svg)](https://hub.docker.com/r/harisekhon/solrcloud-dev) -\n[harisekhon/solrcloud-dev](https://hub.docker.com/r/harisekhon/solrcloud-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/spark.svg)](https://hub.docker.com/r/harisekhon/spark/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/spark.svg)](https://hub.docker.com/r/harisekhon/spark) -\n[harisekhon/spark](https://hub.docker.com/r/harisekhon/spark)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/spotify-tools.svg)](https://hub.docker.com/r/harisekhon/spotify-tools/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/spotify-tools.svg)](https://hub.docker.com/r/harisekhon/spotify-tools) -\n[harisekhon/spotify-tools](https://hub.docker.com/r/harisekhon/spotify-tools)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/superset.svg)](https://hub.docker.com/r/harisekhon/superset/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/superset.svg)](https://hub.docker.com/r/harisekhon/superset) -\n[harisekhon/superset](https://hub.docker.com/r/harisekhon/superset)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/tachyon.svg)](https://hub.docker.com/r/harisekhon/tachyon/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/tachyon.svg)](https://hub.docker.com/r/harisekhon/tachyon) -\n[harisekhon/tachyon](https://hub.docker.com/r/harisekhon/tachyon)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/tcollector.svg)](https://hub.docker.com/r/harisekhon/tcollector/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/tcollector.svg)](https://hub.docker.com/r/harisekhon/tcollector) -\n[harisekhon/tcollector](https://hub.docker.com/r/harisekhon/tcollector)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/tools.svg)](https://hub.docker.com/r/harisekhon/tools/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/tools.svg)](https://hub.docker.com/r/harisekhon/tools) -\n[harisekhon/tools](https://hub.docker.com/r/harisekhon/tools)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/ubuntu-dev.svg)](https://hub.docker.com/r/harisekhon/ubuntu-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/ubuntu-dev.svg)](https://hub.docker.com/r/harisekhon/ubuntu-dev) -\n[harisekhon/ubuntu-dev](https://hub.docker.com/r/harisekhon/ubuntu-dev)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/ubuntu-github.svg)](https://hub.docker.com/r/harisekhon/ubuntu-github/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/ubuntu-github.svg)](https://hub.docker.com/r/harisekhon/ubuntu-github) -\n[harisekhon/ubuntu-github](https://hub.docker.com/r/harisekhon/ubuntu-github)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/ubuntu-java.svg)](https://hub.docker.com/r/harisekhon/ubuntu-java/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/ubuntu-java.svg)](https://hub.docker.com/r/harisekhon/ubuntu-java) -\n[harisekhon/ubuntu-java](https://hub.docker.com/r/harisekhon/ubuntu-java)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/zookeeper.svg)](https://hub.docker.com/r/harisekhon/zookeeper/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/zookeeper.svg)](https://hub.docker.com/r/harisekhon/zookeeper) -\n[harisekhon/zookeeper](https://hub.docker.com/r/harisekhon/zookeeper)\n\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/zookeeper-dev.svg)](https://hub.docker.com/r/harisekhon/zookeeper-dev/builds)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/zookeeper-dev.svg)](https://hub.docker.com/r/harisekhon/zookeeper-dev) -\n[harisekhon/zookeeper-dev](https://hub.docker.com/r/harisekhon/zookeeper-dev)\n\n"
  },
  {
    "path": "Gemfile",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-05-13 15:25:18 +0100 (Fri, 13 May 2022)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                 G e m f i l e\n# ============================================================================ #\n\n# https://bundler.io/gemfile.html\n\n# This isn't automatically installed since ruby code is not much used in this repo\n\n# see also:\n#\n#   setup/gem-packages.txt\n#   setup/gem-packages-desktop.txt  # optional for desktop use\n\nsource 'https://rubygems.org'\n\ngem 'cfn-nag'\ngem 'json'\ngem 'gitlab'\n"
  },
  {
    "path": "Jenkinsfile",
    "content": "//  vim:ts=4:sts=4:sw=4:et:filetype=groovy:syntax=groovy\n//\n//  Author: Hari Sekhon\n//  Date: 2017-06-28 12:39:02 +0200 (Wed, 28 Jun 2017)\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// ========================================================================== //\n//                        J e n k i n s   P i p e l i n e\n// ========================================================================== //\n\n// Epic Jenkinsfile template:\n//\n// https://github.com/HariSekhon/Templates/blob/master/Jenkinsfile\n\n\n// Official Documentation:\n//\n// https://jenkins.io/doc/book/pipeline/syntax/\n//\n// https://www.jenkins.io/doc/pipeline/steps/\n//\n// https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/\n\n\npipeline {\n  // to run on Docker or Kubernetes, see the master Jenkinsfile template listed at the top\n  agent any\n\n  options {\n    timestamps()\n\n    timeout(time: 2, unit: 'HOURS')\n  }\n\n  triggers {\n    cron('H 10 * * 1-5')\n    pollSCM('H/2 * * * *')\n  }\n\n  stages {\n    stage ('Checkout') {\n      steps {\n        checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: '', url: 'https://github.com/HariSekhon/DevOps-Bash-tools']]])\n      }\n    }\n\n    stage('Build') {\n      steps {\n        echo \"Running ${env.JOB_NAME} Build ${env.BUILD_ID} on ${env.JENKINS_URL}\"\n        echo 'Building...'\n        timeout(time: 10, unit: 'MINUTES') {\n          retry(3) {\n//            sh 'apt update -q'\n//            sh 'apt install -qy make'\n//            sh 'make init'\n            sh \"\"\"\n              setup/ci_bootstrap.sh &&\n              make init\n            \"\"\"\n          }\n        }\n        timeout(time: 180, unit: 'MINUTES') {\n          sh 'make ci'\n        }\n      }\n    }\n\n    stage('Test') {\n      options {\n        retry(2)\n      }\n      steps {\n        echo 'Testing...'\n        timeout(time: 120, unit: 'MINUTES') {\n          sh 'make test'\n        }\n      }\n    }\n  }\n}\n"
  },
  {
    "path": "LICENSE",
    "content": "Copyright 2016 Hari Sekhon\n\nPermission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.\n"
  },
  {
    "path": "Makefile",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:56:53 +0000 (Sun, 17 Jan 2016)\n#\n#  vim:ts=4:sts=4:sw=4:noet\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\ninclude Makefile.in\n\nREPO := HariSekhon/DevOps-Bash-tools\n\nCONF_FILES := $(shell sed \"s/\\#.*//; /^[[:space:]]*$$/d\" setup/files.txt)\n\n#CODE_FILES := $(shell find . -type f -name '*.sh' -o -type f -name '.bash*' | sort)\n#CODE_FILES := $(shell git ls-files | grep -E -e '\\.sh$$' -e '\\.bash[^/]*$$' -e '\\.groovy$$' | sort)\nCODE_FILES := $(shell \\\n\tif type git >/dev/null 2>&1; then \\\n\t\tgit ls-files | \\\n\t\tgrep -E -e '\\.sh$$' -e '\\.bash[^/]*$$' -e '\\.groovy$$' | \\\n\t\tsort | \\\n\t\twhile read -r filepath; do \\\n\t\t\ttest -f \"$$filepath\" || continue; \\\n\t\t\ttest -d \"$$filepath\" && continue; \\\n\t\t\ttest -L \"$$filepath\" && continue; \\\n\t\t\techo \"$$filepath\"; \\\n\t\tdone; \\\n\telse \\\n\t\tfind . -type f; \\\n\tfi \\\n)\n\n\nBASH_PROFILE_FILES := $(shell echo .bashrc .bash_profile .bash.d/*.sh)\n\n#.PHONY: *\n\nCURRENT_BRANCH := $(shell git rev-parse --abbrev-ref HEAD)\nTRUNK_BRANCH := $(shell git symbolic-ref refs/remotes/origin/HEAD | sed 's|.*/||')\n\nDEFAULT_TITLE := [GD-00] - merge $(CURRENT_BRANCH) to $(TRUNK_BRANCH)\n\ntitle ?= $(DEFAULT_TITLE)\n\n# ===================\ndefine MAKEFILE_USAGE\n\n  Repo specific options:\n\n    make install                builds all script dependencies, installs AWS CLI, GitHub CLI, symlinks all config files to $$HOME and adds sourcing of bash profile\n\n    make link                   symlinks all config files to $$HOME and adds sourcing of bash profile\n    make unlink                 removes all symlinks pointing to this repo's config files and removes the sourcing lines from .bashrc and .bash_profile\n\n    make python-desktop         installs all Python Pip packages for desktop workstation listed in setup/pip-packages-desktop.txt\n    make perl-desktop           installs all Perl CPAN packages for desktop workstation listed in setup/cpan-packages-desktop.txt\n    make ruby-desktop           installs all Ruby Gem packages for desktop workstation listed in setup/gem-packages-desktop.txt\n    make golang-desktop         installs all Golang packages for desktop workstation listed in setup/go-packages-desktop.txt\n    make nodejs-desktop         installs all NodeJS packages for desktop workstation listed in setup/npm-packages-desktop.txt\n\n    make desktop                installs all of the above + many desktop OS packages listed in setup/\n\n    make mac-desktop            all of the above + installs a bunch of major common workstation software packages like Ansible, Terraform, MiniKube, MiniShift, SDKman, Travis CI, CCMenu, Parquet tools etc.\n    make linux-desktop\n\n    make ls-scripts             print list of scripts in this project, ignoring code libraries in lib/ and .bash.d/\n\n    make github-cli             installs GitHub CLI\n    make kubernetes             installs Kubernetes kubectl and kustomize to ~/bin/\n    make terraform              installs Terraform to ~/bin/\n    make vim                    installs Vundle and plugins\n    make tmux                   installs TMUX TPM and plugin for kubernetes context\n    make ccmenu                 installs and (re)configures CCMenu to watch this and all other major HariSekhon GitHub repos\n    make status                 open the Github Status page of all my repos build statuses across all CI platforms\n\n    make aws                    installs AWS CLI tools\n    make azure                  installs Azure CLI\n    make gcp                    installs Google Cloud SDK\n\n    make aws-shell              sets up AWS Cloud Shell: installs core packages and links configs\n                                (maintains itself across future Cloud Shells via .aws_customize_environment hook)\n    make gcp-shell              sets up GCP Cloud Shell: installs core packages and links configs\n                                (maintains itself across future Cloud Shells via .customize_environment hook)\n    make azure-shell            sets up Azure Cloud Shell (limited compared to gcp-shell, doesn't install OS packages since there is no sudo)\nendef\n\n# not including azure here because it requires interactive prompt and hangs automatic testing of make docker-*\n.PHONY: build\nbuild:\n\t@echo ================\n\t@echo Bash Tools Build\n\t@echo ================\n\t@$(MAKE) git-summary\n\t@$(MAKE) init\n\t@$(MAKE) system-packages\n\t@$(MAKE) aws github-cli\n\n.PHONY: init\ninit: git\n\t@echo \"running init:\"\n\tgit submodule update --init --recursive\n\t@echo\n\n.PHONY: install\ninstall: build\n\t@$(MAKE) link\n\t@$(MAKE) aws\n\t@$(MAKE) gcp\n\t@$(MAKE) github-cli\n\t@$(MAKE) pip\n\n.PHONY: uninstall\nuninstall: unlink\n\t@echo \"Not removing any system packages for safety\"\n\n.PHONY: bash\nbash: link\n\t@:\n\n.PHONY: link\nlink:\n\t@setup/shell_link.sh\n\n.PHONY: unlink\nunlink:\n\t@setup/shell_unlink.sh\n\n.PHONY: mac-desktop\nmac-desktop: desktop\n\t@setup/mac_desktop.sh\n\n.PHONY: mac\nmac: mac-desktop\n\t@:\n\n.PHONY: linux-desktop\nlinux-desktop: desktop\n\t@setup/linux_desktop.sh\n\n.PHONY: linux\nlinux: linux-desktop\n\t@:\n\n.PHONY:\nccmenu:\n\t@setup/ccmenu_setup.sh\n\n.PHONY: desktop\ndesktop: install\n\t@if [ -x /sbin/apk ];        then $(MAKE) apk-packages-desktop; fi\n\t@if [ -x /usr/bin/apt-get ]; then $(MAKE) apt-packages-desktop; fi\n\t@if [ -x /usr/bin/yum ];     then $(MAKE) yum-packages-desktop; fi\n\t@if [ `uname` = Darwin ]; then \\\n\t\tif type brew >/dev/null 2>/dev/null; then \\\n\t\t\t$(MAKE) homebrew-packages-desktop; \\\n\t\tfi; \\\n\tfi\n\t@# do these late so that we have the above system packages installed first to take priority and not install from source where we don't need to\n\t@$(MAKE) perl-desktop\n\t@$(MAKE) golang-desktop\n\t@$(MAKE) nodejs-desktop\n\t@$(MAKE) ruby-desktop\n\t@# no packages any more since jgrep is no longer found\n\t@#$(MAKE) ruby-desktop\n\n.PHONY: apk-packages-desktop\napk-packages-desktop: system-packages\n\t@echo \"Alpine desktop not supported at this time\"\n\t@exit 1\n\n.PHONY: apt-packages-desktop\napt-packages-desktop: system-packages\n\tNO_FAIL=1 NO_UPDATE=1 $(BASH_TOOLS)/packages/apt_install_packages.sh setup/deb-packages-desktop.txt\n\n.PHONY: yum-packages-desktop\nyum-packages-desktop: system-packages\n\tNO_FAIL=1 NO_UPDATE=1 $(BASH_TOOLS)/packages/yum_install_packages.sh setup/rpm-packages-desktop.txt\n\n.PHONY: homebrew-packages-desktop\nhomebrew-packages-desktop: system-packages homebrew\n\t@:\n\n.PHONY: brew-packages-desktop\nbrew-packages-desktop: homebrew-packages-desktop\n\t@:\n\n.PHONY: homebrew\nhomebrew: system-packages brew\n\t@:\n\n.PHONY: brew\nbrew:\n\twhich -a brew || install/install_homebrew.sh\n\twhich -a wget || brew install wget\n\tNO_FAIL=1 NO_UPDATE=1 $(BASH_TOOLS)/packages/brew_install_packages_if_absent.sh setup/brew-packages-desktop.txt\n\tNO_FAIL=1 NO_UPDATE=1 CASK=1 $(BASH_TOOLS)/packages/brew_install_packages_if_absent.sh setup/brew-packages-desktop-casks.txt\n\t@# doesn't pass the packages correctly yet\n\t@#NO_FAIL=1 NO_UPDATE=1 TAP=1 $(BASH_TOOLS)/packages/brew_install_packages.sh setup/brew-packages-desktop-taps.txt\n\tNO_FAIL=1 NO_UPDATE=1 TAP=1 $(BASH_TOOLS)/packages/brew_install_packages.sh setup/brew-packages-desktop-taps.txt\n\n.PHONY: perl-desktop\nperl-desktop: system-packages cpan-desktop\n\t@:\n\n.PHONY: cpan-desktop\ncpan-desktop: cpan\n\tNO_FAIL=1 NO_UPDATE=1 $(BASH_TOOLS)/perl/perl_cpanm_install_if_absent.sh setup/cpan-packages-desktop.txt\n\n.PHONY: golang-desktop\ngolang-desktop: system-packages go-desktop\n\t@:\n\n.PHONY: go-desktop\ngo-desktop: system-packages go\n\t@:\n\n.PHONY: go\ngo:\n\tNO_FAIL=1 $(BASH_TOOLS)/packages/golang_install_if_absent.sh setup/go-packages-desktop.txt\n\n.PHONY: ruby-desktop\nruby-desktop: system-packages gem-desktop\n\t@:\n\n.PHONY: gem-desktop\ngem-desktop: gem\n\tNO_FAIL=1 $(BASH_TOOLS)/packages/ruby_gem_install_if_absent.sh setup/gem-packages-desktop.txt\n\n.PHONY: python-desktop\npython-desktop: system-packages pip-desktop\n\n.PHONY: pip\npip-desktop: pip\n\tPIP=$(PIP) ./python/python_pip_install_if_absent.sh setup/pip-packages-desktop.txt\n\tif uname -s | grep -q Darwin; then \\\n\t\tPIP=$(PIP) ./python/python_pip_install_if_absent.sh setup/pip-packages-mac.txt; \\\n\tfi\n\n.PHONY: nodejs-desktop\nnodejs-desktop: system-packages npm-desktop\n\n.PHONY: npm-desktop\nnpm-desktop: npm\n\t$(BASH_TOOLS)/packages/nodejs_npm_install_if_absent.sh $(BASH_TOOLS)/setup/npm-packages-desktop.txt\n\n.PHONY: aws\naws: system-packages python-version\n\t@if ! command -v aws; then install/install_aws_cli.sh; fi\n#    @$(MAKE) codecommit\n#\n#.PHONY: codecommit\n#codecommit:\n\t@# needed for github_mirror_repos_to_aws_codecommit.sh and dependent GitHub Actions workflows\n\t@if uname -s | grep -q Darwin; then \\\n\t\txargs(){ \\\n\t\t\tgxargs \"$$@\"; \\\n\t\t}; \\\n\tfi; \\\n\tgrep '^git-remote-codecommit' requirements.txt | \\\n\tPIP=$(PIP) xargs --no-run-if-empty ./python/python_pip_install_if_absent.sh || :\n\n.PHONY: aws-shell\naws-shell:\n\t@if [ \"${AWS_EXECUTION_ENV:-}\" != \"CloudShell\" ]; then echo \"Not running inside AWS Cloud Shell\"; exit 1; fi\n\t@$(MAKE) system-packages aws link\n\n.PHONY: azure\nazure: system-packages\n\t@install/install_azure_cli.sh\n\n.PHONY: azure-shell\nazure-shell: link\n\t:\n\n.PHONY: gcp\ngcp: system-packages\n\t@./install/install_gcloud_sdk.sh\n\t@./install/install_cloud_sql_proxy.sh\n\n.PHONY: gcp-shell\ngcp-shell:\n\t@if [ -z \"${DEVSHELL_PROJECT_ID:-}\" ]; then echo \"Not running inside Google Cloud Shell\"; exit 1; fi\n\t@$(MAKE) system-packages link\n\n.PHONY: github-cli\ngithub-cli: ~/bin/gh\n\t@:\n\n~/bin/gh:\n\tinstall/install_github_cli.sh\n\n.PHONY:\ndigital-ocean: ~/bin/doctl\n\t@:\n\n~/bin/doctl:\n\tinstall/install_doctl.sh\n\n.PHONY: kubernetes\nkubernetes: kubectl kustomize\n\t@:\n\n.PHONY: k8s\nk8s: kubernetes\n\t@:\n\n.PHONY: kubectl\nkubectl: ~/bin/kubectl\n\t@:\n\n~/bin/kubectl:\n\tinstall/install_kubectl.sh\n\n.PHONY: kustomize\nkustomize: ~/bin/kustomize\n\t@:\n\n~/bin/kustomize:\n\tinstall/install_kustomize.sh\n\n.PHONY: vim\nvim: ~/.vim/bundle/Vundle.vim\n\t@:\n\n~/.vim/bundle/Vundle.vim:\n\tinstall/install_vundle.sh\n\n.PHONY: tmux\ntmux: ~/.tmux/plugins/tpm ~/.tmux/plugins/kube.tmux\n\t@:\n\n~/.tmux/plugins/tpm:\n\tgit clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm\n\n~/.tmux/plugins/kube.tmux:\n\twget -O ~/.tmux/plugins/kube.tmux https://raw.githubusercontent.com/jonmosco/kube-tmux/master/kube.tmux\n\n.PHONY: test\ntest:\n\t./checks/check_all.sh\n\n.PHONY: clean\nclean:\n\t@rm -fv -- setup/terraform.zip\n\n.PHONY: ls-scripts\nls-scripts:\n\t@$(MAKE) ls | grep -v -e 'lib/' -e '\\.bash'\n\n.PHONY: ls-scripts2\nls-scripts2:\n\t@$(MAKE) ls | grep -v -e 'lib/' -e '\\.bash' -e 'setup/'\n\n.PHONY: wcbashrc\nwcbashrc:\n\t@wc $(BASH_PROFILE_FILES)\n\t@printf \"Total Bash Profile files: \"\n\t@ls $(BASH_PROFILE_FILES) | wc -l\n\n.PHONY: wcbash\nwcbash: wcbashrc\n\t@:\n\n.PHONY: wcbashrc2\nwcbashrc2:\n\t@printf \"Total Bash Profile files: \"\n\t@ls $(BASH_PROFILE_FILES) | wc -l\n\t@printf \"Total line count without # comments: \"\n\t@ls $(BASH_PROFILE_FILES) | xargs sed 's/#.*//;/^[[:space:]]*$$/d' | wc -l\n\n.PHONY: wcbash2\nwcbash2: wcbashrc2\n\t@:\n\n.PHONY: pipreqs-mapping\npipreqs-mapping:\n\t#wget -O resources/pipreqs_mapping.txt https://raw.githubusercontent.com/HariSekhon/pipreqs/mysql-python/pipreqs/mapping\n\twget -O resources/pipreqs_mapping.txt https://raw.githubusercontent.com/bndr/pipreqs/master/pipreqs/mapping\n.PHONY: pip-mapping\npip-mapping: pipreqs-mapping\n\t@:\n\n.PHONY: status-page\nstatus-page:\n\t./cicd/generate_status_page.sh; . .bash.d/git.sh; gitu STATUS.md\n\n.PHONY: dialog-install\ndialog-install:\n\tinstall/install_packages.sh dialog\n\n# Raise Pull Requests from the command line like this:\n#\n#\tYou need GitHub CLI installed ('make' installs it for you) and authenticated eg.:\n#\n#\t\tgh auth login\n#\n#\t\t# https://cli.github.com/manual/gh_auth_login\n#\n#\tExample:\n#\n#\t\tmake pr title=\"Hari code to avoid clicking\"\n#\n.PHONY: pr\npr: dialog-install\n\tgit push --set-upstream origin \"$(CURRENT_BRANCH)\"\n\tif [ -z \"$$GITHUB_PULL_REQUEST_TITLE\" ]; then \\\n\t\tif [ \"$(title)\" = \"$(DEFAULT_TITLE)\" ]; then \\\n\t\t\tGITHUB_PULL_REQUEST_TITLE=\"$$(dialog --inputbox \"Pull Request Title:\" 8 40 \"$(DEFAULT_TITLE)\" 3>&1 1>&2 2>&3)\"; \\\n\t\telse \\\n\t\t\tGITHUB_PULL_REQUEST_TITLE=\"$(title)\"; \\\n\t\tfi; \\\n\tfi; \\\n\texport GITHUB_PULL_REQUEST_TITLE; \\\n\tgithub_pull_request_create.sh \\\n\t\t\"$(REPO)\" \\\n\t\t\"$(CURRENT_BRANCH)\" \\\n\t\t\"$(TRUNK_BRANCH)\"\n\n# raise a PR in one command with Auto-Merge enabled - use this for trivial PRs of low / no impact like MkDocs updates\n.PHONY: auto-pr\nauto-pr: update\n\t@# - if GITHUB_PULL_REQUEST_AUTO_MERGE=true then marks the PR for auto-merge once it is approved and passes pre-requisite checks\n\t@# - if GITHUB_PULL_REQUEST_SQUASH=true while GITHUB_PULL_REQUEST_AUTO_MERGE=true then it marks\n\t@#   the PR's auto-merge to be done using a squash commit to avoid any CLI prompt for how to merge it\n\tGITHUB_PULL_REQUEST_AUTO_MERGE=true \\\n\tGITHUB_PULL_REQUEST_SQUASH=true \\\n\t$(MAKE) pr\n\n# Example:\n#\n#\tmake autopr title=\"Documented something\"\n#\n.PHONY: autopr\nautopr: auto-pr\n\t@:\n\n.PHONY: sync\nsync:\n\tsync_configs_to_adjacent_repos.sh\n"
  },
  {
    "path": "Makefile.in",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2013-02-03 10:25:36 +0000 (Sun, 03 Feb 2013)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#  to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nifneq (\"$(wildcard bash-tools)\", \"\")\n\tBASH_TOOLS := bash-tools\nelse\n\tBASH_TOOLS := .\nendif\n\n# would fail bootstrapping on Alpine\n#SHELL := /usr/bin/env bash\n\nexport PATH := $(PATH):/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/opt/homebrew/bin\n\n# Python breaking backwards compatibility as usual - set this to use historic behaviour and treat the user like an adult\n# this way if the install is run for --user instead of in a virtualenv it'll still work\nexport PIP_BREAK_SYSTEM_PACKAGES := 1\n\nDOCKER_IMAGE := harisekhon/github\n\nifneq (\"$(wildcard /.dockerenv)\", \"\")\n\tINSIDE_DOCKER := 1\nelse\n\tINSIDE_DOCKER :=\nendif\n\nCODE_FILES := $(shell \\\n\tif type git >/dev/null 2>&1; then \\\n\t\tgit ls-files | \\\n\t\twhile read filepath; do \\\n\t\t\ttest -f \"$$filepath\" || continue; \\\n\t\t\ttest -d \"$$filepath\" & continue; \\\n\t\t\ttest -L \"$$filepath\" & continue; \\\n\t\t\techo \"$$filepath\"; \\\n\t\tdone; \\\n\tfi \\\n)\n\nCPANM := cpanm\nexport PIP := pip3\nexport PYTHON := python3\n\nFATPACKS_DIR := fatpacks\n\nSUDO := sudo\nSUDO_PIP := sudo -H\nSUDO_PERL := sudo\n\nPYTHON_VIRTUALENV :=\n\nifdef PERLBREW_PERL\n\t# can't put this here, nor @commented, otherwise gets error - \"commands commence before first target.  Stop.\"\n\t#echo \"Perlbrew environment detected, not calling sudo\"\n\tSUDO_PERL =\nelse\n\tPERLBREW_PERL :=\nendif\n\n# Travis has custom python install earlier in $PATH even in Perl builds so need to install PyPI modules locally to non-system python otherwise they're not found by programs.\n# Perms not set correctly on custom python install in Travis perl build so workaround is done to chown to travis user in .travis.yml\n# Better than modifying $PATH to put /usr/bin first which is likely to affect many other things including potentially not finding the perlbrew installation first\n# Looks like Perl travis builds are now using system Python - do not use TRAVIS env\nifdef VIRTUAL_ENV\n\t#echo \"Virtual Env / Conda detected, not calling sudo\"\n\tSUDO_PIP :=\n\tPYTHON_VIRTUALENV := 1\nendif\nifdef CONDA_DEFAULT_ENV\n\tSUDO_PIP :=\n\tPYTHON_VIRTUALENV := 1\nendif\n\n# must come after to reset SUDO_PERL/SUDO_PIP to blank if root\n# EUID / UID not exported in Make\n# USER not populated in Docker\nifeq '$(shell id -u)' '0'\n\t#echo \"root UID detected, not calling sudo\"\n\tSUDO :=\n\tSUDO_PERL :=\n\tSUDO_PIP :=\nendif\n\n# placeholders to silence check_makefile.sh warnings - should be set in client Makefiles after sourcing\nifndef REPO\n\tREPO := NOTSET\nendif\nifndef ARGS\n\tARGS := NOTSET\nendif\nifndef CONF_FILES\n\tCONF_FILES := NOTSET\nendif\n\ndefine MAKEFILE_USAGE_COMMON\n\n Usage:\n\n  Common Options:\n\n    make help                   show this message\n    make build                  installs all dependencies - OS packages and any language libraries via native tools eg. pip, cpanm, gem, go etc that are not available via OS packages\n    make build-retry            retries 'make build' x 3 until success to try to mitigate temporary upstream repo failures triggering false alerts in CI systems\n    make ci                     prints env, then runs 'build-retry' for more resilient CI builds with debugging\n    make printenv               prints environment variables, CPU cores, OS release, $$PWD, Git branch, hashref etc. Useful for CI debugging\n    make system-packages        installs OS packages only (detects OS via whichever package manager is available)\n    make test                   run tests\n    make clean                  removes compiled / generated files, downloaded tarballs, temporary files etc.\n\n    make submodules             initialize and update submodules to the right release (done automatically by build / system-packages)\n    make init                   same as above, often useful to do in CI systems to get access to additional submodule provided targets such as 'make ci'\n\n    make cpan                   install any modules listed in any cpan-requirements.txt files if not already installed\n    make gem                    install any modules listed in any gem-requirements.txt files if not already installed\n    make npm                    install any modules listed in any npm-requirements.txt files if not already installed\n    make pip                    install any modules listed in any requirements.txt files if not already installed\n\n    make python-compile         compile any python files found in the current directory and 1 level of subdirectory\n    make pycompile\n\n    make github                 open browser at github project\n    make readme                 open browser at github's README\n    make github-url             print github url and copy to clipboard\n    make status                 open browser at Github CI Builds overview Status page for all projects\n\n    make ls                     print list of code files in project\n    make wc                     show counts of files and lines\n\nendef\n    #make ${VENV}                make a virtualenv in the base directory (see VENV)\n    #make pip-install            install python packages in requirements.txt\n    #make git-config             set local git configuration\nexport MAKEFILE_USAGE_COMMON\nexport MAKEFILE_USAGE\n\n# doesn't seem to work\n#.DEFAULT: build\n#\t@echo running default\n#\t$(MAKE) build\n\n# won't be run the first time - will default to first target which will only then initialize submodules\n.PHONY: default\ndefault: git printenv\n\t@$(MAKE) main\n\n.PHONY: printenv\nprintenv: git\n\t@ printf \"CPU Cores: \"; nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null; :\n\t@ # $$USER not always set in sh\n\t@ # printf \"Git hashref: \"; git rev-parse HEAD\n\t@ # printf \"Git hashref: \"; git log --pretty=format:'%H' -n 1\n\t@ # printf \"Git branch:  \"; git branch --show-current # doesn't work on Alpine\n\t@ # printf \"Git branch:  \"; git show-branch --current # prints too many branches\n\t@ # sort --ignore-case switch not available on Alpine, must use sort -f which is available on both Mac and all Linux distros\n\t@ . $(BASH_TOOLS)/lib/ci.sh || : ; \\\n\t\tif is_CI || test -f /.dockerenv; then \\\n\t\t\techo; \\\n\t\t\techo \"USER = `whoami`\"; \\\n\t\t\techo \"PWD = $$PWD\"; \\\n\t\t\techo; \\\n\t\t\tprintf \"Git branch: \"; git rev-parse --abbrev-ref HEAD; \\\n\t\t\tprintf \"Git commit: \"; git log --pretty=format:\"%ai  %cn  %H  %s\" -n 1; echo; \\\n\t\t\techo; \\\n\t\t\tif [ -f /.dockerenv  ]; then \\\n\t\t\t\techo \"Running inside Docker:\"; \\\n\t\t\t\tls -l /.dockerenv 2>/dev/null; \\\n\t\t\tfi; \\\n\t\t\techo; \\\n\t\t\techo \"OS RELEASE:\"; \\\n\t\t\techo; \\\n\t\t\tuname -a || : ; \\\n\t\t\techo; \\\n\t\t\tcat /etc/*release || : ; \\\n\t\t\techo; \\\n\t\t\tunset MAKEFILE_USAGE; \\\n\t\t\tunset MAKEFILE_USAGE_COMMON; \\\n\t\t\tunset TERMCAP; \\\n\t\t\techo; \\\n\t\t\techo \"CI ENVIRONMENT:\"; \\\n\t\t\techo; \\\n\t\t\tenv | grep -vi -e PASS -e TOKEN -e KEY -e SECRET | sort -f; \\\n\t\t\techo; \\\n\t\t\techo; \\\n\t\t\tif which java 2>/dev/null; then \\\n\t\t\t\twhich java; \\\n\t\t\t\tjava -version; \\\n\t\t\t\techo; \\\n\t\t\tfi; \\\n\t\t\techo \"PATH:\"; echo \"$$PATH\" | tr ':' '\\n'; \\\n\t\t\techo; \\\n\t\telse \\\n\t\t\tenv | grep -E 'BUILD|PIPELINE|JOB|STAGE|\\<CI_|^CI=' | grep -v TOKEN || : ; \\\n\t\tfi | cat # stops git commands from entering pager\n\n.PHONY: git-summary\ngit-summary: init\n\t@echo\n\t@echo \"Git summary:\"\n\t@$(BASH_TOOLS)/git/git_summary_line.sh\n\t@echo\n\n.PHONY: ci\nci: printenv\n\t$(MAKE) build-retry\n\n.PHONY: build-retry\nbuild-retry: git\n\t$(BASH_TOOLS)/bin/retry.sh $(MAKE) build\n\n# won't be run the first time - will default to first target which will only then initialize submodules\n.PHONY: main\nmain: printenv\n\t@$(MAKE) build\n\n.PHONY: help\nhelp:\n\t@# this doesn't work because the macro insertion of a multiline literal breaks the line-based make format so we have to export to an env var instead of using natively\n\t@# even the unescaped macro literal in a commented breaks make\n\t@echo \"$$MAKEFILE_USAGE_COMMON $$MAKEFILE_USAGE\" # | less -RFXig # don't use less it will make target tests hang\n\t@echo\n\t@echo \"Now exiting usage help with status code 3 to explicitly prevent silent build failures from stray 'help' arguments\"\n\t@exit 3\n\n.PHONY: usage\nusage: help\n\t@#:\n\n# clever but breaks 'make -n <target>' tests because the exit 3 doesn't actually get called and leads make to think there is a matching target, which then fail to execute\n# catchall - any unrecognized target will print usage\n#%::\n#\t@# don't use less, it will make target tests hang\n#\t@echo Unrecognized option $@; \\\n#\techo; \\\n#\t$(MAKE) usage;\n\n.PHONY: quick\nquick:\n\tQUICK=1 $(MAKE) build\n\n.PHONY: git\ngit:\n\t@# not using install_packages_if_absent.sh as we don't need a package on Mac, it comes with XCode\n\ttype git 2>/dev/null || $(BASH_TOOLS)/packages/install_packages.sh git\n\n.PHONY: submodules\nsubmodules: git\n\t@echo \"checking out any git submodules:\"\n\tgit submodule update --init --recursive\n\t@echo\n\n.PHONY: git-clean\ngit-clean: git\n\t@git clean -n -d\n\t@printf \"\\n\\n%s\" \"If you're happy with this list, run:\"\n\t@printf \"\\n\\n%s\\n\\n\" \"git clean -f -d\"\n\n.PHONY: gitignore\ngitignore:\n\t$(BASH_TOOLS)/git/update_gitignore.io.sh\n\n.PHONY: btest\nbtest: bash-test\n\t@:\n\n.PHONY: bash-test\nbash-test:\n\t$(BASH_TOOLS)/checks/check_all.sh\n\n.PHONY: test\n#test: precommit\ntest: bash-test\n\t@:\n\nprecommit: pre-commit\n\t@:\n\npre-commit:\n\tpre-commit run --all-files\n\n.PHONY: push\npush: test\n\tgit push\n\n.PHONY: system-packages\nsystem-packages: submodules\n\tif [ -x /sbin/apk ];        then $(MAKE) apk-packages; fi\n\tif [ -x /usr/bin/apt-get ]; then $(MAKE) apt-packages; fi\n\t@# /usr/bin/yum is a symlink to dnf-3 on newer RHEL systems, so fails -x /usr/bin/yum\n\tif [ -e /usr/bin/yum ];     then $(MAKE) yum-packages; fi\n\t@# /usr/local/bin/brew    on older macOS\n\t@# /opt/homebrew/bin/brew on newer macOS\n\tif which -a brew && [ `uname` = Darwin ]; then $(MAKE) homebrew-packages; fi\n\n.PHONY: system-packages-perl\nsystem-packages-perl: system-packages\n\tif [ -x /sbin/apk ];        then $(MAKE) apk-packages-perl; fi\n\tif [ -x /usr/bin/apt-get ]; then $(MAKE) apt-packages-perl; fi\n\t@# /usr/bin/yum is a symlink to dnf-3 on newer RHEL systems, so fails -x /usr/bin/yum\n\tif [ -e /usr/bin/yum ];     then $(MAKE) yum-packages-perl; fi\n\n.PHONY: system-packages-python\nsystem-packages-python: system-packages\n\tif [ -x /sbin/apk ];        then $(MAKE) apk-packages-python; fi\n\tif [ -x /usr/bin/apt-get ]; then $(MAKE) apt-packages-python; fi\n\t@# /usr/bin/yum is a symlink to dnf-3 on newer RHEL systems, so fails -x /usr/bin/yum\n\tif [ -e /usr/bin/yum ];     then $(MAKE) yum-packages-python; fi\n\n.PHONY: apk-packages\napk-packages:\n\t# not portable in Alpine sh\n\t#for x in apk-packages{,-perl,-python}{,-dev}.txt; do \\\n\n\tfor x in apk-packages.txt apk-packages-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\t#for x in apk-packages-{optional,cpan,pip}.txt; do \\\n\n\tfor x in apk-packages-optional.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\n.PHONY: apk-packages-perl\napk-packages-perl:\n\tfor x in apk-packages-perl.txt apk-packages-perl-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\t#for x in apk-packages-{optional,cpan,pip}.txt; do \\\n\n\t# don't put comments inside the for loop, breaks syntax expecting 'done'\n\t# no point installing system cpan packages if using perlbrew as they won't be found inside perlbrew\n\tfor x in apk-packages-cpan.txt; do \\\n\t\tif [ -z \"$(PERLBREW_PERL)\" ]; then \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tfi; \\\n\tdone | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\n.PHONY: apk-packages-python\napk-packages-python:\n\tfor x in apk-packages-python.txt apk-packages-python-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\t# no point installing system pip packages when they won't be found in virtualenv and will need to be pip installed anyway\n\tfor x in apk-packages-pip.txt; do \\\n\t\tif [ -z \"$(PYTHON_VIRTUALENV)\" ]; then \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tfi; \\\n\tdone | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apk_install_packages.sh\"\n\n.PHONY: apt-packages\napt-packages:\n\t#for x in deb-packages{,-perl,-python}{,-dev}.txt; do \\\n\n\tfor x in deb-packages.txt deb-packages-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\t#for x in deb-packages-{optional,cpan,pip}.txt; do \\\n\n\tfor x in deb-packages-optional.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\n.PHONY: apt-packages-perl\napt-packages-perl:\n\tfor x in deb-packages-perl.txt deb-packages-perl-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\tfor x in deb-packages-cpan.txt; do \\\n\t\tif [ -z \"$(PERLBREW_PERL)\" ] && \\\n\t\t   [ -z \"$(GOOGLE_CLOUD_SHELL)\" ]; then \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tfi; \\\n\tdone | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\n.PHONY: apt-packages-python\napt-packages-python:\n\tfor x in deb-packages-python.txt deb-packages-python-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\tfor x in deb-packages-pip.txt; do \\\n\t\tif [ -z \"$(PYTHON_VIRTUALENV)\" ] && \\\n\t\t   [ -z \"$(GOOGLE_CLOUD_SHELL)\" ]; then \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tfi; \\\n\tdone  | NO_FAIL=1 NO_UPDATE=1 xargs \"$(BASH_TOOLS)/packages/apt_install_packages.sh\"\n\n.PHONY: yum-packages\nyum-packages:\n\t# needed for Fedora to have find and xargs to use below\n\t\"$(BASH_TOOLS)/packages/yum_install_packages.sh\" findutils\n\n\t# if on Amazon Linux 2 install epel this way\n\tif type -P amazon-linux-extras; then \\\n\t\t$(SUDO) amazon-linux-extras install epel -y; \\\n\tfi\n\t$(BASH_TOOLS)/install/install_epel_repo.sh\n\n\t# installing packages individually to catch package install failure, otherwise yum succeeds even if it misses a package\n\tfor x in rpm-packages.txt rpm-packages-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"\n\tfor x in rpm-packages-optional.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"\n\n.PHONY: yum-packages-perl\nyum-packages-perl:\n\t# installing packages individually to catch package install failure, otherwise yum succeeds even if it misses a package\n\tfor x in rpm-packages-perl.txt rpm-packages-perl-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"\n\tfor x in rpm-packages-cpan.txt; do \\\n\t\tif [ -z \"$(PERLBREW_PERL)\" ]; then \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tfi; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"\n\n.PHONY: yum-packages-python\nyum-packages-python:\n\t# installing packages individually to catch package install failure, otherwise yum succeeds even if it misses a package\n\tfor x in rpm-packages-python.txt rpm-packages-python-dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"\n\t. \"$(BASH_TOOLS)/lib/python.sh\"; \\\n\tset +o pipefail || : ; \\\n\tif ! inside_virtualenv; then \\\n\t\tfor x in rpm-packages-pip.txt; do \\\n\t\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\t\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/yum_install_packages.sh\"; \\\n\tfi\n\n.PHONY: homebrew-packages\nhomebrew-packages:\n\t# Fails if any of the packages are already installed, ignore and continue - if it's a problem the latest build steps will fail with missing headers\n\tfor x in brew-packages.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/brew_install_packages.sh\"\n\t@# fix for OpenSSL 1.0 -> 1.1 library linkage breaking python -c 'import hashlib', which break pips, eg:\n\t@# https://stackoverflow.com/questions/20399331/error-importing-hashlib-with-python-2-7-but-not-with-2-6\n\t$(BASH_TOOLS)/setup/brew_fix_openssl_dependencies.sh\n\n.PHONY: system-packages-remove\nsystem-packages-remove:\n\tif [ -x /sbin/apk ];        then $(MAKE) apk-packages-remove; fi\n\tif [ -x /usr/bin/apt-get ]; then $(MAKE) apt-packages-remove; fi\n\tif [ -x /usr/bin/yum ];     then $(MAKE) yum-packages-remove; fi\n\n.PHONY: apk-packages-remove\napk-packages-remove:\n\tfor x in apk-packages-{,perl-,python-}dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/apk_remove_packages.sh\"\n\t$(SUDO) rm -fr -- /var/cache/apk/*\n\n.PHONY: apt-packages-remove\napt-packages-remove:\n\tfor x in deb-packages-{,perl-,python-}dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/apt_remove_packages.sh\"\n\n.PHONY: yum-packages-remove\nyum-packages-remove:\n\tfor x in rpm-packages-{,perl-,python-}dev.txt; do \\\n\t\tfind . -maxdepth 3 -path \"*/setup/$$x\"; \\\n\tdone | NO_FAIL=1 xargs \"$(BASH_TOOLS)/packages/yum_remove_packages.sh\"\n\n.PHONY: cpan\ncpan::\n\tfind . -maxdepth 3 -path '*/setup/cpan-requirements*.txt' | grep -v cpan-requirements-optional.txt | xargs --no-run-if-empty \"$(BASH_TOOLS)/perl/perl_cpanm_install_if_absent.sh\"\n\t@$(MAKE) cpan-optional\n\n.PHONY: cpan-optional\ncpan-optional::\n\tfind . -maxdepth 3 -path '*/setup/cpan-requirements-optional.txt' | NO_FAIL=1 xargs --no-run-if-empty \"$(BASH_TOOLS)/perl/perl_cpanm_install_if_absent.sh\"\n\n.PHONY: gems\ngems:: gem\n\t@:\n\n.PHONY: gem\ngem::\n\tfind . -maxdepth 3 -path '*/setup/gem-packages.txt' | xargs --no-run-if-empty \"$(BASH_TOOLS)/packages/ruby_gem_install_if_absent.sh\"\n\t@$(MAKE) gem-optional\n\n.PHONY: gem-optional\ngem-optional::\n\tfind . -maxdepth 3 -path '*/setup/gem-packages-optional.txt' | NO_FAIL=1 PIP=\"$(PIP)\" xargs --no-run-if-empty \"$(BASH_TOOLS)/packages/ruby_gem_install_if_absent.sh\"\n\n.PHONY: npm\nnpm::\n\tfind . -maxdepth 3 -path '*/setup/npm-requirements.txt' -o -path '*/setup/npm-packages.txt' | xargs --no-run-if-empty \"$(BASH_TOOLS)/packages/nodejs_npm_install_if_absent.sh\"\n\t@$(MAKE) npm-optional\n\n.PHONY: npm-optional\nnpm-optional::\n\tfind . -maxdepth 3 -path '*/setup/npm-requirements-optional.txt' | NO_FAIL=1 xargs --no-run-if-empty \"$(BASH_TOOLS)/packages/nodejs_npm_install_if_absent.sh\"\n\n.PHONY: pip\npip::\n\tfind . -maxdepth 3 -path '*/requirements.txt' | PIP=\"$(PIP)\" xargs --no-run-if-empty \"$(BASH_TOOLS)/python/python_pip_install_if_absent.sh\"\n\t@$(MAKE) pip-optional\n\n.PHONY: pip-optional\npip-optional::\n\tfind . -maxdepth 3 -path '*/requirements-optional.txt' | NO_FAIL=1 PIP=\"$(PIP)\" xargs --no-run-if-empty \"$(BASH_TOOLS)/python/python_pip_install_if_absent.sh\"\n\n.PHONY: pip-user\npip-user::\n\tPYTHON_USER_INSTALL=1 $(MAKE) pip\n\n.PHONY: fatpacks\nfatpacks:\n\t$(BASH_TOOLS)/perl/perl_generate_fatpacks.sh *.pl\n\t@echo\n\t@if [ -d lib/resources ]; then \\\n\t\tcp -av -- lib/resources fatpacks/; \\\n\tfi\n\t@if $(MAKE) -n fatpacks-local >/dev/null 2>&1; then \\\n\t\techo; \\\n\t\techo \"fatpacks-local target detected, running:\"; \\\n\t\t$(MAKE) fatpacks-local; \\\n\tfi\n\t@echo\n\t@if [ -n \"`ls \"$(FATPACKS_DIR)\"`\" ]; then \\\n\t\ttar czvf fatpacks.tar.gz \"$(FATPACKS_DIR)\"; \\\n\t\techo; \\\n\t\techo \"Generated fatpacks.tar.gz containing $(FATPACKS_DIR)/ directory of perl scripts with all dependencies bundled\"; \\\n\tfi\n\n.PHONY: fatpack\nfatpack: fatpacks\n\t@:\n\n.PHONY: python-compile\npython-compile:\n\t$(BASH_TOOLS)/python/python_compile.sh\n\n.PHONY: pycompile\npycompile: python-compile\n\t@:\n\n.PHONY: python-version\npython-version:\n\t$(BASH_TOOLS)/setup/which_python_installed.sh\n\n.PHONY: golang-version\ngolang-version:\n\t@echo && \\\n\twhich go && \\\n\tls -l `which go` && \\\n\techo && \\\n\tgo version || : ; \\\n\techo\n\n.PHONY: go-version\ngo-version: golang-version\n\t@:\n\n.PHONY: golang-clean\ngolang-clean:\n\t@$(BASH_TOOLS)/packages/golang_rm_binaries.sh\n\n.PHONY: go-clean\ngo-clean: golang-clean\n\t@:\n\n# =======================\n# Nice tricks for pure Python projects\n# - borrowed from https://gist.github.com/bsmith89/c6811893c1cbd2a72cc1d144a197bef2#file-makefile\n\n#VENV = .venv\n#export VIRTUAL_ENV := $(abspath ${VENV})\n\n# putting the venv/bin at the start of the path means that the venv python will be called\n# and the venv libraries used automatically, so no need to 'source .venv/bin/activate' first\n# although it misses the hash flush 'hash -r' that the venv activate script does\n#export PATH := ${VIRTUAL_ENV}/bin:${PATH}\n\n#${VENV}:\n#    python3 -m venv \"$@\"\n\n#pip-install: requirements.txt | ${VENV}\n#    pip install --upgrade -r requirements.txt\n# =======================\n\n.PHONY: sonar\nsonar:\n\tsonar-scanner\n\n.PHONY: update\nupdate: update2\n\t@# putting this here instead of inline dep because otherwise check_makefile.sh will fail the target as build target doesn't exist in this Makefile.in\n\t@$(MAKE) build\n\n.PHONY: update2\nupdate2: update-no-recompile\n\t@:\n\n.PHONY: update-no-recompile\nupdate-no-recompile:\n\tgit pull --no-edit\n\t$(MAKE) submodules\n\n.PHONY: update-submodules\nupdate-submodules:\n\tgit submodule update --init --remote\n.PHONY: updatem\nupdatem: update-submodules\n\t@:\n\n.PHONY: docker-run\ndocker-run:\n\tdocker run -ti --rm ${DOCKER_IMAGE} ${ARGS}\n\n.PHONY: run\nrun: docker-run\n\t@:\n\n.PHONY: concourse\nconcourse:\n\t$(BASH_TOOLS)/cicd/concourse.sh\n\n.PHONY: fly\nfly: concourse\n\t@:\n\n.PHONY: docker-mount\ndocker-mount:\n\t# --privileged=true is needed to be able to:\n\t# mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk\n\tdocker run -ti --rm --privileged=true -v $$PWD:/code ${DOCKER_IMAGE} bash -c \"cd /code; exec bash\"\n\n.PHONY: docker-mount-alpine\ndocker-mount-alpine:\n\t# --privileged=true is needed to be able to:\n\t# mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk\n\tdocker run -ti --rm --privileged=true -v $$PWD:/code ${DOCKER_IMAGE}:alpine bash -c \"cd /code; exec bash\"\n\n.PHONY: docker-mount-debian\ndocker-mount-debian:\n\t# --privileged=true is needed to be able to:\n\t# mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk\n\tdocker run -ti --rm --privileged=true -v $$PWD:/code ${DOCKER_IMAGE}:debian bash -c \"cd /code; exec bash\"\n\n.PHONY: docker-mount-centos\ndocker-mount-centos:\n\t# --privileged=true is needed to be able to:\n\t# mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk\n\tdocker run -ti --rm --privileged=true -v $$PWD:/code ${DOCKER_IMAGE}:centos bash -c \"cd /code; exec bash\"\n\n.PHONY: docker-mount-ubuntu\ndocker-mount-ubuntu:\n\t# --privileged=true is needed to be able to:\n\t# mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk\n\tdocker run -ti --rm --privileged=true -v $$PWD:/code ${DOCKER_IMAGE}:ubuntu bash -c \"cd /code; exec bash\"\n\n.PHONY: mount\nmount: docker-mount\n\t@:\n\n.PHONY: mount-alpine\nmount-alpine: docker-mount-alpine\n\t@:\n\n.PHONY: mount-debian\nmount-debian: docker-mount-debian\n\t@:\n\n.PHONY: mount-centos\nmount-centos: docker-mount-centos\n\t@:\n\n.PHONY: mount-ubuntu\nmount-ubuntu: docker-mount-ubuntu\n\t@:\n\n# checks dockerhub build status for this repo - needs check_dockerhub_repo_build_status.py from Advanced Nagios Plugins Collection to be in $PATH\n.PHONY: dockerhub-status\ndockerhub-status:\n\tcheck_dockerhub_repo_build_status.py -r \"$(DOCKER_IMAGE)\"\n\n# For quick testing only - for actual Dockerfile builds see https://hub.docker.com/u/harisekhon and Dockerfiles source repo https://github.com/HariSekhon/Dockerfiles\n.PHONY: docker-alpine\ndocker-alpine:\n\t$(BASH_TOOLS)/docker/docker_mount_build_exec.sh alpine\n\n.PHONY: docker-debian\ndocker-debian:\n\t$(BASH_TOOLS)/docker/docker_mount_build_exec.sh debian\n\n.PHONY: docker-centos\ndocker-centos:\n\t$(BASH_TOOLS)/docker/docker_mount_build_exec.sh centos\n\n.PHONY: docker-fedora\ndocker-fedora:\n\t$(BASH_TOOLS)/docker/docker_mount_build_exec.sh fedora\n\n.PHONY: docker-ubuntu\ndocker-ubuntu:\n\t$(BASH_TOOLS)/docker/docker_mount_build_exec.sh ubuntu\n\n.PHONY: travis\ntravis:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://travis-ci.org/$(REPO)\"\n\n.PHONY: travis-log\ntravis-log:\n\ttravis_last_log.py --failed $(REPO)\n\n.PHONY: travis-debug\ntravis-debug:\n\ttravis_debug_session.py $(REPO)\n\n.PHONY: browse\nbrowse: github\n\t@:\n\n.PHONY: commitcount\ncommitcount:\n\t@# interestingly, even on 10,000 commit repos, there are no duplicate short hashes shown from:\n\t@# git log --all --pretty=format:\"%h\" | sort | uniq -d\n\t@git log --all --pretty=format:\"%h\" | wc -l\n\n.PHONY: github\ngithub:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://github.com/$(REPO)\"\n\n.PHONY: github-url\ngithub-url:\n\t@. $(BASH_TOOLS)/.bash.d/functions.sh; echo \"https://github.com/$(REPO)\" | tee /dev/stderr | tr -d '\\n' | paste_clipboard\n\n.PHONY: gitlab\ngitlab:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://gitlab.com/$(REPO)\"\n\n.PHONY: gitlab-url\ngitlab-url:\n\t@. $(BASH_TOOLS)/.bash.d/functions.sh; echo \"https://gitlab.com/$(REPO)\" | tee /dev/stderr | tr -d '\\n' | paste_clipboard\n\n.PHONY: bitbucket\nbitbucket:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://bitbucket.org/$(REPO)/src/master/\"\n\n.PHONY: bitbucket-url\nbitbucket-url:\n\t@. $(BASH_TOOLS)/.bash.d/functions.sh; echo \"https://bitbucket.org/$(REPO)/src/master/\" | tee /dev/stderr | tr -d '\\n' | paste_clipboard\n\n.PHONY: status\nstatus:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://bitbucket.org/HariSekhon/DevOps-Bash-tools/src/master/STATUS.md\"\n\n.PHONY: readme\nreadme:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://github.com/$(REPO)/blob/master/README.md\"\n\n.PHONY: issues\nissues:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://github.com/$(REPO)/issues\"\n\n.PHONY: github\ndockerhub:\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; browser \"https://hub.docker.com/u/harisekhon\"\n\n.PHONY: dockerhub-url\ndockerhub-url:\n\t@. $(BASH_TOOLS)/.bash.d/functions.sh; echo \"https://hub.docker.com/u/harisekhon\" | tee /dev/stderr | tr -d '\\n' | paste_clipboard\n\n.PHONY: startrack\nstartrack:\n\t@echo \"Don't run this too much, you will hit an API limit against your IP\"\n\t@. $(BASH_TOOLS)/.bash.d/network.sh; \\\n\tbrowser \"https://seladb.github.io/StarTrack-js/?\\\n\tu=$$(sed 's/\\/.*//' <<< \"$(REPO)\")\\\n\t&r=$$(sed 's/.*\\///' <<< \"$(REPO)\")\"\n\n.PHONY: star\nstar: startrack\n\t@:\n\n.PHONY: allstars\nallstars:\n\t@echo \"Takes a while, don't run this all the time or you will hit an API limit against your IP\"\n\t@REPOS=\"Nagios-Plugins Dockerfiles DevOps-Python-tools DevOps-Perl-tools DevOps-Bash-Tools Nagios-Plugin-Kafka HAProxy-configs\"; \\\n\t. $(BASH_TOOLS)/.bash.d/network.sh; \\\n\tbrowser \"https://seladb.github.io/StarTrack-js/#/preload?\\\n\t$$(\\\n\t\tfor repo in $$REPOS; do \\\n\t\t\tprintf \"%s\" \"&r=HariSekhon,$$repo\"; \\\n\t\tdone | \\\n\t\tsed 's/\\&//'\\\n\t)\"\n\n.PHONY: ls\nls:\n\t@echo $(CODE_FILES) | tr ' ' '\\n' | sort\n\n.PHONY: wc\nwc:\n\tif [ -x wc.sh ]; then ./wc.sh; exit 1; fi\n\t@# CODE_FILES := definitions in Makefiles must not be quoted or will get wc error 'open: File name too long'\n\t@wc -l $(CODE_FILES)\n\t@printf 'Total Lines:\\t\\t\\t'\n\t@cat $(CODE_FILES) | wc -l | sed 's/[[:space:]]//g'\n\t@printf 'Total Lines without # comments:\\t'\n\t@sed 's/#.*//;/^[[:space:]]*$$/d' $(CODE_FILES) | wc -l | sed 's/[[:space:]]//g'\n\t@printf 'Total Files:\\t\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | wc -l | sed 's/[[:space:]]//g'\n\t@printf 'of which not the following:\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -Ev -e '\\.bash' \\\n\t\t\t\t\t\t\t\t\t\t\t\t-e lib/ \\\n\t\t\t\t\t\t\t\t\t\t\t\t-e install/ \\\n\t\t\t\t\t\t\t\t\t\t\t\t-e setup/ \\\n\t\t\t\t\t\t\t\t\t\t\t\t-e 'tests*/' \\\n\t\t\t\t\t\t\t\t\t\t\t\t-e vagrant/ \\\n\t\t\t\t\t\t\t\t\t\t\t\t| wc -l | sed 's/[[:space:]]//g'\n\t@printf 'of which .bash*:\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -c '\\.bash'\n\t@printf 'of which lib/:\\t\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -c lib/\n\t@printf 'of which install/:\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -c install/\n\t@printf 'of which setup/:\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -c setup/\n\t@printf 'of which test(s)/:\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -Ec 'tests*/'\n\t@printf 'of which vagrant/:\\t\\t'\n\t@tr ' ' '\\n' <<< \"$(CODE_FILES)\" | grep -c vagrant/\n\nmdl:\n\t@echo \"Checking Markdown for issues\"\n\t@echo\n\t@if .mdl.rb; then \\\n\t\tmdl -s .mdl.rb *.md; \\\n\telse \\\n\t\tmdl *.md; \\\n\tfi\n\n# finds .swp, would need to port out code lists\n#.PHONY: wcall\n#wcall:\n#\tfind . -type f --not -path '*.git*' -exec cat {} \\; | wc -l\n#\n#.PHONY: wcall\n#wcall:\n#\tfind . -type f -not -path '*.git*' -exec sed 's/#.*//;/^[[:space:]]*$$/d' {} \\; | wc -l\n\n.PHONY: repos\nrepos:\n\t@if ! grep -q \"OTHER_REPOS_START\" README.md || \\\n\t\t! grep -q \"OTHER_REPOS_END\"   README.md || \\\n\t\t! grep -q \"More Core Repos\"   README.md; then \\\n\t\techo \"Adding More Core Repos section to README.md\"; \\\n\t\tprintf '\\n## More Core Repos\\n\\n<!-- OTHER_REPOS_START -->\\n\\n<!-- OTHER_REPOS_END -->\\n' >> README.md; \\\n\tfi; \\\n\tmarkdown_replace_repos.sh\n\nmac-exclude-backups:\n\t@for path in $(MAC_TIME_MACHINE_BACKUP_EXCLUDE_PATHS); do \\\n\t\tif [[ \"$${path:0:1}\" != / ]]; then \\\n\t\t\tpath=\"$$PWD/$$path\"; \\\n\t\tfi; \\\n\t\techo \"Excluding: $$path\"; \\\n\t\tsudo tmutil addexclusion -p \"$$path\"; \\\n\tdone\n\nnobackup:\n\t@if uname | grep -q Darwin; then \\\n\t\t$(MAKE) mac-exclude-backups; \\\n\tfi\n"
  },
  {
    "path": "README.md",
    "content": "# Hari Sekhon - DevOps Bash Tools\n\n[![GitHub stars](https://img.shields.io/github/stars/harisekhon/devops-bash-tools?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/harisekhon/devops-bash-tools?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/network)\n[![LineCount](https://sloc.xyz/github/HariSekhon/DevOps-Bash-tools/?badge-bg-color=2081C2)](https://github.com/boyter/scc/)\n[![Cocomo](https://sloc.xyz/github/HariSekhon/DevOps-Bash-tools/?badge-bg-color=2081C2&category=cocomo)](https://github.com/boyter/scc/)\n[![License](https://img.shields.io/badge/license-MIT-green)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/LICENSE)\n[![My LinkedIn](https://img.shields.io/badge/LinkedIn%20Profile-HariSekhon-blue?logo=data:image/svg%2bxml;base64,PHN2ZyByb2xlPSJpbWciIGZpbGw9IiNmZmZmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48dGl0bGU+TGlua2VkSW48L3RpdGxlPjxwYXRoIGQ9Ik0yMC40NDcgMjAuNDUyaC0zLjU1NHYtNS41NjljMC0xLjMyOC0uMDI3LTMuMDM3LTEuODUyLTMuMDM3LTEuODUzIDAtMi4xMzYgMS40NDUtMi4xMzYgMi45Mzl2NS42NjdIOS4zNTFWOWgzLjQxNHYxLjU2MWguMDQ2Yy40NzctLjkgMS42MzctMS44NSAzLjM3LTEuODUgMy42MDEgMCA0LjI2NyAyLjM3IDQuMjY3IDUuNDU1djYuMjg2ek01LjMzNyA3LjQzM2MtMS4xNDQgMC0yLjA2My0uOTI2LTIuMDYzLTIuMDY1IDAtMS4xMzguOTItMi4wNjMgMi4wNjMtMi4wNjMgMS4xNCAwIDIuMDY0LjkyNSAyLjA2NCAyLjA2MyAwIDEuMTM5LS45MjUgMi4wNjUtMi4wNjQgMi4wNjV6bTEuNzgyIDEzLjAxOUgzLjU1NVY5aDMuNTY0djExLjQ1MnpNMjIuMjI1IDBIMS43NzFDLjc5MiAwIDAgLjc3NCAwIDEuNzI5djIwLjU0MkMwIDIzLjIyNy43OTIgMjQgMS43NzEgMjRoMjAuNDUxQzIzLjIgMjQgMjQgMjMuMjI3IDI0IDIyLjI3MVYxLjcyOUMyNCAuNzc0IDIzLjIgMCAyMi4yMjIgMGguMDAzeiIvPjwvc3ZnPgo=)](https://www.linkedin.com/in/HariSekhon/)\n[![GitHub Last Commit](https://img.shields.io/github/last-commit/HariSekhon/DevOps-Bash-tools?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/commits/master)\n\n[![Codacy](https://app.codacy.com/project/badge/Grade/dffc1bfd13404c95b5a0ab97fd47974e)](https://www.codacy.com/gh/HariSekhon/DevOps-Bash-tools/dashboard)\n[![CodeFactor](https://www.codefactor.io/repository/github/harisekhon/devops-bash-tools/badge)](https://www.codefactor.io/repository/github/harisekhon/devops-bash-tools)\n[![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=alert_status)](https://sonarcloud.io/dashboard?id=HariSekhon_DevOps-Bash-tools)\n[![Maintainability Rating](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=sqale_rating)](https://sonarcloud.io/dashboard?id=HariSekhon_DevOps-Bash-tools)\n[![Reliability Rating](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=reliability_rating)](https://sonarcloud.io/dashboard?id=HariSekhon_DevOps-Bash-tools)\n[![Security Rating](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=security_rating)](https://sonarcloud.io/dashboard?id=HariSekhon_DevOps-Bash-tools)\n[![Vulnerabilities](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=vulnerabilities)](https://sonarcloud.io/summary/new_code?id=HariSekhon_DevOps-Bash-tools)\n\n<!--\nBitBucket exposes HTML comments - open issue - works properly on GitHub/GitLab\ndoesn't detect shell code properly\n[![Lines of Code](https://sonarcloud.io/api/project_badges/measure?project=HariSekhon_DevOps-Bash-tools&metric=ncloc)](https://sonarcloud.io/dashboard?id=HariSekhon_DevOps-Bash-tools)\n-->\n\n[![Linux](https://img.shields.io/badge/OS-Linux-blue?logo=linux)](#hari-sekhon---devops-bash-tools)\n[![Mac](https://img.shields.io/badge/OS-Mac-blue?logo=apple)](#hari-sekhon---devops-bash-tools)\n[![Docker](https://img.shields.io/badge/container-Docker-blue?logo=docker&logoColor=white)](https://hub.docker.com/r/harisekhon/bash-tools)\n[![Dockerfile](https://img.shields.io/badge/repo-Dockerfiles-blue?logo=docker&logoColor=white)](https://github.com/HariSekhon/Dockerfiles)\n[![DockerHub Pulls](https://img.shields.io/docker/pulls/harisekhon/bash-tools?label=DockerHub%20pulls&logo=docker&logoColor=white)](https://hub.docker.com/r/harisekhon/bash-tools)\n[![StarTrack](https://img.shields.io/badge/Star-Track-blue?logo=github)](https://seladb.github.io/StarTrack-js/#/preload?r=HariSekhon,Nagios-Plugins&r=HariSekhon,Dockerfiles&r=HariSekhon,DevOps-Python-tools&r=HariSekhon,DevOps-Perl-tools&r=HariSekhon,DevOps-Bash-tools&r=HariSekhon,HAProxy-configs&r=HariSekhon,SQL-scripts)\n[![StarCharts](https://img.shields.io/badge/Star-Charts-blue?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/STARCHARTS.md)\n\n[![Mac Homebrew](https://img.shields.io/badge/Mac-Homebrew-999999?logo=apple&logoColor=white)](https://brew.sh/)\n[![Alpine](https://img.shields.io/badge/Linux-Alpine-0D597F?logo=alpine%20linux)](https://alpinelinux.org/)\n[![CentOS](https://img.shields.io/badge/Linux-CentOS-262577?logo=centos&logoColor=white)](https://www.centos.org/)\n[![Debian](https://img.shields.io/badge/Linux-Debian-A81D33?logo=debian)](https://www.debian.org/)\n[![Fedora](https://img.shields.io/badge/Linux-Fedora-294172?logo=fedora&logoColor=white)](https://getfedora.org/)\n[![Redhat](https://img.shields.io/badge/Linux-Redhat-EE0000?logo=red%20hat)](https://www.redhat.com/en)\n[![Rocky](https://img.shields.io/badge/Linux-Rocky-10B981?logo=rockylinux&logoColor=white)](https://rockylinux.org/)\n[![Ubuntu](https://img.shields.io/badge/Linux-Ubuntu-E95420?logo=ubuntu&logoColor=white)](https://ubuntu.com/)\n\n<!-- TODO: fix\n[![DockerHub Build Automated](https://img.shields.io/docker/automated/harisekhon/bash-tools?logo=docker&logoColor=white)](https://hub.docker.com/r/harisekhon/bash-tools)\n[![Docker Build Status](https://img.shields.io/docker/cloud/build/harisekhon/bash-tools?logo=docker&logoColor=white)](https://hub.docker.com/r/harisekhon/bash-tools/builds)\n-->\n\n<!--\nofficial badges without logos to differentiate them\n\nthis one I don't trust it'll stick around so using shields version instead\n[![Build Status](https://badges.herokuapp.com/travis/HariSekhon/DevOps-Bash-tools?label=Travis%20CI)](https://travis-ci.org/HariSekhon/DevOps-Bash-tools)\n\nawkward URLs more nicely replaced with shields.io\n\n[![AppVeyor](https://ci.appveyor.com/api/projects/status/u6f97cskcgb30sce/branch/master?svg=true)](https://ci.appveyor.com/project/HariSekhon/devops-bash-tools/branch/master)\n[![Drone](https://cloud.drone.io/api/badges/HariSekhon/DevOps-Bash-tools/status.svg)](https://cloud.drone.io/HariSekhon/DevOps-Bash-tools)\n-->\n\n[![CI Builds Overview](https://img.shields.io/badge/CI%20Builds-Overview%20Page-blue?logo=circleci)](https://harisekhon.github.io/CI-CD/)\n[![Jenkins](https://img.shields.io/badge/Jenkins-ready-blue?logo=jenkins&logoColor=white)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/Jenkinsfile)\n[![Concourse](https://img.shields.io/badge/Concourse-ready-blue?logo=concourse&logoColor=white)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/cicd/.concourse.yml)\n[![GoCD](https://img.shields.io/badge/GoCD-ready-blue?logo=go&logoColor=white)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/cicd/.gocd.yml)\n[![TeamCity](https://img.shields.io/badge/TeamCity-ready-blue?logo=teamcity)](https://github.com/HariSekhon/TeamCity-CI)\n\n[![CircleCI](https://circleci.com/gh/HariSekhon/DevOps-Bash-tools.svg?style=svg)](https://circleci.com/gh/HariSekhon/DevOps-Bash-tools)\n[![BuildKite](https://img.shields.io/buildkite/f11bdd9690a9bac9a8edc6094dc2f2b9af3218a7a15d4ec17d/master?label=BuildKite&logo=buildkite)](https://buildkite.com/hari-sekhon/devops-bash-tools)\n[![AppVeyor](https://img.shields.io/appveyor/build/harisekhon/devops-bash-tools/master?logo=appveyor&label=AppVeyor)](https://ci.appveyor.com/project/HariSekhon/devops-bash-tools/branch/master)\n[![Drone](https://img.shields.io/drone/build/HariSekhon/DevOps-Bash-tools/master?logo=drone&label=Drone)](https://cloud.drone.io/HariSekhon/DevOps-Bash-tools)\n[![Codefresh](https://g.codefresh.io/api/badges/pipeline/harisekhon/GitHub%2FDevOps-Bash-tools?branch=master&key=eyJhbGciOiJIUzI1NiJ9.NWU1MmM5OGNiM2FiOWUzM2Y3ZDZmYjM3.O69674cW7vYom3v5JOGKXDbYgCVIJU9EWhXUMHl3zwA&type=cf-1)](https://g.codefresh.io/pipelines/edit/new/builds?id=5e53eaeea284e010982eaa6e&pipeline=DevOps-Bash-tools&projects=GitHub&projectId=5e52ca8ea284e00f882ea992&context=github&filter=page:1;pageSize:10;timeFrameStart:week)\n[![Cirrus CI](https://img.shields.io/cirrus/github/HariSekhon/DevOps-Bash-tools/master?logo=Cirrus%20CI&label=Cirrus%20CI)](https://cirrus-ci.com/github/HariSekhon/DevOps-Bash-tools)\n[![Semaphore](https://harisekhon.semaphoreci.com/badges/DevOps-Bash-tools.svg)](https://harisekhon.semaphoreci.com/projects/DevOps-Bash-tools)\n[![Buddy](https://img.shields.io/badge/Buddy-ready-1A86FD?logo=buddy)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/buddy.yml)\n[![Shippable](https://img.shields.io/badge/Shippable-legacy-lightgrey?logo=jfrog&label=Shippable)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/shippable.yml)\n[![Travis CI](https://img.shields.io/badge/TravisCI-ready-blue?logo=travis&label=Travis%20CI)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/travis/.travis.yml)\n[![Reviewed by Hound](https://img.shields.io/badge/Reviewed%20by-Hound-8E64B0.svg)](https://houndci.com)\n\n[![Repo on GitHub](https://img.shields.io/badge/repo-GitHub-2088FF?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools)\n[![Repo on GitLab](https://img.shields.io/badge/repo-GitLab-FCA121?logo=gitlab)](https://gitlab.com/HariSekhon/DevOps-Bash-tools)\n[![Repo on Azure DevOps](https://img.shields.io/badge/repo-Azure%20DevOps-0078D7?logo=azure%20devops)](https://dev.azure.com/harisekhon/GitHub/_git/DevOps-Bash-tools)\n[![Repo on BitBucket](https://img.shields.io/badge/repo-BitBucket-0052CC?logo=bitbucket)](https://bitbucket.org/HariSekhon/DevOps-Bash-tools)\n\n[![Azure DevOps Pipeline](https://dev.azure.com/harisekhon/GitHub/_apis/build/status/HariSekhon.DevOps-Bash-tools?branchName=master)](https://dev.azure.com/harisekhon/GitHub/_build/latest?definitionId=1&branchName=master)\n[![GitLab Pipeline](https://img.shields.io/badge/GitLab%20CI-legacy-lightgrey?logo=gitlab)](https://gitlab.com/HariSekhon/DevOps-Bash-tools/pipelines)\n[![BitBucket Pipeline](https://img.shields.io/badge/Bitbucket%20CI-legacy-lightgrey?logo=bitbucket)](https://bitbucket.org/harisekhon/devops-bash-tools/addon/pipelines/home#!/)\n[![AWS CodeBuild](https://img.shields.io/badge/AWS%20CodeBuild-ready-blue?logo=amazon%20aws)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/cicd/buildspec.yml)\n[![GCP Cloud Build](https://img.shields.io/badge/GCP%20Cloud%20Build-ready-blue?logo=google%20cloud&logoColor=white)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/cicd/cloudbuild.yaml)\n\n[![ShellCheck](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/shellcheck.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/shellcheck.yaml)\n[![JSON](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/json.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/json.yaml)\n[![YAML](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/yaml.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/yaml.yaml)\n[![XML](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/xml.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/xml.yaml)\n[![Markdown](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/markdown.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/markdown.yaml)\n[![Validation](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/validate.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/validate.yaml)\n[![Kics](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/kics.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/kics.yaml)\n[![Grype](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/grype.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/grype.yaml)\n[![Semgrep](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/semgrep.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/semgrep.yaml)\n[![Semgrep Cloud](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/semgrep-cloud.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/semgrep-cloud.yaml)\n[![Trivy](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/trivy.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/trivy.yaml)\n\n[![Docker Build (Alpine)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_alpine.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_alpine.yaml)\n[![Docker Build (Debian)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_debian.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_debian.yaml)\n[![Docker Build (Fedora)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_fedora.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_fedora.yaml)\n[![Docker Build (Ubuntu)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_ubuntu.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/docker_bash_ubuntu.yaml)\n\n[![GitHub Actions Ubuntu](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/GitHub%20Actions%20Ubuntu/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22GitHub+Actions+Ubuntu%22)\n[![Mac](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac.yaml)\n[![Mac 11](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac_11.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac_11.yaml)\n[![Mac 12](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac_12.yaml/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions/workflows/mac_12.yaml)\n[![Ubuntu](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Ubuntu/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Ubuntu%22)\n[![Ubuntu 20.04](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Ubuntu%2020.04/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Ubuntu+20.04%22)\n[![Ubuntu 22.04](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Ubuntu%2022.04/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Ubuntu+22.04%22)\n[![Debian](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Debian/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Debian%22)\n[![Debian 10](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Debian%2010/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Debian+10%22)\n[![Debian 11](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Debian%2011/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Debian+11%22)\n[![Debian 12](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Debian%2012/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Debian+12%22)\n[![Fedora](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Fedora/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Fedora%22)\n[![Alpine](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Alpine/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Alpine%22)\n[![Alpine 3](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Alpine%203/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Alpine+3%22)\n\n[![Python 3.7](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Python%203.7/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Python+3.7%22)\n[![Python 3.8](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Python%203.8/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Python+3.8%22)\n[![Python 3.9](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Python%203.9/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Python+3.9%22)\n[![Python 3.10](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Python%203.10/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Python+3.10%22)\n[![Python 3.11](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Python%203.11/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Python+3.11%22)\n<!--\n[![Self Hosted](https://github.com/HariSekhon/DevOps-Bash-tools/workflows/Self%20Hosted/badge.svg)](https://github.com/HariSekhon/DevOps-Bash-tools/actions?query=workflow%3A%22Self+Hosted%22)\n-->\n\n<!-- TODO: https://codecov.io, https://coveralls.io -->\n\n[git.io/bash-tools](https://git.io/bash-tools)\n\n1000+ DevOps Shell Scripts and Advanced Bash environment.\n\nFast, Advanced Systems Engineering, Automation, APIs, shorter CLIs, etc.\n\nHeavily used in many [GitHub repos](https://github.com/search?o=desc&q=user%3Aharisekhon+type%3Arepository&type=Repositories), dozens of [DockerHub builds](https://hub.docker.com/r/harisekhon) ([Dockerfiles](https://github.com/HariSekhon/Dockerfiles)) and 600+ [CI builds](https://harisekhon.github.io/CI-CD/).\n\n## Summary\n\n- Scripts for many popular DevOps technologies, see [Index](#index) below for more details\n- Advanced configs for common tools like [Git](https://git-scm.com/), [vim](https://www.vim.org/), [screen](https://www.gnu.org/software/screen/), [tmux](https://github.com/tmux/tmux/wiki), [PostgreSQL psql](https://www.postgresql.org/) etc...\n- CI configs for most major Continuous Integration products (see [CI builds](https://harisekhon.github.io/CI-CD/) page)\n- CI scripts for a drop-in framework of standard checks to run in all [CI builds](https://harisekhon.github.io/CI-CD/), CI detection, accounting for installation differences across CI environments, root vs user, virtualenvs etc.\n- API scripts auto-handling authentication, tokens and other details to quickly query popular APIs with a few keystrokes just supplying the `/path/endpoint`\n- Advanced Bash environment - `.bashrc` + `.bash.d/*.sh` - aliases, functions, colouring, dynamic Git & shell behaviour enhancements, automatic pathing for installations and major languages like Python, Perl, Ruby, NodeJS, Golang across Linux distributions and Mac. See [.bash.d/README.md](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/.bash.d/README.md)\n- Installs the best systems packages -\n  [AWS CLI](https://aws.amazon.com/cli/),\n  [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest),\n  [GCloud SDK](https://cloud.google.com/sdk),\n  [Digital Ocean CLI](https://docs.digitalocean.com/reference/doctl/),\n  [Terraform](https://www.terraform.io/),\n  [Terragrunt](https://terragrunt.gruntwork.io/),\n  [GitHub CLI](https://github.com/cli/cli),\n  [Kubernetes](https://kubernetes.io/)\n  [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/) &\n  [kustomize](https://kustomize.io/),\n  [Helm](https://helm.sh/),\n  [eksctl](https://eksctl.io/),\n  [Docker-Compose](https://docs.docker.com/compose/),\n  [jq](https://stedolan.github.io/jq/)\n  and many others... extensive package lists for servers and desktops for most major Linux distributions package managers and Mac\n  - `install/` - contains many installation scripts for popular open source software and direct binary downloads from GitHub releases\n  - `configs/` - contains many dot configs for common technologies like ViM, top, Screen, Tmux, MySQL, PostgreSQL etc.\n  - `setup/` - contains setup scripts, package lists, extra configs, Mac OS X settings etc.\n- Utility Libraries used by many hundreds of scripts and [builds](https://harisekhon.github.io/CI-CD/) across [repos](https://github.com/search?o=desc&q=user%3Aharisekhon+type%3Arepository&type=Repositories):\n  - `.bash.d/` - interactive library\n  - `lib/` - scripting and CI library\n- [SQL Scripts](https://github.com/HariSekhon/SQL-scripts) - 100+ scripts for [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), [AWS Athena](https://aws.amazon.com/athena/) + [CloudTrail](https://aws.amazon.com/cloudtrail/), [Google BigQuery](https://cloud.google.com/bigquery)\n- [Templates](https://github.com/HariSekhon/Templates) - templates for common programming languages and build configs\n- [Kubernetes Configs](https://github.com/HariSekhon/Kubernetes-configs) - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks\n\nSee Also: [similar DevOps repos](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/README.md#see-also) in other languages\n\nHari Sekhon\n\nCloud & Big Data Contractor, United Kingdom\n\n(ex-Cloudera, former Hortonworks Consultant)\n\n[![My LinkedIn](https://img.shields.io/badge/LinkedIn%20Profile-HariSekhon-blue?logo=data:image/svg%2bxml;base64,PHN2ZyByb2xlPSJpbWciIGZpbGw9IiNmZmZmZmYiIHZpZXdCb3g9IjAgMCAyNCAyNCIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIj48dGl0bGU+TGlua2VkSW48L3RpdGxlPjxwYXRoIGQ9Ik0yMC40NDcgMjAuNDUyaC0zLjU1NHYtNS41NjljMC0xLjMyOC0uMDI3LTMuMDM3LTEuODUyLTMuMDM3LTEuODUzIDAtMi4xMzYgMS40NDUtMi4xMzYgMi45Mzl2NS42NjdIOS4zNTFWOWgzLjQxNHYxLjU2MWguMDQ2Yy40NzctLjkgMS42MzctMS44NSAzLjM3LTEuODUgMy42MDEgMCA0LjI2NyAyLjM3IDQuMjY3IDUuNDU1djYuMjg2ek01LjMzNyA3LjQzM2MtMS4xNDQgMC0yLjA2My0uOTI2LTIuMDYzLTIuMDY1IDAtMS4xMzguOTItMi4wNjMgMi4wNjMtMi4wNjMgMS4xNCAwIDIuMDY0LjkyNSAyLjA2NCAyLjA2MyAwIDEuMTM5LS45MjUgMi4wNjUtMi4wNjQgMi4wNjV6bTEuNzgyIDEzLjAxOUgzLjU1NVY5aDMuNTY0djExLjQ1MnpNMjIuMjI1IDBIMS43NzFDLjc5MiAwIDAgLjc3NCAwIDEuNzI5djIwLjU0MkMwIDIzLjIyNy43OTIgMjQgMS43NzEgMjRoMjAuNDUxQzIzLjIgMjQgMjQgMjMuMjI3IDI0IDIyLjI3MVYxLjcyOUMyNCAuNzc0IDIzLjIgMCAyMi4yMjIgMGguMDAzeiIvPjwvc3ZnPgo=)](https://www.linkedin.com/in/HariSekhon/)\n<br>*(you're welcome to connect with me on LinkedIn)*\n\n### Quick Setup\n\nTo bootstrap, install packages and link in to your shell profile to inherit all configs, do:\n\n```bash\ncurl -L https://git.io/bash-bootstrap | sh\n```\n\n- Adds sourcing to `.bashrc`/`.bash_profile` to automatically inherit all `.bash.d/*.sh` environment enhancements for all technologies (see [Inventory](#index) below)\n- Symlinks `.*` config dotfiles to `$HOME` for [git](https://git-scm.com/), [vim](https://www.vim.org/), top, [htop](https://hisham.hm/htop/), [screen](https://www.gnu.org/software/screen/), [tmux](https://github.com/tmux/tmux/wiki), [editorconfig](https://editorconfig.org/), [Ansible](https://www.ansible.com/), [PostgreSQL](https://www.postgresql.org/) `.psqlrc` etc. (only when they don't already exist so there is no conflict with your own configs)\n- Installs OS package dependencies for all scripts (detects the OS and installs the right RPMs, Debs, Apk or Mac HomeBrew packages)\n- Installs Python packages\n- Installs [AWS CLI](https://aws.amazon.com/cli/)\n\nTo only install package dependencies to run scripts, simply `cd` to the git clone directory and run `make`:\n\n```shell\ngit clone https://github.com/HariSekhon/DevOps-Bash-tools bash-tools\ncd bash-tools\nmake\n```\n\n`make install` sets your shell profile to source this repo. See [Individual Setup Parts](#individual-setup-parts) below for more install/uninstall options.\n\n## Index\n\n- [Dot Configs](#dot-configs) - `.gitconfig`, `.vimrc`, `.screenrc`, `.tmux.conf`, `.toprc`, `.gitignore`...\n- [Bash Environment & Libraries](#bash-environment--libraries) - `.bashrc`, `.bash.d/` interactive library, `lib/` scripting library\n- [Installation Scripts](#installation-scripts) for many popular open source technologies\n- [Linux & Mac](#linux--mac) - curl OAuth / JWT, LDAP, find duplicate files, SSL certificate get/validate, URL encoding/decoding, Vagrant\n- [Mac & AppleScript](#mac--applescript) - Mac settings and UI automation scripts, send keystrokes, mouse clicks,\n  detect foreground app, switch app, detect locked screen or screensaver, activate screensaver, Hammerspoon system event handlers such as automatically switching audio to be able to Shazam while watching on AirPods\n- [Monitoring](#monitoring) - Grafana, Prometheus, Node Exporter, scripted collection of common Linux & Mac cli\n  monitoring stats and log locations for quick generation of vendor support tarball bundles both locally and over SSH\n- [AWS - Amazon Web Services](#aws---amazon-web-services) - AWS account summary, lots of IAM reports, CIS Benchmark config hardening, EC2, ECR, EKS, Spot termination, S3 access logging, KMS key rotation info, SSM, CloudTrail, CloudWatch billing alarm with SNS notification topic and subscription for email alerts\n- [GCP - Google Cloud Platform](#gcp---google-cloud-platform) - massive GCP auto-inventory, scripts for GCE, GKE, GCR, Secret Manager, BigQuery, Cloud SQL, Cloud Scheduler, Terraform service account creation\n- [Kubernetes](#kubernetes) - massive Kubernetes auto-inventory, cluster management scripts & tricks\n- [Docker](#docker) - Docker API, Dockerhub API, Quay.io API scripts\n- [Databases](#databases) - fast CLI wrappers, instant Docker sandboxes (PostgreSQL, MySQL, MariaDB, SQLite), [SQL scripts](https://github.com/HariSekhon/SQL-scripts), SQL script testers against all versions of a DB, advanced `.psqlrc`\n- [Data](#data) - data tools, converters and format validators for Avro, Parquet, CSV, JSON, INI / Properties files (Java), LDAP LDIF, XML, YAML\n- [Big Data & NoSQL](#big-data--nosql) - Kafka, Hadoop, HDFS, Hive, Impala, ZooKeeper, Cloudera Manager API & Cloudera Navigator API scripts\n- [Git - GitHub, GitLab, Bitbucket, Azure DevOps](#git---github-gitlab-bitbucket-azure-devops) - scripts for Git local & mirror management, GitHub, GitLab & BitBucket APIs\n- [Markdown](#markdown) - generate Markdown indexes and debug `mdl` issues like MD005 inconsistent list indentation in large `README.md` files\n- [CI/CD - Continuous Integration / Continuous Delivery](#cicd---continuous-integration--continuous-deployment) - API scripts & build pipeline configs for most major CI systems:\n  - Jenkins, Concourse, GoCD, TeamCity - one-touch boot & build\n  - Azure DevOps Pipelines, GitHub Actions Workflows, GitLab CI, BitBucket Pipelines, AppVeyor, BuildKite, Travis CI, Circle CI, Codefresh, CodeShip, Drone.io, Semaphore CI, Shippable ...\n  - Terraform Cloud, Octopus Deploy\n  - Checkov / Bridgecrew Cloud\n- [AI & IPaaS](#ai--ipaas) - OpenAI (ChatGPT), Make.com\n- [Internet Services](#internet-services) - Google Maps, Cloudflare, DataDog, Digital Ocean, Kong API Gateway, GitGuardian, Jira, NGrok, Traefik, Pingdom, Wordpress and various pastebins and file upload sites\n- [Java](#java) - Java utilies to debug running Java programs or decompile Java JAR code for deeper debugging\n- [Python](#python) - Python utilities & library management\n- [Perl](#perl) - Perl utilities & library management\n- [Golang](#golang) - Golang utilities\n- [Diagrams](#diagrams) - scripts to generate diagrams from D2lang, MermaidJS and Python Mingrammer source code used in my [HariSekhon/Diagrams-as-Code](https://github.com/HariSekhon/Diagrams-as-Code) repo\n- [Media](#media) - video downloaders & converts, MP3 metadata editing, grouping and ordering of albums and audiobooks, mkv/avi to mp4 converters, 720p video downscaler for posting to social media, download YouTube videos or even entire channels and videos from other social media sites like Twitter / X or Facebook, terminal gif capture\n- [Spotify](#spotify) - 40+ Spotify API scripts for backups, managing playlists, track deduplication, URI conversion, search, add/delete, liked tracks, followed artists, top artists, top tracks etc.\n- [More Linux & Mac](#more-linux--mac) - more systems administration scripts, package installation automation\n- [Builds, Languages & Linting](#builds-languages--linting) - programming language, build system & CI linting\n- [Templates](https://github.com/HariSekhon/Templates) - Templates for AWS, GCP, Terraform, Docker, Jenkins, Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, GitHub Actions, CircleCI, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml etc.\n- [Kubernetes Configs](https://github.com/HariSekhon/Kubernetes-configs) - Kubernetes YAML configs for most common scenarios, including Production Best Practices, Tips & Tricks\n\n### Dot Configs\n\nTop-level dotfiles and `configs/` directory:\n\n- `.*` - dot conf files for lots of common software eg. advanced `.vimrc`, `.gitconfig`, massive `.gitignore`, `.editorconfig`, `.screenrc`, `.tmux.conf` etc.\n  - `.vimrc` - contains many awesome [vim](https://www.vim.org/) tweaks, plus hotkeys for linting lots of different file types in place, including Python, Perl, Bash / Shell, Dockerfiles, JSON, YAML, XML, CSV, INI / Properties files, LDAP LDIF etc without leaving the editor!\n  - `.screenrc` - fancy [screen](https://www.gnu.org/software/screen/) configuration including advanced colour bar, large history, hotkey reloading, auto-blanking etc.\n  - `.tmux.conf` - fancy [tmux](https://github.com/tmux/tmux/wiki) configuration include advanced colour bar and plugins, settings, hotkey reloading etc.\n  - [Git](https://git-scm.com/):\n    - `.gitconfig` - advanced Git configuration\n    - `.gitignore` - extensive Git ignore of trivial files you shouldn't commit\n    - enhanced Git diffs\n    - protections against committing AWS secret keys or merge conflict unresolved files\n\n### Bash Environment & Libraries\n\nTop-level `.bashrc` and `.bash.d/` directory:\n\n- `.bashrc` - shell tuning and sourcing of `.bash.d/*.sh`\n- `.bash.d/*.sh` - thousands of lines of advanced bashrc code, aliases, functions and environment variables for:\n  - [Linux](https://en.wikipedia.org/wiki/Linux) & [Mac](https://en.wikipedia.org/wiki/MacOS)\n  - SCM - [Git](https://git-scm.com/), [Mercurial](https://www.mercurial-scm.org/), [Svn](https://subversion.apache.org)\n  - [AWS](https://aws.amazon.com/)\n  - [GCP](https://cloud.google.com/)\n  - [Docker](https://www.docker.com/)\n  - [Kubernetes](https://kubernetes.io/)\n  - [Kafka](http://kafka.apache.org/)\n  - [Vagrant](https://www.vagrantup.com/)\n  - automatic GPG and SSH agent handling for handling encrypted private keys without re-entering passwords, and lazy evaluation to only prompt key load the first time SSH is called\n  - and lots more - see [.bash.d/README](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/.bash.d/README.md) for a more detailed list\n  - run `make bash` to link `.bashrc`/`.bash_profile` and the `.*` dot config files to your `$HOME` directory to auto-inherit everything\n- `lib/*.sh` - Bash utility libraries full of functions for\n  [Docker](https://www.docker.com/),\n  environment,\n  CI detection ([Travis CI](https://travis-ci.org/), [Jenkins](https://jenkins.io/) etc),\n  port and HTTP url availability content checks etc.\n  Sourced from all my other [GitHub repos](https://github.com/harisekhon) to make setting up Dockerized tests easier.\n\n### Installation Scripts\n\n- `install/install_*.sh` - various simple to use installation scripts for common technologies like:\n  - [AWS CLI](https://aws.amazon.com/cli/)\n  - [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest)\n  - [GCloud SDK](https://cloud.google.com/sdk)\n  - [GitHub CLI](https://cli.github.com/)\n  - [Terraform](https://www.terraform.io/)\n  - [Terragrunt](https://terragrunt.gruntwork.io/)\n  - [Direnv](https://direnv.net/)\n  - [Ansible](https://www.ansible.com/)\n  - [K3s](https://k3s.io)\n  - [MiniKube](https://kubernetes.io/docs/setup/learning-environment/minikube/) (Kubernetes)\n  - [MiniShift](https://www.okd.io/minishift/)\n    ([Redhat OpenShift](https://www.openshift.com/) / [OKD](https://www.okd.io/) dev VMs)\n  - [Maven](https://maven.apache.org/)\n  - [Gradle](https://gradle.org/)\n  - [SBT](https://www.scala-sbt.org/)\n  - [EPEL](https://fedoraproject.org/wiki/EPEL)\n  - [RPMforge](http://repoforge.org/)\n  - [Homebrew](https://brew.sh/)\n  - [Travis CI](https://travis-ci.org/)\n  - [Circle CI](https://circleci.com/)\n  - [AppVeyor](https://www.appveyor.com/)\n  - [BuildKite](https://buildkite.com)\n  - [Avro Tools](https://avro.apache.org/)\n  - [Parquet Tools](https://github.com/apache/parquet-mr/tree/master/parquet-tools)\n  - [Prometheus](https://prometheus.io/)\n  - various JDKs and RDBMS JDBC connector jars\n  - and many more...\n\n### Linux & Mac\n\n`bin/` directory:\n\n- `login.sh` - logs to major Cloud platforms if their credentials are found in the environment, CLIs such as AWS, GCP, Azure, GitHub... Docker registries: DockerHub, GHCR, ECR, GCR, GAR, ACR, Gitlab, Quay...\n- `clean_caches.sh` - cleans out OS package and programming language caches - useful to save space or reduce Docker image size\n- `crypto_dice_rolls.sh` - generates 100 random dice rolls to test a new crypto hardware wallet's fidelity (do not use this for your real crypto seed as your machine could be infected with malware which steals your seed phrase)\n- `delete_duplicate_files.sh` - deletes duplicate files with (N) suffixes, commonly caused by web browser downloads,\n  in the given or current directory. Checks they're exact duplicates of a matching basename file without the (N) suffix with\n  the exact same checksum for safety. Prompts to delete per file. To auto-accept deletions, do\n  `yes | delete_duplicate_files.sh`. This is a fast way of cleaning up your `~/Downloads` directory and can be put your\n  user crontab\n- `disk_speed_read_sequential_dd.sh` - runs a sequential read speed test from the given file using dd and bypassing filesystem cache for a more accurate test\n- `disk_speed_read_random_dd.sh` - runs a random I/O read speed test from the given file using dd and bypassing filesystem cache for a more accurate test\n- `disk_speed_write_sequential_dd.sh` - runs a sequential write speed test to a file in the given or current directory using dd and bypassing filesystem cache for a more accurate test\n- `disk_speed_read_sequential_fio.sh` - runs a sequential read speed test in the current or given directory using fio\n- `disk_speed_read_random_fio.sh` - runs a random I/O read test in the current or given directory using fio\n- `disk_speed_write_sequential_fio.sh` - runs a sequential write speed test to the current or given directory using fio\n- `disk_speed_write_random_fio.sh` - runs a sequential write speed test to the current or given directory using fio\n- `download_url_file.sh` - downloads a file from a URL using wget with no clobber and continue support, or curl with atomic replacement to avoid race conditions. Used by `github/github_download_release_file.sh`, `github_download_release_jar.sh`, and `install/download_*_jar.sh`\n- `curl_auth.sh` - shortens `curl` command by auto-loading your OAuth2 / JWT API token or username & password from environment variables or interactive starred password prompt through a ram file descriptor to avoid placing them on the command line (which would expose your credentials in the process list or OS audit log files). Used by many other adjacent API querying scripts\n- `curl_with_cookies.sh` - extracts cookies for a given URL from your `\\$BROWSER`'s cookie jar and passes them to the `curl` command along with the rest of the args (workaround for older curl builds and Homebrew builds that don't have the newer `--cookies-from-browser functionality)\n- `find_duplicate_files*.sh` - finds duplicate files by size and/or checksum in given directory trees. Checksums are only done on files that already have matching byte counts for efficiency\n- `find_broken_links.sh` - find broken links with delays to avoid tripping defenses\n- `find_broken_symlinks.sh` - find broken symlinks pointing to non-existent files/directories\n- `find_lock.sh` - tries to find if a lockfile is used in the given or current working directory by taking snapshots of the file list before and after a prompt in which you should open/close an application\n- `foreach_path_bin.sh` - runs each binary of the given name found in `$PATH` with the args given. Useful to find all the installed versions of a program in different paths eg. `~/bin/` vs `/usr/local/bin/` eg. `foreach_path_bin.sh terraform --version`\n- `http_duplicate_urls.sh` - find duplicate URLs in a given web page\n- `htmldecode.sh` - decodes HTML encoding. Detects available tools such as Perl, Python or xmlstarlet and uses whatever is available\n- `ldapsearch.sh` - shortens `ldapsearch` command by inferring switches from environment variables\n- `ldap_user_recurse.sh` / `ldap_group_recurse.sh` - recurse Active Directory LDAP users upwards to find all parent groups, or groups downwards to find all nested users (useful for debugging LDAP integration and group-based permissions)\n- `linux_distro_versions.sh` - quickly returns the list of major versions for a given Linux distro\n- `diff_line_threshold.sh` - compares two files vs a line count diff threshold to determine if they are radically different. Used to avoid overwriting files which are not mere updates but completely different files\n- `mv.sh` - moves directory trees resumably and removes the source files as they're copied over. Useful to migrate data from one disk to another, optionally with checksums. Uses rsync and shows the overall % of files transferred and the MB/s data transfer rate\n- `network_gateway.sh` - get the network gateway IP address on Linux or Mac\n- `open.sh` - opens given arg, file or URL using whatever default system opener is available for Linux or Mac\n- `organize_downloads.sh` - moves files of well-known extensions in the `$HOME/Downloads` directory older than 1 week to capitalized subdirectories of their type to keep the `$HOME/Downloads/` directory tidy\n- `copy_to_clipboard.sh` - copies stdin or string arg to system clipboard on Linux or Mac\n- `paste_from_clipboard.sh` - pastes from system clipboard to stdout on Linux or Mac\n- `paste_from_clipboard_upon_changes.sh` - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes\n- `paste_diff_settings.sh` - takes snapshots of before and after clipboard changes and diffs them to show config changes\n- `processes_ram_sum.sh` - sums the RAM usage of all processes matching a given regex in GB to one decimal place\n- `pldd.sh` - parses `/proc` on Linux to show the runtime `.so` loaded dynamic shared libraries a program pid is using. Runtime equivalent of the classic static `ldd` command and because the system `pldd` command often fails to attach to a process\n- `random_select.sh` - selects one of given args at random. Useful for sampling, running randomized subsets of large test suites etc.\n- `random_number.sh` - prints a random integer between two integer arguments (inclusive)\n- `random_string.sh` - prints a random alphanumeric string of a given length\n- `screen_terminal_to_stdout.sh` - dumps the GNU Screen terminal output to stdout\n- `screen_terminal_to_clipboard.sh` - dumps the GNU Screen terminal output to a temp file and copies to clipboard for sharing & debugging purposes\n- `shields_embed_logo.sh` - base64 encodes a given icon file or url and prints the `logo=...` url parameter you need to add the [shields.io](https://shields.io/) badge url\n- `shorten_text_selection.sh` - shortens the selected text in the prior window. Replaces `and` with `&` and crushes out multiple blank lines. I use this for LinkedIn comments due to the short 1250 character limit\n- `shred_file.sh` - overwrites a file 7 times to DoD standards before deleting it to prevent recovery of sensitive information\n- `shred_free_space.sh` - overwrites free space to prevent recovery of sensitive information for files that have already been deleted\n- `split.sh` - split large files into N parts (defaults to the number of your CPU cores) to parallelize operations on them\n- `ssl_get_cert.sh` - gets a remote `host:port` server's SSL cert in a format you can pipe, save and use locally, for example in Java truststores\n- `ssl_verify_cert.sh` - verifies a remote SSL certificate (battle tested more feature-rich version `check_ssl_cert.pl` exists in the [Advanced Nagios Plugins](https://github.com/HariSekhon/Nagios-Plugins) repo)\n- `ssl_verify_cert_by_ip.sh` - verifies SSL certificates on specific IP addresses, useful to test SSL source addresses for CDNs, such as Cloudflare Proxied sources before enabling SSL Full-Strict Mode for end-to-end, or Kubernetes ingresses (see also `curl_k8s_ingress.sh`)\n- `text_filter_ending_substrings.sh` - for a given patterns file of substring endings, print all lines that match in the following files. Uses awk to safely handle all characters as literals, unlike grep, while also maintaining end anchoring which you cannot do using `grep -F`. Optimized awk code uses a bucketing hash for performance to not attempt matching lines which are shorter than patterns, reducing the number of match attempts\n- `tmux_vertical.sh` - launches tmux with N-way vertical shell split or commands given as args in equally balanced vertical panes. Fast way to launch a bunch of shell or commands in an easily reviewable side-by-side way\n- `tmux_horizontal.sh` - same as above but split horizontally\n- `tmux_square.sh` - same as above but with 4 panes in a square tiled view\n- `urlencode.sh` / `urldecode.sh` - URL encode/decode quickly on the command line, in pipes etc.\n- `urlextract.sh` - extracts the URLs from a given string arg, file or standard input\n- `url_extract_redirects.sh` - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the redirected urls instead to stdout\n- `url_replace_redirects.sh` - extracts the URLs from a given string arg, file or standard input, queries each one and outputs the entire contents to stdout with the urls replaced by the redirected urls\n- `urlopen.sh` - opens the URL given as an arg, or first URL found from stdin or a given file.\n  Uses the system's default browser\n- `vagrant_hosts.sh` - generate `/etc/hosts` output from a `Vagrantfile`\n- `vagrant_total_mb.sh` - calculate the RAM committed to VMs in a `Vagrantfile`\n\nSee also [Knowledge Base notes for Linux](https://github.com/HariSekhon/Knowledge-Base/blob/main/linux.md)\nand [Mac](https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md).\n\n### Mac & AppleScript\n\nMac automation scripts to automate the Mac UI and settings\n\n`bin/` directory:\n\n- `mac_diff_settings.sh` - takes before and after snapshots of UI setting changes and diffs them to make it easy to find `defaults` keys to add to `setup/mac_settings.sh` to save settings\n- `mac_restore_file.sh` - checks all the backup mount points for the latest backup that has a given file and then restores it\n- `mac_backup_du_in_progress.sh` - find large files in the currently in-progress Time Machine backup to find out what is taking so long and racking up so many more GB of changes than you expect. This helps discover large but unnecessary files that you might want to exclude using the adjacent script `mac_backup_exclude_paths.sh`\n- `mac_backup_exclude_paths.sh` - excludes many common large caches, docker and VM paths from macOS Time Machine backups\n- `mac_backup_find_excluded_paths.sh` - does a deep search for macOS Time Machine excluded backup paths on file/folder attributes. See [HariSekhon/Knowledge-Base Mac page](https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md#time-machine) for why\n- `mac_rmdir.sh` - safely delete a directory on Mac only if it is empty of actual data, by first removing macOS hidden metadata files and dirs such as `.fseventsd/`, `.Spotlight-V100/` and `.DS_Store` - straight `rmdir` fails otherwise\n- `mac_iso_to_usb.sh` - converts a given ISO file to a USB bootable image and burns it onto a given or detected inserted USB drive\n- `mac_ramdisk.sh` - creates a mac ramdisk of given MB size. Useful for performance, or even testing disk write scripts such as `disk_speed_write_*.sh` without wearing out your SSD\n- `mac_delete_local_snapshots.sh` - deletes local macOS snapshots to free up disk space. When there is a substantial discrepancy between what the `df -h` command and the Finder UI shows, this is often the cause\n- `copy_to_clipboard.sh` - copies stdin or string arg to system clipboard on Linux or Mac\n- `paste_from_clipboard.sh` - pastes from system clipboard to stdout on Linux or Mac\n- `paste_from_clipboard_upon_changes.sh` - pastes from system clipboard to stdout on Linux or Mac whenever the clipboard changes\n- `paste_diff_settings.sh` - Takes snapshots of before and after clipboard changes and diffs them to show config changes\n\n`applescript/` directory:\n\n- `keystrokes.sh` - send N keystroke combinations\n- `mouse_clicks.sh` - send N mouse click combinations to sequence of screen coordinates\n  - `get_mouse_coordinates.sh` - print the current mouse coordinates - to know what to pass to above script\n- `mouse_clicks_remote_desktop.sh` - switches to Microsoft Remote Desktop, waits 10 seconds and then clicks the mouse\n  once a minute to prevent the screensaver from coming on. Workaround to Active Directory Group Policies that don't let\n  you disable the screensaver. Point your mouse to an area that will have no mouse click effect, the Cmd-Tab to Terminal\n  and run this\n- `get_frontmost_process_title.scpt` - detect the frontmost window\n  - to detect if you should send keystrokes / mouse clicks)\n- `set_frontmost_process.scpt` - switch to bring the given app to the foreground to send keystrokes / mouse clicks to it\n  - `browser_get_default.scpt` - get the default configured browser in format passable to Applescript (for  above script)\n- `is_screen_locked.py` - detect if the screen is locked to stop sending keystrokes or mouse clicks\n- `is_screensaver_running.scpt` - detect if the screensaver is running to stop sending keystrokes or mouse clicks\n- `reopen_app.sh` - relaunch a given app\n  (used to reload Shazam to detect DB changes after removing tracks programmatically from its DB)\n- `spotify_app_search.sh` - runs a search in the Spotify App on Mac using Applescript\n- `shazam_app_dump_tracks.sh` - dumps `artist - track` one per line from the Shazam local sqlite DB\n- `shazam_app_delete_track.sh` - deletes a given `\"artist\" \"track\"` from the Shazam local sqlite DB\n- `shazam_search_spotify_then_delete_track.sh` - searches for each Shazam'd track in the local Spotify desktop app,\n  then prompts to delete each track from the local Shazam DB once you've saved it in Spotify.\n  Useful to migrate Shazam'd tracks to Spotify after Apple removed the integration\n- `screensaver_activate.scpt` - activate screensaver\n- `shorten_text_selection.scpt` - shortens the selected text in the prior window. Replaces `and` with `&` and crushes\n  out multiple blank lines. I use this for LinkedIn comments due to the short 1250 character limit\n- `start_app_at_login.sh` - adds an app to the Login items to auto-start\n\nHammerspoon code has been moved to its own repo:\n\n[![Readme Card](https://github-readme-stats.vercel.app/api/pin/?username=HariSekhon&repo=Hammerspoon&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Hammerspoon)\n\nSee also [Mac](https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md) page\nin [HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base).\n\n### Monitoring\n\n`monitoring/` directory:\n\n- `dump_stats.sh` - dumps common command outputs to text files in a local tarball. Useful to collect support information\n  for vendor support cases\n- `grafana_api.sh` - queries the [Grafana](https://grafana.com/) API with authentication\n- `log_timestamp_large_intervals.sh` - finds log lines whose timestamp intervals exceed the given number of seconds and\n  outputs those log lines with the difference between the last and current timestamps. Useful to find actions that are\n  taking a long time from log files such as CI/CD logs\n- `prometheus.sh` - starts [Prometheus](https://prometheus.io/) locally, downloading it if not found in `$PATH`\n- `prometheus_docker.sh` - starts [Prometheus](https://prometheus.io/) in Docker using `docker-compose`\n- `prometheus_node_exporter.sh` - starts Prometheus `node_exporter` locally, downloading it if not found in `$PATH`\n- `ssh_dump_stats.sh` - uses SSH and `dump_stats.sh` to dump common command outputs from remote servers to a local\n  tarball. Useful for vendor support cases\n- `ssh_dump_logs.sh` - Uses SSH to dump logs from server to local text files for uploading to vendor support cases\n\nSee doc pages in [HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base) on Grafana,\nPrometheus, OpenTSDB, InfluxDB etc.\n\n### Databases\n\n`mysql/`, `postgres/`, `sql/` and `bin/` directories:\n\n- [sql/](https://github.com/HariSekhon/SQL-scripts) - 100+ SQL scripts for [PostgreSQL](https://www.postgresql.org/), [MySQL](https://www.mysql.com/), [Google BigQuery](https://cloud.google.com/bigquery) and [AWS Athena](https://aws.amazon.com/athena/) [CloudTrail](https://aws.amazon.com/cloudtrail/) logs integration\n- `sqlite.sh` - one-touch [SQLite](https://www.sqlite.org/index.html), starts sqlite3 shell with sample 'chinook' database loaded\n- `mysql*.sh` - [MySQL](https://www.mysql.com/) scripts:\n  - `mysql.sh` - shortens `mysql` command to connect to [MySQL](https://www.mysql.com/) by auto-populating switches from both standard environment variables like `$MYSQL_TCP_PORT`, `$DBI_USER`, `$MYSQL_PWD` (see [doc](https://dev.mysql.com/doc/refman/8.0/en/environment-variables.html)) and other common environment variables like `$MYSQL_HOST` / `$HOST`, `$MYSQL_USER` / `$USER`, `$MYSQL_PASSWORD` / `$PASSWORD`, `$MYSQL_DATABASE` / `$DATABASE`\n  - `mysql_foreach_table.sh` - executes a SQL query against every table, replacing `{db}` and `{table}` in each iteration eg. `select count(*) from {table}`\n  - `mysql_*.sh` - various scripts using `mysql.sh` for row counts, iterating each table, or outputting clean lists of databases and tables for quick scripting\n  - `mysqld.sh` - one-touch [MySQL](https://www.mysql.com/), boots docker container + drops in to `mysql` shell, with `/sql` scripts mounted in container for easy sourcing eg. `source /sql/<name>.sql`. Optionally loads sample 'chinook' database\n  - see also the [SQL Scripts](https://github.com/HariSekhon/SQL-scripts) repo for many more straight MySQL SQL scripts\n- `mariadb.sh` - one-touch [MariaDB](https://mariadb.org/), boots docker container + drops in to `mysql` shell, with `/sql` scripts mounted in container for easy sourcing eg. `source /sql/<name>.sql`. Optionally loads sample 'chinook' database\n- `postgres*.sh` / `psql.sh` - [PostgreSQL](https://www.postgresql.org/) scripts:\n  - `postgres.sh` - one-touch [PostgreSQL](https://www.postgresql.org/), boots docker container + drops in to `psql` shell, with `/sql` scripts mounted in container for easy sourcing eg. `\\i /sql/<name>.sql`. Optionally loads sample 'chinook' database\n  - `psql.sh` - shortens `psql` command to connect to [PostreSQL](https://www.postgresql.org/) by auto-populating switches from environment variables, using both standard postgres supported environment variables like `$PG*` (see [doc](https://www.postgresql.org/docs/12/libpq-envars.html)) as well as other common environment variables like `$POSTGRESQL_HOST` / `$POSTGRES_HOST` / `$HOST`, `$POSTGRESQL_USER` / `$POSTGRES_USER` / `$USER`, `$POSTGRESQL_PASSWORD` / `$POSTGRES_PASSWORD` / `$PASSWORD`, `$POSTGRESQL_DATABASE` / `$POSTGRES_DATABASE` / `$DATABASE`\n  - `postgres_foreach_table.sh` - executes a SQL query against every table, replacing `{db}`, `{schema}` and `{table}` in each iteration eg. `select count(*) from {table}`\n  - `postgres_*.sh` - various scripts using `psql.sh` for row counts, iterating each table, or outputting clean lists of databases, schemas and tables for quick scripting\n  - `checks/check_sqlfluff.sh` - recursively iterates all SQL code files found in the given or current directory and runs SQLFluff linter against them, inferring the different SQL dialects from each path/filename/extension\n\n### AWS - Amazon Web Services\n\n`aws/` directory:\n\n- [AWS](https://aws.amazon.com/) scripts - `aws_*.sh`:\n  - `aws_profile.sh` - switches to an AWS Profile selected from a convenient interactive menu list of AWS profiles from `$AWS_CONFIG_FILE` - useful when you have lots of AWS work profiles\n    - see also [HariSekhon/Environments](https://github.com/HariSekhon/Environments) for automated switching using direnv when `cd`ing into relevant directories\n  - `aws_cli_create_credential.sh` - creates an AWS service account user for CI/CD or CLI with Admin permissions (or other group or policy), creates an AWS Access Key, saves a credentials CSV and even prints the shell export commands and aws credentials file config to configure your environment to start using it. Useful trick to avoid CLI reauth to `aws sso login` every day.\n  - `aws_terraform_create_credential.sh` - creates a AWS terraform service account with Administrator permissions for Terraform Cloud or other CI/CD systems to run Terraform plan and apply, since no CI/CD systems can work with AWS SSO workflows. Stores the access key as both CSV and prints shell export commands and credentials file config as above\n  - `.envrc-aws` - copy to `.envrc` for [direnv](https://direnv.net/) to auto-load AWS configuration settings such as AWS Profile, Compute Region, EKS cluster kubectl context etc.\n    - calls `.envrc-kubernetes` to set the `kubectl` context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global `~/.kube/config` context\n  - `aws_sso_ssh.sh` - launches local AWS SSO authentication pop-up (if not already authenticated), then scp's the latest resultant `~/.aws/sso/cache/` file to the remote server and SSH's there so that you can use AWS CLI or kubectl to EKS remotely on that server easily, without having to copy and paste the token from remote aws sso login to your local web browser\n  - `aws_terraform_create_s3_bucket.sh` - creates a Terraform S3 bucket for storing the backend state, locks out public access, enables versioning, encryption, and locks out Power Users role and optionally any given user/group/role ARNs via a bucket policy for safety\n  - `aws_terraform_create_dynamodb_table.sh` - creates a Terraform locking table in DynamoDB for use with the S3 backend, plus custom IAM policy which can be applied to less privileged accounts\n  - `aws_terraform_create_all.sh` - runs all of the above, plus also applies the custom DynamoDB IAM policy to the user to ensure if the account is less privileged it can still get the Terraform lock (useful for GitHub Actions environment secret for a read only user to generate Terraform Plans in Pull Request without needing approval)\n  - `aws_terraform_iam_grant_s3_dynamodb.sh` - creates IAM policies to access any S3 buckets and DynamoDB tables with `terraform-state` or `tf-state` in their names, and attaches them to the given user. Useful for limited permissions CI/CD accounts that run Terraform Plan eg. in GitHub Actions pull requests\n  - `aws_account_summary.sh` - prints AWS account summary in `key = value` pairs for easy viewing / grepping of things like `AccountMFAEnabled`, `AccountAccessKeysPresent`, useful for checking whether the root account has MFA enabled and no access keys, comparing number of users vs number of MFA devices etc. (see also `check_aws_root_account.py` in [Advanced Nagios Plugins](https://github.com/HariSekhon/Nagios-Plugins))\n  - `aws_billing_alarm.sh` - creates a [CloudWatch](https://aws.amazon.com/cloudwatch/) billing alarm and [SNS](https://aws.amazon.com/sns/) topic with subscription to email you when you incur charges above a given threshold. This is often the first thing you want to do on an account\n  - `aws_budget_alarm.sh` - creates an [AWS Budgets](https://aws.amazon.com/cloudwatch/) billing alarm and [SNS](https://aws.amazon.com/sns/) topic with subscription to email you when both when you start incurring forecasted charges of over 80% of your budget, and 90% actual usage. This is often the first thing you want to do on an account\n  - `aws_batch_stale_jobs.sh` - lists [AWS Batch](https://aws.amazon.com/batch/) jobs that are older than N hours in a given queue\n  - `aws_batch_kill_stale_jobs.sh` - finds and kills [AWS Batch](https://aws.amazon.com/batch/) jobs that are older than N hours in a given queue\n  - `aws_cloudfront_distribution_for_origin.sh` - returns the AWS CloudFront ARN of the distribution which serves origins containing a given substring. Useful for quickly finding the CloudFront ARN needed to give permissions to a private S3 bucket exposed via CloudFront\n  - `aws_cloudtrails_cloudwatch.sh` - lists [Cloud Trails](https://aws.amazon.com/cloudtrail/) and their last delivery to [CloudWatch](https://aws.amazon.com/cloudwatch/features/) Logs (should be recent)\n  - `aws_cloudtrails_event_selectors.sh` - lists [Cloud Trails](https://aws.amazon.com/cloudtrail/) and their event selectors to check each one has at least one event selector\n  - `aws_cloudtrails_s3_accesslogging.sh` - lists [Cloud Trails](https://aws.amazon.com/cloudtrail/) buckets and their Access Logging prefix and target bucket. Checks [S3 access logging](https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html) is enabled\n  - `aws_cloudtrails_s3_kms.sh` - lists [Cloud Trails](https://aws.amazon.com/cloudtrail/) and whether their [S3](https://aws.amazon.com/s3/) buckets are [KMS](https://aws.amazon.com/kms/) secured\n  - `aws_cloudtrails_status.sh` - lists [Cloud Trails](https://aws.amazon.com/cloudtrail/) status - if logging, multi-region and log file validation enabled\n  - `aws_config_all_types.sh` - lists [AWS Config](https://aws.amazon.com/config/) recorders, checking all resource types are supported (should be true) and includes global resources (should be true)\n  - `aws_config_recording.sh` - lists [AWS Config](https://aws.amazon.com/config/) recorders, their recording status (should be true) and their last status (should be success)\n  - `aws_csv_creds.sh` - prints AWS credentials from a CSV file as shell export statements. Useful to quickly switch your shell to some exported credentials from a service account for testing permissions or pipe to upload to a CI/CD system via an API (eg. `jenkins_cred_add*.sh`, `github_actions_repo*_set_secret.sh`, `gitlab_*_set_env_vars.sh`, `circleci_*_set_env_vars.sh`, `bitbucket_*_set_env_vars.sh`, `terraform_cloud_*_set_vars.sh`, `kubectl_kv_to_secret.sh`). Supports new user and new access key csv file formats.\n  - `aws_codecommit_csv_creds.sh` - prints AWS [CodeCommit](https://aws.amazon.com/codecommit/) Git credentials from a CSV file as shell export statements. Similar use case and chaining as above\n  - `aws_ec2_*.sh` - AWS [EC2](https://aws.amazon.com/ec2/) scripts:\n    - `aws_ec2_instance_*.sh` - AWS EC2 [Instance](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Instances.html) scripts:\n      - `aws_ec2_instance_name_to_id.sh` - looks up an EC2 instance ID from an instance name with extra safety checks that only a single instance ID is returned and a reverse lookup on that instance ID to re-verify it matches the name. This level of safety is important when wanting to terminate an EC2 instance by name. If an instance ID is passed, returns it as is for convenience. Used by adjacent scripts\n      - `aws_ec2_instances.sh` - lists AWS EC2 instances, their DNS names and States in an easy to read table output\n      - `aws_ec2_instance_ip.sh` - determines an EC2 instance IP address, trying first for a public IP, or failing that a private IP\n      - `aws_ec2_instance_clone.sh` - clones an AWS EC2 instance by creating an AMI from the original and then booting a new instance from the AMI with the same settings as the original instance. Useful to testing risky things on a separate EC2 instance, such as Server Administrator recovery of Tableau\n      - `aws_ec2_instance_wait_for_ready.sh` - polls an AWS EC2 instance and waits for it to finish initializing to a ready state. Used by adjacent scripts\n      - `aws_ec2_instance_terminate_by_name.sh` - terminate an AWS EC2 instance by name for convenience, resolves its instance ID, verifies unique and then terminates by ID\n    - `aws_ec2_ami*.sh` - AWS EC2 [AMI](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html) scripts:\n      - `aws_ec2_amis.sh` - list AWS EC2 AMIs belonging to your account in an easy to read table output\n      - `aws_ec2_ami_ids.sh` - lists AWS EC2 AMI IDs only, one per line, to be used in adjacent scripts that creating mapping tables and translate AMI IDs to names in inventory scripts `aws_info_ec2*.sh`\n      - `aws_ec2_ami_name_to_id.sh` - looks up an EC2 AMI ID from a name with extra safety checks that only a single AMI ID is returned and a reverse lookup on that AMI ID to re-verify it matches the name\n      - `aws_ec2_ami_boot.sh` - boots a personal EC2 instance of a given AMI for testing\n      - `aws_ec2_ami_boot_ssh.sh` - boots a personal EC2 instance of a given AMI, determines the public or private IP, and drops you into an SSH shell\n      - `aws_ec2_ami_create_from_instance.sh` - creates an AWS EC2 AMI from an EC2 instance and waits for it to become available for use\n      - `aws_ec2_ami_share_to_account.sh` - shares an AMI with another AWS account. Can specify AMI by name or id\n    - `aws_ec2_ebs_*.sh` - AWS EC2 [EBS](https://aws.amazon.com/ebs/) scripts:\n      - `aws_ec2_ebs_volumes.sh` - list EC2 instances and their EBS volumes in the current region\n      - `aws_ec2_ebs_create_snapshot_and_wait.sh - creates a snapshot of a given EBS volume ID and waits for it to complete with exponential backoff\n      - `aws_ec2_ebs_resize_and_wait.sh - resizes an EBS volume and waits for it to complete modifying and optionally optimizing with exponential backoff\n      - `aws_ec2_ebs_volumes_unattached.sh` - list an unattached EBS volumes in a table format\n    - `aws_ec2_launch_templates_ami_id.sh` - for each Launch Template lists the AMI ID of the latest version. Useful to check EKS upgrades of node groups via Terragrunt have taken effect\n  - `aws_ecr_*.sh` - AWS [ECR](https://aws.amazon.com/ecr/) docker image management scripts:\n    - `aws_ecr_docker_login.sh` - authenticates Docker to AWS ECR, inferring the ECR registry from the current AWS Account ID and Region\n    - `aws_ecr_docker_build_push.sh` - builds a docker image and pushes it to ECR with not just the `latest` docker tag but also the current Git hashref and Git tags\n    - `aws_ecr_list_repos.sh` - lists ECR repos, and their docker image mutability and whether image scanning is enabled\n    - `aws_ecr_list_tags.sh` - lists all the tags for a given ECR docker image\n    - `aws_ecr_newest_image_tags.sh` - lists the tags for the given ECR docker image with the newest creation date (can use this to determine which image version to tag as `latest`)\n    - `aws_ecr_alternate_tags.sh` - lists all the tags for a given ECR docker `image:tag` (use arg `<image>:latest` to see what version / build hashref / date tag has been tagged as `latest`)\n    - `aws_ecr_tag_image.sh` - tags an ECR image with another tag without pulling and pushing it\n    - `aws_ecr_tag_image_by_digest.sh` - same as above but tags an ECR image found via digest (more accurate as reference by existing tag can be a moving target). Useful to recover images that have become untagged\n    - `aws_ecr_tag_latest.sh` - tags a given ECR docker `image:tag` as `latest` without pulling or pushing the docker image\n    - `aws_ecr_tag_branch.sh` - tags a given ECR `image:tag` with the current Git branch without pulling or pushing the docker image\n    - `aws_ecr_tag_datetime.sh` - tags a given ECR docker image with its creation date and UTC timestamp (when it was uploaded to ECR) without pulling or pushing the docker image\n    - `aws_ecr_tag_newest_image_as_latest.sh` - finds and tags the newest build of a given ECR docker image as `latest` without pulling or pushing the docker image\n    - `aws_ecr_tags_timestamps.sh` - lists all the tags and their timestamps for a given ECR docker image\n    - `aws_ecr_tags_old.sh` - lists tags older than N days for a given ECR docker image\n    - `aws_ecr_delete_old_tags.sh` - deletes tags older than N days for a given ECR docker image. Lists the image:tags to be deleted and prompts for confirmation safety\n  - `aws_emr_clusters_last_steps.sh` - shows the last N steps executed on each EMR cluster and their EndTime to find idle clusters that should be removed. Also checks CloudWatch for number of steps running within the last few months to catch directly submitted jobs such as Spark, Hive, Glue or Athena which won't show up in the native steps list\n  - `aws_foreach_profile.sh` - executes a templated command across all AWS named profiles configured in AWS CLIv2, replacing `{profile}` in each iteration. Combine with other scripts for powerful functionality, auditing, setup etc. eg. `aws_kube_creds.sh` to configure `kubectl` config to all EKS clusters in all environments\n  - `aws_foreach_region.sh` - executes a templated command against each AWS region enabled for the current account, replacing `{region}` in each iteration. Combine with AWS CLI or scripts to find resources across regions\n  - `aws_iam_*.sh` - AWS [IAM](https://aws.amazon.com/iam/) scripts:\n    - `aws_iam_password_policy.sh` - prints [AWS password policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html) in `key = value` pairs for easy viewing / grepping (used by `aws_harden_password_policy.sh` before and after to show the differences)\n    - `aws_iam_harden_password_policy.sh` - strengthens [AWS password policy](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_passwords_account-policy.html) according to [CIS Foundations Benchmark](https://d1.awsstatic.com/whitepapers/compliance/AWS_CIS_Foundations_Benchmark.pdf) recommendations\n    - `aws_iam_replace_access_key.sh` - replaces the non-current IAM access key (Inactive, Not Used, longer time since used, or an explicitly given key), outputting the new key as shell export statements (useful for piping to the same tools listed for `aws_csv_creds.sh` above)\n    - `aws_iam_policies_attached_to_users.sh` - finds [AWS IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage.html) directly attached to users (anti-best practice) instead of groups\n    - `aws_iam_policies_granting_full_access.sh` - finds [AWS IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage.html) granting full access (anti-best practice)\n    - `aws_iam_policies_unattached.sh` - lists unattached [AWS IAM policies](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_manage.html)\n    - `aws_iam_policy_attachments.sh` - finds all users, groups and roles where a given IAM policy is attached, so that you can remove all these references in your Terraform code and avoid this error `Error: error deleting IAM policy arn:aws:iam::***:policy/mypolicy: DeleteConflict: Cannot delete a policy attached to entities.`\n    - `aws_iam_policy_delete.sh` - deletes an IAM policy, by first handling all prerequisite steps of deleting all prior versions and all detaching all users, groups and roles\n    - `aws_iam_generate_credentials_report_wait.sh` - generates an AWS IAM [credentials report](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_getting-report.html)\n    - `aws_iam_users.sh` - list your IAM users\n    - `aws_iam_users_access_key_age.sh` - prints AWS users [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) status and age (see also `aws_users_access_key_age.py` in [DevOps Python tools](https://github.com/HariSekhon/DevOps-Python-tools) which can filter by age and status)\n    - `aws_iam_users_access_key_age_report.sh` - prints AWS users [access key](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) status and age using a bulk credentials report (faster for many users)\n    - `aws_iam_users_access_key_last_used.sh` - prints AWS users [access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) last used date\n    - `aws_iam_users_access_key_last_used_report.sh` - same as above using bulk credentials report (faster for many users)\n    - `aws_iam_users_last_used_report.sh` - lists AWS users password/access keys last used dates\n    - `aws_iam_users_mfa_active_report.sh` - lists AWS users password enabled and [MFA](https://aws.amazon.com/iam/features/mfa/) enabled status\n    - `aws_iam_users_without_mfa.sh` - lists AWS users with password enabled but no MFA\n    - `aws_iam_users_mfa_serials.sh` - lists AWS users [MFA](https://aws.amazon.com/iam/features/mfa/) serial numbers (differentiates Virtual vs Hardware MFAs)\n    - `aws_iam_users_pw_last_used.sh` - lists AWS users and their password last used date\n  - `aws_ip_ranges.sh` - get all AWS IP ranges for a given Region and/or Service using the IP range API\n  - `aws_info*.sh`:\n    - `aws_info_all_profiles.sh` - calls `aws_info.sh` for all AWS profiles using `aws_foreach_profile.sh`\n      - `aws_info.sh` - lists AWS deployed resources in the current or specified AWS account profile\n        - `aws_info_ec2.sh` - lists AWS EC2 Instances resources deployed in the current AWS account\n    - `aws_info_ec2_csv.sh` - lists AWS EC2 Instances in quoted CSV format in the current AWS account\n    - `aws_info_ec2_all_profiles_csv.sh` - lists AWS EC2 Instances in quoted CSV format across all configured AWS profiles for their configured region\n  - `aws_eks_cloudwatch_logs.sh` - enables and fetches AWS EKS Master logs via CloudWatch\n  - `aws_eks_ssh_dump_logs.sh` - fetch system logs from EKS Worker Nodes EC2 VMs (eg. for support debug requests by vendors)\n  - `aws_eks_cluster_versions.sh` - iterates EKS clusters to list each AWS EKS cluster name and version in the current account. Combine with `aws_foreach_profile.sh` and `aws_foreach_region.sh` to audit your EKS cluster versions across accounts and regions\n  - `aws_eks_addon_versions.sh` - lists the EKS addon versions available for the given cluster by checking its version before checking addons\n  - `aws_eks_available_ips.sh` - lists the number of available IP addresses in the EKS subnets for the given cluster (5 required for an EKS upgrade)\n  - `aws_eks_ami_create.sh` - creates a custom EKS AMI quickly off the base EKS template and then running a shell script in it before saving it to a new AMI. See also [HariSekhon/Packer](https://github.com/HariSekhon/Packer) for more advanced build\n  - `aws_kms_key_rotation_enabled.sh` - lists [AWS KMS](https://aws.amazon.com/kms/) keys and whether they have key rotation enabled\n  - `aws_kube_creds.sh` - auto-loads all [AWS EKS](https://aws.amazon.com/eks/) clusters credentials in the current --profile and --region so your kubectl is ready to rock on AWS\n  - `aws_kubectl.sh` - runs kubectl commands safely fixed to a given [AWS EKS](https://aws.amazon.com/eks/) cluster using config isolation to avoid concurrency race conditions\n  - `aws_logs_*.sh` - some useful log queries in last N hours (24 hours by default):\n    - `aws_logs_batch_jobs.sh` - lists AWS Batch job submission requests and their callers\n    - `aws_logs_ec2_spot.sh` - lists AWS EC2 Spot fleet creation requests, their caller and first tag value for origin hint\n    - `aws_logs_ecs_tasks.sh` - lists AWS ECS task run requests, their callers and job definitions\n  - `aws_meta.sh` - AWS [EC2 Metadata API](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) query shortcut. See also the official [ec2-metadata](https://aws.amazon.com/code/ec2-instance-metadata-query-tool/) shell script with more features\n  - `aws_nat_gateways_public_ips.sh` - lists the public IPs of all NAT gateways. Useful to give to clients to permit through firewalls for webhooks or similar calls\n  - `aws_rds_list.sh` - list RDS instances with select fields - Name, Status, Engine, AZ, Instance Type, Storage\n  - `aws_rds_open_port_to_my_ip.sh` - adds a security group to an RDS DB instance to open its native database SQL port to your public IP address\n  - `aws_rds_get_version.sh` - quickly retrieve the version of an RDS database to know which JDBC jar version to download using `install/download_*_jdbc.sh` when setting up connections\n  - `aws_route53_check_ns_records.sh` - checks AWS [Route 53](https://aws.amazon.com/route53/) public hosted zones NS servers are delegated in the public DNS hierarchy and that there are no rogue NS servers delegated not matching the Route 53 zone configuration\n  - `aws_sso_accounts.sh` - lists all AWS SSO accounts the current SSO user has access to\n  - `aws_sso_configs.sh` - generates AWS SSO configs for all AWS SSO accounts the currently logged in SSO user has access to\n  - `aws_sso_configs_save.sh` - saves AWS SSO configs generated by `aws_sso_configs.sh` to `~/.aws/config` if they're not already found\n  - `aws_sso_account_id_names.sh` - parses AWS config for AWS SSO and outputs Account IDs and Profile names\n  - `aws_sso_config_duplicate_sections.sh` - lists duplicate AWS SSO config sections that are using the same sso_account_id. Useful to deduplicate configs containing a mix of hand crafted and automatically generated `aws_sso_configs.sh`\n  - `aws_sso_config_duplicate_profile_names.sh` - lists duplicate AWS SSO config profile names that are using the same sso_account_id\n  - `aws_accounts_missing_from_config.sh` - for a list of AWS Account IDs in stdin or files, finds those missing from AWS config\n  - `aws_sso_accounts_missing_from_list.sh` - for a list of AWS Account IDs in stdin or files, finds AWS SSO accounts in AWS config missing from the provided list\n  - `aws_sso_env_creds.sh` - retrieves AWS SSO session credentials in the format of environment export commands for copying to other systems like Terraform Cloud\n  - `aws_sso_role_arn.sh` - determines the currently authenticated AWS SSO user's base role ARN in IAM policy usable format\n  - `aws_sso_role_arns.sh` - prints all AWS SSO role ARNs in IAM policy usable format\n  - `aws_profile_config_add_if_missing.sh` - reads AWS profile config blocks from stdin and appends them to the `~/.aws/config` file if the profile section is not found\n  - `aws_profile_generate_direnvs.sh` - generates subdirectories containing the `config.ini` and `.envrc` for every AWS profile found in the given file or `$AWS_CONFIG_FILE` or `~/.aws/config`. Useful to take a large generated AWS `config.ini` from `aws_sso_configs.sh` and then split it into subdirectories for direnvs\n  - `aws_s3_bucket.sh` - creates an S3 bucket, blocks public access, enables versioning, encryption, and optionally locks out any given user/group/role ARNs via a bucket policy for safety (eg. to stop Power Users accessing a sensitive bucket like Terraform state)\n  - `aws_s3_buckets_block_public_access.sh` - blocks public access to one or more given S3 buckets or files containing bucket names, one per line\n  - `aws_s3_account_block_public_access.sh` - blocks S3 public access at the AWS account level\n  - `aws_s3_check_buckets_public_blocked.sh` - iterates each S3 bucket and checks it has public access fully blocked via policy. Parallelized for speedup\n  - `aws_s3_check_account_public_blocked.sh` - checks S3 public access is blocked at the AWS account level\n  - `aws_s3_sync.sh` - syncs multiple AWS S3 URLs from file lists. Validates S3 URLs, source and destination list lengths matches, and optionally that path suffixes match, to prevent off-by-one human errors spraying data all over the wrong destination paths\n  - `aws_s3_access_logging.sh` - lists [AWS S3](https://aws.amazon.com/s3/) buckets and their [access logging](https://docs.aws.amazon.com/AmazonS3/latest/dev/ServerLogs.html) status\n  - `aws_s3_delete_bucket_with_versions.sh` - deletes a bucket including all versions. Use with caution!\n  - `aws_spot_when_terminated.sh` - executes commands when the [AWS EC2](https://aws.amazon.com/ec2/) instance running this script is notified of [Spot Termination](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-spot-instances.html), acts as a latch mechanism that can be set any time after boot\n  - `aws_sqs_check.sh` - sends a test message to an [AWS SQS](https://aws.amazon.com/sqs/) queue, retrieves it to check and then deletes it via the receipt handle id\n  - `aws_sqs_delete_message.sh` - deletes 1-10 messages from a given [AWS SQS](https://aws.amazon.com/sqs/) queue (to help clear out test messages)\n  - `aws_ssm_put_param.sh` - reads a value from a command line argument or non-echo prompt and saves it to AWS [Systems Manager Parameter Store](https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html). Useful for uploading a password without exposing it on your screen\n  - `aws_secret*.sh` - AWS [Secrets Manager](https://aws.amazon.com/secrets-manager/) scripts:\n    - `aws_secret_list.sh` - returns the list of secrets, one per line\n    - `aws_secret_add.sh` - reads a value from a command line argument or non-echo prompt and saves it to Secrets Manager. Useful for uploading a password without exposing it on your screen\n    - `aws_secret_add_binary.sh` - base64 encodes a given file's contents and saves it to Secrets Manager as a binary secret. Useful for uploading things like QR code screenshots for sharing MFA to recovery admin accounts\n    - `aws_secret_update.sh` - reads a value from a command line argument or non-echo prompt and updates a given Secrets Manager secret. Useful for updating a password without exposing it on your screen\n    - `aws_secret_update_binary.sh` - base64 encodes a given file's contents and updates a given Secrets Manager secret. Useful for updating a QR code screenshot for a root account\n    - `aws_secret_get.sh` - gets a secret value for a given secret from Secrets Manager, retrieving either a secure string or secure binary depending on which is available\n  - `eksctl_cluster.sh` - downloads [eksctl](https://eksctl.io/) and creates an [AWS EKS](https://aws.amazon.com/eks/) Kubernetes cluster\n\nSee also [Knowledge Base notes for AWS](https://github.com/HariSekhon/Knowledge-Base/blob/main/aws.md).\n\n### GCP - Google Cloud Platform\n\n`gcp/` directory:\n\n- [Google Cloud](https://cloud.google.com/) scripts - `gcp_*.sh` / `gce_*.sh` / `gke_*.sh` / `gcr_*.sh` / `bigquery_*.sh`:\n  - `.envrc-gcp` - copy to `.envrc` for [direnv](https://direnv.net/) to auto-load GCP configuration settings such as Project, Region, Zone, GKE cluster kubectl context or any other GCloud SDK settings to shorten `gcloud` commands. Applies to the local shell environment only to avoid race conditions caused by naively changing the global gcloud config at `~/.config/gcloud/active_config`\n    - calls `.envrc-kubernetes` to set the `kubectl` context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global `~/.kube/config` context\n  - `gcp_terraform_create_credential.sh` - creates a service account for [Terraform](https://www.terraform.io/) with full permissions, creates and downloads a credential key json and even prints the `export GOOGLE_CREDENTIALS` command to configure your environment to start using Terraform immediately. Run once for each project and combine with [direnv](https://direnv.net/) for fast easy management of multiple GCP projects\n  - `gcp_ansible_create_credential.sh` - creates an [Ansible](https://www.ansible.com/) service account with permissions on the current project, creates and downloads a credential key json and prints the environment variable to immediately use it\n  - `gcp_cli_create_credential.sh` - creates a GCloud SDK CLI service account with full owner permissions to all projects, creates and downloads a credential key json and even prints the `export GOOGLE_CREDENTIALS` command to configure your environment to start using it. Avoids having to reauth to `gcloud auth login` every day.\n  - `gcp_spinnaker_create_credential.sh` - creates a [Spinnaker](https://spinnaker.io/) service account with permissions on the current project, creates and downloads a credential key json and even prints the Halyard CLI configuration commands to use it\n  - `gcp_info.sh` - huge [Google Cloud](https://cloud.google.com/) inventory of deployed resources within the current project - Cloud SDK info plus all of the following (detects which services are enabled to query):\n    - `gcp_info_compute.sh` - [GCE](https://cloud.google.com/compute/) Virtual Machine instances, [App Engine](https://cloud.google.com/appengine) instances, [Cloud Functions](https://cloud.google.com/functions), [GKE](https://cloud.google.com/kubernetes-engine) clusters, all [Kubernetes](https://kubernetes.io/) objects across all GKE clusters (see `kubernetes_info.sh` below for more details)\n    - `gcp_info_storage.sh` - [Cloud SQL](https://cloud.google.com/sql) info below, plus: [Cloud Storage](https://cloud.google.com/storage) Buckets, [Cloud Filestore](https://cloud.google.com/filestore), [Cloud Memorystore Redis](https://cloud.google.com/memorystore), [BigTable](https://cloud.google.com/bigtable) clusters and instances, [Datastore](https://cloud.google.com/datastore) indexes\n    - `gcp_info_cloud_sql.sh` - [Cloud SQL](https://cloud.google.com/sql) instances, whether their backups are enabled, and all databases on each instance\n      - `gcp_info_cloud_sql_databases.sh` - lists databases inside each [Cloud SQL](https://cloud.google.com/sql) instance. Included in `gcp_info_cloud_sql.sh`\n      - `gcp_info_cloud_sql_backups.sh` - lists backups for each [Cloud SQL](https://cloud.google.com/sql) instance with their dates and status. Not included in `gcp_info_cloud_sql.sh` for brevity. See also `gcp_sql_export.sh` further down for more durable backups to [GCS](https://cloud.google.com/storage)\n      - `gcp_info_cloud_sql_users.sh` - lists users for each running [Cloud SQL](https://cloud.google.com/sql) instance. Not included in `gcp_info_cloud_sql.sh` for brevity but useful to audit users\n    - `gcp_info_networking.sh` - VPC Networks, Addresses, Proxies, Subnets, Routers, Routes, VPN Gateways, VPN Tunnels, Reservations, Firewall rules, Forwarding rules, [Cloud DNS](https://cloud.google.com/dns) managed zones and verified domains\n    - `gcp_info_bigdata.sh` - [Dataproc](https://cloud.google.com/dataproc) clusters and jobs in all regions, [Dataflow](https://cloud.google.com/dataflow) jobs in all regions, [PubSub](https://cloud.google.com/pubsub) messaging topics, [Cloud IOT](https://cloud.google.com/iot-core) registries in all regions\n    - `gcp_info_tools.sh` - [Cloud Source Repositories](https://cloud.google.com/source-repositories), [Cloud Builds](https://cloud.google.com/cloud-build), [Container Registry](https://cloud.google.com/container-registry) images across all major repos (`gcr.io`, `us.gcr.io`, `eu.gcr.io`, `asia.gcr.io`), [Deployment Manager](https://cloud.google.com/deployment-manager) deployments\n    - `gcp_info_auth_config.sh` - Auth Configurations, Organizations & Current Config\n    - `gcp_info_projects.sh` - Projects names and IDs\n    - `gcp_info_services.sh` - Services & APIs enabled\n      - `gcp_service_apis.sh` - lists all available [GCP](https://cloud.google.com/) Services, APIs and their states (enabled/disabled), and provides `is_service_enabled()` function used throughout the adjacent scripts to avoid errors and only show relevant enabled services\n    - `gcp_info_accounts_secrets.sh` - [IAM](https://cloud.google.com/iam) Service Accounts, [Secret Manager](https://cloud.google.com/secret-manager) secrets\n  - `gcp_info_all_projects.sh` - same as above but for all detected projects\n  - `gcp_foreach_project.sh` - executes a templated command across all GCP projects, replacing `{project_id}` and `{project_name}` in each iteration (used by `gcp_info_all_projects.sh` to call `gcp_info.sh`)\n  - `gcp_find_orphaned_disks.sh` - lists orphaned disks across one or more GCP projects (not attached to any compute instance)\n  - `gcp_secret*.sh` - Google [Secret Manager](https://cloud.google.com/secret-manager) scripts:\n    - `gcp_secret_add.sh` - reads a value from a command line argument or non-echo prompt and saves it to GCP Secrets Manager. Useful for uploading a password without exposing it on your screen\n    - `gcp_secret_add_binary.sh` - uploads a binary file to GCP Secrets Manager by base64 encoding it first. Useful for uploading QR code screenshots. Useful for uploading things like QR code screenshots for sharing MFA to recovery admin accounts\n    - `gcp_secret_update.sh` - reads a value from a command line argument or non-echo prompt and updates a given GCP Secrets Manager secret. Useful for uploading a password without exposing it on your screen\n    - `gcp_secret_get.sh` - finds the latest version of a given GCP Secret Manager secret and returns its value. Used by adjacent scripts\n    - `gcp_secret_label_k8s.sh` - labels a given existing GCP secret with the current kubectl cluster name and namespace for later use by `gcp_secrets_to_kubernetes.sh`\n    - `gcp_secrets_to_kubernetes.sh` - loads GCP secrets to Kubernetes secrets in a 1-to-1 mapping. Can specify a list of secrets or auto-loads all GCP secrets with labels `kubernetes-cluster` and `kubernetes-namespace` matching the current `kubectl` context (`kcd` to the right namespace first, see `.bash.d/kubernetes`). See also `kubernetes_get_secret_values.sh` to debug the actual values that got loaded. See also [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) / [External Secrets](https://external-secrets.io/) in my [Kubernetes repo](https://github.com/HariSekhon/Kubernetes-configs)\n    - `gcp_secrets_to_kubernetes_multipart.sh` - creates a Kubernetes secret from multiple GCP secrets (used to put `private.pem` and `public.pem` into the same secret to appear as files on volume mounts for apps in pods to use). See also [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) / [External Secrets](https://external-secrets.io/) in my [Kubernetes repo](https://github.com/HariSekhon/Kubernetes-configs)\n    - `gcp_secrets_labels.sh` - lists GCP Secrets and their labels, one per line suitable for quick views or shell pipelines\n    - `gcp_secrets_update_lable.sh` - updates all GCP secrets in current project matching label key=value with a new label value\n    - `gcp_service_account_credential_to_secret.sh` - creates GCP service account and exports a credential key to GCP Secret Manager (useful to stage or combine with `gcp_secrets_to_kubernetes.sh`)\n  - `gke_*.sh` - Google [Kubernetes Engine](https://cloud.google.com/kubernetes-engine) scripts\n    - `gke_kube_creds.sh` - auto-loads all GKE clusters credentials in the current / given / all projects so your kubectl is ready to rock on GCP\n    - `gke_kubectl.sh` - runs kubectl commands safely fixed to a given GKE cluster using config isolation to avoid concurrency race conditions\n    - `gke_firewall_rule_cert_manager.sh` - creates a GCP firewall rule for a given GKE cluster's masters to access [Cert Manager](https://cert-manager.io/) admission webhook (auto-determines the master cidr, network and target tags)\n    - `gke_firewall_rule_kubeseal.sh` - creates a GCP firewall rule for a given GKE cluster's masters to access [Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) controller for `kubeseal` to work (auto-determines the master cidr, network and target tags)\n    - `gke_nodepool_nodes.sh` - lists all nodes in a given nodepool on the current GKE cluster via kubectl labels (fast)\n    - `gke_nodepool_nodes2.sh` - same as above via GCloud SDK (slow, iterates instance groups)\n    - `gke_nodepool_taint.sh` - taints/untaints all nodes in a given GKE nodepool on the current cluster (see `kubectl_node_taints.sh` for a quick way to see taints)\n    - `gke_nodepool_drain.sh` - drains all nodes in a given nodepool (to decommission or rebuild the node pool, for example with different taints)\n    - `gke_persistent_volumes_disk_mappings.sh` - lists GKE kubernetes persistent volumes to GCP persistent disk names, along with PVC and namespace, useful when investigating, resizing PVs etc.\n  - `gcr_*.sh` - Google [Container Registry](https://cloud.google.com/container-registry) scripts:\n    - `gcr_list_tags.sh` - lists all the tags for a given GCR docker image\n    - `gcr_newest_image_tags.sh` - lists the tags for the given GCR docker image with the newest creation date (can use this to determine which image version to tag as `latest`)\n    - `gcr_alternate_tags.sh` - lists all the tags for a given GCR docker `image:tag` (use arg `<image>:latest` to see what version / build hashref / date tag has been tagged as `latest`)\n    - `gcr_tag_latest.sh` - tags a given GCR docker `image:tag` as `latest` without pulling or pushing the docker image\n    - `gcr_tag_branch.sh` - tags a given GCR docker `image:tag` with the current Git branch without pulling or pushing the docker image\n    - `gcr_tag_datetime.sh` - tags a given GCR docker image with its creation date and UTC timestamp (when it was uploaded or created by [Google Cloud Build](https://cloud.google.com/cloud-build)) without pulling or pushing the docker image\n    - `gcr_tag_newest_image_as_latest.sh` - finds and tags the newest build of a given GCR docker image as `latest` without pulling or pushing the docker image\n    - `gcr_tags_timestamps.sh` - lists all the tags and their timestamps for a given GCR docker image\n    - `gcr_tags_old.sh` - lists tags older than N days for a given GCR docker image\n    - `gcr_delete_old_tags.sh` - deletes tags older than N days for a given GCR docker image. Lists the image:tags to be deleted and prompts for confirmation safety\n    - see also [cloudbuild.yaml](https://github.com/HariSekhon/Templates/blob/master/cloudbuild.yaml) in the [Templates](https://github.com/HariSekhon/Templates) repo\n  - CI/CD on GCP - trigger Google Cloud Build and GKE Kubernetes deployments from orthogonal CI/CD systems like Jenkins / TeamCity:\n    - `gcp_ci_build.sh` - script template for CI/CD to trigger Google Cloud Build to build docker container image with extra datetime and latest tagging\n    - `gcp_ci_deploy_k8s.sh` - script template for CI/CD to deploy GCR docker image to GKE Kubernetes using Kustomize\n  - `gce_*.sh` - Google [Compute Engine](https://cloud.google.com/compute/) scripts:\n    - `gce_foreach_vm.sh` - run a command for each GCP VM instance matching the given name/ip regex in the current GCP project\n    - `gce_host_ips.sh` - prints the IPs and hostnames of all or a regex match of GCE VMs for use in /etc/hosts\n    - `gce_ssh.sh` - Runs `gcloud compute ssh` to a VM while auto-determining its zone first to override any inherited zone config and make it easier to script iterating through VMs\n    - `gcs_ssh_keyscan.sh` - SSH keyscans all the GCE VMs returned from the above `gce_host_ips.sh` script and adds them to `~/.ssh/known_hosts`\n    - `gce_meta.sh` - simple script to query the GCE metadata API from within Virtual Machines\n    - `gce_when_preempted.sh` - GCE VM preemption latch script - can be executed any time to set one or more commands to execute upon preemption\n    - `gce_is_preempted.sh` - GCE VM return true/false if preempted, callable from other scripts\n    - `gce_instance_service_accounts.sh` - lists GCE VM instance names and their service accounts\n  - `gcp_firewall_disable_default_rules.sh` - disables those lax GCP default network \"allow all\" firewall rules\n  - `gcp_firewall_risky_rules.sh` - lists risky GCP firewall rules that are enabled and allow traffic from 0.0.0.0/0\n  - `gcp_sql_*.sh` - [Cloud SQL](https://cloud.google.com/sql) scripts:\n    - `gcp_sql_backup.sh` - creates Cloud SQL backups\n    - `gcp_sql_export.sh` - creates Cloud SQL exports to [GCS](https://cloud.google.com/storage)\n    - `gcp_sql_enable_automated_backups.sh` - enable automated daily Cloud SQL  backups\n    - `gcp_sql_enable_point_in_time_recovery.sh` - enable point-in-time recovery with write-ahead logs\n    - `gcp_sql_proxy.sh` - boots a [Cloud SQL Proxy](https://cloud.google.com/sql/docs/postgres/sql-proxy) to all Cloud SQL instances for fast convenient direct `psql` / `mysql` access via local sockets. Installs Cloud SQL Proxy if necessary\n    - `gcp_sql_running_primaries.sh` - lists primary running Cloud SQL instances\n    - `gcp_sql_service_accounts.sh` - lists Cloud SQL instance service accounts. Useful for copying to [IAM](https://cloud.google.com/iam) to grant permissions (eg. Storage Object Creator for SQL export backups to GCS)\n    - `gcp_sql_create_readonly_service_account.sh` - creates a service account with read-only permissions to Cloud SQL eg. to run export backups to GCS\n    - `gcp_sql_grant_instances_gcs_object_creator.sh` - grants minimal GCS objectCreator permission on a bucket to primary Cloud SQL instances for exports\n  - `gcp_cloud_schedule_sql_exports.sh` - creates Google [Cloud Scheduler](https://cloud.google.com/scheduler) jobs to trigger a [Cloud Function](https://cloud.google.com/functions) via [PubSub](https://cloud.google.com/pubsub) to run [Cloud SQL](https://cloud.google.com/sql) exports to [GCS](https://cloud.google.com/storage) for all [Cloud SQL](https://cloud.google.com/sql) instances in the current GCP project\n    - the Python [GCF](https://cloud.google.com/functions) function is in the [DevOps Python tools](https://github.com/HariSekhon/DevOps-Python-tools) repo\n  - `bigquery_*.sh` - [BigQuery](https://cloud.google.com/bigquery) scripts:\n    - `bigquery_list_datasets.sh` - lists BigQuery datasets in the current GCP project\n    - `bigquery_list_tables.sh` - lists BigQuery tables in a given dataset\n    - `bigquery_list_tables_all_datasets.sh` - lists tables for all datasets in the current GCP project\n    - `bigquery_foreach_dataset.sh` - executes a templated command for each dataset\n    - `bigquery_foreach_table.sh` - executes a templated command for each table in a given dataset\n    - `bigquery_foreach_table_all_datasets.sh` - executes a templated command for each table in each dataset in the current GCP project\n    - `bigquery_table_row_count.sh` - gets the row count for a given table\n    - `bigquery_tables_row_counts.sh` - gets the row counts for all tables in a given dataset\n    - `bigquery_tables_row_counts_all_datasets.sh` - gets the row counts for all tables in all datasets in the current GCP project\n    - `bigquery_generate_query_biggest_tables_across_datasets_by_row_count.sh` - generates a BigQuery SQL query to find the top 10 biggest tables by row count\n    - `bigquery_generate_query_biggest_tables_across_datasets_by_size.sh` - generates a BigQuery SQL query to find the top 10 biggest tables by size\n    - see also the [SQL Scripts](https://github.com/HariSekhon/SQL-scripts) repo for many more straight BigQuery SQL scripts\n  - GCP [IAM](https://cloud.google.com/iam) scripts:\n    - `gcp_service_account*.sh`:\n      - `gcp_service_account_credential_to_secret.sh` - creates GCP service account and exports a credential key to GCP Secret Manager (useful to stage or combine with `gcp_secrets_to_kubernetes.sh`)\n      - `gcp_service_accounts_credential_keys.sh` - lists all service account credential keys and expiry dates, can `grep 9999-12-31T23:59:59Z` to find non-expiring keys\n      - `gcp_service_accounts_credential_keys_age.sh` - lists all service account credential keys age in days\n      - `gcp_service_accounts_credential_keys_expired.sh` - lists expired service account credential keys that should be removed and recreated if needed\n      - `gcp_service_account_members.sh` - lists all members and roles authorized to use any service accounts. Useful for finding GKE Workload Identity mappings\n    - `gcp_iam_*.sh`:\n      - `gcp_iam_roles_in_use.sh` - lists GCP IAM roles in use in the current or all projects\n      - `gcp_iam_identities_in_use.sh` - lists GCP IAM identities (users/groups/serviceAccounts) in use in the current or all projects\n      - `gcp_iam_roles_granted_to_identity.sh` - lists GCP IAM roles granted to identities matching the regex (users/groups/serviceAccounts) in the current or all projects\n      - `gcp_iam_roles_granted_too_widely.sh` - lists GCP IAM roles which have been granted to allAuthenticatedUsers or even worse allUsers (unauthenticated) in one or all projects\n      - `gcp_iam_roles_with_direct_user_grants.sh` - lists GCP IAM roles which have been granted directly to users in violation of best-practice group-based management\n      - `gcp_iam_serviceaccount_members.sh` - lists members with permissions to use each GCP service account\n      - `gcp_iam_serviceaccounts_without_permissions.sh` - finds service accounts without IAM permissionns, useful to detect obsolete service accounts after a 90 day unused permissions clean out\n      - `gcp_iam_workload_identities.sh` - lists GKE Workload Identity integrations, uses `gcp_iam_serviceaccount_members.sh`\n      - `gcp_iam_users_granted_directly.sh` - lists GCP IAM users which have been granted roles directly in violation of best-practice group-based management\n  - `gcs_bucket_project.sh` - finds the GCP project that a given bucket belongs to using the GCP Storage API\n  - `gcs_curl_file.sh` - retrieves a GCS file's contents from a given bucket and path using the GCP Storage API. Useful for starting shell pipelines or being called from other scripts\n  - `firebase_foreach_project.sh` - executes a templated command across all Firebase projects, replacing `{project_id}`, `{project_number}` and `{project_name}` in each iteration\n\nSee also [Knowledge Base notes for GCP](https://github.com/HariSekhon/Knowledge-Base/blob/main/gcp.md).\n\n### Kubernetes\n\n`kubernetes/` directory:\n\n- `.envrc-kubernetes` - copy to `.envrc` for [direnv](https://direnv.net/) to auto-load the right Kubernetes `kubectl` context isolated to current shell to prevent race conditions between shells and scripts caused by otherwise naively changing the global `~/.kube/config` context\n- `aws/eksctl_cluster.sh` - quickly spins up an [AWS EKS](https://aws.amazon.com/eks/) cluster using `eksctl` with some sensible defaults\n- `kubernetes_info.sh` - huge [Kubernetes](https://kubernetes.io/) inventory listing of deployed resources across all namespaces in the current cluster / kube context:\n  - cluster-info\n  - master component statuses\n  - nodes\n  - namespaces\n  - deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers\n  - storage classes, persistent volumes, persistent volume claims\n  - service accounts, resource quotas, network policies, pod security policies\n  - container images running\n  - container images running counts descending\n  - pods  (might be too much detail if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments)\n- `kubectl.sh` - runs kubectl commands safely fixed to a given context using config isolation to avoid concurrency race conditions\n- `kubectl_diff_apply.sh` - generates a kubectl diff and prompts to apply\n- `kustomize_diff_apply.sh` - runs Kustomize build, precreates any namespaces, shows a kubectl diff of the proposed changes, and prompts to apply\n- `kustomize_diff_branch.sh` - runs Kustomize build against the current and target base branch for current or all given directories, then shows the diff for each directory. Useful to detect differences when refactoring, such as switching to tagged bases\n- `kubectl_create_namespaces.sh` - creates any namespaces in yaml files or stdin, a prerequisite for a diff on a blank install, used by adjacent scripts for safety\n- `kubernetes_check_objects_namespaced.sh` - checks Kubernetes yaml(s) for objects which aren't explicitly namespaced, which can easily result in deployments to the wrong namespace. Reads the API resources from your current Kubernetes cluster and if successful excludes cluster-wide objects\n- `kustomize_check_objects_namespaced.sh` - checks Kustomize build yaml output for objects which aren't explicitly namespaced (uses above script)\n- `kubectl_deployment_pods.sh` - gets the pod names with their unpredictable suffixes for a given deployment by querying the deployment's selector labels and then querying pods that match those labels\n- `kubectl_get_all.sh` - finds all namespaced Kubernetes objects and requests them for the current or given namespace. Useful because `kubectl get all` misses a lof of object types\n- `kubectl_get_annotation.sh` - find a type of object with a given annotation\n- `kubectl_restart.sh` - restarts all or filtered deployments/statefulsets in the current or given namespace. Useful when debugging or clearing application problems\n- `kubectl_logs.sh` - tails all containers in all pods or filtered pods in the current or given namespace. Useful when debugging a distributed set of pods in live testing\n- `kubectl_kv_to_secret.sh` - creates a Kuberbetes secret from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n- `kubectl_secret_values.sh` - prints the keys and base64 decoded values within a given Kubernetes secret for quick debugging of Kubernetes secrets. See also: `gcp_secrets_to_kubernetes.sh`\n- `kubectl_secrets_download.sh` - downloads all secrets in current or given namespace to local files of the same name, useful as a backup before migrating to Sealed Secrets\n- `kubernetes_secrets_compare_gcp_secret_manager.sh` - compares each Kubernetes secret to the corresponding secret in GCP Secret Manager. Useful to safety check GCP Secret Manager values align before enabling [External Secrets](https://external-secrets.io/latest/) to replace them\n- `kubernetes_secret_to_external_secret.sh` - generates an [External Secret](https://external-secrets.io/latest/) from an existing Kubernetes secret\n- `kubernetes_secrets_to_external_secrets.sh` - generates [External Secrets](https://external-secrets.io/latest/) from all existing Kubernetes secrets found in the current or given namespace\n- `kubernetes_secret_to_sealed_secret.sh` - generates a [Bitnami Sealed Secret](https://github.com/bitnami-labs/sealed-secrets) from an existing Kubernetes secret\n- `kubernetes_secrets_to_sealed_secrets.sh` - generates [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets) from all existing Kubernetes secrets found in the current or given namespace\n- `kubectl_secrets_annotate_to_be_sealed.sh` - annotates secrets in current or given namespace to allow being overwritten by Sealed Secrets (useful to sync ArgoCD health)\n- `kubectl_secrets_not_sealed.sh` - finds secrets with no SealedSecret ownerReferences\n- `kubectl_secrets_to_be_sealed.sh` - finds secrets pending overwrite by Sealed Secrets with the managed annotation\n- `kubernetes_yaml_strip_live_fields.sh` - strips live fields from Kubernetes yaml object dumps. Useful so you can do `kubectl diff` or `kubectl apply` without hitting annoying errors about immutable fields left in exports from `kubectl get ... -o yaml`\n- `kubernetes_foreach_context.sh` - executes a command across all kubectl contexts, replacing `{context}` in each iteration (skips lab contexts `docker` / `minikube` / `minishift` to avoid hangs since they're often offline)\n- `kubernetes_foreach_namespace.sh` - executes a command across all kubernetes namespaces in the current cluster context, replacing `{namespace}` in each iteration\n  - Can be chained with `kubernetes_foreach_context.sh` and useful when combined with `gcp_secrets_to_kubernetes.sh` to load all secrets from GCP to Kubernetes for the current cluster, or combined with `gke_kube_creds.sh` and `kubernetes_foreach_context.sh` for all clusters!\n- `kubernetes_api.sh` - finds Kubernetes API and runs your curl arguments against it, auto-getting authorization token and auto-populating OAuth authentication header\n- `kubernetes_autoscaler_release.sh` - finds the latest Kubernetes Autoscaler release that matches your local Kubernetes cluster version using kubectl and the GitHub API. Useful for quickly finding the image override version for `eks-cluster-autoscaler-kustomization.yaml` in the [Kubernetes configs](https://github.com/HariSekhon/Kubernetes-configs) repo\n- `kubernetes_etcd_backup.sh` - creates a timestamped backup of the Kubernetes Etcd database for a kubeadm cluster\n- `kubernetes_delete_stuck_namespace.sh` - to forcibly delete those pesky kubernetes namespaces of 3rd party apps like Knative that get stuck and hang indefinitely on the finalizers during deletion\n- `kubeadm_join_cmd.sh` - outputs `kubeadm join` command (generates new token) to join an existing Kubernetes cluster (used in [vagrant kubernetes](https://github.com/HariSekhon/DevOps-Bash-tools/tree/master/vagrant/kubernetes) provisioning scripts)\n- `kubeadm_join_cmd2.sh` - outputs `kubeadm join` command manually (calculates cert hash + generates new token) to join an existing Kubernetes cluster\n- `kubernetes_nodes_ssh_dump_logs.sh` - fetch logs from Kubernetes nodes (eg. for support debug requests by vendors)\n- `kubectl_exec.sh` - finds and execs to the first Kubernetes pod matching the given name regex, optionally specifying the container name regex to exec to, and shows the full generated `kubectl exec` command line for clarity\n- `kubectl_exec2.sh` - finds and execs to the first Kubernetes pod matching given pod filters, optionally specifying the container to exec to, and shows the full generated `kubectl exec` command line for clarity\n- `kubectl_pods_per_node.sh` - lists number of pods per node sorted descending\n- `kubectl_pods_important.sh` - lists important pods and their nodes to check on scheduling\n- `kubectl_pods_colocated.sh` - lists pods from deployments/statefulsets that are colocated on the same node\n- `kubectl_node_labels.sh` - lists nodes and their labels, one per line, easier to read visually or pipe in scripting\n- `kubectl_pods_running_with_labels.sh` - lists running pods with labels matching key=value pair arguments\n- `kubectl_node_taints.sh` - lists nodes and their taints\n- `kubectl_jobs_stuck.sh` - finds Kubernetes jobs stuck for hours or days with no completions\n- `kubectl_jobs_delete_stuck.sh` - prompts for confirmation to delete stuck Kubernetes jobs found by script above\n- `kubectl_images.sh` - lists Kubernetes container images running on the current cluster\n- `kubectl_image_counts.sh` - lists Kubernetes container images running counts sorted descending\n- `kubectl_image_deployments.sh` - lists which deployments, statefulsets or daemonsets container images belong to. Useful to find which deployment, statefulset or daemonset to upgrade to replace a container image eg. when replacing deprecated the k8s.gcr.io registry with registry.k8s.io\n- `kubectl_pod_count.sh` - lists Kubernetes pods total running count\n- `kubectl_pod_labels.sh` - lists Kubernetes pods and their labels, one label per line for easier shell script piping for further actions\n- `kubectl_pod_ips.sh` - lists Kubernetes pods and their pod IP addresses\n- `kubectl_container_count.sh` - lists Kubernetes containers total running count\n- `kubectl_container_counts.sh` - lists Kubernetes containers running counts by name sorted descending\n- `kubectl_pods_dump_*.sh` - dump stats / logs / jstacks from all pods matching a given regex and namespace to txt files for support debugging\n  - `kubectl_pods_dump_stats.sh` - dump stats\n  - `kubectl_pods_dump_logs.sh` - dump logs\n  - `kubectl_pods_dump_jstacks.sh` - dump Java jstacks\n  - `kubectl_pods_dump_all.sh` - calls the above `kubectl_pods_dump_*.sh` scripts for N iterations with a given interval\n- `kubectl_empty_namespaces.sh` - finds namespaces without any of the usual objects using `kubectl get all`\n- `kubectl_delete_empty_namespaces.sh` - removes empty namespaces, uses `kubectl_empty_namespaces.sh`\n- `kubectl_<image>.sh` - quick launch one-off pods for interactive debuggging in Kubernetes\n  - `kubectl_alpine.sh`\n  - `kubectl_busybox.sh`\n  - `kubectl_curl.sh`\n  - `kubectl_dnsutils.sh`\n  - `kubectl_gcloud_sdk.sh`\n  - `kubectl_run_sa.sh` - launch a quick pod with the given service account to test private repo pull & other permissions\n- `kubectl_port_forward.sh` - launches `kubectl port-forward` to a given pod's port with an optional label or name filter. If more than one pod is found, prompts with an interactive dialogue to choose one. Optionally automatically opens the forwarded localhost URL in the default browser\n  - `kubectl_port_forward_spark.sh` - does the above for Spark UI\n- `helm_template.sh` - templates a Helm chart for Kustomize deployments\n- `kustomize_parse_helm_charts.sh` - parses the [Helm](https://helm.sh/) charts from one or more `kustomization.yaml` files into TSV format for further shell pipe processing\n- `kustomize_install_helm_charts.sh` - installs the [Helm](https://helm.sh/) charts from one or more `kustomization.yaml` files the old fashioned Helm CLI way so that tools like [Nova](https://github.com/FairwindsOps/nova) can be used to detect outdated charts (used in [Kubernetes-configs](https://github.com/HariSekhon/Kubernetes-configs) repo's [CI](https://github.com/HariSekhon/Kubernetes-configs/actions/workflows/nova.yaml))\n- `kustomize_update_helm_chart_versions.sh` - updates one or more `kustomization.yaml` files to the latest versions of any charts they contain\n- `kustomize_materialize.sh` - recursively materializes all `kustomization.yaml` to `kustomization.materialized.yaml` in the same directories for scanning with tools like [Pluto](https://github.com/FairwindsOps/pluto) to detect deprecated API objects inherited from embedded Helm charts. Parallelized for performance\n- ArgoCD:\n  - `argocd_auto_sync.sh` - toggle Auto-sync on/off to allow repairs and maintenance operation for a given app and also disables / re-enables the App-of-Apps base apps to stop then re-enabling the app\n  - `argocd_apps_sync.sh` - sync's all [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) apps matching an optional ERE regex filter on their names using the ArgoCD CLI\n  - `argocd_apps_wait_sync.sh` - sync's all [ArgoCD](https://argo-cd.readthedocs.io/en/stable/) apps matching an optional ERE regex filter on their names using the ArgoCD CLI's while also checking their health and operation\n  - `argocd_generate_resource_whitelist.sh` - generates a yaml cluster and namespace resource whitelist for ArgoCD project config. If given an existing yaml, will merge in its original whitelists, dedupe, and write them back into the file using an in-place edit. Useful because ArgoCD 2.2+ doesn't show resources that aren't explicitly allowed, such as ReplicaSets and Pods\n- Pluto:\n  - `pluto_detect_helm_materialize.sh` - recursively materializes all helm `Chart.yaml` and runs [Pluto](https://github.com/FairwindsOps/pluto) on each directory to work around [this issue](https://github.com/FairwindsOps/pluto/issues/444)\n  - `pluto_detect_kustomize_materialize.sh` - recursively materializes all `kustomization.yaml` and runs [Pluto](https://github.com/FairwindsOps/pluto) on each directory to work around [this issue](https://github.com/FairwindsOps/pluto/issues/444)\n  - `pluto_detect_kubectl_dump_objects.sh` - dumps all live Kubernetes objects to /tmp and runs [Pluto](https://github.com/FairwindsOps/pluto) to detect deprecated API objects on the cluster from any source\n- Rancher:\n  - `rancher_api.sh` - queries the Rancher API with authentication\n  - `rancher_kube_creds.sh` - downloads all Rancher clusters credentials into subdirectories matching cluster names, with `.envrc` in each, so a quick `cd` into one and your kubectl is ready to rock\n- see also Google Kubernetes Engine scripts in the [GCP - Google Cloud Platform](https://github.com/HariSekhon/DevOps-Bash-tools/#gcp---google-cloud-platform) section above\n- see also the [Kubernetes configs](https://github.com/HariSekhon/Kubernetes-configs) repo\n\nSee also [Knowledge Base notes for Kubernetes](https://github.com/HariSekhon/Knowledge-Base/blob/main/kubernetes.md).\n\n### Docker\n\n`docker/` directory:\n\n- `docker_*.sh` / `dockerhub_*.sh` - [Docker](https://www.docker.com/) / [DockerHub](https://hub.docker.com/) API scripts:\n  - `dockerhub_api.sh` - queries DockerHub API v2 with or without authentication (`$DOCKERHUB_USER` & `$DOCKERHUB_PASSWORD` / `$DOCKERHUB_TOKEN`)\n  - `docker_api.sh` - queries a Docker Registry with optional basic authentication if `$DOCKER_USER` & `$DOCKER_PASSWORD` are set\n  - `docker_build_hashref.sh` - runs `docker build` and auto-generates docker image name and tag from relative Git path and commit short SHA hashref and a dirty sha suffix if git contents are modified. Useful to compare docker image sizes between your clean and modified versions of `Dockerfile` or contents\n  - `docker_package_check.sh` - runs package installs on major versions of a docker image to check given packages are available before adding them and breaking builds across linux distro versions\n  - `docker_registry_list_images.sh` - lists images in a given private Docker Registry\n  - `docker_registry_list_tags.sh` - lists tags for a given image in a private Docker Registry\n  - `docker_registry_get_image_manifest.sh` - gets a given image:tag manifest from a private Docker Registry\n  - `docker_registry_tag_image.sh` - tags a given image with a new tag in a private Docker Registry via the API without pulling and pushing the image data (must faster and more efficient)\n  - `dockerhub_list_tags.sh` - lists tags for a given DockerHub repo. See also [dockerhub_show_tags.py](https://github.com/HariSekhon/DevOps-Python-tools/blob/master/dockerhub_show_tags.py) in the [DevOps Python tools](https://github.com/HariSekhon/DevOps-Python-tools) repo.\n  - `dockerhub_list_tags_by_last_updated.sh` - lists tags for a given DockerHub repo sorted by last updated timestamp descending\n  - `dockerhub_search.sh` - searches with a configurable number of returned items (older docker cli was limited to 25 results)\n  - `clean_caches.sh` - cleans out OS package and programming language caches, call near end of `Dockerfile` to reduce Docker image size\n  - see also the [Dockerfiles](https://github.com/HariSekhon/Dockerfiles) repo\n- `quay_api.sh` - queries the [Quay.io](https://quay.io/) API with OAuth2 authentication token `$QUAY_TOKEN`\n\nSee also [Knowledge Base notes for Docker](https://github.com/HariSekhon/Knowledge-Base/blob/main/docker.md).\n\n### Data\n\n`data/` directory:\n\n- `avro_tools.sh` - runs Avro Tools jar, downloading it if not already present (determines latest version when\n  downloading)\n- `parquet_tools.sh` - runs Parquet Tools jar, downloading it if not already present (determines latest version\n  when downloading)\n- `csv_header_indices.sh` - list CSV headers with their zero indexed numbers, useful reference when coding against\n  column positions\n- `ini_config_add_if_missing.sh` - reads INI config blocks from stdin and appends them to the specified file if the section is not found. Used by `aws_profile_config_add_if_missing.sh`\n- `ini_config_duplicate_sections.sh` - lists duplicate INI config sections that are using the same value for a given key in the given .ini file\n- `ini_config_duplicate_section_names.sh` - lists duplicate INI config section names that are using the same value for a given key in the given .ini file\n- `ini_grep_section.sh` - prints the named section from a given .ini file to stdout\n- `wordcount.sh` - counts and ranks words by their frequency in file(s) or stdin\n- Data format validation `validate_*.py` from [DevOps Python Tools repo](https://github.com/HariSekhon/DevOps-Python-tools):\n\n  - CSV\n  - JSON\n  - [Avro](https://avro.apache.org/)\n  - [Parquet](https://parquet.apache.org/)\n  - INI / Properties files (Java)\n  - LDAP LDIF\n  - XML\n  - YAML\n\n- `json2yaml.sh` - converts JSON to YAML\n- `yaml2json.sh` - converts YAML to JSON - needed for some APIs like GitLab CI linting\n  (see [Gitlab](#git---github-gitlab-bitbucket-azure-devops) section above)\n\n### Big Data & NoSQL\n\n`bigdata/` and `kafka/` directories:\n\n- `kafka_*.sh` - scripts to make [Kafka](http://kafka.apache.org/) CLI usage easier including auto-setting Kerberos to source TGT from environment and auto-populating broker and zookeeper addresses. These are auto-added to the `$PATH` when `.bashrc` is sourced. For something similar for [Solr](https://lucene.apache.org/solr/), see `solr_cli.pl` in the [DevOps Perl Tools](https://github.com/HariSekhon/DevOps-Perl-tools) repo.\n- `zookeeper*.sh` - [Apache ZooKeeper](https://zookeeper.apache.org/) scripts:\n  - `zookeeper_client.sh` - shortens `zookeeper-client` command by auto-populating the zookeeper quorum from the environment variable `$ZOOKEEPERS` or else parsing the zookeeper quorum from `/etc/**/*-site.xml` to make it faster and easier to connect\n  - `zookeeper_shell.sh` - shortens Kafka's `zookeeper-shell` command by auto-populating the zookeeper quorum from the environment variable `$KAFKA_ZOOKEEPERS` and optionally `$KAFKA_ZOOKEEPER_ROOT` to make it faster and easier to connect\n- `hive_*.sh` / `beeline*.sh` - [Apache Hive](https://hive.apache.org/) scripts:\n  - `beeline.sh` - shortens `beeline` command to connect to [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview) by auto-populating Kerberos and SSL settings, zookeepers for HiveServer2 HA discovery if the environment variable `$HIVE_HA` is set or using the `$HIVESERVER_HOST` environment variable so you can connect with no arguments (prompts for HiveServer2 address if you haven't set `$HIVESERVER_HOST` or `$HIVE_HA`)\n    - `beeline_zk.sh` - same as above for [HiveServer2](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Overview) HA by auto-populating SSL and ZooKeeper service discovery settings (specify `$HIVE_ZOOKEEPERS` environment variable to override). Automatically called by `beeline.sh` if either `$HIVE_ZOOKEEPERS` or `$HIVE_HA` is set (the latter parses `hive-site.xml` for the ZooKeeper addresses)\n  - `hive_foreach_table.sh` - executes a SQL query against every table, replacing `{db}` and `{table}` in each iteration eg. `select count(*) from {table}`\n  - `hive_list_databases.sh` - list Hive databases, one per line, suitable for scripting pipelines\n  - `hive_list_tables.sh` - list Hive tables, one per line, suitable for scripting pipelines\n  - `hive_tables_metadata.sh` - lists a given DDL metadata field for each Hive table (to compare tables)\n  - `hive_tables_location.sh` - lists the data location per Hive table (eg. compare external table locations)\n  - `hive_tables_row_counts.sh` - lists the row count per Hive table\n  - `hive_tables_column_counts.sh` - lists the column count per Hive table\n- ` impala*.sh` - [Apache Impala](https://impala.apache.org/) scripts:\n  - `impala_shell.sh` - shortens `impala-shell` command to connect to [Impala](https://impala.apache.org/) by parsing the Hadoop topology map and selecting a random datanode to connect to its Impalad, acting as a cheap CLI load balancer. For a real load balancer see [HAProxy config for Impala](https://github.com/HariSekhon/HAProxy-configs) (and many other Big Data & NoSQL technologies). Optional environment variables `$IMPALA_HOST` (eg. point to an explicit node or an HAProxy load balancer) and `IMPALA_SSL=1` (or use regular impala-shell `--ssl` argument pass through)\n  - `impala_foreach_table.sh` - executes a SQL query against every table, replacing `{db}` and `{table}` in each iteration eg. `select count(*) from {table}`\n  - `impala_list_databases.sh` - list Impala databases, one per line, suitable for scripting pipelines\n  - `impala_list_tables.sh` - list Impala tables, one per line, suitable for scripting pipelines\n  - `impala_tables_metadata.sh` - lists a given DDL metadata field for each Impala table (to compare tables)\n  - `impala_tables_location.sh` - lists the data location per Impala table (eg. compare external table locations)\n  - `impala_tables_row_counts.sh` - lists the row count per Impala table\n  - `impala_tables_column_counts.sh` - lists the column count per Impala table\n- `hdfs_*.sh` - Hadoop [HDFS](https://en.wikipedia.org/wiki/Apache_Hadoop#Hadoop_distributed_file_system) scripts:\n  - `hdfs_checksum*.sh` - walks an HDFS directory tree and outputs HDFS native checksums (faster) or portable externally comparable CRC32, in serial or in parallel to save time\n  - `hdfs_find_replication_factor_1.sh` / `hdfs_set_replication_factor_3.sh` - finds HDFS files with replication factor 1 / sets HDFS files with replication factor <=2 to replication factor 3 to repair replication safety and avoid no replica alarms during maintenance operations (see also Python API version in the [DevOps Python Tools](https://github.com/HariSekhon/DevOps-Python-tools) repo)\n  - `hdfs_file_size.sh` / `hdfs_file_size_including_replicas.sh` - quickly differentiate HDFS files raw size vs total replicated size\n  - `hadoop_random_node.sh` - picks a random Hadoop cluster worker node, like a cheap CLI load balancer, useful in scripts when you want to connect to any worker etc. See also the read [HAProxy Load Balancer configurations](https://github.com/HariSekhon/HAProxy-configs) which focuses on master nodes\n- `cloudera_*.sh` - [Cloudera](https://www.cloudera.com/) scripts:\n  - `cloudera_manager_api.sh` - script to simplify querying [Cloudera Manager](https://www.cloudera.com/products/product-components/cloudera-manager.html) API using environment variables, prompts, authentication and sensible defaults. Built on top of `curl_auth.sh`\n  - `cloudera_manager_impala_queries*.sh` - queries [Cloudera Manager](https://www.cloudera.com/products/product-components/cloudera-manager.html) for recent [Impala](https://impala.apache.org/) queries, failed queries, exceptions, DDL statements, metadata stale errors, metadata refresh calls etc. Built on top of `cloudera_manager_api.sh`\n  - `cloudera_manager_yarn_apps.sh` - queries [Cloudera Manager](https://www.cloudera.com/products/product-components/cloudera-manager.html) for recent [Yarn](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) apps. Built on top of `cloudera_manager_api.sh`\n  - `cloudera_navigator_api.sh` - script to simplify querying [Cloudera Navigator](https://www.cloudera.com/products/product-components/cloudera-navigator.html) API using environment variables, prompts, authentication and sensible defaults. Built on top of `curl_auth.sh`\n  - `cloudera_navigator_audit_logs.sh` - fetches [Cloudera Navigator](https://www.cloudera.com/products/product-components/cloudera-navigator.html) audit logs for given service eg. hive/impala/hdfs via the API, simplifying date handling, authentication and common settings. Built on top of `cloudera_navigator_api.sh`\n  - `cloudera_navigator_audit_logs_download.sh` - downloads [Cloudera Navigator](https://www.cloudera.com/products/product-components/cloudera-navigator.html) audit logs for each service by year. Skips existing logs, deletes partially downloaded logs on failure, generally retry safe (while true, Control-C, not `kill -9` obviously). Built on top of `cloudera_navigator_audit_logs.sh`\n\nSee also [Knowledge Base notes for Hadoop](https://github.com/HariSekhon/Knowledge-Base/blob/main/hadoop.md).\n\n### Git - GitHub, GitLab, Bitbucket, Azure DevOps\n\n`git/`, `github/`, `gitlab/`, `bitbucket/` and `azure_devops/` directories:\n\n- `git/*.sh` - [Git](https://git-scm.com/) scripts:\n  - `precommit_run_changed_files.sh` - runs pre-commit on all files changed on the current branch vs the default branch. Useful to reproduce `pre-commit` checks that are failing in pull requests to get your PRs to pass\n  - `git_diff_commit.sh` - quickly commits added or updated files to Git, showing a diff and easy enter prompt for each file. Super convenient for fast commits on the command line, and in vim and IDEs via hotkeys\n  - `git_review_push.sh` - shows diff of what would be pushed upstream and prompts to push. Convenient for fast reviewed pushes via vim or IDEs hotkeys\n  - `git_branch_delete_squash_merged.sh` - carefully detects if a squash merged branch you want to delete has no changes with the default trunk branch before deleting it.\n     See [Squash Merges](https://github.com/HariSekhon/Knowledge-Base/blob/main/git.md#squash-merges-require-force-deleting-branches) in knowledge-base about why this is necessary.\n  - `git_tag_release.sh` - creates a Git tag, auto-incrementing a `.N` suffix on the year/month/day date format if no exact version given\n  - `git_foreach_branch.sh` - executes a command on all branches (useful in heavily version branched repos like in my [Dockerfiles](https://github.com/HariSekhon/Dockerfiles) repo)\n  - `git_foreach_repo.sh` - executes a command against all adjacent repos from a given repolist (used heavily by many adjacent scripts)\n  - `git_foreach_modified.sh` - executes a command against each file with git modified status\n  - `git_foreach_repo_replace_readme_actions.sh` - updates the `README.md` badges for GitHub Actions to match the local repo name. Useful to bulk fix copied badges quickly and easily\n  - `git_foreach_repo_update_readme.sh` - git-diff-commits the `README.md` for each Git repo checkout using adjacent `git_foreach_repo.sh` and `git_diff_commit.sh` scripts. Useful to quickly bulk update `README.md` in all your projects, such as when references need updating\n  - `git_push_stats.sh` - shows the Git push stats to the remote origin for the current branch - number of commits and lines of diff, using the following `git_origin_*.sh` scripts:\n  - `git_origin_log_to_push.sh` - shows the Git log in local branch that would be pushed to remote origin\n  - `git_origin_files_to_push.sh` - shows the Git files in local branch that would be pushed to remote origin\n  - `git_origin_diff_to_push.sh` - shows the Git diff of lines in local branch that would be pushed to remote origin\n  - `git_origin_commit_count_to_push.sh` - shows the number of Git commits in local branch that would be pushed to remote origin\n  - `git_origin_line_count_to_push.sh` - shows the Git number of lines changed in local branch that would be pushed to remote origin. These are lines actually added / changed / removed without surrounding context lines\n  - `git_merge_all.sh` / `git_merge_master.sh` / `git_merge_master_pull.sh` - merges updates from master branch to all other branches to avoid drift on longer lived feature branches / version branches (eg. [Dockerfiles](https://github.com/HariSekhon/Dockerfiles) repo)\n  - `git_remotes_add_origin_providers.sh` - auto-creates remotes for the 4 major public repositories ([GitHub](https://github.com/)/[GitLab](https://gitlab.com/)/[Bitbucket](https://bitbucket.org)/[Azure DevOps](https://dev.azure.com/)), useful for `git pull -all` to fetch and merge updates from all providers in one command\n  - `git_remotes_set_multi_origin.sh` - sets up multi-remote origin for unified push to automatically keep the 4 major public repositories in sync (especially useful for [Bitbucket](https://bitbucket.org) and [Azure DevOps](https://dev.azure.com/) which don't have [GitLab](https://gitlab.com/)'s auto-mirroring from [GitHub](https://github.com/) feature)\n  - `git_remotes_set_https_to_ssh.sh` - converts local repo's remote URLs from https to ssh (more convenient with SSH keys instead of https auth tokens, especially since Azure DevOps expires personal access tokens every year)\n  - `git_remotes_set_ssh_to_https.sh` - converts local repo's remote URLs from ssh to https (to get through corporate firewalls or hotels if you travel a lot)\n  - `git_remotes_set_https_creds_helpers.sh` - adds Git credential helpers configuration to the local git repo to use http API tokens dynamically from environment variables if they're set\n  - `git_repos_pull.sh` - pull multiple repos based on a source file mapping list - useful for easily sync'ing lots of Git repos among computers\n  - `git_repos_update.sh` - same as above but also runs the `make update` build to install the latest dependencies, leverages the above script\n  - `git_grep_env_vars.sh` - find environment variables in the current git repo's code base in the format `SOME_VAR` (useful to find undocumented environment variables in internal or open source projects such as ArgoCD eg. [argoproj/argocd-cd #8680](https://github.com/argoproj/argo-cd/pull/8680))\n  - `git_log_empty_commits.sh` - find empty commits in git history (eg. if a `git filter-branch` was run but `--prune-empty` was forgotten, leaking metadata like subjects containing file names or other sensitive info)\n  - `git_log_me.sh` - shows only commits in the Git log done by you. Useful to remind yourself what parts the current Git repo you've been working on for periodic reviews, reports or even updating your CV!\n  - `git_log_me_added.sh` - same as above but only file addition commits\n  - `git_graph_commit_history_gnuplot.sh` - generates GNUplot graphs of Git commits per year and per month for the entire history of the local Git repo checkout\n  - `git_graph_commit_history_mermaidjs.sh` - generates MermaidJS graphs of Git commits per year and per month for the entire history of the local Git repo checkout\n  - `git_graph_commit_times_gnuplot.sh` - generates a GNUplot graph of Git commit times from the current Git repo checkout's `git log`\n  - `git_graph_commit_times_mermaidjs.sh` - generates a MermaidJS graph of Git commit times from the current Git repo checkout's `git log`\n  - `git_graph_commit_times_gnuplot_all_repos.sh` - generates GNUplot graph of the GitHub commit times from all local adjacent Git repo checkouts listed in `setup/repos.txt` using Git log in each checkout\n  - `git_graph_commit_times_mermaidjs_all_repos.sh` - generates MermaidJS graph of the GitHub commit times from all local adjacent Git repo checkouts listed in `setup/repos.txt` using Git log in each checkout\n  - `github_public_lines_of_code.sh` - checks out all public original source GitHub repos for the current or given user and then counts all lines of code for them with breakdowns of languages, files, code, comments and blanks\n  - `git_revert_line.sh` - reverts the first line that matches a given regex from the Git head commit's version of the same line number. Useful to revert some changes caused by over zealous sed'ing scripts, where you want to cherry-pick revert a single line change\n  - `git_files_no_uncommitted_changes.sh` - returns zero if given file(s) don't have uncommitted changes to Git, either staged or unstaged. Useful to be able to iterate over git files with in-place edits only if safe to do so without other uncommitted changes that would be at risk of being lost\n  - `git_files_in_history.sh` - finds all filename / file paths in the git log history, useful for prepping for `git filter-branch`\n  - `git_filter_branch_fix_author.sh` - rewrites Git history to replace author/committer name & email references (useful to replace default account commits). Powerful, read `--help` and `man git-filter-branch` carefully. Should only be used by Git Experts\n  - `git_filter_repo_replace_text.sh` - rewrites Git history to replace a given text to scrub a credential or other sensitive token from history. Refuses to operate on tokens less than 8 chars for safety\n  - `git_submodules_update.sh` - updates all submodules in the local git repo to the latest commit of their detected default trunk branch\n  - `git_submodules_update_repos.sh` - updates submodules for all repos given as args or saved in the `setup/repos.txt` file\n  - `git_askpass.sh` - credential helper script to use environment variables for git authentication\n- `github/*.sh` - [GitHub](https://github.com/) API / CLI scripts:\n  - `github_api.sh` - queries the GitHub [API](https://docs.github.com/en/rest/reference). Can infer GitHub user, repo and authentication token from local checkout or environment (`$GITHUB_USER`, `$GITHUB_TOKEN`)\n  - `github_install_binary.sh` - installs a binary from GitHub releases into $HOME/bin or /usr/local/bin. Auto-determines the latest release if no version specified, detects and unpacks any tarball or zip files\n  - `github_foreach_repo.sh` - executes a templated command for each non-fork GitHub repo, replacing the `{owner}`/`{name}` or `{repo}` placeholders in each iteration\n  - `github_graph_commit_times_gnuplot.sh` - generates GNUplot graph of GitHub commit times from all public GitHub repos for a given user. Fetches the commit data via the GitHub API\n  - `github_graph_commit_times_mermaidjs.sh` - generates MermaidJS graph of the GitHub commit times from all public GitHub repos for a given user. Fetches the commit data via the GitHub API\n  - `github_clone_or_pull_all_repos.sh` - git clones or pulls all repos for a user or organization into directories of the same name under the current directory\n  - `github_download_release_file.sh` - downloads a file from GitHub Releases, optionally determining the latest version, uses `bin/download_url_file.sh`\n  - `github_download_release_jar.sh` - downloads a JAR file from GitHub Releases (used by `install/download_*_jar.sh` for things like [JDBC](https://github.com/HariSekhon/Knowledge-Base/blob/main/jdbc.md) drivers or [Java](#java) [decompilers](https://github.com/HariSekhon/Knowledge-Base/blob/main/java.md#java-decompilers)), optionally determines latest version to download, and finally validates the downloaded file's format\n  - `github_invitations.sh` - lists / accepts repo invitations. Useful to accept a large number of invites to repos generated by automation\n  - `github_mirror_repos_to_gitlab.sh` - creates/syncs GitHub repos to GitLab for migrations or to cron fast free Disaster Recovery, including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases\n  - `github_mirror_repos_to_bitbucket.sh` - creates/syncs GitHub repos to BitBucket for migrations or to cron fast free Disaster Recovery, including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases\n  - `github_mirror_repos_to_aws_codecommit.sh` - creates/syncs GitHub repos to AWS CodeCommit for migrations or to cron fast almost free Disaster Recovery (close to $0 compared to $100-400+ per month for [Rewind BackHub](https://rewind.com/products/backups/github/)), including all branches and tags, plus the repo descriptions. Note this doesn't include PRs/wikis/releases\n  - `github_mirror_repos_to_gcp_source_repos.sh` - creates/syncs GitHub repos to GCP Source Repos for migrations or to cron fast almost free Disaster Recovery (close to $0 compared to $100-400+ per month for [Rewind BackHub](https://rewind.com/products/backups/github/)), including all branches and tags. Note this doesn't include repo description/PRs/wikis/releases\n  - `github_pull_request_create.sh` - creates a Pull Request idempotently by first checking for an existing PR between the branches, and also checking if there are the necessary commits between the branches, to avoid common errors from blindly raising PRs. Useful to automate code promotion across environment branches. Also works across repo forks and is used by `github_repo_fork_update.sh`. Even populates github pull request template and does Jira ticket number replacement from branch prefix\n  - `github_pull_request_preview.sh` - opens a GitHub Pull Request preview page from the current local branch to the given or default branch\n  - `github_push_pr_preview.sh` - pushes to GitHub origin, sets upstream branch, then open a Pull Request preview from current branch to the given or default trunk branch in your browser\n  - `github_push_pr.sh` - pushes to GitHub origin, sets upstream branch, then idemopotently creates a Pull Request from current branch to the given or default trunk branch and opens the generated PR in your browser for review\n  - `github_merge_branch.sh` - merges one branch into another branch via a Pull Request for full audit tracking all changes. Useful to automate feature PRs, code promotion across environment branches, or backport hotfixes from Production or Staging to trunk branches such as master, main, dev or develop\n  - `github_remote_set_upstream.sh` - in a forked GitHub repo's checkout, determine the origin of the fork using GitHub CLI and configure a git remote to the upstream. Useful to be able to easily pull updates from the original source repo\n  - `github_pull_merge_trunk.sh` - pulls the origin or fork upstream repo's trunk branch and merges it into the local branch, In a forked GitHub repo's checkout, determines the origin of the fork using GitHub CLI, configures a git remote to the upstream, pulls the default branch and if on a branch other than the default then merges the default branch to the local current branch. Simplifies and automates keeping your checkout or forked repo up to date with the original source repo to quickly resolve merge conflicts locally and submit updated Pull Requests\n  - `github_forked_add_remote.sh` - quickly adds a forked repo as a remote from an interactive men list of forked repos\n  - `github_forked_checkout_branch.sh` - quickly check out a forked repo's branch from an interactive menu lists of forked repos and their branches\n  - `github_tag_hashref.sh` - Returns the GitHub commit hashref for a given GitHub Actions `owner/repo@tag` or `https://github.com/owner/repo@tag`. Useful for pinning 3rd party GitHub Actions to hashref instead of tag to follow [GitHub Actions Best Practices](https://github.com/HariSekhon/Knowledge-Base/blob/main/github-actions.md#github-actions-best-practices)\n  - `github_actions_foreach_workflow.sh` - executes a templated command for each workflow in a given GitHub repo, replacing `{name}`, `{id}` and `{state}` in each iteration\n  - `github_actions_aws_create_load_credential.sh` - creates an AWS user with group/policy, generates and downloads access keys, and uploads them to the given repo\n  - `github_actions_in_use.sh` - lists GitHub Actions directly referenced in the .github/workflows in the current local repo checkout\n  - `github_actions_in_use_repo.sh` - lists GitHub Actions for a given repo via the API, including following imported reusable workflows\n  - `github_actions_in_use_across_repos.sh` - lists GitHub Actions in use across all your repos\n  - `github_actions_repos_lockdown.sh` - secures GitHub Actions settings across all user repos to only GitHub, verified partners and selected 3rd party actions\n  - `github_actions_repo_set_secret.sh` - sets a secret in the given repo from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `github_actions_repo_env_set_secret.sh` - sets a secret in the given repo and environment from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `github_actions_repo_secrets_overriding_org.sh` - finds any secrets for a repo that are overriding organization level secrets. Useful to combine with `github_foreach_repo.sh` for auditing\n  - `github_actions_repo_restrict_actions.sh` - restricts GitHub Actions in the given repo to only running actions from GitHub and verfied partner companies (.eg AWS, Docker)\n  - `github_actions_repo_actions_allow.sh` - allows select 3rd party GitHub Actions in the given repo\n  - `github_actions_runner.sh` - generates a [GitHub Actions](https://github.com/features/actions) self-hosted runner token for a given Repo or Organization via the GitHub API and then runs a dockerized GitHub Actions runner with the appropriate configuration\n  - `github_actions_runner_local.sh` - downloads, configures and runs a local GitHub Actions Runner for Linux or Mac\n  - `github_actions_runner_token.sh` - generates a GitHub Actions runner token to register a new self-hosted runner\n  - `github_actions_runners.sh` - lists GitHub Actions self-hosted runners for a given Repo or Organization\n  - `github_actions_delete_offline_runners.sh` - deletes offline GitHub Actions self-hosted runners. Useful to clean up short-lived runners eg. Docker, Kubernetes\n  - `github_actions_workflows.sh` - lists GitHub Actions workflows for a given repo (or auto-infers local repository)\n  - `github_actions_workflow_runs.sh` - lists GitHub Actions workflow runs for a given workflow id or name\n  - `github_actions_workflows_status.sh` - lists all GitHub Actions workflows and their statuses for a given repo\n  - `github_actions_workflows_state.sh` - lists GitHub Actions workflows enabled/disabled states (GitHub now disables workflows after 6 months without a commit)\n  - `github_actions_workflows_disabled.sh` - lists GitHub Actions workflows that are disabled. Combine with `github_foreach_repo.sh` to scan all repos to find disabled workflows\n  - `github_actions_workflow_enable.sh` - enables a given GitHub Actions workflow\n  - `github_actions_workflows_enable_all.sh` - enables all GitHub Actions workflows in a given repo. Useful to undo GitHub disabling all workflows in a repo after 6 months without a commit\n  - `github_actions_workflows_trigger_all.sh` - triggers all workflows for the given repo\n  - `github_actions_workflows_cancel_all_runs.sh` - cancels all workflow runs for the given repo\n  - `github_actions_workflows_cancel_waiting_runs.sh` - cancels workflow runs that are in waiting state, eg. waiting for old deployment approvals\n  - `github_actions_log.sh` - outputs the text log for a given GitHub Actions workflow run to the terminal. Fetches the last 10 runs and drops you into an interactive menu to hit enter on the one you want. Useful when the logs are too big for the UI and you have to open it in another tab which is very slow in browser\n  - `github_actions_latest_log.sh` - same as above, but just fetches the latest workflow run log without any prompting\n  - `github_ssh_get_user_public_keys.sh` - fetches a given GitHub user's public SSH keys via the API for piping to `~/.ssh/authorized_keys` or adjacent tools\n  - `github_ssh_get_public_keys.sh` - fetches the currently authenticated GitHub user's public SSH keys via the API, similar to above but authenticated to get identifying key comments\n  - `github_ssh_add_public_keys.sh` - uploads SSH keys from local files or standard input to the currently authenticated GitHub account. Specify pubkey files (default: `~/.ssh/id_rsa.pub`) or read from standard input for piping from adjacent tools\n  - `github_ssh_delete_public_keys.sh` - deletes given SSH keys from the currently authenticated GitHub account by key id or title regex match\n  - `github_gpg_get_user_public_keys.sh` - fetches a given GitHub user's public GPG keys via the API\n  - `github_generate_status_page.sh` - generates a [STATUS.md](https://harisekhon.github.io/CI-CD/) page by merging all the README.md headers for all of a user's non-forked GitHub repos or a given list of any repos etc.\n  - `github_purge_camo_cache.sh` - send HTTP Purge requests to all camo urls (badge caches) for the current or given GitHub repo's landing/README.md page\n  - `github_ip_ranges.sh` - returns GitHub's IP ranges, either all by default or for a select given service such as hooks or actions\n  - `github_sync_repo_descriptions.sh` - syncs GitHub repo descriptions to GitLab & BitBucket repos\n  - `github_release.sh` - creates a GitHub Release, auto-incrementing a `.N` suffix on the year/month/day date format if no exact version given\n  - `github_repo_check_pat_token.sh` - checks the given PAT token can access the given GitHub repo. Useful to test a PAT token used for integrations like ArgoCD\n  - `github_repo_description.sh` - fetches the given repo's description (used by `github_sync_repo_descriptions.sh`)\n  - `github_repo_find_files.sh` - finds files matching a regex in the current or given GitHub repo via the GitHub API\n  - `github_repo_latest_release.sh` - returns the latest release tag for a given GitHub repo via the GitHub API\n  - `github_repo_latest_release_filter.sh` - returns the latest release tag matching a given regex filter for a given GitHub repo via the GitHub API. Useful for getting the latest version of things like Kustomize which has other releases for kyaml\n  - `github_repo_stars.sh` - fetches the stars, forks and watcher counts for a given repo\n  - `github_repo_teams.sh` - fetches the GitHub Enterprise teams and their role permisions for a given repo. Combine with `github_foreach_repo.sh` to audit your all your personal or GitHub organization's repos\n  - `github_repo_collaborators.sh` - fetches a repo's granted users and outside invited collaborators as well as their role permisions for a given repo. Combine with `github_foreach_repo.sh` to audit your all your personal or GitHub organization's repos\n  - `github_repo_protect_branches.sh` - enables branch protections on the given repo. Can specify one or more branches to protect, otherwise finds and applies to any of `master`, `main`, `develop`, `dev`, `staging`, `production`\n  - `github_repos_find_files.sh` - finds files matching a regex across all repos in the current GitHub organization or user account\n  - `github_repo_fork_sync.sh` - sync's current or given fork, then runs `github_repo_fork_update.sh` to cascade changes to major branches via Pull Requests for auditability\n  - `github_repo_fork_update.sh` - updates a forked repo by creating pull requests for full audit tracking and auto-merges PRs for non-production branches\n  - `github_repos_public.sh` - lists public repos for a user or organization. Useful to periodically scan and account for any public repos\n  - `github_repos_disable_wiki.sh` - disables the Wiki on one or more given repos to prevent documentation fragmentation and make people use the centralized documentation tool eg. Confluence or Slite\n  - `github_repos_with_few_users.sh` - finds repos with few or no users (default: 1), which in Enterprises is a sign that a user has created a repo without assigning team privileges\n  - `github_repos_with_few_teams.sh` - finds repos with few or no teams (default: 0), which in Enterprises is a sign that a user has created a repo without assigning team privileges\n  - `github_repos_without_branch_protections.sh` - finds repos without any branch protection rules (use `github_repo_protect_branches.sh` on such repos)\n  - `github_repos_not_in_terraform.sh` - finds all non-fork repos for current or given user/organization which are not found in `$PWD/*.tf` Terraform code\n  - `github_teams_not_in_terraform.sh` - finds all teams for given organization which are not found in `$PWD/*.tf` Terraform code\n  - `github_repos_sync_status.sh` - determines whether each GitHub repo's mirrors on GitLab / BitBucket / Azure DevOps are up to date with the latest commits, by querying all 3 APIs and comparing master branch hashrefs\n  - `github_teams_not_idp_synced.sh` - finds GitHub teams that aren't sync'd from an IdP like Azure AD. These should usually be migrated or removed\n  - `github_user_repos_stars.sh` - fetches the total number of stars for all original source public repos for a given user\n  - `github_user_repos_forks.sh` - fetches the total number of forks for all original source public repos for a given user\n  - `github_user_repos_count.sh` - fetches the total number of original source public repos for a given username\n  - `github_user_followers.sh` - fetches the number of followers for a given username\n  - `github_url_clipboard.sh` - copies a GitHub URL file's contents to the clipboard, converting the URL to a raw GitHub content URL where necessary\n- `gitlab/*.sh` - [GitLab](https://gitlab.com/) API scripts:\n  - `gitlab_api.sh` - queries the GitLab [API](https://docs.gitlab.com/ee/api/api_resources.html). Can infer GitLab user, repo and authentication token from local checkout or environment (`$GITLAB_USER`, `$GITLAB_TOKEN`)\n  - `gitlab_install_binary.sh` - installs a binary from GitLab releases into $HOME/bin or /usr/local/bin. Auto-determines the latest release if no version specified, detects and unpacks any tarball or zip files\n  - `gitlab_push_mr_preview.sh` - pushes to GitLab origin, sets upstream branch, then open a Merge Request preview from current to default branch\n  - `github_push_mr.sh` - pushes to GitLab origin, sets upstream branch, then idemopotently creates a Merge Request from current branch to the given or default trunk branch and opens the generated MR in your browser for review\n  - `gitlab_foreach_repo.sh` - executes a templated command for each GitLab project/repo, replacing the `{user}` and `{project}` in each iteration\n  - `gitlab_project_latest_release.sh` - returns the latest release tag for a given GitLab project (repo) via the GitLab API\n  - `gitlab_project_set_description.sh` - sets the description for one or more projects using the GitLab API\n  - `gitlab_project_set_env_vars.sh` - adds / updates GitLab project-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `gitlab_group_set_env_vars.sh` - adds / updates GitLab group-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `gitlab_project_create_import.sh` - creates a GitLab repo as an import from a given URL, and mirrors if on GitLab Premium (can only manually configure for public repos on free tier, API doesn't support configuring even public repos on free)\n  - `gitlab_project_protect_branches.sh` - enables branch protections on the given project. Can specify one or more branches to protect, otherwise finds and applies to any of `master`, `main`, `develop`, `dev`, `staging`, `production`\n  - `gitlab_project_mirrors.sh` - lists each GitLab repo and whether it is a mirror or not\n  - `gitlab_pull_mirror.sh` - trigger a GitLab pull mirroring for a given project's repo, or auto-infers project name from the local git repo\n  - `gitlab_ssh_get_user_public_keys.sh` - fetches a given GitLab user's public SSH keys via the API, with identifying comments, for piping to `~/.ssh/authorized_keys` or adjacent tools\n  - `gitlab_ssh_get_public_keys.sh` - fetches the currently authenticated GitLab user's public SSH keys via the API\n  - `gitlab_ssh_add_public_keys.sh` - uploads SSH keys from local files or standard input to the currently authenticated GitLab account. Specify pubkey files (default: `~/.ssh/id_rsa.pub`) or read from standard input for piping from adjacent tools\n  - `gitlab_ssh_delete_public_keys.sh` - deletes given SSH keys from the currently authenticated GitLab account by key id or title regex match\n  - `gitlab_validate_ci_yaml.sh` - validates a `.gitlab-ci.yml` file via the GitLab API\n- `bitbucket/*.sh` - [BitBucket](https://bitbucket.org/) API scripts:\n  - `bitbucket_api.sh` - queries the BitBucket [API](https://developer.atlassian.com/bitbucket/api/2/reference/resource/). Can infer BitBucket user, repo and authentication token from local checkout or environment (`$BITBUCKET_USER`, `$BITBUCKET_TOKEN`)\n  - `bitbucket_foreach_repo.sh` - executes a templated command for each BitBucket repo, replacing the `{user}` and `{repo}` in each iteration\n  - `bitbucket_workspace_set_env_vars.sh` - adds / updates Bitbucket workspace-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `bitbucket_repo_set_env_vars.sh` - adds / updates Bitbucket repo-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `bitbucket_repo_set_description.sh` - sets the description for one or more repos using the BitBucket API\n  - `bitbucket_enable_pipelines.sh` - enables the CI/CD pipelines for all repos\n  - `bitbucket_disable_pipelines.sh` - disables the CI/CD pipelines for all repos\n  - `bitbucket_repo_enable_pipeline.sh` - enables the CI/CD pipeline for a given repo\n  - `bitbucket_repo_disable_pipeline.sh` - disables the CI/CD pipeline for a given repo\n  - `bitbucket_ssh_get_public_keys.sh` - fetches the currently authenticated BitBucket user's public SSH keys via the API for piping to `~/.ssh/authorized_keys` or adjacent tools\n  - `bitbucket_ssh_add_public_keys.sh` - uploads SSH keys from local files or standard input to the currently authenticated BitBucket account. Specify pubkey files (default: `~/.ssh/id_rsa.pub`) or read from standard input for piping from adjacent tools\n  - `bitbucket_ssh_delete_public_keys.sh` - uploads SSH keys from local files or standard input to the currently authenticated BitBucket account. Specify pubkey files (default: `~/.ssh/id_rsa.pub`) or read from standard input for piping from adjacent tools\n\nSee also [Knowledge Base notes for Git](https://github.com/HariSekhon/Knowledge-Base/blob/main/git.md).\n\n### Markdown\n\nMaintain your Git `README.md` and similar Markdown documentation well.\n\n- `markdown/*`:\n  - `markdown_generate_index.sh` - generates a markdown index list from the headings in a given markdown file such as `README.md`\n  - `markdown_replace_index.sh` - replaces a markdown index section in a given markdown file using `markdown_generate_index.sh`\n  - `markdown_replace_repos.sh` - replaces the repos block of a given markdown file. Used to keep my GitHub repos Other Repos sections updated\n  - `mdl_list_indentations.sh`- runs Markdownlint `mdl` command and prefixes the spaces count to each offending line of MD005 (inconsistent list indentations). Workaround for [Markdownlint issue #520](https://github.com/markdownlint/markdownlint/issues/520)\n  - `markdown_list_indentations.sh` - prefixes number of spaces before each list item for comparison to MarkdownLint MD005 inconsistent list indentation error\n  - `markdown_columns_to_table.sh` - converts text columns separated by whitespace to a Markdown table with vertically aligned column pipe chars. Combine with scripts like `domains_subdomains_environments.sh` to generate the markdown documentation of your domains and subdomains per project and environment\n  - `markdown_octocat_github_links.sh` - converts GitHub links like\n    `<https://github.com/HariSekhon/Knowledge-Base>` to shorthand links with an OctoCat emoji and without the redundant `https://github.com/` prefix such as\n    [:octocat: HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base)\n  - `markdown_replace_links_with_jsdelivr.sh` - replaces local GitHub repo file links in the given markdown file(s) with JSDelivr CDN links\n\n### CI/CD - Continuous Integration / Continuous Deployment\n\n`jenkins/`, `terraform/`, `teamcity/`, `buildkite/`, `circlci/`, `travis/`, `azure_devops/`, ...,  `cicd/` directories:\n\n- `appveyor_api.sh` - queries [AppVeyor](https://www.appveyor.com/)'s API with authentication\n- `azure_devops/*.sh` - [Azure DevOps](https://dev.azure.com/) scripts:\n  - `azure_devops_api.sh` - queries Azure DevOps's API with authentication\n  - `azure_devops_foreach_repo.sh` - executes a templated command for each Azure DevOps repo, replacing `{user}`, `{org}`, `{project}` and `{repo}` in each iteration\n  - `azure_devops_to_github_migration.sh` - migrates one or all Azure DevOps git repos to GitHub, including all branches and sets the default branch to match via the APIs to maintain the same checkout behaviour\n  - `azure_devops_disable_repos.sh` - disables one or more given Azure DevOps repos (to prevent further pushes to them after migration to GitHub)\n- `circleci/*.sh` - [CircleCI](https://circleci.com/) scripts:\n  - `circleci_api.sh` - queries CircleCI's API with authentication\n  - `circleci_project_set_env_vars.sh` - adds / updates CircleCI project-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `circleci_context_set_env_vars.sh` - adds / updates CircleCI context-level environment variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `circleci_project_delete_env_vars.sh` - deletes CircleCI project-level environment variable(s) via the API\n  - `circleci_context_delete_env_vars.sh` - deletes CircleCI context-level environment variable(s) via the API\n  - `circleci_local_execute.sh` - installs CircleCI CLI and executes `.circleci/config.yml` locally\n  - `circleci_public_ips.sh` - lists [CircleCI](https://circleci.com) public IP addresses via dnsjson.com\n- `codeship_api.sh` - queries [CodeShip](https://codeship.com/)'s API with authentication\n- `drone_api.sh` - queries [Drone.io](https://drone.io/)'s API with authentication\n- `shippable_api.sh` - queries [Shippable](https://www.shippable.com/)'s API with authentication\n- `wercker_app_api.sh` - queries [Wercker](https://app.wercker.com/)'s Applications API with authentication\n- `gocd_api.sh` - queries [GoCD](https://www.gocd.org/)'s API\n- `gocd.sh` - one-touch [GoCD CI](https://www.gocd.org/):\n  - launches in Docker\n  - (re)creates config repo (`$PWD/setup/gocd_config_repo.json`) from which to source pipeline(s) (`.gocd.yml`)\n  - detects and enables agent(s) to start building\n  - call from any repo top level directory with a `.gocd.yml` config (all mine have it), mimicking structure of fully managed CI systems\n- `concourse.sh` - one-touch [Concourse CI](https://concourse-ci.org/):\n  - launches in Docker\n  - configures pipeline from `$PWD/.concourse.yml`\n  - triggers build\n  - tails results in terminal\n  - prints recent build statuses at end\n  - call from any repo top level directory with a `.concourse.yml` config (all mine have it), mimicking structure of fully managed CI systems\n- `fly.sh` - shortens [Concourse](https://concourse-ci.org/) `fly` command to not have to specify target all the time\n- `jenkins/*.sh` - [Jenkins CI](https://jenkins.io/) scripts:\n  - `jenkins.sh` - one-touch [Jenkins CI](https://jenkins.io/):\n    - launches Docker container\n    - installs plugins\n    - validates `Jenkinsfile`\n    - configures job from `$PWD/setup/jenkins-job.xml`\n    - sets Pipeline to git remote origin's `Jenkinsfile`\n    - triggers build\n    - tails results in terminal\n    - call from any repo top level directory with a `Jenkinsfile` pipeline and `setup/jenkins-job.xml` (all mine have it)\n  - `jenkins_api.sh` - queries the Jenkins Rest API, handles authentication, pre-fetches CSFR protection token crumb, supports many environment variables such as `$JENKINS_URL` for ease of use\n    - `jenkins_jobs.sh` - lists Jenkins jobs (pipelines)\n    - `jenkins_foreach_job.sh` - runs a templated command for each Jenkins job\n    - `jenkins_jobs_download_configs.sh` - downloads all Jenkins job configs to xml files of the same name\n    - `jenkins_job_config.sh` - gets or sets a Jenkins job's config\n    - `jenkins_job_description.sh` - gets or sets a Jenkins job's description\n    - `jenkins_job_enable.sh` - enables a Jenkins job by name\n    - `jenkins_job_disable.sh` - disables a Jenkins job by name\n    - `jenkins_job_trigger.sh` - triggers a Jenkins job by name\n    - `jenkins_job_trigger_with_params.sh` - triggers a Jenkins job with parameters which can be passed as `--data KEY=VALUE`\n    - `jenkins_jobs_enable.sh` - enables all Jenkins jobs/pipelines with names matching a given regex\n    - `jenkins_jobs_disable.sh` - disables all Jenkins jobs/pipelines with names matching a given regex\n    - `jenkins_builds.sh` - lists Jenkins latest builds for every job\n    - `jenkins_cred_add_cert.sh` - creates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_add_kubernetes_sa.sh` - creates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_add_secret_file.sh` - creates a Jenkins secret file credential from a file\n    - `jenkins_cred_add_secret_text.sh` - creates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_add_ssh_key.sh` - creates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_add_user_pass.sh` - creates a Jenkins username/password credential\n    - `jenkins_cred_delete.sh` - deletes a given Jenkins credential by id\n    - `jenkins_cred_list.sh` - lists Jenkins credentials IDs and Names\n    - `jenkins_cred_update_cert.sh` - updates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_update_kubernetes_sa.sh` - updates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_update_secret_file.sh` - updates a Jenkins secret file credential from a file\n    - `jenkins_cred_update_secret_text.sh` - updates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_update_ssh_key.sh` - updates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_update_user_pass.sh` - updates a Jenkins username/password credential\n    - `jenkins_cred_set_cert.sh` - creates or updates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_set_kubernetes_sa.sh` - creates or updates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_set_secret_file.sh` - creates or updates a Jenkins secret file credential from a file\n    - `jenkins_cred_set_secret_text.sh` - creates or updates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_set_ssh_key.sh` - creates or updates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_set_user_pass.sh` - creates or updates a Jenkins username/password credential\n  - `jenkins_cli.sh` - shortens `jenkins-cli.jar` command by auto-inferring basic configuations, auto-downloading the CLI if absent, inferrings a bunch of Jenkins related variables like `$JENKINS_URL`, `$JENKINS_CLI_ARGS` and authentication using `$JENKINS_USER`/`$JENKINS_PASSWORD`, or finds admin password from inside local docker container. Used heavily by `jenkins.sh` one-shot setup and the following scripts:\n    - `jenkins_foreach_job_cli.sh` - runs a templated command for each Jenkins job\n    - `jenkins_create_job_parallel_test_runs.sh` - creates a freestyle parameterized test sleep job and launches N parallel runs of it to test scaling and parallelization of [Jenkins on Kubernetes](https://github.com/HariSekhon/Kubernetes-configs#jenkins-on-kubernetes) agents\n    - `jenkins_create_job_check_gcp_serviceaccount.sh` - creates a freestyle test job which runs a GCP Metadata query to determine the GCP serviceaccount the agent pod is operating under to check GKE Workload Identity integration\n    - `jenkins_jobs_download_configs_cli.sh` - downloads all Jenkins job configs to xml files of the same name\n    - `jenkins_cred_cli_add_cert.sh` - creates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_cli_add_kubernetes_sa.sh` - creates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_cli_add_secret_file.sh` - creates a Jenkins secret file credential from a file\n    - `jenkins_cred_cli_add_secret_text.sh` - creates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_cli_add_ssh_key.sh` - creates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_cli_add_user_pass.sh` - creates a Jenkins username/password credential\n    - `jenkins_cred_cli_delete.sh` - deletes a given Jenkins credential by id\n    - `jenkins_cred_cli_list.sh` - lists Jenkins credentials IDs and Names\n    - `jenkins_cred_cli_update_cert.sh` - updates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_cli_update_kubernetes_sa.sh` - updates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_cli_update_secret_file.sh` - updates a Jenkins secret file credential from a file\n    - `jenkins_cred_cli_update_secret_text.sh` - updates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_cli_update_ssh_key.sh` - updates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_cli_update_user_pass.sh` - updates a Jenkins username/password credential\n    - `jenkins_cred_cli_set_cert.sh` - creates or updates a Jenkins certificate credential from a PKCS#12 keystore\n    - `jenkins_cred_cli_set_kubernetes_sa.sh` - creates or updates a Jenkins Kubernetes service account credential\n    - `jenkins_cred_cli_set_secret_file.sh` - creates or updates a Jenkins secret file credential from a file\n    - `jenkins_cred_cli_set_secret_text.sh` - creates or updates a Jenkins secret string credential from a string or a file\n    - `jenkins_cred_cli_set_ssh_key.sh` - creates or updates a Jenkins SSH key credential from a string or an SSH private key file\n    - `jenkins_cred_cli_set_user_pass.sh` - creates or updates a Jenkins username/password credential\n  - `jenkins_password.sh` - gets Jenkins admin password from local docker container. Used by `jenkins_cli.sh`\n  - `jenkins_plugins_latest_versions.sh` - finds the latest versions of given Jenkins plugins. Useful to programmatically upgrade your Jenkins on Kubernetes plugins defined in [values.yaml](https://github.com/HariSekhon/Kubernetes-configs/blob/6d9e34b74d3fa8f353b0fe56e74cea3af439e01a/jenkins/base/values.yaml#L145)\n  - `check_jenkinsfiles.sh` - validates all `*Jenkinsfile*` files in the given directory trees using the online Jenkins validator\n  - See also [Knowledge Base notes for Jenkins](https://github.com/HariSekhon/Knowledge-Base/blob/main/jenkins.md).\n- `teamcity/*.sh` - [TeamCity CI](https://www.jetbrains.com/teamcity/) scripts:\n  - `teamcity.sh` - one-touch [TeamCity CI](https://www.jetbrains.com/teamcity/) cluster:\n    - launches Docker containers with server and 1 agent\n    - click proceed and accept the EULA\n    - waits for server to initialize\n    - waits for agent to register\n    - authorizes agent\n    - creates a VCS Root if `$PWD` has a `.teamcity.vcs.json` / `.teamcity.vcs.ssh.json` / `.teamcity.vcs.oauth.json` and corresponding `$TEAMCITY_SSH_KEY` or `$TEAMCITY_GITHUB_CLIENT_ID`+`$TEAMCITY_GITHUB_CLIENT_SECRET` environment variables\n    - creates a Project and imports all settings and builds from the VCS Root\n    - creates an admin user and an API token for you\n    - see also: [TeamCity CI](https://github.com/HariSekhon/TeamCity-CI) config repo for importing pipelines\n  - `teamcity_api.sh` - queries TeamCity's API, auto-handling authentication and other quirks of the API\n  - `teamcity_create_project.sh` - creates a TeamCity project using the API\n  - `teamcity_create_github_oauth_connection.sh` - creates a TeamCity GitHub OAuth VCS connection in the Root project, useful for bootstrapping projects from VCS configs\n  - `teamcity_create_vcs_root.sh` - creates a TeamCity VCS root from a save configuration (XML or JSON), as downloaded by `teamcity_export_vcs_roots.sh`\n  - `teamcity_upload_ssh_key.sh` - uploads an SSH private key to a TeamCity project (for use in VCS root connections)\n  - `teamcity_agents.sh` - lists TeamCity agents, their connected state, authorized state, whether enabled and up to date\n  - `teamcity_builds.sh` - lists the last 100 TeamCity builds along with the their state (eg. `finished`) and status (eg. `SUCCESS`/`FAILURE`)\n  - `teamcity_buildtypes.sh` - lists TeamCity buildTypes (pipelines) along with the their project and IDs\n  - `teamcity_buildtype_create.sh` - creates a TeamCity buildType from a local JSON configuration (see `teamcity_buildtypes_download.sh`)\n  - `teamcity_buildtype_set_description_from_github.sh` - sync's a TeamCity buildType's description from its Github repo description\n  - `teamcity_buildtypes_set_description_from_github.sh` - sync's all TeamCity buildType descriptions from their GitHub repos where available\n  - `teamcity_export.sh` - downloads TeamCity configs to local JSON files in per-project directories mimicking native TeamCity directory structure and file naming\n  - `teamcity_export_project_config.sh` - downloads TeamCity project config to local JSON files\n  - `teamcity_export_buildtypes.sh` - downloads TeamCity buildType config to local JSON files\n  - `teamcity_export_vcs_roots.sh` - downloads TeamCity VCS root config to local JSON files\n  - `teamcity_projects.sh` - lists TeamCity project IDs and Names\n  - `teamcity_project_set_versioned_settings.sh` - configures a project to track all changes to a VCS (eg. GitHub)\n  - `teamcity_project_vcs_versioning.sh` - quickly toggle VCS versioning on/off for a given TeamCity project (useful for testing without auto-committing)\n  - `teamcity_vcs_roots.sh` - lists TeamCity VCS root IDs and Names\n- `travis/*.sh` - [Travis CI](https://travis-ci.org/) API scripts (one of my all-time favourite CI systems):\n  - `travis_api.sh` - queries the Travis CI API with authentication using `$TRAVIS_TOKEN`\n  - `travis_repos.sh` - lists Travis CI repos\n  - `travis_foreach_repo.sh` - executes a templated command against all Travis CI repos\n  - `travis_repo_build.sh` - triggers a build for the given repo\n  - `travis_repo_caches.sh` - lists caches for a given repo\n  - `travis_repo_crons.sh` - lists crons for a given repo\n  - `travis_repo_env_vars.sh` - lists environment variables for a given repo\n  - `travis_repo_settings.sh` - lists settings for a given repo\n  - `travis_repo_create_cron.sh` - creates a cron for a given repo and branch\n  - `travis_repo_delete_crons.sh` - deletes all crons for a given repo\n  - `travis_repo_delete_caches.sh` - deletes all caches for a given repo (sometimes clears build problems)\n  - `travis_delete_cron.sh` - deletes a Travis CI cron by ID\n  - `travis_repos_settings.sh` - lists settings for all repos\n  - `travis_repos_caches.sh` - lists caches for all repos\n  - `travis_repos_crons.sh` - lists crons for all repos\n  - `travis_repos_create_cron.sh` - creates a cron for all repos\n  - `travis_repos_delete_crons.sh` - deletes all crons for all repos\n  - `travis_repos_delete_caches.sh` - deletes all caches for all repos\n  - `travis_lint.sh` - lints a given `.travis.yml` using the API\n- `buildkite/*.sh` - [BuildKite](https://buildkite.com/) API scripts:\n  - `buildkite_api.sh` - queries the BuildKite API, handling authentication using `$BUILDKITE_TOKEN`\n  - `buildkite_pipelines.sh` - list buildkite pipelines for your `$BUILDKITE_ORGANIZATION` / `$BUILDKITE_USER`\n  - `buildkite_foreach_pipeline.sh` - executes a templated command for each Buildkite pipeline, replacing the `{user}` and `{pipeline}` in each iteration\n  - `buildkite_agent.sh` - runs a buildkite agent locally on Linux or Mac, or in Docker with choice of Linux distros\n  - `buildkite_agents.sh` - lists the Buildkite agents connected along with their hostname, IP, started dated and agent details\n  - `buildkite_pipelines.sh` - lists Buildkite pipelines\n  - `buildkite_create_pipeline.sh` - create a Buildkite pipeline from a JSON configuration (like from `buildkite_get_pipeline.sh` or `buildkite_save_pipelines.sh`)\n  - `buildkite_get_pipeline.sh` - gets details for a specific Buildkite pipeline in JSON format\n  - `buildkite_update_pipeline.sh` - updates a BuildKite pipeline from a configuration provided via stdin or from a file saved via `buildkite_get_pipeline.sh`\n  - `buildkite_patch_pipeline.sh` - updates a BuildKite pipeline from a partial configuration provided as an arg, via stdin, or from a file saved via `buildkite_get_pipeline.sh`\n  - `buildkite_pipeline_skip_settings.sh` - lists the skip intermediate build settings for one or more given BuildKite pipelines\n  - `buildkite_pipeline_set_skip_settings.sh` - configures given or all BuildKite pipelines to skip intermediate builds and cancel running builds in favour of latest build\n  - `buildkite_cancel_scheduled_builds.sh` - cancels BuildKite scheduled builds (to clear a backlog due to offline agents and just focus on new builds)\n  - `buildkite_cancel_running_builds.sh` - cancels BuildKite running builds (to clear them and restart new later eg. after agent / environment change / fix)\n  - `buildkite_pipeline_disable_forked_pull_requests.sh` - disables forked pull request builds on a BuildKite pipeline to protect your build environment from arbitrary code execution security vulnerabilities\n  - `buildkite_pipelines_vulnerable_forked_pull_requests.sh` - prints the status of each pipeline, should all return false, otherwise run the above script to close the vulnerability\n  - `buildkite_rebuild_cancelled_builds.sh` - triggers rebuilds of last N cancelled builds in current pipeline\n  - `buildkite_rebuild_failed_builds.sh` - triggers rebuilds of last N failed builds in current pipeline (eg. after agent restart / environment change / fix)\n  - `buildkite_rebuild_all_pipelines_last_cancelled.sh` - triggers rebuilds of the last cancelled build in each pipeline in the organization\n  - `buildkite_rebuild_all_pipelines_last_failed.sh` - triggers rebuilds of the last failed build in each pipeline in the organization\n  - `buildkite_retry_jobs_dead_agents.sh` - triggers job retries where jobs failed due to killed agents, continuing builds from that point and replacing their false negative failed status with the real final status, slightly better than rebuilding entire jobs which happen under a new build\n  - `buildkite_recreate_pipeline.sh` - recreates a pipeline to wipe out all stats (see url and badge caveats in `--help`)\n  - `buildkite_running_builds.sh` - lists running builds and the agent they're running on\n  - `buildkite_save_pipelines.sh` - saves all BuildKite pipelines in your `$BUILDKITE_ORGANIZATION` to local JSON files in `$PWD/.buildkite-pipelines/`\n  - `buildkite_set_pipeline_description.sh` - sets the description of one or more pipelines using the BuildKite API\n  - `buildkite_set_pipeline_description_from_github.sh` - sets a Buildkite pipeline description to match its source GitHub repo\n  - `buildkite_sync_pipeline_descriptions_from_github.sh` - for all BuildKite pipelines sets each description to match its source GitHub repo\n  - `buildkite_trigger.sh` - triggers BuildKite build job for a given pipeline\n  - `buildkite_trigger_all.sh` - same as above but for all pipelines\n- `terraform_cloud_*.sh` - [Terraform Cloud](https://www.terraform.io/cloud) API scripts:\n  - `terraform_cloud_api.sh` - queries the Cloudflare API, handling authentication using `$TERRAFORM_TOKEN`\n  - `terraform_cloud_ip_ranges.sh` - returns the list of IP ranges for Terraform Cloud\n  - `terraform_cloud_organizations.sh` - lists Terraform Cloud organizations\n  - `terraform_cloud_workspaces.sh` - lists Terraform Cloud workspaces\n  - `terraform_cloud_workspace_vars.sh` - lists Terraform Cloud workspace variables\n  - `terraform_cloud_workspace_set_vars.sh` - adds / updates Terraform workspace-level sensitive environment/terraform variable(s) via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `terraform_cloud_workspace_delete_vars.sh` - deletes one or more Terraform workspace-level variables\n  - `terraform_cloud_varsets.sh` - lists Terraform Cloud variable sets\n  - `terraform_cloud_varset_vars.sh` - lists Terraform Cloud variables in on or all variables sets for the given organization\n  - `terraform_cloud_varset_set_vars.sh` - adds / updates Terraform sensitive environment/terraform variable(s) in a given variable set via the API from `key=value` or shell export format, as args or via stdin (eg. piped from `aws_csv_creds.sh`)\n  - `terraform_cloud_varset_delete_vars.sh` - deletes one or more Terraform variables in a given variable set\n- `terraform_*.sh` - [Terraform](https://www.terraform.io/) scripts:\n  - `terraform_gcs_backend_version.sh` - determines the Terraform state version from the tfstate file in a GCS bucket found in a local given `backend.tf`\n  - `terraform_gitlab_download_backend_variable.sh` - downloads backend.tf from a GitLab CI/CD variable to be able to quickly iterate plans locally\n  - `terraform_import.sh` - finds given resource type in `./*.tf` code or Terraform plan output that are not in Terraform state and imports them\n  - `terraform_import_aws_iam_users.sh` - parses Terraform plan output to import new `aws_iam_user` additions into Terraform state\n  - `terraform_import_aws_iam_groups.sh` - parses Terraform plan output to import new `aws_iam_group` additions into Terraform state\n  - `terraform_import_aws_iam_policies.sh` - parses Terraform plan output to import new `aws_iam_policies` additions, resolves their ARNs and imports them into Terraform state\n  - `terraform_import_aws_sso_permission_sets.sh` - finds all `aws_ssoadmin_permission_set` in `./*.tf` code, resolves the ARNs and imports them to Terraform state\n  - `terraform_import_aws_sso_account_assignments.sh` - parses Terraform plan output to import new `aws_ssoadmin_account_assignment` additions into Terraform state\n  - `terraform_import_aws_sso_managed_policy_attachments.sh` - parses Terraform plan output to import new `aws_ssoadmin_account_assignment` additions into Terraform state\n  - `terraform_import_aws_sso_permission_set_inline_policies.sh` - parses Terraform plan output to import new `aws_ssoadmin_permission_set_inline_policy` additions into Terraform state\n  - `terraform_import_github_repos.sh` - finds all `github_repository` in `./*.tf` code or Terraform plan output that are not in Terraform state and imports them. See also `github_repos_not_in_terraform.sh`\n  - `terraform_import_github_team.sh` - imports a given GitHub team into a given Terraform state resource, by first querying the GitHub API for the team ID needed to import into Terraform\n  - `terraform_import_github_teams.sh` - finds all `github_team` in `./*.tf` code or Terraform plan output that are not in Terraform state, then queries the GitHub API for their IDs and imports them. See also `github_teams_not_in_terraform.sh`\n  - `terraform_import_github_team_repos.sh` - finds all `github_team_repository` in Terraform plan that would be added, then queries the GitHub API for the repos and team IDs and if they both exist then imports them to Terraform state\n  - `terraform_provider_count_sizes.sh` - finds duplicate Terraform providers and their sizes. Useful to find space wastage caused by using Terragrunt without configuring a unified Terraform Plugin Cache\n  - `terraform_resources.sh` - external program to get all resource ids and attribute for a given resource type to work around Terraform splat expression limitation ([#19931](https://github.com/hashicorp/terraform/issues/19931))\n  - `terraform_managed_resource_types.sh` - quick parse of what Terraform resource types are found in `*.tf` files under the current or given directory tree. Useful to give you a quick glance of what services you are managing\n  - `terraform_registry_url_extract.sh` - extracts the Terraform Registry URL in either `tfr://` or `https://registry.terraform.io/` format from a given string, file or standard input. Useful to fast load Terraform Module documentation via editor/IDE hotkeys (see [.vimrc](configs/.vimrc)). Based on `urlextract.sh` above\n  - `terraform_registry_url_to_https.sh` - converts one or more Terraform Registry URLs from `tfr://` to `https://registry.terraform.io/` format\n  - `terraform_registry_url_open.sh` - opens the Terraform Registry URL given as a string arg, file or standard input in either `tfr://` or `https://registry.terraform.io/` format\n  - See also [Knowledge Base notes for Terraform](https://github.com/HariSekhon/Knowledge-Base/blob/main/terraform.md).\n- `checkov_resource_*.sh` - [Checkov](https://www.checkov.io/) resource counts - useful to estimate [Bridgecrew Cloud](https://www.bridgecrew.cloud/) costs which are charged per resource:\n  - `checkov_resource_count.sh` - counts the number of resources Checkov is scanning in the current or given directory\n  - `checkov_resource_count_all.sh` - counts the total number of resources Checkov is scanning across all given repo checkouts\n- `octopus_api.sh` - queries the [Octopus Deploy](https://octopus.com/) API\n- `sonarlint_generate_config.sh` - generates the `.sonarlint/connectedMode.json` config at the root of the Git repo from the `sonar-project.properties` file\n\nSee also [Knowledge Base notes for CI/CD](https://github.com/HariSekhon/Knowledge-Base/blob/main/ci-cd.md).\n\n### AI & IPaaS\n\n`ai/` and `ipaas/` directories:\n\n- `openai_api.sh` - queries the [OpenAI](https://openai.com/) (ChatGPT) API with authentication\n- `make_api.sh` - queries the [Make.com](https://www.make.com) API with authentication\n\n### Internet Services\n\n`internet/`, `cloudflare/`, `pingdom/`, `terraform/` directories:\n\n- Pastebins - uploads files and copies the resulting URL to your clipboard:\n  - code / text only - prompts to approve text / code before upload for safety:\n    - `pastebin.sh` - uploads a file to <https://pastebin.com>, script auto-determines which syntax highlighting to add since API doesn't auto infer\n    - `dpaste.sh` - uploads a file to <https://dpaste.com>, script auto-determines which syntax highlighting to add since API doesn't auto infer\n    - `termbin.sh` - uploads a file to <https://termbin.com> (site has no syntax highlighting)\n  - all files, multimedia or text / code - prompts to approve text / code before upload for safety:\n    - `0x0.sh` - uploads a file to <https://0x0.st> (fast)\n    - `imgur.sh` - uploads an image file to <https://imgur.com>\n    - `file.io.sh` - uploads a file to <https://file.io> with 2 weeks, single download retention\n    - `catbox.sh` - uploads a file to <https://catbox.moe/> with permanent retention (slow)\n    - `litterbox.sh` - uploads a file to <https://litterbox.catbox.moe/> with temporary retention (slow)\n- `digital_ocean_api.sh` / `doapi.sh` - queries the [Digital Ocean](https://www.digitalocean.com/) API with authentication\n  - see also the Digital Ocean CLI `doctl` (`install/install_doctl.sh`)\n- `atlassian_ip_ranges.sh` - lists [Atlassian](https://www.atlassian.com/)'s IPv4 and/or IPv6 cidr ranges via its API\n- `circleci_public_ips.sh` - lists [CircleCI](https://circleci.com) public IP addresses via dnsjson.com\n- `cloudflare_*.sh` - [Cloudflare](https://www.cloudflare.com/) API queries and reports:\n  - `cloudflare_api.sh` - queries the Cloudflare API with authentication\n  - `cloudflare_ip_ranges.sh` - lists Cloudflare's IPv4 and/or IPv6 cidr ranges via its API\n  - `cloudflare_custom_certificates.sh` - lists any custom SSL certificates in a given Cloudflare zone along with their status and expiry date\n  - `cloudflare_dns_records.sh` - lists any Cloudflare DNS records for a zone, including the type and ttl\n  - `cloudflare_dns_records_all_zones.sh` - same as above but for all zones\n  - `cloudflare_dns_record_create.sh` - creates a DNS record in the given domain\n  - `cloudflare_dns_record_update.sh` - updates a DNS record in the given domain\n  - `cloudflare_dns_record_delete.sh` - deletes a DNS record in the given domain\n  - `cloudflare_dns_record_details.sh` - lists the details for a DNS record in the given domain in JSON format for further pipe processing\n  - `cloudflare_dnssec.sh` - lists the Cloudflare DNSSec status for all zones\n  - `cloudflare_firewall_rules.sh` - lists Cloudflare Firewall rules, optionally with filter expression\n  - `cloudflare_firewall_access_rules.sh` - lists Cloudflare Firewall Access rules, optionally with filter expression\n  - `cloudflare_foreach_account.sh` - executes a templated command for each Cloudflare account, replacing the `{account_id}` and `{account_name}` in each iteration (useful for chaining with `cloudflare_api.sh`)\n  - `cloudflare_foreach_zone.sh` - executes a templated command for each Cloudflare zone, replacing the `{zone_id}` and `{zone_name}` in each iteration (useful for chaining with `cloudflare_api.sh`, used by adjacent `cloudflare_*_all_zones.sh` scripts)\n  - `cloudflare_purge_cache.sh` - purges the entire Cloudflare cache\n  - `cloudflare_ssl_verified.sh` - gets the Cloudflare zone SSL verification status for a given zone\n  - `cloudflare_ssl_verified_all_zones.sh` - same as above for all zones\n  - `cloudflare_zones.sh` - lists Cloudflare zone names and IDs (needed for writing Terraform Cloudflare code)\n- `datadog_api.sh` - queries the [DataDog](https://www.datadoghq.com/) API with authentication\n- `dnsjson.sh` - queries dnsjson.com for DNS records\n- `domains_subdomains_environments.sh` - for a given list of domains, deduplicate and print dev / staging subdomains as well as root domain for prod. Used to generate a whole bunch of Ad Tech domains and pixel tracker subdomains for a project. Combine with `markdown_columns_to_table.sh` to generate the markdown documentation for your domains and subomains per project and environment\n- `gitguardian_api.sh` - queries the [GitGuardian](https://www.gitguardian.com/) API with authentication\n- `google_maps_link.sh` - queries for a search string, returns the first hit and then generates a stable fixed place ID url to the result. Useful for sharing in documentation links to places like [HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base) Travel pages\n- `jira_api.sh` - queries [Jira](https://www.atlassian.com/software/jira) API with authentication\n- `kong_api.sh` - queries the [Kong API Gateway](https://docs.konghq.com/gateway/latest/)'s Admin API, handling authentication if enabled\n- `traefik_api.sh` - queries the [Traefik](https://traefik.io/) API, handling authentication if enabled\n- `ngrok_api.sh` - queries the [NGrok](https://ngrok.com/) API with authentication\n- `pingdom_*.sh` - [Pingdom](https://www.pingdom.com/) API queries and reports for status, latency, average response times, latency averages by hour, SMS credits, outages periods and durations over the last year etc.\n  - `pingdom_api.sh` - queries the Solarwinds [Pingdom](https://www.pingdom.com/) API with authentication\n  - `pingdom_foreach_check.sh` - executes a templated command against each Pingdom check, replacing the `{check_id}` and `{check_name}` in each iteration\n  - `pingdom_checks.sh` - show all Pingdom checks, status and latencies\n  - `pingdom_checks_outages.sh` / `pingdom_checks_outages.sh` - show one or all Pingdom checks outage histories for the last year\n  - `pingdom_checks_average_response_times.sh` - shows the average response times for all Pingdom checks for the last week\n  - `pingdom_check_latency_by_hour.sh` / `pingdom_checks_latency_by_hour.sh` - shows the average latency for one or all Pingdom checks broken down by hour of the day, over the last week\n  - `pingdom_sms_credits.sh` - gets the remaining number of Pingdom SMS credits\n- `terraform_cloud_api.sh` - queries [Terraform Cloud](https://www.terraform.io/cloud) API with authentication\n- `terraform_cloud_ip_ranges.sh` - returns the list of IP ranges for [Terraform Cloud](https://www.terraform.io/cloud) via the API, or optionally one or more of the ranges used by different functions\n- `wordpress.sh` - boots Wordpress in docker with a MySQL backend, and increases the upload_max_filesize to be able to restore a real world sized export backup\n- `wordpress_api.sh` - queries the Wordpress API with authentication\n- `wordpress_posts_without_category_tags.sh` - checks posts (articles) for categories without corresponding tags and prints the posts and their missing tags\n\n### Java\n\n`java/` directory:\n\n- `java_show_classpath.sh` - shows Java classpaths, one per line, of currently running Java programs\n- `jvm_heaps*.sh` - show all your Java heap sizes for all running Java processes, and their total MB (for performance tuning and sizing)\n- Java Decompilers:\n  - `java_decompile_jar.sh` - decompiles a Java JAR in /tmp, finds the main class and runs a Java decompiler on its main .class file using `jd_gui.sh`\n  - `jd_gui.sh` - runs Java Decompiler JD GUI, downloading its jar the first time if it's not already present\n  - `bytecode_viwer.sh` - runs Bytecode-Viewer GUI Java decompiler, downloading its jar the first time if it's not already present\n  - `cfr.sh` - runs CFR command line Java decompiler, downloading its jar the first time if it's not already present\n  - `procyon.sh` - runs Procyon command line Java decompiler, downloading its jar the first time if it's not already present\n\nSee also [Knowledge Base notes for Java](https://github.com/HariSekhon/Knowledge-Base/blob/main/java.md)\nand [JVM Performance Tuning](https://github.com/HariSekhon/Knowledge-Base/blob/main/java-jvm-performance-tuning.md).\n\n### Python\n\n`python/` directory:\n\n- `python_compile.sh` - byte-compiles Python scripts and libraries into `.pyo` optimized files\n- `python_pip_install.sh` - bulk installs PyPI modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, VirtualEnvs / Anaconda / GitHub Workflows/ Google Cloud Shell, Mac vs Linux library paths, and ignore failure option\n- `python_pip_install_if_absent.sh` - installs PyPI modules not already in Python libary path (OS or pip installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds\n- `python_pip_install_for_script.sh` - installs PyPI modules for given script(s) if not already installed. Used for dynamic individual script dependency installation in the [DevOps Python tools](https://github.com/HariSekhon/DevOps-Python-tools) repo\n- `python_pip_reinstall_all_modules.sh` - reinstalls all PyPI modules which can fix some issues\n- `pythonpath.sh` - prints all Python libary search paths, one per line\n- `python_find_library_path.sh` - finds directory where a PyPI module is installed - without args finds the Python library base\n- `python_find_library_executable.sh` - finds directory where a PyPI module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your `$PATH`, where `which` won't help)\n- `python_find_unused_pip_modules.sh` - finds PyPI modules that aren't used by any programs in the current directory tree\n- `python_find_duplicate_pip_requirements.sh` - finds duplicate PyPI modules listed for install under the directory tree (useful for deduping module installs in a project and across submodules)\n- `python_translate_import_module.sh` - converts Python import modules to PyPI module names, used by `python_pip_install_for_script.sh`\n- `python_translate_module_to_import.sh` - converts PyPI module names to Python import names, used by `python_pip_install_if_absent.sh` and `python_find_unused_pip_modules.sh`\n- `python_pyinstaller.sh` - creates [PyInstaller](https://pypi.org/project/pyinstaller/) self-contained Python programs with Python interpreter and all PyPI modules included\n- `python_pypi_versions.sh` - prints all available versions of a given PyPi module using the API\n\nSee also [Knowledge Base notes for Python](https://github.com/HariSekhon/Knowledge-Base/blob/main/python.md).\n\n### Perl\n\n`perl/` directory:\n\n- `perl_cpanm_install.sh` - bulk installs CPAN modules from mix of arguments / file lists / stdin, accounting for User vs System installs, root vs user sudo, [Perlbrew](https://perlbrew.pl/) / Google Cloud Shell environments, Mac vs Linux library paths, ignore failure option, auto finds and reads build failure log for quicker debugging showing root cause error in CI builds logs etc\n- `perl_cpanm_install_if_absent.sh` - installs CPAN modules not already in Perl libary path (OS or CPAN installed) for faster installations only where OS packages are already providing some of the modules, reducing time and failure rates in CI builds\n- `perl_cpanm_reinstall_all.sh` - re-installs all CPAN modules. Useful for trying to recompile XS modules on Macs after migration assistant from an Intel Mac to an ARM Silicon Mac leaves your home XS libraries broken as they're built for the wrong architecture\n- `perlpath.sh` - prints all Perl libary search paths, one per line\n- `perl_find_library_path.sh` - finds directory where a CPAN module is installed - without args finds the Perl library base\n- `perl_find_library_executable.sh` - finds directory where a CPAN module's CLI program is installed (system vs user, useful when it gets installed to a place that isn't in your `$PATH`, where `which` won't help)\n- `perl_find_unused_cpan_modules.sh` - finds CPAN modules that aren't used by any programs in the current directory tree\n- `perl_find_duplicate_cpan_requirements.sh` - finds duplicate CPAN modules listed for install more than once under the directory tree (useful for deduping module installs in a project and across submodules)\n- `perl_generate_fatpacks.sh` - creates [Fatpacks](https://metacpan.org/pod/App::FatPacker) - self-contained Perl programs with all CPAN modules built-in\n\nSee also [Knowledge Base notes for Perl](https://github.com/HariSekhon/Knowledge-Base/blob/main/perl.md).\n\n### Golang\n\n`packages/` directory:\n\n- `golang_install.sh` - bulk installs Golang modules from mix of arguments / file lists / stdin\n- `golang_install_if_absent.sh` - same as above but only if the package binary isn't already available in `$PATH`\n- `golang_rm_binaries.sh` - deletes binaries of the same name adjacent to `.go` files. Doesn't delete your `bin/` etc as these are often real deployed applications rather than development binaries\n\n### Diagrams\n\nUsed in [HariSekhon/Diagrams-as-Code](https://github.com/HariSekhon/Diagrams-as-Code)\n\n`diagrams/` directory:\n\n- `d2.sh` - generates a D2lang diagram, using its shebang if present for themes etc, and then opens the resulting image\n- `d2_generate_diagrams.sh` - generates all D2lang `.d2` diagrams found under the current or given directory, git reverting or deleting those that didn't generate properly (to work around [d2lang issue #2367](https://github.com/terrastruct/d2/issues/2367))\n- `mermaidjs_generate_diagrams.sh` - generates all MermaidJS `.mmd` diagrams found under the current or given directory, git reverting or deleting those that didn't generate properly\n- `python_mingrammer_generate_diagrams.sh` - generates all Python Mingrammer `.py` diagrams found under the current or given directory\n\n### Media\n\n`media/` directory:\n\n#### Images\n\n- `image_shrink.sh` - shrinks an image by resizing it (default 50%) to be able to upload it against limits on some websites.\n   Shows the before and after stats and automatically opens the image to check it\n- `image_reduce_quality.sh` - shrinks an image size by reducing its quality (default to 80%) to be able to upload it against limits on some websites.\n   Shows the before and after stats and automatically opens the image to check it\n- `image_trim_pixels.sh` - trims N pixels off one of the sides of an image. Useful to tweak screenshots before sharing them\n- `image_join_vertical.sh` - joins two images top and bottom after matching their widths so they align correctly\n- `image_join_horizontal.sh` - joins two images left and right after matching their heights so they align correctly\n- `imageopen.sh` - opens the given image file using whatever available tool is found on Linux or Mac\n- `d2.sh` - generates [D2lang diagram](https://github.com/HariSekhon/Diagrams-as-Code) and then opens the resulting image\n- `avif_to_png.sh` - converts an Avif image to PNG to be usable on websites that don't support Webp images like LinkedIn\n- `svg_to_png.sh` - converts an SVG image to PNG to be usable on websites that don't support SVG images like LinkedIn, Medium or Reddit\n- `webp_to_png.sh` - converts a Webp image to PNG to be usable on websites that don't support Webp images like Medium. Tries 2 different tools for webp unlike the next generic converter\n- `image_to_png.sh` - try to convert any image to PNG to be usable on fussy websites such as LinkedIn, Medium or Reddit\n\n#### Terminal Gif Capture\n\nEach of these three scripts creates an animated Git from running terminal commands and then opens the resulting gif.\n\n- `ttygif.sh` - uses `ttyrec` and `ttygif`\n- `asciinema.sh` - uses  `asciinema` and `agg`\n- `terminalizer.sh` - uses Terminalizer\n\n#### Audio\n\n- `mp3_set_artist.sh` / `mp3_set_album.sh` - set the artist / album tag for all mp3 files under given directories. Useful for grouping artists/albums and audiobook author/books (eg. for correct importing into Mac's Books.app)\n- `mp3_set_track_name.sh` - set the track name metadata for mp3 files under given directories to follow their filenames. Useful for correctly displaying audiobook progress / chapters etc.\n- `mp3_set_track_order.sh` - set the track order metadata for mp3 files under given directories to follow the lexical file naming order. Useful for correctly ordering album songs and audiobook chapters (eg. for Mac's Books.app). Especially useful for enforcing global ordering on multi-CD audiobooks after grouping into a single audiobook using `mp3_set_album.sh` (otherwise default track numbers in each CD interleave in Mac's Books.app)\n\n#### Video\n\n- `avi_to_mp4.sh` - converts avi files to mp4 using ffmpeg. Useful to be able to play videos on devices like smart TVs that may not recognize newer codecs otherwise\n- `mkv_to_mp4.sh` - converts mkv files to mp4 using ffmpeg. Same use case as above\n- `youtube_download_video.sh` - downloads a YouTube video to mp4 with maximum quality and compatibility usng yt-dlp\n  - `facebook_download_video.sh` - same as above for Facebook\n  - `twitter_download_video.sh` - same as above for Twitter / X\n  - `x_download_video.sh` - same as above for X / Twitter\n- `youtube_download_channel.sh` - downloads all videos from a given YouTube channel using yt-dlp\n- `video_to_720p_mp4` - converts one or more video files to 720p mp4 format using ffmpeg. Useful to make good trade-off of quality vs size for social media sharing\n- `vidopen.sh` - opens the given video file using whatever available tool is found on Linux or Mac\n\nSee also [Knowledge Base notes for MultiMedia](https://github.com/HariSekhon/Knowledge-Base/blob/main/multimedia.md).\n\n### Spotify\n\n40+ [Spotify](https://www.spotify.com/) API scripts (used extensively to manage my [Spotify-Playlists](https://github.com/HariSekhon/Spotify-Playlists) repo).\n\n`spotify/` directory:\n\n- `spotify_playlists*.sh` - list playlists in either `<id> <name>` or JSON format\n- `spotify_playlist_tracks*.sh` - gets playlist contents as track URIs / `Artists - Track` / CSV format - useful for backups or exports between music systems\n- `spotify_backup.sh` - backs up all Spotify playlists as well as the ordered list of playlists\n- `spotify_backup_playlist*.sh` - backs up Spotify playlists to local files in both human readable `Artist - Track` format and Spotify URI format for easy restores or adding to new playlists\n- `spotify_backup_artists_followed.sh` - backs up the list of artists followed to Spotify URI and Names\n- `spotify_search*.sh` - searches Spotify's library for tracks / albums / artists getting results in human readable format, JSON, or URI formats for easy loading to Spotify playlists\n- `spotify_release_year.sh` - searches for a given track or album and finds the original release year\n- `spotify_uri_json.sh` - takes a Spotify URI and dumps its JSON for inspection. You can pass one of the following formats: `spotify:<type>:<alphanumeric_ID>`, `http://open.spotify.com/<type>/<alphanumeric_ID>`, `<alphanumeric_ID>`\n- `spotify_uri_to_name.sh` - converts Spotify track / album / artist URIs to human readable `Artist - Track` or CSV format. Takes Spotify URIs, URL links or just IDs similar to `spotify_uri_json.sh` above. Reads URIs from files or standard input\n- `spotify_create_playlist.sh` - creates a Spotify playlist, either public or private\n- `spotify_rename_playlist.sh` - renames a Spotify playlist\n- `spotify_set_playlists_public.sh` / `spotify_set_playlists_private.sh` - sets one or more given Spotify playlists to public / private\n- `spotify_add_to_playlist.sh` - adds tracks to a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input. Can be combined with many other tools listed here which output Spotify URIs, or appended from other playlists. Can also be used to restore a spotify playlist from backups\n- `spotify_delete_from_playlist.sh` - deletes tracks from a given playlist. Takes a playlist name or ID and Spotify URIs in any form from files or standard input, optionally prefixed with a track position to remove only specific occurrences (useful for removing duplicates from playlists)\n- `spotify_delete_from_playlist_if_in_other_playlists.sh` - deletes tracks from a given playlist if their URIs are found in the subsequently given playlists\n- `spotify_delete_from_playlist_if_track_in_other_playlists.sh` - deletes tracks from a given playlist if their 'Artist - Track' name match are found in the subsequently given playlists (less accurate than exact URI deletion above)\n- `spotify_duplicate_uri_in_playlist.sh` - finds duplicate Spotify URIs in a given playlist (these are guaranteed exact duplicate matches), returns all but the first occurrence and optionally their track positions (zero-indexed to align with the Spotify API for easy chaining with other tools)\n- `spotify_duplicate_tracks_in_playlist.sh` - finds duplicate Spotify tracks in a given playlist (these are idential `Artist - Track` name matches, which may be from different albums / singles)\n- `spotify_delete_duplicate_tracks_in_playlist.sh` - deletes duplicate Spotify tracks (name matched) in a given playlist using `spotify_duplicate_tracks_in_playlist.sh` and `spotify_delete_from_playlist.sh`\n- `spotify_delete_duplicate_track_uris_in_playlist.sh` - deletes duplicate Spotify URI tracks (identical) in a given playlist using `spotify_duplicate_uri_in_playlist.sh` and `spotify_delete_from_playlist.sh`\n- `spotify_delete_any_duplicates_in_playlist.sh` - calls both of the above scripts to first get rid of duplicate URIs and then remove any other duplicates by track name matches\n- `spotify_playlist_tracks_uri_in_year.sh` - finds track URIs in a playlist where their original release date is in a given year or decade (by regex match). This has to do a secondary Spotify track search lookup by name and relies on `normalize_tracknames.pl` from [HariSekhon/Spotify-tools](https://github.com/HariSekhon/Spotify-tools) being built and in the `$PATH`\n- `spotify_playlist_tracks_uri_by_year.sh` -  returns track URIs from the given Spotify playlist for a specific year or range of years. Useful for filtering tracks to add to my best of each decade playlists. More efficient than `spotify_playlist_tracks_uri_in_year.sh`, without dependency on [HariSekhon/Spotify-tools](https://github.com/HariSekhon/Spotify-tools), but it only uses the year of the track version, which if an album track may not be the same year if it was first released as a single earlier\n- `spotify_playlist_tracks_uri_batch_by_year.sh` - Returns all track URIs from the given Spotify playlist(s) grouped by year or decade. Copies each batch to the clipboard, prints to stdout, and prompts to continue before printing the next batch. Useful for filtering tracks to add to my best of each year or decade playlists\n- `spotify_playlist_uri_offset.sh` - finds the offset of a given track URI in a given playlist, useful to find positions to resume processing a large playlist\n- `spotify_top_artists*.sh` - lists your top artists in URI or human readable format\n- `spotify_top_tracks*.sh` - lists top tracks in URI or human readable format\n- `spotify_playlist_top_artists.sh` - returns the top artists for a given Spotify playlist by counting unique track names for each artist. If HariSekhon/Spotify-tools is in the $PATH it uses normalize_tracknames.pl for greater accuracy to collapse multiple versions such as Radio Edit and Album Version to only count that same song once\n- `spotify_liked_tracks*.sh` - lists your `Liked Songs` in URI or human readable formats\n- `spotify_liked_artists*.sh` - list artists from `Liked Songs` in URI or human readable formats\n- `spotify_artists_followed*.sh` - lists all followed artists in URI or human readable formats\n- `spotify_artist_tracks.sh` - gets all track URIs for a given artist, from both albums and single for chain loading to playlists\n- `spotify_follow_artists.sh` - follows artists for the given URIs from files or standard input\n- `spotify_follow_top_artists.sh` - follows all artists in your current Spotify top artists list\n- `spotify_follow_liked_artists.sh` - follows artists with N or more tracks in your `Liked Songs`\n- `spotify_set_tracks_uri_to_liked.sh` - sets a list of spotify track URIs to 'Liked' so they appear in the `Liked Songs` playlist. Useful for marking all the tracks in your best playlists as favourite tracks, or for porting historical `Starred` tracks to the newer `Liked Songs`\n- `spotify_foreach_playlist.sh` - executes a templated command against all playlists, replacing `{playlist}` and `{playlist_id}` in each iteration\n- `spotify_playlist_name_to_id.sh` / `spotify_playlist_id_to_name.sh` - convert playlist names <=> IDs\n- `spotify_playlist_snapshot_id.sh` - returns the Snapshot ID of a given Spotify playlist, useful for detecting whether a playlist has changed or not to skip re-downloading it\n- `spotify_api_token.sh` - gets a Spotify authentication token using either [Client Credentials](https://developer.spotify.com/documentation/general/guides/authorization-guide/#client-credentials-flow) or [Authorization Code](https://developer.spotify.com/documentation/general/guides/authorization-guide/#authorization-code-flow) authentication flows, the latter being able to read/modify private user data, automatically used by `spotify_api.sh`\n- `spotify_api.sh` - query any Spotify [API](https://developer.spotify.com/documentation/web-api/reference/) endpoint with authentication, used by adjacent spotify scripts\n\n### More Linux & Mac\n\n`bin/`, `install/`, `packages/`, `setup/` directories:\n\n- [Linux](https://en.wikipedia.org/wiki/Linux) / [Mac](https://en.wikipedia.org/wiki/MacOS) systems administration scripts:\n  - `install/` - installation scripts for various OS packages (RPM, Deb, Apk) for various Linux distros ([Redhat RHEL](https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux) / [CentOS](https://www.centos.org/) / [Fedora](https://getfedora.org/), [Debian](https://www.debian.org/) / [Ubuntu](https://ubuntu.com/), [Alpine](https://alpinelinux.org/))\n  - install if absent scripts for Python, Perl, Ruby, NodeJS and Golang packages - good for minimizing the number of source code installs by first running the OS install scripts and then only building modules which aren't already detected as installed (provided by system packages), speeding up builds and reducing the likelihood of compile failures\n  - install scripts for tarballs, Golang binaries, random 3rd party installers, [Jython](https://www.jython.org/) and build tools like [Gradle](https://gradle.org/) and [SBT](https://www.scala-sbt.org/) for when Linux distros don't provide packaged versions or where the packaged versions are too old\n  - `packages/` - OS / Distro Package Management:\n    - `install_packages.sh` - installs package lists from arguments, files or stdin on major linux distros and Mac, detecting the package manager and invoking the right install commands, with `sudo` if not root. Works on [RHEL](https://www.redhat.com/en) / [CentOS](https://www.centos.org/) / [Fedora](https://getfedora.org/), [Debian](https://www.debian.org/) / [Ubuntu](https://ubuntu.com/), [Alpine](https://alpinelinux.org/), and [Mac Homebrew](https://brew.sh/). Leverages and supports all features of the distro / OS specific install scripts listed below\n    - `install_packages_if_absent.sh` - installs package lists if they're not already installed, saving time and minimizing install logs / CI logs, same support list as above\n    - Redhat RHEL / CentOS:\n      - `yum_install_packages.sh` / `yum_remove_packages.sh` - installs RPM lists from arguments, files or stdin. Handles Yum + Dnf behavioural differences, calls `sudo` if not root, auto-attempts variations of python/python2/python3 package names. Avoids yum slowness by checking if rpm is installed before attempting to install it, accepts `NO_FAIL=1` env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across RHEL/CentOS/Fedora versions)\n      - `yum_install_packages_if_absent.sh` - installs RPMs only if not already installed and not a metapackage provided by other packages (eg. `vim` metapackage provided by `vim-enhanced`), saving time and minimizing install logs / CI logs, plus all the features of `yum_install_packages.sh` above\n      - `yum_upgrade_packages_if_outdated.sh` - upgrades RPMs only if they're outdated\n      - `rpms_filter_installed.sh` / `rpms_filter_not_installed.sh` - pipe filter packages that are / are not installed for easy script piping\n    - Debian / Ubuntu:\n      - `apt_install_packages.sh` / `apt_remove_packages.sh` - installs Deb package lists from arguments, files or stdin. Auto calls `sudo` if not root, accepts `NO_FAIL=1` env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Debian/Ubuntu distros/versions)\n      - `apt_install_packages_if_absent.sh` - installs Deb packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of `apt_install_packages.sh` above\n      - `apt_upgrade_packages_if_outdated.sh` - upgrades Deb packages only if they're outdated\n      - `apt_wait.sh` - blocking wait on concurrent apt locks to avoid failures and continue when available, mimicking yum's waiting behaviour rather than error'ing out\n      - `debs_filter_installed.sh` / `debs_filter_not_installed.sh` - pipe filter packages that are / are not installed for easy script piping\n    - Alpine:\n      - `apk_install_packages.sh` / `apk_remove_packages.sh` - installs Alpine apk package lists from arguments, files or stdin. Auto calls `sudo` if not root, accepts `NO_FAIL=1` env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across Alpine versions)\n      - `apk_install_packages_if_absent.sh` - installs Alpine apk packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of `apk_install_packages.sh` above\n      - `apk_upgrade_packages_if_outdated.sh` - upgrades Alpine apk packages only if they're outdated\n      - `apk_filter_installed.sh` / `apk_filter_not_installed.sh` - pipe filter packages that are / are not installed for easy script piping\n    - Mac:\n      - `brew_install_packages.sh` / `brew_remove_packages.sh` - installs Mac Hombrew package lists from arguments, files or stdin. Accepts `NO_FAIL=1` env var to ignore unavailable / changed package names (useful for optional packages or attempts for different package names across versions)\n      - `brew_install_packages_if_absent.sh` - installs Mac Homebrew packages only if not already installed, saving time and minimizing install logs / CI logs, plus all the features of `brew_install_packages.sh` above\n      - `brew_upgrade_packages_if_outdated.sh` - upgrades Mac Homebrew packages only if they're outdated\n      - `brew_filter_installed.sh` / `brew_filter_not_installed.sh` - pipe filter packages that are / are not installed for easy script piping\n      - `brew_package_owns.sh` - finds which brew package owns a given filename argument\n- all builds across all my GitHub repos now `make system-packages` before `make pip` / `make cpan` to shorten how many packages need installing, reducing chances of build failures\n\n### Builds, Languages & Linting\n\n`bin/`, `checks/`, `cicd/` or language specific directories:\n\n- `lint.sh` - lints one or more files, auto-determines the file types, parses lint headers and calls appropriate scripts and tools. Integrated with my custom `.vimrc`\n- `run.sh` - runs one or more files, auto-determines the file types, any run or arg headers and executes each file using the appropriate script or CLI tool. Integrated with my custom `.vimrc`\n- `check_*.sh` - extensive collection of generalized tests - these run against all my GitHub repos via [CI](https://harisekhon.github.io/CI-CD/). Some examples:\n\n  - Programming language linting:\n\n    - [Python](https://www.python.org/) (syntax, pep8, byte-compiling, reliance on asserts which can be disabled at runtime, except/pass etc.)\n    - [Perl](https://www.perl.org/)\n    - [Java](https://www.java.com/en/)\n    - [Scala](https://www.scala-lang.org/)\n    - [Ruby](https://www.ruby-lang.org/en/)\n    - [Bash](https://www.gnu.org/software/bash/) / Shell\n    - Misc (whitespace, custom code checks etc.)\n\n  - Build System, Docker & CI linting:\n\n    - [Make](https://www.gnu.org/software/make/)\n    - [Maven](https://maven.apache.org/)\n    - [SBT](https://www.scala-sbt.org/)\n    - [Gradle](https://gradle.org/)\n    - [Travis CI](https://travis-ci.org/)\n    - [Circle CI](https://circleci.com/)\n    - [GitLab CI](https://docs.gitlab.com/ee/ci/)\n    - [Concourse CI](https://concourse-ci.org/)\n    - [Codefresh CI](https://codefresh.io/)\n    - [Dockerfiles](https://docs.docker.com/engine/reference/builder/)\n    - [Docker Compose](https://docs.docker.com/compose/)\n    - [Vagrantfiles](https://www.vagrantup.com/docs/vagrantfile)\n\n## Individual Setup Parts\n\nOptional, only if you don't do the full `make install`.\n\nInstall only OS system package dependencies and [AWS CLI](https://aws.amazon.com/cli/) via Python Pip (doesn't symlink anything to `$HOME`):\n\n```shell\nmake\n```\n\nAdds sourcing to `.bashrc` and `.bash_profile` and symlinks dot config files to `$HOME` (doesn't install OS system package dependencies):\n\n```shell\nmake link\n```\n\nundo via\n\n```shell\nmake unlink\n```\n\nInstall only OS system package dependencies (doesn't include [AWS CLI](https://aws.amazon.com/cli/) or Python packages):\n\n```shell\nmake system-packages\n```\n\nInstall [AWS CLI](https://aws.amazon.com/cli/):\n\n```shell\nmake aws\n```\n\nInstall [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/):\n\n```shell\nmake azure\n```\n\nInstall [GCP GCloud SDK](https://cloud.google.com/sdk) (includes CLI):\n\n```shell\nmake gcp\n```\n\nInstall [GCP GCloud Shell](https://cloud.google.com/shell) environment (sets up persistent OS packages and all home directory configs):\n\n```shell\nmake gcp-shell\n```\n\nInstall generically useful Python CLI tools and modules (includes [AWS CLI](https://aws.amazon.com/cli/), autopep8 etc):\n\n```shell\nmake python\n```\n\n### Full Help\n\n```shell\n> make help\n\n Usage:\n\n  Common Options:\n\n    make help                   show this message\n    make build                  installs all dependencies - OS packages and any language libraries via native tools eg. pip, cpanm, gem, go etc that are not available via OS packages\n    make build-retry            retries 'make build' x 3 until success to try to mitigate temporary upstream repo failures triggering false alerts in CI systems\n    make ci                     prints env, then runs 'build-retry' for more resilient CI builds with debugging\n    make printenv               prints environment variables, CPU cores, OS release, $PWD, Git branch, hashref etc. Useful for CI debugging\n    make system-packages        installs OS packages only (detects OS via whichever package manager is available)\n    make test                   run tests\n    make clean                  removes compiled / generated files, downloaded tarballs, temporary files etc.\n\n    make submodules             initialize and update submodules to the right release (done automatically by build / system-packages)\n    make init                   same as above, often useful to do in CI systems to get access to additional submodule provided targets such as 'make ci'\n\n    make cpan                   install any modules listed in any cpan-requirements.txt files if not already installed\n\n    make pip                    install any modules listed in any requirements.txt files if not already installed\n\n    make python-compile         compile any python files found in the current directory and 1 level of subdirectory\n    make pycompile\n\n    make github                 open browser at github project\n    make readme                 open browser at github's README\n    make github-url             print github url and copy to clipboard\n    make status                 open browser at Github CI Builds overview Status page for all projects\n\n    make ls                     print list of code files in project\n    make wc                     show counts of files and lines\n\n  Repo specific options:\n\n    make install                builds all script dependencies, installs AWS CLI, symlinks all config files to $HOME and adds sourcing of bash profile\n\n    make link                   symlinks all config files to $HOME and adds sourcing of bash profile\n    make unlink                 removes all symlinks pointing to this repo's config files and removes the sourcing lines from .bashrc and .bash_profile\n\n    make python-desktop         installs all Python Pip packages for desktop workstation listed in setup/pip-packages-desktop.txt\n    make perl-desktop           installs all Perl CPAN packages for desktop workstation listed in setup/cpan-packages-desktop.txt\n    make ruby-desktop           installs all Ruby Gem packages for desktop workstation listed in setup/gem-packages-desktop.txt\n    make golang-desktop         installs all Golang packages for desktop workstation listed in setup/go-packages-desktop.txt\n    make nodejs-desktop         installs all NodeJS packages for desktop workstation listed in setup/npm-packages-desktop.txt\n\n    make desktop                installs all of the above + many desktop OS packages listed in setup/\n\n    make mac-desktop            all of the above + installs a bunch of major common workstation software packages like Ansible, Terraform, MiniKube, MiniShift, SDKman, Travis CI, CCMenu, Parquet tools etc.\n    make linux-desktop\n\n    make ls-scripts             print list of scripts in this project, ignoring code libraries in lib/ and .bash.d/\n\n    make github-cli             installs GitHub CLI\n    make kubernetes             installs Kubernetes kubectl and kustomize to ~/bin/\n    make terraform              installs Terraform to ~/bin/\n    make vim                    installs Vundle and plugins\n    make tmux                   installs TMUX TPM and plugin for kubernetes context\n    make ccmenu                 installs and (re)configures CCMenu to watch this and all other major HariSekhon GitHub repos\n    make status                 open the Github Status page of all my repos build statuses across all CI platforms\n\n    make aws                    installs AWS CLI tools\n    make azure                  installs Azure CLI\n    make gcp                    installs Google Cloud SDK\n    make digital-ocean          installs Digital Ocean CLI\n\n    make aws-shell              sets up AWS Cloud Shell: installs core packages and links configs\n                                (maintains itself across future Cloud Shells via .aws_customize_environment hook)\n    make gcp-shell              sets up GCP Cloud Shell: installs core packages and links configs\n                                (maintains itself across future Cloud Shells via .customize_environment hook)\n    make azure-shell            sets up Azure Cloud Shell (limited compared to gcp-shell, doesn't install OS packages since there is no sudo)\n\nNow exiting usage help with status code 3 to explicitly prevent silent build failures from stray 'help' arguments\nmake: *** [help] Error 3\n```\n\n(`make help` exits with error code 3 like most of my programs to differentiate from build success to make sure a stray `help` argument doesn't cause silent build failure with exit code 0)\n\n## Star History\n\n[![Star History Chart](https://api.star-history.com/svg?repos=HariSekhon/DevOps-Bash-tools&type=Date)](https://star-history.com/#HariSekhon/DevOps-Bash-tools&Date)\n\n[git.io/bash-tools](https://git.io/bash-tools)\n\n## More Core Repos\n\n<!-- OTHER_REPOS_START -->\n\n### Knowledge\n\n[![Knowledge-Base](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Knowledge-Base&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Knowledge-Base)\n[![Diagrams-as-Code](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Diagrams-as-Code&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Diagrams-as-Code)\n\n<!--\n\nNot support on GitHub Markdown:\n\n<iframe src=\"https://raw.githubusercontent.com/HariSekhon/HariSekhon/main/knowledge.md\" width=\"100%\" height=\"500px\"></iframe>\n\nDoes nothing:\n\n<embed src=\"https://raw.githubusercontent.com/HariSekhon/HariSekhon/main/knowledge.md\" width=\"100%\" height=\"500px\" />\n\n-->\n\n### DevOps Code\n\n[![DevOps-Bash-tools](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=DevOps-Bash-tools&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/DevOps-Bash-tools)\n[![DevOps-Python-tools](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=DevOps-Python-tools&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/DevOps-Python-tools)\n[![DevOps-Perl-tools](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=DevOps-Perl-tools&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/DevOps-Perl-tools)\n[![DevOps-Golang-tools](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=DevOps-Golang-tools&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/DevOps-Golang-tools)\n\n<!--\n[![Gist Card](https://github-readme-stats-fast.vercel.app/api/gist?id=f8f551332440f1ca8897ff010e363e03)](https://gist.github.com/HariSekhon/f8f551332440f1ca8897ff010e363e03)\n-->\n\n### Containerization\n\n[![Kubernetes-configs](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Kubernetes-configs&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Kubernetes-configs)\n[![Dockerfiles](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Dockerfiles&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Dockerfiles)\n\n### CI/CD\n\n[![GitHub-Actions](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=GitHub-Actions&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/GitHub-Actions)\n[![Jenkins](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Jenkins&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Jenkins)\n\n### Databases - DBA - SQL\n\n[![SQL-scripts](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=SQL-scripts&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/SQL-scripts)\n\n### DevOps Reloaded\n\n[![HAProxy-configs](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=HAProxy-configs&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/HAProxy-configs)\n[![Terraform](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Terraform&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Terraform)\n[![Packer](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Packer&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Packer)\n[![Ansible](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Ansible&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Ansible)\n[![Environments](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Environments&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Environments)\n\n### Monitoring\n\n[![Nagios-Plugins](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Nagios-Plugins&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Nagios-Plugins)\n[![Nagios-Plugin-Kafka](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Nagios-Plugin-Kafka&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Nagios-Plugin-Kafka)\n[![Prometheus](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Prometheus&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Prometheus)\n\n### Templates\n\n[![Templates](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Templates&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Templates)\n[![Template-repo](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Template-repo&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Template-repo)\n\n### Desktop\n\n[![TamperMonkey](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=TamperMonkey&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/TamperMonkey)\n[![Hammerspoon](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Hammerspoon&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Hammerspoon)\n[![MPV-Scripts](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=MPV-Scripts&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/MPV-Scripts)\n\n### Spotify\n\n[![Spotify-tools](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Spotify-tools&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Spotify-tools)\n[![Spotify-playlists](https://github-readme-stats-fast.vercel.app/api/pin/?username=HariSekhon&repo=Spotify-playlists&theme=ambient_gradient&description_lines_count=3)](https://github.com/HariSekhon/Spotify-playlists)\n\nThe rest of my original source repos are\n[here](https://github.com/HariSekhon?tab=repositories&q=&type=source&language=&sort=stargazers).\n\nPre-built Docker images are available on my [DockerHub](https://hub.docker.com/u/harisekhon/)\nand can be re-generated using the my [Dockerfiles](https://github.com/HariSekhon/Dockerfiles) repo.\n\n<!-- OTHER_REPOS_END -->\n"
  },
  {
    "path": "STARCHARTS.md",
    "content": "# GitHub StarCharts\n\n![Original Repos](https://img.shields.io/badge/Repos-20-blue?logo=github)\n![Stars](https://img.shields.io/badge/Stars-7542-blue?logo=github)\n![Forks](https://img.shields.io/badge/Forks-2603-blue?logo=github)\n![Followers](https://img.shields.io/badge/Followers-1568-blue?logo=github)\n[![Azure DevOps Profile](https://img.shields.io/badge/Azure%20DevOps-HariSekhon-0078D7?logo=azure%20devops)](https://dev.azure.com/harisekhon/GitHub)\n[![GitHub Profile](https://img.shields.io/badge/GitHub-HariSekhon-2088FF?logo=github)](https://github.com/HariSekhon)\n[![GitLab Profile](https://img.shields.io/badge/GitLab-HariSekhon-FCA121?logo=gitlab)](https://gitlab.com/HariSekhon)\n[![BitBucket Profile](https://img.shields.io/badge/BitBucket-HariSekhon-0052CC?logo=bitbucket)](https://bitbucket.org/HariSekhon)\n\n[![GitStar Ranking Profile](https://img.shields.io/badge/GitStar%20Ranking-HariSekhon-blue?logo=github)](https://gitstar-ranking.com/HariSekhon)\n\n[git.io/hari-starcharts](https://git.io/hari-starcharts) generated by `github_generate_starcharts.md.sh` in [HariSekhon/DevOps-Bash-tools](https://github.com/HariSekhon/DevOps-Bash-tools)\n\n---\n## Hari Sekhon - DevOps Bash Tools\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/DevOps-Bash-tools?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/DevOps-Bash-tools?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/network)\n\n1000+ DevOps Bash Scripts - AWS, GCP, Kubernetes, Docker, CI/CD, APIs, SQL, PostgreSQL, MySQL, Hive, Impala, Kafka, Hadoop, Jenkins, GitHub, GitLab, BitBucket, Azure DevOps, TeamCity, Spotify, MP3, LDAP, Code/Build Linting, pkg mgmt for Linux, Mac, Python, Perl, Ruby, NodeJS, Golang, Advanced dotfiles: .bashrc, .vimrc, .gitconfig, .screenrc, tmux..\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/DevOps-Bash-tools.svg)](https://starchart.cc/HariSekhon/DevOps-Bash-tools)\n\n---\n## Dockerfiles for DevOps, CI/CD, Big Data & NoSQL\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Dockerfiles)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Dockerfiles?logo=github)](https://github.com/HariSekhon/Dockerfiles/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Dockerfiles?logo=github)](https://github.com/HariSekhon/Dockerfiles/network)\n\n50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Dockerfiles.svg)](https://starchart.cc/HariSekhon/Dockerfiles)\n\n---\n## Advanced Nagios Plugins Collection\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Nagios-Plugins)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Nagios-Plugins?logo=github)](https://github.com/HariSekhon/Nagios-Plugins/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Nagios-Plugins?logo=github)](https://github.com/HariSekhon/Nagios-Plugins/network)\n\n450+ AWS, Hadoop, Cloud, Kafka, Docker, Elasticsearch, RabbitMQ, Redis, HBase, Solr, Cassandra, ZooKeeper, HDFS, Yarn, Hive, Presto, Drill, Impala, Consul, Spark, Jenkins, Travis CI, Git, MySQL, Linux, DNS, Whois, SSL Certs, Yum Security Updates, Kubernetes, Cloudera etc...\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Nagios-Plugins.svg)](https://starchart.cc/HariSekhon/Nagios-Plugins)\n\n---\n## Hari Sekhon - DevOps Python Tools\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/DevOps-Python-tools)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/DevOps-Python-tools?logo=github)](https://github.com/HariSekhon/DevOps-Python-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/DevOps-Python-tools?logo=github)](https://github.com/HariSekhon/DevOps-Python-tools/network)\n\n80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/DevOps-Python-tools.svg)](https://starchart.cc/HariSekhon/DevOps-Python-tools)\n\n---\n## Kubernetes configs\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Kubernetes-configs)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Kubernetes-configs?logo=github)](https://github.com/HariSekhon/Kubernetes-configs/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Kubernetes-configs?logo=github)](https://github.com/HariSekhon/Kubernetes-configs/network)\n\nAdvanced Kubernetes YAML configs - Best Practices, Tips & Tricks, Production-Ready Checklist - experience from several production environments. AWS, GCP, Azure, ArgoCD, GKE, EKS, AKS, Nginx, Traefik, Kong, Cert Manager, CI/CD, Jenkins, Artifactory, TeamCity, GitHub Actions, Cloud SQL, FluxCD, Spinnaker, Selenium Grid, Moon, Helm + Kustomize\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Kubernetes-configs.svg)](https://starchart.cc/HariSekhon/Kubernetes-configs)\n\n---\n## SQL Scripts\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/SQL-scripts)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/SQL-scripts?logo=github)](https://github.com/HariSekhon/SQL-scripts/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/SQL-scripts?logo=github)](https://github.com/HariSekhon/SQL-scripts/network)\n\n100+ SQL Scripts - PostgreSQL, MySQL, Google BigQuery, MariaDB, AWS Athena. DBA, Analytics, DevOps, performance engineering. Google BigQuery ML machine learning classification.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/SQL-scripts.svg)](https://starchart.cc/HariSekhon/SQL-scripts)\n\n---\n## Advanced HAProxy Configs for Big Data, NoSQL, Web and Infrastructure technologies\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/HAProxy-configs)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/HAProxy-configs?logo=github)](https://github.com/HariSekhon/HAProxy-configs/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/HAProxy-configs?logo=github)](https://github.com/HariSekhon/HAProxy-configs/network)\n\n80+ HAProxy Configs for Hadoop, Big Data, NoSQL, Docker, Kubernetes, Elasticsearch, SolrCloud, HBase, MySQL, PostgreSQL, Apache Drill, Hive, Presto, Impala, Hue, ZooKeeper, SSH, RabbitMQ, Redis, Riak, Cloudera, OpenTSDB, InfluxDB, Prometheus, Kibana, Graphite, Rancher etc.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/HAProxy-configs.svg)](https://starchart.cc/HariSekhon/HAProxy-configs)\n\n---\n## Hari Sekhon - Diagrams-as-Code\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Diagrams-as-Code)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Diagrams-as-Code?logo=github)](https://github.com/HariSekhon/Diagrams-as-Code/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Diagrams-as-Code?logo=github)](https://github.com/HariSekhon/Diagrams-as-Code/network)\n\nCloud & DevOps Architecture Diagrams-as-Code in Python and D2 languages\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Diagrams-as-Code.svg)](https://starchart.cc/HariSekhon/Diagrams-as-Code)\n\n---\n## Code & Config Templates\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Templates)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Templates?logo=github)](https://github.com/HariSekhon/Templates/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Templates?logo=github)](https://github.com/HariSekhon/Templates/network)\n\n100+ DevOps Code & Config templates for Kubernetes, AWS, GCP, Terraform, Docker, Packer, Jenkins, CircleCI, GitHub Actions, Lambda, AWS CodeBuild, GCP Cloud Build, Vagrant, Puppet, Python, Bash, Go, Perl, Java, Scala, Groovy, Maven, SBT, Gradle, Make, Jenkinsfile, Makefile, Dockerfile, docker-compose.yml, Vagrantfile, M4 etc...\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Templates.svg)](https://starchart.cc/HariSekhon/Templates)\n\n---\n## Hari Sekhon - DevOps Perl Tools\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/DevOps-Perl-tools)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/DevOps-Perl-tools?logo=github)](https://github.com/HariSekhon/DevOps-Perl-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/DevOps-Perl-tools?logo=github)](https://github.com/HariSekhon/DevOps-Perl-tools/network)\n\n25+ DevOps CLI Tools - Anonymizer, SQL ReCaser (MySQL, PostgreSQL, AWS Redshift, Snowflake, Apache Drill, Hive, Impala, Cassandra CQL, Microsoft SQL Server, Oracle, Couchbase N1QL, Dockerfiles), Hadoop HDFS & Hive tools, Solr/SolrCloud CLI, Nginx stats & HTTP(S) URL watchers for load-balanced web farms, Linux tools etc.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/DevOps-Perl-tools.svg)](https://starchart.cc/HariSekhon/DevOps-Perl-tools)\n\n---\n## Spotify Tools\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Spotify-tools)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Spotify-tools?logo=github)](https://github.com/HariSekhon/Spotify-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Spotify-tools?logo=github)](https://github.com/HariSekhon/Spotify-tools/network)\n\nSpotify Tools - Playlists Backups, Spotify CLI, URI translator, duplication detection / removal, API search queries, API automation etc.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Spotify-tools.svg)](https://starchart.cc/HariSekhon/Spotify-tools)\n\n---\n##  # Jenkins - Advanced Jenkinsfile & Groovy Shared Library\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Jenkins)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Jenkins?logo=github)](https://github.com/HariSekhon/Jenkins/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Jenkins?logo=github)](https://github.com/HariSekhon/Jenkins/network)\n\nJenkins - Advanced Jenkinsfile & Groovy Shared Library of reusable functions and pipelines - including for AWS, GCP, Docker, Kubernetes, ArgoCD, Slack notifications, Git Merge, Terraform, Cloudflare, Jenkins Job Backups, most major Docker registries, DockerHub, GHCR, ECR, GCR, GAR, ACR, GitLab, Quay\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Jenkins.svg)](https://starchart.cc/HariSekhon/Jenkins)\n\n---\n## Hari Sekhon - Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, Security, AWS, GCP etc.\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Knowledge-Base)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Knowledge-Base?logo=github)](https://github.com/HariSekhon/Knowledge-Base/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Knowledge-Base?logo=github)](https://github.com/HariSekhon/Knowledge-Base/network)\n\nIT Knowledge Base from 20 years in DevOps, Linux, Cloud, Big Data, AWS, GCP etc - gradually porting my large private knowledge base to public\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Knowledge-Base.svg)](https://starchart.cc/HariSekhon/Knowledge-Base)\n\n---\n## Hari Sekhon - DevOps Golang Tools\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/DevOps-Golang-tools)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/DevOps-Golang-tools?logo=github)](https://github.com/HariSekhon/DevOps-Golang-tools/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/DevOps-Golang-tools?logo=github)](https://github.com/HariSekhon/DevOps-Golang-tools/network)\n\nDevOps Golang tools\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/DevOps-Golang-tools.svg)](https://starchart.cc/HariSekhon/DevOps-Golang-tools)\n\n---\n## Terraform Templates\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Terraform)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Terraform?logo=github)](https://github.com/HariSekhon/Terraform/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Terraform?logo=github)](https://github.com/HariSekhon/Terraform/network)\n\nTerraform HCL code for AWS / GCP / Azure / GitHub management\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Terraform.svg)](https://starchart.cc/HariSekhon/Terraform)\n\n---\n## GitHub Actions\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/GitHub-Actions)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/GitHub-Actions?logo=github)](https://github.com/HariSekhon/GitHub-Actions/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/GitHub-Actions?logo=github)](https://github.com/HariSekhon/GitHub-Actions/network)\n\nGitHub Actions Reusable Workflows and Master Template\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/GitHub-Actions.svg)](https://starchart.cc/HariSekhon/GitHub-Actions)\n\n---\n## Hari Sekhon - HashiCorp Packer templates\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Packer-templates)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Packer-templates?logo=github)](https://github.com/HariSekhon/Packer-templates/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Packer-templates?logo=github)](https://github.com/HariSekhon/Packer-templates/network)\n\nHashiCorp Packer templates to build portable virtual machines in OVA format for Ubuntu, Debian and Redhat based systems with automated installers Kickstart, Preseed and AutoInstaller / Cloud-Init. Useful for IoT edge sites, Kubernetes base systems and VM appliances to ship to customers\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Packer-templates.svg)](https://starchart.cc/HariSekhon/Packer-templates)\n\n---\n## Hari Sekhon - Spotify Playlists\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/Spotify-Playlists)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/Spotify-Playlists?logo=github)](https://github.com/HariSekhon/Spotify-Playlists/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/Spotify-Playlists?logo=github)](https://github.com/HariSekhon/Spotify-Playlists/network)\n\n240+ playlists, 36,000+ tracks - in both Spotify URI and human-readable formats. Spotify Profile: <https://open.spotify.com/user/harisekhon>. Spotify API tools are submodules of this repo.\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/Spotify-Playlists.svg)](https://starchart.cc/HariSekhon/Spotify-Playlists)\n\n---\n## Hari Sekhon - Perl Library\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/lib)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/lib?logo=github)](https://github.com/HariSekhon/lib/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/lib?logo=github)](https://github.com/HariSekhon/lib/network)\n\nPerl Utility Library for my other repos\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/lib.svg)](https://starchart.cc/HariSekhon/lib)\n\n---\n## CI/CD Status Page\n\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/HariSekhon/CI-CD)\n[![GitHub stars](https://img.shields.io/github/stars/HariSekhon/CI-CD?logo=github)](https://github.com/HariSekhon/CI-CD/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/HariSekhon/CI-CD?logo=github)](https://github.com/HariSekhon/CI-CD/network)\n\nCI/CD Status page for Hari Sekhon's GitHub repos\n\n[![Stargazers over time](https://starchart.cc/HariSekhon/CI-CD.svg)](https://starchart.cc/HariSekhon/CI-CD)\n"
  },
  {
    "path": "STATUS.md",
    "content": "# CI/CD Status Page\n\nMoved to <https://harisekhon.github.io/CI-CD/>\n"
  },
  {
    "path": "ai/openai_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /models\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-10 21:45:48 +0100 (Sat, 10 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the OpenAI API\n\nAutomatically handles authentication via environment variable \\$OPENAI_API_KEY\nIf a member of multiple organizations then you must also set \\$OPENAI_ORGANIZATION_ID\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up your API key here:\n\n    https://platform.openai.com/account/api-keys\n\nIf you are a member of multiple organizations, get the Organization ID here:\n\n    https://platform.openai.com/account/org-settings\n\n\nAPI Reference:\n\n    https://platform.openai.com/docs/api-reference/introduction\n\n\nExamples:\n\n\nList Models:\n\n    ${0##*/} /models\n\nRetrieve Model:\n\n    ${0##*/} /models/{model_id}\n\n    ${0##*/} /models/gpt-3.5-turbo\n\nList Files in our org:\n\n    ${0##*/} /files\n\nGet File metadata:\n\n    ${0##*/} /files/{file_id}\n\nGet File content:\n\n    ${0##*/} /files/{file_id}/content\n\nList fine-tuning jobs:\n\n    ${0##*/} /fine-tunes\n\nRetrieve fine-tune:\n\n    ${0##*/} /fine-tunes/{fine_tune_id}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.openai.com/v1\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined OPENAI_API_KEY\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path//https:\\\\/\\\\/api.openai.com\\/v1}\"\nurl_path=\"${url_path##/}\"\n\nexport TOKEN=\"$OPENAI_API_KEY\"\n\nif [ -n \"${OPENAI_ORGANIZATION_ID:-}\" ]; then\n    CURL_OPTS+=(-H \"OpenAI-Organization: $OPENAI_ORGANIZATION_ID\")\nfi\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "applescript/app_names.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-18 17:03:28 +0400 (Mon, 18 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ $# -gt 0 ]; then\n    cat <<EOF\nPrints sorted list of Applications running in the name format that can be passed to\nthe adjacent script set_frontmost_process.scpt\nEOF\n    exit 3\nfi\n\n\"$srcdir/get_application_names.scpt\" |\nsort -fu # money\n"
  },
  {
    "path": "applescript/browser_close_tab.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 13:05:26 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset defaultBrowser to do shell script \"defaults read \\\\\n    ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure \\\\\n    | awk -F'\\\"' '/http;/{print window[(NR)-1]}{window[NR]=$2}'\"\n\nif defaultBrowser is \"\" or defaultBrowser contains \"safari\" then\n    set defaultBrowser to \"Safari\"\nelse if defaultBrowser contains \"chrome\" then\n    set defaultBrowser to \"Google Chrome\"\nelse if defaultBrowser contains \"firefox\" then\n    set defaultBrowser to \"Firefox\"\nelse\n    set defaultBrowser to \"Unknown\"\nend if\n\n-- doesn't work because tell application is needed at compile time, so best we can do is create a script to launch in a separate process\n--tell application defaultBrowser\n--    keystroke \"w\" using command down\n--end tell\n\n-- doesn't work either\n--set theScript to \"tell application \\\"\" & defaultBrowser & \"\\\" to keystroke \\\"w\\\" using command down\"\n--do shell script \"echo '\" & theScript & \"'\"\n--run script theScript\n\ntell application \"System Events\"\n    set frontmostProcess to first process where it is frontmost\n    tell process defaultBrowser\n        set frontmost to true\n        keystroke \"w\" using command down\n        -- capture the original process and switch it back afterwards instead, just in case we're already in the browser we don't want to Cmd-Tab away\n        --keystroke tab using command down\n    end tell\n    set frontmost of frontmostProcess to true\nend tell\n"
  },
  {
    "path": "applescript/browser_get_default.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 13:05:26 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Returns the default browser in Apple application name usable format, eg. \"Google Chrome\"\n\nset defaultBrowser to do shell script \"defaults read \\\\\n    ~/Library/Preferences/com.apple.LaunchServices/com.apple.launchservices.secure \\\\\n    | awk -F'\\\"' '/http;/{print window[(NR)-1]}{window[NR]=$2}'\"\n\nif defaultBrowser is \"\" or defaultBrowser contains \"safari\" then\n    set defaultBrowser to \"Safari\"\nelse if defaultBrowser contains \"chrome\" then\n    set defaultBrowser to \"Google Chrome\"\nelse if defaultBrowser contains \"firefox\" then\n    set defaultBrowser to \"Firefox\"\nelse\n    set defaultBrowser to \"Unknown\"\nend if\n"
  },
  {
    "path": "applescript/com.harisekhon.wakeup_script.plist",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n  <dict>\n    <key>Label</key>\n    <string>com.harisekhon.wakeup_script</string>\n    <key>Program</key>\n    <string>/Users/hari/github/bash-tools/applescript/wakeup_script.sh</string>\n    <key>WatchPaths</key>\n    <array>\n      <string>/var/db/.AppleSetupDone</string>\n    </array>\n    <key>RunAtLoad</key>\n    <true/>\n  </dict>\n</plist>\n"
  },
  {
    "path": "applescript/get_application_names.scpt",
    "content": "#!/usr/bin/env osascript\n--  vim:ts=4:sts=4:sw=4:et\n--\n--  Author: Hari Sekhon\n--  Date: 2024-10-13 20:26:31 +0300 (Sun, 13 Oct 2024)\n--\n--  https///github.com/HariSekhon/DevOps-Bash-tools\n--\n--  License: see accompanying Hari Sekhon LICENSE file\n--\n--  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n--\n--  https://www.linkedin.com/in/HariSekhon\n--\n\n-- ============================================================================ #\n--                             A p p l e S c r i p t\n-- ============================================================================ #\n\n-- Gets the list of Applications running in the name format that can be passed to\n-- the adjacent script set_frontmost_process.scpt\n\ntell application \"System Events\"\n    set appList to (name of every application process)\nend tell\n\nset output to \"\"\n\nrepeat with appName in appList\n    set output to output & appName & \"\\n\"\nend repeat\n\n-- strip trailing newline\nset output to text 1 thru -2 of output\n\n-- doesn't come out right due to carriage returns\n--do shell script \"echo \" & quoted form of output\n-- even this outputs carriage returns\n--do shell script \"echo \" & quoted form of output & \" | tr '\\r' '\\n'\"\n\n-- annoying pop-up\n--display dialog output as text\n\n-- outputs to stderr instead of stdout, use implicit print of last value instead\n--log output\noutput\n\n-- output is unsorted and sorting in Applescript requires crude in-code sorting like Bubblescript to passing array out\n-- to shell sort and back which results in a string formatting output one character per line BS, just wrap this in quick\n-- shell it's simpler\n"
  },
  {
    "path": "applescript/get_frontmost_process.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 13:05:26 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\ntell application \"System Events\"\n    set frontmostProcess to first process where it is frontmost\n    name of frontmostProcess\nend tell\n"
  },
  {
    "path": "applescript/get_frontmost_process_title.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-12-05 16:30:05 +0000 (Mon, 05 Dec 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Returns the name and title of the current foreground app\n#\n# Borrowed from:\n#\n#   https://stackoverflow.com/questions/5292204/macosx-get-foremost-window-title\n#\n# Tested on macOS 10.7 and 11.7\n\nglobal frontApp, frontAppName, windowTitle\n\nset windowTitle to \"\"\ntell application \"System Events\"\n    set frontApp to first application process whose frontmost is true\n    set frontAppName to name of frontApp\n    tell process frontAppName\n        tell (1st window whose value of attribute \"AXMain\" is true)\n            set windowTitle to value of attribute \"AXTitle\"\n        end tell\n    end tell\nend tell\n\nreturn {frontAppName, windowTitle}\n"
  },
  {
    "path": "applescript/get_mouse_coordinates.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-13 20:43:57 +0100 (Sat, 13 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gets the coordinates of the mouse cursor\n#\n# other options:\n#\n# Cmd-Shift-4   - to take a select area screenshot displays the coordates, then press Esc to cancel\n#\n# MouseTools -location\n\ntell application \"System Events\"\n    # doesn't work due to mouse not being defined\n    set mousePosition to position of the mouse\nend tell\n"
  },
  {
    "path": "applescript/get_mouse_coordinates.sh",
    "content": "#!/usr/bin/env bash\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nMouseTools -location\n"
  },
  {
    "path": "applescript/is_screen_locked.py",
    "content": "#!/usr/bin/env python3\n#  coding=utf-8\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-12-05 22:03:56 +0000 (Mon, 05 Dec 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n\"\"\"\n\nDetects whether the macOS screen is locked\n\nIf locked, prints 'true' and returns exit code 0\nIf unlocked, prints 'false' and returns exit code 1\n\nUseful to avoid automated keystrokes or mouse_clicks while on locked screen which can make it hard to login back in\n\n\"\"\"\n\nfrom __future__ import print_function\n\nimport sys\nimport Quartz\n\nif __name__ == '__main__':\n    # false positive\n    # pylint: disable=no-member\n    d = Quartz.CGSessionCopyCurrentDictionary()\n    if 'CGSSessionScreenIsLocked' in d and d['CGSSessionScreenIsLocked'] == 1:\n        print('true')\n        sys.exit(0)\n    else:\n        print('false')\n        sys.exit(1)\n"
  },
  {
    "path": "applescript/is_screensaver_running.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-12-05 17:44:03 +0000 (Mon, 05 Dec 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Returns 'true' or 'false' as to whether screensaver is running\n#\n# Useful to avoid sending clicks or keystrokes during screensaver, as this makes it hard to login\n#\n# XXX: Caveat: doesn't work on locked screen when screensaver isn't running so not comprehensive\n\ntell application \"System Events\"\n    get running of screen saver preferences\nend tell\n"
  },
  {
    "path": "applescript/keystrokes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-14 17:16:31 +0100 (Sun, 14 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAutomates Keyboard strokes to automate tedious UI actions\n\nPerforms N keyboard key code presses\n\n    https://eastmanreference.com/complete-list-of-applescript-key-codes\n\nSleeps for \\$SLEEP_SECS (default: 1) between clicks to allow UIs to update and perform the next keystroke\n\nStarts each keystroke after \\$START_DELAY seconds (default: 5) to give time to alt-tab back to your UI application and position the cursor\n\nIf given num is negative, will run indefinitely until Control-C'd\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<num> [<keycode> <keycode> <keycode> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nmac_only\n\nnum=\"$1\"\nstart_delay=\"${START_DELAY:-5}\"\nsleep_secs=\"${SLEEP_SECS:-1}\"\n\nif ! [[ \"$num\" =~ ^-?[[:digit:]]+$ ]]; then\n    usage \"invalid non-integer '$num' given for first argument\"\nfi\n\nif ! is_float \"$start_delay\"; then\n    usage \"invalid non-float '$START_DELAY' found in environment for \\$START_DELAY\"\nfi\n\nif ! is_float \"$sleep_secs\"; then\n    usage \"invalid non-float '$SLEEP_SECS' found in environment for \\$SLEEP_SECS\"\nfi\n\nshift || :\n\nread -r -a keys <<< \"$@\"\n\ntimestamp \"waiting for $start_delay secs before starting\"\nsleep \"$start_delay\"\ntimestamp \"starting $num keystrokes\"\necho\n\nfor ((i=1; ; i++)); do\n    # if given num is negative, will run for infinity until Control-C'd\n    if [ \"$num\" -ge 0 ] &&\n       [ \"$i\" -gt \"$num\" ]; then\n        break\n    fi\n    for key in \"${keys[@]}\"; do\n        if [[ \"$key\" =~ ^[[:digit:]][[:digit:]]+$ ]]; then\n            timestamp \"keystroke $i/$num keycode $key\"\n            osascript -e \"tell application \\\"System Events\\\" to key code $key\" || :\n        else\n            timestamp \"keystroke $i/$num key $key\"\n            osascript -e \"tell application \\\"System Events\\\" to keystroke \\\"$key\\\"\"\n        fi\n        sleep \"$sleep_secs.$RANDOM\"  # add $RANDOM up to 1 second jitter to make it harder to spot that this is perfectly automated clicking\n    done\ndone\n"
  },
  {
    "path": "applescript/mouse_clicks.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-13 20:47:12 +0100 (Sat, 13 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Click N times at coordinates x,y\n#\n# waits 5 seconds before starting\n#\n# easier to just use 'MouseTools -leftClick' command line, see adjacent mouse_clicks.sh\n\n# Incomplete, doesn't seem to work, use adjacent mouse_clicks.sh instead which works nicely\n\n\nset N to 10\n\n# see adjacent get_mouse_coordinates.scpt for how to get the mouse coordinates\nset x to 2455\nset y to 1273\n\ndo shell script \"echo waiting 5 secs before starting clicking\"\ndelay 5\n\n# repeat N times\n# want loop iterator variable to print the click we're on\nset i to 0\nrepeat while i < N\n    tell application \"System Events\"\n        click at {x,y}\n    end tell\n    #do shell script \"echo click \" & i\n    #copy \"click \" & i to stdout\n    set i to i + 1\n    delay 1\nend repeat\n\n# looks like last thing printed overwrites all previous output :-/\ncopy \"DONE\" to stdout\n"
  },
  {
    "path": "applescript/mouse_clicks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-14 17:16:31 +0100 (Sun, 14 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAutomates Mouse Clicks to automate tedious UI actions\n\nPerforms N mouse clicks at the sequence of X,Y coordinates given or the current mouse location if no coordinates\n\nSleeps for \\$SLEEP_SECS (default: 1) between clicks to allow UIs to update and perform the next click\n\nStarts clicking after \\$START_DELAYS seconds (default: 5) to give time to alt-tab back to your UI application and position the cursor\n\nIf given num is negative, will run indefinitely until Control-C'd\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<num> [<coordinates> <coordinates> <coordinates> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nmac_only\n\nnum=\"$1\"\nstart_delay=\"${START_DELAY:-5}\"\nsleep_secs=\"${SLEEP_SECS:-1}\"\n\nif ! [[ \"$num\" =~ ^-?[[:digit:]]+$ ]]; then\n    usage \"invalid non-integer '$num' given for first argument\"\nfi\n\nif ! is_float \"$start_delay\"; then\n    usage \"invalid non-float '$START_DELAY' found in environment for \\$START_DELAY\"\nfi\n\nif ! is_float \"$sleep_secs\"; then\n    usage \"invalid non-float '$SLEEP_SECS' found in environment for \\$SLEEP_SECS\"\nfi\n\nshift || :\n\nread -r -a coordinates <<< \"$@\"\n\nif ! type -P cliclick &>/dev/null; then\n    brew install cliclick\nfi\n\nif [ -n \"${coordinates:-}\" ]; then\n    for coordinate in \"${coordinates[@]}\"; do\n        if ! [[ \"$coordinate\" =~ ^[[:digit:]]+,[[:digit:]]+$ ]]; then\n            usage \"invalid coordinate '$coordinate' given - must be in form x,y\"\n        fi\n    done\nfi\n\ntimestamp \"Waiting for $start_delay secs before starting\"\nsleep \"$start_delay\"\ntimestamp \"Starting $num mouse clicks every $sleep_secs\"\necho\n\nfor ((i=1; ; i++)); do\n    # if given num is negative, will run for infinity until Control-C'd\n    if [ \"$num\" -ge 0 ] &&\n       [ \"$i\" -gt \"$num\" ]; then\n        break\n    fi\n    if [ -n \"${coordinates:-}\" ]; then\n        for coordinate in \"${coordinates[@]}\"; do\n            x=\"${coordinate%,*}\"\n            y=\"${coordinate#*,}\"\n            timestamp \"Mouse click $i/$num at $x , $y\"\n            # tool no longer available online\n            #MouseTools -leftClick -x \"$x\" -y \"$y\"\n            cliclick \"c:$x,$y\"\n            sleep \"$sleep_secs.$RANDOM\"\n        done\n    else\n        timestamp \"Mouse click $i/$num at current mouse location\"\n        #MouseTools -leftClick\n        cliclick \"c:.\"\n        sleep \"$sleep_secs.$RANDOM\"  # add $RANDOM up to 1 second jitter to make it harder to spot that this is perfectly automated clicking\n    fi\ndone\n"
  },
  {
    "path": "applescript/mouse_clicks_remote_desktop.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-06 10:55:58 +0700 (Fri, 06 Dec 2024)\n#\n#  https://github.com/HariSekhon\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSwitches to Microsoft Remote Desktop, waits 10 seconds and then clicks the mouse once a minute to prevent the screensaver from coming on\n\nWorkaround to Active Directory Group Policies that don't let you disable the screensaver\n\nPoint the mouse to a safe location with no mouse click effect\n\nThen Cmd-Tab to Terminal, run this and let it switch back to Remote Desktop to keep the session open\nminute to prevent the screensaver from coming on\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nmac_only\n\nexport START_DELAY=\"${START_DELAY:-10}\"\nexport SLEEP_SECS=\"${SLEEP_SECS:-60}\"\n\nif ! is_float \"$START_DELAY\"; then\n    usage \"invalid non-float '$START_DELAY' found in environment for \\$START_DELAY\"\nfi\n\nif ! is_float \"$SLEEP_SECS\"; then\n    usage \"invalid non-float '$SLEEP_SECS' found in environment for \\$SLEEP_SECS\"\nfi\n\ntimestamp \"Switching foreground window to Remote Desktop\"\n\"$srcdir/set_frontmost_process.scpt\" \"Microsoft Remote Desktop\"\n\nexec \"$srcdir/mouse_clicks.sh\" -1\n"
  },
  {
    "path": "applescript/mouse_random_movements.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-06 08:09:48 +0700 (Fri, 06 Dec 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRandomly moves the mouse around the screen\n\nUseful to prevent a screensaver kicking in on a Remote Desktop connection which has Active Directory Group Policies\napplied that doesn't let you disable the screensaver\n\nSleeps for 10 seconds between mouse movements\n\nUPDATE: it turns out this doesn't stop Windows Virtual Desktop from going to screensaver.\n\nWORKAROUND: find a safe area in your WVD session to click without any effect, then Cmd-Tab to Terminal and run this:\n\n    mouse_clicks_remote_desktop.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sleep_seconds> <num_movements>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nmac_only\n\nsleep_secs=\"${1:-10}\"\n\nnum=\"${2:--1}\"\n\nif ! is_float \"$sleep_secs\"; then\n    usage \"invalid non-float argument given for sleep seconds: $sleep_secs\"\nfi\n\nif [ \"$sleep_secs\" -lt 1 ]; then\n    usage \"Sleep seconds cannot be less than 1\"\nfi\n\nif ! [[ \"$num\" =~ ^-?[[:digit:]]+$ ]]; then\n    usage \"invalid non-integer num movements given for first argument: $num\"\nfi\n\nif ! type -P cliclick &>/dev/null; then\n    brew install cliclick\nfi\n\ntimestamp \"Starting random mouse movements\"\necho\n\nscreen_width=\"$(system_profiler SPDisplaysDataType | grep Resolution | awk '{print $2}')\"\nscreen_height=\"$(system_profiler SPDisplaysDataType | grep Resolution | awk '{print $4}')\"\n\nfor ((i=1; ; i++)); do\n    # if given num is negative, will run for infinity until Control-C'd\n    if [ \"$num\" -ge 0 ] &&\n       [ \"$i\" -gt \"$num\" ]; then\n        break\n    fi\n    x=\"$((RANDOM % screen_width))\"\n    y=\"$((RANDOM % screen_height))\"\n    timestamp \"Mouse movement $i/$num at $x , $y\"\n    cliclick \"m:$x,$y\"\n    sleep \"$sleep_secs\"\ndone\n"
  },
  {
    "path": "applescript/reopen_app.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Shazam\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-02 00:44:40 +0300 (Sun, 02 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses Applescript to quit and re-open a given application\n\nWritten to relaunch Shazam after deleting tracks from its DB using adjacent script to reflect the changes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<app>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\napp=\"$1\"\n\nmac_only\n\ntimestamp \"Quitting and re-opening app: $app\"\n\nosascript <<EOF\n    tell application \"$app\" to quit\n    delay 1\n    tell application \"$app\" to activate\nEOF\n"
  },
  {
    "path": "applescript/screensaver_activate.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-12-05 22:21:47 +0000 (Mon, 05 Dec 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Activate the macOS screensaver programmatically\n\ntell application \"ScreenSaverEngine\" to activate\n"
  },
  {
    "path": "applescript/set_frontmost_process.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 13:05:26 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\non run argv\n    tell application \"System Events\"\n        set frontmost of application process (item 1 of argv) to true\n    end tell\nend run\n"
  },
  {
    "path": "applescript/set_mic_internal.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-17 20:04:16 +0800 (Mon, 17 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the Macbook Pro microphone to use the internal mic as sometimes Apple AirPods mic sound muffled to others\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nmac_only\n\nif ! type -P brew &>/dev/null; then\n    \"$srcdir/../install/install_homebrew.sh\"\nfi\n\nif ! type SwitchAudioSource &>/dev/null; then\n    brew install switchaudio-osx\nfi\n\ninternal_mic=\"$(SwitchAudioSource -a | grep Microphone | grep -v -i -e iPhone -e AirPods || :)\"\n\nif [[ $(wc -l <<< \"$internal_mic\") -ge 2 ]]; then\n    die \"ERROR: more than one microphone returned:\n\n$internal_mic\n\"\nfi\n\nif [ -z \"$internal_mic\" ]; then\n    die \"ERROR: failed to determine internal mic\"\nfi\n\n#SwitchAudioSource -t input -s \"MacBook Pro Microphone\"\n#timestamp \"Switching to microphone: $internal_mic\"\nSwitchAudioSource -t input -s \"$internal_mic\"\n"
  },
  {
    "path": "applescript/shazam_app_delete_track.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-02 00:22:03 +0300 (Sun, 02 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a single track from the local Mac's Shazam app sqlite database\n\nThe Shazam app caches this while running so you will need to quit and re-open the app to see this change\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args='\"<artist>\" \"<track>\"'\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nmac_only\n\nartist=\"$1\"\ntrack=\"$2\"\n\ndbpath=\"$(\n    find ~/Library/Group\\ Containers \\\n        -type f \\\n        -path '*/*group.com.shazam/com.shazam.mac.Shazam/ShazamDataModel.sqlite' 2>/dev/null |\n    head -n 1\n    )\"\n\nif [ -z \"$dbpath\" ]; then\n    die \"Error: Could not locate ShazamDataModel.sqlite\"\nfi\n\ntimestamp \"Found Shazam App DB: $dbpath\"\n\ntimestamp \"Backing up DB before deleting\"\nbackup=\"${dbpath}.bak.$(date +%Y%m%d%H%M%S)\"\ncp -v \"$dbpath\" \"$backup\"\ntimestamp \"Backup created at $backup\"\necho >&2\n\ntimestamp \"Deleting track from DB: '$artist - $track'\"\necho >&2\n\n# Delete from ZSHTAGRESULTMO using JOIN with ZSHARTISTMO\n#\n# sqlite3 `.parameter` is fragile and unsafe for arbitrary text,\n# and we end up with all kinds of shell injection and\n# quoting and newline issues with arbitrary data, so pre-generate\n# the variables with escaping using SQLite's own quoting engine\n\n# also fragile\n#artist_sql=$(sqlite3 ':memory:' \"SELECT quote($(\n#    printf \"'%s'\" \"$(printf '%s' \"$artist\" | sed \"s/'/''/g\")\"\n#));\")\n#\n#track_sql=$(sqlite3 ':memory:' \"SELECT quote($(\n#    printf \"'%s'\" \"$(printf '%s' \"$track\" | sed \"s/'/''/g\")\"\n#));\")\n\nartist_sql=\"$(\n  sqlite3 -batch \\\n          -noheader \\\n          -list \\\n          -cmd \".parameter clear\" \\\n          -cmd \".parameter set @v '$(sed \"s/'/''/g\" <<< \"$artist\")'\" \\\n          :memory: \\\n          \"SELECT quote(@v);\"\n)\"\n\ntrack_sql=\"$(\n  sqlite3 -batch \\\n          -noheader \\\n          -list \\\n          -cmd \".parameter clear\" \\\n          -cmd \".parameter set @v '$(sed \"s/'/''/g\" <<< \"$track\")'\" \\\n          :memory: \\\n          \"SELECT quote(@v);\"\n)\"\n\nsqlite3 -batch \\\n        -bail  \\\n        \"$dbpath\" <<SQL\n    DELETE FROM\n        ZSHTAGRESULTMO\n    WHERE\n        Z_PK IN (\n            SELECT\n                r.Z_PK\n            FROM\n                ZSHTAGRESULTMO r\n            JOIN\n                ZSHARTISTMO a\n                    ON\n                a.ZTAGRESULT = r.Z_PK\n            WHERE\n                a.ZNAME = $artist_sql\n              AND\n                r.ZTRACKNAME = $track_sql\n        );\nSQL\n\nif [ -z \"${QUIET:-}\" ]; then\n    timestamp \"You must now quit and re-open the Shazam app to pick up this change\"\nfi\n"
  },
  {
    "path": "applescript/shazam_app_dump_tracks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-01 23:39:41 +0300 (Sat, 01 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\ndefault_dbpath=\"$(\n    find ~/Library/Group\\ Containers \\\n        -type f \\\n        -path '*/*.group.com.shazam/com.shazam.mac.Shazam/ShazamDataModel.sqlite' \\\n        2>/dev/null |\n    head -n 1\n)\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the local Mac's Shazam app sqlite database and outputs all tracks, one per line\nin format:\n\nArtist - Track\n\nUseful for using in conjunction with the adjacent spotify_app_search.sh script\nsince Apple removed Shazam's Spotify integration\n\nCan optionally specify a number of tracks to stop after as an arg or environment variable \\$SHAZAM_APP_DUMP_NUM_TRACKS\n\nA second arg can specify a path to the Shazam SQLite DB, which otherwise defaults to:\n\n    $default_dbpath\n\nTested on Shazam app version 2.11.0 - may need to be modified for other versions as the Shazam DB schema changes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<num_tracks|today|yesterday|week|last:num_days|YYYY-MM-DD> <sqlite_db_path>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nmac_only\n\narg=\"${1:-${SHAZAM_APP_DUMP_NUM_TRACKS:--1}}\"\n\ndbpath=\"${2:-$default_dbpath}\"\n\nif [ ! -f \"$dbpath\" ]; then\n    die \"Error: Could not locate ShazamDataModel.sqlite - File Not Found: $dbpath\"\nfi\n\ntimestamp \"Found Shazam App DB: $dbpath\"\n\nwhere_clause=\"\"\norder_clause=\"ORDER BY r.ZDATE DESC\"\nlimit_clause=\"\"\n\n# macOS Core Data framework stores dates as seconds 2001-01-01 00:00:00 UTC, not unix epoch of 1970\ncoredata_epoch_offset=978307200\n\n# XXX: localtime strftime() may give off comparisons vs UTC stored date timestamps, so avoided for YYYY-MM-DD\n#      keeping local day for today/yesterday/week or last:N though\ncase \"$arg\" in\n    today)\n        where_clause=\"\n            WHERE\n                r.ZDATE >= (\n                    strftime('%s', 'now', 'start of day', 'localtime') - $coredata_epoch_offset\n                )\n        \"\n        order_clause=\"ORDER BY r.ZDATE ASC\"\n        ;;\n    yesterday)\n        where_clause=\"\n            WHERE\n                r.ZDATE >= (\n                    strftime('%s', 'now', 'start of day', '-1 day', 'localtime') - $coredata_epoch_offset\n                )\n            AND\n                r.ZDATE < (\n                    strftime('%s', 'now', 'start of day', 'localtime') - $coredata_epoch_offset\n                )\n        \"\n        order_clause=\"ORDER BY r.ZDATE ASC\"\n        ;;\n    week)\n        where_clause=\"\n            WHERE\n                r.ZDATE >= (\n                    strftime('%s', 'now', 'start of day', '-6 days', 'localtime')\n                    - $coredata_epoch_offset\n                )\n        \"\n        order_clause=\"ORDER BY r.ZDATE ASC\"\n        ;;\n    last:*)\n        days=\"${arg#last:}\"\n\n        if ! [[ \"$days\" =~ ^[[:digit:]]+$ ]]; then\n            die \"Invalid argument for last:N, must be a positive integer: $arg\"\n        fi\n\n        where_clause=\"\n            WHERE\n                r.ZDATE >= (\n                    strftime('%s', 'now', '-$days days', 'localtime')\n                    - $coredata_epoch_offset\n                )\n        \"\n        order_clause=\"ORDER BY r.ZDATE ASC\"\n        ;;\n    ????-??-??)\n        where_clause=\"\n            WHERE\n                r.ZDATE >= (\n                    strftime('%s', '$arg', 'start of day')\n                    - $coredata_epoch_offset\n                )\n            AND\n                r.ZDATE < (\n                    strftime('%s', '$arg', 'start of day', '+1 day')\n                    - $coredata_epoch_offset\n                )\n        \"\n        order_clause=\"ORDER BY r.ZDATE ASC\"\n        ;;\n    *)\n        num=\"$arg\"\n\n        if ! [[ \"$num\" =~ ^-?[[:digit:]]+$ ]]; then\n            die \"Invalid argument given, must be an integer, 'today', or 'yesterday': $arg\"\n        fi\n\n        if [ \"$num\" -gt 0 ]; then\n            limit_clause=\"LIMIT $num\"\n        fi\n        ;;\nesac\n# my ~/.sqliterc forces pretty printing breaking the separator we need so -init /dev/null to ignore it\nsqlite3 \"$dbpath\" -init /dev/null -noheader -separator $'\\t' \\\n\"\n    SELECT\n        a.ZNAME AS artist,\n        r.ZTRACKNAME AS track\n    FROM\n        ZSHTAGRESULTMO r\n    LEFT JOIN\n        ZSHARTISTMO a\n            ON\n        a.ZTAGRESULT = r.Z_PK\n    $where_clause\n    $order_clause\n    $limit_clause;\n\" |\nsed '/^[[:space:]]*$/d' |\nwhile IFS=$'\\t' read -r artist title; do\n    # trim leading/trailing whitespace and replace newlines\n    artist=\"$(tr -d '\\r' <<< \"$artist\" | sed 's/^ *//;s/ *$//')\"\n    title=\"$(tr -d '\\r' <<< \"$title\" | sed 's/^ *//;s/ *$//')\"\n    printf \"%s\\t-\\t%s\\n\" \"$artist\" \"$title\"\ndone |\n# from https://github.com/HariSekhon/DevOps-Perl-tools repo - use it if present in $PATH\nif type -P uniq_order_preserved.pl &>/dev/null; then\n    uniq_order_preserved.pl\nelse\n    cat\nfi\n"
  },
  {
    "path": "applescript/shazam_search_spotify.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-02 00:30:16 +0300 (Sun, 02 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps the local Mac Shazam app's tracks one at a time, searches the Spotify app for each one\n\nShazam to Spotify apps workaround to Apple removing Spotify integration from Shazam\n\nPrompts between each track to proceed to search for the next one\n\nOptionally prompts to delete the track after searh from the Shazam local sqlite DB if this environment variable is set:\n\n    export SHAZAM_APP_DELETE_TRACK_AFTER_SEARCH=1\n\nCan optionally specify a number of tracks to stop after as an arg,\nor a timeframe today/yesterday/week/YYYY-MM-DD,\n\nYou can set this args as an environment variable \\$SHAZAM_APP_DUMP_NUM_TRACKS - the arg takes precedence though\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<num_tracks> <sqlite_db_path>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nmac_only\n\nnum_tracks=\"${1:-${SHAZAM_APP_DUMP_NUM_TRACKS:-1}}\"\nsqlite_db_path=\"${2:-}\"\n\ncase \"$num_tracks\" in\n    # allow only these args to be passed to shazam_app_dump_tracks.sh\n    today|yesterday|week|????-??-??) : ;;\n    *)\n        if ! [[ \"$num_tracks\" =~ ^-?[[:digit:]]+$ ]]; then\n            die \"Invalid argument given, must be an integer or one of today/yesterday/week/YYYY-MM-DD: $num_tracks\"\n        fi\n        ;;\nesac\n\nrelaunch_shazam(){\n    timestamp \"Relaunching Shazam app to reflect removed tracks\"\n    \"$srcdir/reopen_app.sh\" Shazam\n    sleep 2\n    # hit Escape key to to minimize Shazam back to the menu bar\n    osascript -e \"tell application \\\"System Events\\\" to key code 53\"\n    untrap\n    exit\n}\nexport -f relaunch_shazam\n\ntrap_cmd 'relaunch_shazam'\n\nwhile IFS=$'\\t' read -r artist _ track; do\n    \"$srcdir/spotify_app_search.sh\" \"$artist $track\"\n    if [ \"${SHAZAM_APP_DELETE_TRACK_AFTER_SEARCH:-}\" = 1 ]; then\n        timestamp \"Press enter to delete this track from the Shazam DB: $artist - $track\"\n        read -r < /dev/tty\n        QUIET=1 \"$srcdir/shazam_app_delete_track.sh\" \"$artist\" \"$track\"\n    else\n        timestamp \"Press enter to search for next track\"\n        read -r < /dev/tty\n    fi\ndone < <(\n    \"$srcdir/shazam_app_dump_tracks.sh\" \"$num_tracks\" ${sqlite_db_path:+\"$sqlite_db_path\"}\n)\n"
  },
  {
    "path": "applescript/shazam_search_spotify_then_delete_track.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-02 00:30:16 +0300 (Sun, 02 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps the local Mac Shazam app's tracks one at a time, searches the Spotify app for each one,\nand then deletes it from the Shazam local sqlite DB upon an Enter key press to proceed to the next one\n\nShazam to Spotify apps workaround to Apple removing Spotify integration from Shazam\n\nCan optionally specify a number of tracks to stop after as an arg,\nor a timeframe today/yesterday/week/YYYY-MM-DD,\n\nYou can set this args as an environment variable \\$SHAZAM_APP_DUMP_NUM_TRACKS - the arg takes precedence though\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<num_tracks> <sqlite_db_path>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nmac_only\n\nexport SHAZAM_APP_DELETE_TRACK_AFTER_SEARCH=1\n\n\"$srcdir/shazam_search_spotify.sh\" \"$@\"\n"
  },
  {
    "path": "applescript/shorten_text_selection.scpt",
    "content": "#!/usr/bin/env osascript\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-19 01:44:50 +0300 (Mon, 19 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#           Shortens the selected text in the prior window\n# ============================================================================ #\n\n# - Copies the selected text to the clipboard\n# - Replaces \"and\" with \"&\"\n# - Removes multiple blank lines between paragraphs (which result from the pbcopy/pbpaste pipeline otherwise)\n# - Pastes the clipboard text back over the selected text\n#\n# I use this a lot for LinkedIn comments in browser due to the short 1250 character limit\n\n# Tested on macOS 14\n\n# switch to previous window\ntell application \"System Events\"\n\tkey down command\n\tkeystroke tab\n\tkey up command\nend tell\n\ndelay 0.3\n\n# copy the selected text\ntell application \"System Events\"\n\tkeystroke \"c\" using command down\nend tell\n\ndelay 0.1\n\n# - replace occurrences of the word \"and\" with \"&\" using sed with word boundaries\n# - crush out multiple blank lines to a single blank line between paragraphs\n#   - this is correct the pbpaste | pbcopy copying back multiplying the blank lines\ndo shell script \"pbpaste | gsed -E 's/\\\\band\\\\b/\\\\&/g' | cat -s | pbcopy\"\n\ndelay 0.1\n\n# paste the modified text over the original selection\ntell application \"System Events\"\n\tkeystroke \"v\" using command down\nend tell\n"
  },
  {
    "path": "applescript/spotify_app_search.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-01 23:34:33 +0300 (Sat, 01 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a search in the Spotify App on Mac using Applescript\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<search terms>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nmac_only\n\nquery=\"$*\"\n\nquery=\"${query//\\\"/\\\\\\\"}\"\n\ntimestamp \"Telling Spotify app to search for: $query\"\n\nosascript \\\n        -e 'tell application \"Spotify\" to activate' \\\n        -e 'tell application \"System Events\" to keystroke \"l\" using {command down}' \\\n        -e 'delay 0.2' \\\n        -e \"tell application \\\"System Events\\\" to keystroke \\\"$query\\\" & return\"\n"
  },
  {
    "path": "applescript/start_app_at_login.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-06 22:17:22 +0200 (Thu, 06 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds an App to auto-start at Login using Applescript\n\nChecks the /Applications and \\$HOME/Applications for the given app name\n\n(auto-tries both with and without .app extension so you can provide it either way)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<app>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\narg=\"$1\"\n\nHOME=\"${HOME:-$(cd && pwd)}\"\n\napp=\"\"\nfor path in \\\n        \"/Applications/$arg.app\" \\\n        \"/Applications/$arg\" \\\n        \"/$HOME/Applications/$arg.app\" \\\n        \"/$HOME/Applications/$arg\"; do\n    if [ -e \"$path\" ]; then\n        timestamp \"Found '$arg' at '$path'\"\n        app=\"$path\"\n    fi\ndone\n\nif [ -z \"$app\" ]; then\n    die \"App '$arg' not found in /Applications or $HOME/Applications\"\nfi\n\ntimestamp \"Setting '$app' to start at login\"\nosascript -e \"tell application \\\"System Events\\\" to make login item at end with properties {path:\\\"$app\\\", hidden:false}\"\n"
  },
  {
    "path": "applescript/wakeup_script.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-04-17 22:26:11 +0400 (Wed, 17 Apr 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript of actions to run whenever a Mac wakes up from sleep\n\nCurrently this script just flushes the DNS cache to fix Chrome hitting ERR_NOT_FOUND errors when waking up on a VPN\n\nTo set this up, edit the path to this script in the plist xml file and then load it:\n\n    cp -fv $srcdir/com.harisekhon.wakeup_script.plist ~/Library/LaunchAgents/\n    launchctl load $srcdir/com.harisekhon.wakeup_script.plist\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nmac_only\n\n{\n\ntimestamp \"Running Mac wake up script: $0\"\n\ntimestamp \"Flushing DNS cache\"\ndscacheutil -flushcache\n\ntimestamp \"Reloading mDNSResponder\"\nsudo killall -HUP mDNSResponder\n\ntimestamp \"Wake up script completed\"\necho\n\n} 2>&1 | tee -a \"$srcdir/wakeup_script.log\"\n"
  },
  {
    "path": "applescript/world_clock_cities.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2026-02-05 14:53:51 -0300 (Thu, 05 Feb 2026)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Cities to load into the World Clock on macOS\n\n# Was going to load this into Clock via Applescript but this app is extraordinarily bad for scripting\n\nLondon\nPrague\nMadrid\nAmsterdam\nDubai\nBangkok\nCairo\nChisinau\nIstanbul\nTbilisi\nCancun\nBogota\nSantiago\nBuenos Aires\nNew York\nLos Angeles\n#Palo Alto\nSan Franciso\n\n#Marrakesh\n#Kyiv\n#Miami\n#Hanoi\n#Adelaide\n#Athens\n#Pontianak\n#Melbourne\n#Tunis\n"
  },
  {
    "path": "appveyor/appveyor_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-31 13:47:21 +0100 (Tue, 31 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Queries AppVeyor API, auto-populating $APPVEYOR_TOKEN from environment and API url base for convenience\n#\n# https://kevinoid.github.io/appveyor-swagger/bootprint/\n#\n# eg.\n#\n# appveyor_api.sh projects | jq\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Queries the AppVeyor API, auto-populating the base and API tokens from the environment\"\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nif [ -z \"${APPVEYOR_TOKEN:-}\" ]; then\n    usage \"APPVEYOR_TOKEN environment variable is not set (generate this from your Web UI Dashboard -> profile -> API AUTH TOKENS\"\nfi\n\nif [ $# -lt 1 ]; then\n    usage \"no /path given to query in the API\"\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -h|--help) usage\n                   ;;\n   esac\ndone\n\nurl_path=\"${1##/}\"\nshift || :\n\nexport TOKEN=\"$APPVEYOR_TOKEN\"\n\n\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"https://ci.appveyor.com/api/$url_path\" \"$@\"\n"
  },
  {
    "path": "appveyor/appveyor_byoc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 23:16:47 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nif [ -z \"${APPVEYOR_TOKEN:-}\" ]; then\n    echo \"\\$APPVEYOR_TOKEN not found in environment\"\n    exit 1\nfi\n\nexport PATH=\"$PATH:/opt/appveyor/host-agent\"\n\nif ! type -P appveyor-host-agent &>/dev/null; then\n    \"$srcdir/../install/install_appveyor_byoc.sh\"\n    clear\nfi\n\n# leading whitespace break PowerShell commands\npwsh <<EOF\nImport-Module AppVeyorBYOC\nConnect-AppVeyorToComputer -AppVeyorUrl https://ci.appveyor.com -ApiToken $APPVEYOR_TOKEN\nEOF\n\nif is_inside_docker && [ -x /opt/appveyor/host-agent/appveyor-host-agent ]; then\n    cd /opt/appveyor/host-agent\n    /opt/appveyor/host-agent/appveyor-host-agent\nfi\n"
  },
  {
    "path": "appveyor/appveyor_byoc_debian.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 23:16:47 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/utils.sh\"\n\ncd \"$srcdir\"\n\nif [ -z \"${APPVEYOR_TOKEN:-}\" ]; then\n    echo \"\\$APPVEYOR_TOKEN not found in environment\"\n    exit 1\nfi\n\nexport DEBIAN_FRONTEND=noninteractive\n\n#exec docker run -ti --rm -e APPVEYOR_TOKEN -e DEBIAN_FRONTEND -v \"$PWD\":/pwd -w /pwd debian:9 ./appveyor_byoc.sh\n#exec docker run -ti --rm -e APPVEYOR_TOKEN -e DEBIAN_FRONTEND -v \"$PWD\":/pwd -w /pwd harisekhon/appveyor:debian ./appveyor_byoc.sh\nexec docker run -ti --rm -e APPVEYOR_TOKEN -e DEBIAN_FRONTEND harisekhon/appveyor:debian\n"
  },
  {
    "path": "appveyor/appveyor_delete_offline_byoc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 15:48:02 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# eg. HariSekhon\nif [ -z \"${APPVEYOR_ACCOUNT:-}\" ]; then\n    echo \"\\$APPVEYOR_ACCOUNT not defined\"\n    exit 1\nfi\n\necho \"Querying AppVeyor for offline BYOC\"\n\"$srcdir/appveyor_api.sh\" \"account/$APPVEYOR_ACCOUNT/build-clouds\" |\njq -r '.[] | select(.status == \"Offline\") | [.name, .buildCloudId] | @tsv' |\nwhile read -r name id; do\n    echo \"Deleting offline BYOC '$name'\"\n    # obtained from the Network debug tab of making UI calls\n    \"$srcdir/appveyor_api.sh\" \"account/$APPVEYOR_ACCOUNT/build-clouds/$id\" -X DELETE\ndone\n"
  },
  {
    "path": "aws/.aws_customize_environment",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-16 14:56:05 +0000 (Wed, 16 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#           A W S   C l o u d   S h e l l   C u s t o m i z a t i o n\n# ============================================================================ #\n\n# This is designed after the standard GCP CloudShell behaviour which is mimicked via .bash.d/aws-cloudshell.sh\n\n# see log at /var/log/customize_environment\n\n# called as root using sudo by .bash.d/aws-cloudshell.sh\n\nset -euxo pipefail\n\nbash_tools=\"$HOME/github/bash-tools\"\n\nif ! [ -d \"$bash_tools\" ]; then\n    parent_dir=\"${bash_tools%/*}\"\n    mkdir -pv \"$parent_dir\"\n    pushd \"$parent_dir\"\n    git clone https://github.com/HariSekhon/DevOps-Bash-tools \"$bash_tools\"\n    popd\nfi\n\npushd \"$bash_tools\"\n\ngit pull\n\n# not calling gcp-shell which also links because it will link config files to /root\n# instead just run 'make gcp-shell' one time to set up $USER's $HOME\nmake system-packages\n\npopd\n"
  },
  {
    "path": "aws/aws_account_summary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-07 15:08:44 +0000 (Tue, 07 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints AWS account summary in 'key = value' pairs for easy viewing / grepping\n\nUseful information includes whether the root account has MFA enabled and no access keys:\n\nAccountAccessKeysPresent = 0\nAccountMFAEnabled = 1\n\nor comparing number of users to number of MFA devices eg.\n\nMFADevices = 6\nMFADevicesInUse = 6\n...\nUsers = 14\n\n\nIf you don't have AWS Organizations permissions, you'll probably get an error like this, in which case the account name and root account email won't be printed:\n\n    An error occurred (AccessDeniedException) when calling the DescribeAccount operation: You don't have permissions to access this resource.\n\nThis may happen for example when you're using an AWS SSO account that doesn't have privileges at the Organization level to describe the account.\nThis can be safely ignored, the rest of the IAM account summary info containing details such as MFA devices, users and policies etc will be there.\n\n\nSee Also:\n\n    aws_iam_users_mfa_active_report.sh (adjacent)\n    check_aws_root_account.py   -   in The Advanced Nagios Plugins collection (https://github.com/HariSekhon/Nagios-Plugins)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\n# XXX: can't set this to account_id as account summary info only works on a profile basis\nprofile=\"${1:-}\"\n\nif [ -n \"$profile\" ]; then\n    export AWS_PROFILE=\"$profile\"\nfi\n\nexport AWS_DEFAULT_OUTPUT=json\n\naccount_id=\"$(aws sts get-caller-identity --query Account --output text | tr -d '\\r')\"\necho \"AccountID = $account_id\"\n# XXX: might not have permissions to run this one, skip it if so\naccount_info=\"$(aws organizations describe-account --account-id \"$account_id\" || { echo; echo \"Missing permission to describe AWS Organization\"; echo; } >&2 )\"\nif [ -n \"$account_info\" ]; then\n    account_name=\"$(jq -r '.Account.Name' <<< \"$account_info\")\"\n    echo \"AccountName = $account_name\"\n    account_email=\"$(jq -r '.Account.Email' <<< \"$account_info\")\"\n    echo \"AccountEmail = $account_email\"\nfi\naws iam get-account-summary |\njq -r '.SummaryMap | to_entries | map(.key + \" = \" + (.value | tostring)) | .[]' |\nsort\n"
  },
  {
    "path": "aws/aws_accounts_missing_from_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-04 22:01:06 +0700 (Tue, 04 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a list of AWS Account IDs in stdin or files (containing one account id per line),\nfinds those missing from AWS config\n\nYou can override the config file location by setting environment variable AWS_CONFIG_FILE\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files>]\"\n\nhelp_usage \"$@\"\n\n#min_args 0 \"$@\"\n\nHOME=\"${HOME:-$(cd && pwd)}\"\n\naws_config=\"${AWS_CONFIG_FILE:-$HOME/.aws/config}\"\n\n# force functions to log with timestamps\nexport VERBOSE=1\n\naccount_ids=\"$(\n    sed '\n        s/#.*//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d\n    ' \"$@\"\n)\"\n\ntimestamp \"AWS Account IDs not found in: $aws_config\"\necho >&2\nwhile read -r aws_account_id; do\n    if ! is_aws_account_id \"$aws_account_id\"; then\n        warn \"Invalid AWS Account ID: $aws_account_id\"\n    fi\n    if ! grep -Fq \"$aws_account_id\" \"$aws_config\"; then\n        echo \"$aws_account_id\"\n    fi\ndone <<< \"$account_ids\"\n"
  },
  {
    "path": "aws/aws_batch_kill_stale_jobs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-02 17:00:24 +0000 (Tue, 02 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTerminates AWS Batch jobs in a given queue older than N hours (default: 24)\n\nUseful to find and kill jobs that have become too long running, eg. more than 24 hours, or jobs far exceeding their expected time, including jobs that can get stuck with memory allocation errors on shared VMs\n\nMay take a few seconds before the job(s) are actually terminated\n\nRequires AWS CLI to be configured and authenticated\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<queue-name> [<hours>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nqueue=\"$1\"\nhours=\"${2:-24}\"\n\nscript_basename=\"${0##*/}\"\n\n\"$srcdir/aws_batch_stale_jobs.sh\" \"$queue\" \"$hours\" |\njq -r '.[] | [.jobId,.jobName] | @tsv' |\nwhile read -r job_id job_name; do\n    timestamp \"Terminating job id: '$job_id', name: '$job_name'\"\n    aws batch terminate-job --job-id \"$job_id\" --reason \"Job terminated by script $script_basename after running for longer than $hours hours\"\ndone\n"
  },
  {
    "path": "aws/aws_batch_stale_jobs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-02 17:00:24 +0000 (Tue, 02 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS Batch jobs in a given queue older than N hours (default: 24)\n\nIncludes jobs stuck in pending and runnable states as these are usually stuck due to an environment/configuration issue if they've been pending for a long time\n\nUseful to find jobs that have become too long running, eg. more than 24 hours, or jobs far exceeding their expected time, including jobs that can get stuck with memory allocation errors on shared VMs\n\nReturns JSON list of jobs for further processing\n\nRequires AWS CLI to be configured and authenticated\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<queue-name> [<hours>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nqueue=\"$1\"\nhours=\"${2:-24}\"\n\n# doesn't support floats\n#millis=\"$((hours * 3600 * 1000))\"\nmillis=\"$(bc -l <<< \"$hours * 3600 * 1000\" | sed 's/\\..*$//')\"\n\nepoch_millis=\"$(date +%s)000\"\n\nbefore_epoch_millis=\"$((epoch_millis - millis))\"\n\n# --filters only works on newer versions of CLIv2 so jq processing is more reliable across environments:\n#\n#   https://github.com/aws/aws-cli/issues/6526\n#\n#aws batch list-jobs --job-queue \"$queue\" --filters \"name=BEFORE_CREATED_AT,values=$before_epoch\"\n\nfor state in SUBMITTED PENDING RUNNABLE STARTING RUNNING; do\n    aws batch list-jobs --job-queue \"$queue\" --job-status \"$state\" |\n    jq \".jobSummaryList[] | select(.createdAt <= $before_epoch_millis)\"\ndone |\n# slurp items back into an array\njq -s .\n"
  },
  {
    "path": "aws/aws_billing_alarm.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 16:30:33 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets an AWS CloudWatch billing alarm to trigger as soon as you begin incurring any charges\n\nCreates an SNS topic and subscription for the given email address and links it to the above CloudWatch Alarm to email you as soon as your billing charges go over\n\nThe alarm is set in the us-east-1 region (N. Virginia in the web console) because that is where the metric billing data accumulates, regardless of which region you actually use\n\n\nThe first argument sets the alert threshold in USD - an alarm is raised once it goes above that amount\nThe default threshold is 0.00 USD to alert on any charges for safety\n\nThe second argument sets the email address to use in an SNS topic to notify you.\nIf no email is given specified attempts to use the email from your local Git configuration.\nIf neither is available, shows this usage mesage.\n\nXXX: You must also enable Receive Billing Alerts in the Billing Preferences page for the CloudWatch metrics to be populated by AWS Billing:\n\n    https://console.aws.amazon.com/billing/home?#/preferences\n\nSee the created alarm here:\n\n    https://console.aws.amazon.com/cloudwatch/home?region=us-east-1\n\n(notice the region must be us-east-1 as per description above)\n\nSee Also:\n\n    aws_budget_alarm.sh - newer method of doing this using AWS Budgets\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<threshold_amount_in_USD> [<email_address>]\"\n\nhelp_usage \"$@\"\n\nthreshold=\"${1:-0.00}\"\nemail=\"${2:-$(git config user.email || :)}\"\n\n# XXX: region has to be us-east-1 because this is where the billing metric data accumulates regardless of which region you actually use\nregion=\"us-east-1\"\n\nsns_topic=\"AWS_Charges\"\n\nif ! [[ \"$threshold\" =~ ^[[:digit:]]{1,4}(\\.[[:digit:]]{1,2})?$ ]]; then\n    usage \"invalid threshold argument given - must be 0.01 - 9999.99 USD\"\nfi\n\nif is_blank \"$email\"; then\n    usage \"email address not specified and could not determine email from git config\"\nfi\n\ntimestamp \"Creating SNS topic to email '$email' in region '$region'\"\noutput=\"$(aws sns create-topic --name \"$sns_topic\" --region \"$region\" --output json)\"\n\n# \"arn:aws:sns:us-east-1:123456789012:AWS_Charges\"\nsns_topic_arn=\"$(jq -r '.TopicArn' <<< \"$output\")\"\n\necho\n\ntimestamp \"Subscribing email address '$email' to topic '$sns_topic' in region '$region'\"\naws sns subscribe --topic-arn \"$sns_topic_arn\" --protocol email --notification-endpoint \"$email\" --region \"$region\"\n\necho\n\ntimestamp \"Creating CloudWatch Alarm for AWS charges > $threshold USD in region '$region'\"\n# --period 21600 = 6 hours (default)\naws cloudwatch put-metric-alarm --alarm-name \"AWS Charges\" \\\n                                --alarm-description \"Alerts on AWS charges greater than $threshold USD\" \\\n                                --actions-enabled \\\n                                --alarm-actions \"$sns_topic_arn\" \\\n                                --region \"$region\" \\\n                                --namespace \"AWS/Billing\" \\\n                                --metric-name \"EstimatedCharges\" \\\n                                --dimensions \"Name=Currency,Value=USD\" \\\n                                --threshold \"$threshold\" \\\n                                --comparison-operator \"GreaterThanThreshold\" \\\n                                --statistic Maximum \\\n                                --period 21600 \\\n                                --evaluation-periods 1\n"
  },
  {
    "path": "aws/aws_budget.json",
    "content": "{\n\t\"BudgetName\": \"AWS_Charges\",\n\t\"BudgetType\": \"COST\",\n\t\"BudgetLimit\": {\n\t\t\"Amount\": \"<AWS_BUDGET_AMOUNT>\",\n\t\t\"Unit\": \"USD\"\n\t},\n\t\"CostFilters\": {},\n\t\"CostTypes\": {\n\t\t\"IncludeTax\": true,\n\t\t\"IncludeSubscription\": true,\n\t\t\"UseBlended\": false,\n\t\t\"IncludeRefund\": false,\n\t\t\"IncludeCredit\": false,\n\t\t\"IncludeUpfront\": true,\n\t\t\"IncludeRecurring\": true,\n\t\t\"IncludeOtherSubscription\": true,\n\t\t\"IncludeSupport\": true,\n\t\t\"IncludeDiscount\": true,\n\t\t\"UseAmortized\": false\n\t},\n\t\"TimeUnit\": \"MONTHLY\",\n\t\"TimePeriod\": {\n\t\t\"Start\": \"2021-08-01T01:00:00+01:00\",\n\t\t\"End\": \"2087-06-15T01:00:00+01:00\"\n\t}\n}\n"
  },
  {
    "path": "aws/aws_budget_alarm.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-08-02 15:57:07 +0100 (Mon, 02 Aug 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS Budget with an alarm if forecasted to go over 80% of total monthly budget, and another alarm if over 90% of monthly budget\n\nCreates an SNS topic and subscription for the given email address and links it to the above AWS Budgets Alarm to email you as soon as your billing charges are anticipated to go over the threshold. It also modifies the SNS topic's access policy to be accessible from the AWS Budgets service.\n\n\nThe first argument sets the total monthly budget in USD - the 80% and 90% threshold alarms are based on that\nThe default budget is 0.01 USD (will trigger a notification on any expenditure)\n\nThe second argument sets the email address to use in an SNS topic to notify you.\nIf no email is given specified attempts to use the email from your local Git configuration.\nIf neither is available, shows this usage mesage.\n\n\nSee the created AWS Budget here (Global):\n\n    https://console.aws.amazon.com/billing/home#/budgets/overview\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<budget_amount_in_USD> [<email_address>]\"\n\nhelp_usage \"$@\"\n\nbudget=\"${1:-0.01}\"\nemail=\"${2:-$(git config user.email || :)}\"\n\nregion=\"us-east-1\"\n\nsns_topic=\"AWS_Charges\"\n\nif ! [[ \"$budget\" =~ ^[[:digit:]]{1,4}(\\.[[:digit:]]{1,2})?$ ]]; then\n    usage \"invalid budget argument given - must be 0.01 - 9999.99 USD\"\nfi\n\nif is_blank \"$email\"; then\n    usage \"email address not specified and could not determine email from git config\"\nfi\n\ntimestamp \"Creating SNS topic to email '$email' in region '$region'\"\noutput=\"$(aws sns create-topic --name \"$sns_topic\" --region \"$region\" --output json)\"\n\n# \"arn:aws:sns:us-east-1:123456789012:AWS_Charges\"\nsns_topic_arn=\"$(jq -r '.TopicArn' <<< \"$output\")\"\n\necho\n\ntimestamp \"Subscribing email address '$email' to topic '$sns_topic' in region '$region'\"\naws sns subscribe --topic-arn \"$sns_topic_arn\" --protocol email --notification-endpoint \"$email\" --region \"$region\"\n\necho\n\ntimestamp \"Getting account id\"\naccount_id=\"$(aws sts get-caller-identity --query Account --output text)\"\n\necho\n\n# https://docs.aws.amazon.com/cli/latest/reference/sns/set-topic-attributes.html\ntimestamp \"Updating access policy on SNS topic '$sns_topic' to allow AWS Budgets to use it\"\naws sns set-topic-attributes --topic-arn \"$sns_topic_arn\" --attribute-name Policy --attribute-value \"$(sed \"s/<AWS_SNS_ARN>/$sns_topic_arn/; s/<AWS_ACCOUNT_ID>/$account_id/\" \"$srcdir/aws_budget_sns_access_policy.json\")\" --region \"$region\"\n\necho\n\n# https://awscli.amazonaws.com/v2/documentation/api/latest/reference/budgets/create-budget.html\n\ntimestamp \"Checking for existing AWS Budgets\"\nbudgets=\"$(aws budgets describe-budgets --account-id \"$account_id\" --query 'Budgets[*].BudgetName' --output text)\"\n\necho\n\nbudget_name=\"$(jq -r .BudgetName < \"$srcdir/aws_budget.json\")\"\nif grep -Fxq \"$budget_name\" <<< \"$budgets\"; then\n    if [ -n \"${REPLACE_BUDGET:-}\" ]; then\n        timestamp \"deleting budget '$budget' to replace it\"\n        aws budgets delete-budget --account-id \"$account_id\" --budget-name \"$budget_name\"\n        echo\n    else\n        echo \"AWS Budget '$budget' already exists - you must delete it before running this\"\n        exit 0\n    fi\nfi\n\ntimestamp \"Creating AWS Budget with $budget USD budget and 80% forecasted threshold alarm\"\naws budgets create-budget --account-id \"$account_id\" --budget \"$(sed \"s/<AWS_BUDGET_AMOUNT>/$budget/\" \"$srcdir/aws_budget.json\")\" --notifications-with-subscribers \"$(sed \"s/<AWS_SNS_ARN>/$sns_topic_arn/\" \"$srcdir/aws_budget_notification.json\")\"\n"
  },
  {
    "path": "aws/aws_budget_notification.json",
    "content": "[\n\t{\n\t\"Notification\": {\n\t  \"NotificationType\": \"FORECASTED\",\n\t  \"ComparisonOperator\": \"GREATER_THAN\",\n\t  \"Threshold\": 80.0,\n\t  \"ThresholdType\": \"PERCENTAGE\",\n\t  \"NotificationState\": \"ALARM\"\n\t},\n\t\"Subscribers\": [\n\t  {\n\t\t\"SubscriptionType\": \"SNS\",\n\t\t\"Address\": \"<AWS_SNS_ARN>\"\n\t  }\n\t]\n\t},\n\t{\n\t\"Notification\": {\n\t  \"NotificationType\": \"ACTUAL\",\n\t  \"ComparisonOperator\": \"GREATER_THAN\",\n\t  \"Threshold\": 80.0,\n\t  \"ThresholdType\": \"PERCENTAGE\",\n\t  \"NotificationState\": \"ALARM\"\n\t},\n\t\"Subscribers\": [\n\t  {\n\t\t\"SubscriptionType\": \"SNS\",\n\t\t\"Address\": \"<AWS_SNS_ARN>\"\n\t  }\n\t]\n\t},\n\t{\n\t\"Notification\": {\n\t  \"NotificationType\": \"FORECASTED\",\n\t  \"ComparisonOperator\": \"GREATER_THAN\",\n\t  \"Threshold\": 100.0,\n\t  \"ThresholdType\": \"PERCENTAGE\",\n\t  \"NotificationState\": \"ALARM\"\n\t},\n\t\"Subscribers\": [\n\t  {\n\t\t\"SubscriptionType\": \"SNS\",\n\t\t\"Address\": \"<AWS_SNS_ARN>\"\n\t  }\n\t]\n\t},\n\t{\n\t\"Notification\": {\n\t  \"NotificationType\": \"ACTUAL\",\n\t  \"ComparisonOperator\": \"GREATER_THAN\",\n\t  \"Threshold\": 100.0,\n\t  \"ThresholdType\": \"PERCENTAGE\",\n\t  \"NotificationState\": \"ALARM\"\n\t},\n\t\"Subscribers\": [\n\t  {\n\t\t\"SubscriptionType\": \"SNS\",\n\t\t\"Address\": \"<AWS_SNS_ARN>\"\n\t  }\n\t]\n\t}\n]\n"
  },
  {
    "path": "aws/aws_budget_sns_access_policy.json",
    "content": "{\n  \"Version\": \"2008-10-17\",\n  \"Id\": \"__default_policy_ID\",\n  \"Statement\": [\n    {\n      \"Sid\": \"AWSBudgets-notification\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"Service\": \"budgets.amazonaws.com\"\n      },\n      \"Action\": \"SNS:Publish\",\n      \"Resource\": \"<AWS_SNS_ARN>\"\n    },\n    {\n      \"Sid\": \"__default_statement_ID\",\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"*\"\n      },\n      \"Action\": [\n        \"SNS:GetTopicAttributes\",\n        \"SNS:SetTopicAttributes\",\n        \"SNS:AddPermission\",\n        \"SNS:RemovePermission\",\n        \"SNS:DeleteTopic\",\n        \"SNS:Subscribe\",\n        \"SNS:ListSubscriptionsByTopic\",\n        \"SNS:Publish\",\n        \"SNS:Receive\"\n      ],\n      \"Resource\": \"<AWS_SNS_ARN>\",\n      \"Condition\": {\n        \"StringEquals\": {\n          \"AWS:SourceOwner\": \"<AWS_ACCOUNT_ID>\"\n        }\n      }\n    }\n  ]\n}\n"
  },
  {
    "path": "aws/aws_cli_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-20 17:26:21 +0000 (Sat, 20 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS service account for CI/CD automation or AWS CLI to avoid having to re-login every day via SSO with 'aws sso login'\n\nGrants this service account Administator privileges in the current AWS account unless an alternative group or policy is specified\n\nCreates an IAM access key (deleting an older unused key if necessary), writes a CSV just as the UI download would, and outputs both shell export commands and configuration in the format for copying to your AWS profile in ~/.aws/credentials\n\nThe following optional arguments can be given:\n\n- user name         (default: \\$USER-cli)\n- keyfile           (default: ~/.aws/keys/\\${user}_\\${aws_account_id}_accessKeys.csv) - be careful if specifying this, a non-existent keyfile will create a new key, deleting the older of 2 existing keys if necessary to be able to create this\n- group/policy      (default: Admins group or falls through to AdministratorAccess policy - checks for this group name first, or else policy by this name)\n\nThis can also be used as a backup credential - this way if something accidentally happens to your AWS SSO you can still get into your account\n\nIdempotent - safe to re-run, will skip creating a user that already exists or CSV export that already exists\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username> [<group1,group2,policy1,policy2...> <keyfile>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser=\"${1:-$USER-cli}\"\n\n#group=\"${2:-Admins}\"\n#policy=\"${2:-AdministratorAccess}\"\ngroups_or_policies=\"${2:-}\"\ndefault_group=\"Admins\"\ndefault_policy=\"AdministratorAccess\"\n\naws_account_id=\"$(aws_account_id)\"\n\naccess_keys_csv=\"${3:-$HOME/.aws/keys/${user}_${aws_account_id}_accessKeys.csv}\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws_create_user_if_not_exists \"$user\"\n\nexports=\"$(aws_create_access_key_if_not_exists \"$user\" \"$access_keys_csv\")\"\n\ngroup_exists(){\n    # causes a failure in the if policy test condition, probably due to early exit on one of the pipe commands\n    set +o pipefail\n    aws iam list-groups | jq -r '.Groups[].GroupName' | grep -Fixq \"$1\" || return 1\n    set -o pipefail\n}\n\npolicy_exists(){\n    # causes a failure in the if policy test condition, probably due to early exit on one of the pipe commands\n    set +o pipefail\n    aws iam list-policies | jq -r '.Policies[].PolicyName' | grep -Fixq \"$1\" || return 1\n    set -o pipefail\n}\n\ngrant_group_or_policy(){\n    local group_or_policy=\"$1\"\n    if group_exists \"$group_or_policy\"; then\n        group=\"$group_or_policy\"\n        timestamp \"Adding user '$user' to group '$group' on account '$aws_account_id'\"\n        aws iam add-user-to-group --user-name \"$user\" --group-name \"$group\"\n    elif policy_exists \"$group_or_policy\"; then\n        policy=\"$group_or_policy\"\n        timestamp \"Determining ARN for policy '$policy'\"\n        policy_arn=\"$(aws iam list-policies | jq -r \".Policies[] | select(.PolicyName == \\\"$policy\\\") | .Arn\")\"\n        timestamp \"Determined policy ARN:  $policy_arn\"\n        timestamp \"Granting policy '$policy' permissions directly to user '$user' in account '$aws_account_id'\"\n        aws iam attach-user-policy --user-name \"$user\" --policy-arn \"$policy_arn\"\n    else\n        die \"Group/Policy '$group_or_policy' not found in account '$aws_account_id'\"\n    fi\n    echo\n}\n\nif [ -n \"$groups_or_policies\" ]; then\n    for group_or_policy in ${groups_or_policies//,/ }; do\n        grant_group_or_policy \"$group_or_policy\"\n    done\nelse\n    if group_exists \"$default_group\"; then\n        grant_group_or_policy \"$default_group\"\n    elif policy_exists \"$default_policy\"; then\n        grant_group_or_policy \"$policy\"\n    else\n        die \"Neither default group '$default_group', nor default policy '$default_policy' in account '$aws_account_id'\"\n    fi\nfi\n\necho\necho \"Set the following export commands in your environment to begin using this access key in your CLI immediately:\"\necho\necho \"$exports\"\necho\necho \"or add the following to your ~/.aws/credentials file:\"\necho\naws_access_keys_exports_to_credentials <<< \"$exports\"\necho\necho\n"
  },
  {
    "path": "aws/aws_cloudformation_stacks_pending.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 12:22:59 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists CloudFormation stacks not marked as completed\n\nUseful with the 'watch' command or in a loop as a latch (hint: use grep) to check until there are no pending CloudFormation stacks before continuing\n\n\nArguments are fed to AWS CLI eg. to set --region\n\nOutput Format:\n\n<status>    <stack_name>    <stack_description>\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\n\naws cloudformation list-stacks --output json \"$@\" |\njq -r '.StackSummaries[] | [.StackStatus, .StackName, .TemplateDescription] | @tsv' |\n{ grep -Ev '^([[:alnum:]_]+)?COMPLETE' || : ; }\n"
  },
  {
    "path": "aws/aws_cloudfront_distribution_for_origin.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-19 13:54:16 +0700 (Thu, 19 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the AWS CloudFront ARN of the distribution which serves origins containing a given substring\n\nUseful for quickly finding the CloudFront ARN needed to give permissions to a private S3 bucket exposed via CloudFront\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain_substring>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ndomain_substring=\"$1\"\n\njson=\"$(\n    aws cloudfront list-distributions \\\n        --query \"DistributionList.Items[*].{ARN:ARN, DomainNames:Origins.Items[*].DomainName}\" \\\n        --output json\n)\"\n\nif [ \"$json\" = null ]; then\n    echo \"No CloudFront distributions found. Have you set the right \\$AWS_PROFILE environment variable to the correct account?\" >&2\n    exit 1\nfi\n\njq -r \".[] | select(.DomainNames | map(ascii_downcase | contains(\\\"$domain_substring\\\")) | any) | .ARN\" <<< \"$json\"\n"
  },
  {
    "path": "aws/aws_cloudtrails_cloudwatch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-17 16:24:52 +0000 (Fri, 17 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud Trails and their last delivery to CloudWatch Logs (should be recent)\n\nOutput Format:\n\nCloudTrail_Name      LastDeliveryTimestampToCloudWatchLogs (may be null)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\n#echo \"Getting Cloud Trails\" >&2\naws cloudtrail describe-trails |\njq -r '.trailList[].Name' |\nwhile read -r name; do\n    printf '%s\\t' \"$name\"\n    output=\"$(aws cloudtrail get-trail-status --name \"$name\" | jq -r '.LatestcloudwatchLogdDeliveryTime')\"\n    if [ -n \"$output\" ]; then\n        echo \"$output\"\n        echo \"$output\"\n    else\n        echo \"NOT_LOGGING\"\n    fi\ndone |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_cloudtrails_event_selectors.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-17 16:24:52 +0000 (Fri, 17 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud Trails and their event selectors\n\nTo check there is at least one event selector for each trail with IncludeManagementEvents set to true and ReadWriteType set to All\n\nOutput Format:\n\nName      IncludeManagementEvents (boolean)   ReadWriteType (All)     DataResources (optional)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\n#echo \"Getting Cloud Trails\" >&2\naws cloudtrail describe-trails |\njq -r '.trailList[].Name' |\nwhile read -r name; do\n    echo -n \"$name \"\n    aws cloudtrail get-event-selectors --trail-name \"$name\" |\n    jq -r '.EventSelectors[] | [.IncludeManagementEvents, .ReadWriteType, .DataResources[]] | @tsv'\ndone |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_cloudtrails_s3_accesslogging.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-17 16:24:52 +0000 (Fri, 17 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud Trails buckets and their Access Logging prefix and target bucket\n\nOutput Format:\n\nCloudTrail_S3_Bucket      TargetPrefix    TargetBucket\n\nIf access logging isn't configured on the bucket, outputs:\n\nCloudTrail_S3_Bucket      S3_ACCESS_LOGGING_NOT_CONFIGURED\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws cloudtrail describe-trails --query 'trailList[*].S3BucketName' |\njq -r '.[]' |\nwhile read -r name; do\n    printf '%s\\t' \"$name\"\n    output=\"$(aws s3api get-bucket-logging --bucket \"$name\" |\n    jq -r '.LoggingEnabled | [.TargetPrefix, .TargetBucket] | @tsv')\"\n    if [ -z \"$output\" ]; then\n        echo \"S3_ACCESS_LOGGING_NOT_CONFIGURED\"\n    else\n        echo \"$output\"\n    fi\ndone |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_cloudtrails_s3_kms.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-21 18:25:39 +0000 (Tue, 21 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud Trails and whether their S3 buckets are KMS secured\n\nOutput Format:\n\nCloudTrail_Name      S3_KMS_secured (boolean)     KMS_Key_Id\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws cloudtrail describe-trails |\n# more efficient\njq -r '.trailList[] | [.Name, has(\"KmsKeyId\"), .KmsKeyId // \"N/A\"] | @tsv' |\n#jq -r '.trailList[] | [.Name, .KmsKeyId] | @tsv' |\n#while read -r name keyid; do\n#    kms_secured=false\n#    if [ -n \"$keyid\" ]; then\n#        kms_secured=true\n#    else\n#        keyid=\"N/A\"\n#    fi\n#    printf \"%s\\t%s\\t%s\" \"$name\" \"$kms_secured\" \"$keyid\"\n#done |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_cloudtrails_status.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-17 16:24:52 +0000 (Fri, 17 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud Trails and their status - if they're logging, multi-region and log file validation enabled\n\nOutput Format:\n\nName      Logging (boolean)   Multi-Region (boolean)    Logfile Validation (boolean)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\n#echo \"Getting Cloud Trails\" >&2\naws cloudtrail describe-trails |\njq -r '.trailList[] | [.Name, .IsMultiRegionTrail, .LogFileValidationEnabled] | @tsv' |\nwhile read -r name is_multi_region is_validation_enabled; do\n    is_logging=\"$(\n        aws cloudtrail get-trail-status --name \"$name\" |\n        jq -r '.IsLogging'\n    )\"\n    echo \"$name $is_logging $is_multi_region $is_validation_enabled\"\ndone |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_codecommit_csv_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-30 11:13:53 +0100 (Wed, 30 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints AWS CodeCommit Git credentials from a standard AWS HTTPS Git credentials CSV export file as shell export statements\n\n    export GIT_USER=...\n    export GIT_PASSWORD=...\n\nUseful to quickly switch your shell to some exported credentials from a service account for testing permissions\nor pipe to upload to a CI/CD system via an API, eg. the adjacent scripts:\n\n    github_actions_repo*_set_secret.sh\n    gitlab_*_set_env_vars.sh\n    circleci_*_set_env_vars.sh\n    bitbucket_*_set_env_vars.sh\n    terraform_cloud_*_set_vars.sh\n    kubectl_kv_to_secret.sh\n\nExamples:\n\n    # format downloaded from the user's IAM -> Security Credentials -> section HTTPS Git credentials for AWS CodeCommit\n\n    eval \\$(${0##*/} hari_codecommit_credentials.csv)\n\n\nYou can then use these credentials in commands, but note that if the \\$GIT_PASSWORD contains slashes you will need to urlencode it:\n\n    GIT_PASSWORD_URLENCODED=\\\"\\$(urlencode.sh <<< \\\"\\$GIT_PASSWORD\\\")\\\"\n\n    git clone \\\"https://\\$GIT_USER:\\$GIT_PASSWORD_URLENCODED@git-codecommit.eu-west-2.amazonaws.com/v1/repos/myrepo\\\"\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>_codecommit_credentials.csv\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncsv=\"$1\"\n\nif ! [ -f \"$csv\" ]; then\n    die \"ERROR: File not found: $csv\"\nfi\n\n# people may rename their credentials file\n#if ! [[ \"$csv\" =~ ^.+_codecommit_credentials.csv ]]; then\n#    die \"ERROR: Wrong filename, should be in format *_codecommit_credentials.csv\"\n#fi\n\n# XXX: this CSV credentials files come in DOS format unlike other CSV credential downloads for AWS CLI etc.\nif ! tr -d '\\r' < \"$csv\" | grep -Fxq 'User Name,Password'; then\n    die \"ERROR: Expected 'User Name,Password' header not found in file '$csv'\"\nfi\n\nlines=\"$(wc -l \"$csv\" | awk ' {print $1}')\"\n\nif ! [ \"$lines\" -eq 2 ]; then\n    die \"ERROR: wrong number of lines found in CSV credentials file, expected 2, got $lines\"\nfi\n\ntr -d '\\r' < \"$csv\" |\ntail -n 1 |\nawk -F, '{\n    print \"export GIT_USER=\"$1\n    print \"export GIT_PASSWORD=\"$2\n}'\n"
  },
  {
    "path": "aws/aws_config_all_types.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-21 17:20:46 +0000 (Tue, 21 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS Config recorders, checking all resource types are supported (should be true) and includes global resources (should be true)\n\neg.\n\nawsconfig  true  true\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\naws configservice describe-configuration-recorders --output json |\njq -r '.ConfigurationRecorders[] | [.name, .recordingGroup.allSupported, .recordingGroup.includeGlobalResourceTypes] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_config_recording.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-21 17:20:46 +0000 (Tue, 21 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS Config recorders, their recording status (should be true) and their last status (should be success)\n\neg.\n\nawsconfig  true  SUCCESS\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\naws configservice describe-configuration-recorder-status --output json |\njq -r '.ConfigurationRecordersStatus[] | [.name, .recording, .lastStatus] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_csv_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 16:59:48 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints AWS credentials from a standard AWS CSV export file or access key export CSV as shell export statements\n\n    export AWS_ACCESS_KEY_ID=AKIA...\n    export AWS_SECRET_ACCESS_KEY=...\n\nSupports new user and new access key csv file formats eg. 'Download .csv file' when you create an AWS access key in the console\n\nUseful to quickly switch your shell to some exported credentials from a service account for testing permissions\nor pipe to upload to a CI/CD system via an API, eg. the adjacent scripts:\n\n    jenkins_cred_add*.sh\n    github_actions_repo*_set_secret.sh\n    gitlab_*_set_env_vars.sh\n    circleci_*_set_env_vars.sh\n    bitbucket_*_set_env_vars.sh\n    terraform_cloud_*_set_vars.sh\n    kubectl_kv_to_secret.sh\n\nExamples:\n\n    eval \\$(${0##*/} new_user_credentials.csv)  # format downloaded when creating a user\n\n    eval \\$(${0##*/} hari_accessKeys.csv)       # format downloaded when creating an access key\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"credential.csv\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncsv=\"$1\"\n\nif ! [ -f \"$csv\" ]; then\n    die \"ERROR: File not found: $csv\"\nfi\n\nif ! grep -Fq 'AKIA' \"$csv\"; then\n    die \"ERROR: Access Key not found in file '$csv'\"\nfi\n\n# for CSV created at access key creation time\nif tr -d '\\r' < \"$csv\" | grep -Fq 'Access key ID,Secret access key'; then\n    # access keys are prefixed with AKIA, skips header row by selecting the row with the AKIA key\n    awk -F, '/AKIA/{\n        print \"export AWS_ACCESS_KEY_ID=\"$1\n        print \"export AWS_SECRET_ACCESS_KEY=\"$2\n    }' \"$csv\" | tr -d '\\r'\n# for CSV created at user creation time\nelif tr -d '\\r' < \"$csv\" | grep -Fq 'User name,Password,Access key ID,Secret access key,Console login link'; then\n    awk -F, '/AKIA/{\n        print \"export AWS_ACCESS_KEY_ID=\"$3\n        print \"export AWS_SECRET_ACCESS_KEY=\"$4\n    }' \"$csv\" | tr -d '\\r'\nelse\n    die \"ERROR: unrecognized CSV header line, may have changed so code may need an update\"\nfi\n"
  },
  {
    "path": "aws/aws_ec2_ami_boot.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:05:24 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots an EC2 instance from a given AMI for manual debugging\n\nUseful for interactive debugging when creating AMIs\n\n- checks if there is already an EC2 running instance tagged with your name and AMI ID\n  - if yes, reuses it\n  - if not, boots an EC2 instance from the AMI with the given security group, subnet id and SSH key-name given so you can SSH to its ec2-user\n- waits for the EC2 instance to boot\n- waits for the EC2 instance to pass its instance and system checks\n- determines the public or private IP address and outputs it to stdout for use in other scripts\n\nIt's up to you to Terminate the instance as you may want to leave it running and then create an AMI from it when you've finished testing using this script:\n\n    aws_ec2_ami_create_from_instance.sh\n\nYou may want to run this adjacent wrapper script to drop you straight into an SSH prompt:\n\n    aws_ec2_ami_boot_ssh.sh\n\n\nSee Also:\n\n    https://github.com/HariSekhon/Packer\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ami_id> <instance_type> <security_group> <subnet_id> <ssh-key-name> [<instance_profile>]\"\n\nhelp_usage \"$@\"\n\nmin_args 5 \"$@\"\nmax_args 6 \"$@\"\n\nami_id=\"$1\"\ninstance_type=\"$2\"\nsecurity_group=\"$3\"\nsubnet_id=\"$4\"\nssh_key_name=\"$5\"\ninstance_profile=\"${6:-}\"\n\naws_validate_security_group_id \"$security_group\"\n\naws_validate_subnet_id \"$subnet_id\"\n\nif ! is_blank \"$instance_profile\"; then\n    if ! [[ \"$instance_profile\" =~ ^[A-Za-z0-9+=,.@_-]+$ ]]; then\n        die \"Invalid Instance Profile name: $instance_profile\"\n    fi\nfi\n\ninstance_launched=0\n\nuser=\"${USER:-$(whoami)}\"\n\nif is_blank \"$user\"; then\n    die \"Failed to determine username to tag the EC2 instance with\"\nfi\n\nfor((i=1; i <= 100 ; i++)); do\n    instance_name=\"$user-$ami_id\"\n\n    timestamp \"Checking if EC2 instance of AMI already exists: $instance_name\"\n    instance_id=\"$(\n        aws ec2 describe-instances \\\n            --filters \"Name=tag:Name,Values=$instance_name\" \\\n            --query \"Reservations[0].Instances[0].InstanceId\" \\\n            --output text\n    )\"\n\n    if [ \"$instance_id\" != \"None\" ]; then\n        timestamp \"Checking the instance state isn't terminated or shutting down\"\n        instance_state=\"$(\n            aws ec2 describe-instances \\\n                --instance-ids \"$instance_id\" \\\n                --query \"Reservations[0].Instances[0].State.Name\" \\\n                --output text\n        )\"\n        if grep -qi -e terminated -e shutting-down <<< \"$instance_state\"; then\n            timestamp \"This instance is already terminated / shutting down, will try a new instance name\"\n            echo >&2\n            continue\n        fi\n    fi\n\n    if is_blank \"$instance_id\" || [ \"$instance_id\" = \"None\" ]; then\n        timestamp \"Launching EC2 instance: $instance_name\"\n        instance_id=\"$(\n            aws ec2 run-instances \\\n                --image-id \"$ami_id\" \\\n                --count 1 \\\n                --instance-type \"$instance_type\" \\\n                --key-name \"$ssh_key_name\" \\\n                --security-group-ids \"$security_group\" \\\n                --subnet-id \"$subnet_id\" \\\n                --tag-specifications \"ResourceType=instance,Tags=[{Key=Name,Value=$instance_name}]\" \\\n                --query \"Instances[0].InstanceId\" \\\n                --output text\n        )\"\n        timestamp \"Launched instance: $instance_id\"\n    fi\n    instance_launched=1\n    break\ndone\n\nif [ \"$instance_launched\" != 1 ]; then\n    die \"ERROR: Failed to launch instance\"\nfi\n\necho >&2\n\ntimestamp \"Waiting for instance to be running...\"\naws ec2 wait instance-running --instance-ids \"$instance_id\"\ntimestamp \"Instance is running\"\n\necho >&2\n\nget_instance_profile(){\n    local instance_id=\"$1\"\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query \"Reservations[0].Instances[0].IamInstanceProfile.Arn\" \\\n        --output text |\n    sed 's|.*/||'\n}\n\nif ! is_blank \"$instance_profile\"; then\n    instance_profile_attached=0\n    if [ \"$(get_instance_profile \"$instance_id\")\" = \"$instance_profile\" ]; then\n        instance_profile_attached=1\n    else\n        timestamp \"Attaching instance profile: $instance_profile\"\n        aws ec2 associate-iam-instance-profile \\\n                --instance-id \"$instance_id\" \\\n                --iam-instance-profile Name=\"$instance_profile\"\n        echo >&2\n        timestamp \"Waiting for profile to fully attach...\"\n\n        instance_profile_attached=0\n\n        for((i=1; i <= 100 ; i++)); do\n            current_instance_profile=\"$(get_instance_profile \"$instance_id\")\"\n            if [ \"$current_instance_profile\" = \"None\" ]; then\n                timestamp \"No instance profile associated yet...\"\n            elif [ \"$current_instance_profile\" = \"$instance_profile\" ]; then\n                timestamp \"Instance profile attached\"\n                instance_profile_attached=1\n                break\n            else\n                timestamp \"Waiting for instance profile to attach...\"\n            fi\n\n            sleep 3\n        done\n    fi\n    if [ \"$instance_profile_attached\" != 1 ]; then\n        die \"Instance profile failed to attach, gave up waiting\"\n    fi\nfi\n\n\"$srcdir/aws_ec2_wait_for_instance_ready.sh\" \"$instance_id\"\n\ntimestamp \"EC2 instance running: $instance_name\"\necho \"$instance_id\"\n"
  },
  {
    "path": "aws/aws_ec2_ami_boot_ssh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:05:24 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots an EC2 instance from a given AMI, determines the public or private IP, and drops you into an SSH shell\n\nUseful for interactive debugging when creating AMIs\n\n- checks if there is already an EC2 running instance tagged with your name and AMI ID\n  - if yes, reuses it\n  - if not, boots an EC2 instance from the AMI with the given security group, subnet id and SSH key-name given so you can SSH to its ec2-user\n- waits for the EC2 instance to boot\n- waits for the EC2 instance to pass its instance and system checks\n- determines the public or private IP address\n- SSH's to the EC2 instance\n- Assumes ~/.ssh/<ssh-key-name>,pem is present locally to be able to log in to it\n\nIt's up to you to Terminate the instance as you may want to leave it running and then create an AMI from it when you've finished testing using this script:\n\n    aws_ec2_ami_create_from_instance.sh\n\nSee Also:\n\n    https://github.com/HariSekhon/Packer\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ami_id> <instance_type> <security_group> <subnet_id> <ssh-key-name> [<instance_profile>]\"\n\nhelp_usage \"$@\"\n\nmin_args 5 \"$@\"\nmax_args 6 \"$@\"\n\nssh_key_name=\"$5\"\n\ninstance_id=\"$(\"$srcdir/aws_ec2_ami_boot.sh\" \"$@\")\"\n\nip=\"$(\"$srcdir/aws_ec2_instance_ip.sh\" \"$instance_id\")\"\n\ntimestamp \"SSH'ing to EC2 instance\"\necho >&2\n# this is a brand new instance so the SSH host key won't be trusted\nexec ssh \\\n    -i ~/.ssh/\"$ssh_key_name.pem\" \\\n    -o StrictHostKeyChecking=no \\\n    ec2-user@\"$ip\"\n"
  },
  {
    "path": "aws/aws_ec2_ami_create_from_instance.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-12 16:42:30 +0400 (Tue, 12 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS EC2 AMI from an EC2 instance and waits for it to become available for use\n\nOutputs the AMI ID to stdout after it becomes available\n\nUseful to testing risky things on another EC2 vm cloned from that AMI\n\nDoes not reboot the running EC2 instance for safety by default, which means a non-clean filesystem copy\nunless you shut it down first.\n\nTo enforce a reboot of the EC2 instance (be careful in production!) you must set the environment variable:\n\n    export AWS_EC2_REBOOT_INSTANCE=true\n\nSee it in the list of AMIs afterwards using this command:\n\n    aws ec2 describe-images --owners self --query 'Images[*].{ID:ImageId,Name:Name}' --output table\n\nCheck the state of the AMI is finished:\n\n    aws ec2 describe-images --image-ids \\\"\\$AMI_ID\\\" --output table\n\n\nInvestigate instance names quickly using adjacent script aws_ec2_instances.sh\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name> <new_ami_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\ninstance_name=\"$1\"\n\nami_name=\"$2\"\n\ninstance_id=\"$(VERBOSE=1 \"$srcdir/aws_ec2_instance_name_to_id.sh\" \"$instance_name\")\"\nif ! is_instance_id \"$instance_id\"; then\n    die \"Invalid Instance ID returned, failed regex validation: $instance_id\"\nfi\necho >&2\n\nno_reboot=\"--no-reboot\"\n\nif [ \"${AWS_EC2_REBOOT_INSTANCE:-}\" = true ]; then\n    timestamp \"WARNING: AWS_EC2_REBOOT_INSTANCE environment variable is set to true\"\n    echo >&2\n    read -r -p \"Are you sure you want to take down this EC2 instance '$instance_name'? (y/N) \" answer\n    check_yes \"$answer\"\n    no_reboot=\"\"\n    echo >&2\nfi\n\ntimestamp \"Creating AMI '$ami_name' from EC2 instance '$instance_name'\"\nami_id=\"$(\n    aws ec2 create-image --instance-id \"$instance_id\" --name \"$ami_name\" \"$no_reboot\" |\n    jq -r '.ImageId'\n)\"\necho >&2\n\nif is_blank \"$ami_id\" || [ \"$ami_id\" = null ]; then\n    die \"Failed to get AMI ID\"\nelif ! is_ami_id \"$ami_id\"; then\n    die \"Invalid AMI ID returned, failed regex validation: $ami_id\"\nfi\n\n# special variable that increments - use as a built-in timer\nSECONDS=0\n\ntimestamp \"Checking for AMI '$ami_name' to become ready...\"\necho >&2\n\nwhile true; do\n    state=\"$(aws ec2 describe-images --image-ids \"$ami_id\" | jq -r '.Images[0].State')\"\n    if [ \"$state\" = \"available\" ]; then\n        timestamp \"AMI '$ami_name' is now available after $SECONDS seconds\"\n        break\n    elif [ \"$SECONDS\" -gt 1200 ]; then\n        die \"Waited for 20 minutes without AMI becoming available, something is wrong, aborting...\"\n    fi\n    timestamp \"Waiting for AMI '$ami_name' to become ready. State: $state\"\n    sleep 10\ndone\n\necho \"$ami_id\"\n"
  },
  {
    "path": "aws/aws_ec2_ami_ids.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-13 14:42:01 +0400 (Wed, 13 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList AWS EC2 AMI IDs in use in the current or given AWS account, one per line for processing in other scripts\n\nUsed by:\n\n    aws_info_ec2*.sh\n\nSee also:\n\n    aws_ec2_amis.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nif [ $# -gt 0 ]; then\n    aws_profile=\"$1\"\n    shift || :\n    export AWS_PROFILE=\"$aws_profile\"\nfi\n\n\n# false positive - want single quotes for * to be evaluated within AWS query not shell\n# shellcheck disable=SC2016\naws ec2 describe-instances \\\n    --query 'Reservations[*].Instances[*].ImageId' \\\n    --output text |\ntr '[:space:]' '\\n' |\nsort -u |\nsed '/^[[:space:]]*$/d'\n"
  },
  {
    "path": "aws/aws_ec2_ami_name_to_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-13 14:42:01 +0400 (Wed, 13 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns an EC2 AMI ID from name\n\nAdds additional safety checks:\n\n- verifies no more than one AMI ID is returned\n- does a reverse lookup on the AMI ID to verify the name\n- if an AMI ID is passed, returns it as is for convenience\n\nInvestigate AMI names and IDs quickly using adjacent script:\n\n    aws_ec2_amis.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<ami_name>]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nami_name=\"$1\"\n\nif is_ami_id \"$ami_name\"; then\n    log \"Given AMI name is already an AWS AMI ID, outputs as is: $ami_name\"\n    echo \"$ami_name\"\n    exit 0\nfi\n\nlog \"Determining EC2 AMI ID for name '$ami_name'\"\n\nami_id=\"$(\n    aws ec2 describe-images \\\n        --owners self \\\n        --filters \"Name=name,Values=$ami_name\" \\\n        --query 'Images[*].ImageId' \\\n        --output text\n)\"\n\nif is_blank \"$ami_id\"; then\n    die \"No EC2 AMI found with name '$ami_name'\"\nfi\n\nif [ \"$(awk '{print NF}' <<< \"$ami_id\")\" -gt 1 ]; then\n    cat >&2 <<EOF\nMore than 1 AMI ID returned, aborting for safety!\"\n\nAMI IDs found:\n\n$ami_id\nEOF\n    exit 1\nfi\n\nif ! is_ami_id \"$ami_id\"; then\n    die \"Invalid AMI ID returned, failed regex validation: $ami_id\"\nfi\n\nlog \"Determined EC2 AMI ID for name '$ami_name' to be '$ami_id'\"\n\nlog \"Doing reverse lookup on AMI ID for safety\"\n\n# want * to remain in AWS query rather than evaluated in shell\n# shellcheck disable=SC2016\nreturned_name=\"$(\n    aws ec2 describe-images \\\n        --image-ids \"$ami_id\" \\\n        --query 'Images[0].Name' \\\n        --output text\n)\"\n\nif [ \"$returned_name\" != \"$ami_name\" ]; then\n    die \"ERROR: reverse lookup of AMI ID '$ami_id' returned '$returned_name' instead of expected '$ami_name' - aborting for safety\"\nfi\n\nlog \"Reverse lookup on AMI ID for safety correctly returned name '$returned_name'\"\n\n\nlog \"We definitely have the right AMI ID, outputting AMI ID: $ami_id\"\n\necho \"$ami_id\"\n"
  },
  {
    "path": "aws/aws_ec2_ami_share_to_account.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-03 06:01:48 +0700 (Mon, 03 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShares an AMI with another AWS account\n\nUseful to build your AMIs in your CI/CD account and then share to your various projects and environment accounts\n\n\nSee also:\n\n    https://github.com/HariSekhon/Packer\n\n\nFor building the AMI in your CI/CD account\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ami_name_or_id> <aws_account_id>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nami_name_or_id=\"$1\"\naws_account_id=\"$2\"\n\nami_id=\"$(\"$srcdir/aws_ec2_ami_name_to_id.sh\" \"$ami_name_or_id\")\"\n\ntimestamp \"Sharing AMI '$ami_id' with AWS account '$aws_account_id'\"\n\necho\n\naws ec2 modify-image-attribute \\\n    --image-id \"$ami_id\" \\\n    --launch-permission \"Add=[{UserId=$aws_account_id}]\"\n\necho\n\ntimestamp \"AMI '$ami_id' successfully shared with AWS account '$aws_account_id'\"\n"
  },
  {
    "path": "aws/aws_ec2_amis.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-13 14:42:01 +0400 (Wed, 13 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList AWS EC2 AMIs belonging to your account in an easy to read table output\n\nUseful for quickly investigating AMIs\n\nFor full details do:\n\n    aws ec2 describe-images --image-ids \\\"\\$AMI_ID\\\"\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    aws_profile=\"$1\"\n    shift || :\n    export AWS_PROFILE=\"$aws_profile\"\nfi\n\n\n# false positive - want single quotes for * to be evaluated within AWS query not shell\n# shellcheck disable=SC2016\naws ec2 describe-images \\\n    --owners self \\\n    --query 'Images | sort_by(@, &Name)[].{\" ID\":ImageId, \" Name\":Name, \"CreationDate\":CreationDate}' \\\n    --output table\n"
  },
  {
    "path": "aws/aws_ec2_ebs_create_snapshot_and_wait.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-02 10:14:03 +0300 (Fri, 02 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a snapshot of a given EBS volume ID and waits for it to complete with exponential backoff\n\nUseful to take before enlarging EBS volumes as a backup in case anything goes wrong\n\nAutomatically determines the EC2 instance name and prefixes it to the snapshot description\n\n\nUse the adjacent script aws_ec2_ebs_volumes.sh to easily get the volume ID\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ebs_volume_id> <description>\"\n\nhelp_usage \"$@\"\n\n# enforce giving a 2nd arg for description\nnum_args 2 \"$@\"\n\nvolume_id=\"$1\"\ndescription=\"$2\"\n\naws_validate_volume_id \"$volume_id\"\n\nexport MAX_WATCH_SLEEP_SECS=300\n\nexport AWS_DEFAULT_OUTPUT=json\n\ntimestamp \"Finding EC2 instance ID for volume ID '$volume_id'\"\ninstance_id=\"$(aws ec2 describe-volumes --volume-ids \"$volume_id\" --query 'Volumes[*].Attachments[*].InstanceId' --output text)\"\n\ntimestamp \"Finding EC2 instance name for instance ID '$instance_id'\"\n# false positive\n# shellcheck disable=SC2016\ninstance_name=\"$(aws ec2 describe-instances --instance-ids \"$instance_id\" --query 'Reservations[*].Instances[*].{InstanceID:InstanceId,Name:Tags[?Key==`Name`].Value|[0]}' --output json | jq -r '.[].[].Name' | head -n 1)\"\n\n# automatically prefix the description with the instance name\ndescription=\"$instance_name: $description\"\n\ntimestamp \"Taking snapshot of volume '$volume_id' on EC2 instance '$instance_name' with description '$description'\"\nsnapshot=\"$(aws ec2 create-snapshot --volume-id \"$volume_id\" --description \"$description\" --output json)\"\nsnapshot_id=\"$(jq -r '.SnapshotId' <<< \"$snapshot\")\"\necho\n\nget_aws_pending_snapshots(){\n    # false positive\n    # shellcheck disable=SC2016\n    aws ec2 describe-snapshots --snapshot-id \"$snapshot_id\" --query 'Snapshots[?State==`pending`].[VolumeId,SnapshotId,Description,State,Progress]' --output table\n}\n\n# will double this for exponential backoff up to MAX_WATCH_SLEEP_SECS interval\nsleep_secs=5\n\n# loop indefinitely until we explicitly break using time check\nwhile : ; do\n    if [ \"$SECONDS\" -gt 7200 ]; then\n        die \"Timed out waiting 2 hours for EBS snapshot to complete!\"\n    fi\n    if get_aws_pending_snapshots | tee /dev/stderr | grep -Fq \"$description\"; then\n        echo\n        timestamp \"Snapshot still in pending state, waiting $sleep_secs secs before checking again\"\n        sleep \"$sleep_secs\"\n        # exponential backoff\n        sleep_secs=\"$(exponential \"$sleep_secs\" \"$MAX_WATCH_SLEEP_SECS\")\"\n        continue\n    fi\n    break\ndone\n\necho\ntimestamp \"Snapshot completed\"\n"
  },
  {
    "path": "aws/aws_ec2_ebs_resize_and_wait.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-02 10:14:03 +0300 (Fri, 02 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nResizes an EBS volume and waits for it to complete modifying and optionally optimizing with exponential backoff\n\nThis can be done without interruption while the EC2 instance is online\n\nIf you want to wait for the optimization as well as the modification set this environment variable\n\n    export EC2_EBS_WAIT_FOR_OPTIMIZE=true\n\nUse the adjacent script aws_ec2_ebs_volumes.sh to easily get the volume ID\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ebs_volume_id> <size_in_GB>\"\n\nhelp_usage \"$@\"\n\n# enforce giving a 2nd arg for description\nnum_args 2 \"$@\"\n\nvolume_id=\"$1\"\nsize=\"$2\"\n\naws_validate_volume_id \"$volume_id\"\n\nexport MAX_WATCH_SLEEP_SECS=300\n\nexport AWS_DEFAULT_OUTPUT=json\n\ntimestamp \"Resizing volume ID '$volume_id' to $size GB\"\nebs_modification=\"$(aws ec2 modify-volume --volume-id \"$volume_id\" --size 300 --output json)\"\n\nmodification_starttime=\"$(jq -r '.VolumeModification.StartTime' <<< \"$ebs_modification\")\"\necho\n\noptimize_filter=()\nif [ \"${EC2_EBS_WAIT_FOR_OPTIMIZE:-}\" = true ]; then\n    optimize_filter=(--filters \"Name=modification-state,Values=optimizing\")\nfi\n\nget_aws_pending_ebs_modifications(){\n    # cannot quote $optimize_filter because we need it to not create a blank arg if not present\n    # shellcheck disable=SC2086\n    aws ec2 describe-volumes-modifications \\\n        --volume-ids \"$volume_id\" \\\n        --filters \"Name=modification-state,Values=modifying\" \\\n        \"${optimize_filter[@]}\" \\\n        --output text\n}\n\n# will double this for exponential backoff up to MAX_WATCH_SLEEP_SECS interval\nsleep_secs=5\n\n# loop indefinitely until we explicitly break using time check\nwhile : ; do\n    if [ \"$SECONDS\" -gt 7200 ]; then\n        die \"Timed out waiting 2 hours for EBS snapshot to modify and optimize!\"\n    fi\n    if get_aws_pending_ebs_modifications | tee /dev/stderr | grep -Fq \"$modification_starttime\"; then\n        echo\n        timestamp \"EBS modification still in progress, waiting $sleep_secs secs before checking again\"\n        sleep \"$sleep_secs\"\n        # exponential backoff\n        sleep_secs=\"$(exponential \"$sleep_secs\" \"$MAX_WATCH_SLEEP_SECS\")\"\n        continue\n    fi\n    break\ndone\n\necho\ntimestamp \"EBS resize modification completed\"\n"
  },
  {
    "path": "aws/aws_ec2_ebs_volumes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-02 09:18:28 +0300 (Fri, 02 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList EC2 instances and their EBS volumes in the current region\n\nOuptut:\n\n<instance_id>  <instance_name>  <volume_id>  <volume_name>  <volume_size_GB>  <device>  <encrypted>  <delete_on_termination>  <attached/detached>  <attached_time>\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_args \"$@\"\n\ntimestamp \"Getting EC2 instance list\"\nwhile read -r instance_id instance_name; do\n    #echo \"Instance Name: $instance_name, Instance ID: $instance_id\"\n\n    timestamp \"Getting volume list for EC2 instance: $instance_name\"\n\n    if ! is_instance_id \"$instance_id\"; then\n        die \"Invalid Instance ID passed into loop with instance name '$instance_name', failed regex validation: $instance_id\"\n    fi\n\n    # false positive\n    # shellcheck disable=SC2016\n    #volume_ids=\"$(aws ec2 describe-volumes --filters \"Name=attachment.instance-id,Values=$instance_id\" --query 'Volumes[*].[VolumeId, Tags[?Key==`Name`].Value|[0]]' --output text)\"\n    volume_info=\"$(aws ec2 describe-volumes --filters \"Name=attachment.instance-id,Values=$instance_id\" \\\n        --query 'Volumes[*].{VolumeID:VolumeId,Name:Tags[?Key==`Name`].Value|[0],Size:Size,Encrypted:Encrypted,Attachments:Attachments[*]}' --output json | jq -c '.[]')\"\n\n    while read -r volume; do\n        volume_id=\"$(jq   -r '.VolumeID'  <<< \"$volume\")\"\n        volume_name=\"$(jq -r '.Name'      <<< \"$volume\" | grep '.' || echo \"N/A\")\"\n        volume_size=\"$(jq -r '.Size'      <<< \"$volume\")\"\n        encrypted=\"$(jq   -r '.Encrypted' <<< \"$volume\")\"\n\n        while read -r attachment; do\n            device=\"$(jq                -r '.Device'              <<< \"$attachment\")\"\n            status=\"$(jq                -r '.State'               <<< \"$attachment\")\"\n            attach_time=\"$(jq           -r '.AttachTime'          <<< \"$attachment\" | grep '.' || echo \"N/A\")\"\n            delete_on_termination=\"$(jq -r '.DeleteOnTermination' <<< \"$attachment\")\"\n\n            echo \"$instance_id $instance_name $volume_id $volume_name ${volume_size}GB $device $encrypted $delete_on_termination $status $attach_time\"\n        done < <(jq -c '.Attachments[]' <<< \"$volume\")\n    done  <<< \"$volume_info\"\ndone < <(\n    # false positive\n    # shellcheck disable=SC2016\n    aws ec2 describe-instances --query 'Reservations[*].Instances[*].{InstanceID:InstanceId,Name:Tags[?Key==`Name`].Value|[0]}' --output text\n) |\ncolumn -t |\n# sort by instance name and device name\nsort -k2,6\n"
  },
  {
    "path": "aws/aws_ec2_ebs_volumes_unattached.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-29 02:40:31 +0200 (Thu, 29 Aug 2024)\n#\n#  https://github.com/HariSekhon\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList EC2 EBS volumes that are not attached to any instance\n\nOuptut:\n\n<volume_id>  <size>  <availability_zones>  <volume_type>  <state>  <name_tag>, <environment_tag>\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_args \"$@\"\n\ntimestamp \"Getting EC2 EBS volumes not attached to instances\"\necho >&2\n# Volumes[*] should not be shell interpreted\n# shellcheck disable=SC2016\naws ec2 describe-volumes \\\n    --filters 'Name=status,Values=available' \\\n    --query 'Volumes[*].{\n        \"  VolumeID\": VolumeId,\n        \"  VolumeType\": VolumeType,\n        \" Size\": Size,\n        \" State\": State,\n        AvailabilityZone: AvailabilityZone,\n        Name: Tags[?Key==\"Name\"].Value | [0],\n        Environment: Tags[?Key==\"Environment\"].Value | [0],\n        CreateTime: CreateTime\n    }' \\\n    --output table\n"
  },
  {
    "path": "aws/aws_ec2_instance_clone.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-12 16:42:30 +0400 (Tue, 12 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nClones an AWS EC2 instance by creating an AMI from the original and then booting a new instance from the AMI\nwith the same settings as the original instance\n\nUseful to testing risky things on a separate EC2 instance, such as Server Administrator recovery of Tableau\n\nDoes not reboot the running EC2 instance for safety by default, which means a non-clean filesystem copy\nunless you shut it down first.\n\nTo enforce a reboot of the EC2 instance (be careful in production!) you must set the environment variable:\n\n    export AWS_EC2_REBOOT_INSTANCE=true\n\nUses adjacent scripts:\n\n    aws_ec2_ami_create_from_instance.sh\n\n    aws_ec2_instance_name_to_id.sh\n\nInvestigate instance names quickly using adjacent script aws_ec2_instances.sh\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name> <new_instance_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\ninstance_name=\"$1\"\n\nnew_instance_name=\"$2\"\n\nami_name=\"instance-$instance_name-$(date '+%F_%H%M%S')\"\n\n# this script has been updated to wait for the AMI state to become available\nami_id=\"$(\"$srcdir/aws_ec2_ami_create_from_instance.sh\" \"$instance_name\" \"$ami_name\")\"\necho >&2\n\nif ! is_ami_id \"$ami_id\"; then\n    die \"Invalid AMI ID returned, failed regex validation: $ami_id\"\nfi\n\ntimestamp \"Determining instance ID of original EC2 instance '$instance_name'\"\ninstance_id=\"$(\"$srcdir/aws_ec2_instance_name_to_id.sh\" \"$instance_name\")\"\nif ! is_instance_id \"$instance_id\"; then\n    die \"Invalid Instance ID returned, failed regex validation: $instance_id\"\nfi\ntimestamp \"Determined instance ID to be: $instance_id\"\necho >&2\n\ntimestamp \"Determining instance type of original instance\"\ninstance_type=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query 'Reservations[*].Instances[*].InstanceType' \\\n        --output text\n)\"\nif is_blank \"$instance_type\"; then\n    die \"Failed to determine instance type\"\nfi\ntimestamp \"Determined instance type to be: $instance_type\"\necho >&2\n\ntimestamp \"Determining subnet ID of original instance\"\nsubnet_id=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query 'Reservations[*].Instances[*].SubnetId' \\\n        --output text\n)\"\nif is_blank \"$subnet_id\"; then\n    die \"Failed to determine subnet ID\"\nfi\ntimestamp \"Determined subnet ID to be: $subnet_id\"\necho >&2\n\ntimestamp \"Determining key pair name of original instance\"\nkey_name=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query 'Reservations[*].Instances[*].KeyName' \\\n        --output text\n)\"\nif is_blank \"$key_name\"; then\n    die \"Failed to determine key name\"\nfi\ntimestamp \"Determined key pair name to be: $key_name\"\necho >&2\n\ntimestamp \"Determining security group IDs of original instance\"\nsecurity_group_ids=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query 'Reservations[*].Instances[*].SecurityGroups[*].GroupId' \\\n        --output text |\n    tr '\\n' ',' |\n    sed 's/,$//'\n)\"\nif is_blank \"$security_group_ids\"; then\n    die \"Failed to determine security group IDs\"\nfi\ntimestamp \"Determined security group ID to be: $security_group_ids\"\necho >&2\n\ntimestamp \"Launching new EC2 instance from AMI '$ami_name'\"\nnew_instance_id=\"$(\n    aws ec2 run-instances \\\n        --image-id \"$ami_id\" \\\n        --instance-type \"$instance_type\" \\\n        --subnet-id \"$subnet_id\" \\\n        --key-name \"$key_name\" \\\n        --security-group-ids \"$security_group_ids\" \\\n        --tag-specifications \"ResourceType=instance,Tags=[{Key=Name,Value=$new_instance_name}]\" |\n    jq -r '.Instances[0].InstanceId'\n)\"\necho >&2\n\ntimestamp \"Waiting for new EC2 instance '$new_instance_name' ($new_instance_id) to enter running state\"\necho >&2\n\n# special variable that increments - use as a built-in timer\nSECONDS=0\n\nwhile true; do\n    state=\"$(\n        aws ec2 describe-instances \\\n            --instance-ids \"$instance_id\" \\\n            --query 'Reservations[*].Instances[*].State.Name' \\\n            --output text\n    )\"\n\n    if [ \"$state\" = \"running\" ]; then\n        timestamp \"New instance '$new_instance_name' is now running after $SECONDS seconds\"\n        break\n    elif [ \"$SECONDS\" -gt 1200 ]; then\n        die \"Waited for 20 minutes but instance did not enter running state, something is wrong, aborting...\"\n    fi\n    timestamp \"Waiting for instance '$new_instance_name' to enter running state. State: $state\"\n    sleep 10\ndone\n\necho \"$instance_id\"\n"
  },
  {
    "path": "aws/aws_ec2_instance_ip.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:05:24 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the IP address, public or private, or a given EC2 instance by name or instance id\n\nUsed by:\n\n    aws_ec2_ami_boot_ssh.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name_or_id>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ninstance=\"$1\"\n\ntimestamp \"Getting instance public IP for instance: $instance\"\n\ninstance_id=\"$(\"$srcdir/aws_ec2_instance_name_to_id.sh\" \"$instance\")\"\ntimestamp \"Instance ID: $instance_id\"\n\npublic_ip=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query \"Reservations[0].Instances[0].PublicIpAddress\" \\\n        --output text\n)\"\n\nif ! is_blank \"$public_ip\" &&\n   [ \"$public_ip\" != \"None\" ]; then\n    ip=\"$public_ip\"\n    timestamp \"Using instance public IP: $ip\"\nelse\n    timestamp \"No public IP found, getting instance private IP\"\n    private_ip=\"$(\n        aws ec2 describe-instances \\\n            --instance-ids \"$instance_id\" \\\n            --query \"Reservations[0].Instances[0].PrivateIpAddress\" \\\n            --output text\n    )\"\n    ip=\"$private_ip\"\n    timestamp \"Using instance private IP: $ip\"\nfi\n\ntimestamp \"IP address is: $ip\"\necho \"$ip\"\n"
  },
  {
    "path": "aws/aws_ec2_instance_name_to_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-27 11:28:25 +0200 (Tue, 27 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns an EC2 instance ID from name\n\nAdds additional safety checks:\n\n- verifies no more than one instance ID is returned\n- does a reverse lookup on the instance ID to verify the name\n- if an instance ID is passed, returns it as is for convenience\n\nCalled by adjacent scripts like:\n\n    aws_ec2_ami_create_from_instance.sh\n\n    aws_ec2_instance_terminate_by_name.sh\n\nInvestigate instance names and IDs quickly using adjacent script:\n\n    aws_ec2_instances.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ninstance_name=\"$1\"\n\nif [[ \"$instance_name\" =~ ^i-[0-9a-f]{8,17}$ ]]; then\n    log \"Given instance name is already an AWS instance ID, outputs as is: $instance_name\"\n    echo \"$instance_name\"\n    exit 0\nfi\n\nlog \"Determining EC2 instance ID for name '$instance_name'\"\n\ninstance_id=\"$(\n    aws ec2 describe-instances \\\n        --filters \"Name=tag:Name,Values=$instance_name\" \\\n        --query \"Reservations[*].Instances[*].InstanceId\" \\\n        --output text\n)\"\n\nif is_blank \"$instance_id\"; then\n    die \"No EC2 instance found with name '$instance_name'\"\nfi\n\nif [ \"$(awk '{print NF}' <<< \"$instance_id\")\" -gt 1 ]; then\n    cat >&2 <<EOF\nMore than 1 instance ID returned, aborting for safety!\"\n\nInstance IDs found:\n\n$instance_id\nEOF\n    exit 1\nfi\n\nif ! is_instance_id \"$instance_id\"; then\n    die \"Invalid Instance ID returned, failed regex validation: $instance_id\"\nfi\n\nlog \"Determined EC2 instance ID for name '$instance_name' to be '$instance_id'\"\n\nlog \"Doing reverse lookup on instance ID for safety\"\n\n# want * to remain in AWS query rather than evaluated in shell\n# shellcheck disable=SC2016\nreturned_name=\"$(\n    aws ec2 describe-instances \\\n        --instance-ids \"$instance_id\" \\\n        --query 'Reservations[*].Instances[*].Tags[?Key==`Name`].Value' \\\n        --output text\n)\"\n\nif [ \"$returned_name\" != \"$instance_name\" ]; then\n    die \"ERROR: reverse lookup of instance id '$instance_id' returned '$returned_name' instead of expected '$instance_name' - aborting for safety\"\nfi\n\nlog \"Reverse lookup on instance ID for safety correctly returned name '$returned_name'\"\n\n\nlog \"We definitely have the right instance ID, outputting instance ID: $instance_id\"\n\necho \"$instance_id\"\n"
  },
  {
    "path": "aws/aws_ec2_instance_terminate_by_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-27 11:28:25 +0200 (Tue, 27 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTerminate an AWS EC2 instance by name\n\nInvestigate instance names quickly using adjacent script aws_ec2_instances.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ninstance_name=\"$1\"\n\ninstance_id=\"$(VERBOSE=1 \"$srcdir/aws_ec2_instance_name_to_id.sh\" \"$instance_name\")\"\n\nif ! is_instance_id \"$instance_id\"; then\n    die \"Invalid Instance ID returned, failed regex validation: $instance_id\"\nfi\n\necho\n\ntimestamp \"Checking instance state\"\ninstance_state=\"$(\n    aws ec2 describe-instances --instance-ids \"$instance_id\" --query 'Reservations[*].Instances[*].State.Name' --output text\n)\"\nif [ \"$instance_state\" = \"terminated\" ]; then\n    timestamp \"Instance '$instance_name' with id '$instance_id' is already terminated\"\n    exit 0\nelif [ \"$instance_state\" != \"running\" ]; then\n    die \"Instance state '$instance_state' was not expected - is not 'terminated' or 'running' - aborting for safety\"\nfi\necho >&2\n\nread -r -p \"Do you want to terminate instance '$instance_name' with id '$instance_id'? (y/N) \" answer\ncheck_yes \"$answer\"\naws ec2 terminate-instances --instance-ids \"$instance_id\"\n"
  },
  {
    "path": "aws/aws_ec2_instance_wait_for_ready.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:26:34 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPolls an AWS EC2 instance and waits for it to finish initializing to a ready state\n\nUsed by adjacent script aws_eks_ami_create.sh\n\nGet the AWS EC2 Instance ID from the output of the 'aws ec2 run-instances' command - see above script\n\nTimeout secs defaults to 300 if not specified\nCheck interval defaults to 5 seconds if not specified\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_id> [<timeout_secs> <check_interval>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ninstance_id=\"$1\"\n\ntimeout_secs=\"${2:-300}\"\n\ncheck_interval_secs=\"${3:-5}\"\n\naws_validate_instance_id \"$instance_id\"\n\nif ! is_int \"$timeout_secs\"; then\n\tdie \"Invalid Timeout Secs, must be an integer: $timeout_secs\"\nfi\n\nif ! is_int \"$check_interval_secs\"; then\n\tdie \"Invalid Check Interval Secs, must be an integer: $check_interval_secs\"\nfi\n\nif [ \"$timeout_secs\" -lt 1 ]; then\n\tdie \"Invalid Timeout Secs cannot be less than 1: $timeout_secs\"\nfi\n\nif [ \"$check_interval_secs\" -lt 1 ]; then\n\tdie \"Invalid Check Interval Secs cannot be less than 1: $check_interval_secs\"\nfi\n\ntimestamp \"Waiting for AWS EC2 instance to become ready: $instance_id\"\ntimestamp \"Check interval: $check_interval_secs secs\"\ntimestamp \"Timeout max: $timeout_secs secs\"\n\nSECONDS=0\n\ntimestamp \"Checking EC2 instance status...\"\nwhile true; do\n    if [ \"$SECONDS\" -gt \"$timeout_secs\" ]; then\n        die \"ERROR: Timed out waiting $timeout_secs for instance to become ready\"\n    fi\n    instance_checks=\"$(\n        aws ec2 describe-instance-status \\\n            --instance-ids \"$instance_id\" \\\n            --query \"InstanceStatuses[0].InstanceStatus.Status\" \\\n            --output text\n    )\"\n    if [ \"$instance_checks\" = \"ok\" ]; then\n        system_checks=\"$(\n            aws ec2 describe-instance-status \\\n                --instance-ids \"$instance_id\" \\\n                --query \"InstanceStatuses[0].SystemStatus.Status\" \\\n                --output text\n        )\"\n\n        if [ \"$instance_checks\" = \"ok\" ] &&\n           [ \"$system_checks\" = \"ok\" ]; then\n            timestamp \"EC2 instance $instance_id is ready\"\n            break\n        fi\n    fi\n    timestamp \"AWS EC2 instance still initializing...\"\n    sleep \"$check_interval_secs\"\ndone\n"
  },
  {
    "path": "aws/aws_ec2_instances.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-27 11:28:20 +0200 (Tue, 27 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList AWS EC2 instances, their DNS names and States in an easy to read table output\n\nUseful for quickly investigating running instances and comparing to configured FQDN addresses in referencing software\n\nSee also:\n\n    aws_ec2_info.sh - gives similar info but also resolves AMI names and adds an architecture column\n    aws_ec2_info_csv.sh - same as above but in quoted CSV format\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n# false positive - want single quotes for * to be evaluated within AWS query not shell\n# shellcheck disable=SC2016\n#\n# prefixing the column headings with spaces forces them to come first so that we can get the State field\n# in the middle instead of end since AWS CLI seems to sort the columns lexically\naws ec2 describe-instances \\\n    --query 'Reservations[*].Instances[*].{\n                \"  Name\": Tags[?Key==`Name`].Value | [0],\n                \"  ID\": InstanceId,\n                \"  State\": State.Name,\n                \" InstanceType\": InstanceType,\n                \"Public DNS\": publicDnsName,\n                \"Private DNS\": PrivateDnsName\n            }' \\\n    --output table\n"
  },
  {
    "path": "aws/aws_ec2_launch_templates_ami_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-09 20:02:53 +0700 (Thu, 09 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each Launch Template lists the AMI ID of the latest version\n\nUseful to check EKS upgrades of node groups via Terragrunt have taken effect\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n# returns in format:\n#\n#   ID  NAME\n#\nlaunch_templates=\"$(aws ec2 describe-launch-templates --query \"LaunchTemplates[].{Name:LaunchTemplateName,ID:LaunchTemplateId}\" --output text)\"\n\nwhile read -r _id name; do\n    echo -n \"$name \"\n    aws ec2 describe-launch-template-versions --launch-template-name \"$name\" |\n    jq -r '.LaunchTemplateVersions[0].LaunchTemplateData.ImageId'\ndone <<< \"$launch_templates\" # |\n#column -t\n"
  },
  {
    "path": "aws/aws_ecr_alternate_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:30:51 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all alternative tags for the given specific ECR docker image:tag\n\nIf a container has multiple tags (eg. latest, v1, hashref), you can supply '<image>:latest' to see which version has been tagged to 'latest'\n\nEach tag for the given <image>:<tag> is output on a separate line for easy further piping and filtering, including the originally supplied tag\n\nIf no tag is given, assumes 'latest'\n\nIf the image isn't found in GCR, will return nothing and no error code since this is the default GCloud SDK behaviour\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage_tag=\"$1\"\nshift || :\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif [ -z \"$tag\" ] || [ \"$tag\" = \"$image\" ]; then\n    tag=\"latest\"\nfi\n\naws ecr describe-images --repository-name \"$image\" --image-ids \"imageTag=$tag\" \"$@\" |\njq -r '.imageDetails[].imageTags[]' |\nsort\n"
  },
  {
    "path": "aws/aws_ecr_delete_old_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes tags old than N days for a given AWS ECR image\n\nUseful to clean out old CI image builds to save S3 storage costs on old CI images you no longer use\n\n\nPrompts with the list of image:tags that it will delete before proceeding for safety.\n\n\n$usage_aws_cli_required\n\n\nSee Also:\n\n    aws_ecr_tags_old.sh        - used by this script, lists all image:tag older than N days\n    aws_ecr_tags_timestamps.sh - lists tags and timestamps - useful for comparing with the output from aws_ecr_tags_old.sh\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<days_threshold> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nimage=\"$1\"\ndays=\"$2\"\nshift || :\nshift || :\n\nimage_tags=\"$(\"$srcdir/aws_ecr_tags_old.sh\" \"$image\" \"$days\" \"$@\")\"\n\nif [ -z \"$image_tags\" ]; then\n    echo \"No image:tags older than $days old\"\n    exit 0\nfi\n\necho\necho \"List of image:tags that will be deleted:\"\necho\necho \"$image_tags\"\necho\n\nread -r -p 'Are you sure you want to delete these image:tags listed above? (y/N) ' answer\necho\n\nif [ \"$answer\" != \"y\" ]; then\n    echo \"Aborting...\"\n    exit 1\nfi\n\nfor image_tag in $image_tags; do\n    \"$srcdir/aws_ecr_delete_tag.sh\" \"$image_tag\" \"$@\"\ndone\n"
  },
  {
    "path": "aws/aws_ecr_delete_tag.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a tag for the given AWS ECR docker image\n\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage_tag=\"$1\"\nshift || :\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif [ -z \"$tag\" ] || [ \"$tag\" = \"$image\" ]; then\n    usage \"tag not given\"\nfi\n\ntimestamp \"deleting tag '$tag' for ECR image '$image'\"\n# negate the result of the pipe\naws ecr batch-delete-image --repository-name \"$image\" --image-ids \"imageTag=$tag\" \"$@\" |\nif jq -e '.failures[0]'; then\n    exit 1\nfi\n"
  },
  {
    "path": "aws/aws_ecr_docker_build_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-22 15:11:27 +0100 (Fri, 22 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBuilds the local docker image using the Dockerfile in the current directory and pushes it to the AWS ECR registry\n\nTags the docker image with the following and pushes all tags to AWS ECR:\n\n    - latest\n    - Git full hashref\n    - Git branch\n    - any Git tags, if found, for easy versioning support\n    - date (YYYY-MM-DD)\n    - datetimestamp (YYYYMMDDThhmmssZ] in UTC\n\nRequires AWS CLI to be installed and configured, as well as Docker to be running locally\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ecr_registry> <repo>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nECR=\"$1\"\nREPO=\"$2\"\n\nif ! is_aws_ecr_registry \"$ECR\"; then\n    usage \"Invalid ECR address given:  $ECR\"\nfi\n\nif ! [[ \"$REPO\" =~ ^[[:alnum:]/-]+$ ]]; then\n    usage \"Invalid Repo name given:  $REPO\"\nfi\n\nif is_CI; then\n    docker version\n    echo\nfi\n\necho \"* AWS ECR -> Docker login\"\n# $AWS_DEFAULT_REGION should be set in env or profile\n\"$srcdir/aws_ecr_docker_login.sh\" \"$ECR\"\necho\n\necho \"* Determining tags\"\nhashref=\"$(git rev-parse HEAD)\"\ngit_branch=\"$(git rev-parse --abbrev-ref HEAD)\"\ngit_tags=\"$(git tag --points-at HEAD)\"  # can return multiple tags\n# must use date -u switch since --utc only works on Linux and not Mac\ndate=\"$(date -u '+%F')\"\ntimestamp=\"$(date -u '+%FT%H%M%SZ')\"\n\n# adding tags:\n#\ntags=\"\n$git_branch\n$git_tags\n$date\n$timestamp\n\"\necho\n\nexport DOCKER_BUILDKIT=1\n\n# shellcheck disable=SC2046\ndocker build -t \"$ECR/$REPO:$hashref\" . \\\n             --build-arg BUILDKIT_INLINE_CACHE=1 \\\n             --cache-from \"$ECR/$REPO:latest\" \\\n             --cache-from \"$ECR/$REPO:$hashref\" \\\n             $(for tag in $tags; do echo -n \" --cache-from $ECR/$REPO:$tag\"; done)\necho\n\nfor tag in latest $tags; do\n    echo \"* Tagging as '$tag'\"\n    docker tag \"$ECR/$REPO:$hashref\" \"$ECR/$REPO:$tag\"\n    echo\ndone\n\n# pushing latest last intentionally for a more atomic update\nfor tag in \"$hashref\" $tags latest; do\n    echo \"* Pushing tag '$tag'\"\n    docker push \"$ECR/$REPO:$tag\"\n    echo\ndone\n"
  },
  {
    "path": "aws/aws_ecr_docker_login.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-17 02:39:54 +0700 (Tue, 17 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAuthenticates Docker to AWS ECR, inferring the ECR registry from the current AWS Account ID and Region\n\nIf \\$AWS_ACCOUNT_ID and \\$AWS_DEFAULT_REGION are not set in the environment,\ntries to infer them from the current AWS config\n\n\n$usage_aws_cli_required, and also Docker must be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<ecr_registry>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nexport AWS_ACCOUNT_ID=\"${AWS_ACCOUNT_ID:-$(aws_account_id)}\"\n\nexport AWS_DEFAULT_REGION=\"${AWS_DEFAULT_REGION:-$(aws_region)}\"\n\naws_ecr_registry=\"${1:-$(aws_ecr_registry)}\"\n\nif ! is_aws_ecr_registry \"$aws_ecr_registry\"; then\n    die \"Invalid AWS ECR registry: $aws_ecr_registry\"\nfi\n\ntimestamp \"Getting AWS ECR Login password and piping it into Docker for registry: $aws_ecr_registry\"\naws ecr get-login-password |\ndocker login --username AWS --password-stdin \"$aws_ecr_registry\"\n"
  },
  {
    "path": "aws/aws_ecr_list_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-02 19:54:54 +0200 (Tue, 02 Jul 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists ECR repos, their docker image mutability and whether image scanning is enabled\n\nOutput:\n\n<repo_name>    <mutable_tags>    <scanning_enabled>\n\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws ecr describe-repositories |\njq -r '\n    .repositories[] |\n        [\n            .repositoryName,\n            .imageTagMutability,\n            .imageScanningConfiguration.scanOnPush\n         ] |\n    @tsv\n' |\nsort |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_ecr_list_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all tags for the given AWS ECR docker image\n\nEach tag for the given image is output on a separate line for easy further piping and filtering\n\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage=\"$1\"\nshift || :\n\naws ecr list-images --repository \"$image\" \"$@\" |\njq -r '.imageIds[].imageTag | select(.)' |\nsort\n"
  },
  {
    "path": "aws/aws_ecr_newest_image_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the tags for the given AWS ECR docker image with the newest creation date\n\n(eg. for tagging it as 'latest', see adjacent scripts aws_ecr_tag_latest.sh and aws_ecr_tag_newest_image_as_latest.sh)\n\nWhen a docker image has multiple tags (eg. v1, latest) then outputs each tag on a separate line for easy further piping and filtering\n\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage=\"$1\"\nshift || :\n\nnewest_image_timestamp=\"$(aws ecr describe-images --repository-name \"$image\" \"$@\" | jq -r '.imageDetails[].imagePushedAt' | sort -r | head -n1)\"\n\naws ecr describe-images --repository-name \"$image\" \"$@\" |\njq -r \".imageDetails[]? | select(.imagePushedAt == \\\"$newest_image_timestamp\\\") | .imageTags[]?\" |\nsort\n"
  },
  {
    "path": "aws/aws_ecr_tag_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given AWS ECR docker image:tag with the current branch name without pulling and pushing the docker image\n\n\n$usage_aws_cli_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage_tag=\"$1\"\nshift || :\n\nif ! [[ \"$image_tag\" =~ : ]]; then\n    image_tag+=\":latest\"\nfi\n\ndocker_image=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\n\n# Jenkins provides GIT_BRANCH, TeamCity doesn't so normalize and determine it if not automatically set\nif [ -z \"${BRANCH_NAME:-}\" ]; then\n    BRANCH_NAME=\"${GIT_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}\"\nfi\nBRANCH_NAME=\"${BRANCH_NAME##*/}\"\n\nFORCE=1 \"$srcdir/aws_ecr_tag_image.sh\" \"$docker_image:$tag\" \"$BRANCH_NAME\" \"$@\"\n"
  },
  {
    "path": "aws/aws_ecr_tag_datetime.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given ECR docker image with it's creation Date and Timestamp\nwithout pulling and pushing the docker image\n\nThe timestamp is the created time (either uploaded or created by Google Cloud Build)\n\nTags are in the format:\n\nYYYY-MM-DD\nYYYY-MM-DDTHHMMSSZ  (standard ISO UTC time without semi-colons which are invalid in docker tags)\n\nThe timestamp will be normalized to UTC\n\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage_tag=\"$1\"\nshift || :\n\nif ! [[ \"$image_tag\" =~ : ]]; then\n    image_tag+=\":latest\"\nfi\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\n\nif [ -z \"$tag\" ] ||\n   [ \"$tag\" = \"$image\" ] ||\n   [ \"$tag\" = \"$image_tag\" ]; then\n    tag=\"latest\"\nfi\n\ntimestamp=\"$(aws ecr describe-images --repository-name \"$image\" --image-ids \"imageTag=$tag\" \"$@\" | jq -r '.imageDetails[].imagePushedAt' | sort -r | head -n1)\"\nif [ -z \"$timestamp\" ]; then\n    echo \"Failed to determine timestamp from ECR for image '$image' with tag '$tag'\"\n    exit 1\nfi\nif ! [[ \"$timestamp\" =~ ^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}T[[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}[+-][[:digit:]]{2}:[[:digit:]]{2}$ ]]; then\n    echo \"ECR timestamp not in expect YYYY-MM-DDTHH:MM:SS[+-]HH:MM format, API may have changed\"\n    exit 1\nfi\n\n# normalize to UTC\ntimestamp=\"$(date --utc --date=\"$timestamp\" '+%FT%H%M%SZ')\"\n\ndate=\"${timestamp%T*}\"\n\nFORCE=1 \"$srcdir/aws_ecr_tag_image.sh\" \"$image:$tag\" \"$date\" \"$@\"\nFORCE=1 \"$srcdir/aws_ecr_tag_image.sh\" \"$image:$tag\" \"$timestamp\" \"$@\"\n"
  },
  {
    "path": "aws/aws_ecr_tag_image.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-28 11:45:38 +0100 (Mon, 28 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-retag.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags an AWS ECR image with another tag without pulling + pushing the image\n\nIf :<tag> isn't given, assumes 'latest'\nIf the environment variable FORCE is set, will remove the new tag reference to ensure the new tagging takes effect\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_tag_image_by_digest.sh - same as this script but locates the image to tag using a digest\n\n    docker_registry_tag_image.sh - for private Docker Registries\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> <new_tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nimage_tag=\"$1\"\nnew_tag=\"$2\"\nshift || :\nshift || :\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif ! [[ \"$image_tag\" =~ : ]] &&\n   [ \"$tag\" = \"$image\" ]; then\n    tag=\"latest\"\nfi\n\n\ntstamp \"getting manifest for image '$image:$tag'\"\nmanifest=\"$(aws ecr batch-get-image --repository-name \"$image\" --image-ids \"imageTag=$tag\" --query 'images[].imageManifest' --output text \"$@\")\"\nif is_blank \"$manifest\"; then\n    die \"ERROR: no manifest returned, did you specify a valid image tag?\"\nfi\n\nif [ -n \"${FORCE:-}\" ]; then\n    \"$srcdir/aws_ecr_delete_tag.sh\" \"$image:$new_tag\" \"$@\" >/dev/null || :\nfi\ntstamp \"tagging image '$image:$tag' with new tag '$new_tag'\"\naws ecr put-image --repository-name \"$image\" --image-tag \"$new_tag\" --image-manifest \"$manifest\" \"$@\" >/dev/null\n\ntstamp \"tags for image '$image:$tag' are now:\"\naws ecr describe-images --repository-name \"$image\" --output json \"$@\" |\njq -r \".imageDetails[] | select(.imageTags) | select(.imageTags[] == \\\"$tag\\\") | .imageTags[]\"\n"
  },
  {
    "path": "aws/aws_ecr_tag_image_by_digest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-28 11:45:38 +0100 (Mon, 28 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.aws.amazon.com/AmazonECR/latest/userguide/image-retag.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags an AWS ECR image digest with another tag without pulling + pushing the image\n\nUseful to recover an <untagged> image and apply a new tag to it or if you want to more precisely tag an exact image than following another existing tag (which is usually easier but can be a moving target)\n\nIf the environment variable FORCE is set, will remove the new tag reference to ensure the new tagging takes effect\n\n$usage_aws_cli_jq_required\n\n\nSimilar scripts:\n\n    aws_ecr_tag_image.sh - same as this script but locates the image using an existing tag\n\n    docker_registry_tag_image.sh - for private Docker Registries\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> <digest> <new_tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\nimage=\"$1\"\ndigest=\"$2\"\nnew_tag=\"$3\"\n\nif ! [[ \"$digest\" =~ : ]]; then\n    digest=\"sha256:$digest\"\nfi\n\ntstamp \"getting manifest for image '$image' with digest '$digest'\"\nmanifest=\"$(aws ecr batch-get-image --repository-name \"$image\" --image-ids \"imageDigest=$digest\" --query 'images[].imageManifest' --output text \"$@\")\"\nif is_blank \"$manifest\"; then\n    die \"ERROR: no manifest returned, did you specify a valid digest?\"\nfi\n\nif [ -n \"${FORCE:-}\" ]; then\n    \"$srcdir/aws_ecr_delete_tag.sh\" \"$image:$new_tag\" \"$@\" >/dev/null || :\nfi\ntstamp \"tagging image '$image' with digest '$digest' with new tag '$new_tag'\"\naws ecr put-image --repository-name \"$image\" --image-tag \"$new_tag\" --image-manifest \"$manifest\" \"$@\" >/dev/null\n\ntstamp \"tags for image '$image' with digest '$digest' are now:\"\naws ecr describe-images --repository-name \"$image\" --output json \"$@\" |\njq -r \".imageDetails[] | select(.imageDigest) | select(.imageDigest == \\\"$digest\\\") | .imageTags[]\"\n"
  },
  {
    "path": "aws/aws_ecr_tag_latest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given AWS ECR docker image:tag as 'latest' without pulling and pushing the docker image\n\n\n$usage_aws_cli_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image>:<tag> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage_tag=\"$1\"\nshift || :\n\nif ! [[ \"$image_tag\" =~ : ]]; then\n    usage \"tag suffix missing from docker image\"\nfi\n\ndocker_image=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif is_blank \"$tag\"; then\n    usage \"tag suffix is blank\"\nfi\n\nFORCE=1 \"$srcdir/aws_ecr_tag_image.sh\" \"$docker_image:$tag\" \"latest\" \"$@\"\n"
  },
  {
    "path": "aws/aws_ecr_tag_newest_as_latest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the newest build of a given AWS ECR docker image by creation date and tags it as 'latest'\n\nDoes this via metadata API calls to avoid network transfer from any docker pull / docker push\n\nIf an AWS ECR image has multiple tags, will take the longest tag which is assumed to be the most specific and\ntherefore most likely to avoid collisions and race conditions of other tag updates happening concurrently\n\n\n$usage_aws_cli_required\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage=\"$1\"\nshift || :\n\ntags=\"$(\"$srcdir/aws_ecr_newest_image_tags.sh\" \"$image\" \"$@\")\"\n\nif [ -z \"$tags\" ]; then\n    die \"No tags were found for image '$image'... does it exist in ECR?\"\nfi\n\nlongest_tag=\"$(awk '{print length, $0}' <<< \"$tags\" |\n               sort -nr |\n               head -n 1 |\n               awk '{print $2}')\"\n\n\"$srcdir/aws_ecr_tag_latest.sh\" \"$image:$longest_tag\" \"$@\"\n"
  },
  {
    "path": "aws/aws_ecr_tags_old.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists old tags for a given AWS ECR image > \\$days old\n\nThe \\$days threshold defaults to (365 * 2) ie. 2 years old\n\nYou can grep and pipe this output to\n\n    | xargs -L1 aws_ecr_delete_tag.sh\n\nto clean out old CI image builds to save S3 storage costs on old CI images you no longer use\n\n\n$usage_aws_cli_jq_required\n\n\nSee Also:\n\n    aws_ecr_tags_timestamps.sh - lists tags and timestamps - useful for comparing with the output from this script\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<days_threshold> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage=\"$1\"\n\n# 2 years old images by default\ndays_threshold=\"${2:-$((365 * 2))}\"\n\nshift || :\nshift || :\n\ndate_threshold=\"$(date '+%FT%T+00:00' --utc --date=\"$days_threshold days ago\")\"\n\naws ecr describe-images --repository-name \"$image\" \"$@\" |\njq -r \".imageDetails[] |\n       select(.imagePushedAt < \\\"$date_threshold\\\") |\n       [\\\"$image\\\" + \\\":\\\" + .imageTags[]] |\n       .[]\"\n"
  },
  {
    "path": "aws/aws_ecr_tags_timestamps.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest:1.0 stable\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-10 11:53:32 +0000 (Fri, 10 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all tags for the given AWS ECR docker image as well as their image upload timestamp, newest first\n\nOutput Format:\n\n<timestamp>   <tag>\n\nEach timestamp and tag for the given image is output tab separated on a separate line for easy further piping and filtering\n\n\n$usage_aws_cli_jq_required\n\n\nIf you want to remove an extra tag from an existing image:\n\n    aws ecr batch-delete-image --repository-name <image> --image-ids \\\"imageTag=<tag>\\\"\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nimage=\"$1\"\nshift || :\n\naws ecr describe-images --repository \"$image\" \"$@\" |\njq -r '.imageDetails | map(.imagePushedAt + \"\\t\" + .imageTags[]?) | .[]' |\nsort -r\n"
  },
  {
    "path": "aws/aws_eks_addon_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-01 01:52:09 +0700 (Wed, 01 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the EKS addon versions available for the given cluster by checking its version before checking addons\n\nRequires either first arg of the EKS cluster name, or the environment variable \\$EKS_CLUSTER\n\nIf neither are given, checks clusters and if only one is found in account, uses that\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<cluster_name>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncluster=\"${1:-${EKS_CLUSTER:-}}\"\n\nif is_blank \"$cluster\"; then\n    cluster=\"$(aws_eks_cluster_if_only_one)\"\n    if ! is_blank \"$cluster\"; then\n        timestamp \"No cluster specified but only one found in this account, using that: $cluster\"\n    else\n        usage \"Need to define cluster name\"\n    fi\nfi\n\ntimestamp \"Getting cluster version for: $cluster\"\ncluster_version=\"$(aws eks describe-cluster --name \"$cluster\" --query \"cluster.version\" --output text)\"\ntimestamp \"Cluster version: $cluster_version\"\n\naws eks describe-addon-versions \\\n    --kubernetes-version \"$cluster_version\" \\\n    --addon-name vpc-cni \\\n    --query 'addons[].addonVersions[].{Version: addonVersion, Defaultversion: compatibilities[0].defaultVersion}' \\\n    --output table\n"
  },
  {
    "path": "aws/aws_eks_ami_create.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:05:24 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreate a custom EKS AMI quickly off the base EKS template and then running a shell script in it\nbefore saving it to a new AMI\n\n- finds the standard EKS AMI for the given version\n- checks if there is already an EC2 running instance tagged for this\n- boots an EC2 instance from the above AMI with the given security group, subnet id and SSH key-name given so you can SSH to its ec2-user\n- waits for the EC2 instance to boot\n- waits for the EC2 instance to pass its instance and system checks\n- determines the public or private IP address\n- scp's the local script to the instance /tmp\n- SSH's to execute the script (eg. to install the needed things)\n- Assumes ~/.ssh/<ssh-key-name>,pem is present locally to be able to log in to it\n- Creates the AMI\n- Terminates the EC2 instance\n\n\nUses adjacent script:\n\n    aws_ec2_ami_boot.sh\n\n\nYou should really use Packer instead, see\n\n    https://github.com/HariSekhon/Packer\n\nBut this script is an alternative and allowed me to debug something in a pinch\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<custom_ami_name> <eks_version> <instance_type> <security_group> <subnet_id> <ssh-key-name> <script> [<instance_profile>]\"\n\nhelp_usage \"$@\"\n\nmin_args 7 \"$@\"\nmax_args 8 \"$@\"\n\ncustom_ami_name=\"$1\"\neks_version=\"$2\"\ninstance_type=\"$3\"\nsecurity_group=\"$4\"\nsubnet_id=\"$5\"\nssh_key_name=\"$6\"\nscript=\"$7\"  # on local filesystem because its private\ninstance_profile=\"${8:-}\"\n\naws_validate_security_group_id \"$security_group\"\n\naws_validate_subnet_id \"$subnet_id\"\n\nif ! is_blank \"$instance_profile\"; then\n    if ! [[ \"$instance_profile\" =~ ^[A-Za-z0-9+=,.@_-]+$ ]]; then\n        die \"Invalid Instance Profile name: $instance_profile\"\n    fi\nfi\n\ntimestamp \"Getting the latest EKS optimized AMI for $eks_version\"\nbase_ami=\"$(\n    aws ssm get-parameters \\\n        --names \"/aws/service/eks/optimized-ami/$eks_version/amazon-linux-2/recommended/image_id\" \\\n        --query \"Parameters[0].Value\" --output text\n)\"\n\nif is_blank \"$base_ami\"; then\n    die \"Failed to determine EKS AMI for version $eks_version\"\nfi\ntimestamp \"Base EKS AMI is: $base_ami\"\necho >&2\n\ninstance_id=\"$(\"$srcdir/aws_ec2_ami_boot.sh\" \"$base_ami\" \"$instance_type\" \"$security_group\" \"$subnet_id\" \"$ssh_key_name\" ${instance_profile:+\"$instance_profile\"})\"\n\nip=\"$(\"$srcdir/aws_ec2_instance_ip.sh\" \"$instance_id\")\"\n\n# EC2 instance isn't SSM integrated at this point to send commands\n#\n#timestamp \"Copying custom script to instance via SSM\"\n#aws ssm send-command \\\n#    --instance-ids \"$instance_id\" \\\n#    --document-name \"AWS-RunShellScript\" \\\n#    --comment \"Running Custom Install Script\" \\\n#    --parameters \"commands=[\\\"echo '$script' > /tmp/custom-script.sh\\\", \\\"chmod +x /tmp/custom-script.sh\\\"]\"\n#    #--parameters 'commands=[\"curl -O '\"$script_url\"'\", \"bash ./your-custom-script.sh\"]'\n#\n#timestamp \"Running custom install script via SSM\"\n#command_id=\"$(\n#    aws ssm send-command \\\n#        --instance-ids \"$instance_id\" \\\n#        --document-name \"AWS-RunShellScript\" \\\n#        --comment \"Running Custom Install Script\" \\\n#        --parameters 'commands=[\"/tmp/custom-script.sh\"]' \\\n#        --query \"Command.CommandId\" \\\n#        --output text\n#)\"\n\n#timestamp \"Waiting for script execution...\"\n#\"$srcdir/aws_ssm_wait_for_command_to_finish.sh\" \"$command_id\"\n\ntimestamp \"Copying script to instance: $script\"\n\n# this is a brand new instance so the SSH host key won't be trusted\nscp -i ~/.ssh/\"$ssh_key_name.pem\" \\\n    -o StrictHostKeyChecking=no \\\n    \"$script\" \\\n    ec2-user@\"$ip\":/tmp/\necho >&2\n\ninstance_script=\"/tmp/${script##*/}\"\n\ntimestamp \"Executing script on instance: $instance_script\"\nssh -i ~/.ssh/\"$ssh_key_name.pem\" ec2-user@\"$ip\" \"chmod +x $instance_script && $instance_script\"\n\necho >&2\n\n\"$srcdir/aws_ec2_ami_create_from_instance.sh\" \"$instance_id\" \"$custom_ami_name\"\n\n#timestamp \"Creating Custom AMI from running EC2 instance\"\n#custom_ami_id=\"$(\n#    aws ec2 create-image \\\n#        --instance-id \"$instance_id\" \\\n#        --name \"$custom_ami_name\" \\\n#        --no-reboot \\\n#        --query \"ImageId\" \\\n#        --output text\n#)\"\n#timestamp \"Custom AMI creation initiated: $custom_ami_id\"\n\necho >&2\n\ntimestamp \"Terminating temporary EC2 instance: $instance_id\"\naws ec2 terminate-instances --instance-ids \"$instance_id\"\ntimestamp \"Terminated temporary EC2 instance: $instance_id\"\n\ntimestamp \"Waiting for EC2 instance termination\"\naws ec2 wait instance-terminated --instance-ids \"$instance_id\"\ntimestamp \"Instance terminated\"\ntimestamp \"Custom EKS AMI creation completed\"\n"
  },
  {
    "path": "aws/aws_eks_available_ips.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-01 02:37:24 +0700 (Wed, 01 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the number of available IP addresses in the EKS subnets for the given cluster (5 required for an EKS upgrade)\n\nRequires either first arg of the EKS cluster name, or the environment variable \\$EKS_CLUSTER\n\nIf neither are given, checks clusters and if only one is found in account, uses that\n\nOutput should look like this:\n\n---------------------------------------------------\n|                 DescribeSubnets                 |\n+---------------------------+--------------+------+\n|  subnet-067fa8ee8476abbd6 |  eu-west-1a  |  119 |\n|  subnet-0056f7403b17d2b43 |  eu-west-1a  |  121 |\n|  subnet-09586f8fb3addbc8c |  eu-west-1b  |  109 |\n|  subnet-047f3d276a22c6bce |  eu-west-1b  |  118 |\n+---------------------------+--------------+------+\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<cluster_name>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncluster=\"${1:-${EKS_CLUSTER:-}}\"\n\nif is_blank \"$cluster\"; then\n    cluster=\"$(aws_eks_cluster_if_only_one)\"\n    if ! is_blank \"$cluster\"; then\n        timestamp \"No cluster specified but only one found in this account, using that: $cluster\"\n    else\n        usage \"Need to define cluster name\"\n    fi\nfi\n\ntimestamp \"Getting subnets for EKS cluster: $cluster\"\nsubnet_ids=\"$(\n    aws eks describe-cluster --name \"$cluster\" \\\n        --query 'cluster.resourcesVpcConfig.subnetIds' \\\n        --output text\n)\"\necho >&2\n\ntimestamp \"Getting subnets and their AvailableIpAddressCount\"\necho >&2\n# needs splitting, quoting them with strings or comma separators both break in AWS CLI\n# shellcheck disable=SC2086\naws ec2 describe-subnets --subnet-ids $subnet_ids \\\n  --query 'Subnets[*].[SubnetId,AvailabilityZone,AvailableIpAddressCount]' \\\n  --output table\n\n# output should look like this\n#\n#----------------------------------------------------\n#|                  DescribeSubnets                 |\n#+---------------------------+--------------+-------+\n#|  subnet-067fa8ee8476abbd6 |  us-east-1a  |  8184 |\n#|  subnet-0056f7403b17d2b43 |  us-east-1b  |  8153 |\n#|  subnet-09586f8fb3addbc8c |  us-east-1a  |  8120 |\n#|  subnet-047f3d276a22c6bce |  us-east-1b  |  8184 |\n#+---------------------------+--------------+-------+\n"
  },
  {
    "path": "aws/aws_eks_cloudwatch_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-15 06:15:47 +0400 (Tue, 15 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables and fetches AWS EKS Master logs via CloudWatch\n\nIf CloudWatch logging hasn't been enabled already for this EKS cluser's master components, enabling it will take some time and the script will fail to get the log groups because they won't exist yet or you'll get an error like this:\n\n    An error occurred (ResourceNotFoundException) when calling the DescribeLogStreams operation: The specified log group does not exist.\n\nOr this:\n\n    An error occurred (ResourceInUseException) when calling the UpdateClusterConfig operation: Cannot LoggingUpdate because cluster EKS_CLUSTER_NAME currently has update be92dbc8-8406-3188-b1c8-afd40aa3c785 in progress\n\nWait a while and then run it again\n\n\nFor the EKS Node logs, use the adjacent script:\n\n    aws_eks_ssh_dump_logs.sh\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<EKS_CLUSTER_NAME> [<log_line_limit_per_service>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\neks_cluster=\"$1\"\nlimit=\"${2:-10000}\"\n\n#aws eks update-cluster-config --name \"$eks_cluster\" \\\n#        --logging '{\"clusterLogging\":[{\"types\":[\"api\",\"audit\",\"authenticator\",\"scheduler\",\"controllerManager\"],\"enabled\":true}]}'\n\ntimestamp \"Enabling CloudWatch logging on EKS cluster; $eks_cluster\"\naws eks update-cluster-config --name \"$eks_cluster\" \\\n        --logging '{\n            \"clusterLogging\":[\n                {\n                    \"types\": [\n                        \"api\",\n                        \"audit\",\n                        \"authenticator\",\n                        \"scheduler\",\n                        \"controllerManager\"\n                    ],\n                    \"enabled\":true\n                }\n            ]\n        }' || :\necho\n\ntimestamp \"Log Groups:\"\necho\nlog_groups=\"$(aws logs describe-log-groups --log-group-name-prefix \"/aws/eks/$eks_cluster/cluster\" |\n    jq_debug_pipe_dump |\n    jq -Mr '.logGroups[].logGroupName')\"\necho \"$log_groups\"\necho\n\ntstamp=\"$(date '+%F_%H%M%S')\"\n\nfor log_group in $log_groups; do\n\n    timestamp \"Getting log stream names for log_group: $log_group\"\n    echo\n    # usually this\n    #log_stream=\"$(aws logs describe-log-streams --log-group-name \"/aws/eks/$eks_cluster/cluster\")\"\n    log_streams=\"$(aws logs describe-log-streams --log-group-name \"$log_group\" |\n        jq_debug_pipe_dump |\n        jq -Mr '.logStreams[].logStreamName'\n    )\"\n\n    for log_stream in $log_streams; do\n        timestamp \"Getting logs for stream: $log_stream\"\n        logfile=\"${log_group//\\//_}.${log_stream//\\//_}.$tstamp.log\"\n        logfile=\"${logfile#_}\"\n        aws logs get-log-events --log-group-name \"$log_group\" --log-stream-name \"$log_stream\" --limit \"$limit\" |\n        jq_debug_pipe_dump |\n        #jq -Mr '.events[] | [.timestamp + \"  \" .message] | @tsv' > \"$logfile\"\n        jq -Mr '.events[].message' > \"$logfile\"\n        timestamp \"Log stream dumped to file: $logfile\"\n        echo\n    done\n    echo\n\ndone\n\ntimestamp \"Logs fetched, creating compressed tarball\"\necho\n\ntarball=\"$eks_cluster.master.cloudwatch.logs.$tstamp.tar.gz\"\n\ntar czvf \"$tarball\" \"aws_eks_${eks_cluster}_\"*\".$tstamp.log\"\n\necho\ntimestamp \"Generated tarball: $tarball\"\necho\n\ntimestamp \"Completed download of AWS EKS CloudWatch Logs for cluster: $eks_cluster\"\n"
  },
  {
    "path": "aws/aws_eks_cluster_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 12:14:14 +0700 (Wed, 08 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nIterates EKS clusters to list each AWS EKS cluster name and version in the current account\n\nOutput format:\n\n<cluster_name>    <version>\n\nCombine with:\n\n    aws_foreach_region.sh - to audit your cluster versions across regions\n    aws_foreach_profile.sh - to audit your cluster versions across accounts\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws eks list-clusters --output json |\njq -r '.clusters[]' |\nwhile read -r cluster; do\n    aws eks describe-cluster --name \"$cluster\" --output json |\n    jq -r '.cluster | [.name, .version] | @tsv'\ndone\n"
  },
  {
    "path": "aws/aws_eks_ssh_dump_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-15 04:58:14 +0400 (Tue, 15 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFetch system logs from AWS EKS Worker Nodes EC2 VMs (eg. for support debug requests by vendors)\n\nUses the adjacent script:\n\n    $srcdir/../kubernetes/kubernetes_nodes_ssh_dump_logs.sh\n\nRequires Kubectl to be installed and configured to be on the right AWS EKS cluster context as it uses this to determine the nodes\n\nUser - set your SSH_USER environment variable (defaults to 'ec2-user')\n\nSSH key - either set SSH_KEY to the EC2 pem key or add it to a local ssh-agent for passwordless authentication to work\n\nSee here for details:\n\n    $srcdir/../monitoring/ssh_dump_logs.sh --help\n\n\nFor the EKS Master logs, use the adjacent script:\n\n    aws_eks_cloudwatch_logs.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n#eks_node_groups=$(aws eks list-nodegroups --cluster-name \"$EKS_CLUSTER\" --query 'nodegroups' --output text)\n#\n#for node_group in $eks_node_groups; do\n#    instance_ids=$(aws eks describe-nodegroup --cluster-name \"$EKS_CLUSTER\" \\\n#                                              --nodegroup-name \"$node_group\" \\\n#                                              --query 'nodegroup.resources.autoScalingGroups[*].instances[*]' \\\n#                                              --output text)\n#    aws ec2 describe-instances --instance-ids \"$instance_ids\" \\\n#                               --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value]' \\\n#                               --output text\n#done\n\n# set the default EC2 user if nothing is set\nexport SSH_USER=\"${SSH_USER:-ec2-user}\"\n\n# simpler to just get it via kubectl than AWS Commands, reuse this script\n\"$srcdir/../kubernetes/kubernetes_nodes_ssh_dump_logs.sh\"\n"
  },
  {
    "path": "aws/aws_elasticache_serverless_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-16 12:03:31 +0700 (Mon, 16 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuick table of AWS ElastiCache useful fields:\n\nName, Engine, Status, Endpoint Address & Port, Description\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws elasticache describe-serverless-caches \\\n    --query 'ServerlessCaches[*].{\n        \"   Name\":ServerlessCacheName,\n        \"  Engine\":Engine,\n        \"  Status\":Status,\n        \" Endpoint Address\":Endpoint.Address,\n        \" Endpoint Port\":Endpoint.Port,\n        Description:Description\n    }' \\\n    --output table\n"
  },
  {
    "path": "aws/aws_emr_clusters_last_steps.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-02 17:56:07 +0700 (Thu, 02 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the last N steps executed on each EMR cluster and their EndTime to find idle clusters that should be removed\n\nAlso checks CloudWatch for number of steps running within the last few months to catch directly submitted jobs such as\nSpark, Hive, Glue or Athena which won't show up in the native steps list\n\nYou can also check this via the Monitoring tab graphs for the cluster by setting the date range further back\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<last_N_steps>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nnum_steps=\"${1:-5}\"\n\nif ! is_int \"$num_steps\"; then\n    usage \"Number of steps argument must be an integer\"\nfi\n\nif [ \"$num_steps\" = 0 ]; then\n    usage \"Number of steps cannot be zero\"\nfi\n\ntimestamp \"Fetching list of EMR clusters\"\ncluster_ids=$(aws emr list-clusters --query 'Clusters[*].Id' --output text)\n\nif is_blank \"$cluster_ids\"; then\n    die \"No EMR clusters found\"\nfi\n\ntimestamp \"Fetching the last $num_steps steps for each EMR cluster...\"\necho\n\nstart_date=$(date -u --date=\"1 year ago\" +\"%Y-%m-%dT%H:%M:%SZ\")\nend_date=$(date -u +\"%Y-%m-%dT%H:%M:%SZ\")\n\nfor cluster_id in $cluster_ids; do\n    aws emr describe-cluster \\\n        --cluster-id \"$cluster_id\" \\\n        --query 'Cluster.{\n            \"ID\": Id,\n            \"Name\": Name,\n            \"Status\": Status.State\n        }' \\\n        --output table || {\n            warn \"Failed to describe cluster $cluster_id, skipping...\"\n            continue\n    }\n\n    # don't want backtick shell expansion inside AWS --query\n    # shellcheck disable=SC2016\n    steps=\"$(\n        aws emr list-steps --cluster-id \"$cluster_id\" \\\n            --query 'Steps[?Status.Timeline.EndDateTime != `null`]|[].{Name:Name,EndTime:Status.Timeline.EndDateTime}' \\\n            --output json |\n        jq -r \"\n            . |\n            sort_by(.EndTime) |\n            reverse |\n            .[:$num_steps] |\n            .[] |\n            [.Name, .EndTime] |\n            @tsv\n        \"\n    )\"\n\n\n    if is_blank \"$steps\"; then\n        timestamp \"No steps found for this cluster\"\n    else\n        timestamp \"Last 5 steps for cluster:\"\n        echo >&2\n        printf \"    %-50s %s\\n\" \"Step Name\" \"End Time\"\n        echo \"    -------------------------------------------------- --------------------------------\"\n        while IFS=$'\\t' read -r step_name end_time; do\n            printf \"    %-50s %s\\n\" \"$step_name\" \"$end_time\"\n        done <<< \"$steps\"\n    fi\n    echo >&2\n\n    # Get list of metrics to query like this:\n    #\n    #   aws cloudwatch list-metrics --namespace \"AWS/ElasticMapReduce\" --dimensions Name=JobFlowId,Value=\"$cluster_id\"\n    #\n    timestamp \"Apps Running within last $num_steps x 30 day periods\"\n    aws cloudwatch get-metric-statistics \\\n        --namespace \"AWS/ElasticMapReduce\" \\\n        --metric-name \"AppsRunning\" \\\n        --dimensions Name=JobFlowId,Value=\"$cluster_id\" \\\n        --start-time \"$start_date\" \\\n        --end-time \"$end_date\" \\\n        --period $((86400 * 30)) \\\n        --statistics Sum \\\n        --query 'Datapoints[*].[Timestamp,Sum]' \\\n        --output json |\n    jq -r \"sort_by(.[0]) | reverse | .[:$num_steps][] | @tsv\"\n\n    echo >&2\n    echo >&2\ndone\n"
  },
  {
    "path": "aws/aws_foreach_profile.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo 'AWS_PROFILE=$AWS_PROFILE, {profile\\}={profile}'\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-28 16:28:01 +0100 (Wed, 28 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each AWS named profile configured for the local AWS CLIv2\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the profile names and exit after the first iteration\n\nRequires AWS CLIv2 to be installed and configured (older AWS CLIv1 that is installed via pip doesn't support this)\n\nAll arguments become the command template\n\nAWS_PROFILE is set in each iteration and {profile} is replaced in any commands\n\n\neg.\n    ${0##*/} echo 'AWS_PROFILE=\\$AWS_PROFILE, {profile\\\\}={profile}'\n\nTo set up your Kubernetes access to all clusters in all locally configured accounts using adjacent aws_kube_creds.sh script\n\n    ${0##*/} aws_kube_creds.sh\n\n\nBeware that credentials left over in ~/.aws/credentials will be included as profiles and iterated on, you must\ncomment them out in ~/.aws/credentials as well as in ~/.aws/config or \\$AWS_CONFIG_FILE\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nprofiles=\"$(aws configure list-profiles --output text)\"\n\ntotal_profiles=\"$(grep -c . <<< \"$profiles\")\"\n\ni=0\n\nwhile read -r profile; do\n    (( i += 1 ))\n    echo \"# ============================================================================ #\" >&2\n    echo \"# ($i/$total_profiles) AWS profile = $profile\" >&2\n    echo \"# ============================================================================ #\" >&2\n    export AWS_PROFILE=\"$profile\"\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{profile\\}/$profile}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\n    echo >&2\ndone <<< \"$profiles\"\n"
  },
  {
    "path": "aws/aws_foreach_region.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 'echo region is {region}'\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-19 14:59:58 +0100 (Mon, 19 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each AWS region enabled for the current account\n\nYou may want to use this to run an AWS CLI command against all regions to find resources or perform scripting across regions\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the region names and exit after the first iteration\n\nRequires AWS CLI to be installed and configured and 'aws' to be in the \\$PATH\n\nAll arguments become the command template\n\nThe following command template tokens are replaced in each iteration:\n\nAWS Region:     {region}\n\nIf \\$AWS_ALL_REGIONS is defined and not empty then iterates all regions, even those not enabled for the current account\n\n\nExamples:\n\n    ${0##*/} 'echo AWS region is {region}'\n\n    AWS_ALL_REGIONS=1 ${0##*/} 'echo AWS region is {region}'\n\nFind EC2 instances across regions:\n\n    ${0##*/} aws ec2 describe-instances\n\n    ${0##*/} 'aws ec2 describe-instances | jq -r \\\".Reservations | length\\\"'\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\nregions=\"$(\n    # --all-regions iterates all regions whether or not they are enabled for the current account\n    if [ -n \"${AWS_ALL_REGIONS:-}\" ]; then\n        aws ec2 describe-regions --all-regions\n    else\n        aws ec2 describe-regions\n    fi |\n    jq -r '.Regions[] | .RegionName'\n)\"\n\ntotal_regions=\"$(grep -c . <<< \"$regions\")\"\n\ni=0\n\nwhile read -r region; do\n    (( i += 1 ))\n    echo \"# ============================================================================ #\" >&2\n    echo \"# ($i/$total_regions) AWS region = $region\" >&2\n    echo \"# ============================================================================ #\" >&2\n    export AWS_DEFAULT_REGION=\"$region\"\n    export AWS_REGION=\"$region\"\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{region\\}/$region}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\n    echo >&2\ndone <<< \"$regions\"\n"
  },
  {
    "path": "aws/aws_iam_generate_credentials_report_wait.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 10:02:31 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_getting-report.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates an AWS IAM credentials report and waits for it to finish\n\nCalled from adjacent scripts as a dependency so that they can then pull specific information from the report\n\nRequires iam:GenerateCredentialReport on resource: *\n\n\nSee Also:\n\n    more AWS tools in the DevOps Python Tools repo and The Advanced Nagios Plugins Collection:\n\n    - https://github.com/HariSekhon/DevOps-Python-tools\n    - https://github.com/HariSekhon/Nagios-Plugins\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\nSECONDS=0\nMAX_SECONDS=60\n\n# you must run this to generate the report before you can get this info, seems to be ready a couple secs later\nwhile true; do\n    if aws iam generate-credential-report --output json |\n    jq -r .State |\n    tee /dev/stderr |\n    grep -q COMPLETE; then\n        break\n    fi\n    if [ $SECONDS -gt $MAX_SECONDS ]; then\n        echo \"AWS IAM Credentials report failed to return complete within $MAX_SECONDS\"\n        exit 1\n    fi\ndone\n"
  },
  {
    "path": "aws/aws_iam_harden_password_policy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-07 11:57:18 +0000 (Tue, 07 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nStrengthens password policy according to CIS Foundations Benchmark recommendations\n\nView password policy using adjacent script aws_password_policy.sh (automatically called at start and end of this script)\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\necho\necho \"Existing Password Policy:\"\necho\n\"$srcdir/aws_iam_password_policy.sh\"\necho\necho\necho \"Setting Hardened Password Policy:\"\necho\nset -x\naws iam update-account-password-policy --require-uppercase-characters\naws iam update-account-password-policy --require-lowercase-characters\naws iam update-account-password-policy --require-symbols\naws iam update-account-password-policy --require-numbers\naws iam update-account-password-policy --minimum-password-length 14\naws iam update-account-password-policy --password-reuse-prevention 24\naws iam update-account-password-policy --max-password-age 90\nset +x\necho\necho\necho \"New Password Policy:\"\necho\n\"$srcdir/aws_iam_password_policy.sh\"\n"
  },
  {
    "path": "aws/aws_iam_password_policy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-07 11:57:18 +0000 (Tue, 07 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints password policy in 'key = value' pairs for easy viewing / grepping\n\nSee Also:\n\n    aws_iam_harden_password_policy.sh - sets a hardeded password policy along CIS Foundations Benchmark recommendations\n                                        that script calls this one before and after changing the password policy\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\naws iam get-account-password-policy --output json |\njq -r '.PasswordPolicy | to_entries | map(.key + \" = \" + (.value | tostring)) | .[]'\n"
  },
  {
    "path": "aws/aws_iam_policies_attached_to_users.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-07 11:57:18 +0000 (Tue, 07 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds and prints all policies directly attached to users instead of groups\n\nPolicies directly attached to users is against best practice as it is harder to maintain\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n# prefix user column only if there is any output\nprefix_user(){\n    sed \"s/^\\\\(.\\\\)/${user}   &/\"\n    #while read line; do\n    #    if [ -n \"$line\" ]; then\n    #        echo \"$user $line\"\n    #    fi\n    #done\n}\n\nexport AWS_DEFAULT_OUTPUT=json\n\necho \"output will be formatted in to columns at end\" >&2\necho \"getting user list\" >&2\naws iam list-users |\njq -r '.Users[].UserName' |\nwhile read -r user; do\n    echo \"querying user $user\" >&2\n    aws iam list-attached-user-policies --user-name \"$user\" | jq -r '.AttachedPolicies[] | [.PolicyName, .PolicyArn] | @tsv' | prefix_user\n    aws iam list-user-policies --user-name \"$user\" | jq -r '.PolicyNames[] | [.PolicyName, .PolicyArn] | @tsv' | prefix_user\ndone |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_policies_granting_full_access.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-17 16:17:39 +0000 (Fri, 17 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds policies granting full access in JSON format\n\nTakes a while to run (eg. ~18 mins for ~700 policies)\n\nIf stderr is to terminal, prints progress counter in the form of num / total\n\nRecommend to redirect stdout to a file ( > file.txt ) and just watch progress counter on stderr in terminal\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\necho \"Getting policy list\" >&2\npolicies=\"$(\n    # get json to allow to filter later\n    aws iam list-policies |\n    jq -r '.Policies[] | [.Arn, .DefaultVersionId] | @tsv' # | head -n 10 || :\n)\"\n\nnum_policies=\"$(wc -l <<< \"$policies\")\"\nnum_policies=\"${num_policies//[[:space:]]/}\"\n\necho \"Iterating over all $num_policies policies to find policies granting full access (this may take a while)\" >&2\n#{\n#echo '['\n#i=1\nwhile read -r arn version; do\n    echo \"checking $arn version $version\" >&2\n    policy=\"$(aws iam get-policy-version --policy-arn \"$arn\" --version-id \"$version\")\"\n    if {\n        # select any policies where Action is a string or an array containing * from granting all\n        # XXX: if you want to find policies granting full access to a service like S3 just replace '*' with 's3:*'\n        jq -r '.PolicyVersion | select(.Document.Statement[].Action == \"*\")' <<< \"$policy\" 2>/dev/null || :\n        jq -r '.PolicyVersion | select(.Document.Statement[].Action.[] | index(\"*\"))' <<< \"$policy\" 2>/dev/null || :\n       } | grep -q .; then\n        echo \"WARNING: $arn GRANTS FULL ACCESS:\"\n        echo \"$policy\"\n        echo\n    fi\n    # simple but we want progress numbers\n    #echo -n '.' >&2\n    # only print counter if stderr is to terminal\n    #if [ -t 2 ]; then\n    #    printf '\\r%s/%s' \"$i\" \"$num_policies\" >&2\n    #fi\n    #if [ $i -lt \"$num_policies\" ]; then\n    #    echo ','\n    #fi\n    #((i+=1))\ndone <<< \"$policies\"\n#printf '\\n' >&2\n#echo ']'\n#} # |\n# doesn't give full document\n#jq -r '.[].PolicyVersion.Document.Statement[] | select(.Action | index(\"*\"))'\n# gives full document, but not name and doesn't work when Action is string instead of array - doing test in loop now to output arn and handle both cases\n#jq -r '.[].PolicyVersion | select(.Document.Statement[].Action | index(\"*\"))'\n"
  },
  {
    "path": "aws/aws_iam_policies_unattached.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-07 11:57:18 +0000 (Tue, 07 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all unattached policies\n\nThese unattached should probably be cleaned up (removed)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws iam list-policies |\n# get json to allow to filter later\n#jq -r '.Policies[] | select(.IsAttachable==true) | select (.AttachmentCount==0)'\njq -r '.Policies[] | select(.IsAttachable==true) | select (.AttachmentCount==0) | [.PolicyId, .PolicyName, .CreateDate, .UpdateDate] | @tsv' |\nsort -k2 |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_policy_attachments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-15 18:41:42 +0100 (Wed, 15 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all users, groups and roles where a given IAM policy is attached, so that you can remove all these references in your Terraform code and avoid this error:\n\nPlan: 1 to add, 1 to change, 1 to destroy.\nmodule.mymodule.aws_iam_policy.mypolicy: Destroying... [id=arn:aws:iam::***:policy/mypolicy]\n╷\n│Error: error deleting IAM policy arn:aws:iam::***:policy/mypolicy: DeleteConflict: Cannot delete a policy attached to entities.\n│    status code: 409, request id: 1f9ca3ee-48fb-4e5e-9e58-5c266e29e9be\n\n\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npolicy=\"$1\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws_account_id=\"$(aws_account_id)\"\n\npolicy_arn=\"arn:aws:iam::$aws_account_id:policy/$policy\"\n\nfind_entities(){\n    local entity_type=\"$1\"\n    aws iam list-entities-for-policy --policy-arn \"$policy_arn\" |\n    jq_debug_pipe_dump |\n    jq -r \".Policy${entity_type}s[].${entity_type}Name\" |\n    while read -r entity; do\n        printf '%s\\t%s\\n' \"$entity_type\" \"$entity\"\n    done\n}\n\nfind_entities User\nfind_entities Group\nfind_entities Role\n"
  },
  {
    "path": "aws/aws_iam_policy_delete.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-15 18:41:42 +0100 (Wed, 15 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes an IAM policy by first handling all prerequisite steps of deleting all prior versions and detaching all users, groups and roles\n\nBecause doing a straight delete will fail with an error like this:\n\n    An error occurred (DeleteConflict) when calling the DeletePolicy operation: This policy has more than one version. Before you delete a policy, you must delete the policy's versions. The default version is deleted with the policy.\n\nor this:\n\n    An error occurred (DeleteConflict) when calling the DeletePolicy operation: Cannot delete a policy attached to entities.\n\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npolicy=\"$1\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws_account_id=\"$(aws_account_id)\"\n\npolicy_arn=\"arn:aws:iam::$aws_account_id:policy/$policy\"\n\ndetach_entity(){\n    local entity_type=\"$1\"\n    local entity=\"$2\"\n    entity_type=\"$(tr '[:upper:]' '[:lower:]' <<< \"$entity_type\")\"\n    timestamp \"Detaching $entity_type $entity\"\n    aws iam \"detach-${entity_type}-policy\" \"--${entity_type}-name\" \"$entity\" --policy-arn \"$policy_arn\"\n}\n\ndetach_entities(){\n    local entity_type=\"$1\"\n    aws iam list-entities-for-policy --policy-arn \"$policy_arn\" |\n    jq -r \".Policy${entity_type}s[].${entity_type}Name\" |\n    while read -r entity; do\n        detach_entity \"$entity_type\" \"$entity\"\n    done\n}\n\nolder_policy_versions=\"$(aws iam list-policy-versions --policy-arn \"$policy_arn\" |\n                         jq -r '.Versions[] | select(.IsDefaultVersion == false) | .VersionId')\"\n\nfor policy_version_id in $older_policy_versions; do\n    timestamp \"Deleting policy '$policy' version '$policy_version_id'\"\n    aws iam delete-policy-version --policy-arn \"$policy_arn\" --version-id \"$policy_version_id\"\ndone\n\ndetach_entities User\ndetach_entities Group\ndetach_entities Role\n\ntimestamp \"Deleting policy '$policy'\"\naws iam delete-policy --policy-arn \"$policy_arn\"\ntimestamp \"Done\"\n"
  },
  {
    "path": "aws/aws_iam_rename_user_accounts_domains.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-02-06 16:45:18 +0000 (Mon, 06 Feb 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRenames all IAM user accounts from one domain to another\n\nUseful after company mergers / migrations where you go from firstname.lastname@company1.com to firstname.lastname@company2.com\n(even if using mostly AWS SSO you may still need this for your root management account's IAM users)\n\nGroup memberships and permissions are retained\n\nExpects that the user@ prefix portion (eg. first.last@) stays the same before and after company migration\n\nRequires AWS CLI and jq to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<old_domain> <new_domain>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nold_domain=\"$1\"\nnew_domain=\"$2\"\n\naws iam list-users |\njq -r '.Users[].UserName' |\n{ grep \"@$old_domain$\" || : ; } |\nwhile read -r username; do\n    new_username=\"${username/$old_domain/$new_domain}\"\n    timestamp \"Renaming $username -> $new_username\"\n    aws iam update-user --user-name \"$username\" --new-user-name \"$new_username\"\n    echo\ndone\n"
  },
  {
    "path": "aws/aws_iam_replace_access_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-13 22:40:11 +0000 (Mon, 13 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a new IAM access key for the currently authenticated user, outputting the keys as export commands\n\nIf 2 keys already exist, decide which key to replace using these heuristics in this order of preference:\n\n- inactive key\n- unused key\n- oldest usage (ie. not the key we're currently using to authenticate)\n\nAlternatively you can specify an access key id to delete as an argument, but if there is only 1 access key it won't delete it for safety (will output warning)\n\nIf there is only 1 access key, a 2nd key is created but no key is deleted for safety to prevent you cutting yourself off as standard usage is to rotate through 2 access keys\n\nIf the first argument given starts with a dash it is inferred to be an AWS CLI option instead of an access key ID to replace and the above heuristic is used to figure out which key to replace\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_access_key_id> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\naccess_key_id_to_delete=\"\"\nif ! [[ \"${1:-}\" =~ ^- ]]; then\n    access_key_id_to_delete=\"${1:-}\"\n    if [ $# -gt 0 ]; then\n        shift || :\n    fi\nfi\n\nexport AWS_DEFAULT_OUTPUT=json\n\nkeys=\"$(aws iam list-access-keys \"$@\")\"\n\nnum_keys=\"$(jq -r '.AccessKeyMetadata | length' <<< \"$keys\")\"\nif ! [[ \"$num_keys\" =~ ^[[:digit:]]+$ ]]; then\n    die \"Failed to determine number of AWS access keys\"\nfi\n\n# Limited to 2 access keys\nif [ \"$num_keys\" -gt 2 ]; then\n    die \"More than 2 access keys found - code error or AWS has changed the limitations, which affects this logic and requires a code update\"\nelif [ \"$num_keys\" -eq 2 ]; then\n    if [ -z \"$access_key_id_to_delete\" ]; then\n        # figure out which one to delete\n        access_key_ids=\"$(jq -r '.AccessKeyMetadata[].AccessKeyId' <<< \"$keys\")\"\n        last_key_last_used_date=\"\"\n        for access_key_id in $access_key_ids; do\n            key_status=\"$(jq -r \".AccessKeyMetadata[] | select(.AccessKeyId == \\\"$access_key_id\\\") | .Status\" <<< \"$keys\")\"\n            if [ \"$key_status\" = \"Inactive\" ]; then\n                access_key_id_to_delete=\"$access_key_id\"\n                break\n            fi\n            # not passing \"$@\" because if --user-name is specified it is only relevant to other commands\n            last_used_date=\"$(aws iam get-access-key-last-used --access-key-id \"$access_key_id\" |\n                              jq -r '.AccessKeyLastUsed.LastUsedDate')\"\n            if [ \"$last_used_date\" = null ]; then\n                access_key_id_to_delete=\"$access_key_id\"\n                break\n            fi\n            # XXX: Ugly, improve logic here\n            if [ -n \"$last_key_last_used_date\" ]; then\n                # convert both last used to epoch, smaller number is older\n                if [ \"$(date '+%s' --date \"$last_used_date\")\" -le \"$(date '+%s' --date \"$last_key_last_used_date\")\" ]; then\n                    access_key_id_to_delete=\"$access_key_id\"\n                else\n                    # first key must be older\n                    access_key_id_to_delete=\"$(jq -r '.AccessKeyMetadata[0].AccessKeyId' <<< \"$keys\")\"\n                fi\n                break\n            fi\n            last_key_last_used_date=\"$last_used_date\"\n        done\n    fi\n    if [ -z \"$access_key_id_to_delete\" ]; then\n        die \"Couldn't determine which access key to delete, aborting...\"\n    fi\n    timestamp \"Deleting AWS access key '$access_key_id_to_delete'\"\n    #aws iam update-access-key --access-key-id \"$access_key_id_to_delete\" --status Inactive\n    aws iam delete-access-key --access-key-id \"$access_key_id_to_delete\" \"$@\"\n    echo >&2\nfi\n\naws iam create-access-key \"$@\" |\njq -r '\n    .AccessKey |\n    [ \"export AWS_ACCESS_KEY_ID=\" + .AccessKeyId, \"export AWS_SECRET_ACCESS_KEY=\" + .SecretAccessKey ] |\n    join(\"\\n\")\n'\n"
  },
  {
    "path": "aws/aws_iam_users.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-05 17:02:15 +0000 (Thu, 05 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# awless list users\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS users\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n#aws iam list-users | jq -r '.Users[].UserName'\naws iam list-users --query 'Users[*].UserName' --output text | tr '[:space:]' '\\n'\n"
  },
  {
    "path": "aws/aws_iam_users_access_key_age.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-05 17:02:15 +0000 (Thu, 05 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints users access key status and age\n\nSee Also:\n\n    aws_iam_users_access_key_age_report.sh - much quicker version for lots of users\n\n\n    aws_users_access_key_age.py - in DevOps Python Tools which is able to filter by age and status\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\n    awless list accesskeys --format tsv | grep 'years[[:space:]]*$'\n\n\nAWS Config rule compliance:\n\n    https://<region>.console.aws.amazon.com/config/home?region=<region>&v2=true#/rules/details?configRuleName=access-keys-rotated\n\neg.\n\n    https://eu-west-1.console.aws.amazon.com/config/home?region=eu-west-1&v2=true#/rules/details?configRuleName=access-keys-rotated\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\necho \"output will be formatted in to columns at end\" >&2\necho \"getting user list\" >&2\naws iam list-users |\njq -r '.Users[].UserName' |\nwhile read -r username; do\n    echo \"querying user $username\" >&2\n    aws iam list-access-keys --user-name \"$username\" |\n    jq -r '.AccessKeyMetadata[] | [.UserName, .Status, .CreateDate, .AccessKeyId] | @tsv'\ndone |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_users_access_key_age_report.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-05 17:02:15 +0000 (Thu, 05 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints users access key status and age using a credential report (faster for many users)\n\nCSV Output format:\n\nuser,access_key_1_active,access_key_1_last_rotated,access_key_2_active,access_key_2_last_rotated\n\n\nSee Also:\n\n    aws_iam_users_access_key_age.sh\n\n\n    aws_users_access_key_age.py - in DevOps Python Tools which is able to filter by age and status\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\n    awless list accesskeys --format tsv | grep 'years[[:space:]]*$'\n\n\nAWS Config rule compliance:\n\n    https://<region>.console.aws.amazon.com/config/home?region=<region>&v2=true#/rules/details?configRuleName=access-keys-rotated\n\neg.\n\n    https://eu-west-1.console.aws.amazon.com/config/home?region=eu-west-1&v2=true#/rules/details?configRuleName=access-keys-rotated\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/aws_iam_generate_credentials_report_wait.sh\" >&2\n\n# use --decode not -d / -D which varies between Linux and Mac\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    base64_decode=\"base64 -D\"\n#else\n#    base64_decode=\"base64 -d\"\n#fi\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws iam get-credential-report --query 'Content' --output text |\nbase64 --decode |\ncut -d, -f1,9,10,14,15\n"
  },
  {
    "path": "aws/aws_iam_users_access_key_last_used.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 11:21:30 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints users access keys and their last used date\n\nOutput format, tab separated:\n\n<user>    <access_key>   <last_used_date>  <region>\n\n\nSee Also:\n\n    aws_iam_users_access_key_last_used.sh - much quicker version for lots of users\n\n\nSee similar tools in DevOps Python Tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\nif [ $# -gt 0 ]; then\n    users=\"$*\"\nelse\n    echo \"output will be formatted in to columns at end\" >&2\n    echo \"getting user list\" >&2\n    users=\"$(aws iam list-users | jq -r '.Users[].UserName')\"\nfi\n\nwhile read -r username; do\n    echo \"querying user $username\" >&2\n    aws iam list-access-keys --user-name \"$username\" |\n    jq -r '.AccessKeyMetadata[].AccessKeyId' |\n    while read -r access_key; do\n        aws iam get-access-key-last-used --access-key-id \"$access_key\" |\n        jq -r '[.UserName, .AccessKeyLastUsed.LastUsedDate, .AccessKeyLastUsed.Region] | @tsv' |\n#        while read -r user last_used region; do\n#            # if there is no last_used field, 3rd field will be taken from region\n#            #if [ -z \"$region\" ]; then\n#            #    region=\"$last_used\"\n#            #    last_used=\"blank\"\n#            #fi\n#            if [ -z \"$region\" ] && [ \"$last_used\" = \"N/A\" ]; then\n#                region=\"N/A\"\n#            fi\n#            printf '%s\\t%s\\t%s\\t%s\\n' \"$user\" \"$access_key\" \"${last_used:-blank}\" \"$region\"\n#        done\n        awk '{if(NF==2){$3=\"N/A\"}; print $1\"\\t'\"$access_key\"'\\t\"$2\"\\t\"$3}'\n    done\ndone <<< \"$users\" |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_users_access_key_last_used_report.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 11:21:30 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints users access keys and their last used date using a credentials report (faster for many users)\n\nCSV Output format:\n\nuser,access_key_1_active,access_key_1_last_used_date,access_key_2_active,access_key_2_last_used_date\n\n\nSee similar tools in DevOps Python Tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/aws_iam_generate_credentials_report_wait.sh\" >&2\n\n# use --decode not -d / -D which varies between Linux and Mac\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    base64_decode=\"base64 -D\"\n#else\n#    base64_decode=\"base64 -d\"\n#fi\n\naws iam get-credential-report --query 'Content' --output text |\nbase64 --decode |\ncut -d, -f1,9,11,14,16\n"
  },
  {
    "path": "aws/aws_iam_users_last_used_report.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 10:02:31 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS IAM users last used dates for password and access keys\n\nOutput format is CSV with the following headers\n\nuser,password_last_used,access_key_1_last_used_date,access_key_2_last_used_date\n\n\nAdd this to your command pipeline\n\n    | grep -B1 '<root_account>'\n\nto check your root account isn't being used\n\n\nSee similar tools in the DevOps Python Tools repo and The Advanced Nagios Plugins Collection:\n\n    - https://github.com/HariSekhon/DevOps-Python-tools\n    - https://github.com/HariSekhon/Nagios-Plugins\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/aws_iam_generate_credentials_report_wait.sh\" >&2\n\n# use --decode not -d / -D which varies between Linux and Mac\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    base64_decode=\"base64 -D\"\n#else\n#    base64_decode=\"base64 -d\"\n#fi\n\n# not documented in 'aws iam get-credential-report help'\naws iam get-credential-report --query 'Content' --output text |\nbase64 --decode |\ncut -d, -f1,5,11,16\n"
  },
  {
    "path": "aws/aws_iam_users_mfa_active_report.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 10:02:31 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS IAM users password enabled and MFA enabled status\n\nOutput format is CSV with the following headers\n\nuser,password_enabled,mfa_active\n\n\nAdd this to your command pipeline\n\n    | grep -B1 '<root_account>'\n\nto check your root account isn't being used\n\n\nUses the adjacent script aws_iam_generate_credentials_report_wait.sh\n\n\nSee similar tools in the DevOps Python Tools repo and The Advanced Nagios Plugins Collection:\n\n    - https://github.com/HariSekhon/DevOps-Python-tools\n    - https://github.com/HariSekhon/Nagios-Plugins\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/aws_iam_generate_credentials_report_wait.sh\" >&2\n\n# use --decode not -d / -D which varies between Linux and Mac\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    base64_decode=\"base64 -D\"\n#else\n#    base64_decode=\"base64 -d\"\n#fi\n\n# not documented in 'aws iam get-credential-report help'\naws iam get-credential-report --query 'Content' --output text |\nbase64 --decode |\ncut -d, -f1,4,8\n"
  },
  {
    "path": "aws/aws_iam_users_mfa_serials.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-10 17:33:07 +0000 (Fri, 10 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints users MFAs serial numbers to differentiate Virtual vs Hardware MFAs\n\nVirtual MFAs have a SerialNumber in the format:\n\narn:aws:iam::<account_id>:mfa/<mfa>\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws iam list-virtual-mfa-devices |\njq -r '.VirtualMFADevices[] | [.User.UserName, .SerialNumber] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_users_pw_last_used.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-05 17:02:15 +0000 (Thu, 05 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS IAM users and their password last used date\n\nSee Also:\n\n    - check_aws_users_password_last_used.py in the Advanced Nagios Plugins collection\n\n        https://github.com/HariSekhon/Nagios-Plugins\n\n    awless list users\n\n    awless list users --format tsv | awk '{if(\\$4 == \\\"months\\\" || \\$4 == \\\"years\\\") print}'\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws iam list-users |\njq -r '.Users[] | [.UserName, .PasswordLastUsed] | @tsv' |\n#while read -r username password_last_used; do\n#    printf '%s\\t%s\\n' \"$username\" \"${password_last_used:-N/A}\"\n#done |\nawk '{if(NF==1){$2=\"N/A\"}print}' |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_iam_users_without_mfa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 10:02:31 +0000 (Thu, 19 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS IAM users with passwords enabled but without MFA enabled\n\nOutputs a list of users, one per line.\n\n\nUses the adjacent script aws_iam_users_mfa_active_report.sh\n\n\nSee similar tools in the DevOps Python Tools repo and The Advanced Nagios Plugins Collection:\n\n    - https://github.com/HariSekhon/DevOps-Python-tools\n    - https://github.com/HariSekhon/Nagios-Plugins\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/aws_iam_users_mfa_active_report.sh\" |\nawk -F, '$2 !~ \"false\" {print}' |\nsed '/,true$/d' |\ntail -n +2 |\nawk -F, '{print $1}'\n"
  },
  {
    "path": "aws/aws_info.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC1091\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-22 14:20:14 +0400 (Fri, 22 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS deployed resources in the current or specified AWS account profile\n\nWritten to be combined with aws_foreach_profile.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin aws\n\nif [ $# -gt 0 ]; then\n    aws_profile=\"$1\"\n    shift || :\n    export AWS_PROFILE=\"$aws_profile\"\nfi\n\n\ncat <<EOF\n# ============================================================================ #\n#                                 A W S   C L I\n# ============================================================================ #\n\nEOF\n\naws --version\necho\necho\n\n## ============================================================================ #\n. \"$srcdir/aws_info_ec2.sh\"\necho\necho\n\n## ============================================================================ #\n#. \"$srcdir/gcp_info_auth_config.sh\"\n#echo\n#echo\n\n## ============================================================================ #\n#. \"$srcdir/gcp_info_projects.sh\"\n#echo\n#echo\n\n## ============================================================================ #\n#. \"$srcdir/gcp_info_services.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_accounts_secrets.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_compute.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_storage.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_networking.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_bigdata.sh\"\n#echo\n#echo\n#\n## ============================================================================ #\n#. \"$srcdir/gcp_info_tools.sh\"\n#echo\n#echo\n\n# Finished - silencing aws_account_name because you may not have the permissions to AWS Org to retrieve it\ncat <<EOF\n# ============================================================================ #\n# Finished listing resources for AWS account $(aws_account_id) $(aws_account_name 2>/dev/null)\n# ============================================================================ #\nEOF\n"
  },
  {
    "path": "aws/aws_info_all_profiles.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-22 14:22:54 +0400 (Fri, 22 Nov 2024)\n#  long overdue port of the adjacent GCP info scripts from years prior\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGathers AWS Info across all projects\n\nCombines aws_foreach_profile.sh and aws_info.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws_foreach_profile.sh \"'$srcdir/aws_info.sh' '{profile}'\"\n\necho >&2\ntimestamp \"Script Completed Successfully: ${0##*/}\"\n"
  },
  {
    "path": "aws/aws_info_ec2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-22 14:20:14 +0400 (Fri, 22 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS EC2 Instances resources deployed in the current AWS account\n\nWritten to be combined with aws_foreach_project.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin aws\n\nif [ $# -gt 0 ]; then\n    aws_profile=\"$1\"\n    shift || :\n    export AWS_PROFILE=\"$aws_profile\"\nfi\n\n\n# AWS Virtual Machines\ncat >&2 <<EOF\n# ============================================================================ #\n#                   E C 2   V i r t u a l   M a c h i n e s\n# ============================================================================ #\n\nEOF\n\ndeclare -A ami_map\n\ntimestamp \"Getting all unique AMI IDs in use\"\nami_ids=\"$(\"$srcdir/aws_ec2_ami_ids.sh\")\"\n\nwhile read -r ami_id; do\n    [[ -z \"$ami_id\" ]] && continue\n\n    timestamp \"Resolving AMI ID: $ami_id\"\n    ami_name=\"$(\n        aws ec2 describe-images \\\n            --image-ids \"$ami_id\" \\\n            --query 'Images[0].Name' \\\n            --output text\n    )\"\n    if [[ \"$ami_name\" == \"None\" || -z \"$ami_name\" ]]; then\n        timestamp \"WARNING: AMI ID '$ami_id' failed to resolve to AMI Name\"\n    else\n        timestamp \"AMI ID '$ami_id' => '$ami_name'\"\n        ami_map[\"$ami_id\"]=\"$ami_name\"\n    fi\ndone <<< \"$ami_ids\"\n\nsed_script=''\nfor ami_id in \"${!ami_map[@]}\"; do\n    ami_name=\"${ami_map[$ami_id]}\"\n    sed_script+=\"\n    s|[[:space:]]${ami_id}[[:space:]]|$ami_name|g;\"\ndone\n\ntimestamp \"Getting list of EC2 instances with translated AMI IDs to Names\"\ntimestamp \"Putting AMI last as replacing the AMI IDs with Names will mess up the character width alignment of the last column\"\necho >&2\n# shellcheck disable=SC2016\naws ec2 describe-instances \\\n    --query 'Reservations[*].Instances[*].{\n                \"  Name\": Tags[?Key==`Name`].Value | [0],\n                \"  ID\": InstanceId,\n                \"  PrivateIP\": PrivateIpAddress,\n                \"  State\": State.Name,\n                \" InstanceType\": InstanceType,\n                \"AMI\": ImageId,\n                \" Architecture\": Architecture,\n                \" Platform\": PlatformDetails,\n                \" PublicDNS\": publicDnsName,\n                \" PrivateDNS\": PrivateDnsName\n            }' \\\n    --output table |\nsed \"$sed_script\"\n"
  },
  {
    "path": "aws/aws_info_ec2_all_profiles_csv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-22 14:22:54 +0400 (Fri, 22 Nov 2024)\n#  long overdue port of the adjacent GCP info scripts from years prior\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\nscript_basename=\"${0##*/}\"\nscript_basename=\"${script_basename%%.sh}\"\nscript_basename=\"${script_basename%%_csv}\"\n\nlog_timestamp=\"$(date '+%F_%H.%M.%S')\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS EC2 Instances in quoted CSV format across all configured AWS profiles for their configured region\n\nCombines aws_foreach_profile.sh and aws_info_csv.sh\n\nOutputs to both stdout and a file called $script_basename-YYYY-MM-DD_HH.MM.SS.csv\n\nSo that you can diff subsequent runs to see the difference between EC2 VMs that come and go due to AutoScaling Groups\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncsv=\"$script_basename-$log_timestamp.csv\"\ncsv_sorted=\"$script_basename-$log_timestamp-sorted.csv\"\n\nSECONDS=0\n\n# AWS Virtual Machines\ncat >&2 <<EOF\n# ============================================================================ #\n#          A W S   E C 2   I n v e n t o r y   A l l   A c c o u n t s\n# ============================================================================ #\n\nSaving to: $PWD/$csv\n\nSorted and header deduplicated: $PWD/$csv_sorted\n\nEOF\n\n        # don't do this, solved blank columns natively now so it's easier to spot end of line issues if they end in aa comma instead of ,\"\"\n        # see aws_info_ec2_csv.sh where empty fields are now explicitly set to \"\"\n        #s|,$|,\\\"\\\"|;\n\n# aws_info_ec2_csv.sh supports fixing the timestamp so we can combine this later\nexport LOG_TIMESTAMP=\"$log_timestamp\"\n\naws_foreach_profile.sh \"\n    '$srcdir/aws_info_ec2_csv.sh' '{profile}' |\n    sed '\n        s|^|\\\"{profile}\\\",|;\n        1s|^\\\"{profile}\\\"|\\\"AWS_Profile\\\"|;\n    ' # |\n    # this only works on Linux, can't do this fd trick on Mac\n    #tee <(tail -n +2 >> '$csv_sorted')\n\" |\ntee \"$csv\" |\ntail -n +2 |\ntee \"$csv_sorted\"\n\ntmp=\"$(mktemp)\"\n\n# sorting only makes sense when combining a single CSV output format which is why this is EC2 only\n\n# we'd have to combine all the individual CSVs but they have different timestamps, hard to predict, and don't want to make them less granular\nhead -n1 \"$csv\" > \"$tmp\"\n\n#tail -q -n +2 aws_info_ec2-*-\"$LOG_TIMESTAMP.csv\" |\ngrep -v '^\"AWS_Profile\",' \"$csv\" |\nsort -fu >> \"$tmp\"\n\nmv \"$tmp\" \"$csv_sorted\"\n\necho >&2\naws_foreach_profile.sh \"\n    grep -q '^\\\"{profile}\\\",' ||\n    echo 'WARNING: no EC2 Instances found in AWS Profile \\\"{profile}\\\"' '$csv_sorted'\n\" >&2\n\necho >&2\ntimestamp \"Raw CSV: $csv\" >&2\necho >&2\ntimestamp \"Sorted and Header deduplicated CSV: $csv_sorted\" >&2\necho >&2\ntimestamp \"Script Completed Successfully in $SECONDS secs: ${0##*/}\"\n"
  },
  {
    "path": "aws/aws_info_ec2_csv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-22 14:20:14 +0400 (Fri, 22 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\nscript_basename=\"${0##*/}\"\nscript_basename=\"${script_basename%%.sh}\"\nscript_basename=\"${script_basename%%_csv}\"\n\nlog_timestamp=\"${LOG_TIMESTAMP:-$(date '+%F_%H.%M.%S')}\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS EC2 Instances in quoted CSV format in the current AWS account\n\nWritten to be combined with aws_foreach_project.sh\n\nOutputs to both stdout and a file called $script_basename-<AWS_ACCOUNT_ID>-YYYY-MM-DD_HH.MM.SS.csv\n\nSo that you can diff subsequent runs to see the difference between EC2 VMs that come and go due to AutoScaling Groups\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_profile>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin aws\n\nif [ $# -gt 0 ]; then\n    aws_profile=\"$1\"\n    shift || :\n    export AWS_PROFILE=\"$aws_profile\"\nfi\n\naws_account_id=\"$(aws_account_id)\"\n\ncsv=\"$script_basename-$aws_account_id-$log_timestamp.csv\"\n\n# AWS Virtual Machines\ncat >&2 <<EOF\n# ============================================================================ #\n#                       A W S   E C 2   I n s t a n c e s\n# ============================================================================ #\n\nSaving to: $PWD/$csv\n\nEOF\n\ndeclare -A ami_map\n\ntimestamp \"Getting all unique AMI IDs in use\"\nami_ids=\"$(\"$srcdir/aws_ec2_ami_ids.sh\")\"\n\nwhile read -r ami_id; do\n    [[ -z \"$ami_id\" ]] && continue\n\n    timestamp \"Resolving AMI ID: $ami_id\"\n    ami_name=\"$(\n        aws ec2 describe-images \\\n            --image-ids \"$ami_id\" \\\n            --query 'Images[0].Name' \\\n            --output text\n    )\"\n    if [[ \"$ami_name\" == \"None\" || -z \"$ami_name\" ]]; then\n        timestamp \"WARNING: AMI ID '$ami_id' failed to resolve to AMI Name\"\n    else\n        timestamp \"AMI ID '$ami_id' => '$ami_name'\"\n        ami_map[\"$ami_id\"]=\"$ami_name\"\n    fi\ndone <<< \"$ami_ids\"\n\nsed_script=''\nfor ami_id in \"${!ami_map[@]}\"; do\n    ami_name=\"${ami_map[$ami_id]}\"\n    sed_script+=\"\n    s|\\\"$ami_id\\\"|\\\"$ami_name\\\"|g;\"\ndone\n\ntimestamp \"Getting list of EC2 instances\"\n# shellcheck disable=SC2016\njson=\"$(\n    aws ec2 describe-instances \\\n        --query 'Reservations[*].Instances[*].{\n                    \"Name\": Tags[?Key==`Name`].Value | [0],\n                    \"ID\": InstanceId,\n                    \"IP\": PrivateIpAddress,\n                    \"State\": State.Name,\n                    \"InstanceType\": InstanceType,\n                    \"AMI\": ImageId,\n                    \"Architecture\": Architecture,\n                    \"Platform\": PlatformDetails,\n                    \"PublicDNS\": publicDnsName,\n                    \"PrivateDNS\": PrivateDnsName\n                }' \\\n        --output json |\n    jq_debug_pipe_dump\n)\"\n\ntimestamp \"Generating CSV output with AMI images IDs resolved to names\"\necho >&2\necho '\"Instance_ID\",\"Instance_Name\",\"Private_IP_Address\",\"State\",\"Instance_Type\",\"Platform\",\"AMI\",\"Architecture\",\"Private_DNS\",\"Public_DNS\"'\njq -r '\n    .[][] |\n    [ .ID, .Name, .IP, .State, .InstanceType, .Platform, .AMI, .Architecture, .PrivateDNS, .PublicDNS ] |\n    map(if . == null or . == \"\" then \"\" else . end) |\n    @csv\n' <<< \"$json\" |\nsed \"$sed_script\" |\ntee \"$csv\"\n"
  },
  {
    "path": "aws/aws_ip_ranges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: eu-west-1 ROUTE53_HEALTHCHECKS\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-18 17:13:10 +0100 (Fri, 18 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns all the AWS IPs for a given Region and Service using the AWS ip-range json API:\n\n    https://ip-ranges.amazonaws.com/ip-ranges.json\n\nTo get and use these IPs directly in Terraform, see the Cloudflare Firewall module in https://github.com/HariSekhon/Terraform\n\nExamples:\n\n    Lists all regions and their services to filter on:\n\n        ${0##*/} list\n\n    Get all IPs for eu-west-1 region:\n\n        ${0##*/} eu-west-1\n\n    Get all eu-west-1 IPs for EC2, S3 or Route 53 Healthchecks:\n\n        ${0##*/} eu-west-1 EC2\n        ${0##*/} eu-west-1 S3\n        ${0##*/} eu-west-1 ROUTE53_HEALTHCHECKS\n\n    Get global Route 53 Healthcheck IPs:\n\n        ${0##*/} GLOBAL ROUTE53_HEALTHCHECKS\n\n    Get all Route 53 Healthcheck IPs in all regions:\n\n        ${0##*/} all ROUTE53_HEALTHCHECKS\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<region> <service> | list]\"\n\nhelp_usage \"$@\"\n\nurl=\"https://ip-ranges.amazonaws.com/ip-ranges.json\"\nregion=\"${1:-}\"\nservice=\"${2:-}\"\n\n# All regions are lowercase except for GLOBAL\nregion=\"$(tr '[:upper:]' '[:lower:]' <<< \"$region\")\"\nif [ \"$region\" = global ]; then\n    region=GLOBAL\nfi\n# All Services are uppercase\nservice=\"$(tr '[:lower:]' '[:upper:]' <<< \"$service\")\"\n\nif [ \"$region\" = list ]; then\n    curl -sS \"$url\" |\n    jq -r '.prefixes[] | [.region, .service] | @tsv' | sort -u\n    exit 0\nfi\n\ncurl -sSf \"$url\" |\n#jq -r \".prefixes[]\" |\n#if [ -n \"$region\" ] && [ \"$region\" != all ]; then\n#    #jq -r \".prefixes[] | select(.region == \\\"$region\\\") | .ip_prefix\"\n#    jq -r \"select(.region == \\\"$region\\\")\"\n#else\n#    cat\n#fi |\n#if [ -n \"$service\" ] && [ \"$service\" != all ]; then\n#    jq -r \"select(.service == \\\"$service\\\")\"\n#else\n#    cat\n#fi |\n#jq -r '.ip_prefix'\njq -r \"\n    .prefixes[] |\n    if(\\\"$region\\\" != \\\"\\\" and \\\"$region\\\" != \\\"all\\\") then\n        select(.region == \\\"$region\\\")\n    else\n        .\n    end |\n    if(\\\"$service\\\" != \\\"\\\" and \\\"$service\\\" != \\\"all\\\") then\n        select(.service == \\\"$service\\\")\n    else\n        .\n    end |\n    .ip_prefix\n\"  # end jq script\n"
  },
  {
    "path": "aws/aws_kinesis_stream_names.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-16 12:48:36 +0700 (Mon, 16 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly lists the Kinesis stream names\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws kinesis list-streams --query 'StreamNames[]' --output text | tr '[:space:]' '\\n'\n"
  },
  {
    "path": "aws/aws_kms_key_rotation_enabled.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-21 18:32:36 +0000 (Tue, 21 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists KMS keys and whether they have key rotation enabled\n\nOutput Format:\n\nKMS_Key       Rotation_Enabled (boolean)\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws kms list-keys |\njq -r '.Keys[].KeyId' |\nwhile read -r key; do\n    printf '%s\\t' \"$key\"\n    aws kms get-key-rotation-status --key-id \"$key\" |\n    jq -r '.KeyRotationEnabled' || :  # continue leaving blank if no permissions on a given key\ndone\n"
  },
  {
    "path": "aws/aws_kube_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 00:36:09 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates kubectl credentials and contexts for all AWS EKS clusters in the current AWS region\n\nThis is fast way to get set up in new environments, or even just add any new EKS clusters to your existing \\$HOME/.kube/config\n\nWARNING: AWS CLI switches your kubectl context to the last cluster you get credentials for. This can lead to race conditions between other kubectl scripts if they have not forked and isolated their \\$KUBECONFIG. Do not run this while other naive kubectl commands and scripts are running otherwise those non-isolated commands may fire against the wrong kubernetes cluster. See kubectl.sh for more info\n\nCan supply arguments to be passed to AWS CLI to set things like region eg.\n\n    ${0##*/} --region eu-west-2\n\n\n$usage_aws_cli_jq_required\n\n\nSee also:\n\n    kubectl.sh             - isolates kube config to fix kubectl commands to the given cluster to prevent race conditions from applying kubectl changes to the wrong cluster\n    aws_kubectl.sh         - same as above but also gets the credential\n    gke_kube_creds.sh      - same as this script but for GCP GKE clusters\n    rancher_kube_creds.sh  - same as this script but for Rancher Kubernetes clusters\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws eks list-clusters \"$@\" |\njq -r '.clusters[]' |\nwhile read -r cluster; do\n    echo \"Getting AWS EKS credentials for cluster '$cluster':\"\n    aws eks update-kubeconfig --name \"$cluster\" \"$@\"\n    echo\ndone\n"
  },
  {
    "path": "aws/aws_kubectl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 10:56:52 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a kubectl command safely fixed to an AWS EKS cluster by generating an isolated fixed config for the lifetime of this script\n\nAvoids concurrency race conditions with other concurrently executing commands or scripts by avoiding using or changing the global kubectl context\n\nEg. running:\n\n    kubectl config use-context\n            or\n    aws eks update-kubeconfig\n\neither by your hand or in other concurrently executing scripts changes your global kubectl context to run on the given cluster, which could divert your command or concurrently long running scripts in other windows to run kubectl commands on the wrong cluster, leading to cross environment misconfigurations and real world outages (I've seen this personally)\n\nFor frequent more convenient usage you will want to shorten the CLI by copying this script to a local copy in each cluster's yaml config directory and hardcoding the CLUSTER and REGION variables\n\nCould also use main kube config with kubectl switches --cluster / --context (after configuring, see aws_kube_creds.sh), but this is more convenient, especially when hardcoded for the local copy in each cluster's k8s yaml dir\n\n\n$usage_aws_cli_required\n(kubectl is also installed as part of 'make aws')\n\n\nSee Also:\n\n    aws_kube_creds.sh - auto-populates the credentials for all EKS clusters for your kubectl is ready to rock on AWS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<cluster> <zone> <kubectl_options>\"\n\nhelp_usage \"$@\"\n\n\n# ============================================================\n# HARDCODE THIS SECTION FOR SHORTER CLI convenience\n# REMOVE if hardcoding\nmin_args 3 \"$@\"\n\n# fixed to this environment - thou shalt deploy to no other cluster from this script\n\n# HARDCODE THESE for frequent shorter CLI usage\nCLUSTER=\"$1\"  # eg. my-cluster\nREGION=\"$2\"   # eg. us-east-1\n\n# REMOVE if hardcoding\nshift || :\nshift || :\n# ============================================================\n\nkube_config_isolate\n\naws eks update-kubeconfig --name \"$CLUSTER\" --region \"$REGION\"\necho >&2\n\nkubectl \"$@\"\n"
  },
  {
    "path": "aws/aws_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-23 12:14:19 +0000 (Thu, 23 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearch CloudWatch Logs, inserting a more human friendly hours ago optional args to generate the --start-time and --end-time epochs in milliseconds\n\nDefaults to finding logs in the last 24 hours but can optionally take an hours argument to search the last N hours\n\nYou must supply the --log-group-name and --filter-pattern arguments in addition to potentially supplying the hours ago args at the start\n\nExamples:\n\n    ${0##*/} --log-group-name aws-controltower/CloudTrailLogs --filter-pattern '{ ($.eventSource = \\\"batch.amazonaws.com\\\") && ($.eventName = \\\"SubmitJob\\\") }'\n\n    ${0##*/} 48 ...    # start 48 hours ago to present\n\n    ${0##*/} 24 3 ...  # start 24 hours ago up to to 3 hours ago\n\n    ${0##*/} --start-time \\\"\\$(date +%s --date='2021-12-21')000\\\" --end-time \\\"\\$(date +%s --date='2021-12-23')000\\\"  ... # explicitly calculated dates, using standard AWS CLI options (now you see why I default to the simple hours ago optional args)\n\n\nOutput Format in AWS JSON\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<hours_ago_start> <hours_ago_end> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 4 \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\nhours_ago_start=24\nhours_ago_end=0\n\nif [ -n \"${1:-}\" ] &&\n   ! [[ \"${1:-}\" =~ ^- ]]; then\n    hours_ago_start=\"$1\"\n    shift || :\nfi\n\nif [ -n \"${1:-}\" ] &&\n   ! [[ \"${1:-}\" =~ ^- ]]; then\n    hours_ago_end=\"$1\"\n    shift || :\nfi\n\nif ! [[ \"$hours_ago_start\" =~ ^[[:digit:]]+$ ]]; then\n    usage \"invalid value given for hours ago start argument, must be an integer\"\nfi\n\nif ! [[ \"$hours_ago_end\" =~ ^[[:digit:]]+$ ]]; then\n    usage \"invalid value given for hours ago end argument, must be an integer\"\nfi\n\nif ! [[ \"$*\" =~ --start-time ]]; then\n    args+=( --start-time \"$(date '+%s' --date=\"$hours_ago_start hours ago\")000\" )\nfi\nif ! [[ \"$*\" =~ --end-time ]]; then\n    args+=( --end-time \"$(date '+%s' --date=\"$hours_ago_end hours ago\")000\" )\nfi\n\naws logs filter-log-events ${args[@]:+\"${args[@]}\"} \\\n                           \"$@\"\n                           #--max-items 1 \\\n                           # --region eu-west-2  # set AWS_DEFAULT_REGION or pass --region via $@\n                           #--end-time \"$(date '+%s')000\" \\\n"
  },
  {
    "path": "aws/aws_logs_batch_jobs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-23 12:14:19 +0000 (Thu, 23 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches CloudWatch Logs for AWS Batch job submit requests in the last N hours to find who is running large expensive jobs\n\nDefaults to finding logs in the last 24 hours but can optionally take an hours argument to search the last N hours\n\nExample:\n\n    ${0##*/}\n\n    ${0##*/} 48     # 48 hours ago to present\n\n    ${0##*/} 24 3   # 24 hours ago to 3 hours ago\n\n    # explicitly calculated dates in millisecond epochs (hence the suffixed 000), using standard AWS CLI options (now you see why I default to the simple hours ago optional args)\n    ${0##*/} --start-time \\\"\\$(date +%s --date='2021-12-21')000\\\" --end-time \\\"\\$(date +%s --date='2021-12-23')000\\\"\n\n\nOutput Format:\n\n<timestamp>     <job_id>    <user>    <job_name>\neg.\n2021-12-22T19:28:57Z    12a345b6-c789-0de1-f2a3-4b567cdef8ab    hari@domain.com       my_job_1\n2021-12-22T20:22:17Z    123ab4cd-e5f6-789a-b012-34c5d67e8f90    MyBatchRole/1ab2c34567890123d45e678f901a2b34  my_report_postprocess\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<hours_ago_start> <hours_ago_end> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n\"$srcdir/aws_logs.sh\" \"$@\" \\\n                      --log-group-name aws-controltower/CloudTrailLogs \\\n                      --filter-pattern '{ ($.eventSource = \"batch.amazonaws.com\") && ($.eventName = \"SubmitJob\") }' |\njq -r '.events[].message' |\njq_debug_pipe_dump_slurp |\njq -r -s '.[] |\n          [\n            .eventTime,\n            .responseElements.jobId,\n            ( .userIdentity.arn | sub(\"arn:aws:sts::\\\\d+:assumed-role/\"; \"\") | sub(\"AWSReservedSSO_\\\\w+/\"; \"\") ),\n            .responseElements.jobName\n          ] |\n          @tsv'\n"
  },
  {
    "path": "aws/aws_logs_ec2_spot.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-23 12:14:19 +0000 (Thu, 23 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches CloudWatch Logs for AWS EC2 Spot fleet creation requests in the last 24 hours to trace through to services incurring high EC2 charges such as large AWS Batch jobs\n\nDefaults to finding logs in the last 24 hours but can optionally take an hours argument to search the last N hours\n\nExample:\n\n    ${0##*/}\n\n    ${0##*/} 48     # 48 hours ago to present\n\n    ${0##*/} 24 3   # 24 hours ago to 3 hours ago\n\n    # explicitly calculated dates in millisecond epochs (hence the suffixed 000), using standard AWS CLI options (now you see why I default to the simple hours ago optional args)\n    ${0##*/} --start-time \\\"\\$(date +%s --date='2021-12-21')000\\\" --end-time \\\"\\$(date +%s --date='2021-12-23')000\\\"\n\n\nOutput Format:\n\n<timestamp>     <user>    <first_tag_value>\neg.\n2021-12-22T22:37:28Z    AutoScaling     AWSBatch-<name>-asg-12a3b4c5-67d8-9efa-b012-34cde56789f0\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<hours_ago_start> <hours_ago_end> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n\"$srcdir/aws_logs.sh\" \"$@\" \\\n                      --log-group-name aws-controltower/CloudTrailLogs \\\n                      --filter-pattern '{ ($.eventSource = \"ec2.amazonaws.com\") && ($.eventName = \"CreateFleet\") }' |\njq -r '.events[].message' |\njq_debug_pipe_dump_slurp |\njq -r -s '.[] |\n          [\n            .eventTime,\n            ( .userIdentity.principalId | sub(\"^\\\\w+:\"; \"\") ),\n            .requestParameters.CreateFleetRequest.TagSpecification.Tag[0].Value\n          ] |\n          @tsv'\n"
  },
  {
    "path": "aws/aws_logs_ecs_tasks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-23 12:14:19 +0000 (Thu, 23 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches CloudWatch Logs for AWS ECS task run requests in the last 24 hours to trace through to services incurring high EC2 charges such as large AWS Batch jobs\n\nDefaults to finding logs in the last 24 hours but can optionally take an hours argument to search the last N hours, and can optionally take other AWS CLI options\n\nExample:\n\n    ${0##*/}\n\n    ${0##*/} 48     # 48 hours ago to present\n\n    ${0##*/} 24 3   # 24 hours ago to 3 hours ago\n\n    # explicitly calculated dates in millisecond epochs (hence the suffixed 000), using standard AWS CLI options (now you see why I default to the simple hours ago optional args)\n    ${0##*/} --start-time \\\"\\$(date +%s --date='2021-12-21')000\\\" --end-time \\\"\\$(date +%s --date='2021-12-23')000\\\"\n\n\nOutput Format:\n\n<timestamp>     <user>    <task_definition:version>\neg.\n2021-12-23T02:05:34Z    aws-batch       MyJob:11\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<hours_ago_start> <hours_ago_end> <aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n\"$srcdir/aws_logs.sh\" \"$@\" \\\n                      --log-group-name aws-controltower/CloudTrailLogs \\\n                      --filter-pattern '{ ($.eventSource = \"ecs.amazonaws.com\") && ($.eventName = \"RunTask\") }' |\njq -r '.events[].message' |\njq_debug_pipe_dump_slurp |\n# 2021-12-23T02:05:34Z    aws-batch       arn:aws:ecs:eu-west-2:123456789012:task-definition/MyJob:11\njq -r -s '.[] |\n          [\n            .eventTime,\n            ( .userIdentity.principalId | sub(\"^\\\\w+:\"; \"\") ),\n            ( .requestParameters.taskDefinition | sub(\"arn:aws:ecs:[\\\\w-]+:\\\\d+:task-definition/\"; \"\") )\n          ] |\n          @tsv'\n"
  },
  {
    "path": "aws/aws_meta.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 20:41:35 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# intentionally not using lib/ so that this script is standalone and can be easily distributed to VMs without git cloning the whole repo\nusage(){\n    echo \"\nAWS EC2 Metadata API query shortcut\n\n${0##*/} <resource> [curl_options]\n\neg. ${0##*/} public-ipv4\n    ${0##*/} public-hostname\n    ${0##*/} local-ipv4\n    ${0##*/} local-hostname\n    ${0##*/} ami-id\n    ${0##*/} instance-type\n    ${0##*/} placement/availability-zone\n\nSee also ec2-metadata script which is a more complete shell script\n\"\n    exit 3\n}\n\nif [ $# -lt 1 ] ||\n   [[ \"${1:-}\" =~ -.* ]]; then\n    usage\nfi\n\nif ! curl -sS --connect-timeout 2 http://169.254.169.254/ &>/dev/null; then\n    echo\n    echo \"This script must be run from within an EC2 instance as that is the only place the AWS EC2 Metadata API is available\"\n    exit 2\nfi\n\ncurl \"http://169.254.169.254/latest/meta-data/${1:-}\" \"${@:2}\"\n"
  },
  {
    "path": "aws/aws_nat_gateways_public_ips.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-28 12:26:32 +0100 (Thu, 28 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the public IP addresses of your AWS NAT Gateways - useful to give a list of IPs to clients to permit through firewalls for webhooks or similar calls\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\naws ec2 describe-nat-gateways |\njq -r '.NatGateways[].NatGatewayAddresses[].PublicIp'\n"
  },
  {
    "path": "aws/aws_profile.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-17 03:24:18 +0200 (Tue, 17 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSwitches to an AWS Profile selected from a convenient interactive menu list of AWS profiles to choose from\n\nParses \\$AWS_CONFIG_FILE or \\$HOME/.aws/config for the menu list\n\nSkips the menu if there is only 1 AWS profile found in the config\n\nThen sets the AWS_PROFILE and exec's to \\$SHELL to inherit it\n\nConvenient when you have lots of work AWS profiles\n\n\nRequires dialogue menu CLI tool to be installed - attempts to install it via OS package manager if not already found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config_path>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif ! type -P dialog &>/dev/null; then\n    timestamp \"Diaglog not found in \\$PATH, attempting to install via OS package manager\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" dialog\n    echo\nfi\n\n[ -n \"${HOME:-}\" ] || HOME=~\n\nconfig=\"${1:-${AWS_CONFIG_FILE:-$HOME/.aws/config}}\"\n\naws_profile(){\n    local profile=\"${1// }\"\n    if [ -n \"$profile\" ]; then\n        if ! [[ \"$profile\" =~ ^[[:alnum:]_-]+$ ]]; then\n            echo \"invalid profile name given, must be alphanumeric, dashes and underscores allowed\"\n            return 1\n        fi\n        local profile_data\n        profile_data=\"$(aws_get_profile_data \"$profile\")\"\n        [ -n \"$profile_data\" ] ||\n        profile_data=\"$(aws_get_profile_data \"$profile\" \"$config\")\"\n        if [ -z \"$profile_data\" ]; then\n            echo \"profile [$profile] not found in $config!\"\n            return 1\n        fi\n        #aws_clean_env\n        timestamp \"Setting AWS_PROFILE='$profile'\"\n        export AWS_PROFILE=\"$profile\"\n    elif [ -n \"$AWS_PROFILE\" ]; then\n        timestamp \"AWS_PROFILE='$AWS_PROFILE' was already set\"\n    else\n        die \"ERROR: not setting AWS Profile (not found)\"\n    fi\n}\n\naws_get_profile_data(){\n    local profile=\"$1\"\n    local filename=\"${2:-$config}\"\n    sed -n \"/^[[:space:]]*\\\\[\\\\(profile[[:space:]]*\\\\)*$profile\\\\]/,/^[[:space:]]*\\\\[/p\" \"$filename\"\n}\n\nprofiles=\"$(sed -n 's/^[[:space:]]*\\[\\(profile[[:space:]][[:space:]]*\\)*\\(.*\\)\\]/\\2/p' \"$config\" | sort -fu)\"\n\nprofile_menu_items=()\n\nwhile read -r line; do\n\n    # used for counting and string conversion if only a single item\n\n    profile_menu_items+=(\"$line\")\n\n    # passed to dialog because it requires args: tag1 visibletext tag2 visibletext\n    # - by making the second one blank it uses the item as both the tag to be returned\n    # to script as well as the visible text\n\n    profile_menu_tag_items+=(\"$line\" \" \")\n\ndone <<< \"$profiles\"\n\nif [ \"${#profile_menu_items[@]}\" -eq 0 ];then\n    die 'No AWS Profiles found!'\nelif [ \"${#profile_menu_items[@]}\" -eq 1 ];then\n    profile=\"${profile_menu_items[*]}\"\nelse\n    profile=\"$(dialog --menu \"Choose which AWS profile to switch to:\" \"$LINES\" \"$COLUMNS\" \"$LINES\" \"${profile_menu_tag_items[@]}\" 3>&1 1>&2 2>&3)\"\nfi\n\naws_profile \"$profile\"\n\nexec \"$SHELL\"\n"
  },
  {
    "path": "aws/aws_profile_config_add_if_missing.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 20:47:15 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads AWS profile config blocks from stdin and appends them to the ~/.aws/config file if the profile section is not found\n\nYou can override the config file location by setting environment variable AWS_CONFIG_FILE\n\nUses adjacent script:\n\n    ../data/ini_config_add_if_missing.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\naws_config_file=\"${1:-${AWS_CONFIG_FILE:-$HOME/.aws/config}}\"\n\n# designed to support AWS config sections which require a profile prefix\nexport SECTION_PREFIX=\"profile\"\n\n\"$srcdir/../data/ini_config_add_if_missing.sh\" \"$aws_config_file\"\n"
  },
  {
    "path": "aws/aws_profile_generate_direnvs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-24 04:21:37 +0400 (Sun, 24 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates subdirectories containing the config.ini and .envrc for every AWS profile found\nin the given file or \\$AWS_CONFIG_FILE or ~/.aws/config\n\nDoes not overwrite by default for safety, you must export environment variable AWS_PROFILE_DIRENV_OVERWRITE=1\n\nUseful to take a large generated AWS config.ini from script:\n\n    aws_sso_configs.sh\n\nand then split it into subdirectories for direnvs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nconfig=\"${1:-${AWS_CONFIG_FILE:-$HOME/.aws/config}}\"\n\nif ! [ -f \"$config\" ]; then\n    die \"ERROR: file does not exist: $config\"\nfi\n\ngrep -Eo \"^[[:space:]]*\\[profile .+\\]\" \"$config\" |\nsed 's/.*\\[profile//; s/\\].*$//' |\nwhile read -r profile; do\n    if is_blank \"$profile\"; then\n        continue\n    fi\n    if ! [[ \"$profile\" =~ ^[[:alnum:]_-]+$ ]]; then\n        warn \"profile '$profile' failed regex validation, skipping for safety...\"\n        continue\n    fi\n    mkdir -p \"$profile\"\n    subconfig=\"$profile/config.ini\"\n    if ! [ -f \"$subconfig\" ]; then\n        timestamp \"Generating $subconfig\"\n        cat > \"$subconfig\" <<EOF\n#!/usr/bin/env bash\n#\n# Generated using ${0##*/} from:\n#\n#   https://github.com/HariSekhon/DevOps-Bash-tools\n\nEOF\n        \"$srcdir/../data/ini_grep_section.sh\" \"profile $profile\" \"$config\" >> \"$subconfig\"\n        sed -i -e '${/^$/d}' \"$subconfig\"\n        if ! [ -s \"$subconfig\" ]; then\n            die \"Failed to generate $subconfig\"\n        fi\n    fi\n    envrc=\"$profile/.envrc\"\n    if ! [ -f \"$envrc\" ] ||\n       [ \"${AWS_PROFILE_DIRENV_OVERWRITE:-}\" = 1 ]; then\n        # AWS_ACCOUNT_ID is automatically inferred by envrc code from AWS_PROFILE which is all we need\n        #account_id=\"$(grep -Eo '^[[:space:]]*[[:alnum:]_]*account_id[[:space:]]*=[[:space:]][[:digit:]]+' \"$subconfig\" || :)\"\n        #if [ -z \"$account_id\" ]; then\n        #    die \"Failed to determine AWS Account ID from $subconfig\"\n        #fi\n        #echo \"export AWS_ACCOUNT_ID=$aws_account_id\" >> \"$envrc\"\n        timestamp \"Generating $envrc\" # with AWS_PROFILE=$profile\"\n        cat > \"$envrc\" <<EOF\n#!/usr/bin/env bash\n#\n# Generated using ${0##*/} from:\n#\n#   https://github.com/HariSekhon/DevOps-Bash-tools\n\nexport AWS_PROFILE=$profile\n\nexport EKS_CLUSTER=\"\\$(aws eks list-clusters --output json 2>/dev/null | jq -r '.clusters[0]')\"\n#export EKS_NAMESPACE=\n\n#. ../.envrc\n\ngit_root=\"\\$(git rev-parse --show-toplevel)\"\n\n# shellcheck disable=SC1091\n. \"\\$git_root/aws/.envrc\"\nEOF\n    else\n        timestamp \"Direnv configuration already exists, skipping: $envrc\"\n    fi\ndone\n"
  },
  {
    "path": "aws/aws_rds_get_version.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-26 17:02:07 +0200 (Mon, 26 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly retrieve the version of an RDS database matching an AWS instance name or IP address\n\nThis is important to fetch a match JDBC jar file using ../install/download_*_jdbc.sh script\n\n\nWhen an IP address is given, since this is not available in AWS CLI output, has to iterate\nthrough the list or RDS instance URL endpoints doing DNS lookups until it finds it. This is an O(n)\noperation and can be expensive if you have a lot of instances, so you are recommended to supply the\nRDS instance name instead if you know it. This IP resolution functionality was because teams sometimes\nonly gave me the IP address of a database when requesting JDBC setup to them\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<rds_name_or_ip>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrds_instance=\"$1\"\n\n# $ip_regex defined in lib/utils.sh\n# shellcheck disable=SC2154\nif [[ \"$rds_instance\" =~ $ip_regex ]]; then\n    ip=\"$rds_instance\"\n    rds_instance=\"\"\n    while read -r instance fqdn; do\n        timestamp\n        echo -n \"Checking IP of instance '$instance' - $fqdn => \" >&2\n        # check if it's a CNAME and double resolve if so\n        cname=\"$(dig +short \"$fqdn\" CNAME)\"\n        if [ -n \"$cname\" ]; then\n            ip_result=\"$(dig +short \"$cname\" A)\"\n        else\n            ip_result=\"$(dig +short \"$fqdn\" A)\"\n        fi\n        echo \"$ip_result\" >&2\n        if [ \"$ip_result\" = \"$ip\" ]; then\n            timestamp \"Found RDS instance with IP '$ip' to be '$instance'\"\n            rds_instance=\"$instance\"\n            break\n        fi\n    done < <(aws rds describe-db-instances --query 'DBInstances[].[DBInstanceIdentifier,Endpoint.Address]' --output text)\n    if [ -z \"$rds_instance\" ]; then\n        die \"Failed to find RDS instance by IP in region '$(aws_region)' - perhaps IP is wrong or \\$AWS_DEFAULT_REGION is not set correctly?\"\n    fi\nfi\n\n#aws rds describe-db-instances --query \"DBInstances[?DBInstanceIdentifier==\\`$rds_instance\\`].[DBInstanceIdentifier,Engine,EngineVersion]\" --output text\nversion=\"$(aws rds describe-db-instances --query \"DBInstances[?DBInstanceIdentifier==\\`$rds_instance\\`].[EngineVersion]\" --output text)\"\n\nif [ -z \"$version\" ]; then\n    die \"Failed to find RDS instance '$rds_instance' in region '$(aws_region)' - perhaps name is wrong or \\$AWS_DEFAULT_REGION is not set correctly?\"\nfi\n\nif [ \"$(awk '{print NF}' <<< \"$version\")\" -gt 1 ]; then\n    die \"WARNING: more than one version token returned: $version\"\nfi\n\necho \"$version\"\n"
  },
  {
    "path": "aws/aws_rds_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 04:49:35 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList RDS instances with select fields - Name, Status, Engine, AZ, Instance Type, Storage, Endpoint DNS FQDN Address\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<region>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    export AWS_DEFAULT_REGION=\"$1\"\nfi\n\naws rds describe-db-instances \\\n    --query \"DBInstances[*].[DBInstanceIdentifier, DBInstanceStatus, Engine, AvailabilityZone, DBInstanceClass, AllocatedStorage, Endpoint.Address]\" \\\n    --output table\n"
  },
  {
    "path": "aws/aws_rds_open_port_to_my_ip.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-06-27 13:17:51 +0200 (Thu, 27 Jun 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds a security group to an RDS DB instance to open its native database SQL port to your public IP address\n\n- determines your public IP address\n- determines the RDS DB instance's port (usually 3306 for MySQL or 5432 for PostgreSQL)\n- creates a security group containing your username in its name\n- adds a rule to the security group to permit the DB port from your public IP\n- adds the security group to the RDS DB instance\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<instance_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ndb_instance=\"$1\"\n\nuser=\"${USER:-whoami}\"\n\nif is_blank \"$user\"; then\n    die \"Failed to determine username to label the new security group with\"\nfi\n\ntimestamp \"Determining your public IP\"\npublic_ip=\"$(curl -sS https://checkip.amazonaws.com || curl -sS ifconfig.co/json | jq -r '.ip')\"\n\n# $ip_regex is imported from utils.sh which is imported from aws.sh above\n# shellcheck disable=SC2154\nif ! [[ \"$public_ip\" =~ $ip_regex ]]; then\n    die \"Failed to determine your IP address\"\nfi\n\ntimestamp \"Determined your public IP to be '$public_ip'\"\n\ntimestamp \"Determining the port of the database instance '$db_instance'\"\nport=\"$(aws rds describe-db-instances --db-instance-identifier \"$db_instance\" --query 'DBInstances[0].Endpoint.Port' --output text)\"\ntimestamp \"Determined DB port to be '$port'\"\n\nsecurity_group=\"$user-rds-home-access\"\n\ntimestamp \"Determining VPC Id for DB instance '$db_instance'\"\nvpc_id=\"$(aws rds describe-db-instances --db-instance-identifier \"$db_instance\" \\\n                                     --query 'DBInstances[0].DBSubnetGroup.VpcId' --output text)\"\ntimestamp \"Determined VPC Id to be '$vpc_id'\"\n\nif aws ec2 describe-security-groups \\\n        --filters \"Name=vpc-id,Values=vpc-043c8640ca46649b4\" \\\n                  \"Name=group-name,Values=hari-rds-home-access\" \\\n        --query 'SecurityGroups[0].GroupId' \\\n        --output text >/dev/null; then\n    timestamp \"Security group '$security_group' already exists, skipping creation\"\nelse\n    timestamp \"Creating security group '$security_group'\"\n    aws ec2 create-security-group --group-name \"$security_group\" \\\n                                  --description \"Security group for RDS access for $user home IP\" \\\n                                  --vpc-id \"$vpc_id\"\nfi\n\n#security_group_id=\"$(aws ec2 describe-security-groups --group-names \"$security_group\" --query 'SecurityGroups[0].GroupId' --output text)\"\nsecurity_group_id=\"$(aws ec2 describe-security-groups \\\n        --filters \"Name=vpc-id,Values=$vpc_id\" \\\n        \"Name=group-name,Values=hari-rds-home-access\" \\\n        --query 'SecurityGroups[0].GroupId' \\\n        --output text)\"\n\nsecurity_group_rules=\"$(aws ec2 describe-security-groups --group-ids \"$security_group_id\" --query 'SecurityGroups[0].IpPermissions' --output json)\"\n\ncidr=\"$public_ip/32\"\nprotocol=\"tcp\"\n\nsecurity_rule_exists=$(jq -r \\\n    --arg protocol \"$protocol\" \\\n    --arg port \"$port\" \\\n    --arg cidr \"$cidr\" '\n        .[] |\n        select(.IpProtocol == $protocol and\n               .FromPort   == ($port | tonumber) and\n               .ToPort == ($port | tonumber) and\n               .IpRanges[]?.CidrIp == $cidr) |\n        length > 0\n' <<< \"$security_group_rules\")\n\nif [ \"$security_rule_exists\" = \"true\" ]; then\n  timestamp \"Security rule already exists in security group '$security_group', skipping adding it\"\nelse\n    timestamp \"Adding rule to security group '$security_group' opening port $port to your IP '$public_ip'\"\n    aws ec2 authorize-security-group-ingress \\\n                --group-id \"$security_group_id\" \\\n                --protocol tcp \\\n                --port \"$port\" \\\n                --cidr \"$cidr\"\nfi\n\ntimestamp \"Adding security group '$security_group' to RDS instance '$db_instance'\"\naws rds modify-db-instance --db-instance-identifier \"$db_instance\" --vpc-security-group-ids \"$security_group_id\" --apply-immediately >/dev/null\n\ntimestamp \"RDS DB instance '$db_instance' port '$port' is now open to your IP '$public_ip'\"\n"
  },
  {
    "path": "aws/aws_route53_check_ns_records.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-06 12:58:19 +0100 (Wed, 06 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks AWS Route 53 public hosted zones NS records are properly delegated in the public DNS hierarchy to all of the allocated AWS NS servers\n\nCan specify one or more hosted zones space separated, otherwise will find and iterate all public hosted zones in the current AWS account\n\nHelps test your AWS Route 53 setup and debug any imperfections in your NS delegation\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<public_hosted_zones_to_check>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nstatus=0\n\ncheck_zone(){\n    local id=\"$1\"\n    local zone=\"$2\"\n    local aws_ns_servers\n    local aws_ns_status=0\n    local public_ns_status=0\n    timestamp \"Checking domain '$zone'\"\n    aws_ns_servers=\"$(aws route53 get-hosted-zone --id \"$id\" | jq -r '.DelegationSet.NameServers[]')\"\n    #public_delegated_ns_servers=\"$(host -t NS \"$zone\" | awk '{print $4}')\"\n    public_delegated_ns_servers=\"$(dig \"$zone\" NS +short)\"\n    for aws_ns_server in $aws_ns_servers; do\n        if grep -Fxq \"$aws_ns_server.\" <<< \"$public_delegated_ns_servers\"; then\n            timestamp \"AWS NS server '$aws_ns_server' found\"\n        else\n            timestamp \"WARNING: AWS NS server '$aws_ns_server NOT FOUND in live public delegated NS servers for zone '$zone'\"\n            aws_ns_status=1\n            status=1\n        fi\n    done\n    for delegated_ns_server in $public_delegated_ns_servers; do\n        if ! grep -Fxq \"${delegated_ns_server%.}\" <<< \"$aws_ns_servers\"; then\n            timestamp \"WARNING: ROGUE live delegated public NS server '$delegated_ns_server' found, not in list of AWS NS servers for zone '$zone'\"\n            public_ns_status=1\n            status=1\n        fi\n    done\n    if [ $aws_ns_status -eq 0 ]; then\n        timestamp \"AWS hosted zone '$zone' NS servers all accounted for in public DNS hierarchy delegation\"\n    fi\n    if [ $public_ns_status -eq 0 ]; then\n        timestamp \"Domain '$zone' all public NS servers accounted for in AWS hosted zone\"\n    fi\n    echo >&2\n}\n\ntimestamp \"Getting hosted zones list\"\nhosted_zones=\"$(aws route53 list-hosted-zones | jq -r '.HostedZones[] | [.Id, .Name] | @tsv')\"\necho >&2\n\nif [ $# -gt 0 ]; then\n    for zone in \"$@\"; do\n        while read -r id name; do\n            if [ \"${zone%.}\" = \"${name%.}\" ]; then\n                check_zone \"$id\" \"$name\"\n            fi\n        done <<< \"$hosted_zones\"\n    done\nelse\n    while read -r id name; do\n        check_zone \"$id\" \"$name\"\n    done <<< \"$hosted_zones\"\nfi\n\nif [ $status -eq 0 ]; then\n    timestamp \"OK: All zones passed NS delegation validation\"\nelse\n    timestamp \"ERROR: one or more zones failed NS delegation validation\"\n    exit 1\nfi\n"
  },
  {
    "path": "aws/aws_s3_access_logging.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-21 18:13:28 +0000 (Tue, 21 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists S3 buckets and their access logging status\n\nOutput Format:\n\nS3_Bucket      TargetPrefix    TargetBucket\n\nIf access logging isn't configured on a bucket, outputs:\n\nS3_Bucket      S3_ACCESS_LOGGING_NOT_CONFIGURED\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\nbuckets=\"$(aws s3 ls | cut -d' ' -f3-)\"\n\nnum_buckets=\"$(wc -l <<< \"$buckets\")\"\nnum_buckets=\"${num_buckets//[[:space:]]}\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\necho \"Fetching access logging status for each of $num_buckets buckets (this may take a while):\" >&2\nwhile read -r name; do\n    printf '%s\\t' \"$name\"\n    output=\"$(aws s3api get-bucket-logging --bucket \"$name\" |\n    jq -r '.LoggingEnabled | [.TargetPrefix, .TargetBucket] | @tsv')\"\n    if [ -z \"$output\" ]; then\n        echo \"S3_ACCESS_LOGGING_NOT_CONFIGURED\"\n    else\n        echo \"$output\"\n    fi\ndone <<< \"$buckets\" #|\n#sort |\n#column -t\n"
  },
  {
    "path": "aws/aws_s3_account_block_public_access.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-31 10:25:27 +0100 (Tue, 31 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBlocks S3 public access at the AWS Account level\n\nFirst arg must be the AWS Account ID. If not given and \\$AWS_ACCOUNT_ID is set, will use that, otherwise will infer from the current access key\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<aws_account_id>\"\n\nhelp_usage \"$@\"\n\nmax_args 1\n\naws_account_id=\"${1:-}\"\nshift || :\n\nif [ -z \"$aws_account_id\" ]; then\n    if [ -n \"${AWS_ACCOUNT_ID:-}\" ]; then\n        aws_account_id=\"$AWS_ACCOUNT_ID\"\n    else\n        timestamp \"inferring AWS account id from access key\"\n        aws_account_id=\"$(aws_account_id)\"\n        timestamp \"inferred AWS account id to be '$aws_account_id'\"\n    fi\nfi\n\nexport AWS_DEFAULT_OUTPUT=json\n\ntimestamp \"Blocking S3 Public Access for AWS account id '$aws_account_id'\"\naws s3control put-public-access-block --account-id \"$aws_account_id\" --public-access-block-configuration 'BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true'\ntimestamp \"Blocked public access\"\n"
  },
  {
    "path": "aws/aws_s3_bucket.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest-terraform-state\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-27 18:03:32 +0100 (Fri, 27 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an S3 bucket with the following optimizations:\n\n- Blocks Public Access\n- Enables Versioning\n- Enables Server Side Encryption\n- Creates Bucket Policy to lock out any given user/group/role ARNs (optional)\n\nIdempotent: skips bucket creation if already exists, applies versioning, encryption, blocks public access and applies bucket policy if none exists of if \\$OVERWRITE_BUCKET_POLICY is set to any value\n\nRegion: will create the bucket in your configured region, to override locally set \\$AWS_DEFAULT_REGION\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<bucket_name> [<ARNs_to_block_access_from>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbucket=\"$1\"\nshift || :\n\narns_to_block=(\"$@\")\n\nexport AWS_DEFAULT_OUTPUT=json\n\nif aws s3 ls \"s3://$bucket\" &>/dev/null; then\n    timestamp \"Bucket '$bucket' already exists\"\nelse\n    timestamp \"Creating S3 bucket:  $bucket\"\n    aws s3 mb \"s3://$bucket\" || :\nfi\necho >&2\n\ntimestamp \"Enabling S3 versioning\"\naws s3api put-bucket-versioning --bucket \"$bucket\" --versioning-configuration 'Status=Enabled'\ntimestamp \"Versioning enabled\"\necho >&2\n\n#timestamp \"Enabling S3 MFA Delete (only works if you are MFA authenticated)\"\n#if aws s3api put-bucket-versioning --bucket \"$bucket\" --versioning-configuration 'MFADelete=Enabled,Status=Enabled'; then\n#    timestamp \"MFA Delete enabled\"\n#else\n#    timestamp \"WARNING: MFA Delete setting failed, must enable manually if not calling this script from an MFA enabled session\"\n#fi\n#echo >&2\n\ntimestamp \"Enabling S3 server-side encryption\"\naws s3api put-bucket-encryption --bucket \"$bucket\" --server-side-encryption-configuration '{\"Rules\": [{\"ApplyServerSideEncryptionByDefault\": {\"SSEAlgorithm\": \"AES256\"}}]}'\ntimestamp \"Encryption enabled\"\necho >&2\n\n\"$srcdir/aws_s3_buckets_block_public_access.sh\" \"$bucket\"\n\nif [ -n \"${arns_to_block[*]:-}\" ]; then\n    if [ -z \"${OVERWRITE_BUCKET_POLICY:-}\" ] && \\\n       timestamp \"Checking for existing S3 bucket policy\" && \\\n       [ -n \"$(aws s3api get-bucket-policy --bucket \"$bucket\" --query Policy --output text 2>/dev/null)\" ]; then\n        timestamp \"WARNING: bucket policy already exists, not overwriting for safety, must edit manually\"\n    else\n        timestamp \"Creating bucket policy to lock out given ARNs:\"\n        echo >&2\n        for arn in \"${arns_to_block[@]}\"; do\n            printf '\\t%s\\n' \"$arn\" >&2\n        done\n        echo >&2\n        aws s3api put-bucket-policy --bucket \"$bucket\" --policy \"$(cat <<EOF\n{\n  \"Id\": \"Policy1653672260380\",\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Sid\": \"DisallowedUserGroupRoleArns\",\n      \"Action\": \"s3:*\",\n      \"Effect\": \"Deny\",\n      \"Resource\": [\n        \"arn:aws:s3:::$bucket\",\n        \"arn:aws:s3:::$bucket/*\"\n      ],\n      \"Principal\": {\n        \"AWS\": [\n$(\n    for arn in \"${arns_to_block[@]}\"; do\n        # pad by 10 spaces using an empty first arg\n        printf '%10s\"%s\",\\n' \"\" \"$arn\"\n    done |\n    sed '$ s/,$//'\n)\n        ]\n      }\n    }\n  ]\n}\nEOF\n)\"\n        timestamp \"Bucket policy created\"\n    fi\nfi\n"
  },
  {
    "path": "aws/aws_s3_buckets_block_public_access.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest-terraform-state\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-27 18:03:32 +0100 (Fri, 27 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBlocks all public access for given S3 bucket(s) or files containing one bucket per line\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<buckets_or_files>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexport AWS_DEFAULT_OUTPUT=json\n\nblock_bucket_public_access(){\n    local bucket=\"$1\"\n    timestamp \"Enabling S3 Block Public Access for bucket '$bucket'\"\n    aws s3api put-public-access-block --bucket \"$bucket\" --public-access-block-configuration 'BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true'\n    timestamp \"Blocked public access\"\n    echo >&2\n}\n\nfor arg in \"$@\"; do\n    if [ -f \"$arg\" ]; then\n        while read -r bucket; do\n            block_bucket_public_access \"$bucket\"\n        done < \"$arg\"\n    else\n        block_bucket_public_access \"$arg\"\n    fi\ndone\n"
  },
  {
    "path": "aws/aws_s3_check_account_public_blocked.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-31 10:25:27 +0100 (Tue, 31 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks S3 public access is blocked at the AWS Account level\n\nFirst arg must be the AWS Account ID. If not given and \\$AWS_ACCOUNT_ID is set, will use that, otherwise will infer from the current access key\n\nRaw JSON is output to stdout\nOK/WARNING Status is output to stderr\nExits with error code 1 if S3 public access is set and exit code 2 if not fully blocked via policy at the account level\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<aws_account_id>\"\n\nhelp_usage \"$@\"\n\nmax_args 1\n\naws_account_id=\"${1:-}\"\nshift || :\n\nif [ -z \"$aws_account_id\" ]; then\n    if [ -n \"${AWS_ACCOUNT_ID:-}\" ]; then\n        aws_account_id=\"$AWS_ACCOUNT_ID\"\n    else\n        timestamp \"inferring AWS account id from access key\"\n        aws_account_id=\"$(aws_account_id)\"\n        timestamp \"inferred AWS account id to be '$aws_account_id'\"\n    fi\nfi\n\nexport AWS_DEFAULT_OUTPUT=json\n\nshopt -s nocasematch\n\noutput=\"$(aws s3control get-public-access-block --account-id \"$aws_account_id\" || :)\"\nif [ -n \"$output\" ]; then\n    echo \"$output\"\n    # XXX: must align with read command a few lines down\n    policy=\"$(jq -r '.PublicAccessBlockConfiguration | [ .BlockPublicAcls, .IgnorePublicAcls, .BlockPublicPolicy, .RestrictPublicBuckets ] | @tsv' <<< \"$output\")\"\n    read -r BlockPublicAcls IgnorePublicAcls BlockPublicPolicy RestrictPublicBuckets <<< \"$policy\"\n    if [[ \"$BlockPublicAcls\" =~ false ]] ||\n       [[ \"$IgnorePublicAcls\" =~ false ]] ||\n       [[ \"$BlockPublicPolicy\" =~ false ]] ||\n       [[ \"$RestrictPublicBuckets\" =~ false ]]; then\n        echo \"WARNING: Block Public Access policy is not enabled at the account level for AWS Account '$aws_account_id'! \" >&2\n        exit 2\n    else\n        echo \"OK: Block Public Access policy is enabled at the account level for AWS Account '$aws_account_id'\" >&2\n    fi\nelse\n    echo \"WARNING: Block Public Access policy is not set at the account level for AWS Account '$aws_account_id'! \" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "aws/aws_s3_check_buckets_public_blocked.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-31 10:25:27 +0100 (Tue, 31 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nIterates all buckets in the current AWS account and checks their Block Public Access settings\n\nExits with error code 2 if buckets are found which do not have this protection fully set,\nand exits with error code 1 if no buckets found\n\nSet the environment variable QUIET to any value to omit the header and summary warning lines\n\nParallelized to get through bucket list more quickly, set NOPARALLEL env var to any value to serialize for debugging\nFor 64 buckets parallelization took the runtime from 166-168 seconds down to 21-40 seconds\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nmax_args 1\n\nregion=\"${1:-}\"\nshift || :\n\nif [ -n \"$region\" ]; then\n    export AWS_DEFAULT_REGION=\"$region\"\nfi\n\nexport AWS_DEFAULT_OUTPUT=json\n\nnum_buckets=0\nnum_non_compliant_buckets=0\n\nparallelism=\"$(cpu_count)\"\nif [ \"$parallelism\" -gt 20 ]; then\n    # cap the parallelism to not spam AWS API and risk getting blocked\n    parallelism=10\nfi\n\nif [ -n \"${NOPARALLEL:-}\" ]; then\n    parallelism=1\nfi\n\nshopt -s nocasematch\n\nstart_time=\"$(date '+%s')\"\n\nmax_bucket_name_len=\"$(aws s3api list-buckets | jq -rM '.Buckets[].Name | length' | jq -srM 'max')\"\n# XXX: has to be exported for subshell parallel function below to access it\nexport format_string='%-25s\\t%-15s\\t'\"%-${max_bucket_name_len}s\"'\\t%-16s\\t%-17s\\t%-18s\\t%s\\n'\n\nif [ -z \"${QUIET:-}\" ] || is_piped; then\n    # false positive, the format string is carefully constructed\n    # shellcheck disable=SC2059\n    printf \"$format_string\" 'Creation Timestamp' Region Bucket BlockPublicAcls IgnorePublicAcls BlockPublicPolicy RestrictPublicBuckets >&2\nfi\n\ncommands=\"\"\n\nwhile read -r creation_timestamp bucket; do\n    commands+=\"\n    bucket_info '$bucket' '$creation_timestamp'\"\ndone < <(\n    aws s3api list-buckets |\n    jq -r '.Buckets[] | [.CreationDate, .Name] | @tsv'\n)\n\nbucket_info(){\n    [ -n \"${DEBUG:-}\" ] && set -x\n    local bucket=\"$1\"\n    local creation_timestamp=\"$2\"\n    local region\n    local policy\n    region=\"$(aws s3api get-bucket-location --bucket \"$bucket\" || :)\"\n    if [ -n \"$region\" ]; then\n        region=\"$(jq -r '.LocationConstraint' <<< \"$region\")\"\n    else\n        region=\"unknown\"\n        echo \"FAILED to get region for bucket '$bucket', skipping...\" >&2\n    fi\n    policy=\"$(aws s3api get-public-access-block --bucket \"$bucket\" 2>/dev/null || :)\"\n    if [ -n \"$policy\" ]; then\n        # XXX: must align with read command a few lines down\n        policy=\"$(jq -r '.PublicAccessBlockConfiguration | [ .BlockPublicAcls, .IgnorePublicAcls, .BlockPublicPolicy, .RestrictPublicBuckets ] | @tsv' <<< \"$policy\")\"\n    else\n        # Block Access Policy not set\n        policy=\"unset unset unset unset\"\n    fi\n    # XXX: must align with jq command a few lines up\n    read -r BlockPublicAcls IgnorePublicAcls BlockPublicPolicy RestrictPublicBuckets <<< \"$policy\"\n    # false positive, the format string is carefully constructed\n    # shellcheck disable=SC2059\n    # XXX: must align with read command in loop further down, and header line above\n    printf \"$format_string\" \"$creation_timestamp\" \"$region\" \"$bucket\" \"$BlockPublicAcls\" \"$IgnorePublicAcls\" \"$BlockPublicPolicy\" \"$RestrictPublicBuckets\"\n}\nexport -f bucket_info\n\n# XXX: must align with bucket_info() output\nwhile read -r creation_timestamp region bucket BlockPublicAcls IgnorePublicAcls BlockPublicPolicy RestrictPublicBuckets; do\n    # false positive, the format string is carefully constructed\n    # shellcheck disable=SC2059\n    printf \"$format_string\" \"$creation_timestamp\" \"$region\" \"$bucket\" \"$BlockPublicAcls\" \"$IgnorePublicAcls\" \"$BlockPublicPolicy\" \"$RestrictPublicBuckets\"\n    if [[ \"$BlockPublicAcls\" =~ false|unset ]] ||\n       [[ \"$IgnorePublicAcls\" =~ false|unset ]] ||\n       [[ \"$BlockPublicPolicy\" =~ false|unset ]] ||\n       [[ \"$RestrictPublicBuckets\" =~ false|unset ]]; then\n        ((num_non_compliant_buckets+=1))\n    fi\n    ((num_buckets+=1))\ndone < <(parallel -j \"$parallelism\" <<< \"$commands\")\n\nif [ -z \"${QUIET:-}\" ]; then\n    end_time=\"$(date +%s)\"\n    time_taken=\"$((end_time - start_time))\"\n    echo >&2\n    echo \"Time taken: $time_taken secs\" >&2\n    echo >&2\nfi\nif [ $num_buckets -eq 0 ]; then\n    echo \"WARNING: no buckets found\" >&2\n    exit 1\nelif [ $num_non_compliant_buckets -eq 0 ]; then\n    if [ -z \"${QUIET:-}\" ]; then\n        echo \"OK: All $num_buckets buckets found to be compliant blocking public access\" >&2\n    fi\nelse\n    echo \"WARNING: $num_non_compliant_buckets/$num_buckets buckets found without public access blocked!\" >&2\n    exit 2\nfi\n"
  },
  {
    "path": "aws/aws_s3_delete_bucket_with_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-03 09:47:12 +0700 (Tue, 03 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a bucket including all versions\n\nDANGER: this beats the safety mechanisms if you really want to delete some PoC bucket. Use with caution!\n\nRequires AWS CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\n#usage_args=\"[<aws_sso_args>]\"\nusage_args=\"<bucket_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nbucket=\"$1\"\nbucket=\"${bucket#s3://}\"\n\ntimestamp \"Getting object versions for bucket: $bucket\"\nobject_versions=\"$(\n    aws s3api list-object-versions \\\n        --bucket \"$bucket\" \\\n        --query 'Versions[].{Key: Key, VersionId: VersionId}' \\\n        --output text\n)\"\n\nif [ \"$object_versions\" != \"None\" ]; then\n    while read -r key versionId; do\n        timestamp \"Deleting object '$key' version '$versionId'\"\n        aws s3api delete-object --bucket \"$bucket\" --key \"$key\" --version-id \"$versionId\"\n    done <<< \"$object_versions\"\n    echo >&2\nfi\n\ntimestamp \"Getting object deletion markers for bucket: $bucket\"\nobject_deletion_markers=\"$(\n    aws s3api list-object-versions \\\n        --bucket \"$bucket\" \\\n        --query 'DeleteMarkers[].{Key: Key, VersionId: VersionId}' \\\n        --output text\n)\"\n\nif [ \"$object_deletion_markers\" != \"None\" ]; then\n    while read -r key versionId; do\n        timestamp \"Deleting object deletion marker '$key' version '$versionId'\"\n        aws s3api delete-object --bucket \"$bucket\" --key \"$key\" --version-id \"$versionId\"\n    done <<< \"$object_deletion_markers\"\n    echo >&2\nfi\n\ntimestamp \"Deleting bucket: $bucket\"\n#aws s3api delete-bucket --bucket \"$bucket\"\naws s3 rb \"s3://$bucket\"\n"
  },
  {
    "path": "aws/aws_s3_sync.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-05 12:18:36 +0200 (Thu, 05 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSyncs multiple AWS S3 URLs from file lists\n\nNeeded because clients often request copies of data ranges of multiple directories between environment buckets for test data\n\nFor multiple source and destinations specify text files containing the paths, one line per path\n\nFor convenience:\n\n- ignores hash # comment lines\n- strips leading and trailing whitespaces\n- validates each S3 URL's format\n- validates the source and destination list lengths are the same\n- validates each source and destination path suffix is the same\n  - can disable this by 'export AWS_S3_SYNC_DIFFERENT_PATHS=true' before running this script if you really intend for\n    the destination paths to be different to the source paths\n\nThese last two checks help prevent off-by-one human errors missing one path and spraying data to the wrong directories\n\nYou can populate the source and destination path files using native Bash like this:\n\n    echo s3://prod-landing-bucket/transactions/2023-06-{20..30} | tr ' ' '\\n' > sources.txt\n\n    echo s3://uat-landing-bucket/transactions/2023-06-{20..30} | tr ' ' '\\n' > destinations.txt\n\n\nConsider adding the --dryrun option to the end of the script args when running it the first time\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<sources.txt> <destinations.txt> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nlog=\"aws_s3_sync-$(date '+%F_%H.%M.%S').log\"\n\nsources_file=\"$1\"\ndestinations_file=\"$2\"\nshift || :\nshift || :\n\nsources=()\ndestinations=()\n\ndecomment(){\n    sed '\n        s/#.*$//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d\n    ' \"$1\"\n}\n\nvalidate_s3_url(){\n    local url=\"$1\"\n    if ! is_s3_url \"$url\"; then\n        die \"Invalid S3 URL given: $url\"\n    fi\n}\n\n# initially deduplicated this to a load_file() function but it turns out mapfile is only Bash 4+\n# and Bash 3 has no native array passing, requiring array pass-by-name string and ugly evals\nif ! [ -f \"$sources_file\" ]; then\n    die \"File not found: $sources_file\"\nfi\n\ntimestamp \"Capturing log to: $log\"\n{\n\ntimestamp \"Loading sources from file '$sources_file'\"\nwhile IFS= read -r line; do\n    validate_s3_url \"$line\"\n    sources+=(\"$line\")\ndone < <(decomment \"$sources_file\")\nsources_len=\"${#sources[@]}\"\ntimestamp \"$sources_len sources loaded\"\necho\n\nif ! [ -f \"$destinations_file\" ]; then\n    die \"File not found: $destinations_file\"\nfi\ntimestamp \"Loading destinations from file '$destinations_file'\"\nwhile IFS= read -r line; do\n    validate_s3_url \"$line\"\n    destinations+=(\"$line\")\ndone < <(decomment \"$destinations_file\")\ndestinations_len=\"${#destinations[@]}\"\ntimestamp \"$destinations_len destinations loaded\"\necho\n\ntimestamp \"Sanity check: Verifying source and destination list lengths are the same\"\nif [ \"$sources_len\" != \"$destinations_len\" ]; then\n    die \"ERROR: length of sources and destinations arrays of paths are not equal in length: sources ($sources_len) vs destinations ($destinations_len)\"\nfi\n\nif [ \"${AWS_S3_SYNC_DIFFERENT_PATHS:-}\" != true ]; then\n    timestamp \"Sanity check: Verifying source and destination suffix paths are the same\"\n    for ((i=0; i < sources_len; i++)); do\n        src=\"${sources[i]}\"\n        dest=\"${destinations[i]}\"\n        src_path=\"${src#s3://}\"\n        src_path=\"${src_path#*/}\"\n        dest_path=\"${dest#s3://}\"\n        dest_path=\"${dest_path#*/}\"\n        if [ \"$src_path\" != \"$dest_path\" ]; then\n            echo\n            error \"Source path suffix '$src' does not match destination path suffix '$dest'\"\n            echo\n            die \"If this is really intentional, 'export AWS_S3_SYNC_DIFFERENT_PATHS=true' before running this script\"\n        fi\n    done\n    echo\nfi\n\nfor ((i=0; i < sources_len; i++)); do\n    src=\"${sources[i]}\"\n    dest=\"${destinations[i]}\"\n\n    timestamp \"Syncing AWS S3 '$src' to '$dest'\"\n    aws s3 sync \"$src\" \"$dest\" \"$@\"\ndone\necho\n# we've already verified above that $sources_len and $destination_len are the same\ntimestamp \"AWS S3 Sync completed for $sources_len S3 URL paths\"\n\n} 2>&1 |\n# aws s3 sync seems to output \\r messing up the log lines\ntr '\\r' '\\n' |\ntee -a \"$log\"\n"
  },
  {
    "path": "aws/aws_secret_add.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads a value from the command line and saves it to AWS Secrets Manager without echo'ing it on the screen\n\nFirst argument is used as secret name\nSecond argument is used as secret string value if it doesn't start with option swtich double dashes -- and a letter\n    - if this argument is a file, such as an SSH key, reads the file content and saves it as the secret value\n    - if not given prompts for it with a non-echo'ing prompt (recommended for passwords)\nRemaining args are passed directly to 'aws secretsmanager'\n\nExamples:\n\n    ${0##*/} myname\n\n    ${0##*/} myname myvalue\n\n    ${0##*/} myname myvalue --description 'test credential'\n\n    # For accessing in Jenkins via https://plugins.jenkins.io/aws-secrets-manager-credentials-provider/\n    ${0##*/} mysecretstring --tags 'Key=jenkins:credentials:type,Value=string'\n    ${0##*/} mypass         --tags 'Key=jenkins:credentials:type,Value=usernamePassword'  'Key=jenkins:credentials:username,Value=hari'\n    ${0##*/} mysshkey       --tags 'Key=jenkins:credentials:type,Value=sshUserPrivateKey' 'Key=jenkins:credentials:username,Value=hari'\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<secret> --description 'blah' <aws_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\n# perhaps somebody wants a secret value starting with a dash\nif ! [[ \"${2:-}\" =~ ^--[[:alpha:]]+ ]]; then\n    secret=\"${2:-}\"\n    shift || :\nfi\nshift || :\n\nif [ -z \"${secret:-}\" ]; then\n    echo \"Secret not given as second arg\"\n    read_secret\nfi\n\nif [ -f \"$secret\" ]; then\n    secret=\"$(cat \"$secret\")\"\nfi\n\naws secretsmanager create-secret --name \"$name\" --secret-string \"$secret\" \"$@\"\n"
  },
  {
    "path": "aws/aws_secret_add_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds a given binary file to AWS Secrets Manager as base64 - this is only possible via the CLI or SDK as it's not supported in the AWS Console UI at this time\n\nFirst argument is used as secret name\nSecond argument must be a binary file such as a QR Code screenshot - this is converted to base 64 because AWS only permits ASCII characters in this value\nRemaining args are passed directly to 'aws secretsmanager'\n\nExample:\n\n    ${0##*/} mysecret qr-code-screenshot.png\n\n# To retrieve the binary file back:\n\n    aws_secret_get.sh mysecret | base64 --decode > qr-code.png\n\n\nCaveat: your QR pic cannot be too big or complex or it'll result in too large a base64 string which gives this error:\n\n    An error occurred (ValidationException) when calling the CreateSecret operation: 1 validation error detected: Value at 'secretBinary' failed to satisfy constraint: Member must have length less than or equal to 65536\n\nWorkaround: instead of screenshotting, save the QR code png via right-click download, it'll result in a much smaller original png download which will be able to be saved to AWS Secrets Manager\nWarning: this has been tested and works to instantiate other team members virtual MFA on their phones for GitHub.com, even a year later, but Azure AD seems to expire the QR code\n         https://stackoverflow.com/questions/73578781/qr-code-got-expire-with-azure-verifiable-credential\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> <file> [--description 'QR Code for GitHub Account' <aws_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nname=\"$1\"\nfile=\"$2\"\nshift || :\nshift || :\n\nif ! [ -f \"$file\" ]; then\n    die \"File not found: $file\"\nfi\n\naws secretsmanager create-secret --name \"$name\" --secret-binary \"$(base64 \"$file\")\" \"$@\"\n"
  },
  {
    "path": "aws/aws_secret_get.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRetrieves a secret value from a given AWS Secrets Manager secret name\n\nFirst argument is used as secret name\nRemaining args are passed directly to 'aws secretsmanager'\n\nWill check for and output Secret String or Secret Binary\n\nExample:\n\n    ${0##*/} my-secret\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\nshift || :\n\naws secretsmanager get-secret-value --secret-id \"$name\" \"$@\" |\njq -r 'if .SecretString then .SecretString else .SecretBinary end'\n"
  },
  {
    "path": "aws/aws_secret_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists AWS Secrets Manager secrets, one per line\n\nAny args are passed directly to 'aws secretsmanager'\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_options>]\"\n\nhelp_usage \"$@\"\n\naws secretsmanager list-secrets \"$@\" |\njq_debug_pipe_dump |\njq -r '.SecretList[].Name'\n"
  },
  {
    "path": "aws/aws_secret_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads a value from the command line without echo'ing it on the screen and updates the given AWS Secrets Manager secret\n\nFirst argument is used as secret name\nSecond argument is used as secret string value if it doesn't start with option swtich double dashes -- and a letter\n    - if this argument is a file, such as an SSH key, reads the file content and saves it as the secret value\n    - if not given prompts for it with a non-echo'ing prompt (recommended for passwords)\nRemaining args are passed directly to 'aws secretsmanager'\n\nExamples:\n\n    ${0##*/} myname\n\n    ${0##*/} myname myvalue\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<secret> --description 'My changed description' <aws_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\n# perhaps somebody wants a secret value starting with a dash\nif ! [[ \"${2:-}\" =~ ^--[[:alpha:]]+ ]]; then\n    secret=\"${2:-}\"\n    shift || :\nfi\nshift || :\n\nif [ -z \"$secret\" ]; then\n    read_secret\nfi\n\nif [ -f \"$secret\" ]; then\n    secret=\"$(cat \"$secret\")\"\nfi\n\n# put-secret doesn't allow changing the --description or other details\naws secretsmanager update-secret --secret-id \"$name\" --secret-string \"$secret\" \"$@\"\n"
  },
  {
    "path": "aws/aws_secret_update_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a given binary file as base64 to an existing AWS Secrets Manager secret - this is only possible via the CLI or SDK as it's not supported in the AWS Console UI at this time\n\nFirst argument is used as secret name\nSecond argument must be a binary file such as a QR Code screenshot - this is converted to base 64 because AWS only permits ASCII characters in this value\nRemaining args are passed directly to 'aws secretsmanager'\n\nExample:\n\n    ${0##*/} mysecret qr-code-screenshot.png\n\n# To retrieve the binary file back:\n\n    aws_secret_get.sh mysecret | base64 --decode > qr-code.png\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> <file> [--description 'My changed description' <aws_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nname=\"$1\"\nfile=\"$2\"\nshift || :\nshift || :\n\nif ! [ -f \"$file\" ]; then\n    die \"File not found: $file\"\nfi\n\n# put-secret doesn't allow changing the --description or other details\naws secretsmanager update-secret --secret-id \"$name\" --secret-binary \"$(base64 \"$file\")\" \"$@\"\n"
  },
  {
    "path": "aws/aws_spot_when_terminated.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2015\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-07 14:25:06 +0000 (Thu, 07 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://aws.amazon.com/blogs/aws/new-ec2-spot-instance-termination-notices/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# intentionally not using /lib so that this script is standalone and easier to distribute to VMs rather than requiring a full git clone of the repo\nusage(){\n    cat <<EOF\nExecutes the arguments as shell commands when the EC2 instance running this script is notified of Spot Termination\n\nCan be used as a latch mechanism to wait before allowing a shell or script to proceed past calling this script\n\nUsage:\n\n ${0##*/} \"command1; command2; command 3\"        All commands execute in a subshell\n\n ${0##*/}; command 1; command2; command 3        All commands execute in current shell\n                                                                     (notice the semi-colon immediately after the script giving it no argument, merely using it as a latch mechanism)\n\n ${0##*/} command1; command2; command 3          First command executes in the subshell (it's an argument). Commands 2 & 3 execute in the current shell after this script\n\n ${0##*/} 'x=test; echo \\$x'                      Variable is interpolated inside the single quotes at runtime after receiving Spot Termination notice\n\n\nInspired by whenup() / whendown() for hosts and whendone() for processes from interactive bash library .bash.d/* sourced as part of the .bashrc in this repo\n\nYou can trigger this as part of rc.local or similar on an EC2 Spot instance and when it gets the termination notice it'll execute all of its arguments as commands\n\nFor GCP there is a similar adjacent script gce_when_preempted.sh\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\nif ! curl -sS --connect-timeout 2 http://169.254.169.254/ &>/dev/null; then\n    echo \"This script must be run from within an EC2 instance as that is the only place the AWS EC2 Metadata API is available\"\n    exit 2\nfi\n\ntermination_time=\"\"\n\nwhile true; do\n    # regex borrowed from AWS Systems Administration book by O'Reilly and also here:\n    # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-interruptions.html\n    termination_time=\"$(curl -s --max-time 1 http://169.254.169.254/latest/meta-data/spot/termination-time | grep '.*T.*Z' || :)\"\n    if [ -n \"$termination_time\" ]; then\n        break\n    fi\n    echo -n '.'\n    # AWS recommended check interval\n    sleep 5\ndone\n\necho \"Termination Time:  $termination_time\"\necho \"Executing:  $*\"\n# eval'ing so that \"command1; command2\" works as well as 'x=test; echo $x'\neval \"$@\"\n"
  },
  {
    "path": "aws/aws_sqs_check.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-16 19:16:41 +0000 (Wed, 16 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSends a message to a given SQS queue, fetches it and deletes it\n\nQueue URL argument can be copied from SQS queue page and should look similar to:\n\n    https://sqs.<region>.amazonaws.com/<account_number>/myname.fifo\n\n    eg.\n\n    https://sqs.\\$AWS_DEFAULT_REGION.amazonaws.com/\\$AWS_ACCOUNT_ID/myname.fifo\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<queue_url>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nqueue_url=\"$1\"\n\nepoch=\"$(date '+%s')\"\nmessage_group_id=\"${0##*/}\"\nmessage_deduplication_id=\"${0##*/}_$epoch\"\nmessage_body=\"message body from script ${0##*/} pid $$ epoch $epoch\"\n\ntimestamp \"sending SQS test message to queue '$queue_url'\"\naws sqs send-message --queue-url \"$queue_url\" --message-body \"$message_body\" --message-group-id \"$message_group_id\" --message-deduplication-id \"$message_deduplication_id\"\necho >&2\n\nsleep 1\n\ntimestamp \"receiving SQS messages from queue '$queue_url'\"\nmessage_json=\"$(aws sqs receive-message --queue-url \"$queue_url\" --max-number-of-messages=10)\"\necho >&2\n\ntimestamp \"parsing messages for receipt handle\"\nreceipt_handle=\"$(jq -r \".Messages[] | select(.Body == \\\"$message_body\\\") | .ReceiptHandle\" <<< \"$message_json\")\"\necho >&2\n\nif [ -z \"$receipt_handle\" ]; then\n    die \"Message handle not found for message with body '$message_body' in queue '$queue_url'\"\nfi\n\ntimestamp \"deleting test message using receipt handle\"\naws sqs delete-message --queue-url \"$queue_url\" --receipt-handle \"$receipt_handle\"\n"
  },
  {
    "path": "aws/aws_sqs_delete_messages.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-16 19:36:19 +0000 (Wed, 16 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes 1-10 AWS SQS messages from a given SQS queue URL\n\nDefaults to 1 message, max 10 messages\n\n\nQueue URL argument can be copied from SQS queue page and should look similar to:\n\n    https://sqs.<region>.amazonaws.com/<account_number>/myname.fifo\n\n    eg.\n\n    https://sqs.\\$AWS_DEFAULT_REGION.amazonaws.com/\\$AWS_ACCOUNT_ID/myname.fifo\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<queue_url> [<num_messages>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nqueue_url=\"$1\"\nnum_messages=\"${2:-1}\"\n\naws sqs receive-message --queue-url \"$queue_url\" --max-number-of-messages=\"$num_messages\" |\njq -r '.Messages[] | .ReceiptHandle' |\nwhile read -r receipt_handle; do\n    timestamp \"deleting message\"\n    aws sqs delete-message --queue-url \"$queue_url\" --receipt-handle \"$receipt_handle\"\ndone\n"
  },
  {
    "path": "aws/aws_ssm_put_param.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-06 11:40:22 +0000 (Mon, 06 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads a value from the command line and saves it to AWS Systems Manager Parameter Store\n\nfirst argument is used as key - if not given prompts for it\nsecond argument is used as value - if not given prompts for it (recommended for secrets)\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<key> [<value>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nkey=\"$1\"\nsecret=\"${2:-}\"\n\nif [ -z \"$secret\" ]; then\n    read_secret value\nfi\n\naws ssm put-parameter --name \"$key\" --value \"$secret\" --type SecureString --overwrite\n"
  },
  {
    "path": "aws/aws_ssm_wait_for_command.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-31 16:26:34 +0700 (Fri, 31 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPolls an AWS SSM Command invocation and waits for it to finish before returning\n\nUsed by adjacent script aws_eks_ami_create.sh\n\nGet the AWS SSM Command ID from the output of the 'aws ssm send-command' - see above script\n\nTimeout secs defaults to 300 if not specified\nCheck interval defaults to 5 seconds if not specified\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ssm_command_id> [<timeout_secs> <check_interval>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nid=\"$1\"\n\ntimeout_secs=\"${2:-300}\"\n\ncheck_interval_secs=\"${3:-5}\"\n\nif ! is_int \"$timeout_secs\"; then\n\tdie \"Invalid Timeout Secs, must be an integer: $timeout_secs\"\nfi\n\nif ! is_int \"$check_interval_secs\"; then\n\tdie \"Invalid Check Interval Secs, must be an integer: $check_interval_secs\"\nfi\n\nif [ \"$timeout_secs\" -lt 1 ]; then\n\tdie \"Invalid Timeout Secs cannot be less than 1: $timeout_secs\"\nfi\n\nif [ \"$check_interval_secs\" -lt 1 ]; then\n\tdie \"Invalid Check Interval Secs cannot be less than 1: $check_interval_secs\"\nfi\n\ntimestamp \"Timeout max: $timeout_secs secs\"\ntimestamp \"Check interval: $check_interval_secs secs\"\ntimestamp \"Waiting for AWS SSM Command execution to finish: $id\"\n\nSECONDS=0\n\nwhile true; do\n    if [ \"$SECONDS\" -gt \"$timeout_secs\" ]; then\n        die \"ERROR: Timed out waiting $timeout_secs for command to complete\"\n    fi\n    timestamp \"Checking SSM Command status\"\n    status=\"$(\n        aws ssm list-command-invocations \\\n            --command-id \"$id\" \\\n            --query \"CommandInvocations[0].Status\" \\\n            --output text\n    )\"\n\n    if [ \"$status\" = \"Success\" ]; then\n        timestamp \"AWS SSM Command completed successfully\"\n        break\n    elif [ \"$status\" = \"Failed\" ] ||\n         [ \"$status\" = \"TimedOut\" ] ||\n         [ \"$status\" = \"Cancelled\" ]; then\n        die \"ERROR: AWS SSM Command execution failed with status: $status\"\n    else\n        timestamp \"AWS SSM Command still running...\"\n        sleep \"$check_interval_secs\"\n    fi\ndone\n"
  },
  {
    "path": "aws/aws_sso_account_id_names.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-03 17:38:53 +0700 (Mon, 03 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses your \\$AWS_CONFIG_FILE ini config for AWS SSO to output AWS Account IDs and Profile Names\n\nOutput:\n\n<account_id>  <profile_name>\n\n\nThis is one of the most rudimentary scripts in this repo, so it could break if you have a non-uniform\nAWS config file - you must check the results yourself. If you've generated your AWS SSO config using the\nadjacent scripts then it should be fine:\n\n    aws_sso_configs.sh\n\n    aws_sso_configs_save.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<[aws_config_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\naws_config=\"${1:-${AWS_CONFIG_FILE:-$(cd && pwd)/.aws/config}}\"\n\n#grep -e '^\\[profile' \\\n#     -e '^[[:space:]]*sso_account_id' \\\n#    \"$aws_config\" |\n#sed '\n#    N;\n#    s/^[[:space:]]*#.*//;\n#    /^[[:space:]]*$/d;\n#    s/\\[profile //;\n#    s/\\n/ /;\n#    s/^\\([^ ]*\\) \\(.*\\)$/\\2  # \\1/\n#    s/^[[:space:]]*sso_account_id[[:space:]]*=[[:space:]]*//;\n#    s/\\][[:space:]]*$//;\n#'\n\nawk '\n  /^\\[profile / {\n    profile = gensub(/^\\[profile (.*)]$/, \"\\\\1\", 1)\n  }\n  /^[[:space:]]*sso_account_id/ {\n    gsub(/^[[:space:]]*sso_account_id = /, \"\", $0)\n    print $0, profile\n  }\n' \"$aws_config\" |\ncolumn -t\n"
  },
  {
    "path": "aws/aws_sso_accounts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:19:53 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all the AWS SSO accounts you have access to\n\nRequires you to already be logged in to AWS SSO in order to use the access token to list the accounts\n\nIf you are not currently authenticated, with prompt to log you in first\n\nOutput is tab-delimited:\n\n<account_id>    <root_account_email>    <account_name>\n\n\nThe account name is last because it may contain spaces and this is easier to\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\naws_sso_login_if_not_already\n\naccess_token=\"$(aws_sso_token)\"\n\n# awk preprocessing trick to not split the third column name which can contain spaces as then it'd looks weird\naws sso list-accounts --access-token \"$access_token\" |\njq -r '.accountList[] | [.accountId, .emailAddress, .accountName] | @tsv' |\nsort -fuk 3 |\nawk '{printf \"%s\\t%s\\t\", $1, $2; $1=\"\"; $2=\"\"; print}' |\ncolumn -t -s $'\\t'\n"
  },
  {
    "path": "aws/aws_sso_accounts_missing_from_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-04 22:01:06 +0700 (Tue, 04 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a list of AWS Account IDs in stdin or files (containing one account id per line),\nfinds those in AWS config missing from the provided list\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files>]\"\n\nhelp_usage \"$@\"\n\n#min_args 0 \"$@\"\n\nHOME=\"${HOME:-$(cd && pwd)}\"\n\naws_config=\"${AWS_CONFIG_FILE:-$HOME/.aws/config}\"\n\n# force functions to log with timestamps\nexport VERBOSE=1\n\naws_account_ids=\"$(\n    sed '\n        s/#.*//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d\n    ' \"$@\"\n)\"\ntimestamp \"Validating input AWS Accoutn IDs\"\nwhile read -r aws_account_id; do\n    if ! is_aws_account_id \"$aws_account_id\"; then\n        warn \"Invalid AWS Account ID given: $aws_account_id\"\n    fi\ndone <<< \"$aws_account_ids\"\necho >&2\n\ntimestamp \"AWS Accounts missing from provided list vs config: $aws_config\"\necho >&2\nprofile=\"\"\nwhile read -r line; do\n    if [[ \"$line\" =~ ^[[:space:]]*\\[[[:space:]]*profile[[:space:]]+(.+)\\] ]]; then\n        profile=\"${BASH_REMATCH[1]}\"\n        continue\n    elif [[ \"$line\" =~ ^[[:space:]]*sso_account_id[[:space:]]*=[[:space:]]*([[:digit:]]+) ]]; then\n        aws_sso_account_id=\"${BASH_REMATCH[1]}\"\n    else\n        continue\n    fi\n    if ! is_aws_account_id \"$aws_sso_account_id\"; then\n        warn \"Invalid AWS Account ID in config: $aws_sso_account_id\"\n    fi\n    if ! grep -Fxq \"$aws_sso_account_id\" <<< \"$aws_account_ids\"; then\n        echo \"$profile  $aws_sso_account_id\"\n    fi\ndone < <(sed 's/#.*//; /^[[:space:]]*$/d' < \"$aws_config\") |\ncolumn -t\n\necho >&2\ntimestamp \"Done\"\n"
  },
  {
    "path": "aws/aws_sso_cache_expires.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-01 00:30:52 +0700 (Sat, 01 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds when the recent AWS SSO cache expires\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n# shellcheck disable=SC2012\ncreds=\"$(ls -t ~/.aws/sso/cache/* 2>/dev/null | head -n1)\"\n\nlog \"Latest AWS SSO cache file found is: $creds\"\n\n# in case HOME isn't set\n[ -n \"${HOME:-}\" ] || HOME=~\n\nif is_mac; then\n    mod_time=\"$(stat -f %m \"$creds\")\"\nelse  # Linux\n    mod_time=\"$(stat -c %Y \"$creds\")\"\nfi\n\ncurrent_time=\"$(date +%s)\"\n\nseconds_ago=\"$((current_time - mod_time))\"\n\nif [ \"$seconds_ago\" -gt 86400 ]; then\n    echo \"AWS SSO cache expired, $seconds_ago old\"\n    exit 1\nfi\n\nlog \"Checking if AWS SSO is already logged in\"\nif ! is_aws_sso_logged_in; then\n    echo \"AWS SSO cache expired\"\nelse\n    #jq -r '.expiresAt' \"$creds\"\n    #\n    # There is a weird situation where the newest file cache is expired earlier today\n    #\n    # but I have another older one with a timestamp far in the future (more than the usual 1 day of AWS CLI)\n    #\n    # and the AWS CLI is still working, so this is more accurate than the above which shows the\n    #\n    # expired cache from earlier today which is clearly not taking precedence\n    #\n    #   2025-01-31T16:53:34Z\n    #   2025-04-09T10:08:04Z\n    #\n    # this is more accurate in this weird situation\n    jq -r '.expiresAt' ~/.aws/sso/cache/*.json | sort -r | head -n1\nfi\n"
  },
  {
    "path": "aws/aws_sso_config_duplicate_profile_names.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:30:58 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate AWS SSO config profile names that are using the same sso_account_id\n\nUseful to find and remove / comment out an ~/.aws/config with a mix of hand crafted\nand automatically generated AWS SSO configs\n\nYou can override the config file location by setting environment variable AWS_CONFIG_FILE\n\nSee also:\n\n    aws_sso_config_duplicate_sections.sh\n\n    aws_sso_configs.sh - iterates and generates AWS SSO configs for all accounts your currently authenticated user has access to\n\n    aws_sso_configs_save.sh - saves each the above generated configs to ~/.aws/config if they don't already exist by profile name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nconfig=\"${1:-${AWS_CONFIG_FILE:-$HOME/.aws/config}}\"\n\nif ! [ -f \"$config\" ]; then\n    die \"ERROR: file does not exist: $config\"\nfi\n\nduplicate_account_ids=\"$(\n    grep '^[[:space:]]*sso_account_id[[:space:]]*=' \"$config\" |\n    sed 's/.*=[[:space:]]*//' |\n    sort |\n    uniq -d |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nwhile read -r account_id; do\n    if is_blank \"$account_id\"; then\n        continue\n    fi\n    if ! is_int \"$account_id\"; then\n        die \"ERROR: detected invalid AWS Account ID: $account_id\"\n    fi\n    while read -r line; do\n        if is_blank \"$line\"; then\n            continue\n        elif [[ \"$line\" =~ ^[[:space:]]*\\[.+\\] ]]; then\n            section=\"$line\"\n        elif [[ \"$line\" =~ ^[[:space:]]*sso_account_id[[:space:]]*=[[:space:]]*${account_id}[[:space:]]*$ ]]; then\n            echo \"$section\"\n        fi\n    done < \"$config\"\ndone <<< \"$duplicate_account_ids\" |\nsed '\n    s/^[^[]*\\[//;\n    s/^[[:space:]]*profile[[:space:]]*//;\n    s/\\][^]]*$//;\n' |\nsort -fu\n"
  },
  {
    "path": "aws/aws_sso_config_duplicate_sections.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:30:58 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate AWS SSO config sections that are using the same sso_account_id\n\nUseful to find and remove / comment out an ~/.aws/config with a mix of hand crafted\nand automatically generated AWS SSO configs\n\nYou can override the config file location by setting environment variable AWS_CONFIG_FILE\n\nSee also:\n\n    aws_sso_config_duplicate_profile_names.sh\n\n    aws_sso_configs.sh - iterates and generates AWS SSO configs for all accounts your currently authenticated user has access to\n\n    aws_sso_configs_save.sh - saves each the above generated configs to ~/.aws/config if they don't already exist by profile name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nconfig=\"${1:-${AWS_CONFIG_FILE:-$HOME/.aws/config}}\"\n\nif ! [ -f \"$config\" ]; then\n    die \"ERROR: file does not exist: $config\"\nfi\n\nduplicate_account_ids=\"$(\n    grep '^[[:space:]]*sso_account_id[[:space:]]*=' \"$config\" |\n    sed 's/.*=[[:space:]]*//' |\n    sort |\n    uniq -d |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nsection=\"\"\n\nwhile read -r account_id; do\n    if is_blank \"$account_id\"; then\n        continue\n    fi\n    if ! is_int \"$account_id\"; then\n        die \"ERROR: detected invalid AWS Account ID: $account_id\"\n    fi\n    found=0\n    while read -r line; do\n        if is_blank \"$line\"; then\n            if [ \"$found\" = 1 ]; then\n                echo \"$section\"\n                section=\"\"\n                echo\n                found=0\n            fi\n            continue\n        elif [[ \"$line\" =~ ^[[:space:]]*\\[.+\\] ]]; then\n            section=\"$line\"\n        else\n            section+=\"\n$line\"\n            if [[ \"$line\" =~ ^[[:space:]]*sso_account_id[[:space:]]*=[[:space:]]*${account_id}[[:space:]]*$ ]]; then\n                found=1\n            fi\n        fi\n    done < \"$config\"\n    if [ \"$found\" = 1 ]; then\n        echo \"$section\"\n        section=\"\"\n        echo\n        found=0\n    fi\ndone <<< \"$duplicate_account_ids\"\n"
  },
  {
    "path": "aws/aws_sso_configs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:30:58 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates AWS SSO configs for all AWS SSO accounts the currently logged in user has access to\n\nRequires you to already be logged in to AWS SSO in order to use the access token to list the accounts\n\nIf you are not currently authenticated, with prompt to log you in first\n\n\nConfig contents:\n\n.   These assume you are using the same config for each SSO account. Edit the resulting config otherwise\n\n    Start URL - infers from current config\n\n    ROLE - infers from current role. Set AWS_DEFAULT_ROLE, AWS_ROLE or ROLE environment variables to override this in that order of precedence\n\n    REGION - infers from current config. Set the standard AWS environment variable AWS_DEFAULT_REGION, or alternatively AWS_REGION or REGION in that order of precedence, otherwise it will default to eu-west-1 if all of the previous inferences fail\n\n\nUses the adjacent script to get the AWS Account ID and Name for each SSO account:\n\n    aws_sso_accounts.sh\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport aws_default_region=\"eu-west-1\"\n\n# force functions to log with timestamps\nexport VERBOSE=1\n\naws_sso_login_if_not_already\n\naccess_token=\"$(aws_sso_token)\"\n\nrole=\"$(aws_sso_role)\"\necho >&2\n\nregion=\"$(aws_region_from_env)\"\n\nsso_start_url=\"$(aws_sso_start_url)\"\necho >&2\n\nsso_start_region=\"$(aws_sso_start_region)\"\necho >&2\n\ntimestamp \"Getting AWS SSO accounts this account has access to\"\n\"$srcdir/aws_sso_accounts.sh\" |\nwhile read -r id _email name; do\n    name=\"$(tr '[:upper:]' '[:lower:]' <<< \"$name\" | sed 's/[^[:alnum:]]/-/g')\"\n    timestamp \"Looking up available roles for account '$name' ($id)\"\n    roles=\"$(\n        aws sso list-account-roles \\\n            --account-id \"$id\" \\\n            --access-token \"$access_token\" \\\n            --query 'roleList[*].roleName' \\\n            --output text |\n        tr '[:space:]' '\\n'\n    )\"\n    if grep -Fxq \"$role\" <<< \"$roles\"; then\n        timestamp \"Role '$role' is available on account '$name' ($id), using it for config\"\n        sso_role=\"$role\"\n    else\n        timestamp \"Role '$role' not available on account '$name' ($id) - available roles:\"\n        echo >&2\n        echo \"$roles\" >&2\n        echo >&2\n        sso_role=\"$(head -n1 <<< \"$roles\")\"\n        timestamp \"Using first available role '$sso_role' for account '$name' ($id) - edit to another role if necessary\"\n    fi\n    echo >&2\n    cat <<EOF\n[profile $name]\nsso_start_url  = $sso_start_url\nsso_region     = $sso_start_region\nsso_account_id = $id\nsso_role_name  = $sso_role\nregion         = $region\n\nEOF\ndone\n"
  },
  {
    "path": "aws/aws_sso_configs_save.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 20:47:15 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSaves AWS SSO configs generated by aws_sso_configs.sh to ~/.aws/config if they're not already found\n\nSee aws_sso_configs.sh for more details on generating the AWS SSO configs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<aws_config_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n\"$srcdir/aws_sso_configs.sh\" |\n\"$srcdir/aws_profile_config_add_if_missing.sh\" \"$@\"\n"
  },
  {
    "path": "aws/aws_sso_env_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-27 18:23:44 +0100 (Wed, 27 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets AWS SSO session credentials as shell export commands for exporting into another system like Terraform Cloud\n\nExtracts the access token from the AWS SSO cache, then uses that to get credentials for the specified role.\n\nIf role is 'list', will list the available roles and exit so you can see what roles are valid to enter for that argument, eg. AWSAdministratorAccess or AWSPowerUserAccess\n\n\nYou must have already logged in first:\n\n    aws sso login\n\nYou may want to 'export AWS_PROFILE=...' if you have multiple logged in SSO profiles and want to select the right one.\n\nThis script will take the latest creds file from ~/.aws/sso/cache\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<role_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrole=\"$1\"\naccount_id=\"$(aws sts get-caller-identity --query Account --output text)\"\n\n#account_id=\"${AWS_ACCOUNT_ID:-}\"\n#\n#if [ -z \"$account_id\" ]; then\n#    if [ -n \"${AWS_PROFILE:-}\" ]; then\n#        timestamp \"attempting to infer account id from AWS Profile '$AWS_PROFILE'\"\n#        account_id=\"$(sed -n \"/^\\\\[profile $AWS_PROFILE\\\\]/,/sso_account_id/p\" \"${AWS_CONFIG_FILE:-~/.aws/config}\" | awk -F= '/sso_account_id/{print $2}')\"\n#    fi\n#fi\n#if [ -z \"$account_id\" ]; then\n#    min_args 2 \"$@\"\n#    account_id=\"$2\"\n#fi\n\n# shellcheck disable=SC2012\nlatest_cache=\"$(ls -tr ~/.aws/sso/cache/* | sed '/botocore/d' | tail -n 1)\"\n\nread -r region access_token < <(jq -r '[.region, .accessToken] | @tsv' < \"$latest_cache\")\n\nif [ \"$role\" = list ]; then\n    echo \"Roles available for account '$account_id':\"\n    aws sso list-account-roles --account-id \"$account_id\" --access-token \"$access_token\" |\n    jq -r '.roleList[].roleName'\n    exit 0\nfi\n\naws sso get-role-credentials --account-id \"$account_id\" --role-name \"$role\" --region \"$region\" --access-token \"$access_token\" |\njq -r '.roleCredentials | [.accessKeyId, .secretAccessKey, .sessionToken] | @tsv' |\nwhile read -r key secret token; do\n    echo \"export AWS_ACCESS_KEY_ID=$key\"\n    echo \"export AWS_SECRET_ACCESS_KEY=$secret\"\n    echo \"export AWS_SESSION_TOKEN=$token\"\ndone\n"
  },
  {
    "path": "aws/aws_sso_role_arn.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-03 09:47:12 +0700 (Tue, 03 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDetermines the currently authenticated AWS SSO user's base role ARN in IAM policy usable format\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\n#usage_args=\"[<aws_sso_args>]\"\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#num_args 0 \"$@\"\n\n#timestamp \"Checking if AWS SSO is already logged in\"\n#if is_aws_sso_logged_in; then\n#    timestamp \"Already authenticated to AWS SSO, skipping login for speed\"\n#else\n#    timestamp \"Not currently authenticated to AWS SSO, launching login:\"\n#    echo >&2\n#    aws sso login \"$@\"\n#    echo >&2\n#fi\n\nrole_sts=\"$(aws sts get-caller-identity --query Arn --output text)\"\n\n# replace assumed-role with just role\n# strip the /hari@domain.com user suffix which we don't use when referencing the role in all IAM policies\n# since it can be used by many users\n#role_sts_without_user=\"$(\n#    sed '\n#        s|^arn:aws:sts::|arn:aws:iam::|;\n#        s|:assumed-role/|:role/|;\n#        s|/[^/]*$||\n#    ' <<< \"$role_sts\"\n#)\"\nrole_sts_without_user=\"${role_sts%/*}\"\n\nrole_name=\"${role_sts_without_user##*/}\"\n\naws iam list-roles --query 'Roles[*].Arn' --output text | tr '[:space:]' '\\n' |\ngrep \"/$role_name$\"\n"
  },
  {
    "path": "aws/aws_sso_role_arns.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-03 09:47:12 +0700 (Tue, 03 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints all AWS SSO ARNs in IAM policy usable format\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\n#usage_args=\"[<aws_sso_args>]\"\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#num_args 0 \"$@\"\n\n#timestamp \"Checking if AWS SSO is already logged in\"\n#if is_aws_sso_logged_in; then\n#    timestamp \"Already authenticated to AWS SSO, skipping login for speed\"\n#else\n#    timestamp \"Not currently authenticated to AWS SSO, launching login:\"\n#    echo >&2\n#    aws sso login \"$@\"\n#    echo >&2\n#fi\n\naws iam list-roles --query 'Roles[*].Arn' --output text |\ntr '[:space:]' '\\n' |\ngrep '/aws-reserved/sso.amazonaws.com/'\n"
  },
  {
    "path": "aws/aws_sso_ssh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-28 20:19:46 +0700 (Thu, 28 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches local AWS SSO authentication pop-up (if not already authenticated)\n\nThen scp's the latest resultant ~/.aws/sso/cache/ file to the remote server\n\nAnd SSH's there so that you can use AWS CLI or kubectl to EKS remotely on that server easily,\nwithout having to copy and paste the token from remote aws sso login to your local web browser\n\nUseful in enviroments where only a bastion server can access EKS clusters or other AWS services\n\nSince all args go to AWS CLI, if you need to pass SSH options use the environment variable SSH_OPTIONS eg:\n\n    export SSH_OPTIONS=\\\"-i $HOME/.ssh/aws.pem -o StrictHostKeyChecking=no\\\"\n\nBest used when combined with automatically configuring your environment variables for the AWS_PROFILE etc.\nusing direnv or similar\n\nFor code on how to do that, see:\n\n    https://github.com/HariSekhon/Environments\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[user@]server [<aws_sso_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nserver=\"$1\"\nshift || :\n\ntimestamp \"Checking if AWS SSO is already logged in\"\nif is_aws_sso_logged_in; then\n    timestamp \"Already authenticated to AWS SSO, skipping login for speed\"\nelse\n    timestamp \"Not currently authenticated to AWS SSO, launching login:\"\n    echo >&2\n    aws sso login \"$@\"\n    echo >&2\nfi\n\n# in case HOME isn't set\n[ -n \"${HOME:-}\" ] || HOME=~\n\n# AWS SSO uses ~/.aws/sso/cache/ not old credentials file\n#creds=\"${AWS_SHARED_CREDENTIALS_FILE:-$HOME/.aws/credentials}\"\n\n# shellcheck disable=SC2012\ncreds=\"$(ls -t ~/.aws/sso/cache/* | head -n1)\"\n\ntimestamp \"Latest AWS SSO cache file found is: $creds\"\n\nif is_mac; then\n    mod_time=\"$(stat -f %m \"$creds\")\"\nelse  # Linux\n    mod_time=\"$(stat -c %Y \"$creds\")\"\nfi\n\ncurrent_time=\"$(date +%s)\"\n\nseconds_ago=\"$((current_time - mod_time))\"\n\nif [ \"$seconds_ago\" -gt 86400 ]; then\n    warn \"AWS credentials file '$creds' was last modified '$seconds_ago' seconds ago, not within the last 86400 seconds\"\n    warn \"(1 day - the max lifetime of an AWS SSO login cache)\"\n    warn \"ERROR: Not copying credentials file remotely for overwrite safety in case we have the wrong file\"\n    exit 1\nfi\n\n# shellcheck disable=SC2295\ncreds_homepath=\"${creds#$HOME}\"\ncreds_homepath=\"${creds_homepath##/}\"\ncreds_homepath=\"${creds_homepath%/*}\"\n\necho >&2\n\ntimestamp \"Ensuring remote path exists: $creds_homepath\"\n\n# the first time on a new EC2 VM this will fail without pre-creating the directories\n# want splitting and evaluation on client side\n# shellcheck disable=SC2086,SC2029\nssh ${SSH_OPTIONS:-} \"$server\" \"mkdir -pv $creds_homepath/\"\n\necho >&2\n\ntimestamp \"Copying AWS SSO credential cache to $server\"\n\n# want splitting\n# shellcheck disable=SC2086\nscp ${SSH_OPTIONS:-} \"$creds\" \"$server\":\"$creds_homepath/\"\n\necho >&2\n\ntimestamp \"SSH'ing to $server\"\n\n# want splitting\n# shellcheck disable=SC2086\n# the -q option suppressed the Motd from messing up my terminal\nexec ssh ${SSH_OPTIONS:-} \"$server\"\n"
  },
  {
    "path": "aws/aws_terraform_create_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-15 15:21:27 +0100 (Wed, 15 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Terraform CLI machine account, S3 bucket and DynamoDB locking table with policy for restricted accounts\n\nRequired:\n\n- account name      used to suffix S3 bucket, DynamoDB table and policy names\n\nOptional:\n\n- user name         (default: \\$USER-terraform)\n- group or policy   (default: Admins group if found, else AdministratorAccess standard AWS-managed policy)\n- keyfile           (default: ~/.aws/keys/\\${user}_\\${aws_account_id}_accessKeys.csv) - be careful if specifying this, a non-existent keyfile will create a new key, deleting the older of 2 existing keys if necessary to be able to create this\n\nExamples:\n\n    # create buckets, tables and user with all IAM policies for an account called 'myaccount' and a user called 'github-actions-myrepo'\n\n        ${0##*/} myaccount github-actions-terraform  # gets AdministratorAccess by default\n\n        ${0##*/} myaccount github-actions-terraform-plan ReadOnly\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<account_name> [<username> <group_or_policy> <keyfile>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 4 \"$@\"\n\naccount_name=\"$1\"\nuser=\"${2:-$USER-terraform}\"\ngroup_or_policy=\"${3:-}\"\nkeyfile=\"${4:-}\"\n\nbucket=\"terraform-state-$account_name\"\ntable=\"terraform-state-$account_name\"\n\n# must match the name generated in aws_terraform_create_dynamodb_table.sh\ndynamodb_policy=\"Terraform-DynamoDB-Lock-Table-$table\"\n\n\"$srcdir/aws_terraform_create_s3_bucket.sh\" \"$bucket\"\n\necho\n\n\"$srcdir/aws_terraform_create_dynamodb_table.sh\" \"$table\"\n\necho\n\ns3_policy=\"Terraform-S3-Bucket-$bucket\"\n\ntimestamp \"Generating policy document '$s3_policy'\"\ns3_policy_document=\"$(cat <<EOF\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"TerraformS3Bucket\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:GetObject\",\n                \"s3:PutObject\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::$bucket/*\"\n            ]\n        }\n    ]\n}\nEOF\n)\"\n\ntimestamp \"Checking if policy '$s3_policy' exists\"\nset +o pipefail  # early termination from the pipeline will fail the check otherwise\nif aws iam list-policies | jq -r '.Policies[].PolicyName' | grep -Fxq \"$s3_policy\"; then\n    timestamp \"WARNING: policy '$s3_policy' already exists, not creating...\"\nelse\n    timestamp \"Creating Terraform S3 Policy '$s3_policy'\"\n    aws iam create-policy --policy-name \"$s3_policy\" --policy-document \"$s3_policy_document\"\n    timestamp \"Policy created\"\nfi\nset -o pipefail\necho\n\n\"$srcdir/aws_terraform_create_credential.sh\" \"$user\" \"$s3_policy,$dynamodb_policy\"${group_or_policy:+,\"$group_or_policy\"} ${keyfile:+\"$keyfile\"}\n"
  },
  {
    "path": "aws/aws_terraform_create_atlantis_role.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-02 04:27:26 +0700 (Mon, 02 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS IAM role called 'atlantis' with an optional suffix and attaches the Administrator policy\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<atlantis_role_suffix> <atlantis_aws_account_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nsuffix=\"${1:-}\"\n\naws_account_id=\"${2:-}\"\n\nrole_name=\"atlantis\"\n\nif [ -n \"$suffix\" ]; then\n\trole_name=\"$role_name-$suffix\"\nfi\n\nif [ -n \"$aws_account_id\" ]; then\n    if ! is_aws_account_id \"$aws_account_id\"; then\n        usage \"Invalid AWS account ID given for where Atlantis is running, failed reged validation: $aws_account_id\"\n    fi\nelse\n    aws_account_id=\"$(aws_account_id)\"\nfi\n\ntrust_policy=$(cat <<EOF\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\n      \"Effect\": \"Allow\",\n      \"Principal\": {\n        \"AWS\": \"arn:aws:iam::$aws_account_id:root\"\n      },\n      \"Action\": \"sts:AssumeRole\"\n    }\n  ]\n}\nEOF\n)\n\nif aws iam get-role --role-name \"$role_name\" >/dev/null 2>&1; then\n    timestamp \"Role '$role_name' already exists, skipping creation\"\n\texit 0\nfi\n\ntimestamp \"Creating IAM role '$role_name'...\"\naws iam create-role \\\n    --role-name \"$role_name\" \\\n    --assume-role-policy-document \"$trust_policy\" \\\n    --description \"Role for Atlantis with AdministratorAccess\"\necho >&2\n\ntimestamp \"Attaching AdministratorAccess policy to role '$role_name'...\"\naws iam attach-role-policy \\\n    --role-name \"$role_name\" \\\n    --policy-arn \"arn:aws:iam::aws:policy/AdministratorAccess\"\necho >&2\n\ntimestamp \"AWS IAM role '$role_name' created and AdministratorAccess policy attached\"\n"
  },
  {
    "path": "aws/aws_terraform_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-20 17:26:21 +0000 (Sat, 20 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS terraform service account for Terraform Cloud or other CI/CD systems to run terraform plan and apply\n\nGrants this service account Administator privileges in the current AWS account unless a different group or policy is specified as a second argument\n\nCreates an IAM access key (deleting an older unused key if necessary), writes a CSV just as the UI download would, and outputs both shell export commands and configuration in the format for copying to your AWS profile in ~/.aws/credentials\n\nThe following optional arguments can be given:\n\n- user name         (default: \\$USER-terraform)\n- group or policy   (default: Admins group if found, else AdministratorAccess standard AWS-managed policy)\n- keyfile           (default: ~/.aws/keys/\\${user}_\\${aws_account_id}_accessKeys.csv) - be careful if specifying this, a non-existent keyfile will create a new key, deleting the older of 2 existing keys if necessary to be able to create this\n\nIdempotent - safe to re-run, will skip creating a user that already exists or CSV export that already exists\n\nExamples:\n\n    # creates user 'github-actions-MYREPO' with full AdministratorAccess and saves the key in ~/.aws/keys/\n\n        ${0##*/} github-actions-MYREPO\n\n    # creates a read-only user 'github-actions-MYREPO-readonly' with ReadOnlyAccess standard AWS-managed policy attached for GitHub Actions environment secret that can be automatically used in Pull Request workflows without approval, saves key to ~/.aws/keys/\n\n        ${0##*/} github-actions-MYREPO-readonly ReadOnlyAccess\n\n\n$usage_aws_cli_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<username> <group_or_policy> <keyfile>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser=\"${1:-$USER-terraform}\"\nshift || :\n\n# done as part of aws_cli_create_credential.sh now\n#aws_account_id=\"$(aws sts get-caller-identity --query Account --output text)\"\n\n\"$srcdir/aws_cli_create_credential.sh\" \"$user\" \"$@\"  #  AdministratorAccess  \"$HOME/.aws/keys/${user}_${aws_account_id}_accessKeys.csv\" # default location now\n"
  },
  {
    "path": "aws/aws_terraform_create_dynamodb_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest-terraform-state-lock-table\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-27 17:36:29 +0100 (Fri, 27 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a DynamoDB table for Terraform state locking\n\nAlso creates a policy for a permission limited Terraform account to use to lock/unlock the table.\nThis is useful when creating a Read Only account for GitHub Actions environment secret for Pull Requests to not need workflow approval\n\nIdempotent - skips creation of table and policy if they already exist\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<table_name> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ntable=\"$1\"\nshift || :\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws_account_id=\"$(aws_account_id)\"\naws_region=\"$(aws_region)\"\n\ntimestamp \"Checking for existing DynamoDB table '$table'\"\nif aws dynamodb list-tables \"$@\" | jq -r '.TableNames[]' | grep -Fxq \"$table\"; then\n    timestamp \"WARNING: table '$table' already exists in region '$aws_region', not creating...\"\nelse\n    timestamp \"Creating Terraform DynamoDB table '$table' in region '$aws_region'\"\n    aws dynamodb create-table --table-name \"$table\" \\\n                              --key-schema AttributeName=LockID,KeyType=HASH \\\n                              --attribute-definitions AttributeName=LockID,AttributeType=S \\\n                              --billing-mode PAY_PER_REQUEST \\\n                              \"$@\"\n    timestamp \"DynamoDB table created\"\nfi\n\necho\n\npolicy=\"Terraform-DynamoDB-Lock-Table-$table\"\n\ntimestamp \"Generating policy document '$policy'\"\npolicy_document=\"$(cat <<EOF\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"TerraformLock\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"dynamodb:PutItem\",\n                \"dynamodb:DeleteItem\"\n            ],\n            \"Resource\": \"arn:aws:dynamodb:$aws_region:$aws_account_id:table/$table\"\n        }\n    ]\n}\nEOF\n)\"\n\ntimestamp \"Checking for existing policy\"\nset +o pipefail  # early termination from the pipeline will fail the check otherwise\nif aws iam list-policies | jq -r '.Policies[].PolicyName' | grep -Fxq \"$policy\"; then\n    timestamp \"WARNING: policy '$policy' already exists, not creating...\"\nelse\n    timestamp \"Creating Terraform DynamoDB Policy '$policy'\"\n    aws iam create-policy --policy-name \"$policy\" --policy-document \"$policy_document\"\n    timestamp \"DynamoDB policy created\"\nfi\n"
  },
  {
    "path": "aws/aws_terraform_create_s3_bucket.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest-terraform-state\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-27 18:03:32 +0100 (Fri, 27 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an S3 bucket for storing Terraform state with the following optimizations:\n\n- Enables Versioning\n- Enables Encryption\n- Enables Versioning\n- Enables Server Side Encryption\n- Creates Bucket Policy to lock out:\n  - Power Users role\n  - any additional given user/group/role ARNs (optional)\n\nIdempotent: skips bucket creation if already exists, applies versioning, encryption, and applies bucket policy if none exists of if \\$OVERWRITE_BUCKET_POLICY is set to any value\n\nRegion: will create the bucket in your configured region, to override locally set \\$AWS_DEFAULT_REGION\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<bucket_name> [<ARNs_to_block_access_from>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbucket=\"$1\"\nshift || :\n\ntimestamp \"Checking for Power User role\"\npower_user_arn=\"$(aws iam list-roles | jq -r '.Roles[].Arn' | grep -i AWSPowerUserAccess || :)\"\nif [ -n \"$power_user_arn\" ]; then\n    timestamp \"Power User role ARN found:  $power_user_arn\"\nfi\necho >&2\n\n\"$srcdir/aws_s3_bucket.sh\" \"$bucket\" \"$@\" ${power_user_arn:+\"$power_user_arn\"}\n"
  },
  {
    "path": "aws/aws_terraform_iam_grant_s3_dynamodb.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest-terraform-state-lock-table\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-15 17:57:17 +0100 (Wed, 15 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates IAM policies for S3 buckets and DynamoDB tables containing 'terraform-state' or 'tf-state' in their name\n\nAttaches these policies to the specified user, which must already exist (run aws_terraform_create_credential.sh first)\n\nNecessary for limited privilege CI/CD accounts such as GitHub Actions pull request Terraform Plan only workflows using a Read Only account\n\nIdempotent, skips creation of the policies if they already exist\n\n\n$usage_aws_cli_jq_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<user_name> [<aws_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nuser=\"$1\"\nshift || :\n\nexport AWS_DEFAULT_OUTPUT=json\n\naws_account_id=\"$(aws_account_id)\"\n\ndynamodb_policy=\"Terraform-DynamoDB-Lock-Tables\"\ns3_policy=\"Terraform-S3-Buckets\"\n\ntimestamp \"Generating S3 policy document '$s3_policy'\"\ns3_policy_document=\"$(cat <<EOF\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"TerraformS3Bucket\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"s3:GetObject\",\n                \"s3:PutObject\"\n            ],\n            \"Resource\": [\n                \"arn:aws:s3:::*tf-state*\",\n                \"arn:aws:s3:::*terraform-state*\"\n            ]\n        }\n    ]\n}\nEOF\n)\"\n\ntimestamp \"Generating DynamoDB policy document '$dynamodb_policy'\"\ndynamodb_policy_document=\"$(cat <<EOF\n{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": [\n        {\n            \"Sid\": \"TerraformLock\",\n            \"Effect\": \"Allow\",\n            \"Action\": [\n                \"dynamodb:PutItem\",\n                \"dynamodb:DeleteItem\"\n            ],\n            \"Resource\": [\n                \"arn:aws:dynamodb:*:$aws_account_id:table/*terraform-state*\",\n                \"arn:aws:dynamodb:*:$aws_account_id:table/*tf-state*\"\n            ]\n        }\n    ]\n}\nEOF\n)\"\n\necho\n\ncreate_policy(){\n    local policy=\"$1\"\n    local policy_document=\"$2\"\n    timestamp \"Checking if policy '$policy' exists\"\n    set +o pipefail  # early termination from the pipeline will fail the check otherwise\n    if aws iam list-policies | jq -r '.Policies[].PolicyName' | grep -Fxq \"$policy\"; then\n        timestamp \"WARNING: policy '$policy' already exists, not creating...\"\n    else\n        timestamp \"Creating Terraform policy '$policy'\"\n        aws iam create-policy --policy-name \"$policy\" --policy-document \"$policy_document\"\n        timestamp \"Policy created\"\n    fi\n    set -o pipefail\n    echo\n}\n\nattach_policy(){\n    local policy=\"$1\"\n    timestamp \"Attaching policy '$policy' to user '$user'\"\n    timestamp \"Determining ARN for policy '$policy'\"\n    policy_arn=\"$(aws iam list-policies | jq -r \".Policies[] | select(.PolicyName == \\\"$policy\\\") | .Arn\")\"\n    timestamp \"Determined policy ARN:  $policy_arn\"\n    timestamp \"Granting policy '$policy' permissions directly to user '$user' in account '$aws_account_id'\"\n    aws iam attach-user-policy --user-name \"$user\" --policy-arn \"$policy_arn\"\n    echo\n}\n\ncreate_policy \"$dynamodb_policy\" \"$dynamodb_policy_document\"\ncreate_policy \"$s3_policy\" \"$s3_policy_document\"\nattach_policy \"$dynamodb_policy\"\nattach_policy \"$s3_policy\"\n\ntimestamp \"Done\"\n"
  },
  {
    "path": "aws/eksctl_cluster.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 12:10:23 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets up a test AWS EKS cluster using eksctl with 3 worker nodes in a 1-4 node AutoScaling group\n\nTakes about 20 minutes - uses CloudFormation to first create a stack with an EKS cluster management plane, then another stack with a node group,\nand finally configures kubectl config with a context in the form of \\$email@\\$clustername.\\$region.eksctl.io\n\nEnvironment variables to configure:\n\nEKS_CLUSTER - default: 'test'\nEKS_VERSION - default: 1.21 - you should probably set this to the latest supported to avoid having to upgrade later\nAWS_DEFAULT_REGION - default: 'eu-west-2'\nAWS_ZONES - defaults to zones a, b and c in AWS_DEFAULT_REGION (eg. 'eu-west-2a,eu-west-2b,eu-west-2c') - may need to tweak them anyway to work around a lack of capacity in zones. Must match AWS_DEFAULT_REGION\n\nSee Also:\n\n    eksctl.yaml - in HariSekhon/Templates repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<cluster_name> <kubernetes_version> <region> <aws_zones>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif ! command -v eksctl &>/dev/null; then\n    \"$srcdir/../install/install_eksctl.sh\"\n    echo\nfi\n\nEKS_CLUSTER=\"${1:-${EKS_CLUSTER:-test}}\"\nEKS_VERSION=\"${2:-${EKS_VERSION:-1.21}}\"\n# set a default here as needed to infer zones if not set\nAWS_DEFAULT_REGION=\"${3:-${AWS_DEFAULT_REGION:-eu-west2}}\"\nAWS_ZONES=\"${4:-${AWS_DEFAULT_REGION}a,${AWS_DEFAULT_REGION}b,${AWS_DEFAULT_REGION}c}\"\n\n# shellcheck disable=SC2013\nfor zone in ${AWS_ZONES//,/ }; do\n    region=\"${zone::${#zone}-1}\"\n    if [ \"$region\" != \"$AWS_DEFAULT_REGION\" ]; then\n        usage \"invalid zone '$zone' given, must match region '$AWS_DEFAULT_REGION'\"\n    fi\ndone\n\n# cluster will be called \"eksctl-$name-cluster\", in this case \"eksctl-test-cluster\"\ntimestamp \"Creating AWS EKS cluster via eksctl\"\neksctl create cluster --name \"$EKS_CLUSTER\" \\\n                      --version \"$EKS_VERSION\" \\\n                      --region \"$AWS_DEFAULT_REGION\" \\\n                      --zones \"$AWS_ZONES\" \\\n                      --managed \\\n                      --nodegroup-name standard-workers \\\n                        --node-type t3.micro \\\n                        --nodes 3 \\\n                        --nodes-min 1 \\\n                        --nodes-max 4\n"
  },
  {
    "path": "azure-pipelines.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: Sun Feb 23 19:02:10 2020 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#  to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                   A z u r e   D e v O p s   P i p e l i n e\n# ============================================================================ #\n\n# https://aka.ms/yaml\n\ntrigger:\n  - master\n\nvariables:\n  # ubuntu version\n  os_version: '22.04'\n\npool:\n  # there is no /dev/stderr on this azure build!\n  #vmImage: 'ubuntu-latest'\n  #vmImage: 'ubuntu-22.04'\n  vmImage: 'ubuntu-$(os_version)'\n\n# unprivileged container without sudo, cannot install dependencies\n#container: ubuntu:22.04\n\nsteps:\n  - script: cat /etc/*-release\n    displayName: OS Release\n\n  # requires script as first key, otherwise parsing breaks with error message:  Unexpected value 'displayName'\n  - script: env | sort\n    displayName: Environment\n\n  # doesn't work in container due to unprivileged execution and lack of sudo\n  #- script: sudo apt-get update && sudo apt-get install -y git make\n  #  displayName: install git & make\n\n  #- script: make\n  #  displayName: build\n\n  # doesn't work in vmImage build due to lack of access to normal /dev/stderr device\n  # tee: /dev/stderr: No such device or address\n  #- script: make test\n  #  displayName: test\n\n  # hacky workaround to Azure Pipelines ubuntu environment limitations of unprivileged container and no /dev/stderr in vmImage :-(\n  - script: |\n      sudo docker run -v \"$PWD\":/code \"ubuntu:$(os_version)\" /bin/bash -c '\n        set -ex\n        cd /code\n        setup/ci_bootstrap.sh\n        if [ -x setup/ci_git_set_dir_safe.sh ]; then\n          setup/ci_git_set_dir_safe.sh\n        fi\n        make init\n        make ci test\n      '\n    displayName: docker build\n"
  },
  {
    "path": "azure_devops/azure_devops_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /{organization}/{project}/_apis/git/repositories  | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 17:58:16 +0100 (Fri, 23 Oct 2020)\n#\n#  https://azure_devops.com/harisekhon/bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Azure DevOps API\n\nAutomatically handles authentication via environment variables \\$AZURE_DEVOPS_USERNAME / \\$AZURE_DEVOPS_USER\nand \\$AZURE_DEVOPS_TOKEN / \\$AZURE_DEVOPS_PASSWORD\n\nCurrently authentication is done using HTTP basic auth which works for most endpoints according to the docs,\nexcept for account management, for which this code would have to be switched to OAuth\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\n# Set up a personal access token here:\n\n    https://dev.azure.com/\\$AZURE_DEVOPS_USER/_usersSettings/tokens\n        or\n    https://dev.azure.com/\\$AZURE_DEVOPS_ORGANIZATION/_usersSettings/tokens\n\n\n# API Reference:\n\n    https://docs.microsoft.com/en-us/rest/api/azure/devops/\n\n\nExamples:\n\n\n# List a user or organization's Azure DevOps repos:\n\n    ${0##*/} /{organization}/{project}/_apis/git/repositories  | jq .\n\n    ${0##*/} /harisekhon/GitHub/_apis/git/repositories  | jq .\n\n\n# List a user or organization's Azure DevOps Pipelines:\n\n    ${0##*/} /{username}/{project}/_apis/pipelines | jq .\n\n    ${0##*/} /harisekhon/GitHub/_apis/pipelines | jq .\n\n\n# Get a specific pipeline (has an href to the build yaml):\n\n    ${0##*/} /{username}/{project}/_apis/pipelines/{id} | jq .\n\n    ${0##*/} /harisekhon/GitHub/_apis/pipelines/1 | jq .\n\n\nFor convenience the following tokens in the form :token, <token>, {token} are replaced:\n\n\\$AZURE_DEVOPS_USERNAME / \\$AZURE_DEVOPS_USER:                         organization, owner, username, user\n\\$AZURE_DEVOPS_PROJECT or \\$PWD repo's project from remote url:        project\ninferred from \\$PWD repo's remote url:                                repo\n\nThese depend on the environment variables listed above being set or able to infer from local git repo remote urls\notherwise they are not replaced\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://dev.azure.com\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nuser=\"${AZURE_DEVOPS_USERNAME:-${AZURE_DEVOPS_USER:-}}\"\nif [ -z \"$user\" ]; then\n    user=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@dev\\.azure\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//;s/:.*//' || :)\"\n    # curl_auth.sh does this automatically\n    #if [ -z \"$user\" ]; then\n    #    user=\"${USERNAME:${USER:-}}\"\n    #fi\nfi\n\nproject=\"${AZURE_DEVOPS_PROJECT:-}\"\nif [ -z \"$project\" ]; then\n    project=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@dev\\.azure\\.com/{print $2; exit}' | sed 's/.*dev.azure.com\\(:v3\\)*\\/[^/]*\\///; s/\\/.*$//' || :)\"\n    # curl_auth.sh does this automatically\n    #if [ -z \"$user\" ]; then\n    #    user=\"${USERNAME:${USER:-}}\"\n    #fi\nfi\n\nPASSWORD=\"${AZURE_DEVOPS_PASSWORD:-${AZURE_DEVOPS_TOKEN:-}}\"\n\nif [ -z \"${PASSWORD:-}\" ]; then\n    PASSWORD=\"$(git remote -v | awk '/https:\\/\\/[[:alnum:]]+@azure_devops\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//')\"\nfi\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nfi\nexport PASSWORD\n\n#if [ -n \"${PASSWORD:-}\" ]; then\n#    echo \"using authenticated access\" >&2\n#fi\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path//$url_base}\"\nurl_path=\"${url_path##/}\"\n\n# for convenience of straight copying and pasting out of documentation pages\n\nrepo=$(git_repo | sed 's/.*\\///')\n\nif [ -n \"$user\" ]; then\n    url_path=\"${url_path/:organization/$user}\"\n    url_path=\"${url_path/<organization>/$user}\"\n    url_path=\"${url_path/\\{organization\\}/$user}\"\n    url_path=\"${url_path/:owner/$user}\"\n    url_path=\"${url_path/<owner>/$user}\"\n    url_path=\"${url_path/\\{owner\\}/$user}\"\n    url_path=\"${url_path/:username/$user}\"\n    url_path=\"${url_path/<username>/$user}\"\n    url_path=\"${url_path/\\{username\\}/$user}\"\n    url_path=\"${url_path/:user/$user}\"\n    url_path=\"${url_path/<user>/$user}\"\n    url_path=\"${url_path/\\{user\\}/$user}\"\nfi\nif [ -n \"$project\" ]; then\n    url_path=\"${url_path/:project/$project}\"\n    url_path=\"${url_path/<project>/$project}\"\n    url_path=\"${url_path/\\{project\\}/$project}\"\nfi\nurl_path=\"${url_path/:repo/$repo}\"\nurl_path=\"${url_path/<repo>/$repo}\"\nurl_path=\"${url_path/\\{repo\\}/$repo}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "azure_devops/azure_devops_disable_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-13 18:05:20 +0100 (Mon, 13 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.microsoft.com/en-us/rest/api/azure/devops/git/repositories/update?view=azure-devops-rest-6.1#disable-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables one or more given Azure DevOps repos (for after a migration to GitHub to prevent writes to the wrong server being left behind)\n\nFor authentication and other details see:\n\n    azure_devops_api.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> <project> <repo> [<repo2> <repo3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\norg=\"$1\"\nproject=\"$2\"\nshift || :\nshift || :\n\ndisable_repo(){\n    local repo=\"$1\"\n    local id\n    timestamp \"getting repo id for  Azure DevOps organization '$org' project '$project' repo '$repo'\"\n    set +euo pipefail\n    response=\"$(\"$srcdir/azure_devops_api.sh\" \"/$org/$project/_apis/git/repositories/$repo\")\"\n    # shellcheck disable=SC2181\n    if [ $? != 0 ]; then\n        timestamp \"repo not found or already disabled, skipping...\"\n        return 0\n    fi\n    set -euo pipefail\n    id=\"$(jq -e -r .id <<< \"$response\")\"\n    timestamp \"disabling Azure DevOps organization '$org' project '$project' repo '$repo'\"\n    \"$srcdir/azure_devops_api.sh\" \"/$org/$project/_apis/git/repositories/$id?api-version=6.1-preview.1\" -X PATCH -d '{\"isDisabled\": true}' |\n    jq -e -r '[ \"Disabled: \", .isDisabled ] | @tsv'\n}\n\nfor repo in \"$@\"; do\n    disable_repo \"$repo\"\ndone\n"
  },
  {
    "path": "azure_devops/azure_devops_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo user={user} org={org} project={project} name={name} repo={repo}\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-10 14:55:22 +0100 (Fri, 10 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each Azure DevOps repo\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the repo names and exit after the first iteration\n\n\\$AZURE_DEVOPS_ORGANIZATION / \\$AZURE_DEVOPS_USER - the user or organization to iterate the repos on\n\\$AZURE_DEVOPS_PROJECT - the Azure DevOps project to iterate the repo on\n\nThe command template replaces the following for convenience in each iteration:\n\n{organization}, {org} => the organization account being iterated\n{username}, {user}    => the user account being iterated\n{project}             => the project containing the repo\n{name}                => the repo name without the user prefix\n{repo}                => the repo name with the user prefix\n\neg.\n    ${0##*/} echo user={user} project={project} name={name} repo={repo}\n    ${0##*/} echo org={org}   project={project} name={name} repo={repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nuser=\"${AZURE_DEVOPS_USER:-}\"\nuser_or_org=\"${AZURE_DEVOPS_ORGANIZATION:-$user}\"\n\ncheck_env_defined AZURE_DEVOPS_PROJECT\nif is_blank \"$user_or_org\"; then\n    usage \"\\$AZURE_DEVOPS_ORGANIZATION / \\$AZURE_DEVOPS_USER is not defined\"\nfi\n\n\"$srcdir/azure_devops_api.sh\" \"/$user_or_org/$AZURE_DEVOPS_PROJECT/_apis/git/repositories\" |\njq -r '.value[].name' |\nsort |\nwhile read -r name; do\n    repo=\"$user_or_org/$name\"\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $repo\" >&2\n    echo \"# ============================================================================ #\" >&2\n    echo >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{username\\}/${user:-}}\")\n    cmd=(\"${cmd[@]//\\{user\\}/${user:-}}\")\n    cmd=(\"${cmd[@]//\\{organization\\}/${AZURE_DEVOPS_ORGANIZATION:-}}\")\n    cmd=(\"${cmd[@]//\\{org\\}/${AZURE_DEVOPS_ORGANIZATION:-}}\")\n    cmd=(\"${cmd[@]//\\{project\\}/${AZURE_DEVOPS_PROJECT:-}}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "azure_devops/azure_devops_to_github_migration.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-10 14:10:20 +0100 (Fri, 10 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMigrates Azure DevOps repos to GitHub\n\nIdempotent - will find missing repos and migrate any missing ones across\n\nIf specifying a single repo, can optionally rename the destination repo on GitHub\n\nRequirements:\n\n- Azure DevOps and GitHub credentials in your environment variables - see these adjacent scripts for details:\n\n    azure_devops_api.sh --help\n\n    github_api.sh --help\n\n  Azure DevOps organization/project and GitHub organization aren't case sensitive at the time of writing\n\n- Your Git SSH credentials should be set up in both Azure DevOps and GitHub with permissions to clone from Azure and push to GitHub\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<azure_devops_organization> <azure_devops_project> <github_organization> [<repo>] [<new_repo_name>]\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\nazure_devops_organization=\"$1\"\nazure_devops_project=\"$2\"\ngithub_organization=\"$3\"\nazure_repo=\"${4:-}\"\ngithub_repo=\"${5:-}\"\n\nmigrate_repo(){\n    local azure_repo=\"$1\"\n    local github_repo=\"${2:-$azure_repo}\"\n    # mutate naming convention here if required as part of the migration\n    #github_repo=\"${github_repo/old/new}\"\n    timestamp \"migrating '$azure_repo' -> '$github_repo'\"\n    if ! \"$srcdir/../github/github_api.sh\" \"/repos/$github_organization/$github_repo\" &>/dev/null; then\n        timestamp \"creating github repo '$github_repo'\"\n        \"$srcdir/../github/github_api.sh\" \"/orgs/$github_organization/repos\" -X POST -d \"{\\\"name\\\": \\\"$github_repo\\\", \\\"private\\\": true}\" >/dev/null\n    fi\n    # API seems to not update the size field for ages, just do the clone anyway\n    #if \"$srcdir/../github/github_api.sh\" \"/repos/$github_organization/$github_repo\" | jq -e '.size == 0' >/dev/null; then\n    #    timestamp \"repo is empty, cloning across\"\n        tmp=\"/tmp/azure_to_github_migration.$EUID\" #.$$\" - reuse the checkouts for now at the expense of potential race conditions\n        mkdir -p \"$tmp\"\n        pushd \"$tmp\" >/dev/null\n        if ! [ -d \"$azure_repo.git\" ]; then\n            git clone --mirror git@ssh.dev.azure.com:\"v3/$azure_devops_organization/$azure_devops_project/$azure_repo\"\n        fi\n        pushd \"$azure_repo.git\" >/dev/null\n        if ! git remotes | awk '{print $1}' | sort -u | grep -q '^github$'; then\n            timestamp \"adding github remote\"\n            git remote add github git@github.com:\"$github_organization/$github_repo.git\"\n        fi\n        git fetch --all\n        git push github --all\n        # handles situation where github has added commits to the main branch - only handles the 'main' branch, would be more complicated to figure out which branches\n        #if ! git push github --all; then\n        #    git --work-tree=\"$PWD\" checkout main\n        #    git --work-tree=\"$PWD\" pull github main\n        #    git push github --all\n        #fi\n        git push github --tags\n        popd >/dev/null\n        popd >/dev/null\n        timestamp \"getting azure repo default branch\"\n        default_branch=\"$(\"$srcdir/azure_devops_api.sh\" \"/$azure_devops_organization/$azure_devops_project/_apis/git/repositories/$azure_repo\" | jq -r '.defaultBranch' | sed 's/.*\\///')\"\n        timestamp \"setting github repo default branch to '$default_branch'\"\n        \"$srcdir/../github/github_api.sh\" \"/repos/$github_organization/$github_repo\" -X PATCH -d \"{\\\"default_branch\\\": \\\"$default_branch\\\"}\" >/dev/null\n        timestamp \"migrated '$azure_repo' -> '$github_repo'\"\n        echo >&2\n    #fi\n    echo >&2\n}\n\nif [ -n \"$azure_repo\" ]; then\n    migrate_repo \"$azure_repo\" \"$github_repo\"\nelse\n    timestamp \"fetching list of Azure DevOps repos in organization '$azure_devops_organization' project '$azure_devops_project'\"\n    azure_devops_repos=\"$(\"$srcdir/azure_devops_api.sh\" \"/$azure_devops_organization/$azure_devops_project/_apis/git/repositories\" | jq -r '.value[].name')\"\n    timestamp \"fetching list of GitHub repos in organization '$github_organization'\"\n    #github_repos=\"$(\"$srcdir/../github/github_api.sh\" \"/orgs/$github_organization/repos\" | jq -r '.[].name')\"\n    github_repos=\"$(get_github_repos \"$github_organization\" \"is_organization\")\"\n\n    # use GNU grep to avoid buggy Mac -f behaviour\n    if is_mac; then\n        grep(){\n            ggrep \"$@\"\n        }\n    fi\n\n    remaining_repos=\"$(grep -Fxvf <(echo \"$github_repos\") <<< \"$azure_devops_repos\")\"\n\n    # XXX: this will miss if there is any munging on repo naming and will go through the checks to create and populate the repo if empty\n    echo \"Azure DevOps repos missing on GitHub:\"\n    echo\n    echo \"$remaining_repos\"\n    echo\n    for azure_repo in $remaining_repos; do\n        migrate_repo \"$azure_repo\"\n    done\nfi\n"
  },
  {
    "path": "bigdata/beeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to more easily connect to HiveServer2 without having to specify the big JDBC connection string and all options like kerberos principal, ssl etc\n\n\nTested on Hive 1.1.0 on CDH 5.10\n\n\nUseful options for scripting:\n\n  --silent=true\n  --outputformat=tsv2     (tsv is deprecated and single quotes results, tsv2 is recommended and cleaner)\n\n\nSee adjacent hive_*.sh scripts for slightly better versions of these quick command line examples, including better escaping\n\n\nExamples:\n\n\n# List all databases:\n\n  ./beeline.sh --silent=true --outputformat=tsv2 -e 'show databases' | tail -n +2\n\n\n# List all tables in all databases:\n\n  opts=\\\"--silent=true --outputformat=tsv2\\\"; ./beeline.sh \\$opts -e 'show databases' | tail -n +2 | while read db; do ./beeline.sh \\$opts -e \\\"show tables from \\$db\\\" | sed \\\"s/^/\\$db./\\\"; done\n\n\n# Row counts of all tables in all databases:\n\n  opts=\\\"--silent=true --outputformat=tsv2\\\"; ./beeline.sh \\$opts -e 'show databases' | tail -n +2 | while read db; do ./beeline.sh \\$opts -e \\\"show tables from \\$db\\\" | sed \\\"s/^/\\$db./\\\"; done | tail -n +2 | while read table; do printf \\\"%s\\\\t\\\" \\\"\\$table\\\"; ./beeline.sh \\$opts -e \\\"select count(*) from \\$table\\\" | tail -n +2; done | tee row_counts_hive.tsv\n\n\nSee also:\n\n  https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Usinghive-site.xmltoautomaticallyconnecttoHiveServer2\n\n  hive_foreach_table.py / impala_foreach_table.py and similar tools in DevOps Python Tools repo - https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nif [ -n \"${HIVE_HA:-}\" ] ||\n   [ -n \"${HIVE_ZOOKEEPERS:-}\" ]; then\n    exec \"$srcdir/beeline_zk.sh\" \"$@\"\nfi\n\n# not listed in hive-site.xml on edge nodes nor https://github.com/apache/hive/blob/master/data/conf/hive-site.xml\n# must specify in your environment / .bashrc or similar\nif [ -z \"${HIVESERVER2_HOST:-}\" ]; then\n    echo \"HIVESERVER2_HOST environment variable not set\"\n    read -r -p \"Enter HiveServer2 address (FQDN): \" HIVESERVER2_HOST\nfi\n\nopts=\"\"\nif [ -n \"${BEELINE_OPTS:-}\" ]; then\n    opts=\"$opts;$BEELINE_OPTS\"\nfi\n\nset +o pipefail\n# xq -r < hive-site.xml '.configuration.property[] | select(.name == \"hive.server2.use.SSL\") | .value'\nif [ -n \"${HIVESERVER2_SSL:-}\" ] ||\n   grep -A1 'hive.server2.use.SSL' /etc/hive/conf/hive-site.xml 2>/dev/null |\n   grep -q true; then\n    opts=\"$opts;ssl=true\"\n    # works without this but enable if you need\n    #set +o pipefail\n    #trust_file=\"$(find /opt/cloudera/security/jks -maxdepth 1 -name '*-trust.jks' 2>/dev/null | head -n1)\"\n    #set -o pipefail\n    #if [ -f \"$trust_file\" ]; then\n    #    opts=\"$opts;sslTrustStore=$trust_file\"\n    #fi\nfi\n\nrealm=\"${HIVESERVER2_HOST#*.}\"\n\n[ -n \"${VERBOSE:-}\" ] && set -x\nexec beeline -u \"jdbc:hive2://$HIVESERVER2_HOST:10000/default;principal=hive/_HOST@${realm}${opts}\" \"$@\"\n"
  },
  {
    "path": "bigdata/beeline_zk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to more easily connect to HiveServer2 without having to specify the big connection string or look up the ZooKeepers\n\n# see more documentation in the header of the adjacent beeline.sh script\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nhive_site_xml=\"${HIVE_SITE_XML:-/etc/hive/conf/hive-site.xml}\"\n\nset +o pipefail\n# xq -r < hive-site.xml '.configuration.property[] | select(.name == \"hive.zookeeper.quorum\") | .value'\nif [ -z \"${HIVE_ZOOKEEPERS:-}\" ]; then\n    HIVE_ZOOKEEPERS=\"$(grep -A1 hive.zookeeper.quorum \"$hive_site_xml\" 2>/dev/null | grep '<value>' | sed 's/<value>//;s,</value>,,;s/[[:space:]]*//g')\"\n    if [ -z \"${HIVE_ZOOKEEPERS:-}\" ]; then\n        echo \"HIVE_ZOOKEEPERS environment variable not set and couldn't determine from $hive_site_xml (format is zookeeper1.domain.com:2181,zookeeper2.domain.com:2181,zookeeper3.domain.com:2181)\" >&2\n        exit 3\n    fi\nfi\n\n#if ! grep -q -e hive.server2.zookeeper.namespace \"$hive_site_xml\"; then\n#    echo \"Looks like HiveServer2 dynamic discovery is not configured in hive-site.xml, please use beeline.sh adjacent script and set \\$HIVESERVER2_HOST in your environment\"\n#    exit 1\n#fi\n\n# xq -r < hive-site.xml '.configuration.property[] | select(.name == \"hive.zookeeper.namespace\") | .value'\nif [ -z \"${HIVESERVER2_ZOOKEEPER_NAMESPACE:-}\" ]; then\n    # should be under /hiveserver2\n    # hive.zookeeper.namespace -> hive_zookeeper_namespace_hive is a red herring because it merely contains the db/table hierarchy and not the HS2 HA zk discovery info\n    HIVESERVER2_ZOOKEEPER_NAMESPACE=\"$(grep -A1 hive.server2.zookeeper.namespace \"$hive_site_xml\" 2>/dev/null | grep '<value>' | head -n 1 | sed 's/<value>//;s,</value>,,; s/[[:space:]]*//g')\"\n    HIVESERVER2_ZOOKEEPER_NAMESPACE=\"${HIVESERVER2_ZOOKEEPER_NAMESPACE:-hiveserver2}\"\nfi\n\nopts=\"\"\nif [ -n \"${BEELINE_OPTS:-}\" ]; then\n    opts=\"$opts;$BEELINE_OPTS\"\nfi\n\n# xq -r < hive-site.xml '.configuration.property[] | select(.name == \"hive.server2.use.SSL\") | .value'\nif [ -n \"${HIVESERVER2_SSL:-}\" ] ||\n   grep -A1 'hive.server2.use.SSL' /etc/hive/conf/hive-site.xml 2>/dev/null |\n   grep -q true; then\n    opts=\"$opts;ssl=true\"\n    # works without this but enable if you need\n    #set +o pipefail\n    #trust_file=\"$(find /opt/cloudera/security/jks -maxdepth 1 -name '*-trust.jks' 2>/dev/null | head -n1)\"\n    #set -o pipefail\n    #if [ -f \"$trust_file\" ]; then\n    #    opts=\"$opts;sslTrustStore=$trust_file\"\n    #fi\nfi\n\nset -x\nexec beeline -u \"jdbc:hive2://$HIVE_ZOOKEEPERS/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=${HIVESERVER2_ZOOKEEPER_NAMESPACE}${opts}\" \"$@\"\n"
  },
  {
    "path": "bigdata/cloudera_manager_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-02 16:19:20 +0000 (Thu, 02 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/cloudera_manager.sh\n. \"$srcdir/lib/cloudera_manager.sh\"\n\nusage(){\n    cat <<EOF\nScript to query Cloudera Manager API, auto-populating Cloudera Manager host address, cluster name from environment and\nsafely passing credentials via a file descriptor to avoid exposing them in the process list as arguments or OS logging\nhistory\n\nArguments are passed through to curl (eg. -k to not verify internal self-signed SSL certificate)\n\nCombine with jq commands to extract the info you want\n\nEnvironment variables (prompts for address, cluster and password if not passed via environment variables):\n\n\\$CLOUDERA_MANAGER_HOST / \\$CLOUDERA_MANAGER\n\\$CLOUDERA_MANAGER_CLUSTER / \\$CLOUDERA_CLUSTER\n\\$CLOUDERA_MANAGER_SSL (any value enables SSL and changes default port from 7180 to 7183)\n\\$CLOUDERA_MANAGER_USER / \\$CLOUDERA_USER / \\$USER\n\\$CLOUDERA_MANAGER_PASSWORD / \\$CLOUDERA_PASSWORD / \\$USER\n\n./cloudera_manager_api.sh /path\n\nUsed by various adjacent cloudera_manager_*.sh scripts\n\nTested on Cloudera Enterprise 5.10\nEOF\n    exit 3\n}\n\nif [ $# -lt 1 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -h|--help)  usage\n                    ;;\n    esac\ndone\n\nurl_path=\"$1\"\nurl_path=\"/${url_path##/}\"\n\n# remove $1 so we can pass remaining args to curl_auth.sh\nshift || :\n\n# https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cn_navigator_api_overview.html#api-version-compatility\napi_version=\"${CLOUDERA_API_VERSION:-7}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$CLOUDERA_MANAGER/api/v${api_version}${url_path}\" -sS --fail --connect-timeout 5 \"$@\"\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-02 16:19:20 +0000 (Thu, 02 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to fetch Impala queries from Cloudera Manager\n#\n# combine with jq commands to extract just the list of SQL queries from the rich json output\n#\n# ./cloudera_manager_impala_queries.sh | jq -r '.queries[].statement'\n\n# Tested on Cloudera Enterprise 5.10\n\n# Raw JSON output example:\n#\n# {\n#   \"queries\" : [ {\n#     \"queryId\" : \"1234f56ae78ff9:123d45f000000000\",\n#     \"statement\" : \"select count(*) from myDB.myTable -- somecomment\",  # might be just GET_SCHEMAS / GET_TABLES / USE myDB / DESCRIBE FORMATTED `myDB`.`myTable` / REFRESH myDB.myTable etc.\n#     \"queryType\" : \"QUERY\",\n#     \"queryState\" : \"FINISHED\",\n#     \"startTime\" : \"2020-01-02T15:03:44.746Z\",\n#     \"endTime\" : \"2020-01-02T15:33:13.886Z\",\n#     \"rowsProduced\" : null,\n#     \"attributes\" : {\n#       \"admission_result\" : \"Admitted immediately\",\n#       \"admission_wait\" : \"0\",\n#       \"bytes_streamed\" : \"224\",\n#       \"client_fetch_wait_time\" : \"1747483\",\n#       \"client_fetch_wait_time_percentage\" : \"99\",\n#       \"connected_user\" : \"hue\",\n#       \"delegated_user\" : \"hari\",\n#       \"estimated_per_node_peak_memory\" : \"10485760\",\n#       \"file_formats\" : \"PARQUET/NONE\",\n#       \"hdfs_average_scan_range\" : \"3898.6171543326227\",\n#       \"hdfs_bytes_read\" : \"221312800\",\n#       \"hdfs_bytes_read_from_cache\" : \"0\",\n#       \"hdfs_bytes_read_from_cache_percentage\" : \"0\",\n#       \"hdfs_bytes_read_local\" : \"221312800\",\n#       \"hdfs_bytes_read_local_percentage\" : \"100\",\n#       \"hdfs_bytes_read_remote\" : \"0\",\n#       \"hdfs_bytes_read_remote_percentage\" : \"0\",\n#       \"hdfs_bytes_read_short_circuit\" : \"221312800\",\n#       \"hdfs_bytes_read_short_circuit_percentage\" : \"100\",\n#       \"hdfs_scanner_average_bytes_read_per_second\" : \"5.663271900927363E7\",\n#       \"impala_version\" : \"impalad version 2.7.0-cdh5.10.0 RELEASE (build 785a073cd07e2540d521ecebb8b38161ccbd2aa2)\",\n#       \"memory_accrual\" : \"3.1359538E7\",\n#       \"memory_aggregate_peak\" : \"4781060.0\",\n#       \"memory_per_node_peak\" : \"1520435.2\",\n#       \"memory_per_node_peak_node\" : \"<fqdn>:22000\",\n#       \"memory_spilled\" : \"0\",\n#       \"network_address\" : \"<ip_x.x.x.x>:33447\",\n#       \"oom\" : \"false\",\n#       \"planning_wait_time\" : \"147\",\n#       \"planning_wait_time_percentage\" : \"0\",\n#       \"pool\" : \"root.hari_dot_sekhon\",\n#       \"query_status\" : \"OK\",\n#       \"session_id\" : \"12345f9f0eb1b323:18e29b3cb123456d\",\n#       \"session_type\" : \"HIVESERVER2\",\n#       \"stats_missing\" : \"false\",\n#       \"thread_network_receive_wait_time\" : \"21347\",\n#       \"thread_network_send_wait_time\" : \"0\",\n#       \"thread_storage_wait_time\" : \"1511999\"\n#     },\n#   }, {\n#     \"queryId\" : \"1b234f5cdf67890f:c1ad23e400000000\",\n#     \"statement\" : \"USE myDB\",\n#     \"queryType\" : \"DDL\",\n#     \"queryState\" : \"FINISHED\",\n#     \"startTime\" : \"2020-01-02T10:52:33.645Z\",\n#     \"endTime\" : \"2020-01-02T10:52:33.883Z\",\n#     \"rowsProduced\" : null,\n#     \"attributes\" : {\n#       \"admission_result\" : \"Unknown\",\n#       \"client_fetch_wait_time\" : \"11\",\n#       \"client_fetch_wait_time_percentage\" : \"5\",\n#       \"connected_user\" : \"<user>@<domain>\",\n#       \"ddl_type\" : \"USE\",\n#       \"file_formats\" : \"\",\n#       \"impala_version\" : \"impalad version 2.7.0-cdh5.10.0 RELEASE (build 785a073cd07e2540d521ecebb8b381)\",\n#       \"network_address\" : \"<ip_x.x.x.x>:36528\",\n#       \"oom\" : \"false\",\n#       \"original_user\" : \"<user>@<domain>\",\n#       \"planning_wait_time\" : \"203\",\n#       \"planning_wait_time_percentage\" : \"85\",\n#       \"query_status\" : \"OK\",\n#       \"session_id\" : \"1234567d89e01f23:12b34567890d1faf\",\n#       \"session_type\" : \"HIVESERVER2\"\n#       \"stats_missing\" : \"false\",\n#     },\n#     \"user\" : \"<user>\",\n#     \"coordinator\" : {\n#       \"hostId\" : \"e12fc3c4-5678-9f01-2345-678de90123fe\"\n#     },\n#     \"detailsAvailable\" : true,\n#     \"database\" : \"default\",\n#     \"durationMillis\" : 238\n#   ...\n#   } ],\n#   \"warnings\" : [ \"Impala query scan limit reached. Last end time considered is 2019-12-28T12:29:48.471Z\" ]\n# }\n\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/cloudera_manager.sh\n. \"$srcdir/lib/cloudera_manager.sh\"\n\n# defined in lib\n# shellcheck disable=SC2154\ntimestamp \"fetching queries up to now:  $now_timestamp\"\n\n\"$srcdir/cloudera_manager_api.sh\" \"/clusters/$CLOUDERA_CLUSTER/services/impala/impalaQueries?from=1970-01-01T00%3A00%3A00.000Z&to=$now_timestamp&filter=\"\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_ddl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala metadata refresh calls via Cloudera Manager API\n#\n# TSV output format:\n#\n# <time>    <database>  <user>      <statement>\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(.queryType == \"DDL\") |\n       select(.statement | test(\"^\\\\s*(USE|DESCRIBE|GET_TABLES|GET_SCHEMAS|REFRESH)\"; \"i\") | not) |\n       [.startTime, .database, .user, .statement] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_exceptions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala exceptions querievia Cloudera Manager API\n#\n# Tab-separated output format:\n#\n# <time>    <database>  <user>  <statement may be long>     <error>\n\n# Tested on Cloudera Enterprise 5.10\n\n# Caveat: this doesn't catch things like:\n#\n# \"query_status\" : \"\\nFailed to open HDFS file hdfs://nameservice1/user/hive/warehouse/<database>.db/<table>/part-r-00030-1a234567-8bc9-01d2-e345-678fa901b2c3.snappy.parquet\\nError(2): No such file or directory\\n\",\n#\n# because they have\n#\n# queryState\" : \"CREATED\" instead of queryState\" : \"EXCEPTION\"\n#\n# You can instead find those sorts of things using cloudera_manager_impala_queries_metadata_errors.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(.queryState == \"EXCEPTION\") |\n       [.startTime, .database, .user, .statement, .attributes.query_status] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_failed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala query metadata errors via Cloudera Manager API\n#\n# TSV output format:\n#\n# <time>    <database>      <user>      <query error>\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(.attributes.query_status | test(\"failed\"; \"i\")) |\n       [.startTime, .database, .user, .statement, .attributes.query_status] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_metadata.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala metadata refresh calls via Cloudera Manager API\n#\n# TSV output format:\n#\n# <time>    <database>  <user>      <statement>\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(\n            (.attributes.ddl_type == \"RESET_METADATA\")\n            or\n            (.attributes.query_status | test(\"metadata|No such file or directory\"; \"i\"))\n       ) |\n       select(.statement | test(\"^(SELECT|INSERT|UPDATE|DELETE)\"; \"i\") | not) |\n       [.startTime, .database, .user, .statement] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_metadata_errors.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala query metadata errors via Cloudera Manager API\n#\n# TSV output format:\n#\n# <time>    <database>      <user>      <query error>\n\n# Tested on Cloudera Enterprise 5.10\n\n# If encountering race conditions in metadata updates between Impalad daemons connected through a load balancer, you may be interested in setting\n#\n# SET SYNC_DDL=1;\n#\n# in your Impala session before issuing DDL statements or INSERTS into partitioned tables, see\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(.attributes.query_status | test(\"metadata|No such file or directory\"; \"i\")) |\n       [.startTime, .database, .user, .attributes.query_status] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_impala_queries_metadata_refresh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 11:51:09 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to show recent Impala metadata refresh calls via Cloudera Manager API\n#\n# TSV output format:\n#\n# <time>    <database>  <user>      <statement>\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_impala_queries.sh\" |\njq -r '.queries[] |\n       select(.attributes.ddl_type == \"RESET_METADATA\") |\n       [.startTime, .database, .user, .statement] |\n       @tsv'\n"
  },
  {
    "path": "bigdata/cloudera_manager_yarn_apps.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 15:08:10 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to fetch Yarn apps from Cloudera Manager\n#\n# combine with jq commands to extract just the list of SQL queries from the rich json output\n#\n# ./cloudera_manager_yarn_apps.sh | jq -r '.applications[]'\n\n# Tested on Cloudera Enterprise 5.10\n\n# Raw JSON output example:\n#\n# {\n#   \"applications\" : [ {\n#     \"applicationId\" : \"application_1571433934948_123456\",\n#     \"name\" : \"HdfsReplication\",\n#     \"startTime\" : \"2020-01-19T18:17:52.257Z\",\n#     \"user\" : \"hive\",\n#     \"pool\" : \"root.users.hive\",\n#     \"state\" : \"RUNNING\",\n#     \"progress\" : 95.0,\n#     \"attributes\" : {\n#       \"failed_map_attempts\" : \"0\",\n#       \"failed_reduce_attempts\" : \"0\",\n#       \"failed_tasks_attempts\" : \"0\",\n#       \"killed_map_attempts\" : \"0\",\n#       \"killed_reduce_attempts\" : \"0\",\n#       \"killed_tasks_attempts\" : \"0\",\n#       \"map_progress\" : \"100.0\",\n#       \"maps_completed\" : \"20\",\n#       \"maps_pending\" : \"0\",\n#       \"maps_running\" : \"0\",\n#       \"maps_total\" : \"20\",\n#       \"new_map_attempts\" : \"0\",\n#       \"new_reduce_attempts\" : \"0\",\n#       \"new_tasks_attempts\" : \"0\",\n#       \"reduce_progress\" : \"100.0\",\n#       \"reduces_completed\" : \"0\",\n#       \"reduces_pending\" : \"0\",\n#       \"reduces_running\" : \"1\",\n#       \"reduces_total\" : \"1\",\n#       \"running_application_info_retrieval_time\" : \"8.97\",\n#       \"running_map_attempts\" : \"0\",\n#       \"running_reduce_attempts\" : \"1\",\n#       \"running_tasks_attempts\" : \"1\",\n#       \"successful_map_attempts\" : \"20\",\n#       \"successful_reduce_attempts\" : \"0\",\n#       \"successful_tasks_attempts\" : \"20\"\n#       \"tasks_completed\" : \"20\",\n#       \"tasks_pending\" : \"0\",\n#       \"tasks_running\" : \"1\",\n#       \"total_task_num\" : \"21\",\n#       \"tracking_url\" : \"https://<fqdn>:8090/proxy/application_1571433934948_123456/\",\n#       \"uberized\" : \"false\",\n#     },\n#     \"mr2AppInformation\" : { }\n#   }, {\n#     \"applicationId\" : \"application_1571433934948_123456\",\n#     \"name\" : \"oozie:launcher:T=shell:W=blah/blah:A=myuser-node:ID=0009168-200118163518563-oozie-oozi-W\",\n#     \"startTime\" : \"2020-01-23T14:40:48.442Z\",\n#     \"user\" : \"myuser\",\n#     \"pool\" : \"root.stream\",\n#     \"state\" : \"RUNNING\",\n#     \"progress\" : 95.0,\n#     \"attributes\" : {\n#       \"failed_map_attempts\" : \"0\",\n#       \"failed_reduce_attempts\" : \"0\",\n#       \"failed_tasks_attempts\" : \"0\",\n#       \"killed_map_attempts\" : \"0\",\n#       \"killed_reduce_attempts\" : \"0\",\n#       \"killed_tasks_attempts\" : \"0\",\n#       \"map_progress\" : \"100.0\",\n#       \"maps_completed\" : \"0\",\n#       \"maps_pending\" : \"0\",\n#       \"maps_running\" : \"1\",\n#       \"maps_total\" : \"1\",\n#       \"new_map_attempts\" : \"0\",\n#       \"new_reduce_attempts\" : \"0\",\n#       \"new_tasks_attempts\" : \"0\",\n#       \"reduce_progress\" : \"0.0\",\n#       \"reduces_completed\" : \"0\",\n#       \"reduces_pending\" : \"0\",\n#       \"reduces_running\" : \"0\",\n#       \"reduces_total\" : \"0\",\n#       \"running_application_info_retrieval_time\" : \"0.008\",\n#       \"running_map_attempts\" : \"1\",\n#       \"running_reduce_attempts\" : \"0\",\n#       \"running_tasks_attempts\" : \"1\",\n#       \"successful_map_attempts\" : \"0\",\n#       \"successful_reduce_attempts\" : \"0\",\n#       \"successful_tasks_attempts\" : \"0\"\n#       \"tasks_completed\" : \"0\",\n#       \"tasks_pending\" : \"0\",\n#       \"tasks_running\" : \"1\",\n#       \"total_task_num\" : \"1\",\n#       \"tracking_url\" : \"https://<fqdn>:8090/proxy/application_1571433934948_123456/\",\n#       \"uberized\" : \"false\",\n#     },\n#     \"mr2AppInformation\" : { }\n#   },\n#   ...\n#  {\n#     \"applicationId\" : \"application_1571433934948_123456\",\n#     \"name\" : \"my_biz_process.py\",\n#     \"startTime\" : \"2020-01-23T09:26:56.878Z\",\n#     \"endTime\" : \"2020-01-23T09:29:50.705Z\",\n#     \"user\" : \"myETLuser\",\n#     \"pool\" : \"root.users.myETLuser\",\n#     \"state\" : \"FINISHED\",\n#     \"progress\" : 100.0,\n#     \"attributes\" : { },\n#     \"mr2AppInformation\" : { }\n#   } ],\n# }\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/cloudera_manager.sh\n. \"$srcdir/lib/cloudera_manager.sh\"\n\n# defined in lib\n# shellcheck disable=SC2154\ntimestamp \"fetching queries up to now:  $now_timestamp\"\n\n\"$srcdir/cloudera_manager_api.sh\" \"/clusters/$CLOUDERA_CLUSTER/services/yarn/yarnApplications?from=1970-01-01T00%3A00%3A00.000Z&to=$now_timestamp&filter=\"\n"
  },
  {
    "path": "bigdata/cloudera_manager_yarn_apps_failed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 16:13:06 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to fetch failed Yarn Apps / jobs via Cloudera Manager API\n#\n# Dumps the raw JSON for further processing, see cloudera_manager_yarn_apps.sh for the format\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/cloudera_manager_yarn_apps.sh\" |\njq -r '.applications[] | select(.state | test(\"RUNNING|SUCCEEDED|FINISHED|ACCEPTED\") | not)'\n"
  },
  {
    "path": "bigdata/cloudera_navigator_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-04 09:56:42 +0000 (Wed, 04 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/navigator_data_mgmt.html#xd_583c10bfdbd326ba-7dae4aa6-147c30d0933--7b44\n\n# https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/data_mgmt_audit_reports.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/cloudera_navigator.sh\n. \"$srcdir/lib/cloudera_navigator.sh\"\n\nusage(){\n    cat <<EOF\nScript to query Cloudera Navigator API, auto-populating Cloudera Navigator host address, cluster name from environment and\nsafely passing credentials via a file descriptor to avoid exposing them in the process list as arguments or OS logging\nhistory\n\nArguments are passed through to curl (eg. -k to not verify internal self-signed SSL certificate)\n\nCombine with jq commands to extract the info you want\n\nEnvironment variables (prompts for address and password if not passed via environment variables):\n\n\\$CLOUDERA_NAVIGATOR_HOST / \\$CLOUDERA_NAVIGATOR\n\\$CLOUDERA_NAVIGATOR_SSL (any value enables SSL and changes default port from 7186 to 7187)\n\\$CLOUDERA_NAVIGATOR_USER / \\$CLOUDERA_USER / \\$USER\n\\$CLOUDERA_NAVIGATOR_PASSWORD / \\$CLOUDERA_PASSWORD / \\$USER\n\n    ./cloudera_navigator_api.sh /path\n\nUsed by various adjacent cloudera_navigator_*.sh scripts\n\nTested on Cloudera Enterprise 5.10\nEOF\n    exit 3\n}\n\nif [ $# -lt 1 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -h|--help)  usage\n                    ;;\n    esac\ndone\n\nurl_path=\"$1\"\nurl_path=\"/${url_path##/}\"\n\n# remove $1 so we can pass remaining args to curl_auth.sh\nshift || :\n\napi_version=\"${CLOUDERA_API_VERSION:-10}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$CLOUDERA_NAVIGATOR/api/v${api_version}${url_path}\" -sS --fail --connect-timeout 5 \"$@\"\n"
  },
  {
    "path": "bigdata/cloudera_navigator_audit_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-02 16:19:20 +0000 (Thu, 02 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to fetch Cloudera Navigator Audit logs via API\n#\n# See cloudera_navigator_api.sh for base options like Navigator Host, SSL etc\n#\n# I've managed to crash Navigator several times both via the API and the UI trying to get access to > 1 years of historical logs\n# even after increasing the heap by several GB, so I don't recommend you run more than one of these scripts at a time, and\n# try to time bound it to a 1 year interval each time so it is more likely to succeed and less range to restart. I've written an\n# adjacent script called cloudera_navigator_audit_download_logs.sh to manage iterating years and retrying where needed\n#\n# Tested on Cloudera Enterprise 5.10\n\n\n# See the inline documentation for Cloudera Navigator Query filters\n#\n# https://$CLOUDERA_NAVIGATOR_HOST:7187/api-console/index.html#!/audits/getAudits\n#\n# https://$CLOUDERA_NAVIGATOR_HOST:7187/api-console/tutorial.html\n\n# Usage:\n#\n#   ./cloudera_navigator_audit_logs.sh <start_date> <end_date> <query_filter> <curl_options> ...\n\n# Examples:\n#\n# All logs up to now:\n#\n#   ./cloudera_navigator_audit_logs.sh <query> ... > navigator_audit_log.csv\n#\n#\n# Last year of Impala queries (literally today minus 1 year right down to the second):\n#\n#   ./cloudera_navigator_audit_logs.sh \"1 year ago\" service==impala ... > navigator_audit_log_year.csv\n#\n#\n# All Privilege Grants up to now:\n#\n#   ./cloudera_navigator_audit_logs.sh command==GRANT_PRIVILEGE > navigator_audit_log_grants.csv\n#\n#   ./cloudera_navigator_audit_logs.sh command==REVOKE_PRIVILEGE > navigator_audit_log_revokes.csv\n#\n#\n# From Start to End Dates, all hive queries in 2019:\n#\n#   ./cloudera_navigator_audit_logs.sh \"2019-01-01T00:00:00\" \"2020-01-01T00:00:00\" service==hive ... > navigator_audit_log_hive_2019.csv\n#\n#\n# All logs up to now for the Impala service, ignoring the self-signed certificate:\n#\n#   ./cloudera_navigator_audit_logs.sh service==impala -k > navigator_audit_log_impala.csv\n#\n#\n# Since this can easily take an hour or two per year of logs to download, you may want to add progress dots like so:\n#\n#   ./cloudera_navigator_audit_logs.sh service==impala -k | ../bin/progress_dots.sh > navigator_audit_log_impala.csv\n#\n#\n# or if you want full curl interactive progress on stderr:\n#\n#   PROGRESS=1 ./cloudera_navigator_audit_logs.sh service==impala -k > navigator_audit_log_impala.csv\n#\n#\n# XXX: looks like there is a bug in the Navigator API returning only admin commands, not data access, for when start date set to 1970-01-01T00:00:00 - workaround is to use 1970-01-01T00:00:01\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/cloudera_navigator.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/cloudera_navigator.sh\n. \"$srcdir/lib/cloudera_navigator.sh\"\n\n# to use Linux's date -d switch\nif is_mac; then\n    date=\"gdate\"\nelse\n    date=\"date\"\nfi\n\nstart=\"\"\nend=\"\"\n\nif [[ \"${1:-}\" =~ ^[[:digit:]] ]]; then\n    start=\"$1\"\n    shift\nfi\n\nif [[ \"${1:-}\" =~ ^[[:digit:]] ]]; then\n    end=\"$1\"\n    shift\nfi\n\nif [ -z \"$start\" ]; then\n    #start=\"1 year ago\"\n    # XXX: this causes Navigator API to return only admin commands and not SQL queries... weird\n    #start=\"1970-01-01T00:00:00\"\n    # looks like a bug, workaround:\n    start=\"1970-01-01T00:00:01\"\nfi\nstart_epoch_ms=\"$(\"$date\" --utc -d \"$start\" +%s000)\"\n\nif [ -z \"$end\" ]; then\n    end_epoch_ms=\"$now_timestamp\"\nelse\n    end_epoch_ms=\"$(\"$date\" --utc -d \"$end\" +%s000)\"\nfi\n\nstart_date=\"$($date --utc -d \"@${start_epoch_ms%000}\")\"\nend_date=\"$($date --utc -d \"@${end_epoch_ms%000}\")\"\n\n# defined in lib\n# shellcheck disable=SC2154\necho \"fetching audit logs from  '$start_date'  to  '$end_date'\" >&2\n\nquery=\"\"\nif ! [[ \"${1:-}\" =~ ^- ]]; then\n    query=\"${1:-}\"\n    shift\nfi\n\n# don't page through this, dump as whole attachment\nlimit=\"${limit:-10000}\" # max limit\noffset=\"${offset:-0}\"\n\n# CSV format seems to default to attachment=true, ignoring limits and offsets, even when attachment=false\n\n# default in API is JSON\n#format=\"${format:-JSON}\"  # or CSV\nformat=CSV  # only way to get all the records\n\n# attachment will ignore default 10,000 limit and return all results which is what we want - seems to not work on JSON, use CSV format instead, which also seems to ignore limit & offset even with attachment=false\n#\"$srcdir/cloudera_navigator_api.sh\" \"/audits/?query=${query}&startTime=${start_epoch_ms}&endTime=${end_epoch_ms}&format=${format}&limit=$limit&offset=$offset&attachment=false\" \"$@\"\n\"$srcdir/cloudera_navigator_api.sh\" \"/audits/?query=${query}&startTime=${start_epoch_ms}&endTime=${end_epoch_ms}&format=${format}&attachment=true\" \"$@\"\n"
  },
  {
    "path": "bigdata/cloudera_navigator_audit_logs_download.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 12:03:19 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to download all historical audit logs from Cloudera Navigator from 2009 to present\n#\n# 2009 was Cloudera's founding year so we don't search for history past that since it can never exist\n#\n# Uses adjacent cloudera_manager_audit.sh, see comments there for more details\n#\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\nsource \"$srcdir/lib/utils.sh\"\n\ntrap 'tstamp ERROR' exit\n\nservices=\"\nhive\nimpala\nhdfs\nhbase\nscm\n\"\n\n# slighty better compression but takes forever, even slow to decompress\n#compress_cmd=\"bzip2 -9 -c\"\n#compress_cmd=\"gzip -9 -c\"\n#ext=\"gz\"\n\nif [[ \"${1:-}\" =~ ^service== ]]; then\n    single_service=\"${1##service==}\"\n    shift\nfi\n\ndownload_audit_logs(){\n    local year=\"$1\"\n    local month=\"${2#0}\"  # because maths ops + 1 won't work on zero prefixed string, so re-add it later\n    local service=\"$3\"\n    shift; shift; shift\n    if [ \"${#month}\" = 1 ]; then\n        month=\"0$month\"\n    fi\n    local log=\"navigator_audit_${year}-${month}_${service}.csv\"\n    local log_bytes\n    # expand now\n    # shellcheck disable=SC2064\n    trap \"tstamp ERROR; tstamp 'Removing partial log file for restartability without audit gaps: '; rm -fv -- '$log'\" exit\n    if validate_log \"$log\"; then\n        tstamp \"Skipping previously completed log $log...\"\n        echo >&2\n        return\n    else\n        tstamp \"Querying Cloudera Navigator for $year logs for $service\"\n        month=\"${month#0}\"  # because maths ops + 1 won't work on zero prefixed string, so re-add it later\n        if [ \"$month\" = 12 ]; then\n            ((end_year=year+1))\n            end_month=01\n        else\n            ((end_month=month+1))\n            end_year=\"$year\"\n        fi\n        if [ \"${#month}\" = 1 ]; then\n            month=\"0$month\"\n        fi\n        if [ \"${#end_month}\" = 1 ]; then\n            end_month=\"0$end_month\"\n        fi\n        # won't output a newline so the contents of next command will be timestamp prefixed\n        tstamp\n        #time {\n        # don't let a random 401 stop from downloading other logs, can go back and fill in the gaps later by re-running\n        # Navigator returns zero byte logs without headers without error so this || : is not the cause of not catching zero byte logs, which we have to check for separately anyway\n        \"$srcdir/cloudera_navigator_audit_logs.sh\" \"$year-$month-01T00:00:00\" \"$end_year-$end_month-01T00:00:00\" \"service==$service\" \"$@\" | \"$srcdir/../bin/progress_dots.sh\" > \"$log\" || :\n        log_bytes=\"$(stat_bytes \"$log\")\"\n        tstamp \"$log = $log_bytes bytes\"\n        if [ \"$log_bytes\" = 0 ]; then\n            tstamp \"ERROR: Navigator returned zero byte audit log for $log, not even containing the headers row!\"\n            return\n        fi\n        #}\n    fi\n    #local compressed_log=\"$log.$ext\"\n    #if [ -s \"$log\" ]; then\n    if validate_log \"$log\"; then\n        #tstamp \"Compressing audit log:  $log > $compressed_log\"\n        # want splitting\n        # shellcheck disable=SC2086\n        #$compress_cmd \"$log\" > \"$compressed_log\" &\n        :\n    else\n        tstamp \"WARNING: $log doesn't look complete, must check\"\n    fi\n    echo >&2\n}\n\nvalidate_log(){\n    local log=\"$1\"\n    # a single newline in the log file trips this so dive in to deeper checks to make sure we have what looks like enough data\n    if [ -s \"$log\" ]; then\n        local log_bytes\n        log_bytes=\"$(stat_bytes \"$log\")\"\n        tstamp \"$log = $log_bytes bytes\"\n        if [ \"$log_bytes\" = 558 ]; then\n            tstamp \"$log has only headers - inferring there are no logs for that date range\"\n            return 0\n        #elif [ \"$log_bytes\" -gt 10240 ]; then\n        #    tstamp \"Skipping $log since it already exists and is > 10MB\"\n        #    return 0\n        #fi\n        # audit logs start at $year-12-* at the top, and end at the bottom in $year-01-* - partial logs often get cut off\n        # in between, so if we've gotten all the way to January the log is likely complete - tempted to do January 01 but\n        # there will probably be some edge case where a service isn't used on New Year's day or the first few days\n        # because a lot of people take time off around then, so this is more generic to just check for January\n        # can't check for December also being in the log because this would always fail for the current year\n        elif grep -q \"^\\\"$year-$month-0\" \"$log\"; then\n            tstamp \"$log contains logs for $year-$month-0*, looks complete\"\n            return 0\n        fi\n    fi\n    return 1\n}\n\ncurrent_year=\"$(date +%Y)\"\ncurrent_month=\"$(date +%m)\"\n\n# works on Mac but seq on Linux doesn't do reverse, outputs nothing\n#for year in $(seq \"$current_year\" 2009); do\n\n# On Mac tac requires gnu coreutils to be installed via Homebrew\nfor year in $(seq 2009 \"$current_year\" | tac); do\n    for month in {12..1}; do\n        if [ \"$year\" -eq \"$current_year\" ] && [ \"$month\" -gt \"$current_month\" ]; then\n            # Navigator returns forbidden if querying in the future\n            continue\n        fi\n        if [ -n \"${single_service:-}\" ]; then\n            download_audit_logs \"$year\" \"$month\" \"$single_service\" \"$@\"\n        else\n            for service in $services; do\n                download_audit_logs \"$year\" \"$month\" \"$service\" \"$@\"\n            done\n        fi\n    done\ndone\ntstamp \"Finished querying Cloudera Navigator API\"\ntstamp \"Waiting for log compression to finish\"\nwait\ntstamp \"DONE\"\ntrap - exit\n"
  },
  {
    "path": "bigdata/cloudera_navigator_audit_logs_download_retry.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 12:03:19 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Cloudera Navigator fails to download some logs but silently fails without an error or outputting anything, not even the headers\n# so this script loop retries the adjacent script cloudera_navigator_audit_logs_download.sh and checks for zero byte CSVs audit logs and retries\n# until they're all downloaded. In practice, some logs repeatedly get zero bytes so this isn't entirely effective have to cut your losses on the\n# logs that refused to extract from Navigator. Ironically older logs output the headers but not logs, at least indicating that there are no logs\n# rather than just giving blank output which is almost certainly another Cloudera bug\n#\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nsleep_secs=60\n\nif is_mac; then\n    stat_bytes=\"stat -f %z\"\nelse\n    stat_bytes=\"stat -c %s\"\nfi\n\ntime while true; do\n    time \"$srcdir/cloudera_navigator_audit_logs_download.sh\" -k\n    # want splitting of args\n    # shellcheck disable=SC2086\n    # grep -c stops the pipe terminating early causing:\n    # find: `stat' terminated by signal 13\n    num_zero_files=\"$(find . -maxdepth 1 -name 'navigator_audit_*.csv' -exec $stat_bytes {} \\; | grep -c '^0$')\"\n    if [ \"$num_zero_files\" != 0 ]; then\n        echo \"$num_zero_files files detected that have silently errored resulting in zero byte files, sleeping for $sleep_secs before retrying downloads...\"\n        sleep $sleep_secs\n        continue\n    fi\n    echo FINISHED\n    break\ndone\n"
  },
  {
    "path": "bigdata/cloudera_navigator_audit_logs_export_postgresql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-16 14:28:43 +0000 (Mon, 16 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Exports Cloudera Navigator logs from the underlying PostgreSQL database to files in the local directory\n#\n# FILTER environment variable will restrict to matching fully qualified tables (<db>.<schema>.<table>)\n#\n# CSV Output Format is dependent on database columns and can change, but at time of writing was:\n#\n# HDFS logs:\n#\n# id,service_name,username,ip_addr,event_time,operation,src,dest,permissions,allowed,impersonator,delegation_token_id\n#\n# Hive logs:\n#\n# id,event_time,allowed,service_name,username,ip_addr,operation,database_name,object_type,table_name,operation_text,impersonator,resource_path,object_usage_type\n#\n# Impala logs:\n#\n# id,event_time,allowed,service_name,username,impersonator,ip_addr,operation,query_id,session_id,status,database_name,object_type,table_name,privilege,operation_text\n#\n# Tested on AWS RDS PostgreSQL 9.5.15\n\n# For individual table export timings set \\timing in ~/.psqlrc\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nlogdir=\"$PWD/cloudera_navigator_logs\"\n\n# only export tables matching this regex\nexport FILTER='\\.[[:alnum:]]+_audit_events_'\n\n# if you only want Hive + Impala logs to determine table access patterns\n#export FILTER='\\.(hive|impala)_audit_events_'\n\n# don't background gzip's if filesystem < 30GB free as filesystem will fill up faster than gzip can complete and remove original files to free space\nMIN_FILESYSTEM_MB=30000\n\ntstamp \"Exporting Cloudera Navigator logs from PostgreSQL database:\"\necho >&2\n\n# doesn't seem to like \\copy no matter how many backslashes\n#\"$srcdir/../postgres/postgres_foreach_table.sh\" \"\n#select replace('exporting {table}', '\\\"', '');\n#\\\\copy (SELECT * FROM {db}.{schema}.{table}) TO replace('cloudera_navigator_logs/{db}.{schema}.{table}.csv', '\\\"', '') WITH (FORMAT CSV, HEADER);\n#\" \"$@\"\n\ntstamp \"logdir = $logdir\"\necho >&2\nmkdir -pv \"$logdir\"\n\ntime {\n\"$srcdir/../postgres/postgres_list_tables.sh\" \"$@\" |\nwhile read -r db schema table; do\n#    echo \"SELECT 'Exporting $db.$schema.$table' AS progress;\"\n#    echo \"\\\\copy (SELECT * FROM \\\"$db\\\".\\\"$schema\\\".\\\"$table\\\") TO 'cloudera_navigator_logs/$db.$schema.$table.csv' WITH (FORMAT CSV, HEADER);\"\n#done |\n#\"$srcdir/../postgres/psql.sh\" \"$@\"\n    filename=\"$logdir/$db.$schema.$table.csv\"\n    tstamp \"Exporting $db.$schema.$table:  \"\n    rm -fv -- \"$filename\"  # would get overwritten anyway but removing to detect when psql errors out without non-zero exit code\n    psql.sh -c \"\\\\copy (SELECT * FROM \\\"$db\\\".\\\"$schema\\\".\\\"$table\\\") TO '$filename' WITH (FORMAT CSV, HEADER);\"\n    if ! [ -f \"$filename\" ]; then\n        tstamp \"ERROR: EXPORT FAILED\"\n        exit 1\n    fi\n    # empty\n    if ! [ -s \"$filename\" ]; then\n        tstamp \"${filename##*/} is empty, removing...\"\n        rm -f -- \"$filename\"\n        echo >&2\n        continue\n    fi\n    # only a header line\n    if wc -l \"$filename\" | grep -q '^1[[:space:]]'; then\n        tstamp \"${filename##*/} has only header line, removing...\"\n        rm -f -- \"$filename\"\n        echo >&2\n        continue\n    fi\n    # we run out of space without this as logs can easily be dozens of GB per day per service\n    tstamp \"compressing $filename\"\n    filesystem_free_mb=\"$(df -m . | awk '{print $4}' | tail -n 1)\"\n    if [ \"$filesystem_free_mb\" -lt $MIN_FILESYSTEM_MB ]; then\n        # --force overwrite of existing gzip logs\n        gzip -9 --force \"$filename\"\n    else\n        gzip -9 --force \"$filename\" &\n    fi\n    echo >&2\ndone || exit $?\ntstamp \"waiting for background log compression to finish...\"\nwait\necho >&2\ntstamp \"Cloudera Navigator PostgreSQL exports finished\"\necho >&2\n}\n"
  },
  {
    "path": "bigdata/hadoop_random_node.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-16 12:22:40 +0000 (Thu, 16 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to print a random Hadoop node by parsing the Hadoop topology map from /etc\n#\n# Tested on CDH 5.10\n#\n# See also:\n#\n#   find_active_*.py - https://github.com/HariSekhon/DevOps-Python-tools\n#\n#   HAProxy Configs for many Hadoop and other technologies - https://github.com/HariSekhon/HAProxy-configs\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ntopology_map=\"${HADOOP_TOPOLOGY_MAP:-/etc/hadoop/conf/topology.map}\"\n\nif ! [ -f \"$topology_map\" ]; then\n    echo \"File not found: $topology_map. Did you run this on a Hadoop node?\" >&2\n    exit 1\nfi\n\n# returns datanodes in the topology map by omitting nodes with that are masters / namenodes / control nodes\nawk -F'\"' '/<node name=\"[A-Za-z]/{print $2}' \"$topology_map\" |\ngrep -Ev '^[^.]*(name|master|control)' |\nshuf -n 1\n"
  },
  {
    "path": "bigdata/hdfs_checksum.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting HDFS MD5-of-MD5 checksums for each file\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nCapture stdout > file.txt for comparisons\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will skip files with zero bytes to save time since\nthey always return the same anyway:   MD5-of-0MD5-of-0CRC32   00000000000000000000000070bc8f4b72a86921468bf8e8441dce5\n\nCaveats:\n\nThis is slow because the HDFS command startup is slow and is called once per file path so doesn't scale well\nIf you want to skip zero byte files, set environment variable \\$SKIP_ZERO_BYTE_FILES\n\nSee Also:\n\nhadoop_hdfs_files_native_checksums.jy\n\nfrom the adjacent GitHub repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\nI would have written this version in Python but the Snakebite library doesn't support checksum extraction\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ $1=$2=$3=$4=$5=$6=$7=\"\"; print }' |\n#sed 's/^[[:space:]]*//' |\nwhile read -r filepath; do\n    hdfs dfs -checksum \"$filepath\"\ndone\n"
  },
  {
    "path": "bigdata/hdfs_checksum_crc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting portable CRC32 checksums for each file\n(can be used for HDFS vs local comparisons, whereas default MD5-of-MD5 cannot)\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nCapture stdout > file.txt for comparisons\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nThis is slow because the HDFS command startup is slow and is called once per file path\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will skip files with zero bytes to save time since\nthey always return the same checksum anyway.\n\nCaveats:\n\nThis is slow because the HDFS command startup is slow and is called once per file path so doesn't scale well\nIf you want to skip zero byte files, set environment variable \\$SKIP_ZERO_BYTE_FILES to any value\n\nSee Also:\n\nhadoop_hdfs_files_native_checksums.jy\n\nfrom the adjacent GitHub rep (outputs MD5-of-MD5 not CRC32 though):\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\nI would have written this version in Python but the Snakebite library doesn't support checksum extraction\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ $1=$2=$3=$4=$5=$6=$7=\"\"; print }' |\n#sed 's/^[[:space:]]*//' |\nwhile read -r filepath; do\n    hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum \"$filepath\"\ndone\n"
  },
  {
    "path": "bigdata/hdfs_checksum_crc_parallel.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting HDFS MD5-of-MD5 checksums for each file in parallel\nsince dfs command has slow startup\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nCapture stdout | sort > file.txt for comparisons\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will skip files with zero bytes to save time since\nthey always return the same checksum anyway\n\nCaveats:\n\nHDFS command startup is slow and is called once per file so I work around this by launching several dfs commands\nin parallel. Configure parallelism via environment variable PARALLELISM=N where N must be an integer\n\nSee Also:\n\nhadoop_hdfs_files_native_checksums.jy\n\nfrom the adjacent GitHub rep (outputs MD5-of-MD5 not CRC32 though):\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\nI would have written this version in Python but the Snakebite library doesn't support checksum extraction\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\n#PARALLELISM=\"${PARALELLISM:-$(grep -c '^processor[[:space:]]*:' /proc/cpuinfo)}\"\n# don't use all cores because if running on a datanode it might have dozens of cores and overwhelm namenode\n# cap at 10 unless explicitly set\nexport PARALLELISM=\"${PARALLELISM:-10}\"\n\nif ! [[ \"$PARALLELISM\" =~ ^[[:digit:]]+$ ]]; then\n    echo \"PARALLELISM must be set to an integer!\"\n    exit 4\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ $1=$2=$3=$4=$5=$6=$7=\"\"; print }' |\n#sed 's/^[[:space:]]*//' |\nwhile read -r filepath; do\n    echo \"hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum '$filepath'\"\ndone |\nparallel -j \"$PARALLELISM\"\n"
  },
  {
    "path": "bigdata/hdfs_checksum_parallel.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting HDFS MD5-of-MD5 checksums for each file in parallel\nsince dfs command has slow startup\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nCapture stdout | sort > file.txt for comparisons\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will skip files with zero bytes to save time since\nthey always return the same checksum anyway:\n\nMD5-of-0MD5-of-0CRC32   00000000000000000000000070bc8f4b72a86921468bf8e8441dce5\n\nCaveats:\n\nHDFS command startup is slow and is called once per file so I work around this by launching several dfs commands\nin parallel. Configure parallelism via environment variable PARALLELISM=N where N must be an integer\n\nSee Also:\n\nhadoop_hdfs_files_native_checksums.jy\n\nfrom the adjacent GitHub repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\nI would have written this version in Python but the Snakebite library doesn't support checksum extraction\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\n#PARALLELISM=\"${PARALELLISM:-$(grep -c '^processor[[:space:]]*:' /proc/cpuinfo)}\"\n# don't use all cores because if running on a datanode it might have dozens of cores and overwhelm namenode\n# cap at 10 unless explicitly set\nexport PARALLELISM=\"${PARALLELISM:-10}\"\n\nif ! [[ \"$PARALLELISM\" =~ ^[[:digit:]]+$ ]]; then\n    echo \"PARALLELISM must be set to an integer!\"\n    exit 4\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ $1=$2=$3=$4=$5=$6=$7=\"\"; print }' |\n#sed 's/^[[:space:]]*//' |\nwhile read -r filepath; do\n    echo \"hdfs dfs -checksum '$filepath'\"\ndone |\nparallel -j \"$PARALLELISM\"\n"
  },
  {
    "path": "bigdata/hdfs_file_size.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting:\n\n<file_size_of_single_replica>     <filename>\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\n\nusage: ${0##*/} <file_or_directory_paths> [hdfs_dfs_du_options]\n\n\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        # not including -h here because du -h is needed for human readable format\n        --help) usage\n                ;;\n    esac\ndone\n\n# if using -h there will be more columns so remove cols 3 + 4 which are replica sizes eg.\n#\n# 21.7 M  65.0 M  hdfs://nameservice1/user/hive/warehouse/...\n#\n# otherwise will be in format\n#\n# 22713480  68140440  hdfs://nameservice1/user/hive/warehouse/...\n\nhdfs dfs -du \"$@\" |\nawk '{ if($2 ~ /[A-Za-z]/){ $3=\"\"; $4=\"\"} else { $2=\"\" }; print }' |\ncolumn -t\n"
  },
  {
    "path": "bigdata/hdfs_file_size_including_replicas.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting:\n\n<disk_space_for_all_replicas>     <filename>\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\n\nusage: ${0##*/} <file_or_directory_paths> [hdfs_dfs_du_options]\n\n\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        # not including -h here because du -h is needed for human readable format\n        --help) usage\n                ;;\n    esac\ndone\n\n# if using -h there will be more columns so remove cols 1 + 2 and use cols 3 + 4 for sizes including replicas eg.\n#\n# 21.7 M  65.0 M  hdfs://nameservice1/user/hive/warehouse/...\n#\n# otherwise will be in format\n#\n# 22713480  68140440  hdfs://nameservice1/user/hive/warehouse/...\n\nhdfs dfs -du \"$@\" |\nawk '{ if($2 ~ /[A-Za-z]/){ $1=\"\"; $2=\"\" } else { $2=\"\"  }; print  }' |\ncolumn -t\n"
  },
  {
    "path": "bigdata/hdfs_find_replication_factor_1.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments outputting any HDFS files with replication factor 1\n\nThese files cause alerts during single node downtime / maintenance due to missing blocks and\nshould really be set to replication factor 3\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will not list files with zero bytes\nSetting environment variable SET_REPLICATION_FACTOR_3 to any value will set any files found with\nreplication factor 1 back to replication factor 3\n\n\nSee also:\n\nhdfs_find_replication_factor_1.py in DevOps Python tools repo which can\nalso reset these found files back to replication factor 3 to fix the issue\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nset_replication_factor_3(){\n    if [ -n \"${SET_REPLICATION_FACTOR_3:-}\" ]; then\n        xargs --no-run-if-empty hdfs dfs -setrep 3\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ if ($2 == 1) { $1=$2=$3=$4=$5=$6=$7=\"\"; print } }' |\nsed 's/^[[:space:]]*//' |\nset_replication_factor_3\n"
  },
  {
    "path": "bigdata/hdfs_set_replication_factor_3.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-27 16:09:34 +0000 (Wed, 27 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nRecurses HDFS path arguments checking for any files with less than replication factor 3 and resetting them\nback to replication factor 3\n\nThese files cause alerts during single node downtime / maintenance due to missing blocks and\nshould really be set to replication factor 3\n\nCalls HDFS command which is assumed to be in \\$PATH\n\nMake sure to kinit before running this if using a production Kerberized cluster\n\nSetting environment variable SKIP_ZERO_BYTE_FILES to any value will not list files with zero bytes\n\n\nSee also:\n\nhdfs_find_replication_factor_1.sh (adjacent)\n\nand\n\nhdfs_find_replication_factor_1.py in DevOps Python tools repo which can\nalso reset these found files back to replication factor 3 to fix the issue\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n\nusage: ${0##*/} <file_or_directory_paths>\n\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ ^- ]]; then\n    usage\nfi\n\nskip_zero_byte_files(){\n    if [ -n \"${SKIP_ZERO_BYTE_FILES:-}\" ]; then\n        awk '{if($5 != 0) print }'\n    else\n        cat\n    fi\n}\n\nhdfs dfs -ls -R \"$@\" |\ngrep -v '^d' |\nskip_zero_byte_files |\nawk '{ if ($2 < 3) { $1=$2=$3=$4=$5=$6=$7=\"\"; print } }' |\nsed 's/^[[:space:]]*//' |\nxargs --no-run-if-empty hdfs dfs -setrep 3\n"
  },
  {
    "path": "bigdata/hive_foreach_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun SQL query against all Hive tables in all databases via beeline\n\nQuery can contain {db} and {table} placeholders which will be replaced for each table\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\nWARNING: do not run any subshell command reading from standard input, otherwise it will consume the db/table names and exit after the first iteration\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nyou will need to comment out / remove the 'set -o pipefail' to skip errors if you aren't authorized to use\nany of the databases to avoid the script exiting early upon encountering any authorization error such:\n\nERROR: AuthorizationException: User '<user>@<domain>' does not have privileges to access: default   Default Hive database.*.*\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"<query>\\\" [<beeline_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery_template=\"$1\"\nshift || :\n\nopts=\"--silent=true --outputformat=tsv2\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n# shellcheck disable=SC2086\n\"$srcdir/hive_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    \"$srcdir/beeline.sh\" $opts -e \"USE \\`$db\\`; $query\" \"$@\" |\n    tail -n +2\ndone\n"
  },
  {
    "path": "bigdata/hive_list_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList all Hive databases via beeline\n\nOutput Format:\n\n<database_name>\n\n\nFILTER environment variable will restrict to matching databases (if giving <db>.<table>, matches up to the first dot)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nyou will need to comment out / remove '-o pipefail' below to skip errors if you aren't authorized to use\nany of the databases to avoid the script exiting early upon encountering any authorization error such:\n\nERROR: AuthorizationException: User '<user>@<domain>' does not have privileges to access: default   Default Hive database.*.*\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/beeline.sh\" --silent=true --outputformat=tsv2 -e 'SHOW DATABASES' \"$@\" |\ntail -n +2 |\n# awk '{print $1}' |\nwhile read -r db; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db\" =~ ${FILTER%%.*} ]]; then\n        continue\n    fi\n    printf '%s\\n' \"$db\"\ndone\n"
  },
  {
    "path": "bigdata/hive_list_tables.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList Hive tables in all databases via beeline\n\nOutput Format:\n\n<database_name>     <table_name>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor Hive 3.0+ information schema is finally available which is more efficient than iterating per database eg.\n\nSELECT * FROM information_schema.tables\n(table_catalog, table_schema, table_name)\n\nFor Hive < 3.0 - consider using adjacent impala_list_tables.sh instead as it is much faster\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nHive doesn't suffer from db authz issue listing metadata like Impala, which gets:\n\nERROR: AuthorizationException: User '<user>@<domain>' does not have privileges to access: default   Default Hive database.*.*\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nopts=\"--silent=true --outputformat=tsv2\"\n\n# shellcheck disable=SC2086\n\"$srcdir/hive_list_databases.sh\" \"$@\" |\nwhile read -r db; do\n    \"$srcdir/beeline.sh\" $opts -e \"SHOW TABLES FROM \\`$db\\`\" \"$@\" |\n    tail -n +2 |\n    sed \"s/^/$db\t/\"\ndone |\nwhile read -r db table; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db.$table\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\n' \"$db\" \"$table\"\ndone\n"
  },
  {
    "path": "bigdata/hive_tables_column_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints the number of columns per Hive table\n\nOutput Format:\n\n<db>.<table>    <column_count>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor more documentation see the comments at the top of beeline.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nquery_template=\"describe {table}\"\n\nopts=\"--silent=true --outputformat=tsv2\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/hive_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    # shellcheck disable=SC2086\n    if ! \"$srcdir/beeline.sh\" $opts -e \"USE \\`$db\\`; $query\" \"$@\"; then\n        echo \"ERROR running query: $query\" >&2\n        echo \"UNKNOWN\"\n    fi |\n    tail -n +2 |\n    awk '{if(NF == 2){print}}' |\n    wc -l |\n    sed 's/[[:space:]]*//g'\ndone\n"
  },
  {
    "path": "bigdata/hive_tables_locations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShow table location for all tables via beeline\n\nOutput Format:\n\n<db>.<table>    <location>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor more documentation see the comments at the top of beeline.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nexec \"$srcdir/hive_tables_metadata.sh\" Location \"$@\"\n"
  },
  {
    "path": "bigdata/hive_tables_metadata.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrint each table's DDL metadata field eg. Location\n\nOutput Format:\n\n<db>.<table>    <metadata_field>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor more documentation see the comments at the top of beeline.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<metadata_field> [<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nfield=\"$1\"\nshift || :\n\nquery_template=\"describe formatted {table}\"\n\nopts=\"--silent=true --outputformat=tsv2\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/hive_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    # shellcheck disable=SC2086\n    { \"$srcdir/beeline.sh\" $opts -e \"USE \\`$db\\`; $query\" \"$@\" || echo \"ERROR running query: $query\" >&2; } |\n    {  grep \"^$field\" || echo UNKNOWN; } |\n    sed \"s/^$field:[[:space:]]*//; s/[[:space:]]*NULL[[:space:]]*$//\"\ndone\n"
  },
  {
    "path": "bigdata/hive_tables_row_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 11:10:26 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGet row counts for all Hive tables in all databases via beeline\n\nTSV Output format:\n\n<database>.<table>     <row_count>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Hive 1.1.0 on CDH 5.10, 5.16\n\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nyou will need to comment out / remove '-o pipefail' below to skip errors if you aren't authorized to use\nany of the databases to avoid the script exiting early upon encountering any authorization error such:\n\nERROR: AuthorizationException: User '<user>@<domain>' does not have privileges to access: default   Default Hive database.*.*\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<beeline_options>]\"\n\nhelp_usage \"$@\"\n\n\nexec \"$srcdir/hive_foreach_table.sh\" \"SELECT COUNT(*) FROM {db}.{table}\" \"$@\"\n"
  },
  {
    "path": "bigdata/impala_foreach_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2016\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun SQL query against all Impala tables in all databases via impala-shell\n\nQuery can contain {db} and {table} placeholders which will be replaced for each table\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\nWARNING: do not run any subshell command reading from standard input, otherwise it will consume the db/table names and exit after the first iteration\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n'set -o pipefail' is not enabled in order to skip authorization errors such as that documented in impala_list_tables.sh\nand also ignore errors from the 'select count(*)' in the loop as Impala often has metadata errors such as:\n\nERROR: AnalysisException: Failed to load metadata for table: '<table>'\nCAUSED BY: TableLoadingException: Unsupported type 'void' in column '<column>' of table '<table>'\n\n============================================================================ #\n\"'\nWARNINGS: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/hive/warehouse/<database>.db/<table>/1234a5678b90cd1-ef23a45678901234_5678901234_data.10.parq\nError(2): No such file or directory\nRoot cause: RemoteException: File does not exist: /user/hive/warehouse/<database>.db/<table>/1234a5678b90cd1-ef23a45678901234_5678901234_data.10.parq\n        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)\n        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2157)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2127)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040)\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:583)\n        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:94)\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:377)\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)\n        at java.security.AccessController.doPrivileged(Native Method)\n        at javax.security.auth.Subject.doAs(Subject.java:422)\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)\n'\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"<query>\\\" [<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery_template=\"$1\"\nshift || :\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/impala_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    \"$srcdir/impala_shell.sh\" --quiet -Bq \"USE \\`$db\\`; $query\" \"$@\"\ndone\n"
  },
  {
    "path": "bigdata/impala_list_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all Impala databases using adjacent impala_shell.sh script\n\n\nFILTER environment variable will restrict to matching databases (if giving <db>.<table>, matches up to the first dot)\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\n# strip comments after database name, eg.\n# default Default Hive database\n\"$srcdir/impala_shell.sh\" --quiet -Bq 'SHOW DATABASES' \"$@\" |\nawk '{print $1}' |\nwhile read -r db; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db\" =~ ${FILTER%%.*} ]]; then\n        continue\n    fi\n    printf '%s\\n' \"$db\"\ndone\n"
  },
  {
    "path": "bigdata/impala_list_tables.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all Impala tables in all databases using adjacent impala_shell.sh script\n\nOutput Format:\n\n<db>    <table>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nyou will need to comment out / remove 'set -o pipefail' below to skip errors if you aren't authorized to use\nany of the databases to avoid the script exiting early upon encountering any authorization error such:\n\nERROR: AuthorizationException: User '<user>@<domain>' does not have privileges to access: default   Default Hive database.*.*\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/impala_list_databases.sh\" |\nwhile read -r db; do\n    \"$srcdir/impala_shell.sh\" --quiet -Bq \"SHOW TABLES IN \\`$db\\`\" \"$@\" |\n    sed \"s/^/$db\t/\"\ndone |\nwhile read -r db table; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db.$table\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\n' \"$db\" \"$table\"\ndone\n"
  },
  {
    "path": "bigdata/impala_shell.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to more easily connect to Impala without having to find an impalad and repeatedly specify options like -k for kerberos\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nIf using dedicated coordinators then consider setting IMPALA_HOST to one of those explicitly, see\n\n   https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/impala_dedicated_coordinator.html\n\n\nSee also:\n\n  find_active_impalad.py - https://github.com/HariSekhon/DevOps-Python-tools\n\n  HAProxy Configs for Impala and many other technologies - https://github.com/HariSekhon/HAProxy-configs\n\n\nIf you get an error such as:\n\nError connecting: TTransportException, TSocket read 0 bytes\n\nthen check if you need to add --ssl to the command line (or export IMPALA_SSL=1 to do this automatically, eg. put in .bashrc or similar)\n\n\nUseful options for scripting:\n\n  -q --query\n  -B --delimited\n  --output_delimiter=\\\\t   # default\n  --quiet\n\n\nSee adjacent impala_*.sh scripts for slightly better versions of these quick command line examples, including better escaping\n\n\nExamples:\n\n\n# List all databases:\n\n  ./impala_shell.sh -Bq 'show databases' | awk '{print \\$1}'\n\n\n# List all tables in all databases:\n\n  ./impala_shell.sh -Bq 'show databases' | while read db rest; do ./impala_shell.sh -Bq \\\"use \\$db; show tables\\\" | sed \\\"s/^/\\$db./\\\"; done\n\n\n# Row counts for all tables in all databases:\n\n  ./impala_shell.sh --quiet -Bq 'show databases' | while read db rest; do ./impala_shell.sh --quiet -Bq \\\"use \\$db; show tables\\\" | while read table; do printf \\\"%s\\\\t\\\" \\\"\\$db.\\$table\\\"; ./impala_shell.sh --quiet -Bq \\\"use \\$db; SELECT COUNT(*) FROM \\$table\\\"; done; done > row_counts.tsv\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\nopts=\"${IMPALA_OPTS:-}\"\n\ncore_site_xml=\"${HADOOP_CORE_SITE_XML:-/etc/hadoop/conf/core-site.xml}\"\n\n#if ! [ -f \"$core_site_xml\" ]; then\n#    echo \"File not found: $core_site_xml. Did you run this on a Hadoop node?\" >&2\n#    exit 1\n#fi\n\nif [ -n \"${IMPALA_KERBEROS:-}\" ] ||\n   grep -A 1 hadoop.security.authentication \"$core_site_xml\" 2>/dev/null | grep -q kerberos; then\n    opts=\"$opts -k\"\nfi\n\nif [ -n \"${IMPALA_SSL:-}\" ]; then\n    opts=\"$opts --ssl\"\nfi\n\ntopology_map=\"${HADOOP_TOPOLOGY_MAP:-/etc/hadoop/conf/topology.map}\"\n\nif [ -n \"${IMPALA_HOST:-}\" ]; then\n    impalad=\"$IMPALA_HOST\"\nelif [ -f \"$topology_map\" ]; then\n    #echo \"picking random impala from hadoop topology map\" >&2\n    # nodes in the topology map that aren't masters, namenodes, controlnodes etc probably have impalad running on them, so pick one at random to connect to\n    # or alternatively use HAProxy config for load balanced impala clusters - see https://github.com/HariSekhon/HAProxy-configs\n    impalad=\"$(\n        awk -F'\"' '/<node name=\"[A-Za-z]/{print $2}' \"$topology_map\" |\n        grep -Ev '^[^.]*(name|master|control)' |\n        shuf -n 1\n    )\"\nelse\n    impalad=\"$(hostname -f)\"\n    #echo \"IMPALA_HOST not set and topology map '$topology_map' not found, defaulting to local host $impalad\"\nfi\n\n# split opts\n# shellcheck disable=SC2086\nexec impala-shell $opts -i \"$impalad\" \"$@\"\n"
  },
  {
    "path": "bigdata/impala_tables_column_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrint each table's number of columns\n\nOutput Format:\n\n<db>.<table>    <column_count>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nCaveats:\n\n    Hive is more reliable as Impala breaks on some table metadata definitions where Hive doesn't\n\n    Impala is faster than Hive for the first ~1000 tables but then slows down\n    so if you have a lot of tables I recommend you use the Hive version of this instead\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\nquery_template=\"describe {table}\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/impala_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    if ! \"$srcdir/impala_shell.sh\" --quiet -Bq \"USE \\`$db\\`; $query\" \"$@\"; then\n        echo \"ERROR running query: $query\" >&2\n        echo \"UNKNOWN\"\n    fi |\n    awk '{if(NF == 2){print}}' |\n    wc -l |\n    sed 's/[[:space:]]*//g'\ndone\n"
  },
  {
    "path": "bigdata/impala_tables_locations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShow table location for all tables via Impala shell\n\nOutput Format:\n\n<db>.<table>    <location>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nCaveats:\n\n    Hive is more reliable as Impala breaks on some table metadata definitions where Hive doesn't\n\n    Impala is faster than Hive for the first ~1000 tables but then slows down\n    so if you have a lot of tables I recommend you use the Hive version of this instead\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\nexec \"$srcdir/impala_tables_metadata.sh\" Location \"$@\"\n"
  },
  {
    "path": "bigdata/impala_tables_metadata.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# you will almost certainly have to comment out / remove '-o pipefail' to skip authorization errors such as that documented in impala_list_tables.sh\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrint each table's DDL metadata field eg. Location\n\nOutput Format:\n\n<db>.<table>    <field>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nCaveats:\n\n    Hive is more reliable as Impala breaks on some table metadata definitions where Hive doesn't\n\n    Impala is faster than Hive for the first ~1000 tables but then slows down\n    so if you have a lot of tables I recommend you use the Hive version of this instead\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<metadata_field> [<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfield=\"$1\"\nshift || :\n\nquery_template=\"describe formatted {table}\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/impala_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    { \"$srcdir/impala_shell.sh\" --quiet -Bq \"USE \\`$db\\`; $query\" \"$@\" || echo \"ERROR running query: $query\" >&2; } |\n    {  grep \"^$field\" || echo UNKNOWN; } |\n    sed \"s/^$field:[[:space:]]*//; s/[[:space:]]*NULL[[:space:]]*$//\"\ndone\n"
  },
  {
    "path": "bigdata/impala_tables_row_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2016\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for all Impala tables in all databases using adjacent impala_shell.sh script\n\nOutput format:\n\n<database>.<table>     <row_count>\n\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\n\nTested on Impala 2.7.0, 2.12.0 on CDH 5.10, 5.16 with Kerberos and SSL\n\n\nFor more documentation see the comments at the top of impala_shell.sh\n\nFor a better version written in Python see DevOps Python tools repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n'set -o pipefail' is commented out to skip authorization errors such as that documented in impala_list_tables.sh\nand also ignore errors from the 'select count(*)' in the loop as Impala often has metadata errors such as:\n\nERROR: AnalysisException: Failed to load metadata for table: '<table>'\nCAUSED BY: TableLoadingException: Unsupported type 'void' in column '<column>' of table '<table>'\n\n============================================================================ #\n\"'\nWARNINGS: Disk I/O error: Failed to open HDFS file hdfs://nameservice1/user/hive/warehouse/<database>.db/<table>/1234a5678b90cd1-ef23a45678901234_5678901234_data.10.parq\nError(2): No such file or directory\nRoot cause: RemoteException: File does not exist: /user/hive/warehouse/<database>.db/<table>/1234a5678b90cd1-ef23a45678901234_5678901234_data.10.parq\n        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)\n        at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:2157)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2127)\n        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2040)\n        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:583)\n        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:94)\n        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:377)\n        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\n        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)\n        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)\n        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)\n        at java.security.AccessController.doPrivileged(Native Method)\n        at javax.security.auth.Subject.doAs(Subject.java:422)\n        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)\n        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)\n'\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<impala_shell_options>]\"\n\nhelp_usage \"$@\"\n\n\nexec \"$srcdir/impala_foreach_table.sh\" \"SELECT COUNT(*) FROM {db}.{table}\" \"$@\"\n"
  },
  {
    "path": "bigdata/zookeeper_client.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-06 12:09:19 +0000 (Fri, 06 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to more easily connect to ZooKeeper without having to look up the ZooKeeper addresses\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ -z \"${ZOOKEEPERS:-}\" ]; then\n    set +o pipefail\n    ZOOKEEPERS=\"$(find /etc -name '*-site.xml' -exec grep -hA1 zookeeper.quorum {} \\; 2>/dev/null | grep '<value>' | sed 's/<value>//;s,</value>,,;s/[[:space:]]*//g' | sort -u | head -n1)\"\n    set -o pipefail\n    if [ -z \"${ZOOKEEPERS:-}\" ]; then\n        echo \"ZOOKEEPERS environment variable not set (format is zookeeper1.domain.com:2181,zookeeper2.domain.com:2181,zookeeper3.domain.com:2181)\"\n        exit 3\n    fi\nfi\n\nexec zookeeper-client -server \"$ZOOKEEPERS\" \"$@\"\n"
  },
  {
    "path": "bigdata/zookeeper_shell.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Kafka's zookeeper-shell wrapper to auto-populate common options like the zookeeper addresses and znode parent\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154\nexec zookeeper-shell.sh \"${KAFKA_ZOOKEEPERS:-}${KAFKA_ZOOKEEPER_ROOT:-}\" \"$@\"\n"
  },
  {
    "path": "bin/bash_most_used_commands.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-04 15:55:47 -0500 (Sun, 04 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints your 100 most used Bash commands, taking in to account whether your shell history is in basic or extended format\n\nEg. Handles both\n\n    <num> <command>\n\nand\n\n    <num> <date> <time> <command>\n\nBecause shell scripts do not have access to the interactive history, this uses ~/.bash_history file\nand parses out timestamps\n\nCaveat: this will not include current shell's commands which are usually only written upon shell exit\n\nTo include the current shell, do a one liner like this:\n\n    history | awk '{ a[\\$2]++ } END { for(i in a) { print a[i] \\\" \\\" i } }' | sort -rn | head -n 100\n\nOr if you're using timestamped shell history:\n\n    history | awk '{ a[\\$4]++ } END { for(i in a) { print a[i] \\\" \\\" i } }' | sort -rn | head -n 100\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\n#timestamped_history=0\n\n#if history | grep -P '^\\d+\\s+\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2}\\s'; then\n#    timestamped_history=1\n#fi\n\n# doesn't work\n#\"$SHELL\" -c history |\n# if you were piping live history only, but format of ~/.bash_history is different\n#if [ \"$timestamped_history\" = 1 ]; then\n#    awk '{ a[$4]++ } END { for(i in a) { print a[i] \" \" i } }'\n#else\n#    awk '{ a[$2]++ } END { for(i in a) { print a[i] \" \" i } }'\n#fi |\n#\n# filter out timestamp epoch lines prefixed with a hash\nsed '/^#/d' ~/.bash_history |\nawk '{ a[$1]++ } END { for(i in a) { print a[i] \" \" i } }' |\nsort -rn |\nhead -n 100\n"
  },
  {
    "path": "bin/bash_profile_bashrc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-13 00:48:17 +0100 (Sat, 13 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps a trace of the bash profile, .bashrc and all related shell startup costs to a given file\n\nUseful to find out why spawning new bash shells gets slow\n\nI've used this to cut down from 20 seconds to 2 seconds\n\nTips:\n\n- bash completion is expensive and to be mostly avoided\n- avoid iterating on directories with lots of files, use a targeted find command to efficiently pre-filter - this also reduces the amount of shell tracing\n    eg. for x in somedir_full_of_files/*; do [ -d '\\$x' ] || continue; ... ; done   <-- this is expensive and noisy in traces\n- running headtail.py -n 5 on the trace file can be useful, although additional timings added to this file reduce the need\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ntrace_file=\"$1\"\n\ntimestamp \"Running bash interactive with shell tracing - output results to '$trace_file'\"\n# the 'time' command outputs a separating space line already\n#echo >&2\ntime bash -ix -c 'echo' 2>&1 | logger -s 2>\"$trace_file\"\necho >&2\ntimestamp \"Bash profile trace can now be found in the trace file '$trace_file'\"\n"
  },
  {
    "path": "bin/center.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-23 00:02:17 +0000 (Sat, 23 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# borrowed from here:\n#\n# http://codereview.stackexchange.com/questions/94449/text-centering-function-in-bash\n\n# This is only for local use, there is a much better Python version in my DevOps Python Tools repo:\n#\n#  https://github.com/HariSekhon/DevOps-Python-tools\n\nset -euo pipefail\n\ntextsize=\"${#1}\"\n# I want this to only match my hr() function, not 27\" iMac 5K screens\n#width=$(tput cols)\nwidth=\"${2:-${WIDTH:-80}}\"\nspan=\"$(((width + textsize) / 2))\"\nprintf \"%${span}s\\\\n\" \"${1:-NO_MESSAGE_GIVEN_TO_CENTER}\"\n"
  },
  {
    "path": "bin/clean_caches.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-28 17:39:13 +0100 (Tue, 28 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Removes local caches for OS package installations and Programming Language development libraries\n#\n# Useful in Docker builds to reduce image size or just general space cleaning\n#\n# eg.\n#\n# Add this to the end of each of your RUN statements in your Dockerfile to clean up the installation caches and not save them in the Docker layer:\n#\n#   curl -s https://raw.githubusercontent.com/HariSekhon/bash-tools/master/bin/clean_caches.sh | sh\n#\n\nset -eu  #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# not using any of my libraries or path dependencies to allow the above self-contained curl to shell to work for calling from Dockerfiles\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# https://wiki.alpinelinux.org/wiki/Local_APK_cache\n\n# OS package management caches\ncache_list=\"\n/etc/apk/cache\n/var/lib/apt/lists\n/var/cache/apt\n/var/cache/apk\n/var/cache/yum\n\"\n\n# Personal Language Caches we want to remove to save space:\n#\n# .cache => Python pip\n#\n# .cpan  => Perl\n# .cpanm\n#\n# .gem   => Ruby\n#\n# Java / Scala / Groovy:\n#\n# .gradle => Gradle\n# .groovy => Groovy (contains grapes/)\n# .ivy    => Ivy (Sbt / Gradle)\n# .ivy2\n# .m2     => Maven\n# .sbt    => SBT\n\nif [ -z \"${GOPATH-}\" ]; then\n    if [ -d ~/go ]; then\n        GOPATH=~/go\n    else\n        GOPATH=\"notfound\"  # to avoid deleting root level dirs which might be something else\n    fi\nfi\n\n#  ~/.groovy/grapes\npersonal_cache_list=\"\n.cache\n.cpan\n.cpanm\n.gem\n.gradle\n.groovy\ngo/pkg\ngo/src/golang.org\ngo/src/github.com\n$GOPATH/pkg\n$GOPATH/src/golang.org\n$GOPATH/src/github.com\n.ivy\n.ivy2\n.m2\n.sbt\nLibrary/Caches/Homebrew\n\"\n\n\necho \"Deleting Caches\"\n# =====================================\n# Run native OS cache cleaning commands\n#\n# rm -fr is done in the next block\n#if type apk >/dev/null 2>&1 2>&1; then\n#    rm -rf -- /var/cache/apk\nif type apt-get >/dev/null 2>&1; then\n    # could accidentally remove things it shouldn't\n    #apt-get autoremove -y\n    echo \"* apt-get clean\"\n    apt-get clean\nelif type yum >/dev/null 2>&1; then\n    # could accidentally remove things it shouldn't\n    #yum autoremove -y\n    echo \"* yum clean all\"\n    yum clean all\nelif type brew >/dev/null 2>&1; then\n    brew cleanup\nfi\n\necho \"Deleting OS Caches\"\n# =========================\n# Delete OS Cache locations\n#\n# safer than for loop - don't risk word splitting with rm\necho \"$cache_list\" |\nwhile read -r directory; do\n    [ -n \"$directory\" ] || continue\n    [ -e \"$directory\" ] || continue\n    echo \"* removing $directory\"\n    rm -rf -- \"$directory\" || :\n    [ -e \"$directory\" ] || continue\n    # shellcheck disable=SC2039\n    if [ \"${EUID:-${UID:-$(id -u)}}\" != 0 ]; then\n        if type sudo >/dev/null 2>&1; then\n            sudo -n rm -rf -- \"$directory\"\n        fi\n    fi\ndone\n\necho \"Deleting Personal Caches\"\n# =============================================================\n# Delete Personal Cache locations & Programming Language Caches\n#\necho \"$personal_cache_list\" |\nwhile read -r directory; do\n    [ -n \"$directory\" ] || continue\n    # posix sh, not as nice as bash but want to use this script from Alpine docker builds\n    if [ \"$(echo \"$directory\" | cut -c1)\" = / ]; then\n        user_home_cache=\"$directory\"\n    else\n        user_home_cache=~/\"$directory\"\n    fi\n    [ -e \"$user_home_cache\" ] || continue\n    echo \"* removing $user_home_cache\"\n    # ~ more reliable than $HOME which could be unset\n    rm -rf -- \"$user_home_cache\" || :\n    # shellcheck disable=SC2039\n    if [ \"${EUID:-${UID:-$(id -u)}}\" != 0 ]; then\n        if type sudo >/dev/null 2>&1; then\n            # in case user home directory is owned by root, do a late stage removal as root\n            sudo -n rm -rf -- \"$user_home_cache\"\n            [ -e \"/root/$directory\" ] || continue\n            echo \"* removing /root/$directory\"\n            sudo -n rm -rf -- \"/root/$directory\"\n        fi\n    fi\ndone\n\necho \"Finished deleting caches\"\n"
  },
  {
    "path": "bin/cocomo_man_years_estimate.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-14 00:11:35 +0700 (Tue, 14 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCalculate Man Years of Work for a given number of lines of code\n\nUses the conservative formula:\n\n2.4 × ( ( Lines of Code / 1000 ) ^ 1.05 ) / 12\n\nGet lines of code for a project using the cloc tool\n\nOr for all original public repos for a given user, this script:\n\n    ../github/github_public_lines_of_code.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<lines_of_code>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nlines_of_code=\"$1\"\n\nif ! is_int \"$lines_of_code\"; then\n    die \"Lines of Code given is not an integer!\"\nfi\n\nkloc=\"$((lines_of_code / 1000))\"\n\nkloc_power=\"$(bc -l <<< \"e(1.05 * l($kloc))\")\"\n\nbc -l <<< \"2.4 * $kloc_power / 12\"\n"
  },
  {
    "path": "bin/copy_to_clipboard.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-24 01:51:08 +0000 (Sat, 24 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCopies the first argument string or standard input to the system clipboard on Linux or Mac\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    cat <<< \"$1\"\nelse\n    # pass through stdin\n    cat\nfi |\nif is_mac; then\n    pbcopy\nelif is_linux; then\n    xclip\nelse\n    echo \"ERROR: OS is not Darwin/Linux\"\n    return 1\nfi\n"
  },
  {
    "path": "bin/crt_hash.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-14 16:22:09 +0100 (Fri, 14 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses OpenSSL to get the sha256 hash of a certificate crt file\n\nUseful to generating the hash needed for joining a Kubernetes node to a cluster eg.\n\nkubeadm join \\\\\n    --token <token> \\\\\n    --discovery-token-ca-cert-hash sha256:<hash> \\\\\n    k8smaster:6443\n\nTo generate this actual command on a live cluster, run the adjacent kubeadm_join_cmd.sh script (which uses this script)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file.crt>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# eg. /etc/kubernetes/pki/ca.crt\ncrt_file=\"$1\"\n\nif ! [ -f \"$crt_file\" ]; then\n    die \"ERROR: file not found: $crt_file\"\nfi\n\nopenssl x509 -pubkey -in \"$crt_file\" |\nopenssl rsa -pubin -outform der 2>/dev/null |\nopenssl dgst -sha256 -hex |\nsed 's/^.* //'\n"
  },
  {
    "path": "bin/crypto_dice_rolls.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-06-12 14:02:26 +0200 (Thu, 12 Jun 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates random dice rolls to be used for TESTING ONLY of a hardware wallet's fidelity using a site like:\n\n    https://bitcoiner.guide/seed/\n\nDefaults to 100 dice rolls as this is how much entropy you need to test a 24 word BIP-39 seed phrase\nas used by Ledger and similar hardware wallets\n\nDO NOT USE THIS FOR YOUR REAL CRYPTO HOLDINGS - if you machine is compromised with malware\nYOU CAN HAVE YOUR CRYPTO STOLEN\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<num_rolls>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nnum=\"${1:-100}\"\n\n# solves this error on macOS:\n#\n#   tr: Illegal byte sequence\n#\nexport LC_ALL=C\n\ntr -dc 1-6 < /dev/urandom |\nhead -c \"$num\" |\nfold -w5 || :\n#paste -sd ' ' -\n\necho\n"
  },
  {
    "path": "bin/curl_auth.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-01-02 21:08:12 +0000 (Wed, 02 Jan 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# used by utils.sh usage()\n# shellcheck disable=SC2034\nusage_description=\"\nRuns curl with either Kerberos SpNego (if \\$KRB5 is set) or a ram file descriptor using \\$PASSWORD or \\$TOKEN to avoid credentials being exposed via process list or command line logging\n\nRequires prefixing url with http:// or https:// to work on older versions of curl in order to parse hostname\nfor constructing the authentication string to be specific to the host as using netrc default login doesn't always work\n\"\n\n\n# shellcheck source=lib/utils.sh disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by utils.sh usage()\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>] <url>\"\n\nif [ $# -lt 1 ]; then\n    # shellcheck disable=SC2119\n    usage\nfi\n\nfor x in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$x\" in\n        -h|--help) usage\n        ;;\n    esac\ndone\n\ncheck_bin curl\n\nUSERNAME=\"${USERNAME:-${USER:-$(whoami)}}\"\n# for APIs like Codeship codeship_api_token.sh in which you must use basic auth and not pass a token\nif [ -z \"${NO_TOKEN_AUTH:-}\" ]; then\n    TOKEN=\"${TOKEN:-${OAUTH2_TOKEN:-${OAUTH_TOKEN:-${JWT_TOKEN:-}}}}\"\nfi\n\n# Only do password mechanism and netrc_contents workaround if not using Kerberos\nif [ -z \"${KRB5:-${KERBEROS:-}}\" ]; then\n    if is_blank \"${PASSWORD:-}\" &&\n       is_blank \"${TOKEN:-}\"; then\n        pass\n    fi\n\n    # ==============================================\n    # option 1\n\n    # needs newer version of curl, otherwise fall back to parsing the hostname and dynamically created netrc contents\n    # curl 7.64.1 (x86_64-apple-darwin19.0) libcurl/7.64.1 (SecureTransport) LibreSSL/2.8.3 zlib/1.2.11 nghttp2/1.39.2\n    # curl 7.35.0 (x86_64-pc-linux-gnu) libcurl/7.35.0 OpenSSL/1.0.1f zlib/1.2.8 libidn/1.28 librtmp/2.3\n    if is_blank \"${TOKEN:-}\" &&\n       is_curl_min_version 7.64; then\n        netrc_contents=\"default login $USERNAME password $PASSWORD\"\n    fi\n\n    # ==============================================\n    # option 2\n    #\n    #hosts=\"$(awk '{print $1}' < ~/.ssh/known_hosts 2>/dev/null | sed 's/,.*//' | sort -u)\"\n\n    # use built-in echo if availble, cat is slow with ~1000 .ssh/known_hosts\n    #if help echo &>/dev/null; then\n    #    netrc_contents=\"$(for host in $hosts; do echo \"machine $host login $USERNAME password $PASSWORD\"; done)\"\n    #else\n    #    # slow fallback with lots of forks\n    #    netrc_contents=\"$(for host in $hosts; do cat <<< \"machine $host login $USERNAME password $PASSWORD\"; done)\"\n    #fi\n\n    # ==============================================\n    # option 3\n\n    # Instead of generating this for all known hosts above just do it for the host extracted from the args url now\n\n    if is_blank \"${TOKEN:-}\" &&\n       [ -z \"${netrc_contents:-}\" ]; then\n        if ! [[ \"$*\" =~ :// ]]; then\n            usage \"http(s):// not specified in URL\"\n        fi\n\n        # sed https* - because sed on Mac doesn't respect either ? or \\? and otherwise I'd have to use:\n        #\n        # perl -pe 's/https?:\\/\\///'\n        #\n        # or else wrap\n        #\n        # if is_mac; then\n        #       sed(){ gsed \"$@\" }\n        # fi\n        #\n        host=\"$(grep -Eom 1 'https?://[^:\\/[:space:]]+' <<< \"$*\" | sed 's,https*://,,' | head -n1)\"\n\n        netrc_contents=\"machine $host login $USERNAME password $PASSWORD\"\n    fi\nfi\n\n# ==============================================\n\nif [ -n \"${KRB5:-${KERBEROS:-}}\" ]; then\n    command curl -u : --negotiate \"$@\"\nelif ! is_blank \"${TOKEN:-${JWT_TOKEN:-}}\"; then\n    if ! is_blank \"${JWT_TOKEN:-}\"; then\n        auth_header=\"${CURL_AUTH_HEADER:-Authorization: JWT} $JWT_TOKEN\"\n    else\n        # OAuth2\n        auth_header=\"${CURL_AUTH_HEADER:-Authorization: Bearer} $TOKEN\"\n    fi\n    if is_curl_min_version 7.55; then\n        # this trick doesn't work, file descriptor is lost by next line\n        #filedescriptor=<(cat <<< \"Private-Token: $GITLAB_TOKEN\")\n        command curl -H @<(cat <<< \"$auth_header\") \"$@\"\n    else\n        command curl -H \"$auth_header\" \"$@\"\n    fi\nelse\n    command curl --netrc-file <(cat <<< \"$netrc_contents\") \"$@\"\nfi\n"
  },
  {
    "path": "bin/curl_with_cookies.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-10-30 16:57:18 +0300 (Thu, 30 Oct 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns curl using a local cookie extracted from Google Chrome or Firefox\n\nFirst extracts the cookies for the given URL, then passes this directly to curl along with the rest of the args\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"$1\"\n\nshift || :\n\ncheck_bin curl\n\nif ! type -P pycookiecheat &>/dev/null; then\n    pip install pycookiecheat\nfi\n\ncookie=\"$(pycookiecheat \"$url\")\"\n\ncookie_header=\"$(jq -r 'to_entries | map(\"\\(.key)=\\(.value)\") | join(\"; \")' <<< \"$cookie\")\"\n\ncurl -H \"Cookie: $cookie_header\" -L \"$@\" \"$url\"\n"
  },
  {
    "path": "bin/debian_netinstall_pxesetup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-06 03:25:58 +0000 (Wed, 06 Mar 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and sets up the Debian Netboot Install PXE to /private/tftpboot on Mac\n\nArch arg defaults to amd64 is not specified\n\nComplete TFP Solution:\n\n1. Run this script\n\n2. download TftpServer to easily GUI start the macOS built-in tftpd server and you're good to go\n\n3. start ISC DHCP server pointing to this machine\n\n\nSee this page for more info:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/dhcp.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<arch>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\narch=\"${1:-amd64}\"\n\nif ! is_mac; then\n    die \"This is only supported on Mac at this time\"\nfi\n\ndir=\"/private/tftpboot\"\n\nsudo mkdir -p -v \"$dir\"\n\nsudo chown -v \"$USER\" \"$dir\"\n\ncd \"$dir\"\n\n# Adapted from here:\n#\n#   https://wiki.debian.org/PXEBootInstall\n\nmirror=deb.debian.org # A CDN backed by cloudflare and fastly currently\n#arch=amd64\n#arch=i386\n#arch=\"$(get_arch)\"  # we are booting in an x86_64 VM, let it be $1 arg to script to override\ndist=stable\n\nwget \"http://$mirror/debian/dists/$dist/main/installer-$arch/current/images/netboot/netboot.tar.gz\"\nwget \"http://$mirror/debian/dists/$dist/main/installer-$arch/current/images/SHA256SUMS\"\nwget \"http://$mirror/debian/dists/$dist/Release\"\nwget \"http://$mirror/debian/dists/$dist/Release.gpg\"\n\nsha256sum -c <(awk '/netboot\\/netboot.tar.gz/{print $1 \" netboot.tar.gz\"}' SHA256SUMS)\n# netboot.tar.gz: OK\n\n\nsha256sum -c <(awk '/[a-f0-9]{64}[[:space:]].*main\\/installer-'\"$arch\"'\\/current\\/images\\/SHA256SUMS/{print $1 \" SHA256SUMS\"}' Release)\n# SHA256SUMS: OK\n\n#gpg --verify Release.gpg Release\n\ntar zxvf netboot.tar.gz\n"
  },
  {
    "path": "bin/decomment.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-11 12:15:47 +0100 (Fri, 11 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Prints files without comments\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"usage: ${##*/} <files>\"\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nstdin=\"\"\nif [ $# -eq 0 ]; then\n    # slurping is not memory efficient, not suitable for large files, but is the only way to be able to\n    # backtracking testing heads for type of decommenting and still retain full data for processing\n    stdin=\"$(cat)\"\nfi\n\nif [ \"$(head -n1 \"$@\" <<< \"$stdin\" | cut -c 1)\" = \"<\" ]; then\n    if type -P decomment-xml.pl &>/dev/null; then\n        decomment-xml.pl \"$@\" <<< \"$stdin\"\n    else\n        echo \"ERROR: decomment-xml.pl from DevOps Perl Tools repo not found in \\$PATH - ensure you have downloaded and built it before running this against XML files\" >&2\n        exit 1\n    fi\nelif [ \"$(head -n1 \"$@\" <<< \"$stdin\" | cut -f 1-2)\" = \"--\" ]; then\n    sed 's/--.*$//; /^[[:space:]]*$/d' \"$@\" <<< \"$stdin\"\nelse\n    sed 's/#.*$//;\n         s/^[[:space:]]*\\/\\/.*$//;\n         s/[[:space:]]\\/\\/.*$//;\n        /^[[:space:]]*$/d' \"$@\" <<< \"$stdin\"\nfi\n"
  },
  {
    "path": "bin/delete_duplicate_files.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-29 12:46:27 +0000 (Thu, 29 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes duplicate files with (N) suffixes in the given or current directory that are 100% exact duplicate matches by checksum\n\nChecks there is a matching basename file without the (N) suffix with the exact same checksum for safety\n\nPrompts to delete per file. To auto-accept deletions, do\n\n    yes | ${0##*/}\n\nThese sorts of duplicates are commonly caused by clicking Download twice in your web browser\n\nI wrote this to run in local crontab to keep my ~/Downloads directory cleaner without wasting my time to clean it up manually\n\nUses adjacent script find_duplicate_files_by_checksum.sh\n\nTested on macOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\ncd \"${1:-.}\"\n\n# read from file descriptor 3 to not slurp stdin which we need for user prompting\nwhile read -r -u 3 checksum filename; do\n    if [[ \"$filename\" =~ \\([[:digit:]]\\) ]]; then\n        log \"Checking if we can safetly remove: $filename\"\n        file_extension=\"${filename##*.}\"\n        # false positive - this works fine\n        # shellcheck disable=SC2295\n        base_filename=\"${filename% ([[:digit:]]).$file_extension}.$file_extension\"\n        if [ -f \"$base_filename\" ]; then\n            base_filename_checksum=\"$(md5sum \"$base_filename\" | awk '{print $1}')\"\n            if [ \"$base_filename_checksum\" = \"$checksum\" ]; then\n                log \"Base filename exists and checksum matches, removing: $filename\"\n                rm -i -v \"$filename\"\n            else\n                log \"Base filename '$base_filename' checksum '$base_filename_checksum' does not match duplicate file checksum '$checksum', skipping removal for safety\"\n            fi\n        else\n            log \"No matching basename file found: $base_filename\"\n            log \"Skipping removal for safety\"\n        fi\n        log\n    fi\ndone 3< <( \"$srcdir/find_duplicate_files_by_checksum.sh\" )\n"
  },
  {
    "path": "bin/delete_empty_dirs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-27 13:17:04 +0000 (Mon, 27 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes all empty directories under the current or given directory\n\nOn Mac pre-deletes .DS_Store files to prevent otherwise empty directories from being retained\n\nTested on Mac OS X\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nstarting_directory=\"${1:-.}\"\n\n# -d / -depth = depth first traversal - avoids find removing the directory and then trying to recurse it, causing the following error:\n# find: ./somedir: No such file or directory\n# -delete implies -depth but it's riskier because although more efficient avoiding forks for each empty dir, if the selector is wrong such as missing the - before -type you'll lose files / data\n#find \"$starting_directory\" -type d -empty -delete\n\n# Mac's rmdir command doesn't have a --verbose switch for feedback, so use GNU version instead\nif is_mac; then\n    find \"$starting_directory\" -type f -name .DS_Store -delete\n    find \"$starting_directory\" -type d -empty -depth -exec grmdir -v {} \\;\nelse\n    find \"$starting_directory\" -type d -empty -depth -exec rmdir -v {} \\;\nfi\n"
  },
  {
    "path": "bin/diff_line_threshold.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-21 18:57:40 +0100 (Sat, 21 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCompares two files vs a line count diff threshold to determine if they are radically different\n\nUsed to avoid overwriting files which are not mere updates but completely different files\n\nThe max_line_diff threshold is 100 is not specified\n\nUsed by the following scripts to avoid overwriting configs in adjacent repos:\n\n../cicd/sync_ci_to_adjacent_repos.sh\n../cicd/sync_configs_to_adjacent_repos.sh\n\nto avoid overwriting master templates in the Templates repo with CI/CD configs from this repo\n\nIf QUIET environment variable is set to any value then doesn't print anything, to not pollute logs for usage in above scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file1> <file2> [<max_line_diff>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nfile1=\"$1\"\nfile2=\"$2\"\n\nline_threshold=\"${3:-100}\"\n\nif [ -n \"${QUIET:-}\" ]; then\n    timestamp(){\n        :\n    }\nfi\n\ntimestamp \"Comparing File 1: $file1\"\ntimestamp \"Comparing File 2: $file2\"\ntimestamp \"Threshold for line difference: $line_threshold\"\n\ndiff_line_count=$(diff -U 0 \"$file1\" \"$file2\" | grep -cE '^[+-]' || :)\n\ntimestamp \"Total Number of Diff Lines: $diff_line_count\"\n\nif [ \"$diff_line_count\" -ge \"$line_threshold\" ]; then\n    timestamp \"Files '$file1' and '$file2' are radically different with $diff_line_count differing lines >= $line_threshold threshold\"\n    timestamp \"Exit: 1\"\n    exit 1\nelse\n    timestamp \"Files '$file1' and '$file2' are not radically different with only $diff_line_count differing lines < $line_threshold threshold\"\n    timestamp \"Exit: 0\"\nfi\n"
  },
  {
    "path": "bin/disable_swap.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-14 19:26:06 +0100 (Fri, 14 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables Swap on Linux\n\nDesigned to be called modularly from other provisioners to reuse code between scripts\n\neg. Vagrant provisioners that differ by OS / setup can call this from /github/bash-tools/disable_swap.sh within their scripts\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nos=\"$(uname -s)\"\n\nif [ \"$os\" != Linux ]; then\n    echo \"OS '$os' != Linux, aborting disabling swap\"\n    exit 1\nfi\n\n#echo 0 > /proc/sys/vm/swappiness\n\ntimestamp \"Disabling All Swap\"\nswapoff -a\n\ntimestamp \"Commenting out any Swap lines in /etc/fstab\"\nsed -i 's,\\(/.*[[:space:]]none[[:space:]]*swap[[:space:]]\\),#\\1,' /etc/fstab\n\ntimestamp \"Swap disabled\"\n"
  },
  {
    "path": "bin/disk-read-random.fio",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-12-10 00:37:58 -0600 (Wed, 10 Dec 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[global]\ndirectory=./\nsize=2G\ndirect=1\niodepth=32\nioengine=psync\ntime_based=0\nnumjobs=1\ngroup_reporting=1\n\n[rand-read]\nrw=randread\nbs=4k\nfilename=rand_read_testfile\nunlink=1\n"
  },
  {
    "path": "bin/disk-read-sequential.fio",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-12-10 00:37:58 -0600 (Wed, 10 Dec 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[global]\ndirectory=./\nsize=2G\ndirect=1\niodepth=32\nioengine=psync\ntime_based=0\nnumjobs=1\ngroup_reporting=1\n\n[seq-read]\nrw=read\nbs=1M\nfilename=seq_read_testfile\nunlink=1\n"
  },
  {
    "path": "bin/disk-tests.fio",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-12-10 00:37:58 -0600 (Wed, 10 Dec 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[global]\ndirectory=./\nsize=2G\ndirect=1\niodepth=32\nioengine=psync\ntime_based=0\nnumjobs=1\ngroup_reporting=1\n\n; ---------------------------\n; 1. Sequential Read\n; ---------------------------\n[seq-read]\nrw=read\nbs=1M\nfilename=seq_read_testfile\nunlink=1\n\n; ---------------------------\n; 2. Sequential Write\n; ---------------------------\n[seq-write]\nrw=write\nbs=1M\nfilename=seq_write_testfile\nunlink=1\n\n; ---------------------------\n; 3. Random Read\n; ---------------------------\n[rand-read]\nrw=randread\nbs=4k\nfilename=rand_read_testfile\nunlink=1\n\n; ---------------------------\n; 4. Random Write\n; ---------------------------\n[rand-write]\nrw=randwrite\nbs=4k\nfilename=rand_write_testfile\nunlink=1\n"
  },
  {
    "path": "bin/disk-write-random.fio",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-12-10 00:37:58 -0600 (Wed, 10 Dec 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[global]\ndirectory=./\nsize=2G\ndirect=1\niodepth=32\nioengine=psync\ntime_based=0\nnumjobs=1\ngroup_reporting=1\n\n[rand-write]\nrw=randwrite\nbs=4k\nfilename=rand_write_testfile\nunlink=1\n"
  },
  {
    "path": "bin/disk-write-sequential.fio",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-12-10 00:37:58 -0600 (Wed, 10 Dec 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[global]\ndirectory=./\nsize=2G\ndirect=1\niodepth=32\nioengine=psync\ntime_based=0\nnumjobs=1\ngroup_reporting=1\n\n[seq-write]\nrw=write\nbs=1M\nfilename=seq_write_testfile\nunlink=1\n"
  },
  {
    "path": "bin/disk_speed_read_random_dd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a random I/O read test from the given file in order to test the random read speed\n\nUses dd and bypasses filesystem cache for a more accurate test\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\nGenerate a file to read using the adjacent script:\n\n    NODELETE=1 disk_write_speed_sequential_dd.sh\n\nIf a directory is given will look for the following file generated by the above script:\n\n    disk_speed_test_sequential_dd.bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file_or_directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nfile=\"${1:-${PWD:-$(pwd)}}\"\nif [ -d \"$file\" ]; then\n    file=\"$file/disk_speed_test_sequential_dd.bin\"\nfi\nfile=\"${file//\\/\\//\\/}\"\n\ntimestamp \"Random read test from: $file\"\necho >&2\n\ndd if=\"$file\" of=/dev/null bs=64m iflag=direct\n\necho >&2\ntimestamp \"Random read test completed\"\n"
  },
  {
    "path": "bin/disk_speed_read_random_fio.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nfio_config=\"$srcdir/disk-read-random.fio\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a random I/O read test from the current or given directory in order to test the random read speed\n\nUses fio with the settings found in: $fio_config\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-${PWD:-$(pwd)}}\"\n\ncd \"$dir\"\n\ntimestamp \"Random read test in directory: $dir\"\necho >&2\n\nfio \"$fio_config\"\n\necho >&2\ntimestamp \"Random read test completed\"\n"
  },
  {
    "path": "bin/disk_speed_read_sequential_dd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a sequential read test from the given file in order to test the sequential read speed\n\nUses dd and bypasses filesystem cache for a more accurate test\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\nGenerate a file to read using the adjacent script:\n\n    NODELETE=1 disk_write_speed_sequential_dd.sh\n\nIf a directory is given will look for the following file generated by the above script:\n\n    disk_speed_test_sequential_dd.bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file_or_directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nfile=\"${1:-${PWD:-$(pwd)}}\"\nif [ -d \"$file\" ]; then\n    file=\"$file/disk_speed_test_sequential_dd.bin\"\nfi\nfile=\"${file//\\/\\//\\/}\"\n\ntimestamp \"Sequential read test from: $file\"\necho >&2\n\ndd if=\"$file\" of=/dev/null bs=64m iflag=direct\n\necho >&2\ntimestamp \"Sequential read test completed\"\n"
  },
  {
    "path": "bin/disk_speed_read_sequential_fio.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nfio_config=\"$srcdir/disk-read-sequential.fio\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a sequential read test in the current or given directory in order to test the sequential read speed\n\nUses fio with the settings found in: $fio_config\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-${PWD:-$(pwd)}}\"\n\ncd \"$dir\"\n\ntimestamp \"Sequential read test in directory: $dir\"\necho >&2\n\nfio \"$fio_config\"\n\necho >&2\ntimestamp \"Sequential read test completed\"\n"
  },
  {
    "path": "bin/disk_speed_write_random_fio.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nfio_config=\"$srcdir/disk-write-random.fio\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a sequential write test to a file in the given or current directory in order to test the sequential write speed\n\nUses fio with the settings found in: $fio_config\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\nWARNING: Don't re-run this on SSDs frequently as they have a limited number of writes and you'll wear the disk out\nprematurely\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-${PWD:-$(pwd)}}\"\n\ncd \"$dir\"\n\ntimestamp \"Random write test in directory: $dir\"\necho >&2\n\nfio \"$fio_config\"\n\necho >&2\ntimestamp \"Write test completed\"\n"
  },
  {
    "path": "bin/disk_speed_write_sequential_dd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a sequential write test to a file in the given or current directory in order to test the sequential write speed\n\nUses dd and bypasses filesystem cache for a more accurate test\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\nWARNING: Don't re-run this on SSDs frequently as they have a limited number of writes and you'll wear the disk out\nprematurely\n\nRemoves the test file upon completion or Control-C (don't Control-C multiple times or it may not remove file)\n\nSet environment variable NODELETE to any value if you want to retain the file in the case of successful completion\nfor use with the adjacent script:\n\n    disk_read_speed_sequential_dd.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-${PWD:-$(pwd)}}\"\nfile=\"$dir/disk_speed_test_sequential_dd.bin\"\nfile=\"${file//\\/\\//\\/}\"\n\n#if [ -e \"$file\" ]; then\n#    die \"ERROR: file already exists, aborting for safety: $file\"\n#fi\n\ntrap_cmd \"rm -f '$file'\"\n\ntimestamp \"Sequential write test to: $file\"\necho >&2\n\n# conv=sync - pads blocks to the full amount, not needed with /dev/zero which will fully pad\n#dd if=/dev/zero of=\"$file\" bs=1m count=1024\n# write 4GB in higher block size to stress the disk better if it's an SSD\n# oflag=direct - bypass filesystem cache for a more realistic speed test\ndd if=/dev/zero of=\"$file\" bs=64m count=64 oflag=direct\n\necho >&2\ntimestamp \"Write test completed\"\nif [ -n \"${NODELETE:-}\" ]; then\n    timestamp \"Retaining test file: $file\"\n    untrap\nelse\n    timestamp \"Removing test file: $file\"\nfi\n"
  },
  {
    "path": "bin/disk_speed_write_sequential_fio.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 21:21:30 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nfio_config=\"$srcdir/disk-write-sequential.fio\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a sequential write test to a file in the given or current directory in order to test the sequential write speed\n\nUses fio with the settings found in: $fio_config\n\nUseful for testing:\n\n- different disks' speeds\n- different cables' speed with the same disk\n- different ports' speeds on Macs (right ports may be slower)\n\nI wrote this because I discovered a huge performance and estimated time to restore speed difference using\nmacOS Time Machine recovery while using USB 2 vs USB 3 cables with the same SanDisk Extreme Pro SSD external backup disk\n\nWARNING: Don't re-run this on SSDs frequently as they have a limited number of writes and you'll wear the disk out\nprematurely\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-${PWD:-$(pwd)}}\"\n\ncd \"$dir\"\n\ntimestamp \"Sequential write test in directory: $dir\"\necho >&2\n\nfio \"$fio_config\"\n\necho >&2\ntimestamp \"Write test completed\"\n"
  },
  {
    "path": "bin/download_url_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 23:01:46 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads a given URL file using wget or curl\n\nFirst tries wget if available with continuation and noclobber support\n\nAlternatively falls back to using curl with atomic tmp and move to not cause\nrace conditions with files that are in frequent use\n\nDesigned to be called from adjacent download_*.sh scripts to deduplicate code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ndownload_url=\"$1\"\n\ntimestamp \"Downloading: $download_url\"\necho >&2\n\nfilename=\"${download_url##*/}\"\n\nif type -P wget &>/dev/null; then\n    wget -cO \"$filename\" \"$download_url\"\nelse\n    tmpfile=\"$(mktemp)\"\n    curl --fail \"$download_url\" > \"$tmpfile\"\n    unalias mv &>/dev/null || :\n    mv -fv \"$tmpfile\" \"$filename\"\nfi\n\ntimestamp \"Download complete: $filename\"\n"
  },
  {
    "path": "bin/elasticsearch_decommission_node.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-02 15:06:35 +0000 (Mon, 02 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"\nSimple script to trigger a background decommission of an Elasticsearch node from the local cluster\n\nDefaults to connecting to the Elasticsearch cluster via the node localhost:9200\nset \\$ELASTICSEARCH_HOST and \\$ELASTICSEARCH_PORT to override this\nset \\$ELASTICSEARCH_SSL to any value to enable SSL (ignores ssl validation as this is usually self-signed)\n\n${0##*/} <node_ip> [curl_options]\n\neg. ${0##*/} 192.168.1.23\n\"\n    exit 3\n}\n\nif [ $# -ne 1 ]; then\n    usage\nfi\n\nif [[ \"$1\" =~ -.* ]]; then\n    usage\nfi\n\nnode_ip=\"$1\"\nhost=\"${ELASTICSEARCH_HOST:-localhost}\"\nport=\"${ELASTICSEARCH_PORT:-9200}\"\nhttp=http\nif [ -n \"${ELASTICSEARCH_SSL:-}\" ]; then\n    http=https\nfi\n\n# could make this better by checking octets etc like my Python / Perl libraries\n# but don't want this script to get too heavy with dependencies\nif ! [[ \"$node_ip\" =~ ^[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}$ ]]; then\n    echo \"Invalid node IP given: $node_ip\"\n    exit 1\nfi\n\n# want curl opts split\n# shellcheck disable=SC2086\nif curl -X PUT -k \"${@:2}\"\n    \"$http://$host:$port/_cluster/settings\" \\\n     -H 'Content-Type: application/json' \\\n     -d \"{ \\\"transient\\\" :{ \\\"cluster.routing.allocation.exclude._ip\\\" : \\\"$node_ip\\\" } }\"; then\n    printf '\\nSuccess. Now wait for background replication to migrate shards off node %s \\n' \"$1\"\nelse\n    printf '\\nFailed\\n'\nfi\n"
  },
  {
    "path": "bin/exec_interactive.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 12:51:17 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# cannot -set -o pipefail because some docker images version of 'sh' do not support it, namely debian and ubuntu\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# cannot allow set -e because it will cause an exit before the exec to interactive\n(\nexec \"${SHELL:-sh}\" -i 3<<EOF 4<&0 <&3\n  set +e\n    $@\n  exec 3>&- <&4\nEOF\n)\n"
  },
  {
    "path": "bin/file_extensions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-25 04:06:24 +0700 (Sat, 25 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all unique file extensions under the current or given directory\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nfind \"${1:-.}\" -type f -iname '*.*' |\nsed 's/.*\\.//' |\nsort -u\n"
  },
  {
    "path": "bin/find_broken_links.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 13:03:12 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Crawls a URL argument and finds broken links, throttling to 1 link every 2 seconds\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$*\"\n    fi\n    cat <<EOF\n\nusage: ${0##*/} <url>\n\nEOF\n    exit 3\n}\n\nif [ $# != 1 ]; then\n    usage \"no url argument given\"\nfi\n\nurl=\"$1\"\n\nif ! [[ \"$url\" =~ https?:// ]]; then\n    usage \"invalid url argument, must match https?://\"\nfi\n\ntmp=\"$(mktemp)\"\n\n# want splitting\n# shellcheck disable=SC2086\ntrap 'rm -- \"$tmp\"' $TRAP_SIGNALS\n\n# --spider = don't download\n# -r = recursive\n# -nd / --no-directories = don't create local dirs representing structure\n# -nv / --no-verbose = give concise 1 liner information\n# -l 1 = crawl 1 level deep (may need to tune this), set to 'inf' for infinite\n# -w 2 = wait for 2 secs between requests to avoid tripping defenses\n# -H / --span-hosts = follows subdomains + external sites\n# -o \"$tmp\" = output to tmp, now replaced with tee\n# -N = --timestamping = don't download unless newer than local copy, use with mirroring not spidering\n# -nH = -–no-host-directories\n# -P = --directory-prefix (use instead of host directories)\n# -m = --mirror (-r -N -l inf –no-remove-listing)\nwget \\\n     --spider \\\n     -r \\\n     -nd \\\n     -nv \\\n     -l 1 \\\n     -w 2 --random-wait \\\n     -H \\\n     \"$url\" 2>&1 |\ntee \"$tmp\"\nif ! grep -q 'Found no broken links' \"$tmp\"; then\n    echo\n    echo \"Broken links:\"\n    grep -B1 'broken link!' \"$tmp\"\nfi\n"
  },
  {
    "path": "bin/find_broken_symlinks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-10 03:16:48 +0100 (Wed, 10 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds symlinks which point to non-existent files\n\nSee Also:\n\n    checks/check_broken_symlinks.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nexitcode=0\n\nfind \"$dir\" -type l |\nwhile read -r symlink; do\n    target=\"$(readlink -f \"$symlink\")\"\n    if ! [ -e \"$target\" ]; then\n        echo \"Symlink broken: $symlink -> $target\"\n        exitcode=1\n    fi\ndone\n\nexit $exitcode\n"
  },
  {
    "path": "bin/find_duplicate_files_by_checksum.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 13:05:24 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nFinds duplicate files by file checksum\n\nOnly compares files with the same byte counts for efficiency, using the adjacent find_duplicate_files_by_size.sh\nas a pre-filter to speed up the process\n\nOutput format:\n\n<md5_checksum>      <filename>\n\nFor a much more sophisticated duplicate file finder utilizing size, checksums, basenames and\neven partial basenames via regex match see\n\nfind_duplicate_files.py\n\nin the DevOps Python tools repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir1> <dir2> ...]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhelp_usage \"$@\"\n\nlast_checksum=\"\"\nlast_filename=\"\"\nlast_printed=0\n\n# discard size and use checksum as next level filter\n# shellcheck disable=SC2034\n\"$srcdir/find_duplicate_files_by_size.sh\" \"$@\" |\nwhile read -r size filename; do\n    md5sum \"$filename\"  # outputs <checksum> <filename>\ndone |\nsort -k1n |\nwhile read -r checksum filename; do\n    if [ \"$checksum\" = \"$last_checksum\" ]; then\n        if [ \"$last_printed\" = 0 ]; then\n            printf '%s\\t%s\\n' \"$last_checksum\" \"$last_filename\"\n        fi\n        printf '%s\\t%s\\n' \"$checksum\" \"$filename\"\n        last_printed=1\n    else\n        last_printed=0\n    fi\n    last_checksum=\"$checksum\"\n    last_filename=\"$filename\"\ndone\n"
  },
  {
    "path": "bin/find_duplicate_files_by_size.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 12:31:21 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ndefault_exclude_regex='.DS_Store$|\\.nfo$|\\.part$'\n\n# shellcheck disable=SC2034\nusage_description=\"\nFinds duplicate files by file size in bytes\n\nOutput format:\n\n<size_in_bytes>     <filename>\n\nDefault exclusion list ERE regex is:\n\n    $default_exclude_regex\n\nYou can override this by setting environment variable \\$EXCLUDE_REGEX\n\nFor a much more sophisticated duplicate file finder utilizing size, checksums, basenames and\neven partial basenames via regex match see\n\nfind_duplicate_files.py\n\nin the DevOps Python tools repo:\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir1> <dir2> ...]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhelp_usage \"$@\"\n\nexclude_regex=\"${EXCLUDE_REGEX:-$default_exclude_regex}\"\n\nlast_size=\"\"\nlast_filename=\"\"\nlast_printed=0\n\n# GNU coreutils du has bytes, whereas Mac's du only goes to the less granular blocks which is less accurate\nif is_mac; then\n    du(){\n        gdu \"$@\"\n    }\n    export -f du\nfi\n\nwhile read -r size filename; do\n    if [ \"$size\" = \"$last_size\" ]; then\n        if [ \"$last_printed\" = 0 ]; then\n            printf '%s\\t%s\\n' \"$last_size\" \"$last_filename\"\n        fi\n        printf '%s\\t%s\\n' \"$size\" \"$filename\"\n        last_printed=1\n    else\n        last_printed=0\n    fi\n    last_size=\"$size\"\n    last_filename=\"$filename\"\ndone < <(\n    for dir in \"${@:-$PWD}\"; do\n        find \"$dir\" -type f\n    done |\n    # find -print0 breaks this so do it afterwards using tr before passing to xargs -0\n    { grep -Evai -e \"$exclude_regex\" || : ; } |\n    tr '\\n' '\\0' |\n    xargs -0 sh -c 'du -ab \"$@\"' |\n    sort -k1n\n)\n"
  },
  {
    "path": "bin/find_duplicate_lines.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 10:11:25 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find duplicate lines across files\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"<files>\"\n\nfor x in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nfound=0\n\nwhile read -r line; do\n    grep -Fx \"$line\" \"$@\"\n    ((found + 1))\ndone < <(\n    sed 's/#.*//;\n         s/^[[:space:]]*//;\n         s/[[:space:]]*$//;\n         /^[[:space:]]*$/d;' \"$@\" |\n    sort |\n    uniq -d\n)\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "bin/find_hanging_mount_point.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2018-01-12 19:13:34 +0000 (Fri, 12 Jan 2018)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find a hanging mount point on Linux (often caused by NFS)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    cat <<EOF\n\nScript to find a hanging mount point by iterating over each of them in turn, printing the name before reading from them\n\nThe hanging mount point will be the one you get stuck on, so you can debug that one (usually an NFS mount point)\n\nusage:  ${0##*/} [ --include <posix_regex> ] [ --exclude <posix_regex> ]\n\nEOF\n    exit 3\n}\n\ninclude_regex=\".*\"\nexclude_regex=\"\"\n\nuntil [ $# -lt 1 ]; do\n    case $1 in\n        -i|--include) include_regex=\"$2\"\n                      shift\n                      ;;\n        -e|--exclude) exclude_regex=\"$2\"\n                      shift\n                      ;;\n         *) usage\n            ;;\n    esac\n    shift\ndone\n\nif [ \"$(uname -s)\" != \"Linux\" ]; then\n    echo \"Error: this only runs on Linux\"\n    exit 1\nfi\n\nif ! [ -f /proc/mounts ]; then\n    echo \"Error: /proc/mounts not found\"\n    exit 1\nfi\n\necho \"Testing listing in each mount point:\"\necho\nawk '{print $2}' /proc/mounts |\ngrep -Ev ' cgroup ' |\ngrep -E \"$include_regex\" |\n# default blank here would exclude everything, switched to test within loop only if exclude_regex is not blank\n#grep -Ev \"$exclude_regex\" |\nwhile read -r mountpoint; do\n    [ -d \"$mountpoint\" ] || continue\n    if [ \"$mountpoint\"       = \"/proc\"  ] ||\n       [ \"$mountpoint\"       = \"/sys\"   ] ||\n       [ \"$mountpoint\"       = \"/dev\"   ] ||\n       [ \"${mountpoint:0:6}\" = \"/proc/\" ] ||\n       [ \"${mountpoint:0:5}\" = \"/sys/\"  ] ||\n       [ \"${mountpoint:0:5}\" = \"/dev/\"  ]; then\n        continue\n    fi\n    if [[ -n \"$exclude_regex\" && \"$mountpoint\" =~ $exclude_regex ]]; then\n        continue\n    fi\n    echo -n \"$mountpoint:  \"\n    ls -l \"$mountpoint\" &>/dev/null\n    echo \"OK\"\ndone\necho\necho \"Finished\"\n"
  },
  {
    "path": "bin/find_hardlinks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-28 01:16:44 +0000 (Wed, 28 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all hardlinks to a given file by searching the mount point for the same inode number of a file\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nfilename=\"$1\"\n\ninode_number=\"$(stat -f \"%i\" \"$filename\")\"\n\nmount_point=\"$(df \"$filename\" | awk '{print $NF}' | tail -n 1)\"\n\nfind \"$mount_point\" -xdev -inum \"$inode_number\" -print\n"
  },
  {
    "path": "bin/find_lock.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-28 01:50:14 +0100 (Sat, 28 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTries to find if a lockfile is used in the given or current working directory\nby taking snapshots of the file list before and after a prompt in which you should open/close an application\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nif ! [ -d \"$dir\" ]; then\n    usage \"Non-existent directory given: $dir\"\nfi\n\ntimestamp \"Taking 'before' file listing snapshot\"\nfilelist_before=\"$(find \"$dir\" -type f)\"\necho >&2\n\nread -r -p 'Press Enter after opening/closing an application to take another snapshot to see if any files have changed'\n\ntimestamp \"Taking 'after' file listing snapshot\"\nfilelist_after=\"$(find \"$dir\" -type f)\"\necho >&2\n\ntimestamp \"Comparing File Listing Snapshots\"\necho >&2\n\n# so many ugly solutions to only getting lines changed and nothing else\n# if you have a more elegant solution please send it to me\nfilelist_diff=\"$(\n    diff -u <(echo \"$filelist_before\") <(echo \"$filelist_after\") |\n    sed -n '/^[+-]/ p' |\n    # diff fails the pipeline on non-zero exit code if any difference found\n    sed '\n        /^[+-]\\{3\\}/d ;\n        s/^[+-]//;\n    ' |\n    while read -r filename; do\n        readlink -f \"$filename\"\n    done || :\n)\"\n\nif [ -n \"$filelist_diff\" ]; then\n    timestamp \"File existence changes (may or may not be lockfiles):\"\n    echo >&2\n    echo \"$filelist_diff\"\nelse\n    timestamp \"No file existence changes found between snapshots\"\nfi\n"
  },
  {
    "path": "bin/find_symlinks_to_other_directories.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 14:34:11 +0700 (Wed, 08 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds symlinks to other directories under the given path\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<base_path>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [ -L \"$1\" ]; then\n    ls -l \"$1\"\nelse\n    find \"$1\" -type l -exec ls -l {} \\;\nfi |\ngrep -E -- '[[:space:]]->[[:space:]]+.*/' |\nawk '{$1=$2=$3=$4=$5=$6=$7=$8=\"\"; print}'\n"
  },
  {
    "path": "bin/foreach_path_bin.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 13:15:00 +0700 (Wed, 08 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns each binary of the given name found in \\$PATH with the args given\n\nUseful to find all the installed versions of a program in different paths eg. ~/bin/ vs /usr/local/bin/\n\nEg.\n\n    ${0##*/} terraform --version\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"binary <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nbinary=\"$1\"\n\nshift || :\n\n# sed used to regularize /path/to/bin and /path/to//bin caused by multiple path additions some with or without trailing\n# slashes leading to double slashes in the path. By regularizing them we are then able to dedupe them properly\n#\nbinaries=\"$(which -a \"$binary\" | sed 's|//*|/|g' | sort -u)\"\n\nwhile read -r binary; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $binary\" >&2\n    echo \"# ============================================================================ #\" >&2\n    echo >&2\n    ls -l \"$binary\" >&2\n    echo >&2\n    \"$binary\" \"$@\"\n    echo >&2\n    echo >&2\ndone <<< \"$binaries\"\n"
  },
  {
    "path": "bin/grep_or_append.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 12:55:07 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each line passed in via standard input, checks the given file argument and appends each line that doesn't already exist within the file\n\nThis is re-evaluated for each line, so acts as a 'uniq' filter for input lines too\n\nSee vagrant/provision.sh for an example of this in use\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfilename=\"$1\"\n\nwhile read -r line; do\n    # if file doesn't exist will append to create it\n    grep -Fxq \"$line\" \"$filename\" 2>/dev/null ||\n    echo \"$line\" >> \"$filename\"\ndone\n"
  },
  {
    "path": "bin/hackercase.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 'Hacking the Planet!'\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-05 14:35:55 +0300 (Mon, 05 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChange input to Hackercase => h4cK3Rc453 for l33t speak\n\nAccepts a string, file or standard input\n\nOutputs to stderr and copies to clipboard for convenience\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_or_file>]\"\n\nhelp_usage \"$@\"\n\n\nif [ $# -eq 0 ]; then\n    log \"Reading from standard input\"\n    cat \"$*\"\nelse\n    if [[ \"$1\" =~ ^.?/ ]] || [ -f \"$1\" ]; then\n        log \"Reading from files: $*\"\n        cat \"$@\"\n    else\n        echo \"$*\"\n    fi\nfi |\nspasticcase.sh |\nsed '\n    s/a/4/gi;\n    s/b/8/gi;\n    s/e/3/gi;\n    s/i/1/gi;\n    s/o/0/gi;\n    # these reduce readability\n    #s/g/6/gi;\n    #s/s/5/gi;\n    #s/t/7/gi;\n' |\ntee >(\"$srcdir/copy_to_clipboard.sh\")\n# copies to clipboard and also sends to stdout to allow further pipeline processing\n"
  },
  {
    "path": "bin/headtail.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-25 19:03:48 +0100 (Sat, 25 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nHead and Tail input files or standard input\n\nFor a better version written in Python see the adjacent DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\nFor something as simple as just the first and last line, you could instead do:\n\n    sed -n '1p;\\$p'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[-n <num_lines>] [<file1> <file2> ...]\n\n--num-lines     Number of lines to print from each file or stdin (default: 10)\"\n\n\nhelp_usage \"$@\"\n\nfiles=()\n\nuntil [ $# -lt 1 ]; do\n    case \"$1\" in\n     -n|--num)  num_lines=\"${2:-}\"\n                shift || :\n                ;;\n            *)  files+=(\"$1\")\n                ;;\n    esac\n    shift || :\ndone\n\nnum_lines=\"${num_lines:-10}\"\n\ndocsep(){\n    if [ \"${#files[@]}\" -gt 1 ]; then\n        echo '======' >&2\n    fi\n}\n\nif [ \"${#files[@]}\" -gt 0 ]; then\n    for filename in \"${files[@]}\"; do\n        if [ \"$(wc -l \"$filename\" | awk '{print $1}')\" -lt $((2*num_lines)) ]; then\n            cat \"$filename\"\n            continue\n        fi\n        head -n \"$num_lines\" \"$filename\"\n        tail -n \"$num_lines\" \"$filename\"\n        docsep\n    done\nelse\n    output=\"$(cat)\"\n    if [ \"$(wc -l <<< \"$output\" | awk '{print $1}')\" -lt $((2*num_lines)) ]; then\n        echo \"$output\"\n    else\n        head -n \"$num_lines\" <<< \"$output\"\n        tail -n \"$num_lines\" <<< \"$output\"\n    fi\nfi\n"
  },
  {
    "path": "bin/hexencode.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-21 22:50:07 +0700 (Fri, 21 Feb 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick command line Hex encoding\n#\n# Different to URL encoding in that urlencoding will leave dashes,\n# but for Shield.io badges I need to encode the dashes to fit within each badge token\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif uname | grep -q Darwin; then\n    sed(){\n        gsed \"$@\"\n    }\n    export -f sed\nfi\n\nif [ $# -gt 0 ]; then\n    echo \"$@\"\nelse\n    cat\nfi |\nif type -P hexdump &>/dev/null; then\n    hexdump -ve '/1 \" %02x\"' |\n    sed 's/ /%/g'\nelif type -P od &>/dev/null; then\n    od -A n -t x1 |\n    sed 's/[[:space:]]\\+$//; s/[[:space:]]\\+/%/g'\nelse\n    echo \"Neither 'hexdump' nor 'od' commands were found in PATH\" >&2\n    exit 1\nfi |\nsed 's/%0a$//' |\ntr '[:lower:]' '[:upper:]'\necho\n"
  },
  {
    "path": "bin/htmldecode.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  stdin: Driving &amp; road trips\n#\n#  Author: Hari Sekhon\n#  Date: 2025-12-23 22:01:42 -0600 (Tue, 23 Dec 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDecodes HTML encoding\n\nDetects available tools such as Perl, Python or xmlstarlet and uses whatever is available\n\nWorks like a standard filter program, takes file arguments for contents or reads from standard input\n\nWriten to clean up Spotify playlist descriptions such as:\n\n    Driving &amp; road trips ...\n\nto\n\n    Driving & road trips...\n\nUsed by spotify_playlist_description.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ncat \"$@\" |\nif type -P perl &>/dev/null &&\n   perl -MHTML::Entities -e '' &>/dev/null; then\n    log \"Decoding HTML using Perl\"\n    perl -MHTML::Entities -pe 'decode_entities($_)'\nelif type -p python3 &>/dev/null &&\n    log \"Decoding HTML using Python\"\n    python3 -c 'import html' &>/dev/null; then\n    python3 -c 'import sys, html; sys.stdout.write(html.unescape(sys.stdin.read()))'\nelif type -p xmlstarlet; then\n    log \"Decoding HTML using xmlstarlet\"\n    xmlstarlet unesc\nelse\n    echo \"ERROR: neither Perl HTML::Entities nor xmlstarlet are available\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "bin/http_duplicate_urls.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-12-31 17:45:56 +0000 (Sun, 31 Dec 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds duplicate URLs in a given web page\n\nThe output is sorted in ascending order of the number of duplicates found for each URL because on large web pages with lots of duplicates I tend to focus on the most duplicated URLs first which will then be right above your prompt for convenience\n\nFeel free to sort or 'tac' to reverse the order, or you can use tail if you're only interested in the N most duplicated URLs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"$1\"\n\ncurl \"$url\" |\ngrep -Eo 'https?://[^[:space:]\"'\"'\"'<>]+' |\nsort |\nuniq -c |\nsort -k1n |\ngrep -Ev '^[[:space:]]+1[[:space:]]+' || :\n"
  },
  {
    "path": "bin/jsondiff.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 17:00:20 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Diffs 2 JSON files given as arguments\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"usage: ${0##*/} file1.json file2.json\"\n    exit 3\n}\n\nif ! [ $# -eq 2 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ndiff <(jq -S . \"$1\") <(jq -S . \"$2\")\n"
  },
  {
    "path": "bin/keycloak.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-09 17:05:31 +0000 (Wed, 09 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Start a quick local Concourse CI\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nBoots Keycloak in Docker\n\n- boots Keycloak container in Docker\n- configures admin user\n- prints admin credentials\n- prints Keycloak UI URL and opens it in browser\n\n    ${0##*/} [up]\n\n    ${0##*/} down\n\n    ${0##*/} ui     - prints the KeycloakServer URL and automatically opens in browser\n\nIdempotent, you can re-run this and continue from any stage\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nexport KEYCLOAK_HOST=\"${DOCKER_HOST:-localhost}\"\nexport KEYCLOAK_PORT=8080\n\nexport KEYCLOAK_URL=\"http://$KEYCLOAK_HOST:$KEYCLOAK_PORT/admin\"\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/keycloak.yml\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting Keycloak:\"\n    docker-compose up -d \"$@\"\n    echo >&2\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo\n    echo \"Keycloak URL:  $KEYCLOAK_URL\"\n    echo\n    echo \"Keycloak user: ${KEYCLOAK_ADMIN:-admin}\"\n    echo \"Keycloak password: ${KEYCLOAK_ADMIN_PASSWORD:-admin}\"\n    echo\n    open \"$KEYCLOAK_URL\"\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo >&2\n    exit 0\nfi\n\nwhen_url_content 90 \"$KEYCLOAK_URL\" '(?i:keycloak)'\n\nexec \"$0\" ui\n"
  },
  {
    "path": "bin/ldap_group_recurse.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-03-14 19:08:01 +0000 (Thu, 14 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$@\" >&2\n        echo >&2\n    fi\n    cat >&2 <<EOF\n\nRecurses AD LDAP for all users and groups which are members of a given group DN\n\nDumps LDAP user and group objects, follows group nesting\n\nUses Microsoft Active Directory LDAP extension, so is not portable to other LDAP servers\n\nSee the python version in the DevOps Python Tools repo for a more generalized version with nicer control and output\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n\nusage: ${0##*/} <group_dn> [<attribute_filter>]\n\n\nEOF\n    exit 3\n}\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ $# -lt 1 ]; then\n    usage \"no group DN given\"\nfi\n\ngroup_dn=\"$1\"\nshift\n\n\"$srcdir/ldapsearch.sh\" \"(&(|(objectClass=user)(objectClass=group))(memberOf:1.2.840.113556.1.4.1941:=$group_dn))\" \"$@\"\n#\"$srcdir/ldapsearch.sh\" \"(&(objectClass=user)(memberOf:1.2.840.113556.1.4.1941:=$group_dn))\" $@\n#\"$srcdir/ldapsearch.sh\" \"(memberOf:1.2.840.113556.1.4.1941:=$group_dn)\" $@\n"
  },
  {
    "path": "bin/ldap_user_recurse.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-03-14 19:08:01 +0000 (Thu, 14 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$@\" >&2\n        echo >&2\n    fi\n    # multiple ${0##*/} inside here document causes usage to not be rendered, must be a bash bug\n    script=\"${0##*/}\"\n    cat >&2 <<EOF\n\nRecurses AD LDAP for all groups for which a given user DN belongs\n\nDumps LDAP group objects, follows group nesting\n\nUses Microsoft Active Directory LDAP extension, so is not portable to other LDAP servers\n\nSee the python version in the DevOps Python Tools repo for a more generalized version with nicer control and output\n\nhttps://github.com/HariSekhon/DevOps-Python-tools\n\n\nusage: $script <user_dn> [<attribute_filter>]\n\n$script CN=hari,OU=Users,DC=myDomain,DC=com\n\nExample: if you don't know the DN and just want to search on any attribute such as CN, UID or sAMAccountName, then this is useful\n\n$script \\$(./ldapsearch.sh cn=hari dn | awk '/^dn: /{print \\$2; exit}')\n\nEOF\n    exit 3\n}\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ $# -lt 1 ]; then\n    usage \"no user DN given\"\nfi\n\nuser_dn=\"$1\"\nshift\n\n\"$srcdir/ldapsearch.sh\" \"(&(objectClass=group)(member:1.2.840.113556.1.4.1941:=$user_dn))\" \"$@\"\n"
  },
  {
    "path": "bin/ldapsearch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-01 16:49:45 +0000 (Fri, 01 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# USAGE: (will default to your $USER account, infer domain base and prompt for password)\n#\n#  ./ldapsearch.sh <search_query>\n#\n# to test a different account, eg. a service bind account, do\n#\n#  LDAP_USER=<dn_or_email> LDAP_PASSWORD=<password> ./ldapsearch.sh <search_query>\n\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nserver=\"${LDAP_SERVER:-localhost}\"\n\nuri=\"ldap://$server\"\nif [ \"${LDAP_SSL:-}\" = 1 ]; then\n    uri=\"ldaps://$server\"\nfi\n\n# only works on Linux, not Mac\n#domain=\"${DOMAIN:-$(hostname -d)}\"\ndomain=\"${DOMAIN:-$(hostname -f | sed 's/^[^`.]*\\.//')}\"\n\n#base_dn=\"${LDAP_BASE_DN:-dc=$(sed 's/\\./,dc=/g' <<< \"$domain\")}\"\nbase_dn=\"${LDAP_BASE_DN:-dc=${domain//./,dc=}}\"\n\nuser=\"${LDAP_USER:-$USER@$domain}\"\n\n#set +x\nPASS=\"${LDAP_PASSWORD:-${PASSWORD:-${PASS:-}}}\"\n# checks later for Kerberos first, otherwise sets ldapearch -W switch to prompt which is safer\n#if [ -z \"${PASS:-}\" ]; then\n#    pass\n#fi\n\n# shellcheck disable=SC2120\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$@\" >&2\n        echo >&2\n    fi\n    # multiple ${0##*/} inside here document causes usage to not be rendered, must be a bash bug\n    script=\"${0##*/}\"\n    cat >&2 <<EOF\n\nQueries ldap easily using ldapsearch by inferring many common parameters to remove tediousness\n\nUsually only requires setting one or two environment variables, eg. in .bashrc for fast and easy future ldapsearch queries, useful for systems administrators often doing ldap searches or those of use who forget these switches even after over a decade of using ldapsearch\n\n\\$LDAP_SERVER - defaults to localhost. This and \\$LDAP_SSL are probably the minimum you need to set\n\\$LDAP_SSL - optional, enables SSL\n\n\\$LDAP_BASE_DN - infers from local host's domain portion of FQDN (or \\$DOMAIN if set)\n\n\\$LDAP_USER - defaults to using \\$USER@\\$DOMAIN. \\$USER is usually set, \\$DOMAIN is found from 'hostname -f'\n\\$LDAP_PASSWORD / \\$PASSWORD - prompts if not found\n\n\nCaveat:\n\n  $script <dn> - DN based search will not work with ldapsearch - you must use the DN as the \\$LDAP_BASE_DN instead of search filter\n\n\nusage: $script <ldap_filter> [<attribute_filter>]\n\n\nEOF\n    exit 3\n}\n\nfor x in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ \"${LDAP_KRB5:-}\" = 1 ]; then\n    auth_opts=\"-Y GSSAPI\"\nelse\n    auth_opts=\"-x -D $user\"\n    if [ -n \"${PASS:-}\" ]; then\n        auth_opts=\"$auth_opts -w $PASS\"\n    else\n        auth_opts=\"$auth_opts -W\"\n    fi\nfi\n\nif [ -n \"${DEBUG:-}\" ]; then\n    echo\n    # shellcheck disable=SC2001\n    sed \"s/-w[[:space:]]\\\\{1,\\\\}[^[:space:]]\\\\{1,\\\\}/-w '...'/\" <<< \"## ldapsearch -H '$uri' -b '$base_dn' $auth_opts '$*'\"\nfi\n# shellcheck disable=SC2086\nldapsearch -H \"$uri\" -b \"$base_dn\" -o ldif-wrap=no $auth_opts \"$@\"\n"
  },
  {
    "path": "bin/lint.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-20 16:01:28 +0000 (Fri, 20 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Helper script for calling from vim function to run programs or execute with args extraction\n#\n# Runs the value of the 'run:' header from the given file\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/.bash.d/aliases.sh\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/.bash.d/functions.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLints one or more files\n\nAuto-determines the file types, parses any lint headers and calls appropriate scripts and tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"file1 [file2 file3 ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# defer expansion\n# shellcheck disable=SC2016\ntrap_cmd 'exitcode=$?; echo; echo \"Error - Exit Code: $exitcode\"'\n\nfilename=\"$1\"\n\nif ! [ -f \"$filename\" ]; then\n    echo \"File not found: $filename\"\n    exit 1\nfi\n\n# examples:\n#\n# #  lint: k8s\nlint_hint=\"$(parse_lint_hint \"$filename\")\"\n\ndirname=\"$(dirname \"$filename\")\"\nbasename=\"${filename##*/}\"\n\ncd \"$dirname\"\n\n#echo \"Running lint.sh on: $*\"\n#echo\n\nif [ -n \"$lint_hint\" ]; then\n    if [[ \"$lint_hint\" =~ k8s|kubernetes ]]; then\n        #check_yaml.sh \"$basename\"\n        #datree test \"$basename\"\n        check_kubernetes_yaml.sh \"$basename\"\n    else\n        # assume it's a commmand\n        eval \"$lint_hint\" \"$basename\"\n    fi\nelse\n    case \"$basename\" in\n                 *.sh)   shellcheck \"$basename\"\n                        ;;\n             Makefile)  check_makefiles.sh \"$basename\"\n                        ;;\n           Dockerfile)  #hadolint \"$basename\"\n                        check_yaml.sh \"$basename\"\n                        check_dockerfiles.sh \"$basename\"\n                        ;;\n*docker-compose*.y*ml)  #yamllint \"$basename\"\n                        #docker-compose -f \"$basename\" config\n                        check_yaml.sh \"$basename\"\n                        check_docker_compose.sh \"$basename\"\n                        ;;\n                        # TODO: add linting for CloudBuild and Kustomize\n  #  cloudbuild*.y*ml)  yamllint \"$basename\"\n  #                     ;;\n  #kustomization.yaml)  yamllint \"$basename\"\n  #                     ;;\n*.y*ml|autoinstall-user-data)\n                        #yamllint \"$basename\"\n                        check_yaml.sh \"$basename\"\n                        ;;\n              #.envrc)  cd \"$dirname\" && direnv allow .\n               .envrc)  shellcheck \"$basename\"\n                        ;;\n                 *.d2)  d2 fmt \"$basename\"\n                        ;;\n                 *.go)  go fmt -w \"$basename\"\n                        ;;\n                *.lua)  luacheck \"$basename\"\n                        ;;\n                 *.tf)  terraform fmt -diff\n                        terraform validate\n                        ;;\n       terragrunt.hcl)  terragrunt fmt -diff\n                        terragrunt validate\n                        ;;\n *.pkr.hcl|*.pkr.json)  packer init \"$basename\" &&\n                        packer validate \"$basename\" &&\n                        packer fmt -diff \"$basename\"\n                        ;;\n                 *.md)  mdl \"$basename\"\n                        ;;\n             Fastfile) if [[ \"$(readlink -f \"$basename\")\" =~ /fastlane/Fastfile ]]; then\n                            ruby -c \"$basename\"\n                        fi\n                        ;;\n               # this command doesn't exit 1 if the file isn't found\n               #.vimrc)  if ! vim -c \"source $basename\" -c \"q\"; then\n               .vimrc)  if vim -c \"\n                            if !filereadable('$basename') |\n                                echoerr 'Error: File not found'\n                                cquit 1\n                            else\n                                source $basename\n                            endif\n                            \" -c \"q\"; then\n                            echo \"ViM basic lint validation passed\"\n                        else\n                            die \"ViM basic lint validation failed\"\n                        fi\n                        if type -P vint &>/dev/null; then\n                            if vint \"$basename\"; then\n                                echo \"Vint vim script linting passed\"\n                            else\n                                die \"Vint vim script linting failed\"\n                            fi\n                        fi\n                        ;;\n                    *)  die \"Cannot lint unrecognized file type for file: $filename\"\n                        ;;\n    esac\nfi\n\nuntrap\n"
  },
  {
    "path": "bin/linux_distro_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-22 21:24:35 +0100 (Sun, 22 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly returns the list of major versions for a given Linux distro\n\nUsed by programs like ../docker_package_check.sh to get versions faster than iterating lots of tags on the DockerHub API\n\nCurrently supports:\n\n- Alpine\n- Debian\n- Ubuntu\n- Redhat\n- Fedora\n- CentOS\n- Rocky Linux\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<distro_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ndistro=\"$1\"\n\nshopt -s nocasematch\nif [ \"$distro\" = alpine ]; then\n    #curl -sS https://alpinelinux.org/releases/ |\n    curl -sS https://dl-cdn.alpinelinux.org/alpine/ |\n    grep -Eo 'v[[:digit:]]+\\.[[:digit:]]+' |\n    sed 's/^v//; s/[[:space:]]*$//'\nelif [ \"$distro\" = debian ]; then\n    curl -sS https://www.debian.org/releases/ |\n    grep -Eo 'Debian [[:digit:]]+' |\n    sed 's/^Debian //'\nelif [ \"$distro\" = ubuntu ]; then\n    curl -sS https://releases.ubuntu.com/ |\n    grep -Eo 'Ubuntu [[:digit:]]+\\.[[:digit:]]+' |\n    sed 's/^Ubuntu //'\nelif [ \"$distro\" = fedora ]; then\n    curl -sS https://dl.fedoraproject.org/pub/fedora/linux/releases/ |\n    grep -Eo '>[[:digit:]]+/<' |\n    sed 's/^>//; s|/<$||'\nelif [ \"$distro\" = redhat ]; then\n    curl -sS https://access.redhat.com/articles/3078 |\n    grep -Eo 'Red Hat Enterprise Linux [[:digit:]]+' |\n    sed 's/Red Hat Enterprise Linux //; s/[[:space:]]*$//'\nelif [ \"$distro\" = rocky ] || [ \"$distro\" = rockylinux ]; then\n    #curl -sS https://wiki.rockylinux.org/rocky/version/ |\n    #grep -Eo 'Rocky Linux [[:digit:]]+' |\n    #sed 's/Rocky Linux //; s/[[:space:]]*$//'\n    curl -sS https://dl.rockylinux.org/pub/rocky/ |\n    grep -Eo '>[[:digit:]]+/<' |\n    sed 's/^>//; s|/<$||'\nelif [ \"$distro\" = centos ]; then\n    # EOL so just print the known versions\n    echo \"5\n6\n7\n8\"\nelse\n    usage \"Unsupported Linux distro: $distro\"\nfi |\nsort -Vur\n"
  },
  {
    "path": "bin/login.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-21 14:51:31 +0100 (Tue, 21 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns logins to well known Cloud platforms (AWS, GCP, GitHub, DockerHub etc) if any relevant authentication tokens are found for them in the environment\n\nIf targets are given, only runs the logins for those platforms\n\nCurrently supports:\n\n- AWS\n- GCP\n- GitHub CLI\n- Docker Registries:\n  - DockerHub\n  - GitHub Container Registry (GHCR)\n  - Gitlab Container Registry\n  - AWS Elastic Container Registry (ECR)\n  - Azure Container Registry (ACR)\n  - Google Container Registry (GCR)\n  - Google Artifact Registry (GAR)\n  - Quay.io Container Registry (quay)\n\nSee Also:\n\n    login.groovy in repo: https://github.com/HariSekhon/Jenkins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<targets>\"\n\nhelp_usage \"$@\"\n\nall(){\n    dockerhub\n    github\n    ghcr\n    aws\n    gcp\n    azure\n    ecr\n    gcr\n    gar\n    acr\n    gitlab\n    quay\n}\n\ndockerhub(){\n    if [ -n \"${DOCKERHUB_USER:-}\" ] &&\n       [ -n \"${DOCKERHUB_TOKEN:-}\" ]; then\n        echo \"Logging in to DockerHub:\"\n        docker login -u \"$DOCKERHUB_USER\" --password-stdin <<< \"$DOCKERHUB_TOKEN\"\n        echo\n    fi\n}\n\ngithub(){\n    if [ -n \"${GH_TOKEN:-}\" ] ||\n       [ -n \"${GITHUB_TOKEN:-}\" ]; then\n        echo \"GitHub CLI auth:\"\n        # won't log in if these tokens are set, env overrides CLI\n        #if [ -n \"${GH_TOKEN:-}\" ]; then\n        #    gh auth login --with-token <<< \"$GH_TOKEN\"\n        #else\n        #    gh auth login --with-token <<< \"$GITHUB_TOKEN\"\n        #fi\n        gh auth status\n        #echo  # above command already puts a blank line\n    fi\n}\n\nghcr(){\n    if [ -z \"${GITHUB_USER:-}\" ]; then\n        return\n    fi\n    echo \"Logging in to GitHub Container Registry (GHCR):\"\n    if [ -n \"${GH_TOKEN:-}\" ] || [ -n \"${GITHUB_TOKEN:-}\" ]; then\n        docker login ghcr.io -u \"$GITHUB_USER\" --password-stdin <<< \"${GH_TOKEN:-$GITHUB_TOKEN}\"\n    fi\n    echo\n}\n\naws(){\n    if [ -n \"${AWS_ACCESS_KEY_ID:-}\" ] &&\n       [ -n \"${AWS_SECRET_ACCESS_KEY:-}\" ] ||\n       grep -Fxq \"[${AWS_PROFILE:-nonexistent}]\" ~/.aws/credentials 2>/dev/null; then\n        echo \"AWS CLI auth:\"\n        command aws sts get-caller-identity\n        echo\n    fi\n}\n\ngcp(){\n    if [ -n \"${GCP_SERVICEACCOUNT_KEY:-}\" ]; then\n        export CLOUDSDK_CORE_DISABLE_PROMPTS=1\n        echo \"Logging in to Google Cloud:\"\n        #gcloud auth activate-service-account --key-file=<(base64 --decode <<< \"$GCP_SERVICEACCOUNT_KEY\")\n        gcp_login\n        echo\n    fi\n}\n\nazure(){\n    if [ -n \"${AZURE_USER:-}\" ] &&\n       [ -n \"${AZURE_PASSWORD:-}\" ]; then\n        echo \"Logging in to Azure Cloud:\"\n        az login -u \"$AZURE_USER\" -p \"$AZURE_PASSWORD\"\n        echo\n        az ad signed-in-user show\n        echo\n    fi\n}\n\necr(){\n    if [ -z \"${AWS_DEFAULT_REGION:-}\" ]; then\n        AWS_DEFAULT_REGION=\"$(aws_region)\"\n    fi\n    if ! command aws sts get-caller-identity &>/dev/null; then\n        return\n    fi\n    echo \"Logging in to AWS Elastic Container Registry:\"\n    if [ -z \"${AWS_ACCOUNT_ID:-}\" ]; then\n      local AWS_ACCOUNT_ID\n      AWS_ACCOUNT_ID=\"$(command aws sts get-caller-identity | jq -r .Account)\"\n      if [ -z \"$AWS_ACCOUNT_ID\" ]; then\n        echo \"Failed to determine AWS_ACCOUNT_ID\"\n        exit 1\n      fi\n    fi\n    local ECR_TOKEN\n    ECR_TOKEN=\"$(command aws ecr get-login-password --region \"$AWS_DEFAULT_REGION\")\"\n    if [ -z \"$ECR_TOKEN\" ]; then\n      echo \"Failed to get AWS ECR authentication token\"\n      exit 1\n    fi\n    docker login \"${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com\" -u AWS --password-stdin <<< \"$ECR_TOKEN\"\n    echo\n}\n\ngar(){\n    if [ -n \"${GAR_REGISTRY:-}\" ]; then\n        echo \"Logging in to Google Artifact Registry:\"\n        if command -v gcloud &>/dev/null; then\n            gcloud auth configure-docker \"$GAR_REGISTRY\"\n        else\n            echo \"GCloud SDK is not installed, attempting to login with docker directly\" >&2\n            if [ -z \"${GCP_SERVICEACCOUNT_KEY:-}\" ]; then\n                die \"GCP_SERVICEACCOUNT_KEY environment variable not set!\"\n            fi\n            docker login \"$GAR_REGISTRY\" -u _json_key --password-stdin <<< \"$(base64 --decode <<< \"$GCP_SERVICEACCOUNT_KEY\")\"\n        fi\n        echo\n    fi\n}\n\ngcr(){\n    if [ -n \"${GCR_REGISTRY:-}\" ]; then\n        echo \"Logging in to Google Container Registry:\"\n        if command -v gcloud &>/dev/null; then\n            gcloud auth configure-docker \"$GCR_REGISTRY\"\n        else\n            echo \"GCloud SDK is not installed, attempting to login with docker directly\" >&2\n            if [ -z \"${GCP_SERVICEACCOUNT_KEY:-}\" ]; then\n                die \"GCP_SERVICEACCOUNT_KEY environment variable not set!\"\n            fi\n            docker login \"$GCR_REGISTRY\" -u _json_key --password-stdin <<< \"$(base64 --decode <<< \"$GCP_SERVICEACCOUNT_KEY\")\"\n        fi\n        echo\n    fi\n}\n\nacr(){\n    if [ -n \"${ACR_REGISTRY_NAME:-}\" ]; then\n        echo \"Logging in to Azure Container Registry:\"\n        #local TOKEN\n        #TOKEN=\"$(az acr credential show --name \"$registry_name\")\"\n        az acr login --name \"$ACR_REGISTRY_NAME.azurecr.io\"\n        echo\n    fi\n}\n\ngitlab(){\n    if [ -n \"${GITLAB_USER:-}\" ] &&\n       [ -n \"${GITLAB_TOKEN:-}\" ]; then\n        echo \"Logging in to GitLab Container Registry:\"\n        docker login registry.gitlab.com -u \"$GITLAB_USER\" --password-stdin <<< \"$GITLAB_TOKEN\"\n        echo\n    fi\n}\n\nquay(){\n    if [ -n \"${QUAY_USER:-}\" ] &&\n       [ -n \"${QUAY_TOKEN:-}\" ]; then\n        echo \"Logging in to Quay.io Registry:\"\n        docker login quay.io -u \"$QUAY_USER\" --password-stdin <<< \"$QUAY_TOKEN\"\n        echo\n    fi\n}\n\nif [ -n \"$*\" ]; then\n    shopt -s nocasematch\n    for target in \"$@\"; do\n        case \"$target\" in\n            dockerhub)  dockerhub\n                        ;;\n               github)  github\n                        ;;\n                 ghcr)  ghcr\n                        ;;\n                  aws)  aws\n                        ;;\n                  gcp)  gcp\n                        ;;\n                azure)  azure\n                        ;;\n                  ecr)  ecr\n                        ;;\n                  gcr)  gcr\n                        ;;\n                  gar)  gar\n                        ;;\n                  acr)  acr\n                        ;;\n               gitlab)  gitlab\n                        ;;\n                 quay)  quay\n                        ;;\n        esac\n    done\nelse\n    all\nfi\n"
  },
  {
    "path": "bin/lowercase_filename.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-04-26 02:44:42 +0800 (Sat, 26 Apr 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRenames a file to lowercase, using an intermediate file to work around case insensitive filesystems like on macOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nfilename=\"$1\"\n\nif ! [ -f \"$filename\" ]; then\n    die \"File not found: $filename\"\nfi\n\nfilename_lowercase=\"$(tr '[:upper:]' '[:lower:]' <<< \"$filename\")\"\n\ntmpfile=\"$filename.lowercase.tmp\"\n\nmv -fv \"$filename\" \"$tmpfile\"\n\nmv -fv \"$tmpfile\" \"$filename_lowercase\"\n"
  },
  {
    "path": "bin/mac_backup_du_in_progress.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-09 02:25:20 -0500 (Fri, 09 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFind large files in the currently in-progress Time Machine backup to find out what is taking so long\nand racking up so many more GB of changes than you expect\n\nNeeds to be run during a Time Machine backup to be able to find the current '<date>-inprogress' directory\n\nThis helps discover large but unnecessary files that you might want to exclude using the adjacent script:\n\n    mac_backup_exclude_paths.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\nmac_only\n\ntoday=\"$(date '+%F')\"\n# this date command is specific to mac's BSD version of the date command\nyesterday=\"$(command date -v -1d '+%F')\"\n\nshopt -s nullglob\n\nmatches=(/Volumes/*/\"$today\"-*.inprogress)\n\nif [ \"${#matches[@]}\" -eq 0 ]; then\n    timestamp \"No currently dated inprogress dir found\"\n    timestamp \"Attempting to find one dated yesterday in case the currently in-progress backup started before midnight\"\n    matches=(/Volumes/*/\"$yesterday\"-*.inprogress)\n    if [ \"${#matches[@]}\" -eq 0 ]; then\n        echo >&2\n        die \"ERROR: No currently in-progress Time Machine backup directories found for today or yesterday - this must be run during an active Time Machine backup\"\n    fi\nfi\n\ntimestamp \"Found in-progress dir(s):\"\necho >&2\nfor dir in \"${matches[@]}\"; do\n    printf '\\t%s\\n' \"$dir\"\ndone\necho >&2\n\nsudo du -max \"${matches[@]}\" |\nsort -k1n |\ntail -n 1000\n"
  },
  {
    "path": "bin/mac_backup_exclude_paths.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-04-14 04:30:02 +0800 (Mon, 14 Apr 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexcluded_paths=\"\n$HOME/.cache\n$HOME/.cpan\n$HOME/.cpanm\n$HOME/.crc\n$HOME/.docker/machine/cache\n$HOME/.docker/machine/machines\n$HOME/.gem\n$HOME/.gradle\n$HOME/.groovy\n$HOME/.ivy\n$HOME/.ivy2\n$HOME/.m2\n$HOME/.minikube\n$HOME/.sbt\n$HOME/Downloads/TV\n$HOME/Downloads/Transmission\n$HOME/Downloads/YouTube\n$HOME/Library/Application Support/Spotify/PersistentCache\n$HOME/Library/Application Support/multipass/\n$HOME/Library/Caches\n$HOME/Library/Containers/com.docker.docker\n$HOME/Library/Containers/com.utmapp.UTM/Data\n$HOME/Library/Developer/Xcode/DerivedData\n$HOME/Library/Logs/rancher-desktop\n$HOME/Library/Mobile Documents/com~apple~CloudDocs\n$HOME/Library/State/rancher-desktop\n$HOME/VirtualBox VMs\n$HOME/go/bin\n$HOME/go/pkg\n$HOME/go/src/github.com\n$HOME/go/src/golang.org\n/usr/local/texlive\n/Library/Application Support/GarageBand\n/Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate\n/System//Library/AssetsV2/com_apple_MobileAsset_MacSoftwareUpdate\n/macOS Install Data\n/Data/macOS Install Data\n/Volumes/Data/macOS Install Data\n/System//Volumes/Data/macOS Install Data\n\"\n\nif [ -n \"${GOPATH:-}\" ]; then\n    excluded_paths+=\"\n$GOPATH/bin\n$GOPATH/pkg\n$GOPATH/src/github.com\n$GOPATH/src/golang.org\n    \"\nfi\n\nexcluded_paths=\"$(sort -u <<< \"$excluded_paths\" | sed '/^[[:space:]]*$/d')\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExcludes many common large caches, docker and VM paths from macOS Time Machine backups\n\nMust be either run as root or will attempt to use sudo to add each path\n\nSee HariSekhon/Knowledge-Base Mac page for more details on Time Machine path exclusions:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md#time-machine\n\nBy default excludes the following common paths:\n\n$excluded_paths\n\nFind more paths to add to the exclusion list using this command:\n\n    du -max ~ | sort -k1n | tail -n 1000\n\nSee also this next script to find out what is taking so long to populate the currently in-progress backup\nand racking up more GB of changes than you expected to find more things to exclude:\n\n    mac_backup_exclude_paths.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<path> <path2>...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nmac_only\n\ntimestamp \"Excluding paths from macOS Time Machine backups\"\n\nexclude_path(){\n    local path=\"$1\"\n    path=\"${path## }\"\n    path=\"${path%% }\"\n    if is_blank \"$path\"; then\n        return\n    fi\n    timestamp \"Adding path to macOS Time Machine exclusions: $path\"\n    # defined in lib/utils-bourne.sh\n    # shellcheck disable=SC2154\n    \"$sudo\" tmutil addexclusion -p \"$path\"\n}\n\nfor path in \"$@\"; do\n    exclude_path \"$path\"\ndone\n\nwhile read -r path; do\n    exclude_path \"$path\"\ndone <<< \"$excluded_paths\"\n\ntimestamp \"Done\"\n"
  },
  {
    "path": "bin/mac_backup_find_excluded_paths.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-04-14 04:30:02 +0800 (Mon, 14 Apr 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDoes a recursive deep search for macOS Time Machine excluded paths on individual file/folder attributes\nto find items excluded from backups that don't appear in the Time Machine UI\n\nIf a path argument is given, does the deep scan on only that tree\n\nSee HariSekhon/Knowledge-Base Mac page for more details on Time Machine path exclusions for why this is needed:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md#time-machine\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<path>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nmac_only\n\npath=\"${1:-/}\"\n\ntimestamp \"sudo mdfind \\\"com_apple_backup_excludeItem = 'com.apple.backupd'\\\"\"\necho >&2\nsudo mdfind \"com_apple_backup_excludeItem = 'com.apple.backupd'\" |\nsort\necho >&2\n\ntimestamp \"defaults read /Library/Preferences/com.apple.TimeMachine.plist ExcludeByPath\"\necho >&2\ndefaults read /Library/Preferences/com.apple.TimeMachine.plist ExcludeByPath |\nsed '\n    s/^(//;\n    s/)$//;\n    s/^[[:space:]]*//;\n    s/^\"//;\n    s/\",*$//;\n    /^[[:space:]]*$/d;\n' |\nsort\necho >&2\n\ntimestamp \"defaults read /Library/Preferences/com.apple.TimeMachine SkipPaths\"\necho >&2\ndefaults read /Library/Preferences/com.apple.TimeMachine SkipPaths |\nsed '\n    s/^(//;\n    s/)$//;\n    s/^[[:space:]]*//;\n    s/^\"//;\n    s/\",*$//;\n    /^[[:space:]]*$/d;\n' |\nsort\necho >&2\n\ntimestamp \"Doing deep search for xattr excluded paths on each file / directory (this will take a very long time)\"\nsudo find \"$path\" |\nwhile read -r path; do\n    if sudo xattr -p com.apple.metadata:com_apple_backup_excludeItem \"$path\" &>/dev/null; then\n        echo \"$path\"\n    fi\ndone\n"
  },
  {
    "path": "bin/mac_delete_local_snapshots.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-08 08:32:28 -0500 (Thu, 08 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes local macOS snapshots to free up disk space\n\nWhen there is a substantial discrepancy between what the 'df -h' command and the Finder UI shows,\nthis is often the cause\n\nPath defaults to /\n\nRequires being run as root or having sudo privileges\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<path>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nexport path=\"${1:-/}\"\n\nif ! [ -d \"$path\" ]; then\n    die \"ERROR: invalid directory given: $path\"\nfi\n\nshow_disk_space(){\n    #timestamp \"Disk Space Finder Sees:\n    #\"\n    #\n    #diskutil info \"$path\" | grep 'Free Space'\n    #echo\n\n    timestamp \"Disk Space:\"\n    echo\n    df -h \"$path\"\n    echo\n}\n\nshow_disk_space\n\nsnapshots=\"$(\n    tmutil listlocalsnapshots \"$path\" |\n    tail -n +2 |\n    #awk -F. '{print $4}' |\n    #sed '/^[[:space:]]*$/d'\n    #\n    # format is like:\n    #\n    #   com.apple.TimeMachine.2026-02-14-041148.local\n    #   com.apple.TimeMachine.2026-02-14-051247.local\n    #   com.apple.TimeMachine.2026-02-14-174320.local\n    #   com.apple.TimeMachine.2026-02-14-225329.local\n    #   com.apple.TimeMachine.2026-02-15-013603.local\n    #   com.apple.TimeMachine.2026-02-15-023623.local\n    #   com.apple.TimeMachine.2026-02-15-033634.local\n    #   com.apple.os.update-7850C02764D0DA2E70DAEF3595E6971F5B559DB93E53CCD252FAEA3C1110DBCBF1D66824F751C3AFAA37D9761B620610\n    #   com.apple.os.update-F6A31E7EE1E0D94A5A3E654C49952C5BC7E881804C69887CAD1C476EC161E83A\n    #   com.apple.os.update-MSUPrepareUpdate\n    #\n    # update snapshots can't be deleted so just take the date timestamped ones:\n    #\n    #                  2026-02-14-041148\n    command ggrep -oP '\\d{4}-\\d\\d-\\d\\d-\\d+'\n)\"\n\n# because wc -l returns 1 on an empty line due to a \\n newline\nnum_snapshots=\"$(grep -c . <<< \"$snapshots\" || :)\"\n\ntimestamp \"Snapshots to Delete: $num_snapshots\"\necho\n\nif [ \"$num_snapshots\" -lt 1 ]; then\n    timestamp \"No snapshots to delete, exiting.\"\n    exit 0\nfi\n\ntimestamp \"Deleting Time Machine Snapshots\"\necho\n\nwhile read -r snapshot_timestamp; do\n    timestamp \"Deleting local snapshot: $snapshot_timestamp\"\n    sudo tmutil deletelocalsnapshots \"$snapshot_timestamp\"\n    echo\ndone <<< \"$snapshots\"\n\nshow_disk_space\n"
  },
  {
    "path": "bin/mac_diff_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 14:45:38 +0000 (Fri, 16 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTakes snapshots of before and after a settings change to find which 'defaults write' settings\n\nto use in $srcdir/../settings/mac_settings.sh\n\nTakes an initial snapshot to /tmp\n\nThen prompts for you to change the settings in the UI\n\nThen takes another snapshot and diffs the two\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nmac_only\n\nbefore_snapshot_txt=\"/tmp/macos_defaults_before.txt.$$\"\nafter_snapshot_txt=\"/tmp/macos_defaults_after.txt.$$\"\n\ntimestamp \"Taking snapshot to $before_snapshot_txt\"\ndefaults read > \"$before_snapshot_txt\"\necho >&2\n\ntimestamp 'Now change your mac settings in the UI settings'\necho >&2\n\nread -r -p 'Press Enter when finished to take another snapshot and diff them'\necho >&2\n\ntimestamp \"Taking snapshot to $after_snapshot_txt\"\ndefaults read > \"$after_snapshot_txt\"\necho >&2\n\necho \"Diff:\" >&2\necho >&2\n\ndiff -w \"$before_snapshot_txt\" \"$after_snapshot_txt\" &&\necho \"No changes\"\n"
  },
  {
    "path": "bin/mac_gif_preview.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-19 17:13:31 +0700 (Wed, 19 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens Gif in the Mac Finder file viewer preview because the Preview app doesn't show the animation\n\nYou can also open it in a web browser to render the animation if you prefer:\n\n    open -a Safari \\\"\\$file\\\"\n\n    open -a 'Google Chrome' \\\"\\$file\\\"\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file.gif>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nmac_only\n\nfile=\"$1\"\n\nopen -R \"$file\"\n\nSLEEP_SECS=0.5\n\nsleep \"$SLEEP_SECS\"\n\napplescript=\"$srcdir/../applescript\"\n\ntimestamp \"Switching to Finder\"\n\"$applescript/set_frontmost_process.scpt\" Finder\n\nsleep \"$SLEEP_SECS\"\n\ntimestamp \"Checking Finder is the frontmost process\"\nif [ \"$(\"$applescript/get_frontmost_process.scpt\")\" != Finder ]; then\n    die \"Failed to switch to Finder - not the frontmost process\"\nfi\n\n# https://eastmanreference.com/complete-list-of-applescript-key-codes\n\nexport START_DELAY=0  # don't wait before sending keystrokes\n\necho >&2\n\"$applescript/keystrokes.sh\" 1 49  # 1 x space\n"
  },
  {
    "path": "bin/mac_iso_to_usb.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-06 02:09:22 +0000 (Wed, 06 Mar 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts a given ISO file to a USB bootable image and burns it onto a given or detected inserted USB drive\n\nPrompts to insert a USB drive, diffs which device this turns up as, then prompts for confirmation and continues write to it\n\nThe Etcher app is also recommended for a GUI solution:\n\n    https://etcher.balena.io/\n\nUNetbootin is also a classic choice with upstream distribution integration as well as downloaded ISO support\nIf it doesn't detect your USB drive, use Disk Utility to format it as MS-DOS (FAT32) to get it to detect it\n\n    https://unetbootin.github.io/\n\nTested on macOS 14 Sonoma\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<iso_file> [<usb_device>]\"\n\nhelp_usage \"$@\"\n\nmac_only\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\niso=\"$1\"\nusb_drive=\"${2:-}\"\n\nif ! [[ \"$iso\" =~ \\.iso$ ]]; then\n    usage \"specified ISO file does not end in .iso - aborting for safety\"\nfi\n\nif [ -n \"$usb_drive\" ] && ! [[ \"$usb_drive\" =~ ^/dev/ ]]; then\n    usage \"specified device '$usb_drive' does not start with /dev/... aborting for safety\"\nfi\n\nimg=\"${iso%.iso}.img\"\n\ntrap_cmd \"rm -f '$img'\"\n\nif [ -f \"$img\" ]; then\n    timestamp \"IMG file '$img' already exists, skipping for now\"\nelse\n    timestamp \"Converting ISO file '$iso' to '$img'\"\n    hdiutil convert -format UDRW -o \"$img\" \"$iso\"\n    mv \"$img.dmg\" \"$img\"\n    untrap\nfi\n\necho\ntimestamp 'Disks:'\necho\ndisks_before=\"$(diskutil list | tee /dev/stderr)\"\necho\n\nif is_blank \"$usb_drive\"; then\n    read -r -p 'Enter your USB disk and then press Enter'\n    # sleep a couple seconds to give the Mac a chance to detect the inserted USB disk\n    sleep 4\n    echo\n\n    timestamp 'Disks:'\n    echo\n    disks_after=\"$(diskutil list | tee /dev/stderr)\"\n    echo\n\n    echo 'New USB disks detected:'\n    echo\n    usb_drive=\"$(diff -w <(echo \"$disks_before\") <(echo \"$disks_after\") | tee /dev/stderr | grep -Eo '/dev/disk[[:digit:]]+' | head -n 1 || :)\"\n    echo\n    if [ -z \"$usb_drive\" ]; then\n        die 'Failed to detect USB disk'\n    fi\n    echo \"Determined USB drive to be: '$usb_drive'\"\n    echo\nelse\n    echo \"You have selected disk '$usb_drive' from the above list\"\n    echo\nfi\n\nif ! [[ \"$usb_drive\" =~ /dev/disk[[:digit:]]+$ ]]; then\n    die \"USB drive determined to be '$usb_drive' but this does not match expected regex of /dev/diskN - aborting for safety\"\nfi\n\nread -r -p 'Does this look correct? Continue? (y/N) ' answer\n\ncheck_yes \"$answer\"\n\necho\ntimestamp 'Unmounting USB drive'\necho\ndiskutil unmountDisk \"$usb_drive\" || :\necho\n\nraw_usb_drive=\"${usb_drive/\\/disk//rdisk}\"\n\ntimestamp 'Writing to USB drive'\necho\nsudo dd if=\"$img\" of=\"$raw_usb_drive\" bs=1m\necho\n\ntimestamp 'Finished writing, ejecting'\necho\ndiskutil eject \"$usb_drive\"\necho\ntimestamp 'Done'\n"
  },
  {
    "path": "bin/mac_ramdisk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-07-10 15:36:54 +0100 (Tue, 10 Jul 2012)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates and mounts a macOS ramdisk of the given size in MB\n\nTo remove the ramdisk, just run:\n\n    diskutil eject /Volumes/Ramdisk\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<MB_size>]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nMB=\"$1\"\n\nif ! is_int \"$MB\"; then\n    die \"Invalid argument given, MB must be an integer\"\nfi\n\nblocks=\"$((\"$MB\" * 1024 * 1024 / 512))\"\n\nlist_ramdisks() {\n    diskutil list | \\\n    awk '\n        /^\\// {disk=$1}\n        /RAM Disk/ {print disk}\n    ' | \\\n    while read -r d; do\n        mp=$(mount | awk -v d=\"$d\" '$1==d {print $3}')\n        printf \"%s %s\\n\" \"$d\" \"$mp\"\n    done\n}\n\nexisting=\"$(list_ramdisks)\"\n\nif [ -n \"$existing\" ]; then\n    timestamp \"Existing RAM disks found:\"\n    echo \"$existing\"\n    die \"Refusing to create a new RAM disk; please reuse or eject the existing one(s) to avoid leaking ramdisks from repeated runs\"\nfi\n\ntimestamp \"Creating ramdisk of size '$MB' MB => '$blocks' blocks\"\ndisk=\"$(hdiutil attach -nomount ram://$blocks)\"\ntimestamp \"Created: $disk\"\necho\ndiskutil list\necho\ntimestamp \"SAFETY: double check the disk and then run format to mount the Ramdisk with HFS+: $disk\"\necho\necho diskutil erasevolume HFS+ \"Ramdisk\" \"$disk\"\n"
  },
  {
    "path": "bin/mac_restore_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-27 20:04:28 +0700 (Thu, 27 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRestores a file from the latest online Mac Timemachine backup where it exists\n\nPrints the backup disks and then checks the variations of the backup paths mount point paths to find\nthe newest version\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nmac_only\n\nfilename=\"$1\"\nif ! [[ \"$filename\" =~ ^/ ]]; then\n    filename=\"$PWD/$filename\"\nfi\n\ntimestamp \"Backups;\"\necho\ntmutil destinationinfo\necho\n\ntimestamp \"Determining backup mount point\"\nmountpoints=\"$(tmutil destinationinfo | awk -F \" : \" '/^Mount Point/{print $2}')\"\ntimestamp \"Backup Mount Points Online:\n\n$mountpoints\n\"\n\n#timestamp \"Determining latest backup\"\n#latest_backup=\"$(tmutil latestbackup)\"\n\n#if ! [ -d \"$latest_backup\" ]; then\n#    timestamp \"Backup path returned by tmutil not found: $latest_backup\"\n#    timestamp \"Trying alternative path\"\n#    latest_backup=\"$mountpoint/$(tmutil latestbackup | sed 's|.*/||')\"\n#    if ! [ -d \"$latest_backup\" ]; then\n#        timestamp \"Backup path returned by tmutil not found: $latest_backup\"\n#        timestamp \"Trying previous instead of current backup\"\n#        latest_backup=\"${latest_backup%.backup}.previous\"\n#    fi\n#    if ! [ -d \"$latest_backup\" ]; then\n#        die \"Latest backup alternative path not found; $latest_backup\"\n#    fi\n#    timestamp \"Latest Backup: $latest_backup\"\n#fi\n\n# lists backups that are present on the current mount\n#backups=\"$(tmutil listbackups | sed 's|.*/||; s|\\.backup$||; s|\\.previous$||' | tail -r)\"\n# shellcheck disable=SC2012\nbackups=\"$(\n    while read -r mountpoint; do\n        ls -t \"$mountpoint\" |\n        sed '\n            s|\\.backup/*$||;\n            s|\\.previous/*$||;\n            /plist/d;\n        '\n    done <<< \"$mountpoints\" |\n    sort -r\n)\"\nbackupfile=\"\"\nfor backup in $backups; do\n    while read -r mountpoint; do\n        for suffix in backup previous; do\n            backupfile=\"$mountpoint/$backup.$suffix/Data/$filename\"\n            timestamp \"Checking for file: $backupfile\"\n            if [ -f \"$backupfile\" ]; then\n                echo\n                timestamp \"Found backup file: $backupfile\"\n                break 3\n            fi\n        done\n    done <<< \"$mountpoints\"\ndone\nif ! [ -f \"$backupfile\" ]; then\n    die \"ERROR: failed to find $filename in any backups\"\nfi\n\ntimestamp \"Restoring $filename\"\necho\ntmutil restore \"$backupfile\" \"$filename\"\n"
  },
  {
    "path": "bin/mac_rmdir.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-30 23:19:03 -0600 (Sun, 30 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Test:\n#\n#   mkdir -v /tmp/testdir\n#   mkdir -v /tmp/testdir/.fseventsd\n#   mkdir -v /tmp/testdir/.Spotlight-V100\n#   touch -v /tmp/testdir/.DS_Store\n#\n#   mac_rmdir.sh /tmp/testdir\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSafely delete a directory on Mac only if it is empty of actual data,\nby first removing macOS hidden metadata files and dirs such as:\n\n- .fseventsd/\n- .Spotlight-V100/\n- .DS_Store\n\nYou can combine with the find command to clean out an empty directory tree:\n\n    find . -type d -exec \"$srcdir/mac_rmdir.sh\" {} \\;\n\nUsed by adjacent mv.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<directory>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\ndir=\"$1\"\n\nmetadata_dirs=\"\n.fseventsd\n.Spotlight-V100\n\"\n\nwhile read -r subdir; do\n    if is_blank \"$subdir\"; then\n        continue\n    fi\n    if [ -d \"$dir\" ]; then\n        timestamp \"rm -rfv \\\"${dir:?}/${subdir:?}\\\"\"\n        # defensive coding - returns an error if the variables are unset for extra safety\n        # to prevent rm -rf / upon blank variables\n        rm -rfv \"${dir:?}/${subdir:?}\"\n    fi\ndone <<< \"$metadata_dirs\"\n\nif [ -f \"$dir/.DS_Store\" ]; then\n    timestamp \"rm -fv \\\"${dir:?}/.DS_Store\\\"\"\n    rm -fv \"${dir:?}/.DS_Store\"\nfi\n\ntimestamp \"rmdir -v \\\"$dir\\\"\"\nrmdir -v \"$dir\"\n"
  },
  {
    "path": "bin/mv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-26 22:49:25 -0600 (Wed, 26 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMoves directory trees resumably and removes the source directories as they're copied over\n\nUseful to migrate data from one disk to another\n\nIf the CHECKSUM environment variable is set to any non-blank value\nthen also checksums the files on both ends (very slow)\n\nShows the overall % of files transferred and the MB/s data transfer rate\n\nUses rsync so the source and destination directories follows the same principle, do not suffix a slash\nunless you want the contents to be copied without the top level directory name\n\nTo see the overall volume size transfer progress, in another terminal you can run:\n\n    watch df -m /Volumes/One /Volumes/Two\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<source_directory> <destination_directory>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nsrc=\"$1\"\ndest=\"$2\"\n\ntimestamp \"Starting resumable rsync to move files from '$src' to '$dest'\"\necho >&2\nrsync -avh \\\n      --info=progress2,stats \\\n      --remove-source-files \"$src\" \"$dest\" \\\n      --exclude=.Spotlight-V100 \\\n      --exclude=.fseventsd \\\n      ${CHECKSUM:+--checksum}\n      # no point in partial resume if you need to --apend-verify checksum\n      #--partial \\\n      #--no-whole-file \\\n      #--append-verify \\\necho >&2\n\ntimestamp \"Move complete, deleting empty directories from '$src'\"\necho >&2\nif is_mac; then\n    find \"$src\" -type d -exec \"$srcdir/mac_rmdir.sh\" {} \\;\nfi\nfind \"$src\" -type d -empty -delete\necho >&2\n\ntimestamp \"Directory tree move completed\"\n"
  },
  {
    "path": "bin/network_gateway.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-17 01:35:17 -0500 (Sat, 17 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the network gateway in as reliable a way as we can on both Linux and Mac\n\nDesigned to make writing other scripts easier\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\nif type -P ip &>/dev/null; then\n    ip route show default 2>/dev/null |\n    awk '{print $3; exit}'\n# Mac's route command is different, this will only work on Linux\n#elif type -P route &>/dev/null; then\n#    route -n 2>/dev/null |\n#    awk '$1 == \"0.0.0.0\" {print $2; exit}'\nelif type -P netstat &>/dev/null; then\n    netstat -rn 2>/dev/null |\n    awk '\n        $1 == \"Internet:\" { inet = 1; next }\n        $1 == \"Internet6:\" { inet = 0 }\n        inet && ($1 == \"default\" || $1 == \"0.0.0.0\") && $2 ~ /^[0-9.]+$/ {\n            print $2\n            exit\n        }\n    '\nelse\n    die \"Failed to get network gateway\"\nfi\n"
  },
  {
    "path": "bin/open.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google.com\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-15 16:40:43 -0500 (Thu, 15 Jan 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# based off urlopen.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens the file or url given as an arg, or first line from standard input if no arg is given\n\nUses the desktop environment's generic open functionally for Mac or Linux\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nopen(){\n    local openers=(\n        xdg-open\n        gnome-open\n    )\n    local arg=\"$1\"\n    if is_mac; then\n        open \"$arg\"\n    else  # assume Linux\n        for opener in \"${openers[@]}\"; do\n            if type -P \"$opener\" &>/dev/null; then\n                \"$opener\" \"$arg\"\n                return 0\n            fi\n        done\n        die \"ERROR: none of the following desktop openers were found in the \\$PATH:\n\n$(for opener in \"${openers[@]}\"; do echo \"$opener\"; done)\n\nCould not open the given file or url: $arg\n\"\n    fi\n}\nexport -f open\n\nif [ $# -eq 0 ]; then\n    cat\nelse\n    echo \"$1\"\nfi |\n# head -n1 because grep -m 1 can't be trusted and sometimes outputs more matches on subsequent lines\nhead -n1 |\nwhile read -r arg; do\n    open \"$arg\"\ndone\n"
  },
  {
    "path": "bin/oreilly_cover_download.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-25 19:57:12 +0700 (Sat, 25 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads an O'Reilly book cover\n\nGives a nice interactive menu or Book Titles and Animal names to scroll through\n\nUse an ERE regex argument to filter this list to be shorter\n\nIf the regex only matches a single item then skips the interactive menu and downloads it directly\n\n\nRequires dialogue menu CLI tool to be installed - attempts to install it via OS package manager if not already found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<regex_filter>]\"\n\nhelp_usage \"$@\"\n\n#max_args 1 \"$@\"\n\nregex=\"${*:-.}\"\n\n# https://gist.github.com/briandfoy/d68915eb425e1fc4932ceac5cdf2d60d\n#\n# forked to\n#\n# https://gist.github.com/HariSekhon/4374b779ef3f79e5f28cbb8ca0d3b31b\n\n# gh gist view --raw d68915eb425e1fc4932ceac5cdf2d60d --filename oreilly-animals.json > ../resources/oreilly-animals.json\n\njson=\"$srcdir/../resources/oreilly-animals.json\"\n\nif ! type -P dialog &>/dev/null; then\n    timestamp \"Dialog not found in \\$PATH, attempting to install via OS package manager\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" dialog\n    echo\nfi\n\ntimestamp \"Parsing Titles\"\ntitles=\"$(jq -Mr '.[].title' < \"$json\" | grep -Ei \"$regex\" || :)\"\n\ntimestamp \"Parsing Animals\"\nanimals=\"$(jq -Mr '.[].animal' < \"$json\" | grep -Ei \"$regex\" || :)\"\n\nmenu_items=()\nmenu_tag_items=()\n\nwhile read -r line; do\n    # used for counting and string conversion if only a single item matches regex\n    menu_items+=(\"$line\")\n\n    # passed to dialog because it requires args: tag1 visibletext tag2 visibletext\n    # - by making the second one blank it uses the item as both the tag to be returned\n    # to script as well as the visible text\n    menu_tag_items+=(\"$line\" \" \")\n\ndone < <( { echo \"$titles\"; echo \"$animals\"; } | sed '/^[[:space:]]*$/d' | sort -fu )\n\nif [ \"${#menu_items[@]}\" -eq 0 ];then\n    die \"Error: No Book Titles or Animals found matching regex: $regex\"\nelif [ \"${#menu_items[@]}\" -eq 1 ];then\n    selected=\"${menu_items[*]}\"\nelse\n    selected=\"$(dialog --menu \"Choose Book Title or Animal:\" \"$LINES\" \"$COLUMNS\" \"$LINES\" \"${menu_tag_items[@]}\" 3>&1 1>&2 2>&3)\"\nfi\n\ntimestamp \"Selected: $selected\"\n\ntimestamp \"Getting corresponding URL\"\nurl=\"$(jq -r \"limit(1; .[] | select(.title == \\\"$selected\\\" or .animal == \\\"$selected\\\") ) | .cover_src\" < \"$json\")\"\n\ntimestamp \"Downloading: $url\"\n\ndownload_file=\"$selected.${url##*.}\"\n\ntimestamp \"Downloading to file: $download_file\"\n\nwget -nc -O \"$selected.${url##*.}\" \"$url\"\n\ntimestamp \"Download complete\"\n\necho >&2\n\n\"$srcdir/../media/imageopen.sh\" \"$download_file\"\n"
  },
  {
    "path": "bin/organize_downloads.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-18 16:21:32 +0200 (Thu, 18 Jul 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMoves files of well-known extensions in the \\$HOME/Downloads directory older than 1 week\nto capitalized subdirectories of their type to keep the \\$HOME/Downloads/ directory tidy\n\nDesigned to be run either interactively or scheduled via local user 'crontab -e'\n\nCan optionally specify a file extension, otherwise operates on a default list of common files,\naggregating several files of similar types eg. jpeg / png / webp into directories of names like PICS/\nor .tar.* into TARBALLS/\n\nTo change the number of days for which files older than should be moved:\n\n    export DOWNLOADS_ORGANIZE_OLD_DAYS_THRESHOLD=7\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file_extension>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncd ~/Downloads\n\nfile_extension=\"${1:-}\"\n\n# Common file extensions cluttering my ~/Downloads directory\nfile_extensions=(\ncer\ncert\ncrt\ncsv\ndmg\ndoc\ndocx\nimg\niso\njpeg\njpg\njson\nkey\nlog\nodp\np12\npdf\npem\npng\nppt\npptx\nrtf\nsvg\ntar\ntar.bz2\ntar.gz\ntbz2\ntgz\ntxt\nwebp\nxls\nxlsx\nxml\nzip\n)\n\n# for the case match below\nshopt -s nocasematch\n\nmove_files_of_extension(){\n    local file_extension=\"$1\"\n    timestamp \"Processing files or extension '$file_extension'\"\n    echo >&2\n    local subdir\n    case \"$file_extension\" in\n           doc|docx|odp|rtf)    subdir=\"DOC\"\n                                ;;\n                   ppt|pptx)    subdir=\"POWERPOINT\"\n                                ;;\n                   xls|xlsx)    subdir=\"EXCEL\"\n                                ;;\n                    img|iso)    subdir=\"ISO\"\n                                ;;\n                        log)    subdir=\"LOGS\"\n                                ;;\n       crt|cert|key|p12|pem)    subdir=\"SSL_CERTS\"\n                                ;;\n      jpg|jpeg|png|webp|svg)    subdir=\"PICS\"\n                                ;;\ntar|tar.gz|tar.bz2|tgz|tbz2)    subdir=\"TARBALLS\"\n                                ;;\n                          *)    subdir=\"$(tr '[:lower:]' '[:upper:]' <<< \"$file_extension\")\"\n\n    esac\n\n    mkdir -p -v \"$subdir\"\n\n    #yes no |\n    find . -maxdepth 1 \\\n           -type f \\\n           -iname \"*.$file_extension\" \\\n           -mtime +\"${DOWNLOADS_ORGANIZE_OLD_DAYS_THRESHOLD:-7}\" \\\n           -exec mv -iv \"{}\" \"$subdir/\" \\;  # trailing slash is important to enforce directory move and not accidental rename behaviour\n    echo >&2\n}\n\nif [ -n \"$file_extension\" ]; then\n    move_files_of_extension \"$file_extension\"\nelse\n    for file_extension in \"${file_extensions[@]}\"; do\n        move_files_of_extension \"$file_extension\"\n    done\nfi\n"
  },
  {
    "path": "bin/paste_diff_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-01 22:49:15 +0000 (Fri, 01 Mar 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTakes snapshots of before and after clipboard changes and diffs them to show config changes\n\nTakes an initial snapshot to /tmp\n\nThen prompts for you to copy changes to the clipboard (eg. via UI changes and Configuration-as-Code copy)\n\nThen takes another snapshot and diffs the two\n\nExample Use Case:\n\nJenkins to backport settings to JCasC\n\n    1. copy the existing JCasC config from here to clipboard:\n\n        \\$JENKINS_URL/manage/configuration-as-code/viewExport\n\n    2. run this script to save that clipboard to a file\n\n    3. then when prompted, reconfigure Jenkins in the UI\n\n    4. copy the new JCasC config from the page above to the clipboard\n\n    5. press enter to tell this script to continue to past the new clipboard to another file and diff for you\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nbefore_snapshot_txt=\"/tmp/paste_diff_settings.txt.$$\"\nafter_snapshot_txt=\"/tmp/paste_diff_settings.txt.$$\"\n\ntimestamp \"Pasting clipboard to $before_snapshot_txt\"\n\"$srcdir/paste_from_clipboard.sh\" > \"$before_snapshot_txt\"\necho >&2\n\ntimestamp 'Now make your changes and copy your new config to clipboard'\necho >&2\n\nread -r -p 'Press Enter when finished to take another snapshot and diff them'\necho >&2\n\ntimestamp \"Pasting snapshot to $after_snapshot_txt\"\n\"$srcdir/paste_from_clipboard.sh\" > \"$after_snapshot_txt\"\necho >&2\n\necho \"Diff:\" >&2\necho >&2\n\ndiff -w \"$before_snapshot_txt\" \"$after_snapshot_txt\" &&\necho \"No changes\"\n"
  },
  {
    "path": "bin/paste_from_clipboard.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-01 22:56:54 +0000 (Fri, 01 Mar 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPastes from the system clipboard on Linux or Mac to stdout\n\nRedirect this to a file for diffing\n\nUsed by paste_diff_settings.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nif is_mac; then\n    pbpaste\nelif is_linux; then\n    xclip -o\nelse\n    echo \"ERROR: OS is not Darwin/Linux\"\n    return 1\nfi\n"
  },
  {
    "path": "bin/paste_from_clipboard_upon_changes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-29 05:50:44 +0200 (Thu, 29 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWatches for clipboard changes and when detected pastes from the system clipboard on Linux or Mac to stdout\n\nRedirect this to a file for diffing\n\nUses adjacent script paste_from_clipboard.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<polling_interval_seconds>\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\npolling_interval_seconds=\"${1:-0.5}\"\n\nif ! is_float \"$polling_interval_seconds\"; then\n    usage \"First argument for polling interval seconds must be a float\"\nfi\n\nlast_clipboard=\"\"\n\nwhile true; do\n    log \"Checking current clipboard contents\"\n    current_clipboard=\"$(\"$srcdir/paste_from_clipboard.sh\")\"\n    log \"Comparing current clipboard contents to previous clipboard contents\"\n    if [ \"$current_clipboard\" != \"$last_clipboard\" ]; then\n        log \"Clipboard changed, outputting:\"\n        echo \"$current_clipboard\"\n    fi\n    last_clipboard=\"$current_clipboard\"\n    log \"Sleeping for $polling_interval_seconds seconds\"\n    sleep \"$polling_interval_seconds\"\ndone\n"
  },
  {
    "path": "bin/path_revoke_world_writeable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-13 17:23:33 +0100 (Sat, 13 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRevokes world writeable octal permission bit from any directory found in your \\$PATH,\nprinting those directories whose permission bis it changes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ntr ':' '\\n' <<< \"$PATH\" |\nwhile read -r directory; do\n    [ -d \"$directory\" ] || continue\n    # $sudo is assigned in lib depending on whether you have root perms or not\n    # shellcheck disable=SC2154\n    # try without sudo first as you'll probably be able to\n    chmod -v o-w \"$directory\" ||\n    $sudo chmod -v o-w \"$directory\"\ndone\n"
  },
  {
    "path": "bin/pldd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: $$\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-16 02:32:28 +0200 (Mon, 16 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOn Linux parses the /proc/<pid>/maps to list all dyanmic so libraries that a program is using\n\nThe runtime equivalent of the classic Linux ldd command\n\nBecause sometimes the system pldd command gives results like this:\n\n    pldd: cannot attach to process 32781: Operation not permitted\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pid>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\npid=\"$1\"\n\nif ! is_linux; then\n    usage \"ERROR: this script only runs on Linux\"\nfi\n\nmax_pid=\"$(cat /proc/sys/kernel/pid_max)\"\n\nif ! is_int \"$max_pid\"; then\n    die \"ERROR: failed to determine max pid of the system, got: $max_pid\"\nfi\n\nif ! is_int \"$pid\" ||\n   ! [ \"$pid\" -ge 1 ] ||\n   ! [ \"$pid\" -le \"$max_pid\" ]; then\n    die \"Error: PID '$pid' is not in the valid range of 1 to $max_pid\"\nfi\n\nmaps_file=\"/proc/$pid/maps\"\n\nif [ ! -f \"$maps_file\" ]; then\n    echo \"Could not find /proc/$pid/maps\"\n    exit 1\nfi\n\nwhile IFS= read -r line; do\n    # last field contains the file path\n    lib_path=\"$(awk '{print $NF}' <<< \"$line\")\"\n\n    # if the path contains in '.so' its a shared library\n    if [[ \"$lib_path\" =~ \\.so$ ]] ||\n       [[ \"$lib_path\" =~ \\.so\\. ]]; then\n        realpath=$(readlink -f \"$lib_path\" 2>/dev/null)\n        if [ -n \"$realpath\" ]; then\n            echo \"$realpath\"\n        else\n            echo \"$lib_path\"\n        fi\n    fi\ndone < \"$maps_file\" |\nsort -u\n"
  },
  {
    "path": "bin/processes_ram_sum.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: chrome\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-14 18:28:20 +0400 (Thu, 14 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSums the RAM usage of all processes matching a given regex in GB to one decimal place\n\nWriten to find out how much RAM Google Chrome and all its many helper processes were actually consuming\nsince my fancy M3 Max was using 100/128GB RAM and the top few processes only accounted for roughly ~10GB of that\n\nIt turned out Chrome was taking around 52GB for all the many tabs I had open\nwhile IntelliJ and its plugins were taking 17.8GB rather than the 5.8GB reported\nfor the main IntellJ process in Activity Monitor, which is misleading,\nwhile jcef processes were taking another 5.6GB\n\nRegex matches the entire process line return by 'ps aux' so this catches processes running out of installation\ndirectories that may have different names\n\nThis means that\n\n    ${0##*/} google\n\nwill sum both Google Chrome and Google Drive processes which is why\n\n    ${0##*/} google\n\nresults in slightly higher GB sum than\n\n    ${0##*/} chrome\n\n\nTested on Mac, but should work on Linux too\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<process_regex>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nprocess_regex=\"$1\"\n\n# cannot use pgrep because it doesn't output RSS memory used and would necessitate iterating over pids with something like\n#\n#   ps -o rss= -p \"$pid\"\n#\n# which would be lots of needless inefficient process forks\n#\n# shellcheck disable=SC2009\nmemory_kb=\"$(ps aux | grep -Ei \"$process_regex\" | grep -v grep | awk '{sum += $6} END {print sum}')\"\n\n# convert KB to GB and round to one decimal place\nmemory_gb=\"$(awk \"BEGIN {printf \\\"%.1f\\\", $memory_kb / 1024 / 1024}\")\"\n\necho \"Total memory used by all processes matching the '$process_regex' regex: $memory_gb GB\"\n"
  },
  {
    "path": "bin/progress_dots.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-03 17:47:02 +0000 (Tue, 03 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick pipe script to give progress dots on stderr while you pipe or redirect > to a file\n#\n# eg. some_big_command | progress_dots.sh > file.txt\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nLINES_PER_DOT=\"${LINES_PER_DOT:-100}\"\n\nif ! [[ \"$LINES_PER_DOT\" =~ ^[[:digit:]]+$ ]]; then\n    echo \"LINES_PER_DOT must be an integer!\" >&2\n    exit 1\nfi\n\ncount=0\n\nif [ $# -gt 0 ]; then\n    \"$@\"\nelse\n    cat\nfi |\nwhile read -r line; do\n    ((count+=1))\n    perl -e \"if($count % $LINES_PER_DOT == 0){print STDERR '.'}\"\n    printf '%s\\n' \"$line\"\ndone\necho >&2\n"
  },
  {
    "path": "bin/random_number.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 100 110\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-17 12:28:21 +0700 (Tue, 17 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints a random integer between two integer arguments (inclusive)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<min> <max>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nmin=\"$1\"\nmax=\"$2\"\n\nif ! is_int \"$min\"; then\n    usage \"First arg is not an integer: $min\"\nelif ! is_int \"$max\"; then\n    usage \"Second arg is not an integer: $max\"\nfi\n\n#random_number=\"$(shuf -i \"$min-$max\" -n 1)\"\n\nrandom_number=\"$((RANDOM % (max - min + 1) + min))\"\n\necho \"$random_number\"\n"
  },
  {
    "path": "bin/random_select.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: one two three\n#\n#  Author: Hari Sekhon\n#  Date: 2016-08-01 17:53:24 +0100 (Mon, 01 Aug 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints one of the arguments via random selection\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<arg1> <arg2> [<arg3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nindex=0\n\ndeclare -a arg_array\n\nfor arg in \"$@\"; do\n    log \"Saving arg' $arg' at index '$index'\"\n    arg_array[index]=\"$arg\"\n    ((index += 1))\ndone\n\nnum_args=\"${#@}\"\n\nselected_index=\"$((RANDOM % num_args))\"\n\nlog \"Selecting arg index $selected_index\"\necho \"${arg_array[$selected_index]}\"\n"
  },
  {
    "path": "bin/random_string.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 00:45:44 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Generates a random alphanumeric string of the given length\n#\n# pwgen is also a good option, but may not always be installed, hence this is more portable\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints a random alphanumeric string of the given character length\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<num_chars>\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nlen=\"${1:-32}\"\n\nif ! is_int \"$len\"; then\n    usage \"Invalid non-integer string length given as first argument\"\nfi\n\n# fixes illegal byte error in tr / sed etc\nexport LC_ALL=C\n\n#tr -duc 'A-Za-z0-9' < /dev/urandom |\ntr -duc '[:alnum:]' < /dev/urandom |\n#fold -w \"$len\" | head -n1\nhead -c \"$len\" || :  # head returns error code 141 but succeeds\n"
  },
  {
    "path": "bin/retry.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-21 20:41:52 +0000 (Fri, 21 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Had to do this in /bin/sh not bash so that it can be used on bootstrapping Alpine builds\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\ntries=\"${TRIES:-3}\"\n\nusage(){\n    cat <<EOF\n\nusage: ${0##*/} [<num_tries>] <command>\n\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        --help)  usage\n                 ;;\n    esac\ndone\n\n# bash only\n#if [[ \"${1:-}\" =~ ^[[:digit:]]+$ ]]; then\nif echo \"${1:-}\" | grep -Eq '^[[:digit:]]+$'; then\n    tries=\"$1\"\n    shift\nfi\n\nif [ $# -lt 1 ]; then\n    usage\nfi\n\nset +e\n# {1..$tries} doesn't work and `seq` is a needless fork\n# bash only\n#for ((i=0; i < tries; i++)); do\nfor _ in $(seq \"$tries\"); do\n    eval \"$@\"\n    result=$?\n    if [ $result -eq 0 ]; then\n        break\n    fi\ndone\nexit \"$result\"\n"
  },
  {
    "path": "bin/run.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  stdin: Driving &amp; road trips\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-20 16:01:28 +0000 (Fri, 20 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Helper script for calling from vim function to run programs or execute with args extraction\n#\n# Runs the value of the 'run:' header from the given file\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/.bash.d/aliases.sh\"\n\n# shellcheck disable=SC1090,SC1091\n . \"$srcdir/.bash.d/functions.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns one or more files\n\nAuto-determines the file types, any run or arg headers and executes each file using the appropriate script or CLI tool\n\nUseful to call from vim or IDEs via hotkeys to portably standardize quick build testing while editing\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"file1 [file2 file3 ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# defer expansion\n# shellcheck disable=SC2016\ntrap_cmd 'exitcode=$?; echo; echo \"Exit Code: $exitcode\"'\n\nfilename=\"$1\"\nshift || :\n\n# examples:\n#\n# #  run: kubectl apply -f file.yaml\n# // run: go run file.go\n# -- run: psql -f file.sql\nrun_cmd=\"$(parse_run_args \"$filename\")\"\nrun_stdin=\"$(parse_run_stdin \"$filename\")\"\n\n# if stdin: is defined in the file, replace the stdin to this script with it\n# so whichever program is called inherits this stdin\nif not_blank \"$run_stdin\"; then\n    exec <<< \"$run_stdin\"\nfi\n\nfilename=\"$(readlink -f \"$filename\")\"\ndirname=\"$(dirname \"$filename\")\"\nbasename=\"${filename##*/}\"\n\ncd \"$dirname\"\n\ndocker_compose_up(){\n    local dc_args=()\n    local env_file=\"${filename%.*}.env\"\n    if [ -f \"$env_file\" ]; then\n        dc_args+=(--env-file \"$env_file\")\n    fi\n    docker-compose -f \"$filename\" ${dc_args:+\"${dc_args[@]}\"} up\n}\n\nif [ -n \"$run_cmd\" ]; then\n    eval \"$run_cmd\" ${run_stdin:+<<< \"$run_stdin\"}\n# fails to do the open for d2 diagrams\n#elif test -x \"$basename\"; then\n#    ./\"$basename\"\nelse\n    case \"$basename\" in\n             Makefile)  make\n                        ;;\n           Dockerfile)  if [ -f Makefile ]; then\n                            make\n                        else\n                            docker build .\n                        fi\n                        ;;\n*docker-compose*.y*ml)  docker_compose_up\n                        ;;\n              Gemfile)  bundle install\n                        ;;\n                    Fastfile) if [[ \"$(readlink -f \"$basename\")\" =~ /fastlane/Fastfile ]]; then\n                            cd \"..\"\n                            fastlane \"$@\"\n                        fi\n                        ;;\n     cloudbuild*.y*ml)  gcloud builds submit --config \"$basename\" .\n                        ;;\n   kustomization.yaml)  kustomize build --enable-helm\n                        ;;\n               .envrc)  direnv allow .\n                        ;;\n                 *.d2)  if test -x \"$basename\"; then\n                            # use its shebang line to get the settings like --theme or --layout elk eg. for github_actions_cicd.d2 in https://github.com/HariSekhon/Diagrams-as-Code\n                            ./\"$basename\"\n                            # shellcheck disable=SC2012\n                            latest_image=\"$(ls -t \"${basename%.d2}\".{png,svg} 2>/dev/null | head -n1 || :)\"\n                        else\n                            image=\"${basename%.d2}.svg\"\n                            d2 --dark-theme 200 \"$basename\" \"$image\"\n                            latest_image=\"$image\"\n                        fi\n                        open \"$latest_image\"\n                        ;;\n                 *.go)  eval go run \"'$basename'\" \"$(\"$srcdir/lib/args_extract.sh\" \"$basename\")\"\n                        ;;\n                 *.tf)  #terraform plan\n                        terraform apply\n                        ;;\n       terragrunt.hcl)  terragrunt apply\n                        ;;\n *.pkr.hcl|*.pkr.json)  packer init \"$basename\" &&\n                        packer build \"$basename\"\n                        ;;\n                 *.md)  bash -ic \"cd '$dirname'; gitbrowse\"\n                        ;;\n                 *.gv)  file_png=\"${basename%.gv}.png\"\n                        dot -T png \"$basename\" -o \"$file_png\" >/dev/null && open \"$file_png\"\n                        ;;\n            .pylintrc)  pylint ./*.py\n                        ;;\n                    *)  if [[ \"$basename\" =~ /docker-compose/.+\\.ya?ml$ ]]; then\n                            docker_compose_up\n                        elif [[ \"$basename\" =~ \\.ya?ml$ ]] &&\n                           grep -q '^apiVersion:' \"$basename\" &&\n                           grep -q '^kind:'       \"$basename\"; then\n                            # a yaml with these apiVersion and kind fields is almost certainly a kubernetes manifest\n                            kubectl apply -f \"$basename\"\n                            exit 0\n                        fi\n                        if ! [ -x \"$basename\" ]; then\n                            echo \"ERROR: file '$filename' is not set executable!\" >&2\n                            exit 1\n                        fi\n                        if [ \"$(type -P \"$basename\" 2>/dev/null)\" != \"$PWD/$basename\" ]; then\n                            basename=\"./$basename\"\n                        fi\n                        args=\"$(\"$srcdir/lib/args_extract.sh\" \"$basename\")\"\n                        echo \"'$basename'\" \"$args\" >&2\n                        eval \"'$basename'\" \"$args\"\n                        ;;\n    esac\nfi\n"
  },
  {
    "path": "bin/sbtw",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-08-24 15:40:04 +0200 (Thu, 24 Aug 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nSBT_VERSION=\"${SBT_VERSION:-1.3.10}\"\n\nif [ -z \"$JAVA_HOME\" ]; then\n    echo \"\\$JAVA_HOME not set!\"\n    exit 1\nfi\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\n#JAR=\"$HOME/.sbt/sbt-launch-$SBT_VERSION.jar\"\n\nbootdir=\"$HOME/.sbt/boot\"\n\nmkdir -vp \"$bootdir\"\n\nif ! test -d \"$bootdir/sbt-$SBT_VERSION\"; then\n    #URL=\"https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/$SBT_VERSION/sbt-launch.jar\"\n    #URL=\"https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/$SBT_VERSION/jars/sbt-launch.jar\"\n    URL=https://piccolo.link/sbt-1.3.3.tgz\n    echo \"Downloading sbt $SBT_VERSION from $URL\"\n    pushd \"$bootdir\"\n    wget -O \"sbt.tgz\" \"$URL\"\n    tar zxf \"sbt.tgz\"\n    mv -- sbt \"sbt-$SBT_VERSION\"\n    #mv sbt/bin/sbt-launcher.jar \"$JAR\"\n    rm -f -- \"sbt.tgz\"\n    popd\nfi\n\n# calling launcher manually is not necessary\n#exec \"$JAVA_HOME/bin/java\" \\\n#    \"${JVM_ARGS:-}\" \\\n#    -d64 \\\n#    -noverify \\\n#    -Dfile.encoding=UTF8 \\\n#    -Dsbt.boot.directory=\"$HOME/.sbt/boot\" \\\n#    -Xmx1024M -Xss1M -XX:MaxPermSize=256m \\\n#    -XX:+CMSClassUnloadingEnabled \\\n#    -jar \"$JAR\" \"$@\"\n\nexec \"$bootdir/sbt-$SBT_VERSION/bin/sbt\" \"$@\"\n"
  },
  {
    "path": "bin/scan_duplicate_macs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-03-14 18:41:35 +0000 (Thu, 14 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$@\"\n        echo\n    fi\n    cat <<EOF\n\nScans locally attached networks and prints duplicate MAC addresses\n\nUseful for finding duplicate MACs and also finding hosts behind a VIP or similar floating address (VRRP, HSRP)\n\nUses fping to ping all addresses on all local subnets and then checks the local arp cache\n\nCaveat: won't find a VIP on the local host this script is run on\n\nusage: ${0##*/}\n\nEOF\n    exit 3\n}\n\nuntil [ $# -lt 1 ]; do\n    case $1 in\n    -h|--help)  usage\n                ;;\n            *)  usage \"unknown argument: $1\"\n                ;;\n    esac\n    # shellcheck disable=SC2317\n    shift || :\ndone\n\nif [ -z \"${NOSCAN:-}\" ]; then\n    if [ \"$(uname -s)\" = \"Darwin\" ]; then\n        networks=\"$(netstat -rn | awk '/link#/{print $1}' | grep -e '[[:digit:]]\\+\\.[[:digit:]]\\+\\.[[:digit:]]\\+\\.[[:digit:]]\\+')\"\n    else # assume Linux\n        networks=\"$(ip addr | awk '/inet /{print $2}' | grep -v '^127\\.')\"\n    fi\n\n    for network in $networks; do\n        echo \"scanning network $network...\" >&2\n        fping -q -r 0 -c 1 -B 1 -g \"$network\" >&2 || :\n    done\nfi\n\n# Linux\n#arp -e |\n# BSD - more portable, both Linux and Mac support this\narp -a |\n# incomplete seems to only appear on arp on Linux, for both -a and -e formats\n# Linux arp -e\n#awk '!/incomplete/{print $3}' |\n# BSD arp -a (works on Linux too)\nawk '!/incomplete/{print $4}' |\ngrep -vi \"ff:ff:ff:ff:ff:ff\" |\nsort |\nuniq -d |\nwhile read -r mac; do\n    # Linux\n    #arp -e |\n    # BSD - more portable, both Linux and Mac support this\n    arp -a |\n    grep \"$mac\"\ndone\n"
  },
  {
    "path": "bin/screen_terminal_to_clipboard.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-11 01:05:32 -0300 (Wed, 11 Feb 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps the GNU Screen terminal output to a temp file and\ncopies to clipboard for sharing & debugging purposes\n\nUses adjacent scripts:\n\n    screen_terminal_to_stdout.sh\n\n    copy_to_clipboard.sh\n\nYou can pass screen options as args, see screen_terminal_to_stdout.sh for details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<screen_options>]\"\n\nhelp_usage \"$@\"\n\n# indicate to screen_terminal_to_stdout.sh not to remove the term\nexport SCREEN_TERMINAL_NO_DELETE_TEMPFILE=1\n\n\"$srcdir/screen_terminal_to_stdout.sh\" \"$@\" |\n\"$srcdir/copy_to_clipboard.sh\"\n\ntimestamp \"Copied GNU Screen to Clipboard\"\n"
  },
  {
    "path": "bin/screen_terminal_to_stdout.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-11 01:05:32 -0300 (Wed, 11 Feb 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps the GNU Screen terminal output to stdout\n\nRetains the temp file if this environment variable is set to any value:\n\n    export SCREEN_TERMINAL_NO_DELETE_TEMPFILE=1\n\nUsed by adjacent script:\n\n    screen_terminal_to_clipboard.sh\n\nYou can pass screen options as args such as:\n\n    -S <session_name>\n    -p <window_number>\n\nSee window numbers using your 'Ctrl-A + w' hotkey combo or in stdout via this command:\n\n    screen -S <session_name>] -Q windows\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<screen_options>]\"\n\nhelp_usage \"$@\"\n\n# indicate to screen_terminal_to_stdout.sh not to remove the term\nexport SCREEN_TERMINAL_NO_DELETE_TEMPFILE=1\n\ntmpfile=\"$(mktemp)\"\n\nif [ -z \"${SCREEN_TERMINAL_NO_DELETE_TEMPFILE:-}\" ]; then\n    trap_cmd \"rm -f \\\"$tmpfile\\\"\"\nelse\n    timestamp \"Dumping GNU Screen terminal output to tempfile: $tmpfile\"\nfi\n\nscreen -X hardcopy -h \"$tmpfile\" \"$@\"\n\necho >&2\n\ncat \"$tmpfile\"\n"
  },
  {
    "path": "bin/shorten_text_selection.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-19 01:44:50 +0300 (Mon, 19 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShortens the selected text in the prior window\n\n- Copies the selected text to the clipboard\n- Replaces \\\"and\\\" with \\\"&\\\"\n- Removes multiple blank lines between paragraphs (which can result from the copy/paste pipeline otherwise)\n- Pastes the clipboard text back over the selected text\n\nI use this a lot for LinkedIn comments in browser due to the short 1250 character limit\n\nTested on macOS 14, and Debian 11 with Xfce and LDXE desktop environments\n\nDoes not work on Debian 12 Gnome due to wayland lack of virtual keyboard support and even on Debian 12 Gnome on Xorg\nthe keystrokes do not come out properly, not sure why yet, ping me if you have time to figure out why as that's not\nmy day-to-day system to justify spending more time testing on it right now\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\nif is_mac; then\n    exec \"$srcdir/../applescript/shorten_text_selection.scpt\"\nfi\n\nfor bin in xdotool xclip; do\n    if ! type -P \"$bin\" &>/dev/null; then\n        timestamp \"Command '$bin' not found in \\$PATH, attempting to install...\"\n        \"$srcdir/../packages/install_packages.sh\" \"$bin\"\n    fi\ndone\n\ncheck_bin xdotool\ncheck_bin xclip\n\ntimestamp \"Switching back to previous application\"\n# for Debian 12 Gnome\nif [ \"${XDG_SESSION_TYPE:-}\" = wayland ]; then\n    #timestamp \"Detected wayland rather than Xorg, attempting to use ydotool instead\"\n    #if ! type -P ydotool &>/dev/null; then\n    #    timestamp \"Command 'ydotool' not found in \\$PATH, attempting to install...\"\n    #    \"$srcdir/../packages/install_packages.sh\" ydotool\n    #fi\n    ##sudo ydotool key 56:1 15:1 15:0 56:0\n    #sudo ydotool key 56:1    # press Alt\n    #sleep 0.05\n    #sudo ydotool key 15:1    # press Tab\n    #sleep 0.05\n    #sudo ydotool key 15:0    # release Tab\n    #sleep 0.05\n    #sudo ydotool key 56:0    # release Alt\n\n    #timestamp \"Detected wayland rather than Xorg, attempting to use wtype instead\"\n    #if ! type -P wtype &>/dev/null; then\n    #    timestamp \"Command 'wtype' not found in \\$PATH, attempting to install...\"\n    #    \"$srcdir/../packages/install_packages.sh\" wtype\n    #fi\n    #wtype -M alt -k tab -m alt\n    #\n    # Results in error:\n    #\n    #   Compositor does not support the virtual keyboard protocol\n    #\n    # This is because Gnome intentionally disables it for security - you must switch to an X11 session\n    die \"ERROR: wayland based UI is not supported at this time due to lack of virtual keyboard protocol support (intentional by the developers)\"\nelse\n    xdotool keydown alt\n    #xdotool keydown Tab\n    #xdotool keyup Tab\n    xdotool key Tab\n    xdotool keyup alt\nfi\nsleep 0.3\necho\n\ntimestamp \"Copying selected content (Ctrl-C)\"\nxdotool key ctrl+c\nsleep 0.1\necho\n\ntimestamp \"Replacing selected clipboard content - 'and' with '&' and collapse multiple blank lines\"\nxclip -o -selection clipboard |\nsed -E 's/\\band\\b/\\&/g' |\ncat -s |\nxclip -selection clipboard\nsleep 0.1\necho\n\ntimestamp \"Pasting modified clipboard content back to app (Ctrl-v)\"\nxdotool key ctrl+v\n"
  },
  {
    "path": "bin/shred_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-20 10:41:18 +0100 (Sat, 20 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOverwrites a given file using dd to prevent recovery of sensitive information\n\nWARNING: you should only do this on old HDD, not SSD, hard drives which have a limited number of writes - you may shorten an SSD's lifespan if you over use this\n\nUse this only if you've accidentally written some credential / sensitive data to disk and then already deleted it, leaving the file data on disk\nwithout a filename to inode pointer to overwrite just the file\n\nThis prevents file recovery tools from stealing your sensitive data, credentials etc. because otherwise\nthe file data is still there on disk without a filename inode pointer and file recovery tools may be able to recover and steal it\n\nYour disk should also be encrypted anyway for security best practices, in which case you shouldn't need to do this\n\nWorks by writing random data into a large file in the \\$PWD until the disk is full and then deletes the file. This is an old trick from the 2000s\n\nSee also:\n\nOn Mac:\n\n    rm -P   overwrites file 3 times before deleting it (-P switch is available on BSD 'rm' variant only)\n\non Linux:\n\n    srm     from secure-delete package\n    shred\n    wipe\n\n    sfill   works similar to this script except does 2 overwrites - DoD secure standard is 7 overwrites\n\n    sswap   overwrites your swap device\n\n    sdmem   overwrites your RAM to prevent warm boot attacks retrieving sensitive credentials or data\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename> [<passes>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nfile=\"$1\"\npasses=\"${2:-7}\"\n\nif ! [ -f \"$file\" ]; then\n    die \"File '$file' does not exit\"\nfi\n\nfilesize=\"$(du \"$file\" | awk '{print $1}')\"\n\ntimestamp \"Overwriting file '$file'\"\nfor (( i = 0; i < passes; i++ )); do\n    timestamp \"overwrite pass 1...\"\n    dd bs=\"$filesize\" count=1 if=/dev/urandom of=\"$file\"\ndone\n\ntimestamp \"Removing file:\"\nrm -fv \"$file\"\n"
  },
  {
    "path": "bin/shred_free_space.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-20 10:41:18 +0100 (Sat, 20 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShreds free space by overwriting it with random data to prevent recovery of sensitive information\n\nDoes a single overwrite (not very secure), specify 7 passes for DoD secure standard\n\nWARNING: you should only do this on old HDD, not SSD, hard drives which have a limited number of writes - you may shorten an SSD's lifespan if you over use this\n\nUse this only if you've accidentally written some credential / sensitive data to disk and then already deleted it, leaving the file data on disk\nwithout a filename to inode pointer to overwrite just the file\n\nThis prevents file recovery tools from stealing your sensitive data, credentials etc. because otherwise\nthe file data is still there on disk without a filename inode pointer and file recovery tools may be able to recover and steal it\n\nYour disk should also be encrypted anyway for security best practices, in which case you shouldn't need to do this\n\nWorks by writing random data into a large file in the \\$PWD until the disk is full and then deletes the file. This is an old trick from the 2000s\n\nSee also:\n\nOn Mac:\n\n    rm -P   overwrites file 3 times before deleting it (-P switch is available on BSD 'rm' variant only)\n\non Linux:\n\n    srm     from secure-delete package\n    shred\n    wipe\n\n    sfill   works similar to this script except does 2 overwrites - DoD secure standard is 7 overwrites\n\n    sswap   overwrites your swap device\n\n    sdmem   overwrites your RAM to prevent warm boot attacks retrieving sensitive credentials or data\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\" [<dir> <passes>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\ndir=\"${1:-.}\"\npasses=\"${2:-1}\"\n\nif ! [ -d \"$dir\" ]; then\n    die \"Directory '$dir' does not exit\"\nfi\n\ncd \"$dir\"\n\ntmpfile=\"shredfile.binary\"\n\ntrap_cmd \"if [ -f \\\"$tmpfile\\\" ]; then timestamp 'Removing tmpfile \\\"$tmpfile\\\"'; rm -f \\\"$tmpfile\\\"; fi\"\n\n# mac can use 1m or 1M but Linux dd requires 1M capitalized\n#bs='1m'\n#if uname -s | grep -q Darwin; then\n#    bs='1M'\n#fi\n\ntimestamp \"Writing tmpfile '$PWD/$tmpfile'\"\nfor (( i = 0; i < passes; i++ )); do\n    timestamp \"overwrite pass 1...\"\n    # use 1M capitalized for compatibility with both Linux and Mac\n    dd if=/dev/urandom of=\"shredfile.binary\" bs=\"1M\" || :  # will hit out of space error and error out otherwise\n    timestamp \"Removing tmpfile '$tmpfile'\"\n    rm -f \"$tmpfile\"\ndone\n"
  },
  {
    "path": "bin/smart_quotes_replace.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-27 22:28:37 +0800 (Thu, 27 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReplace smart quotes with their regular counterparts using sed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<files_and_sed_options>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# these are supposed to be unquoted quotes\n# shellcheck disable=SC1111\nsed \"\n    s/“/\\\"/g;\n    s/”/\\\"/g;\n    s/‘/'/g;\n    s/’/'/g;\n\" \"$@\"\n"
  },
  {
    "path": "bin/spasticcase.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-17 22:35:54 +0700 (Mon, 17 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChange input to sPaStIcCaSe for replying to people who don't understand economics, demographics or logic\n\nAccepts a string, file or standard input\n\nOutputs to stderr and copies to clipboard for convenience\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_or_file>]\"\n\nhelp_usage \"$@\"\n\ncapitalize_alternate(){\n    sed -E 's/(.)/\\L\\1/g; s/(.)(.)/\\1\\U\\2/g'\n}\n\nif [ $# -eq 0 ]; then\n    log \"Reading from standard input\"\n    cat \"$@\"\nelse\n    if [[ \"$1\" =~ ^.?/ ]] || [ -f \"$1\" ]; then\n        log \"Reading from files: $*\"\n        cat \"$@\"\n    else\n        echo \"$*\"\n    fi\nfi |\ncapitalize_alternate |\ntee >(\"$srcdir/copy_to_clipboard.sh\")\n# copies to clipboard and also sends to stdout to allow further pipeline processing\n"
  },
  {
    "path": "bin/spasticcase2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-17 22:35:54 +0700 (Mon, 17 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChange input to sPaStIcCaSe for replying to people who don't understand economics, demographics or logic\n\nAccepts a string, file or standard input\n\nOutputs to stderr and copies to clipboard for convenience\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_or_file>]\"\n\nhelp_usage \"$@\"\n\ncapitalize_alternate(){\n    awk '{\n        for (i = 1; i <= length($0); i++) {\n            c = substr($0, i, 1);\n            if (i % 2 == 0) {\n                printf toupper(c);\n            } else {\n                printf tolower(c);\n            }\n        }\n        print \"\";\n    }'\n}\n\nif [ $# -eq 0 ]; then\n    log \"Reading from standard input\"\n    cat \"$@\"\nelse\n    if [[ \"$1\" =~ ^.?/ ]] || [ -f \"$1\" ]; then\n        log \"Reading from files: $*\"\n        cat \"$@\"\n    else\n        echo \"$*\"\n    fi\nfi |\ncapitalize_alternate |\ntee >(\"$srcdir/copy_to_clipboard.sh\")\n# copies to clipboard and also sends to stdout to allow further pipeline processing\n"
  },
  {
    "path": "bin/split.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  # false positives\n#  shellcheck disable=SC2178,SC2128\n#\n#  Author: Hari Sekhon\n#  Date: 2019-03-05 18:18:13 +0000 (Tue, 05 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nusage(){\n    if [ -n \"$*\" ]; then\n        echo \"$@\"\n        echo\n    fi\n    cat <<EOF\n\nSplits big file(s) in to \\$PARTS parts (defaults to the number of CPU cores)\n\nUseful for easy parallelizing things that don't easily lend themselves to parallelization like\nanonymize.py from DevOps Python Tools which needs successive ordered anonymization rules\n\nusage: ${0##*/} <files>\n\n-p --parts  Number of parts to split files in to (\\$PARTS, defaults to number of CPU cores)\n-h --help   Show usage and exit\n\nEOF\n    exit 3\n}\n\nif [ $# -eq 0 ]; then\n    usage \"no file arguments given\"\nfi\n\nfor x in \"$@\"; do\n    case \"$x\" in\n        -h|--help)  usage\n                    ;;\n    esac\ndone\n\ncheck_bin split\n#check_bin parallel\n\nparts=\"${PARTS:-}\"\n\nif [ -z \"$parts\" ]; then\n    parts=\"$(cpu_count)\"\nfi\n\nfile_list=\"\"\n\nwhile [ $# -gt 0 ]; do\n    case $1 in\n           -p|--parts)  parts=\"$2\"\n                        shift\n                        ;;\n         -h|--help|-*)  usage\n                        ;;\n                    *)  file_list=\"$file_list $1\"\n                        ;;\n    esac\n    shift\ndone\n\nfor filename in $file_list; do\n    echo \"Splitting $filename in to $parts parts\"\n    if [ \"$(uname -s)\" = \"Darwin\" ]; then\n        linecount=\"$(wc -l < \"$filename\" | awk '{print $1}')\"\n        parts=\"$(bc <<< \"$linecount / $parts\")\"\n        split -l \"$parts\" \"$filename\" \"$filename.\"\n    else\n        split -d -n \"l/$parts\" \"$filename\" \"$filename.\"\n    fi\ndone\n"
  },
  {
    "path": "bin/sqlite.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-10 18:57:07 +0100 (Mon, 10 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick SQLite with the Chinook sample database\n\nDownloads the Chinook database if not present, then drops in to SQLite with it loaded for fast testing\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ndb=chinook.sqlite\n\ncd \"$srcdir\"\n\nif ! [ -f \"$db\" ]; then\n    wget -O \"$db\" 'https://github.com/lerocha/chinook-database/blob/master/ChinookDatabase/DataSources/Chinook_Sqlite.sqlite?raw=true'\nfi\n\nsqlite3 \"$db\"\n"
  },
  {
    "path": "bin/ssl_get_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google.com\n#\n#  Author: Hari Sekhon\n#  Date: 2019-04-11 18:48:01 +0100 (Thu, 11 Apr 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nDumps the SSL certificate blocks for hosts given as arguments, using OpenSSL\n\nPort defaults to 443 if not given\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"host1[:port] [ host2:[:port]] [host3[:port]] ... ]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor host in \"$@\"; do\n    host_port=\"$host\"\n    if ! [[ \"$host_port\" =~ : ]]; then\n        host_port=\"$host_port:443\"\n    fi\n    #if ! openssl s_client -connect \"$host_port\" </dev/null 2>/dev/null | sed -n '/BEGIN/,/END/p'; then\n    #    echo \"ERROR connecting to $host_port\"\n    #    exit 1\n    #fi\n    # sed returns 1\n    openssl s_client -connect \"$host_port\" </dev/null 2>/dev/null | sed -n '/BEGIN/,/END/p' || :\ndone\n"
  },
  {
    "path": "bin/ssl_verify_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google.com yahoo.com\n#\n#  Author: Hari Sekhon\n#  Date: 2019-04-11 18:48:01 +0100 (Thu, 11 Apr 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nVerifies the SSL certificate for each host given as arguments, using OpenSSL\n\nPort defaults to 443 if not specified\n\n\nFor a much better version of this see check_ssl_cert.pl in the Advanced Nagios Plugins Collection:\n\n    check_ssl_cert.pl - checks Expiry days remaining, Domain, Subject Alternative Names, SNI\n\n    https://github.com/HariSekhon/Nagios-Plugins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"host1[:port] [ host2:[:port]] [host3[:port]] ... ]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexitcode=0\n\n# otherwise will silently fail getting openssl output on incorrect host\nset +o pipefail\n\nfor host in \"$@\"; do\n    host_port=\"$host\"\n    if ! [[ \"$host_port\" =~ : ]]; then\n        host_port=\"$host_port:443\"\n    fi\n    # openssl returns 1 regardless of whether host/cert is valid/invalid\n    output=\"$(openssl s_client -connect \"$host_port\" < /dev/null 2>/dev/null |\n              grep Verify |\n              sed 's/^[[:space:]]*//; s/[[:space:]]*$//')\"\n    if [ -n \"${output:-}\" ]; then\n        echo \"$output\"\n        if ! [[ \"$output\" =~ Verify[[:space:]]*return[[:space:]]*code:[[:space:]]*0 ]]; then\n            exitcode=1\n        fi\n    else\n        echo \"Failed to connect\"\n        if [ $exitcode -eq 0 ]; then\n            exitcode=1\n        fi\n    fi\ndone\n"
  },
  {
    "path": "bin/ssl_verify_cert_by_ip.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-22 13:03:06 +0100 (Thu, 22 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nVerifies the SSL certificates for FQDNs at specific IP addresses via curl\n\nUseful to test SSL source addresses for CDNs, such as Cloudflare Proxied sources before enabling SSL Full-Strict Mode for end-to-end, or Kubernetes ingresses (see also curl_k8s_ingress.sh)\n\nPort defaults to 443 if not specified\n\nIf any of the arguments are a file, then reads the contents of that file as the 'FQDN,IP,Port' tuples, one per line either comma or space separated. Useful for bulk testing.\n\n\nFor a better version of this see check_ssl_cert.pl in the Advanced Nagios Plugins Collection:\n\n    check_ssl_cert.pl - checks Expiry days remaining, Domain, Subject Alternative Names, SNI\n\n    https://github.com/HariSekhon/Nagios-Plugins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"fqdn:ip[:port] [ fqdn2:ip[:port]] [fqdn3:ip[:port]] ... ]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexitcode=0\n\n# otherwise will silently fail getting openssl output on incorrect host\nset +o pipefail\n\ncheck_fqdn_ip(){\n    local fqdn=\"$1\"\n    local ip=\"$2\"\n    local port=\"${3:-443}\"\n    local url=\"https://$fqdn\"\n    echo -n \"Checking SSL '$url' at '$ip': \"\n    set +e\n    if curl -s --fail \"$url\" --resolve \"$fqdn:$port:$ip\" &>/dev/null; then\n        echo \"OK\"\n    else\n        echo \"FAILED\"\n        exitcode=1\n    fi\n    set -e\n}\n\nparse_arg(){\n    local arg=\"$1\"\n    fqdn=\"${arg%%:*}\"\n    ip_port=\"${arg#*:}\"\n    ip=\"${ip_port%%:*}\"\n    port=\"${ip_port##*:}\"\n    if [ \"$port\" = \"$ip\" ]; then\n       port=\"\"\n    fi\n    echo \"$fqdn\" \"$ip\" \"$port\"\n}\n\nprocess_arg(){\n    local arg=\"$1\"\n    read -r fqdn ip port < <(parse_arg \"$arg\")\n    check_fqdn_ip \"$fqdn\" \"$ip\" \"$port\"\n}\n\nvalidate_arg(){\n    local arg=\"$1\"\n    read -r fqdn ip port < <(parse_arg \"$arg\")\n    if [ -n \"$port\" ]; then\n        if ! [[ \"$port\" =~ ^[[:digit:]]+$ ]]; then\n            die \"Invalid Port '$port' detected in '$arg'\"\n        fi\n    fi\n    # shellcheck disable=SC2154\n    if ! [[ \"$ip\" =~ ^$ip_regex$ ]]; then\n        die \"Invalid IP '$ip' detected in '$arg'\"\n    fi\n    #if ! [[ \"$fqdn\" =~ ^$domain_regex$ ]]; then\n    #    die \"Invalid FQDN '$fqdn' given in '$arg'\"\n    #fi\n}\n\nparse_file(){\n    local filename=\"$1\"\n    sed 's/#.*//;\n         /^[[:space:]]*$/d;\n         s/[,:]/ /g' \"$filename\"\n}\n\nprocess_file(){\n    local filename=\"$1\"\n    parse_file \"$filename\" |\n    while read -r fqdn ip port; do\n        check_fqdn_ip \"$fqdn\" \"$ip\" \"$port\"\n    done\n}\n\nvalidate_file(){\n    local filename=\"$1\"\n    parse_file \"$filename\" |\n    while read -r fqdn ip port; do\n        validate_arg \"$fqdn:$ip:$port\"\n    done\n}\n\n# pre-validate before running anything, quicker\nfor arg; do\n    if [ -f \"$arg\" ]; then\n        validate_file \"$arg\"\n        continue\n    else\n        validate_arg \"$arg\"\n    fi\ndone\n\nfor arg; do\n    if [ -f \"$arg\" ]; then\n        process_file \"$arg\"\n        continue\n    else\n        process_arg \"$arg\"\n    fi\ndone\n\nexit $exitcode\n"
  },
  {
    "path": "bin/ssl_view_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google.com\n#\n#  Author: Hari Sekhon\n#  Date: 2021-01-27 17:00:21 +0000 (Wed, 27 Jan 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nFetches and translates the remote host's SSL cert to human readable text using OpenSSL\n\nPort defaults to 443 if not given\n\nUses adjacent script ssl_get_cert.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"host[:port]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nhost_port=\"$1\"\n\nif ! [[ \"$host_port\" =~ : ]]; then\n    host_port=\"$host_port:443\"\nfi\n\n\"$srcdir/ssl_get_cert.sh\" \"$host_port\" |\nopenssl x509 -noout -text\n"
  },
  {
    "path": "bin/text_filter_ending_substrings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-03 03:15:56 -0500 (Sat, 03 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a given patterns file of substring endings, print all lines that match in the following files or stdin\n\nUses awk to safely handle all characters as literals, unlike grep, while also maintaining end anchoring\nwhich you cannot do using grep -F.\n\nOptimized awk code uses a bucketing hash for performance to not attempt matching lines which are shorter\nthan patterns, reducing the number of match attempts\n\nBy default matching is case sensitive for accuracy, but if you want case insensitive matching, you can set\nenvironment variable TEXT_MATCH_CASE_INSENSITIVE to 1\n\n    export TEXT_MATCH_CASE_INSENSITIVE=1\n\nYou can also pass subshell content outputs for one of both 'files' using bash file descriptor replacement syntax:\n\n    ${0##*/} <(somecommand) <(somecommand2)\n\nUsed by script:\n\n    $srcdir/../spotify/spotify_delete_from_playlist_if_track_in_other_playlists.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<patterns.txt> [<data.txt> <data2.txt> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor arg; do\n    if ! [ -e \"$arg\" ]; then\n        die \"ERROR: not found: $arg\"\n    fi\ndone\n\npatterns_file=\"$1\"\nshift || :\n\nawk '\n    BEGIN {\n        ci = (ENVIRON[\"TEXT_MATCH_CASE_INSENSITIVE\"] == \"1\")\n    }\n\n    # block only executes for 1st file to build up the patterns hash\n    NR==FNR {\n        pattern = ci ? tolower($0) : $0\n        len = length(pattern)\n        # length bucketing optimization to not attempt to match lines that are shorter than length\n        patterns[len][pattern] = 1\n        #if (len > max) max = len\n        next\n    }\n    {\n        line = ci ? tolower($0) : $0\n        len_line = length(line)\n        for (pattern in patterns) {\n            if (len_line >= pattern) {\n                suffix = substr(line, len_line - pattern + 1)\n                if (suffix in patterns[pattern]) {\n                    print\n                    break\n                }\n            }\n        }\n    }\n' \"$patterns_file\" \"${@:-/dev/stdin}\"\n"
  },
  {
    "path": "bin/tmux_horizontal.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-14 05:27:20 +0300 (Wed, 14 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches tmux and runs the commands given as args in equally balanced horizontal panes\n\nFast way to launch a bunch of commands in an easily reviewable way\n\nIf there is only one arg which is a single digit integer, then launches that many \\$SHELL panes\n\nAutogenerates a new session name in the form of \\$PWD-\\$epoch for uniqueness\n\nExample:\n\n    ${0##*/} htop 'iostat 1' bash\n\n    ${0##*/} bash bash bash\n\n    ${0##*/} 3\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args='\"<command_1>\" \"<command_2>\" [\"<command_3>\" ...]'\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npwd=\"${PWD:-$(pwd)}\"\nepoch=\"$(date +%s)\"\n\n# cannot separate the session name with a dot as this breaks:\n#\n#   tmux has-session -t \"$session\"\n#\n# this command:\n#\n#   tmux has-session -t /Users/hari/mydir.1747191485\n#\n# results in this:\n#\n#   can't find window: /Users/hari/mydir\n#\nsession=\"$pwd-$epoch\"\n\ncmd1=\"$1\"\n\nshift || :\n\nif [ $# -eq 0 ]; then\n   if [[ \"$cmd1\" =~ ^[[:digit:]]$ ]]; then\n        shell=\"${SHELL:-bash}\"\n        count=\"$cmd1\"\n        cmd1=\"$shell\"\n        args=()\n        for ((i = 1; i < count; i++)); do\n            args+=(\"$shell\")\n        done\n        set -- \"${args[@]}\"\n    else\n        usage \"Error: two or more args required unless you are specifying a count of shell panes to launch, otherwise there would be no panes to split\"\n    fi\nfi\n\ntimestamp \"Starting new tmux session in detached mode called '$session' with command: $cmd1\"\ntmux new-session -d -s \"$session\" \"$cmd1\"\n\nif ! tmux has-session -t \"$session\"; then\n    die \"ERROR: tmux session exited too soon from first command: $cmd1\"\nfi\n\nfor cmd; do\n    timestamp \"Splitting the tmux pane horizontally and launching command: $cmd\"\n    tmux split-window -v -t \"$session\":0 \"$cmd\"\ndone\n\ntimestamp \"Balancing the tmux pane layout horizontally for tmux session: $session\"\ntmux select-layout -t \"$session\":0 even-vertical\n\ntimestamp \"Attaching to tmux session: $session\"\ntmux attach-session -t \"$session\"\n"
  },
  {
    "path": "bin/tmux_square.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-14 05:27:20 +0300 (Wed, 14 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches tmux and runs the commands given as args in a square tiled view\n\nFast way to launch a bunch of commands in an easily reviewable way\n\nIf args aren't given, launches a \\$SHELL in each pane, defaulting to bash if \\$SHELL is not set\n\nAutogenerates a new session name in the form of \\$PWD-\\$epoch for uniqueness\n\nExample:\n\n    ${0##*/} htop 'iostat 1' 'ping google.com'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args='\"<command_1>\" \"<command_2>\" \"<command_3>\" \"<command_4>\"'\n\nhelp_usage \"$@\"\n\nmax_args 4 \"$@\"\n\npwd=\"${PWD:-$(pwd)}\"\nepoch=\"$(date +%s)\"\n\n# cannot separate the session name with a dot as this breaks:\n#\n#   tmux has-session -t \"$session\"\n#\n# this command:\n#\n#   tmux has-session -t /Users/hari/mydir.1747191485\n#\n# results in this:\n#\n#   can't find window: /Users/hari/mydir\n#\nsession=\"$pwd-$epoch\"\n\nshell=\"${SHELL:-bash}\"\n\ncmd1=\"${1:-$shell}\"\ncmd2=\"${2:-$shell}\"\ncmd3=\"${3:-$shell}\"\ncmd4=\"${4:-$shell}\"\n\nshift || :\n\ntimestamp \"Starting new tmux session in detached mode called '$session' with command: $cmd1\"\ntmux new-session -d -s \"$session\" \"$cmd1\"\n\nif ! tmux has-session -t \"$session\"; then\n    die \"ERROR: tmux session exited too soon from first command: $cmd1\"\nfi\n\nfor cmd in \"$cmd2\" \"$cmd3\" \"$cmd4\"; do\n    timestamp \"Splitting the tmux window vertically and launching command: $cmd\"\n    tmux split-window -h -t \"$session\":0 \"$cmd\"\ndone\n\ntimestamp \"Balancing the tmux pane layout into a square tiled view for tmux session: $session\"\ntmux select-layout -t \"$session\":0 tiled\n\ntimestamp \"Attaching to tmux session: $session\"\ntmux attach-session -t \"$session\"\n"
  },
  {
    "path": "bin/tmux_vertical.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-05-14 05:27:20 +0300 (Wed, 14 May 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches tmux and runs the commands given as args in horizontally balanced panes\n\nFast way to launch a bunch of commands in an easily reviewable way\n\nIf there is only one arg which is a single digit integer, then launches that many \\$SHELL panes\n\nAutogenerates a new session name in the form of \\$PWD-\\$epoch for uniqueness\n\nExample:\n\n    ${0##*/} htop 'iostat 1'\n\n    ${0##*/} bash bash\n\n    ${0##*/} 2\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args='\"<command_1>\" \"<command_2>\" [\"<command_3>\" ...]'\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npwd=\"${PWD:-$(pwd)}\"\nepoch=\"$(date +%s)\"\n\n# cannot separate the session name with a dot as this breaks:\n#\n#   tmux has-session -t \"$session\"\n#\n# this command:\n#\n#   tmux has-session -t /Users/hari/mydir.1747191485\n#\n# results in this:\n#\n#   can't find window: /Users/hari/mydir\n#\nsession=\"$pwd-$epoch\"\n\ncmd1=\"$1\"\n\nshift || :\n\nif [ $# -eq 0 ]; then\n   if [[ \"$cmd1\" =~ ^[[:digit:]]$ ]]; then\n        shell=\"${SHELL:-bash}\"\n        count=\"$cmd1\"\n        cmd1=\"$shell\"\n        args=()\n        for ((i = 1; i < count; i++)); do\n            args+=(\"$shell\")\n        done\n        set -- \"${args[@]}\"\n    else\n        usage \"Error: two or more args required unless you are specifying a count of shell panes to launch, otherwise there would be no panes to split\"\n    fi\nfi\n\ntimestamp \"Starting new tmux session in detached mode called '$session' with command: $cmd1\"\ntmux new-session -d -s \"$session\" \"$cmd1\"\n\nif ! tmux has-session -t \"$session\"; then\n    die \"ERROR: tmux session exited too soon from first command: $cmd1\"\nfi\n\nfor cmd; do\n    timestamp \"Splitting the tmux window vertically and launching command: $cmd\"\n    tmux split-window -h -t \"$session\":0 \"$cmd\"\ndone\n\ntimestamp \"Balancing the tmux pane layout vertically for tmux session: $session\"\ntmux select-layout -t \"$session\":0 even-horizontal\n\ntimestamp \"Attaching to tmux session: $session\"\ntmux attach-session -t \"$session\"\n"
  },
  {
    "path": "bin/ubuntu_release_version.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-04 18:03:29 +0700 (Tue, 04 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the latest Ubuntu full release version for a given major X.Y release eg. 22.04 => 22.04.6\n\nIf no major version is given, finds the latest major version and then the latest minor full version for that\n\nOutputs the URL to the latest minor version ISO\n\nUsed to update repo:\n\n    https://github.co/HariSekhon/Packer\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nversion=\"${1:-}\"\n\nbase_url=\"https://cdimage.ubuntu.com/releases\"\n\nif is_blank \"$version\"; then\n    timestamp \"No version specified\"\n    timestamp \"Finding latest major Ubuntu release by parsing: $base_url/\"\n    version=\"$(\n        curl -s \"$base_url/\" |\n            grep -oP 'href=\"\\d{2}\\.\\d{2}/\"' |\n            grep -oP '\\d{2}\\.\\d{2}' |\n            sort -V |\n            tail -n 1\n    )\"\n    timestamp \"Latest Ubuntu major version: $version\"\nfi\n\nrelease_url=\"$base_url/$version/release\"\n\ntimestamp \"Finding latest Ubuntu minor release version by parsing: $release_url/\"\nfull_version=\"$(\n    curl -s \"$release_url/\" |\n        grep -oP 'href=\".*?\\.iso\"' |\n        grep -oP '\\d+\\.\\d+(\\.\\d+)?' |\n        sort -V |\n        tail -n 1\n)\"\n\nif is_blank \"$full_version\"; then\n    die \"ERROR: Full version parsing failed for Ubuntu version: $version\"\nfi\n\ntimestamp \"Latest Ubuntu full version for $version:\"\necho >&2\necho \"$full_version\"\n"
  },
  {
    "path": "bin/uniq_chars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-30 12:38:43 +0000 (Thu, 30 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints a list of unique characters in a string\n\nUseful to check large strings like \\$PATH for characters that may be causing a regex match failure\n\nWorks like a standard unix filter program - string is read from a file argument or passed via stdin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file>]\"\n\nhelp_usage \"$@\"\n\n# need GNU switches\nif is_mac; then\n    od(){\n        command god \"$@\"\n    }\nfi\n\n# -c printable characters or backslash escapes\n# -v don't suppress duplicates with a *, we will post process them with sort\n# -w1 only output 1 char per line so we can post process with sort, can be no space between the -w and 1 otherwise it'll try to read the 1 as a file argument\nod -cv -A none -w1 \"$@\" |\nsort -bu\n"
  },
  {
    "path": "bin/url_extract_redirects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-11 01:56:41 +0700 (Sat, 11 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExtracts the URLs from a given string arg, file or standard input,\nqueries each one and outputs the redirected urls instead to stdout\n\nUses adjacent script:\n\n    urlextract.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n\"$srcdir/urlextract.sh\" \"$@\" |\nwhile read -r url; do\n    log \"Resolving final redirect for URL: $url\" >&2\n    # curl follow redirects and print only the final URL\n    # the -I header flag is a bit risky in case the server doesn't give redirects in the headers\n    # but we don't want to waste time downloading large ISO file URLs such as in Packer templates\n    # when trying to find outdated redirected URLs to replace\n    result=\"$(command curl -sSLfI -o /dev/null -w '%{http_code} %{url_effective}' \"$url\" || :)\"\n    http_code=\"$(awk '{print $1}' <<< \"$result\")\"\n    final_url=$(cut -d' ' -f2- <<< \"$result\")\n    if [ \"$http_code\" == 200 ]; then\n        echo \"$final_url\"\n    else\n        warn \"failed to resolve URL, outputting original only: $url\"\n        echo \"$url\"\n    fi\ndone\n"
  },
  {
    "path": "bin/url_replace_redirects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-11 01:56:41 +0700 (Sat, 11 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExtracts the URLs from a given string arg, file or standard input,\nqueries each one and outputs the entire contents to stdout with the urls replaced by the redirected urls\n\nUses adjacent scripts:\n\n    urlextract.sh\n\n    url_extract_redirects.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\narg=\"${1:-}\"\n\ncontents=\"$(\n    if [ $# -eq 0 ]; then\n        cat\n    elif [ -f \"$arg\" ]; then\n        cat \"$arg\"\n    else\n        echo \"$arg\"\n    fi\n)\"\n\nsed_script=\"\"\n\nurls_seen=\"\"\n\ntimestamp \"Resolving URLs in contents to figure out what to replace, this may take some time. To see progress resolving each URL, set environment variable VERBOSE=1\"\nwhile read -r line; do\n    url=\"$(\"$srcdir/urlextract.sh\" <<< \"$line\" 2>/dev/null || :)\"\n    if ! is_blank \"$url\"; then\n        if grep -Fxq \"$url\" <<< \"$urls_seen\"; then\n            log \"Skipping URL already resolved: $url\"\n            continue\n        fi\n        redirect_url=\"$(\"$srcdir/url_extract_redirects.sh\" <<< \"$url\")\"\n        if [ \"$redirect_url\" != \"$url\" ]; then\n            sed_script+=\"\n                s|$url|$redirect_url|g\n            \"\n        fi\n    fi\n    urls_seen+=\"\n$url\"\ndone <<< \"$contents\"\n\n# shellcheck disable=SC2001\nsed_script=\"$(sed \"s/^[[:space:]]*//\" <<< \"$sed_script\")\"\n\n# order sed script by longest lines first to avoid replacing substrings\n# it is safer to replace the longest matches first\nsed_script=\"$(awk '{ print length, $0 }' <<< \"$sed_script\" | sort -nr | cut -d' ' -f2-)\"\n\nsed \"$sed_script\" <<< \"$contents\"\n"
  },
  {
    "path": "bin/urldecode.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: one%20two\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-03 17:47:02 +0000 (Tue, 03 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick command line URL decoding\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nURL decodes the given string argument or standard input\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_to_urldecode>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    echo \"$@\"\nelse\n    cat\nfi |\nif type -P perl &>/dev/null &&\n   perl -MURI::ESCAPE -e '' &>/dev/null; then\n    perl -MURI::Escape -ne 'chomp; print uri_unescape($_) . \"\\n\"'\nelif type -p python3 &>/dev/null &&\n     python3 -c 'from urllib.parse import quote_plus'; then\n    python3 -c 'from urllib.parse import unquote_plus; import sys; print(unquote_plus(sys.stdin.read().rstrip(\"\\n\").rstrip(\"\\r\")))'\nelse\n    echo \"ERROR: neither Perl URI::Escape nor Python3 with UrlLib.Parse are available\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "bin/urlencode.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Sébastien Tellier\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-03 17:47:02 +0000 (Tue, 03 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nURL encodes the given string argument or standard input\n\nOnly escapes characters disallowed in URIs\n\nFigures out which of the relevant tools are available and uses the first one it finds in this order:\n\n- jq\n- Perl URI::ESCAPE\n- Python urllib.parse\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_to_urlencode>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    echo \"$@\"\nelse\n    cat\nfi |\nif type -P jq &>/dev/null; then\n    jq -rn --arg string \"$(cat)\" '$string|@uri'\nelif type -P perl &>/dev/null &&\n   perl -MURI::ESCAPE -e '' &>/dev/null; then\n    perl -MURI::Escape -ne 'chomp; print uri_escape($_) . \"\\n\"'\nelif type -p python3 &>/dev/null &&\n     python3 -c 'from urllib.parse import quote_plus'; then\n     python3 -c 'from urllib.parse import quote_plus; import sys; print(quote_plus(sys.stdin.read().rstrip(\"\\n\").rstrip(\"\\r\")))'\nelse\n    echo \"ERROR: neither Perl URI::Escape nor Python3 with UrlLib.Parse are available\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "bin/urlencode_utf.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Sébastien Tellier\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-03 17:47:02 +0000 (Tue, 03 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nURL encodes the given string argument or standard input\n\nEscapes all non-ASCII characters by converting them to UTF-8 and then encoding them\n\nWritten to solve an issue with the Spotify API\n\nFigures out which of the relevant tools are available and uses the first one it finds in this order:\n\n- jq\n- Perl URI::ESCAPE\n- Python urllib.parse\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_to_urlencode>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    echo \"$@\"\nelse\n    cat\nfi |\nif type -P jq &>/dev/null; then\n    jq -rn --arg string \"$(cat)\" '$string|@uri'\nelif type -P perl &>/dev/null &&\n   perl -MURI::ESCAPE -e '' &>/dev/null; then\n    perl -MURI::Escape -ne 'chomp; print uri_escape_utf8($_) . \"\\n\"'\nelif type -p python3 &>/dev/null &&\n     python3 -c 'from urllib.parse import quote_plus'; then\n     python3 -c '\nfrom urllib.parse import quote_plus\nimport sys\ns = sys.stdin.read().rstrip(\"\\n\").rstrip(\"\\r\")\nprint(quote_plus(s, safe=\"\", encoding=\"utf-8\"))\n'\nelse\n    echo \"ERROR: neither Perl URI::Escape nor Python3 with UrlLib.Parse are available\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "bin/urlextract.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-06 18:59:32 +0100 (Tue, 06 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExtracts the URLs from a given string arg, file or standard input\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\narg=\"${1:-}\"\n\nif [ $# -eq 0 ]; then\n    cat\nelif [ -f \"$arg\" ]; then\n    cat \"$arg\"\nelse\n    echo \"$arg\"\nfi |\n# [] break the regex match, even when escaped \\[\\]\ngrep -Eo 'https?://[[:alnum:]./?&!$#%@*;:+~_=-]+' ||\ndie \"No URLs found\"\n"
  },
  {
    "path": "bin/urlopen.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google.com\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-06 18:59:32 +0100 (Tue, 06 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens the URL given as an arg, or first URL from standard input or a given file\n\nUsed by .vimrc to instantly open a URL on the given line in the editor\n\nVery useful for quickly referencing inline documentation links found throughout my GitHub repos\n\nRespects \\$BROWER environment variable if set, otherwise tries to infer the mechanism on macOS or Linux\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nbrowse(){\n    # some likely catchall browsers on Linux\n    local browsers=(\n        xdg-open\n        sensible-browser\n        x-www-browser\n        gnome-open\n    )\n    local url=\"$1\"\n    if [ -n \"${BROWSER:-}\" ]; then\n        \"$BROWSER\" \"$url\"\n    elif is_mac; then\n        open \"$url\"\n    else  # assume Linux\n        for browser in \"${browsers[@]}\"; do\n            if type -P \"$browser\" &>/dev/null; then\n                \"$browser\" \"$url\" &\n                return 0\n            fi\n        done\n        die \"ERROR: none of the following browsers were found in the \\$PATH:\n\n$(for browser in ${BROWSER:+\"$BROWSER\"} \"${browsers[@]}\"; do echo \"$browser\"; done)\n\nCould not open the URL: $url\n\"\n    fi\n}\nexport -f browse\n\n\"$srcdir/urlextract.sh\" \"$@\" |\n# head -n1 because grep -m 1 can't be trusted and sometimes outputs more matches on subsequent lines\nhead -n1 |\nwhile read -r url; do\n    browse \"$url\"\ndone\n"
  },
  {
    "path": "bin/vault_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-08-12 01:07:03 +0100 (Mon, 12 Aug 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Usage:\n#\n#   source vault_pass.sh\n#\n# to load cred to environment for multiple ansible_playbook_vault runs without having to enter password each time\n#\n# See Also:\n#\n#   the pass() function in .bash.d/functions.sh\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ -n \"${PS1:-}\" ]; then\n    #read -s -r -p \"password: \" VAULT_PASS\n    prompt=\"Enter password: \"\n    password=\"\"\n    while IFS= read -p \"$prompt\" -r -s -n 1 char; do\n        if [[ \"$char\" == $'\\0' ]]; then\n            break\n        fi\n        prompt='*'\n        password=\"${password}${char}\"\n    done\n    echo\n    export VAULT_PASS=\"$password\"\n    echo \"exported \\$VAULT_PASS\"\n    echo \"ready to be called from ansible --vault-id\"\nelif [ -n \"${VAULT_PASS:-}\" ]; then\n    # retuns password from environment to ansible_playbook_vault\n    echo \"$VAULT_PASS\"\nelse\n    echo \"\\$VAULT_PASS not defined - did you forget to define it? Source $0 interactively to set it\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "bitbucket/bitbucket_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 19:45:46 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#  args: /user | jq .\n#  args: /workspaces | jq .\n#  args: /repositories/harisekhon | jq .\n#  args: /repositories/harisekhon/devops-bash-tools/pipelines/ | jq .\n#  args: /repositories/harisekhon/devops-bash-tools -X PUT -H 'Content-Type: application/json' -d '{\"description\": \"some words\"}'\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the BitBucket.org API (v2.0)\n\nCan specify \\$CURL_OPTS for options to pass to curl, or pass them as arguments to the script\n\nAutomatically handles authentication via environment variables \\$BITBUCKET_USERNAME / \\$BITBUCKER_USER and \\$BITBUCKET_TOKEN (which is used as the password, your real BitBucket password isn't accepted by the BitBucket 2.0 API)\nIf either of these are not found, tries to infer from local git repo's bitbucket remotes\n\nYou must set up a personal access token here:\n\nhttps://bitbucket.org/account/settings/app-passwords/\n\n\nAPI Reference:\n\nhttps://developer.atlassian.com/bitbucket/api/2/reference/\nhttps://developer.atlassian.com/bitbucket/api/2/reference/resource/\n\n\nXXX: NOTE: The API is inconsistent in places, some endpoints require a trailing slash otherwise they 404, others cannot have a trailing slash otherwise they 404 the other way\n\n\nExamples:\n\n\n# Get currently authenticated user's workspaces:\n\n    ${0##*/} /workspaces | jq .\n\n\n# Get repos in given workspace:\n\n\n    ${0##*/} /repositories/{workspace}\n\n    ${0##*/} /repositories/harisekhon | jq .\n\n\n# Get a repo's BitBucket Pipelines [oldest first] (must have a slash on the end otherwise gets 404 error):\n\n    ${0##*/} /repositories/{workspace}/{repo_slug}/pipelines/\n\n    ${0##*/} /repositories/harisekhon/devops-bash-tools/pipelines/ | jq .\n\n\n# List repo variables (requires trailing slash or gets 404):\n\n    ${0##*/} /repositories/{workspace}/{repo_slug}/pipelines_config/variables/\n\n    ${0##*/} /repositories/harisekhon/devops-bash-tools/pipelines_config/variables/ | jq .\n\n\n# Create a repo variable (see bitbucket_repo_set_env_vars.sh for a more convenient way of adding/updating env vars, optionally in bulk):\n\n    ${0##*/} /repositories/{workspace}/{repo_slug}/pipelines_config/variables/ -X POST -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n    ${0##*/} /repositories/harisekhon/devops-bash-tools/pipelines_config/variables/ -X POST -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n\n# Update a repo variable (must include braces around the variable uuid but url encode the braces, omitting key still works, providing key overwrites the key field):\n\n    ${0##*/} /repositories/{workspace}/{repo_slug}/pipelines_config/variables/%7B{variable_uuid}%7D -X PUT -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n    ${0##*/} /repositories/harisekhon/devops-bash-tools/pipelines_config/variables/%7B9dc735a8-a3d5-4432-9afc-fad58e368a93%7D -X PUT -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n\n# List workspace variables (notice 2 inconsistencies here: for repos it's pipelines_config with a trailing slash, for workspaces it's pipelines-config with no trailing slash - the slash can result in 404):\n\n    ${0##*/} /workspaces/{workspace}/pipelines-config/variables\n\n    ${0##*/} /workspaces/harisekhon/pipelines-config/variables | jq .\n\n\n# Create a workspace variable (see bitbucket_workspace_set_env_vars.sh for a more convenient way of adding/updating env vars, optionally in bulk):\n# notice the no trailing slash otherwise 404 compared to the repo variable which requires it otherwise gets 404\n# if a variable with this key already exists, will result in {\\\"error\\\": {\\\"message\\\": \\\"Conflict\\\", \\\"detail\\\": \\\"A variable with the key provided already exists for account...\n\n    ${0##*/} /workspaces/{workspace}/pipelines-config/variables -X POST -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n    ${0##*/} /workspaces/harisekhon/pipelines-config/variables -X POST -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n\n# Update a workspace variable (must include braces around the variable uuid but url encode the braces, omitting key still works, providing key overwrites the key field):\n\n    ${0##*/} /workspaces/{workspace}/pipelines_config/variables/%7B{variable_uuid}%7D -X PUT -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n    ${0##*/} /workspaces/harisekhon/pipelines-config/variables/%7Bfc70af25-ec2e-46fd-96bc-d7c2bed3cb4b%7D -X PUT -d '{\\\"key\\\": \\\"mykey\\\", \\\"value\\\": \\\"myvalue\\\", \\\"secured\\\": true }'\n\n\n\n# Update a repo's description:\n\n    ${0##*/} /repositories/{workspace}/{repo_slug} -X PUT -d '{\\\"description\\\": \\\"some words\\\"}'\n\n    ${0##*/} /repositories/harisekhon/devops-bash-tools -X PUT -d '{\\\"description\\\": \\\"some words\\\"}' | jq .\n\n\n# Get currently authenticated user (unfortunately this is less useful than with GitHub / GitLab APIs since you can't use a standard OAuth2 authentication with just the bearer token, and must specify a username to authenticate to the API in the first place):\n\n    ${0##*/} /user | jq .\n\n\nFor convenience you can even copy and paste out of the documentation literally and have the script auto-determine the right settings.\n\nPlaceholders are replaced if the following information is available in environment variables or can be inferred from the local git repo remote urls. The tokens for replacement can be given in the form {token}, <token>, :token\n\n\\$BITBUCKET_USERNAME / \\$BITBUCKET_USER:             owner, username, user\nthe local repo name of the current directory:      repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.bitbucket.org/2.0\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nuser=\"${BITBUCKET_USERNAME:-${BITBUCKET_USER:-}}\"\n# for old App Passwords\nPASSWORD=\"${BITBUCKET_PASSWORD:-${BITBUCKET_TOKEN:-}}\"\n# for API Tokens, but doesn't work yet, gives 401\n#export TOKEN=\"${BITBUCKET_PASSWORD:-${BITBUCKET_TOKEN:-}}\"\n\nif [ -z \"$user\" ]; then\n    echo \"WARNING: \\$BITBUCKET_USERNAME / \\$BITBUCKET_USER not specified, attempting to determine from local remote url\" >&2\n    user=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@bitbucket\\.org/{print $2; exit}' | sed 's|https://||;s/@.*//;s/:.*//' || :)\"\n    # curl_auth.sh does this automatically\n    #if [ -z \"$user\" ]; then\n    #    user=\"${USERNAME:${USER:-}}\"\n    #fi\nfi\n\nif [ -z \"${PASSWORD:-}${TOKEN:-}\" ]; then\n    # if this is still blank then curl_auth.sh will detect and prompt for a password (non echo'd)\n    PASSWORD=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@bitbucket\\.org/{print $2; exit}' | sed 's|https://||;s/@.*//;s/.*://' || :)\"\nfi\nexport PASSWORD\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nelse\n    echo \"WARNING: \\$BITBUCKET_USERNAME / \\$BITBUCKET_USER not specified, and failed to determine from local remote url, will end up using your environment's default \\$USERNAME / \\$USER which may not be the right BitBucket username and can lead to authentication failures - recommend you set \\$BITBUCKET_USERNAME explicitly\" >&2\nfi\n\nurl_path=\"$1\"\nurl_path=\"${url_path##*:\\/\\/bitbucket.org\\/}\"\nurl_path=\"${url_path#2.0}\"\nurl_path=\"${url_path##/}\"\n\n# to support other API versions\n#if [[ \"$url_path\" =~ ^[[:digit:]] ]]; then\n#    url_base=\"${url_base%/2.0}\"\n#fi\n\nshift || :\n\nrepo=$(git_repo 2>/dev/null | sed 's/.*\\///' || :)\n\nif [ -n \"$user\" ]; then\n    url_path=\"${url_path/\\{username\\}/$user}\"\n    url_path=\"${url_path/<username>/$user}\"\n    url_path=\"${url_path/:username/$user}\"\n    url_path=\"${url_path/\\{user\\}/$user}\"\n    url_path=\"${url_path/<user>/$user}\"\n    url_path=\"${url_path/:user/$user}\"\nfi\nurl_path=\"${url_path/\\{repo\\}/$repo}\"\nurl_path=\"${url_path/<repo>/$repo}\"\nurl_path=\"${url_path/:repo/$repo}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "bitbucket/bitbucket_disable_pipelines.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-03 00:14:52 +0100 (Sat, 03 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/bitbucket.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables all Bitbucket CI/CD pipelines via the BitBucket API\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n\"$srcdir/bitbucket_foreach_repo.sh\" \"\n    '$srcdir/bitbucket_repo_disable_pipeline.sh' '{repo}'\n\"\n"
  },
  {
    "path": "bitbucket/bitbucket_enable_pipelines.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-03 00:14:52 +0100 (Sat, 03 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/bitbucket.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables all Bitbucket CI/CD pipelines via the BitBucket API\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n\"$srcdir/bitbucket_foreach_repo.sh\" \"\n    '$srcdir/bitbucket_repo_enable_pipeline.sh' '{repo}'\n\"\n"
  },
  {
    "path": "bitbucket/bitbucket_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo workspace={workspace} name={name} repo={repo}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-30 10:08:07 +0100 (Sun, 30 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each BitBucket repo\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the workspace/user/repo names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{workspace}   =>   the workspace the repo is in\n{name}        =>   the repo name without the workspace prefix\n{repo}        =>   the repo name with the workspace prefix\n\neg.\n    ${0##*/} echo user={user} name={name} repo={repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nget_repos(){\n    get_workspaces |\n    while read -r workspace; do\n        get_workspace_repos \"$workspace\"\n    done\n}\n\nget_workspaces(){\n    local page=1\n    while true; do\n        if ! output=\"$(\"$srcdir/bitbucket_api.sh\" \"/workspaces?page=$page&pagelen=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.values[]' <<< \"$output\")\" ]; then\n            break\n        fi\n        jq -r '.values[].slug' <<< \"$output\"\n        ((page+=1))\n    done\n}\n\nget_workspace_repos(){\n    local workspace=\"$1\"\n    local page=1\n    while true; do\n        if ! output=\"$(\"$srcdir/bitbucket_api.sh\" \"/repositories/$workspace?page=$page&pagelen=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.values[]' <<< \"$output\")\" ]; then\n            break\n        fi\n        jq -r '.values[] | [.workspace.slug, .name, .full_name] | @tsv' <<< \"$output\"\n        ((page+=1))\n    done\n}\n\nget_repos |\nwhile read -r workspace name repo; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $repo\" >&2\n    echo \"# ============================================================================ #\" >&2\n    echo >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{workspace\\}/$workspace}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$name}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "bitbucket/bitbucket_repo_disable_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: pylib\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-03 00:02:53 +0100 (Sat, 03 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/bitbucket.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisable the CI/CD pipeline for a given BitBucket.org repo via the BitBucket API\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nRepo should be specified as <workspace>/<repo> using the slugs which are usually just the same as the username and repo lowercase (they're case sensitive, eg. harisekhon/devops-bash-tools)\n\nRepo workspace prefix, if not set, is assumed to be the username and will use \\$BITBUCKET_USER if available, otherwise will query the BitBucket API to determine it\n\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nrepo=\"$1\"\n\nif ! [[ \"$repo\" =~ / ]]; then\n    log \"No username prefix in repo '$repo', will auto-add it\"\n    # refuse bitbucket_user between function calls for efficiency to save additional queries to the BitBucket API\n    if [ -z \"${bitbucket_user:-}\" ]; then\n        bitbucket_user=\"$(get_bitbucket_user)\"\n    fi\n    repo=\"$bitbucket_user/$repo\"\nfi\n\ntimestamp \"Disabling pipeline for BitBucket repo '$repo'\"\n\n\"$srcdir/bitbucket_api.sh\" \"/repositories/$repo/pipelines_config\" -X PUT -H 'Content-Type: application/json' --data '{ \"enabled\": false }' >/dev/null\n"
  },
  {
    "path": "bitbucket/bitbucket_repo_enable_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: pylib\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-03 00:02:53 +0100 (Sat, 03 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/bitbucket.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables the CI/CD pipeline for a given BitBucket.org repo via the BitBucket API\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nRepo should be specified as <workspace>/<repo> using the slugs which are usually just the same as the username and repo lowercase (they're case sensitive, eg. harisekhon/devops-bash-tools)\n\nRepo workspace prefix, if not set, is assumed to be the username and will use \\$BITBUCKET_USER if available, otherwise will query the BitBucket API to determine it\n\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nrepo=\"$1\"\n\nif ! [[ \"$repo\" =~ / ]]; then\n    log \"No username prefix in repo '$repo', will auto-add it\"\n    # refuse bitbucket_user between function calls for efficiency to save additional queries to the BitBucket API\n    if [ -z \"${bitbucket_user:-}\" ]; then\n        bitbucket_user=\"$(get_bitbucket_user)\"\n    fi\n    repo=\"$bitbucket_user/$repo\"\nfi\n\ntimestamp \"Enabling pipeline for BitBucket repo '$repo'\"\n\n\"$srcdir/bitbucket_api.sh\" \"/repositories/$repo/pipelines_config\" -X PUT -H 'Content-Type: application/json' --data '{ \"enabled\": true }' >/dev/null\n"
  },
  {
    "path": "bitbucket/bitbucket_repo_set_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: github-actions-contexts test\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/bitbucket.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the description of a BitBucket.org repo via the BitBucket API\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nRepo should be specified as <workspace>/<repo> using the slugs which are usually just the same as the username and repo lowercase (they're case sensitive, eg. harisekhon/devops-bash-tools)\n\nRepo workspace prefix, if not set, is assumed to be the username and will use \\$BITBUCKET_USER if available, otherwise will query the BitBucket API to determine it\n\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\n\nExample:\n\n${0##*/} harisekhon/devops-bash-tools    my new description\n\n\nIf no args are given, will read read and description from standard input for easy chaining with other tools, can easily update multiple repositories this way, one repo + description per line:\n\necho <repo> <description> | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> <description>\"\n\nhelp_usage \"$@\"\n\n#min_args 2 \"$@\"\n\nset_repo_description(){\n    local repo=\"$1\"\n    local description=\"${*:2}\"\n\n    if ! [[ \"$repo\" =~ / ]]; then\n        log \"No username prefix in repo '$repo', will auto-add it\"\n        # reuse bitbucket_user between function calls for efficiency to save additional queries to the BitBucket API\n        if [ -z \"${bitbucket_user:-}\" ]; then\n            bitbucket_user=\"$(get_bitbucket_user)\"\n        fi\n        repo=\"$bitbucket_user/$repo\"\n    fi\n\n    timestamp \"Setting BitBucket repo '$repo' description to '$description'\"\n\n    #description=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$description\")\"\n\n    \"$srcdir/bitbucket_api.sh\" \"/repositories/$repo\" -X PUT -H 'Content-Type: application/json' --data \"{ \\\"description\\\": \\\"$description\\\" }\" >/dev/null\n}\n\nif [ $# -gt 0 ]; then\n    if [ $# -lt 2 ]; then\n        usage\n    fi\n    set_repo_description \"$@\"\nelse\n    while read -r repo description; do\n        [ -n \"$repo\" ] || continue\n        [ -n \"$description\" ] || continue\n        set_repo_description \"$repo\" \"$description\"\n    done\nfi\n"
  },
  {
    "path": "bitbucket/bitbucket_repo_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.bitbucket.com/ee/api/pipeline_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates Bitbucket Pipelines repo-level secured environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nWorkspace is case insensitive\nRepo slug is case sensitive and must be in lowercase\nVariable keys are case-sensitive - a change in case will create a new one\n\nExamples:\n\n    ${0##*/} HariSekhon/devops-bash-tools AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} HariSekhon/devops-bash-tools\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} HariSekhon/devops-bash-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<workspace>/<repo_slug> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nworkspace_repo_slug=\"$1\"\nshift || :\n\nexisting_env_vars=\"$(\"$srcdir/bitbucket_api.sh\" \"/repositories/$workspace_repo_slug/pipelines_config/variables/\" | jq -r '.values[] | [.key, .uuid] | @tsv')\"\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    if grep -q \"^${key}[[:space:]]\" <<< \"$existing_env_vars\"; then\n        local variable_uuid\n        variable_uuid=\"$(awk \"/^${key}[[:space:]]/{print \\$2}\" <<< \"$existing_env_vars\" | sed 's/{//;s/}//')\"\n        timestamp \"updating Bitbucket environment variable '$key' in repo '$workspace_repo_slug'\"\n        \"$srcdir/bitbucket_api.sh\" \"/repositories/$workspace_repo_slug/pipelines_config/variables/%7B${variable_uuid}%7D\" -X PUT -d \"{\\\"key\\\": \\\"$key\\\", \\\"value\\\": \\\"$value\\\", \\\"secured\\\": true}\"\n    else\n        timestamp \"adding Bitbucket environment variable '$key' to repo '$workspace_repo_slug'\"\n        \"$srcdir/bitbucket_api.sh\" \"/repositories/$workspace_repo_slug/pipelines_config/variables/\" -X POST -d \"{\\\"key\\\": \\\"$key\\\", \\\"value\\\": \\\"$value\\\", \\\"secured\\\": true}\"\n    fi\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "bitbucket/bitbucket_ssh_add_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.atlassian.com/bitbucket/api/2/reference/resource/users/%7Bselected_user%7D/ssh-keys#post\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds the given SSH public key(s) to the currently authenticated BitBucket.org account via the BitBucket API\n\nIf no SSH public key is given, defaults to using ~/.ssh/id_rsa.pub\n\nIf a dash is given, reads the SSH public key(s) from standard input, ignoring comment lines so you can chain with tools like the adjacent scripts:\n\n    github_ssh_get_user_public_keys.sh\n    gitlab_ssh_get_user_public_keys.sh\n    github_ssh_get_public_keys.sh\n    gitlab_ssh_get_public_keys.sh\n    bitbucket_ssh_get_public_keys.sh\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\nWill return a 400 error if the SSH public key is invalid or has already been added.\nThe script detects already existing keys and skips them to avoid this error\n\nSSH Keys can be found in the Web UI here:\n\n    https://bitbucket.org/account/settings/ssh-keys/\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ssh_public_key_file> [<ssh_public_key_file2>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# this doc is wrong, doesn't accept either account id or uuid, but does accept username\n#\n#   https://developer.atlassian.com/bitbucket/api/2/reference/resource/users/%7Bselected_user%7D/ssh-keys\n#\n#account_id=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.account_id')\"\n#user_uuid=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.uuid')\"\nif [ -n \"${BITBUCKET_USER:-}\" ]; then\n    user=\"$BITBUCKET_USER\"\nelse\n    echo \"# Getting BitBucket user name via API call\"\n    user=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.username')\"\nfi\n\necho \"# Getting existing SSH public keys to skip any keys that already exist (to avoid 400 errors)\" >&2\nssh_public_keys=\"$(\"$srcdir/bitbucket_ssh_get_public_keys.sh\")\"\n\nadd_ssh_public_keys_from_file(){\n    local public_key_file=\"$1\"\n    if [ \"$public_key_file\" = \"-\" ]; then\n        public_key_file=/dev/stdin\n    fi\n    sed 's/#.*//; /^[[:space:]]*$/d' \"$public_key_file\" |\n    while read -r public_key; do\n        [[ \"$public_key\" =~ ^ssh- ]] || die \"invalid SSH key in file '$public_key_file': $public_key\"\n        add_ssh_public_key \"$public_key\"\n    done\n}\n\nadd_ssh_public_key(){\n    local public_key=\"$1\"\n    key=\"$(awk '{print $1\" \"$2}' <<< \"$public_key\")\"\n    if grep -Fq \"$key\" <<< \"$ssh_public_keys\"; then\n        timestamp \"SSH public key already exists, skipping: '$public_key'\"\n        return\n    fi\n    timestamp \"adding SSH public key to currently authenticated BitBucket account: '$public_key'\"\n    # comment field will be populated from the key comment suffix, label is for the UI (which defaults to the comment otherwise but leaves the label field blank)\n    # however, if I load someone else's public key or pipe from another download script, we certainly don't want it misleadingly marking the key as belonging to this machine, so more accurate to just rely on the SSH key comment\n    #\"$srcdir/bitbucket_api.sh\" \"/users/$user/ssh-keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\", \"label\": \"'\"$USER@${HOSTNAME:-$(hostname)}\"'\"}' > /dev/null  # JSON of the newly added key\n    \"$srcdir/bitbucket_api.sh\" \"/users/$user/ssh-keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\"}' |\n    # > /dev/null  # JSON of the newly added key\n    jq -r '[ \"SSH public key added to account\", .owner.display_name, \"-\", .comment ] | @tsv' |\n    tr '\\t' ' ' |\n    timestamp \"$(cat)\"\n    echo >&2\n}\n\nif [ $# -gt 0 ]; then\n    for filename; do\n        add_ssh_public_keys_from_file \"$filename\"\n    done\nelse\n    add_ssh_public_keys_from_file ~/.ssh/id_rsa.pub\nfi\n"
  },
  {
    "path": "bitbucket/bitbucket_ssh_delete_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.atlassian.com/bitbucket/api/2/reference/resource/users/%7Bselected_user%7D/ssh-keys#post\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds the given SSH public key(s) to the currently authenticated BitBucket.org account via the BitBucket API\n\nIf no SSH public key is given, defaults to using ~/.ssh/id_rsa.pub\n\nIf a dash is given, reads the SSH public key(s) from standard input, ignoring comment lines so you can chain with tools like the adjacent scripts:\n\n    github_ssh_get_user_public_keys.sh\n    gitlab_ssh_get_user_public_keys.sh\n    github_ssh_get_public_keys.sh\n    gitlab_ssh_get_public_keys.sh\n    bitbucket_ssh_get_public_keys.sh\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\nWill return a 400 error if the SSH public key is invalid or has already been added.\nThe script detects already existing keys and skips them to avoid this error\n\nSSH Keys can be found in the Web UI here:\n\n    https://bitbucket.org/account/settings/ssh-keys/\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ssh_public_key_file> [<ssh_public_key_file2>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# this doc is wrong, doesn't accept either account id or uuid, but does accept username\n#\n#   https://developer.atlassian.com/bitbucket/api/2/reference/resource/users/%7Bselected_user%7D/ssh-keys\n#\n#account_id=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.account_id')\"\n#user_uuid=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.uuid')\"\nif [ -n \"${BITBUCKET_USER:-}\" ]; then\n    user=\"$BITBUCKET_USER\"\nelse\n    echo \"# Getting BitBucket user name via API call\"\n    user=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.username')\"\nfi\n\necho \"# Getting existing SSH public keys to skip any keys that already exist (to avoid 400 errors)\" >&2\nssh_public_keys=\"$(\"$srcdir/bitbucket_ssh_get_public_keys.sh\")\"\n\nadd_ssh_public_keys_from_file(){\n    local public_key_file=\"$1\"\n    if [ \"$public_key_file\" = \"-\" ]; then\n        public_key_file=/dev/stdin\n    fi\n    sed 's/#.*//; /^[[:space:]]*$/d' \"$public_key_file\" |\n    while read -r public_key; do\n        [[ \"$public_key\" =~ ^ssh- ]] || die \"invalid SSH key in file '$public_key_file': $public_key\"\n        add_ssh_public_key \"$public_key\"\n    done\n}\n\nadd_ssh_public_key(){\n    local public_key=\"$1\"\n    key=\"$(awk '{print $1\" \"$2}' <<< \"$public_key\")\"\n    if grep -Fq \"$key\" <<< \"$ssh_public_keys\"; then\n        timestamp \"SSH public key already exists, skipping: '$public_key'\"\n        return\n    fi\n    timestamp \"adding SSH public key to currently authenticated BitBucket account: '$public_key'\"\n    # comment field will be populated from the key comment suffix, label is for the UI (which defaults to the comment otherwise but leaves the label field blank)\n    # however, if I load someone else's public key or pipe from another download script, we certainly don't want it misleadingly marking the key as belonging to this machine, so more accurate to just rely on the SSH key comment\n    #\"$srcdir/bitbucket_api.sh\" \"/users/$user/ssh-keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\", \"label\": \"'\"$USER@${HOSTNAME:-$(hostname)}\"'\"}' > /dev/null  # JSON of the newly added key\n    \"$srcdir/bitbucket_api.sh\" \"/users/$user/ssh-keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\"}' |\n    # > /dev/null  # JSON of the newly added key\n    jq -r '[ \"SSH public key added to account\", .owner.display_name, \"-\", .comment ] | @tsv' |\n    tr '\\t' ' ' |\n    timestamp \"$(cat)\"\n    echo >&2\n}\n\nif [ $# -gt 0 ]; then\n    for filename; do\n        add_ssh_public_keys_from_file \"$filename\"\n    done\nelse\n    add_ssh_public_keys_from_file ~/.ssh/id_rsa.pub\nfi\n"
  },
  {
    "path": "bitbucket/bitbucket_ssh_get_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 18:53:34 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets the currently authenticated user's SSH key(s) for the BitBucket.org account via the BitBucket API\n\nIf you get an error it's possible you don't have the right token permissions.\nYou can generate a new token with the right permissions here:\n\n    https://bitbucket.org/account/settings/app-passwords/\n\nSSH Keys can be found in the Web UI here:\n\n    https://bitbucket.org/account/settings/ssh-keys/\n\n\nIf \\$BITBUCKET_USER is not set, then first queries the BitBucket API to determine this first\n\nUses the adjacent script bitbucket_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# this doc is wrong, doesn't accept either account id or uuid, but does accept username\n#\n#   https://developer.atlassian.com/bitbucket/api/2/reference/resource/users/%7Bselected_user%7D/ssh-keys\n#\n#account_id=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.account_id')\"\n#user_uuid=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.uuid')\"\nif [ -n \"${BITBUCKET_USER:-}\" ]; then\n    user=\"$BITBUCKET_USER\"\nelse\n    echo \"# Getting BitBucket user name via API call\"\n    user=\"$(\"$srcdir/bitbucket_api.sh\" \"/user\" | jq -r '.username')\"\nfi\n\n# XXX: not handling paging because if you have > 100 SSH keys I want to know what is going on first!\n\necho \"# Fetching SSH Public Key(s) from BitBucket for account:  $user\" >&2\necho \"#\" >&2\n\"$srcdir/bitbucket_api.sh\" \"/users/$user/ssh-keys\" |\njq -r '.values[] | [ .key, .comment, .label ] | @tsv' |\ntr '\\t' ' ' |\nsed \"s|$| (bitbucket.org/$user)|\"\n"
  },
  {
    "path": "bitbucket/bitbucket_workspace_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.bitbucket.com/ee/api/pipeline_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates Bitbucket Pipelines workspace-level secured environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nWorkspace is case-insensitive\nVariable keys are case-sensitive - a change in case will create a new one\n\nExamples:\n\n    ${0##*/} HariSekhon AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} HariSekhon\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} HariSekhon\n\n\nXXX: WARNING: variables at this level can be accessed by all users with 'write' permission for any repository (private or public) that belongs to this workspace\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<workspace> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nworkspace=\"$1\"\nshift || :\n\nexisting_env_vars=\"$(\"$srcdir/bitbucket_api.sh\" \"/workspaces/$workspace/pipelines-config/variables\" | jq -r '.values[] | [.key, .uuid] | @tsv')\"\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    if grep -q \"^${key}[[:space:]]\" <<< \"$existing_env_vars\"; then\n        local variable_uuid\n        variable_uuid=\"$(awk \"/^${key}[[:space:]]/{print \\$2}\" <<< \"$existing_env_vars\" | sed 's/{//;s/}//')\"\n        timestamp \"updating Bitbucket environment variable '$key' in workspace '$workspace'\"\n        \"$srcdir/bitbucket_api.sh\" \"/workspaces/$workspace/pipelines-config/variables/%7B${variable_uuid}%7D\" -X PUT -d \"{\\\"key\\\": \\\"$key\\\", \\\"value\\\": \\\"$value\\\", \\\"secured\\\": true}\"\n    else\n        timestamp \"adding Bitbucket environment variable '$key' to workspace '$workspace'\"\n        \"$srcdir/bitbucket_api.sh\" \"/workspaces/$workspace/pipelines-config/variables\" -X POST -d \"{\\\"key\\\": \\\"$key\\\", \\\"value\\\": \\\"$value\\\", \\\"secured\\\": true}\"\n    fi\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "bitbucket-pipelines.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-24 17:08:57 +0000 (Mon, 24 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                B i t b u c k e t   C I / C D   P i p e l i n e\n# ============================================================================ #\n\n# Reference:\n#\n#   https://support.atlassian.com/bitbucket-cloud/docs/configure-bitbucket-pipelinesyml/\n\n# Languages:\n#\n#   https://confluence.atlassian.com/x/5Q4SMw\n\n# You can specify a custom docker image from Docker Hub as your build environment.\nimage: atlassian/default-image:2\n\npipelines:\n  default:\n    - step:\n        script:\n          - setup/ci_bootstrap.sh\n          - make init\n          - make ci\n          - make test\n"
  },
  {
    "path": "buddy.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-16 14:02:53 +0000 (Mon, 16 Mar 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                B u d d y   C I\n# ============================================================================ #\n\n# https://buddy.works/docs/yaml/yaml-schema\n\n---\n- pipeline: \"Build\"\n  trigger_mode: \"ON_EVERY_PUSH\"\n  ref_name: \"master\"\n  ref_type: \"BRANCH\"\n  target_site_url: \"https://github.com/HariSekhon/DevOps-Bash-tools\"\n  trigger_condition: \"ALWAYS\"\n  actions:\n    - action: \"Execute: make ci test\"\n      type: \"BUILD\"\n      working_directory: \"/buddy/devops-bash-tools\"\n      docker_image_name: \"library/ubuntu\"\n      docker_image_tag: \"18.04\"\n      #setup_commands:\n      # this step gets cached, which results in\n      # E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?\n      #  - apt update\n      #  - apt install -qy git make\n      execute_commands:\n        - setup/ci_bootstrap.sh\n        - make init\n        - make ci\n        - make test\n      volume_mappings:\n        - \"/:/buddy/devops-bash-tools\"\n      shell: \"BASH\"\n      trigger_condition: \"ALWAYS\"\n"
  },
  {
    "path": "buildkite/buildkite_agent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Runs BuildKite Agent\n#\n# see /usr/local/etc/buildkite-agent/buildkite-agent.cfg for config on Mac\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nStarts BuildKite Agent, either locally installed agent, or falls back to Docker agent\n\nRequires \\$BUILDKITE_AGENT_TOKEN, which can be obtained from here:\n\n    https://buildkite.com/organizations/<your_user_or_organization>/agents\n\nEnvironment variables:\n\nBUILDKITE_DOCKER        set to any value to use Docker agent regardless\nBUILDKITE_DOCKER_TAG    set to BuildKite docker agent tag, eg. centos, ubuntu, alpine (default agent latest tag is alpine)\n\"\n\nhelp_usage \"$@\"\n\n# ============================================================================ #\n#                        M a c  /  L i n u x   A g e n t\n# ============================================================================ #\n\nbuildkite_tags=\"os=linux\"\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PATH=\"$PATH:$HOME/.buildkite-agent/bin\"\n\nif type -P buildkite-agent &>/dev/null; then\n    if [ -z \"${BUILDKITE_DOCKER:-}\" ]; then\n        uname_s=\"$(uname -s)\"\n        if [ \"$uname_s\" = Darwin ]; then\n            buildkite_tags=\"os=mac\"\n        elif [ \"$uname_s\" = Linux ]; then\n            buildkite_tags=\"os=linux\"\n            distro=\"$(awk -F= '/^ID=/{print $2}' /etc/*release)\"\n            distro=\"${distro//\\\"/}\"\n            version=\"$(awk -F= '/^VERSION_ID=/{print $2}' /etc/*release | grep -Eom 1 '[[:digit:]]+' | head -n 1)\"\n            buildkite_tags+=\",distro=$distro,version=$version\"\n        else\n            buildkite_tags=\"os=unknown\"\n        fi\n        exec buildkite-agent start --tags \"$buildkite_tags\"\n    fi\nfi\n\n# falls through to Docker agent\n\n# ============================================================================ #\n#                                  D o c k e r\n# ============================================================================ #\n\n# only necessary for Docker, installed agent will have config file containing this token from install/install_buildkite.sh\ncheck_env_defined BUILDKITE_AGENT_TOKEN\n\n# latest, alpine, centos, ubuntu, stable, stable-latest, stable-ubuntu etc - see dockerhub_show_tags.py from adjacent DevOps Python tools repo\ndocker_tag=\"${BUILDKITE_DOCKER_TAG:-latest}\"\nif [ -n \"${BIG:-}\" ] && [ -z \"${BUILDKITE_DOCKER_TAG:-}\" ]; then\n    docker_tag=\"ubuntu\"\nfi\n\nif [ \"$docker_tag\" = \"latest\" ]; then\n    buildkite_tags+=\",distro=alpine\"\nelif [[ \"$docker_tag\" =~ alpine|centos|ubuntu ]]; then\n    buildkite_tags+=\",distro=${docker_tag#stable-}\"\nfi\n\nopts=\"\"\n# for debugging so we can docker exec in to machine and build from cwd\nif [ -n \"${DEBUG:-}\" ]; then\n    opts=\"-v $PWD:/pwd\"\nfi\n\n# want splitting\n# shellcheck disable=SC2086\nexec docker run --rm --name=\"buildkite-agent-$$\" \\\n                $opts \\\n                -e BUILDKITE_AGENT_TOKEN=\"$BUILDKITE_AGENT_TOKEN\" \\\n                buildkite/agent:\"$docker_tag\" start \\\n                    --name=\"${BUILDKITE_AGENT_NAME:-${HOSTNAME:-$(hostname)-$$}}\" \\\n                    --tags \"$buildkite_tags\" \"$@\"\n"
  },
  {
    "path": "buildkite/buildkite_agents.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:33:42 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://buildkite.com/docs/apis/rest-api/agents\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nLists BuildKite Agents via the BuildKite API\n\nOutput format:\n\n<agent_name>    <hostname>    <ip_address>    <started_date>    <user_agent>    <tags>\n\neg.\n\nmyhost.local-1  myhost.local    <ip_x.x.x.x>  2020-09-07T17:07:10.578Z    buildkite-agent/3.20.0.3264 (darwin; amd64) os=mac\nfb3e859b5cb8-1  fb3e859b5cb8    <ip_x.x.x.x>  2020-09-07T17:05:18.562Z    buildkite-agent/3.23.0.x (linux; amd64) os=linux,distro=alpine\narbitrary_name  b03797c47520    <ip_x.x.x.x>  2020-09-07T17:05:38.062Z    buildkite-agent/3.23.0.x (linux; amd64) os=linux,distro=centos\n\nThe 2nd and 3rd agents are running in Docker hence the hostname field is nonsensical, but if you use the adjacent buildkite_agent.sh it'll set\nthe agent name to the same as the hostname for intuitive results and auto-add some tags for os and linux distro\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" \"organizations/{organization}/agents\" \"$@\" |\njq -r '.[] | [.name, .hostname, .ip_address, .created_at, .user_agent, (.meta_data | join(\",\")) ] | @tsv'\n"
  },
  {
    "path": "buildkite/buildkite_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: user | jq\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:17:59 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nQueries BuildKite API, auto-populating \\$BUILDKITE_TOKEN from environment and API url base for convenience\n\nhttps://buildkite.com/docs/apis/rest-api\n\nExamples:\n\n    ${0##*/} /user | jq .\n\n    ${0##*/} /organizations | jq .\n\n    ${0##*/} /organizations/{organization}/pipelines | jq .\n\n    ${0##*/} /organizations/{organization}/pipelines/{pipeline} | jq .\n\n    ${0##*/} /builds | jq .\n\n    ${0##*/} /organizations/{organization}/builds | jq .\n\n    ${0##*/} /organizations/{organization}/pipelines/{pipeline}/builds/<num> | jq .\n\n    ${0##*/} /organizations/{organization}/agents | jq .\n\n    ${0##*/} /organizations/{organization}/emojis | jq .\n\n\nReplacements:\n\n{organization}    \\$BUILDKITE_ORGANIZATION, \\$BUILDKITE_USER, or queries /organizations (if more than one org is found you must set \\$BUILDKITE_ORGANIZATION to avoid ambiguity)\n{pipeline}        \\$BUILDKITE_PIPELINE\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nif [ -z \"${BUILDKITE_TOKEN:-}\" ]; then\n    usage \"BUILDKITE_TOKEN environment variable is not set (generate this from the Web UI -> Personal Settings -> API Access Tokens (https://buildkite.com/user/api-access-tokens)\"\nfi\n\nif [ $# -lt 1 ]; then\n    usage \"no /path given to query in the API\"\nfi\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl_base=\"https://api.buildkite.com/v2\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path#\"$url_base\"}\"\nurl_path=\"${url_path##/}\"\n\nexport TOKEN=\"$BUILDKITE_TOKEN\"\n\n# remember to set this eg. BUILDKITE_ORGANIZATION=\"{organization}\"\nBUILDKITE_ORGANIZATION=\"${BUILDKITE_ORGANIZATION:-${BUILDKITE_USER:-}}\"\n\nif [[ \"$url_path\" =~ {organization} ]]; then\n    organizations=\"$(\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"$url_base/organizations\" | jq -r '.[].slug')\"\n    num_organizations=\"$(wc -w <<< \"$organizations\" | sed 's/[[:space:]]//g')\"\n    if [ \"$num_organizations\" -eq 0 ]; then\n        usage \"\\$BUILDKITE_ORGANIZATION / \\$BUILDKITE_USER not set and could not find any organizations for the \\$BUILDKITE_TOKEN\"\n    elif [ \"$num_organizations\" -eq 1 ]; then\n        BUILDKITE_ORGANIZATION=\"$organizations\"\n        url_path=\"${url_path//\\{organization\\}/$BUILDKITE_ORGANIZATION}\"\n    elif [ \"$num_organizations\" -gt 1 ]; then\n        usage \"\\$BUILDKITE_ORGANIZATION / \\$BUILDKITE_USER not set and found more than 1 organization registered to the \\$BUILDKITE_TOKEN - must specify \\$BUILDKITE_ORGANIZATION to avoid ambiguity\"\n    fi\nfi\n\nif [[ \"$url_path\" =~ {pipeline} ]]; then\n    if [ -z \"$BUILDKITE_PIPELINE\" ]; then\n        usage \"\\$BUILDKITE_PIPELINE is not set, cannot do replacement of {pipeline} given in the url path\"\n    fi\n    url_path=\"${url_path//\\{pipeline\\}/$BUILDKITE_PIPELINE}\"\nfi\n\n\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"$url_base/$url_path\" \"$@\"\n"
  },
  {
    "path": "buildkite/buildkite_cancel_running_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-08 10:53:10 +0100 (Tue, 08 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nCancels BuildKite running builds via its API (to clear them and restart new later eg. after agent / environment change / fix)\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" 'builds?state=running' \"$@\" |\njq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\nwhile read -r name number url; do\n    url=\"${url#https://api.buildkite.com/v2/}\"\n    echo -n \"Cancelling $name build number $number:  \"\n    \"$srcdir/buildkite_api.sh\" \"$url/cancel\" -X PUT |\n    jq -r '.state'\ndone\n"
  },
  {
    "path": "buildkite/buildkite_cancel_scheduled_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:33:42 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nCancels BuildKite scheduled builds via its API (to clear a backlog due to offline agents and just focus on new builds)\n\nYou might have to run this more than once if there are a lot of scheduled builds due to limitations with BuildKite's\nAPI pagination not returning all results or even indicating if there are more results to return\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" 'builds?state=scheduled' \"$@\" |\njq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\nwhile read -r name number url; do\n    url=\"${url#https://api.buildkite.com/v2/}\"\n    echo -n \"Cancelling $name build number $number:  \"\n    \"$srcdir/buildkite_api.sh\" \"$url/cancel\" -X PUT |\n    jq -r '.state'\ndone\n"
  },
  {
    "path": "buildkite/buildkite_create_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 14:38:58 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nCreates a BuildKite pipeline from a JSON configuration provided either as an argument or on stdin\n\nThis JSON file can be created from a configuration downloaded by buildkite_get_pipeline.sh\n\nUsed by buildkite_recreate_pipeline.sh to wipe out old history and reset stats\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"pipeline.json [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -ge 1 ] && [ -f \"$1\" ]; then\n    pipeline_config=\"$(cat \"$1\")\"\n    shift\nelse\n    echo \"config file argument not given, reading config from stdin\"\n    pipeline_config=\"$(cat)\"\nfi\n\nif [ -z \"$BUILDKITE_ORGANIZATION\" ]; then\n    BUILDKITE_ORGANIZATION=\"$(jq -r '.url' <<< \"$pipeline_config\" | sed 's|https://api.buildkite.com/v.*/organizations/||; s|/pipelines/.*||')\"\nfi\n\ncheck_env_defined BUILDKITE_TOKEN\ncheck_env_defined BUILDKITE_ORGANIZATION\n\n#if [ -z \"$pipeline_config\" ]; then\n#    usage \"pipeline config not given\"\n#fi\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/$BUILDKITE_ORGANIZATION/pipelines\" -X POST -d \"$pipeline_config\" \"$@\"\n"
  },
  {
    "path": "buildkite/buildkite_foreach_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:59:00 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nRun a command for each Buildkite pipeline\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the pipeline names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{organization}          => \\$BUILDKITE_ORGANIZATION / \\$BUILDKITE_USER\n{pipeline}              => the pipeline name\n\neg.\n    ${0##*/} echo user={user} name={name} pipeline={pipeline}\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<command>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/buildkite_pipelines.sh\" |\nwhile read -r pipeline; do\n    if [ -n \"${NO_HEADING:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $pipeline\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{pipeline\\}/$pipeline}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\ndone\n"
  },
  {
    "path": "buildkite/buildkite_get_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: devops-bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 14:33:52 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nFetches a BuildKite pipeline's configuration to stdout\n\nUseful for saving the BuildKite pipeline configuration to a local JSON file\n\nThe saved configuration can be loaded via buildkite_create_pipeline.sh\n\nImportant: you probably don't want to commit this pipeline.json to a public Git repo because it contains\nthe webhook URL to triggers builds and could lead to a DoS exploit if publicly disclosed\n\nUsed by buildkite_recreate_pipeline.sh to wipe out old history and reset stats\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    pipeline=\"$1\"\n    shift\nelse\n    pipeline=\"${BUILDKITE_PIPELINE:-${PIPELINE:-}}\"\nfi\n\nif [ -z \"$pipeline\" ]; then\n    usage \"\\$BUILDKITE_PIPELINE not defined and no argument given\"\nfi\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline\" \"$@\" | jq .\n"
  },
  {
    "path": "buildkite/buildkite_patch_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-29 10:23:16 +0100 (Fri, 29 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nPatches a BuildKite pipeline from a partial JSON configuration provided either as an argument or on stdin\n\nThis JSON can be adapted from the download obtained by buildkite_get_pipeline.sh\n\nUsed by buildkite_pipeline_disable_forked_pull_requests.sh to protect your build environment from arbitrary code execution security vulnerabilities via Pull Requests\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline-name> <patchfile.json|json_literal> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined BUILDKITE_ORGANIZATION\n\npipeline=\"$1\"\nshift\n\nif [ $# -ge 1 ]; then\n    if [ -f \"$1\" ]; then\n        pipeline_patch=\"$(cat \"$1\")\"\n        shift\n    elif [[ \"$1\" =~ \"{\" ]]; then\n        pipeline_patch=\"$1\"\n        shift\n    else\n        usage \"patch.json not found and not interpreted as a json literal (missing brace)\"\n    fi\nelse\n    echo \"patch file argument not given, reading patch from stdin\" >&2\n    pipeline_patch=\"$(cat)\"\nfi\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/$BUILDKITE_ORGANIZATION/pipelines/$pipeline\" -X PATCH -d \"$pipeline_patch\" \"$@\"\n"
  },
  {
    "path": "buildkite/buildkite_pipeline_disable_forked_pull_requests.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-29 10:49:08 +0100 (Fri, 29 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nDisables Forked Pull Request builds on a BuildKite pipeline to protect your build environment from arbitrary code execution security vulnerabilities\n\nUses adjacent script buildkite_patch_pipeline.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<pipeline-name> <curl_options>]\"\n\nhelp_usage \"$@\"\n\ncheck_env_defined BUILDKITE_ORGANIZATION\n\npipeline=\"${1:-${BUILDKITE_PIPELINE:-}}\"\n\nif [ -z \"$pipeline\" ]; then\n    usage \"\\$BUILDKITE_PIPELINE not defined and couldn't be determined from JSON config\"\nfi\n\n\"$srcdir/buildkite_patch_pipeline.sh\" \"$pipeline\" \\\n'{\n  \"provider\": {\n    \"settings\": {\n      \"build_pull_request_forks\": false\n    }\n  }\n}'\n"
  },
  {
    "path": "buildkite/buildkite_pipeline_set_skip_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 15:07:11 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets given BuildKite pipelines to skip intermediate queued builds and also cancel running builds in favour of the latest build\n\nIf not pipelines are given, iterates over all pipelines\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<pipeline>]\"\n\nhelp_usage \"$@\"\n\nset_pipeline_settings(){\n    local pipeline=\"$1\"\n    echo \"Setting pipeline '$pipeline' to skip intermediate queued builds and cancel intermediate running builds\"\n    \"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline\" -X PATCH -d '{ \"skip_queued_branch_builds\": true, \"cancel_running_branch_builds\": true }' >/dev/null\n    echo\n    \"$srcdir/buildkite_pipeline_skip_settings.sh\" \"$pipeline\"\n}\n\nif [ $# -gt 0 ]; then\n    for pipeline in \"$@\"; do\n        set_pipeline_settings \"$pipeline\"\n    done\nelse\n    for pipeline in $(\"$srcdir/buildkite_pipelines.sh\"); do\n        set_pipeline_settings \"$pipeline\"\n    done\nfi\n"
  },
  {
    "path": "buildkite/buildkite_pipeline_skip_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 15:07:11 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the cancel settings for a given BuildKite pipeline\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<pipeline>]\"\n\nhelp_usage \"$@\"\n\npipeline=\"${1:-}\"\n\nget_pipeline_settings(){\n    local pipeline=\"$1\"\n    echo \"Pipeline '$pipeline':\"\n    \"$srcdir/buildkite_get_pipeline.sh\" \"$pipeline\" |\n    jq -r '[ .skip_queued_branch_builds,\n             if .skip_queued_branch_builds_filter == \"\" then\n                 \"all\"\n             else\n                 .skip_queued_branch_builds_filter\n             end,\n             .cancel_running_branch_builds,\n             if .cancel_running_branch_builds_filter == \"\" then\n                 \"all\"\n             else\n                 .cancel_running_branch_builds_filter\n             end\n           ] | @tsv' |\n    while read -r skip_queued skip_branches cancel_running cancel_filter; do\n        echo \"Skip Intermediate Builds: $skip_queued\"\n        echo \"Skip Intermediate Branches: $skip_branches\"\n        echo \"Cancel Intermediate Builds: $cancel_running\"\n        echo \"Cancel Intermediate Branches: $cancel_filter\"\n    done\n    echo\n}\n\nif [ $# -gt 0 ]; then\n    for pipeline in \"$@\"; do\n        get_pipeline_settings \"$pipeline\"\n    done\nelse\n    for pipeline in $(\"$srcdir/buildkite_pipelines.sh\"); do\n        get_pipeline_settings \"$pipeline\"\n    done\nfi\n"
  },
  {
    "path": "buildkite/buildkite_pipelines.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nLists BuildKite pipelines in slug format (re-usable in API), one per line to make it easy to iterate on them\n\neg. trigger a build of each pipeline:\n\n./buildkite_pipelines.sh | while read pipeline; do ./buildkite_trigger.sh \\\"\\$pipeline\\\"; done\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines\" | jq -r '.[].slug'\n"
  },
  {
    "path": "buildkite/buildkite_pipelines_vulnerable_forked_pull_requests.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-29 11:34:05 +0100 (Fri, 29 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nQueries all pipelines and prints the setting of executing forked pull requests to show which pipelines are vulnerable to arbitrary code execution\n\nAll pipelines should return 'false' otherwise they are vulnerable and should be updated using buildkite_pipeline_disable_forked_pull_requests.sh\n\nOutput:\n\n    <pipeline_name_slug>    <true/false>\n\n\nUses adjacent scripts buildkite_foreach_pipeline.sh and buildkite_get_pipeline.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_foreach_pipeline.sh\" \\\n    \"'$srcdir/buildkite_get_pipeline.sh' {pipeline} |\n        jq -r '[.slug, .provider.settings.build_pull_request_forks] | @tsv'\"\n"
  },
  {
    "path": "buildkite/buildkite_rebuild_all_pipelines_last_cancelled.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-08 12:08:41 +0100 (Tue, 08 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRebuilds the last cancelled build for each pipeline in BuildKite via its API\n\nSee adjacent scripts for more details:\n\n    buildkite_foreach_pipeline.sh\n    buildkite_rebuild_failed_builds.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_foreach_pipeline.sh\" \"$srcdir/buildkite_rebuild_cancelled_builds.sh\" '{pipeline}' 1\n\n# All Cancelled builds in history\n#\n#\"$srcdir/buildkite_api.sh\" \"builds?state=canceled\" \"$@\" |\n#jq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\n#while read -r name number url; do\n#    url=\"${url#https://api.buildkite.com/v2/}\"\n#    echo -n \"Rebuilding $name build number $number:  \"\n#    \"$srcdir/buildkite_api.sh\" \"$url/rebuild\" -X PUT |\n#    jq -r '.state'\n#done\n"
  },
  {
    "path": "buildkite/buildkite_rebuild_all_pipelines_last_failed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-08 12:08:41 +0100 (Tue, 08 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRebuilds the last failed build on each pipeline in the current organization\n\nSee adjacent scripts for more details:\n\n    buildkite_foreach_pipeline.sh\n    buildkite_rebuild_failed_builds.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_foreach_pipeline.sh\" \"$srcdir/buildkite_rebuild_failed_builds.sh\" '{pipeline}' 1\n\n# All Failed builds in history\n#\n#\"$srcdir/buildkite_api.sh\" \"builds?state=failed\" \"$@\" |\n#jq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\n#while read -r name number url; do\n#    url=\"${url#https://api.buildkite.com/v2/}\"\n#    echo -n \"Rebuilding $name build number $number:  \"\n#    \"$srcdir/buildkite_api.sh\" \"$url/rebuild\" -X PUT |\n#    jq -r '.state'\n#done\n"
  },
  {
    "path": "buildkite/buildkite_rebuild_cancelled_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:59:00 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nRebuilds the last N cancelled builds in BuildKite via its API\n\nUseful to retrying N builds across projects where they may have failed due to agent problems\n\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\npipeline=\"${1:-$(git_repo_name_lowercase)}\"\n\nnum=\"${2:-10}\"\n\nif ! is_int \"$num\"; then\n    usage \"num builds must be an integer\"\nfi\n\nif [ \"$num\" -lt 1 ] || [ \"$num\" -gt 100 ]; then\n    usage \"num builds must be an integer between 1 and 100\"\nfi\n\n# shellcheck disable=SC2154\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline/builds?state=failed&per_page=$num\" |\njq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\nwhile read -r name number url; do\n    url=\"${url#https://api.buildkite.com/v2/}\"\n    echo -n \"Rebuilding $name build number $number:  \"\n    \"$srcdir/buildkite_api.sh\" \"$url/rebuild\" -X PUT |\n    jq -r '.state'\ndone\n"
  },
  {
    "path": "buildkite/buildkite_rebuild_failed_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:59:00 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nRebuilds the last N failed builds in BuildKite via its API\n\nUseful to retrying N builds across projects where they may have failed due to agent problems\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"<pipeline> [<num_builds>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\npipeline=\"${1:-$(git_repo_name_lowercase)}\"\n\nnum=\"${2:-10}\"\n\nif ! is_int \"$num\"; then\n    usage \"num builds must be an integer\"\nfi\n\nif [ \"$num\" -lt 1 ] || [ \"$num\" -gt 100 ]; then\n    usage \"num builds must be an integer between 1 and 100\"\nfi\n\n# shellcheck disable=SC2154\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline/builds?state=failed&per_page=$num\" |\njq -r '.[] | [.pipeline.slug, .number, .url] | @tsv' |\nwhile read -r name number url; do\n    echo -n \"Rebuilding $name build number $number:  \"\n    \"$srcdir/buildkite_api.sh\" \"$url/rebuild\" -X PUT |\n    jq -r '.state'\ndone\n"
  },
  {
    "path": "buildkite/buildkite_rebuild_last_cancelled.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:59:00 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nRebuilds the last cancelled build for each pipeline in BuildKite via its API (after clearing a backlog due to offline agents by cancelling all scheduled builds)\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\nMay fail with Forbidden if your trial account has expired (renew or contact support to switch to free account to get API working again)\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" \"organizations/{organization}/pipelines\" \"$@\" |\njq -r '.[].slug' |\nwhile read -r pipeline; do\n    \"$srcdir/buildkite_api.sh\" \"organizations/{organization}/pipelines/$pipeline/builds?state=canceled\" \"$@\" |\n    jq -r 'limit(1; .[] | [.pipeline.slug, .number, .url] | @tsv)' |\n    while read -r name number url; do\n        url=\"${url#https://api.buildkite.com/v2/}\"\n        echo -n \"Rebuilding $name build number $number:  \"\n        \"$srcdir/buildkite_api.sh\" \"$url/rebuild\" -X PUT |\n        jq\n    done\ndone\n"
  },
  {
    "path": "buildkite/buildkite_recreate_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nRecreates BuildKite pipeline to wipe out history (cancelling all build jobs skews the stats, this is a way of resetting them)\n\nWARNING: the recreated pipeline will have the same name but a newly generated ID, invalidating the existing Status Badge URL\nWARNING: the recreated pipeline will have a different webhook URL, so the GitHub webhook will need to be updated to keep triggering builds\nWorse still, you cannot change the webhook URL back using the saved config as the API ignores the call, which is why the create step cannot set it properly\n\nCreates a <pipeline_name>.json file in the local directory to prevent losing the pipeline configuration\nif the create step fails for any reason after deleting the pipeline (does not pipe the get straight in to the create step)\n\nUses buildkite_get_pipeline.sh and buildkite_create_pipeline.sh adjacent scripts\n\nPipeline name is case sensitive and can be found via:\n\nbuildkite_api.sh /organizations/{organization}/pipelines | jq -r '.[].slug'\n\nOrganization name is also case sensitive and can be found via:\n\nbuildkite_api.sh /organizations | jq -r '.[].slug'\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\npipeline=\"${1:-${BUILDKITE_PIPELINE:-${PIPELINE:-}}}\"\n\nif [ -z \"$pipeline\" ]; then\n    usage \"\\$BUILDKITE_PIPELINE not defined and no argument given\"\nfi\n\nconfig_file=\"$PWD/buildkite-pipeline.$pipeline.json\"\n\ncheck_json_result(){\n    local filename=\"$1\"\n    # prints null\n    #if jq -er '.message' < \"$filename\"; then\n    if jq -er '.message' < \"$filename\" | grep -v '^null$' | grep '[A-Za-z]'; then\n        exit 1\n    fi\n}\n\ntmp=\"$(mktemp)\"\n# want splitting\n# shellcheck disable=SC2086\ntrap 'rm -- \"$tmp\"' $TRAP_SIGNALS\n\necho \"saving pipeline '$pipeline' to local file '$config_file'\"\n\"$srcdir/buildkite_get_pipeline.sh\" \"$pipeline\" > \"$config_file\"\ncheck_json_result \"$config_file\"\n\necho \"deleting pipeline '$pipeline'\"\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline\" -X DELETE \"$@\" | tee \"$tmp\"\ncheck_json_result \"$tmp\"\n\necho \"recreating pipeline '$pipeline'\"\n\"$srcdir/buildkite_create_pipeline.sh\" \"$config_file\" \"$@\" > \"$tmp\"\ncheck_json_result \"$tmp\"\n\necho \"triggering build of recreated pipeline '$pipeline'\"\n\"$srcdir/buildkite_trigger.sh\" \"$pipeline\" > \"$tmp\"\ncheck_json_result \"$tmp\"\n"
  },
  {
    "path": "buildkite/buildkite_retry_jobs_dead_agents.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:59:00 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://buildkite.com/docs/apis/rest-api/builds\n\n# https://buildkite.com/docs/apis/rest-api/jobs\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nTriggers job retries jobs within builds where Failed status was due to a dead or timed out agent, using the BuildKite API\n\nThis is slightly better than retrying the failed builds because it restarts those builds from point of agent failure and replaces the Failed status with the real final status\n\nReally BuildKite should auto-retry in this scenario, which can be configured but is not the default, and this script is a quick workaround to retry failed jobs\n\nhttps://forum.buildkite.community/t/reschedule-builds-on-other-agents-rather-than-fail-builds-when-agents-time-out-or-are-killed-machine-shut-down-or-put-to-sleep/1388\n\nCan optionally specify a pipeline to retry only failed jobs in builds for that pipeline\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<pipeline>]\"\n\nhelp_usage \"$@\"\n\npipeline=\"${1:-}\"\n\nurl_path=\"/builds\"\n\nif [ -n \"$pipeline\" ]; then\n    url_path=\"/{organization}/pipelines/$pipeline/builds\"\nfi\n\n\"$srcdir/buildkite_api.sh\" \"$url_path\" |\njq -r '.[] |\n       select(.state == \"failed\") |\n       select(.jobs[].exit_status == -1) |\n       [.number, (.jobs[] | (select(.exit_status == -1) | .id)), .pipeline.slug ] |\n       @tsv' |\nwhile read -r build_number job_id pipeline_slug; do\n    timestamp \"retrying '$pipeline_slug' build '$build_number' job '$job_id'\"\n    \"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline_slug/builds/$build_number/jobs/$job_id/retry\" -X PUT >/dev/null\ndone\n"
  },
  {
    "path": "buildkite/buildkite_running_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-01 18:33:42 +0100 (Wed, 01 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"\nLists BuildKite running builds via its API\n\nhttps://buildkite.com/docs/apis/rest-api/builds\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_api.sh\" 'builds?state=running' \"$@\" |\njq -r '.[] | [.pipeline.slug, .branch, .number, .commit, .created_at, .jobs[0].agent.name] | @tsv' |\n#while read -r name branch number commit created agent; do\n#    commit=\"${commit:0:8}\"\n#    echo \"$name $branch $number $commit $created $agent\"\n#done |\nsed 's/\\([[:space:]][[:alnum:]]\\{8\\}\\)[[:alnum:]]\\{32\\}[[:space:]]/\\1 /' |\ncolumn -t\n"
  },
  {
    "path": "buildkite/buildkite_save_pipelines.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nsave_dir=\".buildkite-pipelines\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nSaves all BuildKite pipelines in your \\$BUILDKITE_ORGANIZATION to local JSON files in \\$PWD/$save_dir\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\ntimestamp=\"$(date '+%F_%T' | sed 's/:/-/g')\"\n\nmkdir -pv \"$save_dir\"\n\n\"$srcdir/buildkite_pipelines.sh\" \"$@\" |\nwhile read -r pipeline; do\n    json_file=\"$save_dir/$pipeline.json\"\n    echo \"Saving pipeline '$pipeline' to '$json_file'\"\n    \"$srcdir/buildkite_get_pipeline.sh\" \"$pipeline\" \"$@\" > \"$json_file\"\n    if jq -er '.message' < \"$json_file\" | grep -v '^null$' | grep '[A-Za-z]'; then\n        exit 1\n    fi\n    # similar behaviour to bak function\n    cp -av -- \"$json_file\" \"$json_file.bak.$timestamp\"\n    echo\ndone\n"
  },
  {
    "path": "buildkite/buildkite_set_pipeline_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 13:42:12 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the description of a BuildKite pipeline via the BuildKite API\n\nUses the adajcent script buildkite_api.sh, see there for authentication details\n\n\nExample:\n\n    ${0##*/} devops-bash-tools    my new description\n\n\nIf no args are given, will read pipeline_slug and description from standard input for easy chaining with other tools, can easily update multiple repositories this way, one pipeline_slug + description per line:\n\n    echo <pipeline_slug> <description> | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline_slug> <description>\"\n\nhelp_usage \"$@\"\n\n#min_args 2 \"$@\"\n\nset_pipeline_description(){\n    local pipeline_slug=\"$1\"\n    local description=\"${*:2}\"\n\n    timestamp \"Setting BuildKite pipeline '$pipeline_slug' description to '$description'\"\n\n    # this used to work with just -d \"description=$description\" when Accept and Content-Type headers were omitted\n    # but since curl_api_opts auto-sets headers to application/json this must be json or else get 400 bad request error\n    \"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline_slug\" -X PATCH -d '{\"description\": \"'\"$description\"'\"}' >/dev/null\n}\n\nif [ $# -gt 0 ]; then\n    if [ $# -lt 2 ]; then\n        usage\n    fi\n    set_pipeline_description \"$@\"\nelse\n    while read -r pipeline_slug description; do\n        [ -n \"$pipeline_slug\" ] || continue\n        [ -n \"$description\" ] || continue\n        set_pipeline_description \"$pipeline_slug\" \"$description\"\n    done\nfi\n"
  },
  {
    "path": "buildkite/buildkite_set_pipeline_description_from_github.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: devops-bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 14:30:07 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets a given BuildKite pipeline's description to match its source GitHub repo's description\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npipeline=\"$1\"\nshift || :\n\nrepo=\"$(\"$srcdir/buildkite_get_pipeline.sh\" \"$pipeline\" | jq -r .repository)\"\n\nif ! [[ $repo =~ github.com ]]; then\n    die \"ERROR: source repository for pipeline '$pipeline' is not GitHub.com - sync'ing description from other VCS providers is not supported at this time\"\nfi\n\nrepo=\"${repo##github.com/}\"\n\n\"$srcdir/../github/github_repo_description.sh\" \"$repo\" |\nwhile read -r repo description; do\n    \"$srcdir/buildkite_set_pipeline_description.sh\" \"$pipeline\" \"$description\"\ndone\n"
  },
  {
    "path": "buildkite/buildkite_sync_pipeline_descriptions_from_github.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-17 14:12:12 +0000 (Thu, 17 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each BuildKite pipeline, syncs its description to match its source GitHub repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_foreach_pipeline.sh\" \"$srcdir/buildkite_set_pipeline_description_from_github.sh\" \"{pipeline}\"\n"
  },
  {
    "path": "buildkite/buildkite_trigger.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nTriggers a BuildKite build job for a given pipeline\n\nPipeline can be given as an argumnent, or taken from \\$BUILDKITE_PIPELINE / \\$PIPELINE environment variables\n\nOtherwise tries to infer from the current Git repository remote origin URL\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pipeline> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    pipeline=\"$1\"\n    shift\nelse\n    pipeline=\"${BUILDKITE_PIPELINE:-${PIPELINE:-$(git_repo_name_lowercase)}}\"\nfi\n\nif [ -z \"$pipeline\" ]; then\n    usage \"\\$BUILDKITE_PIPELINE not defined and no argument given\"\nfi\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/{organization}/pipelines/$pipeline/builds\" \\\n    -X \"POST\" \\\n    -F \"commit=${BUILDKITE_COMMIT:-HEAD}\" \\\n    -F \"branch=${BUILDKITE_BRANCH:-master}\" \\\n    -F \"message=triggered by Hari Sekhon ${0##*/} script\" \"$@\"\n"
  },
  {
    "path": "buildkite/buildkite_trigger_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nTriggers BuildKite build jobs for all pipelines in the \\$BUILDKITE_ORGANIZATION\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/buildkite_pipelines.sh\" |\nwhile read -r pipeline; do\n    echo \"triggering job for pipeline '$pipeline'\"\n    \"$srcdir/buildkite_trigger.sh\" \"$pipeline\" > /dev/null\ndone\n"
  },
  {
    "path": "buildkite/buildkite_update_pipeline.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 14:38:58 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nUpdates a BuildKite pipeline from a full JSON configuration provided either as an argument or on stdin\n\nThis JSON file can be created from a configuration downloaded by buildkite_get_pipeline.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"pipeline.json [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -ge 1 ] && [ -f \"$1\" ]; then\n    pipeline_config=\"$(cat \"$1\")\"\n    shift\nelse\n    echo \"config file argument not given, reading config from stdin\" >&2\n    pipeline_config=\"$(cat)\"\nfi\n\npipeline=\"$(jq -r '.slug' <<< \"$pipeline_config\" 2>/dev/null || :)\"\npipeline=\"${pipeline:-${BUILDKITE_PIPELINE:-}}\"\n\nif [ -z \"$BUILDKITE_ORGANIZATION\" ]; then\n    BUILDKITE_ORGANIZATION=\"$(jq -r '.url' <<< \"$pipeline_config\" | sed 's|https://api.buildkite.com/v.*/organizations/||; s|/pipelines/.*||')\"\nfi\n\ncheck_env_defined BUILDKITE_ORGANIZATION\n\nif [ -z \"$pipeline\" ]; then\n    usage \"\\$BUILDKITE_PIPELINE not defined and couldn't be determined from JSON config\"\nfi\n\n\"$srcdir/buildkite_api.sh\" \"/organizations/$BUILDKITE_ORGANIZATION/pipelines/$pipeline\" -X PATCH -d \"$pipeline_config\" \"$@\"\n"
  },
  {
    "path": "checks/check_all.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC1090,SC1091\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-11-05 23:29:15 +0000 (Thu, 05 Nov 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -z \"${PROJECT:-}\" ]; then\n    export PROJECT=bash-tools\nfi\n\npushd \"$srcdir\" >/dev/null\n\n# shellcheck source=lib/utils.sh\n. \"lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"lib/docker.sh\"\n\npopd >/dev/null || :\n\nsection \"Running Bash Tools ALL\"\n\n# Breaks on CentOS Docker without this, although works on Debian, Ubuntu and Alpine without\nexport LINES=\"${LINES:-25}\"\nexport COLUMNS=\"${COLUMNS:-80}\"\n\ndeclare_if_inside_docker\n\nbash_tools_start_time=\"$(start_timer)\"\n\n# do help afterwards for Spark to be downloaded, and then help will find and use downloaded spark for SPARK_HOME\n#\"$srcdir/help.sh\"\n\n# don't run this here, it needs to be called explicitly otherwise will fail 'make test deep-clean'\n#\"$srcdir/check_docker_clean.sh\"\n\nif [ -z \"${BASH_EXCLUDED_FILES_FUNCTION:-}\" ]; then\n    if [ -f tests/excluded.sh ]; then\n        export BASH_EXCLUDED_FILES_FUNCTION=tests/excluded.sh\n    fi\nfi\n\n\"$srcdir/check_license_exists.sh\"\n\"$srcdir/check_readme_exists.sh\"\n\n\"$srcdir/check_symlinks.sh\"\n\n\"$srcdir/check_aws_no_git_credentials.sh\"\n\n\"$srcdir/check_git_no_merge_remnants.sh\"\n\n\"$srcdir/check_git_commit_authors.sh\"\n\n\"$srcdir/check_github_actions_workflows_without_checkout.sh\" || :\n\n\"$srcdir/check_github_actions_workflow_injection.sh\"\n\nif [ -z \"${NO_JSON_CHECK:-}\" ]; then\n    \"$srcdir/check_json.sh\"\nfi\n\nif [ -z \"${NO_XML_CHECK:-}\" ]; then\n    \"$srcdir/check_xml.sh\"\n    if [ -n \"${VALIDATE_XML_BASIC:-}\" ]; then\n        if type -P validate_xml.py &>/dev/null; then\n            validate_xml.py .\n        fi\n    fi\nfi\n\nif [ -z \"${NO_YAML_CHECK:-}\" ]; then\n    \"$srcdir/check_yaml.sh\"\n    if [ -n \"${VALIDATE_YAML_BASIC:-}\" ]; then\n        if type -P validate_yaml.py &>/dev/null; then\n            validate_yaml.py .\n        fi\n    fi\nfi\n\n\"$srcdir/check_bash_duplicate_defs.sh\" || :\n\n# duplicate packages eg. in nagios-plugins submodules\n# ./pylib/setup/deb-packages-dev.txt:libkrb5-dev\n# ./lib/setup/deb-packages-dev.txt:libkrb5-dev\n\"$srcdir/check_duplicate_packages.sh\" || :\n\n\"$srcdir/check_duplicate_dependencies.sh\"\n\n\"$srcdir/check_shebang_non_executable.sh\"\n\n# false alerts on the second line of this construct\n#\n# if [ -x $srcdir/script.sh ]; then\n#     $srcdir/script.sh\n#\n#\"$srcdir/check_srcdir_references.sh\"\n\n\"$srcdir/check_bash_syntax.sh\"\n\n# want splitting\n# shellcheck disable=SC2046\n\"$srcdir/check_bash_references.sh\" . $(for x in setup lib; do [ -f \"$x\" ] && echo \"$x\"; done)\n\n\"$srcdir/check_bash_arrays.sh\"\n\n\"$srcdir/check_shell_commands_dash_protections.sh\"\n\n\"$srcdir/check_tests_run_qualified.sh\"\n\n\"$srcdir/check_makefiles.sh\"\n\n\"$srcdir/check_vagrantfiles.sh\"\n\nif [ -z \"${NO_DOCKERFILE_CHECK:-}\" ]; then\n    \"$srcdir/check_dockerfiles.sh\"\nfi\n\nif [ -z \"${NO_DOCKER_COMPOSE_CHECK:-}\" ]; then\n    \"$srcdir/check_docker_compose.sh\"\nfi\n\nif [ -z \"${NO_ANSIBLE_PLAYBOOK_CHECK:-}\" ]; then\n    \"$srcdir/check_ansible_playbooks.sh\"\nfi\n\n# this is usually run after build, no point testing again\nif [ -z \"${NO_MAVEN_POM_CHECK:-}\" ]; then\n    \"$srcdir/check_maven_pom.sh\"\nfi\n\n# this is usually run after build, no point testing again\nif [ -z \"${NO_GRADLE_BUILD_CHECK:-}\" ]; then\n    \"$srcdir/check_gradle_build.sh\"\nfi\n\n# =======\n# XXX: not enabling by default because too simplistic for real projects, likely to cause cross-reference breakages\nif [ -n \"${GROOVYC_CHECK:-}\" ]; then\n    \"$srcdir/check_groovyc.sh\"\nfi\n\nif [ -n \"${JAVAC_CHECK:-}\" ]; then\n    \"$srcdir/check_javac.sh\"\nfi\n# =======\n\nif [ -z \"${NO_PERL_SYNTAX_CHECK:-}\" ]; then\n    \"$srcdir/check_perl_syntax.sh\"\nfi\n\nif [ -z \"${NO_RUBY_SYNTAX_CHECK:-}\" ]; then\n    \"$srcdir/check_ruby_syntax.sh\"\nfi\n\nif [ -z \"${NO_PYTHON_COMPILE:-}\" ]; then\n    \"$srcdir/../python/python_compile.sh\"\nfi\n\nif [ -z \"${NO_PYTHON_MISC_CHECK:-}\" ]; then\n    \"$srcdir/check_python_misc.sh\"\nfi\n\nif [ -z \"${NO_PYTHON_ASSERT_CHECK:-}\" ]; then\n    WARN_ONLY=1 \"$srcdir/check_python_asserts.sh\"\nfi\n\nif [ -z \"${NO_PYTHON_EXCEPTION_PASS_CHECK:-}\" ]; then\n    \"$srcdir/check_python_exception_pass.sh\"\nfi\n\nif [ -z \"${NO_PYTHON_PYLINT_CHECK:-}\" ]; then\n    \"$srcdir/check_python_pylint.sh\"\nfi\n\nif [ -z \"${NO_JAVASCRIPT_ESLINT_CHECK:-}\" ]; then\n    \"$srcdir/check_javascript_eslint.sh\"\nfi\n\n#\"$srcdir/../python/python3.sh\"\n\n# this is usually run after build, no point testing again\n#. \"$srcdir/check_sbt_build.sh\"\n\n\"$srcdir/check_readme_badges.sh\"\n\nif [ -z \"${NO_CIRCLECI_CHECK:-}\" ]; then\n    \"$srcdir/check_circleci_config.sh\"\nfi\nif [ -z \"${NO_CONCOURSE_CHECK:-}\" ]; then\n    \"$srcdir/check_concourse_config.sh\"\nfi\nif [ -z \"${NO_CODEFRESH_CHECK:-}\" ]; then\n    \"$srcdir/check_codefresh_config.sh\"\nfi\nif [ -z \"${NO_DRONE_CHECK:-}\" ]; then\n    \"$srcdir/check_drone_yml.sh\"\nfi\nif [ -z \"${NO_GITLAB_CHECK:-}\" ]; then\n    \"$srcdir/check_gitlab_ci_yml.sh\"\nfi\nif [ -z \"${NO_TRAVISCI_CHECK:-}\" ]; then\n    \"$srcdir/check_travis_yml.sh\" || :  # broken library dependency highline on Fedora\nfi\nif ! is_CI &&\n   [ -n \"${SHIPPABLE_TOKEN:-}\" ]; then\n    \"$srcdir/check_shippable_readme_ids.sh\"\nfi\n\n\"$srcdir/check_tld_chars.sh\"\n\n\"$srcdir/check_no_tabs.sh\"\n\n# too heavy to run all the time, isExcluded on every file has really bad performance\n\"$srcdir/check_whitespace.sh\"\n\n\"$srcdir/check_no_suid_guid_shell_scripts.sh\"\n\n# ========================================\n# Expensive checks, do separately in CI/CD\n#\n#\"$srcdir/check_url_links.sh\"\n\n#\"$srcdir/check_pytools.sh\"\n\n#for script in $(find . -name 'test*.sh'); do\n#    \"$srcdir/$script\" -vvv\n#done\n\ntime_taken \"$bash_tools_start_time\" \"Bash Tools All Checks Completed in\"\nsection2 \"Bash Tools All Checks Completed\"\necho\n"
  },
  {
    "path": "checks/check_ansible_playbooks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 17:24:28 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -type f -name '*playbook.y*ml' | sort)\"\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Ansible Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping Ansible syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping Ansible syntax checks\"\n    echo\nelse\n    if ! command -v ansible-lint &>/dev/null; then\n        echo \"ansible-lint not found in \\$PATH, not running Ansible syntax checks\"\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\n    ansible-lint --version\n    echo\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for x in $filelist; do\n        isExcluded \"$x\" && continue\n        printf \"%-${max_len}s \" \"$x:\"\n        set +eo pipefail\n        output=\"$(ansible-lint \"$x\")\"\n        # shellcheck disable=SC2181\n        if [ $? -eq 0 ]; then\n            echo \"OK\"\n        else\n            echo \"FAILED\"\n            if [ -z \"${QUIET:-}\" ]; then\n                echo\n                echo \"$output\"\n                echo\n            fi\n            if [ -z \"${NOEXIT:-}\" ]; then\n                # shellcheck disable=SC2317\n                return 1 &>/dev/null ||\n                exit 1\n            fi\n        fi\n        set -eo pipefail\n    done\n    time_taken \"$start_time\"\n    section2 \"All Ansible playbook files passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_aws_no_git_credentials.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-18 13:57:12 +0100 (Fri, 18 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# This is a simplistic check, see Semgrep workflows under .github/workflows for more comprehensive check\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"AWS Git credentials scan\"\n\nstart_time=\"$(start_timer)\"\n\nlocation=\"${1:-.}\"\n\nif [ \"$location\" = . ]; then\n    :\nelif [ -d \"$location\" ]; then\n    cd \"$location\"\nelse\n    cd \"$(dirname \"$location\")\"\nfi\n\n# $(pwd) more reliable than $PWD\necho \"checking $(pwd)\"\necho\n\nmatches=\"$(git grep -Ei \\\n    -e 'AWS_ACCESS_KEY_ID[[:space:]]*[=:][\"'\"'\"']*[^$][[:alnum:]]{4,}' \\\n    -e 'AWS_SECRET_ACCESS_KEY[[:space:]]*[=:][\"'\"'\"']*[^$][[:alnum:]]{4,}' \\\n    -e 'AWS_SESSION_TOKEN[[:space:]]*[=]:[\"'\"'\"']*[^$][[:alnum:]]{4,}' \\\n    | grep -Fv 'credentials(' \\\n    || :\n    # credentials excludes Jenkinsfile environment variables sourced from credential sources\n)\"\nif [ -f .gitallowed ]; then\n    # makes no difference, .gitallowed is exempted next anyway\n    #matches=\"$(grep -Ev -f .gitallowed <<< \"$matches\" | grep -Fv -f .gitallowed || :)\"\n    matches=\"$(grep -Ev -f .gitallowed <<< \"$matches\" || :)\"\nfi\n#matches=\"$(grep -Ev -e \"^${0##*/}:[[:space:]]+-e[[:space:]]+'AWS_\" \\\n#                    -e '^.gitallowed:' \\\n#                    <<< \"$matches\" || :)\"\nif [ -n \"$matches\" ]; then\n        # dangerous, fails silently and suppressed legitimate matches\n        #grep -v -f \"$gitallowed\" |\n        #grep -v -e '\\.bash\\.d/aws.sh:' \\\n        #        -e \"${0##*/}:\" |\n    # shellcheck disable=SC2001\n    sed 's/\\(=.....\\).*/\\1....../' <<< \"$matches\"\n    echo\n    echo \"DANGER: potential AWS credentials found in Git!!\"\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"OK: no AWS credentials found in Git\"\necho\n"
  },
  {
    "path": "checks/check_bash_arrays.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2018-08-16 13:47:32 +0100 (Thu, 16 Aug 2018)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks we don't get any bash array variables getting overwritten\n#\n# Written for test_anonymize.sh in DevOps Python & Perl Tools repos which makes heavy use of bash array variables for tests\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nif [ $# -eq 0 ]; then\n    if [ -z \"$(find \"${1:-.}\" -type f -iname '*.sh')\" ]; then\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\nfi\n\nsection \"Bash Array Checks (Duplicate Indices and Syntax Errors)\"\n\ncheck_bash_arrays(){\n    local filename=\"$1\"\n    echo -n \"checking bash arrays:  $1\"\n    set +eo pipefail\n    dups=\"$(grep -o '^[[:space:]]*[[:alnum:]]\\+[[[:digit:]]\\+]=' \"$filename\" | sed 's/[[:space:]]*//' | sort | uniq -d)\"\n    early_equals=\"$(grep -o '[[:alnum:]]\\+=[[[:digit:]]\\+]=' \"$filename\")\"\n    set -eo pipefail\n    if [ -n \"$dups\" ]; then\n        echo \" => Duplicates detected!\"\n        echo \"$dups\"\n        exit 1\n    elif [ -n \"$early_equals\" ]; then\n        echo \" => Invalid array definition detected!\"\n        echo \"$early_equals\"\n        exit 1\n    else\n        echo \" => OK\"\n    fi\n}\n\nrecurse_dir(){\n    for x in $(find \"${1:-.}\" -type f -iname '*.sh' | sort); do\n        # TODO: consider skipping shell scripts which don't contain '#!.*bash'\n        isExcluded \"$x\" && continue\n        check_bash_arrays \"$x\"\n    done\n}\n\nstart_time=\"$(start_timer)\"\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        if [ -d \"$x\" ]; then\n            recurse_dir \"$x\"\n        else\n            check_bash_arrays \"$x\"\n        fi\n    done\nelse\n    recurse_dir .\nfi\n\ntime_taken \"$start_time\"\nsection2 \"All Bash programs passed array duplicates check\"\necho\n"
  },
  {
    "path": "checks/check_bash_duplicate_defs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-08-02 13:30:09 +0100 (Fri, 02 Aug 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks for duplicate function or alias definitions within a given collection of bash scripts - useful for checking across many .bash.d/ includes\n#\n# This isn't caught by shellcheck - for example it's common to alias 'p' to ping, but I also do this for 'kubectl get pods'\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking for Bash duplicate definitions (functions, aliases)\"\n\nstart_time=\"$(start_timer)\"\n\ncheck_duplicate_defs(){\n    echo \"Checking for duplicate definitions in the following files:\"\n    echo\n    for x in \"$@\"; do\n        echo \"$x\"\n    done\n    echo\n    check_duplicate_functions \"$@\"\n    check_duplicate_aliases \"$@\"\n    check_duplicate_aliases_vs_functions \"$@\"\n}\n\nget_functions(){\n    # ignore definitions that happen inside indented blocks such as if/else statements creating conditional functions\n    #grep -Eh '^[[:space:]]*(function[[:space:]]+)?[[:alnum:]-]+[[:space:]]*\\(' \"$@\" 2>/dev/null |\n    grep -Eh '^(function[[:space:]]+)?[[:alnum:]-]+[[:space:]]*\\(' \"$@\" 2>/dev/null |\n    grep -Ev '^[[:space:]]*for[[:space:]]*\\(\\(' |\n    sed 's/^[[:space:]]*\\(function[[:space:]]*\\)*//; s/[[:space:]]*(.*//' |\n    sort\n}\n\nget_aliases(){\n    grep -Eho '^[[:space:]]*alias[[:space:]]+[[:alnum:]]+=' \"$@\" 2>/dev/null |\n    sed 's/^[[:space:]]*alias[[:space:]]*//; s/=$//' |\n    sort\n}\n\ncheck_duplicate_functions(){\n    echo \"* Checking for duplicate function definitions\"\n    echo\n    local function_dups\n    set +o pipefail\n    function_dups=\"$(get_functions \"$@\" | uniq -d)\"\n    if [ -n \"$function_dups\" ]; then\n        echo \"Duplicate functions detected across input files!\"\n        echo\n        for x in $function_dups; do\n            grep -Eno \"^[[:space:]]*(function[[:space:]]+)?${x}[[:space:]]*\\\\(\" \"$@\" 2>/dev/null || :\n        done\n        echo\n        exit 1\n    fi\n}\n\ncheck_duplicate_aliases(){\n    echo \"* Checking for duplicate alias definitions\"\n    echo\n    local alias_dups\n    set +o pipefail\n    alias_dups=\"$(get_aliases \"$@\" | uniq -d)\"\n    if [ -n \"$alias_dups\" ]; then\n        echo \"Duplicate aliases detected across input files!\"\n        echo\n        for x in $alias_dups; do\n            grep -En \"^[[:space:]]*alias[[:space:]]+${x}=\" \"$@\" 2>/dev/null || :\n        done\n        echo\n        echo\n        exit 1\n    fi\n}\n\ncheck_duplicate_aliases_vs_functions(){\n    echo \"* Checking for duplicate alias vs function definitions\"\n    echo\n    local defs\n    local duplicate_defs\n    defs=\"$(get_functions \"$@\")\n$(get_aliases \"$@\")\"\n    duplicate_defs=\"$(sort <<< \"$defs\" | uniq -d)\"\n    if [ -n \"$duplicate_defs\" ]; then\n        echo \"Duplicate function vs alias definitions detected across input files!\"\n        echo\n        for x in $duplicate_defs; do\n            grep -Eno -e \"^[[:space:]]*alias[[:space:]]+${x}=\" \\\n                -e \"^[[:space:]]*(function[[:space:]]+)?${x}[[:space:]]*\\\\(\" \\\n                      \"$@\" 2>/dev/null || :\n        done\n        echo\n        echo\n        exit 1\n    fi\n}\n\nif [ $# -gt 0 ]; then\n    check_duplicate_defs \"$@\"\nelse\n    # only this repo\n    #check_duplicate_defs \"$srcdir/.bashrc\" \"$srcdir\"/.bash.d/*.sh\n    # $HOME + this repo\n    check_duplicate_defs ~/.bashrc ~/.bash.d/*.sh \"$srcdir/../.bashrc\" \"$srcdir\"/.bash.d/*.sh\nfi\n\ntime_taken \"$start_time\"\nsection2 \"No duplicate bash definitions found\"\necho\n"
  },
  {
    "path": "checks/check_bash_references.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-08-04 18:08:01 +0100 (Wed, 04 Aug 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks for broken script references in scripts in the current or given directories, where they are referenced via \\$srcdir/scriptname.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dirs_to_check>]\"\n\nhelp_usage \"$@\"\n\nfor x in \"${@:-.}\"; do\n    pushd \"$x\" &>/dev/null\n    for script in $(git grep --max-depth 0 '^[^#]*srcdir/[[:alnum:]_]*.sh' -- . |\n                    grep -v \"${0##*/}:.*\\\\\\$srcdir/scriptname.sh\" |\n                    grep -Eo 'srcdir/[[:alnum:]_/-]*\\.sh' |\n                    sed 's/srcdir/./g' |\n                    sort -u); do\n        if ! [ -f \"$script\" ]; then\n            echo \"FAILED to find script $script\"\n            exit 1\n        fi\n    done\n    popd &>/dev/null\ndone\n"
  },
  {
    "path": "checks/check_bash_syntax.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-22 20:54:53 +0000 (Fri, 22 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# This really only checks basic syntax, if you're made command errors this won't catch it\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nif [ $# -eq 0 ]; then\n    if [ -z \"$(find \"${1:-.}\" -type f -iname '*.sh')\" ]; then\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\nfi\n\nsection \"Shell Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif ! type -P shellcheck &>/dev/null; then\n    echo \"WARNING: shellcheck not installed, will only do basic checks\"\n    echo\nfi\n\nbash --version\necho\n\ncheck_shell_syntax(){\n    echo -n \"checking shell syntax: $1 \"\n    if grep -q '#!/bin/bas[h]' \"$1\"; then\n        # quotes in middle disrupt warning on our own script\n        echo \"WARNING: '#!\"\"/bin/bash' detected, consider using '#!/usr/bin/env bash' instead\"\n    fi\n    bash -n \"$1\"\n    if type -P shellcheck &>/dev/null; then\n        local basename=\"${1##*/}\"\n        local dirname\n        dirname=\"$(dirname \"$1\")\"\n        # this allows following source hints relative to the source file to be safe to run from any $PWD\n        if ! pushd \"$dirname\" &>/dev/null; then\n            echo \"ERROR: failed to pushd to $dirname\"\n            exit 1\n        fi\n        # -x allows to follow source hints for files not given as arguments\n        shellcheck -x \"$basename\" || :\n        if ! popd &>/dev/null; then\n            echo \"ERROR: failed to popd from $dirname\"\n        fi\n    fi\n    echo \"=> OK\"\n}\n\nrecurse_dir(){\n    for x in $(find \"${1:-.}\" -type f -iname '*.sh' | sort); do\n        isExcluded \"$x\" && continue\n        [[ \"$x\" =~ ${EXCLUDED:-} ]] && continue\n        check_shell_syntax \"$x\"\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        if [ -d \"$x\" ]; then\n            recurse_dir \"$x\"\n        else\n            check_shell_syntax \"$x\"\n        fi\n    done\nelse\n    recurse_dir .\nfi\n\ntime_taken \"$start_time\"\nsection2 \"All Shell programs passed syntax check\"\necho\n"
  },
  {
    "path": "checks/check_caches_clean.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-10-06 00:58:45 +0200 (Fri, 06 Oct 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# Personal Language Caches we want to check have been removed:\n#\n# .cache => Python pip\n#\n# .cpan  => Perl\n# .cpanm\n#\n# .gem   => Ruby\n#\n# Java / Scala / Groovy:\n#\n# .gradle => Gradle\n# .groovy => Groovy (contains grapes/)\n# .ivy    => Ivy (Sbt / Gradle)\n# .ivy2\n# .m2     => Maven\n# .sbt    => SBT\n\npersonal_cache_list=\"\n.cache\n.cpan\n.cpanm\n.gem\n.gradle\n.groovy\n.ivy\n.ivy2\n.m2\n.sbt\n\"\n\nexitcode=0\n\nfor cache in $personal_cache_list; do\n    for home in /root ~; do\n        # This might fail if we're not running as root :-/\n        # consider sudo'ing and find / -type d -name $cache but that might find .cache under some app or something, although we should probably remove that too\n        # for now this is good enough as most docker images are built as root\n        # should test for sudo availability as well\n        if [ -e \"$home/$cache\" ]; then\n            echo \"$home/$cache detected\"\n            exitcode=1\n        fi\n    done\ndone\n\nexit $exitcode\n"
  },
  {
    "path": "checks/check_circleci_config.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-15 00:33:52 +0000 (Fri, 15 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\n#return 0 &>/dev/null || :\n#exit 0\n\nsection \"Circle CI Config Validate\"\n\nstart_time=\"$(start_timer)\"\n\n#if is_travis; then\n#    echo \"Running inside Circle CI, skipping lint check\"\nif is_inside_docker; then\n    echo \"Running inside Docker, skipping Circle lint check\"\nelse\n    if type -P circleci &>/dev/null; then\n        type -P circleci\n        printf \"version: \"\n        circleci version\n        echo\n        find . -path '*/.circleci/config.yml' |\n        while read -r config; do\n            timestamp \"checking CircleCI config: $config\"\n            circleci config validate \"$config\"\n            echo >&2\n        done\n    else\n        echo \"WARNING: skipping Circle check as circleci command not found in \\$PATH ($PATH)\"\n    fi\nfi\n\necho\ntime_taken \"$start_time\"\nsection2 \"Circle CI yaml validation succeeded\"\necho\n"
  },
  {
    "path": "checks/check_codefresh_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 22:47:13 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nconfig=\"codefresh.yml\"\n\nfiles=\"$(find \"${1:-.}\" -name \"$config\")\"\n\nif [ -z \"$files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"C o d e f r e s h\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P codefresh &>/dev/null; then\n    type -P codefresh\n    codefresh version\n    echo\n    if [ -n \"${CODEFRESH_KEY:-}\" ]; then\n        while read -r config; do\n            echo \"Validating $config\"\n            codefresh validate \"$config\" || exit $?\n            echo\n        done <<< \"$files\"\n        time_taken \"$start_time\"\n        section2 \"Codefresh config checks passed\"\n    else\n        echo \"\\$CODEFRESH_KEY not found in environment, can't validate without API authentication, skipping codefresh config checks\"\n    fi\nelse\n    echo \"Codefresh command not found in \\$PATH, skipping codefresh config checks\"\nfi\n\necho\n"
  },
  {
    "path": "checks/check_concourse_config.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-21 19:21:11 +0000 (Sat, 21 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nconfig=\".concourse.yml\"\n\nif [ -z \"$(find \"${1:-.}\" -name \"$config\")\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"C o n c o u r s e\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P fly &>/dev/null; then\n    type -P fly\n    fly --version\n    echo\n    find \"${1:-.}\" -name \"$config\" |\n    while read -r config; do\n        echo \"Validating $config\"\n        fly validate-pipeline -c \"$config\" || exit $?\n        echo\n    done\nelse\n    echo \"Concourse 'fly' command not found in \\$PATH, skipping concourse config checks\"\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Concourse config checks passed\"\necho\n"
  },
  {
    "path": "checks/check_cson.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 17:18:03 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -type f -name '*.cson' | sort)\"\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"CSON Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping CSON syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping CSON syntax checks\"\n    echo\nelse\n    if ! command -v python-cson &>/dev/null; then\n        echo \"python-cson not found in \\$PATH, not running CSON syntax checks\"\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for x in $filelist; do\n        isExcluded \"$x\" && continue\n        printf \"%-${max_len}s \" \"$x:\"\n        set +eo pipefail\n        output=\"$(python-cson -f /dev/null \"$x\" 2>/dev/null)\"\n        # shellcheck disable=SC2181\n        if [ $? -eq 0 ]; then\n            echo \"OK\"\n        else\n            echo \"FAILED\"\n            if [ -z \"${QUIET:-}\" ]; then\n                echo\n                echo \"$output\"\n                echo\n            fi\n            if [ -z \"${NOEXIT:-}\" ]; then\n                # shellcheck disable=SC2317\n                return 1 &>/dev/null ||\n                exit 1\n            fi\n        fi\n        set -eo pipefail\n    done\n    time_taken \"$start_time\"\n    section2 \"All CSON files passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_docker_clean.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-10-06 00:58:45 +0200 (Fri, 06 Oct 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\nif is_inside_docker; then\n    \"$srcdir/check_caches_clean.sh\"\n    if [ -n \"$(find / -type f -name pytools_checks)\" ]; then\n        echo \"pytools_checks detected, should have been removed from docker build\"\n        exit 1\n    fi\nfi\n"
  },
  {
    "path": "checks/check_docker_compose.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-06 13:43:32 +0000 (Mon, 06 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -type f -iname '*docker-compose*.y*ml' -o -type f -ipath '*/docker-compose/*.y*ml' | sort)\"\n\nif [ -z \"$filelist\" ]; then\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Docker Compose Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping docker-compose syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping docker-compose syntax checks\"\n    echo\nelse\n    if ! command -v docker-compose &>/dev/null; then\n        echo \"docker-compose not found in \\$PATH, not running syntax checks\"\n        return 0 &>/dev/null || exit 0\n    fi\n    docker-compose --version\n    echo\n    if is_bitbucket_ci; then\n        echo 'Docker compose version on BitBucket is too old to validate syntax, skipping...'\n        echo\n        exit 0\n    fi\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for docker_compose_path in $filelist; do\n        isExcluded \"$docker_compose_path\" && continue\n        printf \"%-${max_len}s \" \"$docker_compose_path:\"\n        docker_compose_dir=\"$(dirname \"$docker_compose_path\")\"\n        docker_compose_basename=\"${docker_compose_path##*/}\"\n        pushd \"$docker_compose_dir\" &>/dev/null\n        env=()\n        # if there is a matching .env file use it instead of the default, this will fill in missing variables\n        if [ -f \"${docker_compose_basename%.*}.env\" ]; then\n            env+=(--env-file \"${docker_compose_basename%.*}.env\")\n        fi\n        set +eo pipefail\n        output=\"$(docker-compose -f \"$docker_compose_basename\" ${env:+\"${env[@]}\"} config >/dev/null)\"\n        # shellcheck disable=SC2181\n        if [ $? -eq 0 ]; then\n            echo \"OK\"\n        else\n            echo \"FAILED\"\n            if [ -z \"${QUIET:-}\" ]; then\n                echo\n                echo \"$output\"\n                echo\n            fi\n            if [ -z \"${NOEXIT:-}\" ]; then\n                return 1 &>/dev/null || exit 1\n            fi\n        fi\n        set -eo pipefail\n        popd &>/dev/null\n    done\n    time_taken \"$start_time\"\n    section2 \"All docker-compose files passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_dockerfiles.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 16:14:15 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# use hadolint.yaml in same dir as this script unless there is a local $PWD/.hadolint.yaml present\nexport XDG_CONFIG_HOME=\"$srcdir\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -type f -name '*Dockerfile*' | sort)\"\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Dockerfile Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping Dockerfile syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping Dockerfile syntax checks\"\n    echo\nelse\n    if ! command -v hadolint &>/dev/null; then\n        echo \"hadolint not found in \\$PATH, not running Dockerfile syntax checks\"\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\n    hadolint --version\n    echo\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for x in $filelist; do\n        isExcluded \"$x\" && continue\n        printf \"%-${max_len}s \" \"$x:\"\n        set +eo pipefail\n        output=\"$(hadolint \"$x\")\"\n        # shellcheck disable=SC2181\n        if [ $? -eq 0 ]; then\n            echo \"OK\"\n        else\n            echo \"FAILED\"\n            if [ -z \"${QUIET:-}\" ]; then\n                echo\n                echo \"$output\"\n                echo\n            fi\n            if [ -z \"${NOEXIT:-}\" ]; then\n                # shellcheck disable=SC2317\n                return 1 &>/dev/null ||\n                exit 1\n            fi\n        fi\n        set -eo pipefail\n    done\n    time_taken \"$start_time\"\n    section2 \"All Dockerfiles passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_drone_yml.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-22 09:25:44 +0100 (Sat, 22 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\n#return 0 &>/dev/null || :\n#exit 0\n\nsection \"Drone CI Yaml Lint Check\"\n\nstart_time=\"$(start_timer)\"\n\n#if is_travis; then\n#    echo \"Running inside Drone CI, skipping lint check\"\nif is_inside_docker; then\n    echo \"Running inside Docker, skipping Drone CI yaml lint check\"\nelse\n    if type -P drone &>/dev/null; then\n        type -P drone\n        drone --version\n        echo\n        if is_CI; then\n            echo \"using drone from location: $(type -P drone)\"\n            set +e\n        fi\n        find . -name '.drone.yml' |\n        while read -r drone_yml; do\n            echo -n \"Validating $drone_yml => \"\n            drone lint \"$drone_yml\"\n            echo \"OK\"\n        done\n        set -e\n    else\n        echo \"WARNING: skipping Drone CI yaml check as 'drone' command not found in \\$PATH ($PATH)\"\n    fi\nfi\n\necho\ntime_taken \"$start_time\"\nsection2 \"Drone CI yaml validation succeeded\"\necho\n"
  },
  {
    "path": "checks/check_duplicate_dependencies.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2086\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:59:10 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking for duplicate package dependency requirements\"\n\nstart_time=\"$(start_timer)\"\n\npip_requirements_files=\"$(find . -maxdepth 2 -name requirements.txt)\"\n\nif [ -n \"$pip_requirements_files\" ]; then\n    echo \"Pip PyPI requirements files found: \"$pip_requirements_files\n    echo \"checking for duplicates\"\n    \"$srcdir/../python/python_find_duplicate_pip_requirements.sh\" $pip_requirements_files\n    echo\nfi\n\ncpan_requirements_files=\"$(find . -maxdepth 3 -name 'cpan-requirements*.txt')\"\n\nif [ -n \"$cpan_requirements_files\" ]; then\n    echo \"Perl CPAN requirements files found: \"$cpan_requirements_files\n    echo \"checking for duplicates\"\n    \"$srcdir/../perl/perl_find_duplicate_cpan_requirements.sh\" $cpan_requirements_files\n    echo\nfi\n\n#÷for pkg in apk deb rpm brew portage; do\n#÷    # don't check lib and pylib at the same time because they will have duplicates between them\n#÷    for lib in lib pylib; do\n#÷        requirements_files=\"$(find . -maxdepth 3 -name \"$pkg-packages*.txt\" | grep -ve \"/$lib/\" -e \"/bash-tools/\" -e '/pytools_checks/' || :)\"\n#÷\n#÷        if [ -n \"$requirements_files\" ]; then\n#÷            echo \"$pkg requirements files found: \"$requirements_files\n#÷            echo \"checking for duplicates\"\n#÷            \"$srcdir/../bin/find_duplicate_lines.sh\" $requirements_files\n#÷            echo\n#÷        fi\n#÷    done\n#÷done\n\n#\"$srcdir/check_duplicate_packages.sh\"\n\ntime_taken \"$start_time\"\nsection2 \"No duplicate requirements found\"\necho\n"
  },
  {
    "path": "checks/check_duplicate_packages.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:49:47 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find duplicate RPM / Deb / Apk / Brew / Portage packages across files\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<package_list_files>]\"\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nsection \"Duplicate Packages Check\"\n\nstart_time=\"$(start_timer)\"\n\nfound=0\n\nfind_dups(){\n    local duplicate_packages\n    # need word splitting for different files\n    # shellcheck disable=SC2086\n    duplicate_packages=\"$(sed 's/#.*//;\n         s/[<>=].*//;\n         s/^[[:space:]]*//;\n         s/[[:space:]]*$//;\n         /^[[:space:]]*$/d;' $package_files |\n        sort |\n        uniq -d\n    )\"\n    while read -r package; do\n        [ -n \"$package\" ] || continue\n        # need word splitting for different files\n        # shellcheck disable=SC2086\n        grep \"^${package}\\\\([[:space:]]\\\\|$\\\\)\" $package_files\n        ((found+=1))\n    done <<< \"$duplicate_packages\"\n\n}\n\nif [ -n \"$*\" ]; then\n    package_files=\"$*\"\n    find_dups \"$package_files\"\nelse\n    found_files=0\n    for x in rpm deb apk brew portage; do\n        package_files=\"$(find . -maxdepth 3 -name \"$x*-packages*.txt\" | grep -v desktop || :)\"\n        if [ -z \"$package_files\" ]; then\n            continue\n        fi\n        found_files=1\n        echo \"checking for duplicate $x packages\" >&2\n        find_dups \"$package_files\"\n        echo\n    done\n    if [ $found_files -eq 0 ]; then\n        #usage \"No package files found, please specify explicit path to *-packages*.txt\"\n        warn \"No package files found, please specify explicit path to *-packages*.txt\"\n    fi\nfi\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Passed - no duplicate packages found\"\necho\n"
  },
  {
    "path": "checks/check_git_commit_authors.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2015,SC1090,SC1091\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Fri Nov 1 19:04:26 2019 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Various Git log author commit checks against Name / Email / Domain inconsistencies to catch committing with the wrong or default user.name / user.email\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n. \"$srcdir/lib/utils.sh\"\n\n. \"$srcdir/.bash.d/functions.sh\"\n\n. \"$srcdir/.bash.d/git.sh\"\n\nsection \"Git Author Name + Email Checks\"\n\n# XXX: some 3rd party services have changed their names - exclude them from tripping this check here\nignored_names=\"\nsnyk bot\n\"\n\nignored_emails=\"\n@pyup.io$\n@snyk.io$\n\"\n\nif ! isGit .; then\n    echo \"Running from a non-git directory '$PWD' - skipping author + email checks as git log is not available\"\n    echo\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\n#author_name=\"${GIT_AUTHOR_NAME:-${USER:-`whoami`}}\"\n#author_email=\"${GIT_AUTHOR_EMAIL:-${EMAIL:-$(whoami)@$(hostname -f | sed 's/.*@//')}}\"\n#user_name=\"$(git config -l | awk -F= '/user.name/{print $2}')\"\n#user_email=\"$(git config -l | awk -F= '/user.email/{print $2}')\"\n\n# used to check the last N commits (default: last 100 commits, override with $GIT_COMMIT_HISTORY_DEPTH)\n# but figured this is fast enough even on my biggest repo with 10,000 commmits it still returns in a second\n#commit_count=\"${GIT_COMMIT_HISTORY_DEPTH:-100}\"\n\nexitcode=0\n\ngit_log(){\n    #local opts=\"\"\n    #if [[ \"$commit_count\" =~ ^[0-9]+$ ]] &&\n    #   [ \"$commit_count\" -gt 1 ]; then\n    #    opts=\"-n $commit_count\"\n    #fi\n    # need to split opts\n    # shellcheck disable=SC2086\n    #git log --all $opts \"$@\"\n    git log --all \"$@\"\n}\n\ngit_log_names(){\n    # shellcheck disable=SC2119\n    git_log --pretty=format:\"%an\" | toLower | trim | perl -p -e 's/[\\h-]+/ /g' | sort -u\n}\n\ngit_log_emails(){\n    # shellcheck disable=SC2119\n    git_log --pretty=format:\"%ae\" | toLower | trim | sort -u\n}\n\ngit_log_names_emails(){\n    # shellcheck disable=SC2119\n    git_log --pretty=format:\"%an %ae\" | toLower | trim | perl -p -e 's/\\h+/ /g' | sort -u\n}\n\n#names=\"$(git_log_names)\"\nemails=\"$(git_log_emails)\"\nnames_emails=\"$(git_log_names_emails)\"\n\nfilter_ignored_names(){\n    local regex\n    for name in $ignored_names; do\n        regex+=\"|$name\"\n    done\n    regex=\"${regex#|}\"\n    grep -Ev \"$ignored_names\" || :\n}\n\nfilter_ignored_emails(){\n    local regex\n    for email in $ignored_emails; do\n        regex+=\"|$email\"\n    done\n    regex=\"${regex#|}\"\n    grep -Ev \"$ignored_emails\" || :\n}\n\ncheck_multiple_names_per_email(){\n    local err=0\n    emails=\"${emails:-$(git_log_emails)}\"\n    names_emails=\"${names_emails:-(git_log_names_emails)}\"\n    for email in $emails; do\n        #names_for_same_email=\"$(grep \"[[:space:]]$email$\" <<< \"$names_emails\" | perl -p -e 's/\\s\\S*?$//')\"\n        names_for_same_email=\"$(grep -Ei \"[[:space:]]$email$\" <<< \"$names_emails\" |\n                                filter_ignored_emails |\n                                normalize_spaces |\n                                remove_last_column |\n                                sort -u || :)\"\n        # don't quote otherwise have to trim wc output for comparison\n        # shellcheck disable=SC2046\n        if [ $(wc -l <<< \"$names_for_same_email\") -eq 1 ]; then\n            names_for_same_email=\"\"\n        fi\n        check_error \"$names_for_same_email\" \"different names found for same email address '$email'! (misconfigured git config user.name?)\" || err=1\n    done\n    if [ $err -eq 0 ]; then\n        echo \"OK: no differing names for each committed email address\"\n    fi\n}\n\ncheck_multiple_emails_per_name(){\n    local err=0\n    names=\"$(git_log_names | filter_ignored_names)\"\n    # generalize the spaces/dots/dashes to an ERE format regex in case the same name has changed slightly\n    names=\"$(perl -p -e 's/[\\s.-]+/[[:space:].-]*/' <<< \"$names\" | sort -u)\"\n    names_emails=\"${names_emails:-(git_log_names_emails)}\"\n    while read -r name_regex; do\n        # $email_regex defined in lib\n        # shellcheck disable=SC2154\n        emails_for_same_name=\"$(grep -Ei \"^${name_regex}[[:space:]]+$email_regex\" <<< \"$names_emails\" | awk '{print $NF}' | sort -u || :)\"\n        # don't quote otherwise have to trim wc output for comparison\n        # shellcheck disable=SC2046\n        if [ $(wc -l <<< \"$emails_for_same_name\") -eq 1 ]; then\n            emails_for_same_name=\"\"\n        fi\n        check_error \"$emails_for_same_name\" \"different email addresses committed for the same user name '$name_regex'! (misconfigured git config user.email?)\" || err=1\n    done <<< \"$names\"\n    if [ $err -eq 0 ]; then\n        echo \"OK: no differing email addresses committed for each committed user name\"\n    fi\n}\n\ncheck_duplicate_email_prefixes(){\n    emails=\"${emails:-$(git_log_emails)}\"\n    # need to use sed not built-in variable replacement in order to handle multi-line emails\n    # shellcheck disable=SC2001\n    duplicate_email_prefixes=\"$(sed 's/@.*$//;s/\\.//g' <<< \"$emails\" | sort | uniq -d || :)\"\n    # email prefixes normalize hari.sekhon => harisekhon since email accounts like gmail treat them the same, so remember duplicates for harisekhon may include hari.sekhon\n    check_error \"$duplicate_email_prefixes\" \"duplicate email prefixes detected (misconfigured domain name in git user.email?)\" &&\n    echo \"OK: no duplicate email prefixes detected\" || :\n}\n\ncheck_root_committed(){\n    names_emails=\"${names_emails:-$(git_log_names_emails)}\"\n    root_detected=\"$(grep -i '\\<root\\>' <<< \"$names_emails\" || :)\"\n    check_error \"$root_detected\" \"root commits detected\" &&\n    echo \"OK: no root commits detected\" || :\n}\n\ncheck_emails_without_domains(){\n    emails=\"${emails:-$(git_log_emails)}\"\n    # need to use sed not built-in variable replacement in order to handle multi-line emails\n    # shellcheck disable=SC2001\n    domains=\"$(sed 's/.*@//' <<< \"$emails\" | sort -u)\"\n    # $domain_regex defined in lib\n    # shellcheck disable=SC2154\n    non_domain_suffixes=\"$(grep -Ev \"^$domain_regex$\" <<< \"$domains\" || :)\"\n    check_error \"$non_domain_suffixes\" \"non-domain email suffixes detected (misconfigured git user.email defaulting to hostname?)\" &&\n    echo \"OK: no non-domain email suffixes detected\" || :\n}\n\ncheck_single_word_author_names(){\n    names=\"$(git_log_names)\"\n    single_word_author_names=\"$(awk '{if(NF == 1) print $0}' <<< \"$names\" | sort -u)\"\n    check_error \"$single_word_author_names\" \"single word author names detected (misconfigured git user.name?)\" &&\n    echo \"OK: no single word author names detected\" || :\n}\n\ncheck_error(){\n    if [ -n \"$1\" ]; then\n        echo \"WARNING: $2\"\n        echo\n        echo \"$1\"\n        echo\n        exitcode=1\n        return 1\n    fi\n}\n\nif isGit; then\n    check_multiple_names_per_email\n    check_multiple_emails_per_name\n    check_duplicate_email_prefixes\n    check_root_committed\n    check_emails_without_domains\n    # lots of my contrib authors have single word author names\n    # and so do 3rd party services like pyup-bot and ReadmeCritic\n    #check_single_word_author_names\nelse\n    echo \"Not a git repo, skipping...\"\nfi\n\ntime_taken \"$start_time\"\nif [ $exitcode -eq 0 ]; then\n    section2 \"All Git author name / email checks passed\"\nelse\n    section2 \"ERROR: some Git author name / email checks failed\"\nfi\necho\nexit $exitcode\n"
  },
  {
    "path": "checks/check_git_no_merge_remnants.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-06 14:09:39 +0000 (Mon, 06 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking no Git / Diff merge remnants\"\n\nif [ -n \"${1:-}\" ]; then\n    if ! [ -d \"$1\" ]; then\n        echo \"No such file or directory $1\"\n        exit 1\n    fi\n    pushd \"$1\"\nfi\n\nstart_time=\"$(start_timer)\"\n\nregex='^([<]<<<<<<|>>>>>>[>])'\n\necho \"searching for '$regex' under $PWD:\"\necho\n# slow, may scan filesystem containing large files - can waste minutes of time\n#if grep -IER \"$regex\" --devices=skip --exclude-dir={.git} . 2>/dev/null; then\nif git grep -IE \"$regex\" . 2>/dev/null; then\n    echo\n    echo \"FOUND Git / Diff merge remnants!\"\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"No git / diff merge remnants found\"\necho\n"
  },
  {
    "path": "checks/check_github_actions_workflow_injection.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-10 19:09:45 +0100 (Fri, 10 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks the GitHub Actions workflows in the given Git repo checkout don't have any obvious script injection risks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nsection \"GitHub Actions Script Injection Check\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\ngit_root=\"$(git_root)\"\n\nworkflow_dir=\"$git_root/.github/workflows\"\n\n# false positive\n# shellcheck disable=SC2016\nif git grep '^[[:space:]]\\+run:.*${{' \"$workflow_dir/\"*.yaml \"$workflow_dir/\"*.yml; then\n    echo\n    die \"WARNING: possible script injection vectors detected under '$workflow_dir'\"\nelse\n    section2 \"GitHub Actions script injection check passed\"\nfi\n"
  },
  {
    "path": "checks/check_github_actions_workflows_without_checkout.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-21 18:48:21 +0000 (Mon, 21 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds GitHub Actions workflows without checkouts by scanning the .github/workflows/ yaml files\n\nThis is often an error and can be dangerous for security scanning tools as they then fail to scan the real code or generate any security alerts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ndir=\"${1:-}\"\n\nif is_blank \"$dir\"; then\n    if is_in_git_repo; then\n        git_root=\"$(git_root)\"\n        dir=\"$git_root/.github/workflows\"\n    else\n        dir='.'\n    fi\nfi\n\nif ! [[ \"$dir\" =~ /.github/workflows/?$ ]]; then\n    dir+=\"/.github/workflows\"\nfi\n\nif ! [ -d \"$dir\" ]; then\n    die \"ERROR: directory not found: $dir\"\nfi\n\nfilelist=\"$(find \"$dir\" -type f -name '*.y*ml' | sort)\"\n\nsection 'GitHub Actions Workflows without checkout'\n\nstart_time=\"$(start_timer)\"\n\ncount=0\nfiles_without_checkout=\"\"\n\nwhile read -r filename; do\n    if ! [ -f \"$filename\" ]; then\n        die \"ERROR: file not found: $filename\"\n    fi\n    echo \"checking $filename\"\n    if ! grep -Eq '^[^#]*(checkout|clone)' \"$filename\" &&\n       ! grep -Eq '^[[:space:]]+uses:[[:space:]]*.+/.github/workflows/.+@.+' \"$filename\"; then\n        echo\n        echo \"WARNING: no checkout detected in: $filename\"\n        echo\n        files_without_checkout+=\"\n$filename\"\n        ((count+=1))\n    fi\ndone <<< \"$filelist\"\n\necho\nif [ $count -gt 0 ]; then\n    echo \"ERROR: $count files without checkout detected:\"\n    die \"$files_without_checkout\"\nfi\n\ntime_taken \"$start_time\"\nsection2 \"OK: no AWS credentials found in Git\"\necho\n"
  },
  {
    "path": "checks/check_github_codeowners.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-24 14:07:02 +0000 (Thu, 24 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks for any errors in the current or given GitHub repo's committed CODEOWNERS file\n\nBy default checks the CODEOWNERS for the default branch, unless a second argument is given specifying another branch name\n\nRequires GitHub CLI and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo> <ref>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nowner_repo=\"${1:-}\"\nref=\"${2:-}\"\n\nsection 'GitHub CodeOwners check'\n\nstart_time=\"$(start_timer)\"\n\nif is_blank \"$owner_repo\"; then\n    timestamp \"repo not specified, determining self\"\n    owner_repo=\"$(gh api '/repos/{owner}/{repo}' | jq -r .full_name)\"\n    timestamp \"determined repo to be '$owner_repo'\"\n    echo >&2\nfi\n\nif is_blank \"$ref\"; then\n    timestamp \"ref not specified, attempting to determine from current branch\"\n    #ref=\"$(mybranch)\"\n    #timestamp \"determined current branch to be '$ref'\"\n    ref=\"$(gh api \"/repos/$owner_repo\" | jq -r .default_branch)\"\n    timestamp \"determined default branch to be '$ref'\"\n    echo >&2\nfi\n\nurl=\"/repos/$owner_repo/codeowners/errors\"\n\nif ! is_blank \"$ref\"; then\n    url+=\"?ref=$ref\"\nfi\n\ntimestamp \"Checking for CODEOWNERS errors in ref '$ref' via the GitHub API\"\ndata=\"$(gh api \"$url\")\"\n\nerror_count=\"$(jq -r '.error | length' <<< \"$data\")\"\n\nif [ \"$error_count\" -gt 0 ]; then\n    echo \"Error: CODEOWNERS file errors detected for repo '$owner_repo'\"\n    echo\n    jq <<< \"$data\"\n\nfi\n\ntime_taken \"$start_time\"\nsection2 \"OK: no CODEOWNERS errors found for repo '$owner_repo'\"\necho\n"
  },
  {
    "path": "checks/check_gitlab_ci_yml.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 19:34:25 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nyamls=\"$(find \"${1:-.}\" -name .gitlab-ci.yml)\"\n\nif [ -z \"$yamls\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"GitLab CI Yaml Lint Check\"\n\nif [ -z \"${GITLAB_TOKEN:-}\" ]; then\n    echo \"WARNING: \\$GITLAB_TOKEN not found in environment and this API endpoint now requires authentication, skipping...\" >&2\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\nwhile read -r yaml; do\n    printf 'Validating %s:\\t' \"$yaml\"\n    \"$srcdir/../gitlab/gitlab_validate_ci_yaml.sh\" \"$yaml\"\ndone <<< \"$yamls\"\n\ntime_taken \"$start_time\"\nsection2 \"GitLab CI yaml validation succeeded\"\necho\n"
  },
  {
    "path": "checks/check_gradle_build.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-07-25 00:17:36 +0100 (Mon, 25 Jul 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nbuild_files=\"$(find \"${1:-.}\" -name build.gradle)\"\n\nif [ -z \"$build_files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"G r a d l e\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P gradle &>/dev/null; then\n    type -P gradle\n    gradle --version\n    echo\n    grep -v '/build/' <<< \"$build_files\" |\n    sort |\n    while read -r build_gradle; do\n        echo \"Validating $build_gradle\"\n        #gradle -b \"$build_gradle\" -m clean build || exit $?\n        # Gradle 8 doesn't let you specify -b, expects build.gradle\n        dir=\"$(dirname \"$build_gradle\")\"\n        cd \"$dir\"\n        #gradle -b \"$build_gradle\" -m clean build || exit $?\n        gradle -m clean build || exit $?\n    done\nelse\n    echo \"Gradle not found in \\$PATH, skipping gradle checks\"\nfi\n\ntime_taken \"$start_time\"\nsection2 \"All Gradle builds passed dry run checks\"\necho\n"
  },
  {
    "path": "checks/check_groovyc.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-06 18:34:01 +0000 (Thu, 06 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecurses a given directory tree or \\$PWD, finding all Groovy files and validating them using 'groovyc'\n\nUseful for doing basic linting on simple self-contained Groovy scripts such as Jenkins Shared Libraries\n\"\n\nhelp_usage \"$@\"\n\necho \"Checking for Groovy files\"\nfilelist=\"$(for directory in \"${@:-.}\"; do find \"$directory\" -type f -iname '*.groovy'; done | sort -u)\"\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"G r o o v y\"\n\nif ! type -P groovyc &>/dev/null; then\n    echo \"WARNING: groovyc not found in \\$PATH, skipping Groovy checks\"\n    echo\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\ngroovy --version\necho\n\ncheck_groovyc(){\n    local filename=\"$1\"\n    shift || :\n    # this requires far too many function exports for all called CI functions\n    #isExcluded \"$filename\" && return 0\n    if grep -E \"^@NonCPS\" \"$filename\"; then\n        # can't test this Jenkins annotation without 'import com.cloudbees.groovy.cps.NonCPS'\n        return\n    fi\n    echo \"groovyc $filename $*\" >&2\n    classfile_base=\"${filename##*/}\"\n    classfile_base=\"${classfile_base%.groovy}\"\n    # doens't stop class files being left behind in script $PWD, not directory containing \"$filename\"\n    #if ! groovyc --temp /tmp \"$filename\" \"$@\" >&2; then\n    if ! groovyc \"$filename\" \"$@\" >&2; then\n        rm -f -- \"$classfile_base\"*.class\n        echo 1\n        exit 1\n    fi\n    rm -f -- \"$classfile_base\"*.class\n}\n\necho \"building file list\" >&2\ntests=\"$(\n    while read -r filename; do\n        echo \"check_groovyc \\\"$filename\\\"\"\n    done <<< \"$filelist\"\n)\"\n\ncpu_count=\"$(cpu_count)\"\nmultiplier=1  # doesn't get faster increasing this in tests, perhaps even slightly slower due to context switching\nparallelism=\"$((cpu_count * multiplier))\"\n# 4 cores is counted as 8 due to hyperthreading, but this isn't any faster\n#parallelism=\"$((cpu_count / 2))\"\n\necho \"found $cpu_count cores, running $parallelism parallel jobs\"\necho\n\n# export functions to use in parallel\nexport -f check_groovyc\nexport SHELL=/bin/bash  # Debian docker container doesn't set this and defaults to sh, failing to find exported function\n\nset +eo pipefail\ntally=\"$(parallel -j \"$parallelism\" <<< \"$tests\")\"\nexit_code=$?\nset -eo pipefail\n\ncount=\"$(awk '{sum+=$1} END{print sum}' <<< \"$tally\")\"\n\necho >&2\ntime_taken \"$start_time\"\necho >&2\n\nif [ $exit_code -eq 0 ]; then\n    section2 \"Groovy checks passed\"\nelse\n    echo \"ERROR: $count broken groovy files detected!\" >&2\n    echo >&2\n    section2 \"Groovy checks FAILED\"\n    exit 1\nfi\n"
  },
  {
    "path": "checks/check_internet.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-16 15:44:24 -0500 (Fri, 16 Jan 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks the internet is available by multiple tests\n\nWaits for the internet to become available before returning\n\nUseful to run in a blocking latch wait fashion in scripts to ensure the internet is available\nbefore running a big operation like my Spotify Playlists API backups\n\nTests:\n\n- Local Gateway IP is configured (Wifi DHCP has succeeded or we have static details configured)\n- Gateway IP is reachable (ping)\n  - now optional informational, progresses regardless now as some hotel wifi gateway did not return pings\n  even though the end-to-end internet connection was up (router was obviously just set to not respond to ICMP)\n- Public IP is reachable (ping to known major public IP 1.1.1.1)\n- DNS resolution is working (resolves google.com)\n- Public Domain is reachable (ping to google.com)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\nSECONDS=0\n\npublic_ip=\"1.1.1.1\"\ndomain=\"google.com\"\n\nping_count=1\nping_timeout=2\nsleep_seconds=2\n\nget_gateway() {\n    \"$srcdir/../bin/network_gateway.sh\" 2>/dev/null || return 1\n}\n\ncheck_gateway() {\n    gateway_ip=\"$(get_gateway)\" || return 1\n    if ! [ -n \"$gateway_ip\" ]; then\n        timestamp \"FAIL: no Gateway IP detected yet...\"\n        return 1\n    fi\n\n    if ping -c \"$ping_count\" -W \"$ping_timeout\" \"$gateway_ip\" &>/dev/null; then\n        timestamp \"OK: Gateway IP reachable\"\n    else\n        timestamp \"FAIL: Gateway IP '$gateway_ip' unreachable (skipping as some routers are configured to just not respond to ping)\"\n        return 1\n    fi\n}\n\ncheck_public_ip() {\n    if ping -c \"$ping_count\" -W \"$ping_timeout\" \"$public_ip\" &>/dev/null; then\n        timestamp \"OK: Public IP reachable\"\n    else\n        timestamp \"FAIL: Public IP '$public_ip' unreachable\"\n        return\n    fi\n}\n\ncheck_dns() {\n    if type -P getent &>/dev/null; then\n        getent hosts \"$domain\" &>/dev/null\n    elif type -P dig &>/dev/null; then\n        dig +short \"$domain\" &>/dev/null\n    elif type -P nslookup &>/dev/null; then\n        nslookup \"$domain\" &>/dev/null\n    else\n        timestamp \"FAIL: DNS domain '$domain' not resolved\"\n        return 1\n    fi\n    timestamp \"OK: DNS resolved\"\n}\n\ncheck_domain_ping() {\n    if ping -c \"$ping_count\" -W \"$ping_timeout\" \"$domain\" &>/dev/null; then\n        timestamp \"OK: Domain IP reachable\"\n    else\n        timestamp \"FAIL: Domain IP unreachable\"\n        return 1\n    fi\n}\n\ntimestamp \"Detecting Default Gateway IP...\"\nwhile :; do\n    gateway_ip=$(get_gateway)\n    if ! is_blank \"$gateway_ip\"; then\n        break\n    else\n        timestamp \"FAIL: no Gateway IP available yet...\"\n    fi\n    sleep \"$sleep_seconds\"\ndone\n\ntimestamp \"Checking Gateway IP available: $gateway_ip\"\n#while ! check_gateway; do\n# no point wasting 5 tries when the hotel wifi will always fail, it just slows down dependent scripts\n#for ((i=0; i< 5; i++)); do\n#    check_gateway && break\n#    sleep \"$sleep_seconds\"\n#done\ncheck_gateway || :\n\ntimestamp \"Checking Public IP available: $public_ip\"\nwhile ! check_public_ip; do\n    sleep \"$sleep_seconds\"\ndone\n\ntimestamp \"Checking DNS resolution to well known domain: $domain\"\nwhile ! check_dns; do\n    sleep \"$sleep_seconds\"\ndone\n\ntimestamp \"Checking Domain IP reachable: $domain\"\nwhile ! check_domain_ping; do\n    sleep \"$sleep_seconds\"\ndone\n\ntimestamp \"Internet Connection OK within $SECONDS secs\"\nexit 0\n"
  },
  {
    "path": "checks/check_javac.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-06 18:34:01 +0000 (Thu, 06 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecurses a given directory tree or \\$PWD, finding all Java files and validating them using 'javac'\n\nProbably overly simplistic for a Java project, check a real linter instead\n\"\n\nhelp_usage \"$@\"\n\ndirectory=\"${1:-.}\"\nshift ||:\n\nfilelist=\"$(find \"$directory\" -type f -iname '*.java' | sort)\"\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"J a v a\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P javac &>/dev/null; then\n    type -P javac\n    javac -version\n    echo\n    while read -r filename; do\n        isExcluded \"$filename\" && continue\n        echo \"javac $filename $*\"\n        javac \"$filename\" \"$@\"\n    done <<< \"$filelist\"\nelse\n    echo \"WARNING: javac not found in \\$PATH, skipping Java checks\"\n    exit 0\nfi\n\necho\ntime_taken \"$start_time\"\nsection2 \"Java checks passed\"\necho\n"
  },
  {
    "path": "checks/check_javascript_eslint.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-19 19:44:49 +0400 (Tue, 19 Nov 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecurses a given directory tree or \\$PWD, finding all Javascript files and validating them using EsLint\n\"\n\nhelp_usage \"$@\"\n\ndirectory=\"${1:-.}\"\nshift ||:\n\nfiles=\"$(find \"$directory\" -iname '*.js')\"\n\nif [ -z \"$files\" ]; then\n    # this trick allows importing or calling as script\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null || :\n    # shellcheck disable=SC2317\n    exit 0\nfi\n\nsection \"E s L i n t\"\n\nstart_time=\"$(start_timer)\"\n\nif ! type -P eslint &>/dev/null &&\n    type -P npm; then\n    npm install eslint\nfi\n\nif type -P eslint &>/dev/null; then\n    type -P eslint\n    eslint --version\n    echo\n    while read -r filename; do\n        isExcluded \"$filename\" && continue\n        echo \"eslint $filename $*\"\n        eslint \"$filename\" \"$@\"\n    done <<< \"$files\"\n    hr; echo\nfi\n\ntime_taken \"$start_time\"\nsection2 \"EsLint checks passed\"\necho\n"
  },
  {
    "path": "checks/check_jenkinsfiles.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 11:52:44 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nValidates Jenkinsfiles in the directory given as an argument (defaults to \\$PWD) using the Jenkins CLI\n\nRequires a live Jenkins server so not called automatically in the CI framework of this repo\nunless running within Jenkins CI job or the following environment variables are set:\n\nIf running manually then these environment variables must be set:\n\n\\$JENKINS_URL (default: http://localhost:8080)\n        or\n    \\$JENKINS_HOST (default: localhost)\n        and\n    \\$JENKINS_PORT (default: 8080)\n\n\\$JENKINS_USER_ID   / \\$JENKINS_USER\n\\$JENKINS_API_TOKEN / \\$JENKINS_TOKEN /\\$JENKINS_PASSWORD\n\nOnly finds and checks files that match the name glob '*Jenkinsfile*' in the directory paths given, or under the current directory tree if no dirs are specified as args\n\nLimitation: the validator doesn't recognized parameterized pipelines imported via a Jenkins Shared Library. Such valid Jenkinsfiles fail validation with this error: \\\"did not contain the 'pipeline' step'\\\"\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<Jenkinsfiles_dirs>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\njenkinsfiles=()\n\nfor arg in \"${@:-.}\"; do\n    if [ -d \"$arg\" ]; then\n        jenkinsfiles+=( \"$(find \"${1:-.}\" -maxdepth 3 -name '*Jenkinsfile*' | grep -v -e '\\.groovy$')\" )\n    else\n        jenkinsfiles+=(\"$arg\")\n    fi\ndone\n\nif [ -z \"${jenkinsfiles[*]}\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"J e n k i n s f i l e s\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -z \"${JENKINS_URL:-}\" ]; then\n    export JENKINS_URL=\"${JENKINS_HTTPS:-http}://${JENKINS_HOST:-localhost}:${JENKINS_PORT:-8080}\"\nfi\n\nJENKINS_URL=\"${JENKINS_URL%%/}\"\n\n#crumb=\"$(\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"$JENKINS_URL/crumbIssuer/api/json\" | jq -r '.crumb')\"\n\necho \"Validating Jenkinsfiles:\"\necho\nwhile read -r jenkinsfile; do\n    echo -n \"$jenkinsfile => \"\n    #\"$srcdir/../bin/curl_auth.sh\" \"$JENKINS_URL/pipeline-model-converter/validate\" -sS --fail -X POST -F \"jenkinsfile=<Jenkinsfile\" -H \"Jenkins-Crumb: $crumb\"\n    #\"$srcdir/jenkins_api.sh\" \"/pipeline-model-converter/validate\" -X POST -F \"jenkinsfile=<Jenkinsfile\"\n    #\"$srcdir/jenkins_api.sh\" \"/pipeline-model-converter/validate\" -X POST -F \"jenkinsfile=<$jenkinsfile\"\n    # 'export JENKINS_CLI_ARGS=-webSocket' is needed if Jenkins is behind a reverse proxy such as Kubernetes Ingress, otherwise Jenkins CLI hangs\n    \"$srcdir/../jenkins/jenkins_cli.sh\" declarative-linter < \"$jenkinsfile\"\ndone <<< \"$jenkinsfiles\"\n\ntime_taken \"$start_time\"\nsection2 \"Jenkinsfile validation SUCCEEDED\"\necho\n"
  },
  {
    "path": "checks/check_json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 17:18:03 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -type f -name '*.json' | sort)\"\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"JSON Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping JSON syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping JSON syntax checks\"\n    echo\nelse\n    if ! command -v jsonlint &>/dev/null; then\n        echo \"jsonlint not found in \\$PATH, not running JSON syntax checks\"\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\n    type -P jsonlint\n    printf \"version: \"\n    jsonlint --version || :\n    echo\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for x in $filelist; do\n        isExcluded \"$x\" && continue\n        [[ \"$x\" =~ multirecord ]] && continue\n        printf \"%-${max_len}s \" \"$x:\"\n        set +eo pipefail\n        output=\"$(jsonlint \"$x\")\"\n        # shellcheck disable=SC2181\n        if [ $? -eq 0 ]; then\n            echo \"OK\"\n        else\n            echo \"FAILED\"\n            if [ -z \"${QUIET:-}\" ]; then\n                echo\n                echo \"$output\"\n                echo\n            fi\n            if [ -z \"${NOEXIT:-}\" ]; then\n                # shellcheck disable=SC2317\n                return 1 &>/dev/null ||\n                exit 1\n            fi\n        fi\n        set -eo pipefail\n    done\n    time_taken \"$start_time\"\n    section2 \"All JSON files passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_kubernetes_yaml.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-12 11:57:37 +0100 (Thu, 12 May 2022)\n#\n#  https://github.com/HariSekhon/Nagios-Plugins\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks one or more Kubernetes yaml files for linting and schema issues\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"file1.yaml [file2.yaml file3.yaml ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor filename in \"$@\"; do\n    # dereference symlinks, lib/utils.sh uses greadlink if on mac\n    filename=\"$(readlink -m \"$filename\")\"\n\n    \"$srcdir/check_yaml.sh\" \"$filename\"\n\n    if type -P datree &>/dev/null; then\n        # only relevant if named exactly kustomization.yaml\n        if [[ \"$filename\" =~ ^(.*/)kustomization.ya?ml$ ]]; then\n            section \"Datree Kubernetes Kustomize Check\"\n            datree kustomize test \"$(dirname \"$filename\")\"\n        else\n            section \"Datree Kubernetes Check\"\n            datree test --only-k8s-files --ignore-missing-schemas \"$filename\"\n        fi\n        echo\n        section2 \"Datree Kubernetes Check Passed\"\n    fi\ndone\n"
  },
  {
    "path": "checks/check_license_exists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-08 11:10:28 +0000 (Tue, 08 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks that there is a LICENSE or LICENSE.txt at the top of a git repo and that it is not empty\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nsection2 'License file check'\n\nif ! git_root=\"$(git_root)\"; then\n    echo 'Not a git repository, skipping...'\n    echo\n    exit 0\nfi\n\ncd \"$git_root\"\n\nfor x in LICENSE LICENSE.txt; do\n    if [ -s \"$x\" ]; then\n        echo \"OK: $x file found\"\n        exit 0\n    elif [ -f \"$x\" ]; then\n        echo \"WARNING: $x file found but it is empty! \"\n        exit 1\n    fi\ndone\n\necho 'WARNING: LICENSE file not found!'\nexit 1\n"
  },
  {
    "path": "checks/check_makefiles.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 14:07:17 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nmakefiles=\"$(find \"${1:-.}\" -maxdepth 2 -name Makefile -o -name Makefile.in)\"\n\nif [ -z \"$makefiles\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"M a k e\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P make &>/dev/null; then\n    type -P make\n    make --version\n    echo\n    while read -r makefile; do\n        pushd \"$(dirname \"$makefile\")\" >/dev/null\n        echo \"Validating $makefile\"\n        makefile=\"${makefile##*/}\"\n        if grep '#@' \"$makefile\"; then\n            echo \"WARNING: commented lines still visible in $PWD/$makefile\"\n        fi\n        while read -r target; do\n            if [[ \"$target\" =~ wc-?(code|scripts) ]]; then\n                continue\n            fi\n            if ! make -f \"$makefile\" --warn-undefined-variables -n \"$target\" >/dev/null; then\n                echo \"Makefile validation FAILED\"\n                exit 1\n            fi\n        done < <(\n            grep '^[[:alnum:]]\\+:' \"$makefile\" |\n            sort -u |\n            sed 's/:.*$//'\n        ) || :  # without this if no targets are found like in Dockerfiles/jython (which is all inherited) and this will fail set -e silently error out and not check the rest of the Makefiles\n        popd >/dev/null\n    done <<< \"$makefiles\"\nelse\n    echo \"WARNING: 'make' is not installed, skipping...\"\n    echo\n    exit 0\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Makefile validation SUCCEEDED\"\necho\n"
  },
  {
    "path": "checks/check_maven_pom.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-06-30 14:46:43 +0100 (Thu, 30 Jun 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\npoms=\"$(find \"${1:-.}\" -name \"pom.xml\")\"\n\nif [ -z \"$poms\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"M a v e n\"\n\nstart_time=\"$(start_timer)\"\n\nopts=()\n\nif is_CI; then\n    opts+=(-B)\nfi\n\nif type -P mvn &>/dev/null; then\n    type -P mvn\n    mvn --version\n    echo\n    grep -v '/target/' <<< \"$poms\" |\n    while read -r pom; do\n        echo \"Validating $pom\"\n        mvn validate ${opts:+\"${opts[@]}\"} -f \"$pom\" || exit $?\n        echo\n    done\nelse\n    echo \"Maven not found in \\$PATH, skipping maven pom checks\"\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Maven pom checks passed\"\necho\n"
  },
  {
    "path": "checks/check_no_suid_guid_shell_scripts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-12-28 12:24:21 +0000 (Thu, 28 Dec 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking there are no SUID / GUID shell scripts\"\n\nstart_time=\"$(start_timer)\"\n\n# shellcheck source=lib/excluded.sh\n#. \"$srcdir/lib/excluded.sh\"\n\nif is_mac; then\n    find(){\n        gfind \"$@\"\n    }\nfi\n\nsuid_guid_scripts=\"$(\n    find \"${1:-.}\" -type f -iname '*.sh' -perm /4000 -o \\\n                   -type f -iname '*.sh' -perm /2000 |\n    sort |\n    tee /dev/stderr\n)\"\n\nnum_suid_guid=\"$(sed '/^[[:space:]]*$/d' <<< \"$suid_guid_scripts\" | wc -l | sed 's/[[:space:]]*//')\"\n\nif [ \"$num_suid_guid\" -gt 0 ]; then\n    echo\n    echo \"$num_suid_guid files with suid / guid detected!\"\n    # allow to source, ignore if not sourced\n    # shellcheck disable=SC2317\n    return 1 &>/dev/null || :\n    # exit otherwise\n    # shellcheck disable=SC2317\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"SUID / GUID check passed\"\necho\n"
  },
  {
    "path": "checks/check_no_tabs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-12-28 12:24:21 +0000 (Thu, 28 Dec 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking for Tabs (rather than Spaces)\"\n\nstart_time=\"$(start_timer)\"\n\n# shellcheck source=lib/excluded.sh\n. \"$srcdir/lib/excluded.sh\"\n\nprogress_char='-'\n[ -n \"${DEBUG:-}\" ] && progress_char=''\n\nfiles_with_tabs=0\nfor filename in $(find \"${1:-.}\" -type f | grep -Evf \"$srcdir/../resources/whitespace_ignore.txt\" -f \"$srcdir/../resources/tabs_ignore.txt\" | sort); do\n    isExcluded \"$filename\" && continue\n    [[ \"$filename\" =~ .*/check_(no_tabs|whitespace).sh$|.terminal$ ]] && continue\n    printf \"%s\" \"$progress_char\"\n    # \\t aren't working inside character classes for some reason, embedding literal tabs instead\n    output=\"$(grep -EHn '\t' \"$filename\" || :)\"\n    if [ -n \"$output\" ]; then\n        echo\n        echo \"$output\"\n        #let files_with_tabs+=1\n        ((files_with_tabs + 1))\n    fi\ndone\necho\nif [ $files_with_tabs -gt 0 ]; then\n    echo \"$files_with_tabs files with tabs detected!\"\n    # shellcheck disable=SC2317\n    return 1 &>/dev/null ||\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Tabs check passed\"\necho\n"
  },
  {
    "path": "checks/check_perl_syntax.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nset +o pipefail\nfilelist=\"$(find \"${1:-.}\" -maxdepth 2 -type f -iname '*.pl' -o \\\n                                       -type f -iname '*.pm' -o \\\n                                       -type f -iname '*.t' |\n            grep -v /templates/ | sort)\"\nset -o pipefail\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Perl Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif ! type -P perl &>/dev/null; then\n    echo \"Perl is not installed, skipping checks\"\n    exit 0\nfi\n\ntype -P perl\nperl --version\necho\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping perl syntax checks\"\n    echo\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping perl syntax checks\"\n    echo\nelse\n    max_len=0\n    for x in $filelist; do\n        if [ \"${#x}\" -gt \"$max_len\" ]; then\n            max_len=\"${#x}\"\n        fi\n    done\n    # to account for the semi colon\n    ((max_len + 1))\n    for filename in $filelist; do\n        isExcluded \"$filename\" && continue\n        printf \"%-${max_len}s \" \"$filename:\"\n        #$perl -Tc $I_lib $filename\n        # -W too noisy\n        # -Mstrict - flags common DESCRIPTION / VERSION unscoped\n        # -Mdiagnostics\n        # shellcheck disable=SC2154\n        # $perl is set in perl.sh which is included from utils.sh\n        $perl -I . -Tc \"$filename\" 2>&1 | sed \"s,^$filename ,,\"\n    done\n    time_taken \"$start_time\"\n    section2 \"All Perl programs passed syntax check\"\nfi\necho\n"
  },
  {
    "path": "checks/check_python3_compat.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\npip install caniusepython3\necho \"Testing module dependencies for Python 3 compatibility\"\ncaniusepython3 -r requirements.txt\n"
  },
  {
    "path": "checks/check_python_asserts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2018-10-03 12:50:57 +0100 (Wed, 03 Oct 2018)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\n# maxdepth 2 to avoid recursing submodules which have their own checks\nfiles=\"$(find_python_jython_files . -maxdepth 2)\"\n\nif [ -z \"$files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Python - find and alert on any usage of assert outside of /test/\"\n\nstart_time=\"$(start_timer)\"\n\nfound=0\nwhile read -r filename; do\n    type isExcluded &>/dev/null && isExcluded \"$filename\" && echo -n '-' && continue\n    # exclude pytests\n    [[ \"$filename\" = ./test/* ]] && continue\n    echo -n '.'\n    if grep -E '^[[:space:]]+\\bassert\\b' \"$filename\"; then\n        echo\n        echo \"WARNING: $filename contains 'assert'!! This could be disabled at runtime by PYTHONOPTIMIZE=1 / -O / -OO and should not be used!! \"\n        found=1\n        #if ! is_CI; then\n        #    exit 1\n        #fi\n    fi\ndone <<< \"$files\"\n\ntime_taken \"$start_time\"\n\nif [ -n \"${WARN_ONLY:-}\" ]; then\n    section2 \"Python OK - assertions scan finished\"\nelse\n    if [ $found != 0 ]; then\n        exit 1\n    fi\n    section2 \"Python OK - no assertions found in normal code\"\nfi\n\necho\n"
  },
  {
    "path": "checks/check_python_exception_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-13 15:18:19 +0100 (Mon, 13 Nov 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\n# maxdepth 2 to avoid recursing submodules which have their own checks\nfiles=\"$(find_python_jython_files . -maxdepth 2)\"\n\nif [ -z \"$files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Python - finding any usage of exception pass\"\n\nstart_time=\"$(start_timer)\"\n\nwhile read -r filename; do\n    type isExcluded &>/dev/null && isExcluded \"$filename\" && echo -n '-' && continue\n    [[ \"$filename\" =~ /test/ ]] && echo -n '-' && continue\n    echo -n '.'\n    if grep -E -B3 -A1 '^[[:space:]]+\\bpass\\b' \"$filename\" | grep -Eq '^[^#]*\\bexcept\\b'; then\n        echo\n        grep -EHnB 5 -A1 '^[[:space:]]+\\bpass\\b' \"$filename\" | grep -E -5 '^[^#]*\\bexcept\\b'\n        echo\n        echo\n        echo \"WARNING: $filename contains 'pass'!! Check this code isn't being sloppy\"\n        #if ! is_CI; then\n        #    exit 1\n        #fi\n    fi\ndone <<< \"$files\"\n\ntime_taken \"$start_time\"\nsection2 \"Python OK - no except pass usage found\"\necho\n"
  },
  {
    "path": "checks/check_python_misc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-02-16 17:08:18 +0000 (Tue, 16 Feb 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\n# maxdepth 2 to avoid recursing submodules which have their own checks\nfiles=\"$(find_python_jython_files . -maxdepth 2)\"\n\nif [ -z \"$files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Python - finding miscellaneous code issues (calling quit(), self.self references)\"\n\nstart_time=\"$(start_timer)\"\n\nexitcode=0\n\ncheck(){\n    local msg=\"$1\"\n    local regex=\"$2\"\n    local filename=\"$3\"\n    if grep -Eq \"$regex\" \"$filename\"; then\n        echo\n        grep -E \"$regex\" \"$filename\"\n        echo\n        echo\n        echo \"ERROR: $filename contains $msg!! Typo?\"\n        exitcode=1\n    fi\n}\n\nwhile read -r filename; do\n    type isExcluded &>/dev/null && isExcluded \"$filename\" && echo -n '-' && continue\n    echo -n '.'\n    check \"quit() calls\" '^[^#\\.]*\\bquit\\b' \"$filename\"\n    check \"self.self references\" '^[^#]*\\bself\\.self\\b' \"$filename\"\n    #check \"'assert'!! This could be disabled at runtime by PYTHONOPTIMIZE=1 / -O / -OO and should not be used\" '^[[:space:]]+\\bassert\\b' \"$filename\"\ndone <<< \"$files\"\n\nif [ $exitcode != 0 ]; then\n    exit $exitcode\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Python OK - miscellaneous checks passed\"\necho\n"
  },
  {
    "path": "checks/check_python_pep8.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\n# maxdepth 2 to avoid recursing submodules which have their own checks\nfiles=\"$(find_python_jython_files . -maxdepth 2)\"\n\nif [ -z \"$files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Python PEP8 checking all Python / Jython files\"\n\nstart_time=\"$(start_timer)\"\n\n# $sudo defined in lib/util.sh\n# shellcheck disable=SC2154\ntype -P pep8 &>/dev/null || $sudo pip install pep8\ntype -P pep8\npep8 --version\necho\n\nwhile read -r filename; do\n    type isExcluded &>/dev/null && isExcluded \"$filename\" && continue\n    # E265 - spaces after # - I prefer no space it makes it easier to commented code vs actual comments\n    # E402 - import must be at top of file, but I like to do dynamic sys.path.append\n    pep8 --show-source --show-pep8 --max-line-length=120 --ignore=E402,E265 \"$filename\" | more\ndone <<< \"$files\"\n\ntime_taken \"$start_time\"\nsection2 \"Python PEP8 Completed Successfully\"\necho\n"
  },
  {
    "path": "checks/check_python_pylint.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecurses a given directory tree or \\$PWD, finding all Python and Jython files and validating them using PyLint\n\"\n\nhelp_usage \"$@\"\n\ndirectory=\"${1:-.}\"\nshift ||:\n\n# maxdepth 2 to avoid recursing submodules which have their own checks\nfiles=\"$(find_python_jython_files \"$directory\" -maxdepth 2)\"\n\nif [ -z \"$files\" ]; then\n    # this trick allows importing or calling as script\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null || :\n    # shellcheck disable=SC2317\n    exit 0\nfi\n\nsection \"P y L i n t\"\n\nstart_time=\"$(start_timer)\"\n\nif [ -n \"${NOPYLINT:-}\" ]; then\n    echo \"\\$NOPYLINT environment variable set, skipping PyLint error checks\"\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping PyLint error checks\"\nelse\n    if type -P pylint &>/dev/null; then\n        type -P pylint\n        pylint --version\n        echo\n        # Can't do this in one pass because pylint -E raises wrong-import-position when it doesn't individually and refuses to respect --disable\n        #prog_list=\"\n        while read -r filename; do\n            #echo \"checking if $filename is excluded\"\n            isExcluded \"$filename\" && continue\n            #echo \"added $filename for testing\"\n            #prog_list=\"$prog_list $filename\"\n            echo \"pylint -E $filename $*\"\n            pylint -E \"$filename\" \"$@\"\n        done <<< \"$files\"\n        #echo\n        #echo \"Checking for coding errors:\"\n        #echo\n        #echo \"pylint -E $prog_list\"\n        #echo\n        #pylint -E $prog_list\n        hr; echo\n    fi\nfi\n\ntime_taken \"$start_time\"\nsection2 \"PyLint checks passed\"\necho\n"
  },
  {
    "path": "checks/check_pytools.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-09-23 09:16:45 +0200 (Fri, 23 Sep 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir2=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir2/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir2/lib/docker.sh\"\n\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -z \"${PROJECT:-}\" ]; then\n    export PROJECT=bash-tools\nfi\n\nsection \"DevOps Python Tools Checks\"\n\n# must be up here before skipping check so that Dockerfiles can import it\nexport PATH=\"$PATH:$srcdir/pytools_checks:$srcdir/../pytools\"\n\nstart_time=\"$(start_timer)\"\n\nskip_checks=0\nif [ \"$PROJECT\" = \"pytools\" ]; then\n    echo \"detected running in pytools repo, skipping checks here as will be called in bash-tools/check_all.sh...\"\n    skip_checks=1\n#elif [ \"$PROJECT\" = \"Dockerfiles\" ]; then\n#    echo \"detected running in Dockerfiles repo, skipping checks here as will be called in bash-tools/check_all.sh...\"\n#    skip_checks=1\nelif is_inside_docker; then\n    echo \"detected running inside Docker, skipping pytools checks\"\n    skip_checks=1\nfi\n\nif [ $skip_checks = 1 ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\necho -n \"running on branch:  \"\ngit branch | grep '^\\*'\necho\necho \"running in dir:  $PWD\"\necho\n\nget_pytools(){\n    if [ -f \"$srcdir/pytools_checks/Makefile\" ]; then\n        pushd \"$srcdir/pytools_checks\"\n        NOJAVA=1 make update\n        popd\n    else\n        pushd \"$srcdir\"\n        rm -fr -- pytools_checks\n        git clone https://github.com/harisekhon/pytools pytools_checks\n        pushd pytools_checks\n        NOJAVA=1 make\n        popd\n        popd\n    fi\n}\n\nvalidate_yaml_path=\"$(which validate_yaml.py || :)\"\n\n# Ensure we have these at the minimum, these validate_*.py will cover\n# most configuration files as we dynamically find and call any validation programs further down\nif [[ -z \"$validate_yaml_path\" || \"$validate_yaml_path\" =~ pytools_checks ]]; then\n    get_pytools\nfi\n\necho\necho \"Running validation programs:\"\necho\nvalidate_yaml_path=\"$(which validate_yaml.py || :)\"\nif [ -z \"$validate_yaml_path\" ]; then\n    echo \"Failed to find validate_yaml.py in \\$PATH ($PATH)\"\n    exit 1\nfi\npytools_dir=\"$(dirname \"$validate_yaml_path\")\"\nfor validate_program in \"$pytools_dir\"/validate_*.py; do\n    [[ \"$validate_program\" =~ validate_multimedia.py ]] && continue\n    [ -L \"$validate_program\" ] && continue\n    if ! python -V 2>&1 | grep -q 'Python 2' &&\n      [[ \"$validate_program\" =~ validate_cson.py ]]; then\n        echo \"Skipping validate_cson.py on Python 3+ because the cson module hasn't been ported yet\"\n        echo\n        continue\n    fi\n    if [[ -n \"${SKIP_PARQUET:-}\" && \"$validate_program\" =~ .*parquet.* ]]; then\n        echo \"Skipping Parquet checks...\"\n        echo\n        continue\n    fi\n    opts=\"\"\n    if [ \"${validate_program##*/}\" = \"validate_ini.py\" ] ||\n       [ \"${validate_program##*/}\" = \"validate_properties.py\" ]; then\n\n        # upstream zookeeper log4j.properties has duplicate keys in it's config\n        echo \"$validate_program --include 'zookeeper-.*/.*contrib/rest/conf/log4j\\\\.properties' --ignore-duplicate-keys .\"\n        $validate_program --include 'zookeeper-.*/.*contrib/rest/conf/log4j\\.properties' --ignore-duplicate-keys .\n        echo\n\n        # alluxio-site.properies is commented out in Dockerfiles repo due to Alluxio parsing bug\n        # gradle's cache.properties is often empty just a single commented date line\n        echo \"$validate_program --include 'alluxio-site.properties|\\\\.gradle/.+/taskArtifacts/cache\\\\.properties' --allow-empty .\"\n        $validate_program --include 'alluxio-site.properties|\\.gradle/.+/taskArtifacts/cache\\.properties' --allow-empty .\n        echo\n\n        # exclude all of the above which are checked separately with different rules\n        # exclude kafka nagios plugin's /target/resolution-cache/check_kafka/check_kafka_2.10/0.1.0/resolved.xml.properties\n        # do not quote --exclude arg - the quotes will be interpreted literally and would require an eval\n        opts=' --exclude zookeeper-.*/.*contrib/rest/conf/log4j\\.properties|\\.xml\\.properties|alluxio-site.properties|\\.gradle/.+/taskArtifacts/cache\\.properties|\\.gradle/.+/gc.properties'\n\n    fi\n    echo \"${validate_program}$opts: \"\n    #  shellcheck disable=SC2086\n    ${validate_program}$opts .\n    echo\ndone\n\ntime_taken \"$start_time\"\nsection2 \"DevOps Python Tools validations SUCCEEDED\"\necho\n"
  },
  {
    "path": "checks/check_readme_badges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-26 02:03:33 +0000 (Thu, 26 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks for duplicates badge lines, assuming one badge per line as per std layout in headers across all my repos (more git / diff friendly)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking README badges for duplicates and incorrect links\"\n\nignored_lines_regex=\"\nSTATUS.md\nharisekhon/github\nharisekhon/centos-github\nStarTrack\nSTARCHARTS.md\nLinkedIn\nSpotify\nAWS Athena\nMySQL\nPostgreSQL\nMariaDB\nTeamCity\n^=*$\nhttps://aws.amazon.com\nhttps://sonarcloud.io/dashboard\nhttps://hub.docker.com/\nhttps://img.shields.io/badge/\nhttps://github.com/HariSekhon/[[:alnum:]-]*$\n\"\n\nstart_time=\"$(start_timer)\"\n\necho \"checking for duplicates:\"\necho\n\n# uniq -d will cause silent pipe failure without dups otherwise\nset +eo pipefail\n# want splitting for args\n# shellcheck disable=SC2046\nduplicates=\"$(\n    {\n    # exact README lines\n    # shellcheck disable=SC1117\n    \"$srcdir/../git/git_foreach_repo.sh\" \"grep -Eho '\\[\\!\\[.*\\]\\(.*\\)\\]\\(.*\\)' README.md\" |\n    sort |\n    uniq -d\n\n    # any URLs\n    #\"$srcdir/../git/git_foreach_repo.sh\" \"grep -Eho '\\[\\!\\[.*\\]\\(.*\\)\\]\\(.*\\)' README.md\" |\n    #grep -Eo '(http|https)://[a-zA-Z0-9./?=_%:#&,+-]*' |\n    #sort |\n    #uniq -d\n    } |\n    grep -vi $(IFS=$'\\n'; for line in $ignored_lines_regex; do [[ \"$line\" =~ ^[[:space:]]*$ ]] && continue; printf \"%s\" \" -e '$line'\"; done)\n)\"\nset -eo pipefail\n\ngithub_dir=\"$(dirname \"$srcdir\")\"\n\nwhile read -r line; do\n    if [ -z \"${line// }\" ]; then\n        continue\n    fi\n    grep -F --color=yes \"$line\" \"$github_dir\"/*/README.md\n    echo\ndone <<< \"$duplicates\"\n\nif [ -n \"$duplicates\" ]; then\n    exit 1\nelse\n    echo \"No duplicate badge lines found\"\n    echo\nfi\n\ntime_taken \"$start_time\"\nsection2 \"README badge checks passed\"\necho\n"
  },
  {
    "path": "checks/check_readme_exists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-08 11:10:28 +0000 (Tue, 08 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks that there is a LICENSE or LICENSE.txt at the top of a git repo and that it is not empty\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nsection2 'README file check'\n\nif ! git_root=\"$(git_root)\"; then\n    echo 'Not a git repository, skipping...'\n    echo\n    exit 0\nfi\n\ncd \"$git_root\"\n\nfor x in README README.md; do\n    if [ -s \"$x\" ]; then\n        echo \"OK: $x file found\"\n        exit 0\n    elif [ -f \"$x\" ]; then\n        echo \"WARNING: $x file found but it is empty! \"\n        exit 1\n    fi\ndone\n\necho 'WARNING: README file not found!'\nexit 1\n"
  },
  {
    "path": "checks/check_ruby_syntax.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nif [ -z \"$(find \"${1:-.}\" -maxdepth 2 -type f -iname '*.pl' -o -iname '*.pm' -o -iname '*.t')\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"Ruby Syntax Checks\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P ruby &>/dev/null; then\n    type -P ruby\n    ruby --version\n    echo\n    for x in $(find \"${1:-.}\" -maxdepth 2 -type f -iname '*.rb' | sort); do\n        isExcluded \"$x\" && continue\n        echo -n \"$x: \"\n        ruby -c \"$x\"\n    done\n    time_taken \"$start_time\"\n    section2 \"All Ruby programs passed syntax check\"\nelse\n    echo \"WARNING: ruby not found in \\$PATH, skipping ruby syntax checks\"\n    echo\nfi\necho\n"
  },
  {
    "path": "checks/check_sbt_build.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-06-30 14:46:43 +0100 (Thu, 30 Jun 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nbuild_files=\"$(find \"${1:-.}\" -name build.sbt)\"\n\nif [ -z \"$build_files\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"S B T\"\n\nstart_time=\"$(start_timer)\"\n\nif type -P sbt &>/dev/null; then\n    type -P sbt\n    sbt --version\n    echo\n    grep -v '/target/' <<< \"$build_files\" |\n    sort |\n    while read -r build_sbt; do\n        pushd \"$(dirname \"$build_sbt\")\" >/dev/null\n        echo \"Validating $build_sbt\"\n        echo q | sbt reload || exit $?\n        popd >/dev/null\n        echo\n    done\nelse\n    echo \"SBT not found in \\$PATH, skipping maven pom checks\"\nfi\n\ntime_taken \"$start_time\"\nsection2 \"SBT checks passed\"\necho\n"
  },
  {
    "path": "checks/check_shebang_non_executable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-10-06 13:17:14 +0200 (Fri, 06 Oct 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# NFS issues sometimes cause scripts to rewritten from vim without executable bit set, which then gets committed to git by accident\n\nsection \"Finding Non Executable Scripts\"\n\n# These shouldn't be executable even if they have #! lines for syntax reasons\nexceptions='\n\\.bash\\.d\n/lib/\n/\\..+\nenv$\n\\.env\n\\.envrc\nshrc$\n\\..*login$\n\\..*logout$\n\\.bak\n\\.pm$\n'\nexceptions_regex=\"\"\nfor exception in $exceptions; do\n    exceptions_regex=\"$exceptions_regex|$exception\"\ndone\nexceptions_regex=\"(${exceptions_regex#|})\"\n\nfilter_is_git_committed(){\n    while read -r filename; do\n        pushd \"$(dirname \"$filename\")\" &>/dev/null\n        set +o pipefail\n        git status --porcelain \"$filename\" | grep -q '^??' || echo \"$filename\"\n        set -o pipefail\n        popd &>/dev/null\n    done\n}\n\n# only if at start of file, not part why through like %pre / %post sections of anaconda-ks.cfg kickstart file\nfilter_is_shebang(){\n    while read -r filename; do\n        if [ \"$(head -c 2 \"$filename\")\" = '#!' ]; then\n            echo \"$filename\"\n        fi\n    done\n}\n\nset +o pipefail\n# -executable switch not available on Mac\n# trying to build up successive -name options doesn't work and ruins the logic of find, simplify to grep\nnon_executable_scripts=\"$(\n    eval find \"${1:-$PWD}\" -maxdepth 2 -type f -not -perm -u+x |\n    xargs grep -l '^#!' |\n    grep -Ev \"$exceptions_regex\" |\n    filter_is_shebang |\n    filter_is_git_committed |\n    tee /dev/stderr\n)\"\nset -o pipefail\n\necho\nif [ -z \"$non_executable_scripts\" ]; then\n    echo \"OK: no non-executable scripts detected\"\n    exit 0\nelse\n    echo 'FAILED: non-executable scripts detected!'\n    exit 1\nfi\n"
  },
  {
    "path": "checks/check_shell_commands_dash_protections.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-11 12:18:21 +0100 (Mon, 11 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks all files in the Git \\$PWD downwards for commands like cp and mv have double dashes before taking args\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ncheck(){\n    local cmd=\"$1\"\n    command git grep -E \"^[^#]*\\\\<$cmd[[:space:]]+\" |\n    grep -v -e '--' \\\n            -e \"${0##*/}:\" \\\n            -e \"alias $cmd\" \\\n            -e '\\.gitconfig:' \\\n            -e '\\.gitignore:' \\\n            -e '\\.conf:' \\\n    || :\n}\n\nfor cmd in cp mv rm ln; do\n    check \"$cmd\"\ndone\n"
  },
  {
    "path": "checks/check_shellcheck.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-22 20:54:53 +0000 (Fri, 22 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# This really only checks basic syntax, if you're made command errors this won't catch it\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nif [ $# -eq 0 ]; then\n    if [ -z \"$(find \"${1:-.}\" -type f -iname '*.sh')\" ]; then\n        # shellcheck disable=SC2317\n        return 0 &>/dev/null ||\n        exit 0\n    fi\nfi\n\nsection \"ShellCheck\"\n\nstart_time=\"$(start_timer)\"\n\nshellcheck(){\n    echo -n \"shellcheck: $1 \"\n    local basename=\"${1##*/}\"\n    local dirname\n    dirname=\"$(dirname \"$1\")\"\n    # this allows following source hints relative to the source file to be safe to run from any $PWD\n    if ! pushd \"$dirname\" &>/dev/null; then\n        echo \"ERROR: failed to pushd to $dirname\"\n        exit 1\n    fi\n    # -x allows to follow source hints for files not given as arguments\n    command shellcheck -x \"$basename\"\n    if ! popd &>/dev/null; then\n        echo \"ERROR: failed to popd from $dirname\"\n    fi\n    echo \"=> OK\"\n}\n\nrecurse_dir(){\n    for x in $(find \"${1:-.}\" -type f -iname '*.sh' | sort); do\n        isExcluded \"$x\" && continue\n        [[ \"$x\" =~ ${EXCLUDED:-} ]] && continue\n        shellcheck \"$x\"\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        if [ -d \"$x\" ]; then\n            recurse_dir \"$x\"\n        else\n            shellcheck \"$x\"\n        fi\n    done\nelse\n    recurse_dir .\nfi\n\ntime_taken \"$start_time\"\nsection2 \"ShellCheck passed\"\necho\n"
  },
  {
    "path": "checks/check_shippable_readme_ids.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-24 16:06:50 +0000 (Tue, 24 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/utils.sh\"\n\nif [ -z \"${SHIPPABLE_TOKEN:-}\" ]; then\n    echo \"Shippable token not set, skipping shippable check\"\n    exit 0\nfi\necho \"Checking Shippable README.md Project Badge\"\n# https://img.shields.io/shippable/5e52c634d79b7d00077bf5ed/master?label=Shippable)](https://app.shippable.com/github/HariSekhon/DevOps-Bash-tools/dashboard/jobs\nfind . -name README.md -exec grep -Eo 'img.shields.io/shippable/.*app.shippable.com/[^/]+/[^/]+/[^/]+' {} \\; |\nsed 's,img.shields.io/shippable/,,; s,[^[:alnum:]].*app.shippable.com/[^/]*/, ,' |\nwhile read -r id name; do\n    \"$srcdir/../shippable/shippable_projects.sh\" \"$id\" \"$@\" |\n    while read -r id2 owner repo; do\n        if [ \"$id\" != \"$id2\" ]; then\n            echo \"id '$id' != returned id '$id2'\"\n            exit 1\n        fi\n        if [ \"${name}\" != \"$owner/$repo\" ]; then\n            echo \"README.md Shippable badge name '$name' vs ID mismatch ('$owner/$repo')\"\n            exit 1\n        fi\n    done || exit 1\ndone\necho\n"
  },
  {
    "path": "checks/check_sqlfluff.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 01:35:54 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/sqlfluff/sqlfluff\n\n# https://docs.sqlfluff.com/en/stable/reference/cli.html\n\n# https://docs.sqlfluff.com/en/stable/reference/dialects.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/sql.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecursively iterates all SQL code files found in the given or current directory and runs SQLFluff linter against them,\ninferring the different SQL dialects from each path/filename/extension\n\nIf you don't have a mixed SQL tree or repo to lint then you could just do this as a single call to SQLFluff like so\n\n    sqlfluff lint --dialect postgres .\n\nUseful to call in CI/CD or precommit linting\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir> <sqlfluff_options>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\nshift || :\n\nexitcode=0\n\nsection \"SQLFluff\"\n\nif ! type -P sqlfluff &>/dev/null; then\n    timestamp \"SQLFluff not installed, attempting to install it now...\"\n    \"$srcdir/../packages/install_package.sh\" sqlfluff ||\n    pip install sqlfluff\n    echo\nfi\n\necho\nsqlfluff --version\necho\n\nif is_CI; then\n    export VERBOSE=1\nfi\n\nwhile read -r path; do\n    dialect=\"$(infer_sql_dialect_from_path \"$path\" || { echo \"Falling back to ANSI dialect\" >& 2; echo \"ansi\"; } )\"\n    opts=(--dialect \"$dialect\")\n    if is_CI; then\n        opts+=(--nocolor)\n    fi\n    timestamp \"SQLFluffing: $path\"\n    log \"Cmd: sqlfluff lint ${opts[*]} $path\"\n    if ! sqlfluff lint \"${opts[@]}\" \"$path\"; then\n        exitcode=1\n        echo\n    fi\ndone < <(find \"$dir\" -iname '*.sql' -type f)\n\nmsg=\"completed successfully\"\nif [ \"$exitcode\" != 0 ]; then\n    msg=\"found errors!\"\nfi\ntimestamp \"SQLFluffing $msg\"\necho\nhr\nexit \"$exitcode\"\n"
  },
  {
    "path": "checks/check_srcdir_references.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-08 23:46:32 +0100 (Mon, 08 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses shell scripts to find any references to srcdir/path/to/files.sh that are not found from the \\$PWD of the script\n\nWritten to find broken srcdir references after huge repo reorganization\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"arg [<options>]\"\n\nhelp_usage \"$@\"\n\nsection \"Check SrcDir References\"\n\ntrap_cmd \"echo 'FAILED!'\"\n\ngit grep -lE 'srcdir/.+\\.sh' |\ngrep -v '^\\.bash\\.d/' |\ngrep -v 'check_bash_references.sh' |\nwhile read -r scriptpath; do\n    pushd \"$(dirname \"$scriptpath\")\" >/dev/null\n    srcdirpaths=\"$(\n        { grep -v -e ' -x ' -e ' -f ' \"${scriptpath##*/}\" || : ; } |\n        { grep -Eo '^[^#]*\\$srcdir/[^\"'\"'\"'[:space:]]+.sh' || : ; } |\n        { grep -Eo -e 'srcdir/[^\"'\\''[:space:]]+\\.sh' -e 'srcdir/[^\"'\\''[:space:]]*bashrc' || : ; } |\n        sed 's|^srcdir/||'\n    )\"\n    for srcdirpath in $srcdirpaths; do\n        srcdirpath2=\"$(readlink -f \"$srcdirpath\" || :)\"\n        if ! test -f \"$srcdirpath2\"; then\n            echo \"$scriptpath\"\n            timestamp \"broken srcdir path: $srcdirpath\"\n        fi\n    done |\n    sort -u |\n    while read -r scriptpath; do\n        timestamp \"broken scrdir references found in: $scriptpath\"\n        exit 1\n    done\n    #while read -r scriptpath; do\n    #    echo \"$scriptpath:\"\n    #    echo\n    #    grep -E 'srcdir/.+\\.sh' \"${scriptpath##*/}\"\n    #    echo\n    #    echo\n    #    exit 1\n    #done\n    popd >/dev/null\ndone\n\nuntrap\necho \"All srcdir references checked: OK\"\necho\necho\n"
  },
  {
    "path": "checks/check_ssh_keys_encrypted.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2008 (forked from .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks SSH protocol v2 keys\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"SSH Keys encrypted check\"\n\nfor arg in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$arg\" in\n        -*)  usage\n             ;;\n    esac\ndone\n\ncheck_bin openssl\n\nerrors=0\ncheck_ssh_keys_encrypted(){\n    echo \"checking keys:\"\n    echo\n    for key in \"${@:-~/.ssh/id_rsa}\"; do\n        printf \"%s\" \"$key => \"\n        if openssl rsa -noout -passin pass:none -in \"$key\" 2>/dev/null; then\n            echo \"WARNING: your SSH Key '$key' is unprotected. Encrypt it and use SSH agent\"\n            errors=1\n        else\n            echo \"OK\"\n        fi\n    done\n}\n\ncheck_ssh_keys_encrypted \"$@\"\n\nif [ $errors = 1 ]; then\n    exit 1\nfi\n\necho\nhr\necho\n"
  },
  {
    "path": "checks/check_symlinks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-16 10:41:43 +0000 (Sun, 16 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks for broken symlinks that don't point to existing targets\n\nTraverses the given directory or \\$PWD\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<starting_directory>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nbasedir=\"${1:-.}\"\n\nsection \"Symlink Check\"\n\nstart_time=\"$(start_timer)\"\n\nfailing_symlinks=0\nwhile read -r symlink; do\n    if [[ \"$symlink\" =~ /\\.git/ ]]; then\n       continue\n    fi\n    # readlink -f fails to return anything if a parent dir component doesn't exist\n    target=\"$(readlink -m \"$symlink\" || :)\"\n    echo -n '.'\n    # shouldn't happen now switched from readlink -f to readlink -m\n    if [ -z \"$target\" ]; then\n        echo\n        echo \"WARNING: symlink '$symlink' target could not be resolved\" >&2\n        ((failing_symlinks+=1))\n    elif ! [ -e \"$target\" ]; then\n        echo\n        echo \"WARNING: symlink '$symlink' => '$target' - target does not exist\" >&2\n        ((failing_symlinks+=1))\n    fi\ndone < <(find \"$basedir\" -type l)\necho\n\nif [ \"$failing_symlinks\" -gt 0 ]; then\n    echo\n    echo \"ERROR: broken symlinks found!\" >&2\n    echo\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"All Symlink checks passed\"\necho\n"
  },
  {
    "path": "checks/check_tests_run_qualified.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-10-08 16:57:51 +0100 (Sun, 08 Oct 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking all test_*.sh run calls are fully qualified\"\n\nfailed_count=0\nscripts=\"$(find \"${1:-.}\" -maxdepth 2 -type f -name 'test*.sh')\"\n#while read script; do\n# doesn't handle whitespace in names but makes global variable access easier we don't use whitespace in names in unix\nfor script in $scripts; do\n    # this fails files broken on whitespace immediately, will count as 2 of more separate files not found\n    echo -n \"checking script $script => \"\n    if ! [ -f \"$script\" ]; then\n        ((failed_count + 1))\n        echo \"NOT FOUND\"\n        continue\n    fi\n    set +eo pipefail\n                     # don't anchor grep -v as we're prefixing line numbers for convenience\n    # run\n    # run_fail \\d+\n    # run_fail \"\\d+ \\d+ ...\"\n    # run_conn_refused\n    # run_usage\n    # run_404\n    # run_timeout\n    run_fail='run(_[a-z]+[[:digit:]]* \"?[[:digit:][:space:]]+\"?|_conn_refused|_usage|_404|_timeout)?'\n    # docker-compose or docker exec or docker run\n    docker_regex='docker(-compose|[[:space:]]+(exec|run))'\n    # want $bin in regex\n    # shellcheck disable=SC2016\n    suspect_lines=\"$(grep -En '^[[:space:]]*run(_.+)?[[:space:]]+' \"$script\" |\n                     grep -Ev -e \"[[:space:]]*${run_fail}[[:space:]](.*[[:space:]])?(./|(\\\\\\$perl|eval|$docker_regex)[[:space:]])\" \\\n                              -e '[[:space:]]*run_test_versions' \\\n                              -e '[[:space:]]*run_(grep|output)[[:space:]].+(\\$|./)' \\\n                              -e 'ignore_run_unqualified' \\\n                              -e '\\$bin/' \\\n                              # run_grep filter is not that accurate but will do for now\n                              # ignore golang tests where we do run \"$bin/binary\"\n                    )\"\n    set -eo pipefail\n    if [ -n \"$suspect_lines\" ]; then\n        ((failed_count + 1))\n        echo \"Suspect lines detected!\"\n        echo\n        echo \"$suspect_lines\"\n        echo\n    else\n        echo \"OK\"\n    fi\ndone\n\nif [ \"$failed_count\" = 0 ]; then\n    echo\n    echo 'Passed checks'\n    echo\nelse\n    echo\n    echo \"$failed_count scripts with non-qualified command run arguments detected!\"\n    exit 1\nfi\n"
  },
  {
    "path": "checks/check_tld_chars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2018-08-09 21:34:03 +0100 (Thu, 09 Aug 2018)\n#\n#  https://github.com/HariSekhon/pylib\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\ntld_files=\"\ntlds-alpha-by-domain.txt\ncustom_tlds.txt\n\"\n\nsection \"Checking TLDs for suspect chars\"\n\nstart_time=\"$(start_timer)\"\n\nfiles=\"${*:-}\"\nif [ -z \"$files\" ]; then\n    for x in $tld_files; do\n        files=\"$files $(find . -iname \"$x\")\"\n    done\nfi\n\nset +e\nfor x in $files; do\n    #isExcluded \"$x\" && continue\n    echo \"checking $x\"\n    if sed 's/#.*//;/^[[:space:]]$/d;s/[[:alnum:]-]//g' \"$x\" | grep -o '.'; then\n        echo\n        echo \"ERROR: Invalid chars detected in TLD file $x!! \"\n    fi\ndone\n\ntime_taken \"$start_time\"\nsection2 \"Finished checking TLDs for suspect chars\"\n"
  },
  {
    "path": "checks/check_travis_yml.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-15 00:33:52 +0000 (Fri, 15 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\n#return 0 &>/dev/null || :\n#exit 0\n\nsection \"Travis CI Yaml Lint Check\"\n\nif ! [ -f .travis.yml ]; then\n    echo \"No .travis.yml found, skipping Travis CI check\"\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\n#if is_travis; then\n#    echo \"Running inside Travis CI, skipping lint check\"\nif is_inside_docker; then\n    echo \"Running inside Docker, skipping Travis lint check\"\nelse\n    # sometimes ~/.gem/ruby/<version>/bin may not be in $PATH but this succeeds anyway if hashed in shell\n    #if ! type travis &>/dev/null; then\n    for path in ~/.gem/ruby/*/bin; do\n        [ -d \"$path\" ] || continue\n        echo \"adding $path to \\$PATH\"\n        export PATH=\"$PATH:$path\"\n    done\n    if ! type -P travis &>/dev/null; then\n        ruby_version=\"$(ruby --version | awk '{print $2}' | grep -Eo '[[:digit:]]+\\.[[:digit:]]+' | head -n1)\"\n        if bc -l <<< \"$ruby_version < 2.4\" | grep -q 1; then\n            echo \"Ruby version is < 2.4, too old to install Travis CI gem, skipping check\"\n        elif type -P gem &>/dev/null; then\n            # this returns ruby-1.9.3 but using 1.9.1\n            #ruby_version=\"$(ruby --version | awk '{print $2}' | sed 's/p.*//')\"\n            #export PATH=\"$PATH:$HOME/.gem/ruby/$ruby_version/bin\"\n            echo \"installing travis gem... (requires ruby-dev package to be installed)\"\n            # --no-rdoc option not valid on GitHub Workflows macos-latest build\n            #gem install --user-install travis --no-rdoc --no-ri\n            #\"$srcdir/ruby_gem_install_if_absent.sh\" travis\n            # handles SSL linking issues on Mac\n            NONINTERACTIVE=1 \"$srcdir/../install/install_travis.sh\"\n            for path in ~/.gem/ruby/*/bin; do\n                [ -d \"$path\" ] || continue\n                echo \"adding $path to \\$PATH\"\n                export PATH=\"$PATH:$path\"\n            done\n        else\n            echo \"WARNING: skipping Travis install as gem command was not found in \\$PATH\"\n        fi\n        echo\n    fi\n    if type -P travis &>/dev/null; then\n        echo 'Travis path:'\n        echo\n        type -P travis\n        echo\n        echo -n 'Travis version:  '\n        if ! travis version --no-interactive; then\n            echo\n            echo \"WARNING: Travis Gem / install broken, skipping check\"\n            exit 1\n        fi\n        echo\n        echo -n 'Travis lint:  '\n        # Travis CI is getting upstream errors randomly, eg.\n        # server error (500: \"Sorry, we experienced an error.\\n\\nrequest_id:16146a4f-677e-4314-888d-149617bdae2d\\n\")\n        set +e\n        if is_CI; then\n            # get past shell completion install prompt in CI\n            #echo \"n\" | travis lint\n            travis lint --no-interactive\n        else\n            travis lint\n        fi\n        set -e\n    else\n        echo \"WARNING: skipping Travis check as Travis is not installed\"\n    fi\nfi\n\necho\ntime_taken \"$start_time\"\nsection2 \"Travis CI yaml validation succeeded\"\necho\n"
  },
  {
    "path": "checks/check_url_links.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-17 18:11:09 +0000 (Mon, 17 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks for broken URL links in a given file or directory tree\n\nSends HEAD requests and follows redirects - as long as the link redirects and succeeds it'll still pass, as this is most relevant to users and READMEs\n\nAccepts HTTP 2xx/3xx status codes as well as the following to avoid false positives:\n- HTTP 400 (bad request) - eg. a valid API URL may complain we're not sending the required parameters/headers/post body\n- HTTP 401 (unauthorized)\n- HTTP 403 (forbidden)\n- HTTP 405 (method not allowed, ie. HEAD)\n- HTTP 429 (rate limiting)\n\nIgnores:\n- Private addresses (localhost, .local, .svc, .cluster.local)\n- Loopback IP (127.0.0.1)\n- Private IPs (10.x.x.x, 172.16.x.x, 192.168.x.x)\n- APIPA IPs (169.254.x.x)\n\nTo ignore links created with variables or otherwise composed in a way we can't straight test them, you can set URL_LINKS_IGNORED to a list, one per line of the URLs\nTo ignore links without dots in them, ie. not public URLs such as domains or IP addresses, which are most likely internal shortname services, set IGNORE_URLS_WITHOUT_DOTS to any value\n\nIf run in CI, runs 'git ls-files' to avoid scanning other local checkouts or git submodules\nIf you want to filter to specific files only, such a README.md, you can set URL_LINKS_FILE_FILTER='README.md' by name or path or ERE regex\n\nExamples:\n\n    # Scan all URLs in all files under your \\$PWD, or in CI all committed files under your \\$PWD\n\n        ${0##*/}\n\n    # Scan URLs in all files found under the 'src' directory\n\n        ${0##*/} src\n\n    # Scan URLs in all files called README.md under your local directory (local mode only, not CI)\n\n        ${0##*/} . -name README.md\n\n    # Ignore URLs we know won't work because they're samples / fake or constructed with variables we can't determine etc:\n\n        export URL_LINKS_IGNORED='\n            http://myplaceholder\n            nonexistent.com\n            https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK\n            https://some.website.com/downloads/v\\$version/some.tar.gz\n        '\n\n        ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file_or_directory> <find_or_git_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nsection \"URL Link Checks\"\n\nstart_time=\"$(start_timer)\"\n\nstartpath=\"${1:-.}\"\nshift || :\n\ntrap_cmd 'echo >&2'\n\ncheck_bin curl\n\ncheck_url_link(){\n    local url=\"$1\"\n    if [ -n \"${VERBOSE:-}\" ] || [ -n \"${DEBUG:-}\" ]; then\n        echo -n \"$url => \" >&2\n    fi\n    status_code=\"$(command curl -sSILf --retry 3 --retry-delay 2 \"$url\" -o /dev/null -w \"%{http_code}\" 2>/dev/null || :)\"\n    if [ -n \"${VERBOSE:-}\" ] || [ -n \"${DEBUG:-}\" ]; then\n        echo \"$status_code\" >&2\n    else\n        echo -n '.' >&2\n    fi\n    # DockerHub https://registry.hub.docker.com/v2 returns 401\n    # GitHub returns HTTP 429 for too many requests\n    if ! [[ \"$status_code\" =~ ^([23][[:digit:]]{2}|400|401|403|405|429)$ ]]; then\n        echo >&2\n        echo \"Broken Link: $url\" >&2\n        echo >&2\n        echo 1\n        return 1\n    fi\n}\n\n# Mac's BSD grep has a bug around -f ignores\nif is_mac; then\n    grep(){\n        command ggrep \"$@\"\n    }\nfi\n\ntimestamp \"Aggregating unique URLs from files under '$startpath'\"\n# filtering out LinkedIn.com which prevents crawling with HTTP/2 999 code\n#               GitHub returns HTTP 429 for too many requests\n                #-e 'https://github\\.com/marketplace' \\\nurls=\"$(\n    if is_CI; then\n        git ls-files \"$startpath\" \"$@\"\n    else\n        find -L \"$startpath\" -type f \"$@\" |\n        { grep -v -e '/\\.git/' -e '/\\.svn/' -e '/\\.hg/' || : ; }\n    fi |\n    if [ -n \"${URL_LINKS_FILE_FILTER:-}\" ]; then\n        grep -E \"$URL_LINKS_FILE_FILTER\" || :\n    else\n        cat\n    fi |\n    while read -r filename; do\n        [ -f \"$filename\" ] || continue  # protects against symlinks to dirs returned by 'git ls-files'\n        # $url_regex defined in lib/utils.sh\n        # shellcheck disable=SC2154\n        { grep -E \"$url_regex\" \"$filename\" || : ; } |\n        #sed 's/#.*//; /^[[:space:]]*$/d' |\n        { grep -Eiv \\\n             -e '\\$' \\\n             -e 'localhost' \\\n             -e '\\.svc$' \\\n             -e '.local$' \\\n             -e '\\.cluster\\.local' \\\n             -e 'domain\\.com' \\\n             -e 'acmecorp\\.com' \\\n             -e 'example\\.com' \\\n             -e 'linkedin\\.com' \\\n             -e '(169\\.254\\.)' \\\n             -e '(172\\.16\\.)' \\\n             -e '(192\\.168\\.)' \\\n             -e '10\\.[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+' \\\n             -e '127.0.0.1' \\\n             -e '\\.\\.\\.' \\\n             -e 'x\\.x\\.x\\.x' || : ; } |\n        { grep -Eo \"$url_regex\" || : ; } |\n        if [ -n \"${URL_LINKS_IGNORED:-}\" ]; then\n            grep -Eivf <(\n                tr '[:space:]' '\\n' <<< \"$URL_LINKS_IGNORED\" |\n                sed 's/^[[:space:]]*//;\n                     s/[[:space:]]*$//;\n                     /^[[:space:]]*$/d'\n            )\n        else\n            cat\n        fi |\n        if [ -n \"${IGNORE_URLS_WITHOUT_DOTS:-}\" ]; then\n            grep -E 'https?://[^/]+\\.[^/]+' || :\n        else\n            cat\n        fi\n    done |\n    sort -uf\n)\"\nurls=\"${urls##[[:space:]]}\"\nurls=\"${urls%%[[:space:]]}\"\necho >&2\n\nif is_blank \"$urls\"; then\n    echo \"No URLs found\" >&2\n    exit 0\nfi\n\nurl_count=\"$(wc -l <<< \"$urls\" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')\"\n\ntimestamp \"Checking $url_count unique URLs\"\necho >&2\n\ntests=$(\nwhile read -r url; do\n    echo \"check_url_link '$url'\"\ndone <<< \"$urls\"\n)\n\n# export function to use in parallel\nexport -f check_url_link\nexport SHELL=/bin/bash  # Debian docker container doesn't set this and defaults to sh, failing to find exported function\n\nset +eo pipefail\ntally=\"$(parallel -j 10 <<< \"$tests\")\"\nexit_code=$?\nset -eo pipefail\n\nbroken_count=\"$(awk '{sum+=$1} END{print sum}' <<< \"$tally\")\"\n\necho >&2\ntime_taken \"$start_time\"\necho >&2\n\nif [ $exit_code -eq 0 ]; then\n    section2 \"URL links passed\"\nelse\n    echo \"ERROR: $broken_count/$url_count broken links detected!\" >&2\n    echo >&2\n    section2 \"URL Links FAILED\"\n    exit 1\nfi\n"
  },
  {
    "path": "checks/check_vagrantfiles.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 11:52:44 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nvagrantfiles=\"$(find \"${1:-.}\" -maxdepth 3 -name Vagrantfile)\"\n\nif [ -z \"$vagrantfiles\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"V a g r a n t\"\n\nstart_time=\"$(start_timer)\"\n\n# catches $VAGRANT_HOME/Vagrantfile instead of validating the one we are targeting\nunset VAGRANT_HOME\n\nif type -P vagrant &>/dev/null; then\n    type -P vagrant\n    vagrant --version\n    echo\n    echo \"Validating Vagrantfiles:\"\n    echo\n    while read -r vagrantfile; do\n        pushd \"$(dirname \"$vagrantfile\")\" >/dev/null\n\n        # create host directories to avoid this Vagrantfile validation error in GitHub Actions \"CI Mac\" builds:\n        #\n        # vm:\n        # * The host path of the shared folder is missing: ~/github\n        #\n        awk '/^[[:space:]]*config.vm.synced_folder/{print$2}' Vagrantfile |\n        sed 's/,$//; s/\"//g' |\n        sed \"s/'//\" |\n        while read -r directory; do\n            # Mac is silent even with -v and\n            # creates literal ~/ subdirectory path unless eval'd, leaving validation to fail with the real path missing\n            eval mkdir -p -v \"$directory\"\n        done\n\n        echo -n \"$vagrantfile => \"\n        vagrant validate\n        popd >/dev/null\n    done <<< \"$vagrantfiles\"\nelse\n    echo \"WARNING: 'vagrant' is not installed, skipping...\"\n    echo\n    exit 0\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Vagrantfile validation SUCCEEDED\"\necho\n"
  },
  {
    "path": "checks/check_whitespace.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-09 18:50:56 +0000 (Sat, 09 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nsection \"Checking Whitespace\"\n\nstart_time=\"$(start_timer)\"\n\n# shellcheck source=lib/excluded.sh\n. \"$srcdir/lib/excluded.sh\"\n\nprogress_char='-'\n[ -n \"${DEBUG:-}\" ] && progress_char=''\n\nwhitespace_only_files_found=0\ntrailing_whitespace_files_found=0\ntrailing_whitespace_bar_files_found=0\nfor filename in $(find \"${1:-.}\" -type f | grep -Evf \"$srcdir/../resources/whitespace_ignore.txt\" | sort); do\n    isExcluded \"$filename\" && continue\n    printf \"%s\" \"$progress_char\"\n    # [[:space:]] matches \\r in Windows files which we don't want, use explicit character class instead to exclude \\r\n    # works but not efficient\n    #grep -Hn '^[[:space:]]\\+$' <<< \"$(tr -d '\\r' < \"$filename\")\" && let whitespace_only_files_found+=1 || :\n    #grep -Hn '[[:space:]]\\+$'  <<< \"$(tr -d '\\r' < \"$filename\")\" && let trailing_whitespace_files_found+=1 || :\n    #grep -EHn '[[:space:]]{4}\\|[[:space:]]*$'  <<< \"$(tr -d '\\r' < \"$filename\")\" && let trailing_whitespace_bar_files_found+=1 || :\n    # \\t aren't working inside character classes for some reason, embedding literal tabs instead\n    output=\"$(grep -EHn '^[ \t]+$' \"$filename\" || :)\"\n    if [ -n \"$output\" ]; then\n        echo\n        echo \"$output\"\n        ((whitespace_only_files_found + 1))\n    fi\n    output=\"$(grep -EHn '[ \t]+$' \"$filename\" || :)\"\n    if [ -n \"$output\" ]; then\n        echo\n        echo \"$output\"\n        ((trailing_whitespace_files_found + 1))\n    fi\n    output=\"$(grep -EHn '[ \t]{4}\\|[ \t]*$' \"$filename\" || :)\"\n    if [ -n \"$output\" ]; then\n        echo\n        echo \"$output\"\n        ((trailing_whitespace_bar_files_found + 1))\n    fi\ndone\necho\nif [ $whitespace_only_files_found -gt 0 ]; then\n    echo \"$whitespace_only_files_found files with whitespace only lines detected!\"\nfi\nif [ $trailing_whitespace_files_found -gt 0 ]; then\n    echo \"$trailing_whitespace_files_found files with trailing whitespace lines detected!\"\nfi\nif [ $trailing_whitespace_bar_files_found -gt 0 ]; then\n    echo \"$trailing_whitespace_bar_files_found files with trailing whitespace bar lines detected!\"\nfi\nif [ $whitespace_only_files_found -gt 0 ] ||\n   [ $trailing_whitespace_files_found -gt 0 ] ||\n   [ $trailing_whitespace_bar_files_found -gt 0 ]; then\n    # shellcheck disable=SC2317\n    return 1 &>/dev/null ||\n    exit 1\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Whitespace only checks passed\"\necho\n"
  },
  {
    "path": "checks/check_xml.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-13 16:18:51 +0100 (Mon, 13 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n#if [ $# -gt 0 ]; then\n#    filelist=\"$*\"\n#else\n    # can point to test.json as an explicit argument\n    # could check *.json too but in DevOps Python Tools repo multirecord.json would break as this cannot handle multi-json record files\n    #filelist=\"$(find \"${1:-.}\" -type f -name '*.y*ml' -o -type f -name '*.json' | sort)\"\n    filelist=\"$(find \"${1:-.}\" -type f -name '*.xml' | sort)\"\n#fi\n\nif [ -z \"$filelist\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"XML Syntax Checks\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping XML syntax checks\"\n    echo\n    exit 0\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping XML syntax checks\"\n    echo\n    exit 0\nfi\n\n\nif ! command -v xmllint &>/dev/null; then\n    echo \"xmllint not found in \\$PATH, not running XML syntax checks\"\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\ntype -P xmllint\nxmllint --version\necho\n\nexport max_len=0\nfor x in $filelist; do\n    if [ \"${#x}\" -gt \"$max_len\" ]; then\n        max_len=\"${#x}\"\n    fi\ndone\n# to account for the colon\n((max_len + 1))\n\ncheck_xml(){\n    local filename=\"$1\"\n    printf \"%-${max_len}s \" \"$filename:\" >&2\n    set +eo pipefail\n    output=\"$(xmllint \"$filename\")\"\n    result=$?\n    set -eo pipefail\n    # shellcheck disable=SC2181\n    if [ \"$result\" -eq 0 ]; then\n        echo \"OK\" >&2\n    else\n        echo \"FAILED\" >&2\n        if [ -z \"${QUIET:-}\" ]; then\n            echo >&2\n            # shellcheck disable=SC2001\n            sed \"s|^|$filename: |\" <<< \"$output\" >&2\n            echo >&2\n        fi\n        echo 1\n        exit 1\n    fi\n}\n\necho \"building file list\" >&2\ntests=\"$(\n    for filename in $filelist; do\n        #isExcluded \"$filename\" && continue\n        echo \"check_xml $filename\"\n    done\n)\"\n\ncpu_count=\"$(cpu_count)\"\nmultiplier=1  # doesn't get faster increasing this in tests, perhaps even slightly slower due to context switching\nparallelism=\"$((cpu_count * multiplier))\"\n\necho \"found $cpu_count cores, running $parallelism parallel jobs\"\necho\n\n# export functions to use in parallel\nexport -f check_xml\nexport SHELL=/bin/bash  # Debian docker container doesn't set this and defaults to sh, failing to find exported function\n\nset +eo pipefail\ntally=\"$(parallel -j \"$parallelism\" <<< \"$tests\")\"\nexit_code=$?\nset -eo pipefail\n\ncount=\"$(awk '{sum+=$1} END{print sum}' <<< \"$tally\")\"\n\necho >&2\ntime_taken \"$start_time\"\necho >&2\n\nif [ $exit_code -eq 0 ]; then\n    section2 \"All XML files passed syntax check\"\nelse\n    echo \"ERROR: $count broken xml files detected!\" >&2\n    echo >&2\n    section2 \"XML checks failed\"\n    exit 1\nfi\n"
  },
  {
    "path": "checks/check_yaml.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 17:24:58 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# use .yamllint in $PWD or default to $srcdir/yamllint/config\n#export XDG_CONFIG_HOME=\"$srcdir\"\n\n#export YAMLLINT_CONFIG_FILE=\"$srcdir/.config/yamllint/config\"\nexport YAMLLINT_CONFIG_FILE=\"${YAMLLINT_CONFIG_FILE:-$srcdir/../configs/.yamllint.yaml}\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a yaml file or recurses a directory of yamls\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"file1 [file2 file3 ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfilelist=()\n\nfor arg in \"${@:-.}\"; do\n    if [ -d \"$arg\" ]; then\n        filelist+=( \"$(find \"$arg\" -type f -name '*.y*ml' | sort)\" )\n    else\n        filelist+=(\"$arg\")\n    fi\ndone\n\nif [ -z \"${filelist[*]}\" ]; then\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null ||\n    exit 0\nfi\n\nsection \"YAML Syntax Checks\"\n\nif [ -n \"${NOSYNTAXCHECK:-}\" ]; then\n    echo \"\\$NOSYNTAXCHECK environment variable set, skipping YAML syntax checks\"\n    echo\n    exit 0\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping YAML syntax checks\"\n    echo\n    exit 0\nfi\n\n\nif ! command -v yamllint &>/dev/null; then\n    echo \"yamllint not found in \\$PATH, not running YAML syntax checks\"\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\ntype -P yamllint\nyamllint --version\necho\n\nexport max_len=0\nfor x in $filelist; do\n    if [ \"${#x}\" -gt \"$max_len\" ]; then\n        max_len=\"${#x}\"\n    fi\ndone\n# to account for the colon\n((max_len + 1))\n\ncheck_yaml(){\n    local filename=\"$1\"\n    printf \"%-${max_len}s \" \"$filename:\" >&2\n    set +eo pipefail\n    # doesn't pick up the config without an explicit -c ...\n    output=\"$(yamllint -c \"$YAMLLINT_CONFIG_FILE\" \"$filename\")\"\n    result=$?\n    set -eo pipefail\n    # shellcheck disable=SC2181\n    if [ $result -eq 0 ]; then\n        echo \"OK\" >&2\n    else\n        echo \"FAILED\" >&2\n        if [ -z \"${QUIET:-}\" ]; then\n            echo >&2\n            # shellcheck disable=SC2001\n            sed \"s|^|$filename: |\" <<< \"$output\" >&2\n            echo >&2\n        fi\n        echo 1\n        exit 1\n    fi\n}\n\necho \"building file list\" >&2\ntests=\"$(\n    for filename in $filelist; do\n        # very expensive git log and regex matches against every file\n        #isExcluded \"$filename\" && continue\n        echo \"check_yaml $filename\"\n    done\n)\"\n\ncpu_count=\"$(cpu_count)\"\nmultiplier=1  # doesn't get faster increasing this in tests, perhaps even slightly slower due to context switching\nparallelism=\"$((cpu_count * multiplier))\"\n\necho \"found $cpu_count cores, running $parallelism parallel jobs\"\necho\n\n# export functions to use in parallel\nexport -f check_yaml\nexport SHELL=/bin/bash  # Debian docker container doesn't set this and defaults to sh, failing to find exported function\n\nset +eo pipefail\ntally=\"$(parallel -j \"$parallelism\" <<< \"$tests\")\"\nexit_code=$?\nset -eo pipefail\n\ncount=\"$(awk '{sum+=$1} END{print sum}' <<< \"$tally\")\"\n\necho >&2\ntime_taken \"$start_time\"\necho >&2\n\nif [ $exit_code -eq 0 ]; then\n    section2 \"All YAML files passed syntax check\"\nelse\n    echo \"ERROR: $count broken yaml files detected!\" >&2\n    echo >&2\n    section2 \"YAML checks failed\"\n    exit 1\nfi\n"
  },
  {
    "path": "cicd/.concourse.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-21 11:06:48 +0000 (Sat, 21 Mar 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            C o n c o u r s e   C I\n# ============================================================================ #\n\n# https://concourse-ci.org/golang-library-example.html\n\n# https://resource-types.concourse-ci.org/\n# https://concourse-ci.org/resource-types.html\nresources:\n  - name: github\n    icon: github-circle\n    type: git\n    source:\n      uri: https://github.com/HariSekhon/DevOps-Bash-tools\n      branch: master\n  #- name: daily\n  #  type: time\n  #  source:\n  #    interval: 1d\n\n# https://concourse-ci.org/jobs.html\njobs:\n  - name: build\n    public: false\n    plan:\n      - get: github  # from resource above\n        trigger: true\n        #version: every  # build every git commit, default: latest\n      - task: build\n        config:\n          platform: linux\n          image_resource:\n            type: docker-image\n            source:\n              repository: ubuntu\n              tag: latest\n          inputs:\n            - name: github\n              path: code\n          params:\n            CONCOURSE: 1\n          run:\n            path: /bin/bash\n            args:\n              - -c\n              - |\n                cd code &&\n                setup/ci_bootstrap.sh &&\n                make init &&\n                make ci test\n"
  },
  {
    "path": "cicd/.gocd.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-21 11:14:07 +0000 (Sat, 21 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                    G o C D\n# ============================================================================ #\n\n# https://github.com/tomzo/gocd-yaml-config-plugin#setup\n\n# https://docs.gocd.org/current/configuration/configuration_reference.html\n\n---\nformat_version: 3\npipelines:\n  devops-bash-tools:\n    group: defaultGroup\n    label_template: ${COUNT}\n    lock_behavior: none\n    display_order: -1\n    materials:\n      git:\n        git: https://github.com/HariSekhon/DevOps-Bash-tools\n        shallow_clone: false\n        auto_update: true\n        branch: master\n    stages:\n      - build-and-test:\n          fetch_materials: true\n          keep_artifacts: false\n          clean_workspace: false\n          approval:\n            type: success\n            allow_only_on_success: false\n          jobs:\n            #apt-update:\n            #  timeout: 10\n            #  tasks:\n            #  - exec:\n            #      command: apt\n            #      arguments:\n            #        - update\n            #      run_if: passed\n            #install-make:\n            #  timeout: 10\n            #  tasks:\n            #  - exec:\n            #      command: apt\n            #      arguments:\n            #        - install\n            #        - -qy\n            #        - git\n            #        - make\n            #      run_if: passed\n            ci-bootstrap:\n              timeout: 10\n              tasks:\n                - exec:\n                    command: setup/ci_bootstrap.sh\n                    run_if: passed\n            init:\n              timeout: 10\n              tasks:\n                - exec:\n                    command: make\n                    arguments:\n                      - init\n                    run_if: passed\n            build:\n              timeout: 60\n              tasks:\n                - exec:\n                    command: make\n                    arguments:\n                      - ci\n                    run_if: passed\n            test:\n              timeout: 60\n              tasks:\n                - exec:\n                    command: make\n                    arguments:\n                      - test\n                    run_if: passed\n"
  },
  {
    "path": "cicd/buildspec.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-12-19 15:32:28 +0000 (Sat, 19 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           A W S   C o d e B u i l d\n# ============================================================================ #\n\n# References:\n#\n#     https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html\n\n# Operating System should be set to Ubuntu, not Amazon Linux 2\n# - this is both recommended since programming language runtimes are now included in standard image of ubuntu, but also to avoid this error:\n#\n#     /usr/bin/amazon-linux-extras\n#     /root/.pyenv/versions/3.8.3/bin/python: No module named amazon_linux_extras\n\nversion: 0.2\n\n# only on Linux, the user to run as - global setting, alternatively set inside a phase section below for localized user\n#run-as: linux-username\n\nenv:\n  shell: bash\n  # don't store sensitive stuff like AWS secret keys in variables, use parameter-store or secrets-manager\n  # any environment variables replace existing environment variables, ie. beware if setting PATH that it'll replace the existing PATH with a non-interpolated literal\n  # project env vars take precedence over these, with start build vars taking highest precedence\n  #variables:\n  #  DEBUG: \"1\"\n  #exported-variables:\n  #  - DEBUG\n\nphases:\n  # install prerequisites / languages / frameworks / packages to allow build to work\n  install:\n    #commands:\n    #  - setup/ci_bootstrap.sh\n    # languages to install\n    runtime-versions:\n      #java: openjdk11\n      # AWS LTS release of OpenJDK 11\n      java: corretto11\n      golang: 1.14\n      python: 3.8\n      ruby: 2.7\n  # eg. sign in to Amazon ECR or install package dependencies\n  pre_build:\n    commands:\n      - setup/ci_bootstrap.sh\n  build:\n    commands:\n      - echo Build started on `date`\n      - make\n      - echo Build completed on `date`\n      - make test\n      - echo Tests completed on `date`\n"
  },
  {
    "path": "cicd/checkov_resource_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-22 15:20:27 +0000 (Tue, 22 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts the number of resources Checkov is scanning in the current or given directory\n\nUseful to estimate Bridgecrew Cloud costs which are charged per resource\n\nThis can take a while to run on large directories with lots of resources\n\nThe first argument should be a directory (defaults to '.' for \\$PWD)\nThe second argument onwards are passed as-is directly to the 'checkov' command\n\nAlternatively, use a local .checkov.yaml config file to configure checkov settings,\nsuch as multiple directories or skip directories, eg:\n\n    https://github.com/HariSekhon/Templates/blob/master/.checkov.yaml\n\n\nRequires Checkov, awk and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory> <checkov_args>]\"\n\nhelp_usage \"$@\"\n\ncheck_bin awk\ncheck_bin jq\n\n#min_args 1 \"$@\"\n\ndir=\"${1:-.}\"\nshift || :\n\n# user should supply args or local .checkov.yaml if they want additional checkov settings\n#\n# gets lots of errors without --download-external-modules=true, eg:\n# 2022-02-22 15:27:26,308 [MainThread  ] [WARNI]  Failed to download module x/y/z:n.n.n\n#\njson_data=\"$(checkov -d \"$dir\" --download-external-modules true -o json \"$@\")\"\n\njq_count=\"$(\n    jq '\n        if\n            type==\"array\" then .\n        else\n            [.]\n        end |\n        [ .[].summary.resource_count ] |\n        add\n        ' <<< \"$json_data\"\n)\"\n\n# opening brace must be on same line as the /regex_filter/, otherwise will apply to all lines\n# substr to strip the trailing json comma from 'resource_count: <number>,'\nawk_count=\"$(\n    awk '/resource_count/ {\n            resource_count = substr($2, 0, length($2) - 1)\n            total_resource_count += resource_count\n         }\n        END {\n            print total_resource_count\n        }' <<< \"$json_data\"\n)\"\n\nif [ \"$jq_count\" != \"$awk_count\" ]; then\n    die 'Parsing inconsistency detected between jq and awk on the checkov json data. Checkov output format may have changed'\nfi\n\necho \"$jq_count\"\n"
  },
  {
    "path": "cicd/checkov_resource_count_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-22 15:20:27 +0000 (Tue, 22 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts the number of resources Checkov is scanning across all given repos\n\nUseful to estimate Bridgecrew Cloud costs for all repos which are charged per resource\n\nThis can take a while to run on large directories with lots of resources\n\nAny customization to the 'checkov' settings must use local .checkov.yaml config file in each repo root\nsuch as which directories to scan or skip, see this working config for example:\n\n    https://github.com/HariSekhon/Templates/blob/master/.checkov.yaml\n\n\nEach given repo dir should be the root of the repo so that .checkov.yaml can be found\n\n\nUses adjacent script:\n\n    checkov_resource_count.sh\n\n\nRequires Checkov, awk and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo_dir1> [<repo_dir2> <repo_dir3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor dir in \"$@\"; do\n    timestamp \"Scanning Checkov resources in directory '$dir'\"\n    \"$srcdir/checkov_resource_count.sh\" \"$dir\"\ndone |\nawk '{ total += $1 } END { print total }'\n"
  },
  {
    "path": "cicd/cloudbuild.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-12-19 16:27:26 +0000 (Sat, 19 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         G C P   C l o u d   B u i l d\n# ============================================================================ #\n\n# References:\n#\n#     https://cloud.google.com/cloud-build/docs/build-config\n#\n#     https://cloud.google.com/cloud-build/docs/build-debug-locally\n\n\n# gcloud builds submit --config cloudbuild.yaml .\n#\n# cloud-build-local --config cloudbuild.yml --dryrun=false .\n\n# tars $PWD to bucket called ${PROJECT_ID}_cloudbuild\n\ntimeout: 3660s\n\nsteps:\n  - name: ubuntu:18.04\n    entrypoint: bash\n    args:\n      - '-c'\n      - |\n        setup/ci_bootstrap.sh &&\n        make build test\n    timeout: 3600s\n"
  },
  {
    "path": "cicd/codefresh_cancel_delayed_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 18:02:32 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nCancels Codefresh delayed builds using the CodeFresh CLI\n\nRequires Codefresh CLI to be installed and configured (see setup/setup_codefresh.sh)\n\"\n\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhelp_usage \"$@\"\n\ncodefresh get builds -s delayed -l 500 |\ntail -n +2 |\nsed 's/delayed.*//' |\nwhile read -r id name; do\n    echo \"cancelling delayed build '$name' id '$id'\"\n    codefresh terminate \"$id\"\n    echo\ndone\n"
  },
  {
    "path": "cicd/concourse.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-19 19:21:31 +0000 (Thu, 19 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Start a quick local Concourse CI\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nBoots Concourse CI in Docker, and builds the current repo\n\n- boots Concourse container in Docker\n- creates job pipeline from \\$PWD/.concourse.yml\n- unpauses pipeline and job\n- triggers pipeline job\n- follows job build log in CLI\n- prints recent build statuses\n\n    ${0##*/} [up]\n\n    ${0##*/} down\n\n    ${0##*/} ui     - prints the Concourse URL and automatically opens in browser\n\nIdempotent, you can re-run this and continue from any stage\n\nSee Also:\n\n    fly.sh - wraps the fly command specifying target from the environment variable FLY_TARGET to avoid repetition. Also automatically downloads the 'fly' utility if not present in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nexport CONCOURSE_USER=\"${CONCOURSE_USER:-test}\"\nexport CONCOURSE_PASSWORD=\"${CONCOURSE_PASSWORD:-test}\"\n\nexport CONCOURSE_HOST=localhost\nexport CONCOURSE_PORT=8081\n\nprotocol=\"http\"\nif [ -n \"${CONCOURSE_SSL:-}\" ]; then\n    protocol=https\nfi\n\nCONCOURSE_URL=\"$protocol://$CONCOURSE_HOST:$CONCOURSE_PORT\"\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/concourse.yml\"\n\nexport FLY_TARGET=\"ci\"\n\npipeline=\"${PWD##*/}\"\njob=\"$pipeline/build\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nif ! [ -f \"$COMPOSE_FILE\" ]; then\n    timestamp \"downloading Concourse CI docker-compose.yml\" # this is in Git so shouldn't run any more\n    wget -O \"$COMPOSE_FILE\" https://concourse-ci.org/docker-compose.yml\nfi\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting Concourse:\"\n    docker-compose up -d \"$@\"\n    echo >&2\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"Concourse URL:  $CONCOURSE_URL\"\n    echo\n    echo \"Concourse user:     $CONCOURSE_USER\"\n    echo \"Concourse password: $CONCOURSE_PASSWORD\"\n    echo\n    open \"$CONCOURSE_URL\"\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo >&2\n    exit 0\nfi\n\nexport PATH=\"$PATH:\"~/bin\n\nwhen_url_content \"$CONCOURSE_URL\" '(?i:concourse)' # Concourse\necho\n\n# done in fly.sh now\n# which checks for executable which command -v and type -P don't\n# shellcheck disable=SC2230\n#if [ \"$action\" = up ] &&\n#   ! which fly &>/dev/null; then\n#    # fly.sh has ~/bin in $PATH\n#    dir=~/bin\n#    mkdir -pv \"$dir\"\n#    os=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n#    echo \"Downloading fly for OS = $os\"\n#    wget -cO \"$dir/fly\" \"http://$CONCOURSE_HOST:$CONCOURSE_PORT/api/v1/cli?arch=amd64&platform=$os\"\n#    chmod +x \"$dir/fly\"\n#    echo\n#fi\n\ntimestamp \"fly login:\"\n\"$srcdir/fly.sh\" login -c \"$CONCOURSE_URL\" -u \"$CONCOURSE_USER\" -p \"$CONCOURSE_PASSWORD\"\necho\n\nconcourse_yml=\".concourse.yml\"\n\nif ! [ -f \"$concourse_yml\" ]; then\n    timestamp \"Concourse configuration file '$concourse_yml' not found\"\n    echo >&2\n    timestamp \"Skipping loading missing pipeline\"\n    echo >&2\n    timestamp \"Re-run this from the directory containing '$concourse_yml' to auto-load a pipeline\"\n    exit 0\nfi\n\ntimestamp \"updating pipeline: $pipeline\"\ntimestamp \"loading config from $concourse_yml\"\n# fly sp\nset +o pipefail\nyes | \"$srcdir/fly.sh\" set-pipeline -p \"$pipeline\" -c \"$concourse_yml\"\nset -o pipefail\necho\n\ntimestamp \"unpausing pipeline: $pipeline\"\n# fly up\n\"$srcdir/fly.sh\" unpause-pipeline -p \"$pipeline\"\necho\n\ntimestamp \"unpausing job: $job\"\n# fly uj\n\"$srcdir/fly.sh\" unpause-job --job \"$job\"\n\n#\"$srcdir/fly.sh\" trigger-job -j \"$job\"\n#\"$srcdir/fly.sh\" watch -j \"$job\"\n\necho\necho \"Concourse URL:  $CONCOURSE_URL\"\necho\n\n# trigger + watch together\n\"$srcdir/fly.sh\" trigger-job -j \"$job\" -w\n\necho\n\"$srcdir/fly.sh\" builds\n"
  },
  {
    "path": "cicd/coveralls_latest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-12 00:07:36 +0100 (Sun, 12 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nGets the latest Coveralls.io build info for a given repo\n\nIf no repo argument is given, then uses the first GitHub remote from the local git repo\n\nRepo should be qualified with username and is case sensitive\n\neg.\n\ncoveralls_latest.sh HariSekhon/pylib\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"user/repo [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    repo=\"$1\"\nelse\n    repo=\"$(github_user_repo)\"\nfi\n\n# could add ?page=1 to get the latest 10 builds and their coverage changes\ncurl -sSL \"https://coveralls.io/github/$repo.json\" \"$@\"\n# don't pass to jq in case repo doesn't exist you'll get back HTML and a weird\n# | jq\n"
  },
  {
    "path": "cicd/fly.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-21 18:24:59 +0000 (Sat, 21 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nexport PATH=\"$PATH:/usr/local/bin:\"~/bin\n\ntarget=\"${FLY_TARGET:-}\"\n\nopts=()\nif [ -n \"$target\" ]; then\n    opts+=(-t \"$target\")\nfi\n\n# check fly is in $PATH before we exec which would close the shell if called directly, losing the error message\n# which checks for executable which command -v and type -P don't\n# shellcheck disable=SC2230\nif ! which fly &>/dev/null; then\n    if [ -n \"${CONCOURSE_URL:-}\" ] ||\n       [ -n \"${CONCOURSE_HOST:-}\" ]; then\n        protocol=\"http\"\n        if [ -n \"${CONCOURSE_SSL:-}\" ]; then\n            protocol=https\n        fi\n        # give priority to CONCOURSE_URL, otherwise CONCOURSE_HOST must exist and we can construct from there\n        if [ -z \"${CONCOURSE_URL:-}\" ]; then\n            CONCOURSE_URL=\"$protocol://$CONCOURSE_HOST:${CONCOURSE_PORT:-8080}\"\n        fi\n        # this is in $PATH above\n        dir=~/bin\n        mkdir -pv \"$dir\"\n        os=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n        echo \"Downloading fly for OS = $os\" >&2\n        wget -cO \"$dir/fly\" \"http://$CONCOURSE_HOST:$CONCOURSE_PORT/api/v1/cli?arch=amd64&platform=$os\"\n        chmod +x \"$dir/fly\"\n        echo\n    else\n        echo \"'fly' command not found in \\$PATH ($PATH)\" >&2\n        echo >&2\n        echo \"\\$CONCOURSE_URL and \\$CONCOURSE_HOST are both unset, cannot download from concourse automatically\" >&2\n        exit 1\n    fi\nfi\n\nexec fly \"${opts[@]}\" \"$@\"\n"
  },
  {
    "path": "cicd/generate_status_page.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-07 15:01:31 +0000 (Fri, 07 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to generate STATUS.md containing all GitHub repos and DockerHub / Docker Cloud repos build statuses on a single page\n#\n# uses adjacent github_generate_status_page.sh and docker_generate_status_page.sh scripts\n#\n#   GITHUB_USER=HariSekhon DOCKER_USER=harisekhon ./generate_status_page.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ntrap 'echo ERROR >&2' exit\n\ncd \"$srcdir\"\n\nfile=\"STATUS.md\"\n\necho\necho \"Generating STATUS.md\"\necho\n{\n\"$srcdir/../github/github_generate_status_page.sh\"\necho\necho \"---\"\necho\n#\"$srcdir/../docker/docker_generate_status_page.sh\"\necho\necho https://git.io/hari-ci\n} | tee \"$file\"\n\necho\necho\necho \"Generating STARCHARTS.md\"\necho\n\"$srcdir/../github/github_generate_starcharts.md.sh\"\n\ntrap '' exit\n"
  },
  {
    "path": "cicd/gerrit.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-08 13:22:34 +0100 (Wed, 08 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nRuns Gerrit via docker-compose/gerrit.yml\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"arg [<options>]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nGERRIT_VERSION=\"3.1.3\"\n\ngerrit_local(){\n    GERRIT_SITE=~/gerrit_site\n\n    WAR=\"$GERRIT_SITE/gerrit-$GERRIT_VERSION.war\"\n\n    help_usage \"$@\"\n\n    mkdir -pv \"$GERRIT_SITE\"\n\n    wget -O \"$WAR\" \"https://gerrit-releases.storage.googleapis.com/gerrit-$GERRIT_VERSION.war\"\n\n    java -jar \"$WAR\" init --batch --dev -d \"$GERRIT_SITE\"\n\n    # restrict to localhost as per best practice\n    git config --file \"$GERRIT_SITE/etc/gerrit.config\" httpd.listenUrl 'http://localhost:8080'\n\n    \"$GERRIT_SITE/bin/gerrit.sh\" restart\n}\n\ngerrit_docker(){\n    #docker run -d -ti -p 8080:8080 -p 29418:29418 gerritcodereview/gerrit:\"$GERRIT_VERSION\"\n    VERSION=\"$GERRIT_VERSION\" docker-compose -f \"$srcdir/../docker-compose/gerrit.yml\" up -d\n}\n\ngerrit_docker\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    when_ports_available 120 localhost 8080\n    when_url_content 60 'http://localhost:8080' Gerrit\n    open 'http://localhost:8080'\nfi\n"
  },
  {
    "path": "cicd/gerrit_projects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-08 17:53:56 +0100 (Wed, 08 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://gerrit-review.googlesource.com/Documentation/rest-api-projects.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nQueries Gerrit Code Review API for Project List, outputting in CSV format\n\nOutput Format:\n\n<Project ID> , <State> , <Parent Project ID>, <Groups> , <Initial Commit Timestamp>, <Latest Master Commit Timestamp>\n\nThe following environment variables should be set before running:\n\n\\$GERRIT_HOST (default: localhost)\n\\$GERRIT_PORT (default: 8080)\n\\$GERRIT_SSL = 1 (default: blank which is off)\n\\$GERRIT_URL_PREFIX (default: blank, you might need to set /gerrit)\n\n\\$GERRIT_USER\n\\$GERRIT_PASSWORD\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhost=\"${GERRIT_HOST:-localhost}\"\nport=\"${GERRIT_PORT:-8080}\"\n\ncheck_env_defined \"GERRIT_USER\"\ncheck_env_defined \"GERRIT_PASSWORD\"\n\nhelp_usage \"$@\"\n\nprotocol=\"http\"\nif [ -n \"${GERRIT_SSL:-}\" ]; then\n    protocol=\"https\"\nfi\n\nexport USER=\"$GERRIT_USER\"\nexport PASSWORD=\"$GERRIT_PASSWORD\"\n\nbugfix_gerrit_api_output(){\n    sed \"s/^)]}'//\"\n}\n\ncurl_options=\"-sSL $*\"\n\nurl_prefix=\"\"\nif [ -n \"${GERRIT_URL_PREFIX:-}\" ]; then\n    GERRIT_URL_PREFIX=\"${GERRIT_URL_PREFIX#/}\"\n    GERRIT_URL_PREFIX=\"${GERRIT_URL_PREFIX%/}\"\n    url_prefix=\"/$GERRIT_URL_PREFIX\"\nfi\n\n# /a/ prefix for authenticated API access - https://gerrit-review.googlesource.com/Documentation/rest-api.html#authentication\nbase_url=\"$protocol://$host:$port${url_prefix}/a\"\n\ncurl_auth(){\n    local url_path=\"$1\"\n    shift || :\n    # need opt splitting\n    # shellcheck disable=SC2086\n    \"$srcdir/../bin/curl_auth.sh\" \"$base_url/$url_path\" -H 'Content-type: application/json' \"$@\" $curl_options |\n    bugfix_gerrit_api_output\n}\n\ncurl_auth \"projects/\" |\njq -r 'to_entries | .[] | .value | [.id, .state] | @tsv' |\nwhile read -r project_id state; do\n    parent=\"$(curl_auth \"projects/$project_id/parent\" | tr -d '\\n')\"\n    parent=\"${parent//\\\"}\"\n    groups=\"$(curl_auth \"projects/$project_id/access\" | jq -r '([(.groups | to_entries | .[].value.name)] | join(\",\"))')\"\n    groups=\"${groups//\\\"}\"\n    # might not have a timestamp, in which case ignore failures\n    # Also the reflog might get limited and not return the real full git lot to find the initial commit.\n    # It doesn't look like this creation timestamp is available in the /projects/ API directly\n    initial_commit_timestamp=\"$(curl_auth \"projects/$project_id/branches/master/reflog\" | jq -r 'last | .who.date' 2>/dev/null || :)\"\n    initial_commit_timestamp=\"${initial_commit_timestamp//\\\"}\"\n    latest_commit_timestamp=\"$(curl_auth \"projects/$project_id/branches/master/reflog\" | jq -r 'limit(1; .[] | .who.date)' 2>/dev/null || :)\"\n    latest_commit_timestamp=\"${latest_commit_timestamp//\\\"}\"\n    echo \"\\\"$project_id\\\",\\\"$state\\\",\\\"$parent\\\",\\\"$groups\\\",\\\"$initial_commit_timestamp\\\", \\\"$latest_commit_timestamp\\\"\"\ndone\n"
  },
  {
    "path": "cicd/gocd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-27 19:16:35 +0000 (Fri, 27 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nBoots a GoCD CI cluster with server and agent(s) in Docker, and builds the current repo\n\n- boots GoCD server and agent(s) (one by default) in Docker\n- authorizes the agent(s) to begin building\n- loads the config repo if gocd_config_repo.json is found in \\$PWD or \\$PWD/setup/\n- opens the GoCD web UI\n\n    ${0##*/} [up]\n\n    ${0##*/} down\n\n    ${0##*/} ui     - prints the GoCD Server URL and automatically opens in browser\n\nIdempotent, you can re-run this and continue from any stage\n\nSee Also:\n\n    gocd_api.sh - this script makes use of it to handle API calls as part of the setup\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nexport GOCD_URL=\"http://${GOCD_HOST:-localhost}:${GOCD_PORT:-8153}\"\nurl=\"$GOCD_URL/go/pipelines#!/\"\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/gocd.yml\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\n#git_repo=\"$(git remote -v | grep github.com | sed 's/.*github.com/https:\\/\\/github.com/; s/ .*//')\"\n#repo=\"${git_repo##*/}\"\n\n# load .gocd.yaml from this github location\n# doesn't work - see https://github.com/gocd/gocd/issues/7930\n# also, caused gocd-server to be recreated from different repos due to this differing environment variable each time\n# which is not ideal as we want to boot GoCD from any repo and then incrementally add any builds from other repos, or load all via:\n#\n# git_foreach_repo.sh gocd.sh\n#\n#if [ -n \"$git_repo\" ]; then\n#    export CONFIG_GIT_REPO=\"$git_repo\"\n#fi\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting GoCD cluster:\"\n    docker-compose up -d \"$@\"\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"GoCD Server URL:  $GOCD_URL\"\n    \"$srcdir/../bin/urlopen.sh\" \"$GOCD_URL\"\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo >&2\n    exit 0\nfi\n\nwhen_url_content \"$GOCD_URL\" '(?i:gocd)'\necho >&2\n\nSECONDS=0\nmax_secs=300\n# don't use --fail here, it'll exit the loop prematurely\nwhile curl -sS \"$GOCD_URL\" | grep -q 'GoCD server is starting'; do\n    timestamp 'waiting for server to finish starting up and remove message \"GoCD server is starting\"'\n    if [ $SECONDS -gt $max_secs ]; then\n        die \"GoCD server failed to start within $max_secs seconds\"\n    fi\n    sleep 3\ndone\necho >&2\n\n# needs this header, otherwise gets 404\nget_agents(){\n    \"$srcdir/gocd_api.sh\" \"/agents\"\n}\n# TODO: refine this to only connected agents\nget_agent_count(){\n    get_agents |\n    jq '._embedded.agents | length'\n}\n\ntimestamp \"getting list of expected agents\"\nexpected_agents=\"$(docker-compose config | awk '/^[[:space:]]+gocd-agent.*:[[:space:]]*$/ {print $1}' | sed 's/://g; s/[[:space:]]//g; /^[[:space:]]*$/d')\"\nnum_expected_agents=\"$(grep -c . <<< \"$expected_agents\" || :)\"\n\nSECONDS=0\ntimestamp \"Waiting for $num_expected_agents expected agent(s) connect before enabling them:\"\nwhile true; do\n    num_connected_agents=\"$(get_agent_count)\"\n    #if get_agents | grep -q hostname; then\n    if [ \"$num_connected_agents\" -ge \"$num_expected_agents\" ]; then\n        break\n    fi\n    if [ $SECONDS -gt $max_secs ]; then\n        timestamp \"giving up waiting for connected agents after $max_secs\"\n        break\n    fi\n    timestamp \"connected agents: $num_connected_agents\"\n    sleep 3\ndone\necho\n\necho \"Enabling agent(s):\"\necho\nget_agents |\njq -r '._embedded.agents[] | [.hostname, .uuid] | @tsv' |\nwhile read -r hostname uuid; do\n    for expected_agent in $expected_agents; do\n        # grep -f would be easier but don't want to depend on have the GNU version installed and then remapped via func\n        #if [[ \"$hostname\" =~ ^$expected_agent(-[[:digit:]]+)?$ ]]; then\n        if [[ \"$hostname\" =~ ^$expected_agent$ ]]; then\n            timestamp \"enabling expected agent '$hostname' with uuid '$uuid'\"\n            \"$srcdir/gocd_api.sh\" \"/agents/$uuid\" -X PATCH -d '{ \"agent_config_state\": \"Enabled\" }' || :  # don't stop, try enabling all agents\n            echo\n            continue 2\n        fi\n    done\n    timestamp \"WARNING: unauthorized agent '$hostname' was not expected, not automatically enabling\"\ndone\n\necho\ntimestamp \"GoCD is up and ready\"\necho\necho \"GoCD Server URL:  $url\"\necho\nopen \"$url\"\n\nrepo_config=\"\"\nfilename=\"gocd_config_repo.json\"\nif [ -f \"setup/$filename\" ]; then\n    repo_config=\"setup/$filename\"\nelif [ -f \"$filename\" ]; then\n    repo_config=\"$filename\"\nfi\n\nif [ -n \"$repo_config\" ]; then\n    timestamp \"GoCD repo configuration file '$filename' not found\"\n    echo >&2\n    timestamp \"Skipping loading missing pipeline\"\n    echo >&2\n    timestamp \"Re-run this from the directory containing '$filename' or 'setup/$filename' to auto-load a pipeline\"\n    exit 0\nfi\n\ntimestamp \"(re)creating config repo:\"\necho >&2\n\nconfig_repo=\"$(jq -r '.id' \"$repo_config\")\"\n\n# XXX: these config_repo endpoints don't work unless v3 is set\ntimestamp \"deleting config repo if already existing:\"\n\"$srcdir/gocd_api.sh\" \"/admin/config_repos/$config_repo\" \\\n     -H 'Accept:application/vnd.go.cd.v3+json' \\\n     -X DELETE || :\necho >&2\necho >&2\n\n# XXX: these config_repo endpoints don't work unless v3 is set\ntimestamp \"creating config repo:\"\n\"$srcdir/gocd_api.sh\" \"/admin/config_repos\" \\\n     -H 'Accept:application/vnd.go.cd.v3+json' \\\n     -X POST -d @\"$repo_config\"\necho >&2\necho >&2\n"
  },
  {
    "path": "cicd/gocd_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-28 22:33:44 +0000 (Sat, 28 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the GoCD API v6\n\n\\$GOCD_URL or \\$GOCD_HOST must be set to point to the gocd server\n\nIf \\$GOCD_HOST is used, then the following may also be set:\n\n\\$GOCD_PORT - defaults to 8111 and is only used if \\$GOCD_URL is not used\n\\$GOCD_SSL  - defaults to http, any value enables https\n\\$GOCD_API_VERSION - defaults to v6\n\nSets Accept and Content-Type headers to application/json unless specified on the command line arguments.\nBeware that Accept header also sets the \\$GOCD_API_VERSION, so you'd have to do that yourself if overriding via command line args.\n\nImportant: Some API endpoints require older endpoints, such as /admin/config_repos which requires:\n\n    -H 'Accept:application/vnd.go.cd.v3+json'\n\n\nAPI Reference:\n\n    https://api.gocd.org/current/\n\n\nSee Also:\n\n    gocd.sh - boots a GoCD cluster in Docker and makes heavy use of this script against many API endpoints to configure it\n                  convenient way of getting a GoCD API to test this script against, outputs the GOCD_URL and GOCD_TOKEN for you\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path##/}\"\n\n# don't enforce as hard requirements here, instead try alternation further down and construct from what's available\n#check_env_defined \"GOCD_URL\"\n#check_env_defined \"GOCD_HOST\"\n#check_env_defined \"GOCD_TOKEN\"\n\n# not using curl_api_opts because need more control over the accept headers which control API version\nCURL_OPTS=(-sS --fail --connect-timeout 3 \"${CURL_OPTS[@]}\")\n\nif ! [[ \"$*\" =~ Accept: ]]; then\n    if [ -n \"${GOCD_API_VERSION:-}\" ]; then\n        GOCD_API_VERSION=\"${GOCD_API_VERSION#v}\"\n        if ! [[ \"$GOCD_API_VERSION\" =~ ^[[:digit:]]+$ ]]; then\n            usage \"invalid GOCD_API_VERSION defined in environment - must be an integer\"\n        fi\n    else\n        GOCD_API_VERSION=6\n    fi\n    CURL_OPTS+=(-H \"Accept: application/vnd.go.cd.v${GOCD_API_VERSION}+json\")\nfi\nif ! [[ \"$*\" =~ Content-Type: ]]; then\n    CURL_OPTS+=(-H \"Content-Type: application/json\")\nfi\n\nif [ -n \"${GOCD_URL:-}\" ]; then\n    url_base=\"${GOCD_URL%%/}\"\nelse\n    protocol=\"http\"\n    if [ -n \"${GOCD_SSL:-}\" ]; then\n        protocol=\"https\"\n    fi\n    [ -n \"${GOCD_HOST:-}\" ] || usage \"neither \\$GOCD_URL nor \\$GOCD_HOST defined in environment\"\n    host=\"$GOCD_HOST\"\n    port=\"${GOCD_PORT:-8153}\"\n    url_base=\"$protocol://$host:$port\"\nfi\n\nurl_base+=\"/go/api\"\n\nif [ -n \"${GOCD_TOKEN:-}\" ]; then\n    export TOKEN=\"$GOCD_TOKEN\"\n    \"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nelse\n    curl \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nfi\n"
  },
  {
    "path": "cicd/octopus_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-02 19:20:38 +0100 (Tue, 02 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Octopus Deploy API\n\nRequires the following environment variables to be set:\n\n    \\$OCTOPUS_URL / \\$OCTOPUS_CLI_SERVER - including the http scheme eg. http://localhost:8080\n    \\$OCTOPUS_TOKEN / \\$OCTOPUS_CLI_API_KEY - your personal API key generated in your profile\n\n\nGenerate your API key at Profile -> My API Keys, eg:\n\n    http://localhost:8080/app#/Spaces-1/users/me/apiKeys\n\n\nAPI Swagger Reference can be found at \\$OCTOPUS_URL/swaggerui, eg:\n\n    http://localhost:8080/swaggerui/index.html\n\n\nAPI Documentation:\n\n    https://octopus.com/docs/octopus-rest-api\n\n\nExamples:\n\n    # list the other endpoints\n    ${0##*/} /\n\n    ${0##*/} /authentication\n    ${0##*/} /configuration\n    ${0##*/} /dashboard\n    ${0##*/} /dashboardconfiguration\n    ${0##*/} /machines\n    ${0##*/} /featuresConfiguration\n    ${0##*/} /licenses/licenses-current\n    ${0##*/} /packages\n    ${0##*/} /projects\n    ${0##*/} /tasks\n    ${0##*/} /workers\n\n\nSee Also:\n\n    install/install_octo.sh - Installs the Octopus Deploy CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nOCTOPUS_URL=\"${OCTOPUS_URL:-${OCTOPUS_CLI_SERVER:-}}\"\nOCTOPUS_TOKEN=\"${OCTOPUS_TOKEN:-${OCTOPUS_CLI_API_KEY:-}}\"\n\ncheck_env_defined \"OCTOPUS_URL\"\ncheck_env_defined \"OCTOPUS_TOKEN\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nexport TOKEN=\"$OCTOPUS_TOKEN\"\nexport CURL_AUTH_HEADER=\"X-Octopus-ApiKey:\"\n\nurl_base=\"$OCTOPUS_URL/api\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "cicd/run_latest_tests.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-23 17:19:22 +0000 (Sat, 23 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nVERSION=latest \"$srcdir/run_tests.sh\" \"$@\"\n"
  },
  {
    "path": "cicd/run_tests.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-23 17:19:22 +0000 (Sat, 23 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\nsection \"Running Test Scripts\"\n\ndeclare_if_inside_docker\n\nstart_time=\"$(start_timer)\"\n\nfilter=\"${1:-.*}\"\nsearch_dir=\"${2:-.}\"\n\nscripts=\"$(find \"$search_dir\" -maxdepth 2 -type f -iname 'test_*.sh' | grep -E -- \"test_$filter\" | sort -f || :)\"\n\nfor script in $scripts; do\n    date\n    script_start_time=\"$(date +%s)\"\n    echo\n    declare_if_inside_docker\n    # quoting VERSION passes blank which prevents populating with default versions\n    # shellcheck disable=SC2086\n    \"./$script\" ${VERSION:-}\n    echo\n    date\n    echo\n    script_end_time=\"$(date +%s)\"\n    script_time_taken=\"$((script_end_time - script_start_time))\"\n    echo \"Completed in $script_time_taken secs\"\ndone\n\ntime_taken \"$start_time\"\nsection2 \"Test Scripts Completed\"\necho\n"
  },
  {
    "path": "cicd/selenium_hub_wait_ready.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-05-16 10:15:00 +0100 (Sun, 16 May 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWaits for Selenium Grid Hub status to be ready\n\nIf Selenium hub url isn't given, can construct it from\n\n    \\$SELENIUM_HUB_URL\n        or\n    http://\\$SELENIUM_HUB_HOST:\\$SELENIUM_HUB_PORT\n\nIf \\$SELENIUM_HUB_SSL is set, or \\$SELENIUM_HUB_PORT is 443, will enable https\n\nMax secs may use environment variable \\$MAX_SECS, or defaults to 300\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<selenium_grid_hub_url> [<max_secs>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    hub_url=\"$1\"\nelse\n    if [ -n \"${SELENIUM_HUB_URL:-}\" ]; then\n        hub_url=\"$SELENIUM_HUB_URL\"\n    elif [ -n \"${SELENIUM_HUB_HOST:-}\" ]; then\n        hub_url=\"http://$SELENIUM_HUB_HOST:${SELENIUM_HUB_PORT:-4444}\"\n        if [ -n \"${SELENIUM_HUB_SSL:-}\" ] ||\n           [ \"${SELENIUM_HUB_PORT:-}\" = 443 ]; then\n            hub_url=\"https://${hub_url#http://}\"\n        fi\n    else\n        usage \"selenium hub url not given and \\$SELENIUM_HUB_URL / \\$SELENIUM_HUB_HOST not set\"\n    fi\nfi\nmax_secs=\"${2:-${MAX_SECS:-300}}\"\n\nhub_url=\"${hub_url%%/}\"\nhub_url=\"${hub_url%/wd/hub}\"\n\nif [ -n \"$max_secs\" ] &&\n   [[ \"$max_secs\" =~ ^[[:digit:]]+$ ]]; then\n    timestamp \"setting timeout to $max_secs secs\"\n    TMOUT=\"$max_secs\"\n    # shellcheck disable=SC2064\n    trap_cmd \"echo 'Timed out waiting for Selenium Grid Hub to come up after $max_secs secs' >&2; exit 1\"\nfi\n\nwhile :; do\n    status=\"$(curl -sSL \"$hub_url/wd/hub/status\")\"\n    ready=\"$(jq -r '.value.ready' <<< \"$status\" || die \"FAILED to parse Selenium Hub response: $status\" >&2)\"\n    if [[ \"$ready\" =~ true ]]; then\n        timestamp \"Selenium Grid Hub is up\"\n        break\n    fi\n    timestamp 'Waiting for Selenium Grid Hub to be ready'\n    sleep 1\ndone\n\nuntrap\n"
  },
  {
    "path": "cicd/sonarlint_generate_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-25 22:39:30 +0700 (Tue, 25 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates the .sonarlint/connectedMode.json config at the root of the Git repo\nfrom the sonar-project.properties file\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<path/to/sonar-project.properties>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nsonar_project_properties=\"${1:-sonar-project.properties}\"\n\ngit_root=\"$(git_root)\"\n\ntimestamp \"Switching to Git root: $git_root\"\necho\ncd \"$git_root\"\n\ntimestamp \"Parsing $PWD/$sonar_project_properties\"\norg=\"$(awk -F= '/^[[:space:]]*sonar.organization/{print $2}' \"$sonar_project_properties\" | sed 's/[[:space:]]//g')\"\nproject_key=\"$(awk -F= '/^[[:space:]]*sonar.projectKey/{print $2}' \"$sonar_project_properties\" | sed 's/[[:space:]]//g')\"\n\nmkdir -p -v .sonarlint\n\nsonarlint_config=\"$PWD/.sonarlint/connectedMode.json\"\n\ntimestamp \"Writing to $sonarlint_config\"\ncat <<EOF | tee \"$sonarlint_config\"\n{\n    \"sonarCloudOrganization\": \"$org\",\n    \"projectKey\": \"$project_key\"\n}\nEOF\n"
  },
  {
    "path": "cicd/sync_bootstraps_to_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncd \"$srcdir\"\n\nfilelist=\"\nsetup/bootstrap.sh\nsetup/ci_bootstrap.sh\nsetup/ci_git_set_dir_safe.sh\n\"\n\nif [ -n \"$*\" ]; then\n    echo \"$@\"\nelse\n    sed 's/#.*//; s/:/ /' \"$srcdir/../setup/repos.txt\"\nfi |\ngrep -v -e bash-tools -e '^[[:space:]]*$' |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$(tr '[:upper:]' '[:lower:]' <<< \"$repo\")\"\n    fi\n    # filtered above\n    #if ls -lLdi \"$dir\" \"$srcdir\" | awk '{print $1}' | uniq -d | grep -q .; then\n    #    echo \"skipping $dir as it's our directory\"\n    #    continue\n    #fi\n    if ! [ -d \"../$dir\" ]; then\n        echo \"WARNING: repo dir $dir not found, skipping...\"\n        continue\n    fi\n    for filename in $filelist; do\n        target=\"../$dir/$filename\"\n        mkdir -pv \"${target%/*}\"\n        echo \"syncing $filename -> $target\"\n        perl -pe \"\n            s/directory=\\\"(devops-)*bash-tools/directory=\\\"$dir/;\n            s/(devops-)*bash-tools/$repo/i;\n            s/make install/make/;\" \\\n            \"$filename\" > \"$target\"\n        if [ \"$repo\" = \"nagios-plugins\" ]; then\n            perl -pi -e 's/^(\\s*make)$/$1 build zookeeper/' \"$target\"\n        fi\n    done\ndone\n"
  },
  {
    "path": "cicd/sync_ci_to_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSyncs CI configs listed in ../setup/ci.txt to all adjacent repo checkouts listed in ../setup/repos.txt\n\nQuick way of updating CI/CD configs between repos\n\nand then using ../git/github_foreach_repo.sh to commit them\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<one_config_file_to_sync>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncd \"$srcdir/..\"\n\ntmpfile=\"$(mktemp)\"\n\nsed 's/#.*//; s/:/ /' \"$srcdir/../setup/repos.txt\" |\ngrep -vi -e bash-tools \\\n         -e jenkins \\\n         -e github-actions \\\n         -e playlist \\\n         -e sql-scripts \\\n         -e sql-keywords \\\n         -e teamcity \\\n         -e '^[[:space:]]*$' |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$(tr '[:upper:]' '[:lower:]' <<< \"$repo\")\"\n    fi\n    # filtered above\n    #if ls -lLdi \"$dir\" \"$srcdir\" | awk '{print $1}' | uniq -d | grep -q .; then\n    #    timestamp \"skipping $dir as it's our directory\"\n    #    continue\n    #fi\n    if ! [ -d \"../$dir\" ]; then\n        timestamp \"WARNING: repo dir $dir not found, skipping...\"\n        continue\n    fi\n    if [ -n \"$*\" ]; then\n        for filename in \"$@\"; do\n            echo \"$filename\"\n        done\n    else\n        sed 's/#.*//; /^[[:space:]]*$/d' \"$srcdir/../setup/ci.txt\"\n    fi |\n    while read -r filename; do\n        target=\"../$dir/$filename\"\n        if [ -f \"$target\" ] || [ -n \"${NEW:-}\" ]; then\n            :\n        else\n            continue\n        fi\n        perl -pe \"s/(?<!- )(devops-)*bash-tools/$repo/i\" \"$filename\" > \"$tmpfile\"\n        sed -i \"s/directory=\\\"$repo\\\"/directory=\\\"$dir\\\"/g\" \"$tmpfile\"\n        if is_mac; then\n            octal_perms=\"$(stat -f \"%A\" \"$filename\")\"\n        else\n            octal_perms=\"$(stat -c \"%a\" \"$filename\")\"\n        fi\n        chmod \"$octal_perms\" \"$tmpfile\"\n        tmpfile_checksum=\"$(cksum \"$tmpfile\" | awk '{print $1}')\"\n        target_checksum=\"$(cksum \"$target\" | awk '{print $1}')\"\n        if [ \"$tmpfile_checksum\" = \"$target_checksum\" ]; then\n            log \"Skipping CI/CD Config Sync for file due to same checksum: $filename\"\n            continue\n        fi\n        if ! QUIET=1 \"$srcdir/../bin/diff_line_threshold.sh\" \"$filename\" \"$target\"; then\n            timestamp \"Skipping CI/CD Config Sync for file due to large diff: $filename\"\n            continue\n        fi\n        mkdir -pv \"${target%/*}\"\n        timestamp \"Syncing $filename -> $target\"\n        mv \"$tmpfile\" \"$target\"\n    done\ndone\n\"$srcdir/sync_github_actions_workflows_to_adjacent_repos.sh\" \"$@\"\n"
  },
  {
    "path": "cicd/sync_configs_to_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSyncs configs listed in ../setup/files.txt to all adjacent repo checkouts listed in ../setup/repos.txt\n\nQuick way of updating dotfile configs between repos\n\nand then using ../git/github_foreach_repo.sh to commit them\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files_to_sync>]\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncd \"$srcdir/..\"\n\ntmpfile=\"$(mktemp)\"\n\nif [ -n \"$*\" ]; then\n    echo \"$@\"\nelse\n    sed 's/#.*//; s/:/ /' \"$srcdir/../setup/repos.txt\"\nfi |\ngrep -vi -e bash-tools \\\n         -e playlist \\\n         -e '^[[:space:]]*$' |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$(tr '[:upper:]' '[:lower:]' <<< \"$repo\")\"\n    fi\n    if ! [ -d \"../$dir\" ]; then\n        timestamp \"WARNING: repo dir $dir not found, skipping...\"\n        continue\n    fi\n    if [ $# -gt 0 ]; then\n        echo \"$@\" | tr '[:space:]' '\\n'\n    else\n        sed 's/#.*//; /^[[:space:]]*$/d' \"$srcdir/../setup/files.txt\"\n    fi |\n    while read -r filename; do\n        if [ \"$filename\" = .gitignore ]; then\n            continue\n        fi\n        target=\"../$dir/$filename\"\n        if [ -f \"$target\" ] || [ -n \"${NEW:-}\" ]; then\n            :\n        else\n            continue\n        fi\n        if [ -f \"$filename\" ]; then\n            perl -pe \"s/(devops-)*bash-tools/$repo/i\" \"$filename\" > \"$tmpfile\"\n            tmpfile_checksum=\"$(cksum \"$tmpfile\" | awk '{print $1}')\"\n            target_checksum=\"$(cksum \"$target\" | awk '{print $1}')\"\n            if [ \"$tmpfile_checksum\" = \"$target_checksum\" ]; then\n                log \"Skipping Config Sync for file due to same checksum: $filename\"\n                continue\n            fi\n            if ! QUIET=1 \"$srcdir/../bin/diff_line_threshold.sh\" \"$filename\" \"$target\"; then\n                timestamp \"Skipping Config Sync for file due to large diff: $filename\"\n                continue\n            fi\n            mkdir -pv \"${target%/*}\"\n            timestamp \"Syncing $filename -> $target\"\n            mv \"$tmpfile\" \"$target\"\n        else\n            timestamp \"File not found: $filename. Skipping...\"\n        fi\n    done\n    direnv allow \"../$dir\"\ndone\n"
  },
  {
    "path": "cicd/sync_github_actions_workflows_to_adjacent_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSyncs GitHub Actions CI/CD workflows in this repo to all adjacent repo checkouts listed in ../setup/repos.txt\n\nQuick way of updating GitHub Actions CI/CD workflows between repos\n\nand then using ../git/github_foreach_repo.sh to commit them\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<workflow_yaml_1> <workflow_yaml_2> ...]\"\n\nhelp_usage \"$@\"\n\ncd \"$srcdir/../.github/workflows\"\n\ntmpfile=\"$(mktemp)\"\n\nsync_file(){\n    local filename=\"$1\"\n    local repo=\"$2\"\n    local dir=\"${3:-}\"\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    dir=\"$(tr '[:upper:]' '[:lower:]' <<< \"$dir\")\"\n    if ! [ -d \"$srcdir/../../$dir\" ]; then\n        timestamp \"WARNING: repo dir $srcdir/../../$dir not found, skipping...\"\n        return 0\n    fi\n    target=\"$srcdir/../../$dir/.github/workflows/$filename\"\n    targetdir=\"${target%/*}\"\n    mkdir -p -v \"$targetdir\"\n    if [ -f \"$target\" ]; then\n        perl -p -e \"s/(DevOps-)?Bash-tools/$repo/i\" \"$filename\" > \"$tmpfile\"\n        tmpfile_checksum=\"$(cksum \"$tmpfile\" | awk '{print $1}')\"\n        target_checksum=\"$(cksum \"$target\" | awk '{print $1}')\"\n        if [ \"$tmpfile_checksum\" = \"$target_checksum\" ]; then\n            log \"Skipping GitHub Actions CI/CD Config Sync for file due to same checksum: $filename\"\n            return 0\n        fi\n        if ! QUIET=1 \"$srcdir/../bin/diff_line_threshold.sh\" \"$filename\" \"$target\"; then\n            timestamp \"Skipping GitHub Actions CI/CD Config Sync for file due to large diff: $filename\"\n            return 0\n        fi\n        timestamp \"Syncing $filename -> $target\"\n        mv \"$tmpfile\" \"$target\"\n    else\n        log \"File not found: $target. Skipping...\"\n    fi\n}\n\nsed 's/#.*//; s/:/ /' \"$srcdir/../setup/repos.txt\" |\ngrep -v -e bash-tools \\\n        -e '^[[:space:]]*$' |\nwhile read -r repo dir; do\n    if [ $# -gt 1 ]; then\n        for filename in \"$@\"; do\n            sync_file \"$filename\" \"$repo\" \"$dir\"\n        done\n    else\n        for filename in *.yaml; do\n            sync_file \"$filename\" \"$repo\" \"$dir\"\n        done\n    fi\ndone\n"
  },
  {
    "path": "circleci/circleci_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 12:08:21 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the CircleCI API\n\nRequires \\$CIRCLECI_TOKEN in the environment\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here:\n\n    https://app.circleci.com/settings/user/tokens\n\n\nAPI Reference:\n\n    https://circleci.com/docs/api/v2/\n    https://circleci.com/docs/api/v1/\n\n\nThe CircleCI API isn't very mature, so sometimes you need to go back and use the v1.1 API instead of the default v2 API, so if the path starts with /v1.1 it'll use that API instead\n\n\nExamples:\n\n\n# Get currently authenticated user:\n\n    ${0##*/} /me | jq .\n\n\n# List projects (requires v1.1 API, no endpoint in v2 yet):\n\n    ${0##*/} /v1.1/projects | jq .\n\n\n# Get a project:\n\n    ${0##*/} /project/<vcs>/<user_or_org>/<repo>\n\n    ${0##*/} /project/github/HariSekhon/DevOps-Bash-tools | jq .\n\n\n# Get a list of pipeline runs for a project:\n\n    ${0##*/} /project/<vcs>/<user_or_org>/<repo>/pipeline\n\n    ${0##*/} /project/github/HariSekhon/DevOps-Bash-tools/pipeline | jq .\n\n    # just the pipelines triggered by you\n\n    ${0##*/} /project/<vcs>/<user_or_org>/<repo>/pipeline/mine\n\n    ${0##*/} /project/github/HariSekhon/DevOps-Bash-tools/pipeline/mine | jq .\n\n\n# Get environment variables for a project (see circleci_project_set_env_vars.sh to easily set them):\n\n    ${0##*/} /project/<vcs>/<user_or_org>/<repo>/envvar\n\n    ${0##*/} /project/github/HariSekhon/DevOps-Bash-tools/envvar | jq .\n\n# see circleci_project_set_env_vars.sh to easily set these variables\n\n\n# Create / replace a project environment variable:\n\n    ${0##*/} /project/<vcs>/<user_or_org>/<repo>/envvar -X POST -d '{\\\"name\\\": \\\"AWS_ACCESS_KEY_ID\\\", \\\"value\\\": \\\"AKIA...\\\"}'\n\n    ${0##*/} /project/github/HariSekhon/DevOps-Bash-tools/envvar -X POST -d '{\\\"name\\\": \\\"AWS_ACCESS_KEY_ID\\\", \\\"value\\\": \\\"AKIA...\\\"}' | jq .\n\n\n# List contexts for a user or organization (this org id is different to a user id even for a user's context):\n#                                          (organization ID can be found on organization settings page as it is not currently exposed in the API)\n#                                          (find organization ID here: https://app.circleci.com/settings/organization/VCS/MY_USER_OR_ORG/contexts)\n#                                                                  eg. https://app.circleci.com/settings/organization/github/HariSekhon/contexts\n\n    ${0##*/} '/context/<context_id>/environment-variable' | jq .\n\n\n# Use the context ID from the above output to list environment variable secrets from the given context ID\n\n    ${0##*/} /context/c2321a8a-c188-4568-aac7-aef89cb7a6e2/environment-variable | jq .\n\n# see circleci_context_set_env_vars.sh to easily set these variables\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://circleci.com/api/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nexport TOKEN=\"${CIRCLECI_TOKEN:-}\"\nexport CURL_AUTH_HEADER=\"Circle-Token:\"\n\nurl_path=\"$1\"\nshift || :\n\nif [[ \"$url_path\" =~ ^/?v[[:digit:]]+(\\.[[:digit:]]+)?/ ]]; then\n    url_base=\"${url_base%%/v2}\"\nfi\nurl_path=\"${url_path//$url_base}\"\nurl_path=\"${url_path##/}\"\n\n# for convenience of straight copying and pasting out of documentation pages\n\n#repo=$(git_repo | sed 's/.*\\///')\n#\n#if [ -n \"$user\" ]; then\n#    url_path=\"${url_path/:owner/$user}\"\n#    url_path=\"${url_path/<owner>/$user}\"\n#    url_path=\"${url_path/\\{owner\\}/$user}\"\n#    url_path=\"${url_path/:username/$user}\"\n#    url_path=\"${url_path/<username>/$user}\"\n#    url_path=\"${url_path/\\{username\\}/$user}\"\n#    url_path=\"${url_path/:user/$user}\"\n#    url_path=\"${url_path/<user>/$user}\"\n#    url_path=\"${url_path/\\{user\\}/$user}\"\n#fi\n#url_path=\"${url_path/:repo/$repo}\"\n#url_path=\"${url_path/<repo>/$repo}\"\n#url_path=\"${url_path/\\{repo\\}/$repo}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "circleci/circleci_context_delete_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: $CIRCLECI_ORGANIZATION_ID testcontext haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes CircleCI context-level environment variable(s) from args or stdin\n\nIf no third argument is given, reads environment variables from standard input, one key per line or in 'key=value' format or 'export key=value' shell format\n\nYou'll need your Organization ID which isn't currently exposed by the CircleCI API, so you'd need to get this from the contexts page first:\n\n    https://app.circleci.com/settings/organization/<VCS>/<USER_OR_ORG>/contexts\neg.\n    https://app.circleci.com/settings/organization/github/HariSekhon/contexts\n\nExamples:\n\n    ${0##*/} \\$org_id testcontext AWS_ACCESS_KEY_ID...\n\n    export AWS_ACCESS_KEY_ID | ${0##*/} \\$org_id testcontext\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} \\$org_id testcontext\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization_id> <context_name> [<key> <key2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\n# can't get context listing working because can't figure out owner-id and owner-slug wasn't working\norg_id=\"$1\"\ncontext_name=\"$2\"\nshift || :\nshift || :\n\n# quick pagination support to find the context id from the full list of contexts\nwhile true; do\n    page_token=\"\"\n    output=\"$(\"$srcdir/circleci_api.sh\" \"/context?owner-id=$org_id&next_page_token=$page_token\")\"\n    context_id=\"$(jq_debug_pipe_dump <<< \"$output\" | jq -r \".items[] | select(.name == \\\"$context_name\\\") | .id\")\"\n    if [ -n \"$context_id\" ]; then\n        break\n    fi\n    page_token=\"$(jq -r .next_page_token <<< \"$output\")\"\n    if [ -z \"$page_token\" ]; then\n        break\n    fi\ndone\n\nif [ -z \"$context_id\" ]; then\n    die \"Failed to find context with name '$context_name'\"\nfi\n\ndelete_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    timestamp \"deleting environment variable '$key' in context '$context_id'\"\n    \"$srcdir/circleci_api.sh\" \"/context/$context_id/environment-variable/$key\" -X DELETE | jq -r .message\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        delete_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        delete_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "circleci/circleci_context_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: $CIRCLECI_ORGANIZATION_ID testcontext haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates CircleCI context-level environment variable(s) from args or stdin\n\nIf no third argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nYou'll need your Organization ID which isn't currently exposed by the CircleCI API, so you'd need to get this from the contexts page first:\n\n    https://app.circleci.com/settings/organization/<VCS>/<USER_OR_ORG>/contexts\neg.\n    https://app.circleci.com/settings/organization/github/HariSekhon/contexts\n\nExamples:\n\n    ${0##*/} \\$org_id testcontext AWS_ACCESS_KEY_ID=AKIA...\n\n    export AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} \\$org_id testcontext\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} \\$org_id testcontext\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization_id> <context_name> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\n# can't get context listing working because can't figure out owner-id and owner-slug wasn't working\norg_id=\"$1\"\ncontext_name=\"$2\"\nshift || :\nshift || :\n\n# quick pagination support to find the context id from the full list of contexts\nwhile true; do\n    page_token=\"\"\n    output=\"$(\"$srcdir/circleci_api.sh\" \"/context?owner-id=$org_id&next_page_token=$page_token\")\"\n    context_id=\"$(jq_debug_pipe_dump <<< \"$output\" | jq -r \".items[] | select(.name == \\\"$context_name\\\") | .id\")\"\n    if [ -n \"$context_id\" ]; then\n        break\n    fi\n    page_token=\"$(jq -r .next_page_token <<< \"$output\")\"\n    if [ -z \"$page_token\" ]; then\n        break\n    fi\ndone\n\nif [ -z \"$context_id\" ]; then\n    die \"Failed to find context with name '$context_name'\"\nfi\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    timestamp \"adding/updating environment variable '$key' to context '$context_id'\"\n    # shellcheck disable=SC2154\n    # XXX: this is a POST at the project level but a PUT at the context level (else 404 error)\n    \"$srcdir/circleci_api.sh\" \"/context/$context_id/environment-variable/$key\" -X PUT -d \"{\\\"value\\\": \\\"$value\\\"}\" | jq .\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "circleci/circleci_local_execute.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-10 14:51:52 +0000 (Tue, 10 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Runs Circle CI build locally in Docker\n#\n# .circleci/config.yml should contain a docker image and not default machine\n#\n# see local .circleci/config.yml for example\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/utils.sh\"\n\nusage(){\n    echo \"usage: ${0##*/} <job_name>\"\n    exit 3\n}\n\nif ! type -P circleci &>/dev/null; then\n    \"$srcdir/../install/install_circleci.sh\"\nfi\n\nif [ $# -gt 1 ]; then\n    usage\nfi\n\njob_name=\"${1:-build}\"\n\ncircleci config process .circleci/config.yml > process.yml\nexec circleci local execute -c process.yml --job \"$job_name\"\n"
  },
  {
    "path": "circleci/circleci_project_delete_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: github/HariSekhon/DevOps-Bash-tools haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://circleci.com/docs/api/v2/#operation/createEnvVar\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes CircleCI project-level environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one key per line or in 'key=value' format or 'export key=value' shell format\n\nExamples:\n\n    ${0##*/} github/HariSekhon/DevOps-Bash-tools AWS_ACCESS_KEY_ID...\n\n    echo AWS_ACCESS_KEY_ID | ${0##*/} github/HariSekhon/DevOps-Bash-tools\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} github/HariSekhon/DevOps-Bash-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_slug> [<key> <key2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject_slug=\"$1\"\nshift || :\n\nproject_slug=\"${project_slug##/}\"\nproject_slug=\"${project_slug%%/}\"\n\n\nif ! [[ \"$project_slug\" =~ ^[[:alnum:]]+/[[:alnum:]-]+/[[:alnum:]-]+$ ]]; then\n    usage \"project-slug given '$project_slug' does not conform to <vcs>/<user_or_org>/<repo> format\"\nfi\n\ndelete_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    timestamp \"deleting CircleCI environment variable '$key' in project '$project_slug'\"\n    \"$srcdir/circleci_api.sh\" \"/project/$project_slug/envvar/$key\" -X DELETE | jq -r .message\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        delete_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        delete_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "circleci/circleci_project_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: github/HariSekhon/DevOps-Bash-tools haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://circleci.com/docs/api/v2/#operation/createEnvVar\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates CircleCI project-level environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nExamples:\n\n    ${0##*/} github/HariSekhon/DevOps-Bash-tools AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} github/HariSekhon/DevOps-Bash-tools\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} github/HariSekhon/DevOps-Bash-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_slug> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject_slug=\"$1\"\nshift || :\n\nproject_slug=\"${project_slug##/}\"\nproject_slug=\"${project_slug%%/}\"\n\n\nif ! [[ \"$project_slug\" =~ ^[[:alnum:]]+/[[:alnum:]-]+/[[:alnum:]-]+$ ]]; then\n    usage \"project-slug given '$project_slug' does not conform to <vcs>/<user_or_org>/<repo> format\"\nfi\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    # shellcheck disable=SC2154\n    timestamp \"adding/updating CircleCI environment variable '$key' to project '$project_slug'\"\n    # shellcheck disable=SC2154\n    # XXX: this is a POST at the project level but a PUT at the context level (else 404 error)\n    \"$srcdir/circleci_api.sh\" \"/project/$project_slug/envvar\" -X POST -d \"{\\\"name\\\": \\\"$key\\\", \\\"value\\\": \\\"$value\\\"}\" | jq .\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "circleci/circleci_public_ips.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-28 14:22:04 +0100 (Thu, 28 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://circleci.com/docs/2.0/ip-ranges/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists CircleCI public IP addresses eg. for auto-populating firewall rules\n\nYou will also need to enable 'circleci_ip_ranges: true' in your job to use these fixed IPs (requires paid plan):\n\n    https://circleci.com/docs/2.0/configuration-reference/#circleciipranges\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#curl -sSL https://dnsjson.com/all.knownips.circleci.com/A.json |\n#jq -r '.results.records[]' |\n#sort -n\n\n\"$srcdir/../internet/dnsjson.sh\" all.knownips.circleci.com\n"
  },
  {
    "path": "cloudflare/cloudflare_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-21 15:06:10 +0100 (Fri, 21 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# args: /zones | jq .\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Cloudflare API (v4)\n\n\nRequires either \\$CLOUDFLARE_TOKEN or \\$CLOUDFLARE_EMAIL and \\$CLOUDFLARE_API_KEY be available in the environment\n\nToken can be generated here, or the Global API Key can be retrieved from this same page:\n\n    https://dash.cloudflare.com/profile/api-tokens\n\n\nAPI Reference:\n\n    https://api.cloudflare.com/\n\n\nExamples:\n\n\n# Test your token is working:\n\n    ${0##*/} /user/tokens/verify | jq .\n\n\n# List the currently authenticated user:\n\n    ${0##*/} /user | jq .\n\n\n# List accounts:\n\n    ${0##*/} /accounts | jq .\n\n\n# List currently authenticated user's account memberships and permissions:\n\n    ${0##*/} /memberships | jq .\n\n\n# List all zones:\n\n    ${0##*/} /zones | jq .\n\n\n# List all DNS records in a given zone (see cloudflare_dns_records.sh / cloudflare_dns_records_all_zones.sh):\n\n    ${0##*/} /zones/<zone_id>/dns_records\n\n\n# Export your DNS records in Bind config format:\n\n    ${0##*/} /zones/<zone_id>/dns_records/export\n\n\n# DNS analytics reports:\n\n    ${0##*/} /zones/<zone_id>/dns_analytics/report\n    ${0##*/} /zones/<zone_id>/dns_analytics/report/bytime\n\n\n# Details about DNSSEC status and configuration (see cloudflare_dnssec.sh for status across all zones)\n\n    ${0##*/} zones/<zone_id>/dnssec\n\n\n# List DNS Firewall clusters for an account:\n\n    ${0##*/} /accounts/<account_id>/virtual_dns\n\n\n# List the IPv4 and IPv6 cidr ranges for Cloudflare (see cloudflare_ip_ranges.sh for a ready parsed example of this):\n\n    ${0##*/} /ips | jq .\n\n\n# List custom certificates (.result.status and .result.expires_on fields may be of interest):\n\n    ${0##*/} /zones/<zone_id>/custom_certificates\n\n\n# Gets Cloudflare zone SSL verification status for a given zone (see cloudflare_ssl_verified.sh):\n\n    ${0##*/} /zones/<zone_id>/ssl/verification\n\n\n# Get Firewall Rules for a zone:\n\n    ${0##*/} /zones/<zone_id>/firewall/rules\n\n\n# Get Firewall Security Events for a zone:\n\n    ${0##*/} /zones/<zone_id>/security/events\n\n\n# List account rules lists:\n\n    ${0##*/}  /accounts/<account_id>/rules/lists\n\n\n# List load balancer pools:\n\n    ${0##*/} /user/load_balancers/pools\n\n\n# List load balancer monitors:\n\n    ${0##*/} /user/load_balancers/monitors\n\n\n# List load balancers for a zone:\n\n    ${0##*/}  /zones/<zone_id>/load_balancers\n\n\n# Get all cidr ranges owned by the account:\n\n    ${0##*/} /accounts/<account_id>/addressing/prefixes\n\n\n# List healthchecks for a zone:\n\n    ${0##*/} /zones/<zone_id>/healthchecks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.cloudflare.com/client/v4\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path##/}\"\n\nif ! is_blank \"${CLOUDFLARE_TOKEN:-}\"; then\n    export TOKEN=\"$CLOUDFLARE_TOKEN\"\n    \"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nelif ! is_blank \"${CLOUDFLARE_EMAIL:-}\" &&\n     ! is_blank \"${CLOUDFLARE_API_KEY:-}\"; then\n    export X_AUTH_EMAIL=\"X-Auth-Email: $CLOUDFLARE_EMAIL\"\n    export X_AUTH_API_KEY=\"X-Auth-Key: $CLOUDFLARE_API_KEY\"\n    curl \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" -H @<(cat <<< \"$X_AUTH_EMAIL\") -H @<(cat <<< \"$X_AUTH_API_KEY\") \"$@\"\nelse\n    usage \"CLOUDFLARE_TOKEN or CLOUDFLARE_EMAIL/CLOUDFLARE_API_KEY environment variables not specified\"\nfi\n"
  },
  {
    "path": "cloudflare/cloudflare_custom_certificates.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:56:31 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a given Cloudflare zone, list the custom certificates\n\nOutput format:\n\n<id>    <expiry_date>    <status>    <hosts>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<zone_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nzone_id=\"$1\"\n\n\"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/custom_certificates\" | jq -r '.result[] | [ .expires_on, .status, (.hosts | join(\",\")) ] | @tsv'\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_record_create.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-13 21:54:45 +0000 (Tue, 13 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a DNS record in the given domain\n\nYou must specify 'false' for the proxied argument or you will get a 400 error\nas those addresses cannot be proxied by Cloudflare\n\nResolves the domain name to a zone ID first and then submits the request to create the requested record\n\nhttps://developers.cloudflare.com/api/operations/dns-records-for-a-zone-create-dns-record\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain> <hostname> <ip_or_content> <proxied_true_false> <record_type>\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\ndomain=\"$1\"\nhostname=\"$2\"\nip=\"$3\"\nproxied=\"${4:-true}\"\nrecord_type=\"${5:-A}\"\n\nif ! is_bool \"$proxied\"; then\n    usage \"proxied argument must be a boolean (true/false)\"\nfi\n\nzone_id=\"$(\"$srcdir/cloudflare_zones.sh\" |\n           grep -E \"^[[:alnum:]]+[[:space:]]+$domain$\" |\n           sed 's/[[:space:]].*$//' ||\n           die \"Failed to resolved zone id for domain '$domain' - is this the right domain name?\")\"\n\nif [ -z \"$zone_id\" ]; then\n    die \"Zone ID is empty, check code\"\nfi\n\nif [ -n \"${CLOUDFLARE_DNS_RECORD_UPDATE:-}\" ]; then\n    request_type=\"PUT\"\nelse\n    request_type=\"POST\"\nfi\n\n\"$srcdir/cloudflare_api.sh\" \"zones/$zone_id/dns_records\" \\\n                            -X \"$request_type\" \\\n                            -d \"{\n                                \\\"content\\\": \\\"$ip\\\",\n                                \\\"name\\\": \\\"$hostname\\\",\n                                \\\"proxied\\\": $proxied,\n                                \\\"type\\\": \\\"$record_type\\\"\n                            }\"\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_record_delete.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-13 21:54:58 +0000 (Tue, 13 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a DNS record in the given domain\n\nResolves the DNS record and then submits the request to delete the requested record\n\nhttps://developers.cloudflare.com/api/operations/dns-records-for-a-zone-delete-dns-record\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain> <hostname>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\ndomain=\"$1\"\nhostname=\"$2\"\n\nzone_id=\"$(\"$srcdir/cloudflare_zones.sh\" |\n           grep -E \"^[[:alnum:]]+[[:space:]]+$domain$\" |\n           sed 's/[[:space:]].*$//' ||\n           die \"Failed to resolved zone id for domain '$domain' - is this the right domain name?\")\"\n\nif [ -z \"$zone_id\" ]; then\n    die \"Zone ID is empty, check code\"\nfi\n\ndns_record_id=\"$(\n    \"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/dns_records?per_page=50000\" |\n    jq -r '.result[] | [.id, .name] | @tsv' |\n    grep -E \"^[[:alnum:]]+[[:space:]]+$hostname\" |\n    head -n 1 |\n    sed 's/[[:space:]].*$//' ||\n    die \"Failed to record DNS record '$hostname' in domain '$domain' - are the hostname and domain name correct?\"\n)\"\n\n\"$srcdir/cloudflare_api.sh\" \"zones/$zone_id/dns_records/$dns_record_id\" -X DELETE\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_record_details.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-13 21:54:58 +0000 (Tue, 13 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets details for a DNS record in the given domain\n\nDetails are given in JSON for further pipe processing\n\nResolves the DNS record and then submits the request to delete the requested record\n\nhttps://developers.cloudflare.com/api/operations/dns-records-for-a-zone-delete-dns-record\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain> <hostname>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\ndomain=\"$1\"\nhostname=\"$2\"\n\nzone_id=\"$(\"$srcdir/cloudflare_zones.sh\" |\n           grep -E \"^[[:alnum:]]+[[:space:]]+$domain$\" |\n           sed 's/[[:space:]].*$//' ||\n           die \"Failed to resolved zone id for domain '$domain' - is this the right domain name?\")\"\n\nif [ -z \"$zone_id\" ]; then\n    die \"Zone ID is empty, check code\"\nfi\n\ndns_record_id=\"$(\n    \"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/dns_records?per_page=50000\" |\n    jq -r '.result[] | [.id, .name] | @tsv' |\n    grep -E \"^[[:alnum:]]+[[:space:]]+$hostname\" |\n    head -n 1 |\n    sed 's/[[:space:]].*$//' ||\n    die \"Failed to record DNS record '$hostname' in domain '$domain' - are the hostname and domain name correct?\"\n)\"\n\n\"$srcdir/cloudflare_api.sh\" \"zones/$zone_id/dns_records/$dns_record_id\"\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_record_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-13 21:54:45 +0000 (Tue, 13 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a DNS record in the given domain\n\nResolves the domain name to a zone ID first and then submits the request to create the requested record\n\nhttps://developers.cloudflare.com/api/operations/dns-records-for-a-zone-patch-dns-record\n\nUses adjacent cloudflare_dns_record_create.sh\n\nYou might get this error:\n\n\"'    {\"success\":false,\"errors\":[{\"code\":10000,\"message\":\"PUT method not allowed for the api_token authentication scheme\"}]}\n'\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain> <hostname> <ip_or_content> <proxied_true_false> <record_type>\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\nexport CLOUDFLARE_DNS_RECORD_UPDATE=1\n\n\"$srcdir/cloudflare_dns_record_create.sh\" \"$@\"\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_records.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:17:26 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all DNS records for a given Cloudflare zone\n\nResolves the domain name to a zone ID first and then submits the request to list the DNS records in that domain\n\nhttps://api.cloudflare.com/#dns-records-for-a-zone-list-dns-records\n\nOutput format:\n\n<dns_record>    <type>    <ttl>\n\n\nLimitation: only lists the first 50,000 DNS records in a zone. Pagination code addition required if you have a zone larger than this\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domain>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ndomain=\"$1\"\n\nzone_id=\"$(\"$srcdir/cloudflare_zones.sh\" |\n           grep -E \"^[[:alnum:]]+[[:space:]]+$domain$\" |\n           sed 's/[[:space:]].*$//' ||\n           die \"Failed to resolved zone id for domain '$domain' - is this the right domain name?\")\"\n\nif [ -z \"$zone_id\" ]; then\n    die \"Zone ID is empty, check code\"\nfi\n\n\"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/dns_records?per_page=50000\" |\njq -r '.result[] | [.name, .type, .ttl] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "cloudflare/cloudflare_dns_records_all_zones.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:17:26 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all DNS records in Cloudflare across all zones\n\nhttps://api.cloudflare.com/#dns-records-for-a-zone-list-dns-records\n\nOutput format:\n\n<dns_record>    <type>    <ttl>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/cloudflare_foreach_zone.sh\" \"$srcdir/cloudflare_dns_records.sh {zone_id}\"\n"
  },
  {
    "path": "cloudflare/cloudflare_dnssec.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:17:26 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloudflare DnsSec status for all zones\n\nhttps://api.cloudflare.com/#dnssec-dnssec-details\n\nOutput format:\n\n<zone_id>    <zone_name>    <dnssec_status>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/cloudflare_foreach_zone.sh\" \"$srcdir/cloudflare_api.sh /zones/{id}/dnssec | jq -r \\\"[\\\\\\\"{id}\\\\\\\", \\\\\\\"{name}\\\\\\\", .result.status] | @tsv\\\"\"\n"
  },
  {
    "path": "cloudflare/cloudflare_firewall_access_rules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-04 16:00:54 +0000 (Thu, 04 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloudflare Firewall Access Rules for a given Zone ID\n\nIn --verbose mode also also lists the filter expression at the end\n\nOutput:\n\n<id>\n\nUses cloudflare_api.sh - see there for authentication API key details\n\nSee Also:\n\n    cloudflare_zones.sh - to get the zone id argument\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<zone_id> [--verbose]\"\n\nhelp_usage \"$@\"\n\nuntil [ $# -lt 1 ]; do\n    case \"$1\" in\n -v|--verbose)  verbose=1\n                ;;\n           -*)  usage\n                ;;\n            *)  if [ -n \"${zone_id:-}\" ]; then\n                    usage\n                fi\n                zone_id=\"$1\"\n                ;;\n    esac\n    shift || :\ndone\n\nif [ -z \"${zone_id:-}\" ]; then\n    usage \"zone id not defined\"\nfi\n\n\"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/firewall/access_rules/rules\" |\nif [ -n \"${verbose:-}\" ]; then\n    jq -r '.result[] | [.id, .mode, .notes, .configuration.target, .configuration.value] | @tsv'\nelse\n    jq -r '.result[] | [.id, .mode, .notes] | @tsv'\nfi\n"
  },
  {
    "path": "cloudflare/cloudflare_firewall_rules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-04 16:00:54 +0000 (Thu, 04 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloudflare Firewall Rules for a given Zone ID\n\nIn --verbose mode also also lists the filter expression at the end\n\nOutput:\n\n<id>\n\nUses cloudflare_api.sh - see there for authentication API key details\n\nSee Also:\n\n    cloudflare_zones.sh - to get the zone id argument\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<zone_id> [--verbose]\"\n\nhelp_usage \"$@\"\n\nuntil [ $# -lt 1 ]; do\n    case \"$1\" in\n -v|--verbose)  verbose=1\n                ;;\n           -*)  usage\n                ;;\n            *)  if [ -n \"${zone_id:-}\" ]; then\n                    usage\n                fi\n                zone_id=\"$1\"\n                ;;\n    esac\n    shift || :\ndone\n\nif [ -z \"${zone_id:-}\" ]; then\n    usage \"zone id not defined\"\nfi\n\n\"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/firewall/rules\" |\nif [ -n \"${verbose:-}\" ]; then\n    jq -r '.result[] | [.id, .action, .description, .filter.expression] | @tsv'\nelse\n    jq -r '.result[] | [.id, .action, .description] | @tsv'\nfi\n"
  },
  {
    "path": "cloudflare/cloudflare_foreach_account.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo account id = {id} and name = \"{name}\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:08:43 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each Cloudflare account\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the account id/names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{id}   - with the account id   <-- this is the one you want for chaining API queries with cloudflare_api.sh\n{name} - with the account name\n\neg.\n    ${0##*/} 'echo account id = {id} and name = {name}'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/cloudflare_api.sh\" accounts |\njq -r '.result[] | [.id, .name] | @tsv' |\nsed \"s/'/\\\\\\\\'/g\" |\nwhile read -r account_id account_name; do\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $account_id - $account_name\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{account_id\\}/$account_id}\")\n    cmd=(\"${cmd[@]//\\{account_name\\}/$account_name}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$account_id}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$account_name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\ndone\n"
  },
  {
    "path": "cloudflare/cloudflare_foreach_zone.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo zone id = {id} and name = {name}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:08:43 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each Cloudflare zone\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the zone id/names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{id}   - with the zone id   <-- this is the one you want for chaining API queries with cloudflare_api.sh\n{name} - with the zone name\n\neg.\n    ${0##*/} 'echo zone id = {id} and name = {name}'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/cloudflare_api.sh\" zones |\njq -r '.result[] | [.id, .name] | @tsv' |\nsed \"s/'/\\\\\\\\'/g\" |\nwhile read -r zone_id zone_name; do\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $zone_id - $zone_name\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{zone_id\\}/$zone_id}\")\n    cmd=(\"${cmd[@]//\\{zone_name\\}/$zone_name}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$zone_id}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$zone_name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\ndone\n"
  },
  {
    "path": "cloudflare/cloudflare_ip_ranges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 18:42:38 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the Cloudflare IPv4 and/or IPv6 CIDR ranges via the Cloudflare API\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[--ipv4 / --ipv6]\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_switches=\"\n   --ipv4           Output only IPv4 CIDR ranges\n   --ipv6           Output only IPv6 CIDR ranges\n\"\n\nfor arg; do\n    case \"$arg\" in\n        --ipv4)     IPV4_ONLY=1\n                    ;;\n        --ipv6)     IPV6_ONLY=1\n                    ;;\n    esac\ndone\n\nif [ -n \"${IPV4_ONLY:-}\" ] &&\n   [ -n \"${IPV6_ONLY:-}\" ]; then\n    usage \"IPv4 and IPv6 filters are mutually exclusive\"\nfi\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n\"$srcdir/cloudflare_api.sh\" ips |\njq -r '.result.ipv4_cidrs[], .result.ipv6_cidrs[]' |\nif [ -n \"${IPV4_ONLY:-}\" ]; then\n    grep -v -e '[:alpha:]' -e ':'\nelif [ -n \"${IPV6_ONLY:-}\" ]; then\n    grep -e '[:alpha:]' -e ':'\nelse\n    cat\nfi\n"
  },
  {
    "path": "cloudflare/cloudflare_purge_cache.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-04 16:00:54 +0000 (Thu, 04 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPurges EVERYTHING from the Cloudflare cache for the given zone id (which can be obtained from the adjacent cloudflare_zones.sh)\n\nRequires CLOUDFLARE_EMAIL and CLOUDFLARE_TOKEN environment variables to be set\n\nIf CLOUDFLARE_ZONE_ID is set then you can omit the zone_id arg\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<zone_id>]\"\n\nhelp_usage \"$@\"\n\nif [ -n \"${CLOUDFLARE_ZONE_ID:-}\" ]; then\n    zone_id=\"$CLOUDFLARE_ZONE_ID\"\nelse\n    num_args 1 \"$@\"\n\n    zone_id=\"$1\"\nfi\n\ncheck_env_defined \"CLOUDFLARE_EMAIL\"\ncheck_env_defined \"CLOUDFLARE_TOKEN\"\n\n# curl auth will use these to send the token without putting it on the command line / logs\nexport CURL_AUTH_HEADER=\"X-Auth-Key:\"\nexport TOKEN=\"$CLOUDFLARE_TOKEN\"\n\n        # don't use this, let curl_auth.sh do it more securely without leaving breadcrumbs on the local machine\n        #-H \"X-Auth-Key: $CLOUDFLARE_TOKEN\" \\\n\noutput=\"$(\n    \"$srcdir/../bin/curl_auth.sh\" \"https://api.cloudflare.com/client/v4/zones/$zone_id/purge_cache\" \\\n        -sS -X POST \\\n        -H \"Content-Type: application/json\" \\\n        -H \"X-Auth-Email: $CLOUDFLARE_EMAIL\" \\\n        --data '{\"purge_everything\":true}' |\n        jq\n)\"\n\necho \"$output\"\n\njq -e '.success == true' <<< \"$output\"\n"
  },
  {
    "path": "cloudflare/cloudflare_ssl_verified.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 19:09:43 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets Cloudflare zone SSL verification status for a given zone\n\nOutput format:\n\n<hostname>  <status>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<zone>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nzone_id=\"$1\"\n\n\"$srcdir/cloudflare_api.sh\" \"/zones/$zone_id/ssl/verification\" |\njq -r '.result[] | [.hostname, .certificate_status] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "cloudflare/cloudflare_ssl_verified_all_zones.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-02 19:09:43 +0100 (Wed, 02 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets Cloudflare zone SSL verification status for all zones\n\nOutput format:\n\n<hostname>  <status>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/cloudflare_foreach_zone.sh\" \"$srcdir/cloudflare_ssl_verified.sh {zone_id}\"\n"
  },
  {
    "path": "cloudflare/cloudflare_zones.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-04 16:00:54 +0000 (Thu, 04 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloudflare Zone names and IDs (needed for Terraform)\n\nOutput:\n\n<id>    <name>\n\nUses cloudflare_api.sh - see there for authentication API key details\nUsed by cloudflare_dns_record_*.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/cloudflare_api.sh\" /zones |\njq -r '.result[] | [.id, .name] | @tsv'\n"
  },
  {
    "path": "codefresh.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-24 17:43:07 +0000 (Mon, 24 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            C o d e f r e s h   C I\n# ============================================================================ #\n\n# https://codefresh.io/docs/docs/codefresh-yaml/\n\nversion: \"1.0\"\nstages:\n  - \"checkout\"\n  - \"build\"\nsteps:\n  checkout:\n    type: \"git-clone\"\n    description: \"Cloning main repository...\"\n    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'\n    revision: \"${{CF_REVISION}}\"\n    stage: \"checkout\"\n  build:\n    title: Running docker image\n    type: freestyle\n    working_directory: '${{CF_REPO_NAME}}'\n    arguments:\n      image: 'ubuntu:18.04'\n      commands:\n        - setup/ci_bootstrap.sh\n        - make init\n        - make ci\n        - make test\n"
  },
  {
    "path": "codeship/codeship.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2021-04-12 18:33:44 +0100 (Mon, 12 Apr 2021)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                C o d e S h i p\n# ============================================================================ #\n\n# LEGACY - CloudBees has removed free plans as of 1st March 2023\n\n# 3rd party way of doing IaC on CodeShip CI as the free edition doesn't support this\n\n# https://github.com/painless-software/codeship-yaml\n\n# Requires setting up the CodeShip commands like so:\n#\n#   pip install codeship-yaml\n#   codeship-yaml\n#\n# or seaparately in sections:\n#\n# Project Settings > Test Settings > Setup Commands:\n#\n#   pip install codeship-yaml\n#   codeship-yaml install\n#\n# Project Settings > Test Settings > Test Commands:\n#\n#   codeship-yaml before_script script\n#\n# Project Settings > Deployment > (branch name)\n#\n#   codeship-yaml after_success\n\n---\ninstall:\n  # these cause package installation breakages due to GPG or 403 errors, old addresses etc.\n  - sudo rm -fv -- /etc/apt/sources.list.d/cli_assets_heroku_com_branches_stable_apt.list\n  - sudo rm -fv -- /etc/apt/sources.list.d/apache_bintray_com_couchdb_deb.list\n  - sudo rm -fv -- /etc/apt/sources.list.d/www_apache_org_dist_cassandra_debian.list\n  - make\n#before_script:\n#  - somecommand\nscript:\n  - make test\n#after_success:\n#  - echo \"Now we can deploy\"\n"
  },
  {
    "path": "codeship/codeship_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /organizations/{organization_uuid}/projects | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 19:49:17 +0100 (Tue, 13 Oct 2020)\n#\n#  https://codeship.com/harisekhon/bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the CodeShip.com API with authentication\n\nRequired:\n\n    \\$CODESHIP_TOKEN\n        or\n    \\$CODESHIP_USERNAME / \\$CODESHIP_USER and \\$CODESHIP_PASSWORD\n\n    \\$CODESHIP_ORGANIZATION_UUID\n\nSee codeship_api_token.sh for more details\n\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://apidocs.codeship.com/v2\n\n\nExamples:\n\nThe CodeShip API is pretty rudimentary compared to other systems APIs found in adjacent scripts\n\n\n# List projects:\n\n    ${0##*/} /organizations/{organization_uuid}/projects | jq .\n\n# List builds for a project:\n\n    ${0##*/} /organizations/{organization_uuid}/projects/{project_uuid}/builds\n\n    ${0##*/} /organizations/950777b0-5660-0134-8981-1aa13e5eb75e/projects/1b13b490-38c2-0138-ed09-4a04e28a9ab8/builds | jq .\n\n# Get pipelines for a build:\n\n    ${0##*/} /organizations/{organization_uuid}/projects/{project_uuid}/builds/{build_uuid}/pipelines\n\n    ${0##*/} /organizations/950777b0-5660-0134-8981-1aa13e5eb75e/projects/1b13b490-38c2-0138-ed09-4a04e28a9ab8/builds/daec9560-f29c-0138-f24e-0a117489fb82/pipelines | jq .\n\n\nPlaceholders replaced by \\$CODESHIP_ORGANIZATION_UUID:                     {organization}, {organization_uuid}, <organization>, <organization_uuid>, :organization, :organization_uuid\nPlaceholders replaced by \\$CODESHIP_USERNAME / \\$CODESHIP_USER:             :owner, :user, :username, <user>, <username>\nPlaceholders replaced by the local repo name of the current directory:    :repo, <repo>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.codeship.com/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ntoken=\"${CODESHIP_TOKEN:-}\"\norganization_uuid=\"${CODESHIP_ORGANIZATION_UUID:-}\"\n\nif [ -z \"$token\" ] || [ -z \"$organization_uuid\" ]; then\n    output=\"$(GET_ORGANIZATION=1 \"$srcdir/codeship_api_token.sh\")\"\n    token=\"${output%%[[:space:]]*}\"\n    organization_uuid=\"${output##*[[:space:]]}\"\nfi\n\nexport TOKEN=\"$token\"\n\nurl_path=\"$1\"\nshift || :\n\n#url_path=\"${url_path//https:\\/\\/api.codeship.com/v2}\"\nurl_path=\"${url_path##\"$url_base\"}\"\nurl_path=\"${url_path##/}\"\n\n# for convenience of straight copying and pasting out of documentation pages\n\nurl_path=\"${url_path/:organization/$organization_uuid}\"\nurl_path=\"${url_path/<organization>/$organization_uuid}\"\nurl_path=\"${url_path/\\{organization\\}/$organization_uuid}\"\n\nurl_path=\"${url_path/:organization_uuid/$organization_uuid}\"\nurl_path=\"${url_path/<organization_uuid>/$organization_uuid}\"\nurl_path=\"${url_path/\\{organization_uuid\\}/$organization_uuid}\"\n\nurl_path=\"${url_path/:uuid/$organization_uuid}\"\nurl_path=\"${url_path/<uuid>/$organization_uuid}\"\nurl_path=\"${url_path/\\{uuid\\}/$organization_uuid}\"\n\nrepo=$(git_repo | sed 's/.*\\///')\n\nuser=\"${CODESHIP_USERNAME:-${CODESHIP_USER:-}}\"\nif [ -n \"$user\" ]; then\n    url_path=\"${url_path/:owner/$user}\"\n    url_path=\"${url_path/:username/$user}\"\n    url_path=\"${url_path/:user/$user}\"\n    url_path=\"${url_path/<username>/$user}\"\n    url_path=\"${url_path/<user>/$user}\"\nfi\nif [ -n \"${repo:-}\" ]; then\n    url_path=\"${url_path/:repo/$repo}\"\n    url_path=\"${url_path/<repo>/$repo}\"\nfi\n\ncurl_api_opts \"$@\"\n\n# case insensitive regex matching\nshopt -s nocasematch\nif ! [[ \"$*\" =~ Accept: ]]; then\n    CURL_OPTS+=(-H \"Accept: application/json\")\nfi\nif ! [[ \"$*\" =~ Content-Type: ]]; then\n    CURL_OPTS+=(-H \"Content-Type: application/json\")\nfi\n# unset to return to default setting for safety to avoid hard to debug changes of behaviour elsewhere\nshopt -u nocasematch\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "codeship/codeship_api_token.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 20:18:20 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nReturns a CodeShip access token from the CodeShip API using basic authentication\n\nRequires \\$CODESHIP_USERNAME / \\$CODESHIP_USER and \\$CODESHIP_PASSWORD to be defined in the environment\n\nCodeShip user is usually your email address and if using GitHub OAuth sign-in, you'll need to set a normal password via this link:\n\n    https://app.codeship.com/password_reset/new\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\ncurl_api_opts \"$@\"\n\ncheck_env_defined \"CODESHIP_PASSWORD\"\n\nuser=\"${CODESHIP_USERNAME:-${CODESHIP_USER:-}}\"\nif [ -z \"$user\" ]; then\n    die \"\\$CODESHIP_USERNAME / \\$CODESHIP_USER not defined\"\nfi\n\nexport USER=\"$user\"\nexport PASSWORD=\"$CODESHIP_PASSWORD\"\n\n# has to be basic auth, don't allow token to be used as it will result in a 401\noutput=\"$(NO_TOKEN_AUTH=1 \"$srcdir/../bin/curl_auth.sh\" https://api.codeship.com/v2/auth -X POST \"${CURL_OPTS[@]}\" \"$@\")\"\n\ndie_if_error_field \"$output\"\n\nif [ -n \"${DEBUG:-}\" ]; then\n    # pretty print for human convenience / developer review\n    jq . <<< \"$output\" >&2\nfi\n\nif [ -n \"${GET_ORGANIZATION:-}\" ]; then\n    jq -r '[.access_token, .organizations[0].uuid] | @tsv' <<< \"$output\"\nelse\n    jq -r '.access_token' <<< \"$output\"\nfi\n"
  },
  {
    "path": "configs/.Codefresh/cli-config/config.yaml",
    "content": "currentProfile: default\nprofiles:\n  default:\n    output:\n      pretty: false\n      dateFormat: default\n    request:\n      timeout: 30000\n      maxAttempts: 10\n      retryDelay: 500\n    logs:\n      fallback:\n        interval: 5000\n        maxAttempts: 1000\n"
  },
  {
    "path": "configs/.Xdefaults",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Sat Jul 1 00:06:15 2006 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\naterm*background:     black\naterm*foreground:     whitesmoke\naterm*geometry:       72x24\naterm*transparent:    true\naterm*shading:        50\n#Looks nice but slows things down too much\naterm*fading:         70\naterm*cursor:         red\naterm*scrollBar:      false\naterm*transpscrollbar:true\naterm*scrollBar_right:true\naterm*saveLines:      20000\naterm*jumpScroll:     true\naterm*multiscroll:    true\n\nborderlessaterm*background:     black\nborderlessaterm*foreground:     whitesmoke\nborderlessaterm*geometry:       72x24\nborderlessaterm*transparent:    true\nborderlessaterm*shading:        50\n#Looks nice but slows things down too much\nborderlessaterm*fading:         70\nborderlessaterm*cursor:         red\nborderlessaterm*scrollBar:      false\nborderlessaterm*transpscrollbar:true\nborderlessaterm*scrollBar_right:true\nborderlessaterm*saveLines:      20000\nborderlessaterm*jumpScroll:     true\nborderlessaterm*multiscroll:    true\n\naterm80*background:     black\naterm80*foreground:     whitesmoke\naterm80*geometry:       80x24\naterm80*transparent:    true\naterm80*shading:        50\naterm80*fading:         70\naterm80*cursor:         red\naterm80*scrollBar:      false\naterm80*transpscrollbar:true\naterm80*scrollBar_right:true\naterm80*saveLines:      20000\naterm80*jumpScroll:     true\naterm80*multiscroll:    true\n\nborderlessaterm80*background:     black\nborderlessaterm80*foreground:     whitesmoke\nborderlessaterm80*geometry:       80x24\nborderlessaterm80*transparent:    true\nborderlessaterm80*shading:        50\nborderlessaterm80*fading:         70\nborderlessaterm80*cursor:         red\nborderlessaterm80*scrollBar:      false\nborderlessaterm80*transpscrollbar:true\nborderlessaterm80*scrollBar_right:true\nborderlessaterm80*saveLines:      20000\nborderlessaterm80*jumpScroll:     true\nborderlessaterm80*multiscroll:    true\n\nxterm*charClass:      33:48,37:48,45-47:48,38:48\nxterm*background:     black\nxterm*foreground:     white\nxterm*geometry:       80x24\nxterm*saveLines:      20000\nxterm*jumpScroll:     true\nxterm*multiscroll:    true\n#xterm*highlightColor: LightSkyBlue\nxterm*cursorColor:    MediumBlue\nxterm*pointerColor:   MediumBlue\nxterm*internalBorder: 3\nxterm*scrollBar:      false\nxterm*rightScrollBar: false\n"
  },
  {
    "path": "configs/.Xmodmap",
    "content": "keycode 115 = Super_L\nadd Mod4 = Super_L\nkeycode 116 = Super_R\nadd Mod4 = Super_R\nclear Mod1\nkeycode 66 = Alt_L\nkeycode 69 = Alt_R\nadd Mod1 = Alt_L\nadd Mod1 = Alt_R\n"
  },
  {
    "path": "configs/.ansible.cfg",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2014-07-13 16:55:22 +0100 (Sun, 13 Jul 2014)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n[defaults]\ninterpreter_python = /usr/bin/python3\n#forks = 30\n#display_skipped_hosts = False\nlegacy_playbook_variables = no\n# https://docs.ansible.com/ansible/latest/user_guide/intro_getting_started.html#host-key-checking\n# in high security environments instead comment this out and pre-load ~/.ssh/known_hosts via ssh-keyscan to avoid hanging on new or changed hosts\nhost_key_checking = False\n# default command module doesn't support shell variables, pipes or quotes\nmodule_name = shell\n\n[ssh_connection]\npipelining = True\nssh_args = -o ControlMaster=auto -o ControlPersist=30m\n"
  },
  {
    "path": "configs/.athenacli/athenaclirc",
    "content": "# USE defaults from aws cli instead\n\n[aws_profile default]\n# If you are a user of aws cli, you might want to use some configurations of aws cli,\n# please refer to https://athenacli.readthedocs.io/en/latest/awsconfig.html for more infomation.\n#aws_access_key_id = ''\n#aws_secret_access_key = ''\n#region = '' # e.g us-west-2, us-east-1\n\n# Amazon S3 staging directory where query results are stored\n# NOTE: S3 should in the same region as specified above.\n# The format is 's3://<your s3 directory path>'\ns3_staging_dir = ''\n\n[main]\n# log_file location.\nlog_file = ~/.athenacli/app.log\n\n# history_file location.\nhistory_file = '~/.athenacli/history'\n\n# Multi-line mode allows breaking up the sql statements into multiple lines. If\n# this is set to True, then the end of the statements must have a semi-colon.\n# If this is set to False then sql statements can't be split into multiple\n# lines. End of line (return) is considered as the end of the statement.\nmulti_line = True\n\n# Destructive warning mode will alert you before executing a sql statement\n# that may cause harm to the database such as \"drop table\", \"drop database\"\n# or \"shutdown\".\ndestructive_warning = True\n\n# Default log level. Possible values: \"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\"\n# and \"DEBUG\". \"NONE\" disables logging.\nlog_level = INFO\n\n# Timing of sql statments and table rendering.\ntiming = True\n\n# Table format. Possible values: ascii, double, github,\n# psql, plain, simple, grid, fancy_grid, pipe, orgtbl, rst, mediawiki, html,\n# latex, latex_booktabs, textile, moinmoin, jira, vertical, tsv, csv.\n# Recommended: ascii\ntable_format = ascii\n\n# Syntax coloring style. Possible values (many support the \"-dark\" suffix):\n# manni, igor, xcode, vim, autumn, vs, rrt, native, perldoc, borland, tango, emacs,\n# friendly, monokai, paraiso, colorful, murphy, bw, pastie, paraiso, trac, default,\n# fruity.\n# Screenshots at http://mycli.net/syntax\nsyntax_style = default\n\n# Keybindings: Possible values: emacs, vi.\n# Emacs mode: Ctrl-A is home, Ctrl-E is end. All emacs keybindings are available in the REPL.\n# When Vi mode is enabled you can use modal editing features offered by Vi in the REPL.\nkey_bindings = emacs\n\n# Athena prompt\n# \\d - Database name\n# \\r - Region\n# \\D - The full current date\n# \\m - Minutes of the current time\n# \\n - Newline\n# \\P - AM/PM\n# \\R - The current time, in 24-hour military time (0–23)\n# \\s - Seconds of the current time\nprompt = '\\r:\\d> '\nprompt_continuation = '-> '\n\n# enable pager on startup\nenable_pager = True\n\n# Custom colors for the completion menu, toolbar, etc.\n[colors]\n# Completion menus.\nToken.Menu.Completions.Completion.Current = 'bg:#00aaaa #000000'\nToken.Menu.Completions.Completion = 'bg:#008888 #ffffff'\nToken.Menu.Completions.MultiColumnMeta = 'bg:#aaffff #000000'\nToken.Menu.Completions.ProgressButton = 'bg:#003333'\nToken.Menu.Completions.ProgressBar = 'bg:#00aaaa'\n\n# Query results\nToken.Output.Header = 'bold'\nToken.Output.OddRow = ''\nToken.Output.EvenRow = ''\n\n# Selected text.\nToken.SelectedText = '#ffffff bg:#6666aa'\n\n# Search matches. (reverse-i-search)\nToken.SearchMatch = '#ffffff bg:#4444aa'\nToken.SearchMatch.Current = '#ffffff bg:#44aa44'\n\n# The bottom toolbar.\nToken.Toolbar = 'bg:#222222 #aaaaaa'\nToken.Toolbar.Off = 'bg:#222222 #888888'\nToken.Toolbar.On = 'bg:#222222 #ffffff'\n\n# Search/arg/system toolbars.\nToken.Toolbar.Search = 'noinherit bold'\nToken.Toolbar.Search.Text = 'nobold'\nToken.Toolbar.System = 'noinherit bold'\nToken.Toolbar.Arg = 'noinherit bold'\nToken.Toolbar.Arg.Text = 'nobold'\n\n# Favorite queries.\n[favorite_queries]\n"
  },
  {
    "path": "configs/.aws/config",
    "content": "#\n#  vim:ts=4:sts=4:sw=4:et:filetype=dosini\n#\n#  Author: Hari Sekhon\n#  Date: Thu Jul 22 19:17:20 2021 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                   A W S   C L I   C o n f i g u r a t i o n\n# ============================================================================ #\n\n# https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html\n\n# can configure via this command which prompts for access keys, region, output:\n#\n#   aws configure\n\n# make this go nowhere for safety - do not populate ~/.aws/credentials section for [default]\n[default]\nregion = eu-west-2\noutput = json\n\n# root mgmt account of Organization where AWS SSO lives\n[mgmt]\nregion = eu-west-2\noutput = json\n\n# ============================================================================ #\n#                        S S O   E n v i r o n m e n t s\n# ============================================================================ #\n\n# https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html\n\n# can configure via this command which will give you a drop down of the accounts available to fill in the sso account id settings:\n#\n#   aws configure sso\n\n[dev]\nso_start_url = https://a-xxxxxxxxxx.awsapps.com/start  # or custom URL\nsso_region = eu-west-2\nsso_account_id = nnnnnnnnnnnn\nsso_role_name = AWSAdministratorAccess\nregion = eu-west-2\noutput = json\n\n[staging]\nso_start_url = https://a-xxxxxxxxxx.awsapps.com/start\nsso_region = eu-west-2\nsso_account_id = nnnnnnnnnnnn\nsso_role_name = AWSAdministratorAccess\nregion = eu-west-2\noutput = json\n\n[production]\nso_start_url = https://a-xxxxxxxxxx.awsapps.com/start\nsso_region = eu-west-2\nsso_account_id = nnnnnnnnnnnn\nsso_role_name = AWSAdministratorAccess\nregion = eu-west-2\noutput = json\n"
  },
  {
    "path": "configs/.aws/shell/awsshellrc",
    "content": "[aws-shell]\n\n# fuzzy or substring match.\nmatch_fuzzy = True\n\n# vi or emacs key bindings.\nenable_vi_bindings = False\n\n# multi or single column completion menu.\nshow_completion_columns = False\n\n# show or hide the help pane.\nshow_help = True\n\n# visual theme. possible values: manni, igor, xcode, vim,\n# autumn,vs, rrt, native, perldoc, borland, tango, emacs,\n# friendly, monokai, paraiso-dark, colorful, murphy, bw,\n# pastie, paraiso-light, trac, default, fruity.\n# to disable themes, set theme = none\ntheme = vim\n"
  },
  {
    "path": "configs/.checkov.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-02-21 16:53:29 +0000 (Mon, 21 Feb 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          C h e c k o v   c o n f i g\n# ============================================================================ #\n\n# https://github.com/bridgecrewio/checkov#configuration-using-a-config-file\n#\n# This is not well documented but the fields seem to be the same as:\n#\n#   checkov --help\n#\n# See master template at:\n#\n#   https://github.com/HariSekhon/Templates/blob/master/.checkov.yaml\n\n---\ncompact: true\ndirectory:\n  - .\nskip-path:\n  - sql\n  - templates\n  - vagrant/kubernetes/calico.yaml\n  - vagrant/kubernetes/LFS258\ndocker-image: harisekhon/bash-tools\ndownload-external-modules: true  # without this gets lots of annoying warning lines such as '2022-02-22 16:14:40,180 [MainThread  ] [WARNI]  Failed to download module x/y/z:n.n.n'\nframework:\n  - all\nno-guide: true\noutput: cli\nquiet: true\nrepo-id: HariSekhon/DevOps-Bash-tools  # what to report to Bridgecrew Cloud - without this gets annoying duplicate repos such as 'harisekhon_cli_repo/bash-tools'\nskip-suppressions: true\nsoft-fail: true\n"
  },
  {
    "path": "configs/.config/flake8",
    "content": "[flake8]\nignore = E265,E402,F401\nmax-line-length = 120\nexclude = test*/*\nmax-complexity = 10\n"
  },
  {
    "path": "configs/.config/htop/htoprc",
    "content": "# Beware! This file is rewritten by htop when settings are changed in the interface.\n# The parser is also very primitive, and not human-friendly.\nfields=0 48 17 18 38 39 2 46 47 49 1\nsort_key=46\nsort_direction=1\nhide_threads=0\nhide_kernel_threads=1\nhide_userland_threads=0\nshadow_other_users=0\nshow_thread_names=0\nshow_program_path=0\nhighlight_base_name=1\nhighlight_megabytes=1\nhighlight_threads=1\ntree_view=0\nheader_margin=1\ndetailed_cpu_time=0\ncpu_count_from_zero=0\nupdate_process_names=0\naccount_guest_in_cpu_meter=0\ncolor_scheme=0\ndelay=15\nleft_meters=CPU AllCPUs Memory Swap\nleft_meter_modes=1 1 1 1\nright_meters=Tasks LoadAverage Uptime Battery Clock\nright_meter_modes=2 2 2 2 2\n"
  },
  {
    "path": "configs/.config/pycodestyle",
    "content": "[pycodestyle]\ncount = False\nignore = E265, E266, E402\nmax-line-length = 120\nstatistics = True\n"
  },
  {
    "path": "configs/.config/terminalizer/config.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2024-11-14 22:12:36 +0400 (Thu, 14 Nov 2024)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\n# Specify a command to be executed\n# like `/bin/bash -l`, `ls`, or any other commands\n# the default is bash for Linux\n# or powershell.exe for Windows\ncommand: null\n\n# Specify the current working directory path\n# the default is the current working directory path\ncwd: null\n\n# Export additional ENV variables\nenv:\n  recording: true\n\n# Explicitly set the number of columns\n# or use `auto` to take the current\n# number of columns of your shell\ncols: auto\n\n# Explicitly set the number of rows\n# or use `auto` to take the current\n# number of rows of your shell\nrows: auto\n\n# Amount of times to repeat GIF\n# If value is -1, play once\n# If value is 0, loop indefinitely\n# If value is a positive number, loop n times\nrepeat: 0\n\n# Quality\n# 1 - 100\nquality: 100\n\n# Delay between frames in ms\n# If the value is `auto` use the actual recording delays\nframeDelay: auto\n\n# Maximum delay between frames in ms\n# Ignored if the `frameDelay` isn't set to `auto`\n# Set to `auto` to prevent limiting the max idle time\nmaxIdleTime: 2000\n\n# The surrounding frame box\n# The `type` can be null, window, floating, or solid`\n# To hide the title use the value null\n# Don't forget to add a backgroundColor style with a null as type\nframeBox:\n  type: floating\n  title: Terminalizer\n  style:\n    border: 0px black solid\n    # boxShadow: none\n    # margin: 0px\n\n# Add a watermark image to the rendered gif\n# You need to specify an absolute path for\n# the image on your machine or a URL, and you can also\n# add your own CSS styles\nwatermark:\n  imagePath: null\n  style:\n    position: absolute\n    right: 15px\n    bottom: 15px\n    width: 100px\n    opacity: 0.9\n\n# Cursor style can be one of\n# `block`, `underline`, or `bar`\ncursorStyle: block\n\n# Font family\n# You can use any font that is installed on your machine\n# in CSS-like syntax\nfontFamily: \"Monaco, Lucida Console, Ubuntu Mono, Monospace\"\n\n# The size of the font\nfontSize: 12\n\n# The height of lines\nlineHeight: 1\n\n# The spacing between letters\nletterSpacing: 0\n\n# Theme\ntheme:\n  background: \"transparent\"\n  foreground: \"#afafaf\"\n  cursor: \"#c7c7c7\"\n  black: \"#232628\"\n  red: \"#fc4384\"\n  green: \"#b3e33b\"\n  yellow: \"#ffa727\"\n  blue: \"#75dff2\"\n  magenta: \"#ae89fe\"\n  cyan: \"#708387\"\n  white: \"#d5d5d0\"\n  brightBlack: \"#626566\"\n  brightRed: \"#ff7fac\"\n  brightGreen: \"#c8ed71\"\n  brightYellow: \"#ebdf86\"\n  brightBlue: \"#75dff2\"\n  brightMagenta: \"#ae89fe\"\n  brightCyan: \"#b1c6ca\"\n  brightWhite: \"#f9f9f4\"\n"
  },
  {
    "path": "configs/.config/yamllint/config",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-17 17:18:24 +0000 (Mon, 17 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         Y a m l L i n t   C o n f i g\n# ============================================================================ #\n\n---\nextends: default\n\nrules:\n  line-length: disable\n  comments: disable\n  comments-indentation: disable\n  #indentation:\n  #  ignore: |\n  #    */vagrant/kubernetes/calico.yaml\n  #empty-lines:\n  #  ignore: |\n  #    */vagrant/kubernetes/calico.yaml\n"
  },
  {
    "path": "configs/.gemrc",
    "content": "gem: --no-document --no-ri --no-rdoc\n"
  },
  {
    "path": "configs/.grype.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:54:24 +0000 (Mon, 10 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                   G r y p e\n# ============================================================================ #\n\n# https://github.com/anchore/grype#configuration\n\n---\n# enable/disable checking for application updates on startup\n# same as GRYPE_CHECK_FOR_APP_UPDATE env var\ncheck-for-app-update: true\n\n# upon scanning, if a severity is found at or above the given severity then the return code will be 1\n# default is unset which will skip this validation (options: negligible, low, medium, high, critical)\n# same as --fail-on ; GRYPE_FAIL_ON_SEVERITY env var\nfail-on-severity: ''\n\n# the output format of the vulnerability report (options: table, json, cyclonedx)\n# same as -o ; GRYPE_OUTPUT env var\noutput: \"table\"\n\n# suppress all output (except for the vulnerability list)\n# same as -q ; GRYPE_QUIET env var\nquiet: false\n\n# write output report to a file (default is to write to stdout)\n# same as --file; GRYPE_FILE env var\nfile: \"\"\n\n# a list of globs to exclude from scanning, for example:\n# exclude:\n#   - '/etc/**'\n#   - './out/**/*.json'\n# same as --exclude ; GRYPE_EXCLUDE env var\nexclude:\n\n\ndb:\n  # check for database updates on execution\n  # same as GRYPE_DB_AUTO_UPDATE env var\n  auto-update: true\n\n  # location to write the vulnerability database cache\n  # same as GRYPE_DB_CACHE_DIR env var\n  # XXX: this is coming out literally leaves these $XDG_CACHE_HOME dirs everywhere\n  #cache-dir: \"$XDG_CACHE_HOME/grype/db\"\n\n  # URL of the vulnerability database\n  # same as GRYPE_DB_UPDATE_URL env var\n  update-url: \"https://toolbox-data.anchore.io/grype/databases/listing.json\"\n\n\nsearch:\n\n  # the search space to look for packages (options: all-layers, squashed)\n  # same as -s ; GRYPE_SEARCH_SCOPE env var\n  scope: \"squashed\"\n\n\n  # search within archives that do contain a file index to search against (zip)\n  # note: for now this only applies to the java package cataloger\n  # same as GRYPE_PACKAGE_SEARCH_INDEXED_ARCHIVES env var\n  indexed-archives: true\n\n  # search within archives that do not contain a file index to search against (tar, tar.gz, tar.bz2, etc)\n  # note: enabling this may result in a performance impact since all discovered compressed tars will be decompressed\n  # note: for now this only applies to the java package cataloger\n  # same as GRYPE_PACKAGE_SEARCH_UNINDEXED_ARCHIVES env var\n  unindexed-archives: false\n\n\n# options when pulling directly from a registry via the \"registry:\" scheme\nregistry:\n  # skip TLS verification when communicating with the registry\n  # same as GRYPE_REGISTRY_INSECURE_SKIP_TLS_VERIFY env var\n  insecure-skip-tls-verify: false\n  # use http instead of https when connecting to the registry\n  # same as GRYPE_REGISTRY_INSECURE_USE_HTTP env var\n  insecure-use-http: false\n\n  # credentials for specific registries\n  auth:\n    - # the URL to the registry (e.g. \"docker.io\", \"localhost:5000\", etc.)\n      # same as GRYPE_REGISTRY_AUTH_AUTHORITY env var\n      authority: \"\"\n      # same as GRYPE_REGISTRY_AUTH_USERNAME env var\n      username: \"\"\n      # same as GRYPE_REGISTRY_AUTH_PASSWORD env var\n      password: \"\"\n      # note: token and username/password are mutually exclusive\n      # same as GRYPE_REGISTRY_AUTH_TOKEN env var\n      token: \"\"\n    #- ... # note, more credentials can be provided via config file only\n\n\nlog:\n  # use structured logging\n  # same as GRYPE_LOG_STRUCTURED env var\n  structured: false\n\n  # the log level; note: detailed logging suppress the ETUI\n  # same as GRYPE_LOG_LEVEL env var\n  #\n  # prevents specifying -v on the command line, although GRYPE_LOG_LEVEL still works\n  #level: \"error\"\n\n  # location to write the log file (default is not to have a log file)\n  # same as GRYPE_LOG_FILE env var\n  file: \"\"\n"
  },
  {
    "path": "configs/.inputrc",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-08 16:46:24 +0000 (Fri, 08 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# no bell sounds\n#set bell-style none\nset bell-style visible\n\n# tab completion on first press\nset show-all-if-ambiguous on\nset show-all-if-unmodified on\n\n# tab-complete case insensitive is useful sometimes but other times it's easier to differentiate by case early on\n#\n# eg. Dockerfile vs docker-compose.yml\n#\n# changing tab complete behaviour to case insensitive is also counter-intuitive to long time unix users like me,\n# even on Mac's case preserving but insensitive APFS filesystem\n#\n#set completion-ignore-case on\nset completion-ignore-case off # default\n\nset visible-stats on\n\n# trying to avoid alignment issues in emoji filename handling\nset input-meta on\nset output-meta on\nset convert-meta off\nset enable-bracketed-paste on\n"
  },
  {
    "path": "configs/.luacheckrc",
    "content": "--\n--  Author: Hari Sekhon\n--  Date: 2025-10-28 20:43:20 +0300 (Tue, 28 Oct 2025)\n--\n--  vim:ts=4:sts=4:sw=4:et\n--\n--  https///github.com/HariSekhon/DevOps-Bash-tools\n--\n--  License: see accompanying Hari Sekhon LICENSE file\n--\n--  If you're using my code you're welcome to connect with me on LinkedIn\n--  and optionally send me feedback to help steer this or other code I publish\n--\n--  https://www.linkedin.com/in/HariSekhon\n--\n\n-- .luacheckrc\n\n-- Define globals allowed throughout the project\nglobals = {\n   \"vim\",    -- for Neovim plugins\n   \"love\",   -- for LÖVE2D projects\n}\n\n-- Ignore specific warnings by code\nignore = {\n   \"631\",    -- allow line endings without semicolons\n   \"212\",    -- unused argument\n}\n\n-- Allow certain read-only standard globals\nread_globals = {\n   \"os\",\n   \"io\",\n   \"string\",\n   \"table\",\n   \"math\",\n   \"coroutine\",\n   \"debug\",\n   \"utf8\",\n}\n\n-- Set limits\nmax_line_length = 100\nmax_code_line_length = 100\nmax_string_line_length = 120\n\n-- Exclude certain directories from linting\nexclude_files = {\n   \"vendor/**\",\n   \"build/**\",\n   \"node_modules/**\",\n}\n\n-- Treat each file as a separate module\nstd = \"lua54\"\n\n-- Enable showing warning codes in output\ncodes = true\n"
  },
  {
    "path": "configs/.my.cnf",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Mon Mar 24 15:22:28 2008 +0000\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                       M y S Q L   U s e r   C o n f i g\n# ============================================================================ #\n\n# https://dev.mysql.com/doc/refman/8.0/en/option-files.html\n\n# See also:\n#\n#   https://github.com/HariSekhon/SQL-scripts\n\n[client]\nhost=localhost\n# avoid annoying socket errors when connecting to MySQL in docker\nprotocol=tcp\n\n[mysql]\nauto-rehash\nno-beep\nreconnect\n#safe-updates\n#secure-auth  # mysql: [ERROR] unknown option '--secure-auth'.\nshow-warnings\n#sigint-ignore\n"
  },
  {
    "path": "configs/.perlcritic_forbidden_modules",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-29 22:33:20 +0000 (Tue, 29 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nSwitch           Use if, elsif instead of the deprecated Switch module\nFatal            Use autodie instead of Fatal\n"
  },
  {
    "path": "configs/.perlcriticrc",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-29 22:33:20 +0000 (Tue, 29 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nseverity = 5\n# gives full explanations\n#verbose = 11\n\n#verbose = %f: [%p] %m at line %l, column %c.\\n%d\\n\n#\n# this output is ok, but I prefer the default\n#verbose = %f: [%p] %m at line %l, column %c.\\n\n\ntheme = core\n\n[TestingAndDebugging::RequireUseWarnings]\nseverity = 5\n\n# doesn't find this file with $HOME or ~\n#[Modules::ProhibitEvilModules]\n#modules_file = ~/.perlcritic_forbidden_modules\n"
  },
  {
    "path": "configs/.psqlrc",
    "content": "--\n--  Author: Hari Sekhon\n--  Date: 2020-03-16 10:14:28 +0000 (Mon, 16 Mar 2020)\n--\n--  vim:ts=4:sts=4:sw=4:et:filetype=sql\n--\n--  https://github.com/HariSekhon/DevOps-Bash-tools\n--\n--  License: see accompanying Hari Sekhon LICENSE file\n--\n--  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n--\n--  https://www.linkedin.com/in/HariSekhon\n--\n\n-- ========================================================================== --\n--         A d v a n c e d   P o s t g r e S Q L   U s e r   C o n f i g\n-- ========================================================================== --\n\n-- See also:\n--\n--      https://github.com/HariSekhon/SQL-scripts\n\n\n-- PSQL docs:\n--\n--      https://www.postgresql.org/docs/12/app-psql.html\n\n\n--\\! echo; echo .psqlrc loaded; echo\n\\echo\n\\echo '.psqlrc loaded'\n\\echo\n\n--\\set QUIET ON\n-- then later set\n--\\set QUIET OFF\n\n-- ========================================================================== --\n--                                  P r o m p t\n-- ========================================================================== --\n\n-- %M   fqdn\n-- %m   hostname / local for socket\n-- %>   port\n-- %n   username\n-- %x   transaction status - * in transaction, ! when transaction failed, ? when indeterminate\n-- %/   current database\n-- %~   like %/ but ~ for default database\n-- %R   = for connected\n--      ! for disconnected\n--      ^ for single line mode\n-- %#   '#' if superuser, '>' otherwise\n--\n-- %:myvar:     replaced with variable myvar\n-- %`command`   output of shell command\n-- %[%...%]     terminal colour control sequences\n\n------------------\n-- default prompt:\n--\n--              db=#\n--\\set PROMPT1 '%/%R%# '\n\n---------------------------\n-- fancy prompt colorized:\n--                          time          hostname                   serverhost/local user             transaction    database     = #\n\\set PROMPT1 '%[%033[32m%]%`date \"+%T\"` %`hostname -s` %[%033[35;1m%]%m:%[%033[36;1m%]%n %[%033[33;1m%]%x%[%033[;1m%]>%/%[%033[0m%]%R%# '\n\n-------------------------------------------------------------\n-- fancy prompt non-colorized (for PostgreSQL 9.0 - 9.2 environments where colors aren't working - probably a term/env issue):\n--\n--\\set PROMPT1 '%`date \"+%T\"` %`hostname -s` %m:%n %x>%/%R%# '\n\n-------------------------------\n\\set PROMPT2 '[continue] %R > '\n\n-- ========================================================================== --\n\n-- jump to your SQL scripts directory (this is done in postgres.sh)\n--\\cd ~/github/sql\n\n-- separate history files per DB\n\\set HISTFILE ~/.psql_history- :DBNAME\n\n-- default: 500\n\\set HISTSIZE 5000\n\n-- same as in Bash, don't record duplicate commands\n\\set HISTCONTROL ignoredups\n\n-- number of EOF (ctrl-D) to ignore before terminating\n\\set IGNOREEOF 0\n\n-- tab-complete SQL keywords to uppercase\n\\set COMP_KEYWORD_CASE upper\n\n-- args:\n--  default:    none\n--  all         all input lines are printed\n--  queries     queries are printed as they are sent to server\n--  errors      only failed queries are printed\n--\n-- useful to set in scripts\n--\\set ECHO queries\n\n-- echo the hidden queries for things like \\d\n\\set ECHO_HIDDEN ON\n\n-- display NULLs literally instead of as blanks which is confusing\n\\pset null 'NULL'\n\n-- ascii art tables\n\\pset border 2\n\n-- ========================================================================== --\n\n-- set to 'off' if inside a terminal multiplexer (screen, tmux)\n--\\pset pager on\n\n-- leave wrapping to inherit \\$COLUMNS\n--\\pset columns 80\n\n-- enforce vertical results instead of horizontal columns (default: auto - vertical when width > columns)\n--\\x auto\n--\\pset expanded auto\n\n-- replace | with \\t or , to create TSV/CSV output\n--\\pset fieldsep\n\n-- don't align columns for human readability (use when creating TSV/CSV output), default is 'aligned'\n--\\pset unaligned\n\n-- line separator for unaligned output (default: \\n)\n--\\pset recordsep\n\n-- show (N row) count after each query\n\\pset footer on\n\n-- show the encoding\n\\! printf \"Encoding: \"\n\\encoding\n\n-- ========================================================================== --\n\n-- interactively disables transaction rollbacks to fix things interactively, change of default behaviour might confuse though\n--\\set ON_ERROR_ROLLBACK interactive\n\n-- exit SQL script or return to interactive prompt upon first error - use this in production scripts\n--\\set ON_ERROR_STOP on\n\n-- same as -s / --single-step - use this to debug SQL scripts by prompting between executing each line\n--\\set SINGLESTEP on\n\n-- same as -S / --single-line - use newlines as SQL statement terminators so you don't need semicolons\n--\\set SINGLELINE on\n\n-- display context messages from the server - always, errors (default) or never\n--\\set SHOW_CONTEXT errors\n\n-- debug SQL errors: default, verbose or terse\n--\\set VERBOSITY verbose\n\n-- run after an error to get print the last error in verbose format as though VERBOSITY verbose was always set\n--\\errverbose\n\n-- shortcut alias\n\\set eav 'EXPLAIN ANALYZE VERBOSE'\n\n---------------------------------\n-- enable timings for every query\n\\timing on\n\n-- \\! echo\n\\echo\n\n---------\n-- doesn't work on PostgreSQL < 9.1, but doesn't error out so it's fine\n\\conninfo\n\n--\\! echo\n\\echo\n\n\n-- for local settings\n-- XXX: see if there is a way to check for existence and source only then\n--\\i ~/.psqlrc.local\n\n\n-- =========\n-- Variables\n--\n-- \\set myvar literal\n--\n-- call like\n--\n-- :myvar\n--\n-- or interpolate them into queries eg.\n--\n-- SELECT * FROM :tablename;\n--\n-- single quote the value to treat as an SQL literal  :'select'\n-- double quote the value to treat as a SQL identifier :\"tablename\"\n\n\n-- ========================================================================== --\n--                           Q u e r y   A l i a s e s\n-- ========================================================================== --\n\n-- can use be encased in single quotes. otherwise syntax error at the first double quote\n\n\\echo 'Query Aliases & Shortcuts:'\n\\echo\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_backends_per_database.sql\n\\echo '\\t' ':backends' '\\t' 'Number of connected backends per database'\n\\set backends 'SELECT datname,numbackends FROM pg_catalog.pg_stat_database ORDER BY numbackends DESC, datname ASC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_databases_by_size.sql\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_databases_by_size_if_accessible.sql\n\\echo '\\t' ':databases' '\\t' 'Databases by size desc (only ones you can access)'\n--\\set databases 'SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database ORDER BY pg_database_size(datname) DESC;'\n\\set databases 'SELECT d.datname AS Name, pg_catalog.pg_get_userbyid(d.datdba) AS Owner, CASE WHEN pg_catalog.has_database_privilege(d.datname, \\'CONNECT\\') THEN pg_catalog.pg_size_pretty(pg_catalog.pg_database_size(d.datname)) ELSE \\'No Access\\' END AS SIZE FROM pg_catalog.pg_database d ORDER BY CASE WHEN pg_catalog.has_database_privilege(d.datname, \\'CONNECT\\') THEN pg_catalog.pg_database_size(d.datname) ELSE NULL END DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_running_queries.sql\n\\echo '\\t' ':queries' '\\t' 'Running queries and their age'\n--\\set queries 'SELECT pid, age(clock_timestamp(), query_start), usename, application_name, query FROM pg_stat_activity WHERE state != \\'idle\\' AND query NOT ILIKE \\'%pg_stat_activity%\\' ORDER BY query_start DESC;'\n\\set queries 'SELECT pid, age(clock_timestamp(), query_start), datname, usename, application_name, query FROM pg_stat_activity WHERE state != \\'idle\\' ORDER BY query_start DESC;'\n\n\\echo '\\t' ':activity' '\\t' 'All clients and queries'\n\\set activity 'select pid, datname, usename, application_name,client_addr, client_hostname, client_port, query, state from pg_stat_activity;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_locks.sql\n\\echo '\\t' ':locks' '\\t' 'Locks'\n\\set locks 'SELECT t.schemaname, t.relname, l.locktype, page, virtualtransaction, pid, mode, granted FROM pg_locks l, pg_stat_all_tables t WHERE l.relation = t.relid ORDER BY relation ASC; SELECT relation::regclass AS relation_regclass, * FROM pg_locks WHERE NOT granted;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_locks_blocked.sql\n-- https://opensourcedbms.com/dbms/psqlrc-psql-startup-file-for-postgres/\n\\echo '\\t' ':blocked' '\\t' 'Blocked locks'\n\\set blocked 'SELECT bl.pid AS blocked_pid, a.usename AS blocked_user, kl.pid AS blocking_pid, ka.usename AS blocking_user, a.query AS blocked_statement FROM pg_catalog.pg_locks bl JOIN pg_catalog.pg_stat_activity a ON bl.pid = a.pid JOIN pg_catalog.pg_locks kl JOIN pg_catalog.pg_stat_activity ka ON kl.pid = ka.pid ON bl.transactionid = kl.transactionid AND bl.pid != kl.pid WHERE NOT bl.granted;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_tables_row_estimates.sql\n\\echo '\\t' ':rowcounts' '\\t' 'Row count estimates per table'\n\\set rowcounts 'SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_queries_slow.sql\n\\echo '\\t' ':slowqueries' '\\t' 'Slow queries (runtime > 30 secs)'\n\\set slowqueries 'SELECT now() - query_start as \"runtime\", usename, datname, wait_event, state, query FROM pg_stat_activity WHERE now() - query_start > \\'30 seconds\\'::interval ORDER BY runtime DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_sessions.sql\n\\echo '\\t' ':sessions' '\\t' 'Sessions (9.2+)'\n-- backend_type requires PostgreSQL 10+\n--\\set sessions 'SELECT pid, usename, client_addr, client_hostname, client_port, backend_start, query_start, state, backend_type FROM pg_stat_activity ORDER BY backend_type;'\n\\set sessions 'SELECT pid, usename, client_addr, client_hostname, client_port, backend_start, query_start, state FROM pg_stat_activity;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_session_state_count.sql\n-- using extra space because the tab pushes description to the next tab stop and misaligns help\n\\echo '\\t' ':sessionstates ' 'Session counts grouped by state'\n\\set sessionstates 'SELECT count(1), state FROM pg_stat_activity GROUP BY state ORDER BY 1 DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_session_state_count.sql\n\\echo '\\t' ':sessionusers' '\\t' 'Session counts grouped by user'\n\\set sessionusers 'SELECT count(1), usename FROM pg_stat_activity GROUP BY usename ORDER BY 1 DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_tables_by_size.sql\n\\echo '\\t' ':tables' '\\t' 'Tables by size ascending'\n\\set tables 'SELECT nspname, relname, pg_size_pretty(pg_total_relation_size(C.oid)) AS total_size FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE nspname NOT IN (\\'pg_catalog\\', \\'information_schema\\') AND C.relkind <> \\'i\\' AND nspname !~ \\'^pg_toast\\' ORDER BY pg_total_relation_size(C.oid) DESC;'\n\n\\echo\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_query_cache_hit_ratio.sql\n\\echo '\\t' ':qryhits' '\\t' 'Query cache-hit ratio desc'\n\\set qryhits 'CREATE EXTENSION IF NOT EXISTS pg_stat_statements; SELECT calls, rows, shared_blks_hit, shared_blks_read, shared_blks_hit / GREATEST(shared_blks_hit + shared_blks_read, 1)::float AS shared_blks_hit_ratio, local_blks_hit, local_blks_read, local_blks_hit / GREATEST(local_blks_hit + local_blks_read, 1)::float AS local_blks_hit_ratio, query FROM pg_stat_statements ORDER BY shared_blks_hit_ratio DESC, local_blks_hit_ratio DESC, rows DESC LIMIT 100;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_indexes_cache_hit_ratio.sql\n\\echo '\\t' ':idxhits' '\\t' 'Index cache-hit ratio'\n\\set idxhits 'SELECT SUM(idx_blks_read) AS idx_blks_read, SUM(idx_blks_hit)  AS idx_blks_hit, SUM(idx_blks_hit) / GREATEST(SUM(idx_blks_hit) + SUM(idx_blks_read), 1)::float AS ratio FROM pg_statio_user_indexes;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_tables_cache_hit_ratio.sql\n\\echo '\\t' ':tblhits' '\\t' 'Table cache-hit ratio'\n\\set tblhits 'SELECT SUM(heap_blks_read) AS heap_blks_read, SUM(heap_blks_hit)  AS heap_blks_hit, SUM(heap_blks_hit) / GREATEST(SUM(heap_blks_hit) + SUM(heap_blks_read), 1)::float AS ratio FROM pg_statio_user_tables;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_tables_index_usage.sql\n\\echo '\\t' ':tblidxuse' '\\t' 'Table index usage % desc'\n\\set tblidxuse 'SELECT relname AS table, idx_scan / GREATEST(seq_scan + idx_scan, 1) * 100 AS percent_of_times_index_used, n_live_tup AS rows_in_table FROM pg_stat_user_tables WHERE seq_scan + idx_scan > 0 ORDER BY rows_in_table DESC, percent_of_times_index_used DESC;'\n\n\\echo\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_last_analyze.sql\n\\echo '\\t' ':analyzed' '\\t' 'Last analyzed stats (9.3+)'\n\\set analyzed 'SELECT schemaname, relname, n_mod_since_analyze, last_analyze, last_autoanalyze, analyze_count, autoanalyze_count FROM pg_stat_user_tables ORDER BY n_mod_since_analyze DESC, last_analyze DESC, last_autoanalyze DESC;'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_last_vacuum.sql\n\\echo '\\t' ':vacuumed' '\\t' 'Last vaccumed stats (9.1+)'\n\\set vacuumed 'SELECT schemaname, relname, n_live_tup, n_dead_tup, n_dead_tup / GREATEST(n_live_tup + n_dead_tup, 1)::float * 100 AS dead_percentage, last_vacuum, last_autovacuum, vacuum_count, autovacuum_count FROM pg_stat_user_tables WHERE n_dead_tup > 0 ORDER BY n_dead_tup DESC, last_vacuum DESC, last_autovacuum DESC;'\n\n\\echo\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_dirs.sql\n\\echo '\\t' ':dirs' '\\t\\t' 'PostgreSQL config files, data and socket directory locations'\n\\set dirs 'SELECT current_setting(\\'config_file\\') AS \"config_file\", current_setting(\\'hba_file\\') AS \"hba_file\", current_setting(\\'ident_file\\') AS \"ident_file\"; SELECT current_setting(\\'data_directory\\') AS \"data_directory\", current_setting(\\'external_pid_file\\') AS \"external_pid_file\"; SELECT current_setting(\\'unix_socket_directories\\') AS \"unix_socket_directories\", current_setting(\\'unix_socket_permissions\\') AS \"unix_socket_permissions\", current_setting(\\'unix_socket_group\\') AS \"unix_socket_group\";'\n\n-- https://github.com/HariSekhon/SQL-scripts/blob/master/postgres_settings.sql\n\\echo '\\t' ':settings' '\\t' 'PostgreSQL settings (running + from config file)'\n\\set settings 'SELECT name, setting, unit, context FROM pg_settings; select * from pg_file_settings;'\n\n\\echo '\\t' ':uptime' '\\t' 'PostgreSQL postmaster uptime in seconds'\n\\set uptime 'select date_trunc(\\'second\\', current_timestamp - pg_postmaster_start_time() ) as uptime;'\n\n\\echo\n\n\\echo '\\t' ':ls' '\\t\\t' 'List files in $PWD'\n\\echo '\\t' ':ll' '\\t\\t' 'List files in $PWD with timestamps'\n\\echo '\\t' ':lt' '\\t\\t' 'List files in $PWD by timestamps ascending'\n\\set ls '\\\\! ls;'\n\\set ll '\\\\! ls -l;'\n\\set lt '\\\\! ls -ltr;'\n\n\\echo\n\n\\echo '\\t' ':clear' '\\t' 'Clear the screen'\n\\set clear '\\\\! clear;'\n\n\\echo\n\n\\echo '\\t' ':help' '\\t\\t' 'Reload ~/.psqlrc (which prints this helps)'\n\\set help '\\\\i ~/.psqlrc'\n\n\\echo '\\t' ':reload' '\\t' 'Reload ~/.psqlrc'\n\\set reload '\\\\i ~/.psqlrc'\n\n\\echo '\\t' ':r' '\\t\\t' 'Reload ~/.psqlrc'\n\\set r '\\\\i ~/.psqlrc'\n\n\\echo\n\\echo 'Many more excellent PostgreSQL queries are available at:'\n\\echo\n\\echo '\\t' 'https://github.com/HariSekhon/SQL-scripts'\n\\echo\n\\echo\n"
  },
  {
    "path": "configs/.sawsrc",
    "content": "[main]\n\n# Visual theme. Possible values: manni, igor, xcode, vim, autumn, vs, rrt,\n# native, perldoc, borland, tango, emacs, friendly, monokai, paraiso-dark,\n# colorful, murphy, bw, pastie, paraiso-light, trac, default, fruity\ntheme = vim\n\n# Use color output mode.\ncolor_output = True\n\n# Use fuzzy matching mode for resources (default is to use simple substring match).\nfuzzy_match = True\n\n# Use shortcut matching mode\nshortcut_match = True\n\n# log_file location.\nlog_file = ~/.saws.log\n\n# Default log level. Possible values: \"CRITICAL\", \"ERROR\", \"WARNING\", \"INFO\"\n# and \"DEBUG\".\nlog_level = INFO\n"
  },
  {
    "path": "configs/.screenrc",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: circa 2006 - 2008\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                                S c r e e n r c\n# ============================================================================ #\n\n# Reload .screenrc without restarting:\n#\n# Ctrl-a :source $HOME/.screenrc\nbind R source $HOME/.screenrc\n\nstartup_message off\n\ndefscrollback 20000\n\n# attempt to alleviate emoji filename character width readline issues\ndefutf8 on\n\n# enables monitor to notify of change in backgrounded panes (default: off)\n# do not enable - causes flickering ocassionally noticeable, but really awful when running welcome\n#defmonitor on\n\ncrlf off\nvbell off\n\n# default: off - echos command on every new pane\n#verbose on\n\n#version\n\n# redundant vs caption - ends up with 2 lines of the same stuff\n# caption is better when splitting\n#hardstatus string \"...\"\n\n# called only during Power Detach (C-a D D), ie. a detach that sends a HUP signal to screen's parent, results in a logout from the parent shell\npow_detach_msg \"Screen session of \\$LOGNAME \\$:cr:\\$:nl:ended.\"\n\n# set pw to prevent root attaching - if no crypted pw, initially prompts twice for pw\n#password [<crypted_pw>]\n\n#altscreen on\n\n# don't use this, wastes a line at bottom of terminal\n# and if setting string you end up with 2 status lines at bottom since 'caption always' is set further down\n#hardstatus on\n#hardstatus alwayslastline\n\n#term screen-256color\n\n# ============================================================================ #\n# default:\n# shell $SHELL\n#\n# dash prefix starts a login shell\n# shell -$SHELL\n#\n# must be full path because script which relies on $SHELL which is set by this\n#\n# might be a touch expensive as profile logins may do expensive rare initialization operations\n# instead rely on inheriting the profile environment by this point\n#shell -/bin/bash\nshell /bin/bash\n\n# ============================================================================ #\n# defaults to \"Unnamed\" otherwise\n# shelltitle \"\"\n#\n# Auto-Title - makes title the first word after the \"$ \" prompt or sets it to the thing on the other side of | which is simply blank\n#\n# Requires that PS1 have a null shelltitle escape sequence to be able to capture this\n#\n# causes marginal flickering\n#shelltitle '$ |'\n#\n# change prompt to use > instead of $\n# the space before the > is useful to distinguish the PS1 prompt from the PS4 prompt\n# 'screen -t blah' disables dynamic shelltitle on that new pane and cannot reactivate dynamic title even when re-sourcing this config or using functions from .bash.d/screen.sh profile\nshelltitle ' > |'\n\n# ============================================================================ #\n\n# bind    - prefixes ctrl-a\n# bindkey - plain key sequence\n\n# to unbind a key, run the bind again with no command\n\n# no longer exists - left as example\n#bindkey ^f screen -t Tech-Stable ssh tech-stable.harisekhon.com\n\n# open selection list to jump to any URLs in the current pane - this is better done in adjacent vim and tmux configs\n# ugly hack due to screen not supporting piping hardcopy and none of the usual shell / coreutil tricks work\n# eg. - or /dev/stdout or /dev/fd/1 - hardcopy silently fails if not regular file\n#bind u eval \"hardcopy hardcopy.tmp\" \"exec ... urlview hardcopy.tmp\" \"exec rm hardcopy.tmp\"\n#bind u exec ... urlview\n#bind u exec urlview\n\n# ctrl-a ctrl-x\n#bind ^x focus up\n\n# ctrl-a `\nbind ` focus up\n\n# Mac key in similar position to PC key `\n# doesn't work when testing 'ctrl-a §'\n# ignore since that key is slightly awkward and just use the up arrow set above\n#bind § focus up\n\n#bindkey ^` focus up\n# ctrl-a <up_arrow>\n# ctrl-a <down_arrow>\nbindkey \"^A^[OA\" focus up\nbindkey \"^A^[OB\" focus down\n\nbindkey -k k1 select 0  # F1 = screen 0\nbindkey -k k2 select 1  # F2 = screen 1\nbindkey -k k3 select 2  # F3 = screen 2\nbindkey -k k4 select 3  # F4 = screen 3\nbindkey -k k5 select 4  # F5 = screen 4\nbindkey -k k6 select 5  # F6 = screen 5\nbindkey -k k7 select 6  # F7 = screen 6\nbindkey -k k8 copy      # F8 = scrollback mode\nbindkey -k k9 title     # F9 = change title\nbindkey -k k; windowlist -b # F10 = windowlist\nbindkey -k F2 next      # F12 = next\nbindkey -k F1 prev      # F11 = previous\n\n# ============================================================================ #\n\n# %=       - pads space\n# .        - sets background transparent\n# %{-}     - pops one colour off the colour change stack to restore the last colour, useful for consistency\n# $LOGNAME - <user> - needs double quotes\n#\n# can use `` the same as in shell\n\n# Hari's style:\n#\n# transparent magenta <hostname> brightGreen <windows_before_current> brightBlue<num>*<flags> <title> <pop> windows_after_current <pad> brightRed <load> brightGreen <24HR>:<secs> <Day> <dd>/<Mon>/<YYYY>\ncaption always \"%{= .G}[ %{= .m}%H %{-}| %-Lw%?%{r}[%{B}%n%f* %t%{-}]%{-}%?%+Lw %= | %{= .R}%l %{-}| %c:%s %D %d/%M/%Y %{= .G}]\"\n#       splitonly (default - only shows during splits - but we want always to replace hardline and standardize look with or without splits)\n\n# ============================================================================ #\n\n#idle 60 blanker\nidle 0  # off\n\n#blankerprg cmatrix\n# save battery\nblankerprg \"\"\n"
  },
  {
    "path": "configs/.sdkman/etc/config",
    "content": "sdkman_auto_answer=false\nsdkman_auto_selfupdate=false\nsdkman_insecure_ssl=false\nsdkman_curl_connect_timeout=7\nsdkman_curl_max_time=10\nsdkman_beta_channel=false\nsdkman_debug_mode=false\nsdkman_colour_enable=true\n"
  },
  {
    "path": "configs/.sqliterc",
    "content": "--  vim:ts=4:sts=4:sw=4:et\n--\n--  Author: Hari Sekhon\n--  Date: 2020-07-21 20:39:48 +0100 (Tue, 21 Jul 2020)\n--\n--  https://github.com/HariSekhon/DevOps-Bash-tools\n--\n--  License: see accompanying Hari Sekhon LICENSE file\n--\n--  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n--\n--  https://www.linkedin.com/in/HariSekhon\n--\n\n.headers on\n.mode column\n.nullvalue NULL\n-- .prompt \"> \"\n.timer on\n"
  },
  {
    "path": "configs/.terraformrc",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-03-11 04:10:45 +0800 (Tue, 11 Mar 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        T e r r a f o r m   C o n f i g\n# ============================================================================ #\n\n# https://developer.hashicorp.com/terraform/cli/config/config-file\n\n# PLUGIN CACHING:\n#\n#   https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache\n#\n# Necessary to stop Terragrunt wasting tonnes of space\n#\n# You must create this cache dir manually otherwise Terraform will error out:\n#\n#    mkdir -p -v ~/.terraform.d/plugin-cache\n#\n# Otherwise Terraform / Terragrunt runs will error out like this...\n#\n#\n#   There are some problems with the CLI configuration:\n#   ╷\n#   │ Error: The specified plugin cache dir /Users/hari/.terraform.d/plugin-cache cannot be opened: stat /Users/hari/.terraform.d/plugin-cache: no such file or directory\n#   │\n#   ╵\n#\n#   As a result of the above problems, Terraform may not behave as intended.\n#\n#\nplugin_cache_dir   = \"$HOME/.terraform.d/plugin-cache\"\n#\n# Otherwise you could also waste space in these multiples...\n#\n# eg. terraform_provider_count_sizes.sh\n#\n#   30  597M  hashicorp/aws/5.80.0/darwin_arm64/terraform-provider-aws_v5.80.0_x5\n#   7   637M  hashicorp/aws/5.90.1/darwin_arm64/terraform-provider-aws_v5.90.1_x5\n#   4   637M  hashicorp/aws/5.90.0/darwin_arm64/terraform-provider-aws_v5.90.0_x5\n#   3   599M  hashicorp/aws/5.81.0/darwin_arm64/terraform-provider-aws_v5.81.0_x5\n#   2   593M  hashicorp/aws/5.79.0/darwin_arm64/terraform-provider-aws_v5.79.0_x5\n#   1   609M  hashicorp/aws/5.82.2/darwin_arm64/terraform-provider-aws_v5.82.2_x5\n#   1   605M  hashicorp/aws/5.83.1/darwin_arm64/terraform-provider-aws_v5.83.1_x5\n#   1   606M  hashicorp/aws/5.84.0/darwin_arm64/terraform-provider-aws_v5.84.0_x5\n#   1   624M  hashicorp/aws/5.86.0/darwin_arm64/terraform-provider-aws_v5.86.0_x5\n#   1   14M   hashicorp/external/2.3.4/darwin_arm64/terraform-provider-external_v2.3.4_x5\n#   1   14M   hashicorp/local/2.5.2/darwin_arm64/terraform-provider-local_v2.5.2_x5\n#   1   14M   hashicorp/null/3.2.3/darwin_arm64/terraform-provider-null_v3.2.3_x5\n#\n# Output on an Atlantis server pod after deleting all data cache to fix out of space errors and\n# then a single PR run:\n#\n#   14  654M  hashicorp/aws/5.90.1/linux_amd64/terraform-provider-aws_v5.90.1_x5\n#   13  14M   hashicorp/external/2.3.4/linux_amd64/terraform-provider-external_v2.3.4_x5\n#   13  14M   hashicorp/local/2.5.2/linux_amd64/terraform-provider-local_v2.5.2_x5\n#   13  14M   hashicorp/null/3.2.3/linux_amd64/terraform-provider-null_v3.2.3_x5\n#   3   346M  hashicorp/aws/4.67.0/linux_amd64/terraform-provider-aws_v4.67.0_x5\n#   3   621M  hashicorp/aws/5.80.0/linux_amd64/terraform-provider-aws_v5.80.0_x5\n#   3   653M  hashicorp/aws/5.90.0/linux_amd64/terraform-provider-aws_v5.90.0_x5\n#   3   14M   hashicorp/random/3.6.3/linux_amd64/terraform-provider-random_v3.6.3_x5\n#   1   627M  hashicorp/aws/5.82.2/linux_amd64/terraform-provider-aws_v5.82.2_x5\n#   1   630M  hashicorp/aws/5.84.0/linux_amd64/terraform-provider-aws_v5.84.0_x5\n"
  },
  {
    "path": "configs/.tfdocs.d/.terraform-docs.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-01-26 03:04:48 +0700 (Sun, 26 Jan 2025)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/terraform-docs/terraform-docs\n\n---\nformatter: \"\" # this is required\n\nversion: \"\"\n\nheader-from: main.tf\nfooter-from: \"\"\n\nrecursive:\n  enabled: false\n  path: modules\n  include-main: true\n\nsections:\n  hide: []\n  show: []\n\ncontent: \"\"\n\noutput:\n  file: \"\"\n  mode: inject\n  template: |-\n    <!-- BEGIN_TF_DOCS -->\n    {{ .Content }}\n    <!-- END_TF_DOCS -->\n\noutput-values:\n  enabled: false\n  from: \"\"\n\nsort:\n  enabled: true\n  by: name\n\nsettings:\n  anchor: true\n  color: true\n  default: true\n  description: false\n  escape: true\n  hide-empty: false\n  html: true\n  indent: 2\n  lockfile: true\n  read-comments: true\n  required: true\n  sensitive: true\n  type: true\n"
  },
  {
    "path": "configs/.tmux.conf",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-10-08 12:10:02 +0100 (Thu, 08 Oct 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Run 'make tmux' to install TPM before using this config or see git clone at bottom\n\n# ============================================================================ #\n#                      T M U X   C o n f i g u r a t i o n\n# ============================================================================ #\n\n# Ctrl-b r - reloads .tmux.conf\n# remember tmux reuses backend server process which doesn't go down unless you do 'tmux kill-server' (killing tmux server process is an easy reset if you don't want to undo options you've been messing with interactively)\nbind r source-file ~/.tmux.conf \\; display \".tmux.conf reloaded\"\n\n# Ctrl-b b - reloads .bashrc - .bash.d/aliases.sh 'r' enter keystroke is easier to type\n#bind b send-keys \"source ~/.bashrc\" \"Enter\" \\; display \".bashrc reloaded\"\n\n# 'tmux attach' starts new session if none exists\n# when combined with reloading above, however, causing a new session to be created every time\n#new-session\n#attach\n\n# ============================================================================ #\n# change default meta key from Ctrl-b to Ctrl-a to be more like screen\n#\n# seemed like a good idea initially until working on various companies servers and your muscle memory is wrong or\n# you need to embed screen and tmux sessions (eg. screen on laptop, tmux on remote server) without having to double escape meta keys\n#\n#unbind C-b\n#set -g prefix C-a\n#\n# sends meta through to other apps eg. screen\n#bind C-a send-prefix\n#bind a send-prefix\n\n# ============================================================================ #\n#                              S t a t u s   B a r\n# ============================================================================ #\n\n# based on my screen colour style - might be better to leave tmux looking distinct with it's original block green\n# more complex multi-colours now set in window-status-format / window-current-status-format below instead\n# need this to stay set to make sure status-separator doesn't revert to bg=green\n#\n# old tmux on Amazon Linux gives this error:\n#\n#   unknown option: status-style\n#\nset -g status-style \"bg=black fg=brightgreen\"\n\n# ========================================\n# window list - setting to red if panes are synchronized\n# #I - window index\n# #W - window name\nsetw -g window-status-current-format \"#[bg=black fg=red][#{?pane_synchronized,#[bg=brightred fg=white],#[bg=black fg=brightblue]}#I* #W#[bg=black fg=red]]\"\nsetw -g window-status-format         \"#{?pane_synchronized,#[bg=brightred fg=white],#[bg=black fg=brightgreen]} #I #W\"\nsetw -g window-status-separator \" \"\n\n# toggle sending keystrokes to all panes in split - quick way of multi-commanding servers (like cssh back in the day)\nbind y set synchronize-panes\\; display 'synchronize-panes #{?synchronize-panes,on,off}'\n\n# ========================================\nsetw -g monitor-activity on\n#setw -g window-status-activity-style \"bg=black fg=magenta\"\n\n# ========================================\n# Status Left\n\nset -g status-left \"#[bg=black fg=green][ #[fg=magenta]#h #[bg=black fg=green] | #S | \"\nset -g status-left-length 50\n\n# ========================================\n# Status Right\n#\n# #{battery_percentage} - Mac shows battery percentage at top right so don't need this\n# #{online_status} - tmux-plugins/tmux-online-status - causes climbing duplicate status bars at bottom\n# #(bash ~/.tmux/plugins/kube.tmux 250 magenta green) - may not have kubernetes, also expensive to re-execute this every 5 secs\n# Continuum: #{continuum_status}\nset -g status-right \"#[bg=black fg=green]| %H:%M %d/%b/%Y ]\"\n#set -g status-right-length 100\n\n#%if #{TMUX}\n#set -g status-bg red\n#%endif\n\n\n# ============================================================================ #\n#                                S e t t i n g s\n# ============================================================================ #\n\n# set is short for set-option\n# -g = global\n# -s = server / session\n# -u = unsets session to inherit global, or resets global to default with -g\n# -o = prevents resetting an already set value\n# -a = appends to current value\n\n# show all settings:\n#\n# tmux show -g\n\n# don't auto-rename windows\n# set-option -g allow-rename off\n\nset -g status-justify left # or centre\n\nsetw -g alternate-screen on  # default, restore previous contents when exiting interactive app\nset -g bell-action none\nset -g visual-bell off\n\n# renumber windows when one closes\nset -g renumber-windows on\n\n# old tmux on Amazon Linux gives this error:\n#\n#   unknown option: status-style\n#\nset -g focus-events on\n\n# if this line gives you an 'ambiguous' error, upgrade tmux\n#\n# old tmux on Amazon Linux gives this error:\n#\n#   unknown option: status-style\n#\nset -g mouse on\n\n# disable copy-mode dragging\n#unbind -n MouseDrag1Pane\n#unbind -Tcopy-mode MouseDrag1Pane\n\n#set -g base-index 1  # start window count from 1 instead of 0\n#setw -g pane-base-index 1  # start pane count from 1 instead of 0\n\n# ===============================\n# these are set in tmux-sensible\n#\n#setw -g utf8 on\n#set -g status-interval 15  # default: 15 - how often to redraw status and execute tmux plugins\n#set -g default-terminal \"screen-256color\"  # force assuming colors without 'tmux -2' or TERM environment variable\nset -g history-limit 100000  # give us even bigger history than tmux-sensible\n\n\n# ============================================================================ #\n#                             K e y b i n d i n g s\n# ============================================================================ #\n\n# be more like screen to toggle between last window - no longer used as interferes with portal muscle memory when working on various companies servers\n#bind b last-window\n\n#bind-key C-c run-shell \"tmux show-buffer | xclip -i\"\n#bind-key C-v run-shell \"tmux set-buffer \\\\\"$(xclip -o)\\\\\"; tmux paste-buffer\"\n\n# millis to wait a key after meta - default is 500ms\n# set higher for remote servers with latency\n# set lower for local sessions for speed / responsiveness\nset -s escape-time 1\n\n\n# use expanded view - default is choose-tree view which lists sessions collapsed\n#bind s choose-tree -u\n\n# don't recommend changing these from the defaults if you work on other people's servers as a consultant\n#bind | split-window -h\n#bind - split-window -v\n# retain $PWD when splitting\n#bind _ split-window -v -c \"#{pane_current_path}\"\n#bind \\ split-window -h -c \"#{pane_current_path}\"\n\n# ====================================\n# shortcuts to higher numbered windows\nbind F1 selectw -t:10\nbind F2 selectw -t:11\nbind F3 selectw -t:12\nbind F4 selectw -t:13\nbind F5 selectw -t:14\nbind F6 selectw -t:15\nbind F7 selectw -t:16\nbind F8 selectw -t:17\nbind F9 selectw -t:18\nbind F10 selectw -t:19\nbind F11 selectw -t:20\nbind F12 selectw -t:21\n\n#toggle between smallest and largest sizes if a window is visible in multiple places\n#bind F set -w window-size\n\n# switch panes with Alt-arrow\nbind -n M-Left select-pane -L\nbind -n M-Right select-pane -R\nbind -n M-Up select-pane -U\nbind -n M-Down select-pane -D\n\n\n# ============================================================================ #\n#                            T M U X   p l u g i n s\n# ============================================================================ #\n\n# Ctrl-b I to fetch plugins\n\nset -g @plugin 'tmux-plugins/tmux-sensible'  # options everyone should be ok with\nset -g @plugin 'tmux-plugins/tmux-resurrect' # Ctrl-b Ctrl-s to save sessions, Ctrl-b Ctrl-r to restore\nset -g @plugin 'tmux-plugins/tmux-continuum'  # needs tmux-resurrect\nset -g @plugin 'tmux-plugins/tmux-urlview'\n#set -g @plugin 'tmux-plugins/tmux-online-status'  # overkill, just call .bash.d/ functions & aliases when needed\n#set -g @plugin 'tmux-plugins/tmux-pain-control'\n#set -g @plugin 'tmux-plugins/tmux-battery'  # Mac shows this at top right so don't need these bash scripts\n\nset -g @continuum-save-interval '15'  # default: 15, set to 0 to stop automatic saving\nset -g @continuum-restore 'on'  # automatic restore\nset -g @continuum-boot 'on'     # auto-start terminal at boot\nset -g @continuum-boot-options 'fullscreen'\n\n# if icons don't display properly set this\n#set -g @online_icon \"ok\"\n#set -g @offline_icon \"offline!\"\n\n# ============\n\n# Initialize TMUX plugin manager (keep this line at the very bottom of tmux.conf)\n#\n# download TPM like so before launching/reloading tmux:\n#\n# git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm\n#\nrun -b '~/.tmux/plugins/tpm/tpm'\n"
  },
  {
    "path": "configs/.toprc",
    "content": "RCfile for \"top with windows\"\t\t# shameless braggin'\nId:a, Mode_altscr=0, Mode_irixps=1, Delay_time=3.000, Curwin=0\nCPU\tfieldscur=AEHIOQTWKNMbcdfgjplrsuvyzX\n\twinflags=65337, sortindx=10, maxtasks=0\n\tsummclr=2, msgsclr=6, headclr=2, taskclr=6\nTIM\tfieldscur=ABcefgjlrstuvyzMKNHIWOPQDX\n\twinflags=65337, sortindx=12, maxtasks=0\n\tsummclr=6, msgsclr=6, headclr=1, taskclr=1\nMEM\tfieldscur=ANOPQRSTUVbcdefgjlmyzWHIKX\n\twinflags=65337, sortindx=14, maxtasks=0\n\tsummclr=5, msgsclr=5, headclr=4, taskclr=5\nUSR\tfieldscur=AbdEcgfHIjlOPQrstuvyzMKNWX\n\twinflags=65337, sortindx=7, maxtasks=0\n\tsummclr=3, msgsclr=3, headclr=2, taskclr=3\n"
  },
  {
    "path": "configs/.vimrc",
    "content": "\"  vim:ts=4:sts=4:sw=4:tw=0:et\n\"\n\"  Author: Hari Sekhon\n\"  Date: 2006-07-01 22:52:16 +0100 (Sat, 01 Jul 2006)\n\"\n\"  https://github.com/HariSekhon/DevOps-Bash-tools\n\"\n\"  License: see accompanying Hari Sekhon LICENSE file\n\"\n\"  If you're using my code you're welcome to connect with me on LinkedIn\n\"  and optionally send me feedback to help steer this or other code I publish\n\"\n\"  https://www.linkedin.com/in/HariSekhon\n\"\n\n\" ============================================================================ \"\n\"                                   v i m r c\n\" ============================================================================ \"\n\n\" to reload without restarting vim\n\"\n\" :source ~/.vimrc\n\nset encoding=utf-8      \" The encoding displayed\nset fileencoding=utf-8  \" The encoding written to file\n\nscriptencoding utf-8\n\n\" if you cursor location and copy/paste register buffers are not saving\n\" then ensure that your ~/.viminfo file is owned by your user:\n\"\n\"   sudo chown \"$USER\" ~/.viminfo\n\nsyn on\n\nhighlight StatusLine ctermfg=yellow ctermbg=darkgray\nhighlight StatusLineNC ctermfg=darkgrey ctermbg=yellow\nhighlight VertSplit ctermfg=darkgrey ctermbg=yellow\n\"set statusline=%f\\ %h%w%m%r\\ %=%-14.(%l,%c%V%)\\ %P\n\nset visualbell\n\n\" ============================================================================ \"\n\"                               S e t   C o n f i g\n\" ============================================================================ \"\n\n\" show all settable option values and their values\n\"set all\n\nset ai      \" autoindent\nset backspace=indent,eol,start \" fixes not being able to backspace not typed during current insert mode session\nset bg=dark \" background\nset et      \" expandtab\nset ic      \" ignorecase\nset is      \" incsearch\n\"set list   \" visually displays eol, tabs etc so you can always see them\nset ls=2    \" laststatus - Status line 0=off, 1=multi-windows, 2=on\nset listchars=tab:>-,eol:$,trail:.,extends:# \" changes the list characters, makes tabs appear as >---\nset ml      \" modeline - respect the vim: stuff at the stop of files, often off for root\nset mls=15  \" modelines - Controls how many lines to check for modeline, systems often set this to 0\n\"set nocp    \" nocompatible\nset nofen   \" nofoldenable\nset nohls   \" nohlsearch\nset nojs    \" nojoinspaces - only use 1 space even when J joining lines even when line ends in a special char\n\"set nu      \" number (column on left)\nset ru      \" ruler\nset sm      \" showmatch. show matching brackets {}\nset scs     \" smartcase. switch to case sensitive match if uppercase letter is detected\nset si      \" smartindent\nset smd     \" showmode\n\" enabling either one of these causes deleting 4 spaces with each backspace, often over deletes when writing yaml / docs\nset sta     \" smarttab - make 'tab' insert indents instead of tabs at beginning of line\nset sts=4   \" softtabstop\nset sw=4    \" shiftwidth - number of spaces for indentation, should be the same as tabstop really to make tabs and Shift-> the same width\nset ts=4    \" tabstop\nset tw=0    \" textwidth (stops auto wrapping)\nset viminfo='100,<1000,s10,h \" save <1000 lines in the registers instead of <50 lines between files since otherwise I lose lots of lines when deleting and pasting between files\nset wrap    \" line wrapping\n\n\" reload the buffer when file has changed but buffer has not (useful for 'go fmt' / 'git pull' hotkeys from within vim)\nset autoread\n\n\" write buffer on next / prev etc\nset autowrite\n\n\" add comment to next line when using 'o' in command mode\n\" add comment to next line when using Insert mode\nset formatoptions+=or\n\n\" double forward slash means put backup files directly under this directory instead of trying to recreate the directory structure which might fail\n\"set backupdir=.,~//\n\n\" ============================================================================ \"\n\"                                   G U I\n\" ============================================================================ \"\n\n\" see also ~/.gvimrc.local sourcing at bottom of this config\n\n\"behave mswin\nbe xterm\n\n:if has('gui_running')\n    \"colorscheme slate\n    colo slate\n:endif\n\n\n\" ============================================================================ \"\n\"                               P l u g i n s\n\" ============================================================================ \"\n\nfiletype plugin on\nfiletype plugin indent on\n\"filetype off\n\n\" shows what last set ts, ie .vimrc\n\":verbose set ts\n\n\" set scrollbind - in each window then windows will scroll together\n\n\" =======\n\" Vundle\n\"\n\" to install plugins do:\n\"\n\" :PluginInstall\n\n\" set the runtime path to include Vundle and initialize\nset rtp+=~/.vim/bundle/Vundle.vim\n\ncall vundle#begin()\n\" alternatively, pass a path where Vundle should install plugins\n\"call vundle#begin('~/some/path/here')\n\n\" let Vundle manage Vundle, required\nPlugin 'VundleVim/Vundle.vim'\n\nPlugin 'airblade/vim-gitgutter'\n\"Plugin 'fatih/vim-go'\nPlugin 'hashivim/vim-consul'\n\"Plugin 'hashivim/vim-nomadproject'\n\"Plugin 'hashivim/vim-ottoproject'\nPlugin 'hashivim/vim-packer'\nPlugin 'hashivim/vim-terraform'\nPlugin 'hashivim/vim-vagrant'\n\"Plugin 'hashivim/vim-vaultproject'\nPlugin 'vim-syntastic/syntastic'\n\"Plugin 'dense-analysis/ale'\nPlugin 'juliosueiras/vim-terraform-completion'\n\" gets in the way editing more than it helps because it adds doubles of quotes\n\" and braces that often break edits or require more keystrokes to remove than saved\n\"Plugin 'jiangmiao/auto-pairs'\nPlugin 'preservim/nerdcommenter'\nPlugin 'terrastruct/d2-vim'\n\"Plugin 'tpope/vim-commentary'\nPlugin 'tpope/vim-fugitive'\nPlugin 'tpope/vim-surround'\nPlugin 'tmux-plugins/vim-tmux'\n\" https://github.com/motus/pig.vim\n\"Bundle \"motus/pig.vim\"\nPlugin 'wakatime/vim-wakatime'\n\n\" comment at start of line instead of code indentation level\n\" doesn't work: https://github.com/preservim/nerdcommenter/issues/467\nlet g:NERDDefaultAlign = 'left'\nlet g:NERDCommentEmptyLines = 1\n\nlet g:gitgutter_enabled = 0\n\" keep setting if reloading, otherwise default to 1 for enabled\n\"let g:pluginname_setting = get(g:, 'gitgutter_enabled', 1)\n\n\" align settings automatically with Tabularize\nlet g:terraform_align=1\nlet g:terraform_fold_sections=0\n\" auto format *.tf /*.tfvars with 'terraform fmt' or manually calling :TerraformFmt\nlet g:terraform_fmt_on_save=1\n\n\" Syntastic Config\n\"set statusline+=%#warningmsg#\n\"set statusline+=%{SyntasticStatuslineFlag()}\n\"set statusline+=%*\n\n\"let g:syntastic_always_populate_loc_list = 1\n\"let g:syntastic_auto_loc_list = 1\n\"let g:syntastic_check_on_open = 1\n\"let g:syntastic_check_on_wq = 0\n\n\" (Optional) Enable terraform plan to be include in filter\n\"let g:syntastic_terraform_tffilter_plan = 1\n\ncall vundle#end()\n\n\n\" ============================================================================ \"\n\"                           K e y b i n d i n g s\n\" ============================================================================ \"\n\n\"nmap <silent> ;c :call Cformat() <CR>\nnmap <silent> ;a :,!anonymize.py -a <CR>\nnmap          ;A :,!hexanonymize.py --case --hex-only <CR>\nnmap <silent> ;b :!git blame \"%\" <CR>\n\"nmap <silent> ;c :call ToggleComments()<CR>\nnmap <silent> ;c :,!center.py<CR>\nnmap <silent> ;e :,!center.py --space<CR>\nnmap <silent> ;C :,!center.py --unspace<CR>\n\" parses current example line and passes as stdin to bash to quickly execute examples from code - see WriteRunLine() further down for example\n\" messes up interactive vim (disables vim's arrow keys) - calling a terminal reset fixes it\nnmap <silent> ;E :call WriteRunLine() \\| !reset <CR><CR>\nnmap <silent> ;d :r !date '+\\%F \\%T \\%z (\\%a, \\%d \\%b \\%Y)'<CR>kJ\n\"nmap <silent> ;D :Done<CR>\nnmap <silent> ;D :%!decomment.sh \"%\" <CR>\nnmap          ;f :,!fold -s -w 120 \\| sed 's/[[:space:]]*$//'<CR>\nnmap <silent> ;h :call Hr()<CR>\nnmap          ;H :call WriteHelp()<CR>\n\" this inserts Hr literally\n\"imap <silent> <C-H> :Hr<CR>\nnmap <silent> ;I :PluginInstall<CR>\nnmap          ;i :! idea % \\| wq <CR>\nnmap <silent> ;j :JHr<CR>\nnmap          ;k :w \\| ! check_kubernetes_yaml.sh \"%\" <CR>\n\"nmap <silent> ;' :call Sq()<CR>\n\" done automatically on write now\n\"nmap <silent> ;' :call StripTrailingWhiteSpace()<CR>\nnmap <silent> ;' :w \\| !clear; git diff \"%\" <CR>\nnmap          ;m :w \\| call MarkdownIndex() <CR>\nnmap          ;n :w \\| n<CR>\nnmap          ;o :!cd \"%:p:h\" && git log -p \"%:t\" <CR>\nnmap          ;O :call ToggleGutter()<CR>\nnmap          ;p :prev<CR>\n\"nmap          ;P :call TogglePaste()<CR>\n\"should be the same thing according to :help pastetoggle but results in vim startup error 'Not an editor command'\n\"pastetoggle P\nnmap          ;P :set paste!<CR>\nnmap          ;t :set list!<CR>\nnmap          ;q :q<CR>\nnmap          ;r :call WriteRun()<CR>\nnmap          ;R :call WriteRunDebug()<CR>\n\"nmap          ;R :!run.sh %:p<CR>\n\"nmap <silent> ;s :call ToggleSyntax()<CR>\nnmap <silent> ;s :,!sqlcase.pl<CR>\n\"nmap          ;; :call HgGitU()<CR>\n\" command not found\n\"nmap          ;; :! . ~/.bashrc; gitu \"%\" <CR>\n\" sometimes doesn't trigger, see if removing the space helps\n\"nmap          ;; :w!<CR>:call GitUpdateCommit() <CR>\n\" technically more accurate to not leave command mode\nnmap          ;; :w! \\| call GitUpdateCommit() <CR>\n\"nnoremap <nowait> ;; :w! \\| call GitUpdateCommit() <CR>\nnmap          ;/ :w \\| call GitAddCommit() <CR>\nnmap          ;g :w \\| call GitStatus() <CR>\nnmap          ;G :w \\| call GitLogP() <CR>\nnmap          ;L :w \\| ! lint.sh % <CR>\nnmap          ;. :w \\| call GitPull() <CR>\nnmap          ;[ :w \\| call GitPush() <CR>\nnmap          ;, :w \\| s/^/</ \\| s/$/>/ <CR>\n\" write then grep all URLs that are not mine, followed by all URLs that are mine in reverse order to urlview\n\" this is so that 3rd party URLs followed by my URLs from within the body of files get higher priority than my header links\n\"nmap <silent> ;u :w \\| ! bash -c 'grep -vi harisekhon \"%\" ; grep -i harisekhon \"%\" \\| tail -r' \\| urlview \\| <CR>\nnmap <silent> ;u :w \\| ! bash -c '{ urlextract.sh \"%\" ; terraform_registry_url_extract.sh \"%\"; } \\| terraform_registry_url_to_https.sh \\| lines_to_end.sh harisekhon \\| urlview' \\| <CR>\n\" pass current line as stdin to urlview to quickly go to this url\n\" messes up interactive vim (disables vim's arrow keys) - calling a terminal reset fixes it\n\"nmap <silent> ;U :.w !urlview<CR> \\| !reset<CR><CR>\nnmap <silent> ;U :.w !urlopen.sh<CR><CR>\n\" breaks ;; nmap\n\"nmap          ;\\ :source ~/.vimrc<CR>\n\"nmap          ;/ :source ~/.vimrc<CR>\n\"nmap          ;v :source ~/.vimrc<CR>\nnmap          ;v :call SourceVimrc()<CR>\nnmap          ;V :call WriteRunVerbose()<CR>\nnmap          ;w :w<CR>\n\"nmap          ;x :x<CR>\nnmap          ;y :w !pbcopy<CR><CR>\nnmap          ;z :call ToggleDebug()<CR>\nnmap          ;§ :call ToggleScrollLock()<CR>\n\n\"noremap <silent> ,cc :<C-B>silent <C-E>s/^/<C-R>=escape(b:comment_char,'\\/')<CR>/ \\| nohlsearch<CR>\n\"noremap <silent> ,cu :<C-B>silent <C-E>s/^\\V<C-R>=escape(b:comment_char,'\\/')<CR>//e \\| nohlsearch<CR>\n\n\" reloading with these didn't fix above pipe disabling arrow keys but\n\" adding a terminal reset after the pipe command did fix it\n\"noremap <Up>    <Up>\n\"noremap <Down>  <Down>\n\"noremap <Left>  <Left>\n\"noremap <Right> <Right>\n\n\" ============================================================================ \"\n\"                               A u t o c m d\n\" ============================================================================ \"\n\nnmap ;l :echo \"No linting defined for this filetype:\" &filetype<CR>\n\nif has('autocmd')\n  augroup filetype_settings\n\n    \" re-open at last cursor line and center screen on the cursor line\n    \"au BufReadPost * if line(\"'\\\"\") > 1 && line(\"'\\\"\") <= line(\"$\") | exe \"normal! g'\\\"\" | endif\n    autocmd BufReadPost *\n      \\ if line (\"'\\\"\") > 0 && line (\"'\\\"\") <= line (\"$\") |\n      \\    exe \"normal! g`\\\"\" |\n      \\    exe \"normal! g`\\\"zz\" |\n      \\ endif\n\n    \" on write - auto-strip trailing whitespace on lines and remove trailing whitespace only lines end at of file\n    autocmd BufWritePre * %s/\\s\\+$//e | %s#\\($\\n\\s*\\)\\+\\%$##e\n\n    \" writes a vim script that saves and restore folds, ts/sts/sw options etc\n    \"autocmd BufWinLeave *.* mkview\n    \"autocmd BufWinEnter *.* silent loadview\n\n    \" doubles up with nmap ;l\n    \"au BufWritePost *.tf,*.tfvars :!clear; cd \"%:p:h\" && terraform fmt -diff; terraform validate\n\n    \" highlight trailing whitespace\n    \" XXX: doesn't work\n    \"autocmd ColorScheme * highlight ExtraWhitespace ctermbg=red guibg=red\n    \" this works - not using now we auto-strip trailing whitespace above on write anyway and it feels like this constant on/off highlighting slows things down and wastes energy\n    \"autocmd Filetype * match Error /\\s\\+$/\n\n    au BufNewFile,BufRead Makefile set noet\n    au BufNewFile,BufRead *.md set ts=2 sw=2 sts=2 et\n    au BufNewFile,BufRead Jenkinsfile* set filetype=groovy ts=2 sw=2 sts=2 et\n\n    au BufNewFile,BufRead LICENSE set tw=80\n\n    au BufRead,BufNewFile perl set ts=4 sts=4 et\n    au BufRead,BufNewFile *.pl set ts=4 sts=4 et\n\n    \"au BufNew,BufRead *.pp set syntax=conf\n    \" filetype is better than syntax since it figures out indentation and tab completion etc and the ruby is better than conf since it gives more syntax highlighting\n    au BufNew,BufRead *.pp set filetype=ruby sts=2 sw=2 ts=2 et\n\n    au BufNew,BufRead     *.tf  set filetype=terraform sts=2 sw=2 ts=2 et\n    au BufNewFile,BufRead *.hcl set filetype=terraform sts=2 sw=2 ts=2 et\n\n    au BufNew,BufRead *.yml set sts=2 sw=2 ts=2 et\n    au BufNew,BufRead *.yaml set sts=2 sw=2 ts=2 et\n\n    au BufNew,BufRead *.groovy,*.gvy,*.gy,*.gsh set filetype=groovy\n    au BufNew,BufRead *.jsh set filetype=java\n\n    \"au BufNew,BufRead *.rb set filetype=ruby sts=2 sw=2 ts=2\n\n    \" this will disable\n    \"au BufNew,BufRead *.txt set ft=\n    au BufNew,BufRead *.txt hi def link confString  NONE\n\n    au BufRead,BufNewFile *.hta setfiletype html\n\n\"    autocmd FileType c,cpp,java,scala let b:comment_char = '//'\n\"    autocmd FileType sh,perl,python   let b:comment_char = '#'\n\"    autocmd FileType ruby             let b:comment_char = '#'\n\"    autocmd FileType conf,dockerfile,fstab let b:comment_char = '#'\n\"    autocmd FileType sql              let b:comment_char = '--'\n\"    autocmd FileType tex              let b:comment_char = '%'\n\"    autocmd FileType mail             let b:comment_char = '>'\n\"    autocmd FileType vim              let b:comment_char = '\"'\n\n    au BufNew,BufRead *   nmap ;l :w \\| !clear; lint.sh \"%\" <CR>\n\n    \" headtail.py is useful to see the top things to fix and the score on each run and can be found in the\n    \" https://github.com/HariSekhon/Python-DevOps-Tools repo which should be downloaded, run 'make' and add to $PATH\n    au BufNew,BufRead *.py   nmap ;l :w \\| !clear; pylint \"%\" \\| headtail.py <CR>\n    au BufNew,BufRead *.pl   nmap ;l :w \\| !clear; perl -I . -tc \"%\" <CR>\n    au BufNew,BufRead *.rb   nmap ;l :w \\| !clear; ruby -c \"%\" <CR>\n    \" :e reloads the file because autoread isn't working after gofmt in this case\n    au BufNew,BufRead *.go   nmap ;l :w \\| !gofmt -w \"%\" && go build \"%\" <CR>\n    \" breaks waiting to see go build error\n    \" :e<CR>\n\n    \" to make vim autoread after gofmt\n    \" doesn't seem to work, using explicit :e now\n    \"au CursorHold * checktime\n    \"au CursorHold,CursorHoldI * checktime\n    \"au FocusGained,BufEnter * :checktime\n\n    \" TODO: any better groovy/java CLI linters\n    au BufNew,BufRead *.groovy,*.gvy,*.gy,*.gsh  nmap ;l :w \\| !groovyc \"%\"; rm -f -- \"%:p:h\"/*.class <CR>\n\n    \" TODO: often these don't trigger on window switching between different file types\n\n    \" %:t = basename of file\n    au BufNew,BufRead .bash*,*.sh,*.ksh   nmap ;l :w \\| !clear; cd \"%:p:h\" && shellcheck -x -Calways \"%:t\" \\| less -FR <CR>\n    \" for scripts that don't end in .sh like Google Cloud Shell's .customize_environment\n    au FileType sh                        nmap ;l :w \\| !clear; cd \"%:p:h\" && shellcheck -x -Calways \"%:t\" \\| less -FR <CR>\n\n    \" this is replaced by the global fallback mapping after first call\n    \"au BufNewFile,BufRead .vimrc nnoremap ;l :w \\| redraw! \\| call LintVimrc()<CR>\n    au BufRead,BufNewFile .vimrc nnoremap <buffer> ;l :w \\| redraw! \\| call LintVimrc()<CR>\n\n    \" these tools are in the https://github.com/HariSekhon/DevOps-Python-tools & DevOps-Bash-tools repos which should be downloaded, run 'make' and add to $PATH\n    au BufNew,BufRead *.csv        nmap ;l :w \\| !clear; validate_csv.py \"%\" <CR>\n    au BufNew,BufRead *.cson       nmap ;l :w \\| !clear; validate_cson.py \"%\" <CR>\n    au BufNew,BufRead *.d2         nmap ;l :w \\| !clear; d2 fmt \"%\" \\| edit \\| 1s/^# \\!\\//#\\!\\// <CR>\n    au BufNew,BufRead *.json       nmap ;l :w \\| !clear; validate_json.py \"%\"; echo; check_json.sh \"%\" \\| less -FR <CR>\n    au BufNew,BufRead *.ini        nmap ;l :w \\| !clear; validate_ini.py \"%\"; validate_ini2.py \"%\" <CR>\n    \" doesn't work on ansible inventory anyway\n    \"au FileType       ini          nmap ;l :w \\| !clear; validate_ini.py \"%\"; validate_ini2.py \"%\" <CR>\n    au BufNew,BufRead *.php        nmap ;l :w \\| !clear; php5 -l \"%\" <CR>\n    \" this acts as both a validation as well as a fast way of being able to edit the plist\n    \" trying to convert to json results in an error \"invalid object in plist for destination format\"\n    au BufNew,BufRead *.plist      nmap ;l :w \\| !clear; plutil -convert xml1 \"%\" && echo PList OK <CR>\n    au BufNew,BufRead *.properties nmap ;l :w \\| !clear; validate_properties.py \"%\" <CR>\n    au BufNew,BufRead *.ldif       nmap ;l :w \\| !clear; validate_ldap_ldif.py \"%\" <CR>\n    au BufNew,BufRead *.lua        nmap ;l :w \\| !clear; luacheck \"%\" <CR>\n    au BufNew,BufRead *.md         nmap ;l :w \\| !clear; mdl \"%\" \\| less -FR <CR>\n    \"au BufNew,BufRead *.sql        nmap ;l :w \\| !clear; TODO \"%\" \\| less -FR <CR>\n    au BufNew,BufRead *.scala      nmap ;l :w \\| !clear; scalastyle -c \"$bash_tools/scalastyle_config.xml\" \"%\" \\| less -FR <CR>\n    au BufNew,BufRead *.toml       nmap ;l :w \\| !clear; validate_toml.py \"%\" <CR>\n    au BufNew,BufRead *.xml        nmap ;l :w \\| !clear; validate_xml.py \"%\" <CR>\n    \" TODO: needs fix to allow multiple inline yaml docs in 1 file\n    \"au BufNew,BufRead *.yml,*.yaml nmap ;l :w \\| !clear; validate_yaml.py \"%\" <CR>\n    \"au BufNew,BufRead *.yml,*.yaml nmap ;l :w \\| !clear; js-yaml \"%\" >/dev/null && echo YAML OK<CR>\n    au BufNew,BufRead *.yml,*.yaml,autoinstall-user-data nmap ;l :w \\| !clear; yamllint \"%\" && echo YAML OK<CR>\n    au BufNew,BufRead *.tf,*.tf.json,*.tfvars,*.tfvars.json nmap ;l :w \\| call TerraformValidate()<CR>\n    au BufNew,BufRead *.hcl                                 nmap ;l :w \\| call TerragruntValidate()<CR>\n    au BufNew,BufRead *.pkr.hcl,*.pkr.json nmap ;l :w \\| !packer init \"%\" && packer validate \"%\" && packer fmt -diff \"%\" <CR>\n    au BufNew,BufRead *.pkr.hcl,*.pkr.json nmap ;f :w \\| !packer fmt -diff \"%\" <CR>\n\n    \" more specific matches like pom.xml need to come after less specific matches like *.xml as last statement wins\n    au BufNew,BufRead *pom.xml*      nmap ;l :w \\| !clear; mvn validate -f \"%\" \\| less -FR <CR>\n    \" check_makefiles.sh is in this repo which should be added to $PATH\n    au BufNew,BufRead *Makefile*     nmap ;l :w \\| !clear; check_makefiles.sh \"%\" \\| less -FR <CR>\n    au BufNew,BufRead *build.gradle* nmap ;l :w \\| !clear; gradle -b \"%\" -m clean build \\| less -FR <CR> | nmap ;r :!gradle -b \"%\" clean build <CR>\n    au BufNew,BufRead *build.sbt*    nmap ;l :w \\| !clear; cd \"%:p:h\" && echo q \\| sbt reload \"%\" \\| less -FR <CR>\n    au BufNew,BufRead *.travis.yml*  nmap ;l :w \\| !clear; travis lint \"%\" \\| less -FR <CR>\n    au BufNew,BufRead serverless.yml nmap ;l :w \\| !clear; cd \"%:p:h\" && serverless print<CR>\n    au BufNew,BufRead Dockerfile*   nmap ;l :w \\| !clear; hadolint \"%\" \\| less -FR <CR>\n    \"au BufNew,BufRead *docker-compose.y*ml   nmap ;r :w \\| !clear; docker-compose -f \"%\" up<CR>\n    au BufNew,BufRead *docker-compose*.y*ml nmap ;l :w \\| !clear; docker-compose -f \"%\" config \\| less -FR <CR>\n    au BufNew,BufRead Jenkinsfile*  nmap ;l :w \\| !clear; check_jenkinsfiles.sh \"%\" \\| less -FR <CR>\n    \" vagrant validate doesn't take an -f argument so it must be an exact match in order to validate the right thing\n    \" otherwise you will get an error or false positive\n    au BufNew,BufRead Vagrantfile    nmap ;l :w \\| !clear; cd \"%:p:h\" && vagrant validate<CR>\n    au BufNew,BufRead *.circleci/config.yml*  nmap ;l :w \\| !clear; check_circleci_config.sh \\| less -FR <CR>\n    au BufNew,BufRead *circleci_config.yml*   nmap ;l :w \\| !clear; check_circleci_config.sh \\| less -FR <CR>\n    au BufNew,BufRead .pylintrc      nmap ;l :w \\| !clear; pylint ./*.py<CR>\n\n    \"au BufNew,BufRead **/haproxy-configs/*.cfg   nmap ;r :w \\| !clear; haproxy -f \"%:p:h/10-global.cfg\" -f \"%:p:h/20-stats.cfg\" -f \"%\" <CR>\n    au BufNew,BufRead **/haproxy-configs/*.cfg   nmap ;r :w \\| !clear; \"%:p:h/run.sh\" \"%\" <CR>\n    au BufNew,BufRead **/haproxy-configs/*.cfg   nmap ;R :w \\| !clear; DEBUG=1 \"%:p:h/run.sh\" \"%\" <CR>\n\n    \"au BufNew,BufRead fastlane/Fastfile nmap ;r :w \\| !clear; cd \"%:p:h/..\" && fastlane <CR>\n\n    \" if a \"lint:\" header is found then run lint.sh - this allows for more complex file types like Kubernetes yaml\n    \" which can then be linted for yaml as well as k8s schema\n    \" XXX: this is overriding all linting regardless of this expansion - instead use a different hotkey L for fast vs full linting\n    \"if filereadable(expand(\"%:p\")) && match(readfile(expand(\"%:p\")),\"lint:\")\n    \"    au BufNew,BufRead *  nmap ;l :w \\| !clear; lint.sh \"%\" \\| less -FR <CR>\n    \"endif\n  augroup END\nendif\n\n\n\" ============================================================================ \"\n\"                               F u n c t i o n s\n\" ============================================================================ \"\n\n\" avoids this error when trying to run ;-v nmap to re-source this vimrc:\n\"\n\"   E127: Cannot redefine function SourceVimrc: It is in use\n\"\n\"function! SourceVimrc()\n\" This function won't reload as a result, must exit and restart vim\nif ! exists('*SourceVimrc')\n    function SourceVimrc()\n        :source ~/.vimrc\n        let vim_tags = system('grep vim: ' + expand('%') + \" | head -n1 | sed 's/^\\\"[[:space:]]*vim:/set /; s/:/ /g'\")\n        \" this breaks\n        \"echo &vim_tags\n        \"execute \"normal!\" . &vim_tags\n        \":! grep vim: expand(\"%\") | sed 's/\\#//'\n        :echo \"\\n\"\n        :echo 'Currently set options:'\n        :echo \"\\n\"\n        :set ts sts sw et filetype\n    endfunction\nendif\n\n\":! bash -c 'vim -c \"source %\" -c \"q\" && echo \"ViM basic lint validation passed\" || \"ViM basic lint validation failed\"'\n\"\":! if type -P vint &>/dev/null; then vint \"%\"; fi\nif !exists('g:__lintvimrc_defined')\n    let g:__lintvimrc_defined = 1\n    function! LintVimrc()\n      let l:vimrc_path = expand('~/.vimrc')\n\n      echo 'Sourcing ~/.vimrc file...'\n      try\n        execute 'source' l:vimrc_path\n        echohl InfoMsg\n        echo 'Basic Validation Passed: .vimrc'\n        echohl None\n      catch\n        echohl ErrorMsg\n        echo 'Basic Validate Failed while sourcing .vimrc'\n        echo v:exception\n        echo 'At: ' . v:throwpoint\n        echohl None\n        return\n      endtry\n\n      if executable('vint')\n        echo 'Running vint...'\n        let l:vint_output = system('vint ' . l:vimrc_path)\n        if v:shell_error\n          echohl ErrorMsg\n          echo l:vint_output\n          echo 'Vint Validation Failed: .vimrc'\n          echohl None\n        else\n          echohl InfoMsg\n          echo 'Vint Validation Passed: .vimrc'\n          echohl None\n        endif\n      else\n        echohl WarningMsg\n        echo 'Vint not found in PATH, skipping validation'\n        echohl None\n      endif\n    endfunction\nendif\n\nfunction! ToggleSyntax()\n    if exists('g:syntax_on')\n        syntax off\n    else\n        syntax enable\n    endif\nendfunction\n\nfunction! ToggleComments()\n    :let comment_char = '#'\n    :let comment_prefix = '^' . comment_char\n    echo comment_prefix\n    if getline('.') =~# comment_prefix\n        :s/^\\=:comment_char//\n    else\n        :s/^/\\=:comment_char/\n    endif\nendfunction\n\n\" setting this high keeps cursor in middle of screen\n\":set so=999\nfunction! ToggleScrollLock()\n    if &scrolloff > 0\n        :set scrolloff=0\n    else\n        :set scrolloff=999\n    endif\nendfunction\n\n\" simpler to call: set paste!\n\"function! TogglePaste()\n\"    if &paste > 0\n\"        :set nopaste\n\"    else\n\"        :set paste\n\"    endif\n\"endfunction\n\n\" changing this setting has no effect on vim gutter in real time\nfunction! ToggleGutter()\n    :let g:gitgutter_enabled = 0\n    \"if g:gitgutter_enabled > 0\n    \"if get(g:, 'gitgutter_enabled', 0) > 0\n    \"    :let g:gitgutter_enabled = 0\n    \"else\n    \"    :let g:gitgutter_enabled = 1\n    \"endif\nendfunction\n\nfunction! ToggleDebug()\n    if $DEBUG\n        echo 'DEBUG disabled'\n        let $DEBUG=''\n    else\n        echo 'DEBUG enabled'\n        let $DEBUG=1\n    endif\nendfunction\n\n\":command! Hr  :normal a# <ESC>76a=<ESC>a #<ESC>\n\":function! Hr()\n\"    if b:current_syntax eq \"sql\"\n\"        ::normal a-- <ESC>74a=<ESC>a --<ESC>\n\"    else\n\"        :normal a# <ESC>76a=<ESC>a #<ESC>\n\"    endif\n\":endfunction\nfunction! Hr()\n    let width = 80\n\n    let dash_filetypes = [\n                \\ 'sql', 'mysql', 'plsql',\n                \\ 'lua',\n                \\ 'haskell',\n                \\ 'ada'\n                \\ ]\n\n    let hash_filetypes = [\n                \\ 'sh', 'bash', 'zsh',\n                \\ 'python',\n                \\ 'ruby',\n                \\ 'perl',\n                \\ 'make',\n                \\ 'yaml', 'yml',\n                \\ 'toml',\n                \\ 'conf', 'cfg',\n                \\ 'dockerfile'\n                \\ ]\n\n    let ft = &filetype\n\n    if index(dash_filetypes, ft) >= 0\n        let start = '-- '\n        let end   = ' --'\n    elseif index(hash_filetypes, ft) >= 0\n        let start = '# '\n        let end   = ' #'\n    else\n        let start = '# '\n        let end   = ' #'\n    endif\n\n    let fill_len = width - strlen(start) - strlen(end)\n    if fill_len < 0\n        return\n    endif\n\n    let line = start . repeat('=', fill_len) . end\n\n    call append(line('.') - 1, line)\nendfunction\n\n\":function Br()\n\":call Hr()\n\":endfunction\n:command! Br :Hr\n\n\"function JHr()\n\"    s,^,// ========================================================================== //,\n\"endfunction\n\":command JHr :normal a// ========================================================================== //<ESC>lx\n:command! JHr :normal a// <ESC>74a=<ESC>a //<ESC>\n\n:command! Done :normal 37a=<ESC>a DONE <ESC>37a=<ESC>\n\n\" ============================================================================ \"\n\"                            G i t   F u n c t i o n s\n\" ============================================================================ \"\n\n\" works better than a straight nmap which sometimes fails to execute and re-sourcing .vimrc doesn't solve it\n\" without exiting vim - this is buggy behaviour that doesn't seem to happen when using functions instead\n\nfunction! GitUpdateCommit()\n    \":! bash -ic 'cd \"%:p:h\" && gitu \"%:t\" '\n    :! git_diff_commit.sh \"%\"\nendfunction\n\nfunction! GitAddCommit()\n     ! bash -ic 'add \"%\"'\nendfunction\n\nfunction! GitStatus()\n    :! bash -ic 'cd \"%:p:h\" && st'\nendfunction\n\nfunction! GitLogP()\n    :! bash -ic 'cd \"%:p:h\" && git log -p \"%:t\"'\nendfunction\n\nfunction! GitPull()\n    :! bash -ic 'cd \"%:p:h\" && pull'\nendfunction\n\nfunction! GitPush()\n    :! bash -ic 'cd \"%:p:h\" && push'\nendfunction\n\n\" ============================================================================ \"\n\nfunction! MarkdownIndex()\n    :! markdown_replace_index.sh \"%\"\nendfunction\n\n\" superceded by anonymize.py from DevOps Python tools repo, called via hotkey ;a declared above\n\":function RemoveIPs()\n\"    : %s/\\d\\+\\.\\d\\+\\.\\d\\+\\.\\d\\+/<IP_REMOVED>/gc\n\":endfunction\n\"\n\":function RemoveMacs()\n\"    : %s/\\w\\w:\\w\\w:\\w\\w:\\w\\w:\\w\\w:\\w\\w/<MAC_REMOVED>/gc\n\":endfunction\n\"\n\":function RemoveDomains()\n\"    : %s/company1/<DOMAIN_REMOVED>/gci\n\"    : %s/company2/<DOMAIN_REMOVED>/gci\n\":endfunction\n\nfunction! Anonymize()\n    \": call RemoveIPs()\n    \": call RemoveMacs()\n    \": call RemoveDomains()\n    \" Anonymizer is found in adjacent DevOps Python Tools repo which should be added to $PATH\n    \" there is also a faster version in DevOps Perl Tools repo\n    :%!anonymize.py --all\nendfunction\n\n\" StripQuotes()\nfunction! Sq()\n    :s/[\"']//g\nendfunction\n\nfunction! StripTrailingWhiteSpace()\n    :%s/[[:space:]]*$//\nendfunction\n\nfunction! WriteHelp()\n    :w\n    if &filetype ==# 'go'\n        :! go run \"%:p\" --help 2>&1 | less\n    elseif expand('%:t') ==# 'Makefile'\n        :call Make('help')\n    else\n        :! \"%:p\" --help 2>&1 | less\n    endif\nendfunction\n\nfunction! WriteRun()\n    :w\n    if &filetype ==# 'go'\n        \" TODO: consider switching this to go build and then run the binary as\n        \" this gets stdout only at the end so things like welcome.go don't get\n        \" the transition effects when run like this\n        :! eval go run \"%:p\" `$bash_tools/lib/args_extract.sh \"%:p\"` 2>&1 | less\n    \" doesn't work, probably due to no first class support so just get file extension\n    \"elseif &filetype ==# 'tf'\n    elseif expand('%:e') ==# 'tf'\n        \":call TerraformPlan()\n        :call TerraformApply()\n    elseif expand('%:t') =~# '\\.pkr\\.\\(hcl\\|json\\)'\n        :! packer init \"%:p\" && packer build \"%:p\"\n    elseif expand('%:t') ==# 'Makefile'\n        :call Make()\n    elseif expand('%:t') ==# 'Dockerfile'\n        \" \"%:p:h\" is dirname\n        if filereadable(join([expand('%:p:h'), 'Makefile'], '/'))\n            :call Make()\n        else\n            :! docker build '%:p:h'\n        endif\n    elseif expand('%:t') ==# 'Gemfile'\n        \" '%:p:h' is dirname\n        :! cd '%:p:h' && bundle install\n    \"elseif ! empty(matchstr(expand('%:t'), 'cloudbuild.*.yaml'))\n    elseif expand('%:t') =~# 'cloudbuild.*\\.ya\\?ml'\n        :call CloudBuild()\n    elseif expand('%:t') ==# 'kustomization.yaml'\n        :! bash -c 'cd \"%:p:h\" && kustomize build --enable-helm' 2>&1 | less\n    elseif expand('%:t') ==# '.envrc'\n        :! bash -c 'cd \"%:p:h\" && direnv allow .' 2>&1 | less\n    elseif executable('run.sh')\n        \" this only works for scripts\n        \":! eval \"%:p\" `$bash_tools/lib/args_extract.sh \"%:p\"`  2>&1 | less\n        \" but this now works for config files too which can have run headers\n        \" instead of args headers\n        :! \"run.sh\" \"%:p\" 2>&1 | less\n    else\n        echo 'unsupported file type and run.sh not found in PATH'\n    endif\nendfunction\n\nfunction! WriteRunVerbose()\n    :let $VERBOSE=1\n    :call WriteRun()\n    :let $VERBOSE=''\nendfunction\n\nfunction! WriteRunDebug()\n    :let $DEBUG=1\n    :call WriteRun()\n    :let $DEBUG=''\nendfunction\n\nfunction! WriteRunLine()\n    :w\n    if &filetype ==# 'go'\n        \" TODO: consider switching this to go build and then run the binary as\n        \" this gets stdout only at the end so things like welcome.go don't get\n        \" the transition effects when run like this\n        :.w ! sed 's/^[[:space:]]*\\#// ; s|\\$0|%:p| ; s|\\${0\\#\\#\\*\\/}|%:p|' | xargs go run 2>&1 | less\n    elseif expand('%:t') ==# 'Makefile' \" || expand('%:t') ==# 'Makefile.in'\n        let target = split(getline('.'), ':')[0]\n        call Make(target)\n    else\n        \" $0 => ./script.sh  ${0##*/} => ./script.sh\n        \" super crafy hack to get jq options injected via jq() function from interactive profile without doing\n        \" bash -i which causes prompt issues and 'no job control' message, and tee'ing to /dev/stderr instead of\n        \" using set -x which would exposing the inner workings and obscure the executed command\n        .w ! sed 's/^[[:space:]]*\\#// ; s|\\$0|%:p| ; s|\\${0\\#\\#\\*\\/}|%:p|' | bash -c '. $bash_tools/.bash.d/aliases.sh; . $bash_tools/.bash.d/functions.sh; eval \"$(cat | tee /dev/stderr)\"' 2>&1 | less\n    endif\n    :silent exec \"!echo; read -p 'Press enter to return to vim'\"\nendfunction\n\n\" variable number of args like *args in python\nfunction! Make(...)\n    \" '%:p:h' is dirname\n    \":! cd \"%:p:h\" && make join(map(a:000, 'shellescape(v:val)'), ' ')\n    \" this works and nicely pages but only outputs at the end, which is ok for help\n    \" but bad for actual make builds which are long and look like they hang\n    \":echo system('cd ' . expand('%:p:h') . ' && make ' . join(a:000, ''))\n    :exe '! cd \"%:p:h\" && make ' . join(a:000, '') . ' | more'\nendfunction\n\n\" Hashicorp Terraform\nfunction! TerraformValidate()\n    \" remove terraform plan copy-pasted removals for fast backporting\n    :%s/^[[:space:]]*[-~][[:space:]]//e\n    :%s/[[:space:]]->[[:space:]].*$//e\n    :!clear; bash -c 'if [ -d \"%:p:h\"/.terraform ]; then cd \"%:p:h\"; fi; { terraform fmt -diff; terraform validate; } | less -FR'\nendfunction\n\nfunction! TerragruntValidate()\n    :%s/[[:space:]]->[[:space:]].*$//e\n    :!clear; bash -c 'if [ -d \"%:p:h\"/.terraform ]; then cd \"%:p:h\"; fi; { terragrunt hclfmt --terragrunt-diff; terragrunt validate; } | less -FR'\nendfunction\n\nfunction! TerraformPlan()\n    \" '%:p:h' is dirname\n    :! bash -c 'if [ -d \"%:p:h\"/.terraform ]; then cd \"%:p:h\"; fi; terraform plan'\nendfunction\n\nfunction! TerraformApply()\n    :! bash -c 'if [ -d \"%:p:h\"/.terraform ]; then cd \"%:p:h\"; fi; terraform apply'\nendfunction\n\n\" GCP Google Cloud Build\nfunction! CloudBuild()\n    \" '%:p:h' is dirname\n    let cloudbuild_yaml = expand('%:t')\n    :exe '! cd \"%:p:h\" && gcloud builds submit --config ' . cloudbuild_yaml . ' .'\nendfunction\n\n\n\" ============================================================================ \"\n\"                   L o c a l   C o n f i g   S o u r c i n g\n\" ============================================================================ \"\n\n\" either works, requires expand()\n\"let MYLOCALVIMRC = '~/.vimrc.local'\n\"let MYLOCALVIMRC = '$HOME/.vimrc.local'\n\n\" source a config file only if it exists\nfunction! SourceIfExists(file)\n  if filereadable(expand(a:file))\n    exe 'source' a:file\n  endif\nendfunction\n\ncall SourceIfExists('~/.vimrc.local')\ncall SourceIfExists('~/.vim/colors.vim')\n\nif has('gui_running')\n  call SourceIfExists('~/.gvimrc.local')\nendif\n"
  },
  {
    "path": "configs/.wakatime.cfg",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2026-01-08 05:10:45 -0500 (Thu, 08 Jan 2026)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         W a k a T i m e   C o n f i g\n# ============================================================================ #\n\n[settings]\nhide_project = false\noffline = true\nverbose = false\n"
  },
  {
    "path": "configs/.yamllint.yaml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-01-17 17:18:24 +0000 (Mon, 17 Jan 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         Y a m l L i n t   C o n f i g\n# ============================================================================ #\n\n# Master config:\n#\n#   https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/.config/yamllint/config\n\n---\nextends: default\n\nrules:\n  line-length: disable\n  comments: disable\n  comments-indentation: disable\n  colons:\n    ignore: |\n      # need the paths to work as /test/test.yaml as well as /pylib/test/test.yaml\n      test.yaml\n  indentation:\n    ignore: |\n      calico.yaml\n  empty-lines:\n    ignore: |\n      calico.yaml\n"
  },
  {
    "path": "configs/README.md",
    "content": "# Configs\n\nConfig files for various common open source software I use on\n[Linux](https://github.com/HariSekhon/Knowledge-Base/blob/main/linux.md) and\n[Mac](https://github.com/HariSekhon/Knowledge-Base/blob/main/mac.md).\n\nI symlink these to my home directory using this command at the root of this repo:\n\n```shell\nmake link\n```\n\nFiles and directories to symlink are defined in:\n\n[../setup/files.txt](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/setup/files.txt)\n"
  },
  {
    "path": "configs/clamd.conf",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-06-04 02:00:16 +0200 (Wed, 04 Jun 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            C l a m D   C o n f i g\n# ============================================================================ #\n\n# Generated on macOS from command:\n#\n#   clamconf -g clamd.conf >> clamd.conf\n\n##\n## clamd.conf - automatically generated by clamconf 1.4.1\n##\n\n# When AlertExceedsMax is set, files exceeding the MaxFileSize, MaxScanSize, or\n# MaxRecursion limit will be flagged with the virus name starting with\n# \"Heuristics.Limits.Exceeded\".\n# Default: no\n#AlertExceedsMax\n\n# Number of entries the cache can store.\n# Default: 65536\n#CacheSize 65536\n\n# Enable prelude\n# Default: no\n#PreludeEnable\n\n# Name of the analyzer as seen in prewikka\n# Default: disabled\n#PreludeAnalyzerName\n\n# Save all reports to a log file.\n# Default: disabled\n#LogFile /tmp/clamav.log\n\n# By default the log file is locked for writing and only a single\n# daemon process can write to it. This option disables the lock.\n# Default: no\n#LogFileUnlock yes\n\n# Maximum size of the log file.\n# Value of 0 disables the limit.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 1048576\nLogFileMaxSize 100M\n\n# Log time with each message.\n# Default: no\nLogTime yes\n\n# Log all clean files.\n# Useful in debugging but drastically increases the log size.\n# Default: no\n#LogClean yes\n\n# Use the system logger (can work together with LogFile).\n# Default: no\n#LogSyslog yes\n\n# Type of syslog messages.\n# Please refer to 'man syslog' for the facility names.\n# Default: LOG_LOCAL6\n#LogFacility LOG_MAIL\n\n# Enable verbose logging.\n# Default: no\n#LogVerbose yes\n\n# Rotate log file. Requires LogFileMaxSize option set prior to this option.\n# Default: no\n#LogRotate yes\n\n# Log additional information about the infected file, such as its\n# size and hash, together with the virus name.\n# Default: no\n#ExtendedDetectionInfo yes\n\n# Save the process ID to a file.\n# Default: disabled\n#PidFile /run/clamav/clam.pid\n\n# This option allows you to change the default temporary directory.\n# Default: disabled\n#TemporaryDirectory /tmp\n\n# This option allows you to change the default database directory.\n# If you enable it, please make sure it points to the same directory in\n# both clamd and freshclam.\n# Default: /opt/homebrew/var/lib/clamav\n#DatabaseDirectory /var/lib/clamav\n\n# Only load the official signatures published by the ClamAV project.\n# Default: no\n#OfficialDatabaseOnly no\n\n# Return with a nonzero error code if the virus database is older than the specified number of days.\n# Default: disabled\n#FailIfCvdOlderThan -1\n\n# Path to a local socket file the daemon will listen on.\n# Default: disabled\n#LocalSocket /run/clamav/clamd.sock\nLocalSocket /tmp/clamav/clamd.sock\n\n# Sets the group ownership on the unix socket.\n# Default: disabled\n#LocalSocketGroup virusgroup\n\n# Sets the permissions on the unix socket to the specified mode.\n# Default: disabled\nLocalSocketMode 660\n\n# Remove a stale socket after unclean shutdown\n# Default: yes\n#FixStaleSocket yes\n\n# A TCP port number the daemon will listen on.\n# Default: disabled\n#TCPSocket 3310\n\n# By default clamd binds to INADDR_ANY.\n# This option allows you to restrict the TCP address and provide\n# some degree of protection from the outside world.\n# Default: disabled\n#TCPAddr localhost\n\n# Maximum length the queue of pending connections may grow to.\n# Default: 200\n#MaxConnectionQueueLength 30\n\n# Close the STREAM session when the data size limit is exceeded.\n# The value should match your MTA's limit for the maximum attachment size.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 104857600\n#StreamMaxLength 100M\n\n# The STREAM command uses an FTP-like protocol.\n# This option sets the lower boundary for the port range.\n# Default: 1024\n#StreamMinPort 1024\n\n# This option sets the upper boundary for the port range.\n# Default: 2048\n#StreamMaxPort 2048\n\n# Maximum number of threads running at the same time.\n# Default: 10\n#MaxThreads 20\n\n# This option specifies the time (in seconds) after which clamd should\n# timeout if a client doesn't provide any data.\n# Default: 120\n#ReadTimeout 120\n\n# This option specifies the time (in seconds) after which clamd should\n# timeout if a client doesn't provide any initial command after connecting.\n# Default: 30\n#CommandReadTimeout 30\n\n# This option specifies how long to wait (in milliseconds) if the send buffer\n# is full. Keep this value low to prevent clamd hanging.\n# Default: 500\n#SendBufTimeout 200\n\n# Maximum number of queued items (including those being processed by MaxThreads\n# threads). It is recommended to have this value at least twice MaxThreads\n# if possible.\n# WARNING: you shouldn't increase this too much to avoid running out of file\n#  descriptors, the following condition should hold:\n#  MaxThreads*MaxRecursion + MaxQueue - MaxThreads  + 6 < RLIMIT_NOFILE\n#  (usual max for RLIMIT_NOFILE is 1024)\n#\n# Default: 100\n#MaxQueue 200\n\n# This option specifies how long (in seconds) the process should wait\n# for a new job.\n# Default: 30\n#IdleTimeout 60\n\n# Don't scan files/directories whose names match the provided\n# regular expression. This option can be specified multiple times.\n# Default: disabled\n#ExcludePath ^/proc/\n#ExcludePath ^/sys/\n\n# Maximum depth the directories are scanned at.\n# Default: 15\n#MaxDirectoryRecursion 15\n\n# Follow directory symlinks.\n# Default: no\n#FollowDirectorySymlinks no\n\n# Follow symlinks to regular files.\n# Default: no\n#FollowFileSymlinks no\n\n# Scan files and directories on other filesystems.\n# Default: yes\n#CrossFilesystems yes\n\n# This option specifies the time intervals (in seconds) in which clamd\n# should perform a database check.\n# Default: 600\n#SelfCheck 600\n\n# Enable non-blocking (multi-threaded/concurrent) database reloads. This feature\n# will temporarily load a second scanning engine while scanning continues using\n# the first engine. Once loaded, the new engine takes over. The old engine is\n# removed as soon as all scans using the old engine have completed. This feature\n# requires more RAM, so this option is provided in case users are willing to\n# block scans during reload in exchange for lower RAM requirements.\n# Default: yes\n#ConcurrentDatabaseReload yes\n\n# This option allows you to disable clamd's caching feature.\n# Default: no\n#DisableCache no\n\n# Execute a command when virus is found.\n# Use the following environment variables to identify the file and virus names:\n# - $CLAM_VIRUSEVENT_FILENAME\n# - $CLAM_VIRUSEVENT_VIRUSNAME\n# In the command string, '%v' will also be replaced with the virus name.\n# Note: The '%f' filename format character has been disabled and will no longer\n# be replaced with the file name, due to command injection security concerns.\n# Use the 'CLAM_VIRUSEVENT_FILENAME' environment variable instead.\n# For the same reason, you should NOT use the environment variables in the\n# command directly, but should use it carefully from your executed script.\n# Default: disabled\n#VirusEvent /opt/send_virus_alert_sms.sh\n\n# Stop the daemon when libclamav reports an out of memory condition.\n# Default: no\n#ExitOnOOM yes\n\n# Permit use of the ALLMATCHSCAN command.\n# Default: yes\n#AllowAllMatchScan yes\n\n# Don't fork into background.\n# Default: no\n#Foreground no\n\n# Enable debug messages in libclamav.\n# Default: no\n#Debug no\n\n# Don't remove temporary files (for debugging purposes).\n# Default: no\n#LeaveTemporaryFiles no\n\n# Record metadata about the file being scanned.\n# Scan metadata is useful for file analysis purposes and for debugging scan behavior.\n# The JSON metadata will be printed after the scan is complete if Debug is enabled.\n# A metadata.json file will be written to the scan temp directory if LeaveTemporaryFiles is enabled.\n# Default: no\n#GenerateMetadataJson no\n\n# Run the daemon as a specified user (the process must be started by root).\n# Default: disabled\n#User clamav\n\n# With this option enabled ClamAV will load bytecode from the database. It is highly recommended you keep this option on, otherwise you'll miss detections for many new viruses.\n# Default: yes\n#Bytecode yes\n\n# Set bytecode security level.\n# Possible values:\n# \tTrustSigned - trust bytecode loaded from signed .c[lv]d files,\n# \t\t insert runtime safety checks for bytecode loaded from other sources\n# \tParanoid - don't trust any bytecode, insert runtime checks for all\n# Recommended: TrustSigned, because bytecode in .cvd files already has these checks.\n# Default: TrustSigned\n#BytecodeSecurity TrustSigned\n\n# Set bytecode timeout in milliseconds.\n# Default: 10000\n#BytecodeTimeout 10000\n\n# Allow loading bytecode from outside digitally signed .c[lv]d files.\n# Default: no\n#BytecodeUnsigned no\n\n# Set bytecode execution mode.\n# Possible values:\n# \tAuto - automatically choose JIT if possible, fallback to interpreter\n# ForceJIT - always choose JIT, fail if not possible\n# ForceInterpreter - always choose interpreter\n# Test - run with both JIT and interpreter and compare results. Make all failures fatal.\n# Default: Auto\n#BytecodeMode Auto\n\n# Detect Potentially Unwanted Applications.\n# Default: no\n#DetectPUA yes\n\n# Exclude a specific PUA category. This directive can be used multiple times.\n# See https://docs.clamav.net/faq/faq-pua.html for the complete list of PUA\n# categories.\n# Default: disabled\n#ExcludePUA NetTool\n#ExcludePUA PWTool\n\n# Only include a specific PUA category. This directive can be used multiple\n# times.\n# Default: disabled\n#IncludePUA Spy\n#IncludePUA Scanner\n#IncludePUA RAT\n\n# PE stands for Portable Executable - it's an executable file format used\n# in all 32- and 64-bit versions of Windows operating systems. This option\n# allows ClamAV to perform a deeper analysis of executable files and it's also\n# required for decompression of popular executable packers such as UPX or FSG.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanPE yes\n\n# Executable and Linking Format is a standard format for UN*X executables.\n# This option allows you to control the scanning of ELF files.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanELF yes\n\n# Enable the built in email scanner.\n# If you turn off this option, the original files will still be scanned, but\n# without parsing individual messages/attachments.\n# Default: yes\n#ScanMail yes\n\n# Scan RFC1341 messages split over many emails. You will need to\n# periodically clean up $TemporaryDirectory/clamav-partial directory.\n# WARNING: This option may open your system to a DoS attack. Please don't use\n# this feature on highly loaded servers.\n# Default: no\n#ScanPartialMessages no\n\n# With this option enabled ClamAV will try to detect phishing attempts by using\n# signatures.\n# Default: yes\n#PhishingSignatures yes\n\n# Scan URLs found in mails for phishing attempts using heuristics.\n# Default: yes\n#PhishingScanURLs yes\n\n# In some cases (eg. complex malware, exploits in graphic files, and others),\n# ClamAV uses special algorithms to provide accurate detection. This option\n# controls the algorithmic detection.\n# Default: yes\n#HeuristicAlerts yes\n\n# Allow heuristic match to take precedence.\n# When enabled, if a heuristic scan (such as phishingScan) detects\n# a possible virus/phish it will stop scan immediately. Recommended, saves CPU\n# scan-time.\n# When disabled, virus/phish detected by heuristic scans will be reported only\n# at the end of a scan. If an archive contains both a heuristically detected\n# virus/phish, and a real malware, the real malware will be reported.\n# Keep this disabled if you intend to handle \"Heuristics.*\" viruses\n# differently from \"real\" malware.\n# If a non-heuristically-detected virus (signature-based) is found first,\n# the scan is interrupted immediately, regardless of this config option.\n# Default: no\n#HeuristicScanPrecedence yes\n\n# Enable the Data Loss Prevention module.\n# Default: no\n#StructuredDataDetection no\n\n# This option sets the lowest number of Credit Card numbers found in a file\n# to generate a detect.\n# Default: 3\n#StructuredMinCreditCardCount 5\n\n# This option sets the lowest number of Social Security Numbers found\n# in a file to generate a detect.\n# Default: 3\n#StructuredMinSSNCount 5\n\n# With this option enabled the DLP module will search for valid\n# SSNs formatted as xxx-yy-zzzz.\n# Default: yes\n#StructuredSSNFormatNormal yes\n\n# With this option enabled the DLP module will search for valid\n# SSNs formatted as xxxyyzzzz\n# Default: no\n#StructuredSSNFormatStripped no\n\n# Perform HTML/JavaScript/ScriptEncoder normalisation and decryption.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanHTML yes\n\n# This option enables scanning of OLE2 files, such as Microsoft Office\n# documents and .msi files.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanOLE2 yes\n\n# With this option enabled clamav will try to detect broken executables\n# (PE, ELF, & Mach-O) and alert on them with a Broken.Executable heuristic signature.\n# Default: no\n#AlertBrokenExecutables yes\n\n# With this option enabled clamav will try to detect broken media files\n# (JPEG, TIFF, PNG, GIF) and alert on them with a Broken.Media heuristic signature.\n# Default: no\n#AlertBrokenMedia yes\n\n# Alert on encrypted archives and documents (encrypted .zip, .7zip, .rar, .pdf).\n# Default: no\n#AlertEncrypted no\n\n# With this option enabled the DLP module will search for valid Credit Card\n# numbers only. Debit and Private Label cards will not be searched.\n# Default: no\n#StructuredCCOnly no\n\n# Alert on encrypted archives (encrypted .zip, .7zip, .rar).\n# Default: no\n#AlertEncryptedArchive no\n\n# Alert on encrypted documents (encrypted .pdf).\n# Default: no\n#AlertEncryptedDoc no\n\n# With this option enabled OLE2 files with VBA macros, which were not\n# detected by signatures will be marked as \"Heuristics.OLE2.ContainsMacros\".\n# Default: no\n#AlertOLE2Macros no\n\n# Alert on SSL mismatches in URLs, even if they're not in the database.\n# This feature can lead to false positives.\n# Default: no\n#AlertPhishingSSLMismatch\n\n# Alert on cloaked URLs, even if they're not in the database.\n# This feature can lead to false positives.\n# Default: no\n#AlertPhishingCloak no\n\n# Alert on raw DMG image files containing partition intersections.\n# Default: no\n#AlertPartitionIntersection yes\n\n# This option enables scanning within PDF files.\n# If you turn off this option, the original files will still be scanned, but\n# without decoding and additional processing.\n# Default: yes\n#ScanPDF yes\n\n# This option enables scanning within SWF files.\n# If you turn off this option, the original files will still be scanned, but\n# without decoding and additional processing.\n# Default: yes\n#ScanSWF yes\n\n# This option enables scanning xml-based document files supported by libclamav.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanXMLDOCS yes\n\n# This option enables scanning HWP3 files.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanHWP3 yes\n\n# This option enables scanning OneNote files.\n# If you turn off this option, the original files will still be scanned, but\n# without additional processing.\n# Default: yes\n#ScanOneNote yes\n\n# Scan within archives and compressed files.\n# If you turn off this option, the original files will still be scanned, but\n# without unpacking and additional processing.\n# Default: yes\n#ScanArchive yes\n\n# This option enables scanning of image (graphics).\n# If you turn off this option, the original files will still be scanned, but without additional processing.\n# Default: yes\n#ScanImage yes\n\n# This option enables detection by calculating a fuzzy hash of image (graphics)\n# files\n# Signatures using image fuzzy hashes typically match files and documents by\n# identifying images embedded or attached to those files.\n# If you turn off this option, then some files may no longer be detected.\n# Default: yes\n#ScanImageFuzzyHash yes\n\n# This option causes memory or nested map scans to dump the content to disk.\n# If you turn on this option, more data is written to disk and is available\n# when the leave-temps option is enabled at the cost of more disk writes.\n# Default: no\n#ForceToDisk no\n\n# This option sets the maximum amount of time a scan may take to complete.\n# The value of 0 disables the limit.\n# WARNING: disabling this limit or setting it too high may result allow scanning\n# of certain files to lock up the scanning process/threads resulting in a Denial of Service.\n# The value is in milliseconds.\n# Default: 0\n#MaxScanTime 120000\n\n# This option sets the maximum amount of data to be scanned for each input file.\n# Archives and other containers are recursively extracted and scanned up to this\n# value.\n# The value of 0 disables the limit.\n# WARNING: disabling this limit or setting it too high may result in severe\n# damage.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 419430400\n#MaxScanSize 400M\n\n# Files/messages larger than this limit won't be scanned. Affects the input\n# file itself as well as files contained inside it (when the input file is\n# an archive, a document or some other kind of container).\n# The value of 0 disables the limit.\n# WARNING: disabling this limit or setting it too high may result in severe\n# damage to the system.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 104857600\n#MaxFileSize 100M\n\n# Nested archives are scanned recursively, e.g. if a Zip archive contains a RAR\n# file, all files within it will also be scanned. This option specifies how\n# deeply the process should be continued.\n# The value of 0 disables the limit.\n# WARNING: disabling this limit or setting it too high may result in severe\n# damage to the system.\n# Default: 17\n#MaxRecursion 17\n\n# Number of files to be scanned within an archive, a document, or any other\n# container file.\n# The value of 0 disables the limit.\n# WARNING: disabling this limit or setting it too high may result in severe\n# damage to the system.\n# Default: 10000\n#MaxFiles 10000\n\n# This option sets the maximum size of a file to check for embedded PE.\n# Files larger than this value will skip the additional analysis step.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 41943040\n#MaxEmbeddedPE 40M\n\n# This option sets the maximum size of a HTML file to normalize.\n# HTML files larger than this value will not be normalized or scanned.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 41943040\n#MaxHTMLNormalize 40M\n\n# This option sets the maximum size of a normalized HTML file to scan.\n# HTML files larger than this value after normalization will not be scanned.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 8388608\n#MaxHTMLNoTags 8M\n\n# This option sets the maximum size of a script file to normalize.\n# Script content larger than this value will not be normalized or scanned.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 20971520\n#MaxScriptNormalize 20M\n\n# This option sets the maximum size of a ZIP file to reanalyze type recognition.\n# ZIP files larger than this value will skip the step to potentially reanalyze as PE.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 1048576\n#MaxZipTypeRcg 1M\n\n# This option sets the maximum number of partitions of a raw disk image to be scanned.\n# Raw disk images with more partitions than this value will have up to the value number partitions scanned.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# Default: 50\n#MaxPartitions 128\n\n# This option sets the maximum number of icons within a PE to be scanned.\n# PE files with more icons than this value will have up to the value number icons scanned.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# Default: 100\n#MaxIconsPE 100\n\n# This option sets the maximum recursive calls to HWP3 parsing function.\n# HWP3 files using more than this limit will be terminated and alert the user.\n# Scans will be unable to scan any HWP3 attachments if the recursive limit is reached.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may result in severe damage or impact performance.\n# Default: 16\n#MaxRecHWP3 16\n\n# This option sets the maximum calls to the PCRE match function during an instance of regex matching.\n# Instances using more than this limit will be terminated and alert the user but the scan will continue.\n# For more information on match_limit, see the PCRE documentation.\n# Negative values are not allowed.\n# WARNING: setting this limit too high may severely impact performance.\n# Default: 100000\n#PCREMatchLimit 100000\n\n# This option sets the maximum recursive calls to the PCRE match function during an instance of regex matching.\n# Instances using more than this limit will be terminated and alert the user but the scan will continue.\n# For more information on match_limit_recursion, see the PCRE documentation.\n# Negative values are not allowed and values > PCREMatchLimit are superfluous.\n# WARNING: setting this limit too high may severely impact performance.\n# Default: 2000\n#PCRERecMatchLimit 5000\n\n# This option sets the maximum filesize for which PCRE subsigs will be executed.\n# Files exceeding this limit will not have PCRE subsigs executed unless a subsig is encompassed to a smaller buffer.\n# Negative values are not allowed.\n# Setting this value to zero disables the limit.\n# WARNING: setting this limit too high or disabling it may severely impact performance.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 104857600\n#PCREMaxFileSize 100M\n\n# This option specifies a directory or mount point which should be scanned on access. The mount point specified, or the mount point containing the specified directory will be watched, but only notifications will occur. If any directories are specified, this option will preempt the DDD system. It can also be used multiple times.\n# Default: disabled\n#OnAccessMountPath /\n#OnAccessMountPath /home/user\n\n# This option specifies a directory (including all files and directories\n# inside it), which should be scanned on access. This option can\n# be used multiple times.\n# Default: disabled\n#OnAccessIncludePath /home\n#OnAccessIncludePath /students\n\n# This option allows excluding directories from on-access scanning. It can\n# be used multiple times. Only works with DDD system.\n# Default: disabled\n#OnAccessExcludePath /home/bofh\n#OnAccessExcludePath /root\n\n# Use this option to exclude the root UID (0) and allow any processes run under root to access all watched files without triggering scans.\n# Default: disabled\n#OnAccessExcludeRootUID no\n\n# With this option you can exclude specific UIDs. Processes with these UIDs\n# will be able to access all files.\n# This option can be used multiple times (one per line). Using a value of 0 on any line will disable this option entirely. To exclude the root UID please enable the OnAccessExcludeRootUID option.\n# Default: disabled\n#OnAccessExcludeUID 0\n\n# This option allows exclusions via user names when using the on-access scanning client. It can\n# be used multiple times.\n# Default: disabled\n#OnAccessExcludeUname clamuser\n\n# Files larger than this value will not be scanned in on access.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 5242880\n#OnAccessMaxFileSize 5M\n\n# This option toggles the dynamic directory determination system for on-access scanning (Linux only).\n# Default: no\n#OnAccessDisableDDD no\n\n# This option changes fanotify behavior to prevent access attempts on malicious files instead of simply notifying the user (On Access scan only).\n# Default: no\n#OnAccessPrevention yes\n\n# Enables extra scanning and notification after catching certain inotify events. Only works with the DDD system enabled.\n# Default: no\n#OnAccessExtraScanning yes\n\n# Max amount of time (in milliseconds) that the OnAccess client should spend for every connect, send, and receive attempt when communicating with clamd via curl (5s default)\n# Default: 5000\n#OnAccessCurlTimeout 10000L\n\n# Max number of scanning threads to allocate to the OnAccess thread pool at startup--these threads are the ones responsible for creating a connection with the daemon and kicking off scanning after an event has been processed. To prevent clamonacc from consuming all clamd's resources keep this lower than clamd's max threads. Default is 5\n# Default: 5\n#OnAccessMaxThreads 10\n\n# Number of times the OnAccess client will retry a failed scan due to connection problems (or other issues). Defaults to no retries.\n# Default: 0\n#OnAccessRetryAttempts 3\n\n# When using prevention, if this option is turned on, any errors that occur during scanning will result in the event attempt being denied. This could potentially lead to unwanted system behaviour with certain configurations, so the client defaults to off and allowing access events in case of error.\n# Default: no\n#OnAccessDenyOnError yes\n\n# Disable authenticode certificate chain verification in PE files.\n# Default: no\n#DisableCertCheck no\n\n# Deprecated option to enable heuristic alerts (e.g. \"Heuristics.<sig name>\")\n# Default: yes\n#AlgorithmicDetection no\n\n# Default: no\n#BlockMax\n\n# Deprecated option to alert on SSL mismatches in URLs, even if they're not in the database.\n# This feature can lead to false positives.\n# Default: no\n#PhishingAlwaysBlockSSLMismatch no\n\n# Deprecated option to alert on cloaked URLs, even if they're not in the database.\n# This feature can lead to false positives.\n# Default: no\n#PhishingAlwaysBlockCloak no\n\n# Deprecated option to alert on raw DMG image files containing partition intersections.\n# Default: no\n#PartitionIntersection no\n\n# With this option enabled OLE2 files with VBA macros, which were not\n# detected by signatures will be marked as \"Heuristics.OLE2.ContainsMacros\".\n# Default: no\n#OLE2BlockMacros no\n\n# Deprecated option to alert on encrypted archives and documents (encrypted .zip, .7zip, .rar, .pdf).\n# Default: no\n#ArchiveBlockEncrypted no\n"
  },
  {
    "path": "configs/freshclam.conf",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2025-06-04 01:54:39 +0200 (Wed, 04 Jun 2025)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        F r e s h C l a m   C o n f i g\n# ============================================================================ #\n\n# Generated on macOS from command:\n#\n#   clamconf -g freshclam.conf >> freshclam.conf\n\n##\n## freshclam.conf - automatically generated by clamconf 1.4.1\n##\n\n# Maximum size of the log file.\n# Value of 0 disables the limit.\n# You may use 'G' or 'g' for gigabytes (1G = 1g = 1,073,741,824 bytes)\n# 'M' or 'm' for megabytes (1M = 1m = 1048576 bytes)\n# and 'K' or 'k' for kilobytes (1K = 1k = 1024 bytes). To specify the size\n# in bytes just don't use modifiers.\n# Default: 1048576\nLogFileMaxSize 100M\n\n# Log time with each message.\n# Default: no\nLogTime yes\n\n# Use the system logger (can work together with LogFile).\n# Default: no\nLogSyslog yes\n\n# Type of syslog messages.\n# Please refer to 'man syslog' for the facility names.\n# Default: LOG_LOCAL6\n#LogFacility LOG_MAIL\n\n# Enable verbose logging.\n# Default: no\n#LogVerbose yes\n\n# Rotate log file. Requires LogFileMaxSize option set prior to this option.\n# Default: no\n#LogRotate yes\n\n# Save the process ID to a file.\n# Default: disabled\n#PidFile /run/clamav/clam.pid\n\n# This option allows you to change the default database directory.\n# If you enable it, please make sure it points to the same directory in\n# both clamd and freshclam.\n# Default: /opt/homebrew/var/lib/clamav\n#DatabaseDirectory /var/lib/clamav\n\n# Don't fork into background.\n# Default: no\n#Foreground no\n\n# Enable debug messages in libclamav.\n# Default: no\n#Debug no\n\n# Save all reports to a log file.\n# Default: disabled\n#UpdateLogFile /var/log/freshclam.log\n\n# When started by root freshclam will drop privileges and switch to the user\n# defined in this option.\n# Default: clamav\n#DatabaseOwner clamav\n\n# This option defined how many times daily freshclam should check for\n# a database update.\n# Default: 12\n#Checks 24\n\n# Use DNS to verify the virus database version. FreshClam uses DNS TXT records\n# to verify the versions of the database and software itself. With this\n# directive you can change the database verification domain.\n# WARNING: Please don't change it unless you're configuring freshclam to use\n# your own database verification domain.\n# Default: current.cvd.clamav.net\n#DNSDatabaseInfo current.cvd.clamav.net\n\n# DatabaseMirror specifies to which mirror(s) freshclam should connect.\n# You should have at least one entry: database.clamav.net.\n# Default: disabled\nDatabaseMirror database.clamav.net\n\n# This option allows you to easily point freshclam to private mirrors.\n# If PrivateMirror is set, freshclam does not attempt to use DNS\n# to determine whether its databases are out-of-date, instead it will\n# use the If-Modified-Since request or directly check the headers of the\n# remote database files. For each database, freshclam first attempts\n# to download the CLD file. If that fails, it tries to download the\n# CVD file. This option overrides DatabaseMirror, DNSDatabaseInfo\n# and Scripted Updates. It can be used multiple times to provide\n# fall-back mirrors.\n# Default: disabled\n#PrivateMirror mirror1.mynetwork.com\n#PrivateMirror mirror2.mynetwork.com\n\n# This option defines how many attempts freshclam should make before giving up.\n# Default: 3\nMaxAttempts 5\n\n# With this option you can control scripted updates. It's highly recommended to keep them enabled.\n# Default: yes\n#ScriptedUpdates yes\n\n# With this option enabled, freshclam will attempt to load new\n# databases into memory to make sure they are properly handled\n# by libclamav before replacing the old ones. Tip: This feature uses a lot of RAM. If your system has limited RAM and you are actively running ClamD or ClamScan during the update, then you may need to set `TestDatabases no`.\n# Default: yes\n#TestDatabases yes\n\n# By default freshclam will keep the local databases (.cld) uncompressed to\n# make their handling faster. With this option you can enable the compression.\n# The change will take effect with the next database update.\n# Default: no\n#CompressLocalDatabase\n\n# Include an optional signature databases (opt-in). This option can be used multiple times.\n# Default: disabled\n#ExtraDatabase dbname1\n#ExtraDatabase dbname2\n\n# Exclude a standard signature database (opt-out). This option can be used multiple times.\n# Default: disabled\n#ExcludeDatabase dbname1\n#ExcludeDatabase dbname2\n\n# With this option you can provide custom sources (http:// or file://) for database files.\n# This option can be used multiple times.\n# Default: disabled\n#DatabaseCustomURL http://myserver.com/mysigs.ndb\n#DatabaseCustomURL file:///mnt/nfs/local.hdb\n\n# If you're behind a proxy, please enter its address here.\n# Default: disabled\n#HTTPProxyServer your-proxy\n\n# HTTP proxy's port\n# Default: disabled\n#HTTPProxyPort 8080\n\n# A user name for the HTTP proxy authentication.\n# Default: disabled\n#HTTPProxyUsername username\n\n# A password for the HTTP proxy authentication.\n# Default: disabled\n#HTTPProxyPassword pass\n\n# If your servers are behind a firewall/proxy which does a User-Agent\n# filtering you can use this option to force the use of a different\n# User-Agent header.\n# Default: disabled\n#HTTPUserAgent default\n\n# Send the RELOAD command to clamd after a successful update.\n# Default: /opt/homebrew/etc/clamav/clamd.conf\nNotifyClamd yes\n\n# Run a command after a successful database update. Use EXIT_1 to return 1 after successful database update.\n# Default: disabled\n#OnUpdateExecute command\n\n# Run a command when a database update error occurs.\n# Default: disabled\n#OnErrorExecute command\n\n# Run a command when freshclam reports an outdated version.\n# In the command string %v will be replaced with the new version number.\n# Default: disabled\n#OnOutdatedExecute command\n\n# With this option you can provide a client address for the database downloading.\n# Useful for multi-homed systems.\n# Default: disabled\n#LocalIPAddress aaa.bbb.ccc.ddd\n\n# Timeout in seconds when connecting to database server.\n# Default: 30\n#ConnectTimeout 30\n\n# Timeout in seconds when reading from database server. 0 means no timeout.\n# Default: 60\n#ReceiveTimeout 60\n\n# This option enables downloading of bytecode.cvd, which includes additional\n# detection mechanisms and improvements to the ClamAV engine.\n# Default: yes\n#Bytecode yes\n"
  },
  {
    "path": "data/avro_tools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-16 14:09:20 +0200 (Mon, 16 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns Avro Tools jar, downloading it if not already present\n\nUses ../install/download_avro_tools.sh to download the latest version if no version is present\n\nYou can call that script directory if you want to update the version\n\n--help prints this message. For Avro Tools help use no args\n\nRequires Java to be installed to run the avro-tools-<version>.jar\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<avro_tools_args>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\navro_tools_jar=\"$(find \"$srcdir\" -maxdepth 1 -name 'avro-tools-*.jar' | sort -Vr | head -n 1)\"\n\nif [ -z \"$avro_tools_jar\" ] ||\n   # incomplete download, call download again to resume it\n   ! jar tf \"$avro_tools_jar\" &>/dev/null; then\n    pushd \"$srcdir\" 2>/dev/null || die \"Failed to pushd to '$srcdir'\"\n    \"$srcdir/../install/download_avro_tools.sh\"\n    popd &>/dev/null || die \"Failed to return to original dir\"\n    avro_tools_jar=\"$(find \"$srcdir\" -maxdepth 1 -name 'avro-tools-*.jar' | sort -Vr | head -n 1)\"\n    echo\nfi\n\njava -jar \"$avro_tools_jar\" \"$@\"\n"
  },
  {
    "path": "data/csv_header_indices.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-13 10:54:20 +0000 (Mon, 13 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick script to convert CSV header to column numbers for quicker programming reference\n# and reducing human counting errors for large CSVs such as from AWS credential reports\n#\n# eg.\n#\n# ./csv_header_indices.sh <<< \"user,arn,user_creation_time,password_enabled,password_last_used,password_last_changed,password_next_rotation,mfa_active,access_key_1_active,access_key_1_last_rotated,access_key_1_last_used_date,access_key_1_last_used_region,access_key_1_last_used_service,access_key_2_active,access_key_2_last_rotated,access_key_2_last_used_date,access_key_2_last_used_region,access_key_2_last_used_service,cert_1_active,cert_1_last_rotated,cert_2_active,cert_2_last_rotated\"\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nhead -n 1 \"$@\" |\ntr ',' '\\n' |\nnl -v 0  # -v 0 starts indexing at zero as this is what you need when coding\n"
  },
  {
    "path": "data/ini_config_add_if_missing.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 20:47:15 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads INI config blocks from stdin and appends them to the specified file if the section is not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nfile=\"$1\"\n\nmkdir -p -v \"$(dirname \"$file\")\"\n\ntouch \"$file\"\n\nadd_section=0\n\ntimestamp \"Loading any missing ini config sections to $file\"\n\nwhile read -r line; do\n    [[ \"$line\" =~ ^[[:space:]]*$ ]] && continue\n    if [[ $line =~ \\[(.*)\\] ]]; then\n        section=\"${BASH_REMATCH[1]}\"\n        section_shortname=\"$section\"\n        section_brackets=\"[$section]\"\n\n        if [ \"${SECTION_PREFIX:-}\" ]; then\n            # Convert to AWS config-style profile section\n            if ! [[ $section =~ ${SECTION_PREFIX}[[:space:]].+ ]]; then\n                section_brackets=\"[$SECTION_PREFIX $section]\"\n            else\n                section_brackets=\"[$section]\"\n            fi\n            # shellcheck disable=SC2295\n            section_shortname=\"${section#$SECTION_PREFIX}\"\n            section_shortname=\"${section_shortname## }\"\n        fi\n\n        # Check if the config block already exists in the file\n        if grep -Fq \"$section_brackets\" \"$file\"; then\n            timestamp \"Config block '$section_shortname' already found\"\n            add_section=0\n        else\n            timestamp \"Adding missing config block: $section_brackets\"\n            echo >> \"$file\"\n            echo \"$section_brackets\" >> \"$file\"\n            add_section=1\n        fi\n    elif [ \"$add_section\" = 1 ]; then\n        # Append the content of the file\n        echo \"$line\" >> \"$file\"\n    fi\ndone\n"
  },
  {
    "path": "data/ini_config_duplicate_section_names.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:30:58 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate INI config section names that are using the same value for a given key\nin the given .ini file\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file> <key>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nfile=\"$1\"\nkey=\"$2\"\n\nif ! [ -f \"$file\" ]; then\n    die \"ERROR: file does not exist: $file\"\nfi\n\n# check the given key actually exists somewhere\nif ! grep -q \"^[[:space:]]*${key}[[:space:]]*=\" \"$file\"; then\n    die \"ERROR: given key '$key' was not found in the file: $file\"\nfi\n\nduplicate_key_values=\"$(\n    grep \"^[[:space:]]*${key}[[:space:]]*=\" \"$file\" |\n    sed 's/.*=[[:space:]]*//' |\n    sort |\n    uniq -d |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nwhile read -r value; do\n    if is_blank \"$value\"; then\n        continue\n    fi\n    while read -r line; do\n        if is_blank \"$line\"; then\n            continue\n        elif [[ \"$line\" =~ ^[[:space:]]*\\[.+\\] ]]; then\n            section=\"$line\"\n        elif [[ \"$line\" =~ ^[[:space:]]*${key}[[:space:]]*=[[:space:]]*${value}[[:space:]]*$ ]]; then\n            echo \"$section\"\n        fi\n    done < \"$file\"\ndone <<< \"$duplicate_key_values\" |\nsed '\n    s/^[^[]*\\[//;\n    s/^[[:space:]]*//;\n    s/\\][^]]*$//;\n' |\nsort -fu\n"
  },
  {
    "path": "data/ini_config_duplicate_sections.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-20 18:30:58 +0400 (Wed, 20 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate INI config sections that are using the same value for a given key\nin the given .ini file\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file> <key>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nfile=\"$1\"\nkey=\"$2\"\n\nif ! [ -f \"$file\" ]; then\n    die \"ERROR: file does not exist: $file\"\nfi\n\n# check the given key actually exists somewhere\nif ! grep -q \"^[[:space:]]*${key}[[:space:]]*=\" \"$file\"; then\n    die \"ERROR: given key '$key' was not found in the file: $file\"\nfi\n\nduplicate_key_values=\"$(\n    grep \"^[[:space:]]*${key}[[:space:]]*=\" \"$file\" |\n    sed 's/.*=[[:space:]]*//' |\n    sort |\n    uniq -d |\n    sed '/^[[:space:]]*$/d'\n)\"\n\nsection=\"\"\n\nwhile read -r value; do\n    if is_blank \"$value\"; then\n        continue\n    fi\n    found=0\n    while read -r line; do\n        if is_blank \"$line\"; then\n            if [ \"$found\" = 1 ]; then\n                echo \"$section\"\n                section=\"\"\n                echo\n                found=0\n            fi\n            continue\n        elif [[ \"$line\" =~ ^[[:space:]]*\\[.+\\] ]]; then\n            section=\"$line\"\n        else\n            section+=\"\n$line\"\n            if [[ \"$line\" =~ ^[[:space:]]*${key}[[:space:]]*=[[:space:]]*${value}[[:space:]]*$ ]]; then\n                found=1\n            fi\n        fi\n    done < \"$file\"\ndone <<< \"$duplicate_key_values\"\n"
  },
  {
    "path": "data/ini_grep_section.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-24 04:19:14 +0400 (Sun, 24 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints the named section from a given .ini file to stdout\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<section> <file>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nsection=\"$1\"\nfile=\"$2\"\n\nif ! [ -f \"$file\" ]; then\n    die \"ERROR: file does not exist: $file\"\nfi\n\nfound=0\n\nwhile read -r line; do\n    if is_blank \"$line\"; then\n        if [ \"$found\" = 1 ]; then\n            echo \"$section\"\n            section=\"\"\n            echo\n            found=0\n        fi\n        continue\n    elif [[ \"$line\" =~ ^[[:space:]]*\\[$section\\] ]]; then\n        section=\"$line\"\n        found=1\n    elif [ \"$found\" = 1 ]; then\n        section+=\"\n$line\"\n    fi\ndone < \"$file\"\n\n# print if we hit end of file\nif [ \"$found\" = 1 ]; then\n    echo \"$section\"\n    section=\"\"\n    echo\n    found=0\nfi\n"
  },
  {
    "path": "data/json2yaml.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: ../pytools/tests/data/test.json\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-18 01:34:41 +0100 (Tue, 18 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts JSON to YAML using either Perl, Ruby or Python (whichever is available in that order)\n\nJSON can be specified as a filename argument to piped to standard input\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<filename>]\"\n\nhelp_usage \"$@\"\n\njson2yaml(){\n    # needs 3rd party modules installed (YAML::XS, JSON::XS), so check we have both modules first\n    #\n    # Perl module results in json\n    #\n    #   \"ssh_pty\":true\n    #\n    # being translated as\n    #\n    #   ssh_pty: !!perl/scalar:JSON::PP::Boolean 1\n    #\n    # whereas the ruby below comes out properly, so don't use Perl JSON::XS for this any more\n    # They ruby version below also maintains key ordering so it's a more true comparison\n    #\n    #if type -P perl &>/dev/null &&\n    #   perl -MYAML::XS=Load -MJSON::XS=encode_json -e '' &>/dev/null; then\n    #    #perl -MYAML::XS=LoadFile -MJSON::XS=encode_json -e 'for (@ARGV) { for (LoadFile($_)) { print encode_json($_),\"\\n\" } }'\n    #    perl -MYAML::XS=Dump -MJSON::XS=decode_json -e '$/ = undef; print Dump(decode_json(<STDIN>)) . \"\\n\"'\n    # untested, so many transitive dependencies, a couple fail to build\n    #elif type -P catmandu &>/dev/null; then\n    #    catmandu convert JSON to YAML\n    if type -P ruby &>/dev/null &&\n         ruby -r yaml -r json -e '' &>/dev/null; then\n        # don't want variable expansion\n        # shellcheck disable=SC2016\n        ruby -r yaml -r json -e 'puts YAML.dump(JSON.parse(STDIN.read))'\n    # moved to last as typical Python version change problems, breaks across environments with AttributeError: 'module' object has no attribute 'FullLoader'\n    # yaml is a 3rd party library, and in old 2.x versions so was json - only run the Python conversion if we have both libraries installed\n    elif type -P python &>/dev/null &&\n         python -c 'import yaml, json' &>/dev/null; then\n        python -c 'import sys, yaml, json; yaml.safe_dump(json.load(sys.stdin), sys.stdout, default_flow_style=False)'\n    # don't use yq - there are 2 completely different 'yq' which could appear in \\$PATH, so this is unreliable\n    #elif type -P yq &>/dev/null; then\n    else\n        die \"ERROR: unable to convert yaml to json since not one of the following tools were found:  Perl (YAML::XS + JSON::XS), Ruby (json + yaml) or Python (PyYaml)\"\n    fi\n}\n\nif [ $# -gt 0 ]; then\n    for arg; do\n        json2yaml < \"$arg\"\n        echo\n    done\nelse\n    json2yaml\n    echo\nfi\n"
  },
  {
    "path": "data/lines_to_end.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-05 01:05:31 +0700 (Thu, 05 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFilters lines matching the given regex in a given file or stdin and outputs them at the end of stdout\n\nUsed by my .vimrc to push my header links and similar down to the bottom for URL view pop-up to prioritize 3rd party docuementation links\n\nTo enforce the regex matching with case sensitivity:\n\n    export IGNORECASE=0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ere_regex> [<file_or_stdin>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nregex=\"$1\"\narg=\"${2:-}\"\n\nignorecase=1  # true\n\n# testing expected value instead of passing to awk for safety\nif [ \"${IGNORECASE:-}\" = 0 ]; then\n    ignorecase=0  # false\nfi\n\nif [ $# -eq 1 ]; then\n    cat\nelif [ -f \"$arg\" ]; then\n    cat \"$arg\"\nelse\n    echo \"$arg\"\nfi |\n# IGNORECASE requires gawk, not BSD awk, is mapped in lib/utils.sh\nawk -v regex=\"$regex\" \\\n    -v IGNORECASE=\"$ignorecase\" \\\n'\n    $0 ~ regex {\n        matches[++m] = $0\n        next\n    }\n    {\n        print\n    }\n    END {\n        for (i = 1; i <= m; i++) print matches[i]\n    }\n'\n"
  },
  {
    "path": "data/parquet_tools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-16 14:09:20 +0200 (Mon, 16 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns Parquet Tools jar, downloading it if not already present\n\nUses ../install/download_parquet_tools.sh to download the latest version if no version is present\n\nYou can call that script directory if you want to update the version\n\n--help prints this message. There is no Parquet Tools --help, only command args like parquet-cat\n\nRequires Java to be installed to run the parquet-tools-<version>.jar\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<parquet_tools_args>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nparquet_tools_jar=\"$(find \"$srcdir\" -maxdepth 1 -name 'parquet-tools-*.jar' | sort -Vr | head -n 1)\"\n\nif [ -z \"$parquet_tools_jar\" ] ||\n   # incomplete download, call download again to resume it\n   ! jar tf \"$parquet_tools_jar\" &>/dev/null; then\n    pushd \"$srcdir\" 2>/dev/null || die \"Failed to pushd to '$srcdir'\"\n    \"$srcdir/../install/download_parquet_tools.sh\"\n    popd &>/dev/null || die \"Failed to return to original dir\"\n    parquet_tools_jar=\"$(find \"$srcdir\" -maxdepth 1 -name 'parquet-tools-*.jar' | sort -Vr | head -n 1)\"\n    echo\nfi\n\njava -jar \"$parquet_tools_jar\" \"$@\"\n"
  },
  {
    "path": "data/wordcloud.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-23 02:12:38 +0400 (Sat, 23 Nov 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a wordcloud from file(s) or stdin using ImageMagick\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<wordcloud.png>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\npng=\"${1:-wordcloud.png}\"\n\n#wordcounts=\"$(mktemp)\"\n\n#timestamp \"Running wordcount.sh\"\n#\"$srcdir/wordcount.sh\" \"$@\" |\n#awk '$2 !~ /^[^[:alnum:]]+$/' |\n#awk '$2 !~ /^[[:digit:]]+$/' \\\n#> \"$wordcounts\"\n\ntimestamp \"Running wordcloud_cli\"\n#wordcloud_cli --text \"$wordcounts\" --imagefile \"$png\"\nwordcloud_cli --imagefile \"$png\"\n\n#expanded_words=\"$(mktemp)\"\n#\n#timestamp \"Generating word expansion\"\n#awk '{for (i=1; i<=$1; i++) printf $2 \" \";}' < \"$wordcounts\" > \"$expanded_words\"\n#\n#timestamp \"Generating PNG '$png' using ImageMagick\"\n#magick \\\n#    -size 800x600 \\\n#    xc:white \\\n#    -font Arial \\\n#    -pointsize 20 \\\n#    -gravity center \\\n#    -annotate 0 \\\n#    @\"$expanded_words\" \\\n#    \"$png\"\n\ntimestamp \"Opening '$png'\"\n\"$srcdir/../media/imageopen.sh\" \"$png\"\n"
  },
  {
    "path": "data/wordcount.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-24 22:04:05 +0100 (Mon, 24 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a word count list ranked by most used words at the top\n\nWorks like a standard unix filter program - pass in stdin or give it a filename, and outputs to stdout, so you can continue to pipe or redirect to a file as usual\n\nIf the filename arg is a .pdf file then uses pdftotext to dump out the words\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<filename>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#filename=\"$1\"\n\nif [ $# -eq 0 ]; then\n    echo \"Reading from stdin\" >&2\nfi\n\n#output_file=\"$filename.word_frequency.txt\"\n\nshopt -s nocasematch\nif [[ \"${1:-}\" =~ \\.pdf ]]; then\n    pdftotext \"$1\" -\nelse\n    # one of the few legit uses of cat - tr can't process a filename arg or stdin\n    cat \"$@\"\nfi |\n#tr '[:punct:]' ' ' |\ntr '[:space:]' '\\n' |\ntr '[:upper:]' '[:lower:]' |\nsed \"\n    /^[[:space:]]*$/d;\n    #/^$USER$/d;\n    # because sometimes you want to see the occurence of emojis in WhatsApp chats\n    #/^[^[:alnum:]]*$/d;\n\" |\nsort |\nuniq -c |\nsort -k1nr  # > \"$output_file\"\n\n#head -n \"$LINES\" \"$output_file\"\n"
  },
  {
    "path": "data/yaml2json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: .gitlab-ci.yml\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 19:21:27 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts YAML to JSON using either Perl, Ruby or Python (whichever is available in that order)\n\nYAML can be specified as a filename argument to piped to standard input\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<filename>]\"\n\nhelp_usage \"$@\"\n\nyaml2json(){\n    # needs 3rd party modules installed (YAML::XS, JSON::XS), so check we have both modules first\n    if type -P perl &>/dev/null &&\n         perl -MYAML::XS=Load -MJSON::XS=encode_json -e '' &>/dev/null; then\n        #perl -MYAML::XS=LoadFile -MJSON::XS=encode_json -e 'for (@ARGV) { for (LoadFile($_)) { print encode_json($_),\"\\n\" } }'\n        perl -MYAML::XS=Load -MJSON::XS=encode_json -e '$/ = undef; print encode_json(Load(<STDIN>)) . \"\\n\"'\n    elif type -P ruby &>/dev/null &&\n         ruby -r yaml -r json -e '' &>/dev/null; then\n        # don't want variable expansion\n        # shellcheck disable=SC2016\n        ruby -r yaml -r json -e 'puts YAML.load($stdin.read).to_json'\n    # moved to last as typical Python version change problems, breaks across environments with AttributeError: 'module' object has no attribute 'FullLoader'\n    # yaml is a 3rd party library, and in old 2.x versions so was json - only run the Python conversion if we have both libraries installed\n    elif type -P python &>/dev/null &&\n       python -c 'import yaml, json' &>/dev/null; then\n        python -c 'import sys, yaml, json; json.dump(yaml.load(sys.stdin, Loader=yaml.FullLoader), sys.stdout, indent=4)'\n    # don't use yq - there are 2 completely different 'yq' which could appear in \\$PATH, so this is unreliable\n    #elif type -P yq &>/dev/null; then\n    else\n        die \"ERROR: unable to convert yaml to json since not one of the following tools were found:  Perl (YAML::XS + JSON::XS), Ruby (json + yaml) or Python (PyYaml)\"\n    fi\n}\n\nif [ $# -gt 0 ]; then\n    for arg; do\n        yaml2json < \"$arg\"\n        echo\n    done\nelse\n    yaml2json\n    echo\nfi\n"
  },
  {
    "path": "diagrams/d2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-05 19:16:39 +0700 (Wed, 05 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a D2lang diagram, using its shebang if present for themes etc, and then opens the resulting image\n\nIf the .d2 file is set executable, runs it as is,\notherwise checks for shebang, sets executable if present and runs it,\notherwise falls back to a stock d2 build\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file.d2> [<file2.d2>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif ! type -P d2 &>/dev/null; then\n    timestamp \"D2lang binary 'd2' not found in \\$PATH, attempting to install via package manager\"\n    \"$srcdir/../packages/install_packages.sh\" d2\n    echo >&2\nfi\n\nfor arg in \"$@\"; do\n    if ! [ -f \"$arg\" ]; then\n        warn \"not a file, skipping: $arg\"\n        continue\n    fi\n    filename=\"$arg\"\n    if ! [ -s \"$filename\" ]; then\n        warn \"file is empty, skipping: $filename\"\n        continue\n    fi\n    if ! [[ \"$filename\" =~ \\.d2$ ]]; then\n        warn \"not a D2lang file ending in .d2, skipping: $filename\"\n        continue\n    fi\n    header_line=\"$(head -n1 \"$filename\")\"\n    #if [ \"${header_line:0:2}\" = '#!' ]; then\n    if [[ \"$header_line\" =~ ^#!.+d2 ]]; then\n        if ! [ -x \"$filename\" ]; then\n            timestamp \"Shebang detected but not executable, setting executable bit on: $filename\"\n            chmod +x \"$filename\"\n        fi\n        filepath=\"$(readlink -f \"$filename\")\"\n        timestamp \"Running shebang: $filepath\"\n        \"$filepath\"\n    else\n        timestamp \"Running default d2 build: $filename\"\n        d2 \"$filename\"\n    fi\n    file_basename=\"${filename%.*}\"\n    # shellcheck disable=SC2012\n    # ls returns exit 1 when one of the paths isn't found, so ignore its exit code and look for blank\n    generated_image=\"$(ls -t \"$file_basename\".png \"$file_basename\".svg 2>/dev/null | head -n1 || :)\"\n    if is_blank \"$generated_image\"; then\n        die \"Failed to find generated image for: $filename\"\n    fi\n    timestamp \"Opening: $generated_image\"\n    \"$srcdir/imageopen.sh\" \"$generated_image\"\ndone\n"
  },
  {
    "path": "diagrams/d2_generate_diagrams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-22 00:22:08 +0700 (Sat, 22 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates all .d2 diagrams found under the current or given directory\n\nSet SKIP_FILENAME_REGEX environment variable to exclude trying to generate your base templates\nor similar that which are likely to fail for having a superset of example code in them\n\n    export SKIP_FILENAME_REGEX='template.d2|diagram.d2'\n\nPorted from Makefile in:\n\n    https://github.com/HariSekhon/Diagrams-as-Code\n\nbecause started getting more complicated with error handling and revertion on incorrectly generated diagrams,\nfor which a Makefile is not the best to do long multi-line code blocks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<source_dir> <target_dir> <png|svg>]\"\n\nhelp_usage \"$@\"\n\nmax_args 3 \"$@\"\n\nsrc_dir=\"${1:-.}\"\ntarget_dir=\"${2:-.}\"\next=\"${3:-svg}\"\n\ntarget_dir=\"${target_dir%%/}\"\n\nif ! [[ \"$ext\" =~ ^(png|svg)$ ]]; then\n    die \"Invalid extension specified: must be 'png' or 'svg'\"\nfi\n\nif ! type -P d2 &>/dev/null; then\n    \"$srcdir/../install/install_d2.sh\"\nfi\n\necho ======================\necho Generating D2 Diagrams\necho ======================\n\nmkdir -p -v \"$target_dir\"\n\n# workaround to use shebang because d2 doesn't currently support defining the theme in the .d2 file\n# and also doesn't support having a separate images/ directory, see:\n#\n#\thttps://github.com/terrastruct/d2/issues/1286\n#\n#\thttps://github.com/terrastruct/d2/issues/1287\n#\n#\thttps://github.com/terrastruct/d2/issues/1288\n#\n#if [ -x \"$x\" ]; then\n#    ./\"$x\"\n#fi\n\nexitcode=0\n\nwhile read -r filename; do\n    if [ -n \"${SKIP_FILENAME_REGEX:-}\" ]; then\n        if [[ \"$filename\" =~ $SKIP_FILENAME_REGEX ]]; then\n            continue\n        fi\n    fi\n    if [ \"$target_dir\" = . ]; then\n        img=\"${filename%.d2}.$ext\"\n    else\n        basename=\"${filename##*/}\"\n        img=\"$target_dir/${basename%.d2}.$ext\"\n    fi\n    shebang=\"$(\n        head -n 1 \"$filename\" |\n        awk '/^#!\\/.*d2/{print}' |\n        sed 's/^#!//'\n    )\"\n    if [ -z \"$shebang\" ]; then\n        #shebang=\"d2 --theme 200\"\n        shebang=\"d2\"\n    fi\n    timestamp \"Generating: $filename -> $img\"\n    if ! $shebang \"$filename\" \"$img\"; then\n        timestamp \"Failed to generate: $img\"\n        git checkout \"$img\" 2>/dev/null ||\n        rm -fv \"$img\"\n        exitcode=1\n    fi\ndone < <(find \"$src_dir\" -type f -iname '*.d2')\n\nexit \"$exitcode\"\n"
  },
  {
    "path": "diagrams/mermaidjs_generate_diagrams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-22 00:22:08 +0700 (Sat, 22 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates all MermaidJS .mmd diagrams found under the current or given directory\n\nSet SKIP_FILENAME_REGEX environment variable to exclude trying to generate your base templates\nor similar that which are likely to fail for having a superset of example code in them\n\n    export SKIP_FILENAME_REGEX='template.mmd|diagram.mmd'\n\nPorted from Makefile in:\n\n    https://github.com/HariSekhon/Diagrams-as-Code\n\nbecause started getting more complicated with error handling and revertion of incorrectly generated diagrams,\nfor which a Makefile is not the best to do long multi-line code blocks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<source_dir> <target_dir> <png|svg>]\"\n\nhelp_usage \"$@\"\n\nmax_args 3 \"$@\"\n\nsrc_dir=\"${1:-.}\"\ntarget_dir=\"${2:-.}\"\next=\"${3:-svg}\"\n\ntarget_dir=\"${target_dir%%/}\"\n\nif ! [[ \"$ext\" =~ ^(png|svg)$ ]]; then\n    die \"Invalid extension specified: must be 'png' or 'svg'\"\nfi\n\nif ! type -P mmdc &>/dev/null; then\n    \"$srcdir/../install/install_mermaidjs.sh\"\nfi\n\necho =============================\necho Generating MermaidJS Diagrams\necho =============================\n\nmkdir -p -v \"$target_dir\"\n\nexitcode=0\n\nwhile read -r filename; do\n    if [ -n \"${SKIP_FILENAME_REGEX:-}\" ]; then\n        if [[ \"$filename\" =~ $SKIP_FILENAME_REGEX ]]; then\n            continue\n        fi\n    fi\n    if [ \"$target_dir\" = . ]; then\n        img=\"${filename%.mmd}.$ext\"\n    else\n        basename=\"${filename##*/}\"\n        img=\"$target_dir/${basename%.mmd}.$ext\"\n    fi\n    timestamp \"Generating: $filename -> $img\"\n    if ! mmdc -i \"$filename\" -o \"$img\"; then\n        timestamp \"Failed to generate: $img\"\n        git checkout \"$img\" 2>/dev/null ||\n        rm -fv \"$img\"\n        exitcode=1\n    fi\ndone < <(find \"$src_dir\" -type f -iname '*.mmd')\n\nexit \"$exitcode\"\n"
  },
  {
    "path": "diagrams/python_mingrammer_generate_diagrams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-22 00:22:08 +0700 (Sat, 22 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates all Python Mingrammer .py diagrams found under the current or given directory\n\nChecks the python files contain either:\n\nimport diagrams\n\nor\n\nfrom diagrams import ...\n\n\nSet SKIP_FILENAME_REGEX environment variable to exclude trying to generate your base templates\nor similar that which are likely to fail for having a superset of example code in them\n\n    export SKIP_FILENAME_REGEX='template.py|diagram.py'\n\nPorted from Makefile in:\n\n    https://github.com/HariSekhon/Diagrams-as-Code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<source_dir>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n# HariSekhon/Diagrams-as-Code python diagrams are set to auto-open unless this is set\nexport CI=1\n\nsrc_dir=\"${1:-.}\"\n\necho ==========================\necho Generating Python Diagrams\necho ==========================\n\ncd \"$src_dir\"\n\nexitcode=0\n\nwhile read -r filename; do\n    if ! grep -Eq \\\n        -e '^[[:space:]]*import[[:space:]]+diagrams[[:space:]]*$' \\\n        -e '^[[:space:]]*from[[:space:]]+diagrams[[:space:]]+import[[:space:]]+' \\\n        \"$filename\"; then\n        log \"Skipping Python file without any 'diagrams' imports: $filename\"\n        continue\n    fi\n    if [ -n \"${SKIP_FILENAME_REGEX:-}\" ]; then\n        if [[ \"$filename\" =~ $SKIP_FILENAME_REGEX ]]; then\n            continue\n        fi\n    fi\n    timestamp \"Generating: $filename\"\n    if ! python \"$filename\"; then\n        timestamp \"Failed to generate: $filename\"\n        exitcode=1\n    fi\ndone < <(find . -type f -iname '*.py')\n\nexit \"$exitcode\"\n"
  },
  {
    "path": "docker/docker_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: https://hub.docker.com/v2/repositories/harisekhon/hbase/tags | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-14 15:25:12 +0100 (Mon, 14 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries a Docker Registry API v2\n\nSends basic HTTP authentication if the following environment variables are defined:\n\n\\$DOCKER_USERNAME / \\$DOCKER_USER\n\\$DOCKER_PASSWORD / \\$DOCKER_TOKEN\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://docs.docker.com/registry/spec/api/\n\n\nExamples:\n\n# Get all the tags for a given repository called 'harisekhon/hbase' on DockerHub's public registry:\n\n    ${0##*/} https://hub.docker.com/v2/repositories/harisekhon/hbase/tags\n\n\nFor authenticated queries against DockerHub which requires a more complicated OAuth or JWT workflow\nrather than basic authentication, this script calls the adjacent 'dockerhub_api.sh' script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"https://host:port/v2/... [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nuser=\"${DOCKER_USERNAME:-${DOCKER_USER:-}}\"\nPASSWORD=\"${DOCKER_PASSWORD:-${DOCKER_TOKEN:-}}\"\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nfi\nexport PASSWORD\n\nurl=\"$1\"\nshift || :\n\nif [[ \"$url\" =~ hub.docker.com ]]; then\n    \"$srcdir/dockerhub_api.sh\" \"$url\" \"$@\"\nelif [ -n \"${PASSWORD:-}\" ]; then\n    \"$srcdir/../bin/curl_auth.sh\" \"$url\" \"${CURL_OPTS[@]}\" \"$@\"\nelse\n    curl \"$url\" \"${CURL_OPTS[@]}\" \"$@\"\nfi\n"
  },
  {
    "path": "docker/docker_build_hashref.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-10 18:12:25 +0200 (Wed, 10 Jul 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns Docker Build and auto-generates docker image name and tag from relative Git path and commit short SHA\n\nIf it's a dirty checkout with added / modified files then appends '-dirty-<checksum>' to the docker tag\nwhere '<checksum>' is another short hash of the Git status porcelain to differentiate it from the clean unmodified commit version\n\nUseful to compare docker image sizes between your clean and modified versions of Dockerfile or contents\n\nIf you want to set the docker image name instead of letting it default to the git relative root dir, then\nset the environment variable:\n\n    export DOCKER_BUILD_IMAGE_NAME=...\n\nThe dirty commit only hashes which files were added / changed, not their content changes\nIf you really want the hash to change for any differing content changes, then set the environment variable:\n\n    export DOCKER_BUILD_HASH_CONTENTS=1\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<context_path> <docker_build_options>]\"\n\nhelp_usage \"$@\"\n\ncontext_dir=\"${1:-.}\"\nshift || :\ndocker_build_options=(\"$@\")\n\nif ! is_in_git_repo; then\n    die \"Not in a Git repo to auto-determine the Git short sha and relative path for docker image naming\"\nfi\n\ngit_commit_short_sha=\"$(git_commit_short_sha)\"\n\nimage_name=\"${DOCKER_BUILD_IMAGE_NAME:-}\"\n\nif is_blank \"$image_name\"; then\n    git_relative_dir=\"$(git_relative_dir)\"\n\n    image_name=\"${git_relative_dir////--}\"\nfi\n\ndirty=\"\"\nif git status --porcelain | grep -q . ; then\n  if [ \"${DOCKER_BUILD_HASH_CONTENTS:-}\" = 1 ]; then\n    git_root=\"$(git_root)\"\n    # prepend the git root dir to each file because 'git status --porcelain' gives from relative root of repo\n    # then cat each file and pipe them all into md5sum to detect any content differences in the git repo for a unique hashref\n    dirty=\"-dirty-$(git status --porcelain | cut -c 4- | sed \"s|^|$git_root/|\" | xargs cat | md5sum | cut -c 1-7)\"\n  else\n    # if 7 char short hash is good enough for Git then it's good enough for me\n    dirty=\"-dirty-$(git status --porcelain | md5sum | cut -c 1-7)\"\n  fi\nfi\n\nset -x\ndocker build \"$context_dir\" -t \"${image_name}:${git_commit_short_sha}${dirty}\" \"${docker_build_options[@]}\"\n\ndocker images | grep \"^$image_name\"\n"
  },
  {
    "path": "docker/docker_generate_status_page.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-07 15:01:31 +0000 (Fri, 07 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to generate DOCKER_STATUS.md containing all DockerHub / Docker Cloud repo build statuses for a user on a single page\n#\n# Usage:\n#\n# without arguments generates a page containing statuses for all DockerHub repos for your $DOCKER_USER\n#\n#   DOCKER_USER=harisekhon ./github_generate_status_docker.sh\n#\n# with arguments will only generate a page for those repos (repos will not be checked for existence but will get repo not found on the page itself)\n#\n# if not specifying the <user>/ prefix then auto prependeds $DOCKER_USER/\n#\n#   DOCKER_USER=harisekhon ./github_generate_status_page.sh  harisekhon/hbase harisekhon/zookeeper harisekhon/nagios-plugins ...\n#\n#   DOCKER_USER=harisekhon ./github_generate_status_page.sh  hbase zookeeper nagios-plugins ...\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ntrap 'echo ERROR >&2' exit\n\ncd \"$srcdir\"\n\nfile=\"DOCKER_STATUS.md\"\n\nrepolist=\"$*\"\n\n# this leads to confusion as it generates some randomly unexpected output by querying a dockerhub user who happens to have the same name as your local user eg. hari, so force explicit now\n#USER=\"${DOCKER_USER:-${USERNAME:-${USER}}}\"\nif [ -z \"${DOCKER_USER:-}\" ] ; then\n    echo \"\\$DOCKER_USER not set!\"\n    exit 1\nfi\n\nif [ -z \"$repolist\" ]; then\n    repolist=\"$(dockerhub_search.py -n 100 harisekhon | awk '/^harisekhon\\/.*[[:space:]]\\[OK]/{print $1}' | sort)\"\nfi\n\nnum_repos=\"$(wc -l <<< \"$repolist\")\"\nnum_repos=\"${num_repos// /}\"\n\n{\ncat <<EOF\n# Docker Status Page\n\ngenerated by \\`${0##*/}\\` in [HariSekhon/DevOps-Bash-tools](https://github.com/HariSekhon/DevOps-Bash-tools)\n\nThis page relies on shields.io which is slow so a lot of it may not load properly the first time so you may need to do one or more page reloads to get all the badges to load.\n\nEOF\n# don't expand `:latest`\n# shellcheck disable=SC2016\necho \"$num_repos docker repos - \"'`:latest`'\" tag build status:\"\necho\nfor repo in $repolist; do\n    if ! [[ \"$repo\" =~ / ]]; then\n        repo=\"$DOCKER_USER/$repo\"\n    fi\n    echo \"[![Docker Build Status](https://img.shields.io/docker/cloud/build/$repo.svg)](https://hub.docker.com/r/$repo/builds)\"\n    echo \"[![DockerHub Pulls](https://img.shields.io/docker/pulls/$repo.svg)](https://hub.docker.com/r/$repo) -\"\n    echo \"[$repo](https://hub.docker.com/r/$repo)\"\n    echo\ndone\n} | tee \"$file\"\n\ntrap '' exit\n"
  },
  {
    "path": "docker/docker_mount_build_exec.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:14:53 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ $# != 1 ]; then\n    echo \"usage: ${00#*/} <docker_image>\"\n    exit 3\nfi\n\ndocker_image=\"$1\"\n\nscript=\"bin/exec_interactive.sh\"\nif [ -x \"bash-tools/$script\" ]; then\n    script=\"bash-tools/$script\"\nfi\n\n# 'which' command is not available in some bare bones docker images like centos\n# cannot set -u because it results in unbound variable error for $USER\n# cannot set -e because it will exit before the exec to persist\ndocker run -ti --rm -v \"$PWD:/code\" \"$docker_image\" sh -x \"/code/$script\" '\n    cd /code\n    if type apt-get &>/dev/null; then\n        export DEBIAN_FRONTEND=noninteractive\n        apt-get update\n        apt-get install -y git make\n    elif type apk &>/dev/null; then\n        apk add --no-cache git make\n    elif type yum &>/dev/null; then\n        yum install -y git make\n    fi\n    make build test\n'\n"
  },
  {
    "path": "docker/docker_package_check.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: ubuntu pre-commit\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-22 20:47:39 +0100 (Sun, 22 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns package installs on the last 8 major versions of a given docker image to check given packages are available\nbefore adding them and breaking builds across distro versions\n\nUses adjacent scripts:\n\n    ../bin/linux_distro_versions.sh\n\n    ../packages/install_packages.sh\n\nto install the packages using whatever local package manager is detected\n\nSet environment variable MAX_VERSIONS to change the number of distro versions to run against (default: 8)\n\nCurrently only supports DockerHub images\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> <package1> [<package2> <package3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nimage=\"$1\"\nshift || :\n\nmax_versions=\"${MAX_VERSIONS:-8}\"\n\nif ! is_int \"$max_versions\"; then\n    die \"MAX_VERSIONS may only be set to an integer, not: $max_versions\"\nfi\n\ncheck_bin docker\n\nif [[ \"$image\" =~ alpine ]]; then\n    major_versions=\"$(\"$srcdir/../bin/linux_distro_versions.sh\" alpine)\"\nelif [[ \"$image\" =~ debian ]]; then\n    major_versions=\"$(\"$srcdir/../bin/linux_distro_versions.sh\" debian)\"\nelif [[ \"$image\" =~ ubuntu ]]; then\n    major_versions=\"$(\"$srcdir/../bin/linux_distro_versions.sh\" ubuntu)\"\nelif [[ \"$image\" =~ fedora ]]; then\n    major_versions=\"$(\"$srcdir/../bin/linux_distro_versions.sh\" fedora)\"\nelif [[ \"$image\" =~ centos ]]; then\n    major_versions=\"$(\"$srcdir/../bin/linux_distro_versions.sh\" centos)\"\nelse\n    timestamp \"Querying DockerHub for major versions of image '$image'\"\n    major_versions=\"$(\n        \"$srcdir/../docker/dockerhub_list_tags.sh\" \"$image\" |\n        grep -Eo -e '^[[:digit:]]+(\\.[[:digit:]]+)?$' \\\n                 -e '^latest$'\n    )\"\nfi\n\nif grep -Eq '^[[:digit:]]+$' <<< \"$major_versions\"; then\n    timestamp \"Major version tags detected, using only those to save time\"\n    major_versions=\"$(grep -EO -e '^[[:digit:]]+$' -e 'latest' <<< \"$major_versions\" | sort -Vr)\"\nfi\n\nmajor_versions=\"$(head -n \"$max_versions\" <<< \"$major_versions\")\"\n\necho\ntimestamp \"Running for major versions:\"\necho\necho \"$major_versions\"\n\nfor version in $major_versions; do\n    echo\n    timestamp \"Launching docker container for '$image:$version'\"\n    docker run -ti --rm -v \"$srcdir/..\":/pwd -w /pwd \"$image:$version\" packages/install_packages.sh \"$@\"\ndone\n"
  },
  {
    "path": "docker/docker_registry_get_image_manifest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: localhost:5000 centos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 12:05:41 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets the docker manifest for a given image:tag from a Docker Registry via the Docker Registry API v2\n\nIf :<tag> isn't given, assumes 'latest'\n\nSee adjacent docker_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"http(s)://host:port <image>[:<tag>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\ndocker_registry_url=\"$1\"\nimage_tag=\"$2\"\nshift || :\nshift || :\n\nif ! [[ \"$docker_registry_url\" =~ ^(https?://)?[[:alnum:].-]+:[[:digit:]]+/?$ ]]; then\n    usage \"invalid docker registry url: $docker_registry_url\"\nfi\n\nif ! [[ \"$docker_registry_url\" =~ ^https?:// ]]; then\n    docker_registry_url=\"http://$docker_registry_url\"\nfi\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif ! [[ \"$image_tag\" =~ : ]] &&\n   [ \"$tag\" = \"$image\" ]; then\n    tag=\"latest\"\nfi\n\n\"$srcdir/docker_api.sh\" \"$docker_registry_url/v2/$image/manifests/$tag\" -H \"Accept: application/vnd.docker.distribution.manifest.v2+json\"\n"
  },
  {
    "path": "docker/docker_registry_list_images.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 11:21:54 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#  docker run -d -p 5000:5000 --restart always --name registry registry:2\n#  docker tag centos localhost:5000/centos\n#  docker tag ubuntu localhost:5000/ubuntu\n#  docker tag registry:2 localhost:5000/registry:2\n#  docker push localhost:5000/centos\n#  docker push localhost:5000/ubuntu\n#  docker push localhost:5000/registry:2\n#\n#  args: http://localhost:5000\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the images available in a private Docker Registry using the Docker Registry API v2\n\nSee adjacent docker_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"http(s)://host:port\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ndocker_registry_url=\"$1\"\nshift || :\n\nif ! [[ \"$docker_registry_url\" =~ ^(https?://)?[[:alnum:].-]+:[[:digit:]]+/?$ ]]; then\n    usage \"invalid docker registry url: $docker_registry_url\"\nfi\n\nif ! [[ \"$docker_registry_url\" =~ ^https?:// ]]; then\n    docker_registry_url=\"http://$docker_registry_url\"\nfi\n\n\"$srcdir/docker_api.sh\" \"$docker_registry_url/v2/_catalog\" |\njq -r '.repositories[]'\n"
  },
  {
    "path": "docker/docker_registry_list_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: localhost:5000 centos\n#  args: https://hub.docker.com centos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 11:08:01 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists tags for a given Docker Registry image using the Docker Registry API\n\nExample:\n\n    ${0##*/} localhost:5000 centos\n\n    ${0##*/} localhost:5000 ubuntu\n\n    ${0##*/} localhost:5000 harisekhon/hbase\n\n    ${0##*/} https://hub.docker.com centos\n\n    ${0##*/} https://hub.docker.com ubuntu\n\n    ${0##*/} https://hub.docker.com harisekhon/hbase\n\n\nIf the registry given is hub.docker.com, calls dockerhub_list_tags.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"https://host:port <image> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nregistry=\"$1\"\nimage=\"$2\"\nshift || :\nshift || :\n\nif [[ \"$registry\" =~ hub.docker.com ]]; then\n    # calling this unifies the logic around prefixing library/ to official images and takes care of the differing paths between Docker Registry API and DockerHub APIs\n    exec \"$srcdir/dockerhub_list_tags.sh\" \"$image\" \"$@\"\nfi\n\n# now we only have to deal with Docker Registry API\nurl=\"$registry/v2/$image/tags/list\"\n\n\"$srcdir/docker_api.sh\" \"$url\" \"$@\" |\njq -r '.tags[]'\n"
  },
  {
    "path": "docker/docker_registry_tag_image.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: localhost:5000 centos haritest\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 12:05:41 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given image in a Docker Registry with another tag via the Docker Registry API v2\nwithout pulling and pushing the data (much faster and more efficient)\n\nIf :<tag> isn't given, assumes 'latest'\n\nSee adjacent docker_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"http(s)://host:port <image>[:<tag>] <new_tag>\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\ndocker_registry_url=\"$1\"\nimage_tag=\"$2\"\nnew_tag=\"$3\"\nshift || :\nshift || :\nshift || :\n\nif ! [[ \"$docker_registry_url\" =~ ^(https?://)?[[:alnum:].-]+:[[:digit:]]+/?$ ]]; then\n    usage \"invalid docker registry url: $docker_registry_url\"\nfi\n\nif ! [[ \"$docker_registry_url\" =~ ^https?:// ]]; then\n    docker_registry_url=\"http://$docker_registry_url\"\nfi\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif ! [[ \"$image_tag\" =~ : ]] &&\n   [ \"$tag\" = \"$image\" ]; then\n    tag=\"latest\"\nfi\n\nmanifest=\"$(\"$srcdir/docker_registry_get_image_manifest.sh\" \"$docker_registry_url\" \"$image_tag\")\"\n\n\"$srcdir/docker_api.sh\" \"$docker_registry_url/v2/$image/manifests/$new_tag\" -X PUT -d \"$manifest\" -H \"Content-Type: application/vnd.docker.distribution.manifest.v2+json\"\n"
  },
  {
    "path": "docker/dockerhub_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /repositories/harisekhon/hbase/tags | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-14 15:25:12 +0100 (Mon, 14 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the DockerHub.com API v2\n\nAutomatically handles getting an authentication token if you've got auth environment variables:\n\n\\$DOCKERHUB_USERNAME / \\$DOCKERHUB_USER  / \\$DOCKER_USERNAME / \\$DOCKER_USER\n\\$DOCKERHUB_PASSWORD / \\$DOCKERHUB_TOKEN / \\$DOCKER_PASSWORD / \\$DOCKER_TOKEN\n\\$DOCKERHUB_2FA_CODE (if set, does a 2nd level 2FA auth call with this code to get a token before calling the API endpoint)\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here:\n\n    https://hub.docker.com/settings/security\n\n\nAPI Reference:\n\n    https://docs.docker.com/registry/spec/api/\n\nDockerHub doesn't respect a lot of the Docker Registry API spec, eg. .../_catalog and .../tags/list both don't work and get 404s,\nso may need to experiment more than with your own private docker registry\n\n\nExamples:\n\n# Get all the tags for a given repository called 'harisekhon/hbase':\n\n    ${0##*/} /repositories/harisekhon/hbase/tags\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\n#url_base=\"https://registry.hub.docker.com/v2\"\nurl_base=\"https://hub.docker.com/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nuser=\"${DOCKERHUB_USERNAME:-${DOCKERHUB_USER:-${DOCKER_USERNAME:-${DOCKER_USER:-}}}}\"\nPASSWORD=\"${DOCKERHUB_PASSWORD:-${DOCKERHUB_TOKEN:-${DOCKER_PASSWORD:-${DOCKER_TOKEN:-}}}}\"\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nfi\nexport PASSWORD\n\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path#https:\\/\\/registry.hub.docker.com}\"\nurl_path=\"${url_path#https:\\/\\/hub.docker.com}\"\n# replace // with /\nurl_path=\"${url_path//\\/\\/\\//\\/}\"\nurl_path=\"${url_path#/v2}\"\nurl_path=\"${url_path##/}\"\n\nif [ -n \"${PASSWORD:-}\" ]; then\n    # since DockerHub has many different API addresses it's easier to use JWT which isn't limited to a predefined service address\n    JWT=1\n    if [ -n \"${JWT:-}\" ]; then\n        #output=\"$(\"$srcdir/../bin/curl_auth.sh\" https://hub.docker.com/v2/users/login/ \\\n        output=\"$(curl https://hub.docker.com/v2/users/login/ \\\n                       -X POST \\\n                       \"${CURL_OPTS[@]}\" \\\n                       -d '{\"username\": \"'\"$user\"'\", \"password\": \"'\"$PASSWORD\"'\"}'\n        )\"\n        token=\"$(jq -r .token <<< \"$output\")\"\n        if [ -n \"${DOCKERHUB_2FA_CODE:-}\" ]; then\n            output=\"$(curl https://hub.docker.com/v2/users/2fa-login/ \\\n                           -X POST \\\n                           \"${CURL_OPTS[@]}\" \\\n                           -d '{\"login_2fa_token\": \"'\"$token\"'\", \"code\": \"'\"$DOCKERHUB_2FA_CODE\"'\"}'\n            )\"\n            token=\"$(jq -r .token <<< \"$output\")\"\n        fi\n        # automatically picked up by curl_auth.sh further down\n        export JWT_TOKEN=\"$token\"\n    else\n        # OAuth2\n        output=\"$(\"$srcdir/../bin/curl_auth.sh\" https://auth.docker.io/token -X GET \\\n                                         -H 'Content-Type: application/x-www-form-urlencoded' \\\n                                         -H 'Www-Authenticate: Bearer realm=\"https://auth.docker.io/token\",service=\"hub.docker.com\"' \\\n                                         -d \"grant_type=password&access_type=online&client_id=${0##*}&service=hub.docker.com\" # alternative: registry.docker.io\n                                         #-d \"grant_type=password&access_type=online&client_id=${0##*}&service=hub.docker.io&username=$user&password=$PASSWORD\"\n                                         #-H 'Www-Authenticate: Bearer realm=\"https://auth.docker.io/token\",service=\"hub.docker.com\",scope=\"repository:myuser/myimage:pull,push\"'\n        )\"\n        token=\"$(jq -r .access_token <<< \"$output\")\"\n        export TOKEN=\"$token\"\n    fi\n    if [ -z \"$token\" ] || [ \"$token\" = null ]; then\n        die \"Authentication failed: $output\"\n    fi\n    \"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nelse\n    # proceed without authentication\n    curl \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nfi\n"
  },
  {
    "path": "docker/dockerhub_build_status.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-03 14:21:19 +0100 (Fri, 03 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gets last build status for a DockerHub repo\n#\n# eg.\n#\n# dockerhub_build_status.sh harisekhon/nagios-plugins | jq\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"Gets last build status for a DockerHub repo\"\n\n# shellcheck disable=SC2034\nusage_args=\"<user/repo>\"\n\nif [ $# -lt 1 ]; then\n    usage\nfi\n\nrepo=\"$1\"\n\ncurl -sSL \"https://hub.docker.com/api/build/v1/source?image=$repo\"\n"
  },
  {
    "path": "docker/dockerhub_list_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: centos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-14 15:43:04 +0100 (Mon, 14 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists tags for a given DockerHub image using the DockerHub API\n\nExample:\n\n    ${0##*/} centos\n\n    ${0##*/} ubuntu\n\n    ${0##*/} harisekhon/hbase\n\n\nSee also:\n\n- dockerhub_show_tags.py in the DevOps Python tools repo:  https://github.com/HariSekhon/DevOps-Python-tools\n\n    ddockerhub_show_tags.py harisekhon/hbase\n\n- Skopeo\n\n    skopeo inspect docker://harisekhon/hbase | jq -r '.RepoTags[]'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"repo/image [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_image=\"$1\"\nshift || :\n\nif ! [[ \"$repo_image\" =~ / ]]; then\n    repo_image=\"library/$repo_image\"\nfi\n\nget_tags(){\n    local url_path=\"$1\"\n    local output\n    shift || :\n    output=\"$(\"$srcdir/dockerhub_api.sh\" \"$url_path\" \"$@\")\"\n    jq -r '.results[].name' <<< \"$output\"\n    next=\"$(jq -r .next <<< \"$output\")\"\n    if [ -n \"$next\" ] && [ \"$next\" != null ]; then\n        get_tags \"$next\"\n    fi\n}\n\nget_tags \"/repositories/$repo_image/tags\" \"$@\"\n"
  },
  {
    "path": "docker/dockerhub_list_tags_by_last_updated.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: centos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-14 15:43:04 +0100 (Mon, 14 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists tags for a given DockerHub image using the DockerHub API, sorted by last updated timestamp descending (newest at the top)\n\nExample:\n\n    ${0##*/} centos\n\n    ${0##*/} ubuntu\n\n    ${0##*/} harisekhon/hbase\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"repo/image [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_image=\"$1\"\nshift || :\n\nif ! [[ \"$repo_image\" =~ / ]]; then\n    repo_image=\"library/$repo_image\"\nfi\n\nget_tags(){\n    local url_path=\"$1\"\n    local output\n    shift || :\n    output=\"$(\"$srcdir/dockerhub_api.sh\" \"$url_path\" \"$@\")\"\n    jq -r '.results | sort_by(.last_updated) | reverse | .[] | [.name, .last_updated] | @tsv' <<< \"$output\"\n    next=\"$(jq -r .next <<< \"$output\")\"\n    if [ -n \"$next\" ] && [ \"$next\" != null ]; then\n        get_tags \"$next\"\n    fi\n}\n\nget_tags \"/repositories/$repo_image/tags\" \"$@\" |\nsort -k2r |\ncolumn -t\n"
  },
  {
    "path": "docker/dockerhub_repo_set_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-04-22 17:36:34 +0100 (Fri, 22 Apr 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the description summary of a DockerHub.com repo via the DockerHub API\n\nUseful to sync GitHub repo description to DockerHub repo description\n\nUses the adajcent script dockerhub_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nExample:\n\n    ${0##*/} HariSekhon/my-repo    my new description\n\n\nIf no second arg is given, will read repo description from standard input\n\n    echo <description> | ${0##*/} HariSekhon/my-repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner/repo> [<description>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nowner_repo=\"$1\"\ndescription=\"${*:2}\"\n\nif [ $# -lt 2 ]; then\n    echo \"reading repo summary description from stdin\" >&2\n    description=\"$(cat)\"\nfi\n\ntimestamp \"Setting DockerHub repo '$owner_repo' description to '$description'\"\n\n# don't URL encode this as it's inside JSON\n#description=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$description\")\"\n# just strip quotes to protect the JSON\ndescription=\"${description//\\\"/}\"\n\n\"$srcdir/dockerhub_api.sh\" \"/repositories/$owner_repo\" -X PATCH --data \"{ \\\"description\\\": \\\"$description\\\" }\"\n"
  },
  {
    "path": "docker/dockerhub_repo_set_readme.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-04-22 17:36:34 +0100 (Fri, 22 Apr 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the full README description of a DockerHub.com repo via the DockerHub API\n\nUseful to sync GitHub README to DockerHub README\n\nUses the adajcent script dockerhub_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nExample:\n\n    ${0##*/} HariSekhon/my-repo    my new readme description\n\n\nIf no second arg is given, will read repo description from standard input\n\n    cat README.md | ${0##*/} HariSekhon/my-repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner/repo> [<description>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nowner_repo=\"$1\"\ndescription=\"${*:2}\"\n\nif [ $# -lt 2 ]; then\n    echo \"reading repo full description from stdin\" >&2\n    description=\"$(cat)\"\nfi\n\ntimestamp \"Setting DockerHub repo '$owner_repo' description to '$description'\"\n\n# don't URL encode this as it's inside JSON\n#description=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$description\")\"\n# just strip quotes to protect the JSON\ndescription=\"${description//\\\"/}\"\n\n\"$srcdir/dockerhub_api.sh\" \"/repositories/$owner_repo\" -X PATCH --data \"{ \\\"full_description\\\": \\\"$description\\\" }\"\n"
  },
  {
    "path": "docker/dockerhub_search.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-14 16:14:36 +0100 (Mon, 14 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTool to search DockerHub repos and return a configurable number of results using the Docker API\n\nMimics 'docker search' results format but more flexible\n\nOlder Docker CLI didn't support configuring the returned number of search results and always returned 25:\n\nhttps://github.com/docker/docker/issues/23055\n\nSee also:\n\n    dockerhub_search.py\n\nin the DevOps Python tools repo - https://github.com/HariSekhon/DevOps-Python-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"search terms\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery=\"\"\nverbose=0\nlimit=25\n\nuntil [ $# -lt 1 ]; do\n    case $1 in\n    -h|--help)  usage\n                ;;\n   -l|--limit)  limit=\"$2\"\n                shift || :\n                ;;\n -v|--verbose)  verbose=1\n                ;;\n           -*)  usage \"unknown argument: $1\"\n                ;;\n            *)  query+=\"%20$1\"\n                ;;\n    esac\n    shift || :\ndone\n\nif ! is_int \"$limit\"; then\n    usage \"--num is not an integer\"\nfi\n\nquery=\"${query#%20}\"\n\npage=1\n\n# starter value, will be overriden on first iteration\nnum_pages=100\n\nresults=0\n\nprintf '%-30s   %-45s   %-7s   %-8s   %-10s\\n' \"NAME\" \"DESCRIPTION\" \"STARS\" \"OFFICIAL\" \"AUTOMATED\"\n\nwhile [ $results -lt \"$limit\" ] &&\n      [ \"$page\" -le \"$num_pages\" ]; do\n    output=\"$(curl -sSL --fail --connect-timeout 3 \"https://index.docker.io/v1/search?q=$query&page=${page}&n=100\")\"\n    num_pages=\"$(jq -r .num_pages <<< \"$output\")\"\n    ((page+=1))\n    while read -r name stars official automated description; do\n        ((results += 1))\n        if [ $results -gt \"$limit\" ]; then\n            break 2\n        fi\n        if [ \"${#description}\" -gt 45 ]; then\n            description=\"${description:0:42}...\"\n        fi\n        if [ \"$official\" = true ]; then\n            official=\"[OK]\"\n        else\n            official=\"\"\n        fi\n        if [ \"$automated\" = true ]; then\n            automated=\"[OK]\"\n        else\n            automated=\"\"\n        fi\n        printf '%-30s   %-45s   %-7s   %-8s   %-10s\\n' \"$name\" \"$description\" \"$stars\" \"$official\" \"$automated\"\n    done < <(jq -r '.results[] | [.name, .star_count, .is_official, .is_automated, .description] | @tsv' <<< \"$output\")\ndone\n\nif [ $verbose = 1 ]; then\n    echo\n    echo \"Results Shown: $results\"\n    echo \"Total Results: $(jq -r .num_results <<< \"$output\")\"\nfi\n"
  },
  {
    "path": "docker/quay_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /users/harisekhon | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 23:43:00 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Quay.com API\n\nRequires \\$QUAY_TOKEN to be set in the environment\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up your OAuth2 access token here:\n\n    https://quay.io/user/<username>?tab=robots\n\n\nAPI Reference:\n\n    https://docs.quay.io/api/\n\n\nAPI Explorer:\n\n    https://docs.quay.io/api/swagger/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://quay.io/api/v1\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined \"QUAY_TOKEN\"\n\nexport TOKEN=\"$QUAY_TOKEN\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path//https:\\/\\/quay.io\\/api\\/v1/}\"\nurl_path=\"${url_path##/}\"\n\n\"$srcdir/../bin/curl_auth.sh\" -L \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "docker-compose/circleci.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-13 18:59:29 +0000 (Mon, 13 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://circleci.com/docs/2.0/runner-installation-docker/\n\n# Set up runner token:\n#\n#   circleci runner resource-class create <namespace>/<resource> \"description\" --generate-token\n#\n# eg.\n#\n#   circleci runner resource-class create harisekhon/docker-runner \"Docker Runner\" --generate-token\n\n# For a much more advanced setup ready to go on Kubernetes with auto-scaling, see\n#\n#   https://github.com/HariSekhon/Kubernetes-configs\n\nversion: '2.2'\n\nservices:\n  circleci:\n    image: circleci/runner:launch-agent\n    # or replace with an image with built-in dependencies - see https://github/harisekhon/Dockerfiles\n    #image: harisekhon/circleci-runner\n    environment:\n      CIRCLECI_API_TOKEN: $CIRCLECI_API_TOKEN     # use the token output by the 'runner resource-class create' command above\n      CIRCLECI_RESOURCE_CLASS: ${CIRCLECI_RESOURCE_CLASS:-harisekhon/docker-runner}  # <namespace>/<resource>\n"
  },
  {
    "path": "docker-compose/concourse.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-21 11:14:07 +0000 (Sat, 21 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# based off concourse quickstart\n#\n# https://concourse-ci.org/docker-compose.yml\n\nversion: '3'\n\nservices:\n  concourse-db:\n    hostname: concourse-db\n    image: postgres\n    environment:\n      POSTGRES_DB: concourse\n      POSTGRES_PASSWORD: ${PGPASSWORD:-concourse_password}\n      POSTGRES_USER: ${PGUSER:-concourse_user}\n      PGDATA: /database\n\n  concourse:\n    hostname: concourse\n    image: concourse/concourse:${VERSION:-6.0}\n    command: quickstart\n    privileged: true\n    depends_on: [concourse-db]\n    ports:\n      - ${CONCOURSE_PORT:-8080}:8080\n    #volumes:\n    #  - .:/pwd\n    environment:\n      CONCOURSE_POSTGRES_HOST: concourse-db\n      CONCOURSE_POSTGRES_USER: ${PGUSER:-concourse_user}\n      CONCOURSE_POSTGRES_PASSWORD: ${PGPASSWORD:-concourse_password}\n      CONCOURSE_POSTGRES_DATABASE: concourse\n      CONCOURSE_EXTERNAL_URL: http://${CONCOURSE_HOST:-localhost}:${CONCOURSE_PORT:-8080}\n      CONCOURSE_ADD_LOCAL_USER: ${CONCOURSE_USER:-test}:${CONCOURSE_PASSWORD:-test}\n      CONCOURSE_MAIN_TEAM_LOCAL_USER: ${CONCOURSE_USER:-test}\n      CONCOURSE_WORKER_BAGGAGECLAIM_DRIVER: overlay\n#    healthcheck:\n#      # doesn't have curl\n#      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080\"]\n#      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n#      test: \"${DOCKER_HEALTHCHECK:-curl -f http://localhost:8080}\"\n#      interval: 30s\n#      #start_period: 30s  # version 3.4+\n#      timeout: 10s\n#      retries: 5\n"
  },
  {
    "path": "docker-compose/gerrit.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-08 13:36:32 +0100 (Wed, 08 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nversion: '3'\n\nservices:\n  gerrit:\n    hostname: gerrit\n    image: gerritcodereview/gerrit:${VERSION:-latest}\n    ports:\n      - 8080:8080\n      - 29418:29418\n"
  },
  {
    "path": "docker-compose/gocd.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-27 17:37:46 +0000 (Fri, 27 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nversion: '3'\n\nservices:\n  gocd-server:\n    hostname: gocd-server\n    image: gocd/gocd-server:v${VERSION:-20.2.0}\n    ports:\n      - 8153:8153\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8153/go/api/v1/health\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:8153/go/api/v1/health}\"\n      interval: 30s\n      #start_period: 30s  # version 3.4+\n      timeout: 10s\n      retries: 5\n    #environment:\n    #  CONFIG_GIT_REPO: ${CONFIG_GIT_REPO:-https://github.com/HariSekhon/DevOps-Bash-tools}\n    #  CONFIG_GIT_BRANCH: ${CONFIG_GIT_BRANCH:-master}\n\n  # XXX: containers should be called gocd-agent as gocd.sh uses this to count the number of expected agents and wait for them to connect before authorizing them\n  gocd-agent:\n    # XXX: hostname must be set to gocd-agent to be recognized as an expected agent\n    hostname: gocd-agent\n    image: gocd/gocd-agent-ubuntu-18.04:v${VERSION:-20.2.0}\n    ports:\n      - 8152:8152\n    # needs root to be able to install OS package dependencies - 'sudo' not installed\n    user: root:root\n    environment:\n      GO_SERVER_URL: http://gocd-server:8153/go\n      AGENT_STARTUP_ARGS: -Dgo.agent.status.api.bind.host=0.0.0.0\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8152/health/v1/isConnectedToServer\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:8152/health/v1/isConnectedToServer}\"\n      interval: 30s\n      #start_period: 30s  # version 3.4+\n      timeout: 10s\n      retries: 5\n    restart: on-failure\n    links:\n      - gocd-server\n\n#  gocd-agent2:\n#    image: gocd/gocd-agent-ubuntu-18.04:v${VERSION:-20.2.0}\n#    # needs root to be able to install OS package dependencies - 'sudo' not installed\n#    user: root:root\n#    environment:\n#      GO_SERVER_URL: http://gocd-server:8153/go\n#    restart: on-failure\n#    links:\n#      - gocd-server\n"
  },
  {
    "path": "docker-compose/jenkins.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-28 01:49:38 +0000 (Sat, 28 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/docker/blob/master/README.md\n\nversion: '3'\n\nservices:\n  jenkins-server:\n    hostname: jenkins-server\n    # official repo abandoned - too old for plugins to load - unusable\n    #image: jenkins:2.60.3\n    # community version\n    #image: jenkins/jenkins:${VERSION:-2.228}\n    image: jenkins/jenkins:${VERSION:-lts}\n    # jenkinsci/jenkins is deprecated in favour of jenkins/jenkins but jenkins org doesn't have blueocean\n    # this is based off Alpine - Jenkinsfile is for Debian/Ubuntu systems like standard Jenkins, just use stock image with plugin\n    #image: jenkinsci/blueocean:${VERSION:-1.22.0}\n    ports:\n      - ${JENKINS_PORT:-8080}:8080\n      - 50000:50000\n    # doesn't auto-install, done explicitly by jenkins.sh now\n    #volumes:\n    #  - ./setup/jenkins-plugins.txt:/usr/share/jenkins/ref/plugins.txt\n    user: root:root\n    # gets 403 without -Djenkins.install.runSetupWizard=false\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/login\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:8080/login}\"\n      interval: 30s\n      start_period: 30s  # version 3.4+\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n    environment:\n      # skips creating admin user or prompting to install plugins\n      #JAVA_OPTS: -Dhudson.footerURL=https://github.com/harisekhon -Djenkins.install.runSetupWizard=false\n      JAVA_OPTS: -Dhudson.footerURL=https://github.com/harisekhon\n#  jenkins-agent:\n#    hostname: jenkins-agent\n#    # doesn't have an lts tag at the moment\n#    image: jenkins/inbound-agent:latest\n#    restart: unless-stopped\n#    environment:\n#      JENKINS_AGENT_NAME: agent1\n#      JENKINS_URL: http://jenkins-server:8080\n#      # need to get this from the web UI, then uncomment and set in environment - it's not visible in the Rest API :'-(\n#      JENKINS_SECRET: ${JENKINS_SECRET:-none}\n"
  },
  {
    "path": "docker-compose/keycloak.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-09 17:28:12 +0000 (Wed, 09 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.keycloak.org/getting-started/getting-started-docker\n\nversion: '3'\n\nservices:\n  keycloak:\n    hostname: keycloak\n    command:\n      - start-dev\n    image: quay.io/keycloak/keycloak:${VERSION:-17.0.0}\n    ports:\n      - ${KEYCLOAK_PORT:-8080}:8080\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/admin\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:8080/admin}\"\n      interval: 30s\n      start_period: 30s  # version 3.4+\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n    environment:\n      KEYCLOAK_ADMIN: ${KEYCLOAK_ADMIN:-admin}\n      KEYCLOAK_ADMIN_PASSWORD: ${KEYCLOAK_ADMIN_PASSWORD:-admin}\n"
  },
  {
    "path": "docker-compose/octopus-deploy.env",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-08-04 15:42:17 +0100 (Thu, 04 Aug 2022)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://octopus.com/docs/installation/octopus-server-linux-container/docker-compose-linux\n#\n# https://hub.docker.com/r/octopusdeploy/octopusdeploy\n\n\n# Define the password for the SQL database. This also must be set in the DB_CONNECTION_STRING value.\n#\n# XXX: warning - do not change this password after container's initial creation, otherwise results in the following error:\n#\n#   Logon       Error: 18456, Severity: 14, State: 8.\n#   Logon       Login failed for user 'sa'. Reason: Password did not match that for the login provided. [CLIENT: <ip_x.x.x.x>]\n#\n# Would need to delete the containers and docker volumes to reset\n#\n# 2nd startup breaks similarly to above if doing this, can't set dynamically because it'll change each boot and mismatch\n#SA_PASSWORD=\"$(openssl rand 16 | base64)\"\n#\n# annoyingly long to figure out and type type\n#SA_PASSWORD=\"$HOSTNAME-$USER-$PWD\"\n#\n# python-dotenv fails to parse this\n#echo \"Generated password: '$SA_PASSWORD'\"\n#\n# leaving blank results in:\n#\n# blank password results in error:\n#\n#   ERROR: Unable to set system administrator password: Password validation failed. The password does not meet SQL Server password policy requirements because it is too short. The password must be at least 8 characters..\n#\n#SA_PASSWORD=\n#\n# trying ease of use results in this error:\n#\n#   ERROR: Unable to set system administrator password: Password validation failed. The password does not meet SQL Server password policy requirements because it is not complex enough. The password must be at least 8 characters long and contain characters from three of the following four sets: Uppercase letters, Lowercase letters, Base 10 digits, and Symbols..\n#\n#SA_PASSWORD=testdbpass\n#SA_PASSWORD=ComplexDBPassw0rd\n# set to the same as what should have been the default password anyway according to https://hub.docker.com/r/octopusdeploy/octopusdeploy\nSA_PASSWORD=0ct@pu55!\n\n# Tag for the Octopus Deploy Server image. Use \"latest\" to pull the latest image or specify a specific tag\n#OCTOPUS_SERVER_TAG=latest\n\n# Sql Server image. Set this variable to the version you wish to use. Default is to use the latest.\n#SQL_IMAGE=mcr.microsoft.com/mssql/server\n\n# The default created user username for login to the Octopus Server\n#\n# leaving blank results in this error when ADMIN_PASSWORD is set:\n#\n#   ERROR: A new admin password was provided but no admin username was specified.\n#\n#ADMIN_USERNAME=\n# should have been the default user anyway according to https://hub.docker.com/r/octopusdeploy/octopusdeploy\nADMIN_USERNAME=admin\n\n# It is highly recommended this value is changed as it's the default user password for login to the Octopus Server\n#\n# ease of use results in this error:\n#\n#   Octopus.Shared.ControlledFailureException: The password was too weak. Please try including a mix of numbers, uppercase and lowercase letters, and special characters.\n#\n#ADMIN_PASSWORD=admin\n#\n# too much typing but what can you do...\n# doesn't work - env var is taken literally and not evaluated\n#ADMIN_PASSWORD=\"$SA_PASSWORD\"\n#\n# empty results in:\n#\n#   There are no authentication providers enabled. Learn about enabling authentication providers\n#\n#ADMIN_PASSWORD=\n# should have been the default password anyway according to https://hub.docker.com/r/octopusdeploy/octopusdeploy\nADMIN_PASSWORD=0ct@pu55!\n\n# Email associated with the default created user. If empty will default to octopus@example.local\nADMIN_EMAIL=\n\n# Accept the Microsoft Sql Server Eula found here: https://go.microsoft.com/fwlink/?linkid=857698\n#ACCEPT_EULA=Y\n\n# Use of this Image means you must accept the Octopus Deploy Eula found here: https://octopus.com/company/legal\n#ACCEPT_OCTOPUS_EULA=Y\n\n# Unique Server Node Name - If left empty will default to the machine Name\nOCTOPUS_SERVER_NODE_NAME=\n\n# Database Connection String. If using database in sql server container, it is highly recommended to change the password.\n#\n# set in the docker compose file now to deduplicate via reuse of environment variable\n#DB_CONNECTION_STRING=Server=db,1433;Database=OctopusDeploy;User=sa;Password=THE_SA_PASSWORD_DEFINED_ABOVE\n\n# Your License key for Octopus Deploy. If left empty, it will try and create a free license key for you\nOCTOPUS_SERVER_BASE64_LICENSE=\n\n# Octopus Deploy uses a master key for encryption of your databse. If you're using an external database that's already been setup for Octopus Deploy,\n# you can supply the master key to use it.\n# If left blank, a new master key will be generated with the database creation.\n# Create a new master key with the command: openssl rand 16 | base64\n#\n#\n# if you don't set a Master key it seems to get generated each boot, resulting in this error:\n#\n#   Octopus.Core.Security.MasterKey.MasterKeyMismatchException: Failed to decrypt the Octopus Server certificate. This usually indicates the wrong master key is being used to read encrypted data.\n#\n#\n# must be a base64 string, otherwise results in this error:\n#\n#   octopus-server_1    | The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.\n#   octopus-server_1    | System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.\n#\n#MASTER_KEY=TestKey_only_safe_if_limited_to_localhost\n#\n#\n# results in error:\n#\n#   octopus-server_1    | Specified key is not a valid size for this algorithm.\n#   octopus-server_1    | System.Security.Cryptography.CryptographicException: Specified key is not a valid size for this algorithm.\n#\n# echo TestKey_only_safe_if_limited_to_localhost | base64\n#MASTER_KEY=VGVzdEtleV9vbmx5X3NhZmVfaWZfbGltaXRlZF90b19sb2NhbGhvc3QK\n#\n# openssl rand 16 | base64\nMASTER_KEY=Wl3rNdWWe2VLVYAXY/sTsA==\n\n# The API Key to set for the administrator. If this is set and no password is provided then a service account user will be created.\n# If this is set and a password is also set then a standard user will be created.\n#\n# NOTE: There is a known issue when providing both the ADMIN_PASSWORD and ADMIN_API_KEY that prevents the Administrator from logging in.\n# This will be resolved in a future version of Octopus. See: https://github.com/OctopusDeploy/Issues/issues/6629 for further details.\n#\nADMIN_API_KEY=\n\n# Docker-In-Docker is used to support worker container images. It can be disabled by setting DISABLE_DIND to Y.\n# The container only requires the privileged setting if DISABLE_DIND is set to N.\nDISABLE_DIND=Y\nPRIVILEGED=false\n\n# Octopus can be run either as the user root or as octopus.\nOCTOPUS_USER=octopus\n"
  },
  {
    "path": "docker-compose/octopus-deploy.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2022-08-04 14:53:17 +0100 (Thu, 04 Aug 2022)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://octopus.com/docs/installation/octopus-server-linux-container/docker-compose-linux\n\n# https://hub.docker.com/r/octopusdeploy/octopusdeploy\n\n# Reset:\n#\n#   docker rm -f bash-tools_octopus-server_1 bash-tools_octopus-db-mssql_1\n#   docker volume ls -q | xargs docker volume rm\n\n---\nversion: '3'\nservices:\n  octopus-db-mssql:\n    image: mcr.microsoft.com/mssql/server\n    environment:\n      SA_PASSWORD: ${SA_PASSWORD}\n      ACCEPT_EULA: 'Y'\n    ports:\n      - 1401:1433\n    healthcheck:\n      test: [\"CMD\", \"/opt/mssql-tools/bin/sqlcmd\", \"-U\", \"sa\", \"-P\", \"${SA_PASSWORD}\", \"-Q\", \"select 1\"]\n      interval: 10s\n      retries: 10\n    volumes:\n      - sqlvolume:/var/opt/mssql\n  octopus-server:\n    image: octopusdeploy/octopusdeploy:${OCTOPUS_SERVER_TAG:-latest}\n    privileged: ${PRIVILEGED}  # needs true if DISABLE_DIND=N in environment section\n    #user: octopus  # or 'root'\n    user: ${OCTOPUS_USER}\n    environment:\n      ACCEPT_EULA: 'Y'\n      #OCTOPUS_SERVER_NODE_NAME: ${OCTOPUS_SERVER_NODE_NAME}\n      DB_CONNECTION_STRING: Server=octopus-db-mssql,1433;Database=OctopusDeploy;User=sa;Password=${SA_PASSWORD}\n      ADMIN_USERNAME: ${ADMIN_USERNAME}  # default: admin\n      ADMIN_PASSWORD: ${ADMIN_PASSWORD}  # default: 0ct@pu55! - but instead blank seems to disable authentication provider, see env file for error message\n      ADMIN_EMAIL: ${ADMIN_EMAIL}  # defaults to octopus@example.local\n      OCTOPUS_SERVER_BASE64_LICENSE: ${OCTOPUS_SERVER_BASE64_LICENSE}  # default: creates a free 12 month license\n      MASTER_KEY: ${MASTER_KEY}  # if blank will generate one when the DB is created\n      ADMIN_API_KEY: ${ADMIN_API_KEY}\n      DISABLE_DIND: ${DISABLE_DIND}\n    ports:\n      - 8080:8080\n      - 11111:10943\n    depends_on:\n      - octopus-db-mssql\n    volumes:\n      - repository:/repository\n      - artifacts:/artifacts\n      - taskLogs:/taskLogs\n      - cache:/cache\n      - import:/import\nvolumes:\n  repository:\n  artifacts:\n  taskLogs:\n  cache:\n  import:\n  sqlvolume:\n"
  },
  {
    "path": "docker-compose/prometheus.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 16:42:37 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nversion: '3'\n\nservices:\n  prometheus:\n    hostname: prometheus\n    image: prom/prometheus:${VERSION:-latest}\n    ports:\n      - ${PROMETHEUS_PORT:-9090}:9090\n    volumes:\n      - ${PROMETHEUS_CONFIG:-./prometheus.yml}:/etc/prometheus/prometheus.yml\n    #environment:\n#    healthcheck:\n#      # doesn't have curl\n#      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:9090\"]\n#      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n#      test: \"${DOCKER_HEALTHCHECK:-curl -f http://localhost:9090}\"\n#      interval: 30s\n#      #start_period: 30s  # version 3.4+\n#      timeout: 10s\n#      retries: 5\n"
  },
  {
    "path": "docker-compose/teamcity.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-24 15:40:47 +0000 (Tue, 24 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# XXX: you must set an external RDBMS for production deployments (I use Cloud SQL)\n\nversion: '3'\n\nservices:\n  teamcity-server:\n    hostname: teamcity-server\n    # 2GB docker image\n    image: jetbrains/teamcity-server:${VERSION:-2020.2.1}\n    ports:\n      - 8111:8111\n    restart: unless-stopped\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8111\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:8111}\"\n      interval: 30s\n      start_period: 30s\n      timeout: 10s\n      retries: 5\n    environment:\n      #TEAMCITY_SERVER_MEM_OPTS: \"-Xmx2g -XX:MaxPermSize=270m -XX:ReservedCodeCacheSize=350m\"\n      TEAMCITY_SERVER_MEM_OPTS: \"-Xmx512m\"\n    volumes:\n      # mount existing TeamCity CI configs - this works from a VCS integrated repo checkout too\n      #- ../../teamcity/.teamcity/HariSekhonGitHubProjects:/data/teamcity_server/datadir/config/projects/HariSekhonGitHubProjects\n      # for server.xml override:\n      #- teamcity/conf:/opt/teamcity/conf\n      #- ./teamcity/teamcity-database.properties:/data/teamcity_server/datadir/config/database.properties\n      - $HOME/teamcity/server/datadir:/data/teamcity_server/datadir\n      - $HOME/teamcity/server/logs:/opt/teamcity/logs\n  teamcity-agent:\n    hostname: teamcity-agent\n    # 1.3GB docker image\n    image: jetbrains/teamcity-agent:${VERSION:-2020.2.1}\n    # need the linux-sudo image for Docker-in-Docker - but this doesn't work due to iptables initialization error\n    #image: jetbrains/teamcity-agent:${VERSION:-2020.2.1}-linux-sudo\n    ports:\n      - 9090:9090\n    # needs root to be able to install OS package dependencies - 'sudo' not installed unless using the linux-sudo suffix, which is a lot bigger and similar result\n    # Since version 2020.1, TeamCity agent Docker images run under a non-root user by default\n    user: root:root\n    environment:\n      SERVER_URL: http://teamcity-server:8111\n      # AGENT_NAME is used to set the reported agent name and by teamcity.sh to automatically authorize the agent(s)\n      AGENT_NAME: agent1\n      # currently authorized via API in teamcity.sh\n      #AGENT_TOKEN:\n      # need the linux-sudo image for Docker-in-Docker\n      DOCKER_IN_DOCKER: start\n    healthcheck:\n      #test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:9090\"]\n      # export DOCKER_HEALTHCHECK=/bin/true in development to bypass healthcheck\n      test: \"${DOCKER_HEALTHCHECK:-curl http://localhost:9090}\"\n      interval: 30s\n      start_period: 30s\n      timeout: 10s\n      retries: 5\n    restart: unless-stopped\n    links:\n      - teamcity-server\n    volumes:\n      # persist the buildAgent.properties containing the agent id and autosaved authorization token\n      - $HOME/teamcity/agent1/conf:/data/teamcity_agent/conf\n      #\n      # checked out sources - this needs 3GB to run builds so needs to be mounted - the docker image has 60GB and the mounts appear to only get 1GB not properly mapped to host but must be stuck in Docker VM overlay so omitting it\n      - $HOME/teamcity/agent1/work:/opt/buildagent/work\n      #\n      - $HOME/teamcity/agent1/logs:/opt/buildagent/logs\n      #\n      # internal build agent caches\n      - $HOME/teamcity/agent1/system:/opt/buildagent/system\n      #\n      # plugins\n      - $HOME/teamcity/agent1/plugins:/opt/buildagent/plugins\n"
  },
  {
    "path": "docker-compose/wordpress.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2023-12-04 12:24:30 +0000 (Mon, 04 Dec 2023)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n---\nversion: '3.1'\nservices:\n  wordpress:\n    image: wordpress:${VERSION:-latest}\n    restart: always\n    ports:\n      - 8080:80\n    environment:\n      WORDPRESS_DB_HOST: db\n      WORDPRESS_DB_USER: exampleuser\n      WORDPRESS_DB_PASSWORD: examplepass\n      WORDPRESS_DB_NAME: exampledb\n    volumes:\n      - wordpress:/var/www/html\n  db:\n    image: mysql:5.7\n    restart: always\n    environment:\n      MYSQL_DATABASE: exampledb\n      MYSQL_USER: exampleuser\n      MYSQL_PASSWORD: examplepass\n      MYSQL_RANDOM_ROOT_PASSWORD: '1'\n    volumes:\n      - db:/var/lib/mysql\nvolumes:\n  wordpress:\n  db:\n"
  },
  {
    "path": "drone/drone_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 23:27:44 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#  args: /user | jq -C .\n#  args: /user/repos | jq -C .\n#  args: /repos/HariSekhon/DevOps-Bash-tools/builds | jq -C .\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Drone.io API\n\nCan specify \\$CURL_OPTS for options to pass to curl, or pass them as arguments to the script\n\nAutomatically handles authentication via environment variable \\$DRONE_TOKEN\n\n\nGet your personal access token here:\n\n    https://cloud.drone.io/account\n\n\nAPI Reference:\n\n    https://docs.drone.io/api/overview/\n\n\nExamples:\n\n\n# Get currently authenticated user:\n\n    ${0##*/} /user\n\n\n# List repos registered in Drone:\n\n    ${0##*/} /user/repos\n\n\n# List your Drone builds for a repo (case sensitive):\n\n    ${0##*/} /repos/{owner}/{repo}/builds\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://cloud.drone.io/api\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path##*:\\/\\/cloud.drone.io\\/api}\"\nurl_path=\"${url_path##/}\"\nurl_path=\"${url_path##api}\"\n\nexport TOKEN=\"$DRONE_TOKEN\"\n\n# this trick doesn't work, file descriptor is lost by next line\n#filedescriptor=<(cat <<< \"Private-Token: $DRONE_TOKEN\")\n# this works\n#curl \"${CURL_OPTS[@]}\" -H @<(cat <<< \"Authorization: Bearer $DRONE_TOKEN\") \"$url_base/$url_path\" \"$@\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "drone/drone_docker_runner.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-22 09:32:49 +0100 (Sat, 22 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a Drone.io CI Runner in Docker\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\ncheck_env_defined \"DRONE_SERVER\"\ncheck_env_defined \"DRONE_RPC_SECRET\"\n\nhelp_usage \"$@\"\n\ndocker run \\\n  -v /var/run/docker.sock:/var/run/docker.sock \\\n  -e DRONE_RPC_PROTO=https \\\n  -e DRONE_RPC_HOST=\"$DRONE_SERVER\" \\\n  -e DRONE_RPC_SECRET=\"$DRONE_RPC_SECRET\" \\\n  -e DRONE_RUNNER_CAPACITY=2 \\\n  -e DRONE_RUNNER_NAME=\"$HOSTNAME\" \\\n  -p 3000:3000 \\\n  --restart always \\\n  --name drone-runner \\\n  drone/drone-runner-docker:1\n"
  },
  {
    "path": "drone/drone_docker_server.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-22 09:32:49 +0100 (Sat, 22 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a Drone.io CI Server in Docker\n\nSee here to set up the GitHub integration credentials\n\nhttps://docs.drone.io/server/provider/github/\n\nNeed to set \\$DRONE_RPC_SECRET to some value for the Runners to authenticate to\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\ncheck_env_defined \"DRONE_GITHUB_CLIENT_ID\"\ncheck_env_defined \"DRONE_GITHUB_CLIENT_SECRET\"\ncheck_env_defined \"DRONE_RPC_SECRET\"\n\nhelp_usage \"$@\"\n\ndocker run \\\n  --volume=/var/lib/drone:/data \\\n  --env=DRONE_GITHUB_CLIENT_ID=\"$DRONE_GITHUB_CLIENT_ID\" \\\n  --env=DRONE_GITHUB_CLIENT_SECRET=\"$DRONE_GITHUB_CLIENT_SECRET\" \\\n  --env=DRONE_RPC_SECRET=\"$DRONE_RPC_SECRET\" \\\n  --env=DRONE_SERVER_HOST=\"localhost\" \\\n  --env=DRONE_SERVER_PROTO=\"https\" \\\n  --publish=80:80 \\\n  --publish=443:443 \\\n  --restart=always \\\n  --detach=true \\\n  --name=drone \\\n  drone/drone:1\n"
  },
  {
    "path": "gcp/.customize_environment",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-19 16:43:00 +0000 (Tue, 19 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#           G C P   C l o u d   S h e l l   C u s t o m i z a t i o n\n# ============================================================================ #\n\n# see log at /var/log/customize_environment\n\n# /google/devshell/customize_environment_done is touched when completed\n\n# called as root by GCP\n\nset -euxo pipefail\n\nbash_tools=\"$HOME/github/bash-tools\"\n\nif ! [ -d \"$bash_tools\" ]; then\n    parent_dir=\"${bash_tools%/*}\"\n    mkdir -pv \"$parent_dir\"\n    pushd \"$parent_dir\"\n    git clone https://github.com/HariSekhon/DevOps-Bash-tools \"$bash_tools\"\n    popd\nfi\n\nmkdir -pv ~/.cloudshell\n\n# suppress apt-get warnings about being ephemeral, that is what this script is for\ntouch ~/.cloudshell/no-apt-get-warning\n\npushd \"$bash_tools\"\n\ngit pull\n\n# not calling gcp-shell which also links because it will link config files to /root\n# instead just run 'make gcp-shell' one time to set up $USER's $HOME\nmake system-packages\n\npopd\n"
  },
  {
    "path": "gcp/.gcloudignore",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-09-28 09:58:44 +0100 (Mon, 28 Sep 2020)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           . g c l o u d i g n o r e\n# ============================================================================ #\n\n# https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore\n\n# ============================================\n# Ignore uploading these to Google Cloud Build, Cloud Functions, App Engine etc.\n\n#\n# there is a much bigger list in the adjacent .gitignore, some of which may be worth porting in to here too\n#\n# XXX: tested, cannot use blacklists like in .dockerignore because this breaks uploading anything, regardless of the blacklist coming before or after the !whitelist items\n#      use of either * or .* breaks Google Cloud Functions upload which end up with empty zips even with !main.py and !requirements.txt whitelist exclusions before or after\n#      so this definitely doesn't support the features of .gitignore and .dockerignore\n# *\n# .*\n\n# since .* doesn't work, unlike .dockerignore we must specify the common dot files to exclude below\n\n# .git seems to be ignored by default judging by cloud build uploaded tarball size\n.git/\n.hg/\n.svn/\n.gcloudignore\n.gitignore\n.dockerignore\n\n# descends directories matching basenames like .gitignore, doesn't need **/ prefix like .dockerignore\n\n# CI stuff you'd usuall commit but don't want to upload - there are more dot files with auth that wouldn't commit borrowed from massive adjacent .gitignore further down in the dotfiles section\nJenkinsfile\nazure-pipelines.y*ml\nbitbucket-pipelines.y*ml\nbuddy.y*ml\ncodefresh.y*ml\nshippable.y*ml\nwercker.y*ml\ngocd_config_repo.json\njenkins-job.xml\nhadolint.y*ml\nscalastyle_config.xml\nyamllint/\n# contains webhook URL which should not be committed publicly\nbuildkite-pipeline*.json\n\n# ========================================\n# Based on the massive adjacent .gitignore\n\ngit/\ngithub/\ngitolite*/\ngitroot/\nmercurial/\nhg/\nhgroot/\nsvn/\nsvnroot/\nsubversion/\n\ndist/\nnode_modules/\nvendor/\nlogs/\ngit/\ngithub/\nvagrant/\nvenv/\ndebs/\nrpms/\n\n*#*#\n*.a\n*.avi\n*.bak\n*.bak.*\n*.bin\n*.bkp\n*.class\n*.dump\n*.flv\n*.kdb\n*.lock\n*.local\n*.log\n*.macports-saved_*\n*.mp3\n*.mp4\n*.mpeg\n*.mpg\n*.o\n*.orig\n*.out\n*.part\n*.pyc\n*.pyo\n*.stderr\n*.stdout\n*.swo\n*.swp\n*.tmp\n*.wmv\n*~\ntmp.*\n~*\n\n# since .* doesn't work we must specify the common dot files to exclude below\n\n.*.local\n.*.env\n\n.a\n.activator/\n.adobe\n.agent.env\n.aliaslists\n.android/\n.anyconnect\n.ApacheDirectoryStudio/\n.atftp_history\n.atom/\n.audacious/\n.awless/\n.aws/\n.bash*\n.cache\n.cassandra/\n.cbq_history\n.ccm/\n.CFUserTextEncoding\n# Codefresh config contains API Key\n.cfconfig\n# Circle CI API Token stored in .circleci/cli.yml\n.circleci/\n.Codefresh/\n.compiz\n.conda/\n.config\n# contains repo_token / COVERALLS_REPO_TOKEN\n.coveralls.yml\n.cpan/\n.cpanm/\n.cups/\n.data\n.dbshell\n.dbus\n.dcos/\n.DCOPserver_*\n.devcenter/\n.docker*\n.dropbox/\n.DS_Store\n.dvdcss/\n# contains things like auth tokens\n.envrc*\n.env*\n.erlang.cookie\n.evolution\n.fluxbox/\n.flyrc\n.fontconfig\n.fseventsd\n.gaim/\n.gem\n.gimp-*\n.github_actions_runner/\n.gitk\n.gmvault/\n.gnome\n.gnome2\n.gnome2_private\n.gnupg*\n.gpg*\n.gradle\n.groovy\n.gstreamer-*\n.gvfs\n.htoprc\n.ICEauthority\n.IdeaIC*\n.idea\n.inkscape-etc/\n.ion3\n.ipython/\n.irb-save-history\n.irb_history\n# contains creds\n.iredisrc\n.ivy2/\n.jline-jython.history\n.kde\n.kodos\n.kube/\n.ldapvi_history\n.lesshst\n.links2\n.local\n.m2/\n.macports/\n.macports/\n.macromedia\n.matplotlib/\n.mcop\n.mcoprc\n.minikube/\n.minishift/\n.minishift.env\n.mozilla\n.mtools/\n.mysql_history\n.nbprofiler/\n.neo4j_shell_history\n.npm/\n.octave_hist\n.openoffice.org\n.openoffice.org2\n.oracle_jre_usage/\n.ovftool.ssldb\n.Qsync/\n.parallel/\n# PostgreSQL password file\n.pgpass\n# stores credentials\n.pig_history\n.pki\n.pentaho/\n.psql_history\n.pulse\n.puppet/\n.pwm3\n.PyCharm*/\n.pylint*\n.python*\n.python-eggs/\n.python_history\n.qt\n.qicon\n.qnicon\n# contains usernames and passwords\n.rabbitmqadmin.conf\n.RData\n.recently-used\n.recently-used.xbel\n.rediscli_history\n.Rhistory\n.rnd\n.rstudio-desktop/\n.rbenv/\n# 'rbenv local' $PWD version file\n.ruby-version\n.sbt/\n.scala_history\n.sdkman/\n# Semaphore CI - contains auth token\n.sem.yaml\n.serverauth.*\n.sh_history\n.Skype\n.snowsql/\n# Snowflake password stored in plaintext in here\n.snowsql/config\n.spark*\n.spark_history\n.Spotlight-*\n.spumux/\n.sqlite_history\n.sqlline/history\n.ssh/\n.ssh*\n.subversion/\n# stores creds\n.subversion/auth\n.svn\n.swatch_script.*\n.TemporaryItems\n.terraform.d/\n.terragrunt\n.themes\n.thumbnails\n.tilda/\n.tmux/\n.tomboy\n.tomboy.log\n.Trash\n.Trashes\n.travis/\n.vagrant\n.vagrant.d/\n.vboxclient*\n.vim/\n.viminfo\n.vnc/\n.wapi\n.wget-hsts\n.wine\n.wireshark-etc/\n.wireshark/\n.wmii*\n.Xauthority\n.xine\n.xmms\n.xsession-errors\n.zenmap-etc/pango/pangorc\n\nBox Documents/\nDesktop/\nDocuments/\nDownloads/\ndrive/\nDropbox/\neclipse/\nGoogle*Drive\nmbox\nMovies\nMusic\nQsync/\nPictures\nvagrant/\nvenv/\nVirtualBox VMs/\nVirtualBoxShared\nvisualvm*\nwordlists/\n\n*.doc\n*.docx\n*.msg\n*.pages\n*.ppt\n*.pptx\n*.rtf\n*.wpd\n*.wps\n*.xls\n*.xlsx\n"
  },
  {
    "path": "gcp/bigquery_foreach_dataset.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo \"dataset_id =  \\'{dataset}\\'\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 08:54:54 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExecute a command against all Google BigQuery dataset IDs in the current project\n\nCommand can contain {dataset} placeholder which will be replaced with the id for each dataset\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the dataset IDs and exit after the first iteration\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncommand_template=\"$*\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/bigquery_list_datasets.sh\" |\nwhile read -r dataset_id; do\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        hr\n        echo \"Dataset = $dataset_id\"\n        hr\n    fi >&2\n    #printf '%s\\t' \"$dataset_id\"\n    command=\"${command_template//\\{dataset_id\\}/$dataset_id}\"\n    command=\"${command//\\{dataset\\}/$dataset_id}\"\n    eval \"$command\"\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo\n    fi >&2\ndone\n"
  },
  {
    "path": "gcp/bigquery_foreach_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bigquery-public-data.github_repos echo \"project = {project}, dataset = {dataset} / schema = {schema}, table = {table}\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 08:54:54 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExecute a command against all Google BigQuery tables in a given dataset in the current project\n\nCommand can contain {project}, {dataset} / {schema} and {table} placeholders which will be replaced for each table\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the project/dataset/table names and exit after the first iteration\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project>.]<dataset> <command>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\n# validated in bigquery_list_tables.sh\ndataset=\"$1\"\nshift || :\n\ncommand_template=\"$*\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/bigquery_list_tables.sh\" \"$dataset\" |\nwhile read -r project dataset table; do\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        hr\n        echo \"Project = $project, Dataset/Schema = $dataset, Table = $table\"\n        hr\n    fi >&2\n    command=\"${command_template//\\{project\\}/$project}\"\n    command=\"${command//\\{dataset\\}/$dataset}\"\n    command=\"${command//\\{schema\\}/$dataset}\"\n    command=\"${command//\\{table\\}/$table}\"\n    eval \"$command\"\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo\n    fi >&2\ndone\n"
  },
  {
    "path": "gcp/bigquery_foreach_table_all_datasets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo \"project = {project}, dataset = {dataset} / schema = {schema}, table = {table}\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 15:18:52 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExecute a command against all Google BigQuery tables in all datasets in the current project\n\nCommand can contain {project}, {dataset} / {schema} and {table} placeholders which will be replaced for each table\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the dataset names and exit after the first iteration\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncommand_template=\"$*\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\n\"$srcdir/bigquery_list_datasets.sh\" |\nwhile read -r dataset; do\n    \"$srcdir/bigquery_foreach_table.sh\" \"$dataset\" \"$command_template\"\ndone\n"
  },
  {
    "path": "gcp/bigquery_generate_query_biggest_tables_across_datasets_by_row_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 10:11:27 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates an SQL query to find the Top 10 biggest tables by row count across all BigQuery datasets in the current GCP project\n\nPrints the SQL query to standard output. Pipe this to the 'bq query' command to execute\n\nRequires GCloud SDK to be installed and authorized\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nunion_all=\"$(NO_HEADING=1 \"$srcdir/bigquery_foreach_dataset.sh\" \"echo 'SELECT * FROM {dataset_id}.__TABLES__ UNION ALL'\")\"\n\nunion_all=\"${union_all%UNION ALL}\"\n\nquery=\"\nWITH ALL__TABLES__ AS (\n$union_all\n)\nSELECT\n  project_id,\n  dataset_id,\n  table_id,\n  row_count,\n  ROUND(size_bytes/pow(10,9),2) as size_gb,\n  TIMESTAMP_MILLIS(creation_time) AS creation_time,\n  TIMESTAMP_MILLIS(last_modified_time) as last_modified_time,\n  CASE\n    WHEN type = 1 THEN 'table'\n    WHEN type = 2 THEN 'view'\n  ELSE NULL\n  END AS type\nFROM\n  ALL__TABLES__\nORDER BY\n  row_count DESC,\n  size_gb DESC\nLIMIT 10;\n\"\n\necho \"$query\"\n"
  },
  {
    "path": "gcp/bigquery_generate_query_biggest_tables_across_datasets_by_size.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 10:11:27 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates an SQL query to find the Top 10 biggest tables by row count across all BigQuery datasets in the current GCP project\n\nPrints the SQL query to standard output. Pipe this to the 'bq query' command to execute\n\nRequires GCloud SDK to be installed and authorized\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nunion_all=\"$(NO_HEADING=1 \"$srcdir/bigquery_foreach_dataset.sh\" \"echo 'SELECT * FROM {dataset_id}.__TABLES__ UNION ALL'\")\"\n\nunion_all=\"${union_all%UNION ALL}\"\n\nquery=\"\nWITH ALL__TABLES__ AS (\n$union_all\n)\nSELECT\n  project_id,\n  dataset_id,\n  table_id,\n  ROUND(size_bytes/pow(10,9),2) as size_gb,\n  row_count,\n  TIMESTAMP_MILLIS(creation_time) AS creation_time,\n  TIMESTAMP_MILLIS(last_modified_time) as last_modified_time,\n  CASE\n    WHEN type = 1 THEN 'table'\n    WHEN type = 2 THEN 'view'\n  ELSE NULL\n  END AS type\nFROM\n  ALL__TABLES__\nORDER BY\n  size_gb DESC,\n  row_count DESC\nLIMIT 10;\n\"\n\necho \"$query\"\n"
  },
  {
    "path": "gcp/bigquery_list_datasets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 08:54:54 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the BigQuery datasets in the current GCP project, one per line\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#set +e\n#output=\"$(bq ls --quiet --headless --format=json)\"\n## shellcheck disable=SC2181\n#if [ $? != 0 ]; then\n#    echo \"$output\" >&2\n#    exit 1\n#fi\n#set -e\nbq ls --quiet --headless --format=json |\njq -r '.[].datasetReference.datasetId'\n"
  },
  {
    "path": "gcp/bigquery_list_tables.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bigquery-public-data.github_repos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 14:46:21 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all BigQuery tables in a given dataset by querying BigQuery's Information Schema for that dataset\n\nTo list tables from a dataset in another project, just prefix the project eg. <project>.<dataset>\n\nOutput Format:\n\n<project>   <dataset>   <table>\n\nFILTER environment variable will restrict to matching tables (matches against fully qualified table name <project>.<dataset>.<table>)\n\nLimited to 10,000 table names by default (increase max_rows in script if you have a bigger dataset than this)\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project>.]<dataset>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ndataset=\"$1\"\n\nif ! [[ \"$dataset\" =~ ^[[:alnum:]_.-]+$ ]]; then\n    die \"invalid dataset name given (must contain only alphanumeric, dots, dashes and underscores):  $dataset\"\nfi\n\n# XXX: you might need to edit this\nmax_rows=10000\n\nset +e\noutput=\"$(bq query --quiet --headless --format=prettyjson --max_rows \"$max_rows\" --nouse_legacy_sql 'select table_catalog, table_schema, table_name FROM `'\"$dataset\"'.INFORMATION_SCHEMA.TABLES`;')\"\n# shellcheck disable=SC2181\nif [ $? != 0 ]; then\n    echo \"$output\" >&2\n    exit 1\nfi\nset -e\njq -r '.[] | [.table_catalog, .table_schema, .table_name] | @tsv' <<< \"$output\" |\nwhile read -r project dataset table; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$project.$dataset.$table\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\t%s\\n' \"$project\" \"$dataset\" \"$table\"\ndone\n"
  },
  {
    "path": "gcp/bigquery_list_tables_all_datasets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 14:46:21 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all BigQuery tables in all datasets in the current project using the BigQuery Information Schema for each dataset\n\nOutput Format:\n\n<project>   <dataset>   <table>\n\nFILTER environment variable will restrict to matching tables (matches against fully qualified table name <dataset>.<schema>.<table>)\n\nLimited to 10,000 table names by default (increase max_rows in script if you have a bigger dataset than this)\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/bigquery_list_datasets.sh\" |\nwhile read -r dataset; do\n    \"$srcdir/bigquery_list_tables.sh\" \"$dataset\"\ndone\n"
  },
  {
    "path": "gcp/bigquery_table_row_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bigquery-public-data.github_repos.commits\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 15:32:21 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for a given BigQuery table\n\nOutput:\n\n<project>.<dataset>.<table>     <row_count>\n\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project>.]<dataset>.<table> <command>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# validated in bigquery_list_tables.sh\ntable=\"$1\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\noutput=\"$(bq query --quiet --headless --format=prettyjson --nouse_legacy_sql \"SELECT COUNT(*) AS row_count FROM \\`$table\\`;\")\"\n# shellcheck disable=SC2181\nif [ $? != 0 ]; then\n    echo \"$output\" >&2\n    exit 1\nfi\njq -r '.[].row_count' <<< \"$output\" |\nwhile read -r row_count; do\n    printf '%s\\t%s\\n' \"$table\" \"$row_count\"\ndone\n"
  },
  {
    "path": "gcp/bigquery_tables_row_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bigquery-public-data.github_repos\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 15:32:21 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for all BigQuery tables in the given dataset\n\nOutput:\n\n<project>.<dataset>.<table>     <row_count>\n\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project>.]<dataset>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# validated in bigquery_list_tables.sh\ndataset=\"$1\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\nexport NO_HEADING=1\n\n\"$srcdir/bigquery_foreach_table.sh\" \"$dataset\" \"$srcdir/bigquery_table_row_count.sh\" \"{project}.{dataset}.{table}\"\n"
  },
  {
    "path": "gcp/bigquery_tables_row_counts_all_datasets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-25 15:32:21 +0100 (Fri, 25 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for all BigQuery tables in all datasets in the current project\n\nOutput:\n\n<project>.<dataset>.<table>     <row_count>\n\n\nRequires GCloud SDK which must be configured and authorized for the project\n\nTested on Google BigQuery\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\nexport NO_HEADING=1\n\n\"$srcdir/bigquery_foreach_table_all_datasets.sh\" \"$srcdir/bigquery_table_row_count.sh\" \"{project}.{dataset}.{table}\"\n"
  },
  {
    "path": "gcp/firebase_foreach_project.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo project id is {id}, number is '{number}', name is '{name}'\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-27 00:58:45 +0700 (Thu, 27 Feb 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each Google Firebase project in the current account\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the project id/names and exit after the first iteration\n\nRequires Firebase CLI to be installed and configured be in the \\$PATH\n\nAll arguments become the command template\n\nThe following command template tokens are replaced in each iteration:\n\nProject ID:     {id}     {project_id}\nProject Number: {number} {project_number}\nProject Name:   {name}   {project_name}\n\n\neg.\n    ${0##*/} 'echo Firebase project has id {id}, number {number} and name {name}'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\nprojects=\"$(\n    firebase projects:list --json |\n    jq -r '\n        .result[] |\n        [.projectId, .projectNumber, .displayName] |\n        @tsv\n    '\n)\"\n\ntotal_projects=\"$(grep -c . <<< \"$projects\")\"\n\ni=0\n\nwhile read -r project_id project_number project_name; do\n    (( i += 1 ))\n    echo \"# ======================================================================================================== #\" >&2\n    echo \"# ($i/$total_projects) Firebase Project ID = $project_id -- Number = $project_number -- Name = $project_name\" >&2\n    echo \"# ======================================================================================================== #\" >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{project_id\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{project_name\\}/$project_name}\")\n    cmd=(\"${cmd[@]//\\{project_number\\}/$project_number}\")\n    cmd=(\"${cmd[@]//\\{project\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$project_name}\")\n    cmd=(\"${cmd[@]//\\{number\\}/$project_number}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\n    echo >&2\ndone <<< \"$projects\"\n"
  },
  {
    "path": "gcp/gce_foreach_vm.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo project id is {project_id}, VM name is '{name}', VM IP is '{ip}' in region '$region' zone '$zone'\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-20 15:14:12 +0000 (Tue, 20 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each GCP VM instance matching the given name/ip regex in the current GCP project\n\nThis is powerful so use carefully!\n\nUseful when combined with gce_ssh.sh using IAP to run some command on all instances\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the project id/names and exit after the first iteration\n\nRequires GCloud SDK to be installed and configured and 'gcloud' to be in the \\$PATH\n\nThe first argument is the VM Name or IP filter as an ERE regex to run against only those matching VMs\n\nAll remaining arguments become the command template\n\nThe following command template tokens are replaced in each iteration:\n\nVM Name:      {vm_name}     {name}\nVM IP:        {vm_ip}       {ip}\nProject ID:   {project_id}  {project}\nRegion:       {region}\nZone:         {zone}\n\neg.\n    ${0##*/} '.*' 'echo VM name {name} has ip {ip} in project {project_id} region {region} zone {zone}'\n\n\nGet all the SSH Keys of the VMs into your known_hosts so SSH doesn't prompt the first time and you don't have to ssh -o StrictHostKeyChecking=no\n\n    ${0##*/} '.*' 'ssh-keyscan {ip} >> ~/.ssh/known_hosts'\n\nIf using SSH, stop it from eating each next record by putting < /dev/null at the end of the ssh command:\n\n    gce_foreach_vm.sh solr-be \\\"ssh hari_sekhon_domain_com@{ip} 'hostname; ifconfig | grep inet' < /dev/null\\\"\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name_or_ip_filter_ERE> <command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nfilter=\"$1\"\nshift || :\n\n# for informational purposes in the header only\nproject_id=\"$(gcloud config list --format='get(core.project)')\"\nregion=\"$(gcloud config list --format='get(compute.region)')\"\n\ngcloud compute instances list --format='value(name,networkInterfaces[0].networkIP,zone)' |\ngrep -E \"$filter\" |\nwhile read -r name ip zone; do\n    echo \"# ========================================================================================================= #\" >&2\n    echo \"# GCP VM = $name - IP = $ip, Project $project_id - Zone: $zone\" >&2\n    echo \"# ========================================================================================================= #\" >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{vm_name\\}/$name}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$name}\")\n    cmd=(\"${cmd[@]//\\{vm_ip\\}/$ip}\")\n    cmd=(\"${cmd[@]//\\{ip\\}/$ip}\")\n    cmd=(\"${cmd[@]//\\{project_id\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{project\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{region\\}/$region}\")\n    cmd=(\"${cmd[@]//\\{zone\\}/$zone}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\n    echo >&2\ndone\n"
  },
  {
    "path": "gcp/gce_host_ips.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-28 12:40:41 +0000 (Wed, 28 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOutputs the hostnames and IP addresses of all or a given regex filter of Google Compute Engine hosts\nin the current project in a format that can be piped or copied into /etc/hosts\n\nUseful to doing direct SSH to hosts and especially for Ansible to speed up not going through IAP which is slow\n\n\nOutput:\n\n<ip>    <vm_hostname>\n<ip>    <vm_hostname2>\n<ip>    <vm_hostname3>\n\n\nUsed by gce_ssh_keyscan.sh which you might also want to use to pre-populate your SSH known_hosts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<regex_filter> <gcloud_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#max_args 1 \"$@\"\n\nregex=\"${1:-.*}\"\nshift || :\n\ngcloud compute instances list --filter=\"name ~ $regex\" --format='get(networkInterfaces[0].networkIP, name)' \"$@\" |\ncolumn -t |\nsort -k2\n"
  },
  {
    "path": "gcp/gce_instance_service_accounts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-01 15:51:36 +0100 (Tue, 01 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor all or a given list of instances in the current GCP project, list the instance names and their service accounts\n\nGCloud SDK is required, and will attempt to infer the zone based on the instance name, but sometimes it cannot, in\nwhich cases you advised to set the compute/zone yourself (gcloud config set compute/zone ...)\n\nCaveat: slow - would write a Python version but the API is so bad and hardly documented that it's not worth the trouble\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<instances>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nlist_vm_service_account(){\n    local instance=\"$1\"\n    local zone=\"${2:-}\"\n    local opts=()\n    if [ -n \"$zone\" ]; then\n        opts+=(--zone \"$zone\")\n    fi\n    # outputs the results to stderr when no zone is specified\n    gcloud compute instances describe \"$instance\" \"${opts[@]}\" --format='table[no-heading](name, serviceAccounts.email.join(\",\"))' 2>&1 #|\n    # GCloud SDK changes behaviour to error out if piped!\n    #grep -Ev '^No zone specified. Using zone \\[[[:alnum:]-]+\\] for instance:' || :\n}\n\nif [ $# -gt 0 ]; then\n    for arg; do\n        list_vm_service_account \"$arg\"\n    done\nelse\n    gcloud compute instances list --format='table[no-heading](name, serviceAccounts.email.join(\",\"))'\nfi\n"
  },
  {
    "path": "gcp/gce_is_preempted.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-21 16:02:54 +0000 (Thu, 21 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"\nChecks GCE Metadata API for the preemption status and returns preempted / not preempted and 0 or 1 exit code\n\nCan combine in a shutdown script to do something different if preempted vs a normal shutdown\n\nhttps://cloud.google.com/compute/docs/instances/create-start-preemptible-instance\n\nhttps://cloud.google.com/compute/docs/shutdownscript\n\nusage:\n\n${0##*/}\n\"\n    exit 3\n}\n\nif [ $# -ne 0 ]; then\n    usage\nfi\n\n#if ! curl -sS --connect-timeout 2 -H \"Metadata-Flavor: Google\" \"http://metadata.google.internal/computeMetadata/v1/\" &>/dev/null; then\n#    echo \"This script must be run from within a GCE instance as that is the only place the GCP GCE Metadata API is available\"\n#    exit 2\n#fi\n\nif output=\"$(curl -sS -H \"Metadata-Flavor: Google\" \"http://metadata.google.internal/computeMetadata/v1/instance/preempted\")\"; then\n    if grep -q TRUE <<< \"$output\"; then\n        echo \"preempted\"\n        exit 0\n    else\n        echo \"not preempted\"\n        exit 1\n    fi\nelse\n    echo \"FAILED to query GCE Metadata API, not running inside GCE?\"\n    exit 2\nfi\n"
  },
  {
    "path": "gcp/gce_meta.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-21 15:54:25 +0000 (Thu, 21 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick script to make it a little easier to query the Google Cloud GCE Metadata API\n#\n# Must be run from inside GCE otherwise will fail with an error like this:\n#\n# curl: (6) Could not resolve host: metadata.google.internal\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"\nSimple query wrapper to GCE's Metadata API\n\nLists resources if no argument given\n\n${0##*/} <resource>\n\neg. ${0##*/} instance/scheduling/preemptible\n\"\n    exit 3\n}\n\nif [ $# -gt 1 ] ||\n   [[ \"${1:-}\" =~ -.* ]]; then\n    usage\nfi\n\ncurl -sS -H \"Metadata-Flavor: Google\" \"http://metadata.google.internal/computeMetadata/v1/${1:-}\"\n\n# above doesn't output a trailing newline, when using in shell we usually want this\necho\n"
  },
  {
    "path": "gcp/gce_ssh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-18 06:44:52 +0000 (Sun, 18 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns 'gcloud compute ssh' to a VM while auto-determining its zone first to override any inherited zone config and make it easier to script iterating through VMs\n\nOtherwise if \\$CLOUDSDK_COMPUTE_ZONE environment is inherited (eg. via .envrc) pointing to a different zone it results in this error:\n\n    ERROR: (gcloud.compute.ssh) Could not fetch resource:\n     - The resource 'projects/<MY_PROJECT>/zones/<ZONE>/instances/<VM_NAME>' was not found\n\nor if in the wrong project or region you can be interactively prompted for a zone\n\nYour GCP project and region should already be set in your current 'gcloud config',\nor export CLOUDSDK_CORE_PROJECT and CLOUDSDK_CORE_REGION environment variables,\nor supply explicit --project ... and --region ... arguments to this script\n\nIf the VM zone isn't found it resolves the project and region to remind you that you're probably in the wrong project / region\nwhile displaying them to make it more obvious that you've inherited the wrong config, to save you some debugging time\nand stopping you from getting stuck on the interactive zone prompt\n\nExample iteration if you don't have direct access or SSH keys to a client's VMs,\nyou can use this to SSH for loop like so using the standard gcloud compute ssh argument of '--command':\n\n    for x in {1..10}; do gce_ssh.sh vm-\\$x --command 'sudo systemctl restart MYAPP.service'; echo; done\n\nYou can also use an IP address of the VM for convenience which will get resolved to a VM name\n\nRequires GCloud SDK to be installed, configured and authenticated\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<vm_name_or_ip> [<gcloud_sdk_args>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nvm_name=\"$1\"\nshift || :\n\nunset CLOUDSDK_COMPUTE_ZONE\n\nif [[ \"$vm_name\" =~ ^[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+$ ]]; then\n    ip=\"$vm_name\"\n    timestamp \"Resolving IP '$ip' to VM name\"\n    vm_name=\"$(gcloud compute instances list --filter=\"networkInterfaces[0].networkIP: $ip\" --format='value(name)')\"\n    if [ -z \"$vm_name\" ]; then\n        die \"Failed to resolve '$ip' to VM name\"\n    fi\nfi\n\n# If gcloud config's compute/zone is set, then actively determines the zone of the VM first and overrides it specifically\n# Better to let it try to figure it out and exit with an explicit error reminding what project and region you are in\n#if gcloud config get compute/zone 2>/dev/null | grep -q .; then\n    timestamp \"Determining zone for VM '$vm_name'\"\n    #zone=\"$(gcloud compute instances list | awk \"/^${vm_name}[[:space:]]/ {print \\$2}\")\"\n    zone=\"$(gcloud compute instances list --filter=\"name=$vm_name\" --format='value(zone)')\"\n    if [ -z \"$zone\" ]; then\n        die \"Failed to determine zone for VM name '$vm_name' - perhaps VM name is incorrect?\nor wrong project ('$(gcloud config get core/project 2>/dev/null)')?\nor wrong region ('$(gcloud config get compute/region 2>/dev/null)')?\"\n    fi\n#fi\n\n# would auto-determine the zone if in the right project and region but otherwise will interactively prompt\n# - this is why we auto-populate the zone above to give a very explicit error out while showing the currently inherited project and region\ntimestamp \"gcloud compute ssh '$vm_name' ${zone:+--zone \"'$zone'\"} $*\"\ngcloud compute ssh \"$vm_name\" ${zone:+--zone \"$zone\"} \"$@\"\n"
  },
  {
    "path": "gcp/gce_ssh_keyscan.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-28 12:40:41 +0000 (Wed, 28 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSSH keyscans all the GCE VM hosts that match the given regex filter to add them to SSH known_hosts\n\nUseful to doing direct SSH to hosts and especially for Ansible to speed up not going through IAP which is slow\nand not having them prompt to accept the SSH keys\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<regex_filter> <gcloud_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#max_args 1 \"$@\"\n\nssh_known_hosts_file=~/.ssh/known_hosts\n\n\"$srcdir/gce_host_ips.sh\" \"$@\" |\nwhile read -r ip hostname; do\n    ssh-keyscan \"$ip\" >> \"$ssh_known_hosts_file\"\n    ssh-keyscan \"$hostname\" >> \"$ssh_known_hosts_file\"\ndone\n"
  },
  {
    "path": "gcp/gce_when_preempted.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-21 16:02:54 +0000 (Thu, 21 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nWaits and watches GCE Metadata API for the preemption trigger before continuing to execute its arguments\n\nAlternative to shutdown scripts or if you want to set up an interactive CLI latch on preemption\n\nhttps://cloud.google.com/compute/docs/shutdownscript\n\nUsage:\n\n ${0##*/} \"command1; command2; command 3\"        All commands execute in a subshell\n\n ${0##*/}; command 1; command2; command 3        All commands execute in current shell\n                                                                     (notice the semi-colon immediately after the script giving it no argument, merely using it as a latch mechanism)\n\n ${0##*/} command1; command2; command 3          First command executes in the subshell (it's an argument). Commands 2 & 3 execute in the current shell after this script\n\n ${0##*/} 'x=test; echo \\$x'                      Variable is interpolated inside the single quotes at runtime after receiving Spot Termination notice\n\n\nInspired by whenup() / whendown() for hosts and whendone() for processes from interactive bash library .bash.d/* sourced as part of the .bashrc in this repo\n\nFor AWS there is a similar adjacent script aws_spot_when_terminated.sh\n\nEOF\n    exit 3\n}\n\nif [[ \"${1:-}\" =~ -.* ]]; then\n    usage\nfi\n\nif ! curl -sS -H \"Metadata-Flavor: Google\" \"http://metadata.google.internal/computeMetadata/v1/\" &>/dev/null; then\n    echo \"This script must be run from within a GCE instance as that is the only place the GCP GCE Metadata API is available\"\n    exit 2\nfi\n\nif output=\"$(curl -sS -H \"Metadata-Flavor: Google\" \"http://metadata.google.internal/computeMetadata/v1/instance/preempted?wait_for_change=true\")\"; then\n    if grep -q TRUE <<< \"$output\"; then\n        \"$@\"\n    else\n        echo \"not preempted, skipping commands\"\n        exit 1\n    fi\nelse\n    echo \"FAILED to query GCE Metadata API, not running inside GCE?\"\n    exit 2\nfi\n"
  },
  {
    "path": "gcp/gcp_ansible_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-23 17:50:33 +0000 (Fri, 23 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an Ansible service account for the Ansible GCP playbook inventory plugin, in the current or specified GCP\nproject, then creates and downloads the credentials json and even prints the command to configure your environment\nto start using Ansible immediately\n\nexport GOOGLE_CREDENTIALS=\\$HOME/.gcloud/\\$name-\\$project-credential.json\n\nThe following optional arguments can be given:\n\n- service account name prefix   (default: \\$USER-ansible)\n- credential file path          (default: \\$HOME/.gcloud/\\$name-\\$project-credential.json)\n- project                       (default: \\$CLOUDSDK_CORE_PROJECT or gcloud config's currently configured project setting core.project)\n\nIdempotent - safe to re-run, will skip service accounts and keyfiles that already exist\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<name> <credential.json> <project>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nname=\"${1:-$USER-ansible}\"\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"${3:-${CLOUDSDK_CORE_PROJECT:-$(gcloud config list --format='get(core.project)')}}\"\n\nnot_blank \"$project\" || die \"ERROR: no project specified and \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\nkeyfile=\"${2:-$HOME/.gcloud/$name-$project-credential.json}\"\n\ngcp_create_serviceaccount_if_not_exists \"$name\" \"$project\" \"$USER's service account for Ansible deployments\"\n\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\ngcp_create_credential_if_not_exists \"$service_account\" \"$keyfile\"\n\necho \"Granting Owner permissions to service account '$service_account' on project '$project'\"\n# some projects may require --condition=None in non-interactive mode\ngcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role=roles/owner --condition=None >/dev/null\n\nkeyfile=\"$(readlink -e \"$keyfile\")\"\n\n# https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_inventory.html\n#\n#   either\n#\n#       GCE_CREDENTIALS_FILE_PATH\n#   or\n#       GCP_SERVICE_ACCOUNT_FILE\n#\n#   optional\n#\n#       GCP_SERVICE_ACCOUNT_FILE\necho\necho \"Set this in your environment to use Ansible now:\"\necho\necho \"export GCE_CREDENTIALS_FILE_PATH=$keyfile\"\necho\n"
  },
  {
    "path": "gcp/gcp_ci_build.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-02 18:58:29 +0000 (Tue, 02 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp_ci.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a GCP CI Build using Google Cloud Build\n\nEnvironment variables to set in the CI/CD system:\n\nAPP                     - name of the app / docker image to build\nBUILD                   - should be automatically set to the Git hash on Jenkins or TeamCity\nCLOUDSDK_CORE_PROJECT   - project ID of your GCP project\nGCP_SERVICEACCOUNT_KEY  - the contents of a credentials.json for a serviceaccount with permissions to GCR + GKE\n\nPrimarily written for Jenkins and TeamCity, but should work with minor alterations in other CI/CD tools (see lib/gcp_ci.sh which infers branch and build details)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\n# CLOUDSDK_CORE_PROJECT and APP should be set by the CI/CD system\n# BUILD is inferred from the Git commit that triggered the CI/CD system\ncheck_env_defined \"APP\"\ncheck_env_defined \"BUILD\"\ncheck_env_defined \"CLOUDSDK_CORE_PROJECT\"\ncheck_env_defined \"GCP_SERVICEACCOUNT_KEY\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nset -x\n\nprintenv\n\n# if you project has a uniform naming that maps to branches, enable and tune this function from lib/gcp_ci.sh\n#set_gcp_project\n\n# provide a credentials.json file argument to this function or provide if in a GCP_SERVICEACCOUNT_KEY environment variable via the CI/CD system\n# necessary so you can log in to different projects and maintain IAM permissions isolation for safety\n# do not use the same serviceaccount with permissions across projects, you can cross contaminate and make mistakes, deploy the wrong environment etc.\ngcp_login\n\n# if it's not already built, submit a Google Cloud Build to build the docker images\nif ! tags_exist_for_container_image \"$APP\"; then\n    # $BUILD is auto-populated via lib/gcp_ci.sh from GIT_COMMIT in Jenkins or BUILD_VCS_NUMBER in TeamCity\n    #gcloud_builds_submit \"${BUILD}\"\n\n    gcloud builds submit --project \"$CLOUDSDK_CORE_PROJECT\" --config \"cloudbuild.yaml\" --substitutions _REGISTRY=\"gcr.io/$CLOUDSDK_CORE_PROJECT,_BUILD=$BUILD\" --timeout=3600\n\n    \"$srcdir/gcr_tag_datetime.sh\" \"$CLOUDSDK_CORE_PROJECT/$APP:$BUILD\"\n    \"$srcdir/gcr_tag_latest.sh\"   \"$CLOUDSDK_CORE_PROJECT/$APP:$BUILD\"\nfi\n"
  },
  {
    "path": "gcp/gcp_ci_deploy_k8s.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-02 18:58:40 +0000 (Tue, 02 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp_ci.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a GCP CI/CD Deploy to GKE Kubernetes\n\nEnvironment variables to set in the CI/CD system:\n\nAPP                     - name of your application\nBUILD                   - should be automatically set to the Git hash on Jenkins or TeamCity\nENVIRONMENT             - dev/staging/production - corresponding to a local k8s/<environment> directory in the checkout\nCLUSTER_NAME            - name of your GKE cluster\nCLOUDSDK_CORE_PROJECT   - project ID of your GCP project\nCLOUDSDK_COMPUTE_REGION - GCP region of your GKE cluster\nGCP_SERVICEACCOUNT_KEY  - the contents of a credentials.json for a serviceaccount with permissions to run Google Cloud Build\n\nPrimarily written for Jenkins and TeamCity, but should work with minor alterations in other CI/CD tools (see lib/gcp_ci.sh which infers branch and build details)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\n# BUILD is inferred from the Git commit that triggered the CI/CD system\n# The rest of these should be set by the CI/CD system\ncheck_env_defined \"APP\"\ncheck_env_defined \"BUILD\"\ncheck_env_defined \"ENVIRONMENT\"\ncheck_env_defined \"CLOUDSDK_CORE_PROJECT\"\ncheck_env_defined \"CLOUDSDK_COMPUTE_REGION\"\ncheck_env_defined \"GKE_CLUSTER\"\ncheck_env_defined \"GCP_SERVICEACCOUNT_KEY\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nset -x\n\nkube_config_isolate\n\nprintenv\n\n# auto-infer CLOUDSDK_CORE_PROJECT if not set in environment\n# requires a uniform predictable project naming convention that maps to Git branches, tune this function in lib/gcp_ci.sh\n# won't override CLOUDSDK_CORE_PROJECT\n#set_gcp_project\n\n# auto-infer CLOUDSDK_COMPUTE_REGION if not set in environment\n# put logic or default region in lib/gcp_ci.sh function\n# won't override CLOUDSDK_COMPUTE_REGION\n#set_gcp_compute_region europe-west1\n\n# provide a credentials.json file argument to this function or provide it in a GCP_SERVICEACCOUNT_KEY environment variable via the CI/CD system\n# necessary so you can log in to different projects and maintain IAM permissions isolation for safety\n# do not use the same serviceaccount with permissions across projects, you can cross contaminate and make mistakes, deploy the wrong environment etc.\ngcp_login\n\ngke_login \"$GKE_CLUSTER\"\n\ncd \"k8s/$ENVIRONMENT\"\n\nreplace_latest_with_build \"$BUILD\"\n\ndownload_kustomize\n\nkustomize_kubernetes_deploy \"${K8_APP:-$APP}\" \"${K8_NAMESPACE:-$APP}\"\n"
  },
  {
    "path": "gcp/gcp_cli_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-20 17:26:21 +0000 (Sat, 20 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP service account for GCloud SDK CLI to avoid having to re-login every day with 'gcloud auth login'\n\nGrants this service account 'owner' rights to all projects\n\nCreates and downloads a json credential and even prints the commands to activate the credential\n\nexport GOOGLE_CREDENTIALS=\\\"\\$HOME/.gcloud/\\$name-\\$project-credential.json\\\"\ngcloud auth activate-service-account --key-file=\\\"\\$GOOGLE_CREDENTIALS\\\"\n\nThe following optional arguments can be given:\n\n- service account name prefix   (default: \\$USER-cli)\n- credential file path          (default: \\$HOME/.gcloud/\\$name-\\$project-credential.json)\n- project                       (default: \\$CLOUDSDK_CORE_PROJECT or gcloud config's currently configured project setting core.project)\n\nThis can also be used as a backup credential - this way if something accidentally happens to your primary user account\nor this service account, you can always use the other to repair access without having to rely on colleagues who be away\n\nIdempotent - safe to re-run, will skip service accounts and keyfiles that already exist\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<name> <credential.json> <project>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nname=\"${1:-$USER-cli}\"\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"${3:-${CLOUDSDK_CORE_PROJECT:-$(gcloud config list --format='get(core.project)')}}\"\n\nnot_blank \"$project\" || die \"ERROR: no project specified and \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\nkeyfile=\"${2:-$HOME/.gcloud/$name-$project-credential.json}\"\n\ngcp_create_serviceaccount_if_not_exists \"$name\" \"$project\" \"$USER's service account for CLI usage\"\n\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\ngcp_create_credential_if_not_exists \"$service_account\" \"$keyfile\"\n\ntimestamp \"Granting Owner permissions to service account '$service_account' on all projects\"\n\nfor project in $(gcloud projects list --format='get(project_id)'); do\n    timestamp \"Granting Owner permissions to service account '$service_account' on project '$project'\"\n    # some projects may require --condition=None in non-interactive mode\n    gcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role='roles/owner' --condition=None >/dev/null\ndone\n\nkeyfile=\"$(readlink -e \"$keyfile\")\"\n\necho\necho \"Set this in your environment to use this long-term credential in your CLI:\"\necho\necho \"export GOOGLE_CREDENTIALS=$keyfile\"\necho\necho \"gcloud auth activate-service-account --key-file=\\\"\\$GOOGLE_CREDENTIALS\\\"\"\necho\n"
  },
  {
    "path": "gcp/gcp_cloud_schedule_sql_exports.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-16 10:51:45 +0100 (Fri, 16 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Python-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates Cloud Scheduler PubSub jobs to trigger Cloud SQL export jobs for every database\nin every running non-replica Cloud SQL instance in the current project\n\nSQL instances can optionally be specified, otherwise iterates all running non-replica SQL instances\n(only running instances can export, otherwise will error out)\n(only non-replicas should be exported, replicas will likely fail due to conflict with replication recovery)\n\nConsider scheduling cron to be before / after the Automated Backups window and before the Google Maintenance Window for each instance\n\nOptional environment variables and their defaults:\n\n\\$BUCKET                  \\${project_id}-sql-backups\n\\$PUBSUB_TOPIC            cloud-sql-backups\n\\$CLOUD_SCHEDULER_CRON    0 2 * * *\n\\$TIMEZONE                Etc/UTC\n\\$CLOUD_SCHEDULER_REPLACE if set to any value will delete and recreate the Cloud Scheduler job (gcloud prompts before deleting each job)\n\nCaveat: if the scheduler kicks off a subsequent database export on the same SQL instance, it will fail to launch if the previous one hasn't finished yet, so the \\$CLOUD_SCHEDULER_CRON will increment the hour field for each successive database on the same SQL instance\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nproject_id=\"$(gcloud config list --format=\"value(core.project)\")\"\nnot_blank \"$project_id\" || die \"ERROR: GCloud SDK core.project property not set in config\"\n\nbucket=\"${BUCKET:-${project_id}-sql-backups}\"\ncron=\"${CLOUD_SCHEDULER_CRON:-0 2 * * *}\"\ntopic=\"${PUBSUB_TOPIC:-cloud-sql-backups}\"\ntimezone=\"${TIMEZONE:-Etc/UTC}\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    # XXX: can only list databases and export from running instances\n    #      can only export from non-replicas (see gcp_sql_export.sh for details)\n    timestamp \"Getting all SQL instance in the current project\"\n    sql_instances=\"$(gcloud sql instances list --format=json |\n                     jq -r '.[] | select(.instanceType != \"READ_REPLICA_INSTANCE\") | select(.state == \"RUNNABLE\") | .name')\"\nfi\n\nfor sql_instance in $sql_instances; do\n    timestamp \"Getting all databases on SQL instance '$sql_instance'\"\n    databases=\"$(gcloud sql databases list --instance=\"$sql_instance\" --format='get(name)')\"\n    echo >&2\n    instance_cron=\"$cron\"\n    for database in $databases; do\n        # skip information schema, not allowed to dump this on MySQL, fails with:\n        # ERROR: (gcloud.sql.export.sql) [ERROR_RDBMS] mysqldump: Dumping 'information_schema' DB content is not supported\n        [ \"$database\" = \"information_schema\" ] && continue\n        # skip these MySQL built-in DBs too\n        [ \"$database\" = \"sys\" ] && continue\n        [ \"$database\" = \"performance_schema\" ] && continue\n        job_name=\"cloud-sql-backup--$sql_instance--$database\"\n        if [ -n \"${CLOUD_SCHEDULER_REPLACE:-}\" ]; then\n            timestamp \"Deleting Cloud Scheduler job for instance '$sql_instance' database '$database' if exists\"\n            gcloud scheduler jobs delete \"$job_name\" || :\n            echo >&2\n        fi\n        timestamp \"Creating Cloud Scheduler job for instance '$sql_instance' database '$database' with cron '$instance_cron'\"\n        gcloud scheduler jobs create pubsub \"$job_name\" --schedule \"$instance_cron\" --topic \"$topic\" --message-body '{ \"database\": \"'\"$database\"'\", \"instance\": \"'\"${sql_instance}\"'\", \"project\": \"'\"$project_id\"'\", \"bucket\": \"'\"$bucket\"'\" }' --time-zone \"$timezone\" --description \"Triggers Cloud SQL export of instance '$sql_instance' database '$database' via a PubSub message trigger to a Cloud Function\"\n        echo >&2\n        # increments the second hour field, resets the clock if we hit midnight\n        instance_cron=\"$(awk '{if($2 > 22){$2 -= 24}; print $1\" \"$2+1\" \"$3\" \"$4\" \"$5}' <<< \"$instance_cron\")\"\n    done\ndone\n"
  },
  {
    "path": "gcp/gcp_find_orphaned_disks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 10:22:49 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds orphaned disks across one or more GCP Projects using GCloud SDK\n\nThis is done by finding disks in each project with no 'users' (instances attached)\n\nOutput format:\n\n<project_id>    <zone>    <type>    <sizeGB>    <disk_name>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    tr '[:space:]' '\\n' <<< \"$@\"\nelse\n    gcloud projects list --format=\"get(project_id)\"\nfi |\nwhile read -r project_id; do\n    # XXX: for disks that have always been in use lastDetachTimestamp may not exist and result in false positives\n    gcloud compute disks list --format=\"table[no-heading](name, zone.basename(), type.basename(), sizeGb, users.join(','))\" --project \"$project_id\" |\n    while read -r disk zone type size users; do\n        [ -n \"$users\" ] && continue\n        printf '%s\\t%s\\t%s\\t%s\\t%s\\n' \"$project_id\" \"$zone\" \"$type\" \"$size\" \"$disk\"\n    done\ndone\n"
  },
  {
    "path": "gcp/gcp_firewall_disable_default_rules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-01-20 13:04:42 +0000 (Wed, 20 Jan 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables GCP default-allow-* rules in the default network as these are too lax permitting all of the internet to access VMs\n\nSkips the default-allow-internal rule though to avoid breaking internal infrastructure connections, you'd have to change that yourself if wanted.\n\nIn reality you probably shouldn't be using the default network anyway, and disabling these rules is just good practice to tighten up your GCP configuration.\n\n\nGCP Firewall Documentation:\n\n    https://cloud.google.com/vpc/docs/firewalls#more_rules_default_vpc\n\n\nRequires GCloud SDK to be installed and configured to the right project\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ngcloud compute firewall-rules list --filter 'name: default-allow-* AND network: default' --format='value(name)' |\ngrep -v '^default-allow-internal$'  |\nwhile read -r firewall_rule_name; do\n    timestamp \"disabling firewall-rule $firewall_rule_name\"\n    gcloud compute firewall-rules update --disabled \"$firewall_rule_name\"\n    echo\ndone\n"
  },
  {
    "path": "gcp/gcp_firewall_risky_rules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-05 14:48:56 +0000 (Fri, 05 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all risky GCP Firewall rules enabled in the current project that are open with source range 0.0.0.0/0 (whole internet can pass through)\n\nSuch rules should be rare, eg:\n\n- your public website (and even then only if not using full Cloudflare Proxied protection)\n\n- GKE ingress generated GCP firewall rules (too open by default)\n    - to lock them down see my adjacent Kubernetes-templates repo's service.yaml:\n\n    https://github.com/HariSekhon/Kubernetes-templates/blob/master/service.yaml\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_id>\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nopts=()\nif [ -n \"${1:-}\" ]; then\n    opts+=(--project \"$1\")\nfi\n\ngcloud compute firewall-rules list --format='table[no-heading](name,source_ranges)' --filter='disabled=false' \"${opts[@]}\" |\n{ grep \"'0.0.0.0/0'\" || : ; }\n"
  },
  {
    "path": "gcp/gcp_foreach_project.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo project id is {id}, name is '{name}'\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-25 16:39:17 +0100 (Tue, 25 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each GCP project in the current account\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the project id/names and exit after the first iteration\n\nRequires GCloud SDK to be installed and configured and 'gcloud' to be in the \\$PATH\n\nAll arguments become the command template\n\nThe following command template tokens are replaced in each iteration:\n\nProject ID:     {id}    {project_id}\nProject Name:   {name}  {project_name}\n\n\neg.\n    ${0##*/} 'echo GCP project has id {id} and name {name}'\n\n\nFor a more useful example, see:\n\n    gcp_info_all_projects.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# don't need to capture and replace project in config as done in environment variable now as it's much safer as it's limited to only this script and its child processes\n#current_project=\"$(gcloud config list --format=\"value(core.project)\")\"\n#if [ -n \"$current_project\" ]; then\n#    # want interpolation now not at exit\n#    # shellcheck disable=SC2064\n#    trap \"gcloud config set project '$current_project'\" EXIT\n#else\n#    trap \"gcloud config unset project\" EXIT\n#fi\n\nprojects=\"$(gcloud projects list --format=\"value(project_id,name)\")\"\n\ntotal_projects=\"$(grep -c . <<< \"$projects\")\"\n\ni=0\n\nwhile read -r project_id project_name; do\n    (( i += 1 ))\n    echo \"# ============================================================================ #\" >&2\n    echo \"# ($i/$total_projects) GCP Project ID = $project_id -- Name = $project_name\" >&2\n    echo \"# ============================================================================ #\" >&2\n    # XXX: this would cause a concurrency race condition bug between other scripts and sessions that could be dangerous\n    #gcloud config set project \"$project_id\"\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{project_id\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{project_name\\}/$project_name}\")\n    cmd=(\"${cmd[@]//\\{project\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$project_id}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$project_name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\n    echo >&2\ndone <<< \"$projects\"\n"
  },
  {
    "path": "gcp/gcp_iam_identities_in_use.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-12 14:13:44 +0000 (Fri, 12 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuick list of unique GCP IAM identities (users/groups/serviceAccounts) in use in the current GCP project or across all projects\n\nYou can optionally specify the GCP project, otherwise infers your currently set core.project\n\nIf you specify 'all' for project, will return a sorted superset list from all projects\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nproject=\"${1:-}\"\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\nget_identities(){\n    local project=\"$1\"\n    gcloud projects get-iam-policy \"$project\" --format=json |\n    jq -r '.bindings[].members[]'\n}\n\nif [ \"$project\" = \"all\" ] ;then\n    for project in $(gcloud projects list --format='get(project_id)'); do\n        get_identities \"$project\"\n    done\nelse\n    get_identities \"$project\"\nfi |\nsort -u\n"
  },
  {
    "path": "gcp/gcp_iam_roles_granted_to_identity.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-19 11:23:50 +0000 (Fri, 19 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all roles in the given or current project which are assigned to any identity (user/group/serviceAccount) matching the given regex\n\nFor best matching the identity should be in the standard GCP format eg.\n\n    user:hari@mydomain.com\n\n    group:platform-engineering@mydomain.com\n\n    serviceAccount:myproject-gke-sa@myproject.iam.gserviceaccount.com\n    serviceAccount:myproject@appspot.gserviceaccount.com\n\nbut is interpreted as a regex so you could just use a substring match like 'hari' and that would find roles containing any users/groups/serviceAccounts with that string in them\n\n    ${0##*/} hari\n\nYou could also use this to find any roles granted on a user-basis against group-oriented policy eg.\n\n    ${0##*/} ^user:\n\nFind roles granted too widely to all authenticated users, searching across all your projects:\n\n    ${0##*/} group:allAuthenticatedUsers all\n\nFind roles granted too widely to all unauthenticated users, searching across all your projects:\n\n    ${0##*/} group:allUsers all\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<identity_regex> [<project_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nregex=\"$1\"\nproject=\"${2:-}\"\n\nif is_blank \"$regex\"; then\n    usage \"identity regex cannot be blank\"\nfi\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\nget_roles(){\n    local project=\"$1\"\n    gcloud projects get-iam-policy \"$project\" --format=json |\n    jq -r \".bindings[] | select(.members[] | test(\\\"$regex\\\")) | .role\"\n}\n\nif [ \"$project\" = \"all\" ] ;then\n    for project in $(gcloud projects list --format='get(project_id)'); do\n        get_roles \"$project\"\n    done\nelse\n    get_roles \"$project\"\nfi |\nsort -u\n"
  },
  {
    "path": "gcp/gcp_iam_roles_granted_too_widely.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-19 12:26:49 +0000 (Fri, 19 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds roles granted too widely in the current / given GCP project, or search across all your projects\n\nUses the adjacent script gcp_iam_roles_granted_to_identity.sh to search for the special GCP groups allAuthenticatedUsers and allUsers which should usually not be granted to anything\n\nSee gcp_iam_roles_granted_to_identity.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nproject=(\"${1:-}\")\n\necho \"Roles granted to all authenticated users:\"\n\"$srcdir/gcp_iam_roles_granted_to_identity.sh\" group:allAuthenticatedUsers \"${project[@]}\"\necho\necho \"Roles granted to even unauthenticated users:\"\n\"$srcdir/gcp_iam_roles_granted_to_identity.sh\" group:allUsers \"${project[@]}\"\necho\n"
  },
  {
    "path": "gcp/gcp_iam_roles_in_use.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-12 14:13:44 +0000 (Fri, 12 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuick list of unique project-level GCP IAM roles in use in the current GCP project or across all projects\n\nUseful for quick lookups of IAM policy role names which are different from the human readable names in the GCP UI\n\neg. when backporting GCP IAM permissions to Terraform:\n\n    gcloud projects get-iam-policy \\\"\\$(gcloud config list --format='get(core.project)')\\\" > /tmp/iam_policy.yaml\n    ${0##*/} > /tmp/iam_roles_reference.txt\n    vim -O /tmp/iam_policy.yaml /tmp/iam_roles_reference.txt\n    # in another window edit the Terraform IAM and screen/tmux back and forth from this reference\n\n\nNOTICE: does not include roles assigned to say individual GCS buckets, only those assigned at the project level to users, groups or serviceAccounts.\n\n\nYou can optionally specify the GCP project, otherwise infers your currently set core.project\n\nIf you specify 'all' for project, will return a sorted superset list from all projects\n\nIf the role you're looking for isn't currently in use, you may want to browse all role names in SDK format for use in Terraform via:\n\n  gcloud iam roles list --format='get(name)'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nproject=\"${1:-}\"\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\nget_roles(){\n    local project=\"$1\"\n    gcloud projects get-iam-policy \"$project\" --format=json |\n    #awk '/^[[:space:]]+role:[[:space:]]/{print $2}'\n    jq -r '.bindings[].role'\n}\n\nif [ \"$project\" = \"all\" ] ;then\n    for project in $(gcloud projects list --format='get(project_id)'); do\n        get_roles \"$project\"\n    done\nelse\n    get_roles \"$project\"\nfi |\nsort -u\n"
  },
  {
    "path": "gcp/gcp_iam_roles_with_direct_user_grants.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-19 11:59:27 +0000 (Fri, 19 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFind GCP roles that have been granted directly to a user in the current / given GCP project\n\nUseful to find IAM permissions in violation of best practice group-oriented management\n\nIf you set the environment variable \\$DUMP_ROLES to any value, will output a JSON output for each role so you can see the offending user members\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nproject=\"${1:-}\"\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\ngcloud projects get-iam-policy \"$project\" --format=json |\njq -r \".bindings[] | select(.members[] | test(\\\"^user:\\\")) | .role\" |\nsort -u |\nif [ -n \"${DUMP_ROLES:-}\" ]; then\n    while read -r role; do\n        gcloud projects get-iam-policy \"$project\" --format=json |\n        jq -r \".bindings[] | select(.role == \\\"$role\\\")\"\n    done\nelse\n    cat\nfi\n"
  },
  {
    "path": "gcp/gcp_iam_serviceaccount_members.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-01 15:51:14 +0100 (Tue, 01 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP service account members eg. to find GKE Workload Identity integrations\n\nOutput:\n\n    <service_account>   <role>  <member>\n\neg.\n\n    jenkins-agent@MyProject.iam.gserviceaccount.com      roles/iam.workloadIdentityUser  serviceAccount:MyProject.svc.id.goog[jenkins/jenkins-agent]\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -eq 1 ]; then\n    export CLOUDSDK_CORE_PROJECT=\"$1\"\nelif [ $# -eq 0 ]; then\n    :\nelse\n    usage\nfi\n\nserviceaccounts=\"$(gcloud iam service-accounts list --format='value(name.basename())')\"\n\nfor serviceaccount in $serviceaccounts; do\n    gcloud iam service-accounts get-iam-policy \"$serviceaccount\" --format json |\n    jq -r \"\n        .bindings[]? |\n        { \\\"role\\\": .role, \\\"member\\\": .members[] } |\n        [ \\\"$serviceaccount\\\", .role, .member ] |\n        @tsv\n    \"\ndone\n"
  },
  {
    "path": "gcp/gcp_iam_serviceaccounts_without_permissions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-12 14:13:44 +0000 (Fri, 12 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds GCP service accounts without any remaining IAM permissions - useful to find and remove old serviceaccounts after 90 day unused IAM permissions clean out\n\nYou can optionally specify the GCP project, otherwise infers your currently set core.project\n\nIf you specify 'all' for project, will return a sorted superset list from all projects\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nproject=\"${1:-}\"\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\nget_unused_identities(){\n    local project=\"$1\"\n    identities_in_use=\"$(\"$srcdir/gcp_iam_identities_in_use.sh\" \"$project\" | grep '^serviceAccount:' | sed 's/^[^:]*://')\"\n    gcloud iam service-accounts list --format='get(email)' --project \"$project\" |\n    while read -r email; do\n        # don't include <project>@appspot.gserviceaccount.com, but this was manageable in console\n        #if ! [[ \"$email\" =~ \\.iam\\.gserviceaccount\\.com$ ]]; then\n        #    continue\n        #fi\n        if ! grep -Fixq \"$email\" <<< \"$identities_in_use\"; then\n            echo \"$email\"\n        fi\n    done\n\n}\n\nif [ \"$project\" = \"all\" ] ;then\n    for project in $(gcloud projects list --format='get(project_id)'); do\n        get_unused_identities \"$project\"\n    done\nelse\n    get_unused_identities \"$project\"\nfi |\nsort -u\n"
  },
  {
    "path": "gcp/gcp_iam_users_granted_directly.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-19 12:06:42 +0000 (Fri, 19 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds GCP IAM users which have been granted roles on an individual basis in the current or given GCP project\n\nUseful to find best practice violations where users have been granted roles instead of group-based management\n\nThis is slightly more useful than the adjacent gcp_iam_roles_with_direct_user_grants.sh since this is listed in the GCP Console UI per user on the IAM permissions page\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nproject=\"${1:-}\"\n\nif is_blank \"$project\"; then\n    project=\"$(gcloud config list --format='get(core.project)')\"\nfi\n\nnot_blank \"$project\" || die \"ERROR: no project specified and GCloud SDK core.project property not set in config\"\n\ngcloud projects get-iam-policy \"$project\" --format=json |\njq -r \".bindings[].members[] | select(test(\\\"^user:\\\"))\" |\nsort -u\n"
  },
  {
    "path": "gcp/gcp_iam_workload_identities.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-01 16:01:05 +0100 (Tue, 01 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList GCP Workload Identity service account integrations\n\nUses adjacent script gcp_iam_serviceaccount_members.sh, see there for output format\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/gcp_iam_serviceaccount_members.sh\" \"$@\" |\ngrep roles/iam.workloadIdentityUser\n"
  },
  {
    "path": "gcp/gcp_info.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC1090,SC1091\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gather common GCP environment info for quickly surveying new client environments\n#\n# Requires:\n#\n# - GCloud CLI to be available and configured 'gcloud init'\n#   (or just use Cloud Shell, will prompt you to set the project if it's not already)\n# - API services to be enabled (or to select Y to enable them when prompted)\n# - Billing to be enabled in order to enable API services\n#\n# Tested with Google Cloud SDK installed locally\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154,SC1117\nusage_description=\"\nLists GCP deployed resources in the current or specified GCP Project\n\nMake sure that you run this from an authorized network so things like kubectl don't hang\n\nLists in this order (categories broadly reflect the GCP Console grouping of services):\n\n    - GCloud SDK version\n    - Auth, Organizations & Config:\n      - Organizations\n      - Auth Configurations\n      - Current Configuration & Properties\n    - Projects:\n      - Project Names & IDs\n      - Current Project\n      - checks project is set to continue with the following\n    - Services & APIs:\n      - Enabled Services & API\n      - collectors all available services to only show enabled services from this point onwards\n    - Accounts & Secrets:\n      - IAM Service Accounts\n      - Secrets Manager secrets\n    - Compute:\n      - GCE Virtual Machines\n      - App Engine instances\n      - Cloud Functions\n      - GKE Clusters\n      - Kubernetes, for every GKE cluster:\n        - cluster-info\n        - master component statuses\n        - nodes\n        - namespaces\n        - deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers\n        - services, ingresses\n        - jobs, cronjobs\n        - storage classes, persistent volumes, persistent volume claims\n        - service accounts, resource quotas, network policies, pod security policies\n        - container images running\n        - container images running counts descending\n        - pods  # might be too much detail if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments\n    - Storage:\n      - Cloud SQL instances\n      - Cloud SQL backups enabled\n      - Cloud Storage Buckets\n      - Cloud Filestore\n      - Cloud Memorystore Redis\n      - BigTable clusters and instances\n      - Datastore Indexes\n    - Networks:\n      - VPC Networks\n      - Addresses\n      - Proxies\n      - Subnets\n      - Routers\n      - Routes\n      - VPN Gateways\n      - VPN Tunnels\n      - Reservations\n      - Firewall Rules & Forwarding Rules\n      - DNS managed zones & verified domains\n    - Big Data:\n      - Dataproc clusters       (all regions)\n      - Dataproc jobs           (all regions)\n      - Dataflow jobs           (all regions)\n      - PubSub topics\n      - Cloud IOT Registries    (all regions)\n    - Tools:\n      - Cloud Source Repositories\n      - Cloud Builds\n      - Container Registry Images\n      - Deployment Manager\n\nThis is useful in so many ways. Aside from a general inventory / overview for a new client, you might be interested in tracking down a specific IP address by outputting this to a file and then running grepping for the IPs:\n\n    ${0##*/} | tee output.txt && grep -E '[[:digit:]]+(\\.[[:digit:]]+){3}' output.txt\n\n$gcp_info_noninteractive_help\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# GCloud SDK tools versions\ncat <<EOF\n# ============================================================================ #\n#                              G C l o u d   S D K\n# ============================================================================ #\n\nEOF\n\ngcloud version\n#echo\n#gsutil version -l\n#echo\n#bq version\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_auth_config.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_projects.sh\"\necho\necho\n\n# ============================================================================ #\n# this is done after gcp_info_projects.sh because that enforces having a project set\necho \"LISTING INFO FOR PROJECT:  $(gcloud info --format=\"get(config.project)\")\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_services.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_accounts_secrets.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_compute.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_storage.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_networking.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_bigdata.sh\"\necho\necho\n\n# ============================================================================ #\n. \"$srcdir/gcp_info_tools.sh\"\necho\necho\n\n# Finished\ncat <<EOF\n# ============================================================================ #\n# Finished listing resources for GCP Project $(gcloud config list --format=\"value(core.project)\")\n# ============================================================================ #\nEOF\n"
  },
  {
    "path": "gcp/gcp_info_accounts_secrets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP IAM Service Accounts & Secrets Manager secrets deployed in the current GCP Project\n\nLists in this order:\n\n    - IAM Service Accounts\n    - Secrets Manager secrets\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# Service Accounts\ncat <<EOF\n# ============================================================================ #\n#                        S e r v i c e   A c c o u n t s\n# ============================================================================ #\n\nEOF\n\ngcp_info \"Service Accounts\" gcloud iam service-accounts list\n\n\ncat <<EOF\n\n\n# ============================================================================ #\n#                         I A M   P e r m i s s i o n s\n# ============================================================================ #\n\nEOF\n\n\"$srcdir/gcp_iam_roles_granted_too_widely.sh\"\n\n\n# Secrets\ncat <<EOF\n\n\n# ============================================================================ #\n#                                 S e c r e t s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled secretmanager.googleapis.com; then\n    gcp_info \"GCP Secrets\" gcloud secrets list\nelse\n    echo \"Secrets Manager API (secretmanager.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_all_projects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-25 16:46:24 +0100 (Tue, 25 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGathers GCP Info across all projects\n\nCombines gcp_foreach_project.sh and gcp_info.sh\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ngcp_foreach_project.sh \"'$srcdir/gcp_info.sh' '{id}'\"\n"
  },
  {
    "path": "gcp/gcp_info_auth_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Auth, Organizations and Config\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# Organizations\ncat <<EOF\n# ============================================================================ #\n#                           O r g a n i z a t i o n s\n# ============================================================================ #\n\nEOF\n\ngcp_info \"Organizations\" gcloud organizations list\n\n\n# Auth & Config\ncat <<EOF\n\n\n# ============================================================================ #\n#                           A u t h   &   C o n f i g\n# ============================================================================ #\n\nEOF\n\ngcp_info \"GCloud SDK Configurations\" gcloud config configurations list\necho\n\n# list credentialed accounts and show which one is active - dupliates info from configurations so not needed\n#gcloud auth list\n#echo\n\necho \"=========================\"\necho \"GCloud SDK Configuration:\"\necho\ngcloud config list\n"
  },
  {
    "path": "gcp/gcp_info_bigdata.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Big Data resources deployed in the current GCP Project\n\nLists in this order:\n\n    - Dataproc clusters       (all regions)\n    - Dataproc jobs           (all regions)\n    - Dataflow jobs           (all regions)\n    - PubSub topics\n    - Cloud IOT registries    (all regions)\n\nEnvironment variables of regions to shortcut scanning all regions, comma or space separated:\n\nGCE_REGIONS - for Dataproc clusters and jobs\nIOT_REGIONS - for Cloud IOT registries\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# Dataproc clusters\ncat <<EOF\n# ============================================================================ #\n#                       D a t a p r o c   C l u s t e r s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled dataproc.googleapis.com; then\n    # because --region=all doesn't work\n    # inherit gce_regions if set elsewhere, eg. gcp_info_compute.sh called first when running gcp_info.sh\n    gce_regions=\"${GCE_REGIONS:-${gce_regions:-$(gcloud compute regions list --format='table[no-heading](name)')}}\"\n    gce_regions=\"${gce_regions//,/ }\"\n    gcp_info \"Dataproc clusters: global\"      gcloud dataproc clusters list --region=\"global\"\n    for region in $gce_regions; do\n        gcp_info \"Dataproc clusters: $region\" gcloud dataproc clusters list --region=\"$region\"\n    done\nelse\n    echo \"Dataproc API (dataproc.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Dataproc jobs\ncat <<EOF\n\n\n# ============================================================================ #\n#                           D a t a p r o c   J o b s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled dataproc.googleapis.com; then\n    # because --region=all doesn't work\n    # re-use gce_regions from above\n    gcp_info \"Dataproc jobs: global\"          gcloud dataproc jobs list --region=\"global\"\n    for region in $gce_regions; do\n        gcp_info \"Dataproc jobs: $region\"     gcloud dataproc jobs list --region=\"$region\"\n    done\nelse\n    echo \"Dataproc API (dataproc.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Dataflow jobs\ncat <<EOF\n\n\n# ============================================================================ #\n#                           D a t a f l o w   J o b s\n# ============================================================================ #\n\nEOF\n\n# works even when set to disabled:\n#\n# DISABLED  dataflow.googleapis.com   Dataflow API\n#\n#if is_service_enabled dataflow.googleapis.com; then\n    # --region=all      actually works here unlike dataproc and cloud iot\n    # --status=active   to see only running jobs\n    gcp_info \"Dataflow jobs\" gcloud dataflow jobs list --region=all --status=all\n#else\n#    echo \"Dataflow API (dataflow.googleapis.com) is not enabled, skipping...\"\n#fi\n\n\n# PubSub topics\ncat <<EOF\n\n\n# ============================================================================ #\n#                           P u b S u b   T o p i c s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled pubsub.googleapis.com; then\n    gcp_info \"Cloud PubSub topics\" gcloud pubsub topics list\nelse\n    echo \"Cloud PubSub API (pubsub.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Cloud IOT\ncat <<EOF\n\n\n# ============================================================================ #\n#                               C l o u d   I O T\n# ============================================================================ #\n\nEOF\n\n#iot_supported_regions=\"\n#asia-east1\n#europe-west1\n#us-central1\n#\"\n\n# get dynamically in case they add a region\n# ERROR: (gcloud.iot.registries.list) NOT_FOUND: The cloud region 'projects/$GOOGLE_PROJECT_ID/locations/all' (location 'all') isn't supported. Valid regions: {asia-east1,europe-west1,us-central1}\niot_regions=\"${IOT_REGIONS:-$(gcloud iot registries list --region=\"all\" 2>&1 | sed 's/.*{//; s/}//; s/,/ /g' || :)}\"\niot_regions=\"${iot_regions//,/ }\"\n\nif is_service_enabled cloudiot.googleapis.com; then\n    for region in $iot_regions; do\n        gcp_info \"Cloud IOT registries: $region\" gcloud iot registries list --region=\"$region\"\n    done\nelse\n    echo \"Cloud IOT API ( cloudiot.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_cloud_sql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Cloud SQL instances and the status of backups for each SQL instance in the current GCP Project\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# Cloud SQL instances\ncat <<EOF\n# ============================================================================ #\n#                     C l o u d   S Q L   I n s t a n c e s\n# ============================================================================ #\n\nEOF\n\n# might need this one instead sqladmin.googleapis.com\nif is_service_enabled sql-component.googleapis.com; then\n    gcp_info \"Cloud SQL instances\" gcloud sql instances list\n    gcp_info \"Cloud SQL backups enabled\"   gcloud sql instances list --format=\"table(name, settings.backupConfiguration.enabled: label='BACKUPS_ENABLED', settings.backupConfiguration.pointInTimeRecoveryEnabled, settings.backupConfiguration.replicationLogArchivingEnabled, settings.backupConfiguration.startTime)\"\n    echo\n    \"$srcdir/gcp_info_cloud_sql_databases.sh\"\nelse\n    echo \"Cloud SQL API (sql-component.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_cloud_sql_backups.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Cloud SQL instances and the status of backups for each SQL instance in the current GCP Project\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\ncat <<EOF\n# ============================================================================ #\n#                       C l o u d   S Q L   B a c k u p s\n# ============================================================================ #\n\nEOF\n\n# might need this one instead sqladmin.googleapis.com\nif is_service_enabled sql-component.googleapis.com; then\n    gcloud sql instances list --format='get(name)' |\n    while read -r instance; do\n        gcp_info \"Cloud SQL backups for instance '$instance'\" gcloud sql backups list --instance \"$instance\"\n    done\nelse\n    echo \"Cloud SQL API (sql-component.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_cloud_sql_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 11:55:20 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Cloud SQL databases on each SQL instance in the current GCP Project\n\nOnly works on running instances since it requires querying the DB. We skip non-running instances as otherwise the GCloud SDK errors out\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\ncat <<EOF\n# ============================================================================ #\n#                     C l o u d   S Q L   D a t a b a s e s\n# ============================================================================ #\n\nEOF\n\n# might need this one instead sqladmin.googleapis.com\nif is_service_enabled sql-component.googleapis.com; then\n    gcloud sql instances list --format='get(name)' --filter 'state = RUNNABLE' |\n    while read -r instance; do\n        gcp_info \"Cloud SQL databases for instance '$instance'\" gcloud sql databases list --instance \"$instance\"\n    done\nelse\n    echo \"Cloud SQL API (sql-component.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_cloud_sql_users.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 11:16:01 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Cloud SQL users for each running SQL instance in the current GCP Project\n\nOnly works on running instances since it requires querying the DB. We skip non-running instances as otherwise the GCloud SDK errors out\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\ncat <<EOF\n# ============================================================================ #\n#                         C l o u d   S Q L   U s e r s\n# ============================================================================ #\n\n# running instances only\n\nEOF\n\n# might need this one instead sqladmin.googleapis.com\nif is_service_enabled sql-component.googleapis.com; then\n    gcloud sql instances list --format='get(name)' --filter 'state = RUNNABLE' |\n    while read -r instance; do\n        gcp_info \"Cloud SQL users for instance '$instance'\" gcloud sql users list --instance \"$instance\"\n    done\nelse\n    echo \"Cloud SQL API (sql-component.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_compute.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Compute resources deployed in the current GCP Project\n\nLists in this order:\n\n    - GCE Virtual Machines\n    - App Engine instances\n    - Cloud Functions\n    - GKE Clusters\n    - Kubernetes, for every GKE cluster:\n      - cluster-info\n      - master component statuses\n      - nodes\n      - namespaces\n      - deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers\n      - storage classes, persistent volumes, persistent volume claims\n      - service accounts, resource quotas, network policies, pod security policies\n      - pods  # might be too volumous if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments\n\n$gcp_info_noninteractive_help\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\nDoes not apply to Kubernetes info\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# GCE Virtual Machines\ncat <<EOF\n# ============================================================================ #\n#                        V i r t u a l   M a c h i n e s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled compute.googleapis.com; then\n    #gcloud compute machine-types list\n\n    gcp_info \"GCE instances\" gcloud compute instances list --sort-by=ZONE\nelse\n    echo \"GCE API (compute.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# App instances\ncat <<EOF\n\n\n# ============================================================================ #\n#                    A p p   E n g i n e   i n s t a n c e s\n# ============================================================================ #\n\nEOF\n\n# works even when all of these are disabled:\n#\n# DISABLED  accessapproval.googleapis.com                         Access Approval API\n# DISABLED  appengine.googleapis.com                              App Engine Admin API\n# DISABLED  appengineflex.googleapis.com                          Google App Engine Flexible Environment\n\napp_engine_details(){\n    gcp_info \"App Engine\" gcloud app describe || return 0\n\n    #if is_service_enabled appengine.googleapis.com; then\n        # errors out if App Engine hasn't been created in the project yet\n        gcp_info \"App Engine instances\" gcloud app instances list || return 0\n    #else\n    #    echo \"GAE API (appengine.googleapis.com) is not enabled, skipping...\"\n    #fi\n}\napp_engine_details\n\n\n# Cloud Functions\ncat <<EOF\n\n\n# ============================================================================ #\n#                         C l o u d   F u n c t i o n s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled cloudfunctions.googleapis.com; then\n    gcp_info \"Cloud Functions\" gcloud functions list\nelse\n    echo \"Cloud Functions API (cloudfunctions.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Cloud Run\ncat <<EOF\n\n\n# ============================================================================ #\n#                               C l o u d   R u n\n# ============================================================================ #\n\nEOF\n\n# confirm this one\nif is_service_enabled run.googleapis.com; then\n    gcp_info \"Cloud Run services\" gcloud run services list\nelse\n    echo \"Cloud RUN API (run.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# ============================================================================ #\necho\necho\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/gcp_info_gke.sh\"\n"
  },
  {
    "path": "gcp/gcp_info_gke.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GKE resources deployed in the current GCP Project\n\nLists in this order:\n\n    - GKE Clusters\n    - Kubernetes, for every GKE cluster:\n      - cluster-info\n      - master component statuses\n      - nodes\n      - namespaces\n      - deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers\n      - storage classes, persistent volumes, persistent volume claims\n      - service accounts, resource quotas, network policies, pod security policies\n      - pods  # might be too volumous if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\nDoes not apply to Kubernetes info\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# GKE clusters\ncat <<EOF\n# ============================================================================ #\n#                            G K E   C l u s t e r s\n# ============================================================================ #\n\nEOF\n\nlist_gke_clusters(){\n    if is_service_enabled container.googleapis.com; then\n        gcp_info \"GKE clusters\" gcloud container clusters list\n        if ! gcloud container clusters list | grep -q .; then\n            echo \"No GKE clusters found, skipping listing GKE deployments and resources\"\n            return 0\n        fi\n    else\n        echo \"Google Kubernetes Engine API (container.googleapis.com) is not enabled, skipping...\"\n        return\n    fi\n    list_gke_deployments\n}\n\n\n# Kubernetes\nlist_gke_deployments(){\n    cat <<EOF\n\n\n# ============================================================================ #\n#                         G K E   D e p l o y m e n t s\n# ============================================================================ #\nEOF\n    gcloud container clusters list --format='value(name,zone)' |\n    while read -r cluster zone; do\n        cat <<EOF\n\n    # ======================================================= #\n    # GKE Cluster: $cluster\n    # ======================================================= #\n\nEOF\n        gcloud container clusters get-credentials \"$cluster\" --zone \"$zone\"\n        echo\n        \"$srcdir/../kubernetes/kubernetes_info.sh\"\n    done\n}\n\n\nlist_gke_clusters\n"
  },
  {
    "path": "gcp/gcp_info_networking.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP network resources deployed in the current GCP Project\n\nLists in this order:\n\n    - Networks:\n      - VPC Networks\n      - Addresses\n      - Proxies\n      - Subnets\n      - Routers\n      - Routes\n      - VPN Gateways\n      - VPN Tunnels\n      - Reservations\n    - Firewall Rules & Forwarding Rules\n    - DNS managed zones & verified domains\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# Networking\ncat <<EOF\n# ============================================================================ #\n#                              N e t w o r k i n g\n# ============================================================================ #\n\nEOF\n\ngcp_info \"Networks\"               gcloud compute networks list\n\ngcp_info \"Addresses\"              gcloud compute addresses list\n\ngcp_info \"Target Pools\"           gcloud compute target-pools list\n\ngcp_info \"HTTP Proxies\"           gcloud compute target-http-proxies list\n\ngcp_info \"HTTPS Proxies\"          gcloud compute target-https-proxies list\n\ngcp_info \"SSL Proxies\"            gcloud compute target-ssl-proxies list\n\ngcp_info \"TCP Proxies\"            gcloud compute target-tcp-proxies list\n\ngcp_info \"URL Maps\"               gcloud compute url-maps list\n\ngcp_info \"Subnets\"                gcloud compute networks subnets list --sort-by=NETWORK,REGION\n\ngcp_info \"Subnets usable for GKE\" gcloud container subnets list-usable --sort-by=NETWORK,REGION\n\ngcp_info \"Routers\"                gcloud compute routers list\n\ngcp_info \"Routes\"                 gcloud compute routes list\n\ngcp_info \"VPN Gateways\"           gcloud compute vpn-gateways list\n\ngcp_info \"VPN Tunnels\"            gcloud compute vpn-tunnels list\n\ngcp_info \"Reservations\"           gcloud compute reservations list\n\n\n# Firewalls\ncat <<EOF\n\n\n# ============================================================================ #\n#                               F i r e w a l l s\n# ============================================================================ #\n\nEOF\n\ngcp_info \"Firewall Rules\"   gcloud compute firewall-rules list\n                            # same output\n                            #gcloud compute firewall-rules list --sort-by=NETWORK\n\ngcp_info \"Forwarding Rules\" gcloud compute forwarding-rules list\n\n\n# DNS\ncat <<EOF\n\n\n# ============================================================================ #\n#                                     D N S\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled dns.googleapis.com; then\n    gcp_info \"DNS Managed Zones\" gcloud dns managed-zones list\n\n    gcp_info \"Domains verified\"  gcloud domains list-user-verified\nelse\n    echo \"Cloud DNS API (dns.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_projects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Projects and checks project is configured\n\nCan optionally specify a project id to switch to (will switch back to original project on any exit except kill -9)\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# Project\ncat <<EOF\n# ============================================================================ #\n#                                P r o j e c t s\n# ============================================================================ #\n\nEOF\n\ngcp_info \"GCP Projects\" gcloud projects list\necho\necho \"Checking project is configured...\"\n# unreliable only errors when not initially set, but gives (unset) if you were to 'gcloud config unset project'\n#if ! gcloud config get-value project &>/dev/null; then\n# ok, but ugly and format dependent\n#if ! gcloud config list | grep '^project[[:space:]]='; then\n# best\nif ! gcloud info --format=\"get(config.project)\" | grep -q .; then\n    cat <<EOF\n\nERROR: You need to configure your Google Cloud project first\n\nSelect one from the project IDs above:\n\ngcloud config set project <id>\nEOF\n    exit 1\nfi\n"
  },
  {
    "path": "gcp/gcp_info_services.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP services & APIs enabled in the current GCP Project\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# Services & APIs Enabled\ncat <<EOF\n# ============================================================================ #\n#                 S e r v i c e s   &   A P I s   E n a b l e d\n# ============================================================================ #\n\nEOF\n\nif ! type is_service_enabled &>/dev/null; then\n    echo \"getting list of all services & APIs (will use this to determine which services to list based on what is enabled)\" >&2\n    # shellcheck disable=SC1090,SC1091\n    . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n    echo >&2\n    echo >&2\nfi\n\ngcp_info \"Services Enabled\" gcloud services list --enabled\n"
  },
  {
    "path": "gcp/gcp_info_storage.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP storage resources deployed in the current GCP Project\n\nLists in this order:\n\n    - Cloud SQL instances\n    - Cloud SQL backups enabled\n    - Cloud Storage Buckets\n    - Cloud Filestore\n    - Cloud Memorystore Redis\n    - BigTable clusters and instances\n    - Datastore Indexes\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\"$srcdir/gcp_info_cloud_sql.sh\"\n\n# Cloud Storage Buckets\ncat <<EOF\n\n\n# ============================================================================ #\n#                                 B u c k e t s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled storage-component.googleapis.com; then\n    gsutil ls\nelse\n    echo \"Cloud Storage API (storage-component.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Cloud Filestore\ncat <<EOF\n\n\n# ============================================================================ #\n#                         C l o u d   F i l e s t o r e\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled file.googleapis.com; then\n    gcp_info \"Cloud Filestore instances\" gcloud filestore instances list\nelse\n    echo \"Cloud Filestore API (file.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Cloud MemoryStore Redis\ncat <<EOF\n\n\n# ============================================================================ #\n#                 C l o u d   M e m o r y s t o r e   R e d i s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled redis.googleapis.com; then\n    gcp_info \"Cloud Memorystore Redis instances\" gcloud redis instances list --region all\nelse\n    echo \"Cloud Memorystore Redis API (redis.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# BigTable clusters and instances\ncat <<EOF\n\n\n# ============================================================================ #\n#                                B i g T a b l e\n# ============================================================================ #\n\nEOF\n\n# works even with these disabled:\n#\n# DISABLED  bigtable.googleapis.com                               Cloud Bigtable API\n# DISABLED  bigtableadmin.googleapis.com                          Cloud Bigtable Admin API\n# DISABLED  bigtabletableadmin.googleapis.com                     Cloud Bigtable Table Admin API\n#\n# if is_service_enabled bigtable.googleapis.com; then\n    gcp_info \"BigTable clusters\"  gcloud bigtable clusters list\n    gcp_info \"BigTable instances\" gcloud bigtable instances list\n#else\n#    echo \"BigTable API (bigtable.googleapis.com) is not enabled, skipping...\"\n#fi\n\n\n# Datastore Indexes\ncat <<EOF\n\n\n# ============================================================================ #\n#                       D a t a s t o r e   I n d e x e s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled datastore.googleapis.com; then\n    # may error out if doesn't exist\n    gcp_info \"Cloud Datastore indexes\" gcloud datastore indexes list || :\nelse\n    echo \"Datastore API datastore.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_info_tools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP 'Tools category' resources deployed resources in the current GCP Project\n\nLists in this order:\n\n    - Cloud Source Repositories\n    - Cloud Builds\n    - Container Registry Images\n    - Deployment Manager\n\nCan optionally specify a project id using the first argument, otherwise uses currently configured project\n\n$gcp_info_formatting_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_bin gcloud\n\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\n    export CLOUDSDK_CORE_PROJECT=\"$project_id\"\nfi\n\n\n# shellcheck disable=SC1090,SC1091\ntype is_service_enabled &>/dev/null || . \"$srcdir/gcp_service_apis.sh\" >/dev/null\n\n\n# Source Repos\ncat <<EOF\n# ============================================================================ #\n#                            S o u r c e   R e p o s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled sourcerepo.googleapis.com; then\n    gcloud source repos list\nelse\n    echo \"Source Repos API (sourcerepo.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Cloud Builds\ncat <<EOF\n\n\n# ============================================================================ #\n#                            C l o u d   B u i l d s\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled cloudbuild.googleapis.com; then\n    gcp_info \"Cloud Builds\" gcloud builds list\nelse\n    echo \"Cloud Builds API (cloudbuild.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Container Registry Images\ncat <<EOF\n\n\n# ============================================================================ #\n#               C o n t a i n e r   R e g i s t r y   I m a g e s\n# ============================================================================ #\n\nEOF\n\nproject=\"$(gcloud info --format=\"get(config.project)\")\"\n\nif is_service_enabled containerregistry.googleapis.com; then\n    gcp_info \"Google Container Registry Images: gcr.io/$project\" gcloud container images list\n    for x in us eu asia; do\n        repository=\"$x.gcr.io/$project\"\n        gcp_info \"Google Container Registry Images: $repository\" gcloud container images list --repository \"$repository\"\n    done\nelse\n    echo \"Container Registry API (containerregistry.googleapis.com) is not enabled, skipping...\"\nfi\n\n\n# Deployment Manager\ncat <<EOF\n\n\n# ============================================================================ #\n#                      D e p l o y m e n t   M a n a g e r\n# ============================================================================ #\n\nEOF\n\nif is_service_enabled deploymentmanager.googleapis.com; then\n    gcp_info \"Deployment Manager deployments\" gcloud deployment-manager deployments list\nelse\n    echo \"Deployment Manager API (deploymentmanager.googleapis.com) is not enabled, skipping...\"\nfi\n"
  },
  {
    "path": "gcp/gcp_secret_add.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads a value from the command line and saves it to GCP Secret Manager without echo'ing it on the screen\n\nFirst argument is used as secret name\nSecond argument is used as secret string value\n    - if this argument is a file, such as an SSH key, reads the file content and saves it as the secret value\n    - if not given prompts for it with a non-echo'ing prompt (recommended for passwords)\nRemaining args are passed directly to 'gcloud secrets'\n\n\nIf you get an error like this:\n\nERROR: (gcloud.secrets.create) FAILED_PRECONDITION: Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/123456789012] attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.\n- '@type': type.googleapis.com/google.rpc.PreconditionFailure\n  violations:\n  - description: Constraint constraints/gcp.resourceLocations violated for [orgpolicy:projects/123456789012]\n      attempting to create a secret in [global]. For more information, see https://cloud.google.com/resource-manager/docs/organization-policy/defining-locations.\n    subject: orgpolicy:projects/123456789012\n    type: constraints/gcp.resourceLocations\n\n\nThen just append the following gcloud secrets args when calling this script to set a location (change --locations to your preferred):\n\n    --replication-policy user-managed --locations europe-west2\n\n\n$usage_gcloud_sdk_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<secret> <gcloud_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\nsecret=\"${2:-}\"\nshift || :\nshift || :\n\nif [ -z \"$secret\" ]; then\n    read_secret\n    #if [ -f \"$secret\" ]; then\n    #    read -p \"Given secret has been found as a local filename, are you sure you want to add this file?\" answer\n    #    if ! is_yes \"$answer\"; then\n    #        die 'Aborting...'\n    #    fi\n    #fi\nfi\n\nif [ -f \"$secret\" ]; then\n    gcloud secrets create \"$name\" --data-file \"$secret\" \"$@\"\nelse\n    gcloud secrets create \"$name\" --data-file - \"$@\" <<< \"$secret\"\nfi\n"
  },
  {
    "path": "gcp/gcp_secret_add_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds a given binary file to GCP Secret Manager as base64\n\nFirst argument is used as secret name\nSecond argument must be a binary file such as a QR Code screenshot - this is converted to base 64\nRemaining args are passed directly to 'gcloud secrets'\n\nExample:\n\n    ${0##*/} mysecret qr-code-screenshot.png\n\n# To retrieve the binary file back:\n\n    gcp_secret_get.sh mysecret | base64 --decode > qr-code.png\n\nWarning: this has been tested and works to instantiate other team members virtual MFA on their phones for GitHub.com, even a year later, but Azure AD seems to expire the QR code\n         https://stackoverflow.com/questions/73578781/qr-code-got-expire-with-azure-verifiable-credential\n\n\n$usage_gcloud_sdk_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<secret> <gcloud_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nname=\"$1\"\nfile=\"$2\"\nshift || :\nshift || :\n\nif ! [ -f \"$file\" ]; then\n    die \"File not found: $file\"\nfi\n\nbase64 \"$file\" |\ngcloud secrets create \"$name\" --data-file - \"$@\"\n"
  },
  {
    "path": "gcp/gcp_secret_get.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the latest version of a given GCP Secret Manager secret and fetches its value\n\nFirst argument is used as secret name - if not given prompts for it\nSecond or more args are passed to 'gcloud secrets'\n\n\n$usage_gcloud_sdk_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nname=\"$1\"\n\nlatest_version=\"$(\n    gcloud secrets versions list \"$name\" --filter='state = enabled' --format='value(name)' |\n    sort -k1nr |\n    head -n1\n)\"\n\ngcloud secrets versions access \"$latest_version\" --secret=\"$name\"\n"
  },
  {
    "path": "gcp/gcp_secret_label_k8s.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds kubernetes-cluster and kubernetes-namespace labels to a given GCP Secret Manager secret based on the current kubectl context for later use by the gcp_secrets_to_kubernetes.sh script\n\nFirst argument is used as secret name - if not given prompts for it\nSecond or more args are passed to 'gcloud secrets'\n\n\nGCloud SDK and kubectl must both be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<gcloud_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\nshift || :\n\n# assumes the context and cluster name are the same which they usually are for AWS, GCP, docker-desktop and minikube\nread -r kubernetes_cluster kubernetes_namespace \\\n    < <(kubectl config get-contexts | awk '/*/{print $2\" \"$NF}')\n\ngcloud secrets update \"$name\" --update-labels=\"kubernetes-cluster=$kubernetes_cluster,kubernetes-namespace=$kubernetes_namespace\" \"$@\"\n"
  },
  {
    "path": "gcp/gcp_secret_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-24 12:40:18 +0000 (Wed, 24 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads a value from the command line without echo'ing it on the screen and updates the given GCP Secret Manager secret\n\nFirst argument is used as secret name - if not given prompts for it\nSecond argument is used as secret string value\n    - if this argument is a file, such as an SSH key, reads the file content and saves it as the secret value\n    - if not given prompts for it with a non-echo'ing prompt (recommended for passwords)\nThird or more args are passed to 'gcloud secrets'\n\n\n$usage_gcloud_sdk_required\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<name> [<secret> <gcloud_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\nsecret=\"${2:-}\"\nshift || :\nshift || :\n\nif [ -z \"$secret\" ]; then\n    read_secret\nfi\n\nif [ -f \"$secret\" ]; then\n    secret=\"$(cat \"$secret\")\"\nfi\n\ngcloud secrets versions add \"$name\" --data-file - \"$@\" <<< \"$secret\"\n"
  },
  {
    "path": "gcp/gcp_secrets_labels.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-28 23:37:29 +0100 (Wed, 28 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GCP Secrets labels in tabular form suitable for quick reviews and shell pipelines\n\nDoes not list secrets without labels as this is primarily a label review tool\n\n\nOutput Format:\n\n<secret1_name>   <label_key1>=<label_value1>\n<secret1_name>   <label_key2>=<label_value2>\n<secret2_name>   <label_key1>=<label_value1>\n...\n\n\nRequires GCloud SDK to be installed and configured\n\n\nSee Also:\n\n    gcp_secrets_update_label.sh - updates all matching labels of a given value with a new value (eg. when migrating Kubernetetes namespaces)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ngcloud secrets list --format=json  |\njq -r '\n    .[] |\n    select(.labels) |\n    {\"name\": .name, \"label\": (.labels | to_entries[]) } |\n    [.name, .label.key + \"=\" + .label.value] |\n    @tsv' |\ncolumn -t\n"
  },
  {
    "path": "gcp/gcp_secrets_to_kubernetes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-04 10:55:43 +0100 (Fri, 04 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLoads given list of GCP Secret Manager secrets to the current Kubernetes cluster with the same name\n\nIf no secrets are specified, then finds all secrets in the current project with a label of kubernetes-cluster that\nmatches the current kubectl context's cluster and which do not have the label kubernetes-multi-part-secret set (as\nthese must be combined using gcp_secrets_to_kubernetes_multipart.sh instead)\n\nFor each secret, checks for a label called 'kubernetes-namespace', and if set, then creates the secret in that namespace,\notherwise loads to the current namespace. This can even be a comma separated list of namespaces and will create the secret in each one\n\nRemember to execute this from the right GCP project configured to get the right secrets\nand with the right Kubernetes context selected to load to the right cluster\n\nTo avoid concurrency race conditions between kubectl commands this script will isolate the current kubernetes context\nenvironment in this script before beginning the load so that all secrets are loaded to the right cluster regardless of\nany other naive kubernetes processes that might change the global kubectl context to point to a different cluster\n\nSee Also:\n\n    Alternatives:\n\n        Sealed Secrets   - https://github.com/bitnami-labs/sealed-secrets\n\n        External Secrets - https://external-secrets.io/\n\n            both of which are available in my Kubernetes repo - https://github.com/HariSekhon/Kubernetes-configs\n\n    gke_kube_creds.sh - to create the Kubernetes cluster contexts in kubectl for your GKE clusters - you need to be using the correct kubectl context before running this script\n\n    gcp_secrets_to_kubernetes_multipart.sh - for creating more complex multi-key-value secrets\n\n    kubernetes_get_secret_values.sh - for checking what was auto-loaded into a given kubernetes secret\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<secret_name> <secret_name2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nkube_config_isolate\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"$(gcloud config list --format='get(core.project)' || :)\"\nexport CLOUDSDK_CORE_PROJECT=\"${CLOUDSDK_CORE_PROJECT:-$project}\"\nnot_blank \"$CLOUDSDK_CORE_PROJECT\" || die \"ERROR: no project specified and \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\n\n# there's no -o jsonpath / -o namespace / -o cluster as of Kubernetes 1.15 so have to just print columns\nkubectl_context=\"$(kubectl config get-contexts \"$(kubectl config current-context)\" --no-headers)\"\ncurrent_cluster=\"$(awk '{print $3}' <<< \"$kubectl_context\")\"\ncurrent_namespace=\"$(awk '{print $5}' <<< \"$kubectl_context\")\"\ncurrent_namespace=\"${current_namespace:-default}\"\n\nload_secret(){\n    local secret=\"$1\"\n    local namespace\n    namespaces=\"$(gcloud secrets describe \"$secret\" --format='get(labels.kubernetes-namespace)')\"\n    namespaces=\"${namespaces:-$current_namespace}\"\n    tr ',' '\\n' <<< \"$namespaces\" |\n    while read -r namespace; do\n        [ -z \"$namespace\" ] && continue\n        namespace=\"${NAMESPACE_OVERRIDE:-$namespace}\"\n        if kubectl get secret \"$secret\" -n \"$namespace\" &>/dev/null; then\n            timestamp \"kubernetes secret '$secret' already exists in namespace '$namespace', skipping creation...\"\n            return\n        fi\n        ## secrets created without a value are an odd use case but it has happened, so ignore and load blank value\n        #latest_version=\"$(get_latest_secret_version \"$secret\" || :)\"\n        #if [ -n \"$latest_version\" ]; then\n        #    value=\"$(gcloud secrets versions access \"$latest_version\" --secret=\"$secret\")\"\n        #else\n        #    timestamp \"WARNING: no versions found for GCP secret '$secret', using blank secret value\"\n        #    value=\"\"\n        #fi\n        value=\"$(\"$srcdir/gcp_secret_get.sh\" \"$secret\")\"\n        timestamp \"creating kubernetes secret '$secret' in namespace '$namespace'\"\n        # kubectl create secret automatically base64 encodes the $value\n        # if you did this in yaml you'd have to base64 encode it yourself in the yaml\n        #         could alternatively make this --from-literal=\"value=$value\"\n        kubectl create secret generic \"$secret\" --from-literal=\"$secret=$value\" -n \"$namespace\"\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for arg; do\n        load_secret \"$arg\"\n    done\nelse\n    gcloud secrets list --format='value(name)' \\\n                        --filter=\"labels.kubernetes-cluster=$current_cluster \\\n                                  AND NOT \\\n                                  labels.kubernetes-multipart-secret ~ . \\\n                                  AND NOT \\\n                                  labels.kubernetes-multi-part-secret ~ .\" |\n    while read -r secret; do\n        load_secret \"$secret\"\n    done\nfi\n"
  },
  {
    "path": "gcp/gcp_secrets_to_kubernetes_multipart.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-04 10:55:43 +0100 (Fri, 04 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLoads a given list of GCP Secret Manager secrets to a single Kubernetes secret\n\nUse the following command to see secrets you may want to combine and load to Kubernetes (eg. jwt-private-pem + jwt-public-pem):\n\n    gcloud secrets list\n\nLoads to the specified explicit Kubernetes namespace since multiple GCP secrets labels might conflict\n\nSee Also:\n\n    Alternatives:\n\n        Sealed Secrets   - https://github.com/bitnami-labs/sealed-secrets\n\n        External Secrets - https://external-secrets.io/\n\n            both of which are available in my Kubernetes repo - https://github.com/HariSekhon/Kubernetes-configs\n\n    gcp_secrets_to_kubernetes.sh - for linear 1-to-1 secret auto-loading\n\n    kubernetes_get_secret_values.sh - for checking what was auto-loaded into a given kubernetes secret\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubernetes_namespace> <kubernetes_secret_name> <gcp_secret_1> [<gcp_secret_2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\nkube_config_isolate\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"$(gcloud config list --format='get(core.project)' || :)\"\nexport CLOUDSDK_CORE_PROJECT=\"${CLOUDSDK_CORE_PROJECT:-$project}\"\nnot_blank \"$CLOUDSDK_CORE_PROJECT\" || die \"ERROR: \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\n\nnamespace=\"$1\"\nkubernetes_secret=\"$2\"\nshift || :\nshift || :\n\nif kubectl get secret \"$kubernetes_secret\" -n \"$namespace\" &>/dev/null; then\n    timestamp \"Kubernetes secret '$kubernetes_secret' already exists in namespace '$namespace', skipping creation...\"\n    exit 0\nfi\n\n# auto base64 encodes the $value - you must base64 encode it yourself if putting it in via yaml\nkubectl_cmd=\"kubectl create secret generic '$kubernetes_secret' -n '$namespace'\"\n\nfor gcp_secret; do\n    value=\"$(\"$srcdir/gcp_secret_get.sh\" \"$gcp_secret\")\"\n\n    # this is all really annoying because there is no right answer to this convention\n    # because GCP Secret Manager won't let you use dots so mangle jwt-private-pem => jwt-private.pem\n    #gcp_secret=\"$(perl -pe 's/[_-]pem$/.pem/' <<< \"$gcp_secret\")\"\n    # actually apps expect just private.pem and public.pem in the secret mounted directory\n    if [[ \"$gcp_secret\" =~ -private-pem$ ]]; then\n        key=\"private.pem\"\n    elif [[ \"$gcp_secret\" =~ -public-pem$ ]]; then\n        key=\"public.pem\"\n    else\n        key=\"$gcp_secret\"\n    fi\n\n    kubectl_cmd+=\" --from-literal='$key=$value'\"\ndone\n\ntimestamp \"creating kubernetes secret '$kubernetes_secret' in namespace '$namespace' from GCP secrets: $*\"\neval \"$kubectl_cmd\"\n"
  },
  {
    "path": "gcp/gcp_secrets_update_label.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-20 15:25:01 +0100 (Tue, 20 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates GCP Secrets in current project with a matching metadata label key=value to a new value\n\nUsed this to track Kubernetes cluster and namespace in GCP Secrets metadata in combination with\ngcp_secrets_to_kubernetes.sh\n\nThis script is useful for say bulk moving secrets to a new namespace by updating all the kubernetes-namespace labels:\n\n    ${0##*/} kubernetes-namespace <oldname> <newname>\n\n\nRequires GCloud SDK to be installed and configured\n\nSee Also:\n\n    gcp_secrets_labels.sh - lists labels one per line to review before/after mass changing label values\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<label_key> <old_value> <new_value>\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\nlabel_key=\"$1\"\nold_value=\"$2\"\nnew_value=\"$3\"\n\ngcloud_export_active_configuration\n\nexport CLOUDSDK_CORE_PROJECT=\"${CLOUDSDK_CORE_PROJECT:-$(gcloud info --format=\"get(config.project)\")}\"\n\nsecrets=\"$(gcloud secrets list --filter \"labels.$label_key=$old_value\" --format='value(name)')\"\n\nfor secret in $secrets; do\n    gcloud secrets update \"$secret\" --update-labels \"$label_key=$new_value\"\ndone\n"
  },
  {
    "path": "gcp/gcp_service_account_credential_to_secret.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-16 17:02:01 +0000 (Mon, 16 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP service account and exports a credentials key to Google Secret Manager\n\nThis is useful because it can be repeatedly sourced from there, eg. loaded to Kubernetes\n\nShould use the service account's short name (the prefix before the @ symbol)\n\nSee Also:\n\n    gcp_secrets_to_kubernetes.sh\n\nIdempotent - skips service account creation if already exists and credential key export if already exists in Google Secret Manager\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<service_account_name> [<project> <description>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"$1\"\n# will reconstruct the full id / service account email using the project and naming convention\nname=\"${name%%@*}\"\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"$(gcloud config list --format='get(core.project)' || :)\"\nexport CLOUDSDK_CORE_PROJECT=\"${CLOUDSDK_CORE_PROJECT:-$project}\"\nnot_blank \"$CLOUDSDK_CORE_PROJECT\" || die \"ERROR: \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\n\ndescription=\"${3:-}\"\n\nkeyfile=\"/tmp/$name-$project-credential.json.$$\"\ntrap_cmd \"rm -f -- '$keyfile'\"\n\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\nif gcloud iam service-accounts list --format='get(email)' | grep -Fxq \"$service_account\"; then\n    timestamp \"Service account '$service_account' already exists\"\nelse\n    timestamp \"Creating service account '$name' in project '$project'\"\n    gcloud iam service-accounts create \"$name\" --description=\"$description\"\nfi\n\nsecret_name=\"${name}-credential\"\nif gcloud secrets list --format='value(name)' | grep -Fxq \"$secret_name\"; then\n    timestamp \"GCP Secret '$secret_name' already exists\"\nelse\n    timestamp \"Exporting service account '$name' credential key to GCP Secret '$secret_name' in project '$project'\"\n    gcloud iam service-accounts keys create \"$keyfile\" --iam-account=\"$service_account\" --key-file-type=\"json\"\n    gcloud secrets create \"$secret_name\" --data-file=\"$keyfile\"\nfi\n"
  },
  {
    "path": "gcp/gcp_service_account_members.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-22 16:34:20 +0100 (Thu, 22 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists one or all GCP service accounts with their members and roles\n\nUseful for finding which users have ability to impersonate a service account and more importantly for GKE Workload Identity mappings\n\nService accounts must be given by email as standard, eg:\n\n    <name>@<project>.iam.gserviceaccount.com\n\nOutput:\n\n    <service_account_email>     <member>      <role>\n\neg.\n\n<name>@<project>.iam.gserviceaccount.com  serviceAccount:jenkins@<project>.iam.gserviceaccount.com    roles/iam.serviceAccountUser\n<name>@<project>.iam.gserviceaccount.com  serviceAccount:<project>.svc.id.goog[k8s_namespace/k8s_sa]  roles/iam.workloadIdentityUser\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<service_account_email1> <service_account_email2>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    tr ' ' '\\n' <<< \"$@\"\nelse\n    gcloud iam service-accounts list --format='get(email)'\nfi |\nwhile read -r service_account_email; do\n    gcloud iam service-accounts get-iam-policy \"$service_account_email\" --format=json |\n    jq -r \".bindings[]? | [ \\\"$service_account_email\\\", .members[]?, .role] | @tsv\"\ndone #|\n# tidier but delays output - can pipe to this column yourself if you want this trade off\n#column -t\n"
  },
  {
    "path": "gcp/gcp_service_accounts_credential_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-28 14:02:13 +0000 (Wed, 28 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList all service account credential keys in the current GCP project along with their creation and expiry dates\n\nExcludes built-in system managed keys which are hidden in the Console UI anyway and are not actionable\nor in scope for a key policy audit.\n\nTo get a list of all user managed keys with no expiry set, you can simply grep the output from this script:\n\n    ${0##*/} | grep 9999-12-31T23:59:59Z\n\n\nOutput Format:\n\n<key_id>  <created_date>  <expiry_date>  <service_account_email>\n\n\nRequires GCloud SDK to be installed and configured for your project\n\n\nSee Also:\n\n    gcp_service_account_credential_keys.py\n\nin the DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nservice_accounts=\"$(gcloud iam service-accounts list --format='get(email)')\"\n\nfor service_account in $service_accounts; do\n    gcloud iam service-accounts keys list --iam-account \"$service_account\" \\\n                                          --format='table[no-heading](name.basename(), validAfterTime, validBeforeTime)' \\\n                                          --filter='keyType != SYSTEM_MANAGED' |\n    # suffixing is better for alignment as service account email lengths are the only variable field and otherwise\n    # this comes out all misaligned or we have to pipe through column -t with no progress output,\n    # leaving appearance of a long O(n) hang before results\n    #sed \"s/^/$service_account    /\"\n    sed \"s/$/  $service_account/\"\ndone\n"
  },
  {
    "path": "gcp/gcp_service_accounts_credential_keys_age.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-28 14:02:13 +0000 (Wed, 28 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList all service account credential keys age in the current GCP project\n\nExcludes built-in system managed keys which are hidden in the Console UI anyway and are not actionable\nor in scope for a key policy audit.\n\n\nOutput Format:\n\n<key_id>  <age_in_days>  <service_account_email>\n\n\nRequires GCloud SDK to be installed and configured for your project\n\n\nSee Also:\n\n    gcp_service_account_credential_keys.py\n\nin the DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nservice_accounts=\"$(gcloud iam service-accounts list --format='get(email)')\"\n\nnow=\"$(date '+%s')\"\n\nfor service_account in $service_accounts; do\n    gcloud iam service-accounts keys list --iam-account \"$service_account\" \\\n                                          --format='table[no-heading](name.basename(), validAfterTime)' \\\n                                          --filter='keyType != SYSTEM_MANAGED' |\n    while read -r id creation_date; do\n        creation_epoch=\"$(date --date \"$creation_date\" '+%s')\"\n        age_days=$(( (now - creation_epoch) / 86400))\n        printf '%s  %4d\\n' \"$id\" \"$age_days\"\n    done |\n    # suffixing is better for alignment as service account email lengths are the only variable field and otherwise\n    # this comes out all misaligned or we have to pipe through column -t with no progress output,\n    # leaving appearance of a long O(n) hang before results\n    #sed \"s/^/$service_account    /\"\n    sed \"s/$/  $service_account/\"\ndone\n"
  },
  {
    "path": "gcp/gcp_service_accounts_credential_keys_expired.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-28 14:02:13 +0000 (Wed, 28 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList all service account credential keys that are expired in the current GCP project\n\nExcludes built-in system managed keys which are hidden in the Console UI anyway and are not actionable\nor in scope for a key policy audit, and don't expire\n\n\nOutput Format:\n\n<key_id>  <expiry_date>  <service_account_email>\n\n\nRequires GCloud SDK to be installed and configured for your project\n\n\nSee Also:\n\n    gcp_service_account_credential_keys.py\n\nin the DevOps Python tools repo:\n\n    https://github.com/HariSekhon/DevOps-Python-tools/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nservice_accounts=\"$(gcloud iam service-accounts list --format='get(email)')\"\n\nnow=\"$(date '+%s')\"\n\nfor service_account in $service_accounts; do\n    gcloud iam service-accounts keys list --iam-account \"$service_account\" \\\n                                          --format='table[no-heading](name.basename(), validBeforeTime)' \\\n                                          --filter='keyType != SYSTEM_MANAGED' |\n    while read -r id expiry_date; do\n        expiry_epoch=\"$(date --date \"$expiry_date\" '+%s')\"\n        if [ \"$expiry_epoch\" -le \"$now\" ]; then\n            printf '%s  %s\\n' \"$id\" \"$expiry_date\"\n        fi\n    done |\n    # suffixing is better for alignment as service account email lengths are the only variable field and otherwise\n    # this comes out all misaligned or we have to pipe through column -t with no progress output,\n    # leaving appearance of a long O(n) hang before results\n    #sed \"s/^/$service_account    /\"\n    sed \"s/$/  $service_account/\"\ndone\n"
  },
  {
    "path": "gcp/gcp_service_apis.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-26 16:59:50 +0100 (Wed, 26 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList GCP Services, APIs and their states:\n\nOutput:\n\n<state>     <api>   <name>\n\nUseful to check whether a service API, used by gcp_info_services.sh\n\nContains the function is_service_enabled() for sourcing - takes as an argument the API name eg. for GKE specify\n\n    is_service_enabled container.googleapis.com\n\nwhere container.googleapis.com is the API corresponding to GKE, as you can see in the output from this script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# Don't change order of headings here as is_services_enabled() below depends on this\n\nservices_list=\"$(gcloud services list --available --format \"table[no-heading](state, config.name, config.title)\")\"\n\necho \"$services_list\"\n\nis_service_enabled(){\n    # must be the api path, eg. file.googleapis.com\n    local service=\"$1\"\n    service=\"${service//./\\.}\"  # escape dots for grep\n    grep -Eqi \"^ENABLED[[:space:]]+$service\" <<< \"$services_list\"\n}\n"
  },
  {
    "path": "gcp/gcp_spinnaker_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-02 17:09:07 +0100 (Wed, 02 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP service account for Spinnaker deployments in the current or specified GCP project, then\ncreates and downloads the credentials json and even prints the commands to configure your Spinnaker Halyard CLI:\n\nhal config storage gcs edit --project \\$PROJECT --bucket-location \\$BUCKET_LOCATION --json-path \\$KEYFILE\nhal config storage edit --type gcs\n\nhal config artifact gcs account add \\$PROJECT --json-path \\$KEYFILE\nhal config artifact gcs enable\n\n\nThe following optional arguments can be given:\n\n- credential file path          (default: \\$HOME/.gcloud/spinnaker-\\$project-credential.json)\n- project                       (default: \\$CLOUDSDK_CORE_PROJECT or gcloud config's currently configured project setting core.project)\n\nIdempotent - safe to re-run, will skip service accounts and keyfiles that already exist\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential.json> <project>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nname=\"spinnaker\"\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"${2:-${CLOUSDK_CORE_PROJECT:-$(gcloud config list --format='get(core.project)')}}\"\n\nnot_blank \"$project\" || die \"ERROR: no project specified and \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\nkeyfile=\"${1:-$HOME/.gcloud/$name-$project-credential.json}\"\n\ngcp_create_serviceaccount_if_not_exists \"$name\" \"$project\" \"Spinnaker service account for deployments\"\n\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\ngcp_create_credential_if_not_exists \"$service_account\" \"$keyfile\"\n\n# XXX: this could probably be limited to just the bucket rather than the entire project, this is how Spinnaker docs do it, so unclear if there are further requirements for this high privileges\necho \"Granting Storage Admin permissions to service account '$service_account' on project '$project'\"\n# some projects may require --condition=None in non-interactive mode\ngcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role roles/storage.admin --condition=None >/dev/null\n\nkeyfile=\"$(readlink -e \"$keyfile\")\"\n\ncat <<EOF\n\n\nYou can now continue to configure Spinnaker Halyard CLI with these details:\n\nNeeded to store Application settings + Pipelines:\n\nhal config storage gcs edit --project \"$project\" --bucket-location \"\\$BUCKET_LOCATION\" --json-path \"$keyfile\"\nhal config storage edit --type gcs\n\nOptional - only needed if using artifacts stored on GCS:\n\nhal config artifact gcs account add \"$project\" --json-path \"$keyfile\"\nhal config artifact gcs enable\nEOF\n"
  },
  {
    "path": "gcp/gcp_sql_backup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 09:58:18 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cloud.google.com/sql/docs/postgres/backup-recovery/backups\n#\n# https://cloud.google.com/sql/docs/postgres/backup-recovery/backing-up#gcloud\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a manual backup of one or more Cloud SQL database instances in the current project\n\nSQL instances can optionally be specified, otherwise iterates all running non-replica SQL instances\n\nTo enabled automated daily backups, run the adjacent script:\n\n    gcp_sql_enable_automated_backups.sh\n\nCloud SQL backups are deleted with SQL instances so while these are convenient, you should also do full exports to GCS using the adjacent script:\n\n    gcp_sql_export.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    # XXX: backups cannot be run for stopped instances or read replicas:\n    # ERROR: (gcloud.sql.backups.create) HTTPError 400: This operation is not valid for this instance.\n    # ERROR: (gcloud.sql.backups.create) HTTPError 400: Backups cannot be enabled for read replica instances.\n    sql_instances=\"$(gcloud sql instances list --format=json |\n                     jq -r '.[] | select(.instanceType != \"READ_REPLICA_INSTANCE\") | select(.state == \"RUNNABLE\") | .name')\"\nfi\n\nfor sql_instance in $sql_instances; do\n    timestamp \"Backing up SQL instance '$sql_instance':\"\n    gcloud sql backups create --instance=\"$sql_instance\"  # --async\n    echo >&2\ndone\ntimestamp \"Cloud SQL Backups Completed\"\n"
  },
  {
    "path": "gcp/gcp_sql_create_readonly_service_account.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-16 12:01:31 +0100 (Fri, 16 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP service account in the current project called \\${GOOGLE_SERVICE_ACCOUNT}@\\${project_id}.iam.gserviceaccount.com\n\nGOOGLE_SERVICE_ACCOUNT defaults to 'cloud-function-sql-backup'\n\nGrants it permissions:\n\n    - Cloud SQL Client\n    - Cloud SQL Viewer\n\n\nThis is necessary to set up Cloud SQL export backups to GCS using the adjacent scripts. See\n\n    gcp_cloud_schduler_sql_exports.sh\n\nSee the Cloud Function at:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\nThis script is idempotent and safe to re-run even if the service account already exists and has permissions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nproject=\"$(gcloud config list --format=\"value(core.project)\")\"\nnot_blank \"$project\" || die \"ERROR: GCloud SDK core.project property not set\"\nname=\"${GOOGLE_SERVICE_ACCOUNT:-cloud-function-sql-backup}\"\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\nif ! gcloud iam service-accounts list | grep -q \"$service_account\"; then\n    timestamp \"Creating GCP service account '$service_account'\"\n    gcloud iam service-accounts create \"$name\" --description \"Exports Cloud SQL data to GCS\"\n    echo >&2\nfi\n\ntimestamp \"Granting CloudSQL Client to service account '$service_account'\"\ngcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role=roles/cloudsql.client\necho >&2\n\ntimestamp \"Granting CloudSQL Client to service account '$service_account'\"\ngcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role=roles/cloudsql.viewer\necho >&2\n\n# XXX: need to grant the Cloud SQL instance service accounts objectCreator to the bucket, not this cloud function's service account\n#      see instead adjacent script:\n#\n#      gcp_sql_grant_instances_gcs_object_creator.sh\n#\n#timestamp \"Granting Storage Object Creator on bucket '$bucket' to service account '$service_account'\"\n#gsutil iam ch \"serviceAccount:$service_account:objectCreator\" \"gs://$bucket\"\n"
  },
  {
    "path": "gcp/gcp_sql_enable_automated_backups.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 09:58:18 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cloud.google.com/sql/docs/postgres/backup-recovery/backups\n#\n# https://cloud.google.com/sql/docs/postgres/backup-recovery/backing-up#gcloud\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnable Cloud SQL automated daily backups\n\nSQL instances can optionally be specified, otherwise iterates all running non-replica SQL instances\n\nTo take an immediate backup, run the adjacent script:\n\n    gcp_sql_backup.sh\n\nCloud SQL backups are deleted with SQL instances so while these are convenient, you should also do full exports to GCS using the adjacent script:\n\n    gcp_sql_export.sh\n\n\nBackup maintenance windows are 4 hour blocks - set the CLOUD_SQL_BACKUP_START_TIME environment variable in HH:MM format (UTC) to change it from the default 01:00 am\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    # XXX: backups cannot be run for read replicas, and nor can this be enabled for stopped instances which would result in:\n    # ERROR: (gcloud.sql.instances.patch) HTTPError 400: The incoming request contained invalid data.\n    sql_instances=\"$(gcloud sql instances list --format=json |\n                     jq -r '.[] | select(.instanceType != \"READ_REPLICA_INSTANCE\") | select(.state == \"RUNNABLE\") | .name')\"\nfi\n\nbackup_start_time=\"${CLOUD_SQL_BACKUP_START_TIME:-01:00}\"\n\nfor sql_instance in $sql_instances; do\n    timestamp \"Enabling automated daily backups for SQL instance '$sql_instance' at '$backup_start_time' UTC:\"\n    gcloud sql instances patch \"$sql_instance\" --backup-start-time \"$backup_start_time\"\n    echo >&2\ndone\ntimestamp \"Cloud SQL automated backups configured for '$backup_start_time' UTC\"\n"
  },
  {
    "path": "gcp/gcp_sql_enable_point_in_time_recovery.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 09:58:18 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cloud.google.com/sql/docs/postgres/backup-recovery/pitr#gcloud\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnable Cloud SQL point-in-time recovery\n\nSQL instances can optionally be specified, otherwise iterates all running non-replica SQL instances\n\nThis automatically creates a backup at the time of enabling, which takes a few minutes during which your database may be unavailable.\n\nWARNING: Enabling/Disabling point-in-time recovery restarts the instance, causing an outage, so this script will prompt for confirmation before proceeding.\n\nRequires automated backups to already be enabled, otherwise you'll get this error:\n\n    ERROR: (gcloud.sql.instances.patch) HTTPError 400: Invalid request: Point in time recovery must be disabled when backup is disabled.\n\nSee adjacent script to enable automated backups first:\n\n    gcp_sql_enable_automated_backups.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    # XXX: backups cannot be run for read replicas, and nor can this be enabled for stopped instances which would result in:\n    # ERROR: (gcloud.sql.instances.patch) HTTPError 400: The incoming request contained invalid data.\n    sql_instances=\"$(gcloud sql instances list --format=json |\n                     jq -r '.[] | select(.instanceType != \"READ_REPLICA_INSTANCE\") | select(.state == \"RUNNABLE\") | .name')\"\nfi\n\necho \"Cloud SQL instances on which point-in-time recovery will be enabled:\"\necho\nfor sql_instance in $sql_instances; do\n    echo \"$sql_instance\"\ndone\necho\necho\necho \"WARNING: Enabling or disabling point-in-time recovery restarts the instance, causing an outage\"\necho\necho \"WARNING: Do not do this on production databases during business hours\"\necho\nread -r -p \"Are you sure you want to proceed to enable point-in-time recovery including restarting the above Cloud SQL instances? (y/N) \" answer\necho\nif ! [[ \"$answer\" =~ ^(y|yes)$ ]]; then\n    echo \"Aborting...\"\n    exit 1\nfi\n\nfor sql_instance in $sql_instances; do\n    timestamp \"Enabling point-in-time recovery for SQL instance '$sql_instance':\"\n    gcloud sql instances patch \"$sql_instance\" --enable-point-in-time-recovery\n    echo >&2\ndone\ntimestamp \"Cloud SQL instances point-in-time recovery enabled\"\n"
  },
  {
    "path": "gcp/gcp_sql_export.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 09:58:18 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cloud.google.com/sql/docs/postgres/import-export/exporting\n#\n# https://cloud.google.com/sql/docs/postgres/import-export\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExports all running non-replica SQL database instances in the current project to GCS\n\nGCS bucket name must be specified\n\nSQL instances can optionally be specified, otherwise iterates all running non-replica SQL instances\n(only running instances can export, otherwise will error out)\n(only non-replicas should be exported, replicas will likely fail due to conflict with replication recovery)\n\nAll databases for each SQL instance will be exported to the GCS bucket with file names in the format:\n\n    gs://<bucket>/backups/sql/<sql_instance>--<database_name>--<date_timestamp>.sql.gz\n\nRequirements:\n\n- GCS bucket must already exist\n- Each SQL instance's service account will be granted the minimal role Storage Object Creator to the specified bucket to allow the export to succeed\n\n\n- Do not back up Replicas as it the export will likely conflict with the ongoing recovery operation like so:\n\n    ERROR: (gcloud.sql.export.sql) [ERROR_RDBMS] pg_dump: Dumping the contents of table \\\"<myTable>\\\" failed: PQgetResult() failed.\n    pg_dump: Error message from server: ERROR:  canceling statement due to conflict with recovery\n\n\nFor busy heavily utilized production databases this may put a strain on their resources or take a long time due to contention leading to the export command erroring out with a timeout waiting like so:\n\n    Exporting Cloud SQL instance...failed.\n    ERROR: (gcloud.sql.export.sql) Operation https://sqladmin.googleapis.com/sql/v1beta4/projects/<project>/operations/<instance_id> is taking longer than expected. You can continue waiting for the operation by running \\`gcloud beta sql operations wait --project <project> <instance_id>\\`\n\n\nIn this case your options are modify the export command to use --async or use the --offload flag to use serverless export (will take 5 minutes longer to spin up a replica and additional charges will apply)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<gcs_bucket> [<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nmin_args 1 \"$@\"\n\ngcs_bucket=\"$1\"\nshift || :\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    sql_instances=\"$(\"$srcdir/gcp_sql_running_primaries.sh\")\"\nfi\n\ntimestamp \"Granting SQL instance(s) objectCreator on GCS bucket '$gcs_bucket'\"\n# want splitting\n# shellcheck disable=SC2086\n\"$srcdir/gcp_sql_grant_instances_gcs_object_creator.sh\" \"$gcs_bucket\" $sql_instances\necho >&2\n\ntimestamp \"Exporting SQL instance(s) to GCS bucket '$gcs_bucket'\"\nfor sql_instance in $sql_instances; do\n    echo >&2\n    timestamp \"Getting list of databases for SQL instance '$sql_instance'\"\n    databases=\"$(gcloud sql databases list --instance=\"$sql_instance\" --format='get(name)')\"\n    for database in $databases; do\n        # skip information schema, not allowed to dump this on MySQL, fails with:\n        # ERROR: (gcloud.sql.export.sql) [ERROR_RDBMS] mysqldump: Dumping 'information_schema' DB content is not supported\n        [ \"$database\" = \"information_schema\" ] && continue\n        # skip these MySQL built-in DBs too\n        [ \"$database\" = \"sys\" ] && continue\n        [ \"$database\" = \"performance_schema\" ] && continue\n        timestamp \"Exporting SQL instance '$sql_instance' database '$database'\"\n        # adding .gz will auto-encrypt the bucket\n        gcloud sql export sql \"$sql_instance\" \"gs://$gcs_bucket/backups/sql/$sql_instance--$database--$(date '+%F_%H%M').sql.gz\" --database \"$database\"  # --offload\n    done\ndone\necho >&2\ntimestamp \"Cloud SQL exports to GCS bucket '$gcs_bucket' completed\"\n"
  },
  {
    "path": "gcp/gcp_sql_grant_instances_gcs_object_creator.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-16 12:01:31 +0100 (Fri, 16 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGrants all running non-replica SQL database instances in the current project objectCreator access to a GCS bucket for backup exports\n\nCan specify an explicit list to grant if wanting to grant to non-running instances or only a subset of instances\n\nThis is necessary to back up Cloud SQL to GCS using the adjacent scripts. See\n\n    gcp_sql_export.sh\n    gcp_cloud_scheduler_sql_exports.sh\n\nSee the Cloud Function at:\n\n    https://github.com/HariSekhon/DevOps-Python-tools\n\n\nThis script is idempotent and safe to re-run\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<gcs_bucket> [<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nmin_args 1 \"$@\"\n\ngcs_bucket=\"$1\"\nshift || :\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    sql_instances=\"$(\"$srcdir/gcp_sql_running_primaries.sh\")\"\nfi\n\n# Need to grant the Cloud SQL instance service accounts objectCreator to the bucket, not this cloud function account\n#timestamp \"Granting Storage Object Creator on bucket '$bucket' to service account '$service_account'\"\n#gsutil iam ch \"serviceAccount:$service_account:objectCreator\" \"gs://$bucket\"\n\nfor sql_instance in $sql_instances; do\n    service_account=\"$(gcloud sql instances describe \"$sql_instance\" --format='get(serviceAccountEmailAddress)')\"\n    timestamp \"Granting instance '$sql_instance' service account '$service_account' objectCreator role to the backup bucket '$gcs_bucket'\"\n    gsutil iam ch \"serviceAccount:$service_account:objectCreator\" \"gs://$gcs_bucket\"\n    echo >&2\ndone\ntimestamp \"Cloud SQL grants to GCS bucket '$gcs_bucket' completed\"\n"
  },
  {
    "path": "gcp/gcp_sql_list_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-14 18:28:03 +0100 (Wed, 14 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Cloud SQL instances and their databases in the current GCP project\n\nCan specify one or more Cloud SQL instances, otherwise finds and iterates all instances in the current project\n\n\nOutput Format:\n\n<host_sql_instance>     <database_name>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    # XXX: can only list databases for running instances, otherwise:\n    #      ERROR: (gcloud.sql.databases.list) HTTPError 400: Invalid request: Invalid request since instance is not running.\n    sql_instances=\"$(gcloud sql instances list --format=json | jq -r '.[] | select(.state == \"RUNNABLE\") | .name')\"\nfi\n\nfor sql_instance in $sql_instances; do\n    #gcloud sql databases list --instance=\"$sql_instance\" --format='get(name)'\n    gcloud sql databases list --instance=\"$sql_instance\" --format='table[no-heading](instance, name)'\ndone\n"
  },
  {
    "path": "gcp/gcp_sql_proxy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-01 15:37:49 +0000 (Mon, 01 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nSOCKDIR=~/cloud_sql.socks\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens a Cloud SQL Proxy for all instances in given GCP Project(s) or all GCP Projects\n\nPopulates \\$HOME/cloud_sql.socks/ directory with all your Cloud SQL instances sockets\n\nMakes it quicker and easier to connect than slow 'gcloud sql connect' commands (which also requires a public IP address attached to your SQL instance)\n\n\nUsage:\n\n    ${0##*/}\n\n    psql -h ~/cloud_sql.socks/<instance> ...\n\n    mysql -S ~/cloud_sql.socks/<instance> ...\n\n\n\nAuto-installs Google Cloud SQL Proxy if not found in \\$PATH\n\n\nRestart to pick up any new Cloud SQL instances\n\nUseful to quick interactive DBA work\n\nFor production long-lived proxying, use dedicated Cloud SQL Proxy instances with service account credentials (avoid application default login expiry or restarts for picking up new instances)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<projects>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    projects=\"${*:-}\"\nelse\n    projects=\"$(gcloud projects list --format='get(project_id)')\"\nfi\nnot_blank \"$projects\" || die \"ERROR: no project specified and GCloud SDK core.project property not set\"\n\nexport PATH=\"$PATH:\"~/bin\n\nif ! type -P cloud_sql_proxy &>/dev/null; then\n    \"$srcdir/../install/install_cloud_sql_proxy.sh\"\nfi\n\nmkdir -p -v \"$SOCKDIR\"\n\nprojects=\"${projects//[[:space:]]/,}\"\n\n# prompt for Application Default credentials if not already found\nif ! gcloud auth application-default print-access-token &>/dev/null; then\n    gcloud auth application-default login\nfi\n\ncmd=(cloud_sql_proxy -projects \"$projects\" -dir \"$SOCKDIR\")\n\necho \"${cmd[*]}\"\n\"${cmd[@]}\"\n"
  },
  {
    "path": "gcp/gcp_sql_running_primaries.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-16 12:01:31 +0100 (Fri, 16 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all running non-replica Cloud SQL instance names\n\nUseful to get a list to iterate on for backups, exports, grants etc.\n\nUsed by adjacent scripts:\n\n    gcp_sql_export.sh\n    gcp_sql_grant_instances_gcs_object_creator.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# XXX: only running instances can do exports, otherwise will error out\n# XXX: only non-replicas back up correctly due to conflicts with ongoing replication recovery operations - see description above for details\n# gcloud sql instances list --format='get(name)' --filter 'STATUS=runnable' | grep -v -- '-replica$'\n# better to not rely on the name having a '-replica' suffix and instead use the JSON instanceType field to exclude replicas\ngcloud sql instances list --format=json | jq -r '.[] | select(.instanceType != \"READ_REPLICA_INSTANCE\") | select(.state == \"RUNNABLE\") | .name'\n"
  },
  {
    "path": "gcp/gcp_sql_service_accounts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-13 09:58:18 +0100 (Tue, 13 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists service accounts for each Cloud SQL instance in the current project\n\nUse this list to copy in to IAM and grant Storage Object Creator on a bucket in order to run backup SQL Exports using the adjacent gcp_sql_export.sh script\n\nOutput Format:\n\n<sql_instance_name>     <sql_instance_service_account>\n\nSQL instances can be specified as arguments, otherwise lists service accounts for all SQL instances in the current project\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<sql_instance1> <sql_instance2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nsql_instances=\"$*\"\n\nif [ -z \"$sql_instances\" ]; then\n    sql_instances=\"$(gcloud sql instances list --format='get(name)')\"\nfi\n\nfor sql_instance in $sql_instances; do\n    gcloud sql instances describe \"$sql_instance\" --format=json |\n    jq -r '[.name, .serviceAccountEmailAddress] | @tsv'\ndone\n"
  },
  {
    "path": "gcp/gcp_terraform_create_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-28 17:02:13 +0000 (Wed, 28 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP service account for Terraform deployments in the current or specified GCP project, then\ncreates and downloads the credentials json and even prints the command to configure your environment to start using Terraform immediately:\n\nexport GOOGLE_CREDENTIALS=\\$HOME/.gcloud/\\$name-\\$project-credential.json\n\nThe following optional arguments can be given:\n\n- service account name prefix   (default: \\$USER-terraform)\n- credential file path          (default: \\$HOME/.gcloud/\\$name-\\$project-credential.json)\n- project                       (default: \\$CLOUDSDK_CORE_PROJECT or gcloud config's currently configured project setting core.project)\n\nIdempotent - safe to re-run, will skip service accounts and keyfiles that already exist\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<name> <credential.json> <project>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nname=\"${1:-$USER-terraform}\"\n\n# XXX: sets the GCP project for the duration of the script for consistency purposes (relying on gcloud config could lead to race conditions)\nproject=\"${3:-${CLOUDSDK_CORE_PROJECT:-$(gcloud config list --format='get(core.project)')}}\"\n\nnot_blank \"$project\" || die \"ERROR: no project specified and \\$CLOUDSDK_CORE_PROJECT / GCloud SDK config core.project value not set\"\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\nkeyfile=\"${2:-$HOME/.gcloud/$name-$project-credential.json}\"\n\ngcp_create_serviceaccount_if_not_exists \"$name\" \"$project\" \"$USER's service account for Terraform deployments\"\n\nservice_account=\"$name@$project.iam.gserviceaccount.com\"\n\ngcp_create_credential_if_not_exists \"$service_account\" \"$keyfile\"\n\necho \"Granting Owner permissions to service account '$service_account' on project '$project'\"\n# some projects may require --condition=None in non-interactive mode\ngcloud projects add-iam-policy-binding \"$project\" --member=\"serviceAccount:$service_account\" --role=roles/owner --condition=None >/dev/null\n\nkeyfile=\"$(readlink -e \"$keyfile\")\"\n\necho\necho \"Set this in your environment to use Terraform now:\"\necho\necho \"export GOOGLE_CREDENTIALS=$keyfile\"\necho\n"
  },
  {
    "path": "gcp/gcr_alternate_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox:latest\n#  args: gcr.io/google-containers/busybox:latest\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all alternative tags for the given specific GCR docker image:tag\n\nIf a container has multiple tags (eg. latest, v1, hashref), you can supply '<image>:latest' to see which version has been tagged to 'latest'\n\nEach tag for the given <image>:<tag> is output on a separate line for easy further piping and filtering, including the originally supplied tag\n\nIf no tag is given, assumes 'latest'\n\nIf the image isn't found in GCR, will return nothing and no error code since this is the default GCloud SDK behaviour\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>[:<tag>]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage_tag=\"$1\"\n\nif ! [[ \"$image_tag\" =~ gcr\\.io ]]; then\n    image_tag=\"gcr.io/$image_tag\"\nfi\n\n# $gcr_image_tag_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image_tag\" =~ ^($gcr_image_regex|$gcr_image_tag_regex)$ ]]; then\n    usage \"unrecognized GCR image name - should be in a format matching this regex: ^$gcr_image_regex$ or ^$gcr_image_tag_regex$\"\nfi\n\nimage=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\nif [ -z \"$tag\" ] || [ \"$tag\" = \"$image\" ]; then\n    tag=\"latest\"\nfi\n\ngcloud container images list-tags \"$image\" --format='csv[no-heading,delimiter=\"\\n\"](tags[])' --filter=\"tags=$tag\" # |\n#while read -r tag; do\n#    printf \"%s:%s\\n\" \"$image\" \"$tag\"\n#done\n"
  },
  {
    "path": "gcp/gcr_delete_old_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox $((365 * 4))\n#  args: gcr.io/google-containers/busybox $((365 * 4))\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-14 12:13:34 +0000 (Mon, 14 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes tags old than N days for a given GCR image\n\nUseful to clean out old CI image builds to save GCS storage costs on old CI images you no longer use\n\n\nPrompts with the list of image:tags that it will delete before proceeding for safety.\n\n\nSee Also:\n\n    gcr_tags_old.sh        - used by this script, lists all image:tag older than N days\n    gcr_tags_timestamps.sh - lists tags and timestamps - useful for comparing with the output from gcr_tags_old.sh\n\n    gcr_*.sh - scripts for Google Container Registry\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> <days_threshold>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nimage_tags=\"$(\"$srcdir/gcr_tags_old.sh\" \"$@\")\"\n\ndays=\"$2\"\n\nif [ -z \"$image_tags\" ]; then\n    echo \"No image:tags older than $days old\"\n    exit 0\nfi\n\necho\necho \"List of image:tags that will be deleted:\"\necho\necho \"$image_tags\"\necho\n\nread -r -p 'Are you sure you want to delete these image:tags listed above? (y/N) ' answer\necho\n\nif [ \"$answer\" != \"y\" ]; then\n    echo \"Aborting...\"\n    exit 1\nfi\n\nxargs gcloud container images delete -q --force-delete-tags <<< \"$image_tags\"\n"
  },
  {
    "path": "gcp/gcr_list_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox\n#  args: gcr.io/google-containers/busybox\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all tags for the given GCR docker image\n\nEach tag for the given image is output on a separate line for easy further piping and filtering\n\nIf the image isn't found in GCR, will return nothing and no error code since this is the default GCloud SDK behaviour\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>:<tag>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\nif ! [[ \"$image\" =~ gcr\\.io ]]; then\n    image=\"gcr.io/$image\"\nfi\n\n# $gcr_image_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image\" =~ ^$gcr_image_regex$ ]]; then\n    usage \"unrecognized GCR image name - should be in a format matching this regex: ^$gcr_image_regex$\"\nfi\n\ngcloud container images list-tags \"$image\" --format='csv[no-heading,delimiter=\"\\n\"](tags[])' |\n# grep -v changes the error code depending on whether it managed to filter out any blank lines, so preferring sed here\nsed '/^[[:space:]]*$/d'\n"
  },
  {
    "path": "gcp/gcr_newest_image_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox\n#  args: gcr.io/google-containers/busybox\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the tags for the given GCR docker image with the newest creation date\n\n(eg. for tagging it as 'latest', see adjacent scripts gcr_tag_latest.sh and gcr_tag_newest_image_as_latest.sh)\n\nWhen a docker image has multiple tags (eg. v1, latest) then outputs each tag on a separate line for easy further piping and filtering\n\nIf the image isn't found in GCR, will return nothing and no error code since this is the default GCloud SDK behaviour\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\nif ! [[ \"$image\" =~ gcr\\.io ]]; then\n    image=\"gcr.io/$image\"\nfi\n\n# $gcr_image_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image\" =~ ^$gcr_image_regex$ ]]; then\n    usage \"unrecognized GCR image name - should be in a format matching this regex: ^$gcr_image_regex$\"\nfi\n\ngcloud container images list-tags \"$image\" --sort-by=\"~timestamp\" --limit 1 --format='csv[no-heading,delimiter=\"\\n\"](tags[])' # |\n#while read -r tag; do\n#    printf \"%s:%s\\n\" \"$image\" \"$tag\"\n#done\n"
  },
  {
    "path": "gcp/gcr_tag_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox:latest\n#  args: gcr.io/google-containers/busybox:latest\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given GCR docker image:tag with the current branch name without pulling and pushing the docker image\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>:<tag>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage_tag=\"$1\"\n\nif ! [[ \"$image_tag\" =~ gcr\\.io ]]; then\n    image_tag=\"gcr.io/$image_tag\"\nfi\n\n# $gcr_image_tag_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image_tag\" =~ ^$gcr_image_tag_regex$ ]]; then\n    usage \"unrecognized GCR image:tag name - should be in a format matching this regex: ^$gcr_image_tag_regex$\"\nfi\n\ndocker_image=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\n\n# Jenkins provides GIT_BRANCH, TeamCity doesn't so normalize and determine it if not automatically set\nif [ -z \"${BRANCH_NAME:-}\" ]; then\n    BRANCH_NAME=\"${GIT_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}\"\nfi\nBRANCH_NAME=\"${BRANCH_NAME##*/}\"\n\necho \"tagging docker image $docker_image:$tag with branch '$BRANCH_NAME'\"\n# --quiet otherwise prompts Y/n which would hang build\ngcloud container images add-tag --quiet \"$docker_image:$tag\" \"$docker_image:$BRANCH_NAME\"\n"
  },
  {
    "path": "gcp/gcr_tag_datetime.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox:latest\n#  args: gcr.io/google-containers/busybox:latest\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given GCR docker image with it's creation Date and Timestamp\nwithout pulling and pushing the docker image\n\nThe timestamp is the created time (either uploaded or created by Google Cloud Build)\n\nTags are in the format:\n\nYYYY-MM-DD\nYYYY-MM-DDTHHMMSSZ  (standard ISO UTC time without semi-colons which are invalid in docker tags)\n\nThe timestamp will be normalized to UTC\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>[:<tag>]\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage_tag=\"$1\"\n\nif ! [[ \"$image_tag\" =~ gcr\\.io ]]; then\n    image_tag=\"gcr.io/$image_tag\"\nfi\n\n# $gcr_image_optional_tag_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image_tag\" =~ ^$gcr_image_optional_tag_regex$ ]]; then\n    usage \"unrecognized GCR image:tag name - should be in a format matching this regex: ^$gcr_image_optional_tag_regex$\"\nfi\n\ndocker_image=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\n\nif [ -z \"$tag\" ] ||\n   [ \"$tag\" = \"$docker_image\" ] ||\n   [ \"$tag\" = \"$image_tag\" ]; then\n    tag=\"latest\"\nfi\n\ntimestamp=\"$(gcloud container images list-tags \"$docker_image\" --limit 10 --format='get(timestamp.datetime)' --filter=\"tags=$tag\")\"\nif [ -z \"$timestamp\" ]; then\n    echo \"Failed to determine timestamp from GCR for image '$docker_image' with tag '$tag'\"\n    exit 1\nfi\nif ! [[ \"$timestamp\" =~ ^[[:digit:]]{4}-[[:digit:]]{2}-[[:digit:]]{2}[[:space:]][[:digit:]]{2}:[[:digit:]]{2}:[[:digit:]]{2}[+-][[:digit:]]{2}:[[:digit:]]{2}$ ]]; then\n    echo \"ECR timestamp not in expect YYYY-MM-DD HH:MM:SS[+-]HH:MM format, API may have changed\"\n    exit 1\nfi\n\n# normalize to UTC\ntimestamp=\"$(date --utc --date=\"$timestamp\" '+%FT%H%M%SZ')\"\n\ndate=\"${timestamp%T*}\"\n\necho \"tagging docker image $docker_image:$tag with extra tags: $date $timestamp\"\n# --quiet otherwise prompts Y/n which would hang build\ngcloud container images add-tag --quiet \\\n    \"$docker_image:$tag\" \\\n        \"$docker_image:$date\" \\\n        \"$docker_image:$timestamp\"\n        #\"$docker_image:latest\"  # there is also gcr_tag_latest.sh adjacent now\n"
  },
  {
    "path": "gcp/gcr_tag_latest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox:latest\n#  args: gcr.io/google-containers/busybox:latest\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTags a given GCR docker image:tag as 'latest' without pulling and pushing the docker image\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>:<tag>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage_tag=\"$1\"\n\nif ! [[ \"$image_tag\" =~ gcr\\.io ]]; then\n    image_tag=\"gcr.io/$image_tag\"\nfi\n\n# $gcr_image_tag_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image_tag\" =~ ^$gcr_image_tag_regex$ ]]; then\n    usage \"unrecognized GCR image:tag name - should be in a format matching this regex: ^$gcr_image_tag_regex$\"\nfi\n\ndocker_image=\"${image_tag%%:*}\"\ntag=\"${image_tag##*:}\"\n\necho \"tagging docker image $docker_image:$tag as 'latest'\"\n# --quiet otherwise prompts Y/n which would hang build\ngcloud container images add-tag --quiet \"$docker_image:$tag\" \"$docker_image:latest\"\n"
  },
  {
    "path": "gcp/gcr_tag_newest_image_as_latest.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox\n#  args: gcr.io/google-containers/busybox\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 17:17:35 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the newest build of a given GCR docker image by creation date and tags it as 'latest'\n\nDoes this via metadata API calls to avoid network transfer from any docker pull / docker push\n\nIf a GCR image has multiple tags, will take the longest tag which is assumed to be the most specific and\ntherefore most likely to avoid collisions and race conditions of other tag updates happening concurrently\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\nif ! [[ \"$image\" =~ gcr\\.io ]]; then\n    image=\"gcr.io/$image\"\nfi\n\n# $gcr_image_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image\" =~ ^$gcr_image_regex$ ]]; then\n    usage \"unrecognized GCR image name - should be in a format matching this regex: ^$gcr_image_regex$\"\nfi\n\ntags=\"$(\"$srcdir/gcr_newest_image_tags.sh\" \"$@\")\"\n\nif [ -z \"$tags\" ]; then\n    die \"No tags were found for image '$image'... does it exist in GCR?\"\nfi\n\nlongest_tag=\"$(awk '{print length, $0}' <<< \"$tags\" |\n               sort -nr |\n               head -n 1 |\n               awk '{print $2}')\"\n\n\"$srcdir/gcr_tag_latest.sh\" \"$image:$longest_tag\"\n"
  },
  {
    "path": "gcp/gcr_tags_old.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox\n#  args: gcr.io/google-containers/busybox\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-14 12:13:34 +0000 (Mon, 14 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists old tags for a given GCR image > \\$days old\n\nThe \\$days threshold defaults to (365 * 2) ie. 2 years old\n\nYou can grep and pipe this output to\n\n    | xargs gcloud container images delete -q --force-delete-tags\n\nto clean out old CI image builds to save GCS storage costs on old CI images you no longer use\n\n\nSee Also:\n\n    gcr_tags_timestamps.sh - lists tags and timestamps - useful for comparing with the output from this script\n\n    gcr_*.sh - scripts for Google Container Registry\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> [<days_threshold>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nimage=\"$1\"\n\nif ! [[ \"$image\" =~ gcr\\.io ]]; then\n    image=\"gcr.io/$image\"\nfi\n\n# $gcr_image_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image\" =~ ^$gcr_image_regex$ ]]; then\n    usage \"unrecognized gcr image name - should be in a format matching this regex: ^$gcr_image_regex$\"\nfi\n\n\n# 2 years old images by default\ndays_threshold=\"${2:-$((365 * 2))}\"\n\ndate_threshold=\"$(date --date=\"$days_threshold days ago\")\"\n\ngcloud container images list-tags \"$image\" \\\n                                  --limit 999999 \\\n                                  --sort-by=TIMESTAMP \\\n                                  --filter=\"timestamp.datetime < '$date_threshold'\" \\\n                                  --format=json |\njq -r \"map(\\\"$image\\\" + \\\":\\\" + .tags[]) | .[]\"\n"
  },
  {
    "path": "gcp/gcr_tags_timestamps.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: google-containers/busybox\n#  args: gcr.io/google-containers/busybox\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-15 14:52:47 +0100 (Tue, 15 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gcp.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all tags for the given GCR docker image as well as their image upload timestamp\n\nOutput Format:\n\n<timestamp>   <tag>\n\nEach timestamp and tag for the given image is output tab separated on a separate line for easy further piping and filtering\n\nIf the image isn't found in GCR, will return nothing and no error code since this is the default GCloud SDK behaviour\n\n\nSimilar scripts:\n\n    aws_ecr_*.sh - scripts for AWS Elastic Container Registry\n\n    gcr_*.sh - scripts for Google Container Registry\n\n\nRequires GCloud SDK to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[gcr.io/]<project_id>/<image>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\nif ! [[ \"$image\" =~ gcr\\.io ]]; then\n    image=\"gcr.io/$image\"\nfi\n\n# $gcr_image_regex is defined in lib/gcp.sh\n# shellcheck disable=SC2154\nif ! [[ \"$image\" =~ ^$gcr_image_regex$ ]]; then\n    usage \"unrecognized GCR image name - should be in a format matching this regex: ^$gcr_image_regex$\"\nfi\n\ngcloud container images list-tags \"$image\" --format=json |\njq -r 'map(.timestamp.datetime + \"\\t\" + .tags[]) | .[]'\n"
  },
  {
    "path": "gcp/gcs_bucket_project.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 01:00:11 +0000 (Fri, 16 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the GCP Storage JSON API to find which GCP Project a given GCS bucket belongs to\n\nOutput:\n\n<project_id>    <project_name>\n\nRequires GCloud SDK and jq to be installed as well as GCloud SDK being already authenticated with an account with permission to storage.buckets.get and resourcemanager.projects.get in the project where the bucket lives (use a GCP owner account which has access to all your projects)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<bucket_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nbucket=\"$1\"\n\naccess_token=\"$(gcloud auth print-access-token || die \"Failed to get GCP access token - you must be authenticated first\")\"\n\nexport CURL_OPTS='-sS'\nexport TOKEN=\"$access_token\"\n\n# https://cloud.google.com/storage/docs/json_api/v1/buckets/get\nproject_number=\"$(\n    \"$srcdir/../bin/curl_auth.sh\" $CURL_OPTS \"https://storage.googleapis.com/storage/v1/b/$bucket\" |\n    jq -r '.projectNumber'\n)\"\n\ngcloud projects list --filter=\"PROJECT_NUMBER=$project_number\" --format=\"value(PROJECT_ID, NAME)\"\n"
  },
  {
    "path": "gcp/gcs_curl_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 01:11:31 +0000 (Fri, 16 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRetrieves a GCS file's contents from a given bucket and path using GCP authentication to the Storage JSON API\n\nUseful for starting shell pipelines or being called from other scripts such as terraform_gcs_backend_version.sh\n\nRequires GCloud SDK and jq being installed as well as GCloud SDK being already authenticated with an account with permission to the bucket\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<bucket> <file_path>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nbucket=\"$1\"\nfilepath=\"$2\"\n# there should be no leading slash - that would lead to a Not Found error\nfilepath=\"${2##/}\"\n# must urlencode /paths/to/file\nfilepath=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$filepath\")\"\n# alternatives\n# from DevOps-Python-tools repo if in $PATH\n#filepath=\"$(urlencode.py <<< \"$filepath\")\"\n# from DevOps-Perl-tools repo if in $PATH\n#filepath=\"$(urlencode.pl <<< \"$filepath\")\"\n\naccess_token=\"$(gcloud auth print-access-token || die \"Failed to get GCP access token - you must be authenticated first\")\"\n\nexport CURL_OPTS='-sS'\nexport TOKEN=\"$access_token\"\n\n\"$srcdir/../bin/curl_auth.sh\" $CURL_OPTS \"https://storage.googleapis.com/storage/v1/b/$bucket/o/$filepath?alt=media\"\n"
  },
  {
    "path": "gcp/gke_firewall_rule_cert_manager.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-15 12:21:25 +0000 (Tue, 15 Dec 2020)\n#\n#  https://github.com/HariSekhon/work\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP firewall rule to permit Cert Manager access on port 10250 for the admission webhook\n\nDetermines a given GKE cluster's master cidr block, network and target tags\n\nSolves this error:\n\n    Internal error occurred: failed calling admission webhook ... the server is currently unable to handle the request\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project> <cluster>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\n# project must be given explicitly to fix all subsequent gcloud commands to the right cluster to avoid concurrency race conditions of any other scripts or commands in adjacent windows from switching configs and causing these commands to go to the wrong project\nproject=\"$1\"\ncluster_name=\"$2\"\n\nPORT=10250\n\nfirewall_rule_name=\"gke-$cluster_name-masters-to-cert-manager\"\n\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\ntimestamp \"Getting details for cluster '$cluster_name'\"\n#gcloud container clusters describe \"$cluster\"\nmaster_cidr_block=\"$(gcloud container clusters describe \"$cluster_name\" --format='get(privateClusterConfig.masterIpv4CidrBlock)')\"\nnetwork=\"$(gcloud container clusters describe \"$cluster_name\" --format='value(networkConfig.network.basename())')\"\necho\necho \"Determined cluster '$cluster_name' master cidr block to be '$master_cidr_block'\"\necho \"Determined cluster '$cluster_name' network to be '$network'\"\necho\n\ntimestamp \"Listing firewall rules for cluster '^gke-$cluster_name':\"\necho\ngcloud compute firewall-rules list \\\n                                --filter \"name~^gke-$cluster_name\" \\\n                                --format 'table(\n                                    name,\n                                    network,\n                                    direction,\n                                    sourceRanges.list():label=SRC_RANGES,\n                                    allowed[].map().firewall_rule().list():label=ALLOW,\n                                    targetTags.list():label=TARGET_TAGS\n                                )'\necho\n\ntimestamp \"Getting target tags\"\ntarget_tags=\"$(gcloud compute firewall-rules list --filter \"name~^gke-$cluster_name\" --format 'get(targetTags.list())' | sort -u)\"\n\ntimestamp \"Determined target tags to be:\"\necho\necho \"$target_tags\"\necho\n\nif gcloud compute firewall-rules list --filter \"name=$firewall_rule_name\" --format 'get(name)' | grep -q .; then\n    timestamp \"GCP firewall rule '$firewall_rule_name' for cert manager already exists. If this is not working for you, check the target tags, port etc haven't changed\"\nelse\n    timestamp \"Adding a GCP firewall rule called '$firewall_rule_name' to permit GKE cluster '$cluster_name' master nodes to access cert manager pods on port $PORT:\"\n    gcloud compute firewall-rules create \"$firewall_rule_name\" \\\n                                           --network \"$network\" \\\n                                           --action ALLOW \\\n                                           --direction INGRESS \\\n                                           --source-ranges \"$master_cidr_block\" \\\n                                           --rules TCP:\"$PORT\" \\\n                                           --target-tags \"$target_tags\"  # is one in my testing, might need editing if more than one\n    echo\n    echo \"Done.\"\nfi\n"
  },
  {
    "path": "gcp/gke_firewall_rule_kubeseal.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-28 12:54:35 +0100 (Thu, 28 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GCP firewall rule to permit kubeseal to communicate with sealed-secrets-controller service\n\nDetermines a given GKE cluster's master cidr block, network and target tags\n\nhttps://github.com/bitnami-labs/sealed-secrets/blob/main/docs/GKE.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project> <cluster_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\n# project must be given explicitly to fix all subsequent gcloud commands to the right cluster to avoid concurrency race conditions of any other scripts or commands in adjacent windows from switching configs and causing these commands to go to the wrong project\nproject=\"$1\"\ncluster_name=\"$2\"\n\nPORT=8080\n\nfirewall_rule_name=\"gke-$cluster_name-masters-to-kubeseal\"\n\nexport CLOUDSDK_CORE_PROJECT=\"$project\"\n\ntimestamp \"Getting details for cluster '$cluster_name'\"\nmaster_ipv4_cidr=\"$(gcloud container clusters describe \"$cluster_name\" --format='get(privateClusterConfig.masterIpv4CidrBlock)')\"\nnetwork=\"$(gcloud container clusters describe \"$cluster_name\" --format='value(networkConfig.network.basename())')\"\necho\necho \"Determined cluster '$cluster_name' master cidr block to be '$master_ipv4_cidr'\"\necho \"Determined cluster '$cluster_name' network to be '$network'\"\necho\n\ntimestamp \"Listing firewall rules for cluster '^gke-$cluster_name':\"\necho\ngcloud compute firewall-rules list \\\n                                --filter \"name~^gke-$cluster_name\" \\\n                                --format 'table(\n                                    name,\n                                    network,\n                                    direction,\n                                    sourceRanges.list():label=SRC_RANGES,\n                                    allowed[].map().firewall_rule().list():label=ALLOW,\n                                    targetTags.list():label=TARGET_TAGS\n                                )'\necho\n\ntimestamp \"Getting target tags\"\ntarget_tags=\"$(gcloud compute firewall-rules list --filter \"name~^gke-$cluster_name\" --format 'get(targetTags.list())' | sort -u)\"\n\ntimestamp \"Determined target tags to be:\"\necho\necho \"$target_tags\"\necho\n\nif gcloud compute firewall-rules list --filter \"name=$firewall_rule_name\" --format 'get(name)' | grep -q .; then\n    echo \"GCP firewall rule '$firewall_rule_name' for kubeseal already exists. If this is not working for you, check the target tags, port etc haven't changed\"\nelse\n    timestamp \"Adding a GCP firewall from master called '$firewall_rule_name' to permit IPv4 cidr '$master_ipv4_cidr' network '$network' to target tags '$target_tags'\"\n    gcloud compute firewall-rules create \"$firewall_rule_name\" \\\n                                         --network \"$network\" \\\n                                         --allow \"tcp:$PORT\" \\\n                                         --source-ranges \"$master_ipv4_cidr\" \\\n                                         --target-tags \"$target_tags\" \\\n                                         --priority 1000\n    echo\n    echo \"Done\"\nfi\n"
  },
  {
    "path": "gcp/gke_kube_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-25 15:54:23 +0100 (Tue, 25 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates kubectl credentials and contexts for all GKE clusters in the current or given GCP project\n\nThis is fast way to get set up in new environments, or even just add any new GKE clusters to your existing \\$HOME/.kube/config\n\nIf the argument given is 'all', will run for all GCP projects using gcp_foreach_project.sh\n\nWARNING: GCloud SDK switches your kubectl context to the last cluster you get credentials for. This can lead to race conditions between other kubectl scripts if they have not forked and isolated their \\$KUBECONFIG. Do not run this while other naive kubectl commands and scripts are running otherwise those non-isolated commands may fire against the wrong kubernetes cluster. See kubectl.sh for more info\n\nSee also:\n\n    kubectl.sh             - isolates kube config to fix kubectl commands to the given cluster to prevent race conditions applying kubectl changes to the wrong cluster\n    gke_kubectl.sh         - same as above but also gets the credential\n    gcp_foreach_project.sh - iterates all locally configured projects, used by the 'all' argument\n    aws_kube_creds.sh      - same as this script but for AWS EKS\n    rancher_kube_creds.sh  - same as this script but for Rancher kubernetes clusters\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\nproject_id=\"${1:-}\"\n\nif [ -n \"${project_id:-}\" ]; then\n    if [ \"$project_id\" = all ]; then\n        \"$srcdir/gcp_foreach_project.sh\" \"${BASH_SOURCE[0]}\"\n        exit 0\n    else\n        export CLOUDSDK_CORE_PROJECT=\"$project_id\"\n    fi\nfi\n\ngcloud container clusters list --format='value(name,zone)' |\nwhile read -r cluster zone; do\n    echo \"Getting GKE credentials for cluster '$cluster' in zone '$zone':\"\n    gcloud container clusters get-credentials \"$cluster\" --zone \"$zone\"\n    echo\ndone\n"
  },
  {
    "path": "gcp/gke_kubectl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-03 14:24:44 +0000 (Tue, 03 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a kubectl command safely fixed to a GKE cluster by generating an isolated fixed config for the lifetime of this script\n\nAvoids concurrency race conditions with other concurrently executing commands or scripts by avoiding using or changing the global kubectl context\n\nEg. running:\n\n    kubectl config use-context\n            or\n    gcloud container clusters get-credentials\n\neither by your hand or in other concurrently executing scripts changes your global kubectl context to run on the given cluster, which could divert your command or concurrently long running scripts in other windows to run kubectl commands on the wrong cluster, leading to cross environment misconfigurations and real world outages (I've seen this personally)\n\nIf GKE_CONTEXT is set in the environment and matches a pre-existing context, skips pulling GKE creds for speed and noise reduction.\n\nIf GKE_CONTEXT is not set then requires the following to be set in the environment in order to obtain the credentials to the GKE cluster (will try to auto-infer from gcloud config if not set):\n\nCLOUDSDK_CORE_PROJECT       - project containing your GKE cluster\nCLOUDSDK_COMPUTE_REGION     - region containing your GKE cluster\nCLOUDSDK_CONTAINER_CLUSTER  - name of your GKE cluster\n\nIf the CLOUDSDK variables are not set and cannot be inferred from gcloud config, then errors out. If they are set though, they may be pointing to the wrong project or region so it is recommended to set them\n\nFor frequent more convenient usage you will want to shorten the CLI by copying this script to a local copy in each cluster's yaml config directory and hardcoding the GKE_CONTEXT (use gke_kube_creds.sh to pre-populate the context and credentials) or CLOUDSDK_CORE_PROJECT, CLOUDSDK_COMPUTE_REGION and CLOUDSDK_CONTAINER_CLUSTER variables if pulling GKE creds.\n\nCould also use main kube config with kubectl switches --cluster / --context (after configuring, see gke_kube_creds.sh), but this is more convenient, especially when hardcoded for the local copy in each cluster's k8s yaml dir\n\n\nSee Also:\n\n    gke_kube_creds.sh - auto-populates the credentials for all GKE clusters for your kubectl is ready to rock on GCP\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_options>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# ============================================================\n# HARDCODE THIS SECTION FOR SHORTER CLI convenience\n# REMOVE if hardcoding\n\n#GKE_CONTEXT=gke_<myproject>_<myregion>_<clustername>\n\nif [ -z \"${GKE_CONTEXT:-}\" ]; then\n\n    # fixed to this environment - thou shalt deploy to no other cluster from this script\n\n    # HARDCODE THESE for frequent shorter CLI usage\n    #CLOUDSDK_CORE_PROJECT=myproject\n    #CLOUDSDK_COMPUTE_REGION=europe-west1\n    #CLOUDSDK_CONTAINER_CLUSTER=\"$2\"  # eg. <myproject>-europe-west1\n\n    CLOUDSDK_CORE_PROJECT=\"${CLOUDSDK_CORE_PROJECT:-$(gcloud config list --format=\"get(core.project)\")}\"\n    CLOUDSDK_COMPUTE_REGION=\"${CLOUDSDK_COMPUTE_REGION:-$(gcloud config list --format=\"get(compute.region)\")}\"\n    CLOUDSDK_CONTAINER_CLUSTER=\"${CLOUDSDK_CONTAINER_CLUSTER:-$(gcloud config list --format=\"get(container.cluster)\")}\"\n    check_env_defined CLOUDSDK_CORE_PROJECT\n    check_env_defined CLOUDSDK_COMPUTE_REGION\n    check_env_defined CLOUDSDK_CONTAINER_CLUSTER\n\n    # if set and available in original kube config, will just copy config and switch to this context (faster and less noisy than re-pulling creds from GKE)\n    GKE_CONTEXT=\"gke_${CLOUDSDK_CORE_PROJECT}_${CLOUDSDK_COMPUTE_REGION}_${CLOUDSDK_CONTAINER_CLUSTER}\"\nfi\n# ============================================================\n\nkube_config_isolate\n\nif ! gcloud auth application-default print-access-token >/dev/null; then\n    gcloud auth application-default login\nfi\n\n# if original kube config contains the context, copy and reuse it (faster and less noisy than re-pulling the creds from GKE), especially when called in script iterations\nif [ -n \"${GKE_CONTEXT:-}\" ] &&\n   kubectl config get-contexts -o name | grep -Fxq \"$GKE_CONTEXT\"; then\n    # switch context if not already the current context (avoids repeating \"switching context\" output noise when this script it called iteratively in loop by other scripts)\n    if [ \"$(kubectl config current-context)\" != \"$GKE_CONTEXT\" ]; then\n        kubectl config use-context \"$GKE_CONTEXT\" >&2\n    fi\nelse\n    gcloud container clusters get-credentials \"$CLOUDSDK_CONTAINER_CLUSTER\" --region \"$CLOUDSDK_COMPUTE_REGION\" --project \"$CLOUDSDK_CORE_PROJECT\" >&2\n    echo >&2\nfi\n\nkubectl \"$@\"\n"
  },
  {
    "path": "gcp/gke_nodepool_drain.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 17:44:30 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDrains all Kubernetes nodes from a given GKE cluster's nodepool\n\nUseful to decommission an entire nodepool to delete or recreate it (eg. with different taints)\n\nYou must have a second node pool with sufficient capacity / autoscaling max nodes to be able to accommodate the evicted pods\n\n- finds instance groups in a node pool\n- finds nodes in each instance group\n- disables autoscaling for the node pool (to prevent booting new nodes to migrate pods to)\n- cordons all nodes (to prevent pod migrations to other nodes in the same pool)\n- drains pods from each node in sequence\n\nRequires:\n\n    - GCloud SDK to be installed and configured\n      - requires core/project and compute/region to be set in your gcloud config\n        or else environment variables CLOUDSDK_CORE_PROJECT and CLOUDSDK_COMPUTE_REGION\n    - uses adjacent scripts:\n      - gke_nodepool_nodes2.sh - lists all nodes in the given nodepool\n      - gke_kubectl.sh - for safe kubectl with isolated context\n\nIf gcloud config container/cluster or CLOUDSDK_CONTAINER_CLUSTER are set then you don't have to specify the cluster name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<cluster_name>] <node_pool_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [ $# -eq 2 ]; then\n    export GCLOUD_CONTAINER_CLUSTER=\"$1\"\n    node_pool=\"$2\"\nelif [ $# -eq 1 ]; then\n    node_pool=\"$1\"\nelse\n    usage\nfi\n\nnodes=\"$(VERBOSE=1 \"$srcdir/gke_nodepool_nodes2.sh\" \"$node_pool\")\"\n\necho >&2\ntimestamp \"disabling autoscaling for node pool '$node_pool'\"\ngcloud container node-pools update --no-enable-autoscaling \"$node_pool\"\n\necho >&2\ntimestamp \"cordoning nodes:\"\nfor node in $nodes; do\n    #timestamp \"cordoning node '$node'\"\n    \"$srcdir/gke_kubectl.sh\" cordon \"$node\"\ndone\n\nforce=\"\"\nif [ \"${FORCE:-1}\" ]; then\n    force=\"--force\"\nfi\n\nfor node in $nodes; do\n    echo >&2\n    timestamp \"draining node '$node'\"\n    \"$srcdir/gke_kubectl.sh\" drain \"$node\" $force --ignore-daemonsets  # &  # could parallelize this - respects pod disruption budgets\ndone\n"
  },
  {
    "path": "gcp/gke_nodepool_nodes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 17:44:30 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all Kubernetes nodes for a given nodepool on the current GKE cluster by filtering for the corresponding label\n\nRequires kubectl to be installed and configured\n\nIf you run this on a non-GKE cluster, will return no nodes as there will be no nodes with the matching labels\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<node_pool_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nnode_pool=\"$1\"\n\nkubectl get nodes -l cloud.google.com/gke-nodepool=\"$node_pool\" -o name\n"
  },
  {
    "path": "gcp/gke_nodepool_nodes2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 17:44:30 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all Kubernetes nodes in a given GKE cluster's nodepool\n\n- finds instance groups in a node pool\n- finds nodes in each instance group\n\nRequires:\n\n    - GCloud SDK to be installed and configured\n      - requires core/project and compute/region to be set in your gcloud config\n        or else environment variables CLOUDSDK_CORE_PROJECT and CLOUDSDK_COMPUTE_REGION\n\nIf gcloud config container/cluster or CLOUDSDK_CONTAINER_CLUSTER are set then you don't have to specify the cluster name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<cluster_name>] <node_pool_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [ $# -eq 2 ]; then\n    export GCLOUD_CONTAINER_CLUSTER=\"$1\"\n    node_pool=\"$2\"\nelif [ $# -eq 1 ]; then\n    node_pool=\"$1\"\nelse\n    usage\nfi\n\nif [ -n \"${VERBOSE:-}\" ]; then\n    timestamp \"finding instance groups in node pool '$node_pool'\"\nfi\ninstance_groups=\"$(\n    gcloud container node-pools describe \"$node_pool\" --format=json |\n    jq -r '.instanceGroupUrls[] | sub(\"^.*/\"; \"\")'\n)\"\n\nif [ -n \"${VERBOSE:-}\" ]; then\n    echo >&2\nfi\nfor instance_group in $instance_groups; do\n    if [ -n \"${VERBOSE:-}\" ]; then\n        #timestamp \"finding zone of instance group '$instance_group'\"\n        timestamp \"finding nodes in instance group '$instance_group'\"\n    fi\n    zone=\"$(gcloud compute instance-groups list --filter=\"name=$instance_group\" --format='get(zone)' | sed 's|^.*/||')\"\n    gcloud compute instance-groups list-instances \"$instance_group\" --zone \"$zone\" --format='value(NAME)'  # doesn't find it without zone, and NAME must be capitalized too\ndone\n"
  },
  {
    "path": "gcp/gke_nodepool_taint.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 20:35:02 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTaints/untaints all nodes in the given GKE nodepool on the current cluster with your taint spec\n\nSee:\n\n    kubectl taint nodes --help\n\nfor the taint spec\n\nThis is just slightly easier than typing:\n\n    kubectl taint nodes -l cloud.google.com/gke-nodepool=<node_pool_name> ...\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<node_pool_name> <taint_spec>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nnode_pool=\"$1\"\nshift || :\n\n#kube_config_isolate\n\n#nodes=\"$(VERBOSE=1 \"$srcdir/gke_nodepool_nodes.sh\" \"$node_pool\")\"\n\nkubectl taint nodes -l cloud.google.com/gke-nodepool=\"$node_pool\" \"$@\"\n"
  },
  {
    "path": "gcp/gke_persistent_volume_disk_mappings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 13:05:48 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOn GKE prints the list of Kubernetes volumes to GCP persistent disk name mappings using kubectl\n\nAny args are passed straight to kubectl as is\n\nOutput:\n\n    <namespace>    <pvc_name>    <kubernetes_persistent_volume>    <gcp_persistent_disk>    <zone>\n\nThis is useful to investigate GCP disks when testing disk resizing\n\nSee Also:\n\n    https://github.com/HariSekhon/Kubernetes-templates\n\n        storageclass-gcp-*-resizeable.yml\n\nKubectl is expect to be installed, configured and pointing to the correct cluster context\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_args>]\"\n\nhelp_usage \"$@\"\n\n# the spec line needs to be one line because newlines in the jsonpath come out literally\nkubectl get pv \"$@\" -o jsonpath='\n    {range .items[*]}\n        {.spec.claimRef.namespace}{\"\\t\"}{.spec.claimRef.name}{\"\\t\"}{.metadata.name}{\"\\t\"}{.spec.gcePersistentDisk.pdName}{\"\\t\"}{.metadata.labels.failure-domain\\.beta\\.kubernetes\\.io/zone}{\"\\n\"}\n    ' |\ncolumn -t\n"
  },
  {
    "path": "git/git_askpass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  run: GIT_USERNAME=hari-s GIT_PASSWORD=testpass git_askpass.sh get\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-24 15:35:06 +0100 (Wed, 24 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://git-scm.com/docs/gitcredentials\n\n# https://git-scm.com/docs/git-credential\n\nset -euo pipefail\n# doesn't seem to pass through DEBUG environment variable when called via 'git credential fill' - will need to set -x explicitly\n[ -n \"${DEBUG:-}\" ] && set -x\n#set -x\n\nusage(){\n    cat <<EOF\nGIT_ASKPASS credential script to allow loading credentials from environment variables to Git dynamically\n\nThe \\$GIT_ASKPASS environment variable should be set to the location of this script to have Git call it automatically\n\nThis program is designed to be called by the 'git' command in the form of:\n\n    git credential fill\n\nFull example command:\n\n    echo url=https://github.com | GIT_ASKPASS=$0 git credential fill\n\nwhich calls this script like so:\n\n    ${0##*/} get\n\n\nEnvironment variables used if available, in precedence order from left to right:\n\n    username = \\$GIT_USERNAME, \\$GIT_USER\n    password = \\$GIT_TOKEN, \\$GIT_PASSWORD\n\n\nusage: ${0##*/} get\nEOF\n    exit 3\n}\n\nif [ $# -ne 1 ] || [[ $* =~ - ]]; then\n    usage\nfi\n\nusername_variables=\"\nGIT_USERNAME\nGIT_USER\n\"\n\npassword_variables=\"\nGIT_TOKEN\nGIT_PASSWORD\n\"\n\noutput_variable(){\n    local key=\"$1\"\n    local variables=\"$2\"\n    for var in $variables; do\n        if [ -n \"${!var:-}\" ]; then\n            echo \"$key=${!var}\"\n            break\n        fi\n    done\n}\n\nif [ \"$1\" = get ]; then\n    output_variable username \"$username_variables\"\n    output_variable password \"$password_variables\"\n\n# have observed Git version 2.27.0 in ArgoCD calling the GIT_ASKPASS program twice with these 2 first arguments:\n#\n#   'Username for '\\''https://github.com'\\''\n#\n# and\n#\n#   'Password for '\\''https://github.com'\\''\n#\n# and then taking the entire first line returned as the value\nelif [[ \"$*\" =~ Username ]]; then\n    output_variable username \"$username_variables\" | sed 's/^username=//'\nelif [[ \"$*\" =~ Password ]]; then\n    output_variable password \"$password_variables\" | sed 's/^password=//'\nfi\n"
  },
  {
    "path": "git/git_branch_delete_squash_merged.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-27 12:06:49 +0200 (Tue, 27 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCarefully detects if a Git squash merged branch you want to delete has no changes vs the default trunk branch before deleting it\n\nDoing a branch force delete without these checks risks losing all unpushed code changes on a branch\n\nYou need to run this on a clean trunk branch as it will check for any changed files (only untracked files are ignored)\n\nIf you don't think you can lose code by deleting the wrong branch just consider what happens when you repeatedly\nwork on area, branching each time but haven't been able to automatically prune old branches. You'll use similar names\nand then may be confused about which is the old one to delete, run 'git branch -D' on the wrong one and congrats\nyou've just permanently lost all your unpushed work on that branch\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<branch>\"\n# TODO: consider allowing a second arg to explicitly specify the trunk branch\n#       - shouldn't be needed in most sane cases though\n#       - client who has dual trunk repo with prod deployments running off both develop and master\n#         is a historical legacy mistake\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbranch=\"$1\"\n\ndefault_trunk_branch=\"$(default_branch)\"\n\ntimestamp \"Checking out branch '$branch'\"\n# this actually wipes out any previous the merge state test\n# - if script were to crash out, simply re-running would be enough\ngit checkout \"$branch\"\necho\n\ntimestamp \"Pulling latest changes to branch '$branch'\"\ngit pull ||\ntimestamp \"Pull failed but that may be ok because the upstream branch may have been auto-deleted upon merge, continuing...\"\necho\n\ntimestamp \"Checking out default trunk branch '$default_trunk_branch'\"\n# this actually wipes out any previous the merge state test\n# - if script were to crash out, simply re-running would be enough\ngit checkout \"$default_trunk_branch\"\necho\n\ntimestamp \"Pulling latest changes to default trun branch '$default_trunk_branch'\"\ngit pull\necho\n\ntimestamp \"Checking '$branch' branch contents vs '$default_trunk_branch' using a pseudo merge for any diffs\"\n# this git command only outputs to stderr so we must capture to stdout to test, and then re tee back to stderr\n# to give good user visibility of the state message\noutput=\"$(git merge --no-ff --no-commit \"$branch\" 2>&1 | tee /dev/stderr)\"\necho\n\n# check for any changes in the checkout but ignore untracked files\nif [ -z \"$(git status --porcelain | sed '/^??/d')\" ]; then\n    timestamp \"No content changes detected\"\nelse\n    timestamp \"Changes detected from branch '$branch' or dirty checkout - branch may not be fully merged\"\n    timestamp \"Aborting for safety\"\n    exit 1\nfi\n\n# \"Automatic merge went well\" is not enough - hence we need the content change check above\n# and use this as a mere state confirmation here if we don't get an \"Already up to date\" message\nif [[ \"$output\" =~ Already\\ up\\ to\\ date|Automatic\\ merge\\ went\\ well ]]; then\n    timestamp \"Branch is safe to delete\"\n    timestamp \"Deleting branch '$branch'\"\n    git branch -D \"$branch\"\nelse\n    timestamp \"WARNING: branch '$branch' was not detected as safely merged into trunk '$default_trunk_branch'\"\n    echo\nfi\ngit merge --abort\n"
  },
  {
    "path": "git/git_clean_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nrun(){\n    local repofile=\"$1\"\n    echo \"processing repos from file: $repofile\"\n    sed 's/#.*//; s/:/ /; /^[[:space:]]*$/d' < \"$repofile\" |\n    while read -r _ dir; do\n        #if ! echo \"$repo\" | grep -q \"/\"; then\n        #    repo=\"HariSekhon/$repo\"\n        #fi\n        if [ -d \"$dir\" ]; then\n            pushd \"$dir\"\n            # make update does git pull but if that mechanism is broken then this first git pull will allow the repo to self-fix itself\n            if grep -q '^clean:' Makefile; then\n                make clean\n            fi\n            popd\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        run \"$x\"\n    done\nelse\n    run \"$srcdir/../setup/repos.txt\"\nfi\n"
  },
  {
    "path": "git/git_diff_commit.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-08-05 15:49:27 +0100\n#  (migrated out of .bash.d/git.sh for use in IntelliJ)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly commits added or updated files to Git, showing a diff and easily enter prompt for each file\n\nCommits with a generic \\\"added \\$filename\\\" or \\\"updated \\$filename\\\" commit message\n\nLazy but awesome for lots of daily quick intermediate commit saves\n\nOriginally used in .bash.d/git.sh as a function git() and vimrc hotkey\n\nPorted to external script be callable from IntelliJ as an External Tool because it's less keystrokes\nand no mouse movement compared to IntelliJ's own hot key git commit tooling\n\nIf no args are given, then git diffs and commits against the current working directory\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files_or_directories>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nresolve_symlinks(){\n    local readlink=readlink\n    if is_mac; then\n        readlink=greadlink\n        if ! type -P greadlink >&/dev/null; then\n            \"$srcdir/../packages/brew_install_package.sh\" coreutils  # for greadlink\n        fi\n    fi\n    for x in \"$@\"; do\n        \"$readlink\" -m \"$x\"\n    done\n}\n\n# Using this trick to be able to call this script from an IntelliJ hotkey which doesn't allocate /dev/tty\n# which prevents using my simpler trick of reading from /dev/tty to avoid the while loops from eating /dev/stdin\n#\n# duplicate original /dev/stdin file description 0 into file descriptor 3\n# before using while read line loops (for safe filename processing)\n# because those looks consume the /dev/stdin\nexec 3<&0\n\ngit_diff_commit(){\n    local basedir\n    for filename in \"${@:-.}\"; do\n        if [ \"$filename\" != . ]; then\n            # TODO: detect link changes and commit them too\n            filename=\"$(resolve_symlinks \"$filename\")\"\n        fi\n        basedir=\"$(dirname \"$filename\")\"\n        pushd \"$basedir\" > /dev/null\n        git_status_porcelain=\"$(git status --porcelain -s \"${filename##*/}\")\"\n        added_files=\"$(\n            grep -e '^?' -e '^A' <<< \"$git_status_porcelain\" |\n            sed 's/^...//; s/^\"//; s/\"$//' || :\n            # stripping leading and trailing quotes because git adds them when the filename contains spaces,\n            # but we do line handling on the filename so don't need this and it breaks later processing\n            # as the quotes become taken literally\n        )\"\n        while read -r added_filename; do\n            is_blank \"$added_filename\" && continue\n            basename=\"${added_filename##*/}\"\n            git add \"$basename\"\n            diff=\"$(git diff --color=always -- \"$added_filename\"\n                    git diff --cached --color=always -- \"$added_filename\")\"\n            echo \"$diff\" | less -FR\n            echo\n            # read doesn't print when using a redirect so have to print ourself\n            printf \"Hit enter to commit added file '%s' or Control-C to cancel: \" \"$added_filename\"\n            #read -r -p \"Hit enter to commit added file '$added_filename' or Control-C to cancel\" _ <&3  # read from dup’d stdin, not eaten by the loop\n            # discard the save variable, call it _ to signify this\n            read -r _ <&3  # read from dup’d stdin, not eaten by the loop\n            echo\n            echo \"committing added file $added_filename\"\n            git commit -m \"added $basename\" -- \"$added_filename\"\n        done <<< \"$added_files\"\n        changed_files=\"$(\n            grep -e '^M' -e '^.M' <<< \"$git_status_porcelain\" |\n            sed 's/^...//; s/^\"//; s/\"$//' || :\n            # stripping leading and trailing quotes because git adds them when the filename contains spaces,\n            # but we do line handling on the filename so don't need this and it breaks later processing\n            # as the quotes become taken literally\n        )\"\n        while read -r changed_filename; do\n            is_blank \"$changed_filename\" && continue\n            basename=\"${changed_filename##*/}\"\n            diff=\"$(git diff --color=always -- \"$changed_filename\"\n                    git diff --cached --color=always -- \"$changed_filename\")\"\n            if [ -z \"$diff\" ]; then\n                continue\n            fi\n            echo \"$diff\" | less -FR\n            echo\n            # read doesn't print when using a redirect so have to print ourself\n            printf \"Hit enter to commit updated file '%s' or Control-C to cancel: \" \"$changed_filename\"\n            # read doesn't print when using a redirect\n            #read -r -p \"Hit enter to commit updated file '$changed_filename' or Control-C to cancel\" _ <&3  # read from dup’d stdin, not eaten by the loop\n            # discard the save variable, call it _ to signify this\n            read -r _ <&3  # read from dup’d stdin, not eaten by the loop\n            echo\n            git add -- \"$changed_filename\"\n            echo \"committing updated file $changed_filename\"\n            git commit -m \"updated $basename\" -- \"$changed_filename\"\n        done <<< \"$changed_files\"\n        popd >&/dev/null || :\n    done\n}\n\nfor target in \"${@:-.}\"; do\n    git_diff_commit \"$target\"\ndone\n\ntimestamp \"Git Diff Commit completed\"\n"
  },
  {
    "path": "git/git_files_in_history.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-15 08:58:02 +0000 (Sat, 15 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all unique file paths in Git history\n\nUseful for prepraring to 'git filter-branch' eg. for repo splicing\n\nMust be run from within a git repository, assumes the 'git' command is installed and in the \\$PATH\n\nIf you only want current files, use instead 'git ls-files'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# technically --all isn't that relevant to git filter-branch as it'll list things on other branches, but in terms of listing all files in history, it's more correct to include it as it won't break filter-branch derived usage\n# -e '^[[:alpha:]]+:' filters out Author: , Date: , Merge: etc.\ngit log --all --name-only --no-color |\ngrep -Ev -e '^commit' -e '^[[:alpha:]]+:' -e '^Date:' -e '^[[:space:]]' -e '^[[:space:]]*$' |\nsort -u\n"
  },
  {
    "path": "git/git_files_last_modified.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-18 08:40:45 +0100 (Sat, 18 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nFor each file or directory argument given, finds the last modified date in the git log and orders the output descending\n\nIf no arguments are given, assumes to use \\$PWD\n\nRequires git to be in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files_or_dirs>]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nunset GIT_PAGER\n\nfor dir in \"${@:-.}\"; do\n    # faster & more efficient but git log format doesn't have filename in format string\n    #git ls-files -z \"$dir\" |\n    #xargs -0 -L 1 git log --format=\"%ai %H\"\n    if [ -n \"${VERBOSE:-}\" ]; then\n        echo -n \"iterating $dir \" >&2\n    fi\n    # works for files too\n    git ls-files \"$dir\" |\n    while read -r filename; do\n        if [ -n \"${VERBOSE:-}\" ]; then\n            echo -n '.' >&2\n        fi\n        # aI - 2020-07-17T21:52:55+01:00\n        # ai - 2020-07-17 21:52:55 +0100\n        # escape % in filename to come out literally\n        name=\"${filename//%/%%}\"\n        git log --format=\"%ai   %H  $name\" \"$filename\"\n    done\n    if [ -n \"${VERBOSE:-}\" ]; then\n        echo >&2\n    fi\ndone |\nsort -r\n"
  },
  {
    "path": "git/git_files_no_uncommitted_changes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-04-06 22:12:20 +0800 (Sun, 06 Apr 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns zero if given file(s) don't have uncommitted changes to Git, either staged or unstaged\n\nUseful to be able to iterate over git files with in-place edits only if safe to do so\nwithout other uncommitted changes that would be at risk of being lost\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<files>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexitcode=0\n\nfor filename; do\n    if ! [ -f \"$filename\" ]; then\n        die \"File not found: $filename\"\n    fi\n    dirname=\"$(dirname \"$filename\")\"\n    basename=\"${filename##*/}\"\n    cd \"$dirname\"\n    if ! git status --porcelain \"$basename\" | grep .; then\n        echo \"No uncommited changes: $filename\"\n    else\n        exitcode=1\n        echo \"UNCOMMITTED changes discovered: $filename\"\n    fi\n    cd - >/dev/null\ndone\n\nexit \"$exitcode\"\n"
  },
  {
    "path": "git/git_filter_branch_fix_author.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-04 17:46:52 +0000 (Fri, 04 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRewrites the current Git branch using git filter-branch to change the Author/Committer name and/or email address\n\nFor each commit in the current branch history, if both:\n\n    - the Author or Committer Name matches the <old_name>\n    - the Author or Committer Email matches the <old_email>\n\nthen both the Author and Committer names and emails are set to <new_name> and <new_email>\n\n<git_options> - passed literally to git filter-branch after -- can use this to only rewrite a revision range, eg. <starting_hashref>..<ending_hashref>\n\nMust be called from the top level directory of the repository\n\nYou may still see the old name/email in the local repo's git log, test by cloning to a new repo and if happy then park the old checkout and checkout clean\n\n\nDANGER: this rewrites Git history.\n        Do not use this carelessly as it rewrites Git history.\n        Always have a backup.\n        Do not do this on pushed branches unless you are an Expert and intend to\n\nIf there is already a git filter-branch rewrite backup in .git/refs/original, git filter-branch will refuse to proceed - specify \\$FORCE_GIT_REWRITE=1 in the environment to force the rewrite\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<old_email> <new_email> [<new_name>] [<git_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nold_email=\"$1\"\nnew_email=\"$2\"\nnew_name=\"${3:-}\"\nshift || :\nshift || :\nif [ -n \"$new_name\" ]; then\n    shift || :\nfi\n\nopts=()\nif [ -n \"${FORCE_GIT_REWRITE:-}\" ]; then\n    opts+=(-f)\nfi\n\nfor x in \"$old_email\" \"$new_email\"; do\n    # email_regex is defined in lib/utils.sh\n    # shellcheck disable=SC2154\n    # <Hari Sekhon@TeamCity> doesn't match - technically this may not match real emails so be a bit more lax\n    #if ! [[ \"$x\" =~ $email_regex ]]; then\n    if ! [[ \"$x\" =~ @ ]]; then\n        die \"Invalid email '$x' given\"\n    fi\ndone\n\necho\necho \"DANGER!!!\"\necho\necho -n \"You are about to replace all Author and Committer references from '$old_email' to '$new_email'\"\nif [ -n \"$new_name\" ]; then\n    echo \" and change the name field to '$new_name'\"\nfi\necho\nread -r -p \"DANGER: are you absolutely sure? (y/N)  \" answer\necho\n\nis_yes \"$answer\" || die \"Aborting\"\n\ntimestamp \"Starting git filter-branch replacement\"\necho\n\ngit filter-branch \"${opts[@]}\" --tag-name-filter cat --env-filter \\\n    \"\n    if [ \\\"\\$GIT_AUTHOR_EMAIL\\\"    = '$old_email' ] ||\n       [ \\\"\\$GIT_COMMITTER_EMAIL\\\" = '$old_email' ]; then\n        if [ -n '$new_name' ]; then\n            export GIT_AUTHOR_NAME='$new_name'\n            export GIT_COMMITTER_NAME='$new_name'\n        fi\n        export GIT_AUTHOR_EMAIL='$new_email'\n        export GIT_COMMITTER_EMAIL='$new_email'\n    fi\n    \" \\\n    -- --all \"$@\"\n"
  },
  {
    "path": "git/git_filter_repo_replace_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-01-30 19:05:59 +0000 (Tue, 30 Jan 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRewrites the Git history using git filter-repo to replace a given token with another to scrub history\n\n<git_options> - passed literally to 'git filter-repo', can use this to only rewrite a revision range, eg. <starting_hashref>..<ending_hashref>\n\n\nDANGER: this rewrites Git history.\n        Do not use this carelessly as it rewrites Git history.\n        Always have a backup.\n        Do not do this on pushed branches unless you are an Expert and intend to\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<old_token> [<new_token>] [<git_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nold_token=\"$1\"\nnew_token=\"$2\"\nshift || :\nshift || :\n\nopts=()\nif [ -n \"${FORCE_GIT_REWRITE:-}\" ]; then\n    opts+=(-f)\nfi\n\nif [ \"${#old_token}\" -lt 8 ]; then\n    echo \"\nDANGER: refusing to replace a token less than 8 characters because this is likely to match other tokens and cause serious unintended consequences which could destroy your code base\n\nIf you really intend to do this and known what you're doing, edit this code to bypass\n\" >&2\n    exit 1\nfi\n\necho\necho \"DANGER!!!\"\necho\necho -n \"You are about to replace all substrings of '$old_token' to '$new_token' - this could seriously damage you code base if you careless use a token that matches elsewhere\"\necho\necho \"Have you taken a backup?\"\necho\nread -r -p \"DANGER: are you absolutely sure? (y/N)  \" answer\necho\n\nis_yes \"$answer\" || die \"Aborting\"\n\ntimestamp \"Starting git filter-repo replacement\"\necho\n\ngit filter-repo \"${opts[@]}\" --replace-text <(echo \"$old_token==>$new_token\") \"$@\"\n"
  },
  {
    "path": "git/git_foreach_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-21 10:43:47 +0100 (Tue, 21 Nov 2017)\n#\n#  https://github.com/HariSekhon/Dockerfiles\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# access to useful functions and aliases\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/.bash.d/aliases.sh\"\n#. \"$srcdir/.bash.d/functions.sh\"\n. \"$srcdir/.bash.d/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nRun a command against each Git branch in the current repo checkout\n\nChecks out each branch and runs the command, before returning to the original branch\n\nIf the command fails on a branch, this script will exit and leave you checked out on that branch\n\nOne use case for this is merging your trunk branch into all your feature branches in an automated manner\n\nThis is powerful so use carefully!\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nforeachbranch \"$@\"\n"
  },
  {
    "path": "git/git_foreach_modified.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-02 19:54:09 +0100 (Tue, 02 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# access to useful functions and aliases\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/aliases.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/functions.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nRuns any arguments as a command against each file with a Git modified status\n\nThe filename will be appended to the end of each command in each iteration\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor filename in $(git status --porcelain | awk '/^.M/{print $NF}'); do\n    \"$@\" \"$filename\"\ndone\n"
  },
  {
    "path": "git/git_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo owner={owner} repo={repo} is found at dir={dir}\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# access to useful functions and aliases\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/aliases.sh\"\n#\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/functions.sh\"\n#\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/git.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against all GitHub repos while changing to their directories on disk\n\nAny repos which are not checked out locally adjacent to this repo are skipped\n\nAll arguments become the command template\n\nThe command template replaces the following for convenience in each iteration:\n\n{owner} - repo owner eg. HariSekhon\n{repo}  - repo name without the user/org prefix (eg. DevOps-Bash-tools)\n{dir}   - directory on disk (eg. /Users/hari/github/bash-tools)\n\neg. ${0##*/} 'echo {repo} is found at {dir}'\n\nor more usefully when chained with the other adjacent github_*.sh / gitlab_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n#git_url=\"${GIT_URL:-https://github.com}\"\n\n#git_base_dir=~/github\n\n#mkdir -pv \"$git_base_dir\"\n\n#cd \"$git_base_dir\"\n\nopts=(\"${OPTS:-}\")\nif [ -z \"${NO_TEST:-}\" ]; then\n    opts+=(\"test\")\nfi\n\nrepofile=\"$srcdir/../setup/repos.txt\"\n\nrepolist=\"${REPOS:-}\"\nif [ -n \"$repolist\" ]; then\n    :\nelif [ -f \"$repofile\" ]; then\n    log \"processing repos from file: $repofile\" >&2\n    repolist=\"$(sed 's/#.*//; /^[[:space:]]*$/d' < \"$repofile\")\"\nelse\n    log \"fetching repos from GitHub repo list\" >&2\n    repolist=\"$(curl -sSL https://raw.githubusercontent.com/HariSekhon/bash-tools/master/setup/repos.txt | sed 's/#.*//')\"\nfi\n\nexecute_repo(){\n    local owner_repo=\"$1\"\n    shift || :\n    local cmd=(\"$@\")\n    local owner\n    local repo=\"${owner_repo##*/}\"\n    repo_dir=\"$repo\"\n    repo_dir=\"${repo_dir##*:}\"\n    repo_dir=\"$srcdir/../../$repo_dir\"\n    repo=\"${repo%%:*}\"\n    if ! [ -d \"$repo_dir\" ]; then\n        #git clone \"$git_url/$repo\" \"$repo_dir\"\n        return\n    fi\n    if grep -q \"/\" <<< \"$owner_repo\"; then\n        owner=\"${owner_repo%/*}\"\n    else\n        #owner_repo=\"HariSekhon/$repo\"\n        owner=\"$(cd \"$repo_dir\"; git remotes | awk '{print $2}' | sed 's|/[^/]*$||; s|/[^/]*/_git||; s|.*[/:]||' | head -n1)\"\n        if [ -z \"$owner\" ]; then\n            die \"Failed to find owner for repo '$repo' at dir '$repo_dir'\"\n        fi\n        owner_repo=\"$owner/$repo\"\n    fi\n    pushd \"$repo_dir\" >/dev/null\n    repo_dir=\"$PWD\"\n    if [ -z \"${GIT_FOREACH_REPO_NO_HEADERS:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $owner_repo - $repo_dir\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"${cmd[@]//\\{owner\\}/$owner}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{dir\\}/$repo_dir}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    if [[ \"${cmd[*]}\" =~ github_.*.sh|gitlab_.*.sh|bitbucket_*.sh ]]; then\n        # throttle hitting the GitHub / GitLab / Bitbucket APIs too often as they may error out\n        sleep 0.05\n    fi\n    popd >/dev/null\n    if [ -z \"${GIT_FOREACH_REPO_NO_HEADERS:-}\" ]; then\n        echo >&2\n    fi\n}\n\nfor repo in $repolist; do\n    execute_repo \"$repo\" \"$@\"\ndone\n"
  },
  {
    "path": "git/git_foreach_repo_replace_readme_actions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 18:01:04 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates the README.md badges for GitHub Actions to match the local repo name\n\nUseful to bulk fix copied badges quickly and easily\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nsed=\"sed\"\nif is_mac; then\n    sed=\"gsed\"\nfi\n\n\"$srcdir/git_foreach_repo.sh\" \"$sed\"' -i \"s|{owner}/[[:alnum:]_-]*/actions|{owner}/{repo}/actions|g\" README.md'\n"
  },
  {
    "path": "git/git_foreach_repo_update_readme.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 18:01:04 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGit Diff Commits the README.md for each Git repo checkout using adjacent git_foreach_repo.sh and git_diff_commit.sh scripts\n\nUseful to quickly bulk update README.md in all your projects, such as when references need updating\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nsed=\"sed\"\nif is_mac; then\n    sed=\"gsed\"\nfi\n\n\"$srcdir/git_foreach_repo.sh\" \"$srcdir/git_diff_commit.sh README.md\"\n"
  },
  {
    "path": "git/git_graph_commit_history_gnuplot.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-03 04:55:02 +0300 (Thu, 03 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\ncode_month=\"git_commits_per_month.gnuplot\"\ncode_year=\"git_commits_per_year.gnuplot\"\ndata_month=\"data/git_commits_per_month.dat\"\ndata_year=\"data/git_commits_per_year.dat\"\nimage_year=\"images/git_commits_per_year.png\"\nimage_month=\"images/git_commits_per_month.png\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates GNUplot graphs of Git commits per year and per month for the entire history of the local Git repo checkout\n\nGenerates the following files:\n\n    $code_month - Code\n    $code_year  - Code\n\n    $data_month - Data\n    $data_year  - Data\n\n    $image_month - Image\n    $image_year  - Image\n\nA MermaidJS version of this script is adjacent at:\n\n    git_graph_commit_history_mermaidjs.sh\n\nRequires Git and GNUplot to be installed to generate the graphs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_log_paths_to_check_only>]\"\n\nhelp_usage \"$@\"\n\nif ! is_in_git_repo; then\n    die \"Error: Not inside a git repository!\"\nfi\n\nfor x in $code_month  \\\n         $code_year   \\\n         $data_month  \\\n         $data_year   \\\n         $image_month \\\n         $image_year; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\n# output git commits as simple YYYY-MM and then just sort and count them by month\ntimestamp \"Calculating commit counts per month from the Git log\"\ngit log --date=format:'%Y-%m' --pretty=format:'%ad' \"$@\" |\nsort |\nuniq -c |\nawk '{print $2, $1}' > \"$data_month\"\ntimestamp \"Wrote data: $data_month\"\necho\n\ntimestamp \"Calculating commit counts per year from the Git log\"\ngit log --date=format:'%Y' --pretty=format:'%ad' \"$@\" |\nsort |\nuniq -c |\nawk '{print $2, $1}' > \"$data_year\"\ntimestamp \"Wrote data: $data_year\"\necho\n\ngnuplot_common_settings=\"\n#\n# Generated by ${0##*/}\n#\n# from https://github.com/HariSekhon/DevOps-Bash-tools\n#\nset terminal pngcairo size 1280,720 enhanced font 'Arial,14'\nset ylabel 'Number of Commits'\nset grid\nset xtics rotate by -45\nset boxwidth 0.5 relative\nset style fill solid\nset datafile separator ' '\n\"\n#set xtics auto  # cannot find a way to make this show every year\n\ntimestamp \"Generating GNUplot code for Commits per Month\"\nsed '/^[[:space:]]*$/d' > \"$code_month\" <<EOF\n$gnuplot_common_settings\nset title \"Git Commits per Month\"\nset xlabel \"Month-Year\"\nset format x \"%b %Y\"\nset xdata time\nset timefmt \"%Y-%m\"\nset output \"$image_month\"\nplot \"$data_month\" using 1:2 with boxes title 'Commits'\nEOF\ntimestamp \"Generated GNUplot code: $code_month\"\necho\n\ntimestamp \"Generating GNUplot code for Commits per Year\"\nsed '/^[[:space:]]*$/d' > \"$code_year\" <<EOF\n$gnuplot_common_settings\nset title \"Git Commits per Year\"\nset xlabel \"Year\"\n# results in X axis labels every 2 years\n#set xdata time\n#set timefmt \"%Y\"\n#set format x \"%Y\"\n# trick to get X axis labels for every year\nstats \"$data_year\" using 1 nooutput\nset xrange [STATS_min:STATS_max]\nset xtics 1\nset output \"$image_year\"\nplot \"$data_year\" using 1:2 with boxes title 'Commits'\nEOF\ntimestamp \"Generated GNUplot code: $code_year\"\n\necho\n\ntimestamp \"Generating bar chart for Commits per Month\"\ngnuplot \"$code_month\"\ntimestamp \"Generated bar chart image: $image_month\"\necho\n\ntimestamp \"Generating bar chart for Commits per Year\"\ngnuplot \"$code_year\"\ntimestamp \"Generated bar chart image: $image_year\"\necho\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image_month\"\n\"$srcdir/../media/imageopen.sh\" \"$image_month\"\n\ntimestamp \"Opening: $image_year\"\n\"$srcdir/../media/imageopen.sh\" \"$image_year\"\n"
  },
  {
    "path": "git/git_graph_commit_history_mermaidjs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-04 03:03:56 +0300 (Fri, 04 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\ncode_month=\"git_commits_per_month.mmd\"\ncode_year=\"git_commits_per_year.mmd\"\n\ndata_month=\"data/git_commits_per_month.dat\"\ndata_year=\"data/git_commits_per_year.dat\"\n\nimage_month=\"images/git_commits_per_month.svg\"\nimage_year=\"images/git_commits_per_year.svg\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates MermaidJS graphs of Git commits per year and per month for the entire history of the local Git repo checkout\n\nGenerates the MermaidJS code and then uses MermaidJS CLI to generate the images\n\n    $code_month - Code\n    $code_year  - Code\n\n    $data_month - Data\n    $data_year  - Data\n\n    $image_month - Image\n    $image_year  - Image\n\nA GNUplot version of this script is adjacent at:\n\n    git_graph_commit_history_gnuplot.sh\n\nRequires Git and MermaidJS CLI (mmdc) to be installed to generate the graphs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_log_paths_to_check_only>]\"\n\nhelp_usage \"$@\"\n\ncheck_bin mmdc\n\nif ! is_in_git_repo; then\n    die \"Error: Not inside a git repository!\"\nfi\n\nfor x in $code_month  \\\n         $code_year   \\\n         $data_month  \\\n         $data_year   \\\n         $image_month \\\n         $image_year; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\n# output git commits as simple YYYY-MM and then just sort and count them by month\ntimestamp \"Calculating commit counts per month from the Git log\"\ngit log --date=format:'%Y-%m' --pretty=format:'%ad' \"$@\" |\nsort |\nuniq -c |\nawk '{print $2\" \"$1}' > \"$data_month\"\ntimestamp \"Wrote data: $data_month\"\necho\n\ntimestamp \"Calculating commit counts per year from the Git log\"\ngit log --date=format:'%Y' --pretty=format:'%ad' \"$@\" |\nsort |\nuniq -c |\nawk '{print $2\" \"$1}' > \"$data_year\"\ntimestamp \"Wrote data: $data_year\"\necho\n\nexport -f parse_file_col_to_csv\n\ntimestamp \"Generating MermaidJS code for Commits per Month\"\ncat > \"$code_month\" <<EOF\nxychart-beta\n    title \"Git Commits per Month\"\n    x-axis [ $(parse_file_col_to_csv \"$data_month\" 1) ]\n    y-axis \"Number of Commits\"\n    bar    [ $(parse_file_col_to_csv \"$data_month\" 2) ]\n    %%line [ $(parse_file_col_to_csv \"$data_month\" 2) ]\nEOF\ntimestamp \"Generated MermaidJS code: $code_month\"\necho\n\ntimestamp \"Generating MermaidJS code for Commits per Year\"\ncat > \"$code_year\" <<EOF\nxychart-beta\n    title \"Git Commits per Year\"\n    x-axis [ $(parse_file_col_to_csv \"$data_year\" 1) ]\n    y-axis \"Number of Commits\"\n    bar    [ $(parse_file_col_to_csv \"$data_year\" 2) ]\n    %%line [ $(parse_file_col_to_csv \"$data_year\" 2) ]\nEOF\ntimestamp \"Generated MermaidJS code: $code_year\"\necho\n\ntimestamp \"Generating bar chart for Commits per Month\"\nmmdc -i \"$code_month\" -o \"$image_month\" -t dark --quiet # -b transparent\ntimestamp \"Generated bar chart image: $image_month\"\necho\n\ntimestamp \"Generating bar chart for Commits per Year\"\nmmdc -i \"$code_year\" -o \"$image_year\" -t dark --quiet # -b transparent\ntimestamp \"Generated bar chart image: $image_year\"\necho\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image_month\"\n\"$srcdir/../media/imageopen.sh\" \"$image_month\"\n\ntimestamp \"Opening: $image_month\"\n\"$srcdir/../media/imageopen.sh\" \"$image_year\"\n"
  },
  {
    "path": "git/git_graph_commit_times_gnuplot.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-03 10:41:23 +0300 (Thu, 03 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\ncode=\"git_commit_times.gnuplot\"\ndata=\"data/git_commit_times.dat\"\nimage=\"images/git_commit_times.png\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a GNUplot graph of Git commit times from the current Git repo checkout's git log\n\nGenerates the following files:\n\n    $code - Code\n\n    $data  - Data\n\n    $image - Image\n\nA MermaidJS version of this script is adjacent at:\n\n    git_graph_commit_times_mermaidjs.sh\n\nRequires Git and GNUplot to be installed to generate the graph\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nif ! is_in_git_repo; then\n    die \"ERROR: must be run from a git repo checkout as it relies on the 'git log' command\"\nfi\n\nfor x in $code \\\n         $data \\\n         $image; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\ngit_repo=\"$(git_repo)\"\n\ntimestamp \"Running inside checkout of Git repo: $git_repo\"\ntimestamp \"Fetching Hour of all commits from Git log\"\ngit log --date=format:'%H' --pretty=format:'%ad' |\nsort |\nuniq -c |\nawk '{print $2\" \"$1}' > \"$data\"\necho\n\ntimestamp \"Generating GNUplot code for Commits per Hour\"\nsed '/^[[:space:]]*$/d' > \"$code\" <<EOF\n#\n# Generated by ${0##*/}\n#\n# from https://github.com/HariSekhon/DevOps-Bash-tools\n#\nset terminal pngcairo size 1280,720 enhanced font \"Arial,14\"\nset title \"$git_repo - Git Commits by Hour\"\nset xlabel \"Hour of Day (author's local time)\"\nset ylabel \"Number of Commits\"\nset grid\n#set xtics rotate by -45\nset boxwidth 0.8 relative\nset style fill solid\nset datafile separator \" \"\n# results in X axis labels every 2 years\n#set xdata time\n#set timefmt \"%H\"\n#set format x \"%H\"\n# trick to get X axis labels for every year\nstats \"$data\" using 1 nooutput\nset xrange [STATS_min:STATS_max]\nset xtics 1\nset output \"$image\"\nplot \"$data\" using 1:2 with boxes title 'Commits'\nEOF\ntimestamp \"Generated GNUplot code: $code\"\n\ntimestamp \"Generating bar chart for Commits per Hour\"\ngnuplot \"$code\"\ntimestamp \"Generated bar chart image: $image\"\necho\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "git/git_graph_commit_times_gnuplot_all_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-03 10:41:23 +0300 (Thu, 03 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\nrepolist=\"$(readlink -f \"$srcdir/../setup/repos.txt\")\"\n\ncode=\"git_commit_times_all_repos.gnuplot\"\ndata=\"data/git_commit_times_all_repos.dat\"\nimage=\"images/git_commit_times_all_repos.png\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a GNUplot graph of Git commit times from all adjacent Git repos listed in:\n\n    $repolist\n\nGenerates the following files:\n\n    $code - Code\n\n    $data  - Data\n\n    $image - Image\n\nA MermaidJS version of this script is adjacent at:\n\n    git_graph_commit_times_mermaidjs_all_repos.sh\n\nThese adjacent scripts perform a similar function but using GitHub API commit data:\n\n    ../github/github_graph_commit_times_gnuplot.sh\n\n    ../github/github_graph_commit_times_mermaidjs.sh\n\nA Golang version of this program which uses the GitHub API can be found here:\n\n    https://github.com/HariSekhon/GitHub-Graph-Commit-Times\n\nRequires Git and GNUplot to be installed to generate the graph\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nfor x in $code \\\n         $data \\\n         $image; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\nolder_than(){\n    local file=\"$1\"\n    local days=\"$2\"\n    if ! [ -f \"$file\" ]; then\n        return 0\n    fi\n    if find \"$file\" -mtime +\"$days\" -print -quit | grep -q . ; then\n        return 0\n    fi\n    return 1\n}\n\nif ! older_than \"$data\" 7; then\n    timestamp \"Using cached data since it is less than 7 days old: $data\"\nelse\n    timestamp \"Getting list of Git repo checkout directories from: $repolist\"\n    repo_dirs=\"$(sed 's/#.*//; s/.*://; /^[[:space:]]*$/d' \"$repolist\")\"\n\n    timestamp \"Found repos: $(wc -l <<< \"$repo_dirs\" | sed 's/[[:space:]]/g')\"\n    echo\n\n    while read -r repo_dir; do\n        repo_dir=\"$(readlink -f \"$srcdir/../../$repo_dir\")\"\n        timestamp \"Entering repo dir: $repo_dir\"\n        pushd \"$repo_dir\" &>/dev/null || die \"Failed to pushd to: $repo_dir\"\n        timestamp \"Fetching Hour of all commits from Git log\"\n        git log --date=format:'%H' --pretty=format:'%ad'\n        popd &>/dev/null || die \"Failed to popd from: $repo_dir\"\n        echo\n    done <<< \"$repo_dirs\" |\n    sort |\n    uniq -c |\n    awk '{print $2\" \"$1}' > \"$data\"\n    echo\nfi\n\ntimestamp \"Generating GNUplot code for Commits per Hour\"\nsed '/^[[:space:]]*$/d' > \"$code\" <<EOF\n#\n# Generated by ${0##*/}\n#\n# from https://github.com/HariSekhon/DevOps-Bash-tools\n#\nset terminal pngcairo size 1280,720 enhanced font \"Arial,14\"\nset title \"Git Commits by Hour\"\nset xlabel \"Hour of Day (author's local time)\"\nset ylabel \"Number of Commits\"\nset grid\n#set xtics rotate by -45\nset boxwidth 0.8 relative\nset style fill solid\nset datafile separator \" \"\n# results in X axis labels every 2 years\n#set xdata time\n#set timefmt \"%H\"\n#set format x \"%H\"\n# trick to get X axis labels for every year\nstats \"$data\" using 1 nooutput\nset xrange [STATS_min:STATS_max]\nset xtics 1\nset output \"$image\"\nplot \"$data\" using 1:2 with boxes title 'Commits'\nEOF\ntimestamp \"Generated GNUplot code: $code\"\n\ntimestamp \"Generating bar chart for Commits per Hour\"\ngnuplot \"$code\"\ntimestamp \"Generated bar chart image: $image\"\necho\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "git/git_graph_commit_times_mermaidjs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-04 03:03:56 +0300 (Fri, 04 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\ncode=\"git_commit_times.mmd\"\ndata=\"data/git_commit_times.dat\"\nimage=\"images/git_commit_times.svg\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a MermaidJS graph of Git commit times from the current Git repo checkout's git log\n\nGenerates the MermaidJS code and then uses MermaidJS CLI to generate the image\n\n    $code - Code\n\n    $data - Data\n\n    $image - Image\n\nA GNUplot version of this script is adjacent at:\n\n    git_graph_commit_times_gnuplot.sh\n\nRequires Git and MermaidJS CLI (mmdc) to be installed to generate the graphs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncheck_bin mmdc\n\nif ! is_in_git_repo; then\n    die \"ERROR: must be run from a git repo checkout as it relies on the 'git log' command\"\nfi\n\ngit_repo=\"$(git_repo)\"\n\ntimestamp \"Running inside checkout of Git repo: $git_repo\"\ntimestamp \"Fetching Hour of all commits from Git log\"\ngit log --date=format:'%H' --pretty=format:'%ad' |\nsort |\nuniq -c |\nawk '{print $2\" \"$1}' > \"$data\"\necho\n\nexport -f parse_file_col_to_csv\n\ntimestamp \"Generating MermaidJS code for bar chart of commit times\"\ncat > \"$code\" <<EOF\nxychart-beta\n    title \"$git_repo - Git Commits by Hour\"\n    x-axis [ $(parse_file_col_to_csv \"$data\" 1) ]\n    y-axis \"Number of Commits\"\n    bar    [ $(parse_file_col_to_csv \"$data\" 2) ]\n    %%line [ $(parse_file_col_to_csv \"$data\" 2) ]\nEOF\ntimestamp \"Generated MermaidJS code\"\necho\n\ntimestamp \"Generating MermaidJS bar chart image: $image\"\nmmdc -i \"$code\" -o \"$image\" -t dark --quiet # -b transparent\ntimestamp \"Generated MermaidJS image: $image\"\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening generated bar chart\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "git/git_graph_commit_times_mermaidjs_all_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-04 03:03:56 +0300 (Fri, 04 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\nrepolist=\"$(readlink -f \"$srcdir/../setup/repos.txt\")\"\n\ncode=\"git_commit_times_all_repos.mmd\"\ndata=\"data/git_commit_times_all_repos.dat\"\nimage=\"images/git_commit_times_all_repos.svg\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a MermaidJS graph of Git commit times from all adjacent Git repos listed in:\n\n    $repolist\n\nGenerates the MermaidJS code and then uses MermaidJS CLI to generate the image\n\n    $code - Code\n\n    $data - Data\n\n    $image - Image\n\nA GNUplot version of this script is adjacent at:\n\n    git_graph_commit_times_gnuplot_all_repos.sh\n\nThese adjacent scripts perform a similar function but using GitHub API commit data:\n\n    ../github/github_graph_commit_times_gnuplot.sh\n\n    ../github/github_graph_commit_times_mermaidjs.sh\n\nA Golang version of this program which uses the GitHub API can be found here:\n\n    https://github.com/HariSekhon/GitHub-Graph-Commit-Times\n\nRequires Git and MermaidJS CLI (mmdc) to be installed to generate the graphs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncheck_bin mmdc\n\nfor x in $code \\\n         $data \\\n         $image; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\nolder_than(){\n    local file=\"$1\"\n    local days=\"$2\"\n    if ! [ -f \"$file\" ]; then\n        return 0\n    fi\n    if find \"$file\" -mtime +\"$days\" -print -quit | grep -q . ; then\n        return 0\n    fi\n    return 1\n}\n\nif ! older_than \"$data\" 7; then\n    timestamp \"Using cached data since it is less than 7 days old: $data\"\nelse\n    timestamp \"Getting list of Git repo checkout directories from: $repolist\"\n    repo_dirs=\"$(sed 's/#.*//; s/.*://; /^[[:space:]]*$/d' \"$repolist\")\"\n\n    timestamp \"Found repos: $(wc -l <<< \"$repo_dirs\" | sed 's/[[:space:]]//g')\"\n    echo\n\n    while read -r repo_dir; do\n        repo_dir=\"$(readlink -f \"$srcdir/../../$repo_dir\")\"\n        timestamp \"Entering repo dir: $repo_dir\"\n        pushd \"$repo_dir\" &>/dev/null || die \"Failed to pushd to: $repo_dir\"\n        timestamp \"Fetching Hour of all commits from Git log\"\n        git log --date=format:'%H' --pretty=format:'%ad'\n        popd &>/dev/null || die \"Failed to popd from: $repo_dir\"\n        echo\n    done <<< \"$repo_dirs\" |\n    sort |\n    uniq -c |\n    awk '{print $2\" \"$1}' > \"$data\"\nfi\necho\n\nexport -f parse_file_col_to_csv\n\ntimestamp \"Generating MermaidJS code for bar chart of commit times\"\ncat > \"$code\" <<EOF\nxychart-beta\n    title \"Git Commits by Hour\"\n    x-axis [ $(parse_file_col_to_csv \"$data\" 1) ]\n    y-axis \"Number of Commits\"\n    bar    [ $(parse_file_col_to_csv \"$data\" 2) ]\n    %%line [ $(parse_file_col_to_csv \"$data\" 2) ]\nEOF\ntimestamp \"Generated MermaidJS code\"\necho\n\ntimestamp \"Generating MermaidJS bar chart image: $image\"\nmmdc -i \"$code\" -o \"$image\" -t dark --quiet # -b transparent\ntimestamp \"Generated MermaidJS image: $image\"\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening generated bar chart\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "git/git_grep_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-13 11:15:06 +0100 (Mon, 13 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGit Grep's tokens that look like environment variables (UPPER_UPPER) out of the current code base underneath the given dir or \\$PWD\n\nUseful to find out environment variables supported in a code base when they're not well documented\n\nWon't find short one-piece environment variables like DEBUG because otherwise we'd also return HTTPS and all sorts of other irrelevant tokens and noise.\n\nOriginally written to document ArgoCD's environment variables for better administration\n\n    https://github.com/argoproj/argo-cd/pull/8680\n\nFiles or extensions to exclude can optionally be specified as args, and must be valid ERE regex that match the file path suffix, filenames and file extension literals will usually be fine\n\n\nExamples:\n\nYou may want to exclude other files like Dockerfiles, Makefiles to just focus on environment variables supported in the actual code.\nYou can mix and match any of the following argument examples.\n\nExclude Dockerfiles and Makefiles:\n\n    ${0##*/} Dockerfile Makefile\n\n\nTo exclude all Dockerfiles like Dockerfile and Dockerfile.dev, Dockerfile.prod etc. this ERE regex must be quoted to not expand in the shell before being passed to this script:\n\n    ${0##*/} 'Dockerfile[[:alnum:].-]*'\n\n\nTo exclude all GitHub Actions workflows:\n\n    ${0##*/} '.github/workflows/[[:alnum:].-]+'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir> <files_or_extensions_to_ignore>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ndir=\"${1:-}\"\n\nif [ -n \"$dir\" ] && [ -d \"$dir\" ]; then\n    shift || :\n    cd \"$dir\"\nfi\n\nfiles_or_extensions_to_ignore=(\n\"$@\"\nmd\nasc\ncrt\nkey\nhttp\ncss\ngo.sum\nsvg\ntsx\nLICENSE\n)\n\n# XXX: tune this if needed, currently the trade off is will find things like ARGOCD_<blah> but won't find something without an underscore like DEBUG=1\n# looser, finds more, but also a lot more noise\n#environment_variable_bre_regex='[[:upper:]_]\\{4,\\}[^[:alnum:]_]'\nenvironment_variable_bre_regex='[[:upper:]]\\{2,\\}_[[:upper:]_]\\{3,\\}'\n\ndockerfile_keywords=\"$(sed 's/#.*//; /^[[:space:]]*$/d' \"$srcdir/lib/dockerfile_keywords.txt\")\"\n\n# shellcheck disable=SC2207\nfile_content_prefixes_to_ignore=(\nMakefile:.PHONY\n    $(\n        for keyword in $dockerfile_keywords; do\n            echo \"Dockerfile:$keyword\"\n        done\n    )\n\n)\n\n# generate large single regex because too many -e switches don't work on grep on Mac (bug?)\nfiles_or_extensions_regex=\"$(printf \"%s|\" \"${files_or_extensions_to_ignore[@]}\" | sed 's/|$//')\"\nfile_content_prefixes_regex=\"$(printf \"%s|\" \"${file_content_prefixes_to_ignore[@]}\" | sed 's/|$//')\"\n\n# XXX: must disable git grep color or will break filter processing due to escape codes not matching exclusion filters\n# want arg splitting\n# shellcheck disable=SC2046\ngit grep -I --color=never \"$environment_variable_bre_regex\" |\n# don't put quote inside the echo, it'll put literal quotes that will then fail to match to filter out\ngrep -Ev -e \"(^|/|\\\\.)($files_or_extensions_regex):\" -e \"$file_content_prefixes_regex\" |\nsort -ui |\nif is_piped; then\n    grep \"$environment_variable_bre_regex\"\nelse\n    # restore colour to make it easier to see the important bits of the lines\n    grep \"$environment_variable_bre_regex\" --color=yes |\n    less -RFX\nfi\n"
  },
  {
    "path": "git/git_log_empty_commits.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-17 16:41:04 +0100 (Thu, 17 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a Git log of any empty commits\n\nUseful to detect if there are empty commits in Git history\n\nThis happens when rewriting Git history with filtering, such as forking a git repo and running 'git filter-branch' but forgetting to include the --prune-empty switch\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nfor sha in $(git rev-list --min-parents=1 --max-parents=1 --all); do\n    if [ \"$(git rev-parse \"${sha}^{tree}\")\" = \"$(git rev-parse \"${sha}^1^{tree}\")\" ]; then\n        # same output as git log -p -1\n        #git show \"$sha\"\n        #git log -p -1 --graph \"$sha\"\n        # only hash and subject, not enough info\n        #git log --oneline \"$sha\"\n        #git log -1 --format='%h | %ai | %an | %s' \"$sha\" | column -t -s '|' || break\n        git show --format='%h | %ai | %an | %s' \"$sha\" | column -t -s '|' || break\n    fi\ndone\n"
  },
  {
    "path": "git/git_log_me.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-20 14:50:09 +0700 (Thu, 20 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows only commits in the Git log done by you\n\nUseful to remind yourself what parts the current Git repo you've been working on\nfor periodic reviews, reports or even updating your CV!\n\nFilters the Git log for your Git configured username and email address\n\nIf you have a global Git config using your personal email address but work repo specific overrides using your work email\nthis script will include to filter for both which will also catch commits that may have been accidentally committed\nbefore you overrode your Git email in work repo or if using Squash Merges in GitHub UI that defaulted to the wrong email\n\nPasses all args through to git log so you can add additional filters eg. to show everything added by me:\n\n    ${0##*/} --all --diff-filter=A\n\nor for compact output:\n\n    ${0##*/} --oneline\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_log_options>]\"\n\nhelp_usage \"$@\"\n\n# git config --list outputs both global and local repo overrides,\n# resulting in multiple user.email outputs for a work repo using local override\nauthors=\"$(git config -l | awk -F= '/^user.(name|email)/ {print $2}' | sort -u)\"\n\nauthor_opts=()\n\nwhile read -r author; do\n    author_opts+=(--author \"$author\")\ndone <<< \"$authors\"\n\ngit log \"${author_opts[@]}\" \"$@\"\n"
  },
  {
    "path": "git/git_log_me_added.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-20 14:50:09 +0700 (Thu, 20 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows only file addition commits in the Git log done by you\n\nUseful to remind yourself what parts the current Git repo you've added - for periodic reviews, reports\nor even updating your CV!\n\nFilters the Git log for your Git configured username and email address using adjacent script:\n\n    git_log_me.sh\n\nPasses all args through to git log so you can add additional filters eg.\n\n    ${0##*/} --all\n\n    ${0##*/} --oneline\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_log_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/git_log_me.sh\" --diff-filter=A \"$@\"\n"
  },
  {
    "path": "git/git_merge_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-21 10:43:47 +0100 (Tue, 21 Nov 2017)\n#\n#  https://github.com/HariSekhon/Dockerfiles\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# For Git < 2.0 may need to set:\n#\n# git config merge.defaultToUpstream true\n\nforeachbranch 'git merge --no-edit'\n\ngit checkout master\n"
  },
  {
    "path": "git/git_merge_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Feb 1 11:29:15 2021 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMerges one Git branch into another branch in a local Git checkout\n\nDesigned to be called by a CI build system to automatically backport via merge any changes in Staging branch into Dev branch eg.\n\n    ${0##*/} staging dev\n\nRequires executing inside a Git SSH cloned repo and an SSH key being present on the CI Agent that executes this job.\nSet the CI job to clone the repo via SSH and use the CI system's secrets mechanism for the SSH key.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<from_branch> <to_branch>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nfrom_branch=\"$1\"\nto_branch=\"$2\"\n\nset -x\n\nif is_CI; then\n    :  # running in CI as expected\n    # needed to check in\n    # XXX: can edit this to your company domain and team email address\n    git config user.email \"platform-engineering@localhost\"\n    git config user.name \"$(CI_name)\"  # lib/ci.sh CI_name function will return something appropriate\n#else\n#    echo \"This script is designed to only be run from CI!!\"\n#    exit 1\nfi\n\n# only apply to own repo\ncd \"$(dirname \"$0\")\"\n\ngit config core.sparseCheckout false\n\ngit status\n\n# doesn't exist yet\ncat ~/.ssh/known_hosts || :\n\nmkdir -pv ~/.ssh\n\n# needed for git pull to work\nssh-keyscan github.com >> ~/.ssh/known_hosts\n\n## needed to get list of remote branches before checking one out locally\ngit fetch\n\ngit checkout \"$to_branch\" --force\ngit pull --no-edit\ngit merge \"origin/$from_branch\" --no-edit\n\ngit push\n"
  },
  {
    "path": "git/git_merge_master.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-21 10:43:47 +0100 (Tue, 21 Nov 2017)\n#\n#  https://github.com/HariSekhon/Dockerfiles\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# For Git < 2.0 may need to set:\n#\n# git config merge.defaultToUpstream true\n\nforeachbranch 'git merge --no-edit && git merge master --no-edit'\n\ngit checkout master\n"
  },
  {
    "path": "git/git_merge_master_pull.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-21 10:43:47 +0100 (Tue, 21 Nov 2017)\n#\n#  https://github.com/HariSekhon/Dockerfiles\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# For Git < 2.0 may need to set:\n#\n# git config merge.defaultToUpstream true\n\nforeachbranch 'git fetch && git merge --no-edit && git merge master --no-edit'\n\ngit checkout master\n"
  },
  {
    "path": "git/git_origin_commit_count_to_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-23 03:23:55 +0700 (Thu, 23 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the number of Git commits in local branch that would be pushed to remote origin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n\"$srcdir/git_origin_log_to_push.sh\" --oneline |\nwc -l |\nsed 's/[[:space:]]*//g'\n"
  },
  {
    "path": "git/git_origin_diff_to_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-23 03:23:55 +0700 (Thu, 23 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the Git diff of lines in local branch that would be pushed to remote origin\n\nYou can give git diff options\n\nExample:\n\n    ${0##*/} --color=always | less -FR\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_diff_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nlog \"Determining current branch\"\ncurrent_branch=\"$(current_branch)\"\nlog \"Current branch: $current_branch\"\n\nlog \"Checking we have an origin\"\ngit remote get-url origin >/dev/null\n\n#git diff \"origin/$current_branch..\"\n\n# these next two work even if you haven't pushed this branch yet, but the first one can result in the following error\n#\n#git diff \"$@\" \"origin..\"\n#\n#    fatal: ambiguous argument 'origin..': unknown revision or path not in the working tree.\n#    Use '--' to separate paths from revisions, like this:\n#    'git <command> [<revision>...] -- [<file>...]'\n\ngit diff \"$@\" \"FETCH_HEAD..HEAD\"\n"
  },
  {
    "path": "git/git_origin_files_to_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-10 20:59:47 +0800 (Mon, 10 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the Git files in local branch that would be pushed to remote origin\n\nYou can give git diff options\n\nExample:\n\n    ${0##*/} --color=always | less -FR\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_diff_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nlog \"Determining current branch\"\ncurrent_branch=\"$(current_branch)\"\nlog \"Current branch: $current_branch\"\n\nlog \"Checking we have an origin\"\ngit remote get-url origin >/dev/null\n\n#git diff --name-status \"origin/$current_branch..\"\n\n# these next two work even if you haven't pushed this branch yet, but the first one can result in the following error\n#\n#git diff --name-status \"$@\" \"origin..\"\n#\n#    fatal: ambiguous argument 'origin..': unknown revision or path not in the working tree.\n#    Use '--' to separate paths from revisions, like this:\n#    'git <command> [<revision>...] -- [<file>...]'\n\ngit diff --name-status \"$@\" \"FETCH_HEAD..HEAD\"\n"
  },
  {
    "path": "git/git_origin_line_count_to_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-23 03:23:55 +0700 (Thu, 23 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the Git number of lines changed in local branch that would be pushed to remote origin\n\nThese are lines actually added / changed / removed without surrounding context lines\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\n\"$srcdir/git_origin_diff_to_push.sh\" --unified=0 |\nsed -n '/^[+-]/ {/^\\(---\\|+++\\)/!p}' |\nsed '$ { /^[[:space:]]*$/d }' |\nwc -l |\nsed 's/[[:space:]]*//g'\n"
  },
  {
    "path": "git/git_origin_log_to_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-23 03:23:55 +0700 (Thu, 23 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the Git log in local branch that would be pushed to remote origin\n\nYou can give git log options like --oneline\n\nExample:\n\n    ${0##*/} --oneline | wc -l\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<git_log_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nlog \"Determining current branch\"\ncurrent_branch=\"$(current_branch)\"\nlog \"Current branch: $current_branch\"\n\nlog \"Checking we have an origin\"\ngit remote get-url origin >/dev/null\n\n#git log \"origin/$current_branch..\"\n\n# these next two work even if you haven't pushed this branch yet, but the first one can result in the following error\n#\n#git log \"$@\" \"origin..\"\n#\n#    fatal: ambiguous argument 'origin..': unknown revision or path not in the working tree.\n#    Use '--' to separate paths from revisions, like this:\n#    'git <command> [<revision>...] -- [<file>...]'\n\ngit log \"$@\" \"FETCH_HEAD..HEAD\"\n"
  },
  {
    "path": "git/git_pull_make_repos.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# during 'curl | bash' ... $BASH_SOURCE isn't set\nif [ -n \"${BASH_SOURCE[0]:-}\" ]; then\n    srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nfi\n\ngit_url=\"${GIT_URL:-https://github.com}\"\n\nmake=\"${MAKE:-make}\"\nbuild=\"${BUILD:-build}\"\n\ngit_base_dir=~/github\n\nmkdir -pv \"$git_base_dir\"\n\ncd \"$git_base_dir\"\n\nopts=\"${OPTS:-}\"\nif [ -z \"${NO_TEST:-}\" ]; then\n    opts=\"$opts test\"\nfi\n\nif [ -z \"${JAVA_HOME:-}\" ]; then\n    set +e\n    JAVA_HOME=\"$(which java 2>/dev/null)/..\"\n    if [ -z \"${JAVA_HOME:-}\" ]; then\n        JAVA_HOME=\"$(type java 2>/dev/null | sed 's/java is //; s/hashed //; s/[()]//g')\"\n    fi\n    set -e\n    if [ -z \"${JAVA_HOME:-}\" ]; then\n        JAVA_HOME=\"/usr\"\n    fi\nfi\n\nif ! type -P git &>/dev/null ||\n   ! type -P make &>/dev/null &&\n   [ -n \"${srcdir:-}\" ]; then\n#    if type -P yum &>/dev/null; then\n#        yum install -y git make\n#    elif type -P apt-get &>/dev/null; then\n#        apt-get update\n#        apt-get install -y --no-install-recommends git make\n#    elif type -P apk &>/dev/null; then\n#        apk update\n#        apk add git make\n#    fi\n    \"$srcdir/../packages/install_packages.sh\" git make\nfi\n\nif [ -n \"${REPOS:-}\" ]; then\n    tr '[:space:]' '\\n' <<< \"$REPOS\"\nelif [ -n \"${srcdir:-}\" ]; then\n    sed 's/#.*//; s/:/ /; /^[[:space:]]*$/d' < \"$srcdir/../setup/repos.txt\"\nelse\n    echo \"\\$REPOS not set and \\$srcdir not set/available, possibly due to 'curl ... | bash' usage, cannot determine list of repos to pull and build\" >&2\n    exit 1\nfi |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    if ! echo \"$repo\" | grep -q \"/\"; then\n        repo=\"HariSekhon/$repo\"\n    fi\n    if ! [ -d \"$dir\" ]; then\n        git clone \"$git_url/$repo\" \"$dir\"\n    fi\n    pushd \"$dir\"\n    git pull --no-edit\n    git submodule update --init\n    #  shellcheck disable=SC2086\n    if [ -z \"${NOBUILD:-}\" ] &&\n       [ -z \"${NO_BUILD:-}\" ]; then\n        \"$make\" \"$build\" $opts\n    fi\n    if [ -f /.dockerenv ]; then\n        for x in system-packages-remove clean deep-clean; do\n            if grep -q \"^$x:\" Makefile bash-tools/Makefile.in 2>/dev/null; then\n                $make \"$x\"\n            fi\n        done\n    fi\n    popd\ndone\n"
  },
  {
    "path": "git/git_push_stats.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-23 03:23:55 +0700 (Thu, 23 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the Git push stats to the remote origin for the current branch - number of commits and lines of diff\n\nUtilizes adjacent scripts:\n\n    git_origin_commit_count_to_push.sh\n\n    git_origin_line_count_to_push.sh\n\n    git_origin_diff_to_push.sh\n\n    git_origin_files_to_push.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nnum_commits=\"$(\"$srcdir/git_origin_commit_count_to_push.sh\")\"\n\nnum_lines_changed=\"$(\"$srcdir/git_origin_line_count_to_push.sh\")\"\n\nnum_lines_diff=\"$(\"$srcdir/git_origin_diff_to_push.sh\" | wc -l | sed 's/[[:space:]]*//g')\"\n\nfiles=\"$(\"$srcdir/git_origin_files_to_push.sh\")\"\n\nfiles_added=\"$(grep -c '^A' <<< \"$files\" || :)\"\n\nfiles_modified=\"$(grep -c '^M' <<< \"$files\" || :)\"\n\nfiles_deleted=\"$(grep -c '^D' <<< \"$files\" || :)\"\n\nfiles_renamed=\"$(grep -c '^R' <<< \"$files\" || :)\"\n\nfiles_other=\"$(grep -c -v '^[AMDR]' <<< \"$files\" || :)\"\n\nfiles_total=\"$(grep -c '.' <<< \"$files\" || :)\"\n\ncat <<EOF\nStats for Push to Origin:\n\n    $(git remote -v | awk '/^origin[[:space:]]/{print $2; exit}')\n\nNumber of Commits: $num_commits\n\nNumber of Lines Changed: $num_lines_changed\n\nNumber of Lines Diff: $num_lines_diff\n\nFiles Added:    $files_added\nFiles Modified: $files_modified\nFiles Deleted:  $files_deleted\nFiles Renamed:  $files_renamed\nFiles Other:    $files_other\nFiles Total:    $files_total\n\nEOF\n\nif [ \"$files_total\" != \"$((files_added + files_modified + files_deleted + files_renamed))\" ]; then\n    echo\n    warn \"Total Files != ( Added + Modified + Deleted + Renamed )\n\nCheck what the 'Other Files' are\n\"\nfi\n"
  },
  {
    "path": "git/git_remotes_add_origin_providers.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets up Git remotes to one or more of the major public Git Repos - GitHub, GitLab, Bitbucket or Azure DevOps\n\nUseful to:\n\n1. easily pull / push to each provider by name\n2. fetch and merge updates from all Repo providers via a single command:\n\n    git pull --all\n\nSee Also:\n\n    git_remotes_set_multi_origin.sh  - for push to all\n    git_sync_repos_upstream.sh       - for sync'ing all repos to another provider\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"github|gitlab|bitbucket|azure|all\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nmin_args 1 \"$@\"\n\nname=\"${1:-}\"\n\nif [ -z \"$name\" ]; then\n    usage\nfi\n\nadd_remote_repo(){\n    local name=\"$1\"\n    local domain=\"unconfigured\"\n    # used by log statements\n    # shellcheck disable=SC2034\n    local VERBOSE=1\n    if git remote -v | grep -Eq \"^${name}[[:space:]]\"; then\n        log \"$name remote already configured, skipping...\"\n        return 0\n    fi\n    # loads domain and variables user and token if available via environment variables\n    git_provider_env \"$name\"\n    log \"$name remote not configured, configuring...\"\n    set +o pipefail\n    url=\"$(git remote -v | awk '{print $2}' | grep -Ei \"$domain\" | head -n 1)\"\n    if [ -n \"$url\" ]; then\n        log \"copied existing remote url for $name as is including any access tokens to named remote $name\"\n    else\n        url=\"$(git remote -v | awk '{print $2}' | grep -Ei 'bitbucket.org|github.com|gitlab.com|dev.azure.com' | head -n 1 | perl -pe \"s/^((\\\\w+:\\\\/\\\\/)?(git@)?)(.+@)?[^\\\\/:]+/\\$1$domain/\")\"\n        # XXX: Azure DevOps has non-uniform URLs compared to the 3 main Git repos\n        if [ \"$name\" = \"azure\" ]; then\n            url=\"$(git_to_azure_url \"$url\")\"\n        else\n            # undo weird Azure DevOps url components if we happen to infer URL from an Azure DevOps url\n            url=\"$(azure_to_git_url \"$url\")\"\n        fi\n        # XXX: shouldn't really print full url below in case it has an http access token in it that we don't want appearing as plaintext on the screen\n        log \"inferring $name URL to be $url\"\n        log \"adding remote $name with url $url\"\n        if [[ \"$url\" =~ ^https:// ]]; then\n            if [ -n \"${user:-}\" ] && [ -n \"${token:-}\" ]; then\n                log \"added authentication credentials from environment\"\n                url=\"https://$user:$token@${url##https://}\"\n            fi\n        elif ! [[ \"$url\" =~ git@ ]]; then\n            url=\"git@${url##ssh:\\/\\/}\"\n        fi\n    fi\n    set -o pipefail\n    git remote add \"$name\" \"$url\"\n    #log \"pulling from $name to merge if necessary\"\n    #git pull --no-edit \"$name\" master\n    #log \"pushing to $name remote\"\n    #git push \"$name\" master\n}\n\nif [ \"$name\" = \"github\" ] ||\n   [ \"$name\" = \"gitlab\" ] ||\n   [ \"$name\" = \"bitbucket\" ] ||\n   [ \"$name\" = \"azure\" ]; then\n    add_remote_repo \"$name\"\n    echo >&2\n    git remote -v | sed 's|://.*@|://|'\nelif [ \"$name\" = \"all\" ]; then\n    for name in github gitlab bitbucket azure; do\n        add_remote_repo \"$name\"\n    done\n    echo >&2\n    git remote -v | sed 's|://.*@|://|'\nelse\n    usage\nfi\n"
  },
  {
    "path": "git/git_remotes_set_https_creds_helpers.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-09 02:48:01 +0000 (Fri, 09 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates Git credential helper config for the major Git hosting providers if their tokens are found in the local environment\n\nSupport GitHub, GitLab, Bitbucket and Azure DevOps\n\nChecks for the following environment variables and if they're set then set up the credentials helper for dynamic HTTPS credentials\n\nThis is useful because Azure DevOps personal access tokens have a maximum expiry of 1 year so you don't want to hardcode them across two dozen git repo checkouts but rather all inherit the auth from the environment\n\nIf you travel and lot, sometimes you can only git push through HTTPS due to egress port filtering in hotels or corporate firewalls\n\nWhen combined with the adjacent scripts\n\n    git_remotes_set_ssh_to_https.sh\n        and\n    git_foreach_repo.sh\n\nthis allows you to quickly switch many git repo checkouts from SSH to HTTPS and have them all inherit the environment auth tokens\n\nIf the GIT_GLOBAL_CONFIG environment variable is set to any value, then sets it at the global user config level instead of in the local repo\n\nIf no provider is found, checks for the following environment variables and sets them automatically in the local git repo if they're both found\n\nGitHub:\n\n    GITHUB_USER\n    GITHUB_TOKEN / GH_TOKEN\n\nGitLab:\n\n    GITLAB_USER\n    GITLAB_TOKEN\n\nBitbucket:\n\n    BITBUCKET_USER\n    BITBUCKET_TOKEN\n\nAzure DevOps:\n\n    AZURE_DEVOPS_USER\n    AZURE_DEVOPS_TOKEN\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<github|gitlab|bitbucket|azure|all>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nprovider=\"${1:-all}\"\n\n#global=\"${2:+--global}\"\nglobal=\"${GIT_GLOBAL_CONFIG:+--global}\"\n\ngithub_cred_helper(){\n    # GH_TOKEN is higher precedence in GitHub CLI so do the same here for consistency to avoid non-intuitive auth problems where one works and the other doesn't using different tokens\n    if [ -n \"${GH_TOKEN:-${GITHUB_TOKEN:-}}\" ]; then\n        timestamp \"Setting credential helper for GitHub\"\n        # shellcheck disable=SC2016,SC2086\n        for fqdn in github.com gist.github.com; do\n            # username is not needed in practice when using a token, neither on github.com, nor on gist.github.com after the first time for https://<username>@github.com/...\n            # but put it back in because that first time leads to problems\n            #git config $global \"credential.https://$fqdn.helper\" '!f() { sleep 1; echo \"password=${GH_TOKEN:-${GITHUB_TOKEN}}\"; }; f'\n            # env vars need to be evaluated dynamically not here\n            # $global must not be quoted because when it is empty this will break the command with a '' arg\n            git config $global \"credential.https://$fqdn.helper\" '!f() { sleep 1; echo \"username=${GITHUB_USER}\"; echo \"password=${GH_TOKEN:-${GITHUB_TOKEN}}\"; }; f'\n        done\n    else\n        timestamp \"NOT setting credential helper for GitHub since \\$GH_TOKEN / \\$GITHUB_TOKEN not found in environment\"\n    fi\n}\n\ngitlab_cred_helper(){\n    #if [ -n \"${GITHUB_USER:-}\" ]; then\n        if [ -n \"${GITLAB_TOKEN:-}\" ]; then\n            timestamp \"Setting credential helper for GitLab\"\n            # shellcheck disable=SC2016,SC2086\n            #git config credential.https://gitlab.com.helper '!f() { sleep 1; echo \"username=${GITLAB_USER}\"; echo \"password=${GITLAB_TOKEN}\"; }; f'\n            # $global must not be quoted because when it is empty this will break the command with a '' arg\n            git config $global credential.https://gitlab.com.helper '!f() { sleep 1; echo \"password=${GITLAB_TOKEN}\"; }; f'\n        else\n            timestamp \"NOT setting credential helper for GitLab since \\$GITLAB_TOKEN not found in environment\"\n        fi\n    #fi\n}\n\nbitbucket_cred_helper(){\n    #if [ -n \"${GITHUB_USER:-}\" ]; then\n        if [ -n \"${BITBUCKET_TOKEN:-}\" ]; then\n            timestamp \"Setting credential helper for Bitbucket\"\n            # shellcheck disable=SC2016,SC2086\n            #git config credential.https://bitbucket.org.helper '!f() { sleep 1; echo \"username=${BITBUCKET_USER}\"; echo \"password=${BITBUCKET_TOKEN}\"; }; f'\n            # $global must not be quoted because when it is empty this will break the command with a '' arg\n            git config $global credential.https://bitbucket.org.helper '!f() { sleep 1; echo \"password=${BITBUCKET_TOKEN}\"; }; f'\n        else\n            timestamp \"NOT setting credential helper for Bitbucket since \\$BITBUCKET_TOKEN not found in environment\"\n        fi\n    #fi\n}\n\nazure_devops_cred_helper(){\n    #if [ -n \"${GITHUB_USER:-}\" ]; then\n        if [ -n \"${AZURE_DEVOPS_TOKEN:-}\" ]; then\n            timestamp \"Setting credential helper for Azure DevOps\"\n            # shellcheck disable=SC2016,SC2086\n            #git config credential.https://dev.azure.com.helper '!f() { sleep 1; echo \"username=${AZURE_DEVOPS_USER}\"; echo \"password=${AZURE_DEVOPS_TOKEN}\"; }; f'\n            # $global must not be quoted because when it is empty this will break the command with a '' arg\n            git config $global credential.https://dev.azure.com.helper '!f() { sleep 1; echo \"password=${AZURE_DEVOPS_TOKEN}\"; }; f'\n        else\n            timestamp \"NOT setting credential helper for Azure DevOps since \\$AZURE_DEVOPS_TOKEN not found in environment\"\n        fi\n    #fi\n}\n\nif [ \"$provider\" = all ]; then\n    github_cred_helper\n    gitlab_cred_helper\n    bitbucket_cred_helper\n    azure_devops_cred_helper\nelif [ \"$provider\" = github ]; then\n    github_cred_helper\nelif [ \"$provider\" = gitlab ]; then\n    gitlab_cred_helper\nelif [ \"$provider\" = bitbucket ]; then\n    bitbucket_cred_helper\nelif [ \"$provider\" = azure_devops ]; then\n    azure_devops_cred_helper\nelse\n    usage \"unrecognized provider specified: $provider\"\nfi\n"
  },
  {
    "path": "git/git_remotes_set_https_to_ssh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-06 10:47:18 +0100 (Tue, 06 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChanges all of the current repo's remote URLs from https:// to git@ (SSH)\n\nHas some extra rules for conversion to Azure DevOps ssh path format since this differs from standard GitHub / GitLab / Bitbucket type paths\n\"\n\nhelp_usage \"$@\"\n\ntopdir=\"$(git_root)\"\n\ncd \"$topdir\"\n\ncp -iv -- .git/config \".git/config.$(date +%F_%H%M%S).bak\"\n\n# XXX: only replace first / with : since we're setting to git@ prefix without ssh:// prefix - if ssh://git@ then it uses slashes throughout\nperl -pi -e 's/(https:\\/\\/[^\\/]+)\\//\\1:/;\n             s/https:\\/\\/(.+\\@)?/git\\@/;\n             ' .git/config\n\nazure_devops_url=\"$(grep -m1 '^[[:space:]]*url[[:space:]]*=[[:space:]]*.*dev.azure.com' .git/config | sed 's/.*url[[:space:]]*=[[:space:]]*//; s/[[:space:]]*$//' || :)\"\n\nif [ -n \"$azure_devops_url\" ]; then\n    azure_devops_url2=\"$(git_to_azure_url \"$azure_devops_url\")\"\n\n    sed -i.bak \"s|$azure_devops_url|$azure_devops_url2|\" .git/config\nfi\n\necho >&2\ngit remotes -v\n"
  },
  {
    "path": "git/git_remotes_set_multi_origin.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets up Git multi-origin to one or more of the major public Git repos - GitHub, GitLab, Bitbucket or Azure DevOps\nfor the local checkout so that each push sync's to multiple provider's repos\n\nTo pull from multiple Repo providers in a single command:\n\n    git pull --all\n\nsee the related script:\n\n    git_remotes_add_origin_providers.sh\n\nSee Also:\n\n    git_remotes_add_origin_providers.sh  - creates remotes for easy individual pull/pushes or git pull --all\n    git_sync_repos_upstream.sh           - for sync'ing all repos to another provider\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"github|gitlab|bitbucket|azure|all\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nmin_args 1 \"$@\"\n\nadd_origin_url(){\n    local name=\"$1\"\n    local domain=\"unconfigured\"\n    # loads domain and variables user and token if available via environment variables\n    git_provider_env \"$name\"\n    if git remote -v | grep -Eq \"^origin[[:space:]]+.+$domain\"; then\n        echo \"$name remote already configured in origin, skipping...\"\n        return 0\n    fi\n    echo \"$name remote not configured in origin, configuring...\"\n    set +o pipefail\n    url=\"$(git remote -v | awk '{print $2}' | grep -Ei \"$domain\" | head -n 1)\"\n    if [ -n \"$url\" ]; then\n        echo \"copied existing remote url for $name as is including any access tokens to named remote $name\"\n    else\n        url=\"$(\n            git remote -v |\n            awk '{print $2}' |\n            grep -Ei 'bitbucket.org|github.com|gitlab.com|dev.azure.com' |\n            head -n 1 |\n            perl -pe \"\n                s/^(\\\\w+:\\\\/\\\\/)[^\\\\/]+/\\$1$domain/;\n                s/^(git@)[^:]+/\\$1$domain/\n            \"\n        )\"\n        # XXX: Azure DevOps has non-uniform URLs compared to the 3 main Git repos\n        if [ \"$name\" = \"azure\" ]; then\n            url=\"$(git_to_azure_url \"$url\")\"\n        else\n            # undo weird Azure DevOps url components if we happen to infer URL from an Azure DevOps url\n            url=\"$(azure_to_git_url \"$url\")\"\n        fi\n        # XXX: shouldn't really print full url below in case it has an http access token in it that we don't want appearing as plaintext on the screen, but git remote -v will print it later on anyway\n        echo \"inferring $name URL to be $url\"\n        echo \"adding additional origin remote for $name with url $url\"\n    fi\n    set -o pipefail\n    # --push is willing to add duplicates, prefer error out (should never happen as we check for existing remote url)\n    #git remote set-url --add --push origin \"$url\"\n    git remote set-url --add origin \"$url\"\n}\n\nfor name in \"$@\"; do\n    if [ \"$name\" = \"github\" ] ||\n       [ \"$name\" = \"gitlab\" ] ||\n       [ \"$name\" = \"bitbucket\" ] ||\n       [ \"$name\" = \"azure\" ]; then\n        add_origin_url \"$name\"\n        echo >&2\n        # TMI\n        #git remote show origin\n        git remote -v | grep '^origin' | sed 's|://.*@|://|'\n    elif [ \"$name\" = \"all\" ]; then\n        for name2 in github gitlab bitbucket azure; do\n            add_origin_url \"$name2\"\n        done\n        echo >&2\n        #git remote show origin\n        git remote -v | grep '^origin' | sed 's|://.*@|://|'\n    else\n        usage\n    fi\ndone\n"
  },
  {
    "path": "git/git_remotes_set_ssh_to_https.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-06 10:47:18 +0100 (Tue, 06 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChanges all of the current repo's remote URLs from ssh:// or git@ (SSH) to https://\n\nHas some extra rules for conversion to Azure DevOps https path format since this differs from standard GitHub / GitLab / Bitbucket type paths\n\nIf you travel and lot, sometimes you can only git push through HTTPS due to egress port filtering in hotels or corporate firewalls\n\nUsed to include authentication tokens in the generated URLs if found in the environment\n\nHowever, it's better to instead run the below script to get the HTTPS API tokens dynamically from environment variables:\n\n    git_remotes_set_https_creds_helpers.sh\n\"\n\nhelp_usage \"$@\"\n\ntopdir=\"$(git_root)\"\n\ncd \"$topdir\"\n\ncp -iv -- .git/config \".git/config.$(date +%F_%H%M%S).bak\"\n\n# XXX: only replace first / with : if git@, if ssh://git@ then it uses slashes throughout\nperl -pi -e 's/(\\bgit@[^:]+):/\\1\\//;\n             s/ssh:\\/\\/(.+@)?/https:\\/\\//;\n             s/\\bgit@/https:\\/\\//;\n             ' .git/config\n\nazure_devops_url=\"$(grep -m1 '^[[:space:]]*url[[:space:]]*=[[:space:]]*.*dev.azure.com' .git/config |\n                    sed 's/.*url[[:space:]]*=[[:space:]]*//; s/[[:space:]]*$//' || :)\"\n\nif [ -n \"$azure_devops_url\" ]; then\n    azure_devops_url2=\"$(git_to_azure_url \"$azure_devops_url\")\"\n\n    sed -i.bak \"s|$azure_devops_url|$azure_devops_url2|\" .git/config\nfi\n\n# not using this now - use git_remotes_set_https_creds_helpers.sh instead\n#embed_tokens(){\n#    for x in github gitlab bitbucket azure; do\n#        git_provider_env \"$x\"\n#        # variables loaded by git_provider_env()\n#        # inject user:token for https authentication\n#        # shellcheck disable=SC2154\n#        perl -pi -e \"s/(?<!\\\\@)$domain/$user:$token\\\\@$domain/;\" .git/config\n#        # remove prefix if there is no $token eg. ':<blank>@'\n#        # strip : prefix if there is no $user\n#        perl -pi -e 's/\\/\\/[^:]*:\\@/\\/\\//;\n#                     s/\\/\\/:/\\/\\//;' .git/config\n#    done\n#}\n\necho >&2\ngit remotes -v\n"
  },
  {
    "path": "git/git_repos.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 18:54:17 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nrepofile=\"$srcdir/../setup/repos.txt\"\n\nif [ -f \"$repofile\" ]; then\n    echo \"processing repos from local file: $repofile\" >&2\n    cat \"$repofile\"\nelse\n    echo \"fetching repos from GitHub repos.txt:\" >&2\n    curl -sSL https://raw.githubusercontent.com/HariSekhon/bash-tools/master/setup/repos.txt\nfi |\nsed 's/#.*//; s/.*://; /^[[:space:]]*$/d'\n"
  },
  {
    "path": "git/git_repos_pull.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Pulls all my Git repos listed in setup/repos.txt to ~/github/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ \"${GIT_HTTPS:-}\" ]; then\n    git_url=\"${GIT_URL:-https://github.com/}\"\n    if [ -n \"${GITHUB_TOKEN:-}\" ] && ! [[ \"$git_url\" =~ @ ]]; then\n        git_url=\"https://$GITHUB_TOKEN@${git_url#https://}\"\n    fi\nelse\n    git_url=\"${GIT_URL:-git@github.com:}\"\nfi\n\ngit_base_dir=~/github\n\nmkdir -pv \"$git_base_dir\"\n\ncd \"$git_base_dir\"\n\nsed 's/#.*//; s/:/ /; /^[[:space:]]*$/d' \"$srcdir/../setup/repos.txt\" |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    if ! echo \"$repo\" | grep -q \"/\"; then\n        repo=\"HariSekhon/$repo\"\n    fi\n    if [ -d \"$dir\" ]; then\n        pushd \"$dir\"\n        # make update does git pull but if that mechanism is broken then this first git pull will allow the repo to self-fix itself\n        git pull --no-edit\n        popd\n    else\n        git clone \"${git_url}${repo}\" \"$dir\"\n    fi\ndone\n"
  },
  {
    "path": "git/git_repos_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Pulls all my Git repos listed in setup/repos.txt to ~/github/ and runs a 'make update' build\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ngit_url=\"${GIT_URL:-https://github.com}\"\n\ngit_base_dir=~/github\n\nmkdir -pv \"$git_base_dir\"\n\ncd \"$git_base_dir\"\n\nsed 's/#.*//; s/:/ /; /^[[:digit:]]*$/d' \"$srcdir/../setup/repos.txt\" |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    if ! echo \"$repo\" | grep -q \"/\"; then\n        repo=\"HariSekhon/$repo\"\n    fi\n    if [ -d \"$dir\" ]; then\n        pushd \"$dir\"\n        # make update does git pull but if that mechanism is broken then this first git pull will allow the repo to self-fix itself\n        if [ -n \"${QUICK:-}\" ] ||\n           [ -n \"${NOBUILD:-}\" ] ||\n           [ -n \"${NO_BUILD:-}\" ]; then\n            make update-no-recompile || exit 1\n        else\n            make update\n        fi\n        popd\n    else\n        git clone \"$git_url/$repo\" \"$dir\"\n    fi\ndone\n"
  },
  {
    "path": "git/git_revert_line.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-19 18:48:29 +0400 (Tue, 19 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReverts the first line that matches a given regex from the Git head commit's version of the same line number\n\nUseful to revert some changes caused by over zealous sed'ing scripts, where you want to cherry-pick revert a single line change\n\nOnly reverts the first matching line for safety in case you do something stupid like passing . and end up losing all uncommitted changes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<file> <ere_regex>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nfile=\"$1\"\nregex=\"$2\"\n\ntimestamp \"Determining line number in file '$file' that matches regex: $regex\"\nline_number=\"$(grep -En \"$regex\" \"$file\" | head -n1 | cut -d: -f1 || :)\"\n\nif ! is_int \"$line_number\"; then\n    die \"Failed to find line matching regex: $regex\"\nfi\n\ntimestamp \"Matched line number: $line_number\"\n\ntimestamp \"Determining original line from HEAD revision\"\noriginal_line=$(git show HEAD:\"$file\" | sed -n \"${line_number}p\")\nif is_blank \"$original_line\"; then\n    die \"Failed to find original line in HEAD revision matching line number: $line_number\"\nfi\ntimestamp \"Original line found: $original_line\"\n\noriginal_line=\"${original_line//\\//\\\\/}\"\noriginal_line=\"${original_line//\\!/\\\\!}\"\noriginal_line=\"${original_line//\\&/\\\\&}\"\n\ntimestamp \"Replacing line number $line_number\"\nsed -i \"${line_number}s/.*/$original_line/\" \"$file\"\ntimestamp \"Replaced line $line_number from HEAD revision\"\n"
  },
  {
    "path": "git/git_review_push.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-08-05 15:49:27 +0100\n#  (migrated out of .bash.d/git.sh and gitconfig for use in IntelliJ)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows a git diff of what would be pushed and if you hit enter then pushes to origin remote\n\nSee also git_remotes_set_multi_origin.sh if you want to be able to push to multiple origin remotes for resilience\nby using different platforms eg. GitHub, GitLab, Azure DevOps and Bitbucket\n\nLazy but awesome for lots of daily quick intermediate diff-review-pushes workflow\n\nMigrated from .gitconfig alias and vimrc hotkey\n\nPorted to external script be callable from IntelliJ as an External Tool because it's less keystrokes\nand no mouse movement compared to IntelliJ's own hot key git commit tooling\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ntimestamp \"Pulling to ensure we have the latest remote contents\"\necho >&2\ngit pull\necho >&2\n\ntimestamp \"Comparing HEAD vs FETCH_HEAD\"\necho >&2\ntimestamp \"Checking git log\"\ncommits=\"$(git log --oneline FETCH_HEAD..HEAD)\"\necho >&2\n\nif is_blank \"$commits\"; then\n    timestamp \"No changes to push\"\n    exit 0\nfi\n\ntimestamp \"Getting diff\"\ndiff=\"$(git diff --color=always FETCH_HEAD..HEAD)\"\n\n{\n    echo\n    \"$srcdir/git_push_stats.sh\"\n    echo \"$diff\"\n} | less -FR\n\nif is_blank \"$diff\"; then\n    timestamp \"No changes to push, but commit difference (empty commits?)\"\n    echo >&2\nfi\n\nread -r -p \"Push to origin remote? (Y/n) \" answer\necho >&2\n\ncheck_yes \"${answer:-Y}\"\n\ngit push\n\necho >&2\ntimestamp \"Git Diff Review Push completed\"\n"
  },
  {
    "path": "git/git_set_dir_safe.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-03 20:07:09 +0100 (Wed, 03 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the current or given git directory and submodules to be marked as safe\n\nNecessary for some CI/CD systems like Azure DevOps Pipelines which have incorrect ownership on the git checkout dir triggering this error:\n\n    fatal: detected dubious ownership in repository at '/code/sql'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# this script is standalone without lib dependency so can be called directly from bootstrapped CI before submodules, since that is the exact problem that needs to be solved to allow CI/CD systems with incorrect ownership of the checkout directory to be able to checkout the necessary git submodules\n\"$srcdir/../setup/ci_git_set_dir_safe.sh\" \"${1:-.}\"\n"
  },
  {
    "path": "git/git_submodules_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates all Git submodules in the current Git repo to the latest default branch commit and commits them\n\nThis is also useful for simpler use cases:\n\n    git submodule foreach --recursive 'git checkout master && git pull'\n\nBut this script will figure out a mix or repos on master vs main vs develop branches as trunk\nand has better output and error handling\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nif ! is_git_repo .; then\n\tdie 'ERROR: Not in a Git repository!'\nfi\n\ngit_root=\"$(git_root)\"\n\ncd \"$git_root\"\n\ngit pull --no-edit\n\nsubmodules=\"$(\n    git submodule |\n    awk '{print $2}'\n)\"\n\nif is_blank \"$submodules\"; then\n    echo \"No Git Submodules detected\"\n    exit 0\nfi\n\ntimestamp \"Git submodules detected:\"\necho >&2\necho \"$submodules\" >&2\necho >&2\n\nfor submodule in $(git submodule | awk '{print $2}'); do\n\t[ -d \"$submodule\" ] || continue\n\t[ -L \"$submodule\" ] && continue\n    timestamp \"Updating submodule: $submodule\"\n    echo >&2\n\tpushd \"$submodule\" ||\n        die \"ERROR: Failed to pushd to submodule directory: $submodule\"\n\tgit stash\n\tgit checkout \"$(default_branch)\"\n\tgit pull --no-edit\n\tgit submodule update --init --remote\n\tgit submodule update --recursive\n\tpopd\n    echo >&2\ndone\n\nfor submodule in $(git submodule | awk '{print $2}'); do\n\t[ -d \"$submodule\" ] || continue\n\t[ -L \"$submodule\" ] && continue\n    timestamp \"Committing submodule: $submodule\"\n    echo >&2\n\tif ! git status \"$submodule\" |\n\t\t grep -q nothing; then\n\t\tgit commit -m \"updated $submodule\" \"$submodule\" ||\n\t\tdie \"ERROR: Failed to commit submodule update\"\n\tfi\n    echo >&2\ndone\n\necho\n\nfor submodule in $(git submodule | awk '{print $2}'); do\n\t[ -d \"$submodule\" ] || continue\n\t[ -L \"$submodule\" ] && continue\n    timestamp \"Committing submodule: $submodule\"\n    echo >&2\n\tpushd \"$submodule\" ||\n        die \"ERROR: Failed to pushd to submodule directory: $submodule\"\n\tgit stash pop || :\n\tpopd\n    echo >&2\ndone\n"
  },
  {
    "path": "git/git_submodules_update_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:14:06 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\nrepofile=\"$srcdir/../setup/repos.txt\"\nrepofile=\"$(readlink -f \"$repofile\")\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates all Git repos given as args or listed in file:\n\n    $repofile\n\n\nUses adjacent script:\n\n    git_submodules_update.sh\n\n\nEnvironment Variables:\n\n    GIT_BASE_DIR - default: $HOME/github - checks out the repos to this location if they are not already present\n\n    GIT_URL - default: https://github.com\n\n    GIT_OWNER - default: HariSekhon - used only when omitting the owner/ prefix of owner/repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#num_args 0 \"$@\"\n\ngit_url=\"${GIT_URL:-https://github.com}\"\n\nHOME=\"${HOME:-$(cd && pwd)}\"\n\ngit_base_dir=\"${GIT_BASE_DIR:-$HOME/github}\"\n\ngit_owner=\"${GIT_OWNER:-HariSekhon}\"\n\nmkdir -pv \"$git_base_dir\"\n\ncd \"$git_base_dir\"\n\nif [ $# -gt 0 ]; then\n    repolist=\"$*\"\nelse\n    repolist=\"${*:-${REPOS:-}}\"\n    if [ -n \"$repolist\" ]; then\n        :\n    elif [ -f \"$repofile\" ]; then\n        echo \"processing repos from file: $repofile\"\n        repolist=\"$(sed 's/#.*//; /^[[:space:]]*$/d' < \"$repofile\")\"\n    else\n        echo \"fetching repos from GitHub repo list\"\n        repolist=\"$(curl -sSL https://raw.githubusercontent.com/HariSekhon/bash-tools/master/setup/repos.txt | sed 's/#.*//')\"\n    fi\nfi\n\nrun(){\n    local repolist=\"$*\"\n    timestamp \"Updating Git submodules for all repos: $repolist\"\n    echo >&2\n    for repo in $repolist; do\n        repo_dir=\"${repo##*/}\"\n        repo_dir=\"${repo_dir##*:}\"\n        repo=\"${repo%%:*}\"\n        if ! echo \"$repo\" | grep -q \"/\"; then\n            repo=\"$git_owner/$repo\"\n        fi\n        echo \"========================================\"\n        timestamp \"Updating $repo\"\n        echo \"========================================\"\n        if ! [ -d \"$repo_dir\" ]; then\n            git clone \"$git_url/$repo\" \"$repo_dir\"\n        fi\n        pushd \"$repo_dir\"\n        \"$srcdir/git_submodules_update.sh\"\n        # make update does git pull but if that mechanism is broken then this first git pull will allow the repo to self-fix itself\n        #git pull --no-edit\n        #git submodule update --init --remote --recursive --force ||\n        #for submodule in $(git submodule | awk '{print $2}'); do\n        #    if [ -d \"$submodule\" ] && ! [ -L \"$submodule\" ]; then\n        #        pushd \"$submodule\" || continue\n        #        git stash\n        #        default_branch=\"$(default_branch)\"\n        #        git checkout \"$default_branch\"\n        #        git pull --no-edit ||\n        #        git reset --hard origin/\"$default_branch\"\n        #        git submodule update\n        #        popd\n        #    fi\n        #    echo\n        #done\n        #for submodule in $(git submodule | awk '{print $2}'); do\n        #    echo \"committing latest hashref for submodule $submodule\"\n        #    git ci -m \"updated submodule $submodule\" \"$submodule\" || :\n        #    echo\n        #done\n        echo\n        if [ -z \"${NO_PUSH:-}\" ]; then\n            if [ -n \"${REVIEW_PUSH:-}\" ]; then\n                \"$srcdir/git_review_push.sh\"\n            else\n                git push\n            fi\n        fi\n        popd\n        echo\n    done\n}\n\nrun \"$repolist\"\n"
  },
  {
    "path": "git/git_summary_line.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-31 19:29:38 +0100 (Tue, 31 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Prints concise git log for $PWD repo - used by all repos headers to signify their release in CI logs\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# setting TERM and --no-pager are attempted workarounds to breakage in CircleCI, hangs with 'WARNING: terminal is not fully functional' (press RETURN)\nexport TERM=xterm\ngit --no-pager log -n 1 --pretty=format:'>>> %H  %ai  (%an)  %s%n'\n"
  },
  {
    "path": "git/git_sync_repos_upstream.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-13 15:36:35 +0000 (Thu, 13 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSyncs all adjacent repos from setup/repos.txt to one of the upstreams GitHub / GitLab / BitBucket / Azure DevOps\n\nAnother trick would be to set the remote origin to contain all 3 URLs so each push goes to all 3 repos every time\n\neg.\n\n    git remote set-url --add origin <url>\n\n    git remote set-url --add origin https://bitbucket.org/HariSekhon/DevOps-Bash-tools\n\nSee git_remotes_set_multi_origin.sh for an auto-inferred implementation of this\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"github|gitlab|bitbucket|azure\"\n\nname=\"${1:-}\"\n\nif [ -z \"$name\" ]; then\n    usage\nfi\n\nif [ \"$name\" = \"github\" ]; then\n    domain=github.com\nelif [ \"$name\" = \"gitlab\" ]; then\n    domain=gitlab.com\nelif [ \"$name\" = \"bitbucket\" ]; then\n    domain=bitbucket.org\nelif [ \"$name\" = \"azure\" ]; then\n    domain=dev.azure.com\nelse\n    usage\nfi\n\n#sed 's/#.*//; s/:/ /; /^[[:space:]]*$/d' \"$srcdir/../setup/repos.txt\" |\necho \"DevOps-Golang-tools go-tools\" |\nwhile read -r repo dir; do\n    if [ -z \"$dir\" ]; then\n        dir=\"$repo\"\n    fi\n    dir=\"../$dir\"\n    if ! [ -d \"$dir\" ]; then\n        echo \"WARNING: repo dir $dir not found, skipping...\"\n        continue\n    fi\n    echo\n    echo \"Sync'ing repo $repo to $name\"\n    name=\"${name%.*}\"\n    pushd \"$dir\" &>/dev/null\n    if ! git remote | grep -q \"$name\"; then\n        echo \"$name remote not configured, configuring...\"\n        set +o pipefail\n        url=\"$(git remote -v | awk '{print $2}' | grep -Ei \"$domain\" | head -n 1)\"\n        if [ -n \"$url\" ]; then\n            echo \"copied existing remote url for $name as is including any access tokens to named remote $name\"\n        else\n            url=\"$(git remote -v | awk '{print $2}' | grep -Ei 'bitbucket.org|github.com|gitlab.com|dev.azure.com' | head -n 1 | perl -pe \"s/^((\\\\w+:\\\\/\\\\/)?(git@)?)[^\\\\/:]+/\\$1$domain/\")\"\n            # XXX: Azure DevOps has non-uniform URLs compared to the 3 main Git repos\n            if [ \"$name\" = \"azure\" ]; then\n                url=\"$(git_to_azure_url \"$url\")\"\n            else\n                # undo weird Azure DevOps url components if we happen to infer URL from an Azure DevOps url\n                url=\"$(azure_to_git_url \"$url\")\"\n            fi\n            echo \"inferring $name URL to be $url\"\n            # don't put this below and it'd print your access token to screen from existing remote\n            echo \"adding remote $name with url $url\"\n        fi\n        set -o pipefail\n        git remote add \"$name\" \"$url\"\n    fi\n    echo \"pulling from $name to merge if necessary\"\n    git pull --no-edit \"$name\" master\n    echo \"pushing to $name remote\"\n    git push \"$name\" master\n    popd &>/dev/null\ndone\n"
  },
  {
    "path": "git/git_tag_release.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-12 14:18:52 +0100 (Tue, 12 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Git Tag for the current repo checkout, auto-incrementing the default vYYYY.NN release if one isn't given\n\nThe first argument is an optional version, which is recommended to set to vN.N.N eg. v1.0.0 as per semantic versioning standards\n\nIf the first argument is 'day' or 'date', will determine the next available release in the format vYYYYMMDD.NN where NN is incremented from 1\nIf the first argument is 'month', will determine the next available release in the format vYYYYMM.NN\nIf the first argument is 'year', will determine the next available release in the format vYYYY.NN\n\nIf no argument is given, defaults to generating a 'year' version in the format vYYYY.NN\n\nThese formats don't have dashes in them like ISO dates so that if you move from YYYY to YYYYMM format or YYYYMMDD format, software will recognize the newer format as the highest version number ie. the latest version\n\n\nRequires the Git command to be installed\n\nDon't forget to 'git push --tags' to send the tag upstream\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version> <commit>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-year}\"\ncommit=\"${2:-HEAD}\"\nshift || :\nshift || :\n\ngenerate_version=0\nprefix='v'\nif [ -n \"${NO_GIT_RELEASE_PREFIX:-}\" ]; then\n    prefix=''\nfi\n\nif [ \"$version\" = year ]; then\n    version=\"${prefix}$(date '+%Y')\"\n    generate_version=1\nelif [ \"$version\" = month ]; then\n    version=\"${prefix}$(date '+%Y%m')\"\n    generate_version=1\nelif [ \"$version\" = day ] || [ \"$version\" = date ]; then\n    version=\"${prefix}$(date '+%Y%m%d')\"\n    generate_version=1\nfi\n\nif [ \"$generate_version\" = 1 ]; then\n    existing_tags=\"$(git tags)\"\n\n    number=\"$(grep -Eo \"^$version\"'\\.\\d+' <<< \"$existing_tags\" | head -n 1 | sed \"s/^$version\\\\.//\" || echo 1)\"\n\n    # increment the number\n    while grep -Fxq \"$version.$number\" <<< \"$existing_tags\"; do\n        ((number+=1))\n        if [ $number -gt 9999 ]; then\n            die \"FAILED to find unused tag version in format '$version.NN'\"\n        fi\n    done\n\n    version+=\".$number\"\nfi\n\ntimestamp \"Generating tag '$version' on commit '$commit'\"\ngit tag \"$version\" \"$commit\"\ntimestamp \"Generated release tag '$version'\"\n"
  },
  {
    "path": "git/gitguardian_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /health | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-14 11:35:49 +0100 (Wed, 14 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the GitGuardian.com API\n\nAutomatically handles authentication via environment variable \\$GITGUARDIAN_TOKEN\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here - where nnnnnn is your personal account number seen in your dashboard URLs\n\n    https://dashboard.gitguardian.com/workspace/nnnnnn/api/personal-access-tokens\n\n\nAPI Reference:\n\n    https://api.gitguardian.com/docs\n\n\nExamples:\n\n\n    # Get health (check API key is valid):\n\n        ${0##*/} /health | jq .\n\n    # Lists Secret Incidents:\n\n        ${0##*/} /incidents/secrets | jq .\n\n    # Get Specific Secret Incident:\n\n        ${0##*/} /incidents/secrets/{incident_id} | jq .\n\n    # List Secret Occurrences:\n\n        ${0##*/} /occurrences/secrets | jq .\n\n    # List Sources - repos and their open incidence counts, last scanned date, health (safe/at_risk), visibility (public/private), type (eg. github):\n\n        ${0##*/} /sources | jq .\n\n    # List Members:\n\n        ${0##*/} /members | jq .\n\n    # List Invitations:\n\n        ${0##*/} /invitations | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.gitguardian.com/v1\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path//https:\\/\\/api.gitguardian.com/v1}\"\nurl_path=\"${url_path##/}\"\n\ncheck_env_defined GITGUARDIAN_TOKEN\n\nexport TOKEN=\"$GITGUARDIAN_TOKEN\"\n\nexport CURL_AUTH_HEADER=\"Authorization: Token\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "git/gitignore.io_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Wed Sep 18 14:34:59 2019 +0100\n#        (forked from .bash.d/git.sh)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads the latest gitignore.io ignore lists to standard output via gitignore.io's API\n\nUseful as a large base to populate your .gitignore, and then sprinkle a few customizations\n\nSee the massive .gitignore in this repo for an example of this being used\n\n\nArgs should be a comma / space separate list of languages and frameworks for which to retrieve gitignore config for\n\n\nExamples:\n\n\n# List all available languages, configs and frameworks:\n\n    ${0##*/} list\n\n\n# Get the .gitignore config for the C, Python and Perl programming languages\n\n    ${0##*/} c,python,perl\n\n\n# Get the .gitignore for a whole bunch of programming languages you use, plus Maven, SBT, Gradle, Ansible and Homebrew ...\n\n    ${0##*/} c python perl ruby java scala go maven sbt gradle ansible\n\n\n# List all available languages, configs and frameworks that aren't in the list passed in from standard input. eg. find only new and missing options to add to your configs\n\n    ${0##*/} missing <<< 'your_existing_list_comma_or_space_or_newline_separated'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<langs>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ngitignore_api(){\n    local url;\n    local langs;\n    local options=();\n    local args=();\n    local commas_to_newlines=\"cat\";\n    for arg in \"$@\"; do\n        if [ \"$arg\" = -- ]; then\n            options+=(\"$arg\");\n        else\n            args+=(\"$arg\");\n        fi;\n    done;\n    langs=\"$(IFS=, ; echo \"${args[*]}\")\";\n    url=\"https://www.gitignore.io/api/$langs\";\n    if [ \"$langs\" = \"list\" ]; then\n        commas_to_newlines=\"tr ',' '\\\\n'\";\n    fi;\n    {\n        if hash curl 2> /dev/null; then\n            curl -sSL \"${options[@]}\" \"$url\";\n        else\n            if hash wget 2> /dev/null; then\n                wget -O - \"${options[*]}\" \"$url\";\n            fi;\n        fi\n    } | eval \"$commas_to_newlines\";\n    echo\n}\n\nif [ \"$*\" = \"missing\" ] || [ \"$*\" = \"new\" ]; then\n    if is_mac; then\n        # Mac's grep is buggy and fails to exclude things it should from file, randomly missing some without any special characters or whitespace or anything else to explain it, but GNU grep works correctly with grep -v -f\n        grep(){\n            command ggrep \"$@\"\n        }\n    fi\n    my_list=\"$(cat)\"\n    gitignore_api list |\n    grep -Fvx -f <(tr ',' ' ' <<< \"${my_list%%#*}\" | tr '[:space:]' '\\n' | sed 's/[[:space:]]*//g; /^[[:space:]]*$/d')\nelse\n    gitignore_api \"$@\"\nfi\n"
  },
  {
    "path": "git/precommit_run_changed_files.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-16 22:08:31 +0200 (Fri, 16 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns pre-commit on all files changed on the current branch vs the default branch\n\nUseful to reproduce pre-commit checks that are failing in pull requests to get your PRs to pass\n\nWARNING: \\$HOME/.mdlrc appears to be cumulative and will hide errors in git checkouts with local .mdlrc\n         you will only realize this when your Pull Request checks fail on GitHub, so you may need to edit\n         or delete your \\$HOME/.mdlrc to stop it masking your local pre-commit testing\n\nRequires git and pre-commit to be installed and must be run on the feature branch in the git repo checkout\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<pre-commit args>]\"\n\nhelp_usage \"$@\"\n\n#num_args 0 \"$@\"\n\ngit_root=\"$(git_root)\"\n\necho \"cd $git_root\"\ncd \"$git_root\"\n\ndefault_branch=\"$(git symbolic-ref refs/remotes/origin/HEAD | sed 's|.*/||')\"\n\nchanged_files=()\nwhile IFS= read -r filename; do\n    changed_files+=(\"$filename\")\ndone < <(git diff --name-only \"$default_branch\"..)\n\nopts=(--show-diff-on-failure --color=auto --files)\n\necho\necho \"pre-commit run ${opts[*]} ${changed_files[*]} $*\"\necho\npre-commit run \"${opts[@]}\" \"${changed_files[@]}\" \"$@\"\n"
  },
  {
    "path": "git/update_gitignore.io.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-24\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Updated gitignore based on existing gitignore.io API in the file\n#\n# gitignore.io\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nheader=\"Created by https://www.gitignore.io/api/\"\n\nset +eo pipefail\nurl=\"$(grep \"$header\" .gitignore)\"\nset -eo pipefail\n\nif [ -z \"$url\" ]; then\n    echo\n    echo \"No gitignore.io API url found in .gitignore\"\n    echo\n    echo \"You need to add the gitignore.io content to .gitignore initially to choose your selections, this script merely runs an update based on the API url recorded in the comments\"\n    exit 1\nfi\n\nurl=\"$(head -n1 <<< \"${url/*https:/https:}\")\"\n\nsed_regex=\"${header//\\//\\\\/}\"\nsed -i.bak \"/$sed_regex/,\\$d\" .gitignore\n\ncurl -sS \"$url\" |\nsed 's/[[:space:]]*$//' |\nsed -n \"/$sed_regex/,\\$p\" >> .gitignore\n"
  },
  {
    "path": "github/github_actions_aws_create_load_credential.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-11 11:47:26 +0000 (Fri, 11 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates an AWS user, generates and downloads an access key and uploads it to the given GitHub repo\n\nAWS Access Keys are stored/staged in ~/.aws/keys/ - so re-running this from a new account/machine will have no choice but to recreate the access keys and upload the new key as the secret key is only available once at creation time\n\nRequires AWS CLI and GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> <group_or_policy_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nowner_repo=\"$1\"\ngroup_or_policy=\"$2\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"Invalid GitHub owner/repo given: $owner_repo\"\nfi\n\n#owner=\"${owner_repo%/*}\"\nrepo=\"${owner_repo##*/}\"\n\n#if ! gh repo list \"$owner\" --json name -q \".[] | select(.name == \\\"$repo\\\") | .name\" | grep -Fxq \"$repo\"; then\nif ! gh repo view \"$owner_repo\" >/dev/null; then\n    die \"GitHub repo '$owner_repo' was not found!\"\nfi\n\n#aws_account_id=\"$(aws sts get-caller-identity --query Account --output text)\"\naws_account_id=\"$(aws_account_id)\"\n\nkeyfile=~/.aws/keys/\"${repo}_${aws_account_id}_accessKeys.csv\"\n\nuser=\"github-actions-$repo\"\n\n\"$srcdir/../aws/aws_cli_create_credential.sh\" \"$user\" \"$group_or_policy\" \"$keyfile\"\n\"$srcdir/../aws/aws_csv_creds.sh\" \"$keyfile\" |\n\"$srcdir/github_actions_repo_set_secret.sh\" \"$owner_repo\"\n"
  },
  {
    "path": "github/github_actions_delete_offline_runners.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-05 12:44:44 +0000 (Fri, 05 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/actions#delete-a-self-hosted-runner-group-from-an-organization\n#\n# https://docs.github.com/en/rest/reference/actions#delete-a-self-hosted-runner-from-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes offline GitHub Actions self-hosted runners for the given Repo or Organization via the GitHub API\n\nUses github_actions_runners.sh to find offline runners\n\nSee Also:\n\n    github_actions_runner.sh - generates a token and launches a runner for a GitHub Organization or Repo\n\n    https://github.com/HariSekhon/Kubernetes-configs - for running GitHub Actions Runners in Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo_or_organization>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_or_org=\"$1\"\nshift\n\n\"$srcdir/github_actions_runners.sh\" \"$repo_or_org\" |\nawk '$2~/offline/{print $1}' |\nwhile read -r id; do\n    if [[ \"$repo_or_org\" =~ / ]]; then\n        prefix=\"repos\"\n    else # is an org\n        prefix=\"orgs\"\n    fi\n    timestamp \"deleting offline runner with id '$id'\"\n    \"$srcdir/github_api.sh\" \"/$prefix/$repo_or_org/actions/runners/$id\" -X DELETE\ndone\n"
  },
  {
    "path": "github/github_actions_foreach_workflow.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: devops-bash-tools echo user={user} repo={repo} name={name} workflow_name={workflow} id={id}\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-27 11:21:14 +0000 (Sat, 27 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each GitHub Actions workflow in the given repo\n\nAll arguments after the repo become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the workflow id/names and exit after the first iteration\n\n\\$GITHUB_ORGANIZATION / \\$GITHUB_USER - the user or organization to iterate the repos on - if not specified then determines the currently authenticated github user from your \\$GITHUB_TOKEN\n\nThe command template replaces the following for convenience in each iteration:\n\n{organization}, {org} => the organization account being iterated\n{username}, {user}    => the user account being iterated\n{repo}                => the repo name with the user prefix\n{workflow}, {name}    => the workflow name\n{id}, {workflow_id}   => the workflow id\n\neg.\n    ${0##*/} devops-bash-tools echo user={user} repo={repo} name={name} workflow_name={workflow} id={id}\n    ${0##*/} devops-bash-tools echo org={org}   repo={repo} name={name} workflow_name={workflow} id={id}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> <command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nrepo=\"$1\"\nshift || :\n\n\"$srcdir/github_actions_workflows.sh\" \"$repo\" |\njq -r '.workflows[] | [.id, .name] | @tsv' |\nwhile read -r id workflow; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $workflow\" >&2\n    echo \"# ============================================================================ #\" >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{username\\}/${user:-}}\")\n    cmd=(\"${cmd[@]//\\{user\\}/${user:-}}\")\n    cmd=(\"${cmd[@]//\\{organization\\}/${GITHUB_ORGANIZATION:-}}\")\n    cmd=(\"${cmd[@]//\\{org\\}/${GITHUB_ORGANIZATION:-}}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$workflow}\")\n    cmd=(\"${cmd[@]//\\{workflow\\}/$workflow}\")\n    cmd=(\"${cmd[@]//\\{workflow_id\\}/$id}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$id}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "github/github_actions_in_use.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:53:11 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all GitHub Actions directly referenced in the .github/workflows in the current local repo checkout\n\nThis is useful to combine with\n\n    github_actions_repo_actions_allow.sh\n        and\n    github_actions_repos_lockdown.sh\n\nAdd these actions you need to ~/.github/workflows/allowed-actions.txt and then run github_actions_repos_lockdown.sh\n\nCaveat: does not detect GitHub Actions referenced in reusable workflows - for that instead use github_actions_in_use_repo.sh which follows GitHub Reusable Workflows references in GitHub via the API\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ngit_root=\"$(git_root)\"\n\n# filtering out anything with .github in it as that will be a .github/workflows/file.yaml, not an action\ngrep -ERho '^[^#]+[[:space:]]uses:.+@[^#[:space:]]+' \"$git_root/.github/workflows/\" |\nsed '\n    s/^[^#]*[[:space:]]uses:[[:space:]]*//;\n    s/#.*$//;\n    s/[[:space:]]*$//;\n    /\\.github/d\n' |\nsort -fu\n"
  },
  {
    "path": "github/github_actions_in_use_across_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:58:49 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all GitHub Actions in use from the .github/workflows directory across all workflows in all original source repos for the user or organization\n\nThis is useful to combine with github_actions_repo_actions_allow.sh and github_actions_repos_lockdown.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser_or_org=\"${1:-${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}}\"\n\nget_github_repos \"$user_or_org\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    \"$srcdir/github_actions_in_use_repo.sh\" \"$user_or_org/$repo\"\n    echo >&2\ndone # |\n#sort -u\n"
  },
  {
    "path": "github/github_actions_in_use_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:58:49 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all GitHub Actions in use in the given repo under .github/workflows/ in the default branch using the GitHub API\n\nThis is useful to combine with github_actions_repo_actions_allow.sh and github_actions_repos_lockdown.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo=\"$1\"  # in owner/repo format\n\nif ! is_github_owner_repo \"$repo\"; then\n    usage \"invalid repo format specified, expected owner/repo format\"\nfi\n\nprocess_contents(){\n    local data\n    data=\"$(cat)\"\n    if [ \"$(jq -r '.type' <<< \"$data\")\" != file ]; then\n        return 0\n    fi\n    jq -r '.content' <<< \"$data\" |\n    base64 --decode\n}\n\ngrep_github_actions(){\n    # filtering out anything with .github in it as that will be a .github/workflows/file.yaml, not an action\n    { grep -Eo '^[^#]+[[:space:]]uses:.+@[^#[:space:]]+' || : ; } |\n    sed '\n        s/^[^#]*[[:space:]]uses:[[:space:]]*//;\n        s/#.*$//;\n        s/[[:space:]]*$//;\n        /\\.github/d;\n    ' |\n    sort -fu\n}\n\ntimestamp \"Checking repo '$repo'\"\n# if this gets a 404 then skip there is no such dir and no workflows\nif \"$srcdir/github_api.sh\" \"/repos/$repo/contents/.github/workflows\" &>/dev/null; then\n    timestamp \"GitHub workflows dir found, fetching file list\"\n    \"$srcdir/github_api.sh\" \"/repos/$repo/contents/.github/workflows\" |\n    #jq -r '.[] | select(.type == \"file\") | .download_url' |  # this would probably only work for public repos, we need to use the API address to reuse github_api.sh\n    jq -r '.[].name' |\n    while read -r filename; do\n        [[ \"$filename\" =~ \\.ya?ml$ ]] || continue\n        #timestamp \"Checking '$repo' => ${download_url##*/}\"\n        timestamp \"Checking '$repo' => $filename\"\n\n        # Find directly used GitHub Actions\n        data=\"$(\"$srcdir/github_api.sh\" \"/repos/$repo/contents/.github/workflows/$filename\")\"\n        contents=\"$(process_contents <<< \"$data\")\"\n        grep_github_actions <<< \"$contents\"\n\n        # Find GitHub Actions in remote workflow files\n        { grep -Eo '^[^#]+[[:space:]]uses:.+/.github/workflows/.+@[^#[:space:]]+' || : ; } <<< \"$contents\" |\n        sed 's/^[^#]*[[:space:]]uses:[[:space:]]*//; s/#.*$//;' |\n        while read -r reusable_workflow; do\n            timestamp \"Checking reusable workflow: $reusable_workflow\"\n            reusable_workflow=\"${reusable_workflow/.github/contents/.github}\"\n            reusable_workflow=\"${reusable_workflow/@/?ref=}\"\n            \"$srcdir/github_api.sh\" \"/repos/$reusable_workflow\" |\n            process_contents |\n            grep_github_actions\n        done\n    done\nelse\n    timestamp \"No GitHub workflows found in repo '$repo'\"\nfi\n"
  },
  {
    "path": "github/github_actions_latest_log.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-21 23:16:42 +0800 (Fri, 21 Mar 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOutputs the text log of the latest GitHub Actions workflow run to the terminal\n\nUseful when the logs are too big for the UI and you have to open it in another tab which is very slow in browser\n\nUses adjacent script:\n\n    github_actions_log.sh\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nowner_repo=\"${1:-}\"\n\nif [ -n \"$owner_repo\" ]; then\n    is_github_owner_repo \"$owner_repo\" || die \"Invalid GitHub owner/repo given: $owner_repo\"\nelse\n    owner_repo=\"$(github_owner_repo)\"\nfi\n\n\"$srcdir/github_actions_log.sh\" \"$owner_repo\" 1\n"
  },
  {
    "path": "github/github_actions_log.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-21 23:16:42 +0800 (Fri, 21 Mar 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\ndefault_num_workflow_runs=10\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOutputs the text log for a given GitHub Actions workflow run to the terminal\n\nFetches the last $default_num_workflow_runs runs and drops you into an interactive menu to hit enter on the one you want\n\nUseful when the logs are too big for the UI and you have to open it in another tab which is very slow in browser\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo> <num_workflow_runs>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nowner_repo=\"${1:-}\"\nnum_workflow_runs=\"${2:-$default_num_workflow_runs}\"\n\nargs=()\nif [ -n \"$owner_repo\" ]; then\n    is_github_owner_repo \"$owner_repo\" || die \"Invalid GitHub owner/repo given: $owner_repo\"\n    args+=(-R \"$owner_repo\")\nfi\n\nif ! is_int \"$num_workflow_runs\"; then\n    usage \"Second arg for number of workflow runs to fetch has to be an integer, got: $num_workflow_runs\"\nfi\nif [ \"$num_workflow_runs\" -lt 1 ]; then\n    usage \"Second arg for number of workflow runs to fetch cannot be less than 1, got: $num_workflow_runs\"\nfi\n\nif ! type -P dialog &>/dev/null; then\n    timestamp \"Diaglog not found in \\$PATH, attempting to install via OS package manager\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" dialog\n    echo\nfi\n\ntimestamp \"Fetching last $num_workflow_runs workflow runs\"\nworkflow_runs=\"$(gh run list -L \"$num_workflow_runs\" ${args:+\"${args[@]}\"})\"\n\n# header is not printed in a pipe\n#header=\"$(head -n1 <<< \"$workflow_runs\")\"\n#workflow_runs=\"$(tail -n +2 <<< \"$workflow_runs\")\"\n\ntimestamp \"Generating menu items\"\nwhile read -r line; do\n\n    # used for counting and string conversion if only a single item\n\n    menu_items+=(\"$line\")\n\n    # passed to dialog because it requires args: tag1 visibletext tag2 visibletext\n    # - by making the second one blank it uses the item as both the tag to be returned\n    # to script as well as the visible text\n\n    menu_tag_items+=(\"$line\" \" \")\n\ndone <<< \"$workflow_runs\"\n\nif [ \"${#menu_items[@]}\" -eq 0 ];then\n    die 'No workflow runs found!'\nelif [ \"${#menu_items[@]}\" -eq 1 ];then\n    workflow_run=\"${menu_items[*]}\"\nelse\n    workflow_run=\"$(dialog --menu \"Choose which workflow run to fetch the logs for\" \"$LINES\" \"$COLUMNS\" \"$LINES\" \"${menu_tag_items[@]}\" 3>&1 1>&2 2>&3)\"\nfi\n\ntimestamp \"Parsing ID for selected workflow run: $workflow_run\"\nworkflow_id=\"$(awk '{for(i=1;i<=NF;i++) if($i ~ /^[0-9]{11}$/) print $i}' <<< \"$workflow_run\")\"\n\nif is_blank \"$workflow_id\"; then\n    die \"Failed to parse the workflow ID from the selected item\"\nfi\n\ntimestamp \"Fetching logs for workflow ID '$workflow_id'\"\ngh run view \"$workflow_id\" ${args:+\"${args[@]}\"} --log\n"
  },
  {
    "path": "github/github_actions_repo_actions_allow.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:00:10 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\nALLOW_FILE=\"$srcdir/.github/workflows/actions-allowed.txt\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAllows select 3rd party GitHub Actions in the given repo using the GitHub API\n\nIf you have an Organization, I recommend you set this organization-wide instead, but for individual users this is handy to automate tightenting up your security\n\nThe list of actions is taken from the adjacent file '${ALLOW_FILE#$srcdir/}'\n\nSee Also:\n\n    github_actions_repos_lockdown.sh - applies this to all repos\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<user>/<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\nrepo=\"$(perl -pne 's|^https://github.com/||i' <<< \"$repo\")\"\nrepo=\"${repo##/}\"\n\nactions_allowed=\"$(sed 's/#.*//;\n                        s/^[[:space:]]*//;\n                        s/[[:space:]]*$//;\n                        /^[[:space:]]*$/d;' \\\n                        \"$ALLOW_FILE\")\"\n\nif [ -z \"$actions_allowed\" ]; then\n    die \"No 3rd party actions found in the allow file '$ALLOW_FILE'\"\nfi\n\ntimestamp \"Enabling the following 3rd party GitHub Actions on repo '$repo':\"\necho\necho \"$actions_allowed\"\necho\n\n# convert to array of strings for the API\n#actions_allowed_json_array=\"$(printf \"[%s]\" \"$(while read -r action; do echo \"\\\"$action\\\", \"; done <<< \"$actions_allowed\" | sed 's/, //')\")\"\n\nactions_allowed_json_array=\"[\"\nwhile read -r action; do\n    actions_allowed_json_array+=\"\\\"$action\\\", \"\ndone <<< \"$actions_allowed\"\nactions_allowed_json_array=\"${actions_allowed_json_array%, }\"\nactions_allowed_json_array+=\"]\"\n\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/permissions/selected-actions\" -X PUT -d \"{\\\"patterns_allowed\\\": $actions_allowed_json_array}\"  # don't quote array, it contains the quotes already\n"
  },
  {
    "path": "github/github_actions_repo_env_set_secret.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon/devops-bash-tools testenv haritest=stuff haritest2=stuff2\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/ee/api/project_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates GitHub repo environment-level secrets from args or stdin, for use in GitHub Actions\n\nIf 'owner/repo' isn't specified as the first argument, attempts to infer from the local git repo\nThe next argument must be the environment name\nIf no subsequent key=value pair arguments are given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nExamples:\n\n    ${0##*/} HariSekhon/DevOps-Bash-tools testenv AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} HariSekhon/DevOps-Bash-tools testenv\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | gh secret set -R HariSekhon/DevOps-Bash-tools --env testenv -f -\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} HariSekhon/DevOps-Bash-tools testenv\n\n        aws_csv_creds.sh credentials_exported.csv | gh secret set -R HariSekhon/DevOps-Bash-tools --env testenv -f -\n\n\nRequires the GitHub CLI 'gh' to be installed and available in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner_repo_or_id>] <environment_name> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\n# requires the GitHub CLI\ncheck_bin gh\n\nmin_args 1 \"$@\"\n\nowner_repo=''\n\nif [ $# -eq 1 ]; then\n    environment_name=\"$1\"\nelif [ $# -gt 1 ] && [[ \"$2\" =~ = ]]; then\n    environment_name=\"$1\"\nelse\n    owner_repo=\"$1\"\n    environment_name=\"$2\"\n    shift || :\nfi\nshift || :\n\nowner_repo=\"${owner_repo:-$(get_github_repo)}\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    usage \"owner_repo given '$owner_repo' does not conform <user_or_org>/<repo> format\"\nfi\n\n# don't need to check for existing secrets as the API is a set (add/update) operation anyway\n#existing_secrets=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner_repo/tonkasim/actions/secrets\" | jq -r '.secrets[].name')\"\n\n# needed to create an encrypted value against the repo's public key before upload\n#repo_public_key_data=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner_repo/actions/secrets/public-key\")\"\n#repo_public_key=\"$(jq -r '.key' <<< \"$repo_public_key_data\")\"\n#repo_public_key_id=\"$(jq -r '.key_id' <<< \"$repo_public_key_data\")\"\n\nadd_secret(){\n    local key_value=\"$1\"\n    parse_export_key_value \"$key_value\"\n    # shellcheck disable=SC2154\n    timestamp \"setting GitHub secret '$key' in repo '$owner_repo' environment '$environment_name'\"\n    # XXX: no way to generate this encrypted value in Bash at this time, all language bindings but no Libsodium CLI so use GitHub CLI instead\n    # https://docs.github.com/en/rest/reference/actions#create-or-update-an-environment-secret\n    #\n    # XXX: there is some kind of bug in GitHub CLI - the secret doesn't work when fed through stdin, only through --body\n    #      https://github.com/cli/cli/issues/5031\n    #command gh secret set -R \"$owner_repo\" --env \"$environment_name\" \"$key\" <<< \"$value\"\n    # shellcheck disable=SC2154\n    # $value is defined in parse_export_key_value()\n    command gh secret set -R \"$owner_repo\" --env \"$environment_name\" \"$key\" --body \"$value\"\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_secret \"$arg\"\n    done\nelse\n    while read -r line; do\n        [ -n \"$line\" ] || continue\n        add_secret \"$line\"\n    done\nfi\n"
  },
  {
    "path": "github/github_actions_repo_restrict_actions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 18:44:00 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRestricts the allowed GitHub Actions in the given repo to only those created by GitHub and verified partner companies (eg. AWS, Docker) using the GitHub API\n\nIf you have an Organization, I recommend you set this organization-wide instead, but for individual users this is handy to automate tightenting up your security\n\nSee Also:\n\n    github_actions_repo_actions_allow.sh - applies this to all repos\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<user>/<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\nrepo=\"$(perl -pne 's|^https://github.com/||i' <<< \"$repo\")\"\nrepo=\"${repo##/}\"\n\ntimestamp \"Restricting GitHub Actions to selected actions on repo '$repo'\"\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/permissions\" -X PUT -d '{\"enabled\":true, \"allowed_actions\": \"selected\"}'\n\ntimestamp \"Enabling GitHub and Verified Partners actions on repo '$repo'\"\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/permissions/selected-actions\" -X PUT -d '{\"github_owned_allowed\":true, \"verified_allowed\": true}'\n"
  },
  {
    "path": "github/github_actions_repo_secrets_overriding_org.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-14 17:16:30 +0000 (Mon, 14 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds any GitHub Actions secrets in the current or given repo that are overriding Organization-level secrets\n\nUseful to audit across repos when combined with github_foreach_repo.sh\n\nRequires GitHub CLI to be installed and configured with a token with permissions to the organization as well as the repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nowner_repo=\"${1:-}\"\n# set the default outside of the ${1:-} to avoid issues with braces and escaping them becoming literal slashes\nif is_blank \"$owner_repo\"; then\n    owner_repo='{owner}/{repo}'\nfi\n\norg=\"$(gh api \"/repos/$owner_repo\" -q '.organization.login')\"\n\nif [ -z \"$org\" ]; then\n    die \"Repo is not owned by an organization\"\nfi\n\norg_secrets=\"$(gh api \"/orgs/$org/actions/secrets\" | jq -r '.secrets[].name')\"\n\nif [ -z \"$org_secrets\" ]; then\n    warn \"No organization secrets found\"\n    exit 0\nfi\n\nrepo_secrets=\"$(gh api \"/repos/$owner_repo/actions/secrets\" | jq -r '.secrets[].name')\"\n\nwhile read -r repo_secret; do\n    [ -z \"$repo_secret\" ] && continue\n    if grep -Fxq \"$repo_secret\" <<< \"$org_secrets\"; then\n        echo \"$repo_secret\"\n    fi\ndone <<< \"$repo_secrets\"\n"
  },
  {
    "path": "github/github_actions_repo_set_secret.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon/devops-bash-tools haritest=stuff haritest2=stuff2\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/ee/api/project_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates GitHub repo-level secrets from key=value args or stdin, for use in GitHub Actions\n\nIf 'owner/repo' isn't specified as the first argument, attempts to infer from the local git repo\nIf no subsequent key=value pair arguments are given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nExamples:\n\n    ${0##*/} HariSekhon/DevOps-Bash-tools AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} HariSekhon/DevOps-Bash-tools\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | gh secret set -R HariSekhon/DevOps-Bash-tools -f -\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} HariSekhon/DevOps-Bash-tools\n\n        aws_csv_creds.sh credentials_exported.csv | gh secret set -R HariSekhon/DevOps-Bash-tools -f -\n\n\nRequires the GitHub CLI 'gh' to be installed and available in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner_repo_or_id> <key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\n# requires the GitHub CLI\ncheck_bin gh\n\n#min_args 1 \"$@\"\n\nif [ -n \"${1:-}\" ] && ! [[ \"$1\" =~ = ]]; then\n    owner_repo=\"$1\"\n    shift || :\nfi\nowner_repo=\"${owner_repo:-$(get_github_repo)}\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    usage \"owner_repo '$owner_repo' does not conform <user_or_org>/<repo> format\"\nfi\n\n# don't need to check for existing secrets as the API is a set (add/update) operation anyway\n#existing_secrets=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner_repo/tonkasim/actions/secrets\" | jq -r '.secrets[].name')\"\n\n# needed to create an encrypted value against the repo's public key before upload\n#repo_public_key_data=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner_repo/actions/secrets/public-key\")\"\n#repo_public_key=\"$(jq -r '.key' <<< \"$repo_public_key_data\")\"\n#repo_public_key_id=\"$(jq -r '.key_id' <<< \"$repo_public_key_data\")\"\n\nadd_secret(){\n    local key_value=\"$1\"\n    parse_export_key_value \"$key_value\"\n    # shellcheck disable=SC2154\n    timestamp \"setting GitHub secret '$key' in repo '$owner_repo'\"\n    # https://docs.github.com/en/rest/reference/actions#create-or-update-a-repository-secret--code-samples\n    # won't work, gets error:\n    # {\n    #   \"message\": \"Invalid request.\\n\\n\\\"key_id\\\" wasn't supplied.\",\n    #   \"documentation_url\": \"https://docs.github.com/rest/reference/actions#create-or-update-a-repository-secret\"\n    # }\n    #\"$srcdir/github_api.sh\" \"/repos/$owner_repo/actions/secrets/$key\" \\\n    #    -X PUT \\\n    #    -H 'Accept: application/vnd.github.v3+json' \\\n    #    -d \"{\\\"encrypted_value\\\":\\\"$value\\\"}\"\n    #    XXX: no way to generate this encrypted value in Bash at this time, all language bindings but no Libsodium CLI so use GitHub CLI instead\n    #\n    # XXX: there is some kind of bug in GitHub CLI - the secret doesn't work when fed through stdin, only through --body\n    #      https://github.com/cli/cli/issues/5031\n    #command gh secret set -R \"$owner_repo\" \"$key\" <<< \"$value\"\n    # shellcheck disable=SC2154\n    command gh secret set -R \"$owner_repo\" \"$key\" --body \"$value\"\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_secret \"$arg\"\n    done\nelse\n    while read -r line; do\n        [ -n \"$line\" ] || continue\n        add_secret \"$line\"\n    done\nfi\n"
  },
  {
    "path": "github/github_actions_repos_lockdown.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-26 19:22:46 +0000 (Wed, 26 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSecures the allowed GitHub Actions across all repos for the user\n\nUses:\n        github_actions_repo_restrict_actions.sh\n        github_actions_repo_actions_allow.sh\n\nIf you have an Organization, I recommend you set this organization-wide instead, but for individual users this is handy to automate tightenting up your security\n\nTODO: restrict token permissions to default to minimal content read only, but the GitHub API does not support managing this at time of writing\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser=\"$(get_github_user)\"\n\nget_github_repos \"$user\" |\nwhile read -r repo; do\n    github_actions_repo_restrict_actions.sh \"$user/$repo\"\n    github_actions_repo_actions_allow.sh \"$user/$repo\"\n    echo\ndone\n"
  },
  {
    "path": "github/github_actions_runner.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon/spark-apps\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-05 04:25:16 +0000 (Fri, 05 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-an-organization\n#\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a GitHub Actions Runner token for the given Repo or Organization via the GitHub API and then runs a Dockerized GitHub Actions runner with the appropriate configuration\n\nSee Also:\n\n    https://github.com/HariSekhon/Kubernetes-configs - for running GitHub Actions Runners in Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo_or_organization> [<runner_config_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_or_org=\"$1\"\nshift\n\ntoken=\"$(\"$srcdir/github_actions_runner_token.sh\" \"$repo_or_org\")\"\n\ndocker run -ti \\\n           --rm \\\n           -v /var/run/docker.sock:/var/run/docker.sock \\\n           harisekhon/github-actions-runner \\\n           --url \"https://github.com/$repo_or_org\" \\\n           --token \"$token\" \\\n           --unattended \"$@\"\n"
  },
  {
    "path": "github/github_actions_runner_local.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-08 22:15:59 +0100 (Wed, 08 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nDownloads, configures and runs GitHub Actions Runner to run on the local machine\n\n# XXX: WARNING: GitHub advises self-hosted runners only be used for Private repos to prevent arbitrary code execution on your runners via Pull Requests (or you can set 'Allow local actions only' in the Actions Runners configuration in the repo or org)\n\nRepo URL can be supplied via \\$GITHUB_ACTIONS_REPO, otherwise attempts to infer from the local checkout's git remote url\n\nToken can be supplied as either first argument or via environment variable \\$GITHUB_ACTIONS_RUNNER_TOKEN\n\nVersion can be specified via the environment variable \\$GITHUB_ACTIONS_RUNNER_VERSION\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<GITHUB_ACTIONS_RUNNER_TOKEN>]\"\n\nhelp_usage \"$@\"\n\nVERSION=\"${GITHUB_ACTIONS_RUNNER_VERSION:-${GITHUB_ACTIONS_VERSION:-${VERSION:-2.284.0}}}\"\n\nGITHUB_ACTIONS_RUNNER_TOKEN=\"${1:-${GITHUB_ACTIONS_RUNNER_TOKEN:-}}\"\n\ncheck_env_defined GITHUB_ACTIONS_RUNNER_TOKEN\n\ngithub_repo=\"${GITHUB_ACTIONS_REPO:-}\"\nif [ -z \"$github_repo\" ]; then\n    echo \"GITHUB_ACTIONS_REPO not defined, inferring from local repository\"\n    github_repo_url=\"$(git remote -v | awk '/github/{print $2}' | head -n 1 | git_repo_strip_auth)\"\n    if [ -z \"$github_repo_url\" ]; then\n        echo \"Failed to infer github repo url from local git repository\"\n        exit 1\n    fi\n    github_repo_url=\"${github_repo_url#ssh://}\"\n    github_repo_url=\"${github_repo_url/://}\"\n    if ! [[ \"$github_repo_url\" =~ https?:// ]]; then\n        github_repo_url=\"https://$github_repo_url\"\n    fi\n    echo \"inferred github repo url as: $github_repo_url\"\nfi\n\ncd \"$srcdir\"\n\ndir=\".github_actions_runner\"\n\nmkdir -pv \"$dir\"\n\ncd \"$dir\"\n\nos=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\nif [ \"$os\" = darwin ]; then\n    os=osx\nfi\n\ntar=\"actions-runner-$os-x64-$VERSION.tar.gz\"\n\nif ! [ -f \"$tar\" ]; then\n    url=\"https://github.com/actions/runner/releases/download/v$VERSION/actions-runner-$os-x64-$VERSION.tar.gz\"\n    curl -O -L \"$url\"\nfi\n\nif ! [ -f config.sh ]; then\n    tar xzf \"$tar\" || rm -fv -- \"$tar\"\nfi\n\nif ! [ -f .credentials ] ||\n   ! [ -f .runner ]; then\n    ./config.sh remove || :\n    set +o pipefail\n    yes \"\" | ./config.sh --url \"$github_repo_url\" --token \"$GITHUB_ACTIONS_RUNNER_TOKEN\"\nfi\n\n./run.sh\n"
  },
  {
    "path": "github/github_actions_runner_token.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon/spark-apps\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 16:21:52 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-an-organization\n#\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a GitHub Actions runner token to register a new self-hosted runner for the given Repo or Organization via the GitHub API\n\nSee Also:\n\n    github_actions_runner.sh - generates a token and launches a runner for a GitHub Organization or Repo\n\n    https://github.com/HariSekhon/Kubernetes-configs - for running GitHub Actions Runners in Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo_or_organization>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_or_org=\"$1\"\nshift\n\nif [[ \"$repo_or_org\" =~ / ]]; then\n    prefix=\"repos\"\nelse # it's an org\n    prefix=\"orgs\"\nfi\n\n\"$srcdir/github_api.sh\" \"/$prefix/$repo_or_org/actions/runners/registration-token\" -X POST |\njq -r '.token'\n"
  },
  {
    "path": "github/github_actions_runners.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 16:21:52 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-an-organization\n#\n# https://docs.github.com/en/rest/reference/actions#create-a-registration-token-for-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GitHub Actions self-hosted runners for the given Repo or Organization via the GitHub API\n\nOutput Format:\n\n<id>    <online/offline>    <busy_boolean>    <os>    <name>    <label1,label2...>\n\nSee Also:\n\n    github_actions_runner.sh - generates a token and launches a runner for a GitHub Organization or Repo\n\n    https://github.com/HariSekhon/Kubernetes-configs - for running GitHub Actions Runners in Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo_or_organization>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nrepo_or_org=\"$1\"\nshift\n\nif [[ \"$repo_or_org\" =~ / ]]; then\n    prefix=\"repos\"\nelse # it's an org\n    prefix=\"orgs\"\nfi\n\"$srcdir/github_api.sh\" \"/$prefix/$repo_or_org/actions/runners\" |\njq -r '.runners[] | [.id, .status, .busy, .os, .name, ([.labels[].name]|join(\",\")) ] | @tsv' |\nsort -n |\ncolumn -t\n"
  },
  {
    "path": "github/github_actions_workflow_enable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-28 11:11:51 +0000 (Sun, 28 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables a GitHub Actions workflow via the API\n\nUse this to enable a workflow programmatically.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> <workflow_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nrepo=\"$1\"\nworkflow_id=\"$2\"\n\nUSER=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\n\nif ! [[ $repo =~ / ]]; then\n    repo=\"$USER/$repo\"\nfi\n\ntimestamp \"Enabling github actions workflow with id '$workflow_id'\"\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/workflows/$workflow_id/enable\" -X PUT\ntimestamp \"workflow id '$workflow_id' enabled'\"\n"
  },
  {
    "path": "github/github_actions_workflow_runs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: ci_ubuntu\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 16:21:52 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nReturns GitHub Actions Workflow runs for a given Workflow ID (or name.yaml) in json format via the API\n\nWorkflow ID can be either a number (see output of adjacent github_actions_workflows.sh), or the name of the workflow yaml file, with or without the .yaml extension\n\nIf no repo arg is given and is inside a git repo then takes determines the repo from the first git remote listed\n\n\\$REPO and \\$WORKFLOW_ID environment variables are also supported with positional args taking precedence\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<workflow_id> [<repo>]\"\n\nhelp_usage \"$@\"\n\nworkflow_id=\"${1:-${WORKFLOW_ID:-}}\"\n\nrepo=\"${2:-${REPO:-}}\"\n\nif [ -z \"$workflow_id\" ]; then\n    usage \"workflow_id not specified\"\nfi\n\nif ! [[ \"$workflow_id\" =~ ^[[:digit:]]+$ ]]; then\n    if ! [[ \"$workflow_id\" =~ \\.ya?ml$ ]]; then\n        workflow_id=\"$workflow_id.yaml\"\n    fi\nfi\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\nif [ -z \"$repo\" ]; then\n    usage \"repo not specified and couldn't determine from git remote command\"\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif ! [[ $repo =~ / ]]; then\n    repo=\"$USER/$repo\"\nfi\n\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/workflows/$workflow_id/runs\"  # | | jq '.workflow_runs[0:10]'\n"
  },
  {
    "path": "github/github_actions_workflows.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 16:21:52 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns GitHub Actions Workflows json via the API\n\nIf no repo arg is given and is inside a git repo then takes determines the repo from the first git remote listed\n\nOptional workflow id as second parameter will filter to just that workflow\n\n\\$REPO and \\$WORKFLOW_ID environment variables are also supported with positional args taking precedence\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> [<workflow_id>]\"\n\nhelp_usage \"$@\"\n\nrepo=\"${1:-${REPO:-}}\"\n\nworkflow_id=\"${2:-${WORKFLOW_ID:-}}\"\nif [ -n \"$workflow_id\" ]; then\n    workflow_id=\"/$workflow_id\"\nfi\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\nif [ -z \"$repo\" ]; then\n    usage \"repo not specified and couldn't determine from git remote command\"\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nUSER=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\nPASSWORD=\"${GITHUB_PASSWORD:-${GITHUB_TOKEN:-${PASSWORD:-}}}\"\n\nif ! [[ $repo =~ / ]]; then\n    repo=\"$USER/$repo\"\nfi\n\n# XXX: would need to iterate pages if you have more than 100 workflows\n\"$srcdir/github_api.sh\" \"/repos/$repo/actions/workflows$workflow_id?per_page=100\" -L # | jq -r '.workflows[].path' | sed 's|.github/workflows/||;s|\\.yaml$||'\n"
  },
  {
    "path": "github/github_actions_workflows_cancel_all_runs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-09 18:39:14 +0000 (Wed, 09 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCancels all runs in the current or given GitHub repo\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/<repo>]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nargs=()\nif [ $# -gt 0 ]; then\n    repo=\"$1\"\n    args+=(-R \"$repo\")\nfi\n\ngh run list -L 200 ${args:+\"${args[@]}\"} \\\n            --json name,status,databaseId \\\n            -q '.[] | select(.status != \"completed\") | [.databaseId, .name] | @tsv' |\n#xargs -L1 echo gh run cancel ${args:+\"${args[@]}\"} |\nwhile read -r id name; do\n    timestamp \"Cancelling workflow: $name\"\n    echo gh run cancel ${args:+\"${args[@]}\"} \"$id\"\ndone |\nparallel -j 10\n"
  },
  {
    "path": "github/github_actions_workflows_cancel_waiting_runs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-08 16:15:45 +0100 (Wed, 08 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCancels runs waiting for deployment environment approval in the current or given GitHub repo (these can build up from pushes)\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/<repo>]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nargs=()\nif [ $# -gt 0 ]; then\n    repo=\"$1\"\n    args+=(-R \"$repo\")\nfi\n\ngh run list -L 200 ${args:+\"${args[@]}\"} \\\n            --json name,status,databaseId \\\n            -q '.[] | select(.status == \"waiting\") | [.databaseId, .name] | @tsv' |\nwhile read -r id name; do\n    timestamp \"Cancelling workflow: $name\"\n    echo gh run cancel ${args:+\"${args[@]}\"} \"$id\"\ndone |\nparallel -j 10\n"
  },
  {
    "path": "github/github_actions_workflows_disabled.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-28 11:11:51 +0000 (Sun, 28 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GitHub Actions Workflows enabled/disable state via the API\n\nOutput format:\n\n<workflow_id>    <enabled/disable>   <workflow_name>\n\n\nTo scan all repos to find any disabled workflows, such as developers disabling code scanning workflows that are bugging them in PRs, do:\n\n    github_foreach_repo.sh github_actions_workflows_disabled.sh {owner}/{repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> [<workflow_id>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/github_actions_workflows.sh\" \"$@\" |\njq -r '.workflows[] | select(.state != \"active\") | [.id, .state, .name ] | @tsv'\n"
  },
  {
    "path": "github/github_actions_workflows_enable_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-28 11:11:51 +0000 (Sun, 28 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables all GitHub Actions workflows via the API\n\nUse to re-enable all your workflows as GitHub has started disabling workflows in repos after 6 months without a commit\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(get_github_repo)\"\nfi\n\nUSER=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\n\nif ! [[ $repo =~ / ]]; then\n    repo=\"$USER/$repo\"\nfi\n\ntimestamp \"Enabling all github actions workflows in repo '$repo'\"\n\"$srcdir/github_actions_foreach_workflow.sh\" \"$repo\" \"$srcdir/github_actions_workflow_enable.sh\" \"$repo\" \"{id}\"\n"
  },
  {
    "path": "github/github_actions_workflows_rerun_failed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-22 23:45:18 +0000 (Tue, 22 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRe-runs the last failed run of each workflow for the current or given GitHub repo\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nowner_repo=\"${1:-}\"\n\nargs=()\nif [ -n \"$owner_repo\" ]; then\n    is_github_owner_repo \"$owner_repo\" || die \"Invalid GitHub owner/repo given: $owner_repo\"\n    args+=(-R \"$owner_repo\")\nfi\n\ngh run list -L 200 ${args:+\"${args[@]}\"} \\\n            --json name,status,conclusion,workflowDatabaseId,databaseId \\\n            -q '.[] |\n                select(.status == \"completed\") |\n                select(.conclusion != \"success\") |\n                select(.conclusion != \"skipped\") |\n                [.databaseId, .name] |\n                @tsv' |\nsort -u -k 2,3 |\nwhile read -r id name; do\n    timestamp \"re-triggering workflow run: $name ($id)\"\n    echo \"gh run rerun ${args:+\"${args[*]}\"} $id\"\ndone |\nparallel -j 5\n"
  },
  {
    "path": "github/github_actions_workflows_state.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-28 11:11:51 +0000 (Sun, 28 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GitHub Actions Workflows enabled/disable state via the API\n\nOutput format:\n\n<workflow_id>    <enabled/disable>   <workflow_name>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> [<workflow_id>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/github_actions_workflows.sh\" \"$@\" |\njq -r '.workflows[] | [.id, .state, .name ] | @tsv'\n"
  },
  {
    "path": "github/github_actions_workflows_status.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 16:21:52 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GitHub Actions Workflows run status via the API\n\nIf no repo arg is given and is inside a git repo then takes determines the repo from the first git remote listed\n\n\\$REPO and \\$WORKFLOW_ID environment variables are also supported with positional args taking precedence\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> [<workflow_id>]\"\n\nhelp_usage \"$@\"\n\nworkflows=\"$(\n    \"$srcdir/github_actions_workflows.sh\" \"$@\" |\n    jq -r '.workflows[].path' |\n    sed 's|.github/workflows/||;s|\\.yaml$||'\n)\"\n\n\nfor workflow_name in $workflows; do\n    {\n    output=\"$(\n        printf '%s\\t' \"$workflow_name\"\n        \"$srcdir/github_actions_workflow_runs.sh\" \"$workflow_name\" |\n        jq -r 'limit(1; .workflow_runs[] | select(.status == \"completed\") | .conclusion)'\n    )\"\n    echo \"$output\"\n    } &\ndone |\nsort |\ncolumn -t\n"
  },
  {
    "path": "github/github_actions_workflows_status2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-03 14:45:29 +0000 (Thu, 03 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the last completed status of all GitHub Actions workflows\n\nRequires GitHub CLI to be installed and configured with a \\$GITHUB_TOKEN\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nseen=()\n\ngh run list -L 200 --json  name,status,conclusion \\\n                   -q '.[] |\n                       select(.status == \"completed\") |\n                       [.conclusion, .name] | @tsv' |\nwhile read -r conclusion name; do\n    if ! printf '%s\\n' \"${seen[@]}\" | grep -Fixq \"$name\"; then\n        printf '%-10s\\t%s\\n' \"$conclusion\" \"$name\"\n        seen+=(\"$name\")\n    fi\ndone\n"
  },
  {
    "path": "github/github_actions_workflows_trigger_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-09 18:39:14 +0000 (Wed, 09 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTriggers all workflows in the current or given GitHub repo\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/<repo> <gh_options>]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nowner_repo='{owner}/{repo}'\nargs=()\nif [ $# -gt 0 ]; then\n    repo=\"$1\"\n    shift || :\n    owner_repo=\"$repo\"\n    args+=(-R \"$repo\")\nfi\n\ngh api \"/repos/$owner_repo/actions/workflows\" -q '.workflows[].name' |\nwhile read -r workflow_name; do\n    timestamp \"Triggering workflow: $workflow_name\"\n    gh workflow run ${args:+\"${args[@]}\"} \"$workflow_name\" \"$@\"\ndone\n"
  },
  {
    "path": "github/github_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /user | jq .\n#  args: /users/HariSekhon/repos | jq .\n#  args: /repos/HariSekhon/DevOps-Bash-tools/actions/workflows | jq.\n#  args: /repos/:user/:repo/actions/workflows | jq.\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-12 23:43:00 +0000 (Wed, 12 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the GitHub.com API\n\nAutomatically handles authentication via environment variables \\$GITHUB_USERNAME / \\$GITHUB_USER\nand \\$GH_TOKEN / \\$GITHUB_TOKEN / \\$GITHUB_PASSWORD (password is deprecated)\n\nOptional: \\$GITHUB_USER - used for some replacement tokens, prevents having to search git remotes or query the API for it\n          \\$GH_HOST / \\$GITHUB_HOST - FQDN used to point to self-hosts GitHub Enterprise servers\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here - may need to click 'Enable SSO' next to each token to access corporation organizations with SSO (eg. Azure AAD SSO):\n\n    https://github.com/settings/tokens\n\n\nAPI Reference:\n\n    https://docs.github.com/en/rest/reference\n\n\nExamples:\n\n\n# Get currently authenticated user:\n\n    ${0##*/} /user\n\n\n# Get a specific user:\n\n    ${0##*/} /users/HariSekhon\n\n\n# List a user's GitHub repos:\n\n    ${0##*/} /users/HariSekhon/repos\n\n\n# List an organization's GitHub repos:\n\n    ${0##*/} /orgs/MyOrg/repos\n\n\n# Check this specific PAT token can access a specific repo (eg. for testing integration from ArgoCD):\n\n    GH_TOKEN=... ${0##*/} /repos/<owner>/<repo>\n\n\n# Get the GitHub Actions workflows for a given repo:\n\n    ${0##*/} /repos/HariSekhon/DevOps-Bash-tools/actions/workflows\n\n# List repository invitations:\n\n    ${0##*/} /user/repository_invitations\n\n# For convenience you can even copy and paste out of the documentation literally and have the script auto-determine the right settings. The following tokens in the form :token, <token>, {token} are replaced:\n\n# Placeholders replaced by \\$GITHUB_USERNAME / \\$GITHUB_USER:                 owner, username, user\n# Placeholders replaced by the local repo name of the current directory:    repo\n\n    ${0##*/} /repos/:user/:repo/actions/workflows\n\n    ${0##*/} /users/{username}/settings/billing/actions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://${GH_HOST:-${GITHUB_HOST:-api.github.com}}\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\n# false positive, this works\n# shellcheck disable=SC2295\nurl_path=\"${url_path##$url_base}\"\nurl_path=\"${url_path##/}\"\n\nuser=\"${GITHUB_USERNAME:-${GITHUB_USER:-}}\"\n\nif [ -z \"$user\" ]; then\n    user=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@github\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//;s/:.*//' || :)\"\n    # curl_auth.sh does this automatically\n    #if [ -z \"$user\" ]; then\n    #    user=\"${USERNAME:${USER:-}}\"\n    #fi\nfi\n\nPASSWORD=\"${GH_TOKEN:-${GITHUB_TOKEN:-${GITHUB_PASSWORD:-}}}\"\n\nif [ -z \"${PASSWORD:-}\" ]; then\n    PASSWORD=\"$(git remote -v | awk '/https:\\/\\/[[:alnum:]]+@github\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//')\"\nfi\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nfi\nexport PASSWORD\n\n#if [ -n \"${PASSWORD:-}\" ]; then\n#    echo \"using authenticated access\" >&2\n#fi\n\n# for convenience of straight copying and pasting out of documentation pages\n\nrepo=$(git_repo | sed 's/.*\\///' || :)\n\nif [ -n \"$user\" ]; then\n    url_path=\"${url_path/:owner/$user}\"\n    url_path=\"${url_path/<owner>/$user}\"\n    #url_path=\"${url_path/\\{owner\\}/$user}\"\n    url_path=\"${url_path/:username/$user}\"\n    url_path=\"${url_path/<username>/$user}\"\n    url_path=\"${url_path/\\{username\\}/$user}\"\n    url_path=\"${url_path/:user/$user}\"\n    url_path=\"${url_path/<user>/$user}\"\n    url_path=\"${url_path/\\{user\\}/$user}\"\nfi\nurl_path=\"${url_path/:repo/$repo}\"\nurl_path=\"${url_path/<repo>/$repo}\"\n#url_path=\"${url_path/\\{repo\\}/$repo}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "github/github_clone_or_pull_all_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-01 00:55:07 +0100 (Thu, 01 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGit clones all repos for a given owner (user or organization)\n\nClones repos to the current directory using the same directory names as each repo\n\nBy default will clone via SSH. Set the environment variable GIT_HTTPS to any value to use the HTTPS protocol to clone instead\n\nIf the directory already exists, enters, switches to the default branch and does a 'git pull'\n\nFor an organization, you must set the environment variable \\$GITHUB_ORGANIZATION, otherwise by default will attempt to determine your github user via \\$GITHUB_USER or your currently authenticated GitHub API user as defined from the utility script github_api.sh\n\nRequires Git and the adjacent github_api.sh script to be installed and configured for authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nowner=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\n\ngithub_url=\"git@github.com:\"\nif [ -n \"${GIT_HTTPS:-}\" ]; then\n    github_url=\"https://github.com/\"\nfi\n\ntrap_cmd 'echo Exited with error'\n\nget_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    if [ -d \"$repo\" ]; then\n        timestamp \"GitHub repo directory '$repo' already exists, entering directory to update checkout\"\n        pushd \"$repo\" >/dev/null\n        echo\n        branch=\"$(default_branch)\"\n        timestamp \"Switching to default branch '$branch'\"\n        if git status --porcelain | grep -q '^.M'; then\n            echo\n            timestamp \"Stashing changes in progress\"\n            git stash\n            echo\n        fi\n        git checkout \"$branch\"\n        echo\n        timestamp \"Git Pull\"\n        git pull\n        if git stash list | grep -q .; then\n            timestamp \"Popping stash\"\n            git stash pop 2>/dev/null || :\n            echo\n        fi\n        echo\n        timestamp \"Leaving directory\"\n        popd >/dev/null\n    else\n        timestamp \"Cloning repo '$owner/$repo' to directory '$repo'\"\n        git clone \"$github_url\"\"$owner/$repo\"\n    fi\n    echo\n    hr\n    echo\ndone\nuntrap\ntimestamp \"All github repos cloned and up to date for owner '$owner'\"\n"
  },
  {
    "path": "github/github_download_release_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: https://github.com/pgjdbc/pgjdbc/releases/download/REL{version}/postgresql-{version}.jar latest\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 23:01:46 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads the given or latest version of a GitHub release file\n\nIf the file url template contains {version} placeholders but no explicit version then the placeholds will be replaced with the latest version which will be automatically determined from GitHub releases\n\nIf the file URL does not contain {version} placeholders then downloads the URL as given\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\nDesigned to be called from download_github_jar.sh and similar adjacent scripts to deduplicate code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url> [<version>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nurl=\"$1\"\nversion=\"${2:-latest}\"\n\nurl=\"${url## }\"\nurl=\"${url%% }\"\n\nshopt -s nocasematch\n# defined in lib/github.sh\n# shellcheck disable=SC2154\nif ! [[ \"$url\" =~ ^$github_release_url_regex ]]; then\n    die \"ERROR: Invalid URL given, not a GitHub release URL: $url\"\nfi\nshopt -u nocasematch\n\nif [[ \"$url\" =~ {version} ]]; then\n    github_owner_repo=\"$(sed 's|^https://github.com/||i; s|/releases/.*$||i' <<< \"$url\")\"\n    if [ \"$version\" = latest ]; then\n        timestamp \"Determining latest release of $github_owner_repo available\"\n        version=\"$(github_repo_latest_release.sh \"$github_owner_repo\")\"\n        # bash built-in variable substitution cannot replace all alpha prefix chars - tested this already\n        # shellcheck disable=SC2001\n        version=\"$(sed 's/^[[:alpha:]]*//' <<< \"$version\")\"\n        timestamp \"Determined latest version of $github_owner_repo to be version '$version'\"\n    fi\n    url=\"${url//\\{version\\}/$version}\"\nfi\n\n\"$srcdir/../bin/download_url_file.sh\" \"$url\"\n"
  },
  {
    "path": "github/github_download_release_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: https://github.com/pgjdbc/pgjdbc/releases/download/REL{version}/postgresql-{version}.jar latest\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 23:01:46 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads the given or latest version of a GitHub release file\n\nIf the file url template contains {version} placeholders but no explicit version then the placeholds will be replaced with the latest version which will be automatically determined from GitHub releases\n\nIf the file URL does not contain {version} placeholders then downloads the URL as given\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\nDesigned to be called from download_github_jar.sh and similar adjacent scripts to deduplicate code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url> [<version>]\"\n\nhelp_usage \"$@\"\n\ncheck_bin file\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nurl=\"$1\"\nversion=\"${2:-latest}\"\n\nif ! [[ \"$url\" =~ \\.jar$ ]]; then\n    die \"ERROR: given URL is not for a JAR file: $url\"\nfi\n\n\"$srcdir/github_download_release_file.sh\" \"$url\" \"$version\"\necho >&2\n\ntimestamp \"Validating JAR content format\"\nfilename=\"${url##*/}\"\nfilename=\"${filename//\\{version\\}/*}\"\n# have to risk splitting to get globbing to work on latest file\n# shellcheck disable=SC2086,SC2046\noutput=\"$(file --brief $(ls -tr $filename))\"\nexpected_format=\"^Zip archive data|^Java archive data \\(JAR\\)\"\nif ! [[ \"$output\" =~ $expected_format ]]; then\n    die \"ERROR: JAR failed 'file' type matching, expected '$expected_format', got: '$output'\"\nfi\ntimestamp \"JAR file format validation passed\"\n"
  },
  {
    "path": "github/github_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo owner={owner} repo={repo}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-30 10:08:07 +0100 (Sun, 30 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each original non-fork GitHub repo\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the repo names and exit after the first iteration\n\n\\$GITHUB_ORGANIZATION / \\$GITHUB_USER - the user or organization to iterate the repos on - if not specified then determines the currently authenticated github user from your \\$GITHUB_TOKEN\n\nThe command template replaces the following for convenience in each iteration:\n\n{owner}               => the user or organization that owns the repo eg. HariSekhon or MyOrg, whichever owns the repo\n{repo}                => the repo name without the owner prefix eg. DevOps-Bash-tools\n\neg.\n    ${0##*/} echo owner={owner}  name={name} repo={repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nowner=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\n\nget_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    owner_repo=\"$owner/$repo\"\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $owner_repo\" >&2\n    echo \"# ============================================================================ #\" >&2\n    echo >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{owner\\}/$owner}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "github/github_forked_add_remote.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-07 00:27:46 +0300 (Wed, 07 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEasily adds a git remote to a GitHub forked repo selected from a menu list\n\nIf executed from the checkout of GitHub repo, determines if this is a fork and if so finds the upstream forks to choose from\n\nOr the owner/repo can be specified explicitly, with the shorthand '.' meaning the current repo's forks, regardless of whether it is itself a fork\n\n\nRequires GitHub CLI to be installed and authenticated, as well as jq\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner/repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_github_origin\n\nif [ $# -gt 0 ]; then\n    owner_repo=\"$1\"\n    shift\n    if [ \"$owner_repo\" = . ]; then\n        timestamp \"Determining self owner/repo\"\n        owner_repo=\"$(github_owner_repo)\"\n        timestamp \"Determined self owner/repo to be: $owner_repo\"\n    fi\nelif is_github_fork; then\n    timestamp \"Detected running in the checkout of a forked repo\"\n    timestamp \"Determining upstream origin repo\"\n    owner_repo=\"$(github_upstream_owner_repo || die \"Not a forked repo?\")\"\n    timestamp \"Determined upstream origin repo to be: $owner_repo\"\nelse\n    timestamp \"Determining owner/repo\"\n    owner_repo=\"$(github_owner_repo)\"\n    timestamp \"Determined owner/repo to be: $owner_repo\"\nfi\n\ntimestamp \"Retrieving forked repos for: $owner_repo\"\nfork_list=\"$(gh api \"repos/$owner_repo/forks\" --paginate --jq '.[].full_name')\"\n\nif is_blank \"$fork_list\"; then\n    die \"No forks found for repo: '$owner_repo\"\nfi\n\nfork_owner_repo=\"$(fzf --prompt=\"Select a Fork: \" --height=40% --border --ansi <<< \"$fork_list\")\"\n\n# default remote name\nfork_remote=\"${fork_owner_repo//\\//_}\"\n\nif is_blank \"$fork_remote\"; then\n    die \"Git remote name cannot be blank!\"\nfi\n\ntimestamp \"Determining git base url\"\necho\ngithub_url_base=\"$(\n    git remote -v |\n    awk '/origin/ { print $2; exit }' |\n    sed 's|\\(.*github.com[:/]\\).*|\\1|'\n)\"\n\nfork_remote_url=\"${github_url_base}${fork_owner_repo}\"\nfork_remote_url_escaped=\"${fork_remote_url//\\//\\\\/}\"\n\nif git remote -v | awk '{print $2}' | grep -Eq \"^$fork_remote_url(\\\\.git)?$\"; then\n    timestamp \"Existing git remote found with fork url '$fork_remote_url', skipping adding remote\"\n    fork_remote=\"$(git remote -v | awk \"/[[:space:]]${fork_remote_url_escaped}[[:space:].]/ {print \\$1; exit}\")\"\n    if is_blank \"$fork_remote\"; then\n        die \"Failed to parse existing git remote for url: $fork_remote_url_escaped\"\n    fi\n    timestamp \"Use existing fork remote: $fork_remote\"\nelse\n    fork_remote=\"$(dialog --inputbox \"Enter name of git remote to create:\" 8 40 \"$fork_remote\" 3>&1 1>&2 2>&3)\"\n    if git remote -v | grep -q \"^${fork_remote}[[:space:]]\"; then\n        timestamp \"Git remote '$fork_remote' already exists, not creating\"\n    else\n        timestamp \"Adding git remote '$fork_remote' to be able to pull directly from forked repo\"\n        git remote add \"$fork_remote\" \"$fork_remote_url\"\n    fi\nfi\necho\n\ntimestamp \"Fetching repo '$fork_owner_repo' from git remote: $fork_remote\"\ngit fetch \"$fork_remote\"\n"
  },
  {
    "path": "github/github_forked_checkout_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-07 00:27:46 +0300 (Wed, 07 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets up a branch from a GitHub forked repo selected from a menu list\n\nIf executed from the checkout of GitHub repo, determines if this is a fork and if so finds the upstream forks to pull from\n\nOr the owner/repo can be specified explicitly, with the shorthand '.' meaning the current repo's forks, regardless of whether it is itself a fork\n\n\nRequires GitHub CLI to be installed and authenticated, as well as jq\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner/repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_github_origin\n\nif [ $# -gt 0 ]; then\n    owner_repo=\"$1\"\n    shift\n    if [ \"$owner_repo\" = . ]; then\n        timestamp \"Determining self owner/repo\"\n        owner_repo=\"$(github_owner_repo)\"\n        timestamp \"Determined self owner/repo to be: $owner_repo\"\n    fi\nelif is_github_fork; then\n    timestamp \"Detected running in the checkout of a forked repo\"\n    timestamp \"Determining upstream origin repo\"\n    owner_repo=\"$(github_upstream_owner_repo || die \"Not a forked repo?\")\"\n    timestamp \"Determined upstream origin repo to be: $owner_repo\"\nelse\n    timestamp \"Determining owner/repo\"\n    owner_repo=\"$(github_owner_repo)\"\n    timestamp \"Determined owner/repo to be: $owner_repo\"\nfi\n\ntimestamp \"Retrieving forked repos for: $owner_repo\"\nfork_list=\"$(gh api \"repos/$owner_repo/forks\" --paginate --jq '.[].full_name')\"\n\nif is_blank \"$fork_list\"; then\n    die \"No forks found for repo: '$owner_repo\"\nfi\n\nfork_owner_repo=\"$(fzf --prompt=\"Select a Fork: \" --height=40% --border --ansi <<< \"$fork_list\")\"\n\ntimestamp \"Retrieving branches for chosen fork: $fork_owner_repo\"\nbranches=\"$(gh api -H \"Accept: application/vnd.github.v3+json\" \"/repos/$fork_owner_repo/branches\" --paginate -q '.[].name')\"\n\nif is_blank \"$branches\"; then\n    die \"No branches found or unable to fetch branches\"\nfi\n\nbranch=$(fzf --prompt=\"Select a Branch from fork '$fork_owner_repo': \" --height=40% --border --ansi <<< \"$branches\")\n\nfork_remote=\"${fork_owner_repo//\\//_}\"\n\ntimestamp \"Determining git base url\"\necho\ngithub_url_base=\"$(\n    git remote -v |\n    awk '/origin/ { print $2; exit }' |\n    sed 's|\\(.*github.com[:/]\\).*|\\1|'\n)\"\n\nfork_remote_url=\"${github_url_base}${fork_owner_repo}\"\nfork_remote_url_escaped=\"${fork_remote_url//\\//\\\\/}\"\n\nif git remote -v | awk '{print $2}' | grep -Eq \"^$fork_remote_url(\\\\.git)?$\"; then\n    timestamp \"Existing git remote found with fork url '$fork_remote_url', reusing that\"\n    fork_remote=\"$(git remote -v | awk \"/[[:space:]]${fork_remote_url_escaped}[[:space:].]/ {print \\$1; exit}\")\"\n    if is_blank \"$fork_remote\"; then\n        die \"Failed to parse existing git remote for url: $fork_remote_url_escaped\"\n    fi\n    timestamp \"Using existing fork remote: $fork_remote\"\nelif git remote -v | grep -q \"^${fork_remote}[[:space:]]\"; then\n    timestamp \"Git remote '$fork_remote' already exists, not creating\"\nelse\n    timestamp \"Adding git remote '$fork_remote' to be able to pull directly from forked repo\"\n    git remote add \"$fork_remote\" \"$fork_remote_url\"\nfi\necho\n\ntimestamp \"Fetching branch '$branch' from fork repo '$fork_owner_repo'\"\ngit fetch \"$fork_remote\" \"$branch\"\n\ngit checkout -b \"$fork_remote/$branch\" \"$fork_remote/$branch\"\n"
  },
  {
    "path": "github/github_generate_starcharts.md.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-07 23:00:59 +0000 (Mon, 07 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\ntop_N=20\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to generate STARCHARTS.md containing the star graphs of the Top N GitHub repos on a single page\n\n\n# Examples:\n\n\n    Without arguments queries for all non-fork repos for your \\$GITHUB_USER and iterate them up to $top_N to generate the page\n\n        GITHUB_USER=HariSekhon ./github_generate_status_page.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ntrap 'echo ERROR >&2' exit\n\nfile=\"STARCHARTS.md\"\n\n# this leads to confusion as it generates some randomly unexpected output by querying a github user who happens to have the same name as your local user eg. hari, so force explicit now\n#USER=\"${GITHUB_USER:-${USERNAME:-${USER}}}\"\nif [ -z \"${GITHUB_USER:-}\" ] ; then\n    GITHUB_USER=\"$(get_github_user || :)\"\nfi\nif is_blank \"${GITHUB_USER:-}\" || [ \"$GITHUB_USER\" = null ]; then\n    echo \"\\$GITHUB_USER not set!\"\n    exit 1\nfi\n\nget_repos(){\n    page=1\n    while true; do\n        echo \"fetching GitHub repos - page $page\" >&2\n        if ! output=\"$(\"$srcdir/github_api.sh\" \"/users/$GITHUB_USER/repos?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        # use authenticated requests if you are hitting the API rate limit - this is automatically done above now if USER/PASSWORD GITHUB_USER/GITHUB_PASSWORD/GITHUB_TOKEN environment variables are detected\n        # eg. CURL_OPTS=\"-u harisekhon:$GITHUB_TOKEN\" ...\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq -r '.[] | select(.fork | not) | select(.private | not) | [.full_name, .stargazers_count, .forks] | @tsv' <<< \"$output\"\n        ((page+=1))\n    done\n}\n\nrepolist=\"$(get_repos | grep -v spark-apps | sort -k2nr -k3nr | awk '{print $1}' | head -n \"$top_N\")\"\n\nnum_repos=\"$(wc -w <<< \"$repolist\")\"\nnum_repos=\"${num_repos// /}\"\n\n# make portable between linux and mac\nhead(){\n    if [ \"$(uname -s)\" = Darwin ]; then\n        # from brew's coreutils package (installed by 'make')\n        ghead \"$@\"\n    else\n        command head \"$@\"\n    fi\n}\n\ntempfile=\"$(mktemp)\"\ntrap 'echo ERROR >&2; rm -vf -- \"$tempfile\"' exit\n\n{\nactual_repos=0\n\ntotal_stars=0\ntotal_forks=0\necho \"getting followers\" >&2\nfollowers=\"$(\"$srcdir/github_api.sh\" /users/harisekhon | jq -r .followers)\"\n\necho \"---\" >&2\nfor repo in $repolist; do\n    if ! [[ \"$repo\" =~ / ]]; then\n        repo=\"$GITHUB_USER/$repo\"\n    fi\n    echo \"fetching GitHub repo info for '$repo'\" >&2\n    repo_json=\"$(\"$srcdir/github_api.sh\" \"/repos/$repo\")\"\n    description=\"$(jq -r .description <<< \"$repo_json\")\"\n    stars=\"$(jq -r .stargazers_count <<< \"$repo_json\")\"\n    forks=\"$(jq -r .forks <<< \"$repo_json\")\"\n    #watchers=\"$(jq -r .watchers <<< \"$repo_json\")\"\n    ((total_stars += stars))\n    ((total_forks += forks))\n    #((total_watchers += watchers))\n    echo \"fetching GitHub README.md for '$repo'\" >&2\n    echo \"---\"\n    echo \"---\" >&2\n    title=\"$(curl -sS --fail \"https://raw.githubusercontent.com/$repo/master/README.md\" | { head -n1 | sed 's/^#*//'; cat >/dev/null; } )\"\n    title=\"${title## }\"\n    title=\"${title%% }\"\n    printf '## %s\\n' \"$title\"\n    printf '\n[![Repo on GitHub](https://img.shields.io/badge/GitHub-repo-blue?logo=github)](https://github.com/%s)\n[![GitHub stars](https://img.shields.io/github/stars/%s?logo=github)](https://github.com/%s/stargazers)\n[![GitHub forks](https://img.shields.io/github/forks/%s?logo=github)](https://github.com/%s/network)\n' \"$repo\" \"$repo\" \"$repo\" \"$repo\" \"$repo\"\n    echo\n    #printf '%s\\n' \"Link:  [$repo](https://github.com/$repo)\"\n    #echo\n    printf '%s\\n' \"$description\"\n    echo\n    printf '%s\\n' \"[![Stargazers over time](https://starchart.cc/$repo.svg)](https://starchart.cc/$repo)\"\n    echo\n    ((actual_repos+=1))\ndone\n} > \"$tempfile\"\n\nif [ \"$num_repos\" != \"$actual_repos\" ]; then\n    echo \"ERROR: differing number of target github repos ($num_repos) vs actual repos ($actual_repos)\"\n    exit 1\nfi\n\n{\ncat <<EOF\n# GitHub StarCharts\n\n![Original Repos](https://img.shields.io/badge/Repos-$actual_repos-blue?logo=github)\n![Stars](https://img.shields.io/badge/Stars-$total_stars-blue?logo=github)\n![Forks](https://img.shields.io/badge/Forks-$total_forks-blue?logo=github)\n![Followers](https://img.shields.io/badge/Followers-$followers-blue?logo=github)\n[![Azure DevOps Profile](https://img.shields.io/badge/Azure%20DevOps-HariSekhon-0078D7?logo=azure%20devops)](https://dev.azure.com/harisekhon/GitHub)\n[![GitHub Profile](https://img.shields.io/badge/GitHub-HariSekhon-2088FF?logo=github)](https://github.com/HariSekhon)\n[![GitLab Profile](https://img.shields.io/badge/GitLab-HariSekhon-FCA121?logo=gitlab)](https://gitlab.com/HariSekhon)\n[![BitBucket Profile](https://img.shields.io/badge/BitBucket-HariSekhon-0052CC?logo=bitbucket)](https://bitbucket.org/HariSekhon)\n\n[![GitStar Ranking Profile](https://img.shields.io/badge/GitStar%20Ranking-HariSekhon-blue?logo=github)](https://gitstar-ranking.com/HariSekhon)\n\n[git.io/hari-starcharts](https://git.io/hari-starcharts) generated by \\`${0##*/}\\` in [HariSekhon/DevOps-Bash-tools](https://github.com/HariSekhon/DevOps-Bash-tools)\n\nEOF\ncat \"$tempfile\"\n} | tee \"$file\"\n\nrm -f -- \"$tempfile\"\n\nuntrap\n"
  },
  {
    "path": "github/github_generate_status_page.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-07 15:01:31 +0000 (Fri, 07 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\ntop_N=\"${TOP_N:-100}\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to generate a Markdown page containing the headers and CI/CD status badges of the Top N rated by stars GitHub repos\n\n\nExamples:\n\n\n    Without arguments queries for all non-fork repos for your \\$GITHUB_ORGANIZATION or \\$GITHUB_USER and iterates up to $top_N of them to generate the page\n\n        ./github_generate_status_page.sh\n\n\n    With arguments will query those repo's README.md at the top level - if omitting the owner prefix will prepend \\$GITHUB_ORGANIZATION/ or \\$GITHUB_USER/\n\n        GITHUB_USER=HariSekhon ./github_generate_status_page.sh  DevOps-Python-tools  DevOps-Perl-tools SomeOtherCoolGuy/his-repo\n\n        GITHUB_ORGANIZATION=my-org ./github_generate_status_page.sh  HariSekhon/DevOps-Python-tools  some-org-repo\n\nSupported Environment Variables:\n\n    EXCLUDE_REPOS - an ERE regex or repos to exclude\n    TOP_N         - an integer of the top N number of repos to include, ranked by star count\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user/repo1> <user/repo2> ...]\"\n\nhelp_usage \"$@\"\n\ntrap 'echo ERROR >&2' exit\n\nis_int \"$top_N\" || die \"Invalid TOP_N '$top_N' specified, must be an integer\"\n\nrepolist=\"$*\"\n\n# this leads to confusion as it generates some randomly unexpected output by querying a github user who happens to have the same name as your local user eg. hari, so force explicit now\n#USER=\"${GITHUB_USER:-${USERNAME:-${USER}}}\"\nGITHUB_USER=\"${GITHUB_USER:-$(get_github_user || :)}\"\nif is_blank \"${GITHUB_USER:-}\" || [ \"$GITHUB_USER\" = null ]; then\n    die \"\\$GITHUB_USER not set and could not infer user from token!\"\nfi\nOWNER=\"${GITHUB_ORGANIZATION:-$GITHUB_USER}\"\nif is_blank \"${OWNER:-}\" || [ \"$OWNER\" = null ]; then\n    die \"\\$GITHUB_ORGANIZATION / \\$GITHUB_USER not set and could not infer user from token!\"\nfi\n\nprefix=\"users\"\nif [ -n \"${GITHUB_ORGANIZATION:-}\" ]; then\n    prefix=\"orgs\"\nfi\n\nget_repos(){\n    page=1\n    while true; do\n        echo \"fetching GitHub repos - page $page\" >&2\n        if ! output=\"$(\"$srcdir/github_api.sh\" \"/$prefix/$OWNER/repos?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        # use authenticated requests if you are hitting the API rate limit - this is automatically done above now if USER/PASSWORD GITHUB_USER/GITHUB_PASSWORD/GITHUB_TOKEN environment variables are detected\n        # eg. CURL_OPTS=\"-u harisekhon:$GITHUB_TOKEN\" ...\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq -r '.[] | select(.fork | not) | select(.private | not) | [.full_name, .stargazers_count, .forks] | @tsv' <<< \"$output\"\n        ((page+=1))\n    done\n}\n\noriginal_sources=0\n\nif [ -z \"$repolist\" ]; then\n    repolist=\"$(get_repos |\n                if [ -n \"${EXCLUDE_REPOS:-}\" ]; then\n                    grep -Ev \"$EXCLUDE_REPOS\" || :\n                else\n                    cat\n                fi |\n                sort -k2nr -k3nr |\n                awk '{print $1}' |\n                head -n \"$top_N\")\"\n    original_sources=1\nfi\n\nnum_repos=\"$(wc -w <<< \"$repolist\")\"\nnum_repos=\"${num_repos// /}\"\n\n#echo \"$repolist\" >&2\n\n# make portable between linux and mac\nhead(){\n    if [ \"$(uname -s)\" = Darwin ]; then\n        # from brew's coreutils package (installed by 'make')\n        ghead \"$@\"\n    else\n        command head \"$@\"\n    fi\n}\n\ntempfile=\"$(mktemp)\"\ntempfile_summary=\"$(mktemp)\"\ntempfile_header=\"$(mktemp)\"\n#trap 'echo ERROR >&2; rm -f \"$tempfile\" \"$tempfile_summary\" \"$tempfile_header\"' exit\n\n{\nactual_repos=0\n\ntotal_stars=0\ntotal_forks=0\necho \"getting followers\" >&2\nfollowers=\"$(\"$srcdir/github_api.sh\" \"/$prefix/$OWNER\" | jq -r .followers)\"\n\necho \"---\" >&2\nfor repo in $repolist; do\n    if ! [[ \"$repo\" =~ / ]]; then\n        repo=\"$OWNER/$repo\"\n    fi\n    echo \"fetching GitHub repo info for '$repo'\" >&2\n    repo_json=\"$(\"$srcdir/github_api.sh\" \"/repos/$repo\")\"\n    description=\"$(jq -r .description <<< \"$repo_json\")\"\n    stars=\"$(jq -r .stargazers_count <<< \"$repo_json\")\"\n    forks=\"$(jq -r .forks <<< \"$repo_json\")\"\n    #watchers=\"$(jq -r .watchers <<< \"$repo_json\")\"\n    ((total_stars += stars))\n    ((total_forks += forks))\n    #((total_watchers += watchers))\n    echo \"fetching GitHub README.md for '$repo'\" >&2\n    echo \"---\"\n    echo \"---\" >&2\n    {\n        #perl -e '$/ = undef; my $content=<STDIN>; $content =~ s/<!--[^>]+-->//gs; print $content' |\n        curl -sS \"https://raw.githubusercontent.com/$repo/master/README.md\" |\n        perl -pe '$/ = undef; s/<!--[^>]+-->//gs' |\n        sed -n '1,/^[^\\[[:space:]<=-]/ p' |\n        head -n -1 |\n        #perl -ne 'print unless /=============/;' |\n        grep -v \"===========\" |\n        sed '1 s/^[^#]/# &/'\n    } |\n    {\n        read -r title\n        printf '%s\\n' \"$title\"\n        echo\n        printf '%s\\n' \"Link:  [$repo](https://github.com/$repo)\"\n        echo\n        printf '%s\\n' \"$description\"\n        cat\n    }\n    # works for Link which is a limited format, but couldn't expect to safely inject a description line pulled from the GitHub API because all sorts of characters could be contained which break this, so instead doing this via the brace piping trick above\n    # \\\\ escapes the newlines to allow them inside the sed for literal replacement since \\n doesn't work\n    #sed \"2 s|^|\\\\\n#\\\\\n#Link:  [$repo](https://github.com/$repo)\n#|\"\n    echo\n    ((actual_repos+=1))\ndone\n} > \"$tempfile\"\n\nif [ \"$num_repos\" != \"$actual_repos\" ]; then\n    echo \"ERROR: differing number of target github repos ($num_repos) vs actual repos ($actual_repos)\"\n    exit 1\nfi\n\nlines_of_code_counts=\"$(\n    grep -Ei 'img.shields.io/badge/lines%20of%20code-[[:digit:]]+(\\.[[:digit:]]+)?k' \"$tempfile\" |\n    sed 's|.*img.shields.io/badge/lines%20of%20code-||; s/[[:alpha:]].*$//'|\n    tr '\\n' '+' |\n    sed 's/+$//' || :\n)\"\n# on Mac piping to | bc -l works, but on Linux breaks, but <<< works\nlines_of_code=\"$(bc -l <<< \"$lines_of_code_counts\" || echo \"unknown\")\"\n\nhosted_build_regex='\\[.+('\nhosted_build_regex+='travis-ci.+\\.svg'\nhosted_build_regex+='|github\\.com/.+/workflows/.+/badge\\.svg'\nhosted_build_regex+='|dev\\.azure\\.com/.+/_apis/build/status'\nhosted_build_regex+='|app\\.codeship\\.com/projects/.+/status'\nhosted_build_regex+='|appveyor\\.com/api/projects/status'\nhosted_build_regex+='|circleci\\.com/.+\\.svg'\nhosted_build_regex+='|cloud\\.drone\\.io/api/badges/.+/status.svg'\nhosted_build_regex+='|g\\.codefresh\\.io/api/badges/pipeline/'\nhosted_build_regex+='|api\\.shippable\\.com/projects/.+/badge'\nhosted_build_regex+='|app\\.wercker\\.com/status/'\nhosted_build_regex+='|img\\.shields\\.io/.*/buildspec.yml'\nhosted_build_regex+='|img\\.shields\\.io/.*/cloudbuild.yaml'\nhosted_build_regex+='|img\\.shields\\.io/.+/pipeline'\nhosted_build_regex+='|img\\.shields\\.io/.+/build/'\nhosted_build_regex+='|img\\.shields\\.io/buildkite/'\nhosted_build_regex+='|img\\.shields\\.io/cirrus/'\nhosted_build_regex+='|img\\.shields\\.io/docker/build/'\nhosted_build_regex+='|img\\.shields\\.io/docker/cloud/build/'\nhosted_build_regex+='|img\\.shields\\.io/travis/'\nhosted_build_regex+='|img.shields.io/badge/Shippable'\nhosted_build_regex+='|img.shields.io/badge/TravisCI'\nhosted_build_regex+='|img\\.shields\\.io/shippable/'\nhosted_build_regex+='|img\\.shields\\.io/wercker/ci/'\nhosted_build_regex+='|app\\.buddy\\.works/.*/pipelines/pipeline/.*/badge.svg'\nhosted_build_regex+='|img\\.shields\\.io/badge/Buddy'\nhosted_build_regex+='|\\.semaphoreci\\.com/badges/'\nhosted_build_regex+='|api\\.netlify\\.com/api/v1/badges/'\nhosted_build_regex+=')'\n# to check for any badges missed, just go\n#grep -Ev \"$hosted_build_regex\" README.md\n\nself_hosted_build_regex='\\[\\!\\[[^]]+\\]\\(.*\\)\\]\\(.*/blob/master/('\nself_hosted_build_regex+='Jenkinsfile'\nself_hosted_build_regex+='|.concourse.yml'\nself_hosted_build_regex+='|.gocd.yml'\nself_hosted_build_regex+=')\\)'\nself_hosted_build_regex+='|img\\.shields\\.io/badge/TeamCity'\n\nnum_CI_systems=22\n\nis_owner_harisekhon(){\n    shopt -s nocasematch\n    [[ \"$OWNER\" =~ ^HariSekhon$ ]]\n}\n\nif is_owner_harisekhon; then\n    cat >> \"$tempfile_summary\" <<EOF\n[![Generate README](https://github.com/HariSekhon/CI-CD/actions/workflows/readme.yaml/badge.svg)](https://github.com/HariSekhon/CI-CD/actions/workflows/readme.yaml)\n[![pages-build-deployment](https://github.com/HariSekhon/CI-CD/actions/workflows/pages/pages-build-deployment/badge.svg)](https://github.com/HariSekhon/CI-CD/actions/workflows/pages/pages-build-deployment)\n[![Netlify Status](https://api.netlify.com/api/v1/badges/853ef60c-c01b-4b83-99ba-8fda541f850f/deploy-status)](https://app.netlify.com/sites/harisekhon/deploys)\n![Last Generated](https://img.shields.io/badge/Last%20Generated-$(date +%F |\n                                                                  # \"$srcdir/../bin/urlencode.sh\" |\n                                                                  # need to escape dashes to avoid shields.io interpreting them as field separators\n                                                                  sed 's/-/--/g')-yellowgreen?logo=github)\n[![GitHub Last Commit](https://img.shields.io/github/last-commit/HariSekhon/CI-CD?logo=github)](https://github.com/HariSekhon/CI-CD/commits/master)\nEOF\nfi\n\nif [ -n \"${DEBUG:-}\" ]; then\n    echo\n    echo \"Hosted Builds:\"\n    echo\n    grep -Eh \"$hosted_build_regex\" \"$tempfile\" \"$tempfile_header\" || :\n    echo\n    echo \"Self-Hosted Builds:\"\n    echo\n    grep -Eh \"$self_hosted_build_regex\" \"$tempfile\" \"$tempfile_header\" || :\nfi >&2\nnum_hosted_builds=\"$(grep -Ehc \"$hosted_build_regex\" \"$tempfile\" \"$tempfile_summary\" | tr '\\n' '+' | sed 's/+$/\\n/' | bc -l | sed 's/[[:space:]]//g')\"\nnum_self_hosted_builds=\"$(grep -Ehc \"$self_hosted_build_regex\" \"$tempfile\" \"$tempfile_summary\" | tr '\\n' '+' | sed 's/+$/\\n/' | bc -l | sed 's/[[:space:]]//g')\"\n\nnum_builds=$((num_hosted_builds + num_self_hosted_builds))\n\ncat > \"$tempfile_header\" <<EOF\n# CI/CD Status Page\n\n![Original Repos](https://img.shields.io/badge/Repos-$actual_repos-blue?logo=github)\n![Stars](https://img.shields.io/badge/Stars-$total_stars-blue?logo=github)\n![Forks](https://img.shields.io/badge/Forks-$total_forks-blue?logo=github)\n![Followers](https://img.shields.io/badge/Followers-$followers-blue?logo=github)\nEOF\n\nif ! is_blank \"$lines_of_code\" &&\n   [ \"$lines_of_code\" != unknown ]; then\n    cat >> \"$tempfile_header\" <<EOF\n![Lines of Code](https://img.shields.io/badge/lines%20of%20code-${lines_of_code}k-lightgrey?logo=codecademy)\nEOF\nfi\n\ncat >> \"$tempfile_header\" <<EOF\n[![GitStar Ranking Profile](https://img.shields.io/badge/GitStar%20Ranking-$OWNER-blue?logo=github)](https://gitstar-ranking.com/$OWNER)\nEOF\n\nif is_owner_harisekhon; then\n    cat >> \"$tempfile_header\" <<-EOF\n[![StarCharts](https://img.shields.io/badge/Star-Charts-blue?logo=github)](https://github.com/HariSekhon/DevOps-Bash-tools/blob/master/STARCHARTS.md)\nEOF\nfi\n\nif is_owner_harisekhon; then\n    # XXX: workaround to include Readme Generate, GitHub Pages and Netlify builds here because we cannot precalculate them from $tempfile_header as they are added after the CI Builds count is needed here\n    # XXX: must update this if editing this header to add/remove any builds\n    ((num_CI_systems += 1))  # for Netlify\n    cat >> \"$tempfile_header\" <<EOF\n\n[![Azure DevOps Profile](https://img.shields.io/badge/Azure%20DevOps-HariSekhon-0078D7?logo=azure%20devops)](https://dev.azure.com/harisekhon/GitHub)\n[![GitHub Profile](https://img.shields.io/badge/GitHub-HariSekhon-2088FF?logo=github)](https://github.com/HariSekhon)\n[![GitLab Profile](https://img.shields.io/badge/GitLab-HariSekhon-FCA121?logo=gitlab)](https://gitlab.com/HariSekhon)\n[![BitBucket Profile](https://img.shields.io/badge/BitBucket-HariSekhon-0052CC?logo=bitbucket)](https://bitbucket.org/HariSekhon)\n[![GitHub Pages](https://img.shields.io/badge/GitHub-Pages-2088FF?logo=github)](https://harisekhon.github.io/CI-CD/)\n[![Netlify](https://img.shields.io/badge/Netlify-site-00C7B7?logo=netlify)](https://harisekhon.netlify.app/)\n\n[![CI/CD Builds](https://img.shields.io/badge/CI%2FCD%20Builds-$num_builds-blue?logo=circleci)](https://harisekhon.github.io/CI-CD/)\n$(cat \"$tempfile_summary\")\nEOF\nfi\n\n#if is_owner_harisekhon; then\n#    cat <<EOF\n#\n#[git.io/hari-ci](https://git.io/hari-ci) generated by \\`${0##*/}\\` in [HariSekhon/DevOps-Bash-tools](https://github.com/HariSekhon/DevOps-Bash-tools)\n#\n#This page usually loads better on [BitBucket](https://bitbucket.org/harisekhon/ci-cd/src/master/README.md) due to less aggressive proxy timeouts cutting off badge loading than GitHub / GitLab\n#\n#EOF\n#\n#fi\n\ncat \"$tempfile_header\"\necho\nprintf \"%s \" \"$num_repos\"\nif [ \"$original_sources\" = 1 ]; then\n    printf \"original source \"\nfi\nprintf 'git repos with %s CI builds (%s hosted, %s self-hosted) across %s different CI/CD systems:\\n\\n' \"$num_builds\" \"$num_hosted_builds\" \"$num_self_hosted_builds\" \"$num_CI_systems\"\ncat \"$tempfile\"\nprintf '\\n%s git repos summarized with %s CI builds (%s hosted, %s self-hosted) across %s different CI/CD systems\\n' \"$actual_repos\" \"$num_builds\" \"$num_hosted_builds\" \"$num_self_hosted_builds\" \"$num_CI_systems\"\n\ntrap '' exit\n"
  },
  {
    "path": "github/github_gpg_get_user_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-17 09:32:20 +0100 (Mon, 17 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/users#list-gpg-keys-for-a-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nFetches a GitHub user's public GPG key(s) via the GitHub API\n\nUser can be given as first argument, or environment variables \\$GITHUB_USER or \\$USER\n\n${0##*/} <user>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 1 ]; then\n    usage\nelif [ $# -eq 1 ]; then\n    user=\"$1\"\nelif [ -n \"${GITHUB_USER:-}\" ]; then\n    user=\"$GITHUB_USER\"\nelif [ -n \"${USER:-}\" ]; then\n    if [[ \"$USER\" =~ hari|sekhon ]]; then\n        user=harisekhon\n    else\n        user=\"$USER\"\n    fi\nelse\n    usage\nfi\n\necho \"# Fetching GPG Public Key(s) from GitHub for account:  $user\" >&2\necho \"#\" >&2\ncurl -sS --fail \"https://api.github.com/users/$user/gpg_keys\" |\njq -r '.[].public_key'\n"
  },
  {
    "path": "github/github_graph_commit_times_gnuplot.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-03 10:41:23 +0300 (Thu, 03 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\ncode=\"git_commit_times_all_repos.gnuplot\"\ndata=\"data/git_commit_times_all_repos.dat\"\nimage=\"images/git_commit_times_all_repos.png\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a GNUplot graph of Git commit times from all GitHub repos for a given user using the GitHub API\n\nGenerates the following files:\n\n    $code - Code\n\n    $data     - Data\n\n    $image     - Image\n\nA MermaidJS version of this program is adjacent in:\n\n    github_graph_commit_times_mermaidjs.sh\n\nA Golang version of this program can be found here:\n\n    https://github.com/HariSekhon/GitHub-Graph-Commit-Times\n\nFetching GitHub commits via the API is slow so if there is a data cache updated in the last 7 days then uses that\nto save time re-fetching the same data. Delete it if you want to refresh\n\nRequires GitHub CLI and GNUplot to be installed and GH_TOKEN to be present\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nusername=\"$1\"\n\nfor x in $code \\\n         $data \\\n         $image; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\nif file_modified_in_last_days \"$data\" 7; then\n    timestamp \"Using cached data: $data\"\nelse\n    timestamp \"Fetching list of GitHub repos\"\n    repos=\"$(get_github_repos \"$username\")\"\n    timestamp \"Found repos: $(wc -l <<< \"$repos\" | sed 's/[[:space:]]//g')\"\n    echo\n\n    while read -r repo; do\n        timestamp \"Fetching commit times for repo: $repo\"\n        gh api -H \"Accept: application/vnd.github.v3+json\" \"/repos/$username/$repo/commits\" --paginate |\n        jq -r '.[].commit.author.date[11:13]'\n    done <<< \"$repos\" |\n    sort |\n    uniq -c |\n    awk '{print $2\" \"$1}' > \"$data\"\nfi\necho\n\ntimestamp \"Generating GNUplot code for Commits per Hour\"\nsed '/^[[:space:]]*$/d' > \"$code\" <<EOF\n#\n# Generated by ${0##*/}\n#\n# from https://github.com/HariSekhon/DevOps-Bash-tools\n#\nset terminal pngcairo size 1280,720 enhanced font \"Arial,14\"\nset title \"Git Commits by Hour\"\nset xlabel \"Hour of Day (UTC)\"\nset ylabel \"Number of Commits\"\nset grid\n#set xtics rotate by -45\nset boxwidth 0.8 relative\nset style fill solid\nset datafile separator \" \"\n# results in X axis labels every 2 years\n#set xdata time\n#set timefmt \"%H\"\n#set format x \"%H\"\n# trick to get X axis labels for every year\nstats \"$data\" using 1 nooutput\nset xrange [STATS_min:STATS_max]\nset xtics 1\nset output \"$image\"\nplot \"$data\" using 1:2 with boxes title 'Commits'\nEOF\ntimestamp \"Generated GNUplot code: $code\"\n\ntimestamp \"Generating bar chart for Commits per Hour\"\ngnuplot \"$code\"\ntimestamp \"Generated bar chart image: $image\"\necho\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "github/github_graph_commit_times_mermaidjs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-03 10:41:23 +0300 (Thu, 03 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\ncode=\"git_commit_times_all_repos.mmd\"\ndata=\"data/git_commit_times_all_repos.dat\"\nimage=\"images/git_commit_times_all_repos.svg\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGraphs the GitHub commit times from all public GitHub repos for a given user\n\nFetches the commit data via the GitHub API and generates a bar chart using MermaidJS\n\nGenerates the MermaidJS code and then uses MermaidJS CLI to generate the image\n\n    $code - Code\n\n    $data - Data\n\n    $image - Image\n\nA GNUplot version of this program is adjacent in:\n\n    github_graph_commit_times_gnuplot.sh\n\nA Golang version of this program can be found here:\n\n    https://github.com/HariSekhon/GitHub-Graph-Commit-Times\n\nFetching GitHub commits via the API is slow so if there is a data cache updated in the last 7 days then uses that\nto save time re-fetching the same data. Delete it if you want to refresh\n\nRequires GitHub CLI and MermaidJS CLI to be installed and GH_TOKEN to be present\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_bin mmdc\n\nusername=\"$1\"\n\nfor x in $code \\\n         $data \\\n         $image; do\n    mkdir -p -v \"$(dirname \"$x\")\"\ndone\n\nif file_modified_in_last_days \"$data\" 7; then\n    timestamp \"Using cached data: $data\"\nelse\n    timestamp \"Fetching list of GitHub repos\"\n    repos=\"$(get_github_repos \"$username\")\"\n    timestamp \"Found repos: $(wc -l <<< \"$repos\" | sed 's/[[:space:]]//g')\"\n    echo\n\n    while read -r repo; do\n        timestamp \"Fetching commit times for repo: $repo\"\n        gh api -H \"Accept: application/vnd.github.v3+json\" \"/repos/$username/$repo/commits\" --paginate |\n        jq -r '.[].commit.author.date[11:13]'\n    done <<< \"$repos\" |\n    sort |\n    uniq -c |\n    awk '{print $2\" \"$1}' > \"$data\"\nfi\necho\n\nexport -f parse_file_col_to_csv\n\ntimestamp \"Generating MermaidJS code for Commits per Hour\"\ncat > \"$code\" <<EOF\nxychart-beta\n    title \"Git Commits by Hour\"\n    x-axis [ $(parse_file_col_to_csv \"$data\" 1) ]\n    y-axis \"Number of Commits\"\n    bar    [ $(parse_file_col_to_csv \"$data\" 2) ]\n    %%line [ $(parse_file_col_to_csv \"$data\" 2) ]\nEOF\ntimestamp \"Generated MermaidJS code: $code\"\necho\n\ntimestamp \"Generating MermaidJS bar chart for Commits per Hour\"\nmmdc -i \"$code\" -o \"$image\" -t dark --quiet # -b transparent\ntimestamp \"Generated MermaidJS image: $image\"\n\nif is_CI; then\n    exit 0\nfi\n\ntimestamp \"Opening: $image\"\n\"$srcdir/../media/imageopen.sh\" \"$image\"\n"
  },
  {
    "path": "github/github_install_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls a GitHub repo's release binary to \\$HOME/bin or /usr/local/bin, unpacking it from a tarball or zip if necessary\n\nIf the release file is a tarball or zip file then it'll auto-unpack it, but you must specify the path to the binary in the unpack\n\nIf version is not specified, determine the latest release and installs that\n\nIf the release URL title/path is more complicated than the convention of following the version number, such as is the case for Kustomize, then you'd need to call install_binary.sh with the URL path instead of using this script, see install/install_kustomize.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> <release_file_tarball_zip> [<version> <path/to/unpacked/binary> <install_path>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nowner_repo=\"$1\"\n\nrelease_file=\"$2\"\n\nversion=\"${3:-latest}\"\n\nbinary=\"${4:-}\"\n\ninstall_path=\"${5:-}\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nrelease_file=\"${release_file//\\{version\\}/${version#v}}\"\nbinary=\"${binary//\\{version\\}/${version#v}}\"\n\n\"$srcdir/../packages/install_binary.sh\" \"https://github.com/$owner_repo/releases/download/$version/$release_file\" ${binary:+\"$binary\"} ${install_path:+\"$install_path\"}\n"
  },
  {
    "path": "github/github_invitations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-22 17:43:43 +0000 (Tue, 22 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists / Accepts GitHub repo invitations\n\nAny arguments given are assumed to be invitation IDs or repositories and those are accepted.\nIf the argument 'all' is given, accepts all invitations.\n\nUseful to clear a bulk of invitations generated by automation code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<id> <id2> ... ]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfor (( page=1; page <= 100; page++)); do\n    if ! output=\"$(\"$srcdir/github_api.sh\" \"/user/repository_invitations?page=$page&per_page=100\")\"; then\n        echo \"ERROR\" >&2\n        exit 1\n    fi\n    jq_debug_pipe_dump <<< \"$output\" |\n    jq -r '.[] | select(.expired == false) | [.id, .repository.full_name] | @tsv'\n    github_result_has_more_pages \"$output\" || break\ndone |\nwhile read -r id owner_repo; do\n    if [ $# -gt 0 ]; then\n        for arg in \"$@\"; do\n            if [ \"$arg\" = \"all\" ] ||\n               [ \"$arg\" = \"$id\" ] ||\n               [ \"$arg\" = \"$owner_repo\" ]; then\n                timestamp \"accepting invitation '$id' for repo '$owner_repo'\"\n                \"$srcdir/github_api.sh\" \"/user/repository_invitations/$id\" -X PATCH\n                echo >&2\n                continue\n            fi\n        done\n    else\n        printf '%s\\t%s\\n' \"$id\" \"$owner_repo\"\n    fi\ndone\n"
  },
  {
    "path": "github/github_ip_ranges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: actions\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 14:57:23 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nip_services=(\n    actions\n    api\n    dependabot\n    git\n    hooks\n    importer\n    pages\n    web\n)\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns all the Atlassian IPs ranges in CIDR form\n\n    https://api.github.com/meta\n\nCan specify one of the following services, otherwise returns all:\n\n$(tr ' ' '\\n' <<< \"${ip_services[*]}\" | sed 's/^/    /')\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[service]\"\n\nhelp_usage \"$@\"\n\nip_service=\"${1:-}\"\n\nurl=\"https://api.github.com/meta\"\n\ndata=\"$(curl -sSf \"$url\")\"\n\nif [ \"$ip_service\" ]; then\n    jq -r \".${ip_service}[]\" <<< \"$data\"\nelse\n    for service in \"${ip_services[@]}\"; do\n        jq -r \".${service}[]\" <<< \"$data\"\n    done\nfi |\nsort -nu\n"
  },
  {
    "path": "github/github_merge_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-17 11:32:45 +0000 (Thu, 17 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMerges one branch into another in the current or given repo, creating and merging Pull Requests for full audit tracking all changes\n\nUseful to automate code promotion across environment branches or to automatically backport hotfixes from Production branch to dev / trunk branches\n\nAlso works across repo forks if the head branch contains an '<owner>:' prefix\n\nUseful Git terminology reminder:\n\nThe HEAD branch is the branch you want to merge FROM, eg. 'my-feature-branch'\nThe BASE branch is the branch you want to merge INTO, eg. 'master' or 'main'\n\nRequires GitHub CLI to be installed and configured\n\nDepends on adjacent script:\n\n    github_pull_request_create.sh\n\nUsed by adjacent script:\n\n    github_repo_fork_update.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo>] <from_head_branch> <to_base_branch>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\nmax_args 3 \"$@\"\n\noutput=\"$(\"$srcdir/github_pull_request_create.sh\" \"$@\")\"\n\nif [ -n \"$output\" ]; then\n    url=\"$(parse_pull_request_url \"$output\")\"\n    timestamp \"Merging Pull Request:  $url\"\n    gh pr merge --merge \"$url\"\n    echo >&2\nfi\n"
  },
  {
    "path": "github/github_mirror_repos_to_aws_codecommit.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-29 19:19:43 +0100 (Tue, 29 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# gets absolute rather than relative path, for when we pushd later, otherwise relative $srcdir references will break\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMirrors all or given repos from GitHub to AWS CodeCommit via AWS CLI and Git HTTPS mirror clones\n\nUseful to create/sync GitHub repos to AWS CodeCommit for migration or to cron for fast almost free DR purposes\n(almost \\$0 AWS charges compared to \\$100-\\$400+ per month for Rewind / BackHub)\n\nIncludes repo descriptions and all branches and tags, but not PRs/Wikis/Releases\n\nEspecially useful to backup dynamic environments where people are adding new repos all the time, avoids having to maintain configurations as finds and iterates all non-fork repos by default\nCan't use Terraform to dynamically create these backups because a simple commented/deleted code mistake would bypass prevent_destroy and delete your backup repos as well as your originals!\n\n    https://github.com/hashicorp/terraform/issues/17599\n\nCron this script as per your preferred backup schedule\n\nIf no repos are given, iterates all non-fork repos for the current user or GitHub organization\n\nEach repo will have the same name in AWS as it does on GitHub\n\nFor source GitHub accounts, requires:\n\n    - \\$GITHUB_TOKEN\n    - \\$GITHUB_ORGANIZATION, \\$GITHUB_USER or else infers owner of the \\$GITHUB_TOKEN\n\nFor AWS CodeCommit requires:\n\n    - \\$AWS_DEFAULT_REGION\n    - AWS Credentials:\n      - AWS CLI configured with CodeCommit full access to create repositories (\\$AWS_PROFILE, \\$AWS_ACCESS_KEY_ID, \\$AWS_SECRET_ACCESS_KEY etc.)\n      - \\$AWS_GIT_USER and \\$AWS_GIT_PASSWORD\n          or\n      - Python Pip git-remote-codecommit module to be installed to use AWS CLI credentials\n\nIn a GitHub Organization, only repos the user can read will be mirrored, others won't be returned in the list of GitHub repos to even try (as an outside collaborator user)\n\n\nIf \\$CLEAR_CACHE=true, deletes the /tmp cache and uses a fresh clone mirror. This can sometimes clear push errors.\n\nIf \\$FORCE_MIRROR=true, runs a mirror operation (overwrites refs and deletes removed branches). Not the default for safety.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<repo1> <repo2> <repo3> ...]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\ncheck_env_defined \"AWS_DEFAULT_REGION\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntimestamp \"Starting GitHub to AWS CodeCommit mirroring\"\necho >&2\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nowner=\"${GITHUB_ORGANIZATION:-$user}\"\n\nif is_blank \"$owner\"; then\n    die \"Failed to determine GitHub owner\"\nfi\n\nif [ $# -gt 0 ]; then\n    repos=\"$*\"\nelse\n    timestamp \"Getting list of all non-fork GitHub repos owned by '$owner'\"\n    repos=\"$(get_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\")\"\n    echo >&2\nfi\n\n# not using mktemp because we want to reuse this staging area between runs for efficiency\ntmpdir=\"/tmp/github_mirror_to_aws_codecommmit/$owner\"\n\nif [ \"${CLEAR_CACHE:-}\" = true ]; then\n    timestamp \"Removing cache: $tmpdir\"\n    rm -fr -- \"$tmpdir\"\nfi\n\ntimestamp \"Switching to '$tmpdir' directory for mirror staging\"\nmkdir -p -v \"$tmpdir\"\ncd \"$tmpdir\"\necho >&2\n\nsucceeded=0\nfailed=0\n\nmirror_repo(){\n    local repo=\"$1\"\n    local description\n    # in case we need to mutate the names later, such as working around dots in repo names eg. \".github\"\n    local aws_repo=\"$repo\"\n\n    timestamp \"Checking AWS repo '$aws_repo' exists\"\n    if ! aws codecommit list-repositories | jq -r '.repositories[].repositoryName' | grep -Fxq \"$aws_repo\" >/dev/null; then\n        timestamp \"Creating AWS repo '$aws_repo'\"\n        aws codecommit create-repository --repository-name \"$aws_repo\" || return 1\n        echo >&2\n    fi\n\n    timestamp \"Checking GitHub repo for description to copy\"\n    description=\"$(\"$srcdir/github_repo_description.sh\" \"$owner/$repo\" | sed \"s/^${repo}[[:space:]]*//\")\"\n    if [ -n \"$description\" ]; then\n        timestamp \"Setting AWS repo '$aws_repo' description to '$description'\"\n        aws codecommit update-repository-description --repository-name \"$aws_repo\" --repository-description \"$description\"\n    fi\n\n    if [ -d \"$repo.git\" ]; then\n        timestamp \"Using existing clone in directory '$repo.git'\"\n        pushd \"$repo.git\" >/dev/null || return 1\n        git remote update origin || return 1\n    else\n        timestamp \"Cloning GitHub repo to directory '$repo.git'\"\n        git clone --mirror \"https://$user:$GITHUB_TOKEN@github.com/$owner/$repo.git\" || return 1\n        pushd \"$repo.git\" >/dev/null || return 1\n    fi\n\n    if ! git remote -v | awk '{print $1}' | grep -Fxq aws; then\n        timestamp \"Adding AWS remote origin\"\n        if [ -n \"${AWS_GIT_USER:-}\" ] &&\n           [ -n \"${AWS_GIT_PASSWORD:-}\" ]; then\n            timestamp \"Using AWS git user and url encoded password\"\n            AWS_GIT_PASSWORD_URLENCODED=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$AWS_GIT_PASSWORD\")\"\n            git remote add aws \"https://$AWS_GIT_USER:$AWS_GIT_PASSWORD_URLENCODED@git-codecommit.$AWS_DEFAULT_REGION.amazonaws.com/v1/repos/$repo\"\n        else\n            timestamp \"Using AWS credentials via git-remote-codecommit\"\n            git remote add aws \"codecommit::$AWS_DEFAULT_REGION://$repo\"\n        fi\n    fi\n\n    if [ \"${FORCE_MIRROR:-}\" = true ]; then\n        # more dangerous, force overwrites remote repo refs\n        timestamp \"Force mirroring to AWS CodeCommit (overwrite)\"\n        git push --mirror aws || return 1\n    else\n        timestamp \"Pushing all branches to AWS CodeCommit\"\n        git push --all aws || return 1  # XXX: without return 1 the function ignores errors, even with set -e inside the function\n\n        timestamp \"Pushing all tags to AWS CodeCommit\"\n        git push --tags aws || return 1\n    fi\n\n    # TODO: if AWS CodeCommit supports protected branches in future\n    #timestamp \"Enabling branch protections on AWS mirror repo '$aws_repo'\"\n    #\"$srcdir/aws_codecommit_protect_branches.sh\" \"$aws_repo\"\n\n    timestamp \"Getting GitHub repo '$repo' default branch\"\n    local default_branch\n    default_branch=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner/$repo\" | jq -r '.default_branch')\"\n    timestamp \"Setting AWS CodeCommit repo '$aws_repo' default branch to '$default_branch'\"\n    aws codecommit update-default-branch --repository-name \"$aws_repo\" --default-branch-name \"$default_branch\"\n\n    popd >/dev/null || return 1\n    echo >&2\n    ((succeeded+=1))\n}\n\nfailed_repos=\"\"\n\nfor repo in $repos; do\n    if [[ \"$repo\" =~ / ]]; then\n        die \"Repo '$repo' should be specified without owner prefix\"\n    fi\n    if ! mirror_repo \"$repo\"; then\n        popd >/dev/null || :\n        timestamp \"Mirroring failed, clearing cache and trying again\"\n        rm -fr -- \"$tmpdir/$repo.git\"\n        if ! mirror_repo \"$repo\"; then\n            echo >&2\n            timestamp \"ERROR: Failed to mirror repo '$repo' to AWS\"\n            failed_repos+=\" $repo\"\n            echo >&2\n            ((failed+=1))\n        fi\n    fi\ndone\n\nif [ $failed -gt 0 ]; then\n    timestamp \"ERROR: $failed GitHub repos failed to mirror to AWS ($succeeded succeeded). Failed repos: $failed_repos\"\n    exit 1\nfi\n\ntimestamp \"GitHub to AWS mirroring completed successfully for $succeeded repos\"\n"
  },
  {
    "path": "github/github_mirror_repos_to_bitbucket.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-22 10:47:11 +0000 (Tue, 22 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# gets absolute rather than relative path, for when we pushd later, otherwise relative $srcdir references will break\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMirrors all or given repos from GitHub to BitBucket via APIs and HTTPS mirror clones\n\nUseful to create/sync GitHub repos to BitBucket for migration or to cron for fast free DR purposes\n\nIncludes repo descriptions and all branches and tags, but not PRs/Wikis/Releases\n\nEspecially useful to backup dynamic environments where people are adding new repos all the time, avoids having to maintain configurations as finds and iterates all non-fork repos by default\nCan't use Terraform to dynamically create these backups because a simple commented/deleted code mistake would bypass prevent_destroy and delete your backup repos as well as your originals!\n\n    https://github.com/hashicorp/terraform/issues/17599\n\nCron this script as per your preferred backup schedule\n\nIf no repos are given, iterates all non-fork repos for the current user or GitHub organization\n\nEach repo will have the same name in BitBucket as it does on GitHub\nBitBucket workspace is assumed to be the same as the BitBucket user name (set \\$BITBUCKET_WORKSPACE to override)\n\nRequires \\$GITHUB_TOKEN AND \\$BITBUCKET_TOKEN to be set\n\nIn a GitHub Organization, only repos the user can read will be mirrored, others won't be returned in the list of GitHub repos to even try (as an outside collaborator user)\n\nSource GitHub and Destination BitBucket accounts, in order or priority:\n\n    \\$GITHUB_ORGANIZATION, \\$GITHUB_USER or owner of the \\$GITHUB_TOKEN\n    \\$BITBUCKET_OWNER, \\$BITBUCKET_USER or the owner of the \\$BITBUCKET_TOKEN\n    \\$BITBUCKET_WORKSPACE - the container where the repositories are created, can auto-determine if there is only one workspace owned by the \\$BITBUCKET_USER\n\nIf \\$CLEAR_CACHE=true, deletes the /tmp cache and uses a fresh clone mirror. This can sometimes clear push errors.\n\nIf \\$FORCE_MIRROR=true, runs a mirror operation (overwrites refs and deletes removed branches). Not the default for safety.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<repo1> <repo2> <repo3> ...]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\ncheck_env_defined \"BITBUCKET_TOKEN\"\n#check_env_defined \"BITBUCKET_WORKSPACE\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntimestamp \"Starting GitHub to BitBucket mirroring\"\necho >&2\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nowner=\"${GITHUB_ORGANIZATION:-$user}\"\nbitbucket_owner=\"${BITBUCKET_OWNER:-${BITBUCKET_USER:-$(\"$srcdir/../bitbucket/bitbucket_api.sh\" /user | jq -r .username)}}\"\nbitbucket_workspace=\"${BITBUCKET_WORKSPACE:-}\"\n\nif is_blank \"$owner\"; then\n    die \"Failed to determine GitHub owner\"\nfi\nif is_blank \"$bitbucket_owner\"; then\n    die \"Failed to determine BitBucket owner\"\nfi\n\nif is_blank \"$bitbucket_workspace\"; then\n    timestamp \"Attempting to auto-determine BitBucket workspace from user if only one exists\"\n    bitbucket_workspace=\"$(\"$srcdir/../bitbucket/bitbucket_api.sh\" '/user/permissions/workspaces?q=permission=\"owner\"' | jq -r '.values[].workspace.slug')\"\n    if is_blank \"$bitbucket_workspace\"; then\n        die \"Failed to determine BitBucket workspace\"\n    fi\n    timestamp \"Determined BitBucket workspace to be '$bitbucket_workspace'\"\n    echo >&2\nfi\n\nif [ $# -gt 0 ]; then\n    repos=\"$*\"\nelse\n    timestamp \"Getting list of all non-fork GitHub repos owned by '$owner'\"\n    repos=\"$(get_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\")\"\n    echo >&2\nfi\n\n# not using mktemp because we want to reuse this staging area between runs for efficiency\ntmpdir=\"/tmp/github_mirror_to_bitbucket/$owner\"\n\nif [ \"${CLEAR_CACHE:-}\" = true ]; then\n    timestamp \"Removing cache: $tmpdir\"\n    rm -fr -- \"$tmpdir\"\nfi\n\ntimestamp \"Switching to '$tmpdir' directory for mirror staging\"\nmkdir -p -v \"$tmpdir\"\ncd \"$tmpdir\"\necho >&2\n\nsucceeded=0\nfailed=0\n\nmirror_repo(){\n    local repo=\"$1\"\n    # BitBucket doesn't allow repo name like .github, only alnum, dashes and underscores, and not starting with unusual characters either\n    bitbucket_repo=\"$repo\"\n    bitbucket_owner_repo=\"$bitbucket_owner/$bitbucket_repo\"\n\n    timestamp \"Checking BitBucket repo '$bitbucket_owner/$bitbucket_repo' exists\"\n    if ! \"$srcdir/../bitbucket/bitbucket_api.sh\" \"/repositories/$bitbucket_owner_repo\" >/dev/null; then\n        timestamp \"Creating BitBucket repo '$bitbucket_owner/$bitbucket_repo'\"\n        \"$srcdir/../bitbucket/bitbucket_api.sh\" \"/repositories/$bitbucket_owner_repo\" -X POST -d \"{ \\\"scm\\\": \\\"git\\\", \\\"key\\\": \\\"$bitbucket_workspace\\\" }\" >/dev/null || return 1\n        echo >&2\n    fi\n\n    timestamp \"Checking GitHub repo for description to copy\"\n    \"$srcdir/github_repo_description.sh\" \"$owner/$repo\" |\n    sed \"s/^$repo/$bitbucket_repo/\" |\n    # timestamp not needed here as bitbucket_project_set_description.sh will output if it is setting the repo description\n    \"$srcdir/../bitbucket/bitbucket_repo_set_description.sh\"\n\n    if [ -d \"$repo.git\" ]; then\n        timestamp \"Using existing clone in directory '$repo.git'\"\n        pushd \"$repo.git\" >/dev/null || return 1\n        git remote update origin || return 1\n    else\n        timestamp \"Cloning GitHub repo to directory '$repo.git'\"\n        git clone --mirror \"https://$user:$GITHUB_TOKEN@github.com/$owner/$repo.git\" || return 1\n        pushd \"$repo.git\" >/dev/null || return 1\n    fi\n\n    if ! git remote -v | awk '{print $1}' | grep -Fxq bitbucket; then\n        timestamp \"Adding BitBucket remote origin\"\n        git remote add bitbucket \"https://$bitbucket_owner:$BITBUCKET_TOKEN@bitbucket.org/$bitbucket_owner/$bitbucket_repo.git\"\n    fi\n\n    if [ \"${FORCE_MIRROR:-}\" = true ]; then\n        # more dangerous, force overwrites remote repo refs\n        timestamp \"Force mirroring to Bitbucket (overwrite)\"\n        git push --mirror bitbucket || return 1\n    else\n        timestamp \"Pushing all branches to BitBucket\"\n        git push --all bitbucket || return 1  # XXX: without return 1 the function ignores errors, even with set -e inside the function\n\n        timestamp \"Pushing all tags to BitBucket\"\n        git push --tags bitbucket || return 1\n    fi\n\n    #timestamp \"Getting GitHub repo '$repo' default branch\"\n    #local default_branch\n    #default_branch=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner/$repo\" | jq -r '.default_branch')\"\n    #timestamp \"Setting BitBucket repo '$aws_repo' default branch to '$default_branch'\"\n    #\n    # XXX: returns 200 OK but doesn't change the default branch - might be bug in BitBucket API\n    #\n    #       https://community.atlassian.com/t5/Bitbucket-questions/Setting-the-default-branch-with-REST-API/qaq-p/1336362#U1989031\n    #\n    #\"$srcdir/../bitbucket/bitbucket_api.sh\" \"/repositories/$bitbucket_owner/$bitbucket_repo\" -X PUT -d '{\"mainbranch\": { \"name\": \"'\"$default_branch\"'\", \"type\": \"branch\" } }' >/dev/null\n\n    popd >/dev/null || return 1\n    echo >&2\n    ((succeeded+=1))\n}\n\nfailed_repos=\"\"\n\nfor repo in $repos; do\n    if [[ \"$repo\" =~ / ]]; then\n        die \"Repo '$repo' should be specified without owner prefix\"\n    fi\n    if ! mirror_repo \"$repo\"; then\n        popd >/dev/null || :\n        timestamp \"Mirroring failed, clearing cache and trying again\"\n        rm -fr -- \"$tmpdir/$repo.git\"\n        if ! mirror_repo \"$repo\"; then\n            echo >&2\n            timestamp \"ERROR: Failed to mirror repo '$repo' to BitBucket\"\n            failed_repos+=\" $repo\"\n            echo >&2\n            ((failed+=1))\n        fi\n    fi\ndone\n\nif [ $failed -gt 0 ]; then\n    timestamp \"ERROR: $failed GitHub repos failed to mirror to BitBucket ($succeeded succeeded). Failed repos: $failed_repos\"\n    exit 1\nfi\n\ntimestamp \"GitHub to BitBucket mirroring completed successfully for $succeeded repos\"\n"
  },
  {
    "path": "github/github_mirror_repos_to_gcp_source_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-31 11:06:02 +0100 (Thu, 31 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# gets absolute rather than relative path, for when we pushd later, otherwise relative $srcdir references will break\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMirrors all or given repos from GitHub to GCP Source Repos via GCloud SDK and Git HTTPS mirror clones\n\nUseful to create/sync GitHub repos to GCP Source Repos for migration or to cron for fast almost free DR purposes\n(almost \\$0 GCP charges compared to \\$100-\\$400+ per month for Rewind / BackHub)\n\nIncludes all branches and tags, but not description/PRs/Wikis/Releases\n\nEspecially useful to backup dynamic environments where people are adding new repos all the time, avoids having to maintain configurations as finds and iterates all non-fork repos by default\nCan't use Terraform to dynamically create these backups because a simple commented/deleted code mistake would bypass prevent_destroy and delete your backup repos as well as your originals!\n\n    https://github.com/hashicorp/terraform/issues/17599\n\nCron this script as per your preferred backup schedule\n\nUnfortunately GCloud SDK doesn't currently support configuring GCP's GitHub automatic mirroring, which is only available through the UI Console, making it unsuitable for automation\nCloud Source Repos are extremely rudimentary with no features and only really suitable for mirror trigger repos or as backups, and can't even configure a different default branch\n\nIf no repos are given, iterates all non-fork repos for the current user or GitHub organization\n\nEach repo will have the same name in GCP Source Repos to avoid clashing with manually configured GCP automatically mirroring functionality which creates repos in the format 'github_{owner}_{repo}'\nAny non-letter leading characters and non alphanumeric/dots/dashs/underscores will be removed to meet GCP Source Repos naming conventions eg. '.github' -> 'github' repo\n\nFor source GitHub accounts, requires:\n\n    - \\$GITHUB_TOKEN\n    - \\$GITHUB_ORGANIZATION, \\$GITHUB_USER or else infers owner of the \\$GITHUB_TOKEN\n\nFor GCP Source Repos requires:\n\n    - \\$CLOUDSDK_CORE_PROJECT\n    - GCloud SDK installed and authenticated\n\nIn a GitHub Organization, only repos the user can read will be mirrored, others won't be returned in the list of GitHub repos to even try (as an outside collaborator user)\n\n\nIf \\$CLEAR_CACHE=true, deletes the /tmp cache and uses a fresh clone mirror. This can sometimes clear push errors.\n\nIf \\$FORCE_MIRROR=true, runs a mirror operation (overwrites refs and deletes removed branches). Not the default for safety.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<repo1> <repo2> <repo3> ...]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\ncheck_env_defined \"CLOUDSDK_CORE_PROJECT\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntimestamp \"Starting GitHub to GCP Source Repos mirroring\"\necho >&2\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nowner=\"${GITHUB_ORGANIZATION:-$user}\"\n\nif is_blank \"$owner\"; then\n    die \"Failed to determine GitHub owner\"\nfi\n\nif [ $# -gt 0 ]; then\n    repos=\"$*\"\nelse\n    timestamp \"Getting list of all non-fork GitHub repos owned by '$owner'\"\n    repos=\"$(get_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\")\"\n    echo >&2\nfi\n\n# not using mktemp because we want to reuse this staging area between runs for efficiency\ntmpdir=\"/tmp/github_mirror_to_gcp_source_repos/$owner\"\n\nif [ \"${CLEAR_CACHE:-}\" = true ]; then\n    timestamp \"Removing cache: $tmpdir\"\n    rm -fr -- \"$tmpdir\"\nfi\n\ntimestamp \"Switching to '$tmpdir' directory for mirror staging\"\nmkdir -p -v \"$tmpdir\"\ncd \"$tmpdir\"\necho >&2\n\nsucceeded=0\nfailed=0\n\naccount=\"$(gcloud config list --format='get(core.account)')\"\n\nmirror_repo(){\n    local repo=\"$1\"\n    #local description\n    # XXX: same naming convention of GCP's GitHub mirroring so the two mechanisms can interoperate\n    #      must be lowercase because GSR is case sensitive and mirroring will create a separate repo otherwise\n    #local gcp_repo=\"github_${owner}_$repo\"\n    #local gcp_repo=\"$(tr '[:upper:]' '[:lower:]' <<< \"$gcp_repo\")\"\n    local gcp_repo=\"$repo\"\n    gcp_repo=\"$(sed 's/^[^[:alpha:]]*//; s/[^[:alnum:]._-]//' <<< \"$gcp_repo\")\"\n\n    timestamp \"Checking GCP repo '$gcp_repo' exists\"\n    if ! gcloud source repos list --format='value(name)' | grep -Fxq \"$gcp_repo\" >/dev/null; then\n        timestamp \"Creating GCP source repo '$gcp_repo'\"\n        gcloud source repos create \"$gcp_repo\" || return 1\n        echo >&2\n    fi\n\n    # XXX: GCP Source Repos don't support descriptions yet\n    #timestamp \"Checking GitHub repo for description to copy\"\n    #description=\"$(\"$srcdir/github_repo_description.sh\" \"$owner/$repo\" | sed \"s/^${repo}[[:space:]]*//\")\"\n    #if [ -n \"$description\" ]; then\n    #    timestamp \"Setting GCP repo '$gcp_repo' description to '$description'\"\n    #    # gcloud source repos update --description ... no such command yet\n    #fi\n\n    if [ -d \"$repo.git\" ]; then\n        timestamp \"Using existing clone in directory '$repo.git'\"\n        pushd \"$repo.git\" >/dev/null || return 1\n        git remote update origin || return 1\n    else\n        timestamp \"Cloning GitHub repo to directory '$repo.git'\"\n        git clone --mirror \"https://$user:$GITHUB_TOKEN@github.com/$owner/$repo.git\" || return 1\n        pushd \"$repo.git\" >/dev/null || return 1\n    fi\n\n    if ! git remote -v | awk '{print $1}' | grep -Fxq gcp; then\n        timestamp \"Adding GCP remote origin\"\n        git remote add gcp \"https://source.developers.google.com/p/$CLOUDSDK_CORE_PROJECT/r/$gcp_repo\"\n        # having a blank helper before the real help prevents:\n        # bad input: ..........\n        git config --replace-all credential.https://source.developers.google.com/.helper ''\n        git config --add         credential.https://source.developers.google.com/.helper \"!gcloud auth git-helper --account=$account --ignore-unknown \\$@\"\n    fi\n\n    if [ \"${FORCE_MIRROR:-}\" = true ]; then\n        # more dangerous, force overwrites remote repo refs\n        timestamp \"Force mirroring to GCP Source Repo (overwrite)\"\n        git push --mirror gcp || return 1\n    else\n        timestamp \"Pushing all branches to GCP Source Repo\"\n        git push --all gcp || return 1  # XXX: without return 1 the function ignores errors, even with set -e inside the function\n\n        timestamp \"Pushing all tags to GCP Source Repo\"\n        git push --tags gcp || return 1\n    fi\n\n    # TODO: if GCP Source Repos supports protected branches in future\n    #timestamp \"Enabling branch protections on GCP mirror repo '$gcp_repo'\"\n    #\"$srcdir/gcp_source_repo_protect_branches.sh\" \"$gcp_repo\"\n\n    # XXX: GCloud SDK is too rudimentary, doesn't support setting default branch\n    #timestamp \"Getting GitHub repo '$repo' default branch\"\n    #local default_branch\n    #default_branch=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner/$repo\" | jq -r '.default_branch')\"\n    #timestamp \"Setting GCP Source Repo '$gcp_repo' default branch to '$default_branch'\"\n    # # gcloud source repos update  ... so default branch switch\n\n    popd >/dev/null || return 1\n    echo >&2\n    ((succeeded+=1))\n}\n\nfailed_repos=\"\"\n\nfor repo in $repos; do\n    if [[ \"$repo\" =~ / ]]; then\n        die \"Repo '$repo' should be specified without owner prefix\"\n    fi\n    if ! mirror_repo \"$repo\"; then\n        popd >/dev/null || :\n        timestamp \"Mirroring failed, clearing cache and trying again\"\n        rm -fr -- \"$tmpdir/$repo.git\"\n        if ! mirror_repo \"$repo\"; then\n            echo >&2\n            timestamp \"ERROR: Failed to mirror repo '$repo' to GCP\"\n            failed_repos+=\" $repo\"\n            echo >&2\n            ((failed+=1))\n        fi\n    fi\ndone\n\nif [ $failed -gt 0 ]; then\n    timestamp \"ERROR: $failed GitHub repos failed to mirror to GCP ($succeeded succeeded). Failed repos: $failed_repos\"\n    exit 1\nfi\n\ntimestamp \"GitHub to GCP mirroring completed successfully for $succeeded repos\"\n"
  },
  {
    "path": "github/github_mirror_repos_to_gitlab.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-22 10:47:11 +0000 (Tue, 22 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# gets absolute rather than relative path, for when we pushd later, otherwise relative $srcdir references will break\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nMirrors all or given repos from GitHub to GitLab via APIs and HTTPS mirror clones\n\nUseful to create/sync GitHub repos to GitLab for migration or to cron for fast free DR purposes\n\nIncludes repo descriptions and all branches and tags, but not PRs/Wikis/Releases\n\nEspecially useful to backup dynamic environments where people are adding new repos all the time, avoids having to maintain configurations as finds and iterates all non-fork repos by default\nCan't use Terraform to dynamically create these backups because a simple commented/deleted code mistake would bypass prevent_destroy and delete your backup repos as well as your originals!\n\n    https://github.com/hashicorp/terraform/issues/17599\n\nCron this script as per your preferred backup schedule\n\nIf no repos are given, iterates all non-fork repos for the current user or GitHub organization\n\nEach repo will have the same name in GitLab as it does on GitHub, but characters other than alphanumeric/dash/underscores will be replaced by underscore,\nand any leading special characters will be removed to meet GitLab's repo naming requirements eg. a repo called '.test' on GitHub will mirrored to just 'test' on GitLab\n\nRequires \\$GITHUB_TOKEN AND \\$GITLAB_TOKEN to be set\n\nIn a GitHub Organization, only repos the user can read will be mirrored, others won't be returned in the list of GitHub repos to even try (as an outside collaborator user)\n\nSource GitHub and Destination GitLab accounts, in order or priority:\n\n    \\$GITHUB_ORGANIZATION, \\$GITHUB_USER or owner of the \\$GITHUB_TOKEN\n    \\$GITLAB_OWNER, \\$GITLAB_USER or the owner of the \\$GITLAB_TOKEN\n\nIf \\$CLEAR_CACHE=true, deletes the /tmp cache and uses a fresh clone mirror. This can sometimes clear push errors.\n\nIf \\$FORCE_MIRROR=true, runs a mirror operation (overwrites refs and deletes removed branches). Not the default for safety.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<repo1> <repo2> <repo3> ...]\"\n\ncheck_env_defined \"GITHUB_TOKEN\"\ncheck_env_defined \"GITLAB_TOKEN\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntimestamp \"Starting GitHub to GitLab mirroring\"\necho >&2\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nowner=\"${GITHUB_ORGANIZATION:-$user}\"\ngitlab_owner=\"${GITLAB_OWNER:-${GITLAB_USER:-$(\"$srcdir/../gitlab/gitlab_api.sh\" /user | jq -r .username)}}\"\n\nif is_blank \"$owner\"; then\n    die \"Failed to determine GitHub owner\"\nfi\nif is_blank \"$gitlab_owner\"; then\n    die \"Failed to determine GitLab owner\"\nfi\n\n#timestamp \"Getting GitLab id in case we need to create any repos in GitLab\"\n#gitlab_id=\"$(\"$srcdir/../gitlab/gitlab_api.sh\" \"/users?username=$gitlab_owner\" | jq -r '.[0].id')\"\n#echo >&2\n#if is_blank \"$gitlab_id\"; then\n#    die \"Failed to determine GitLab id\"\n#fi\n\nif [ $# -gt 0 ]; then\n    repos=\"$*\"\nelse\n    timestamp \"Getting list of all non-fork GitHub repos owned by '$owner'\"\n    repos=\"$(get_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\")\"\n    echo >&2\nfi\n\n# not using mktemp because we want to reuse this staging area between runs for efficiency\ntmpdir=\"/tmp/github_mirror_to_gitlab/$owner\"\n\nif [ \"${CLEAR_CACHE:-}\" = true ]; then\n    timestamp \"Removing cache: $tmpdir\"\n    rm -fr -- \"$tmpdir\"\nfi\n\ntimestamp \"Switching to '$tmpdir' directory for mirror staging\"\nmkdir -p -v \"$tmpdir\"\ncd \"$tmpdir\"\necho >&2\n\nsucceeded=0\nfailed=0\n\nmirror_repo(){\n    local repo=\"$1\"\n    # GitLab doesn't allow repo name like .github, only alnum, dashes and underscores, and not starting with unusual characters either\n    gitlab_repo=\"$(sed 's/[^[:alnum:]_-]/_/g; s/^[^[:alnum:]]*//' <<< \"$repo\")\"\n    gitlab_owner_repo=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$gitlab_owner/$gitlab_repo\")\"\n\n    timestamp \"Checking GitLab repo '$gitlab_owner/$gitlab_repo' exists\"\n    if ! \"$srcdir/../gitlab/gitlab_api.sh\" \"/projects/$gitlab_owner_repo\" >/dev/null; then\n        timestamp \"Creating GitLab repo '$gitlab_owner/$gitlab_repo'\"\n        # only available for admins\n        #\"$srcdir/../gitlab/gitlab_api.sh\" \"/projects/user/$gitlab_id\" -X POST -d \"{ \\\"name\\\": \\\"$gitlab_repo\\\", \\\"visibility\\\": \\\"private\\\" }\" >/dev/null\n        \"$srcdir/../gitlab/gitlab_api.sh\" \"/projects\" -X POST -d \"{ \\\"name\\\": \\\"$gitlab_repo\\\", \\\"visibility\\\": \\\"private\\\" }\" >/dev/null || return 1\n        echo >&2\n    fi\n\n    timestamp \"Checking GitHub repo for description to copy\"\n    \"$srcdir/github_repo_description.sh\" \"$owner/$repo\" |\n    sed \"s/^$repo/$gitlab_repo/\" |\n    # timestamp not needed here as gitlab_project_set_description.sh will output if it is setting the repo description\n    \"$srcdir/../gitlab/gitlab_project_set_description.sh\"\n\n    if [ -d \"$repo.git\" ]; then\n        timestamp \"Using existing clone in directory '$repo.git'\"\n        pushd \"$repo.git\" >/dev/null || return 1\n        git remote update origin || return 1\n    else\n        timestamp \"Cloning GitHub repo to directory '$repo.git'\"\n        git clone --mirror \"https://$user:$GITHUB_TOKEN@github.com/$owner/$repo.git\" || return 1\n        pushd \"$repo.git\" >/dev/null || return 1\n    fi\n\n    if ! git remote -v | awk '{print $1}' | grep -Fxq gitlab; then\n        timestamp \"Adding GitLab remote origin\"\n        git remote add gitlab \"https://$gitlab_owner:$GITLAB_TOKEN@gitlab.com/$gitlab_owner/$gitlab_repo.git\"\n    fi\n\n    if [ \"${FORCE_MIRROR:-}\" = true ]; then\n        # more dangerous, force overwrites remote repo refs\n        timestamp \"Force mirroring to GitLab (overwrite)\"\n        git push --mirror gitlab || return 1\n    else\n        timestamp \"Pushing all branches to GitLab\"\n        git push --all gitlab || return 1  # XXX: without return 1 the function ignores errors, even with set -e inside the function\n\n        timestamp \"Pushing all tags to GitLab\"\n        git push --tags gitlab || return 1\n    fi\n\n    timestamp \"Enabling branch protections on GitLab mirror repo '$gitlab_owner/$gitlab_repo'\"\n    \"$srcdir/../gitlab/gitlab_project_protect_branches.sh\" \"$gitlab_owner/$gitlab_repo\"\n\n    timestamp \"Getting GitHub repo '$repo' default branch\"\n    local default_branch\n    default_branch=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner/$repo\" | jq -r '.default_branch')\"\n    timestamp \"Setting GitLab repo '$gitlab_owner/$gitlab_repo' default branch to '$default_branch'\"\n    \"$srcdir/../gitlab/gitlab_api.sh\" \"/projects/$gitlab_owner_repo\" -X PUT -d '{\"default_branch\": \"'\"$default_branch\"'\"}' >/dev/null\n\n    popd >/dev/null || return 1\n    echo >&2\n    ((succeeded+=1))\n}\n\nfailed_repos=\"\"\n\nfor repo in $repos; do\n    if [[ \"$repo\" =~ / ]]; then\n        die \"Repo '$repo' should be specified without owner prefix\"\n    fi\n    if ! mirror_repo \"$repo\"; then\n        popd >/dev/null || :\n        timestamp \"Mirroring failed, clearing cache and trying again\"\n        rm -fr -- \"$tmpdir/$repo.git\"\n        if ! mirror_repo \"$repo\"; then\n            echo >&2\n            timestamp \"ERROR: Failed to mirror repo '$repo' to GitLab\"\n            failed_repos+=\" $repo\"\n            echo >&2\n            ((failed+=1))\n        fi\n    fi\ndone\n\nif [ $failed -gt 0 ]; then\n    timestamp \"ERROR: $failed GitHub repos failed to mirror to GitLab ($succeeded succeeded). Failed repos: $failed_repos\"\n    exit 1\nfi\n\ntimestamp \"GitHub to GitLab mirroring completed successfully for $succeeded repos\"\n"
  },
  {
    "path": "github/github_public_lines_of_code.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-13 23:48:38 +0700 (Mon, 13 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks out all public original source GitHub repos for the current or given user\nand then counts all lines of code for them with breakdowns of languages, files,\ncode, comments and blanks\n\nUses GitHub CLI and cloc - attempts to install them if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nowner=\"${1:-}\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nif ! type -P gh &>/dev/null; then\n    timestamp \"GitHub CLI not found, attempting to install...\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" gh ||\n    \"$srcdir/../install/install_github_cli.sh\"\n    echo\nfi\n\nif ! type -P cloc &>/dev/null; then\n    timestamp \"cloc not found, attempting to install...\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" cloc\n    echo\nfi\n\ntimestamp \"Getting list of public original GitHub repo urls\"\ngithub_repo_urls=\"$(get_github_repo_urls ${owner:+\"$owner\"} --visibility public --source)\"\nnum_github_repo_urls=\"$(wc -l <<< \"$github_repo_urls\" | sed 's/[[:space:]]*//g')\"\ntimestamp \"Found $num_github_repo_urls GitHub repos\"\necho\n\n# not using /tmp and mktemp because want to cache long term for performance\n# and clean checkouts are useful for all sorts of things and will be reused by other scripts\nbasedir=\"$HOME/github/clean_checkouts\"\n\nmkdir -p -v \"$basedir\"\n\ntimestamp \"Switching to dir: $basedir\"\ncd \"$basedir\"\necho\n\ntimestamp \"Checking out all public original GitHub repos\"\necho\nwhile read -r github_repo_url; do\n    dir=\"${github_repo_url##*/}\"\n    if [ -d \"$dir\" ]; then\n        timestamp \"Pulling $github_repo_url\"\n        pushd \"$dir\"\n        git pull\n        popd\n    else\n        timestamp \"Cloning $github_repo_url\"\n        git clone \"$github_repo_url\"\n    fi\n    echo\ndone <<< \"$github_repo_urls\"\ntimestamp \"Cloning done\"\necho\necho\ntimestamp \"Counting lines of code:\"\necho\necho\ntimestamp \"Cloc:\"\necho\ncloc .\necho\necho\ntimestamp \"Scc:\"\necho\nscc .\n"
  },
  {
    "path": "github/github_pull_merge_trunk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-06-19 19:12:25 +0200 (Wed, 19 Jun 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPulls the origin or forks upstream repo's trunk branch and merges it into the local branch\n\nIn a forked GitHub repo's checkout, determines the origin of the fork using GitHub CLI,\nconfigures a git remote to the upstream, pulls the default branch and if on a branch other than the default\nthen merges the default branch to the local current branch\n\nSimplifies and automates keeping your cloned or forked repo up to date with the original source repo to quickly resolve\nmerge conflicts locally and submit updated Pull Requests\n\nSet environment variable GIT_REBASE=true if you want the pulls and merge to current branch to rebase instead of merge commit...\nif you really love violating VCS history integrity! Personally, not a fan. You can also end up in rebase hell for a series of\ncommits that a default merge commit would have auto-resolved\n\nRead:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/git.md#the-evils-of-rebasing\n\nRequires GitHub CLI to be installed and authenticated, as well as jq\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner/repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_github_origin\n\nif [ $# -gt 0 ]; then\n    upstream_owner_repo=\"$1\"\n    shift\nelse\n    if is_github_fork; then\n        upstream_owner_repo=\"$(github_upstream_owner_repo || die \"Not a forked repo?\")\"\n    fi\nfi\n\nrebase=\"${GIT_REBASE:-false}\"\n\nif ! is_bool \"$rebase\"; then\n    usage \"GIT_REBASE environment variable must be set to 'true' or 'false'\"\nfi\n\nif is_github_fork; then\n    \"$srcdir/github_remote_set_upstream.sh\" \"$upstream_owner_repo\"\n    echo\nfi\n\ndefault_branch=\"$(default_branch)\"\n\n# should be a straight fast-forward\ntimestamp \"Pulling default branch '$default_branch' from origin\"\necho\n# --no-edit option isn't available to rebase\ngit pull origin \"$default_branch\" --rebase=\"$rebase\" --no-edit\necho\n\nif is_github_fork; then\n    current_branch=\"$(git rev-parse --abbrev-ref HEAD)\"\n\n    if [ \"$current_branch\" = \"$default_branch\" ]; then\n        timestamp \"Pulling default branch '$default_branch' from upstream $upstream_owner_repo\"\n        git pull upstream \"$default_branch\" --no-edit\n    else\n        timestamp \"Fetching default branch '$default_branch' from upstream $upstream_owner_repo\"\n        echo\n        git fetch upstream \"$default_branch:$default_branch\"\n        echo\n        if [ \"$rebase\" = true ]; then\n            timestamp \"Rebasing current branch '$current_branch' from default branch '$default_branch'\"\n            git rebase \"$default_branch\"\n        else\n            timestamp \"Merging default branch '$default_branch' into current branch '$current_branch'\"\n            git merge \"$default_branch\" --no-edit\n        fi\n    fi\nfi\n"
  },
  {
    "path": "github/github_pull_request_create.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-17 11:32:45 +0000 (Thu, 17 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GitHub Pull Request idempotently by first checking for an existing PR between the branches,\nand also checking if there are the necessary commits between the branches, to avoid common errors from blindly raising PRs\n\nUseful to automate audited code promotion across environments (eg. Staging branch -> Production branch)\n\nAlso works across repo forks if the head branch contains an '<owner>:' prefix\n\nUseful Git terminology reminder:\n\nThe HEAD branch is the branch you want to merge FROM, eg. 'my-feature-branch'\nThe BASE branch is the branch you want to merge INTO, eg. 'master' or 'main'\n\nIf \\$GITHUB_PULL_REQUEST_TITLE or \\$GITHUB_PULL_REQUEST_BODY environment variables are set, those values will be used to generate the pull request\nIf \\$GITHUB_PULL_REQUEST_BODY is not set, but .github/pull_request_template.md is present at the top of the current checkout repo, will use that for the body. If the template body contains Jira ticket number token templates such as AA-XXXXX or AA-NNNNN and the current branch is prefixed with a matching alpha-number token, then those tokens will be replaced with the real Jira number from the branch\n\nIf \\$GITHUB_PULL_REQUEST_AUTO_MERGE is set to 'true' then marks the pull request to be automatically merged once its pre-requisites like checks and peer review approval are passed\n\nIf \\$GITHUB_PULL_REQUEST_SQUASH is set to any value, then marks the pull request to use a squash commit, otherwise defaults to merge\n\nTo raise PRs across forks do:\n\n    ${0##*/} <upstream_repo> <your_forked_owner>:<your_forked_repo>:<your_feature_branch> <upstream_repo_base_trunk_branch_to_merge_to>\n\n\nRequires GitHub CLI to be installed and configured\n\nUsed by adjacent scripts:\n\n    github_merge_branch.sh\n    github_repo_fork_update.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo> <from_head_branch> <to_base_branch>]\"\n\nhelp_usage \"$@\"\n\n#min_args 2 \"$@\"\nmax_args 3 \"$@\"\n\nowner_repo=\"\"\nif [ $# -eq 3 ]; then\n    owner_repo=\"$1\"\n    shift || :\nfi\n\nhead=\"${1:-}\"\nbase=\"${2:-}\"\n\nif is_blank \"$owner_repo\"; then\n    if ! is_in_git_repo; then\n        die \"Repo not specified and not in a git repository checkout to infer it\"\n    fi\n    owner_repo='{owner}/{repo}'\nfi\n\nrepo_data=\"$(gh api \"/repos/$owner_repo\")\"\n\nowner=\"$(jq -r '.owner.login' <<< \"$repo_data\")\"\nrepo=\"$(jq -r '.name' <<< \"$repo_data\")\"\n\nif is_blank \"$base\"; then\n    timestamp \"Base branch not specified, inferring to be default branch from repo\"\n    base=\"$(jq -r '.default_branch' <<< \"$repo_data\")\"\n    timestamp \"Using default branch '$base' as base branch\"\nfi\n\nif is_blank \"$head\"; then\n    if ! is_in_git_repo; then\n        die \"Head branch not specified and not in a git repository checkout to infer it\"\n    fi\n    checkout_owner_repo=\"$(gh api '/repos/{owner}/{repo}' | jq -r '.full_name')\"\n    if [ \"$owner/$repo\" != \"$checkout_owner_repo\" ]; then\n        die \"ERROR: Head branch not specified and current git repository checkout we are within ($checkout_owner_repo) does not match the target repo ($owner/$repo), so cannot use local branch name to infer it\"\n    fi\n    timestamp \"Head branch not specified, inferring to be current branch from repo checkout\"\n    head=\"$(git rev-parse --abbrev-ref HEAD)\"\n    timestamp \"Head branch was inferred from local git checkout branch to be '$head'\"\n    if [ \"$head\" = \"$base\" ]; then\n        die \"Cannot create pull request from head branch '$head' to base branch '$base' because they are the same branch! \"\n    fi\nfi\n\nif [[ \"$head\" =~ : ]]; then\n    head_owner=\"${head%%:*}\"\n    head_name=\"${head##*:}\"\nelse\n    head_owner=\"$owner\"\n    head_name=\"$head\"\nfi\n\nbody_template=\"\"\nif [ \"$owner_repo\" = '{owner}/{repo}' ]; then\n    # we are raising for the local repo\n    git_root=\"$(git_root)\"\n    pr_template=\"$git_root/.github/pull_request_template.md\"\n    if [ -f \"$pr_template\" ]; then\n        body_template=\"$(cat \"$pr_template\")\"\n        # if branch prefix matches an AA-XXXXX or AA-NNNNN token placeholder in the body, replace it\n        branch_prefix=\"$(grep -Eom1 '^[[:alpha:]]{2,}-[[:digit:]]{3,}(-|$)' <<< \"$head\" | sed 's/-$//' || :)\"\n        if [[ \"$branch_prefix\" =~ ^[[:alpha:]]{2,}-[[:digit:]]{3,}$ ]]; then\n            match1=\"${branch_prefix//[[:digit:]]/X}\"\n            match2=\"${branch_prefix//[[:digit:]]/N}\"\n            # doesn't work for replacements below\n            #shopt -s nocasematch\n            #body_template=\"${body_template//$match1/$branch_prefix}\"\n            #body_template=\"${body_template//$match2/$branch_prefix}\"\n            #shopt -u nocasematch\n            # shellcheck disable=SC2001\n            body_template=\"$(sed \"s/\\\\($match1\\\\|$match2\\\\)/$branch_prefix/gi\" <<< \"$body_template\")\"\n        fi\n    fi\nfi\n\ntitle=\"${GITHUB_PULL_REQUEST_TITLE:-Merge $head branch into $base branch}\"\nbody=\"${GITHUB_PULL_REQUEST_BODY:-${body_template:-Created automatically by script \\`${0##*/}\\` in the [DevOps Bash tools](https://github.com/HariSekhon/DevOps-Bash-tools) repo}}\"\n\ntotal_commits=\"$(gh api \"/repos/$owner/$repo/compare/$base...$head\" -q '.total_commits')\"\n\nget_pr_url(){\n    local existing_pr\n    existing_pr=\"$(gh pr list -R \"$owner/$repo\" \\\n        --json baseRefName,changedFiles,commits,headRefName,headRepository,headRepositoryOwner,isCrossRepository,number,state,title,url \\\n        -q \".[] |\n            select(.baseRefName == \\\"$base\\\") |\n            select(.headRefName == \\\"$head_name\\\") |\n            select(.headRepositoryOwner.login == \\\"$head_owner\\\")\n    \")\"\n    if [ -n \"$existing_pr\" ]; then\n        jq -r '.url' <<< \"$existing_pr\"\n    else\n        echo \"\"\n    fi\n}\n\nif [ \"$total_commits\" -gt 0 ]; then\n    # check for existing PR between these branches before creating another\n    existing_pr_url=\"$(get_pr_url)\"\n    if [ -n \"$existing_pr_url\" ]; then\n        timestamp \"Branch '$base' already has an existing pull request from '$head', skipping creating PR:\"\n        echo >&2\n        echo \"$existing_pr_url\"\n        echo >&2\n    else\n        timestamp \"Creating Pull Request from head '$head' into base branch '$base'\"\n        # --no-maintainer-edit is important, otherwise member ci account gets error (and yes there is a double 'Fork collab' error in GitHub CLI's error message):\n        # pull request create failed: GraphQL: Fork collab Fork collab can't be granted by someone without permission (createPullRequest)\n        gh pr create -R \"$owner/$repo\" \\\n                     --base  \"$base\"   \\\n                     --head  \"$head\"   \\\n                     --title \"$title\"  \\\n                     --body  \"$body\"   \\\n                     --no-maintainer-edit\n    fi\n    if [ \"${GITHUB_PULL_REQUEST_AUTO_MERGE:-}\" = true ]; then\n        pr_url=\"$(get_pr_url)\"\n        # not supporting Rebase on purpose - see https://medium.com/@harisekhon/the-evils-of-git-rebasing-beec34a607c7\n        merge_type=\"--merge\"\n        if [ \"${GITHUB_PULL_REQUEST_SQUASH:-}\" = true ]; then\n            merge_type=\"--squash\"\n        fi\n        gh pr merge \"$pr_url\" --auto \"$merge_type\"\n    fi\n    echo >&2\nelse\n    timestamp \"Branch '$base' is already up to date with upstream, skipping creating PR\"\n    echo >&2\nfi\n"
  },
  {
    "path": "github/github_pull_request_preview.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-15 11:16:44 +0100 (Fri, 15 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens a GitHub Pull Request preview page from the current local branch to the default branch\n\nOptionally you can specify the head and base target branch yourself as arguments\n\nUseful to call from aliases/functions to quickly open a PR. See .bash.d/git.sh where this is used via github_push_pr_preview.sh to automate this workflow with a handful of keystrokes\n\nPrints the Pull Request URL, and opens it for you in your default browser\n\nAssumes that GitHub is the remote origin, and checks for this for safety\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<target_base_branch> <head_branch>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 2 \"$@\"\n\ncheck_github_origin\n\nowner_repo=\"$(github_origin_owner_repo)\"\n\n# checks are done inside github_origin_owner_repo() now\n#if [ -z \"$owner_repo\" ]; then\n#    die 'Failed to find origin remote pointing to github.com! Are we in a github checkout?'\n#fi\n\ndefault_branch=\"${1:-$(default_branch)}\"\ncurrent_branch=\"${2:-$(current_branch)}\"\n\n#url=\"https://github.com/$owner_repo/pull/new/$branch\"\n# from your current branch to the default branch by default\nurl=\"https://github.com/$owner_repo/compare/$default_branch...$current_branch\"\n\necho\necho \"Pull Request URL:\"\necho\nprintf '\\t%s\\n' \"$url\"\n\necho\necho \"Opening Pull Request\"\n\"$srcdir/../bin/urlopen.sh\" \"$url\"\n#elif [ -n \"${BROWSER:-}\" ]; then\n#    echo\n#    echo \"Opening Pull Request using \\$BROWSER\"\n#    \"$BROWSER\" \"$url\"\n#else\n#    echo\n#    echo \"\\$BROWSER environment variable not set and not on Mac to use default browser, not opening browser\"\n#fi\n"
  },
  {
    "path": "github/github_purge_camo_cache.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-08 18:00:53 +0100 (Mon, 08 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPurges all GitHub camo caches for things like CI/CD badges\n\nIf no owner/repo arg is given, attempts to determine the GitHub repo from the current git repo's origin\n\nhttps://docs.github.com/en/authentication/keeping-your-account-and-data-secure/about-anonymized-urls#removing-an-image-from-camos-cache\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    owner_repo=\"$1\"\nelse\n    if is_github_origin; then\n        owner_repo=\"$(github_origin_owner_repo)\"\n    else\n        usage \"no <owner>/<repo> arg given and current directory doesn't appear to be a GitHub clone to auto-determine it\"\n    fi\nfi\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    usage \"Invalid owner/repo given: $owner_repo\"\nfi\n\nurl=\"https://github.com/$owner_repo\"\n\ntimestamp \"Fetching: $url\"\ncurl -sL \"$url\" |\ngrep -Eo '<img src=\"https?://camo.githubusercontent.com/[^\"]+' |\nsed -e 's/<img src=\"//' |\nwhile read -r camo_url; do\n    timestamp \"Purging: $camo_url\"\n    echo \"curl -sX PURGE '$camo_url' &>/dev/null\"\ndone |\nparallel -j 10\n"
  },
  {
    "path": "github/github_push_pr.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-15 11:16:44 +0100 (Fri, 15 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPushes the current branch to GitHub origin, sets upstream branch, then raises a Pull Request to the given or default branch\n\nIf \\$GITHUB_MERGE_PULL_REQUEST=true then will automatically merge the pull request as well\n\nIf \\$GITHUB_MERGE_PULL_REQUEST_AS_ADMIN=true then will merge as admin to bypass branch merge protection\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<target_base_branch> <title> <description>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 3 \"$@\"\n\ncheck_github_origin\n\nbase_branch=\"${1:-$(default_branch)}\"\nexport GITHUB_PULL_REQUEST_TITLE=\"${2:-}\"\nexport GITHUB_PULL_REQUEST_BODY=\"${3:-}\"\n\ncurrent_branch=\"$(current_branch)\"\n\ngit push --set-upstream origin \"$(current_branch)\"\n\necho\noutput=\"$(\"$srcdir/github_pull_request_create.sh\" \"$current_branch\" \"$base_branch\")\"\necho \"$output\"\necho\n\nif [ -z \"$output\" ]; then\n    die \"Pull request not created\"\nfi\n\nurl=\"$(parse_pull_request_url \"$output\")\"\n\nif [ \"${GITHUB_MERGE_PULL_REQUEST:-}\" = true ]; then\n    args=\"\"\n    if [ \"${GITHUB_MERGE_PULL_REQUEST_AS_ADMIN:-}\" = true ]; then\n        args=\"--admin\"\n    fi\n    timestamp \"Merging Pull Request:  $url\"\n    gh pr merge --merge \"$url\" $args\nfi\n\nif is_mac; then\n    echo \"Opening Pull Request\"\n    open \"$url\"\nelif [ -n \"${BROWSER:-}\" ]; then\n    echo \"Opening Pull Request using \\$BROWSER\"\n    \"$BROWSER\" \"$url\"\nelse\n    echo \"\\$BROWSER environment variable not set and not on Mac to use default browser, not opening browser\"\nfi\n"
  },
  {
    "path": "github/github_push_pr_preview.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-15 11:16:44 +0100 (Fri, 15 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPushes the current branch to GitHub origin, setting upstream branch, then opens a Pull Request preview from this to the given or default branch\n\nAssumes that GitHub is the remote origin, and checks for this for safety\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<target_base_branch> <head_branch>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 2 \"$@\"\n\ncheck_github_origin\n\ncurrent_branch=\"$(current_branch)\"\n\ngit push --set-upstream origin \"$current_branch\"\n\n\"$srcdir/github_pull_request_preview.sh\" \"$@\"\n"
  },
  {
    "path": "github/github_release.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-12 14:18:52 +0100 (Tue, 12 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GitHub Release and Git Tag for the current repo checkout, auto-incrementing the default vYYYY.NN release if one isn't given\n\nDetermines the GitHub repository to create a release in from the local checkout from which it is executed,\nunless \\$GITHUB_OWNER_REPO is set in the environment or '-R <owner>/<repo>' are given as the final args\n\nThe first argument is the version, which is recommended to set to vN.N.N eg. v1.0.0 as per semantic versioning standards\n\nIf the first argument is 'day' or 'date', will determine the next available release in the format vYYYYMMDD.NN where NN is incremented from 1\nIf the first argument is 'month', will determine the next available release in the format vYYYYMM.NN\nIf the first argument is 'year', will determine the next available release in the format vYYYY.NN\n\nIf no argument is given, defaults to generating a 'year' version in the format vYYYY.NN\n\nThese formats don't have dashes in them like ISO dates so that if you move from YYYY to YYYYMM format or YYYYMMDD format, GitHub will recognize the newer format as the Latest release\n\nIf you later return to short format releases of just year or month, GitHub won't detect them as the Latest release (determined via testing).\n\nWARNING: if you delete a GitHub release, the tag is left in the repo. If you then create a new release automatically defaulting to the version that was just deleted and it reuses the old git tag, you could end up with a release pointing to an old tag rather than the current commit\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version> <title> <description> <gh_cli_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-year}\"\ntitle=\"${2:-}\"\ndescription=\"${3:-}\"\nshift || :\nshift || :\nshift || :\n\nowner_repo=()\nif [ -n \"${GITHUB_OWNER_REPO:-}\" ]; then\n    owner_repo=(-R \"$GITHUB_OWNER_REPO\")\nfi\n\ngenerate_version=0\nprefix='v'\nif [ -n \"${NO_GITHUB_RELEASE_PREFIX:-}\" ]; then\n    prefix=''\nfi\n\nif [ \"$version\" = year ]; then\n    version=\"${prefix}$(date '+%Y')\"\n    generate_version=1\nelif [ \"$version\" = month ]; then\n    version=\"${prefix}$(date '+%Y%m')\"\n    generate_version=1\nelif [ \"$version\" = day ] || [ \"$version\" = date ]; then\n    version=\"${prefix}$(date '+%Y%m%d')\"\n    generate_version=1\nfi\n\nif [ \"$generate_version\" = 1 ]; then\n    latest_releases=\"$(gh release list ${owner_repo:+\"${owner_repo[@]}\"} -L 200 --exclude-drafts \"$@\" | awk '{print $1}')\"\n\n    number=\"$(grep -Eo \"^$version\"'\\.\\d+' <<< \"$latest_releases\" | head -n 1 | sed \"s/^$version\\\\.//\" || echo 1)\"\n\n    # increment the number\n    while grep -Fxq \"$version.$number\" <<< \"$latest_releases\"; do\n        ((number+=1))\n        if [ $number -gt 9999 ]; then\n            die \"FAILED to find unused release in format '$version.NN'\"\n        fi\n    done\n\n    version+=\".$number\"\nfi\n\nif is_blank \"$title\"; then\n    title=\"$version\"\nfi\n\ngh release create ${owner_repo:+\"${owner_repo[@]}\"} \"$version\" --title \"$version\" --notes \"$description\" \"$@\"\n"
  },
  {
    "path": "github/github_remote_set_upstream.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-31 18:50:34 +0300 (Wed, 31 Jul 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nIn a forked GitHub repo's checkout, determine the origin of the fork using GitHub CLI and configure a git remote to the upstream\n\nUseful to be able to easily pull updates from the original source repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner/repo>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\ncheck_github_origin\n\nif [ $# -gt 0 ]; then\n    upstream_owner_repo=\"$1\"\n    shift\nelse\n    upstream_owner_repo=\"$(github_upstream_owner_repo || die \"Not a forked repo?\")\"\nfi\n\n# follow the clone checkout scheme eg. https:// or ssh://\n#\n# this helps remote workers like me avoiding proxy / wifi hotspot restrictions using HTTPS clones instead of SSH\ntimestamp \"Determining git base url\"\necho\ngithub_url_base=\"$(\n    git remote -v |\n    awk '/origin/ { print $2; exit }' |\n    sed 's|\\(.*github.com[:/]\\).*|\\1|'\n)\"\n\nif git remote -v | grep -q '^upstream'; then\n    timestamp \"Git remote 'upstream' already exists, not creating\"\nelse\n    timestamp \"Adding git remote 'upstream' to be able to pull directly from original source repo we are forked from\"\n    git remote add upstream \"${github_url_base}${upstream_owner_repo}\"\nfi\n"
  },
  {
    "path": "github/github_repo_add_collaborator.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-31 17:37:12 +0100 (Thu, 31 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/collaborators\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds a given user as a collaborate to the given GitHub repo with the specified permission (eg. for CI/CD machine accounts as outside invited collaborators) via GitHub API\n\nPerm should usually be one of:\n\n    pull\n    push\n    admin\n    maintain\n    triage\n\nThis is most useful for GitHub Enterprise Organization repos to add a CI/CD machine account programmatically, especially when combined with github_foreach_repo.sh.\nAlternatively if you don't have SSO enforced you can add the machine account directly as a member of the Organization with default access to all repos\n\nSee Also:\n\n    github_repo_collaborators.sh - show collaborators and their permissions to a given repo\n    github_invitations.sh        - show or accept repos invites programmatically\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> <user> <permission>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nowner_repo=''\n\nif [ $# -gt 3 ]; then\n    owner_repo=\"$1\"\n    user=\"$2\"\n    perm=\"$3\"\nelif [ $# -eq 2 ]; then\n    user=\"$1\"\n    perm=\"$2\"\nelse\n    usage \"invalid number of args, must be 2 or 3\"\nfi\n\nif [ -z \"$owner_repo\" ]; then\n    owner_repo=\"$(get_github_repo)\"\nfi\n\ntimestamp \"Adding collaborator '$user' to GitHub repo '$owner_repo'\"\n\"$srcdir/github_api.sh\" \"/repos/$owner_repo/collaborators/$user\" -X PUT -d '{\"permission\": \"'\"$perm\"'\" }'\n"
  },
  {
    "path": "github/github_repo_check_pat_token.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-16 13:38:59 +0700 (Mon, 16 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks the given PAT token can access the given GitHub repo\n\nUseful to test the PAT token for integrations like ArgoCD\n\nThe token can be given as a second arg or it infers it from one of the environment varibles in this order of precedence:\n\n    \\$GH_TOKEN\n    \\$GITHUB_TOKEN\n\n\nRequires GitHub CLI to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> [<token>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\nowner_repo=\"$1\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"Invalid GitHub <owner>/<repo> given, failed regex match: $owner_repo\"\nfi\n\nexport GH_TOKEN=\"${2:-${GH_TOKEN:-${GITHUB_TOKEN:-}}}\"\n\nif is_blank \"$GH_TOKEN\"; then\n    die \"GH_TOKEN is blank and no second arg given for token\"\nfi\n\n# follows what github_api.sh infers to use as the token\ntoken_ending=\"${GH_TOKEN:(-4)}\"\n# or with a preceding space but this is not obvious and someone might remove the space, exposing the token to the screen\n#token_ending=\"${GH_TOKEN: -4}\"\n\necho \"Token ending: ...$token_ending\"\necho\necho -n \"Login: \"\n# error message is:\n#\n#   curl: (22) The requested URL returned error: 401\n#\n#\"$srcdir/github_api.sh\" /user | jq -r '.login'\n#\n# this error message is better:\n#\n#   gh: Bad credentials (HTTP 401)\n#\ngh api /user | jq -r '.login'\necho\ntimestamp \"Checking PAT token can access repo '$owner_repo'\"\necho\n# error message:\n#\n#   curl: (22) The requested URL returned error: 404\n#\n#result=\"$(\"$srcdir/github_api.sh\" \"/repos/$owner_repo\" | jq -r '.full_name')\"\n#\n# error message:\n#\n#   gh: Not Found (HTTP 404)\n#\nresult=\"$(gh api \"/repos/$owner_repo\" | jq -r '.full_name')\"\nif [ \"$result\" = null ] || is_blank \"$result\"; then\n    die \"ERROR: PAT token failed to access repo '$owner_repo'\"\nelse\n    timestamp \"Successfull accessed GitHub repo: $result\"\nfi\n"
  },
  {
    "path": "github/github_repo_collaborators.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 11:15:51 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/collaborators\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists a GitHub repo's collaborators (users granted access via teams, or outside invited collaborators) and their role name permissions using the GitHub API\n\nIf no repo is given, infers from local repo's git remotes\n\nOutput format:\n\n<repo>  <user1>   <permission>\n<repo>  <user2>   <permission>\n\nThis is most useful for GitHub Enterprise repos that are part of an organization to audit access across repos, especially when combined with github_foreach_repo.sh.\nFor many personal repos you'll likely only see your own user with admin permissions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(get_github_repo)\"\nfi\n\n\"$srcdir/github_api.sh\" \"/repos/$repo/collaborators\" |\njq -r \".[] | [\\\"$repo\\\", .login, .role_name] | @tsv\" |\ncolumn -t\n"
  },
  {
    "path": "github/github_repo_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 11:15:51 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the description of a given GitHub repo using the GitHub API\n\nIf no repo is given, infers from local repo's git remotes\n\nOutput format:\n\n<repo>  <description>\n\nCan be piped to the stdin of gitlab_project_set_description.sh\n\nThe given repo must be the current URL - cannot be a previous repo name link\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\nrepo=\"$(perl -pne 's|^https://github.com/||i' <<< \"$repo\")\"\nrepo=\"${repo##/}\"\n\n\"$srcdir/github_api.sh\" \"/repos/$repo\" |\njq -r '[.name, .description] | @tsv'\n"
  },
  {
    "path": "github/github_repo_find_files.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Dockerfile\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-11 09:26:55 +0100 (Tue, 11 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds files matching the given name in the current or given GitHub repo using the GitHub CLI\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename_regex> [<owner>/<repo>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfilename=\"$1\"\nowner_repo=\"${2:-\":owner/:repo\"}\"\n\ngh api \"/repos/$owner_repo/git/trees/HEAD?recursive=1\" |\njq -r \".tree[]?.path | select(. | test(\\\"$filename\\\") )\"\n"
  },
  {
    "path": "github/github_repo_fork_sync.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-17 00:12:59 +0100 (Fri, 17 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSync's the current fork repo or from the original source repo sync's all repos matching the given ERE regex\n\nFirst calls a repo sync of the trunk branch, then and calls github_repo_fork_update.sh to raise Pull Requests from trunk to the major branches\n\nIf not running in a fork checkout but the \\$GITHUB_FORK_REGEX environment is set to any value, then the first argument regex can be omitted\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<fork_owner_repo_regex>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nregex=\"${1:-${GITHUB_FORK_REGEX:-}}\"\n\ntimestamp \"Determining if running from within a forked repo checkout\"\nis_fork=\"$(gh api \"/repos/{owner}/{repo}\" -q '.fork')\"\n\nif [ \"$is_fork\" = true ]; then\n    timestamp \"Confirmed running from within a forked repo checkout\"\n    fork_repos='{owner}/{repo}'\nelse\n    timestamp \"Not running from a forked repo checkout\"\n    if [ -z \"$regex\" ]; then\n        usage \"not running in a fork repo and no regex given to select a fork to sync\"\n    fi\n\n    timestamp \"Getting all forked repos matching regex '$regex'\"\n    set +o pipefail\n    fork_repos=\"$(gh api '/repos/{owner}/{repo}/forks' -q '.[].full_name' | grep -Ei \"$regex\")\"\n    set -o pipefail\n\n    if [ -z \"$fork_repos\" ]; then\n        die \"Failed to find an forked repos matching regex '$regex'\"\n    fi\nfi\n\nfor owner_repo in $fork_repos; do\n    echo\n    timestamp \"Sync'ing fork $owner_repo\"\n    gh repo sync \"$owner_repo\"\n    echo\n    timestamp \"Raising Pull Requests to major branches for fork $owner_repo\"\n    \"$srcdir/github_repo_fork_update.sh\" \"$owner_repo\"\ndone\n\ntimestamp \"Fork Sync done\"\n\n#gh workflow -R \"$fork_repo\" run fork-update-pr.yaml -f debug=false\n#\n#sleep 5\n#\n#id=\"$(gh run list --workflow=fork-update-pr.yaml -R \"$fork_repo\" -L 1 --json databaseId --jq '.[].databaseId')\"\n#\n#gh run watch \"$id\" -R \"$fork_repo\"\n"
  },
  {
    "path": "github/github_repo_fork_update.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-14 11:32:21 +0000 (Mon, 14 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\nBRANCHES_TO_PR_DEFAULT=\"\n    master\n    main\n    develop\n    dev\n    staging\n    production\n\"\n\n# make this more explicit for user\nBRANCHES_TO_AUTOMERGE_DEFAULT=\"\"\n#    master\n#    main\n#    develop\n#    dev\n#    staging\n#\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates the current or given repo fork via Pull Requests for full audit tracking\n\nCreates Pull Requests for branches given as arguments or set in \\$BRANCHES_TO_PR, or else by default the following branches if they are found:\n$BRANCHES_TO_PR_DEFAULT\n\nAuto-merges the PRs for branches set in \\$BRANCHES_TO_AUTOMERGE or the following default branches:\n\n${BRANCHES_TO_AUTOMERGE_DEFAULT:-<none>}\n\nRequires GitHub CLI to be installed and configured\n\nSee also:\n\n    github_repo_fork_recreate.sh - recreates a forked repo to clean out PRs and reset branches to be clean fast-forward merges\n    gh repo sync - sync's a repo but lacks the auditing of Pull Requests\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>/<repo> <branch1> <branch2> <branch3> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nowner_repo=\"${1:-}\"\nshift || :\nbranches=\"${*:-${BRANCHES_TO_PR:-$BRANCHES_TO_PR_DEFAULT}}\"\nbranches_to_automerge=\"${BRANCHES_TO_AUTOMERGE:-$BRANCHES_TO_AUTOMERGE_DEFAULT}\"\n\nif is_blank \"$owner_repo\"; then\n    if ! is_in_git_repo; then\n        die \"No repo given and not in a git repository checkout to infer it\"\n    fi\n    owner_repo='{owner}/{repo}'\nfi\n\nrepo_data=\"$(gh api \"/repos/$owner_repo\")\"\n\nis_fork=\"$(jq -r '.fork' <<< \"$repo_data\")\"\nowner=\"$(jq -r '.owner.login' <<< \"$repo_data\")\"\nrepo=\"$(jq -r '.name' <<< \"$repo_data\")\"\n\nif [ \"$is_fork\" != true ]; then\n    die \"Repo '$owner/$repo' is not a forked repo, cannot raise a pull request from an original source repo\"\nfi\n\nfork_source_owner=\"$(jq -r '.source.owner.login' <<< \"$repo_data\")\"\nfork_source_repo=\"$(jq -r '.source.full_name' <<< \"$repo_data\")\"\nfork_source_default_branch=\"$(jq -r '.source.default_branch' <<< \"$repo_data\")\"\n\nfork_repo_branches=\"$(get_github_repo_branches \"$owner/$repo\")\"\nsource_repo_branches=\"$(get_github_repo_branches \"$fork_source_repo\")\"\n\nfor branch in $branches; do\n    # use function to iterate pages\n    #if ! gh api \"/repos/$owner/$repo/branches?per_page=100\" -q '.[].name' | grep -Fxq \"$branch\"; then\n    if ! grep -Fxq \"$branch\" <<< \"$fork_repo_branches\"; then\n        timestamp \"No local fork branch '$branch' found, skipping PR\"\n        echo >&2\n        continue\n    fi\n\n    # use function to iterate pages\n    #if gh api \"/repos/$fork_source_repo/branches?per_page=100\" -q '.[].name' | grep -Fxq \"$branch\"; then\n    if grep -Fxq \"$branch\" <<< \"$source_repo_branches\"; then\n        fork_source_branch=\"$branch\"\n    else\n        fork_source_branch=\"$fork_source_default_branch\"\n    fi\n\n    base=\"$branch\"\n    head=\"$fork_source_owner:$fork_source_branch\"\n\n    if tr '[:space:]' '\\n' <<< \"$branches_to_automerge\" | sed '/^[[:space:]]*$/d' | grep -Fxq \"$branch\"; then\n        \"$srcdir/github_merge_branch.sh\" \"$owner_repo\" \"$head\" \"$base\"\n    else\n        \"$srcdir/github_pull_request_create.sh\" \"$owner_repo\" \"$head\" \"$base\"\n    fi\ndone\n"
  },
  {
    "path": "github/github_repo_latest_release.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: anchore/grype\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 18:33:13 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the latest release name/version for a given 'owner/repo' via the GitHub API\n\nIf a repo has no releases, gets a 404 error\n\nRequires curl and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo>\"\n\nhelp_usage \"$@\"\n\ncheck_bin curl\ncheck_bin jq\n\nmin_args 1 \"$@\"\n\nowner_repo=\"$1\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"Invalid owner/repo argument given: $owner_repo\"\nfi\n\nif [ -n \"${GITHUB_TOKEN:-}\" ]; then\n    \"$srcdir/github_api.sh\" \"/repos/$owner_repo/releases/latest\"\nelse\n    curl -sSL --fail \"https://api.github.com/repos/$owner_repo/releases/latest\"\nfi |\njq_debug_pipe_dump |\njq -e -r .tag_name ||\ndie \"Failed to determine latest release\"\n"
  },
  {
    "path": "github/github_repo_latest_release_filter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: kubernetes-sigs/kustomize kustomize\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 18:33:13 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the latest release name/version for a given 'owner/repo' and release regex filter via the GitHub API\n\nThis is because some repos like that of Kustomize (kubernetes-sigs/kustomize) have releases for different components\nso the latest release API endpoint is not sufficient for the use case of only returning the latest Kustomize release\n\nIf a repo has no releases, gets a 404 error\n\nRequires curl and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> <regex>\"\n\nhelp_usage \"$@\"\n\ncheck_bin curl\ncheck_bin jq\n\nmin_args 2 \"$@\"\n\nowner_repo=\"$1\"\nregex_filter=\"$2\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"Invalid owner/repo argument given: $owner_repo\"\nfi\n\nif [ -n \"${GITHUB_TOKEN:-}\" ]; then\n    \"$srcdir/github_api.sh\" \"/repos/$owner_repo/releases\"\nelse\n    curl -sSL --fail \"https://api.github.com/repos/$owner_repo/releases\"\nfi |\njq_debug_pipe_dump |\njq -r \"limit(1; .[] | select(.tag_name | test(\\\"$regex_filter\\\")) | .tag_name)\"\n"
  },
  {
    "path": "github/github_repo_protect_branches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-14 15:44:55 +0100 (Tue, 14 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/repos#update-branch-protection\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\nsettings='\n{\n    \"allow_force_pushes\": false,\n    \"allow_deletions\": false,\n    \"enforce_admins\": true,\n    \"required_status_checks\": null,\n    \"required_pull_request_reviews\": null,\n    \"restrictions\": null\n}\n'\n\ndefault_branches_to_protect=\"\n    main\n    master\n    develop\n    dev\n    staging\n    production\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables branch protection for one or more branches in the given GitHub repo (prevents deleting the branch or force pushing over it)\n\nIf no branch is specified, then applies branches protections to any of the following branches if they're found:\n$default_branches_to_protect\n\nXXX: Beware this could reset certain protection settings on the branch when run, such as enabling/disabling PR approvals due to the way the API bundles them together.\n     This is the complete list of settings sent, which you'd need to modify near the top of this code to change:\n\n$(jq . <<< \"$settings\")\n\n\nFor authentication and other details see:\n\n    github_api.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner> <repo> [<branch> <branch2> <branch3> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 2 \"$@\"\n\nowner=\"${1:-}\"\nrepo=\"${2:-}\"\nshift || :\nshift || :\n\n#check_env_defined GH_TOKEN\n\nif ! is_blank \"$owner\" &&\n   !  is_blank \"$repo\"; then\n    owner_repo=\"$owner/$repo\"\nelse\n    timestamp \"No GitHub owner/repo specified - determining from current Git checkout\"\n    if ! is_in_git_repo; then\n        usage \"Did not specify owner and repo and not in a Git repo to try to infer it\"\n    fi\n    owner_repo=\"$(get_github_repo || :)\"\n    if is_blank \"$owner_repo\"; then\n        owner_repo=\"$(github_owner_repo)\"\n    fi\n    timestamp \"Inferred GitHub repo to be: $owner_repo\"\nfi\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"ERROR: invalid GitHub owner/repo provided, failed regex validation: $owner_repo\"\nfi\n\nprotect_repo_branch(){\n    local branch=\"$1\"\n    timestamp \"protecting GitHub repo '$owner_repo' branch '$branch'\"\n    \"$srcdir/github_api.sh\" \"/repos/$owner_repo/branches/$branch/protection\" -X PUT -d \"$settings\" >/dev/null\n    timestamp \"protection applied to branch '$branch'\"\n}\n\nif [ $# -gt 0 ]; then\n    for branch in \"$@\"; do\n        protect_repo_branch \"$branch\"\n    done\nelse\n    timestamp \"no branches specified, getting branch list\"\n    branches=\"$(get_github_repo_branches \"$owner_repo\")\"\n    for branch in $default_branches_to_protect; do\n        timestamp \"checking for branch '$branch'\"\n        if grep -Fxq \"$branch\" <<< \"$branches\"; then\n            protect_repo_branch \"$branch\"\n        fi\n    done\nfi\n"
  },
  {
    "path": "github/github_repo_stars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 11:15:51 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists a GitHub repo's stars and forks using the GitHub API\n\nIf no repo is given, infers from local repo's git remotes\n\nOutput format:\n\n<repo>  <stars>  <forks>  <watchers>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\n\"$srcdir/github_api.sh\" \"/repos/$repo\" |\njq -r '[.name, .stargazers_count, .forks, .watchers] | @tsv'\n"
  },
  {
    "path": "github/github_repo_teams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 11:15:51 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists a GitHub repo's teams and their role name permissions using the GitHub API\n\nIf no repo is given, infers from local repo's git remotes\n\nOutput format:\n\n<repo>  <team1>  <permission>\n<repo>  <team2>  <permission>\n\nThis is most useful for GitHub Enterprise repos that are part of an organization to audit access across repos, especially when combined with github_foreach_repo.sh.\nFor many personal repos where you haven't invited collaborators, you will get no results, just as you will see in the GitHub UI a message like \\\"You haven't invited any collaborators yet\\\" on the page https://github.com/<user>/<repo>/settings/access\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\n\"$srcdir/github_api.sh\" \"/repos/$repo/teams\" |\njq -r \".[] | [\\\"$repo\\\", .slug, .permission] | @tsv\" |\ncolumn -t\n"
  },
  {
    "path": "github/github_repos_disable_rebase.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-14 15:44:55 +0100 (Tue, 14 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/repos#update-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables Pull Request Rebase Merges on one or more given GitHub repos\n\nFor authentication and other details see:\n\n    github_api.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner> <repo> [<repo2> <repo3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nowner=\"$1\"\nshift || :\n\ndisable_pr_rebasing(){\n    local repo=\"$1\"\n    timestamp \"disabling PR rebasing on repo '$owner/$repo'\"\n    \"$srcdir/github_api.sh\" \"/repos/$owner/$repo\" -X PATCH -d '{\"allow_rebase_merge\": false}' |\n    jq -e -r '[ \"Rebase Merging: \", .allow_rebase_merge ] | @tsv'\n}\n\nfor repo in \"$@\"; do\n    disable_pr_rebasing \"$repo\"\ndone\n"
  },
  {
    "path": "github/github_repos_disable_wiki.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-14 15:44:55 +0100 (Tue, 14 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/repos#update-a-repository\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables the Wiki on one or more given GitHub repos (to prevent scattered documentation instead of using say Confluence or Slite)\n\nFor authentication and other details see:\n\n    github_api.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> <repo> [<repo2> <repo3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\norg=\"$1\"\nshift || :\n\ndisable_repo_wiki(){\n    local repo=\"$1\"\n    timestamp \"disabling wiki on GitHub organization '$org' repo '$repo'\"\n    \"$srcdir/github_api.sh\" \"/repos/$org/$repo\" -X PATCH -d '{\"has_wiki\": false}' |\n    jq -e -r '[ \"Wiki Enabled: \", .has_wiki ] | @tsv'\n}\n\nfor repo in \"$@\"; do\n    disable_repo_wiki \"$repo\"\ndone\n"
  },
  {
    "path": "github/github_repos_find_files.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Dockerfile\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-11 09:26:55 +0100 (Tue, 11 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds files matching the given name across all repos in the current organization or user using the GitHub API & CLI\n\nOutput Format:\n\n<owner>/<repo>    <file_path>\n<owner>/<repo2>   <file_path2>\n\n\nRequires GitHub CLI to be installed and configured, as well as the adjacent github_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename_regex>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfilename=\"$1\"\n\nowner=\"${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}\"\n\nget_github_repos \"$owner\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    gh api \"/repos/$owner/$repo/git/trees/HEAD?recursive=1\" |\n    jq -r \".tree[]?.path | select(. | test(\\\"$filename\\\") )\" |\n    sed $\"s|^|$owner/$repo\\\\t|\"\ndone\n"
  },
  {
    "path": "github/github_repos_not_in_terraform.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 16:00:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all non-fork GitHub repos for the current or given user or organization which are not found in ./*.tf code\n\nUseful to catch if anyone has created any unmanaged repos\n\nRelies on the GitHub repo name matching the terraform repo identifier, or in cases of repos with a leading dot such as '.github', without the dot prefix\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<owner>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nowner=\"${1:-}\"\n\nrepos=\"$(\n    gh repo list ${owner:+\"$owner\"} \\\n            -L 99999 \\\n            --json name,isFork \\\n            -q '.[] | select(.isFork == false) | .name' |\n    sort -f\n)\"\n\nfor repo in $repos; do\n    # literal terraform resource github_repository are easy to find, but assumes the resource name is the same as the repo name\n    # search without the dot prefix which isn't allowed in Terraform code identifiers\n    grep -Eq '^[[:space:]]*resource[[:space:]]+\"github_repository\"[[:space:]]+\"'\"${repo#.}\"'\"' ./*.tf ||\n    # but if using a module such as github_repo (https://github.com/HariSekhon/Terraform) then need to find names in a repos.tf file, not very portable, may need tuning if you do something different\n    grep -Eq \"^[[:space:]]+name[[:space:]]*=[[:space:]]*\\\"$repo\\\"[[:space:]]*$\" repos.tf 2>/dev/null ||\n    echo \"$repo\"\ndone\n"
  },
  {
    "path": "github/github_repos_public.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-04 11:20:07 +0000 (Thu, 04 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists a GitHub user or organization's public repositories using the GitHub API\n\nUseful to periodically scan for any accidentally public repos\n\n\\$GITHUB_ORGANIZATION takes precedence over \\$GITHUB_USER\n\nSee github_api.sh for further authentication details\n\nSee also:\n\n    github_repo_teams.sh - to list groups and permissions to each repo (combine with github_foreach_repo.sh to audit all repos)\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nuser_or_org=\"${GITHUB_ORGANIZATION:-$user}\"\n\nget_github_repos \"$user_or_org\" \"${GITHUB_ORGANIZATION:-}\" \"select(.private != true)\"\n"
  },
  {
    "path": "github/github_repos_sync_status.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-31 18:05:24 +0100 (Mon, 31 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each non-fork GitHub repo, checks corresponding GitLab / BitBucket / Azure DevOps repos to check they are in sync with the same master hashref\n\nUseful for checking if GitLab repo mirroring has broken or if BitBucket / Azure DevOps haven't been pushed to be kept in sync with master GitHub repos\n\n\nOutput Format:\n\nRepo: <user>/<repo>    GitHub: <sha_hash>        GitLab: <sha_hash>        BitBucket: <sha_hash>        Azure_Devops: <sha_hash>        In-Sync: <boolean>\n\nOutput Format If selecting only one of --gitlab / --bitbucket / --azure-devops:\n\nRepo: <user>/<repo>    GitHub: <sha_hash>        GitLab: <sha_hash>        In-Sync: <boolean>\n\nRepo: <user>/<repo>    GitHub: <sha_hash>        BitBucket: <sha_hash>     In-Sync: <boolean>\n\nRepo: <user>/<repo>    GitHub: <sha_hash>        Azure_DevOps: <sha_hash>  In-Sync: <boolean>\n\n\nArguments can be specified to check only select repos, and options can be specified to compare each GitHub repo to its\nGitLab or BitBucket counterpart of the same name (by default checks both)\n\nBy default this will check through all GitHub repos - this can be pages of hundreds of lines of GitHub repos, so for\nefficiency you may want to explicitly specify the repos as args or in cases where not all repos are mirrored to GitLab\nor pushed to BitBucket\n\nUser name is assumed to be the same across GitHub / GitLab / BitBucket. If it's not, make sure to specify \\$GITLAB_USER / \\$BITBUCKET_USER to override\n\nCaveats:\n\n- due to a limitation of the BitBucket API requiring username + token, cannot auto-determine username so if your\n\\$BITBUCKET_USERNAME / \\$BITBUCKET_USER / \\$USER etc is incorrect, this'll fail to authenticate and won't find the relevant repo,\nreturning 'None' for the hashref and 'In-Sync: False'. You must ensure your BitBucket username is correct\n\n- the BitBucket workspace is assumed to be the same as the username\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[options] [<repos_to_check>]\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_switches=\"\n-g --gitlab         Compare each GitHub repo with its GitLab counterpart\n-b --bitbucket      Compare each GitHub repo with its BitBucket counterpart\n-a --azure-devops   Compare each GitHub repo with its Azure DevOps counterpart\n-d --date           Compare by date of latest commit instead of hashref\n-l --long-hashrefs  Use long 40 char hashrefs, not 8 char abbreviated ones\n\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepos=()\ncheck_gitlab=0\ncheck_bitbucket=0\ncheck_azure_devops=0\ncompare_by_date=0\nlong_hashrefs=0\n\nfor arg; do\n    case \"$arg\" in\n        -g|--gitlab)    check_gitlab=1\n                        ;;\n     -b|--bitbucket)    check_bitbucket=1\n                        ;;\n  -a|--azure-devops)    check_azure_devops=1\n                        ;;\n          -d|--date)    compare_by_date=1\n                        shift || :\n                        ;;\n -l|--long-hashrefs)    long_hashrefs=1\n                        shift || :\n                        ;;\n         -D|--debug)    export DEBUG=1\n                        shift || :\n                        ;;\n                 -*)    usage\n                        ;;\n                  *)    repos+=(\"$arg\")\n                        ;;\n    esac\ndone\n\nif [ $compare_by_date = 1 ] && [ $long_hashrefs = 1 ]; then\n    usage \"--date and --long-hashrefs are mutually exclusive\"\nfi\n\ncheck_gitlab(){\n    [ $check_gitlab = 1 ]       && return 0\n    [ $check_bitbucket = 0 ]    || return 1\n    [ $check_azure_devops = 0 ] || return 1\n    return 0\n}\n\ncheck_bitbucket(){\n    [ $check_bitbucket = 1 ]    && return 0\n    [ $check_gitlab = 0 ]       || return 1\n    [ $check_azure_devops = 0 ] || return 1\n    return 0\n}\n\ncheck_azure_devops(){\n    [ $check_azure_devops = 1 ] && return 0\n    [ $check_gitlab = 0 ]       || return 1\n    [ $check_bitbucket = 0 ]    || return 1\n    return 0\n}\n\ngithub_user=\"$(get_github_user)\"\n\n# need gitlab user\nif check_gitlab; then\n    gitlab_user=\"${GITLAB_USERNAME:-${GITLAB_USER:-}}\"\n    if [ -z \"$gitlab_user\" ]; then\n        gitlab_user=\"$(\"$srcdir/../gitlab/gitlab_api.sh\" \"/user\" | jq -r '.username')\"\n        gitlab_user=\"${gitlab_user:-<user>}\"\n    fi\nfi\n\ncheck_repos(){\n    for repo in \"$@\"; do\n        # very concise gives exactly the head hashref but no date\n        #github_commits=\"$(\"$srcdir/github_api.sh\" \"/repos/$github_user/$repo/git/ref/heads/master\")\"\n        github_commits=\"$(\"$srcdir/github_api.sh\" \"/repos/$github_user/$repo/commits/master?per_page=1\")\"\n        if [ $compare_by_date = 1 ]; then\n            # GitHub returns Z time\n            github_master_ref=\"$(jq -r '.commit.committer.date' <<< \"$github_commits\")\"\n        else\n            #github_master_ref=\"$(jq -r '.object.sha' <<< \"$github_commits\")\"\n            github_master_ref=\"$(jq -r '.sha' <<< \"$github_commits\")\"\n            if [ $long_hashrefs = 0 ]; then\n                github_master_ref=\"${github_master_ref:0:8}\"\n            fi\n        fi\n        # don't printf as we go because it's harder to debug, instead collect the line and print in one go\n        line=\"$(printf '%-26s\\tGitHub: %s    ' \"$github_user/$repo\" \"$github_master_ref\")\"\n        in_sync=True\n        if check_gitlab; then\n            #gitlab_commits=\"$(\"$srcdir/../gitlab/gitlab_api.sh\" \"/projects/${gitlab_user}%2F$repo/repository/branches/master\" 2>/dev/null || :)\"\n            gitlab_commits=\"$(\"$srcdir/../gitlab/gitlab_api.sh\" \"/projects/${gitlab_user}%2F$repo/repository/commits?ref_name=master&per_page=1\" 2>/dev/null || :)\"\n            if [ $compare_by_date = 1 ]; then\n                # GitHub returns current timezone eg. .000+01:00\n                gitlab_master_ref=\"$(jq -r '.[0].committed_date' <<< \"$gitlab_commits\")\"\n                if [ -n \"$gitlab_master_ref\" ] && [ \"$gitlab_master_ref\" != null ]; then\n                    gitlab_master_ref=\"$(date --utc -d \"$gitlab_master_ref\" '+%FT%TZ')\"\n                fi\n            else\n                # or .commit.short_id - only GitLab gives this short hashref in the API, we'll just truncate all of them to 8 chars for output\n                # for /repository/branches/master endpoint\n                #gitlab_master_ref=\"$(jq -r '.commit.id' <<< \"$gitlab_commits\")\"\n                gitlab_master_ref=\"$(jq -r '.[0].id' <<< \"$gitlab_commits\")\"\n                if [ $long_hashrefs = 0 ]; then\n                    gitlab_master_ref=\"${gitlab_master_ref:0:8}\"\n                fi\n            fi\n            gitlab_master_ref=\"${gitlab_master_ref:-None}\"\n            line+=\"$(printf \"GitLab: %-${#github_master_ref}s    \" \"$gitlab_master_ref\")\"\n            if [ \"$gitlab_master_ref\" != \"$github_master_ref\" ]; then\n                in_sync=False\n            fi\n        fi\n        if check_bitbucket; then\n            #bitbucket_commits=$(\"$srcdir/../bitbucket/bitbucket_api.sh\" \"/repositories/<user>/$repo/refs/branches/master?pagelen=1\" 2>/dev/null || : )\"\n            bitbucket_commits=\"$(\"$srcdir/../bitbucket/bitbucket_api.sh\" \"/repositories/<user>/$repo/commits/master?pagelen=1\" 2>/dev/null || : )\"\n            if [ $compare_by_date = 1 ]; then\n                # BitBucket returns +00:00 timezone\n                bitbucket_master_ref=\"$(jq -r '.values[0].date' <<< \"$bitbucket_commits\")\"\n                if [ -n \"$bitbucket_master_ref\" ] && [ \"$bitbucket_master_ref\" != null ]; then\n                    bitbucket_master_ref=\"$(date --utc -d \"$bitbucket_master_ref\" '+%FT%TZ')\"\n                fi\n            else\n                # for refs/branches/master endpoint\n                #bitbucket_master_ref=\"$(jq -r '.target.hash' <<< \"$bitbucket_commits\" || echo None)\"\n                bitbucket_master_ref=\"$(jq -r '.values[0].hash' <<< \"$bitbucket_commits\")\"\n                if [ $long_hashrefs = 0 ]; then\n                    bitbucket_master_ref=\"${bitbucket_master_ref:0:8}\"\n                fi\n            fi\n            bitbucket_master_ref=\"${bitbucket_master_ref:-None}\"\n            line+=\"$(printf \"BitBucket: %-${#github_master_ref}s    \" \"$bitbucket_master_ref\")\"\n            if [ \"$bitbucket_master_ref\" != \"$github_master_ref\" ]; then\n                in_sync=False\n            fi\n        fi\n        if check_azure_devops; then\n            # Bug in Azure DevOps API: returns commits in timestamped order to second resolution,so if two commits have the same timestamp, you may get the wrong one\n            # https://developercommunity.visualstudio.com/content/problem/508507/azure-devops-rest-api-returns-commits-in-incorrect.html\n            azure_devops_commits=\"$(\"$srcdir/../azure_devops/azure_devops_api.sh\" \"/{user}/{project}/_apis/git/repositories/$repo/commits?api-version=6.1-preview.1&searchCriteria.\\$top=1&searchCriteria.compareVersion.version=master&searchCriteria.compareVersion.versionType=branch\"  2>/dev/null || :)\"\n            azure_devops_commits=\"$(\"$srcdir/../azure_devops/azure_devops_api.sh\" \"/{user}/{project}/_apis/git/repositories/$repo/commits?\\$top=1&branch=master\"  2>/dev/null || :)\"\n            if [ $compare_by_date = 1 ]; then\n                azure_devops_master_ref=\"$(jq -r '.value[0].committer.date' <<< \"$azure_devops_commits\")\"\n                if [ -n \"$azure_devops_master_ref\" ] && [ \"$azure_devops_master_ref\" != null ]; then\n                    azure_devops_master_ref=\"$(date --utc -d \"$azure_devops_master_ref\" '+%FT%TZ')\"\n                fi\n            else\n                azure_devops_master_ref=\"$(jq -r '.value[0].commitId' <<< \"$azure_devops_commits\")\"\n                if [ $long_hashrefs = 0 ]; then\n                    azure_devops_master_ref=\"${azure_devops_master_ref:0:8}\"\n                fi\n            fi\n            azure_devops_master_ref=\"${azure_devops_master_ref:-None}\"\n            line+=\"$(printf \"Azure_DevOps: %-${#github_master_ref}s    \" \"$azure_devops_master_ref\")\"\n            if [ \"$azure_devops_master_ref\" != \"$github_master_ref\" ]; then\n                in_sync=False\n            fi\n        fi\n        printf '%sIn-Sync: %s\\n' \"$line\" \"$in_sync\"\n    done\n}\n\nif [ \"${#repos[@]}\" -gt 0 ]; then\n    check_repos \"${repos[@]}\"\nelse\n    # want splitting\n    # shellcheck disable=SC2046\n    check_repos $(get_github_repos \"$github_user\")\nfi\n"
  },
  {
    "path": "github/github_repos_with_few_teams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 1\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-29 18:08:23 +0000 (Wed, 29 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds GitHub repo with few or no teams, which in Enterprises is a sign that a user has created a repo without assigning team privileges\n\nThe default teams threshold if not given is 0\n\nOutput format:\n\n<org>/<repo3>\n<org>/<repo5>\n...\n\n\nOutput format if \\$VERBOSE is set (timestamped logs are sent to stderr):\n\n2021-12-29 18:09:57  checking repos for '<org>' with <= 0 teams\n2021-12-29 18:09:57  checking repo: <org>/<repo1>\n2021-12-29 18:09:57  checking repo: <org>/<repo2>\n2021-12-29 18:09:58  checking repo: <org>/<repo3>\n<org>/<repo>  <team1>  <permission>\n<org>/<repo>  <team2>  <permission>\n2021-12-29 18:10:00  checking repo: <org>/<repo4>\n...\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<minimum_number_of_teams>]\"\n\nhelp_usage \"$@\"\n\nminimum_number_of_teams=\"${1:-0}\"\n\nif ! is_int \"$minimum_number_of_teams\"; then\n    usage \"invalid number of teams argument '$minimum_number_of_teams', must be an integer\"\nfi\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nuser_or_org=\"${GITHUB_ORGANIZATION:-$user}\"\n\nlog \"checking repos for '$user_or_org' with <= $minimum_number_of_teams teams\"\n\nget_github_repos \"$user_or_org\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    log \"checking repo: $user_or_org/$repo\"\n    \"$srcdir/github_api.sh\" \"/repos/$user_or_org/$repo/teams\" |\n     jq -r \"select(length <= $minimum_number_of_teams) | \\\"$repo\\\"\"\ndone\n"
  },
  {
    "path": "github/github_repos_with_few_users.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 2\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-29 18:08:23 +0000 (Wed, 29 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/git.sh\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds GitHub repo with few users, which in Enterprises is a sign that a user has created a repo without assigning team privileges\n\nThe default user threshold if not given is 1\n\n\nOutput format:\n\n<org>/<repo3>\n<org>/<repo5>\n...\n\n\nOutput format if \\$VERBOSE is set (timestamped logs are sent to stderr):\n\n2021-12-29 18:09:57  checking repos for '<org>' with <= 1 users\n2021-12-29 18:09:57  checking repo: <org>/<repo1>\n2021-12-29 18:09:57  checking repo: <org>/<repo2>\n2021-12-29 18:09:58  checking repo: <org>/<repo3>\n<org>/<repo3>\n2021-12-29 18:10:00  checking repo: <org>/<repo4>\n...\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<minimum_number_of_users>]\"\n\nhelp_usage \"$@\"\n\nminimum_number_of_users=\"${1:-1}\"\n\nif ! is_int \"$minimum_number_of_users\"; then\n    usage \"invalid number of users argument '$minimum_number_of_users', must be an integer\"\nfi\n\nuser=\"${GITHUB_USER:-$(get_github_user)}\"\nuser_or_org=\"${GITHUB_ORGANIZATION:-$user}\"\n\nlog \"checking repos for '$user_or_org' with <= $minimum_number_of_users users\"\n\nget_github_repos \"$user_or_org\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r repo; do\n    log \"checking repo: $user_or_org/$repo\"\n    \"$srcdir/github_api.sh\" \"/repos/$user_or_org/$repo/collaborators\" |\n     jq -r \"select(length <= $minimum_number_of_users) | \\\"$repo\\\"\"\ndone\n"
  },
  {
    "path": "github/github_repos_without_branch_protections.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-14 15:44:55 +0100 (Tue, 14 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/repos#update-branch-protection\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds repos for the given user or organization that have no branch protections enabled\n\n\\$GITHUB_ORGANIZATION must be set if querying an organization\n\nFor authentication and other details see:\n\n    github_api.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization_or_owner>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nuser_or_org=\"${1:-${GITHUB_ORGANIZATION:-${GITHUB_USER:-$(get_github_user)}}}\"\n\nget_github_repos \"$user_or_org\" \"${GITHUB_ORGANIZATION:-}\" |\nwhile read -r name; do\n    repo=\"$user_or_org/$name\"\n    page=1\n    protected_branches=\"\"\n    while true; do\n        if ! output=\"$(\"$srcdir/github_api.sh\" \"/repos/$repo/branches?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        protected_branches=\"$protected_branches\n                            $(jq_debug_pipe_dump <<< \"$output\" |\n                              jq -r '.[] | select(.protected == true)')\"\n        ((page+=1))\n    done\n    if is_blank \"$protected_branches\"; then\n        echo \"$name\"\n    fi\ndone\n"
  },
  {
    "path": "github/github_ssh_add_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/free-pro-team@latest/rest/reference/users#create-a-public-ssh-key-for-the-authenticated-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds the given SSH public key(s) to the currently authenticated GitHub account via the GitHub API\n\nIf no SSH public key is given, defaults to using ~/.ssh/id_rsa.pub\n\nIf a dash is given, reads the SSH public key(s) from standard input, ignoring comment lines so you can chain with tools like the adjacent scripts:\n\n    github_ssh_get_user_public_keys.sh\n    gitlab_ssh_get_user_public_keys.sh\n    github_ssh_get_public_keys.sh\n    gitlab_ssh_get_public_keys.sh\n    bitbucket_ssh_get_public_keys.sh\n\nWill return a 422 error if the SSH public key is invalid or has already been added.\nThe script detects already existing keys and skips them to avoid this error\n\nSSH Keys can be found in the Web UI here:\n\n    https://github.com/settings/keys\n\nIf you get a 404 error it is likely that your \\$GITHUB_TOKEN doesn't have the write:public_key permission\nYou can edit the permissions your token has here:\n\n    https://github.com/settings/tokens\n\nUses the adjacent script github_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ssh_public_key_file> [<ssh_public_key_file2>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\necho \"# Getting existing SSH public keys to skip any keys that already exist (to avoid 400 errors)\" >&2\nssh_public_keys=\"$(\"$srcdir/github_ssh_get_public_keys.sh\")\"\n\nadd_ssh_public_keys_from_file(){\n    local public_key_file=\"$1\"\n    if [ \"$public_key_file\" = \"-\" ]; then\n        public_key_file=/dev/stdin\n    # sed will give this error anyway\n    #elif ! [ -f \"$public_key_file\" ]; then\n    #    die \"ERROR: file not found: $public_key_file\"\n    fi\n    sed 's/#.*//; /^[[:space:]]*$/d' \"$public_key_file\" |\n    while read -r public_key; do\n        [[ \"$public_key\" =~ ^ssh- ]] || die \"invalid SSH key in file '$public_key_file': $public_key\"\n        add_ssh_public_key \"$public_key\"\n    done\n}\n\nadd_ssh_public_key(){\n    local public_key=\"$1\"\n    key=\"$(awk '{print $1\" \"$2}' <<< \"$public_key\")\"\n    comment=\"$(awk '{$1=\"\"; $2=\"\"; print}' <<< \"$public_key\" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')\"\n    if grep -Fq \"$key\" <<< \"$ssh_public_keys\"; then\n        timestamp \"SSH public key already exists, skipping: '$public_key'\"\n        return\n    fi\n    timestamp \"adding SSH public key to currently authenticated GitHub account: '$public_key'\"\n    # don't assume the key is from the local machine, it's likely this will be used by admins or chained with other tools that download from GitHub / BitBucket and upload to GitHub for synchronization, in which case the local machine is not related to the key\n    #\"$srcdir/github_api.sh\" \"/user/keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\", \"title\": \"'\"$USER@${HOSTNAME:-$(hostname)}\"'\"}' > /dev/null  # JSON of the newly added key\n    \"$srcdir/github_api.sh\" \"/user/keys\" -X POST -H \"Content-Type: application/json\" -d '{\"title\": \"'\"$comment\"'\", \"key\": \"'\"$public_key\"'\"}' |\n    jq -r '[ \"SSH public key \\\"\" +  .title + \"\\\" was added to account as id \" + (.id | tostring) ] | @tsv' |\n    tr '\\t' ' ' |\n    timestamp \"$(cat)\"\n    echo >&2\n}\n\nif [ $# -gt 0 ]; then\n    for filename; do\n        add_ssh_public_keys_from_file \"$filename\"\n    done\nelse\n    add_ssh_public_keys_from_file ~/.ssh/id_rsa.pub\nfi\n"
  },
  {
    "path": "github/github_ssh_delete_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/free-pro-team@latest/rest/reference/users#delete-a-public-ssh-key-for-the-authenticated-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes given SSH public key(s) from the currently authenticated GitHub account via the GitHub API\n\nAccepts either a key ID or a case insensitive ERE regex to match against the key's title.\nIf multiple key titles match the given regex, deletes all of them.\nIf no keys match, does nothing.\n\nSSH Keys can be found in the Web UI here:\n\n    https://github.com/settings/keys\n\nIf you get a 404 error it's likely that your \\$GITHUB_TOKEN doesn't have the admin:public_key permission\nYou can edit the permissions your token has here:\n\n    https://github.com/settings/tokens\n\nUses the adjacent script github_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<key_id_or_title_regex>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfilter=\"$1\"\n\ntimestamp \"Getting SSH public keys\"\n\"$srcdir/github_api.sh\" \"/user/keys\" |\njq -r '.[] | [.id, .title] | @tsv' |\nwhile read -r key_id title; do\n    if [ \"$key_id\" = \"$filter\" ] ||\n       grep -Eqi \"$filter\" <<< \"$title\"; then\n        \"$srcdir/github_api.sh\" \"/user/keys/$key_id\" -X DELETE\n        timestamp \"Deleted SSH key with id '$key_id' and title '$title'\"\n    fi\ndone\n"
  },
  {
    "path": "github/github_ssh_get_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-18\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/free-pro-team@latest/rest/reference/users#list-public-ssh-keys-for-the-authenticated-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    cat <<EOF\nFetches the currently authenticated GitHub user's public SSH key(s) via the GitHub API\n\nSSH Keys can be found in the Web UI here:\n\n    https://github.com/settings/keys\n\nIf you get a 404 error it is likely that your \\$GITHUB_TOKEN doesn't have the read:public_key permission\nYou can edit the permissions your token has here:\n\n    https://github.com/settings/tokens\n\nTo get other named users's public SSH keys see:\n\n    github_ssh_get_user_public_keys.sh\n\n\n${0##*/}\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        *)  usage\n            ;;\n    esac\ndone\n\necho \"# Getting user login for tagging the keys\" >&2\nuser=\"$(\"$srcdir/github_api.sh\" \"/user\" | jq -r '.login')\"\n\n# XXX: not handling paging because if you have > 100 SSH keys I want to know what is going on first!\n\necho \"# Fetching SSH Public Key(s) from GitHub for the currently authenticated account\" >&2\necho \"#\" >&2\n\"$srcdir/github_api.sh\" \"/user/keys\" |\n# doesn't give any more info\n#jq -r '.[].id' |\n#while read -r id; do\n#    \"$srcdir/github_api.sh\" \"/user/keys/$id\" |\n#    jq .\n#done\njq -r '.[] | [.key, .title] | @tsv' |\ntr '\\t' ' ' |\nsed \"s|$| (github.com/$user)|\"\n"
  },
  {
    "path": "github/github_ssh_get_user_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-18\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/users#list-public-keys-for-a-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nFetches a GitHub user's public SSH key(s) via the GitHub API\n\nUser can be given as first argument, otherwise falls back to using environment variables \\$GITHUB_USER or \\$USER\n\nSSH Keys can be found in the Web UI here:\n\n    https://github.com/settings/keys\n\nTo retrieve SSH keys with comments, you'd need to use API authentication, see\n\n    github_ssh_get_public_keys.sh\n\n\n${0##*/} <user>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 1 ]; then\n    usage\nelif [ $# -eq 1 ]; then\n    user=\"$1\"\nelif [ -n \"${GITHUB_USER:-}\" ]; then\n    user=\"$GITHUB_USER\"\nelif [ -n \"${USER:-}\" ]; then\n    if [[ \"$USER\" =~ hari|sekhon ]]; then\n        user=harisekhon\n    else\n        user=\"$USER\"\n    fi\nelse\n    usage\nfi\n\n# XXX: not handling paging because if you have > 100 SSH keys I want to know what is going on first!\n\necho \"# Fetching SSH Public Key(s) from GitHub for account:  $user\" >&2\necho \"#\" >&2\ncurl -sS --fail \"https://api.github.com/users/$user/keys\" |\njq -r '.[].key' |\nsed \"s/$/ $user (github.com)/\"\n"
  },
  {
    "path": "github/github_ssh_get_user_public_keys2.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-18\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/rest/reference/users#list-public-keys-for-a-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nFetches a GitHub user's public SSH key(s) via HTTP\n\nUser can be given as first argument, or environment variables \\$GITHUB_USER or \\$USER\n\nTechnically should use the GitHub API, see instead:  github_ssh_get_user_public_keys.sh\n\n\n${0##*/} <user>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 1 ]; then\n    usage\nelif [ $# -eq 1 ]; then\n    user=\"$1\"\nelif [ -n \"${GITHUB_USER:-}\" ]; then\n    user=\"$GITHUB_USER\"\nelif [ -n \"${USER:-}\" ]; then\n    if [[ \"$USER\" =~ hari|sekhon ]]; then\n        user=harisekhon\n    else\n        user=\"$USER\"\n    fi\nelse\n    usage\nfi\n\n\necho \"# Fetching SSH Public Key(s) from GitHub for account:  $user\" >&2\necho \"#\" >&2\ncurl -sS --fail \"https://github.com/$user.keys\" |\nsed \"s|$| github.com/$user|\"\n"
  },
  {
    "path": "github/github_sync_repo_descriptions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 11:23:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSync GitHub repo descriptions to GitLab and BitBucket repos of the same name\n\nQueries the GitHub API for each repo's description, then pushes that description to repos of\nthe same name on GitLab via the GitLab API\n\nIf repos are given as arguments, then only sync's those repos, otherwise queries the GitHub API and iterates all repos\n\nFor more details see github_repo_description.sh, gitlab_project_set_description.sh for tuning options around authentication, user/organization for each site etc.\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user/repo1> <user/repo2> ...]\"\n\nhelp_usage \"$@\"\n\nexport GIT_FOREACH_REPO_NO_HEADERS=1\n\nif [ -n \"$*\" ]; then\n    for repo; do\n        \"$srcdir/github_repo_description.sh\" \"$repo\"\n    done\nelse\n    \"$srcdir/github_foreach_repo.sh\" \"github_repo_description.sh '{owner}/{repo}'\" 2>/dev/null\nfi |\nwhile read -r repo description; do\n    \"$srcdir/../gitlab/gitlab_project_set_description.sh\" <<< \"$repo $description\"\n    \"$srcdir/../bitbucket/bitbucket_repo_set_description.sh\" <<< \"$repo $description\"\ndone\n"
  },
  {
    "path": "github/github_tag_hashref.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-26 01:27:22 +0400 (Sat, 26 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the GitHub commit hashref for a given GitHub Actions owner/repo@tag or https://github.com/owner/repo@tag\n\nUseful for pinning 3rd party GitHub Actions to hashref instead of tag to follow GitHub Actions Best Practices:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/github-actions.md#github-actions-best-practices\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner/action@v1.2,3>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\narg=\"$1\"\n\narg=\"${arg## }\"\narg=\"${arg%% }\"\n\nif ! [[ \"$arg\" =~ ^https://github.com/ ]]; then\n    arg=\"${arg#https://github.com/}\"\nfi\n\nif ! [[ \"$arg\" =~ @ ]]; then\n    usage \"no @<tag> found in arg: $arg\"\nfi\n\ntag=\"${arg##*@}\"\narg=\"${arg%%@*}\"\n\nif ! is_github_owner_repo \"$arg\"; then\n    usage \"arg does not match expected regex for GitHub owner/repo format: $arg\"\nfi\n\ngithub_url=\"https://github.com/$arg\"\n\ngit ls-remote --tags \"$github_url\" \"$tag\" |\ngrep -E \"^[[:alnum:]]+[[:space:]]+refs/tags/$tag\" |\nawk '{print $1}'\n"
  },
  {
    "path": "github/github_teams_not_idp_synced.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 17:14:19 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all GitHub organization teams that are not sync'd from na IdP like Azure AD (these should probbly be replaced/migrated/deleted if using IdP integration)\n\nOrg can be given as an arg or taken from environment variable \\$GITHUB_ORGANIZATION\n\nif \\$QUIET is set then won't print progress to stderr, just the non-IdP sync'd teams tn stdout\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<org>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\norg=\"${1:-${GITHUB_ORGANIZATION:-}}\"\n\nif is_blank \"$org\"; then\n    usage \"Organization not defined\"\nfi\n\nfor((page=1;; page++)); do\n    if [ \"$page\" -gt 100 ]; then\n        die \"Hit over 100 pages of teams, possible infinite loop, exiting...\"\n    fi\n    if [ -z \"${QUIET:-}\" ]; then\n        timestamp \"getting list of teams page $page\"\n    fi\n    data=\"$(gh api \"/orgs/$org/teams?per_page=100&page=$page\" | jq_debug_pipe_dump)\"\n    if jq_is_empty_list <<< \"$data\"; then\n        break\n    fi\n    jq -r '.[].slug' <<< \"$data\" |\n    while read -r team; do\n        if [ -z \"${QUIET:-}\" ]; then\n            timestamp \"checking team '$team'\"\n        fi\n        team_mappings=\"$(gh api \"/orgs/$org/teams/$team/team-sync/group-mappings\" | jq_debug_pipe_dump)\"\n        if jq -e 'select((.groups | length) == 0)' <<< \"$team_mappings\" >/dev/null; then\n            if [ -z \"${QUIET:-}\" ]; then\n                timestamp \"WARNING: team '$team' is not sync'd' from an IdP!\"\n            fi\n            echo \"$team\"\n        fi\n    done\n    if jq -e 'length < 100' <<< \"$data\" >/dev/null; then\n        break\n    fi\ndone\n"
  },
  {
    "path": "github/github_teams_not_in_terraform.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 16:00:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all GitHub organization teams which are not found in ./*.tf code\n\nUseful to catch teams sync'd from an IdP like AAD that require referencing in team repo assignments\n\nRequires the GitHub Organization to be specified as an arg or found in \\$GITHUB_ORGANIZATION environment variable\n\nThis relies on the GitHub team slug matching the terraform team identifier\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\norg=\"${1:-${GITHUB_ORGANIZATION:-}}\"\n\nif is_blank \"$org\"; then\n    usage \"Organization not specifieid\"\nfi\n\nteams=\"$(\n    for((page=1; ; page++)); do\n        data=\"$(gh api \"/orgs/$org/teams?per_page=100&page=$page\")\"\n        if [ \"$(jq -r 'length' <<< \"$data\")\" -lt 1 ]; then\n            break\n        fi\n        jq -r '.[].slug' <<< \"$data\"\n        ((page+=1))\n        if [ \"$page\" -gt 1000 ]; then\n            die \"Hit 1000 pages of 100 teams per page, there is probably a bug\"\n        fi\n    done |\n    sort -f\n)\"\n\nfor team in $teams; do\n    grep -Eq '^[[:space:]]*resource[[:space:]]+\"github_team\"[[:space:]]+\"'\"$team\"'\"' ./*.tf ||\n    echo \"$team\"\ndone\n"
  },
  {
    "path": "github/github_url_clipboard.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-25 02:27:53 +0200 (Fri, 25 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCopies a GitHub URL file's contents to the clipboard,\nconverting the URL to a raw GitHub content URL where necessary\n\nDesigned to quickly and easily copy link content into HariSekhon/Knowledge-Base repo\neg. for HariSekhon/SQL-scripts references to be easily copyable code blocks without having to follow links\n\nGitHub URL can be passed as an arg or read from standard input\n\nTip: add this to a hotkey in your editor or IDE\n\nLimitation: only tested on public GitHub repos\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_to_repo_file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nurl=\"$(\"$srcdir/../bin/urlextract.sh\" \"$@\" | head -n1)\"\n\nif ! [[ \"$url\" =~ ^https://(github\\.com|raw.githubusercontent.com)/ ]]; then\n    usage \"Non-GitHub URL passed as first argument, must start with https://github.com or https://raw.githubusercontent.com/\"\nfi\n\nif [[ \"$url\" =~ github\\.com ]]; then\n    url_orig=\"$url\"\n    url=\"${url//github.com/raw.githubusercontent.com}\"\n    # need more advanced replace\n    # shellcheck disable=SC2001\n    #url=\"$(sed 's|/blob/[^/]*/||' <<< \"$url\")\"\n    url=\"${url//\\/blob\\//\\/}\"\n    log \"Converted '$url_orig'\n                            to '$url'\"\nfi\n\ncurl -sSf \"$url\" |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\n\nbytes=\"$(\"$srcdir/../bin/paste_from_clipboard.sh\" | wc -c | sed 's/[[:space:]]//g')\"\n\ntimestamp \"Copied '$url' to clipboard: $bytes bytes\"\n"
  },
  {
    "path": "github/github_user_followers.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 03:35:51 +0200 (Wed, 21 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the number of followers for a given username\n\nOutput format:\n\n<number_of_followers>\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nusername=\"${1:-}\"\n\ngh api \"users/$username\" --jq '.followers'\n"
  },
  {
    "path": "github/github_user_repos_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 03:35:51 +0200 (Wed, 21 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the total number of original source public GitHub repos for a given username\n\nOutput format:\n\n<number_of_original_public_repos>\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<username> <public_only_flag>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nusername=\"${1:-}\"\n\nis_public=\"${2:-}\"\n\nselect_private=\"select(.)\"\nif [ -n \"${is_public:-}\" ]; then\n    select_private=\"select(.isPrivate | not)\"\nfi\n\ngh repo list \\\n    ${username:+\"$username\"} \\\n    --limit 99999 \\\n    --json isFork,isPrivate \\\n    --jq \"\n        [\n            .[] |\n            select(.isFork | not) |\n            $select_private\n        ] |\n        length\n    \"\n"
  },
  {
    "path": "github/github_user_repos_forks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 03:35:51 +0200 (Wed, 21 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the total number of forks for all original source public GitHub repos for a given username\n\nOutput format:\n\n<total_repo_forks>\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nusername=\"${1:-}\"\n\ngh repo list \"$username\" --limit 99999 \\\n                         --json isFork,forkCount \\\n                         --jq '\n                            [.[] |\n                            select(.isFork | not) |\n                            .forkCount] |\n                            add'\n"
  },
  {
    "path": "github/github_user_repos_stars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 03:35:51 +0200 (Wed, 21 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the total number of stars for all original source public GitHub repos for a given username\n\nOutput format:\n\n<total_repo_stars>\n\n\nRequires GitHub CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<username>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nusername=\"${1:-}\"\n\ngh repo list \"$username\" --limit 99999 \\\n                         --json stargazerCount,isFork \\\n                         --jq '\n                            [.[] |\n                            select(.isFork | not) |\n                            .stargazerCount] |\n                            add'\n"
  },
  {
    "path": "github/gitio.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 16:05:56 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a git.io shortcut URL for your GitHub project\n\nBeware that the first shortname for a URL will stick, ignoring subsequent requests\n\nSee here for more details:\n\nhttps://github.blog/2011-11-10-git-io-github-url-shortener/\n\nhttps://aletheia.icu/~badt/git-io/\n\nhttps://gist.github.com/dikiaap/01f5e2ba3c738012aef0a8f524a6e207\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<github_user>/<github_repo> <shortname>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\ngithub_repo=\"$1\"\n\nshortname=\"$2\"\n\nif ! [[ \"$github_repo\" =~  / ]]; then\n    usage\nfi\n\nif ! [[ \"$github_repo\" =~ https?:// ]]; then\n    github_repo=\"https://github.com/$github_repo\"\nfi\n\ncurl https://git.io/ -i -F \"url=$github_repo\" -F \"code=$shortname\"\n"
  },
  {
    "path": "gitlab/gitlab_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 23:27:44 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#  args: /user | jq .\n#  args: /users/:id/projects | jq .\n#  args: /users/$(gitlab_api.sh /users?username=harisekhon | jq -r .[].id) | jq .\n#  args: /users/HariSekhon/projects | jq .\n#  args: /projects/:id | jq .\n#  args: /projects/HariSekhon%2FDevOps-Bash-tools/pipelines | jq .\n#  args: /projects/:id/pipelines | jq .\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the GitLab.com API (v4)\n\nAutomatically handles authentication via environment variable \\$GITLAB_TOKEN\n\nOptional: \\$GITLAB_USER - used for some replacement tokens, prevents having to search git remotes or query the API for it\n          \\$GITLAB_HOST - URL to point to self-hosts GitLab servers\n\nCan specify \\$CURL_OPTS for options to pass to curl, or pass them as arguments to the script\n\n\nYou must set up a personal access token here:\n\n    https://gitlab.com/-/user_settings/personal_access_tokens\n\n\nAPI Reference:\n\n    https://docs.gitlab.com/ee/api/api_resources.html\n\n\nExamples:\n\n\n# Get currently authenticated user:\n\n    ${0##*/} /user | jq .\n\n\n# List a user's GitLab projects (repos):\n\n    ${0##*/} /users/HariSekhon/projects | jq .\n\n\n# Update a project's description:\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools -X PUT -d 'description=test'\n\n    # Specify project ID or name (url-encoded otherwise will return 404 and fail to find project)\n\n\n# List a project's protected branches:\n\n    ${0##*/} /projects/HariSekhon%2fDevOps-Bash-tools/protected_branches | jq .\n\n\n# List a user's GitLab groups (contexts for sharing environment variables across projects):\n\n    ${0##*/} /groups | jq .\n\n\n# List a project's CI pipeline environment variables (careful this even returns 'masked' variable values in plaintext):\n\n    ${0##*/} /projects/HariSekhon%2fDevOps-Bash-tools/variables\n\n\n# Delete all environment variables for a given CI pipeline (see also gitlab_project_set_env_vars.sh to load them):\n\n    ${0##*/} /projects/HariSekhon%2fDevOps-Bash-tools/variables | jq -r '.[].key' | while read -r key; do ${0##*/} \\\"/projects/HariSekhon%2fDevOps-Bash-tools/variables/\\$key\\\" -X DELETE; done\n\n\n# List a group's CI pipeline environment variables (careful this even returns 'masked' variable values in plaintext):\n\n    ${0##*/} /groups/test6765/variables | jq .\n\n\n# List a project's CI pipeline runs, sorted by newest run first:\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools/pipelines\n\n\n# List a project's jobs (contains the status and pipeline reference):\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools/jobs\n\n\n# List a project's jobs for a specific pipeline:\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools/pipelines/<pipeline_id>/jobs\n\n\n# List a project's deployments:\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools/deployments\n\n\n# Get details for a single job:\n\n    ${0##*/} /projects/:id/jobs/:job_id\n\n\n# Get a project's remote mirrors:\n\n    ${0##*/} /projects/HariSekhon%2FDevOps-Bash-tools/remote_mirrors\n\n\n# Get log for a specific job:\n\n    ${0##*/} /projects/:id/jobs/:job_id/trace\n\n\n# List recent events such as pushes, by the currently authenticated user:\n\n    ${0##*/} /events\n\n\nFor convenience you can even copy and paste out of the documentation literally and have the script auto-determine the right settings (due to the context variation of the GitLAB API documentation tokens this is only done for users and projects only at this time)\n\nThe following placeholders are replaced if the environment variables are available or inferred fro mthe local git repo. The format can be one of {token}, <token> :token\n\n\\$GITLAB_USERNAME / \\$GITLAB_USER:                           owner, username, user, /users/:id\nlocal repo name of the current directory:                  repo\nlocal full 'user/repo' name of the current directory:      project, /projects/:id\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"${GL_HOST:-${GITLAB_HOST:-https://gitlab.com}}/api/v4\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\n# false positive, this works\n# shellcheck disable=SC2295\nurl_path=\"${url_path##$url_base}\"\nurl_path=\"${url_path##/}\"\n\n# for convenience of straight copying and pasting out - but documentation uses :id in different contexts to mean project id or user id so this is less useful than in github_api.sh\n\nuser=\"${GITLAB_USERNAME:-${GITLAB_USER:-}}\"\nif [ -z \"$user\" ]; then\n    user=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@gitlab\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//;s/:.*//' || :)\"\nfi\n\nif [ -n \"$user\" ]; then\n    export USERNAME=\"$user\"\nfi\n\nif [ -z \"${GITLAB_TOKEN:-}\" ]; then\n    GITLAB_TOKEN=\"$(git remote -v 2>/dev/null | awk '/https:\\/\\/.+@gitlab\\.com/{print $2; exit}' | sed 's|https://||;s/@.*//;s/.*://' || :)\"\nfi\n\nif [ -z \"$GITLAB_TOKEN\" ]; then\n    usage \"GITLAB_TOKEN not defined and could not infer from local repo\"\nfi\n\nproject=\"$(git_repo 2>/dev/null || :)\"\nrepo=\"$(sed 's/.*\\///' <<< \"$project\")\"\nproject=\"${project//\\//%2F}\" # cheap url encode slash\n\nif [ -n \"$user\" ]; then\n    url_path=\"${url_path/\\{owner\\}/$user}\"\n    url_path=\"${url_path/<owner>/$user}\"\n    url_path=\"${url_path/:owner/$user}\"\n    url_path=\"${url_path/\\{username\\}/$user}\"\n    url_path=\"${url_path/<username>/$user}\"\n    url_path=\"${url_path/:username/$user}\"\n    url_path=\"${url_path/\\{user\\}/$user}\"\n    url_path=\"${url_path/<user>/$user}\"\n    url_path=\"${url_path/:user/$user}\"\nfi\nurl_path=\"${url_path/\\{repo\\}/$repo}\"\nurl_path=\"${url_path/<repo>/$repo}\"\nurl_path=\"${url_path/:repo/$repo}\"\nurl_path=\"${url_path/\\{project\\}/$project}\"\nurl_path=\"${url_path/<project>/$project}\"\nurl_path=\"${url_path/:project/$project}\"\nurl_path=\"${url_path/projects\\/:id/projects\\/$project}\"\nurl_path=\"${url_path/users\\/:id/users\\/$user}\"\n\nexport TOKEN=\"$GITLAB_TOKEN\"\n\n# can also leave out to use OAuth compliant header \"Authorization: Bearer <token>\"\nexport CURL_AUTH_HEADER=\"Private-Token:\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "gitlab/gitlab_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo user={user} name={name} repo={project}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-30 10:08:07 +0100 (Sun, 30 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each GitLab project / repo\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the repo names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{username}, {user}    =>    your authenticated user\n{name}                =>    the repo / project name without the user prefix\n{project}, {repo}     =>    the repo / project name with the user prefix\n\neg.\n    ${0##*/} echo user={user} name={name} repo={project}\n    ${0##*/} echo user={user} name={name} repo={repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [ -n \"${GITLAB_USER:-}\" ]; then\n    user=\"$GITLAB_USER\"\nelse\n    # get currently authenticated user\n    user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\nfi\n\nget_repos(){\n    local page=1\n    while true; do\n        if ! output=\"$(\"$srcdir/gitlab_api.sh\" \"/users/$user/projects?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq -r '.[] | select(.fork | not) | [.path, .path_with_namespace] | @tsv' <<< \"$output\"\n        ((page+=1))\n    done\n}\n\nget_repos |\nwhile read -r name repo; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $repo\" >&2\n    echo \"# ============================================================================ #\" >&2\n    echo >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{username\\}/$user}\")\n    cmd=(\"${cmd[@]//\\{user\\}/$user}\")\n    cmd=(\"${cmd[@]//\\{project\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "gitlab/gitlab_get_user_ssh_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 18:13:02 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/users.html#list-ssh-keys-for-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\nusage(){\n    cat <<EOF\nFetches a given GitLab user's public SSH key(s) via the GitLab API\n\nUser can be given as first argument, otherwise falls back to using environment variables \\$GITLAB_USER or \\$USER\n\nSSH Keys can be found in the Web UI here:\n\n    https://gitlab.com/profile/keys\n\n\n${0##*/} <user>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 1 ]; then\n    usage\nelif [ $# -eq 1 ]; then\n    user=\"$1\"\nelif [ -n \"${GITLAB_USER:-}\" ]; then\n    user=\"$GITLAB_USER\"\nelif [ -n \"${USER:-}\" ]; then\n    if [[ \"$USER\" =~ hari|sekhon ]]; then\n        user=harisekhon\n    else\n        user=\"$USER\"\n    fi\nelse\n    usage\nfi\n\n# XXX: not handling paging because if you have > 100 SSH keys I want to know what is going on first!\n\necho \"# Fetching SSH Public Key(s) from GitLab for account:  $user\" >&2\necho \"#\" >&2\n# authenticated query is not necessary, gets more information than GitHub regardless, which I use to add standard SSH key description suffix back (useful if loading these keys to other systems to know what their descriptions are)\ncurl -sS --fail \"https://gitlab.com/api/v4/users/$user/keys\" |\njq -r '.[] | [.key, .title] | @tsv' |\ntr '\\t' ' '\n"
  },
  {
    "path": "gitlab/gitlab_get_user_ssh_public_keys2.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 18:25:33 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/users.html#list-ssh-keys\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    cat <<EOF\nFetches a GitLab user's public SSH key(s) via HTTP\n\nUser can be given as first argument, or environment variables \\$GITLAB_USER or \\$USER\n\nTechnically should use the GitLab API, see instead:  gitlab_ssh_get_user_public_keys.sh\n\n${0##*/} <user>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 1 ]; then\n    usage\nelif [ $# -eq 1 ]; then\n    user=\"$1\"\nelif [ -n \"${GITLAB_USER:-}\" ]; then\n    user=\"$GITLAB_USER\"\nelif [ -n \"${USER:-}\" ]; then\n    if [[ \"$USER\" =~ hari|sekhon ]]; then\n        user=harisekhon\n    else\n        user=\"$USER\"\n    fi\nelse\n    usage\nfi\n\n\necho \"# Fetching SSH Public Key(s) from GitLab for account:  $user\" >&2\necho \"#\" >&2\ncurl -sS --fail \"https://gitlab.com/$user.keys\"\n"
  },
  {
    "path": "gitlab/gitlab_group_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: testgroup haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-13 18:10:59 +0000 (Thu, 13 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/group_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates GitLab CI group-level masked environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nIf GITLAB_VARIABLES_PROTECTED=1 is set in the environment, then will create a protected environment variable which is only available to protected branches or tags\nIf GITLAB_VARIABLES_UNMASKED=1 is set in the environment, then environment variables will not be masked\n\nExamples:\n\n    ${0##*/} testgroup AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} testgroup\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} testgroup\n\n\nXXX: Caveat - GitLab only masks variables 8 characters or longer and they are retrievable in plaintext via the API - better to use a secrets value if possible\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<group_slug_or_id> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ngroup_name=\"$1\"\nshift || :\n\ntimestamp \"retrieving group id for group name '$group_name'\"\n# paginate groups to find the group id of the matching group_name\npage=1\nwhile true; do\n    output=\"$(\"$srcdir/gitlab_api.sh\" \"/groups?page=$page&per_page=100\")\"\n    group_id=\"$(jq_debug_pipe_dump <<< \"$output\" | jq -r \".[] | select(.name == \\\"$group_name\\\") | .id\")\"\n    if [ -n \"$group_id\" ]; then\n        break\n    fi\n    if ! jq -e '.[]' <<< \"$output\" &>/dev/null; then\n        break\n    fi\n    ((page+=1))\ndone\n\nif [ -z \"$group_id\" ]; then\n    die \"Failed to find ID for group name '$group_name'\"\nfi\n\nexisting_env_vars=\"$(\"$srcdir/gitlab_api.sh\" \"groups/$group_id/variables\" | jq -r '.[].key')\"\n\nprotected=false\nif [ -n \"${GITLAB_VARIABLES_PROTECTED:-}\" ]; then\n    protected=true\nfi\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    local masked=true\n    # shellcheck disable=SC2154\n    if [ \"${GITLAB_VARIABLES_UNMASKED:-}\" = 1 ]; then\n        echo \"WARNING: value for key '$key' will not be masked because GITLAB_VARIABLES_UNMASKED is set in the environment\" >&2\n        masked=false\n    elif [ \"${#value}\" -lt 8 ]; then  # avoids 400 errors from the API if sending < 8 chars with masked=true\n        echo \"WARNING: value for key '$key' is less than 8 characters so can't be masked in GitLab\" >&2\n        masked=false\n    fi\n    # shellcheck disable=SC2154\n    if grep -Fxq \"$key\" <<< \"$existing_env_vars\"; then\n        timestamp \"updating GitLab environment variable '$key' in group name '$group_name' (id: '$group_id')\"\n        \"$srcdir/gitlab_api.sh\" \"groups/$group_id/variables/$key?masked=$masked&protected=$protected\" -X PUT \\\n            -F \"value=$value\" \\\n            -H 'Content-Type: multipart/form-data'\n    else\n        timestamp \"adding GitLab environment variable '$key' to group '$group_name' (id: '$group_id')\"\n        \"$srcdir/gitlab_api.sh\" \"groups/$group_id/variables?masked=$masked&protected=$protected\" -X POST \\\n            -F \"key=$key\" \\\n            -F \"value=$value\" \\\n            -H 'Content-Type: multipart/form-data'\n            #-F \"environment_scope=*\"  # only available in Premium\n            # must override the default -H 'Content-Type: application/json' in curl_api_opts() in lib/utils.sh  to avoid 400 or 406 errors from the API\n    fi |\n    jq '.value = \"REDACTED\"'  # echo's back the variable value in plaintext\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "gitlab/gitlab_install_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: gitlab-org/cli glab_{version}_macOS_x86_64.tar.gz latest bin/glab\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls a GitLab project repo's release binary to \\$HOME/bin or /usr/local/bin, unpacking it from a tarball or zip if necessary\n\nIf the release file is a tarball or zip file then it'll auto-unpack it, but you must specify the path to the binary in the unpack\n\nIf version is not specified, determine the latest release and installs that\n\nIf the release URL title/path is more complicated than the convention of following the version number, such as is the case for Kustomize, then you'd need to call install_binary.sh with the URL path instead of using this script, see install/install_kustomize.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo> <release_file_tarball_zip> [<version> <path/to/unpacked/binary> <install_path>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nowner_repo=\"$1\"\n\nrelease_file=\"$2\"\n\nversion=\"${3:-latest}\"\n\nbinary=\"${4:-}\"\n\ninstall_path=\"${5:-}\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitLab API\"\n    version=\"$(\"$srcdir/gitlab_project_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nrelease_file=\"${release_file//\\{version\\}/${version#v}}\"\nbinary=\"${binary//\\{version\\}/${version#v}}\"\n\n#                                        https://gitlab.com/gitlab-org/cli/-/releases/v1.36.0/downloads/glab_1.36.0_macOS_x86_64.tar.gz\n\"$srcdir/../packages/install_binary.sh\" \"https://gitlab.com/$owner_repo/-/releases/$version/downloads/$release_file\" ${binary:+\"$binary\"} ${install_path:+\"$install_path\"}\n"
  },
  {
    "path": "gitlab/gitlab_project_create_import.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-21 18:37:01 +0000 (Mon, 21 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a GitLab project (repo) as a mirror import from another repo URL\n\nIf on GitLab Premium will configure auto-mirroring to stay in-sync too (can only manually configure for public repos on free tier, API doesn't support configuring even public repos on free)\n\nURL can contain authentication information in the form:\n\n    https://<username>:<password>@<url>/<owner>/<repo>\neg.\n    ${0##*/} devops-bash-tools https://harisekhon:mypass@github.com/harisekhon/devops-bash-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_name> <repo_url_to_mirror>\"\n\nhelp_usage \"$@\"\n\nname=\"$1\"\nimport_url=\"$2\"\n\n#if [ -n \"${GITLAB_USER:-}\" ]; then\n#    user=\"$GITLAB_USER\"\n#else\n#    # get currently authenticated user\n#    user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\n#fi\n\n#user=\"$(\"$srcdir/gitlab_api.sh\" \"/users?username=$user\" | jq -r '.[0].id')\"\n\ntimestamp \"Creating GitLab repo '$name'\"\n#\"$srcdir/gitlab_api.sh\" \"/projects/user/$user\" -X POST \\\n# this will mirror automatically on Premium\n\"$srcdir/gitlab_api.sh\" \"/projects\" -X POST \\\n    -d \"{\n    \\\"name\\\": \\\"$name\\\",\n    \\\"import_url\\\": \\\"$import_url\\\",\n    \\\"mirror\\\": true\n}\"\necho >&2\n\n# not needed, create mirror option does the same\n# XXX: only available in Premium unfortunately\n#timestamp \"Configuring repo mirroring from '$import_url'\"\n#\"$srcdir/gitlab_api.sh\" \"/projects/$user/$name/mirror/pull\" -X POST \\\n#    -d \"{\n#    \\\"import_url\\\": \\\"$import_url\\\",\n#    \\\"mirror\\\": true\n#}\"\n"
  },
  {
    "path": "gitlab/gitlab_project_latest_release.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: gitlab-org/cli\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 18:33:13 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the latest release name/version for a given 'owner/repo' project via the GitLab API\n\nIf a project repo has no releases, gets a 404 error\n\nRequires curl and jq to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<owner>/<repo>\"\n\nhelp_usage \"$@\"\n\ncheck_bin curl\ncheck_bin jq\n\nmin_args 1 \"$@\"\n\nowner_repo=\"$1\"\n\nif ! is_github_owner_repo \"$owner_repo\"; then\n    die \"Invalid owner/repo argument given: $owner_repo\"\nfi\n\nowner_repo=\"${owner_repo//\\//%2F}\"\n\nif [ -n \"${GITHUB_TOKEN:-}\" ]; then\n    CURL_OPTS=\"-ssL --fail\" \"$srcdir/gitlab_api.sh\" \"/projects/$owner_repo/releases/permalink/latest\"\nelse\n    curl -sSL --fail \"https://api.github.com/projects/$owner_repo/permalink/releases/latest\"\nfi |\njq_debug_pipe_dump |\njq -e -r .tag_name ||\ndie \"Failed to determine latest release\"\n"
  },
  {
    "path": "gitlab/gitlab_project_mirrors.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-31 18:00:41 +0100 (Mon, 31 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists GitLab projects (repos) and whether or not they are mirrors\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nif [ -n \"${GITLAB_USER:-}\" ]; then\n    user=\"$GITLAB_USER\"\nelse\n    # get currently authenticated user\n    user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\nfi\n\n{\n    page=1\n    while true; do\n        if ! output=\"$(\"$srcdir/gitlab_api.sh\" \"/users/$user/projects?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq -r '.[] | [.path, if has(\"mirror\") then .mirror else false end ] | @tsv' <<< \"$output\"\n        ((page+=1))\n    done\n} | column -t\n"
  },
  {
    "path": "gitlab/gitlab_project_protect_branches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\ndefault_branches_to_protect=\"\n    main\n    master\n    develop\n    dev\n    staging\n    production\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables branch protection for one or more branches in the given GitLab project (repo) to prevents deleting the branch or force pushing over it\n\nIf no branch is specified, the applies branches protections to any of the following branches if they're found:\n$default_branches_to_protect\n\nProject can be the full project name (eg. HariSekhon/DevOps-Bash-tools) or the project ID\n\nProject username prefix can be omitted, will use \\$GITLAB_USER if available, otherwise will query the GitLab API to determine the user owning the \\$GITLAB_TOKEN\n\nAutomatically url encodes the project name and description for you since the GitLab API will return 404 and fail to find the project name if not url encoded\n\nUses the adajcent script gitlab_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project> [<branch> <branch2> <branch3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject=\"$1\"\nshift || :\n\nif ! [[ \"$project\" =~ / ]]; then\n    log \"No username prefix in project '$project', will auto-add it\"\n    log \"Attempting to infer username\"\n    if [ -n \"${GITLAB_USER:-}\" ]; then\n        gitlab_user=\"$GITLAB_USER\"\n        log \"Using username '$gitlab_user' from \\$GITLAB_USER\"\n    else\n        log \"Querying GitLab API for currently authenticated username\"\n        gitlab_user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\n        log \"GitLab API returned username '$gitlab_user'\"\n    fi\n    project=\"$gitlab_user/$project\"\nfi\n\n# url-encode project name otherwise GitLab API will fail to find project and return 404\nproject_name=\"$project\"\nproject=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$project\")\"\n\nallow_force_push=false\ncode_owner_approval_required=true\n\nprotect_project_branch(){\n    local branch=\"$1\"\n    timestamp \"protecting GitLab project '$project_name' branch '$branch'\"\n    # gets  409 error if there is already branch protection, so remove and reapply it to ensure it is applied with these settings\n    \"$srcdir/gitlab_api.sh\" \"/projects/$project/protected_branches/$branch\" -X DELETE &>/dev/null || :\n    #if \"$srcdir/gitlab_api.sh\" \"/projects/$project/protected_branches/$branch\" &>/dev/null; then\n        #timestamp \"patching existing GitLab branch protection\"\n        # XXX: GitLab API ignores PATCH'ing allow_force_push, only works for code_owner_approval_required - the only way to enforce allow_force_push=false is to remove and recreate the branch protection\n        #\"$srcdir/gitlab_api.sh\" \"/projects/$project/protected_branches/$branch?allow_force_push=$allow_force_push&code_owner_approval_required=$code_owner_approval_required\" -X PATCH >/dev/null\n    #fi\n    \"$srcdir/gitlab_api.sh\" \"/projects/$project/protected_branches?name=$branch&allow_force_push=$allow_force_push&code_owner_approval_required=$code_owner_approval_required\" -X POST >/dev/null\n    timestamp \"protection applied to branch '$branch'\"\n}\n\nget_gitlab_project_branches(){\n    local project=\"$1\"\n    local branches\n    local page=1\n    for ((page=1; page < 100; page++)); do\n        branches=\"$(\"$srcdir/gitlab_api.sh\" \"/projects/$project/repository/branches?page=$page&per_page=100\" | jq_debug_pipe_dump | jq -r '.[].name')\"\n        if [ -z \"$branches\" ]; then\n            break\n        fi\n        echo \"$branches\"\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for branch in \"$@\"; do\n        protect_project_branch \"$branch\"\n    done\nelse\n    timestamp \"no branches specified, getting branch list\"\n    branches=\"$(get_gitlab_project_branches \"$project\")\"\n    for branch in $default_branches_to_protect; do\n        timestamp \"checking for branch '$branch'\"\n        if grep -Fxq \"$branch\" <<< \"$branches\"; then\n            timestamp \"protecting branch '$branch'\"\n            protect_project_branch \"$branch\"\n        fi\n    done\nfi\n"
  },
  {
    "path": "gitlab/gitlab_project_set_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the description of a GitLab.com project via the GitLab API\n\nUses the adajcent script gitlab_api.sh, see there for authentication details\n\n\\$CURL_OPTS can be set to provide extra arguments to curl\n\n\nProject can be the full project name (eg. HariSekhon/DevOps-Bash-tools) or the project ID\n\nProject username prefix can be omitted, will use \\$GITLAB_USER if available, otherwise will query the GitLab API to determine it\n\nAutomatically url encodes the project name and description for you since the GitLab API will return 404 and fail to find the project name if not url encoded\n\n\nExample:\n\n    ${0##*/} HariSekhon/DevOps-Bash-tools    my new description\n\n\nIf no args are given, will read project and description from standard input for easy chaining with other tools, can easily update multiple repositories this way, one project + description per line:\n\n    echo <project> <description> | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project> <description>\"\n\nhelp_usage \"$@\"\n\n#min_args 2 \"$@\"\n\nset_project_description(){\n    local project=\"$1\"\n    local description=\"${*:2}\"\n\n    if ! [[ \"$project\" =~ / ]]; then\n        log \"No username prefix in project '$project', will auto-add it\"\n        # reuse gitlab_user between function calls for efficiency to save additional queries to the GitLab API\n        if [ -z \"${gitlab_user:-}\" ]; then\n            log \"Attempting to infer username\"\n            if [ -n \"${GITLAB_USER:-}\" ]; then\n                gitlab_user=\"$GITLAB_USER\"\n                log \"Using username '$gitlab_user' from \\$GITLAB_USER\"\n            else\n                log \"Querying GitLab API for currently authenticated username\"\n                gitlab_user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\n                log \"GitLab API returned username '$gitlab_user'\"\n            fi\n        fi\n        project=\"$gitlab_user/$project\"\n    fi\n\n    timestamp \"Setting GitLab project '$project' description to '$description'\"\n\n    # url-encode project name otherwise GitLab API will fail to find project and return 404\n    project=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$project\")\"\n    # don't URL encode this now that it is inside JSON\n    #description=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$description\")\"\n    # just strip quotes to protect the JSON\n    description=\"${description//\\\"/}\"\n\n    # this used to work with just -d \"description=$description\" when Accept and Content-Type headers were omitted\n    # but since curl_api_opts auto-sets headers to application/json this must be json or else get 400 bad request error\n    \"$srcdir/gitlab_api.sh\" \"/projects/$project\" -X PUT --data \"{ \\\"description\\\": \\\"$description\\\" }\" >/dev/null\n}\n\nif [ $# -gt 0 ]; then\n    if [ $# -lt 2 ]; then\n        usage\n    fi\n    set_project_description \"$@\"\nelse\n    while read -r project description; do\n        [ -n \"$project\" ] || continue\n        [ -n \"$description\" ] || continue\n        set_project_description \"$project\" \"$description\"\n    done\nfi\n"
  },
  {
    "path": "gitlab/gitlab_project_set_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon/DevOps-Bash-tools haritest=stuff\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-03 17:41:23 +0000 (Fri, 03 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/project_level_variables.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates GitLab CI project-level masked environment variable(s) from args or stdin\n\nIf no second argument is given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nIf GITLAB_VARIABLES_PROTECTED=1 is set in the environment, then will create a protected environment variable which is only available to protected branches or tags\nIf GITLAB_VARIABLES_UNMASKED=1 is set in the environment, then environment variables will not be masked\n\nExamples:\n\n    ${0##*/} github/HariSekhon/DevOps-Bash-tools AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} HariSekhon/DevOps-Bash-tools\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} HariSekhon/DevOps-Bash-tools\n\n\nXXX: Caveat - GitLab only masks variables 8 characters or longer and they are retrievable in plaintext via the API - better to use a secrets value if possible\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_slug_or_id> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject_slug=\"$1\"\nshift || :\n\nif ! [[ \"$project_slug\" =~ ^[[:digit:]]+$|^[[:alnum:]-]+/[[:alnum:]-]+$ ]]; then\n    usage \"project-slug given '$project_slug' does not conform to id or <user_or_org>/<repo> format\"\nfi\n\nproject_slug=\"${project_slug//\\//%2F}\"\n\nexisting_env_vars=\"$(\"$srcdir/gitlab_api.sh\" \"projects/$project_slug/variables\" | jq -r '.[].key')\"\n\nprotected=false\nif [ -n \"${GITLAB_VARIABLES_PROTECTED:-}\" ]; then\n    protected=true\nfi\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    local masked=true\n    # shellcheck disable=SC2154\n    if [ \"${GITLAB_VARIABLES_UNMASKED:-}\" = 1 ]; then\n        echo \"WARNING: value for key '$key' will not be masked because GITLAB_VARIABLES_UNMASKED is set in the environment\" >&2\n        masked=false\n    elif [ \"${#value}\" -lt 8 ]; then  # avoids 400 errors from the API if sending < 8 chars with masked=true\n        echo \"WARNING: value for key '$key' is less than 8 characters so can't be masked in GitLab\" >&2\n        masked=false\n    fi\n    # shellcheck disable=SC2154\n    if grep -Fxq \"$key\" <<< \"$existing_env_vars\"; then\n        timestamp \"updating GitLab environment variable '$key' in project '$project_slug'\"\n        \"$srcdir/gitlab_api.sh\" \"projects/$project_slug/variables/$key?masked=$masked&protected=$protected\" -X PUT \\\n            -F \"value=$value\" \\\n            -H 'Content-Type: multipart/form-data'\n    else\n        timestamp \"adding GitLab environment variable '$key' to project '$project_slug'\"\n                                          # could also use a project_id\n        \"$srcdir/gitlab_api.sh\" \"projects/$project_slug/variables?masked=$masked&protected=$protected\" -X POST \\\n            -F \"key=$key\" \\\n            -F \"value=$value\" \\\n            -H 'Content-Type: multipart/form-data'\n            # must override the default -H 'Content-Type: application/json' in curl_api_opts() in lib/utils.sh  to avoid 400 or 406 errors from the API\n    fi |\n    jq '.value = \"REDACTED\"'  # echo's back the variable value in plaintext\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "gitlab/gitlab_pull_mirror.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-01 13:54:41 +0100 (Tue, 01 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTrigger a GitLab mirror pull of a given project's repo\n\nProject can be specified as a Project ID or a Project name\n\nIf no repo is given, tries to determine from the current git checkout\n\nOutputs '200' indicating OK if successful\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ -n \"${GITLAB_USER:-}\" ]; then\n    user=\"$GITLAB_USER\"\nelse\n    # get currently authenticated user\n    user=\"$(\"$srcdir/gitlab_api.sh\" /user | jq -r .username)\"\nfi\n\nif [ $# -gt 0 ]; then\n    project=\"$1\"\n    shift || :\nelse\n    project=\"$(git_repo)\"\nfi\n\nif [[ \"$project\" =~ / ]]; then\n    project=\"${project//\\//%2F}\"\nelif [[ \"$project\" =~ ^[[:digit:]]$ ]]; then\n    :\nelse\n    project=\"$user%2F$project\"\nfi\n\n\"$srcdir/gitlab_api.sh\" \"/projects/$project/mirror/pull\" -X POST \"$@\"\necho\n"
  },
  {
    "path": "gitlab/gitlab_push_mr.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 21:58:25 +0000 (Fri, 16 Feb 2024)\n#\n#  https://gitlab.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gitlab.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPushes the current branch to GitLab origin, sets upstream branch, then raises a Pull Request to the given or default branch\n\nIf \\$GITLAB_MERGE_PULL_REQUEST=true then will automatically merge the pull request as well\n\n\"\n#If \\$GITLAB_MERGE_PULL_REQUEST_AS_ADMIN=true then will merge as admin to bypass branch merge protection\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<target_base_branch> <title> <description>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 3 \"$@\"\n\ncheck_gitlab_origin\n\nbase_branch=\"${1:-$(default_branch)}\"\ntitle=\"${2:-}\"\ndescription=\"${3:-}\"\n\ngit push --set-upstream origin \"$(current_branch)\"\n\necho\n\ncurrent_branch=\"$(current_branch)\"\n\n# check for existing MR first\nexisting_mr=\"$(glab mr list -s \"$current_branch\" -t \"$base_branch\" | grep -F \"$current_branch\" || :)\"\nif [ -n \"$existing_mr\" ]; then\n    timestamp \"Merge Request already exists, skipping creation\"\n    echo\n    echo \"$existing_mr\"\n    id=\"${existing_mr%%[[:space:]]*}\"\n    id=\"${id#!}\"\n    url=\"$(glab mr view \"$id\" | grep '^url:[[:space:]]' | sed 's/^url:[[:space:]]*//')\"\nelse\n    output=\"$(glab mr create ${title:+--title \"$title\"} ${description:+--description \"$description\"} \"${description:---fill}\" --yes --source-branch \"$current_branch\" --target-branch \"$base_branch\" --remove-source-branch)\"\n    echo \"$output\"\n    echo\n    if [ -z \"$output\" ]; then\n        die \"Pull request not created\"\n    fi\n    url=\"$(parse_pull_request_url \"$output\")\"\nfi\necho\n\nif [ \"${GITLAB_MERGE_PULL_REQUEST:-}\" = true ]; then\n    #args=\"\"\n    #if [ \"${GITLAB_MERGE_PULL_REQUEST_AS_ADMIN:-}\" = true ]; then\n    #    args=\"--admin\"\n    #fi\n    timestamp \"Merging Pull Request:  $url\"\n    id=\"${url##*/}\"\n    glab mr merge \"$id\" --yes\nfi\n\nif is_mac; then\n    echo \"Opening Pull Request\"\n    open \"$url\"\nelif [ -n \"${BROWSER:-}\" ]; then\n    echo \"Opening Pull Request using \\$BROWSER\"\n    \"$BROWSER\" \"$url\"\nelse\n    echo \"\\$BROWSER environment variable not set and not on Mac to use default browser, not opening browser\"\nfi\n"
  },
  {
    "path": "gitlab/gitlab_push_mr_preview.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-15 11:16:44 +0100 (Fri, 15 Jul 2022)\n#\n#  https://gitlab.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/gitlab.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPushes the current branch to GitLab origin, setting upstream branch, then opens a Pull Request preview from current to default branch\n\nAssumes that GitLab is the remote origin, and checks for this for safety\n\n\nRequires GitLab CLI to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\ncheck_gitlab_origin\n\ncurrent_branch=\"$(current_branch)\"\n\ngit push --set-upstream origin \"$current_branch\"\n\nglab mr create --web --fill\n"
  },
  {
    "path": "gitlab/gitlab_ssh_add_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/users.html#add-ssh-key-for-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds the given SSH public key(s) to the currently authenticated GitLab account via the GitLab API\n\nIf no SSH public key is given, defaults to using ~/.ssh/id_rsa.pub\n\nIf a dash is given, reads the SSH public key(s) from standard input, ignoring comment lines so you can chain with tools like the adjacent scripts:\n\n    github_ssh_get_user_public_keys.sh\n    gitlab_ssh_get_user_public_keys.sh\n    github_ssh_get_public_keys.sh\n    gitlab_ssh_get_public_keys.sh\n    bitbucket_ssh_get_public_keys.sh\n\nWill return a 400 error if the SSH public key is invalid or has already been added.\nThe script detects already existing keys and skips them to avoid this error\n\nSSH Keys can be found in the Web UI here:\n\n    https://gitlab.com/profile/keys\n\nIf you get a 404 error it is likely that your \\$GITLAB_TOKEN doesn't have the 'api' (full) permissions\nYou can regenerate a token with the right permissions here:\n\n    https://gitlab.com/profile/personal_access_tokens\n\nUses the adjacent script gitlab_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<ssh_public_key_file> [<ssh_public_key_file2>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\necho \"# Getting existing SSH public keys to skip any keys that already exist (to avoid 400 errors)\" >&2\nssh_public_keys=\"$(\"$srcdir/gitlab_ssh_get_public_keys.sh\")\"\n\nadd_ssh_public_keys_from_file(){\n    local public_key_file=\"$1\"\n    if [ \"$public_key_file\" = \"-\" ]; then\n        public_key_file=/dev/stdin\n    fi\n    sed 's/#.*//; /^[[:space:]]*$/d' \"$public_key_file\" |\n    while read -r public_key; do\n        [[ \"$public_key\" =~ ^ssh- ]] || die \"invalid SSH key in file '$public_key_file': $public_key\"\n        add_ssh_public_key \"$public_key\"\n    done\n}\n\nadd_ssh_public_key(){\n    local public_key=\"$1\"\n    key=\"$(awk '{print $1\" \"$2}' <<< \"$public_key\")\"\n    comment=\"$(awk '{$1=\"\"; $2=\"\"; print}' <<< \"$public_key\" | sed 's/^[[:space:]]*//; s/[[:space:]]*$//')\"\n    if grep -Fq \"$key\" <<< \"$ssh_public_keys\"; then\n        timestamp \"SSH public key already exists, skipping: '$public_key'\"\n        return\n    fi\n    timestamp \"adding SSH public key to currently authenticated GitLab account: '$public_key'\"\n    # don't assume the key is from the local machine, it's likely this will be used by admins or chained with other tools that download from GitHub / BitBucket and upload to GitLab for synchronization, in which case the local machine is not related to the key\n    #\"$srcdir/gitlab_api.sh\" \"/user/keys\" -X POST -H \"Content-Type: application/json\" -d '{\"key\": \"'\"$public_key\"'\", \"title\": \"'\"$USER@${HOSTNAME:-$(hostname)}\"'\"}' > /dev/null  # JSON of the newly added key\n    \"$srcdir/gitlab_api.sh\" \"/user/keys\" -X POST -H \"Content-Type: application/json\" -d '{\"title\": \"'\"$comment\"'\", \"key\": \"'\"$public_key\"'\"}' |\n    jq -r '\"SSH public key \\\"\" + .title + \"\\\" was added to account as key id \" + (.id|tostring)' |\n    timestamp \"$(cat)\"\n    echo >&2\n}\n\nif [ $# -gt 0 ]; then\n    for filename; do\n        add_ssh_public_keys_from_file \"$filename\"\n    done\nelse\n    add_ssh_public_keys_from_file ~/.ssh/id_rsa.pub\nfi\n"
  },
  {
    "path": "gitlab/gitlab_ssh_delete_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://gitlab.com/harisekhon/bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/users.html#delete-ssh-key-for-current-user\n#\n# Could also delete for a named user if you have admin permissions:\n#\n# https://docs.gitlab.com/ee/api/users.html#delete-ssh-key-for-given-user\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes given SSH public key(s) from the currently authenticated GitLab account via the GitLab API\n\nAccepts either a key ID or a case insensitive ERE regex to match against the key's title.\nIf multiple key titles match the given regex, deletes all of them.\nIf no keys match, does nothing.\n\nSSH Keys can be found in the Web UI here:\n\n    https://gitlab.com/profile/keys\n\nIf you get a 404 error it is likely that your \\$GITLAB_TOKEN doesn't have the 'api' (full) permissions\nYou can regenerate a token with the right permissions here:\n\n    https://gitlab.com/profile/personal_access_tokens\n\nUses the adjacent script gitlab_api.sh, see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<key_id_or_title_regex>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfilter=\"$1\"\n\ntimestamp \"Getting SSH public keys\"\n\"$srcdir/gitlab_api.sh\" \"/user/keys\" |\njq -r '.[] | [.id, .title] | @tsv' |\nwhile read -r key_id title; do\n    if [ \"$key_id\" = \"$filter\" ] ||\n       grep -Eqi \"$filter\" <<< \"$title\"; then\n        \"$srcdir/gitlab_api.sh\" \"/user/keys/$key_id\" -X DELETE\n        timestamp \"Deleted SSH key with id '$key_id' and title '$title'\"\n    fi\ndone\n"
  },
  {
    "path": "gitlab/gitlab_ssh_get_public_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 18:13:02 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/users.html#list-ssh-keys\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    cat <<EOF\nFetches the currently authenticated GitLab user's public SSH key(s) via the GitLab API\n\nSSH Keys can be found in the Web UI here:\n\n    https://gitlab.com/profile/keys\n\nIf you get a 404 error it is likely that your \\$GITLAB_TOKEN doesn't have the 'read_api' or 'api' (full) permissions\nYou can regenerate a token with the right permissions here:\n\n    https://gitlab.com/profile/personal_access_tokens\n\nTo get other named users's public SSH keys see:\n\n    gitlab_ssh_get_user_public_keys.sh\n\n\n${0##*/}\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        *)  usage\n            ;;\n    esac\ndone\n\n# XXX: not handling paging because if you have > 100 SSH keys I want to know what is going on first!\n\necho \"# Fetching SSH Public Key(s) from GitLab for currently authenticated account\" >&2\necho \"#\" >&2\n# authenticated query\n\"$srcdir/gitlab_api.sh\" \"/user/keys\" |\njq -r '.[] | [.key, .title] | @tsv'\n"
  },
  {
    "path": "gitlab/gitlab_validate_ci_yaml.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 18:39:12 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.gitlab.com/ee/api/lint.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nValidates a given GitLab CI config via the GitLab APIv4\n\nGitlab now requires authentication for this endpoint, see gitlab_api.sh for details\n\nExample:\n\n${0##*/} .gitlab-ci.yml\n\n\nIf no arg is given, will look for .gitlab-ci.yml in the current directory\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path/to/.gitlab-ci.yml\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ngitlab_ci_yml=\".gitlab-ci.yml\"\n\nif [ $# -gt 0 ]; then\n    filename=\"$1\"\n    if ! [[ \"$filename\" =~ $gitlab_ci_yml$ ]]; then\n        die \"invalid filename given, must be called '$gitlab_ci_yml', instead got:  $filename\"\n    fi\nelif [ -f \"$gitlab_ci_yml\" ]; then\n    filename=\"$gitlab_ci_yml\"\nelse\n    usage \"no filename given and no file '$gitlab_ci_yml' found in the current directory\"\nfi\n\ncontent=\"$(sed 's/#.*//; /^[[:space:]]*$/d' \"$filename\")\"\n\njson_content=\"$(\"$srcdir/../bin/yaml2json.sh\" <<< \"$content\")\"\n\n# escape quotes and must not have any indentations and be on one line\njson_content_escaped=\"$(sed 's/\"/\\\\\"/g;s/^[[:space:]]*//;' <<< \"$json_content\" | tr -d '\\n')\"\n\n# now needs to be authenticated\n#result=\"$(curl -sS --fail https://gitlab.com/api/v4/ci/lint -X POST --header \"Content-Type: application/json\" --data \"{\\\"content\\\": \\\"$json_content_escaped\\\"}\" | jq -r .status)\"\nresult=\"$(\"$srcdir/gitlab_api.sh\" /ci/lint -X POST -d \"{\\\"content\\\": \\\"$json_content_escaped\\\"}\" | jq -r .status)\"\necho \"$result\"\nif [ \"$result\" != valid ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "hadolint.yaml",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-01 16:14:15 +0100 (Tue, 01 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# sourced by check_dockerfiles.sh if no \\$PWD/.hadolint.yaml is found\n\nignored:\n  # Maintainer is deprecated\n  - DL4000\n  # FROM latest - dev images build on upstream latest tag intentionally\n  - DL3007\n  # apt-get install versions need not be pinned\n  - DL3008\n  # apk add versions need not be pinned\n  - DL3018\n  #- SC1010\n  # - for dev images it's ok to use both curl and wget as they are dependencies of different scripts\n  - DL4001\n\ntrustedRegistries:\n  - docker.io\n  #- my-company.com:5000\n"
  },
  {
    "path": "images/README.md",
    "content": "# Images\n\nThese are resource images.\n\nFor code check the adjacent [../media](../media) directory.\n"
  },
  {
    "path": "install/download_avro_tools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-16 13:47:47 +0200 (Mon, 16 Sep 2024)\n#  (ported from Knowledge Base avro page)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Apache Avro Tools jar or an explicitly given version\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\n#version=\"1.12.0\"\nversion=\"${1:-latest}\"\n\ndownloads_url='https://repo1.maven.org/maven2/org/apache/avro/avro-tools'\n\n# ERE format for grep -E\nversion_regex='<a href=\"[[:digit:]]+.[[:digit:]]+.[[:digit:]]+/\" title=\"[[:digit:]]+.[[:digit:]]+.[[:digit:]]+/\">([[:digit:]]+.[[:digit:]]+.[[:digit:]]+)/</a>'\n\n# Should match these:\n#\n# <a href=\"1.10.0/\" title=\"1.10.0/\">1.10.0/</a>\n# <a href=\"1.10.1/\" title=\"1.10.1/\">1.10.1/</a>\n# <a href=\"1.10.2/\" title=\"1.10.2/\">1.10.2/</a>\n# <a href=\"1.11.0/\" title=\"1.11.0/\">1.11.0/</a>\n# <a href=\"1.11.1/\" title=\"1.11.1/\">1.11.1/</a>\n# <a href=\"1.11.2/\" title=\"1.11.2/\">1.11.2/</a>\n# <a href=\"1.11.3/\" title=\"1.11.3/\">1.11.3/</a>\n# <a href=\"1.12.0/\" title=\"1.12.0/\">1.12.0/</a>\n# <a href=\"1.5.0/\" title=\"1.5.0/\">1.5.0/</a>\n# <a href=\"1.5.1/\" title=\"1.5.1/\">1.5.1/</a>\n# <a href=\"1.5.2/\" title=\"1.5.2/\">1.5.2/</a>\n# <a href=\"1.5.3/\" title=\"1.5.3/\">1.5.3/</a>\n# <a href=\"1.5.4/\" title=\"1.5.4/\">1.5.4/</a>\n# <a href=\"1.6.0/\" title=\"1.6.0/\">1.6.0/</a>\n# <a href=\"1.6.1/\" title=\"1.6.1/\">1.6.1/</a>\n# <a href=\"1.6.2/\" title=\"1.6.2/\">1.6.2/</a>\n# <a href=\"1.6.3/\" title=\"1.6.3/\">1.6.3/</a>\n# <a href=\"1.7.0/\" title=\"1.7.0/\">1.7.0/</a>\n# <a href=\"1.7.1/\" title=\"1.7.1/\">1.7.1/</a>\n# <a href=\"1.7.2/\" title=\"1.7.2/\">1.7.2/</a>\n# <a href=\"1.7.3/\" title=\"1.7.3/\">1.7.3/</a>\n# <a href=\"1.7.4/\" title=\"1.7.4/\">1.7.4/</a>\n# <a href=\"1.7.5/\" title=\"1.7.5/\">1.7.5/</a>\n# <a href=\"1.7.6/\" title=\"1.7.6/\">1.7.6/</a>\n# <a href=\"1.7.7/\" title=\"1.7.7/\">1.7.7/</a>\n# <a href=\"1.8.0/\" title=\"1.8.0/\">1.8.0/</a>\n# <a href=\"1.8.1/\" title=\"1.8.1/\">1.8.1/</a>\n# <a href=\"1.8.2/\" title=\"1.8.2/\">1.8.2/</a>\n# <a href=\"1.9.0/\" title=\"1.9.0/\">1.9.0/</a>\n# <a href=\"1.9.1/\" title=\"1.9.1/\">1.9.1/</a>\n# <a href=\"1.9.2/\" title=\"1.9.2/\">1.9.2/</a>\n\n\nif [ \"$version\" = \"latest\" ]; then\n    timestamp \"Determining latest Avro Tools version from $downloads_url\"\n    versions=\"$(\n        curl -sS \"$downloads_url/\" |\n        grep -Eo \"$version_regex\" |\n        sed 's|</a>[[:space:]]*$||; s|^.*>||; s|/$||'\n    )\"\n    version=\"$(sort -Vr <<< \"$versions\" | head -n 1)\"\n    timestamp \"Determined latest Avro Tools version to be $version\"\nfi\n\ndownload_url=\"$downloads_url/$version/avro-tools-$version.jar\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n"
  },
  {
    "path": "install/download_azul_openjdk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 10:13:38 +0200 (Wed, 21 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads Azul OpenJDK version for Linux x86_64\n\nIf Java version is not specified, automatically determines latest version to download\n\nFor Linux distributions, see also this doc to install native packages:\n\n    https://docs.azul.com/core/install/debian\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<java_version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\njava_version=\"${1:-}\"\n\nif is_mac; then\n    grep(){\n        command ggrep \"$@\"\n    }\nfi\n\nurl_base=\"https://cdn.azul.com/zulu/bin\"\n\ntimestamp \"Fetching directory listing of versions from: $url_base/\"\ntimestamp \"(slow, takes 30 seconds due to large listing, please wait)\"\ndirectory_listing=\"$(curl -sS \"$url_base/\")\"\n\nif is_blank \"$java_version\" || [ \"$java_version\" = latest ]; then\n    latest=true\n    timestamp \"Java version not specified, attempting to find latest JDK version\"\n    # super brittle to pass a web page that will change\n    versions=\"$(grep -P -o 'jdk\\K\\d+' <<< \"$directory_listing\" || die \"Failed to parse JDK versions out of directory listing\")\"\n    java_version=\"$(sort -nr <<< \"$versions\" | head -n1 || :)\"\n    is_blank \"$java_version\" && die \"Failed to parse JDK version from list of versions\"\n    timestamp \"Determined latest JDK version to be $java_version\"\nfi\n\nparse_download_paths(){\n    grep -Eo -e \"/zulu/bin/zulu${java_version}[^>]+jdk[^>]+-linux_x64.tar.gz\" \\\n             -e \"/zulu/bin/zulu[^>]+jdk${java_version}[^>]*-linux_x64.tar.gz\" \\\n             <<< \"$directory_listing\" ||\n    die \"Failed to parse any download URLs for java version $java_version - upstream format may have changed?\"\n}\n\nstrip_beta_paths(){\n    local download_paths=\"$1\"\n    # links are relative\n    sed '/beta/d' <<< \"$download_paths\" | tail -n1 | sed 's|^/zulu/bin/||'\n}\n\ntimestamp \"Parsing download path from directory listing for JDK version '$java_version'\"\ndownload_paths=\"$(parse_download_paths)\"\ndownload_path=\"$(strip_beta_paths \"$download_paths\")\"\n\nif [ -z \"$download_path\" ]; then\n    if [ \"$latest\" = true ]; then\n        # iterate back 3 JDK versions to find a non-beta release\n        for ((i=1; i <= 3; i++)); do\n            timestamp \"No non-beta JDK versions found, attempting to try for previous JDK version (attempt $i of 3)\"\n            # take the previous Java version if the current one has only beta releases\n            java_version=\"$(grep -P -o 'jdk\\K\\d+' <<< \"$directory_listing\" |\n                            sort -unr |\n                            awk -v N=\"$java_version\" '$1 < N' |\n                            head -n1 || :)\"\n            if [ -z \"$java_version\" ]; then\n                die \"ERROR: failed to determine previous JDK version\"\n            fi\n            timestamp \"Inferred previous JDK version to be '$java_version'\"\n            timestamp \"Parsing download path from directory listing for previous JDK version '$java_version'\"\n            # links are relative\n            download_paths=\"$(parse_download_paths)\"\n            download_path=\"$(strip_beta_paths \"$download_paths\")\"\n            if [ -n \"$download_path\" ]; then\n                break\n            fi\n        done\n    fi\n    if [ -z \"$download_path\" ]; then\n        die \"Failed to parse any download URL for java version '$java_version'\"\n    fi\nfi\n\ndownload_url=\"$url_base/$download_path\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n"
  },
  {
    "path": "install/download_bytecode_viewer_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 20:57:31 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Bytecode-Viewer java command line decompiler jar or an explicitly given version\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nversion=\"${1:-latest}\"\n\ngithub_owner_repo=\"Konloch/bytecode-viewer\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/v{version}/Bytecode-Viewer-{version}.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_cfr_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 20:57:31 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest CFR java command line decompiler jar or an explicitly given version\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nversion=\"${1:-latest}\"\n\ngithub_owner_repo=\"leibnitz27/cfr\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/{version}/cfr-{version}.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_jd_gui_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 20:57:31 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest JD GUI jar or an explicitly given version\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nversion=\"${1:-latest}\"\n\ngithub_owner_repo=\"java-decompiler/jd-gui\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/v{version}/jd-gui-{version}-min.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_mssql_jdbc_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Mon Aug 26 16:57:51 2024 +0200\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Microsoft SQL Server JDBC jar or an explicitly given version\n\nUseful to get the jar to upload to data integration 3rd party directories or Docker images or Kubernetes\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\nJRE version defaults to 8\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version> <jre_version>]\"\n\n#version=\"${1:-42.2.18}\"\nversion=\"${1:-latest}\"\njre_version=\"${2:-8}\"\n\ngithub_owner_repo=\"microsoft/mssql-jdbc\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/v{version}/mssql-jdbc-{version}.jre$jre_version.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_mysql_jdbc_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-24 11:54:52 +0000 (Tue, 24 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest MySQL JDBC jar or an explicitly given version\n\nUseful to get the jar to upload to data integration 3rd party directories or Docker images or Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nlatest=\"9.0.0\"\n\n#version=\"${1:-8.0.22}\"\nversion=\"${1:-$latest}\"\n\n# TODO: figure out a way of determining what the latest MySQL JDBC connector version is\n\nif [ \"$version\" = \"$latest\" ]; then\n    download_url=\"https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-j-$version.tar.gz\"\nelse\n    download_url=\"https://downloads.mysql.com/archives/get/p/3/file/mysql-connector-j-$version.tar.gz\"\nfi\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\necho >&2\n\ntimestamp \"Untarring\"\ntarball=\"${download_url##*/}\"\ntar zxvf \"$tarball\" -C . --strip 1 \"mysql-connector-j-$version/mysql-connector-j-$version.jar\"\necho >&2\n\ntimestamp \"Extracted mysql-connector-j-$version.jar\"\n"
  },
  {
    "path": "install/download_openjdk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-21 10:13:38 +0200 (Wed, 21 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads OpenJDK version for Linux x86_64\n\nIf Java version is not specified, automatically determines latest version to download\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<java_version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\njava_version=\"${1:-}\"\n\nif is_mac; then\n    grep(){\n        command ggrep \"$@\"\n    }\nfi\n\nif is_blank \"$java_version\"; then\n    timestamp \"Java version not specified, attempting to find latest JDK version\"\n    timestamp \"Fetching downloads page\"\n    # super brittle to pass a web page that will change so just take the first JDK number which is likely to be the production release\n    download_page=\"$(curl -sS \"https://jdk.java.net/\")\"\n    timestamp \"Parsing downloads page\"\n    java_version=\"$(grep -P -o -m 1 'JDK \\K\\d+' <<< \"$download_page\" || die \"Failed to parse JDK version out of web page\")\"\n    timestamp \"Determined latest JDK version to be $java_version\"\nfi\n\njava_version_url=\"https://jdk.java.net/java-se-ri/$java_version\"\n\nif [ \"$java_version\" = 8 ]; then\n    java_version_url+=\"-MR6\"\nelif [ \"$java_version\" = 11 ]; then\n    java_version_url+=\"-MR3\"\nelif [ \"$java_version\" = 17 ]; then\n    java_version_url+=\"-MR1\"\nfi\n\ntimestamp \"Parsing download URL from Java version URL: $java_version_url\"\ndownload_page=\"$(curl -sS \"$java_version_url\")\"\ndownload_url=\"$(grep -Eom1 \"https://download.java.net/openjdk/(open)?jdk$java_version.+linux-x64.*.tar.gz\" <<< \"$download_page\" ||\n                die \"Failed to parse download URL\")\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n"
  },
  {
    "path": "install/download_parquet_tools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-16 13:47:47 +0200 (Mon, 16 Sep 2024)\n#  (ported from Knowledge Base parquet page)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Apache Parquet Tools jar or an explicitly given version\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\n#version=\"1.11.2\"\nversion=\"${1:-latest}\"\n\ndownloads_url='https://repo1.maven.org/maven2/org/apache/parquet/parquet-tools'\n\n# ERE format for grep -E\nversion_regex='<a href=\"[[:digit:]]+.[[:digit:]]+.[[:digit:]]+/\" title=\"[[:digit:]]+.[[:digit:]]+.[[:digit:]]+/\">([[:digit:]]+.[[:digit:]]+.[[:digit:]]+)/</a>'\n\n# Should match these:\n#\n# <a href=\"1.10.0/\" title=\"1.10.0/\">1.10.0/</a>\n# <a href=\"1.10.1/\" title=\"1.10.1/\">1.10.1/</a>\n# <a href=\"1.11.0/\" title=\"1.11.0/\">1.11.0/</a>\n# <a href=\"1.11.1/\" title=\"1.11.1/\">1.11.1/</a>\n# <a href=\"1.11.2/\" title=\"1.11.2/\">1.11.2/</a>\n# <a href=\"1.7.0/\" title=\"1.7.0/\">1.7.0/</a>\n# <a href=\"1.8.0/\" title=\"1.8.0/\">1.8.0/</a>\n# <a href=\"1.8.1/\" title=\"1.8.1/\">1.8.1/</a>\n# <a href=\"1.8.2/\" title=\"1.8.2/\">1.8.2/</a>\n# <a href=\"1.8.3/\" title=\"1.8.3/\">1.8.3/</a>\n# <a href=\"1.9.0/\" title=\"1.9.0/\">1.9.0/</a>\n\nif [ \"$version\" = \"latest\" ]; then\n    timestamp \"Determining latest Parquet Tools version from $downloads_url\"\n    versions=\"$(\n        curl -sS \"$downloads_url/\" |\n        grep -Eo \"$version_regex\" |\n        sed 's|</a>[[:space:]]*$||; s|^.*>||; s|/$||'\n    )\"\n    version=\"$(sort -Vr <<< \"$versions\" | head -n 1)\"\n    timestamp \"Determined latest Parquet Tools version to be $version\"\nfi\n\ndownload_url=\"$downloads_url/$version/parquet-tools-$version.jar\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n"
  },
  {
    "path": "install/download_postgres_jdbc_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-27 14:21:49 +0000 (Fri, 27 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest PostgreSQL JDBC jar or an explicitly given version\n\nUseful to get the jar to upload to data integration 3rd party directories or Docker images or Kubernetes\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\n#version=\"${1:-42.2.18}\"\nversion=\"${1:-latest}\"\n\ngithub_owner_repo=\"pgjdbc/pgjdbc\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/REL{version}/postgresql-{version}.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_procyon_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 23:01:46 +0200 (Wed, 28 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Procyon java command line decompiler jar or an explicitly given version\n\nVersion defaults to 'latest' in which case it determines the latest version from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nversion=\"${1:-latest}\"\n\ngithub_owner_repo=\"mstrobel/procyon\"\n\n\"$srcdir/../github/github_download_release_jar.sh\" \"https://github.com/$github_owner_repo/releases/download/v{version}/procyon-decompiler-{version}.jar\" \"$version\"\n"
  },
  {
    "path": "install/download_vertica_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-27 14:12:50 +0200 (Tue, 27 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuickly determines and downloads latest Vertica JDBC jar or an explicitly given version\n\nUseful to get the jar to upload to data integration 3rd party directories or Docker images or Kubernetes\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\n#version=\"24.2.0\"\nversion=\"${1:-latest}\"\n\ndownloads_url=\"https://www.vertica.com/download/vertica/client-drivers/\"\n\n# ERE format for grep -E\njar_url_regex=\"https://www.vertica.com/client_drivers/[[:digit:].x-]+/[[:digit:].-]+/vertica-jdbc-[[:digit:].-]+.jar\"\n\n# Should match these:\n#\n# https://www.vertica.com/client_drivers/24.2.x/24.2.0-1/vertica-jdbc-24.2.0-1.jar\n# https://www.vertica.com/client_drivers/24.1.x/24.1.0-0/vertica-jdbc-24.1.0-0.jar\n# https://www.vertica.com/client_drivers/23.4.x/23.4.0-0/vertica-jdbc-23.4.0-0.jar\n# https://www.vertica.com/client_drivers/23.3.x/23.3.0-0/vertica-jdbc-23.3.0-0.jar\n# https://www.vertica.com/client_drivers/12.0.x/12.0.4-0/vertica-jdbc-12.0.4-0.jar\n# https://www.vertica.com/client_drivers/11.1.x/11.1.1-0/vertica-jdbc-11.1.1-0.jar\n# https://www.vertica.com/client_drivers/11.0.x/11.0.2-0/vertica-jdbc-11.0.2-0.jar\n# https://www.vertica.com/client_drivers/10.1.x/10.1.1-0/vertica-jdbc-10.1.1-0.jar\n# https://www.vertica.com/client_drivers/10.0.x/10.0.1-0/vertica-jdbc-10.0.1-0.jar\n# https://www.vertica.com/client_drivers/9.3.x/9.3.1-0/vertica-jdbc-9.3.1-0.jar\n# https://www.vertica.com/client_drivers/9.2.x/9.2.1-0/vertica-jdbc-9.2.1-0.jar\n# https://www.vertica.com/client_drivers/9.2.x/9.2.1-0/vertica-jdbc-9.2.1-0.jar\n# https://www.vertica.com/client_drivers/9.1.x/9.1.1-0/vertica-jdbc-9.1.1-0.jar\n# https://www.vertica.com/client_drivers/9.0.x/9.0.1-0/vertica-jdbc-9.0.1-0.jar\n# https://www.vertica.com/client_drivers/8.1.x/8.1.1-0/vertica-jdbc-8.1.1-0.jar\n# https://www.vertica.com/client_drivers/8.0.x/8.0.1/vertica-jdbc-8.0.1-0.jar\n# https://www.vertica.com/client_drivers/7.2.x/7.2.3-0/vertica-jdbc-7.2.3-0.jar\n\ntimestamp \"Fetching list of Vertica JDBC JAR download URLs from $downloads_url\"\njar_urls=\"$(\n    curl -sS \"$downloads_url\" |\n    grep -Eo \"$jar_url_regex\"\n)\"\n\nif [ \"$version\" = \"latest\" ]; then\n    timestamp \"Determining latest version from list of JAR download urls\"\n    download_url=\"$(head -n1 <<< \"$jar_urls\")\"\n    timestamp \"Determined latest JAR version to be ${download_url##*/}\"\nelse\n    timestamp \"Checking if requested version '$version' is available\"\n    download_url=\"$(grep \"$version\" <<< \"$jar_urls\" | head -n 1 || :)\"\n    if [ -z \"$download_url\" ]; then\n        echo\n        echo \"ERROR: Vertica JDBC JAR version '$version' not found\" >&2\n        echo\n        echo \"Here are the list of available versions:\"\n        echo\n        sed 's/.*vertica-jdbc-//; s/-[[:digit:]]\\+.jar$//' <<< \"$jar_urls\"\n        echo\n        exit 1\n    fi\nfi\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n"
  },
  {
    "path": "install/getawless.sh",
    "content": "#!/bin/bash\nset -e\n\n# Download latest awless binary from Github\n\nARCH_UNAME=\"$(uname -m)\"\nif [[ \"$ARCH_UNAME\" == \"x86_64\" ]]; then\n\tARCH=\"amd64\"\nelse\n\tARCH=\"386\"\nfi\n\nEXT=\"tar.gz\"\n\nif [[ \"$OSTYPE\" == \"linux-gnu\" ]]; then\n\tOS=\"linux\"\nelif [[ \"$OSTYPE\" == \"darwin\"* ]]; then\n\tOS=\"darwin\"\nelif [[ \"$OSTYPE\" == \"win32\" ]] || [[ \"$OSTYPE\" == \"cygwin\" ]] || [[ \"$OSTYPE\" == \"msys\" ]] ; then\n\tOS=\"windows\"\n\tEXT=\"zip\"\nelse\n\techo \"No awless binary available for OS '$OSTYPE'. You may want to use go to install awless with 'go get -u github.com/wallix/awless'\"\n  exit\nfi\n\n# broken SSL cert as documented here - https://github.com/wallix/awless/issues/278\n#LATEST_VERSION=`curl -fs https://updates.awless.io | grep -oE \"v[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}\"`\nLATEST_VERSION=\"$(curl -s https://api.github.com/repos/wallix/awless/releases/latest | awk '/\"tag_name\":/ {print $2}' | sed 's/[\",]//g')\"\n\nFILENAME=awless-$OS-$ARCH.$EXT\n\nDOWNLOAD_URL=\"https://github.com/wallix/awless/releases/download/$LATEST_VERSION/$FILENAME\"\n\necho \"Downloading awless from $DOWNLOAD_URL\"\n\nif ! curl --fail -o \"$FILENAME\" -L \"$DOWNLOAD_URL\"; then\n    exit\nfi\n\necho \"\"\necho \"extracting $FILENAME to ./awless\"\n\nif [[ \"$OS\" == \"windows\" ]]; then\n\techo 'y' | unzip $FILENAME > /dev/null\nelse\n\ttar -xzf $FILENAME\nfi\n\necho \"removing $FILENAME\"\nrm -- \"$FILENAME\"\nchmod +x ./awless\n\necho \"\"\necho \"awless successfully installed to ./awless\"\necho \"\"\necho \"don't forget to add it to your path, with, for example, 'sudo mv -- awless /usr/local/bin/' \"\necho \"\"\necho \"then, for autocompletion, run:\"\necho \"    [bash] echo 'source <(awless completion bash)' >> ~/.bashrc\"\necho \"    OR\"\necho \"    [zsh]  echo 'source <(awless completion zsh)' >> ~/.zshrc\"\n"
  },
  {
    "path": "install/install_android_commandlinetools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-27 02:18:03 +0700 (Thu, 27 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and installs the latest Android Command Line Tools to ~/Android/Sdk/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncd /tmp\n\ntimestamp \"Determining OS\"\nos=\"$(uname | tr '[:upper:]' '[:lower:]')\"\nif [ \"$os\" = darwin ]; then\n    os=mac\nfi\ntimestamp \"OS is: $os\"\n\nzip=\"commandlinetools-$os-11076708_latest.zip\"\n\ntimestamp \"Downloading Android platform tools for OS '$os'\"\nwget -Nc \"https://dl.google.com/android/repository/$zip\"\n\nmkdir -p -v ~/Android/Sdk\n\ntimestamp \"Unzipping Android SDK platform tools to $HOME/Android/Sdk (overwrite)\"\necho\nunzip -o \"$zip\" -d ~/Android/Sdk\necho\n\n#rm -fr ~/Android/Sdk/cmdline-tools/latest\n\nmkdir -p -v ~/Android/Sdk/cmdline-tools/latest\necho\n\n#mv -fv ~/Android/Sdk/cmdline-tools/* ~/Android/Sdk/cmdline-tools/latest/ || :\n\n# unpacks to ~/Android/Sdk/cmdline-tools/{bin,lib} but the sdkmanager insists on finding ~/Android/Sdk/cmdline-tools/latest/bin\ntimestamp \"Moving ~/Android/Sdk/cmdline-tools/ to ~/Android/Sdk/cmdline-tools/latest/\"\nrsync -a --remove-source-files ~/Android/Sdk/cmdline-tools/ ~/Android/Sdk/cmdline-tools/latest/ --exclude latest\necho\n\ntimestamp \"Removing empty directories under ~/Android/Sdk/cmdline-tools/\"\nfind /Users/hari/Android/Sdk/cmdline-tools/ -type d -empty -delete\n\ncat <<EOF\n\nNow set these environment variables in your shell:\n\n    export ANDROID_HOME=\"\\$HOME/Android/Sdk\"\n    export PATH=\"\\$PATH:\\$ANDROID_HOME/cmdline-tools/latest/bin\"\nEOF\n"
  },
  {
    "path": "install/install_android_sdk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-27 02:04:11 +0700 (Thu, 27 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and installs the latest Android SDK to ~/Android/Sdk/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncd /tmp\n\ntimestamp \"Determining OS\"\nos=\"$(uname | tr '[:upper:]' '[:lower:]')\"\ntimestamp \"OS is: $os\"\n\nzip=\"platform-tools-latest-$os.zip\"\n\ntimestamp \"Downloading Android platform tools for OS '$os'\"\nwget -Nc \"https://dl.google.com/android/repository/$zip\"\n\nmkdir -p -v ~/Android/Sdk\n\ntimestamp \"Unzipping Android SDK platform tools to $HOME/Android/Sdk (overwrite)\"\necho\nunzip -o \"$zip\" -d ~/Android/Sdk\n\ncat <<EOF\n\nNow set these environment variables in your shell:\n\n    export ANDROID_HOME=\"\\$HOME/Android/Sdk\"\n    export PATH=\"\\$PATH:\\$ANDROID_HOME/platform-tools\"\nEOF\n"
  },
  {
    "path": "install/install_ansible.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n# command -v catches aliases, not suitable\n#\n#  Author: Hari Sekhon\n#  Date: 2019/09/20\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Ansible on Mac / Linux\n#\n# https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html\n#\n# Prompts for sudo if using OS system packages\n#\n# If falling back to Python PIP then if running as root installs to System, otherwise installs to local --user library\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nos=\"$(uname -s)\"\necho \"OS detected as $os\"\necho\n\nsudo=sudo\npip_opts=\"--user\"\nif [ $EUID -eq 0 ]; then\n    sudo=\"\"\n    pip_opts=\"\"\nfi\n\nif [ -z \"${UPDATE_ANSIBLE:-}\" ]; then\n    if type -P ansible &>/dev/null; then\n        echo \"Ansible already installed\"\n        echo\n        echo \"To update ansible, set the below and then re-run this script\"\n        echo\n        echo \"export UPDATE_ANSIBLE=1\"\n        exit 0\n    fi\nfi\n\necho \"Installing Ansible\"\necho\nif [ \"$os\" = \"Darwin\" ]; then\n    brew update\n    brew install ansible\nelif [ \"$os\" = \"Linux\" ]; then\n    if type -P dnf &>/dev/null; then\n        echo \"Installing via DNF\"\n        $sudo dnf install -y ansible\n    elif type -P yum &>/dev/null; then\n        echo \"Installing via Yum\"\n        $sudo yum install -y ansible\n    elif type -P apt &>/dev/null; then\n        echo \"Installing via APT\"\n        if grep -q Ubuntu /etc/*release; then\n            opts=\"-o DPkg::Lock::Timeout=1200\"\n            $sudo apt update $opts\n            $sudo apt install -y $opts software-properties-common\n            $sudo apt-add-repository -y --update ppa:ansible/ansible\n            $sudo apt update $opts\n            $sudo apt install -y $opts ansible\n        else\n            # assume Debian\n            line='deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main'\n            if ! grep -Fq \"$line\" /etc/apt/sources.list; then\n                echo \"$line\" >> /etc/apt/sources.list\n            fi\n            $sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367\n            $sudo apt update $opts\n            $sudo apt install -y $opts ansible\n        fi\n    elif type -P apk &>/dev/null; then\n        echo \"Installing via Apk\"\n        $sudo apk update\n        $sudo apk add ansible\n    elif type -P emerge &>/dev/null; then\n        echo \"Installing via Emerge\"\n        $sudo emerge -av app-admin/ansible\n    elif type -P pip &>/dev/null; then\n        echo \"Installing via Pip\"\n        pip install $pip_opts ansible\n    else\n        echo \"Couldn't find Linux package manager!'\"\n        exit 1\n    fi\nelif type -P pip &>/dev/null; then\n    echo \"Unsupported OS, installing via Pip\"\n    pip install $pip_opts ansible\nelse\n    echo \"Unsupported OS and pip not available!\"\n    exit 2\nfi\n"
  },
  {
    "path": "install/install_appveyor_byoc.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 22:38:36 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs PowerShell on Mac and various Linux distros\n#\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux\n\n# Mac - could do this, but following standard powershell install and calling from there works too\n#\n# https://www.appveyor.com/docs/byoc/mac/\n#\n# HOMEBREW_HOST_AUTH_TKN=<host-authorization-token> HOMEBREW_APPVEYOR_URL=https://ci.appveyor.com brew install appveyor/brew/appveyor-host-agent\n#\n# shows up in brew services either way:\n#\n# brew services list\n\n# Linux:\n#\n# https://www.appveyor.com/docs/byoc/linux/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif ! type -P pwsh &>/dev/null; then\n    \"$srcdir/install_powershell.sh\"\nfi\n\n# AppVeyor host dependencies\n# sysvinit-tools on RHEL, but appveyor byoc looks for dpkg so is probably only compatible with debian based distributions\nif grep -Eiq 'debian|ubuntu' /etc/*release; then\n    opts=\"-o DPkg::Lock::Timeout=1200\"\n    apt-get update $opts\n    apt-get install -y $opts libcap2-bin libterm-ui-perl sudo sysvinit-utils\nfi\n\n# leading whitespace break PowerShell commands\npwsh << EOF\nInstall-Module AppVeyorBYOC -Scope CurrentUser -Force; Import-Module AppVeyorBYOC\nEOF\n"
  },
  {
    "path": "install/install_argocd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-09-17 16:27:28 +0100 (Fri, 17 Sep 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls ArgoCD CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\n#export RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" argoproj/argo-cd \"argocd-{os}-{arch}\" \"$version\"\n\necho\nargocd version --client\n"
  },
  {
    "path": "install/install_awless.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-19 11:26:11\n#  (moved from Makefile)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs AWLess\n\n# You might need to first:\n#\n# yum install -y epel-release\n# yum install -y gcc git make python-pip which\n#\n# this is automatically done first when called via 'make aws' at top level of this repo\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/ci.sh\"\n\nsection \"Installing AWLess\"\n\nmkdir -p ~/bin\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PATH=\"$PATH:/usr/local/bin\"\nexport PATH=\"$PATH:$HOME/bin\"\n\nif type -P awless &>/dev/null; then\n    echo \"Awless already installed\"\nelse\n    echo\n    echo \"==========================\"\n    echo \"Awless is unmaintained now, installing only optimistically and ignoring failures...\"\n    echo \"==========================\"\n    echo\n    set +e\n    echo \"Installing AWLess\"\n    if is_mac; then\n        # this brew install fails on Linux even when brew is installed and works for SAM CLI\n        brew tap wallix/awless\n        echo\n        brew install awless\n    else\n        if ! curl -sS https://updates.awless.io >/dev/null; then\n            #echo\n            #echo \"AWLess SSL certificate still expired, must install manually until fixed\"\n            \"$srcdir/getawless.sh\"\n        else\n            curl -sS https://raw.githubusercontent.com/wallix/awless/master/getawless.sh | bash\n        fi\n        mv -iv -- awless ~/bin/\n    fi\nfi\n\n# Awless is usually installed to /usr/local/bin/awless\n\ncat <<EOF\n\nDone\n\nInstalled to:\n\n$(type -P awless)\n\nEOF\n"
  },
  {
    "path": "install/install_aws_cli.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-19 11:26:11\n#  (moved from Makefile)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs AWS CLI, eksctl and ECS CLI\n\n# You might need to first:\n#\n# yum install -y epel-release\n# yum install -y gcc git make python-pip which\n#\n# this is automatically done first when called via 'make aws' at top level of this repo\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/ci.sh\"\n\nsection \"Installing AWS CLI\"\n\nmkdir -p ~/bin\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PATH=\"$PATH:/usr/local/bin\"\nexport PATH=\"$PATH:$HOME/bin\"\n\n#if type -P aws &>/dev/null; then\n#    echo \"AWS CLI already installed\"\nif type -P apk &>/dev/null; then\n    if ! grep -q 'https://dl-cdn.alpinelinux.org/alpine/edge/testing' /etc/apk/repositories; then\n        echo -e -n \"\\n@testing https://dl-cdn.alpinelinux.org/alpine/edge/testing\" >> /etc/apk/repositories\n    fi\n    apk add aws-cli-v2@testing --no-cache\nelse\n    echo \"Installing AWS CLI\"\n    # old AWS CLI v1 - doesn't support AWS SSO\n    #PYTHON_USER_INSTALL=1 \"$srcdir/../python/python_pip_install.sh\" awscli\n    pushd /tmp\n    #curl \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\" -o \"awscli-bundle.zip\"\n    #wget -c \"https://s3.amazonaws.com/aws-cli/awscli-bundle.zip\"\n    #unzip -o awscli-bundle.zip\n    # needs to find Python 3 first in the path to work\n    #PATH=\"/usr/local/opt/python/libexec/bin:$PATH\" ./awscli-bundle/install -b ~/bin/aws\n    if is_mac; then\n        wget -c \"https://awscli.amazonaws.com/AWSCLIV2.pkg\" -O \"AWSCLIV2.pkg\"\n        # defined in utils.sh lib\n        # shellcheck disable=SC2154\n        $sudo installer -pkg AWSCLIV2.pkg -target /\n        rm -fr -- AWSCLIV2.pkg\n    else\n        wget -c \"https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip\" -O \"awscliv2.zip\"\n        unzip -o awscliv2.zip\n        # defined in utils.sh lib\n        # shellcheck disable=SC2154\n        $sudo ./aws/install --update\n        rm -fr -- aws awscliv2.zip\n    fi\n    popd\n    echo\n    echo -n \"AWS CLI version: \"\n    aws --version\n    echo\nfi\n\n\"$srcdir/install_eksctl.sh\"\n\nif type -P ecs-cli &>/dev/null; then\n    echo \"ECS CLI already installed\"\nelse\n    echo \"Installing AWS ECS CLI\"\n    if is_mac; then\n        wget -O ~/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-darwin-amd64-latest\n    else\n        wget -O ~/bin/ecs-cli https://amazon-ecs-cli.s3.amazonaws.com/ecs-cli-linux-amd64-latest\n    fi\n    chmod +x ~/bin/ecs-cli\nfi\n\n# AWS CLI usually installs to ~/.local/bin/aws on Linux or ~/Library/Python/2.7/bin on Mac\n\ncat <<EOF\n\nDone\n\nInstalled locations:\n\n$(type -P aws)\n$(type -P ecs-cli)\n$(type -P eksctl)\n\nEOF\n"
  },
  {
    "path": "install/install_aws_ebcli.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-17 16:02:53 +0100 (Tue, 17 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Elastic Beanstalk CLI\n#\n# https://github.com/aws/aws-elastic-beanstalk-cli-setup\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/ci.sh\"\n\nsection \"Install Elastic Beanstalk CLI\"\n\nmkdir -pv ~/github\n\ncd ~/github\n\nif [ -d aws-elastic-beanstalk-cli-setup ]; then\n    pushd  aws-elastic-beanstalk-cli-setup\n    git pull\n    popd\nelse\n    git clone https://github.com/aws/aws-elastic-beanstalk-cli-setup.git\nfi\n\npython3=\"$(type -P python3)\"\n\nif is_linux; then\n    if type -P apt-get; then\n        opts=\"-o DPkg::Lock::Timeout=1200\"\n        apt-get install -y $opts \\\n            build-essential \\\n            zlib1g-dev \\\n            libssl-dev \\\n            libncurses-dev \\\n            libffi-dev \\\n            libsqlite3-dev \\\n            libreadline-dev \\\n            libbz2-dev\n    elif type -P yum; then\n        yum group install -y \"Development Tools\"\n        yum install -y \\\n            zlib-devel \\\n            openssl-devel \\\n            ncurses-devel \\\n            libffi-devel \\\n            sqlite-devel.x86_64 \\\n            readline-devel.x86_64 \\\n            bzip2-devel.x86_64\n    fi\nelif is_mac; then\n    brew install zlib openssl readline\n    CFLAGS=\"-I$(brew --prefix openssl)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include\"\n    LDFLAGS=\"-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib\"\n    export CFLAGS\n    export LDFLAGS\nfi\n\n\"$python3\" ./aws-elastic-beanstalk-cli-setup/scripts/ebcli_installer.py -p \"$python3\"\n\n# -p /usr/local/bin/python3 avoids this error:\n#\n#    Traceback (most recent call last):\n#      File \"/Library/Python/2.7/site-packages/virtualenv.py\", line 37, in <module>\n#        import ConfigParser\n#    ModuleNotFoundError: No module named 'ConfigParser'\n#\n#    During handling of the above exception, another exception occurred:\n#\n#    Traceback (most recent call last):\n#      File \"/Library/Python/2.7/site-packages/virtualenv.py\", line 39, in <module>\n#        import configparser as ConfigParser\n#      File \"/Library/Python/2.7/site-packages/configparser/__init__.py\", line 11, in <module>\n#        raise ImportError('This package should not be accessible on Python 3. '\n#    ImportError: This package should not be accessible on Python 3. Either you are trying to run from the python-future src folder or your installation of python-future is corrupted.\n"
  },
  {
    "path": "install/install_aws_sam_cli.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-19 11:26:11\n#  (moved from Makefile)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs AWS SAM CLI\n\n# You might need to first:\n#\n# yum install -y epel-release\n# yum install -y gcc git make python-pip which\n#\n# this is automatically done first when called via 'make aws' at top level of this repo\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/ci.sh\"\n\nsection \"Installing AWS SAM CLI\"\n\nmkdir -p ~/bin\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PATH=\"$PATH:/usr/local/bin\"\nexport PATH=\"$PATH:$HOME/bin\"\n\nif grep -q Alpine /etc/os-release 2>/dev/null; then\n    echo \"Skipping SAM CLI install on Alpine as Homebrew installer is broken on Alpine, see https://github.com/Homebrew/homebrew-core/issues/49813\"\n    exit 0\nfi\n\nif grep -q 'CentOS release [1-6][\\.[:space:]]' /etc/system-release 2>/dev/null; then\n    echo \"Skipping SAM CLI install on RHEL/CentOS < 7 as Homebrew installer no longer supports it, see https://github.com/Homebrew/brew/issues/7583#issuecomment-640379742\"\n    exit 0\nfi\n\nif is_CI && ! type -P brew && ! is_curl_min_version 7.41; then\n    echo \"Skipping SAM CLI install due to curl version < 7.41 - HomeBrew won't install, which AWS SAM CLI depends on, so skipping to avoid breaking older CI builds...\"\n    exit 0\nfi\n\n# when installing homebrew this doesn't detect the missing directory so doesn't add it to path, and after homebrew this needs to be called again\nload_homebrew_path(){\n    local directory\n    # root installs to first one, user installs to the latter\n    for directory in /home/linuxbrew/.linuxbrew/bin ~/.linuxbrew/bin; do\n        if [ -d \"$directory\" ]; then\n            export PATH=\"$PATH:$directory\"\n        fi\n    done\n}\n\nload_homebrew_path\n\nif type -P sam &>/dev/null; then\n    echo \"AWS SAM CLI already installed\"\nelse\n    # installs on Linux too as it is the AWS recommended method to install SAM CLI\n    \"$srcdir/install_homebrew.sh\"\n    load_homebrew_path\n    echo\n\n    echo \"Installing AWS SAM CLI\"\n    # AWS installs SAM CLI the same way on Linux + Mac\n    brew tap aws/tap\n    echo\n    # ignore this failure:\n    # ==> Downloading https://api.github.com/repos/aws/aws-sam-cli/tarball/v1.3.1\n    # ==> Downloading from https://codeload.github.com/aws/aws-sam-cli/legacy.tar.gz/v\n    # curl: (22) The requested URL returned error: 404 Not Found\n    set +e\n    brew install aws-sam-cli\nfi\n\n# AWS SAM CLI command installs to the standard HomeBrew directory\n#\n# On Mac that will be /usr/local/bin/sam\n# On Linux it will be /home/linuxbrew/.linuxbrew/bin for root or ~/.linuxbrew/bin for users\n\ncat <<EOF\n\nDone\n\nInstalled to:\n\n$(type -P sam)\n\nEOF\n"
  },
  {
    "path": "install/install_azure_cli.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n# shellcheck disable=SC2230\n# command -v catches aliases, not suitable\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 17:38:12 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Azure CLI\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\necho \"Installing Azure CLI\"\necho\n\nsudo=\"\"\nif [ $EUID != 0 ]; then\n    sudo=sudo\n    #if ! type -P $sudo &>/dev/null; then\n    #    echo \"not root and $sudo command not available, skipping Azure CLI install\"\n    #    exit 0\n    #fi\nfi\n\nuname_s=\"$(uname -s)\"\n#mkdir -p ~/bin\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n#[ -n \"${HOME:-}\" ] || HOME=~\n#export PATH=\"$PATH:/usr/local/bin\"\n#export PATH=\"$PATH:$HOME/bin\"\n\ninstall_azure_cli(){\n    if type -P apt-get &>/dev/null; then\n        curl -sL https://aka.ms/InstallAzureCLIDeb | $sudo bash\n    elif type -P yum &>/dev/null; then\n        # Needs Python 3\n#        if ! type -P python3 &>/dev/null; then\n#            echo \"Python 3 dependency not found, skipping\"\n#            exit 0\n#        fi\n        if ! yum list python3 &>/dev/null; then\n            echo \"Python 3 not available in package repos, cannot install Azure CLI, skipping...\"\n            exit 0\n        fi\n        $sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc\n        $sudo cat > /etc/yum.repos.d/azure-cli.repo <<EOF\n[azure-cli]\nname=Azure CLI\nbaseurl=https://packages.microsoft.com/yumrepos/azure-cli\nenabled=1\ngpgcheck=1\ngpgkey=https://packages.microsoft.com/keys/microsoft.asc\nEOF\n        \"$srcdir/../packages/yum_install_packages.sh\" azure-cli\n    elif [ \"$uname_s\" = Darwin ]; then\n        # outputs progress dots to prevent CI builds from terminating after 10 mins without output\n        \"$srcdir/../packages/brew_install_packages.sh\" azure-cli\n    elif [ \"$uname_s\" = Linux ]; then\n        if ! [ -t 1 ]; then\n            echo \"Non-interactive terminal, cannot install Azure CLI using Microsoft's install script as it assumes to pass /dev/tty to second-level python script, skipping...\"\n            exit 0\n        fi\n        if type -P apk &>/dev/null; then\n            # only works on Alpine 3 - Alpine 2.x doesn't support --no-cache and nor does it have Python 3 package dependency which Azure CLI requires\n            \"$srcdir/../packages/apk_install_packages.sh\" curl python3 python3-dev alpine-sdk musl-dev libffi-dev # openssl-dev conflicts with libressl-dev\n        fi\n        yes \"\" | curl -L https://aka.ms/InstallAzureCli | $sudo bash\n    echo\n        echo \"OS '$uname_s' is not Mac / Linux - not supported\"\n        exit 1\n    fi\n}\n\nif type -P az &>/dev/null; then\n    echo \"Azure CLI already installed\"\nelse\n    install_azure_cli\n    echo\nfi\n"
  },
  {
    "path": "install/install_azure_devops_cli.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n# shellcheck disable=SC2230\n# command -v catches aliases, not suitable\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 17:38:12 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Azure CLI\n#\n# https://learn.microsoft.com/en-us/azure/devops/cli/?view=azure-devops\n\n# XXX: Note - as of May 2024 Azure DevOps CLI only supports the cloud not the on premise server\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\necho \"Installing Azure DevOps CLI\"\necho\n\n\"$srcdir/install_azure_cli.sh\"\necho\n\necho \"Installing Azure DevOps CLI Extension\"\naz extension add --name azure-devops\necho\n\necho \"Checking Azure DevOps CLI Extension is installed\"\naz extension show --name azure-devops\necho\n\necho \"Azure DevOps CLI extension installation complete\"\necho\n\necho \"Next step configure your default organization and project to avoid having to specify it in each command\"\necho\necho \"Example:\"\necho\necho \"  az devops configure --defaults organization=https://dev.azure.com/harisekhon project=GitHub\"\necho\necho \"Show configuration:\"\necho\necho \"  az devops configure -l\"\n"
  },
  {
    "path": "install/install_bazel.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-16 18:53:36 +0100 (Fri, 16 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nBAZEL_VERSION=\"${1:-3.2.0}\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nif type -P bazel &>/dev/null; then\n    if bazel version | grep -q \"^Build label: $BAZEL_VERSION$\"; then\n        echo \"Bazel is already installed and the right version: $BAZEL_VERSION\"\n        exit 0\n    fi\nfi\n\nplatform=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n\ncd /tmp\n\ncurl -fLO \"https://github.com/bazelbuild/bazel/releases/download/${BAZEL_VERSION}/bazel-${BAZEL_VERSION}-installer-$platform-x86_64.sh\"\n\nchmod +x \"bazel-${BAZEL_VERSION}-installer-$platform-x86_64.sh\"\n./\"bazel-${BAZEL_VERSION}-installer-$platform-x86_64.sh\" --user  # --user installs to ~/bin and sets the .bazelrc path to ~/.bazelrc\n\nbazel version\n"
  },
  {
    "path": "install/install_bazelisk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-07-16 18:53:36 +0100 (Fri, 16 Jul 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Bazelisk CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.10.1}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" bazelbuild/bazelisk 'bazelisk-{os}-{arch}' \"$version\"\n"
  },
  {
    "path": "install/install_buildkite.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-11 17:49:09 +0000 (Wed, 11 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs BuildKite using Homebrew on Mac or direct download to ~/bin otherwise\n\n# https://buildkite.com/organizations/hari-sekhon/agents#setup-linux\n#\n# https://buildkite.com/organizations/hari-sekhon/agents#setup-macos\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\nsection \"Installing BuildKite Agent\"\n\n# path on Mac\nexport PATH=\"$PATH:/usr/local/bin\"\n\nif is_linux; then\n    # unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n    [ -n \"${HOME:-}\" ] || HOME=~\n    export PATH=\"$PATH:$HOME/.buildkite-agent/bin\"\nfi\n\nif [ -z \"${BUILDKITE_AGENT_TOKEN:-}\" ]; then\n    echo \"BUILDKITE_AGENT_TOKEN environment variable not defined\"\n    exit 1\nfi\n\nif type -P buildkite-agent &>/dev/null; then\n    echo \"** buildkite-agent is already installed\"\nelif is_mac; then\n    brew tap buildkite/buildkite\n    brew install buildkite-agent\nelse\n    TOKEN=\"$BUILDKITE_AGENT_TOKEN\" bash -c \"$(curl -sL https://raw.githubusercontent.com/buildkite/agent/master/install.sh)\"\nfi\n\nif is_mac; then\n    config=/usr/local/etc/buildkite-agent/buildkite-agent.cfg\n    if [ -f \"$config\" ]; then\n        if grep -q \"$BUILDKITE_AGENT_TOKEN\" \"$config\"; then\n            echo \"** \\$BUILDKITE_AGENT_TOKEN already found in config\"\n        else\n            echo\n            echo \"** injecting buildkite token into config: $config\"\n            echo\n            sed -i.bak \"s/^token=.*/token=$BUILDKITE_AGENT_TOKEN/\" \"$config\"\n        fi\n    fi\n    # run as a background agent\n    # brew services start buildkite/buildkite/buildkite-agent\n    # run as foreground agent\n    buildkite-agent start\nelse\n    ~/.buildkite-agent/bin/buildkite-agent start\nfi\n"
  },
  {
    "path": "install/install_cert_manager_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 18:51:40 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Cert Manager CLI 'cmctl'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.6.1}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/../github/github_install_binary.sh\" cert-manager/cert-manager 'cmctl-{os}-{arch}.tar.gz' \"$version\" cmctl\n\necho\ncmctl version --client\n"
  },
  {
    "path": "install/install_circleci.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-10 14:34:41 +0000 (Tue, 10 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Circle CI using Homebrew on Mac or direct download to ~/bin otherwise\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\nsection \"Installing Circle CI\"\n\nif type -P circleci &>/dev/null; then\n    echo \"circleci already installed\"\n    echo\n    exit 0\nfi\n\nif is_mac; then\n    \"$srcdir/../packages/brew_install_packages.sh\" circleci\nelse\n    curl -fLSs https://circle.ci/cli | DESTDIR=~/bin bash\nfi\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nif ! is_CI && [ -t 1 ]; then\n    circleci setup\nfi\n"
  },
  {
    "path": "install/install_circleci_runner.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-13 18:23:41 +0000 (Mon, 13 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://circleci.com/docs/2.0/runner-installation/#installation\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads the CircleCI Runner launch agent to ~/bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nos=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"  # linux or darwin\narch=\"$(uname -m)\"  # x86_64 or amd64\nif [ \"$arch\" = x86_64 ]; then\n    arch=\"amd64\"\nfi\nplatform=\"$os/$arch\"\n\nbase_url=\"https://circleci-binary-releases.s3.amazonaws.com/circleci-launch-agent\"\nagent_version=$(curl -sS --fail \"${base_url}/release.txt\")\n\ntimestamp \"Using CircleCI Launch Agent version $agent_version\"\necho >&2\n\nprefix=~/bin/circleci-runner\n\nmkdir -p \"$prefix/workdir\"\n\ntmp=\"$(mktemp -d)\"\n\ncd \"$tmp\"\n\ntimestamp \"Downloading and verifying CircleCI Launch Agent Binary\"\ncurl -sSL --fail \"$base_url/$agent_version/checksums.txt\" -o checksums.txt\nfile=\"$(grep -F \"$platform\" checksums.txt | cut -d ' ' -f 2 | sed 's/^.//')\"\necho >&2\n\nmkdir -p \"$platform\"\n\ntimestamp \"Downloading CircleCI Launch Agent: $file\"\ncurl -sS --fail --compressed -L \"$base_url/$agent_version/$file\" -o \"$file\"\necho >&2\n\ntimestamp \"Verifying CircleCI Launch Agent download\"\ngrep \"$file\" checksums.txt |\nsha256sum --check\n\nchmod +x \"$file\"\n\ncp -v -- \"$file\" \"$prefix/circleci-launch-agent\"\n"
  },
  {
    "path": "install/install_clairctl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-15 06:00:33 +0100 (Mon, 15 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Clairctl\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-4.6.1}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" \"quay/clair\" \"clairctl-{os}-{arch}\" \"$version\"\n"
  },
  {
    "path": "install/install_cliclick.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-14 17:01:04 +0100 (Sun, 14 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs CliClick on Mac OS X\n#\n# https://github.com/BlueM/cliclick\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nVERSION=\"4.0.1\"\n\nif [ \"$(uname -s)\" != Darwin ]; then\n    echo \"Operating System is not Mac, cannot install MouseTools which is for Mac, aborting...\"\n    exit 0\nfi\n\nif type -P cliclick &>/dev/null; then\n    echo \"CliClick already installed\"\n    exit 0\nfi\n\ncd /tmp\n\nwget -O \"cliclick-$VERSION.zip\" \"https://github.com/BlueM/cliclick/archive/$VERSION.zip\"\n\nunzip -o \"cliclick-$VERSION.zip\"\n\ncd \"cliclick-$VERSION\"\n\necho\necho \"Building cliclick\"\nmake\n\necho\nmv -iv -- cliclick ~/bin\n\necho\nrm -fr -- \"cliclick-$VERSION\" \"cliclick-$VERSION.zip\"\n\necho \"cliclick is now available at ~/bin - ensure that location is in $PATH\"\n"
  },
  {
    "path": "install/install_cloud_sql_proxy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-01 15:01:24 +0000 (Mon, 01 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cloud.google.com/sql/docs/postgres/sql-proxy\n\n# Installs Google Cloud SQL Proxy to $HOME/bin\n#\n# only supports 64-bit Linux / Mac (who uses 32-bit any more?)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif type -P cloud_sql_proxy 2>/dev/null; then\n    echo \"Google Cloud SQL Proxy is already installed, skipping install...\"\n    exit 0\nfi\n\nos=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n\nurl=\"https://dl.google.com/cloudsql/cloud_sql_proxy.$os.amd64\"\n\ntmpfile=\"$(mktemp)\"\n\necho \"Downloading Google Cloud SQL Proxy\"\nif type wget &>/dev/null; then\n    wget -qO \"$tmpfile\" \"$url\"\nelif type curl &>/dev/null; then\n    curl -sS \"$url\" > \"$tmpfile\"\nelse\n    echo \"Error: neither wget nor curl were found in your \\$PATH, cannot download cloud_sql_proxy\"\n    exit 1\nfi\n\necho \"setting executable\"\nchmod +x \"$tmpfile\"\n\necho \"moving to ~/bin\"\nmv -fv -- \"$tmpfile\" ~/bin/cloud_sql_proxy\n\necho \"Done. Don't forget to add $HOME/bin to \\$PATH\"\n"
  },
  {
    "path": "install/install_cloudbees.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 11:25:50 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls CloudBees Core CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"${1:-latest}\"\n\n\"$srcdir/../packages/install_binary.sh\" 'https://s3.amazonaws.com/cloudbees-core-cli/master/cloudbees-{os}-{arch}.tar.gz' cloudbees\n"
  },
  {
    "path": "install/install_coder_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-03 23:25:02 +0200 (Wed, 03 Jul 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the Coder CLI via GitHub binary releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-2.13.0}\"\nversion=\"${1:-latest}\"\n\n# on Mac just runs:\n#\n#   brew install coder/coder/coder\n#\n# which we can do via ../setup/brew-packages-desktop-taps.txt\n#\n#curl -L https://coder.com/install.sh | sh\n\nexport ARCH_X86_64=amd64\nexport ARCH_ARM64=amd64\n\nexport RUN_VERSION_ARG=1\n\next=\"tar.gz\"\nif is_mac; then\n    ext=\"zip\"\nfi\n\n\"$srcdir/../github/github_install_binary.sh\" coder/coder \"coder_{version}_{os}_{arch}.$ext\" \"$version\" coder\n"
  },
  {
    "path": "install/install_container-diff.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 10:27:12 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls container-diff to \\$HOME/bin\n\nDocumentation:\n\n    https://github.com/GoogleContainerTools/container-diff\n\"\n\nhelp_usage \"$@\"\n\ncd /tmp\n\nif is_mac; then\n    url=https://storage.googleapis.com/container-diff/latest/container-diff-darwin-amd64\nelse\n    url=https://storage.googleapis.com/container-diff/latest/container-diff-linux-amd64\nfi\n\nwget -O container-diff \"$url\"\n\nchmod +x container-diff\n\nmkdir -pv ~/bin\n\nmv -iv -- container-diff ~/bin\n"
  },
  {
    "path": "install/install_crictl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-28 14:23:46 +0100 (Sun, 28 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls crictl CLI for CRI-compatible container runtimes such as containerd\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-v1.27.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" kubernetes-sigs/cri-tools \"crictl-v{version}-{os}-{arch}.tar.gz\" \"$version\" crictl\n"
  },
  {
    "path": "install/install_d2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-02 21:13:23 +0100 (Tue, 02 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the D2 diagram scripting language\n\n    https://d2lang.com/\n\n    https://github.com/terrastruct/d2/blob/master/docs/INSTALL.md\n\n    https://github.com/terrastruct/tala#installation\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# This shows that on Mac it'll simply install d2 via homebrew\n#\n#   curl -fsSL https://d2lang.com/install.sh | sh -s -- --dry-run\n\n#if is_mac; then\n#    brew install d2\n#else\n    if ! \"$srcdir/../packages/install_packages.sh\" d2; then\n\n        if ! type -P curl &>/dev/null; then\n            \"$srcdir/../packages/install_packages.sh\" curl\n        fi\n\n        curl -fsSL https://d2lang.com/install.sh | sh -s --\n    fi\n#fi\n\n# ==========================================\n# Install proprietary Tala layout engine too\n#curl -fsSL https://d2lang.com/install.sh | sh -s -- --tala\n#\n# belay that order, ends up with horrible:\n#\n#   WARNING: THIS COPY OF TALA IS UNLICENSED AND IS FOR EVALUATION PURPOSES ONLY\n#\n# for every run of d2 even when not using the tala layout\n# ==========================================\n\n# can also install via go but won't get the man page\n#\n# go install oss.terrastruct.com/d2@latest\n"
  },
  {
    "path": "install/install_datree.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-04-14 10:51:19 +0100 (Thu, 14 Apr 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncurl https://get.datree.io | /bin/bash\necho\n\nif [ -n \"${DATREE_TOKEN:-}\" ]; then\n    echo \"\\$DATATREE_TOKEN found, configuring...\"\n    datree config set token \"$DATREE_TOKEN\"\nfi\n\necho\necho -n \"Datree version: \"\ndatree version\n"
  },
  {
    "path": "install/install_diff-so-fancy.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-23 22:30:21 +0100 (Mon, 23 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs diff-so-fancy on Mac and Linux\n#\n# .bashrc will detect and use diff-so-fancy if it is available in the $PATH,\n# by setting $GIT_PAGER\n#\n# ~/.gitconfig will take precedence though if a pager is explicitly specified there\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif type -P diff-so-fancy &>/dev/null; then\n    echo \"diff-so-fancy is already installed!\"\n    exit 0\nfi\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    if ! type -P brew &>/dev/null; then\n        \"$srcdir/install_homebrew.sh\"\n    fi\n    brew update\n    brew install diff-so-fancy\nelse\n#     if [ \"$(uname -s)\" = Linux ]; then\n#         if ! type -P npm &>/dev/null; then\n#           if type -P dnf &>/dev/null; then\n#               dnf install -y npm\n#           elif type -P yum &>/dev/null; then\n#               yum install -y npm\n#           elif type -P apt-get &>/dev/null; then\n#               # not available on Debian yet it seems, but present on Ubuntu\n#               apt-get update\n#               apt-get install -y npm\n#           elif type -P apk &>/dev/null; then\n#               apk update\n#               apk add npm\n#           fi\n#       fi\n#   fi\n    # Not using NPM as it is broken on both CentOS and Ubuntu\n    #\n    # https://github.com/so-fancy/diff-so-fancy/issues/350\n    #\n    # # npm install diff-so-fancy\n    # /\n    # └── diff-so-fancy@1.2.7\n    #\n    # npm WARN enoent ENOENT: no such file or directory, open '/package.json'\n    # npm WARN !invalid#1 No description\n    # npm WARN !invalid#1 No repository field.\n    # npm WARN !invalid#1 No README data\n    # npm WARN !invalid#1 No license field.\n    #\n    #if type -P npm &>/dev/null; then\n    #    npm install diff-so-fancy\n    #else\n        echo \"Downloading diff-so-fancy fatpack to ~/bin\"\n        mkdir -pv ~/bin\n        cd ~bin\n        wget https://raw.githubusercontent.com/so-fancy/diff-so-fancy/master/third_party/build_fatpack/diff-so-fancy\n        chmod +x diff-so-fancy\n    #fi\nfi\n"
  },
  {
    "path": "install/install_direnv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-10 13:09:20 +0300 (Sat, 10 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstall Direnv using the online install\n\nThe online install will install direnv to your local user writable \\$PATH\neven if there is a direnv already in the \\$PATH\n\nThis standardized install_<name>.sh script will check for direnv in \\$PATH and skip the install if found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_args \"$@\"\n\nif type -P direnv &>/dev/null; then\n    echo \"direnv is already installed at '$(type -P direnv)', skipping install\"\n    exit 0\nfi\n\n# clean PATH because the direnv installer will write the 'direnv' binary to the first available user writeable path\n# and we don't want it putting it somewhere like ~/github/bash-tools - mixed in with git repo and scripts\nPATH=\"$HOME/bin:$(tr ':' '\\n' <<< \"$PATH\" | grep -e '^/bin' -e '^/usr' | tr '\\n' ':' | sed 's/:$//')\"\nexport PATH\n\n# ensure we have at least one user writable directory\nmkdir -p -v ~/bin\n\ncurl -sfL https://direnv.net/install.sh | bash\necho\nversion=\"$(direnv version)\"\necho \"Direnv version: $version\"\n"
  },
  {
    "path": "install/install_docker_buildx.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-14 11:29:58 +0000 (Fri, 14 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Docker Buildx plugin to ~/.docker/cli-plugins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-latest}\"\n\nowner_repo=\"docker/buildx\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\n    [[ \"$version\" =~ ^v ]] || version=\"v$version\"\nfi\n\ninstall_dir=~/.docker/cli-plugins\n\nmkdir -p -v \"$install_dir\"\n\n\"$srcdir/../github/github_install_binary.sh\" docker/buildx \"buildx-$version.{os}-{arch}\" \"$version\" docker-buildx \"$install_dir/docker-buildx\"\n"
  },
  {
    "path": "install/install_docker_compose.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-19 19:31:41 +0000 (Thu, 19 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Docker Compose\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\narch=\"$(get_arch)\"\nif [ \"$arch\" = amd64 ]; then\n    arch=x86_64\nfi\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" docker/compose \"docker-compose-{os}-$arch\" \"$version\"\n"
  },
  {
    "path": "install/install_docker_scan.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-15 09:19:28 +0000 (Sat, 15 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Docker Scan plugin to ~/.docker/cli-plugins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nversion=\"${1:-latest}\"\n\ninstall_dir=~/.docker/cli-plugins\n\nmkdir -p -v \"$install_dir\"\n\n\"$srcdir/../github/github_install_binary.sh\" docker/scan-cli-plugin 'docker-scan_{os}_{arch}' \"$version\" \"$install_dir/docker-scan\"\n"
  },
  {
    "path": "install/install_dockerhub_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-04-22 17:22:15 +0100 (Fri, 22 Apr 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls DockerHub CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.30.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" docker/hub-tool \"hub-tool-{os}-{arch}.tar.gz\" \"$version\" hub-tool/hub-tool\n"
  },
  {
    "path": "install/install_dockle.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Dockle\n\nhttps://github.com/goodwithtech/dockle\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.4.11}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86=32bit\nexport ARCH_X86_64=64bit\nexport OS_DARWIN=macOS\nexport OS_LINUX=Linux\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" goodwithtech/dockle \"dockle_{version}_{os}-{arch}.tar.gz\" \"$version\" dockle\n"
  },
  {
    "path": "install/install_doctl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-05 23:50:40 +0100 (Tue, 05 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Digital Ocean CLI\n\nIf you're on Mac and you have \\$DIGITAL_OCEAN_TOKEN set in your environment, will configure it for the default context automatically if no access tokens are configured in your 'Library/Application Support/doctl/config.yaml'\nIf you have a token \\$DIGITALOCEAN_ACCESS_TOKEN this won't be needed because doctl will pick it up automatically so you don't need to configure the config.yml\n\nGenerate a personal access token here:\n\n    https://cloud.digitalocean.com/account/api/tokens\n\nCLI Command Reference:\n\n    https://docs.digitalocean.com/reference/doctl/reference/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.78.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" digitalocean/doctl 'doctl-{version}-{os}-{arch}.tar.gz' \"$version\" doctl\n\nDIGITAL_OCEAN_TOKEN=\"${DIGITAL_OCEAN_TOKEN:-${DIGITALOCEAN_TOKEN:-}}\"\n\nif [ -n \"${DIGITAL_OCEAN_TOKEN:-}\" ] &&\n    # if $DIGITALOCEAN_ACCESS_TOKEN is there the output will change ti 'Validating token... OK' and expect script will break\n   [ -z \"${DIGITALOCEAN_ACCESS_TOKEN:-}\" ]; then\n    if is_mac; then\n        if ! grep -Eq 'access-token: .{3,}' \"$HOME/Library/Application Support/doctl/config.yaml\"; then\n            echo\n            echo \"Setting up authentication\"\n            echo\n            \"$srcdir/doctl_auth_init.exp\"\n        fi\n    fi\nfi\n"
  },
  {
    "path": "install/install_drone.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-01 12:18:29 +0000 (Tue, 01 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Drone CI CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" harness/drone-cli 'drone_{os}_{arch}.tar.gz' \"$version\" drone\n"
  },
  {
    "path": "install/install_eksctl.sh",
    "content": "#!/usr/bin/env bash\n# vim:ts=4:sts=4:sw=4:et\n# shellcheck disable=SC2230\n# command -v catches aliases, not suitable\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 11:59:42 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#\"$srcdir/install_homebrew.sh\"\n#brew tap weaveworks/tap\n#brew install weaveworks/tap/eksctl\n#brew upgrade eksctl\n#brew link --overwrite eksctl\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls AWS eksctl CLI from WeaveWorks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" weaveworks/eksctl 'eksctl_{os}_{arch}.tar.gz' \"$version\" eksctl\n"
  },
  {
    "path": "install/install_eksup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 08:28:50 +0700 (Wed, 08 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls EKSup tool to analyze EKS clusters for upgrades\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-0.9.0}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86_64=x86_64\nexport ARCH_ARM64=aarch64\n\nexport RUN_VERSION_OPT=1\n\n# awful naming convention variations:\n#\n#   https://github.com/clowdhaus/eksup/releases\n#\nif is_mac; then\n    basename=\"eksup-v{version}-{arch}-apple-{os}\"\nelse\n    if uname -m | grep arm; then\n        basename=\"eksup-v{version}-{arch}-unknown-{os}-gnueabihf\"\n    else\n        basename=\"eksup-v{version}-{arch}-unknown-{os}-musl\"\n    fi\nfi\n\n\n\"$srcdir/../github/github_install_binary.sh\" clowdhaus/eksup \"$basename.tar.gz\" \"$version\" \"$basename/eksup\"\n"
  },
  {
    "path": "install/install_epel_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-17 11:11:52 +0000 (Sun, 17 Feb 2019)\n#        2013-07-17 21:27:25 +0100 (Wed, 17 Jul 2013)\n#           (found older version)\n#        2012-06-25 15:20:39 +0100\n#           (originally an alias in .bashrc)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ \"${NO_FAIL:-}\" ]; then\n    set +eo pipefail\nfi\n\nif grep -qi 'NAME=.*Fedora' /etc/*release; then\n    echo \"Detected Fedora, skipping epel install...\"\n    exit 0\nfi\n\nif rpm -q epel-release; then\n    echo \"EPEL rpm is already installed, skipping...\"\n    exit 0\nfi\n\n[ $EUID -eq 0 ] && sudo=\"\" || sudo=sudo\n\nif $sudo yum repolist | grep -qi '\\<epel\\>'; then\n    # accounts for custom internal EPEL mirrors which should have epel in the name\n    echo \"EPEL yum repo already detected in yum repolist, skipping...\"\n    exit 0\nfi\n\nif ! $sudo yum install -y epel-release; then\n    rpm -qi wget || yum install -y wget\n    major_release=\"$(grep -ho '[[:digit:]]' /etc/*release | head -n1)\"\n    wget -t 5 --retry-connrefused -O /tmp/epel.rpm \"https://dl.fedoraproject.org/pub/epel/epel-release-latest-$major_release.noarch.rpm\"\n    $sudo rpm -ivh /tmp/epel.rpm\n    rm -f -- /tmp/epel.rpm\nfi\n"
  },
  {
    "path": "install/install_etcd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-26 01:32:46 +0100 (Fri, 26 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Etcd\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-v3.5.9}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"etcd-io/etcd\"\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\next=\"tar.gz\"\nif [ \"$os\" = \"darwin\" ]; then\n    ext=\"zip\"\nfi\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nurl=\"https://github.com/$owner_repo/releases/download/$version/etcd-$version-$os-$arch.$ext\"\n\npackage=\"/tmp/etcd.$$.$ext\"\n\ncurl -sSLf \"$url\"  -o \"$package\"\n\nif [ \"$ext\" = \"tar.gz\" ]; then\n    sudo tar -zxv --strip-components=1 -C /usr/local/bin/\nelse\n    tmpdir=\"/tmp/etcd.$$\"\n    mkdir -p -v \"$tmpdir\"\n    echo\n    unzip -o -d \"$tmpdir\" \"$package\"\n    unalias rm 2>/dev/null || :\n    rm -fv \"$package\"\n    echo\n    unalias mv 2>/dev/null || :\n    sudo mv -fv \"$tmpdir/etcd-$version-$os-$arch/\"{etcd,etcdctl,etcdutl} /usr/local/bin/\n    rm -fr \"$tmpdir\"\nfi\n\necho\n/usr/local/bin/etcd --version\necho\n/usr/local/bin/etcdctl version\necho\n/usr/local/bin/etcdutl version\n"
  },
  {
    "path": "install/install_firebase_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-17 04:27:10 +0700 (Mon, 17 Feb 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Firebase CLI binary from GitHub releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-13.31.1}\"\nversion=\"${1:-latest}\"\n\nexport OS_DARWIN=macos\n\nexport RUN_VERSION_OPT=1\n\n#\"$srcdir/../packages/install_binary.sh\" \"https://firebase.tools/bin/{os}/$version\" firebase\n\n#https://github.com/firebase/firebase-tools/releases/download/v13.31.1/firebase-tools-macos\n\"$srcdir/../github/github_install_binary.sh\" firebase/firebase-tools \"firebase-tools-{os}\" \"$version\" firebase\n"
  },
  {
    "path": "install/install_fly.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-03 02:14:22 +0000 (Sun, 03 Mar 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Concourse Fly CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" concourse/concourse \"fly-{version}-{os}-{arch}.tgz\" \"$version\" \"fly-{version}-{os}-{arch}/bin/fly\"\n"
  },
  {
    "path": "install/install_fossa_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-05-22 12:48:07 +0100 (Sun, 22 May 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/fossas/fossa-cli\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncurl -H 'Cache-Control: no-cache' https://raw.githubusercontent.com/fossas/fossa-cli/master/install-latest.sh | bash\n"
  },
  {
    "path": "install/install_gcloud_sdk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-14 20:56:43 +0000 (Thu, 14 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install Google Cloud SDK - uses packages where available or else installs to $HOME via the script installer\n#\n# You may want to install manually if using Google App Engine as the docs say to not use the packaged versions of gcloud\n# when installing the app engine add-ons (probably due to path write permissions)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nexport PATH=\"$PATH:$HOME/google-cloud-sdk/bin\"\n\nif type -P gcloud &>/dev/null; then\n    echo \"GCloud already installed, skipping...\"\n    exit 0\nfi\n\n#apt_version=\"123.0.0-0\"\n#yum_version=\"123.0.0\"\n\n# installs to $HOME/google-cloud-sdk\ninstall_root_dir=~\n\napt_optional_packages=\"\ngoogle-cloud-sdk-app-engine-python \\\ngoogle-cloud-sdk-app-engine-python-extras \\\ngoogle-cloud-sdk-app-engine-java \\\ngoogle-cloud-sdk-app-engine-go \\\ngoogle-cloud-sdk-datalab \\\ngoogle-cloud-sdk-datastore-emulator \\\ngoogle-cloud-sdk-gke-gcloud-auth-plugin \\\ngoogle-cloud-sdk-pubsub-emulator \\\ngoogle-cloud-sdk-cbt \\\ngoogle-cloud-sdk-cloud-build-local \\\ngoogle-cloud-sdk-bigtable-emulator \\\nkubectl\n\"\n\nyum_optional_packages=\"\ngoogle-cloud-sdk-app-engine-python\ngoogle-cloud-sdk-app-engine-python-extras\ngoogle-cloud-sdk-app-engine-java\ngoogle-cloud-sdk-app-engine-go\ngoogle-cloud-sdk-bigtable-emulator\ngoogle-cloud-sdk-datalab\ngoogle-cloud-sdk-datastore-emulator\ngoogle-cloud-sdk-cbt\ngoogle-cloud-sdk-cloud-build-local\ngoogle-cloud-sdk-gke-gcloud-auth-plugin\ngoogle-cloud-sdk-pubsub-emulator\n\"\n# conflicts with kubernetes-client package\n#kubectl\n\nsudo=sudo\n[ $EUID -eq 0 ] && sudo=\"\"\n\necho \"Installing Google Cloud SDK\"\nif type -P yum &>/dev/null; then\n    $sudo tee -a /etc/yum.repos.d/google-cloud-sdk.repo << EOF\n    [google-cloud-sdk]\nname=Google Cloud SDK\nbaseurl=https://packages.cloud.google.com/yum/repos/cloud-sdk-el7-x86_64\nenabled=1\ngpgcheck=1\nrepo_gpgcheck=1\ngpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg\n       https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg\nEOF\n    yum install -y google-cloud-sdk #-$yum_version\n    # want splitting to single line\n    # shellcheck disable=SC2086\n    yum install -y $yum_optional_packages\n# https://cloud.google.com/sdk/docs/downloads-apt-get\nelif type -P apt-get &>/dev/null; then\n    google_cloud_sdk_source=\"deb [signed-by=/usr/share/keyrings/cloud.google.gpg] https://packages.cloud.google.com/apt cloud-sdk main\"\n    grep -Fq \"$google_cloud_sdk_source\" /etc/apt/sources.list.d/google-cloud-sdk.list 2>/dev/null ||\n    $sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list <<< \"$google_cloud_sdk_source\"\n    opts=\"-o DPkg::Lock::Timeout=1200\"\n    $sudo apt-get install -y $opts apt-transport-https ca-certificates gnupg\n    curl -sS https://packages.cloud.google.com/apt/doc/apt-key.gpg |\n        $sudo apt-key --keyring /usr/share/keyrings/cloud.google.gpg add -\n    $sudo apt-get update $opts\n    $sudo apt-get install -y $opts google-cloud-sdk #=$apt_version\n    # want splitting to single line\n    # shellcheck disable=SC2086\n    $sudo apt-get install -y $opts $apt_optional_packages\nelif [[ \"$(uname -s)\" =~ Darwin|Linux ]]; then\n    # https://cloud.google.com/sdk/docs/downloads-interactive\n    install_script=\"$(mktemp -t gcloud_installer.sh.XXXXXX)\"\n    curl -sS https://sdk.cloud.google.com > \"$install_script\"\n    chmod +x \"$install_script\"\n    \"$install_script\" --disable-prompts --install-dir=\"$install_root_dir\"\n    gcloud components install kubectl\n    if [ \"$(uname -s)\" = Darwin ]; then\n        gcloud components install gke-gcloud-auth-plugin\n        gke-gcloud-auth-plugin --version\n    fi\nelse\n    echo \"Unsupported OS '$(uname -s)'\" >&2\n    exit 1\nfi\n\n# requires interactive prompts\n#echo \"Initializing gcloud...\"\n#gcloud init\necho\necho \"Done. You will need to run 'gcloud init' to set up your profile.\"\n"
  },
  {
    "path": "install/install_github_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls GitHub CLI\n\nOnce installed, configure authentication by creating a Personal Access Token (PAT) here:\n\n    https://github.com/settings/tokens\n\nand then exporting that as an environment variable - either GH_TOKEN or GITHUB_TOKEN (the former has higher precedence so is recommended):\n\n    export GH_TOKEN=...\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86_64=amd64\nexport OS_DARWIN=macOS\n\next=\"tar.gz\"\nif is_mac; then\n    ext=\"zip\"\nfi\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" cli/cli \"gh_{version}_{os}_{arch}.$ext\" \"$version\" \"gh_{version}_{os}_{arch}/bin/gh\"\n"
  },
  {
    "path": "install/install_github_codeql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-26 17:34:18 +0100 (Tue, 26 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system\n#\n# https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls GitHub CodeQL CLI + bundle to \\$HOME/bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.10.1}\"\n#version=\"${1:-latest}\"\n\nos=\"$(get_os)\"\nif [ \"$os\" = darwin ]; then\n    os=osx\nfi\n\n# github/codeql only has source not binary, and it cannot be extracted via github_install_binary.sh - it must be run from the bundle, which is ~500MB\n\n                          # 64 - there are no i386 or other binaries\ntarball=\"codeql-bundle-${os}64.tar.gz\"\n\n#tmp=\"$(mktemp -d)\"\n#trap_cmd 'rm -fr \"$tmp\"'\n\n# better for caching, use flock instead\ntmp=/tmp\n\ncd \"$tmp\"\n\ntimestamp \"Downloading latest codeql tarball\"\necho\nwget -cO \"$tarball\" \"https://github.com/github/codeql-action/releases/latest/download/$tarball\"\necho\n\ntimestamp \"Extracting tarball\"\necho\nrm -fr -- ./codeql\ntar xvzf ./\"$tarball\"  # -C ~/bin/  # mv is more atomic, otherwise concurrent operations using codeql might break\necho\n\nunalias rm &>/dev/null || :\nunalias mv &>/dev/null || :\n\ntimestamp \"Removing tarball\"\necho\nrm -fv -- \"$tarball\"\necho\n\ntimestamp \"Moving CodeQL to \\$HOME/bin/\"\necho\nrm -fr -- ~/bin/codeql\nmv -fv -- codeql/ ~/bin/\necho\n\necho \"Version:\"\n~/bin/codeql/codeql version\necho\necho \"Don't forget to add \\$HOME/bin/codeql to \\$PATH\"\necho\n"
  },
  {
    "path": "install/install_github_ssh_keys.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-16\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs the SSH Key(s) from user's GitHub to the local $HOME/.ssh/authorized_keys\n#\n# Uses $GITHUB_USER or $USER as the expected GitHub user\n#\n# set $AUTHORIZED_KEYS to specify an alternative ssh keys location\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nhome=\"${HOME:-$(cd && pwd)}\"\nauthorized_keys=\"${AUTHORIZED_KEYS:-$home/.ssh/authorized_keys}\"\n\n\"$srcdir/../github/github_ssh_get_user_public_keys.sh\" |\nwhile read -r ssh_key; do\n    # skip comment lines\n    [ -z \"$(sed 's/#.*//; /^[[:space:]]*$/d' <<< \"$ssh_key\")\" ] && continue\n    echo \"Processing key:  $ssh_key\"\n    #algo_hash=\"$(awk '{print $1\" \"$2}' <<< \"$ssh_key\")\"\n    algo_hash=\"${ssh_key%%==*}\"\n    if [ -f \"$authorized_keys\" ] &&\n        grep -Fq \"$algo_hash\" \"$authorized_keys\"; then\n        echo \"Key already found in $authorized_keys, skipping...\"\n    else\n        echo \"Adding key to $authorized_keys\"\n        mkdir -vp \"$(dirname \"$authorized_keys\")\"\n        echo \"$ssh_key from GitHub\" >> \"$authorized_keys\"\n    fi\n    echo\n    echo \"ensuring correct 0600 permissions applied to $authorized_keys\"\n    chmod 0600 \"$authorized_keys\"\n    echo\ndone\necho Done\n"
  },
  {
    "path": "install/install_gitlab_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls GitLab CLI\n\nOnce installed, configure authentication by creating a Personal Access Token (PAT) here:\n\n    https://gitlab.com/-/user_settings/personal_access_tokens\n\nand then exporting that as an environment variable:\n\n    export GITLAB_TOKEN=...\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nversion=\"${1:-latest}\"\n\next=\"tar.gz\"\n\nexport ARCH_X86_64=amd64\nexport OS_DARWIN=macOS\n\nexport RUN_VERSION_ARG=1\n\n# https://gitlab.com/gitlab-org/cli/-/releases/v1.36.0/downloads/glab_1.36.0_macOS_x86_64.tar.gz\n\n\"$srcdir/../gitlab/gitlab_install_binary.sh\" gitlab-org/cli \"glab_{version}_{os}_{arch}.$ext\" \"$version\" \"bin/glab\"\n"
  },
  {
    "path": "install/install_golang.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-01-03 19:13:35 +0000 (Sun, 03 Jan 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Downloads Golang to ~/bin\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Golang to ~/bin\n\nUses GitHub CLI (installs it if not already installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-1.15.6}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"golang/go\"\n\ninstall_location=~/bin\n\nuname_s=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n\ntmp_tar=\"$(mktemp)\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub CLI API\"\n    # Golang has no GitHub releases but all tags\n    #version=\"$(\"$srcdir/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    type -P gh &>/dev/null || \"$srcdir/install_github_cli.sh\"\n    version=\"$(gh api \"repos/$owner_repo/tags\" \\\n                --jq '\n                    .[] |\n                    select(.name | test(\"^go[0-9]\")) |\n                    .name\n                ' --paginate |\n                head -n1 |\n                sed 's/^go//' || :)\"\n    if [ -z \"$version\" ]; then\n        die \"Failed to determine latest version of $owner_repo\"\n    fi\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nurl=\"https://golang.org/dl/go$version.$uname_s-amd64.tar.gz\"\n\nmkdir -p -v \"$install_location\"\n\necho \"$(date '+%F %T')  Downloading $url\"\nwget -cO \"$tmp_tar\" \"$url\"\n\necho \"$(date '+%F %T')  Unpacking to $install_location\"\ntar zxf \"$tmp_tar\" -C \"$install_location\"\n\necho\necho \"Golang installed to $install_location\"\necho\necho \"To make use of it set the following:\"\necho\necho \"export PATH=\\\"$install_location/go/bin:\\$PATH\\\"\"\necho \"export GOROOT=\\\"$install_location/go\\\"\"\n"
  },
  {
    "path": "install/install_gonogo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 15:57:02 +0700 (Wed, 08 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Fairwinds GoNoGo kubernetes upgrade checker tool\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.30}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" FairwindsOps/gonogo 'gonogo_{version}_{os}_{arch}.tar.gz' \"$version\" gonogo\n"
  },
  {
    "path": "install/install_gradle.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-08-01 10:17:55 +0100 (Mon, 01 Aug 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"================================================================================\"\necho \"                           G r a d l e   I n s t a l l\"\necho \"================================================================================\"\n\nGRADLE_VERSION=${1:-${GRADLE_VERSION:-7.3.3}}\n\nam_root(){\n    [ \"${EUID:-${UID:-$(id -n)}}\" = 0 ]\n}\nif am_root; then\n    BASE=/opt\nelse\n    BASE=~/bin\nfi\n\necho\ndate '+%F %T  Starting...'\nstart_time=\"$(date +%s)\"\necho\n\nif [ -e \"$BASE/gradle\" ]; then\n    echo \"$BASE/gradle already exists, skipping download\"\nelse\n    mkdir -pv \"$BASE\"\n    cd \"$BASE\"\n    wget -c -t 10 --retry-connrefused \"https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip\"\n    unzip \"gradle-$GRADLE_VERSION-bin.zip\"\n    rm -f -- \"gradle-$GRADLE_VERSION-bin.zip\"\n    echo\n    echo \"Gradle Install done\"\nfi\nif am_root; then\n    ln -sv -- \"gradle-$GRADLE_VERSION\" gradle\n    if ! [ -e /etc/profile.d/gradle.sh ]; then\n        echo \"Adding /etc/profile.d/gradle.sh\"\n        # shell execution tracing comes out in the file otherwise\n        set +x\n        cat >> /etc/profile.d/gradle.sh <<EOF\nexport GRADLE_HOME=/opt/gradle\nexport PATH=\\$PATH:\\$GRADLE_HOME/bin\nEOF\n    fi\nelse\n    ln -fsv -- ~/bin/\"gradle-$GRADLE_VERSION\"/bin/gradle ~/bin\n    echo \"Ensure you have ~/bin set in your \\$PATH\"\nfi\n\necho\ndate '+%F %T  Finished'\necho\nend_time=\"$(date +%s)\"\ntime_taken=\"$((end_time - start_time))\"\necho \"Completed in $time_taken secs\"\necho\necho \"==================================================\"\necho \"            Gradle Install Completed\"\necho \"==================================================\"\necho\n"
  },
  {
    "path": "install/install_groovy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-08-01 10:17:55 +0100 (Mon, 01 Aug 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"================================================================================\"\necho \"                           G r o o v y   I n s t a l l\"\necho \"================================================================================\"\n\nGROOVY_VERSION=${1:-${GROOVY_VERSION:-4.0.0}}\n\nam_root(){\n    [ \"${EUID:-${UID:-$(id -n)}}\" = 0 ]\n}\nif am_root; then\n    BASE=/opt\nelse\n    BASE=~/bin\nfi\n\ndate '+%F %T  Starting...'\nstart_time=\"$(date +%s)\"\necho\n\nif ! [ -e \"$BASE/groovy\" ]; then\n    mkdir -p \"$BASE\"\n    cd \"$BASE\"\n    wget -t 100 --retry-connrefused \"https://groovy.jfrog.io/artifactory/dist-release-local/groovy-zips/apache-groovy-binary-$GROOVY_VERSION.zip\"\n    unzip \"apache-groovy-binary-$GROOVY_VERSION.zip\"\n    ln -sv -- \"groovy-$GROOVY_VERSION\" groovy\n    rm -f -- \"apache-groovy-binary-$GROOVY_VERSION.zip\"\n    echo\n    echo \"Groovy Install done\"\nelse\n    echo \"$BASE/groovy already exists - doing nothing\"\nfi\nif am_root; then\n    if [ -d /etc/profile.d/groovy.sh ]; then\n        if ! [ -e /etc/profile.d/groovy.sh ]; then\n            echo \"Adding /etc/profile.d/groovy.sh\"\n            # shell execution tracing comes out in the file otherwise\n            set +x\n            cat >> /etc/profile.d/groovy.sh <<EOF\n    export GROOVY_HOME=/opt/groovy\n    export PATH=\\$PATH:\\$GROOVY_HOME/bin\nEOF\n        fi\n    fi\nelse\n    echo \"Ensure you have ~/bin/groovy/bin set in your \\$PATH\"\nfi\n\necho\ndate '+%F %T  Finished'\necho\nend_time=\"$(date +%s)\"\ntime_taken=\"$((end_time - start_time))\"\necho \"Completed in $time_taken secs\"\necho\necho \"==================================================\"\necho \"            Groovy Install Completed\"\necho \"==================================================\"\necho\n"
  },
  {
    "path": "install/install_grype.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Grype\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"${1:-0.30.0}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"anchore/grype\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\n    [[ \"$version\" =~ ^v ]] || version=\"v$version\"\nfi\n\n\ncurl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh |\nsh -s -- -b ~/bin \"$version\"\n"
  },
  {
    "path": "install/install_helm.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-29 17:38:41 +0100 (Fri, 29 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Helm CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"${1:-3.7.1}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"helm/helm\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    version=\"${version#v}\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://get.helm.sh/helm-v$version-{os}-{arch}.tar.gz\" \"{os}-{arch}/helm\"\n"
  },
  {
    "path": "install/install_homebrew.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-12\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install Homebrew on Mac OS X or Linux (used by AWS CLI on Linux)\n#\n# doesn't install on CentOS 6 any more\n#\n# https://github.com/Homebrew/brew/issues/7583#issuecomment-640379742\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif type -P brew &>/dev/null; then\n    echo \"HomeBrew already installed, skipping install...\"\nelse\n    echo \"===================\"\n    echo \"Installing HomeBrew\"\n    echo \"===================\"\n    echo\n    # root installs to first one, user installs to the latter\n    for x in /home/linuxbrew/.linuxbrew/bin ~/.linuxbrew/bin; do\n        if [ -d \"$x\" ]; then\n            export PATH=\"$PATH:$x\"\n        fi\n    done\n    # can't install until homebrew is installed, and these should already be present\n    #\"$srcdir/../packages/install_packages_if_absent.sh\" bash curl git sudo\n    cmds=\"\"\n    if [ \"$(uname -s)\" = Linux ]; then\n        # if we're in debug mode enable set -x inside the HomeBrew script so we can see what inside it is causing breakage\n        if [ -n \"${DEBUG_HOMEBREW:-}\" ]; then\n            cmds=\"set -x\"\n        fi\n        {\n            echo \"$cmds\"\n            # LinuxBrew has migrated to HomeBrew now\n            curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh\n            # but this requires newer curl and fails many CI builds - https://github.com/Homebrew/install/issues/367\n            #curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh\n        } |\n        {\n        # XXX: requires 'sudo' command to install now no matter whether run as root or a regular user :-/\n        if [ \"$EUID\" -eq 0 ]; then\n            user=linuxbrew\n            echo \"Installing HomeBrew on Linux as user $user\"\n            # Alpine has adduser\n            id \"$user\" 2>/dev/null || useradd \"$user\" || adduser -D \"$user\"\n            mkdir -p -v \"/home/$user\"\n            chown -R \"$user\" \"/home/$user\"\n            # can't just pass bash, and -s shell needs to be fully qualified path\n            su \"$user\" -s /bin/bash\n        else\n            echo \"Installing HomeBrew on Linux as user ${USER:-whoami}\"\n            # newer versions of HomeBrew require bash not sh due to use of [[\n            bash\n        fi\n        }\n    else\n        echo \"Installing HomeBrew on Mac as user ${USER:-whoami}\"\n        # now deprecated and replaced with the shell version below\n        #curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install | ruby\n        bash -c \"$(echo \"$cmds\"; curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)\"\n    fi\nfi\n"
  },
  {
    "path": "install/install_infoblox_ova.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230,SC2317\n#\n#  Author: Hari Sekhon\n#  Date: early 2019\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Download InfoBlox DDI OVA and calls it to trigger the import to VirtualBox pop up\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"InfoBlox DDI doesn't boot when the specs are downsized to fit on a Mac. You would need to install the OVA to a full VMware with 128GB RAM etc\"\necho \"Rest of script will not execute, but the code is left for reference\"\n\nexit 1\n\nINFOBLOX_URL=https://go.infoblox.com/downloads/vnios/8.4.4/nios-8.4.4-386831-2019-08-02-03-45-48-ddi.ova\n\nVM_NAME=\"${VM_NAME:-VNIOS System}\"\n\nCPUS=\"${CPUS:-2}\"\nRAM=\"${RAM:-4096}\"  # MB\nVRAM=\"${VRAM:-12}\"  # Video RAM MB\n\necho \"Downloading InfoBlox OVA to ~/Downloads/\"\n\ncd ~/Downloads/\n\ninfoblox_ova=\"${INFOBLOX_URL##*/}\"\n\nif [ \"$infoblox_ova\" ]; then\n    echo \"Found $infoblox_ova, skipping download\"\nelse\n    wget \"$INFOBLOX_URL\"\nfi\n\nif vboxmanage list vms | grep \"^\\\"$VM_NAME\\\"\"; then\n    echo \"$VM_NAME VM already found, skipping import\"\nelse\n    # vboxmanage import nios-8.4.4-386831-2019-08-02-03-45-48-ddi.ova\n    #VBoxManage: error: Cannot import until the license agreement listed above is accepted.\n    echo \"Importing $infoblox_ova\"\n    # could set $VM_NAME here but want to leave it as the default import\n    vboxmanage import --vsys 0 --vmname \"$VM_NAME\" --eula accept \"$infoblox_ova\"\nfi\n\nif vboxmanage showvminfo \"$VM_NAME\" | grep -q \"^State:[[:space:]]*running\"; then\n    echo \"VM already running, skipping configure and start\"\nelse\n    echo \"Configuring $VM_NAME with $CPUS CPUs, $RAM MB RAM, $VRAM MB Video RAM\"\n    vboxmanage modifyvm \"$VM_NAME\" --cpus \"$CPUS\" --memory \"$RAM\" --vram \"$VRAM\"\n\n    echo \"Starting $VM_NAME\"\n    vboxmanage startvm \"$VM_NAME\"\nfi\n"
  },
  {
    "path": "install/install_intellij_plugins.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-06-12 18:15:16 +0200 (Wed, 12 Jun 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\nplugins_txt=\"$srcdir/../setup/intellij-plugins.txt\"\n\nplugins_txt=\"$(readlink -f \"$plugins_txt\" || die \"Failed to find file: $srcdir/../setup/intellij-plugins.txt\")\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls IntelliJ Plugins listed in $plugins_txt\n\nEdit this file to add/comment/uncomment lines to select the plugins you want\nand then run this script to install them all in one shot\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\ntimestamp \"Parsing Plugin List\"\necho\nplugin_list=\"$(\n    sed '\n        s/#.*//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d;\n    ' \"$plugins_txt\" |\n    while read -r plugin; do\n        if [[ \"$plugin\" =~ [[:space:]] ]]; then\n            echo \"'$plugin'\"\n        else\n            echo \"$plugin\"\n        fi\n    done\n)\"\n\ntimestamp \"Plugins List:\"\necho\necho \"$plugin_list\"\necho\n\ntimestamp \"Installing Plugins\"\n# want to interpret quotes\n# shellcheck disable=SC2086\neval idea installPlugins $plugin_list\n\ntimestamp \"Plugins Installation Complete\"\n"
  },
  {
    "path": "install/install_java.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-28 15:47:41 +0100 (Tue, 28 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/bash-tools/lib/utils.sh\"\n\nexport PATH=\"$srcdir:$srcdir/..:$PATH\"\n\nif type -P yum &>/dev/null; then\n    # RHEL / CentOS does the right thing and pulls in the current version\n    #java\n    #java-headless\n    install_packages.sh java-headless  # won't install to $PATH, make sure to add /usr/lib/jvm/jre/bin/ to $PATH (jre is a symlink)\nelif type -P apt-get &>/dev/null; then\n    # Debian / Ubuntu\n    #openjdk-8-jre-headless  # smaller than openjdk-11 package (127 vs 200 MB) and more tested\n    #openjdk-11-jre-headless\n    install_packages.sh openjdk-11-jre-headless\nelif type -P apk &>/dev/null; then\n    # Alpine\n    #openjdk8-jre\n    #openjdk9-jre-headless\n    #openjdk10-jre-headless\n    #openjdk11-jre-headless\n    install_packages.sh openjdk8-jre\nfi\n"
  },
  {
    "path": "install/install_jfrog_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-14 16:44:47 +0000 (Tue, 14 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://jfrog.com/getcli/\n\n# https://www.jfrog.com/confluence/display/CLI/JFrog+CLI\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\ntmp=\"$(mktemp -d)\"\ncd \"$tmp\"\n\n# both give the same actual binary\n\n# installs as 'jfrog' in ~/bin\n#curl -fL https://getcli.jfrog.io | bash -s v2\n#mv -iv -- jfrog ~/bin/jfrog\n\n# installs as 'jf' in /usr/local/bin\ncurl -fL \"https://install-cli.jfrog.io\" | sh\n\necho\njf --version\n\nif [ -n \"${JFROG_TOKEN:-}\" ]; then\n    echo\n    jf setup \"$JFROG_TOKEN\"\nfi\n"
  },
  {
    "path": "install/install_jx.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-16 08:11:53 +0100 (Wed, 16 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://jenkins-x.io/docs/install-setup/install-binary/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls JenkinsX CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" jenkins-x/jx 'jx-{os}-{arch}.tar.gz' \"$version\" jx\n\necho >&2\njx version --short\n"
  },
  {
    "path": "install/install_k3d.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-01 00:05:05 +0100 (Sat, 01 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls K3d wrapper for k3s mini kubernetes distribution\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version_tag>]\"\n\nhelp_usage \"$@\"\n\nversion=\"${1:-}\"\n\nmax_args \"$@\"\n\nif [ -n \"$version\" ]; then\n    export TAG=\"$version\"\nfi\n\ntimestamp \"Installing K3d\"\n\ncurl -sSf https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash\n"
  },
  {
    "path": "install/install_k3s.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-16 10:15:52 +0100 (Tue, 16 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# XXX: doesn't install on new M1 Apple chip\n#\n#   [ERROR]  Can not find systemd or openrc to use as a process supervisor for k3s\n#\n# GitHub issue:\n#\n#   https://github.com/k3s-io/k3s/issues/734\n#\n# Workaround - use k3d instead:\n#\n#   install_k3d.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls K3s mini kubernetes distribution and adds it to the kubectl config\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nk3s_yaml=\"/etc/rancher/k3s/k3s.yaml\"\nkubeconfig=\"${KUBECONFIG:-~/.kube/config}\"\n\ntimestamp \"Installing K3s\"\n\ncurl -sfL https://get.k3s.io | sh -\n\ntimestamp \"Copying $k3s_yaml to $kubeconfig so we can use regular 'kubectl' instead of 'k3s kubectl'\"\n\nmkdir -pv \"$(dirname \"$kubeconfig\")\"\n\ncat \"$k3s_yaml\" >> \"$kubeconfig\"\n"
  },
  {
    "path": "install/install_k6.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-08 15:17:15 +0100 (Fri, 08 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the K6 load testing CLI by Grafana Labs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.39.0}\"\nversion=\"${1:-latest}\"\n\n# Mac packages has both macos instead of darwin, and zip instead of tarball\nexport OS_DARWIN=macos\next=\"tar.gz\"\n\nif is_mac; then\n    ext=\"zip\"\nfi\n\npackage=\"k6-v{version}-{os}-{arch}.$ext\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" grafana/k6 \"$package\" \"$version\" \"k6-v{version}-{os}-{arch}/k6\" k6\n"
  },
  {
    "path": "install/install_keeper_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-05 14:29:41 +0700 (Wed, 05 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and runs the Keeper CLI installer on Mac\n\nIf not on Mac, tries to install Keeper CLI via Python pip\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<version>\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-17.0.8}\"\nversion=\"${1:-latest}\"\n\nif ! is_mac; then\n    pip3 install keepercommander\n    exit 0\nfi\n\nexport OS_DARWIN=mac\n\nowner_repo=\"Keeper-Security/Commander\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nif ! [[ \"$version\" =~ ^v ]]; then\n    version=\"v$version\"\nfi\n\narch=\"$(get_arch)\"\n\ndownload_url=\"https://github.com/$owner_repo/releases/download/$version/keeper-commander-mac-$arch-$version.pkg\"\nfile=\"${download_url##*/}\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n\n\"$srcdir/../bin/open.sh\" \"$file\"\n"
  },
  {
    "path": "install/install_kics.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-01 13:25:37 +0000 (Tue, 01 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kics 1.5.1 (the last version to support downloadable binaries)\n\nIt's recommended to use the docker image instead now\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# the last version to support downloadable binaries\nversion=\"${1:-1.5.1}\"\n\nowner_repo=\"Checkmarx/kics\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\n    version=\"${version#v}\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\nif [ \"$arch\" = \"amd64\" ]; then\n    arch=\"x64\"\nfi\n\n# tarballs unpacks locally so create dir\ninstalldir=~/bin/\"kics_${version}\"\n\nmkdir -pv \"$installdir\"\n\ncd \"$installdir\"\n\ntarball=\"kics_${version}_${os}_$arch.tar.gz\"\n\n# wget isn't available on GCloud SDK container\ncurl -sSLf -o \"$tarball\" \"https://github.com/Checkmarx/kics/releases/download/v$version/$tarball\"\necho\n\necho \"unpacking tarball to: $PWD\"\ntar zxf \"$tarball\"\necho\n\necho \"removing tarball:\"\nrm -fv -- \"$tarball\"\necho\n\necho \"symlinking install dir:\"\nln -sfhv -- \"$installdir\" ~/bin/kics ||\n# GCloud SDK version of 'ln' command doesn't have the -h switch\nln -sfv -- \"$installdir\" ~/bin/kics\necho\n\necho \"Ensure $HOME/bin/kics is added to your \\$PATH\"\n"
  },
  {
    "path": "install/install_kind.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-13 10:03:47 +0100 (Sat, 13 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kind (Kubernetes in Docker)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" kubernetes-sigs/kind 'kind-{os}-{arch}' \"$version\"\n"
  },
  {
    "path": "install/install_knative_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-16 01:46:54 +0100 (Sun, 16 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://knative.dev/docs/client/install-kn/\n#\n# https://knative.dev/docs/client/kn-plugins/#list-of-knative-plugins\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Knative CLI on Mac using Brew for kn and func\n\nThen determines the latest versions of the following plugins and installs them from the GitHub release pages:\n\n    func\n    kn-operator\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<func_version>] [<options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-latest}\"\n\ntimestamp \"Installing knative/client/kn\"\nbrew install knative/client/kn\necho\n\ntimestamp \"Installing knative-sandbox/kn-plugins/quickstart\"\nbrew install knative-sandbox/kn-plugins/quickstart\necho\n\ntimestamp \"Open brew tap knative-sandbox/kn-plugins\"\nbrew tap knative-sandbox/kn-plugins\necho\n\ntimestamp \"Installing func\"\nbrew install func\necho\n\ntimestamp \"Downloading func latest release as fn plugin\"\n\n\"$srcdir/../github/github_install_binary.sh\" knative/func \"func_{os}_{arch}\" \"$version\" kn-func\n\n\"$srcdir/../github/github_install_binary.sh\" knative-sandbox/kn-plugin-operator \"kn-operator-{os}-{arch}\" \"$version\" kn-operator\n\necho \"Knative plugins installed:\"\necho\nkn plugin list\necho\n\necho -n \"Knative func plugin version: \"\nkn func version\n"
  },
  {
    "path": "install/install_kops.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-10 10:49:47 +0100 (Wed, 10 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kubernetes Kops CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.24.1}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" kubernetes/kops 'kops-{os}-{arch}' \"$version\"\n"
  },
  {
    "path": "install/install_kubectl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 12:08:44 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://kubernetes.io/docs/tasks/tools/install-kubectl/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kubernetes 'kubectl' CLI\n\nInstalls kubectl binary to /usr/local/bin/ or \\$HOME/bin/ depending on your permissions\n\nFor systems with package managers it may be automatically installed using the packages for each\npackage manager:\n\n    $srcdir/../setup/*packages*.txt\n\nOfficial documentation:\n\n    https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-latest}\"\n\nif [ \"$version\" = latest ]; then\n    version=\"$(curl -sSL https://dl.k8s.io/release/stable.txt)\"\n    version=\"${version#v}\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\n# gives this error:\n# WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.\n# The connection to the server localhost:8080 was refused - did you specify the right host or port?\n#export RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://dl.k8s.io/release/v$version/bin/{os}/{arch}/kubectl\"\n\necho\nif [ -w /usr/local/bin ]; then\n    echo \"/usr/local/bin/kubectl version:\"\n    echo\n    /usr/local/bin/kubectl version --client\nelse\n    echo \"${HOME:-~}/bin/kubectl version:\"\n    echo\n    ~/bin/kubectl version --client\nfi\n"
  },
  {
    "path": "install/install_kubectl_plugin_cert_manager.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 12:08:44 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://cert-manager.io/docs/usage/kubectl-plugin/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kubernetes 'kubectl' plugin for cert-manager\n\nAlso pre-installs kubectl if not already present\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.1.0}\"\nversion=\"${1:-latest}\"\n\nbinary=\"kubectl-cert_manager\"\n\nif ! type -P kubectl &>/dev/null; then\n    timestamp \"Kubectl not installed, pre-installing...\"\n    echo\n    \"$srcdir/install_kubectl.sh\"\n    echo\nfi\n\n# can rusult in error trying to contact k8s cluster\n#export RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" cert-manager/cert-manager 'kubectl-cert_manager-{os}-{arch}.tar.gz' \"$version\" \"$binary\"\n"
  },
  {
    "path": "install/install_kubectl_plugin_convert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-01 04:14:29 +0700 (Wed, 01 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the latest version of the kubectl-convert plugin\n\nAlso pre-installs kubectl if not already present\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nif ! type -P kubectl &>/dev/null; then\n    timestamp \"Kubectl not installed, pre-installing...\"\n    echo\n    \"$srcdir/install_kubectl.sh\"\n    echo\nfi\n\ntimestamp \"Getting latest stable version number\"\nversion=\"$(curl -L -s https://dl.k8s.io/release/stable.txt)\"\ntimestamp \"Stable version is: $version\"\n\necho\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\n\ntimestamp \"Downloading kubectl-convert version '$version' for '$os' arch '$arch'\"\necho\nwget -c \"https://dl.k8s.io/release/$version/bin/$os/$arch/kubectl-convert\"\necho\ntimestamp \"Downloaded binary\"\n\necho\n\ntimestamp \"Getting SHA checksum\"\necho\nwget -c \"https://dl.k8s.io/release/$version/bin/$os/$arch/kubectl-convert.sha256\"\necho\ntimestamp \"Downloaded checksum\"\n\necho\n\ntimestamp \"Checking SHA checksum\"\necho \"$(cat kubectl-convert.sha256) kubectl-convert\" |\nsha256sum --check #<<< \"$(cat kubectl-convert.sha256) kubectl-convert\"\n\necho\n\n# doesn't support file://\n#\"$srcdir/../packages/install_binary.sh\" \"file://$PWD/kubectl-convert\"\n\n# user might not have sudo rights - just install to $HOME/bin if they want it in /usr/local/bin they can sudo this script\n#\n#sudo=\"\"\n#[ \"$EUID\" = 0 ] || sudo=sudo\n#\n#$sudo install -o root -g root -m 0755 kubectl-convert /usr/local/bin/kubectl-convert\n\nif [ \"$EUID\" = 0 ]; then\n    timestamp \"Installing to /usr/local/bin\"\n    echo\n    # on Mac breaks as expects a numeric UID\n    #install -v -o root -g root -m 0755 kubectl-convert /usr/local/bin/kubectl-convert\n    install -v -m 0755 kubectl-convert /usr/local/bin/kubectl-convert\nelse\n    timestamp \"Installing to $HOME/bin\"\n    echo\n    mkdir -p -v ~/bin\n    install -v -m 0755 kubectl-convert ~/bin/kubectl-convert\nfi\necho\n\ntimestamp \"Checking the plugin is installed properly by calling it through kubectl\"\necho\n\nkubectl convert --help\n\nrm kubectl-convert\nrm kubectl-convert.sha256\n"
  },
  {
    "path": "install/install_kubectl_plugin_krew.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Wed Jan 8 05:46:28 2025 +0700\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the latest version of the kubectl krew plugin\n\nAlso pre-installs kubectl if not already present\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"${KREW_ROOT:-$HOME/.krew}/bin:$PATH\"\n\nif ! type -P kubectl &>/dev/null; then\n    timestamp \"Kubectl not installed, pre-installing...\"\n    echo\n    \"$srcdir/install_kubectl.sh\"\n    echo\nfi\n\ntmp=\"$(mktemp -d)\"\n\ncd \"$tmp\"\n\nos=\"$(get_os)\"\n\narch=\"$(get_arch)\"\n\nkrew=\"krew-${os}_${arch}\"\n\ntimestamp \"Downloading latest $krew.tar.gz\"\ncurl -fsSLO \"https://github.com/kubernetes-sigs/krew/releases/latest/download/$krew.tar.gz\"\necho\n\ntimestamp \"Extracting $krew.tar.gz\"\necho\n\ntar zxvf \"$krew.tar.gz\"\n\necho\n\ntimestamp \"Installing krew\"\n\"./$krew\" install krew\necho\n\ntimestamp \"Install Complete\"\n"
  },
  {
    "path": "install/install_kubent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-01 03:26:36 +0700 (Wed, 01 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kube-No-Trouble from GitHub binary releases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-0.7.3}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86_64=amd64\nexport OS_DARWIN=darwin\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" doitintl/kube-no-trouble \"kubent-{version}-{os}-{arch}.tar.gz\" \"$version\" \"kubent\"\necho\n"
  },
  {
    "path": "install/install_kubescape.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-11 10:21:17 +0100 (Tue, 11 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ncurl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash\n"
  },
  {
    "path": "install/install_kubeseal.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kubeseal for Bitnami Sealed Secrets\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.18.1}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" bitnami-labs/sealed-secrets \"kubeseal-{version}-{os}-{arch}.tar.gz\" \"$version\" kubeseal\n"
  },
  {
    "path": "install/install_kubevious.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-23 23:29:32 +0100 (Tue, 23 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kubevious CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-v1.0.55}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86_64=x64\nexport OS_DARWIN=macos\n# has a different binary for Alpine - probably due to compilation against musl instead of glibc\nif [ -f /etc/alpine-release ]; then\n    export OS_LINUX=alpine\nfi\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" kubevious/cli \"kubevious-{os}-{arch}\" \"$version\"\n"
  },
  {
    "path": "install/install_kustomize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 12:08:44 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://kubernetes-sigs.github.io/kustomize/installation/binaries/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Kustomize\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"${1:-4.4.1}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"kubernetes-sigs/kustomize\"\n\n# XXX can't use github_install_binary.sh because Kustomize repo intermixes kustomize and kyaml releases, so much filter the latest version, then strip the kustomize/ prefix in the version tag\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release_filter.sh\" \"$owner_repo\" \"kustomize\")\"\n    version=\"${version##*/}\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\n    [[ \"$version\" =~ ^v ]] || version=\"v$version\"\nfi\n\n# now installs to /private and fails as user :-/\n#curl -s \"https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh\"  | bash\n\n# Kustomize 3.5.x gets an error like this when using kustomization.yaml with a double slash in the URL:\n#\n#     Error: accumulating resources: accumulateFile \"accumulating resources from 'github.com/argoproj/argo-cd//manifests/cluster-install?ref=v2.0.3': evalsymlink failure on '/tmp/git@repo/argocd/overlay/github.com/argoproj/argo-cd/manifests/cluster-install?ref=v2.0.3' : lstat /tmp/git@repo/argocd/overlay/github.com: no such file or directory\", accumulateDirector: \"recursed accumulation of path '/tmp/kustomize-881686007/repo': accumulating resources: accumulateFile \\\"accumulating resources from '../namespace-install': evalsymlink failure on '/tmp/kustomize-881686007/namespace-install' : lstat /tmp/kustomize-881686007/namespace-install: no such file or directory\\\", loader.New \\\"Error loading ../namespace-install with git: url lacks host: ../namespace-install, dir: evalsymlink failure on '/tmp/kustomize-881686007/namespace-install' : lstat /tmp/kustomize-881686007/namespace-install: no such file or directory, get: invalid source string: ../namespace-install\\\"\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://github.com/$owner_repo/releases/download/kustomize%2F$version/kustomize_${version}_{os}_amd64.tar.gz\" kustomize\n"
  },
  {
    "path": "install/install_maven.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-08-01 10:17:55 +0100 (Mon, 01 Aug 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"================================================================================\"\necho \"                           M a v e n   I n s t a l l\"\necho \"================================================================================\"\n\nMAVEN_VERSION=${1:-${MAVEN_VERSION:-3.3.9}}\n\nBASE=/opt\n\necho\ndate '+%F %T  Starting...'\nstart_time=\"$(date +%s)\"\necho\n\nif ! [ -e \"$BASE/maven\" ]; then\n    mkdir -p \"$BASE\"\n    cd \"$BASE\"\n    wget -t 100 --retry-connrefused \"https://archive.apache.org/dist/maven/maven-3/$MAVEN_VERSION/binaries/apache-maven-$MAVEN_VERSION-bin.tar.gz\"\n    tar zxvf \"apache-maven-$MAVEN_VERSION-bin.tar.gz\"\n    ln -sv -- \"apache-maven-$MAVEN_VERSION\" maven\n    rm -f -- \"apache-maven-$MAVEN_VERSION-bin.tar.gz\"\n    echo\n    echo \"Maven Install done\"\nelse\n    echo \"$BASE/maven already exists - doing nothing\"\nfi\nif ! [ -e /etc/profile.d/maven.sh ]; then\n    echo \"Adding /etc/profile.d/maven.sh\"\n    # shell execution tracing comes out in the file otherwise\n    set +x\n    cat >> /etc/profile.d/maven.sh <<EOF\nexport MAVEN_HOME=/opt/maven\nexport PATH=\\$PATH:\\$MAVEN_HOME/bin\nEOF\nfi\n\necho\ndate '+%F %T  Finished'\necho\nend_time=\"$(date +%s)\"\ntime_taken=\"$((end_time - start_time))\"\necho \"Completed in $time_taken secs\"\necho\necho \"==================================================\"\necho \"            Maven Install Completed\"\necho \"==================================================\"\necho\n"
  },
  {
    "path": "install/install_mermaidjs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-22 01:06:35 +0700 (Sat, 22 Feb 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the MermaidJS diagram scripting language CLI\n\nInstalls NodeJS system package if not already installed,\nand then the npm module @mermaid-js/mermaid-cli\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/../packages/install_packages_if_absent.sh\" nodejs\n\n\"$srcdir/../packages/nodejs_npm_install_if_absent.sh\" @mermaid-js/mermaid-cli\n"
  },
  {
    "path": "install/install_minikube.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: early 2019\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs MiniKube on Mac - needs VirtualBox to be installed first\n\n#set -euo pipefail\nset -u\n\n#[ -n \"${DEBUG:-}\" ] &&\nset -x\n\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    if ! type -P minikube &>/dev/null; then\n        if ! type -P brew &>/dev/null; then\n            echo \"HomeBrew needs to be installed first, trying to install now\"\n            \"$srcdir/install_homebrew.sh\"\n        fi\n        brew update\n        brew cask install minikube\n        brew install docker-machine-driver-xhyve\n    fi\n    brew_prefix=\"$(brew --prefix)\"\n    sudo chown root:wheel \"$brew_prefix\"/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve\n    sudo chmod u+s \"$brew_prefix\"/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve\n    if [ -z \"${NO_START:-}\" ] &&\n       [ -z \"${QUICK:-}\" ]    &&\n       ! minikube status | grep -i Running; then\n        minikube start\n    fi\nelse\n    echo \"Only Mac is supported at this time\"\n    exit 1\nfi\n"
  },
  {
    "path": "install/install_minishift.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: Aug 2019\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs MiniShift on Mac - needs VirtualBox to be installed first\n\n#set -euo pipefail\nset -u\n\n#[ -n \"${DEBUG:-}\" ] &&\nset -x\n\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    if ! type -P minishift &>/dev/null; then\n        if ! type -P brew &>/dev/null; then\n            echo \"HomeBrew needs to be installed first, trying to install now\"\n            \"$srcdir/install_homebrew.sh\"\n        fi\n        brew update\n        brew cask install minishift\n        brew cask install --force minishift\n        brew install docker-machine-driver-xhyve\n    fi\n    brew_prefix=\"$(brew --prefix)\"\n    sudo chown root:wheel \"$brew_prefix\"/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve\n    sudo chmod u+s \"$brew_prefix\"/opt/docker-machine-driver-xhyve/bin/docker-machine-driver-xhyve\n    if [ -z \"${NO_START:-}\" ] &&\n       [ -z \"${QUICK:-}\" ]; then\n       # quicker to just try, it'll tell you if it's already running\n       #! minishift status | grep -i Running; then\n        minishift start --vm-driver=virtualbox\n    fi\n    # .bash.d/kubernetes.sh automatically sources this so 'oc' command is available in all new shells\n    # fails if no running minishift VM - in which case remove the .minishift.env to avoid errors on every new shell\n    minishift oc-env > ~/.minishift.env || rm -f -- ~/.minishift.env\nelse\n    echo \"Only Mac is supported at this time\"\n    exit 1\nfi\n"
  },
  {
    "path": "install/install_mousetools.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-14 15:47:33 +0100 (Sun, 14 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# XXX: looks like this tool has disappeared from the website\n\n# Installs MouseTools on Mac OS X\n#\n# http://www.hamsoftengineering.com/codeSharing/MouseTools/MouseTools.html\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ \"$(uname -s)\" != Darwin ]; then\n    echo \"Operating System is not Mac, cannot install MouseTools which is for Mac, aborting...\"\n    exit 0\nfi\n\nif type -P MouseTools &>/dev/null; then\n    echo \"MouseTools already installed\"\n    exit 0\nfi\n\ncd /tmp\n\nwget -O MouseTools.zip http://www.hamsoftengineering.com/assets/MouseTools.zip\n\nunzip -o -- MouseTools.zip\n\nmv -iv -- MouseTools ~/bin\n\nrm -- MouseTools.zip\n\necho\necho \"MouseTools is now available at ~/bin - ensure that location is in $PATH\"\n"
  },
  {
    "path": "install/install_ngrok.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-11 10:07:36 +0000 (Tue, 11 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls NGrok CLI\n\nIf \\$NGROK_AUTHTOKEN or \\$NGROK_TOKEN is set, configures authentication using that token, in that order of precedence\n\nhttps://dashboard.ngrok.com/get-started/your-authtoken\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-v3-stable}\"\n\nexport RUN_VERSION_ARG=1\n\n# XXX: passing 'v2-stable' as version the URL gets updated but it still downloads v3, not sure how to predict this :-/\n\"$srcdir/../packages/install_binary.sh\" \"https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-$version-{os}-{arch}.zip\" \"ngrok\"\n\nTOKEN=\"${NGROK_AUTHTOKEN:-${NGROK_TOKEN:-}}\"\n\nif [ -n \"${TOKEN:-}\" ]; then\n    echo\n    timestamp \"found, \\$NGROK_TOKEN, configuring authentication\"\n    ngrok config add-authtoken \"$TOKEN\"\n    timestamp \"authentication configured\"\nfi\n"
  },
  {
    "path": "install/install_nova.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Fairwinds Nova\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-3.0.2}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" FairwindsOps/nova 'nova_{version}_{os}_{arch}.tar.gz' \"$version\" nova\n"
  },
  {
    "path": "install/install_octo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-17 16:45:49 +0100 (Wed, 17 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstall Octopus Deploy CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-9.0.0}\"\n\nexport PATH+=':'~/bin\n\nexport OS_DARWIN=osx\nexport ARCH_X86_64=x64\n\n\"$srcdir/../packages/install_binary.sh\" \"https://download.octopusdeploy.com/octopus-tools/$version/OctopusTools.$version.{os}-{arch}.tar.gz\" octo\n\n# mkdir -pv ~/.bash.autocomplete.d/\n#octo install-autocomplete --shell bash --dryRun > ~/.bash.autocomplete.d/octo.sh\n#\n# gets this error:\n#\n#    Octopus CLI, version 9.0.0\n#\n#    DRY RUN\n#    Installing auto-complete scripts for Bash\n#    Installing scripts in /Users/hari/.bashrc\n#    Updating profile at /Users/hari/.bashrc\n#    Preview of script changes:\n#\n#    System.IndexOutOfRangeException: Index was outside the bounds of the array.\n#       at Serilog.Parsing.MessageTemplateParser.ParsePropertyToken(Int32 startAt, String messageTemplate, Int32& next)\n#       at Serilog.Parsing.MessageTemplateParser.Tokenize(String messageTemplate)+MoveNext()\n#       at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)\n#       at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)\n#       at Serilog.Events.MessageTemplate..ctor(String text, IEnumerable`1 tokens)\n#       at Serilog.Parsing.MessageTemplateParser.Parse(String messageTemplate)\n#       at Serilog.Core.Pipeline.MessageTemplateCache.Parse(String messageTemplate)\n#       at Serilog.Parameters.MessageTemplateProcessor.Process(String messageTemplate, Object[] messageTemplateParameters, MessageTemplate& parsedTemplate, IEnumerable`1& properties)\n#       at Serilog.Core.Logger.Write(LogEventLevel level, Exception exception, String messageTemplate, Object[] propertyValues)\n#       at Serilog.Core.Logger.Information(Exception exception, String messageTemplate, Object[] propertyValues)\n#       at Serilog.Core.Logger.Information(String messageTemplate, Object[] propertyValues)\n#       at Octopus.CommandLine.CommandOutputProvider.Information(String template, Object[] propertyValues)\n#       at Octopus.CommandLine.ShellCompletion.ShellCompletionInstaller.Install(Boolean dryRun)\n#       at Octopus.CommandLine.Commands.InstallAutoCompleteCommand.Execute(String[] commandLineArguments)\n#       at Octopus.Cli.CliProgram.Run(String[] args)\n#    Exit code: -3\n"
  },
  {
    "path": "install/install_oh-my-zsh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-01 21:01:50 +0000 (Sun, 01 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"downloading Oh-my-Zsh...\"\nsh -c \"$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)\"\n"
  },
  {
    "path": "install/install_openssh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-15 18:19:22 +0100 (Wed, 15 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install OpenSSH - callable via curl to bash from builds\n#\n# Written for use in AppVeyor to install OpenSSH if not installed already to work around issues:\n#\n# https://github.com/appveyor/ci/issues/3373\n#\n# https://github.com/appveyor/ci/issues/3384\n#\n# has since been added to AppVeyor's own scripts:\n#\n# https://github.com/appveyor/ci/pull/3385\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\nif ! command -v sshd &>/dev/null; then\n    if command -v apt-get &>/dev/null; then\n        opts=\"-o DPkg::Lock::Timeout=1200\"\n        $sudo apt-get update $opts\n        $sudo apt-get install -y $opts openssh-server\n    elif command -v yum &>/dev/null; then\n        $sudo yum install -y openssh-server\n    elif command -v apk &>/dev/null; then\n        $sudo apk update\n        $sudo apk add openssh-server\n    fi\nfi\n"
  },
  {
    "path": "install/install_oracle_client.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-12 01:54:30 +0300 (Sat, 12 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\ndownloads_url=\"https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html\"\n\nopt_base=\"/opt/oracle\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the Oracle InstantClient packages including SQLPlus, JDBC, ODDC, SDK and Tools\n\nIf the first arg is 'list' then just lists discovered versions and exits so you can choose the version you want if not using latest\n\nVersion can be a prefix major or major.minor number and it'll take whatever the first patch version of that release which is found\n\nSome RPM versions will not install on some versions of Redhat based systems due to compatibility, you are advised to try the latest version\n\nYou can also look here and fetch manually if you really want:\n\n    $downloads_url\n\nOn RHEL systems installs all RPMs\n\nOn non-RHEL systems installs all Zips to $opt_base\n\nIf you get this error (eg. on Amazon Linux 2 when trying to install Oracle Client version 23):\n\n    Error: Invalid version flag: or\n\nthen install an older version by passing it an arg of a major version and letting the script figure out the rest, eg.\n\n    ${0##*/} 21\n\"\n# On Mac it ignores the version arg and always installs the latest DMGs\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nversion=\"${1:-latest}\"\n\n#if ! is_linux && ! is_mac; then\n#    die \"ERROR: only Linux and Mac are supported at this time\"\n#fi\n\nif ! is_linux; then\n    die \"ERROR: only Linux is supported at this time\"\nfi\n\ntimestamp \"Installing Oracle InstantClient\"\necho\n\nif is_linux && ! am_root; then\n    die \"You need to be root on Linux to run this script as it installs software\"\nfi\n\nif is_linux; then\n    # xargs is needed by install_packages_if_absent.sh\n    if ! type -P xargs &>/dev/null; then\n        \"$srcdir/../packages/install_packages.sh\" findutils  # contains xargs for RHEL8\n    fi\n    \"$srcdir/../packages/install_packages_if_absent.sh\" curl ca-certificates\n\n    timestamp \"Fetching downloads page: $downloads_url\"\n    downloads_page=\"$(curl \"$downloads_url\")\"\n    echo\n\n    timestamp \"Parsing available versions\"\n    versions=\"$(\n        grep -Eo \"oracle-instantclient[^'\\\"]*basic[^'\\\"]*rpm\" <<< \"$downloads_page\" |\n        sed '\n            s/^.*basic[[:alpha:]]*-//;\n            s/-.*$//; s/.rpm$//;\n            s/linuxx64//;\n            /^[[:space:]]*$/d\n        ' |\n        sort -Vur\n        #die \"ERROR: No versions parsed from downloads url: $downloads_url\"\n    )\"\n    echo\n\n    versions_found=\"Versions found:\n\n    $versions\n    \"\n\n    if [ \"$version\" = list ]; then\n        echo \"$versions_found\"\n        exit 1\n    fi\n\n    log \"$versions_found\"\nfi\n\nparse_rpms_version(){\n    local version=\"$1\"\n    local rhel_version\n    rhel_version=\"$(awk -F= '/^VERSION=/{print $2}' /etc/*release | sed 's/\"//g')\"\n    local rpms\n    # Oracle for some reason doesn't prefix the hrefs with 'https:' must add it ourselves\n    # XXX: SECURITY: this regex must remain as tight as possible known chars as these packages are passed to the CLI yum install command further down\n    #      and this could open you to a Shell Injection attack if you mess with this\n    #      KNOWN GOOD CHARS WHITELISTING ONLY - NO '.' sloppiness can be allowed here !!\n    rpms=\"$(\n        grep -Eo \"//download.oracle.com/[[:alnum:]/._-]+/oracle-instantclient[[:alnum:]._-]+${version}[[:alnum:]._-]+\\.rpm\" <<< \"$downloads_page\" |\n        # remove basiclite rpm since it clashes with basic rpm\n        sed '/basiclite/d' |\n        sed 's|^|https:|' ||\n        die \"ERROR: failed to parse RPM packages for version '$version' from downloads page\"\n    )\"\n    if grep -qi amazon /etc/*-release; then\n        timestamp \"Detected Amazon Linux, ignoring version\"\n        rhel_version=\"\"\n    fi\n    if is_int \"$rhel_version\"; then\n        timestamp \"Determined RHEL version to be: $rhel_version\"\n    else\n        timestamp \"Failed to determine RHEL version, will attempt to use latest RHEL RPM version available (best effort for Fedora and Amazon Linux)\"\n        rhel_version=\"$(\n            # -m 1 still outputs 2 matches from the first line :-/\n            grep -Eom1 '\\.el[[:digit:]]+\\.x86_64.rpm' <<< \"$rpms\" |\n            tr ' ' '\\n' |\n            head -n 1 |\n            sed 's/^\\.el//; s/\\..*$//' || :\n            #die \"Failed to parse latest RHEL version from available RPMs\"\n        )\"\n        # older versions of client do not have el suffixes\n        #if ! is_int \"$rhel_version\"; then\n        #    die \"ERROR: failed to parse latest RHEL version from available RPMs, got unexpected non-integer: '$rhel_version'\"\n        #fi\n    fi\n    if is_int \"$rhel_version\"; then\n        timestamp \"Parsing RPMs for RHEL $rhel_version\"\n        grep -E \"\\\\.el$rhel_version\\\\.x86_64\\\\.rpm\" <<< \"$rpms\" ||\n        die \"ERROR: failed to parse RPM packages for RHEL version $rhel_version\"\n    else\n        echo \"$rpms\"\n    fi\n}\n\nparse_zips_version(){\n    local version=\"$1\"\n    # Oracle for some reason doesn't prefix the hrefs with 'https:' must add it ourselves\n    # XXX: SECURITY: this regex must remain as tight as possible known chars as these zips are passed to the CLI unzip command further down\n    #      and this could open you to a Shell Injection attack if you mess with this\n    #      KNOWN GOOD CHARS WHITELISTING ONLY - NO '.' sloppiness can be allowed here !!\n    grep -Eo \"//download.oracle.com/[[:alnum:]/._-]+${version}[[:alnum:]._-]*\\.zip\" <<< \"$downloads_page\" |\n    # remove basiclite zip since it clashes with basic zip\n    sed '/basiclite/d' |\n    sed 's|^|https:|' ||\n    die \"ERROR: failed to parse ZIP packages for version '$version' from downloads page\"\n}\n\nif is_mac; then\n    arch=\"$(get_arch)\"\n    dmgs=\"\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-basic-macos-$arch.dmg\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-sqlplus-macos-$arch.dmg\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-tools-macos-$arch.dmg\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-sdk-macos-$arch.dmg\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-jdbc-macos-$arch.dmg\n        https://download.oracle.com/otn_software/mac/instantclient/instantclient-odbc-macos-$arch.dmg\n    \"\nelif [ \"$version\" = latest ]; then\n    timestamp \"Using version 'latest'\"\n    version=\"$(head -n1 <<< \"$versions\")\"\n    timestamp \"Latest version determined to be: $version\"\n    # use permalink to latest version\n    rpms=\"\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-basic-linuxx64.rpm\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-sqlplus-linuxx64.rpm\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-tools-linuxx64.rpm\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-devel-linuxx64.rpm\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-jdbc-linuxx64.rpm\n        https://download.oracle.com/otn_software/linux/instantclient/oracle-instantclient-odbc-linuxx64.rpm\n    \"\n    zips=\"\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-basic-linuxx64.zip\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-sqlplus-linuxx64.zip\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-tools-linuxx64.zip\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-sdk-linuxx64.zip\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-jdbc-linuxx64.zip\n        https://download.oracle.com/otn_software/linux/instantclient/instantclient-odbc-linuxx64.zip\n    \"\nelse\n    if ! grep -Fq \"$version\" <<< \"$versions\"; then\n        die \"ERROR: requested version '$version' does not match list of available versions:\n\n$versions\n\"\n    fi\n    version=\"${version%%.}\"\n    version=\"$(\n        grep -Eo -m 1 \"^${version}(\\.[[:digit:].]+$)*$\" <<< \"$versions\" ||\n        die \"ERROR: Failed to find version prefixed with: $version\"\n    )\"\n    timestamp \"Requested version matched available version: $version\"\nfi\n\nif is_mac; then\n    timestamp \"Installing DMGs on Mac\"\n    cd ~/Downloads\n    for dmg in $dmgs; do\n        timestamp \"Fetching DMG: $dmg\"\n        wget -c \"$dmg\"\n        open \"${dmg##*/}\"\n    done\nelif type -P yum &>/dev/null; then\n    if [ -z \"${rpms:-}\" ]; then\n        # we did not set the permalink rpms because script wasn't passed latest\n        timestamp \"On RHEL-based system, determining RPM package to install\"\n        rpms=\"$(parse_rpms_version \"$version\")\"\n    fi\n    echo\n    timestamp \"Installing RPMs\"\n    # want splitting\n    # shellcheck disable=SC2086\n    yum install -y $rpms\nelse\n    # libaio shared library is needed by sqlplus\n    \"$srcdir/../packages/install_packages_if_absent.sh\" wget unzip\n    if grep -q Ubuntu /etc/*-release; then\n        \"$srcdir/../packages/install_packages_if_absent.sh\" libaio-dev\n    elif grep -q Debian /etc/*-release; then\n        \"$srcdir/../packages/install_packages_if_absent.sh\" libaio1\n    elif grep -qi redhat /etc/*-release; then\n        \"$srcdir/../packages/install_packages_if_absent.sh\" libaio\n    fi\n    if [ -z \"${zips:-}\" ]; then\n        # we did not set the permalink zips because script wasn't passed latest\n        timestamp \"On non-RHEL-based system, falling back to using zips\"\n        zips=\"$(parse_zips_version \"$version\")\"\n    fi\n    timestamp \"Installing zips\"\n    mkdir -p -v \"$opt_base\"\n    cd \"$opt_base\"\n    # want splitting\n    # shellcheck disable=SC2086\n    for zip in $zips; do\n        wget -c \"$zip\"\n        # don't overwrite existing files for safety to not risk breaking an existing installation\n        unzip -n \"${zip##*/}\"\n    done\n    echo\n    timestamp \"Linking newest instantclient_* installation to $PWD/instantclient to make a more stable path for LD_LIBRARY_PATH\"\n    # want newer one linked, easier than using find for this\n    # shellcheck disable=SC2012\n    ln -sfv \"$(ls -dt instantclient_* | head -n1)\" instantclient\n    echo\n    if [ -f /usr/lib/x86_64-linux-gnu/libaio.so.1t64 ]; then\n        if ! [ -f /usr/lib/x86_64-linux-gnu/libaio.so.1 ]; then\n            # Ubuntu doesn't work without this\n            timestamp \"Linking /usr/lib/x86_64-linux-gnu/libaio.so.1t64 to /usr/lib/x86_64-linux-gnu/libaio.so.1 because sqlplus looks for libaio.so.1 and fails otherwise\"\n            ln -sv /usr/lib/x86_64-linux-gnu/libaio.so.1t64 /usr/lib/x86_64-linux-gnu/libaio.so.1\n        fi\n        echo\n    fi\n    echo\n    echo \"IMPORTANT: you will need to set this in your environment for Oracle programs like sqlplus to know where to find its own shared libraries\"\n    echo\n    echo \"  export LD_LIBRARY_PATH=/opt/oracle/instantclient\"\nfi\n\necho\ntimestamp \"Oracle Client installed successfully\"\n"
  },
  {
    "path": "install/install_oracle_sql_developer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 07:19:37 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Oracle SQL Developer IDE\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nversion=\"${1:-23.1.1.345.2114}\"\n\nexport OS_DARWIN=macos\nexport ARCH_ARM64=aarch64\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\n\nif type -P yum &>/dev/null; then\n    timestamp \"Installing SQL Developer on RHEL based system\"\n    yum install -y \"https://download.oracle.com/otn_software/java/sqldeveloper/sqldeveloper-$version.noarch.rpm\"\nelif is_mac; then\n    timestamp \"Installing SQL Developer on Mac\"\n    url=\"https://download.oracle.com/otn_software/java/sqldeveloper/sqldeveloper-$version-$os-$arch.app.zip\"\n    zip=\"${url##*/}\"\n    timestamp \"Downloading: $url\"\n    wget -c \"$url\"\n    echo\n    timestamp \"Unzipping: $zip\"\n    unzip -o \"$zip\"\n    echo\n    sql_developer=\"SQLDeveloper.app\"\n    if ! [ -d \"$sql_developer\" ]; then\n        die \"Failed to find expected extracted directory: $sql_developer\"\n    fi\n    timestamp \"Moving $sql_developer to /Applications\"\n    echo\n    if [ -d \"/Applications/$sql_developer\" ]; then\n        timestamp \"Removing old SQL Developer installation\"\n        rm -fr \"/Applications/$sql_developer\"\n        echo\n    fi\n    mv -fv \"$sql_developer\" /Applications\n    echo\n    timestamp \"Opening $sql_developer\"\n    open -a \"$sql_developer\"\nelse\n    die \"Unsupport OS - not Mac or RHEL based\"\nfi\n"
  },
  {
    "path": "install/install_oracle_sqlcl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-13 00:59:42 +0300 (Sun, 13 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads the latest Oracle SQLcl command line client to /usr/local/ and links it to /usr/local/bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ntimestamp \"Installing Oracle SQLcl command line client\"\necho\n\n# sometimes on Mac /usr/local is writeable by the user, in which case don't enforce sudo\nif ! [ -w /usr/local ] || ! [ -w /usr/local/bin ]; then\n#if ! am_root; then\n    die \"ERROR: must be root to run this script as it will download and unpack to /usr/local\"\nfi\n\n# permalink to latest release\ndownload_url=\"https://download.oracle.com/otn_software/java/sqldeveloper/sqlcl-latest.zip\"\ninstall_base=\"/usr/local\"\n\ntimestamp \"Downloading: $download_url\"\nwget -c \"$download_url\"\necho\n\n# unsure the files are created as rwxr-xr-x octal permissions otherwise users will get this error trying to run the sql / sqlcl wrapper:\n#\n#   Error: Could not find or load main class oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#   Caused by: java.lang.ClassNotFoundException: oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#\n# no, it's not this, the stupid zip is unpacking the files with 0640 permissions\n#umask 0022\n\ntimestamp \"Unzipping to $install_base/\"\nunzip -n sqlcl-latest.zip -d \"$install_base/\"\necho\n\ntimestamp \"Fixing library permissions\"\n# don't print this in case it scares users\n#timestamp \"Fixing stupid default 0640 permissions on '$install_base/sqlcl/lib/*' to avoid this error:\n#\n#Error: Could not find or load main class oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#Caused by: java.lang.ClassNotFoundException: oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#\"\nchmod -R o+r \"$install_base/sqlcl/lib\"\necho\n\n# clashes with GNU parallel which installs an 'sql' program in the path so link this to sqlcl to avoid path priority clashes\n#if ! [ -e /usr/local/bin/sql ]; then\n#    timestamp \"Linking $install_base/sqlcl/bin/sql to /usr/local/bin/ for \\$PATH convenience\"\n#    ln -sv \"$install_base/sqlcl/bin/sql\" /usr/local/bin/\n#    echo\n#fi\n\n# Symlinking sql to /usr/local/bin doesn't work as the script naively so not follow it's own symlink refernce\n# to find its adjacent libraries, resuling in this error:\n#\n#   Error: Could not find or load main class oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#   Caused by: java.lang.ClassNotFoundException: oracle.dbtools.raptor.scriptrunner.cmdline.SqlCli\n#\n#if ! [ -e /usr/local/bin/sqlcl ]; then\n#    timestamp \"Linking $install_base/sqlcl/bin/sql to /usr/local/bin/sqlcl for \\$PATH convenience\"\n#    ln -sv \"$install_base/sqlcl/bin/sql\" /usr/local/bin/sqlcl\n#    echo\n#fi\n\n# Instead, create a stub script instead\n\nstub_script=\"/usr/local/bin/sqlcl\"\nif ! [ -e \"$stub_script\" ]; then\n    timestamp \"Creating stub script for \\$PATH convenience: $stub_script\"\n    cat > \"$stub_script\" <<EOF\n#!/usr/bin/env bash\nset -euo pipefail\n#cd \"$install_base/sqlcl/bin\"\n#./sql \"\\$@\"\n\"$install_base/sqlcl/bin/sql\" \"\\$@\"\nEOF\n    chmod +x \"$stub_script\"\n    echo\nfi\n\ntimestamp \"Completed installation of SQLcl oracle client\"\necho\n#timestamp \"Don't forget to add /usr/local/sqlcl/bin to your \\$PATH and check for clashes with other programs called 'sql' in your path (GNU Parallels puts one in /usr/local/bin/ for example)\"\ntimestamp \"Call SQLcl as 'sqlcl' which should be in your \\$PATH now\"\necho\necho \"SQLcl version:\"\nsqlcl -version\n"
  },
  {
    "path": "install/install_packer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-28 16:27:26 +0100 (Sun, 28 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Hashicorp Packer\n\nHomebrew no longer has recent Packer versions past 1.9.4 due to a change in license to be non-free\n\nSo this downloads newer versions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.8.7}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"hashicorp/packer\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    version=\"${version#v}\"  # https://releases.hashicorp.com doesn't use v prefix but github does\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\n# gives just version number\n#export RUN_VERSION_OPT=1\n# prefixes with Packer\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://releases.hashicorp.com/packer/$version/packer_${version}_{os}_{arch}.zip\" packer\n"
  },
  {
    "path": "install/install_parquet-tools.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-17\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Parquet Tools to local ~/bin\n#\n# add ~/bin/parquet-tools-* to $PATH (automatically detected and done via advanced bashrc in this repo)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nPARQUET_VERSION=\"${PARQUET_VERSION:-1.5.0}\"\n\nzipfile=\"parquet-tools-$PARQUET_VERSION-bin.zip\"\nzipdir=\"${zipfile%-bin.zip}\"\n\nURL=\"${URL:-http://search.maven.org/remotecontent?filepath=com/twitter/parquet-tools/$PARQUET_VERSION/$zipfile}\"\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nbin=\"${BIN:-$HOME/bin}\"\n\nmkdir -pv \"$bin\"\n\ncd \"$bin\"\n\nif type -P parquet-cat &>/dev/null; then\n    echo \"parquet-tools already found in \\$PATH:\"\n    echo\n    dirname \"$(which parquet-cat)\"\nelif [ -f \"$zipdir/parquet-cat\" ]; then\n    echo \"parquet-tools already installed in local dir ($PWD/$zipdir)\"\nelse\n    echo \"parquet-tools not found in \\$PATH, nor in the local ${zipfile%.zip} directory\"\n    echo\n    echo \"downloading parquet-tools to $bin\"\n    wget -t 100 --retry-connrefused -c -O \"$zipfile\" \"$URL\"\n    echo\n    echo \"unzipping parquet-tools\"\n    unzip -- \"$zipfile\"\n    echo\n    echo \"chmod'ing 0755 parquet-tools-*\"\n    chmod 0755 parquet-tools-*\n    echo\n    echo \"removing zipfile\"\n    rm -f -- \"$zipfile\"\n    echo\n    echo \"Done\"\nfi\n\necho\necho \"Ensure $bin/${zipfile%.zip} is in the \\$PATH (it's auto-detected in new shells if sourcing this repo's .bashrc)\"\necho\n"
  },
  {
    "path": "install/install_pluto.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Fairwinds Pluto\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-5.6.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" FairwindsOps/pluto 'pluto_{version}_{os}_{arch}.tar.gz' \"$version\" pluto\n"
  },
  {
    "path": "install/install_polaris.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Fairwinds Polaris\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-5.1.0}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" FairwindsOps/polaris 'polaris_{os}_{arch}.tar.gz' \"$version\" polaris\n"
  },
  {
    "path": "install/install_powershell.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 22:38:36 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs PowerShell on Mac and various Linux distros\n#\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\nif type -P pwsh &>/dev/null; then\n    echo \"PowerShell is already installed\"\n    exit 0\nfi\n\nif is_mac; then\n    brew cask install powershell\nelif is_linux; then\n    # Ubuntu 16.04 and 18.04 - 18.10 onwards use Snap, will require updates\n    # 16.04 works, 18.04 has broken package dependency\n    if grep -q '^DISTRIB_ID=Ubuntu' /etc/*release; then\n        \"$srcdir/install_powershell_ubuntu.sh\"\n#        if grep -E '^DISTRIB_RELEASE=(19|18\\.10)' /etc/*release; then\n#            # assigned in lib\n#            # shellcheck disable=SC2154\n#            $sudo snap install powershell --classic\n#        else\n#            version=\"$(awk -F= '/^DISTRIB_RELEASE=/{print $2}' /etc/*release)\"\n#            $sudo apt-get update\n#            $sudo apt-get install -y wget apt-transport-https\n#            wget -q \"https://packages.microsoft.com/config/ubuntu/$version/packages-microsoft-prod.deb\"\n#            $sudo dpkg -i packages-microsoft-prod.deb\n#            $sudo apt-get update\n#            $sudo apt-get install -y powershell\n#        fi\n    # works on Debian 8 & 9\n    # Debian 10 not supported yet, only in Preview\n    # https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7#debian-10\n    elif grep -q '^ID=debian' /etc/*release; then\n        \"$srcdir/install_powershell_debian.sh\"\n#        # works in Stretch but not in Jessie\n#        #codename=\"$(awk -F= '/^VERSION_CODENAME=/{print $2}' /etc/*release)\"\n#        codename=\"$(grep -Eo '^VERSION=\"[[:digit:]]* \\(.+\\)\"' /etc/*release | sed 's/.*(//; s/)//; s/\"//g')\"\n#        $sudo apt-get update\n#        $sudo apt-get install -y curl gnupg apt-transport-https\n#        curl https://packages.microsoft.com/keys/microsoft.asc | $sudo apt-key add -\n#        $sudo sh -c \"echo 'deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-debian-$codename-prod $codename main' > /etc/apt/sources.list.d/microsoft.list\"\n#        $sudo apt-get update\n#        $sudo apt-get install -y powershell\n    elif grep -qi 'redhat' /etc/*release; then\n        \"$srcdir/install_powershell_rhel.sh\"\n#        version=\"$(awk -F= '/^VERSION_ID=/{print $2}' /etc/*release)\"\n#        version=\"${version//\\\"/}\"\n#        if [ \"$version\" != 7 ]; then\n#            echo \"Unsupported RHEL/CentOS version\"\n#            exit 1\n#        fi\n#        curl \"https://packages.microsoft.com/config/rhel/$version/prod.repo\" | $sudo tee /etc/yum.repos.d/microsoft.repo\n#        $sudo yum install -y powershell\n    else\n        echo \"Unsupported Linux distribution\"\n        exit 1\n    fi\nelse\n    echo \"Unsupported OS\"\n    exit 1\nfi\n"
  },
  {
    "path": "install/install_powershell_debian.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 22:38:36 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs PowerShell on Debian\n#\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\nif type -P pwsh &>/dev/null; then\n    echo \"PowerShell is already installed\"\n    exit 0\nfi\n\n# works on Debian 8 & 9\n# Debian 10 not supported yet, only in Preview\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux?view=powershell-7#debian-10\nif ! grep -q '^ID=debian' /etc/*release; then\n    echo \"Not Debian\"\n    exit 1\nfi\n\n# works in Stretch but not in Jessie\n#codename=\"$(awk -F= '/^VERSION_CODENAME=/{print $2}' /etc/*release)\"\ncodename=\"$(grep -Eo '^VERSION=\"[[:digit:]]* \\(.+\\)\"' /etc/*release | sed 's/.*(//; s/)//; s/\"//g')\"\nopts=\"-o DPkg::Lock::Timeout=1200\"\n$sudo apt-get update $opts\n$sudo apt-get install -y $opts curl gnupg apt-transport-https\ncurl https://packages.microsoft.com/keys/microsoft.asc | $sudo apt-key add -\n$sudo sh -c \"echo 'deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-debian-$codename-prod $codename main' > /etc/apt/sources.list.d/microsoft.list\"\n$sudo apt-get update $opts\n$sudo apt-get install -y $opts powershell\n"
  },
  {
    "path": "install/install_powershell_rhel.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 22:38:36 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs PowerShell on Ubuntu\n#\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\nif type -P pwsh &>/dev/null; then\n    echo \"PowerShell is already installed\"\n    exit 0\nfi\n\nif ! grep -qi 'redhat' /etc/*release; then\n    echo \"Not RHEL / CentOS\"\n    exit 1\nfi\n\nversion=\"$(awk -F= '/^VERSION_ID=/{print $2}' /etc/*release)\"\nversion=\"${version//\\\"/}\"\n\nif [ \"$version\" != 7 ]; then\n    echo \"Unsupported RHEL/CentOS version\"\n    exit 1\nfi\n\ncurl \"https://packages.microsoft.com/config/rhel/$version/prod.repo\" | $sudo tee /etc/yum.repos.d/microsoft.repo\n\n$sudo yum install -y powershell\n"
  },
  {
    "path": "install/install_powershell_ubuntu.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 22:38:36 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs PowerShell on Ubuntu\n#\n# https://docs.microsoft.com/en-us/powershell/scripting/install/installing-powershell-core-on-linux\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\nif type -P pwsh &>/dev/null; then\n    echo \"PowerShell is already installed\"\n    exit 0\nfi\n\nif ! grep -q '^DISTRIB_ID=Ubuntu' /etc/*release; then\n    echo \"Not Ubuntu\"\n    exit 1\nfi\n\n# Ubuntu 16.04 and 18.04 - 18.10 onwards use Snap, will require updates\n# 16.04 works, 18.04 has broken package dependency\nif grep -E '^DISTRIB_RELEASE=(19|18\\.10)' /etc/*release; then\n    # assigned in lib\n    # shellcheck disable=SC2154\n    $sudo snap install powershell --classic\nelse\n    version=\"$(awk -F= '/^DISTRIB_RELEASE=/{print $2}' /etc/*release)\"\n    opts=\"-o DPkg::Lock::Timeout=1200\"\n    $sudo apt-get update $opts\n    $sudo apt-get install -y $opts wget apt-transport-https\n    wget -q \"https://packages.microsoft.com/config/ubuntu/$version/packages-microsoft-prod.deb\"\n    $sudo dpkg -i packages-microsoft-prod.deb\n    $sudo apt-get update $opts\n    $sudo apt-get install -y $opts powershell\nfi\n"
  },
  {
    "path": "install/install_prometheus.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version> <component_name>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.54.1}\"\nversion=\"${1:-latest}\"\nname=\"${2:-prometheus}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" prometheus/\"$name\" \"$name-{version}.{os}-{arch}.tar.gz\" \"$version\" \"$name-{version}.{os}-{arch}/$name\"\n"
  },
  {
    "path": "install/install_prometheus_alertmanager.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Alert Manager\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.27.0}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" alertmanager\n"
  },
  {
    "path": "install/install_prometheus_blackbox_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Blackbox Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.25.0}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" blackbox_exporter\n"
  },
  {
    "path": "install/install_prometheus_consul_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Consul Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.12.1}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" consul_exporter\n"
  },
  {
    "path": "install/install_prometheus_graphite_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Graphite Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.15.2}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" graphite_exporter\n"
  },
  {
    "path": "install/install_prometheus_memcached_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Memcached Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.14.4}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" memcached_exporter\n"
  },
  {
    "path": "install/install_prometheus_mysqld_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus MySQLd Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.15.1}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" mysqld_exporter\n"
  },
  {
    "path": "install/install_prometheus_node_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Node Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.8.2}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" node_exporter\n"
  },
  {
    "path": "install/install_prometheus_push_gateway.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Push Gateway\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-1.10.0}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" pushgateway\n"
  },
  {
    "path": "install/install_prometheus_statsd_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 02:32:00 +0300 (Tue, 08 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Prometheus Statsd Exporter\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.27.2}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" statsd_exporter\n"
  },
  {
    "path": "install/install_promlens.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-07 23:06:24 +0300 (Mon, 07 Oct 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Promlens - the Prometheus web-based PromQL query builder, analyzer, and visualizer\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.3.0}\"\nversion=\"${1:-latest}\"\n\n\"$srcdir/install_prometheus.sh\" \"$version\" promlens\n"
  },
  {
    "path": "install/install_pulumi_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-21 09:46:37 +0100 (Thu, 21 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# installs to ~/.pulumi/bin/\ncurl -fsSL https://get.pulumi.com | sh\necho\n~/.pulumi/bin/pulumi version\n"
  },
  {
    "path": "install/install_rancher_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-05-01 23:31:03 +0400 (Wed, 01 May 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Rancher CLI\n\nAs of version 2.8.3 there is no macOS Arm binary available for download, so downloads the amd64 binary which works\n\nOnce installed, configure authentication by creating a personal Access Key and Secret Key here:\n\n    https://\\$RANCHER_HOST:\\$RANCHER_PORT/dashboard/account\n\nand then authenticating using them like so:\n\n    rancher login \\\"https://\\$RANCHER_HOST\\\" --token \\\"\\$RANCHER_ACCESS_KEY:\\$RANCHER_SECRET_KEY\\\"\n\nSee Rancher knowledge base page for CLI usage info and examples:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/rancher.md\n\nFor Rancher itself, see Kubernetes configs with Kustomize and Helm in the repo:\n\n    https://github.com/HariSekhon/Kubernetes-configs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.8.3}\"\nversion=\"${1:-latest}\"\n\nexport ARCH_X86_64=amd64\nexport ARCH_ARM64=arm  # not present so override back to amd64 below and use rosetta\nexport ARCH_OVERRIDE=amd64\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" rancher/cli \"rancher-{os}-{arch}-v{version}.tar.gz\" \"$version\" \"rancher-v{version}/rancher\"\n"
  },
  {
    "path": "install/install_rpmforge.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2012-06-25 15:20:39 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ \"${NO_FAIL:-}\" ]; then\n    set +eo pipefail\nfi\n\nif grep -qi \"NAME=Fedora\" /etc/*release; then\n    echo \"Detected Fedora, skipping rpmforge install...\"\n    exit 0\nfi\n\nif rpm -q rpmforge-release; then\n    echo \"rpmforge-release rpm is already installed, skipping...\"\n    #exit 0\nfi\n\nif yum repolist | grep -qi '\\<rpmforge\\>'; then\n    echo \"rpmforge yum repo already detected in yum repolist, skipping...\"\n    #exit 0\nfi\n\n[ $EUID -eq 0 ] && sudo=\"\" || sudo=\"\"\n\nrpm -qi wget || yum install -y wget\n\nmajor_release=\"$(grep -ho '[[:digit:]]' /etc/*release | head -n1)\"\narch=\"$(uname -m)\"\n\nrpm_url=\"$(\n    curl -sS http://repoforge.org/use/ |\n    grep -Eo \"http://repository.it4i.cz/mirrors/repoforge/redhat/el$major_release/en/$arch/.*\\\\.$arch\\\\.rpm\"\n)\"\n\nwget -t 5 --retry-connrefused -O /tmp/repoforge.rpm \"$rpm_url\"\n$sudo rpm -ivh /tmp/repoforge.rpm\nrm -f -- /tmp/repoforge.rpm\n"
  },
  {
    "path": "install/install_rvm.sh",
    "content": "#!/bin/sh\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-04 16:36:18 +0100 (Fri, 04 Oct 2019)\n#        (circa 2016 originally)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Ruby RVM\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif type apk >/dev/null 2>&1; then\n    apk --no-cache add bash curl procps\nelif type apt-get >/dev/null 2>&1; then\n    opts=\"-o DPkg::Lock::Timeout=1200\"\n    apt-get update $opts\n    apt-get install -y $opts curl procps\nelif type yum >/dev/null 2>&1; then\n    echo \"rhel based systems aleady have curl\"\nfi\n\nexec bash <<EOF\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ngpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3 7D2BAF1CF37B13E2069D6956105BD0E739499BDB\n\ncurl -sSL https://get.rvm.io | bash -s stable --rails\n\nEOF\n"
  },
  {
    "path": "install/install_sbt.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-08-01 10:17:55 +0100 (Mon, 01 Aug 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"================================================================================\"\necho \"                              S B T   I n s t a l l\"\necho \"================================================================================\"\n\nSBT_VERSION=${1:-${SBT_VERSION:-1.6.1}}\n\nam_root(){\n    # shellcheck disable=SC2039\n    [ \"${EUID:-${UID:-$(id -n)}}\" = 0 ]\n}\nif am_root; then\n    BASE=/opt\n    sudo=\"\"\nelse\n    BASE=~/bin\n    sudo=sudo\nfi\n\necho\ndate '+%F %T  Starting...'\nstart_time=\"$(date +%s)\"\necho\n\nif command -v yum 2>/dev/null; then\n    curl -sSL https://www.scala-sbt.org/sbt-rpm.repo |\n        $sudo tee /etc/yum.repos.d/sbt-rpm.repo\n    $sudo yum install -y java-sdk\n    $sudo yum install -y --nogpgcheck sbt\nelif command -v apt-get 2>/dev/null; then\n    opts=\"-o DPkg::Lock::Timeout=1200\"\n    $sudo apt-get update $opts\n    openjdk=\"$(apt-cache search openjdk | grep -Eo 'openjdk-[[:digit:]]+-jdk' | head -n1)\"\n    $sudo apt-get install -y $opts \"$openjdk\" scala gnupg2\n    echo \"deb https://repo.scala-sbt.org/scalasbt/debian all main\" |\n        $sudo tee /etc/apt/sources.list.d/sbt.list\n    echo \"deb https://repo.scala-sbt.org/scalasbt/debian /\" |\n        $sudo tee /etc/apt/sources.list.d/sbt_old.list\n    $sudo apt-get install -y $opts apt-transport-https curl gnupg\n    curl -sL \"https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823\" |\n        ${sudo:+$sudo -H} gpg --no-default-keyring --keyring gnupg-ring:/etc/apt/trusted.gpg.d/scalasbt-release.gpg --import\n    $sudo chmod 644 /etc/apt/trusted.gpg.d/scalasbt-release.gpg\n    $sudo apt-get update $opts\n    $sudo apt-get install -y $opts sbt\nelse\n    echo \"No mainstream package managers detected, doing tarball install\"\n    if ! [ -e \"$BASE/sbt\" ]; then\n        mkdir -p \"$BASE\"\n        cd \"$BASE\"\n        wget -t 10 --retry-connrefused \"https://github.com/sbt/sbt/releases/download/v$SBT_VERSION/sbt-$SBT_VERSION.tgz\" && \\\n        tar zxvf \"sbt-$SBT_VERSION.tgz\" && \\\n        rm -f -- \"sbt-$SBT_VERSION.tgz\"\n        echo\n        echo \"SBT Install done\"\n    else\n        echo \"$BASE/sbt already exists - doing nothing\"\n    fi\n    if am_root; then\n        if ! [ -e /etc/profile.d/sbt.sh ]; then\n            echo \"Adding /etc/profile.d/sbt.sh\"\n            # shell execution tracing comes out in the file otherwise\n            set +x\n            cat >> /etc/profile.d/sbt.sh <<EOF\nexport SBT_HOME=/opt/sbt\nexport PATH=\\$PATH:\\$SBT_HOME/bin\nEOF\n        fi\n    else\n        echo \"Ensure you have ~/bin/sbt/bin set in your \\$PATH\"\n    fi\nfi\n\necho\ndate '+%F %T  Finished'\necho\nend_time=\"$(date +%s)\"\ntime_taken=\"$((end_time - start_time))\"\necho \"Completed in $time_taken secs\"\necho\necho \"==================================================\"\necho \"              SBT Install Completed\"\necho \"==================================================\"\necho\n"
  },
  {
    "path": "install/install_sdkman.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-04 16:36:18 +0100 (Fri, 04 Oct 2019)\n#        (circa 2016 originally)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs SDKMan\n\n# https://sdkman.io/install\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ncurl -sS \"https://get.sdkman.io\" | bash\n"
  },
  {
    "path": "install/install_sdkman_all_sdks.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-08 16:48:17 +0100 (Tue, 08 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs SDKman's most commonly used build tools e.g Java, Scala, Groovy + Maven, SBT, Gradle\n#\n# you may need to run ./install_sdkman.sh first\n\nset -eo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nsdks=\"\njava\nscala\ngroovy\nmaven\ngradle\nsbt\n\"\n\nif ! type sdk &>/dev/null; then\n    \"$srcdir/install_sdkman.sh\"\nfi\n\nif [ -s ~/.sdkman/bin/sdkman-init.sh ]; then\n    # shellcheck disable=SC1090,SC1091\n    . ~/.sdkman/bin/sdkman-init.sh\nfi\n\nfor x in $sdks; do\n    set +o pipefail\n    yes | sdk install \"$x\"\ndone\n"
  },
  {
    "path": "install/install_semaphore_ci.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-15 13:09:19 +0100 (Wed, 15 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\ncheck_env_defined SEMAPHORE_CI_TOKEN\ncheck_env_defined SEMAPHORE_CI_ORGANIZATION\n\ncurl https://storage.googleapis.com/sem-cli-releases/get.sh | bash\n\nsem connect \"${SEMAPHORE_CI_ORGANIZATION}\".semaphoreci.com \"$SEMAPHORE_CI_TOKEN\"\n\necho\necho \"Done\"\necho\necho \"Semaphore CI CLI installed\"\necho\nsem version\necho\necho \"Run 'sem help' for options\"\n"
  },
  {
    "path": "install/install_serverless.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-21 10:46:30 +0100 (Wed, 21 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls serverless.com binary to ~/.serverless/bin/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nif [ -n \"${FORCE_INSTALL:-}\" ] || ! type -P serverless &>/dev/null; then\n    curl -o- -L https://slss.io/install | bash\nelse\n    echo \"serverless is already installed. To upgrade run 'serverless upgrade'\"\nfi\n\n# configure by first run of:\n#\n#   serverless\n#\n# uninstall via:\n#\n#   serverless uninstall\n\necho\necho \"Serverless Version:\"\necho\n\nserverless --version\n"
  },
  {
    "path": "install/install_spotifycontrol.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-17 11:28:16 +0000 (Tue, 17 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Update: using Shpotify now which is installed via adjacent brew packages list\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls SpotifyControl CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/../packages/install_binary.sh\" \"https://raw.githubusercontent.com/dronir/SpotifyControl/master/SpotifyControl\"\n"
  },
  {
    "path": "install/install_squirrel_sql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-13 15:40:23 +0300 (Sun, 13 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nconfig=\"$srcdir/../setup/squirrelsql-install-options.xml\"\nconfig=\"$(readlink -f \"$config\")\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstall SQuirreL SQL Client\n\nAllows you to install with a custom config:\n\n    $config\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nversion=\"${1:-latest}\"\n\nowner_repo=\"squirrel-sql-client/squirrel-sql-stable-releases\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    version=\"${version%%-*}\"\n    timestamp \"latest version is '$version'\"\nfi\nis_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\n\nexport RUN_VERSION_ARG=1\n\nif is_mac; then\n    os=\"MACOSX\"\nelse\n    os=\"standard\"\nfi\n\ncd /tmp\n\ndownload_url=\"https://github.com/squirrel-sql-client/squirrel-sql-stable-releases/releases/download/$version-installer/squirrel-sql-$version-$os-install.jar\"\n\n\"$srcdir/../bin/download_url_file.sh\" \"$download_url\"\n\ninstall_jar=\"${download_url##*/}\"\n\njava -jar \"./$install_jar\" -options-system \"$config\"  # file with settings or where to install from the homebrew\n\nif is_mac; then\n    timestamp \"Launching SQuirreL\"\n    #open -a SQuirreLSQL\n    # this is where is installs to\n    open -a /Applications/SQuirreLSQL.app\nfi\n"
  },
  {
    "path": "install/install_syft.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Syft\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"${1:-0.35.0}\"\nversion=\"${1:-latest}\"\n\nowner_repo=\"anchore/syft\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\n    [[ \"$version\" =~ ^v ]] || version=\"v$version\"\nfi\n\ncurl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh |\nsh -s -- -b ~/bin \"$version\"\n"
  },
  {
    "path": "install/install_talosctl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-03-06 13:21:08 +0000 (Mon, 06 Mar 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.talos.dev/v1.3/introduction/quickstart/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Talos 'talosctl' client binary\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ncurl -sSL https://talos.dev/install | sh\n"
  },
  {
    "path": "install/install_terraform.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019/09/20\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Terraform\n\nCan optionally specify an exact version to install instead of latest (auto-determines latest release)\n\"\n\n#If the 'terraform' binary is already found on \\$PATH, aborts for safety as Terraform version upgrades affect the state file\n#Set UPDATE_TERRAFORM=1 in the environment to upgrade the Terraform version\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-${TERRAFORM_VERSION:-${VERSION:-0.12.29}}}\"\n#version=\"${1:-${TERRAFORM_VERSION:-${VERSION:-0.14.5}}}\"\n#version=\"${1:-${TERRAFORM_VERSION:-${VERSION:-1.1.2}}}\"\nversion=\"${1:-${TERRAFORM_VERSION:-latest}}\"\n\nowner_repo=\"hashicorp/terraform\"\n\nif [ \"$version\" = latest ]; then\n    timestamp \"determining latest version of '$owner_repo' via GitHub API\"\n    version=\"$(\"$srcdir/../github/github_repo_latest_release.sh\" \"$owner_repo\")\"\n    version=\"${version#v}\"\n    timestamp \"latest version is '$version'\"\nelse\n    is_semver \"$version\" || die \"non-semver version argument given: '$version' - should be in format: N.N.N\"\nfi\n\n#if [ -z \"${UPDATE_TERRAFORM:-}\" ]; then\n#    # command -v catches aliases, not suitable\n#    # shellcheck disable=SC2230\n#    if type -P \"$binary\" &>/dev/null; then\n#        echo \"Terraform binary '$binary' is already installed and available in \\$PATH\"\n#        echo\n#        echo \"To add or overwrite regardless, set the below variable and then re-run this script:\"\n#        echo\n#        echo \"export UPDATE_TERRAFORM=1\"\n#        exit 0\n#    fi\n#fi\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://releases.hashicorp.com/terraform/$version/terraform_${version}_{os}_{arch}.zip\" terraform\n"
  },
  {
    "path": "install/install_terraformer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 17:42:56 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Grype\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version> <specific>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.8.18}\"\nversion=\"${1:-latest}\"\nasset=\"${2:-all}\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../github/github_install_binary.sh\" GoogleCloudPlatform/terraformer \"terraformer-$asset-{os}-{arch}\" \"$version\" terraformer\n"
  },
  {
    "path": "install/install_terragrunt.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-07 19:08:50 +0100 (Thu, 07 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Terragrunt on Mac / Linux\n#\n# If running as root, installs to /usr/local/bin\n#\n# If running as non-root, installs to $HOME/bin\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Terragrunt\n\nCan optionally specify an exact version to install instead of latest (auto-determines latest release)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-${TERRAGRUNT_VERSION:-${VERSION:-0.38.4}}}\"\nversion=\"${1:-${TERRAGRUNT_VERSION:-latest}}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" gruntwork-io/terragrunt 'terragrunt_{os}_{arch}' \"$version\" terragrunt\n\n#echo\n#terragrunt --install-autocomplete\n"
  },
  {
    "path": "install/install_tfenv.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 12:44:50 +0700 (Wed, 08 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls tfenv for managing Terraform versions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"$HOME/.tfenv/bin:$PATH\"\n\nif is_mac; then\n    brew install tfenv\nelse\n    if [ -d ~/.tfenv ]; then\n        cd ~/.tfenv\n        timestamp \"Git pulling in existing ~/.tfenv dir\"\n        echo\n        git pull\n        echo\n    else\n        timestamp \"Git cloning the tfenv github repo to ~/.tfenv\"\n        echo >&2\n        git clone --depth=1 https://github.com/tfutils/tfenv.git ~/.tfenv\n    fi\n    echo\n    echo \"Don't forget to add ~/.tfenv/bin to your \\$PATH in your ~/.bashrc or similar\"\n    echo\n    echo \"(already done if sourcing DevOps-Bash-tools repo's .bashrc using .bash.d/terraform.sh)\"\n    echo\nfi\necho\n\necho -n \"tfenv version: \"\ntfenv version-name\necho\n"
  },
  {
    "path": "install/install_tfsec.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-06 15:09:39 +0000 (Thu, 06 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls TFSec CLI by AquaSec\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.63.1}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" aquasecurity/tfsec 'tfsec-{os}-{arch}' \"$version\"\n"
  },
  {
    "path": "install/install_tgswitch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-08 12:54:27 +0700 (Wed, 08 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls tgswitch for managing Terragrunt versions\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport HOME=\"${HOME:-$(cd && pwd)}\"\n\nexport PATH=\"$HOME/bin:$PATH\"\n\n#if is_mac; then\n#    brew install warrensbox/tap/tgswitch\n#else\n    # Tries to install to /usr/local/bin/ and gets permission denied\n    #curl -L https://raw.githubusercontent.com/warrensbox/tgswitch/release/install.sh | bash\n    \"$srcdir/../github/github_install_binary.sh\" warrensbox/tgswitch \"tgswitch_{version}_{os}_{arch}.tar.gz\" \"$version\" \"tgswitch\"\n#fi\n\necho\necho -n \"Terragrunt \"\ntgswitch --version\n"
  },
  {
    "path": "install/install_tkn.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-15 19:34:06 +0100 (Fri, 15 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Tekton CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-2.4.0}\"\nversion=\"${1:-latest}\"\n\nexport OS_DARWIN=Darwin\n\nif is_mac; then\n    export ARCH_OVERRIDE=all\nfi\n\nexport RUN_VERSION_ARG=1\n\npackage=\"tkn_{version}_{os}_{arch}.tar.gz\"\n\n\"$srcdir/../github/github_install_binary.sh\" tektoncd/cli \"$package\" \"$version\" tkn\n"
  },
  {
    "path": "install/install_travis.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: Tue Sep 17 16:41:02 2019 +0100\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs the Travis CI gem to home dir, logs in and generates a $TRAVIS_TOKEN\n#\n# putting the $TRAVIS_TOKEN in your environment is useful for the travis tools available in\n#\n#  https://github.com/HariSekhon/DevOps-Python-tools\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/ci.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/os.sh\"\n\n# fix version of Travis CI since it has a dependency on SSL and we need to detect and switch the SSL version for certain Mac environments, eg. Semaphore CI\nexport GEM_OPTS=\"-v 1.8.13\"\n\nif is_mac; then\n    #if ! [ -f /usr/local/opt/openssl/lib/libssl.1.0.0.dylib ]; then\n    if is_semaphore_ci; then\n        echo \"Switching OpenSSL to version 1.0.2t on Mac to avoid SSL build errors for Travis CI gem\" >&2\n        #brew switch openssl 1.0.2t\n        brew reinstall ruby\n    fi\nfi\n\n\"$srcdir/../packages/ruby_gem_install_if_absent.sh\" travis\n\n# add ruby to paths temporarily (logic borrowed from advanced bashrc code in .bash.d/paths.sh)\nfor ruby_bin in $(find ~/.gem/ruby -maxdepth 2 -name bin -type d 2>/dev/null | tac); do\n    export PATH=\"$PATH:$ruby_bin\"\ndone\n\nif ! is_CI &&\n   [ -z \"${QUICK:-}\" ] &&\n   [ -z \"${NONINTERACTIVE:-}\" ]; then\n    travis login\n    travis token\nfi\n"
  },
  {
    "path": "install/install_trivy.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-10 19:07:22 +0000 (Mon, 10 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Trivy by AquaSec\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-0.22.0}\"\nversion=\"${1:-latest}\"\n\nexport OS_DARWIN=macOS\nexport OS_LINUX=Linux\nexport ARCH_X86_64=\"64bit\"\nexport ARCH_X86=\"32bit\"\nexport ARCH_ARM64=\"ARM64\"  # Trivy packages uppercase arm\nexport ARCH_ARM=\"ARM\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" aquasecurity/trivy \"trivy_{version}_{os}-{arch}.tar.gz\" \"$version\" trivy\n"
  },
  {
    "path": "install/install_vertica_vsql_client.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-10 01:19:15 +0200 (Tue, 10 Sep 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://kubernetes.io/docs/tasks/tools/install-kubectl/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Vertica VSQL Client on Linux x86_64\n\nInstalls vqsql binary to /usr/local/bin/ or \\$HOME/bin/ depending on your permissions\n\nOffical Documentation:\n\n    https://docs.vertica.com/23.4.x/en/connecting-to/using-vsql/installing-vsql-client/\n\nDownload URLs:\n\n    https://www.vertica.com/download/vertica/client-drivers/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"24.2.0\"\nversion=\"${1:-latest}\"\n\n# Required by the vsql binary\n#\n# Invalid locale: run the \"locale\" command and check for warnings\n#export LANG=\"end_US.UTF-8\"\nexport LC_ALL=\"C.UTF-8\"\n\nlibxcrypt_package=\"\"\nif type -P rpm &>/dev/null; then\n    libxcrypt_package=\"libxcrypt-compat\"\nelif type -P apt-get &>/dev/null; then\n    libxcrypt_package=\"libxcrypt-compat\"\n    # or\n    #libxcypt_package=\"libcrypt1\"\nelse\n    timestamp \"WARNING: unknown package manager, not RPM or Apt based, not downloading the libxcrypt dependency\"\n    echo\nfi\n\nif [ -n \"$libxcrypt_package\" ]; then\n    timestamp \"Installing libxcrypt dependency\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" \"$libxcrypt_package\"\n    echo\nfi\n\ndownloads_url=\"https://www.vertica.com/download/vertica/client-drivers/\"\n\n# ERE format for grep -E\ntar_url_regex=\"https://www.vertica.com/client_drivers/[[:digit:].x-]+/[[:digit:].-]+/vertica-client-[[:digit:].-]+.x86_64.tar.gz\"\n\n# Should match these:\n#\n# https://www.vertica.com/client_drivers/24.2.x/24.2.0-1/vertica-client-24.2.0-1.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/24.1.x/24.1.0-0/vertica-client-24.1.0-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/23.4.x/23.4.0-0/vertica-client-23.4.0-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/23.3.x/23.3.0-0/vertica-client-23.3.0-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/12.0.x/12.0.4-0/vertica-client-12.0.4-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/11.1.x/11.1.1-0/vertica-client-11.1.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/11.0.x/11.0.2-0/vertica-client-11.0.2-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/10.1.x/10.1.1-0/vertica-client-10.1.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/10.0.x/10.0.1-0/vertica-client-10.0.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/9.3.x/9.3.1-0/vertica-client-9.3.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/9.2.x/9.2.1-0/vertica-client-9.2.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/9.1.x/9.1.1-0/vertica-client-9.1.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/9.0.x/9.0.1-0/vertica-client-9.0.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/8.1.x/8.1.1-0/vertica-client-8.1.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/8.0.x/8.0.1/vertica-client-8.0.1-0.x86_64.tar.gz\n# https://www.vertica.com/client_drivers/7.2.x/7.2.3-0/vertica-client-7.2.3-0.x86_64.tar.gz\n\ntimestamp \"Fetching list of Vertica tarball download URLs from $downloads_url\"\ntar_urls=\"$(\n    curl -sS \"$downloads_url\" |\n    grep -Eo \"$tar_url_regex\"\n)\"\n\nif [ \"$version\" = \"latest\" ]; then\n    timestamp \"Determining latest version from list of tarball download urls\"\n    download_url=\"$(head -n1 <<< \"$tar_urls\")\"\n    timestamp \"Determined latest tarball version to be ${download_url##*/}\"\nelse\n    timestamp \"Checking if requested version '$version' is available\"\n    download_url=\"$(grep \"$version\" <<< \"$tar_urls\" | head -n 1 || :)\"\n    if [ -z \"$download_url\" ]; then\n        echo\n        echo \"ERROR: Vertica Client tarball version '$version' not found\" >&2\n        echo\n        echo \"Here are the list of available versions:\"\n        echo\n        sed 's/.*vertica-client-//; s/-[[:digit:]]\\+.x86_64.tar.gz$//' <<< \"$tar_urls\"\n        echo\n        exit 1\n    fi\nfi\n\nexport RUN_VERSION_OPT=1\n\n#timestamp \"Downloading from: $download_url\"\n\"$srcdir/../packages/install_binary.sh\" \"$download_url\" opt/vertica/bin/vsql\n\n# automatically run by install_binary.sh when RUN_VERSION_OPT=1 is set above\n#if [ -w /usr/local/bin ]; then\n#    /usr/local/bin/vsql --version\n#else\n#    ~/bin/vsql --version\n#fi\n\necho\necho \"You may need to also set your locale, such as putting this in your \\$HOME/.bashrc:\"\necho\necho \"    export LC_ALL=$LC_ALL\"\necho\n"
  },
  {
    "path": "install/install_vertica_vsql_client_rpm.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-10 01:19:15 +0200 (Tue, 10 Sep 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://kubernetes.io/docs/tasks/tools/install-kubectl/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Vertica VSQL Client RPM on a Redhat-based Linux x86_64\n\nIf FIPS=true environment variable is set then downloads and installs the FIPS compliant RPM instead\n\nOffical Documentation:\n\n    https://docs.vertica.com/23.4.x/en/connecting-to/client-libraries/client-drivers/install-config/fips/installing-fips-client-driver-odbc-and-vsql/\n\nDownload URLs:\n\n    https://www.vertica.com/download/vertica/client-drivers/\n\nTested on Rocky Linux 8, 9\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n#version=\"24.2.0\"\nversion=\"${1:-latest}\"\n\n# Required by the vsql binary\n#\n# Invalid locale: run the \"locale\" command and check for warnings\n#export LANG=\"end_US.UTF-8\"\nexport LC_ALL=\"C.UTF-8\"\n\nlibxcrypt_package=\"\"\nif type -P rpm &>/dev/null; then\n    # already available on older Rocky Linux 8 from this package\n    if ! rpm -q libxcrypt &>/dev/null; then\n        libxcrypt_package=\"libxcrypt-compat\"\n    fi\nelse\n    die \"ERROR: running on a non-RPM system\"\nfi\n\nif [ -n \"$libxcrypt_package\" ]; then\n    timestamp \"Installing libxcrypt dependency\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" \"$libxcrypt_package\"\n    echo\nfi\n\ndownloads_url=\"https://www.vertica.com/download/vertica/client-drivers/\"\n\nfips=\"\"\nif [ \"${FIPS:-}\" = true ]; then\n    fips=\"-fips\"\nfi\n\n# ERE format for grep -E\nrpm_url_regex=\"https://www.vertica.com/client_drivers/[[:digit:].x-]+/[[:digit:].-]+/vertica-client$fips-[[:digit:].-]+.x86_64.rpm\"\n\n# Should match either these regular RPMs:\n#\n# https://www.vertica.com/client_drivers/24.2.x/24.2.0-1/vertica-client-24.2.0-1.x86_64.rpm\n# https://www.vertica.com/client_drivers/24.1.x/24.1.0-0/vertica-client-24.1.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/23.4.x/23.4.0-0/vertica-client-23.4.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/23.3.x/23.3.0-0/vertica-client-23.3.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/12.0.x/12.0.4-0/vertica-client-12.0.4-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/11.1.x/11.1.1-0/vertica-client-11.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/11.0.x/11.0.2-0/vertica-client-11.0.2-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/10.1.x/10.1.1-0/vertica-client-10.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/10.0.x/10.0.1-0/vertica-client-10.0.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.3.x/9.3.1-0/vertica-client-9.3.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.2.x/9.2.1-0/vertica-client-9.2.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.1.x/9.1.1-0/vertica-client-9.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.0.x/9.0.1-0/vertica-client-9.0.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/8.1.x/8.1.1-0/vertica-client-8.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/8.0.x/8.0.1/vertica-client-8.0.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/7.2.x/7.2.3-0/vertica-client-7.2.3-0.x86_64.rpm\n#\n# or if FIPS=true is set these RPMs:\n#\n# https://www.vertica.com/client_drivers/24.2.x/24.2.0-1/vertica-client-fips-24.2.0-1.x86_64.rpm\n# https://www.vertica.com/client_drivers/24.1.x/24.1.0-0/vertica-client-fips-24.1.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/23.4.x/23.4.0-0/vertica-client-fips-23.4.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/23.3.x/23.3.0-0/vertica-client-fips-23.3.0-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/12.0.x/12.0.4-0/vertica-client-fips-12.0.4-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/11.1.x/11.1.1-0/vertica-client-fips-11.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/11.0.x/11.0.2-0/vertica-client-fips-11.0.2-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/10.1.x/10.1.1-0/vertica-client-fips-10.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.2.x/9.2.1-0/vertica-client-fips-9.2.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.1.x/9.1.1-0/vertica-client-fips-9.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/9.0.x/9.0.1-0/vertica-client-fips-9.0.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/8.1.x/8.1.1-0/vertica-client-fips-8.1.1-0.x86_64.rpm\n# https://www.vertica.com/client_drivers/8.0.x/8.0.1/vertica-client-fips-8.0.1-0.x86_64.rpm\n\ntimestamp \"Fetching list of Vertica RPM download URLs from $downloads_url\"\nrpm_urls=\"$(\n    curl -sS \"$downloads_url\" |\n    grep -Eo \"$rpm_url_regex\"\n)\"\n\nif [ \"$version\" = \"latest\" ]; then\n    timestamp \"Determining latest version from list of RPM download urls\"\n    download_url=\"$(head -n1 <<< \"$rpm_urls\")\"\n    timestamp \"Determined latest RPM version to be ${download_url##*/}\"\nelse\n    timestamp \"Checking if requested version '$version' is available\"\n    download_url=\"$(grep \"$version\" <<< \"$rpm_urls\" | head -n 1 || :)\"\n    if [ -z \"$download_url\" ]; then\n        echo\n        echo \"ERROR: Vertica Client RPM FIPS version '$version' not found\" >&2\n        echo\n        echo \"Here are the list of available versions:\"\n        echo\n        sed 's/.*vertica-client-//; s/-[[:digit:]]\\+.x86_64.rpm$//' <<< \"$rpm_urls\"\n        echo\n        exit 1\n    fi\nfi\n\ntimestamp \"Installing from: $download_url\"\necho\n\nyum install -y \"$download_url\"\necho\n\n/opt/vertica/bin/vsql --version\n\necho\necho \"You may need to also set your locale, such as putting this in your \\$HOME/.bashrc:\"\necho\necho \"    export LC_ALL=$LC_ALL\"\necho\n"
  },
  {
    "path": "install/install_vundle.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-03 12:14:36 +0000 (Fri, 03 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Vim plugin manager Vundle to $HOME/.vim/bundle/Vundle.vim\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif ! type -P vim &>/dev/null; then\n    echo \"Vim not installed, aborting...\"\n    exit 1\nfi\n\ntarget=~/.vim/bundle/Vundle.vim\n\nmkdir -pv \"${target%/*}\"\n\nif ! [ -e \"$target\" ]; then\n    git clone https://github.com/gmarik/Vundle.vim.git \"$target\"\nfi\n\ndate \"+%F %T  Installing Vim Vundle plugins...\"\n# this tends to mess up the terminal and requires a reset afterwards\nvim --not-a-term +PluginInstall +qall >/dev/null\ndate \"+%F %T  Finished installing Vim Vundle plugins\"\n"
  },
  {
    "path": "install/install_wercker_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-01 21:01:50 +0000 (Sun, 01 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Wercker CLI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\nexport RUN_VERSION_ARG=1\n\n\"$srcdir/../packages/install_binary.sh\" \"https://s3.amazonaws.com/downloads.wercker.com/cli/stable/{os}_{arch}/wercker\"\n"
  },
  {
    "path": "install/install_yq.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-13 08:24:29 +0100 (Sat, 13 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls 'yq'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nexport PATH=\"$PATH:$HOME/bin\"\n\nhelp_usage \"$@\"\n\n#version=\"${1:-4.27.2}\"\nversion=\"${1:-latest}\"\n\nexport RUN_VERSION_OPT=1\n\n\"$srcdir/../github/github_install_binary.sh\" mikefarah/yq 'yq_{os}_{arch}' \"$version\"\n"
  },
  {
    "path": "internet/0x0.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 06:52:16 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://0x0.st and copies the resulting URL to your clipboard\n\nIf the content is ASCII then prompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nDoes not do this for non-ASCII files since we can't print media content to the terminal\n\nExpiry defaults to 24 hours\n\nRecommended: for text use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: for code - decomment.sh\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename> [<expiry_hours>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://0x0.st\"\n\nfile=\"$1\"\nexpiry=\"${2:-${OXO_EXPIRY:-24}}\"\n\nif ! [[ \"$expiry\" =~ ^[[:digit:]]+$ ]]; then\n    usage \"Invalid value for expiry arg, must be an integer of hours\"\nfi\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n#    timestamp \"reading from file: $file\"\n#fi\n\nif file \"$file\" | grep -q ASCII; then\n    content=\"$(cat \"$file\")\"\n    echo\n\n    cat <<EOF\nHere is what will be pasted to https://0x0.st:\n\n$content\n\nEOF\n\n    read -r -p \"Continue? [y/N] \" answer\n    echo\n\n    check_yes \"$answer\"\n    echo\nfi\n\n{\ncommand curl -sSlf \"$url\" \\\n             -F \"file=@$file\" \\\n             -F \"expires=$expiry\" ||\n    {\n        timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n        command curl -sSl \"$url\" \\\n                     -F \"file=@$file\" \\\n                     -F \"expires=$expiry\"\n        echo\n        exit 1\n    }\n} |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/atlassian_ip_ranges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 16:18:10 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFetches and parses Atlassian's IP ranges API, outputting the CIDR ranges, one per line\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[--ipv4 / --ipv6]\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_switches=\"\n   --ipv4           Output only IPv4 CIDR ranges\n   --ipv6           Output only IPv6 CIDR ranges\n\"\n\nhelp_usage \"$@\"\n\nfor arg; do\n    case \"$arg\" in\n        --ipv4)     IPV4_ONLY=1\n                    ;;\n        --ipv6)     IPV6_ONLY=1\n                    ;;\n    esac\ndone\n\nif [ -n \"${IPV4_ONLY:-}\" ] &&\n   [ -n \"${IPV6_ONLY:-}\" ]; then\n    usage \"IPv4 and IPv6 filters are mutually exclusive\"\nfi\n\nurl=\"https://ip-ranges.atlassian.com/\"\n\ncurl -sSf \"$url\" |\n#jq -r '.items[] | [.network, .mask_len | tostring ] | join(\"/\")'\njq -r '.items[] | .cidr' |\nif [ -n \"${IPV4_ONLY:-}\" ]; then\n    grep -v -e '[:alpha:]' -e ':'\nelif [ -n \"${IPV6_ONLY:-}\" ]; then\n    grep -e '[:alpha:]' -e ':'\nelse\n    cat\nfi\n"
  },
  {
    "path": "internet/catbox.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 06:57:02 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://catbox.moe/ and copies the resulting URL to your clipboard\n\nSlow, use 0x0.sh instead\n\nIf the content is ASCII then prompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nDoes not do this for non-ASCII files since we can't print media content to the terminal\n\nRetention is PERMANENT. Use litterbox.sh if you want temporary upload\n\nRecommended: for text use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: for code - decomment.sh\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://catbox.moe/user/api.php\"\n\nfile=\"$1\"\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n#    timestamp \"reading from file: $file\"\n#fi\n\nif file \"$file\" | grep -q ASCII; then\n    content=\"$(cat \"$file\")\"\n    echo\n\n    cat <<EOF\nHere is what will be pasted to https://catbox.moe/:\n\n$content\n\nEOF\n\n    read -r -p \"Continue? [y/N] \" answer\n    echo\n\n    check_yes \"$answer\"\n    echo\nfi\n\n{\ncommand curl -sSlf \"$url\" \\\n     -F \"reqtype=fileupload\" \\\n     -F \"fileToUpload=@$file\" ||\n\n    {\n        timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n        command curl -sSl \"$url\" \\\n             -F \"reqtype=fileupload\" \\\n             -F \"fileToUpload=@$file\"\n        echo\n        exit 1\n    }\n} |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/datadog_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /v1/validate\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-03 12:23:03 +0000 (Thu, 03 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the DataDog API\n\nRequires \\$DD_API_KEY / \\$DATADOG_TOKEN to be set in the environment\nIf \\$DD_APP_KEY is set, will also be sent in requests\n\nCreate an organization API key here:\n\n    https://app.datadoghq.eu/organization-settings/api-keys\n\n        (hint: copy the key, not the key id):\n\nCreate \\$DD_APP_KEY here:\n\n    https://app.datadoghq.eu/organization-settings/application-keys\n        or\n    https://app.datadoghq.eu/personal-settings/application-keys\n\n\nIf not using default US site, must also set \\$DATADOG_HOST, eg:\n\n    export DATADOG_HOST=https://api.datadoghq.eu\n\nFor site names, see:\n\n    https://docs.datadoghq.com/api/latest/authentication/\n\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://docs.datadoghq.com/api/latest/\n\n\nDefaults to /v2 API, but if /vN is specified, then uses that API instead, eg. prefixing /v1 to API endpoints that are only available in v1 such as /v1/validate or /v1/org\n\n\nExamples:\n\n# Validate your API token is working:\n\n    ${0##*/} /v1/validate | jq .\n\n# List users:\n\n    ${0##*/} /users | jq .\n        or\n    ${0##*/} /v1/user | jq .\n\n\n# List your organizations:\n\n    ${0##*/} /v1/org | jq .\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"${DATADOG_HOST:-https://api.datadoghq.com}/api/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nexport TOKEN=\"${DD_API_KEY:-${DATADOG_TOKEN:-}}\"\nexport CURL_AUTH_HEADER=\"DD-API-KEY:\"\n\nif [ -n \"${DD_APP_KEY:-}\" ]; then\n    CURL_OPTS+=(-H \"DD-APPLICATION-KEY: $DD_APP_KEY\")\nfi\n\nurl_path=\"$1\"\nshift || :\n\nif [[ \"$url_path\" =~ ^/?v[[:digit:]]+/ ]]; then\n    url_base=\"${url_base%%/v2}\"\nfi\nurl_path=\"${url_path//$url_base}\"\nurl_path=\"${url_path##/}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "internet/digital_ocean_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /account | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-06 00:22:15 +0100 (Wed, 06 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Digital Ocean API\n\nAutomatically handles authentication via environment variables \\$DIGITALOCEAN_ACCESS_TOKEN, \\$DIGITALOCEAN_TOKEN or \\$DIGITAL_OCEAN_TOKEN in that order of priority\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nGenerate a personal access token here:\n\n    https://cloud.digitalocean.com/account/api/tokens\n\n\nAPI Reference:\n\n    https://docs.digitalocean.com/reference/api/\n\n\nDefaults to /v2 API, but if /vN is specified, then uses that API instead, eg. prefixing /v1 to API endpoints that are only available in /v1 or if a new /v3 API gets released\n\n\nExamples:\n\n    # Get your account details:\n\n        ${0##*/} /account | jq .\n\n    # Get your account balance:\n\n        ${0##*/} /customers/my/balance | jq .\n\n    # List the projects in your account:\n\n        ${0##*/} /projects | jq .\n\n    # List the SSH keys in your account:\n\n        ${0##*/} /account/keys | jq .\n\n    # List the actions taken on your account:\n\n        ${0##*/} /actions | jq .\n\n    # List all Regions - datacentres, features, and machine sizes:\n\n        ${0##*/} /regions | jq .\n\n    # List your Kubernetes clusters:\n\n        ${0##*/} /kubernetes/clusters | jq .\n\n    # List the load balancers in your account:\n\n        ${0##*/} /load_balancers | jq .\n\n    # List all block storage volumes:\n\n        ${0##*/} /volumes | jq .\n\n    # List all droplets:\n\n        ${0##*/} /droplets | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.digitalocean.com/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nDIGITALOCEAN_ACCESS_TOKEN=\"${DIGITALOCEAN_ACCESS_TOKEN:-${DIGITALOCEAN_TOKEN:-${DIGITAL_OCEAN_TOKEN:-}}}\"\n\ncheck_env_defined DIGITALOCEAN_ACCESS_TOKEN\n\nexport TOKEN=\"${DIGITALOCEAN_ACCESS_TOKEN:-}\"\n\nurl_path=\"${1:-}\"\nshift || :\n\nif [[ \"$url_path\" =~ ^/?v[[:digit:]]+/ ]]; then\n    url_base=\"${url_base%%/v2}\"\nfi\nurl_path=\"${url_path//$url_base}\"\nurl_path=\"${url_path##/}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "internet/dnsjson.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: all.knownips.circleci.com\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-28 14:22:04 +0100 (Thu, 28 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries dnsjson.com for a given DNS record and prints the IP addresses\n\nDefaults to a type A dns record if not given\n\nReturns nothing if the DNS record is not found as dnsjson.com returns a blank result set in that case\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<dns_address> [<dns_record_type>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\naddress=\"$1\"\ntype=\"${2:-A}\"\n\ncurl -sSL \"https://dnsjson.com/$address/$type.json\" |\njq -r '.results.records[]' |\nsort -n\n"
  },
  {
    "path": "internet/domains_subdomains_environments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-24 17:31:03 +0700 (Mon, 24 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a given list of domains, deduplicate and print dev / staging subdomains as well as root domain for prod\n\nUsed to generate a whole bunch of Ad Tech domains and pixel tracker subdomains for a project\n\nSet environment variable SUBDOMAIN to alter the subdomain prefix:\n\n    export SUBDOMAIN='ads'\n\nSet environment variable ENVIRONMENTS to alter the subdomain suffixes:\n\n    export ENVIRONMENTS='dev staging'\n\nOutput:\n\n    ads-dev.<domain>     ads-staging.<domain>     ads.<domain>\n    ads-dev.<domain2>    ads-staging.<domain2>    ads.<domain2>\n    ads-dev.<domain3>    ads-staging.<domain3>    ads.<domain3>\n    ads-dev.<domain4>    ads-staging.<domain4>    ads.<domain4>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<domains_or_files_containing_domains>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nsubdomain=\"${SUBDOMAIN:-ads}\"\n\nenvironments=\"${ENVIRONMENTS:-\ndev\nstaging\n}\"\n\nif [ $# -eq 0 ]; then\n    log \"${0##*/}: Reading from stdin\"\n    cat\nelse\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            log \"${0##*/}: Reading from file: $arg\"\n            cat \"$arg\"\n        else\n            echo \"$arg\"\n        fi\n    done\nfi |\nsort -u |\nwhile read -r domain; do\n    # $domain_regex is defined in lib/util.sh\n    # shellcheck disable=SC2154\n    if ! [[ \"$domain\" =~ ^$domain_regex$ ]]; then\n        die \"Failed domain regex validation: $domain\"\n    fi\n    for environment in $environments; do\n        echo -n \"$subdomain-$environment.$domain \"\n    done\n    echo \"$subdomain.$domain\"\ndone |\ncolumn -t\n"
  },
  {
    "path": "internet/dpaste.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 06:06:59 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://dpaste.com and copies the resulting URL to your clipboard\n\nPrompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nExpiry defaults to 1 day\n\nRecommended: use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: decomment.sh\n\nSyntax Highlighting: tries to auto-infer it in this script for some common formats based on the file extension.\n                     You can override this as an argument\n\nSee values for syntax highlighting here:\n\n    https://dpaste.com/syntaxes/\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename> [<expiry> <format>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://dpaste.com/api/v2/\"\n\nfile=\"$1\"\nexpiry=\"${2:-${DPASTE_EXPIRY:-1}}\"\nformat=\"${3:-text}\"  # syntax highlighting\n\nif ! [[ \"$expiry\" =~ ^[[:digit:]]+$ ]]; then\n    usage \"Invalid value for expiry arg, must be an integer of days\"\nfi\n\nif ! file \"$file\" | grep -q ASCII; then\n    die \"This is only for text files like code. For non-text files use adjacent 0x0.sh\"\nfi\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n    timestamp \"reading from file: $file\"\n#fi\n\ncontent=\"$(cat \"$file\")\"\necho\n\ncat <<EOF\nHere is what will be dpaste-ed:\n\n$content\n\nEOF\n\nread -r -p \"Continue? [y/N] \" answer\necho\n\ncheck_yes \"$answer\"\necho\n\nif [ \"$format\" = text ]; then\n    ext=\"${file##*.}\"\n\n    shopt -s nocasematch\n\n    # adapted from:\n    #\n    #   https://dpaste.com/api/syntax-choices/\n    #\n    case \"$ext\" in\n        apache | log) format=apache ;;\n        apt) format=apt_sources ;;\n        as | actionscript) format=actionscript ;;\n        asm | s) format=asm ;;\n        bat | cmd) format=dos ;;\n        bib) format=bibtext ;;\n        clj | cljs | cljr | cljc | cljd | edn) format=clojure ;;\n        cpp | h | hpp) format=cpp ;;\n        cs) format=csharp ;;\n        el) format=emacs-lisp ;;\n        eml | email) format=email ;;\n        fs) format=fsharp ;;\n        groovy | gvy | gy | gsh) format=groovy ;;\n        hs | lhs) format=haskell ;;\n        java | jsh ) format=java ;;\n        js) format=javascript ;;\n        kt | kts | kexe | klib) format=kotlin ;;\n        lsp) format=lisp ;;\n        m) format=matlab ;;\n        md) format=markdown ;;\n        ml) format=ocaml ;;\n        pas) format=pascal ;;\n        php | php3 | php4) format=php ;;\n        pl | pm | t) format=perl ;;\n        plt | gnu | gpi | gih) format=gnuplot ;;\n        pp) format=puppet ;;\n        psql) format=psql ;;\n        postgres | postgresql) format=postgresql ;;\n        r) format=rsplus ;;\n        rb) format=ruby ;;\n        rs) format=rust ;;\n        sc | scala) format=scala ;;\n        scpti | scptd) format=applescript ;;\n        sh | bash) format='bash' ;;\n        spec) format=rpmspec ;;\n        ssh/config) format=sshconfig ;;\n        tex) format=latex ;;\n        ts) format=typescript ;;\n        v) format=verilog ;;\n        vb) format=vb.net ;;\n        vbs | vbscript) format=vbscript ;;\n        vim | vimscript) format=vim ;;\n        yml | yaml) format=yaml ;;\n        # the rest try for straight matches - some of these should probably be moved above with more file extension variations\n        #\n        # Populated from:\n        #\n        #   curl https://dpaste.com/api/syntax-choices/ | jq -r 'keys[]' | tr '\\n' '|'\n        abap|abnf|actionscript3|ada|adl|agda|aheui|alloy|ambienttalk|amdgpu|ampl|ansys|antlr|antlr-actionscript|antlr-cpp|antlr-csharp|antlr-java|antlr-objc|antlr-perl|antlr-python|antlr-ruby|apacheconf|apl|applescript|arduino|arrow|arturo|asc|asn1|aspectj|aspx-cs|aspx-vb|asymptote|augeas|autohotkey|autoit|awk|bare|basemake|batch|bbcbasic|bbcode|bc|bdd|befunge|berry|bibtex|blitzbasic|blitzmax|blueprint|bnf|boa|boo|boogie|bqn|brainfuck|bst|bugs|c|c-objdump|ca65|cadl|camkes|capdl|capnp|carbon|cbmbas|cddl|ceylon|cfc|cfengine3|cfm|cfs|chaiscript|chapel|charmci|cheetah|cirru|clay|clean|clojure|clojurescript|cmake|cobol|cobolfree|coffeescript|comal|common-lisp|componentpascal|console|coq|cplint|cpp-objdump|cpsa|cr|crmsh|croc|cryptol|csharp|csound|csound-document|csound-score|css|css+django|css+genshitext|css+lasso|css+mako|css+mozpreproc|css+myghty|css+php|css+ruby|css+smarty|css+ul4|cuda|cypher|cython|d|d-objdump|dart|dasm16|dax|debcontrol|debsources|delphi|desktop|devicetree|dg|diff|django|docker|doscon|dpatch|dtd|duel|dylan|dylan-console|dylan-lid|earl-grey|easytrieve|ebnf|ec|ecl|eiffel|elixir|elm|elpi|emacs-lisp|erb|erl|erlang|evoque|execline|extempore|ezhil|factor|fan|fancy|felix|fennel|fift|fish|flatline|floscript|forth|fortran|fortranfixed|foxpro|freefem|fsharp|fstar|func|futhark|gap|gap-console|gas|gcode|gdscript|genshi|genshitext|gherkin|glsl|gnuplot|go|golo|gooddata-cl|gosu|graphql|graphviz|groff|gsql|gst|haml|handlebars|haskell|haxe|haxeml|hexdump|hlsl|hsail|hspec|html|html+cheetah|html+django|html+evoque|html+genshi|html+handlebars|html+lasso|html+mako|html+myghty|html+ng2|html+php|html+smarty|html+twig|html+ul4|html+velocity|http|hybris|hylang|i6t|icon|idl|idris|iex|igor|inform6|inform7|ini|io|ioke|irc|isabelle|j|jags|jasmin|javascript+cheetah|javascript+django|javascript+lasso|javascript+mako|javascript+mozpreproc|javascript+myghty|javascript+php|javascript+ruby|javascript+smarty|jcl|jlcon|jmespath|js+genshitext|js+ul4|jsgf|jslt|json|jsonld|jsonnet|jsp|jsx|julia|juttle|k|kal|kconfig|kmsg|koka|kotlin|kql|kuin|lasso|ldapconf|ldif|lean|less|lighttpd|lilypond|limbo|liquid|literate-agda|literate-cryptol|literate-haskell|literate-idris|livescript|llvm|llvm-mir|llvm-mir-body|logos|logtalk|lsl|lua|macaulay2|make|mako|maql|mask|mason|mathematica|matlab|matlabsession|maxima|mcfunction|mcschema|meson|mime|minid|miniscript|mips|modelica|modula2|monkey|monte|moocode|moonscript|mosel|mozhashpreproc|mozpercentpreproc|mql|mscgen|mupad|mxml|myghty|mysql|nasm|ncl|nemerle|nesc|nestedtext|newlisp|newspeak|ng2|nginx|nimrod|nit|nixos|nodejsrepl|notmuch|nsis|numpy|nusmv|objdump|objdump-nasm|objective-c|objective-c++|objective-j|ocaml|octave|odin|omg-idl|ooc|opa|openedge|openscad|output|pacmanconf|pan|parasail|pawn|peg|perl|perl6|phix|pig|pike|pkgconfig|plpgsql|pointless|pony|portugol|postgres-explain|postscript|pot|pov|powershell|praat|procfile|prolog|promql|properties|protobuf|prql|psysh|ptx|pug|puppet|pwsh-session|py+ul4|py2tb|pycon|pypylog|pytb|python|python2|q|qbasic|qlik|qml|qvto|racket|ragel|ragel-c|ragel-cpp|ragel-d|ragel-em|ragel-java|ragel-objc|ragel-ruby|rbcon|rconsole|rd|reasonml|rebol|red|redcode|registry|resourcebundle|rexx|rhtml|ride|rita|rng-compact|roboconf-graph|roboconf-instances|robotframework|rql|rsl|rst|rust|sarl|sas|sass|savi|scaml|scdoc|scheme|scilab|scss|sed|sgf|shen|shexc|sieve|silver|singularity|slash|slim|slurm|smali|smalltalk|smarty|smithy|sml|snbt|snobol|snowball|solidity|sophia|sp|sparql|spice|splus|sql|sql+jinja|sqlite3|squidconf|srcinfo|ssp|stan|stata|supercollider|swift|swig|systemd|systemverilog|tads3|tal|tap|tasm|tcl|tcsh|tcshcon|tea|teal|teratermmacro|termcap|terminfo|terraform|text|thrift|ti|tid|tlb|tls|tnt|todotxt|toml|trac-wiki|trafficscript|treetop|tsql|turtle|twig|typoscript|typoscriptcssdata|typoscripthtmldata|ucode|ul4|unicon|unixconfig|urbiscript|usd|vala|vcl|vclsnippets|vctreestatus|velocity|verifpal|verilog|vgl|vhdl|visualprolog|visualprologgrammar|vyper|wast|wdiff|webidl|wgsl|whiley|wikitext|wowtoc|wren|x10|xml|xml+cheetah|xml+django|xml+evoque|xml+lasso|xml+mako|xml+myghty|xml+php|xml+ruby|xml+smarty|xml+ul4|xml+velocity|xorg.conf|xpp|xquery|xslt|xtend|xul+mozpreproc|yaml+jinja|yang|yara|zeek|zephir|zig|zone) format=\"$ext\" ;;\n    esac\nfi\n\n#filename_encoded=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$file\")\"\n\n#content=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$content\" | tr -d '\\n')\"\n\n{\n# try twice, fall back to trying without the API syntax highlighting selection in case it is wrong as this can result in\n#\ncommand curl -sSLf \"$url\" \\\n             -F \"expiry_days=$expiry\" \\\n             -F \"syntax=$format\" \\\n             -F \"content=<-\" <<< \"$content\" ||\n\n    command curl -sSLf \"$url\" \\\n                 -F \"expiry_days=$expiry\" \\\n                 -F \"content=<-\" <<< \"$content\" ||\n\n        {\n            timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n            command curl -sSL \"$url\" \\\n                         -F \"expiry_days=$expiry\" \\\n                         -F \"content=<-\" <<< \"$content\"\n            echo\n            exit 1\n        }\n} |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/file.io.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 07:20:10 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://file.io/ and copies the resulting URL to your clipboard\n\nIf the content is ASCII then prompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nDoes not do this for non-ASCII files since we can't print media content to the terminal\n\nRetention is a single download by design only and up to 2 weeks availability\n\nRecommended: for text use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: for code - decomment.sh\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://file.io\"\nfile=\"$1\"\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n#    timestamp \"reading from file: $file\"\n#fi\n\nif file \"$file\" | grep -q ASCII; then\n    content=\"$(cat \"$file\")\"\n    echo\n\n    cat <<EOF\nHere is what will be pasted to https://file.io/:\n\n$content\n\nEOF\n\n    read -r -p \"Continue? [y/N] \" answer\n    echo\n\n    check_yes \"$answer\"\n    echo\nfi\n\n{\ncommand curl -sSlf \"$url\" \\\n             -F \"file=@$file\" ||\n\n    {\n        timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n        command curl -sSl \"$url\" \\\n                     -F \"file=@$file\" ||\n        echo\n        exit 1\n    }\n} |\njq_debug_pipe_dump |\njq -r .link |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/google_maps_link.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-12-18 14:25:17 -0600 (Thu, 18 Dec 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries for a search string, returns the first hit and then generates a stable fixed place ID url to the result\n\nUseful for sharing in documentation links to places like HariSekhon/Knowledge-Base Travel pages\n\nA Google Maps API Key is required otherwise you will get an error like this:\n\n\t{\n\t   \\\"error_message\\\" : \\\"You must use an API key to authenticate each request to Google Maps Platform APIs. For additional information, please refer to http://g.co/dev/maps-no-account\\\",\n\t   \\\"html_attributions\\\" : [],\n\t   \\\"results\\\" : [],\n\t   \\\"status\\\" : \\\"REQUEST_DENIED\\\"\n\t}\n\nor this if you haven't enabled billing or your payment method has expired, causing the billing account to be disabled:\n\n\t{\n\t  \\\"error_message\\\": \\\"You must enable Billing on the Google Cloud Project at https://console.cloud.google.com/project/_/billing/enable Learn more at https://developers.google.com/maps/gmp-get-started\\\",\n\t  \\\"html_attributions\\\": [],\n\t  \\\"results\\\": [],\n\t  \\\"status\\\": \\\"REQUEST_DENIED\\\"\n\t}\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<query>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery=\"$*\"\n\ncheck_env_defined GOOGLE_MAPS_API_KEY\n\nencoded_query=\"$(\n    printf '%s' \"$query\" |\n    jq -sRr @uri\n)\"\n\nplace_id=\"$(\n    curl -s \\\n        \"https://maps.googleapis.com/maps/api/place/textsearch/json?query=${encoded_query}&key=${GOOGLE_MAPS_API_KEY}\" |\n    jq_debug_pipe_dump |\n    jq -r '.results[0].place_id // empty'\n)\"\n\nif [ -z \"$place_id\" ]; then\n    echo \"No place ID found\" >&2\n    exit 1\nfi\n\nprintf 'https://www.google.com/maps/place/?q=place_id:%s\\n' \"$place_id\"\n"
  },
  {
    "path": "internet/imgur.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 07:20:10 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads an image file to https://imgur.com/ and copies the resulting URL to your clipboard\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image_file>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nIMGUR_CLIENT_ID=\"${IMGUR_CLIENT_ID:-DevOps-Bash-tools}\"\n\nurl=\"https://api.imgur.com/3/image\"\nfile=\"$1\"\n\nif ! file \"$file\" | grep -qi image; then\n    die \"Only image files may be uploaded\"\nfi\n\nresult=\"$(\n    command curl -sSl \"$url\" \\\n             -H \"Authorization: Client-ID $IMGUR_CLIENT_ID\" \\\n             -F \"image=@$file\"\n)\"\n\njq_debug_pipe_dump <<< \"$result\" > /dev/null\n\nsuccess=\"$(jq -r .success <<< \"$result\")\"\nif [ \"$success\" != true ]; then\n    error=\"$(jq -r .data.error <<< \"$result\")\"\n    die \"ERROR: $error\"\nfi\n\njq -r '.data.link' <<< \"$result\" |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\n"
  },
  {
    "path": "internet/jira_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-11 16:24:49 +0000 (Fri, 11 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Jira API\n\nRequires following environment variables to be defined:\n\n    \\$JIRA_USER  - eg. hari.sekhon@domain.com\n    \\$JIRA_TOKEN\n    \\$JIRA_DOMAIN - eg. mycompany (if your Jira URLs in your browser are https://mycompany.atlassian.net)\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here:\n\n    https://id.atlassian.com/manage-profile/security/api-tokens\n\n\nAPI Reference:\n\n    https://developer.atlassian.com/cloud/jira/platform/rest/v3/intro/\n\n    https://developer.atlassian.com/cloud/jira/platform/rest/v2/intro/\n\n\nThis defaults to API v3 but to use an older API such as v2 simply prefix the path with /2/\n\n\nExamples:\n\n# List Users:\n\n    ${0##*/} /2/users/search | jq -r '.[].displayName' | sort -fu\n\n# List Groups (admin privileges required otherwise gets 403 error):\n\n    ${0##*/} /2/group | jq .\n\n# List Projects:\n\n    ${0##*/} /project/search | jq .\n\n# List Issues:\n\n    ${0##*/} /2/search?jql= | jq .\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncheck_env_defined JIRA_DOMAIN\ncheck_env_defined JIRA_USER\ncheck_env_defined JIRA_TOKEN\n\nurl_base=\"https://$JIRA_DOMAIN.atlassian.net/rest/api/3\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nexport USERNAME=\"$JIRA_USER\"\nexport PASSWORD=\"$JIRA_TOKEN\"\n\nurl_path=\"$1\"\nshift || :\n\nif [[ \"$url_path\" =~ ^/?[[:digit:]]+(\\.[[:digit:]]+)?/ ]]; then\n    url_base=\"${url_base%%/3}\"\nfi\nurl_path=\"${url_path//$url_base}\"\nurl_path=\"${url_path##/}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "internet/kong_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /license/report | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-07 23:36:30 +0100 (Fri, 07 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Kong Gateway Admin API\n\nSee also the 'deck' command\n\nAutomatically handles authentication via environment variable \\$KONG_TOKEN if available\n\nKONG_API_HOST must be set\nKONG_API_PORT defaults to 8433 for SSL\nKONG_API_PROTOCOL defaults to 'https'\n\nKONG_TOKEN (optional) if Admin API is secured with RBAC (Enterprise only)\n\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://docs.konghq.com/gateway/latest/admin-api/\n\n\nExamples:\n\n# Check health status:\n\n    ${0##*/} /status | jq .\n\n\n# List Admin API endpoints:\n\n    ${0##*/} /endpoints | jq .\n\n\n# List services:\n\n    ${0##*/} /services | jq .\n\n\n# Query a specific service:\n\n    ${0##*/} /services/{service name or id} | jq .\n\n\n# List routes:\n\n    ${0##*/} /routes | jq .\n\n\n# Query a specific service:\n\n    ${0##*/} /routes/{route name or id} | jq .\n\n\n# List consumers:\n\n    ${0##*/} /consumers | jq .\n\n\n# Install a plugin:\n\n    ${0##*/} /plugins -X POST -d 'name=rate-limiting' | jq .\n\n\n# List all plugins (only shows user installed plugins, not bundled ones it seems):\n\n    ${0##*/} /plugins | jq .\n\n\n# List enabled plugins including bundled ones:\n\n    ${0##*/} /plugins/enabled | jq .\n\n\n# List upstreams (backend services):\n\n    ${0##*/} /upstreams | jq .\n\n\n# Query one upstream\n\n    ${0##*/} /upstreams/{upstream name or id} | jq .\n\n\n# List all targets for an upstream backend service:\n\n    ${0##*/} /upstreams/{name or id}/targets/all | jq .\n\n\n# List workspaces:\n\n    ${0##*/} /workspaces | jq .\n\n\n# List RBAC users:\n\n    ${0##*/} /rbac/users | jq .\n\n\n# List admins:\n\n    ${0##*/} /admins | jq .\n\n\n# Get the current license details:\n\n    ${0##*/} /license/report | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncheck_env_defined KONG_API_HOST\n\nhost=\"$KONG_API_HOST\"\nport=\"${KONG_API_PORT:-8443}\"\nprotocol=\"${KONG_API_PROTOCOL:-https}\"\nurl_base=\"$protocol://$host:$port\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl=\"$url_path\"\nif ! [[ \"$url_path\" =~ :// ]]; then\n    url_path=\"${url_path##/}\"\n    url=\"$url_base/$url_path\"\nfi\n\nexport CURL_AUTH_HEADER=\"Kong-Admin-Token:\"\nexport TOKEN=\"${KONG_TOKEN:-${TOKEN:-no_token_given}}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "internet/litterbox.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 07:12:37 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://litterbox.catbox.moe/ and copies the resulting URL to your clipboard\n\nSlow, use 0x0.sh instead\n\nIf the content is ASCII then prompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nDoes not do this for non-ASCII files since we can't print media content to the terminal\n\nExpiry defaults to 24 hours\n\nRecommended: for text use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: for code - decomment.sh\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://litterbox.catbox.moe/resources/internals/api.php\"\n\nfile=\"$1\"\nexpiry=\"${2:-${LITTERBOX_EXPIRY:-24}}\"\n\nif ! [[ \"$expiry\" =~ ^(1|12|24|72)$ ]]; then\n    usage \"Invalid value for expiry hours arg, must be one of: 1, 12, 24, 72\"\nfi\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n#    timestamp \"reading from file: $file\"\n#fi\n\nif file \"$file\" | grep -q ASCII; then\n    content=\"$(cat \"$file\")\"\n    echo\n\n    cat <<EOF\nHere is what will be pasted to https://litterbox.catbox.moe/:\n\n$content\n\nEOF\n\n    read -r -p \"Continue? [y/N] \" answer\n    echo\n\n    check_yes \"$answer\"\n    echo\nfi\n\n{\ncommand curl -sSlf \"$url\" \\\n     -F \"reqtype=fileupload\" \\\n     -F \"time=${expiry}h\" \\\n     -F \"fileToUpload=@$file\" ||\n\n    {\n        timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n        command curl -sSl \"$url\" \\\n             -F \"reqtype=fileupload\" \\\n             -F \"time=${expiry}h\" \\\n             -F \"fileToUpload=@$file\"\n        echo\n        exit 1\n    }\n} |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/ngrok_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /api_keys | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-11 09:03:03 +0100 (Sat, 11 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the NGrok.com API\n\nAutomatically handles authentication via environment variable \\$NGROK_API_TOKEN\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here - may need to click 'Enable SSO' next to each token to access corporation organizations with SSO (eg. Azure AAD SSO):\n\n    https://dashboard.ngrok.com/api\n\n\nAPI Reference:\n\n    https://ngrok.com/docs/api\n\n\nExamples:\n\nList API Keys:\n\n    ${0##*/} /api_keys | jq .\n\nList Agents:\n\n    ${0##*/} /agent_ingresses | jq .\n\nList Tunnels:\n\n    ${0##*/} /tunnel_sessions | jq .\n\nList FailOver Backends:\n\n    ${0##*/} /backends/failover | jq .\n\nList HTTP Response Backends:\n\n    ${0##*/} /backends/http_response | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.ngrok.com\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\ncheck_env_defined \"NGROK_API_TOKEN\"\n\nexport TOKEN=\"${NGROK_API_TOKEN}\"\n\nurl_path=\"${1:-}\"\nshift || :\n\nurl_path=\"${url_path//https:\\/\\/api.ngrok.com}\"\nurl_path=\"${url_path##/}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" -H \"Ngrok-Version: 2\" \"${CURL_OPTS[@]}\" \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "internet/pastebin.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2010-05-18 10:40:36 +0100 (Tue, 18 May 2010)\n#  (just discovered in private repo and ported here)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://pastebin.com and copies the resulting URL to your clipboard\n\nPrompts to confirm the content before uploading for your safe review as this defaults to PUBLIC but not listed\n\nExpiry defaults to 1 day\n\nRequired: an API key in environment variable PASTEBIN_API_KEY\n\nRecommended: use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: decomment.sh\n\nSyntax Highlighting: the API doesn't infer syntax highlighting based on the filename extension,\n                     so tries to auto-infer it in this script for some common formats based on the file extension.\n                     You can override this as an argument\n\nSee values for parameters here:\n\n    https://pastebin.com/doc_api#4\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\nIf you use this a lot you will hit this error from the API:\n\n    Bad API request, Post limit, maximum pastes per 24h reached\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename> [<expiry> <private> <format>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined PASTEBIN_API_KEY\n\nurl=\"https://pastebin.com/api/api_post.php\"\n\nfile=\"$1\"\nexpiry=\"${2:-${PASTEBIN_EXPIRY:-1D}}\"\nprivate=\"${3:-1}\"  # 1=unlisted (default), 0=public, 2=private\nformat=\"${4:-text}\"  # syntax highlighting\n\nif ! [[ \"$private\" =~ ^(0|1|2)$ ]]; then\n    usage \"Invalid value for private arg, must be one of: 0, 1 or 2 for public, unlisted or private respectively\"\nfi\n\nif ! [[ \"$expiry\" =~ ^[[:digit:]]+[[:alpha:]]$ ]]; then\n    usage \"Invalid value for expiry arg, must be in format: <integer><uppercase_unit_of_time_character>\"\nfi\nexpiry=\"$(tr '[:lower:]' '[:upper:]' <<< \"$expiry\")\"\n\nif ! file \"$file\" | grep -q ASCII; then\n    die \"This is only for text files like code. For non-text files use adjacent 0x0.sh\"\nfi\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n    timestamp \"reading from file: $file\"\n#fi\n\ncontent=\"$(cat \"$file\")\"\necho\n\ncat <<EOF\nHere is what will be pastebin-ed:\n\n$content\n\nEOF\n\nread -r -p \"Continue? [y/N] \" answer\necho\n\ncheck_yes \"$answer\"\necho\n\nif [ \"$format\" = text ]; then\n    ext=\"${file##*.}\"\n\n    shopt -s nocasematch\n\n    # adapted from:\n    #\n    #   https://pastebin.com/doc_api#5\n\n    case \"$ext\" in\n        apache | log) format=apache ;;\n        apt) format=apt_sources ;;\n        as | actionscript) format=actionscript ;;\n        asm | s) format=asm ;;\n        bat | cmd) format=dos ;;\n        bib) format=bibtext ;;\n        clj | cljs | cljr | cljc | cljd | edn) format=clojure ;;\n        cpp | h | hpp) format=cpp ;;\n        cs) format=csharp ;;\n        el) format=emacs-lisp ;;\n        eml | email) format=email ;;\n        erl | hrl) format=erlang ;;\n        fs) format=fsharp ;;\n        groovy | gvy | gy | gsh) format=groovy ;;\n        hs | lhs) format=haskell ;;\n        java | jsh ) format=java ;;\n        js) format=javascript ;;\n        kt | kts | kexe | klib) format=kotlin ;;\n        lsp) format=lisp ;;\n        m) format=matlab ;;\n        md) format=markdown ;;\n        ml) format=ocaml ;;\n        pas) format=pascal ;;\n        php | php3 | php4) format=php ;;\n        pl | pm | t) format=perl ;;\n        plt | gnu | gpi | gih) format=gnuplot ;;\n        pp) format=puppet ;;\n        psql | postgres | postgresql) format=postgresql ;;\n        r) format=rsplus ;;\n        rb) format=ruby ;;\n        rs) format=rust ;;\n        sc | scala) format=scala ;;\n        scpti | scptd) format=applescript ;;\n        sh | bash) format='bash' ;;\n        spec) format=rpmspec ;;\n        ssh/config) format=sshconfig ;;\n        tex) format=latex ;;\n        ts) format=typescript ;;\n        v) format=verilog ;;\n        vb) format=vbnet ;;\n        vbs | vbscript) format=vbscript ;;\n        vim | vimscript) format=vim ;;\n        yml | yaml) format=yaml ;;\n        # the rest try for straight matches - some of these should probably be moved above with more file extension variations\n        abap|actionscript3|ada|aimms|algol68|applescript|arduino|arm|asp|asymptote|autoconf|autohotkey|autoit|avisynth|awk|bascomavr|basic4gl|bibtex|b3d|blitzbasic|bmx|bnf|boo|bf|c|c_winapi|cpp-winapi|cpp-qt|c_loadrunner|caddcl|cadlisp|ceylon|cfdg|c_mac|chaiscript|chapel|cil|klonec|klonecpp|cmake|cobol|coffeescript|cfm|css|cuesheet|d|dart|dcl|dcpu16|dcs|delphi|oxygene|diff|div|dot|e|ezt|ecmascript|eiffel|epc|euphoria|falcon|filemaker|fo|f1|fortran|freebasic|freeswitch|gambas|gml|gdb|gdscript|genero|genie|gettext|go|godot-glsl|gwbasic|haxe|hicest|hq9plus|html4strict|html5|icon|idl|ini|inno|intercal|io|ispfpanel|j|jcl|jquery|json|julia|Julia|kixtart|ksp|ldif|lb|lsl2|lisp|llvm|locobasic|logtalk|lolcode|lotusformulas|lotusscript|lscript|lua|m68k|magiksf|make|mapbasic|markdown|mercury|metapost|mirc|mmix|mk-61|modula2|modula3|68000devpac|mpasm|mxml|mysql|nagios|netrexx|newlisp|nginx|nim|nsis|oberon2|objeck|objc|ocaml|ocaml-brief|octave|pf|glsl|oorexx|oobas|oracle8|oracle11|oz|parasail|parigp|pascal|pawn|pcre|per|perl|perl6|phix|php-brief|pic16|pike|pixelbender|pli|plsql|postscript|povray|powerbuilder|powershell|proftpd|progress|prolog|properties|providex|purebasic|pycon|python|pys60|q|q/kdb+|qbasic|qml|racket|rails|rbs|rebol|reg|rexx|robots|roff|sas|scheme|scilab|scl|sdlbasic|smalltalk|smarty|spark|sparql|sqf|sql|standardml|StandardML|stonescript|sclang|swift|systemverilog|tsql|tcl|teraterm|texgraph|thinbasic|typoscript|unicon|uscript|upc|urbi|vala|vedit|verilog|vhdl|visualfoxpro|visualprolog|whitespace|whois|winbatch|xbasic|xml|xojo|xorg_conf|xpp|yara|z80|zxbasic) format=\"$ext\" ;;\n    esac\nfi\n\nfilename_encoded=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$file\")\"\n\n#content=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$content\" | tr -d '\\n')\"\n\n{\n# try twice, fall back to trying without the API paste format in case it is wrong as this can result in\n#\n#   Bad API request, invalid api_paste_format\n#\ncommand curl -X POST -sSLf \"$url\" \\\n     -d \"api_option=paste\" \\\n     -d \"api_dev_key=$PASTEBIN_API_KEY\" \\\n     -d \"api_paste_name=$filename_encoded\" \\\n     -d \"api_paste_code=$content\" \\\n     -d \"api_paste_expire_date=$expiry\" \\\n     -d \"api_paste_private=$private\" \\\n     -d \"api_paste_format=$format\" ||\n\n    command curl -X POST -sSLf \"$url\" \\\n         -d \"api_option=paste\" \\\n         -d \"api_dev_key=$PASTEBIN_API_KEY\" \\\n         -d \"api_paste_name=$filename_encoded\" \\\n         -d \"api_paste_code=$content\" \\\n         -d \"api_paste_expire_date=$expiry\" \\\n         -d \"api_paste_private=$private\" ||\n\n        {\n            timestamp \"FAILED: repeating without the curl -f switch to get the error from the API:\"\n            command curl -X POST -sSL \"$url\" \\\n                 -d \"api_option=paste\" \\\n                 -d \"api_dev_key=$PASTEBIN_API_KEY\" \\\n                 -d \"api_paste_name=$filename_encoded\" \\\n                 -d \"api_paste_code=$content\" \\\n                 -d \"api_paste_expire_date=$expiry\" \\\n                 -d \"api_paste_private=$private\"\n            echo\n            exit 1\n        }\n} |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/shields_embed_logo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-16 19:53:07 +0200 (Fri, 16 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBase64 encodes a given icon file or url and prints the logo=... url parameter you need to add the shields.io badge url\n\nIf you pass a URL will download the URL and base64 the content\n\nOtherwise download the icon to your local disk eg. using SimpleIcons.org and pass that as a parameter\n\nThis is only tested with svg files at this time\n\nExample:\n\n        wget -nc https://raw.githubusercontent.com/simple-icons/simple-icons/e8de041b64586c0c532f9ea5508fd8e29d850937/icons/linkedin.svg\n\n        ${0##*/} linkedin.svg\n\n    or\n\n        ${0##*/} https://raw.githubusercontent.com/simple-icons/simple-icons/e8de041b64586c0c532f9ea5508fd8e29d850937/icons/linkedin.svg\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename_or_url>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nicon=\"$1\"\n\nif [[ \"$icon\" =~ ^https?:// ]]; then\n    content=\"$(curl -sS \"$icon\")\"\nelif [ -f \"$icon\" ]; then\n    content=\"$(cat \"$icon\")\"\nelse\n    usage \"first argument needs to be a URL or a local file\"\nfi\n\ndata=\"$(base64 <<< \"$content\")\"\n\necho \"logo=data:image/svg%2bxml;base64,$data\"\n"
  },
  {
    "path": "internet/termbin.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-07 06:07:05 +0400 (Thu, 07 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads a file to https://termbin.com and copies the resulting URL to your clipboard\n\nPrompts to confirm the content before uploading for your safe review as this is PUBLIC\n\nRecommended: use anonymize.py or anonymize.pl from the adjacent DevOps-Python-tools or DevOps-Perl-tools repos\n\nOptional: decomment.sh\n\nThere is no syntax highlighting on https://termbin.com\n\nKnowledge Base page: https://github.com/HariSekhon/Knowledge-Base/blob/main/upload-sites.md\n\nRequires nc or ncat or netcat to be installed\n\nAttempts to install netcat on common systems like RHEL / Debian / Ubuntu / Alpine / Mac if needed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfile=\"$1\"\n\nif ! file \"$file\" | grep -q ASCII; then\n    die \"This is only for text files like code. For non-text files use adjacent 0x0.sh\"\nfi\n\nnc=nc\n\nif type -P nc &>/dev/null; then\n    :\nelif type -P ncat &>/dev/null; then\n    nc=ncat\nelif type -P netcat &>/dev/null; then\n    nc=netcat\nelif is_mac; then\n    \"$srcdir/../packages/brew_install_packages.sh\" netcat\nelif type -P yum &>/dev/null ||\n     type -P apt-get &>/dev/null; then\n    \"$srcdir/../packages/install_packages.sh\" nc\nelif type -P apk &>/dev/null; then\n    \"$srcdir/../packages/apk_install_packages.sh\" netcat-openbsd\nelse\n    die \"No netcat program installed: nc, ncat or netcat\"\nfi\n\n# Do not allow reading from stdin because it does not allow the prompt safety\n#if [ \"$file\" = '-' ]; then\n#    timestamp \"reading from stdin\"\n    #file=\"/dev/stdin\"\n#else\n    timestamp \"reading from file: $file\"\n#fi\n\ncontent=\"$(cat \"$file\")\"\necho\n\ncat <<EOF\nHere is what will be uploaded to termbin:\n\n$content\n\nEOF\n\nread -r -p \"Continue? [y/N] \" answer\necho\n\ncheck_yes \"$answer\"\necho\n\n\"$nc\" termbin.com 9999 <<< \"$content\" |\ntee /dev/stderr |\n\"$srcdir/../bin/copy_to_clipboard.sh\"\necho\n"
  },
  {
    "path": "internet/traefik_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /api/version | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-30 03:27:47 +0100 (Sun, 30 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Traefik API\n\nAutomatically handles authentication via environment variable \\$TRAEFIK_TOKEN if available\n\nTRAEFIK_API_HOST must be set\nTRAEFIK_API_PORT defaults to '443'\nTRAEFIK_API_PROTOCOL defaults to 'https'\n\nTRAEFIK_TOKEN (optional) if API is secured with a middleware JWT authentication token\nIf using HTTP basic auth set TRAEFIK_TOKEN=' ' with a blank and instead set USERNAME/PASSWORD environment variables, otherwise it'll use \\$USERNAME from your shell and prompt for a password\n\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://doc.traefik.io/traefik/operations/api/\n\n\nFor convenience you may omit the /api prefix and it will be added automatically\n\n\nExamples:\n\n# Get the API version:\n\n    ${0##*/} /api/version | jq .\n\n# Overview stats of http, tcp, enabled features and providers:\n\n    ${0##*/} /api/overview | jq .\n\n# List entrypoints:\n\n    ${0##*/} /api/entrypoints | jq .\n\n# Get only the 'websecure' entrypoint:\n\n    ${0##*/} /api/entrypoints/websecure | jq .\n\n# List HTTP routers:\n\n    ${0##*/} /api/http/routers | jq .\n\n# List TCP routers:\n\n    ${0##*/} /api/tcp/routers | jq .\n\n# Get only the 'ping@internal' HTTP entrypoint:\n\n    ${0##*/} /api/http/routers/ping@internal | jq .\n\n# List HTTP middlewares:\n\n    ${0##*/} /api/http/middlewares | jq .\n\n# List TCP middlewares:\n\n    ${0##*/} /api/tcp/middlewares | jq .\n\n# Get only the 'traefik-strip-prefix-catch-all@kubernetescrd' HTTP middleware:\n\n    ${0##*/} /api/http/middlewares/traefik-strip-prefix-catch-all@kubernetescrd | jq .\n\n# List HTTP services:\n\n    ${0##*/} /api/http/services | jq .\n\n# List TCP services:\n\n    ${0##*/} /api/tcp/services | jq .\n\n# Get only the 'dashboard@internal' HTTP service:\n\n    ${0##*/} /api/http/services/dashboard@internal | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncheck_env_defined TRAEFIK_API_HOST\n\nhost=\"$TRAEFIK_API_HOST\"\nport=\"${TRAEFIK_API_PORT:-443}\"\nprotocol=\"${TRAEFIK_API_PROTOCOL:-https}\"\nurl_base=\"$protocol://$host:$port\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\nurl_path=\"${url_path#/}\"\nif ! [[ \"$url_path\" =~ ^api ]]; then\n    url_path=\"api/$url_path\"\nfi\n\nurl=\"$url_path\"\nif ! [[ \"$url_path\" =~ :// ]]; then\n    url_path=\"${url_path##/}\"\n    url=\"$url_base/$url_path\"\nfi\n\nexport TOKEN=\"${TRAEFIK_TOKEN:-${TOKEN:-no_token_given}}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "internet/wordpress.htaccess",
    "content": "php_value upload_max_filesize 3072M\nphp_value post_max_size 3072M\nphp_value memory_limit 3072M\nphp_value max_execution_time 300\nphp_value max_input_time 300\n"
  },
  {
    "path": "internet/wordpress.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-12-04 12:21:15 +0000 (Mon, 04 Dec 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://hub.docker.com/_/wordpress\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# dipping into interactive library for opening browser to Wordpress\n# XXX: order is important here because there is an interactive library of retry() and a scripting library version of retry() and we want the latter, which must be imported second\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../.bash.d/network.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nexport WORDPRESS_URL=\"http://${DOCKER_HOST:-localhost}:8080\"\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport DOCKER_CONTAINER=\"$COMPOSE_PROJECT_NAME-wordpress-1\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/wordpress.yml\"\nexport WORDPRESS_HTACCESS_FILE=\"$srcdir/wordpress.htaccess\"\nexport WORDPRESS_HTACCESS_PATH=\"/var/www/html/.htaccess\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots a quick Wordpress blog container\n\nCopies .htaccess settings from the adjacent file to increase upload sizes for restoring backups:\n\n    $WORDPRESS_HTACCESS_FILE\n\nOpens the Wordpress URL in the default browser:\n\n    $WORDPRESS_URL\n\nTo boot a specific version of Wordpress:\n\nexport VERSION=6.4\n\nNote some plugins will break the Live Preview of themes and must be temporarily deactivated to see them, such as:\n\nAntispam Bee\nBroken Link Checker\nJetpack Boost\nIONOS Security\nYoast SEO plugin\n\nTested on Wordpress 6.4\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting Wordpress:\"\n    docker-compose up -d \"$@\"\n    echo\n    when_url_content 60 \"$WORDPRESS_URL\" '.*'\n    echo\n    timestamp \"Copying $WORDPRESS_HTACCESS_FILE into wordpress container $WORDPRESS_HTACCESS_PATH\"\n    docker cp \"$WORDPRESS_HTACCESS_FILE\" \"$DOCKER_CONTAINER\":\"$WORDPRESS_HTACCESS_PATH\"\n    echo\n    exec \"${BASH_SOURCE[0]}\" ui\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"Wordpress URL:  $WORDPRESS_URL\"\n    open \"$WORDPRESS_URL\"\nelse\n    docker-compose \"$action\" \"$@\"\n    echo\nfi\n"
  },
  {
    "path": "internet/wordpress_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /users/1 | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2024-01-23 10:54:30 +0000 (Tue, 23 Jan 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Wordpress API v2\n\nRequires WORDPRESS_URL, WORDPRESS_USER and WORDPRESS_PASSWORD environment variables to be defined\n\nAutomatically adds the /wp-json/wp/v2/ prefix for convenience\n\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up an application password and export it in the shell environment variable WORDPRESS_PASSWORD\n\n    https://make.wordpress.org/core/2020/11/05/application-passwords-integration-guide/\n\n\nAPI Reference:\n\n    https://developer.wordpress.org/rest-api/reference/\n\n\nTo return only a subset of fields add '?fields=field1,field2...' to the /url/path\n\n    https://developer.wordpress.org/rest-api/using-the-rest-api/global-parameters/\n\n\nYou will need to handle pagination in the calling scripts:\n\n    https://developer.wordpress.org/rest-api/using-the-rest-api/pagination/\n\n\nExamples:\n\n\n# Get users:\n\n    ${0##*/} /users\n\n    # get a specific user by user id 1\n    ${0##*/} /users/1\n\n\n# Get categories:\n                                                   # or .name\n    ${0##*/} /categories?per_page=100 | jq -r '.[].slug'\n\n\n# Get comments:\n\n    ${0##*/} /comments?per_page=100 | jq .\n\n    # get a specific comment\n    ${0##*/} /comments/<id>\n\n\n# Get media (images metadata):\n\n    ${0##*/} /media | jq .\n\n\n# Get pages:\n\n    ${0##*/} /pages | jq .\n\n\n# Get posts (articles):\n\n    ${0##*/} /posts | jq .\n\n\n# Get post revisions:\n\n    ${0##*/} /posts/<id>/revisions | jq .\n\n# Get posts that match a given search term:\n\n    ${0##*/} /search?search=sekhon | jq .\n\n# Get tags:\n                                                   # or .name\n    ${0##*/} /tags?per_page=100 | jq -r '.[].slug'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined WORDPRESS_URL\ncheck_env_defined WORDPRESS_USER\ncheck_env_defined WORDPRESS_PASSWORD\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_base=\"$WORDPRESS_URL\"\nif ! [[ \"$url_base\" =~ ^https:// ]] && ! [[ \"$url_base\" =~ http://localhost[:/] ]]; then\n    usage \"must use an HTTPS url for WORDPRESS_URL for safety unless using localhost\"\nfi\n\n# more generic but needless fork\n#url_path=\"$(sed 's|https?://[^/]*' <<< \"$url_path\")\"\n#\n# not needed, works without this ugly hack\n# shellcheck disable=SC2295\n#url_path=\"${url_path##\"\"$WORDPRESS_URL\"\"}\"\nurl_path=\"${url_path##$WORDPRESS_URL}\"\n\nurl_path=\"${url_path##/}\"\nif [[ \"$url_path\" =~ /v1/|/wp-site-health/ ]]; then\n    WORDPRESS_NO_PREFIX=1\nfi\nif [ -z \"${WORDPRESS_NO_PREFIX:-}\" ]; then\n    url_path=\"${url_path##wp-json/}\"\n    url_path=\"${url_path##wp/}\"\n    url_path=\"${url_path##v2/}\"\n    url_path=\"wp-json/wp/v2/$url_path\"\nfi\n\nexport USERNAME=\"$WORDPRESS_USER\"\nexport PASSWORD=\"$WORDPRESS_PASSWORD\"\n\n# useless it requires ID which is not your $WORDPRESS_USERNAME\n#url_path=\"${url_path/<user_id>/$WORDPRESS_USER}\"\n#url_path=\"${url_path/\\{user_id\\}/$WORDPRESS_USER}\"\n#url_path=\"${url_path/<user>/$WORDPRESS_USER}\"\n#url_path=\"${url_path/\\{user\\}/$WORDPRESS_USER}\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "internet/wordpress_plugins_markdown.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-12-14 19:44:36 -0600 (Sun, 14 Dec 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a Markdown list of Wordpress plugins for documentation purposes\n\nYou must run this from inside the Wordpress installation root\n\nRuns PHP to get the PluginURI which is not exposed by the 'wp plugin list' command\n\nTested on Wordpress 6\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nno_more_args \"$@\"\n\nwp core version\n\n# don't want expansion\n# shellcheck disable=SC2016\nwp eval '\nrequire_once ABSPATH . \"wp-admin/includes/plugin.php\";\n\n$plugins = get_plugins();\n\nforeach ($plugins as $plugin) {\n  $name = $plugin[\"Name\"];\n  $url  = $plugin[\"PluginURI\"] ?: \"https://wordpress.org/plugins/\" . sanitize_title($name) . \"/\";\n  $desc = rtrim($plugin[\"Description\"], \".\");\n\n  echo \"- [$name]($url) - $desc\\n\";\n}\n'\n"
  },
  {
    "path": "internet/wordpress_posts_without_category_tags.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-01-23 20:15:52 +0000 (Tue, 23 Jan 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each Wordpress post (article), checks its Categories and if a corresponding Tag of the same slug exists but isn't assigned to the post, prints it\n\nUseful check if you want to have matching tags for a tag cloud that are aligned with your categories\n\nUses wordpress_api.sh - see its --help for details on what environment variables need to be set:\n\n        $srcdir/wordpress_api.sh --help\n\nLimitation: doesn't iterate more than 100 categories or tags, extend code if you really have that many to process\n\nRequires curl to be installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nno_more_args \"$@\"\n\napi=\"$srcdir/wordpress_api.sh\"\n\n# XXX: not bothered iterating categories or tags - expand this if you have more than 100 categories or tags\ntimestamp \"getting list of available categories\"\ncategory_map=\"$(\"$api\" /categories?per_page=100 | jq -r '.[] | [.id, .slug] | @tsv')\"\ntimestamp \"getting list of available tags\"\ntag_map=\"$(\"$api\" /tags?per_page=100 | jq -r '.[] | [.id, .slug] | @tsv')\"\n\nfor((page=1;; page++)); do\n    if [ \"$page\" -gt 100 ]; then\n        die \"Hit over 100 pages of teams, possible infinite loop, exiting...\"\n    fi\n    if [ -z \"${QUIET:-}\" ]; then\n        timestamp \"getting list of posts: page $page\"\n    fi\n    data=\"$(\"$api\" \"/posts?per_page=100&page=$page\" | jq_debug_pipe_dump)\"\n    if jq_is_empty_list <<< \"$data\"; then\n        break\n    fi\n    jq -r '.[] | [.id, .slug] | @tsv' <<< \"$data\" |\n    while read -r post_id post_slug; do\n        #if [ -z \"${QUIET:-}\" ]; then\n        #    timestamp \"checking post '$post_slug'\"\n        #fi\n        category_ids=\"$(jq -r \".[] | select(.id == $post_id) | .categories[]\" <<< \"$data\")\"\n        tag_ids=\"$(jq -r \".[] | select(.id == $post_id) | .tags[]\" <<< \"$data\")\"\n        #new_tags=\"$tag_ids\"\n        for category_id in $category_ids; do\n            category_slug=\"$(awk \"/^${category_id}[[:space:]]/{print \\$2}\" <<< \"$category_map\")\"\n            tag_slug=\"$category_slug\"\n            tag_id=\"$(awk \"/[[:space:]]$tag_slug$/{print \\$1}\" <<< \"$tag_map\" || :)\"\n            [ -z \"$tag_id\" ] && continue\n            if ! grep -Fxq \"$tag_id\" <<< \"$tag_ids\"; then\n                timestamp \"post '$post_slug' is missing tag '$tag_slug'\"\n                #new_tags+=$'\\n'\"$tag_id\"\n                #new_tags_array=\"[ $(tr '\\n' ',' <<< \"$new_tags\" | sed 's/,$//') ]\"\n                #timestamp \"adding post '$post_slug' tag '$tag_slug': $new_tags_array\"\n                #data=\"$(\"$api\" \"/posts/$post_id\" -X POST -d \"{ \\\"tags\\\": $new_tags_array }\" | jq_debug_pipe_dump)\"\n            fi\n        done\n        #post_data=\"$(\"$api\" / | jq_debug_pipe_dump)\"\n    done\n    if jq -e 'length < 100' <<< \"$data\" >/dev/null; then\n        break\n    fi\ndone\n"
  },
  {
    "path": "ipaas/make_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-06-10 22:44:21 +0100 (Sat, 10 Jun 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Make.com API\n\nAutomatically handles authentication via environment variable \\$MAKE_API_KEY\nMust also set the \\$MAKE_ZONE eg. 'eu1'\n\nReplaces {organizationId} in the URL with \\$MAKE_ORGANIZATION_ID if set, otherwise queries the API and uses the first organization returned by the API\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up your API token here:\n\n    https://eu1.make.com/user/api\n\n\nAPI Reference:\n\n    https://www.make.com/en/api-documentation\n\n\nExamples:\n\nSome of these may not work on the free plan, see this issue:\n\n    https://community.make.com/t/erro-403-permission-denied-forbidden-to-use-token-authorization-for-this-organization/9946\n\n\nPing the API:\n\n    ${0##*/} /ping\n\nGet currently authenticated user:\n\n    ${0##*/} /users/me | jq .\n\nList users / teams / organizations / license:\n\n    # 403 - VPN access only\n    #${0##*/} /admin/users\n    #${0##*/} /admin/teams\n    #${0##*/} /admin/organizations\n    #${0##*/} /admin/system-settings/default-license\n\nList Connections:\n\n    ${0##*/} /connections?teamId=...\n\nList Data Stores:\n\n    ${0##*/} /data-stores?teamId=...\n\nList Hooks:\n\n    ${0##*/} /hooks?teamId=...\n\nList Notifications:\n\n    ${0##*/} /notifications | jq .\n\nList User's Organizations (shows API limit here):\n\n    ${0##*/} /organizations | jq .\n\nGet Organization Details:\n\n    ${0##*/} /organizations/{organizationId}\n\nList Scenarios:\n\n    ${0##*/} /scenarios?teamId=...\n\nList Teams:\n\n    ${0##*/} /teams?organizationId={organizationId}\n\nList Templates:\n\n    ${0##*/} /templates\n\nList Users:\n\n    ${0##*/} /users?organizationId={organizationId} | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined MAKE_API_KEY\ncheck_env_defined MAKE_ZONE\n\nurl_base=\"https://$MAKE_ZONE.make.com/api/v2\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path//https:\\\\/\\\\/*.make.com\\/api/v2}\"\nurl_path=\"${url_path##/}\"\n\nexport CURL_AUTH_HEADER=\"Authorization: Token\"\nexport TOKEN=\"$MAKE_API_KEY\"\n\nif [[ \"$url_path\" =~ {organizationId} ]]; then\n    if [ -z \"${MAKE_ORGANIZATION_ID:-}\" ]; then\n        MAKE_ORGANIZATION_ID=\"$(\"$0\" /organizations | jq -Mr '.organizations[0].id')\"\n    fi\n    url_path=\"${url_path/\\{organizationId\\}/$MAKE_ORGANIZATION_ID}\"\nfi\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "java/bytecode_viewer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 21:02:58 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWrapper to download and run the Bytecode-Viewer command line java decompiler\n\nExamples:\n\n    ${0##*/} Main.class\n\n    ${0##*/} myapp.jar\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jar_or_java.class>\"\n\n# let see the help from tool instead\n#help_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbytecode_viewer_jar=\"$srcdir/Bytecode-Viewer.jar\"\n\nif ! [ -f \"$bytecode_viewer_jar\" ]; then\n    pushd \"$srcdir\"\n    ../install/download_bytecode_viewer_jar.sh\n    timestamp\n    echo -n \"Symlinking: \" >&2\n    ln -sv Bytecode-Viewer-*.jar \"${bytecode_viewer_jar##*/}\"\n    popd\nfi\n\njava -jar \"$bytecode_viewer_jar\" \"$@\"\n"
  },
  {
    "path": "java/cfr.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 21:02:58 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWrapper to download and run the CFR command line java decompiler\n\nExamples:\n\n    ${0##*/} Main.class > Main.java\n\n    ${0##*/} myapp.jar\n\n    ${0##*/} myapp.jar --outputdir ./myapp-java-code/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jar_or_java.class>\"\n\n# let see the help from tool instead\n#help_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncfr_jar=\"$srcdir/cfr.jar\"\n\nif ! [ -f \"$cfr_jar\" ]; then\n    pushd \"$srcdir\"\n    ../install/download_cfr_jar.sh\n    timestamp\n    echo -n \"Symlinking: \" >&2\n    ln -sv cfr-*.jar \"${cfr_jar##*/}\"\n    popd\nfi\n\njava -jar \"$cfr_jar\" \"$@\"\n"
  },
  {
    "path": "java/java_decompile_jar.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 21:07:22 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDecompiles a Java jar\n\nUnpacks it in /tmp, finds the Main-Class from the META-INF/MANIFEST.MF\nand runs the Java decompiler JD GUI on its main .class file\n\nUses adjacent script ./jd_gui.sh\nwhich uses ../install/download_jd_gui_jar.sh if the JD GUI jar is not already available\n\nRequire Java to already be installed and in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jar>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njar=\"$1\"\n\nreadlink(){\n    if is_mac; then\n        command greadlink \"$@\"\n    else\n        command readlink \"$@\"\n    fi\n}\n\nif ! [[ \"$jar\" =~ \\.jar$ ]]; then\n    die \"ERROR: arg given is not a JAR file: $jar\"\nfi\n\nif ! [[ -f \"$jar\" ]]; then\n    die \"ERROR: jar file not found: $jar\"\nfi\n\njar=\"$(readlink -f \"$jar\")\"\nif is_mac; then\n    jar=\"${jar/\\/private\\/tmp/\\/tmp}\"\nfi\n\nif [ \"$jar\" != \"/tmp/${jar##*/}\" ]; then\n    timestamp \"Copying JAR to /tmp:\"\n    echo >&2\n    unalias cp >&/dev/null || :\n    cp -fv \"$jar\" /tmp/\n    echo >&2\nfi\n\ncd /tmp/\n\njar=\"${jar##*/}\"\n\ntimestamp \"Unpacking JAR\"\n#echo >&2\ntar zxf \"$jar\"\n#echo >&2\necho >&2\n\ntimestamp \"Determining main class\"\nmain_class=\"$(awk '/Main-Class:/{print $2}' META-INF/MANIFEST.MF)\"\necho >&2\nif is_blank \"$main_class\"; then\n    die \"ERROR: failed to find Main-Class from unpacked jar's META-INF/MANIFEST.MF\"\nfi\ntimestamp \"Determined main class to be: $main_class\"\necho >&2\n\nmain_class_file=\"${main_class//./\\/}\"\n\ntimestamp \"Running decompiler on main class: $main_class_file\"\necho >&2\n\"$srcdir/jd_gui.sh\" \"$main_class_file\"\n"
  },
  {
    "path": "java/java_show_classpath.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2178,SC2001,SC2128,SC2179\n#\n#  Author: Hari Sekhon\n#  Date: 2024-07-23 20:43:59 +0300 (Tue, 23 Jul 2024)\n#  Ported from Perl version\n#  Original Date: 2013-02-11 11:50:00 +0000 (Mon, 11 Feb 2013)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints all the command line classpaths of Java processes\n\nOptionally filter Java processes by a giving a regex\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<command_process_regex_ere_format>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n# ERE format for [[\ncommand_regex=\"${1:-.*}\"\n\nif ! is_regex \"$command_regex\"; then\n    echo \"Invalid command regex supplied: $command_regex\"\n    exit 1\nfi\n\n# use GNU ps on Mac to maintain compatibility\n# doesn't work in subshells\n#if is_mac; then\n#    ps(){\n#        gps \"$@\"\n#    }\n#    export -f ps\n#fi\n\nis_jps_output(){\n    # JPS output format\n    [[ \"$1\" =~ ^([[:digit:]]+)[[:space:]]+([[:alnum:]]*)$ ]]\n}\n\n#replace_classpath_with_token(){\n#    sed 's/[[:space:]]-\\(cp\\|classpath\\)[[:space:]=]\\+.\\+[[:space:]]/ <CLASSPATHS> /' <<< \"$1\"\n#}\n\nshow_cli_classpath(){\n    local cmd=\"$1\"\n    [[ \"$cmd\" =~ java ]] || return\n    args=\"${cmd#*java }\"\n    #cmd=\"$(replace_classpath_with_token \"$cmd\")\"\n    echo\n    echo \"command:  $cmd\"\n    echo\n    count=0\n    # XXX: will break upon directories in classpath containing a space, but this should be rare on unix systems\n    classpaths=\"$(grep -Eo -- '-(cp|classpath)[=[:space:]]+(.+)[[:space:]]' <<< \"$args\" || die \"Failed to find classpath in args '$args'\")\"\n    classpaths=\"${classpaths## }\"\n    classpaths=\"${classpaths##-cp}\"\n    classpaths=\"${classpaths##-classpath}\"\n    classpaths=\"${classpaths##=}\"\n    IFS=':' read -r -a classpaths <<< \"$classpaths\"\n    for classpath in \"${classpaths[@]}\"; do\n        [ -z \"$classpath\" ] && continue\n        echo \"classpath:  $classpath\"\n        count=$((count + 1))\n    done\n    [ $count -gt 0 ] && echo\n    echo \"$count classpath(s) found\"\n    echo\n}\n\nshow_jinfo_classpath(){\n    local cmd=\"$1\"\n    if is_jps_output \"$cmd\"; then\n        #pid=\"${cmd%% *}\" # first token is pid\n        pid=\"${BASH_REMATCH[1]}\"\n        # skip Jps itself which may be lingering in the process list from our call before this function\n        [ \"${BASH_REMATCH[2]:-}\" = Jps ] && return\n        if [ -z \"${BASH_REMATCH[2]:-}\" ]; then\n            cmd+=\" <embedded JVM no classname from JPS output>\"\n        fi\n        echo \"JPS:     $cmd\"\n        echo \"command: $(ps -f -p \"$pid\" | tail -n +2 | sed 's/^[[:space:]]*//')\"\n        echo\n    else\n        if ! [[ \"$cmd\" =~ java ]]; then\n            # shellcheck disable=SC2028\n            echo \"skipping $cmd since it doesn't match regex 'java'\"\n            return\n        fi\n        #cmd=\"$(replace_classpath_with_token \"$cmd\")\"\n        echo\n        echo \"command:  $cmd\"\n        if [[ \"$cmd\" =~ ^[[:space:]]*[[:alnum:]]+[[:space:]]+[[:digit:]]+[[:space:]]+ ]]; then\n            echo \"ps -ef input detected\"\n        elif [[ \"$cmd\" =~ ^[[:space:]]*[[:digit:]]+[[:space:]]+[[:alnum:]]+[[:space:]]+ ]]; then\n            echo \"ps aux input detected\"\n        else\n            echo \"Invalid input to show_jinfo_classpath, expecting '<pid> <classname>' or '<pid> <user> <cmd>' or 'ps -ef' or 'ps aux' input\"\n            return\n        fi\n        pid=\"$(grep -Eo '^[[:digit:]]+' <<< \"$cmd\")\"\n    fi\n    output=\"$(jinfo \"$pid\")\"\n    found_classpath=0\n    while IFS= read -r line; do\n        # breaks on this content:\n        #\n        #   /Users/hari/Library/Application Support/JetBrains/IdeaIC2023.3/plugins/sonarlint-intellij/sloop/lib/error_prone_annotations-2.18.0.jar\n        #\n        #if [[ \"$line\" =~ error ]]; then\n        #    echo \"jinfo error detected attaching to process id $pid\"\n        #    echo \"$line\"\n        #    return\n        #fi\n        if [[ \"$line\" =~ ^java.class.path[[:space:]]*=[[:space:]]* ]]; then\n            line=\"${line#*=}\"\n            line=\"${line# }\"\n            count=0\n            # jinfo escapes the colon separators with backslashes\n            line=\"$(sed 's/\\\\:/:/g' <<< \"$line\")\"\n            IFS=':' read -r -a classpaths <<< \"$line\"\n            for classpath in \"${classpaths[@]}\"; do\n                [ -z \"$classpath\" ] && continue\n                echo \"classpath:  $classpath\"\n                count=$((count + 1))\n            done\n            [ $count -gt 0 ] && echo\n            echo \"$count classpath(s) found\"\n            echo\n            found_classpath=1\n            break\n        fi\n    done <<< \"$output\"\n    if [ $found_classpath -eq 0 ]; then\n        echo \"Failed to find java classpath in output from jinfo!\"\n    fi\n    hr\n}\n\nif ! process_list=\"$(jps 2>/dev/null)\"; then\n    echo \"WARNING: jps failed, perhaps not in \\$PATH? (\\$PATH = $PATH)\" >&2\n    echo \"WARNING: Falling back to ps command, may be less accurate\" >&2\n    process_list=\"$(ps -e -o pid,user,command)\"\nfi\n\nwhile IFS= read -r line; do\n    log \"input: $line\"\n    if is_jps_output \"$line\"; then\n        if [ -n \"$command_regex\" ]; then\n            if [[ \"$line\" =~ $command_regex ]]; then\n                show_jinfo_classpath \"$line\"\n            fi\n        else\n            show_jinfo_classpath \"$line\"\n        fi\n    elif [[ \"$line\" =~ java[[:space:]].*$command_regex ]]; then\n        #show_jinfo_classpath \"$line\"\n        show_cli_classpath \"$line\"\n    fi\ndone <<< \"$process_list\"\n"
  },
  {
    "path": "java/jd_gui.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 21:02:58 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWrapper to download and run Java Decompiler GUI\n\nExamples:\n\n    ${0##*/} myapp.jar\n\n    ${0##*/} Main.class\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<java.class>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njd_gui_jar=\"$srcdir/jd-gui-min.jar\"\n\nif ! [ -f \"$jd_gui_jar\" ]; then\n    pushd \"$srcdir\"\n    ../install/download_jd_gui_jar.sh\n    timestamp\n    echo -n \"Symlinking: \" >&2\n    ln -sv jd-gui-*-min.jar \"${jd_gui_jar##*/}\"\n    popd\nfi\n\njava -jar \"$jd_gui_jar\" \"$@\"\n"
  },
  {
    "path": "java/jvm_heaps.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-01 16:54:55 +0000 (Fri, 01 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Shows the Java heaps for all Java processes\n#\n# Sun JDK 8, OpenJDK 11 and IBM JDK all treat the last -Xmx on the command line as the actual one, so we are going with that\n#\n# You can check this is true on your specific implementation like so:\n#\n# java -version; java -Xmx1G -XX:+PrintFlagsFinal -Xmx2G 2>/dev/null | grep MaxHeapSize\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    # numfmt seems to only work on capitalized unit suffixes\n    numfmt(){ gnumfmt \"$@\"; }\nfi\n\n# jps only available with JDK, not JRE\n# picks up other things like Adobe crash reporter with lots of java related env vars - which is wrong, we only care about CLI args to java procs\n# shellcheck disable=SC2009\n#pgrep -l -f java |\nps -ef |\ngrep java |\ngrep -v -e '[[:space:][:alpha:]]grep[[:space:]]' -e 'sed[[:space:]]' |\nsed 's/.*[[:space:]]-Xmx/-Xmx/' |\nsed 's/[[:space:]].*[[:space:]]\\([[:alnum:].]\\)/ \\1/' |\ncolumn -t\n"
  },
  {
    "path": "java/jvm_heaps_total_mb.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-01 16:54:55 +0000 (Fri, 01 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Calculates the total combined RAM in MB allocated to all Java heap sizes on all JVM processes running on the local machine\n#\n# Requires recent version of GNU coreutils to be installed for correct MB conversions (you might need to 'brew upgrade coreutils' on Mac)\n#\n# You should account for at least ~20% overhead on top of this to account for non-heap Java RAM usage eg. off-heap allocations, JIT optimizations, JNI code, GC (especially G1) etc...\n#\n# Sun JDK 8, OpenJDK 11 and IBM JDK all treat the last -Xmx on the command line as the actual one, so we are going with that\n#\n# You can check this is true on your specific implementation like so:\n#\n# java -version; java -Xmx1G -XX:+PrintFlagsFinal -Xmx2G 2>/dev/null | grep MaxHeapSize\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    # numfmt seems to only work on capitalized unit suffixes\n    numfmt(){ gnumfmt \"$@\"; }\nfi\n\n# picks up other things like Adobe crash reporter with lots of java related env vars - which is wrong, we only care about CLI args to java procs\n# shellcheck disable=SC2009\n#pgrep -l -f java |\nps -ef |\ngrep java |\ngrep -v -e '[[:space:][:alpha:]]grep[[:space:]]' -e 'sed[[:space:]]' |\nsed 's/.*[[:space:]]-Xmx/-Xmx/' |\ngrep --color=no -o -- '-Xmx[^[:space:]]*' |\nsed 's/-Xmx//' |\nnumfmt --from=iec |\nawk '{sum += $1} END {print sum}' |\nnumfmt --to-unit=Mi # newers versions (eg. 8.30) need Mi for correct unit conversion,\n                    # but older versions (eg. 8.23) crash when given Mi as target unit\n                    # eg.  'Abort trap: 6' - if this happens to you, 'brew upgrade coreutils' to resolve it\n"
  },
  {
    "path": "java/procyon.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 23:03:58 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWrapper to download and run the Procyon command line java decompiler\n\nExamples:\n\n    ${0##*/} Main.class > Main.java\n\n    ${0##*/} myapp.jar\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jar_or_java.class>\"\n\n# let see the help from tool instead\n#help_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nprocyon_jar=\"$srcdir/procyon-decompiler.jar\"\n\nif ! [ -f \"$procyon_jar\" ]; then\n    pushd \"$srcdir\"\n    ../install/download_procyon_jar.sh\n    timestamp\n    echo -n \"Symlinking: \" >&2\n    ln -sv procyon-*.jar \"${procyon_jar##*/}\"\n    popd\nfi\n\njava -jar \"$procyon_jar\" \"$@\"\n"
  },
  {
    "path": "jenkins/README.md",
    "content": "# Jenkins scripts\n\nThe Bash scripts can execute from the git clone, but the `*.groovy` scripts needs to be pasted into the Jenkins Console at `$JENKINS_URL/script` which you can browse to:\n\nJenkins -> Manage Jenkins -> Script Console\n"
  },
  {
    "path": "jenkins/jenkins.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-28 11:49:22 +0000 (Sat, 28 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Start a quick local Concourse CI\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nBoots Jenkins CI in Docker, and builds the current repo\n\n- boots Jenkins container in Docker\n- installs plugins\n- prints admin credentials\n- creates job from \\$PWD/setup/jenkins-job.xml\n  - using pipeline from \\$PWD/Jenkinsfile\n- enables job\n- builds job\n- prints Jenkins UI URL and opens it in browser\n\n    ${0##*/} [up]\n\n    ${0##*/} down\n\n    ${0##*/} ui     - prints the Jenkins Server URL and automatically opens in browser\n\nIdempotent, you can re-run this and continue from any stage\n\nSee Also:\n\n    jenkins_cli.sh - this script makes heavy use of it to handle API calls with authentication as part of the setup\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nexport JENKINS_HOST=\"${DOCKER_HOST:-localhost}\"\nexport JENKINS_PORT=8080\n\nexport JENKINS_URL=\"http://$JENKINS_HOST:$JENKINS_PORT\"\ncli=\"$srcdir/jenkins_cli.sh\"\n\n#repo=\"${PWD##*/}\"\n#git_repo=\"$(git remote -v | grep github.com | sed 's/.*github.com/https:\\/\\/github.com/; s/ .*//')\"\ngit_repo=\"$(git_repo)\"\nrepo=\"${git_repo##*/}\"\njob=\"$repo\"\njob_xml=\"setup/jenkins-job.xml\"\n\nJenkinsfile=Jenkinsfile\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/jenkins.yml\"\n\nplugins_txt=\"$srcdir/../setup/jenkins-plugins.txt\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nprint_creds(){\n    password=\"$(\"$srcdir/jenkins_password.sh\" || :)\"\n\n    if [ -n \"$password\" ]; then\n    cat <<EOF\n\nJenkins Login username:  admin\nJenkins Login password:  $password\n\nEOF\n    fi\n}\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting Jenkins:\"\n    docker-compose up -d \"$@\"\n    echo >&2\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"Jenkins URL:  $JENKINS_URL\"\n    print_creds\n    open \"$JENKINS_URL\"\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo >&2\n    exit 0\nfi\n\nwhen_jenkins_up(){\n    when_url_content 90 \"$JENKINS_URL/login\" '(?i:jenkins|hudson)'\n    echo\n}\n\nwhen_jenkins_up\n\ntimestamp \"Installing plugins\"\n# would be slow to do this via jenkins-cli\n# sed 's/#.*//; s/:.*//; /^[[:space:]]*$/d' setup/jenkins-plugins.txt | while read plugin; do jenkins_cli.sh install-plugin \"$plugin\"; done\n#\n# Old: deprecated, but still more portable across existing versions\ndocker-compose exec -T jenkins-server /usr/local/bin/install-plugins.sh < \"$plugins_txt\" ||\n# New: later switch to\n{\n    docker-compose cp \"$plugins_txt\" jenkins-server:/ &&\n    docker-compose exec -T jenkins-server /bin/jenkins-plugin-cli -f \"${plugins_txt##*/}\"\n} || :  # exits 1 warning about plugins already existing\necho\n\n# if this fails then the the CLI commands will also fail because they use a similar mechanism to find the admin password from the container to authenticate with the Jenkins API\nprint_creds\n\nif ! \"$cli\" list-plugins | grep -q .; then\n    timestamp \"Restarting Jenkins to pick up plugins:\"\n    #\"$cli\" restart\n    #when_ports_down 300 \"$JENKINS_HOST\" \"$JENKINS_PORT\"\n    docker-compose restart jenkins-server \"$@\"\n\n    when_jenkins_up\nfi\n\nSECONDS=0\nwhile [ \"$SECONDS\" -lt 300 ]; do\n    if \"$cli\" list-plugins | grep -q .; then\n        echo\n        break\n    fi\n    timestamp \"waiting for Jenkins to finish initializing and list plugins\"\n    sleep 1\ndone\n\nfor filename in \"$Jenkinsfile\" \"$job_xml\"; do\n    if ! [ -f \"$filename\" ]; then\n        timestamp \"Jenkins configuration file '$filename' not found\"\n        echo >&2\n        timestamp \"Skipping loading missing pipeline\"\n        echo >&2\n        timestamp \"Re-run this from the root directory of a repo containing '$Jenkinsfile' and '$job_xml' to auto-load a pipeline\"\n        exit 0\n    fi\ndone\n\n# XXX: this fails if the plugins haven't been loaded properly\n# XXX: you'll also get this error trying to run the job's pipeline:\n#      java.lang.NoSuchMethodError: No such DSL method 'pipeline' found among steps\ntimestamp \"Validating Jenkinsfile\"\n\"$cli\" declarative-linter < \"$Jenkinsfile\"\necho\n\ntimestamp \"Creating / Updating job - $job:\"\nif \"$cli\" list-jobs | grep -q \"^$job$\"; then\n    timestamp \"job already exists, updating...\"\n    \"$cli\" update-job \"$job\" < \"$job_xml\"\nelse\n    timestamp \"job does not exist, creating...\"\n    \"$cli\" create-job \"$job\" < \"$job_xml\"\nfi\necho\n\ntimestamp \"Enabling job - $job:\"\n\"$cli\" enable-job \"$job\"\necho\n\n# -f waits for build\n# -v prints build contents\ntimestamp \"Building job - $job and tailing output:\"\n\"$cli\" build -f -v \"$job\"\n\n#echo \"Tailing last build:\"\n#\"$cli\" console \"$job\" -f\n"
  },
  {
    "path": "jenkins/jenkins_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-01 19:08:00 +0100 (Tue, 01 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Jenkins Rest API\n\nHandles obtaining the CRSF protection token Jenkins-Crumb for you\n\nCan specify \\$CURL_OPTS for options to pass to curl, or pass them as arguments to the script\n\nRequires either \\$JENKINS_URL or \\$JENKINS_HOST + \\$JENKINS_PORT which defaults to localhost and port 8080\n\nIf you require SSL, specify full \\$JENKINS_URL\n\nAutomatically handles authentication via environment variables:\n\n- \\$JENKINS_USER_ID / \\$JENKINS_USERNAME / \\$JENKINS_USER\n- \\$JENKINS_API_TOKEN / \\$JENKINS_TOKEN  / \\$JENKINS_PASSWORD\n\nIf using JENKINS_PASSWORD, obtains the Jenkins-Crumb cookie from a pre-request\n\nOn Jenkins 2.176.2 onwards, you must set JENKINS_TOKEN instead of using a password, see\n\n    https://www.jenkins.io/doc/upgrade-guide/2.176/#SECURITY-626\n\nExample:\n\n    # List Credentials in the Global System Provider Store:\n\n        ${0##*/} '/credentials/store/system/domain/_/api/json?tree=credentials\\\\[id\\\\]' | jq .\n\n    # Retrieve the config of a given credential by its ID, in this case 'haritest':\n\n        ${0##*/} '/credentials/store/system/domain/_/credential/haritest/config.xml'\n\n    # Delete a Jenkins job/pipeline called 'test':\n\n        ${0##*/} /job/test/ -X DELETE -i\n\n    # See many adjacent jenkins_*.sh scripts that use this script to make many common actions easier\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncurl_api_opts \"$@\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl_path=\"${1:-}\"\nurl_path=\"${url_path##/}\"\n\nshift || :\n\nJENKINS_URL=\"${JENKINS_URL:-http://${JENKINS_HOST:-localhost}:${JENKINS_PORT:-8080}}\"\nshopt -s nocasematch\nif ! [[ \"$JENKINS_URL\" =~ https?:// ]]; then\n    JENKINS_URL=\"http://$JENKINS_URL\"\nfi\nshopt -u nocasematch\nJENKINS_URL=\"${JENKINS_URL%%/}\"\n\nexport USERNAME=\"${JENKINS_USER_ID:-${JENKINS_USERNAME:-${JENKINS_USER:-}}}\"\nJENKINS_TOKEN=\"${JENKINS_API_TOKEN:-${JENKINS_TOKEN:-}}\"\nif [ -n \"${JENKINS_TOKEN:-}\" ]; then\n    export PASSWORD=\"$JENKINS_TOKEN\"\nelse\n    export PASSWORD=\"${JENKINS_PASSWORD:-${JENKINS_TOKEN:-}}\"\n    crumb=\"$(\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"$JENKINS_URL/crumbIssuer/api/json\" | jq -r '.crumb')\"\n    CURL_OPTS+=(-H \"Jenkins-Crumb: $crumb\")\nfi\n\n\"$srcdir/../bin/curl_auth.sh\" \"$JENKINS_URL/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\"\n"
  },
  {
    "path": "jenkins/jenkins_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-30 10:42:58 +0100 (Thu, 30 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the latest builds and their statuses for all the Jenkins jobs/pipelines\n\nOutput Format:\n\n<timestamp>    <duration>    <build_number>    <result>    <name>\n\n\nTested on Jenkins 2.319\n\nUses jenkins_api.sh - see there for authentication and connection details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\njenkins_api.sh '/api/json?tree=jobs\\[name,lastBuild\\[number,timestamp,duration,building,result\\]\\]&pretty=true' |\njq -r '.jobs[] |\n        [\n            .lastBuild.timestamp,\n            .lastBuild.number,\n            .lastBuild.building,\n            .lastBuild.result,\n            .lastBuild.duration,\n            .name\n        ] | @tsv' |\nsort -r |\nwhile read -r timestamp_millis number building result duration_millis name; do\n    if ! [[ \"$timestamp_millis\" =~ ^[[:digit:]]+$ ]]; then\n        # not builds have been run for this pipeline yet, populate defaults to avoid errors\n        name=\"$timestamp_millis $number $building $result $duration_millis $name\"\n        timestamp_millis=0\n        number=\"0\"\n        result=\"N/A\"\n        duration_millis=0\n    fi\n    timestamp=$((timestamp_millis / 1000))\n    datestamp=\"$(date -d \"@$timestamp\" '+%FT%T')\"\n    if [ \"$building\" = true ]; then\n        result=\"BUILDING\"  # result is blank otherwise\n        name=\"$duration_millis $name\"  # this eats the first word of name if the prior column is blank\n        duration=\"$(($(date '+%s') - timestamp))\"\n    else\n        duration=\"$((duration_millis / 1000))\"\n    fi\n    duration=\"$(seconds_to_hours \"$duration\")\"\n    printf '%s\\t%12s\\t%8d\\t%-10s\\t%s\\n' \"$datestamp\" \"$duration\" \"$number\" \"$result\" \"$name\"\ndone\n"
  },
  {
    "path": "jenkins/jenkins_clear_build_history.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2021-04-08 19:16:17 +0100 (Thu, 08 Apr 2021)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\n// WARNING: you can run into all sorts of state problems with the Milestone plugin preventing your builds from running due to a remnant reference to a higher build number - this also happens on job restores which start from build #1 again\n//\n//\t\t\t[2022-08-02T16:14:46.501Z] Trying to pass milestone 0\n//\t\t\t[2022-08-02T16:14:46.501Z] Canceled since build #3814 already got here\n//\n//\t\t\thttps://issues.jenkins.io/browse/JENKINS-38641\n//\n//\t\t\tone fix is of course to do\n//\n//\t\t\t\tjob.nextBuildNumber = 3815\n\n// XXX: Edit this to the name of your job pipeline\nString jobName = \"My Dev Pipeline\"\n\n// Simple Job\n//def job = Jenkins.instance.getItem(jobName)\n//\n// XXX: must specify the branch for Multibranch Pipelines\n//def job = Jenkins.instance.getItemByFullName(\"$jobName/master\")\n//\n// but this method can also handle simple jobs so use it for everything\ndef job = Jenkins.instance.getItemByFullName(jobName)\n\n// deletes the previous build history\njob.getBuilds().each { it.delete() }\n\n// XXX: don't reset this build number as it can break the Milestone plugin for this job, see comment above for more details\n//job.nextBuildNumber = 1\n\njob.save()\n"
  },
  {
    "path": "jenkins/jenkins_clear_build_history_all_jobs.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2021-04-08 19:16:17 +0100 (Thu, 08 Apr 2021)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\n// XXX: WARNING: you can run into all sorts of state problems with the Milestone plugin preventing your builds from running due to a remnant reference to a higher build number - this also happens on job restores which start from build #1 again\n//\n//\t\t\t[2022-08-02T16:14:46.501Z] Trying to pass milestone 0\n//\t\t\t[2022-08-02T16:14:46.501Z] Canceled since build #3814 already got here\n//\n//\t\t\thttps://issues.jenkins.io/browse/JENKINS-38641\n//\n//\t\t\tone fix is of course to do\n//\n//\t\t\t\tjob.nextBuildNumber = 3815\n//\n//\t\t\tbut you'd either need to do that for only one specific job (see adjacent script jenkins_clear_build_history.groovy) or you'd need to set the build number to something higher than any of the jobs got to\n\njenkins.model.Jenkins.instance.items.findAll().each {\n\tprintln\n\tdef jobName = it.name\n\tdef job = Jenkins.instance.getItem(jobName)\n\tjob.getBuilds().each { it.delete() }\n\tjob.nextBuildNumber = 1\n\tjob.save()\n}\n"
  },
  {
    "path": "jenkins/jenkins_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: help\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-28 12:56:30 +0000 (Sat, 28 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWraps the Jenkins CLI and handles using various environment variables for connection and authentication\n\nDownloads the jenkins-cli.jar from the \\$JENKINS_URL on first run\n\nConfiguration Environment Variables:\n\n(in order of descending precedence and matches jenkins-cli.jar)\n\n    JENKINS_URL  - https://jenkins.domain.com\n        or\n    JENKINS_HOST\n    JENKINS_PORT\n    JENKINS_SSL     (any value enables SSL)\n\n    JENKINS_USER_ID\n    JENKINS_TOKEN\n    (tries a username of admin if neither of the above two variables are set)\n\n    JENKINS_API_TOKEN\n    JENKINS_TOKEN\n    JENKINS_PASSWORD\n    (will attempt to find the password via jenkins_password.sh as a last ditch attempt if not set)\n\n    JENKINS_CLI_ARGS    (optional - any switches you want passed to the jenkins-cli.jar)\n\nIMPORTANT: if Jenkins is behind a reverse proxy such as Kubernetes Ingress, you will probably need to add the '-webSocket' argument, otherwise it Jenkins CLI will hang\n\n      Tip: set this in JENKINS_CLI_ARGS to not have to specify it every time\n\nUsage:\n\n    jenkins_cli.sh --help\n\n\nEverything else is passed directly to jenkins-cli.jar\n\n\n Examples:\n\n   # See all Jenkins CLI options:\n\n       jenkins_cli.sh help\n\n   # Get Jenkins version:\n\n       jenkins_cli.sh version\n\n   # Show your authenticated user:\n\n       jenkins_cli.sh who-am-i\n\n   # List Plugins:\n\n       jenkins_cli.sh list-plugins\n\n   # List Jobs (Pipelines):\n\n       jenkins_cli.sh list-jobs\n\n   # List Credential Providers:\n\n       jenkins_cli.sh list-credentials-providers\n\n   # List Credentials in the default global in-built credential store:\n\n       jenkins_cli.sh list-credentials system::system::jenkins\n\n   # Dump all Credentials configurations in the default in-build credentials store in XML format:\n\n       jenkins_cli.sh list-credentials-as-xml system::system::jenkins\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# could set this to DNS CNAME 'jenkins', but often these says we run in docker\nDEFAULT_JENKINS_HOST=localhost\nDEFAULT_JENKINS_PORT=8080\n\njava=\"${JAVA:-java}\"\njar=\"$srcdir/jenkins-cli.jar\"\n\n# shellcheck disable=SC2230\nif ! which \"$java\" &>/dev/null; then\n    echo \"$java not found in PATH (\\$PATH)\"\n    exit 1\nfi\n\nif [ -z \"${JENKINS_URL:-}\" ]; then\n    host=\"${JENKINS_HOST:-${HOST:-$DEFAULT_JENKINS_HOST}}\"\n    port=\"${JENKINS_PORT:-${PORT:-$DEFAULT_JENKINS_PORT}}\"\n    http=\"http\"\n    if [ -n \"${JENKINS_SSL:-}\" ]; then\n        http=\"https\"\n    fi\n    export JENKINS_URL=\"$http://$host:$port\"\nfi\n\nJENKINS_USER=\"${JENKINS_USER_ID:-${JENKINS_USER:-admin}}\"\n\nJENKINS_PASSWORD=\"${JENKINS_API_TOKEN:-${JENKINS_TOKEN:-${JENKINS_PASSWORD:-}}}\"\nif [ -z \"${JENKINS_PASSWORD:-}\" ]; then\n    JENKINS_PASSWORD=\"$(\"$srcdir/jenkins_password.sh\" || :)\"\nfi\n\nif ! [ -s \"$jar\" ]; then\n    if type -P wget &>/dev/null; then\n        wget -cO \"$jar\" \"$JENKINS_URL/jnlpJars/jenkins-cli.jar\"\n    else\n        curl -sSf >\"$jar\" \"$JENKINS_URL/jnlpJars/jenkins-cli.jar\"\n    fi\nfi\n\n# -s \"$JENKINS_URL\" is implicit\n# cannot load jenkins job from stdin if doing this\n#java -jar \"$jar\" -auth @/dev/fd/0 \"$@\" <<< \"$JENKINS_USER:$JENKINS_PASSWORD\"\n#java -jar \"$jar\" -auth \"$JENKINS_USER:$JENKINS_PASSWORD\" \"$@\"\n\n# want splitting\n# shellcheck disable=SC2086\njava -jar \"$jar\" -s \"$JENKINS_URL\" -auth @<(cat <<< \"$JENKINS_USER:$JENKINS_PASSWORD\") ${JENKINS_CLI_ARGS:-} \"$@\"\n"
  },
  {
    "path": "jenkins/jenkins_count_jobs.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2021-04-08 19:16:17 +0100 (Thu, 08 Apr 2021)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\njenkins.model.Jenkins.instance.items.findAll().size\n"
  },
  {
    "path": "jenkins/jenkins_create_job_check_gcp_serviceaccount.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-07 18:12:00 +0000 (Wed, 07 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\njob_name='check-gcp-serviceaccount'\njob_xml=\"$srcdir/../setup/jenkins-job-check-gcp-serviceaccount.xml\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a freestyle test job which runs a GCP Metadata query to determine the GCP serviceaccount the agent pod is operating under   to check GKE Workload Identity integration\n\nTriggers the job immediately to get the serviceaccount result\n\nThe job configuration where this is specified is in:\n\n    $job_xml\n\nUses the adjacent script jenkins_cli.sh\n\nJenkins authentication token and environment variables must be set - see jenkins_cli.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport JENKINS_OVERWRITE_JOB=1\n\n\"$srcdir/jenkins_create_run_job.sh\" \"$job_name\" \"$job_xml\"\n"
  },
  {
    "path": "jenkins/jenkins_create_job_parallel_test_runs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-07 18:12:00 +0000 (Wed, 07 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\njob_name='check-sleep-parallel'\njob_xml=\"$srcdir/../setup/jenkins-job-sleep-parallel-parameterized.xml\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a freestyle parameterized test sleep job and launches N parallel runs of it to test scaling and parallelization of Jenkins on Kubernetes agents\n\nRuns 10 jobs by default which run for 10 minutes each\n\nThe job configuration where this is specified is in:\n\n    $job_xml\n\nUses the adjacent script jenkins_cli.sh\n\nJenkins authentication token and environment variables must be set - see jenkins_cli.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<number_of_job_runs_to_launch>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nnum_jobs=\"${1:-10}\"\n\nif ! is_int \"$num_jobs\"; then\n    usage \"argument must be an integer\"\nelif [ \"$num_jobs\" -gt 100 ]; then\n    usage \"argument is greater than 100, this seems like a costly mistake. Edit the protection in this script if you really want to do this\"\nfi\n\nif ! [ -f \"$job_xml\" ]; then\n    die \"Job config file not found: $job_xml\"\nfi\n\ntimestamp \"Checking if job '$job_name' already exists'\"\nif jenkins_cli.sh list-jobs | grep -q \"^$job_name$\"; then\n    timestamp \"Job '$job_name' already exists, skipping creation\"\nelse\n    timestamp \"Job '$job_name' does not exist yet, creating...\"\n    \"$srcdir/jenkins_cli.sh\" create-job \"$job_name\" < \"$job_xml\"\nfi\n\nfor ((i=1; i <= \"$num_jobs\"; i++)); do\n    timestamp \"Launching job $i\"\n    \"$srcdir/jenkins_cli.sh\" build \"$job_name\" -p \"UNIQUE_VALUE=Run $i\"\ndone\n"
  },
  {
    "path": "jenkins/jenkins_create_run_job.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: check-gcp-serviceaccount ../setup/jenkins-job-check-gcp-serviceaccount.xml\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-07 18:12:00 +0000 (Wed, 07 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins job from a given xml file and triggers it to run immediately\n\nIf the job already exists, skips creation\nIf the job already exists and the enviroment variable JENKINS_OVERWRITE_JOB is set to any value it updates the job definition\n\nUses the adjacent script jenkins_cli.sh\n\nJenkins authentication token and environment variables must be set - see jenkins_cli.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> <job_xml_file> [<jenkins_cli_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\njob_name=\"$1\"\njob_xml=\"$2\"\nshift || :\nshift || :\n\nif ! [ -f \"$job_xml\" ]; then\n    die \"Job config file not found: $job_xml\"\nfi\n\ntimestamp \"Checking if job '$job_name' already exists'\"\nif jenkins_cli.sh list-jobs | grep -q \"^$job_name$\"; then\n    if [ -n \"${JENKINS_OVERWRITE_JOB:-}\" ]; then\n        timestamp \"Job '$job_name' already exists, updating to ensure latest version\"\n        \"$srcdir/jenkins_cli.sh\" update-job \"$job_name\" < \"$job_xml\"\n    else\n        timestamp \"Job '$job_name' already exists, skipping creation\"\n    fi\nelse\n    timestamp \"Job '$job_name' does not exist yet, creating...\"\n    \"$srcdir/jenkins_cli.sh\" create-job \"$job_name\" < \"$job_xml\"\nfi\n\ntimestamp \"Triggering job '$job_name'\"\n\"$srcdir/jenkins_cli.sh\" build \"$job_name\" \"$@\"\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to create a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if ! [ -f \"$keystore\" ]; then\n        die \"keystore file '$keystore' not found!\"\n    fi\n    timestamp \"Reading keystore file '$keystore'\"\n    local keystore_contents\n    keystore_contents=\"$(base64 \"$keystore\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <keyStoreSource class=\\\"com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\\$UploadedKeyStoreSource\\\">\n    <uploadedKeystoreBytes>$keystore_contents</uploadedKeystoreBytes>\n  </keyStoreSource>\n  <password>$keystore_password</password>\n</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\"\n    timestamp \"Creating Jenkins certificate keystore credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$keystore\" ]; then\n    create_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        create_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-k8s-sa \"My Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\nfor _ in {1..4}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\n  <scope>GLOBAL</scope>\n  <id>$id</id>\n  <description>$description</description>\n</org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\"\n    timestamp \"Creating Jenkins kubernetes service account credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$id\" ]; then\n    create_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        create_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if ! [ -f \"$secret\" ]; then\n        die \"secret file '$secret' not found!\"\n    fi\n    timestamp \"Reading secret file '$secret'\"\n    secret=\"$(cat \"$secret\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Creating Jenkins secret file credential '$id' in store '$store' domain '$domain_name'\"\n    # can break on files with DOS line endings\n    #\"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    # this works either way\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d \"$xml\" ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$secret_file\" ]; then\n    create_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        create_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-text mySecretText \"\" \"\" \"My Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if [ -f \"$secret\" ]; then\n        timestamp \"Reading secret from file '$secret'\"\n        secret=\"$(cat \"$secret\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Creating Jenkins secret text credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$secret\" ]; then\n    create_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        create_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if [ -f \"$private_key\" ]; then\n        timestamp \"Reading private key from file '$private_key'\"\n        private_key=\"$(cat \"$private_key\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <usernameSecret>false</usernameSecret>\n  <privateKeySource class=\\\"com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\\$DirectEntryPrivateKeySource\\\">\n    <privateKey>$private_key</privateKey>\n  </privateKeySource>\n</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\"\n    timestamp \"Creating Jenkins ssh key credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$private_key\" ]; then\n    create_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        create_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_add_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-user-pass hari mypassword \"\" \"\" \"My Test Username + Password\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-user-pass' appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-user-pass', username 'hari' and password 'mypassword':\n\n        ${0##*/} hari-user-pass hari mypassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari mypassword '' '' 'My Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=mypassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\ncreate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <password>$password</password>\n  <usernameSecret>false</usernameSecret>\n</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\"\n    timestamp \"Creating Jenkins username/password credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/createCredentials\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$password\" ]; then\n    create_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        create_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to create a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if ! [ -f \"$keystore\" ]; then\n        die \"keystore file '$keystore' not found!\"\n    fi\n    timestamp \"Reading keystore file '$keystore'\"\n    local keystore_contents\n    keystore_contents=\"$(base64 \"$keystore\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <keyStoreSource class=\\\"com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\\$UploadedKeyStoreSource\\\">\n    <uploadedKeystoreBytes>$keystore_contents</uploadedKeystoreBytes>\n  </keyStoreSource>\n  <password>$keystore_password</password>\n</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\"\n    timestamp \"Creating Jenkins certificate keystore credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$keystore\" ]; then\n    create_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        create_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-k8s-sa \"My Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\n\ncreate_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\n  <scope>GLOBAL</scope>\n  <id>$id</id>\n  <description>$description</description>\n</org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\"\n    timestamp \"Creating Jenkins kubernetes service account credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$id\" ]; then\n    create_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        create_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if ! [ -f \"$secret\" ]; then\n        die \"secret file '$secret' not found!\"\n    fi\n    timestamp \"Reading secret file '$secret'\"\n    secret=\"$(cat \"$secret\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Creating Jenkins secret file credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$secret_file\" ]; then\n    create_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        create_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-text mySecretText \"\" \"\" \"My Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\ncreate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if [ -f \"$secret\" ]; then\n        timestamp \"Reading secret from file '$secret'\"\n        secret=\"$(cat \"$secret\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Creating Jenkins secret text credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$secret\" ]; then\n    create_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        create_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\ncreate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if [ -f \"$private_key\" ]; then\n        timestamp \"Reading private key from file '$private_key'\"\n        private_key=\"$(cat \"$private_key\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <usernameSecret>false</usernameSecret>\n  <privateKeySource class=\\\"com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\\$DirectEntryPrivateKeySource\\\">\n    <privateKey>$private_key</privateKey>\n  </privateKeySource>\n</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\"\n    timestamp \"Creating Jenkins ssh key credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$private_key\" ]; then\n    create_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        create_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_add_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-user-pass hari mypassword \"\" \"\" \"My Test Username + Password\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with -user-pass appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-user-pass', username 'hari' and password 'mypassword':\n\n        ${0##*/} hari-user-pass hari mypassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari mypassword '' '' 'My Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=mypassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\ncreate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <password>$password</password>\n  <usernameSecret>false</usernameSecret>\n</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\"\n    timestamp \"Creating Jenkins username/password credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" create-credentials-by-xml \"$store\" \"$domain\" <<< \"$xml\"\n    timestamp \"Secret '$id' created\"\n}\n\nif [ -n \"$password\" ]; then\n    create_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        create_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_delete.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-user-pass\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a Jenkins Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<credential_id> [<store> <domain>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nid=\"${1:-}\"\nstore=\"${2:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${3:-${JENKINS_SECRET_DOMAIN:-_}}\"\n\ndomain_name=\"$domain\"\nif [ \"$domain_name\" = '_' ]; then\n    domain_name='GLOBAL'\nfi\n\ntimestamp \"Deleting Jenkins credential '$id' in store '$store' domain '$domain_name'\"\n\"$srcdir/jenkins_cli.sh\" delete-credentials \"$store\" \"$domain\" \"$id\"\ntimestamp \"Secret '$id' deleted\"\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 19:27:21 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Jenkins credentials in the given credential store using jenkins_cli.sh\n\nDefaults to the system::system::jenkins global in-built store\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses adjacent jenkins_cli.sh - see there for more details on required connection settings / environment variables\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nstore=\"${1:-system::system::jenkins}\"\n\n# 'export JENKINS_CLI_ARGS=-webSocket' is needed if Jenkins is behind a reverse proxy such as Kubernetes Ingress, otherwise Jenkins CLI hangs\n\"$srcdir/jenkins_cli.sh\" list-credentials \"$store\"\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to create a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_cert.sh\" \"$id\" \"$keystore\" \"$keystore_password\" \"$store\" \"$domain\" \"$description\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_cert.sh\" \"$id\" \"$keystore\" \"$keystore_password\" \"$store\" \"$domain\" \"$description\"\n    fi\n}\n\nif [ -n \"$keystore\" ]; then\n    set_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        set_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-k8s-sa \"My Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\n\nset_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_kubernetes_sa.sh\" \"$id\" \"$description\" \"$store\" \"$domain\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_kubernetes_sa.sh\" \"$id\" \"$description\" \"$store\" \"$domain\"\n    fi\n}\n\nif [ -n \"$id\" ]; then\n    set_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        set_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_secret_file.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_secret_file.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\"\n    fi\n}\n\nif [ -n \"$secret_file\" ]; then\n    set_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        set_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-text mySecretText \"\" \"\" \"My Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_secret_text.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_secret_text.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\"\n    fi\n}\n\nif [ -n \"$secret\" ]; then\n    set_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        set_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nset_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_ssh_key.sh\" \"$id\" \"$user\" \"$private_key\" \"$store\" \"$domain\" \"$description\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_ssh_key.sh\" \"$id\" \"$user\" \"$private_key\" \"$store\" \"$domain\" \"$description\"\n    fi\n}\n\nif [ -n \"$private_key\" ]; then\n    set_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        set_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_set_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-user-pass hari mypassword \"\" \"\" \"My Test Username + Password\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with -user-pass appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-user-pass', username 'hari' and password 'mypassword':\n\n        ${0##*/} hari-user-pass hari mypassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari mypassword '' '' 'My Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=mypassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nset_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    if \"$srcdir/jenkins_cli.sh\" get-credentials-as-xml \"$store\" \"$domain\" \"$id\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_cli_update_user_pass.sh\" \"$id\" \"$user\" \"$password\" \"$store\" \"$domain\" \"$description\"\n    else\n        \"$srcdir/jenkins_cred_cli_add_user_pass.sh\" \"$id\" \"$user\" \"$password\" \"$store\" \"$domain\" \"$description\"\n    fi\n}\n\nif [ -n \"$password\" ]; then\n    set_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        set_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Updated Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to update a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # update a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Updated Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if ! [ -f \"$keystore\" ]; then\n        die \"keystore file '$keystore' not found!\"\n    fi\n    timestamp \"Reading keystore file '$keystore'\"\n    local keystore_contents\n    keystore_contents=\"$(base64 \"$keystore\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <keyStoreSource class=\\\"com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\\$UploadedKeyStoreSource\\\">\n    <uploadedKeystoreBytes>$keystore_contents</uploadedKeystoreBytes>\n  </keyStoreSource>\n  <password>$keystore_password</password>\n</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\"\n    timestamp \"Updating Jenkins certificate keystore credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$keystore\" ]; then\n    update_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        update_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-k8s-sa \"My Updated Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\n\nupdate_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\n  <scope>GLOBAL</scope>\n  <id>$id</id>\n  <description>$description</description>\n</org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\"\n    timestamp \"Updating Jenkins kubernetes service account credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$id\" ]; then\n    update_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        update_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Updated Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if ! [ -f \"$secret\" ]; then\n        die \"secret file '$secret' not found!\"\n    fi\n    timestamp \"Reading secret file '$secret'\"\n    secret=\"$(cat \"$secret\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Updating Jenkins secret file credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$secret_file\" ]; then\n    update_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        update_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-secret-text mySecretText \"\" \"\" \"My Updated Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if [ -f \"$secret\" ]; then\n        timestamp \"Reading secret from file '$secret'\"\n        secret=\"$(cat \"$secret\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Updating Jenkins secret text credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$secret\" ]; then\n    update_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        update_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My Updated SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nupdate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if [ -f \"$private_key\" ]; then\n        timestamp \"Reading private key from file '$private_key'\"\n        private_key=\"$(cat \"$private_key\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <usernameSecret>false</usernameSecret>\n  <privateKeySource class=\\\"com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\\$DirectEntryPrivateKeySource\\\">\n    <privateKey>$private_key</privateKey>\n  </privateKeySource>\n</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\"\n    timestamp \"Updating Jenkins ssh key credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$private_key\" ]; then\n    update_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        update_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_cli_update_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-cli-user-pass hari-updated-user my-updated-password '' '' 'My Updated Username + Password'\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system::system::jenkins' provider store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with -user-pass appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'hari-user-pass' to have username 'hari' and password 'myNewPassword':\n\n        ${0##*/} hari-user-pass hari myNewPassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari myNewPassword '' '' 'My Updated Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=myNewPassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system::system::jenkins}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\n\nupdate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <password>$password</password>\n  <usernameSecret>false</usernameSecret>\n</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\"\n    timestamp \"Updating Jenkins username/password credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_cli.sh\" update-credentials-by-xml \"$store\" \"$domain\" \"$id\" <<< \"$xml\"\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$password\" ]; then\n    update_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        update_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_delete.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-user-pass\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a Jenkins Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<credential_id> <store> <domain> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nstore=\"${2:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${3:-${JENKINS_SECRET_DOMAIN:-_}}\"\nfor _ in {1..3}; do shift || : ; done\ncurl_args=(\"$@\")\n\ndomain_name=\"$domain\"\nif [ \"$domain_name\" = '_' ]; then\n    domain_name='GLOBAL'\nfi\ntimestamp \"Deleting Jenkins username/password credential '$id' in store '$store' domain '$domain_name'\"\n\"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X DELETE ${curl_args:+\"${curl_args[@]}\"}\ntimestamp \"Secret '$id' deleted\"\n"
  },
  {
    "path": "jenkins/jenkins_cred_get.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: haritest\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets the config for a given Jenkins credentials by its ID from the given credential store and domain\n\nUseful to retrieve the configuration of a credential to update it or create a similar credential\n\nDefaults to the 'system' provider's store and global domain '_'\n\nTo obtain the credential ID, use jenkins_cred_list.sh\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<id> [<store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nid=\"$1\"\nstore=\"${2:-system}\"\ndomain=\"${3:-_}\"\nfor _ in {1..3}; do shift || : ; done\n\n\"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" \"$@\"\necho\n"
  },
  {
    "path": "jenkins/jenkins_cred_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Jenkins Credentials in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nThe returned credential IDs are what you should be specifying in your Jenkinsfile pipeline:\n\n    environment {\n        MYVAR = credentials('some-id')\n    }\n\nSee master Jenkinsfile for more examples:\n\n    https://github.com/HariSekhon/Jenkins/blob/master/Jenkinsfile\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nstore=\"${1:-system}\"\ndomain=\"${2:-_}\"\nshift || :\nshift || :\n\n\"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/api/json?tree=credentials\\\\[id\\\\]\" \"$@\" |\njq -r '.credentials[].id' |\nsort -f\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to create a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_cert.sh\" \"$id\" \"$keystore\" \"$keystore_password\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_cert.sh\" \"$id\" \"$keystore\" \"$keystore_password\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$keystore\" ]; then\n    set_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        set_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-k8s-sa \"My Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\nfor _ in {1..4}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_kubernetes_sa.sh\" \"$id\" \"$description\" \"$store\" \"$domain\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_kubernetes_sa.sh\" \"$id\" \"$description\" \"$store\" \"$domain\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$id\" ]; then\n    set_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        set_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_secret_file.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_secret_file.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$secret_file\" ]; then\n    set_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        set_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-text mySecretText \"\" \"\" \"My Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_secret_text.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_secret_text.sh\" \"$id\" \"$secret\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$secret\" ]; then\n    set_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        set_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_ssh_key.sh\" \"$id\" \"$user\" \"$private_key\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_ssh_key.sh\" \"$id\" \"$user\" \"$private_key\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$private_key\" ]; then\n    set_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        set_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_set_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-user-pass hari mypassword \"\" \"\" \"My Test Username + Password\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates or Updates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-user-pass' appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # create a credential with id 'hari-user-pass', username 'hari' and password 'mypassword':\n\n        ${0##*/} hari-user-pass hari mypassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari mypassword '' '' 'My Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=mypassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nset_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    if \"$srcdir/jenkins_cred_get.sh\" \"$id\" \"$store\" \"$domain\" &>/dev/null; then\n        \"$srcdir/jenkins_cred_update_user_pass.sh\" \"$id\" \"$user\" \"$password\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    else\n        \"$srcdir/jenkins_cred_add_user_pass.sh\" \"$id\" \"$user\" \"$password\" \"$store\" \"$domain\" \"$description\" ${curl_args:+\"${curl_args[@]}\"}\n    fi\n}\n\nif [ -n \"$password\" ]; then\n    set_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        set_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_cert.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-cert ~/Downloads/test.p12 \"\" \"\" \"\" \"My Updated Cert Keystore\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Certificate Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and PKCS#12 keystore file aren't given as arguments, then reads from stdin, reading in\n'ID=/path/to/keystore Description' or standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # If you want to update a sample p12 file, you can do it like this:\n\n        openssl req -x509 -nodes -newkey rsa:2048 -keyout test.key -out test.crt -subj '/CN=test.com'\n        openssl pkcs12 -export -inkey test.key -in test.crt -passout pass: -out test.p12\n\n    # update a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} my-certificate-keystore ~/Downloads/test.p12 '' '' 'My Certificate Keystore'\n\n    # or piped from standard input:\n\n        # export KEYSTORE_PASSWORD, JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-certificate-keystore=~/Downloads/test.p12 | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <keystore> <keystore_password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nkeystore=\"${2:-}\"\nkeystore_password=\"${3:-${KEYSTORE_PASSWORD:-}}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local keystore=\"$value\"\n    if ! [ -f \"$keystore\" ]; then\n        die \"keystore file '$keystore' not found!\"\n    fi\n    timestamp \"Reading keystore file '$keystore'\"\n    local keystore_contents\n    keystore_contents=\"$(base64 \"$keystore\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <keyStoreSource class=\\\"com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl\\$UploadedKeyStoreSource\\\">\n    <uploadedKeystoreBytes>$keystore_contents</uploadedKeystoreBytes>\n  </keyStoreSource>\n  <password>$keystore_password</password>\n</com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl>\"\n    timestamp \"Updating Jenkins certificate keystore credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$keystore\" ]; then\n    update_credential \"$id\"=\"$keystore\" \"$description\"\nelse\n    while read -r id_keystore description; do\n        update_credential \"$id_keystore\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_kubernetes_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-k8s-sa \"My Updated Kubernetes ServiceAccount\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Kubernetes Service Account Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id is not given as an argument, then reads from stdin, reading in ID=DESCRIPTION format\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5, Kubernetes plugin 1.29.2, and Kubernetes Credential plugin 0.8.0\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'my-k8s-sa':\n\n        ${0##*/} my-k8s-sa 'My Description'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo my-k8s-sa=my description | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <description> <store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\ndescription=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\nfor _ in {1..4}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local id=\"$1\"\n    local description=\"${2:-}\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\n  <scope>GLOBAL</scope>\n  <id>$id</id>\n  <description>$description</description>\n</org.jenkinsci.plugins.kubernetes.credentials.FileSystemServiceAccountCredential>\"\n    timestamp \"Updating Jenkins kubernetes service account credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$id\" ]; then\n    update_credential \"$id\" \"$description\"\nelse\n    while read -r id description; do\n        update_credential \"$id\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_secret_file.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-file ~/.aws/keys/circleci_accessKeys.csv \"\" \"\" \"My Update Secret File\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Secret File Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret file aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for shell piping\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'aws-access-key-csv', and file ~/.aws/keys/jenkins_accessKeys.csv:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-csv ~/.aws/keys/jenkins_accessKeys.csv '' '' 'My AWS Access Key CSV'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo aws-access-key-csv=~/.aws/keys/jenkins_accessKeys.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret_file=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if ! [ -f \"$secret\" ]; then\n        die \"secret file '$secret' not found!\"\n    fi\n    timestamp \"Reading secret file '$secret'\"\n    secret=\"$(cat \"$secret\")\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Updating Jenkins secret file credential '$id' in store '$store' domain '$domain_name'\"\n    # can break on files with DOS line endings\n    #\"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    # this works either way\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d \"$xml\" ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$secret_file\" ]; then\n    update_credential \"$id\"=\"$secret_file\" \"$description\"\nelse\n    while read -r id_secretFile description; do\n        update_credential \"$id_secretFile\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_secret_text.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-secret-text mySecretText \"\" \"\" \"My Updated Secret Text\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Secret Text Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id and secret text aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'aws-access-key-id', and text 'AKIA...':\n\n        ${0##*/} aws-access-key-id AKIA...\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} aws-access-key-id AKIA... '' '' 'My AWS Access Key ID'\n\n    # or piped from standard input:\n\n        # export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'AWS_ACCESS_KEY_ID=AKIA...' | ${0##*/}\n        echo 'AWS_SECRET_ACCESS_KEY=...' | ${0##*/}\n\n    # using aws_csv_creds.sh to load the credentials from a standard AWS credential download:\n\n        aws_csv_creds.sh ~/.aws/keys/downloaded_access_key_creds.csv | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <secret_text> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nsecret=\"${2:-}\"\nstore=\"${3:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${4:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${5:-}\"\nfor _ in {1..5}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local key_value=\"$1\"\n    local description=\"${2:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local id=\"$key\"\n    # shellcheck disable=SC2154\n    local secret=\"$value\"\n    if [ -f \"$secret\" ]; then\n        timestamp \"Reading secret from file '$secret'\"\n        secret=\"$(cat \"$secret\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <secret>$secret</secret>\n</org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl>\"\n    timestamp \"Updating Jenkins secret text credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$secret\" ]; then\n    update_credential \"$id\"=\"$secret\" \"$description\"\nelse\n    while read -r id_secret description; do\n        update_credential \"$id_secret\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-ssh-key hari ~/.ssh/id_rsa \"\" \"\" \"My Updated SSH Key\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins SSH Key Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and private_key aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-ssh-key' appended\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'hari-ssh-key', username 'hari' and load the private key contents from my ~/.ssh/id_rsa file:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-ssh-key hari ~/.ssh/id_rsa '' '' 'My SSH Key'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin if not using system global store\n\n        echo hari=~/.ssh/id_rsa | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <private_key_or_file> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\nprivate_key=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local private_key=\"$value\"\n    if [ -f \"$private_key\" ]; then\n        timestamp \"Reading private key from file '$private_key'\"\n        private_key=\"$(cat \"$private_key\")\"\n    fi\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\n  <scope>$domain</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <usernameSecret>false</usernameSecret>\n  <privateKeySource class=\\\"com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey\\$DirectEntryPrivateKeySource\\\">\n    <privateKey>$private_key</privateKey>\n  </privateKeySource>\n</com.cloudbees.jenkins.plugins.sshcredentials.impl.BasicSSHUserPrivateKey>\"\n    timestamp \"Updating Jenkins ssh key credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$private_key\" ]; then\n    update_credential \"$id\" \"$user\"=\"$private_key\" \"$description\"\nelse\n    while read -r id user_privatekey description; do\n        if [ -z \"${user_privatekey:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_privatekey=\"$id\"\n            id=\"${id%%=*}-ssh-key\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        update_credential \"$id\" \"$user_privatekey\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_cred_update_user_pass.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: hari-test-api-user-pass hari mypassword \"\" \"\" \"My Updated Username + Password\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://github.com/jenkinsci/credentials-plugin/blob/master/docs/user.adoc#creating-a-credentials\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates a Jenkins Username/Password Credential in the given credential store and domain\n\nDefaults to the 'system' provider's store and global domain '_'\n\nIf credential id, user and password aren't given as arguments, then reads from stdin, reading in KEY=VALUE\nor standard shell export format - useful for piping from tools like aws_csv_creds.sh\n\nIf standard input does not have an id field, the id will be generated from the username lowercased with '-user-pass' appended.\n\nIn cases where you are reading secrets from stdin, you can set the store and domain via the environment variables\n\\$JENKINS_SECRET_STORE and \\$JENKINS_SECRET_DOMAIN\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExamples:\n\n    # update a credential with id 'hari-user-pass', username 'hari' and password 'mypassword':\n\n        ${0##*/} hari-user-pass hari mypassword\n\n    # with a description, leaving the store and domain as the default global one:\n\n        ${0##*/} hari-user-pass hari mypassword '' '' 'My Username + Password'\n\n    # or piped from standard input:\n\n        #export JENKINS_SECRET_STORE and JENKINS_SECRET_DOMAIN if using stdin but not using system global store\n\n        echo 'hari=mypassword' | ${0##*/}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<credential_id> <user> <password> <store> <domain> <description> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nid=\"${1:-}\"\nuser=\"${2:-}\"\npassword=\"${3:-}\"\nstore=\"${4:-${JENKINS_SECRET_STORE:-system}}\"\ndomain=\"${5:-${JENKINS_SECRET_DOMAIN:-_}}\"\ndescription=\"${6:-}\"\nfor _ in {1..6}; do shift || : ; done\ncurl_args=(\"$@\")\n\nupdate_credential(){\n    local id=\"$1\"\n    local key_value=\"$2\"\n    local description=\"${3:-}\"\n    parse_export_key_value \"$key_value\"\n    # key/value are exported by above function\n    # shellcheck disable=SC2154\n    local user=\"$key\"\n    # shellcheck disable=SC2154\n    local password=\"$value\"\n    local domain_name=\"$domain\"\n    if [ \"$domain_name\" = '_' ]; then\n        domain_name='GLOBAL'\n    fi\n    local xml=\"<com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\n  <scope>$domain_name</scope>\n  <id>$id</id>\n  <description>$description</description>\n  <username>$user</username>\n  <password>$password</password>\n  <usernameSecret>false</usernameSecret>\n</com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl>\"\n    timestamp \"Updating Jenkins username/password credential '$id' in store '$store' domain '$domain_name'\"\n    \"$srcdir/jenkins_api.sh\" \"/credentials/store/$store/domain/$domain/credential/$id/config.xml\" -X POST -H \"Content-Type: application/xml\" -d @<(cat <<< \"$xml\") ${curl_args:+\"${curl_args[@]}\"}\n    timestamp \"Secret '$id' updated\"\n}\n\nif [ -n \"$password\" ]; then\n    update_credential \"$id\" \"$user\"=\"$password\" \"$description\"\nelse\n    while read -r id user_password; do\n        if [ -z \"${user_password:-}\" ] && [[ \"$id\" =~ = ]]; then\n            user_password=\"$id\"\n            id=\"${id%%=*}-user-pass\"\n            id=\"$(tr '[:upper:]' '[:lower:]' <<< \"$id\")\"\n        else\n            timestamp \"WARNING: invalid line detected, skipping creating credential\"\n            continue\n        fi\n        update_credential \"$id\" \"$user_password\" \"$description\"\n    done < <(sed 's/^[[:space:]]*export[[:space:]]*//; /^[[:space:]]*$/d')\nfi\n"
  },
  {
    "path": "jenkins/jenkins_creds_cli_xml_dump.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 19:27:21 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps the XML configs of all credentials in the global system store using jenkins_cli.sh\n\nTested on Jenkins 2.319 with Credentials plugin 2.5\n\nUses adjacent jenkins_cli.sh - see there for more details on required connection settings / environment variables\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nstore=\"${1:-system::system::jenkins}\"\n\n# -webSocket is needed if Jenkins is behind a reverse proxy such as Kubernetes Ingress, otherwise Jenkins CLI hangs\n#\"$srcdir/jenkins_cli.sh\" -webSocket list-credentials-as-xml system::system::jenkins\n\"$srcdir/jenkins_cli.sh\" list-credentials-as-xml \"$store\"\n"
  },
  {
    "path": "jenkins/jenkins_foreach_job.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo \"job='{job}'\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each Jenkins Job name obtained via the Jenkins API\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the job names and exit after the first iteration\n\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\nThe command template replaces the following for convenience in each iteration:\n\n{job}    => the job name\n\n\nTested on Jenkins 2.319\n\n\nExample:\n\n    ${0##*/} echo \\\"job='{job}'\\\"\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/jenkins_jobs.sh\" |\nwhile read -r job; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $job\" >&2\n    echo \"# ============================================================================ #\" >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{job\\}/$job}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "jenkins/jenkins_foreach_job_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo \"job='{job}'\"\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each Jenkins Job name obtained via the Jenkins CLI\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the job names and exit after the first iteration\n\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\nThe command template replaces the following for convenience in each iteration:\n\n{job}    => the job name\n\n\nTested on Jenkins 2.319\n\n\nExample:\n\n    ${0##*/} echo \\\"job='{job}'\\\"\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/jenkins_cli.sh\" list-jobs |\nwhile read -r job; do\n    echo \"# ============================================================================ #\" >&2\n    echo \"# $job\" >&2\n    echo \"# ============================================================================ #\" >&2\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{job\\}/$job}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo >&2\ndone\n"
  },
  {
    "path": "jenkins/jenkins_job_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets or Sets the Jenkins job/pipeline config via the Jenkins API\n\nTested on Jenkins 2.319 and 2.426\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\n\nExample:\n\n    # Get the current job's configuration:\n\n        ${0##*/} myJob\n\n    # Set a new job configuration:\n\n        ${0##*/} myJob myConfig.xml\n\n    # Pretty-print through xmllint:\n\n        ${0##*/} myJob | xmllint --format\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<config.xml>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nconfig_file=\"${2:-}\"\n\nif [ -n \"$config_file\" ]; then\n    if ! [ -f \"$config_file\" ]; then\n        die \"Config file not found: $config_file\"\n    fi\n    \"$srcdir/jenkins_api.sh\" \"/job/$job/config.xml\" -X POST -d @\"$config_file\"\n    timestamp \"Set Jenkins job '$job' config\"\nelse\n    \"$srcdir/jenkins_api.sh\" \"/job/$job/config.xml\"\n    # because there isn't a newline at the end\n    echo\nfi\n"
  },
  {
    "path": "jenkins/jenkins_job_description.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets or Sets the description of a given Jenkins job/pipeline via the Jenkins API\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<description>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nshift || :\ndescription=\"$*\"\n\nif [ -n \"$description\" ]; then\n    \"$srcdir/jenkins_api.sh\" \"/job/$job/description?description=$description\" -X POST\n    timestamp \"Set Jenkins job '$job' description to '$description'\"\nelse\n    \"$srcdir/jenkins_api.sh\" \"/job/$job/description\"\n    # because there isn't a newline at the end\n    echo\nfi\n"
  },
  {
    "path": "jenkins/jenkins_job_disable.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2022-01-25 19:26:05 +0000 (Tue, 25 Jan 2022)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\n// XXX: Edit this to the name of your job pipeline\ndef jobName = \"My Dev Pipeline\"\ndef job = Jenkins.instance.getItem(jobName)\njob.setDisabled(true)\njob.isDisabled()\n"
  },
  {
    "path": "jenkins/jenkins_job_disable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables a Jenkins job/pipeline via the Jenkins API\n\nTested on Jenkins 2.319 and 2.246\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nshift || :\n\ntimestamp \"Disabling Jenkins job '$job'\"\n\"$srcdir/jenkins_api.sh\" \"/job/$job/disable\" -X POST \"$@\"\n"
  },
  {
    "path": "jenkins/jenkins_job_enable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables a Jenkins job/pipeline via the Jenkins API\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nshift || :\n\ntimestamp \"Enabling Jenkins job '$job'\"\n\"$srcdir/jenkins_api.sh\" \"/job/$job/enable\" -X POST \"$@\"\n"
  },
  {
    "path": "jenkins/jenkins_job_trigger.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTriggers a Jenkins build job via the Jenkins API\n\nThe Jenkins job configuration option 'Trigger builds remotely (e.g., from scripts)' can be used to generate a token just for this job/pipeline, or you can use your personal API token\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nshift || :\n\n\"$srcdir/jenkins_api.sh\" \"/job/$job/build?cause=Triggered by ${0##*/}\" -X POST \"$@\"\ntimestamp \"Triggered Jenkins job '$job'\"\n"
  },
  {
    "path": "jenkins/jenkins_job_trigger_with_params.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTriggers a Jenkins build job with parameters via the Jenkins API\n\nTo specify parameters, add --data KEY=VALUE format arguments\n\nThe Jenkins job configuration option 'Trigger builds remotely (e.g., from scripts)' can be used to generate a token just for this job/pipeline, or you can use your personal API token\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob=\"$1\"\nshift || :\n\n\"$srcdir/jenkins_api.sh\" \"/job/$job/buildWithParameters?cause=Triggered by ${0##*/}\" -X POST \"$@\"\ntimestamp \"Triggered Jenkins job '$job'\"\n"
  },
  {
    "path": "jenkins/jenkins_jobs.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2021-04-08 19:16:17 +0100 (Thu, 08 Apr 2021)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\n//jenkins.model.Jenkins.instance.items.findAll().each { println it.name } // return the whole array, don't want that, just return num\njenkins.model.Jenkins.instance.items.findAll().each { println it.displayName }.size\n"
  },
  {
    "path": "jenkins/jenkins_jobs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Jenkins Job/Pipeline Names via the Jenkins API\n\nTested on Jenkins 2.319 and 2.246\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<store> <domain> <curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/jenkins_api.sh\" '/api/json?tree=jobs\\[name\\]&pretty=true' |\njq -r '.jobs[].name' |\nsort -f\n"
  },
  {
    "path": "jenkins/jenkins_jobs_disable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-23 01:13:35 +0000 (Fri, 23 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDisables all Jenkins jobs/pipelines with names matching a given regex via the Jenkins API\n\nTested on Jenkins 2.319 and 2.246\n\nRemember to quote the job name regex filter to stop it matching your local files eg. '.*' to not match '. .. .envrc'\n\nUses the adjacent jenkins_job_disable.sh jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name_regex_filter> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [[ \"$*\" =~ \\.\\ \\.\\. ]]; then\n    die \"You've specified an unquoted .* that has match . and .. directories - remember to quote your regex!\"\nfi\n\njob_name_regex_filter=\"$1\"\nshift || :\n\ntimestamp \"Getting job list\"\n\"$srcdir/jenkins_jobs.sh\" \"$@\" |\nwhile read -r job_name; do\n    if [[ \"$job_name\" =~ $job_name_regex_filter ]]; then\n        \"$srcdir/jenkins_job_disable.sh\" \"$job_name\" \"$@\"\n    fi\ndone\n"
  },
  {
    "path": "jenkins/jenkins_jobs_disabled.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2022-01-25 19:26:05 +0000 (Tue, 25 Jan 2022)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// Lists all disabled Jenkins jobs\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\njenkins.model.Jenkins.instance.items.findAll { it.isDisabled() }.each { println it.name }.size\n"
  },
  {
    "path": "jenkins/jenkins_jobs_download_configs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads all Jenkins job configs to files of the same name in the current directory via the Jenkins API\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ntimestamp \"Downloading all Jenkins job configs to current directory: $PWD\"\n\n\"$srcdir/jenkins_foreach_job.sh\" \"\n    '$srcdir/jenkins_job_config.sh' '{job}' > '{job}.xml' &&\n    echo 'Downloaded config to file: $PWD/{job}.xml'\n    \"\n"
  },
  {
    "path": "jenkins/jenkins_jobs_download_configs_cli.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-06-28 18:34:34 +0100 (Tue, 28 Jun 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads all Jenkins job configs to files of the same name in the current directory via the Jenkins CLI\n\nTested on Jenkins 2.319\n\nUses the adjacent jenkins_cli.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ntimestamp \"Downloading all Jenkins job configs to current directory: $PWD\"\n\n# something in jenkins_cli.sh is causing the loop to terminate early with no more lines\n\"$srcdir/jenkins_foreach_job_cli.sh\" \"\n    '$srcdir/jenkins_cli.sh' get-job '{job}' </dev/null > '{job}.xml' &&\n    echo >> '{job}.xml' &&\n    echo 'Downloaded config to file: $PWD/{job}.xml'\n    \"\n"
  },
  {
    "path": "jenkins/jenkins_jobs_enable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-23 01:13:35 +0000 (Fri, 23 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://wiki.jenkins-ci.org/display/JENKINS/Remote+access+API\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables all Jenkins jobs/pipelines with names matching a given regex via the Jenkins API\n\nTested on Jenkins 2.319 and 2.246\n\nRemember to quote the job name regex filter to stop it matching your local files eg. '.*' to not match '. .. .envrc'\n\nUses the adjacent jenkins_job_enable.sh jenkins_api.sh - see there for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name_regex_filter> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif [[ \"$*\" =~ \\.\\ \\.\\. ]]; then\n    die \"You've specified an unquoted .* that has match . and .. directories - remember to quote your regex!\"\nfi\n\njob_name_regex_filter=\"$1\"\nshift || :\n\ntimestamp \"Getting job list\"\n\"$srcdir/jenkins_jobs.sh\" \"$@\" |\nwhile read -r job_name; do\n    if [[ \"$job_name\" =~ $job_name_regex_filter ]]; then\n        \"$srcdir/jenkins_job_enable.sh\" \"$job_name\" \"$@\"\n    fi\ndone\n"
  },
  {
    "path": "jenkins/jenkins_jobs_status.groovy",
    "content": "//\n//  Author: Hari Sekhon\n//  Date: 2022-01-25 19:26:05 +0000 (Tue, 25 Jan 2022)\n//\n//  vim:ts=4:sts=4:sw=4:noet\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\n// List all Jenkins jobs prefix with whether they are enabled or disabled\n\n// Paste this into $JENKINS_URL/script\n//\n// Jenkins -> Manage Jenkins -> Script Console\n\njenkins.model.Jenkins.instance.items.findAll().each { println \"disabled: ${it.isDisabled()}\\t\\tname: ${it.name}\" }.size\n"
  },
  {
    "path": "jenkins/jenkins_password.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-28 15:16:18 +0000 (Sat, 28 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gets the Jenkins initial admin password out of the Docker Compose jenkins-server container or Kubernetes jenkins helm deployment secret\n\n# For docker-compose you might need the compose project name matching the instantiation, eg.\n#\n#   export COMPOSE_PROJECT_NAME=\"bash-tools\"\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nconfig=\"$srcdir/../docker-compose/jenkins.yml\"\n\nif [ -n \"${JENKINS_PASSWORD:-}\" ]; then\n    echo \"using \\$JENKINS_PASSWORD from environment\" >&2\n\n# weird bug on macOS 14.1 where grep -q is no longer matching but grep is, so >/dev/null instead\nelif docker ps 2>/dev/null | grep jenkins-server >/dev/null; then\n\n    # </dev/null stops this swallowing stdin which we need for jenkins_cli.sh create-job\n    JENKINS_PASSWORD=\"$(docker-compose -f \"$config\" exec -T jenkins-server cat /var/jenkins_home/secrets/initialAdminPassword </dev/null)\"\n\n#elif kubectl get po -n jenkins -l app.kubernetes.io/component=jenkins-controller -o name 2>/dev/null | grep -q .; then\n    #pod=\"$(kubectl get po -n jenkins -l app.kubernetes.io/component=jenkins-controller -o name)\"\n    # doesn't exist\n    #JENKINS_PASSWORD=\"$(kubectl exec -ti -n jenkins \"$pod\" -- cat /var/jenkins_home/secrets/initialAdminPassword)\"\n\nelif kubectl get secret -n jenkins jenkins &>/dev/null; then\n    JENKINS_PASSWORD=\"$(kubectl get secret -n jenkins jenkins -o jsonpath=\"{.data.jenkins-admin-password}\" | base64 --decode)\"\n\nelif [ -f /var/jenkins_home/secrets/initialAdminPassword ]; then\n    JENKINS_PASSWORD=\"$(cat /var/jenkins_home/secrets/initialAdminPassword)\"\nfi\n\nif [ -z \"${JENKINS_PASSWORD:-}\" ]; then\n    echo \"ERROR: failed to determine JENKINS_PASSWORD from environment, docker-compose or kubernetes\" >&2\n    exit 1\nfi\n\n# if sourced, export JENKINS_PASSWORD, if subshell, echo it\n#if [[ \"$_\" != \"$0\" ]]; then\n    export JENKINS_PASSWORD\n#else\n    echo -n \"$JENKINS_PASSWORD\"  # no newline so we can pipe straight to pbcopy / xclip or similar\n#fi\n"
  },
  {
    "path": "jenkins/jenkins_plugins_latest_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: kubernetes\n#\n#  Author: Hari Sekhon\n#  Date: 2024-01-31 02:32:56 +0000 (Wed, 31 Jan 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the latest versions of Jenkins plugins given by args by querying:\n\n    https://updates.jenkins.io/current/update-center.actual.json\n\nUsed by update_plugin_versions.sh script in https://github.com/HariSekhon/Kubernetes-configs/tree/master/jenkins/base\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<plugin>\"\n\nhelp_usage \"$@\"\n\nany_opt_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"https://updates.jenkins.io/current/update-center.actual.json\"\n\nlog \"* downloading json\"\njson=\"$(curl -sSfL \"$url\" || die \"failed to fetch json from '$url'\")\"\n\nlog \"* checking not blank\"\n# extremely poor performance for large 3M json download\n#if is_blank \"$json\"; then\nif [ -z \"$json\" ]; then\n    die \"json returned from url '$url' is blank!\"\nfi\n\n\nlog \"* parsing json\"\n#version=\"$(jq -r \".plugins[] | select(.name == \\\"$plugin\\\") | .version\" <<< \"$json\")\"\nplugin_versions=\"$(jq -r '.plugins[] | .name + \":\" + .version' <<< \"$json\")\"\n\nexitcode=0\n\nlog \"* printing plugin:version for each found plugin\"\nfor plugin in \"$@\"; do\n    if ! grep \"^$plugin:\" <<< \"$plugin_versions\"; then\n        warning \"plugin '$plugin' not found! Have you provided the right plugin slug?\"\n        exitcode=1\n    fi\ndone\n\n#echo \"$version\"\n\nexit $exitcode\n"
  },
  {
    "path": "kafka/kafka_acls.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nkafka-acls.sh $bootstrap_server \"$@\"\n"
  },
  {
    "path": "kafka/kafka_cli_jaas.conf",
    "content": "//  vim:ts=4:sts=4:sw=4:et\n//\n//  Author: Hari Sekhon\n//  Date: 2016-06-28 14:39:32 +0100 (Tue, 28 Jun 2016)\n//\n//  https://github.com/HariSekhon/DevOps-Bash-tools\n//\n//  License: see accompanying Hari Sekhon LICENSE file\n//\n//  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n//\n//  https://www.linkedin.com/in/HariSekhon\n//\n\nKafkaClient {\n    com.sun.security.auth.module.Krb5LoginModule required\n    useKeyTab=false\n    useTicketCache=true\n    renewTGT=false\n    doNotPrompt=true\n    serviceName=\"kafka\";\n};\n"
  },
  {
    "path": "kafka/kafka_configs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nkafka-configs.sh $kafka_zookeeper \"$@\"\n"
  },
  {
    "path": "kafka/kafka_console_consumer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nexec kafka-console-consumer.sh $bootstrap_server \"$@\"\n"
  },
  {
    "path": "kafka/kafka_console_producer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nexec kafka-console-producer.sh $broker_list \"$@\"\n"
  },
  {
    "path": "kafka/kafka_consumer_groups.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nexec kafka-consumer-groups.sh $bootstrap_server \"$@\"\n"
  },
  {
    "path": "kafka/kafka_consumer_perf_test.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nexec kafka-consumer-perf-test.sh $broker_list \"$@\"\n"
  },
  {
    "path": "kafka/kafka_producer_perf_test.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\nexec kafka-producer-perf-test.sh \"$@\"\n"
  },
  {
    "path": "kafka/kafka_topics.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-07-30 15:16:58 +0100 (Tue, 30 Jul 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# sources heap, kerberos, brokers, zookeepers etc\n# shellcheck source=.bash.d/kafka.sh\n. \"$srcdir/.bash.d/kafka.sh\"\n\n# old version of kafka used --zookeeper, deprecated now\n#kafka-topics.sh $kafka_zookeeper \"$@\"\n\n# it's assigned in .bash.d/kafka.sh\n# shellcheck disable=SC2154,SC2086\nexec kafka-topics.sh $bootstrap_server \"$@\"\n"
  },
  {
    "path": "kics.config",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2023-05-05 18:05:53 +0100 (Fri, 05 May 2023)\n#\n#  vim:ts=2:sts=2:sw=2:et:filetype=yaml\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             K i c s   C o n f i g\n# ============================================================================ #\n\n# https://github.com/Checkmarx/kics/blob/master/docs/configuration-file.md\n\n---\n#path: assets/iac_samples\nverbose: true\nlog-file: true\n#type:\n#  - Dockerfile\n#  - Kubernetes\n#queries-path: \"assets/queries\"\nexclude-paths:\n  # ignore submodules - handle them in the source repos only\n  - bash-tools/\n  - github-actions/\n  - haproxy-configs/\n  - jenkins/\n  - kubernetes-templates/\n  - lib/\n  - pylib/\n  - spotify-tools/\n  - sql/\n  - sql-keywords/\n  - templates/\n  - terraform-templates/\n#output-path: \"results\"\n"
  },
  {
    "path": "kubernetes/argocd_apps_sync.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-08 20:31:59 +0100 (Mon, 08 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSync all ArgoCD apps matching an optional given ERE regex filter\n\nRequires ArgoCD CLI to be installed and configured for authentication\n\nYou may also want to set some environment variables such as:\n\n    export ARGOCD_SERVER='argocd.domain.com'\n    export ARGOCD_OPTS='--grpc-web --timeout 600'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<name_filter>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nfilter=\"${1:-.*}\"\n\nargocd app list |\nawk '{print $1}' |\n{ grep -E \"$filter\" || : ; } |\nwhile read -r app; do\n    echo \"Syncing app '$app':\"\n    echo\n    cmd=(argocd app sync \"$app\")\n    echo \"${cmd[*]}\"\n    echo\n    \"${cmd[@]}\"\n    echo\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/argocd_apps_wait_sync.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-08 20:31:59 +0100 (Mon, 08 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSync all ArgoCD apps matching an optional given ERE regex filter\n\nRequires ArgoCD CLI to be installed and configured for authentication\n\nYou may also want to set some environment variables such as:\n\n    export ARGOCD_SERVER='argocd.domain.com'\n    export ARGOCD_OPTS='--grpc-web --timeout 600'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<name_filter>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nfilter=\"${1:-.*}\"\n\nargocd app list |\nawk '{print $1}' |\n{ grep -E \"$filter\" || : ; } |\nwhile read -r app; do\n    echo \"Syncing app '$app':\"\n    echo\n    cmd=(argocd app wait \"$app\" --sync --operation --health)\n    echo \"${cmd[*]}\"\n    \"${cmd[@]}\"\n    echo\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/argocd_auto_sync.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-08-10 21:06:30 +0100 (Thu, 10 Aug 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnables/Disable ArgoCD Auto-Sync\n\nNeed to be able to manually fix apps sometimes, without ArgoCD reverting the changes before they're testing and committed\n\nIf ArgoCD apps of the name 'argocd' and 'apps' are found, then toggles them too to prevent cascading auto-sync re-enabling via the App-of-Apps pattern\n(see https://github.com/HariSekhon/Kubernetes-configs)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"on/off [<app>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_bin argocd\ncheck_env_defined \"ARGOCD_SERVER\"\n\nif [ \"$1\" = on ]; then\n    automated_or_none=\"automated\"\n    msg=\"enabling auto-sync on app\"\nelif [ \"$1\" = off ]; then\n    automated_or_none=\"none\"\n    msg=\"disabling auto-sync on app\"\nelse\n    usage \"invalid first arg given, must be either 'on' or 'off'\"\nfi\n\n# custom env var, found in .envrc in Kubernetes and DevOps-Bash-tools .envrc files\nif [ -n \"${ARGOCD_APP:-}\" ]; then\n    app=\"$ARGOCD_APP\"\nelif [ $# -eq 2 ]; then\n    app=\"$2\"\nelse\n    usage \"\\$ARGOCD_APP not defined and no second arg passed for app to enable/disable auto-sync for\"\nfi\n\napps=\"$(argocd app list -o name | sed 's|argocd/||')\"\n\n# App-of-Apps which would re-enable the app must be disabled first\nfor base_app in argocd apps; do\n    if grep -Fxq 'argocd' <<< \"$apps\"; then\n        timestamp \"$msg '$base_app'\"\n        argocd app set \"$base_app\" --sync-policy \"$automated_or_none\"\n    fi\n    echo >&2\ndone\n\ntimestamp \"$msg '$app'\"\nargocd app set \"$app\" --sync-policy \"$automated_or_none\"\n"
  },
  {
    "path": "kubernetes/argocd_generate_resource_whitelist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-10 18:23:34 +0100 (Wed, 10 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates YAML list of ArgoCD resources to whitelist for both Cluster and Namespace for an ArgoCD argoproj.io/v1alpha1 AppProject\n\nThis is needed if using whitelists in ArgoCD 2.2+ because they restrict showing the child objects unless they are whitelists if using whitelists\n\nDumps all API object resources from the current Kubernetes cluster context into YAML format to put straight into argocd-project.yaml\n\nIf an argocd-project.yaml is given, parses it for existing resource whitelist, merges the two lists in sorted order by Kind and then updates the yaml with an in-place edit\nYou should have revision controlled your yaml file in Git before you do this - both as a backup of the last good version as well as to see the difference of the edited file\n\nRequires kubectl to be installed and configured, and also yq if a yaml file is given\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<argocd-project.yaml>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nyaml_file=\"${1:-}\"\n\napi_resources=\"$(\n    # there is no -o json as of Kubernetes 1.27\n    kubectl api-resources |\n    # strip header line\n    tail -n +2 || :  # ignore this error:\n    # error: unable to retrieve the complete list of server APIs: external.metrics.k8s.io/v1beta1: received empty response for: external.metrics.k8s.io/v1beta1\n)\"\n\nif [ -n \"$yaml_file\" ]; then\n    [ -f \"$yaml_file\" ] || die \"Given YAML file does not exist: $yaml_file\"\nfi\n\necho \"clusterResourceWhitelist:\"\nawk '$(NF-1) ~/false/ {print $NF\" \"$(NF-2)}' <<< \"$api_resources\" |\nif [ -n \"$yaml_file\" ]; then\n    # pass through api_resources and strip trailing /v1 as it's redundant vs file contents\n    sed 's|/v1$| |'\n    # add resources from yaml\n    yq '.spec.clusterResourceWhitelist[] |\n        [ .kind, .group | sub(\"^$\", \"v1\")] |\n        @tsv' \"$yaml_file\"\nelse\n    # pass through api_resources and strip trailing /v1 as it's redundant vs file contents\n    sed 's|/v1$| |'\nfi |\ncolumn -t |\nsort -u |\nwhile read -r kind group; do\n    # ArgoCD doesn't show pods for v1 Pod, only '' Pod in the project resource whitelist\n    if [ \"$group\" = \"v1\" ]; then\n        group=\"''\"\n    fi\n    echo \"  - group: $group\"\n    echo \"    kind: $kind\"\ndone |\nif [ -n \"$yaml_file\" ]; then\n    yq -i \".spec.clusterResourceWhitelist = $(yq '.' -o json)\" \"$yaml_file\"\nelse\n    cat\nfi\necho\necho \"namespaceResourceWhitelist:\"\nawk '$(NF-1) ~/true/ {print $NF\" \"$(NF-2)}' <<< \"$api_resources\" |\nif [ -n \"$yaml_file\" ]; then\n    # pass through api_resources and strip trailing /v1 as it's redundant vs file contents\n    sed 's|/v1$| |'\n    # add resources from yaml\n    yq '.spec.namespaceResourceWhitelist[] |\n        [ .kind, .group | sub(\"^$\", \"v1\")] |\n        @tsv' \"$yaml_file\"\nelse\n    # pass through api_resources and strip trailing /v1 as it's redundant vs file contents\n    sed 's|/v1$| |'\nfi |\ncolumn -t |\nsort -u |\nwhile read -r kind group; do\n    if [ \"$group\" = \"v1\" ]; then\n        group=\"''\"\n    fi\n    echo \"  - group: $group\"\n    echo \"    kind: $kind\"\ndone |\nif [ -n \"$yaml_file\" ]; then\n    yq -i \".spec.namespaceResourceWhitelist = $(yq '.' -o json)\" \"$yaml_file\"\nelse\n    cat\nfi\n"
  },
  {
    "path": "kubernetes/argocd_namespace_resource_whitelist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-18 11:43:50 +0100 (Fri, 18 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFilter program to generate ArgoCD namespace resource whitelist from a given Kubernetes YAML or Kustomize build output\n\nYAML can be supplied as a file argument or via standard input. If no file is given, waits for stdin like a standard unix filter program\n\nOutputs YAML for the namespaceResourceWhitelist section of argocd-project.yaml\n\nA full argocd-project.yaml is already provided at the URL below with all the most common object permissions already populated via the output from this script against my production environment\n\n    https://github.com/HariSekhon/Kubernetes-configs\n\nUses adjacent script kubernetes_resource_types.sh\n\nTested on ArgoCD 2.0.3\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file.yaml> <file2.yaml> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\necho \"  namespaceResourceWhitelist:\"\n\"$srcdir/kubernetes_resource_types.sh\" \"$@\" |\nwhile read -r group kind; do\n    # Cluster resources, ignore these\n    if [[ \"$kind\" =~ Namespace|PriorityClass|StorageClass ]]; then\n        continue\n    fi\n    group=\"${group%/*}\"\n    if [ \"$group\" = v1 ]; then\n        group=\"\"\n    fi\n    if [ \"$group\" = \"\" ]; then\n        group=\"''\"\n    fi\n    echo \"  - group: $group\"\n    echo \"    kind: $kind\"\ndone\n"
  },
  {
    "path": "kubernetes/argocd_password.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-07 15:45:38 +0000 (Fri, 07 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gets the ArgoCD initial admin password from the environment or Kubernetes secret\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -n \"${ARGOCD_PASSWORD:-}\" ]; then\n    echo \"using \\$ARGOCD_PASSWORD from environment\" >&2\nelif kubectl get secret -n argocd argocd-initial-admin-secret &>/dev/null; then\n    ARGOCD_PASSWORD=\"$(kubectl -n argocd get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 --decode)\"\nfi\n\nif [ -z \"${ARGOCD_PASSWORD:-}\" ]; then\n    echo \"ERROR: failed to determine ARGOCD_PASSWORD from environment or kubernetes\" >&2\n    exit 1\nfi\n\n# if sourced, export ARGOCD_PASSWORD, if subshell, echo it\n#if [[ \"$_\" != \"$0\" ]]; then\n    export ARGOCD_PASSWORD\n#else\n    echo -n \"$ARGOCD_PASSWORD\"  # no newline so we can pipe straight to pbcopy / xclip or similar\n#fi\n"
  },
  {
    "path": "kubernetes/curl_k8s_ingress.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-03 18:11:23 +0100 (Thu, 03 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCurls the current Kubernetes cluster's Nginx Ingress controller's external IP address using the given URL\n\nUseful for debugging Ingress Routing + SSL certificates directly by bypassing DNS -> CDN addresses such as Cloudflare\n\nRequires kubectl to be installed and configured with the target cluster selected as the current context\nin order to find the Ingress Controller's external IP address\n\nTips:\n\n    - Make sure to specify the correct prefix eg. https:// and path suffix so that you don't hit a 302 redirect back to the CDN address\n    - Use '-kv' curl switches to see the cert and header info\n\nPrints the curl command inferred as it is used so you can see what you're doing\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"$1\"\nshift || :\n\nshopt -s nocasematch\n\nif ! [[ \"$url\" =~ :// ]]; then\n    url=\"http://$url\"\nfi\n\n#shopt -s extglob\n#host=\"${url#http?(s)://}\"\n#shopt -u extglob\n\nhost=\"${url#http://}\"\nhost=\"${host#https://}\"\n\n# strip /path\nhost=\"${host%%[;/]*}\"\n\n# if url prefix doesn't strip successfully, the above line results in host='https:'\nif [[ \"$host\" =~ : ]]; then\n    die \"Host parse failure from url: '$url', inferred host as '$host'\"\nfi\n\n# TODO: extend this to detect and return the IPs of other ingress controllers if deployed eg. older nginx-ingress, traefik, haproxy etc.\n#\n# XXX: on some clusters appears under '.spec.loadBalancerIP' and others '.status.loadBalancer.ingress[].ip' so try both\nIP=\"$(kubectl get svc ingress-nginx-controller -n ingress-nginx -o 'jsonpath={.spec.loadBalancerIP}{.status.loadBalancer.ingress[].ip}')\"\nif [ -z \"$IP\" ]; then\n    # support_msg defined in lib/utils-bourne.sh\n    # shellcheck disable=SC2154\n    die \"Failed to determine Kubernetes ingress IP address - possibly not using ingress-nginx, in which case this code needs extending. $support_msg\"\nfi\n\nif [[ \"$url\" =~ ^http:// ]]; then\n    port=80\nelse\n    port=443\nfi\n\n# this works for ingress path routing but not SSL verification, forcing you to use -k and not debugging the certificate\n#url=\"${url/:\\/\\/$host/://$IP}\"\n#curl -H \"Host: $host\" \"$url\" \"$@\"\n\ncmd=(curl \"$url\" --resolve \"$host:$port:$IP\" \"$@\")\necho \"${cmd[*]}\"\n\"${cmd[@]}\"\n"
  },
  {
    "path": "kubernetes/datree_kustomize_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-02 12:35:16 +0100 (Tue, 02 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all kustomization.yml/yaml files under the current or given path and runs datree kustomize test for each one\n\nRetains the highest number exit code to ensure no earlier errors are lost\n\nTo pass arguments to Kustomize instead of Datree, you must use --\n\nExample:\n\n    ${0##*/} . -- --enable-helm\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<path> <datree_options>]\"\n\ncheck_env_defined \"DATREE_TOKEN\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# XXX: this code has now been added to the official Datree docs:\n#\n#   https://hub.datree.io/integrations/kustomize-support#testing-multiple-kustomizations\n\npath=\"${1:-.}\"\nshift || :\n\nfinal_exit_code=0\n\nwhile read -r kustomization; do\n    dir=\"$(dirname \"$kustomization\")\"\n    echo \"Datree Kustomization Test: $kustomization\"\n    set +e\n    datree kustomize test \"$dir\" \"$@\"\n    exitcode=$?\n    set -e\n    if [ \"$exitcode\" -gt \"$final_exit_code\" ]; then\n        final_exit_code=\"$exitcode\"\n    fi\n    # Datree outputs enough space at the end of each run already\n    #echo\ndone < <(find \"$path\" -type f -name 'kustomization.y*ml')\n\nif [ \"$final_exit_code\" = 0 ]; then\n    echo \"Success\"\nelse\n    echo \"Violations found, returning exit code $final_exit_code\"\nfi\nexit \"$final_exit_code\"\n"
  },
  {
    "path": "kubernetes/helm_template.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-26 17:05:53 +0000 (Fri, 26 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTemplates a given Helm chart for local declaratively tracked and auto-repaired Kustomize deployments\n\nUsed to quickly update various GitOps configuration versions - see https://github.com/HariSekhon/Kubernetes-configs\n\n\nRequires helm to be installed\n\nTested on Helm 3.7.1\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<repo-url> <repo/chart> [--version <version> -n <namespace> -f values.yaml ... <helm_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nrepo_url=\"$1\"\nchart=\"$2\"\nshift || :\nshift || :\n\n# doesn't really matter what this repo name is as long as it's consistent and can infer what it should be from the chart call\nrepo=\"${chart%%/*}\"\nrelease_name=\"${chart##*/}\"  # for simplicity the release name can be the same as the chart, which is fine for most cases\n\nif ! helm repo list | grep -q \"^${repo}[[:space:]]\"; then\n    helm repo add \"$repo\" \"$repo_url\"\nfi\n\nhelm template \"$release_name\" \"$chart\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubeadm_join_cmd.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-14 16:30:08 +0100 (Fri, 14 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates the kubeadm join command for an already existing Kubernetes cluster where the initial kubeadm init join command token has already expired\n\nkubeadm is assumed to be working and available in the \\$PATH\n\nTested on Kubernetes 1.18.1\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntoken=\"$(kubeadm token generate)\"\n\nkubeadm token create \"$token\" --ttl 2h --print-join-command\n"
  },
  {
    "path": "kubernetes/kubeadm_join_cmd2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-14 16:30:08 +0100 (Fri, 14 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates the kubeadm join command for an already existing Kubernetes cluster where the initial kubeadm init join command token has already expired\n\nDetermines the certificate hash, and generates a new temporary token with which to join\n\nkubeadm is assumed to be working and available in the \\$PATH\n\nTested on Kubernetes 1.18.1\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[options]\n\n-m --master     Master host address (default: local hostname/fqdn)\n-p --port       Master port (default: 6443)\n-c --crt        CA crt file (default: /etc/kubernetes/pki/ca.crt)\n\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# defaults\nmaster=\"$(hostname -f)\"\nport=\"6443\"\nca_crt=\"/etc/kubernetes/pki/ca.crt\"\n\nwhile [ $# -gt 0 ]; do\n    case \"$1\" in\n      -m|--master)  master=\"$2\"\n                    shift\n                    ;;\n      -p|--port)    port=\"$2\"\n                    shift\n                    ;;\n         -c|--crt)  ca_crt=\"$2\"\n                    shift\n                    ;;\n                *)  usage\n                    ;;\n    esac\n    shift\ndone\n\nif ! is_int \"$port\"; then\n    usage \"port given is not an integer: $port\"\nfi\n\n# ca_crt path validated in here\ncrt_hash=\"$(\"$srcdir/../bin/crt_hash.sh\" \"$ca_crt\")\"\n\ntoken=\"$(kubeadm token create)\"\n\ncat <<EOF\nkubeadm join \\\\\n    --token \"$token\" \\\\\n    --discovery-token-ca-cert-hash \"sha256:$crt_hash\" \\\\\n    \"$master\":$port\nEOF\n"
  },
  {
    "path": "kubernetes/kubectl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-03 14:24:44 +0000 (Tue, 03 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a kubectl command safely fixed to a specific Kubernetes context by using an isolated fixed config for the lifetime of this script\n\nAvoids concurrency race conditions with other concurrently executing commands or scripts by avoiding using or changing the global kubectl context\n\nEg. running:\n\n    kubectl config use-context '<name>'\n\neither by your hand or in other concurrently executing scripts changes your global kubectl context to run on the given cluster, which could divert your command or concurrently long running scripts in other windows to run kubectl commands on the wrong cluster, leading to cross environment misconfigurations and real world outages (I've seen this personally)\n\nFor frequent more convenient usage you will want to shorten the CLI by copying this script to a local copy in each cluster's yaml config directory and hardcoding the CONTEXT variable\n\nThe kubectl context specified should already be configured in your primary kubectl config which is copied to an isolated config to fix it before running your kubectl actions\n\nCould also use explicit kubectl switches --cluster / --context, but this is more convenient, especially when hardcoded for the local copy in each cluster's k8s yaml dir\n\n\nSee Also:\n\n    - aws_kubectl.sh - similar to this script but also downloads the AWS EKS credential\n    - gke_kubectl.sh - similar to this script but also downloads the GCP GKE credential\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_context> <kubectl_options>\"\n\nhelp_usage \"$@\"\n\n# ============================================================\n# REMOVE AND HARDCODE THIS SECTION FOR SHORTER CLI convenience\nmin_args 2 \"$@\"\n\n# fixed to this kubectl context - thou shalt deploy to no other cluster context from this script\n\n# HARDCODE THIS for frequent shorter CLI usage\nCONTEXT=\"$1\"\n\n# REMOVE THIS IF HARDCODING\nshift || :\n# ============================================================\n\nkube_config_isolate\n\n# switch context if not already the current context (avoids repeating \"switching context\" output noise when this script it called iteratively in loop by other scripts)\nkube_context \"$CONTEXT\"\n\nkubectl \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_alpine.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick Alpine pod on Kubernetes to debug things\n\nShares the same alpine pod for successive invocations of this script for speed\n\nArguments become options to 'kubectl run'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nimage=\"alpine\"\n\nname=\"alpine-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_busybox.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick busybox pod on Kubernetes to debug networking / dns\n\nShares the same busybox pod for successive invocations of this script for speed\n\nArguments become options to 'kubectl run'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nimage=\"busybox\"\n\nname=\"busybox-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_container_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-21 15:30:28 +0100 (Thu, 21 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts the total number of running Docker containers on the current Kubernetes cluster in the current namespace\n\nSpecify the --namespace or use --all-namespaces to get the total across all namespaces, including the kube-system namespace\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<--namespace blah | --all-namespaces>\"\n\nhelp_usage \"$@\"\n\nkubectl get pods \"$@\" --field-selector status.phase=Running -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.name}{\"\\n\"}' |\nwc -l |\nsed 's/[[:space:]]*//'\n"
  },
  {
    "path": "kubernetes/kubectl_container_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-21 15:30:28 +0100 (Thu, 21 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes containers running counts by name sorted descending on the current Kubernetes cluster in the current namespace\n\nSpecify the --namespace or use --all-namespaces to get the total across all namespaces, including the kube-system namespace\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<--namespace blah | --all-namespaces>\"\n\nhelp_usage \"$@\"\n\nkubectl get pods \"$@\" --field-selector status.phase=Running -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.name}{\"\\n\"}' |\nsort |\nuniq -c |\nsort -k1nr\n"
  },
  {
    "path": "kubernetes/kubectl_create_namespaces.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-12 15:34:19 +0000 (Thu, 12 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReads kubernetes yaml from stdin, extracts all namespace names and creates the namespaces via kubectl in the current context\n\nThis is needed because on blank installs doing a 'kubectl diff' can result in the following error:\n\n    Error from server (NotFound): namespaces \\\"blah\\\" not found\n\nInstead you can first precreate the namespaces if they don't already exist, after which the diff will succeed:\n\n    ${0##*/} file.yaml file2.yaml ...\n\nor\n\n    kustomize build | ${0##*/}\n\n\nSince this script applies to the current kubectl context, it is best used as part of other scripts such as kustomize_diff_apply.sh where the kube config and context are isolated and set to avoid race conditions by depending on global kube config which could be concurrently naively changed during execution by other scripts/shells\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file.yaml> <file2.yaml> ...]\"\n\nhelp_usage \"$@\"\n\nno_more_opts \"$@\"\n\nnamespaces=\"$(awk '/^[[:space:]]*namespace:[[:space:]]*[a-z0-9]([-a-z0-9]*[a-z0-9])?[[:space:]]*$/{print $2}' \"$@\" | sort -u)\"\n\ncurrent_namespaces=\"$(kubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{\"\\n\"}')\"\n\nfor namespace in $namespaces; do\n    if grep -Fxq \"$namespace\" <<< \"$current_namespaces\"; then\n        echo \"namespace '$namespace' aleady exists\"\n    else\n        kubectl create namespace \"$namespace\"\n    fi\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_curl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick curl pod on Kubernetes with SSL support to debug websites / SSL / ingresses\n\nShares the same curl pod for successive invocations of this script for speed\n\nArguments become options to 'kubectl run'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nimage=\"curlimages/curl\"\n\nname=\"curl-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_delete_empty_namespaces.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-18 16:53:40 +0100 (Thu, 18 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds empty namespaces in the current Kubernetes cluster context and deletes them\n\nUses adjacent script kubectl_empty_namespaces.sh\n\nKubectl must be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\n\"$srcdir/kubectl_empty_namespaces.sh\" |\nwhile read -r namespace; do\n    timestamp \"Deleting empty namespace '$namespace'\"\n    kubectl delete namespace \"$namespace\"\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_deployment_pods.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-12 14:41:48 +0200 (Mon, 12 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the pods of a deployment by querying the deployment's spec.selector.matchLabels\nand then querying pods with matching selectors\n\nRequires kubectl to be installed and configured, as well as jq installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<deployment_name> [-n <namespace>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ndeployment=\"$1\"\nshift || :\n\nread -r -a label_args < <(\n    kubectl get deployment \"$deployment\" \"$@\" -o jsonpath='{.spec.selector.matchLabels}' |\n    jq -r 'to_entries[] | \"-l \\(.key)=\\(.value)\"'\n)\n\nkubectl get pods \"${label_args[@]}\" \"$@\" -o name\n"
  },
  {
    "path": "kubernetes/kubectl_diff_apply.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-18 11:09:30 +0000 (Wed, 18 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n# not using anything from here directly\n#. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns kubectl diff and apply, prompting to accept any changes before applying\n\nMust be given kubectl args to provide the input yaml(s), eg:\n\n    ${0##*/} -f file.yaml\n\n    kustomize build | ${0##*/} -f -\n\nHighly recommended to specify --context to avoid race conditions in global kube config which could apply to the wrong cluster. See kubectl.sh for more details\n\n    ${0##*/} -f file.yaml --context=gke_...\n\nSee Also:\n\n    kustomize_diff_apply.sh - runs kustomize build, precreates namespaces, and then runs this diff and apply\n\n    kubectl_create_namespaces.sh - creates any namespaces in the yaml inputs to allow an initial diff to succeed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_options>\"\n\nhelp_usage \"$@\"\n\nstdin=0\nyaml=\"\"\n\n# there is no space before -f in $* if given as first argument\n#if [[ \"$*\" =~ [[:space:]]-f[[:space:]]*- ]]; then\nif [[ \"$*\" =~ -f[[:space:]]*- ]]; then\n    stdin=1\nfi\n\nif [ $stdin -eq 1 ]; then\n    yaml=\"$(cat)\"\nfi\n\necho \"Diff from live running cluster:\"\necho\n\nif kubectl diff \"$@\" <<< \"$yaml\"; then\n    echo \"No changes to deploy\"\n    exit 0\nfi\n\necho\nread -r -p \"Deploy the above changes? (y/N) \" answer < /dev/tty  # < $(tty)\nshopt -s nocasematch\nif [[ ! \"$answer\" =~ ^y|yes$ ]]; then\n    echo \"Aborting...\"\n    exit 1\nfi\nshopt -u nocasematch\necho\necho\n\nkubectl apply \"$@\" <<< \"$yaml\"\n"
  },
  {
    "path": "kubernetes/kubectl_dnsutils.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick tutum/dnsutils pod on Kubernetes to debug networking / dns\n\nShares the same dnsutils pod for successive invocations of this script for speed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nimage=\"tutum/dnsutils\"\n\nname=\"dnsutils-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_empty_namespaces.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-18 16:53:40 +0100 (Thu, 18 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds empty namespaces in the current Kubernetes cluster context\n\nOutputs empty namespaces one per line to stdout, while outputting timestamped progress to stderr, so stdout can still be\npassed straight in a command pipe to act on those namespaces\n\nUsed by adjacent script kubectl_delete_empty_namespaces.sh\n\nKubectl must be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\ntimestamp \"Getting list of namespaces\"\n#kubectl get ns -o name |\n#sed 's|namespace/||' |\nkubectl get namespaces -o jsonpath='{range .items[*]}{.metadata.name}{\"\\n\"}' |\nwhile read -r namespace; do\n    if [[ \"$namespace\" =~ ^kube- ]]; then\n        continue\n    fi\n    timestamp \"Checking namespace '$namespace'\"\n    # quicker, but doesn't return all objects - should return the usual candidates though\n    objects=\"$(kubectl get all -n \"$namespace\" -o name)\"\n    # slow but safer\n    #objects=\"$(\"$srcdir/kubectl_get_all.sh\" -n \"$namespace\" | sed 's/#.*//| /^[[:space:]]*$/d')\"\n    if [ -z \"$objects\" ]; then\n        echo \"$namespace\"\n    fi\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_exec.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 11:06:23 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nKubectl exec's to the first pod matching the given regex name and optional pod filters\n\nUseful to not have to first look up deployment pod names which have random string suffixes\nand quickly jump in to any pod in a namespace/deployment to debug a web farm etc.\n\nShows the full auto-generated 'kubectl exec' command for clarity\n\nExecs to bash if available, otherwise /bin/sh\n\nFirst arg is the pod's name as an extended regex (ERE)\nOptional second arg is the container's name as an extended regex (ERE)\nSubsequent args from the first dash are passed straight to 'kubectl get pods' to set namespace, label filters etc.\nIf there is no 3rd arg onwards, passes --all-namespaces to 'kubectl get pods' in order to find a matching pod, otherwise you should specify --namespace\n\nExamples:\n\n    ${0##*/} nginx\n\n    ${0##*/} nginx -n prod\n\n    ${0##*/} nginx -n prod -l app=nginx\n\n    ${0##*/} nginx sidecar-container -n prod -l app=nginx\n\nSee also:\n\n    kubectl_exec2.sh\n\nfor a version with one less arg that works strictly on pod filters\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<pod_name_regex> [<container_name_regex>] [<pod_filters>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npod_regex=\"$1\"\nshift || :\n\ncontainer_regex=\".\"\n\nif [ $# -gt 0 ]; then\n    if ! [[ \"$1\" =~ ^- ]]; then\n        container_regex=\"$1\"\n        shift || :\n    fi\nfi\n\nkubectl cluster-info &>/dev/null || die \"Failed to connect to Kubernetes cluster!\"\n\npod=\"$(kubectl get pods \"${@:---all-namespaces}\" -o 'jsonpath={range .items[*]}{.metadata.namespace}{\"\\t\"}{.metadata.name}{\"\\n\"}' | tr ' ' '\\n' | grep -E -m 1 \"$pod_regex\" || :)\"\n\nif [ -z \"$pod\" ]; then\n    die \"No matching pods found, perhaps you forgot to pass the --namespace? (tip: specify -A / --all-namespaces if lazy, it'll filter by pod name)\"\nfi\n\n# auto-determine namespace because specifying it is annoying\n# done via extended jsonpath above now to make 1 less query\n#namespace=\"$(kubectl get pods --all-namespaces | grep -E \"^[^[:space:]]+[[:space:]]+[^[:space:]]*${pod}\" | awk '{print $1; exit}' || :)\"\n\n#if [ -z \"$namespace\" ]; then\n#    die \"failed to auto-determine namespace for pod '$pod'\"\n#fi\n\nnamespace=\"${pod%%[[:space:]]*}\"\npod=\"${pod##*[[:space:]]}\"\n\n# auto-determine container from regex if given or just take first container\ncontainer=\"$(kubectl get pods -n \"$namespace\" \"$pod\" -o 'jsonpath={range .spec.containers[*]}{.name}{\"\\n\"}' | grep -m 1 \"$container_regex\" | awk '{print $1}' || :)\"\n\nif [ -z \"$container\" ]; then\n    die \"failed to get container name matching regex '$container_regex' for pod '$pod'\"\nfi\n\ncmd=(kubectl exec -ti --namespace \"$namespace\" \"$pod\" --container \"$container\" -- /bin/sh -c 'if type bash >/dev/null 2>&1; then exec bash; else exec sh; fi')\necho \"${cmd[@]}\"\n\"${cmd[@]}\"\n"
  },
  {
    "path": "kubernetes/kubectl_exec2.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 11:06:23 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nKubectl exec's to the first pod matching the given filter criteria\n\nUseful to not have to first look up deployment pod names which have random string suffixes\nand quickly jump in to any pod in a namespace/deployment to debug a web farm etc.\n\nShows the full auto-generated 'kubectl exec' command for clarity\n\nExecs to bash if available, otherwise /bin/sh\n\nFirst arg is the optional pod container name (if no container is specified we'll pick the first one and show you in the kubectl output)\nSubsequent args from the first dash are passed straight to 'kubectl get pods' to set namespace, label filters etc.\n\nExamples:\n\n    ${0##*/} -n prod -l app=nginx\n\n    ${0##*/} sidecar-container -n prod -l app=nginx\n\n\nSee also:\n\n    kubectl_exec.sh\n\nfor a different approaching using just a partial pod name and optional partial container name, auto-determines the namespace, it's simpler and less typing in most cases\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<container_name>] [<pod_filters>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncontainer=\"\"\n\nif [ $# -gt 0 ]; then\n    if ! [[ \"$1\" =~ ^- ]]; then\n        container=\"$1\"\n        shift\n    fi\nfi\n\nkubectl cluster-info &>/dev/null || die \"Failed to connect to Kubernetes cluster!\"\n\npod=\"$(kubectl get pods \"$@\" -o 'jsonpath={.items[0].metadata.name}' 2>/dev/null || :)\"\n\nif [ -z \"$pod\" ]; then\n    die \"No matching pods found, perhaps you forgot to pass the --namespace? (tip: specify -A / --all-namespaces if lazy, it'll filter by pod name)\"\nfi\n\n# auto-determine namespace because specifying it is annoying\nnamespace=\"$(kubectl get pods --all-namespaces | grep -E \"^[^[:space:]]+[[:space:]]+[^[:space:]]*${pod}\" | awk '{print $1; exit}' || :)\"\n\nif [ -z \"$container\" ]; then\n    # auto-determine first container to show explicitly what we're connecting to\n    container=\"$(kubectl get pods -n \"$namespace\" \"$pod\" -o 'jsonpath={.spec.containers[*].name}' | grep -m 1 \".\" | awk '{print $1}' || :)\"\nfi\n\ncmd=(kubectl exec -ti --namespace \"$namespace\" \"$pod\" --container \"$container\" -- /bin/sh -c 'if type bash >/dev/null 2>&1; then exec bash; else exec sh; fi')\necho \"${cmd[@]}\"\n\"${cmd[@]}\"\n"
  },
  {
    "path": "kubernetes/kubectl_gcloud_sdk.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick GCloud SDK pod on Kubernetes to debug networking / dns\n\nShares the same gcloud-sdk pod for successive invocations of this script for speed\n\nArguments become options to 'kubectl run'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nimage=\"gcr.io/google.com/cloudsdktool/cloud-sdk\"\n\nname=\"gcloud-sdk-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_get_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-28 13:45:11 +0100 (Mon, 28 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFind all Kubernetes API namespaced resource types and queries the current namespace for all of them\n\nUseful to scan the current namespace for all resources since 'kubectl get all' only returns select API objects\n\nCan set KUBECTL_GET_ALL_SEPARATOR environment variable, useful like this:\n\n    KUBECTL_GET_ALL_SEPARATOR='---' ${0##*/} --all-namespaces -o yaml > dump.yaml\n\nUseful to be able to scan live Kubernetes objects with file linting tools like Pluto to detect deprecated live objects on the cluster affecting your Kubernetes cluster upgrades\n\n\nRequires Kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[-n <namespace> | -A | --all-namespaces] [ <kubectl_get_options> ]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nnon_gettable_resources=\"\nbindings\nlocalsubjectaccessreviews\nlocalsubjectaccessreviews.authorization.k8s.io\nselfsubjectaccessreviews.authorization.k8s.io\nselfsubjectrulesreviews.authorization.k8s.io\nsubjectaccessreviews.authorization.k8s.io\ntokenreviews.authentication.k8s.io\n\"\n\nif [[ \"$*\" =~ -A|--all-namespaces ]]; then\n    kubectl api-resources -o name\nelse\n    kubectl api-resources --namespaced=true -o name\nfi |\nsort -fu |\nwhile read -r resource; do\n    # Error from server (MethodNotAllowed): the server does not allow this method on the requested resource\n    for non_gettable_resource in $non_gettable_resources; do\n        [ \"$resource\" = \"$non_gettable_resource\" ] && continue 2\n    done\n    if [ -n \"${KUBECTL_GET_ALL_SEPARATOR:-}\" ]; then\n        echo \"$KUBECTL_GET_ALL_SEPARATOR\"\n    fi\n    echo \"# $resource:\" >&2\n    kubectl get \"$resource\" \"$@\" 2>&1 |\n    sed '/No resources found/d'\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_get_annotation.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-01 14:51:44 +0100 (Mon, 01 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns objects of a given type that have the given annotation set, optionally to a given value\n\nFiltering by label is easy, but when you want to filter by annotation this is trickier\n\nThis script makes it easy to do in the current namespace\n\nOutput:\n\n<namespace>    <name>    <annotation>=<value>\n\n\nKubectl needs to be installed in the \\$PATH and configured with the right context\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<object_kind> <annotation> [<value> <kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nkind=\"$1\"\nannotation=\"$2\"\nvalue=\"${3:-}\"\nshift || :\nshift || :\nshift || :\n\nannotation_escaped=\"${annotation//./\\\\.}\"\nvalue_escaped=\"${value//\\\"/\\\\\\\"}\"\n\n# can't decompose this line across lines unfortunately for clarity, fails to parse\nif [ -n \"$value\" ]; then\n    kubectl get \"$kind\" \"$@\" -o jsonpath=\"{range .items[?(@.metadata.annotations.$annotation_escaped==\\\"$value_escaped\\\")]}{.metadata.namespace}{\\\"\\\\t\\\"}{.metadata.name}{\\\"\\\\t$annotation=\\\"}{.metadata.annotations.$annotation_escaped}{\\\"\\\\n\\\"}\" | column -t\nelse\n    kubectl get \"$kind\" \"$@\" -o jsonpath=\"{range .items[?(@.metadata.annotations.$annotation_escaped)]}{.metadata.namespace}{\\\"\\\\t\\\"}{.metadata.name}{\\\"\\\\t$annotation=\\\"}{.metadata.annotations.$annotation_escaped}{\\\"\\\\n\\\"}\" | column -t\nfi\n"
  },
  {
    "path": "kubernetes/kubectl_image_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-09 11:37:14 +0100 (Wed, 09 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes container images running counts sorted descending across all namespaces\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkubectl get pods --all-namespaces --field-selector status.phase=Running -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{\"\\n\"}' |\nsort |\nuniq -c |\nsort -k1nr\n"
  },
  {
    "path": "kubernetes/kubectl_image_deployments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-09 11:37:14 +0100 (Wed, 09 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes container images running on the cluster along with their deployments, statefulsets of daemonsets that they belong to\n\nUseful to find which deployment, stateful or daemonset to upgrade such hunting down an images such as when the kubernetes project deprecated the k8s.gcr.io registry in favour of registry.k8s.io\n\nOutput:\n\n<api>    <kind>    <namespace>    <name>    <image:tag>\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkubectl get deploy,sts,ds --all-namespaces -o json |\n# jq trick {} expands out the sub-array elements\njq -r '\n    .items[] |\n        {\n            \"api\":.apiVersion,\n            \"kind\": .kind,\n            \"namespace\": .metadata.namespace,\n            \"name\": .metadata.name,\n            # jq trick having a list here causes the rest of this map to be duplicated one per list item\n            \"images\": [.spec.template.spec.initContainers[]?.image, .spec.template.spec.containers[]?.image] | flatten[]\n        } |\n        [\n            .api,\n            .kind,\n            .namespace,\n            .name,\n            .images\n        ] | @tsv\n' |\nsort -k3 -u |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kubectl_images.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-09 11:37:14 +0100 (Wed, 09 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes container images running on the current cluster\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n# some similar queries:\n#\n# https://kubernetes.io/docs/tasks/access-application-cluster/list-all-running-container-images/\n\nkubectl get pods --all-namespaces --field-selector status.phase=Running -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{\"\\n\"}' |\nsort -u\n"
  },
  {
    "path": "kubernetes/kubectl_jobs_delete_stuck.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-11 15:45:57 +0100 (Fri, 11 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds and deletes stuck Kubernetes jobs using the criteria specified in the adjacent script kubernetes_find_stuck_jobs.sh which this script calls\n\nShows the jobs and prompts for confirmation before deleting them\n\nThis assumes that your jobs are being run by cronjobs and will be recreated\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nstuck_jobs=\"$(kubectl_jobs_stuck.sh \"$@\")\"\nheader=\"$(head -n1 <<< \"$stuck_jobs\")\"\nstuck_jobs=\"$(tail -n +2 <<< \"$stuck_jobs\")\"\n\nif [ -z \"$stuck_jobs\" ]; then\n    exit 0\nfi\n\necho \"Jobs to delete:\"\necho\necho \"$header\"\necho \"$stuck_jobs\"\necho\nread -r -p \"Are you sure you want to delete these jobs? (y/N) \" answer\n\nshopt -s nocasematch\n\nif ! [[ \"$answer\" =~ y|yes ]]; then\n    timestamp \"Aborting...\"\n    exit 1\nfi\n\ntimestamp \"Deleting stuck kubernetes jobs\"\n\nawk '!/^NAME/{print $1}' <<< \"$stuck_jobs\" |\nxargs kubectl delete jobs \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_jobs_stuck.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-11 15:37:03 +0100 (Fri, 11 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds stuck Kubernetes jobs based on them matching all of following criteria:\n\n1. 0 completions\n2. duration is in hours or days\n3. duration matches age (has been stuck since start)\n\nArgs are passed to kubectl as per normal (eg. specify -n for namespace)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nkubectl get jobs \"$@\" |\nwhile read -r name completions duration age; do\n    if [[ \"$completions\" =~ ^0 ]] &&\n       [[ \"$duration\" =~ h|d|[[:digit:]]{3}m ]] &&\n       [ \"$duration\" = \"$age\" ]; then\n        echo \"$name $completions $duration $age\"\n    elif [ \"$name\" = NAME ]; then\n        echo \"$name $completions $duration $age\"\n    fi\ndone |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kubectl_kv_to_secret.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-03-16 18:40:25 +0000 (Wed, 16 Mar 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Kuberbetes secret from key=value args or stdin\n\nIf no third or more arguments are given, reads environment variables from standard input, one per line in 'key=value' format or 'export key=value' shell format\n\nExamples:\n\n    ${0##*/} mynamespace mysecret AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} mynamespace mysecret\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} mynamespace mysecret\n\n\nSee Also:\n\n    kubectl_secret_values.sh - dumps all key value pairs, base64 decoded, for a given Kubernetes secret\n\n\n\nRequires kubectl to be installed and available in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<k8s_namespace> <k8s_secret> [<key1>=<value1> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nnamespace=\"$1\"\nsecret_name=\"$2\"\nshift || :\nshift || :\n\nkube_config_isolate\n\nkey_values=()\n\nadd_kv(){\n    local key_value=\"$1\"\n    parse_export_key_value \"$key_value\"\n    key_values+=(\"--from-literal=$key=$value\")\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_kv \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_kv \"$line\"\n    done\nfi\n\nkubectl create secret generic -n \"$namespace\" \"$secret_name\" \"${key_values[@]}\"\n"
  },
  {
    "path": "kubernetes/kubectl_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-11 14:44:05 +0100 (Thu, 11 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTails logs from several pods a the same time in the current or given namespace\n\nUseful for debugging and quickly checking which components in a distributed app are having issues\n\nCan optionally specify an ERE regex filter on the pod name\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <filter>]\"\n\nnamespace=\"${1:-}\"\nfilter=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\n# kill all child processes by session id on cleanup, which is less likely to be changed than parent id\nif is_mac; then\n    # want late evaluation\n    # shellcheck disable=SC2016\n    trap_cmd 'kill $(ps -g $$ -o pid=) 2>/dev/null'\nelse\n    # shellcheck disable=SC2016\n    trap_cmd 'kill $(ps -s $$ -o pid=) 2>/dev/null'\nfi\n\n# avoiding subshell pipe into while loop otherwise 'wait' below won't wait\nwhile read -r pod_name; do\n    # pod_name is like pod/argocd-server-7558b87667-4fmx7\n    # and can be passed to kubectl as is\n    kubectl logs --tail 10 -f \"$pod_name\" --all-containers=true &\ndone < <( kubectl get pods -o name | grep -E \"$filter\" || : )\n\nwait\n"
  },
  {
    "path": "kubernetes/kubectl_node_labels.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-19 10:28:40 +0100 (Mon, 19 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes nodes and their labels, one per line (more convenient for shell piping)\n\nOutput Format:\n\n<node_name>     <label_key>=<label_value>\n<node_name>     <label_key2>=<label_value2>\n<node_name2>     <label_key>=<label_value>\n...\n\n\nThis format is much easier to read and work with while scripting than the single line monstrocity returned by:\n\n    kubectl get nodes --show-labels\n\n\nRequires kubectl to be installed and configured\n\n\nIf you only want to see a a list of unique available labels:\n\n    ${0##*/} | awk '{print \\$2}' | sort -u\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\nkubectl get nodes -o json |\njq -r '\n    .items[] |\n    { \"name\": .metadata.name, \"label\": .metadata.labels | to_entries[] } |\n    select(.label) |\n    [ .name, .label.key + \"=\" + .label.value ] |\n    @tsv\n' |\nsort -k1,2\n"
  },
  {
    "path": "kubernetes/kubectl_node_taints.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 20:35:30 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes nodes and their taints, one taint per line\n\nOutput Format:\n\n<node_name>     <key>=<value>:<effect>\n\n\nRequires kubectl to be installed and configured\n\n\nSee Also:\n\n    kubectl_node_labels.sh - list nodes and their labels, one label per line\n\n    kubectl taint nodes -l key=value <taint_spec>\neg.\n    kubectl taint nodes -l cloud.google.com/gke-preemptible=true preemptible=true:NoSchedule\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nnum_args 0 \"$@\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\nkubectl get nodes -o json |\njq -r '\n    .items[] |\n    { \"name\": .metadata.name, \"taint\": .spec.taints[]? } |\n    select(.taint) |\n    [ .name, .taint.key + \"=\" + .taint.value + \":\" + .taint.effect ] |\n    @tsv\n'\n"
  },
  {
    "path": "kubernetes/kubectl_pod_count.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-10-21 15:30:28 +0100 (Thu, 21 Oct 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts the total number of running pods on the current Kubernetes cluster in the current namespace\n\nSpecify the --namespace or use --all-namespaces to get the total across all namespaces, including the kube-system namespace\n\nRequires kubectl to be in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<--namespace blah | --all-namespaces>\"\n\nhelp_usage \"$@\"\n\nkubectl get pods \"$@\" --field-selector status.phase=Running -o jsonpath='{range .items[*]}{.metadata.name}{\"\\n\"}' |\nwc -l |\nsed 's/[[:space:]]*//'\n"
  },
  {
    "path": "kubernetes/kubectl_pod_ips.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-27 15:56:59 +0100 (Tue, 27 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes pods and their pod IPs, one per line (more convenient for shell piping)\n\nSpecify kubectl options like --namespace '...' or --all-namespaces are arguments like you normally would\n\nOutput Format:\n\n<namespace>     <pod_name>     <ip>\n<namespace>     <pod_name2>    <ip>\n...\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\nkubectl get pods -o json \"$@\" |\njq -r '\n    .items[] |\n    [.metadata.namespace, .metadata.name, .status.podIP] |\n    @tsv\n' |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kubectl_pod_labels.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-27 15:56:59 +0100 (Tue, 27 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes pods and their labels, one per line (more convenient for shell piping)\n\nSpecify kubectl options like --namespace '...' or --all-namespaces are arguments like you normally would\n\nOutput Format:\n\n<namespace>     <pod_name>     <label_key>=<label_value>\n<namespace>     <pod_name>     <label_key2>=<label_value2>\n<namespace>     <pod_name2>    <label_key>=<label_value>\n...\n\n\nThis format is much easier to read and work with while scripting than multiple labels return on a single line by:\n\n    kubectl get pods --show-labels\n\n\nRequires kubectl to be installed and configured\n\n\nIf you only want to see a a list of unique available labels:\n\n    ${0##*/} | awk '{print \\$2}' | sort -u\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\nkubectl get pods -o json \"$@\" |\njq -r '\n    .items[] |\n    {\"namespace\": .metadata.namespace, \"name\": .metadata.name, \"label\": .metadata.labels | to_entries[] } |\n    [.namespace, .name, .label.key + \"=\" + .label.value ] |\n    @tsv\n' |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kubectl_pods_colocated.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 16:37:29 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFind Kubernetes pods belonging to the same deployments / statefulsets that are colocated on the same nodes to check up on pod anti-affinity scheduling against sharing nodes\n\nTakes an ERE regex argument for deployments / statefulsets to check (use \\\".\\\" or leave blank to check all of them)\n\nSubsequent arguments are passed straight to kubectl to be able to specify namespaces, labels etc.\n\nTo check for any colocated placements of any deployments or statefulsets across all namespaces:\n\n    ${0##*/} . --all-namespaces\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<deployment_or_statefulset_regex>] [<kubectl_options]>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nregex=\"${1:-.}\"\nshift || :\n\nif is_mac; then\n    # use GNU uniq to get the -D switch to print all repeated lines\n    uniq(){\n        guniq \"$@\"\n    }\nfi\n\nfor x in deploy sts; do\n    kubectl get \"$x\" \"$@\" -o name |\n    grep -E \"$regex\" |\n    sed 's|.*\\/||' |\n    while read -r name; do\n        # awk strips the last 2 cols, adds 1st column namespace to end so uniq can check it too\n        # sort by the node column 8\n        # show lines with duplicate by node\n        # strip last column which is duplicate of namespace column\n        kubectl get po \"$@\" -o wide |\n        grep -E \"^[^[:space:]]+[[:space:]]+$name-[[:alnum:]]+-[[:alnum:]]+[[:space:]]\" |\n        awk 'NF{NF-=2}; {print $0\" \"$1}' |\n        column -t |\n        sort -k8 |\n        uniq -f7 -D |\n        awk 'NF{NF-=1}1'\n    done\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_pods_dump_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-23 10:09:56 +0200 (Fri, 23 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns adjacent kubectl_pod_dump_*.sh scripts a given number of times at given intervals\n\nThis is useful to collect support debug information to give to upstream support vendors\n\nnum_iterations    - defaults to 1\ninterval_seconds  - defaults to 300 seconds\n\nIf a JDK is specified as a 5th argument then runs an optional\n\n    kubectl_pods_dump_jstacks.sh\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jdk> [<num_iterations> <interval_seconds> <namespace> <pod_name_regex>]\"\n\nhelp_usage \"$@\"\n\nmin_args 0 \"$@\"\nmax_args 5 \"$@\"\n\njdk=\"$1\"\nnum_iterations=\"${2:-1}\"\ninterval_seconds=\"${3:-300}\"\nnamespace=\"${4:-}\"\npod_name_regex=\"${5:-.}\"\n\nif ! is_int \"$num_iterations\"; then\n    die \"First arg - num_iterations - must be an integer!\"\nfi\n\nif ! is_int \"$interval_seconds\"; then\n    die \"Second arg - interval_seconds - must be an integer!\"\nfi\n\nif ! [ \"$interval_seconds\" -gt 0 ]; then\n    die \"Second arg - interval_seconds - must be an integer greater than zero!\"\nfi\n\n# canonicalize to full path so we can find it from a new support-bundle dir\njdk=\"$(readlink -f \"$jdk\")\"\n\nsupport_bundle_dir=\"support-bundle-$(date '+%F_%H%M')\"\n\nmkdir -p -v \"$support_bundle_dir\"\n\ncd \"$support_bundle_dir\"\n\nif [ -n \"$jdk\" ]; then\n    if ! [ -f \"$jdk/bin/jstack\" ]; then\n        die \"Fifth arg - jdk - must be a directory containing a JDK with jstack!\"\n    fi\n    if ! [ -x \"$jdk/bin/jstack\" ]; then\n        die \"jstack within given jdk directory is not executable!\"\n    fi\nfi\n\nfor ((i=1; i <= num_iterations; i++)); do\n    timestamp \"Running iteration $i/$num_iterations of dump all:\"\n    echo\n    cmd=(\"$srcdir/kubectl_pods_dump_jstacks.sh\" \"$jdk\" ${namespace:+\"$namespace\" \"$pod_name_regex\"})\n    timestamp \"Running: ${cmd[*]}\"\n    \"${cmd[@]}\"\n    echo\n    cmd=(\"$srcdir/kubectl_pods_dump_stats.sh\" ${namespace:+\"$namespace\" \"$pod_name_regex\"})\n    timestamp \"Running: ${cmd[*]}\"\n    \"${cmd[@]}\"\n    echo\n    cmd=(\"$srcdir/kubectl_pods_dump_logs.sh\" ${namespace:+\"$namespace\" \"$pod_name_regex\"})\n    timestamp \"Running: ${cmd[*]}\"\n    \"${cmd[@]}\"\n    echo\n    timestamp \"Finished iteration $i of dump all\"\n    if [ \"$num_iterations\" -gt \"$i\" ]; then\n        timestamp \"Waiting for $interval_seconds seconds before next dump iteration\"\n        sleep \"$interval_seconds\"\n    fi\n    echo\n    echo\ndone\ntimestamp \"All dump iterations completed\"\necho >&2\ntimestamp \"Tarring $support_bundle_dir\"\ncd ..\ntarball=\"$support_bundle_dir.tar.gz\"\ntar czvf \"$tarball\" \"$support_bundle_dir\"\necho >&2\ntimestamp \"Support bundle ready: $tarball\"\n"
  },
  {
    "path": "kubernetes/kubectl_pods_dump_jstacks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-22 11:58:37 +0200 (Thu, 22 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps Kubernetes Java JStacks to text files for every pod that matches a given regex in the given namespace\n\nUseful for debugging Spark jobs on Kubernetes\n\n\nDumps command outputs to files of this name format:\n\nkubectl-pod-jstack.YYYY-MM-DD-HHSS.POD_NAME.txt\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<jdk_dir> [<namespace> <pod_name_regex>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 3 \"$@\"\n\njdk=\"$1\"\nnamespace=\"${2:-}\"\npod_name_regex=\"${3:-.}\"\n\nif ! [ -f \"$jdk/bin/jstack\" ]; then\n    die \"Fifth arg - jdk - must be a directory containing a JDK with jstack!\"\nfi\nif ! [ -x \"$jdk/bin/jstack\" ]; then\n    die \"jstack within given jdk directory is not executable!\"\nfi\n\nkube_config_isolate\n\nif [ -n \"$namespace\" ]; then\n    echo \"Switching to namespace '$namespace'\";\n    kubectl config set-context \"$(kubectl config current-context)\" --namespace \"$namespace\"\n    echo\nfi\n\nkubectl get pods -o name |\nsed 's|pod/||' |\ngrep -E \"$pod_name_regex\" |\nwhile read -r pod; do\n    echo\n    #kubectl exec\"$pod\" -- \"rm -fr /tmp/jdk\"\n    # want expansion in pod not local shell, and ignore && && || it works\n    # shellcheck disable=SC2016,SC2015\n    timestamp \"Dumping pod jstack: $pod\" &&\n    output_file=\"kubectl-pod-jstack.$(date '+%F_%H%M').$pod.txt\" &&\n    timestamp \"Copying jdk '$jdk' to pod /tmp\" &&\n    kubectl cp \"$jdk/\" \"$pod\":/tmp/jdk &&\n    timestamp \"Dumping JStack inside pod\" &&\n    kubectl exec \"$pod\" -- \\\n        bash -c '\n            java_pid=\"$(pgrep java | tee /dev/stderr)\"\n            java_pids=($java_pid)\n            if [ \"${#java_pids[@]}\" -gt 1 ]; then\n                  echo \"WARNING: more than one Java PID returned: ${java_pids[*]}\"\n            elif [ \"${#java_pids[@]}\" -eq 0 ]; then\n                echo \"WARNING: no Java PID found (perhaps you need to regex filter to only pods running Java processes?\" |\n                tee /tmp/jstack-output.txt /dev/stderr\n                exit 0\n            fi\n            /tmp/jdk/bin/jstack \"$java_pid\" > /tmp/jstack-output.txt\n        ' &&\n    timestamp \"Copying /tmp/jstack-output.txt to local machine\" &&\n    kubectl cp \"$pod\":/tmp/jstack-output.txt \"$output_file\" &&\n    timestamp \"Dumped pod jstack to file: $output_file\" ||\n    warn \"Failed to collect jstack for pod '$pod'\"\n    # XXX: because race condition - pods can go away during execution and we still want to collect the rest of the pods\ndone\ntimestamp \"JStack dumps completed\"\n"
  },
  {
    "path": "kubernetes/kubectl_pods_dump_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-22 11:58:37 +0200 (Thu, 22 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps Kubernetes pod logs to text files for every pod that matches a given regex in the given namespace\n\nUseful for debugging Spark jobs on Kubernetes\n\n\nDumps command outputs to files of this name format:\n\nkubectl-pod-log.YYYY-MM-DD-HHSS.POD_NAME.txt\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <pod_name_regex>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nnamespace=\"${1:-}\"\npod_name_regex=\"${2:-.}\"\n\nkube_config_isolate\n\nif [ -n \"$namespace\" ]; then\n    echo \"Switching to namespace '$namespace'\";\n    kubectl config set-context \"$(kubectl config current-context)\" --namespace \"$namespace\"\n    echo\nfi\n\nkubectl get pods -o name |\nsed 's|pod/||' |\ngrep -E \"$pod_name_regex\" |\nwhile read -r pod; do\n    echo\n    tstamp=\"$(date '+%F_%H%M')\"\n    # ignore && && || it works\n    # shellcheck disable=SC2015\n    timestamp \"Dumping pod stdout log: $pod\" &&\n    stdout_file=\"kubectl-pod-log.$tstamp.$pod.txt\" &&\n    kubectl logs \"$pod\" > \"$stdout_file\" &&\n    timestamp \"Dumped pod stdout log to file: $stdout_file\" ||\n    warn \"Failed to collect stdout log for pod '$pod'\"\n    # XXX: because race condition - pods can go away during execution and we still want to collect the rest of the pods\n\n    #for log in messages dmesg; do\n    #    log_file=\"kubectl-$log-log.$tstamp.$pod.txt\"\n    #    timestamp \"Dumping pod $log log: $pod\" &&\n    #    kubectl cp \"$pod\" \"/var/log/$log\" \"$log_file\" &&\n    #    timestamp \"Dumped pod $log log to file: $log_file\" ||\n    #    warn \"Failed to collect stdout log for pod '$pod'\"\n    #done\ndone\necho\ntimestamp \"Log dumps completed\"\n"
  },
  {
    "path": "kubernetes/kubectl_pods_dump_stats.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-22 11:58:37 +0200 (Thu, 22 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDumps common Linux command outputs for every pod that matches a given regex in the given namespace\n\nUseful for debugging Spark jobs on Kubernetes\n\n\nDumps command outputs to files of this name format:\n\nkubectl-pod-dump-output.YYYY-MM-DD-HHSS.POD_NAME.txt\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <pod_name_regex>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nnamespace=\"${1:-}\"\npod_name_regex=\"${2:-.}\"\n\nkube_config_isolate\n\nif [ -n \"$namespace\" ]; then\n    echo \"Switching to namespace '$namespace'\";\n    kubectl config set-context \"$(kubectl config current-context)\" --namespace \"$namespace\"\n    echo\nfi\n\nkubectl get pods -o name |\nsed 's|pod/||' |\ngrep -E \"$pod_name_regex\" |\nwhile read -r pod; do\n    echo\n    # ignore && && || it works\n    # shellcheck disable=SC2015\n    timestamp \"Running stats commands on pod: $pod\" &&\n    output_file=\"kubectl-pod-stats.$(date '+%F_%H%M').$pod.txt\" &&\n    # Copied from ../bin/dump_stats.sh for servers\n    #\n    # Most of these won't be available inside a pod, but we can only try...\n    #\n    # exec'ing stderr to be send to stdout to be captured in the log sections of why the stats are not available eg.\n    #\n    #   bash: line 25: iostat: command not found\n    #   bash: line 35: mpstat: command not found\n    #   bash: line 39: sar: command not found\n    #   bash: line 44: sar: command not found\n    #   bash: line 68: netstat: command not found\n    #   bash: line 73: lsof: command not found\n    #\n    kubectl exec \"$pod\" -- bash -c '\n        exec 2>&1\n\n        echo \"Dumping common command outputs\"\n        echo\n        echo \"Disk Space:\"\n        echo\n        df -h\n        echo\n        echo\n        echo \"Uname:\"\n        echo\n        uname -a\n        echo\n        echo\n        echo \"Uptime:\"\n        echo\n        uptime\n        echo\n        echo\n        echo \"RAM in GB:\"\n        echo\n        free -g\n        echo\n        echo\n        echo \"IOstat:\"\n        echo\n        iostat -x 1 5\n        echo\n        echo\n        echo \"lsblk:\"\n        echo\n        lsblk\n        echo\n        echo\n        echo \"MPstat:\"\n        echo\n        mpstat -P ALL 1 5\n        echo\n        echo\n        echo \"SAR 1 sec intervals x 5:\"\n        echo\n        sar -u 1 5\n        echo\n        echo\n        echo \"SAR -A:\"\n        echo\n        sar -A\n        echo\n        echo\n        echo \"Top snapshot with Threads:\"\n        echo\n        top -H -b -n 1\n        echo\n        echo\n        echo \"VMstat:\"\n        echo\n        vmstat 1 5\n        echo\n        echo\n        echo \"Process List:\"\n        echo\n        ps -ef\n        echo\n        echo\n        echo \"Process List Alternative - ps auxf:\"\n        echo\n        ps auxf\n        echo\n        echo\n        echo \"Netstat:\"\n        echo\n        netstat -an\n        echo\n        echo\n        echo \"LSOF:\"\n        echo\n        lsof -n -O\n        echo\n        echo\n        echo \"Dmesg:\"\n        echo\n        dmesg\n    ' >\"$output_file\" &&\n    echo &&\n    timestamp \"Dumped stats commands outputs to file: $output_file\" &&\n    echo &&\n    echo ||\n    warn \"Failed to collect stats for pod '$pod'\"\n    # XXX: because race condition - pods can go away during execution and we still want to collect the rest of the pods\ndone\ntimestamp \"Stats dumps completed\"\n"
  },
  {
    "path": "kubernetes/kubectl_pods_important.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-04-16 16:14:04 +0100 (Fri, 16 Apr 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nList important Kubernetes pods with their nodes to check on if they are obeying the scheduling requirements you want\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nimportant_pods_regex=\"\nkube-dns\ningress-nginx\nnginx-ingress\ncert-manager\njenkins\nteamcity\n-[[:digit:]]+$\n\"\n\nimportant_pods_regex=\"$(sed '/^[[:space:]]*$/d' <<< \"$important_pods_regex\" | tr '\\n' '|')\"\nimportant_pods_regex=\"${important_pods_regex%|}\"\n\nkubectl get pods --all-namespaces -o wide |\ngrep -Ei \"$important_pods_regex\"\n"
  },
  {
    "path": "kubernetes/kubectl_pods_per_node.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-08 09:51:05 +0100 (Tue, 08 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets the number of pods per node sorted descending\n\nRequires kubectl in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\n#kubectl get pods --all-namespaces -o json |\n#jq -r '.items[] | .spec.nodeName' |\nkubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.spec.nodeName}{\"\\n\"}' |\nsort |\nuniq -c |\nsort -k1nr\n"
  },
  {
    "path": "kubernetes/kubectl_pods_running_with_labels.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-20 17:52:17 +0200 (Tue, 20 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes running pods with labels matching key=value pair arguments, returning one pod name per line (convenient for shell piping)\n\nAll arguments after the first dash prefixed argument are treated as options to kubectl. This is where you should specify -n <namespace> or --all-namespaces\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"key=value key2=value2 ... [<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nkube_config_isolate\n\nlabel_args=()\n\nvalidate_label(){\n    local arg=\"$1\"\n    if ! [[ \"$arg\" =~ ^[[:alnum:]-]+=[[:alnum:]-]+$ ]]; then\n        die \"Invalid label key=value pair given (does not match regex validation): $arg\"\n    fi\n}\n\nuntil [ $# -lt 1 ]; do\n    case $1 in\n        -*) break\n            ;;\n         *) validate_label \"$1\"\n            label_argss+=(-l \"$1\")\n            ;;\n    esac\n    shift || :\ndone\n\nkubectl get pods \"${label_args[@]}\" \\\n                 --field-selector=status.phase=Running \\\n                 -o json \"$@\" | \\\n  jq -r '\n    .items[] |\n    select(.status.containerStatuses[0].state.running != null) |\n    select(.spec.containers[].image |\n      contains(\"artifacthub.informaticacloud.com\")) |\n    .metadata.name\n  '\n"
  },
  {
    "path": "kubernetes/kubectl_port_forward.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-01 14:06:16 +0200 (Sun, 01 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches kubectl port-forward to a pod\n\nOptional second argument may specify a grep ERE regex on the pod line or\na more specific key=value kubernetes label (the latter is preferable)\n\nIf more than one matching pod is found, prompts with an interactive dialogue to choose one\n\nIf more than one pod port is found, you will be prompted with another interactive dialog\nunless you can define the environment variable POD_PORT instead\n\nIf OPEN_URL environment variable is set and this script is not run over SSH then automatically\nopens the UI on localhost URL in the default browser\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <pod_line_ERE_regex_or_key=value_label>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nnamespace=\"${1:-}\"\nfilter=\"${2:-}\"\n#filter_label=()\nfilter_label=\"\"\ngrep_filter=\"\"\n\nif [[ \"$filter\" =~ = ]]; then\n    #filter_label=(-l \"$filter\")\n    #\n    # this works on Bash 3 set -u but breaks on Bash 4\n    #\n    #   \"${filter_label[@]}\"\n    #\n    # the fix on Bash 4 is\n    #\n    #   \"${filter_label[@:-]}\"\n    #\n    # but that isn't recognized on Bash 3\n    #\n    # portable workaround, convert to string and don't quote\n    #\n    filter_label=\"-l $filter\"\nelif [ -n \"$filter\" ]; then\n    grep_filter=\"$filter\"\nfi\n\nkube_config_isolate\n\ntimestamp \"Getting pods that match filter '$filter'\"\n# need splitting due to above behavioural difference between Bash 3 and Bash 4\n# shellcheck disable=SC2086\npods=\"$(\n    kubectl get pods ${namespace:+-n \"$namespace\"} \\\n                     $filter_label \\\n                     --field-selector=status.phase=Running |\n    if [ -n \"$grep_filter\" ]; then\n        grep -E \"$grep_filter\"\n    else\n        cat\n    fi |\n    tail -n +2\n)\"\n\nif [ -z \"$pods\" ]; then\n    die \"No matching pods found\"\nfi\n\nnum_lines=\"$(wc -l <<< \"$pods\")\"\n\nif [ \"$num_lines\" -eq 1 ]; then\n    timestamp \"Only one matching Kubernetes pod found\"\n    pod=\"$(awk '{print $1}' <<< \"$pods\")\"\nelif [ \"$num_lines\" -gt 1 ]; then\n    timestamp \"Multiple Kubernetes pods found, launching selection menu\"\n    menu_items=()\n    while read -r line; do\n        menu_items+=(\"$line\" \"\")\n    done <<< \"$pods\"\n    if [ \"${#menu_items[@]}\" -eq 0 ]; then\n        die \"Pod menu generation failed: empty\"\n    fi\n    chosen_pod=\"$(dialog --menu \"Choose which Kubernetes pod to forward to:\" \"$LINES\" \"$COLUMNS\" \"$LINES\" \"${menu_items[@]}\" 3>&1 1>&2 2>&3)\"\n    if [ -z \"$chosen_pod\" ]; then\n        timestamp \"Cancelled, aborting...\"\n        exit 1\n    fi\n    pod=\"$(awk '{print $1}' <<< \"$chosen_pod\")\"\nelse\n    die \"ERROR: No matching pods found\"\nfi\n\nif [ -n \"${POD_PORT:-}\" ]; then\n    if ! is_port \"$POD_PORT\"; then\n        die \"Environment variable POD_PORT must be a valid port number integer\"\n    fi\n    pod_port=\"$POD_PORT\"\nelse\n    pod_port=\"$(kubectl get pod  ${namespace:+-n \"$namespace\"} \"$pod\" -o jsonpath='{.spec.containers[*].ports[*].containerPort}')\"\nfi\n\nif [ -z \"$pod_port\" ]; then\n    die \"Failed to determine port for pod '$pod'\"\nfi\n\nif [ \"$(awk '{print NF}' <<< \"$pod_port\")\" -gt 1 ]; then\n    timestamp \"More than one port found on the pod, prompting for selection\"\n    menu_items=()\n    while read -r line; do\n        menu_items+=(\"$line\" \"\")\n    done < <(tr ' ' '\\n' <<< \"$pod_port\" | sed '/^[[:space:]]*$/d')\n    if [ \"${#menu_items[@]}\" -eq 0 ]; then\n        die \"Port menu generation failed: empty\"\n    fi\n    chosen_port=\"$(dialog --menu \"Choose which port to forward to:\" \"$LINES\" \"$COLUMNS\" \"$LINES\" \"${menu_items[@]}\" 3>&1 1>&2 2>&3)\"\n    if [ -z \"$chosen_port\" ]; then\n        timestamp \"Cancelled, aborting...\"\n        exit 1\n    fi\n    pod_port=\"$chosen_port\"\nfi\n\nlocal_port=\"$pod_port\"\n\nif [ \"$local_port\" -lt 1024 ]; then\n    if [ \"$local_port\" = 80 ]; then\n        local_port=8080\n    elif [ \"$local_port\" = 443 ]; then\n        local_port=8443\n    else\n        local_port=\"$((local_port + 1000))\"\n    fi\nfi\nlocal_port=\"$(next_available_port \"$pod_port\")\"\n\ntimestamp \"Launching port forwarding to pod '$pod' port '$pod_port' to local port '$local_port'\"\nkubectl port-forward --address 127.0.0.1 ${namespace:+-n \"$namespace\"} \"$pod\" \"$local_port\":\"$pod_port\" &\n\npid=\"$!\"\n\nsleep 2\n\nif ! kill -0 \"$pid\" 2>/dev/null; then\n    die \"ERROR: kubectl port-forward exited\"\nfi\n\nif [ -z \"${SSH_CONNECTION:-}\" ]; then\n    echo\n    url=\"http://localhost:$local_port\"\n    timestamp \"Port-forwarded UI is now available at: $url\"\n\n    if [ -n \"${OPEN_URL:-}\" ]; then\n        echo\n        timestamp \"Opening URL:  $url\"\n        \"$srcdir/../bin/urlopen.sh\" \"$url\"\n    fi\nfi\n"
  },
  {
    "path": "kubernetes/kubectl_port_forward_spark.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-01 14:06:16 +0200 (Sun, 01 Sep 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLaunches kubectl port-forward to Spark driver pod for Spark UI\n\nIf more than one Spark driver pod is found, prompts with an interactive dialogue to choose one\n\nOn Mac automatically opens the Spark UI on localhost URL in the default browser\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nnamespace=\"${1:-}\"\n\nexport OPEN_URL=1\n\nexport POD_PORT=4040\n\n\"$srcdir/kubectl_port_forward.sh\" \"$namespace\" spark-role=driver\n"
  },
  {
    "path": "kubernetes/kubectl_rerun_job.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-06 09:10:40 +0000 (Thu, 06 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBackups up and then re-runs a given Kubernetes job\n\nBackups are taken to the source directory of this script under the .kubectl_job_definitions/ directory before re-creating the job,\njust in case, as a delete operation must happen before the re-creation and if the re-creation fails the job definition would otherwise be lost\n\nExample:\n\n    ${0##*/} cert-manager-startupapicheck -n cert-manager\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<job_name> [<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\njob_name=\"$1\"\nshift || :\n\nnamespace=\"\"\nargs=()\n\nuntil [ $# -lt 1 ]; do\n    case $1 in\n      -n|--namespace)  namespace=\"${2:-}\"\n                       shift || :\n                       ;;\n                   *)  args+=(\"$1\")\n                       ;;\n    esac\n    shift || :\ndone\n\n# if namespace was not set then try to infer from current context\nif [ -z \"$namespace\" ]; then\n    namespace=\"$(kubectl config get-contexts | awk '/^\\*/{print $5}')\"\nfi\n# if not set in context, must be 'default'\nnamespace=\"${namespace:-default}\"\n\n#jobdir=\"${PWD:-$(pwd)}/.kubectl_job_definitions\"\njobdir=\"$srcdir/.kubectl_job_definitions\"\nmkdir -p -v \"$jobdir\"\n\njobfile=\"$jobdir/$namespace.$job_name.json\"\n\n# backup the job because job recreation will fail after it is deleted if any of the fields prevent re-creation\n# remove the fields that prevent recreation\nif ! [ -f \"$jobfile\" ]; then\n    kubectl get job \"$job_name\" ${namespace:+--namespace \"$namespace\"} ${args:+\"${args[@]}\"} -o json |\n    jq -Mr 'del(.spec.selector)' |\n    jq -Mr 'del(.spec.template.metadata.labels)' \\\n    > \"$jobfile\"\nfi\n\nkubectl replace --force -f \"$jobfile\"\n"
  },
  {
    "path": "kubernetes/kubectl_restart.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-11 14:34:46 +0100 (Thu, 11 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRestarts all deployments and statefulsets in the current or given namespace\n\nUseful for example to restart all the ArgoCD components if it's stuck\n\nCan optionally specify an ERE regex filter on the deployment or statefulset name\n\n\nRequires kubectl to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <filter>]\"\n\nhelp_usage \"$@\"\n\nnamespace=\"${1:-}\"\nfilter=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nkubectl get deploy,sts -o name |\n{ grep -E \"$filter\" || : ; } |\nwhile read -r type_name; do\n    # type_name is like deployment.apps/argocd-server or statefulset.apps/argocd-application-controller\n    # and can be passed to kubectl as is\n    kubectl rollout restart \"$type_name\"\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_rollout_history_all_deployments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-14 11:42:58 +0100 (Wed, 14 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShow the rollout history of all Kubernetes deployments in the current cluster context\n\nExcludes kube-system\n\nUseful to see if people have been using change-cause properly\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nkubectl get deploy -A -o custom-columns='name:.metadata.name,namespace:.metadata.namespace' |\ntail -n +2 |\ngrep -v kube-system |\nwhile read -r name namespace; do\n    kubectl rollout history \"deploy/$name\" -n \"$namespace\"\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_run_sa.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-28 15:04:50 +0100 (Fri, 28 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns a quick pod on Kubernetes with the given serviceaccount to test private repo pull & other permissions\n\nShares the same pod for successive invocations of this script for speed\n\nArguments become options to 'kubectl run'\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<serviceaccount> [<image> <kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nservice_account=\"$1\"\nimage=\"${2:-busybox}\"\nshift || :\nshift || :\n\nname=\"${image##*/}\"\nname=\"${name%%:*}\"\nname+=\"-${USER:-$(whoami)}\"\n\nrun_static_pod \"$name\" \"$image\" --overrides=\"{ \\\"spec\\\": { \\\"serviceAccount\\\": \\\"$service_account\\\" }  }\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubectl_secret_values.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-08 12:52:38 +0100 (Tue, 08 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSimply reads out and base64 decodes secret values to quickly debug them\n\nIterates each key within the Kubernetes secret and prints them line by line with their contained value\n\nOutput format:\n\n<secret_key1>   <decoded_value>\n<secret_key2>   <decoded_value>\n\nGood counterpart to check on what has been auto-loaded from GCP Secret Manager\nto Kubernetes secrets by gcp_secrets_to_kubernetes.sh\n\nSee Also:\n\n    gcp_secrets_to_kubernetes.sh - loads from GCP Secret Manager to Kubernetes secret\n\n    kubectl_kv_to_secret,sh - creates a Kuberbetes secret from key=value args or stdin, environment export commands (eg. piped from aws_csv_creds.sh)\n\n\nAdditional arguments are assumed to be kubectl options, useful for specifying --namespace\n\n\nRequires kubectl in \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<secret_name> [<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n#kubectl get secret stackdriver-api-key -o 'jsonpath={.data.stackdriver-api-key}' | base64 -D\nkubectl get secret \"$@\" -o json |\n# @base64d works nicely on jq 1.6 but not available on 1.5\n#jq -r '.data | to_entries[] | [.key, (.value | @base64d) ] | @tsv''\njq -r '.data | to_entries[] | [.key, .value] | @tsv' |\nwhile read -r key value; do\n    # use --decode not -d / -D which varies between Linux and Mac\n    printf '%s\\t%s\\n' \"$key\" \"$(base64 --decode <<< \"$value\")\"\ndone\n"
  },
  {
    "path": "kubernetes/kubectl_secrets_annotate_to_be_sealed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-29 19:15:08 +0100 (Fri, 29 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAnnotate all secrets in the current or given namespace to allow Sealed Secret overwrite\n\nUseful to fix update conflicts in ArgoCD where the sealed secret can't be unmapped\n\nThis is usually done as part of the migration script kubernetes_secrets_to_sealed_secrets.sh but if reapplying or updating secrets then this may be needed\n\nSee Also:\n\n    kubernetes_secret_to_sealed_secret.sh\n    kubernetes_secrets_to_sealed_secrets.sh\n    kubectl_secrets_download.sh (to take a backup of secrets first)\n\n\nRequires kubectl to be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nnamespace=\"${1:-}\"\ncontext=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nkubectl get secrets |\n# don't touch the default generated service account tokens for safety\ngrep -v kubernetes.io/service-account-token |\n# remove header\ngrep -v '^NAME[[:space:]]' |\nawk '{print $1}' |\nwhile read -r secret; do\n    timestamp \"Annotating secret '$secret' to be managed by sealed-secrets controller\"\n    kubectl annotate secrets \"$secret\" 'sealedsecrets.bitnami.com/managed=true' --overwrite\n    echo\ndone\ntimestamp Done\n"
  },
  {
    "path": "kubernetes/kubectl_secrets_download.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-01 16:49:06 +0100 (Mon, 01 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads all Kubernetes secrets in the current or given namespace to files in the local directory named secret-<name>.yaml\n\nUseful for backing up all your live secrets before migrating to Sealed Secrets or External Secrets\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubectl_options>]\"\n\nhelp_usage \"$@\"\n\nkubectl get secrets --no-headers -o custom-columns=\":metadata.name\" \"$@\" |\nwhile read -r secret; do\n    filename=\"secret-$secret.$(date '+%F_%H-%M-%S').yaml\"\n    timestamp \"Downloading secret '$secret' to '$filename'\"\n    kubectl get secrets \"$secret\" \"$@\" -o yaml > \"$filename\"\ndone\n\necho \"Done\"\n"
  },
  {
    "path": "kubernetes/kubectl_secrets_not_sealed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-01 14:51:44 +0100 (Mon, 01 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns secrets not yet managed by Sealed Secrets by querying for secrets without an SealedSecret owner reference\n\nOutput:\n\n<namespace>     <secret_name>\n\n\nKubectl and jq need to be installed in the \\$PATH and kubectl configured with the right context\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_options>\"\n\nhelp_usage \"$@\"\n\n# can't decompose this line across lines unfortunately for clarity, fails to parse\n# gets secrets with a SealedSecret owner\n#kubectl get secrets -o jsonpath='{range .items[?(@.metadata.ownerReferences[*].kind == \"SealedSecret\")]}{.metadata.name}{\"\\n\"}'\n\n# secrets with no owner references at all\nkubectl get secrets \"$@\" -o json |\njq -r '\n    .items[] |\n    select(.metadata.ownerReferences | not) |\n    [.metadata.namespace, .metadata.name] |\n    @tsv'\n\n# secrets with metadata.ownerReferences that don't have a SealedSecret kind (TODO: this query needs more testing)\nkubectl get secrets \"$@\" -o json |\njq -r '\n    .items[] |\n    select(.metadata.ownerReferences) |\n    select(.metadata.ownerReferences[].kind == \"SealedSecret\" | not) |\n    [.metadata.namespace, .metadata.name] |\n    @tsv'\n"
  },
  {
    "path": "kubernetes/kubectl_secrets_to_be_sealed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-01 15:10:29 +0100 (Mon, 01 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns all secrets that have the annotation sealedsecrets.bitnami.com/managed=\\\"true\\\" set, ie. waiting to be replaced by Bitnami Sealed Secrets\n\nUseful to track progress in migrations to Sealed Secrets\n\nSee Also:\n\n    kubernetes_secrets_to_sealed_secrets.sh - sets the annotation for the secret to be overwritten\n\nRequires kubectl to be install in the \\$PATH and configured with the right context\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_options>\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/kubectl_get_annotation.sh\" secrets sealedsecrets.bitnami.com/managed '\"true\"' \"$@\"\n"
  },
  {
    "path": "kubernetes/kubernetes_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-08-02 19:38:43 +0100 (Fri, 02 Aug 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# You might also be interested in find_active_kubernetes_api.py in DevOps Python tools repo - https://github.com/HariSekhon/DevOps-Python-tools\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# used by utils.sh usage()\n# shellcheck disable=SC2034\nusage_description=\"Auto-determines the Kubernetes API server and kube-system API Token to make curl calls to K8S easier\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=.bash.d/kubernetes.sh\n. \"$srcdir/.bash.d/kubernetes.sh\"\n\n# used by utils.sh usage()\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\ntoken=\"$(k8s_get_token)\"\napi_server=\"$(k8s_get_api)\"\n\npath=\"$1\"\nshift || :\n\n# could also extract the k8s certs from ~/.kube/config (not shown in kubectl config view, would have to json parse outside), and then do\n# curl \"$api_server\" --cert encoded.crt --key encoded.key --cacert encoded-ca.crt\n\nexport TOKEN=\"$token\"\n\n# XXX: have to use -k to not verify the certificate here because often it is self-signed\n#\"$srcdir/../bin/curl_auth.sh\" -k \"$api_server$path\" \"$@\"\n\"$srcdir/../bin/curl_auth.sh\" \"$api_server$path\" \"$@\"\n"
  },
  {
    "path": "kubernetes/kubernetes_autoscaler_release.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-26 11:19:24 +0000 (Fri, 26 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the latest Kubernetes Autoscaler release that matches your local kubnernetes cluster version using kubectl and the GitHub API\n\nUseful to populate eks-cluster-autoscaler-kustomization.yaml image override in https://github.com/HariSekhon/Kubernetes-configs\n\n\nRequires Kubectl to be installed and configured, as well as jq\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<major.minor version>]\"\n\nhelp_usage \"$@\"\n\nversion=\"${1:-}\"\n\nif [ -n \"$version\" ]; then\n    if ! [[ \"$version\" =~ ^[[:digit:]]+\\.[[:digit:]]+$ ]]; then\n        usage \"invalid kubernetes major.minor version given\"\n    fi\nelse\n    version=\"$( kubectl version -o json | jq -r '.serverVersion | .major + \".\" + .minor | gsub(\"[+]\"; \"\")' )\"\nfi\n\noutput=\"$(curl -sS https://api.github.com/repos/kubernetes/autoscaler/releases?per_page=100)\"\necho -n \"Kubernetes version: \"\njq -r '\n    first(\n        .[] |\n        select(.tag_name | test(\"cluster-autoscaler-'\"$version\"'\")) |\n        .tag_name\n    ) |\n    gsub(\"cluster-autoscaler-\"; \"\")\n' <<< \"$output\"\n#echo\n#echo -n \"Helm Chart: \"\n#echo \"Kubernetes versions / Charts:\"\n#echo\n#jq -r '\n#    .[] |\n#    select(.tag_name | test(\"cluster-autoscaler\")) |\n#    .tag_name |\n#    gsub(\"cluster-autoscaler-chart-\"; \"chart \") |\n#    gsub(\"cluster-autoscaler-\"; \"kubernetes \")\n#' <<< \"$output\"\necho\nif type -P helm &>/dev/null; then\n    echo \"Helm Chart: \"\n    echo\n    if ! helm repo list | grep -q '^cluster-autoscaler[[:space:]]'; then\n        helm repo add cluster-autoscaler https://kubernetes.github.io/autoscaler\n    fi\n    helm search repo cluster-autoscaler -l |\n    grep -m 1 \"$version\"\nfi\n"
  },
  {
    "path": "kubernetes/kubernetes_check_objects_namespaced.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-13 08:08:02 +0100 (Sat, 13 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks Kubernetes yaml for objects without namespaces specified, which can easily result in deployments to the wrong namespace\n\nUseful to find common mistakes in YAMLs or Helm chart templates pulled through Kustomize that don't have namespaces unless you specify it in the kustomization.yaml\n\nTakes yaml as standard input or file arguments\n\nSome objects are cluster-wide so should be ignored, but most objects should be namespaced. Will query your current Kubernetes cluster\nfor such objects, and if successful will exclude them to reduce noise\n\nRequires kubectl to be installed and configured, as well as 'yq' version 4.18.1+\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file1.yaml> <file2.yaml> ...]\"\n\nhelp_usage \"$@\"\n\nyaml_objects_without_namespace=\"$(yq 'select(.metadata.namespace == null)' -- \"$@\")\"\n\n# -o name is not usable for kind filtering later, and output doesn't support any other column as of 1.18, so must extract the KIND last column\nobjects_not_namespaced=\"$(kubectl api-resources --namespaced=false --no-headers | awk '{print $NF}' 2>/dev/null || :)\"\n\nif [ -n \"$objects_not_namespaced\" ]; then\n    regex=\"$(tr '\\n' '|' <<< \"$objects_not_namespaced\" | sed 's/|$//')\"\n    yaml_objects_without_namespace=\"$(yq 'select(.kind | test(\"^('\"$regex\"')$\") | not)' <<< \"$yaml_objects_without_namespace\")\"\nfi\n\nnum_objects_without_namespace=\"$(grep -c '^kind:' <<< \"$yaml_objects_without_namespace\" || :)\"\n\nif [ \"$num_objects_without_namespace\" = 0 ]; then\n    echo \"OK: all objects have a namespace specified\" >&2\nelse\n    echo \"WARNING: $num_objects_without_namespace objects detected without namespace:\" >&2\n    echo >&2\n    echo \"$yaml_objects_without_namespace\"\n    echo >&2\n    echo \"WARNING: $num_objects_without_namespace objects detected without namespace!\" >&2\n    exit 1\nfi\n"
  },
  {
    "path": "kubernetes/kubernetes_delete_stuck_namespace.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-04-16 22:26:17 +0100 (Sun, 16 Apr 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nForcibly deletes a Kubernetes namespace that is stuck deleting\n\nWritten to get rid of Knative namespaces knative-eventing, knative-serving and kourier-system which were all permanently stuck trying to delete\n\nDoesn't do the actual deletion of the namespace, but deletes the finalizers to allow a namespace already stuck in deletion to be removed\n\nWARNING: do not run this on a normal healthy namespace that you care about, as it will wipe out the finalizers\n\nVarious solutions to this problem can be found here:\n\n    https://stackoverflow.com/questions/52369247/namespace-stuck-as-terminating-how-i-removed-it\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<namespace>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nnamespace=\"$1\"\n\ntimestamp \"Stuck resources in namespace '$namespace':\"\nkubectl api-resources --verbs=list --namespaced -o name |\nxargs -n 1 kubectl get --show-kind --ignore-not-found -n \"$namespace\"\n\necho\n\ntimestamp \"Attempting to remove the finalizers:\"\nkubectl get namespace \"$namespace\" -o json |\ntr -d \"\\n\" |\nsed \"s/\\\"finalizers\\\": \\[[^]]\\+\\]/\\\"finalizers\\\": []/\" |\nkubectl replace --raw \"/api/v1/namespaces/$namespace/finalize\" -f -\n"
  },
  {
    "path": "kubernetes/kubernetes_etcd_backup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 10:35:08 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBacks up Kubernetes Etcd database on a kubeadm cluster to etcd-kubernetes-backup-DATETIMESTAMP.tar.gz, containing the Etcd database snapshot and PKI certs\n\nRequires 'etcdctl' to be in \\$PATH\n\nWhen restoring, you must restore all nodes because the restore will override the cluster id and member id so the nodes won't communicate on partial nodes restore. Restores of lost nodes require new node has the same IP address\n\nTested on Etcd v3\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nbackup_timestamp=\"$(date '+%F_%H%M')\"\n\nbackup_dir=\"etcd-snapshot-$backup_timestamp.db\"\nbackup_tar=\"etcd-kubernetes-backup-$backup_timestamp.tar.gz\"\n\nexport ETCDCTL_API=3\n\ntimestamp \"backing up Etcd database to directory $backup_dir\"\n# should be root\netcdctl snapshot save \"$backup_dir\" \\\n                      --cacert /etc/kubernetes/pki/etcd/server.crt \\\n                      --cert   /etc/kubernetes/pki/etcd/ca.crt \\\n                      --key    /etc/kubernetes/pki/etcd/ca.key\necho >&2\ntimestamp \"checking Etcd backup\"\netcdctl --write-out table snapshot status \"$backup_dir\"\n\necho >&2\ntimestamp \"tar'ing Etcd backup and /etc/kubernetes/pki/etc certs\"\ntar cvzf \"$backup_tar\" \"$backup_dir\" /etc/kubernetes/pki/etc\n"
  },
  {
    "path": "kubernetes/kubernetes_foreach_context.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bash -c 'echo -n \"kubectl context: \"; kubectl config current-context; echo \"context: {context}\"'\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-26 16:18:30 +0100 (Wed, 26 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each configured Kubernetes kubectl context (cluster)\n\nCan chain with kubernetes_foreach_namespace.sh\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the context names and exit after the first iteration\n\nAll arguments become the command template\n\nReplaces {context} if present in the command template with the current kubectl context name in each iteration, but often this isn't necessary to specify explicitly given the kubectl context is set within each iteration for each context for convenience running short commands local to the context\n\neg.\n    ${0##*/} kubectl get pods\n\nSince lab contexts like Docker Desktop, Minikube etc are often offline and likely to hang, they are skipped. Deleted GKE clusters you'll need to remove from your kubeconfig yourself before calling this\n\nSee Also: gke_kube_creds.sh to auto-create all your contexts for all clusters on Google Kubernetes Engine!\n\n\nRequires 'kubectl' to be configured and available in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nkube_config_isolate\n\n# don't need to store this any more as we now switch the KUBECONFIG which is only used for the lifetime of this script\n#original_context=\"$(kubectl config current-context)\"\n\nkubectl config get-contexts -o name |\nwhile read -r context; do\n    if [[ \"$context\" =~ docker|minikube|minishift ]]; then\n        echo \"Skipping context '$context'...\"\n        echo\n        continue\n    fi\n    echo \"# ============================================================================ #\" >&2\n    echo \"# Kubernetes context = $context\" >&2\n    echo \"# ============================================================================ #\" >&2\n    # shellcheck disable=SC2064  # want interpolation now\n    # XXX: no longer reset because we isolate the environment above via redirecting KUBECONFIG and simply let it expire at the end of this script\n    #trap \"echo; echo 'Reverting to original context:' ; kubectl config use-context '$original_context'\" EXIT\n    kubectl config use-context \"$context\"\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{context\\}/$context}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/kubernetes_foreach_namespace.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bash -c 'echo -n \"kubectl context: \"; kubectl config get-contexts | awk \"/^\\*/{print \\$5\\\" \\\"\\$3}\"; echo \"namespace: {namespace}\"'\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-08 19:20:40 +0100 (Tue, 08 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each Kubernetes namespace on the current cluster / kubectl context\n\nCan chain with kubernetes_foreach_context.sh\n\nThis is powerful so use carefully!\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the namespace names and exit after the first iteration\n\nAll arguments become the command template\n\nReplaces {namespace} if present in the command template with the namespace in each iteration, but often this isn't necessary to specify explicitly given the kubectl context's namespace is set within each iteration for convenience running short commands local to the namespace\n\neg.\n    ${0##*/} gcp_secrets_to_kubernetes.sh\n\n\nRequires 'kubectl' to be configured and available in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nkube_config_isolate\n\ncurrent_context=\"$(kubectl config current-context)\"\n\n# there is no -o jsonpath/namespace so must just get column\n# don't need to store this any more as we now switch the KUBECONFIG which is only used for the lifetime of this script\n#original_namespace=\"$(kubectl config get-contexts \"$current_context\" --no-headers | awk '{print $5}')\"\n\nset_namespace(){\n    local namespace=\"$1\"\n    kubectl config set-context \"$current_context\" --namespace \"$namespace\"\n}\n\n#kubectl get namespaces -o name | sed 's,namespace/,,' |\nkubectl get namespaces -o 'jsonpath={range .items[*]}{.metadata.name}{\"\\n\"}' |\nwhile read -r namespace; do\n    #if [[ \"$context\" =~ kube-system ]]; then\n    #    echo \"Skipping context '$context'...\"\n    #    echo\n    #    continue\n    #fi\n    echo \"# ============================================================================ #\" >&2\n    echo \"# Kubernetes namespace = $namespace, content = $current_context\" >&2\n    echo \"# ============================================================================ #\" >&2\n    # shellcheck disable=SC2064  # want interpolation now\n    # XXX: no longer reset because we isolate the environment above via redirecting KUBECONFIG and simply let it expire at the end of this script\n    #trap \"echo; echo 'Reverting context to original namespace: $original_namespace' ; set_namespace '$original_namespace'\" EXIT\n    set_namespace \"$namespace\"\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{namespace\\}/$namespace}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/kubernetes_info.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#  (forked from gcp_info.sh)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Kubernetes info and deployed resources across all namespaces in the current cluster / kube context\n\nCan optionally specify a different kubernetes context to switch to (will switch back to original context on any exit except kill -9)\n\nLists:\n\n- cluster-info\n- master component statuses\n- nodes\n- namespaces\n- deployments, replicasets, replication controllers, statefulsets, daemonsets, horizontal pod autoscalers\n- services, ingresses\n- jobs, cronjobs\n- storage classes, persistent volumes, persistent volume claims\n- service accounts, resource quotas, network policies, pod security policies\n- container images running\n- container images running counts descending\n- pods  # might be too much detail if you have high replica counts, so done last, comment if you're sure nobody has deployed pods outside deployments\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<kubernetes_context>]\"\n\nhelp_usage \"$@\"\n\nkube_config_isolate\n\ncurrent_context=\"$(kubectl config current-context)\"\n\nif [ $# -gt 0 ]; then\n    context=\"$1\"\n    # want interpolation now\n    # shellcheck disable=SC2064\n    trap \"echo; kubectl config use-context '$current_context'\" EXIT\n    # kubectl declares this just fine\n    #echo \"switching to kubernetes context '$context'\"\n    kubectl config use-context \"$context\"\n    echo\nfi\n\necho \"kubectl context: $(kubectl config current-context)\"\necho\nkubectl cluster-info\necho\nkubectl get componentstatuses\necho\nkubectl get nodes -o wide\necho\nkubectl get namespaces\necho\nkubectl get --all-namespaces deployments,replicasets,replicationcontrollers,statefulsets,daemonsets,horizontalpodautoscalers\necho\nkubectl get --all-namespaces services,ingresses\necho\necho\nkubectl get --all-namespaces jobs,cronjobs\necho\necho\nkubectl get --all-namespaces storageclasses,persistentvolumes,persistentvolumeclaims\necho\necho\nkubectl get --all-namespaces serviceaccounts,resourcequotas,networkpolicies,podsecuritypolicies\necho\necho\necho \"# Running container images:\"\necho\n\"$srcdir/kubectl_images.sh\"\necho\necho\necho \"# Running container image counts:\"\necho\n\"$srcdir/kubectl_image_counts.sh\"\necho\necho\n# pods might be too numerous with high replica counts and low value info, but there is always a chance that people launched pods without deployments, you can comment it out if you're confident that isn't the case\nkubectl get --all-namespaces pods\n"
  },
  {
    "path": "kubernetes/kubernetes_nodes_ssh_dump_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-15 04:58:14 +0400 (Tue, 15 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFetch logs from Kubernetes nodes (eg. for support debug requests by vendors)\n\nUses the adjacent script:\n\n    $srcdir/../monitoring/ssh_dump_logs.sh\n\nRequires Kubectl to be installed and configured to be on the right Kubernetes cluster context as it uses this to\ndetermine the nodes\n\nUser - set your SSH_USER environment variable if your login user is different to your local \\$USER\n\nSSH key - if you need to set to an alternative SSH key or\n          else add it to a local ssh-agent for passwordless authentication to work\n\nSee here for details:\n\n    $srcdir/../monitoring/ssh_dump_logs.sh --help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ntimestamp \"Getting Kubernetes nodes via kubectl\"\nnodes=\"$(kubectl top nodes --no-headers | awk '{print $1}')\"\nnum_nodes=\"$(wc -l <<< \"$nodes\" | sed 's/[[:space:]]//g')\"\ntimestamp \"Found $num_nodes nodes\"\necho\n\n# want splitting\n# shellcheck disable=SC2086\n\"$srcdir/../monitoring/ssh_dump_logs.sh\" $nodes\n"
  },
  {
    "path": "kubernetes/kubernetes_resource_types.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-06-18 11:28:40 +0100 (Fri, 18 Jun 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFilter program to get all unique Kubernetes resources types out of a Kubernetes yaml or Kustomize build output\n\nYaml can be supplied as a file argument or via standard input. If no file is given, waits for stdin like a standard unix filter program\n\nUseful to find objects to grant an ArgoCD project permissions to manage for an app you are adding to ArgoCD\n\nOutput Format:\n\n<group>     <object_kind>\n\nSorted by object kind\n\neg.\n\nv1                          ConfigMap\nbatch/v1beta1               CronJob\napps/v1                     Deployment\nautoscaling/v1              HorizontalPodAutoscaler\nextensions/v1beta1          Ingress\nv1                          Namespace\npolicy/v1beta1              PodDisruptionBudget\nscheduling.k8s.io/v1        PriorityClass\nv1                          Service\nv1                          ServiceAccount\napps/v1                     StatefulSet\nstorage.k8s.io/v1           StorageClass\nautoscaling.k8s.io/v1beta2  VerticalPodAutoscaler\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file.yaml> <file2.yaml> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nawk '/^(api|kind)/{print $2}' \"$@\" |\n# sed N joins every 2 lines\nsed 'N;s/\\n/ /' |\nsort -k2 -u |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kubernetes_secret_to_external_secret_gcp.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-07-26 00:38:43 +0100 (Wed, 26 Jul 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Kubernetes external secret yaml from a given secret in the current or given namespace\n\nAssumes each secret has a key of the same name\n\n- generates external secret yaml - if GENERATE_YAML_ONLY env var is set to any value, then stops here\n- checks the GCP Secret Manager secret exists\n  - if it doesn't, creates it\n  - if it does, validates that its content matches the existing secret in Kubernetes\n  - if there is a difference and OVERWRITE_GCP_SECRET env var is set to any value, creates a new version and uses that - XXX: use this carefully\n- creates external secret in the same namespace\n\nUseful to migrate existing secrets to external secrets referencing GCP Secret Manager\n\nSee kubernetes_secrets_to_external_secrets_gcp.sh to quickly migrate all your secrets to external secrets\n\nUse kubectl_secrets_download.sh to take a backup of existing kubernetes secrets first\n\nXXX: you should probably omit committing secrets generated by Cert Manager (eg. *-tls)\n\n\nRequires kubectl and GCloud SDK to both be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<secret_name> [<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_bin kubectl\ncheck_bin gcloud\n\nsecret=\"$1\"\nnamespace=\"${2:-}\"\ncontext=\"${3:-}\"\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nif [ -z \"${namespace:-}\" ]; then\n    namespace=\"$(kube_current_namespace)\"\nfi\n\nyaml_file=\"external-secret-$secret.yaml\"\n\ntimestamp \"Generating external secret for secret '$secret'\"\n\nk8s_secret_json=\"$(kubectl get secret \"$secret\" -o json)\"\n\nif [ -z \"$k8s_secret_json\" ]; then\n    timestamp \"ERROR: failed to get Kubernetes secret json\"\n    exit 1\nfi\n\nkeys=\"$(jq -r '.data | keys[]' <<< \"$k8s_secret_json\")\"\nif [ -z \"$keys\" ]; then\n    timestamp \"ERROR: fails to get keys for secret\"\n    exit 1\nfi\nnum_keys=\"$(wc -l <<< \"$keys\" | sed 's/[[:space:]]//g')\"\nif [ \"$num_keys\" != 1 ]; then\n    timestamp \"ERROR: more than 1 key in secret, not handling\"\n    exit 1\nfi\n\n# if the secret has a dash in it, then you need to quote it whether .data.\"$secret\" or .data[\"$secret\"]\nk8s_secret_value=\"$(jq -r \".data[\\\"$secret\\\"]\" <<< \"$k8s_secret_json\" | base64 --decode)\"\n\nif [ -z \"$k8s_secret_value\" ]; then\n    timestamp \"ERROR: failed to get Kubernetes secret value for '$secret' key '$secret'\"\n    exit 1\nfi\n\n# https://github.com/HariSekhon/Kubernetes-configs/blob/master/external-secret.yaml\nyaml=\"---\napiVersion: external-secrets.io/v1beta1\nkind: ExternalSecret\nmetadata:\n  name: $secret\n  namespace: $namespace\nspec:\n  refreshInterval: 1h\n  secretStoreRef:\n    name: gcp-secret-manager\n    #kind: SecretStore\n    kind: ClusterSecretStore\n  target:\n    name: $secret\n    creationPolicy: Merge\n  data:\n    - secretKey: $secret  # key within k8s secret\n      remoteRef:\n        key: $secret  # GCP Secret Manager secret\n\"\n\n# remove the sed line if you want to leave the comments in the generated files\necho \"$yaml\" \\\n    | sed 's/#.*$//; s/[[:space:]]*$//; /^[[:space:]]*$/d' \\\n    > \"$yaml_file\"\n\nif [ -n \"${GENERATE_YAML_ONLY:-}\" ]; then\n    exit 0\nfi\n\ntimestamp \"Generated external-secret yaml file:  $yaml_file\"\n\ntimestamp \"Checking GCP Secret Manager for secret '$secret'\"\n\nif gcloud secrets list --format='value(name)' | grep -Fxq \"$secret\"; then\n    timestamp \"GCP secret '$secret' already exists\"\n    timestamp \"Checking Kubernetes secret '$secret' content matches GCP secret '$secret' content\"\n    timestamp \"Getting GCP secret '$secret' value\"\n    gcp_secret_value=\"$(\"$srcdir/../gcp/gcp_secret_get.sh\" \"$secret\")\"\n    # if it's GCP service account key\n    # false positive - trivy:ignore:gcp-service-account doesn't work\n    # trivy:ignore:gcp-service-account\n    if grep -Fq '\"type\": \"service_account\"' <<< \"$gcp_secret_value\"; then\n        # doesn't work\n        #if [ \"$(jq -Mr <<< \"$gcp_secret_value\")\" = \"$(jq -Mr <<< \"$k8s_secret_value\")\" ]; then\n        # if there are no whitespace differences between the service account keys then accept\n        if [ -n \"$(diff -w <(echo \"$gcp_secret_value\") <(echo \"$k8s_secret_value\") )\" ]; then\n            timestamp \"ERROR: GCP secret service account json does not match existing Kubernetes secret service account json\"\n            exit 1\n        fi\n    elif [ \"$gcp_secret_value\" = \"$k8s_secret_value\" ]; then\n        timestamp \"GCP and Kubernetes secret values match\"\n    elif [ \"${OVERWRITE_GCP_SECRET:-}\" ]; then\n        timestamp \"UPDATING GCP secret '$secret' value\"\n        gcloud secrets versions add \"$secret\" --data-file=- <<< \"$k8s_secret_value\"\n    else\n        timestamp \"ERROR: GCP secret value does not match existing Kubernetes secret value - careful manual reconciliation required\"\n        exit 1\n    fi\nelse\n    timestamp \"GCP secret '$secret' doesn't exist\"\n    timestamp \"CREATING GCP secret '$secret' from the content of the Kubernetes secret '$secret'\"\n    \"$srcdir/../gcp/gcp_secret_add.sh\" \"$secret\" \"$k8s_secret_value\"\n    timestamp \"GCP secret '$secret' created\"\nfi\n\ntimestamp \"Applying external secret '$secret'\"\n\nkubectl apply -f \"$yaml_file\"\n"
  },
  {
    "path": "kubernetes/kubernetes_secret_to_sealed_secret.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-29 19:15:08 +0100 (Fri, 29 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Kubernetes sealed secret from a given secret in the current or given namespace\n\n- generates sealed secret yaml\n- annotates existing secret to be able to be managed by sealed secrets\n- creates sealed secret in the same namespace\n\nUseful to migrate existing secrets to sealed secrets which are safe to commit to Git\n\nSee kubernetes_secrets_to_sealed_secrets.sh to quickly migrate all your secrets to sealed secrets\n\nUse kubectl_secrets_download.sh to take a backup of existing kubernetes secrets first\n\n\nRequires kubectl and kubeseal to both be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<secret_name> [<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nsecret=\"$1\"\nnamespace=\"${2:-}\"\ncontext=\"${3:-}\"\n\nif [[ \"$secret\" =~ kubernetes\\.io/service-account-token ]]; then\n    echo \"WARNING: skipping touching secret '$secret' for safety\"\n    exit 0\nfi\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nyaml=\"sealed-secret-$secret.yaml\"\n\ntimestamp \"Generating sealed secret for secret '$secret'\"\n\nkubectl get secret \"$secret\" -o yaml |\nkubeseal -o yaml > \"$yaml\"\n\ntimestamp \"Generated:  $yaml\"\n\ntimestamp \"Annotating secret '$secret' to be managed by sealed-secrets controller\"\n\nkubectl annotate secrets \"$secret\" 'sealedsecrets.bitnami.com/managed=true' --overwrite\n\ntimestamp \"Applying sealed secret '$secret'\"\n\nkubectl apply -f \"$yaml\"\n"
  },
  {
    "path": "kubernetes/kubernetes_secrets_compare_gcp_secret_manager.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-07-26 00:38:43 +0100 (Wed, 26 Jul 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCompares each Kubernetes secret to GCP Secret Manager\n\nChecks for each key in the kubernetes secret:\n\n- that the kubernetes secret key exists in GCP Secret Manager\n- that the kubernetes secret key value matches the value of the latest version in GCP Secret Manager\n\nUseful to verify before enabling pulling external secrets from GCP Secret Manager\n\nSee kubernetes_secrets_to_external_secrets_gcp.sh to quickly migrate all your secrets to external secrets\n\nUse kubectl_secrets_download.sh to take a backup of existing kubernetes secrets first\n\n\nRequires kubectl and GCloud SDK to both be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\ncheck_bin kubectl\ncheck_bin gcloud\n\nnamespace=\"${1:-}\"\ncontext=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nif [ -z \"${namespace:-}\" ]; then\n    namespace=\"$(kube_current_namespace)\"\nfi\n\nsecrets=\"$(\n    kubectl get secrets |\n    grep -v '^NAME[[:space:]]' |\n    awk '{print $1}'\n)\"\n\n#max_len=0\n#while read -r secret; do\n#    if [ \"${#secret}\" -gt \"$max_len\" ]; then\n#        max_len=\"${#secret}\"\n#    fi\n#done <<< \"$secrets\"\n\n# shellcheck disable=SC2317\ncheck_secret(){\n    local secret=\"$1\"\n    local secret_json\n    local secret_type\n    secret_json=\"$(kubectl get secret \"$secret\" -o json)\"\n    secret_type=\"$(jq -r '.type' <<< \"$secret_json\")\"\n    if [ \"$secret_type\" = \"kubernetes.io/service-account-token\" ]; then\n        print_result \"$secret\" \"n/a\" \"n/a\" \"skip_k8s_service_account\"\n        return\n    fi\n    if [ \"$secret_type\" = \"kubernetes.io/tls\" ]; then\n        tls_cert_manager_issuer=\"$(jq -r '.metadata.annotations.\"cert-manager.io/issuer-name\"' <<< \"$secret_json\")\"\n        if [ -n \"$tls_cert_manager_issuer\" ]; then\n            print_result \"$secret\" \"n/a\" \"n/a\" \"skip_tls_cert_manager\"\n            return\n        fi\n    fi\n\n    local keys\n    keys=\"$(jq -r '.data | keys[]' <<< \"$secret_json\")\"\n    if [ -z \"$keys\" ]; then\n        print_result \"$secret\" \"n/a\" \"n/a\" \"FAILED_TO_GET_SECRET_KEYS\"\n        return 1\n    fi\n    local num_keys\n    num_keys=\"$(wc -l <<< \"$keys\" | sed 's/[[:space:]]//g')\"\n    for key in $keys; do\n        if [ \"$num_keys\" -eq 1 ] || [ \"$key\" = \"$secret\" ]; then\n            gcp_secret=\"$secret\"\n        else\n            gcp_secret=\"$secret-$(tr -C '[:alnum:]-\\n' '-' <<< \"$key\")\"\n        fi\n        check_key \"$secret\" \"$key\" \"$gcp_secret\" \"$secret_json\"\n    done\n}\n\n# shellcheck disable=SC2317\ncheck_key(){\n    local secret=\"$1\"\n    local key=\"$2\"\n    local gcp_secret=\"$3\"\n    local secret_json=\"$4\"\n    local k8s_secret_value\n    local gcp_secret_value\n    local result\n    # if the secret has a dash in it, then you need to quote it whether .data.\"$secret\" or .data[\"$secret\"]\n    k8s_secret_value=\"$(jq -r \".data[\\\"$key\\\"]\" <<< \"$secret_json\" | base64 --decode)\"\n    if [ -z \"$k8s_secret_value\" ]; then\n        print_result \"$secret\" \"$key\" \"$gcp_secret\" \"FAILED_TO_GET_K8S_KEY_VALUE\"\n        return 1\n    fi\n\n    if ! gcloud secrets list --format='value(name)' | grep -Fxq \"$gcp_secret\"; then\n        print_result \"$secret\" \"$key\" \"$gcp_secret\" \"MISSING_ON_GCP\"\n        return 1\n    else\n        gcp_secret_value=\"$(\"$srcdir/../gcp/gcp_secret_get.sh\" \"$gcp_secret\")\"\n        # if it's GCP service account key\n        # false positive - trivy:ignore:gcp-service-account doesn't work\n        # trivy:ignore:gcp-service-account\n        if grep -Fq '\"type\": \"service_account\"' <<< \"$gcp_secret_value\"; then\n            if [ -n \"$(diff -w <(echo \"$gcp_secret_value\") <(echo \"$k8s_secret_value\") )\" ]; then\n                print_result \"$secret\" \"$key\" \"$gcp_secret\" \"MISMATCHED_GCP_SERVICE_ACCOUNT_VALUE\"\n                return 1\n            else\n                print_result \"$secret\" \"$key\" \"$gcp_secret\" \"ok_gcp_service_account_value\"\n            fi\n        elif [ \"$gcp_secret_value\" = \"$k8s_secret_value\" ]; then\n            print_result \"$secret\" \"$key\" \"$gcp_secret\" \"ok_gcp_value_matches\"\n        else\n            print_result \"$secret\" \"$key\" \"$gcp_secret\" \"MISMATCHED_GCP_VALUE\"\n            return 1\n        fi\n    fi\n}\n\n# have all calls standardize the different results to allow column -t alignment ans sorting at the end\n# shellcheck disable=SC2317\nprint_result(){\n    local secret=\"$1\"\n    local key=\"$2\"\n    local gcp_secret=\"$3\"\n    local result=\"$4\"\n    echo \"Kubernetes secret '$secret' key '$key' == GCP secret '$gcp_secret' => $result\"\n}\n\nexport srcdir\nexport -f check_secret\nexport -f check_key\nexport -f print_result\nwhile read -r secret; do\n    echo \"check_secret '$secret'\"\ndone <<< \"$secrets\" |\nparallel |\ncolumn -t |\nsort -k11r\n"
  },
  {
    "path": "kubernetes/kubernetes_secrets_to_external_secrets_gcp.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-07-26 00:38:43 +0100 (Wed, 26 Jul 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates Kubernetes external secrets yamls from all existing Kubernetes secrets in the current or given namespace\n\nFor every secret, performs the following actions via kubernetes_secrets_to_external_secret_gcp.sh:\n\n- generates external secret yaml\n- checks the GCP Secret Manager secret exists\n  - if it doesn't, creates it\n  - if it does, validates that its content matches the existing secret in Kubernetes\n- creates external secret in the same namespace\n- omits:\n  - type kubernetes.io/service-account-token\n  - type tls with Cert Manager annotation\n\nUseful to migrate existing secrets to external secrets referencing GCP Secret Manager\n\nUse kubectl_secrets_download.sh to take a backup of existing kubernetes secrets first\n\n\nRequires kubectl and GCloud SDK to both be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\ncheck_bin kubectl\n\nnamespace=\"${1:-}\"\ncontext=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nif [ \"${namespace:-}\" ]; then\n    namespace=\"$(kube_current_namespace)\"\nfi\n\nfor secret in $(kubectl get secrets -o name | sed 's|^secret/||'); do\n    secret_json=\"$(kubectl get secret \"$secret\" -o json)\"\n    secret_type=\"$(jq -r '.type' <<< \"$secret_json\")\"\n    if [ \"$secret_type\" = \"kubernetes.io/service-account-token\" ]; then\n        timestamp \"Skipping touching service account token secret '$secret' for safety\"\n        echo\n        continue\n    fi\n    if [ \"$secret_type\" = \"kubernetes.io/tls\" ]; then\n        tls_cert_manager_issuer=\"$(jq -r '.metadata.annotations.\"cert-manager.io/issuer-name\"' <<< \"$secret_json\")\"\n        if [ -n \"$tls_cert_manager_issuer\" ]; then\n            timestamp \"Skipping touching tls secret '$secret' because its managed by Cert Manager\"\n            echo\n            continue\n        fi\n    fi\n    \"$srcdir/kubernetes_secret_to_external_secret_gcp.sh\" \"$secret\"\n    echo\ndone\n"
  },
  {
    "path": "kubernetes/kubernetes_secrets_to_sealed_secrets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-29 19:15:08 +0100 (Fri, 29 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates Kubernetes sealed secrets from all Kubernetes secrets in the current or given namespace\n\nIterates all non-service-account-token secrets, and for each one:\n\n    - generates sealed secret yaml\n    - annotates existing secret to be able to be managed by sealed secrets\n    - creates sealed secret in the same namespace\n\nUseful to migrate existing secrets to sealed secrets which are safe to commit to Git\n\nUse kubectl_secrets_download.sh to take a backup of existing kubernetes secrets first\n\n    XXX: you should probably omit committing secrets generated by Cert Manager (eg. *-tls)\n\n\nRequires kubectl and kubeseal to both be in the \\$PATH and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<namespace> <context>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nnamespace=\"${1:-}\"\ncontext=\"${2:-}\"\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\nif [ -n \"$namespace\" ]; then\n    kube_namespace \"$namespace\"\nfi\n\nkubectl get secrets |\n# don't touch the default generated service account tokens for safety\ngrep -v kubernetes.io/service-account-token |\n# remove header\ngrep -v '^NAME[[:space:]]' |\nawk '{print $1}' |\nwhile read -r secret; do\n    \"$srcdir/kubernetes_secret_to_sealed_secret.sh\" \"$secret\" ${namespace:+\"$namespace\"} ${context:+\"$context\"}\n    echo\ndone\n\necho \"Completed. Don't forget to commit all sealed-secrets-*.yaml to Git\"\n"
  },
  {
    "path": "kubernetes/kubernetes_yaml_strip_live_fields.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-18 23:49:51 +0700 (Wed, 18 Dec 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nStrips live fields from Kubernetes yaml objects\n\nUseful so you can do 'kubectl diff' or 'kubectl apply' without hitting annoying errors about immutable fields\nleft in exports from 'kubectl get ... -o yaml'\n\nReads from files or standard input and outputs to standard output like a standard unix filter program for shell piping\n\n\nImmutable fields removed:\n\n    - metadata.creationTimestamp,\n    - metadata.uid,\n    - metadata.resourceVersion,\n    - metadata.selfLink,\n    - metadata.generation,\n    - status,\n    - spec.clusterIP,\n    - spec.loadBalancerIP,\n    - spec.template.metadata.creationTimestamp\n\"\n#- metadata.annotations (optional toggle with --strip-annotations)\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nstrip_yaml(){\n\tyq eval \\\n\t\t'del(\n\t\t  .metadata.creationTimestamp,\n\t\t  .metadata.uid,\n\t\t  .metadata.resourceVersion,\n\t\t  .metadata.selfLink,\n\t\t  .metadata.generation,\n\t\t  .status,\n\t\t  .spec.clusterIP,\n\t\t  .spec.loadBalancerIP,\n\t\t  .spec.template.metadata.creationTimestamp\n\t\t)' -\n}\n\nif [ $# -eq 0 ]; then\n    log \"Reading from standard input\"\n\tcat \"$@\" | strip_yaml\nelse\n\tfor file; do\n        strip_yaml < \"$file\"\n    done\nfi\n"
  },
  {
    "path": "kubernetes/kustomize_check_objects_namespaced.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-13 08:08:02 +0100 (Sat, 13 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nKustomize builds and checks the resulting Kubernetes yaml for objects without namespaces specified, which can easily result in deployments to the wrong namespace\n\nUseful to find common mistakes in YAMLs or Helm chart templates pulled through Kustomize that don't have namespaces unless you specify it explicitly in the kustomization.yaml\n\nUses the adjacent script kubernetes_check_objects_namespaces.sh\n\nRequires yq version 4.18.1+\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\nyaml=\"$(kustomize build --enable-helm)\"\n\n\"$srcdir/kubernetes_check_objects_namespaced.sh\" <<< \"$yaml\"\n"
  },
  {
    "path": "kubernetes/kustomize_diff_apply.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-18 11:09:30 +0000 (Wed, 18 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns kustomize build, precreates any namespaces, prompts you with a diff to the changes, and then applies if you accept the prompt\n\nMust be run from the Kustomize directory\n\nUses adjacent scripts:\n\n    kubectl_diff_apply.sh\n    kubectl_create_namespaces.sh\n\nIf a kubectl context is given as an arg, uses adjacent kubectl.sh to prevent race conditions, see kubectl.sh for more details\n\n\nRequires Kustomize 4.x for --enable-helm support\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<kubectl_context> <kubectl_options>\"\n\nhelp_usage \"$@\"\n\ncontext=\"${1:-}\"\nshift || :\n\nkube_config_isolate\n\nif [ -n \"$context\" ]; then\n    kube_context \"$context\"\nfi\n\necho \"Deploying Kustomize kubernetes configs from directory: $PWD\"\necho\n\nyaml=\"$(kustomize build --enable-helm)\"\n\necho \"Pre-creating any namespaces so that diff can succeed\"\n\"$srcdir/kubectl_create_namespaces.sh\" <<< \"$yaml\"\n\n\"$srcdir/kubectl_diff_apply.sh\" -f - <<< \"$yaml\"\n\n#echo \"Diff from live running cluster:\"\n#echo\n#\n#if kubectl diff -f - <<< \"$yaml\"; then\n#    echo \"No changes to deploy\"\n#    exit 0\n#fi\n#\n#echo\n#read -r -p \"Deploy the above changes? (y/N) \" answer\n#shopt -s nocasematch\n#if [[ ! \"$answer\" =~ ^y|yes$ ]]; then\n#    echo \"Aborting...\"\n#    exit 1\n#fi\n#shopt -u nocasematch\n#echo\n#echo\n#\n#kubectl apply -f - \"$@\" <<< \"$yaml\"\n"
  },
  {
    "path": "kubernetes/kustomize_diff_branch.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-14 16:57:44 +0100 (Thu, 14 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns kustomize build in the current vs the target base Git branch, for all directories given as args\n\nUseful to validate kustomization.yaml refactoring, such as changing bases, switching to tagged external bases and wanting to ensure the refactor is a no-op\n\nIf no directories are given, assumes to kustomize build in the local directory\n\n\nRequires Kustomize 4.x for --enable-helm support\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<git_branch> [<dir1> <dir2> <dir3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\noriginal_branch=\"$(current_branch)\"\nbase_branch=\"$1\"\nshift || :\n\ndirs=(\"$@\")\nif [ -z \"${dirs[*]:-}\" ]; then\n    dirs+=(\"$PWD\")\nfi\n\ntrap_cmd \"popd &>/dev/null; git checkout '$original_branch' &>/dev/null; git stash pop &>/dev/null || :\"\n\ntimestamp \"Collecting kustomize build outputs for directories in current branch '$original_branch': $*\"\necho\nfor dir in \"${dirs[@]}\"; do\n    mkdir -p \"/tmp/$dir.$$\"\n    pushd \"$dir\"\n    kustomize build --enable-helm > \"/tmp/$dir.$$/head\"\n    popd\ndone\necho\n\ntimestamp \"Stashing any uncommitted changes in current branch that might prevent a branch switch\"\necho\ngit stash\necho\n\ntimestamp \"Switching to base branch: $base_branch\"\ngit checkout --quiet \"$base_branch\"\necho\n\ntimestamp \"Collecting kustomize build outputs for directories in base branch '$base_branch': $*\"\necho\nfor dir in \"${dirs[@]}\"; do\n    pushd \"$dir\"\n    kustomize build --enable-helm > \"/tmp/$dir.$$/base\"\n    popd\ndone\necho\n\ntimestamp \"Switching back to original branch: $original_branch\"\ngit checkout --quiet \"$original_branch\"\necho\n\ntimestamp \"Restoring any stashed changes\"\necho\ngit stash pop || :\necho\n\ntimestamp \"Differences per directory from base branch '$base_branch' to head current branch '$original_branch':\"\necho\nfor dir in \"${dirs[@]}\"; do\n    echo \"Directory: $dir\"\n    echo\n    diff \"/tmp/$dir.$$/base\" \"/tmp/$dir.$$/head\" || :\n    echo\n    echo\ndone\n\nuntrap\n"
  },
  {
    "path": "kubernetes/kustomize_install_helm_charts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-01 10:54:32 +0100 (Thu, 01 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls the Helm Charts from one or more Kustomize kustomization.yaml files using Helm CLI so that tools like Nova can be run on the live helm releases to detect outdated charts\n\nAll arguments are passed straight to yq and must be kustomization.yaml files\n\nIf no argument is given attempts to use a kustomization.yaml in the current working directory for convenience\n\nEnvironment variables:\n  - if \\$SKIP_EXISTING_HELM_INSTALLATIONS is set to any value, then will skip those installations (useful for CI/CD retries without failing on existing installation from previous run)\n  - if \\$SKIP_ERRORS is set to any value, will ignore failures to install each helm chart, such as webhooks failing to contact cert-manager (useful for CI/CD runs where you just want the charts installed to test outdated release versions with Nova such as https://github.com/HariSekhon/Kubernetes-configs/actions/workflows/kustomize-nova.yaml)\n\nUses adjacent script kustomize_parse_helm_charts.sh and is used in CI/CD GitHub Actions for repo:\n\n    https://github.com/HariSekhon/Kubernetes-configs CI/CD GitHub Actions\n\n\nRequires Helm and yq to be installed and installs them if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"kustomization.yaml [kustomization2.yaml ...]\"\n\nhelp_usage \"$@\"\n\nany_opt_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntype -P helm &>/dev/null || \"$srcdir/../install/install_helm.sh\"\ntype -P yq &>/dev/null || \"$srcdir/../install/install_yq.sh\"\n\n# if there are no repositories to show will return exit code 1 so || :\nhelm_repos=\"$(helm repo list -o yaml | yq -r '.[] | [.name, .url] | @tsv' || :)\"\n\nif [ -n \"${SKIP_EXISTING_HELM_INSTALLATIONS:-}\" ]; then\n    helm_installations=\"$(helm ls -A -o json | jq -r '.[].name')\"\nfi\n\n# slow to do for every run, leave this as a rarely needed exercise for the caller\n#echo\n#helm repo update\n#echo\n\nfor kustomization in \"${@:-kustomization.yaml}\"; do\n    pushd \"$(dirname \"$kustomization\")\" >/dev/null\n    kustomization=\"${kustomization##*/}\"\n    \"$srcdir/kustomize_parse_helm_charts.sh\" \"$kustomization\" |\n    while read -r repo_url name version values_file; do\n        if [ -n \"${SKIP_EXISTING_HELM_INSTALLATIONS:-}\" ]; then\n            if grep -Fxq \"$name\" <<< \"$helm_installations\"; then\n                timestamp \"Skipping existing Helm installation: $name\"\n                continue\n            fi\n        fi\n        if [ \"$values_file\" = null ]; then\n            values_file=\"\"\n        fi\n        if ! grep -Eq \"^${name}[[:space:]]+${repo_url}[[:space:]]*$\" <<< \"$helm_repos\"; then\n            timestamp \"Adding Helm repo '$repo_url' as name '$name'\"\n            # might fail here if you've already installed a repo with this name\n            helm repo add \"$name\" \"$repo_url\" || die \"adding repo '$name' with url '$url' failed, fix your repos as we don't want to remove/modify your existing repos if there is a repo name clash\"\n        fi\n        timestamp \"Installing Helm chart '$name' version '$version' from '$repo_url'\"\n        if [ -n \"${SKIP_ERRORS:-}\" ]; then\n            set +e\n        fi\n        helm install \"$name\" \"$name/$name\" --version \"$version\" --create-namespace --namespace \"$name\" ${values_file:+--values \"$values_file\"}\n        if [ -n \"${SKIP_ERRORS:-}\" ]; then\n            set -e\n        fi\n        echo\n    done\n    popd >/dev/null\ndone\n"
  },
  {
    "path": "kubernetes/kustomize_materialize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-01-30 14:26:58 +0000 (Mon, 30 Jan 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecursively finds all Kustomizations and materializes the full resultant YAML in an adjacent file called kustomization.materialized.yaml in each directory\n\nUseful for checking the YAML with linting tools like FairwindsOps Pluto to detect deprecated API objects inherited from embedded Helm charts affecting your Kubernetes cluster upgrades\n\nParallelized for performance, with Helm support enabled, requires 'kustomize' binary to be in the \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nkustomize_materialize(){\n    kustomization_path=\"$1\"\n    echo \"$kustomization_path\"\n    pushd \"$(dirname \"$kustomization_path\")\" >/dev/null\n    #if [[ \"$kustomization\" =~ ^eks- ]]; then\n    #    echo \"Skipping $kustomization\"\n    #    echo\n    #    continue\n    #fi\n    kustomize build --enable-helm > \"kustomization.materialized.yaml\"\n    echo \"Materialized YAML -> $PWD/kustomization.materialized.yaml\"\n    popd >/dev/null\n    echo\n}\nexport -f kustomize_materialize\n\nfind \"$dir\" -name kustomization.yaml |\nwhile read -r kustomization_path; do\n    echo \"kustomize_materialize '$kustomization_path'\"\ndone |\nparallel\n"
  },
  {
    "path": "kubernetes/kustomize_parse_helm_charts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-01 10:54:32 +0100 (Thu, 01 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses the Helm Charts from one or more Kustomize kustomization.yaml files into a TSV format for post-processing such as installing\n\nUseful to install the Helm charts the old fashioned non-GitOps way via Helm CLI so that tools like Nova can be run on the live helm releases to detect old outdated charts\n\nAll arguments are passed straight to yq and must be kustomization.yaml files or valid --options\n\n\nOutput Format:\n\n<repo_url>    <chart_name>     <chart_version>    <values_file>\n\n\nAny field not found or commented out like valuesFile in kustomization.yaml will return 'null' for that field\n\n\nUsed by adjacent script kustomize_install_helm_charts.sh in CI/CD GitHub Actions for repo:\n\n    https://github.com/HariSekhon/Kubernetes-configs\n\n\nRequires yq and installs it if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"kustomization.yaml kustommization2.yaml...\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ntype -P yq &>/dev/null || \"$srcdir/../install/install_yq.sh\"\n\nyq '.helmCharts[] | [.repo, .name, .version, .valuesFile] | @tsv' \"$@\" --no-doc --no-colors |\nsed '/^[[:space:]]*$/d' |\nsort -u |\ncolumn -t\n"
  },
  {
    "path": "kubernetes/kustomize_update_helm_chart_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-01 10:54:32 +0100 (Thu, 01 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates one of more one Kustomize kustomization.yaml files with the latest version of the charts\n\nAll arguments are passed straight to yq and must be kustomization.yaml files\n\nIf no argument is given attempts to use a kustomization.yaml in the current working directory for convenience\n\nUses adjacent script kustomize_parse_helm_charts.sh\n\n\nRequires Helm and yq to be installed and installs them if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"kustomization.yaml [kustomization2.yaml ...]\"\n\nhelp_usage \"$@\"\n\nany_opt_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ntype -P helm &>/dev/null || \"$srcdir/../install/install_helm.sh\"\ntype -P yq &>/dev/null || \"$srcdir/../install/install_yq.sh\"\n\n# if there are no repositories to show will return exit code 1 so || :\nhelm_repos=\"$(helm repo list -o yaml | yq -r '.[] | [.name, .url] | @tsv' || :)\"\n\n# slow to do for every run, leave this as a rarely needed exercise for the caller\n#echo\n#helm repo update\n#echo\n\nfor kustomization in \"${@:-kustomization.yaml}\"; do\n    pushd \"$(dirname \"$kustomization\")\" >/dev/null\n    kustomization=\"${kustomization##*/}\"\n    \"$srcdir/kustomize_parse_helm_charts.sh\" \"$kustomization\" |\n    while read -r repo_url name version _values_file; do\n        if ! grep -Eq \"^${name}[[:space:]]+${repo_url}[[:space:]]*$\" <<< \"$helm_repos\"; then\n            timestamp \"Adding Helm repo '$repo_url' as name '$name'\"\n            # might fail here if you've already installed a repo with this name\n            helm repo add \"$name\" \"$repo_url\" || die \"adding repo '$name' with url '$repo_url' failed, fix your repos as we don't want to remove/modify your existing repos if there is a repo name clash\"\n        fi\n        timestamp \"Finding latest Helm chart '$name' version from '$repo_url'\"\n        latest_version=\"$(helm search repo \"$name\" | awk \"/^$name\\/${name}[[:space:]]/{print \\$2}\")\"\n        #helm install \"$name\" \"$name/$name\" --version \"$version\" --create-namespace --namespace \"$name\" ${values_file:+--values \"$values_file\"}\n        if [ \"$version\" != \"$latest_version\" ]; then\n            timestamp \"Updating '$kustomization' chart '$name' from version '$version' to version '$latest_version'\"\n            # more accurate but unfortunately strips out --- and all blank spacing lines\n            #yq -i \".helmCharts[select(.name == \\\"$name\\\")].version = \\\"$latest_version\\\"\"  \"$kustomization\"\n            # for revision controlled kustomization.yaml this is good enough\n            sed -i \"s/ version:[[:space:]]*$version/ version: $latest_version/\" \"$kustomization\"\n        fi\n        echo\n    done\n    popd >/dev/null\ndone\n"
  },
  {
    "path": "kubernetes/pluto_detect_helm_materialize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-10 12:32:19 +0300 (Sat, 10 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecursively finds all local Helm Charts and materializes the full resultant YAML in an adjacent file called helm.materialized.yaml in each directory\n\nThe runs 'pluto detect-files -d .' in each directory to detect deprecated API objects inherited from embedded Helm charts affecting your Kubernetes cluster upgrades\n\nIf you are using internal private Helm repos, you will need to add them to your machine before running this\n\nPluto is run per directory as a workaround for this recursion issue:\n\n    https://github.com/FairwindsOps/pluto/issues/444\n\nParallelized for performance\n\nRequires 'helm', 'pluto' and 'yq' binaries to be in the \\$PATH - will attempt to install them if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nfor x in helm pluto yq; do\n    if ! type -P \"$x\" &>/dev/null; then\n        \"$srcdir/../install_$x.sh\"\n    fi\ndone\n\npluto_detect_helm_materialize(){\n    chart_path=\"$1\"\n    hr\n    timestamp \"$chart_path\"\n    pushd \"$(dirname \"$chart_path\")\" >/dev/null\n    #if [[ \"$chart\" =~ ^eks- ]]; then\n    #    echo \"Skipping $chart\"\n    #    echo\n    #    continue\n    #fi\n    helm dependency build\n    name=\"$(yq .name Chart.yaml)\"\n    helm template \"$name\" . > \"chart.materialized.yaml\"\n    timestamp \"Materialized Helm YAML -> $PWD/chart.materialized.yaml\"\n    pluto detect-files -d .\n    popd >/dev/null\n    echo >&2\n}\nexport -f pluto_detect_helm_materialize\nexport -f hr\nexport -f timestamp\n\nfind \"$dir\" -name Chart.yaml |\nwhile read -r chart_path; do\n    echo \"pluto_detect_helm_materialize '$chart_path'\"\ndone |\nparallel\n"
  },
  {
    "path": "kubernetes/pluto_detect_kubectl_dump_objects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-02-02 15:23:26 +0000 (Thu, 02 Feb 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds live deprecated objects in the current Kubernetes cluster\n\nDumps all Kubernetes objects from the current kubectl cluster context to a directory in /tmp and then runs Pluto against it to detect deprecated API objects affecting your Kubernetes cluster upgrades\n\nNewer versions of Pluto can 'pluto detect-all-in-cluster', but on real-world clusters I've found that this finds different deprecated API objects compared to detect-files on dumped objects, so this script has been updated to do both for comparison\n\nSee here for more details:\n\n    https://github.com/FairwindsOps/pluto/issues/461\n\nRequires 'kubectl' and a recent 'pluto' binary to be in the \\$PATH (newer Pluto is required for the second pass of detect-all-in-cluster), as well as the kubectl context configured and set as current context\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nexport KUBECTL_GET_ALL_SEPARATOR='---'\n\ndir=\"/tmp/${0##*/}.$$\"\ndumpfile=\"$dir/all.yaml\"\n\nmkdir -pv \"$dir\"\n\ntimestamp \"Dumping all live Kubernetes objects to $dumpfile (this will take a few minutes)\"\n\"$srcdir/kubectl_get_all.sh\" --all-namespaces -o yaml > \"$dumpfile\"\necho >&2\n\ntimestamp \"Scanning dumped objects with Pluto\"\necho >&2\n# returns exit code 3 when deprecated objects are found so doesn't run the next detect-all-in-cluster without ignoring the exit code\npluto detect-files -d \"$dir\" || :\necho >&2\n\ntimestamp \"Scanning live cluster as real-world testing shows this finds different results\"\necho >&2\npluto detect-all-in-cluster\n"
  },
  {
    "path": "kubernetes/pluto_detect_kustomize_materialize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-01-30 14:26:58 +0000 (Mon, 30 Jan 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRecursively finds all Kustomizations and materializes the full resultant YAML in an adjacent file called kustomization.materialized.yaml in each directory\n\nThe runs 'pluto detect-files -d .' in each directory to detect deprecated API objects inherited from embedded Helm charts affecting your Kubernetes cluster upgrades\n\nPluto is run per directory as a workaround for this recursion issue:\n\n    https://github.com/FairwindsOps/pluto/issues/444\n\nParallelized for performance, with Helm support enabled.\n\nRequires 'kustomize' and 'pluto' binaries to be in the \\$PATH - will attempt to install them if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\nmax_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nfor x in kustomize pluto; do\n    if ! type -P \"$x\" &>/dev/null; then\n        \"$srcdir/../install_$x.sh\"\n    fi\ndone\n\npluto_detect_kustomize_materialize(){\n    kustomization_path=\"$1\"\n    hr\n    timestamp \"$kustomization_path\"\n    pushd \"$(dirname \"$kustomization_path\")\" >/dev/null\n    #if [[ \"$kustomization\" =~ ^eks- ]]; then\n    #    echo \"Skipping $kustomization\"\n    #    echo\n    #    continue\n    #fi\n    kustomize build --enable-helm > \"kustomization.materialized.yaml\"\n    timestamp \"Materialized Kustomize YAML -> $PWD/kustomization.materialized.yaml\"\n    pluto detect-files -d .\n    popd >/dev/null\n    echo >&2\n}\nexport -f pluto_detect_kustomize_materialize\nexport -f hr\nexport -f timestamp\n\nfind \"$dir\" -name kustomization.yaml |\nwhile read -r kustomization_path; do\n    echo \"pluto_detect_kustomize_materialize '$kustomization_path'\"\ndone |\nparallel\n"
  },
  {
    "path": "kubernetes/rancher_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /nodes | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2024-05-01 23:39:25 +0400 (Wed, 01 May 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Rancher API\n\nAutomatically handles authentication via environment variables \\$RANCHER_ACCESS_KEY and \\$RANCHER_SECRET_KEY\n\nRequires \\$RANCHER_HOST and optionally \\$RANCHER_PORT (default: 443) to be set\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here - may need to click 'Enable SSO' next to each token to access corporation organizations with SSO (eg. Azure AAD SSO):\n\n    https://\\$RANCHER_HOST:\\$RANCHER_PORT/dashboard/account\n\n\nAPI Reference:\n\n    https://ranchermanager.docs.rancher.com/api/api-reference\n\n\nExamples:\n\n    List API endpoints\n\n        ${0##*/} / | jq .\n\n    List clusters:\n\n        ${0##*/} /clusters | jq .\n\n    List nodes:\n\n        ${0##*/} /nodes | jq .\n\n    List projects:\n\n        ${0##*/} /projects | jq .\n\n    List settings:\n\n        ${0##*/} /settings | jq .\n\n    List users:\n\n        ${0##*/} /users | jq .\n\n    List groups:\n\n        ${0##*/} /groups | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncheck_env_defined RANCHER_HOST\ncheck_env_defined RANCHER_ACCESS_KEY\ncheck_env_defined RANCHER_SECRET_KEY\n\nurl_base=\"https://$RANCHER_HOST:${RANCHER_PORT:-443}/v3\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\n# false positive, this works\n# shellcheck disable=SC2295\nurl_path=\"${url_path##$url_base}\"\nurl_path=\"${url_path##/v3}\"\nurl_path=\"${url_path##/}\"\n\nexport USER=\"$RANCHER_ACCESS_KEY\"\nexport PASSWORD=\"$RANCHER_SECRET_KEY\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\" |\njq_debug_pipe_dump\n"
  },
  {
    "path": "kubernetes/rancher_kube_creds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-05-01 23:50:24 +0400 (Wed, 01 May 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/aws.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads Rancher kubectl configs for all kubernetes clusters in Rancher\nto a directory structure matching the cluster names under the current or given directory\n\nThis is fastest way to get set up with local kubectl access in new Rancher environments\n\nGenerates a .envrc in each directory to quickly auto-load using direnv if no .envrc already exists\n\nAlso generates a local kubeconfig.all in the top level directory with its own .envrc if you want to use that instead\n\nRequires Rancher CLI to be set up and configured, as well as jq\n\n\nSee also:\n\n    install_rancher_cli.sh\n    aws_kube_creds.sh  - same as this script but for AWS EKS clusters\n    gke_kube_creds.sh  - same as this script but for GCP GKE clusters\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<directory> <cluster1> <cluster2> <cluster3>...]\"\n\nhelp_usage \"$@\"\n\nif ! type -P rancher &>/dev/null; then\n    \"$srcdir/../install/install_rancher_cli.sh\"\nfi\n\ndir=\"${1:-.}\"\nshift || :\n\ncd \"$dir\" || die \"Failed to cd to '$dir'\"\n\nif [ $# -gt 0 ]; then\n    clusters=\"$*\"\nelse\n    timestamp \"Getting list of Rancher clusters\"\n    clusters=\"$(rancher clusters ls --format json | jq -r '.Cluster.name')\"\n    echo\nfi\n\nfor cluster in $clusters; do\n    timestamp \"Getting kubeconfig for Rancher cluster '$cluster'\"\n    mkdir -p \"$cluster\"\n    base=\"$PWD/$cluster\"\n    kubeconfig=\"$base/kubeconfig\"\n    envrc=\"$base/.envrc\"\n    rancher clusters kubeconfig \"$cluster\" > \"$kubeconfig\"\n    timestamp \"Downloaded '$kubeconfig'\"\n    if ! [ -f \"$envrc\" ]; then\n        timestamp \"Generating '$envrc'\"\n        cat >> \"$envrc\" <<-EOF\n            export KUBECONFIG=\"\\$PWD/kubeconfig\"\nEOF\n        if type -P direnv &>/dev/null; then\n            timestamp \"Direnv allowing '$envrc'\"\n            direnv allow \"$envrc\"\n        fi\n    fi\n    echo\ndone\n\ntimestamp \"Finished downloading Rancher kubeconfigs\"\n\necho\ntimestamp \"Merging configs into $PWD/kubeconfig.all\"\n# shellcheck disable=SC2012\n# find . -maxdepth 2 -name kubeconfig\nKUBECONFIG=$(ls \"$PWD\"/*/kubeconfig | tr '\\n' ':')\nexport KUBECONFIG\nkubectl config view --merge --flatten > kubeconfig.all\necho\n\nif ! [ -f .envrc ]; then\n    timestamp \"Generating $PWD/.envrc\"\n    cat >> .envrc <<-EOF\n    export KUBECONFIG=\"\\$PWD/kubeconfig.all\"\nEOF\n    direnv allow .envrc\n    echo\nfi\n\ntimestamp \"Done\"\n"
  },
  {
    "path": "lib/README.md",
    "content": "Utility Library\n===============\n\nFunctions abstracted from top level scripts.\n\nThere are more functions and aliases with interactive focus in the [.bash.d/](https://github.com/HariSekhon/DevOps-Bash-tools/tree/master/.bash.d) directory at the top level.\n"
  },
  {
    "path": "lib/args_extract.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-20 16:01:28 +0000 (Fri, 20 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Helper script for calling from vim function to run programs with args extraction\n#\n# Returns the value of the 'args:' header from the given file\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# sed is horribly non-portable between Linux and Mac - must use gsed or perl\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    # requires coreutils to be installed\n#    sed(){ gsed \"$@\"; }\n#fi\n\n# #  args: <output this bit>\n# // args: <output this bit>\n#sed -n '/^[[:space:]]*\\(#\\|\\/\\/\\)[[:space:]]*args:/ s/^[[:space:]]*\\(#\\|\\/\\/\\)[[:space:]]*args:[[:space:]]// p' \"$1\"\n# or\nperl -ne 'if(/^\\s*(#|\\/\\/)\\s*args:/){s/^\\s*(#|\\/\\/)\\s*args:\\s*//; print $_; exit}' \"$1\"\n"
  },
  {
    "path": "lib/aws.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-11 13:53:11 +0000 (Fri, 11 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nlibdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$libdir/utils.sh\"\n\n# used in client scripts\n# shellcheck disable=SC2034\nusage_aws_cli_required=\"Requires AWS CLI to be installed and configured (run 'make aws && aws configure')\"\n# shellcheck disable=SC2034\nusage_aws_cli_jq_required=\"Requires AWS CLI to be installed and configured, as well as jq  (run 'make aws && aws configure')\"\n\n# shortest AWS Region length is 9 for eu-west-N\n#\n#   aws ec2 describe-regions --query \"Regions[].{Name:RegionName}\" --output text |\n#   awk '{ if (length < min || NR == 1) min = length } END { print min }'\n#\naws_ecr_regex='[[:digit:]]{12}.dkr.ecr.[[:alnum:]-]{9,}.amazonaws.com'\naws_account_id_regex='[[:digit:]]{12}'\naws_region_regex='[a-z]{2}-[a-z]+-[[:digit:]]'\ninstance_id_regex='i-[0-9a-fA-F]{17}'\nami_id_regex='ami-[0-9a-fA-F]{8}([0-9a-fA-F]{9})?'\n# S3 URL regex with s3:// prefix\ns3_regex='s3:\\/\\/([a-z0-9][a-z0-9.-]{1,61}[a-z0-9])\\/(.+)$|^s3:\\/\\/([a-z0-9][a-z0-9.-]{1,61}[a-z0-9])\\/([a-z0-9][a-z0-9.-]{1,61}[a-z0-9])\\/(.+)'\naws_sg_regex=\"sg-[0-9a-f]{8,17}\"\naws_subnet_regex=\"subnet-[0-9a-f]{8,17}\"\n\nis_aws_sso_logged_in(){\n    aws sts get-caller-identity &>/dev/null\n}\n\naws_account_id(){\n    aws sts get-caller-identity --query Account --output text\n}\n\naws_account_name(){\n    # you may not have permission to the AWS Org in which case this will return an error:\n    #\n    #   An error occurred (AccessDeniedException) when calling the DescribeAccount operation: You don't have permissions to access this resource.\n    #\n    aws organizations describe-account --account-id \"$AWS_ACCOUNT_ID\" --query \"Account.Name\" --output text\n}\n\naws_region(){\n    # region actually used by the CLI, but some accounts running scripts may not have permissions to this\n    # aws ec2 describe-availability-zones --output text --query 'AvailabilityZones[0].[RegionName]'\n    if [ -n \"${AWS_DEFAULT_REGION:-}\" ]; then\n        echo \"$AWS_DEFAULT_REGION\"\n        return\n    fi\n    region=\"$(aws configure get region || :)\"\n    if [ -z \"$region\" ]; then\n        region=\"$(aws ec2 describe-availability-zones --query \"AvailabilityZones[0].RegionName\" --output text || :)\"\n    fi\n    if [ -z \"$region\" ]; then\n        die \"FAILED to get AWS region in aws_region() function in lib/aws.sh\"\n    fi\n    if ! is_aws_region \"$region\"; then\n        die \"Invalid AWS Region returned in lib/aws.sh, failed regex validation: $region\"\n    fi\n    echo \"$region\"\n}\n\naws_ecr_registry(){\n    local aws_ecr_registry=\"$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com\"\n    if ! is_aws_ecr_registry \"$aws_ecr_registry\"; then\n        die \"Failed to generated AWS ECR registry correctly, failed regex: $aws_ecr_registry\"\n    fi\n    echo \"$aws_ecr_registry\"\n}\n\nis_aws_ecr_registry(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$aws_ecr_regex$ ]]\n}\n\naws_user_exists(){\n    local user=\"$1\"\n    aws iam list-users | jq -e -r \".Users[] | select(.UserName == \\\"$user\\\")\" >/dev/null\n}\n\naws_create_user_if_not_exists(){\n    local user=\"$1\"\n    if aws_user_exists \"$user\"; then\n        timestamp \"User '$user' already exists\"\n    else\n        timestamp \"Creating user '$user'\"\n        aws iam create-user --user-name \"$user\"\n    fi\n}\n\naws_create_access_key_if_not_exists(){\n    local user=\"$1\"\n    local access_keys_csv=\"$2\"\n    mkdir -pv \"$(dirname \"$access_keys_csv\")\"\n    if [ -f \"$access_keys_csv\" ] && grep -Fq AKIA \"$access_keys_csv\"; then\n        timestamp \"Access Keys CSV '$access_keys_csv' already exists\"\n        \"$libdir/../aws/aws_csv_creds.sh\" \"$access_keys_csv\"\n    else\n        local exports\n        timestamp \"Creating access key, removing an old one if necessary\"\n        exports=\"$(\"$libdir/../aws/aws_iam_replace_access_key.sh\" --user-name \"$user\")\"\n        aws_access_keys_to_csv <<< \"$exports\" >> \"$access_keys_csv\"\n        timestamp \"Created access key and saved to CSV:  $access_keys_csv\"\n        echo \"$exports\"\n    fi\n}\n\n# reads export commands and outputs CSV file format to stdout to save\naws_access_keys_to_csv(){\n    local env_var\n    local access_key\n    local secret_key\n    while read -r line; do\n        is_blank \"$line\" && continue\n        env_var=\"${line%%#*}\"\n        env_var=\"${env_var##[[:space:]]}\"\n        env_var=\"${env_var##export}\"\n        env_var=\"${env_var##[[:space:]]}\"\n        if ! [[ \"$env_var\" =~ ^[[:alpha:]][[:alnum:]_]+=.+$ ]]; then\n            die \"invalid environment key=value argument passed to aws_access_keys_to_csv(): $env_var\"\n        fi\n        key=\"${env_var%%=*}\"\n        value=\"${env_var#*=}\"\n        if [ \"$key\" = \"AWS_ACCESS_KEY_ID\" ]; then\n            access_key=\"$value\"\n        elif [ \"$key\" = \"AWS_SECRET_ACCESS_KEY\" ]; then\n            secret_key=\"$value\"\n        else\n            die \"unexpected key '$key' passed to aws_access_keys_to_csv() - only expected AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY\"\n        fi\n    done\n    if is_blank \"$access_key\"; then\n        die \"aws_access_keys_to_csv(): failed to parse access key\"\n    fi\n    if is_blank \"$secret_key\"; then\n        die \"aws_access_keys_to_csv(): failed to parse secret key\"\n    fi\n    echo \"Access key ID,Secret access key\"  # header line to match the AWS console UI\n    echo \"$access_key,$secret_key\"\n}\n\n# reads export commands and outputs ~/aws/credentials file format to stdout to save\naws_access_keys_exports_to_credentials(){\n    local profile=\"${AWS_PROFILE:-default}\"\n    local env_var\n    local key\n    local value\n    echo \"[$profile]\"\n    while read -r line; do\n        is_blank \"$line\" && continue\n        env_var=\"${line%%#*}\"\n        env_var=\"${env_var##[[:space:]]}\"\n        env_var=\"${env_var##export}\"\n        env_var=\"${env_var##[[:space:]]}\"\n        if ! [[ \"$env_var\" =~ ^[[:alpha:]][[:alnum:]_]+=.+$ ]]; then\n            die \"invalid environment key=value argument passed to aws_access_keys_exports_to_credentials(): $env_var\"\n        fi\n        key=\"${env_var%%=*}\"\n        value=\"${env_var#*=}\"\n        echo \"$(tr '[:upper:]' '[:lower:]' <<< \"$key\")=$value\"\n    done\n}\n\n# returns one cluster name per line\naws_eks_clusters(){\n    aws eks list-clusters --query 'clusters' --output text |\n    tr '[:space:]' '\\n' |\n    sed '/^[[:space:]]*$/d'\n}\n\naws_eks_cluster_if_only_one(){\n    local eks_clusters\n    eks_clusters=\"$(aws_eks_clusters)\"\n    num_eks_clusters=\"$(grep -c . <<< \"$eks_clusters\")\"\n    if [ \"$num_eks_clusters\" = 1 ]; then\n        echo \"$eks_clusters\"\n    fi\n}\n\naws_validate_volume_id(){\n    local volume_id=\"$1\"\n    if ! [[ \"$volume_id\" =~ ^vol-[[:alnum:]]{17}$ ]]; then\n        usage \"\n    Invalid volume ID given, expected format: vol-xxxxxxxxxxxxxxxxx,\n                                   but given: $volume_id\"\n    fi\n}\n\naws_sso_login_if_not_already(){\n    if ! is_aws_sso_logged_in; then\n        # output to stderr so that if we are collecting the output from this script,\n        # we do not collect any output from sso login\n        aws sso login 2>&1\n    fi\n}\n\naws_sso_cache(){\n    # find is not as good for finding the sorted latest cache file\n    # shellcheck disable=SC2012\n    ls -t ~/.aws/sso/cache/*.json | head -n1\n}\n\naws_sso_token(){\n    local sso_cache_file\n    sso_cache_file=\"$(aws_sso_cache)\"\n    jq -r .accessToken < \"$sso_cache_file\"\n}\n\naws_sso_role(){\n    role=\"${AWS_DEFAULT_ROLE:-${AWS_ROLE:-${ROLE:-}}}\"\n    if ! is_blank \"$role\"; then\n        log \"Using role from environment variable: $role\"\n        echo \"$role\"\n    else\n        log \"Determining role from currently authenticated AWS SSO role\"\n        role=\"$(\n        aws sts get-caller-identity --query Arn --output text |\n        sed '\n            s|^arn:aws:sts:.*:assumed-role/AWSReservedSSO_||;\n            s|_[[:alnum:]]\\{16\\}/.*$||;\n        '\n    )\"\n        log \"Determined role to be: $role\"\n        echo \"$role\"\n    fi\n}\n\naws_sso_start_url(){\n    local sso_start_url\n    log \"Determining SSO Start URL\"\n    sso_start_url=\"$(aws configure get sso_start_url || :)\"\n    if is_blank \"$sso_start_url\"; then\n        # shouldn't fall through to this\n        log \"Failed to determine SSO Start URL from 'aws configure', falling back to trying the highest occurence in sso cache file\"\n        local sso_cache_file\n        sso_cache_file=\"$(aws_sso_cache)\"\n        sso_start_url=\"$(\n            jq -Mr '.startUrl' \"$sso_cache_file\" |\n            sed '/^null$/d' |\n            sort |\n            uniq -c |\n            sort -nr |\n            awk '{print \\$2; exit}'\n        )\"\n    fi\n    if ! is_url \"$sso_start_url\"; then\n        die \"Invalid AWS SSO Start URL returned in lib/aws.sh, failed regex validation: $sso_start_url\"\n    fi\n    log \"Determined SSO Start URL: $sso_start_url\"\n    echo \"$sso_start_url\"\n}\n\naws_sso_start_region(){\n    log \"Determining SSO Start Region from config\"\n    local sso_cache_file\n    sso_cache_file=\"$(aws_sso_cache)\"\n    local sso_start_region\n    sso_start_region=\"$(jq -Mr '.region' \"$sso_cache_file\")\"\n    if ! is_aws_region \"$sso_start_region\"; then\n        die \"Invalid AWS SSO Start Region returned, failed regex validation: $sso_start_region\"\n    fi\n    log \"Determined SSO Start Region: $sso_start_region\"\n    echo \"$sso_start_region\"\n}\n\naws_region_from_env(){\n    local region\n    region=\"${AWS_DEFAULT_REGION:-${AWS_REGION:-${REGION:-}}}\"\n    if ! is_blank \"$region\"; then\n        log \"Using region from environment variable: $region\"\n    else\n        region=\"$(aws_region)\"\n        if ! is_blank \"$region\"; then\n            log \"Inferred region to be: $region\"\n        elif ! is_blank \"${aws_default_region:-}\"; then\n            region=\"$aws_default_region\"\n            log \"Defaulting to using region: $region\"\n        else\n            echo \"AWS region not found from environment variables or AWS mechanism\" >&2\n            return 1\n        fi\n    fi\n    if ! is_aws_region \"$region\"; then\n        die \"Invalid AWS region, failed regex validation: $region\"\n    fi\n    echo \"$region\"\n}\n\nis_aws_account_id(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$aws_account_id_regex$ ]]\n}\n\nis_aws_region(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$aws_region_regex$ ]]\n}\n\nis_s3_url(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$s3_regex$ ]]\n}\n\nis_instance_id(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$instance_id_regex$ ]]\n}\nis_ami_id(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$ami_id_regex$ ]]\n}\n\naws_validate_ami_id() {\n    local arg=\"$1\"\n    if ! is_instance_id \"$arg\"; then\n        die \"Invalid EC2 AMI ID: $arg\"\n    fi\n}\n\naws_validate_instance_id() {\n    local arg=\"$1\"\n    if ! is_instance_id \"$arg\"; then\n        die \"Invalid EC2 Instance ID: $arg\"\n    fi\n}\n\naws_validate_security_group_id() {\n    local arg=\"$1\"\n    if ! is_aws_security_group_id \"$arg\"; then\n        die \"Invalid Security Group ID: $arg\"\n    fi\n}\n\nis_aws_security_group_id() {\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$aws_sg_regex$ ]]\n}\n\naws_validate_subnet_id() {\n    local arg=\"$1\"\n    if ! is_aws_subnet_id \"$arg\"; then\n        die \"Invalid Subnet ID: $arg\"\n    fi\n}\n\nis_aws_subnet_id() {\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$aws_subnet_regex$ ]]\n}\n\n#aws_get_cred_path(){\n#    # unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n#    [ -n \"${HOME:-}\" ] || HOME=~\n#    local aws_credentials=\"${AWS_SHARED_CREDENTIALS_FILE:-$HOME/.aws/credentials}\"\n#    local aws_config=\"${AWS_CONFIG_FILE:-$HOME/.aws/config}\"\n#    local boto=\"${BOTO_CONFIG:-$HOME/.boto}\"\n#    local credentials_file\n#    if [ -f \"$aws_credentials\" ]; then\n#        credentials_file=\"$aws_credentials\"\n#    # older boto creds\n#    elif [ -f \"$boto\" ]; then\n#        credentials_file=\"$boto\"\n#    elif [ -f \"$aws_config\" ]; then\n#        credentials_file=\"$aws_config\"\n#    else\n#        echo \"no credentials found - didn't find $aws_credentials or $boto or $aws_config\" 2>/dev/null\n#        return 1\n#    fi\n#    echo \"$credentials_file\"\n#}\n#aws_credentials_file=\"$(aws_get_cred_path)\"\n"
  },
  {
    "path": "lib/bitbucket.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-16 09:52:29 +0100 (Sun, 16 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nlib_srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$lib_srcdir/utils.sh\"\n\nget_bitbucket_user(){\n    # reuse bitbucket_user between function calls for efficiency to save additional queries to the BitBucket API\n    if [ -z \"${bitbucket_user:-}\" ]; then\n        log \"Attempting to infer username\"\n        if [ -n \"${BITBUCKET_USER:-}\" ]; then\n            bitbucket_user=\"$BITBUCKET_USER\"\n            log \"Using username '$bitbucket_user' from \\$BITBUCKET_USER\"\n        else\n            log \"Querying BitBucket API for currently authenticated username\"\n            bitbucket_user=\"$(\"$lib_srcdir/../bitbucket/bitbucket_api.sh\" /user | jq -r .username)\"\n            log \"BitBucket API returned username '$bitbucket_user'\"\n        fi\n    fi\n    echo \"$bitbucket_user\"\n}\n"
  },
  {
    "path": "lib/ci.sh",
    "content": "#!/bin/sh\n#  shellcheck disable=SC2128,SC2230,SC1090\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Python-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# Azure DevOps env var, chain to standard debug across all programs, scripts and repos\nif [ \"${SYSTEM_DEBUG:-}\" = \"true\" ]; then\n    export DEBUG=1\nfi\n\n# https://jenkins.io/doc/book/pipeline/jenkinsfile/#using-environment-variables\nis_jenkins(){\n    # also BUILD_ID, BUILD_NUMBER, BUILD_URL but less specific, caught in is_CI generic\n    # JENKINS_URL is set in client side environment for various CLIs and scripts, can't rely on just that\n    if [ -n \"${JENKINS_URL:-}\" ] &&\n       [ -n \"${EXECUTOR_NUMBER:-}\" ] &&\n       [ -n \"${JOB_NAME:-}\" ] &&\n       [ -n \"${BUILD_ID:-}\" ] &&\n       [ -n \"${BUILD_URL:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://docs.travis-ci.com/user/environment-variables/#default-environment-variables\nis_travis(){\n    # also TRAVIS_JOB_ID\n    if [ -n \"${TRAVIS:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://www.appveyor.com/docs/environment-variables/\nis_appveyor(){\n    # also CI but not really specific, caught in is_CI generic\n    if [ -n \"${APPVEYOR:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://support.atlassian.com/bitbucket-cloud/docs/variables-and-secrets/\nis_bitbucket_ci(){\n    # also CI but not really specific, caught in is_CI generic\n    if [ -n \"${BITBUCKET_BUILD_NUMBER:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://buddy.works/docs/pipelines/environment-variables#default-environment-variables\nis_buddy_works_ci(){\n    # also BUDDY but a bit too generic for my liking\n    if [ -n \"${BUDDY_PIPELINE_ID:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://buildkite.com/docs/pipelines/environment-variables\nis_buildkite(){\n    # also CI but not really specific, caught in is_CI generic\n    if [ -n \"${BUILDKITE:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# Concourse team refuse to add this variable like all the other normal CI systems\n# so it relies on you having added it yourself (see .concourse.yml in any of my repos for example)\nis_concourse(){\n    if [ -n \"${CONCOURSE:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://circleci.com/docs/2.0/env-vars/#built-in-environment-variables\nis_circle_ci(){\n    # also CI but not really specific, caught in is_CI generic\n    # also CIRCLE_JOB\n    if [ -n \"${CIRCLECI:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://cirrus-ci.org/guide/writing-tasks/\nis_cirrus_ci(){\n    # also CI but not really specific, caught in is_CI generic\n    if [ -n \"${CIRRUS_CI:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://codefresh.io/docs/docs/codefresh-yaml/variables/\nis_codefresh(){\n    # also CI but not really specific, caught in is_CI generic\n    if [ -n \"${CF_BUILD_ID:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://documentation.codeship.com/basic/builds-and-configuration/set-environment-variables/#default-environment-variables\nis_codeship(){\n    # also CI and other generic CI_ env vars caught in is_CI generic\n    # formerly codeship\n    if [ \"${CI_NAME:-}\" = \"CodeShip\" ]; then\n        return 0\n    fi\n    return 1\n}\n\nis_drone_io(){\n    # also CI and other generic CI_ env vars caught in is_CI generic\n    if [ -n \"${DRONE:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://help.github.com/en/actions/configuring-and-managing-workflows/using-environment-variables#default-environment-variables\nis_github_workflow(){\n    if [ -n \"${GITHUB_ACTIONS:-}\" ] ||\n       [ -n \"${GITHUB_WORKFLOW:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\nis_github_action(){\n    is_github_workflow\n}\n\n# https://docs.gitlab.com/ee/ci/variables/predefined_variables.html\nis_gitlab_ci(){\n    # also CI and other generic CI_ env vars caught in is_CI generic\n    if [ -n \"${GITLAB_CI:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://docs.gocd.org/current/faq/dev_use_current_revision_in_build.html\nis_gocd(){\n    if [ -n \"${GO_PIPELINE_NAME:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://scrutinizer-ci.com/docs/build/environment-variables\nis_scrutinizer_ci(){\n    # also CI but specific to Scrutinizer, caught in is_CI generic\n    if [ -n \"${SCRUTINIZER:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# incomplete documentation - CI code framework in this repo reveals a lot more env vars\n# https://help.semmle.com/wiki/display/SD/Environment+variables\nis_semmle(){\n    if [ -n \"${SEMMLE_PLATFORM:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://docs.semaphoreci.com/ci-cd-environment/environment-variables/\nis_semaphore_ci(){\n    # too generic, might clash with something else in future\n    #if [ -n \"${SEMAPHORE:-}\" ]; then\n    if [ -n \"${SEMAPHORE_PIPELINE_ID:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# http://docs.shippable.com/ci/env-vars/#stdEnv\nis_shippable_ci(){\n    # also CI and CONTINUOUS_INTEGRATION but not really specific to Shippable, caught in is_CI generic\n    # also $SHIPPABLE_JOB_ID / $SHIPPABLE_JOB_NUMBER\n    if [ -n \"${SHIPPABLE:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://www.jetbrains.com/help/teamcity/predefined-build-parameters.html\nis_teamcity_ci(){\n    # also BUILD_NUMBER, but less specific, caught in is_CI generic\n    if [ -n \"${TEAMCITY_VERSION:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\nis_teamcity(){ is_teamcity_ci; }\n\n# same as Azure DevOps\n# https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables\nis_tfs_ci(){\n    if [ -n \"${TF_BUILD:-}\" ]; then\n        return 0\n    fi\n    return 1\n}\n\n# https://docs.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables\n#\n# docs aren't great, testing shows these env vars of interest:\n#\n# AGENT_NAME=Azure Pipelines 2\n# POWERSHELL_DISTRIBUTION_CHANNEL=Azure-DevOps-ubuntu18\n# SYSTEM_TASKDEFINITIONSURI=https://dev.azure.com/harisekhon/\n# SYSTEM_TEAMFOUNDATIONCOLLECTIONURI=https://dev.azure.com/harisekhon/\n# SYSTEM_TEAMFOUNDATIONSERVERURI=https://dev.azure.com/harisekhon/\n# TF_BUILD=True\n#\nis_azure_devops(){\n    if is_tfs_ci; then\n        return 0\n    fi\n    return 1\n}\n\nis_CI(){\n    if [ -n \"${CI:-}\" ] ||\n       [ -n \"${CI_NAME:-}\" ] ||\n       [ -n \"${CONTINUOUS_INTEGRATION:-}\" ] ||\n       [ -n \"${BUILD_ID:-}\" ] ||\n       [ -n \"${BUILD_NUMBER:-}\" ] ||\n       [ -n \"${BUILD_URL:-}\" ] ||\n       is_jenkins ||\n       is_teamcity ||\n       is_travis ||\n       is_circle_ci ||\n       is_github_workflow ||\n       is_gitlab_ci ||\n       is_azure_devops ||\n       is_appveyor ||\n       is_bitbucket_ci ||\n       is_buildkite ||\n       is_buddy_works_ci ||\n       is_codeship ||\n       is_codefresh ||\n       is_concourse ||\n       is_cirrus_ci ||\n       is_drone_io ||\n       is_gocd ||\n       is_scrutinizer_ci ||\n       is_semmle ||\n       is_semaphore_ci ||\n       is_shippable_ci ||\n       is_tfs_ci; then\n        return 0\n    fi\n    return 1\n}\n\nCI_name(){\n    if [ -n \"${CI_NAME:-}\" ]; then\n        echo \"$CI_NAME\"\n    elif is_jenkins; then\n        echo \"Jenkins\"\n    elif is_teamcity; then\n        echo \"TeamCity\"\n    elif is_travis; then\n        echo \"Travis CI\"\n    elif is_circle_ci; then\n        echo \"Circle CI\"\n    elif is_github_workflow; then\n        echo \"GitHub Actions\"\n    elif is_gitlab_ci; then\n        echo \"Gitlab CI\"\n    elif is_azure_devops; then\n        echo \"Azure DevOps\"\n    elif is_appveyor; then\n        echo \"AppVeyor\"\n    elif is_bitbucket_ci; then\n        echo \"Bitbucket Pipelines\"\n    elif is_buildkite; then\n        echo \"BuildKite\"\n    elif is_buddy_works_ci; then\n        echo \"Buddy CI\"\n    elif is_codeship; then\n        echo \"CodeShip\"\n    elif is_codefresh; then\n        echo \"Codefresh\"\n    elif is_concourse; then\n        echo \"Concourse\"\n    elif is_cirrus_ci; then\n        echo \"Cirrus CI\"\n    elif is_drone_io; then\n        echo \"Drone CI\"\n    elif is_gocd; then\n        echo \"GoCD\"\n    elif is_scrutinizer_ci; then\n        echo \"Scrutinizer\"\n    elif is_semmle; then\n        echo \"Semmle\"\n    elif is_semaphore_ci; then\n        echo \"Semaphore CI\"\n    elif is_shippable_ci; then\n        echo \"Shippable\"\n    elif is_tfs_ci; then\n        echo \"TFS CI\"\n    elif is_CI; then\n        echo \"CI\"\n    else\n        echo \"Not a recognized CI system\"\n        return 1\n    fi\n}\n\nif is_travis; then\n    #export DOCKER_HOST=\"${DOCKER_HOST:-localhost}\"\n    export HOST=\"${HOST:-localhost}\"\nfi\n\nif is_CI; then\n    # used by Artifactory JFrog CLI to disable interactive prompts and progress bar - https://www.jfrog.com/confluence/display/CLI/JFrog+CLI#JFrogCLI-EnvironmentVariables\n    export CI=true\nfi\n"
  },
  {
    "path": "lib/cloudera_manager.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 15:08:10 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick lib to be sourced for auto-populating Cloudera Manager details in a script\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nCLOUDERA_MANAGER_HOST=\"${CLOUDERA_MANAGER_HOST:-${CLOUDERA_MANAGER:-${CLOUDERA_HOST:-${HOST:-}}}}\"\nCLOUDERA_MANAGER_PORT=\"${CLOUDERA_MANAGER_PORT:-${CLOUDERA_PORT:-${PORT:-7180}}}\"\n\nCLOUDERA_MANAGER_HOST=\"${CLOUDERA_MANAGER_HOST##*://}\"\nCLOUDERA_MANAGER_HOST=\"${CLOUDERA_MANAGER_HOST%%/*}\"\nCLOUDERA_MANAGER_HOST=\"${CLOUDERA_MANAGER_HOST%%:*}\"\n\nif [ -z \"$CLOUDERA_MANAGER_HOST\" ]; then\n    read -r -p 'Enter Cloudera Manager host URL: ' CLOUDERA_MANAGER\nfi\n\nCLOUDERA_MANAGER=\"http://$CLOUDERA_MANAGER_HOST:$CLOUDERA_MANAGER_PORT\"\n\nexport USER=\"${CLOUDERA_MANAGER_USER:-${CLOUDERA_USER:-${USER:-}}}\"\nexport PASSWORD=\"${CLOUDERA_MANAGER_PASSWORD:-${CLOUDERA_PASSWORD:-${PASSWORD:-}}}\"\n\nif [ -n \"${CLOUDERA_MANAGER_SSL:-}\" ]; then\n    CLOUDERA_MANAGER=\"https://${CLOUDERA_MANAGER#*://}\"\n    if [[ \"$CLOUDERA_MANAGER\" =~ :7180$ ]]; then\n        CLOUDERA_MANAGER=\"${CLOUDERA_MANAGER%:7180}:7183\"\n    fi\nfi\n\n# seems to work on CM / CDH 5.10.0 even when cluster is set to 'blah' but probably shouldn't rely on that\nCLOUDERA_CLUSTER=\"${CLOUDERA_CLUSTER:-${CLOUDERA_MANAGER_CLUSTER:-}}\"\nif [ -z \"${CLOUDERA_CLUSTER:-}\" ]; then\n    read -r -p 'Enter Cloudera Manager Cluster name: ' CLOUDERA_CLUSTER\nfi\n\n# 2020-01-02T16%3A17%3A57.514Z\n# url encoding : => %3A seems to be done automatically by curl so not bothering to urlencode here\n# shellcheck disable=SC2034\nnow_timestamp=\"$(date '+%Y-%m-%dT%H:%M:%S.000Z')\"\n"
  },
  {
    "path": "lib/cloudera_navigator.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-01-23 15:08:10 +0000 (Thu, 23 Jan 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick lib to be sourced for auto-populating Cloudera Navigator details in a script\n\n# Tested on Cloudera Enterprise 5.10\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nCLOUDERA_NAVIGATOR_HOST=\"${CLOUDERA_NAVIGATOR_HOST:-${CLOUDERA_NAVIGATOR:-${CLOUDERA_HOST:-${HOST:-}}}}\"\nCLOUDERA_NAVIGATOR_PORT=\"${CLOUDERA_NAVIGATOR_PORT:-${CLOUDERA_PORT:-${PORT:-7186}}}\"\n\nCLOUDERA_NAVIGATOR_HOST=\"${CLOUDERA_NAVIGATOR_HOST##*://}\"\nCLOUDERA_NAVIGATOR_HOST=\"${CLOUDERA_NAVIGATOR_HOST%%/*}\"\nCLOUDERA_NAVIGATOR_HOST=\"${CLOUDERA_NAVIGATOR_HOST%%:*}\"\n\nif [ -z \"$CLOUDERA_NAVIGATOR_HOST\" ]; then\n    read -r -p 'Enter Cloudera Navigator host URL: ' CLOUDERA_NAVIGATOR_HOST\nfi\n\nexport USER=\"${CLOUDERA_NAVIGATOR_USER:-${CLOUDERA_USER:-${USER:-}}}\"\nexport PASSWORD=\"${CLOUDERA_NAVIGATOR_PASSWORD:-${CLOUDERA_PASSWORD:-${PASSWORD:-}}}\"\n\nCLOUDERA_NAVIGATOR=\"http://$CLOUDERA_NAVIGATOR_HOST:$CLOUDERA_NAVIGATOR_PORT\"\n\nif [ -n \"${CLOUDERA_NAVIGATOR_SSL:-}\" ]; then\n    CLOUDERA_NAVIGATOR=\"https://${CLOUDERA_NAVIGATOR#*://}\"\n    if [[ \"$CLOUDERA_NAVIGATOR\" =~ :7186$ ]]; then\n        CLOUDERA_NAVIGATOR=\"${CLOUDERA_NAVIGATOR%:7186}:7187\"\n    fi\nfi\n\n# shellcheck disable=SC2034\n# Navigator expects timestamp in millis\nnow_timestamp=\"$(date +%s000)\"\n"
  },
  {
    "path": "lib/dbshell.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2034\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-05 13:42:41 +0100 (Wed, 05 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Utility library for the postgres / mysql / mariadb scripts at top level\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")/..\" && pwd)\"\n\nsql_scripts=\"$srcdir/sql\"\nif [ -d \"$srcdir/../sql\" ]; then\n    sql_scripts=\"$srcdir/../sql\"\nfi\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nsql_mount_description=\"\nSQL  scripts     => /sql  <- session \\$PWD for convenient sql sourcing\nBash scripts     => /bash\nhost \\$PWD        => /pwd\n\\$HOME/github     => /github\n\"\n\ndocker_sql_mount_switches=\" \\\n    -v $srcdir:/bash \\\n    -v $sql_scripts:/sql \\\n    -v $HOME/github:/github \\\n    -v $PWD:/pwd \\\n\"\n\n# MySQL 5.5 container\n#\n# MySQL init process done. Ready for start up.\n# ...\n# 200808 19:30:58 [Note] mysqld: ready for connections.\nwait_for_mysql_ready(){\n    local container_name=\"$1\"\n    local max_secs=90\n    local num_lines=50\n    local tries=0\n    SECONDS=0\n    while true; do\n        ((tries+=1))\n        if [ $((tries % 5)) = 1 ]; then\n            timestamp 'waiting for mysqld to be ready to accept connections before connecting mysql shell...'\n        fi\n        if docker_container_exited \"$container_name\"; then\n            echo \"ERROR: '$container_name' container died!\" >&2\n            echo >&2\n            docker logs \"$container_name\"\n            exit 1\n        fi\n        if docker logs --tail \"$num_lines\" \"$container_name\" 2>&1 |\n            grep -i -A \"$num_lines\" \\\n                 -e 'Entrypoint.*Ready' \\\n                 -e 'MySQL init process done' |\n            grep -q \\\n                 -e 'mysqld.*ready for connections' \\\n                 -e 'mysqld.*ready to accept connections'; then\n            break\n        fi\n        sleep 1\n        if [ $SECONDS -gt $max_secs ]; then\n            timestamp \"container '$container_name' failed to become ready for connections within $max_secs secs, check logs (format may have changed):\"\n            echo >&2\n            docker logs \"$container_name\"\n            exit 1\n        fi\n    done\n}\n\nwait_for_postgres_ready(){\n    local container_name=\"$1\"\n    local max_secs=60\n    local num_lines=50\n    # PostgreSQL 84:\n    #\n    # PostgreSQL stand-alone backend 8\n    # ...\n    # LOG:  database system is ready to accept connections\n    #\n    #\n    # PostgreSQL 11.8:\n    #\n    # PostgreSQL init process complete; ready for start up.\n    # ...\n    # 2020-08-09 21:56:04.824 GMT [1] LOG:  database system is ready to accept connections\n    #\n    SECONDS=0\n    while true; do\n        timestamp 'waiting for postgres to be ready to accept connections before connecting psql...'\n        if docker_container_exited \"$container_name\"; then\n            echo \"ERROR: '$container_name' container died!\" >&2\n            echo >&2\n            docker logs \"$container_name\"\n            exit 1\n        fi\n        if docker logs --tail \"$num_lines\" \"$container_name\" 2>&1 |\n           grep -E -A \"$num_lines\" \\\n           -e 'PostgreSQL init.*(ready|complete)' \\\n           -e 'PostgreSQL stand-alone backend 8' |\n           grep -q 'ready to accept connections'; then\n            break\n        fi\n        sleep 1\n        if [ $SECONDS -gt $max_secs ]; then\n            echo \"PostgreSQL failed to become ready for connections within $max_secs secs, check logs (format may have changed):\"\n            echo\n            docker logs \"$container_name\"\n            exit 1\n        fi\n    done\n}\n\ndocker_rm_when_last_connection(){\n    local scriptname=\"$1\"\n    local container_name=\"$2\"\n    [ -z \"${DOCKER_NO_DELETE:-}\" ] || return\n    if [ \"$(lsof -lnt \"$scriptname\" | grep -c .)\" -lt 2 ]; then\n    #if [ \"$(pgrep -lf \"bash.*${0##*/}\" | grep -c .)\" -lt 2 ]; then\n    #if [ \"$(ps -ef | grep -c \"[b]ash.*${0##*/}\")\" -lt 2 ]; then\n        timestamp \"last session closing, deleting container:\"\n        docker rm -f -- \"$container_name\"\n    fi\n}\n\nstrip_requires_db_pre(){\n    sed 's/.*Requires[[:space:]].*'\"$db//\"\n}\n\nstrip_requires_db_post(){\n    sed '\n        s/.*Requires[[:space:]]*//;\n        s/'\"$db\"'[[:space:]]*$//\n    '\n}\n\n# detect version headers and only run if the version corresponds\nskip_min_version(){\n    local db=\"$1\"\n    local version=\"$2\"\n    local sql_file=\"$3\"\n    local min_version\n    local inclusive=\"\"\n    if grep -Eiom 1 -- '--[[:space:]]*Requires[[:space:]]+'\"$db\"'[[:space:]]*N/A' \"$sql_file\"; then\n        timestamp \"skipping script '$sql_file' due to N/A version\"\n        return 0\n    fi\n    # some versions of sed don't support +, so stick to *\n    #                                             Requires PostgreSQL 9.2+\n    #                                             Requires 9.2 <= PostgreSQL <= 9.5\n    min_version=\"$(grep -Eiom 1 -- '--[[:space:]]*Requires[[:space:]]+'\"$db\"'[[:space:]]+(>=?)?[[:space:]]*[[:digit:]]+(\\.[[:digit:]]+)?' \"$sql_file\" | strip_requires_db_pre || :)\"\n    if [ -z \"$min_version\" ]; then\n        min_version=\"$(grep -Eiom 1 -- '--[[:space:]]*Requires[[:space:]]+[[:digit:]]+(\\.[[:digit:]]+)?[[:space:]]*<=?[[:space:]]*'\"$db\" \"$sql_file\" | strip_requires_db_post || :)\"\n    fi\n    if [ -n \"$min_version\" ] &&\n       [ \"$version\" != latest ]; then\n        # shellcheck disable=SC1001\n        if [[ \"$min_version\" =~ \\= ]] ||\n           ! [[ \"$min_version\" =~ [\\<\\>] ]]; then\n            inclusive=\"=\"\n        fi\n        min_version=\"${min_version/>}\"\n        min_version=\"${min_version/<}\"\n        min_version=\"${min_version/=}\"\n        min_version=\"${min_version//[[:space:]]}\"\n        is_float \"$min_version\" || die \"code error: non-float '$min_version' parsed in skip_min_version()\"\n        skip_msg=\"skipping script '$sql_file' due to min required version >$inclusive $min_version\"\n        is_float \"$version\" || die \"code error: non-float '$version' passed to skip_min_version()\"\n        if [ -n \"$inclusive\" ]; then\n            if bc_bool \"$version < $min_version\"; then\n                timestamp \"$skip_msg\"\n                return 0\n            fi\n        else\n            if bc_bool \"$version <= $min_version\"; then\n                timestamp \"$skip_msg\"\n                return 0\n            fi\n        fi\n    fi\n    return 1\n}\n\n# detect version headers and only run if the version corresponds\nskip_max_version(){\n    local db=\"$1\"\n    local version=\"$2\"\n    local sql_file=\"$3\"\n    local max_version\n    local inclusive=\"\"\n    if grep -Eiom 1 -- '--[[:space:]]*Requires[[:space:]]+'\"$db\"'[[:space:]]*N/A' \"$sql_file\"; then\n        timestamp \"skipping script '$sql_file' due to N/A version\"\n        return 0\n    fi\n    # some versions of sed don't support +, so stick to *\n    #                                             Requires PostgreSQL <= 9.1\n    max_version=\"$(grep -Eiom 1 -- '--[[:space:]]*Requires[[:space:]]+.*'\"$db\"'[[:space:]]+<=?[[:space:]]*[[:digit:]]+(\\.[[:digit:]]+)?' \"$sql_file\" | strip_requires_db_pre || :)\"\n    if [ -n \"$max_version\" ]; then\n        if [[ \"$max_version\" =~ = ]]; then\n            inclusive=\"=\"\n        fi\n        max_version=\"${max_version/<}\"\n        max_version=\"${max_version/>}\"\n        max_version=\"${max_version/=}\"\n        max_version=\"${max_version//[[:space:]]}\"\n        is_float \"$max_version\" || die \"code error: non-float '$max_version' parsed in skip_max_version()\"\n        skip_msg=\"skipping script '$sql_file' due to max required version <$inclusive $max_version\"\n        # we can't know if the latest version has moved beyond the max version, so assume it has to be safe and return true to skip\n        if [ \"$version\" = latest ]; then\n            timestamp \"$skip_msg\"\n            return 0\n        fi\n        is_float \"$version\" || die \"code error: non-float '$version' passed to skip_max_version()\"\n        # if version is an int, eg. 12, implies 12.x, so make the requirement to be greater than or equal to the next integer up\n        # since 12.3 > 12 will return to skip, but humanly we want <= 12 to allow all versions of 12, only skipping for 13+\n        if [[ \"$max_version\" =~ ^[[:digit:]]+$ ]]; then\n            ((max_version+=1))\n            inclusive=\"\"\n        fi\n        if [ -n \"$inclusive\" ]; then\n            if bc_bool \"$version > $max_version\"; then\n                timestamp \"$skip_msg\"\n                return 0\n            fi\n        else\n            if bc_bool \"$version >= $max_version\"; then\n                timestamp \"$skip_msg\"\n                return 0\n            fi\n        fi\n    fi\n    return 1\n}\n"
  },
  {
    "path": "lib/docker.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-02-07 22:42:47 +0000 (Sun, 07 Feb 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_bash_tools_docker=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=1090,SC1091\n. \"$srcdir_bash_tools_docker/utils.sh\"\n\ndocker_compose_quiet=\"\"\nif is_CI; then\n    docker_compose_quiet=\"--quiet\"\nfi\n\nis_docker_installed(){\n    if type -P docker &>/dev/null; then\n        return 0\n    fi\n    return 1\n}\n\nis_docker_available(){\n    #[ -n \"${TRAVIS:-}\" ] && return 0\n    if is_docker_installed; then\n        if docker info &>/dev/null; then\n            return 0\n        fi\n    fi\n    #echo \"Docker not available\"\n    return 1\n}\n\nis_docker_compose_available(){\n    #[ -n \"${TRAVIS:-}\" ] && return 0\n    if type -P docker-compose &>/dev/null; then\n        return 0\n    fi\n    #echo \"Docker Compose not available\"\n    return 1\n}\n\ncheck_docker_available(){\n    if ! is_docker_installed; then\n        echo 'WARNING: Docker not found, skipping checks!!!'\n        exit 0\n    fi\n    if ! is_docker_available; then\n        echo 'WARNING: Docker not available, skipping checks!!!'\n        exit 0\n    fi\n    if ! is_docker_compose_available; then\n        echo \"WARNING: Docker Compose not found in \\$PATH, skipping checks!!!\"\n        exit 0\n    fi\n    # alternative\n    #export DOCKER_SERVICE=\"$(ps -o comm= $PPID)\"\n    export DOCKER_SERVICE=\"${BASH_SOURCE[1]#*test_}\"\n    export DOCKER_SERVICE=\"${DOCKER_SERVICE%.sh}\"\n    export DOCKER_CONTAINER=\"${COMPOSE_PROJECT_NAME:-docker}\"\n    # for Docker Machine but not Docker for Mac\n    # nagios-plugins -> nagiosplugins\n    export DOCKER_CONTAINER=\"${DOCKER_CONTAINER//-}\"\n    export DOCKER_CONTAINER=\"${DOCKER_CONTAINER}_${DOCKER_SERVICE}_1\"\n    # srcdir is defined in client scripts\n    # shellcheck disable=SC2154\n    export COMPOSE_FILE=\"$srcdir/docker/$DOCKER_SERVICE-docker-compose.yml\"\n}\n\nis_docker_container_running(){\n    local containers\n    containers=\"$(docker ps)\"\n    if [ -n \"${DEBUG:-}\" ]; then\n        echo \"Containers Running:\n$containers\n\"\n    fi\n    #if grep -q \"[[:space:]]$1$\" <<< \"$containers\"; then\n    if [[ \"$containers\" =~ [[:space:]]$1$ ]]; then\n        return 0\n    fi\n    return 1\n}\n\nis_inside_docker(){\n    test -f /.dockerenv\n}\n\n# KUBERNETES_PORT=tcp://<ip_x.x.x.x>:443\n# KUBERNETES_PORT_443_TCP=tcp://<ip_x.x.x.x>:443\n# KUBERNETES_PORT_443_TCP_ADDR=<ip_x.x.x.x>\n# KUBERNETES_PORT_443_TCP_PORT=443\n# KUBERNETES_PORT_443_TCP_PROTO=tcp\n# KUBERNETES_SERVICE_HOST=<ip_x.x.x.x>\n# KUBERNETES_SERVICE_PORT=443\n# KUBERNETES_SERVICE_PORT_HTTPS=443\nis_inside_kubernetes(){\n    [ -n \"${KUBERNETES_PORT:-}\" ]\n}\n\ndeclare_if_inside_docker(){\n    if is_inside_docker; then\n        echo\n        echo \"(running in Docker container $(hostname -f))\"\n        echo\n    fi\n}\n\ndocker_compose_pull(){\n    if [ -z \"${KEEPDOCKER:-}\" ]; then\n        if is_CI || [ -n \"${DOCKER_PULL:-}\" ]; then\n            VERSION=\"${version:-}\" docker-compose pull $docker_compose_quiet || :\n        fi\n    fi\n}\n\ndocker_compose_path_version(){\n    local path=\"$1\"\n    local dir_base=\"$2\"\n    if [ -z \"${DOCKER_SERVICE:-}\" ]; then\n        echo \"Error: \\$DOCKER_SERVICE has not been set in environment yet, was check_docker_available() called first?\"\n        exit 1\n    fi\n    set +e\n    local version\n    version=\"$(docker-compose exec \"$DOCKER_SERVICE\" ls \"$path\" -1 --color=no |\n                     grep --color=no -- \"$dir_base.*[[:digit:]]\" |\n                     tr -d '\\r' |\n                     tee /dev/stderr |\n                     tail -n 1 |\n                     sed \"s/$dir_base//\"\n                    )\"\n    set -e\n    if [ -z \"$version\" ]; then\n        echo \"Error: failed to find docker compose path version from path $path for $dir_base!\"\n        exit 1\n    fi\n    echo \"$version\"\n}\n\ndocker_compose_version_test(){\n    local name=\"${1:-}\"\n    local version=\"${2:-}\"\n    if [ -z \"$name\" ]; then\n        \"ERROR: missing first arg for name to docker_compose_version_test()\"\n        exit 1\n    fi\n    if [ -z \"$version\" ]; then\n        \"ERROR: missing second arg for version to docker_compose_version_test()\"\n        exit 1\n    fi\n    if [ \"$version\" = \"latest\" ]; then\n        echo \"latest version, fetching latest version from DockerHub master branch\"\n        local version\n        version=\"$(dockerhub_latest_version \"$name\")\"\n        echo \"expecting version '$version'\"\n    fi\n    hr\n    found_version=\"$(docker_compose_path_version / \"$name\"-)\"\n    echo \"found $name version $found_version\"\n    hr\n    if [[ \"$found_version\" =~ $version ]]; then\n        echo \"$name docker version matches expected (found '$found_version', expected '$version')\"\n    else\n        echo \"Docker container version does not match expected version! (found '$found_version', expected '$version')\"\n        exit 1\n    fi\n}\n\ndocker_compose_port(){\n    local env_var=\"${1:-}\"\n    local name=\"${2:-}\"\n    if [ -z \"$env_var\" ]; then\n        echo \"ERROR: docker_compose_port() first arg \\$1 was not supplied for \\$env_var\"\n        exit 1\n    fi\n    if [ -z \"$name\" ]; then\n        name=\"$env_var\"\n    fi\n    name=\"$name port\"\n    if ! [[ \"$env_var\" =~ .*_PORT$ ]]; then\n        #env_var=\"$(tr '[[:lower:]]' '[[:upper:]]' <<< \"$env_var\")_PORT\"\n        # doesn't work on Mac\n        #env_var=\"$(sed 's/.*/\\U&/;s/[^[:alnum:]]/_/g' <<< \"$env_var\")_PORT\"\n        # shellcheck disable=SC2001\n        env_var=\"$(sed 's/[^[:alnum:]]/_/g' <<< \"$env_var\" | tr '[:lower:]' '[:upper:]')_PORT\"\n    fi\n    if [ -z \"${DOCKER_SERVICE:-}\" ]; then\n        echo \"ERROR: \\$DOCKER_SERVICE is not set, cannot run docker_compose_port()\"\n        exit 1\n    fi\n    set +u\n    if eval [ -z \\$\"${env_var}_DEFAULT\" ]; then\n        echo \"ERROR: ${env_var}_DEFAULT is not set, cannot run docker_compose_port()\"\n        exit 1\n    fi\n    set -u\n    # shellcheck disable=SC2006,SC2116\n    eval printf \"\\\"$name -> $`echo \"${env_var}_DEFAULT\"` => \\\"\"\n    # shellcheck disable=SC2006,SC2116,SC2046\n    export \"$env_var\"=\"$(eval docker-compose port \"$DOCKER_SERVICE\" $`echo \"${env_var}_DEFAULT\"` | sed 's/.*://')\"\n    if eval [ -z \\$\"$env_var\" ]; then\n        echo \"ERROR: failed to get port mapping for $env_var\"\n        exit 1\n    fi\n    if eval [ -z \\$\"$env_var\" ]; then\n        echo \"FAILED got no port mapping for $env_var... did the container crash?\"\n        exit 1\n    fi\n    if eval ! [[ \\$\"$env_var\" =~ ^[[:digit:]]+$ ]]; then\n        echo -n \"ERROR: failed to get port mapping for $env_var - non-numeric port '\"\n        eval echo -n \\$\"$env_var\"\n        echo \"' returned, possible parse error\"\n        exit 1\n    fi\n    eval echo \"\\$$env_var\"\n}\n\ndocker_exec(){\n    if [ -n \"${DOCKER_SKIP_EXEC:-}\" ]; then\n        echo \"skipping docker exec: $*\"\n        return 0\n    fi\n    local user=\"\"\n    if [ -n \"${DOCKER_USER:-}\" ]; then\n        user=\" --user $DOCKER_USER\"\n    fi\n    local MNTDIR=\"${DOCKER_MOUNT_DIR:-}\"\n    if [ -n \"$MNTDIR\" ]; then\n        MNTDIR=\"$MNTDIR/\"\n    fi\n    if [ -z \"${DOCKER_JAVA_HOME:-}\" ]; then\n        # shellcheck disable=SC2086,SC2145\n        run docker exec -i $user \"$DOCKER_CONTAINER\" \"$MNTDIR\"\"$@\"\n    else\n        local cmds=\"export JAVA_HOME=$DOCKER_JAVA_HOME\n$MNTDIR$*\"\n        echo  \"docker exec -i$user \\\"$DOCKER_CONTAINER\\\" /bin/bash <<EOF\n        $cmds\nEOF\"\n        # use run rather than run++ and plain docker exec so that it inherits ERRCODE\n        # shellcheck disable=SC2086\n        run docker exec -i $user \"$DOCKER_CONTAINER\" /bin/bash <<EOF\n        $cmds\nEOF\n    fi\n}\n\ndocker_compose_exec(){\n    local user=\"\"\n    if [ -n \"${DOCKER_USER:-}\" ]; then\n        user=\" --user $DOCKER_USER\"\n    fi\n    local MNTDIR=\"${DOCKER_MOUNT_DIR:-}\"\n    if [ -n \"$MNTDIR\" ]; then\n        MNTDIR=\"$MNTDIR/\"\n    fi\n    if [ -z \"${DOCKER_JAVA_HOME:-}\" ]; then\n        # shellcheck disable=SC2086\n        run docker-compose exec $user \"$DOCKER_SERVICE\" \"$MNTDIR$*\"\n    else\n        local cmds=\"export JAVA_HOME=$DOCKER_JAVA_HOME\n$MNTDIR$*\"\n        echo  \"docker-compose exec$user \\\"$DOCKER_SERVICE\\\" /bin/bash <<EOF\n        $cmds\nEOF\"\n        # shellcheck disable=SC2086\n        run docker-compose exec $user \"$DOCKER_SERVICE\" /bin/bash <<EOF\n        $cmds\nEOF\n    fi\n}\n\ndockerhub_latest_version(){\n    repo=\"${1-}\"\n    if [ -z \"$repo\" ]; then\n        echo \"Error: no repo passed to dockerhub_latest_version for first arg\"\n    fi\n    set +e\n    local version\n    version=\"$(curl -sS \"https://raw.githubusercontent.com/HariSekhon/Dockerfiles/master/$repo/Dockerfile\" | awk -F= '/^ARG[[:space:]]+[A-Za-z0-9_]+_VERSION=/ {print $2; exit}')\"\n    set -e\n    if [ -z \"$version\" ]; then\n        version='.*'\n    fi\n    echo \"$version\"\n}\n\ndocker_container_image(){\n    local container_name=\"$1\"\n    docker ps --filter \"name=^$container_name$\" --format '{{.Image}}'\n}\n\ndocker_container_exists(){\n    local container_name=\"$1\"\n    docker ps -a --filter \"name=^$container_name$\" --format '{{.Status}}' | grep -q .\n}\n\ndocker_container_exited(){\n    local container_name=\"$1\"\n    docker ps -a --filter \"name=^$container_name$\" --format '{{.Status}}' | grep -Eqi 'Dead|Exited'\n}\n\ndocker_container_not_running(){\n    local container_name=\"$1\"\n    docker ps -a --filter \"name=^$container_name$\" --format '{{.Status}}' | grep -Eqi 'Created|Paused|Dead|Exited'\n}\n\ndocker_pull(){\n    local docker_image=\"$1\"\n    local version\n    local opts\n    if is_CI or ! is_interactive; then\n        if [ \"$(docker version --format '{{.Client.Version}}' | grep -o '[[:digit:]]*' | head -n 1)\" -gt 18 ]; then\n            opts=\"-q\"\n        else\n            # Travis CI has Docker 18 without the -q / --quiet switch\n            opts=\"> /dev/null\"\n        fi\n    fi\n    if [[ \"$docker_image\" =~ : ]]; then\n        version=\"${docker_image#*:}\"\n        docker_image=\"${docker_image%:*}\"\n    else\n        local version=\"$2\"\n    fi\n    # don't pull if we already have the image locally - both to save time and remove internet dependency if running on a laptop\n    if ! docker images --filter reference=\"$docker_image:$version\" --format '{{.Repository}}:{{.Tag}}' | grep -q .; then\n        # want splitting\n        # shellcheck disable=SC2086\n        eval docker pull ${opts:-} \"$docker_image:$version\"\n    fi\n}\n\nexternal_docker(){\n    [ -n \"${EXTERNAL_DOCKER:-}\" ] && return 0 || return 1\n}\n\nlaunch_container(){\n    local image=\"${1:-${DOCKER_IMAGE}}\"\n    local container=\"${2:-${DOCKER_CONTAINER}}\"\n    local ports=\"${*:3}\"\n    if [ -n \"${TRAP:-}\" ] || is_CI; then\n        trap_container \"$container\"\n    fi\n    if external_docker; then\n        echo \"External Docker detected, skipping container creation...\"\n        return 0\n    else\n        [ -n \"${DOCKER_HOST:-}\" ] && echo \"using docker address '$DOCKER_HOST'\"\n        if ! is_docker_available; then\n            echo \"WARNING: Docker not found, cannot launch container $container\"\n            return 1\n        fi\n        # reuse container it's faster\n        #docker rm -f -- \"$container\" &>/dev/null\n        #sleep 1\n        if [[ \"$container\" = *test* ]]; then\n            docker rm -f -- \"$container\" &>/dev/null || :\n        fi\n        if ! is_docker_container_running \"$container\"; then\n            # This is just to quiet down the CI logs from useless download clutter as docker pull/run doesn't have a quiet switch as of 2016 Q3\n            if is_CI; then\n                # pipe to cat tells docker that stdout is not a tty, switches to non-interactive mode with less output\n                { docker pull \"$image\" || :; } | cat\n            fi\n            port_mappings=\"\"\n            for port in $ports; do\n                port_mappings=\"$port_mappings -p $port:$port\"\n            done\n            echo -n \"starting container: \"\n            # need tty for sudo which Apache startup scripts use while SSH'ing localhost\n            # eg. hadoop-start.sh, hbase-start.sh, mesos-start.sh, spark-start.sh, tachyon-start.sh, alluxio-start.sh\n            # shellcheck disable=SC2086\n            docker run -d -t --name \"$container\" ${DOCKER_OPTS:-} $port_mappings \"$image\" ${DOCKER_CMD:-}\n            hr\n            echo \"Running containers:\"\n            docker ps\n            hr\n            #echo \"waiting $startupwait seconds for container to fully initialize...\"\n            #sleep $startupwait\n        else\n            echo \"Docker container '$container' already running\"\n        fi\n    fi\n    if [ -n \"${ENTER:-}\" ]; then\n        docker exec -ti \"$DOCKER_CONTAINER\" bash\n    fi\n}\n\ndelete_container(){\n    local container=\"${1:-$DOCKER_CONTAINER}\"\n    local msg=\"${2:-}\"\n    echo\n    if [ -z \"${NODELETE:-}\" ] && ! external_docker; then\n        if [ -n \"$msg\" ]; then\n            echo \"$msg\"\n        fi\n        echo -n \"Deleting container \"\n        docker rm -f -- \"$container\"\n        untrap\n    fi\n}\n\ntrap_container(){\n    local container=\"${1:-$DOCKER_CONTAINER}\"\n    # shellcheck disable=SC2154,SC2086\n    trap 'result=$?; '\"delete_container $container 'trapped exit, cleaning up container'\"' || : ; exit $result' $TRAP_SIGNALS\n}\n\ndocker_rmi_dangling_layers(){\n    # want splitting\n    # shellcheck disable=SC2046\n    docker rmi $(docker images -q --filter dangling=true) 2>/dev/null || :\n}\n\ndocker_rmi_grep(){\n    docker images |\n    grep -Ei -- \"^$1\" |\n    awk '{print $1\":\"$2}' |\n    xargs docker rmi --force || :\n}\n\n# to be called at end of scripts as well as trap function\ndocker_image_cleanup(){\n    local docker_images\n    if [ -n \"${COMPOSE_FILE:-}\" ] && [ -f \"$COMPOSE_FILE\" ]; then\n        docker_images=\"$(grep '^[[:space:]]*image' \"$COMPOSE_FILE\" | sed 's/.*image:[[:space:]]*//; s/:.*//')\"\n    fi\n    if [ -n \"${docker_images:-}\" ]; then\n        for docker_image in $docker_images; do\n            docker_rmi_grep \"$docker_image\" || :\n        done\n    fi\n    docker_rmi_dangling_layers\n}\n"
  },
  {
    "path": "lib/dockerfile_keywords.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Mon Nov 2 17:26:28 2015 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et:filetype=conf\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                     D o c k e r f i l e   K e y w o r d s\n# ============================================================================ #\n\n# Copied from https://github.com/HariSekhon/SQL-keywords\n\nFROM\nMAINTAINER\nRUN\nCMD\nLABEL\nEXPOSE\nENV\nARG\nADD\nCOPY\nENTRYPOINT\nVOLUME\nUSER\nWORKDIR\nONBUILD\nSTOPSIGNAL\nHEALTHCHECK\nSHELL\n"
  },
  {
    "path": "lib/excluded.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# intended only to be sourced by lib/utils.sh\n#\n# split from lib/utils.sh as this is specific to this repo\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif type isExcluded &>/dev/null; then\n    return 0\nfi\n\nif [ -n \"${BASH_EXCLUDE_FILES_FUNCTION:-}\" ] &&\n   [ -f \"$BASH_EXCLUDE_FILES_FUNCTION\" ] &&\n   [[ \"$BASH_EXCLUDE_FILES_FUNCTION\" =~ \\.sh$ ]] &&\n   grep -q 'isExcluded()' \"$BASH_EXCLUDE_FILES_FUNCTION\"; then\n    # shellcheck disable=SC1090\n    . \"$BASH_EXCLUDE_FILES_FUNCTION\"\n    return 0\nfi\n\nisExcluded(){\n    local prog=\"$1\"\n    # this really is anything beginning with a star\n    # shellcheck disable=SC2049\n    [[ \"$prog\" =~ ^\\* ]] && return 0\n    [[ \"$prog\" =~ ^# ]]  && return 0\n    #[[ \"$prog\" =~ /\\. ]] && return 0\n    [[ \"$prog\" =~ /\\.git/ ]] && return 0\n    [[ \"$prog\" =~ ^\\.git/ ]] && return 0\n    [[ \"$prog\" =~ \\.external_modules/ ]] && return 0\n    [[ \"$prog\" =~ /templates/ ]] && return 0\n    [[ \"$prog\" =~ TODO ]] && return 0\n    [[ \"$prog\" =~ /inc/Module/.*\\.pm ]] && return 0\n    # imported, minimal editing restricted to essentials only\n    [[ \"$prog\" =~ getawless.sh ]] && return 0\n    if [ -n \"${EXCLUDE_FILES_REGEX:-}\" ]; then\n        if [[ \"$prog\" =~ $EXCLUDE_FILES_REGEX ]]; then\n            return 0\n        fi\n    fi\n    # optimization to avoid fork\n    #if ! uname -s | grep -q Darwin; then\n    if ! [[ \"${OSTYPE:-}\" == darwin* ]]; then\n        if [[ \"$prog\" =~ /applescript/ ]]; then\n            return 0\n        fi\n    fi\n    #if is_CI &&\n    #   git rev-parse --is-inside-work-tree &>/dev/null; then\n    #    echo \"Running in CI and within Git repo checkout, skipping submodules\"\n    #    # sufficient as long as no submodule dirs have spaces in them\n    #    #submodules=\"$(git submodule | awk '{print $2}')\"\n    #    submodules=\"$(git submodule | cut -c 43- | cut -f 1 -d'(')\"\n    #    for submodule_dir in $submodules; do\n    #        if [[ \"$prog\" =~ /$submodule_dir/ ]]; then\n    #            return 0\n    #        fi\n    #    done\n    #fi\n    # this external git check is expensive, skip it when in CI as using fresh git checkouts\n    #is_CI && return 1\n    ## shellcheck disable=SC2230\n    #if type -P git &>/dev/null; then\n    #    commit=\"$(git log \"$prog\" 2>/dev/null | head -n1 | grep 'commit' || :)\"\n    #    if [ -z \"$commit\" ]; then\n    #        return 0\n    #    fi\n    #fi\n    return 1\n}\n"
  },
  {
    "path": "lib/gcp.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC1090,SC2034\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-13 19:38:39 +0100 (Thu, 13 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\ngcp_lib_srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$gcp_lib_srcdir/utils.sh\"\n\ngcp_info_noninteractive_help=\"You should only this script non-interactively / in pipes once you have tested it locally on the command line because some services may prompt you for choices, eg. Cloud Run, which you should save to your GCloud SDK config settings first\"\n\ngcp_info_formatting_help=\"In interactive mode, prints fancy boxes around GCP info to be easier on the eye. In non-interactive mode it skips formatting so it's easier to parse or process with other commands like grep / awk etc. Non-interactive mode happens automatically when the output is piped or redirected to a file or another command, or when \\$NO_FORMATTING=1 is set in environment\"\n\nusage_gcloud_sdk_required=\"GCloud SDK must be installed and configured\"\n\nif is_piped || [ -n \"${NO_FORMATTING:-}\" ]; then\n    gcloud_formatting=''\nelse\n    # want deferred expansion\n    # shellcheck disable=SC2016\n    gcloud_formatting='\"[box,title=\\\"$title\\\"]\"'\nfi\n\n# not anchoring here any more so that we can use these to compose more complex regex - client should anchor regex in matches\ngcr_image_regex='([^\\.]+\\.)?gcr\\.io/[^/]+/[^:]+'\ngcr_image_tag_regex=\"$gcr_image_regex:.+\"\ngcr_image_optional_tag_regex=\"$gcr_image_regex(:.+)?\"\n\ngcp_info(){\n    local title=\"$1\"\n    shift || :\n    if [ -z \"$gcloud_formatting\" ]; then\n        #perl -e \"print '=' x (${#title} + 1);\"\n        for ((i=0; i <= ${#title}; i++)); do\n            #printf '='\n            # built-in - not as portable eg. sh, but given we explicitly execute in bash should be ok\n            echo -n '='\n        done\n        echo\n        echo \"$title:\"\n    fi\n    if [[ \"$*\" =~ --format ]]; then\n        # eval formatting for table only to get late evaluated $title\n        \"${@//--format=table\\(/--format=table$(eval echo \"$gcloud_formatting\")\\(}\"\n    else\n        # formatting has to be eval'd in order to pick up latest $title as a late binding\n        # better than eval'ing the entire command line to evaluate $title in the formatting string interpolation\n        \"$@\" --format=\"$(eval echo \"$gcloud_formatting\")\" || return\n        if [ -z \"$gcloud_formatting\" ]; then\n            echo\n        fi\n    fi\n}\n\n# avoid race conditions on changing the configuration\n# (it's still possible to change the settings within the configuration, use CLOUDSDK_CORE_PROJECT and similar overrides on an as needed basis)\ngcloud_export_active_configuration(){\n    local active_configuration\n    active_configuration=\"$(gcloud config configurations list --format='get(name)' --filter='is_active = True')\"\n    export CLOUDSDK_ACTIVE_CONFIG_NAME=\"$active_configuration\"\n}\n\ngcp_serviceaccount_exists(){\n    local service_account=\"$1\"\n    gcloud iam service-accounts list --format='get(email)' --filter=\"email:$service_account\" |\n    grep -Fxq \"$service_account\"\n}\n\ngcp_create_serviceaccount_if_not_exists(){\n    local name=\"$1\"\n    local project=\"$2\"\n    local description=\"${3:-}\"\n    local service_account=\"$name@$project.iam.gserviceaccount.com\"\n\n    if gcp_serviceaccount_exists \"$service_account\"; then\n        echo \"Service account '$service_account' already exists\" >&2\n    else\n        gcloud iam service-accounts create \"$name\" --description=\"$description\" --project \"$project\"\n    fi\n}\n\ngcp_create_credential_if_not_exists(){\n    local serviceaccount=\"$1\"\n    local keyfile=\"$2\"\n    mkdir -pv \"$(dirname \"$keyfile\")\"\n    if [ -f \"$keyfile\" ]; then\n        echo \"Credentials keyfile '$keyfile' already exists\" >&2\n    else\n        gcloud iam service-accounts keys create \"$keyfile\" --iam-account=\"$service_account\" --key-file-type=\"json\"\n    fi\n}\n\n# necessary so you can log in to different projects and maintain IAM permissions isolation for safety\n# do not use the same serviceaccount with permissions across projects, you can cross contaminate and make mistakes, deploy the wrong environment etc.\ngcp_login(){\n    local credentials_json=\"${1:-}\"\n    if [ -n \"$credentials_json\" ] &&\n       [ -f \"$credentials_json\" ]; then\n        gcloud auth activate-service-account --key-file=\"$credentials_json\"\n    elif [ -n \"${GCP_SERVICEACCOUNT_KEY:-}\" ]; then\n        # XXX: it's hard to copy the contents of this around so it's easiest to do via:\n        #\n        #   base64 credentials.json | pbcopy\n        #\n        # and then paste that into the CI/CD environment variables for the build\n        #\n        gcloud auth activate-service-account --key-file=<(base64 --decode <<< \"$GCP_SERVICEACCOUNT_KEY\")\n    else\n        die \"no credentials.json file passed to gcp_login() and \\$GCP_SERVICEACCOUNT_KEY not set in environment\"\n    fi\n}\n\ngke_login(){\n    local cluster_name=\"$1\"\n    # if running the CI build on the same k8s cluster as the deployment will go to - this is often not the case and not reliable to be detected either since we are often running these builds inside docker images and it would rely on correctly configuring the environment variables to be able to detect this. Instead just open the GKE's cluster's master networks to the projects external NAT IP\n    #local opts=(--internal-ip)\n    local opts=()\n    gcloud container clusters get-credentials \"$cluster_name\" \"${opts[@]}\"\n}\n\nenable_kaniko(){\n    #gcloud config set builds/use_kaniko True\n    export CLOUDSDK_BUILDS_USE_KANIKO=True\n}\n"
  },
  {
    "path": "lib/gcp_ci.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-02 18:59:07 +0000 (Tue, 02 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                    G C P   C I   S h a r e d   L i b a r y\n# ============================================================================ #\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# shellcheck disable=SC1090\n. \"$srcdir/gcp.sh\"\n\n# ============================================================================ #\n#       J e n k i n s   /   T e a m C i t y   B r a n c h   +   B u i l d\n# ============================================================================ #\n\n# Jenkins provides $BRANCH_NAME only in MultiBranch builds, otherwise use $GIT_BRANCH\n# TeamCity doesn't provide this so will fall back to git rev-parse\nif [ -z \"${BRANCH_NAME:-}\" ]; then\n    BRANCH_NAME=\"${GIT_BRANCH:-$(git rev-parse --abbrev-ref HEAD)}\"\nfi\nBRANCH_NAME=\"${BRANCH_NAME##*/}\"\n\n# Jenkins provides $GIT_COMMIT, TeamCity provides $BUILD_VCS_NUMBER\nBUILD=\"${GIT_COMMIT:-${BUILD_VCS_NUMBER:-$(git show-ref --hash HEAD)}}\"\n\n# ============================================================================ #\n#                    G C P   P r o j e c t   +   R e g i o n\n# ============================================================================ #\n\n# If Project isn't set in the CI/CD environment (safest way as it doesn't have a race condition with global on-disk gcloud config), then infer it\nset_gcp_project(){\n    # in Jenkins the branch is prefixed with origin/\n    if [ -z \"${CLOUDSDK_CORE_PROJECT:-}\" ]; then\n        if [[ \"$BRANCH_NAME\" =~ ^(dev|staging)$ ]]; then\n            export CLOUDSDK_CORE_PROJECT=\"$MYPROJECT-$BRANCH_NAME\"\n        elif [ \"$BRANCH_NAME\" = production ]; then\n            # production project has a non-uniform project id, so override it\n            export CLOUDSDK_CORE_PROJECT=\"$MYPROJECT-prod\"\n        else\n            # assume it is a feature branch being deployed to the Dev project\n            export CLOUDSDK_CORE_PROJECT=\"$MYPROJECT-dev\"\n        fi\n    fi\n}\n\nset_gcp_compute_region(){\n    local region=\"${1:-europe-west1}\"\n    if [ -z \"${CLOUDSDK_COMPUTE_REGION:-}\" ]; then\n        # Set default region where you GKE cluster is going to be\n        export CLOUDSDK_COMPUTE_REGION=\"$region\"\n        # Do an override if you have a project that is in a different region to the rest\n    #    if [ \"$CLOUDSDK_CORE_PROJECT\" = \"$MYPROJECT-staging\" ]; then\n    #        # Staging's location is different\n    #        export CLOUDSDK_COMPUTE_REGION=\"europe-west4\"\n    #    fi\n    fi\n}\n\n# ============================================================================ #\n#           Print the Environment in every build for easier debugging\n# ============================================================================ #\n\nprintenv(){\n    echo \"Environment:\"\n    env | sort\n}\n\n# ============================================================================ #\n#                               F u n c t i o n s\n# ============================================================================ #\n\nlist_container_tags(){\n    local image=\"$1\"\n    local build=\"$2\"  # Git hashref that triggered CI build\n    # --format=text returns blank if no match tag for the docker image exists, which is convenient for testing such tags_exist_for_container_image() below\n    gcloud container images list-tags --filter=\"tags:${build}\" --format=text \"gcr.io/$CLOUDSDK_CORE_PROJECT/$image\"\n    # will get this error if you try running this is via DooD, switching to normal K8s pod template in pipeline solves this:\n    # ERROR: gcloud crashed (MetadataServerException): HTTP Error 404: Not Found\n}\n\ntags_exist_for_container_image(){\n    # since list_container_tags returns blank if this build hashref doesn't exist, we can use this as a simple test\n    [ -n \"$(list_container_tags \"$APP\" \"$BUILD\")\" ]\n}\n\ngcloud_builds_submit(){\n    local build=\"$1\"\n    local cloudbuild_yaml=\"${2:-cloudbuild.yaml}\"\n    gcloud builds submit --project \"$CLOUDSDK_CORE_PROJECT\" --config \"$cloudbuild_yaml\" --substitutions _REGISTRY=\"gcr.io/$CLOUDSDK_CORE_PROJECT,_BUILD=$build\" --timeout=3600\n    # will get this error if you try running this is via DooD, switching to normal K8s pod template in pipeline solves this:\n    # ERROR: gcloud crashed (MetadataServerException): HTTP Error 404: Not Found\n}\n\n# yamls contain tag 'latest', we replace this with the build hashref matching the docker images just built as part of the build pipeline\n# Better is to use CI/CD to update the kustomization.yaml with the hashref as part of a GitOps workflow - see my Jenkins shared library https://github.com/HariSekhon/Jenkins/tree/master/vars\nreplace_latest_with_build(){\n    local build=\"$1\"\n    sed -i \"s/\\\\([[:space:]]newTag:[[:space:]]*\\\"*\\\\)latest/\\\\1$build/g\" -- kustomization.yaml\n    sed -i \"s/commit=latest/commit=$build/g\" -- kustomization.yaml\n}\n\ndownload_kustomize(){\n    #curl -s \"https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh\"  | bash\n    # better to fix version in case later versions change behaviour or syntax\n    curl -o kustomize --location https://github.com/kubernetes-sigs/kustomize/releases/download/v3.1.0/kustomize_3.1.0_linux_amd64\n    chmod u+x ./kustomize\n}\n\nkubernetes_deploy(){\n    local app=\"$1\"\n    local namespace=\"$2\"\n    kubectl apply -f .\n    kubectl rollout status \"deployment/$app\" -n \"$namespace\" --timeout=120s\n}\n\n# old, use ArgoCD instead now, see Kubernetes repo:\n#\n#   https://github.com/HariSekhon/Kubernetes-configs\n#\nkustomize_kubernetes_deploy(){\n    local app=\"$1\"\n    local namespace=\"$2\"\n    # append to PATH to be able to find just downloaded ./kustomize\n    PATH=\"$PATH:.\" kubectl_create_namespaces.sh\n    # XXX: DANGER: this would replace $RABBITMQ_HOME - needs more testing to support 'feature staging' / 'feature dev' - but envsubst doesn't support default values\n    #PATH=\"$PATH:.\" kustomize build . | envsubst | kubectl apply -f -\n    PATH=\"$PATH:.\" kustomize build . | kubectl apply -f -\n    kubectl annotate \"deployment/$app\" -n \"$namespace\" kubernetes.io/change-cause=\"$(date '+%F %H:%M')  CI deployment: app $app build ${BUILD:-}\"\n    local deployments\n    deployments=\"$(kubectl get deploy,sts -n \"$namespace\" --output name)\"\n    # $deployment contains deployment.apps/ or statefulset.apps/ prefixes\n    trap 'echo \"ERROR: kubernetes $namespace $deployment is BROKEN - possible misconfiguration or bad code is preventing pods from coming up after a reasonable timeframe of retries, please see GKE container logs\" >&2' EXIT\n    # using a global shared timeout rather than a --timeout=\"${timeout}s\" for each kubectl rollout as that multiplies by the amount of deployments and statefulsets which should have been working\n    TMOUT=600\n    for deployment in $deployments; do\n        kubectl rollout status \"$deployment\" -n \"$namespace\"\n    done\n    trap '' EXIT\n\n    # could also run the deployment via Google Cloud Build\n    #gcloud builds submit --project \"$CLOUDSDK_CORE_PROJECT\" --config=\"../../cloudbuild-deploy.yaml\" --no-source\n}\n"
  },
  {
    "path": "lib/git.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2017-11-21 10:45:41 +0100 (Tue, 21 Nov 2017)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Export of useful Git utility functions from years gone by\n#\n# far more git functions are available in the interactive library .bash.d/git.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_git=\"${srcdir:-}\"\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$srcdir/utils.sh\"\n\nis_git_repo(){\n    local target=${1:-.};\n    if [ -d \"$target/.git\" ]; then\n        return 0;\n    else\n        if [ -f \"$target\" ] && [ -d \"${target%/*}/.git\" ]; then\n            return 0;\n        else\n            if [ -d \"$target\" ]; then\n                pushd \"$target\" > /dev/null || return 1;\n                if git status >&/dev/null; then\n                    popd >&/dev/null;\n                    return 0;\n                fi;\n            else\n                pushd \"$(dirname \"$target\")\" > /dev/null || return 1;\n                if git status >&/dev/null; then\n                    popd >&/dev/null;\n                    return 0;\n                fi;\n            fi;\n            popd >&/dev/null;\n            return 2;\n        fi;\n    fi\n}\n\ngit_repo(){\n    # give preference for origin, then GitHub, GitLab, Bitbucket, Azure DevOps in that order\n    local remotes\n    remotes=\"$( git remote -v 2>/dev/null)\"\n    {\n        awk 'BEGIN {IGNORECASE=1} $1 ~ /origin/ {print}' <<< \"$remotes\"\n        awk 'BEGIN {IGNORECASE=1} $1 ~ /github/ {print}' <<< \"$remotes\"\n        awk 'BEGIN {IGNORECASE=1} $1 ~ /gitlab/ {print}' <<< \"$remotes\"\n        awk 'BEGIN {IGNORECASE=1} $1 ~ /bitbucket/ {print}' <<< \"$remotes\"\n        awk 'BEGIN {IGNORECASE=1} $1 ~ /azure/ {print}' <<< \"$remotes\"\n        echo \"$remotes\"\n    } |\n    awk '{print $2}' |\n    head -n1 |\n    sed '\n        s,.*://,,;\n        s/.*@//;\n        s/[^:/]*[:/]//;\n        s/\\.git$//;\n        s|^/||;\n        s|/[^/]*/_git/|/|;\n    '\n}\n\ngit_repo_name(){\n    git_repo |\n    sed 's|.*/||'\n}\n\ngit_repo_name_lowercase(){\n    git_repo_name |\n    tr '[:upper:]' '[:lower:]'\n}\n\ngit_root(){\n    git rev-parse --show-toplevel\n}\n\ngit_root_basedir(){\n    local git_root\n    git_root=\"$(git_root)\"\n    echo \"${git_root##*/}\"\n}\n\ngit_relative_dir(){\n    if ! is_in_git_repo; then\n        echo \"Error: not in a git repo when trying to determine git_relative_dir()\" >&2\n    fi\n    local git_root\n    local git_root_basedir\n    git_root=\"$(git_root)\"\n    git_root_basedir=\"$(git_root_basedir)\"\n    # false positive - this works in Bash (tested on Mac)\n    # shellcheck disable=SC2295\n    echo \"$git_root_basedir/${PWD##$git_root/}\"\n}\n\nis_in_git_repo(){\n    #git_root &>/dev/null\n    git rev-parse --is-inside-work-tree &>/dev/null\n}\n\nis_file_tracked_in_git(){\n    local filename=\"$1\"\n    git ls-files --error-unmatch -- \"$filename\" &>/dev/null\n}\n\ngit_commit_short_sha(){\n    git rev-parse --short HEAD\n}\n\ncurrent_branch(){\n    git rev-parse --abbrev-ref HEAD\n}\n\ndefault_branch(){\n    git remote show origin |\n    sed -n '/HEAD branch/ s/.\\+:[[:space:]]\\+//p'\n}\n\nallbranches(){\n    if type -P uniq_order_preserved.pl &>/dev/null; then\n        local uniq=uniq_order_preserved.pl\n    else\n        local uniq=\"sort | uniq\"\n    fi\n    # this only shows local branches, to show all remote ones do\n    # git ls-remote | awk '/\\/heads\\//{print $2}' | sed 's,refs/heads/,,'\n    # shellcheck disable=SC2086\n    git branch -a | clean_branch_name | eval $uniq\n}\n\nclean_branch_name(){\n    sed '\n        s/^\\* // ;\n        s/.*\\/// ;\n        s/^[[:space:]]*// ;\n        s/[[:space:]]*$// ;\n        s/.*[[:space:]]// ;\n        s/)[[:space:]]*//\n    '\n}\n\nforeachbranch(){\n    local start_branch\n    start_branch=$(git branch | grep '^\\*' | clean_branch_name);\n    local branches\n    branches=\"$(allbranches)\";\n    if [ \"$start_branch\" != \"master\" ]; then\n        branches=\"$(sed \"1,/$start_branch/d\" <<< \"$branches\")\";\n    fi;\n    local branch;\n    for branch in $branches; do\n        hr\n        if [ -z \"${FORCEMASTER:-}\" ] && [ \"$branch\" = \"master\" ]; then\n            echo \"skipping master branch for safety (set FORCEMASTER=1 environment variable to override)\"\n            continue\n        fi\n        if [ -n \"${BRANCH_FILTER:-}\" ] && ! grep -E \"$BRANCH_FILTER\" <<< \"$branch\"; then\n            continue\n        fi\n        echo \"$branch:\"\n        # shellcheck disable=SC2294\n        if git branch | grep -Fq --color=auto \"$branch\"; then\n            git checkout \"$branch\"\n        else\n            git checkout --track \"origin/$branch\";\n        fi &&\n        eval \"$@\" || return\n        echo\n    done\n    git checkout \"$start_branch\"\n}\n\nmybranch(){\n    #git branch | awk '/^\\*/ {print $2; exit}'\n    git rev-parse --abbrev-ref HEAD\n}\n\n# shouldn't need to use this any more, git_check_branches_upstream.py from DevOps Python Tools repo has a --fix flag which will do this for all branches if they have no upstream set - https://github.com/HariSekhon/DevOps-Python-tools\nset_upstream(){\n    git branch --set-upstream-to \"origin/$(mybranch)\" \"$(mybranch)\"\n}\n\n# load domain, user and token variables from environment\n# don't export variables, they are used as globals in calling script but shouldn't be visible to child processes\n# shellcheck disable=SC2034\ngit_provider_env(){\n    local name=\"$1\"\n    if [ \"$name\" = \"github\" ]; then\n        domain=github.com\n        user=\"${GITHUB_USERNAME:-${GITHUB_USER:-}}\"\n        token=\"${GITHUB_TOKEN:-${GITHUB_PASSWORD:-}}\"\n    elif [ \"$name\" = \"gitlab\" ]; then\n        domain=gitlab.com\n        user=\"${GITLAB_USERNAME:-${GITLAB_USER:-}}\"\n        token=\"${GITLAB_TOKEN:-${GITLAB_PASSWORD:-}}\"\n    elif [ \"$name\" = \"bitbucket\" ]; then\n        domain=bitbucket.org\n        user=\"${BITBUCKET_USERNAME:-${BITBUCKET_USER:-}}\"\n        token=\"${BITBUCKET_APP_PASSWORD:-${BITBUCKET_TOKEN:-${BITBUCKET_PASSWORD:-}}}\"\n    elif [ \"$name\" = \"azure\" ]; then\n        domain=dev.azure.com\n        user=\"${AZURE_DEVOPS_USERNAME:-${AZURE_DEVOPS_USER:-}}\"\n        token=\"${AZURE_DEVOPS_TOKEN:-${AZURE_DEVOPS_PASSWORD:-}}\"\n    fi\n}\n\n# Azure DevOps has non-uniform URLs compared to the 3 main Git repos so here are general conversion rules used by git_remotes_add_origin_providers.sh / git_remotes_set_multi_origin.sh\ngit_to_azure_url(){\n    local url=\"$1\"\n    # XXX: you should set $AZURE_DEVOPS_PROJECT in your environment or call your project GitHub as I have - there is no portable way to infer this from other repos since they don't have this hierarchy level - querying the API might work if there is only a single project, but this might get overly complicated if requiring additional authentication\n    project=\"${AZURE_DEVOPS_PROJECT:-}\"\n    if [ -z \"$project\" ]; then\n        timestamp \"WARNING: \\$AZURE_DEVOPS_PROJECT not set, defaulting to project name 'GitHub'\"\n        project=\"GitHub\"\n    fi\n    url=\"${url/git@dev.azure.com/git@ssh.dev.azure.com}\"\n    url=\"${url%.git}\"\n    # don't match on ssh.dev.azure.com because ssh. prefix might not be stripped before this point, eg. from git_remotes_set_ssh_to_https.sh\n    if [[ \"$url\" =~ git@|ssh:// ]]; then\n        url=\"${url/\\/_git\\//\\/}\"\n        if ! [[ \"$url\" =~ v3/ ]]; then\n            if [[ \"$url\" =~ ^ssh:// ]]; then\n                # add v3/ if not in URL already\n                url=\"$(perl -pn -e 's/^(ssh:\\/\\/[^\\/]+)\\/(?!v3\\/)/$1\\/v3\\//' <<< \"$url\")\"\n            else\n                url=\"$(perl -pn -e 's/:\\/?/:v3\\//' <<< \"$url\")\"\n            fi\n        fi\n        # if 4 sections then it's already in Azure format of [:/]v3/username/project/repo\n        # - in that case just lowercase the username\n        # else\n        # - otherwise also inject the project just before repo name to conform to weird Azure DevOps urls\n        if grep -Eq '[:/][^./]+/[^/]+/[^/]+/[^/]+$' <<< \"$url\"; then\n            url=\"$(perl -pe \"s/(\\\\/[^\\\\/]+)(\\\\/[^\\\\/]+\\\\/[^\\\\/]+)$/\\\\L\\$1\\\\E\\$2/\" <<< \"$url\")\"\n        else\n            url=\"$(perl -pe \"s/(\\\\/[^\\\\/]+)(\\\\/[^\\\\/]+)$/\\\\L\\$1\\\\E\\\\/$project\\$2/\" <<< \"$url\")\"\n        fi\n    else # https\n        url=\"${url/ssh.dev.azure.com/dev.azure.com}\"\n        url=\"${url/\\/v3\\//\\/}\"\n        url=\"${url/:v3\\//\\/}\"\n        if ! [[ \"$url\" =~ /_git/ ]]; then\n            url=\"$(perl -pe 's/(\\/[^\\/]+)$/\\/_git$1/' <<< \"$url\")\"\n        fi\n        # match exactly\n        # shellcheck disable=SC2076\n        if ! [[ \"$url\" =~ \"/$project/\" ]]; then\n            url=\"$(perl -pe \"s/\\\\/_git\\\\//\\\\/$project\\\\/_git\\\\//\" <<< \"$url\")\"\n        fi\n    fi\n    # duplicate slashes break Azure DevOps URLs but resist the urge for simple fix replacing // with / as this would break ssh:// and https://\n    echo \"$url\"\n}\n\nazure_to_git_url(){\n    local url=\"$1\"\n    url=\"${url/:v3\\//:}\"\n    #url=\"${url/\\/_git\\//\\/}\"\n    # XXX: strip the middle component out from Azure URLs that aren't found in other major Git providers like GitHub / GitLab / Bitbucket:\n    #\n    #   git@ssh.dev.azure.com:v3/harisekhon/GitHub/DevOps-Bash-tools\n    #   https://dev.azure.com/harisekhon/GitHub/_git/DevOps-Bash-tools\n    #\n    url=\"$(perl -pe 's/([\\/:][^\\/:]+)\\/[^\\/]+\\/_git(\\/[^\\/]+)$/$1$2/' <<< \"$url\")\"\n    echo \"$url\"\n}\n\nsrcdir=\"$srcdir_git\"\n"
  },
  {
    "path": "lib/github.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo user={user} name={name} repo={repo}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-30 10:08:07 +0100 (Sun, 30 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_github_lib=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir_github_lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir_github_lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_github_cli_required=\"Requires GitHub CLI to be installed and configured, as well as jq\"\n\n# ERE format regex\ngithub_pull_request_url_regex='https://github.com/[[:alnum:]_-]+/[[:alnum:]_-]+/pull/[[:digit:]]+'\n# shellcheck disable=SC2034\ngithub_release_url_regex='https://github.com/[[:alnum:]_-]+/[[:alnum:]_-]+/releases/download/[^/]+/'\n\nget_github_repo(){\n    git remote -v 2>/dev/null |\n    grep -E \"github\\.[[:alnum:].-]+[/:]\" |\n    awk '{print $2}' |\n    head -n1 |\n    sed '\n        s,.*://,,;\n        s/.*@//;\n        s/[^:/]*[:/]//;\n        s/\\.git\\/*$//;\n        s|^/||;\n    '\n}\n\nis_github_origin(){\n    git remote -v |\n    # permitting generic domain regex for those self-hosting their own github enterprise servers\n    grep -Eq \"^origin[[:space:]].+github\\.[[:alnum:].-]+[/:]\"\n}\n\nis_github_fork(){\n    local is_fork\n    is_fork=\"$(gh repo view --json isFork -q '.isFork')\"\n    if [ \"$is_fork\" = true ]; then\n        return 0\n    fi\n    return 1\n}\n\ngithub_owner_repo(){\n    local owner_repo\n    owner_repo=\"$(gh repo view --json owner,name -q '.owner.login + \"/\" + .name')\"\n    if ! is_github_owner_repo \"$owner_repo\"; then\n        echo \"GitHub owner/repo '$owner_repo' does not match expected format returned in github_owner_repo()\" >&2\n        return 1\n    fi\n    echo \"$owner_repo\"\n}\n\ncheck_github_origin(){\n    if ! is_github_origin; then\n        die 'GitHub is not set as remote origin in current repo!'\n    fi\n}\n\ngithub_origin_owner_repo(){\n    local owner_repo\n    owner_repo=\"$(\n        git remote -v |\n        grep -Em1 '^origin.*github\\.[[:alnum:].-]+[/:]' |\n        sed '\n            s|.*github\\.[[:alnum:].-]*[:/]||;\n            s/\\.git.*//;\n            s/[[:space:]].*//\n        ' ||\n        :\n    )\"\n    if ! is_github_owner_repo \"$owner_repo\"; then\n        echo \"GitHub owner/repo '$owner_repo' does not match expected format returned in github_origin_owner_repo()\" >&2\n        return 1\n    fi\n    echo \"$owner_repo\"\n}\n\ngithub_repo_set_default(){\n    # this command has poor behaviour - returns exit code 0 whether set or not, and the error string is sent to stdout instead of stderr, although it returns blank when in a pipe\n    #\n    #   https://github.com/cli/cli/issues/9398\n    #\n    # returns blank when in a pipe\n    if ! gh repo set-default --view | grep -q '.'; then\n        local origin\n        origin=\"$(git remote -v | awk '/^origin[[:space:]]/{print $2; exit}')\"\n        if [ -n \"$origin\" ]; then\n            timestamp \"GitHub CLI setting repo default to '$origin'\"\n            gh repo set-default \"$origin\"\n        else\n            gh repo set-default --view >&2\n            echo \"GitHub CLI default repo not set\" >&2\n            return 1\n        fi\n    fi\n}\n\ngithub_upstream_owner_repo(){\n    local owner_repo\n    github_repo_set_default\n    owner_repo=\"$(gh repo view --json parent | jq -r '.parent | .owner.login + \"/\" + .name')\"\n    if [ \"$owner_repo\" = / ]; then\n        echo \"Failed to determine upstream owner/repo\" >&2\n        return 1\n    elif ! is_github_owner_repo \"$owner_repo\"; then\n        echo \"GitHub upstream owner/repo '$owner_repo' does not match expected format\" >&2\n        return 1\n    fi\n    echo \"$owner_repo\"\n}\n\nis_github_owner_repo(){\n    local repo=\"$1\"\n    # .github repo is valid\n    [[ \"$repo\" =~ ^[[:alnum:]-]+/[[:alnum:]._-]+$ ]]\n}\n\nget_github_user(){\n    if [ -n \"${GITHUB_USER:-}\" ]; then\n        echo \"$GITHUB_USER\"\n    else\n        # get currently authenticated user\n        \"$srcdir_github_lib/../github/github_api.sh\" /user | jq -r .login\n    fi\n}\n\ngithub_result_has_more_pages(){\n    local output=\"$1\"\n    if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n        return 1\n    elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n        exit 1\n    fi\n    return 0\n}\n\n# pass owner as first arg and then any other --options\nget_github_repo_urls(){\n    local owner=\"${1:-}\"\n    gh repo list \"$@\" \\\n        --limit 9999999 \\\n        --json url \\\n        --jq '.[].url'\n        # pass these from the client to retain flexibility\n        #--visibility public \\\n        #--source \\\n}\n\nget_github_repos(){\n    local owner=\"${1:-}\"\n    if [ -z \"$owner\" ]; then\n        owner=\"$(get_github_user)\"\n    fi\n    local is_org=\"${2:-}\"\n    local filter=\"${3:-.}\"\n    local prefix\n    if [ -n \"$is_org\" ]; then\n        prefix=\"orgs\"\n    else\n        prefix=\"users\"\n    fi\n    local page=1\n    while true; do\n        if ! output=\"$(\"$srcdir_github_lib/../github/github_api.sh\" \"/$prefix/$owner/repos?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq_debug_pipe_dump <<< \"$output\" |\n        jq -r \".[] | select(.fork | not) | $filter | .name\"\n        ((page+=1))\n    done\n}\n\nget_github_repo_branches(){\n    local repo=\"$1\"\n    local page=1\n    while true; do\n        if ! output=\"$(\"$srcdir_github_lib/../github/github_api.sh\" \"/repos/$repo/branches?page=$page&per_page=100\")\"; then\n            echo \"ERROR\" >&2\n            exit 1\n        fi\n        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n            break\n        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n            exit 1\n        fi\n        jq_debug_pipe_dump <<< \"$output\" |\n        jq -r \".[].name\"\n        ((page+=1))\n    done\n}\n\nparse_pull_request_url(){\n    local text=\"$1\"\n    grep -Eom1 \"$github_pull_request_url_regex\" <<< \"$text\" ||\n    die \"Failed to parse GitHub pull request URL\"\n}\n"
  },
  {
    "path": "lib/gitlab.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: echo user={user} name={name} repo={repo}\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-30 10:08:07 +0100 (Sun, 30 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_gitlab_lib=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir_gitlab_lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir_gitlab_lib/git.sh\"\n\n# shellcheck disable=SC2034\nusage_gitlab_cli_required=\"Requires GitLab CLI to be installed and configured, as well as jq\"\n\n# ERE format regex\ngitlab_pull_request_url_regex='https://gitlab.com/[[:alnum:]/_-]+/pull/[[:digit:]]+'\n\nget_gitlab_repo(){\n    git remote -v 2>/dev/null |\n    grep -E \"gitlab\\.[[:alnum:].-]+[/:]\" |\n    awk '{print $2}' |\n    head -n1 |\n    sed '\n        s,.*://,,;\n        s/.*@//;\n        s/[^:/]*[:/]//;\n        s/\\.git$//;\n        s|^/||;\n    '\n}\n\nis_gitlab_origin(){\n    git remote -v |\n    # permitting generic domain regex for those self-hosting their own gitlab servers\n    grep -Eq \"^origin.*gitlab\\.[[:alnum:].-]+[/:]\"\n}\n\ncheck_gitlab_origin(){\n    if ! is_gitlab_origin; then\n        die 'GitLab is not set as remote origin in current repo!'\n    fi\n}\n\ngitlab_origin_owner_repo(){\n    local owner_repo\n    owner_repo=\"$(\n        git remote -v |\n        grep -Em1 '^origin.*gitlab\\.[[:alnum:].-]+[/:]' |\n        sed '\n            s|.*gitlab\\.[[:alnum:].-]*[:/]||;\n            s/\\.git.*//;\n            s/[[:space:]].*//\n        ' ||\n        :\n    )\"\n    is_gitlab_owner_repo \"$owner_repo\" || die \"<owner>/<repo> '$owner_repo' does not match expected format\"\n    echo \"$owner_repo\"\n}\n\nis_gitlab_owner_repo(){\n    local repo=\"$1\"\n    # .gitlab repo is valid\n    [[ \"$repo\" =~ ^[[:alnum:]-]+/[[:alnum:]._-]+$ ]]\n}\n\nget_gitlab_user(){\n    if [ -n \"${GITLAB_USER:-}\" ]; then\n        echo \"$GITLAB_USER\"\n    else\n        # get currently authenticated user\n        \"$srcdir_gitlab_lib/../gitlab/gitlab_api.sh\" /user | jq -r .username\n    fi\n}\n\n#gitlab_result_has_more_pages(){\n#    local output=\"$1\"\n#    if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n#        return 1\n#    elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n#        exit 1\n#    fi\n#    return 0\n#}\n#\n#get_gitlab_repos(){\n#    local owner=\"${1:-}\"\n#    if [ -z \"$owner\" ]; then\n#        owner=\"$(get_gitlab_user)\"\n#    fi\n#    local is_org=\"${2:-}\"\n#    local filter=\"${3:-.}\"\n#    local prefix\n#    if [ -n \"$is_org\" ]; then\n#        prefix=\"orgs\"\n#    else\n#        prefix=\"users\"\n#    fi\n#    local page=1\n#    while true; do\n#        if ! output=\"$(\"$srcdir_gitlab_lib/../gitlab/gitlab_api.sh\" \"/$prefix/$owner/repos?page=$page&per_page=100\")\"; then\n#            echo \"ERROR\" >&2\n#            exit 1\n#        fi\n#        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n#            break\n#        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n#            exit 1\n#        fi\n#        jq_debug_pipe_dump <<< \"$output\" |\n#        jq -r \".[] | select(.fork | not) | $filter | .name\"\n#        ((page+=1))\n#    done\n#}\n#\n#get_gitlab_repo_branches(){\n#    local repo=\"$1\"\n#    local page=1\n#    while true; do\n#        if ! output=\"$(\"$srcdir_gitlab_lib/../gitlab_api.sh\" \"/repos/$repo/branches?page=$page&per_page=100\")\"; then\n#            echo \"ERROR\" >&2\n#            exit 1\n#        fi\n#        if [ -z \"$(jq '.[]' <<< \"$output\")\" ]; then\n#            break\n#        elif jq -r '.message' <<< \"$output\" >&2 2>/dev/null; then\n#            exit 1\n#        fi\n#        jq_debug_pipe_dump <<< \"$output\" |\n#        jq -r \".[].name\"\n#        ((page+=1))\n#    done\n#}\n#\n#parse_pull_request_url(){\n#    local text=\"$1\"\n#    grep -Eom1 \"$gitlab_pull_request_url_regex\" <<< \"$text\" ||\n#    die \"Failed to parse GitLab pull request URL\"\n#}\n"
  },
  {
    "path": "lib/kubernetes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-03-02 18:59:16 +0000 (Tue, 02 Mar 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# XXX: prevents race conditions from changes in global context\nkube_config_isolate(){\n    local tmp=\"/tmp/.kube\"\n    local default_kubeconfig=\"${HOME:-$(cd ~ && pwd)}/.kube/config\"\n    local original_kubeconfig=\"${KUBECONFIG:-$default_kubeconfig}\"\n\n    mkdir -pv \"$tmp\"\n\n    kubeconfig=\"$tmp/config.${EUID:-$UID}.$$\"\n\n    if [ -f \"$original_kubeconfig\" ]; then\n        cp -f -- \"$original_kubeconfig\" \"$kubeconfig\"\n    elif [ -f \"$default_kubeconfig\" ]; then\n        cp -f -- \"$default_kubeconfig\" \"$kubeconfig\"\n    elif [ -f \"$PWD/.kube/config\" ]; then\n        cp -f -- \"$PWD/.kube/config\" \"$kubeconfig\"\n    fi\n\n    export KUBECONFIG=\"$kubeconfig\"\n}\n\n# run 'kubectl config use-context' only if not already on the desired context, in order to minimize noise\nkube_context(){\n    local context=\"$1\"\n    local namespace=\"${2:-}\"\n    local current_context\n    current_context=\"$(kube_current_context)\"\n    if [ \"$context\" != \"$current_context\" ]; then\n        kubectl config use-context \"$context\" >&2\n    fi\n    kube_namespace \"$namespace\"\n}\n\nkube_current_context(){\n    kubectl config current-context\n}\n\nkube_current_namespace(){\n    kubectl config get-contexts |\n    { grep -F \"$(kube_current_context)\" || die \"Failed to get current context\"; } |\n    awk '{print $NF}'\n}\n\nkube_namespace(){\n    local namespace=\"$1\"\n    local current_namespace\n    current_namespace=\"$(kube_current_namespace)\"\n    if [ \"$namespace\" != \"$current_namespace\" ]; then\n        local current_context\n        current_context=\"$(kube_current_context)\"\n        kubectl config set-context \"$current_context\" --namespace \"$namespace\" >&2\n    fi\n}\n\nrun_static_pod(){\n    local name=\"$1\"\n    local image=\"$2\"\n    shift || :\n    shift || :\n    local pod_json\n    pod_json=\"$(kubectl get pod \"$name\" \"$@\" -o json 2>/dev/null || :)\"\n\n    local cmd=(/bin/sh -c 'if type bash >/dev/null 2>&1; then exec bash; else exec sh; fi')\n\n    launch_static_pod(){\n        exec kubectl run -ti --rm --restart=Never \"$name\" --image=\"$image\" \"$@\" -- \"${cmd[@]}\"\n    }\n\n    if [ -n \"$pod_json\" ]; then\n        if jq -e '.status.phase == \"Running\"' <<< \"$pod_json\" >/dev/null; then\n            kubectl exec -ti \"$name\" \"$@\" -- \"${cmd[@]}\"\n        elif jq -e '.status.phase == \"Succeeded\" or .status.phase == \"Failed\"' <<< \"$pod_json\" >/dev/null; then\n            kubectl delete pod \"$name\" \"$@\"\n            sleep 3\n            launch_static_pod \"$@\"\n        # This is what shows as ContainerCreating in kubectl get pods\n        elif jq -e '.status.phase == \"Pending\"' <<< \"$pod_json\" >/dev/null; then\n            echo \"Pod pending...\"\n            sleep 3\n            run_static_pod \"$name\" \"$image\" \"$@\"\n        else\n            echo \"ERROR: Pod already exists. Check its state and remove it?\"\n            kubectl get pod \"$name\" \"$@\"\n            return 1\n        fi\n    else\n        launch_static_pod \"$@\"\n    fi\n}\n"
  },
  {
    "path": "lib/mac.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-17 20:21:43 +0100 (Mon, 17 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# designed to be included from utils.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif ! is_mac; then\n    return\nfi\n\nif ! type tac &>/dev/null; then\n    tac(){\n        gtac \"$@\"\n    }\nfi\n"
  },
  {
    "path": "lib/mp3.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-22 23:56:40 +0100 (Wed, 22 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$srcdir/utils.sh\"\n\n# used in client scripts\n# shellcheck disable=SC2034\nmp3_usage_behaviour_msg=\"If no directory arguments are given, works on MP3s under \\$PWD\n\nOnly proposes MP3 files within direct subdirectories to reduce the chance of accidentally running on an entire top level MP3 archive\n\nShows the list of MP3 files that would be affected before running the metadata update and prompts for confirmation before proceeding for safety\"\n\nget_mp3_files(){\n    local mp3_files\n    mp3_files=\"$(\n        for dir in \"$@\"; do\n            find \"$dir\" -maxdepth 2 -iname '*.mp3'\n        done\n    )\"\n    if is_blank \"$mp3_files\"; then\n        echo \"No MP3 files found\" >&2\n        exit 1\n    fi\n    echo \"$mp3_files\"\n}\n"
  },
  {
    "path": "lib/os.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nis_linux(){\n    if [ \"$(uname -s)\" = \"Linux\" ]; then\n        return 0\n    fi\n    return 1\n}\n\nis_mac(){\n    if [ \"$(uname -s)\" = \"Darwin\" ]; then\n        return 0\n    fi\n    return 1\n}\n\nis_windows(){\n    case \"$(uname -s)\" in\n        CYGWIN*|MINGW*|MSYS*)   return 0\n                                ;;\n    esac\n    return 1\n}\n\nlinux_only(){\n    if ! is_linux; then\n        die \"Only Linux is supported\"\n    fi\n}\n\nmac_only(){\n    if ! is_mac; then\n        die \"Only macOS is supported\"\n    fi\n}\n\nwindows_only(){\n    if ! is_windows; then\n        die \"Only Windows is supported\"\n    fi\n}\n\nget_os(){\n    # shellcheck disable=SC3043\n    local os >/dev/null 2>&1 || :\n    if [ -n \"${OS_DARWIN:-}\" ]; then\n        if is_mac; then\n            os=\"$OS_DARWIN\"\n        fi\n    elif [ -n \"${OS_LINUX:-}\" ]; then\n        if is_linux; then\n            os=\"$OS_LINUX\"\n        fi\n    else\n        os=\"$(uname -s | tr '[:upper:]' '[:lower:]')\"\n    fi\n    echo \"$os\"\n}\n\nget_arch(){\n    # shellcheck disable=SC3043\n    local arch >/dev/null 2>&1 || :\n    arch=\"$(uname -m)\"\n    if [ \"$arch\" = x86_64 ]; then\n        arch=amd64  # files are conventionally usually named amd64 not x86_64\n    fi\n    #if [ \"$arch\" = aarch64 ]; then\n    #    arch=arm64\n    #fi\n    if [ -n \"${ARCH_X86_64:-}\" ]; then\n        if [ \"$arch\" = amd64 ] || [ \"$arch\" = x86_64 ]; then\n            arch=\"$ARCH_X86_64\"\n        fi\n    fi\n    if [ -n \"${ARCH_X86:-}\" ]; then\n        if [ \"$arch\" = i386 ]; then\n            arch=\"$ARCH_X86\"\n        fi\n    fi\n    if [ -n \"${ARCH_ARM64:-}\" ]; then\n        if [ \"$arch\" = arm64 ]; then\n            arch=\"$ARCH_ARM64\"\n        fi\n    fi\n    if [ -n \"${ARCH_ARM:-}\" ]; then\n        if [ \"$arch\" = arm ]; then\n            arch=\"$ARCH_ARM\"\n        fi\n    fi\n    if [ -n \"${ARCH_OVERRIDE:-}\" ]; then\n        arch=\"$ARCH_OVERRIDE\"\n    fi\n    echo \"$arch\"\n}\n"
  },
  {
    "path": "lib/packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:30:31 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#  Used on Alpine so needs to be /bin/sh\n\nset +eu  #o pipefail\nif [ \"${SHELL##*/}\" = bash ]; then\n    # shellcheck disable=SC2039\n    set -o pipefail\nfi\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# used in client code\n# shellcheck disable=SC2034\npackage_args_description=\"Takes a list of packages as arguments or via stdin,\nand for any arguments that are plaintext files,\nreads the packages from those given files (one package per line)\"\n\npackages=\"\"\n\n_process_package_args(){\n    for arg; do\n        if [ -f \"$arg\" ] && file \"$arg\" | grep -q ASCII; then\n            echo \"adding packages from file:  $arg\" >&2\n            # Bourne shell doesn't have arrays otherwise would use them here\n            packages=\"$packages $(sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d; /^[^[:alnum:]]/d' \"$arg\")\"\n            echo >&2\n        else\n            packages=\"$packages $arg\"\n        fi\n    done\n    # Homebrew tap package lists are in format \"tap package\" and those lines should not be split\n    echo \"$packages\" |\n    if [ -n \"${HOMEBREW_PACKAGES_TAP:-}\" ]; then\n        cat\n    else\n        # filter out commands in scripts which have spaces between tokens like 'cat <<END' or 'brew install'\n        # get rid of EOF / END type heredocs endings, using perl because Mac's sed has weak regex\n        #     /^[[:upper:]]{2,3}$/d'\n        tr ' ' '\\n' |\n        sed 's/^[[:space:]]*//; s/[[:space:]]*$//' |\n        #perl -p -e 's/^[[:upper:]]{3}$//' |\n        #sed '/[^[:space:]][[:space:]][^[:space:]]/d' |\n        sort -u || :\n    fi |\n    sed 's/^[[:space:]]*//;\n         s/[[:space:]]*$//;\n         /^[[:space:]]*$/d' || :\n}\n\nprocess_package_args(){\n    if [ $# -gt 0 ]; then\n        _process_package_args \"$@\"\n    else\n        #echo \"reading packages from stdin\" >&2\n        # need splitting\n        # shellcheck disable=SC2046\n        _process_package_args $(cat)\n    fi\n}\n\ninstalled_apk(){\n    apk info 2>/dev/null\n}\n\ninstalled_debs(){\n    dpkg-query -W -f '${db:Status-Abbrev}\\t${binary:Package}\\n' |\n    awk '/^i/{print $2}' |\n    sed 's/:.*$//' |\n    sort -u || :\n}\n\ninstalled_rpms(){\n    rpm -qa --queryformat '%{RPMTAG_NAME}\\n'\n}\n\nrpms_filter_provided(){\n    while read -r rpm; do\n        # accounts for vim being provided by vim-enhanced, so we don't try to install the metapackage again and again\n        rpm -q --whatprovides \"$rpm\" >/dev/null 2>&1 &&\n        echo \"$rpm\"\n    done\n}\n\nrpms_filter_not_provided(){\n    while read -r rpm; do\n        # accounts for vim being provided by vim-enhanced, so we don't try to install the metapackage again and again\n        rpm -q --whatprovides \"$rpm\" >/dev/null 2>&1 ||\n        echo \"$rpm\"\n    done\n}\n"
  },
  {
    "path": "lib/perl.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_bash_tools_perl=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# shellcheck disable=SC1090\n. \"$srcdir_bash_tools_perl/ci.sh\"\n\n# to avoid perl: warning: Falling back to the standard locale (\"C\")\n#\n# might have to run this on Debian/Ubuntu:\n#\n# sudo locale-gen en_US.UTF-8\n#\n# breaks on some systems, probably need to install something for setlocale\n#export LANGUAGE=\"${LANGUAGE:-en_US.UTF-8}\"\n#export LANG=\"${LANG:-en_US.UTF-8}\"\n#export LC_ALL=\"${LC_ALL:-en_US.UTF-8}\"\n\n# Taint code doesn't use PERL5LIB, use -I instead\n#I_lib=\"\"\n\nperl=\"perl\"\n\n# shellcheck disable=SC2230\nif ! type -P $perl &>/dev/null; then\n    if is_CI; then\n        if is_mac; then\n            find=\"gfind\"\n        else\n            find=\"find\"\n        fi\n        echo \"WARNING: $perl not found in \\$PATH ($PATH)\"\n        echo\n        echo \"searching filesystem for Perl:\"\n        echo\n        \"$find\" / -type f -name perl -executable 2>/dev/null\n        echo\n    fi >&2\n    return 0 &>/dev/null || :\n    exit 0\nfi\n\nif [ -n \"${PERLBREW_PERL:-}\" ]; then\n\n    PERL_VERSION=\"${PERLBREW_PERL}\"\n    export PERL_VERSION=\"${PERLBREW_PERL/perl-/}\"\n    sudo=\"\"\n\n    # For Travis CI which installs modules locally\n#    export PERL5LIB=$(echo \\\n#        ${PERL5LIB:-.} \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/site_perl/$PERL_VERSION/x86_64-linux \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/site_perl/$PERL_VERSION/darwin-2level \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/site_perl/$PERL_VERSION \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/$PERL_VERSION/x86_64-linux \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/$PERL_VERSION/darwin-2level \\\n#        $PERLBREW_ROOT/perls/$PERLBREW_PERL/lib/$PERL_VERSION \\\n#        | tr '\\n' ':'\n#    )\n\n    # gets this error when not specifying full perl path:\n        # Can't load '/home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/x86_64-linux/auto/re/re.so' for module re: /home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/x86_64-linux/auto/re/re.so: undefined symbol: PL_valid_types_IVX at /home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/XSLoader.pm line 68.\n        # at /home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/x86_64-linux/re.pm line 85.\n        # Compilation failed in require at /home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/File/Basename.pm line 44.\n        # BEGIN failed--compilation aborted at /home/travis/perl5/perlbrew/perls/5.16/lib/5.16.3/File/Basename.pm line 47.\n        # Compilation failed in require at ./check_riak_diag.pl line 25.\n        # BEGIN failed--compilation aborted at ./check_riak_diag.pl line 25.\n\n    #perl=\"$PERLBREW_ROOT/perls/$PERLBREW_PERL/bin/perl $I_lib\"\n\n    # don't want dollars to expand\n    # shellcheck disable=SC2016\n    PERL_MAJOR_VERSION=\"$($perl -v | $perl -ne '/This is perl (\\d+), version (\\d+),/ && print \"$1.$2\"')\"\nelse\n    PERL_VERSION=\"$(perl --version | grep -Eom 1 'v[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+' | sed 's/^v//')\"\n    # don't want dollars to expand\n    # shellcheck disable=SC2016\n    PERL_MAJOR_VERSION=\"$($perl -v | $perl -ne '/This is perl (\\d+), version (\\d+),/ && print \"$1.$2\"')\"\nfi\n\nI_lib=\"\"\nPERL5LIB=\"${PERL5LIB:-}\"\n# because PERL5LIB is not respected in Taint mode, but -I is because it's more explicit\nfor x in ${PERL5LIB//:/ }; do\n    I_lib+=\"-I $x \"\ndone\n# this breaks a lot of stuff because client code rightly assumes to run \"$perl\"\n#perl=\"$perl $I_lib\"\n\nexport sudo\nexport PERL_VERSION\nexport PERL_MAJOR_VERSION\n"
  },
  {
    "path": "lib/python.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-03 13:14:22 +0100 (Fri, 03 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_bash_tools_python=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# shellcheck disable=SC1090\n. \"$srcdir_bash_tools_python/ci.sh\"\n\n# shellcheck disable=SC1090\n. \"$srcdir_bash_tools_python/os.sh\"\n\n# need all the paths for when Pip gets installed locally\nif ! type add_PATHS &>/dev/null ; then\n    # shellcheck disable=SC1090\n    . \"$srcdir_bash_tools_python/../.bash.d/python.sh\"\nfi\n\n# set to true for debugging CI builds like Semaphore CI's weird Python environment on Mac where it defaults to /usr/bin/python (2.7)\n# but /usr/local/opt/python/libexec/bin/pip (python 3.7) or /usr/local/bin/pip3 (python 3.8), causing library installation vs runtime import mismatches\n# (now checked for further below to catch early and highlight the root cause)\n#if [ -n \"${DEBUG:-}\" ]; then\n#    if is_semaphore_ci; then\n#        echo\n#        echo \"Python and Pip installations:\"\n#        # very slow, pushes build past 1 hour\n#        for x in python python2 python3 pip pip2 pip3; do\n#            find / -type f -name \"$x\" -exec ls -l {} \\; -o \\\n#                   -type l -name \"$x\" -exec ls -l {} \\; 2>/dev/null || :\n#        done\n#        echo\n#        echo\n#    fi\n#fi\n\nif [ -n \"${PYTHON:-}\" ]; then\n    python=\"$PYTHON\"\nelse\n    #python=\"$(command -v \"$python\" || command -v \"python3\" || command -v \"python2\" || :)\"\n    # XXX: bug, on new M1 Macs command -v appears to return 'python' where it is not installed, possibly inherited, whereas we need it to fail to fall through to python3\n    #python=\"$(command -v python 2>/dev/null || :)\"\n    python=\"$(type -P python 2>/dev/null || :)\"\n    python2=\"$(type -P python2 2>/dev/null || :)\"\n    python3=\"$(type -P python3 2>/dev/null || :)\"\n    if [ -z \"$python\" ]; then\n        if [ -n \"${python3:-}\" ]; then\n            python=\"$python3\"\n        elif [ -n \"${python2:-}\" ]; then\n            python=\"$python2\"\n        else\n            echo \"ERROR: 'command -v python' failed to find python\" >&2\n            exit 1\n        fi\n    fi\nfi\n\nif [ -n \"${PIP:-}\" ]; then\n    pip=\"$PIP\"\nelse\n    pip=\"$(command -v pip 2>/dev/null || :)\"\n    pip2=\"$(command -v pip2 2>/dev/null || :)\"\n    pip3=\"$(command -v pip3 2>/dev/null || :)\"\n    if [ -z \"$pip\" ]; then\n        if [ -n \"$pip3\" ]; then\n            pip=\"$pip3\"\n        elif [ -n \"${pip2:-}\" ]; then\n            pip=\"$pip2\"\n        else\n            echo \"pip not found, falling back to trying just 'pip'\" >&2\n            pip=pip\n        fi\n    fi\nfi\n\ninstall_pip_manually(){\n    if is_mac; then\n        brew install openssl || :\n        brew_prefix=\"$(brew --prefix)\"\n        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n    fi\n    if \"$python\" -V 2>&1 | grep -q '^Python 2'; then\n        curl -sS https://bootstrap.pypa.io/pip/2.7/get-pip.py -o get-pip.py\n    else\n        curl -sS https://bootstrap.pypa.io/get-pip.py -o get-pip.py\n    fi\n    \"$python\" get-pip.py\n    rm -f -- get-pip.py\n}\n\n# Needed on Semaphore Mac builds and also Ubuntu 20.04 LTS\n#\n#if is_semaphore_ci; then\n#    echo \"Semaphore CI detected, installing manually to avoid non-SSL version\"\n#    install_pip_manually\nif ! command -v \"$pip\" >/dev/null 2>&1; then\n    echo \"pip not installed, trying to install manually...\"\n    install_pip_manually\nfi\n\n# bad idea, programs with /usr/env/python will often call python 2 and fail to find pip modules\n#if [[ \"$pip\" =~ pip3$ ]] &&\n#   \"$python\" -V 2>&1 | grep -q 'Python 2'; then\n#   [ -L \"$python\" ] &&\n##    python3=\"$(command -v python3 2>/dev/null || :)\"\n##    if [ -n \"$python3\" ]; then\n##        echo \"Symlinking python to python3 to match pip3:\"\n##        ln -sfv \"$python3\" \"$python\"\n##    fi\n#    if command -v python3 &>/dev/null; then\n#        python=\"$(command -v python3)\"\n#    fi\n#elif [[ \"$python\" =~ python3 ]] &&\n#    \"$pip\" -V 2>&1 | grep -qi 'python[ /]2'; then\n#    if command -v pip3 &>/dev/null; then\n#        pip=\"$(command -v pip3)\"\n#    fi\n#fi\n\n# replace with fully qualified, which aids in debugging different CI environments\npip=\"$(command -v \"$pip\")\"\n\nrecursion_depth=0\ncheck_python_pip_versions_match(){\n    ((recursion_depth+=1))\n    if [ $recursion_depth -gt 5 ]; then\n        echo \"recursing too deep in $srcdir_bash_tools_python/python.sh, non-trivial python vs pip versions mismatch!\"\n        exit 1\n    fi\n    set +eo pipefail\n    # split steps for easier CI debugging in DEBUG mode\n    python_version=\"$(\"$python\" -V 2>&1)\"\n    python_version=\"$(<<< \"$python_version\" grep -Eom1 '[[:digit:]]+\\.[[:digit:]]+')\"\n    export python_major_version=\"${python_version%%.*}\"\n    pip_python_version=\"$(\"$pip\" -V 2>&1)\"\n    pip_python_version=\"$(<<< \"$pip_python_version\" grep -Eom1 '\\(python [[:digit:]]+\\.[[:digit:]]+\\)' | sed 's/(python[[:space:]]*//; s/)//')\"\n    export pip_python_major_version=\"${pip_python_version%%.*}\"\n    set -eo pipefail\n\n    if [ -n \"${python_version:-}\" ] &&\n       [ -n \"${pip_python_version:-}\" ]; then\n        if [ \"$python_version\" != \"$pip_python_version\" ]; then\n            if [ \"${python_version:0:1}\" = 3 ] &&\n               [ -n \"${pip3:-}\" ]; then\n                pip=\"$pip3\"\n                check_python_pip_versions_match\n            elif [ \"${python_version:0:1}\" = 2 ]; then\n                if [ -n \"${pip2:-}\" ]; then\n                    pip=\"$pip2\"\n                else\n                    # python2-pip removed from Ubuntu / Alpine repos :-(\n                    install_pip_manually\n                    pip=\"$(command -v \"pip\")\"\n                fi\n                check_python_pip_versions_match\n            # switching to python3 will lead programs with /usr/env/python defaulting to python 2 to fail to find pip modules\n            #elif [ \"${python_version:0:1}\" = 2 ] &&\n            #     [ -n \"${python3:-}\" ]; then\n            #    python=\"$python3\"\n            #    check_python_pip_versions_match\n            else\n                echo\n                echo \"Python major version '$python_version' != Pip major version '$pip_python_version' !!\"\n                echo\n                echo \"python = $python\"\n                echo \"pip    = $pip\"\n                echo\n                echo \"Python PyPI modules will not be installed to the correct site-packages and will lead to import failures later on\"\n                echo\n                echo \"Fix your \\$PATH or \\$PYTHON / \\$PIP to be aligned to the same installation\"\n                echo\n                exit 1\n            fi\n        fi\n    fi\n}\n\ncheck_python_pip_versions_match\n\ninside_virtualenv(){\n    if [ -n \"${VIRTUAL_ENV:-}\" ] ||\n       #[ -n \"${PYENV_ROOT:-}\" ] ||\n       [ -n \"${CODESHIP_VIRTUALENV:-}\" ] ||\n       [ -n \"${CONDA_DEFAULT_ENV:-}\" ]; then\n        return 0\n    elif [ -n \"${PYENV_ROOT:-}\" ]; then\n        if command -v \"$python\" | grep -q \"$PYENV_ROOT\" &&\n           command -v \"$pip\" | grep -q \"$PYENV_ROOT\"; then\n            return 0\n        fi\n    # GitHub Actions Python versions\n    elif command -v \"$python\" | grep -Eqi '/hostedtoolcache/'; then\n    #or\n    #elif command -v \"$pip\" | grep -Eqi '/hostedtoolcache/'; then\n        return 0\n    # CircleCI uses /opt/circleci/.pyenv/shims/python\n    # Codeship path when using virtualenv\n    elif command -v \"$python\" | grep -Eqi '/\\.pyenv/|/shims/|/home/'; then\n        return 0\n    elif command -v \"$pip\" | grep -Eqi '/\\.pyenv/|/shims/|/home/'; then\n        return 0\n    fi\n    return 1\n}\n\nfind_python_files(){\n    local startpath=\"${1:-.}\"\n    shift || :\n    find \"$startpath\" \"$@\" -type f -iname '*.py' |\n    sort\n}\nfind_jython_files(){\n    local startpath=\"${1:-.}\"\n    shift || :\n    find \"$startpath\" \"$@\" -type f -iname '*.jy' |\n    sort\n}\nfind_python_jython_files(){\n    local startpath=\"${1:-.}\"\n    shift || :\n    find \"$startpath\" \"$@\" -type f -iname '*.py' -o \\\n                           -type f -iname '*.jy' |\n    sort\n}\n"
  },
  {
    "path": "lib/ruby.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-16 21:51:27 +0100 (Thu, 16 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir_bash_tools_python=\"$(dirname \"${BASH_SOURCE[0]}\")\"\n\n# shellcheck disable=SC1090\n#. \"$srcdir_bash_tools_python/ci.sh\"\n\n# shellcheck disable=SC1090\n#. \"$srcdir_bash_tools_python/os.sh\"\n\ninside_ruby_virtualenv(){\n    # $HOME/.rbenv/shims/ruby\n    if inside_rvm || inside_rbenv; then\n        return 0\n    fi\n    return 1\n}\n\n# https://github.com/rbenv/rbenv#command-reference\ninside_rbenv(){\n    # $HOME/.rbenv/shims/ruby\n    # this could be true and still set to system\n    #if command -v ruby | grep -q -e '/\\.rbenv/'; then\n    #\n    #rbenv local - $PWD/.ruby-version\n    #rbenv global - ~/.rbenv/version\n    # gem env home exposes this in one\n    if [ -n \"${RBENV_VERSION:-}\" ] ||\n       gem env home 2>/dev/null | grep -q -e '/\\.rbenv/'; then\n        # technically should check if $RBENV_ROOT is set and also 'rbenv root' but this would be more expensive and should always be .rbenv anyway\n        return 0\n    fi\n    return 1\n}\n\ninside_rvm(){\n    if [[ \"${GEM_HOME:-}${MY_RUBY_HOME:-}\" =~ /\\.rvm/ ]] ||\n        command -v ruby | grep -q -e '/\\.rvm/'; then\n        return 0\n    fi\n    return 1\n}\n"
  },
  {
    "path": "lib/spotify.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 00:26:08 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nlibdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$libdir/utils.sh\"\n\noffset=\"${SPOTIFY_OFFSET:-0}\"\n# conservative, some API endpoints can take 100, others 50 - this is the safe choice but will go faster for those APIs if you can set 100 where appropriate\nlimit=\"${SPOTIFY_LIMIT:-50}\"\n\nif ! [[ \"$offset\" =~ ^[[:digit:]]+$ ]]; then\n    echo \"Invalid \\$SPOTIFY_OFFSET = $offset found in environment\" >&2\n    exit 1\nfi\n\nif ! [[ \"$limit\" =~ ^[[:digit:]]+$ ]]; then\n    echo \"Invalid \\$SPOTIFY_LIMIT = $limit found in environment\" >&2\n    exit 1\nfi\n\nspotify_token(){\n    # tokens with authorization have length 281, tokens without 219, so if there is an unauthorized token in the environment, regenerate it to avoid 401 errors\n    # [ -a ] works in bash and is the most concise way of doing this\n    # shellcheck disable=SC2166\n    if [ -z \"${SPOTIFY_ACCESS_TOKEN:-}\" ] ||\n       [ -n \"${SPOTIFY_PRIVATE:-}\" -a \"${#SPOTIFY_ACCESS_TOKEN}\" -lt 280 ]; then\n        SPOTIFY_ACCESS_TOKEN=\"$(\"$libdir/../spotify/spotify_api_token.sh\")\"\n    fi\n    export SPOTIFY_ACCESS_TOKEN\n}\n\nspotify_user(){\n    spotify_user=\"${spotify_user:-${SPOTIFY_USER:-}}\"\n    if [ -z \"$spotify_user\" ]; then\n        if [ -n \"${SPOTIFY_PRIVATE:-}\" ]; then\n            spotify_user=\"$(SPOTIFY_PRIVATE=1 \"$libdir/../spotify_api.sh\" \"/v1/me\" | jq -r '.id')\"\n        else\n            usage \"\\$SPOTIFY_USER not defined, and not using SPOTIFY_PRIVATE to auto-infer user from an authorized token\"\n        fi\n    fi\n}\n\n# used by client scripts\n# shellcheck disable=SC2034\nusage_auth_msg=\"Requires \\$SPOTIFY_ACCESS_TOKEN, or \\$SPOTIFY_ID and \\$SPOTIFY_SECRET to be defined in the environment\"\n\n# srcdir defined in client scripts\n# shellcheck disable=SC2034,SC2154\nusage_token_private=\"export SPOTIFY_ACCESS_TOKEN=\\\"\\$(SPOTIFY_PRIVATE=1 '$libdir/../spotify_api_token.sh')\\\"\"\n\n# shellcheck disable=SC2034\nusage_playlist_help=\"See spotify_playlists.sh --help for details on accessing private playlists\"\n\n# shellcheck disable=SC2034\nusage_auth_help=\"See spotify_api_token.sh --help for authentication details and setting up your SPOTIFY_ID, SPOTIFY_SECRET and callback URL\"\n\nget_next(){\n    jq -r '.next' <<< \"$*\"\n}\n\nis_local_uri(){\n    [[ \"$1\" =~ ^spotify:local:|open.spotify.com/local/ ]]\n}\n\nis_spotify_playlist_id(){\n    local playlist_id=\"${1:-}\"\n    #if [ -z \"$playlist_id\" ]; then\n    #    die \"no playlist id passed to function is_spotify_playlist_id()\"\n    #fi\n    [[ \"$playlist_id\" =~ ^[A-Za-z0-9]{22}$ ]]\n}\n\nis_spotify_playlist_snapshot_id(){\n    local snapshot_id=\"${1:-}\"\n    [[ \"$snapshot_id\" =~ ^[A-Za-z0-9+/]{20,}={0,2}$ ]]\n}\n\nvalidate_spotify_uri(){\n    local uri=\"$1\"\n    if ! [[ \"$uri\" =~ ^(spotify:(track|album|artist|episode):|^https?://open.spotify.com/(track|album|artist|episode)/)?[[:alnum:]]+(\\?.+)?$ ]]; then\n        echo \"Invalid URI provided: $uri\" >&2\n        return 1\n    fi\n    # My Love Island playlist has a mix of tracks and episodes so this breaks\n    #\n    #   Invalid URI type 'track' vs URI 'spotify:episode:4Dl83m4Ibate3jn0hJFMzX'\n    #\n    #if [[ \"$uri\" =~ open.spotify.com/|^spotify: ]]; then\n    #    if ! [[ \"$uri\" =~ open.spotify.com/${uri_type:-track}|^spotify:${uri_type:-track} ]]; then\n    #        echo \"Invalid URI type '${uri_type:-track}' vs URI '$uri'\" >&2\n    #        return 1\n    #    fi\n    #fi\n    uri=\"${uri##*[:/]}\"\n    uri=\"${uri%%\\?*}\"\n    echo \"$uri\"\n}\n"
  },
  {
    "path": "lib/sql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-11 01:45:42 +0300 (Fri, 11 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n# shellcheck disable=SC2034\nsrcdir_sql_lib=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# try to infer the SQL dialect from the filename or path hoping there is a clue in a naming convention somewhere along the path of directory of file naming convention\n#\n# written to be able to auto-infer what the --dialect arg to sqlfluff should be for check_sqlfluff.sh script but can be reused for other SQL tools as this is a generally useful thing to be able to infer\n#\ninfer_sql_dialect_from_path(){\n    local path=\"$1\"\n    local basename=\"${path##*/}\"\n    if [[ \"$path\" =~ mysql ]] ||\n       [[ \"$path\" =~ mariadb ]]; then\n        echo \"mysql\"\n        return\n    # postgres covers postgresql via substring match, no need for postgresql specific match\n    elif [[ \"$path\" =~ postgres ]]; then\n        echo \"postgres\"\n        return\n    elif [[ \"$path\" =~ oracle ]] ||\n         [[ \"$basename\" =~ \\.plsql$ ]]; then\n        echo \"oracle\"\n        return\n    elif [[ \"$path\" =~ mssql ]] ||\n         # do not path match tsql as it could have unintended consequences of matching the end of unrelated <word>sql\n         #[[ \"$path\" =~ tsql ]] ||\n         [[ \"$path\" =~ transactsql ]] ||\n         [[ \"$path\" =~ microsoft ]] ||\n         [[ \"$basename\" =~ \\.tsql$ ]]; then\n        echo \"mssql\"\n        return\n    elif [[ \"$path\" =~ athena ]]; then\n        echo \"athena\"\n        return\n    elif [[ \"$path\" =~ bigquery ]]; then\n        echo \"bigquery\"\n        return\n    elif [[ \"$path\" =~ databricks ]] ||\n         [[ \"$path\" =~ sparksql ]]; then\n        echo \"sparksql\"\n        return\n    elif [[ \"$path\" =~ hive ]]; then\n         # don't try this as HSQLDB uses this file extension and most Hive practioners don't (I used to be one back in Cloudera and Hortonworks days for much of the 2010s)\n         #[[ \"$path\" =~ \\.hsql ]]; then\n        echo \"hive\"\n        return\n    elif [[ \"$path\" =~ redshift ]] ||\n         [[ \"$basename\" =~ \\.rd$ ]]; then\n        echo \"vertica\"\n        return\n    elif [[ \"$path\" =~ vertica ]] ||\n         [[ \"$basename\" =~ \\.vsql$ ]]; then\n        echo \"vertica\"\n        return\n    elif [[ \"$path\" =~ teradata ]] ||\n         [[ \"$basename\" =~ \\.td$ ]]; then\n        echo \"teradata\"\n        return\n    elif [[ \"$path\" =~ greenplum ]] ||\n         [[ \"$basename\" =~ \\.gpd$ ]]; then\n        echo \"greenplum\"\n        return\n    elif [[ \"$path\" =~ clickhouse ]] ||\n         [[ \"$basename\" =~ \\.ch$ ]]; then\n        echo \"greenplum\"\n        return\n    elif [[ \"$path\" =~ duckdb ]] ||\n         [[ \"$basename\" =~ \\.duck$ ]]; then\n        echo \"greenplum\"\n        return\n    else\n        # Remaining dialects that we don't have special multi-rules or priorities for, shorten the code with a simple direct mapping loop, reference:\n        #\n        #   https://docs.sqlfluff.com/en/stable/reference/dialects.html\n        #\n            # I use words like materialize often and don't trust it to be inferred as an SQL dialect\n            #materialize \\\n        for dialect in \\\n            db2 \\\n            exasol \\\n            snowflake \\\n            soql \\\n            sqlite \\\n            trino \\\n            ; do\n            if [[ \"$path\" =~ $dialect ]]; then\n                echo \"$dialect\"\n                return\n            fi\n        done\n\n    fi\n    echo \"Failed to detect SQL dialect from path '$path'\" >&2\n    return 1\n}\n"
  },
  {
    "path": "lib/travis.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 10:25:34 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nlibdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$libdir/git.sh\"\n\nget_travis_user(){\n    \"$libdir/../travis_api.sh\" /user | jq -r '.login'\n}\n\ntravis_prefix_repo(){\n    local repo=\"$1\"\n    local user\n    if ! [[ \"$repo\" =~ /|%2F ]]; then\n        user=\"$(get_travis_user)\"\n        repo=\"$user/$repo\"\n    fi\n    echo \"$repo\"\n}\n\ntravis_url_encode_repo(){\n    local repo=\"$1\"\n    repo=\"${repo//\\//%2F}\"\n    echo \"$repo\"\n}\n\ntravis_prefix_encode_repo(){\n    local repo=\"${1:-}\"\n    if [ -z \"$repo\" ]; then\n        repo=\"$(git_repo)\"\n    fi\n    repo=\"$(travis_prefix_repo \"$repo\")\"\n    repo=\"$(travis_url_encode_repo \"$repo\")\"\n    echo \"$repo\"\n}\n"
  },
  {
    "path": "lib/utils-bourne.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir_bash_tools_utils_bourne=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\n\nif [ \"${bash_tools_utils_bourne_imported:-0}\" = 1 ]; then\n    return 0\nfi\nbash_tools_utils_bourne_imported=1\n\nam_root(){\n    # shellcheck disable=SC2039,SC3028\n    [ \"${EUID:-${UID:-$(id -u)}}\" -eq 0 ]\n}\n\nif am_root; then\n    sudo=\"\"\nelse\n    sudo=sudo\nfi\nexport sudo\n\nexport support_msg=\"Please raise a GitHub Issue at https://github.com/HariSekhon/DevOps-Bash-tools/issues\"\n"
  },
  {
    "path": "lib/utils.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2128,SC2230,SC1090\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir_bash_tools_utils=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ \"${bash_tools_utils_imported:-0}\" = 1 ]; then\n    return 0\nfi\nbash_tools_utils_imported=1\n\n. \"$srcdir_bash_tools_utils/utils-bourne.sh\"\n\nexport PATH=\"$PATH:/usr/local/bin\"\n\n. \"$srcdir_bash_tools_utils/ci.sh\"\n. \"$srcdir_bash_tools_utils/docker.sh\"\n. \"$srcdir_bash_tools_utils/os.sh\"\n. \"$srcdir_bash_tools_utils/mac.sh\"\n. \"$srcdir_bash_tools_utils/perl.sh\"\n. \"$srcdir_bash_tools_utils/../.bash.d/colors.sh\"\n#. \"$srcdir_bash_tools_utils/python.sh\"\n#. \"$srcdir_bash_tools_utils/ruby.sh\"\n\n# consider adding ERR as set -e handler, not inherited by shell funcs / cmd substitutions / subshells without set -E\nexport TRAP_SIGNALS=\"INT QUIT TRAP ABRT TERM EXIT\"\n\n# prevents illegal byte encoding errors when piping to filenames with unicode characters\n# doesn't work in CentOS 8 docker, gets this error\n# bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)\n#export LC_ALL=en_US.UTF-8\n# but this works\nexport LANG=en_US.UTF-8\n\nif [ -z \"${run_count:-}\" ]; then\n    run_count=0\nfi\nif [ -z \"${total_run_count:-}\" ]; then\n    total_run_count=0\nfi\n\n# ERE format (egrep / grep -E)\n#\n# used in client scripts\n# shellcheck disable=SC2034\ndomain_regex='(([A-Za-z0-9](-?[A-Za-z0-9])*)\\.)+[A-Za-z]{2,}'\n# shellcheck disable=SC2034\nemail_regex='[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}'\n# shellcheck disable=SC2034\nip_regex='[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}'\n# shellcheck disable=SC2034\nurl_regex='https?://[[:alnum:]@%\\._\\+~#?&/=-]+'  # TODO: improve\n\nwrong_port=1111\n\ndie(){\n    echo \"$@\" >&2\n    exit 1\n}\n\nhr(){\n    echo \"================================================================================\"\n}\n\nhr2(){\n    echo \"==================================================\"\n}\n\nhr3(){\n    echo \"========================================\"\n}\n\nsection(){\n    name=\"$*\"\n    hr\n    \"$srcdir_bash_tools_utils/../bin/center.sh\" \"$@\"\n    hr\n    if [ -n \"${PROJECT:-}\" ]; then\n        echo \"PROJECT: $PROJECT\"\n    fi\n    if is_inside_docker; then\n        echo \"(running inside docker)\"\n    fi\n    echo\n}\n\nsection2(){\n    hr2\n    hr2echo \"$@\"\n    hr2\n    echo\n}\n\nsection3(){\n    hr3\n    hr3echo \"$@\"\n    hr3\n    echo\n}\n\nhr2echo(){\n    \"$srcdir_bash_tools_utils/../bin/center.sh\" \"$@\" 50\n}\n\nhr3echo(){\n    \"$srcdir_bash_tools_utils/../bin/center.sh\" \"$@\" 40\n}\n\n#set +o pipefail\n#spark_home=\"$(ls -d tests/spark-*-bin-hadoop* 2>/dev/null | head -n 1)\"\n#set -o pipefail\n#if [ -n \"$spark_home\" ]; then\n#    export SPARK_HOME=\"$spark_home\"\n#fi\n\n# shellcheck disable=SC1090\ntype isExcluded &>/dev/null || . \"$srcdir_bash_tools_utils/excluded.sh\"\n\n\ncheck_bin(){\n    local bin=\"${1:-}\"\n    if ! type -P \"$bin\" &>/dev/null; then\n        echo \"command '$bin' not found in \\$PATH ($PATH)\" >&2\n        if is_CI; then\n            timestamp \"Running in CI, searching entire system for '$bin'\"\n            find / -type f -name \"$bin\" 2>/dev/null\n        fi\n        exit 1\n    fi\n}\n\n\ncheck_output(){\n    local expected=\"$1\"\n    # shellcheck disable=SC2178\n    local cmd=\"${*:2}\"\n    # do not 2>&1 it will cause indeterministic output even with python -u so even tail -n1 won't work\n    echo \"check_output:  $cmd\"\n    echo \"expecting:     $expected\"\n    local output\n    output=\"$($cmd)\"\n    # intentionally not quoting so that we can use things like google* glob matches for google.com and google.co.uk\n    # shellcheck disable=SC2053\n    if [[ \"$output\" = $expected ]]; then\n        echo \"SUCCESS:       $output\"\n    else\n        die \"FAILED:        $output\"\n    fi\n    echo\n}\n\ncheck_exit_code(){\n    local exit_code=$?\n    local expected_exit_codes\n    expected_exit_codes=\"$*\"\n    local failed\n    failed=1\n    for e in $expected_exit_codes; do\n        if [ \"$exit_code\" = \"$e\" ]; then\n            echo \"got expected exit code: $e\"\n            failed=0\n        fi\n    done\n    if [ \"$failed\" != 0 ]; then\n        echo \"WRONG EXIT CODE RETURNED! Expected: '$expected_exit_codes', got: '$exit_code'\" >&2\n        return 1\n    fi\n}\n\ntick_msg(){\n    # defined in .bash.d/colors.sh\n    # shellcheck disable=SC2154\n    echo -e \"${bldgrn}✓ ${txtrst}$*\"\n}\n\ncpu_count(){\n    local cpu_count\n    if type -P nproc &>/dev/null; then\n        nproc\n        return\n    elif is_mac; then\n        cpu_count=\"$(sysctl -n hw.ncpu)\"\n    else\n        #cpu_count=\"$(awk '/^processor/ {++n} END {print n+1}' /proc/cpuinfo)\"\n        cpu_count=\"$(grep -c '^processor[[:space:]]*:' /proc/cpuinfo)\"\n    fi\n    echo \"$cpu_count\"\n}\n\nhas_terminal(){\n    [ -t 0 ]\n}\n\nis_tty(){\n    has_terminal\n}\n\nis_piped(){\n    ! [ -t 1 ]\n}\n\n# beware this results false in scripts, has_terminal is probably what you want\nis_interactive(){\n    if [ -n \"${PS1:-}\" ]; then\n        return 0\n    fi\n    case \"$-\" in\n        *i*) return 0\n    esac\n    return 1\n}\n\nfile_newer_than_mins(){\n    local mins=\"$1\"\n    local file=\"$2\"\n    local mtime\n    if ! is_int \"$mins\"; then\n        die \"Error: non-integer passed as first arg to file_newer_than_mins() function\"\n    fi\n    local secs=\"$((mins * 60))\"\n\n    if is_mac; then\n        mtime=$(stat -f %m \"$file\")\n    else  # assume Linux GNU stat\n        mtime=$(stat -c %Y \"$file\")\n    fi\n\n    (( \"$(date +%s)\" - mtime <= secs ))\n}\n\n# XXX: there are other tarball extensions for other compression algorithms but these are the 2 very standard ones we always use: gzip or bz2\nhas_tarball_extension(){\n    local filename=\"$1\"\n    # .tgz\n    # .tbz\n    # .tar.gz\n    # .tar.bz2\n    #[[ \"$filename\" =~ \\.(tgz|tbz|tar(\\.(gz|bz2))?)$ ]]\n    has_tarball_gzip_extension \"$filename\" ||\n    has_tarball_bzip2_extension \"$filename\"\n}\n\nhas_tarball_gzip_extension(){\n    local filename=\"$1\"\n    # .tgz\n    # .tar.gz\n    [[ \"$filename\" =~ \\.(tgz|tar\\.gz)$ ]]\n}\n\nhas_tarball_bzip2_extension(){\n    local filename=\"$1\"\n    # .tbz\n    # .tar.bz2\n    [[ \"$filename\" =~ \\.(tbz|tar\\.bz2)$ ]]\n}\n\ncurl(){\n    local opts=()\n    if is_piped || is_CI; then\n        opts+=(-sSf)\n    fi\n    command curl ${opts:+\"${opts[@]}\"} \"$@\"\n}\n\nwget(){\n    local opts=(-c)\n    if is_piped || is_CI; then\n        opts+=(--no-verbose)  # -q suppresses the error messages we need to debug, leading to silent exits\n    fi\n    command wget ${opts:+\"${opts[@]}\"} \"$@\"\n}\n\ndownload(){\n    local url=\"$1\"\n    local download_file=\"${2:-${url##*/}}\"\n    if type -P wget &>/dev/null; then\n        wget -O \"$download_file\" \"$url\"\n    elif type -P curl &>/dev/null; then\n        curl -sSLf -o \"$download_file\" \"$url\"\n    else\n        die \"wget / curl not installed - cannot download\"\n    fi\n}\n\nis_latest_version(){\n    # permit .* as we often replace version if latest with .* to pass regex version tests, which allows this to be called any time\n    if [ \"$version\" = \"latest\" ] || [ \"$version\" = \".*\" ]; then\n        return 0\n    fi\n    return 1\n}\n\ncurl_version(){\n    curl --version | awk '{print $2; exit}' | grep -Eom1 '[[:digit:]]+\\.[[:digit:]]+'\n}\nis_curl_min_version(){\n    # shellcheck disable=SC2178\n    local target_version=\"$1\"\n    local curl_version\n    curl_version=\"$(curl_version)\"\n    #bc_bool \"$curl_version >= $target_version\"\n    is_min_version \"$curl_version\" \"$target_version\"\n}\n\ngolang_version(){\n    go version | grep -Eom1 '[[:digit:]]+\\.[[:digit:]]+'\n}\ngo_version(){\n    golang_version\n}\n\nis_golang_min_version(){\n    # shellcheck disable=SC2178\n    local target_version=\"$1\"\n    local go_version\n    go_version=\"$(go_version)\"\n    is_min_version \"$go_version\" \"$target_version\"\n}\nis_go_min_version(){\n    is_golang_min_version \"$@\"\n}\n\nis_min_version(){\n    local IFS=.\n    # shellcheck disable=SC2206\n    local version=($1)\n    # shellcheck disable=SC2206\n    local target_version=($2)\n    local i\n    for ((i=0; i < ${#target_version[@]}; i++)); do\n        if [[ -z \"${version[i]:-}\" ]]; then\n            # fill empty fields with zeros\n            version[i]=0\n        fi\n        if (( target_version[i] > version[i] )); then\n            return 1\n        fi\n    done\n    return 0\n}\n\nis_semver(){\n    # shellcheck disable=SC2178\n    local version=\"$1\"\n    local allowed_prefix=\"${2:-v}\"\n    [[ \"$version\" =~ ^${allowed_prefix}?[[:digit:]]+\\.[[:digit:]]+\\.[[:digit:]]+$ ]]\n}\n\nbc_bool(){\n    # bc returns 1 when expression is true and zero otherwise, but this is counterintuitive\n    # to regular shell scripting, let's use the actual output 1 for true, 0 for false\n    # echo rather than <<< to show expression to evaluate in trace debugging\n    echo \"$@\" | bc -l | grep -q 1\n}\n\ncurl_api_opts(){\n    # arrays can't be exported so have to pass as a string and then split to array\n    if [ -n \"${CURL_OPTS:-}\" ]; then\n        read -r -a CURL_OPTS <<< \"${CURL_OPTS[@]}\" # this @ notation works for both strings and arrays in case a future version of bash do export arrays this should still work\n    else\n        #read -r -a CURL_OPTS <<< \"-sS --fail --connect-timeout 10\"\n        CURL_OPTS=(-sS --fail --connect-timeout 10)\n    fi\n\n    # case insensitive regex matching\n    shopt -s nocasematch\n    # XML by default :-/\n    if ! [[ \"$* ${CURL_OPTS[*]:-}\" =~ Accept: ]]; then\n        CURL_OPTS+=(-H \"Accept: application/json\")\n    fi\n    if ! [[ \"$* ${CURL_OPTS[*]:-}\" =~ Content-Type: ]]; then\n        CURL_OPTS+=(-H \"Content-Type: application/json\")\n    fi\n    # unset to return to default setting for safety to avoid hard to debug changes of behaviour elsewhere\n    shopt -u nocasematch\n    export CURL_OPTS\n}\n\n# useful for cutting down on number of noisy docker tests which take a long time but more importantly\n# cause the CI builds to fail with job logs > 4MB\nci_sample(){\n    local versions\n    versions=\"${*:-}\"\n    # longer time limits on GitHub Workflows than other CI systems like Travis so don't sample, run everything\n    if is_github_workflow; then\n        echo \"$versions\"\n    elif [ -n \"${SAMPLE:-}\" ] || is_CI; then\n        if [ -n \"$versions\" ]; then\n            local a\n            IFS=' ' read -r -a a <<< \"$versions\"\n            local highest_index\n            highest_index=\"${#a[@]}\"\n            local random_index\n            random_index=\"$((RANDOM % highest_index))\"\n            echo \"${a[$random_index]}\"\n        else\n            return 1\n        fi\n    else\n        if [ -n \"$versions\" ]; then\n            echo \"$versions\"\n        fi\n    fi\n    return 0\n}\n\ntrap_cmd(){\n    # shellcheck disable=SC2064,SC2086\n    trap \"$@\" $TRAP_SIGNALS\n}\n\nuntrap(){\n    # shellcheck disable=SC2086\n    trap - $TRAP_SIGNALS\n}\n\nplural(){\n    plural=\"s\"\n    local num\n    num=\"${1:-}\"\n    if [ \"$num\" = 1 ]; then\n        plural=\"\"\n    fi\n}\n\nplural_str(){\n    local parts=(\"$@\")\n    plural ${#parts[@]}\n}\n\n# =================================\n#\n# these functions are too clever and dynamic but save a lot of duplication in nagios-plugins test_*.sh scripts\n#\nprint_debug_env(){\n    echo\n    echo \"Environment for Debugging:\"\n    echo\n    if [ -n \"${VERSION:-}\" ]; then\n        echo \"VERSION: $VERSION\"\n        echo\n    fi\n    if [ -n \"${version:-}\" ]; then\n        echo \"version: $version\"\n        echo\n    fi\n    # multiple name support for MySQL + MariaDB variables\n    for name in \"$@\"; do\n        name=\"$(tr '[:lower:]' '[:upper:]' <<< \"$name\")\"\n        #eval echo \"export ${name}_PORT=$`echo ${name}_PORT`\"\n        # instead of just name_PORT, find all PORTS in environment and print them\n        # while read line to preserve CASSANDRA_PORTS=7199 9042\n        env | grep -E -- \"^$name.*_\" | grep -v -e 'DEFAULT=' -e 'VERSIONS=' | sort | while read -r env_var; do\n            # sed here to quote export CASSANDRA_PORTS=7199 9042 => export CASSANDRA_PORTS=\"7199 9042\"\n            eval echo \"'export $env_var'\" | sed 's/=/=\"/;s/$/\"/'\n        done\n        echo\n    done\n}\n\ntrap_debug_env(){\n    local name\n    name=\"$1\"\n    # stop CI systems from running out of space due to accumulated docker images as that causes build failures\n    if is_CI &&\n       ! type trap_function &>/dev/null &&\n       type docker_image_cleanup &>/dev/null; then\n        trap_function(){\n            # shellcheck disable=SC2317\n            docker_image_cleanup\n        }\n    fi\n    # shellcheck disable=SC2086,SC2154\n    trap 'result=$?; type trap_function >/dev/null 2>/dev/null && trap_function; print_debug_env '\"$*\"'; untrap; exit $result' $TRAP_SIGNALS\n}\n\nis_debug(){\n    [ -n \"${DEBUG:-}\" ]\n}\n\nis_verbose(){\n    [ -n \"${VERBOSE:-}\" ]\n}\n\npass(){\n    read_secret \"password\"\n    export PASSWORD=\"$secret\"\n}\n\nread_secret(){\n    secret=\"\"\n    prompt=\"Enter ${1:-secret value}: \"\n    # doesn't echo any characters to the screen even in commands\n    #stty -echo\n    # this gives stars feedback which is nicer\n    while IFS= read -p \"$prompt\" -r -s -n 1 char; do\n        if [[ \"$char\" == $'\\0' ]]; then\n            break\n        fi\n        prompt='*'\n        secret=\"${secret}${char}\"\n    done\n    #stty echo\n    echo\n    export secret\n}\n\nif is_mac; then\n    awk(){\n        # needed for awk -v IGNORECASE=1 to work for case insensitive regex\n        command gawk \"$@\"\n    }\n    grep(){\n        command ggrep \"$@\"\n    }\n    readlink(){\n        command greadlink \"$@\"\n    }\n    date(){\n        command gdate \"$@\"\n    }\n    sed(){\n        command gsed \"$@\"\n    }\n    xargs(){\n        command gxargs \"$@\"\n    }\nfi\n\n# fails interactive import without this\nfunction run++ () {\n    #if [[ \"$run_count\" =~ ^[[:digit:]]+$ ]]; then\n        ((run_count+=1))\n    #fi\n}\n\nrun(){\n    if [ -n \"${ERRCODE:-}\" ]; then\n        run_fail \"$ERRCODE\" \"$@\"\n    else\n        run++\n        echo \"$@\"\n        \"$@\"\n        # run_fail does it's own hr\n        hr\n    fi\n}\n\nrun_conn_refused(){\n    echo \"checking connection refused:\"\n    ERRCODE=2 run_grep \"Connection refused|Can't connect|Could not connect to|ConnectionClosed\" \"$@\" -H localhost -P \"$wrong_port\"\n}\n\nrun_404(){\n    echo \"checking 404 Not Found:\"\n    ERRCODE=2 run_grep \"404 Not Found\" \"$@\"\n}\n\nrun_timeout(){\n    echo \"checking timeout:\"\n    ERRCODE=3 run_grep \"timed out\" \"$@\"\n}\n\nrun_usage(){\n    echo \"checking usage / parsing:\"\n    ERRCODE=3 run_grep \"usage: \" \"$@\"\n}\n\nrun_output(){\n    local expected_output\n    expected_output=\"$1\"\n    shift\n    run++\n    echo \"$@\"\n    set +e\n    check_output \"$expected_output\" \"$@\"\n    set -e\n    hr\n}\n\nrun_fail(){\n    local expected_exit_code\n    expected_exit_code=\"$1\"\n    shift\n    run++\n    echo \"$@\"\n    set +e\n    \"$@\"\n    # intentionally don't quote $expected_exit_code so that we can pass multiple exit codes through first arg and have them expanded here\n    # shellcheck disable=SC2086\n    check_exit_code $expected_exit_code || exit 1\n    set -e\n    hr\n}\n\nrun_grep(){\n    local egrep_pattern\n    egrep_pattern=\"$1\"\n    shift\n    expected_exit_code=\"${ERRCODE:-0}\"\n    run++\n    echo \"$@\"\n    set +eo pipefail\n    # pytools programs write to stderr, must test this for connection refused type information\n    output=\"$(\"$@\" 2>&1)\"\n    if ! check_exit_code \"$expected_exit_code\"; then\n        echo \"$output\"\n        exit 1\n    fi\n    set -e\n    # this must be egrep -i because (?i) modifier does not work\n    echo \"> | tee /dev/stderr | grep -Eqi '$egrep_pattern'\"\n    echo \"$output\" | tee /dev/stderr | grep -Eqi -- \"$egrep_pattern\"\n    set -o pipefail\n    hr\n}\n\nrun_test_versions(){\n    local name\n    name=\"$1\"\n    local test_func\n    test_func=\"$(tr '[:upper:]' '[:lower:]' <<< \"test_${name/ /_}\")\"\n    local VERSIONS\n    VERSIONS=\"$(tr '[:lower:]' '[:upper:]' <<< \"${name/ /_}_VERSIONS\")\"\n    # shellcheck disable=SC2006\n    local test_versions\n    # shellcheck disable=SC2046,SC2006,SC2116\n    test_versions=\"$(eval ci_sample $`echo \"$VERSIONS\"`)\"\n    local test_versions_ordered\n    test_versions_ordered=\"$test_versions\"\n    if [ -z \"${NO_VERSION_REVERSE:-}\" ]; then\n        # tail -r works on Mac but not Travis CI Ubuntu Trusty\n        # shellcheck disable=SC2119\n        test_versions_ordered=\"$(tr ' ' '\\n' <<< \"$test_versions\" | tac | tr '\\n' ' ')\"\n    fi\n    local start_time\n    start_time=\"$(start_timer \"$name tests\")\"\n    for version in $test_versions_ordered; do\n        version_start_time=\"$(start_timer \"$name test for version:  $version\")\"\n        run_count=0\n        eval \"$test_func\" \"$version\"\n        if [ $run_count -eq 0 ]; then\n            echo \"NO TEST RUNS DETECTED!\" >&2\n            exit 1\n        fi\n        ((total_run_count+=run_count))\n        time_taken \"$version_start_time\" \"$name version '$version' tests completed in\"\n        echo\n    done\n\n    if [ -n \"${NOTESTS:-}\" ]; then\n        print_debug_env \"$name\"\n    else\n        untrap\n        timestamp \"All $name tests succeeded for versions: $test_versions\"\n        echo >&2\n        timestamp \"Total Tests run: $total_run_count\"\n        time_taken \"$start_time\" \"All version tests for $name completed in\"\n        echo\n    fi\n    echo\n}\n\n# examples:\n#\n# #  run: kubectl apply -f file.yaml\n# // run: go run file.go\n# -- run: psql -f file.sql\nparse_run_args(){\n    perl -ne 'if(/^\\s*(#|\\/\\/|--)\\s*run:/){s/^\\s*(#|\\/\\/)\\s*run:\\s*//; print $_; exit}' \"$@\"\n}\n\n# examples:\n#\n# #  stdin: Driving & road trip playlist\nparse_run_stdin(){\n    perl -ne 'if(/^\\s*(#|\\/\\/|--)\\s*stdin:/){s/^\\s*(#|\\/\\/)\\s*stdin:\\s*//; print $_; exit}' \"$@\"\n}\n\n# example:\n#\n# lint: k8s\nparse_lint_hint(){\n    perl -ne 'if(/^\\s*(#|\\/\\/|--)\\s*lint:/){s/^\\s*(#|\\/\\/)\\s*lint:\\s*//; print $_; exit}' \"$@\"\n}\n\n# =================================\n\nstat_bytes(){\n    if is_mac; then\n        stat -f %z \"$@\"\n    else\n        stat -c %s \"$@\"\n    fi\n}\n\ntimestamp(){\n    local ts\n    ts=\"$(date '+%F %T')\"\n    if [ $# -gt 0 ]; then\n        printf \"%s  %s\\n\" \"$ts\" \"$*\" >&2\n    else\n        printf \"%s  \" \"$ts\" >&2\n    fi\n}\ntstamp(){ timestamp \"$@\"; }\n\nseconds_to_hours(){\n    local secs=\"$1\"\n    printf '%d:%02d:%02d\\n' $((secs/3600)) $((secs%3600/60)) $((secs%60))\n}\n\nwarn(){\n    timestamp \"WARNING: $*\"\n}\nwarning(){\n    warn \"$@\"\n}\n\nerror(){\n    timestamp \"ERROR: $*\"\n}\n\nlog(){\n    if is_verbose; then\n        timestamp \"$@\"\n    fi\n}\n\ntrim(){\n    local str=\"$1\"\n    # easier\n    #sed '\n    #    s/^[[:space:]]*//;\n    #    s/[[:space:]]*$//;\n    #' <<< \"$str\"\n\n    # more efficient without process fork to sed\n    #\n    # not using shopt -s extglob because then I have to track it its prior state\n    # from parent/client calling script and restore it\n    #\n    #str=\"${str##+([[:space:]])}\"  # trim leading\n    #str=\"${str%%+([[:space:]])}\"  # trim trailing\n    #\n    # trim leading whitespace\n    str=\"${str#\"${str%%[![:space:]]*}\"}\"\n    # trim trailing whitespace\n    str=\"${str%\"${str##*[![:space:]]}\"}\"\n\n    # strip literal \\n at edges\n    # leading\n    while [[ \"$str\" == '\\n'* ]]; do\n        str=\"${str#\\\\n}\"\n    done\n\n    # trailing\n    while [[ \"$str\" == *'\\n' ]]; do\n        str=\"${str%\\\\n}\"\n    done\n\n    echo \"$str\"\n}\n# used in subshells to capture output so export it\nexport -f trim\n\nclear_current_line(){\n    # Terminal Codes:\n    #\n    # \\r       - position cursor back to column 0\n    # \\033[K   - ESC + [K = clear line\n    #\n    printf \"\\r\\033[K\"\n}\n\nclear_previous_line(){\n    # Terminal Codes:\n    #\n    # \\033[1A  - ESC + [1A = move cursor up one line\n    # \\033[2K  - ESC + [2K = clear entire line\n    # \\r       - position cursor back to column 0\n    #\n    printf \"\\033[1A\\033[2K\\r\"\n}\n\nstart_timer(){\n    tstamp \"Starting $*\n\"\n    date '+%s'\n}\n\ntime_taken(){\n    echo\n    local start_time\n    # workaround if 2>&1 captures message from start_timer and only want final epoch timestamp\n    start_time=\"${1//*[[:space:]]}\"\n    shift\n    local time_taken\n    local msg\n    msg=\"${*:-Completed in}\"\n    tstamp \"Finished\"\n    echo\n    local end_time\n    end_time=\"$(date +%s)\"\n    time_taken=\"$((end_time - start_time))\"\n    echo \"$msg $time_taken secs\"\n    echo\n}\n\n# args may be passed in client code\n# shellcheck disable=SC2120\nstartupwait(){\n    startupwait=\"${1:-30}\"\n    if is_CI; then\n        ((startupwait*=2))\n    fi\n}\n# trigger to set a sensible default if we forget, as it is used\n# as a fallback in when_ports_available and when_url_content below\n# shellcheck disable=SC2119\nstartupwait\n\nnext_available_port(){\n    local local_port=\"${1:-1024}\"\n    local next_port\n    while netstat -lnt | grep -q \":$local_port \"; do\n        next_port=\"$((local_port + 1))\"\n        timestamp \"Local port '$local_port' in use, trying next port '$next_port'\"\n        local_port=\"$next_port\"\n        if [ \"$local_port\" -gt 65535 ]; then\n            die \"ERROR: No local port found available\"\n        fi\n    done\n    echo \"$local_port\"\n}\n\nwhen_pingable(){\n    local host=\"$1\"\n    ping -o \"$host\"\n}\n\nwhen_ports_available(){\n    local max_secs=\"${1:-}\"\n    if ! [[ \"$max_secs\" =~ ^[[:digit:]]+$ ]]; then\n        max_secs=\"$startupwait\"\n    else\n        shift\n    fi\n    local host=\"${1:-}\"\n    local ports=\"${*:2}\"\n    local retry_interval=\"${RETRY_INTERVAL:-1}\"\n    if [ -z \"$host\" ]; then\n        echo \"$FUNCNAME: host \\$2 not set\" >&2\n        exit 1\n    elif [ -z \"$ports\" ]; then\n        echo \"$FUNCNAME: ports \\$3 not set\" >&2\n        exit 1\n    else\n        for port in $ports; do\n            if ! [[ \"$port\" =~ ^[[:digit:]]+$ ]]; then\n                echo \"$FUNCNAME: invalid non-numeric port argument '$port'\" >&2\n                exit 1\n            fi\n        done\n    fi\n    if ! [[ \"$retry_interval\" =~ ^[[:digit:]]+$ ]]; then\n        echo \"$FUNCNAME: invalid non-numeric \\$RETRY_INTERVAL '$retry_interval'\" >&2\n        exit 1\n    fi\n    # Mac nc doesn't have -z switch like Linux GNU version and we can't rely on one being found first in $PATH\n    #local nc_cmd=\"nc -vw $retry_interval $host <<< ''\"\n    #cmd=\"\"\n    #for x in $ports; do\n    #    cmd=\"$cmd $nc_cmd $x &>/dev/null && \"\n    #done\n    #local cmd=\"${cmd% && }\"\n    # shellcheck disable=SC2086\n    plural_str $ports\n    timestamp \"waiting for up to $max_secs secs for port$plural '$ports' to become available, retrying at $retry_interval sec intervals\"\n    #echo \"cmd: ${cmd// \\&\\>\\/dev\\/null}\"\n    local found=0\n    if type -P nc &>/dev/null; then\n        # Mac nc doesn't have -z switch like Linux GNU version\n        nc_opts=\"\"\n        if nc --help 2>&1 | grep -q GNU; then\n            nc_opts=\"-z\"\n        fi\n        try_number=0\n        # special built-in that increments for script runtime, reset to zero exploit it here\n        SECONDS=0\n        # bash will interpolate from string for correct numeric comparison and safer to quote vars\n        while [ \"$SECONDS\" -lt \"$max_secs\" ]; do\n            ((try_number+=1))\n            for port in $ports; do\n                if ! nc -v -w \"$retry_interval\" $nc_opts \"$host\" \"$port\" < /dev/null &>/dev/null; then\n                    timestamp \"$try_number waiting for host '$host' port '$port'\"\n                    sleep \"$retry_interval\"\n                    break\n                fi\n                found=1\n            done\n            if [ $found -eq 1 ]; then\n                break\n            fi\n        done\n        if [ $found -eq 1 ]; then\n            timestamp \"host '$host' port$plural '$ports' available after $SECONDS secs\"\n        else\n            timestamp \"host '$host' port$plural '$ports' still not available after '$max_secs' secs, giving up waiting\"\n            return 1\n        fi\n    else\n        timestamp \"WARNING: nc command not found in \\$PATH, cannot check port availability, skipping port checks, tests may fail due to race conditions on service availability\"\n        timestamp \"sleeping for '$max_secs' secs instead\"\n        sleep \"$max_secs\"\n    fi\n}\n\n# Do not use this on docker containers\n# docker mapped ports still return connection succeeded even when the process mapped to them is no longer listening inside the container!\n# must be the result of docker networking\nwhen_ports_down(){\n    local max_secs=\"${1:-}\"\n    if ! [[ \"$max_secs\" =~ ^[[:digit:]]+$ ]]; then\n        max_secs=\"$startupwait\"\n    else\n        shift\n    fi\n    local host=\"${1:-}\"\n    local ports=\"${*:2}\"\n    local retry_interval=\"${RETRY_INTERVAL:-1}\"\n    if [ -z \"$host\" ]; then\n        echo \"$FUNCNAME: host \\$2 not set\" >&2\n        return 1\n    elif [ -z \"$ports\" ]; then\n        echo \"$FUNCNAME: ports \\$3 not set\" >&2\n        return 1\n    else\n        for port in $ports; do\n            if ! [[ \"$port\" =~ ^[[:digit:]]+$ ]]; then\n                echo \"$FUNCNAME: invalid non-numeric port argument '$port'\" >&2\n                return 1\n            fi\n        done\n    fi\n    if ! [[ \"$retry_interval\" =~ ^[[:digit:]]+$ ]]; then\n        echo \"$FUNCNAME: invalid non-numeric \\$RETRY_INTERVAL '$retry_interval'\" >&2\n        return 1\n    fi\n    #local max_tries=$(($max_secs / $retry_interval))\n    # Mac nc doesn't have -z switch like Linux GNU version\n    nc_opts=\"\"\n    if nc --help 2>&1 | grep -q GNU; then\n        nc_opts=\"-z\"\n    fi\n    local nc_cmd=\"nc -v -w $retry_interval $nc_opts $host < /dev/null\"\n    # shellcheck disable=SC2178\n    local cmd=\"\"\n    for x in $ports; do\n        # shellcheck disable=SC2178\n        cmd=\"$cmd ! $nc_cmd $x &>/dev/null && \"\n    done\n    # shellcheck disable=SC2178\n    cmd=\"${cmd% && }\"\n    # shellcheck disable=SC2086\n    plural_str $ports\n    timestamp \"waiting for up to $max_secs secs for port$plural '$ports' to go down, retrying at $retry_interval sec intervals\"\n    timestamp \"cmd: ${cmd// \\&\\>\\/dev\\/null}\"\n    local down=0\n    if type -P nc &>/dev/null; then\n        #for((i=1; i <= $max_tries; i++)); do\n        try_number=0\n        # special built-in that increments for script runtime, reset to zero exploit it here\n        SECONDS=0\n        # bash will interpolate from string for correct numeric comparison and safer to quote vars\n        while [ \"$SECONDS\" -lt \"$max_secs\" ]; do\n            ((try_number+=1))\n            timestamp \"$try_number trying host '$host' port(s) '$ports'\"\n            if eval \"$cmd\"; then\n                down=1\n                break\n            fi\n            sleep \"$retry_interval\"\n        done\n        if [ $down -eq 1 ]; then\n            timestamp \"host '$host' port$plural '$ports' down after $SECONDS secs\"\n        else\n            timestamp \"host '$host' port$plural '$ports' still not down after '$max_secs' secs, giving up waiting\"\n            return 1\n        fi\n    else\n        timestamp \"WARNING: nc command not found in \\$PATH, cannot check for ports down, skipping port checks, tests may fail due to race conditions on service availability\"\n        timestamp \"sleeping for '$max_secs' secs instead\"\n        sleep \"$max_secs\"\n    fi\n}\n\nwhen_url_content(){\n    local max_secs=\"${1:-}\"\n    if ! [[ \"$max_secs\" =~ ^[[:digit:]]+$ ]]; then\n        max_secs=\"$startupwait\"\n    else\n        shift\n    fi\n    local url=\"${1:-}\"\n    local expected_regex=\"${2:-}\"\n    local args=\"${*:3}\"\n    local retry_interval=\"${RETRY_INTERVAL:-1}\"\n    if [ -z \"$url\" ]; then\n        echo \"$FUNCNAME: url \\$2 not set\" >&2\n        exit 1\n    elif [ -z \"$expected_regex\" ]; then\n        echo \"$FUNCNAME: expected content \\$3 not set\" >&2\n        exit 1\n    fi\n    if ! [[ \"$retry_interval\" =~ ^[[:digit:]]+$ ]]; then\n        echo \"$FUNCNAME: invalid non-numeric \\$RETRY_INTERVAL '$retry_interval'\" >&2\n        exit 1\n    fi\n    #local max_tries=$(($max_secs / $retry_interval))\n    timestamp \"waiting up to $max_secs secs at $retry_interval sec intervals for HTTP interface to come up with expected regex content: '$expected_regex'\"\n    found=0\n    #for((i=1; i <= $max_tries; i++)); do\n    try_number=0\n    # special built-in that increments for script runtime, reset to zero exploit it here\n    SECONDS=0\n    # bash will interpolate from string for correct numeric comparison and safer to quote vars\n    if type -P curl &>/dev/null; then\n        while [ \"$SECONDS\" -lt \"$max_secs\" ]; do\n            ((try_number+=1))\n            timestamp \"$try_number trying $url\"\n            #\n            # tac reads full content to prevent grep closing stdin from causing this curl error:\n            #\n            #   curl: (23) Failure writing output to destination\n            #\n            # ignore tac exit code from breaking the pipefail\n            #\n            # shellcheck disable=SC2086,SC2119\n            if curl -skL --connect-timeout 1 --max-time 5 ${args:-} \"$url\" | { tac || : ; } | grep -Eq -- \"$expected_regex\"; then\n                timestamp \"URL content detected '$expected_regex'\"\n                found=1\n                break\n            fi\n            sleep \"$retry_interval\"\n        done\n        if [ $found -eq 1 ]; then\n            timestamp \"URL content found after $SECONDS secs\"\n        else\n            timestamp \"URL content still not available after '$max_secs' secs, giving up waiting\"\n            return 1\n        fi\n    else\n        timestamp \"WARNING: curl command not found in \\$PATH, cannot check url content, skipping content checks, tests may fail due to race conditions on service availability\"\n        timestamp \"sleeping for '$max_secs' secs instead\"\n        sleep \"$max_secs\"\n    fi\n}\n\nretry(){\n    local max_secs=\"${1:-}\"\n    local max_retries=\"${MAX_RETRIES:-10}\"\n    local retry_interval=\"${RETRY_INTERVAL:-1}\"\n    shift\n    if ! [[ \"$max_secs\" =~ ^[[:digit:]]+$ ]]; then\n        die \"ERROR: non-integer '$max_secs' passed to $FUNCNAME() for \\$1\"\n    fi\n    if ! [[ \"$retry_interval\" =~ ^[[:digit:]]+$ ]]; then\n        die \"$FUNCNAME: invalid non-numeric \\$RETRY_INTERVAL '$retry_interval'\"\n    fi\n    local negate=\"\"\n    expected_return_code=\"${ERRCODE:-0}\"\n    if [ \"$1\" == '!' ]; then\n        negate=1\n        shift\n    fi\n    local cmd=(\"$@\")\n    if [ -z \"$*\" ]; then\n        die \"ERROR: no command passed to $FUNCNAME() for \\$3\"\n    fi\n    #echo \"retrying for up to $max_secs secs at $retry_interval sec intervals:\"\n    try_number=0\n    SECONDS=0\n    while true; do\n        ((try_number+=1))\n        #echo -n \"try $try_number:  \"\n        set +e\n        \"${cmd[@]}\"\n        returncode=$?\n        set -e\n        if [ -n \"$negate\" ]; then\n            if [ $returncode != 0 ]; then\n                RETRY_INFO_MSG=\"$(timestamp \"Command failed after $SECONDS secs\" 2>&1)\"\n                export RETRY_INFO_MSG\n                break\n            fi\n        elif [ \"$returncode\" = \"$expected_return_code\" ]; then\n            RETRY_INFO_MSG=\"$(timestamp \"Command succeeded with expected exit code of $expected_return_code after $SECONDS secs\" 2>&1)\"\n            export RETRY_INFO_MSG\n            break\n        fi\n        if [ \"$try_number\" -gt \"$max_retries\" ]; then\n            timestamp \"FAILED: giving up after $max_retries retries\"\n            return 1\n        fi\n        if [ \"$SECONDS\" -gt \"$max_secs\" ]; then\n            timestamp \"FAILED: giving up after $max_secs secs\"\n            return 1\n        fi\n        if [ -n \"${RETRY_EXPONENTIAL_BACKOFF:-}\" ]; then\n            sleep \"$(( retry_interval * (2 ** try_number) ))\"\n        else\n            sleep \"$retry_interval\"\n        fi\n    done\n}\n\n\ntimeout(){\n    if is_mac; then\n        gtimeout \"$@\"\n    else\n        timeout \"$@\"\n    fi\n}\n\n\nusage(){\n    local args=\"\"\n    local switches=\"\"\n    local description=\"\"\n    if [ -n \"${usage_args:-}\" ]; then\n        args=\"$usage_args\"\n    fi\n    if [ -n \"${usage_switches:-}\" ]; then\n        switches=\"$usage_switches\"\n        switches=\"${switches##[[:space:]]}\"\n        switches=\"${switches%%[[:space:]]}\"\n    fi\n    if [ -n \"${usage_description:-}\" ]; then\n        description=\"$usage_description\n\"\n    fi\n    if [ -n \"$*\" ]; then\n        echo \"$*\" >&2\n        echo >&2\n    fi\n    cat >&2 <<EOF\n$description\nusage: ${0##*/} $args\n\n$switches\n-h --help           Print usage help and exit\nEOF\n    exit 3\n}\n\nmin_args(){\n    local min=\"$1\"\n    shift || :\n    if [ $# -lt \"$min\" ]; then\n        usage \"error: missing arguments\"\n    fi\n}\n\nmax_args(){\n    local max=\"$1\"\n    shift || :\n    if [ $# -gt \"$max\" ]; then\n        usage \"error: too many arguments, expected $max, got $#\"\n    fi\n}\n\nnum_args(){\n    local num=\"$1\"\n    shift || :\n    min_args \"$num\" \"$@\"\n    max_args \"$num\" \"$@\"\n}\n\nhelp_usage(){\n    for arg; do\n        case \"$arg\" in\n            -h|-help|--help)  usage\n                              ;;\n        esac\n    done\n}\n\nany_opt_usage(){\n    for arg; do\n        case \"$arg\" in\n            -*)  usage\n                 ;;\n        esac\n    done\n}\nno_more_opts(){\n    any_opt_usage \"$@\"\n}\n\nno_more_args(){\n    if [ -n \"${1:-}\" ]; then\n        usage \"too many args given:  $*\"\n    fi\n}\n\nno_args(){\n    if [ -n \"${1:-}\" ]; then\n        usage \"args given where none expected:  $*\"\n    fi\n}\n\ncheck_env_defined(){\n    local env=\"$1\"\n    if [ -z \"${!env:-}\" ]; then\n        usage \"\\$$env not defined\"\n    fi\n}\n\nis_yes(){\n    shopt -s nocasematch\n    if [[ \"$1\" =~ ^(y|yes)$ ]]; then\n        shopt -u nocasematch\n        return 0\n    else\n        shopt -u nocasematch\n        return 1\n    fi\n}\n\ncheck_yes(){\n    local answer=\"$1\"\n    if ! is_yes \"$answer\"; then\n        echo \"Aborting...\" >&2\n        exit 1\n    fi\n}\n\nis_int(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^[[:digit:]]+$ ]]\n}\n\nis_float(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^[[:digit:]]+(\\.[[:digit:]]+)?$ ]]\n}\n\nis_bool(){\n    local arg=\"$1\"\n    # intentionally not making this case insensitive in case APIs are touchy about this\n    # calling script can set case matching insensitivity if needed\n    [[ \"$arg\" =~ ^true|false$ ]]\n}\n\n# [[ regex\nis_regex(){\n    local regex=\"$1\"\n    # right side must not be quoted in order to be properly interpreted as regex\n    [[ \"$regex\" =~ $regex ]]\n}\n\nis_port(){\n    local port=\"$1\"\n    if ! is_int \"$port\"; then\n        return 1\n    elif [ \"$port\" -lt 1 ]; then\n        return 1\n    elif [ \"$port\" -gt 65535 ]; then\n        return 1\n    fi\n}\n\nis_url(){\n    local arg=\"$1\"\n    [[ \"$arg\" =~ ^$url_regex$ ]]\n}\n\nexponential(){\n    local int=\"$1\"\n    local max=\"${2:-}\"\n    if ! is_int \"$int\"; then\n        echo \"ERROR: non-integer passed as first arg to exponential() function!\" >&2\n        return 1\n    fi\n    if [ -n \"$max\" ]; then\n        if ! is_int \"$max\"; then\n            echo \"ERROR: non-integer passed as second arg to exponential() function!\" >&2\n            return 1\n        fi\n        if [ \"$int\" -ge \"$max\" ]; then\n            echo \"$max\"\n            return\n        fi\n    fi\n    local result\n    result=\"$((int * 2))\"\n    if [ -n \"$max\" ]; then\n        if [ \"$result\" -gt \"$max\" ]; then\n            result=\"$max\"\n        fi\n    fi\n    echo \"$result\"\n}\n\nparse_export_key_value(){\n    local env_var=\"$1\"\n    env_var=\"${env_var%%#*}\"\n    env_var=\"${env_var##[[:space:]]}\"\n    env_var=\"${env_var##export}\"\n    env_var=\"${env_var##[[:space:]]}\"\n    env_var=\"${env_var%%[[:space:]]}\"\n    # when using this in kubectl_kv_to_secret.sh we may want 'some-thing=ENV_VAR'\n    # to be able to name the Kubernetes secret key, eg. buildkite-agent-token=\"$BUILDKITE_AGENT_TOKEN\"\n    #if ! [[ \"$env_var\" =~ ^[[:alpha:]][[:alnum:]_]+=.+$ ]]; then\n    #    die \"invalid environment key=value argument given: $env_var\"\n    #fi\n    # shellcheck disable=SC2034\n    key=\"${env_var%%=*}\"\n    # shellcheck disable=SC2034\n    value=\"${env_var#*=}\"\n}\n\n# ============================================================================ #\n#                                   JSON utils\n# ============================================================================ #\n\n# extremely poor performance on large 3MB json string from https://updates.jenkins.io/current/update-center.actual.json\n# seems to hang, not sure why yet, avoid and use a simpler test in that case\nis_blank(){\n    local arg=\"${*:-}\"\n    arg=\"${arg##[[:space:]]}\"\n    arg=\"${arg%%[[:space:]]}\"\n    [ -z \"$arg\" ]\n    # or\n    #[[ \"$arg\" =~ ^[[:blank:]]*$ ]]\n}\n\nnot_blank(){\n    ! is_blank \"$*\"\n}\n\nis_null(){\n    is_blank \"${*:-}\" || [ \"$*\" = null ]\n}\n\nnot_null(){\n    ! is_null \"$*\"\n}\n\nhas_error_field(){\n    # shellcheck disable=SC2181\n    if [ $? != 0 ]; then\n        return 0\n    elif [ \"$(jq -r '.error' <<< \"$*\" || :)\" != null ]; then\n        return 0\n    elif [ \"$(jq -r '.errors' <<< \"$*\" || :)\" != null ]; then\n        return 0\n    elif [ \"$(jq -r '.error_description' <<< \"$*\" || :)\" != null ]; then\n        return 0\n    fi\n    return 1\n}\n\ndie_if_error_field(){\n    if [ -z \"$*\" ]; then\n        echo \"no json string passed to die_if_error_field()\" >&2\n        exit 1\n    fi\n    if has_error_field \"$*\"; then\n        echo \"ERROR: $*\" >&2\n        exit 1\n    fi\n}\n\nwarn_if_error_field(){\n    if [ -z \"$*\" ]; then\n        echo \"no json string passed to warn_if_error_field()\" >&2\n        exit 1\n    fi\n    if has_error_field \"$*\"; then\n        echo \"WARNING: $*\" >&2\n    fi\n}\n\n# ==============================\n\n# not wrapping this because in some rare cases we may need to pipe through jq_debug_pipe_dump_slurp instead and this would break (eg. aws_logs_*.sh)\n#jq(){\n#    jq_debug_pipe_dump |\n#    jq \"$@\"\n#}\n\n# pipe debugging filter commands, straight passthrough whether debug mode is enabled or not\njq_debug_pipe_dump(){\n    if [ -n \"${DEBUG:-}\" ]; then\n        data=\"$(cat)\"\n        jq -r . <<< \"$data\" >&2 || :\n        cat <<< \"$data\"\n    else\n        cat  # needed for straight passthrough in non-debug mode\n    fi\n}\n\njq_debug_pipe_dump_slurp(){\n    if [ -n \"${DEBUG:-}\" ]; then\n        data=\"$(cat)\"\n        jq -r -s . <<< \"$data\" >&2 || :\n        call <<< \"$data\"\n    else\n        cat  # needed for straight passthrough in non-debug mode\n    fi\n}\n\njq_is_empty_list(){\n    jq -e 'length == 0' >/dev/null\n}\n# ==============================\n\n# parse a .dat file's column to CSV - used to generate data for embedding into MermaidJS mmd config in:\n#\n# git/git_graph_*_mermaidjs.sh\n#   and\n# github/github_graph_*_mermaidjs.sh\n#\nparse_file_col_to_csv(){\n    local data_file=\"$1\"\n    local field=\"$2\"\n    awk \"{print \\$$field}\" \"$data_file\" |\n    tr '\\n' ',' |\n    sed 's/,/, /g; s/, $//'\n}\n\nfile_modified_in_last_days(){\n    local file=\"$1\"\n    local days=\"$2\"\n    if ! is_int \"$days\"; then\n        die \"Non-integer passed as second arg to file_modified_in_last_days()\"\n    fi\n    if ! [ -f \"$file\" ]; then\n        return 1\n    elif find \"$file\" -mtime -\"$days\" -print | grep -q .; then\n        return 0\n    else\n        local days_ago_in_seconds\n        days_ago_in_seconds=\"$(date -d \"$days days ago\" '+%s')\"\n        if is_mac; then\n            if [ \"$(stat -f '%m' \"$file\")\" -ge \"$days_ago_in_seconds\" ]; then\n                return 0\n            else\n                return 1\n            fi\n        elif [ \"$(stat -c '%Y' \"$file\")\" -ge \"$days_ago_in_seconds\" ]; then\n            return 0\n        else\n            return 1\n        fi\n    fi\n}\n"
  },
  {
    "path": "markdown/markdown_columns_to_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-24 17:45:18 +0700 (Mon, 24 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts text columns separated by whitespace to a Markdown table with vertically aligned column pipe chars\n\nReads from a file or standard input\n\nIf you don't want the first line to be used as the table header and instead generate placeholder 'Column N' headers:\n\n  export NO_HEADER_ROW=1\n\nEg.\n\n    SUBDOMAIN='myproject' domains_subdomains_environments.sh | NO_HEADER_ROW=1 ${0##*/}\n\nOR with a manual input (I've spaced them just for clarity here,\n   the script will align columns even if the spacing is different regardless)\n\n    ${0##*/} <<< EOF\n        Dev             Staging             Production\n        dev.domain.com  staging.domain.com  prod.domain.com\n        dev.domain2.com staging.domain2.com prod.domain2.com\n        dev.domain3.com staging.domain3.com prod.domain3.com\n    EOF\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nif [ $# -eq 1 ]; then\n    if ! [ -f \"$1\" ]; then\n        die \"File not found: $1\"\n    fi\n    log \"${0##*/} Reading from file: $1\"\n    filename_or_stdin=\"$1\"\nelif [ $# -eq 0 ]; then\n    log \"${0##*/} Reading from stdin\"\n    filename_or_stdin=\"-\"\nelse\n    usage \"More than one arg given\"\nfi\n\n# do not want string interpolation messing with awk variables\n# shellcheck disable=SC2016\nawk_script='\nBEGIN {\n    max_cols = 0\n    use_first_row_as_header = (ENVIRON[\"NO_HEADER_ROW\"] == \"\" || ENVIRON[\"NO_HEADER_ROW\"] == \"0\")\n}\n\nNR == 1 {\n    for (i = 1; i <= NF; i++) {\n        len = length($i)\n        if (use_first_row_as_header) {\n            header_name[i] = $i\n            max_width[i] = len\n        } else {\n            if (len > max_width[i]) {\n                max_width[i] = len\n            }\n            data[NR][i] = $i\n        }\n        if (NF > max_cols) max_cols = NF\n    }\n}\n\nNR > 1 || (NR == 1 && !use_first_row_as_header) {\n    for (i = 1; i <= NF; i++) {\n        len = length($i)\n        if (len > max_width[i]) {\n            max_width[i] = len\n        }\n        data[NR][i] = $i\n    }\n    if (NF > max_cols) max_cols = NF\n}\n\nEND {\n    header = \"|\"\n    separator = \"|\"\n    for (i = 1; i <= max_cols; i++) {\n        if (use_first_row_as_header) {\n            header_name[i] = (i in header_name) ? header_name[i] : sprintf(\"Column %d\", i)\n        } else {\n            header_name[i] = sprintf(\"Column %d\", i)\n        }\n        if (length(header_name[i]) > max_width[i]) {\n            max_width[i] = length(header_name[i])\n        }\n        header = header sprintf(\" %-\" max_width[i] \"s |\", header_name[i])\n        separator = separator sprintf(\" %s |\", sprintf(\"%*s\", max_width[i], sprintf(\"%*s\", max_width[i], \"\")))\n    }\n    gsub(/ /, \"-\", separator)\n    print header\n    print separator\n\n    for (row = (use_first_row_as_header ? 2 : 1); row <= NR; row++) {\n        line = \"|\"\n        for (col = 1; col <= max_cols; col++) {\n            value = (col in data[row]) ? data[row][col] : \"\"\n            line = line sprintf(\" %-\" max_width[col] \"s |\", value)\n        }\n        print line\n    }\n}\n'\n\nawk \"$awk_script\" \"$filename_or_stdin\"\n"
  },
  {
    "path": "markdown/markdown_generate_index.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-14 17:53:42 +0100 (Sun, 14 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGenerates a markdown index list from the headings in a given markdown file such as README.md\n\nIf no file is given but README.md is found in the \\$PWD, then uses that\n\nDefaults to 2 space indentation, but if you want to override that to shut up mdl rule, you can:\n\n    export MARKDOWN_INDENTATION=3\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<README.md>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nmarkdown_file=\"${1:-README.md}\"\n\n# prefer 2 but working around this annoying:\n#\n#   https://github.com/markdownlint/markdownlint/blob/main/docs/RULES.md#md007---unordered-list-indentation\n#\nindent_width=\"${MARKDOWN_INDENTATION:-2}\"\n\nif ! is_int \"$indent_width\"; then\n    die \"Indent width must be a valid integer, not: $indent_width\"\nfi\n\nif ! [ -f \"$markdown_file\" ]; then\n    die \"File not found: $markdown_file\"\nfi\n\n# since we now strip ```code``` blocks we must ensure that they match otherwise this code cannot\n# reliably run as it'll result in stripping out valid headings\n\nif [ \"$(( $(grep -c '^[[:space:]]*```' \"$markdown_file\") % 2))\" != 0 ]; then\n    die \"Error - uneven number of code blocks found in file: $markdown_file\"\nfi\n\n# sed strip out ```code``` blocks to avoid # comments inside code blocks from going into the index\n# tail -n +2 takes off the first line which is the header we definitely don't want in the index\n# false positive\n# shellcheck disable=SC2016\nsed '/^[[:space:]]*```/,/^[[:space:]]*```/d' \"$markdown_file\" |\n# strip out oneline html comment because next sed will strip to end of file otherwise\nsed '/<!--.*-->/d;' |\n# strip out <!-- commented out --> sections\nsed '/<!--/,/-->/d' |\ngrep -E -e '^#+[[:space:]]' -e '<h[[:digit:]] id=' |\ntail -n +2 |\n# don't include main headings\n#sed '/^#[[:space:]]/d' |\n# don't include the Index title itself either\nsed '/^[#[:space:]]*Index$/d' |\nwhile read -r line; do\n    # support HTML tags (currently only on same line)\n    if [[ \"$line\" =~ \\<h[[:digit:]][[:space:]]+id= ]]; then\n        level=${line#<h}\n        level=${level%% *}\n        title=${line#*>}\n        title=${title%%</h*}\n        link=${line#*id=\\\"}\n        link=${link%%\\\"*}\n    else\n        level=\"$(grep -Eo '^#+' <<< \"$line\" | tr -d '[:space:]' | wc -c)\"\n        level=\"${level//[[:space:]]}\"\n        #if [ \"$level\" -gt 3 ]; then\n        #    continue\n        #fi\n        #title=\"${line##*# }\"\n        title=\"$(sed '\n            # strip anchor prefixes\n            s/^##* //;\n            # change text links like [ZooKeeper](zookeeper.md) to just ZooKeeper\n            s/\\[\\([^]]\\+\\)\\]([[:alnum:]:\\/#?.=-]\\+)/\\1/g;\n        ' <<< \"$line\")\"\n        # create relative links of just the anchor and not the repo URL prefix, it's more portable\n        link=\"$(\n            sed '\n                s/^#*[[:space:]]*//;\n            ' <<< \"$title\" |\n            tr '[:upper:]' '[:lower:]' |\n            sed '\n                s/[^[:alnum:][:space:]_-]//g;\n                s/[[:space:]-]/-/g;\n                s/^/#/;\n            '\n        )\"\n    fi\n    indentation=$(( indent_width * ( level - 2 ) ))\n    if [ \"$indentation\" -gt 0 ]; then\n        printf \"%${indentation}s\" \" \"\n    fi\n    printf -- \"- [%s](%s)\\n\" \"$title\" \"$link\"\ndone\n"
  },
  {
    "path": "markdown/markdown_list_indentations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-24 15:08:43 +0700 (Mon, 24 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrefixes number of spaces before each list item for comparison to MarkdownLint MD005 inconsistent list indentation error\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<markdown_files>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor filename; do\n    # shellcheck disable=SC2016,SC2001\n    cat -n \"$filename\" |\n    # not expressions - false positive\n    sed '/[[:space:]]```/,/[[:space:]]```/d' |\n    # strip out oneline html comment because next sed will strip to end of file otherwise\n    sed '/<!--.*-->/d;' |\n    # strip out <!-- commented out --> sections\n    sed '/<!--/,/-->/d' |\n    sed \"s/^[[:space:]]*/$filename /\" |\n    # match any line in format:\n    #\n    #   <filename_non_space> <line_number> - list item\n    awk '/^[^[:space:]]+[[:space:]][[:digit:]]+[[:space:]][[:space:]]*(-|[[:digit:]]+\\.)/ {print}' |\n    while read -r line; do\n        sed \"s/^${filename}[[:space:]]\\+[[:digit:]]\\+[[:space:]]//\" <<< \"$line\" |\n        # print the number of spaces before list items\n        awk '{print length($0) - length(substr($0, match($0, /[^ ]/)))}' |\n        #tr -d '\\n'\n        tr '\\n' ' '\n        echo \"$line\"\n    done\ndone\n"
  },
  {
    "path": "markdown/markdown_octocat_github_links.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-04-06 21:53:41 +0800 (Sun, 06 Apr 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts <https://github.com/...> links in Markdown to shorthand links with an OctoCat emoji\nand without the redundant https://github.com/ prefix\n\neg.\n\n    <https://github.com/HariSekhon/Knowledge-Base>\n\n        would become\n\n    [:octocat: HariSekhon/Knowledge-Base](https://github.com/HariSekhon/Knowledge-Base)\n\nThis looks nicer and is shorter when rendered\n\nIf given a markdown file it will in-place edit the file to replace the references\n\nIf passed as string or stdin it will print to stdout\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file_or_string> <file2_or_string2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# replace <https://github.com/owner/repo> with [:octocat: owner/repo](https://github.com/owner/repo)\n#\n# ignores GitHub links that shouldn't be changed like https://github.com/settings/... in my Knowledge-Base repo\n#\n# ignores #readme or similar anchor suffixes and ?tab= parameters in the capture for the link text,\n# but retains them for the link target\nregex_script='\n    /github\\.com\\/settings\\// n;\n    s|<\\(https://github.com/\\([^/]*/[^/?>#]*\\)[^>]*\\)>|[:octocat: \\2](\\1)|g\n'\n\nif [ $# -gt 0 ]; then\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            sed -i \"$regex_script\" \"$arg\"\n        else\n            sed \"$regex_script\" <<< \"$arg\"\n        fi\n    done\nelse\n    warn \"Reading from stdin\"\n    sed \"$regex_script\"\nfi\n"
  },
  {
    "path": "markdown/markdown_replace_index.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-17 03:06:53 +0200 (Sat, 17 Aug 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReplaces the index block of a given markdown file\n\nUses the adjacent script markdown_generate_index.sh\n\nRequires the markdown file to have lines with\n\n<!-- INDEX_START -->\n\n    and\n\n<!-- INDEX_END -->\n\nlines to demark the index block\n\n\nIf no file is given but README.md is found in the \\$PWD, then uses that\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<README.md>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nmarkdown_file=\"${1:-README.md}\"\n\nindex_tmp=\"$(mktemp)\"\n\nmarkdown_tmp=\"$(mktemp)\"\n\nif ! [ -f \"$markdown_file\" ]; then\n    die \"File not found: $markdown_file\"\nfi\n\n# check the tags existing in the markdown file otherwise we can't do anything\nfor x in INDEX_START INDEX_END; do\n    if ! grep -q \"<!--.*$x.*-->\" \"$markdown_file\"; then\n        die \"Markdown file '$markdown_file' is missing the index boundary comment <!--.*$x.*-->\"\n    fi\ndone\n\ntimestamp \"Generating new index for file: $markdown_file\"\n\n\"$srcdir/markdown_generate_index.sh\" \"$markdown_file\" > \"$index_tmp\"\n\ntimestamp \"Replacing index in file: $markdown_file\"\n\nsed -n \"\n    1,/INDEX_START/p\n\n    /INDEX_START/ a\n\n    /INDEX_START/,/INDEX_END/ {\n        /INDEX_START/ {\n            r $index_tmp\n        }\n    }\n\n    /INDEX_END/ i\n\n    /INDEX_END/,$ p\n\" \"$markdown_file\" > \"$markdown_tmp\"\n\n#unalias mv &>/dev/null || :\nmv -f \"$markdown_tmp\" \"$markdown_file\"\n\n#unalias rm &>/dev/null || :\nrm -f \"$index_tmp\"\n\ntimestamp \"Replaced index in file: $markdown_file\"\n"
  },
  {
    "path": "markdown/markdown_replace_links_with_jsdelivr.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-11 17:58:27 -0300 (Wed, 11 Feb 2026)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/github.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReplaces local GitHub repo file links in the given markdown file(s) with JSDelivr CDN links\n\nHandles Markdown and URL links in the format:\n\n[Name](/path)\n\nimg src=\\\"/path\\\"\n\nExplicit URLs are not converted yet\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<markdown_files>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nprocess_markdown_file(){\n    local filename=\"$1\"\n    local dirname\n    local basename\n    dirname=\"$(dirname \"$filename\")\"\n    basename=\"${filename##*/}\"\n    [ -f \"$filename\" ] || die \"File not found: $filename\"\n    pushd \"$dirname\" &>/dev/null\n    if ! is_git_repo; then\n        die \"Not inside a Git Repo - will not be able to determine GitHub repo name to construct JSDelivr URL\"\n    fi\n    owner_repo=\"$(get_github_repo)\"\n    if is_blank \"$owner_repo\"; then\n        die \"Failed to determine GitHub repo for file: $filename\"\n    fi\n    current_branch=\"$(current_branch)\"\n    sed -i \"\n        s|\\\\(img src=[\\\"']\\\\)/|\\\\1https://cdn.jsdelivr.net/gh/$owner_repo@$current_branch/|;\n        s|\\\\([[^\\\\]].*](\\\\)/|\\\\1https://cdn.jsdelivr.net/gh/$owner_repo@$current_branch/|;\n    \" \"$basename\"\n    popd &>/dev/null\n}\n\nif [ $# -eq 0 ]; then\n    process_markdown_file README.md\nelse\n    for filename; do\n        process_markdown_file \"$filename\"\n    done\nfi\n"
  },
  {
    "path": "markdown/markdown_replace_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-09-04 14:42:02 +0200 (Wed, 04 Sep 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\ndefault_url=\"https://raw.githubusercontent.com/HariSekhon/HariSekhon/main/README.md\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReplaces the repos block of a given markdown file\n\nPulls from here by default if no url is specified:\n\n    $default_url\n\nRequires the markdown file to have lines with\n\n<!-- OTHER_REPOS_START -->\n\n    and\n\n<!-- OTHER_REPOS_END -->\n\nlines to demark the repos block\n\n\nIf no file is given but README.md is found in the \\$PWD, then uses that\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<README.md> <url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nmarkdown_file=\"${1:-README.md}\"\nurl=\"${2:-$default_url}\"\n\nother_repos_tmp=\"$(mktemp)\"\nother_repos_tmp2=\"$(mktemp)\"\n\nmarkdown_tmp=\"$(mktemp)\"\n\nif ! [ -f \"$markdown_file\" ]; then\n    die \"File not found: $markdown_file\"\nfi\n\n# check the tags existing in the markdown file otherwise we can't do anything\nfor x in OTHER_REPOS_START OTHER_REPOS_END; do\n    if ! grep -q \"<!--.*$x.*-->\" \"$markdown_file\"; then\n        die \"Markdown file '$markdown_file' is missing the other repos section boundary comment <!--.*$x.*-->\"\n    fi\ndone\n\ntimestamp \"Fetching other repos section for file '$markdown_file' from $url\"\n\nif [ -f \"$url\" ]; then\n    cp \"$url\" \"$other_repos_tmp\"\nelse\n    curl -sS \"$url\" > \"$other_repos_tmp\"\nfi\n\nif [ -z \"$other_repos_tmp\" ]; then\n    die \"URL returned empty content: $url\"\nfi\n\nfor x in REPOS_START REPOS_END; do\n    if ! grep -q \"<!--.*$x.*-->\" \"$other_repos_tmp\"; then\n        die \"URL content is missing the other repos section boundary comment <!--.*$x.*-->\"\n    fi\ndone\n\n# copy out content between REPOS_START and REPOS_END lines\nsed -n '/REPOS_START/,/REPOS_END/{\n    /REPOS_START/,/REPOS_END/{\n        /REPOS_START/b;\n        /REPOS_END/b;\n        p;\n    }\n}' \"$other_repos_tmp\" |\n# strip leading and trailing whitespace lines\nsed '/./,$!d' |\ntac |\nsed '/./,$!d' |\ntac > \"$other_repos_tmp2\"\n\n#unalias mv &>/dev/null || :\nmv -f \"$other_repos_tmp2\" \"$other_repos_tmp\"\n\ntimestamp \"Replacing other other repos section in file: $markdown_file\"\n\nsed -n \"\n    1,/OTHER_REPOS_START/p\n\n    /OTHER_REPOS_START/ a\n\n    /OTHER_REPOS_START/,/OTHER_REPOS_END/ {\n        /OTHER_REPOS_START/ {\n            r $other_repos_tmp\n        }\n    }\n\n    /OTHER_REPOS_END/ i\n\n    /OTHER_REPOS_END/,$ p\n\" \"$markdown_file\" > \"$markdown_tmp\"\n\nmv -f \"$markdown_tmp\" \"$markdown_file\"\n\n#unalias rm &>/dev/null || :\nrm -f \"$other_repos_tmp\"\n\ntimestamp \"Replaced other repos section in file: $markdown_file\"\n"
  },
  {
    "path": "markdown/mdl_list_indentations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-02-24 15:08:43 +0700 (Mon, 24 Feb 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns Markdownlint mdl command and prefixes the spaces count to each offending line of MD005\n(inconsistent list indentations)\n\nWorkaround for:\n\n    https://github.com/DavidAnson/markdownlint/issues/1514\n\nTested on MarkdownLint 0.13.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<markdownlint_args_such_as_filenames>\"\n\nhelp_usage \"$@\"\n\nmdl \"$@\" |\nawk '/: MD005/{print $1}' |\nsed 's/:/ /g' |\nwhile read -r filename line_number; do\n    sed -n \"${line_number} p\" \"$filename\" |\n    awk '{print length($0) - length(substr($0, match($0, /[^ ]/)))}' |\n    tr -d '\\n'\n    echo -n \":$filename:$line_number:\"\n    sed -n \"${line_number} p\" \"$filename\"\ndone\n"
  },
  {
    "path": "media/asciinema.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-14 19:20:36 +0400 (Thu, 14 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Gif from running terminal commands using asciinema and agg\n\nand then opens the resulting gif\n\nIf on Mac uses your \\$BROWSER or Google Chrome since Mac Preview shows the individual gif frames\ninstead of the running gif\n\nAlso resizes the terminal to be a standard 80x25 characters before it begins. Set the environment variable\nNO_RESIZE_TERMINAL to any value to avoid this\n\nThe colours are more vibrant in ttygif.sh but it also records the Terminal title, which this doesn't\n\nSee also adjacent scripts:\n\n    ttygif.sh\n\n    terminalizer.sh\n\n\nRequires asciinema and agg to be installed, attempts to install them via your package manager if not found in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<output.gif>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n# tty-2024-11-14_19:49:39.gif displays as tty-2024-11-14_19/49/39.gif in Finder\n# so use dot separators instead like native Mac screenshots\ngif=\"${1:-asciinema-$(date '+%F_%H.%M.%S').gif}\"\nif ! [[ \"$gif\" =~ \\.gif$ ]]; then\n    gif=\"$gif.gif\"\nfi\nrecording_file=\"/tmp/asciinema.$$.cast\"\n\nif ! type -P asciinema &>/dev/null; then\n    \"$srcdir/../packages/install_packages.sh\" asciinema\nfi\n\nif ! type -P agg &>/dev/null; then\n    \"$srcdir/../packages/install_packages.sh\" agg\nfi\n\nif [ -z \"${NO_RESIZE_TERMINAL:-}\" ]; then\n    resize -s 25 80\nfi\n\nclear\n\ntimestamp \"Now run your commands\"\n\nasciinema rec \"$recording_file\"\n\nagg \"$recording_file\" \"$gif\"\n\nscreenshot_dir=~/Desktop/Screenshots\n\nif [ -d \"$screenshot_dir\" ]; then\n    timestamp \"Moving $gif to $screenshot_dir\"\n    mv -iv \"$gif\" \"$screenshot_dir/\"\n    gif=\"$screenshot_dir/$gif\"\n    echo\nfi\n\ntimestamp \"Gif is now available as: $gif\"\n\nif is_mac; then\n    open -a \"${BROWSER:-Google Chrome}\" \"$gif\"\nelse\n    \"$srcdir/imageopen.sh\" \"$gif\"\nfi\n"
  },
  {
    "path": "media/avi_to_mp4.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-19 19:42:22 +0100 (Tue, 19 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts one or more AVI files given or found recursively under given paths or current directory to mp4 format using ffmpeg\n\nUseful to be able to stream videos to devices like smart TVs that may otherwise not understand the codecs used in the original format\n\nNames the generated files the same except with the '.avi' extension replaced with '.mp4'\n\nSkips files which already have a corresponding adjacent '.mp4' file present to be able to resume partial directory\nconversions, and also removes partially complete files for consistency using bash trapping\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files_or_directories>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ncheck_bin ffmpeg ||\n\"$srcdir/../packages/install_packages.sh\" ffmpeg\n\nSECONDS=0\n\ntime \\\nfor basedir in \"${@:-.}\"; do\n    while read -r filepath; do\n        mp4_filepath=\"${filepath%.avi}.mp4\"\n        if ! [ -s \"$mp4_filepath\" ]; then\n            # shellcheck disable=SC2016\n            trap_cmd 'echo; echo \"removing partially done file:\"; rm -fv \"$mp4_filepath\"; untrap'\n            timestamp \"converting $filepath => $mp4_filepath\"\n            #time nice ffmpeg -i \"$filepath\" \"$mp4_filepath\" < /dev/null  # don't let the ffmpeg command eat the incoming filenames\n            time nice ffmpeg -i \"$filepath\" -vcodec copy -acodec copy -scodec mov_text -movflags +faststart \"$mp4_filepath\" < /dev/null\n            echo\n        fi\n    done < <(find \"$basedir\" -type f -iname '*.avi')\ndone\n\necho\necho \"All conversions completed in $SECONDS secs\"\nuntrap\n"
  },
  {
    "path": "media/avif_to_png.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-06 20:51:19 +0400 (Wed, 06 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts a Avif image to PNG to be usable on websites that don't support Avif images like LinkedIn\n\nOpens the converted PNG image to verify it\n\nRequires ImageMagick to be installed, attempts to install it if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\navif=\"$1\"\n\nshopt -s nocasematch\nif ! [[ \"$avif\" =~ \\.avif$ ]]; then\n    die \"ERROR: file passed does not have an .avif file extension: $avif\"\nfi\n\n# because shopt -s nocasematch doesn't work on bash native string manipulation during testing\n# shellcheck disable=SC2001\npng=\"$(sed 's/\\.avif$//i' <<< \"$avif\").png\"\n\nif [ -f \"$png\" ]; then\n    die \"$png already exists, aborting...\"\nfi\n\nconverted=0\n\nconvert(){\n    if [ -f \"$png\" ]; then\n        die \"$png already exists, aborting...\"\n    fi\n    # can add other tools here later if wanted\n    if type -P magick &>/dev/null; then\n        timestamp \"Converting '$avif' to '$png' using ImageMagick\"\n        magick \"$avif\" \"$png\"\n        return 0\n    fi\n    timestamp \"No tool found installed to convert avif to png\"\n    return 1\n}\n\nif convert; then\n    converted=1\nelse\n    \"$srcdir/../packages/install_packages.sh\" imagemagick ||\n    # can add other tools here later if wanted\n    #\"$srcdir/../packages/install_packages.sh\" anothertool ||\n    die \"Failed to install ImageMagick to convert Avif to PNG\"\n\n    if is_mac; then\n        \"$srcdir/../packages/install_package_if_absent.sh\" libheif\n    fi\n\n    if convert; then\n        converted=1\n    fi\nfi\n\nif [ \"$converted\" = 1 ]; then\n    if [ -f \"$png\" ]; then\n        timestamp \"Conversion complete, file available: $png\"\n    else\n        die \"Conversion failed. Did not find expected file: $png\"\n    fi\nelse\n    die \"Conversion failed\"\nfi\n\ntimestamp \"Opening image: $png\"\n\"$srcdir/imageopen.sh\" \"$png\"\n"
  },
  {
    "path": "media/image_join_vertical.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-28 14:37:45 +0200 (Wed, 28 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nJoins two images vertically after matching their widths so they align correctly\n\nIf the third arg is not given then outputs to joined_image.png\n\nRequires ImageMagick to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<top_image> <bottom_image> [<output_image>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\nmax_args 3 \"$@\"\n\ncheck_bin magick\n\ntop_image=\"$1\"\nbottom_image=\"$2\"\ndefault_output_image=\"joined_image.png\"\noutput_image=\"${3:-$default_output_image}\"\n\ntop_image_width=\"$(magick identify -format \"%w\" \"$top_image\")\"\nbottom_image_width=\"$(magick identify -format \"%w\" \"$bottom_image\")\"\n\nif [ \"$top_image_width\" -lt \"$bottom_image_width\" ]; then\n    resized_image=\"${bottom_image%.*}.resized.${bottom_image##*.}\"\n    timestamp \"Resizing bottom image '$bottom_image' which has width '$bottom_image_width' to match '$top_image' width of '$top_image_width' => $resized_image\"\n    # convert or magick convert is deprecated in ImageMagick version 7 (IMv7) - use just 'magick' instead now\n    #magick convert \"$bottom_image\" -resize \"$top_image_width!\" \"$resized_image\"\n    magick \"$bottom_image\" -resize \"$top_image_width\" \"$resized_image\"\n    bottom_image=\"$resized_image\"\n    echo >&2\nelif [ \"$top_image_width\" -gt \"$bottom_image_width\" ]; then\n    resized_image=\"${top_image%.*}.resized.${top_image##*.}\"\n    timestamp \"Resizing top image '$top_image' which has width '$top_image_width' to match '$bottom_image' width of '$bottom_image_width' => $resized_image\"\n    # deprecated, see comment above\n    #magick convert \"$top_image\" -resize \"$bottom_image_width!\" \"$resized_image\"\n    magick \"$top_image\" -resize \"$bottom_image_width\" \"$resized_image\"\n    top_image=\"$resized_image\"\n    echo >&2\nelif [ \"$top_image_width\" -eq \"$bottom_image_width\" ]; then\n    timestamp \"Image widths already match, joining as is\"\n    echo >&2\nelse\n    die \"ERROR: logic error, please check code\"\nfi\n\nif [ \"$output_image\" != \"$default_output_image\" ] &&\n   [ -e \"$output_image\" ]; then\n    die \"ERROR: output image '$output_image' already exists, not overwriting for safety\"\nfi\n\ntimestamp \"Joining top image '$top_image' and bottom image '$bottom_image' into output image '$output_image'\"\n# deprecated, see comment above\n#magick convert \"$top_image\" \"$bottom_image\" -append \"$output_image\"\nmagick \"$top_image\" \"$bottom_image\" -append \"$output_image\"\necho >&2\ntimestamp \"Stacked image created: $output_image\"\necho >&2\n\ntimestamp \"Opening image: $output_image\"\n\"$srcdir/imageopen.sh\" \"$output_image\"\n"
  },
  {
    "path": "media/image_reduce_quality.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-12-24 01:31:35 -0600 (Wed, 24 Dec 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShrinks an image size by reducing its quality (default to 80%) to be able to upload it against limits on some websites\n\nQuickly written to be able to upload a 4.2MB passport pic to the Copa airline flight to Panama as its limit was 4MB\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<iamge_file> [<percentage_to_reduce_to>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\ninput_image=\"$1\"\npercentage=\"${2:-80%}\"\npercentage=\"${percentage%%%}\"\n\nif ! is_int \"$percentage\"; then\n    die \"Second arg for percentage must be an integer\"\nfi\n\nif [ \"$percentage\" -lt 2 ] || [ \"$percentage\" -gt 98 ]; then\n    die \"Percentage must be between 2 and 98 %\"\nfi\n\noutput_image=\"${input_image%.*}.quality.$percentage%.${input_image##*.}\"\n\ntimestamp \"Reducing image quality of '$input_image' to $percentage% => '$output_image'\"\necho >&2\n\nmagick \"$input_image\" -quality \"$percentage\" \"$output_image\"\n\ntimestamp \"Before vs After:\"\necho >&2\n\n{\nmagick identify \"$input_image\"\nmagick identify \"$output_image\"\n} | column -t\n\n\"$srcdir/imageopen.sh\" \"$output_image\"\n"
  },
  {
    "path": "media/image_shrink.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-12-24 01:31:35 -0600 (Wed, 24 Dec 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShrinks an image by resizing it (default 50%) to be able to upload it against limits on some websites\n\nQuickly written to be able to upload a 4.2MB passport pic to the Copa airline flight to Panama as its limit was 4MB\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<iamge_file> [<percentage_to_shrink_to>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\nmax_args 2 \"$@\"\n\ninput_image=\"$1\"\npercentage=\"${2:-50%}\"\npercentage=\"${percentage%%%}\"\n\nif ! is_int \"$percentage\"; then\n    die \"Second arg for percentage must be an integer\"\nfi\n\nif [ \"$percentage\" -lt 2 ] || [ \"$percentage\" -gt 98 ]; then\n    die \"Percentage must be between 2 and 98 %\"\nfi\n\noutput_image=\"${input_image%.*}.size.$percentage%.${input_image##*.}\"\n\ntimestamp \"Shrinking image '$input_image' to $percentage% => '$output_image'\"\necho >&2\n\nmagick \"$input_image\" -resize \"$percentage\"% \"$output_image\"\n\ntimestamp \"Before vs After:\"\necho >&2\n\n{\nmagick identify \"$input_image\"\nmagick identify \"$output_image\"\n} | column -t\n\n\"$srcdir/imageopen.sh\" \"$output_image\"\n"
  },
  {
    "path": "media/image_to_png.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-06 20:51:19 +0400 (Wed, 06 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts an image to PNG to be usable on websites that don't support the original image format,\nsuch as LinkedIn, Medium or Reddit\n\nOpens the converted PNG image to verify it\n\nRequires ImageMagick to be installed, attempts to install it if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\n# because shopt -s nocasematch doesn't work on bash native string manipulation during testing\n# shellcheck disable=SC2001\npng=\"$(sed 's/\\.[^.]\\+$//i' <<< \"$image\").png\"\n\nif [ -f \"$png\" ]; then\n    die \"$png already exists, aborting...\"\nfi\n\nconverted=0\n\nconvert(){\n    if [ -f \"$png\" ]; then\n        die \"$png already exists, aborting...\"\n    fi\n    # can add other tools here later if wanted\n    if type -P magick &>/dev/null; then\n        timestamp \"Converting '$image' to '$png' using ImageMagick\"\n        magick \"$image\" \"$png\"\n        return 0\n    fi\n    timestamp \"No tool found installed to convert image to png\"\n    return 1\n}\n\nif convert; then\n    converted=1\nelse\n    \"$srcdir/../packages/install_packages.sh\" imagemagick ||\n    # can add other tools here later if wanted\n    #\"$srcdir/../packages/install_packages.sh\" anothertool ||\n    die \"Failed to install ImageMagick to convert image to PNG\"\n\n    if is_mac; then\n        \"$srcdir/../packages/install_package_if_absent.sh\" libheif\n    fi\n\n    if convert; then\n        converted=1\n    fi\nfi\n\nif [ \"$converted\" = 1 ]; then\n    if [ -f \"$png\" ]; then\n        timestamp \"Conversion complete, file available: $png\"\n    else\n        die \"Conversion failed. Did not find expected file: $png\"\n    fi\nelse\n    die \"Conversion failed\"\nfi\n\ntimestamp \"Opening image: $png\"\n\"$srcdir/imageopen.sh\" \"$png\"\n"
  },
  {
    "path": "media/image_trim_pixels.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-02 08:36:02 +0700 (Thu, 02 Jan 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTrims N pixels off one of the sides of an image\n\nUseful to tweak screenshots before sharing them\n\nCreates a new file and automatically opens the new image to check the result\n\nFirst arg is the image file to edit\n\nSecond arg picks a side - options are one of:\n\n    - top\n    - bottom\n    - left\n    - right\n\nThird arg is the number of pixels to trim off (default: 1)\n\n\nRequires ImageMagick to be installed, attempts to install it if not already installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image> <side> [<number_of_pixels>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\ncheck_bin magick\n\nimage=\"$1\"\nside=\"$2\"\npixels=\"${3:-1}\"\n\n# name the new file the same as the old file but with _trimmed suffixed just before the file extension\n#\n#   eg. my-file.png => my-file_trimmed.png\n#\noutput_image=\"${image%.*}_trimmed.${image##*.}\"\n\nif ! is_int \"$pixels\"; then\n    usage \"Pixels must be an integer\"\nfi\n\nif ! type -P magick &>/dev/null; then\n    \"$srcdir/../packages/install_packages.sh\" imagemagick\nfi\n\ntimestamp \"Input image: $image\"\ntimestamp \"Output image: $output_image\"\n\nif [ \"$side\" = top ]; then\n    timestamp \"Trimming $pixels pixels off the top\"\n    # '-crop +0+2' tells ImageMagick to leave the width (0), but shift the image down by 2 pixels (+2),\n    # effectively trimming 2 pixels from the top\n    #\n    # '+repage' resets the virtual canvas metadata, so it doesn't retain the original canvas size.\n    magick \"$image\" -crop +0+\"$pixels\" +repage \"$output_image\"\nelif [ \"$side\" = bottom ]; then\n    timestamp \"Trimming $pixels pixels off the bottom\"\n    magick \"$image\" -crop +0+\"$pixels\" +repage \"$output_image\"\n    magick \"$image\" -gravity South -chop 0x\"$pixels\" \"$output_image\"\nelif [ \"$side\" = right ]; then\n    timestamp \"Trimming $pixels pixels off the right\"\n    # '-gravity East' tells it to keep to the left - like driving in the UK!\n    magick \"$image\" -gravity East -chop \"$pixels\"x0 \"$output_image\"\nelif [ \"$side\" = left ]; then\n    timestamp \"Trimming $pixels pixels off the left\"\n    magick \"$image\" -gravity West -chop \"$pixels\"x0 \"$output_image\"\nelse\n    usage \"Invalid side selected, must be one of: top, bottom, left, right\"\nfi\n\ntimestamp \"Opening image: $output_image\"\n\"$srcdir/imageopen.sh\" \"$output_image\"\n"
  },
  {
    "path": "media/imageopen.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-06 18:59:32 +0100 (Tue, 06 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens the given image file using whatever available tool is found on Linux or Mac\n\nUsed by the following scripts:\n\n    ./image_join_stack.sh - to automatically open the stacked image\n\n    ../git/git_graph_commit_history_gnuplot.sh - to automatically open the generated bar chart images\n    ../git/git_graph_commit_history_mermaidjs.sh\n    ../git/git_graph_commit_times_gnuplot.sh\n    ../git/git_graph_commit_times_mermaidjs.sh\n    ../github/github_graph_commit_times_gnuplot.sh\n    ../github/github_graph_commit_times_mermaidjs.sh\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<image_file>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nimage=\"$1\"\n\n# Will be tried in this order\nlinux_commands=(\n    xdg-open\n    gnome-open\n    eog\n    feh\n    display\n    gthumb\n    sxiv\n)\n\nif is_mac; then\n    open \"$image\"\nelse  # assume Linux\n    found=0\n    for linux_command in \"${linux_commands[@]}\"; do\n        if type -P \"$linux_command\" &>/dev/null; then\n            found=1\n            \"$linux_command\" \"$image\" &\n            break\n        fi\n    done\n    if [ \"$found\" != 1 ]; then\n        die \"ERROR: none of the following Linux commands to open image were found: ${linux_commands[*]}\"\n    fi\nfi\n"
  },
  {
    "path": "media/mkv_to_mp4.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-19 19:42:22 +0100 (Tue, 19 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts any MKV files given or found recursively under given paths or current directory to mp4 format using ffmpeg\n\nUseful to be able to stream videos to devices like smart TVs that may otherwise not understand the codecs used in the original format\n\nNames the generated files the same except with the '.mkv' extension replaced with '.mp4'\n\nSkips files which already have a corresponding adjacent '.mp4' file present to be able to resume partial directory\nconversions, and also removes partially complete files for consistency using bash trapping\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<files_or_directories>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ncheck_bin ffmpeg ||\n\"$srcdir/../packages/install_packages.sh\" ffmpeg\n\nSECONDS=0\n\ntime \\\nfor basedir in \"${@:-.}\"; do\n    while read -r filepath; do\n        mp4_filepath=\"${filepath%.mkv}.mp4\"\n        #if [ -n \"${FORCE_OVERWRITE:-}\" ] ||\n        if ! [ -s \"$mp4_filepath\" ]; then\n            # shellcheck disable=SC2016\n            trap_cmd 'echo; echo \"removing partially done file:\"; rm -fv \"$mp4_filepath\"; untrap'\n            timestamp \"converting $filepath => $mp4_filepath\"\n            if [ -n \"${QUICK:-}\" ]; then\n                time nice ffmpeg -i \"$filepath\" -vcodec copy -acodec copy -scodec mov_text -movflags +faststart \"$mp4_filepath\" < /dev/null  # don't let the ffmpeg command eat the incoming filenames\n                # -sn disabled subtitle stream\n                #time nice ffmpeg -i \"$filepath\" -vcodec copy -acodec copy -sn -movflags +faststart \"$mp4_filepath\" < /dev/null  # don't let the ffmpeg command eat the incoming filenames\n            else\n                # the default encoder doesn't work on TV for some stuff\n                time nice ffmpeg -i \"$filepath\" \"$mp4_filepath\" < /dev/null  # don't let the ffmpeg command eat the incoming filenames\n\n                # don't copy codec as is but transcode into something your TV can understand / pause / forward\n                #time nice ffmpeg -i \"$filepath\" -c:v libx264 -movflags +faststart \"$mp4_filepath\" < /dev/null\n\n                # none of this worked either\n                #time nice ffmpeg -i \"$filepath\" \\\n                #    -vcodec mpeg4 \\\n                #    -acodec aac \\\n                #    -sn \\\n                #    -movflags +faststart \\\n                #    \"$mp4_filepath\" < /dev/null\n                # -vcodec libxvid # really pixelated even at 1GB -> 1GB conversion\n                # -vcodec mpeg4   # works but extremely pixelated\n                #   -qscale:v 17    # even with this, and the files come out huge, bigger than MKV originals\n                #-acodec mp3   # aac is native and better quality\n                #\n                # none of these work on my TV\n                #-vcodec libx264\n                #-vcodec mpeg2 # not found - must be too old and not compiled in\n            fi\n            echo >&2\n        fi\n    done < <(find \"$basedir\" -type f -iname '*.mkv')\ndone\n\necho >&2\ntimestamp \"All conversions completed in $SECONDS secs\"\nuntrap\n"
  },
  {
    "path": "media/mp3_set_album.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-21 11:36:49 +0100 (Tue, 21 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/mp3.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd / Modify album metadata across all MP3 files in the given directories to group albums or audiobooks for Mac's Books.app\n\n$mp3_usage_behaviour_msg\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"album name\\\" [<dir1> <dir2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_bin id3v2\n\nalbum=\"$1\"\n\nshift || :\n\n# used to pipe file list inline which is more comp sci 101 correct but that could create a race condition on second\n# evaluation of file list changing after confirmation prompt, and RAM is cheap, so better to use a static list of files\n# stored in ram and operate on that since it'll never be that huge anyway\n\nmp3_files=\"$(get_mp3_files \"${@:-$PWD}\")\"\n\necho \"List of MP3 files to set album = '$album':\"\necho\necho \"$mp3_files\"\necho\n\nread -r -p \"Are you happy to set the album metadata on all of the above mp3 files to '$album'? (y/N) \" answer\n\ncheck_yes \"$answer\"\n\necho\n\nwhile read -r mp3; do\n    echo \"setting album '$album' on '$mp3'\"\n    id3v2 --album \"$album\" \"$mp3\"\ndone <<< \"$mp3_files\"\n"
  },
  {
    "path": "media/mp3_set_artist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-21 11:36:49 +0100 (Tue, 21 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC2154\n. \"$srcdir/lib/mp3.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd / Modify artist metadata across all MP3 files in the given directories to edit albums or group audiobooks for Mac's Books.app\n\n$mp3_usage_behaviour_msg\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"artist name\\\" [<dir1> <dir2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_bin id3v2\n\nartist=\"$1\"\n\nshift || :\n\n# used to pipe file list inline which is more comp sci 101 correct but that could create a race condition on second\n# evaluation of file list changing after confirmation prompt, and RAM is cheap, so better to use a static list of files\n# stored in ram and operate on that since it'll never be that huge anyway\n\nmp3_files=\"$(get_mp3_files \"${@:-$PWD}\")\"\n\necho \"List of MP3 files to set artist = '$artist':\"\necho\necho \"$mp3_files\"\necho\n\nread -r -p \"Are you happy to set the artist metadata on all of the above mp3 files to '$artist'? (y/N) \" answer\n\ncheck_yes \"$answer\"\n\necho\n\nwhile read -r mp3; do\n    echo \"setting artist '$artist' on '$mp3'\"\n    id3v2 --artist \"$artist\" \"$mp3\"\ndone <<< \"$mp3_files\"\n"
  },
  {
    "path": "media/mp3_set_track_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-21 11:36:49 +0100 (Tue, 21 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/mp3.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd / Modify track name metadata to match MP3 filenames for all MP3s in the given directories to improve display appearance when playing\n\nThe track name metadata is taken to be the basename of the MP3 file without the extension\n\n$mp3_usage_behaviour_msg\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir1> <dir2> ...]\"\n\nhelp_usage \"$@\"\n\ncheck_bin id3v2\n\nshift || :\n\n# used to pipe file list inline which is more comp sci 101 correct but that could create a race condition on second\n# evaluation of file list changing after confirmation prompt, and RAM is cheap, so better to use a static list of files\n# stored in ram and operate on that since it'll never be that huge anyway\n\nmp3_files=\"$(get_mp3_files \"${@:-$PWD}\")\"\n\necho \"List of MP3 files => track names to be set:\"\n\necho\n\nshopt -s nocasematch\n\nmp3_regex='([^/]+).mp3$'\n\nget_track_name(){\n    local mp3=\"$1\"\n    local artist\n    #track_name=\"$mp3\"\n    #track_name=\"${track_name##*/}\"\n    #track_name=\"${track_name%.mp3}\"\n    #track_name=\"${track_name%.MP3}\"\n    # case insensitive matching without subshelling\n    if [[ \"$mp3\" =~ $mp3_regex ]]; then\n        track_name=\"${BASH_REMATCH[1]}\"\n    else\n        die \"failed to regex match filename '$mp3' to generate track name\"\n    fi\n    artist=\"$(id3v2 -l \"$mp3\" |\n        grep '[[:space:]]Artist:[[:space:]]' | sed 's/.*[[:space:]]Artist:[[:space:]]*//; s/[[:space:]]*$//'\n    )\"\n    track_name=\"${track_name#$artist}\"\n    track_name=\"${track_name#[[:space:]]*-[[:space:]]*}\"\n    echo \"$track_name\"\n}\n\nwhile read -r mp3; do\n    track_name=\"$(get_track_name \"$mp3\")\"\n    echo \"'$mp3' => '$track_name'\"\ndone <<< \"$mp3_files\"\n\necho\n\nread -r -p \"Are you happy to set the track name metadata on all of the above mp3 files? (y/N) \" answer\n\ncheck_yes \"$answer\"\n\necho\n\nwhile read -r mp3; do\n    track_name=\"$(get_track_name \"$mp3\")\"\n    echo \"setting track name metadata on '$mp3' to '$track_name'\"\n    id3v2 --song \"$track_name\" \"$mp3\"\ndone <<< \"$mp3_files\"\n"
  },
  {
    "path": "media/mp3_set_track_order.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-21 11:36:49 +0100 (Tue, 21 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/mp3.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd / Modify track number metadata across all MP3 files in the given directories to restructure audiobooks to be contiguous for Mac's Books.app\n\n$mp3_usage_behaviour_msg\n\nMP3 filenames should be in lexical order before running this\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir1> <dir2> ...]\"\n\nhelp_usage \"$@\"\n\ncheck_bin id3v2\n\n# used to pipe file list inline which is more comp sci 101 correct but that could create a race condition on second\n# evaluation of file list changing after confirmation prompt, and RAM is cheap, so better to use a static list of files\n# stored in ram and operate on that since it'll never be that huge anyway\n\nmp3_files=\"$(get_mp3_files \"${@:-$PWD}\")\"\n\necho \"List of MP3 files and their metadata track ordering:\"\n\necho\n\ntotal=0\n\nwhile read -r mp3; do\n    [ -n \"$mp3\" ] || continue\n    ((total+=1))\ndone <<< \"$mp3_files\"\n\n{\n    position=0;\n    while read -r mp3; do\n        [ -n \"$mp3\" ] || continue\n        ((position+=1))\n        printf '%s\\t%s\\n' \"$position/$total\" \"$mp3\"\n    done\n} <<< \"$mp3_files\"\n\necho\n\nread -r -p 'Are you happy with this track metadata ordering? (y/N) ' answer\n\ncheck_yes \"$answer\"\n\necho\n\n{\n    position=0;\n    while read -r mp3; do\n        [ -n \"$mp3\" ] || continue\n        ((position+=1))\n        echo \"setting track order $position/$total on '$mp3'\"\n        id3v2 --track \"$position/$total\" \"$mp3\"\n    done\n} <<< \"$mp3_files\"\n"
  },
  {
    "path": "media/svg_to_png.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-06 12:20:08 +0300 (Sun, 06 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts an SVG image to PNG to be usable on websites that don't support SVG images like LinkedIn, Medium or Reddit\n\nOpens the converted PNG image to verify it\n\nRequires one of Inkscape, ImageMagik or rsvg-convert - will attempt to use whichever is already installed\nor install one of them if none are found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nsvg=\"$1\"\n\nshopt -s nocasematch\nif ! [[ \"$svg\" =~ \\.svg$ ]]; then\n    die \"ERROR: file passed does not have an .svg file extension: $svg\"\nfi\n\n# because shopt -s nocasematch doesn't work on bash native string manipulation during testing\n# shellcheck disable=SC2001\npng=\"$(sed 's/\\.svg$//i' <<< \"$svg\").png\"\n\nconverted=0\n\nconvert(){\n    if [ -f \"$png\" ]; then\n        die \"$png already exists, aborting...\"\n    fi\n    if type -P magick &>/dev/null; then\n        timestamp \"Converting '$svg' to '$png' using ImageMagick\"\n        magick \"$svg\" \"$png\"\n        return 0\n    elif rsvg-convert &>/dev/null; then\n        timestamp \"Converting '$svg' to '$png' using rsvg-convert\"\n        rsvg-convert \"$svg\" -o \"$png\"\n        return 0\n    # heavy, try it last\n    elif type -P inkscape &>/dev/null; then\n        timestamp \"Converting '$svg' to '$png' using Inkscape\"\n        inkscape \"$svg\" --export-filename=\"$png\"\n        return 0\n    fi\n    return 1\n}\n\nif convert; then\n    converted=1\nelse\n    \"$srcdir/../packages/install_packages.sh\" imagemagick ||\n    # package on Debian / Ubuntu\n    \"$srcdir/../packages/install_packages.sh\" rsvg-convert ||\n    \"$srcdir/../packages/install_packages.sh\" inkscape ||\n    die \"Failed to install any of the usual tools to convert SVG to PNG\"\n\n    if convert; then\n        converted=1\n    fi\nfi\n\nif [ \"$converted\" = 1 ]; then\n    if [ -f \"$png\" ]; then\n        timestamp \"Conversion complete, file available: $png\"\n    else\n        die \"Conversion failed. Did not find expected file: $png\"\n    fi\nelse\n    die \"Conversion failed\"\nfi\n\ntimestamp \"Opening image: $png\"\n\"$srcdir/imageopen.sh\" \"$png\"\n"
  },
  {
    "path": "media/terminalizer.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-14 19:20:36 +0400 (Thu, 14 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Gif from running terminal commands using Terminalizer\n\nand then opens the resulting gif\n\nTerminalizer is highly configurable but extremely slow to generate the gif compared to the adjacent\nttygif.sh or asciinema.sh scripts\n\nIf on Mac uses your \\$BROWSER or Google Chrome since Mac Preview shows the individual gif frames\ninstead of the running gif\n\nAlso resizes the terminal to be a standard 80x25 characters before it begins. Set the environment variable\nNO_RESIZE_TERMINAL to any value to avoid this\n\nThe colours are more vibrant in ttygif.sh but it also records the Terminal title, which this doesn't\n\nSee also faster adjacent scripts:\n\n    ttygif.sh\n\n    asciinema.sh\n\n\nRequires terminalizer to be installed, attempts to install it via NPM if not found in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<output.gif>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n# tty-2024-11-14_19:49:39.gif displays as tty-2024-11-14_19/49/39.gif in Finder\n# so use dot separators instead like native Mac screenshots\ngif=\"${1:-terminalizer-$(date '+%F_%H.%M.%S').gif}\"\nif ! [[ \"$gif\" =~ \\.gif$ ]]; then\n    gif=\"$gif.gif\"\nfi\n\nrecording_file=\"/tmp/terminalizer.$$\"\n\nif ! type -P terminalizer &>/dev/null; then\n    if ! type -P npm &>dev/null; then\n        \"$srcdir/../packages/install_packages.sh\" terminalizer\n    fi\n    npm install terminalizer\nfi\n\nif [ -z \"${NO_RESIZE_TERMINAL:-}\" ]; then\n    resize -s 25 80\nfi\n\nclear\n\ntimestamp \"Now run your commands\"\n\nterminalizer record \"$recording_file\" --skip-sharing\n\nterminalizer render \"$recording_file\" -o \"$gif\"\n\nscreenshot_dir=~/Desktop/Screenshots\n\nif [ -d \"$screenshot_dir\" ]; then\n    timestamp \"Moving $gif to $screenshot_dir\"\n    mv -iv \"$gif\" \"$screenshot_dir/\"\n    gif=\"$screenshot_dir/$gif\"\n    echo\nfi\n\ntimestamp \"Gif is now available as: $gif\"\n\nif is_mac; then\n    open -a \"${BROWSER:-Google Chrome}\" \"$gif\"\nelse\n    \"$srcdir/imageopen.sh\" \"$gif\"\nfi\n"
  },
  {
    "path": "media/ttygif.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-11-14 19:20:36 +0400 (Thu, 14 Nov 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Gif from running terminal commands using ttyrec and ttygif\n\nand then opens the resulting gif\n\nIf on Mac uses your \\$BROWSER or Google Chrome since Mac Preview shows the individual gif frames\ninstead of the running gif\n\nAlso resizes the terminal to be a standard 80x25 characters before it begins. Set the environment variable\nNO_RESIZE_TERMINAL to any value to avoid this\n\nSee also adjacent scripts:\n\n    asciinema.sh\n\n    terminalizer.sh\n\n\nRequires ttygif to be installed, attempts to install it via your package manager if not found in \\$PATH\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<output.gif>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n# tty-2024-11-14_19:49:39.gif displays as tty-2024-11-14_19/49/39.gif in Finder\n# so use dot separators instead like native Mac screenshots\ngif=\"${1:-tty-$(date '+%F_%H.%M.%S').gif}\"\nif ! [[ \"$gif\" =~ \\.gif$ ]]; then\n    gif=\"$gif.gif\"\nfi\n\nttyrec_recording_file=\"/tmp/ttyrec.$$\"\n\nif ! type -P ttygif &>/dev/null; then\n    \"$srcdir/../packages/install_packages.sh\" ttygif\nfi\n\nif [ -z \"${NO_RESIZE_TERMINAL:-}\" ]; then\n    resize -s 25 80\nfi\n\nclear\n\n# Makes the PS1 prompt too long which eats into the screen recording terminal space and makes it wrap\n# Instead of this, just move the tty.gif at the end\n#if [ -d ~/Desktop/Screenshots ]; then\n#    timestamp \"Switching to ~/Desktop/Screenshots directory\"\n#    cd ~/Desktop/Screenshots\n#fi\n\ntimestamp \"Now run your commands\"\n\nttyrec \"$ttyrec_recording_file\"\n\n#export TTYGIF_DEBUG=1\n\nttygif \"$ttyrec_recording_file\" -f  # -f includes the terminal window border instead of leaving it half height cut off\n\n# the default filename created is tty.gif\nmv -iv tty.gif \"$gif\"\n\nscreenshot_dir=~/Desktop/Screenshots\n\nif [ -d \"$screenshot_dir\" ]; then\n    timestamp \"Moving $gif to $screenshot_dir\"\n    mv -iv \"$gif\" \"$screenshot_dir/\"\n    gif=\"$screenshot_dir/$gif\"\n    echo\nfi\n\ntimestamp \"Gif is now available as: $gif\"\n\nif is_mac; then\n    open -a \"${BROWSER:-Google Chrome}\" \"$gif\"\nelse\n    \"$srcdir/imageopen.sh\" \"$gif\"\nfi\n"
  },
  {
    "path": "media/video_to_720p_mp4.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-19 19:42:22 +0100 (Tue, 19 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts one or more video files to 720p mp4 format using ffmpeg\n\nUseful to make good trade-off of quality vs size for social media sharing\n\nNames the generated files the same except with the file extension replaced with '.720p.mp4'\n\nSkips files which already have a corresponding adjacent '.720.mp4' file for safety\n\nWill fail if given a video that is already below that resolution\n\nRemoves partially complete files for consistency using bash trapping\n\nInstalls ffmpeg using OS package manager if not already installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<video_files>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nif ! type -P ffmpeg &>/dev/null; then\n    timestamp \"ffmpeg not found in \\$PATH, attempting to install...\"\n    echo\n    \"$srcdir/../packages/install_packages.sh\" ffmpeg\n    echo\nfi\n\ncheck_bin ffmpeg\n\nSECONDS=0\n\ntime \\\nfor filepath in \"$@\"; do\n    mp4_filepath=\"${filepath%.*}.720p.mp4\"\n    if [ -s \"$mp4_filepath\" ]; then\n        timestamp \"File already exists, skipping: $mp4_filepath\"\n    else\n        # shellcheck disable=SC2016\n        trap_cmd 'echo; echo \"removing partially done file:\"; rm -fv \"$mp4_filepath\"; untrap'\n        timestamp \"converting $filepath => $mp4_filepath\"\n        time nice ffmpeg -i \"$filepath\" -vf \"scale=-1:720\" -c:v libx264 -crf 23 -preset medium -c:a copy -movflags +faststart \"$mp4_filepath\"\n        echo >&2\n    fi\n    \"$srcdir/vidopen.sh\" \"$mp4_filepath\"\ndone\n\necho >&2\ntimestamp \"All conversions completed in $SECONDS secs\"\nuntrap\n"
  },
  {
    "path": "media/vidopen.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-01-26 08:01:28 +0700 (Sun, 26 Jan 2025)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens the given video file using whatever available tool is found on Linux or Mac\n\nOn Mac, the following environment variable can alter behaviour:\n\n    DEFAULT_VIDEO_PLAYER - the name of the video player to use eg. 'QuickTime Player'\n    BACKGROUND_VIDEO     - if set to any value, opens in the background (does not automatically play),\n                           used in scripts like those below to prevent long-running downloads from popping into the\n                           foreground interrupting your workflow and having to Cmd-Tab back every time\n    PLAY_VIDEO           - if set to any value, starts playing the video (currently only tested on 'QuickTime Player')\n\nUsed by the following scripts:\n\n    youtube_download_video.sh\n    twitter_download_video.sh\n    x_download_video.sh\n    video_to_720p_mp4\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<video_file> [<player_specific_commandline_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nvideo=\"$1\"\nshift || :\n\n# Will be tried in this order\nlinux_commands=(\n    mpv\n    mplayer\n    vlc\n    ffplay\n    totem\n    xine\n)\n    #smplayer  # requires args unlike the others: smplayer -send-action play_current_file\n\nif is_mac; then\n    opts=()\n    # if the video player has a binary in \\$PATH then use it, otherwise assume it's an /Application/NAME.app\n    if type -P \"${DEFAULT_VIDEO_PLAYER-}\" &>/dev/null; then\n        if [ \"$DEFAULT_VIDEO_PLAYER\" = mpv ]; then\n            if [ -z \"${PLAY_VIDEO:-}\" ]; then\n                opts+=(\"--pause=yes\")\n            fi\n            if [ -z \"${BACKGROUND_VIDEO:-}\" ]; then\n                opts+=(\"--focus-on=never\")\n            fi\n        fi\n        \"$DEFAULT_VIDEO_PLAYER\" \"${opts[@]}\" \"$@\" -- \"$video\" &\n    else\n        if [ -n \"${BACKGROUND_VIDEO:-}\" ]; then\n            opts+=(-g)\n        fi\n        if [ -n \"${DEFAULT_VIDEO_PLAYER:-}\" ]; then\n            opts+=(-a \"$DEFAULT_VIDEO_PLAYER\")\n        fi\n        open \"${opts[@]}\" \"$@\" \"$video\"\n        if [ -n \"${PLAY_VIDEO:-}\" ]; then\n            osascript -e \"tell application \\\"${DEFAULT_VIDEO_PLAYER:-QuickTime Player}\\\" to play document 1\"\n        fi\n    fi\nelse  # assume Linux\n    found=0\n    for linux_command in \"${linux_commands[@]}\"; do\n        if type -P \"$linux_command\" &>/dev/null; then\n            found=1\n            \"$linux_command\" \"$@\" -- \"$video\" &\n            break\n        fi\n    done\n    if [ \"$found\" != 1 ]; then\n        die \"ERROR: none of the following Linux commands to open video were found: ${linux_commands[*]}\"\n    fi\nfi\n"
  },
  {
    "path": "media/webp_to_png.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-06 12:20:08 +0300 (Sun, 06 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts a Webp image to PNG to be usable on websites that don't support Webp images like Medium\n\nOpens the converted PNG image to verify it\n\nRequires either ImageMagick or dwebp to be installed, attempts to them in this order if not found\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nwebp=\"$1\"\n\nshopt -s nocasematch\nif ! [[ \"$webp\" =~ \\.webp$ ]]; then\n    die \"ERROR: file passed does not have an .webp file extension: $webp\"\nfi\n\n# because shopt -s nocasematch doesn't work on bash native string manipulation during testing\n# shellcheck disable=SC2001\npng=\"$(sed 's/\\.webp$//i' <<< \"$webp\").png\"\n\nif [ -f \"$png\" ]; then\n    die \"$png already exists, aborting...\"\nfi\n\nconverted=0\n\nconvert(){\n    if [ -f \"$png\" ]; then\n        die \"$png already exists, aborting...\"\n    fi\n    if type -P magick &>/dev/null; then\n        timestamp \"Converting '$webp' to '$png' using ImageMagick\"\n        magick \"$webp\" \"$png\"\n        return 0\n    elif type -P dwebp &>/dev/null; then\n        timestamp \"Converting '$webp' to '$png' using dwebp\"\n        dwebp \"webp\" -o \"$png\"\n        return 0\n    fi\n    timestamp \"No tool found installed to convert webp to png\"\n    return 1\n}\n\nif convert; then\n    converted=1\nelse\n    \"$srcdir/../packages/install_packages.sh\" imagemagick ||\n    \"$srcdir/../packages/install_packages.sh\" webp ||\n    die \"Failed to install any of the usual tools to convert Webp to PNG\"\n\n    if convert; then\n        converted=1\n    fi\nfi\n\nif [ \"$converted\" = 1 ]; then\n    if [ -f \"$png\" ]; then\n        timestamp \"Conversion complete, file available: $png\"\n    else\n        die \"Conversion failed. Did not find expected file: $png\"\n    fi\nelse\n    die \"Conversion failed\"\nfi\n\ntimestamp \"Opening image: $png\"\n\"$srcdir/imageopen.sh\" \"$png\"\n"
  },
  {
    "path": "media/youtube_download_channel.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-09-17 01:42:27 +0100 (Sun, 17 Sep 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads all videos from an entire YouTube channel using yt-dlp\n\nInstalls yt-dlp (for downloading) and ffmpeg (for conversions) via OS package manager if not already installed\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<youtube_channel_url>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n#\"$srcdir/../packages/install_packages_if_absent.sh\" yt-dlp ffmpeg\n\n# in case installed manually but not in package manager\nfor cmd in yt-dlp ffmpeg; do\n    if ! type -P \"$cmd\" &>/dev/null; then\n        timestamp \"$cmd not found in \\$PATH, attempting to install...\"\n        echo\n        \"$srcdir/../packages/install_packages.sh\" \"$cmd\"\n        echo\n    fi\n    check_bin \"$cmd\"\ndone\n\n# https://github.com/yt-dlp/yt-dlp#output-template\n\n#yt-dlp -f mp4 -c -w -o \"%(upload_date)s - %(title)s.%(ext)s\" -v \"$1\"\n\n# -c --continue\n# -w --no-overwrite\n# -o --output format file name\n# -v --verbose (debug output)\n#\n# --format mp4 \\\n#\n# --format best - results in poor quality video in testing due to video only and audio only combinations\n#\n# --format \"bestvideo+bestaudio/best\" - unfortunately this results in a file that macOS QuickTime can't open natively\n#                                       (although VLC can but then VLC was always the best)\n#\n#       bestvideo+bestaudio: downloads the best video and audio streams separately and merges them (requires ffmpeg or avconv)\n#       /best: falls back to the best single file if the video+audio combination isn't available\n#\n# for maximum compatibility specify compatible formats\nyt-dlp \\\n    --format \"bestvideo[ext=mp4][vcodec^=avc1]+bestaudio[ext=m4a]/best[ext=mp4]\" \\\n    --merge-output-format mp4 \\\n    --continue \\\n    --no-overwrite \\\n    --output \"%(autonumber)s - %(title)s.%(ext)s\" \\\n    ${DEBUG:+--verbose} \\\n    \"$1\"\n"
  },
  {
    "path": "media/youtube_download_video.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-09-17 01:42:27 +0100 (Sun, 17 Sep 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# default to trying get the best video and audio format we can\ndefault_format=\"bestvideo[ext=mp4][vcodec^=avc1]+bestaudio/best[ext=mp4]\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads a YouTube, Facebook or Twitter / X video to mp4 with maximum quality and compatibility using yt-dlp\n\nOpens the video automatically for you to check the download\n\nEven resumes downloads after network interruptions or a Control-C and re-run the command later\n\nInstalls yt-dlp (for downloading) and ffmpeg (for conversions) via OS package manager if not already installed\n\nUsed to automatically tries to upgrade the yt-dlp and ffmpeg packages first as sites like YouTube update their site\nregularly breaking this and requiring a yt-dlp update - but this broke on slow connections, so it's more efficient\nin day to day usage to just upgrade them as a one off manually if the downloading fails\n\n(You should only install yt-dlp and ffmpeg using your OS package manager for this auto-upgrading to work,\notherwise you'll be having to upgrade yt-dlp often yourself manually.)\n\nIf you run into a error determining a video format to download such as this:\n\n    WARNING: [youtube] RVVDi1PHgw4: nsig extraction failed: Some formats may be missing\n\nIf you run into this error:\n\n    ERROR: [youtube] ...: Sign in to confirm you’re not a bot.\n\nThen set this in your shell first with the name of your browser:\n\n    export COOKIES_FROM_BROWSER=chrome\n\nYou can specify the download format using args eg. --format, but you can also set this in environment variable used\nby this script:\n\n    export YT_DLP_FORMAT='best[height<=720]'\n\nThis can make hundreds of MB of size difference vs 1080p or greater, reducing a long download wait time.\n\nBetter would be to add the '[height<=720]' to the default format which is:\n\n    export YT_DLP_FORMAT='$default_format'\n\n        so it becomes:\n\n    export YT_DLP_FORMAT='bestvideo[ext=mp4][vcodec^=avc1][height<=720]+bestaudio/best[ext=mp4][height<=720]'\n\nTip: set this in direnv to have smaller quicker downloads using less disk space by limiting to good enough 720p,\nespecially for videos you're just going to quickly watch one time with the mpvd alias to delete the file after one\nwatch, see this documentation:\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/direnv.md\n\n    https://github.com/HariSekhon/Knowledge-Base/blob/main/mpv.md\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<video_url> [<filename>.mp4 <yt-dlp-args>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n#max_args 2 \"$@\"\n\nurl=\"$1\"\nfile_basename_without_ext=\"${2:-%(title)s}\"\nshift || :\nshift || :\n\noutput_filename=\"$file_basename_without_ext.%(ext)s\"\n\n# override the format with direnv in your ~/Downloads or ~/Downloads/YouTube dir\n# dir if you want faster downloads at a good enough format:\n#\n#   eg. export YT_DLP_FORMAT='best[height=720]'\n#\nformat=\"${YT_DLP_FORMAT:-$default_format}\"\n\nexport HOMEBREW_NO_ENV_HINTS=1\n\n# In case installed manually but not in package manager...\n#\n# Update: expect user to be using package manager so we can auto-upgrade yt-dlp which is needed often\n#\n#for cmd in yt-dlp ffmpeg; do\n#    if ! type -P \"$cmd\" &>/dev/null; then\n#        timestamp \"$cmd not found in \\$PATH, attempting to install...\"\n#        echo\n#        \"$srcdir/../packages/install_packages.sh\" \"$cmd\"\n#        echo\n#    fi\n#    check_bin \"$cmd\"\n#done\n\n\"$srcdir/../packages/install_packages_if_absent.sh\" yt-dlp ffmpeg\n\n# painfully slow blocks downloads on slow connections because retrieving new Homebrew package lists is expensive\n#\"$srcdir/../packages/upgrade_packages_if_outdated.sh\" yt-dlp ffmpeg\n\n# https://github.com/yt-dlp/yt-dlp#output-template\n\n# -c --continue\n# -w --no-overwrite\n# -o --output format file name\n# -v --verbose (debug output)\n#\n# --format mp4 \\\n#\n# --format best - results in poor quality video in testing due to video only and audio only combinations\n#\n# --format \"bestvideo+bestaudio/best\" - unfortunately this results in a file that macOS QuickTime can't open natively\n#                                       (although VLC can but then VLC was always the best)\n#\n#       bestvideo+bestaudio: downloads the best video and audio streams separately and merges them (requires ffmpeg or avconv)\n#       /best: falls back to the best single file if the video+audio combination isn't available\n#\n# for maximum compatibility specify compatible formats\n#\n#    --output \"%(title)s.%(ext)s\" \\\n#\n# Increased retries to 50 because hotel wifi sucks around the world, which is why you want to download the videos\n# for reliable offline play in the first place (or watching on the plane)\nyt-dlp \\\n    --format \"$format\" \\\n    --merge-output-format mp4 \\\n    --continue \\\n    --no-overwrite \\\n    --retries 50 \\\n    --output \"$output_filename\" \\\n    ${DEBUG:+--verbose} \\\n    ${COOKIES_FROM_BROWSER:+--cookies-from-browser \"$COOKIES_FROM_BROWSER\"} \\\n    \"$@\" \\\n    \"$url\"\n\nif [ \"${2:-}\" ]; then\n    # quicker and should always be the arg and .mp4 due to the --format options above\n    filename=\"$file_basename_without_ext.mp4\"\nelse\n    # if the filename isn't specified, we can infer it since no filename specified means no path specified so\n    # we can infer it to be the most recent file with an mp4 extension in $PWD\n    # shellcheck disable=SC2012\n    # this doesn't work reliably, the timestamp of a newer downloaded video file can be older, resulting in opening\n    # the wrong video\n    #\"$srcdir/vidopen.sh\" \"$(ls -t ./*.mp4 | head -n1)\"\n    timestamp \"Determining download filename\"\n    # \"$format\" is only needed here for it to return the right file extension\n    # in the \"$output_filename\" format eg. '.mp4' instead of '.webm'\n    filename=\"$(\n        yt-dlp --get-filename \\\n               --format \"$format\" \\\n               --merge-output-format mp4 \\\n               --output \"$output_filename\" \\\n               ${COOKIES_FROM_BROWSER:+--cookies-from-browser \"$COOKIES_FROM_BROWSER\"} \\\n               \"$@\" \\\n               \"$url\"\n    )\"\nfi\nif ! [ -f \"$filename\" ]; then\n    die \"Failed to find expected output file: $filename\"\nfi\ntimestamp \"Touching file timestamp to make it easier to find when browsing\"\ntouch -- \"$filename\"\n#if is_mac; then\n#    timestamp \"Showing in Finder\"\n#    open -R \"$filename\"\n#fi\nif [ -z \"${NO_VIDEO_OPEN:-}\" ]; then\n    timestamp \"Opening video file: $filename\"\n    \"$srcdir/vidopen.sh\" \"$filename\"\nfi\ntimestamp \"Download Complete: $filename\"\n"
  },
  {
    "path": "monitoring/dump_stats.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-26 14:38:31 +0200 (Mon, 26 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage_description=\"\nDumps common command outputs to text files in a local tarball\n\nUseful to collect support information for vendor support cases\n\nScp'd by ssh_dump_stats.sh to collect from remote servers\n\n\nCreates a tarball in this name format:\n\nstats-bundle.YYYY-MM-DD-HHSS.tar.gz\n\n\nIf this script is not run as root then prefixes sudo to every command to ensure that it succeeds\n\n\nExport LSOF=true to dump the lsof output which is otherwise skipped by default because on a production VM it resulted in 599M of output which aside from taking 30 seconds was ballooning the even compressed tarball to 47MB instead of 108KB\n\n\nIf NO_REMOVE_STATS_DIR environment variable is set to any value then does not remove the intermediate stats-bundle.YYYY-MM-DD-HHSS directory\n\n\nusage: ${0##*/}\n\"\n\nif [ $# -gt 0 ]; then\n    echo \"$usage_description\"\n    exit 3\nfi\n\ntstamp=\"$(date '+%F_%H%M')\"\n\nstats_bundle_dir=\"stats-bundle-$tstamp\"\n\nmkdir -p -v \"$stats_bundle_dir\"\n\ncd \"$stats_bundle_dir\"\n\nsudo=\"\"\nif [ \"$EUID\" -ne 0 ]; then\n    sudo=sudo\nfi\n\nmac=false\nif uname -s | grep Darwin; then\n    mac=true\nfi\n\ntimestamp(){\n    printf \"%s\\n\" \"$(date '+%F %T')  $*\" >&2\n}\n\ndump(){\n    local name=\"$1\"\n    shift\n    local cmd=(\"$name\")\n    if [ $# -gt 0 ]; then\n        cmd=(\"$@\")\n    fi\n    cmd_name=\"${cmd[0]}\"\n    if ! type -P \"$cmd_name\" &>/dev/null; then\n        timestamp \"Command '$cmd_name' not found, skipping...\"\n        return\n    fi\n    log_file=\"$name-output.$tstamp.txt\"\n    # ignore && && || it works\n    # shellcheck disable=SC2015\n    timestamp \"Collecting $name output\" >&2\n    $sudo \"${cmd[@]}\" > \"$log_file\"\n    timestamp  \"Collected $name output to file: $log_file\"\n    echo >&2\n}\n\n# ============================================================================ #\n#                    Commands that work on both Linux and Mac\n# ============================================================================ #\n\ndump_common(){\n\n    timestamp \"Dumping common command outputs\"\n    echo >&2\n    # -g for GB only works on Mac\n    #dump df df -g\n    dump df df -h\n    dump dmesg\n    # very expensive - takes 30 seconds and results in a 599MB output file on a production informatica secure agent server VM\n    if [ \"${LSOF:-}\" = true ]; then\n        dump lsof lsof -n -O\n    fi\n    dump netstat netstat -an\n    dump ps_ef ps -ef\n    dump uname uname -a\n    dump uptime\n\n}\n\n# ============================================================================ #\n#                         Commands that only work on Mac\n# ============================================================================ #\n\ndump_mac(){\n\n    timestamp \"Dumping Mac specific command outputs\"\n    echo >&2\n    dump diskutil_list diskutil list\n    dump iostat iostat -c 5\n    dump memory_pressure\n    dump ps_auxf ps aux\n    dump top top -l 1\n    dump top_mpstat top -l 1 -stats pid,command,cpu,th,pstate,time,cpu -ncols 16\n    #dump vmstat  # not available on my macOS 14\n}\n\n# ============================================================================ #\n#                        Commands that only work on Linux\n# ============================================================================ #\n\ndump_linux(){\n\n    timestamp \"Dumping Linux specific command outputs\"\n    echo >&2\n    dump free free -g\n    dump iostat iostat -x 1 5\n    dump lsblk\n    dump mpstat mpstat -P ALL 1 5\n    dump ps_auxf ps auxf\n    dump sar_5 sar -u 1 5\n    dump sar_all sar -A\n    dump top top -H -b -n 1\n    dump vmstat vmstat 1 5\n}\n\n# ============================================================================ #\n\ndump_common\n\nif [ \"$mac\" = true ]; then\n    dump_mac\nelse # assume Linux as the default case\n    dump_linux\nfi\n\ntimestamp \"Finished collection\"\necho >&2\n\ntarball=\"$stats_bundle_dir.tar.gz\"\n\ntimestamp \"Creating compressed tarball '$tarball' for easier collection and space savings\"\n\ncd ..\n\ntar czvf \"$tarball\" \"$stats_bundle_dir\"\necho\n\nif [ -z \"${NO_REMOVE_STATS_DIR:-}\" ]; then\n    timestamp \"Removing directory: $stats_bundle_dir\"\n    rm -fr -- \"$stats_bundle_dir\"\n    echo >&2\nfi\n\ntimestamp \"Collect tarball at: $PWD/$tarball\"\n"
  },
  {
    "path": "monitoring/log_timestamp_large_intervals.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-05-20 12:48:08 +0400 (Mon, 20 May 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds log lines whose timestamp intervals exceed the given number of seconds (default: 30)\nand outputs those log lines with the difference between the last and current timestamps\n\nUseful to find actions that are taking a long time from log files such as CI/CD logs\n\nExpects each log line in the file to be prefixed with the following format:\n\nYYYY-MM-DDTHH:MM:SS\n\n(year month day T hour minute second)\n\nThis is a fairly standard log timestamp used by tools like Azure DevOps Pipeline logs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<logfile> [<minimum_gap_seconds>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nlogfile=\"${1:-/dev/stdin}\"\nminimum_secs_gap=\"${2:-30}\"\n\n# prefix format - must match the line strip inside the while loop below\n#format=\"%Y-%M-%DT%H:%M:%S\"\n\nlast_epoch=0\ncurrent_epoch=0\nlast_line=\"\"\n\nwhile read -r line; do\n    # XXX: adjust this for other timestamp formats / lengths\n    timestamp=\"${line:0:19}\"  # must match the timestamp prefix length\n    #timestamp=\"$(awk '{print $1}' <<< \"$line\")\"  # needlessly expensive on forks and awk\n    #timestamp=\"${line%%[[:space:]]*}\"  # cheaper, take only the first token, but timestamps can have no spaces in them in place of the T separator\n    current_epoch=\"$(date -d \"$timestamp\" +\"%s\")\"\n\n    if [[ \"$last_epoch\" -eq 0 ]]; then\n        last_epoch=\"$current_epoch\";\n    fi\n\n    gap_secs=\"$((current_epoch - last_epoch))\"\n\n    if [[ \"$gap_secs\" -gt \"$minimum_secs_gap\" ]]; then\n        # redundant information, use for debugging only\n        #echo \"$gap_secs secs = $(date -d \"@$last_epoch\" +\"$format\") => $(date -d \"@$current_epoch\" +\"$format\")\"\n        #echo \"$gap_secs secs\"\n        #if [ \"$gap_secs\" -gt 60 ]; then\n            mins=\"$((gap_secs / 60))\"\n            secs=\"$((gap_secs % 60))\"\n            echo \"$mins mins $secs secs\"\n        #fi\n        echo \"$last_line\"\n        echo \"$line\"\n        echo\n    fi\n\n    last_epoch=\"$current_epoch\"\n    last_line=\"$line\"\ndone < \"$logfile\"\n"
  },
  {
    "path": "monitoring/prometheus.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 05:24:37 +0300 (Tue, 08 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns Prometheus locally\n\nInstalls it to \\$PATH if not already available\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<prometheus_args>]\"\n\nhelp_usage \"$@\"\n\nif ! type -P prometheus &>/dev/null; then\n    \"$srcdir/../install/install_prometheus.sh\"\nfi\n\nprometheus \"$@\"\n"
  },
  {
    "path": "monitoring/prometheus_docker.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 16:36:29 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots Prometheus in Docker with the given configuration (default: \\$PWD/prometheus.yml)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ] [<prometheus.yml>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/prometheus.yml\"\nexport PROMETHEUS_URL=\"http://localhost:9090\"\nexport PROMETHEUS_CONFIG=\"${2:-$srcdir/../setup/prometheus.yml}\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting Prometheus:\"\n    docker-compose up -d \"$@\"\n    echo >&2\n    when_url_content \"$PROMETHEUS_URL\" '(?i:prometheus)'\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" ui\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo >&2\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"Prometheus URL:  $PROMETHEUS_URL\"\n    echo\n    if is_mac; then\n        open \"$PROMETHEUS_URL\"\n    fi\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo >&2\n    exit 0\nfi\n"
  },
  {
    "path": "monitoring/prometheus_node_exporter.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-10-08 05:24:37 +0300 (Tue, 08 Oct 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns local Prometheus Node Exporter\n\nInstalls it to \\$PATH if not already available\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<node_exporter_args>]\"\n\nhelp_usage \"$@\"\n\nif ! type -P node_exporter &>/dev/null; then\n    \"$srcdir/../install/install_prometheus_node_exporter.sh\"\nfi\n\nnode_exporter \"$@\"\n"
  },
  {
    "path": "monitoring/ssh_dump_logs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-26 14:38:31 +0200 (Mon, 26 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/kubernetes.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses SSH to dump logs to local text files for servers given as arguments for uploading to vendor support cases\n\nCollects:\n\n/var/log/messages\n/var/log/dmesg\n\n\nDumps logs to files of this name format:\n\nlog.YYYY-MM-DD-HHSS.<server>.<log>.txt\n\n\nFor each server that is not prefixed by root@, uses sudo in ssh to copy the log file to the user's home directory first\nto work around root file permissions issues and chown it to the login user\n\n\nRequires SSH client to be installed and configured to preferably passwordless ssh key access\n\n\nTuning Authentication - SSH User and SSH Key\n\nIf either of the following environment variables are set they will be added to the SSH commands:\n\n    SSH_USER\n    SSH_KEY\n\nTo use a different SSH key - for example because you are iterating AWS EC2 servers, you can must either set SSH_KEY\nor else add that EC2 SSH key to your ssh-agent - see this doc:\n\nhttps://github.com/HariSekhon/Knowledge-Base/blob/main/ssh.md#use-ssh-agent\n\nThen run this script as usual\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<server1> [<server2> <server3>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nssh_known_hosts=~/.ssh/known_hosts\n\ntimestamp \"SSH Keyscanning nodes to prevent getting stuck on host key prompts\"\necho\nfor user_server in \"$@\"; do\n    server=\"${user_server##*@}\"\n    timestamp \"SSH keyscan '$server'\"\n    ssh-keyscan \"$server\" |\n    while read -r line; do\n        grep -Fxq \"$line\" \"$ssh_known_hosts\" ||\n        echo \"$line\" >> \"$ssh_known_hosts\"\n    done\n    echo\ndone\necho\n\ntstamp=\"$(date '+%F_%H%M')\"\n\nfor user_server in \"$@\"; do\n    echo\n    server=\"${user_server##*@}\"\n    user=\"${user_server%%@*}\"\n    if [ \"$user\" = \"$server\" ]; then\n        user=\"${SSH_USER:-}\"\n    fi\n    #for log in messages secure dmesg; do\n    for log in messages dmesg; do\n        # do not change this format without also changing the format of the gunzip and tar operations further down\n        log_file=\"log.$tstamp.$server.$log.txt.gz\"\n        # ignore && && || it works\n        # shellcheck disable=SC2015\n        timestamp \"Dumping server '$server' log: $log\" &&\n        sudo=\"\"\n        if ! [[ \"$server\" =~ ^root@ ]]; then\n            sudo=sudo\n        fi\n        # want client side expansion\n        # shellcheck disable=SC2029\n        ssh ${SSH_KEY:+-i \"$SSH_KEY\"} ${user:+\"$user@\"}\"$server\" \"\n            $sudo gzip -c /var/log/$log > ~/$log_file &&\n            $sudo chown -v \\$USER ~/$log_file\n        \" &&\n        scp ${SSH_KEY:+-i \"$SSH_KEY\"} ${user:+\"$user@\"}\"$server\":\"./$log_file\" .\n        timestamp \"Dumped server '$server' log to file: $log_file\" ||\n        warn \"Failed to get '$server' log: $log\"\n        # XXX: because race condition - AWS EC2 Spot instances or GCP Preemptible instances can go away\n        # during execution and we still want to collect the rest of the servers without erroring out completely\n    done\ndone\necho\n\ntimestamp \"Tarballing all logs into a support bundle for easy sharing with vendors\"\necho\ntimestamp \"Gunzipping logs to achieve better overall compression\"\necho\ngunzip \"log.$tstamp\".*.txt.gz\necho\ntarball=\"logs.$tstamp.tar.gz\"\ntimestamp \"Tarballing logs to: $tarball\"\necho\ntar czvf \"$tarball\" \"log.$tstamp\".*.txt\necho\ntimestamp \"Tarball ready: $tarball\"\necho\ntimestamp \"Log dumps completed\"\n"
  },
  {
    "path": "monitoring/ssh_dump_stats.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-26 14:38:31 +0200 (Mon, 26 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses SSH to dump common command outputs from remote servers to a local tarball. Useful for vendor support cases\n\nCopies adjacent dump_stats.sh script to the remote server, executes it, and collects the resulting tarball back to the local machine\n\nThe collected tarball will in this name format:\n\n<server>.stats-bundle.YYYY-MM-DD-HHSS.tar.gz\n\n\nLsof output is not included by default as it is very voluminous, see dump_stats.sh for details and to enable it\n\n\nRequires SSH client to be installed and configured to preferably passwordless ssh key access\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<server1> [<server2> <server3>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nfor server in \"$@\"; do\n    scp \"$srcdir/dump_stats.sh\" \"$server\":\n    # doesn't work - might have to be allowed in sshd_config which is not portable by default\n    #ssh -o SendEnv=DEBUG \"$server\" \"\n    # want client side expansion\n    # shellcheck disable=SC2029,SC2015\n    ssh \"$server\" \"\n        export DEBUG=\\\"${DEBUG:-}\\\"\n        export LSOF=\\\"${LSOF:-}\\\"\n        chmod +x dump_stats.sh &&\n        ./dump_stats.sh\n    \" &&\n    latest_tarball=\"$(ssh \"$server\" \"ls -tr stats-bundle-*.tar.gz | tail -n 1\")\" &&\n    timestamp \"Collecting tarball '$latest_tarball'\" &&\n    scp \"$server\":\"$latest_tarball\" \"$server.$latest_tarball\" &&\n    timestamp \"Collected tarball\" &&\n    timestamp \"Removing tarball on server to save space\" &&\n    ssh \"$server\" \"rm -fv -- ./$latest_tarball\" ||\n    warn \"Failed to collect tarball from server '$server'\"\n    # XXX: because race condition - spot instances can go away during execution\n    # and we still want to collect the rest of the servers\ndone\n\ntimestamp \"Stats dumps completed\"\n"
  },
  {
    "path": "mysql/mariadb.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-05 19:49:48 +0100 (Wed, 05 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://hub.docker.com/_/mariadb\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\n# defined in lib/dbshell.sh\n# shellcheck disable=SC2154\nshell_description=\"$sql_mount_description\n\nSource a sql script:\n\nsource mysql_info.sql\n\n\nGet shell access:\n\n\\\\! bash\n\n\nList available SQL scripts:\n\n\\\\! ls -l mysql*.sql\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots a quick MariaDB docker container and drops you in to the 'mysql' shell\n\nMultiple invocations of this script will connect to the same MariaDB container if already running\nand the last invocation of this script to exit from the mysql shell will delete that container\n\nMariaDB version can be specified using the first argument, or the \\$MARIADB_VERSION environment variable,\notherwise 'latest' is used\n\nVersions to use can be found from the following URL:\n\nhttps://hub.docker.com/_/mariadb?tab=tags\n\nor programmatically on the command line (see DevOps Python tools repo):\n\ndockerhub_show_tags.py mariadb\n\n\nOptions to the 'mysql' shell command can be given using the \\$MYSQL_OPTS environment variable\n\nAutomatically creates shared bind mount points from host to container for convenience:\n$shell_description\n\n\nTested on MariaDB 5.5, 10.0 - 10.5\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>] [options]\n\n-n  --name  NAME    Docker container name to use (default: mariadb)\n-p  --port  PORT    Expose MariaDB port 3306 on given port number\n-d  --no-delete     Don't delete the container upon the last mysql session closing (\\$DOCKER_NO_DELETE)\n-r  --restart       Force a restart of a clean MariaDB instance (\\$MARIADB_RESTART)\n-s  --sample        Load sample Chinook database (\\$LOAD_SAMPLE)\"\n\nhelp_usage \"$@\"\n\ndocker_image=mariadb\nport=\"\"\ndocker_opts=\"\"\n\npassword=\"${MYSQL_ROOT_PASSWORD:-${MYSQL_PWD:-${MYSQL_PASSWORD:-${PASSWORD:-test}}}}\"\n\nwhile [ $# -gt 0 ]; do\n    # DOCKER_NO_DELETE used by functions from lib\n    # shellcheck disable=SC2034\n    case \"$1\" in\n      -n| --name)   container_name=\"$2\"\n                    shift\n                    ;;\n      -p| --port)   port=\"$2\"\n                    [[ \"$port\" =~ ^[[:digit:]]*$ ]] || die \"invalid --port '$port' given\"\n                    shift\n                    ;;\n     -s|--sample)   LOAD_SAMPLE_DB=1\n                    ;;\n    -r|--restart)   MARIADB_RESTART=1\n                    ;;\n  -d|--no-delete)   DOCKER_NO_DELETE=1\n                    ;;\n               *)   version=\"$1\"\n                    ;;\n    esac\n    shift\ndone\n\ncontainer_name=\"${container_name:-${MARIADB_CONTAINER_NAME:-mariadb}}\"\nversion=\"${version:-${MARIADB_VERSION:-latest}}\"\n\nif [ -n \"$port\" ]; then\n    docker_opts=\"-p $port:5432\"\nfi\n\ndb=\"$srcdir/chinook.mysql\"\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ] &&\n   ! [ -f \"$db\" ]; then\n    timestamp \"downloading sample 'chinook' database\"\n    wget -qcO \"$db\" 'https://github.com/lerocha/chinook-database/blob/master/ChinookDatabase/DataSources/Chinook_MySql.sql?raw=true'\n    #iconv -f ISO-8859-1 -t UTF-8 \"$db\" > \"$db.utf8\"\nfi\n\n# kill existing if we have specified a different version than is running\ndocker_image_version=\"$(docker_container_image \"$container_name\")\"\nif [ -n \"$docker_image_version\" ] &&\n   [ \"$docker_image_version\" != \"$docker_image:$version\" ]; then\n    MARIADB_RESTART=1\nfi\n\n# remove existing non-running container so we can boot a new one\nif docker_container_not_running \"$container_name\"; then\n    MARIADB_RESTART=1\nfi\n\nif [ -n \"${MARIADB_RESTART:-}\" ]; then\n    # ensures version is correct before we kill any existing test env to switch versions to minimize downtime\n    timestamp \"docker pull $docker_image:$version\"\n    docker_pull \"$docker_image:$version\"\n\n    timestamp \"killing existing MariaDB container:\"\n    docker rm -f -- \"$container_name\" 2>/dev/null || :\nfi\n\nif ! docker_container_exists \"$container_name\"; then\n    timestamp \"booting MariaDB container from image '$docker_image:$version':\"\n    # defined in lib/dbshell.sh\n    # shellcheck disable=SC2154,SC2086\n    docker run -d \\\n        --name \"$container_name\" \\\n        $docker_opts \\\n        -e MYSQL_ROOT_PASSWORD=\"$password\" \\\n        $docker_sql_mount_switches \\\n        \"$docker_image\":\"$version\"\n        #-v \"$srcdir/../setup/mysql/conf.d/my.cnf:/etc/mysql/conf.d/\" \\\nfi\n\nwait_for_mysql_ready \"$container_name\"\necho\n\ntimestamp \"linking shell profile for .my.cnf\"\ndocker exec \"$container_name\" bash -c \"cd /bash && setup/shell_link.sh &>/dev/null\" || :\n\n# yes expand now\n# shellcheck disable=SC2064\ntrap \"echo ERROR; echo; echo; [ -z '${DEBUG:-}' ] || docker logs '$container_name'\" EXIT\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ]; then\n    dbname=\"${db##*/}\"\n    dbname=\"${dbname%%.*}\"\n    timestamp \"loading $dbname database\"\n    #docker exec -i \"$container_name\" mysql -u root -p\"$password\" ${MYSQL_OPTS:-} -e \"CREATE DATABASE IF NOT EXISTS $dbname\"\n    timestamp \"loading data (this may take a minute)\"\n    # shellcheck disable=SC2086\n    docker exec -i -e MYSQL_PWD=\"$password\" \"$container_name\" mysql -u root ${MYSQL_OPTS:-} < \"${db}\"\n    timestamp \"done\"\n    echo >&2\nfi\n\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    cat <<EOF\n$shell_description\n\nEOF\nfi\n\n# cd to /sql to make sourcing easier without /sql/ path prefix\ndocker_exec_opts=\"-w /sql -i\"\n\n# allow non-interactive piped automation to avoid tty errors eg.\n# for sql in mysql*.sql; do echo \"source $sql\"; done | mariadb.sh\n# normally you would just 'mariadb.sh mysql*.sql' but this is used by mariadb_test_scripts.sh\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    docker_exec_opts+=\" -t\"\nfi\n\n# want opt splitting\n# shellcheck disable=SC2154,SC2086\ndocker exec -e MYSQL_PWD=\"$password\" $docker_exec_opts \"$container_name\" mysql -u root ${MYSQL_OPTS:-}\n\nuntrap\n\ndocker_rm_when_last_connection \"$0\" \"$container_name\"\n"
  },
  {
    "path": "mysql/mariadb_test_scripts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2028\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-09 10:42:23 +0100 (Sun, 09 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\nmariadb_versions=\"\n5.5\n10.0\n10.1\n10.2\n10.3\n10.4\n10.5\nlatest\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns all of the scripts given as arguments against multiple MariaDB versions using docker\n\nUses mariadb.sh to boot a MariaDB docker environment and pipe source statements in to the container\n\nSources each script in MariaDB in the order given\n\nRuns against a list of MariaDB versions from the first of the following conditions:\n\n- If \\$MARIADB_VERSIONS environment variable is set, then only tests against those versions in the order given, space or comma separated, with 'x' used as a wildcard (eg. '5.x , 10.x')\n- If \\$GET_DOCKER_TAGS is set and dockerhub_show_tags.py is found in the \\$PATH (from DevOps Python tools repo), then uses it to fetch the latest live list of version tags available from the dockerhub API, reordering by newest first\n- Falls back to the following pre-set list of versions, reordering by newest first:\n\n$(tr ' ' '\\n' <<< \"$mariadb_versions\" | grep -v '^[[:space:]]*$')\n\nIf a script has a headers such as:\n\n-- Requires MariaDB N.N (same as >=)\n-- Requires MariaDB >= N.N\n-- Requires MariaDB >  N.N\n-- Requires MariaDB <= N.N\n-- Requires MariaDB <  N.N\n\nthen will only run that script on the specified versions of MariaDB\n\nThis is for convenience so you can test a whole repository such as my SQL-scripts repo just by running against all scripts and have this code figure out the combinations of scripts to run vs versions, eg:\n\n${0##*/} mysql_*.sql\n\nIf no script files are given as arguments, then searches \\$PWD for scripts named in the formats:\n\nmysql*.sql\nmaria*.sql\nmariadb*.sql\n*.mysql\n\n\nTested on MariaDB 5.5, 10.0 - 10.5\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"script1.sql [script2.sql ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport MARIADB_CONTAINER_NAME=\"${MARIADB_CONTAINER_NAME:-mariadb-test-scripts}\"\n\nif [ $# -gt 0 ]; then\n    scripts=(\"$@\")\nelse\n    shopt -s nullglob\n    scripts=(mysql*.sql maria*.sql mariadb*.sql *.mysql)\nfi\n\nif [ ${#scripts[@]} -lt 1 ]; then\n    usage \"no scripts given and none found in current working directory matching the patterns: mysql*.sql / maria*.sql / mariadb*.sql / *.mysql\"\nfi\n\nfor sql_file in \"${scripts[@]}\"; do\n    [ -f \"$sql_file\" ] || die \"ERROR: file not found: $sql_file\"\ndone\n\necho \"Testing ${#scripts[@]} MariaDB scripts:\"\necho\nfor sql_file in \"${scripts[@]}\"; do\n    echo \"$sql_file\"\ndone\necho\n\nget_mariadb_versions(){\n    if [ -n \"${GET_DOCKER_TAGS:-}\" ]; then\n        echo \"checking if dockerhub_show_tags.py is available:\" >&2\n        echo >&2\n        if type -P dockerhub_show_tags.py 2>/dev/null; then\n            echo >&2\n            echo \"dockerhub_show_tags.py found, executing to get latest list of MariaDB docker version tags\" >&2\n            echo >&2\n            mariadb_versions=\"$(dockerhub_show_tags.py mariadb |\n                                grep -Eo -e '[[:space:]][[:digit:]]{1,2}\\.[[:digit:]]' \\\n                                         -e '^[[:space:]*latest[[:space:]]*$' |\n                                sed 's/[[:space:]]//g' |\n                                sort -u -t. -k1n -k2n)\"\n            echo \"found MariaDB versions:\" >&2\n            echo \"$mariadb_versions\"\n            echo >&2\n            return\n        fi\n    fi\n    echo \"$mariadb_versions\" |\n    tr ' ' '\\n' |\n    grep -v '^[[:space:]]*$' |\n    if is_CI; then\n        echo \"CI detected - using randomized sample of MariaDB versions to test against:\" >&2\n        {\n        shuf | head -n 1\n        echo 5.5  # most problematic / incompatible versions should always be tested\n        echo 10.0\n        echo latest\n        } | sort -unr -t. -k1,2\n    else\n        echo \"using default list of MariaDB versions to test against:\" >&2\n        cat\n    fi\n    echo >&2\n}\n\nif [ -n \"${MARIADB_VERSIONS:-}\" ]; then\n    versions=\"\"\n    MARIADB_VERSIONS=\"${MARIADB_VERSIONS//,/ }\"\n    for version in $MARIADB_VERSIONS; do\n        if [[ \"$version\" =~ x ]]; then\n            versions+=\" $(grep \"${version//x/.*}\" <<< \"$mariadb_versions\" |\n                          sort -u -t. -k1n -k2 |\n                          tac ||\n                          die \"version '$version' not found\")\"\n        else\n            versions+=\" $version\"\n        fi\n    done\n    mariadb_versions=\"$(tr ' ' '\\n' <<< \"$versions\" | grep -v '^[[:space:]]*$')\"\n    echo \"using given MariaDB versions:\"\nelse\n    mariadb_versions=\"$(get_mariadb_versions | tac)\"\nfi\n\necho \"$mariadb_versions\"\necho\n\nfor version in $mariadb_versions; do\n    hr\n    echo \"Executing scripts against MariaDB version '$version'\": >&2\n    echo >&2\n    {\n    # comes out first, not between scripts\n    #echo '\\! printf \"================================================================================\\n\"'\n    echo 'SELECT VERSION();'\n    for sql_file in \"${scripts[@]}\"; do\n        if skip_min_version \"MariaDB\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        if skip_max_version \"MariaDB\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        # no effect\n        #echo\n        # comes out first instead of with scripts\n        #echo \"\\\\! printf '\\nscript %s:' '$sql_file'\"\n        echo \"select '$sql_file' as script;\"\n        # instead of dealing with pathing issues, prefixing /pwd or depending on the scripts being in the sql/ directory\n        #echo \"source $sql_file\"\n        cat \"$sql_file\"\n        #echo \"\\\\! printf '\\n\\n'\"\n    done\n    } |\n    # forcing mysql shell --table output as though interactive\n    MYSQL_OPTS=\"--table\" \\\n    command time \\\n    \"$srcdir/mariadb.sh\" \"$version\" --restart\n    echo >&2\n    timestamp \"Succeeded testing ${#scripts[@]} scripts for MariaDB $version\"\n    echo >&2\n    echo >&2\ndone\necho >&2\necho >&2\ntimestamp \"All MariaDB tests passed for all scripts on all versions:  $(tac <<< \"$mariadb_versions\" | tr '\\n' ' ')\"\n"
  },
  {
    "path": "mysql/mysql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:15:38 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to more easily connect to MySQL without having to repeatedly specify options like host, username and password\n\nLeverages standard MySQL options as well as others likely to be found in the environment\n\nhttps://dev.mysql.com/doc/refman/8.0/en/environment-variables.html\n\nSee also - GNU sql\n\nTested on MySQL 8.0.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<mysql_options>]\"\n\n#help_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfor arg; do\n    case \"$arg\" in\n        --help) usage\n                ;;\n    esac\ndone\n\n#opts=(${MYSQL_OPTS:-})\nread -r -a opts MYSQL_OPTS\n\nMYSQL_HOST=\"${MYSQL_HOST:-${HOST:-}}\"\nif [ -n \"${MYSQL_HOST:-}\" ]; then\n    opts+=(-h \"$MYSQL_HOST\")\nfi\n\nMYSQL_PORT=\"${MYSQL_TCP_PORT:-${MYSQL_PORT:-${PORT:-}}}\"\nif [ -n \"${MYSQL_PORT:-}\" ]; then\n    opts+=(-P \"$MYSQL_PORT\")\nfi\n\nMYSQL_USER=\"${MYSQL_USER:-${DBI_USER:-${USER:-}}}\"\nif [ -n \"${MYSQL_USER:-}\" ]; then\n    opts+=(-u \"$MYSQL_USER\")\nfi\n\nMYSQL_PASSWORD=\"${MYSQL_PWD:-${MYSQL_PASSWORD:-${PASSWORD:-}}}\"\nif [ -n \"${MYSQL_PASSWORD:-}\" ]; then\n    echo \"WARNING: not auto-adding -p<password> for safety\" >&2\n    echo \"exporting MYSQL_PWD instead but this is deprecated and may not work in future versions\" >&2\n    #opts+=(-p\"$MYSQL_PASSWORD\")\n    export MYSQL_PWD=\"$MYSQL_PASSWORD\"\nfi\n\nMYSQL_DATABASE=\"${MYSQL_DATABASE:-${DATABASE:-}}\"\nif [ -n \"${MYSQL_DATABASE:-}\" ]; then\n    opts+=(-d \"$MYSQL_DATABASE\")\nfi\n\n# split opts\n# shellcheck disable=SC2086\nexec mysql \"${opts[@]}\" \"$@\"\n"
  },
  {
    "path": "mysql/mysql_foreach_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 22:54:48 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun SQL query against all MySQL tables in all databases via mysql.sh\n\nQuery can contain {db} and {table} placeholders which will be replaced for each table\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\nAuto-skips information_schema, performance_schema, sys and mysql databases for safety\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the db/table names and exit after the first iteration\n\n\nTested on MySQL 8.0.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"<query>\\\" [<mysql_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery_template=\"$1\"\nshift || :\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\nAUTOFILTER=1 \"$srcdir/mysql_list_tables.sh\" \"$@\" |\nwhile read -r db table; do\n    printf '%s.%s\\t' \"$db\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\`$db\\`}\"\n    query=\"${query//\\{table\\}/\\`$table\\`}\"\n    \"$srcdir/mysql.sh\" -s -D \"$db\" -e \"$query\" \"$@\"\ndone\n"
  },
  {
    "path": "mysql/mysql_list_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 20:21:42 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all MySQL databases using adjacent mysql.sh script\n\nFILTER environment variable will restrict to matching databases (if giving <db>.<table>, matches up to the first dot)\n\nAUTOFILTER if set to any value skips information_schema, performance_schema, sys and mysql databases\n\nTested on MySQL 8.0.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<mysql_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/mysql.sh\" -s -e 'SELECT DISTINCT(table_schema) FROM information_schema.tables ORDER BY table_schema;' \"$@\" |\nsed 's/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' |\nif [ -n \"${AUTOFILTER:-}\" ]; then\n    grep -Ev '^(information_schema|performance_schema|sys|mysql)$'\nelse\n    cat\nfi |\nwhile read -r db; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db\" =~ ${FILTER%%.*} ]]; then\n        continue\n    fi\n    printf '%s\\n' \"$db\"\ndone\n"
  },
  {
    "path": "mysql/mysql_list_tables.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 20:22:28 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all MySQL tables using adjacent mysql.sh script\n\nFILTER environment variable will restrict to matching tables (matches against fully qualified table name <db>.<table>)\n\nAUTOFILTER if set to any value skips information_schema, performance_schema, sys and mysql databases\n\nTested on MySQL 8.0.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<mysql_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/mysql.sh\" -s -e \"SELECT table_schema, TABLE_NAME FROM information_schema.tables;\" \"$@\" |\nsed 's/|//g; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' |\nif [ -n \"${AUTOFILTER:-}\" ]; then\n    grep -Ev '^(information_schema|performance_schema|sys|mysql)[[:space:]]'\nelse\n    cat\nfi |\nwhile read -r db table; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db.$table\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\n' \"$db\" \"$table\"\ndone\n"
  },
  {
    "path": "mysql/mysql_tables_row_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-13 11:05:05 +0000 (Fri, 13 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for all MySQL tables in all databases using adjacent mysql.sh script\n\nTSV Output format:\n\n<database>.<table>     <row_count>\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<table>)\n\nTested on MySQL 8.0.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<mysql_options>]\"\n\nhelp_usage \"$@\"\n\n\nexec \"$srcdir/mysql_foreach_table.sh\" \"SELECT COUNT(*) FROM {db}.{table}\" \"$@\"\n"
  },
  {
    "path": "mysql/mysql_test_scripts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2028\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-09 10:42:23 +0100 (Sun, 09 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\nmysql_versions=\"\n5.5\n5.6\n5.7\n8.0\nlatest\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns all of the scripts given as arguments against multiple MySQL versions using docker\n\nUses mysqld.sh to boot a mysql docker environment and pipe source statements in to the container\n\nSources each script in MySQL in the order given\n\nRuns against a list of MySQL versions from the first of the following conditions:\n\n- If \\$MYSQL_VERSIONS environment variable is set, then only tests against those versions in the order given, space or comma separated, with 'x' used as a wildcard (eg. '5.x , 8.x')\n- If \\$GET_DOCKER_TAGS is set and dockerhub_show_tags.py is found in the \\$PATH (from DevOps Python tools repo), then uses it to fetch the latest live list of version tags available from the dockerhub API, reordering by newest first\n- Falls back to the following pre-set list of versions, reordering by newest first:\n\n$(tr ' ' '\\n' <<< \"$mysql_versions\" | grep -v '^[[:space:]]*$')\n\nIf a script has a headers such as:\n\n-- Requires MySQL N.N (same as >=)\n-- Requires MySQL >= N.N\n-- Requires MySQL >  N.N\n-- Requires MySQL <= N.N\n-- Requires MySQL <  N.N\n\nthen will only run that script on the specified versions of MySQL\n\nThis is for convenience so you can test a whole repository such as my SQL-scripts repo just by running against all scripts and have this code figure out the combinations of scripts to run vs versions, eg:\n\n${0##*/} mysql_*.sql\n\nIf no script files are given as arguments, then searches \\$PWD for scripts named in the formats:\n\nmysql*.sql\n*.mysql\n\n\nTested on MySQL 5.5, 5.6, 5.7, 8.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"script1.sql [script2.sql ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport MYSQL_CONTAINER_NAME=\"${MYSQL_CONTAINER_NAME:-mysql-test-scripts}\"\n\nif [ $# -gt 0 ]; then\n    scripts=(\"$@\")\nelse\n    shopt -s nullglob\n    scripts=(mysql*.sql *.mysql)\nfi\n\nif [ ${#scripts[@]} -lt 1 ]; then\n    usage \"no scripts given and none found in current working directory matching the patterns: mysql*.sql / *.mysql\"\nfi\n\nfor sql_file in \"${scripts[@]}\"; do\n    [ -f \"$sql_file\" ] || die \"ERROR: file not found: $sql_file\"\ndone\n\necho \"Testing ${#scripts[@]} MySQL scripts:\"\necho\nfor sql_file in \"${scripts[@]}\"; do\n    echo \"$sql_file\"\ndone\necho\n\nget_mysql_versions(){\n    if [ -n \"${GET_DOCKER_TAGS:-}\" ]; then\n        echo \"checking if dockerhub_show_tags.py is available:\" >&2\n        echo >&2\n        if type -P dockerhub_show_tags.py 2>/dev/null; then\n            echo\n            echo \"dockerhub_show_tags.py found, executing to get latest list of MySQL docker version tags\" >&2\n            echo >&2\n            mysql_versions=\"$(dockerhub_show_tags.py mysql |\n                                grep -Eo -e '[[:space:]][[:digit:]]{1,2}\\.[[:digit:]]' \\\n                                         -e '^[[:space:]*latest[[:space:]]*$' |\n                                sed 's/[[:space:]]//g' |\n                                sort -u -t. -k1n -k2n)\"\n            echo \"found MySQL versions:\" >&2\n            echo >&2\n            echo \"$mysql_versions\"\n            return\n        fi\n    fi\n    echo \"$mysql_versions\" |\n    tr ' ' '\\n' |\n    grep -v '^[[:space:]]*$' |\n    if is_CI; then\n        echo \"CI detected - using randomized sample of MySQL versions to test against:\" >&2\n        {\n        shuf | head -n 1\n        echo 5.5  # most problematic / incompatible versions should always be tested\n        echo 5.6\n        echo latest\n        } | sort -unr -t. -k1,2\n    else\n        echo \"using default list of MySQL versions to test against:\" >&2\n        cat\n    fi\n    echo >&2\n}\n\nif [ -n \"${MYSQL_VERSIONS:-}\" ]; then\n    versions=\"\"\n    MYSQL_VERSIONS=\"${MYSQL_VERSIONS//,/ }\"\n    for version in $MYSQL_VERSIONS; do\n        if [[ \"$version\" =~ x ]]; then\n            versions+=\" $(grep \"${version//x/.*}\" <<< \"$mysql_versions\" |\n                          sort -u -t. -k1n -k2 |\n                          tac ||\n                          die \"version '$version' not found\")\"\n        else\n            versions+=\" $version\"\n        fi\n    done\n    mysql_versions=\"$(tr ' ' '\\n' <<< \"$versions\" | grep -v '^[[:space:]]*$')\"\n    echo \"using given MySQL versions:\"\nelse\n    mysql_versions=\"$(get_mysql_versions | tac)\"\nfi\n\necho \"$mysql_versions\"\necho\n\nfor version in $mysql_versions; do\n    hr\n    echo \"Executing scripts against MySQL version '$version'\": >&2\n    echo >&2\n    {\n    echo 'SELECT VERSION();'\n    for sql_file in \"${scripts[@]}\"; do\n        if skip_min_version \"MySQL\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        if skip_max_version \"MySQL\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        # comes out first, not between scripts\n        #echo '\\! printf \"================================================================================\\n\"'\n        # no effect\n        #echo\n        # comes out first instead of with scripts\n        #echo \"\\\\! printf '\\nscript %s:' '$sql_file'\"\n        echo \"select '$sql_file' as script;\"\n        # instead of dealing with pathing issues, prefixing /pwd or depending on the scripts being in the sql/ directory\n        # just cat them in to the shell instead as it's more portable\n        #echo \"source $sql_file\"\n        cat \"$sql_file\"\n        #echo \"\\\\! printf '\\n\\n'\"\n    done\n    } |\n    # forcing mysql shell --table output as though interactive\n    MYSQL_OPTS=\"--table\" \\\n    command time \\\n    \"$srcdir/mysqld.sh\" \"$version\" --restart\n    echo >&2\n    timestamp \"Succeeded testing ${#scripts[@]} scripts for MySQL $version\"\n    echo >&2\n    echo >&2\ndone\necho >&2\necho >&2\ntimestamp \"All MySQL tests passed for all scripts on all versions:  $(tac <<< \"$mysql_versions\" | tr '\\n' ' ')\"\n"
  },
  {
    "path": "mysql/mysqld.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-05 19:49:48 +0100 (Wed, 05 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://hub.docker.com/_/mysql\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\n# defined in lib/dbshell.sh\n# shellcheck disable=SC2154\nshell_description=\"$sql_mount_description\n\nSource a sql script:\n\nsource mysql_info.sql\n\n\nGet shell access:\n\n\\\\! bash\n\n\nList available SQL scripts:\n\n\\\\! ls -l mysql*.sql\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots a quick MySQL docker container and drops you in to the 'mysql' shell\n\nMultiple invocations of this script will connect to the same MySQL container if already running\nand the last invocation of this script to exit from the mysql shell will delete that container\n\nMySQL version can be specified using the first argument, or the \\$MYSQL_VERSION environment variable,\notherwise 'latest' is used\n\nVersions to use can be found from the following URL:\n\nhttps://hub.docker.com/_/mysql?tab=tags\n\nor programmatically on the command line (see DevOps Python tools repo):\n\ndockerhub_show_tags.py mysql\n\n\nOptions to the 'mysql' shell command can be given using the \\$MYSQL_OPTS environment variable\n\nAutomatically creates shared bind mount points from host to container for convenience:\n$shell_description\n\n\nTested on MySQL 5.5, 5.6, 5.7, 8.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>] [options]\n\n-n  --name  NAME    Docker container name to use (default: mysql)\n-p  --port  PORT    Expose MySQL port 3306 on given port number\n-d  --no-delete     Don't delete the container upon the last mysql session closing (\\$DOCKER_NO_DELETE)\n-r  --restart       Force a restart of a clean MySQL instance (\\$MYSQL_RESTART)\n-s  --sample        Load sample Chinook database (\\$LOAD_SAMPLE)\"\n\nhelp_usage \"$@\"\n\ndocker_image=mysql\nport=\"\"\ndocker_opts=\"\"\n\npassword=\"${MYSQL_ROOT_PASSWORD:-${MYSQL_PWD:-${MYSQL_PASSWORD:-${PASSWORD:-test}}}}\"\n\nwhile [ $# -gt 0 ]; do\n    # DOCKER_NO_DELETE used by functions from lib\n    # shellcheck disable=SC2034\n    case \"$1\" in\n      -n| --name)   container_name=\"$2\"\n                    shift\n                    ;;\n      -p| --port)   port=\"$2\"\n                    [[ \"$port\" =~ ^[[:digit:]]*$ ]] || die \"invalid --port '$port' given\"\n                    shift\n                    ;;\n     -s|--sample)   LOAD_SAMPLE_DB=1\n                    ;;\n    -r|--restart)   MYSQL_RESTART=1\n                    ;;\n  -d|--no-delete)   DOCKER_NO_DELETE=1\n                    ;;\n               *)   version=\"$1\"\n                    ;;\n    esac\n    shift\ndone\n\ncontainer_name=\"${container_name:-${MYSQL_CONTAINER_NAME:-mysql}}\"\nversion=\"${version:-${MYSQL_VERSION:-latest}}\"\n\nif [ -n \"$port\" ]; then\n    docker_opts=\"-p $port:5432\"\nfi\n\ndb=\"$srcdir/chinook.mysql\"\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ] &&\n   ! [ -f \"$db\" ]; then\n    timestamp \"downloading sample 'chinook' database\"\n    wget -qcO \"$db\" 'https://github.com/lerocha/chinook-database/blob/master/ChinookDatabase/DataSources/Chinook_MySql.sql?raw=true'\n    #iconv -f ISO-8859-1 -t UTF-8 \"$db\" > \"$db.utf8\"\nfi\n\n# kill existing if we have specified a different version than is running\ndocker_image_version=\"$(docker_container_image \"$container_name\")\"\nif [ -n \"$docker_image_version\" ] &&\n   [ \"$docker_image_version\" != \"$docker_image:$version\" ]; then\n    MYSQL_RESTART=1\nfi\n\n# remove existing non-running container so we can boot a new one\nif docker_container_not_running \"$container_name\"; then\n    MYSQL_RESTART=1\nfi\n\nif [ -n \"${MYSQL_RESTART:-}\" ]; then\n    # ensures version is correct before we kill any existing test env to switch versions to minimize downtime\n    timestamp \"docker pull $docker_image:$version\"\n    docker_pull \"$docker_image:$version\"\n\n    timestamp \"killing existing MySQL container:\"\n    docker rm -f -- \"$container_name\" 2>/dev/null || :\nfi\n\nif ! docker_container_exists \"$container_name\"; then\n    timestamp \"booting MySQL container from image '$docker_image:$version':\"\n    # defined in lib/dbshell.sh\n    # shellcheck disable=SC2154,SC2086\n    docker run -d \\\n        --name \"$container_name\" \\\n        $docker_opts \\\n        -e MYSQL_ROOT_PASSWORD=\"$password\" \\\n        $docker_sql_mount_switches \\\n        \"$docker_image\":\"$version\"\n        #-v \"$srcdir/../setup/mysql/conf.d/my.cnf:/etc/mysql/conf.d/\" \\\n\nfi\n\nwait_for_mysql_ready \"$container_name\"\necho\n\ntimestamp \"linking shell profile for .my.cnf\"\ndocker exec \"$container_name\" bash -c \"cd /bash && setup/shell_link.sh &>/dev/null\" || :\n\n# yes expand now\n# shellcheck disable=SC2064\ntrap \"echo ERROR; echo; echo; [ -z '${DEBUG:-}' ] || docker logs '$container_name'\" EXIT\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ]; then\n    dbname=\"${db##*/}\"\n    dbname=\"${dbname%%.*}\"\n    timestamp \"loading $dbname database\"\n    #docker exec -i \"$container_name\" mysql -u root -p\"$password\" ${MYSQL_OPTS:-} -e \"CREATE DATABASE IF NOT EXISTS $dbname\"\n    timestamp \"loading data (this may take a minute)\"\n    # shellcheck disable=SC2086\n    docker exec -i -e MYSQL_PWD=\"$password\" \"$container_name\" mysql -u root ${MYSQL_OPTS:-} < \"${db}\"\n    timestamp \"done\"\n    echo >&2\nfi\n\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    cat <<EOF\n$shell_description\n\nEOF\nfi\n\n# cd to /sql to make sourcing easier without /sql/ path prefix\ndocker_exec_opts=\"-w /sql -i\"\n\n# allow non-interactive piped automation to avoid tty errors eg.\n# for sql in mysql*.sql; do echo \"source $sql\"; done | mysqld.sh\n# normally you would just 'mysqld.sh mysql*.sql' but this is used by mysql_test_scripts.sh\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    docker_exec_opts+=\" -t\"\nfi\n\n# want opt splitting\n# shellcheck disable=SC2086\ndocker exec -e MYSQL_PWD=\"$password\" $docker_exec_opts \"$container_name\" mysql -u root ${MYSQL_OPTS:-}\n\nuntrap\n\ndocker_rm_when_last_connection \"$0\" \"$container_name\"\n"
  },
  {
    "path": "packages/apk_filter_installed.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 00:05:26 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  #o pipefail  # not available in POSIX sh\nif [ \"${SHELL##*/}\" = bash ]; then\n    # shellcheck disable=SC2039\n    set -o pipefail\nfi\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2154\nusage(){\n    cat <<EOF\nChecks a given list of APK packages and returns those already installed\n\n$package_args_description\n\nTested on Alpine\n\n\nusage: ${0##*/} <packages>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\ninstalled_packages=\"$(mktemp)\"\ntrap 'rm -f -- \"$installed_packages\"' EXIT\n\ninstalled_apk > \"$installed_packages\"\n\nprocess_package_args \"$@\" |\ngrep -Fx -f \"$installed_packages\" || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/apk_filter_not_installed.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 00:05:26 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  #o pipefail  # not available in POSIX sh\nif [ \"${SHELL##*/}\" = bash ]; then\n    # shellcheck disable=SC2039\n    set -o pipefail\nfi\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2154\nusage(){\n    cat <<EOF\nChecks a given list of APK packages and returns those not installed\n\n$package_args_description\n\nTested on Alpine\n\n\nusage: ${0##*/} <packages>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\ninstalled_packages=\"$(mktemp)\"\ntrap 'rm -f -- \"$installed_packages\"' EXIT\n\ninstalled_apk > \"$installed_packages\"\n\nprocess_package_args \"$@\" |\ngrep -vFx -f \"$installed_packages\" || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/apk_install_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install Apk packages in a forgiving way - useful for installing Perl CPAN and Python PyPI modules that may or may not be available\n#\n# combine with later use of the following scripts to only build packages that aren't available in the Linux distribution:\n#\n# perl_cpanm_install_if_absent.sh\n# python_pip_install_if_absent.sh\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils-bourne.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\nusage(){\n    echo \"Installs Alpine APK package lists\"\n    echo\n    echo \"Takes a list of apk packages as arguments or via stdin, and for any arguments that are plaintext files, reads the packages from those given files (one package per line)\"\n    echo\n    echo \"usage: ${0##*/} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor x in \"$@\"; do\n    case \"$x\" in\n        -*) usage\n            ;;\n    esac\ndone\n\necho \"Installing Apk Packages\"\n\npackages=\"\"\n\nprocess_args(){\n    for arg in \"$@\"; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"$packages\" ]; then\n    exit 0\nfi\n\n# uniq\npackages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\n\necho \"Packages to be installed:\"\necho\necho \"$packages\" | tr ' ' '\\n'\necho\n\nopts=\"\"\nif is_CI; then\n    #opts=\"--quiet\"  # doesn't print packages installed but still has a progress bar\n    opts=\"--no-progress\"  # prints packages installed but not progress bar filling up logs\nfi\n\n# sudo set in lib/utils-bourne.sh\n# shellcheck disable=SC2154\n[ -n \"${NO_UPDATE:-}\" ] || $sudo apk update $opts\n\n# [[ ]] and <<< not available in sh\n#if echo \"$packages\" | grep -q openssl-dev; then\n#    if apk info | grep -q libressl-dev; then\n#        echo \"openssl-dev is incompatible with currently installed libressl-dev, trying to uninstall libressl-dev before proceeding...\"\n#        apk del libressl-dev  # will break if mariadb-dev is installed, this probably isnt't the right place to do this anyway...\n#    fi\n#fi\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    for package in $packages; do\n        $sudo apk add $opts \"$package\" || :\n    done\nelse\n    # shellcheck disable=SC2086\n    $sudo apk add $opts $packages\nfi\n"
  },
  {
    "path": "packages/apk_install_packages_if_absent.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2154\nusage(){\n    cat <<EOF\nInstalls Alpine APK package lists if the packages aren't already installed\n\n$package_args_description\n\nTested on Alpine\n\n\nusage: ${0##*/} <packages>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\n\nprocess_package_args \"$@\" |\n\"$srcdir/apk_filter_not_installed.sh\" |\nxargs -r \"$srcdir/apk_install_packages.sh\"\n"
  },
  {
    "path": "packages/apk_remove_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Remove Apk packages in a forgiving way - useful for uninstalling development packages no longer needed eg. to minimize size of Docker images\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"Removing Apk Packages\"\n\npackages=\"\"\nfor arg; do\n    if [ -f \"$arg\" ]; then\n        echo \"adding packages from file:  $arg\"\n        packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n        echo\n    else\n        packages=\"$packages $arg\"\n    fi\n    # uniq\n    packages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\ndone\n\nif [ -z \"$packages\" ]; then\n    exit 0\nfi\n\nSUDO=\"\"\n# $EUID isn't available in /bin/sh in Alpine\n# shellcheck disable=SC2039\n[ \"${EUID:-$(id -u)}\" != 0 ] && SUDO=sudo\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    # shellcheck disable=SC2086\n    if ! $SUDO apk del $packages; then\n        for package in $packages; do\n            $SUDO apk del \"$package\" || :\n        done\n    fi\nelse\n    # shellcheck disable=SC2086\n    $SUDO apk del $packages\nfi\n"
  },
  {
    "path": "packages/apk_upgrade_packages_if_outdated.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-02 23:14:31 -0300 (Mon, 02 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2154\nusage(){\n    cat <<EOF\nUpgrades Alpine APK package lists if the packages are outdated\n\nThis is slow because it has to do an 'apk update' to get the latest package list first\nwhich can take 15 seconds if you're not on a fast connection\n\n$package_args_description\n\nNot yet tested on Alpine as new docker images have no outdated packages\n\n\nusage: ${0##*/} <packages>\nEOF\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*)     usage\n                ;;\n    esac\ndone\n\napk update\n\nupgradeable_packages=\"$(apk version -l '<')\"\n\nprocess_package_args \"$@\" |\nwhile read -r package; do\n    if echo \"$upgradeable_packages\" | grep -q \"^$package-[[:digit:]]\"; then\n        echo \"$package\"\n    fi\ndone |\nxargs -r apk upgrade\n"
  },
  {
    "path": "packages/apt_install_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install Deb packages in a forgiving way - useful for installing Perl CPAN and Python PyPI modules that may or may not be available\n#\n# combine with later use of the following scripts to only build packages that aren't available in the Linux distribution:\n#\n# perl_cpanm_install_if_absent.sh\n# python_pip_install_if_absent.sh\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils-bourne.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\nusage(){\n    echo \"Installs Debian / Ubuntu deb package lists\"\n    echo\n    echo \"Takes a list of deb packages as arguments or via stdin, and for any arguments that are plaintext files, reads the packages from those given files (one package per line)\"\n    echo\n    echo \"usage: ${0##*/} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\necho \"Installing Deb Packages\"\n\nexport NEEDRESTART_MODE=a\nexport DEBIAN_FRONTEND=noninteractive\n\n#apt=\"apt\"\napt=\"apt-get\"\n\nif ! type \"$apt\" >/dev/null 2>&1; then\n    echo \"$apt not found in \\$PATH ($PATH), cannot install apt packages!\"\n    exit 1\nfi\n\nopts=\"-o DPkg::Lock::Timeout=1200\"\nif [ -f /.dockerenv ]; then\n    echo \"running inside docker, not installing recommended extra packages unless specified to save space\"\n    opts=\"$opts --no-install-recommends\"\nfi\nif is_CI; then\n    echo \"running in CI quiet mode\"\n    opts=\"$opts -q\"\n    echo\n    echo \"/etc/apt/sources.list:\"\n    cat /etc/apt/sources.list || :\n    echo\n    for x in /etc/apt/sources.list.d/*; do\n        [ -f \"$x\" ] || continue\n        echo \"$x:\"\n        cat \"$x\"\n        echo\n    done\nfi\n#if is_semmle; then\n#    # sudo: no tty present and no askpass program specified\n#    echo \"Semmle detected, not running package installs as this is a docker user environment without sudo privs or tty\"\n#    exit 0\n#fi\n\npackages=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"$packages\" ]; then\n    exit 0\nfi\n\n# uniq\npackages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\n\necho\necho \"Packages to be installed:\"\necho\necho \"$packages\" | tr ' ' '\\n'\necho\n\n# requires fuser which might not already be installed, catch-22 situation if wanting to use this for everything including bootstraps\n#\"$srcdir/apt_wait.sh\"\n\n# sudo set in lib/utils-bourne.sh\n# want splitting of $opts\n# shellcheck disable=SC2154,SC2086\n[ -n \"${NO_UPDATE:-}\" ] || $sudo \"$apt\" update $opts\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    # shellcheck disable=SC2086\n    for package in $packages; do\n        #\"$srcdir/apt_wait.sh\"\n        $sudo \"$apt\" install -y $opts \"$package\" || :\n    done\nelse\n    #\"$srcdir/apt_wait.sh\"\n    # shellcheck disable=SC2086\n    $sudo \"$apt\" install -y $opts $packages\nfi\n"
  },
  {
    "path": "packages/apt_install_packages_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 18:38:39 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Debian / Ubuntu deb package lists if the packages aren't already installed\n\n$package_args_description\n\nTested on Debian and Ubuntu\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nexport DEBIAN_FRONTEND=noninteractive\n\nprocess_package_args \"$@\" |\n\"$srcdir/debs_filter_not_installed.sh\" |\nxargs --no-run-if-empty \"$srcdir/apt_install_packages.sh\"\n"
  },
  {
    "path": "packages/apt_remove_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Remove and Purge Deb packages in a forgiving way - useful for uninstalling development packages no longer needed eg. to minimize size of Docker images\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\necho \"Removing Deb Packages\"\n\nexport DEBIAN_FRONTEND=noninteractive\n\n#apt=\"apt\"\napt=\"apt-get\"\n\nif ! type \"$apt\" >/dev/null 2>&1; then\n    echo \"$apt not found in \\$PATH ($PATH), cannot install apt packages!\"\n    exit 1\nfi\n\nopts=\"--no-install-recommends\"\nif is_CI; then\n    echo \"running in CI quiet mode\"\n    opts=\"$opts -qq\"\nfi\n\npackages=\"\"\nfor arg; do\n    if [ -f \"$arg\" ]; then\n        echo \"adding packages from file:  $arg\"\n        packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n        echo\n    else\n        packages=\"$packages $arg\"\n    fi\n    # uniq\n    packages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\ndone\n\nif [ -z \"$packages\" ]; then\n    exit 0\nfi\n\nSUDO=\"\"\n# shellcheck disable=SC2039\n[ \"${EUID:-$(id -u)}\" != 0 ] && SUDO=sudo\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    # shellcheck disable=SC2086\n    if ! $SUDO \"$apt\" purge -y $packages; then\n        for package in $packages; do\n            $SUDO \"$apt\" purge -y \"$package\" || :\n        done\n    fi\nelse\n    # shellcheck disable=SC2086\n    $SUDO \"$apt\" purge -y $packages\nfi\n"
  },
  {
    "path": "packages/apt_set_lock_timeout.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-15 03:01:21 +0800 (Sat, 15 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConfigures APT lock timeouts so APT install commands don't just error out\n\nTimeout seconds defaults to 1200 if not specified\n\nCreates or appends to file:\n\n    /etc/\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<seconds>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nseconds=\"${1:-1200}\"\n\nsetting=\"DPkg::Lock::Timeout\"\n\nline=\"$setting \\\"$seconds\\\";\"\n\nfile=\"/etc/apt/apt.conf.d/99timeout\"\n\nif grep -q \"^[[:space:]]*${setting}[[:space:]]\" \"$file\" 2>/dev/null; then\n    sed -i \"s/[[:space:]]*$setting.*/$line/\" \"$file\"\nelse\n    echo \"$line\" >> \"$file\"\nfi\n"
  },
  {
    "path": "packages/apt_upgrade_packages_if_outdated.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-02 23:48:50 -0300 (Mon, 02 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpgrades Debian / Ubuntu deb package lists if the packages are outdated\n\nRefuses to upgrade/install packages which aren't already installed\n\n$package_args_description\n\nTested on Debian with apt\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nexport DEBIAN_FRONTEND=noninteractive\n\napt update\n\n# unlike other distros, it's faster and easier to just run the upgrade on the packages and let apt figure it out\n#upgradeable_packages=\"$(apt list --upgradable)\"\n\nprocess_package_args \"$@\" |\n#while read -r package; do\n#    if echo \"$upgradeable_packages\" | grep -Eq \"^$package(/|[[:space:]])\"; then\n#        echo \"$package\"\n#    fi\n#done |\nxargs --no-run-if-empty apt install --only-upgrade -y\n"
  },
  {
    "path": "packages/apt_wait.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-18 20:03:00 +0100 (Thu, 18 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Blocking wait on apt-get locks to allow multiple apt-get calls to wait instead of exiting with errors\n#\n# Not really a new idea, there are similar implementations eg.\n#\n# https://gist.github.com/tedivm/e11ebfdc25dc1d7935a3d5640a1f1c90\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nsleep_secs=1\n\nlocks=\"\n/var/lib/dpkg/lock\n/var/lib/apt/lists/lock\n\"\n\nunattended_upgrade_log=\"/var/log/unattended-upgrades/unattended-upgrades.log\"\n\nsudo=\"\"\n[ $EUID = 0 ] || sudo=sudo\n\ncheck_bin(){\n    if ! type -P \"$1\" &>/dev/null; then\n        echo \"$1 not found in \\$PATH ($PATH)\" >&2\n        exit 1\n    fi\n}\n\ncheck_bin apt-get\ncheck_bin fuser\n\nif [ -n \"$sudo\" ]; then\n    check_bin \"$sudo\"\nfi\n\nwhile true; do\n    for lock in $locks; do\n        if $sudo fuser \"$lock\" &>/dev/null; then\n            echo \"apt lock in use ($lock), waiting...\" >&2\n            sleep \"$sleep_secs\"\n            continue 2\n        fi\n    done\n    if [ -f \"$unattended_upgrade_log\" ] &&\n       $sudo fuser \"$unattended_upgrade_log\" &>/dev/null; then\n        echo \"apt unattended upgrade log in use ($unattended_upgrade_log), waiting...\" >&2\n        sleep \"$sleep_secs\"\n        continue\n    fi\n    break\ndone\n"
  },
  {
    "path": "packages/brew_filter_in_setup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: vim bash git\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 12:12:43 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Mac Homebrew packages and returns those that are already recorded in setup/brew-packages-*.txt\n\n$package_args_description\n\nSupports TAP=1 environment variables for checking taps list instead\n\nTested on Mac Homebrew\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\n# process_package_args requires much more specific env var to disambiguate\nHOMEBREW_PACKAGES_TAP=\"${TAP:-}\"\n\nprocess_package_args \"$@\" |\nif [ -n \"${HOMEBREW_PACKAGES_TAP:-}\" ]; then\n    while read -r tap package; do\n        grep -Eq \"^${tap}[[:space:]]+$package$\" <(sed 's/#.*//; /^[[:digit:]]*$/d' \"$srcdir/../setup/\"brew-packages*taps.txt) &&\n        echo \"$tap $package\"\n    done\nelse\n    # do not quote cask, blank quotes break shells and there will never be any token splitting anyway\n    # shellcheck disable=SC2046\n    tr ' ' '\\n' |\n    # Mac's grep is buggy, doesn't matches utimer unless sort -r to try it before '^r$' - but then gives false positives on other packages\n    #grep -Fx -f <(sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' \"$srcdir/../setup/\"brew-packages*.txt | sort -ur)\n    command ggrep -Fx -f <(sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' \"$srcdir/../setup/\"brew-packages*.txt)\nfi\n"
  },
  {
    "path": "packages/brew_filter_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 00:42:27 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Mac Homebrew packages and returns those already installed\n\n$package_args_description\n\nSupport TAP=1 or CASK=1 environment variables for checking taps or casks respectively\n\nTested on Mac Homebrew\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\n# process_package_args requires much more specific env var to disambiguate\nHOMEBREW_PACKAGES_TAP=\"${TAP:-}\"\n\nprocess_package_args \"$@\" |\nif [ -n \"${HOMEBREW_PACKAGES_TAP:-}\" ]; then\n    if [ -n \"${NO_FAIL:-}\" ]; then\n        set +e\n    fi\n    installed_packages=\"$(brew list)\"\n    while read -r tap package; do\n        set +e  # grep causes pipefail exit code breakages in calling code when it doesn't match\n        grep -Fxq \"$package\" <<< \"$installed_packages\" &&\n        echo \"$tap $package\"\n    done\nelse\n    # do not quote cask, blank quotes break shells and there will never be any token splitting anyway\n    # shellcheck disable=SC2046\n    tr ' ' '\\n' |\n    grep -Fx -f <(brew $([ -z \"${CASK:-}\" ] || echo cask) list) || :  # grep causes pipefail exit code breakages in calling when it doesn't match\nfi\n"
  },
  {
    "path": "packages/brew_filter_not_in_setup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: vim bash git\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 12:12:43 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Mac Homebrew packages and returns those that are not recorded in setup/brew-packages-*.txt\n\n$package_args_description\n\nSupports TAP=1 environment variables for checking taps list instead\n\nTested on Mac Homebrew\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\n# process_package_args requires much more specific env var to disambiguate\nHOMEBREW_PACKAGES_TAP=\"${TAP:-}\"\n\nprocess_package_args \"$@\" |\nif [ -n \"${HOMEBREW_PACKAGES_TAP:-}\" ]; then\n    while read -r tap package; do\n        grep -Eq \"^#?${tap}[[:space:]]+$package$\" <(sed 's/#.*//; /^[[:digit:]]*$/d' \"$srcdir/../setup/\"brew-packages*taps.txt) ||\n        echo \"$tap $package\"\n    done\nelse\n    # do not quote cask, blank quotes break shells and there will never be any token splitting anyway\n    # shellcheck disable=SC2046\n    tr ' ' '\\n' |\n    # Mac's grep is buggy, doesn't matches utimer unless sort -r to try it before '^r$' - but then gives false positives on other packages\n    #grep -vFx -f <(sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' \"$srcdir/../setup/\"brew-packages*.txt | sort)\n    command ggrep -vFx -f <(sed 's/#.*//; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' \"$srcdir/../setup/\"brew-packages*.txt) |\n    while read -r package; do\n        grep -Eqi \"^#${package}([[:space:]]|$)\" \"$srcdir/../setup/\"brew-packages*.txt || echo \"$package\"\n    done |\n    sort -u\nfi\n"
  },
  {
    "path": "packages/brew_filter_not_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 00:42:27 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Mac Homebrew packages and returns those not installed\n\n$package_args_description\n\nSupports TAP=1 or CASK=1 environment variables for checking taps or casks respectively\n\nTested on Mac Homebrew\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nprocess_package_args \"$@\" |\nif [ -n \"${TAP:-}\" ]; then\n    if [ -n \"${NO_FAIL:-}\" ]; then\n        set +e\n    fi\n    installed_packages=\"$(brew list)\"\n    while read -r tap package; do\n        set +e  # grep causes pipefail exit code breakages in calling code when it doesn't match\n        grep -Fxq \"$package\" <<< \"$installed_packages\" ||\n        echo \"$tap $package\"\n    done\nelse\n    # do not quote cask, blank quotes break shells and there will never be any token splitting anyway\n    # shellcheck disable=SC2046\n    tr ' ' '\\n' |\n    grep -vFx -f <(brew $([ -z \"${CASK:-}\" ] || echo cask) list) || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\nfi\n"
  },
  {
    "path": "packages/brew_install_packages.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2086\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Mac OSX - HomeBrew install packages in a forgiving way\n\nset -eu #o pipefail  # undefined in /bin/sh\n[ -n \"${DEBUG:-}\" ] && set -x\n\nusage(){\n    echo \"Installs Mac Homebrew package lists\"\n    echo\n    echo \"Takes a list of brew packages as arguments or .txt files containing lists of packages (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor x in \"$@\"; do\n    case \"$x\" in\n        -*) usage\n            ;;\n    esac\ndone\n\necho \"Installing Mac HomeBrew Packages\"\n\npackages=\"\"\n\nprocess_args(){\n    for arg in \"$@\"; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n        if [ -z \"${TAP:-}\" ]; then\n            # uniq\n            packages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\n# Sudo is not required as running Homebrew as root is extremely dangerous and no longer supported as\n# Homebrew does not drop privileges on installation you would be giving all build scripts full access to your system\n\nbrew_update_opts=\"\"\nif [ -n \"${TRAVIS:-}\" ]; then\n    brew_update_opts=\"-v\"\nfi\nif [ -z \"${NO_UPDATE:-}\" ]; then\n    if [ -n \"${NO_FAIL:-}\" ]; then\n        set +e #o pipefail  # undefined in /bin/sh\n    fi\n    echo \"Updating Homebrew\"\n    # brew update takes a long time and doesn't output anything so background it and output progress dots every 5 secs\n    # to make sure Travis CI doesn't terminate the job for lack of output activity after 10 mins (used to fail builds)\n    brew update $brew_update_opts &\n    while jobs | grep -Eq '[[:space:]]+Running[[:space:]]+brew[[:space:]]+update'; do\n        # /bin/sh doesn't support -e\n        #echo -n .\n        printf .\n        sleep 5\n    done\n    set -e #o pipefail  # undefined in /bin/sh\nfi\n\nif [ -n \"${TAP:-}\" ]; then\n    # convert to array\n    # need splitting\n    # shellcheck disable=SC2206\n    packages_array=($packages)\n    if [ -n \"${NO_FAIL:-}\" ]; then\n        set +e\n    fi\n    for((i=0; i < ${#packages_array[@]}; i+=2)); do\n        tap=\"${packages_array[$i]}\"\n        package=\"${packages_array[(($i+1))]}\"\n        brew tap \"$tap\"\n        brew install \"$package\"\n    done\n    exit\nelse\n    packages=\"$(tr ' ' '\\n' <<< \"$packages\" | sort -u | tr '\\n' ' ')\"\nfi\n\nopts=\"\"\nif [ -n \"${CASK:-}\" ]; then\n    opts=\"--cask\"\nfi\n\necho\necho \"Packages to be installed:\"\necho\ntr ' ' '\\n' <<< \"$packages\"\necho\n\n# Fails if any of the packages are already installed, so you'll probably want to ignore and continue and detect missing\n# package later in build system if it's a problem eg. resulting in missing headers later in build\nif [ -n \"${NO_FAIL:-}\" ]; then\n    for package in $packages; do\n        brew install $opts \"$package\" || :\n    done\nelse\n    # want splitting\n    # shellcheck disable=SC2086\n    brew install $opts $packages\nfi\n"
  },
  {
    "path": "packages/brew_install_packages_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2086\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 19:03:51 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Mac OSX - HomeBrew install packages in a forgiving way\n\nset -eu #o pipefail  # undefined in /bin/sh\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Mac Homebrew package lists if the packages aren't already installed\n\n$package_args_description\n\nTested on macOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nprocess_package_args \"$@\" |\n\"$srcdir/brew_filter_not_installed.sh\" |\ngxargs --no-run-if-empty \"$srcdir/brew_install_packages.sh\"\n"
  },
  {
    "path": "packages/brew_package_owns.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-08-08 16:16:51 +0300 (Thu, 08 Aug 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds which Homebrew package owns a given file on Mac\n\nBrew doesn't have a command to do this\n\nFirst checks if the argument is a binary on the \\$PATH, and if so attempts to find it in the Homebrew base Cellar\n\nIf it is it does a cheap parse and print of the package name\n\nDoes the same for NodeJS modules under Homebrew's lib/node_modules/\n\nIf it cannot find it the cheap way in those directories then prints a warning\nand continues to iterates all packages one by one to find it, using the argument as a grep ERE regex\n\nThis is a very expensive O(n) operation and should be a last resort!\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nexecutable_or_regex=\"$1\"\n\n# must not has a trailing slash for later regex matching which appends a slash\nbrew_basedir=\"/opt/homebrew\"\n\npath=\"$(which \"$executable_or_regex\" 2>/dev/null || :)\"\nif [ -n \"$path\" ]; then\n    absolute_path=\"$(greadlink -f \"$path\")\"\n\n    if [[ \"$absolute_path\" =~ ^$brew_basedir/Cellar/ ]]; then\n        # false positive, this correctly interpolates in Bash on Mac\n        # shellcheck disable=SC2295\n        package=\"${absolute_path##${brew_basedir}/Cellar/}\"\n        package=\"${package%%/*}\"\n        if [ -n \"$package\" ]; then\n            echo \"$package\"\n            exit 0\n        fi\n    fi\n\n    if [[ \"$absolute_path\" =~ ^$brew_basedir/lib/node_modules/ ]]; then\n        # false positive, this correctly interpolates in Bash on Mac\n        # shellcheck disable=SC2295\n        package=\"${absolute_path##${brew_basedir}/lib/node_modules/}\"\n        package=\"${package%%/*}\"\n        if [ -n \"$package\" ]; then\n            echo \"NodeJS package: $package\"\n            exit 0\n        fi\n    fi\n\nfi\n\nwarn \"failed to find '$executable_or_regex' as a binary in the Homebrew Cellar\"\nwarn \"now proceeding to a very slow expensive iterative search of every installed brew package\"\n\nbrew list |\nwhile read -r package; do\n    # trade a small amount of RAM to detect when this command gets a Control-C\n    # so that we don't have to keep hitting Control-C to break execution of this script\n    # false positive - actually breaks out of the loop in Bash on Mac\n    # shellcheck disable=SC2106\n    package_contents=\"$(brew list \"$package\" || break)\"\n    if grep -Eq \"$executable_or_regex\" <<< \"$package_contents\"; then\n        echo \"$package\"\n    fi\ndone\n"
  },
  {
    "path": "packages/brew_upgrade_packages.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 21:54:59 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows all outdated Homebrew packages and prompts to upgrade them\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\ntimestamp \"updating Homebrew's package list\"\n\nbrew update\n\necho\n\ntimestamp \"Listing outdated brew packages\"\n\necho\n\nbrew outdated\n\necho\n\nread -r -p \"Continue? (Y/n) \" answer\n\nshopt -s nocasematch\nif ! [[ \"$answer\" =~ ^(y|yes)?$ ]]; then\n    exit 1\nfi\n\necho\n\ntimestamp \"Upgrading brew packages\"\n\nbrew upgrade\n\necho\n\ntimestamp \"Cleaning up (removing cached downloads)\"\n\necho\n\n# lots of 'Warning: Skipping <name>: most recent version <x.y.z> not installed'\nbrew cleanup -s 2>/dev/null || :\n"
  },
  {
    "path": "packages/brew_upgrade_packages_if_outdated.sh",
    "content": "#!/usr/bin/env bash\n#  shellcheck disable=SC2086\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 19:03:51 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Mac OSX - HomeBrew upgrades packages if they're outdated\n\nset -eu #o pipefail  # undefined in /bin/sh\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpgrades Mac Homebrew package lists if the packages are outdated\n\nIgnores packages that aren't recognized, outputting a warning to stderr\n\n$package_args_description\n\nTested on macOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nprocess_package_args \"$@\" |\nwhile read -r package; do\n    output=\"$(brew outdated \"$package\" 2>&1)\"\n    result=$?\n    if [ \"$result\" = 0 ]; then\n        continue\n    fi\n    if [[ \"$output\" =~ \"Error: No available formula\" ]]; then\n        echo \"$output\" >&2\n        continue\n    fi\n    echo \"$package\"\ndone |\ngxargs --no-run-if-empty brew upgrade\n"
  },
  {
    "path": "packages/debs_filter_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:25:39 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Deb packages and returns those already installed\n\n$package_args_description\n\nTested on Debian and Ubuntu\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nexport DEBIAN_FRONTEND=noninteractive\n\nprocess_package_args \"$@\" |\ngrep -Fx -f <(installed_debs) || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/debs_filter_not_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:25:39 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of Deb packages and returns those not installed\n\n$package_args_description\n\nTested on Debian and Ubuntu\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nexport DEBIAN_FRONTEND=noninteractive\n\nprocess_package_args \"$@\" |\ngrep -vFx -f <(installed_debs) || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/golang_install.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ngo=\"${GO:-go}\"\n\nusage(){\n    echo \"Installs Golang tools, taking in to account library paths\"\n    echo\n    echo \"Takes a list of go tool names as arguments or .txt files containing lists of tools (one per line)\"\n    echo\n    echo \"To skip Golang tools that fail to install, set this in environment:\"\n    echo\n    echo \"export GOLANG_SKIP_FAILURES=1\"\n    echo\n    echo \"usage: ${0##*} <list_of_tools>\"\n    echo\n    exit 3\n}\n\nfor x in \"$@\"; do\n    case \"$x\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ngo_tools=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding golang tools from file:  $arg\"\n            go_tools=\"$go_tools $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            go_tools=\"$go_tools $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${go_tools// }\" ]; then\n    usage\nfi\n\ngo_tools=\"$(tr ' ' ' \\n' <<< \"$go_tools\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing Golang tools\"\necho\n\nopts=\"\"\nif [ -n \"${TRAVIS:-}\" ]; then\n    echo \"running in quiet mode\"\n    opts=\"-q\"\nfi\n\nenvopts=\"\"\nif [ \"$(uname -s)\" = \"Darwin\" ]; then\n    if type -P brew &>/dev/null; then\n        # usually /usr/local\n        brew_prefix=\"$(brew --prefix)\"\n        # needed to build Crypt::SSLeay\n        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n        # need to send OPENSSL_INCLUDE and OPENSSL_LIB through sudo explicitly using prefix\n        envopts=\"OPENSSL_INCLUDE=$OPENSSL_INCLUDE OPENSSL_LIB=$OPENSSL_LIB\"\n    fi\nfi\n\nfor go_tool in $go_tools; do\n    go_tool=\"${go_tool#http?://}\"\n    if ! [[ \"$go_tool\" =~ @ ]]; then\n        go_tool+=\"@latest\"\n    fi\n    echo \"$envopts $go install $opts $go_tool\"\n    # want splitting of opts and tools\n    # shellcheck disable=SC2086\n    if [ -n \"${GOLANG_SKIP_FAILURES:-}\" ]; then\n        set +eo pipefail\n        eval $envopts \"$go\" install $opts \"$go_tool\"\n        set +eu pipefail\n    fi\ndone\n"
  },
  {
    "path": "packages/golang_install_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    echo \"Installs Golang tools not already installed\"\n    echo\n    echo \"Leverages adjacent golang_install.sh which takes in to account library paths etc\"\n    echo\n    echo \"Takes a list of go tool names as arguments or .txt files containing lists of tools (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_tools>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ngo_tools=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding golang tools from file:  $arg\"\n            go_tools=\"$go_tools $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            go_tools=\"$go_tools $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${go_tools// }\" ]; then\n    usage\nfi\n\ngo_tools=\"$(tr ' ' ' \\n' <<< \"$go_tools\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing Golang tools that are not already installed\"\necho\n\nif [ -n \"${GOPATH:-}\" ]; then\n    export PATH=\"$PATH:$GOPATH/bin\"\nfi\n\nfor go_tool in $go_tools; do\n    go_bin=\"${go_tool##*/}\"\n    path=\"$(type -P \"$go_bin\" 2>/dev/null || :)\"\n    if [ -n \"$path\" ]; then\n        echo \"go tool '$go_tool' ($go_bin => $path) already installed, skipping...\"\n    else\n        echo \"installing go tool '$go_tool'\"\n        echo\n        \"$srcdir/golang_install.sh\" \"$go_tool\"\n    fi\ndone\n"
  },
  {
    "path": "packages/golang_rm_binaries.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-15 17:27:41 +0100 (Fri, 15 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Finds binaries adjacent to Golang .go source files in given directories and removes them\n#\n# Doesn't remove $GOPATH/bin stuff, only adjacent .go compiles\n# because $GOPATH/bin stuff is often 'go get' downloaded programs that we want to keep\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n#srcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/utils.sh\"\n\nbasedir=\"${1:-$PWD}\"\n\necho\necho \"Removing Golang binaries adjacent to .go files under $basedir:\"\necho\n\n#directories=\"$(\n#    find \"$basedir\" -type f -name '*.go' |\n#    sed 's/[^/].*.go$//' |\n#    sort -u\n#)\"\n\n#while read -r directory; do\n#done <<< \"$directories\"\n\nrm_if_binary(){\n    set +o pipefail\n    file --mime \"$1\" |\n    # can't use -l because it gives (standard input) instead of filename,\n    # must get the filename from the file --mime output instead\n    grep 'charset=binary' |\n    grep -v '[[:space:]]inode/directory;[[:space:]]' |\n    sed 's/:.*//' |\n    xargs rm -fv --\n}\n\n# Finds and removes 'foo' binary for 'foo.go' adjacent compiled programs\nfind \"$basedir\" -type f -name '*.go' |\ngrep -Eve '/src/github\\.com/' -e '/src/golang\\.org/' |\nwhile read -r filename; do\n    filename=\"${filename%.go}\"\n    rm_if_binary \"$filename\"\ndone\n\ncheck_src_dir(){\n    for x in \"$1\"/*; do\n        if ! [ -d \"$x\" ]; then\n            continue\n        fi\n        binfile=\"$x/${x##*/}\"\n        if [ -f \"$binfile\" ]; then\n            rm_if_binary \"$binfile\"\n        fi\n    done\n}\n\nfind \"$basedir\" -type d -name 'src' |\nwhile read -r src_dir; do\n    check_src_dir \"$src_dir\"\ndone\n\nif [ \"src\" = \"$(basename \"$(cd \"$basedir\" && pwd)\")\" ]; then\n    check_src_dir \"$basedir\"\nfi\n"
  },
  {
    "path": "packages/install_binary.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 'https://s3.amazonaws.com/cloudbees-core-cli/master/cloudbees-{os}-{arch}.tar.gz' cloudbees\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 11:25:17 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and installs a binary from a given URL and extraction path to /usr/local/bin if run as root or ~/bin if run as user\n\nThe URL can be parameterized with {os} and {arch} tokens to be replaced by the current OS (linux/darwin) or Architecture (amd64/arm)\n\nEnvironment overrides for {os} and {arch} if their values are Darwin, Linux, x86_64 or i386 respectively, in this order of priority:\n\nexport OS_DARWIN=macOS    # because some packages use 'macOS' while others use 'darwin'\nexport OS_LINUX=Linux     # because some packages titlecase this\n\nexport ARCH_X86_64=amd64  # because some packages use 'amd64' instead of 'x86_64'\nexport ARCH_X86=x86       # because some packages use 'x86' instead of 'i386'\nexport ARCH_ARM64=ARM64   # because some packages use uppercase 'ARM64' instead of 'arm64'\nexport ARCH_ARM=ARM       # because some packages use uppercase 'ARM' instead of 'arm'\nexport ARCH_OVERRIDE=all  # because some packages use a random universal arch rather than 'x86_64' or 'i386'\n\nIf a zip or tarball is given, will be unpacked in /tmp and the binary path specified will be copied to bin\n\nAn optional binary destination can be given to name the file - if the file name is an absolute path will place it there instead of /usr/local/bin or ~/bin\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<url> [<binary_path_in_zip_or_tarball> <install_path>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl=\"$1\"\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\n\nurl=\"${url//\\{os\\}/$os}\"\nurl=\"${url//\\{arch\\}/$arch}\"\n\npackage=\"${url##*/}\"\ntmp=\"/tmp/install_binary.$$\"\ndownload_file=\"$tmp/$package\"\n\nif [[ \"$package\" =~ \\.zip$ ]] || has_tarball_extension \"$package\"; then\n    if [ $# -lt 2 ]; then\n        usage \"binary file path must be specified if downloading a tarball or zip file ('$package')\"\n    fi\n    binary=\"$2\"\n    binary=\"${binary//\\{os\\}/$os}\"\n    binary=\"${binary//\\{arch\\}/$arch}\"\nfi\n\nmkdir -p -v \"$tmp\"\n\ntrap_cmd \"rm -f -- '$download_file'\"\n\ncd \"$tmp\"\n\ntimestamp \"Downloading: $url\"\ndownload \"$url\" \"$download_file\"\n\nif has_tarball_extension \"$package\"; then\n    timestamp \"Extracting tarball package\"\n    if ! type -P tar &>/dev/null; then\n        \"$srcdir/install_packages.sh\" tar\n    fi\n    if has_tarball_gzip_extension \"$package\"; then\n        tar xvzf \"$download_file\"\n    elif has_tarball_bzip2_extension \"$package\"; then\n        tar xvjf \"$download_file\"\n    fi\n    if ! [ -f \"$binary\" ]; then\n        die \"Failed to find binary '$binary' in unpacked tarball '$download_file' - is the given binary filename / path correct?\"\n    fi\n    download_file=\"$binary\"\n    echo\nelif [[ \"$package\" =~ \\.zip$ ]]; then\n    timestamp \"Extracting zip package\"\n    if ! type -P unzip &>/dev/null; then\n        \"$srcdir/install_packages.sh\" unzip\n    fi\n    unzip -o \"$download_file\"\n    if ! [ -f \"$binary\" ]; then\n        die \"Failed to find binary '$binary' in unpacked zip '$download_file' - is the given binary filename / path correct?\"\n    fi\n    download_file=\"$binary\"\nfi\n\ntimestamp \"Setting executable: $download_file\"\nchmod +x \"$download_file\"\necho\n\ndestination=\"${3:-${2:+${2##*/}}}\"\nif [ -z \"$destination\" ]; then\n    destination=\"${download_file##*/}\"\n    # no longer suffixing with $$, used in $tmp instead\n    #destination=\"${destination%%.$$}\"\n    # if there are any -darwin-amd64 or -amd64-darwin suffixes remove them either way around (this is why $os is stripped before and after)\n    destination=\"${destination%%-$os}\"\n    destination=\"${destination%%_$os}\"\n    destination=\"${destination%%-$arch}\"\n    destination=\"${destination%%_$arch}\"\n    destination=\"${destination%%-x86_64}\"  # $arch is amd64, we must check to strip this explicitly extra\n    destination=\"${destination%%_x86_64}\"\n    destination=\"${destination%%-$os}\"\n    destination=\"${destination%%_$os}\"\nfi\n\nif ! [[ \"$destination\" =~ ^/ ]]; then\n    if am_root || [ -w /usr/local/bin ]; then\n        destination=\"/usr/local/bin/$destination\"\n    else\n        destination=~/bin/\"$destination\"\n    fi\nfi\ninstall_path=\"${destination%/*}\"\nif [ -e \"$install_path\" ] && ! [ -d \"$install_path\" ]; then\n    die \"ERROR: install path $install_path is not a directory, aborting!\"\nfi\nmkdir -p -v \"$install_path\"\n#echo\n\ntimestamp \"Moving to install dir:\"\n# common alias mv='mv -i' would force a prompt we don't want, even with -f\nunalias mv &>/dev/null || :\nmv -fv -- \"$download_file\" \"$destination\"\necho\n\ntimestamp \"Installation complete\"\n\nif [ -n \"${RUN_VERSION_ARG:-}\" ]; then\n    echo\n    \"$destination\" version\nelif [ -n \"${RUN_VERSION_OPT:-}\" ]; then\n    echo\n    \"$destination\" --version\nfi\n"
  },
  {
    "path": "packages/install_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-09 23:35:44 +0000 (Mon, 09 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu #o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\ncheck_bin(){\n    command -v \"$1\" >/dev/null 2>/dev/null\n}\n\nif check_bin apk; then\n    \"$srcdir/apk_install_packages.sh\" \"$@\"\nelif check_bin apt-get; then\n    \"$srcdir/apt_install_packages.sh\" \"$@\"\nelif check_bin yum; then\n    \"$srcdir/yum_install_packages.sh\" \"$@\"\nelif check_bin brew; then\n    \"$srcdir/brew_install_packages.sh\" \"$@\"\nelse\n    echo \"ERROR: No recognized package manager found to install packages with\"\n    exit 1\nfi\n"
  },
  {
    "path": "packages/install_packages_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 17:28:41 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Caveat: doesn't catch metapackages\n#\n# eg. vim on centos is resolved to vim-enhanced and doesn't match to prevent trying to install again\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\npackages=(\"$@\")\n\ncheck_bin(){\n    type -P \"$@\" &>/dev/null\n}\n\nif check_bin apk; then\n    \"$srcdir/apk_install_packages_if_absent.sh\" \"${packages[@]}\"\nelif check_bin apt-get dpkg; then\n    \"$srcdir/apt_install_packages_if_absent.sh\" \"${packages[@]}\"\nelif check_bin yum rpm; then\n    \"$srcdir/yum_install_packages_if_absent.sh\" \"${packages[@]}\"\nelif check_bin brew; then\n    \"$srcdir/brew_install_packages_if_absent.sh\" \"${packages[@]}\"\nelse\n    echo \"Unsupported OS / Package Manager\"\n    exit 1\nfi\n"
  },
  {
    "path": "packages/nodejs_npm_install.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-23 10:41:56 +0100 (Wed, 23 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nnpm=\"${NPM:-npm}\"\n\nusage(){\n    echo \"Installs NodeJS NPM packages\"\n    echo\n    echo \"Automatically set to install globally if run as root\"\n    echo\n    echo \"Takes a list of npm package names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\npackages=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${packages// }\" ]; then\n    usage\nfi\n\npackages=\"$(tr ' ' ' \\n' <<< \"$packages\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing NodeJS NPM packages\"\necho\n\nopts=\"\"\n#if [ -n \"${TRAVIS:-}\" ]; then\n#    echo \"running in quiet mode\"\n#    opts=\"-q\"\n#fi\n\nenvopts=\"\"\nif [ \"$(uname -s)\" = \"Darwin\" ]; then\n    if type -P brew &>/dev/null; then\n        # usually /usr/local\n        brew_prefix=\"$(brew --prefix)\"\n        # needed to build Crypt::SSLeay\n        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n        # need to send OPENSSL_INCLUDE and OPENSSL_LIB through sudo explicitly using prefix\n        envopts=\"OPENSSL_INCLUDE=$OPENSSL_INCLUDE OPENSSL_LIB=$OPENSSL_LIB\"\n    fi\nfi\n\nif [ $EUID = 0 ]; then\n    opts=\"-g\"\nfi\n\necho \"$envopts $npm install $opts $packages\"\n# want splitting of opts and packages\n# shellcheck disable=SC2086\neval $envopts \"$npm\" install $opts $packages\n"
  },
  {
    "path": "packages/nodejs_npm_install_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-23 10:37:20 +0100 (Wed, 23 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nnpm=\"${NPM:-npm}\"\n\nusage(){\n    echo \"Installs NodeJS NPM packages not already installed\"\n    echo\n    echo \"Leverages adjacent nodejs_npm_install.sh which will automatically set to install globally if run as root\"\n    echo\n    echo \"Takes a list of npm packages as arguments or .txt files containing lists of packages (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\npackages=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\n#if [ -z \"${packages// }\" ]; then\n#    usage\n#fi\n\npackages=\"$(tr ' ' ' \\n' <<< \"$packages\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing NodeJS NPM packages that are not already installed\"\necho\n\n#installed_packages=\"$(npm_config_parseable=true $npm list 2>/dev/null | tail -n +2 | sed 's,.*/,,')\"\n\nfor package in $packages; do\n    #if grep -Fxq \"$package\" <<< \"$installed_packages\"; then\n    # less efficient than above but more likely to not try to re-install packages\n    set +o pipefail\n    if npm_config_parseable=true $npm list 2>/dev/null |\n        tail -n +2 |\n        sed 's,.*/,,' |\n        grep -Fxq \"$package\"; then\n        echo \"nodejs npm package '$package' already installed, skipping...\"\n    else\n        echo \"installing nodejs npm package '$package'\"\n        echo\n        \"$srcdir/nodejs_npm_install.sh\" \"$package\"\n    fi\ndone\n"
  },
  {
    "path": "packages/rpms_filter_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:53:52 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of RPM packages and returns those already installed\n\n$package_args_description\n\nTested on CentOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nprocess_package_args \"$@\" |\ngrep -Fx -f <(installed_rpms) || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/rpms_filter_not_installed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:53:52 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks a given list of RPM packages and returns those not installed\n\n$package_args_description\n\nTested on CentOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\nprocess_package_args \"$@\" |\ngrep -vFx -f <(installed_rpms) || :  # grep causes pipefail exit code breakages in calling code when it doesn't match\n"
  },
  {
    "path": "packages/ruby_gem_install.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/os.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ruby.sh\"\n\ngem=\"${GEM:-gem}\"\nopts=\"${GEM_OPTS:-}\"\n\nif type -P \"$gem\" &>/dev/null; then\n    gem=\"$(type -P \"$gem\")\"\nfi\n\nusage(){\n    echo \"Installs Ruby Gems, taking in to account library paths\"\n    echo\n    echo \"Takes a list of ruby gem names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_gems>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ngems=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding gems from file:  $arg\"\n            gems=\"$gems $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            gems=\"$gems $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${gems// }\" ]; then\n    usage\nfi\n\ngems=\"$(tr ' ' ' \\n' <<< \"$gems\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing Ruby Gems\"\necho\n\nif is_CI; then\n    #echo \"running in quiet mode\"\n    opts=\"-q\"\nfi\n\nenvopts=\"\"\nif is_mac; then\n    if type -P brew &>/dev/null; then\n        # usually /usr/local\n        brew_prefix=\"$(brew --prefix)\"\n        # needed to build Crypt::SSLeay\n        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n        # need to send OPENSSL_INCLUDE and OPENSSL_LIB through sudo explicitly using prefix\n        envopts=\"OPENSSL_INCLUDE=$OPENSSL_INCLUDE OPENSSL_LIB=$OPENSSL_LIB\"\n    fi\nfi\n\nsudo=\"\"\n# don't use --user-install when using RVM because it will cause programs to error out in the RVM environments, breaking builds in Travis CI, Circle CI, AppVeyor etc\nif [ $EUID != 0 ] &&\n   ! inside_ruby_virtualenv; then\n    #sudo=sudo\n    opts=\"$opts --user-install\"\nfi\n\necho \"$sudo $envopts $gem install $opts $gems\"\n# want splitting of opts and gems\n# shellcheck disable=SC2086\neval $sudo $envopts \"$gem\" install $opts $gems\n"
  },
  {
    "path": "packages/ruby_gem_install_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n#ruby=\"${RUBY:-ruby}\"\ngem_cmd=\"${GEM:-gem}\"\nif type -P \"$gem_cmd\" &>/dev/null; then\n    gem_cmd=\"$(type -P \"$gem_cmd\")\"\nfi\n\nusage(){\n    echo \"Installs Ruby Gems not already installed\"\n    echo\n    echo \"Leverages adjacent ruby_gem_install.sh which takes in to account library paths, rubybrew envs etc\"\n    echo\n    echo \"Takes a list of ruby gem names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_gems>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ngems=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding gems from file:  $arg\"\n            gems=\"$gems $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            gems=\"$gems $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${gems// }\" ]; then\n    usage\nfi\n\ngems=\"$(tr ' ' ' \\n' <<< \"$gems\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing Ruby Gems that are not already installed\"\necho\n\n#installed_gems=\"$(\"$gem_cmd\" list --no-versions | grep -ve '^[[:space:]]*$' -e '^\\*')\"\n\nfor gem in $gems; do\n    #if \"$ruby\" -e \"require '$gem'\" &>/dev/null; then\n    #if grep -Fxq \"$gem\" <<< \"$installed_gems\"; then\n    # less efficient than above but more likely to not try to re-install packages\n    if \"$gem_cmd\" list --no-versions |\n        grep -ve '^[[:space:]]*$' -e '^\\*' |\n        grep -Fxq \"$gem\"; then\n        echo \"ruby gem '$gem' already installed, skipping...\"\n    else\n        echo \"installing ruby gem '$gem'\"\n        echo\n        \"$srcdir/ruby_gem_install.sh\" \"$gem\"\n    fi\ndone\n"
  },
  {
    "path": "packages/upgrade_packages_if_outdated.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-02 22:22:43 -0300 (Mon, 02 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Caveat: doesn't catch metapackages\n#\n# eg. vim on centos is resolved to vim-enhanced and doesn't match to prevent trying to upgrade again\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\npackages=(\"$@\")\n\ncheck_bin(){\n    type -P \"$@\" &>/dev/null\n}\n\nif check_bin apk; then\n    \"$srcdir/apk_upgrade_packages_if_outdated.sh\" \"${packages[@]}\"\nelif check_bin apt-get dpkg; then\n    \"$srcdir/apt_upgrade_packages_if_outdated.sh\" \"${packages[@]}\"\nelif check_bin yum rpm; then\n    \"$srcdir/yum_upgrade_packages_if_outdated.sh\" \"${packages[@]}\"\nelif check_bin brew; then\n    \"$srcdir/brew_upgrade_packages_if_outdated.sh\" \"${packages[@]}\"\nelse\n    echo \"Unsupported OS / Package Manager\"\n    exit 1\nfi\n"
  },
  {
    "path": "packages/yum_install_packages.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Install RPM packages in a forgiving way - useful for installing Perl CPAN and Python PyPI modules that may or may not be available\n#\n# combine with later use of the following scripts to only build packages that aren't available in the Linux distribution:\n#\n# perl_cpanm_install_if_absent.sh\n# python_pip_install_if_absent.sh\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nusage(){\n    echo \"Installs Yum RPM package lists\"\n    echo\n    echo \"Takes a list of yum packages as arguments or via stdin, and for any arguments that are plaintext files, reads the packages from those given files (one package per line)\"\n    echo\n    echo \"usage: ${0##/*} <list_of_packages>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\npackages=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then  # && file \"$arg\" | grep -q ASCII  # file not available by default and may not be installed\n            echo \"adding packages from file:  $arg\"\n            packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            packages=\"$packages $arg\"\n        fi\n        # uniq\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\necho \"Installing RPM Packages\"\n\nif [ -z \"${packages// }\" ]; then\n    exit 0\nfi\n\npackages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | sed '/^[[:space:]]*$/d')\"\n\n# RHEL8 ruining things with no default python and lots of python package renames\n# - handling systematically rather than exploding out all my repos package lists\nif [ -n \"${NO_FAIL:-}\" ]; then\n    if grep -q '^REDHAT_SUPPORT_PRODUCT_VERSION=\"8\"' /etc/*release 2>/dev/null; then\n        if grep -q 'python-' <<< \"$packages\"; then\n            # shellcheck disable=SC2001\n            packages=\"$packages\n$(sed 's/^python-/python2-/' <<< \"$packages\")\n$(sed 's/^python-/python3-/' <<< \"$packages\")\n$(sed 's/^python[23]-/python2-/' <<< \"$packages\")\n$(sed 's/^python[23]-/python3-/' <<< \"$packages\")\"\n            echo \"Expanding Python packages out to: $packages\"\n        fi\n    fi\nfi\n\n# CentOS is EOL - but for some reason CentOS 7 works without this, while CentOS 6 and 8 need archive\nif grep -qi 'NAME=.*CentOS' /etc/*release && grep -q '^VERSION=\"[68]\"$' /etc/*release; then\n    echo \"CentOS EOL detected, replacing yum base URL to vault to re-enable package installs\"\n    sed -i 's/^[[:space:]]*mirrorlist/#mirrorlist/' /etc/yum.repos.d/CentOS-*\n    sed -i 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' /etc/yum.repos.d/CentOS-*\nfi\n\necho\necho \"Packages to be installed:\"\necho\ntr ' ' '\\n' <<< \"$packages\"\necho\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    if type -P dnf &>/dev/null; then\n        # dnf exits if any of the packages aren't found so do them individually and ignore failures\n        for package in $packages; do\n            # sudo set in lib/utils-bourne.sh\n            # shellcheck disable=SC2154\n            rpm -q \"$package\" || $sudo dnf install -y \"$package\" || :\n        done\n    else\n        # shellcheck disable=SC2086\n        $sudo yum install -y $packages || :\n    fi\nelse\n    # dnf exists with a proper error code on any error so faster to do all packages together\n    if type -P dnf &>/dev/null; then\n        # want splitting\n        # shellcheck disable=SC2086\n        dnf install -y $packages\n    else\n        # must install separately to check install succeeded because yum install returns 0 when some packages installed and others didn't\n        for package in $packages; do\n            rpm -q \"$package\" || $sudo yum install -y \"$package\"\n        done\n    fi\nfi\n"
  },
  {
    "path": "packages/yum_install_packages_if_absent.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 18:32:06 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nInstalls Yum RPM package lists if the packages aren't already installed\n\n$package_args_description\n\nTested on CentOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\n# get xargs if it's not installed since we call it below in the shell pipeline\nrpm -q findutils &>/dev/null ||\nyum install -y findutils\n\nprocess_package_args \"$@\" |\n\"$srcdir/rpms_filter_not_installed.sh\" |\nrpms_filter_not_provided |\nxargs --no-run-if-empty \"$srcdir/yum_install_packages.sh\"\n"
  },
  {
    "path": "packages/yum_remove_packages.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 21:31:10 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Remove RPM packages in a forgiving way - useful for uninstalling development packages no longer needed eg. to minimize size of Docker images\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"Removing RPM Packages\"\n\npackages=\"\"\nfor arg; do\n    if [ -f \"$arg\" ]; then\n        echo \"adding packages from file:  $arg\"\n        packages=\"$packages $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n        echo\n    else\n        packages=\"$packages $arg\"\n    fi\n    # uniq\n    packages=\"$(echo \"$packages\" | tr ' ' ' \\n' | sort -u | tr '\\n' ' ')\"\ndone\n\nif [ -z \"$packages\" ]; then\n    exit 0\nfi\n\nSUDO=\"\"\n# shellcheck disable=SC2039\n[ \"${EUID:-$(id -u)}\" != 0 ] && SUDO=sudo\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    # shellcheck disable=SC2086\n    if ! $SUDO yum remove -y $packages; then\n        for package in $packages; do\n            if rpm -q \"$package\"; then\n                $SUDO yum remove -y \"$package\" || :\n            fi\n        done\n    fi\nelse\n    # must install separately to check install succeeded because yum install returns 0 when some packages installed and others didn't\n    for package in $packages; do\n        rpm -q \"$package\" || $SUDO yum remove -y \"$package\"\n    done\nfi\n"
  },
  {
    "path": "packages/yum_upgrade_packages_if_outdated.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-03 00:01:41 -0300 (Tue, 03 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/packages.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpgrades Yum RPM package lists if the packages are outdated\n\n$package_args_description\n\nTested on CentOS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<packages>\"\n\nhelp_usage \"$@\"\n\n# get xargs if it's not installed since we call it below in the shell pipeline\nrpm -q findutils &>/dev/null ||\nyum install -y findutils\n\n# quicker and simpler to just let yum/dnf determine that it's not already installed\n#\n# dnf outputs something like:\n#\n#   No match for argument: wget\n#   No match for argument: nonexistentpackage\n#\n# regardless of whether there is a potential package upsteam like wget, or not like nonexistentpackage\n#\n#upgradeable_packages=\"$(yum check-update)\"\n\nprocess_package_args \"$@\" |\n#while read -r package; do\n#    if echo \"$upgradeable_packages\" | grep -Eq \"^$package(.|[[:space:]])\"; then\n#        echo \"$package\"\n#    fi\n#done |\nxargs --no-run-if-empty yum upgrade -y\n"
  },
  {
    "path": "perl/perl_cpanm_install.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/perl.sh\"\n\nCPANM=\"${CPANM:-cpanm}\"\n\nusage(){\n    echo \"Installs Perl CPAN modules using Cpanm, taking in to account library paths, perlbrew envs etc\"\n    echo\n    echo \"Takes a list of perl module names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_modules>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ncpan_modules=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding cpan modules from file:  $arg\"\n            cpan_modules=\"$cpan_modules $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            cpan_modules=\"$cpan_modules $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${cpan_modules// }\" ]; then\n    usage\nfi\n\ncpan_modules=\"$(tr ' ' ' \\n' <<< \"$cpan_modules\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing CPAN Modules\"\necho\n\nopts=\"${CPANM_OPTS:-}\"\nif is_CI; then\n    echo \"running in quiet mode for CI to minimize log noise\"\n    opts=\"-q\"\nfi\n\nenvopts=\"\"\nif [ \"$(uname -s)\" = \"Darwin\" ]; then\n    if type -P brew &>/dev/null; then\n        # usually /usr/local\n        brew_prefix=\"$(brew --prefix)\"\n        # needed to build Crypt::SSLeay\n        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n        # need to send OPENSSL_INCLUDE and OPENSSL_LIB through sudo explicitly using prefix\n        envopts=\"OPENSSL_INCLUDE=$OPENSSL_INCLUDE OPENSSL_LIB=$OPENSSL_LIB\"\n    fi\n\n    # auto CPATH fix for compiling XS modules with Mac's EXTERN.h\n    #\n    # solves this:\n    #\n    # ./xshelper.h:34:10: fatal error: 'EXTERN.h' file not found\n    # #include <EXTERN.h>\n    #          ^~~~~~~~~~\n    # 1 error generated.\n    #\n    for directory in /Library/Developer/CommandLineTools/SDKs/MacOSX*.sdk/System/Library/Perl/\"$PERL_MAJOR_VERSION\"/darwin-thread-multi-2level; do\n        if [ -f \"$directory/CORE/EXTERN.h\" ]; then\n            export CPATH=\"${CPATH:-}:$directory/CORE\"\n        fi\n    done\n    if [ -n \"${CPATH:-}\" ]; then\n        envopts=\"${envopts} CPATH=$CPATH\"\n    fi\nfi\n\nsudo=\"\"\nif [ -n \"${PERL_USER_INSTALL:-}\" ] ||\n   [ -n \"${PERLBREW_PERL:-}\" ] ||\n   [ -n \"${GOOGLE_CLOUD_SHELL:-}\" ]; then\n    sudo=\"\"\nelif [ $EUID != 0 ]; then\n    sudo=sudo\nfi\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    for cpan_module in $cpan_modules; do\n        echo \"$sudo $envopts $CPANM --notest $opts $cpan_module\"\n        # want splitting of opts\n        # shellcheck disable=SC2086\n        # 'env' prevent a command not found error if sudo isn't used from the space OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\" prefix\n        env $sudo $envopts \"$CPANM\" --notest $opts \"$cpan_module\" || :\n    done\nelse\n    echo \"$sudo $envopts $CPANM --notest $opts $cpan_modules\"\n    # want splitting of opts and modules\n    # shellcheck disable=SC2086\n    # 'env' prevent a command not found error if sudo isn't used from the space OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\" prefix\n    if ! env $sudo $envopts \"$CPANM\" --notest $opts $cpan_modules; then\n        echo\n        echo \"reading latest cpanm build.log for details:\"\n        echo\n        set +o pipefail\n        $sudo find ~/.cpanm/work -type f -name build.log -print0 | xargs -0 ls -tr | tail -n1 | xargs $sudo cat\n        # build log is still in user's home dir even when using sudo\n        find ~/.cpanm/work -type f -name build.log -print0 | xargs -0 ls -tr | tail -n1 | xargs cat\n        exit 1\n    fi\nfi\n"
  },
  {
    "path": "perl/perl_cpanm_install_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:48:29 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nperl=\"${PERL:-perl}\"\n\nusage(){\n    echo \"Installs Perl CPAN modules not already installed using Cpanm\"\n    echo\n    echo \"Leverages adjacent perl_cpanm_install.sh which takes in to account library paths, perlbrew envs etc\"\n    echo\n    echo \"Takes a list of perl module names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_modules>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\ncpan_modules=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding cpan modules from file:  $arg\"\n            cpan_modules=\"$cpan_modules $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            cpan_modules=\"$cpan_modules $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${cpan_modules// }\" ]; then\n    usage\nfi\n\ncpan_modules=\"$(tr ' ' ' \\n' <<< \"$cpan_modules\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing CPAN Modules that are not already installed\"\necho\n\nfor cpan_module in $cpan_modules; do\n    perl_module=\"${cpan_module%%@*}\"\n    if \"$perl\" -e \"use $perl_module;\" &>/dev/null; then\n        echo \"perl cpan module '$perl_module' already installed, skipping...\"\n    else\n        echo \"installing perl cpan module '$perl_module'\"\n        echo\n        \"$srcdir/perl_cpanm_install.sh\" \"$cpan_module\"\n    fi\ndone\n"
  },
  {
    "path": "perl/perl_cpanm_reinstall_all.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-01-30 21:00:19 +0000 (Tue, 30 Jan 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nCPANM=\"${CPANM:-cpanm}\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRe-Installs all currently installed Perl CPAN modules using Cpanm, taking in to account library paths, perlbrew envs etc\n\nUseful for trying to recompile XS modules on Macs after migration assistant from an Intel Mac to an ARM Silicon Mac leaves your home XS libraries broken as they're built for the wrong architecture\n\nexport PERL_USER_INSTALL=1\n\nto prevent it installing to the system but just overriding the home dir modules in this case\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\necho \"Re-Installing ALL CPAN Modules\"\necho\n\nexport CPANM_OPTS=\"--reinstall\"\n\ncpan -l |\nsed $'s/\\t/@/; s/@undef$//' |\nxargs \"$srcdir/perl_cpanm_install.sh\"\n"
  },
  {
    "path": "perl/perl_find_duplicate_cpan_requirements.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:52:24 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find duplicate Perl CPAN module requirements across files\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<setup/cpan_requirements.txt>]\"\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 0 ]; then\n    requirements_files=\"$*\"\nelse\n    # might be more confusing in perl tools to find unused modules in subdirs like perl lib, so just stick to local\n    #requirements_files=\"$(find . -name cpan-requirements.txt)\"\n    requirements_files=\"$(find . -maxdepth 2 -name cpan-requirements.txt)\"\n    if [ -z \"$requirements_files\" ]; then\n        usage \"No requirements files found, please specify explicit path to cpan-requirements.txt\"\n    fi\nfi\n\nfound=0\n\n# want splitting of requirements files to separate files\n# shellcheck disable=SC2086\nduplicate_cpan_modules=\"$(\n    sed 's/#.*//;\n         s/@.*//;\n         s/^[[:space:]]*//;\n         s/[[:space:]]*$//;\n         /^[[:space:]]*$/d;' $requirements_files |\n    sort |\n    uniq -d\n)\"\n\nwhile read -r module; do\n    [ -z \"$module\" ] && continue\n    # want splitting of requirements files to separate files\n    # shellcheck disable=SC2086\n    grep \"^$module\\\\>\" $requirements_files\n    ((found + 1))\ndone <<< \"$duplicate_cpan_modules\"\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "perl/perl_find_library_executable.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-15 09:59:01 +0100 (Sun, 15 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Returns the first argument that is found as an executable in the $PATH\n#\n# Useful for Macs to find where libraries executable scripts like fatpacks are, which may get installed locally in $HOME/perl5/bin to avoid Mac OS X System Integrity Protection\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFind where one or more CLI programs are installed by searching all the perl library locations\n\nEspecially useful when perl tools get installed to places not in your \\$PATH - where the 'which' command can't help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<program> [<program2> <program3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nperl=\"${PERL:-perl}\"\n\nperl_path=\"\"\n\nwhile read -r path; do\n    bin=\"${path%/lib/perl*/site-packages*}/bin\"\n    if [ -d \"$bin\" ]; then\n        perl_path=\"$perl_path:$bin\"\n    fi\ndone < <(\"$perl\" -e 'print join(\"\\n\", @INC);')\n\nif [ -d ~/perl5/bin ]; then\n    perl_path=\"$perl_path:\"~/perl5/bin\nfi\n\nexport PATH=\"$perl_path:$PATH\"\nset +o pipefail\nfound=\"$(type -P \"$@\" | head -n 1)\"\nset -o pipefail\n\nif [ -n \"$found\" ]; then\n    echo \"$found\"\nelse\n    echo \"no perl executable was found matching any of: $*\" >&2\n    echo \"\\$PATH searched was: $PATH\" >&2\n    if is_CI; then\n        echo\n        echo \"running in CI detected, attempting to search all paths\" >&2\n        for x in \"$@\"; do\n            echo \"searching for $x:\" >&2\n            find / -type f -name \"$x\" 2>/dev/null || :\n        done\n    fi\n    exit 1\nfi\n"
  },
  {
    "path": "perl/perl_find_library_path.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-27 17:54:37 +0100 (Fri, 27 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Shows the path to Perl libraries given as arguments\n#\n# There is a better version of this in the adjacent DevOps Perl Tools repo called perl_find_library_path.pl\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nperl=\"${PERL:-perl}\"\n\nfind_perl_module(){\n    cpan_module=\"$1\"\n    # shellcheck disable=SC2016\n    \"$perl\" -M\"$cpan_module\" -e '$path='\"$cpan_module\"'; $path =~ s/::/\\//g; $path =~ s/$/.pm/; print \"$INC{$path}\\n\";'\n}\n\nif [ $# -eq 0 ]; then\n    find_perl_module \"File::Basename\" | sed 's,/File/Basename.pm$,,'\nfi\n\nfor arg; do\n    find_perl_module \"$arg\"\ndone\n"
  },
  {
    "path": "perl/perl_find_unused_cpan_modules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:52:24 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find unused Perl CPAN module requirements in the current directory tree\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<setup/cpan_requirements.txt>]\"\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 0 ]; then\n    requirements_files=\"$*\"\nelse\n    # might be more confusing in perl tools to find unused modules in subdirs like perl lib, so just stick to local\n    #requirements_files=\"$(find . -name cpan-requirements.txt)\"\n    requirements_files=\"$(find . -maxdepth 2 -name cpan-requirements.txt)\"\n    if [ -z \"$requirements_files\" ]; then\n        usage \"No requirements files found, please specify explicit path to cpan-requirements.txt\"\n    fi\nfi\n\nfound=0\n\n# want splitting of requirements files to separate files\n# shellcheck disable=SC2086\ncpan_modules=\"$(\n    sed 's/#.*//;\n         s/@.*//;\n         s/^[[:space:]]*//;\n         s/[[:space:]]*$//;\n         /^[[:space:]]*$/d;' $requirements_files |\n    sort -u\n)\"\n\nwhile read -r module; do\n        # grep -R is sloooow by comparison to git grep\n    if ! \\\n        git grep \"^[[:space:]]*\\\\(use\\\\|require\\\\|import\\\\)[[:space:]]\\\\+$module\" |\n        grep -v 'cpan-requirements.*.txt' |\n        grep -q .; then\n            echo \"$module\"\n            ((found + 1))\n    fi\ndone <<< \"$cpan_modules\"\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "perl/perl_generate_fatpacks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-08 18:20:47 +0100 (Tue, 08 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Uses App::Fatpacker to package all perl scripts given as args in to self-contained scripts with dependencies contained\n#\n# fatpack doesn't bundle compiled XS code, but is smart enough to replace those references eg. 'use JSON::XS;' becomes 'use JSON;' in the outputted self-contained script\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\noutput_dir=\"fatpacks\"\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\nexport PERL5LIB=\"${PERL5LIB:-}:$HOME/perl5/lib/perl5\"\nexport PATH=\"${PATH:-}:$HOME/perl5/bin\"\n# for Mac\nfor x in /usr/local/Cellar/perl/*/bin; do\n    if [ -d \"$x\" ]; then\n        export PATH=\"$PATH:$x\"\n    fi\ndone\n\nusage(){\n    echo \"Generates self-contained versions of Perl scripts using App::FatPacker\"\n    echo\n    echo \"Takes a list of perl scripts as arguments or .txt files containing lists of scripts (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_scripts>\"\n    echo\n    exit 3\n}\n\nperl_scripts=\"\"\nfor x in \"$@\"; do\n    if [[ \"$x\" =~ .*\\.txt ]]; then\n        echo \"adding perl scripts from file:  $x\"\n        perl_scripts=\"$perl_scripts $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$x\")\"\n        echo\n    else\n        perl_scripts=\"$perl_scripts $x\"\n    fi\n    perl_scripts=\"$(tr ' ' ' \\n' <<< \"$perl_scripts\" | sort -u | tr '\\n' ' ')\"\ndone\n\nfor x in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$1\" in\n        -*) usage\n            ;;\n    esac\ndone\n\nif [ -z \"$perl_scripts\" ]; then\n    # shellcheck disable=SC2119\n    usage\nfi\n\nsection \"FatPacking Perl Scripts\"\n\nif is_inside_docker; then\n    echo \"Detected running inside Docker, skipping building fatpacks...\"\n    exit 0\nfi\n\ncheck_bin fatpack\n\n# want expansion\n# shellcheck disable=SC2086\ntrap 'echo ERROR' $TRAP_SIGNALS\n\nmkdir -pv \"$output_dir\"\n\necho \"Generating App::FatPacker self-contained perl scripts with all dependencies:\"\necho\ni=0\nfor perl_script in $perl_scripts; do\n    [ -f \"$perl_script\" ] || continue\n    ((i+=1))\n    dest=\"$output_dir/${perl_script%.pl}.fatpack.pl\"\n    echo \"$perl_script -> $dest\"\n    # re-run without error messages suppressed if it fails\n    fatpack pack \"$perl_script\" 2>/dev/null > \"$dest\" ||\n    fatpack pack \"$perl_script\" > \"$dest\"\n    chmod +x \"$dest\"\ndone\necho\necho \"Generated $i fatpacked scripts under fatpacks/ directory\"\n#echo\n#echo \"Generating tarball of fatpacked scripts\"\n#echo\n#tar czf fatpacks.tar.gz fatpacks/\n#echo\n#echo \"Done! Generated fatpacks.tar.gz containing $i fatpacked scripts with all dependencies included\"\nuntrap\n"
  },
  {
    "path": "perl/perl_generate_par_binaries.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-16 15:24:45 +0000 (Sun, 16 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Uses PAR::Packer to compile all perl scripts given as args in to binaries\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/docker.sh\n. \"$srcdir/lib/docker.sh\"\n\noutput_dir=\"bin\"\n\nusage(){\n    echo \"Generates binaries from Perl scripts using PAR::Packer\"\n    echo\n    echo \"Takes a list of perl scripts as arguments or .txt files containing lists of scripts (one per line)\"\n    echo\n    echo \"usage: ${0##*} <list_of_scripts>\"\n    echo\n    exit 3\n}\n\nperl_scripts=\"\"\nfor x in \"$@\"; do\n    if [[ \"$x\" =~ .*\\.txt ]]; then\n        echo \"adding perl scripts from file:  $x\"\n        perl_scripts=\"$perl_scripts $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$x\")\"\n        echo\n    else\n        perl_scripts=\"$perl_scripts $x\"\n    fi\n    perl_scripts=\"$(tr ' ' ' \\n' <<< \"$perl_scripts\" | sort -u | tr '\\n' ' ')\"\ndone\n\nfor x in \"$@\"; do\n    # shellcheck disable=SC2119\n    case \"$1\" in\n        -*) usage\n            ;;\n    esac\ndone\n\nif [ -z \"$perl_scripts\" ]; then\n    # shellcheck disable=SC2119\n    usage\nfi\n\nsection \"Generating PAR binaries from Perl Scripts\"\n\nif is_inside_docker; then\n    echo \"Detected running inside Docker, skipping building par binaries...\"\n    exit 0\nfi\n\ncheck_bin pp\n\n# want expansion\n# shellcheck disable=SC2086\ntrap 'echo ERROR' $TRAP_SIGNALS\n\nmkdir -pv \"$output_dir\"\n\necho \"Generating App::FatPacker self-contained perl scripts with all dependencies:\"\necho\ni=0\nfor perl_script in $perl_scripts; do\n    ((i+=1))\n    dest=\"$output_dir/${perl_script%.pl}\"\n    echo \"$perl_script -> $dest\"\n    # -c runs it but breaks on taint mode enable scripts with the usual: \"-T\" is on the #! line, it must also be used on the command line at\n    # re-run without error messages suppressed if it fails\n    pp -o \"$dest\" \"$perl_script\" 2>/dev/null ||\n    pp -o \"$dest\" \"$perl_script\"\n    #chmod +x \"$dest\"\ndone\necho\necho \"Generated $i PAR binaries under bin/ directory\"\nuntrap\n"
  },
  {
    "path": "perl/perlpath.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-27 17:12:39 +0100 (Fri, 27 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Prints the Perl @INC paths, one per line\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# copied from old bashrc function which is now ported under .bash.d/ too, but given here for convenience in case you are not running the full bash profile\nperl -e 'print join(\"\\n\", @INC);'\n"
  },
  {
    "path": "pingdom/pingdom_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:10:46 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.pingdom.com/api/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Pingdom API (v3.1)\n\nCan specify \\$CURL_OPTS for options to pass to curl, or pass them as arguments to the script\n\nAutomatically handles authentication via environment variable \\$PINGDOM_TOKEN\n\n\nYou must set up an access token here:\n\n    https://my.pingdom.com/app/api-tokens\n\n\nAPI Reference:\n\n    https://docs.pingdom.com/api/\n\n\nExamples:\n\n\n# Get Pingdom checks and statuses:\n\n    ${0##*/} /checks\n\n\n# Get detailed info on a specific check:\n\n    ${0##*/} /checks/<id>\n\n\n# Get average uptime / response time for a specific check:\n\n    ${0##*/} /summary.average/<check_id>\n\n\n# Get Pingdom actions such as status change emails and SMS (newest first):\n\n    ${0##*/} /actions\n\n\n# Get your configured Maintenance windows and occurrences:\n\n    ${0##*/} /maintenance\n    ${0##*/} /maintenance.occurrences\n\n\n# Recent Outages (last week by default, see pingdom_check_outages.sh for last year):\n\n    ${0##*/} /summary.outage/<check_id>\n\n\n# List all Pingdom probe servers\n\n    ${0##*/} /probes\n\n\n# Get check results from all probes for a given check id (find which geographies a sight is being affected in). You will need to correlate this to the probe ids from from the /probes query\n\n    ${0##*/} /results/<check_id>\n\n\n# List teams and their members:\n\n    ${0##*/} /alerting/teams\n\n\n# List alert contacts:\n\n    ${0##*/} /alerting/contacts\n\n\n# Get Credits remaining:\n\n    ${0##*/} /credits\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.pingdom.com/api/3.1\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined \"PINGDOM_TOKEN\"\n\n#curl_api_opts json headers breaks the Pingdom API calls\nif [ -n \"${CURL_OPTS:-}\" ]; then\n    read -r -a CURL_OPTS <<< \"${CURL_OPTS[@]}\" # this @ notation works for both strings and arrays in case a future version of bash do export arrays this should still work\nelse\n    CURL_OPTS=(-sS --fail --connect-timeout 3)\nfi\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path##/}\"\n\nexport TOKEN=\"$PINGDOM_TOKEN\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n\n# args: /checks | jq .\n# args: /checks/<check_id>\n# args: /checks/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n# args: /alerting/contacts | jq .\n# args: /alerting/teams | jq .\n# args: /credits | jq .\n# args: /actions | jq .\n# args: /summary.average/<checkid>\n# args: /summary.average/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n# args: /summary.outage/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n# args: /summary.hoursofday/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n# args: /summary.performance/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n# args: /maintenance | jq .\n# args: /maintenance.occurrences | jq .\n# args: /probes | jq .\n# args: /analysis/<check_id> | jq .\n# args: /analysis/$(pingdom_api.sh /checks | jq -r 'first(.checks[].id)') | jq .\n"
  },
  {
    "path": "pingdom/pingdom_check_latency_by_hour.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 15:59:30 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.pingdom.com/api/#tag/Summary.hoursofday\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets average latency per hour of the day, averaged over the last week, for a given Pingdom check via the Pingdom API\n\nOutput format - quoted CSV:\n\n<hour_of_day>,<average_latency_in_ms>\n\nFor TSV format, set TSV=1 environment variable\n\n\nTo find check IDs:\n\n    pingdom_checks.sh\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<check_id> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_id=\"$1\"\nshift || :\n\n{\necho \"hour,latency_ms\"\n# could set from=<epoch>&to=<epoch> - defaults to 1 week to now\n\"$srcdir/pingdom_api.sh\" \"/summary.hoursofday/$check_id\" \"$@\" |\njq -r '.hoursofday[] | [.hour, .avgresponse] | @csv'\n} |\nif [ -n \"${TSV:-}\" ]; then\n    column -t -s , |\n    sed 's/\"//g'\nelse\n    cat\nfi\n"
  },
  {
    "path": "pingdom/pingdom_check_outages.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:14:16 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the status periods (outages) for a given Pingdom check for the last year (now - 12 months) via the Pingdom API\n\nOutput format - quoted CSV:\n\n\\\"<status>\\\",\\\"<duration_in_seconds>\\\",\\\"<from_timestamp>\\\",\\\"<to_timestamp>\\\"\n\nFor TSV format, set TSV=1 environment variable\n\n\nTo find check IDs:\n\n    pingdom_checks.sh\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<check_id> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_id=\"$1\"\nshift || :\n\nepoch_1_year_ago=\"$(date \"+%s\" -d \"1 year ago\")\"\n\n\"$srcdir/pingdom_api.sh\" \"/summary.outage/$check_id?from=$epoch_1_year_ago\" \"$@\" |\njq -r '.summary.states[] | [.status, .timefrom, .timeto] | @tsv' |\nwhile read -r status from to; do\n    printf '\"%s\",\"%s\",\"%s\",\"%s\"\\n' \"$status\" \"$((to - from))\" \"$(date -d @\"$from\")\" \"$(date -d @\"$to\")\"\ndone |\nif [ -n \"${TSV:-}\" ]; then\n    column -t -s , |\n    sed 's/\"//g'\nelse\n    cat\nfi\n"
  },
  {
    "path": "pingdom/pingdom_checks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:14:16 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the Pingdom checks and their status via the Pingdom API\n\nOutput format:\n\n<check_id>  <check_name>      <type>      <hostname>    <status>    <last_response_time_in_ms>\n\n\nFrom https://docs.pingdom.com/api/#section/Best-Practices/Use-common-sense:\n\n\\\"Don't check for the status of a check every other second. The highest check resolution is one minute. Checking more often than that won't give you much of an advantage\\\"\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/pingdom_api.sh\" /checks \"$@\" |\njq -r \".checks[] | [.id, .name, .type, .hostname, .status, .lastresponsetime] | @csv\" |\ncolumn -t -s , |\nsed 's/\"//g'\n"
  },
  {
    "path": "pingdom/pingdom_checks_average_response_times.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:14:16 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://docs.pingdom.com/api/#tag/Summary.average\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the average response times for all Pingdom checks over the last week via the Pingdom API\n\nOutput format - quoted CSV:\n\n\\\"<id>\\\",\\\"<name>\\\",\\\"<type>\\\",\\\"<hostname>\\\",\\\"<status>\\\",\\\"<average_response_time_in_ms>\\\"\n\nFor TSV format, set TSV=1 environment variable. TSV output must wait until until the end after every check's query has returned, in order to get the column alignment correct\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\nepoch_1_week_ago=\"$(date \"+%s\" -d \"1 week ago\")\"\n\n\"$srcdir/pingdom_api.sh\" /checks \"$@\" |\njq -r \".checks[] | [.id, .type, .hostname, .status, .name] | @tsv\" |\nwhile read -r id type hostname status name; do\n    printf '\"%s\",\"%s\",\"%s\",\"%s\",\"%s\",\"%s\"\\n' \"$id\" \"$name\" \"$type\" \"$hostname\" \"$status\" \\\n    \"$(\"$srcdir/pingdom_api.sh\" \"/summary.average/$id?from=$epoch_1_week_ago\" \"$@\" | jq '.summary.responsetime.avgresponse')\"\ndone |\nif [ -n \"${TSV:-}\" ]; then\n    column -t -s , |\n    sed 's/\"//g'\nelse\n    cat\nfi\n"
  },
  {
    "path": "pingdom/pingdom_checks_latency_by_hour.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:14:16 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nGets the average latency per hour of the day, averaged over the last week, for all Pingdom checks via the Pingdom API\n\nOutput format - quoted CSV:\n\n<hour_of_day>,<average_latency_in_ms>\n\nFor TSV format, set TSV=1 environment variable\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/pingdom_foreach_check.sh\" \"$srcdir/pingdom_check_latency_by_hour.sh\" \"{id}\"\n"
  },
  {
    "path": "pingdom/pingdom_checks_outages.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 13:14:16 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the status periods (outages) for all Pingdom checks for the last year (now - 12 months) via the Pingdom API\n\nOutput format - quoted CSV:\n\n\\\"<status>\\\",\\\"<duration_in_seconds>\\\",\\\"<from_timestamp>\\\",\\\"<to_timestamp>\\\"\n\nFor TSV format, set TSV=1 environment variable\n\nIf you only want the outages just pipe it through:\n\n    | grep -Ev '^\\\"?up'\n\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/pingdom_foreach_check.sh\" \"$srcdir/pingdom_check_outages.sh\" \"{id}\"\n"
  },
  {
    "path": "pingdom/pingdom_foreach_check.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bash -c 'echo \"check id = {id} and name = {name}\"'\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 15:25:27 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command against each Pingdom check\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the check id/names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{id}   - with the check id   <-- this is the one you want for chaining API queries with pingdom_api.sh\n{name} - with the check name\n\neg.\n    ${0##*/} 'echo check id = {id} and name = {name}'\n\nFor real usage examples, see:\n\n    pingdom_checks_outages.sh\n    pingdom_checks_latency_by_hour.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/pingdom_api.sh\" /checks |\njq -r '.checks[] | [.id, .name] | @tsv' |\nwhile read -r check_id check_name; do\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $check_id - $check_name\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{check_id\\}/$check_id}\")\n    cmd=(\"${cmd[@]//\\{check_name\\}/$check_name}\")\n    cmd=(\"${cmd[@]//\\{id\\}/$check_id}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$check_name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\ndone\n"
  },
  {
    "path": "pingdom/pingdom_sms_credits.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-24 14:12:31 +0100 (Mon, 24 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the number of Pingdom available SMS credits checks via the Pingdom API\n\n\\$PINGDOM_TOKEN must be defined in the environment for authentication\n\nSee adjacent pingdom_api.sh for more details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n\"$srcdir/pingdom_api.sh\" /credits \"$@\" |\njq -r '.credits.availablesms'\n"
  },
  {
    "path": "postgres/postgres.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-05 13:42:41 +0100 (Wed, 05 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://hub.docker.com/_/postgres\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\n# defined in lib/dbshell.sh\n# shellcheck disable=SC2154\nshell_description=\"$sql_mount_description\n\nSource a sql script:\n\n\\\\i postgres_info.sql\n\n\nGet shell access:\n\n\\\\!\n\n\nList available SQL scripts:\n\n\\\\! ls -l postgres*.sql\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBoots a quick PostgreSQL docker container and drops you in to the 'psql' shell\n\nMultiple invocations of this script will connect to the same Postgres container if already running\nand the last invocation of this script to exit from the psql shell will delete that container\n\nPostgresSQL version can be specified using the first argument, or the \\$POSTGRES_VERSION environment variable,\notherwise 'latest' is used\n\nVersions to use can be found from the following URL:\n\nhttps://hub.docker.com/_/postgres?tab=tags\n\nor programmatically on the command line (see DevOps Python tools repo):\n\ndockerhub_show_tags.py postgres\n\n\nAutomatically creates shared bind mount points from host to container for convenience:\n$shell_description\n\n\nTested on PostgreSQL 8.4, 9.x, 10.x, 11.x, 12.x, 13.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>] [options]\n\n-n  --name  NAME    Docker container name to use (default: postgres)\n-p  --port  PORT    Expose PostgreSQL port 5432 on given port number\n-d  --no-delete     Don't delete the container upon the last psql session closing (\\$DOCKER_NO_DELETE)\n-r  --restart       Force a restart of a clean PostgreSQL instance (\\$POSTGRES_RESTART)\n-s  --sample        Load sample Chinook database (\\$LOAD_SAMPLE)\"\n\nhelp_usage \"$@\"\n\ndocker_image=postgres\nport=\"\"\ndocker_opts=\"\"\n\npassword=\"${PGPASSWORD:-${POSTGRESQL_PASSWORD:-${POSTGRES_PASSWORD:-${PASSWORD:-test}}}}\"\n\nwhile [ $# -gt 0 ]; do\n    # DOCKER_NO_DELETE used by functions from lib\n    # shellcheck disable=SC2034\n    case \"$1\" in\n      -n| --name)   container_name=\"$2\"\n                    shift\n                    ;;\n      -p| --port)   port=\"$2\"\n                    [[ \"$port\" =~ ^[[:digit:]]*$ ]] || die \"invalid --port '$port' given\"\n                    shift\n                    ;;\n     -s|--sample)   LOAD_SAMPLE_DB=1\n                    ;;\n    -r|--restart)   POSTGRES_RESTART=1\n                    ;;\n  -d|--no-delete)   DOCKER_NO_DELETE=1\n                    ;;\n               *)   version=\"$1\"\n                    ;;\n    esac\n    shift\ndone\n\ncontainer_name=\"${container_name:-${POSTGRES_CONTAINER_NAME:-postgres}}\"\nversion=\"${version:-${POSTGRESQL_VERSION:-${POSTGRES_VERSION:-latest}}}\"\n\nif [ -n \"$port\" ]; then\n    docker_opts=\"-p $port:5432\"\nfi\n\ndb=\"$srcdir/chinook.psql\"\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ] &&\n   ! [ -f \"$db.utf8\" ]; then\n    timestamp \"downloading sample 'chinook' database\"\n    wget -qcO \"$db\" 'https://github.com/lerocha/chinook-database/blob/master/ChinookDatabase/DataSources/Chinook_PostgreSql.sql?raw=true'\n    iconv -f ISO-8859-1 -t UTF-8 \"$db\" > \"$db.utf8\"\nfi\n\n# kill existing if we have specified a different version than is running\ndocker_image_version=\"$(docker_container_image \"$container_name\")\"\nif [ -n \"$docker_image_version\" ] &&\n   [ \"$docker_image_version\" != \"$docker_image:$version\" ]; then\n    POSTGRES_RESTART=1\nfi\n\n# remove existing non-running container so we can boot a new one\nif docker_container_not_running \"$container_name\"; then\n    POSTGRES_RESTART=1\nfi\n\nif [ -n \"${POSTGRES_RESTART:-}\" ]; then\n    # ensures version is correct before we kill any existing test env to switch versions to minimize downtime\n    timestamp \"docker pull $docker_image:$version\"\n    docker_pull \"$docker_image:$version\"\n\n    timestamp \"killing existing container:\"\n    docker rm -f -- \"$container_name\" 2>/dev/null || :\nfi\n\nif ! docker_container_exists \"$container_name\"; then\n    timestamp \"booting PostgreSQL container from image '$docker_image:$version':\"\n    # defined in lib/dbshell.sh\n    # shellcheck disable=SC2154,SC2086,SC2046\n    docker run -d \\\n        --name \"$container_name\" \\\n        $docker_opts \\\n        -e POSTGRES_PASSWORD=\"$password\" \\\n        -v \"$srcdir/../setup/postgresql.conf:/etc/postgresql/postgresql.conf\" \\\n        $docker_sql_mount_switches \\\n        \"$docker_image\":\"$version\" \\\n        $(if [ \"${version:0:1}\" = 8 ] || [ \"${version:0:3}\" = '9.0' ]; then echo postgres; fi) \\\n        -c 'config_file=/etc/postgresql/postgresql.conf'\n        # can't mount postgresql.conf here because it prevents /var/lib/postgresql/data from being initialized\n        #-v \"$srcdir/../setup/postgresql.conf:/var/lib/postgresql/data/postgresql.conf\"\nfi\n\nwait_for_postgres_ready \"$container_name\"\necho\n\ntimestamp \"linking shell profile for .psqlrc\"\ndocker exec \"$container_name\" bash -c \"cd /bash && setup/shell_link.sh &>/dev/null\" || :\n\n# yes expand now\n# shellcheck disable=SC2064\ntrap \"echo ERROR; echo; echo; [ -z '${DEBUG:-}' ] || docker logs '$container_name'\" EXIT\n\nif [ -n \"${LOAD_SAMPLE_DB:-}\" ]; then\n    dbname=\"${db##*/}\"\n    dbname=\"${dbname%%.*}\"\n    timestamp \"loading $dbname database\"\n    # psql -c doesn't allow mixing SQL and psql meta-commands, must pipe in\n    # create database if not exists equiv in postgres                                                         # \\gexec executes each column returned as a SQL statement\n    echo \"SELECT 'CREATE DATABASE $dbname' WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '$dbname')\\\\gexec\" |\n    docker exec -i -e PGOPTIONS=\"-c client_min_messages=WARNING\" \"$container_name\" psql -U postgres\n    timestamp \"loading data (this may take a minute)\"\n    docker exec -e PGOPTIONS=\"-c client_min_messages=WARNING\" \"$container_name\" psql -U postgres -q -d \"$dbname\" -f \"/bash/${db##*/}.utf8\"\n    timestamp \"done\"\n    echo >&2\nfi\n\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    cat <<EOF\n$shell_description\n\nEOF\nfi\n\n# cd to /sql to make sourcing easier without /sql/ path prefix\ndocker_exec_opts=\"-w /sql -i\"\n\n# allow non-interactive piped automation to avoid tty errors eg.\n# for sql in postgres*.sql; do echo \"source $sql\"; done | postgres.sh\n# normally you would just 'postgres.sh postgres*.sql' but this is used by postgres_test_scripts.sh\nif has_terminal && [ -z \"${DOCKER_NO_TERMINAL:-}\" ]; then\n    docker_exec_opts+=\" -t\"\nfi\n\n# want opt splitting\n# shellcheck disable=SC2086\ndocker exec $docker_exec_opts \"$container_name\" /bash/psql_colorized.sh -U postgres\n\nuntrap\n\ndocker_rm_when_last_connection \"$0\" \"$container_name\"\n"
  },
  {
    "path": "postgres/postgres_foreach_table.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:50:46 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun SQL query against all PostgreSQL tables in all databases via psql.sh\n\nQuery can contain {db}, {schema} and {table} placeholders which will be replaced for each table\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<schema>.<table>)\n\nAuto-skips information_schema and pg_catalog schemas for safety\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the db/schema/table names and exit after the first iteration\n\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"<query>\\\" [<psql_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery_template=\"$1\"\nshift || :\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\nAUTOFILTER=1 \"$srcdir/postgres_list_tables.sh\" \"$@\" |\nwhile read -r db schema table; do\n    printf '%s.%s.%s\\t' \"$db\" \"$schema\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\\"$db\\\"}\"\n    query=\"${query//\\{schema\\}/\\\"$schema\\\"}\"\n    query=\"${query//\\{table\\}/\\\"$table\\\"}\"\n    # doing \\c $db is noisy\n    \"$srcdir/psql.sh\" -q -t -d \"$db\" -c \"$query\" \"$@\"\n    # weird situation on RDS PostgreSQL, hanging all night trying to select count(*) a table, happens on many tables, skip them like so and carry on\n    #timeout -k 10 60 \"$srcdir/psql.sh\" -q -t -d \"$db\" -c \"$query\" \"$@\" || echo\ndone |\nsed '/^[[:space:]]*$/d'\n"
  },
  {
    "path": "postgres/postgres_foreach_table_timeout.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:50:46 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -eu  # -o pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun SQL query against all PostgreSQL tables in all databases via psql.sh\n\nQuery can contain {db}, {schema} and {table} placeholders which will be replaced for each table\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<schema>.<table>)\n\nAuto-skips information_schema and pg_catalog schemas for safety\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the db/schema/table names and exit after the first iteration\n\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\\\"<query>\\\" [<psql_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nquery_template=\"$1\"\nshift || :\n\n# exit the loop subshell if you Control-C\ntrap 'exit 130' INT\n\ntimeout=\"\"\nif [ -n \"${TABLE_TIMEOUT:-}\" ]; then\n    if ! [[ \"$TABLE_TIMEOUT\" =~ ^[[:digit:]]+$ ]]; then\n        usage \"invalid TABLE_TIMEOUT environment variable, must be an integer!\"\n    fi\n    timeout=\"timeout -k 10 $TABLE_TIMEOUT\"\nfi\n\nAUTOFILTER=1 \"$srcdir/postgres_list_tables.sh\" \"$@\" |\nwhile read -r db schema table; do\n    printf '%s.%s.%s\\t' \"$db\" \"$schema\" \"$table\"\n    query=\"${query_template//\\{db\\}/\\\"$db\\\"}\"\n    query=\"${query//\\{schema\\}/\\\"$schema\\\"}\"\n    query=\"${query//\\{table\\}/\\\"$table\\\"}\"\n    # weird situation on RDS PostgreSQL, hanging all night trying to select count(*) a table, happens on many tables\n    set +e\n    # time them out, skip them and carry on\n    # doing \\c $db is noisy, using -d $db instead\n    $timeout \"$srcdir/psql.sh\" -q -t -d \"$db\" -c \"$query\" \"$@\"\n    result=$?\n    set -e\n    if [ $result -ne 0 ]; then\n        if [ $result -eq 124 ] &&\n           [ -n \"$timeout\" ]; then\n            echo\n        else\n            exit $result\n        fi\n    fi\ndone |\nsed '/^[[:space:]]*$/d'\n"
  },
  {
    "path": "postgres/postgres_list_databases.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:40:27 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all PostgreSQL databases using adjacent psql.sh script\n\nFILTER environment variable will restrict to matching databases (if giving <db>.<table>, matches up to the first dot)\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/psql.sh\" -q -t -c 'SELECT DISTINCT(table_catalog) FROM information_schema.tables ORDER BY table_catalog;' \"$@\" |\nsed 's/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' |\nwhile read -r db; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db\" =~ ${FILTER%%.*} ]]; then\n        continue\n    fi\n    printf '%s\\n' \"$db\"\ndone\n"
  },
  {
    "path": "postgres/postgres_list_schemas.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:40:27 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all PostgreSQL schemas using adjacent psql.sh script\n\nFILTER environment variable will restrict to matching schemas (matches against fully qualified schema name <db>.<schema>)\n\nAUTOFILTER if set to any value skips information_schema and pg_catalog schemas\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/psql.sh\" -q -t -c \"SELECT DISTINCT table_catalog, table_schema FROM information_schema.tables ORDER BY table_catalog, table_schema;\" \"$@\" |\nsed 's/|//g; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' |\nif [ -n \"${AUTOFILTER:-}\" ]; then\n    grep -Ev '[[:space:]](information_schema|pg_catalog)$'\nelse\n    cat\nfi |\nwhile read -r db schema; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db.$schema\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\n' \"$db\" \"$schema\"\ndone\n"
  },
  {
    "path": "postgres/postgres_list_tables.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:40:27 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all PostgreSQL tables using adjacent psql.sh script\n\nFILTER environment variable will restrict to matching tables (matches against fully qualified table name <db>.<schema>.<table>)\n\nAUTOFILTER if set to any value skips information_schema and pg_catalog schemas\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\nhelp_usage \"$@\"\n\n\n#\"$srcdir/psql.sh\" -q -t -c \"SELECT table_catalog || '.' || table_schema || '.' || table_name FROM information_schema.tables ORDER BY table_catalog, table_schema, table_name;\" \"$@\" |\n\"$srcdir/psql.sh\" -q -t -c \"SELECT table_catalog, table_schema, table_name FROM information_schema.tables ORDER BY table_catalog, table_schema, table_name;\" \"$@\" |\nsed 's/|//g; s/^[[:space:]]*//; s/[[:space:]]*$//; /^[[:space:]]*$/d' |\n#while read -r table; do\nif [ -n \"${AUTOFILTER:-}\" ]; then\n    grep -Ev '[[:space:]](information_schema|pg_catalog)[[:space:]]'\nelse\n    cat\nfi |\nwhile read -r db schema table; do\n    if [ -n \"${FILTER:-}\" ] &&\n       ! [[ \"$db.$schema.$table\" =~ $FILTER ]]; then\n        continue\n    fi\n    printf '%s\\t%s\\t%s\\n' \"$db\" \"$schema\" \"$table\"\ndone\n"
  },
  {
    "path": "postgres/postgres_tables_row_counts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-12-10 11:33:52 +0000 (Tue, 10 Dec 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCounts rows for all PostgreSQL tables in all databases using adjacent psql.sh script\n\nTSV Output format:\n\n<database>.<schema>.<table>     <row_count>\n\nFILTER environment variable will restrict to matching fully qualified tables (<db>.<schema>.<table>)\n\nTested on AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\nhelp_usage \"$@\"\n\n\n\"$srcdir/postgres_foreach_table.sh\" \"SELECT COUNT(*) FROM {db}.{schema}.{table}\" \"$@\" |\nsed '/^[[:space:]]*$/d'\n"
  },
  {
    "path": "postgres/postgres_test_scripts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  shellcheck disable=SC2028\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-09 10:42:23 +0100 (Sun, 09 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/dbshell.sh\"\n\npostgres_versions=\"\n8.4\n9.0\n9.1\n9.2\n9.3\n9.4\n9.5\n9.6\n10.0\n10.1\n10.2\n10.3\n10.4\n10.5\n10.6\n10.7\n10.8\n10.9\n11.0\n11.1\n11.2\n11.3\n11.4\n11.5\n11.6\n11.7\n11.8\n12.0\n12.1\n12.2\n12.3\n12.4\n13.0\nlatest\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns all of the scripts given as arguments against multiple PostgreSQL versions using docker\n\nUses postgres.sh to boot a PostgreSQL docker environment and pipe source statements in to the container\n\nSources each script in PostgreSQL in the order given\n\nRuns against a list of PostgreSQL versions from the first of the following conditions:\n\n- If \\$POSTGRES_VERSIONS environment variable is set, then only tests against those versions in the order given, space or comma separated, with 'x' used as a wildcard (eg. '10.x , 11.x , 12.x')\n- If \\$GET_DOCKER_TAGS is set and dockerhub_show_tags.py is found in the \\$PATH (from DevOps Python tools repo), then uses it to fetch the latest live list of version tags available from the dockerhub API, reordering by newest first\n- Falls back to the following pre-set list of versions, reordering by newest first:\n\n$(tr ' ' '\\n' <<< \"$postgres_versions\" | grep -v '^[[:space:]]*$')\n\nIf a script has a headers such as:\n\n-- Requires PostgreSQL N.N (same as >=)\n-- Requires PostgreSQL >= N.N\n-- Requires PostgreSQL >  N.N\n-- Requires PostgreSQL <= N.N\n-- Requires PostgreSQL <  N.N\n\nthen will only run that script on the specified versions of PostgreSQL\n\nThis is for convenience so you can test a whole repository such as my SQL-scripts repo just by running against all scripts and have this code figure out the combinations of scripts to run vs versions, eg:\n\n${0##*/} postgres_*.sql\n\nIf no script files are given as arguments, then searches \\$PWD for scripts named in the formats:\n\npostgres*.sql\n*.psql\n\n\nTested on PostgreSQL 8.4, 9.x, 10.x, 11.x, 12.x, 13.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"script1.sql [script2.sql ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport POSTGRES_CONTAINER_NAME=\"${POSTGRES_CONTAINER_NAME:-postgres-test-scripts}\"\n\nif [ $# -gt 0 ]; then\n    scripts=(\"$@\")\nelse\n    shopt -s nullglob\n    scripts=(postgres*.sql *.psql)\nfi\n\nif [ ${#scripts[@]} -lt 1 ]; then\n    usage \"no scripts given and none found in current working directory matching the patterns: postgres*.sql / *.psql\"\nfi\n\nfor sql_file in \"${scripts[@]}\"; do\n    [ -f \"$sql_file\" ] || die \"ERROR: file not found: $sql_file\"\ndone\n\necho \"Testing ${#scripts[@]} PostgreSQL scripts:\"\necho\nfor sql_file in \"${scripts[@]}\"; do\n    echo \"$sql_file\"\ndone\necho\n\nget_postgres_versions(){\n    if [ -n \"${GET_DOCKER_TAGS:-}\" ]; then\n        echo \"checking if dockerhub_show_tags.py is available:\" >&2\n        echo >&2\n        if type -P dockerhub_show_tags.py 2>/dev/null; then\n            echo >&2\n            echo \"dockerhub_show_tags.py found, executing to get latest list of PostgreSQL docker version tags\" >&2\n            echo >&2\n            postgres_versions=\"$(dockerhub_show_tags.py postgres |\n                                grep -Eo '[[:space:]][[:digit:]]{1,2}\\.[[:digit:]]' -e '^[[:space:]*latest[[:space:]]*$' |\n                                sed 's/[[:space:]]//g' |\n                                grep -v \"8.4\" |\n                                sort -u -t. -k1n -k2n)\"\n            echo \"found PostgreSQL versions:\" >&2\n            echo >&2\n            echo \"$postgres_versions\"\n            return\n        fi\n    fi\n    echo \"$postgres_versions\" |\n    tr ' ' '\\n' |\n    grep -v '^[[:space:]]*$' |\n    if is_CI; then\n        echo \"CI detected - using randomized sample of PostgreSQL versions to test against:\" >&2\n        {\n        shuf | head -n 1\n        echo 9.1  # most problematic / incompatible versions should always be tested\n        echo 9.5\n        echo latest\n        } | sort -unr -t. -k1,2\n    else\n        echo \"using default list of PostgreSQL versions to test against:\" >&2\n        cat\n    fi\n    echo >&2\n}\n\nif [ -n \"${POSTGRESQL_VERSIONS:-${POSTGRES_VERSIONS:-}}\" ]; then\n    versions=\"\"\n    POSTGRES_VERSIONS=\"${POSTGRESQL_VERSIONS:-$POSTGRES_VERSIONS}\"\n    POSTGRES_VERSIONS=\"${POSTGRES_VERSIONS//,/ }\"\n    for version in $POSTGRES_VERSIONS; do\n        if [[ \"$version\" =~ x ]]; then\n            versions+=\" $(grep \"${version//x/.*}\" <<< \"$postgres_versions\" |\n                          sort -u -t. -k1n -k2 |\n                          tac ||\n                          die \"version '$version' not found\")\"\n        else\n            versions+=\" $version\"\n        fi\n    done\n    postgres_versions=\"$(tr ' ' '\\n' <<< \"$versions\" | grep -v '^[[:space:]]*$')\"\n    echo \"using given PostgreSQL versions:\" >&2\nelse\n    postgres_versions=\"$(get_postgres_versions | tac)\"\nfi\n\necho \"$postgres_versions\"\necho\n\nfor version in $postgres_versions; do\n    hr\n    echo \"Executing scripts against PostgreSQL version '$version'\": >&2\n    echo >&2\n    {\n    echo 'SELECT VERSION();'\n    for sql_file in \"${scripts[@]}\"; do\n        if skip_min_version \"PostgreSQL\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        if skip_max_version \"PostgreSQL\" \"$version\" \"$sql_file\"; then\n            continue\n        fi\n        echo '\\! printf \"================================================================================\\n\"'\n        # no effect\n        #echo\n        echo '\\set ON_ERROR_STOP true'\n        # ugly\n        #echo \"select '$sql_file' as script;\"\n        echo \"\\\\! printf '\\\\nscript %s:\\\\n\\\\n' '$sql_file'\"\n        # instead of dealing with pathing issues, prefixing /pwd or depending on the scripts being in the sql/ directory\n        #echo \"\\\\i $sql_file\"\n        cat \"$sql_file\"\n        echo \"\\\\! printf '\\\\n\\\\n'\"\n    done\n    } |\n    command time \"$srcdir/postgres.sh\" \"$version\" --restart\n    echo >&2\n    timestamp \"Succeeded testing ${#scripts[@]} scripts for PostgreSQL $version\"\n    echo >&2\n    echo >&2\ndone\necho >&2\necho >&2\ntimestamp \"All PostgreSQL tests passed for all scripts on all versions:  $(tac <<< \"$postgres_versions\" | tr '\\n' ' ')\"\n"
  },
  {
    "path": "postgres/psql.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-12 16:15:38 +0000 (Thu, 12 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nScript to more easily connect to PostgreSQL without having to repeatedly specify options like host, username and password\n\nLeverages standard PostgresSQL options as well as others likely to be found in the environment\n\nhttps://www.postgresql.org/docs/9.0/libpq-envars.html\n\nSee also - GNU sql\n\nTested on PostgreSQL 8.4, 9.x, 10.x, 11.x, 12.x, 13.0\n          AWS RDS PostgreSQL 9.5.15\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\n# would catch -h but this is a legit option we should let through\n#help_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfor arg; do\n    case \"$arg\" in\n        --help) usage\n                ;;\n    esac\ndone\n\nopts=\"${POSTGRES_OPTS:-}\"\n\nPOSTGRES_HOST=\"${PGHOST:-${POSTGRESQL_HOST:-${POSTGRES_HOST:-${HOST:-}}}}\"\nif [ -n \"${POSTGRES_HOST:-}\" ]; then\n    # more intuitive to see in the process list what is going on\n    #export PGHOST=\"$POSTGRES_HOST\"\n    opts=\"$opts -h $POSTGRES_HOST\"\nfi\n\nPOSTGRES_PORT=\"${PGPORT:-${POSTGRESQL_PORT:-${POSTGRES_PORT:-${PORT:-}}}}\"\nif [ -n \"${POSTGRES_PORT:-}\" ]; then\n    # more intuitive to see in the process list what is going on\n    #export PGPORT=\"$POSTGRES_PORT\"\n    opts=\"$opts -p $POSTGRES_PORT\"\nfi\n\nPOSTGRES_USER=\"${PGUSER:-${POSTGRESQL_USER:-${POSTGRES_USER:-${USER:-}}}}\"\nif [ -n \"${POSTGRES_USER:-}\" ]; then\n    # more intuitive to see in the process list what is going on\n    #export PGUSER=\"$POSTGRES_USER\"\n    opts=\"$opts -U $POSTGRES_USER\"\nfi\n\nPOSTGRES_PASSWORD=\"${PGPASSWORD:-${POSTGRESQL_PASSWORD:-${POSTGRES_PASSWORD:-${PASSWORD:-}}}}\"\nif [ -n \"${POSTGRES_PASSWORD:-}\" ]; then\n    # can't do this and wouldn't want to as it'd expose it in the password list\n    #opts=\"$opts -U $POSTGRES_PASSWORD\"\n    export PGPASSWORD=\"$POSTGRES_PASSWORD\"\nfi\n\nPOSTGRES_DATABASE=\"${PGDATABASE:-${POSTGRESQL_DATABASE:-${POSTGRES_DATABASE:-${DATABASE:-}}}}\"\nif [ -n \"${POSTGRES_DATABASE:-}\" ]; then\n    # more intuitive to see in the process list what is going on\n    #export PGDATABASE=\"$POSTGRES_DATABASE\"\n    opts=\"$opts -d $POSTGRES_DATABASE\"\nfi\n\n# split opts\n# shellcheck disable=SC2086\nexec psql $opts \"$@\"\n"
  },
  {
    "path": "postgres/psql_colorized.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-20 12:21:38 +0100 (Tue, 20 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRuns 'psql' with colorized output\n\nUses adjacent psql.sh, see there for more details\n\nTested on PostgreSQL 8.4, 9.x, 10.x, 11.x, 12.x, 13.0\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<psql_options>]\"\n\n# would catch -h but this is a legit option we should let through\n#help_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfor arg; do\n    case \"$arg\" in\n        --help) usage\n                ;;\n    esac\ndone\n\nGREEN=\"$(echo -e '\\033[0;32m')\"\n# RED=\"$(echo -e '\\033[1;31m')\"\nNOCOLOR=\"$(echo -e '\\033[0m')\"\n\n#export LESS=\"-iMSx4 -FXR\"\nexport LESS=\"-RFXig --tabs=4\"\n\nif type -P less &>/dev/null; then\n    pager=\"less\"\nelse\n    pager=\"more\"\nfi\n\nsed=\"sed\"\nif is_mac; then\n    # doesn't work inside PAGER evaluation\n    #sed(){\n    #    command gsed \"$@\"\n    #}\n    sed=\"command gsed\"\nfi\n\n                # can't colourize errors because they go to stderr and 2>&1 at start of pager breaks everything\n                #s/\\(ERROR:\\)/$RED\\1$NOCOLOR/g;\n                #s/^\\(\\.psqlrc loaded\\)/$GREEN\\1$NOCOLOR/;\nexport PAGER=\"$sed '\n                s/^\\\\(([0-9]\\\\+ rows*)\\\\)/$GREEN\\\\1$NOCOLOR/;\n                s/^\\\\(-\\\\[\\\\ RECORD\\\\ [0-9]\\\\+\\\\ \\\\][-+]\\\\+\\\\)/$GREEN\\\\1$NOCOLOR/;\n                s/|/$GREEN|$NOCOLOR/g;\n                s/^\\\\([-+]\\\\+\\\\)/$GREEN\\\\1$NOCOLOR/\n                ' 2>&1 |\n              $pager\"\n\n\"$srcdir/psql.sh\" --pset pager=always \"$@\"\n"
  },
  {
    "path": "python/pygmentize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-02 21:50:00 +0000 (Sat, 02 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n#set -euo pipefail\n#[ -n \"${DEBUG:-}\" ] && set -x\n\n# idea from https://superuser.com/questions/117841/when-reading-a-file-with-less-or-more-how-can-i-get-the-content-in-colors\n\nif [ $# -gt 0 ]; then\n    case \"$1\" in\n        *.ad[asb]|\\\n        *.asm|\\\n        *.awk|\\\n        *.axp|\\\n        *.diff|\\\n        *.ebuild|\\\n        *.eclass|\\\n        *.groff|\\\n        *.hh|\\\n        *.inc|\\\n        *.java|\\\n        *.js|\\\n        *.lsp|\\\n        *.l|\\\n        *.m4|\\\n        *.pas|\\\n        *.patch|\\\n        *.php|\\\n        *.pl|\\\n        *.pm|\\\n        *.pod|\\\n        *.pov|\\\n        *.ppd|\\\n        *.py|\\\n        *.p|\\\n        *.rb|\\\n        *.sh|\\\n        *.sql|\\\n        *.xml|\\\n        *.xps|\\\n        *.xsl|\\\n        *.[ch]pp|\\\n        *.[ch]xx|\\\n        *.[ch]\\\n            )   pygmentize -f 256 \"$1\"\n                ;;\n\n        .bash*) pygmentize -f 256 -l sh \"$1\"\n                ;;\n\n        *)\n            if grep -q '#!.*bash' \"$1\" 2> /dev/null; then\n                pygmentize -f 256 -l sh \"$1\"\n            else\n                exit 1\n            fi\n    esac\nelse\n    pygmentize -f 256 -g\nfi\n\nexit 0\n"
  },
  {
    "path": "python/python_compile.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2015-05-25 01:38:24 +0100 (Mon, 25 May 2015)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/python.sh\"\n\npython=\"${PYTHON:-${python:-python}}\"\n\nset +o pipefail\nfilelist=\"$(find \"${1:-.}\" -maxdepth 2 -type f -iname '*.py' -o -iname '*.jy' | grep -v /templates/ | sort)\"\nset -o pipefail\n\nif [ -z \"$filelist\" ]; then\n    echo \"no Python / Jython files found to compile\"\n    echo\n    echo \"usage: ${0##*/} <python_file_or_directory>\"\n    echo\n    # shellcheck disable=SC2317\n    return 0 &>/dev/null || :\n    # shellcheck disable=SC2317\n    exit 0\nfi\n\nsection \"Compiling Python / Jython files\"\n\nif ! type -P \"$python\" &>/dev/null; then\n    echo \"$python not found in \\$PATH, skipping compile\"\n    exit 0\nfi\n\nstart_time=\"$(start_timer)\"\n\n# opts:\n#\n# -O  - optimize\n# -3  - warn on Python 3 incompatibilies that 2to3 cannot easily fix\n# -t  - warn on inconsistent use of tabs\n\nopts=()\n\nif \"$python\" -V 2>&1 | grep -q 'Python 2'; then\n    opts+=(-3)\nfi\n\nif ! is_travis &&\n   ! type -P pypy &>/dev/null &&\n   ! type -P python | grep -qi pypy; then\n       opts+=(-t)\nfi\n\nif [ -n \"${NOCOMPILE:-}\" ]; then\n    echo \"\\$NOCOMPILE environment variable set, skipping python compile\"\nelif [ -n \"${QUICK:-}\" ]; then\n    echo \"\\$QUICK environment variable set, skipping python compile\"\nelse\n    if [ -n \"${FAST:-}\" ]; then\n        # want opt expansion\n        # shellcheck disable=SC2086\n        \"$python\" ${opts:+\"${opts[@]}\"} -m compileall \"${1:-.}\" || :\n    else\n        for x in $filelist; do\n            type isExcluded &>/dev/null && isExcluded \"$x\" && continue\n            echo \"compiling $x\"\n            # want opt expansion\n            # shellcheck disable=SC2086\n            \"$python\" -O ${opts:+\"${opts[@]}\"} -m py_compile \"$x\"\n        done\n    fi\nfi\n\ntime_taken \"$start_time\"\nsection2 \"Finished compiling Python / Jython files\"\necho\n"
  },
  {
    "path": "python/python_find_duplicate_pip_requirements.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:49:47 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find duplicate Python Pip / PyPI module requirements across files\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<pip_requirements_files>]\"\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nfound=0\n\nif [ -n \"$*\" ]; then\n    requirements_files=\"$*\"\nelse\n    requirements_files=\"$(find . -maxdepth 2 -name requirements.txt)\"\n    if [ -z \"$requirements_files\" ]; then\n        usage \"No requirements files found, please specify explicit path to requirements.txt\"\n    fi\nfi\n\n# need word splitting for different files\n# shellcheck disable=SC2086\nsed 's/#.*//;\n     s/[<>=].*//;\n     s/^[[:space:]]*//;\n     s/[[:space:]]*$//;\n     /^[[:space:]]*$/d;' $requirements_files |\nsort |\nuniq -d |\nwhile read -r module ; do\n    # need word splitting for different files\n    # shellcheck disable=SC2086\n    grep \"^${module}[<>=]\" $requirements_files\n    ((found + 1))\ndone\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "python/python_find_library_executable.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-15 09:59:01 +0100 (Sun, 15 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Returns the first argument that is found as an executable in the $PATH, with preference given to local Python library installation\n#\n# Useful for Macs to find where libraries executable scripts like nosetests are, which may get installed locally in $HOME/Library/Python/2.7/bin to avoid Mac OS X System Integrity Protection\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck source=lib/utils.sh\n#. \"$srcdir/lib/python.sh\"\n\npython=\"${PYTHON:-python}\"\npython=\"$(type -P \"$python\" || die \"'$python' not found\")\"\n\npython_path=\"\"\n\n# finds weird things like this to make $PATH work:\n#\n# /usr/local/Cellar/numpy/1.14.5/libexec/nose/bin\n#\n# Since nose is available in brew Cellar's numpy, it doesn't get installed to ~/Library and isn't found otherwise.\n# Rather than do an --ignore-installed which could cause all sorts of other issues, just use it from wherever it is found\n#\n# $python defined in lib/python.sh\n# shellcheck disable=SC2154\nwhile read -r path; do\n    bin=\"${path%/lib/python*/site-packages*}/bin\"\n    if [ -d \"$bin\" ]; then\n        python_path=\"$bin:$python_path\"\n    fi\ndone < <(\"$python\" -c 'from __future__ import print_function; import sys; print(\"\\n\".join(reversed(sys.path)))' | sed '/^[[:space:]]*$/d')\n\nfor path in ~/Library/Python/*; do\n    bin=\"$path/bin\"\n    [ -d \"$bin\" ] || continue\n    python_path=\"$bin:$python_path\"\ndone\n\n# prioritise our default python at the beginning of the $PATH\npython_version=\"$(\"$python\" -c 'from __future__ import print_function; import sys; print(\"{}.{}\".format(sys.version_info.major, sys.version_info.minor))')\"\nbin=~/Library/Python/\"$python_version\"/bin\n\nif [ -d \"$bin\" ]; then\n    python_path=\"$bin:$python_path\"\nfi\n\nif [ $# -lt 1 ]; then\n    cat <<EOF\nFind where one or more CLI programs are installed by searching all the Python library locations\n\nEspecially useful when Python tools get installed to places not in your \\$PATH - where the 'which' command can't help\n\nusage: ${0##*/} <program> [<program2> <program3> ...]\n\nEOF\n    exit 1\nfi\n\nexport PATH=\"$python_path:$PATH\"\nset +o pipefail\nfound=\"$(type -P \"$@\" | head -n 1)\"\nset -o pipefail\n\nif [ -n \"$found\" ]; then\n    echo \"$found\"\nelse\n    echo \"no Python executable was found matching any of: $*\" >&2\n    echo \"\\$PATH searched was: $PATH\" >&2\n    if is_CI; then\n        echo\n        echo \"running in CI detected, attempting to search all paths\" >&2\n        for x in \"$@\"; do\n            echo \"searching for $x:\" >&2\n            find / -type f -name \"$x\" 2>/dev/null || :\n        done\n    fi\n    exit 1\nfi\n"
  },
  {
    "path": "python/python_find_library_path.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-27 17:54:37 +0100 (Fri, 27 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Shows the path to Python libraries given as arguments\n#\n# There is a better version of this in the adjacent DevOps Python Tools repo called python_find_library_path.py\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\npython=\"${PYTHON:-python}\"\npython=\"$(type -P \"$python\" || die \"'$python' not found\")\"\n\nfind_python_sys_path(){\n    cat <<EOF |\nfrom __future__ import print_function\n# more likely to be right than \\$USER\nimport getpass\nimport os\nimport sys\nuser = getpass.getuser()\nfor path in sys.path:\n    # don't return local /Users/\\$USER/Library/Python/2.7/lib/python/site-packages\n    # as that is not the source of sys\n    if user in path:\n        continue\n    if 'Python.framework' in path:\n        path = path.rsplit('{}lib{}'.format(os.sep, os.sep))[0]\n        print(path)\n        break\n    elif path.endswith('/site-packages'):\n        print(path)\n        break\nEOF\n    \"$python\" #|\n    #sed 's,/python/*[[:digit:].]*/site-packages,,'\n}\n\nif [ $# -eq 0 ]; then\n    find_python_sys_path\nfi\n\nfor arg; do\n    if [ \"$arg\" = \"sys\" ]; then\n        find_python_sys_path\n    else\n        \"$python\" -c \"from __future__ import print_function; import $arg; print($arg.__file__)\"\n    fi\ndone\n"
  },
  {
    "path": "python/python_find_unused_pip_modules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-27 11:49:47 +0000 (Wed, 27 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to find unused Python Pip / PyPI modules in the current git directory tree\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh disable=SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<requirements.txt>]\"\n\nfor x in \"$@\"; do\n    case \"$x\" in\n    -h|--help)  usage\n                ;;\n    esac\ndone\n\nif [ $# -gt 0 ]; then\n    requirements_files=(\"$@\")\nelse\n    # might be more confusing in pytools to find unused modules in subdirs like pylib, so just stick to local\n    #requirements_files=\"$(find . -name requirements.txt)\"\n    requirements_files=(\"requirements.txt\")\n    if [ \"${#requirements_files[@]}\" = 0 ]; then\n        usage \"No requirements files found, please specify explicit path to requirements.txt\"\n    fi\nfi\n\nfound=0\n\npip_modules=\"$(\n    sed 's/#.*//;\n     s/[<>=].*//;\n     s/^[[:space:]]*//;\n     s/[[:space:]]*$//;\n     /^[[:space:]]*$/d;' \"${requirements_files[@]}\" |\n     sort -u |\n     \"$srcdir/python_translate_module_to_import.sh\"\n)\"\n\nwhile read -r module; do\n        # grep -R is sloooow by comparison to git grep\n        #grep -R \"import[[:space:]]\\\\+$module\\\\|from[[:space:]]\\\\+$module[[:space:]]\\\\+import[[:space:]]\\\\+\" . |\n    if ! \\\n        git grep \"import[[:space:]]\\\\+$module\\\\|from[[:space:]]\\\\+$module\\\\([[:alnum:]\\\\.]\\\\+\\\\)\\\\?[[:space:]]\\\\+import[[:space:]]\\\\+\" |\n        grep -v requirements.txt |\n        grep -q .; then\n            echo \"$module\"\n            ((found + 1))\n    fi\ndone <<< \"$pip_modules\"\n\nif [ $found -gt 0 ]; then\n    exit 1\nfi\n"
  },
  {
    "path": "python/python_indices.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-19 18:41:58 +0000 (Thu, 19 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Quick script to convert Python lists to indices for quicker programming debugging / referencing\n#\n# eg. copy it to stdin from the python debug output - used when figuring out log component indicies\n#\n# ./python_indices.sh <<< \"['one', 'two', 'three']\"\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\ncat \"$@\" |\npython -c '\nfrom __future__ import print_function\nimport ast\n#import json\nimport sys\nstdin = sys.stdin.read()\nmy_list = ast.literal_eval(stdin)\n#my_list = json.loads(stdin)\ni = 0\nfor item in my_list:\n    print(i, item)\n    i += 1\n'\n"
  },
  {
    "path": "python/python_pip_install.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:56:24 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs to --user on Mac to avoid System Integrity Protection built in to OS X El Capitan and later\n#\n# Also detects and sets up OpenSSL and Kerberos library paths on Mac when using HomeBrew\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/ci.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/python.sh\"\n\n# want arg splitting\n# shellcheck disable=SC2206\nopts=(${PIP_OPTS:-})\n\nusage(){\n    echo \"Installs Python PyPI modules using Pip, taking in to account library paths, virtual envs etc\"\n    echo\n    echo \"Takes a list of python module names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"You may need to set this in your environment to install to system or user libraries in newer versions of pip:\"\n    echo\n    echo \"  export PIP_BREAK_SYSTEM_PACKAGES=1\"\n    echo\n    echo\n    echo \"usage: ${0##*} <list_of_modules>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\npip_modules=()\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding pip modules from file:  $arg\"\n            # want splitting\n            # shellcheck disable=SC2207\n            pip_modules+=($(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\"))\n            echo\n        else\n            pip_modules+=(\"$arg\")\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${pip_modules[*]}\" ]; then\n    usage\nfi\n\n# want splitting\n# shellcheck disable=SC2207\npip_modules=($(tr '[:space:]' ' \\n' <<< \"${pip_modules[@]}\" | sort -u | tr '\\n' ' '))\n\necho \"Installing Python PyPI Modules\"\necho\n\nif is_CI; then\n    #echo \"running in quiet mode for CI to minimize log noise\"\n    opts+=(-q)\nfi\n\nsudo=\"\"\nif inside_virtualenv; then\n    echo \"inside virtualenv, not using sudo\"\n    sudo=\"\"\nelif [ $EUID != 0 ]; then\n    sudo=\"sudo --preserve-env=PIP_BREAK_SYSTEM_PACKAGES\"\nfi\n\nuser_opt(){\n    if inside_virtualenv; then\n        echo \"inside virtualenv, ignoring --user switch which wouldn't work\"\n        sudo=\"\"\n    else\n        opts+=(--user)\n        sudo=\"\"\n    fi\n}\n\nenvopts=()\nexport LDFLAGS=\"\"\nif [ \"$(uname -s)\" = \"Darwin\" ]; then\n    # setting these caused compile errors failing to find stdio.h when pip installing requests-kerberos\n#    if type -P brew &>/dev/null; then\n#        # usually /usr/local\n#        brew_prefix=\"$(brew --prefix)\"\n#\n#        export OPENSSL_INCLUDE=\"$brew_prefix/opt/openssl/include\"\n#        export OPENSSL_LIB=\"$brew_prefix/opt/openssl/lib\"\n#\n#        export LDFLAGS=\"${LDFLAGS:-} -L$brew_prefix/lib\"\n#        export CFLAGS=\"${CFLAGS:-} -I$brew_prefix/include\"\n#        export CPPFLAGS=\"${CPPFLAGS:-} -I$brew_prefix/include\"\n#\n#        # for OpenSSL\n#        export LDFLAGS=\"${LDFLAGS:-} -L$OPENSSL_LIB\"\n#        export CFLAGS=\"${CFLAGS:-} -I$OPENSSL_INCLUDE\"\n#        export CPPFLAGS=\"${CPPFLAGS:-} -I$OPENSSL_INCLUDE\"\n#\n#        # for Kerberos\n#        export LDFLAGS=\"${LDFLAGS:-} -L$brew_prefix/opt/krb5/lib\"\n#        export CFLAGS=\"${CFLAGS:-} -I$brew_prefix/opt/krb5/include -I $brew_prefix/opt/krb5/include/krb5\"\n#        export CPPFLAGS=\"${CPPFLAGS:-} -I$brew_prefix/opt/krb5/include -I $brew_prefix/opt/krb5/include/krb5\"\n#\n#        #export CPATH=\"${CPATH:-}:$brew_prefix/lib\"\n#        #export LIBRARY_PATH=\"${LIBRARY_PATH:-}:$brew_prefix/lib\"\n#\n#        # need to send OPENSSL_INCLUDE and OPENSSL_LIB through sudo explicitly using prefix\n#        envopts=(OPENSSL_INCLUDE=\"$OPENSSL_INCLUDE\" OPENSSL_LIB=\"$OPENSSL_LIB\") # LDFLAGS=\"$LDFLAGS\" CFLAGS=\"$CFLAGS\" CPPFLAGS=\"$CPPFLAGS\")\n#    fi\n    # avoids Mac's System Integrity Protection built in to OS X El Capitan and later\n    user_opt\nelif [ -n \"${PYTHON_USER_INSTALL:-}\" ] ||\n     [ -n \"${GOOGLE_CLOUD_SHELL:-}\" ]; then\n    user_opt\nfi\n\nif [ -n \"${NO_FAIL:-}\" ]; then\n    for pip_module in \"${pip_modules[@]}\"; do\n        # pip defined in lib/python.sh\n        # shellcheck disable=SC2154\n        echo \"$sudo $pip install ${opts[*]:-} $pip_module\"\n        # want splitting of opts\n        # shellcheck disable=SC2068\n        $sudo ${envopts[@]:-} \"$pip\" install ${opts[@]:-} \"$pip_module\"\n    done\nelse\n    echo \"$sudo $pip install ${opts[*]:-} ${pip_modules[*]}\"\n    # want splitting of opts and modules\n    # shellcheck disable=SC2068\n    $sudo ${envopts[@]:-} \"$pip\" install ${opts[@]:-} \"${pip_modules[@]}\"\nfi\n"
  },
  {
    "path": "python/python_pip_install_for_script.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-05-07 16:28:58 +0100 (Fri, 07 May 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all Python modules installed in a script and installs them if not already present\n\nUses adjacent scripts:\n\n    python_translate_import_to_pip.sh\n    python_pip_install_if_absent.sh\n\n\nWill break on custom local modules since they won't be found on PyPI, use --exclude <regex> to avoid that. Regex is in ERE format\n\nEg.\n    ${0##*/} *.py --exclude harisekhon  # excludes my personal local library which is not on PyPI\n\n\nIf you supply .pyc or .pyo filenames, it will infer to .py instead. This is useful if calling from a Makefile looking for .pyc or .pyo dynamic targets\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<script1.py> [<script2.py> <script3.py>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nscripts=()\nexclude_regex=\"\"\n\nuntil [ $# -lt 1 ]; do\n    case \"$1\" in\n        -e|--exclude)   exclude_regex=\"${2:-}\"\n                        shift || :\n                        ;;\n                  -*)   usage\n                        ;;\n                    *)  if [[ \"$1\" =~ \\.py[co] ]]; then\n                            filename=\"$1\"\n                            filename=\"${filename%.pyc}\"\n                            filename=\"${filename%.pyo}\"\n                            filename=\"$filename.py\"\n                            scripts+=(\"$filename\")\n                        else\n                            scripts+=(\"$1\")\n                        fi\n                        ;;\n    esac\n    shift || :\ndone\n\nif [ ${#scripts[@]} -eq 0 ]; then\n    usage\nfi\n\npip_modules_import=\"$(\n    grep -Eh '^[[:space:]]*import[[:space:]]' \"${scripts[@]}\" |\n    awk '{print $2}' |\n    tr -d '\\r' |\n    sort -u\n)\"\npip_modules_from_import=\"$(\n    grep -Eho '^[[:space:]]*from[[:space:]][^[[:space:]_]+[[:space:]]+import[[:space:]]+[^[:space:]]+$' \"${scripts[@]}\" |\n    sed 's/^[[:space:]]*from[[:space:]]*//; s/[[:space:]][[:space:]]*import[[:space:]].*$//' |\n    sort -u\n)\"\n\npip_modules=\"$(\n    sort -u <<-EOF |\n    $pip_modules_import\n    $pip_modules_from_import\nEOF\n    if [ -n \"$exclude_regex\" ]; then\n        grep -Ev \"$exclude_regex\" || :\n    else\n        cat\n    fi\n)\"\n\npip_modules=\"$(\"$srcdir/python_translate_import_to_module.sh\" <<< \"$pip_modules\")\"\n\n\"$srcdir/python_pip_install_if_absent.sh\" <<< \"$pip_modules\"\n"
  },
  {
    "path": "python/python_pip_install_if_absent.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-15 13:56:24 +0000 (Fri, 15 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Avoids trying to install / upgrade pip modules that have been installed with system packages to avoid errors like the following:\n#\n# Cannot uninstall 'beautifulsoup4'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/python.sh\"\n\nusage(){\n    echo \"Installs Python PyPI modules not already installed using Pip\"\n    echo\n    echo \"Leverages adjacent python_pip_install.sh which takes in to account library paths, virtual envs etc\"\n    echo\n    echo \"Takes a list of python module names as arguments or .txt files containing lists of modules (one per line)\"\n    echo\n    echo \"You may need to set this in your environment to install to system or user libraries in newer versions of pip:\"\n    echo\n    echo \"  export PIP_BREAK_SYSTEM_PACKAGES=1\"\n    echo\n    echo\n    echo \"usage: ${0##*} <list_of_modules>\"\n    echo\n    exit 3\n}\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\npip_modules=\"\"\n\nprocess_args(){\n    for arg; do\n        if [ -f \"$arg\" ]; then\n            echo \"adding pip modules from file:  $arg\"\n            pip_modules=\"$pip_modules $(sed 's/#.*//;/^[[:space:]]*$$/d' \"$arg\")\"\n            echo\n        else\n            pip_modules=\"$pip_modules $arg\"\n        fi\n    done\n}\n\nif [ $# -gt 0 ]; then\n    process_args \"$@\"\nelse\n    # shellcheck disable=SC2046\n    process_args $(cat)\nfi\n\nif [ -z \"${pip_modules// }\" ]; then\n    usage\nfi\n\npip_modules=\"$(tr ' ' ' \\n' <<< \"$pip_modules\" | sort -u | tr '\\n' ' ')\"\n\necho \"Installing Python PyPI Modules that are not already installed\"\necho\n\n# doesn't solve the problem on CircleCI:\n#\n# Traceback (most recent call last):\n#   File \"/home/circleci/.local/bin/pip\", line 5, in <module>\n#     from pip._internal.cli.main import main\n# ImportError: No module named pip._internal.cli.main\n#\n#if is_CI; then\n#    echo \"attempting to upgrade pip to solve common CI/CD problems\"\n#    sudo='sudo'\n#    if [ \"${EUID:-${UID:-$(id -u)}}\" = 0 ]; then\n#        sudo=''\n#    fi\n#    \"$sudo\" \"$python\" -m pip install --upgrade pip || :\n#    echo\n#fi\n\nfor pip_module in $pip_modules; do\n    python_module=\"$(\"$srcdir/python_translate_module_to_import.sh\" <<< \"$pip_module\")\"\n\n    # pip module often pull in urllib3 which result in errors like the following so ignore it\n    #:\n    # Cannot uninstall 'urllib3'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n    #\n    #echo \"checking if python module '$python_module' is installed\"\n    # assigned in lib/python.sh\n    # shellcheck disable=SC2154\n    if \"$python\" -c \"import $python_module\" &>/dev/null; then\n        echo \"python module '$python_module' already installed, skipping...\"\n    else\n        echo \"installing python module '$python_module'\"\n        echo\n        \"$srcdir/python_pip_install.sh\" \"$pip_module\"\n    fi\ndone\n"
  },
  {
    "path": "python/python_pip_reinstall_all_modules.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-29 17:57:47 +0000 (Sat, 29 Feb 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Reinstalls all Python Pip modules which is often the fix for the python startup error\n#\n# \"Illegal instruction: 4\"\n#\n# which is often caused by some corrupted module or incompatability vs local CPU architecture\n# which recompiling often solves\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nif [ -n \"${PIP:-}\" ]; then\n    pip=\"$PIP\"\nelse\n    pip=pip\n    if ! type -P \"pip\" &>/dev/null; then\n        echo \"pip not found, falling back to pip2\"\n        pip=pip2\n    fi\nfi\nopts=\"${PIP_OPTS:-}\"\n\n# want splitting\n# shellcheck disable=SC2086\n\"$pip\" $opts freeze | PIP_OPTS=\"--force-reinstall\" xargs \"$srcdir/python_pip_install.sh\"\n"
  },
  {
    "path": "python/python_pyinstaller.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-10 16:01:58 +0100 (Thu, 10 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Creates PyInstaller self-contained bundles of all python files given as args or all python files in the current and first subdirectory (to avoid creating bundles of libraries)\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck source=lib/utils.sh\n. \"$srcdir/lib/utils.sh\"\n\nfilelist=\"$(find \"${1:-.}\" -maxdepth 2 -type f -iname '*.py')\"\n\nif [ -z \"$filelist\" ]; then\n    echo \"no Python\"\n    echo\n    echo \"usage: ${0##*/} <python_file_or_directory>\"\n    echo\n    return 0 &>/dev/null || :\n    exit 0\nfi\n\nsection \"Compiling and bundling Python code using PyInstaller\"\n\nstart_time=\"$(start_timer)\"\n\nopts=(${PYINSTALL_OPTS:+\"$PYINSTALLER_OPTS\"})\n\nopts+=(-y)\nif [ -d pylib ]; then\n    opts+=(--paths pylib)\n    if [ -d pylib/resources ]; then\n        for filename in pylib/resources/*; do\n            opts+=(--add-data \"$filename:resources\")\n        done\n    fi\nfi\n\nfor filename in $filelist; do\n    type isExcluded &>/dev/null && isExcluded \"$filename\" && continue\n    echo \"compiling $filename => dist/$filename\"\n    # want opt expansion\n    # shellcheck disable=SC2086\n    pyinstaller ${opts:+\"${opts[@]}\"} \"$filename\"\n    echo\ndone\n\ntime_taken \"$start_time\"\nsection2 \"Finished PyInstaller compilation\"\necho\n"
  },
  {
    "path": "python/python_pypi_versions.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: superset\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-19 15:58:35 +0100 (Sun, 19 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nLists the versions of a Python package on PyPI\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<package_name> [<curl_options>]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\npackage=\"$1\"\n\nshift || :\n\nresponse=\"$(curl -sSL \"https://pypi.org/pypi/$package/json\" \"$@\")\"\n\nif ! jq -r '.releases | keys | .[]' <<< \"$response\"; then\n    cat >&2 <<EOF\n$response\nEOF\n    exit 1\nfi\n"
  },
  {
    "path": "python/python_translate_import_to_module.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: yaml git\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-19 01:55:24 +0000 (Tue, 19 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTranslates Python import statement module names to pip module names\n\nUsed by python_pip_install_for_script.sh to parse import statements to pip modules to check they're installed\n\n\nReads from standard input if no args are given\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<module1> <module2> ...]\"\n\nhelp_usage \"$@\"\n\n\nmappings=\"$srcdir/../resources/pipreqs_mapping.txt\"\n\nif ! [ -f \"$mappings\" ]; then\n    wget -O \"$mappings\" https://raw.githubusercontent.com/bndr/pipreqs/master/pipreqs/mapping\nfi\n\nsed_script=\"$(\n    tr ':' ' ' < \"$mappings\" |\n    while read -r import_name module_name rest; do\n        if ! [[ \"$import_name\" =~ ^[A-Za-z0-9/_.-]+$ ]]; then\n            echo \"import name '$import_name' did not match expected alphanumeric regex!\" >&2\n            continue\n        fi\n        if ! [[ \"$module_name\" =~ ^[A-Za-z0-9_.-]+$ ]]; then\n            echo \"import module name '$module_name' did not match expected alphanumeric regex!\" >&2\n            continue\n        fi\n        echo \"s|^$import_name$|$module_name|;\"\n    done\n)\"\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        if [ -f \"$x\" ]; then\n            cat \"$x\"\n        else\n            echo \"$x\"\n        fi\n    done\nelse\n    cat\nfi |\nsed \"$sed_script\" |\n    # - import names replace dashes with underscores\nsed '\n    s/_/-/g;\n'\n"
  },
  {
    "path": "python/python_translate_module_to_import.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: PyYAML GitPython\n#\n#  Author: Hari Sekhon\n#  Date: 2019-02-19 01:55:24 +0000 (Tue, 19 Feb 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTranslates Python pip modules to import names\n\nUsed by adjacent script python_pip_install_if_absent.sh to check if a module is available somewhere in the path by trying to import it\n\n\nReads from standard input if no args are given\n\"\n\n# shellcheck disable=SC2034\nusage_args=\"[<module1> <module2> ...]\"\n\nhelp_usage \"$@\"\n\n\nmappings=\"$srcdir/../resources/pipreqs_mapping.txt\"\n\nif ! [ -f \"$mappings\" ]; then\n    wget -O \"$mappings\" https://raw.githubusercontent.com/bndr/pipreqs/master/pipreqs/mapping\nfi\n\nsed_script=\"$(\n    tr ':' ' ' < \"$mappings\" |\n    while read -r import_name module_name rest; do\n        if ! [[ \"$import_name\" =~ ^[A-Za-z0-9/_.-]+$ ]]; then\n            echo \"import name '$import_name' did not match expected alphanumeric regex!\" >&2\n            continue\n        fi\n        if ! [[ \"$module_name\" =~ ^[A-Za-z0-9_.-]+$ ]]; then\n            echo \"import module name '$module_name' did not match expected alphanumeric regex!\" >&2\n            continue\n        fi\n        echo \"s|^$module_name$|$import_name|;\"\n    done\n)\"\n\nif [ $# -gt 0 ]; then\n    for x in \"$@\"; do\n        if [ -f \"$x\" ]; then\n            cat \"$x\"\n        else\n            echo \"$x\"\n        fi\n    done\nelse\n    cat\nfi |\nsed 's/[>=].*$//;' |\n    # these have been replaced my pipreqs mapping file\n    #s/beautifulsoup4/bs4/;\n    #s/PyYAML/yaml/;\n    #s/GitPython/git/;\n    #s/traceback2/traceback/;\nsed \"$sed_script\" |\n    # general rules:\n    # - import names don't have python-* prefixes\n    # - import names don't have *-python suffixes\n    # - import names replace dashes with underscores\n    # - psycopg2-binary -> psycopg2\nsed '\n    s/^python-//;\n    s/-*python$//;\n    s/-binary$//;\n    s/-/_/g;\n    s/\\[.*\\]//\n' # |\n    # generally lowercase but a couple of exceptions - these are in mappings file so we don't execute this conversion any more\n#tr '[:upper:]' '[:lower:]' |\n#sed '\n#    s/mysqldb/MySQLdb/;\n#    s/krbv/krbV/;\n#'\n"
  },
  {
    "path": "python/pythonpath.sh",
    "content": "#!/usr/bin/env bash\n# shellcheck disable=SC2230\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-27 17:12:39 +0100 (Fri, 27 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Prints the Python sys.path, one per line\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# copied from old bashrc function which is now ported under .bash.d/ too, but given here for convenience in case you are not running the full bash profile\npython -c 'from __future__ import print_function; import sys; [print(_) for _ in sys.path if _]'\n"
  },
  {
    "path": "requirements.txt",
    "content": "aws-consoler>=1.1.0\ncheckov>=2.0.618\n#git-remote-codecommit>=1.16\ngrip>=4.6.1\njsonlint>=0.1\npycookiecheat==0.8.0\npylint>=1.9.5\n# can only be installed on macOS - moved to setup/pip-packages-mac.txt\n#pyobjc-framework-Quartz>=9.0.1\nsemgrep>=0.78.0\nspeedtest-cli>=2.1.3\nsqlfluff>=3.2.2\nwakatime>=14.0.1\nyamllint>=1.15.0\n"
  },
  {
    "path": "resources/oreilly-animals.json",
    "content": "[\n  {\n    \"animal\": \"12-Wired Bird of Paradise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155452/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155452.do\",\n    \"title\": \"Mobile Design and Development\"\n  },\n  {\n    \"animal\": \"3-Banded Armadillo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024491/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024491.do\",\n    \"title\": \"Windows PowerShell for Developers\"\n  },\n  {\n    \"animal\": \"Aardvark\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007065/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007065.do\",\n    \"title\": \"Jakarta Commons Cookbook\"\n  },\n  {\n    \"animal\": \"Aardwolf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029786/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029786.do\",\n    \"title\": \"Clojure Cookbook\"\n  },\n  {\n    \"animal\": \"Addax, aka Screwhorn Antelope\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804855/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804855.do\",\n    \"title\": \"Ubuntu: Up and Running\"\n  },\n  {\n    \"animal\": \"Adjutant (Storks)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030027/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030027.do\",\n    \"title\": \"Social eCommerce\"\n  },\n  {\n    \"animal\": \"Aegina Citrea, narcomedusae, jellyfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033783/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033783.do\",\n    \"title\": \"BioBuilder\"\n  },\n  {\n    \"animal\": \"African Civet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596519650/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596519650.do\",\n    \"title\": \"JRuby Cookbook\"\n  },\n  {\n    \"animal\": \"African Crowned Crane aka Grey Crowned Crane\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023975/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023975.do\",\n    \"title\": \"C# 5.0 Pocket Reference\"\n  },\n  {\n    \"animal\": \"African Crowned Crane aka Grey Crowned Crane\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527433/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527433.do\",\n    \"title\": \"Programming C# 3.0\"\n  },\n  {\n    \"animal\": \"African Crowned Crane aka Grey Crowned Crane\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596159849/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596159849.do\",\n    \"title\": \"Programming C# 4.0\"\n  },\n  {\n    \"animal\": \"African Elephant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510008/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510008.do\",\n    \"title\": \"MCSA on Windows Server 2003 Core Exams in a Nutshell\"\n  },\n  {\n    \"animal\": \"African Elephant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000301/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000301.do\",\n    \"title\": \"MCSE: Windows 2000 Exams in a Nutshell\"\n  },\n  {\n    \"animal\": \"African Elephant, young\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033448/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033448.do\",\n    \"title\": \"Hadoop: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"African Forest Buffalo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920218494/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920218494.do\",\n    \"title\": \"Semantic Software Design\"\n  },\n  {\n    \"animal\": \"African Palm Civet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033998/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033998.do\",\n    \"title\": \"CSS Refactoring\"\n  },\n  {\n    \"animal\": \"African Rock Python\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596158118/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596158118.do\",\n    \"title\": \"Programming Python\"\n  },\n  {\n    \"animal\": \"African Sacred Ibis\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596158019/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596158019.do\",\n    \"title\": \"Google Analytics\"\n  },\n  {\n    \"animal\": \"African Soft Shell Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025740/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025740.do\",\n    \"title\": \"Developing with Couchbase Server\"\n  },\n  {\n    \"animal\": \"African Spoonbill, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003616/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003616.do\",\n    \"title\": \"ADO.NET in a Nutshell\"\n  },\n  {\n    \"animal\": \"African Tent Tortoise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005955/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005955.do\",\n    \"title\": \"Classic Shell Scripting\"\n  },\n  {\n    \"animal\": \"African Wild Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023227/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023227.do\",\n    \"title\": \"PayPal APIs: Up and Running\"\n  },\n  {\n    \"animal\": \"Agamidae Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009762/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009762.do\",\n    \"title\": \"SQL Cookbook\"\n  },\n  {\n    \"animal\": \"Agile antechinus (Mouselike)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020950/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020950.do\",\n    \"title\": \"Automating ActionScript Projects with Eclipse and Ant\"\n  },\n  {\n    \"animal\": \"Alactaga, Long-Eared Jerboa\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022671/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022671.do\",\n    \"title\": \"Just Spring Integration\"\n  },\n  {\n    \"animal\": \"Alaskan Plaice, aka flatfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028062/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028062.do\",\n    \"title\": \"Learning PHP Design Patterns\"\n  },\n  {\n    \"animal\": \"Alleghany Hellbender Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030515/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030515.do\",\n    \"title\": \"Introduction to Machine Learning with Python\"\n  },\n  {\n    \"animal\": \"Alligator\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527754/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527754.do\",\n    \"title\": \"Java Generics and Collections\"\n  },\n  {\n    \"animal\": \"Alpaca\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920012689/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920012689.do\",\n    \"title\": \"Intermediate Perl\"\n  },\n  {\n    \"animal\": \"Alpine Chamois\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596806033/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596806033.do\",\n    \"title\": \"HTML5: Up and Running\"\n  },\n  {\n    \"animal\": \"American Bison\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003876/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003876.do\",\n    \"title\": \"Java Extreme Programming Cookbook\"\n  },\n  {\n    \"animal\": \"American Bittern bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023869/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023869.do\",\n    \"title\": \"HLSL and Pixel Shaders for XAML Developers\"\n  },\n  {\n    \"animal\": \"American Crocodile\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032991/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032991.do\",\n    \"title\": \"Crafting the InfoSec Playbook\"\n  },\n  {\n    \"animal\": \"American Crocodile, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006204/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006204.do\",\n    \"title\": \"Java Examples in a Nutshell\"\n  },\n  {\n    \"animal\": \"American Lobster\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029151/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029151.do\",\n    \"title\": \"Lift Cookbook\"\n  },\n  {\n    \"animal\": \"American Marsh Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021292/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021292.do\",\n    \"title\": \"Introduction to Tornado\"\n  },\n  {\n    \"animal\": \"American Ostrich\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030287/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030287.do\",\n    \"title\": \"Learning Scala\"\n  },\n  {\n    \"animal\": \"American Oystercatcher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021216/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021216.do\",\n    \"title\": \"ActionScript Developer's Guide to Robotlegs\"\n  },\n  {\n    \"animal\": \"American Robin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026907/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026907.do\",\n    \"title\": \"MySQL High Availability\"\n  },\n  {\n    \"animal\": \"Amur Hedgehog (aka common europaeus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029557/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029557.do\",\n    \"title\": \"Data Structures and Algorithms with JavaScript\"\n  },\n  {\n    \"animal\": \"Andean Condor (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920075837/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920075837.do\",\n    \"title\": \"Cloud Native Infrastructure\"\n  },\n  {\n    \"animal\": \"Angler Fish (aka monkfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028833/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028833.do\",\n    \"title\": \"Releasing HTML5 Games for Windows 8\"\n  },\n  {\n    \"animal\": \"Angora Goat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100148/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100148.do\",\n    \"title\": \"JUNOS Cookbook\"\n  },\n  {\n    \"animal\": \"Angular Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920045113/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920045113.do\",\n    \"title\": \"Principles of Data Wrangling\"\n  },\n  {\n    \"animal\": \"Antarctic Giant Petrel aka Giant Fulmar (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024750/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024750.do\",\n    \"title\": \"Programming Grails\"\n  },\n  {\n    \"animal\": \"Aoudad, aka Barbary sheep\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007379/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007379.do\",\n    \"title\": \"Perl 6 and Parrot Essentials\"\n  },\n  {\n    \"animal\": \"Appaloosa Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002039/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002039.do\",\n    \"title\": \"Apache: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Arabian Camel, aka Dromedary\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003104/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003104.do\",\n    \"title\": \"Computer Science & Perl Programming\"\n  },\n  {\n    \"animal\": \"Arabian Camel, aka Dromedary\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018476/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018476.do\",\n    \"title\": \"Perl Pocket Reference\"\n  },\n  {\n    \"animal\": \"Arabian Camel, aka Dromedary, Head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002411/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002411.do\",\n    \"title\": \"Perl in a Nutshell\"\n  },\n  {\n    \"animal\": \"Arabian Horse, spotted\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596154493/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596154493.do\",\n    \"title\": \"Linux in a Nutshell\"\n  },\n  {\n    \"animal\": \"Arched Duck, aka Whistling Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021681/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021681.do\",\n    \"title\": \"What's New in Adobe AIR 3\"\n  },\n  {\n    \"animal\": \"Arctic Cod Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004798/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004798.do\",\n    \"title\": \"Programming SQL Server 2005\"\n  },\n  {\n    \"animal\": \"Arctic Tern\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003807/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003807.do\",\n    \"title\": \"Programming ColdFusion MX\"\n  },\n  {\n    \"animal\": \"Argus Pheasant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032335/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032335.do\",\n    \"title\": \"SVG Essentials\"\n  },\n  {\n    \"animal\": \"Armadillo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003432/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003432.do\",\n    \"title\": \"Essential System Administration\"\n  },\n  {\n    \"animal\": \"Armadillo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004491/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004491.do\",\n    \"title\": \"Essential System Administration Pocket Reference\"\n  },\n  {\n    \"animal\": \"Armadillo, Nine-banded\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028529/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028529.do\",\n    \"title\": \"Doing Data Science\"\n  },\n  {\n    \"animal\": \"Armed Hermit Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920221432/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920221432.do\",\n    \"title\": \"Learning Apache OpenWhisk\"\n  },\n  {\n    \"animal\": \"Ascension Frigate (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920175131/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920175131.do\",\n    \"title\": \"Cloud Native DevOps with Kubernetes\"\n  },\n  {\n    \"animal\": \"Asian Civet, Zibeth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020356/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020356.do\",\n    \"title\": \"Graphics and Animation on iOS\"\n  },\n  {\n    \"animal\": \"Asian Painted Frog (aka Chubby Frog)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000608/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000608.do\",\n    \"title\": \"Windows Me Annoyances\"\n  },\n  {\n    \"animal\": \"Asp Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928435/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928435.do\",\n    \"title\": \"ASP in a Nutshell\"\n  },\n  {\n    \"animal\": \"Asp Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927506/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927506.do\",\n    \"title\": \"Developing ASP Components\"\n  },\n  {\n    \"animal\": \"Asse Caama Fox, aka Cape Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020349/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020349.do\",\n    \"title\": \"Writing Game Center Apps in iOS\"\n  },\n  {\n    \"animal\": \"Atlantic Cod (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028536/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028536.do\",\n    \"title\": \"Enterprise Data Workflows with Cascading\"\n  },\n  {\n    \"animal\": \"Atlantic Herring (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029229/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029229.do\",\n    \"title\": \"Anonymizing Health Data\"\n  },\n  {\n    \"animal\": \"Atlantic Wreckfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596515812/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596515812.do\",\n    \"title\": \"Programming Amazon Web Services\"\n  },\n  {\n    \"animal\": \"Australasian Gannet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034285/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034285.do\",\n    \"title\": \"Swift Development with Cocoa\"\n  },\n  {\n    \"animal\": \"Australasian Snapper\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025344/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025344.do\",\n    \"title\": \"Developing Backbone.js Applications\"\n  },\n  {\n    \"animal\": \"Australian Bee-Eater Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028925/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028925.do\",\n    \"title\": \"RESTful Java with JAX-RS 2.0\"\n  },\n  {\n    \"animal\": \"Australian Magpie (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920056379/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920056379.do\",\n    \"title\": \"Think Data Structures\"\n  },\n  {\n    \"animal\": \"Axis Deer\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102425/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102425.do\",\n    \"title\": \"Fonts & Encodings\"\n  },\n  {\n    \"animal\": \"Axolotl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022350/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022350.do\",\n    \"title\": \"Machine Learning for Email\"\n  },\n  {\n    \"animal\": \"Aye-Aye (lemur)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000981/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000981.do\",\n    \"title\": \"Exim: The Mail Transfer Agent\"\n  },\n  {\n    \"animal\": \"Azure Jay (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920254812/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920254812.do\",\n    \"title\": \"Building Intelligent Cloud Applications\"\n  },\n  {\n    \"animal\": \"Babirusa Wild Pig aka Sulawesi Babirusa\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780937175026/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780937175026.do\",\n    \"title\": \"Programming with curses\"\n  },\n  {\n    \"animal\": \"Baboon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007171/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007171.do\",\n    \"title\": \"Exchange Server Cookbook\"\n  },\n  {\n    \"animal\": \"Badger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004767/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004767.do\",\n    \"title\": \"Perl Template Toolkit\"\n  },\n  {\n    \"animal\": \"Bailey's Shrew\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022602/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022602.do\",\n    \"title\": \"Testing in Scala\"\n  },\n  {\n    \"animal\": \"Baloenceps Shoebill (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021698/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021698.do\",\n    \"title\": \"What's New in Flash Player 11\"\n  },\n  {\n    \"animal\": \"Banded Angelfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029175/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029175.do\",\n    \"title\": \"Learning MySQL and MariaDB\"\n  },\n  {\n    \"animal\": \"Banded Broadbill\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009571/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009571.do\",\n    \"title\": \"Understanding MySQL Internals\"\n  },\n  {\n    \"animal\": \"Banded Linsang, Delundung Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514433/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514433.do\",\n    \"title\": \"Web 2.0 Architectures\"\n  },\n  {\n    \"animal\": \"Bank Vole\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002473/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002473.do\",\n    \"title\": \"Jython Essentials\"\n  },\n  {\n    \"animal\": \"Barbary Ape, Barbary Macaque\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005399/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005399.do\",\n    \"title\": \"JavaServer Faces\"\n  },\n  {\n    \"animal\": \"Barbary Partridge\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920181576/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920181576.do\",\n    \"title\": \"Deep Learning from Scratch\"\n  },\n  {\n    \"animal\": \"Barbastelle Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021988/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021988.do\",\n    \"title\": \"802.11n: A Survival Guide\"\n  },\n  {\n    \"animal\": \"Barbel Flying Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026099/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026099.do\",\n    \"title\": \"Building Web, Cloud, and Mobile Solutions with F#\"\n  },\n  {\n    \"animal\": \"Bare-tailed Woolly Opossum\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033059/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033059.do\",\n    \"title\": \"Introducing GitHub\"\n  },\n  {\n    \"animal\": \"Bare-throated Bellbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002022/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002022.do\",\n    \"title\": \"Programming Jabber\"\n  },\n  {\n    \"animal\": \"Barred Bandicoot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027393/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027393.do\",\n    \"title\": \"Bandit Algorithms for Website Optimization\"\n  },\n  {\n    \"animal\": \"Barred Owl, flying\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004033/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004033.do\",\n    \"title\": \"Kerberos: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Basilisk Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001865/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001865.do\",\n    \"title\": \"Network Troubleshooting Tools\"\n  },\n  {\n    \"animal\": \"Basket Star (starfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033905/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033905.do\",\n    \"title\": \"MPLS in the SDN Era\"\n  },\n  {\n    \"animal\": \"Bear Paw Clam\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025481/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025481.do\",\n    \"title\": \"Understanding Computation\"\n  },\n  {\n    \"animal\": \"Bear on hind legs\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565920378/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565920378.do\",\n    \"title\": \"SCO UNIX in a Nutshell\"\n  },\n  {\n    \"animal\": \"Beaver\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009533/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009533.do\",\n    \"title\": \"Essential Microsoft Operations Manager\"\n  },\n  {\n    \"animal\": \"Bee-eater Bird, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002916/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002916.do\",\n    \"title\": \"XPath and XPointer\"\n  },\n  {\n    \"animal\": \"Beech Marten\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030690/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030690.do\",\n    \"title\": \"Building Polyfills\"\n  },\n  {\n    \"animal\": \"Beisa Oryx, or East African Oryx (Antelope)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927254/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927254.do\",\n    \"title\": \"Writing Word Macros\"\n  },\n  {\n    \"animal\": \"Belgian Shepherd\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004439/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004439.do\",\n    \"title\": \"Macintosh Troubleshooting Pocket Guide for Mac OS\"\n  },\n  {\n    \"animal\": \"Belgian Tervuren Sheepdog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019923/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019923.do\",\n    \"title\": \"Gradle Beyond the Basics\"\n  },\n  {\n    \"animal\": \"Beluga Whale\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920036791/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920036791.do\",\n    \"title\": \"Docker Cookbook\"\n  },\n  {\n    \"animal\": \"Berkshire Pig, or Common Pig\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920044383/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920044383.do\",\n    \"title\": \"Programming Pig\"\n  },\n  {\n    \"animal\": \"Bighorn Sheep\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003135/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003135.do\",\n    \"title\": \"Perl Cookbook\"\n  },\n  {\n    \"animal\": \"Bilby, Rabbit-eared Bandicoot (Macrotis lagotis)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002763/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002763.do\",\n    \"title\": \"Java Data Objects\"\n  },\n  {\n    \"animal\": \"Binturong\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030546/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030546.do\",\n    \"title\": \"Apache Oozie\"\n  },\n  {\n    \"animal\": \"Black Cockatoo, aka Palm Cockatoo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026532/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026532.do\",\n    \"title\": \"Async in C# 5.0\"\n  },\n  {\n    \"animal\": \"Black Crow 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001001/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001001.do\",\n    \"title\": \"Running Weblogs with Slash\"\n  },\n  {\n    \"animal\": \"Black Fox Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020387/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020387.do\",\n    \"title\": \"SharePoint 2010 for Project Management\"\n  },\n  {\n    \"animal\": \"Black Grouse (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028789/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028789.do\",\n    \"title\": \"Effective Akka\"\n  },\n  {\n    \"animal\": \"Black Jaguar\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527228/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527228.do\",\n    \"title\": \"Cisco IOS Cookbook\"\n  },\n  {\n    \"animal\": \"Black Leopard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004569/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004569.do\",\n    \"title\": \"Advanced Perl Programming\"\n  },\n  {\n    \"animal\": \"Black Lion Marmoset\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025849/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025849.do\",\n    \"title\": \"Learning Agile\"\n  },\n  {\n    \"animal\": \"Black Retriever Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026839/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026839.do\",\n    \"title\": \"Resilience and Reliability on AWS\"\n  },\n  {\n    \"animal\": \"Black Tree Kangaroo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035367/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035367.do\",\n    \"title\": \"Knockout.js\"\n  },\n  {\n    \"animal\": \"Black giant squirrel with a nut, aka Javan Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596803742/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596803742.do\",\n    \"title\": \"Java: The Good Parts\"\n  },\n  {\n    \"animal\": \"Black swan, detail of head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565921122/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565921122.do\",\n    \"title\": \"Programming with GNU Software\"\n  },\n  {\n    \"animal\": \"Black-Billed Australian Bustard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018438/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018438.do\",\n    \"title\": \"Data Mashups in R\"\n  },\n  {\n    \"animal\": \"Black-backed Jackal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032090/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032090.do\",\n    \"title\": \"UX Strategy\"\n  },\n  {\n    \"animal\": \"Black-faced Spider Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920047391/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920047391.do\",\n    \"title\": \"Programming Beyond Practices\"\n  },\n  {\n    \"animal\": \"Black-headed Caique (parrot, bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920052012/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920052012.do\",\n    \"title\": \"OpenShift for Developers\"\n  },\n  {\n    \"animal\": \"Black-headed gull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003197/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003197.do\",\n    \"title\": \"C# & VB.NET Conversion Pocket Reference\"\n  },\n  {\n    \"animal\": \"Black-winged Kite\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920042501/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920042501.do\",\n    \"title\": \"Cloud Foundry: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Blackbuck Antelope\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596522315/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596522315.do\",\n    \"title\": \"Even Faster Web Sites\"\n  },\n  {\n    \"animal\": \"Blesbok (African antelope)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001780/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001780.do\",\n    \"title\": \"Perl & LWP\"\n  },\n  {\n    \"animal\": \"Blood Pheasant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004217/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004217.do\",\n    \"title\": \"RELAX NG\"\n  },\n  {\n    \"animal\": \"Bloodhounds, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001612/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001612.do\",\n    \"title\": \"Learning Carbon\"\n  },\n  {\n    \"animal\": \"Blowfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514471/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514471.do\",\n    \"title\": \"CJKV Information Processing\"\n  },\n  {\n    \"animal\": \"Blue Jay in flight\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020189/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020189.do\",\n    \"title\": \"Planning for IPv6\"\n  },\n  {\n    \"animal\": \"Blue Rock Thrush (blue)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920051923/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920051923.do\",\n    \"title\": \"Designing with Sound\"\n  },\n  {\n    \"animal\": \"Blue Swimming Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027966/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027966.do\",\n    \"title\": \"XML and InDesign\"\n  },\n  {\n    \"animal\": \"Blue Tilapia (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920063698/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920063698.do\",\n    \"title\": \"Learning TensorFlow\"\n  },\n  {\n    \"animal\": \"Blue Wildebeest, aka Brindled Gnu\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021155/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021155.do\",\n    \"title\": \"Geolocation in iOS\"\n  },\n  {\n    \"animal\": \"Blue and Gold Macaws\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008680/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008680.do\",\n    \"title\": \"Switching to VoIP\"\n  },\n  {\n    \"animal\": \"Blue carpenter Bee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037057/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037057.do\",\n    \"title\": \"Data Visualization with Python and JavaScript\"\n  },\n  {\n    \"animal\": \"Blue-Lipped Tree Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032861/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032861.do\",\n    \"title\": \"Data Wrangling with Python\"\n  },\n  {\n    \"animal\": \"Blue-faced Leceister Sheep\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920187714/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920187714.do\",\n    \"title\": \"Practical Time Series Analysis\"\n  },\n  {\n    \"animal\": \"Blueback Char Trout\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925724/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925724.do\",\n    \"title\": \"Internet Core Protocols: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Bluebird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596805791/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596805791.do\",\n    \"title\": \"Building iPhone Apps with HTML, CSS, and JavaScript\"\n  },\n  {\n    \"animal\": \"Bluefin Tuna\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028932/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028932.do\",\n    \"title\": \"Active Directory Cookbook\"\n  },\n  {\n    \"animal\": \"Bluejay\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003593/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003593.do\",\n    \"title\": \"Writing Excel Macros with VBA\"\n  },\n  {\n    \"animal\": \"Boa Constrictor snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922266/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922266.do\",\n    \"title\": \"Software Portability with imake\"\n  },\n  {\n    \"animal\": \"Boarhound Dog aka Great Dane \",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920010159/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920010159.do\",\n    \"title\": \"Network Warrior\"\n  },\n  {\n    \"animal\": \"Boatbill Heron\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002503/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002503.do\",\n    \"title\": \"Programming .NET Web Services\"\n  },\n  {\n    \"animal\": \"Bobac, aka Steppe Marmot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527037/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527037.do\",\n    \"title\": \"Essential CVS\"\n  },\n  {\n    \"animal\": \"Bobtail Squid\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033424/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033424.do\",\n    \"title\": \"Effective Computation in Physics\"\n  },\n  {\n    \"animal\": \"Bohemian Waxwing 2, looking over shoulder\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596523053/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596523053.do\",\n    \"title\": \"JUNOS High Availability\"\n  },\n  {\n    \"animal\": \"Booted Racket Tail Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804381/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804381.do\",\n    \"title\": \"PHP: The Good Parts\"\n  },\n  {\n    \"animal\": \"Boston Terrier\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928411/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928411.do\",\n    \"title\": \"AppleScript in a Nutshell\"\n  },\n  {\n    \"animal\": \"Boston Terrier\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102111/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102111.do\",\n    \"title\": \"AppleScript: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Botta's Pocket Gopher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920046516/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920046516.do\",\n    \"title\": \"Introducing Go\"\n  },\n  {\n    \"animal\": \"Bowhead Whale\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035671/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035671.do\",\n    \"title\": \"Using Docker\"\n  },\n  {\n    \"animal\": \"Box Turtle  (NOT a tortoise)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024132/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024132.do\",\n    \"title\": \"Windows PowerShell Cookbook\"\n  },\n  {\n    \"animal\": \"Box Turtle  (NOT a tortoise)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024248/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024248.do\",\n    \"title\": \"Windows PowerShell Pocket Reference\"\n  },\n  {\n    \"animal\": \"Brazilian Guinea Pigs, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028215/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028215.do\",\n    \"title\": \"Coding with Coda\"\n  },\n  {\n    \"animal\": \"Brazilian Striped Hog-nosed Skunk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920054290/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920054290.do\",\n    \"title\": \"Ransomware\"\n  },\n  {\n    \"animal\": \"Bridled Common Murre, aka Guillemot (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028802/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028802.do\",\n    \"title\": \"Distributed Network Data\"\n  },\n  {\n    \"animal\": \"Brittle Starfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596520656/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596520656.do\",\n    \"title\": \"Getting Started with Flex 3\"\n  },\n  {\n    \"animal\": \"Bronzewing Pigeons, Pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024446/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024446.do\",\n    \"title\": \"Web Workers\"\n  },\n  {\n    \"animal\": \"Brown Long-Eared Bat, aka Geoffroys Nyctophile\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025948/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025948.do\",\n    \"title\": \"Web Audio API\"\n  },\n  {\n    \"animal\": \"Brown Noddy (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035053/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035053.do\",\n    \"title\": \"High Performance Android Apps\"\n  },\n  {\n    \"animal\": \"Brown Trout\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920073994/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920073994.do\",\n    \"title\": \"Streaming Systems\"\n  },\n  {\n    \"animal\": \"Brown-Eared Pheasant Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019824/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019824.do\",\n    \"title\": \"HTML5 Media\"\n  },\n  {\n    \"animal\": \"Brush-tailed Rat Kangaroo, with baby in its pouch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518189/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518189.do\",\n    \"title\": \"Erlang Programming\"\n  },\n  {\n    \"animal\": \"Brush-tailed Rat Kangaroo, with baby in its pouch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020240/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020240.do\",\n    \"title\": \"Erlang by Example with Cesarini and Thompson\"\n  },\n  {\n    \"animal\": \"Brushtail Possum, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001186/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001186.do\",\n    \"title\": \"Subclassing and Hooking with Visual Basic\"\n  },\n  {\n    \"animal\": \"Bucking Bronco Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005900/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005900.do\",\n    \"title\": \"Linux Device Drivers\"\n  },\n  {\n    \"animal\": \"Bufflehead Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518448/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518448.do\",\n    \"title\": \"Programming Visual Basic 2008\"\n  },\n  {\n    \"animal\": \"Bull with horns, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804886/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804886.do\",\n    \"title\": \"LPI Linux Certification in a Nutshell\"\n  },\n  {\n    \"animal\": \"Bulldog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510039/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510039.do\",\n    \"title\": \"ScreenOS Cookbook\"\n  },\n  {\n    \"animal\": \"Bullfinch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024033/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024033.do\",\n    \"title\": \"Programming F# 3.0\"\n  },\n  {\n    \"animal\": \"Bullhead Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022923/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022923.do\",\n    \"title\": \"Programming Computer Vision with Python\"\n  },\n  {\n    \"animal\": \"Bumblebee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024811/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024811.do\",\n    \"title\": \"Berglund and McCullough on Mastering Cassandra for Architects\"\n  },\n  {\n    \"animal\": \"Bumblebee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925984/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925984.do\",\n    \"title\": \"Oracle 8i Internal Services\"\n  },\n  {\n    \"animal\": \"Bumblebee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030553/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030553.do\",\n    \"title\": \"Relational Theory for Computer Professionals\"\n  },\n  {\n    \"animal\": \"Bumpy Toad\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100094/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100094.do\",\n    \"title\": \"Monad (AKA PowerShell)\"\n  },\n  {\n    \"animal\": \"Burbot (fish), aka Quappe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030348/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030348.do\",\n    \"title\": \"Using Flume\"\n  },\n  {\n    \"animal\": \"Burro, Mule\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025726/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025726.do\",\n    \"title\": \"Getting Started with Mule Cloud Connect\"\n  },\n  {\n    \"animal\": \"Burro, Mule\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025726/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025726.do\",\n    \"title\": \"Getting Started with Mule Cloud Connect\"\n  },\n  {\n    \"animal\": \"Burrowing Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018094/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018094.do\",\n    \"title\": \"Mastering Search Analytics\"\n  },\n  {\n    \"animal\": \"Bush Antelope (aka Yellow back duiker)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025139/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025139.do\",\n    \"title\": \"ClojureScript: Up and Running\"\n  },\n  {\n    \"animal\": \"Bush Wren\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019909/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019909.do\",\n    \"title\": \"Building and Testing with Gradle\"\n  },\n  {\n    \"animal\": \"Bushbuck (Guib)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000127/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000127.do\",\n    \"title\": \"Managing IMAP\"\n  },\n  {\n    \"animal\": \"Bushmaster Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013228/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013228.do\",\n    \"title\": \"Programming Amazon EC2\"\n  },\n  {\n    \"animal\": \"Butterfly Blenny Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024873/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024873.do\",\n    \"title\": \"Drupal for Designers\"\n  },\n  {\n    \"animal\": \"Butterfly, Charaxes Brutus (White-barred), in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596802288/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596802288.do\",\n    \"title\": \"Search Patterns\"\n  },\n  {\n    \"animal\": \"C Elegans Worm\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926646/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926646.do\",\n    \"title\": \"Developing Bioinformatics Computer Skills\"\n  },\n  {\n    \"animal\": \"Cabot's Tragopan (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920043072/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920043072.do\",\n    \"title\": \"SVG Text Layout\"\n  },\n  {\n    \"animal\": \"Caiman Crocodile\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926219/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926219.do\",\n    \"title\": \"Python Programming On Win32\"\n  },\n  {\n    \"animal\": \"Camel, aka Dromedary\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004927/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004927.do\",\n    \"title\": \"Programming Perl\"\n  },\n  {\n    \"animal\": \"Canadian Goose (3), flying wings down\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003975/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003975.do\",\n    \"title\": \".NET & XML\"\n  },\n  {\n    \"animal\": \"Canadian Lynx\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926592/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926592.do\",\n    \"title\": \"Delphi in a Nutshell\"\n  },\n  {\n    \"animal\": \"Cape Hunting Dog, aka African Wild Dog, Painted Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527310/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527310.do\",\n    \"title\": \"Rails Cookbook\"\n  },\n  {\n    \"animal\": \"Cape Lion\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021599/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021599.do\",\n    \"title\": \"Mac OS X Lion Pocket Guide\"\n  },\n  {\n    \"animal\": \"Cape Petrel bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020004/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020004.do\",\n    \"title\": \"HTML5 Geolocation\"\n  },\n  {\n    \"animal\": \"Capuchin Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155803/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155803.do\",\n    \"title\": \"Google Advertising Tools\"\n  },\n  {\n    \"animal\": \"Capybara\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021469/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021469.do\",\n    \"title\": \"C# Database Basics\"\n  },\n  {\n    \"animal\": \"Card Players\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020103/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020103.do\",\n    \"title\": \"Privacy and Big Data\"\n  },\n  {\n    \"animal\": \"Cardinal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920000723/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920000723.do\",\n    \"title\": \"MacRuby: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Caribou, Reindeer with full antlers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001506/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001506.do\",\n    \"title\": \"Designing Large Scale Lans\"\n  },\n  {\n    \"animal\": \"Caribou, aka Reindeer with full antlers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001278/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001278.do\",\n    \"title\": \"T1: A Survival Guide\"\n  },\n  {\n    \"animal\": \"Carolina Parakeets, pair perched on a branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005023/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005023.do\",\n    \"title\": \"Building Wireless Community Networks\"\n  },\n  {\n    \"animal\": \"Carp\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102357/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102357.do\",\n    \"title\": \"Building Scalable Web Sites\"\n  },\n  {\n    \"animal\": \"Carpet Chameleon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027089/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027089.do\",\n    \"title\": \"Designing Multi-Device Experiences\"\n  },\n  {\n    \"animal\": \"Caspian Tern, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024002/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024002.do\",\n    \"title\": \"Mobile Development with C#\"\n  },\n  {\n    \"animal\": \"Cassowary (flightless bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025320/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025320.do\",\n    \"title\": \"Android Developer Tools Essentials\"\n  },\n  {\n    \"animal\": \"Cat Leaves 2&3\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528102/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528102.do\",\n    \"title\": \"Designing Web Navigation\"\n  },\n  {\n    \"animal\": \"Catbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920235583/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920235583.do\",\n    \"title\": \"Strategic Writing for UX\"\n  },\n  {\n    \"animal\": \"Catfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004286/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004286.do\",\n    \"title\": \"VB.NET Language Pocket Reference\"\n  },\n  {\n    \"animal\": \"Catfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101527/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101527.do\",\n    \"title\": \"Visual Basic 2005 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Catholic priests\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021872/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021872.do\",\n    \"title\": \"Ethics of Big Data\"\n  },\n  {\n    \"animal\": \"Central American Turkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596157630/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596157630.do\",\n    \"title\": \"Windows 7 Annoyances\"\n  },\n  {\n    \"animal\": \"Chacma Baboon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006334/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006334.do\",\n    \"title\": \"Windows Server Cookbook\"\n  },\n  {\n    \"animal\": \"Chameleon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518851/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518851.do\",\n    \"title\": \"SQL in a Nutshell\"\n  },\n  {\n    \"animal\": \"Chamois\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004859/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004859.do\",\n    \"title\": \"Extreme Programming Pocket Guide\"\n  },\n  {\n    \"animal\": \"Channel-billed Cuckoo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025405/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025405.do\",\n    \"title\": \"Just Spring Data Access\"\n  },\n  {\n    \"animal\": \"Channel-billed Toucan in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518394/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518394.do\",\n    \"title\": \"Designing Gestural Interfaces\"\n  },\n  {\n    \"animal\": \"Chanting Falcon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021797/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021797.do\",\n    \"title\": \"Google Power Search\"\n  },\n  {\n    \"animal\": \"Checkered Wrasse (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035350/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035350.do\",\n    \"title\": \"ASP.NET MVC 5 with Bootstrap and Knockout.js\"\n  },\n  {\n    \"animal\": \"Cheetah\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926998/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926998.do\",\n    \"title\": \"Programming the Perl DBI\"\n  },\n  {\n    \"animal\": \"Chick and egg, hatching, baby chicken\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004200/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004200.do\",\n    \"title\": \"Learning XML\"\n  },\n  {\n    \"animal\": \"Chimpanzee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920039006/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920039006.do\",\n    \"title\": \"Big Data for Chimps\"\n  },\n  {\n    \"animal\": \"Chimpanzee, Head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007959/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007959.do\",\n    \"title\": \"UML 2.0 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Chinese Pangolin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920206767/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920206767.do\",\n    \"title\": \"Creating Augmented and Virtual Realities\"\n  },\n  {\n    \"animal\": \"Chough Thrush aka Pander's Ground Jay\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033622/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033622.do\",\n    \"title\": \"IPv6 Address Planning\"\n  },\n  {\n    \"animal\": \"Chough bird (crow)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018407/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018407.do\",\n    \"title\": \"Scaling CouchDB\"\n  },\n  {\n    \"animal\": \"Cicada 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027737/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027737.do\",\n    \"title\": \"Oracle Essentials\"\n  },\n  {\n    \"animal\": \"Climbing Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004255/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004255.do\",\n    \"title\": \"Windows XP Pocket Reference\"\n  },\n  {\n    \"animal\": \"Climbing Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009007/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009007.do\",\n    \"title\": \"Windows XP in a Nutshell\"\n  },\n  {\n    \"animal\": \"Clown Butterfly Fish (aka Ornate Butterflyfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006419/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006419.do\",\n    \"title\": \"Eclipse\"\n  },\n  {\n    \"animal\": \"Clown Butterfly Fish (aka Ornate Butterflyfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100650/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100650.do\",\n    \"title\": \"Eclipse IDE Pocket Guide\"\n  },\n  {\n    \"animal\": \"Clown Butterfly Fish (aka Ornate Butterflyfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897215528/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897215528.do\",\n    \"title\": \"Eclipse IDE kurz & gut \"\n  },\n  {\n    \"animal\": \"Clydesdale Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920014348/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920014348.do\",\n    \"title\": \"HBase: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Coatimundi\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565921160/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565921160.do\",\n    \"title\": \"C++ The Core Language\"\n  },\n  {\n    \"animal\": \"Coatimundi\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003647/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003647.do\",\n    \"title\": \"Planning for Web Services: Obstacles and Opportunities\"\n  },\n  {\n    \"animal\": \"Cobra snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002398/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002398.do\",\n    \"title\": \"Dreamweaver in a Nutshell\"\n  },\n  {\n    \"animal\": \"Cochin Chickens, aka Cochin China Fowl, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020196/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020196.do\",\n    \"title\": \"Practical JIRA Administration\"\n  },\n  {\n    \"animal\": \"Cockatiel (Nymphilus Navae, Hollandae)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020578/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020578.do\",\n    \"title\": \"Natural Language Annotation for Machine Learning\"\n  },\n  {\n    \"animal\": \"Cockatiel (or Parakeet), with crowning feathers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006846/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006846.do\",\n    \"title\": \"Word Pocket Guide\"\n  },\n  {\n    \"animal\": \"Coelacanth fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002992/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002992.do\",\n    \"title\": \"BLAST\"\n  },\n  {\n    \"animal\": \"Collie dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007614/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007614.do\",\n    \"title\": \"C++ Cookbook\"\n  },\n  {\n    \"animal\": \"Comber Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920233626/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920233626.do\",\n    \"title\": \"Learning CoreDNS\"\n  },\n  {\n    \"animal\": \"Common Agouti, Aguti\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026457/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026457.do\",\n    \"title\": \"Getting Started with Arduino and iOS\"\n  },\n  {\n    \"animal\": \"Common Agouti, Aguti\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021179/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021179.do\",\n    \"title\": \"iOS Sensor Apps with Arduino\"\n  },\n  {\n    \"animal\": \"Common Bittern, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022220/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022220.do\",\n    \"title\": \"Programming Entity Framework: Code First\"\n  },\n  {\n    \"animal\": \"Common Creeper (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028192/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028192.do\",\n    \"title\": \"Computer Science Programming Basics in Ruby\"\n  },\n  {\n    \"animal\": \"Common Death Adder\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920179603/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920179603.do\",\n    \"title\": \"Cybersecurity Ops with bash\"\n  },\n  {\n    \"animal\": \"Common Flying Dragon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920225010/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920225010.do\",\n    \"title\": \"Terraform: Up & Running\"\n  },\n  {\n    \"animal\": \"Common Garter Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007256/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007256.do\",\n    \"title\": \"Windows XP Cookbook\"\n  },\n  {\n    \"animal\": \"Common Goat (male)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025283/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025283.do\",\n    \"title\": \"Accessible EPUB 3\"\n  },\n  {\n    \"animal\": \"Common Goat (male)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024897/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024897.do\",\n    \"title\": \"EPUB 3 Best Practices\"\n  },\n  {\n    \"animal\": \"Common Hare\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026235/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026235.do\",\n    \"title\": \"Programming for PaaS\"\n  },\n  {\n    \"animal\": \"Common Hill Myna Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596806163/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596806163.do\",\n    \"title\": \"Building the Realtime User Experience\"\n  },\n  {\n    \"animal\": \"Common Loon aka Great Northern Diver (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920097471/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920097471.do\",\n    \"title\": \"Deep Learning Cookbook\"\n  },\n  {\n    \"animal\": \"Common Nighthawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596515089/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596515089.do\",\n    \"title\": \"Website Optimization\"\n  },\n  {\n    \"animal\": \"Common Palm Civet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920266624/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920266624.do\",\n    \"title\": \"Concurrency in C# Cookbook\"\n  },\n  {\n    \"animal\": \"Common Pheasant, female hen\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024484/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024484.do\",\n    \"title\": \"Civic Apps Competition Handbook\"\n  },\n  {\n    \"animal\": \"Common Rat, aka Brown Rat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596154516/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596154516.do\",\n    \"title\": \"Bioinformatics Programming Using Python\"\n  },\n  {\n    \"animal\": \"Common Raven\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155148/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155148.do\",\n    \"title\": \"Complete Web Monitoring\"\n  },\n  {\n    \"animal\": \"Common Small-Spotted Genet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008796/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008796.do\",\n    \"title\": \"Excel Scientific and Engineering Cookbook\"\n  },\n  {\n    \"animal\": \"Common Snipe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008659/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008659.do\",\n    \"title\": \"Web Mapping Illustrated\"\n  },\n  {\n    \"animal\": \"Common Starling\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024217/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024217.do\",\n    \"title\": \"Introducing Starling\"\n  },\n  {\n    \"animal\": \"Comoro Cuckoo Roller (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033202/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033202.do\",\n    \"title\": \"Programming Google App Engine with Java\"\n  },\n  {\n    \"animal\": \"Condor, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020417/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020417.do\",\n    \"title\": \"Virtualization: A Manager's Guide\"\n  },\n  {\n    \"animal\": \"Conger Eel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001032/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001032.do\",\n    \"title\": \"COM & .NET Component Services\"\n  },\n  {\n    \"animal\": \"Conger Eel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033240/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033240.do\",\n    \"title\": \"Salt Essentials\"\n  },\n  {\n    \"animal\": \"Coot birds with baby chicks\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020011/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020011.do\",\n    \"title\": \"Making Isometric Social Real-Time Games with HTML5, CSS3, and JavaScript\"\n  },\n  {\n    \"animal\": \"Copper Sweeper Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022510/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022510.do\",\n    \"title\": \"What's New in CSS3\"\n  },\n  {\n    \"animal\": \"Coral Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006525/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006525.do\",\n    \"title\": \"Essential ActionScript 2.0\"\n  },\n  {\n    \"animal\": \"Coral Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526948/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526948.do\",\n    \"title\": \"Essential ActionScript 3.0\"\n  },\n  {\n    \"animal\": \"Coral bed with Sea Anemones\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100797/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100797.do\",\n    \"title\": \"Linux Kernel in a Nutshell\"\n  },\n  {\n    \"animal\": \"Corn Bunting (large)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000059/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000059.do\",\n    \"title\": \"Palm OS Network Programming\"\n  },\n  {\n    \"animal\": \"Cornetfish or Tobacco-pipe Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024088/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024088.do\",\n    \"title\": \"HTML5 and JavaScript Web Apps\"\n  },\n  {\n    \"animal\": \"Corsac Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004231/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004231.do\",\n    \"title\": \"Objective-C Pocket Reference\"\n  },\n  {\n    \"animal\": \"Cotton-top Tamarin Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101879/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101879.do\",\n    \"title\": \"Ajax on Java\"\n  },\n  {\n    \"animal\": \"Cowboy Domino Players\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002558/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002558.do\",\n    \"title\": \"Understanding Linux Network Internals\"\n  },\n  {\n    \"animal\": \"Cowboy Man on Rearing Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007607/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007607.do\",\n    \"title\": \"Running Linux\"\n  },\n  {\n    \"animal\": \"Cowboy and Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005481/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005481.do\",\n    \"title\": \"Linux Network Administrator's Guide\"\n  },\n  {\n    \"animal\": \"Cowboy wranglers, Ropers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005832/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005832.do\",\n    \"title\": \"Linux Unwired\"\n  },\n  {\n    \"animal\": \"Cowboys and Natives, Playing Cards\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007584/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007584.do\",\n    \"title\": \"Linux in a Windows World\"\n  },\n  {\n    \"animal\": \"Cowboys and horses\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101831/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101831.do\",\n    \"title\": \"SUSE Linux\"\n  },\n  {\n    \"animal\": \"Cowboys at a campfire\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003913/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003913.do\",\n    \"title\": \"Linux Security Cookbook\"\n  },\n  {\n    \"animal\": \"Cowboys group shootout\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005818/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005818.do\",\n    \"title\": \"Understanding Open Source and Free Software Licensing\"\n  },\n  {\n    \"animal\": \"Cowboys herding cattle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009526/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009526.do\",\n    \"title\": \"Linux System Administration\"\n  },\n  {\n    \"animal\": \"Cowboys herding cattle, steers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005702/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005702.do\",\n    \"title\": \"High Performance Linux Clusters with OSCAR, Rocks, OpenMosix, and MPI\"\n  },\n  {\n    \"animal\": \"Cowboys in a doorway\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005696/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005696.do\",\n    \"title\": \"Linux iptables Pocket Reference\"\n  },\n  {\n    \"animal\": \"Cowboys with Horses and Wagon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006709/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006709.do\",\n    \"title\": \"Linux Server Security\"\n  },\n  {\n    \"animal\": \"Cowboys with Pack Mule\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101046/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101046.do\",\n    \"title\": \"Linux Desktop Pocket Guide\"\n  },\n  {\n    \"animal\": \"Cowboys, horse, cow, roping cattle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526825/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526825.do\",\n    \"title\": \"Fedora Linux\"\n  },\n  {\n    \"animal\": \"Cowry Snail\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050308/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050308.do\",\n    \"title\": \"Microservice Architecture\"\n  },\n  {\n    \"animal\": \"Coyote\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002442/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002442.do\",\n    \"title\": \"BEEP:  The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Cozumel Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920123880/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920123880.do\",\n    \"title\": \"Designing Web APIs\"\n  },\n  {\n    \"animal\": \"Crab Spider\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003579/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003579.do\",\n    \"title\": \"Webmaster in a Nutshell\"\n  },\n  {\n    \"animal\": \"Crab-eating Mongoose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101220/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101220.do\",\n    \"title\": \"Access Data Analysis Cookbook\"\n  },\n  {\n    \"animal\": \"Crab-eating Opossum\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526955/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526955.do\",\n    \"title\": \"ActionScript 3.0 Cookbook\"\n  },\n  {\n    \"animal\": \"Crawfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002572/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002572.do\",\n    \"title\": \"VB.NET Core Classes in a  Nutshell\"\n  },\n  {\n    \"animal\": \"Crayfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920053170/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920053170.do\",\n    \"title\": \"Practical Machine Learning with H2O\"\n  },\n  {\n    \"animal\": \"Cream-Colored Courser (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021049/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021049.do\",\n    \"title\": \"What Is HTML5?\"\n  },\n  {\n    \"animal\": \"Crested Agouti\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032625/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032625.do\",\n    \"title\": \"OpenStack Operations Guide\"\n  },\n  {\n    \"animal\": \"Crested Caracara Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926493/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926493.do\",\n    \"title\": \"Windows System Policy Editor\"\n  },\n  {\n    \"animal\": \"Crested Eagle (Morphnus guianensis)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927131/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927131.do\",\n    \"title\": \"Windows 2000 Administration in a Nutshell\"\n  },\n  {\n    \"animal\": \"Crested Grebe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009496/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009496.do\",\n    \"title\": \"Programming Visual Basic 2005\"\n  },\n  {\n    \"animal\": \"Crested Ibis Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021247/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021247.do\",\n    \"title\": \"Designing for XOOPS\"\n  },\n  {\n    \"animal\": \"Crested Pardalotte bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022053/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022053.do\",\n    \"title\": \"Developing with Google+\"\n  },\n  {\n    \"animal\": \"Crested Pelican\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034063/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034063.do\",\n    \"title\": \"Responsive Typography\"\n  },\n  {\n    \"animal\": \"Crested Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029816/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029816.do\",\n    \"title\": \"Designing Mobile Payment Experiences\"\n  },\n  {\n    \"animal\": \"Crested Porcupine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003944/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003944.do\",\n    \"title\": \"Secure Programming Cookbook for C and C++\"\n  },\n  {\n    \"animal\": \"Crested Screamer (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033776/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033776.do\",\n    \"title\": \"Programming Chrome Apps\"\n  },\n  {\n    \"animal\": \"Crested Shriketit, Falconelle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021575/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021575.do\",\n    \"title\": \"REST API Design Rulebook\"\n  },\n  {\n    \"animal\": \"Crested Whistling Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920242598/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920242598.do\",\n    \"title\": \"Kubernetes Patterns\"\n  },\n  {\n    \"animal\": \"Cuckoo Pheasant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025832/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025832.do\",\n    \"title\": \"Learning JavaScript Design Patterns\"\n  },\n  {\n    \"animal\": \"Cuckoo Roller, aka Courol (Leptosomus discolor), pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022381/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022381.do\",\n    \"title\": \"MongoDB and PHP\"\n  },\n  {\n    \"animal\": \"Cuckoo on branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897215207/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897215207.do\",\n    \"title\": \"PHP kurz & gut\"\n  },\n  {\n    \"animal\": \"Cuttlefish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030140/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030140.do\",\n    \"title\": \"Ethernet Switches\"\n  },\n  {\n    \"animal\": \"Cuttlefish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004019/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004019.do\",\n    \"title\": \"Flash Remoting: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Dachshunds, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596807740/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596807740.do\",\n    \"title\": \"Building Wireless Sensor Networks\"\n  },\n  {\n    \"animal\": \"Darter, Australian (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003388/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003388.do\",\n    \"title\": \".NET Windows Forms in a Nutshell\"\n  },\n  {\n    \"animal\": \"Darter, Australian (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003210/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003210.do\",\n    \"title\": \"Programming .NET Windows Applications\"\n  },\n  {\n    \"animal\": \"Daurian Pika (Ochotona dauurica)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026860/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026860.do\",\n    \"title\": \"Puppet Types and Providers\"\n  },\n  {\n    \"animal\": \"Demoiselle Pomacentrus Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101626/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101626.do\",\n    \"title\": \"Learning WCF\"\n  },\n  {\n    \"animal\": \"Desmarest's Hutia (rodent)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034469/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034469.do\",\n    \"title\": \"RESTful Rails Development\"\n  },\n  {\n    \"animal\": \"Dhole Dog, aka Asiatic wild dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596801984/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596801984.do\",\n    \"title\": \"Programming Windows Azure\"\n  },\n  {\n    \"animal\": \"Diana Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920014201/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920014201.do\",\n    \"title\": \"Programming Social Applications\"\n  },\n  {\n    \"animal\": \"Doctor Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024699/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024699.do\",\n    \"title\": \"Testable JavaScript\"\n  },\n  {\n    \"animal\": \"Dolium Shell, aka Conch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003227/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003227.do\",\n    \"title\": \"RADIUS\"\n  },\n  {\n    \"animal\": \"Domestic Cat and Kitten (pair)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023913/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023913.do\",\n    \"title\": \"Active Directory\"\n  },\n  {\n    \"animal\": \"Domestic Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006426/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006426.do\",\n    \"title\": \"Real World Web Services\"\n  },\n  {\n    \"animal\": \"Donkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008697/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008697.do\",\n    \"title\": \"Cisco IOS in a Nutshell\"\n  },\n  {\n    \"animal\": \"Donkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923201/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923201.do\",\n    \"title\": \"Managing IP Networks with Cisco Routers\"\n  },\n  {\n    \"animal\": \"Dorcas Gazelle, aka Ariel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024156/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024156.do\",\n    \"title\": \"Windows Server 2012: Up and Running\"\n  },\n  {\n    \"animal\": \"Dormouse with nut\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920010890/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920010890.do\",\n    \"title\": \"Using Drupal\"\n  },\n  {\n    \"animal\": \"Double-Crested Cormorant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001469/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001469.do\",\n    \"title\": \"Object-Oriented Programming with Visual Basic .NET\"\n  },\n  {\n    \"animal\": \"Dove, in flight\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002121/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002121.do\",\n    \"title\": \"Postfix: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Dragonfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922686/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922686.do\",\n    \"title\": \"Oracle Design: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Dragonfly OR Myrmeleon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003364/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003364.do\",\n    \"title\": \"Oracle in a Nutshell\"\n  },\n  {\n    \"animal\": \"Duckbill Platypus\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005436/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005436.do\",\n    \"title\": \"Web Database Applications with PHP and MySQL\"\n  },\n  {\n    \"animal\": \"Dung Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002336/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002336.do\",\n    \"title\": \"Oracle RMAN Pocket Reference\"\n  },\n  {\n    \"animal\": \"Dwarf Mongoose, Kusimanse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021513/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021513.do\",\n    \"title\": \"MongoDB and Python\"\n  },\n  {\n    \"animal\": \"Eared Seal, aka Sea Lion\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922471/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922471.do\",\n    \"title\": \"Learning VBScript\"\n  },\n  {\n    \"animal\": \"Eastern Chipmunks, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005566/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005566.do\",\n    \"title\": \"STL Pocket Reference\"\n  },\n  {\n    \"animal\": \"Eastern Gray Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023982/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023982.do\",\n    \"title\": \"JavaScript for PHP Developers\"\n  },\n  {\n    \"animal\": \"Eastern Honey Bee Group\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920056072/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920056072.do\",\n    \"title\": \"Mastering Ethereum\"\n  },\n  {\n    \"animal\": \"Eastern Kingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024194/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024194.do\",\n    \"title\": \"Developing Business Intelligence Apps for SharePoint\"\n  },\n  {\n    \"animal\": \"Egyptian Goose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003609/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003609.do\",\n    \"title\": \"Mastering Visual Studio .NET\"\n  },\n  {\n    \"animal\": \"Egyptian Yellow-billed Kite (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920045106/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920045106.do\",\n    \"title\": \"Agile Application Security\"\n  },\n  {\n    \"animal\": \"Eider Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028857/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028857.do\",\n    \"title\": \"Functional JavaScript\"\n  },\n  {\n    \"animal\": \"Electric Catfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596519216/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596519216.do\",\n    \"title\": \"RESTful .NET\"\n  },\n  {\n    \"animal\": \"Elegant Hyla Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596153618/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596153618.do\",\n    \"title\": \"grep Pocket Reference\"\n  },\n  {\n    \"animal\": \"Eleonora's Falcon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596157081/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596157081.do\",\n    \"title\": \"SEO Warrior\"\n  },\n  {\n    \"animal\": \"Elephant Seal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022466/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022466.do\",\n    \"title\": \"Big Data Glossary\"\n  },\n  {\n    \"animal\": \"Elephant Shrew, crouching\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001285/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001285.do\",\n    \"title\": \"Python & XML\"\n  },\n  {\n    \"animal\": \"Elephant-nosed fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920049111/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920049111.do\",\n    \"title\": \"Electronics Cookbook\"\n  },\n  {\n    \"animal\": \"Emu Wren\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025382/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025382.do\",\n    \"title\": \"Macintosh Terminal Pocket Guide\"\n  },\n  {\n    \"animal\": \"Emu, large and fluffy\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927162/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927162.do\",\n    \"title\": \"Mastering Perl/Tk\"\n  },\n  {\n    \"animal\": \"Emu, young\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925175/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925175.do\",\n    \"title\": \"Perl/Tk Pocket Reference\"\n  },\n  {\n    \"animal\": \"English Leceister Sheep\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032472/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032472.do\",\n    \"title\": \"Learning MCollective\"\n  },\n  {\n    \"animal\": \"English Setter Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020875/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020875.do\",\n    \"title\": \"Managing Infrastructure with Puppet\"\n  },\n  {\n    \"animal\": \"Ermine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596159788/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596159788.do\",\n    \"title\": \"jQuery Cookbook\"\n  },\n  {\n    \"animal\": \"Ermine 1 (part of a Pair)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001438/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001438.do\",\n    \"title\": \"Java and XSLT\"\n  },\n  {\n    \"animal\": \"Eskimo Dogs, aka Huskies\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005016/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005016.do\",\n    \"title\": \"Inside .Mac\"\n  },\n  {\n    \"animal\": \"Euploea Midamus Butterfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008642/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008642.do\",\n    \"title\": \"Learning MySQL\"\n  },\n  {\n    \"animal\": \"Eurasian Eagle-Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027713/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027713.do\",\n    \"title\": \"JavaScript Enlightenment\"\n  },\n  {\n    \"animal\": \"Eurasian Eagle-owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032298/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032298.do\",\n    \"title\": \"Thoughtful Machine Learning\"\n  },\n  {\n    \"animal\": \"Eurasian Magpie\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920047568/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920047568.do\",\n    \"title\": \"Stream Processing with Apache Spark\"\n  },\n  {\n    \"animal\": \"European Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004712/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004712.do\",\n    \"title\": \"sendmail Cookbook\"\n  },\n  {\n    \"animal\": \"European Bear\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000400/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000400.do\",\n    \"title\": \"Java Servlet Programming\"\n  },\n  {\n    \"animal\": \"European Common Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528089/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528089.do\",\n    \"title\": \"Windows Vista Pocket Reference\"\n  },\n  {\n    \"animal\": \"European Common Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527075/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527075.do\",\n    \"title\": \"Windows Vista in a Nutshell\"\n  },\n  {\n    \"animal\": \"European Common Starling\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920107361/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920107361.do\",\n    \"title\": \"Unity Game Development Cookbook\"\n  },\n  {\n    \"animal\": \"European Garden Spider\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920233145/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920233145.do\",\n    \"title\": \"Graph Algorithms\"\n  },\n  {\n    \"animal\": \"European Golden Plover\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920001416/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920001416.do\",\n    \"title\": \"Closure: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"European Hamster\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030157/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030157.do\",\n    \"title\": \"Bioinformatics Data Skills\"\n  },\n  {\n    \"animal\": \"European Ibex, aka Alpine Ibex (goat)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002695/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002695.do\",\n    \"title\": \"Java Web Services\"\n  },\n  {\n    \"animal\": \"European Ibex, aka Alpine Ibex (goat)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003999/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003999.do\",\n    \"title\": \"Java Web Services in a Nutshell\"\n  },\n  {\n    \"animal\": \"European Night Heron\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920001317/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920001317.do\",\n    \"title\": \"Junos Security\"\n  },\n  {\n    \"animal\": \"European Partridge\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596806767/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596806767.do\",\n    \"title\": \"JavaScript Patterns\"\n  },\n  {\n    \"animal\": \"European Polecat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034131/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034131.do\",\n    \"title\": \"Learning Puppet 4\"\n  },\n  {\n    \"animal\": \"European Roller\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028086/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028086.do\",\n    \"title\": \"Building Modular Cloud Apps with OSGi\"\n  },\n  {\n    \"animal\": \"European Tree Frog - climbing\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002800/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002800.do\",\n    \"title\": \"NetBeans: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"European Wildcat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028901/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028901.do\",\n    \"title\": \"ZooKeeper\"\n  },\n  {\n    \"animal\": \"European common toad\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924178/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924178.do\",\n    \"title\": \"Windows 98 Annoyances\"\n  },\n  {\n    \"animal\": \"Fan-footed Gecko, aka Wall Gecko\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923249/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923249.do\",\n    \"title\": \"Learning Perl on Win32 Systems\"\n  },\n  {\n    \"animal\": \"Fan-tailed Raven (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920043614/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920043614.do\",\n    \"title\": \"Intelligence-Driven Incident Response\"\n  },\n  {\n    \"animal\": \"Favonia Octonema, radiata\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026464/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026464.do\",\n    \"title\": \"Java EE 6 Pocket Guide\"\n  },\n  {\n    \"animal\": \"Female Garden Spider 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006013/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006013.do\",\n    \"title\": \"Oracle Regular Expressions Pocket Reference\"\n  },\n  {\n    \"animal\": \"Female Garden Spider 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000189/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000189.do\",\n    \"title\": \"Oracle and Open Source\"\n  },\n  {\n    \"animal\": \"Fennec fox (Desert Fox)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005726/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005726.do\",\n    \"title\": \"Java Servlet & JSP Cookbook\"\n  },\n  {\n    \"animal\": \"Ferret\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527853/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527853.do\",\n    \"title\": \"Ferret\"\n  },\n  {\n    \"animal\": \"Finhorse, Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027867/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027867.do\",\n    \"title\": \"Bootstrap\"\n  },\n  {\n    \"animal\": \"Fish, Seawolf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022978/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022978.do\",\n    \"title\": \"Juniper Networks Warrior\"\n  },\n  {\n    \"animal\": \"Flamecrest (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920063445/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920063445.do\",\n    \"title\": \"Natural Language Processing with PyTorch\"\n  },\n  {\n    \"animal\": \"Flamingo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527402/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527402.do\",\n    \"title\": \"Dynamic HTML: The Definitive Reference\"\n  },\n  {\n    \"animal\": \"Flat-Headed Cat, kitten\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003883/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003883.do\",\n    \"title\": \"Essential Blogging\"\n  },\n  {\n    \"animal\": \"Florida Cricket Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035831/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035831.do\",\n    \"title\": \"Learning AngularJS\"\n  },\n  {\n    \"animal\": \"Florida Panther, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924888/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924888.do\",\n    \"title\": \"Java Foundation Classes in a Nutshell\"\n  },\n  {\n    \"animal\": \"Fly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000660/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000660.do\",\n    \"title\": \"Unix for Oracle DBAs Pocket Reference\"\n  },\n  {\n    \"animal\": \"Flying Dragon (lizard)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003128/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003128.do\",\n    \"title\": \"Games, Diversions & Perl Culture\"\n  },\n  {\n    \"animal\": \"Flying Dragon (lizard)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002060/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002060.do\",\n    \"title\": \"Programming Web Services with Perl\"\n  },\n  {\n    \"animal\": \"Flying Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002008/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002008.do\",\n    \"title\": \"Programming Visual Basic for the Palm OS\"\n  },\n  {\n    \"animal\": \"Flying Fox Fruit Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510299/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510299.do\",\n    \"title\": \"sendmail\"\n  },\n  {\n    \"animal\": \"Flying Fox Fruit Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922785/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922785.do\",\n    \"title\": \"sendmail Desktop Reference\"\n  },\n  {\n    \"animal\": \"Flying Gurnard Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920000426/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920000426.do\",\n    \"title\": \"Getting Started with Google Wave\"\n  },\n  {\n    \"animal\": \"Flying Gurnard Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596806019/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596806019.do\",\n    \"title\": \"Google Wave: Up and Running\"\n  },\n  {\n    \"animal\": \"Forest Dormouse, aka Wood Dormouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030782/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030782.do\",\n    \"title\": \"Building Web Apps with Ember.js\"\n  },\n  {\n    \"animal\": \"Forked-mouse  Lemur\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529369/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529369.do\",\n    \"title\": \"Adding Ajax\"\n  },\n  {\n    \"animal\": \"Forktailed Nightjar (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032168/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032168.do\",\n    \"title\": \"Responsive Theming for Drupal\"\n  },\n  {\n    \"animal\": \"Fossa (type of cat)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030058/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030058.do\",\n    \"title\": \"Hands-On Sencha Touch 2\"\n  },\n  {\n    \"animal\": \"Four-horned Cottus fish (aka sculpin)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026136/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026136.do\",\n    \"title\": \"ZeroMQ\"\n  },\n  {\n    \"animal\": \"Four-lined Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033219/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033219.do\",\n    \"title\": \"Programming Google App Engine with Python\"\n  },\n  {\n    \"animal\": \"Fox Terrier Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020264/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020264.do\",\n    \"title\": \"Planning and Managing Drupal Projects\"\n  },\n  {\n    \"animal\": \"French Grunt fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920141105/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920141105.do\",\n    \"title\": \"Visualizing Streaming Data\"\n  },\n  {\n    \"animal\": \"Frigatebird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009069/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009069.do\",\n    \"title\": \"Programming MapPoint in .NET\"\n  },\n  {\n    \"animal\": \"Frilled Coquette Hummingbird, perched\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928381/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928381.do\",\n    \"title\": \"DHCP for Windows 2000\"\n  },\n  {\n    \"animal\": \"Frilled Lizard \\\"dragon\\\"\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000523/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000523.do\",\n    \"title\": \"Creating Applications with Mozilla\"\n  },\n  {\n    \"animal\": \"Fringed Gecko, aka Flying Gekko\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529857/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529857.do\",\n    \"title\": \"Flex 3 Cookbook\"\n  },\n  {\n    \"animal\": \"Frontier Man on Rope with Axe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006617/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006617.do\",\n    \"title\": \"Managing Security with Snort & IDS Tools\"\n  },\n  {\n    \"animal\": \"Frontier Man using a telescope\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518165/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518165.do\",\n    \"title\": \"Security Monitoring\"\n  },\n  {\n    \"animal\": \"Fruit Bat with young\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920012337/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920012337.do\",\n    \"title\": \"Introducing Regular Expressions\"\n  },\n  {\n    \"animal\": \"Galago Lemur aka Senegal Galago\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002435/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002435.do\",\n    \"title\": \"Learning Wireless Java\"\n  },\n  {\n    \"animal\": \"Galago aka bushbaby (Senegal Galago),  head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002534/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002534.do\",\n    \"title\": \"J2ME in a Nutshell\"\n  },\n  {\n    \"animal\": \"Galapagos Land Iguana\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029335/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029335.do\",\n    \"title\": \"PHP Cookbook\"\n  },\n  {\n    \"animal\": \"Galapagos Tortoise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006365/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006365.do\",\n    \"title\": \"Upgrading to PHP 5\"\n  },\n  {\n    \"animal\": \"Gang-Gang Cockatoo (Callocephalum Galeatum)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022060/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022060.do\",\n    \"title\": \"Designing Data Visualizations\"\n  },\n  {\n    \"animal\": \"Gannet, Head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927148/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927148.do\",\n    \"title\": \"Excel 2000 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Gannet, perched\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514525/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514525.do\",\n    \"title\": \"Excel 2007 Pocket Guide\"\n  },\n  {\n    \"animal\": \"Garden Dormouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026174/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026174.do\",\n    \"title\": \"Just Hibernate\"\n  },\n  {\n    \"animal\": \"Garden Spider\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518769/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518769.do\",\n    \"title\": \"Building Social Web Applications\"\n  },\n  {\n    \"animal\": \"Garganey Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021407/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021407.do\",\n    \"title\": \"20 Recipes for Programming MVC 3\"\n  },\n  {\n    \"animal\": \"Gemmous Dragonet, fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022633/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022633.do\",\n    \"title\": \"Building HTML5 Games with ImpactJS\"\n  },\n  {\n    \"animal\": \"Geoffrey's Spider Monkeys\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920251897/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920251897.do\",\n    \"title\": \"Learning Chaos Engineering\"\n  },\n  {\n    \"animal\": \"German Shepherd\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020226/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020226.do\",\n    \"title\": \"An Introduction to MapReduce with Pete Warden\"\n  },\n  {\n    \"animal\": \"Giant Anteater\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596517335/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596517335.do\",\n    \"title\": \"Maven: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Giant Bamboo Ladybird Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920053576/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920053576.do\",\n    \"title\": \"Learning FPGAs\"\n  },\n  {\n    \"animal\": \"Giant Green Sea Anemone\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510503/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510503.do\",\n    \"title\": \"Building a Web 2.0 Portal with ASP.NET 3.5\"\n  },\n  {\n    \"animal\": \"Giant Petrel Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527563/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527563.do\",\n    \"title\": \"Programming .NET 3.5\"\n  },\n  {\n    \"animal\": \"Giant Squid\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001629/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001629.do\",\n    \"title\": \"Squid: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Glassfish, aka ambassis vaivasensis\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030614/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030614.do\",\n    \"title\": \"Java EE 7 Essentials\"\n  },\n  {\n    \"animal\": \"Glossy Starling\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101213/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101213.do\",\n    \"title\": \"Unicode Explained\"\n  },\n  {\n    \"animal\": \"Gnu aka Wildebeest\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924963/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924963.do\",\n    \"title\": \"GNU Emacs Pocket Reference\"\n  },\n  {\n    \"animal\": \"Gnu aka Wildebeest\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006488/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006488.do\",\n    \"title\": \"Learning GNU Emacs\"\n  },\n  {\n    \"animal\": \"Gold Carp Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020561/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020561.do\",\n    \"title\": \"Elastic Beanstalk\"\n  },\n  {\n    \"animal\": \"Golden Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007348/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007348.do\",\n    \"title\": \"JBoss at Work: A Practical Guide\"\n  },\n  {\n    \"animal\": \"Golden Jackal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101091/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101091.do\",\n    \"title\": \"Web Site Cookbook\"\n  },\n  {\n    \"animal\": \"Golden Lion Tamarin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025528/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025528.do\",\n    \"title\": \"Appcelerator Titanium: Up and Running\"\n  },\n  {\n    \"animal\": \"Golden Oriole in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596515096/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596515096.do\",\n    \"title\": \"Painting the Web\"\n  },\n  {\n    \"animal\": \"Golden Pheasant, in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155100/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155100.do\",\n    \"title\": \"Google SketchUp Cookbook\"\n  },\n  {\n    \"animal\": \"Goldfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102098/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102098.do\",\n    \"title\": \"Learning C# 2005\"\n  },\n  {\n    \"animal\": \"Goliath Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528249/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528249.do\",\n    \"title\": \"A+, Network+, Security+ Exams in a Nutshell\"\n  },\n  {\n    \"animal\": \"Goosander Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027577/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027577.do\",\n    \"title\": \"SDN: Software Defined Networks\"\n  },\n  {\n    \"animal\": \"Gooseneck Barnacles\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920051879/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920051879.do\",\n    \"title\": \"Working with Static Sites\"\n  },\n  {\n    \"animal\": \"Gorgeted Bird of Paradise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596153984/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596153984.do\",\n    \"title\": \"JUNOS Enterprise Switching\"\n  },\n  {\n    \"animal\": \"Gorilla Babies\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102081/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102081.do\",\n    \"title\": \"UML 2.0 Pocket Reference\"\n  },\n  {\n    \"animal\": \"Gorilla, pair (mother & baby)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009823/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009823.do\",\n    \"title\": \"Learning UML 2.0\"\n  },\n  {\n    \"animal\": \"Grasshopper group\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804831/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804831.do\",\n    \"title\": \"iPhone 3D Programming\"\n  },\n  {\n    \"animal\": \"Gray Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037194/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037194.do\",\n    \"title\": \"Hack and HHVM\"\n  },\n  {\n    \"animal\": \"Gray Wolf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005634/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005634.do\",\n    \"title\": \"JavaServer Pages\"\n  },\n  {\n    \"animal\": \"Gray Wolf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002312/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002312.do\",\n    \"title\": \"JavaServer Pages Pocket Reference\"\n  },\n  {\n    \"animal\": \"Gray-headed Lovebirds\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034339/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034339.do\",\n    \"title\": \"I Heart Logs\"\n  },\n  {\n    \"animal\": \"Great Black Cockatoo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780937175224/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780937175224.do\",\n    \"title\": \"termcap and terminfo\"\n  },\n  {\n    \"animal\": \"Great Cormorant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029571/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029571.do\",\n    \"title\": \"Java Web Services: Up and Running\"\n  },\n  {\n    \"animal\": \"Great Gray Shrike\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013693/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013693.do\",\n    \"title\": \"Getting the Most Out of Google Apps for Business\"\n  },\n  {\n    \"animal\": \"Great Gray Shrike\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596801601/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596801601.do\",\n    \"title\": \"Using Google App Engine\"\n  },\n  {\n    \"animal\": \"Great Plover\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007577/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007577.do\",\n    \"title\": \".NET Compact Framework Pocket Guide\"\n  },\n  {\n    \"animal\": \"Great Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024767/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024767.do\",\n    \"title\": \"Spring Data\"\n  },\n  {\n    \"animal\": \"Great White Heron\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596521196/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596521196.do\",\n    \"title\": \"Using SQLite\"\n  },\n  {\n    \"animal\": \"Great White Pelican\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029519/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029519.do\",\n    \"title\": \"Apache Sqoop Cookbook\"\n  },\n  {\n    \"animal\": \"Great-Eared Goatsucker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920017776/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920017776.do\",\n    \"title\": \"Making Embedded Systems\"\n  },\n  {\n    \"animal\": \"Greater Honeyguide\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005030/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005030.do\",\n    \"title\": \"Perl Debugger Pocket Reference\"\n  },\n  {\n    \"animal\": \"Greater Kudu (Antelope)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510374/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510374.do\",\n    \"title\": \"Programming WPF\"\n  },\n  {\n    \"animal\": \"Greater Racket-tail Drongo, Dicrurus paradiseus\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021711/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021711.do\",\n    \"title\": \"Mobile HTML5\"\n  },\n  {\n    \"animal\": \"Greek Tortoise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025245/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025245.do\",\n    \"title\": \"Maintainable JavaScript\"\n  },\n  {\n    \"animal\": \"Green Broadbill, aka Calyptomena Verdis\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026280/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026280.do\",\n    \"title\": \"Learning from jQuery\"\n  },\n  {\n    \"animal\": \"Green Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596523015/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596523015.do\",\n    \"title\": \"Ruby Best Practices\"\n  },\n  {\n    \"animal\": \"Green Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100278/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100278.do\",\n    \"title\": \"GDB Pocket Reference\"\n  },\n  {\n    \"animal\": \"Green Monkey 1 (adult holding a baby)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002053/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002053.do\",\n    \"title\": \"Perl and XML\"\n  },\n  {\n    \"animal\": \"Green Sandpiper\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920229889/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920229889.do\",\n    \"title\": \"Programming Kubernetes\"\n  },\n  {\n    \"animal\": \"Green woodpecker (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026921/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026921.do\",\n    \"title\": \"Packet Guide to Voice over IP\"\n  },\n  {\n    \"animal\": \"Green-tailed Jacamar, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024309/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024309.do\",\n    \"title\": \"The Little Book on CoffeeScript\"\n  },\n  {\n    \"animal\": \"Grey Heron (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920047995/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920047995.do\",\n    \"title\": \"Efficient R Programming\"\n  },\n  {\n    \"animal\": \"Grey-headed Woodpecker (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920049159/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920049159.do\",\n    \"title\": \"Building Maintainable Software, Java Edition\"\n  },\n  {\n    \"animal\": \"Greyhound\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529307/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529307.do\",\n    \"title\": \"High Performance Web Sites\"\n  },\n  {\n    \"animal\": \"Greylag Goose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527730/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527730.do\",\n    \"title\": \"C# 3.0 Design Patterns\"\n  },\n  {\n    \"animal\": \"Griffin Vulture\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018483/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018483.do\",\n    \"title\": \"Machine Learning for Hackers\"\n  },\n  {\n    \"animal\": \"Grizzly Bear\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155940/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155940.do\",\n    \"title\": \"CSS Cookbook\"\n  },\n  {\n    \"animal\": \"Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928626/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928626.do\",\n    \"title\": \"HTTP Pocket Reference\"\n  },\n  {\n    \"animal\": \"Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925090/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925090.do\",\n    \"title\": \"HTTP: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Growling Lions (pair)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101497/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101497.do\",\n    \"title\": \"Java and XML\"\n  },\n  {\n    \"animal\": \"Guanaco\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920158059/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920158059.do\",\n    \"title\": \"Programming TypeScript\"\n  },\n  {\n    \"animal\": \"Guinea Pigs, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007393/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007393.do\",\n    \"title\": \"NUnit Pocket Reference\"\n  },\n  {\n    \"animal\": \"Guinea fowl, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021735/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021735.do\",\n    \"title\": \"Programming Interactivity\"\n  },\n  {\n    \"animal\": \"Guitarfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005207/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005207.do\",\n    \"title\": \"ASP.NET in a Nutshell\"\n  },\n  {\n    \"animal\": \"Guitarfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529567/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529567.do\",\n    \"title\": \"Programming ASP.NET 3.5\"\n  },\n  {\n    \"animal\": \"Gyr Falcon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926769/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926769.do\",\n    \"title\": \"Designing and Programming CICS Applications\"\n  },\n  {\n    \"animal\": \"Haddock fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022282/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022282.do\",\n    \"title\": \"Essential iOS Build and Release\"\n  },\n  {\n    \"animal\": \"Hairtail Silvery Fish, aka scabbardfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024040/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024040.do\",\n    \"title\": \"Programming ASP.NET MVC 4\"\n  },\n  {\n    \"animal\": \"Hamadryas Baboon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002251/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002251.do\",\n    \"title\": \"Embedding Perl in HTML with Mason\"\n  },\n  {\n    \"animal\": \"Hamadryas Butterfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920062776/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920062776.do\",\n    \"title\": \"Learning Perl 6\"\n  },\n  {\n    \"animal\": \"Hammerhead Shark\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009090/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009090.do\",\n    \"title\": \".NET Gotchas\"\n  },\n  {\n    \"animal\": \"Hanuman Langur Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926318/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926318.do\",\n    \"title\": \"Win32 API Programming with Visual Basic\"\n  },\n  {\n    \"animal\": \"Harbour Seal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041597/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041597.do\",\n    \"title\": \"Introduction to JavaScript Object Notation\"\n  },\n  {\n    \"animal\": \"Harlequin Longhorn Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596520731/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596520731.do\",\n    \"title\": \"Java SOA Cookbook\"\n  },\n  {\n    \"animal\": \"Harp Seal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032465/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032465.do\",\n    \"title\": \"iOS 7 Programming Fundamentals\"\n  },\n  {\n    \"animal\": \"Harpy Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018315/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018315.do\",\n    \"title\": \"25 Recipes for Getting Started with R\"\n  },\n  {\n    \"animal\": \"Harpy Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022008/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022008.do\",\n    \"title\": \"R in a Nutshell\"\n  },\n  {\n    \"animal\": \"Harrier, Foxhound\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033899/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033899.do\",\n    \"title\": \"BeagleBone Cookbook\"\n  },\n  {\n    \"animal\": \"Harvest Mouse, looking up\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000967/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000967.do\",\n    \"title\": \"Python Standard Library\"\n  },\n  {\n    \"animal\": \"Hatteria Great Fringed Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596801694/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596801694.do\",\n    \"title\": \"RESTful Web Services Cookbook\"\n  },\n  {\n    \"animal\": \"Hawfinch bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021230/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021230.do\",\n    \"title\": \"Mapping with Drupal\"\n  },\n  {\n    \"animal\": \"Hawksbill Sea Turtle 3\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510336/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510336.do\",\n    \"title\": \"Version Control with Subversion\"\n  },\n  {\n    \"animal\": \"Hawksbill Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020837/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020837.do\",\n    \"title\": \"Getting Started with CouchDB\"\n  },\n  {\n    \"animal\": \"Hawksbill Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001957/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001957.do\",\n    \"title\": \"Learning the Korn Shell\"\n  },\n  {\n    \"animal\": \"Hedgehog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596517724/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596517724.do\",\n    \"title\": \"Harnessing Hibernate\"\n  },\n  {\n    \"animal\": \"Helmet Cockatoo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019893/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019893.do\",\n    \"title\": \"50 Tips and Tricks for MongoDB Developers\"\n  },\n  {\n    \"animal\": \"Helmet Shrike Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020516/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020516.do\",\n    \"title\": \"Packet Guide to Core Network Protocols\"\n  },\n  {\n    \"animal\": \"Hermit crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102074/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102074.do\",\n    \"title\": \"Programming .NET Components\"\n  },\n  {\n    \"animal\": \"Herring\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101855/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101855.do\",\n    \"title\": \"Practical Perforce\"\n  },\n  {\n    \"animal\": \"Herring Gull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927049/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927049.do\",\n    \"title\": \"Outlook 2000 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Herring Gull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004446/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004446.do\",\n    \"title\": \"Outlook Pocket Guide\"\n  },\n  {\n    \"animal\": \"Hippopotamus, seated\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925779/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925779.do\",\n    \"title\": \"JavaScript Application Cookbook\"\n  },\n  {\n    \"animal\": \"Hoffman's Sloth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028468/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028468.do\",\n    \"title\": \"RESTful Web APIs\"\n  },\n  {\n    \"animal\": \"Honey Badger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920140399/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920140399.do\",\n    \"title\": \"Programming Bitcoin\"\n  },\n  {\n    \"animal\": \"Honey Bee or Hornet 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922372/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922372.do\",\n    \"title\": \"Oracle Performance Tuning\"\n  },\n  {\n    \"animal\": \"Honey Bee or Hornet 3\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002688/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002688.do\",\n    \"title\": \"Oracle SQL Tuning Pocket Reference\"\n  },\n  {\n    \"animal\": \"Honey Buzzard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021377/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021377.do\",\n    \"title\": \"Building on SugarCRM\"\n  },\n  {\n    \"animal\": \"Honeybees and Honeycomb\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033158/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033158.do\",\n    \"title\": \"Building Microservices\"\n  },\n  {\n    \"animal\": \"Hooded Crow\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596809577/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596809577.do\",\n    \"title\": \"Real World Instrumentation with Python\"\n  },\n  {\n    \"animal\": \"Hooded Seal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022626/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022626.do\",\n    \"title\": \"Exploring Everyday Things with R and Ruby\"\n  },\n  {\n    \"animal\": \"Hoopoe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102401/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102401.do\",\n    \"title\": \"Flash 8 Cookbook\"\n  },\n  {\n    \"animal\": \"Horned Frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527624/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527624.do\",\n    \"title\": \"Windows Vista Annoyances\"\n  },\n  {\n    \"animal\": \"Horned Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006099/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006099.do\",\n    \"title\": \"Ant: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Horned Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002619/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002619.do\",\n    \"title\": \"Learning the Unix Operating System\"\n  },\n  {\n    \"animal\": \"Horned Screamer Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596519254/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596519254.do\",\n    \"title\": \"LINQ Pocket Reference\"\n  },\n  {\n    \"animal\": \"Horned Trunk Fish aka Honeycomb Cowfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021810/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021810.do\",\n    \"title\": \"Getting Started with OAuth 2.0\"\n  },\n  {\n    \"animal\": \"Horse Mackerel (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032922/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032922.do\",\n    \"title\": \"Developing Web Components\"\n  },\n  {\n    \"animal\": \"Horse, Thoroughbred*, trotting or running\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002275/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002275.do\",\n    \"title\": \"Practical mod_perl\"\n  },\n  {\n    \"animal\": \"Horseshoe Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100520/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100520.do\",\n    \"title\": \"802.11 Wireless Networks: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"House Martin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596801311/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596801311.do\",\n    \"title\": \"Mercurial: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Howler Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514082/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514082.do\",\n    \"title\": \"JavaScript & DHTML Cookbook\"\n  },\n  {\n    \"animal\": \"Hummingbird Moth 1\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000882/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000882.do\",\n    \"title\": \"Java Programming with Oracle JDBC\"\n  },\n  {\n    \"animal\": \"Hummingbird, Sun Gem flying\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516178/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516178.do\",\n    \"title\": \"The Ruby Programming Language\"\n  },\n  {\n    \"animal\": \"Hummingbird, pointed down\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006761/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006761.do\",\n    \"title\": \"Better, Faster, Lighter Java\"\n  },\n  {\n    \"animal\": \"Hungarian Grey Bull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037040/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037040.do\",\n    \"title\": \"Blockchain\"\n  },\n  {\n    \"animal\": \"Hunting cat (domestic cat)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920012221/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920012221.do\",\n    \"title\": \"Physics for Game Developers\"\n  },\n  {\n    \"animal\": \"Hydra Porpita Pacifica\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025573/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025573.do\",\n    \"title\": \"Monitoring with Ganglia\"\n  },\n  {\n    \"animal\": \"Hyrax, side view\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001537/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001537.do\",\n    \"title\": \"Using SANs and NAS\"\n  },\n  {\n    \"animal\": \"I'iwi (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920042266/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920042266.do\",\n    \"title\": \"React: Up & Running\"\n  },\n  {\n    \"animal\": \"Iberian Lynx (Lynx pardinus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155209/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155209.do\",\n    \"title\": \"iPhone Open Application Development\"\n  },\n  {\n    \"animal\": \"Impala, Common (Antelope)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033936/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033936.do\",\n    \"title\": \"Getting Started with Impala\"\n  },\n  {\n    \"animal\": \"Indian Civet aka Weasel Malacca, small\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026587/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026587.do\",\n    \"title\": \"Building Node Applications with MongoDB and Backbone\"\n  },\n  {\n    \"animal\": \"Indian Monitor (Lizard)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050773/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050773.do\",\n    \"title\": \"Practical Monitoring\"\n  },\n  {\n    \"animal\": \"Irish Setter\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004620/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004620.do\",\n    \"title\": \"Cocoa in a Nutshell\"\n  },\n  {\n    \"animal\": \"Italian Greyhound\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518745/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518745.do\",\n    \"title\": \"Universal Design for Web Applications\"\n  },\n  {\n    \"animal\": \"Ivory-billed Woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924154/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924154.do\",\n    \"title\": \"ADO:  ActiveX Data Objects\"\n  },\n  {\n    \"animal\": \"Jabiru Stork\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527211/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527211.do\",\n    \"title\": \"XSLT\"\n  },\n  {\n    \"animal\": \"Jacana bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000509/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000509.do\",\n    \"title\": \"Server Load Balancing\"\n  },\n  {\n    \"animal\": \"Japanese Badger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033332/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033332.do\",\n    \"title\": \"Hadoop Security\"\n  },\n  {\n    \"animal\": \"Japanese Rhino Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037231/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037231.do\",\n    \"title\": \"Essential Cybersecurity Science\"\n  },\n  {\n    \"animal\": \"Japanese Sable\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920014607/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920014607.do\",\n    \"title\": \"jQuery Mobile: Up and Running\"\n  },\n  {\n    \"animal\": \"Japanese Waxwing Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027409/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027409.do\",\n    \"title\": \"Heroku: Up and Running\"\n  },\n  {\n    \"animal\": \"Javan Rhinoceros\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920011460/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920011460.do\",\n    \"title\": \"JavaScript Pocket Reference\"\n  },\n  {\n    \"animal\": \"Javan Rhinoceros\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920017080/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920017080.do\",\n    \"title\": \"Tom Hughes-Croucher on Node\"\n  },\n  {\n    \"animal\": \"Javan Tiger, in the bush\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027829/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027829.do\",\n    \"title\": \"Java 7 Pocket Guide\"\n  },\n  {\n    \"animal\": \"Jellyfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001193/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001193.do\",\n    \"title\": \"Programming Web Services with XML-RPC\"\n  },\n  {\n    \"animal\": \"Jellyfish chrysaora\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024729/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024729.do\",\n    \"title\": \"WebGL: Up and Running\"\n  },\n  {\n    \"animal\": \"Jerboa\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920014225/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920014225.do\",\n    \"title\": \"HTML5 Mobile Web Development\"\n  },\n  {\n    \"animal\": \"Jerboa\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026259/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026259.do\",\n    \"title\": \"Programming the Mobile Web\"\n  },\n  {\n    \"animal\": \"Jersey Cow\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004361/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004361.do\",\n    \"title\": \"C Pocket Reference\"\n  },\n  {\n    \"animal\": \"Jersey Cow\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897212381/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897212381.do\",\n    \"title\": \"C kurz & gut\"\n  },\n  {\n    \"animal\": \"Jersey Cow\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923065/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923065.do\",\n    \"title\": \"Practical C Programming\"\n  },\n  {\n    \"animal\": \"John Dory Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924840/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924840.do\",\n    \"title\": \"Java 2D Graphics\"\n  },\n  {\n    \"animal\": \"Kaka, Nestor Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034421/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034421.do\",\n    \"title\": \"R Packages\"\n  },\n  {\n    \"animal\": \"Kakapo, aka Owl Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034407/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034407.do\",\n    \"title\": \"R for Data Science\"\n  },\n  {\n    \"animal\": \"Kanchil Mouse Deer\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596521271/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596521271.do\",\n    \"title\": \"XMPP: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Kangaroo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780937175774/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780937175774.do\",\n    \"title\": \"Power Programming with RPC\"\n  },\n  {\n    \"animal\": \"Kangaroo Rat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001230/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001230.do\",\n    \"title\": \"Building Java Enterprise Applications\"\n  },\n  {\n    \"animal\": \"Kangaroo nibbling a tree branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020981/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020981.do\",\n    \"title\": \"Getting Started with Roo\"\n  },\n  {\n    \"animal\": \"Kestrel Common\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920016038/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920016038.do\",\n    \"title\": \"HTML5 Cookbook\"\n  },\n  {\n    \"animal\": \"Kestrel Sparrow Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008819/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008819.do\",\n    \"title\": \"Developing Feeds with RSS and Atom\"\n  },\n  {\n    \"animal\": \"Kestrel Sparrow Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526979/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526979.do\",\n    \"title\": \"What Are Syndication Feeds\"\n  },\n  {\n    \"animal\": \"King Charles Spaniel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920010265/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920010265.do\",\n    \"title\": \"Learning the iOS 4 SDK for JavaScript Programmers\"\n  },\n  {\n    \"animal\": \"King Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021087/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021087.do\",\n    \"title\": \"Practical JIRA Plugins\"\n  },\n  {\n    \"animal\": \"King Penguins (trio)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529321/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529321.do\",\n    \"title\": \"Programming Collective Intelligence\"\n  },\n  {\n    \"animal\": \"King Vulture\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007072/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007072.do\",\n    \"title\": \"SpamAssassin\"\n  },\n  {\n    \"animal\": \"Kit Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920052555/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920052555.do\",\n    \"title\": \"Applied Text Analysis with Python\"\n  },\n  {\n    \"animal\": \"Kite\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596802363/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596802363.do\",\n    \"title\": \"Data Analysis with Open Source Tools\"\n  },\n  {\n    \"animal\": \"Kiwi bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780937175309/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780937175309.do\",\n    \"title\": \"Checking C Programs with Lint\"\n  },\n  {\n    \"animal\": \"Knysna Turaco (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920045335/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920045335.do\",\n    \"title\": \"SVG Animations\"\n  },\n  {\n    \"animal\": \"Koala Bear, on a Branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029274/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029274.do\",\n    \"title\": \"HTML5 Pocket Reference\"\n  },\n  {\n    \"animal\": \"Koala Bear, without branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527327/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527327.do\",\n    \"title\": \"HTML & XHTML: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Krait Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526894/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526894.do\",\n    \"title\": \"Programming Flex 2\"\n  },\n  {\n    \"animal\": \"Krait Snake\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516215/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516215.do\",\n    \"title\": \"Programming Flex 3\"\n  },\n  {\n    \"animal\": \"Kudu antelope, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526733/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526733.do\",\n    \"title\": \"XAML in a Nutshell\"\n  },\n  {\n    \"animal\": \"Kuhl's Flying Gecko\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596805623/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596805623.do\",\n    \"title\": \"Flex 4 Cookbook\"\n  },\n  {\n    \"animal\": \"Kultarr (jumping jerboa)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026877/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026877.do\",\n    \"title\": \"Sencha Touch 2 Up and Running\"\n  },\n  {\n    \"animal\": \"Ladybug\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925168/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925168.do\",\n    \"title\": \"Oracle Database Administration: The Essential Refe\"\n  },\n  {\n    \"animal\": \"Lammergeir, aka Bearded Eagle, aka Bearded Vulture\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041429/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041429.do\",\n    \"title\": \"Security for Web Developers\"\n  },\n  {\n    \"animal\": \"Land Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002978/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002978.do\",\n    \"title\": \"TCP/IP Network Administration\"\n  },\n  {\n    \"animal\": \"Land Snail\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008956/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008956.do\",\n    \"title\": \"SSH, The Secure Shell: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Lantern Fly, flying\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006327/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006327.do\",\n    \"title\": \"Mastering Oracle SQL\"\n  },\n  {\n    \"animal\": \"Lapland Longspur bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023241/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023241.do\",\n    \"title\": \"Sakai OAE Deployment and Management\"\n  },\n  {\n    \"animal\": \"Lapwing Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027102/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027102.do\",\n    \"title\": \"An Introduction to iOS Programming\"\n  },\n  {\n    \"animal\": \"Lapwing Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028642/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028642.do\",\n    \"title\": \"Learning iOS Programming\"\n  },\n  {\n    \"animal\": \"Large Claw Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020295/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020295.do\",\n    \"title\": \"Design and Prototyping for Drupal\"\n  },\n  {\n    \"animal\": \"Large Indian Civet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030492/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030492.do\",\n    \"title\": \"Client-Server Web Apps with JavaScript and Java\"\n  },\n  {\n    \"animal\": \"Large Spotted Genet, aka Blotched Genett\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033592/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033592.do\",\n    \"title\": \"Using WebPageTest\"\n  },\n  {\n    \"animal\": \"Leaf-nosed Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008451/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008451.do\",\n    \"title\": \"sendmail 8.13 Companion\"\n  },\n  {\n    \"animal\": \"Least Weasel (Mustela nivalis)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596153595/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596153595.do\",\n    \"title\": \"iPhone Forensics\"\n  },\n  {\n    \"animal\": \"Least Weasel (Mustela nivalis), head and shoulders\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009878/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009878.do\",\n    \"title\": \"Web Design in a Nutshell\"\n  },\n  {\n    \"animal\": \"Leatherback Sea Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022428/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022428.do\",\n    \"title\": \"VMware Cookbook\"\n  },\n  {\n    \"animal\": \"Lemur in color (Verreauxs Sifaka)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007652/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007652.do\",\n    \"title\": \"Ambient Findability\"\n  },\n  {\n    \"animal\": \"Leonberger Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022664/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022664.do\",\n    \"title\": \"Using Mac OS X Lion Server\"\n  },\n  {\n    \"animal\": \"Leopard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529819/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529819.do\",\n    \"title\": \"Mac OS X Leopard Pocket Guide\"\n  },\n  {\n    \"animal\": \"Leopard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596520632/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596520632.do\",\n    \"title\": \"Mac OS X for Unix Geeks (Leopard)\"\n  },\n  {\n    \"animal\": \"Leopard Ground Squirrel aka Hood's Marmot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027041/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027041.do\",\n    \"title\": \"MongoDB Applied Design Patterns\"\n  },\n  {\n    \"animal\": \"Lesser Spotted Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030713/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030713.do\",\n    \"title\": \"Java 8 Lambdas\"\n  },\n  {\n    \"animal\": \"Lettered Aracari (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026525/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026525.do\",\n    \"title\": \"eXist\"\n  },\n  {\n    \"animal\": \"Liger, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004941/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004941.do\",\n    \"title\": \"Sequence Analysis in a Nutshell: A Guide to Tools\"\n  },\n  {\n    \"animal\": \"Lilac-Breasted Roller (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033851/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033851.do\",\n    \"title\": \"User Story Mapping\"\n  },\n  {\n    \"animal\": \"Lime Tree Sphinx Moth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596809539/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596809539.do\",\n    \"title\": \"Introduction to Search with Sphinx\"\n  },\n  {\n    \"animal\": \"Lion (male)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005689/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005689.do\",\n    \"title\": \"Hardcore Java\"\n  },\n  {\n    \"animal\": \"Lion-tailed Macaque aka Wanderoo aka Lion-tailed Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516482/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516482.do\",\n    \"title\": \"Dojo: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Lioness\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022404/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022404.do\",\n    \"title\": \"What's New in Java 7\"\n  },\n  {\n    \"animal\": \"Lioness, facing back\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003715/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003715.do\",\n    \"title\": \"Web Privacy with P3P\"\n  },\n  {\n    \"animal\": \"Little Blue Heron\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920269885/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920269885.do\",\n    \"title\": \"Practical Automated Machine Learning on Azure\"\n  },\n  {\n    \"animal\": \"Little Green Bee-eater (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024651/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024651.do\",\n    \"title\": \"Understanding Context\"\n  },\n  {\n    \"animal\": \"Little Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023456/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023456.do\",\n    \"title\": \"Learning Android\"\n  },\n  {\n    \"animal\": \"Little Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528126/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528126.do\",\n    \"title\": \"Mastering Regular Expressions\"\n  },\n  {\n    \"animal\": \"Little Pied Cormorant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030133/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030133.do\",\n    \"title\": \"JavaMail API\"\n  },\n  {\n    \"animal\": \"Llama\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920014430/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920014430.do\",\n    \"title\": \"Randal Schwartz on Learning Perl\"\n  },\n  {\n    \"animal\": \"Locust aka grasshopper (not flying)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100575/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100575.do\",\n    \"title\": \"DNS and BIND\"\n  },\n  {\n    \"animal\": \"Locust aka grasshopper (not flying)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020158/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020158.do\",\n    \"title\": \"DNS and BIND on IPv6\"\n  },\n  {\n    \"animal\": \"Loggerhead Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025733/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025733.do\",\n    \"title\": \"Getting Started with Couchbase Server\"\n  },\n  {\n    \"animal\": \"Long Centipede\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923751/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923751.do\",\n    \"title\": \"Oracle Built-in Packages\"\n  },\n  {\n    \"animal\": \"Long-Eared Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018421/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018421.do\",\n    \"title\": \"JavaScript Web Applications\"\n  },\n  {\n    \"animal\": \"Long-beaked Echidna, aka Porcupine Echidna\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026914/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026914.do\",\n    \"title\": \"Scala Cookbook\"\n  },\n  {\n    \"animal\": \"Long-eared Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024972/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024972.do\",\n    \"title\": \"Git Pocket Guide\"\n  },\n  {\n    \"animal\": \"Long-eared Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024774/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024774.do\",\n    \"title\": \"McCullough and Berglund on Mastering Advanced Git\"\n  },\n  {\n    \"animal\": \"Long-eared Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920017462/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920017462.do\",\n    \"title\": \"McCullough and Berglund on Mastering Git\"\n  },\n  {\n    \"animal\": \"Long-eared Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022862/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022862.do\",\n    \"title\": \"Version Control with Git\"\n  },\n  {\n    \"animal\": \"Long-tailed Chinchilla\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596522537/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596522537.do\",\n    \"title\": \"Search Engine Optimization for Flash\"\n  },\n  {\n    \"animal\": \"Lovebird in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013716/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013716.do\",\n    \"title\": \"Designing Mobile Interfaces\"\n  },\n  {\n    \"animal\": \"Lowland Paca or Spotted Cavy\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025085/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025085.do\",\n    \"title\": \"Hadoop Operations\"\n  },\n  {\n    \"animal\": \"Luna Moth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155261/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155261.do\",\n    \"title\": \"Palm webOS\"\n  },\n  {\n    \"animal\": \"Macaw Parrot, Head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924932/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924932.do\",\n    \"title\": \"Lingo in a Nutshell\"\n  },\n  {\n    \"animal\": \"Madagascar Harrier Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028048/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028048.do\",\n    \"title\": \"High Performance Browser Networking\"\n  },\n  {\n    \"animal\": \"Madagascar blossoms, fruit branch in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516253/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516253.do\",\n    \"title\": \"Designing Web Interfaces\"\n  },\n  {\n    \"animal\": \"Magpie\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029281/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029281.do\",\n    \"title\": \"Building iPhone and iPad Electronic Projects\"\n  },\n  {\n    \"animal\": \"Mahi-Mahi fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029397/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029397.do\",\n    \"title\": \"Efficient Android Threading\"\n  },\n  {\n    \"animal\": \"Malabar Giant Squirrel, aka Indian Giant Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920039952/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920039952.do\",\n    \"title\": \"Building Applications on Mesos\"\n  },\n  {\n    \"animal\": \"Malachite Kingfisher (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920061977/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920061977.do\",\n    \"title\": \"Practical Tableau\"\n  },\n  {\n    \"animal\": \"Malay Fox-Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021162/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021162.do\",\n    \"title\": \"Basic Sensors in iOS\"\n  },\n  {\n    \"animal\": \"Malayan Badger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021964/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021964.do\",\n    \"title\": \"MySQL Troubleshooting \"\n  },\n  {\n    \"animal\": \"Male Fencer 2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006266/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006266.do\",\n    \"title\": \"Mastering FreeBSD and OpenBSD Security\"\n  },\n  {\n    \"animal\": \"Maleo Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022886/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022886.do\",\n    \"title\": \"Building Android Apps with HTML, CSS, and JavaScript\"\n  },\n  {\n    \"animal\": \"Mammoth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928466/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928466.do\",\n    \"title\": \"Practical PostgreSQL\"\n  },\n  {\n    \"animal\": \"Man and Bubble\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005658/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005658.do\",\n    \"title\": \"Understanding the Linux Kernel\"\n  },\n  {\n    \"animal\": \"Man in Flying Machine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026891/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026891.do\",\n    \"title\": \"Linux System Programming\"\n  },\n  {\n    \"animal\": \"Man sharpening tool with machine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100759/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100759.do\",\n    \"title\": \"Knoppix Pocket Reference\"\n  },\n  {\n    \"animal\": \"Man with stethoscope Invention\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100063/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100063.do\",\n    \"title\": \"Internet Forensics\"\n  },\n  {\n    \"animal\": \"Man working on Log Jam\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927308/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927308.do\",\n    \"title\": \"Managing RAID on Linux\"\n  },\n  {\n    \"animal\": \"Manatee\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033196/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033196.do\",\n    \"title\": \"Hadoop Application Architectures\"\n  },\n  {\n    \"animal\": \"Mandrill Baboon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007539/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007539.do\",\n    \"title\": \"Windows Server 2003 Security Cookbook\"\n  },\n  {\n    \"animal\": \"Maned Sheep (aka Barbary sheep)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013044/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013044.do\",\n    \"title\": \"Supercharged JavaScript Graphics\"\n  },\n  {\n    \"animal\": \"Mantis Shrimp\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033950/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033950.do\",\n    \"title\": \"Data Algorithms\"\n  },\n  {\n    \"animal\": \"Marabou Stork\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003272/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003272.do\",\n    \"title\": \"Learning XSLT\"\n  },\n  {\n    \"animal\": \"Marbled Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021728/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021728.do\",\n    \"title\": \"iOS 5 Programming Cookbook\"\n  },\n  {\n    \"animal\": \"Marine Invertebrate, aka Jellyfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007829/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007829.do\",\n    \"title\": \"Java Threads\"\n  },\n  {\n    \"animal\": \"Markhor Goat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920121718/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920121718.do\",\n    \"title\": \"Optimizing Java\"\n  },\n  {\n    \"animal\": \"Marmoset\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006549/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006549.do\",\n    \"title\": \"AspectJ Cookbook\"\n  },\n  {\n    \"animal\": \"Marsh Harrier\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596522513/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596522513.do\",\n    \"title\": \"Adobe AIR 1.5 Cookbook\"\n  },\n  {\n    \"animal\": \"Martial Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032571/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032571.do\",\n    \"title\": \"eCommerce in the Cloud\"\n  },\n  {\n    \"animal\": \"Mastiff Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002350/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002350.do\",\n    \"title\": \"Building Cocoa Applications: A Step by Step Guide\"\n  },\n  {\n    \"animal\": \"Merlin, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024224/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024224.do\",\n    \"title\": \"Safe C++\"\n  },\n  {\n    \"animal\": \"Methona Butterfly 2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005870/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005870.do\",\n    \"title\": \"Oracle PL/SQL for DBAs\"\n  },\n  {\n    \"animal\": \"Middle Spotted Woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100896/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100896.do\",\n    \"title\": \"MySQL Stored Procedure Programming\"\n  },\n  {\n    \"animal\": \"Mimic Butterfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596519681/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596519681.do\",\n    \"title\": \"MediaWiki\"\n  },\n  {\n    \"animal\": \"Miniature Pinscher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001261/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001261.do\",\n    \"title\": \"VBScript Pocket Reference\"\n  },\n  {\n    \"animal\": \"Miniature Pinscher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004880/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004880.do\",\n    \"title\": \"VBScript in a Nutshell\"\n  },\n  {\n    \"animal\": \"Mining Women - one holding shovel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007522/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007522.do\",\n    \"title\": \"Exploring the JDS Linux Desktop\"\n  },\n  {\n    \"animal\": \"Mink\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924918/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924918.do\",\n    \"title\": \"LDAP System Administration\"\n  },\n  {\n    \"animal\": \"Minnow 2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928404/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928404.do\",\n    \"title\": \"COM+ Programming with Visual Basic\"\n  },\n  {\n    \"animal\": \"Mississippi Kite\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023968/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023968.do\",\n    \"title\": \"eBay Commerce Cookbook\"\n  },\n  {\n    \"animal\": \"Moholi Bushbaby\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925458/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925458.do\",\n    \"title\": \"Managing Microsoft Exchange Server\"\n  },\n  {\n    \"animal\": \"Mollusk (in shell)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004279/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004279.do\",\n    \"title\": \"J2EE Design Patterns\"\n  },\n  {\n    \"animal\": \"Monitor Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006563/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006563.do\",\n    \"title\": \"Essential PHP Security\"\n  },\n  {\n    \"animal\": \"Monkey (not a Chimpanze)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565920903/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565920903.do\",\n    \"title\": \"Exploring Expect\"\n  },\n  {\n    \"animal\": \"Monkey (not a Chimpanze)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924987/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924987.do\",\n    \"title\": \"Tcl/Tk Pocket Reference\"\n  },\n  {\n    \"animal\": \"Monkfish, aka sea monk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518462/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518462.do\",\n    \"title\": \"Learning ASP.NET 3.5\"\n  },\n  {\n    \"animal\": \"Montagu's Harrier (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920056294/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920056294.do\",\n    \"title\": \"Getting Started with Varnish Cache\"\n  },\n  {\n    \"animal\": \"Moor Frogs\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100711/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100711.do\",\n    \"title\": \"Visual Basic 2005 Jumpstart\"\n  },\n  {\n    \"animal\": \"Moose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529949/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529949.do\",\n    \"title\": \"Apache Cookbook\"\n  },\n  {\n    \"animal\": \"Mouflon Goat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920223771/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920223771.do\",\n    \"title\": \"Practical Data Science with SAP\"\n  },\n  {\n    \"animal\": \"Mouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924192/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924192.do\",\n    \"title\": \"CGI Programming with Perl\"\n  },\n  {\n    \"animal\": \"Mouse, walking (actually this is an elephant shrew)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927315/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927315.do\",\n    \"title\": \"GIMP Pocket Reference\"\n  },\n  {\n    \"animal\": \"Mousebird (Senegal Coly)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033011/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033011.do\",\n    \"title\": \"Getting Started with Bluetooth Low Energy\"\n  },\n  {\n    \"animal\": \"Moving Leaf insect\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008857/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008857.do\",\n    \"title\": \"Oracle SQL*Plus Pocket Reference\"\n  },\n  {\n    \"animal\": \"Moving Leaf insect\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007461/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007461.do\",\n    \"title\": \"Oracle SQL*Plus: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Murex Shell\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514242/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514242.do\",\n    \"title\": \"Programming ASP.NET AJAX\"\n  },\n  {\n    \"animal\": \"Murex Shell\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596526726/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596526726.do\",\n    \"title\": \"Programming Atlas\"\n  },\n  {\n    \"animal\": \"Musk Deer\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019886/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019886.do\",\n    \"title\": \"Building Web Apps for Google TV\"\n  },\n  {\n    \"animal\": \"Musk Meminna aka Sri Lankan Spotted Chevrotain (deer)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024347/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024347.do\",\n    \"title\": \"SharePoint Apps with LightSwitch\"\n  },\n  {\n    \"animal\": \"Musk Shrew\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023630/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023630.do\",\n    \"title\": \"Regular Expressions Cookbook\"\n  },\n  {\n    \"animal\": \"Musky Octopus\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920167433/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920167433.do\",\n    \"title\": \"Programming Quantum Computers\"\n  },\n  {\n    \"animal\": \"Musky rat-kangaroo aka potoroo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030508/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030508.do\",\n    \"title\": \"Learning jQuery Deferreds\"\n  },\n  {\n    \"animal\": \"Namaqua Sand Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032519/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032519.do\",\n    \"title\": \"Fluent Python\"\n  },\n  {\n    \"animal\": \"Narwhal\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102302/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102302.do\",\n    \"title\": \"Podcasting Pocket Guide\"\n  },\n  {\n    \"animal\": \"Natter-jack Toad\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920044994/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920044994.do\",\n    \"title\": \"Getting Started with SQL\"\n  },\n  {\n    \"animal\": \"New Zealand Kaka Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026266/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026266.do\",\n    \"title\": \"HTML5 Canvas\"\n  },\n  {\n    \"animal\": \"Newfoundland Dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596803032/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596803032.do\",\n    \"title\": \"Developing Large Web Applications\"\n  },\n  {\n    \"animal\": \"Newfoundland Dog, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923584/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923584.do\",\n    \"title\": \"VB & VBA in a Nutshell: The Language\"\n  },\n  {\n    \"animal\": \"Nicobar Pigeon, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155988/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155988.do\",\n    \"title\": \"flex & bison\"\n  },\n  {\n    \"animal\": \"Night Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000448/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000448.do\",\n    \"title\": \"Designing Active Server Pages\"\n  },\n  {\n    \"animal\": \"Nile Valley Sunbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029601/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029601.do\",\n    \"title\": \"RaphaelJS\"\n  },\n  {\n    \"animal\": \"Nilotic Monitor Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020615/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020615.do\",\n    \"title\": \"Quick Guide to Flash Catalyst\"\n  },\n  {\n    \"animal\": \"Non-Striped Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920043676/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920043676.do\",\n    \"title\": \"Client-Side Data Storage\"\n  },\n  {\n    \"animal\": \"North African Wild Ass\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001667/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001667.do\",\n    \"title\": \"Hardening Cisco Routers\"\n  },\n  {\n    \"animal\": \"North American Bullfrog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003074/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003074.do\",\n    \"title\": \"Mastering Perl for Bioinformatics\"\n  },\n  {\n    \"animal\": \"Northern Blue Devil (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920044949/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920044949.do\",\n    \"title\": \"Org Design for Design Orgs\"\n  },\n  {\n    \"animal\": \"Northern Crested Newt\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920232582/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920232582.do\",\n    \"title\": \"Machine Learning Pocket Reference\"\n  },\n  {\n    \"animal\": \"Northern Harrier Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006686/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006686.do\",\n    \"title\": \"RT Essentials\"\n  },\n  {\n    \"animal\": \"Northern Hawk Owl in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514556/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514556.do\",\n    \"title\": \"Visualizing Data\"\n  },\n  {\n    \"animal\": \"Norway Rat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006891/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006891.do\",\n    \"title\": \"Unit Test Frameworks\"\n  },\n  {\n    \"animal\": \"Nudibranch Facelina Auriculata\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920215691/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920215691.do\",\n    \"title\": \"Strengthening Deep Neural Networks\"\n  },\n  {\n    \"animal\": \"Nutcracker Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514839/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514839.do\",\n    \"title\": \"Web Security Testing Cookbook\"\n  },\n  {\n    \"animal\": \"Oarfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035343/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035343.do\",\n    \"title\": \"Deep Learning\"\n  },\n  {\n    \"animal\": \"Octopus\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028987/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028987.do\",\n    \"title\": \"Ethernet: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Octopus\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002459/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002459.do\",\n    \"title\": \"Java Management Extensions\"\n  },\n  {\n    \"animal\": \"Octopus, Graneledone\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920049500/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920049500.do\",\n    \"title\": \"Designing Across Senses\"\n  },\n  {\n    \"animal\": \"Opah fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025436/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025436.do\",\n    \"title\": \"Opa: Up and Running\"\n  },\n  {\n    \"animal\": \"Openbill Stork\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003517/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003517.do\",\n    \"title\": \"Shared Source CLI Essentials\"\n  },\n  {\n    \"animal\": \"Ophiops Lizards 2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596517359/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596517359.do\",\n    \"title\": \"The ActionScript 3.0 Quick Reference Guide: For Developers and Designers Using Flash\"\n  },\n  {\n    \"animal\": \"Opossum\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004101/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004101.do\",\n    \"title\": \"DNS & Bind Cookbook\"\n  },\n  {\n    \"animal\": \"Opossum Mouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020127/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020127.do\",\n    \"title\": \"Redis Cookbook\"\n  },\n  {\n    \"animal\": \"Orange-Winged Amazon Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028574/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028574.do\",\n    \"title\": \"Hands-On Programming with R\"\n  },\n  {\n    \"animal\": \"Orangutan\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565929432/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565929432.do\",\n    \"title\": \"Managing The Windows 2000 Registry\"\n  },\n  {\n    \"animal\": \"Oribi (small antelope)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920043584/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920043584.do\",\n    \"title\": \"Etudes for ClojureScript\"\n  },\n  {\n    \"animal\": \"Oriental Flying Gurnard aka Searobin (Fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032489/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032489.do\",\n    \"title\": \"The Uncertain Web\"\n  },\n  {\n    \"animal\": \"Ornate Chorus frogs (Cystignathus Ornatus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920010326/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920010326.do\",\n    \"title\": \"Jenkins: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Ornate Monitor Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041528/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041528.do\",\n    \"title\": \"Site Reliability Engineering\"\n  },\n  {\n    \"animal\": \"Osprey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002787/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002787.do\",\n    \"title\": \"Java & XML Data Binding\"\n  },\n  {\n    \"animal\": \"Otter Civet, aka Cynogale\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100735/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100735.do\",\n    \"title\": \"Analyzing Business Data with Excel\"\n  },\n  {\n    \"animal\": \"Owls, aka Little Owls, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897215351/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897215351.do\",\n    \"title\": \"Reguläre Ausdrücke kurz & gut\"\n  },\n  {\n    \"animal\": \"Owls, pair sitting on a stump\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514273/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514273.do\",\n    \"title\": \"Regular Expression Pocket Reference\"\n  },\n  {\n    \"animal\": \"Oystercatcher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565921320/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565921320.do\",\n    \"title\": \"Using csh & tcsh\"\n  },\n  {\n    \"animal\": \"Pacuma Toadfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028994/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028994.do\",\n    \"title\": \"Building Hybrid Android Apps with Java and JavaScript\"\n  },\n  {\n    \"animal\": \"Painted Parakeet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920189817/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920189817.do\",\n    \"title\": \"Generative Deep Learning\"\n  },\n  {\n    \"animal\": \"Painted Snipe, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013754/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013754.do\",\n    \"title\": \"Clojure Programming\"\n  },\n  {\n    \"animal\": \"Palm Civet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005382/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005382.do\",\n    \"title\": \"Office 2003 XML\"\n  },\n  {\n    \"animal\": \"Pampas Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804817/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804817.do\",\n    \"title\": \"Cocoa and Objective-C: Up and Running\"\n  },\n  {\n    \"animal\": \"Pampas Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002374/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002374.do\",\n    \"title\": \"SAX2\"\n  },\n  {\n    \"animal\": \"Papuan Hornbill (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029564/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029564.do\",\n    \"title\": \"Speaking JavaScript\"\n  },\n  {\n    \"animal\": \"Paradise Whydah (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920038481/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920038481.do\",\n    \"title\": \"Elegant SciPy\"\n  },\n  {\n    \"animal\": \"Parakeet, head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924895/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924895.do\",\n    \"title\": \"Word 2000 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020318/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020318.do\",\n    \"title\": \"Developing BlackBerry Tablet Applications with Flex 4.5\"\n  },\n  {\n    \"animal\": \"Parrotfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020622/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020622.do\",\n    \"title\": \"Programming Razor\"\n  },\n  {\n    \"animal\": \"Partridge 2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009731/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009731.do\",\n    \"title\": \"Integrating Excel and Access\"\n  },\n  {\n    \"animal\": \"Passenger Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596522056/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596522056.do\",\n    \"title\": \"Java Message Service\"\n  },\n  {\n    \"animal\": \"Peach-Faced Lovebird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018261/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018261.do\",\n    \"title\": \"21 Recipes for Mining Twitter\"\n  },\n  {\n    \"animal\": \"Peacock Flounder\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920174462/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920174462.do\",\n    \"title\": \"Database Internals\"\n  },\n  {\n    \"animal\": \"Peacock Moth (Saturnia Pyri)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516130/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516130.do\",\n    \"title\": \"Learning OpenCV\"\n  },\n  {\n    \"animal\": \"Peacock head, aka Peafowl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007645/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007645.do\",\n    \"title\": \"XML in a Nutshell\"\n  },\n  {\n    \"animal\": \"Peacock, aka Indian Peafowl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013457/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013457.do\",\n    \"title\": \"Website Architecture and Design With XML\"\n  },\n  {\n    \"animal\": \"Peacock, aka Indian Peafowl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100506/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100506.do\",\n    \"title\": \"XML Pocket Reference\"\n  },\n  {\n    \"animal\": \"Pearl Spotted Barbet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510411/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510411.do\",\n    \"title\": \"Learning Flash Media Server 2\"\n  },\n  {\n    \"animal\": \"Pearl Spotted Barbet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596515904/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596515904.do\",\n    \"title\": \"Learning Flash Media Server 3\"\n  },\n  {\n    \"animal\": \"Peccary\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027638/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027638.do\",\n    \"title\": \"AWS System Administration\"\n  },\n  {\n    \"animal\": \"Pen-tailed Tree-Shrew\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023128/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023128.do\",\n    \"title\": \"Understanding PaaS\"\n  },\n  {\n    \"animal\": \"Pencilled Lark\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025887/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025887.do\",\n    \"title\": \"What is Dart?\"\n  },\n  {\n    \"animal\": \"Pennant-winged Nightjar\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003555/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003555.do\",\n    \"title\": \"XSL-FO\"\n  },\n  {\n    \"animal\": \"Percheron Draft Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005191/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005191.do\",\n    \"title\": \"Jakarta Struts Pocket Reference\"\n  },\n  {\n    \"animal\": \"Percheron Draft Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006518/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006518.do\",\n    \"title\": \"Programming Jakarta Struts\"\n  },\n  {\n    \"animal\": \"Pere David's Deer, aka elaphure\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025122/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025122.do\",\n    \"title\": \"MapReduce Design Patterns\"\n  },\n  {\n    \"animal\": \"Persian Greyhound\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596804053/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596804053.do\",\n    \"title\": \"Windows 7: Up and Running\"\n  },\n  {\n    \"animal\": \"Peruvian Spider Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527440/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527440.do\",\n    \"title\": \"Ajax on Rails\"\n  },\n  {\n    \"animal\": \"Petit Basset Griffon Vendeen, dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018247/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018247.do\",\n    \"title\": \"Writing and Querying MapReduce Views in CouchDB\"\n  },\n  {\n    \"animal\": \"Philippine Fairy-bluebird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050568/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050568.do\",\n    \"title\": \"Mastering Azure Analytics\"\n  },\n  {\n    \"animal\": \"Phoebe Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028277/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028277.do\",\n    \"title\": \"JavaScript Testing with Jasmine\"\n  },\n  {\n    \"animal\": \"Pied Avocet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008000/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008000.do\",\n    \"title\": \"Windows Server 2003 Network Administration\"\n  },\n  {\n    \"animal\": \"Pied Kingfisher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514334/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514334.do\",\n    \"title\": \"MySQL in a Nutshell\"\n  },\n  {\n    \"animal\": \"Pied Kingfisher, on branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002114/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002114.do\",\n    \"title\": \"Managing & Using MySQL\"\n  },\n  {\n    \"animal\": \"Pied Kingfisher, on branch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514266/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514266.do\",\n    \"title\": \"MySQL Pocket Reference\"\n  },\n  {\n    \"animal\": \"Pied Wagtail (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034520/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034520.do\",\n    \"title\": \"Git for Teams\"\n  },\n  {\n    \"animal\": \"Pigfooted Bandicoot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002886/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002886.do\",\n    \"title\": \"Java NIO\"\n  },\n  {\n    \"animal\": \"Pine Grosbeak\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023005/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023005.do\",\n    \"title\": \"Programming Android\"\n  },\n  {\n    \"animal\": \"Pink-headed Warbler\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920040156/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920040156.do\",\n    \"title\": \"Frontend Architecture for Design Systems\"\n  },\n  {\n    \"animal\": \"Pintail Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025429/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025429.do\",\n    \"title\": \"Getting Started with D3\"\n  },\n  {\n    \"animal\": \"Pionus Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596159801/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596159801.do\",\n    \"title\": \"Building Web Reputation Systems\"\n  },\n  {\n    \"animal\": \"Pipefish, aka Billfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030799/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030799.do\",\n    \"title\": \"Full Stack Web Development with Backbone.js\"\n  },\n  {\n    \"animal\": \"Piping Crow (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028000/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028000.do\",\n    \"title\": \"Understanding and Using C Pointers\"\n  },\n  {\n    \"animal\": \"Pipistrelle Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028970/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028970.do\",\n    \"title\": \"Feedback Control for Computer Systems\"\n  },\n  {\n    \"animal\": \"Plaice fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024149/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024149.do\",\n    \"title\": \"Designing for Scalability with Erlang/OTP\"\n  },\n  {\n    \"animal\": \"Polar Bear Color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034674/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034674.do\",\n    \"title\": \"Information Architecture\"\n  },\n  {\n    \"animal\": \"Pole Shrimp\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007102/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007102.do\",\n    \"title\": \"Eclipse Cookbook\"\n  },\n  {\n    \"animal\": \"Pomeranian dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155902/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155902.do\",\n    \"title\": \"CouchDB: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Poodles no tail (pair)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000196/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000196.do\",\n    \"title\": \"Java Internationalization\"\n  },\n  {\n    \"animal\": \"Porcelain Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007553/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007553.do\",\n    \"title\": \"Designing Embedded Hardware\"\n  },\n  {\n    \"animal\": \"Potto (loris)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006105/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006105.do\",\n    \"title\": \"Managing Projects with GNU Make\"\n  },\n  {\n    \"animal\": \"Prairie dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002367/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002367.do\",\n    \"title\": \"JXTA in a Nutshell\"\n  },\n  {\n    \"animal\": \"Praying Mantis\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006211/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006211.do\",\n    \"title\": \"Oracle Application Server 10g Essentials\"\n  },\n  {\n    \"animal\": \"Proboscis Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926479/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926479.do\",\n    \"title\": \"Perl for Web Site Management\"\n  },\n  {\n    \"animal\": \"Pronghorn\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021667/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021667.do\",\n    \"title\": \"Functional Programming for Java Developers\"\n  },\n  {\n    \"animal\": \"Puffer fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025993/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025993.do\",\n    \"title\": \"Building a Windows IT Infrastructure in the Cloud\"\n  },\n  {\n    \"animal\": \"Puffins, group of three\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925298/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925298.do\",\n    \"title\": \"Virtual Private Networks\"\n  },\n  {\n    \"animal\": \"Pukeko aka Swamphen (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920049487/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920049487.do\",\n    \"title\": \"Prototyping for Designers\"\n  },\n  {\n    \"animal\": \"Puma, aka mountain lion, sitting\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025665/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025665.do\",\n    \"title\": \"OS X Mountain Lion Pocket Guide\"\n  },\n  {\n    \"animal\": \"Pumpkinseed Sunfish, aka Bream\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101770/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101770.do\",\n    \"title\": \"Visual Basic 2005 Cookbook\"\n  },\n  {\n    \"animal\": \"Purple Capped Lory, Head closeup\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927520/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927520.do\",\n    \"title\": \"MCSD in a Nutshell\"\n  },\n  {\n    \"animal\": \"Purple Sandpiper\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920261704/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920261704.do\",\n    \"title\": \"Cloud Native\"\n  },\n  {\n    \"animal\": \"Purple-crowned Fairy Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920224341/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920224341.do\",\n    \"title\": \"D3 for the Impatient\"\n  },\n  {\n    \"animal\": \"Purple-naped Lory\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033226/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033226.do\",\n    \"title\": \"Getting Started with OpenShift\"\n  },\n  {\n    \"animal\": \"Purple-tailed Wood Hoopoe\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041498/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041498.do\",\n    \"title\": \"Automating Junos Administration\"\n  },\n  {\n    \"animal\": \"Pygmy Flying Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034186/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034186.do\",\n    \"title\": \"Web Content Management\"\n  },\n  {\n    \"animal\": \"Pygmy Piculet bird (Ochraceous Piculet)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021063/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021063.do\",\n    \"title\": \"Building Mobile Applications with Java\"\n  },\n  {\n    \"animal\": \"Rabbit\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021421/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021421.do\",\n    \"title\": \"Parallel R\"\n  },\n  {\n    \"animal\": \"Rabbit, crouching\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925397/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925397.do\",\n    \"title\": \"Crossing Platforms A Macintosh/Windows Phrasebook\"\n  },\n  {\n    \"animal\": \"Raccoons, Pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565921177/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565921177.do\",\n    \"title\": \"Applying RCS and SCCS\"\n  },\n  {\n    \"animal\": \"Rainbow Lorikeet in color\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596154790/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596154790.do\",\n    \"title\": \"Effective UI\"\n  },\n  {\n    \"animal\": \"Rat Kangaroo, aka Brush-tailed Bettong\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023999/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023999.do\",\n    \"title\": \"Orchard CMS: Up and Running\"\n  },\n  {\n    \"animal\": \"Red & Yellow Barbet (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920038382/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920038382.do\",\n    \"title\": \"Graphing Data with R\"\n  },\n  {\n    \"animal\": \"Red Ant Male\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514105/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514105.do\",\n    \"title\": \"Oracle PL/SQL Best Practices\"\n  },\n  {\n    \"animal\": \"Red Ant Male\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897215382/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897215382.do\",\n    \"title\": \"Oracle PL/SQL kurz & gut\"\n  },\n  {\n    \"animal\": \"Red Colobus Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002190/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002190.do\",\n    \"title\": \"Perl Graphics Programming\"\n  },\n  {\n    \"animal\": \"Red Comb Seastar (starfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920140610/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920140610.do\",\n    \"title\": \"Asterisk: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Red Deer 2 (doe back)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008406/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008406.do\",\n    \"title\": \"Essential SNMP\"\n  },\n  {\n    \"animal\": \"Red Firefish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001759/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001759.do\",\n    \"title\": \"Java and SOAP\"\n  },\n  {\n    \"animal\": \"Red Fox\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102432/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102432.do\",\n    \"title\": \"Programming Firefox\"\n  },\n  {\n    \"animal\": \"Red Gurnard Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022596/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022596.do\",\n    \"title\": \"Application Security for the Android Platform\"\n  },\n  {\n    \"animal\": \"Red Kite (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920157199/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920157199.do\",\n    \"title\": \"Practical Cloud Security\"\n  },\n  {\n    \"animal\": \"Red Panda\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596153823/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596153823.do\",\n    \"title\": \"Programming the Semantic Web\"\n  },\n  {\n    \"animal\": \"Red Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924529/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924529.do\",\n    \"title\": \"Java RMI\"\n  },\n  {\n    \"animal\": \"Red-Crested Wood Quails\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596154172/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596154172.do\",\n    \"title\": \"Enterprise Development with Flex\"\n  },\n  {\n    \"animal\": \"Red-Shouldered Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920012269/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920012269.do\",\n    \"title\": \"High Performance Drupal\"\n  },\n  {\n    \"animal\": \"Red-billed Blue Magpie (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032564/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032564.do\",\n    \"title\": \"Designing and Developing for Google Glass\"\n  },\n  {\n    \"animal\": \"Red-billed Streamertail Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596523190/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596523190.do\",\n    \"title\": \"iPhone SDK Application Development\"\n  },\n  {\n    \"animal\": \"Red-breasted Goose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050735/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050735.do\",\n    \"title\": \"Mastering CloudForms Automation\"\n  },\n  {\n    \"animal\": \"Red-breasted Merganser (duck)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920042204/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920042204.do\",\n    \"title\": \"The Enterprise Big Data Lake\"\n  },\n  {\n    \"animal\": \"Red-footed falcon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518370/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518370.do\",\n    \"title\": \"AIR for Javascript Developers Pocket Guide\"\n  },\n  {\n    \"animal\": \"Red-headed Woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920216032/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920216032.do\",\n    \"title\": \"Programming PyTorch for Deep Learning\"\n  },\n  {\n    \"animal\": \"Redmullett Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009748/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009748.do\",\n    \"title\": \"XSLT Cookbook\"\n  },\n  {\n    \"animal\": \"Reeves's Pheasant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002527/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002527.do\",\n    \"title\": \"XML Schema\"\n  },\n  {\n    \"animal\": \"Reinwardt's gliding frog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924864/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924864.do\",\n    \"title\": \"Windows 98 in a Nutshell\"\n  },\n  {\n    \"animal\": \"Rhinoceros Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514983/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514983.do\",\n    \"title\": \"Real World Haskell\"\n  },\n  {\n    \"animal\": \"Ribbon Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021278/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021278.do\",\n    \"title\": \"Getting Started with RStudio\"\n  },\n  {\n    \"animal\": \"Right Whale\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596516499/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596516499.do\",\n    \"title\": \"Natural Language Processing with Python\"\n  },\n  {\n    \"animal\": \"Ring Dove\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022251/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022251.do\",\n    \"title\": \"Code Simplicity\"\n  },\n  {\n    \"animal\": \"Ring-tailed Cat (Bassariscus astutus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596157616/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596157616.do\",\n    \"title\": \"HTML & CSS: The Good Parts\"\n  },\n  {\n    \"animal\": \"Ring-tailed Cat (bassaris)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100940/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100940.do\",\n    \"title\": \"Beyond Java\"\n  },\n  {\n    \"animal\": \"Ring-tailed Lemur\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005559/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005559.do\",\n    \"title\": \"AI for Game Developers\"\n  },\n  {\n    \"animal\": \"Ringneck Parakeet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002909/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002909.do\",\n    \"title\": \"802.11 Security\"\n  },\n  {\n    \"animal\": \"River Kingfisher (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033103/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033103.do\",\n    \"title\": \"High Performance Responsive Design\"\n  },\n  {\n    \"animal\": \"Roadrunner, greater\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025719/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025719.do\",\n    \"title\": \"Dart: Up and Running\"\n  },\n  {\n    \"animal\": \"Rock Dove, aka Blue Rock Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565928565/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565928565.do\",\n    \"title\": \"Palm OS Programming\"\n  },\n  {\n    \"animal\": \"Rock Dove, aka Blue Rock Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026358/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026358.do\",\n    \"title\": \"Vagrant: Up and Running\"\n  },\n  {\n    \"animal\": \"Rock Ptarmigan (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920179337/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920179337.do\",\n    \"title\": \"Data Science from Scratch\"\n  },\n  {\n    \"animal\": \"Rock Thrush, Songbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925366/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925366.do\",\n    \"title\": \"Web Caching\"\n  },\n  {\n    \"animal\": \"Roe Deer, trio\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028352/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028352.do\",\n    \"title\": \"Learning R\"\n  },\n  {\n    \"animal\": \"Rook (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920038320/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920038320.do\",\n    \"title\": \"Juniper QFX10000 Series\"\n  },\n  {\n    \"animal\": \"Rooster and Hens\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009083/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009083.do\",\n    \"title\": \"SharePoint User's Guide\"\n  },\n  {\n    \"animal\": \"Rosalie Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001223/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001223.do\",\n    \"title\": \"Oracle DBA Checklists Pocket Reference\"\n  },\n  {\n    \"animal\": \"Rose-Breasted Grosbeak, Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020424/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020424.do\",\n    \"title\": \"Social Network Analysis for Startups\"\n  },\n  {\n    \"animal\": \"Rose-crowned Fruit Dove\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033707/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033707.do\",\n    \"title\": \"Effective Modern C++\"\n  },\n  {\n    \"animal\": \"Rosefish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920223764/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920223764.do\",\n    \"title\": \"Mastering Spark with R\"\n  },\n  {\n    \"animal\": \"Rosy Feather Star\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528461/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528461.do\",\n    \"title\": \"ActionScript 3.0 Design Patterns\"\n  },\n  {\n    \"animal\": \"Rosy Starling, aka Rose-Colored Pastor Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021223/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021223.do\",\n    \"title\": \"APIs: A Strategy Guide\"\n  },\n  {\n    \"animal\": \"Rosy feather star\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018551/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018551.do\",\n    \"title\": \"Asterisk Cookbook\"\n  },\n  {\n    \"animal\": \"Rough-Legged Buzzard/Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020530/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020530.do\",\n    \"title\": \"Building Hypermedia APIs with HTML5 and Node\"\n  },\n  {\n    \"animal\": \"Royal King Flycatcher\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013884/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013884.do\",\n    \"title\": \"Developing Android Applications with Adobe AIR\"\n  },\n  {\n    \"animal\": \"Ruddy Shelduck (duck)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529598/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529598.do\",\n    \"title\": \"Windows Vista Administration: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Ruff Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021117/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021117.do\",\n    \"title\": \"Developing iOS Applications with Flex 4.5\"\n  },\n  {\n    \"animal\": \"Ruffed Grouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032984/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032984.do\",\n    \"title\": \"Customizing Chef\"\n  },\n  {\n    \"animal\": \"Rufous Treepie (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028888/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028888.do\",\n    \"title\": \"Google Compute Engine\"\n  },\n  {\n    \"animal\": \"Rufous-collared Kingfisher bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025252/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025252.do\",\n    \"title\": \"Mobile JavaScript Application Development\"\n  },\n  {\n    \"animal\": \"Rufous-necked Weaver, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920016045/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920016045.do\",\n    \"title\": \"Canvas Pocket Reference\"\n  },\n  {\n    \"animal\": \"Rufous-necked Weaver, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920016182/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920016182.do\",\n    \"title\": \"jQuery Pocket Reference\"\n  },\n  {\n    \"animal\": \"Running Roadrunner\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596159863/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596159863.do\",\n    \"title\": \"iPhone Game Development\"\n  },\n  {\n    \"animal\": \"Russian Desman (semiaquatic mammal)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920053262/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920053262.do\",\n    \"title\": \"Refactoring JavaScript\"\n  },\n  {\n    \"animal\": \"Russian Greyhound, dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021360/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021360.do\",\n    \"title\": \"Using the HTML5 Filesystem API\"\n  },\n  {\n    \"animal\": \"Sabine's_Gull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920203940/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920203940.do\",\n    \"title\": \"Istio: Up and Running\"\n  },\n  {\n    \"animal\": \"Sacred Ibis\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924338/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924338.do\",\n    \"title\": \"Tcl/Tk in a Nutshell\"\n  },\n  {\n    \"animal\": \"Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013471/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013471.do\",\n    \"title\": \"SQL Pocket Guide\"\n  },\n  {\n    \"animal\": \"Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005733/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005733.do\",\n    \"title\": \"SQL Tuning\"\n  },\n  {\n    \"animal\": \"Salim Ali's Fruit Bat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024736/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024736.do\",\n    \"title\": \"Programming Robots with ROS\"\n  },\n  {\n    \"animal\": \"Salmon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030836/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030836.do\",\n    \"title\": \"CSS Fonts\"\n  },\n  {\n    \"animal\": \"Salmon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032250/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032250.do\",\n    \"title\": \"CSS Text\"\n  },\n  {\n    \"animal\": \"Salmon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027614/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027614.do\",\n    \"title\": \"Selectors, Specificity, and the Cascade\"\n  },\n  {\n    \"animal\": \"Salmon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027621/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027621.do\",\n    \"title\": \"Values, Units, and Colors\"\n  },\n  {\n    \"animal\": \"Salmon2\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041658/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041658.do\",\n    \"title\": \"Transitions and Animations in CSS\"\n  },\n  {\n    \"animal\": \"Saltwater Crocodile\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032380/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032380.do\",\n    \"title\": \"Automating Microsoft Azure Infrastructure Services\"\n  },\n  {\n    \"animal\": \"Salvin's Prion (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920038467/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920038467.do\",\n    \"title\": \"Learning Virtual Reality\"\n  },\n  {\n    \"animal\": \"Sambar (deer)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920056669/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920056669.do\",\n    \"title\": \"Modern Java Recipes\"\n  },\n  {\n    \"animal\": \"Sand Dollar\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101428/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101428.do\",\n    \"title\": \"Java Enterprise in a Nutshell\"\n  },\n  {\n    \"animal\": \"Sand Grouse, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020202/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020202.do\",\n    \"title\": \"Python and AWS Cookbook\"\n  },\n  {\n    \"animal\": \"Sand Martin bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023777/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023777.do\",\n    \"title\": \"Cloud Architecture Patterns\"\n  },\n  {\n    \"animal\": \"Sand Star (starfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004323/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004323.do\",\n    \"title\": \"WebLogic: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Sanderling (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021506/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021506.do\",\n    \"title\": \"What Is Node?\"\n  },\n  {\n    \"animal\": \"Sapphirine Gurnard Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596521967/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596521967.do\",\n    \"title\": \"Learning XNA 3.0\"\n  },\n  {\n    \"animal\": \"Sapphirine Gurnard Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013709/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013709.do\",\n    \"title\": \"Learning XNA 4.0\"\n  },\n  {\n    \"animal\": \"Sargassumfish Frogfish, aka Anglerfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030829/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030829.do\",\n    \"title\": \"Learning SPARQL\"\n  },\n  {\n    \"animal\": \"Sawfish Shark, maybe Longtooth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025986/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025986.do\",\n    \"title\": \"Effective Monitoring and Alerting\"\n  },\n  {\n    \"animal\": \"Scallop Shell\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005139/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005139.do\",\n    \"title\": \"PC Hardware in a Nutshell\"\n  },\n  {\n    \"animal\": \"Scarab Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565929487/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565929487.do\",\n    \"title\": \"Oracle SQL*Loader: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Scarlet Minivet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101039/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101039.do\",\n    \"title\": \"PHPUnit Pocket Guide\"\n  },\n  {\n    \"animal\": \"Scops Owl, aka Little Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027690/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027690.do\",\n    \"title\": \"DOM Enlightenment\"\n  },\n  {\n    \"animal\": \"Scorpion Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897217294/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897217294.do\",\n    \"title\": \"Das Xen Kochbuch\"\n  },\n  {\n    \"animal\": \"Scottish Deerhound dog\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020332/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020332.do\",\n    \"title\": \"Concurrent Programming in Mac OS X and iOS\"\n  },\n  {\n    \"animal\": \"Scrawled Butterflyfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026365/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026365.do\",\n    \"title\": \"Parallel and Concurrent Programming in Haskell\"\n  },\n  {\n    \"animal\": \"Sea Lion and Seals\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002701/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002701.do\",\n    \"title\": \"Network Security with OpenSSL\"\n  },\n  {\n    \"animal\": \"Sea Otter\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006396/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006396.do\",\n    \"title\": \"Automating System Administration with Perl\"\n  },\n  {\n    \"animal\": \"Sea Pen Cnidarian\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897218802/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897218802.do\",\n    \"title\": \"Praxisbuch Nagios (German Animal)\"\n  },\n  {\n    \"animal\": \"Sea Pen, Pteroeides spinosum\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920031109/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920031109.do\",\n    \"title\": \"Designing Connected Products\"\n  },\n  {\n    \"animal\": \"Sea Sponge\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000950/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000950.do\",\n    \"title\": \"Programming Web  Services with SOAP\"\n  },\n  {\n    \"animal\": \"Sea shell (Venus comb, aka Thorny Woodcock)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100643/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100643.do\",\n    \"title\": \"ASP.NET 2.0 Cookbook\"\n  },\n  {\n    \"animal\": \"Seahorse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924536/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924536.do\",\n    \"title\": \"Mastering Algorithms with C\"\n  },\n  {\n    \"animal\": \"Secretary Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002633/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002633.do\",\n    \"title\": \"Practical RDF\"\n  },\n  {\n    \"animal\": \"Senegal Lioness\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006358/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006358.do\",\n    \"title\": \"Essential Mac OS X Panther Server Administration\"\n  },\n  {\n    \"animal\": \"Senegal Lioness\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529543/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529543.do\",\n    \"title\": \"Mac OS X Tiger Server Administration\"\n  },\n  {\n    \"animal\": \"Seriema Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020172/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020172.do\",\n    \"title\": \"Developing Android Applications with Flex 4.5\"\n  },\n  {\n    \"animal\": \"Seychelles Blue Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596807252/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596807252.do\",\n    \"title\": \"Programming Entity Framework\"\n  },\n  {\n    \"animal\": \"Sheep, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005672/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005672.do\",\n    \"title\": \"CVS Pocket Reference\"\n  },\n  {\n    \"animal\": \"Sheldrake duck, aka Shelduck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030331/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030331.do\",\n    \"title\": \"Becoming Functional\"\n  },\n  {\n    \"animal\": \"Sheriff man, wearing hat and badge  head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005894/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005894.do\",\n    \"title\": \"Learning Red Hat Enterprise Linux & Fedora\"\n  },\n  {\n    \"animal\": \"Shetland Pony, rearing on hind legs\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518899/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518899.do\",\n    \"title\": \"Apache 2 Pocket Reference\"\n  },\n  {\n    \"animal\": \"Short-Beaked Echidna\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030928/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030928.do\",\n    \"title\": \"Data Push Apps with HTML5 SSE\"\n  },\n  {\n    \"animal\": \"Short-Eared Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596802806/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596802806.do\",\n    \"title\": \"High Performance JavaScript\"\n  },\n  {\n    \"animal\": \"Short-Legged Goose, aka Ross's Goose\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024422/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024422.do\",\n    \"title\": \"Bad Data Handbook\"\n  },\n  {\n    \"animal\": \"Short-Toed Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922518/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922518.do\",\n    \"title\": \"Windows NT in a Nutshell\"\n  },\n  {\n    \"animal\": \"Short-eared Elephant Shrew\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920046189/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920046189.do\",\n    \"title\": \"Concurrency in Go\"\n  },\n  {\n    \"animal\": \"Short-tailed Pangolin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024057/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024057.do\",\n    \"title\": \"Practical Computer Vision with SimpleCV\"\n  },\n  {\n    \"animal\": \"Short-toed Snake Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920041535/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920041535.do\",\n    \"title\": \"Design Leadership\"\n  },\n  {\n    \"animal\": \"Shorthorn Bull\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018254/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018254.do\",\n    \"title\": \"Data Source Handbook\"\n  },\n  {\n    \"animal\": \"Shoveler Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007669/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007669.do\",\n    \"title\": \"Programming Excel with VBA and .NET\"\n  },\n  {\n    \"animal\": \"Shrew, aka Common Tree-Shrew\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920015956/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920015956.do\",\n    \"title\": \"Node: Up and Running\"\n  },\n  {\n    \"animal\": \"Shrimp\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005054/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005054.do\",\n    \"title\": \".NET Framework Essentials\"\n  },\n  {\n    \"animal\": \"Siberian Husky (dog)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920057741/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920057741.do\",\n    \"title\": \"Designing Bots\"\n  },\n  {\n    \"animal\": \"Siberian Jay\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028482/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028482.do\",\n    \"title\": \"JSON at Work\"\n  },\n  {\n    \"animal\": \"Siberian Tiger, prowling\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030409/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030409.do\",\n    \"title\": \"Clojure Inside Out\"\n  },\n  {\n    \"animal\": \"Siberian Tiger, seated\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009144/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009144.do\",\n    \"title\": \"Mac OS X Tiger Pocket Guide\"\n  },\n  {\n    \"animal\": \"Siberian Viper (snake)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920065555/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920065555.do\",\n    \"title\": \"Machine Learning and Security\"\n  },\n  {\n    \"animal\": \"Silkworm Moth\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927537/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927537.do\",\n    \"title\": \"Oracle Net8 Configuration and Troubleshooting\"\n  },\n  {\n    \"animal\": \"Silkworm on a leaf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565921153/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565921153.do\",\n    \"title\": \"PThreads Programming\"\n  },\n  {\n    \"animal\": \"Silver Bass (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009656/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009656.do\",\n    \"title\": \"Learning the bash Shell\"\n  },\n  {\n    \"animal\": \"Silver Moony Fish (psettus rhombeus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021452/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021452.do\",\n    \"title\": \"Building Web Applications with Erlang\"\n  },\n  {\n    \"animal\": \"Siren Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005146/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005146.do\",\n    \"title\": \"ActionScript for Flash MX Pocket Reference\"\n  },\n  {\n    \"animal\": \"Siren Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003968/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003968.do\",\n    \"title\": \"ActionScript for Flash MX: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Sitting Tiger, Head and shoulders\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009434/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009434.do\",\n    \"title\": \"Mac OS X Tiger in a Nutshell\"\n  },\n  {\n    \"animal\": \"Six-banded Armadillo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033714/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033714.do\",\n    \"title\": \"The Architecture of Privacy\"\n  },\n  {\n    \"animal\": \"Sixshafted Bird of Paradise\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596155476/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596155476.do\",\n    \"title\": \"Programming the iPhone User Experience\"\n  },\n  {\n    \"animal\": \"Skua seabird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024835/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024835.do\",\n    \"title\": \"Getting Started with Storm\"\n  },\n  {\n    \"animal\": \"Skunk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023234/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023234.do\",\n    \"title\": \"Hacking and Securing iOS Applications\"\n  },\n  {\n    \"animal\": \"Slender Loris \\\"Awk\\\"\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922259/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922259.do\",\n    \"title\": \"sed & awk\"\n  },\n  {\n    \"animal\": \"Slender Loris \\\"Awk\\\"\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003524/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003524.do\",\n    \"title\": \"sed and awk Pocket Reference\"\n  },\n  {\n    \"animal\": \"Slender-horned Gazelle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002541/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002541.do\",\n    \"title\": \"BGP\"\n  },\n  {\n    \"animal\": \"Slow Loris\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596518172/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596518172.do\",\n    \"title\": \"Facebook Cookbook\"\n  },\n  {\n    \"animal\": \"Smooth-billed Ani (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920044970/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920044970.do\",\n    \"title\": \"Think DSP\"\n  },\n  {\n    \"animal\": \"Smoothhound Shark\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004422/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004422.do\",\n    \"title\": \"Programming .NET Security\"\n  },\n  {\n    \"animal\": \"Snake Knot, vipers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025016/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025016.do\",\n    \"title\": \"Twisted Network Programming Essentials\"\n  },\n  {\n    \"animal\": \"Snakeneck Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003869/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003869.do\",\n    \"title\": \"Learning Visual Basic .NET\"\n  },\n  {\n    \"animal\": \"Snipe Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000783/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000783.do\",\n    \"title\": \"Practical VoIP Using VOCAL\"\n  },\n  {\n    \"animal\": \"Snow Leopard, crouching\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596802738/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596802738.do\",\n    \"title\": \"Mac OS X Snow Leopard Pocket Guide\"\n  },\n  {\n    \"animal\": \"Snow Leopard, sitting\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101060/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101060.do\",\n    \"title\": \"Tomcat: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Snowy Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920215707/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920215707.do\",\n    \"title\": \"Think Julia\"\n  },\n  {\n    \"animal\": \"Softshell Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009342/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009342.do\",\n    \"title\": \"IPv6 Network Administration\"\n  },\n  {\n    \"animal\": \"Soldier with Bayonet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007911/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007911.do\",\n    \"title\": \"Snort Cookbook\"\n  },\n  {\n    \"animal\": \"Soldiers or rangers, with rifles\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007164/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007164.do\",\n    \"title\": \"SELinux\"\n  },\n  {\n    \"animal\": \"Sollarium Shell\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596521004/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596521004.do\",\n    \"title\": \"Mathematica Cookbook\"\n  },\n  {\n    \"animal\": \"Song Thrush (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032366/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032366.do\",\n    \"title\": \"Mobile and Web Messaging\"\n  },\n  {\n    \"animal\": \"Songbirds (Buntings), pair, perched top\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529581/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529581.do\",\n    \"title\": \"SharePoint 2007: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Sonnerats Jungle Fowl (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920167860/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920167860.do\",\n    \"title\": \"Deep Learning for the Life Sciences\"\n  },\n  {\n    \"animal\": \"South African Python\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033431/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033431.do\",\n    \"title\": \"Cython\"\n  },\n  {\n    \"animal\": \"Southern Flying Squirrel, aka Virginian\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034377/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034377.do\",\n    \"title\": \"60 Recipes for Apache CloudStack\"\n  },\n  {\n    \"animal\": \"Southern Ground Hornbill (formerly African)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005467/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005467.do\",\n    \"title\": \"Samba Pocket Reference\"\n  },\n  {\n    \"animal\": \"Southern Ground Hornbill (formerly African)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007690/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007690.do\",\n    \"title\": \"Using Samba\"\n  },\n  {\n    \"animal\": \"Sparrow Hawk\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022343/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022343.do\",\n    \"title\": \"High Performance MySQL\"\n  },\n  {\n    \"animal\": \"Spider Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897216099/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897216099.do\",\n    \"title\": \"Web TV - AV-Streaming im Internet\"\n  },\n  {\n    \"animal\": \"Spider Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004088/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004088.do\",\n    \"title\": \"Java Swing\"\n  },\n  {\n    \"animal\": \"Spiny Lobster (rock lobster)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002244/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002244.do\",\n    \"title\": \"Web Services Essentials\"\n  },\n  {\n    \"animal\": \"Spiny-Tailed Gecko\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024101/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024101.do\",\n    \"title\": \"Getting Started with Windows 8 Apps\"\n  },\n  {\n    \"animal\": \"Spix Macaw (parrot, bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050858/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050858.do\",\n    \"title\": \"Creating Web Animations\"\n  },\n  {\n    \"animal\": \"Splendor Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924383/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924383.do\",\n    \"title\": \"Oracle Scripts\"\n  },\n  {\n    \"animal\": \"Spot-billed Toucanet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100087/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100087.do\",\n    \"title\": \"XSLT 1.0 Pocket Reference\"\n  },\n  {\n    \"animal\": \"Spot-fin Porcupine Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026785/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026785.do\",\n    \"title\": \"Juniper SRX Series\"\n  },\n  {\n    \"animal\": \"Spotted Cuckoo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100674/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100674.do\",\n    \"title\": \"PHP in a Nutshell\"\n  },\n  {\n    \"animal\": \"Spotted Cuscus (Spilocuscus maculatus)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013303/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013303.do\",\n    \"title\": \"YUI 3 Cookbook\"\n  },\n  {\n    \"animal\": \"Spotted Dalmatian\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927179/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927179.do\",\n    \"title\": \"Lotus Domino Administration  in a Nutshell\"\n  },\n  {\n    \"animal\": \"Spotted Eagle (Greater)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924796/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924796.do\",\n    \"title\": \"Programming Internet Email\"\n  },\n  {\n    \"animal\": \"Spotted Goby, fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021124/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021124.do\",\n    \"title\": \"Google AdWords\"\n  },\n  {\n    \"animal\": \"Spotted Hyena\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529314/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529314.do\",\n    \"title\": \"Securing Ajax Applications\"\n  },\n  {\n    \"animal\": \"Spotted Nathura, aka Great Tinamou (mountain hen, Perdiz Grande)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028741/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028741.do\",\n    \"title\": \"User-Centered Design\"\n  },\n  {\n    \"animal\": \"Spotted Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000738/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000738.do\",\n    \"title\": \"Solaris 8 Administrator's Guide\"\n  },\n  {\n    \"animal\": \"Spotted Thick-knee (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920045816/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920045816.do\",\n    \"title\": \"Going GAS\"\n  },\n  {\n    \"animal\": \"Springbok (one)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926165/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926165.do\",\n    \"title\": \"Database Programming with JDBC & Java\"\n  },\n  {\n    \"animal\": \"Springbok (one)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004576/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004576.do\",\n    \"title\": \"JDBC Pocket Reference\"\n  },\n  {\n    \"animal\": \"Springer Spaniel (dog)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020301/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020301.do\",\n    \"title\": \"Drupal Development Tricks for Designers\"\n  },\n  {\n    \"animal\": \"Springhaas, aka spring hare\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920027072/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920027072.do\",\n    \"title\": \"Python Cookbook\"\n  },\n  {\n    \"animal\": \"Springhare, or Western Brush Wallaby\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020752/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020752.do\",\n    \"title\": \"Creating HTML5 Animations with Flash and Wallaby\"\n  },\n  {\n    \"animal\": \"Squat Lobster\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920052265/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920052265.do\",\n    \"title\": \"Zero Trust Networks\"\n  },\n  {\n    \"animal\": \"Squirrel Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021193/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021193.do\",\n    \"title\": \"Beginning NFC\"\n  },\n  {\n    \"animal\": \"Squirrel Treefrog aka Treetoad\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020585/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020585.do\",\n    \"title\": \"jQuery Mobile\"\n  },\n  {\n    \"animal\": \"Stag Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926745/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926745.do\",\n    \"title\": \"Oracle PL/SQL Programming: A Developer's Workbook\"\n  },\n  {\n    \"animal\": \"Staghound\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001735/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001735.do\",\n    \"title\": \"Perl Best Practices\"\n  },\n  {\n    \"animal\": \"Star-nosed Mole\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003159/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003159.do\",\n    \"title\": \"C# Essentials\"\n  },\n  {\n    \"animal\": \"Steppe Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920050964/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920050964.do\",\n    \"title\": \"The SEO Battlefield\"\n  },\n  {\n    \"animal\": \"Stickleback Fish, Fifteen-Spine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020806/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020806.do\",\n    \"title\": \"Getting Started with GEO, CouchDB, and Node.js\"\n  },\n  {\n    \"animal\": \"Stickleback Fish, Fifteen-Spine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924666/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924666.do\",\n    \"title\": \"Windows 2000 Performance  Guide\"\n  },\n  {\n    \"animal\": \"Stingray\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596513979/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596513979.do\",\n    \"title\": \"Learning ASP.NET 2.0 with AJAX\"\n  },\n  {\n    \"animal\": \"Stormy Petrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920015116/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920015116.do\",\n    \"title\": \"Programming HTML5 Applications\"\n  },\n  {\n    \"animal\": \"Straw-necked Ibis (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033868/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033868.do\",\n    \"title\": \"Modern PHP\"\n  },\n  {\n    \"animal\": \"Striped Burrfish aka Swell Burrfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030676/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030676.do\",\n    \"title\": \"Designing for Emerging Technologies\"\n  },\n  {\n    \"animal\": \"Striped Hyena\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004002/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004002.do\",\n    \"title\": \"Mac OS X for Java Geeks\"\n  },\n  {\n    \"animal\": \"Striped Red Mullet (fish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030720/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030720.do\",\n    \"title\": \"Think Bayes\"\n  },\n  {\n    \"animal\": \"Sturgeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596515201/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596515201.do\",\n    \"title\": \"Enterprise Rails\"\n  },\n  {\n    \"animal\": \"Suffolk Punch (Horse)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920039761/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920039761.do\",\n    \"title\": \"Database Reliability Engineering\"\n  },\n  {\n    \"animal\": \"Sugar Squirrel, Biak Glider\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920025955/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920025955.do\",\n    \"title\": \"Web Performance Daybook Volume 2 \"\n  },\n  {\n    \"animal\": \"Sumatran Tiger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024798/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024798.do\",\n    \"title\": \"Berglund and McCullough on Mastering Grails 101\"\n  },\n  {\n    \"animal\": \"Sumatran Tiger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030393/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030393.do\",\n    \"title\": \"Functional Thinking\"\n  },\n  {\n    \"animal\": \"Sumatran Tiger\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009137/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009137.do\",\n    \"title\": \"Running Mac OS X Tiger\"\n  },\n  {\n    \"animal\": \"Sun Conure (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034445/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034445.do\",\n    \"title\": \"Getting Started with OpenBTS\"\n  },\n  {\n    \"animal\": \"Sun Star Radiata (starfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023708/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023708.do\",\n    \"title\": \"20 Recipes for Programming PhoneGap\"\n  },\n  {\n    \"animal\": \"Sunda Slow Loris\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920039747/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920039747.do\",\n    \"title\": \"Calm Technology\"\n  },\n  {\n    \"animal\": \"Superb Lyre Bird, menura superba (antique name)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022503/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022503.do\",\n    \"title\": \"Making Musical Apps\"\n  },\n  {\n    \"animal\": \"Superb Starling (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920035848/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920035848.do\",\n    \"title\": \"Creating a Data-Driven Organization\"\n  },\n  {\n    \"animal\": \"Sword-billed Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001728/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001728.do\",\n    \"title\": \"Web Performance Tuning\"\n  },\n  {\n    \"animal\": \"Swordfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002848/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002848.do\",\n    \"title\": \"System Performance Tuning\"\n  },\n  {\n    \"animal\": \"Tadpole of a Greenfrog (sketch)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000806/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000806.do\",\n    \"title\": \"Beginning Perl for Bioinformatics\"\n  },\n  {\n    \"animal\": \"Taguan Flying Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005221/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005221.do\",\n    \"title\": \"Java Database Best Practices\"\n  },\n  {\n    \"animal\": \"Tailorbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020523/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020523.do\",\n    \"title\": \"Packet Guide to Routing and Switching\"\n  },\n  {\n    \"animal\": \"Tamandua Collared Anteater\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002732/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002732.do\",\n    \"title\": \"Access Database Design & Programming\"\n  },\n  {\n    \"animal\": \"Tamandua, aka Lesser Anteater\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006785/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006785.do\",\n    \"title\": \"Access Cookbook\"\n  },\n  {\n    \"animal\": \"Tamandua, aka Lesser Anteater\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021483/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021483.do\",\n    \"title\": \"PDF Explained\"\n  },\n  {\n    \"animal\": \"Tarpan Horses, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021322/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021322.do\",\n    \"title\": \"Learning Rails 3\"\n  },\n  {\n    \"animal\": \"Tarpan Horses, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002169/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002169.do\",\n    \"title\": \"XML Publishing with AxKit\"\n  },\n  {\n    \"animal\": \"Tarsier, full-body, standing on hind feet, b/w engraving\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529833/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529833.do\",\n    \"title\": \"Learning the vi and Vim Editors\"\n  },\n  {\n    \"animal\": \"Tarsier, full-body, standing on hind feet, b/w engraving\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100292/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100292.do\",\n    \"title\": \"Unix in a Nutshell\"\n  },\n  {\n    \"animal\": \"Tarsier, full-body, standing on hind feet, b/w engraving\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920010913/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920010913.do\",\n    \"title\": \"vi and Vim Editors Pocket Reference\"\n  },\n  {\n    \"animal\": \"Tawny Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926288/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926288.do\",\n    \"title\": \"qmail\"\n  },\n  {\n    \"animal\": \"Tegu Lizard\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004750/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004750.do\",\n    \"title\": \"RTF Pocket Guide\"\n  },\n  {\n    \"animal\": \"Tengmalm's Owl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920015963/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920015963.do\",\n    \"title\": \"Junos Enterprise Routing\"\n  },\n  {\n    \"animal\": \"Tenrec\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021674/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021674.do\",\n    \"title\": \"Deploying OpenStack\"\n  },\n  {\n    \"animal\": \"Thick-Billed Coot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022459/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022459.do\",\n    \"title\": \"ActionScript Developer's Guide to PureMVC\"\n  },\n  {\n    \"animal\": \"Thick-knee bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022442/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022442.do\",\n    \"title\": \"What is EPUB 3?\"\n  },\n  {\n    \"animal\": \"Thick-tailed Greater Galago (Garnett's)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029687/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029687.do\",\n    \"title\": \"Functional Thinking\"\n  },\n  {\n    \"animal\": \"Thirteen-lined Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002985/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002985.do\",\n    \"title\": \"C++ In a Nutshell\"\n  },\n  {\n    \"animal\": \"Thirteen-lined Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004965/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004965.do\",\n    \"title\": \"C++ Pocket Reference\"\n  },\n  {\n    \"animal\": \"Thirteen-lined Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897212626/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897212626.do\",\n    \"title\": \"C++ kurz & gut\"\n  },\n  {\n    \"animal\": \"Thirteen-lined Ground Squirrel\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004194/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004194.do\",\n    \"title\": \"Practical C++ Programming\"\n  },\n  {\n    \"animal\": \"Thornback Crab\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023074/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023074.do\",\n    \"title\": \"Statistics in a Nutshell\"\n  },\n  {\n    \"animal\": \"Thread-winged Lacewing, aka Antlion\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002107/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002107.do\",\n    \"title\": \"Perl for Oracle DBAs\"\n  },\n  {\n    \"animal\": \"Three-toed Woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565922594/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565922594.do\",\n    \"title\": \"Managing Mailing Lists\"\n  },\n  {\n    \"animal\": \"Tiger Butterfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596517748/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596517748.do\",\n    \"title\": \"JavaScript: The Good Parts\"\n  },\n  {\n    \"animal\": \"Tiger Salamander\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920045960/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920045960.do\",\n    \"title\": \"Identity, Authentication, and Access Management in OpenStack\"\n  },\n  {\n    \"animal\": \"Tiger sitting or lying\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001575/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001575.do\",\n    \"title\": \"Java Security\"\n  },\n  {\n    \"animal\": \"Tim O'Reilly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020080/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020080.do\",\n    \"title\": \"Tim O'Reilly in a Nutshell\"\n  },\n  {\n    \"animal\": \"Tits, (birds) various species: Crested, Great, etc.\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920052074/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920052074.do\",\n    \"title\": \"Building Microservices with ASP.NET Core\"\n  },\n  {\n    \"animal\": \"Toad\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009717/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009717.do\",\n    \"title\": \"Toad Pocket Reference for Oracle\"\n  },\n  {\n    \"animal\": \"Toucan\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000646/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000646.do\",\n    \"title\": \"Programming with Qt\"\n  },\n  {\n    \"animal\": \"Toy Duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037880/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037880.do\",\n    \"title\": \"Arduino: A Technical Reference\"\n  },\n  {\n    \"animal\": \"Toy Rabbit, a mechanical toy\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920013310/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920013310.do\",\n    \"title\": \"Processing and Arduino in Tandem\"\n  },\n  {\n    \"animal\": \"Toy Rabbit, a mechanical toy\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018377/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018377.do\",\n    \"title\": \"Processing and Arduino in Tandem: Audio Visualizer\"\n  },\n  {\n    \"animal\": \"Toy Rabbit, a mechanical toy\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018353/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018353.do\",\n    \"title\": \"Processing and Arduino in Tandem: Video Mixer\"\n  },\n  {\n    \"animal\": \"Tragopan, aka horned pheasant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007713/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007713.do\",\n    \"title\": \"Jakarta Struts Cookbook\"\n  },\n  {\n    \"animal\": \"Tree\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002657/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002657.do\",\n    \"title\": \"MySQL Reference Manual\"\n  },\n  {\n    \"animal\": \"Tree\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008260/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008260.do\",\n    \"title\": \"OpenOffice.org Writer\"\n  },\n  {\n    \"animal\": \"Tree\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005160/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005160.do\",\n    \"title\": \"The Complete FreeBSD\"\n  },\n  {\n    \"animal\": \"Tree\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596529185/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596529185.do\",\n    \"title\": \"Using Moodle\"\n  },\n  {\n    \"animal\": \"Tree Porcupine, perching on top\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925106/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925106.do\",\n    \"title\": \"Managing NFS and NIS\"\n  },\n  {\n    \"animal\": \"Tree Swift\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020394/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020394.do\",\n    \"title\": \"Just Spring\"\n  },\n  {\n    \"animal\": \"Tree Swift\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565927568/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565927568.do\",\n    \"title\": \"Transact-SQL Cookbook\"\n  },\n  {\n    \"animal\": \"Trigger Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920018308/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920018308.do\",\n    \"title\": \"Scaling MongoDB\"\n  },\n  {\n    \"animal\": \"Trojan Horse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565926820/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565926820.do\",\n    \"title\": \"Malicious Mobile Code\"\n  },\n  {\n    \"animal\": \"Trunk Fish, species unknown, possibly Aracana Aurita\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023258/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023258.do\",\n    \"title\": \"Node for Front-End Developers\"\n  },\n  {\n    \"animal\": \"Tsetse Fly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005177/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005177.do\",\n    \"title\": \"Oracle Data Dictionary Pocket Reference\"\n  },\n  {\n    \"animal\": \"Tubularia Coronata (Radiata)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920029199/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920029199.do\",\n    \"title\": \"Learning Responsive Web Design\"\n  },\n  {\n    \"animal\": \"Tufted Coquette Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033578/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033578.do\",\n    \"title\": \"Designing for Performance\"\n  },\n  {\n    \"animal\": \"Tule Elk \",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920118428/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920118428.do\",\n    \"title\": \"Building Products for the Enterprise\"\n  },\n  {\n    \"animal\": \"Turnstone, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023159/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023159.do\",\n    \"title\": \"jQuery UI\"\n  },\n  {\n    \"animal\": \"Turtle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923218/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923218.do\",\n    \"title\": \"Using and Managing PPP\"\n  },\n  {\n    \"animal\": \"Turtle Dove\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101008/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101008.do\",\n    \"title\": \"Zero Configuration Networking: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Two Brown Bears, mother with cub\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005047/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005047.do\",\n    \"title\": \"Programming Flash Communication Server\"\n  },\n  {\n    \"animal\": \"Two-Winged Flying Fish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033929/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033929.do\",\n    \"title\": \"Becoming a Better Programmer\"\n  },\n  {\n    \"animal\": \"Umbrellabird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783868991536/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783868991536.do\",\n    \"title\": \"Online-Shops mit OXID eShop\"\n  },\n  {\n    \"animal\": \"Vervain Hummingbird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032502/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032502.do\",\n    \"title\": \"Lightweight Django\"\n  },\n  {\n    \"animal\": \"Victorian Crowned Pigeon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920077916/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920077916.do\",\n    \"title\": \"Design for How People Think\"\n  },\n  {\n    \"animal\": \"Violet Crossfish (starfish)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920146667/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920146667.do\",\n    \"title\": \"Managing Kubernetes\"\n  },\n  {\n    \"animal\": \"Violet Ground Beetle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920037019/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920037019.do\",\n    \"title\": \"Understanding Industrial Design\"\n  },\n  {\n    \"animal\": \"Vulturine Guineafowl\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596003692/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596003692.do\",\n    \"title\": \"XForms Essentials\"\n  },\n  {\n    \"animal\": \"Wahlberg's Honeyguide (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032397/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032397.do\",\n    \"title\": \"Learning Chef\"\n  },\n  {\n    \"animal\": \"Wall Gecko (Cape Tarentola)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897216594/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897216594.do\",\n    \"title\": \"Webentwicklung mit CakePHP\"\n  },\n  {\n    \"animal\": \"Wallaby, with baby joey in pouch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596158033/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596158033.do\",\n    \"title\": \"Enterprise JavaBeans 3.1\"\n  },\n  {\n    \"animal\": \"Wallaby, with baby joey in pouch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004187/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004187.do\",\n    \"title\": \"WebSphere 4.0 AEs Workbook for Enterprise Java Beans\"\n  },\n  {\n    \"animal\": \"Wallaby, with baby joey in pouch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004170/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004170.do\",\n    \"title\": \"Weblogic Server 6.1 Workbook for Enterprise Java Beans\"\n  },\n  {\n    \"animal\": \"Wallachian Sheep aka Cretan sheep\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920026013/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920026013.do\",\n    \"title\": \"Node.js for PHP Developers\"\n  },\n  {\n    \"animal\": \"Wallcreeper bird 2, pointed down\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565924017/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565924017.do\",\n    \"title\": \"Transact-SQL Programming\"\n  },\n  {\n    \"animal\": \"Wandering Albatross\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006853/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006853.do\",\n    \"title\": \"Securing Windows Server 2003\"\n  },\n  {\n    \"animal\": \"Wandering Albatross\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514112/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514112.do\",\n    \"title\": \"Windows Server 2008: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Wandering Chaetodon, aka Crisscross butterflyFish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596521073/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596521073.do\",\n    \"title\": \"Learning C# 3.0\"\n  },\n  {\n    \"animal\": \"Warrior on Horseback\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021490/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021490.do\",\n    \"title\": \"Inside Cyber Warfare\"\n  },\n  {\n    \"animal\": \"Warriors, Group\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005450/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005450.do\",\n    \"title\": \"Security Warrior\"\n  },\n  {\n    \"animal\": \"Wasp Nest\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008994/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008994.do\",\n    \"title\": \"Oracle Utilities Pocket Reference\"\n  },\n  {\n    \"animal\": \"Water Buffalo\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102296/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102296.do\",\n    \"title\": \"MCSE Core Elective Exams in a Nutshell\"\n  },\n  {\n    \"animal\": \"Water Rail (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920034292/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920034292.do\",\n    \"title\": \"Living Clojure\"\n  },\n  {\n    \"animal\": \"Water Spider\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596100490/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596100490.do\",\n    \"title\": \"Oracle DBA Pocket Guide\"\n  },\n  {\n    \"animal\": \"Waterbuck Antelope\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920017547/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920017547.do\",\n    \"title\": \"Programming Google App Engine\"\n  },\n  {\n    \"animal\": \"Western Bluebird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920033561/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920033561.do\",\n    \"title\": \"Discussing Design\"\n  },\n  {\n    \"animal\": \"Western Rosella Parakeet\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920137733/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920137733.do\",\n    \"title\": \"Fundamentals of Data Visualization\"\n  },\n  {\n    \"animal\": \"Whale Shark\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596000455/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596000455.do\",\n    \"title\": \"Web Security, Privacy & Commerce\"\n  },\n  {\n    \"animal\": \"Whip-Poor-Will, bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920019664/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920019664.do\",\n    \"title\": \"Sinatra: Up and Running\"\n  },\n  {\n    \"animal\": \"White Pelican\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596101237/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596101237.do\",\n    \"title\": \"Learning Windows Server 2003\"\n  },\n  {\n    \"animal\": \"White Pelican\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596004040/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596004040.do\",\n    \"title\": \"Windows Server 2003 in a Nutshell\"\n  },\n  {\n    \"animal\": \"White Rabbit\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596527501/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596527501.do\",\n    \"title\": \"Java I/O\"\n  },\n  {\n    \"animal\": \"White-Handed Gibbons\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920057895/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920057895.do\",\n    \"title\": \"Collaborative Product Design\"\n  },\n  {\n    \"animal\": \"White-Necked Raven\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002305/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002305.do\",\n    \"title\": \"DNS on Windows 2000\"\n  },\n  {\n    \"animal\": \"White-Necked Raven\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005627/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005627.do\",\n    \"title\": \"DNS on Windows Server 2003\"\n  },\n  {\n    \"animal\": \"White-Throated Dipper Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596519193/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596519193.do\",\n    \"title\": \"FBML Essentials\"\n  },\n  {\n    \"animal\": \"White-bellied Parrot (bird) aka Green-thighed Parrot\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920058687/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920058687.do\",\n    \"title\": \"DevOps with OpenShift\"\n  },\n  {\n    \"animal\": \"White-breasted Nuthatch\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596154622/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596154622.do\",\n    \"title\": \"Twitter API: Up and Running\"\n  },\n  {\n    \"animal\": \"White-cheeked Turaco\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596523107/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596523107.do\",\n    \"title\": \"Data-Driven Services with Silverlight 2\"\n  },\n  {\n    \"animal\": \"White-crested Helmetshrike (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032151/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032151.do\",\n    \"title\": \"JavaScript with Promises\"\n  },\n  {\n    \"animal\": \"White-sided Dolphin\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920223788/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920223788.do\",\n    \"title\": \"Kubernetes: Up and Running\"\n  },\n  {\n    \"animal\": \"White-tailed Eagle\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925670/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925670.do\",\n    \"title\": \"Writing Apache Modules with Perl and C\"\n  },\n  {\n    \"animal\": \"Whitebar Surgeonfish\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920020134/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920020134.do\",\n    \"title\": \"Codermetrics\"\n  },\n  {\n    \"animal\": \"Widow Bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024460/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024460.do\",\n    \"title\": \"Resource-Oriented Computing with NetKernel\"\n  },\n  {\n    \"animal\": \"Wild Canary\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514808/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514808.do\",\n    \"title\": \"Intel Threading Building Blocks\"\n  },\n  {\n    \"animal\": \"Wild Goat, aka Iberian Ibex (Capra pyrenaica)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596520717/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596520717.do\",\n    \"title\": \"Rails Pocket Reference\"\n  },\n  {\n    \"animal\": \"Wild Goat, aka Iberian Ibex (Capra pyrenaica)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596522018/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596522018.do\",\n    \"title\": \"Rails: Up and Running\"\n  },\n  {\n    \"animal\": \"Wild Goat, aka Iberian Ibex (Capra pyrenaica), Head\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002145/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002145.do\",\n    \"title\": \"Ruby in a Nutshell\"\n  },\n  {\n    \"animal\": \"Wildcat, aka Jungle Cat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007300/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007300.do\",\n    \"title\": \"Killer Game Programming in Java\"\n  },\n  {\n    \"animal\": \"Willow Ptarmigan, aka Willow Grouse\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920024514/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920024514.do\",\n    \"title\": \"Accessibility Handbook\"\n  },\n  {\n    \"animal\": \"Winged Ant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596009557/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596009557.do\",\n    \"title\": \"New Features in Oracle 9i\"\n  },\n  {\n    \"animal\": \"Winged Ant\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007706/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007706.do\",\n    \"title\": \"Oracle Initialization Parameters Pocket Reference\"\n  },\n  {\n    \"animal\": \"Wolf\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565923980/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565923980.do\",\n    \"title\": \"Mastering Algorithms with Perl\"\n  },\n  {\n    \"animal\": \"Wolverine\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007836/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007836.do\",\n    \"title\": \"Time Management for System Administrators\"\n  },\n  {\n    \"animal\": \"Woman Blacksmith\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596102487/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596102487.do\",\n    \"title\": \"Linux Networking Cookbook\"\n  },\n  {\n    \"animal\": \"Woman on Circus Trapeze\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007942/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007942.do\",\n    \"title\": \"Network Security Tools\"\n  },\n  {\n    \"animal\": \"Wombat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514075/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514075.do\",\n    \"title\": \"Essential SharePoint 2007\"\n  },\n  {\n    \"animal\": \"Wombat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596514075/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596514075.do\",\n    \"title\": \"Essential SharePoint 2007\"\n  },\n  {\n    \"animal\": \"Wombat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920149033/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920149033.do\",\n    \"title\": \"Hands-On Unsupervised Learning Using Python\"\n  },\n  {\n    \"animal\": \"Women Armament Workers\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596006402/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596006402.do\",\n    \"title\": \"Linux Cookbook\"\n  },\n  {\n    \"animal\": \"Women in Masquerade, pair\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596008789/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596008789.do\",\n    \"title\": \"Digital Identity\"\n  },\n  {\n    \"animal\": \"Wood Duck, aka Carolina duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596805012/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596805012.do\",\n    \"title\": \"DocBook 5: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Wood Duck, aka Carolina duck\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9781565925809/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9781565925809.do\",\n    \"title\": \"DocBook: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Wood Rat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920028154/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920028154.do\",\n    \"title\": \"Learning Python\"\n  },\n  {\n    \"animal\": \"Woolly Monkey\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596528386/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596528386.do\",\n    \"title\": \"Ajax: The Definitive Guide\"\n  },\n  {\n    \"animal\": \"Wreathed Hornbill (bird)\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032823/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032823.do\",\n    \"title\": \"Data Science at the Command Line\"\n  },\n  {\n    \"animal\": \"Wryneck - Old World woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783955613709/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783955613709.do\",\n    \"title\": \"GIMP kurz & gut\"\n  },\n  {\n    \"animal\": \"Wryneck - Old World woodpecker\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9783897215535/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9783897215535.do\",\n    \"title\": \"Gimp kurz & gut\"\n  },\n  {\n    \"animal\": \"Yak\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920032304/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920032304.do\",\n    \"title\": \"Accumulo\"\n  },\n  {\n    \"animal\": \"Yellow Baboon\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596007430/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596007430.do\",\n    \"title\": \"JUnit Pocket Guide\"\n  },\n  {\n    \"animal\": \"Yellowjacket\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596005276/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596005276.do\",\n    \"title\": \"Optimizing Oracle Performance\"\n  },\n  {\n    \"animal\": \"Zanick aka Meerkat\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920022473/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920022473.do\",\n    \"title\": \"HTML5 for Publishers\"\n  },\n  {\n    \"animal\": \"Zebra\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596510329/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596510329.do\",\n    \"title\": \"Advanced Rails\"\n  },\n  {\n    \"animal\": \"Zebra Butterfly\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596001803/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596001803.do\",\n    \"title\": \"Learning Oracle PL/SQL\"\n  },\n  {\n    \"animal\": \"Zebu Cow\",\n    \"cover_src\": \"https://covers.oreilly.com/images/9780596002756/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/9780596002756.do\",\n    \"title\": \"IP Routing\"\n  },\n  {\n    \"animal\": \"hornets, wasps and hive\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920023555/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920023555.do\",\n    \"title\": \"Programming Hive\"\n  },\n  {\n    \"animal\": \"sea anemone\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920030485/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920030485.do\",\n    \"title\": \"WebSocket\"\n  },\n  {\n    \"animal\": \"white wagtail bird\",\n    \"cover_src\": \"https://covers.oreilly.com/images/0636920021759/cat.gif\",\n    \"link\": \"https://shop.oreilly.com/product/0636920021759.do\",\n    \"title\": \"Developing Enterprise iOS Applications\"\n  }\n]\n"
  },
  {
    "path": "resources/pipreqs_mapping.txt",
    "content": "AG_fft_tools:agpy\nANSI:pexpect\nAdafruit:Adafruit_Libraries\nApp:Zope2\nAsterisk:py_Asterisk\nBB_jekyll_hook:bitbucket_jekyll_hook\nBanzai:Banzai_NGS\nBeautifulSoupTests:BeautifulSoup\nBioSQL:biopython\nBuildbotStatusShields:BuildbotEightStatusShields\nComputedAttribute:ExtensionClass\nCrypto:pycryptodome\nCryptodome:pycryptodomex\nFSM:pexpect\nFiftyOneDegrees:51degrees_mobile_detector_v3_wrapper\nGeoBaseMain:GeoBasesDev\nGeoBases:GeoBasesDev\nGlobals:Zope2\nHelpSys:Zope2\nIPython:ipython\nKittens:astro_kittens\nLevenshtein:python_Levenshtein\nLifetime:Zope2\nMethodObject:ExtensionClass\nOFS:Zope2\nOpenGL:PyOpenGL\nOpenSSL:pyOpenSSL\nPIL:Pillow\nProducts:Zope2\nPyWCSTools:astLib\nPyxides:astro_pyxis\nQtCore:PySide\nS3:s3cmd\nSCons:pystick\nShared:Zope2\nSignals:Zope2\nStemmer:PyStemmer\nTesting:Zope2\nTopZooTools:topzootools\nTreeDisplay:DocumentTemplate\nWorkingWithDocumentConversion:aspose_pdf_java_for_python\nZPublisher:Zope2\nZServer:Zope2\nZTUtils:Zope2\naadb:auto_adjust_display_brightness\nabakaffe:abakaffe_cli\nabiosgaming:abiosgaming.py\nabiquo:abiquo_api\nabl:abl.cssprocessor\nabl:abl.robot\nabl:abl.util\nabl:abl.vpath\nabo:abo_generator\nabris_transform:abris\nabstract:abstract.jwrotator\nabu:abu.admin\nac_flask:AC_Flask_HipChat\nacg:anikom15\nacme:acme.dchat\nacme:acme.hello\nacted:acted.projects\naction:ActionServer\nactionbar:actionbar.panel\nactivehomed:afn\nactivepapers:ActivePapers.Py\naddress_book:address_book_lansry\nadi:adi.commons\nadi:adi.devgen\nadi:adi.fullscreen\nadi:adi.init\nadi:adi.playlist\nadi:adi.samplecontent\nadi:adi.slickstyle\nadi:adi.suite\nadi:adi.trash\nadict:aDict2\naditam:aditam.agent\naditam:aditam.core\nadiumsh:adium_sh\nadjector:AdjectorClient\nadjector:AdjectorTracPlugin\nadkit:Banner_Ad_Toolkit\nadmin_tools:django_admin_tools\nadminishcategories:adminish_categories\nadminsortable:django_admin_sortable\nadspygoogle:adspygoogle.adwords\nadvancedcaching:agtl\nadytum:Adytum_PyMonitor\naffinitic:affinitic.docpyflakes\naffinitic:affinitic.recipe.fakezope2eggs\naffinitic:affinitic.simplecookiecuttr\naffinitic:affinitic.verifyinterface\naffinitic:affinitic.zamqp\nafpy:afpy.xap\nagatesql:agate_sql\nageliaco:ageliaco.recipe.csvconfig\nagent_http:agent.http\nagora:Agora_Client\nagora:Agora_Fountain\nagora:Agora_Fragment\nagora:Agora_Planner\nagora:Agora_Service_Provider\nagoraplex:agoraplex.themes.sphinx\nagsci:agsci.blognewsletter\nagx:agx.core\nagx:agx.dev\nagx:agx.generator.buildout\nagx:agx.generator.dexterity\nagx:agx.generator.generator\nagx:agx.generator.plone\nagx:agx.generator.pyegg\nagx:agx.generator.sql\nagx:agx.generator.uml\nagx:agx.generator.zca\nagx:agx.transform.uml2fs\nagx:agx.transform.xmi2uml\naimes:aimes.bundle\naimes:aimes.skeleton\naio:aio.app\naio:aio.config\naio:aio.core\naio:aio.signals\naiohs2:aio_hs2\naioroutes:aio_routes\naios3:aio_s3\nairbrake:airbrake_flask\nairship:airship_icloud\nairship:airship_steamcloud\nakamai:edgegrid_python\nalation:alation_api\nalba_client:alba_client_python\nalburnum:alburnum_maas_client\nalchemist:alchemist.audit\nalchemist:alchemist.security\nalchemist:alchemist.traversal\nalchemist:alchemist.ui\nalchemyapi:alchemyapi_python\nalerta:alerta_server\nalexandria_upload:Alexandria_Upload_Utils\nalibaba:alibaba_python_sdk\naliyun:aliyun_python_sdk\naliyuncli:alicloudcli\naliyunsdkacs:aliyun_python_sdk_acs\naliyunsdkbatchcompute:aliyun_python_sdk_batchcompute\naliyunsdkbsn:aliyun_python_sdk_bsn\naliyunsdkbss:aliyun_python_sdk_bss\naliyunsdkcdn:aliyun_python_sdk_cdn\naliyunsdkcms:aliyun_python_sdk_cms\naliyunsdkcore:aliyun_python_sdk_core\naliyunsdkcrm:aliyun_python_sdk_crm\naliyunsdkcs:aliyun_python_sdk_cs\naliyunsdkdrds:aliyun_python_sdk_drds\naliyunsdkecs:aliyun_python_sdk_ecs\naliyunsdkess:aliyun_python_sdk_ess\naliyunsdkft:aliyun_python_sdk_ft\naliyunsdkmts:aliyun_python_sdk_mts\naliyunsdkocs:aliyun_python_sdk_ocs\naliyunsdkoms:aliyun_python_sdk_oms\naliyunsdkossadmin:aliyun_python_sdk_ossadmin\naliyunsdkr-kvstore:aliyun_python_sdk_r_kvstore\naliyunsdkram:aliyun_python_sdk_ram\naliyunsdkrds:aliyun_python_sdk_rds\naliyunsdkrisk:aliyun_python_sdk_risk\naliyunsdkros:aliyun_python_sdk_ros\naliyunsdkslb:aliyun_python_sdk_slb\naliyunsdksts:aliyun_python_sdk_sts\naliyunsdkubsms:aliyun_python_sdk_ubsms\naliyunsdkyundun:aliyun_python_sdk_yundun\nallattachments:AllAttachmentsMacro\nallocine:allocine_wrapper\nallowedsites:django_allowedsites\nalm:alm.solrindex\naloft:aloft.py\nalpacalib:alpaca\nalphabetic:alphabetic_simple\nalphasms:alphasms_client\naltered:altered.states\nalterootheme:alterootheme.busycity\nalterootheme:alterootheme.intensesimplicity\nalterootheme:alterootheme.lazydays\nalurinium:alurinium_image_processing\nalxlib:alx\namara3:amara3_iri\namara3:amara3_xml\namazon:AmazonAPIWrapper\namazon:python_amazon_simple_product_api\nambikesh1349-1:ambikesh1349_1\nambilight:AmbilightParty\namifs:amifs_core\namiorganizer:ami_organizer\namitu:amitu.lipy\namitu:amitu_putils\namitu:amitu_websocket_client\namitu:amitu_zutils\namltlearn:AMLT_learn\namocrm:amocrm_api\namqpdispatcher:amqp_dispatcher\namqpstorm:AMQP_Storm\nanalytics:analytics_python\nanalyzedir:AnalyzeDirectory\nancientsolutions:ancientsolutions_crypttools\nanderson_paginator:anderson.paginator\nandroid_clean_app:android_resource_remover\nanel_power_control:AnelPowerControl\nangus:angus_sdk_python\nannalist_root:Annalist\nannogesiclib:ANNOgesic\nansible-role-apply:ansible_role_apply\nansibledebugger:ansible_playbook_debugger\nansibledocgen:ansible_docgen\nansibleflow:ansible_flow\nansibleinventorygrapher:ansible_inventory_grapher\nansiblelint:ansible_lint\nansiblerolesgraph:ansible_roles_graph\nansibletools:ansible_tools\nanthill:anthill.exampletheme\nanthill:anthill.skinner\nanthill:anthill.tal.macrorenderer\nanthrax:AnthraxDojoFrontend\nanthrax:AnthraxHTMLInput\nanthrax:AnthraxImage\nantisphinx:antiweb\nantispoofing:antispoofing.evaluation\nantlr4:antlr4_python2_runtime\nantlr4:antlr4_python3_runtime\nantlr4:antlr4_python_alt\nanybox:anybox.buildbot.openerp\nanybox:anybox.nose.odoo\nanybox:anybox.paster.odoo\nanybox:anybox.paster.openerp\nanybox:anybox.recipe.sysdeps\nanybox:anybox.scripts.odoo\napiclient:google_api_python_client\napitools:google_apitools\napm:arpm\napp_data:django_appdata\nappconf:django_appconf\nappd:AppDynamicsDownloader\nappd:AppDynamicsREST\nappdynamics_bindeps:appdynamics_bindeps_linux_x64\nappdynamics_bindeps:appdynamics_bindeps_linux_x86\nappdynamics_bindeps:appdynamics_bindeps_osx_x64\nappdynamics_proxysupport:appdynamics_proxysupport_linux_x64\nappdynamics_proxysupport:appdynamics_proxysupport_linux_x86\nappdynamics_proxysupport:appdynamics_proxysupport_osx_x64\nappium:Appium_Python_Client\nappliapps:applibase\nappserver:broadwick\narchetypes:archetypes.kss\narchetypes:archetypes.multilingual\narchetypes:archetypes.schemaextender\narm:ansible_role_manager\narmor:armor_api\narmstrong:armstrong.apps.related_content\narmstrong:armstrong.apps.series\narmstrong:armstrong.cli\narmstrong:armstrong.core.arm_access\narmstrong:armstrong.core.arm_layout\narmstrong:armstrong.core.arm_sections\narmstrong:armstrong.core.arm_wells\narmstrong:armstrong.dev\narmstrong:armstrong.esi\narmstrong:armstrong.hatband\narmstrong:armstrong.templates.standard\narmstrong:armstrong.utils.backends\narmstrong:armstrong.utils.celery\narstecnica:arstecnica.raccoon.autobahn\narstecnica:arstecnica.sqlalchemy.async\narticle-downloader:article_downloader\nartifactcli:artifact_cli\narvados:arvados_python_client\narvados_cwl:arvados_cwl_runner\narvnodeman:arvados_node_manager\nasana_to_github:AsanaToGithub\nasciibinary:AsciiBinaryConverter\nasd:AdvancedSearchDiscovery\naskbot:askbot_tuan\naskbot:askbot_tuanpa\nasnhistory:asnhistory_redis\naspen_jinja2_renderer:aspen_jinja2\naspen_tornado_engine:aspen_tornado\nasprise_ocr_api:asprise_ocr_sdk_python_api\naspy:aspy.refactor_imports\naspy:aspy.yaml\nasterisk:asterisk_ami\nasts:add_asts\nasymmetricbase:asymmetricbase.enum\nasymmetricbase:asymmetricbase.fields\nasymmetricbase:asymmetricbase.logging\nasymmetricbase:asymmetricbase.utils\nasyncirc:asyncio_irc\nasyncmongoorm:asyncmongoorm_je\nasyncssh:asyncssh_unofficial\nathletelist:athletelistyy\natm:automium\natmosphere:atmosphere_python_client\natom:gdata\natomic:AtomicWrite\natomisator:atomisator.db\natomisator:atomisator.enhancers\natomisator:atomisator.feed\natomisator:atomisator.indexer\natomisator:atomisator.outputs\natomisator:atomisator.parser\natomisator:atomisator.readers\natreal:atreal.cmfeditions.unlocker\natreal:atreal.filestorage.common\natreal:atreal.layouts\natreal:atreal.mailservices\natreal:atreal.massloader\natreal:atreal.monkeyplone\natreal:atreal.override.albumview\natreal:atreal.richfile.preview\natreal:atreal.richfile.qualifier\natreal:atreal.usersinout\natsim:atsim.potentials\nattractsdk:attract_sdk\naudio:audio.bitstream\naudio:audio.coders\naudio:audio.filters\naudio:audio.fourier\naudio:audio.frames\naudio:audio.lp\naudio:audio.psychoacoustics\naudio:audio.quantizers\naudio:audio.shrink\naudio:audio.wave\naufrefer:auf_refer\nauslfe:auslfe.formonline.content\nauspost:auspost_apis\nauth0:auth0_python\nauth_server_client:AuthServerClient\nauthorize:AuthorizeSauce\nauthzpolicy:AuthzPolicyPlugin\nautobahn:autobahn_rce\navatar:geonode_avatar\nawebview:android_webview\nazure:azure_common\nazure:azure_mgmt_common\nazure:azure_mgmt_compute\nazure:azure_mgmt_network\nazure:azure_mgmt_nspkg\nazure:azure_mgmt_resource\nazure:azure_mgmt_storage\nazure:azure_nspkg\nazure:azure_servicebus\nazure:azure_servicemanagement_legacy\nazure:azure_storage\nb2gcommands:b2g_commands\nb2gperf:b2gperf_v1.3\nb2gperf:b2gperf_v1.4\nb2gperf:b2gperf_v2.0\nb2gperf:b2gperf_v2.1\nb2gperf:b2gperf_v2.2\nb2gpopulate:b2gpopulate_v1.3\nb2gpopulate:b2gpopulate_v1.4\nb2gpopulate:b2gpopulate_v2.0\nb2gpopulate:b2gpopulate_v2.1\nb2gpopulate:b2gpopulate_v2.2\nb3j0f:b3j0f.annotation\nb3j0f:b3j0f.aop\nb3j0f:b3j0f.conf\nb3j0f:b3j0f.sync\nb3j0f:b3j0f.utils\nbabel:Babel\nbabelglade:BabelGladeExtractor\nbackplane:backplane2_pyclient\nbackport_abcoll:backport_collections\nbackports:backports.functools_lru_cache\nbackports:backports.inspect\nbackports:backports.pbkdf2\nbackports:backports.shutil_get_terminal_size\nbackports:backports.socketpair\nbackports:backports.ssl\nbackports:backports.ssl_match_hostname\nbackports:backports.statistics\nbadgekit:badgekit_api_client\nbadlinks:BadLinksPlugin\nbael:bael.project\nbaidu:baidupy\nbalrog:buildtools\nbaluhn:baluhn_redux\nbamboo:bamboo.pantrybell\nbamboo:bamboo.scaffold\nbamboo:bamboo.setuptools_version\nbamboo:bamboo_data\nbamboo:bamboo_server\nbambu:bambu_codemirror\nbambu:bambu_dataportability\nbambu:bambu_enqueue\nbambu:bambu_faq\nbambu:bambu_ffmpeg\nbambu:bambu_grids\nbambu:bambu_international\nbambu:bambu_jwplayer\nbambu:bambu_minidetect\nbambu:bambu_navigation\nbambu:bambu_notifications\nbambu:bambu_payments\nbambu:bambu_pusher\nbambu:bambu_saas\nbambu:bambu_sites\nbanana:Bananas\nbanana:banana.maya\nbang:bangtext\nbarcode:barcode_generator\nbark:bark_ssg\nbarking_owl:BarkingOwl\nbart:bart_py\nbasalt:basalt_tasks\nbase62:base_62\nbasemap:basemap_Jim\nbash:bash_toolbelt\nbashutils:Python_Bash_Utils\nbasic_http:BasicHttp\nbasil:basil_daq\nbatchapps:azure_batch_apps\nbcrypt:python_bcrypt\nbeaker:Beaker\nbeetsplug:beets\nbegin:begins\nbenchit:bench_it\nbeproud:beproud.utils\nbfillings:burrito_fillings\nbigjob:BigJob\nbillboard:billboard.py\nbinstar_build_client:anaconda_build\nbinstar_client:anaconda_client\nbiocommons:biocommons.dev\nbirdhousebuilder:birdhousebuilder.recipe.conda\nbirdhousebuilder:birdhousebuilder.recipe.docker\nbirdhousebuilder:birdhousebuilder.recipe.redis\nbirdhousebuilder:birdhousebuilder.recipe.supervisor\nblender26-meshio:pymeshio\nbootstrap:BigJob\nborg:borg.localrole\nbow:bagofwords\nbpdb:bpython\nbqapi:bisque_api\nbraces:django_braces\nbriefscaster:briefs_caster\nbrisa_media_server/plugins:brisa_media_server_plugins\nbrkt_requests:brkt_sdk\nbroadcastlogging:broadcast_logging\nbrocadetool:brocade_tool\nbronto:bronto_python\nbrownie:Brownie\nbrowsermobproxy:browsermob_proxy\nbrubeckmysql:brubeck_mysql\nbrubeckoauth:brubeck_oauth\nbrubeckservice:brubeck_service\nbrubeckuploader:brubeck_uploader\nbs4:beautifulsoup4\nbson:pymongo\nbst:bst.pygasus.core\nbst:bst.pygasus.datamanager\nbst:bst.pygasus.demo\nbst:bst.pygasus.i18n\nbst:bst.pygasus.resources\nbst:bst.pygasus.scaffolding\nbst:bst.pygasus.security\nbst:bst.pygasus.session\nbst:bst.pygasus.wsgi\nbtable:btable_py\nbtapi:bananatag_api\nbtceapi:btce_api\nbtcebot:btce_bot\nbtsync:btsync.py\nbuck:buck.pprint\nbud:bud.nospam\nbudy:budy_api\nbuffer:buffer_alpaca\nbuggd:bug.gd\nbugle:bugle_sites\nbugspots:bug_spots\nbugzilla:python_bugzilla\nbugzscout:bugzscout_py\nbuildTools:ajk_ios_buildTools\nbuildnotifylib:BuildNotify\nbuildout:buildout.bootstrap\nbuildout:buildout.disablessl\nbuildout:buildout.dumppickedversions\nbuildout:buildout.dumppickedversions2\nbuildout:buildout.dumprequirements\nbuildout:buildout.eggnest\nbuildout:buildout.eggscleaner\nbuildout:buildout.eggsdirectories\nbuildout:buildout.eggtractor\nbuildout:buildout.extensionscripts\nbuildout:buildout.locallib\nbuildout:buildout.packagename\nbuildout:buildout.recipe.isolation\nbuildout:buildout.removeaddledeggs\nbuildout:buildout.requirements\nbuildout:buildout.sanitycheck\nbuildout:buildout.sendpickedversions\nbuildout:buildout.threatlevel\nbuildout:buildout.umask\nbuildout:buildout.variables\nbuildslave:buildbot_slave\nbuiltins:pies2overrides\nbumper:bumper_lib\nbumple:bumple_downloader\nbundesliga:bundesliga_cli\nbundlemaker:bundlemanager\nburpui:burp_ui\nbusyflow:busyflow.pivotal\nbuttercms-django:buttercms_django\nbuzz:buzz_python_client\nbvc:buildout_versions_checker\nbvggrabber:bvg_grabber\nbyond:BYONDTools\nbzETL:Bugzilla_ETL\nbzlib:bugzillatools\nbzrlib:bzr\nbzrlib:bzr_automirror\nbzrlib:bzr_bash_completion\nbzrlib:bzr_colo\nbzrlib:bzr_killtrailing\nbzrlib:bzr_pqm\nc2c:c2c.cssmin\nc2c:c2c.recipe.closurecompile\nc2c:c2c.recipe.cssmin\nc2c:c2c.recipe.jarfile\nc2c:c2c.recipe.msgfmt\nc2c:c2c.recipe.pkgversions\nc2c:c2c.sqlalchemy.rest\nc2c:c2c.versions\nc2c_recipe_facts:c2c.recipe.facts\ncabalgata:cabalgata_silla_de_montar\ncabalgata:cabalgata_zookeeper\ncache_utils:django_cache_utils\ncaptcha:django_recaptcha\ncartridge:Cartridge\ncassandra:cassandra_driver\ncassandralauncher:CassandraLauncher\ncc42:42qucc\ncerberus:Cerberus\nchameleon:Chameleon\ncharmtools:charm_tools\nchef:PyChef\nchip8:c8d\ncjson:python_cjson\nclassytags:django_classy_tags\ncloghandler:ConcurrentLogHandler\nclonevirtualenv:virtualenv_clone\ncloud-insight:al_cloudinsight\ncloud_admin:adminapi\ncloudservers:python_cloudservers\nclusterconsole:cerebrod\nclustersitter:cerebrod\ncms:django_cms\ncolander:ba_colander\ncolors:ansicolors\ncompile:bf_lc3\ncompose:docker_compose\ncompressor:django_compressor\nconcurrent:futures\nconfigargparse:ConfigArgParse\nconfigparser:pies2overrides\ncontracts:PyContracts\ncoordination:BigJob\ncopyreg:pies2overrides\ncorebio:weblogo\ncouchapp:Couchapp\ncouchdb:CouchDB\ncouchdbcurl:couchdb_python_curl\ncourseradownloader:coursera_dl\ncow:cow_framework\ncreole:python_creole\ncreoleparser:Creoleparser\ncrispy_forms:django_crispy_forms\ncronlog:python_crontab\ncrontab:python_crontab\nctff:tff\ncups:pycups\ncurator:elasticsearch_curator\ncurl:pycurl\ndaemon:python_daemon\ndare:DARE\ndateutil:python_dateutil\ndawg:DAWG\ndecouple:python-decouple\ndeb822:python_debian\ndebian:python_debian\ndemo:webunit\ndeployer:juju_deployer\ndepot:filedepot\ndevtools:tg.devtools\ndgis:2gis\ndhtmlparser:pyDHTMLParser\ndigitalocean:python_digitalocean\ndistribute_setup:ez_setup\ndistutils2:Distutils2\ndjango:Django\ndjango_hstore:amitu_hstore\ndjangobower:django_bower\ndjcelery:django_celery\ndjkombu:django_kombu\ndjorm_pgarray:djorm_ext_pgarray\ndns:dnspython\ndocgen:ansible_docgenerator\ndocker:docker_py\ndogpile:dogpile.cache\ndogpile:dogpile.core\ndogshell:dogapi\ndot_parser:pydot\ndot_parser:pydot2\ndot_parser:pydot3k\ndotenv:python-dotenv\ndpkt:dpkt_fix\ndsml:python_ldap\ndurationfield:django_durationfield\ndzclient:datazilla\neasybuild:easybuild_framework\neditor:python_editor\nelasticluster:azure_elasticluster\nelasticluster:azure_elasticluster_current\nelftools:pyelftools\nelixir:Elixir\nem:empy\nemlib:empy\nenchant:pyenchant\nencutils:cssutils\nengineio:python_engineio\nenum:enum34\nephem:pyephem\nerrorreporter:abl.errorreporter\nesplot:beaker_es_plot\nexample:adrest\nexamples:tweepy\nez_setup:pycassa\nfabfile:Fabric\nfabric:Fabric\nfaker:Faker\nfdpexpect:pexpect\nfedora:python_fedora\nfias:ailove_django_fias\nfiftyone_degrees:51degrees_mobile_detector\nfive:five.customerize\nfive:five.globalrequest\nfive:five.intid\nfive:five.localsitemanager\nfive:five.pt\nflasher:android_flasher\nflask:Flask\nflask_frozen:Frozen_Flask\nflask_redis:Flask_And_Redis\nflaskext:Flask_Bcrypt\nflvscreen:vnc2flv\nfollowit:django_followit\nforge:pyforge\nformencode:FormEncode\nformtools:django_formtools\nfourch:4ch\nfranz:allegrordf\nfreetype:freetype_py\nfrontmatter:python_frontmatter\nftpcloudfs:ftp_cloudfs\nfuntests:librabbitmq\nfuse:fusepy\nfuzzy:Fuzzy\ngabbi:tiddlyweb\ngen_3dwallet:3d_wallet_generator\ngendimen:android_gendimen\ngenshi:Genshi\ngeohash:python_geohash\ngeonode:GeoNode\ngeoserver:gsconfig\ngeraldo:Geraldo\ngetenv:django_getenv\ngeventwebsocket:gevent_websocket\ngflags:python_gflags\ngit:GitPython\ngithub:PyGithub\ngitpy:git_py\nglobusonline:globusonline_transfer_api_client\ngoogle:protobuf\ngoogleapiclient:google_api_python_client\ngrace-dizmo:grace_dizmo\ngrammar:anovelmous_grammar\ngrapheneapi:graphenelib\ngreplin:scales\ngridfs:pymongo\ngrokcore:grokcore.component\ngslib:gsutil\nhamcrest:PyHamcrest\nharpy:HARPy\nhawk:PyHawk_with_a_single_extra_commit\nhaystack:django_haystack\nhgext:mercurial\nhggit:hg_git\nhglib:python_hglib\nho:pisa\nhola:amarokHola\nhoover:Hoover\nhostlist:python_hostlist\nhtml:pies2overrides\nhtmloutput:nosehtmloutput\nhttp:pies2overrides\nhvad:django_hvad\nkrbv:krbV\ni99fix:199Fix\nigraph:python_igraph\nimdb:IMDbPY\nimpala:impyla\ninmemorystorage:ambition_inmemorystorage\nipaddress:backport_ipaddress\njaraco:jaraco.timing\njaraco:jaraco.util\njinja2:Jinja2\njiracli:jira_cli\njohnny:johnny_cache\njpgrid:python_geohash\njpiarea:python_geohash\njpype:JPype1\njpypex:JPype1\njsonfield:django_jsonfield\njstools:aino_jstools\njupyterpip:jupyter_pip\njwt:PyJWT\nkazoo:asana_kazoo\nkernprof:line_profiler\nkeyczar:python_keyczar\nkeyedcache:django_keyedcache\nkeystoneclient:python_keystoneclient\nkickstarter:kickstart\nkss:kss.core\nkuyruk:Kuyruk\nlangconv:AdvancedLangConv\nlava:lava_utils_interface\nlazr:lazr.authentication\nlazr:lazr.restfulclient\nlazr:lazr.uri\nldap:python_ldap\nldaplib:adpasswd\nldapurl:python_ldap\nldif:python_ldap\nlib2or3:2or3\nlib3to2:3to2\nlibaito:Aito\nlibbe:bugs_everywhere\nlibbucket:bucket\nlibcloud:apache_libcloud\nlibfuturize:future\nlibgenerateDS:generateDS\nlibmproxy:mitmproxy\nlibpasteurize:future\nlibsvm:7lk_ocr_deploy\nlisa:lisa_server\nloadingandsaving:aspose_words_java_for_python\nlocust:locustio\nlogbook:Logbook\nlogentries:buildbot_status_logentries\nlogilab:logilab_mtconverter\nmachineconsole:cerebrod\nmachinesitter:cerebrod\nmagic:python_magic\nmako:Mako\nmanifestparser:ManifestDestiny\nmarionette:marionette_client\nmarkdown:Markdown\nmarks:pytest_marks\nmarkupsafe:MarkupSafe\nmavnative:pymavlink\nmemcache:python_memcached\nmetacomm:AllPairs\nmetaphone:Metafone\nmetlog:metlog_py\nmezzanine:Mezzanine\nmigrate:sqlalchemy_migrate\nmimeparse:python_mimeparse\nminitage:minitage.paste\nminitage:minitage.recipe.common\nmissingdrawables:android_missingdrawables\nmkrst_themes:2lazy2rest\nmockredis:mockredispy\nmodargs:python_modargs\nmodel_utils:django_model_utils\nmodels:asposebarcode\nmodels:asposestorage\nmoksha:moksha.common\nmoksha:moksha.hub\nmoksha:moksha.wsgi\nmoneyed:py_moneyed\nmongoalchemy:MongoAlchemy\nmonthdelta:MonthDelta\nmopidy:Mopidy\nmopytools:MoPyTools\nmptt:django_mptt\nmrbob:mr.bob\nmsgpack:msgpack_python\nmutations:aino_mutations\nmws:amazon_mws\nmysql:mysql_connector_repackaged\nMySQLdb:MySQL-python\nnative_tags:django_native_tags\nndg:ndg_httpsclient\nnereid:trytond_nereid\nnested:baojinhuan\nnester:Amauri\nnester:abofly\nnester:bssm_pythonSig\nnovaclient:python_novaclient\noauth2_provider:alauda_django_oauth\noauth2client:oauth2client\nodf:odfpy\nometa:Parsley\nopenid:python_openid\nopensearchsdk:ali_opensearch\noslo_i18n:oslo.i18n\noslo_serialization:oslo.serialization\noslo_utils:oslo.utils\noss:alioss\noss:aliyun_python_sdk_oss\noss:aliyunoss\noutput:cashew\nowslib:OWSLib\npacketdiag:nwdiag\npaho:paho_mqtt\npaintstore:django_paintstore\nparler:django_parler\npast:future\npaste:PasteScript\npath:forked_path\npath:path.py\npaver:Paver\npeak:ProxyTypes\npicasso:anderson.picasso\npicklefield:django-picklefield\npilot:BigJob\npivotal:pivotal_py\nplayhouse:peewee\nplivoxml:plivo\nplone:plone.alterego\nplone:plone.api\nplone:plone.app.blob\nplone:plone.app.collection\nplone:plone.app.content\nplone:plone.app.contentlisting\nplone:plone.app.contentmenu\nplone:plone.app.contentrules\nplone:plone.app.contenttypes\nplone:plone.app.controlpanel\nplone:plone.app.customerize\nplone:plone.app.dexterity\nplone:plone.app.discussion\nplone:plone.app.event\nplone:plone.app.folder\nplone:plone.app.i18n\nplone:plone.app.imaging\nplone:plone.app.intid\nplone:plone.app.layout\nplone:plone.app.linkintegrity\nplone:plone.app.locales\nplone:plone.app.lockingbehavior\nplone:plone.app.multilingual\nplone:plone.app.portlets\nplone:plone.app.querystring\nplone:plone.app.redirector\nplone:plone.app.registry\nplone:plone.app.relationfield\nplone:plone.app.textfield\nplone:plone.app.theming\nplone:plone.app.users\nplone:plone.app.uuid\nplone:plone.app.versioningbehavior\nplone:plone.app.viewletmanager\nplone:plone.app.vocabularies\nplone:plone.app.widgets\nplone:plone.app.workflow\nplone:plone.app.z3cform\nplone:plone.autoform\nplone:plone.batching\nplone:plone.behavior\nplone:plone.browserlayer\nplone:plone.caching\nplone:plone.contentrules\nplone:plone.dexterity\nplone:plone.event\nplone:plone.folder\nplone:plone.formwidget.namedfile\nplone:plone.formwidget.recurrence\nplone:plone.i18n\nplone:plone.indexer\nplone:plone.intelligenttext\nplone:plone.keyring\nplone:plone.locking\nplone:plone.memoize\nplone:plone.namedfile\nplone:plone.outputfilters\nplone:plone.portlet.collection\nplone:plone.portlet.static\nplone:plone.portlets\nplone:plone.protect\nplone:plone.recipe.zope2install\nplone:plone.registry\nplone:plone.resource\nplone:plone.resourceeditor\nplone:plone.rfc822\nplone:plone.scale\nplone:plone.schema\nplone:plone.schemaeditor\nplone:plone.session\nplone:plone.stringinterp\nplone:plone.subrequest\nplone:plone.supermodel\nplone:plone.synchronize\nplone:plone.theme\nplone:plone.transformchain\nplone:plone.uuid\nplone:plone.z3cform\nplonetheme:plonetheme.barceloneta\npng:pypng\npolymorphic:django_polymorphic\nportalocker:ConcurrentLogHandler\npostmark:python_postmark\npowerprompt:bash_powerprompt\nprefetch:django-prefetch\nprintList:AndrewList\nprogressbar:progressbar2\nprogressbar:progressbar33\nprovider:django_oauth2_provider\npuresasl:pure_sasl\npwiz:peewee\npxssh:pexpect\npy7zlib:pylzma\npyAMI:pyAMI_core\npyarsespyder:arsespyder\npyasdf:asdf\npyaspell:aspell_python_ctypes\npybb:pybbm\npybloomfilter:pybloomfiltermmap\npyccuracy:Pyccuracy\npyck:PyCK\npycrfsuite:python_crfsuite\npydispatch:PyDispatcher\npygeolib:pygeocoder\npygments:Pygments\npygraph:python_graph_core\npyjon:pyjon.utils\npyjsonrpc:python_jsonrpc\npykka:Pykka\npylogo:PyLogo\npylons:adhocracy_Pylons\npymagic:libmagic\npymycraawler:Amalwebcrawler\npynma:AbakaffeNotifier\npyphen:Pyphen\npyrimaa:AEI\npysideuic:PySide\npysqlite2:adhocracy_pysqlite\npysqlite2:pysqlite\npythongettext:python_gettext\npythonjsonlogger:python_json_logger\npyutilib:PyUtilib\npyximport:Cython\nqs:qserve\nquadtree:python_geohash\nqueue:future\nquickapi:django_quickapi\nquickunit:nose_quickunit\nrackdiag:nwdiag\nradical:radical.pilot\nradical:radical.utils\nreStructuredText:Zope2\nreadability:readability_lxml\nreadline:gnureadline\nrecaptcha_works:django_recaptcha_works\nrelstorage:RelStorage\nreportapi:django_reportapi\nreprlib:pies2overrides\nrequests:Requests\nrequirements:requirements_parser\nrest_framework:djangorestframework\nrestclient:py_restclient\nretrial:async_retrial\nreversion:django_reversion\nrhaptos2:rhaptos2.common\nrobot:robotframework\nrobots:django_robots\nrosdep2:rosdep\nrsbackends:RSFile\nruamel:ruamel.base\ns2repoze:pysaml2\nsaga:saga_python\nsaml2:pysaml2\nsass:libsass\nsassc:libsass\nsasstests:libsass\nsassutils:libsass\nsayhi:alex_sayhi\nscalrtools:scalr\nscikits:scikits.talkbox\nscratch:scratchpy\nscreen:pexpect\nscss:pyScss\nsdict:dict.sorted\nsdk_updater:android_sdk_updater\nsekizai:django_sekizai\nsendfile:pysendfile\nserial:pyserial\nsetuputils:astor\nshapefile:pyshp\nshapely:Shapely\nsika:ahonya_sika\nsingleton:pysingleton\nsittercommon:cerebrod\nskbio:scikit_bio\nsklearn:scikit_learn\nslugify:unicode_slugify\nsmarkets:smk_python_sdk\nsnappy:ctypes_snappy\nsocketio:gevent_socketio\nsocketserver:pies2overrides\nsockjs:sockjs_tornado\nsocks:SocksiPy_branch\nsolr:solrpy\nsolution:Solution\nsorl:sorl_thumbnail\nsouth:South\nsphinx:Sphinx\nsphinx_pypi_upload:ATD_document\nsphinxcontrib:sphinxcontrib_programoutput\nsqlalchemy:SQLAlchemy\nsrc:atlas\nsrc:auto_mix_prep\nstats_toolkit:bw_stats_toolkit\nstatsd:dogstatsd_python\nstdnum:python_stdnum\nstoneagehtml:StoneageHTML\nstorages:django_storages\nstubout:mox\nsuds:suds_jurko\nswiftclient:python_swiftclient\nsx:pisa\ntabix:pytabix\ntaggit:django_taggit\ntasksitter:cerebrod\ntastypie:django_tastypie\nteamcity:teamcity_messages\ntelebot:pyTelegramBotAPI\ntempita:Tempita\ntenjin:Tenjin\ntermstyle:python_termstyle\ntest:pytabix\nthclient:treeherder_client\nthreaded_multihost:django_threaded_multihost\nthreecolor:3color_Press\ntidylib:pytidylib\ntkinter:future\ntlw:3lwg\ntoredis:toredis_fork\ntornadoredis:tornado_redis\ntower_cli:ansible_tower_cli\ntrac:Trac\ntracopt:Trac\ntranslation_helper:android_localization_helper\ntreebeard:django_treebeard\ntrytond:trytond_stock\ntsuru:tsuru_circus\ntvrage:python_tvrage\ntw2:tw2.core\ntw2:tw2.d3\ntw2:tw2.dynforms\ntw2:tw2.excanvas\ntw2:tw2.forms\ntw2:tw2.jit\ntw2:tw2.jqplugins.flot\ntw2:tw2.jqplugins.gritter\ntw2:tw2.jqplugins.ui\ntw2:tw2.jquery\ntw2:tw2.sqla\ntwisted:Twisted\ntwitter:python_twitter\ntxclib:transifex_client\nu115:115wangpan\nunidecode:Unidecode\nuniverse:ansible_universe\nusb:pyusb\nuseless:useless.pipes\nuserpass:auth_userpass\nutilities:automakesetup.py\nutkik:aino_utkik\nuwsgidecorators:uWSGI\nvalentine:ab\nvalidate:configobj\nversion:chartio\nvirtualenvapi:ar_virtualenv_api\nvyatta:brocade_plugins\nwebdav:Zope2\nweblogolib:weblogo\nwebob:WebOb\nwebsocket:websocket_client\nwebtest:WebTest\nwerkzeug:Werkzeug\nwheezy:wheezy.caching\nwheezy:wheezy.core\nwheezy:wheezy.http\nwikklytext:tiddlywebwiki\nwinreg:future\nwinrm:pywinrm\nworkflow:Alfred_Workflow\nwsmeext:WSME\nwtforms:WTForms\nwtfpeewee:wtf_peewee\nxdg:pyxdg\nxdist:pytest_xdist\nxmldsig:pysaml2\nxmlenc:pysaml2\nxmlrpc:pies2overrides\nxmpp:xmpppy\nxstatic:XStatic_Font_Awesome\nxstatic:XStatic_jQuery\nxstatic:XStatic_jquery_ui\nyaml:PyYAML\nz3c:z3c.autoinclude\nz3c:z3c.caching\nz3c:z3c.form\nz3c:z3c.formwidget.query\nz3c:z3c.objpath\nz3c:z3c.pt\nz3c:z3c.relationfield\nz3c:z3c.traverser\nz3c:z3c.zcmlhook\nzmq:pyzmq\nzopyx:zopyx.textindexng3\n"
  },
  {
    "path": "resources/tabs_ignore.txt",
    "content": "# Posix Regex\n# used on top of whitespace_ignore.txt\nMakefile\n.*\\.go$\n.*\\.json$\n.*\\.log$\n.*\\.xml$\n"
  },
  {
    "path": "resources/whitespace_ignore.txt",
    "content": "# Posix Regex\n/Makefile$\n/\\.\n\\.pyc$\n\\.pyo$\n\\.jyc$\n\\.jyo$\n\\.zip$\n\\.tar$\n\\.tgz$\n\\.tbz$\n\\.gz$\n\\.bz2$\n\\.class$\n\\.plist$\n\\.svg$\n\\.png$\n\\.jpg$\n\\.jpeg$\n/target/\n/fatpacks/\n/build/\n/gradle/\n/gradlew\n/mvnw\ndependency-reduced-pom.xml\n/[[:alnum:]-]+-[[:digit:]\\.]+/\n/cover_db/\ncheck_memcached_stats.cfg-pnp4nagios\n/spark-.*-bin-hadoop.*/\n-bin\\b\n"
  },
  {
    "path": "scalastyle_config.xml",
    "content": "<scalastyle commentFilter=\"enabled\">\n <name>Scalastyle standard configuration</name>\n <check level=\"warning\" class=\"org.scalastyle.file.FileTabChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.FileLengthChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxFileLength\"><![CDATA[800]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.file.HeaderMatchesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"header\"><![CDATA[// Copyright (C) 2011-2012 the original author or authors.\n// See the LICENCE.txt file distributed with this work for additional\n// information regarding copyright ownership.\n//\n// Licensed under the Apache License, Version 2.0 (the \"License\");\n// you may not use this file except in compliance with the License.\n// You may obtain a copy of the License at\n//\n// http://www.apache.org/licenses/LICENSE-2.0\n//\n// Unless required by applicable law or agreed to in writing, software\n// distributed under the License is distributed on an \"AS IS\" BASIS,\n// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n// See the License for the specific language governing permissions and\n// limitations under the License.]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.SpacesAfterPlusChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.WhitespaceEndOfLineChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.SpacesBeforePlusChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.FileLineLengthChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxLineLength\"><![CDATA[160]]></parameter>\n   <parameter name=\"tabSize\"><![CDATA[4]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ClassNamesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[[A-Z][A-Za-z]*]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ObjectNamesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[[A-Z][A-Za-z]*]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.PackageObjectNamesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[^[a-z][A-Za-z]*$]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.EqualsHashCodeChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.IllegalImportsChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"illegalImports\"><![CDATA[sun._,java.awt._]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ParameterNumberChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxParameters\"><![CDATA[8]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.MagicNumberChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"ignore\"><![CDATA[-1,0,1,2,3]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NoWhitespaceBeforeLeftBracketChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NoWhitespaceAfterLeftBracketChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ReturnChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NullChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NoCloneChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NoFinalizeChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.CovariantEqualsChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.StructuralTypeChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.RegexChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[println]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NumberOfTypesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxTypes\"><![CDATA[30]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.CyclomaticComplexityChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maximum\"><![CDATA[10]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.UppercaseLChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.SimplifyBooleanExpressionChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.IfBraceChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"singleLineAllowed\"><![CDATA[true]]></parameter>\n   <parameter name=\"doubleLineAllowed\"><![CDATA[false]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.MethodLengthChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxLength\"><![CDATA[50]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.MethodNamesChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[^[a-z][A-Za-z0-9]*$]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.NumberOfMethodsInTypeChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"maxMethods\"><![CDATA[30]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.PublicMethodsHaveTypeChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.NewLineAtEofChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.file.NoNewLineAtEofChecker\" enabled=\"false\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.WhileChecker\" enabled=\"false\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.VarFieldChecker\" enabled=\"false\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.VarLocalChecker\" enabled=\"false\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.RedundantIfChecker\" enabled=\"false\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.TokenChecker\" enabled=\"false\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[println]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.DeprecatedJavaChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.EmptyClassChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ClassTypeParameterChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"regex\"><![CDATA[^[A-Z_]$]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.UnderscoreImportChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.LowercasePatternMatchChecker\" enabled=\"true\"></check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.MultipleStringLiteralsChecker\" enabled=\"true\">\n  <parameters>\n   <parameter name=\"allowed\"><![CDATA[2]]></parameter>\n   <parameter name=\"ignoreRegex\"><![CDATA[^\"\"$]]></parameter>\n  </parameters>\n </check>\n <check level=\"warning\" class=\"org.scalastyle.scalariform.ImportGroupingChecker\" enabled=\"true\"></check>\n</scalastyle>"
  },
  {
    "path": "scripts/README.md",
    "content": "# Scripts\n\nThis directory is for quick hack scripts that are not as generic tools as the rest of this repo.\n\nThey usually have something specific to a situation or environment that may be personal to me and are not intended to used as-is.\n"
  },
  {
    "path": "scripts/git_capitalize_urls.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-04-28 19:22:01 +0100 (Thu, 28 Apr 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReplaces URLs in git files with capitalized equivalents for uniformity and easier reading\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nsed_script=\"\ns|https\\\\?://www.linkedin.com/in/HariSekhon|https://www.linkedin.com/in/HariSekhon|gi;\ns|https\\\\?://ghcr.io/harisekhon/|https://ghcr.io/HariSekhon/|gi;\n\n$(\n    sed 's/#.*//; /^[[:space:]]*$/d; s/:/ /' \"$srcdir/../setup/repos.txt\" |\n    while read -r repo shortname; do\n        if [ -n \"$repo\" ]; then\n            echo \"s|https\\\\?://github.com/harisekhon/$repo|https://github.com/HariSekhon/$repo|gi;\"\n        fi\n        if [ -n \"$shortname\" ]; then\n            echo \"s|https\\\\?://github.com/harisekhon/$shortname$|https://github.com/HariSekhon/$repo|gi;\"\n            echo \"s|https\\\\?://github.com/harisekhon/$shortname/|https://github.com/HariSekhon/$repo/|gi;\"\n        fi\n        echo\n    done\n)\"\n\necho \"Running sed script on all Git files:\"\necho\necho \"$sed_script\"\necho\n\nfor filename in $(git ls-files); do\n    [ -f \"$filename\" ] || continue\n    [ -d \"$filename\" ] && continue\n    [ -L \"$filename\" ] && continue\n    echo \"$filename\"\ndone |\nif is_mac; then\n    xargs gsed -i \"$sed_script\"\nelse\n    xargs sed -i \"$sed_script\"\nfi\n"
  },
  {
    "path": "scripts/spotify_commit_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-03 17:14:30 +0100 (Fri, 03 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nIterates over and commits all downloaded Spotify playlists in \\$PWD or playlists/ directory,\nshowing diffs and then committing each in turn\n\nFirst shows only the net removals in standard Spotify URIs + Track Name for a playlist\nto check if anything has been lost from a playlist (additions don't need as much scrutiny)\n\nDrastically reduces net removals list for human review by omitting duplicate URI removals\n(checks if URI is present in the spotify format playlist) or URI replacements for same song\n(either URI remapping or single vs album)\n\nIf there are no net removals then auto-commits the playlist\n\nOtherwise shows the list of net removals in both Spotify URI and Track name formats\nfollowed by the full human readable playlist diff and spotify URI diff underneath\n\nIf satisfactory, hitting enter at the end of the playlist diff will commit both\nthe Spotify URI and human readable playlist simultaneously\n\nRequires DevOps-Perl-tools to be in \\$PATH for diffnet.pl\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<playlist>]\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/.bash.d/git.sh\"\n\nhelp_usage \"$@\"\n\nspotify_token || :\n\nexec 3<&0\n\ncommit_playlist(){\n    playlist=\"$1\"\n    if ! [ -f \"$playlist\" ] ||\n       ! [ -f \"spotify/$playlist\" ] ||\n       [[ \"$playlist\" =~ \\.txt$ ]]; then\n        return\n    fi\n    timestamp \"Checking playlist: $playlist\"\n    line_count_human_playlist=\"$(wc -l < \"$playlist\" | sed 's/[[:space:]]//g')\"\n    line_count_spotify_playlist=\"$(wc -l < \"spotify/$playlist\" | sed 's/[[:space:]]//g')\"\n    if [ \"$line_count_human_playlist\" != \"$line_count_spotify_playlist\" ]; then\n        warn \"ERROR: line counts between '$playlist' and 'spotify/$playlist' do not match! Partially interrupted download?\"\n    fi\n    if git status -s -- \"$playlist\" \"spotify/$playlist\" | grep -q '^[?A]'; then\n        git add \"$playlist\" \"spotify/$playlist\"\n        if [ -f \"$playlist.description\" ]; then\n            git add \"$playlist.description\"\n            git ci -m \"added $playlist spotify/$playlist\" \"$playlist\" \"spotify/$playlist\" \"$playlist.description\"\n        else\n            git ci -m \"added $playlist spotify/$playlist\" \"$playlist\" \"spotify/$playlist\"\n        fi\n        return\n    fi\n    if ! git status -s -- \"$playlist\" \"spotify/$playlist\" | grep -q '^.M'; then\n        return\n    fi\n    net_removals=\"$(find_net_removals \"$playlist\")\"\n    if [ -z \"$net_removals\" ]; then\n        # XXX: double safety check on the top level Artist-Tracks file which might have been partially modified\n        #      even if the spotify/playlist file has no net removals\n        local stats\n        local added_num\n        local removed_num\n        stats=\"$(git diff --numstat \"$playlist\")\"\n        added_num=\"$(awk '{print $1}' <<< \"$stats\")\"\n        removed_num=\"$(awk '{print $2}' <<< \"$stats\")\"\n        if [ \"$removed_num\" -gt \"$added_num\" ]; then\n            die \"ERROR: tracks have been removed from playlist '$playlist' top level file which were not removed from 'spotify/$playlist' - partial interrupted playlist download or files committed separately?\"\n        fi\n        echo \"Auto-committing playlist '$playlist' as no net removals\"\n        echo\n        git add -- \"$playlist\" \"spotify/$playlist\"\n        git ci -m \"updated $playlist spotify/$playlist\" -- \"$playlist\" \"spotify/$playlist\"\n        echo\n        return\n    fi\n    echo \"Net Removals from playlist '$playlist' (could be replaced with different track version):\"\n    echo\n    echo \"$net_removals\"\n    echo\n    #read -r -p \"Hit enter to see full human and spotify diffs or Control-C to exit\"\n    echo \"Hit enter to see full human and spotify diffs or Control-C to exit\"\n    read -r <&3\n    echo\n    git diff \"$playlist\" \"spotify/$playlist\"\n    echo\n    echo \"Hit enter to commit playlist '$playlist' or Control-C to exit\"\n    read -r <&3\n    echo\n    git add \"$playlist\" \"spotify/$playlist\"\n    git ci -m \"updated $playlist\"\n}\n\nfind_net_removals(){\n    local playlist=\"$1\"\n    # stop grep breaking everything when no removals\n    git diff \"spotify/$playlist\" |\n    diffnet.pl |\n    { grep ^- || :; } |\n    sed 's/^-//' |\n    while read -r uri; do\n        if grep -Fxq \"$uri\" \"spotify/$playlist\"; then\n            if [ -n \"${VERBOSE:-}\" ]; then\n                echo \"skipping removed duplicate URI '$uri' which is present in spotify/$playlist\" >&2\n            fi\n            continue\n        fi\n        track=\"$(\"$srcdir/../spotify/spotify_uri_to_name.sh\" <<< \"$uri\")\"\n        if grep -Fxq \"$track\" \"$playlist\"; then\n            if [ -n \"${VERBOSE:-}\" ]; then\n                echo \"skipping removed URI for track '$track' which is found in $playlist (must have been replaced with a different URI)\" >&2\n            fi\n            continue\n        fi\n        printf '%s\\t%s\\n' \"$uri\" \"$track\"\n    done\n}\n\nif [ $# -gt 0 ]; then\n    for playlist in \"$@\"; do\n        commit_playlist \"$playlist\"\n    done\nelse\n    if ! [[ \"$PWD\" =~ playlists ]]; then\n        cd playlists\n    fi\n    git status --porcelain |\n    { grep '^.M' || :; } |\n    sed 's/^...//; s,spotify/,,; s/^\"//; s/\"$//' |\n    sort -u |\n    while read -r playlist; do\n        commit_playlist \"$playlist\"\n    done\nfi\n"
  },
  {
    "path": "scripts/spotify_commit_rename_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-04 13:14:15 +0100 (Sat, 04 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nCommits and then Renames a Spotify playlist\n\nWhen Spotify playlist names change and you want to commit the updates which were downloaded to a new non-committed playlist file,\nthis script will copy the new playlist file given as the first argument over the old playlist filen given as the second argument,\nthen commit the original playlist file to check the history for removals using spotify_commit_playlists.sh, and then rename the old\nplaylist file to the new playlist file to align with the spotify_backup*.sh exports in future\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<old_playlist_name> <new_playlist_name>\"\n\nhelp_usage \"$@\"\n\nnum_args 2 \"$@\"\n\nold=\"$1\"\nnew=\"$2\"\n\n# Copy the newly downloaded playlist files over the old git committed ones\n# so that we can then commit them under the old name before renaming the safely\n# committed playlist to the new name\nmv -vf -- \"$new\" \"$old\"\nmv -vf -- \"spotify/$new\" \"spotify/$old\"\n\n\"$srcdir/spotify_commit_playlists.sh\" \"$old\"\n\n\"$srcdir/spotify_rename_playlist_files.sh\" \"$old\" \"$new\"\n"
  },
  {
    "path": "scripts/spotify_rename_playlist_files.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-04 13:14:15 +0100 (Sat, 04 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../.bash.d/git.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nRenames a Spotify playlist in both the \\$PWD and \\$PWD/spotify/ directories\nto keep the Spotify backups in sync\n\nIf <new_playlist_name> is a directory or ends with /, the file is moved into\nthat directory keeping the same filename (rename to <dir>/<old_filename>)\n\nWith optional third arg <subdir> (path-mapped backups): renames under\n\\$PWD/<subdir>/ and \\$PWD/spotify/<subdir>/ instead, invoke from backup base\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<old_playlist_name> <new_playlist_name> [<subdir>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nold=\"$1\"\nnew=\"$2\"\nsubdir=\"${3:-}\"\n\n# the replacement of slashes breaks the later logic to test for destination being a directory\n#old=\"$(\"$srcdir/../spotify/spotify_playlist_to_filename.sh\" \"$old\")\"\n#new=\"$(\"$srcdir/../spotify/spotify_playlist_to_filename.sh\" \"$new\")\"\n\n# optional subdir for path-mapped backups: playlists in base/Subdir/, URIs in base/spotify/Subdir/\nif [ -n \"$subdir\" ]; then\n    prefix=\"$subdir/\"\n    spotify_prefix=\"spotify/$subdir/\"\nelse\n    prefix=\"\"\n    spotify_prefix=\"spotify/\"\nfi\n\n# if new is a directory or ends in /, treat as directory: rename to same filename under that path\nif [[ \"$new\" == */ ]] || [[ -d \"${prefix}$new\" ]]; then\n    mkdir -p -v \"$new\"\n    new_dest=\"${new%/}/$old\"\n    dest_prefix=\"\"\n    spotify_dest_prefix=\"spotify/\"\n    mkdir -p -v \"${spotify_dest_prefix}$new\"\nelse\n    new_dest=\"$new\"\n    dest_prefix=\"$prefix\"\n    spotify_dest_prefix=\"$spotify_prefix\"\nfi\n\n# the gitrename function in lib/git.sh has been updated to preserve the new file\n# and restore it after the move to then git diff and commit any updates\ngitrename \"${prefix}$old\" \"${dest_prefix}$new_dest\"\n\ngitrename \"${spotify_prefix}$old\" \"${spotify_dest_prefix}$new_dest\"\n\nif [ -f \"aggregations/${prefix}$old\" ]; then\n    gitrename \"aggregations/${prefix}$old\" \"aggregations/${dest_prefix}$new_dest\"\nfi\n\nif [ -f \"${prefix}$old.description\" ]; then\n    gitrename \"${prefix}$old.description\" \"${dest_prefix}$new_dest.description\"\nfi\n\nif [ -f \"id/${prefix}$old.id.txt\" ]; then\n    mv -v \"id/${prefix}$old.id.txt\" \"id/${dest_prefix}$new_dest.id.txt\"\nfi\n"
  },
  {
    "path": "scripts/update_bash_tools_references.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Makefile README.md Jenkinsfile *.yaml *.yml */*.sh *.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-10 00:44:52 +0100 (Wed, 10 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUpdates references to bash-tools/ or bash_tools in repos using this as a submodule to account for the repo re-org\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<files>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# uniform prefixes on scripts matching their directory names\nprefixes='\ngit\npython\nperl\ngithub\ngitlab\nbitbucket\nazure_devops\nkubernetes\ncircleci\nbuildkite\njenkins\nmysql\npostgres\nmp3\npingdom\nshippable\nteamcity\nspotify\nterraform\ntravis\nvagrant\nwercker\nkafka\naws\ngcp\ndocker\ndrone\ncodeship\n'\n\n# directory script_prefix mappings\nmappings='\nlib utils.sh\nchecks check\nchecks run\n\ngcp gke\ngcp gce\ngcp gcr\ngcp bigquery\n\nbigdata cloudera\nbigdata hadoop\nbigdata hdfs\nbigdata hive\nbigdata impala\nbigdata beeline\nbigdata zookeeper\n\nbin azure_info.sh\nbin center.sh\nbin clean_caches.sh\nbin crt_hash.sh\nbin csv_header_indices.sh\nbin curl_auth.sh\nbin decomment.sh\nbin delete_empty_dirs.sh\nbin disable_swap.sh\nbin elasticsearch_decommission_node.sh\nbin exec_interactive.sh\nbin find\nbin grep_or_append.sh\nbin headtail.sh\nbin json2yaml.sh\nbin jsondiff.sh\nbin jvm\nbin keycloak.sh\nbin ldap\nbin ldapsearch.sh\nbin lint.sh\nbin login.sh\nbin progress_dots.sh\nbin random\nbin retry.sh\nbin run.sh\nbin sbtw\nbin scan_duplicate_macs.sh\nbin shellcheck.sh\nbin split.sh\nbin sqlite.sh\nbin ssl\nbin uniq_chars.sh\nbin url\nbin vault\nbin word_frequency.sh\nbin yaml2json.sh\n\ninternet atlassian_ip_ranges.sh\ninternet datadog_api.sh\ninternet digital_ocean_api.sh\ninternet dnsjson.sh\ninternet jira_api.sh\ninternet kong_api.sh\ninternet ngrok_api.sh\ninternet traefik_api.sh\n\nconfigs .vimrc\n\ncicd checkov_resource_count.sh\ncicd checkov_resource_count_all.sh\ncicd codefresh_cancel_delayed_builds.sh\ncicd concourse.sh\ncicd coveralls_latest.sh\ncicd fly.sh\ncicd generate_status_page.sh\ncicd gerrit.sh\ncicd gerrit_projects.sh\ncicd gocd.sh\ncicd gocd_api.sh\ncicd octopus_api.sh\ncicd run_latest_tests.sh\ncicd run_tests.sh\ncicd selenium_hub_wait_ready.sh\ncicd sync_bootstraps_to_adjacent_repos.sh\ncicd sync_ci_to_adjacent_repos.sh\ncicd sync_configs_to_adjacent_repos.sh\n\nkubernetes argocd\nkubernetes datree\nkubernetes helm\nkubernetes kubeadm\nkubernetes kubectl\nkubernetes kustomize\nkubernetes pluto\nkubernetes prometheus.sh\n\npackages install\npackages apk\npackages apt\npackages brew\npackages debs\npackages rpms\npackages nodejs\npackages golang\npackages ruby\npackages yum\n'\n\ntimestamp \"Replacing references\"\n\nsed_script=\"$(\n        for prefix in $prefixes; do\n            echo \"s|bash-tools/${prefix}_|bash-tools/$prefix/${prefix}_|g;\"\n            echo \"s|bash_tools/${prefix}_|bash_tools/$prefix/${prefix}_|g;\"\n            echo \"s|master/${prefix}_|master/$prefix/${prefix}_|g;\"\n        done\n        while read -r directory prefix; do\n            if [ -z \"$directory\" ]; then\n                continue\n            fi\n            # catch prefixes\n            echo \"s|bash-tools/${prefix}_|bash-tools/$directory/${prefix}_|g;\"\n            echo \"s|bash_tools/${prefix}_|bash_tools/$directory/${prefix}_|g;\"\n            echo \"s|master/${prefix}_|master/$directory/${prefix}_|g;\"\n            # catch whole scripts\n            echo \"s|bash-tools/${prefix}\\\\>|bash-tools/$directory/${prefix}|g;\"\n            echo \"s|bash_tools/${prefix}\\\\>|bash_tools/$directory/${prefix}|g;\"\n            echo \"s|master/${prefix}\\\\>|master/$directory/${prefix}|g;\"\n        done <<< \"$mappings\"\n)\"\n\nsed -i \"$sed_script\" \"$@\"\n\necho >&2\ntimestamp \"Search and Replace Done\"\necho >&2\ntimestamp \"Remaining references to check:\"\necho >&2\n\nignore_regex=\"$(for prefix in $prefixes; do echo -n \"$prefix|\"; done)\"\nignore_regex=\"/(${ignore_regex%|})/\"\n\n# need splitting\n# shellcheck disable=SC2046\ngit grep -e 'bash_tools/.' -e 'bash-tools/.' |\ngrep -Ev -e /setup/ \\\n        -e \"$ignore_regex\" \\\n        -e /bin/ \\\n        -e /checks/ \\\n        -e /lib/ \\\n        -e bash-tools/Makefile.in\n"
  },
  {
    "path": "search/solr_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /api | jq .\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-20 10:22:32 +0000 (Tue, 20 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nWorks with the Solr API\n\nAutomatically handles authentication if environment variable \\$SOLR_PASSWORD or \\$SOLR_TOKEN are available\n\nSOLR_HOST must be set\nSOLR_PORT defaults to '8983'\nSOLR_PROTOCOL defaults to 'http'\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nAPI Reference:\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/config-api.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/request-parameters-api.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/managed-resources.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/collections-api.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/configsets-api.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/coreadmin-api.html\n\n    https://solr.apache.org/guide/solr/latest/configuration-guide/v2-api.html\n\n\nFor convenience you may omit the /api prefix and it will be added automatically\n\n\nExamples:\n\n    ${0##*/} /solr/admin/collections?action=CLUSTERSTATUS\n\n    ${0##*/} /solr/admin/collections?action=LIST\n\n    ${0##*/} '/solr/admin/cores?action=LIST&distrib=false'\n\n    Use this in scripts to abstract away the solr connection and authentication details via centralized environment variables such as a .envrc file\n\n    ${0##*/} '/solr/admin/collections?action=CREATE&name=\\$collection&numShards=\\$NUM_SHARDS&maxShardsPerNode=\\$MAX_SHARDS_PER_NODE&replicationFactor=3&wt=xml&collection.configName=\\$collection&autoAddReplicas=true'\n\n    See Collections API for more creating / modifying collections / splitting shards:\n\n        https://solr.apache.org/guide/solr/latest/configuration-guide/collections-api.html\n\n\nSee Also\n\n    Solr CLI - https://github.com/HariSekhon/DevOps-Perl-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\ncheck_env_defined SOLR_HOST\n\nhost=\"$SOLR_HOST\"\nport=\"${SOLR_PORT:-8983}\"\nprotocol=\"${SOLR_PROTOCOL:-http}\"\nurl_base=\"$protocol://$host:$port\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"$1\"\nshift || :\nurl_path=\"${url_path#/}\"\n# might want to query /solr/admin or /api\n#if ! [[ \"$url_path\" =~ ^api ]]; then\n#    url_path=\"api/$url_path\"\n#fi\n\nurl=\"$url_path\"\nif ! [[ \"$url_path\" =~ :// ]]; then\n    url_path=\"${url_path##/}\"\n    url=\"$url_base/$url_path\"\nfi\n\nexport USERNAME=\"${SOLR_USERNAME:-${SOLR_USER:-${USERNAME:-}}}\"\nexport PASSWORD=\"${SOLR_PASSWORD:-}\"\nexport TOKEN=\"${SOLR_TOKEN:-${TOKEN:-}}\"\n\nif not_blank \"$PASSWORD\" || not_blank \"$TOKEN\"; then\n    \"$srcdir/curl_auth.sh\" \"$url\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"} \"$@\"\nelse\n    # CURL_OPTS adds -H 'Accept: application/json' -H 'Content-Type: application/json' and other defaults from lib/utils.sh\n    curl \"$url\" ${CURL_OPTS:+\"${CURL_OPTS[@]}\"}  \"$@\"\nfi |\n# XML shouldn't be returned due to above CURL_OPTS giving -H 'Accept: application/json'\njq_debug_pipe_dump\n"
  },
  {
    "path": "search/solr_collection_check_exists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-25 23:53:53 +0000 (Sun, 25 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nChecks if a given Solr collection exists via the Solr API\n\nUses the adjacent script solr_api.sh - see it for required environment variables and authentication\n\nSee Also\n\n    Solr CLI - https://github.com/HariSekhon/DevOps-Perl-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<collection_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncollection=\"$1\"\nshift || :\n\nif \"$srcdir/solr_api.sh\" \"/solr/admin/collections?action=LIST\" | jq -r '.collections[]' | grep -Fxq \"$collection\"; then\n    timestamp \"Solr collection '$collection' exists\"\nelse\n    timestamp \"Solr collection '$collection' does not exist\"\n    exit 1\nfi\n"
  },
  {
    "path": "search/solr_collection_create_if_not_exists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-25 23:53:53 +0000 (Sun, 25 Feb 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Solr Collection via the Solr API if it does not already exist\n\nUses the adjacent script solr_api.sh - see it for required environment variables and authentication\n\nAssume a config of the same name already exists in ZooKeeper\n\nSet environment variables\n\nNUM_SHARDS              defaults to 3\nREPLICATION_FACTOR      defaults to 3\nNUM_SHARDS_PER_NODE     defaults to 9\n\nSee Also\n\n    Solr CLI - https://github.com/HariSekhon/DevOps-Perl-tools\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<collection_name> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncollection=\"$1\"\nshift || :\n\nNUM_SHARDS=\"${NUM_SHARDS:-3}\"\nMAX_SHARDS_PER_NODE=\"${MAX_SHARDS_PER_NODE:-9}\"\nREPLICATION_FACTOR=\"${REPLICATION_FACTOR:-3}\"\n\ntimestamp \"Checking if collection '$collection' already exists\"\nif \"$srcdir/solr_collection_check_exists.sh\" \"$collection\"; then\n    timestamp \"Skipping create\"\nelse\n    \"$srcdir/solr_api.sh\" \"/solr/admin/collections?action=CREATE&name=$collection&numShards=$NUM_SHARDS&maxShardsPerNode=$MAX_SHARDS_PER_NODE&replicationFactor=$REPLICATION_FACTOR&wt=xml&collection.configName=$collection&autoAddReplicas=true\"\nfi\n"
  },
  {
    "path": "setup/Hari.terminal",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n<plist version=\"1.0\">\n<dict>\n\t<key>BackgroundColor</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGjCwwTVSRudWxs0w0ODxAREldO\n\tU1doaXRlXE5TQ29sb3JTcGFjZVYkY2xhc3NNMCAwLjg5OTk5OTk4ABADgALSFBUWF1ok\n\tY2xhc3NuYW1lWCRjbGFzc2VzV05TQ29sb3KiFhhYTlNPYmplY3QIERokKTI3SUxRU1dd\n\tZGx5gI6Qkpeiq7O2AAAAAAAAAQEAAAAAAAAAGQAAAAAAAAAAAAAAAAAAAL8=\n\t</data>\n\t<key>Bell</key>\n\t<false/>\n\t<key>CursorBlink</key>\n\t<true/>\n\t<key>CursorColor</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGjCwwTVSRudWxs0w0ODxARElVO\n\tU1JHQlxOU0NvbG9yU3BhY2VWJGNsYXNzTxAhMC4yMTk2MDc4NiAwLjk5NjA3ODQ5IDAu\n\tMTUyOTQxMTgAEAKAAtIUFRYXWiRjbGFzc25hbWVYJGNsYXNzZXNXTlNDb2xvcqIWGFhO\n\tU09iamVjdAgRGiQpMjdJTFFTV11kand+oqSmq7a/x8oAAAAAAAABAQAAAAAAAAAZAAAA\n\tAAAAAAAAAAAAAAAA0w==\n\t</data>\n\t<key>CursorType</key>\n\t<integer>0</integer>\n\t<key>Font</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGkCwwVFlUkbnVsbNQNDg8QERIT\n\tFFZOU1NpemVYTlNmRmxhZ3NWTlNOYW1lViRjbGFzcyNAKAAAAAAAABAQgAKAA1pBbmRh\n\tbGVNb25v0hcYGRpaJGNsYXNzbmFtZVgkY2xhc3Nlc1ZOU0ZvbnSiGRtYTlNPYmplY3QI\n\tERokKTI3SUxRU1heZ253foWOkJKUn6SvuL/CAAAAAAAAAQEAAAAAAAAAHAAAAAAAAAAA\n\tAAAAAAAAAMs=\n\t</data>\n\t<key>FontAntialias</key>\n\t<false/>\n\t<key>ProfileCurrentVersion</key>\n\t<real>2.0699999999999998</real>\n\t<key>SelectionColor</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGmCwwXHR4lVSRudWxs1Q0ODxAR\n\tEhMUFRZcTlNDb21wb25lbnRzVU5TUkdCXE5TQ29sb3JTcGFjZV8QEk5TQ3VzdG9tQ29s\n\tb3JTcGFjZVYkY2xhc3NPEB4wLjAxNjgwNDE3NzMxIDAuMTk4MzUwOTk1OCAxIDFPECo2\n\tLjU3NDc2NDg0OGUtMDUgMC4wMDE4MDEwMTM5NDcgMC45OTgyMjg1NQAQAYACgAXTGBkR\n\tGhscVE5TSURVTlNJQ0MQB4ADgARPEQxIAAAMSExpbm8CEAAAbW50clJHQiBYWVogB84A\n\tAgAJAAYAMQAAYWNzcE1TRlQAAAAASUVDIHNSR0IAAAAAAAAAAAAAAAAAAPbWAAEAAAAA\n\t0y1IUCAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAARY3BydAAAAVAAAAAzZGVzYwAAAYQAAABsd3RwdAAAAfAAAAAUYmtwdAAAAgQAAAAU\n\tclhZWgAAAhgAAAAUZ1hZWgAAAiwAAAAUYlhZWgAAAkAAAAAUZG1uZAAAAlQAAABwZG1k\n\tZAAAAsQAAACIdnVlZAAAA0wAAACGdmlldwAAA9QAAAAkbHVtaQAAA/gAAAAUbWVhcwAA\n\tBAwAAAAkdGVjaAAABDAAAAAMclRSQwAABDwAAAgMZ1RSQwAABDwAAAgMYlRSQwAABDwA\n\tAAgMdGV4dAAAAABDb3B5cmlnaHQgKGMpIDE5OTggSGV3bGV0dC1QYWNrYXJkIENvbXBh\n\tbnkAAGRlc2MAAAAAAAAAEnNSR0IgSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAASc1JHQiBJ\n\tRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAAAAAAAAAAAAFhZWiAAAAAAAADzUQABAAAAARbMWFlaIAAAAAAAAAAAAAAAAAAAAABY\n\tWVogAAAAAAAAb6IAADj1AAADkFhZWiAAAAAAAABimQAAt4UAABjaWFlaIAAAAAAAACSg\n\tAAAPhAAAts9kZXNjAAAAAAAAABZJRUMgaHR0cDovL3d3dy5pZWMuY2gAAAAAAAAAAAAA\n\tABZJRUMgaHR0cDovL3d3dy5pZWMuY2gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAuSUVDIDYxOTY2LTIuMSBEZWZhdWx0\n\tIFJHQiBjb2xvdXIgc3BhY2UgLSBzUkdCAAAAAAAAAAAAAAAuSUVDIDYxOTY2LTIuMSBE\n\tZWZhdWx0IFJHQiBjb2xvdXIgc3BhY2UgLSBzUkdCAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAGRlc2MAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2\n\tNi0yLjEAAAAAAAAAAAAAACxSZWZlcmVuY2UgVmlld2luZyBDb25kaXRpb24gaW4gSUVD\n\tNjE5NjYtMi4xAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAB2aWV3AAAAAAATpP4AFF8u\n\tABDPFAAD7cwABBMLAANcngAAAAFYWVogAAAAAABMCVYAUAAAAFcf521lYXMAAAAAAAAA\n\tAQAAAAAAAAAAAAAAAAAAAAAAAAKPAAAAAnNpZyAAAAAAQ1JUIGN1cnYAAAAAAAAEAAAA\n\tAAUACgAPABQAGQAeACMAKAAtADIANwA7AEAARQBKAE8AVABZAF4AYwBoAG0AcgB3AHwA\n\tgQCGAIsAkACVAJoAnwCkAKkArgCyALcAvADBAMYAywDQANUA2wDgAOUA6wDwAPYA+wEB\n\tAQcBDQETARkBHwElASsBMgE4AT4BRQFMAVIBWQFgAWcBbgF1AXwBgwGLAZIBmgGhAakB\n\tsQG5AcEByQHRAdkB4QHpAfIB+gIDAgwCFAIdAiYCLwI4AkECSwJUAl0CZwJxAnoChAKO\n\tApgCogKsArYCwQLLAtUC4ALrAvUDAAMLAxYDIQMtAzgDQwNPA1oDZgNyA34DigOWA6ID\n\trgO6A8cD0wPgA+wD+QQGBBMEIAQtBDsESARVBGMEcQR+BIwEmgSoBLYExATTBOEE8AT+\n\tBQ0FHAUrBToFSQVYBWcFdwWGBZYFpgW1BcUF1QXlBfYGBgYWBicGNwZIBlkGagZ7BowG\n\tnQavBsAG0QbjBvUHBwcZBysHPQdPB2EHdAeGB5kHrAe/B9IH5Qf4CAsIHwgyCEYIWghu\n\tCIIIlgiqCL4I0gjnCPsJEAklCToJTwlkCXkJjwmkCboJzwnlCfsKEQonCj0KVApqCoEK\n\tmAquCsUK3ArzCwsLIgs5C1ELaQuAC5gLsAvIC+EL+QwSDCoMQwxcDHUMjgynDMAM2Qzz\n\tDQ0NJg1ADVoNdA2ODakNww3eDfgOEw4uDkkOZA5/DpsOtg7SDu4PCQ8lD0EPXg96D5YP\n\tsw/PD+wQCRAmEEMQYRB+EJsQuRDXEPURExExEU8RbRGMEaoRyRHoEgcSJhJFEmQShBKj\n\tEsMS4xMDEyMTQxNjE4MTpBPFE+UUBhQnFEkUahSLFK0UzhTwFRIVNBVWFXgVmxW9FeAW\n\tAxYmFkkWbBaPFrIW1hb6Fx0XQRdlF4kXrhfSF/cYGxhAGGUYihivGNUY+hkgGUUZaxmR\n\tGbcZ3RoEGioaURp3Gp4axRrsGxQbOxtjG4obshvaHAIcKhxSHHscoxzMHPUdHh1HHXAd\n\tmR3DHeweFh5AHmoelB6+HukfEx8+H2kflB+/H+ogFSBBIGwgmCDEIPAhHCFIIXUhoSHO\n\tIfsiJyJVIoIiryLdIwojOCNmI5QjwiPwJB8kTSR8JKsk2iUJJTglaCWXJccl9yYnJlcm\n\thya3JugnGCdJJ3onqyfcKA0oPyhxKKIo1CkGKTgpaymdKdAqAio1KmgqmyrPKwIrNitp\n\tK50r0SwFLDksbiyiLNctDC1BLXYtqy3hLhYuTC6CLrcu7i8kL1ovkS/HL/4wNTBsMKQw\n\t2zESMUoxgjG6MfIyKjJjMpsy1DMNM0YzfzO4M/E0KzRlNJ402DUTNU01hzXCNf02NzZy\n\tNq426TckN2A3nDfXOBQ4UDiMOMg5BTlCOX85vDn5OjY6dDqyOu87LTtrO6o76DwnPGU8\n\tpDzjPSI9YT2hPeA+ID5gPqA+4D8hP2E/oj/iQCNAZECmQOdBKUFqQaxB7kIwQnJCtUL3\n\tQzpDfUPARANER0SKRM5FEkVVRZpF3kYiRmdGq0bwRzVHe0fASAVIS0iRSNdJHUljSalJ\n\t8Eo3Sn1KxEsMS1NLmkviTCpMcky6TQJNSk2TTdxOJU5uTrdPAE9JT5NP3VAnUHFQu1EG\n\tUVBRm1HmUjFSfFLHUxNTX1OqU/ZUQlSPVNtVKFV1VcJWD1ZcVqlW91dEV5JX4FgvWH1Y\n\ty1kaWWlZuFoHWlZaplr1W0VblVvlXDVchlzWXSddeF3JXhpebF69Xw9fYV+zYAVgV2Cq\n\tYPxhT2GiYfViSWKcYvBjQ2OXY+tkQGSUZOllPWWSZedmPWaSZuhnPWeTZ+loP2iWaOxp\n\tQ2maafFqSGqfavdrT2una/9sV2yvbQhtYG25bhJua27Ebx5veG/RcCtwhnDgcTpxlXHw\n\tcktypnMBc11zuHQUdHB0zHUodYV14XY+dpt2+HdWd7N4EXhueMx5KnmJeed6RnqlewR7\n\tY3vCfCF8gXzhfUF9oX4BfmJ+wn8jf4R/5YBHgKiBCoFrgc2CMIKSgvSDV4O6hB2EgITj\n\thUeFq4YOhnKG14c7h5+IBIhpiM6JM4mZif6KZIrKizCLlov8jGOMyo0xjZiN/45mjs6P\n\tNo+ekAaQbpDWkT+RqJIRknqS45NNk7aUIJSKlPSVX5XJljSWn5cKl3WX4JhMmLiZJJmQ\n\tmfyaaJrVm0Kbr5wcnImc951kndKeQJ6unx2fi5/6oGmg2KFHobaiJqKWowajdqPmpFak\n\tx6U4pammGqaLpv2nbqfgqFKoxKk3qamqHKqPqwKrdavprFys0K1ErbiuLa6hrxavi7AA\n\tsHWw6rFgsdayS7LCszizrrQltJy1E7WKtgG2ebbwt2i34LhZuNG5SrnCuju6tbsuu6e8\n\tIbybvRW9j74KvoS+/796v/XAcMDswWfB48JfwtvDWMPUxFHEzsVLxcjGRsbDx0HHv8g9\n\tyLzJOsm5yjjKt8s2y7bMNcy1zTXNtc42zrbPN8+40DnQutE80b7SP9LB00TTxtRJ1MvV\n\tTtXR1lXW2Ndc1+DYZNjo2WzZ8dp22vvbgNwF3IrdEN2W3hzeot8p36/gNuC94UThzOJT\n\t4tvjY+Pr5HPk/OWE5g3mlucf56noMui86Ubp0Opb6uXrcOv77IbtEe2c7ijutO9A78zw\n\tWPDl8XLx//KM8xnzp/Q09ML1UPXe9m32+/eK+Bn4qPk4+cf6V/rn+3f8B/yY/Sn9uv5L\n\t/tz/bf//0h8gISJaJGNsYXNzbmFtZVgkY2xhc3Nlc1xOU0NvbG9yU3BhY2WiIyRcTlND\n\tb2xvclNwYWNlWE5TT2JqZWN00h8gJidXTlNDb2xvcqImJAAIABEAGgAkACkAMgA3AEkA\n\tTABRAFMAWgBgAGsAeAB+AIsAoACnAMgA9QD3APkA+wECAQcBDQEPAREBEw1fDWQNbw14\n\tDYUNiA2VDZ4Now2rAAAAAAAAAgEAAAAAAAAAKAAAAAAAAAAAAAAAAAAADa4=\n\t</data>\n\t<key>ShowActiveProcessArgumentsInTitle</key>\n\t<true/>\n\t<key>ShowCommandKeyInTitle</key>\n\t<false/>\n\t<key>ShowDimensionsInTitle</key>\n\t<true/>\n\t<key>ShowRepresentedURLInTitle</key>\n\t<true/>\n\t<key>ShowRepresentedURLPathInTitle</key>\n\t<true/>\n\t<key>ShowShellCommandInTitle</key>\n\t<false/>\n\t<key>ShowTTYNameInTitle</key>\n\t<false/>\n\t<key>ShowWindowSettingsNameInTitle</key>\n\t<false/>\n\t<key>TextBoldColor</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGmCwwXHR4lVSRudWxs1Q0ODxAR\n\tEhMUFRZcTlNDb21wb25lbnRzVU5TUkdCXE5TQ29sb3JTcGFjZV8QEk5TQ3VzdG9tQ29s\n\tb3JTcGFjZVYkY2xhc3NPEBIwIDAuOTkxNDM5NDAyMSAxIDFPECcwLjEyNzcwMDAzMDgg\n\tMC45OTk3NzQ1MTU2IDAuOTk5MDQ3NTE3OAAQAYACgAXTGBkRGhscVE5TSURVTlNJQ0MQ\n\tB4ADgARPEQxIAAAMSExpbm8CEAAAbW50clJHQiBYWVogB84AAgAJAAYAMQAAYWNzcE1T\n\tRlQAAAAASUVDIHNSR0IAAAAAAAAAAAAAAAAAAPbWAAEAAAAA0y1IUCAgAAAAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAARY3BydAAAAVAAAAAz\n\tZGVzYwAAAYQAAABsd3RwdAAAAfAAAAAUYmtwdAAAAgQAAAAUclhZWgAAAhgAAAAUZ1hZ\n\tWgAAAiwAAAAUYlhZWgAAAkAAAAAUZG1uZAAAAlQAAABwZG1kZAAAAsQAAACIdnVlZAAA\n\tA0wAAACGdmlldwAAA9QAAAAkbHVtaQAAA/gAAAAUbWVhcwAABAwAAAAkdGVjaAAABDAA\n\tAAAMclRSQwAABDwAAAgMZ1RSQwAABDwAAAgMYlRSQwAABDwAAAgMdGV4dAAAAABDb3B5\n\tcmlnaHQgKGMpIDE5OTggSGV3bGV0dC1QYWNrYXJkIENvbXBhbnkAAGRlc2MAAAAAAAAA\n\tEnNSR0IgSUVDNjE5NjYtMi4xAAAAAAAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAFhZWiAA\n\tAAAAAADzUQABAAAAARbMWFlaIAAAAAAAAAAAAAAAAAAAAABYWVogAAAAAAAAb6IAADj1\n\tAAADkFhZWiAAAAAAAABimQAAt4UAABjaWFlaIAAAAAAAACSgAAAPhAAAts9kZXNjAAAA\n\tAAAAABZJRUMgaHR0cDovL3d3dy5pZWMuY2gAAAAAAAAAAAAAABZJRUMgaHR0cDovL3d3\n\tdy5pZWMuY2gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAAAZGVzYwAAAAAAAAAuSUVDIDYxOTY2LTIuMSBEZWZhdWx0IFJHQiBjb2xvdXIgc3Bh\n\tY2UgLSBzUkdCAAAAAAAAAAAAAAAuSUVDIDYxOTY2LTIuMSBEZWZhdWx0IFJHQiBjb2xv\n\tdXIgc3BhY2UgLSBzUkdCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAGRlc2MAAAAAAAAALFJl\n\tZmVyZW5jZSBWaWV3aW5nIENvbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAA\n\tACxSZWZlcmVuY2UgVmlld2luZyBDb25kaXRpb24gaW4gSUVDNjE5NjYtMi4xAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAAAB2aWV3AAAAAAATpP4AFF8uABDPFAAD7cwABBMLAANc\n\tngAAAAFYWVogAAAAAABMCVYAUAAAAFcf521lYXMAAAAAAAAAAQAAAAAAAAAAAAAAAAAA\n\tAAAAAAKPAAAAAnNpZyAAAAAAQ1JUIGN1cnYAAAAAAAAEAAAAAAUACgAPABQAGQAeACMA\n\tKAAtADIANwA7AEAARQBKAE8AVABZAF4AYwBoAG0AcgB3AHwAgQCGAIsAkACVAJoAnwCk\n\tAKkArgCyALcAvADBAMYAywDQANUA2wDgAOUA6wDwAPYA+wEBAQcBDQETARkBHwElASsB\n\tMgE4AT4BRQFMAVIBWQFgAWcBbgF1AXwBgwGLAZIBmgGhAakBsQG5AcEByQHRAdkB4QHp\n\tAfIB+gIDAgwCFAIdAiYCLwI4AkECSwJUAl0CZwJxAnoChAKOApgCogKsArYCwQLLAtUC\n\t4ALrAvUDAAMLAxYDIQMtAzgDQwNPA1oDZgNyA34DigOWA6IDrgO6A8cD0wPgA+wD+QQG\n\tBBMEIAQtBDsESARVBGMEcQR+BIwEmgSoBLYExATTBOEE8AT+BQ0FHAUrBToFSQVYBWcF\n\tdwWGBZYFpgW1BcUF1QXlBfYGBgYWBicGNwZIBlkGagZ7BowGnQavBsAG0QbjBvUHBwcZ\n\tBysHPQdPB2EHdAeGB5kHrAe/B9IH5Qf4CAsIHwgyCEYIWghuCIIIlgiqCL4I0gjnCPsJ\n\tEAklCToJTwlkCXkJjwmkCboJzwnlCfsKEQonCj0KVApqCoEKmAquCsUK3ArzCwsLIgs5\n\tC1ELaQuAC5gLsAvIC+EL+QwSDCoMQwxcDHUMjgynDMAM2QzzDQ0NJg1ADVoNdA2ODakN\n\tww3eDfgOEw4uDkkOZA5/DpsOtg7SDu4PCQ8lD0EPXg96D5YPsw/PD+wQCRAmEEMQYRB+\n\tEJsQuRDXEPURExExEU8RbRGMEaoRyRHoEgcSJhJFEmQShBKjEsMS4xMDEyMTQxNjE4MT\n\tpBPFE+UUBhQnFEkUahSLFK0UzhTwFRIVNBVWFXgVmxW9FeAWAxYmFkkWbBaPFrIW1hb6\n\tFx0XQRdlF4kXrhfSF/cYGxhAGGUYihivGNUY+hkgGUUZaxmRGbcZ3RoEGioaURp3Gp4a\n\txRrsGxQbOxtjG4obshvaHAIcKhxSHHscoxzMHPUdHh1HHXAdmR3DHeweFh5AHmoelB6+\n\tHukfEx8+H2kflB+/H+ogFSBBIGwgmCDEIPAhHCFIIXUhoSHOIfsiJyJVIoIiryLdIwoj\n\tOCNmI5QjwiPwJB8kTSR8JKsk2iUJJTglaCWXJccl9yYnJlcmhya3JugnGCdJJ3onqyfc\n\tKA0oPyhxKKIo1CkGKTgpaymdKdAqAio1KmgqmyrPKwIrNitpK50r0SwFLDksbiyiLNct\n\tDC1BLXYtqy3hLhYuTC6CLrcu7i8kL1ovkS/HL/4wNTBsMKQw2zESMUoxgjG6MfIyKjJj\n\tMpsy1DMNM0YzfzO4M/E0KzRlNJ402DUTNU01hzXCNf02NzZyNq426TckN2A3nDfXOBQ4\n\tUDiMOMg5BTlCOX85vDn5OjY6dDqyOu87LTtrO6o76DwnPGU8pDzjPSI9YT2hPeA+ID5g\n\tPqA+4D8hP2E/oj/iQCNAZECmQOdBKUFqQaxB7kIwQnJCtUL3QzpDfUPARANER0SKRM5F\n\tEkVVRZpF3kYiRmdGq0bwRzVHe0fASAVIS0iRSNdJHUljSalJ8Eo3Sn1KxEsMS1NLmkvi\n\tTCpMcky6TQJNSk2TTdxOJU5uTrdPAE9JT5NP3VAnUHFQu1EGUVBRm1HmUjFSfFLHUxNT\n\tX1OqU/ZUQlSPVNtVKFV1VcJWD1ZcVqlW91dEV5JX4FgvWH1Yy1kaWWlZuFoHWlZaplr1\n\tW0VblVvlXDVchlzWXSddeF3JXhpebF69Xw9fYV+zYAVgV2CqYPxhT2GiYfViSWKcYvBj\n\tQ2OXY+tkQGSUZOllPWWSZedmPWaSZuhnPWeTZ+loP2iWaOxpQ2maafFqSGqfavdrT2un\n\ta/9sV2yvbQhtYG25bhJua27Ebx5veG/RcCtwhnDgcTpxlXHwcktypnMBc11zuHQUdHB0\n\tzHUodYV14XY+dpt2+HdWd7N4EXhueMx5KnmJeed6RnqlewR7Y3vCfCF8gXzhfUF9oX4B\n\tfmJ+wn8jf4R/5YBHgKiBCoFrgc2CMIKSgvSDV4O6hB2EgITjhUeFq4YOhnKG14c7h5+I\n\tBIhpiM6JM4mZif6KZIrKizCLlov8jGOMyo0xjZiN/45mjs6PNo+ekAaQbpDWkT+RqJIR\n\tknqS45NNk7aUIJSKlPSVX5XJljSWn5cKl3WX4JhMmLiZJJmQmfyaaJrVm0Kbr5wcnImc\n\t951kndKeQJ6unx2fi5/6oGmg2KFHobaiJqKWowajdqPmpFakx6U4pammGqaLpv2nbqfg\n\tqFKoxKk3qamqHKqPqwKrdavprFys0K1ErbiuLa6hrxavi7AAsHWw6rFgsdayS7LCsziz\n\trrQltJy1E7WKtgG2ebbwt2i34LhZuNG5SrnCuju6tbsuu6e8IbybvRW9j74KvoS+/796\n\tv/XAcMDswWfB48JfwtvDWMPUxFHEzsVLxcjGRsbDx0HHv8g9yLzJOsm5yjjKt8s2y7bM\n\tNcy1zTXNtc42zrbPN8+40DnQutE80b7SP9LB00TTxtRJ1MvVTtXR1lXW2Ndc1+DYZNjo\n\t2WzZ8dp22vvbgNwF3IrdEN2W3hzeot8p36/gNuC94UThzOJT4tvjY+Pr5HPk/OWE5g3m\n\tlucf56noMui86Ubp0Opb6uXrcOv77IbtEe2c7ijutO9A78zwWPDl8XLx//KM8xnzp/Q0\n\t9ML1UPXe9m32+/eK+Bn4qPk4+cf6V/rn+3f8B/yY/Sn9uv5L/tz/bf//0h8gISJaJGNs\n\tYXNzbmFtZVgkY2xhc3Nlc1xOU0NvbG9yU3BhY2WiIyRcTlNDb2xvclNwYWNlWE5TT2Jq\n\tZWN00h8gJidXTlNDb2xvcqImJAAIABEAGgAkACkAMgA3AEkATABRAFMAWgBgAGsAeAB+\n\tAIsAoACnALwA5gDoAOoA7ADzAPgA/gEAAQIBBA1QDVUNYA1pDXYNeQ2GDY8NlA2cAAAA\n\tAAAAAgEAAAAAAAAAKAAAAAAAAAAAAAAAAAAADZ8=\n\t</data>\n\t<key>TextColor</key>\n\t<data>\n\tYnBsaXN0MDDUAQIDBAUGBwpYJHZlcnNpb25ZJGFyY2hpdmVyVCR0b3BYJG9iamVjdHMS\n\tAAGGoF8QD05TS2V5ZWRBcmNoaXZlctEICVRyb290gAGmCwwXHR4lVSRudWxs1Q0ODxAR\n\tEhMUFRZcTlNDb21wb25lbnRzVU5TUkdCXE5TQ29sb3JTcGFjZV8QEk5TQ3VzdG9tQ29s\n\tb3JTcGFjZVYkY2xhc3NPEBIwLjk5OTk5NjAwNjUgMSAxIDFPEBEwLjk5OTk5NTA1Mjgg\n\tMSAxABABgAKABdMYGREaGxxUTlNJRFVOU0lDQxAHgAOABE8RDEgAAAxITGlubwIQAABt\n\tbnRyUkdCIFhZWiAHzgACAAkABgAxAABhY3NwTVNGVAAAAABJRUMgc1JHQgAAAAAAAAAA\n\tAAAAAAAA9tYAAQAAAADTLUhQICAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAABFjcHJ0AAABUAAAADNkZXNjAAABhAAAAGx3dHB0AAAB8AAA\n\tABRia3B0AAACBAAAABRyWFlaAAACGAAAABRnWFlaAAACLAAAABRiWFlaAAACQAAAABRk\n\tbW5kAAACVAAAAHBkbWRkAAACxAAAAIh2dWVkAAADTAAAAIZ2aWV3AAAD1AAAACRsdW1p\n\tAAAD+AAAABRtZWFzAAAEDAAAACR0ZWNoAAAEMAAAAAxyVFJDAAAEPAAACAxnVFJDAAAE\n\tPAAACAxiVFJDAAAEPAAACAx0ZXh0AAAAAENvcHlyaWdodCAoYykgMTk5OCBIZXdsZXR0\n\tLVBhY2thcmQgQ29tcGFueQAAZGVzYwAAAAAAAAASc1JHQiBJRUM2MTk2Ni0yLjEAAAAA\n\tAAAAAAAAABJzUkdCIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAWFlaIAAAAAAAAPNRAAEAAAABFsxYWVogAAAA\n\tAAAAAAAAAAAAAAAAAFhZWiAAAAAAAABvogAAOPUAAAOQWFlaIAAAAAAAAGKZAAC3hQAA\n\tGNpYWVogAAAAAAAAJKAAAA+EAAC2z2Rlc2MAAAAAAAAAFklFQyBodHRwOi8vd3d3Lmll\n\tYy5jaAAAAAAAAAAAAAAAFklFQyBodHRwOi8vd3d3LmllYy5jaAAAAAAAAAAAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABkZXNjAAAAAAAAAC5JRUMgNjE5\n\tNjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAAAAAAAC5J\n\tRUMgNjE5NjYtMi4xIERlZmF1bHQgUkdCIGNvbG91ciBzcGFjZSAtIHNSR0IAAAAAAAAA\n\tAAAAAAAAAAAAAAAAAAAAZGVzYwAAAAAAAAAsUmVmZXJlbmNlIFZpZXdpbmcgQ29uZGl0\n\taW9uIGluIElFQzYxOTY2LTIuMQAAAAAAAAAAAAAALFJlZmVyZW5jZSBWaWV3aW5nIENv\n\tbmRpdGlvbiBpbiBJRUM2MTk2Ni0yLjEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHZp\n\tZXcAAAAAABOk/gAUXy4AEM8UAAPtzAAEEwsAA1yeAAAAAVhZWiAAAAAAAEwJVgBQAAAA\n\tVx/nbWVhcwAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAo8AAAACc2lnIAAAAABDUlQg\n\tY3VydgAAAAAAAAQAAAAABQAKAA8AFAAZAB4AIwAoAC0AMgA3ADsAQABFAEoATwBUAFkA\n\tXgBjAGgAbQByAHcAfACBAIYAiwCQAJUAmgCfAKQAqQCuALIAtwC8AMEAxgDLANAA1QDb\n\tAOAA5QDrAPAA9gD7AQEBBwENARMBGQEfASUBKwEyATgBPgFFAUwBUgFZAWABZwFuAXUB\n\tfAGDAYsBkgGaAaEBqQGxAbkBwQHJAdEB2QHhAekB8gH6AgMCDAIUAh0CJgIvAjgCQQJL\n\tAlQCXQJnAnECegKEAo4CmAKiAqwCtgLBAssC1QLgAusC9QMAAwsDFgMhAy0DOANDA08D\n\tWgNmA3IDfgOKA5YDogOuA7oDxwPTA+AD7AP5BAYEEwQgBC0EOwRIBFUEYwRxBH4EjASa\n\tBKgEtgTEBNME4QTwBP4FDQUcBSsFOgVJBVgFZwV3BYYFlgWmBbUFxQXVBeUF9gYGBhYG\n\tJwY3BkgGWQZqBnsGjAadBq8GwAbRBuMG9QcHBxkHKwc9B08HYQd0B4YHmQesB78H0gfl\n\tB/gICwgfCDIIRghaCG4IggiWCKoIvgjSCOcI+wkQCSUJOglPCWQJeQmPCaQJugnPCeUJ\n\t+woRCicKPQpUCmoKgQqYCq4KxQrcCvMLCwsiCzkLUQtpC4ALmAuwC8gL4Qv5DBIMKgxD\n\tDFwMdQyODKcMwAzZDPMNDQ0mDUANWg10DY4NqQ3DDd4N+A4TDi4OSQ5kDn8Omw62DtIO\n\t7g8JDyUPQQ9eD3oPlg+zD88P7BAJECYQQxBhEH4QmxC5ENcQ9RETETERTxFtEYwRqhHJ\n\tEegSBxImEkUSZBKEEqMSwxLjEwMTIxNDE2MTgxOkE8UT5RQGFCcUSRRqFIsUrRTOFPAV\n\tEhU0FVYVeBWbFb0V4BYDFiYWSRZsFo8WshbWFvoXHRdBF2UXiReuF9IX9xgbGEAYZRiK\n\tGK8Y1Rj6GSAZRRlrGZEZtxndGgQaKhpRGncanhrFGuwbFBs7G2MbihuyG9ocAhwqHFIc\n\texyjHMwc9R0eHUcdcB2ZHcMd7B4WHkAeah6UHr4e6R8THz4faR+UH78f6iAVIEEgbCCY\n\tIMQg8CEcIUghdSGhIc4h+yInIlUigiKvIt0jCiM4I2YjlCPCI/AkHyRNJHwkqyTaJQkl\n\tOCVoJZclxyX3JicmVyaHJrcm6CcYJ0kneierJ9woDSg/KHEooijUKQYpOClrKZ0p0CoC\n\tKjUqaCqbKs8rAis2K2krnSvRLAUsOSxuLKIs1y0MLUEtdi2rLeEuFi5MLoIuty7uLyQv\n\tWi+RL8cv/jA1MGwwpDDbMRIxSjGCMbox8jIqMmMymzLUMw0zRjN/M7gz8TQrNGU0njTY\n\tNRM1TTWHNcI1/TY3NnI2rjbpNyQ3YDecN9c4FDhQOIw4yDkFOUI5fzm8Ofk6Njp0OrI6\n\t7zstO2s7qjvoPCc8ZTykPOM9Ij1hPaE94D4gPmA+oD7gPyE/YT+iP+JAI0BkQKZA50Ep\n\tQWpBrEHuQjBCckK1QvdDOkN9Q8BEA0RHRIpEzkUSRVVFmkXeRiJGZ0arRvBHNUd7R8BI\n\tBUhLSJFI10kdSWNJqUnwSjdKfUrESwxLU0uaS+JMKkxyTLpNAk1KTZNN3E4lTm5Ot08A\n\tT0lPk0/dUCdQcVC7UQZRUFGbUeZSMVJ8UsdTE1NfU6pT9lRCVI9U21UoVXVVwlYPVlxW\n\tqVb3V0RXklfgWC9YfVjLWRpZaVm4WgdaVlqmWvVbRVuVW+VcNVyGXNZdJ114XcleGl5s\n\tXr1fD19hX7NgBWBXYKpg/GFPYaJh9WJJYpxi8GNDY5dj62RAZJRk6WU9ZZJl52Y9ZpJm\n\t6Gc9Z5Nn6Wg/aJZo7GlDaZpp8WpIap9q92tPa6dr/2xXbK9tCG1gbbluEm5rbsRvHm94\n\tb9FwK3CGcOBxOnGVcfByS3KmcwFzXXO4dBR0cHTMdSh1hXXhdj52m3b4d1Z3s3gReG54\n\tzHkqeYl553pGeqV7BHtje8J8IXyBfOF9QX2hfgF+Yn7CfyN/hH/lgEeAqIEKgWuBzYIw\n\tgpKC9INXg7qEHYSAhOOFR4Wrhg6GcobXhzuHn4gEiGmIzokziZmJ/opkisqLMIuWi/yM\n\tY4zKjTGNmI3/jmaOzo82j56QBpBukNaRP5GokhGSepLjk02TtpQglIqU9JVflcmWNJaf\n\tlwqXdZfgmEyYuJkkmZCZ/JpomtWbQpuvnByciZz3nWSd0p5Anq6fHZ+Ln/qgaaDYoUeh\n\ttqImopajBqN2o+akVqTHpTilqaYapoum/adup+CoUqjEqTepqaocqo+rAqt1q+msXKzQ\n\trUStuK4trqGvFq+LsACwdbDqsWCx1rJLssKzOLOutCW0nLUTtYq2AbZ5tvC3aLfguFm4\n\t0blKucK6O7q1uy67p7whvJu9Fb2Pvgq+hL7/v3q/9cBwwOzBZ8Hjwl/C28NYw9TEUcTO\n\txUvFyMZGxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzR\n\tvtI/0sHTRNPG1EnUy9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i\n\t3ynfr+A24L3hROHM4lPi2+Nj4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/vs\n\thu0R7ZzuKO6070DvzPBY8OXxcvH/8ozzGfOn9DT0wvVQ9d72bfb794r4Gfio+Tj5x/pX\n\t+uf7d/wH/Jj9Kf26/kv+3P9t///SHyAhIlokY2xhc3NuYW1lWCRjbGFzc2VzXE5TQ29s\n\tb3JTcGFjZaIjJFxOU0NvbG9yU3BhY2VYTlNPYmplY3TSHyAmJ1dOU0NvbG9yoiYkAAgA\n\tEQAaACQAKQAyADcASQBMAFEAUwBaAGAAawB4AH4AiwCgAKcAvADQANIA1ADWAN0A4gDo\n\tAOoA7ADuDToNPw1KDVMNYA1jDXANeQ1+DYYAAAAAAAACAQAAAAAAAAAoAAAAAAAAAAAA\n\tAAAAAAANiQ==\n\t</data>\n\t<key>UseBrightBold</key>\n\t<true/>\n\t<key>VisualBell</key>\n\t<false/>\n\t<key>VisualBellOnlyWhenMuted</key>\n\t<false/>\n\t<key>WindowTitle</key>\n\t<string>Term</string>\n\t<key>columnCount</key>\n\t<integer>215</integer>\n\t<key>name</key>\n\t<string>Hari</string>\n\t<key>rowCount</key>\n\t<integer>64</integer>\n\t<key>shellExitAction</key>\n\t<integer>1</integer>\n\t<key>type</key>\n\t<string>Window Settings</string>\n</dict>\n</plist>\n"
  },
  {
    "path": "setup/R-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2014-12-29 21:10:12 +0000 (Mon, 29 Dec 2014)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# R packages to install from CRAN\n\narules # Apriori Association Rules\ndata.table\ndevtools\ne1071 # Naiive Bayes\nggplot2\nhttr\njsonlite\nreshape\nRJava\nROCR # used with Naiive Bayes\nrpart # Decision Trees\nrpart.plot\nslidify\n"
  },
  {
    "path": "setup/README.md",
    "content": "# Setup\n\n- Package lists used by the `make` build at the top level\n  - OS packages - Mac OS X / macOS and Linux distributions (RHEL/CentOS, Debian/Ubuntu, Alpine)\n  - language libraries for Python, Perl, Golang\n- Simple installation scripts are found in the `../install/install_*.sh` directory for common technologies that make it\n  very easy to install software\n- the top level `Makefile` installs these packages using `make` for the core CI/CD list or `make desktop` for all the\n  extra goodies you might use on your Desktop or Laptop\n- configurations in this directory and `../configs` directory make it easy to set up a new laptop / workstation\n  every time you work for a new company\n- `make link` symlinks all the `../configs` listed in the `files.txt` in this directory\n\nAlmost nothing should need to be done manually, a single `make desktop` command and you should be up and running.\n\n## Mac XCode for Developers\n\nRun `git` - it will automatically prompt to install XCode if needed. If git runs, XCode is likely already installed.\n"
  },
  {
    "path": "setup/alternatives_set_python.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-02-21 09:52:05 +0000 (Fri, 21 Feb 2020)\n#  forked from pylib's Makefile from:\n#  Original Date: 2013-01-06 15:45:00 +0000 (Sun, 06 Jan 2013)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\n#$sudo ln -sv -- `type -P python2` /usr/local/bin/python\n\nalternatives(){\n    # not available on Alpine\n    if type -P alternatives &>/dev/null; then\n        $sudo alternatives --set python \"$@\"\n    else\n        $sudo ln -sv -- \"$1\" /usr/local/bin/python\n    fi\n}\n\nif ! type -P python &>/dev/null; then\n    set +e\n    python2=\"$(type -P python2 2>/dev/null)\"\n    python3=\"$(type -P python3 2>/dev/null)\"\n    set -e\n    if [ -n \"$python3\" ]; then\n        echo \"alternatives: setting python -> $python3\"\n        # using function wrapper instead for Alpine\n        #$sudo alternatives --set python \"$python3\"\n        alternatives \"$python3\"\n    elif [ -n \"$python2\" ]; then\n        echo \"alternatives: setting python -> $python2\"\n        # using function wrapper instead for Alpine\n        #$sudo alternatives --set python \"$python2\"\n        alternatives \"$python2\"\n    fi\nfi\n\nif ! type -P pip; then\n    set +e\n    pip2=\"$(type -P pip2 2>/dev/null)\"\n    pip3=\"$(type -P pip3 2>/dev/null)\"\n    set -e\n    if [ -f /usr/local/bin/pip ]; then\n        echo \"/usr/local/bin/pip already exists, not symlinking - check your \\$PATH includes /usr/local/bin (\\$PATH = $PATH)\"\n    elif [ -n \"$pip3\" ]; then\n        $sudo ln -sv -- \"$pip3\" /usr/local/bin/pip\n    elif [ -n \"$pip2\" ]; then\n        $sudo ln -sv -- \"$pip2\" /usr/local/bin/pip\n    else\n        $sudo easy_install pip || :\n    fi\nfi\n\necho\npython -V\necho\n\"$srcdir/pip_fix_version.sh\"\npip -V\necho\n"
  },
  {
    "path": "setup/apk-packages-desktop.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                      Alpine Desktop Package Requirements\n# ============================================================================ #\n\nnetcat-openbsd\nperl-html-parser\nskopeo\nsteghide\ntokei  # lines of code counter\nxmlstartlet\n"
  },
  {
    "path": "setup/apk-packages-optional.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          Alpine Package Requirements\n# ============================================================================ #\n\n# Packages that don't error if failing to install\n\n# Python is a mess and breaks with this on latest Alpine 3, so do as best effort\n#\n#   python (missing):\n# ERROR: unsatisfiable constraints:\n#     required by: world[python]\n\npython\npython2\npython3\npy-pip\npy2-pip\npy3-pip\n\n# available from Alpine 3.20 only\npre-commit\n"
  },
  {
    "path": "setup/apk-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          Alpine Package Requirements\n# ============================================================================ #\n\nbash\nbc\ncoreutils\ncurl\nfindutils\ngcc\ngit\ngo\ngrep\njq\nlibxml2-utils  # needed for xmllint\nmake\nopenssh-client\nparallel\nperl\nperl-app-cpanminus\nruby-dev # to build Travis CI gem\nwget\nwhich\nzip\nunzip\n\n# causes errors now so moved to apk-packages-optional.txt to allow failures\n#\n#   python (missing):\n# ERROR: unsatisfiable constraints:\n#     required by: world[python]\n#python\n#py-pip\n"
  },
  {
    "path": "setup/atom-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-03-29 00:54:51 +0000 (Sun, 29 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                           A t o m   P a c k a g e s\n# ============================================================================ #\n\nlinter-jenkins\n"
  },
  {
    "path": "setup/bootstrap.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-16 10:33:03 +0100 (Wed, 16 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Alpine / Wget:\n#\n#   wget -O- https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/setup/bootstrap.sh | sh\n#\n# Curl:\n#\n#   curl https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/setup/bootstrap.sh | sh\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(dirname \"$0\")\"\n\nrepo=\"https://github.com/HariSekhon/DevOps-Bash-tools\"\n\ndirectory=\"bash-tools\"\n\nsudo=\"\"\n[ \"$(whoami)\" = \"root\" ] || sudo=sudo\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    echo \"Bootstrapping on Mac OS X:  $repo\"\n    if ! type brew >/dev/null 2>&1; then\n        curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install | $sudo ruby\n    fi\nelif [ \"$(uname -s)\" = Linux ]; then\n    echo \"Bootstrapping on Linux:  $repo\"\n    if type apk >/dev/null 2>&1; then\n        $sudo apk --no-cache add bash git make curl wget\n    elif type apt-get >/dev/null 2>&1; then\n        if [ -n \"${CI:-}\" ]; then\n            export DEBIAN_FRONTEND=noninteractive\n        fi\n        opts=\"-o DPkg::Lock::Timeout=1200\"\n        if [ -z \"${PS1:-}\" ]; then\n            opts=\"$opts -qq\"\n        fi\n        $sudo apt-get update  $opts\n        $sudo apt-get install $opts -y git make curl wget --no-install-recommends\n    elif type yum >/dev/null 2>&1; then\n        if grep -qi 'NAME=.*CentOS' /etc/*release; then\n            echo \"CentOS EOL detected, replacing yum base URL to vault to re-enable package installs\"\n            $sudo sed -i 's/^[[:space:]]*mirrorlist/#mirrorlist/' /etc/yum.repos.d/CentOS-Linux-*\n            $sudo sed -i 's|^#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|' /etc/yum.repos.d/CentOS-Linux-*\n        fi\n        $sudo yum install -y git make curl wget\n    else\n        echo \"Package Manager not found on Linux, cannot bootstrap\"\n        exit 1\n    fi\nelse\n    echo \"Only Mac & Linux are supported for conveniently bootstrapping all install scripts at this time\"\n    exit 1\nfi\n\nif [ \"${srcdir##*/}\" = setup ]; then\n    cd \"$srcdir/..\"\nelse\n    # if this is an empty directory eg. a cache mount, then remove it to get a proper checkout\n    rmdir \"$directory\" 2>/dev/null || :\n    if [ -d \"$directory\" ]; then\n        cd \"$directory\"\n        git pull\n    else\n        git clone \"$repo\" \"$directory\"\n        cd \"$directory\"\n    fi\nfi\n\nif [ -z \"${NO_MAKE:-}\" ]; then\n    make install\nfi\n"
  },
  {
    "path": "setup/brew-packages-desktop-casks.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2011-06-01 16:58:33 +0000 (Wed, 01 Jun 2011)\n#\n\n# run: CASK=1 ../packages/brew_install_packages.sh brew-packages-desktop-casks.txt\n\nalfred\nanaconda\natom\nbarrier\nblackhole-2ch  # to create multi-output sound device to shazam songs from shows while using headphones\nccmenu\n#claude-code  # use native installer which auto-updates: curl -fsSL https://claude.ai/install.sh | bash\ncodeship/taps/jet\ndocker\nflux-app\n#insomnia  # Kong Rest API testing client desktop app\n#iterm2\n#firefox-developer-edition\ngoogle-chrome\ngoogle-cloud-sdk\n#google-earth-pro\ngrandperspective  # like Disk Inventory X - GUI disk space analysis\nhammerspoon  # system event handler\n#intellij-idea-ce  # in main packages now\nkeepassxc\nkeka\nminikube\nminishift\nmultipass  # instant Ubuntu VMs\n#omnidisksweeper  # GUI disk space analysis\npodman\npostman\npowershell\nprotonvpn\nraycast\nspotify\ntransmission\ntor-browser\nvagrant\nvirtualbox\nvnc-viewer\nwireshark  # installs GUI, without cask only install CLI utils like tshark\n"
  },
  {
    "path": "setup/brew-packages-desktop-taps.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-12-19 15:08:01 +0000 (Thu, 19 Dec 2019)\n#\n\n# run: TAP=1 ../packages/brew_install_packages.sh brew-packages-desktop-taps.txt\n\n# tap                   package\n\nhashicorp/tap           vault\nhashicorp/tap           packer\n\nwarrensbox/tap          tgswitch\n\ncoder/coder             coder\n\nariga/tap               atlas\n\nCheckmarx/tap           kics\n\nFairwindsOps/tap        polaris\nFairwindsOps/tap        pluto\nFairwindsOps/tap        nova\n\nderailed/k9s            k9s\n\n#anchore/grype           grype\n#anchore/syft            syft\n\n# do via install script to load token\n#buildkite/buildkite     buildkite\n\n# BuildKite CLI - for running builds locally\nbuildkite/cli           bk\ndbcli/tap               athenacli\nwallix/awless           awless\n#wercker/wercker         wercker-cli  # EOL - github repo password protected now, causes install to hang on username/password prompt\ngolangci/tap            golangci-lint\n\n#fluxcd/tap              flux\n\n#codefresh-io/cli        codefresh\n#sambadevi/powerlevel9k  powerlevel9k\n\n#tektoncd/tools         tektoncd-cli\n\n# don't install this, see install_d2.sh for details why\n#terrastruct/tap         tala\n\n# analyzes an AWS EKS cluster for upgrade requirements\nclowdhaus/taps          eksup\n\nteamookla/speedtest     speedtest  # https://www.speedtest.net/\n\n# to download different versions of XCode from the command line\nxcodesorg/made          xcodes\n"
  },
  {
    "path": "setup/brew-packages-desktop.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2011-06-01 16:58:33 +0000 (Wed, 01 Jun 2011)\n#\n\n# Things to install by hand\n#\n# Xcode - running 'git' is enough to trigger the install\n#\n# Many of these are now available in brew casks - TODO: test and move to brew-packages-desktop-casks.txt\n#\n# Chrome\n# Docker Desktop\n# Google Drive\n# RStudio\n# Sophos Anti-Virus\n# ClamXav\n# Colloquay\n# JConsole\n# Reggy\n# Remote Desktop Connection\n# Skype\n# Spotify\n# Sublime Text 2\n# SynergyOSX\n# TunnelBlick\n# VLC\n# WindowWrangler?\n# Wireshark\n# iPhone Explorer\n\n# Cask installed projects are in adjacent file brew-packages-desktop-casks.txt\n\n# Install from formula on GitHub\nhttps://raw.githubusercontent.com/jingweno/spotctl/master/Formula/spotctl.rb\n\n# Install HomeBrew Desktop packages - one per line\nack\nact  # runs GitHub Actions locally\nagg  # converts asciicast terminal recording to gif\namazon-ecs-cli\n#amazon-sam-cli  # no longer found\nansible\nansible-lint\naria2  # optionally used by xcodes for downloading, supports resuming and segmented downloading\narchey4  # displays system info in text\nargocd\nasciinema  # records terminal to asciicast\nasdf\n#atomicparsley  # mp4 cli tool\nautoconf\nautojump\nautossh\n#awsume\n#aws-iam-authenticator\nbash  # version 5, not old version 3 from Apple due to licensing\n#bash-completion  # slows down spawning new shells which I do frequently\n#bazel\nbazelisk  # bazel wrapper downloader like mvnw/gradlew, addes 'bazel' to $PATH so conflicts with bazel package above\nbrew-cask-completion\n#duckdb  # open-source in-memory columnar RDBMS, see Knowledge Base for more details: https://github.com/HariSekhon/Knowledge-Base\ncfssl\n#cfn-lint\ncheckov\ncheckstyle\ncircleci\nclamav\n#clair  # container vulnerability scanner by quay.io - this is the server, not clairctl\ncliclick\ncmatrix\ncloc  # count lines of code - code / comment / blank\ncolordiff\ncolima  # CLI Docker container runtime (use 'docker' client after 'colima start')\ncosign\ncowsay\n#coreutils # in brew-packages.txt\ncsshx\n#ctags\nctop  # container top\ncursor  # AI editor\ncursor-cli\n#cyberduck  # GUI client for FTP, SFTP, WebDAV, Amazon S3, Google Drive etc.\nd2  # declarative diagram language\ndateutils  # strptime, dateadd, dateconv, datediff, dategrep, dateseq, datesort, datetest, datezone - https://www.fresse.org/dateutils/\ndbeaver-community\ndhcping\ndhcpdump\ndialog  # create a curses interactive menu\ndiff-so-fancy\ndiff-pdf\n#diffutils  # GNU diff, diff3, sdiff, cmp commands - links to /opt/homebrew/bin/ and depending on $PATH ordering may supplant Mac's BSD diff\ndirenv\ndisk-inventory-x\ndive  # analyze docker image layers\ndnsmasq\ndnstop\ndocker-completion\ndocker-compose-completion\ndoctl  # Digital Ocean CLI\ndos2unix\ndrone-cli\nduti  # displays or set the default application for a given file extension or URL\ne2fsprogs\n#ec2-api-tools\neksctl\nentr  # execute commands upon file changes\nexpect\netcd\nexiftool  # view image metadata\nexiv2     # view image metadata\n#ext4fuse  # to read Linux partitions eg. on external hard drives\nfastlane  # for Mobile CI/CD development\nffmpeg\n#filezilla  # GUI ftp client - not found\nfio  # disk I/O speed test\nfirebase-cli\nfish\nflyscrape\nfortune\nfping  # scriptable fast ping for multiple hosts\nfswatch\nfzf\ngcc\ngem-completion\n#gh  # GitHub CLI - moved to core packages\ngimp\n#git  # comes with macOS XCode\ngit-credential-manager  # prompts for HTTPS creds in browser and caches them locally to HTTPS checkouts\ngit-lfs\ngit-filter-repo  # recommended to use instead of 'git filter-branch' in its man page\n#gnu-sed  # in brew-packages.txt\n#gnupg\ngnupg2\ngnuplot\ngping  # graphs pings in terminal interactive ascii art\ngradle\n#gradle-completion  # forces bash-completion which slows down my shells\n#grafana\ngraphviz  # for the 'dot' command (use with terraform graph), and also github.com/HariSekhon/Diagrams-as-Code\ngron  # flattens JSON to be greppable\n#groovy  # see near end\n#groovysdk  # see near end\nhadolint\nhaproxy\nhelm\nhexyl  # hex command line viewer - https://github.com/sharkdp/hexyl\nhtop\nhttping  # ping-like tool for HTTP requests\nhttp-toolkit  # HTTP intercepting debugger\nhub\nhugo  # static website generator - https://gohugo.io/getting-started/quick-start/\nid3v2  # mp3 tag cli editor\n#id3lib  # mp3 tag manipulation\niftop\nimagemagick\n#inetutils  # GNU telnet, ftp, tftp, inetd, syslog, ping, traceroute, whois - prefixed with g\ninfracost  # Cloud cost estimator for Terraform code\nipcalc\nioping  # monitor I/O latency in real-time\niperf\nipython\n#isc-dhcp  # the classic DHCP daemon, use for bootstrapping PXE installing Debian Linux or similar as per https://github.com/HariSekhon/Knowledge-Base/blob/main/dhcp.md\nintellij-idea-ce\njsonlint  # in brew-packages.txt\njenkins-x/jx/jx\njnv  # json interactive viewer (pipe kubectl into it)\n#jpeg\n#jq  # in brew-packages.txt\njruby\njython\nk3d  # wrapper for k3s\nk6\nkind  # Kubernetes in Docker\nkompose\nknative/client/kn\nknative-sandbox/kn-plugins/quickstart\nkops\nkubecolor\n#kubectl  # old version, download binary or use GCloud SDK version\nkubernetes-cli\nkubeseal  # Bitnami Sealed Secrets\nkubevious  # validation linter for Kubernetes yaml\nkustomize\n#lame  # mp3 encoder\nlaunchctl-completion\n#libpng\n#libtiff\n#libusb\nlolcat\nluacheck  # lua script linting\nlynx\n#lzo\n#lz4\nm4  # classic macro processor\n#makedepend  # Makefile dependencies for C\nmas  # command line interface to the App Store to install/upgrade apps\n#maven\nmaven-completion\nmaven-shell\n#md5sha1sum  # clashes with coreutils\nmedia-info  # read mp3 tags\nmercurial\n#mermaid-cli  # old version - see instead https://github.com/HariSekhon/Knowledge-Base/blob/main/mermaidjs.md\nubuntu/microk8s/microk8s\nmidnight-commander\nminio\n#mitmproxy\nmkcert  # simple tool to make locally trusted development certificates - used by spotify_api_token.sh for https localhost callback\nmosh\n# nagios-plugins clashes\nmonitoring-plugins\nmplayer\nmpssh\nmpv\nmtr\nmycli\nmysql\nncdu  # ncurse disk usage - interactive disk space analysis\nneovim\nnetcat\nngrep\nnmap\n#node  # NodeJS\nnss\n#ntfstool  # for accessing NTFS volumes - unsigned requires Security approval and installing kernel driver\n#octave\n#oniguruma  # regex library\n#openjpeg\nopenai-whisper\n#osxfuse  # required to be installed before 'ext4fuse' package\np7zip\npacker-completion\npandoc\n#parallel  # in brew-packages.txt\nparquet-tools\n#pcre\n#pcre2\nperltidy\npgcli\npgformatter\npigz  # parallel gzip\npinentry\npinentry-mac\npip-completion\npipenv\npipx\n#pk11-kit\npluto  # by FairwindsOps - detect outdated Kubernetes API objects\npostgresql\npre-commit\nprestosql\n#protobuf\npssh\npstree\npup  # parse html from command line\npwgen\n#pygments  # in pip-packages-desktop.txt\n#python  # in brew-packages.txt\n#pipenv\n#pyenv\nqemu\nr\nrancher-cli\nrbenv\n#rclone  # like rsync for S3 / GCS / SFTP / HDFS / SMB / HTTP\nrdesktop\n#redis\n#readline  # in brew-packages.txt\n#restic  # backup program, tracks versions, backups to a local repo dir / SFTP / Restic server http(s) / S3 / GCS / MinIO / Rclone\nrlwrap  # gives readline wrapping support (commonly used with Oracle SQL*Plus)\nrsync  # newer rsync 3 than macOS rsync 2 - has new/different switches\n#rtmpdump\nruby  # Mac ruby is too old to install Travis CI\nruby-completion\n#sbt  # see near end\n#scala  # see near end\nscreen  # get newer version for -Q switch, will use different sockets directory\ns3cmd\nscalastyle\nscc  # Succinct Code Counter - lines and Cocomo man effort estimate\nsemgrep\nsiege\n#six  # Python 2 to 3 conversion\nskaffold\nskopeo\nshpotify\nsmartmontools\nsnappy\nsonar-completion\nsonar-scanner\nsqlite\nsqlfluff  # linter\nsqlparse  # SQL reformatter\nsquirrelsql\nssllabs-scan\nsslscan\nssh-copy-id\nstats  # fancier Activity Monitor\n#steghide  # looks like it's been removed\n#subversion  # pulls in brew version of Perl which breaks library installations due to this horror - https://github.com/toddr/IO-Tty/issues/25\nswitchaudio-osx\ntcpping  # ping-like tool for TCP requests\n#terraform  # use tfenv with direnv to match your company's version and avoid accidentally upgrading the tfstate\n#terragrunt  # old, use install/install_terragrunt.sh to grab latest version to ~/bin instead\n#terraformer  # large, install only if actively using\ntfenv  # Terraform version manager inspired by rbenv\ntflint # Terraform linter\ntfsec  # Terraform static analysis by AquaSec\nthrift\ntidy-html5\ntmux\ntokei  # lines of code counter\ntree\nttygif  # records terminal and creates gif\n#ttyrec  # pulled as dependency of ttygif\nurlview  # used by tmux plugin tmux-urlview, and also my advanced .vimrc\nunix2dos\n#unrar  # no longer available\nutimer\nvagrant-completion\nvault  # Hashicorp\n# old Mac vim not\nvim  # Mac's vim not syntax highlighting Bash properly on macOS Big Sur 11.7 https://github.com/vim/vim/issues/11937 - watch brew version pulls in a lot of dependencies though including Perl which breaks library installations due to this horror - https://github.com/toddr/IO-Tty/issues/25\nvint  # vim script linter (for me to lint my big .vimrc)\n#visual-studio-code  # the package formerly known as vscode\nwakatime-cli\nwatch\nwatchexec  # run commands when files change\n#wget  # in brew-packages.txt\n#whois  # in brew-packages.txt\n#windows-app  # replaces microsoft-remote-desktop\nwireshark  # only install tshark CLI, must --cask for Wireshark\n#webp\n#x264\n#x265\nxclip\nxcodesorg/made/xcodes\nxmlstarlet  # XML command line utilities such as html decoding, used by htmldecode.sh\n#xvid\nxz\nyamllint  # in brew-packages.txt\n#yara\nyoutube-dl\n#zeromq\n#zlib\n#zstd\nzoom\nzsh\n\n# runtime only\n#groovy\n# GDK - use SDKMan instead (install/install_sdkman.sh)\n#groovysdk\n# done by SDKMan now\n#maven\n#sbt\n#scala\n\n# no longer available\n#jwhois\n\n# clashes with docker and needs unlinking - don't use this any more anyway since Docker Desktop arrived\n#docker-machine-completion\n\n# automake\n# autoconf\n# Mac already supplies a libtool, so this gets prefixed with a 'g'\n# libtool\n"
  },
  {
    "path": "setup/brew-packages-ignore.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2024-03-07 16:40:15 +0000 (Thu, 07 Mar 2024)\n#\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# These are packages that are installed as dependencies but we want to ignore in the output of:\n#\n#       brew_packages_not_saved.sh\n#\n# Usually because they're libraries which we are not going to track and install manually\n#\n# List populated from:\n#\n#       brew_packages_not_saved.sh >> brew-packages-ignore.txt\n#\n# followed by piping the following section through 'sort -fu'\n\nabseil\naom\naribb24\nassimp\nblack\nbrotli\nc-ares\nca-certificates\ncairo\ncapstone\ncffi\ncjson\ndav1d\ndbus\ndouble-conversion\ndtc\nflac\nfontconfig\nfreetype\nfrei0r\nfribidi\ngd\ngdbm\ngdk-pixbuf\ngettext\nghostscript\ngiflib\nglib\ngmp\ngnutls\ngpgme\ngraphite2\ngroff\ngts\nharfbuzz\nhighway\nhunspell\nhwloc\nicu4c\nimath\nimlib2\nisl\njansson\njasper\njbig2dec\njet\njpeg-turbo\njpeg-xl\njson-c\njsoncpp\nleptonica\nlima\nlittle-cms2\nmbedtls\nmd4c\nmpdecimal\nmpfr\nnetpbm\nnettle\nnpth\nnspr\nnss\nopenblas\nopencore-amr\nopenexr\nopenvino\nopus\np11-kit\npango\npixman\npkg-config\npoppler\nprotobuf-c\npsutils\npugixml\npycparser\nqt\nrav1e\nrubberband\nruby-build\ns-lang\nsdl2\nshared-mime-info\nspeex\nspeexdsp\nsrt\nsvt-av1\ntbb\ntcl-tk\ntesseract\ntheora\ntree-sitter\nuchardet\nutf8proc\nvde\nwxwidgets\nxorgproto\nzimg\n"
  },
  {
    "path": "setup/brew-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                    Mac OS X - Homebrew Package Requirements\n# ============================================================================ #\n\nawscli\nazure-cli\ncmake\ncoreutils\ncpanminus\ndirenv\nfindutils\ngawk\ngh  # GitHub CLI\ngit-secrets\ngnu-sed\ngnu-tar\ngo\ngrep  # gnu version better than Mac version\njsonlint\njq\njwhois\nlibxml2  # needed for xmllint\nopenssl\nparallel\npython  # Mac usually comes with Python, but pip was missing in Semaphore CI\npython-setuptools  # because distutils was removed in Python 3.12 :-(\nreadline\nshellcheck\n#terraform # old 0.11 masking newer 0.12 from install_terraform.sh\nwget\nwhois\nyamllint\nyq\n\n# Mac already provides unzip - brew is keg only and doesn't link anyway to avoid clashes\n#unzip\n"
  },
  {
    "path": "setup/brew_fix_openssl_dependencies.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-26 15:19:31 +0000 (Tue, 26 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Reinstalls Python via HomeBrew to fix OpenSSL library linkage break upon OpenSSL 1.0 => OpenSSL 1.1 upgrade caused by krb5\n\n# fixes this breakage:\n#\n# python -c 'import hashlib', which break pips, eg:\n#\n# https://stackoverflow.com/questions/20399331/error-importing-hashlib-with-python-2-7-but-not-with-2-6\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif [ \"$(uname -s)\" != Darwin ]; then\n    echo \"Not a Mac system, aborting...\"\n    exit 1\nfi\n\nif type -P brew &>/dev/null; then\n    # doesn't work - xcode-select can't detect it, let it install itself to fix\n    #if ! which python ; then sudo mkdir -pv /usr/local/bin ; sudo ln -sfv /usr/bin/python3 /usr/local/bin/python; fi\n    # grep -q causes a pipefail via early pipe close which exits the script early without fixing\n    if python -c 'import hashlib' 2>&1 | tee /dev/stderr | grep 'unsupported hash type'; then\n        echo \"Attempting to upgrade homebrew packages depending on upgraded OpenSSL linkage break\"\n        if python -V 2>&1 | tee /dev/stderr | grep '^Python 2'; then\n            echo \"upgrading python@2\"\n            brew upgrade python@2\n        else\n            echo \"upgrading python\"\n            brew upgrade python\n        fi\n        echo \"finding packages dependent on openssl\"\n        #dependent_packages=\"$(brew deps --installed | awk -F: '/:.*openssl/{print $1}')\"\n        dependent_packages=\"$(brew uses openssl --installed)\"\n        # trick to flatten cheaply\n        # shellcheck disable=SC2086\n        echo \"upgrading packages dependent on openssl:  \" $dependent_packages\n        # want package splitting\n        # shellcheck disable=SC2086\n        brew upgrade $dependent_packages\n    fi\nelse\n    echo \"Not a HomeBrew system, aborting...\"\n    exit 1\nfi\n"
  },
  {
    "path": "setup/brew_packages_not_saved.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-03-06 05:14:07 +0000 (Wed, 06 Mar 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all locally installed brew packages which are not saved into the adjacent brew*.txt lists\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"arg [<options>]\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\ncd \"$srcdir\"\n\nsaved_packages=\"$(\n    {\n        sed 's/[[:space:]]#.*//; /^[[:space:]]*$/d' \\\n            brew-packages.txt \\\n            brew-packages-desktop*.txt \\\n            brew-packages-ignore.txt |\n        awk '{print $NF}'\n\n        grep -Eho '^#[[:alnum:]-]+' \\\n            brew-packages.txt \\\n            brew-packages-desktop*.txt |\n        sed 's/^#//'\n\n        grep -Eho '^#[[:alnum:]-]+/[[:alnum:]-]+[[:space:]]+[[:alnum:]-]+[[:space:]]*$' \\\n            brew-packages-desktop-taps.txt |\n        sed 's/[[:space:]]*$//; s/.*[[:space:]]//'\n    } |\n    sort -fu\n)\"\nnum_packages_excluded=\"$(wc -l <<< \"$saved_packages\" | sed 's/[[:space:]]*//')\"\n\ntimestamp \"Excluding '$num_packages_excluded' packages\"\necho >&2\n\nbrew list |\nsort -fu |\ngrep -xvf <(echo \"$saved_packages\") |\nwhile read -r package; do\n    [[ \"$package\" =~ ^lib ]] && continue\n    [[ \"$package\" =~ ^python- ]] && continue\n    #[[ \"$package\" =~ ^python@ ]] && continue\n    [[ \"$package\" =~ @ ]] && continue\n    # easy but naive, creates two forks per package which is slow\n    # doing an single inline grep -v before this loop\n    #if ! grep -Eq \"^#?$package([[:space:]].+)?$\" brew*.txt; then\n        #if grep -Fxq \"$package\" brew-packages-ignore.txt; then\n            echo \"$package\"  # not found in setup/brew*\"\n        #fi\n    #fi\ndone\n"
  },
  {
    "path": "setup/ccmenu_cp_plist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-02 20:53:06 +0000 (Sat, 02 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nplist_dir=\"Library/Containers/net.sourceforge.cruisecontrol.CCMenu/Data/Library/Preferences\"\n\nplist_file=\"net.sourceforge.cruisecontrol.CCMenu.plist\"\n\ncd \"$srcdir/..\"\n\ncp -vf -- ~/\"$plist_dir/$plist_file\" \"$PWD/$plist_dir/$plist_file\"\nplutil -convert xml1 \"$PWD/$plist_dir/$plist_file\"\necho\n\necho \"git diff $PWD/$plist_dir/$plist_file\"\ngit diff \"$PWD/$plist_dir/$plist_file\"\n"
  },
  {
    "path": "setup/ccmenu_setup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-16 01:19:58 +0100 (Wed, 16 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nbin=/Applications/CCMenu.app/Contents/MacOS/CCMenu\n\nplist_dir=\"Library/Containers/net.sourceforge.cruisecontrol.CCMenu/Data/Library/Preferences\"\n\nplist_file=\"net.sourceforge.cruisecontrol.CCMenu.plist\"\n\n\"$srcdir/../install/install_homebrew.sh\"\necho\n\nif ! [ -f \"$bin\" ]; then\n    echo \"=================\"\n    echo \"Installing CCMenu\"\n    echo \"=================\"\n    brew cask install ccmenu\n    echo\nfi\n\n#if ! pgrep CCMenu &>/dev/null; then\n#    echo \"ensuring a first run has been done before replacing config, starting CCMenu\"\n#    # need to ensure it's started before overwriting the config\n#    \"$bin\" &\n#    echo\n#    sleep 2\n#fi\n\n#echo \"Downloading CCMenu configuration from GitHub release\"\n#wget -c -O ~/Library/Containers/net.sourceforge.cruisecontrol.CCMenu/Data/Library/Preferences/net.sourceforge.cruisecontrol.CCMenu.plist \\\n#           https://github.com/HariSekhon/DevOps-Bash-tools/releases/download/ccmenu/net.sourceforge.cruisecontrol.CCMenu.plist\n\ncd \"$srcdir/..\"\n\necho \"Killing CCMenu\"\npkill -f \"$bin\" || :\necho\n\nwhile pgrep CCMenu &>/dev/null; do\n    echo \"waiting for CCMenu to go down\"\n    sleep 1\n    if [ $SECONDS -gt 30 ]; then\n        echo \"Timed out waiting for CCMenu to go down\"\n        exit 1\n    fi\ndone\n\n#mkdir -pv ~/\"$plist_dir/\"\n#rm -f ~/\"$plist_dir/$plist_file\"\n#echo \"Removing ~/Library/Containers/net.sourceforge.cruisecontrol.CCMenu/Container.plist\"\n#rm -f ~/Library/Containers/net.sourceforge.cruisecontrol.CCMenu/Container.plist\n# these don't take immediate effect due to caching, so load via 'defaults' instead\n#echo \"Linking CCMenu configuration:\"\n#ln -svf \"$PWD/$plist_dir/$plist_file\" ~/\"$plist_dir/$plist_file\"\n#echo \"Copying CCMenu configuration:\"\n#cp -vf \"$PWD/$plist_dir/$plist_file\" ~/\"$plist_dir/$plist_file\"\necho \"Loading CCMenu configuration\"\n# give file or pipe via stdin\n#defaults import net.sourceforge.cruisecontrol.CCMenu \"$PWD/$plist_dir/$plist_file\"\ndefaults import net.sourceforge.cruisecontrol.CCMenu - < \"$PWD/$plist_dir/$plist_file\"\necho\n\necho \"Starting CCMenu\"\n#\"$bin\" &\n#disown\nopen -a CCMenu\n\necho \"Done\"\n"
  },
  {
    "path": "setup/ci.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-10-10 17:42:54 +0100 (Thu, 10 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                         C I   C o n f i g   F i l e s\n# ============================================================================ #\n\n# Files to sync to adjacent repos, one per line\n\nJenkinsfile\n.appveyor.yml\n.buildkite/pipeline.yml\n.circleci/config.yml\n.cirrus.yml\n.drone.yml\n.gitlab-ci.yml\n.mdlrc\n.mdl.rb\n.pre-commit-config.yaml\n.pylintrc\n#.scrutinizer.yml\n.semaphore/semaphore.yml\n#.travis.yml\nazure-pipelines.yml\nbitbucket-pipelines.yml\nbuddy.yml\n\ncicd/.concourse.yml\ncicd/.gocd.yml\ncicd/buildspec.yml\ncicd/cloudbuild.yaml\n\ncodefresh.yml\nkics.config\nsonar-project.properties\n\nsetup/gocd_config_repo.json\nsetup/jenkins-job.xml\nsetup/ci_bootstrap.sh\n\nteamcity/.teamcity.vcs.json\nteamcity/.teamcity.vcs.oauth.json\nteamcity/.teamcity.vcs.ssh.json\n\n# defunct\n#codeship/codeship.yml\n#shippable/shippable.yml\n#wercker/wercker.yml\n#wercker/.werckerignore\n"
  },
  {
    "path": "setup/ci_bootstrap.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-02 17:43:35 +0100 (Tue, 02 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Designed to bootstrap all CI systems with retries to make sure the networking, package lists and package repos works before proceeding\n#\n# Minimizes CI build failures due to temporary networking blips, which happens more often than you would think when you have a large number of CI builds across a lot of disparate systems\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nmax_tries=10\ninterval=60 # secs\n\nsudo=\"\"\n# EUID undefined in posix sh\n#[ $EUID = 0 ] || sudo=sudo\n[ \"$(whoami)\" = root ] || sudo=sudo\n\nretry(){\n    # no local in posix sh\n    count=0\n    while true; do\n        # no let or bare (()) in posix sh, must discard output rather than execute it\n        _=$((count+=1))\n        printf \"%s  try %d:  \" \"$(date '+%F %T')\" \"$count\"\n        echo \"$*\"\n        \"$@\" &&\n        break;\n        echo\n        if [ \"$count\" -ge \"$max_tries\" ]; then\n            echo \"$count tries failed, aborting...\"\n            exit 1\n        fi\n        echo \"sleeping for $interval secs before retrying\"\n        sleep \"$interval\"\n        echo\n    done\n}\n\nif [ \"$(uname -s)\" = Darwin ]; then\n    echo \"Bootstrapping Mac\"\n    # removing adjacent dependency to be able to curl from github to avoid submodule circular dependency (git / submodule / install git & make)\n    #retry \"$srcdir/../install/install_homebrew.sh\"\n    if command -v brew 2>&1; then\n        # fix for CI runners on Mac with shallow homebrew clone - which is failing all the BuildKite builds\n        for git_root in /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core /usr/local/Homebrew/Library/Taps/homebrew/homebrew-cask; do\n            if [ -d \"$git_root\" ]; then\n                # find out if Homebrew is a shallow git checkout and if so fix it\n                if [ -f \"$(git -C \"$git_root\" rev-parse --git-dir)/shallow\" ] ||\n                   [ \"$(git -C \"$git_root\" rev-parse --is-shallow-repository)\" = true ]; then\n                    git -C \"$git_root\" fetch --unshallow\n                fi\n            fi\n        done\n        retry brew update\n    fi\nelif [ \"$(uname -s)\" = Linux ]; then\n    echo \"Bootstrapping Linux\"\n    if type apk >/dev/null 2>&1; then\n        retry $sudo apk update\n        retry $sudo apk add --no-progress bash git make\n    elif type apt-get >/dev/null 2>&1; then\n        opts=\"-q -o DPkg::Lock::Timeout=1200\"\n        retry $sudo apt-get update  $opts\n        retry $sudo apt-get install $opts -y git make\n    elif type yum >/dev/null 2>&1; then\n        #retry $sudo yum makecache\n        retry $sudo yum install -qy git make\n    else\n        echo \"Package Manager not found on Linux, cannot bootstrap\"\n        exit 1\n    fi\nelse\n    echo \"Only Mac & Linux are supported for conveniently bootstrapping all install scripts at this time\"\n    exit 1\nfi\n\n#retry make init\n\n# not calling make because in some CI systems we call 'make ci' which includes retries but in others with more restrictive build minutes we only run 'make' for a single shot build\n#\n#make\n"
  },
  {
    "path": "setup/ci_git_set_dir_safe.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-08-03 20:07:09 +0100 (Wed, 03 Aug 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Necessary for some CI/CD systems like Azure DevOps Pipelines which have incorrect ownership on the git checkout dir triggering this error:\n#\n#    fatal: detected dubious ownership in repository at '/code/sql'\n\n# standalone script without lib dependency so it can be called directly from bootstrapped CI before submodules, since that is the exact problem that needs to be solved to allow CI/CD systems with incorrect ownership of the checkout directory to be able to checkout the necessary git submodules\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ndir=\"${1:-$srcdir/..}\"\n\ncd \"$dir\"\n\necho \"Setting directory as safe: $PWD\"\ngit config --global --add safe.directory \"$PWD\"\n\nwhile read -r submodule_dir; do\n    dir=\"$PWD/$submodule_dir\"\n    echo \"Setting directory as safe: $dir\"\n    git config --global --add safe.directory \"$dir\"\ndone < <(git submodule | awk '{print $2}')\n\necho \"Done\"\n"
  },
  {
    "path": "setup/cpan-packages-desktop.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2013-02-18 10:48:12 +0000 (Mon, 18 Feb 2013)\n#\n#  vim:ts=4:sw=4:et\n\nApp::Ack\nApp::cpanminus\nApp::FatPacker\n# useful for converting JSON to YAML\nCatmandu\nDBD::mysql\nIO::Socket::SSL\nJSON\nJSON::XS\nLWP::Simple\nLWP::UserAgent\nMail::IMAPClient\nNet::DNS\nNet::SSH::Expect\nPerl::Critic\nTest::Kwalitee\nNet::SSLeay\nSMS::AQL\nTest::More\nText::Unidecode\nThrift::BinaryProtocol\nThrift::BufferedTransport\nThrift::Socket\nURI::Escape\nXML::Simple\n"
  },
  {
    "path": "setup/cpan-requirements-optional.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:noet\n#\n#  Author: Hari Sekhon\n#  Date: 2013-01-06 15:45:00 +0000 (Sun, 06 Jan 2013)\n#\n#  https://github.com/HariSekhon/lib\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                       Optional CPAN Module Dependencies\n# ============================================================================ #\n\n# needed for PAR::Packer\nExtUtils::Embed\n\n# needed by perl_generate_par_binaries.sh\nPAR::Packer\n\n# for json2yaml.sh / yaml2json.sh\nJSON::XS\nYAML::XS\n"
  },
  {
    "path": "setup/cpan-requirements.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:noet\n#\n#  Author: Hari Sekhon\n#  Date: 2013-01-06 15:45:00 +0000 (Sun, 06 Jan 2013)\n#\n#  https://github.com/HariSekhon/lib\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            CPAN Module Dependencies\n# ============================================================================ #\n\n# needed by bash-tools/perl_generate_fatpacks.sh\nApp::FatPacker\n"
  },
  {
    "path": "setup/deb-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Sat Apr 7 13:23:57 2007 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Deb desktop packages - one per line\n\n# needs extra repos\n# ansible\n\n#mwcollect\n#nessus\n#PEAR-PHP_Shell\nacpitool\nadtool\nair\nairsnort\nalsa-utils\naosd-cat\napache2-utils  # for ab (apache bench)\napp-dicts/dictd-dicts\napp-vim/perl-support\napt-file\napt-spy\narpd\narping\narptables\narpwatch\nat\naterm\natftpd\naudacious\nautocutsel\nautojump\nautossh\nazureus\nbash-completion\nbastille\nbiew\nbittorrent\nbonnie++\nbridge-utils\nbtscanner\nbum\ncatdoc\nccrypt\nchkrootkit\nchntpw\ncksfv\nclusterssh\ncmospwd\nconky\ncowsay\ncryptcat\ncsync2\n#curl # in deb-packages.txt\ncvs\ndavl\nddrescue\ndebconf-utils  # for debconf-get-selections --installer\ndebootstrap\ndebsecan\ndenyhosts\ndhcping\ndia\ndialog\ndnspython\ndnstop\ndnstracer\ndosfstools\ndrbd\ndriftnet\ndsniff\ndynagen\ndynamips\ne2fsprogs\nemacs\neterm\neterm-themes\netherape\nethstatus\nethtool\nettercap-gtk\nevince\nexiv2  # view image metadata\nfcrackzip\nfiglet\nfilelight\nfluxbox\nforemost\nfortune-mod\nfping\nftp\nfzf\ngaim\ngames-misc/fortune-mod\ngcc\ngdb\ngdm\ngdm-themes\ngftp\ngimp\n#git # in deb-packages.txt\ngit-core\ngitk\ngkrellm\ngksu\ngnupg\ngnupg-agent\ngparted\ngpm\ngraphviz  # for the 'dot' command (use with terraform graph), and also github.com/HariSekhon/Diagrams-as-Code\ngrub\ngt5\ngvim\nharden-doc\nheartbeat\nhexedit\nhexyl  # hex viewer - https://github.com/sharkdp/hexyl\nhoneyd\nhostapd\nhping2\nhping3\nhtop\nhttping\nhunt\niceweasel\nifstat\nike-scan\nimagemagick\nincron\niotop\niozone\nipcalc\nipmitool\niptables\niptraf\nipvsadm\nipython\nirssi\njedit\njhead\nkde\nkhexedit\nkismet\nkodos\nkolourpaint\nkubecolor\nksh\nkubernetes  # only on Ubuntu - not kubeadm, 'kubernetes install' - for microk8s or Canonical K8S distribution on workstation, VMware, Openstack, AWS, Azure, Google, and Oracle cloud\nkubetail\nkvirc\nlabrea\nldap-utils\nldapscripts\nldapvi\nlibc6-dev\nlibgeoip1\nlibhtml-parser-perl\nlibimage-exiftool-perl\nliblist-allutils-perl\nliblist-moreutils-perl\nlibnet-cidr-perl\nlibnet-netmask-perl\nlibnet-ssh-perl\nlibperl-critic-perl\nlibssl-dev\nlinks\nlinux-gazette-all\nlogrotate\nlogwatch\nlokkit\nlsat\nlshw\nlsof\nlsscsi\nluma\nlvm2\nlynx\nm4\nmacchanger\nmagicrescue\nmailx\nmboxgrep\nmc\nmdadm\nmemdump\nmercurial\nmosh\nmost\nmp3info\nmtr\nmultitail\nmycli\nnagios-nrpe-plugin\nnagios-plugins\nnasm\nnbd-client\nnbd-server\nnbtscan\nncftp\nnemesis\nnepenthes\nnet-snmp\nnet-tools  # netstat\nnetbeans\nnetcat\nnetdiscover\nnetperf\nnetsed\nnetselect\nnetselect-apt\nnfs-kernel-server\nnfs-utils\nngrep\nnikto\nnmap\nntfs3g\nntop\nntp\nntpdate\nopenipmi\nopenldap\nopenssh-server\nopenvpn\nopenvpn-blacklist\np0f\npaketto\nparted\npasco\npastebinit\npciutils\nperl-doc\nperltidy\npgcli\nphp5-cli\nphrack-all\npidgin\npidgin-sipe\npigz  # parallel gzip\npinentry-gtk\npiozone\n#pipenv\npound\npsh\npsmisc  # for pstree\npssh\npv\npwgen\npychecker\npydns\npyflakes\npygobject\npylint\npython-doc\npython-pygments\nrancid\nrancid-cgi\nrancid-core\nrancid-utils\nrats\nrbenv\nrdesktop\nroot-tail\nrouter-audit-tool\nrsync\nsbd\nscanlogd\nscanssh\nscreen\nsec\nsecure-delete\nsguil-client\nsguil-sensor\nsguil-server\nsiege\nsipcalc\nslay\nsleuthkit\nslocate\nslurm\nsmb4k\nsnortsam\nspeedtest\nsplint\nsquid\nsquid-graph\nssmtp\nsteghide\nstrace\nstunnel\nsubversion\nsucrack\nsudo\nsun-java6-plugin\nswatch\nsys-apps/ack\nsyslog-ng\nsysstat\nsysv-rc-conf\ntcpdump\ntcptrack\ntct\ntftp\ntftpd\ntilda\ntoilet\ntokei  # lines of code counter\ntomboy\ntraceroute\ntshark\n#unzip # in deb-packages.txt\nusbutils\nvim-full\nvirt-manager\nvlc\nvnc4server\nwin32codecs\nwine\nwipe\nwireless-tools\nwireshark\nwmaker\nwmaker-data\nwmctrl\nwpasupplicant\nx86info\nxbattbar-acpi\nxchat\nxclip\nxdotool\nxfce4-artwork\nxine-ui\nxkbset\nxmlstartlet\nxosd-bin\nxosview\nxpdf\nxprobe\nxscreensaver\nxsel\nxserver-xorg\nxterm\nzenity\nzip\nzsh\n"
  },
  {
    "path": "setup/deb-packages-optional.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-04-10 13:35:58 +0100 (Fri, 10 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                             Deb Packages Optional\n# ============================================================================ #\n\ndirenv\nshellcheck\nyq\n\n# prevents this error:\n# bash-tools/lib/utils.sh: line 41: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory\nlocales-all\n\n# on Ubuntu but not Debian\ngolang-race-detector-runtime\n\n# Ubuntu 20.04 has Python3, python installs Python2 but python-pip package is no longer available\npython\npython3\npython-pip\npython3-pip\npython-setuptools # needed by 'pip install awscli'\n\n# only available from Debian 12 and Ubuntu 22.04\npre-commit\n\n# to give readline wrapping support to Oracle SQL*Plus\nrlwrap\n\n# Debian 12+ and Ubuntu 24+\nsqlfluff\n"
  },
  {
    "path": "setup/deb-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          Deb Package Requirements\n# ============================================================================ #\n\nbash\nbc\nbuild-essential\nca-certificates\ncpanminus\ncurl\n#direnv\ngit\n#gnupg\ngolang\ngroovy  # needed by check_groovyc.sh in Jenkins shared library CI/CD workflow in GitHub Actions\niputils-ping\njq\nless\nlibperl-dev    # needed for PAR:Packer\nlibxml2-utils  # needed for xmllint\nmake\nopenssh-client\nparallel\nperl\nprocps\npsmisc  # for fuser for apt_wait.sh\nruby-dev  # to build Travis CI gem\ntime\nwget\n# included in distro\n#which\nzip\nunzip\n"
  },
  {
    "path": "setup/debian.experimental.pref",
    "content": "# 0 < P < 100: causes a version to be installed only if there is no\n# installed version of the package\n\nPackage: *\nPin: release a=experimental\nPin-Priority: 1\n"
  },
  {
    "path": "setup/debian.stable.pref",
    "content": "# 500 <= P < 990: causes a version to be installed unless there is a\n# version available belonging to the target release or the installed\n# version is more recent\n\nPackage: *\nPin: release a=stable\nPin-Priority: 900\n"
  },
  {
    "path": "setup/debian.testing.pref",
    "content": "# 100 <= P < 500: causes a version to be installed unless there is a\n# version available belonging to some other distribution or the installed\n# version is more recent\n\nPackage: *\nPin: release a=testing\nPin-Priority: 400\n"
  },
  {
    "path": "setup/debian.unstable.pref",
    "content": "# 0 < P < 100: causes a version to be installed only if there is no\n# installed version of the package\n\nPackage: *\nPin: release a=unstable\nPin-Priority: 50\n"
  },
  {
    "path": "setup/docker-images.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-19 12:12:35 +0000 (Tue, 19 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            Docker Images to preload\n# ============================================================================ #\n\nalpine\nbusybox\ncentos\ncentos:8\ncentos:7\ndebian\ndebian:10 # buster\n#debian:9 # stretch\n#debian:8 # jessie\n#debian:7 # wheezy\nfedora\nubuntu\nubuntu:18.04\n#ubuntu:14.04\n#gentoo/stage3-amd64\n\nmcr.microsoft.com/azure-cli\ngoogle/cloud-sdk\n\n#mysql\n#mysql:5.5\n#mysql:5.6\n#mysql:5.7\npostgres  # needed by concourse\n\nconcourse/concourse\njenkins\n#sonarqube\n\nharisekhon/github\nharisekhon/pytools\nharisekhon/perl-tools\nharisekhon/bash-tools\nharisekhon/nagios-plugins\nharisekhon/alpine-java\nharisekhon/centos-java\nharisekhon/debian-java\nharisekhon/fedora-java\nharisekhon/ubuntu-java\nharisekhon/centos-scala\nharisekhon/fedora-scala\nharisekhon/alpine-dev\nharisekhon/centos-dev\nharisekhon/debian-dev\nharisekhon/fedora-dev\nharisekhon/ubuntu-dev\nharisekhon/alpine-github\nharisekhon/centos-github\nharisekhon/debian-github\nharisekhon/fedora-github\nharisekhon/ubuntu-github\n\n# pypy\n# perl\n# golang\n# ruby\n# jruby\n# node\n# clojure\n# julia\n# erlang\n\n#memcached\n# java # only openjdk due to Oracle removing system distributor license, no Linux ships Oracle's Java any more\n#redis\n#redis:3.2-alpine\n\n#elasticsearch\n# preload for nagios plugins tests\n#elasticsearch:1.4\n#elasticsearch:1.5\n#elasticsearch:1.6\n#elasticsearch:1.7\n#elasticsearch:2.0\n#elasticsearch:2.2\n#elasticsearch:2.3\n#elasticsearch:2.4\n#elasticsearch:5.0\n## using my own kafka container\n# spotify/kafka\n# wurstmeister/kafka\n# confluent/kafka\n\n#cassandra\n#couchbase\n#nginx\n#nginx:1.10\n#nginx:1.11.0\n#rabbitmq\n#rabbitmq:3.4\n#rabbitmq:3.5\n#rabbitmq:3.6\n#influxdb\n#solr\n#solr:5\n#solr:6\n#mongo\n#mongo:2.6\n#mongo:3.0\n#mongo:3.2\n#mongo:3.3\n#neo4j\n#neo4j:2.3\n#neo4j:3.0\n# icinga/icinga2\n\n# couchdb\n# hazelcast/hazelcast\n# percona\n# jetty\n# logstash\n# kibana\n# celery\n\n# exciting low latency distributed SQL product\n#crate\n\n# hopsoft/graphite-statsd\n# grafana/grafana\n# icinga/icinga2\n# sequenceiq/spark\n# sequenceiq/drill\n# there is no latest tag for velvia/spark-jobserver :-(\n# velvia/spark-jobserver:0.6.0\n\n#opower/opentsdb\n# petergrace/opentsdb-docker\n#ceph/daemon\n#docker.bintray.io/jfrog/artifactory-oss\n\n# Needs VM with 8GB of RAM, just use VirtualBox\n#cloudera/quickstart\n\n#cloudera/impala-dev:minimal\n# hortonworks/ambari-agent\n# hortonworks/ambari-server\n# mesosphere/marathon\n# mesosphere/swarm\n# mesosphere/presto-server\n# mesosphere/presto-cli\n"
  },
  {
    "path": "setup/docker_bootstrap.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-16 10:33:03 +0100 (Wed, 16 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Builds a git repo (taken from the first /github/<name> component in $PATH) inside a docker image at /github and then runs tests and cleans the caches to minimize the docker image size\n\n# Alpine / Wget:\n#\n#   wget -O- https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/setup/docker_bootstrap.sh | sh\n#\n# Curl:\n#\n#   curl https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/setup/docker_bootstrap.sh | sh\n\nset -eux\nif [ -n \"${SHELL:-}\" ] && [ \"${SHELL##*/}\" = \"bash\" ]; then\n    set -o pipefail\nfi\n\nbasedir=\"/github\"\n\nrepo=\"$(echo \"$PATH\" | tr ':' '\\n' | grep \"^$basedir/\" | sed 's|/github/||' | head -n1)\"\n\nif [ -z \"$repo\" ]; then\n    echo \"ERROR: could not determine repo from \\$PATH components, no /github/<name> was found in \\$PATH: $PATH\"\n    exit 1\nfi\n\nmkdir -pv \"$basedir\"\n\ncd \"$basedir\"\n\n# type -P doesn't work in bourne shell\nif ! command -v curl >/dev/null 2>&1; then\n    if command -v apt-get >/dev/null 2>&1; then\n        export DEBIAN_FRONTEND=noninteractive\n        opts=\"-o DPkg::Lock::Timeout=1200\"\n        apt-get update  $opts\n        apt-get install $opts -y curl  # --no-install-recommends  # will omit ca-certificates which will break the ability to curl the bootstrap script further down\n    elif command -v yum >/dev/null 2>&1; then\n        yum install -y curl\n    elif command -v apk >/dev/null 2>&1; then\n        apk add --no-cache curl\n    fi\nfi\n\n# bourne shell won't detect subshell failure, so better to break this to detectable parts\n#curl -sSf \"https://raw.githubusercontent.com/HariSekhon/$repo/master/setup/bootstrap.sh\" | sh\n\ntrap 'command rm -fv -- /bootstrap.sh /clean_caches.sh' INT QUIT TRAP ABRT TERM EXIT\n\ncurl -sSf \"https://raw.githubusercontent.com/HariSekhon/$repo/master/setup/bootstrap.sh\" > /bootstrap.sh\n\nsh /bootstrap.sh\n\ncd \"$basedir/$repo\"\n\nif [ -z \"${NO_TESTS:-}\" ]; then\n    make test\nfi\n\ncurl -sSf https://raw.githubusercontent.com/HariSekhon/DevOps-Bash-tools/master/bin/clean_caches.sh > /clean_caches.sh\n\nsh /clean_caches.sh\n"
  },
  {
    "path": "setup/download_cassandra.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-28 15:39:09 +0100 (Tue, 28 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Installs Java and downloads Cassandra to $PWD\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/bash-tools/lib/utils.sh\"\n\nif ! type -P wget &>/dev/null ||\n     type -P apk; then  # Alpine built-in wget isn't good enough\n    \"$srcdir/../packages/install_packages.sh\" wget\nfi\n\n#/usr/lib/jvm/jre/bin/\nif ! type -P java &>/dev/null; then\n    \"$srcdir/../install/install_java.sh\"\nfi\n\nCASSANDRA_VERSION=\"${CASSANDRA_VERSION:-3.11.4}\"\n\nTAR=\"apache-cassandra-$CASSANDRA_VERSION-bin.tar.gz\"\n\nurl=\"http://www.apache.org/dyn/closer.lua?filename=cassandra/$CASSANDRA_VERSION/$TAR&action=download\"\n\nurl_archive=\"http://archive.apache.org/dist/cassandra/$CASSANDRA_VERSION/$TAR\"\n\necho \"Downloading Cassandra tarball:\"\n# --max-redirect - some apache mirrors redirect a couple times and give you the latest version instead\n#                  but this breaks stuff later because the link will not point to the right dir\n#                  (and is also the wrong version for the tag)\nwget -t 10 --max-redirect 1 --retry-connrefused -O \"$TAR\" \"$url\" || \\\nwget -t 10 --max-redirect 1 --retry-connrefused -O \"$TAR\" \"$url_archive\"\n\necho \"Extracting tarball:\"\ntar zxf \"$TAR\"\n\necho \"Removing tarball:\"\nrm -fv -- \"$TAR\"\n\n# check tarball was extracted to the right place, helps ensure it's the right version and the link will work\nif ! test -d \"apache-cassandra-$CASSANDRA_VERSION\"; then\n    echo \"apache-cassandra-$CASSANDRA_VERSION directory not found from extracted tarball!\"\n    exit 1\nfi\n\necho \"Symlinking cassandra:\"\nln -sv -- \"apache-cassandra-$CASSANDRA_VERSION\" cassandra\n\nif [ -f /.dockerenv ]; then\n    echo \"Running inside docker, removing docs:\"\n    rm -rf -- cassandra/doc cassandra/javadoc\nfi\n"
  },
  {
    "path": "setup/download_openjdk11.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-01-05 15:41:31 +0000 (Wed, 05 Jan 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads and unpacks OpenJDK to \\$PWD\n\nPrint the JAVA_HOME you need\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<version>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nversion=\"${1:-11.0.11+9}\"\n\nos=\"$(get_os)\"\narch=\"$(get_arch)\"\n\n# AdoptOpenJDK deviate from the common terms\nif [ \"$os\" = darwin ]; then\n    os=\"mac\"\nfi\nif [ \"$arch\" = amd64 ]; then\n    arch=\"x64\"\nfi\n\n#version=\"$(\"$srcdir/../urlencode.sh\" <<< \"$version\")\"\nversion2=\"$(tr '+' '_' <<< \"$version\")\"\nurl=\"https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-$version/OpenJDK11U-jdk_${arch}_${os}_hotspot_${version2}.tar.gz\"\n\ntimestamp \"Downloading JDK version '$version'\"\necho\ndownload \"$url\"\necho\n\ntarget_dir=\"${PWD:-$(pwd)}/jdk-$version\"\nif is_mac; then\n    target_dir+=\"/Contents/Home\"\nfi\n\nif [ -d \"$target_dir\" ]; then\n    timestamp \"Target directory already exists, aborting\"\nelse\n    timestamp \"Extracting tarball\"\n    echo\n    tar xvf \"${url##*/}\"\nfi\necho\n\ntimestamp \"Done\"\necho\ncat <<EOF\n\nSet your environment as follows:\n\nexport JAVA_HOME=\"$target_dir\"\nexport PATH=\"\\$PATH:\\$JAVA_HOME/bin\"\n\nEOF\n"
  },
  {
    "path": "setup/files.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-10-10 17:42:54 +0100 (Thu, 10 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            C o n f i g   F i l e s\n# ============================================================================ #\n\n# Files to link to $HOME, one per line\n\n.editorconfig\n.envrc\n.envrc-python\n.gitconfig\n.gitignore\n# top-level .mdlrc is cumulative and can mask issues in project repos\n# instead use an .mdlrc and .mdl.rb in each repo individually\n#.mdlrc\n#.mdl.rb\n# home dir is too heavy for this right now\n#.pre-commit-config.yaml\n.pylintrc\nconfigs/.ansible.cfg\nconfigs/.athenacli/athenaclirc\n#.aws/config\nconfigs/.aws/shell/awsshellrc\naws/.aws_customize_environment\nconfigs/.Codefresh/cli-config/config.yaml\nconfigs/.config/flake8\nconfigs/.config/htop/htoprc\nconfigs/.config/pycodestyle\nconfigs/.config/terminalizer/config.yml\nconfigs/.config/yamllint/config\nconfigs/.gemrc\nconfigs/.git-templates/git-secrets/hooks/*\nconfigs/.grype.yaml\n#configs/.hammerspoon  # moved to its own repo\nconfigs/.inputrc\nconfigs/.luacheckrc\nconfigs/.my.cnf\nconfigs/.perlcriticrc\nconfigs/.perlcritic_forbidden_modules\nconfigs/.psqlrc\nconfigs/.rabbitmqadmin.conf\nconfigs/.sawsrc\nconfigs/.screenrc\nconfigs/.sdkman/etc/config\nconfigs/.sqliterc\nconfigs/.terraformrc\nconfigs/.tfdocs.d/.terraform-docs.yml\nconfigs/.tmux.conf\nconfigs/.toprc\nconfigs/.vimrc\n#configs/.wakatime.cfg  # copied because Wakatime IntelliJ plugin doesn't accept env var and insists on writing to this file and don't want to accidentally commit it\nconfigs/.Xdefaults\nconfigs/.Xmodmap\ngcp/.customize_environment\nscalastyle_config.xml\n"
  },
  {
    "path": "setup/gem-packages-desktop.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2013-02-18 10:47:57 +0000 (Mon, 18 Feb 2013)\n#\n#  vim:ts=4:sw=4:et\n\ngitlab\ngist\njgrep\nlolcat  # colourizes ascii text\nsqlint\nrubocop  # ruby linter\n\n# installed via homebrew now, can't find gem any more\n#rbenv\n\n# mdl needs ruby >= 2.4.0 whereas older Macs may have 2.3.0 - if so install manually via:\n#\n#       brew install ruby\n#\n#       GEM=/usr/local/Cellar/ruby/*/bin/gem  ./ruby_gem_install.sh mdl\n#\n# will be picked up by new shells via .bash.d/paths.sh automatically detecting and adding ~/.gem/ruby/*/bin\n#\n# on macOS Catalina I now have Ruby 2.6 so this can install normally\nmdl\n"
  },
  {
    "path": "setup/gem-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2013-02-18 10:47:57 +0000 (Mon, 18 Feb 2013)\n#\n#  vim:ts=4:sw=4:et\n\njson\n"
  },
  {
    "path": "setup/go-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-06 17:09:33 +0100 (Fri, 06 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# List of Golang software to install via 'go install', one per line, and if not @version is suffixed, will assume @latest\n\ngolang.org/x/tools/cmd/goimports\ngithub.com/remind101/assume-role\ngithub.com/songgao/colorgo\ngithub.com/cjbassi/gotop\ngithub.com/mkouhei/gosh\ngithub.com/rakyll/hey\n#github.com/PaulRosset/previs\n#github.com/wercker/wercker\n"
  },
  {
    "path": "setup/gocd_config_repo.json",
    "content": "{\n  \"id\": \"bash-tools\",\n  \"plugin_id\": \"yaml.config.plugin\",\n  \"material\": {\n    \"type\": \"git\",\n    \"attributes\": {\n  \t\"url\": \"https://github.com/HariSekhon/DevOps-Bash-tools\",\n  \t\"branch\": \"master\",\n  \t\"auto_update\": true\n    }\n  },\n  \"configuration\": [\n    {\n      \"key\": \"file_pattern\",\n      \"value\": \"cicd/*.gocd.y*ml\"\n    }\n  ],\n  \"rules\": [\n    {\n  \t\"directive\": \"allow\",\n  \t\"action\": \"*\",\n  \t\"type\": \"*\",\n  \t\"resource\": \"*\"\n    }\n  ]\n}\n"
  },
  {
    "path": "setup/intellij-plugins.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2024-06-12 08:18:51 +0200 (Wed, 12 Jun 2024)\n#\n#  vim:ts=4:sts=4:sw=4:et:filetype=conf\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                        I n t e l l i J   P l u g i n s\n# ============================================================================ #\n\n# Taken from https://github.com/HariSekhon/Knowledge-Base/blob/main/intellij.md\n\n\n# ============================================================================ #\n#        D o c k e r ,   K u b e r n e t e s   a n d   T e r r a f o r m\n# ============================================================================ #\n\n# Docker - https://plugins.jetbrains.com/plugin/7724-docker\nDocker\n\n# Terraform and HCL - https://plugins.jetbrains.com/plugin/7808-terraform-and-hcl\norg.intellij.plugins.hcl\n\n# only available in Ultimate Edition :-(\n# Kubernetes - https://plugins.jetbrains.com/plugin/10485-kubernetes\n#com.intellij.kubernetes\n\n\n# ============================================================================ #\n#                               L a n g u a g e s\n# ============================================================================ #\n\n# Shell Script - https://plugins.jetbrains.com/plugin/13122-shell-script\ncom.jetbrains.sh\n\n# Bash Support  - https://plugins.jetbrains.com/plugin/4230-bashsupport\nBashSupport\n\n# Perl - https://plugins.jetbrains.com/plugin/7796-perl\ncom.perl5\n\n# Go Linter (https://plugins.jetbrains.com/plugin/12496-go-linter\ncom.ypwang.plugin.go-linter\n\n# See Python, NodeJS and Java sections further down\n\n\n# ============================================================================ #\n#        C o r e   E d i t i n g ,   G i t   &   F i l e   F o r m a t s\n# ============================================================================ #\n\n# .ignore - https://plugins.jetbrains.com/plugin/7495--ignore  - supports various `.ignore` files for different technologies\nmobi.hsz.idea.gitignore\n\n# Code Glance - https://plugins.jetbrains.com/plugin/7275-codeglance/  - adds a minimap of the file\nnet.vektah.codeglance\n\n# Grep Console - https://plugins.jetbrains.com/plugin/7125-grep-console\nGrepConsole\n\n# BrowseWordAtCaret - https://plugins.jetbrains.com/plugin/201-browsewordatcaret\nBrowseWordAtCaret\n\n# Editor Config - https://plugins.jetbrains.com/plugin/7294-editorconfig/\norg.editorconfig.editorconfigjetbrains\n\n# GitLink - https://plugins.jetbrains.com/plugin/8183-gitlink/  - shortcut to open files on GitHub and other hosted repo providers\nuk.co.ben-gibson.remote.repository.mapper\n\n# Git Toolbox - https://plugins.jetbrains.com/plugin/7499-gittoolbox/  - automatic fetches, show status vs upsteam origin\nzielu.gittoolbox\n\n# CSV Editor - https://plugins.jetbrains.com/plugin/10037-csv-editor\nnet.seesharpsoft.intellij.plugins.csv\n\n# JSON Parser - https://plugins.jetbrains.com/plugin/10650-json-parser  - validate & format JSON strings\ncom.godwin.json.parser\n\n# CamelCase - https://plugins.jetbrains.com/plugin/7160-camelcase\nde.netnexus.camelcaseplugin\n\n# RegexpTester - https://plugins.jetbrains.com/plugin/2917-regexp-tester\norg.intellij.RegexpTester\n\n# Database Navigator - https://plugins.jetbrains.com/plugin/1800-database-navigator\nDBN\n\n# Markdown Navigator Enhanced - https://plugins.jetbrains.com/plugin/7896-markdown-navigator-enhanced/\ncom.vladsch.idea.multimarkdown\n\n#  Zero Width Characters locator - https://plugins.jetbrains.com/plugin/7448-zero-width-characters-locator  - find characters that could break your code\ncom.ultrahob.zerolength.plugin\n\n# Env files support - https://plugins.jetbrains.com/plugin/9525--env-files-support\nru.adelf.idea.dotenv\n\n# String Manipulation - https://plugins.jetbrains.com/plugin/2162-string-manipulation\nString Manipulation\n\n# Rainbow Brackets - https://plugins.jetbrains.com/plugin/10080-rainbow-brackets\nizhangzhihao.rainbow.brackets\n\n# Rainbow CSV - https://plugins.jetbrains.com/plugin/12896-rainbow-csv\ncom.andrey4623.rainbowcsv\n\n# Indent Rainbow - https://plugins.jetbrains.com/plugin/13308-indent-rainbow\nindent-rainbow.indent-rainbow\n\n# Return Highlighter - https://plugins.jetbrains.com/plugin/13303-return-highlighter\ncom.github.lppedd.idea-return-highlighter\n\n\n# ============================================================================ #\n#                             U s a g e   S t a t s\n# ============================================================================ #\n\n# WakaTime - https://plugins.jetbrains.com/plugin/7425-wakatime  - stats on your usage\ncom.wakatime.intellij.plugin\n\n# Code Time - https://plugins.jetbrains.com/plugin/10687-code-time/  - stats on your usage\ncom.softwareco.intellij.plugin\n\n# Statistic - https://plugins.jetbrains.com/plugin/4509-statistic  - shows project stats, files, line count etc.\nStatistic\n\n\n# ============================================================================ #\n#                                   C l o u d\n# ============================================================================ #\n\n# AWS ToolKit - https://plugins.jetbrains.com/plugin/11349-aws-toolkit - Amazon CodeWhisperer integration\naws.toolkit\n\n# Azure Toolkit for IntelliJ - https://plugins.jetbrains.com/plugin/8053-azure-toolkit-for-intellij\ncom.microsoft.tooling.msservices.intellij.azure\n\n# Google Cloud Code - https://plugins.jetbrains.com/plugin/8079-gemini-code-assist-cloud-code\ncom.google.gct.core\n\n\n# ============================================================================ #\n#                                   C I / C D\n# ============================================================================ #\n\n# Jenkins Pipeline Linter - https://plugins.jetbrains.com/plugin/15699-jenkins-pipeline-linter\ncom.github.mikesafonov.jenkins-linter-idea-plugin\n\n# Jenkins Control - https://plugins.jetbrains.com/plugin/6110-jenkins-control\n#Jenkins Control Plugin\n\n# Groovy - https://plugins.jetbrains.com/plugin/1524-groovy - for your Groovy Shared Library functions eg. https://github.com/HariSekhon/Jenkins\n#org.intellij.groovy  # sourced in the JVM section further down\n\n# Teamcity - https://plugins.jetbrains.com/plugin/1820-teamcity - just use Jenkins instead, see Jenkins-on_kubernetes: https://github.com/HariSekhon/Kubernetes-configs#Jenkins-on-Kubernetes\n#Jetbrains TeamCity Plugin\n\n# SonarLint - https://plugins.jetbrains.com/plugin/7973-sonarlint - use with SonarQube / SonarCloud\norg.sonarlint.idea\n\n# Synk Security - https://plugins.jetbrains.com/plugin/10972-snyk-security\n#io.snyk.snyk-intellij-plugin\n\n\n# ============================================================================ #\n#                O p t i o n a l   -   N i c e   t o   H a v e s\n# ============================================================================ #\n\n# IDEA Features Trainer - https://plugins.jetbrains.com/plugin/8554-ide-features-trainer  - teaches you the IDE\ntraining\n\n# Key Promoter X - https://plugins.jetbrains.com/plugin/9792-key-promoter-x  - teaches you keyboard shortcut when you click with the mouse\n\"Key Promoter X\"\n\n# Material Theme UI - https://plugins.jetbrains.com/plugin/8006-material-theme-ui\ncom.chrisrm.idea.MaterialThemeUI\n\n# Extra Icons - https://plugins.jetbrains.com/plugin/11058-extra-icons  - adds icons for different file types\nlermitage.intellij.extra.icons\n\n# Atom Material Icons - https://plugins.jetbrains.com/plugin/10044-atom-material-icons  - nicer file icons\ncom.mallowigi\n\n# Yet another emoji support - https://plugins.jetbrains.com/plugin/12512-yet-another-emoji-support\ncom.github.shiraji.yaemoji\n\n# Mongo - https://plugins.jetbrains.com/plugin/7141-mongo-plugin\n#\"Mongo Plugin\"  # who uses Mongo any more?\n\n# Pieces - https://plugins.jetbrains.com/plugin/17328-pieces--save-search-share--reuse-code-snippets  - code snippets - you should be using libraries but unfortunately some languages have boilerplate for which library do not solve the repetition between programs. See also  HariSekhon/Templates - https://github.com/HariSekhon/Templates\n\n\n# ============================================================================ #\n#                                  P y t h o n\n# ============================================================================ #\n\n# Python - https://plugins.jetbrains.com/plugin/7322-python-community-edition - contains better support for Jython than PyCharm - cross-language navigation, completion and refactoring\nPythonCore\n\n# Requirements - https://plugins.jetbrains.com/plugin/10837-requirements\nru.meanmail.plugin.requirements\n\n# PyLint - https://plugins.jetbrains.com/plugin/11084-pylint/\ncom.leinardi.pycharm.pylint\n\n# Mypy - https://plugins.jetbrains.com/plugin/11086-mypy/\ncom.leinardi.pycharm.mypy\n\n# Live Coding in Python - https://plugins.jetbrains.com/plugin/9742-live-coding-in-python/\nio.github.donkirkby.livepycharm\n\n# Python Enhancements - https://plugins.jetbrains.com/plugin/10194-python-enhancements/\ncom.pythondce\n\n# Python Security - https://plugins.jetbrains.com/plugin/13609-python-security\norg.tonybaloney.security.pycharm-security\n\n# Python Annotations - https://plugins.jetbrains.com/plugin/12035-python-annotations\nru.meanmail.plugin.pyannotations\n\n\n# ============================================================================ #\n#                                  N o d e J S\n# ============================================================================ #\n\n# NodeJS - https://plugins.jetbrains.com/plugin/6098#node-js\nNodeJS\n\n# ESLint - https://plugins.jetbrains.com/plugin/7494#eslint\ncom.wix.eslint\n\n# Prettier - https://plugins.jetbrains.com/plugin/10456#prettier\nintellij.prettierJS\n\n# Quokka - https://plugins.jetbrains.com/plugin/9667#quokka  - rapid prototyping playground\nquokka.js\n\n\n# ============================================================================ #\n#                   Java / Groovy / Scala / Kotlin & JVM Tools\n# ============================================================================ #\n\n# Groovy - https://plugins.jetbrains.com/plugin/1524-groovy\norg.intellij.groovy\n\n# Scala - https://plugins.jetbrains.com/plugin/1347-scala\norg.intellij.scala\n\n# Kotlin - https://plugins.jetbrains.com/plugin/6954-kotlin\norg.jetbrains.kotlin\n\n# Maven Helper - https://plugins.jetbrains.com/plugin/7179-maven-helper\nMavenRunHelper\n\n# SBT - https://plugins.jetbrains.com/plugin/5007-sbt\nSBT\n\n# Gradle - https://plugins.jetbrains.com/plugin/13112-gradle\ncom.intellij.gradle\n\n# Gradle/Maven Navigation - https://plugins.jetbrains.com/plugin/9857-gradle-maven-navigation\ntv.twelvetone.gradle.plugin.navigation\n\n# Sprint Boot Assistant - https://plugins.jetbrains.com/plugin/17747-spring-boot-assistant\ndev.flikas.idea.spring.boot.assistant.plugin\n# Lombok - https://plugins.jetbrains.com/plugin/6317-lombok  - automates generating getters/setters etc. Project Lombok - https://projectlombok.org/\nLombook Plugin\n# JRebel - https://plugins.jetbrains.com/plugin/4441-jrebel-and-xrebel  - auto-reload code changes\n# XRebel - https://plugins.jetbrains.com/plugin/4441-jrebel-and-xrebel/  - performance profiling\nJRebelPlugin\n\n\n# ============================================================================ #\n#                               D e b u g g i n g\n# ============================================================================ #\n\n# SpotBugs - https://plugins.jetbrains.com/plugin/14014-spotbugs\n#org.jetbrains.plugins.spotbugs\n\n# LiveEdit - https://plugins.jetbrains.com/plugin/7007-live-edit  - shows changes instantly for JavaScript, HTML, can enable for NodeJS etc.\n#com.intellij.plugins.html.instantEditing\n\n# Lightrun - https://plugins.jetbrains.com/plugin/16477-lightrun  - for live running code debugging using  Lightrun - https://lightrun.com/\n#com.lightrun.idea.plugin.saas.LightrunPlugin\n\n# Rookout - https://plugins.jetbrains.com/plugin/12637-rookout\n#com.rookout.intellij-plugin\n\n\n# ============================================================================ #\n#                              A I   P l u g i n s\n# ============================================================================ #\n\n# JetBrains AI Assistant - https://plugins.jetbrains.com/plugin/22282-jetbrains-ai-assistant\n\n# AWS ToolKit - https://plugins.jetbrains.com/plugin/11349-aws-toolkit  - Amazon CodeWhisperer integration\n\n# TabNine - https://plugins.jetbrains.com/plugin/12798-tabnine-ai-code-completion--chat-in-java-js-ts-python--more  - AI code suggestions\ncom.tabnine.TabNine\n\n# Codota AI Autocomplete for Java and JavaScript - https://plugins.jetbrains.com/plugin/7638-codota-ai-autocomplete-for-java-and-javascript\n # `idea installPlugins com.codota.csp.intellij`\n\n# GitHub CoPilot - https://plugins.jetbrains.com/plugin/17718-github-copilot\n\n# AI Coding Assistant - https://plugins.jetbrains.com/plugin/20724-ai-coding-assistant\n\n# Codiumate - https://plugins.jetbrains.com/plugin/21206-codiumate--code-test-and-review-with-confidence--by-codiumai  - CodiumAI integration\n"
  },
  {
    "path": "setup/jenkins-job-check-gcp-serviceaccount.xml",
    "content": "<?xml version='1.1' encoding='UTF-8'?>\n<project>\n  <actions/>\n  <description>Runs a job to fetch the GCP serviceaccount being used by the default pod on GKE to test Workload Identity integration</description>\n  <keepDependencies>false</keepDependencies>\n  <properties>\n    <jenkins.model.BuildDiscarderProperty>\n      <strategy class=\"hudson.tasks.LogRotator\">\n        <daysToKeep>7</daysToKeep>\n        <numToKeep>100</numToKeep>\n        <artifactDaysToKeep>-1</artifactDaysToKeep>\n        <artifactNumToKeep>-1</artifactNumToKeep>\n      </strategy>\n    </jenkins.model.BuildDiscarderProperty>\n  </properties>\n  <scm class=\"hudson.scm.NullSCM\"/>\n  <canRoam>true</canRoam>\n  <disabled>false</disabled>\n  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>\n  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>\n  <triggers/>\n  <concurrentBuild>true</concurrentBuild>\n  <builders>\n    <hudson.tasks.Shell>\n      <command>echo Environment:\necho\nenv | sort\necho\necho Starting\necho &quot;Fetching GCP Metadata:&quot;\ncurl -sS -H &quot;Metadata-Flavor: Google&quot; -w &quot;\\n&quot; http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/email\necho Finished</command>\n      <configuredLocalRules/>\n    </hudson.tasks.Shell>\n  </builders>\n  <publishers/>\n  <buildWrappers/>\n</project>\n"
  },
  {
    "path": "setup/jenkins-job-sleep-parallel-parameterized.xml",
    "content": "<?xml version='1.1' encoding='UTF-8'?>\n<project>\n  <actions/>\n  <description>Runs a sleep job to allow testing parallelization and auto-scaling of Jenkins agents on Kubernetes</description>\n  <keepDependencies>false</keepDependencies>\n  <properties>\n    <jenkins.model.BuildDiscarderProperty>\n      <strategy class=\"hudson.tasks.LogRotator\">\n        <daysToKeep>7</daysToKeep>\n        <numToKeep>100</numToKeep>\n        <artifactDaysToKeep>-1</artifactDaysToKeep>\n        <artifactNumToKeep>-1</artifactNumToKeep>\n      </strategy>\n    </jenkins.model.BuildDiscarderProperty>\n    <hudson.model.ParametersDefinitionProperty>\n      <parameterDefinitions>\n        <hudson.model.StringParameterDefinition>\n          <name>UNIQUE_VALUE</name>\n          <description>Enter anything unique to queue multiple copies of this job to force parallelization</description>\n          <defaultValue>default value</defaultValue>\n          <trim>false</trim>\n        </hudson.model.StringParameterDefinition>\n        <hudson.model.ChoiceParameterDefinition>\n          <name>SLEEP_MINUTES</name>\n          <choices class=\"java.util.Arrays$ArrayList\">\n            <a class=\"string-array\">\n              <string>10</string>\n              <string>5</string>\n              <string>1</string>\n              <string>20</string>\n              <string>30</string>\n            </a>\n          </choices>\n        </hudson.model.ChoiceParameterDefinition>\n      </parameterDefinitions>\n    </hudson.model.ParametersDefinitionProperty>\n  </properties>\n  <scm class=\"hudson.scm.NullSCM\"/>\n  <canRoam>true</canRoam>\n  <disabled>false</disabled>\n  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>\n  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>\n  <triggers/>\n  <concurrentBuild>true</concurrentBuild>\n  <builders>\n    <hudson.tasks.Shell>\n      <command>echo Environment:\necho\nenv | sort\necho\necho Starting\necho &quot;UNIQUE_VALUE=$UNIQUE_VALUE&quot;\nSLEEP_SECONDS=&quot;$(($SLEEP_MINUTES * 60))&quot;\necho &quot;Sleeping for $SLEEP_MINUTES minutes ($SLEEP_SECONDS seconds)&quot;\nsleep &quot;$SLEEP_SECONDS&quot;\necho Finished</command>\n      <configuredLocalRules/>\n    </hudson.tasks.Shell>\n  </builders>\n  <publishers/>\n  <buildWrappers/>\n</project>\n"
  },
  {
    "path": "setup/jenkins-job.xml",
    "content": "<?xml version='1.1' encoding='UTF-8'?>\n<flow-definition plugin=\"workflow-job@2.38\">\n  <actions>\n    <org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobAction plugin=\"pipeline-model-definition@1.6.0\"/>\n    <org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobPropertyTrackerAction plugin=\"pipeline-model-definition@1.6.0\">\n      <jobProperties/>\n      <triggers>\n        <string>hudson.triggers.SCMTrigger</string>\n        <string>hudson.triggers.TimerTrigger</string>\n      </triggers>\n      <parameters/>\n      <options/>\n    </org.jenkinsci.plugins.pipeline.modeldefinition.actions.DeclarativeJobPropertyTrackerAction>\n  </actions>\n  <description></description>\n  <keepDependencies>false</keepDependencies>\n  <properties>\n    <hudson.plugins.jira.JiraProjectProperty plugin=\"jira@3.0.14\"/>\n    <com.coravy.hudson.plugins.github.GithubProjectProperty plugin=\"github@1.29.5\">\n      <projectUrl>https://github.com/HariSekhon/DevOps-Bash-tools/</projectUrl>\n      <displayName></displayName>\n    </com.coravy.hudson.plugins.github.GithubProjectProperty>\n    <org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>\n      <triggers>\n        <hudson.triggers.TimerTrigger>\n          <spec>H 10 * * 1-5</spec>\n        </hudson.triggers.TimerTrigger>\n        <hudson.triggers.SCMTrigger>\n          <spec>H/2 * * * *</spec>\n          <ignorePostCommitHooks>false</ignorePostCommitHooks>\n        </hudson.triggers.SCMTrigger>\n      </triggers>\n    </org.jenkinsci.plugins.workflow.job.properties.PipelineTriggersJobProperty>\n  </properties>\n  <definition class=\"org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition\" plugin=\"workflow-cps@2.80\">\n    <scm class=\"hudson.plugins.git.GitSCM\" plugin=\"git@4.2.2\">\n      <configVersion>2</configVersion>\n      <userRemoteConfigs>\n        <hudson.plugins.git.UserRemoteConfig>\n          <url>https://github.com/HariSekhon/DevOps-Bash-tools</url>\n        </hudson.plugins.git.UserRemoteConfig>\n      </userRemoteConfigs>\n      <branches>\n        <hudson.plugins.git.BranchSpec>\n          <name>*/master</name>\n        </hudson.plugins.git.BranchSpec>\n      </branches>\n      <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>\n      <submoduleCfg class=\"list\"/>\n      <extensions/>\n    </scm>\n    <scriptPath>Jenkinsfile</scriptPath>\n    <lightweight>true</lightweight>\n  </definition>\n  <triggers/>\n  <disabled>false</disabled>\n</flow-definition>"
  },
  {
    "path": "setup/jenkins-plugins.txt",
    "content": "# Basic list of plugins to get started, generated by:\n#\n#   jenkins_cli.sh list-plugins | awk '{print $1\":\"$NF}' | sort > setup/jenkins_plugins.txt\n#\n# For a more comprehensive list of awesome plugins, see:\n#\n#   https://github.com/HariSekhon/Kubernetes-configs/blob/master/jenkins-values.yaml\n#\nblueocean:1.22.0\nbuild-timeout:1.19.1\ncommand-launcher:1.4\n#convert-to-pipeline:1.0\ngit-client:3.2.1\ngit-server:1.9\ngit:4.2.2\ngithub-api:1.106\ngithub-branch-source:2.6.0\ngithub:1.29.5\ngradle:1.36\njunit:1.28\ntimestamper:1.11.2\nworkflow-aggregator:2.6\n"
  },
  {
    "path": "setup/linux_desktop.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-16 10:33:03 +0100 (Wed, 16 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nrun(){\n    echo \"Running $srcdir/$1\"\n    QUICK=1 \"$srcdir/$1\"\n    echo\n    echo\n    echo \"================================================================================\"\n}\n\ninstall_scripts=\"\ninstall_ansible.sh\ninstall_diff-so-fancy.sh\ninstall_gcloud_sdk.sh\ninstall_sdkman.sh\ninstall_sdkman_all_sdks.sh\ninstall_terraform.sh\ninstall_vundle.sh\n\"\n# don't use this much any more\n#install_parquet-tools.sh\n# Legacy CI\n#install_travis.sh\n\nfor x in $install_scripts; do\n    run \"$x\"\ndone\nif [[ \"$USER\" =~ hari|sekhon ]]; then\n    run install_github_ssh_key.sh\nfi\necho\n\"$srcdir/shell_link.sh\"\n"
  },
  {
    "path": "setup/mac_delete_routes_on_interface.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-27 12:11:13 +0100 (Thu, 27 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOn Mac deletes all routes relating to a given network interface\n\nUseful to clear stale route entries left by vpn clients such as AWS Client VPN that may interfere with networking or reconnections\n\n(You may also need to restart your wifi after disconnecting AWS VPN Client - appears to be buggy behaviour)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<interface>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ninterface=\"$1\"\n\nis_mac || usage \"ERROR: this script is only designed to run on Mac\"\n\nnetstat -rn |\nawk \"/$interface/{print \\$1}\" |\nwhile read -r net; do\n    # defined in lib/utils.sh\n    # shellcheck disable=SC2154\n    $sudo route delete -net \"$net\" -ifp \"$interface\"\ndone\n"
  },
  {
    "path": "setup/mac_desktop.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-16 10:33:03 +0100 (Wed, 16 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\necho \"Install XCode CLI tools\"\nxcode-select --install || :  # ignore if already installed\n\nrun(){\n    echo \"Running $srcdir/$1\"\n    QUICK=1 \"$srcdir/$1\"\n    echo\n    echo\n    echo \"================================================================================\"\n}\n\necho\n\"$srcdir/shell_link.sh\"\n\n# homebrew script must be first\ninstall_scripts=\"\ninstall_homebrew.sh\ninstall_ansible.sh\ninstall_diff-so-fancy.sh\ninstall_gcloud_sdk.sh\ninstall_minikube.sh\ninstall_minishift.sh\ninstall_sdkman.sh\ninstall_sdkman_all_sdks.sh\ninstall_terraform.sh\ninstall_vundle.sh\n\"\n# don't use this much any more\n#install_parquet-tools.sh\n# Legacy CI\n#install_travis.sh\n\nfor x in $install_scripts; do\n    run \"$x\"\ndone\nif [[ \"$USER\" =~ hari|sekhon ]]; then\n    run install_github_ssh_key.sh\nfi\n"
  },
  {
    "path": "setup/mac_diff_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-29 00:12:43 +0100 (Tue, 29 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Script to diff settings changes and then drop in to the new settings json to explore\n#\n# Speeds up experimentation and key collecting for adjacent script mac_settings.sh\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ndir=\"$srcdir/mac_settings\"\n\nmkdir -pv \"$dir\"\n\nbefore_settings=\"$dir/settings-before-$(date '+%F_%H%M%S').json\"\n\ndefaults read > \"$before_settings\"\n\necho \"Settings Saved\"\necho\necho \"Now make your changes\"\necho\necho \"Press Enter when ready to collect the changes difference\"\n\nread -r\n\nafter_settings=\"$dir/settings-after-$(date '+%F_%H%M%S').json\"\n\ndefaults read > \"$after_settings\"\n\ndiff \"$before_settings\" \"$after_settings\" | less -+F -i || :\n\nexec \"${EDITOR:-vim}\" \"$after_settings\"\n"
  },
  {
    "path": "setup/mac_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-01 21:01:50 +0000 (Sun, 01 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                            M a c   S e t t i n g s\n# ============================================================================ #\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# ============================================================================ #\n#                  B a c k u p   B e f o r e   A p p l y i n g\n# ============================================================================ #\n\n# If you do mess something up, you can also go in to Time Machine and get the ~/Library/Preferences/ plist back,\n# 'defaults import' it and then restart the daemons\n\nbackup_dir=\"$srcdir/mac_settings\"\n\nmkdir -pv \"$backup_dir\"\n\nbackup=\"$backup_dir/settings-backup-$(date '+%F_%H%M%S')-$HOSTNAME.json\"\n\necho \"creating ~/.bash_sessions_disable touchfile to stop 'Restore sessions' on every shell open\"\ntouch ~/.bash_sessions_disable\necho\n\necho \"backing up mac settings to $backup before applying new settings\"\ndefaults read > \"$backup\"\n\n# ============================================================================ #\n# Set Dark Theme without requiring restart\n\n# will prompt to allow Terminal to control System Events which is useful to enable anyway for Apple Scripting eg. ../applescript/*.scpt\nosascript -e 'tell app \"System Events\" to tell appearance preferences to set dark mode to 1'\n\n# toggle between light and dark theme by setting it to the opposite of its current setting\n#osascript -e 'tell app \"System Events\" to tell appearance preferences to set dark mode to not dark mode'\n\n# ============================================================================ #\n\n# References:\n#\n# https://ss64.com/osx/defaults.html\n#\n# https://github.com/herrbischoff/awesome-macos-command-line\n\n# TIP: how to figure out settings keys\n#\n# 1. defaults read > settings.json\n# 2. Make your changes in the UI\n# 3. defaults read > settings2.json\n# 4. diff settings.json settings2.json\n\n# This now automated using the adjacent script mac_diff_settings.sh\n# which saves copies of the before and after configs and diffs them,\n# before dropping in to the new config to explore the full settings paths\n\n# \"Apple Global Domain\" === NSGlobalDomain\ndefaults write NSGlobalDomain AppleActionOnDoubleClick Maximize\ndefaults write NSGlobalDomain AppleInterfaceStyle -string Dark\ndefaults write NSGlobalDomain AppleEnableMenuBarTransparency -bool true\ndefaults write NSGlobalDomain AppleSpacesSwitchOnActivate 1\ndefaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool false\ndefaults write NSGlobalDomain NSQuitAlwaysKeepsWindows -bool true\ndefaults write NSGlobalDomain ApplePressAndHoldEnabled -bool false\ndefaults write NSGlobalDomain AppleMetricUnits -bool true\ndefaults write NSGlobalDomain AppleTemperatureUnit -string \"Celsius\"\ndefaults write NSGlobalDomain com.apple.sound.beep.flash -bool false\n\ndefaults write com.apple.menuextra.battery ShowPercent -string \"YES\"\n\ndefaults write com.apple.menuextra.clock DateFormat -string \"EEE d MMM  HH:mm\"\ndefaults write com.apple.menuextra.clock FlashDateSeparators -bool false\ndefaults write com.apple.menuextra.clock IsAnalog -bool false\n\n# Remove Shutdown & Restart buttons at login window\n#defaults write com.apple.loginwindow ShutDownDisabled -bool true\n#defaults write com.apple.loginwindow RestartDisabled -bool true\n#defaults write com.apple.loginwindow LoginwindowText \"Your Message\"\n\n# turn off the annoying \"Application Downloaded from Internet\" quarantine warning\n#defaults write com.apple.LaunchServices LSQuarantine -bool false\n\n# disable annoying crash prompt\ndefaults write com.apple.CrashReporter DialogType none  # set to 'prompt' to restore\n\n# disable dashboard widgets (saves RAM)\ndefaults write com.apple.dashboard mcx-disabled -boolean true\n\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.TimeMachine\" -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.airport\"     -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.appleuser\"   -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.battery\"     -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.bluetooth\"   -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.volume\"      -bool true\ndefaults write com.apple.systemuiserver \"NSStatusItem Visible com.apple.menuextra.clock\"       -bool true\n\n\n# ============================================================================ #\n#                                K e y b o a r d\n# ============================================================================ #\n\n# fast keyboard repeat and low delay\ndefaults write NSGlobalDomain InitialKeyRepeat -int 15\ndefaults write NSGlobalDomain KeyRepeat -int 2\n\n# make Fn key show hotkeys in touch bar on newer Macs\ndefaults write com.apple.touchbar.agent.PresentationModeFnModes.appWithControlStrip -string fullControlStrip\n\n# ============================================================================ #\n#                                T r a c k p a d\n# ============================================================================ #\n\n# ============\n# tap to click\ndefaults write com.apple.trackpad forceClick -bool false\ndefaults write com.apple.AppleMultitouchTrackpad Clicking -bool false\ndefaults write com.apple.AppleMultitouchTrackpad USBMouseStopsTrackpad -int 0\ndefaults write com.apple.driver.AppleBluetoothMultitouch.trackpad Clicking -bool false\n\ndefaults write com.apple.AppleMultitouchTrackpad TrackpadThreeFingerTapGesture -int 0\ndefaults write com.apple.driver.AppleBluetoothMultitouch.trackpad TrackpadThreeFingerTapGesture -int 0\n\n# ============================\n# tap to click on login screen\ndefaults -currentHost write NSGlobalDomain com.apple.mouse.tapBehavior -int 1\ndefaults write NSGlobalDomain com.apple.mouse.tapBehavior -int 1\n\n# OS version on login screen\ndefaults write com.apple.loginwindow AdminHostInfo HostName\n\n# ==============\n# Trackpad Speed - choose an int/float in the range 0-3:\n# 0 = slowest\n# 3 = fastest\ndefaults write com.apple.trackpad.scaling -float 3\n\n# ===========\n# right-click\ndefaults write com.apple.AppleMultitouchTrackpad TrackpadRightClick -bool true\ndefaults write com.apple.driver.AppleBluetoothMultitouch.trackpad TrackpadRightClick -bool true\n# click bottom right corner for right-click - doesn't work\n#defaults write com.apple.AppleMultitouchTrackpad TrackpadCornerSecondaryClick -int 2\n#defaults write com.apple.driver.AppleBluetoothMultitouch.trackpad TrackpadCornerSecondaryClick -int 2\n\n# ===============\n# Haptic feedback\n# 0: Light\n# 1: Medium\n# 2: Firm\ndefaults write com.apple.AppleMultitouchTrackpad.FirstClickThreshold -int 0\ndefaults write com.apple.AppleMultitouchTrackpad.SecondClickThreshold -int 0\ndefaults write com.apple.AppleMultitouchTrackpad.ForceSuppressed -int 0\ndefaults write com.apple.AppleMultitouchTrackpad ActuationStrength -int 0\n\n# =======================================\n# Enable “natural” (Lion-style) scrolling\ndefaults write NSGlobalDomain com.apple.swipescrolldirection -bool true\n\n# Maximize window when double clicking the bar\ndefaults write NSGlobalDomain AppleActionOnDoubleClick -string Maximize\n\n# ============================================================================ #\n#                                T e r m i n a l\n# ============================================================================ #\n\n# use the custom 'Hari' profile\ndefaults write com.apple.Terminal \"Default Window Settings\" -string \"Hari\"\ndefaults write com.apple.Terminal \"Startup Window Settings\" -string \"Hari\"\n\n# ===============\n# open maximized:\n# - MacBook Pro 15\" 2013 has 204 x 59\n# - MacBook Pro 15\" 2018 has 239 x 72\n# - if you're running a 13\" Macbook Pro this will be different again, let's infer the size using the current terminal\n# - $LINES and $COLUMNS aren't automatically available in a non-interactive script shell, so if not set fall back to 'tput'\n#   - tput falls back to 80 cols x 24 lines - too conservative\n#     - now instead relying on $COLUMNS and $LINES being exported by shell profile .bash.d/env.sh\n#       - this relies on user having already maximized your Terminal window before running this\nCOLUMNS=\"${COLUMNS:-$(tput cols)}\"\nLINES=\"${LINES:-$(tput lines)}\"\n# try to squeaze the terminal right to the edges\n((COLUMNS+=1))\n((LINES+=1))\n\n# this is a blob and cannot be descended in to using 'defaults' command :'-(\n# so we load from a saved Terminal -> Preferences -> Profiles -> Hari -> Settings -> Export file and override the $LINES and $COLUMNS\nmac_terminal_settings=\"$(cat \"$srcdir/Hari.terminal\")\"\nmac_terminal_settings=\"${mac_terminal_settings/<integer>239</<integer>$COLUMNS<}\"\nmac_terminal_settings=\"${mac_terminal_settings/<integer>72</<integer>$LINES<}\"\ndefaults write com.apple.Terminal \"Window Settings\" -dict-add Hari \"$mac_terminal_settings\"\n\n# ============================================================================ #\n#                              T a s k   B a r\n# ============================================================================ #\n\ndefaults write com.apple.controlcenter \"NSStatusItem Visible Bluetooth\" -int 1\n\n# ============================================================================ #\n#               S c r e e n s a v e r   &   H o t   C o r n e r s\n# ============================================================================ #\n\n# require password after screensaver\ndefaults write com.apple.screensaver askForPassword -bool true\n\n# gives 5 secs grace before requiring password in case you accidentally hit a hot corner\ndefaults write com.apple.screensaver askForPasswordDelay -int 5\n\n# bottom left Hot Corner activates Screensaver\ndefaults write com.apple.dock wvous-bl-corner -int 5\ndefaults write com.apple.dock wvous-bl-modifier -int 0\n\n# top right Hot Corner puts display to sleep\ndefaults write com.apple.dock wvous-tr-corner -int 10\ndefaults write com.apple.dock wvous-tr-modifier -int 0\n\n# ============================================================================ #\n#                             S c r e e n s h o t s\n# ============================================================================ #\n\n# save screenshots to Desktop\nmkdir -p -v ~/Desktop/Screenshots\n# put them under a Screenshots folder, it's cleaner than having them all over your background\ndefaults write com.apple.screencapture location -string ~/Desktop/Screenshots\n\ndefaults write com.apple.screencapture \"show-thumbnail\" -bool \"true\"\n\n# save in PNG format (default, other options: BMP, GIF, JPG, PDF, TIFF)\ndefaults write com.apple.screencapture type -string \"png\"\n#defaults write com.apple.screencapture type JPG\n\n# default screenshot file name\n# defaults write com.apple.screencapture name \"myScreenShot\"\n\n# disable drop shadows in screenshots\ndefaults write com.apple.screencapture disable-shadow -bool true\n\n# killall SystemUIServer\n\n# ============================================================================ #\n#                                    D o c k\n# ============================================================================ #\n\n# auto-hide Dock\ndefaults write com.apple.dock autohide -bool true\n\n# make Dock appear on all screens - this is important so that cmd-tab window switching appears there too as it follows the Dock screen\ndefaults write com.apple.dock appswitcher-all-displays -bool true\n\n# hide non-active apps\n# XXX: careful this wipes out your Dock and reversing it to false doesn't restore your Dock items\n# If you mess up your Dock it is difficult to regenerate programmatically, so I suggest you go in to Time Machine and get the ~/Library/Preferences/com.apple.dock.plist back, 'defaults import' and then killall Dock to get back your icons\n#defaults write com.apple.dock static-only -bool true\n\n# pop-up instantly\ndefaults write com.apple.dock autohide-delay         -float 0  # secs\n# disappear instantly\ndefaults write com.apple.dock autohide-time-modifier -float 0\n\n# speed up Mission Control animations\ndefaults write com.apple.dock expose-animation-duration -float 0.12\n\n# revert to default dock delays\n#defaults delete com.apple.dock autohide-delay\n#defaults delete com.apple.dock autohide-time-modifier\n\n# stop apps saving to iCloud by default\ndefaults write NSGlobalDomain NSDocumentSaveNewDocumentsToCloud -bool false\n\n# enable Dashboard with widgets\n#defaults write com.apple.dashboard mcx-disabled -boolean false\n\n# make hidden app icons translucent\ndefaults write com.apple.Dock showhidden -bool true\n\n# show indicator lights for open applications in the dock\ndefaults write com.apple.dock show-process-indicators -bool true\n\n# disable bouncing icons\ndefaults write com.apple.dock no-bouncing -bool true\n\n#defaults write com.apple.dock minimize-to-application -bool true\n# don't minimize to applications, it's more obvious when they're on the far right\ndefaults write com.apple.dock minimize-to-application -bool false\n\ndefaults write com.apple.dock mineffect scale  # faster than 'genie'\n\n# don't leave closed apps in the dock\ndefaults write com.apple.dock show-recents -bool no\ndefaults write com.apple.dock recent-apps -array # intentionally empty\n\n#killall Dock\n\n# ============================================================================ #\n#                           M i s c e l l a n e o u s\n# ============================================================================ #\n\n# Use network time\nsudo systemsetup -setusingnetworktime on\n\n# auto-restart after a system freeze\nsudo systemsetup -setrestartfreeze on\n\n# restore windows after reboot\ndefaults write com.apple.systempreferences NSQuitAlwaysKeepsWindows -bool true\ndefaults write -g NSQuitAlwaysKeepsWindows -bool true\n\n# Check for software updates daily, not just once per week\ndefaults write com.apple.SoftwareUpdate ScheduleFrequency -int 1\n\n# Disable local Time Machine snapshots\n#sudo tmutil disable local\n\ndefaults write com.apple.TimeMachine AutoBackup -int 1\n# default 3600 = 1 hour backup interval\n#sudo defaults write /System/Library/Launch Daemons/com.apple.backupd-auto StartInterval -int 1800\n\n# Increase sound quality for Bluetooth headphones/headsets\ndefaults write com.apple.BluetoothAudioAgent \"Apple Bitpool Min (editable)\" -int 40\n\n# Expanded print options\ndefaults write -g PMPrintingExpandedStateForPrint -bool true\ndefaults write -g PMPrintingExpandedStateForPrint2 -bool true\n\n# Expand file save dialog\ndefaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode -bool true\ndefaults write NSGlobalDomain NSNavPanelExpandedStateForSaveMode2 -bool true\n\n# ============================================================================ #\n#                                  F i n d e r\n# ============================================================================ #\n\n# TODO: set Finder to columns that are auto-wide\n\n# hide Desktop icons\n#defaults write com.apple.finder CreateDesktop -bool false\n\n# set Desktop as the default location for new Finder windows\n#defaults write com.apple.finder NewWindowTarget -string \"PfDe\"\n# set Downloads as the default location for new Finder windows\ndefaults write com.apple.finder NewWindowTarget -string \"PfLo\"\ndefaults write com.apple.finder NewWindowTargetPath -string \"file://${HOME}/Downloads/\"\ndefaults write com.apple.finder NSNavLastRootDirectory -string \"file://${HOME}/Downloads/\"\n\n# default to Details view\ndefaults write com.apple.finder FXPreferredViewStyle        -string Nlsv\ndefaults write com.apple.finder TrashViewSettings           -string Nlsv\ndefaults write com.apple.finder SearchRecentsSavedViewStyle -string Nlsv\n\n# allow copying from Quick Look preview\ndefaults write com.apple.finder QLEnableTextSelection -bool true\n\n# show the ~/Library folder\nchflags nohidden ~/Library\n\n# show Status Bar\ndefaults write com.apple.finder ShowStatusBar -bool true\n\n# Empty Trash securely by default\ndefaults write com.apple.finder EmptyTrashSecurely -bool true\n\n# Automatically remove Trash > 30 days\ndefaults write com.apple.finder FXRemoveOldTrashItems -bool true\n\n# don't disable the warning before emptying the Trash in case you hit Cmd-Del then Cmd-Shift-Del data could be lost\ndefaults write com.apple.finder WarnOnEmptyTrash -bool true\n\n# show hidden files by default\ndefaults write com.apple.finder AppleShowAllFiles -bool true\n\n# show all filename extensions\ndefaults write NSGlobalDomain AppleShowAllExtensions -bool true\n\n# show the scroll bar all the time\ndefaults write NSGlobalDomain AppleShowScrollBars -string WhenScrolling # or Automatic or Always\n\n# close always confirms changes\ndefaults write NSGlobalDomain NSCloseAlwaysConfirmsChanges -bool true\n\n# show path bar\ndefaults write com.apple.finder ShowPathbar -bool true\n\n# display full POSIX path as Finder window title\ndefaults write com.apple.finder _FXShowPosixPathInTitle -bool true\n\n# When performing a search, search the current folder by default\ndefaults write com.apple.finder FXDefaultSearchScope -string \"SCcf\"\n\n# disable the warning when changing a file extension\ndefaults write com.apple.finder FXEnableExtensionChangeWarning -bool false\n\n# Show icons for hard drives, servers, and removable media on the desktop\ndefaults write com.apple.finder ShowExternalHardDrivesOnDesktop -bool true\ndefaults write com.apple.finder ShowHardDrivesOnDesktop         -bool true\ndefaults write com.apple.finder ShowMountedServersOnDesktop     -bool true\ndefaults write com.apple.finder ShowRemovableMediaOnDesktop     -bool true\n\n# Avoid creating .DS_Store files on network volumes\ndefaults write com.apple.desktopservices DSDontWriteNetworkStores -bool true\n# Don't create .DS_Store files on USB drives - probably don't want this if you use external hard drives a lot\n#defaults write com.apple.desktopservices DSDontWriteUSBStores -bool true\n\n\n# Automatically open a new Finder window when a volume is mounted\ndefaults write com.apple.frameworks.diskimages auto-open-ro-root -bool true\ndefaults write com.apple.frameworks.diskimages auto-open-rw-root -bool true\ndefaults write com.apple.finder    OpenWindowForNewRemovableDisk -bool true\n\n# Enable 'Quit' menu item in Finder\n#defaults write com.apple.finder QuitMenuItem -bool true\ndefaults write com.apple.finder QuitMenuItem -bool false\n\n#killall Finder\n\n# ============================================================================ #\n#                                 P r e v i e w\n# ============================================================================ #\n\n# set to false to not re-open previous documents from last Preview session\ndefaults write com.apple.Preview NSQuitAlwaysKeepsWindows -bool true\n\n#killall Preview\n\n# ============================================================================ #\n#                       A u t o m a t i c   U p d a t e s\n# ============================================================================ #\n\n# Auto check for updates\ndefaults write com.apple.SoftwareUpdate AutomaticCheckEnabled -bool true\n\n# Auto download updates in background\ndefaults write com.apple.SoftwareUpdate AutomaticDownload -bool true\n\n# Don't install App updates automatically\ndefaults write com.apple.commerce AutoUpdate -bool false\n\n# Don't install MacOS updates automatically\ndefaults write com.apple.commerce AutoUpdateRestartRequired -bool false\n\n# Don't install Security updates automatically\ndefaults write com.apple.SoftwareUpdate CriticalUpdateInstall -bool false\n\n# ============================================================================ #\n\n# Enable AirDrop over Ethernet and on unsupported Macs running Lion\n#defaults write com.apple.NetworkBrowser BrowseAllInterfaces -bool true\n\n# Change indexing order and disable some search results\n#defaults write com.apple.spotlight orderedItems -array \\\n#   '{\"enabled\" = 1;\"name\" = \"APPLICATIONS\";}' \\\n#   '{\"enabled\" = 1;\"name\" = \"SYSTEM_PREFS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"DIRECTORIES\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"PDF\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"FONTS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"DOCUMENTS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MESSAGES\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"CONTACT\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"EVENT_TODO\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"IMAGES\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"BOOKMARKS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MUSIC\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MOVIES\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"PRESENTATIONS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"SPREADSHEETS\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"SOURCE\";}' \\\n#   '{\"enabled\" = 1;\"name\" = \"MENU_DEFINITION\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MENU_OTHER\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MENU_CONVERSION\";}' \\\n#   '{\"enabled\" = 1;\"name\" = \"MENU_EXPRESSION\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MENU_WEBSEARCH\";}' \\\n#   '{\"enabled\" = 0;\"name\" = \"MENU_SPOTLIGHT_SUGGESTIONS\";}'\n\n# ============================================================================ #\n\necho \"Some settings won't take effect until you restart the processes. When you've saved your work and are ready, run:\n\nsudo killall Finder\nsudo killall Dock\nsudo killall cfprefsd\nsudo killall SystemUIServer\nsudo killall Terminal\n\"\n"
  },
  {
    "path": "setup/mac_spotlight_config_optimize.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-11-11 19:48:55 +0100 (Tue, 11 Nov 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/../lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConfigures Apple Spotlight to be more minimalistic and exclude a lot of rubbish results like web pages\n\n- Enables indexing (needed by Raycast even if replacing Spotlight Search, see HariSekhon/Knowledge repo's Mac page)\n- Excludes noisy folders (Downloads, Pictures, external Volumes)\n- Disables Siri Suggestions & web results to give priority to local apps (or just use Raycast or Alfred, see Mac page)\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nnum_args 0 \"$@\"\n\nmac_only\n\ntimestamp \"Enabling Spotlight indexing\"\nsudo mdutil -a -i on\necho >&2\n\ntimestamp \"Clearing Spotlight's temporary caches\"\nsudo mdutil -E /\necho >&2\n\ntimestamp \"Adding common folders to Spotlight Privacy list...\"\n\n# skip these folder from indexing\nexcluded_paths=(\n  \"/System\"\n  \"/Library\"\n  \"/private\"\n  \"/Volumes\"\n  \"$HOME/Downloads\"\n  \"$HOME/Movies\"\n  \"$HOME/Music\"\n  \"$HOME/Pictures\"\n)\n\nfor path in \"${excluded_paths[@]}\"; do\n    if [ -d \"$path\" ]; then\n        timestamp \"Excluding path: $path\"\n        sudo mdutil -i off \"$path\" >/dev/null 2>&1 || :\n    fi\ndone\necho >&2\n\ntimestamp \"Disabling Siri & web suggestions in Spotlight\"\ndefaults write com.apple.Spotlight SiriSuggestionsEnabled -bool false\ndefaults write com.apple.Siri SuggestionsEnabled -bool false\ndefaults write com.apple.lookup.shared LookupSuggestionsDisabled -bool true\ndefaults write com.apple.Spotlight OrderedItems -array \\\n  '{\"enabled\" = 1; \"name\" = \"APPLICATIONS\";}' \\\n  '{\"enabled\" = 1; \"name\" = \"DOCUMENTS\";}' \\\n  '{\"enabled\" = 0; \"name\" = \"MENU_DEFINITION\";}' \\\n  '{\"enabled\" = 0; \"name\" = \"MENU_OTHER\";}' \\\n  '{\"enabled\" = 0; \"name\" = \"MENU_WEBSEARCH\";}' \\\n  '{\"enabled\" = 0; \"name\" = \"MENU_SPOTLIGHT_SUGGESTIONS\";}'\necho >&2\n\ntimestamp \"Restarting Spotlight services...\"\nkillall mds >/dev/null 2>&1 || :\nkillall mds_stores >/dev/null 2>&1 || :\necho >&2\n\necho \"Spotlight config optimized\"\necho\necho \"- Indexing active for Applications and Documents only\"\necho \"- Siri and web suggestions disabled\"\necho \"- Noisy folders excluded\"\necho\necho \"You can verify with: mdutil -s /\"\n"
  },
  {
    "path": "setup/mas-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2026-01-29 17:32:16 -0400 (Thu, 29 Jan 2026)\n#\n#  vim:ts=4:sts=4:sw=4:et:filetype=conf\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#              M a s   A p p   S t o r e   P a c k a g e   L i s t\n# ============================================================================ #\n\n# App Store packages to install on a new Mac using Mas\n#\n# Mas is the open source command line interface to the App Store\n\n# For more details on Mas, see HariSekhon/Knowledge-Base page on Mas:\n#\n#   https://github.com/HariSekhon/Knowledge-Base/blob/main/mas.md\n\n# Generated using this command to strip out the version numbers:\n#\n#   mas list | cut -c-40\n\n# Install using command:\n#\n#   sed 's/#.*//' mas-packages.txt | awk '{print $1}' | xargs mas install\n\n 490461369  Bandwidth+\n 500154009  Bitdefender Virus Scanner\n 425264550  Blackmagic Disk Speed Test\n1376878040  BlueWallet\n6740840706  Braver Search\n1457309557  Converter\n 663592361  DuckDuckGo\n 935235287  Encrypto\n 406056744  Evernote\n 408981434  iMovie\n 408981381  iPhoto\n 409183694  Keynote\n1295203466  Microsoft Remote Desktop\n 409203825  Numbers\n1579499874  OpenSpeedTest-Server\n 409201541  Pages\n 897118787  Shazam\n 803453959  Slack\n1153157709  Speedtest\n 747648890  Telegram\n 310633997  WhatsApp\n1455463454  WiFi Speed Test\n 497799835  Xcode\n"
  },
  {
    "path": "setup/npm-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-18 12:53:03 +0000 (Mon, 18 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# NodeJS desktop packages - one per line\n\n@mermaid-js/mermaid-cli\ncodefresh\njs-yaml\nnetlify-cli\nnodemon\nterminalizer\n"
  },
  {
    "path": "setup/npm-packages.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-11-18 12:53:03 +0000 (Mon, 18 Nov 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# NodeJS packages - one per line\n\n#js-yaml  # in npm-packages-desktop.txt\njsonlint\n"
  },
  {
    "path": "setup/pip-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Wed Feb 3 16:44:26 2016 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# use install_ansible.sh instead now which will try to use OS package manager first and only use pip if not available\n# if OS package manager version is too old then consider installing newer version via pip\n#ansible\n\nansible-lint\nathenacli\nautopep8\naws-shell\nawscli\nbeautifulsoup4\nbitbucket-cli\ncfn-lint\ncodeship-yaml\nflake8\n#gitlab-python  # Python 3 only by the look of it\ngitsome\n# tries to uninstall thrift -> ERROR: Cannot uninstall 'thrift'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n# just use impala-shell from CDH parcel on edge node instead\n# impala-shell\niredis\nkazoo\nkube-shell\nmycli\nnose\npgcli\npipenv\npipx\n# pep8 has been renamed to pycodestyle\npycodestyle\npyflakes\npygments\npyhcl  # Hashicorp's config language parser - returns dict and comes with cli tool to parse hcl to json - hcltool INFILE [OUTFILE]\nPyInstaller\npylint\npytest\npython-cson\nrequests\ntabulate\nsaws\nPyYAML\nyamllint\nyq\n\n#avro\n#hadoopy\n#pymongo\n"
  },
  {
    "path": "setup/pip-packages-mac.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-13 01:50:26 +0100 (Sat, 13 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# can only be installed on macOS\n\npyobjc-framework-Quartz\n"
  },
  {
    "path": "setup/pip_fix_version.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-02-26 09:31:22 +0000 (Fri, 26 Feb 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Downgrades Pip version if Python < 3 because pip has dropped support for the widely used Python 2.7\n#\n# Avoids this error:\n#\n# Traceback (most recent call last):\n#   File \"/usr/lib64/python2.7/runpy.py\", line 162, in _run_module_as_main\n#     \"__main__\", fname, loader, pkg_name)\n#   File \"/usr/lib64/python2.7/runpy.py\", line 72, in _run_code\n#     exec code in run_globals\n#   File \"/usr/lib/python2.7/site-packages/pip/__main__.py\", line 21, in <module>\n#     from pip._internal.cli.main import main as _main\n#   File \"/usr/lib/python2.7/site-packages/pip/_internal/cli/main.py\", line 60\n#     sys.stderr.write(f\"ERROR: {exc}\")\n#                                    ^\n\nset -eu\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif ! pip --version >/dev/null 2>&1; then\n    echo \"pip --version failed\"\n    if python --version 2>&1 | grep -q '^Python 2'; then\n        echo \"Python is legacy version 2\"\n        echo \"Installing pip < 21.0 to be able to run\"\n        easy_install 'pip < 21.0'\n    fi\nfi\n"
  },
  {
    "path": "setup/portage-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Tue Oct 17 20:25:14 2006 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gentoo Portage packages to install - one per line\n\n# additional packages on top of portage-packages-server.txt\n\n# hal dep required for network manager and is hal masked, review this\n# openvas is masked, review this\n#arptables\n#bastille\n#ksh\n#metasploit\n#mwcollect\n#nessus\n#networkmanager\n#noscript\n#openvas\n#xsupplicant\nacct\nacpi\nacpitool\nacroread\nadtool\nair\nairsnort\nalsa-utils\napachetop\napp-admin/analog\napp-admin/ansible\napp-dicts/dictd-dicts\n#app-emacs/gentoo-syntax\n#app-vim/gentoo-syntax\napp-vim/perl-support\nargus\narp-sk\narpd\narping\narpoison\narptools\narpwatch\naspell\n#at\natftpd\naudacious\naudacious-plugins\naudacious-themes\naudit\nauthforce\nautobook\nautocutsel\nautopsy\naxel\nbaghira\n#bash-completion\nbcrypt\nbcwipe\nbiew\n#bind-tools\nbitchx\n#bittorrent\nbonnie++\nbridge-utils\ncatdoc\nccrypt\ncfengine\ncfengine-syntax\nchkrootkit\nchntpw\ncksfv\nclusterssh\n#cmospwd\ncommonbox-styles\ncommonbox-styles-extra\nconky\ncowsay\ncracklib-words\ncryptcat\ncsync2\nctypes\ncurl\ncvs\nd4x\ndavl\ndcfldd\ndclock\ndd-rescue\nddrescue\ndebootstrap\n#denyhosts\ndev-libs/geoip\ndev-perl/HTML-Parser\ndev-perl/List-MoreUtils\ndev-perl/Net-Netmask\ndev-perl/net-ssh-perl\ndev-perl/Perl-Critic\ndev-perl/perltidy\ndev-util/git\ndev-xml/xmlstarlet\ndhcpd-syntax\ndhcping\ndia\ndialog\ndictd\ndjbdns-man\ndnspython\ndnstop\ndnstracer\ndnswalk\ndosdetector\ndosfstools\ndrbd\ndriftnet\ndsniff\ndynagen\ndynamips\n#e2fsprogs\nemacs\neterm\neterm-themes\netherape\nethstatus\n#ethtool\nettercap\nevince\nfiglet\nfilelight\nfirewalk\nfluxbox\nfluxbox-styles-fluxmod\nfluxbox-syntax\nforemost\nfortune-mod\nfping\nfragroute\nftp\ngames-misc/fortune-mod\ngdb\ngdm\ngdm-themes\ngentoo-artwork\ngentoo-artwork-livecd\ngentoo-bashcomp\ngentoo-xcursors\n#gentoolkit\ngftp\ngimp\ngit\ngit-core\ngitk\ngkrellm\ngksu\nglsof\ngnu-netcat\ngnupg\ngparted\ngpm\ngrdesktop\ngrub\ngspoof\ngt5\ngvim\ngxmessage\nheartbeat\nhexedit\n#hexyl  # hex viewer - https://github.com/sharkdp/hexyl - requires sudo eselect repository enable dm9pZCAq && sudo emerge --sync dm9pZCAq && sudo emerge sys-apps/hexyl::dm9pZCAq\nhoneyd\nhostapd\nhping\nhtml2text\nhtop\nhttperf\nhttping\nhttrack\nhunt\nhydra\nifstat\niftop\nike-scan\nimagemagick\nincron\niozone\nip-sentinel\nipcalc\niperf\nipmitool\nipsorcery\niptables\niptraf\niputils\nipython\nirssi\njad-bin\njedit\njhead\njohntheripper\nk3b\nkde\nkhexedit\nkismet\nknock\nkodos\nkrdc\nkuake\nkuroo\nkvirc\nlabrea\nlibextractor\nlinkchecker\nlinklint\nlinks\nlinux-kernel-in-a-nutshell\nlogrotate\nlogsurfer+\nlogwatch\nlsat\nlshw\n#lsof\nlsscsi\nluma\nlvm2\nlynx\nm4\nmac-robber\nmacchanger\nmagicrescue\n#mailx\nmboxgrep\nmc\nmd5deep\nmdadm\nmdcrack\nmedia-gfx/exiftool  # view image metadata\nmedia-gfx/imagemagick\nmemdump\nmercurial\nmost\nmount-cifs\nmozilla-firefox-bin\nmozilla-thunderbird-bin\nmp3info\nmplayer\nmtr\nmultitail\nmysql-gui-tools\nnagios-plugins\nnagios-syntax\nnasm\nnbaudit\nnbd\nnbtscan\nncftp\nnemesis\nnepenthes\nnet-analyzer/amap\nNet-CIDR\nnet-im/psi\nNet-RawIP\nnet-snmp\nnet-tools\nnetbeans\nnetcat\nnetdiscover\nnethogs\nnetkit-telnetd\nnetperf\nnetscape-flash\nnetsed\nnetselect\nnettop\nnfs-kernel-server\nnfs-utils\nngrep\nnikto\nnipper\nnmap\nnmbscan\nnsat\nntfs3g\nntfsprogs\nntop\n#ntp\n#ntp-syntax\nopenipmi\nopenldap\n#openssh\nopenvpn\nopenvpn-blacklist\nophcrack\np0f\npackETH\npackit\npaketto\n#pam-syntax\npam_usb\nparos\nparted\npasco\npastebinit\n#pciutils\nPEAR-PHP_Shell\nperl-doc\npidgin\npidgin-sipe\npigz  # parallel gzip\npinentry\npiozone\npipebench\npipenv\npipeworks\npkcrack\n#portsentry\npound\npscan\npsh\npsi-themes\npssh\npv\npwgen\npychecker\npydns\npyflakes\npygobject\npylint\nqtparted\nrainbowcrack\nrancid\nrancid-cgi\nrancid-core\nrancid-utils\nrar\nrats\nrdesktop\nredhat-artwork\nrkhunter\nroot-tail\nrouter-audit-tool\nrsync\nsamba\nsbd\n#scanlogd\nscanssh\nscreen\nsec\nsecure-delete\nsguil-client\nsguil-sensor\nsguil-server\nsharutils\nsiege\nsipcalc\nsleuthkit\n#slocate\nslurm\nsmartmontools\nsmb4k\nsnort\nsnortsam\nSOAP-Lite\nsocat\nsplint\nsquid\nsquid-graph\nsshfs-fuse\nssmtp\nsteghide\nstrace\nstunnel\nsubversion\n#sudo\nswatch\nsys-apps/ack\nsys-fs/fuse\nsys-process/iotop\n#syslog-ng\nsysstat\ntcpdump\ntcpreplay\ntcpslice\ntcptrack\ntct\ntenshi\ntftp\ntftpd\nthcrut\ntilda\ntoilet\ntomboy\n#traceroute\ntrafshow\ntruecrypt\ntsclient\ntshark\ntwill\ntwisted\nunrar\nunzip\nupnpscan\n#usbutils\nusbview\n#vim\nvirt-manager\n#vixie-cron\nvlock\nvmware-player\nvnc\nvncsnapshot\nwebfuzzer\nweblint\nwebscarab\nwhoischk\nwin32codecs\nwine\nwipe\nwireless-tools\nwireshark\nwmaker\nwmaker-data\nwmctrl\nx11-terms/aterm\nx86info\nxchat\nxclip\nxdotool\nxev\nxfe\nxine-ui\nxkbset\nxlsclients\nxorg-x11\nxosd\nxosview\nxprobe\nxrefresh\nxrestop\nxrootconsole\nxscreensaver\nxsel\nxterm\nxvid\nyakuake\nyersinia\nzenity\nzip\nzsh\n"
  },
  {
    "path": "setup/portage-packages-dockapps.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Fri Oct 20 16:44:03 2006 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gentoo Portage packages to install - one per line\n\nasmon\nmountapp\nwmbio\nwmblob\nwmbutton\nwmclockmon\nwmcpuload\nwmcube\nwmdots\nwmdiskmon\nwmdrawer\nwmfire\nwmfsm\nwmhdplop\nwmifinfo\nwminet\nwmix\nwmlongrun\nwmMatrix\nwmmemmon\nwmmisc\nwmmon\nwmnd\nwmnetload\nwmnetmon\nwmpiki\nwmpower\nwmtop\nwmwave\nxscreensaver-app\n"
  },
  {
    "path": "setup/portage-packages-extras.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Tue Oct 17 20:25:14 2006 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gentoo Portage packages to install - one per line\n\n# additional packages on top of portage-packages-server.txt and portage-packages-desktop.txt\n\nabs-guide\nalien\napp-misc/beagle\nbar\nbing\nbittorrent\nbwmon\ncherrypy\nckermit\ncmospwd\ncutecom\ndjbdns\nechoping\nflasm\nganglia\nhdparm\nhtdig\nhtmltidy\ngnome-power-manager\nkrusader\niplog\niptstate\nisic\nkchmviewer\nkolourpaint\nlinpopup\nlinux-docs\nlinux-gazette-all\nlinux-logo\nlogsentry\nmakepasswd\nmbrowse\nminicom\nmp3_check\nndiswrapper\nnet-mail/imapsync\nnetwag\nnetwox\nopera\npbnj\npflogsumm\nphrack-all\nporthole\npowertop\nrain\nraccess\nseamonkey\nsiphon\nsmokeping\nsniffit\nssldump\nsslwrap\ntmpwatch\nvalgrind\nwatchpid\nwebalizer\nwebmin\nwhowatch\nwindowmaker\nx11-themes/windowmaker-themes\n#wmctrl\nxbatt\nxbattbar\nxchm\nxconsole\nxdiskusage\nxpdf\nxpenguins\nxsimpsons\n"
  },
  {
    "path": "setup/portage-packages-server.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Tue May 27 20:29:57 2008 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Gentoo Portage packages to install - one per line\n\napp-emacs/gentoo-syntax\napp-vim/gentoo-syntax\nat\nbash-completion\nbind-tools\ndenyhosts\ne2fsprogs\nethtool\ngentoolkit\nlsof\nmailx\nntp\nntp-syntax\nopenssh\npam-syntax\npciutils\nportsentry\nscanlogd\nslocate\nsudo\nsyslog-ng\ntraceroute\nusbutils\nvim\nvixie-cron\n"
  },
  {
    "path": "setup/postgresql.conf",
    "content": "# vim:ts=8:sts=8:sw=8:et\n# -----------------------------\n# PostgreSQL configuration file\n# -----------------------------\n#\n# This file consists of lines of the form:\n#\n#   name = value\n#\n# (The \"=\" is optional.)  Whitespace may be used.  Comments are introduced with\n# \"#\" anywhere on a line.  The complete list of parameter names and allowed\n# values can be found in the PostgreSQL documentation.\n#\n# The commented-out settings shown in this file represent the default values.\n# Re-commenting a setting is NOT sufficient to revert it to the default value;\n# you need to reload the server.\n#\n# This file is read on server startup and when the server receives a SIGHUP\n# signal.  If you edit the file on a running system, you have to SIGHUP the\n# server for the changes to take effect, run \"pg_ctl reload\", or execute\n# \"SELECT pg_reload_conf()\".  Some parameters, which are marked below,\n# require a server shutdown and restart to take effect.\n#\n# Any parameter can also be given as a command-line option to the server, e.g.,\n# \"postgres -c log_connections=on\".  Some parameters can be changed at run time\n# with the \"SET\" SQL command.\n#\n# Memory units:  kB = kilobytes        Time units:  ms  = milliseconds\n#                MB = megabytes                     s   = seconds\n#                GB = gigabytes                     min = minutes\n#                TB = terabytes                     h   = hours\n#                                                   d   = days\n\n\n#------------------------------------------------------------------------------\n# FILE LOCATIONS\n#------------------------------------------------------------------------------\n\n# The default values of these variables are driven from the -D command-line\n# option or PGDATA environment variable, represented here as ConfigDir.\n\n#data_directory = 'ConfigDir'           # use data in another directory\n                                        # (change requires restart)\n#hba_file = 'ConfigDir/pg_hba.conf'     # host-based authentication file\n                                        # (change requires restart)\n#ident_file = 'ConfigDir/pg_ident.conf' # ident configuration file\n                                        # (change requires restart)\n\n# If external_pid_file is not explicitly set, no extra PID file is written.\n#external_pid_file = ''                 # write an extra PID file\n                                        # (change requires restart)\n\n\n#------------------------------------------------------------------------------\n# CONNECTIONS AND AUTHENTICATION\n#------------------------------------------------------------------------------\n\n# - Connection Settings -\n\nlisten_addresses = '*'\n                                        # comma-separated list of addresses;\n                                        # defaults to 'localhost'; use '*' for all\n                                        # (change requires restart)\n#port = 5432                            # (change requires restart)\n#max_connections = 100                  # (change requires restart)\n#superuser_reserved_connections = 3     # (change requires restart)\n#unix_socket_directories = '/tmp'       # comma-separated list of directories\n                                        # (change requires restart)\n#unix_socket_group = ''                 # (change requires restart)\n#unix_socket_permissions = 0777         # begin with 0 to use octal notation\n                                        # (change requires restart)\n#bonjour = off                          # advertise server via Bonjour\n                                        # (change requires restart)\n#bonjour_name = ''                      # defaults to the computer name\n                                        # (change requires restart)\n\n# - TCP settings -\n# see \"man 7 tcp\" for details\n\n#tcp_keepalives_idle = 0                # TCP_KEEPIDLE, in seconds;\n                                        # 0 selects the system default\n#tcp_keepalives_interval = 0            # TCP_KEEPINTVL, in seconds;\n                                        # 0 selects the system default\n#tcp_keepalives_count = 0               # TCP_KEEPCNT;\n                                        # 0 selects the system default\n#tcp_user_timeout = 0                   # TCP_USER_TIMEOUT, in milliseconds;\n                                        # 0 selects the system default\n\n# - Authentication -\n\n#authentication_timeout = 1min          # 1s-600s\n#password_encryption = md5              # md5 or scram-sha-256\n#db_user_namespace = off\n\n# GSSAPI using Kerberos\n#krb_server_keyfile = ''\n#krb_caseins_users = off\n\n# - SSL -\n\n#ssl = off\n#ssl_ca_file = ''\n#ssl_cert_file = 'server.crt'\n#ssl_crl_file = ''\n#ssl_key_file = 'server.key'\n#ssl_ciphers = 'HIGH:MEDIUM:+3DES:!aNULL' # allowed SSL ciphers\n#ssl_prefer_server_ciphers = on\n#ssl_ecdh_curve = 'prime256v1'\n#ssl_min_protocol_version = 'TLSv1'\n#ssl_max_protocol_version = ''\n#ssl_dh_params_file = ''\n#ssl_passphrase_command = ''\n#ssl_passphrase_command_supports_reload = off\n\n\n#------------------------------------------------------------------------------\n# RESOURCE USAGE (except WAL)\n#------------------------------------------------------------------------------\n\n# - Memory -\n\n#shared_buffers = 32MB                  # min 128kB\n                                        # (change requires restart)\n#huge_pages = try                       # on, off, or try\n                                        # (change requires restart)\n#temp_buffers = 8MB                     # min 800kB\n#max_prepared_transactions = 0          # zero disables the feature\n                                        # (change requires restart)\n# Caution: it is not advisable to set max_prepared_transactions nonzero unless\n# you actively intend to use prepared transactions.\n#work_mem = 4MB                         # min 64kB\n#maintenance_work_mem = 64MB            # min 1MB\n#autovacuum_work_mem = -1               # min 1MB, or -1 to use maintenance_work_mem\n#max_stack_depth = 2MB                  # min 100kB\n#shared_memory_type = mmap              # the default is the first option\n                                        # supported by the operating system:\n                                        #   mmap\n                                        #   sysv\n                                        #   windows\n                                        # (change requires restart)\n#dynamic_shared_memory_type = posix     # the default is the first option\n                                        # supported by the operating system:\n                                        #   posix\n                                        #   sysv\n                                        #   windows\n                                        #   mmap\n                                        # (change requires restart)\n\n# - Disk -\n\n#temp_file_limit = -1                   # limits per-process temp file space\n                                        # in kB, or -1 for no limit\n\n# - Kernel Resources -\n\n#max_files_per_process = 1000           # min 25\n                                        # (change requires restart)\n\n# - Cost-Based Vacuum Delay -\n\n#vacuum_cost_delay = 0                  # 0-100 milliseconds (0 disables)\n#vacuum_cost_page_hit = 1               # 0-10000 credits\n#vacuum_cost_page_miss = 10             # 0-10000 credits\n#vacuum_cost_page_dirty = 20            # 0-10000 credits\n#vacuum_cost_limit = 200                # 1-10000 credits\n\n# - Background Writer -\n\n#bgwriter_delay = 200ms                 # 10-10000ms between rounds\n#bgwriter_lru_maxpages = 100            # max buffers written/round, 0 disables\n#bgwriter_lru_multiplier = 2.0          # 0-10.0 multiplier on buffers scanned/round\n#bgwriter_flush_after = 0               # measured in pages, 0 disables\n\n# - Asynchronous Behavior -\n\n#effective_io_concurrency = 1           # 1-1000; 0 disables prefetching\n#max_worker_processes = 8               # (change requires restart)\n#max_parallel_maintenance_workers = 2   # taken from max_parallel_workers\n#max_parallel_workers_per_gather = 2    # taken from max_parallel_workers\n#parallel_leader_participation = on\n#max_parallel_workers = 8               # maximum number of max_worker_processes that\n                                        # can be used in parallel operations\n#old_snapshot_threshold = -1            # 1min-60d; -1 disables; 0 is immediate\n                                        # (change requires restart)\n#backend_flush_after = 0                # measured in pages, 0 disables\n\n\n#------------------------------------------------------------------------------\n# WRITE-AHEAD LOG\n#------------------------------------------------------------------------------\n\n# - Settings -\n\n#wal_level = replica                    # minimal, replica, or logical\n                                        # (change requires restart)\n#fsync = on                             # flush data to disk for crash safety\n                                        # (turning this off can cause\n                                        # unrecoverable data corruption)\n#synchronous_commit = on                # synchronization level;\n                                        # off, local, remote_write, remote_apply, or on\n#wal_sync_method = fsync                # the default is the first option\n                                        # supported by the operating system:\n                                        #   open_datasync\n                                        #   fdatasync (default on Linux)\n                                        #   fsync\n                                        #   fsync_writethrough\n                                        #   open_sync\n#full_page_writes = on                  # recover from partial page writes\n#wal_compression = off                  # enable compression of full-page writes\n#wal_log_hints = off                    # also do full page writes of non-critical updates\n                                        # (change requires restart)\n#wal_init_zero = on                     # zero-fill new WAL files\n#wal_recycle = on                       # recycle WAL files\n#wal_buffers = -1                       # min 32kB, -1 sets based on shared_buffers\n                                        # (change requires restart)\n#wal_writer_delay = 200ms               # 1-10000 milliseconds\n#wal_writer_flush_after = 1MB           # measured in pages, 0 disables\n\n#commit_delay = 0                       # range 0-100000, in microseconds\n#commit_siblings = 5                    # range 1-1000\n\n# - Checkpoints -\n\n#checkpoint_timeout = 5min              # range 30s-1d\n#max_wal_size = 1GB\n#min_wal_size = 80MB\n#checkpoint_completion_target = 0.5     # checkpoint target duration, 0.0 - 1.0\n#checkpoint_flush_after = 0             # measured in pages, 0 disables\n#checkpoint_warning = 30s               # 0 disables\n\n# - Archiving -\n\n#archive_mode = off             # enables archiving; off, on, or always\n                                # (change requires restart)\n#archive_command = ''           # command to use to archive a logfile segment\n                                # placeholders: %p = path of file to archive\n                                #               %f = file name only\n                                # e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'\n#archive_timeout = 0            # force a logfile segment switch after this\n                                # number of seconds; 0 disables\n\n# - Archive Recovery -\n\n# These are only used in recovery mode.\n\n#restore_command = ''           # command to use to restore an archived logfile segment\n                                # placeholders: %p = path of file to restore\n                                #               %f = file name only\n                                # e.g. 'cp /mnt/server/archivedir/%f %p'\n                                # (change requires restart)\n#archive_cleanup_command = ''   # command to execute at every restartpoint\n#recovery_end_command = ''      # command to execute at completion of recovery\n\n# - Recovery Target -\n\n# Set these only when performing a targeted recovery.\n\n#recovery_target = ''           # 'immediate' to end recovery as soon as a\n                                # consistent state is reached\n                                # (change requires restart)\n#recovery_target_name = ''      # the named restore point to which recovery will proceed\n                                # (change requires restart)\n#recovery_target_time = ''      # the time stamp up to which recovery will proceed\n                                # (change requires restart)\n#recovery_target_xid = ''       # the transaction ID up to which recovery will proceed\n                                # (change requires restart)\n#recovery_target_lsn = ''       # the WAL LSN up to which recovery will proceed\n                                # (change requires restart)\n#recovery_target_inclusive = on # Specifies whether to stop:\n                                # just after the specified recovery target (on)\n                                # just before the recovery target (off)\n                                # (change requires restart)\n#recovery_target_timeline = 'latest'    # 'current', 'latest', or timeline ID\n                                # (change requires restart)\n#recovery_target_action = 'pause'       # 'pause', 'promote', 'shutdown'\n                                # (change requires restart)\n\n\n#------------------------------------------------------------------------------\n# REPLICATION\n#------------------------------------------------------------------------------\n\n# - Sending Servers -\n\n# Set these on the master and on any standby that will send replication data.\n\n#max_wal_senders = 10           # max number of walsender processes\n                                # (change requires restart)\n#wal_keep_segments = 0          # in logfile segments; 0 disables\n#wal_sender_timeout = 60s       # in milliseconds; 0 disables\n\n#max_replication_slots = 10     # max number of replication slots\n                                # (change requires restart)\n#track_commit_timestamp = off   # collect timestamp of transaction commit\n                                # (change requires restart)\n\n# - Master Server -\n\n# These settings are ignored on a standby server.\n\n#synchronous_standby_names = '' # standby servers that provide sync rep\n                                # method to choose sync standbys, number of sync standbys,\n                                # and comma-separated list of application_name\n                                # from standby(s); '*' = all\n#vacuum_defer_cleanup_age = 0   # number of xacts by which cleanup is delayed\n\n# - Standby Servers -\n\n# These settings are ignored on a master server.\n\n#primary_conninfo = ''                  # connection string to sending server\n                                        # (change requires restart)\n#primary_slot_name = ''                 # replication slot on sending server\n                                        # (change requires restart)\n#promote_trigger_file = ''              # file name whose presence ends recovery\n#hot_standby = on                       # \"off\" disallows queries during recovery\n                                        # (change requires restart)\n#max_standby_archive_delay = 30s        # max delay before canceling queries\n                                        # when reading WAL from archive;\n                                        # -1 allows indefinite delay\n#max_standby_streaming_delay = 30s      # max delay before canceling queries\n                                        # when reading streaming WAL;\n                                        # -1 allows indefinite delay\n#wal_receiver_status_interval = 10s     # send replies at least this often\n                                        # 0 disables\n#hot_standby_feedback = off             # send info from standby to prevent\n                                        # query conflicts\n#wal_receiver_timeout = 60s             # time that receiver waits for\n                                        # communication from master\n                                        # in milliseconds; 0 disables\n#wal_retrieve_retry_interval = 5s       # time to wait before retrying to\n                                        # retrieve WAL after a failed attempt\n#recovery_min_apply_delay = 0           # minimum delay for applying changes during recovery\n\n# - Subscribers -\n\n# These settings are ignored on a publisher.\n\n#max_logical_replication_workers = 4    # taken from max_worker_processes\n                                        # (change requires restart)\n#max_sync_workers_per_subscription = 2  # taken from max_logical_replication_workers\n\n\n#------------------------------------------------------------------------------\n# QUERY TUNING\n#------------------------------------------------------------------------------\n\n# - Planner Method Configuration -\n\n#enable_bitmapscan = on\n#enable_hashagg = on\n#enable_hashjoin = on\n#enable_indexscan = on\n#enable_indexonlyscan = on\n#enable_material = on\n#enable_mergejoin = on\n#enable_nestloop = on\n#enable_parallel_append = on\n#enable_seqscan = on\n#enable_sort = on\n#enable_tidscan = on\n#enable_partitionwise_join = off\n#enable_partitionwise_aggregate = off\n#enable_parallel_hash = on\n#enable_partition_pruning = on\n\n# - Planner Cost Constants -\n\n#seq_page_cost = 1.0                    # measured on an arbitrary scale\n#random_page_cost = 4.0                 # same scale as above\n#cpu_tuple_cost = 0.01                  # same scale as above\n#cpu_index_tuple_cost = 0.005           # same scale as above\n#cpu_operator_cost = 0.0025             # same scale as above\n#parallel_tuple_cost = 0.1              # same scale as above\n#parallel_setup_cost = 1000.0   # same scale as above\n\n#jit_above_cost = 100000                # perform JIT compilation if available\n                                        # and query more expensive than this;\n                                        # -1 disables\n#jit_inline_above_cost = 500000         # inline small functions if query is\n                                        # more expensive than this; -1 disables\n#jit_optimize_above_cost = 500000       # use expensive JIT optimizations if\n                                        # query is more expensive than this;\n                                        # -1 disables\n\n#min_parallel_table_scan_size = 8MB\n#min_parallel_index_scan_size = 512kB\n#effective_cache_size = 4GB\n\n# - Genetic Query Optimizer -\n\n#geqo = on\n#geqo_threshold = 12\n#geqo_effort = 5                        # range 1-10\n#geqo_pool_size = 0                     # selects default based on effort\n#geqo_generations = 0                   # selects default based on effort\n#geqo_selection_bias = 2.0              # range 1.5-2.0\n#geqo_seed = 0.0                        # range 0.0-1.0\n\n# - Other Planner Options -\n\n#default_statistics_target = 100        # range 1-10000\n#constraint_exclusion = partition       # on, off, or partition\n#cursor_tuple_fraction = 0.1            # range 0.0-1.0\n#from_collapse_limit = 8\n#join_collapse_limit = 8                # 1 disables collapsing of explicit\n                                        # JOIN clauses\n#force_parallel_mode = off\n#jit = on                               # allow JIT compilation\n#plan_cache_mode = auto                 # auto, force_generic_plan or\n                                        # force_custom_plan\n\n\n#------------------------------------------------------------------------------\n# REPORTING AND LOGGING\n#------------------------------------------------------------------------------\n\n# - Where to Log -\n\n#log_destination = 'stderr'             # Valid values are combinations of\n                                        # stderr, csvlog, syslog, and eventlog,\n                                        # depending on platform.  csvlog\n                                        # requires logging_collector to be on.\n\n# This is used when logging to stderr:\n#logging_collector = off                # Enable capturing of stderr and csvlog\n                                        # into log files. Required to be on for\n                                        # csvlogs.\n                                        # (change requires restart)\n\n# These are only used if logging_collector is on:\n#log_directory = 'log'                  # directory where log files are written,\n                                        # can be absolute or relative to PGDATA\n#log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'        # log file name pattern,\n                                        # can include strftime() escapes\n#log_file_mode = 0600                   # creation mode for log files,\n                                        # begin with 0 to use octal notation\n#log_truncate_on_rotation = off         # If on, an existing log file with the\n                                        # same name as the new log file will be\n                                        # truncated rather than appended to.\n                                        # But such truncation only occurs on\n                                        # time-driven rotation, not on restarts\n                                        # or size-driven rotation.  Default is\n                                        # off, meaning append to existing files\n                                        # in all cases.\n#log_rotation_age = 1d                  # Automatic rotation of logfiles will\n                                        # happen after that time.  0 disables.\n#log_rotation_size = 10MB               # Automatic rotation of logfiles will\n                                        # happen after that much log output.\n                                        # 0 disables.\n\n# These are relevant when logging to syslog:\n#syslog_facility = 'LOCAL0'\n#syslog_ident = 'postgres'\n#syslog_sequence_numbers = on\n#syslog_split_messages = on\n\n# This is only relevant when logging to eventlog (win32):\n# (change requires restart)\n#event_source = 'PostgreSQL'\n\n# - When to Log -\n\n#log_min_messages = warning             # values in order of decreasing detail:\n                                        #   debug5\n                                        #   debug4\n                                        #   debug3\n                                        #   debug2\n                                        #   debug1\n                                        #   info\n                                        #   notice\n                                        #   warning\n                                        #   error\n                                        #   log\n                                        #   fatal\n                                        #   panic\n\n#log_min_error_statement = error        # values in order of decreasing detail:\n                                        #   debug5\n                                        #   debug4\n                                        #   debug3\n                                        #   debug2\n                                        #   debug1\n                                        #   info\n                                        #   notice\n                                        #   warning\n                                        #   error\n                                        #   log\n                                        #   fatal\n                                        #   panic (effectively off)\n\n#log_min_duration_statement = -1        # -1 is disabled, 0 logs all statements\n                                        # and their durations, > 0 logs only\n                                        # statements running at least this number\n                                        # of milliseconds\n\n#log_transaction_sample_rate = 0.0      # Fraction of transactions whose statements\n                                        # are logged regardless of their duration. 1.0 logs all\n                                        # statements from all transactions, 0.0 never logs.\n\n# - What to Log -\n\n#debug_print_parse = off\n#debug_print_rewritten = off\n#debug_print_plan = off\n#debug_pretty_print = on\n#log_checkpoints = off\n#log_connections = off\n#log_disconnections = off\n#log_duration = off\n#log_error_verbosity = default          # terse, default, or verbose messages\n#log_hostname = off\n#log_line_prefix = '%m [%p] '           # special values:\n                                        #   %a = application name\n                                        #   %u = user name\n                                        #   %d = database name\n                                        #   %r = remote host and port\n                                        #   %h = remote host\n                                        #   %p = process ID\n                                        #   %t = timestamp without milliseconds\n                                        #   %m = timestamp with milliseconds\n                                        #   %n = timestamp with milliseconds (as a Unix epoch)\n                                        #   %i = command tag\n                                        #   %e = SQL state\n                                        #   %c = session ID\n                                        #   %l = session line number\n                                        #   %s = session start timestamp\n                                        #   %v = virtual transaction ID\n                                        #   %x = transaction ID (0 if none)\n                                        #   %q = stop here in non-session\n                                        #        processes\n                                        #   %% = '%'\n                                        # e.g. '<%u%%%d> '\n#log_lock_waits = off                   # log lock waits >= deadlock_timeout\n#log_statement = 'none'                 # none, ddl, mod, all\n#log_replication_commands = off\n#log_temp_files = -1                    # log temporary files equal or larger\n                                        # than the specified size in kilobytes;\n                                        # -1 disables, 0 logs all temp files\n#log_timezone = 'GMT'\n\n#------------------------------------------------------------------------------\n# PROCESS TITLE\n#------------------------------------------------------------------------------\n\n#cluster_name = ''                      # added to process titles if nonempty\n                                        # (change requires restart)\n#update_process_title = on\n\n\n#------------------------------------------------------------------------------\n# STATISTICS\n#------------------------------------------------------------------------------\n\n# - Query and Index Statistics Collector -\n\n#track_activities = on\n#track_counts = on\n#track_io_timing = off\n#track_functions = none                 # none, pl, all\n#track_activity_query_size = 1024       # (change requires restart)\n#stats_temp_directory = 'pg_stat_tmp'\n\n\n# - Monitoring -\n\n#log_parser_stats = off\n#log_planner_stats = off\n#log_executor_stats = off\n#log_statement_stats = off\n\n\n#------------------------------------------------------------------------------\n# AUTOVACUUM\n#------------------------------------------------------------------------------\n\n#autovacuum = on                        # Enable autovacuum subprocess?  'on'\n                                        # requires track_counts to also be on.\n#log_autovacuum_min_duration = -1       # -1 disables, 0 logs all actions and\n                                        # their durations, > 0 logs only\n                                        # actions running at least this number\n                                        # of milliseconds.\n#autovacuum_max_workers = 3             # max number of autovacuum subprocesses\n                                        # (change requires restart)\n#autovacuum_naptime = 1min              # time between autovacuum runs\n#autovacuum_vacuum_threshold = 50       # min number of row updates before\n                                        # vacuum\n#autovacuum_analyze_threshold = 50      # min number of row updates before\n                                        # analyze\n#autovacuum_vacuum_scale_factor = 0.2   # fraction of table size before vacuum\n#autovacuum_analyze_scale_factor = 0.1  # fraction of table size before analyze\n#autovacuum_freeze_max_age = 200000000  # maximum XID age before forced vacuum\n                                        # (change requires restart)\n#autovacuum_multixact_freeze_max_age = 400000000        # maximum multixact age\n                                        # before forced vacuum\n                                        # (change requires restart)\n#autovacuum_vacuum_cost_delay = 2ms     # default vacuum cost delay for\n                                        # autovacuum, in milliseconds;\n                                        # -1 means use vacuum_cost_delay\n#autovacuum_vacuum_cost_limit = -1      # default vacuum cost limit for\n                                        # autovacuum, -1 means use\n                                        # vacuum_cost_limit\n\n\n#------------------------------------------------------------------------------\n# CLIENT CONNECTION DEFAULTS\n#------------------------------------------------------------------------------\n\n# - Statement Behavior -\n\n#client_min_messages = notice           # values in order of decreasing detail:\n                                        #   debug5\n                                        #   debug4\n                                        #   debug3\n                                        #   debug2\n                                        #   debug1\n                                        #   log\n                                        #   notice\n                                        #   warning\n                                        #   error\n#search_path = '\"$user\", public'        # schema names\n#row_security = on\n#default_tablespace = ''                # a tablespace name, '' uses the default\n#temp_tablespaces = ''                  # a list of tablespace names, '' uses\n                                        # only default tablespace\n#default_table_access_method = 'heap'\n#check_function_bodies = on\n#default_transaction_isolation = 'read committed'\n#default_transaction_read_only = off\n#default_transaction_deferrable = off\n#session_replication_role = 'origin'\n#statement_timeout = 0                  # in milliseconds, 0 is disabled\n#lock_timeout = 0                       # in milliseconds, 0 is disabled\n#idle_in_transaction_session_timeout = 0        # in milliseconds, 0 is disabled\n#vacuum_freeze_min_age = 50000000\n#vacuum_freeze_table_age = 150000000\n#vacuum_multixact_freeze_min_age = 5000000\n#vacuum_multixact_freeze_table_age = 150000000\n#vacuum_cleanup_index_scale_factor = 0.1        # fraction of total number of tuples\n                                                # before index cleanup, 0 always performs\n                                                # index cleanup\n#bytea_output = 'hex'                   # hex, escape\n#xmlbinary = 'base64'\n#xmloption = 'content'\n#gin_fuzzy_search_limit = 0\n#gin_pending_list_limit = 4MB\n\n# - Locale and Formatting -\n\n#datestyle = 'iso, mdy'\n#intervalstyle = 'postgres'\n#timezone = 'GMT'\n#timezone_abbreviations = 'Default'     # Select the set of available time zone\n                                        # abbreviations.  Currently, there are\n                                        #   Default\n                                        #   Australia (historical usage)\n                                        #   India\n                                        # You can create your own file in\n                                        # share/timezonesets/.\n#extra_float_digits = 1                 # min -15, max 3; any value >0 actually\n                                        # selects precise output mode\n#client_encoding = sql_ascii            # actually, defaults to database\n                                        # encoding\n\n# These settings are initialized by initdb, but they can be changed.\n#lc_messages = 'C'                      # locale for system error message\n                                        # strings\n#lc_monetary = 'C'                      # locale for monetary formatting\n#lc_numeric = 'C'                       # locale for number formatting\n#lc_time = 'C'                          # locale for time formatting\n\n# default configuration for text search\n#default_text_search_config = 'pg_catalog.simple'\n\n# - Shared Library Preloading -\n\nshared_preload_libraries = 'pg_stat_statements' # (change requires restart)\n#local_preload_libraries = ''\n#session_preload_libraries = ''\n#jit_provider = 'llvmjit'               # JIT library to use\n\n# - Other Defaults -\n\n#dynamic_library_path = '$libdir'\n\n\n#------------------------------------------------------------------------------\n# LOCK MANAGEMENT\n#------------------------------------------------------------------------------\n\n#deadlock_timeout = 1s\n#max_locks_per_transaction = 64         # min 10\n                                        # (change requires restart)\n#max_pred_locks_per_transaction = 64    # min 10\n                                        # (change requires restart)\n#max_pred_locks_per_relation = -2       # negative values mean\n                                        # (max_pred_locks_per_transaction\n                                        #  / -max_pred_locks_per_relation) - 1\n#max_pred_locks_per_page = 2            # min 0\n\n\n#------------------------------------------------------------------------------\n# VERSION AND PLATFORM COMPATIBILITY\n#------------------------------------------------------------------------------\n\n# - Previous PostgreSQL Versions -\n\n#array_nulls = on\n#backslash_quote = safe_encoding        # on, off, or safe_encoding\n#escape_string_warning = on\n#lo_compat_privileges = off\n#operator_precedence_warning = off\n#quote_all_identifiers = off\n#standard_conforming_strings = on\n#synchronize_seqscans = on\n\n# - Other Platforms and Clients -\n\n#transform_null_equals = off\n\n\n#------------------------------------------------------------------------------\n# ERROR HANDLING\n#------------------------------------------------------------------------------\n\n#exit_on_error = off                    # terminate session on any error?\n#restart_after_crash = on               # reinitialize after backend crash?\n#data_sync_retry = off                  # retry or panic on failure to fsync\n                                        # data?\n                                        # (change requires restart)\n\n\n#------------------------------------------------------------------------------\n# CONFIG FILE INCLUDES\n#------------------------------------------------------------------------------\n\n# These options allow settings to be loaded from files other than the\n# default postgresql.conf.  Note that these are directives, not variable\n# assignments, so they can usefully be given more than once.\n\n#include_dir = '...'                    # include files ending in '.conf' from\n                                        # a directory, e.g., 'conf.d'\n#include_if_exists = '...'              # include file only if it exists\n#include = '...'                        # include file\n\n\n#------------------------------------------------------------------------------\n# CUSTOMIZED OPTIONS\n#------------------------------------------------------------------------------\n\n# Add settings for extensions here\n"
  },
  {
    "path": "setup/prometheus.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: Wed Jan 5 16:53:05 2022 +0000\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/Templates\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                              P r o m e t h e u s\n# ============================================================================ #\n\n---\nglobal:\n  scrape_interval: 15s # default: 1 minute\n  evaluation_interval: 15s # evaluate rules every 15 sess (default: 1 minute)\n  #scrape_timeout: 10s\n\n#alerting:\n#  alertmanagers:\n#    - static_configs:\n#        - targets:\n          # - alertmanager:9093\n\n# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.\n#rule_files:\n  # - \"first_rules.yml\"\n  # - \"second_rules.yml\"\n\nscrape_configs:\n  # job_name adds label `job=<job_name>` to timeseries metrics\n  - job_name: prometheus\n    #metrics_path: /metrics\n    #scheme: http\n    #honor_timestamps: true\n    #enable_compression: true\n    #follow_redirects: true\n    #enable_http2: true\n    static_configs:\n      - targets:\n          - localhost:9090\n\n  - job_name: demo\n    static_configs:\n      - targets:\n          - demo.promlabs.com:10000\n          - demo.promlabs.com:10001\n          - demo.promlabs.com:10002\n"
  },
  {
    "path": "setup/python_install_snakebite.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-06 10:37:29 +0000 (Fri, 06 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Tries to installs Python snakebite module for Python 3 or Python 2 downgrading each time to try another version\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n\"$srcdir/../python/python_pip_install.sh\" \"snakebite-py3[kerberos]\" ||\n\"$srcdir/../python/python_pip_install.sh\" \"snakebite-py3\" ||\n\"$srcdir/../python/python_pip_install.sh\" \"snakebite[kerberos]\" ||\n\"$srcdir/../python/python_pip_install.sh\" \"snakebite\"\n"
  },
  {
    "path": "setup/python_mac_upgrade_ssl_fix.sh",
    "content": "#!/usr/bin/env bash\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-16\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# from https://stackoverflow.com/questions/24287239/abort-trap-6-when-running-a-python-script\n\nset -euo pipefail\n\nsudo=\"\"\n[ $EUID -eq 0 ] || sudo=sudo\n\n#brew update && brew upgrade && brew install openssl\ncd /usr/local/Cellar/openssl/*/lib\nsudo cp -- libssl.1.0.0.dylib libcrypto.1.0.0.dylib /usr/local/lib/\ncd /usr/local/lib\n[ -f libssl.dylib ] ||\n    $sudo ln -s -- libssl.1.0.0.dylib libssl.dylib\n[ -f libcrypto.dylib ] ||\n    $sudo ln -s -- libcrypto.1.0.0.dylib libcrypto.dylib\n#pip3 install --upgrade packagename\n"
  },
  {
    "path": "setup/repos.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-01-17 12:15:39 +0000 (Sun, 17 Jan 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Repos to git clone or update, one per line\n#\n# Format:\n#\n#  <user_or_org>/<repo>[:local_dir]\n#\n# eg.\n#\n#  HariSekhon/Nagios-Plugins\n#\n# if user or organization prefix is not specified then defaults to the HariSekhon account\n#\n# \":local_dir\" is optional in case you want to shorten the local directory name, lowercase it or change it for any reason such as daily convenience\n#\n\nHariSekhon:harisekhon\n#Knowledge-Base:knowledge\nEnvironments:environments\nTamperMonkey:tampermonkey\nHammerspoon:hammerspoon\nMPV-Scripts:mpvscripts\n# order repos by dependent submodule repos first\nGitHub-Actions-Contexts:contexts\nSQL-keywords:sql-keywords\nGitHub-Actions:github-actions\nJenkins:jenkins\nVagrant:vag\nSQL-scripts:sql\nKubernetes-configs:k8s\nTerraform:terraform\nPacker:pack\nTemplates:templates\nTeamCity-CI:teamcity\nDevOps-Bash-tools:bash-tools\n\n# libraries first - if these don't work then the dependent repos beneath them won't work\nlib\npylib\nlib-java:libj\nNagios-Plugin-Kafka:nagios-plugin-kafka\n\nAnsible:ansible\nDiagrams-as-Code:diagrams\nDevOps-Golang-tools:go-tools\nDevOps-Perl-tools:perl-tools\nDevOps-Python-tools:pytools\nSpotify-tools:spotify-tools\nSpotify-Playlists:playlists\n#Spark-Apps:spark-apps\nTemplate-repo:trepo\n\nHAProxy-configs:haproxy-configs\n\nDockerfiles\n\nNagios-Plugins:nagios-plugins\n"
  },
  {
    "path": "setup/rpm-packages-desktop.txt",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Tue Mar 18 21:20:52 2008 +0000\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# RPM desktop packages - one per line\n\n#mwcollect\nacpi\nacpitool\nair\nairsnort\nalsa-utils\nansible\n#app-dicts/dictd-dicts\n#app-vim/perl-support\narpd\narpwatch\naterm\natftpd\nautocutsel\nautojump\nautojump-zsh\nautossh\nbash-completion\nbittorrent\nbridge-utils\ncatdoc\nchntpw\nclusterssh\ncmospwd\ncowsay\ncsync2\ncvs\ndavl\ndebootstrap\ndenyhosts\n#dev-perl/Net-Netmask\n#dev-perl/net-ssh-perl\n#dev-perl/Perl-Critic\n#dev-perl/perltidy\ndia\ndialog\ndnspython\ndrbd\ndsniff\ndynagen\ndynamips\nemacs\neterm-themes\nettercap-gtk\nevince\nfiglet\nfilelight\nfluxbox\nforemost\nfortune-mod\nfzf\n#games-misc/fortune-mod\ngdb\ngeoip\ngimp\n#git # in rpm-packages.txt\ngit-core\ngitk\ngkrellm\ngksu\ngraphviz  # for the 'dot' command (use with terraform graph), and also github.com/HariSekhon/Diagrams-as-Code\ngt5\ngvim\nheartbeat\nhexedit\nhexyl  # hex viewer - https://github.com/sharkdp/hexyl\nhoneyd\nhostapd\nImageMagick\nincron\niozone\niotop\nipcalc\nipmitool\niptraf\nipvsadm\nipython\nirssi\njedit\njhead\nkdebase\nkdenetwork\nkhexedit\nkodos\nksh\nkubernetes-client\nkvirc\nlshw\nlsscsi\nluma\nlvm2\nm4\nmagicrescue\nman\nman-pages\nmboxgrep\nmdadm\nmemdump\nmercurial\nmosh\nmp3info\nmycli\nmysql\nMySQL-python\nmysql-server\nmytop\nnagios-plugins\nnagios-plugins-mysql\nnasm\nnbd\nnbtscan\nnc\nnepenthes\nNet-CIDR\nnet-snmp\nnetbeans\nnetperf\nnetselect\nnfs-kernel-server\nnfs-utils\nnmap\nntfs3g\nopenipmi\nopenldap\nopenvpn\nopenvpn-blacklist\npaketto\npasco\npastebinit\npdfgrep\nPEAR-PHP_Shell\nperl-doc\nperl-HTML-Parser\nperl-Image-ExifTool\npidgin\npidgin-sipe\npigz  # parallel gzip\npiozone\npipenv\npound\npsh\npsmisc  # for pstree\npssh\npv\npydns\npygobject\npylint\npython-pygments\nrancid\nrancid-cgi\nrancid-core\nrancid-utils\nrbenv\nrdesktop\nrouter-audit-tool\nrpm-build\nrpmlint\nsbd\n#screen  # removed in favour of tmux :'-(\nsguil-client\nsguil-sensor\nsguil-server\nskopeo\nsipcalc\nslurm\nsnort\nsnortsam\nsquid\nsquid-graph\nstrace\nsubversion\nsys-apps/ack\ntftp\ntftpd\ntilda\ntoilet\ntokei  # lines of code counter\ntomboy\ntshark\nvirt-manager\nvlc\nvnc\nwin32codecs\nwine\nwireshark\nwmaker\nwmaker-data\nwmctrl\nx86info\nxclip\nxdotool\nxkbset\nxmlstartlet\nxosview\nxprobe\nxsel\nxterm\nzenity\n"
  },
  {
    "path": "setup/rpm-packages-optional.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                       Optional RPM Package Requirements\n# ============================================================================ #\n\n# no longer available on RHEL8 :-(\nShellCheck\n\n# only available on Fedora for now\ndirenv\n\n# RHEL7 only - kubernetes meta packages pulls in kubernetes-client, kubernetes-master, kubernetes-node, docker, python etc..\nkubernetes-client\n\n# not available on RHEL6\nperl-App-cpanminus\nperl-CPAN  # needed in RHEL6 to build cpanm to build rest\n\npython\npython-pip\npython2\npython2-pip\npython3\npython3-pip\n\n# required in Fedora, but not available in RHEL6\nhostname\n\n# not available in RHEL 6 repos\nparallel\n\n# not available as of Rocky Linux 9 or any version of CentOS\n# available from Fedora 31 onwards\npre-commit\n\nprocps     # RHEL <= 7 (RHEL 7 maps procps to procps-ng)\nprocps-ng  # RHEL 8\n\n# to give readline wrapping support to Oracle SQL*Plus\nrlwrap\n\n# Not available in any RHEL based system as of Fedora 40\n#sqlfluff\n"
  },
  {
    "path": "setup/rpm-packages.txt",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2019-03-16 20:35:14 +0000 (Sat, 16 Mar 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying LICENSE file\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                          RPM Package Requirements\n# ============================================================================ #\n\nbash\nbc\n#binutils  # contains old C tools like 'strings'\nbind-utils # host / nslookup / dig commands\ncurl\n#direnv\ngcc\ngit\ngolang\nfindutils  # find and xargs\niputils\njq\nlibxml2  # needed for xmllint - usually already installed\nmake\nopenssh-clients\nparallel\nperl\n#perl-App-cpanminus  # not available on RHEL6, moved to rpm-packages-optional.txt\n#ShellCheck  # not available in RHEL8, moved to rpm-packages-optional.txt\nruby-devel  # to build Travis CI gem\nwget\nwhich\nzip\nunzip\n"
  },
  {
    "path": "setup/setup_codefresh.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-04 21:22:03 +0100 (Sat, 04 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nif ! type -P codefresh &>/dev/null; then\n    if [ \"$(uname -s)\" = Darwin ]; then\n        echo \"Installing codefresh via HomeBrew\"\n        brew tap codefresh-io/cli\n        brew install codefresh\n    else\n        echo \"Installing codefresh via npm\"\n        npm install codefresh\n    fi\nfi\n\nif [ -z \"${CODEFRESH_KEY:-}\" ]; then\n    echo \"\\$CODEFRESH_KEY is not defined\"\n    exit 1\nfi\n\n# generate API key - https://g.codefresh.io/user/settings\necho \"creating codefresh auth context\"\ncodefresh auth create-context --api-key \"$CODEFRESH_KEY\"\n"
  },
  {
    "path": "setup/shell_link.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-18 23:21:50 +0100 (Wed, 18 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks if .bashrc and .bash_profile are sourced, otherwise injects source lines in to the $HOME directory equivalents\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nbash_tools=\"$srcdir/..\"\n\nconf_files=()\nwhile IFS= read -r line; do\n    conf_files+=(\"$line\")\ndone < <(\n    \"$srcdir/../bin/decomment.sh\" \"$bash_tools/setup/files.txt\"\n)\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n#[ -n \"${HOME:-}\" ] || HOME=~\nHOME=\"${HOME:-$(cd && pwd)}\"\n\nsetup_file(){\n    local filename=\"$1\"\n    if grep -Eq \"^[[:space:]]*(source|\\\\.)[[:space:]]+$bash_tools/$filename\" \"$HOME/$filename\" 2>/dev/null; then\n        echo \"$filename already sourced in $HOME/$filename\"\n    else\n        echo \"injecting into ~/$filename: source $bash_tools/$filename\"\n        echo \"source $bash_tools/$filename\" >> \"$HOME/$filename\"\n    fi\n}\n\nsetup_file .bashrc\nsetup_file .bash_profile\nsetup_file .bash_logout\n\nsetup_file .zshrc\nsetup_file .zprofile\nsetup_file .zshenv\nsetup_file .zlogin\nsetup_file .zlogout\n\necho\necho \"Symlinking dot files to \\$HOME directory: $HOME\"\necho\n\nopts=\"\"\nif [ -n \"${FORCE:-}\" ]; then\n    opts=\"-f\"\nfi\n\nfix_link(){\n    local path=\"$1\"\n    local target=\"$2\"\n\n    if [ -L \"$path\" ]; then\n        local current\n        current=\"$(readlink \"$path\")\"\n\n        if [ ! -e \"$path\" ] || [ \"$current\" != \"$target\" ]; then\n            echo \"WARNING: Removing stale symlink: $path\"\n            rm -f -- \"$path\"\n        fi\n    fi\n}\n\n#find_file(){\n#    local filename=\"$1\"\n#    if [ -e \"$filename\" ]; then\n#        echo \"$filename\"\n#    else\n#        root_filename=\"$bash_tools/$filename\"\n#        if [ -e \"$root_filename\" ]; then\n#            echo \"$root_filename\"\n#        else\n#            echo \"ERROR: cannot find source for $filename\" >&2\n#            exit 1\n#        fi\n#    fi\n#}\n\nif uname -s | grep -q Darwin; then\n    readlink(){\n        command greadlink \"$@\"\n    }\n    export -f readlink\nfi\n\nsymlink(){\n    local path=\"$1\"\n    if [[ \"$path\" =~ / ]]; then\n        filename=\"${path##*/}\"\n        srcdir=\"${path%/*}\"\n        destdir=\"${srcdir#configs}\"\n        destdir=\"${destdir##/}\"\n        destdir=\"${destdir%%/}\"\n        sourcepath=\"$bash_tools${srcdir+/$srcdir}/$filename\"  # if dirname, insert /dirname in middle\n        destpath=\"$HOME${destdir+/\"$destdir\"/}\"               # if dirname, append /dirname to dest\n        # remove double slashes to clean the path\n        sourcepath=\"${sourcepath/\\/\\//\\/}\"\n        destpath=\"${destpath/\\/\\//\\/}\"\n        destpath=\"${destpath%%/}\"\n        mkdir -pv \"$destpath\"\n        sourcepath=\"$(readlink -f \"$sourcepath\")\"\n        fix_link \"$destpath/$filename\" \"$sourcepath\"\n        # want opt expansion\n        # shellcheck disable=SC2086\n        ln -sv $opts -- \"$sourcepath\" \"$destpath\" || :\n    else\n        fix_link \"$HOME/$filename\" \"$bash_tools/$filename\"\n        # want opt expansion\n        # shellcheck disable=SC2086\n        ln -sv $opts -- \"$bash_tools/$filename\" \"$HOME/\" || :\n        # if we link .vimrc then run the vundle install and get plugins to prevent vim errors every startup\n        if [ \"$filename\" = .vimrc ]; then\n            \"$srcdir/../install/install_vundle.sh\" || :\n        fi\n    fi\n}\n\n for filename in \"${conf_files[@]}\"; do\n     if [[ \"$filename\" =~ \\* ]]; then\n         for expanded_filename in \"$bash_tools\"/$filename; do\n            if [[ \"$expanded_filename\" =~ \\* ]]; then\n                echo \"ERROR: failed to expand glob: $bash_tools/$filename\"\n                exit 1\n            fi\n            # shellcheck disable=SC2295\n            expanded_filename=\"${expanded_filename#$bash_tools/}\"\n            symlink \"$expanded_filename\"\n         done\n     else\n         symlink \"$filename\"\n     fi\n done\n\nfix_link \"$HOME/.gitignore_global\" \"$bash_tools/.gitignore\"\n# want opt expansion\n# shellcheck disable=SC2086\nln -sv $opts -- \"$bash_tools/.gitignore\" \"$HOME/.gitignore_global\" || :\n\n# drop my personal Git username and email local file in this repo into home dir\nif [[ \"${USER:-}\" =~ ^hari$|harisekhon|hsekhon ]]; then\n    fix_link \"$HOME/.gitconfig.local\" \"$bash_tools/.gitconfig.local\"\n    ln -sv -- \"$bash_tools/.gitconfig.local\" \"$HOME/\" || :\nfi\n"
  },
  {
    "path": "setup/shell_unlink.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2019-09-18 23:21:50 +0100 (Wed, 18 Sep 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Checks if .bashrc and .bash_profile are sourced, otherwise injects source lines in to the $HOME directory equivalents\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nbash_tools=\"$srcdir/..\"\n\nconf_files=()\nwhile IFS= read -r line; do\n    conf_files+=(\"$line\")\ndone < <(\n    \"$srcdir/../bin/decomment.sh\" \"$bash_tools/setup/files.txt\"\n)\n\n# unreliable that HOME is set, ensure shell evaluates to the right thing before we use it\n[ -n \"${HOME:-}\" ] || HOME=~\n\necho \"removing symlinks to dot files in \\$HOME directory: $HOME\"\necho\n\nfor filename in $conf_files .gitignore_global; do\n    filename=\"${filename#configs}\"\n    filename=\"${filename#/}\"\n    if [ -L ~/\"$filename\" ]; then\n        rm -fv -- ~/\"$filename\" || :\n    fi\ndone\n\necho\n\nremove_sourcing(){\n    local filename=\"$1\"\n    if ! grep -Eq \"^[[:space:]]*(source|\\\\.)[[:space:]]+$bash_tools/$filename\" ~/\"$filename\" 2>/dev/null; then\n        echo \"$filename not currently sourced in ~/$filename\"\n    else\n        echo \"in-place editing ~/$filename to remove sourcing of $bash_tools/$filename\"\n        local bash_tools_escaped=\"${bash_tools//\\//\\\\/}\"\n        local filename_escaped=\"${filename//\\//\\\\/}\"\n        perl -ni\".bak-$(date '+%F_%H%M')\" -e \"print unless /(source|\\\\.)[[:space:]]+$bash_tools_escaped\\\\/$filename_escaped/\" ~/\"$filename\"\n    fi\n}\n\nremove_sourcing .bashrc\nremove_sourcing .bash_profile\n"
  },
  {
    "path": "setup/squirrelsql-install-options.xml",
    "content": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<AutomatedInstallation langpack=\"eng\">\n<com.izforge.izpack.panels.hello.HelloPanel id=\"HelloPanel_0\"/>\n<com.izforge.izpack.panels.htmlinfo.HTMLInfoPanel id=\"HTMLInfoPanel_1\"/>\n<com.izforge.izpack.panels.target.TargetPanel id=\"TargetPanel_2\">\n<installpath>/Applications/SQuirreLSQL.app</installpath>\n</com.izforge.izpack.panels.target.TargetPanel>\n<com.izforge.izpack.panels.packs.PacksPanel id=\"PacksPanel_3\">\n<pack index=\"0\" name=\"Base\" selected=\"true\"/>\n<pack index=\"1\" name=\"Standard\" selected=\"true\"/>\n<pack index=\"2\" name=\"Optional Plugin - Mac OS Plugin (older Mac versions only)\" selected=\"true\"/>\n<pack index=\"3\" name=\"Optional Plugin - Greenplum\" selected=\"true\"/>\n<pack index=\"4\" name=\"Optional Plugin - WIKI table configurations\" selected=\"true\"/>\n<pack index=\"5\" name=\"Optional Plugin - Swing Violation Dedector\" selected=\"true\"/>\n<pack index=\"6\" name=\"Optional Plugin - Multi Source\" selected=\"true\"/>\n<pack index=\"7\" name=\"Optional Plugin - Vertica\" selected=\"true\"/>\n<pack index=\"8\" name=\"Optional Plugin - DB2\" selected=\"true\"/>\n<pack index=\"9\" name=\"Optional Plugin - Derby\" selected=\"true\"/>\n<pack index=\"10\" name=\"Optional Plugin - Firebird \" selected=\"true\"/>\n<pack index=\"11\" name=\"Optional Plugin - Hibernate\" selected=\"true\"/>\n<pack index=\"12\" name=\"Optional Plugin - H2 \" selected=\"true\"/>\n<pack index=\"13\" name=\"Optional Plugin - Informix \" selected=\"true\"/>\n<pack index=\"14\" name=\"Optional Plugin - Microsoft SQL Server \" selected=\"true\"/>\n<pack index=\"15\" name=\"Optional Plugin - MySQL \" selected=\"true\"/>\n<pack index=\"16\" name=\"Optional Plugin - Netezza \" selected=\"true\"/>\n<pack index=\"17\" name=\"Optional Plugin - Oracle\" selected=\"true\"/>\n<pack index=\"18\" name=\"Optional Plugin - PostgreSQL \" selected=\"true\"/>\n<pack index=\"19\" name=\"Optional Plugin - Session Scripts\" selected=\"true\"/>\n<pack index=\"20\" name=\"Optional Plugin - Smart Tools \" selected=\"true\"/>\n<pack index=\"21\" name=\"Optional Plugin - SQL Parametrisation \" selected=\"true\"/>\n<pack index=\"22\" name=\"Optional Plugin - SQL Replace \" selected=\"true\"/>\n<pack index=\"23\" name=\"Optional Plugin - SQL Validator \" selected=\"true\"/>\n<pack index=\"24\" name=\"Optional Plugin - Sybase \" selected=\"true\"/>\n<pack index=\"25\" name=\"Optional Plugin - High resolution icon \" selected=\"true\"/>\n<pack index=\"26\" name=\"Optional Plugin - Internationalization \" selected=\"true\"/>\n<pack index=\"27\" name=\"Optional Plugin - Intersystems Cache Plugin \" selected=\"true\"/>\n<pack index=\"28\" name=\"Optional Translation - Brazilian Portuguese\" selected=\"true\"/>\n<pack index=\"29\" name=\"Optional Translation - Bulgarian\" selected=\"true\"/>\n<pack index=\"30\" name=\"Optional Translation - Czech\" selected=\"true\"/>\n<pack index=\"31\" name=\"Optional Translation - Simplified Chinese\" selected=\"true\"/>\n<pack index=\"32\" name=\"Optional Translation - French\" selected=\"true\"/>\n<pack index=\"33\" name=\"Optional Translation - German\" selected=\"true\"/>\n<pack index=\"34\" name=\"Optional Translation - Italian\" selected=\"true\"/>\n<pack index=\"35\" name=\"Optional Translation - Japanese\" selected=\"true\"/>\n<pack index=\"36\" name=\"Optional Translation - Korean\" selected=\"true\"/>\n<pack index=\"37\" name=\"Optional Translation - Polish\" selected=\"true\"/>\n<pack index=\"38\" name=\"Optional Translation - Russian\" selected=\"true\"/>\n<pack index=\"39\" name=\"Optional Translation - Spanish\" selected=\"true\"/>\n</com.izforge.izpack.panels.packs.PacksPanel>\n<com.izforge.izpack.panels.install.InstallPanel id=\"InstallPanel_4\"/>\n<com.izforge.izpack.panels.finish.FinishPanel id=\"FinishPanel_5\"/>\n</AutomatedInstallation>\n"
  },
  {
    "path": "setup/teamcity/teamcity-database.properties",
    "content": "#Mon Nov 30 22:18:13 GMT 2020\nconnectionUrl=jdbc\\:hsqldb\\:file\\:$TEAMCITY_SYSTEM_PATH/buildserver\n"
  },
  {
    "path": "setup/teamcity-mysql-setup.sql",
    "content": "--\n--  Author: Hari Sekhon\n--  Date: 2020-11-24 11:25:23 +0000 (Tue, 24 Nov 2020)\n--\n--  vim:ts=2:sts=2:sw=2:et:filetype=sql\n--\n--  https://github.com/HariSekhon/DevOps-Bash-tools\n--\n--  License: see accompanying Hari Sekhon LICENSE file\n--\n--  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n--\n--  https://www.linkedin.com/in/HariSekhon\n--\n\n-- https://www.jetbrains.com/help/teamcity/setting-up-an-external-database.html#MySQL\n\ncreate database teamcity collate utf8_bin;\ncreate user teamcity identified by 'teamcity';\ngrant all privileges on teamcity.* to teamcity;\ngrant process on *.* to teamcity;\n"
  },
  {
    "path": "setup/upgrade_gradle_wrapper.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-31 16:20:04 +0100 (Sun, 31 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\nGRADLE_VERSION=\"${GRADLE_VERSION:-6.4.1}\"\n\n./gradlew wrapper --gradle-version=\"$GRADLE_VERSION\" --distribution-type=bin\n"
  },
  {
    "path": "setup/which_python_installed.sh",
    "content": "#!/bin/sh\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-05-16 16:59:11 +0100 (Sat, 16 May 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -u\n[ -n \"${DEBUG:-}\" ] && set -x\n\necho \"Python / Pip versions installed:\"\necho\n\n# executing in sh where type is not available\n#type -P python\nfor x in python python2 python3 pip pip2 pip3; do\n    cmdpath=\"$(command -v \"$x\" 2>/dev/null)\"\n    if [ -n \"$cmdpath\" ]; then\n        printf \"%s\" \"$cmdpath => \"\n        \"$cmdpath\" -V\n    fi\ndone\necho\nfor x in python python2 python3 pip pip2 pip3; do\n    cmdpath=\"$(command -v \"$x\" 2>/dev/null)\"\n    if [ -n \"$cmdpath\" ]; then\n        ls -l \"$cmdpath\"\n    fi\ndone\n"
  },
  {
    "path": "shippable/shippable.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-23 23:20:54 +0000 (Sun, 23 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# http://docs.shippable.com/platform/workflow/config/\n\n# http://docs.shippable.com/ci/advancedOptions/environmentVariables/\n\nlanguage: none\n\nbranches:\n  only:\n    - master\n\nbuild:\n  ci:\n    # workaround to broken repos\n    # W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://downloads.apache.org/cassandra/debian 311x InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY E91335D77E3E87CB\n    # W: GPG error: http://dl.yarnpkg.com/debian stable Release: The following signatures were invalid: KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1580619281  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983  KEYEXPIRED 1507181400  KEYEXPIRED 1546376218  KEYEXPIRED 1546372003  KEYEXPIRED 1580619281  KEYEXPIRED 1580607983\n    # E: The repository 'http://dl.yarnpkg.com/debian stable Release' is no longer signed.\n    # bash-tools/Makefile.in:272: recipe for target 'apt-packages' failed\n    - rm -fv -- /etc/apt/sources.list.d/cassandra.sources.list*\n    - rm -fv -- /etc/apt/sources.list.d/yarn.list*\n    # Basho repo is giving a '402 payment required' error\n    # https://github.com/Shippable/support/issues/5172\n    - rm -fv -- /etc/apt/sources.list.d/basho_riak.list\n    #- shippable_retry make\n    - setup/ci_bootstrap.sh\n    - make init\n    - make ci\n    - make test\n\nintegrations:\n  notifications:\n    - integrationName: email\n      type: email\n      on_success: never\n      on_failure: never\n      on_pull_request: never\n"
  },
  {
    "path": "shippable/shippable_account_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-24 15:27:07 +0000 (Tue, 24 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# pending support ticket around permissions issue\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Returns Shippable accounts for the given \\$SHIPPABLE_TOKEN\n\nCaveat: this API endpoint only works for paid accounts :-(\n\nhttps://github.com/Shippable/support/issues/5068\n\"\n\nhelp_usage \"$@\"\n\n#curl -sSH 'Accept: application/json' 'https://api.shippable.com/projects?sortBy=createdAt&sortOrder=-1&ownerAccountIds=harisekhon'\n\"$srcdir/shippable_api.sh\" '/accounts'\n"
  },
  {
    "path": "shippable/shippable_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-24 15:36:53 +0000 (Tue, 24 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Queries the Shippable API, auto-populating the base and API tokens from the environment\"\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nif [ -z \"${SHIPPABLE_TOKEN:-}\" ]; then\n    usage \"SHIPPABLE_TOKEN environment variable is not set (generate this from your Web UI Dashboard -> profile -> API AUTH TOKENS\"\nfi\n\nif [ $# -lt 1 ]; then\n    usage \"no /path given to query in the API\"\nfi\n\nhelp_usage \"$@\"\n\nurl_path=\"${1##/}\"\nshift || :\n\nexport TOKEN=\"$SHIPPABLE_TOKEN\"\n\n# non-standard auth header\nexport CURL_AUTH_HEADER=\"Authorization: apiToken\"\n\n\"$srcdir/../bin/curl_auth.sh\" -sS --fail \"https://api.shippable.com/$url_path\" \"$@\"\n"
  },
  {
    "path": "shippable/shippable_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-24 15:36:53 +0000 (Tue, 24 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Returns recent Shippable build results\n\nCaveat: this API endpoint only works for paid accounts :-(\n\nhttps://github.com/Shippable/support/issues/5068\n\"\n\nif [ -z \"${SHIPPABLE_ACCOUNT_ID:-}\" ]; then\n    usage \"SHIPPABLE_ACCOUNT_ID environment variable is not set\n\nGet this value from your Web UI Dashboard - it's in your dashboard url:\n\nhttps://app.shippable.com/accounts/<THIS_BIT_IS_YOUR_ACCOUNT_ID>/dashboard\n\"\nfi\n\nhelp_usage \"$@\"\n\n\"$srcdir/shippable_api.sh\" \"/accounts/$SHIPPABLE_ACCOUNT_ID/runStatus\" \"$@\"\n#jq -r \"$jq_query\"\n"
  },
  {
    "path": "shippable/shippable_project_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-04-15 17:29:40 +0100 (Wed, 15 Apr 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_id> [<curl_options>]\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Returns recent Shippable build results for a given project\n\nSpecify the project ID as the first argument or have \\$SHIPPABLE_PROJECT_ID environment variable defined\n\nThis works for free accounts whereas the adjacent shippable_builds.sh requires a paid account\n\nThis is the same API endpoints that shields.io uses for it's badges\n\nThe statusCode field returned has integers that correspond to statuses\n\nhttp://docs.shippable.com/ci/build_status/\n\"\n\nSHIPPABLE_PROJECT_ID=\"${1:-${SHIPPABLE_PROJECT_ID:-}}\"\n\ncheck_env_defined SHIPPABLE_PROJECT_ID\n\nhelp_usage \"$@\"\n\n# this will enforce a $SHIPPABLE_TOKEN which isn't necessary as this is a public endpoint\n#\"$srcdir/shippable_api.sh\" \"/projects/$SHIPPABLE_PROJECT_ID/branchRunStatus\" \"$@\"\ncurl -sSH 'Accept: application/json' \"https://api.shippable.com/projects/$SHIPPABLE_PROJECT_ID/branchRunStatus\" \"$@\"\n#jq -r \"$jq_query\"\n"
  },
  {
    "path": "shippable/shippable_projects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-24 15:36:53 +0000 (Tue, 24 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_description=\"Returns a list of Shippable projects ids and names, or a single project name if the project id is given as an arg\"\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nif [ -z \"${SHIPPABLE_ACCOUNT_ID:-}\" ]; then\n    usage \"SHIPPABLE_ACCOUNT_ID environment variable is not set (get this value from your Web UI Dashboard)\"\nfi\n\nhelp_usage \"$@\"\n\nproject_id=\"\"\nif [ $# -gt 0 ]; then\n    project_id=\"$1\"\n    shift || :\nfi\n\nif [ -n \"$project_id\" ] &&\n   ! [[ \"$project_id\" =~ ^[[:alnum:]]+$ ]]; then\n    usage \"invalid project id '$project_id' argument specified, must be alphanumeric\"\nfi\n\njq_query_common='[.id, .sourceRepoOwner.login, .name] | @tsv'\n# API returns list without project id or hashmap with project id\njq_query=\".[] | $jq_query_common\"\nif [ -n \"$project_id\" ]; then\n    jq_query=\"$jq_query_common\"\nfi\n\n\"$srcdir/shippable_api.sh\" \"/projects/$project_id?sortBy=createdAt&sortOrder=-1&ownerAccountIds=$SHIPPABLE_ACCOUNT_ID\" \"$@\" |\njq -r \"$jq_query\"\n"
  },
  {
    "path": "sonar-project.properties",
    "content": "#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2016-07-19 18:31:17 +0100 (Tue, 19 Jul 2016)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               S o n a r Q u b e\n# ============================================================================ #\n\nsonar.host.url=https://sonarcloud.io\n\n# Required metadata\nsonar.organization=harisekhon\nsonar.projectName=DevOps-Bash-tools\nsonar.projectKey=HariSekhon_DevOps-Bash-tools\nsonar.projectVersion=1.0\n\nsonar.projectDescription=DevOps-Bash-tools\n\nsonar.links.homepage=https://github.com/HariSekhon/DevOps-Bash-tools\nsonar.links.scm=https://github.com/HariSekhon/DevOps-Bash-tools\nsonar.links.issue=https://github.com/HariSekhon/DevOps-Bash-tools/issues\nsonar.links.ci=https://github.com/HariSekhon/DevOps-Bash-tools/actions\n\n# directories to scan (defaults to sonar-project.properties dir otherwise)\nsonar.sources=.\n\n#sonar.language=py\n\nsonar.sourceEncoding=UTF-8\n\n#sonar.exclusions=**/tests/**\nsonar.exclusions=**/zookeeper-*/**/*\n"
  },
  {
    "path": "spotify/spotify_add_artist_to_backlog_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-07-08 17:11:29 +0200 (Tue, 08 Jul 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nplaylist=\"Discover Backlog\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches for the tracks of a given artist and adds them to the \\\"$playlist\\\" playlist\n\nBecause Spotify's UI is horrible to try to get all the tracks from the discographies and add them to a playlist manually\n\nBy defaults limits to 1000 tracks, which should cover all of the artist's work\n\nIf you need to change this limit, set:\n\n    export SPOTIFY_SEARCH_LIMIT=5000\n\nExpects the \\\"$playlist\\\" playlist to already exist\n\nUses adjacent scripts:\n\n    spotify_search_uri.sh\n\n    spotify_add_to_playlist.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<artist>\"\n\nhelp_usage \"$@\"\n\nnum_args 1 \"$@\"\n\nartist=\"$1\"\n\nexport SPOTIFY_SEARCH_LIMIT=\"${SPOTIFY_SEARCH_LIMIT:-1000}\"\n\n\"$srcdir/spotify_search_uri.sh\" artist:\"$artist\" |\ntee >(\"$srcdir/spotify_uri_to_name.sh\") |\n\"$srcdir/spotify_add_to_playlist.sh\" \"$playlist\"\n"
  },
  {
    "path": "spotify/spotify_add_to_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 18:02:26 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds Spotify URIs to a given playlist\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\nCan take file(s) with URIs as arguments or read from standard input for chaining with other tools\n\nUseful for chaining with other 'spotify_*_uri.sh' tools (eg. spotify_playlist_tracks_uri.sh, spotify_search_uri.sh) or loading from saved spotify format playlists (eg. HariSekhon/Spotify-Playlists github repo)\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> [<file1> <file2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\nplaylist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\")\"\n\n# playlist ID obtained from 'spotify_playlists.sh'\nurl_path=\"/v1/playlists/$playlist_id/tracks\"\n\ncount=0\n\nadd_to_playlist(){\n    if [ $# -lt 1 ]; then\n        echo \"Error: no IDs passed to add_to_playlist()\" >&2\n        exit 1\n    fi\n    local id_array=\"\"\n    for id in \"$@\"; do\n        # requires explicit track URI type since could also be episodes added to playlist\n        id_array+=\"\\\"spotify:track:$id\\\", \"\n    done\n    id_array=\"${id_array%, }\"\n    timestamp \"adding ${#@} tracks to playlist '$playlist_name'\"\n    \"$srcdir/spotify_api.sh\" \"$url_path\" -X POST -d '{\"uris\": '\"[$id_array]}\" >/dev/null  # ignore the { \"spotify_snapshot\": ... } json output\n    ((count+=${#@}))\n}\n\nadd_file_URIs(){\n    declare -a ids\n    ids=()\n    while read -r track_uri; do\n        if is_blank \"$track_uri\"; then\n            continue\n        fi\n        if is_local_uri \"$track_uri\"; then\n            continue\n        fi\n        id=\"$(validate_spotify_uri \"$track_uri\")\"\n\n        ids+=(\"$id\")\n\n        if [ \"${#ids[@]}\" -ge 50 ]; then\n            add_to_playlist \"${ids[@]}\"\n            sleep 1\n            ids=()\n        fi\n    done < \"$filename\"\n\n    if [ \"${#ids[@]}\" -eq 0 ]; then\n        return\n    fi\n    add_to_playlist \"${ids[@]}\"\n}\n\nfor filename in \"${@:-/dev/stdin}\"; do\n    add_file_URIs \"$filename\"\ndone\n\ntimestamp \"$count tracks added to playlist '$playlist_name'\"\n"
  },
  {
    "path": "spotify/spotify_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: /v1/users/harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-23 17:17:18 +0100 (Tue, 23 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the given Spotify API endpoint\n\nAPI endpoints you can query with this code:\n\nhttps://developer.spotify.com/documentation/web-api/reference/\n\nBreaking Changes Feb 2026 - several API endpoints have been REMOVED :-(:\n\n    https://developer.spotify.com/documentation/web-api/references/changes/february-2026\n\n    https://developer.spotify.com/documentation/web-api/tutorials/february-2026-migration-guide\n\nFor private user data endpoints you must export SPOTIFY_PRIVATE=1\n\n$usage_auth_help\n\n\nExamples:\n\nspotify_api.sh /v1/users/harisekhon\n\nSPOTIFY_PRIVATE=1 spotify_api.sh /v1/me/tracks\n\nUsed by adjacent spotify_*.sh scripts for more serious usage\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/url/path [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl_path=\"$1\"\n\nshift || :\n\nif [[ \"$url_path\" =~ /v1/me/ ]]; then\n    export SPOTIFY_PRIVATE=1\nfi\n\nspotify_token\n\nurl_base=\"https://api.spotify.com\"\n# shellcheck disable=SC2295\nurl_path=\"${url_path##$url_base}\"\nurl_path=\"${url_path##/}\"\n\nexport TOKEN=\"$SPOTIFY_ACCESS_TOKEN\"\n\nif not_blank \"${DEBUG:-}\"; then\n    # No point adding backoff-retry logic during HTTP 429 Too Many Requests failures\n    # as the Spotify backoff period specified in the Retry-After header is 14 hours,\n    # although it seem to not apply to all endpoints consistently, some other lookups still work\n    #\n    # Here we just want to see how long is left in the retry-after header by enabling DEBUG mode\n    #\n    if ! \"$srcdir/../bin/curl_auth.sh\" -sSL --fail \"$url_base/$url_path\" \"$@\"; then\n        \"$srcdir/../bin/curl_auth.sh\" -i -s \"$url_base/$url_path\" \"$@\" >&2\n        output=\"$(\"$srcdir/../bin/curl_auth.sh\" -i \"$url_base/$url_path\" \"$@\")\"\n        if [[ \"$output\" =~ HTTP/2[[:space:]]+429|retry-after: ]]; then\n            retry_after=\"$(awk '/^retry-after: /{print $2}' <<< \"$output\" | tr -d '\\r')\"\n            if ! is_int \"$retry_after\"; then\n                die \"ERROR: expected an integer for the retry-after header, got: $retry_after\"\n            fi\n            if is_mac; then\n                date(){\n                    command gdate \"$@\"\n                }\n            fi\n            echo \"Retry-After header says you can retry after $retry_after seconds\" >&2\n            date -u -d \"@$retry_after\" '+Retry after: %H hours %M minutes %S seconds' >&2\n            date -d \"+$retry_after seconds\" '+Retry at: %Y-%m-%d %H:%M:%S' >&2\n        fi\n        exit 1\n    fi\nelse\n    # the Spotify API is very unreliable and often gets 502 errors\n    # seen 20 x HTTP 500 errors from the API in a row :-/\n    # so give it many tries before giving up\n    MAX_RETRIES=\"30\" retry 300 \"$srcdir/../bin/curl_auth.sh\" -sSL --fail \"$url_base/$url_path\" \"$@\"\nfi\n"
  },
  {
    "path": "spotify/spotify_api_token.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-23 17:59:52 +0100 (Tue, 23 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nReturns a Spotify access token from the Spotify API, using app credentials and a Spotify cookie\n\nThis token is needed to access the rest of the Spotify API endpoints\n\nRequires \\$SPOTIFY_ID and \\$SPOTIFY_SECRET to be defined in the environment\n\nDue to quirks of the Spotify API, by default returns a non-interactive access token that cannot access private user data\n\nTo get a token to access the private user data API endpoints:\n\nexport SPOTIFY_PRIVATE=1\n\nSince this requires a Spotify login cookie for the authentication workflow to request a token, it will either:\n\n1. Check pycookiecheat is available in the \\$PATH, and if so:\n   - use it extract your \\$BROWSER's Spotify cookie\n   - use the cookie to query the Spotify API authentication workflow endpoint\n   OR if pycookiecheat is not available...\n2. Launch an interactive browser pop-up to use your browser's Spotify cookie\n   - on macOS it'll auto close the new tab and return you to the previously active window using Applescript automation\n\nThis script will capture the authentication workflow output using a redirect to a local loopback callback and then use\nthat to further request an actual Spotify API token that can be used by other programs and print that to stdout\n\nMany scripts utilize this code and will automatically generate the authentication tokens for you if you have\n\\$SPOTIFY_ID and \\$SPOTIFY_SECRET environment variables set so you usually don't need to call this yourself\n\n\nFor private tokens which require authorization pop-ups, if you want to avoid these on every run of these spotify\nscripts, you can preload a private authorized token in to your shell for an hour like so:\n\nexport SPOTIFY_ACCESS_TOKEN=\\\"\\$(SPOTIFY_PRIVATE=1 '$srcdir/../spotify/spotify_api_token.sh')\n\n\nGenerate an App client ID and secret for SPOTIFY_ID and SPOTIFY_SECRET environment variables here:\n\nhttps://developer.spotify.com/dashboard/applications\n\nMake sure to add a callback URL of exactly 'http://127.0.0.1:12345/callback'\nwithout the quotes to be able to generate private tokens\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#if [ -n \"${SPOTIFY_ACCESS_TOKEN:-}\" ] &&\n#   [ -n \"${SPOTIFY_ACCESS_TOKEN//[[:space:]]}\" ]; then\n#    echo \"$SPOTIFY_ACCESS_TOKEN\"\n#    exit 0\n#fi\n\ncheck_env_defined \"SPOTIFY_ID\"\ncheck_env_defined \"SPOTIFY_SECRET\"\n\n# encode spaces as %20 or +\nscope=\"${SPOTIFY_TOKEN_SCOPE:-\napp-remote-control\nplaylist-modify-private\nplaylist-modify-public\nplaylist-read-collaborative\nplaylist-read-private\nstreaming\nuser-follow-modify\nuser-follow-read\nuser-library-modify\nuser-library-read\nuser-modify-playback-state\nuser-read-currently-playing\nuser-read-email\nuser-read-playback-position\nuser-read-playback-state\nuser-read-private\nuser-read-recently-played\nuser-top-read\n}\n\"\n# perl -pe doesn't really work here, hard to remove leading/trailing ++ without slurp to real var\n#scope=\"$(perl -e '$str = do { local $/; <STDIN> }; $str =~ s/\\s+/\\+/g; $str =~ s/^\\++//; $str =~ s/\\++$//; print $str' <<< \"$scope\")\"\n# simpler\nscope=\"$(tr '\\n' '+' <<< \"$scope\" | sed 's/^+//; s/+*$//')\"\n\n# ============================================================================ #\n# Client Credentials method - the most suitable to scripting but doesn't grant access to user data :-/\n#\n#   https://developer.spotify.com/documentation/general/guides/authorization-guide/#client-credentials-flow\n#\n\n# If we only need a public access token we do this simpler workflow call and exit after printing the token\n\nif is_blank \"${SPOTIFY_PRIVATE:-}\"; then\n    output=\"$(\n        NO_TOKEN_AUTH=1 \\\n        USERNAME=\"$SPOTIFY_ID\" \\\n        PASSWORD=\"$SPOTIFY_SECRET\" \\\n        \"$srcdir/../bin/curl_auth.sh\" \\\n        -sSL \\\n        -X 'POST' \\\n        -d 'grant_type=client_credentials' \\\n        -d \"scope=$scope\" \\\n        https://accounts.spotify.com/api/token \\\n        \"$@\"\n    )\"\n    #die_if_error_field \"$output\"\n    jq -r '.access_token' <<< \"$output\"\n    exit \"$?\"\nfi\n\n# ============================================================================ #\n# Callback for Private authz catch\n\ncallback_port=12345\n\nredirect_uri=\"http://127.0.0.1:$callback_port/callback\"\nredirect_uri_encoded=\"$(printf '%s' \"$redirect_uri\" | jq -s -R -r @uri)\"\n\n# Spotify not respecting the localhost cert anyway since it doesn't get that far on the redirect with the following error:\n#\n#   INVALID_CLIENT: Insecure redirect URI\n#\n# Must use loopback with http anyway\n\n#callback_key=\"$(mktemp /tmp/spotify_callback.key.XXXXXXXXXX)\"\n#callback_crt=\"$(mktemp /tmp/spotify_callback.crt.XXXXXXXXXX)\"\n\n##callback_key=\"$srcdir/spotify_callback.key\"\n#callback_key=\"$srcdir/localhost-key.pem\"\n##callback_crt=\"$srcdir/spotify_callback.crt\"\n#callback_crt=\"$srcdir/localhost.pem\"\n#callback_p12=\"$srcdir/localhost.p12\"\n##callback_cnf=\"$srcdir/spotify_callback_openssl.cnf\"\n#callback_p12_password=\"spotify\"\n#\n## generate self-signed certificate if it doesn't already exist\n#if ! [ -f \"$callback_key\" ] ||\n#   ! [ -f \"$callback_crt\" ]; then\n#    timestamp \"Creating a spotify callback OpenSSL certificate\"\n##    openssl=openssl\n#    if is_mac; then\n#        MAC=1\n#        openssl=\"$(brew --prefix openssl)/bin/openssl\"\n#    fi\n##    openssl version -a\n##    #\"$openssl\" req -newkey rsa:2048 -nodes \\\n##    #               -keyout \"$callback_key\" \\\n##    #               -x509 -days 3650 \\\n##    #               -out \"$callback_crt\" \\\n##    #               -subj \"/CN=localhost\" 2>/dev/null\n##    \"$openssl\" req -x509 -nodes -days 3650 \\\n##                   -newkey rsa:2048 \\\n##                   -keyout \"$callback_key\" \\\n##                   -out \"$callback_crt\" \\\n##                   -config \"$callback_cnf\" \\\n##                   2>/dev/null\n#    mkcert -install\n#    mkcert localhost  # this overwrites existing localhost.pem and localhost-key.pem\n#    timestamp \"Creating a p12 certificate bundle to import into keychain\"\n#    \"$openssl\" pkcs12 -export \\\n#                      -name spotify_callback \\\n#                      ${MAC:+-legacy -keypbe PBE-SHA1-3DES -certpbe PBE-SHA1-3DES -macalg sha1} \\\n#                      -inkey \"$callback_key\" \\\n#                      -in \"$callback_crt\" \\\n#                      -out \"$callback_p12\" \\\n#                      -passout pass:\"$callback_p12_password\"\n#                      # do not use these switches - tried with libressl but macOS keychain refuses to import\n#                      #-nomac \\\n#                      #-keypbe NONE \\\n#                      #-certpbe NONE\n#    #timestamp \"Checking the p12 isn't encrypted\" # actually needed to be encrypted to be accepted into the keychain\n#    # Expected lines: \"MAC: sha256, ...\" and \"Key bag\" (NOT \"Shrouded Keybag\" and NOT \"Warning: MAC is absent!\")\n#    #\"$openssl\" pkcs12 -in \"$callback_p12\" -noout -info -passin pass:\"$callback_p12_password\"\n#    if is_mac; then\n#        # macOS won't trust the local root cert without the key, so must generate and import the p12\n#        timestamp \"Importing p12 into keychain: $callback_p12\"\n#        sudo security import \"$callback_p12\" \\\n#                             -k /Library/Keychains/System.keychain \\\n#                             -P \"$callback_p12_password\" \\\n#                             -A\n#        timestamp \"Importing cert into keychain: $callback_crt\"\n#        sudo security add-trusted-cert \\\n#                        -d \\\n#                        -r trustRoot \\\n#                        -k /Library/Keychains/System.keychain \\\n#                        \"$callback_crt\"\n#        timestamp \"Verifying cert is trusted: $callback_crt\"\n#        security verify-cert -c \"$callback_crt\"\n#    else\n#        warn \"you must import this certificate to be trusted by your browser: $callback_crt\"\n#    fi\n#fi\n\ncallback(){\n    {\n    log \"Waiting to catch callback\"\n    local timestamp\n    timestamp=\"$(date '+%F %T')\"\n    local netcat_switches\n    netcat_switches=\"-l localhost $callback_port\"\n    # GNU netcat has different switches :-/\n    # also errors out so we have to ignore its error code\n    if nc --version 2>&1 | grep -q GNU; then\n        netcat_switches=\"-l -p $callback_port --close\"\n    fi\n    # TODO: add a mutex wait lock here, UPDATE: can't remember why I wrote this now, there is a wait at the end for this\n    pkill -9 -f \"^nc $netcat_switches$\" || :\n    trap_cmd \"pkill -9 -f '^nc $netcat_switches$'\"\n    #trap_cmd \"pkill -9 -f '^openssl s_server .* -accept $callback_port'\"\n    #log \"Killing any existing openssl listener if there is already one running on port: $callback_port\"\n    #pkill -9 -f \"^openssl s_server .* -accept $callback_port\" || :\n    sleep 1\n    #local response\n    # need opt splitting\n    # shellcheck disable=SC2086\n    #response=\"$(openssl s_server \\\n    #                -quiet \\\n    #                -key \"$callback_key\" \\\n    #                -cert \"$callback_crt\" \\\n    #                -accept 12345 \\\n    #                -www 2>/dev/null <<EOF || :\n    response=\"$(nc $netcat_switches <<EOF || :\nHTTP/1.1 200 OK\n\n$timestamp  Spotify token accepted, now return to command line to use Spotify API tools\nEOF\n    )\"\n    log \"Callback Caught\"\n\n    local code\n    log \"Response: $response\"\n    log\n    code=\"$(grep -Eo \"GET.*code=([^?]+)\" <<< \"$response\" | sed 's/.*code=//; s/[&[:space:]].*$//' || :)\"\n    if is_blank \"$code\"; then\n        echo \"failed to parse code, authentication failure or authorization denied?\"\n        exit 1\n    fi\n    log \"Parsed code: $code\"\n    log\n    log \"Requesting API token using code\"\n\n    # or send client_id + client_secret fields in POST body - using curl_auth.sh now to avoid this appearing in process list / logs\n    #basic_auth_token=\"$(base64 <<< \"$SPOTIFY_ID:$SPOTIFY_SECRET\")\"\n\n    #curl -H \"Authorization: Basic $basic_auth_token\" -d grant_type=authorization_code -d code=\"$code\" -d redirect_uri=\"$redirect_uri\" https://accounts.spotify.com/api/token\n    # curl_auth.sh prevents auth token appearing in process list\n    local output\n    # debugging\n    #printf 'DEBUG raw redirect_uri=<%s>\\n' \"$redirect_uri\" >&2\n    output=\"$(\n        NO_TOKEN_AUTH=1 \\\n        USERNAME=\"$SPOTIFY_ID\" \\\n        PASSWORD=\"$SPOTIFY_SECRET\" \\\n        \"$srcdir/../bin/curl_auth.sh\" \\\n        https://accounts.spotify.com/api/token \\\n        -sSL \\\n        -d grant_type=authorization_code \\\n        -d redirect_uri=\"$redirect_uri\" \\\n        -d code=\"$code\" \\\n        #-d code_verifier=\"$code_verifier\"\n    )\"\n\n    # output everything that isn't the token to stderr as it's almost certainly user information or errors and we don't want that to be captured by client scripts\n    } >&2\n    echo \"$output\"\n}\n\n# ============================================================================ #\n# Implicit Grant Method\n#\n#   https://developer.spotify.com/documentation/general/guides/authorization-guide/#implicit-grant-flow\n#\n#output=\"$(curl -sSL -X GET \"https://accounts.spotify.com/authorize?client_id=$SPOTIFY_ID&redirect_uri=$redirect_uri&scope=$scope&response_type=token\")\"\n\n# ============================================================================ #\n# Authorization Code Flow\n#\n#   https://developer.spotify.com/documentation/general/guides/authorization-guide/#authorization-code-flow\n#\n#output=\"$(curl -sSL -X GET \"https://accounts.spotify.com/authorize?client_id=$SPOTIFY_ID&redirect_uri=$redirect_uri&scope=$scope&response_type=code\")\"\n\n# ============================================================================ #\n# Authorization Code Flow with Proof Key for Code Exchange (PKCE)\n#\n#   https://developer.spotify.com/documentation/general/guides/authorization-guide/#authorization-code-flow-with-proof-key-for-code-exchange-pkce\n\n#code_verifier=\"$(\n#    head -c 64 /dev/urandom |\n#    base64 |\n#    tr '+/' '-_' |\n#    tr -d '='\n#)\"\n#\n#code_challenge=$(\n#    printf '%s' \"$code_verifier\" |\n#    openssl dgst -sha256 -binary |\n#    base64 |\n#    tr '+/' '-_' |\n#    tr -d '=' |\n#    tr -d '\\n' |\n#    tr -d '[:space:]'\n#)\n#\n#if [ \"$(uname -s)\" = Darwin ]; then\n#    sha1sum(){\n#        command shasum \"$@\"\n#    }\n#fi\n\n# ============================================================================ #\n# using Authorization Code Flow\n\napplescript=\"$srcdir/../applescript\"\n\n# Authorization Code Flow with PKCE\nif not_blank \"${SPOTIFY_PRIVATE:-}\"; then\n    # clean up subprocesses to prevent netcat from being left behind as an orphan and blocking future runs\n    # shellcheck disable=SC2064\n    trap \"kill -- -$$\" EXIT\n    callback |\n    jq_debug_pipe_dump |\n    jq -r '.access_token' &\n    sleep 1\n    if ! pgrep -q -P $$; then\n        die \"Callback exited prematurely, port $callback_port may have been already bound, not launching authorization to prevent possible credential interception\"\n    fi\n    trap -- EXIT\n    {\n    # authorization code flow with PKCE\n    #url=\"https://accounts.spotify.com/authorize?client_id=$SPOTIFY_ID&redirect_uri=$redirect_uri_encoded&scope=$scope&response_type=code&code_challenge_method=S256&code_challenge=$code_challenge\"\n    # without PKCE\n    url=\"https://accounts.spotify.com/authorize?client_id=$SPOTIFY_ID&redirect_uri=$redirect_uri_encoded&scope=$scope&response_type=code\"\n    # implicit grant flow would use response_type=token, but this requires an SSL connection in the redirect URI\n    # and would complicate things with localhost SSL server certificate management\n    if type -P pycookiecheat &>/dev/null; then\n        # this calls the Spotify url using your browser cookies, and automatically redirects to the local callback handler\n        # to collect and request the token without having to fire up the browser pop-up\n        \"$srcdir/../bin/curl_with_cookies.sh\" \"$url\"\n    elif is_mac; then\n        log \"URL: $url\"\n        frontmost_process=\"$(\"$applescript/get_frontmost_process.scpt\")\"\n        \"$srcdir/../bin/urlopen.sh\" \"$url\"\n        # if using PKCE, need to add code to save and reuse refresh_token, otherwise it results in a fresh authorization page each time\n        # send Tab, Tab, Tab, Space to accept the new prompt page\n        #START_DELAY=1 SLEEP_SECS=1 \"$srcdir/../applescript/keystrokes.sh\" 1 48 48 48 49\n        # don't close the tab too fast or the token isn't passed to the local callback handler\n        wait\n        \"$applescript/browser_close_tab.scpt\"\n        \"$applescript/set_frontmost_process.scpt\" \"$frontmost_process\"\n    else\n        echo\n        echo \"Go to the following URL in your browser, authorize and then the token will be output on the command line:\"\n        echo\n        echo \"$url\"\n        echo\n        \"$srcdir/../bin/urlopen.sh\" \"$url\"\n        echo\n        wait\n    fi\n    } >&2\nfi\n"
  },
  {
    "path": "spotify/spotify_artist_tracks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Sia\"\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-16 19:09:57 +0000 (Tue, 16 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/#/operations/get-an-artists-albums\n#\n# https://developer.spotify.com/documentation/web-api/reference/#/operations/get-an-albums-tracks\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC2154\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOutputs track URIs for a given Spotify artist\n\nArtist argument can be an artist name or preferably an artist ID (get this from the app -> Share -> Copy link to artist)\n\nUseful to chain with the following scripts:\n\n    spotify_add_to_playlist.sh\n    spotify_delete_any_duplicates_in_playlist.sh\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<spotify_artist>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nartist=\"$1\"\n\nshift || :\n\nif is_blank \"$artist\"; then\n    usage \"artist not defined\"\nfi\n\nspotify_token\n\nif [ \"${#artist}\" = 22 ]; then\n    artist_id=\"$artist\"\nelse\n    timestamp \"resolving artist '$artist' to artist ID\"\n    artist_id=\"$(SPOTIFY_SEARCH_TYPE=artist SPOTIFY_SEARCH_LIMIT=50 \"$srcdir/spotify_search_json.sh\" \"$artist\" | jq -r \".artists.items[] | select(.name | ascii_downcase == \\\"$artist\\\") | .id\" | head -n1)\"\n    timestamp \"got artist ID '$artist_id'\"\necho >&2\nfi\n\n# $offset defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/artists/$artist_id/albums?limit=50&offset=$offset&include_groups=album,single\"  # API limit max is 50\n\ntimestamp \"getting list of albums for artist\"\nalbums=\"$(\n    while not_null \"$url_path\"; do\n        output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n        #die_if_error_field \"$output\"\n        url_path=\"$(get_next \"$output\")\"\n        jq -r '.items[] | [.id, .name] | @tsv' <<< \"$output\"\n    done\n)\"\necho >&2\n\noffset=0\nwhile read -r album_id album_name; do\n    timestamp \"getting list of tracks for album '$album_name'\"\n    # $offset defined in lib/spotify.sh\n    # shellcheck disable=SC2154\n    url_path=\"/v1/albums/$album_id/tracks?limit=50&offset=$offset\"  # API limit max is 50\n\n    while not_null \"$url_path\"; do\n        output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n        #die_if_error_field \"$output\"\n        url_path=\"$(get_next \"$output\")\"\n        #jq -r '.items[] | [.id, .name] | @tsv' <<< \"$output\"\n        jq -r '.items[].id' <<< \"$output\"\n    done\ndone <<< \"$albums\"\n"
  },
  {
    "path": "spotify/spotify_artists_followed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 21:06:46 +0100 (Fri, 23 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/follow/get-followed/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Followed Artists via the Spotify API\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/following?type=artist&offset=$offset&limit=$limit\"\n\noutput(){\n    jq -r '.artists.items[] | select(.name != \"\") | .name' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(jq -r '.artists.next' <<< \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_artists_followed_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 21:06:46 +0100 (Fri, 23 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/follow/get-followed/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Followed Artists URIs via the Spotify API\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/following?type=artist&offset=$offset&limit=$limit\"\n\noutput(){\n    jq -r '.artists.items[].uri' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(jq -r '.artists.next' <<< \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_artists_followed_uri_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-03 03:06:50 -0300 (Tue, 03 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/follow/get-followed/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Followed Artists URIs and Names via the Spotify API\n\nOutput:\n\n<URI> <Name>\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/following?type=artist&offset=$offset&limit=$limit\"\n\noutput(){\n    jq -r '\n        .artists |\n        .items[] |\n        [ .uri, .name ] |\n        @tsv\n    ' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(jq -r '.artists.next' <<< \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_backup.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-02 19:11:12 +0100 (Thu, 02 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOne-touch Spotify Backup of all or selected Spotify playlists\n\nIf playlist args are given then backs up only those playlists\n\nWithout args, backs up all public playlists\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<playlist> <playlist2> ...]\"\n\nhelp_usage \"$@\"\n\nif is_blank \"${SPOTIFY_BACKUP_DIR:-}\"; then\n    if [[ \"$PWD\" =~ playlists ]]; then\n        export SPOTIFY_BACKUP_DIR=\"$PWD\"\n    else\n        export SPOTIFY_BACKUP_DIR=\"$PWD/playlists\"\n    fi\nfi\n\nsection \"Running Spotify Playlists Backup\"\n\nspotify_token\n\nSECONDS=0\n\nmkdir -pv \"$SPOTIFY_BACKUP_DIR/spotify\"\n\nif [ $# -gt 0 ]; then\n    timestamp \"Backing up selected playlist(s):\"\n    echo\n    for playlist in \"$@\"; do\n        printf '%s  ' \"$(date '+%F %T')\"\n        \"$srcdir/spotify_backup_playlist.sh\" \"$playlist\"\n    done\n    exit 0\nfi\n\n\"$srcdir/spotify_backup_artists_followed.sh\"\necho >&2\n\n\"$srcdir/spotify_backup_playlists_list.sh\"\necho >&2\n\n\"$srcdir/spotify_backup_playlists.sh\"\necho >&2\n\ntimestamp \"Spotify Backup completed in $SECONDS seconds\"\n"
  },
  {
    "path": "spotify/spotify_backup_artists_followed.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-07 00:30:25 +0000 (Sat, 07 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads the list of Spotify artists followed\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nif is_blank \"${SPOTIFY_BACKUP_DIR:-}\"; then\n    if [[ \"$PWD\" =~ playlists ]]; then\n        export SPOTIFY_BACKUP_DIR=\"$PWD\"\n    else\n        export SPOTIFY_BACKUP_DIR=\"$PWD/playlists\"\n    fi\nfi\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nSECONDS=0\n\nmkdir -pv \"$SPOTIFY_BACKUP_DIR/spotify\"\n\ntimestamp \"Backing up Spotify artists followed to: $SPOTIFY_BACKUP_DIR/spotify/artists_followed.txt\"\ntmp=\"$(mktemp)\"\n\"$srcdir/spotify_artists_followed_uri_name.sh\" | sort -k2 -f > \"$tmp\"\nmv -f \"$tmp\" \"$SPOTIFY_BACKUP_DIR/spotify/artists_followed.txt\"\n\ntimestamp \"Stripping Spotify artists followed URIs to create: $SPOTIFY_BACKUP_DIR/artists_followed.txt\"\nawk '{$1=\"\"; print}' \"$SPOTIFY_BACKUP_DIR/spotify/artists_followed.txt\" |\nsed 's/^[[:space:]]//' > \"$SPOTIFY_BACKUP_DIR/artists_followed.txt\"\n"
  },
  {
    "path": "spotify/spotify_backup_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 17:39:04 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBacks up a given Spotify playlist to text files in both Spotify URI and human readable formats\n\nSpotify track URI format can be copied and pasted back in to Spotify to restore a playlist to a previous state\n(for example if you accidentally deleted a track and didn't do an immediate Ctrl-Z / Cmd-Z)\n\nSpotify track URI format can also be combined with spotify_add_to_playlist.sh to restore or add to another playlist\n\nCaches metadata locally in .spotify_metadata/ directory. Tracks playlist name to automatically rename playlist files\nand playlist snapshot ID to avoid re-downloading playlists which haven't changed, reducing the number of Spotify API\ncalls and therefore the likelihood of hitting HTTP 429 Too Many Requests throttling errors\n\nIf a second argument is given with the Spotify Snapshot_ID, this avoids having to fetch the playlist's metadata,\ngreatly speeding up iterative downloads of all playlists\n\nThe environment variable SPOTIFY_PLAYLIST_FORCE_DOWNLOAD can be set to any value to force a playlist to redownload\nand ignore the last downloaded snapshot ID optimization above\n\nIf the playlist name has changed Detects the playlist file is committed to Git, then calls:\n\n    spotify_rename_playlist_files.sh\n\nto rename the corresponding playlist, spotify and description files to match the current name and then restore the\nchanges on top of the rename so you can then git diff to see what has changed if anything aside from the rename itself\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<playlist_id> <snapshot_id> <curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nif [ \"$playlist\" = liked ] || [ \"$playlist\" = saved ]; then\n    playlist=\"Liked Songs\"\nfi\nliked(){\n    [ \"$playlist\" = \"Liked Songs\" ]\n}\n\nplaylist_id=\"${2:-}\"\nsnapshot_id=\"${3:-}\"\n\nshift || :\nshift || :\nshift || :\n\nspotify_user\n\nif not_blank \"${SPOTIFY_BACKUP_DIR:-}\"; then\n    backup_dir=\"$SPOTIFY_BACKUP_DIR\"\nelif [[ \"$PWD\" =~ playlist ]]; then\n    backup_dir=\"$PWD\"\nelse\n    backup_dir=\"$PWD/playlists\"\nfi\n\n# messes up output per line when iterating all playlists via spotify_backup_playlists.sh\n#log \"Backing up to directory: $backup_dir\"\n\nbackup_dir_base=\"$backup_dir\"\nbackup_dir_spotify_base=\"$backup_dir/spotify\"\nbackup_dir_metadata=\"$backup_dir/.spotify_metadata\"\nunchanged_playlist=0\n\nmkdir -vp \"$backup_dir_base\"\nmkdir -vp \"$backup_dir_spotify_base\"\nmkdir -vp \"$backup_dir_metadata\"\n\n# load mappings once\nload_path_mappings(){\n    local base_dir=\"$1\"\n    local mappings_file=\"$backup_dir_base/.path_mappings.txt\"\n\n    # can't make this an associative array because we want to support multiple rules per dir path\n    dirs=()\n    regexes=()\n\n    [ -f \"$mappings_file\" ] || return 0\n\n    while IFS= read -r line; do\n        line=\"${line%%#*}\"\n        line=\"$(sed 's/^[[:space:]]*//;s/[[:space:]]*$//' <<< \"$line\")\"\n        [ -z \"$line\" ] && continue\n\n        # if file is split by tabs, prefer this\n        if [[ \"$line\" == *$'\\t'* ]]; then\n            dir=\"${line%%$'\\t'*}\"\n            regex=\"${line#*$'\\t'}\"\n            dir=\"$(trim \"$dir\")\"\n            regex=\"$(trim \"$regex\")\"\n        # but fall back to parsing multiple spaces as field separator between dir and regex\n        else\n            dir=\"$(sed -E 's/[[:space:]]{2,}.*//' <<< \"$line\")\"\n            regex=\"$(sed -E 's/^.*[[:space:]]{2,}//' <<< \"$line\")\"\n        fi\n        # dir should always be populated as it comes first\n        [ -z \"$dir\" ] && continue\n        [ -z \"$regex\" ] && continue\n\n        dirs+=(\"$dir\")\n        regexes+=(\"$regex\")\n    done < \"$mappings_file\"\n}\n\nget_path_mapping_subdir(){\n    local base_dir=\"$1\"\n    local playlist_name=\"$2\"\n    local debug_mappings=\"${DEBUG_MAPPINGS:-}\"\n\n    if [ -z \"${path_map_dirs_loaded:-}\" ]; then\n        load_path_mappings \"$base_dir\"\n        path_map_dirs_loaded=1\n    fi\n\n    local i\n\n    # try exact match first as it's faster than regex and allows us just use simpler literal lines\n    for (( i=0; i < ${#regexes[@]} ; i++ )); do\n        if [ \"$playlist_name\" = \"${regexes[$i]}\" ]; then\n            if [ -n \"$debug_mappings\" ]; then\n                echo \"literal match: [${regexes[$i]}]\" >&2\n                echo \"playlist:      [$playlist]\"      >&2\n                echo \"dir:           [${dirs[$i]}]\"    >&2\n                echo\n            fi\n            echo \"${dirs[$i]}\"\n            return 0\n        fi\n    done\n\n    # regex fallback\n    for (( i=0; i  <${#regexes[@]} ; i++ )); do\n        if [[ \"$playlist_name\" =~ ${regexes[$i]} ]]; then\n            if [ -n \"$debug_mappings\" ]; then\n                echo \"regex:    [${regexes[$i]}]\" >&2\n                echo \"playlist: [$playlist]\"      >&2\n                echo \"dir:      [${dirs[$i]}]\"    >&2\n                echo\n            fi\n            echo \"${dirs[$i]}\"\n            return 0\n        fi\n    done\n}\nexport -f get_path_mapping_subdir\n\napply_path_mapping(){\n    local playlist_name=\"$1\"\n    path_mapping_subdir=\"$(get_path_mapping_subdir \"$backup_dir_base\" \"$playlist_name\")\"\n    if [ -n \"$path_mapping_subdir\" ]; then\n        backup_dir=\"$backup_dir_base/$path_mapping_subdir\"\n        backup_dir_spotify=\"$backup_dir_base/spotify/$path_mapping_subdir\"\n    else\n        backup_dir=\"$backup_dir_base\"\n        backup_dir_spotify=\"$backup_dir_spotify_base\"\n    fi\n    mkdir -vp \"$backup_dir\"\n    mkdir -vp \"$backup_dir_spotify\"\n}\n\nif liked; then\n    playlist_name=\"Liked Songs\"\nfi\n\nif liked; then\n    export SPOTIFY_PRIVATE=1\nfi\nspotify_token\n\nSECONDS=0\n\n# uses the local spotify/playlists.txt file cache as it's faster than iterating the API for all playlists\n# with spotify_playlist_name_to_id.sh to find the ID of matching playlist name\nplaylist_name_to_id(){\n    local playlist=\"$1\"\n    # spotify/playlists.txt is generated up front by spotify_backup_playlists.sh\n    # before it calls this script against each playlist\n    local playlists_cache=\"spotify/playlists.txt\"\n    if [ -f \"$playlists_cache\" ]; then\n        playlist_id=\"$(\n            awk -v name=\"$playlist\" '\n                {\n                    id = $1\n                    $1 = \"\"\n                    sub(/^[[:space:]]+/, \"\")\n                    if ($0 == name) {\n                        print id\n                        exit\n                    }\n                }\n            ' \"$playlists_cache\"\n        )\"\n\n        if [ -n \"$playlist_id\" ] && [[ \"$playlist_id\" =~ ^[A-Za-z0-9]{22}$ ]]; then\n            log \"cache hit: $playlist_id\"\n            echo \"$playlist_id\"\n            return\n        fi\n    fi\n    log \"cache miss: processing to use API lookup\"\n    SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\"\n}\nexport -f playlist_name_to_id\n\nclean_trackname(){\n    local artist_track=\"$1\"\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    ' <<< \"$artist_track\"\n}\nexport -f clean_trackname\n\ntimestamp\n\nif liked; then\n    echo -n \"$playlist_name \"\n\n    filename=\"$(\"$srcdir/spotify_playlist_to_filename.sh\" <<< \"$playlist_name\")\"\n    apply_path_mapping \"$playlist_name\"\n\n    # Caching behaviour\n    # if we pass the second arg snapshot ID just use that to save an API call\n    if ! is_blank \"$snapshot_id\"; then\n        liked_added_at=\"$snapshot_id\"\n    else\n        liked_added_at=\"$(\"$srcdir/spotify_api.sh\" \"/v1/me/tracks?limit=1\" | jq -r '.items[0].added_at')\"\n    fi\n    liked_metadata_dir=\"$backup_dir_metadata/liked\"\n    mkdir -pv \"$liked_metadata_dir\"\n    liked_added_cache=\"$liked_metadata_dir/added_at\"\n    if [ -f \"$liked_added_cache\" ] &&\n       [ \"$liked_added_at\" = \"$(cat \"$liked_added_cache\")\" ] &&\n       is_blank \"${SPOTIFY_PLAYLIST_FORCE_DOWNLOAD:-}\"; then\n        echo -n '=> Latest Added Timestamp Unchanged'\n    else\n        echo -n \"=> URIs \"\n        #trap_cmd \"cd \\\"$backup_dir_spotify\\\" && git checkout \\\"$filename\\\" &>/dev/null\"\n        #\"$srcdir/spotify_liked_tracks_uri.sh\" \"$@\" | sort -f > \"$backup_dir_spotify/$filename\"\n        #untrap\n        # better to just use atomic moves so we can ./commit.sh even while this is running\n        # without being prompted with net removals by partially completed downloads\n        track_tmp=\"$(mktemp)\"\n        uri_tmp=\"$(mktemp)\"\n        #\"$srcdir/spotify_liked_tracks_uri.sh\" \"$@\" | sort -f > \"$tmp\"\n        #mv -f -- \"$tmp\" \"$backup_dir_spotify/$filename\"\n        # open output files once\n        exec 3>>\"$uri_tmp\"\n        exec 4>>\"$track_tmp\"\n        count=0\n        \"$srcdir/spotify_liked_uri_artist_track.sh\" |\n        while read -r uri track; do\n            if ! validate_spotify_uri \"$uri\" &>/dev/null &&\n               ! is_local_uri \"$uri\" ; then\n                die \"Invalid Spotify URI returned: '$uri', for track: $track\"\n            fi\n            echo \"$uri\" >&3\n            clean_trackname \"$track\" >&4\n\n            count=$((count + 1))\n            if [ $((count % 100)) -eq 0 ]; then\n                echo -n '.'\n            fi\n        done\n        # wipe out the progress dots and reprint the current line\n        clear_current_line\n        timestamp\n        echo -n \"$playlist_name => Description => URIs \"\n        #mv -f \"$track_tmp\" \"$backup_dir/$filename\"\n        #mv -f \"$uri_tmp\" \"$backup_dir_spotify/$filename\"\n        # XXX: sort the Liked URI and track orderings - although this breaks the fidelity between the playlist <=> spotify/playlist formats,\n        #      it's necessary to avoid recurring large diffs as Spotify seems to change the output ordering of this\n        sort -f \"$track_tmp\" > \"$backup_dir/$filename\"\n        sort -f \"$uri_tmp\" > \"$backup_dir_spotify/$filename\"\n        rm -f \"$track_tmp\"\n        rm -f \"$uri_tmp\"\n        # try to avoid hitting HTTP 429 rate limiting\n        sleep 0.1\n        num_track_uris=\"$(wc -l < \"$backup_dir_spotify/$filename\" | sed 's/[[:space:]]*//')\"\n        num_tracks=\"$(wc -l < \"$backup_dir/$filename\" | sed 's/[[:space:]]*//')\"\n\n        if [ \"$num_tracks\" != \"$num_track_uris\" ]; then\n            die \"ERROR: differing number of tracks ($num_tracks) vs URIs ($num_track_uris) detected for Liked Songs\"\n        fi\n\n        echo -n \"OK ($num_track_uris) => Tracks \"\n        #trap_cmd \"cd \\\"$backup_dir\\\" && git checkout \\\"$filename\\\" &>/dev/null\"\n        #\"$srcdir/spotify_liked_tracks.sh\" \"$@\" | sort -f > \"$backup_dir/$filename\"\n        #tmp=\"$(mktemp)\"\n        #\"$srcdir/spotify_liked_tracks.sh\" \"$@\" | sort -f > \"$tmp\"\n        #mv -f -- \"$tmp\" \"$backup_dir/$filename\"\n        #untrap\n        echo \"$liked_added_at\" > \"$liked_added_cache\"\n        echo -n 'OK'\n    fi\nelse\n    if ! is_blank \"$playlist_id\"; then\n        playlist_name=\"$playlist\"\n    else\n        playlist_id=\"$(playlist_name_to_id \"$playlist\")\"\n        # if we were passed a playlist_id instead of name as first arg to avoid one lookup,\n        # do a reverse lookup to get the name\n        playlist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\" \"$@\")\"\n    fi\n\n    echo -n \"$playlist_name\"\n\n    filename=\"$(\"$srcdir/spotify_playlist_to_filename.sh\" <<< \"$playlist_name\")\"\n    apply_path_mapping \"$playlist_name\"\n\n    # XXX: bugfix for 'illegal byte sequence error' for weird unicode chars in the filename\n    #filename=\"$(sed 's/[^[:alnum:][:space:]!\"$&'\"'\"'()+,.\\/:<_|–\\∕-]/-/g' <<< \"$filename\")\"\n\n    #echo \"Saving to filename: $filename\"\n\n    playlist_metadata_dir=\"$backup_dir_metadata/$playlist_id\"\n    mkdir -p \"$playlist_metadata_dir\"\n\n    playlist_metadata_name_file=\"$playlist_metadata_dir/name\"\n    playlist_metadata_filename_file=\"$playlist_metadata_dir/filename\"\n    playlist_metadata_snapshot_id_file=\"$playlist_metadata_dir/snapshot_id\"\n\n    #playlist_json=\"$(\"$srcdir/spotify_playlist_json.sh\" \"$playlist_id\")\"\n\n    # If we pass the second arg snapshot ID just use that to save an API call\n    if is_blank \"$snapshot_id\"; then\n        # optimization to pull only the fields we need without the first 100 tracks\n        playlist_json=\"$(\"$srcdir/spotify_api.sh\" \"/v1/playlists/$playlist_id?fields=snapshot_id,description\")\"\n\n        # debug code for when we hit HTTP 429 Too Many Requests back off errors\n        #if [ -n \"${SPOTIFY_DUMP_HEADERS:-}\" ]; then\n        #    \"$srcdir/../bin/curl_auth.sh\" -i \"$url_base/$url_path\" \"$@\" | sed '/^[[:space:]]*$/,$d' >&2\n        #    exit 1\n        #fi\n\n        echo -n \" => Description \"\n\n        description_file=\"$backup_dir/$filename.description\"\n\n        # playlist descriptions are HTML encoded\n        jq -r '.description' <<< \"$playlist_json\" | tr -d '\\n' | \"$srcdir/../bin/htmldecode.sh\" > \"$description_file\"\n\n        if [ -f \"$description_file\" ]; then\n            # if file is blank then no description is set, remove the useless file\n            if ! [ -s \"$description_file\" ]; then\n                rm -f -- \"$description_file\"\n                echo -n \"None\"\n            else\n                echo -n \"OK\"\n            fi\n        fi\n\n        snapshot_id=\"$(jq -r '.snapshot_id' <<< \"$playlist_json\" | tr -d '\\n')\"\n    fi\n\n    # renaming a playlist or changing its description also changes the snapshot ID,\n    # not just adding/removing/reordering tracks, triggering a full re-download and rename handling logic\n    if [ -f \"$playlist_metadata_snapshot_id_file\" ] &&\n       [ \"$snapshot_id\" = \"$(cat \"$playlist_metadata_snapshot_id_file\")\" ] &&\n       [ -f \"$backup_dir/$filename\" ] &&\n       [ -f \"$backup_dir_spotify/$filename\" ] &&\n       is_blank \"${SPOTIFY_PLAYLIST_FORCE_DOWNLOAD:-}\"; then\n        echo -n ' => Snapshot ID unchanged'\n        unchanged_playlist=1\n    else\n        # reset to the last good version to avoid having partial files which will offer bad commits of removed tracks\n        echo -n \" => URIs \"\n        #trap_cmd \"cd \\\"$backup_dir_spotify\\\" && git checkout \\\"$filename\\\" &>/dev/null\"\n        #\"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist_id\" \"$@\" > \"$backup_dir_spotify/$filename\"\n        track_tmp=\"$(mktemp)\"\n        uri_tmp=\"$(mktemp)\"\n        #\"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist_id\" \"$@\" > \"$tmp\"\n        #mv -f \"$tmp\" \"$backup_dir_spotify/$filename\"\n        #untrap\n        # open output files once\n        exec 3>>\"$uri_tmp\"\n        exec 4>>\"$track_tmp\"\n        count=0\n        \"$srcdir/spotify_playlist_tracks_uri_artist_track.sh\" \"$playlist_id\" \"$@\" |\n        # TODO: consider replacing this with a tee to two streaming commands to avoid so many executions\n        while read -r uri track; do\n            if ! validate_spotify_uri \"$uri\" &>/dev/null &&\n               ! is_local_uri \"$uri\" ; then\n                die \"Invalid Spotify URI returned: '$uri', for track: $track\"\n            fi\n            echo \"$uri\" >&3\n            clean_trackname \"$track\" >&4\n\n            count=$((count + 1))\n            if [ $((count % 100)) -eq 0 ]; then\n                echo -n '.'\n            fi\n        done\n        # wipe out the progress dots and reprint the current line\n        clear_current_line\n        timestamp\n        echo -n \"$playlist_name => Description => URIs \"\n        mv -f \"$track_tmp\" \"$backup_dir/$filename\"\n        mv -f \"$uri_tmp\" \"$backup_dir_spotify/$filename\"\n        # try to avoid hitting HTTP 429 rate limiting\n        sleep 0.1\n        num_track_uris=\"$(wc -l < \"$backup_dir_spotify/$filename\" | sed 's/[[:space:]]*//')\"\n        num_tracks=\"$(wc -l < \"$backup_dir/$filename\" | sed 's/[[:space:]]*//')\"\n\n        if [ \"$num_tracks\" != \"$num_track_uris\" ]; then\n            die \"ERROR: differing number of tracks ($num_tracks) vs URIs ($num_track_uris) detected for playlist: $playlist\"\n        fi\n\n        echo -n \"OK ($num_track_uris) => Tracks \"\n\n        # reset to the last good version to avoid having partial files which will offer bad commits of removed tracks\n        # no longer needed as we use > tmp && atomic move below now instead since it's cleaner\n        #trap_cmd \"cd \\\"$backup_dir\\\" && git checkout \\\"$filename\\\" &>/dev/null\"\n        #\"$srcdir/spotify_playlist_tracks.sh\" \"$playlist_id\" \"$@\" > \"$backup_dir/$filename\"\n        #untrap\n\n        # better to just use atomic moves so we can ./commit.sh even while this is running\n        # without being prompted with net removals by partially completed downloads\n        #tmp=\"$(mktemp)\"\n        # sometimes there are tracks that have blank names due to spotify data issues\n        #\"$srcdir/spotify_playlist_tracks.sh\" \"$playlist_id\" \"$@\" | sed '/^[[:space:]]*$/d' > \"$tmp\"\n        #\"$srcdir/spotify_playlist_tracks.sh\" \"$playlist_id\" \"$@\" > \"$tmp\"\n        #mv -f \"$tmp\" \"$backup_dir/$filename\"\n        echo -n 'OK'\n\n        old_filename=\"$(\n            if [ -f \"$playlist_metadata_filename_file\" ]; then\n                cat \"$playlist_metadata_filename_file\"\n            fi\n        )\"\n\n        if not_blank \"$old_filename\" &&\n           [ \"$backup_dir/$filename\" != \"$backup_dir/$old_filename\" ]; then\n\n            echo -n \" => playlist RENAMED\"\n\n            # with path mapping, renames are under base: Subdir/ and spotify/Subdir/; run from backup base\n            if [ -n \"${path_mapping_subdir:-}\" ]; then\n                cd \"$backup_dir_base\"\n            else\n                cd \"$backup_dir\"\n            fi\n\n            # if we're in a git repo and the old filename is git managed, then rename it\n            #\n            # optionally using a local rename.sh script if present - useful script hook which could have\n            # some more specific handling of corresponding files under management - *.description, spotify/ or\n            # .spotify/metadata/ files\n            #\n            # in my case this just calls spotify_rename_playlist_files.sh in this repo so it's the same, but a\n            # potentially useful hook script to leave in, and the rename.sh abstraction is simpler\n            if is_in_git_repo &&\n               is_file_tracked_in_git \"$old_filename\"; then\n                echo -n \" => updating files... \"\n                if [ -x ./rename.sh ]; then\n                    ./rename.sh \"$old_filename\" \"$filename\" ${path_mapping_subdir:+$path_mapping_subdir}\n                else\n                    \"$srcdir/../scripts/spotify_rename_playlist_files.sh\" \"$old_filename\" \"$filename\" ${path_mapping_subdir:+$path_mapping_subdir}\n                fi\n            fi\n\n            cd -\n        fi\n        # save all the metadata for comparison in the next run\n        echo \"$playlist_name\" > \"$playlist_metadata_name_file\"\n        echo \"$filename\"      > \"$playlist_metadata_filename_file\"\n        echo \"$snapshot_id\"   > \"$playlist_metadata_snapshot_id_file\"\n        # try to avoid hitting HTTP 429 rate limiting\n        sleep 0.1\n    fi\nfi\necho \" => $SECONDS secs\"\n# used by HariSekhon/Spotify-Playlists scripts to remove the many lines of unchanged playlists output\n# so I can see only what has changed and where I am spending time to optimize things\nif [ \"$unchanged_playlist\" = 1 ] && ! is_blank \"${QUIET_UNCHANGED_PLAYLISTS:-}\"; then\n    clear_previous_line\nfi\n"
  },
  {
    "path": "spotify/spotify_backup_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 17:39:04 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nBacks up Spotify playlists for a given user to text files in both Spotify URI and human readable formats\n\nSpotify track URI format can be copied and pasted back in to Spotify to restore a playlist to a previous state\n(for example if you accidentally deleted a track and didn't do an immediate Ctrl-Z / Cmd-Z)\n\nSpotify track URI format can also be combined with spotify_add_to_playlist.sh to restore or add to another playlist\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<spotify_user> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# used by spotify_user function below\n# shellcheck disable=SC2034\nspotify_user=\"${1:-${SPOTIFY_USER:-}}\"\n\n# avoids spotify_backup_playlist.sh having to resolve this itself each time\nexport SPOTIFY_USER=\"$spotify_user\"\n\nshift || :\n\nspotify_user\n\nif not_blank \"${SPOTIFY_BACKUP_DIR:-}\"; then\n    backup_dir=\"$SPOTIFY_BACKUP_DIR\"\nelif [[ \"$PWD\" =~ playlist ]]; then\n    backup_dir=\"$PWD\"\nelse\n    backup_dir=\"$PWD/playlists\"\nfi\n\nSECONDS=0\ntimestamp \"Backing up Spotify playlists to $backup_dir\"\necho >&2\nmkdir -vp \"$backup_dir\"\n\nspotify_token\n\n# for spotify_backup_playlist.sh to inherit, saving executions to recalculate on each iteration\nexport SPOTIFY_BACKUP_DIR=\"$backup_dir\"\n\n# stop spotify_foreach_playlist.sh from printing the playlist name as this results in duplicate output\nexport SPOTIFY_FOREACH_NO_PRINT_PLAYLIST_NAME=1\nexport SPOTIFY_FOREACH_NO_NEWLINE=1\n\nplaylist_file=\"$PWD/.spotify_metadata/playlists.txt\"\n\nif [ -f \"$playlist_file\" ] &&\n    file_newer_than_mins 5 \"$playlist_file\"; then\n    timestamp \"Spotify playlist file '$playlist_file' was updated < 5 minutes ago, using it for playlist name <=> ID lookups\"\n    while read -r playlist_id snapshot_id playlist_name; do\n        \"$srcdir/spotify_backup_playlist.sh\" \"$playlist_name\" \"$playlist_id\" \"$snapshot_id\" \"$@\"\n    done < \"$playlist_file\"\nelse\n    \"$srcdir\"/spotify_foreach_playlist.sh \"\n        '$srcdir/spotify_backup_playlist.sh' \\\"{playlist_name}\\\" '{playlist_id}' '{snapshot_id}'\n    \" \"$spotify_user\" \"$@\"\nfi\nif [ -n \"${SPOTIFY_PRIVATE:-}\" ] &&\n   is_blank \"${NO_LIKED_PLAYLIST:-}\"; then\n    \"$srcdir/spotify_backup_playlist.sh\" liked \"$@\"\nfi\necho >&2\ntimestamp \"Spotify playlists backup finished in $SECONDS seconds\"\n"
  },
  {
    "path": "spotify/spotify_backup_playlists_list.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-02 19:11:12 +0100 (Thu, 02 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads the list of Spotify playlists\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nif is_blank \"${SPOTIFY_BACKUP_DIR:-}\"; then\n    if [[ \"$PWD\" =~ playlists ]]; then\n        export SPOTIFY_BACKUP_DIR=\"$PWD\"\n    else\n        export SPOTIFY_BACKUP_DIR=\"$PWD/playlists\"\n    fi\nfi\n\nspotify_token\n\nSECONDS=0\n\nmkdir -pv \"$SPOTIFY_BACKUP_DIR/spotify\"\n\ntimestamp \"Dumping list of Spotify playlists to $SPOTIFY_BACKUP_DIR/.spotify_metadata/playlists.txt\"\ntmp=\"$(mktemp)\"\nSPOTIFY_PLAYLIST_SNAPSHOT_ID=1 \"$srcdir/spotify_playlists.sh\" > \"$tmp\"\nmv -f \"$tmp\" \"$SPOTIFY_BACKUP_DIR/.spotify_metadata/playlists.txt\"\n\ntimestamp \"Stripping spotify playlist Snapshot IDs from $SPOTIFY_BACKUP_DIR/.spotify_metadata/playlists.txt => $SPOTIFY_BACKUP_DIR/spotify/playlists.txt\"\nawk '{$2=\"\"; print}' \"$SPOTIFY_BACKUP_DIR/.spotify_metadata/playlists.txt\" |\nsed 's/  / /' > \"$SPOTIFY_BACKUP_DIR/spotify/playlists.txt\"\n\ntimestamp \"Stripping spotify playlist IDs from $SPOTIFY_BACKUP_DIR/spotify/playlists.txt => $SPOTIFY_BACKUP_DIR/playlists.txt\"\ntmp=\"$(mktemp)\"\nsed 's/^[^[:space:]]*[[:space:]]*//' \"$SPOTIFY_BACKUP_DIR/spotify/playlists.txt\" > \"$tmp\"\nmv -f \"$tmp\" \"$SPOTIFY_BACKUP_DIR/playlists.txt\"\n\ntimestamp \"Spotify playlists list downloaded\"\n"
  },
  {
    "path": "spotify/spotify_callback_openssl.cnf",
    "content": "[req]\ndefault_bits       = 2048\nprompt             = no\ndefault_md         = sha256\ndistinguished_name = dn\nx509_extensions    = ext\n\n[dn]\nCN = localhost\n\n[ext]\nsubjectAltName = @alt_names\n\n[alt_names]\nDNS.1 = localhost\nIP.1  = 127.0.0.1\nIP.2  = ::1\n"
  },
  {
    "path": "spotify/spotify_create_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 22:11:57 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a Spotify playlist\n\nBy default the created playlist will be public\n\nIf SPOTIFY_PRIVATE=1 is set in the environment then the created playlist will be private\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\npublic=true\n\nif [ -n \"${SPOTIFY_PRIVATE:-}\" ]; then\n    public=false\nfi\n\n# creating a playlist requires an authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nspotify_user\n\n# $spotify_user is defined by spotify_user() from library\n# shellcheck disable=SC2154\nurl_path=\"/v1/users/$spotify_user/playlists\"\n\n\"$srcdir/spotify_api.sh\" \"$url_path\" -X POST -H \"Content-Type: application/json\" -d \"{ \\\"name\\\": \\\"$playlist\\\", \\\"public\\\": $public }\"\n"
  },
  {
    "path": "spotify/spotify_delete_any_duplicates_in_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-11 13:34:47 +0100 (Fri, 11 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes any duplicates by either URI or 'Artist - Track' name\n\nThis is a heavy option that will blast duplicate URIs and then also look for duplicate track name matches\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\ntimestamp \"calling spotify_delete_duplicates_in_playlist.sh:\"\n\"$srcdir/spotify_delete_duplicates_in_playlist.sh\" \"$@\"\necho >&2\ntimestamp \"calling spotify_delete_tracks_duplicates_in_playlist.sh:\"\n\"$srcdir/spotify_delete_duplicate_tracks_in_playlists.sh\" \"$@\"\n"
  },
  {
    "path": "spotify/spotify_delete_duplicate_track_uris_in_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: test\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 19:05:25 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes duplicate Spotify URIs in a given playlist\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\nTo see which URIs would be deleted, you can first run spotify_duplicate_uri_in_playlist.sh <playlist_name_or_id> and optionally pipe that through spotify_uri_to_name.sh to translate to human readable names eg for a playlist called 'test':\n\nspotify_duplicate_uri_in_playlist.sh MyPlaylist\n\nspotify_duplicate_uri_in_playlist.sh MyPlaylist | spotify_uri_to_name.sh\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> [<playlist2> <playlist3> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nexport SPOTIFY_DUPLICATE_TRACK_POSITIONS=1\n\nfor playlist in \"$@\"; do\n    # this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\n    playlist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n    timestamp \"Finding and deleting duplicates in playlist \\\"$playlist\\\" by exact URI match:\"\n    \"$srcdir/spotify_duplicate_uri_in_playlist.sh\" \"$playlist_id\" |\n    \"$srcdir/spotify_delete_from_playlist.sh\" \"$playlist_id\"\ndone\n"
  },
  {
    "path": "spotify/spotify_delete_duplicate_tracks_in_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: test\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 19:05:25 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes duplicate Spotify tracks in a given playlist (by Artist - Track name match, may be from different albums / singles and not 100% identical performances)\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\nexport SPOTIFY_DUPLICATE_TRACK_POSITIONS=1\n\ntimestamp \"Finding and deleting duplicate tracks in playlist \\\"$playlist\\\" by exact \\\"Artist - Track\\\" name match:\"\nduplicates=\"$(\"$srcdir/spotify_duplicate_tracks_in_playlist.sh\" \"$playlist_id\")\"\n\nif [ -z \"$duplicates\" ]; then\n    timestamp \"No duplicate track names found\"\n    exit 0\nfi\n\ncount=\"$(wc -l <<< \"$duplicates\" | sed 's/[[:space:]]*//g')\"\n\nif [ -z \"${SPOTIFY_NO_CONFIRM:-}\" ]; then\n    if is_interactive; then\n        timestamp \"Duplicates to remove:\"\n        echo >&2\n\n        while read -r position uri; do\n            printf '%s\\t' \"$position\"\n            \"$srcdir/spotify_uri_to_name.sh\" <<< \"$uri\"\n        done <<< \"$duplicates\"\n        echo\n\n        read -r -p \"Are you sure you want to delete these $count tracks from playlist \\\"$playlist\\\"? (y/N) \" answer\n        echo >&2\n\n        shopt -s nocasematch\n\n        if ! [[ \"$answer\" =~ y|yes ]]; then\n            timestamp \"Aborting...\"\n            exit 1\n        fi\n    fi\nfi\n\ntimestamp \"Deleting $count duplicate tracks\"\necho >&2\n\"$srcdir/spotify_delete_from_playlist.sh\" \"$playlist_id\" <<< \"$duplicates\"\n"
  },
  {
    "path": "spotify/spotify_delete_from_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 19:05:25 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes Spotify URIs from a given playlist\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\nCan take file(s) with URIs as arguments or read from standard input for chaining with other tools\n\nInput formats can be IDs or any standard Spotify URI format, eg:\n\nspotify:track:<ID>\nhttps://open.spotify.com/track/<ID>\n<ID>\n\nor can be prefixed with track position in the playlist (zero-indexed) if you only want to delete a single instance of the song (useful when removing only duplicates), separated by either space:\n\n<track_position>      spotify:track:<ID>\n<track_position>      https://open.spotify.com/track/<ID>\n<track_position>      <ID>\n\nor a colon:\n\n<track_position>:spotify:track:<ID>\n<track_position>:https://open.spotify.com/track/<ID>\n<track_position>:<ID>\n\n\nUseful for chaining with other tools (eg. spotify_playlist_tracks_uri.sh / spotify_search_uri.sh in this repo, or\ntracks_already_in_playlists.sh in the HariSekhon/Spotify-Playlists github repo) or loading from saved spotify format\nplaylists (eg. TODO playlists dumped by spotify_backup*.sh / spotify_playlist_tracks_uri.sh)\n\nCaveat: won't check the tracks are already in the playlist, will simply fire batch delete API calls and count the number of tracks requested to be removed, so repeated runs of the same URIs fed in will give the same results, which might mislead you to thinking they weren't remove the first time around, when they've already been removed\n\nIf you set environment variable SPOTIFY_RESOLVE_TRACKS_DELETED it'll resolve and print the names of the tracks its deleting for debugging purposes (this is more expensive as it requires one extra API call per 50 tracks, tripling the number of API calls required)\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> [<file1> <file2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\nplaylist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\")\"\n\n# playlist ID obtained from 'spotify_playlists.sh'\nurl_path=\"/v1/playlists/$playlist_id/tracks\"\n\ncount=0\n\nsnapshot_id=\"\"\n\n# Takes track IDs or track position:ID for more specific deletes\ndelete_from_playlist(){\n    if [ $# -lt 1 ]; then\n        echo \"Error: no IDs passed to delete_from_playlist()\" >&2\n        exit 1\n    fi\n    local uri_array=\"\"\n    local track_position\n    local id\n    if [ -n \"${SPOTIFY_RESOLVE_TRACKS_DELETED:-}\" ]; then\n        for id in \"$@\"; do\n            if [[ \"$id\" =~ ^[[:digit:]]+: ]]; then\n                id=\"${id#*:}\"\n            fi\n            echo \"spotify:track:$id\"\n        done |\n        \"$srcdir/spotify_uri_to_name.sh\"\n    fi\n    for id in \"$@\"; do\n        if [[ \"$id\" =~ ^[[:digit:]]+: ]]; then\n            # extract first column for track position\n            track_position=\"${id%%:*}\"\n            # don't try to calculate this, the numbers aren't as predicted during testing, use snapshot consistency instead\n            #((track_position-=count))\n            # keep zero-indexed for compatability with other tools - no longer necessary, they return zero indexed now\n            #((track_position-=1)) # convert one-indexed (eg. from grep) to zero-indexed for Spotify API\n            id=\"${id#*:}\"\n            # requires explicit track URI type since could also be episodes added to playlist\n            uri_array+=\"{\\\"uri\\\": \\\"spotify:track:$id\\\", \\\"positions\\\": [$track_position]}, \"\n            if [ -z \"$snapshot_id\" ]; then\n                # get the Snapshot ID of the playlist before we start deleting for consistency and use this for all rounds of deletions\n                snapshot_id=\"$(\"$srcdir/spotify_api.sh\" \"${url_path%/tracks}?fields=snapshot_id\" | jq -r '.snapshot_id')\"\n            fi\n        else\n            # requires explicit track URI type since could also be episodes added to playlist\n            uri_array+=\"{\\\"uri\\\": \\\"spotify:track:$id\\\"}, \"\n        fi\n    done\n    uri_array=\"${uri_array%, }\"\n    timestamp \"removing ${#@} tracks from playlist '$playlist_name'\"\n    json_payload='{\"tracks\": '\"[$uri_array]\"\n    #if [ -n \"$snapshot_id\" ] && [ \"$snapshot_id\" != null ]; then\n    # let it send null and fail as we should never get back null\n    if [ -n \"$snapshot_id\" ]; then\n        json_payload+=\", \\\"snapshot_id\\\": \\\"$snapshot_id\\\"\"\n    fi\n    json_payload+=\"}\"\n    local output\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" -X DELETE -d \"$json_payload\")\"\n    #die_if_error_field \"$output\"\n    warn_if_error_field \"$output\"\n    ((count+=${#@}))\n    # don't take the new snapshot ID - use the one from before we start deleting for consistency otherwise the second round of deletes will fail\n    #snapshot_id=\"$(jq -r '.snapshot_id' <<< \"$output\")\"\n    #if is_blank \"$snapshot_id\"; then\n    #    die \"Spotify API returned blank snapshot id, please investigate with DEBUG=1 mode\"\n    #fi\n    #if [ \"$snapshot_id\" = null ]; then\n    #    die \"Spotify API returned snapshot_id '$snapshot_id', please investigate with DEBUG=1 mode\"\n    #fi\n    # slow down a bit to try to reduce hitting Spotify API rate limits and getting 429 errors on large playlists\n    sleep 1\n}\n\ndelete_URIs_from_playlist(){\n    declare -a ids\n    ids=()\n    while read -r track_uri; do\n        track_position=\"\"\n        if [[ \"$track_uri\" =~ ^[[:digit:]]+[:[:space:]]+ ]]; then\n            # extract first column for track position\n            track_position=\"${track_uri%%[:[:space:]]*}\"\n\n            # remove first column of track position\n            track_uri=\"${track_uri#*[:[:space:]]}\"\n\n            # strip any remaining leading whitespace without subshelling to sed\n            track_uri=\"${track_uri#\"${track_uri%%[!:[:space:]]*}\"}\"\n        fi\n        if is_blank \"$track_uri\"; then\n            continue\n        fi\n        if is_local_uri \"$track_uri\"; then\n            continue\n        fi\n        if [ -n \"${SPOTIFY_DELETE_IGNORE_IRREGULAR_IDS:-}\" ]; then\n            id=\"$(validate_spotify_uri \"$track_uri\" || :)\"\n            if ! is_spotify_playlist_id \"$id\"; then\n                timestamp \"skipping deleting irregular ID '$id'\"\n                continue\n            fi\n        else\n            id=\"$(validate_spotify_uri \"$track_uri\")\"\n        fi\n\n        if [ -n \"$track_position\" ]; then\n            ids+=(\"$track_position:$id\")\n        else\n            ids+=(\"$id\")\n        fi\n\n        if [ \"${#ids[@]}\" -eq 100 ]; then\n            delete_from_playlist \"${ids[@]}\"\n            ids=()\n        fi\n    done\n\n    if [ \"${#ids[@]}\" -gt 0 ]; then\n        delete_from_playlist \"${ids[@]}\"\n    fi\n}\n\ndelete_URIs_from_playlist < <(\n    cat \"${@:-/dev/stdin}\" |\n    sort -u\n)\n\ntimestamp \"$count tracks deleted from playlist '$playlist_name'\"\n"
  },
  {
    "path": "spotify/spotify_delete_from_playlist_if_in_other_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 17:29:53 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes tracks from the given playlist if their URIs are found in the subsequently given playlists\n\nThis is useful to delete things from TODO playlists that are already in a bunch of other playlists\n\nThe first playlist is the one to delete the tracks in, this will be your TODO playlist\n\nSubsequent playlist args are the source playlists to check for already existing tracks\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> <playlist_name_or_id> [<playlist_name_or_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nplaylist_to_delete_from=\"$1\"\nshift || :\n\ntimestamp \"Preparing to delete from playlist: $playlist_to_delete_from\"\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# BSD grep has a bug in grep -f, rely on GNU grep instead\nif is_mac; then\n    grep(){\n        # handles grep -f properly\n        command ggrep \"$@\"\n    }\nfi\n\nfor playlist in \"$@\"; do\n    timestamp \"Getting track URIs from other playlist for exact matching: $playlist\"\n    \"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist\"\ndone |\nsort -u |\ngrep -Fxf <(\n    timestamp \"Getting track URIs from target playlist to delete from: $playlist_to_delete_from\"\n    \"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist_to_delete_from\"\n) |\n\"$srcdir/spotify_delete_from_playlist.sh\" \"$playlist_to_delete_from\"\n"
  },
  {
    "path": "spotify/spotify_delete_from_playlist_if_track_in_other_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-07-22 17:00:36 +0100 (Fri, 22 Jul 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes tracks from the given playlist if their 'Artist - Track' name matches exactly tracks found in the subsequently given playlists\n\nThis is useful to delete things from TODO playlists that are already in a bunch of other playlists\n\nThe first playlist is the one to delete the tracks in, this will be your TODO playlist\n\nSubsequent playlist args are the source playlists to check for already existing tracks\n\nCaveat: this is not as accurate as the default adjacent script spotify_delete_from_playlist_if_in_other_playlists.sh which only deletes on exact URI matches\n        because you can have different versions of the same song with same 'Artist - Track' name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> <playlist_name_or_id> [<playlist_name_or_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nplaylist_to_delete_from=\"$1\"\nshift || :\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# BSD grep has a bug in grep -f, rely on GNU grep instead\nif is_mac; then\n    grep(){\n        # handles grep -f properly\n        command ggrep \"$@\"\n    }\n    sed(){\n        # handles + matching properly\n        command gsed \"$@\"\n    }\nfi\n\n# switched to optimized awk filtering instead to avoid all regex character issues and also enforce ending anchoring\n#grep -Ff <(\n\"$srcdir/../bin/text_filter_ending_substrings.sh\" \\\n<(\n    for playlist in \"$@\"; do\n        timestamp \"Getting list of Artist - Track names from source playlist: $playlist\"\n        \"$srcdir/spotify_playlist_tracks.sh\" \"$playlist\"\n    done |\n    sort -u #|\n    # No longer needed since switching to optimized awk filtering using text_filter_ending_substrings.sh\n    # XXX: anchor the track name as as suffix regex to prevent it accidentally removing '... (Remix)' or similar alternate versions of tracks\n    #      this is a trade off vs using -F for regex safety - I am betting the fringe conditional of a track name having some character that\n    #      breaks when used as a regex, possibly resulting in missing an odd track, is far less common than the other failure scenario of\n    #      matching substings of track names. If this becomes a problem we can add escaping to special characters interpreted by regex\n    #sed 's/$/$/'\n) \\\n<(\n    # Returns a list one per line in the format:\n    #\n    #   URI \\t Artist - Track\n    #\n    timestamp \"Getting list of URI + Artist - Track names from target playlist: $playlist_to_delete_from\"\n    \"$srcdir/spotify_playlist_tracks_uri_artist_track.sh\" \"$playlist_to_delete_from\"\n) |\n# get just the URIs of matching tracks\nsed $'s/\\t.*$//' |\n\"$srcdir/spotify_delete_from_playlist.sh\" \"$playlist_to_delete_from\"\n"
  },
  {
    "path": "spotify/spotify_duplicate_tracks_in_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: test | tee /dev/stderr | spotify_uri_to_name.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 19:05:25 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate Spotify tracks in a given playlist\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\nYou can combine this with spotify_uri_to_name.sh to see the duplicate track names eg. for a playlist called 'My Playlist':\n\n${0##*/} 'My Playlist'\n\nIf \\$SPOTIFY_DUPLICATE_TRACK_POSITIONS is set then outputs the track position (zero-indexed to align with the Spotify API) as the first column with the URI as the second column\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n# playlists max out at only around ~8000 tracks so this is safe to do in ram\ntracklist_URIs=\"$(\"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist_id\")\"\n\ntracklist_tracks=\"$(\"$srcdir/spotify_uri_to_name.sh\" <<< \"$tracklist_URIs\")\"\n\nduplicate_tracks=\"$(sort <<< \"$tracklist_tracks\" | uniq -d)\"\n\nwhile read -r track_name; do\n    [ -n \"$track_name\" ] || continue\n    grep -Fxn \"$track_name\" <<< \"$tracklist_tracks\" |\n    tail -n +2\ndone <<< \"$duplicate_tracks\" |\nif [ -n \"${SPOTIFY_DUPLICATE_TRACK_POSITIONS:-}\" ]; then\n    sed 's/:/ /' |\n    while read -r track_position track_name; do\n        uri=\"$(sed -n \"${track_position}p\" <<< \"$tracklist_URIs\")\"\n        ((track_position -= 1))\n        echo \"$track_position $uri\"\n    done |\n    column -t\nelse\n    sed 's/^[[:digit:]]*://'\nfi\n"
  },
  {
    "path": "spotify/spotify_duplicate_uri_in_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: test | tee /dev/stderr | spotify_uri_to_name.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-24 19:05:25 +0100 (Fri, 24 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists duplicate Spotify URIs in a given playlist\n\nPlaylist must be specified as the first argument and can be either a Spotify playlist ID or a full playlist name (see spotify_playlists.sh)\n\nYou can combine this with spotify_uri_to_name.sh to see the duplicate track names eg. for a playlist called 'My Playlist':\n\n${0##*/} 'My Playlist' | spotify_uri_to_name.sh\n\nIf \\$SPOTIFY_DUPLICATE_TRACK_POSITIONS is set then also outputs the track position (zero-indexed to align with the Spotify API) as the first column with the URI as the second column\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nshift || :\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n# playlists max out at only around ~8000 tracks so this is safe to do in ram\ntracklist_URIs=\"$(\"$srcdir/spotify_playlist_tracks_uri.sh\" \"$playlist_id\")\"\n\nduplicate_URIs=\"$(sort <<< \"$tracklist_URIs\" | uniq -d)\"\n\nwhile read -r uri; do\n    [ -n \"$uri\" ] || continue\n    grep -Fxn \"$uri\" <<< \"$tracklist_URIs\" |\n    tail -n +2\ndone <<< \"$duplicate_URIs\" |\nif [ -n \"${SPOTIFY_DUPLICATE_TRACK_POSITIONS:-}\" ]; then\n    sed 's/:/ /' |\n    while read -r track_position uri; do\n        ((track_position-=1))\n        echo \"$track_position $uri\"\n    done |\n    column -t\nelse\n    sed 's/^[[:digit:]]*://'\nfi\n"
  },
  {
    "path": "spotify/spotify_filename_to_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: < <(find ../playlists/spotify/ -type f)\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 17:39:04 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nNormalizes a Spotify playlist filename provided as arg(s) or stdin to an original playlist name\n\nThis is used because playlists with slashes have them converted to unicode equivalent by spotify_playlist_to_filename.sh, so when searching for playlists in spotify_playlist_name_to_id.sh, I use this script to reverse the process to allow for using auto-completed filenames as playlist names and still having everything resolve correctly\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_filename>\"\n\nhelp_usage \"$@\"\n\nnormalize(){\n    # strip folder name\n    sed 's|.*/||' |\n    # replace unicode forward slash needed for storing as filename with the original ascii version in the real playlist name\n    tr '∕' '/'\n}\n\nif not_blank \"$*\"; then\n    normalize <<< \"$*\"\nelse\n    normalize  # from stdin\nfi\n"
  },
  {
    "path": "spotify/spotify_follow_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 21:06:46 +0100 (Fri, 23 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/follow/get-followed/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTakes a list of Spotify Artists in URI format and sets them to be followed\n\nUseful when combined with adjacent script:\n\n    spotify_liked_artists_uri.sh\n\nto automatically follow all artists for which you have liked tracks, or combined with a shell pipeline to only like artists with at least 3 liked tracks:\n\n    spotify_liked_artists_uri.sh | sort | uniq -c | sort -k1nr | grep\n\n\n$usage_auth_help\n\n\nCan specify either one or more files with a list of spotify artist URIs, otherwise reads URIs from standard input\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<file1> <file2> ...]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/following?type=artist\"\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\ncount=0\n\nfollow_artists(){\n    if [ $# -lt 1 ]; then\n        echo \"Error: no artist IDs passed to follow_artists()\" >&2\n        exit 1\n    fi\n    local ids=\"\"\n    for id in \"$@\"; do\n        ids+=\"$id,\"\n    done\n    ids=\"${ids%,}\"\n    timestamp \"following ${#@} artists\"\n    \"$srcdir/spotify_api.sh\" \"$url_path&ids=$ids\" -X PUT #>/dev/null  # ignore the { \"spotify_snapshot\": ... } json output\n    ((count+=${#@}))\n}\n\nadd_file_URIs(){\n    declare -a ids\n    ids=()\n    while read -r artist_uri; do\n        if is_blank \"$artist_uri\"; then\n            continue\n        fi\n        if is_local_uri \"$artist_uri\"; then\n            continue\n        fi\n        id=\"$(uri_type=artist validate_spotify_uri \"$artist_uri\")\"\n\n        ids+=(\"$id\")\n\n        if [ \"${#ids[@]}\" -ge 50 ]; then\n            follow_artists \"${ids[@]}\"\n            sleep 1\n            ids=()\n        fi\n    done < \"$filename\"\n\n    if [ \"${#ids[@]}\" -eq 0 ]; then\n        return\n    fi\n    follow_artists \"${ids[@]}\"\n}\n\nfor filename in \"${@:-/dev/stdin}\"; do\n    add_file_URIs \"$filename\"\ndone\n\ntimestamp \"$count artists followed\"\n"
  },
  {
    "path": "spotify/spotify_follow_liked_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-26 15:50:49 +0000 (Mon, 26 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFollows Spotify artists with N or more tracks in your Liked Songs using the Spotify API\n\nThe threshold for the number of Liked Songs for an artist to be followed defaults to 5 Liked Songs but this can be overriden using the first argument\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<threshold_number_of_Liked_Songs>\"\n\nhelp_usage \"$@\"\nno_more_opts \"$@\"\n\nthreshold=\"${1:-5}\"\n\nis_int \"$threshold\" || usage \"threshold given is not an integer\"\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n\"$srcdir/spotify_liked_artists_uri.sh\" |\nsort |\nuniq -c |\nsort -k1nr |\nwhile read -r num uri; do\n    if [ \"$num\" -ge 5 ]; then\n        echo \"$uri\"\n    fi\ndone |\n\"$srcdir/spotify_follow_artists.sh\"\n"
  },
  {
    "path": "spotify/spotify_follow_top_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-26 15:50:49 +0000 (Mon, 26 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFollows all artists in your Spotify top artists list using the Spotify API\n\nUseful to ensure that you are following the artists you most listen to\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\nno_more_opts \"$@\"\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n\"$srcdir/spotify_top_artists_uri.sh\" |\n\"$srcdir/spotify_follow_artists.sh\"\n"
  },
  {
    "path": "spotify/spotify_foreach_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 16:33:04 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExecutes a command per Spotify playlist for a given user\n\nThe command must be quoted as the first argument and is templated, replacing the placeholders {playlist} and {playlist_id} in the command string\n\nUseful for combining with other spotify_*.sh scripts, such as downloading all the 'Artist - Track' names or all the Spotify URIs as backups\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the playlist id/names and exit after the first iteration\n\nRequires \\$SPOTIFY_USER be set in the environment or else given as the second arg\n\n$usage_playlist_help\n\n$usage_auth_help\n\n\nExamples:\n\n./spotify_foreach_playlist.sh './spotify_playlist_tracks.sh {playlist_id} > playlists/{playlist}.txt' harisekhon\n\n./spotify_foreach_playlist.sh './spotify_playlist_tracks_uri.sh {playlist_id} > playlist-backups/{playlist}.txt' harisekhon\n\n(see spotify_backup_playlists.sh for an even better implementation of this above one liner using this script)\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command to execute per playlist> [<spotify_user> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncommand_template=\"$1\"\nshift || :\n\nspotify_token\n\n# trigger spotify_playlists.sh to return an extra middle column with the snapshot ID\n# which we pass to spotify_backup_playlist.sh to avoid re-downloading many playlists\nexport SPOTIFY_PLAYLIST_SNAPSHOT_ID=1\n\nplaylists=\"$(\"$srcdir/spotify_playlists.sh\" \"$@\")\"\n\ntotal_playlists=\"$(grep -c . <<< \"$playlists\")\"\n\ni=0\n\nwhile read -r playlist_id snapshot_id playlist; do\n    (( i += 1 ))\n    if is_blank \"${SPOTIFY_FOREACH_NO_PRINT_PLAYLIST_NAME:-}\"; then\n        printf '%s/%s  %s\\t' \"$i\" \"$total_playlists\" \"$playlist\"\n    fi\n    if ! is_spotify_playlist_snapshot_id \"$snapshot_id\"; then\n        die \"ERROR: invalid Spotify snapshot_id '$snapshot_id' detected for playlist: $playlist\"\n    fi\n    # handle danger - done at playlist level not command level because we need late command evaluation in spotify_backup_playlists.sh\n    # this works, tested on Ke$ha playlist and `echo injected`\n    playlist=\"${playlist//$/\\\\$}\"\n    playlist=\"${playlist//\\`/}\"\n    playlist=\"${playlist//\\\\\"/\\\\\\\\\"}\"\n    # shellcheck disable=SC1003\n    playlist=\"${playlist//'/\\\\'}\"\n    cmd=\"${command_template//\\{playlist_id\\}/$playlist_id}\"\n    cmd=\"${cmd//\\{playlist\\}/$playlist}\"\n    cmd=\"${cmd//\\{snapshot_id\\}/$snapshot_id}\"\n    eval \"$cmd\"\n    if is_blank \"${SPOTIFY_FOREACH_NO_NEWLINE:-}\"; then\n        printf '\\n'\n    fi\ndone <<< \"$playlists\"\n"
  },
  {
    "path": "spotify/spotify_liked_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 21:06:46 +0100 (Fri, 23 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the all artist names from tracks in Liked Songs\n\nUseful to see how many tracks are liked from each artist:\n\n    ${0##*/} |\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\n# max 50 artists per request\nurl_path=\"/v1/me/tracks?limit=$limit&offset=$offset\"\n\noutput(){\n    #jq -r . <<< \"$output\"\n    jq -r '.items[].track.artists[].name' <<< \"$output\"\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_liked_artists_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-23 21:06:46 +0100 (Fri, 23 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the artist URIs of all artists in Liked Songs\n\nUseful for piping to spotify_follow_artists.sh\n\nTo see the list of Liked artists by name, it's faster to pipeline all the URIs to spotify_uri_to_name.sh which batches translations in groups of 50, then sort and count at the very end by name (it's also potentially more accurate where more than one URI converts to the same artist name), or just run:\n\n    spotify_liked_artists.sh\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\n# max 50 artists per request\nurl_path=\"/v1/me/tracks?limit=$limit&offset=$offset\"\n\noutput(){\n    #jq -r . <<< \"$output\"\n    jq -r '.items[].track.artists[].uri' <<< \"$output\"\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_liked_tracks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Liked Songs (aka Saved Tracks) via the Spotify API\n\nOutput format:\n\nArtist - Track\n\nor if \\$SPOTIFY_CSV environment variable is set then:\n\n\\\"Artist\\\",\\\"Track\\\"\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/tracks?limit=$limit&offset=$offset\"\n\noutput(){\n    # some tracks come out with blank artists and track name, skip these using select(name != \"\") filter to avoid blank lines\n    # unfortunately some tracks actually do come out with blank artist and track name, this must be a bug inside Spotify, but\n    # filtering it like this throws off the line counts verification and also the track might be blank but the artist might not be\n    if not_blank \"${SPOTIFY_CSV:-}\"; then\n        #jq -r '.items[].track | select(.name != \"\") | [([.artists[].name] | join(\", \")), .name] | @csv'\n        jq -r '.items[].track | [([.artists[].name] | join(\", \")), .name] | @csv'\n    else\n        #jq -r '.items[].track | select(.name != \"\") | [([.artists[].name] | join(\", \")), \"-\", .name] | @tsv'\n        jq -r '.items[].track | [([.artists[].name] | join(\", \")), \"-\", .name] | @tsv'\n    fi <<< \"$output\" |\n    tr '\\t' ' ' |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d;\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_liked_tracks_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the URIs of the current Spotify user's Liked Songs (aka Saved Tracks) via the Spotify API\n\nSpotify track URIs can be used as backups to restore a playlist's contents or copying to a new playlist\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/tracks?limit=$limit&offset=$offset\"\n\noutput(){\n    jq -r '.items[] | [.track.uri] | @tsv' <<< \"$output\"\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_liked_uri_artist_track.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the URIs and 'Artist - Track' names of the current user's Liked Songs (aka Saved Tracks) via the Spotify API\n\nOutput format:\n\nURI \\\\t Artist - Track\n\n\nUsed by spotify_backup_playlist.sh instead of spotify_liked_tracks.sh and spotify_liked_tracks_uri.sh separately\nbecause this halves tbe number of API calls to postprocess rather than fetch separate lists - this is a big\noptimization given I have nearly 100,000 tracks under management which requires a lot of paging\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/tracks?limit=$limit&offset=$offset\"\n\noutput(){\n    jq -r '.items[].track | [.uri, ([.artists[]?.name] | join(\", \")), \"-\", .name] | @tsv' <<< \"$output\" |\n    sed $'s/\\t/|/' |\n    tr '\\t' ' ' |\n    tr '|' '\\t' |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]$/d;\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-31 10:07:23 +0000 (Sat, 31 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns only the artist names for the tracks in a given Spotify playlist\n\nPlaylist argument can be a playlist name or a playlist ID (get this from spotify_playlists.sh)\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\n\nOutputs 1 artist per line, so tracks with multiple artists become multiple artist lines\n\nThis is useful for piping to 'sort | uniq -c' to:\n\n1. Find top artists per playlist by counting the number of tracks per artist\n2. Find top blacklisted artists by doing the above for my Blacklist playlist (done the Spotify-Playlists repo)\n\nExample:\n\n    SPOTIFY_PRIVATE=1 spotify_playlist_artists.sh Blacklist | sort | uniq -c | sort -k1nr\n\n\nIn these use cases though, this newer script is better as it normalizes track names and outputs the ranked order:\n\n    spotify_playlist_top_artists.sh\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\noutput(){\n    jq -r '.items[].track | select(.artists) | .artists[].name' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d;\n    '\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_id_to_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: 2ddv4fdbsD7WOnmY30g40i | tee /dev/stderr | spotify_playlist_name_to_id.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-03 00:25:24 +0100 (Fri, 03 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses Spotify API to translate a Spotify public playlist ID to a name\n\nIf a spotify playlist name is given instead of an ID, returns it as is\n\nA single playlist ID can be given as an argument, or a list can be passed via stdin\n\nNeeded by several other adjacent spotify tools\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_id> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id_to_name(){\n    local playlist_id=\"$1\"\n    shift || :\n    # if it's not a playlist id, scan all playlists and take the ID of the first matching playlist name\n    if is_spotify_playlist_id \"$playlist_id\"; then\n        playlist_name=\"$(\n            \"$srcdir/spotify_api.sh\" \"/v1/playlists/$playlist_id\" \"$@\" |\n            jq -r '.name' |\n            sed 's/[[:space:]]*$//' || :\n        )\"\n        # it turns out a playlist name can be blank :-/\n        #if is_blank \"$playlist_name\" || [ \"$playlist_name\" = null ]; then\n        if is_blank \"$playlist_name\"; then\n            echo \"$playlist_id\"\n            die \"Error: playlist name is blank for playlist ID: $playlist_id\"\n        elif [ \"$playlist_name\" = \"$playlist_id\" ]; then\n            die \"Error: playlist name resolved to the same as ID - this might be an edge case / bug and requires investigation\"\n        elif [ \"$playlist_name\" = null ]; then\n            die \"Error: failed to find playlist name matching ID '$playlist_id'\"\n        fi\n        echo \"$playlist_name\"\n    else\n        echo \"$playlist_id\"\n    fi\n}\n\nspotify_token\n\nif [ $# -gt 0 ]; then\n    playlist_id=\"$1\"\n    shift || :\n    playlist_id_to_name \"$playlist_id\" \"$@\"\nelse\n    while read -r playlist_id; do\n        playlist_id_to_name \"$playlist_id\" \"$@\"\n    done\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns Spotify API output for a given playlist\n\nPlaylist argument can be a playlist name or ID (see spotify_playlists.sh)\n\nCaveat: limited to 50 public playlists due to Spotify API, must specify OFFSET=50 to get next 50.\n        This script does not iterate each page automatically because the output would be nonsensical\n        multiple json outputs so you must iterate yourself and process each json result in turn\n        For an example of how to do this and process multiple paged requests see spotify_playlist_tracks.sh\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist id not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\n\"$srcdir/spotify_api.sh\" \"/v1/playlists/$playlist_id?limit=$limit&offset=$offset\" \"$@\"\n"
  },
  {
    "path": "spotify/spotify_playlist_name_to_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"My Shazam Tracks\" | tee /dev/stderr | spotify_playlist_id_to_name.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-03 00:25:24 +0100 (Fri, 03 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUses Spotify API to translate a Spotify public playlist name to ID\n\nIf a Spotify playlist ID is given, returns it as is (this is for coding convenience when calling from other scripts)\n\nNeeded by several other adjacent spotify tools\n\nThis is quite a slow O(n) operation as it has to iterate through all playlists until it finds a matching name to\nretrieve its ID and is therefore used as a last resort by my adjacent scripts\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#if is_mac; then\n#    awk(){\n#        command gawk \"$@\"\n#    }\n#fi\n\n# causes way too many random problems to allow partial substring matching, wastes time debugging, better to fail\nexport SPOTIFY_PLAYLIST_EXACT_MATCH=1\n\nplaylist_name_to_id(){\n    local playlist_name=\"$1\"\n    shift || :\n    # if it's not a playlist id, scan all playlists and take the ID of the first matching playlist name\n    if is_spotify_playlist_id \"$playlist_name\"; then\n        echo \"$playlist_name\"\n    else\n        # If we've auto-completed a playlist name from the filename, replace the unicode slashes with the real ones\n        if [ -f \"$playlist_name\" ]; then\n            playlist_name=\"$(\"$srcdir/spotify_filename_to_playlist.sh\" <<< \"$playlist_name\")\"\n        fi\n        # works but could get needlessly complicated to escape all possible regex special chars, switching to partial string match instead\n        #playlist_regex=\"${playlist_id//\\//\\\\/}\"\n        #playlist_regex=\"${playlist_regex//\\(/\\\\(}\"\n        #playlist_regex=\"${playlist_regex//\\)/\\\\)}\"\n                       #awk \"BEGIN{IGNORECASE=1} /${playlist_regex//\\//\\\\/}/ {print \\$1; exit}\" || :)\"\n        playlist_id=\"$(SPOTIFY_PLAYLISTS_ALL=1 \"$srcdir/spotify_playlists.sh\" \"$@\" |\n                        if [ \"${SPOTIFY_PLAYLIST_EXACT_MATCH:-}\" ]; then\n                            # do not tr [:upper:] [:lower:] as this invalidates the ID which is case insensitive\n                            # save last case sensitive setting, ignore return code which will error if not already set\n                            last_nocasematch=\"$(shopt -p nocasematch || :)\"\n                            shopt -s nocasematch\n                            while read -r id name; do\n                                if [[ \"$name\" = \"$playlist_name\" ]]; then\n                                   echo \"$id\"\n                                   break\n                                fi\n                            done\n                            # restore last case sensitive setting\n                            eval \"$last_nocasematch\"\n                        else\n                            grep -Fi -m1 \"$playlist_name\" |\n                            awk '{print $1}'\n                        fi || :\n        )\"\n        if is_blank \"$playlist_id\"; then\n            echo \"Error: failed to find playlist ID matching given playlist name '$playlist_name'\" >&2\n            exit 1\n        fi\n        if ! is_spotify_playlist_id \"$playlist_id\"; then\n            die \"ERROR: playlist id '$playlist_id' does not match expected regex\"\n        fi\n        echo \"$playlist_id\"\n    fi\n}\n\nspotify_token\n\nif [ $# -gt 0 ]; then\n    playlist_name=\"$1\"\n    shift || :\n    playlist_name_to_id \"$playlist_name\" \"$@\"\nelse\n    while read -r playlist_name; do\n        playlist_name_to_id \"$playlist_name\" \"$@\"\n    done\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_snapshot_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: harisekhon\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-03 16:25:30 -0300 (Tue, 03 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the Snapshot ID of a given Spotify playlist\n\nUsed by Spotify backup scripts like blacklisted_artists.sh in HariSekhon/Spotify-Playlists\nto skip re-downloading an up to date playlist\n\nIf you pass 'liked' then retrieves the Last Added timestamp which is the equivalent for 'Liked Songs'\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<playlist_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nplaylist=\"$1\"\nif [ \"$playlist\" = liked ] || [ \"$playlist\" = saved ]; then\n    playlist=\"Liked Songs\"\nfi\nliked(){\n    [ \"$playlist\" = \"Liked Songs\" ]\n}\n\nplaylist_id=\"${2:-}\"\n\nshift || :\nshift || :\n\nspotify_user\n\nif liked; then\n    export SPOTIFY_PRIVATE=1\nfi\n\nspotify_token\n\nSECONDS=0\n\nif liked; then\n    \"$srcdir/spotify_api.sh\" \"/v1/me/tracks?limit=1\" |\n    jq -r '.items[0].added_at'\nelse\n    if is_blank \"$playlist_id\"; then\n        playlist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n    fi\n    # optimization to pull only the fields we need without the first 100 tracks\n    \"$srcdir/spotify_api.sh\" \"/v1/playlists/$playlist_id?fields=snapshot_id\" |\n    jq -r '.snapshot_id'\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_to_filename.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: < ../playlists/playlists.txt\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 17:39:04 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nNormalizes a Spotify playlist name provided as arg(s) or stdin to a valid filename\n\nReplaces invalid / dangerous characters with underscores\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name>\"\n\nhelp_usage \"$@\"\n\n#normalize(){\n#    # replace forward slash with unicode version so we can store playlist files that look like the real thing\n#    # but avoid the breakage caused by directory separator\n#    tr '/' '∕'\n#    #tr '/[:space:]' '_'\n#    # requires Perl 5.10+\n#    #perl -pe 's/[\\h\\/]/_/g'\n#    #perl -pe 's/!//g'\n#    #perl -pe 's/[^\\w\\v-]/_/g'\n#}\n\nsanitize_filename() {\n    # replace forward slash with unicode version so we can store playlist files that look like the real thing\n    # but avoid the breakage caused by directory separator\n    tr '/' '∕' |\n    perl -CS -Mutf8 -p -e '\n        s{[\\x00-\\x09\\x0B-\\x1F\\x7F]}{_}g;  # control chars except \\n\n        s{[^\\p{Print}\\p{Emoji}\\n]}{_}g;   # non-printable, non-emoji, keep \\n\n        s/\\s+$/\\n/;                       # no trailing spaces, happens in emoji suffix playlists\n    ' |\n    if is_windows; then\n        perl -CS -Mutf8 -p -e '\n            s{[\\\\/:*?\"<>|]}{_}g;  # Windows-invalid filename characters\n            s/[\\s.]+$/\\n/;        # trailing space or dot not valid on Windows\n        '\n    else\n        cat\n    fi\n}\n\nif not_blank \"$*\"; then\n    sanitize_filename <<< \"$*\"\nelse\n    sanitize_filename  # from stdin\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_top_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Favourites 💯 😎\"\n#  args: 3iRkPfmGAPH9zOrOwPOibk\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-20 23:02:35 -0500 (Tue, 20 Jan 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the top artists for a given Spotify playlist by counting unique track names for each artist\n\nIf HariSekhon/Spotify-tools is in the \\$PATH it uses normalize_tracknames.pl for greater accuracy to\ncollapse multiple versions such as Radio Edit and Album Version to only count that same song once\n\nPlaylist argument can be a playlist name or a playlist ID (get this from spotify_playlists.sh)\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\n\nOutput format:\n\n<unique_track_count> <artist>\n<unique_track_count> <artist2>\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\nprint_output(){\n    jq -r '\n      .items[]\n      | .track\n      | select(.name and .artists)\n      | .name as $track\n      | .artists[]\n      | [.name, $track]\n      | @tsv\n    ' <<< \"$output\"\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    print_output\n    # slow down a bit to try to reduce hitting Spotify API rate limits and getting HTTP 429 Too Many Requests on large playlists\n    #sleep 0.1\ndone |\n# if HariSekhon/Spotify-tools is in the \\$PATH, use this to deduplicate variations of the same track\nif type -P normalize_tracknames.pl &>/dev/null; then\n    normalize_tracknames.pl\nelse\n    cat\nfi |\nsort -u |\nif [ -n \"${DEBUG_ARTIST_TRACKS:-}\" ]; then\n    cat\nelse\n    cut -f1 |\n    sed '/^[[:space:]]*$/d' |\n    sort |\n    uniq -c |\n    sort -nr\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC2154\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns track names in a given Spotify playlist\n\nPlaylist argument can be a playlist name or a playlist ID (get this from spotify_playlists.sh)\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\n\nOutput format:\n\nArtist - Track\n\nor if \\$SPOTIFY_CSV environment variable is set then:\n\n\\\"Artist\\\",\\\"Track\\\"\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\nprint_output(){\n    # If you set \\$SPOTIFY_PLAYLIST_TRACKS_UNAVAILABLE=1 then will only output tracks that are unavailable (greyed out on Spotify)\n    # Can feed this in to spotify_delete_from_playlist.sh to crop them from TODO / Discover Backlog type playlists\n    #if [ -n \"${SPOTIFY_PLAYLIST_TRACKS_UNAVAILABLE:-}\" ]; then\n        # XXX: this isn't reliable, some tracks are still available when these fields are both empty :-/\n        # and debug dumps comparing tracks shows there are no other fields to differentiate whether a track is available or not\n    #    jq -r '.items[] | select(.track.uri) | select((.track.available_markets | length) == 0) | select((.track.album.available_markets | length) == 0)' <<< \"$output\"\n    #else\n    if not_blank \"${SPOTIFY_CSV:-}\"; then\n        jq -r '\n            .items[].track |\n            [ ( [ .artists[]?.name ] | join(\", \") ), .name ] |\n            @csv\n        '\n    else\n        jq -r '\n            .items[].track |\n            [ ( [ .artists[]?.name ] | join(\", \") ), \"-\", .name ] |\n            @tsv\n        '\n    fi <<< \"$output\" |\n    tr '\\t' ' ' |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    print_output\n    # slow down a bit to try to reduce hitting Spotify API rate limits and getting HTTP 429 Too Many Requests on large playlists\n    #sleep 0.1\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns track URIs for the given Spotify playlist\n\nPlaylist argument can be a playlist name or ID (see spotify_playlists.sh)\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\nSpotify track URIs can be used:\n- as backups to restore a playlist's contents\n- copied to new playlists\n- set to Liked (spotify_set_tracks_uri_to_liked.sh)\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\nprint_output(){\n    #jq -r '.' <<< \"$output\"\n\n    # If you set \\$SPOTIFY_PLAYLIST_TRACKS_UNAVAILABLE=1 then will only output tracks that are unavailable (greyed out on Spotify)\n    # Can feed this in to spotify_delete_from_playlist.sh to crop them from TODO / Discover Backlog type playlists\n    #if [ -n \"${SPOTIFY_PLAYLIST_TRACKS_UNAVAILABLE:-}\" ]; then\n        # XXX: this isn't reliable, some tracks are still available when these fields are both empty :-/\n        # and debug dumps comparing tracks shows there are no other fields to differentiate whether a track is available or not\n    #    jq -r '.items[] | select(.track.uri) | select((.track.available_markets | length) == 0) | select((.track.album.available_markets | length) == 0)' <<< \"$output\"\n    #else\n    jq -r '.items[] | select(.track.uri) | .track.uri' <<< \"$output\"\n    #fi\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    print_output\n    # slow down a bit to try to reduce hitting Spotify API rate limits and getting 429 errors on large playlists\n    #sleep 0.1\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks_uri_artist_track.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns track URIs and 'Artist - Track' names in a given Spotify playlist\n\nPlaylist argument can be a playlist name or a playlist ID (get this from spotify_playlists.sh)\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\n\nOutput format:\n\nURI \\\\t Artist - Track\n\n\n$usage_playlist_help\n\n\nUsed by spotify_backup_playlist.sh instead of spotify_playlist_tracks.sh and spotify_playlist_tracks_uri.sh separately\nbecause this halves tbe number of API calls to postprocess rather than fetch separate lists - this is a big\noptimization given I have nearly 100,000 tracks under management which requires a lot of paging\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\n\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\noutput(){\n    jq -r '\n        .items[].track |\n        [ .uri, ( [.artists[]?.name] | join(\", \") ), \"-\", .name ] |\n        @tsv\n    ' <<< \"$output\" |\n    sed $'s/\\t/|/' |\n    tr '\\t' ' ' |\n    tr '|' '\\t' |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//;\n        /^[[:space:]]*$/d;\n    '\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks_uri_batch_by_year.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-10 23:44:07 -0300 (Tue, 10 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns all track URIs from the given Spotify playlist(s) grouped by year or decade\n\nCopies each batch to the clipboard, prints to stdout, and prompts to continue\nbefore printing the next batch\n\nSet the environment variable TRACK_URIS_BY_DECADE to any value for decade batching\n\nUseful for filtering tracks to add to my best of each year or decade playlists\n\nPlaylist argument can be a playlist name or ID (see spotify_playlists.sh)\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> [<playlist2> <playlist3>]\"\n\nhelp_usage \"$@\"\n\nif [ $# -eq 0 ]; then\n    usage \"playlist not defined\"\nfi\n\nspotify_token\n\ntmpfile=\"$(mktemp)\"\ntrap_cmd \"rm -f \\\"$tmpfile\\\"\"\n\n# collect to tmpfile:\n#\n#   year    canonical_id    release_date    uri\n#\ncollect_output() {\n    jq -r '\n        .items[]\n        | select(.track?.uri)\n        | .track as $t\n        | ($t.album.release_date // \"\") as $rd\n        | select($rd | length >= 4)\n        | ($rd[0:4]) as $year\n        | select($year | test(\"^[0-9]{4}$\"))\n        | ($t.linked_from.id // $t.id) as $cid\n        | \"\\($year)\\t\\($cid)\\t\\($rd)\\t\\($t.uri)\"\n    ' <<< \"$output\" >> \"$tmpfile\"\n}\n\nprocess_playlist(){\n    local playlist=\"$1\"\n    timestamp \"Processing playlist: $playlist\"\n    playlist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n    # $offset defined in lib/spotify.sh\n    # shellcheck disable=SC2154\n    url_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\n    while not_null \"$url_path\"; do\n        output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\")\"\n        url_path=\"$(get_next \"$output\")\"\n        collect_output\n        # slow down a bit to try to reduce hitting Spotify API rate limits and getting 429 errors on large playlists\n        sleep 0.1\n    done\n}\n\nfor arg; do\n    process_playlist \"$arg\"\ndone\necho\n\nif [ -n \"${TRACK_URIS_BY_DECADE:-}\" ]; then\n    grouped=\"$(\n        awk -F'\\t' '\n            {\n                decade = substr($1,1,3) \"0s\";\n                print decade \"\\t\" $2 \"\\t\" $3 \"\\t\" $4\n            }\n        ' \"$tmpfile\" |\n        sort -k2,2 -k3,3 |\n        sort -u -k2,2 |\n        sort -k1,1 |\n        cut -f1,4\n    )\"\nelse\n    grouped=\"$(\n        sort -k2,2 -k3,3 \"$tmpfile\" |\n        sort -u -k2,2 |\n        sort -k1,1 |\n        cut -f1,4\n    )\"\nfi\n\ncurrent=\"\"\n\nbatchfile=\"$(mktemp)\"\ntrap_cmd \"rm -f \\\"$tmpfile\\\" \\\"$batchfile\\\"\"\n\nwhile IFS=$'\\t' read -r label uri; do\n    # when we move to the next year or decade, dump the current batchfile and reset it for the next batch\n    if [ \"$label\" != \"$current\" ] && [ -n \"$current\" ]; then\n        {\n            echo \"=== $current ===\"\n            echo\n            tee >( \"$srcdir/../bin/copy_to_clipboard.sh\" ) < \"$batchfile\"\n            echo\n        } | less -F\n        printf \"Press ENTER to continue...\" >&2\n        # read < tty avoids reading from the loop stdin but requires interactive TTY support\n        # otherwise we need to do the trick from my doc here which is a bit more exec tricky:\n        #\n        # https://github.com/HariSekhon/Knowledge-Base/blob/main/bash.md#wait-for-a-terminal-prompt-from-inside-a-while-loop\n        #\n        # need to handle SIGINT explicitly to allow Control-C from this read < tty\n        trap 'echo; exit 130' INT\n        read -r _ < /dev/tty || exit 130\n        echo\n        # clear batchfile\n        : > \"$batchfile\"\n    fi\n\n    current=\"$label\"\n    printf '%s\\n' \"$uri\" >> \"$batchfile\"\ndone <<< \"$grouped\"\n\n# Final batch\nif [ -s \"$batchfile\" ]; then\n    {\n        echo\n        echo \"=== $current ===\"\n        echo\n        tee >( \"$srcdir/../bin/copy_to_clipboard.sh\" ) < \"$batchfile\"\n        echo\n    } | less -F\nfi\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks_uri_by_year.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Upbeat & Sexual Pop\"\n#  args: 64OO67Be8wOXn6STqHxexr\n#\n#  Author: Hari Sekhon\n#  Date: 2026-01-29 18:40:46 -0400 (Thu, 29 Jan 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns track URIs from the given Spotify playlist for a specific year or range of years\n\nUseful for filtering tracks to add to my best of each decade playlists\n\nPlaylist argument can be a playlist name or ID (see spotify_playlists.sh)\n\nThe year can be one of:\n\n- an integer\n- a year range in the format '<start>-<end>' eg. '2000-2009'\n- an entire decade such as '1980s', '1990s', '2000s', '2010s', '2020s'\n- a decade range such as '1990s-2000s'\n\n\\$SPOTIFY_PLAYLIST can be used from environment if no first argument is given\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> <year_or_range> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nplaylist_id=\"${1:-${SPOTIFY_PLAYLIST:-}}\"\nyear_arg=\"${2:-}\"\nshift || :\nshift || :\n\nif is_blank \"$playlist_id\"; then\n    usage \"playlist not defined\"\nfi\n\nif is_blank \"$year_arg\"; then\n    usage \"year or range not defined\"\nfi\n\ndecade_to_range() {\n    local decade=\"$1\"\n    local start=\"${decade%s}\"   # remove trailing 's'\n    echo \"$start $((start + 9))\"\n}\n\nif [[ \"$year_arg\" =~ ^([0-9]{4})$ ]]; then\n    # Single year\n    year_start=\"$year_arg\"\n    year_end=\"$year_arg\"\nelif [[ \"$year_arg\" =~ ^([0-9]{4})-([0-9]{4})$ ]]; then\n    # Year range\n    year_start=\"${BASH_REMATCH[1]}\"\n    year_end=\"${BASH_REMATCH[2]}\"\nelif [[ \"$year_arg\" =~ ^([0-9]{4})s$ ]]; then\n    # Single decade\n    read -r year_start year_end < <(decade_to_range \"$year_arg\")\nelif [[ \"$year_arg\" =~ ^([0-9]{4}s)-([0-9]{4}s)$ ]]; then\n    # Decade range\n    read -r year_start _ < <(decade_to_range \"${BASH_REMATCH[1]}\")\n    read -r _ year_end < <(decade_to_range \"${BASH_REMATCH[2]}\")\nelse\n    usage \"invalid year or range: '$year_arg'\"\nfi\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist_id\" \"$@\")\"\n\n# $offset defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\nprint_output(){\n    #jq -r '.items[] | select(.track.uri) | .track.uri' <<< \"$output\"\n    # filter tracks by release year, works for singles, EPs, albums\n    jq -r --arg start \"$year_start\" --arg end \"$year_end\" '\n        .items[]\n        | select(.track.uri)\n        | select(.track.album.release_date | test(\"^[0-9]{4}\"))\n        | .track as $t\n        | ($t.album.release_date[0:4] | tonumber) as $year\n        | select($year >= ($start | tonumber) and $year <= ($end | tonumber))\n        | $t.uri\n    ' <<< \"$output\"\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    print_output\n    # slow down a bit to try to reduce hitting Spotify API rate limits and getting 429 errors on large playlists\n    #sleep 0.1\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_tracks_uri_in_year.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: \"The 70s\" 197.\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-18 12:07:54 +0000 (Wed, 18 Nov 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-playlist/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the URIs of all tracks in a playlist which originated in the given year regex\n\nUseful to find tracks from a given year or decade from existing playlists to create more specialised playlists\n\nThe regex is an anchored BRE format regex. Typically you'll want to provide a simple 4-digit year or 3 digits and a dot for a decade\n\n\nExample:\n\nThis returns everything in MyPlaylist that had an original release date in the 1980s:\n\n    ${0##*/} MyPlaylist 198.\n\nEven if the track in the playlist is from a compilation released later, it will find the original release date by doing\na targeted search for that track using the Spotify API\n\n\nCaveat: not every single track is findable in the Spotify Search API - there are some rare edge cases where certain versions won't be found even with broader simpler searches (eg. Nadia Ali  Call My Name Spencer Hill Radio Edit).\n        Such tracks are now skipped rather than raising errors as it's better to take the 99% success rate than deal with the rare data issues in the Spotify API\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> <year_regex> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nplaylist=\"$1\"\nyear_regex=\"$2\"\nshift || :\nshift || :\n\nif is_blank \"$playlist\"; then\n    usage \"playlist not defined\"\nfi\n\nif is_blank \"$year_regex\"; then\n    usage \"year regex not defined\"\nfi\n\n# allow filtering private playlists\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    # XXX: very important - do not quote, this only works in very simple cases and doesn't work in artist:\"quoted\" track:\"quoted\" or even \"artist:blah track:blah\" formats\n    # quotes are part of the data in track names, so this is highly breakable, using artist: track: is the more specific method anyway, just stick with that\n    jq -r '.items[].track |\n           [ .uri,\n             \"artist:\" + ([.artists[].name] | join(\" \")),\n             \"track:\" + .name\n           ] | @tsv' <<< \"$output\" |\n    while read -r uri search_criteria; do\n        #log \"searching for year - $search_criteria\"\n        artist_search_terms=\"${search_criteria%%track:*}\"\n        artist=\"${artist_search_terms#artist:}\"\n        track=\"${search_criteria##*track:}\"\n        log \"searching for year for:  $artist - $track\"\n        # Spotify API can handle single quotes (strips them out) but breaks if you URL encode them! So remove them rather than URL encode\n        artist=\"${artist//\\'/}\"\n        # to be able to query artists with unicode characters eg. Blue Öyster Cult\n        artist=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$artist\")\"\n        # strip single quotes as raw single quotes qork but url encoding single quotes breaks the API\n        track=\"${track//\\'/}\"\n        # requires HariSekhon/Spotify-tools to be in the \\$PATH\n        track=\"$(normalize_tracknames.pl <<< \"$track\" | \"$srcdir/../bin/urlencode.sh\")\"\n        # XXX: this track isn't found with but is if you leave off the artist: and track: prefixes then you find the other versions of it which are found in the API - this is a better trade off than finding nothing\n        #\n        #      artist:Nadia track:Call My Name\n        #\n        #      Sultan,Sultan + Shepard,Nadia Ali - Call My Name - Spencer & Hill Remix\n        #\n        log \"searching for year - $artist - $track\"\n        #year=\"$(\"$srcdir/spotify_release_year.sh\" \"$artist\" \"$track\")\"\n        # XXX: hack to ignore fringe cases where search doesn't find anything due to weird characters or encoding issues or edge cases like 'Nadia Ali      Call My Name (Spencer & Hill Radio Edit) (feat. Nadia Ali)' which doesn't appear in even basic searches such as 'Nadia Ali   Call My Name Radio Edit') even though other versions do\n        year=\"$(\"$srcdir/spotify_release_year.sh\" \"$artist\" \"$track\" || :)\"\n        if [ -z \"$year\" ]; then\n            log \"failed to find year for:  $artist - $track\"\n            if [ -n \"${DEBUG:-}\" ]; then\n                echo >&2\n            fi\n            continue\n        fi\n        log \"got year $year\"\n        if [[ \"$year\" =~ ^$year_regex$ ]]; then\n            echo \"$uri\" |\n            if [ -n \"${DEBUG:-}\" ]; then\n                \"$srcdir/spotify_uri_to_name.sh\"\n                echo >&2\n            else\n                cat\n            fi\n        fi\n    done\ndone\n"
  },
  {
    "path": "spotify/spotify_playlist_uri_offset.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: Dance∕Pop∕House∕Trance∕DnB∕Electronica∕Gym spotify:track:5Vy3sdJZJ6AnDChSfNtKs8\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-19 21:23:19 +0000 (Thu, 19 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nPrints the offset of a given spotify track URI in a given playlist\n\nUseful to find the offset to use to continue processing a large partially processed playlist because Spotify API tokens are only valid for 1 hour and some playlists with several thousand tracks take longer than that to process eg. spotify_playlist_tracks_uri_in_year.sh)\n\nPlaylist can be given as a name or id\n\nURI must be in the form of spotify:track:uri as this is the native format in the playlist we are looking for\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist> <spotify:track:uri> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nplaylist=\"$1\"\nuri=\"$2\"\nshift || :\nshift || :\n\nif is_blank \"$playlist\"; then\n    usage \"playlist not defined\"\nfi\n\nif is_blank \"$uri\"; then\n    usage \"URI not defined\"\nfi\n\n# discard the id returned\nvalidate_spotify_uri \"$uri\" >/dev/null\n\nif ! [[ \"$uri\" =~ ^spotify:track:[[:alnum:]]+$ ]]; then\n    usage \"URI must be in spotify:track:xxxxx... format\"\nfi\n\nplaylist_id=\"$(\"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\" \"$@\")\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id/tracks?limit=100&offset=$offset\"\n\noutput(){\n    jq -r '.items[] | select(.track.uri) | .track.uri' <<< \"$output\"\n}\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    jq -r '.items[].track.uri' <<< \"$output\" |\n    while read -r this_uri; do\n        if [ \"$this_uri\" = \"$uri\" ]; then\n            echo \"$offset\"\n            exit 0\n        fi\n        ((offset += 1))\n    done\n    url_path=\"$(get_next \"$output\")\"\ndone\nexit 1\n"
  },
  {
    "path": "spotify/spotify_playlists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 09:30:53 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-list-users-playlists/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the list of Spotify playlists\n\nOutput Format:\n\n<playlist_id>   <playlist_name>\n\nIf SPOTIFY_PLAYLIST_SNAPSHOT_ID=1 environment variable is set, then returns a 2nd column with the snapshot ID.\nThis addition was made as an optimization to avoid having to do an extra API call per playlist when running\nspotify_backup_playlist.sh / spotify_backup_playlists.sh\n\n\\$SPOTIFY_USER must be defined in environment or given as first arg unless \\$SPOTIFY_PRIVATE=1 is set,\nin which case it's inferred from the auth token\n\nBy default the Spotify API returns only public playlists owned by given Spotify user and that have been explicitly\nadded to their profile\n\nThis is counter-intuitive\n\nTo get the public playlists not explicitly added to a user's profile you need to use private API access mode (SPOTIFY_PRIVATE=1)\n\nTo get all playlists including private playlists - export SPOTIFY_PRIVATE=1\nTo get only private playlists                    - export SPOTIFY_PRIVATE_ONLY=1 (implicitly adds SPOTIFY_PRIVATE=1)\nTo get only public playlists                     - export SPOTIFY_PUBLIC_ONLY=1\nTo get all public playlists including those not added to profile - export SPOTIFY_PRIVATE=1 SPOTIFY_PUBLIC_ONLY=1\n\nTo also get followed playlists - export SPOTIFY_PLAYLISTS_FOLLOWED=1\nTo get only followed playlists - export SPOTIFY_PLAYLISTS_FOLLOWED_ONLY=1\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<spotify_user> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif [ -n \"${SPOTIFY_PLAYLISTS_FOLLOWED_ONLY:-}\" ]; then\n    export SPOTIFY_PLAYLISTS_FOLLOWED=1\nfi\n\nif [ -n \"${SPOTIFY_PRIVATE_ONLY:-}\" ]; then\n    export SPOTIFY_PRIVATE=1\nfi\n\nif not_blank \"${SPOTIFY_PUBLIC_ONLY:-}\" &&\n   not_blank \"${SPOTIFY_PRIVATE_ONLY:-}\"; then\n    die 'ERROR: Cannot set both SPOTIFY_PUBLIC_ONLY and SPOTIFY_PRIVATE_ONLY environment variables - they are mutually exclusive!'\nfi\n\n# because otherwise the Spotify API only returns the public playlists that have been explicitly added to the profile\n#if [ -n \"${SPOTIFY_PUBLIC_ONLY:-}\" ]; then\n#    export SPOTIFY_PRIVATE=1\n#fi\n\nspotify_user=\"${1:-${SPOTIFY_USER:-}}\"\n\n# will infer from token if $SPOTIFY_PRIVATE=1\nspotify_user\n\nif is_blank \"$spotify_user\"; then\n    # /v1/me/playlists gets an authorization error and '/v1/users/me/playlists' returns the wrong user, an actual literal user called 'me'\n    #user=\"me\"\n    usage \"user not specified\"\nfi\n\nshift || :\n\n# redirects to fetching your own playlists instead of the named user if you happen to have the a stronger spotify private access token\n#if not_blank \"${SPOTIFY_PRIVATE:-}\"; then\n#    # /v1/me/playlists gets an authorization error and '/v1/users/me/playlists' returns the wrong user, an actual literal user called 'me'\n#    # $limit/$offset defined in lib/spotify.sh\n#    # shellcheck disable=SC2154\n#    url_path=\"/v1/me/playlists?limit=$limit&offset=0\"  # never use $offset - it will prevent playlists being found\n#else\n    # $limit/$offset defined in lib/spotify.sh\n    # shellcheck disable=SC2154\n    url_path=\"/v1/users/$spotify_user/playlists?limit=$limit&offset=0\"  # never use $offset\n#fi\n\noutput(){\n    jq '.items[]' <<< \"$output\" |\n    if not_blank \"${SPOTIFY_PUBLIC_ONLY:-}\"; then\n        jq 'select(.public == true)'\n    elif not_blank \"${SPOTIFY_PRIVATE_ONLY:-}\"; then\n        jq 'select(.public != true)'\n    else\n        cat\n    fi |\n    if [ -n \"${SPOTIFY_PLAYLISTS_FOLLOWED_ONLY:-}\" ]; then\n        jq \"select(.owner.id != \\\"$spotify_user\\\")\"\n    elif [ -n \"${SPOTIFY_PLAYLISTS_FOLLOWED:-}\" ]; then\n        cat\n    else\n        jq \"select(.owner.id == \\\"$spotify_user\\\")\"\n    fi |\n    if [ -n \"${SPOTIFY_PLAYLIST_SNAPSHOT_ID:-}\" ]; then\n        jq -r \"[.id, .snapshot_id, .name] | @tsv\"\n    else\n        jq -r \"[.id, .name] | @tsv\"\n    fi |\n    sed 's/[[:space:]]*$//'\n}\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone |\n# dedupe by playlist ID in the first column as there are occasional duplicates returned by Spotify API\nawk '!seen[$1]++'\n"
  },
  {
    "path": "spotify/spotify_playlists_json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-24 01:17:21 +0100 (Wed, 24 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/playlists/get-list-users-playlists/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC2154\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the list of Spotify public playlists in raw JSON format for the given Spotify user\n\n\\$SPOTIFY_USER can be used from the evironment if no first argument is given\n\nCaveat: limited to 50 public playlists due to Spotify API, must specify OFFSET=50 to get next 50.\n        This script does not iterate each page automatically because the output would be nonsensical\n        multiple json outputs so you must iterate yourself and process each json result in turn\n        For an example of how to do this and process multiple paged requests see spotify_playlists.sh\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<spotify_username> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nif not_blank \"${SPOTIFY_PRIVATE:-}\"; then\n    # $limit/$offset defined in lib/spotify.sh\n    # shellcheck disable=SC2154\n    url_path=\"/v1/me/playlists?limit=$limit&offset=$offset\"\nelse\n    user=\"${1:-${SPOTIFY_USER:-}}\"\n    if is_blank \"$user\"; then\n        # /v1/me/playlists gets an authorization error and '/v1/users/me/playlists' returns the wrong user, an actual literal user called 'me'\n        #user=\"me\"\n        usage \"user not specified\"\n    fi\n    # $limit/$offset defined in lib/spotify.sh\n    # shellcheck disable=SC2154\n    url_path=\"/v1/users/$user/playlists?limit=$limit&offset=$offset\"\nfi\n\nshift || :\n\n\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\"\n"
  },
  {
    "path": "spotify/spotify_release_year.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-17 17:09:17 +0000 (Tue, 17 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds the original release year of a given track or album via a search query of the top 10 results and taking the oldest release date\n\nEspecially useful to find the original dates of songs that get re-released in 'Greatest Hits' type albums\n(eg. use in script to find songs from X decade regardless of which copy of the songs are in your playlist)\n\nThe search must be as specific as possible for accurate results and should include both the artist name as well as the track/album,\npreferably with the artist specified as artist:<name> eg.\n\n    <track_name> artist:<artist_name>\n\n    <album_name> artist:<artist_name>\n\nExamples:\n\n    Tracks:\n\n        ${0##*/} sweet harmony artist:the beloved\n\n        ${0##*/} artist:the beloved track:sweet harmony\n\n        ${0##*/} kylie on a night like this\n\n    Albums:\n\n        SPOTIFY_SEARCH_TYPE=album ${0##*/} happiness artist:the beloved\n\n        SPOTIFY_SEARCH_TYPE=album ${0##*/} artist:the beloved album:happiness\n\nFor more details on search query syntax see spotify_search_json.sh\n\n\nCaveat: this is only as accurate as Spotify's data which is usually fairly good, but if Spotify has only managed to license a song via a later compilation album then we can only report the earliest release date which may not be the real original release for a classic. An example of this is\n\n    artist:\\\"Carl Douglas\\\"  track:\\\"Kung Fu Fighting\\\"\n\nWhich I know is a 70s song but Spotify only has it on compilations dated 2001 onwards, leading to an incorrect result in that rare case.\n\nIn a rare case you may need to tune the number of results from which the original date is inferred to more than 10 using the SPOTIFY_SEARCH_LIMIT environment variable but in my testing the first 10 results have always contained the original version from which to infer the oldest date,\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<track_name> artist:<artist>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nexport SPOTIFY_SEARCH_LIMIT=\"${SPOTIFY_SEARCH_LIMIT:-10}\"\n\n\"$srcdir/spotify_search_json.sh\" \"$@\" |\nif [ \"${SPOTIFY_SEARCH_TYPE:-track}\" = track ]; then\n    jq -r \".tracks.items[] | .album.release_date\"\nelif [ \"${SPOTIFY_SEARCH_TYPE:-}\" = album ]; then\n    jq -r \".albums.items[] | .release_date\"\nelse\n    die \"unsupported SPOTIFY_SEARCH_TYPE='$SPOTIFY_SEARCH_TYPE' - must be either track or album\"\nfi |\n# lexical oldest of YYYY or YYYY-mm-dd will be first\nsort |\nhead -n1 |\n# a few release dates have higher granularity, strip everything after the year YYYY-mm-dd\nsed 's/-.*$//' |\n# this is only here to make the exit code error 1 if nothing is found\ngrep --color=no .\n"
  },
  {
    "path": "spotify/spotify_rename_playlist.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 23:26:15 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRenames a given Spotify playlist\n\nThe Spotify playlist to rename can be identified by name or ID\n\n\n$usage_playlist_help\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> <new_playlist_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nplaylist=\"$1\"\n\nnew_playlist_name=\"$2\"\nnew_playlist_name=\"${new_playlist_name//\\\"}\"\n\n# modifying a playlist requires an authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\n# this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\nplaylist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\nplaylist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\")\"\n\n# shellcheck disable=SC2154\nurl_path=\"/v1/playlists/$playlist_id\"\n\ntimestamp \"renaming playlist '$playlist_name' to '$new_playlist_name'\"\n\"$srcdir/spotify_api.sh\" \"$url_path\" -X PUT -H \"Content-Type: application/json\" -d \"{ \\\"name\\\": \\\"$new_playlist_name\\\" }\"\n"
  },
  {
    "path": "spotify/spotify_search.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Foo Fighters\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-05 14:33:55 +0100 (Sun, 05 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/search/search/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches the Spotify API and returns the first N tracks / artists / albums that match the given search expression\n\nSee this page for documentation on how to write query expressions:\n\n    https://developer.spotify.com/documentation/web-api/reference/search/search/\n\nExamples:\n\nFind tracks called 'arlandria' by artist 'foo fighters':\n\n    ${0##*/} artist:foo fighters track:arlandria\n\nFind top 5 matching artists with 'foo' in the name:\n\n    SPOTIFY_SEARCH_TYPE=artist SPOTIFY_SEARCH_LIMIT=5 ${0##*/} foo\n\n\nNon-ASCII characters will cause the Spotify API to break with HTTP 400 errors - you must URL encode them first\neg. the artist Sébastien Tellier. You can use the ../bin/urlencode.sh script on the term first\n\n\nUses the adjacent script spotify_search_json.sh which supports the following environment variable options:\n\n\\$SPOTIFY_SEARCH_TYPE  = track # default\n                        artist\n                        album\n\n\\$SPOTIFY_SEARCH_LIMIT = 1 # default\n\n\\$SPOTIFY_SEARCH_OFFSET = 0 # default\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"'<search_expression>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/spotify_search_json.sh\" \"$@\" |\n\nif [ \"${SPOTIFY_SEARCH_TYPE:-}\" = \"artist\" ]; then\n    jq -r '.artists.items[].name'\nelif [ \"${SPOTIFY_SEARCH_TYPE:-}\" = \"album\" ]; then\n    jq -r '.albums.items[] | [([.artists[].name] | join(\",\")), \"-\", .name ] | @tsv' |\n    tr '\\t' ' '\nelse\n    jq -r '.tracks.items[] | [([.artists[].name] | join(\",\")), \"-\", .name ] | @tsv' |\n    tr '\\t' ' '\nfi\n"
  },
  {
    "path": "spotify/spotify_search_alternate_track_uris.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-05 14:33:55 +0100 (Sun, 05 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/search\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns track URIs for the other tracks that have the same Artist + Track name as the given track URI(s)\nby first resolving the given URI(s) artist + track name(s), then searching for and returning the URIs of the results\n\nAccepts URIs as either args or from standard input\n\nReturns up to 10 URIs of tracks with the same Artist + Track matching\n\nUses spotify_search_json.sh - see its help for more details on search capabilities and limitations\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<track_uri_or_files>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nexport SPOTIFY_SEARCH_LIMIT=10\n\nspotify_token\n\nSPOTIFY_TSV=1 \"$srcdir/spotify_uri_to_name.sh\" \"$@\" |\nwhile IFS=$'\\t' read -r artist track; do\n    # URL encode to solve non-ASCII characters breaking the Spotify API calls eg. artist:\"Sébastien Tellier\"\n    artist=\"$(urlencode.sh <<< \"$artist\")\"\n    track=\"$(urlencode.sh <<< \"$track\")\"\n    \"$srcdir/spotify_search_uri.sh\" artist:\"$artist\" track:\"$track\"\ndone\n"
  },
  {
    "path": "spotify/spotify_search_json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Foo Fighters\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-05 14:33:55 +0100 (Sun, 05 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/search/search/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches the Spotify API and returns the first N tracks / artists / albums that match the given search expression\n\nSee this page for documentation on how to write query expressions:\n\n    https://developer.spotify.com/documentation/web-api/reference/search\n\nExample:\n\n    ./${0##*/} artist:foo fighters track:arlandria\n\n    ./${0##*/} artist:\\\"Foo Fighters\\\" track:arlandria\n\nAPI JSON is returned, used by adjacent spotify_search*.sh scripts\n\nThere is no way in Spotify API to anchor search terms, they are always fuzzy and will therefore return things like\nremixes of a given song along with its original version\n\nNon-ASCII characters will cause the Spotify API to break with HTTP 400 errors - you must URL encode them first\neg. the artist Sébastien Tellier. You can use the ../bin/urlencode.sh script on the term first\n\nEnvironment variable options:\n\n\\$SPOTIFY_SEARCH_TYPE  = track # default\n                        artist\n                        album\n\n\\$SPOTIFY_SEARCH_LIMIT = 1 # default, official API limit 10, this code truncates to max 50 to avoid API errors\n\n\\$SPOTIFY_SEARCH_OFFSET = 0 # default\n\n\nCaveat: the Spotify API returns unicode characters eg.\n\n    Blue Öyster Cult - (Don't Fear) The Reaper\n\nbut these same unicode characters when fed back in to the Spotify Search API find no entries and only work if you were feed in simple characters ie. convert the Öyster to Oyster or url encode them before passing them to this script.\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"'<search_expression>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n# let user use full query and put artist: / album: / track: prefixes themselves\n#if [ $# -gt 1 ]; then\n#    artist=\"$1\"\n#    # encode spaces between search terms with %20 or +\n#    artist=\"${artist// /+}\"\n#    shift || :\n#fi\n#track=\"$*\"\n#track=\"${track// /+}\"\n\n# this substitution doesn't work on $* so have to do it in 2 steps\nsearch_terms=\"$*\"\nshopt -s extglob\n# replace multiple spaces with a single plus, requires extglob\nsearch_terms=\"${search_terms//+([[:space:]])/%20}\"\n\n# quotes break search unless urlencoded - but url encoding the entire search string using urlencode.sh breaks everything so just replace the quotes\n# XXX: also quoting only seems to work for simple queries, not highly specific ones like artist:blah track:blah which don't seem to work whether you try artist:\"blah\" track:\"blah\" or \"artist:blah track:blah\" - so best to leave them off\n# XXX: this escaping is necessary for when tracks have quotes inside their names - ie. part of the data, not part of the search tricks\nsearch_terms=\"${search_terms//\\\"/%22}\"\n\n# looks like rather than URL encoding single quotes the Spotify API strips them out - replacing with urlencoded fails to match, but if you strip them out or just leave them in for the Spotify API itself to strip out then it returns the correct results\n#search_terms=\"${search_terms//\\'/%27}\"\n#search_terms=\"${search_terms//\\'/}\"\n\n# brackets also work in searches - don't get complicated here\n#search_terms=\"${search_terms//[^[:alnum:][:space:]:%-]/}\"\n\nspotify_token\n\nsearch_type=\"${SPOTIFY_SEARCH_TYPE:-track}\"\nif [ -n \"${SPOTIFY_SEARCH_LIMIT:-}\" ]; then\n    limit=\"$SPOTIFY_SEARCH_LIMIT\"\n    if ! [[ \"$limit\" =~ ^[[:digit:]]+$ ]]; then\n        echo \"Invalid \\$SPOTIFY_SEARCH_LIMIT = $limit found in environment\" >&2\n        exit 1\n    fi\nelif [ -n \"${DEBUG:-}\" ]; then\n    limit=20\nelse\n    limit=1\nfi\n\noffset=\"${SPOTIFY_SEARCH_OFFSET-0}\"\n\n# causes errors if trying to query more than the max limit of 50\nif [ \"$limit\" -gt 50 ]; then\n    limit=50\nfi\n\nurl_path=\"/v1/search?type=$search_type&offset=$offset&limit=$limit\"\n\n# \"Access token has no market information\" - can't see how to set this in docs\n#if [ -z \"${NO_MARKET:-}\" ]; then\n#    # restrict to tracks available to the local market - this is more useful for being able to immediately use the returned URI to put in a playlist\n#    url_path+=\"&market=from_token\"\n#fi\n\n# quoting will preserve order of terms but will probably not work when putting artist and track name in the args\n#url_path+=\"&q=\\\"$search_terms\\\"\"\nurl_path+=\"&q=$search_terms\"\n#url_path+=\"&q=artist:$artist+track:$track\"\n\n\"$srcdir/spotify_api.sh\" \"$url_path\"\n"
  },
  {
    "path": "spotify/spotify_search_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Foo Fighters\" | tee /dev/stderr | spotify_uri_to_name.sh\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-05 14:33:55 +0100 (Sun, 05 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/search/search/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSearches the Spotify API and returns the Spotify URIs for the first N tracks / artists / albums that match the given search expression\n\nSee this page for documentation on how to write query expressions:\n\n    https://developer.spotify.com/documentation/web-api/reference/search\n\nExample:\n\n./${0##*/} artist:foo fighters track:arlandria\n\n\nUses the adjacent script spotify_search_json.sh which supports the following environment variable options:\n\n\\$SPOTIFY_SEARCH_TYPE  = track # default\n                        artist\n                        album\n\n\\$SPOTIFY_SEARCH_LIMIT = 1 # default\n\n\\$SPOTIFY_SEARCH_OFFSET = 0 # default\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"'<search_expression>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/spotify_search_json.sh\" \"$@\" |\n\n#if [ \"${SPOTIFY_SEARCH_TYPE:-}\" = \"artist\" ]; then\n#    jq -r '.artists.items[].uri'\n#elif [ \"${SPOTIFY_SEARCH_TYPE:-}\" = \"album\" ]; then\n#    jq -r '.albums.items[].uri'\n#else\n#    jq -r '.tracks.items[].uri'\n#fi\n\njq -r \".${SPOTIFY_SEARCH_TYPE:-track}s.items[].uri\"\n"
  },
  {
    "path": "spotify/spotify_set_playlists_private.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"Driving (Auto-generated as reverse of Upbeat & Sexual Pop)\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 23:26:15 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets given Spotify playlists to private\n\nAccepts Spotify playlist names or IDs as either arguments or on standard input, one playlist per line\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> [<playlist2> <playlist3>]\"\n\nhelp_usage \"$@\"\n\n# modifying a playlist requires an authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nset_playlist_private(){\n    local playlist=\"$1\"\n    local playlist_id\n    local playlist_name\n\n    # this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\n    playlist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n    playlist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\")\"\n\n    url_path=\"/v1/playlists/$playlist_id\"\n\n    timestamp \"setting playlist '$playlist_name' to private\"\n    \"$srcdir/spotify_api.sh\" \"$url_path\" -X PUT -H \"Content-Type: application/json\" -d \"{ \\\"public\\\": false }\"\n}\n\nif [ $# -gt 0 ]; then\n    for playlist in \"$@\"; do\n        set_playlist_private \"$playlist\"\n    done\nelse\n    while read -r playlist; do\n        set_playlist_private \"$playlist\"\n    done\nfi\n"
  },
  {
    "path": "spotify/spotify_set_playlists_public.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: \"My Shazam tracks\"\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-23 23:26:15 +0100 (Thu, 23 Jul 2020)\n#\n#  https://github.com/HariSekhon/Spotify-Playlists\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets given Spotify playlists to public\n\nAccepts Spotify playlist names or IDs as either arguments or on standard input, one playlist per line\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<playlist_name_or_id> [<playlist2> <playlist3>]\"\n\nhelp_usage \"$@\"\n\n# modifying a playlist requires an authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nset_playlist_public(){\n    local playlist=\"$1\"\n    local playlist_id\n    local playlist_name\n\n    # this script returns the ID if it's already in the correct format, otherwise queries and returns the playlist ID for the playlist\n    playlist_id=\"$(SPOTIFY_PLAYLIST_EXACT_MATCH=1 \"$srcdir/spotify_playlist_name_to_id.sh\" \"$playlist\")\"\n\n    playlist_name=\"$(\"$srcdir/spotify_playlist_id_to_name.sh\" \"$playlist_id\")\"\n\n    url_path=\"/v1/playlists/$playlist_id\"\n\n    timestamp \"setting playlist '$playlist_name' to public\"\n    \"$srcdir/spotify_api.sh\" \"$url_path\" -X PUT -H \"Content-Type: application/json\" -d \"{ \\\"public\\\": true }\"\n}\n\nif [ $# -gt 0 ]; then\n    for playlist in \"$@\"; do\n        set_playlist_public \"$playlist\"\n    done\nelse\n    while read -r playlist; do\n        set_playlist_public \"$playlist\"\n    done\nfi\n"
  },
  {
    "path": "spotify/spotify_set_tracks_uri_to_liked.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: ../playlists/spotify/Starred | spotify_uri_to_name.sh\n#  args: ../playlists/spotify/Starred | tee /dev/stderr | spotify_uri_to_name.sh\n#  args: ../playlists/spotify/Starred\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 00:13:30 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/save-tracks-user/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nSets the given list of Spotify track URIs to Liked Songs via the Spotify API\n\nTrack URIs can be given in the form of a file argument or passed via standard input\n\nSample use cases:\n- convert Spotify's previous Starred tracks to Liked Songs (Spotify change Starred a normal playlist and created the new Liked Songs)\n- mark all the songs of your favourite playlists as Liked Songs\n\nspotify_set_tracks_uri_to_liked.sh Starred\n\nOutputs the URIs it has marked as Liked Songs so that you can track the progress.\nYou can combine this with spotify_uri_to_name.sh to get human readable progress, eg:\n\nspotify_set_tracks_uri_to_liked.sh Starred | spotify_uri_to_name.sh\n\nfor debugging to see both the URIs processed and the human readable names you can do:\n\nspotify_set_tracks_uri_to_liked.sh Starred | tee /dev/stderr | spotify_uri_to_name.sh\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<filename> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nurl_path=\"/v1/me/tracks?ids=\"\n\n# requires authorized token\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\ndeclare -a ids\nids=()\n\nset_to_liked(){\n    local ids\n    # join array arg on commas\n    { local IFS=','; ids=\"$*\"; }\n    if [ -z \"$ids\" ]; then\n        return\n    fi\n    timestamp \"Liking tracks in batch:\"\n    echo >&2\n    tr ',' '\\n' <<< \"$ids\" |\n    sed 's/^/spotify:track:/' |\n    \"$srcdir/spotify_uri_to_name.sh\" || :\n    echo >&2\n    \"$srcdir/spotify_api.sh\" \"$url_path${ids}\" -X PUT\n}\n\nif [ $# -gt 0 ]; then\n    if [ -f \"$1\" ]; then\n        filename=\"$1\"\n        shift || :\n    else\n        echo \"first argument is not a present file, reading from stdin\" >&2\n    fi\nelse\n    filename=/dev/stdin\nfi\n\nwhile read -r track_uri; do\n    if is_blank \"$track_uri\"; then\n        continue\n    fi\n    if is_local_uri \"$track_uri\"; then\n        continue\n    fi\n    #echo \"$track_uri\"\n    id=\"$(validate_spotify_uri \"$track_uri\")\"\n\n    ids+=(\"$id\")\n\n    if [ \"${#ids[@]}\" -ge 50 ]; then\n        set_to_liked \"${ids[@]}\"\n        ids=()\n    fi\ndone < \"$filename\"\n\nset_to_liked \"${ids[@]}\"\n"
  },
  {
    "path": "spotify/spotify_top_artists.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 10:50:37 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Top Artists via the Spotify API\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/top/artists?offset=$offset&limit=$limit\"\n\noutput(){\n    # some tracks come out with blank artists and track name, skip these using select(name != \"\") filter to avoid blank lines\n    jq -r '.items[] | select(.name != \"\") | .name' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_top_artists_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 10:50:37 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Top Artists in URI format via the Spotify API\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/top/artists?offset=$offset&limit=$limit\"\n\noutput(){\n    # some tracks come out with blank artists and track name, skip these using select(name != \"\") filter to avoid blank lines\n    jq -r '.items[] | select(.name != \"\") | .uri' <<< \"$output\" |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_top_tracks.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 10:50:37 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Top Tracks via the Spotify API\n\nOutput format:\n\nArtist - Track\n\nor if \\$SPOTIFY_CSV environment variable is set then:\n\n\\\"Artist\\\",\\\"Track\\\"\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/top/tracks?offset=$offset&limit=$limit\"\n\noutput(){\n    # some tracks come out with blank artists and track name, skip these using select(name != \"\") filter to avoid blank lines\n    if not_blank \"${SPOTIFY_CSV:-}\"; then\n        jq -r '.items[] | select(.name != \"\") | [([.artists[].name] | join(\", \")), .name] | @csv'\n    else\n        jq -r '.items[] | select(.name != \"\") | [([.artists[].name] | join(\", \")), \"-\", .name] | @tsv'\n    fi <<< \"$output\" |\n    tr '\\t' ' ' |\n    sed '\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_top_tracks_uri.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-20 10:50:37 +0100 (Mon, 20 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/library/get-users-saved-tracks/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the current Spotify user's Top Tracks in URI format via the Spotify API\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# defined in lib/spotify.sh\n# shellcheck disable=SC2154\nurl_path=\"/v1/me/top/tracks?offset=$offset&limit=$limit\"\n\noutput(){\n    jq -r '.items[] | .uri' <<< \"$output\"\n}\n\nexport SPOTIFY_PRIVATE=1\n\nspotify_token\n\nwhile not_null \"$url_path\"; do\n    output=\"$(\"$srcdir/spotify_api.sh\" \"$url_path\" \"$@\")\"\n    #die_if_error_field \"$output\"\n    url_path=\"$(get_next \"$output\")\"\n    output\ndone\n"
  },
  {
    "path": "spotify/spotify_uri_json.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: https://open.spotify.com/track/2aq9tUKdAy8kOXhpGtWBfp\n#\n#  Author: Hari Sekhon\n#  Date: 2026-02-11 00:16:53 -0300 (Wed, 11 Feb 2026)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn\n#  and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTakes a Spotify URI and dumps its JSON for inspection\n\nYou can pass one of the following formats:\n\nspotify:<type>:<alphanumeric_ID>\n\nhttp://open.spotify.com/<type>/<alphanumeric_ID>\n\n<alphanumeric_ID>\n\n\nwhere <type> is track / episode / album / artist\n\nThese IDs are 22 chars, but this is length is not enforced in case the Spotify API changes\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<spotify_uri> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nuri=\"${1:-}\"\nshift || :\n\nspotify_token\n\ninfer_uri_type(){\n    local uri=\"$1\"\n\n    if [[ \"$uri\" =~ ^spotify:(track|album|artist|episode): ]]; then\n        echo \"${BASH_REMATCH[1]}\"\n    elif [[ \"$uri\" =~ ^https?://open.spotify.com/(track|album|artist|episode)/ ]]; then\n        echo \"${BASH_REMATCH[1]}\"\n    else\n        # fallback\n        echo \"${SPOTIFY_URI_TYPE:-track}\"\n    fi\n}\n\ntype=\"$(infer_uri_type \"$uri\")\"\n\nid=\"$(validate_spotify_uri \"$uri\")\"\n\nurl=\"/v1/${type}s/$id\"\n\noutput=\"$(\"$srcdir/spotify_api.sh\" \"$url\" \"$@\")\"\n\njq . <<< \"$output\"\n"
  },
  {
    "path": "spotify/spotify_uri_to_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  args: ../../playlists/spotify/Rocky\n#\n#  Author: Hari Sekhon\n#  Date: 2020-06-25 22:28:51 +0100 (Thu, 25 Jun 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.spotify.com/documentation/web-api/reference/tracks/get-several-tracks/\n#\n# https://developer.spotify.com/documentation/web-api/reference/albums/get-several-albums/\n#\n# https://developer.spotify.com/documentation/web-api/reference/artists/get-several-artists/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/spotify.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTakes Spotify URIs and converts them to Track, Album or Artist names using the Spotify API\n\nSpotify URIs are read from file arguments or standard input and can accept any of the following forms for convenience:\n\nspotify:<type>:<alphanumeric_ID>\nhttp://open.spotify.com/<type>/<alphanumeric_ID>\n<alphanumeric_ID>\n\nwhere <type> is track / episode / album / artist\n\nThese IDs are 22 chars, but this is length is not enforced in case the Spotify API changes\n\nOutput format (depending on whether it's a track, an album or an artist URI):\n\nArtist - Track\nArtist - Album\nArtist\n\nor if \\$SPOTIFY_CSV environment variable is set then:\n\n\\\"Artist\\\",\\\"Track\\\"\n\\\"Artist\\\",\\\"Album\\\"\n\\\"Artist\\\"\n\nUseful for saving Spotify playlists in a format that is easier to understand, revision control changes or export to\nother music systems\n\nor if \\$SPOTIFY_TSV environment variable is set then:\n\nArtist \\\\t Track\n\nUseful for post-processing in scripts like spotify_search_alternate_track_uris.sh that want to be sure which was the\nartist and which was the track component\n\nThe first argument that doesn't correspond to a file and all subsequent arguements are fed as-is to curl as options\n\n\n$usage_auth_help\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<uri_or_files>] [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n# try to avoid hitting HTTP 429 Too Many Requests as this leads to long ban periods of ~14 hours\n# throttle by this many seconds between bulk query requests\nsleep_secs=\"0.5\"\n\ndeclare -a curl_options\ncurl_options=()\n\nspotify_token\n\ninfer_uri_type(){\n    local uri=\"$1\"\n    if [[ \"$uri\" =~ ^spotify:(track|album|artist|episode): ]]; then\n        echo \"${BASH_REMATCH[1]}\"\n    elif [[ \"$uri\" =~ ^https?://open.spotify.com/(track|album|artist|episode)/ ]]; then\n        echo \"${BASH_REMATCH[1]}\"\n    else\n        # default fallback\n        echo \"${SPOTIFY_URI_TYPE:-track}\"\n    fi\n}\n\nconvert(){\n    # associative array: type -> comma-separated IDs\n    declare -A batch_ids\n    declare -A batch_counts\n\n    while read -r uri || [ -n \"$uri\" ]; do\n        [ -z \"$uri\" ] && continue\n\n        # skip local URIs first\n        if is_local_uri \"$uri\"; then\n            # flush all current batches before outputting local\n            for t in \"${!batch_ids[@]}\"; do\n                query_bulk_type \"$t\" \"${batch_ids[$t]}\"\n                batch_ids[\"$t\"]=\"\"\n            done\n            output_local_uri \"$uri\"\n            continue\n        fi\n\n        # determine type of URI\n        type=\"$(infer_uri_type \"$uri\")\"\n        id=\"$(validate_spotify_uri \"$uri\")\"\n\n        # append to batch\n        batch_ids[\"$type\"]+=\"$id,\"\n        batch_counts[\"$type\"]=$(( ${batch_counts[\"$type\"]:-0} + 1 ))\n\n        # flush batch if it reaches 50\n        if [ \"${batch_counts[$type]}\" -ge 50 ]; then\n            query_bulk_type \"$type\" \"${batch_ids[$type]}\"\n            batch_ids[\"$type\"]=\"\"\n            batch_counts[\"$type\"]=0\n        fi\n    done\n\n    # flush remaining batches\n    for t in \"${!batch_ids[@]}\"; do\n        [ -n \"${batch_ids[$t]}\" ] && query_bulk_type \"$t\" \"${batch_ids[$t]}\"\n    done\n}\n\n# bulk query the correct endpoint per type to reduce the number of queries and to\n# both improve performance and try to avoid the dredded HTTP 429 Too Many Requests 14 hour ban\nquery_bulk_type(){\n    local type=\"$1\"\n    local ids_csv=\"${2%,}\"  # remove trailing comma\n    [ -z \"$ids_csv\" ] && return\n\n    local url_base=\"/v1/${type}s\"\n\n    if [ \"${#curl_options[@]}\" -gt 0 ]; then\n        \"$srcdir/spotify_api.sh\" \"$url_base?ids=$ids_csv\" \"${curl_options[@]}\"\n    else\n        \"$srcdir/spotify_api.sh\" \"$url_base?ids=$ids_csv\"\n    fi |\n    output  # pipe into jq output()\n    sleep \"$sleep_secs\"\n}\n\noutput_local_uri(){\n    local uri=\"$1\"\n    if [[ \"$uri\" =~ ^spotify:local: ]]; then\n        uri=\"${uri#spotify:local:}\"\n        artist=\"${uri%%:*}\"\n        uri=\"${uri#*:}\"\n        uri=\"${uri#*:}\"\n        uri=\"${uri%:*}\"\n    elif [[ \"$uri\" =~ open.spotify.com/local/ ]]; then\n        uri=\"${uri#http://open.spotify.com/local/}\"\n        artist=\"${uri%%/*}\"\n        uri=\"${uri#*/}\"\n        uri=\"${uri#*/}\"\n        uri=\"${uri%/*}\"\n    else\n        echo \"Unrecognized track URI format: $uri\"\n        exit 1\n    fi\n    track=\"${uri//+/ }\"\n    if not_blank \"$artist\"; then\n        artist=\"${artist//+/ }\"\n        track=\"$artist - $track\"\n    fi\n    \"$srcdir/../bin/urldecode.sh\" <<< \"$track\"\n}\n\n# This breaks on playlists with mixed URI types such as track + episode in my Love Island playlist so\n# push this logic out of bash and to jq where it can be handled in a better unified way\n#\n#output(){\n#    if [[ \"$output\" =~ \\\"(tracks|albums|artists|episodes)\\\"[[:space:]]*:[[:space:]]+\\[[[:space:]]*null[[:space:]]*\\] ]]; then\n#        echo \"no matching $uri_type URI found - did you specify an incorrect URI or wrong \\$SPOTIFY_URI_TYPE for that URI?\" >&2\n#        return\n#    fi\n#    local conversion=\"@tsv\"\n#    if not_blank \"${SPOTIFY_CSV:-}\"; then\n#        conversion=\"@csv\"\n#    fi\n#    if [ \"$uri_type\" = track ]; then\n#        output_artist_item\n#    elif [ \"$uri_type\" = artist ]; then\n#        jq -r \".${uri_type}s[] | [([.name] | join(\\\", \\\"))] | $conversion\"\n#    elif [ \"$uri_type\" = album ]; then\n#        output_artist_item\n#    else\n#        echo \"URI type '$uri' parsing not implemented\" >&2\n#        exit 1\n#    fi <<< \"$output\" |\n#    clean_output\n#}\n\n# Handled in unified output() function now depending on if fields are detected in jq\n#\n#output_artist_item(){\n#    if not_blank \"${SPOTIFY_CSV:-}\"; then\n#        # some tracks come out with blank artists and track name, skip these using select(name != \"\") filter to avoid blank lines\n#        # unfortunately some tracks actually do come out with blank artist and track name, this must be a bug inside Spotify, but\n#        # filtering it like this throws off the line counts verification and also the track might be blank but the artist might not be\n#        #jq -r \".${uri_type}s[] | select(.name != \\\"\\\") | [([.artists[].name] | join(\\\", \\\")), .name] | $conversion\"\n#        jq -r \".${uri_type}s[] | [([.artists[].name] | join(\\\", \\\")), .name] | $conversion\"\n#    else\n#        #jq -r \".${uri_type}s[] | select(.name != \\\"\\\") | [([.artists[].name] | join(\\\", \\\")), \\\"-\\\", .name] | $conversion\"\n#        jq -r \".${uri_type}s[] | [([.artists[].name] | join(\\\", \\\")), \\\"-\\\", .name] | $conversion\"\n#    fi\n#}\n\noutput(){\n    local conversion=\"@tsv\"\n    if not_blank \"${SPOTIFY_CSV:-}\"; then\n        conversion=\"@csv\"\n    fi\n\n    jq -r '\n    # 1) Playlist items\n    (\n      .tracks?\n      | select(type == \"object\")\n      | .items[]?\n      | (.track // .episode)\n      | select(. != null)\n    ),\n\n    # 2) Bulk tracks\n    (\n      .tracks?\n      | select(type == \"array\")\n      | .[]?\n    ),\n\n    # 3) Bulk episodes\n    (\n      .episodes[]?\n    ),\n\n    # 4) Single track / episode object\n    (\n      select(type == \"object\" and (.type == \"track\" or .type == \"episode\"))\n    )\n\n    | if .type == \"track\" then\n        [\n          (.artists | map(.name) | join(\", \")),\n          \"-\",\n          .name\n        ]\n      elif .type == \"episode\" then\n        [\n          .show.name,\n          \"-\",\n          .name\n        ]\n      else\n        empty\n      end\n    | '\"$conversion\"'\n    ' |\n    clean_output\n}\nexport -f output\n\nclean_output(){\n    if [ -n \"${SPOTIFY_TSV:-}\" ]; then\n        sed $'s/\\t-\\t/\\t/'\n    else\n        tr '\\t' ' '\n    fi |\n    sed '\n        s/^[[:space:]]*-//;\n        s/^[[:space:]]*//;\n        s/[[:space:]]*$//\n    '\n}\n\nfiles=()\ntmp=\"$(mktemp)\"\n\nfor filename in \"$@\"; do\n    if [ -f \"$filename\" ]; then\n        files+=(\"$filename\")\n        shift || :\n    elif [[ \"$filename\" =~ ^spotify: ]]; then\n        echo \"$filename\" >> \"$tmp\"\n        shift || :\n    else\n        break\n    fi\ndone\n\nif [ -s \"$tmp\" ]; then\n    files+=(\"$tmp\")\nfi\n\nif [ $# -gt 0 ]; then\n    curl_options=(\"$@\")\nfi\n\nif [ -n \"${files[*]:-}\" ]; then\n    for filename in \"${files[@]}\"; do\n        convert < \"$filename\"\n    done\nelse\n    convert  # read from stdin\nfi\n"
  },
  {
    "path": "teamcity/.teamcity.vcs.json",
    "content": "{\n  \"id\": \"TeamCity\",\n  \"name\": \"TeamCity\",\n  \"vcsName\": \"jetbrains.git\",\n  \"href\": \"/httpAuth/app/rest/vcs-roots/id:TeamCity\",\n  \"project\": {\n    \"id\": \"_Root\",\n    \"name\": \"<Root project>\",\n    \"description\": \"Contains all other projects\",\n    \"href\": \"/httpAuth/app/rest/projects/id:_Root\",\n    \"webUrl\": \"http://localhost:8111/project.html?projectId=_Root\"\n  },\n  \"properties\": {\n    \"count\": 9,\n    \"property\": [\n      {\n        \"name\": \"agentCleanFilesPolicy\",\n        \"value\": \"ALL_UNTRACKED\"\n      },\n      {\n        \"name\": \"agentCleanPolicy\",\n        \"value\": \"ON_BRANCH_CHANGE\"\n      },\n      {\n        \"name\": \"authMethod\",\n        \"value\": \"ANONYMOUS\"\n      },\n      {\n        \"name\": \"branch\",\n        \"value\": \"master\"\n      },\n      {\n        \"name\": \"ignoreKnownHosts\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"submoduleCheckout\",\n        \"value\": \"CHECKOUT\"\n      },\n      {\n        \"name\": \"url\",\n        \"value\": \"https://github.com/HariSekhon/TeamCity-CI\"\n      },\n      {\n        \"name\": \"useAlternates\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"usernameStyle\",\n        \"value\": \"USERID\"\n      }\n    ]\n  },\n  \"vcsRootInstances\": {\n    \"href\": \"/httpAuth/app/rest/vcs-root-instances?locator=vcsRoot:(id:TeamCity)\"\n  }\n}\n"
  },
  {
    "path": "teamcity/.teamcity.vcs.oauth.json",
    "content": "{\n  \"id\": \"TeamCity\",\n  \"name\": \"TeamCity\",\n  \"vcsName\": \"jetbrains.git\",\n  \"href\": \"/httpAuth/app/rest/vcs-roots/id:TeamCity\",\n  \"project\": {\n    \"id\": \"_Root\",\n    \"name\": \"<Root project>\",\n    \"description\": \"Contains all other projects\",\n    \"href\": \"/httpAuth/app/rest/projects/id:_Root\",\n    \"webUrl\": \"http://localhost:8111/project.html?projectId=_Root\"\n  },\n  \"properties\": {\n    \"count\": 9,\n    \"property\": [\n      {\n        \"name\": \"agentCleanFilesPolicy\",\n        \"value\": \"ALL_UNTRACKED\"\n      },\n      {\n        \"name\": \"agentCleanPolicy\",\n        \"value\": \"ON_BRANCH_CHANGE\"\n      },\n      {\n        \"name\": \"authMethod\",\n        \"value\": \"PASSWORD\"\n      },\n      {\n        \"name\": \"branch\",\n        \"value\": \"master\"\n      },\n      {\n        \"name\": \"ignoreKnownHosts\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"submoduleCheckout\",\n        \"value\": \"CHECKOUT\"\n      },\n      {\n        \"name\": \"url\",\n        \"value\": \"https://github.com/HariSekhon/TeamCity-CI\"\n      },\n      {\n        \"name\": \"useAlternates\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"username\",\n        \"value\": \"HariSekhon\"\n      },\n      {\n        \"name\": \"usernameStyle\",\n        \"value\": \"USERID\"\n      }\n    ]\n  },\n  \"vcsRootInstances\": {\n    \"href\": \"/httpAuth/app/rest/vcs-root-instances?locator=vcsRoot:(id:TeamCity)\"\n  }\n}\n"
  },
  {
    "path": "teamcity/.teamcity.vcs.ssh.json",
    "content": "{\n  \"id\": \"TeamCity\",\n  \"name\": \"TeamCity\",\n  \"vcsName\": \"jetbrains.git\",\n  \"href\": \"/app/rest/vcs-roots/id:TeamCity\",\n  \"project\": {\n    \"id\": \"_Root\",\n    \"name\": \"<Root project>\",\n    \"description\": \"Contains all other projects\",\n    \"href\": \"/app/rest/projects/id:_Root\",\n    \"webUrl\": \"http://localhost:8111/project.html?projectId=_Root\"\n  },\n  \"properties\": {\n    \"count\": 11,\n    \"property\": [\n      {\n        \"name\": \"agentCleanFilesPolicy\",\n        \"value\": \"ALL_UNTRACKED\"\n      },\n      {\n        \"name\": \"agentCleanPolicy\",\n        \"value\": \"ON_BRANCH_CHANGE\"\n      },\n      {\n        \"name\": \"authMethod\",\n        \"value\": \"TEAMCITY_SSH_KEY\"\n      },\n      {\n        \"name\": \"branch\",\n        \"value\": \"refs/heads/master\"\n      },\n      {\n        \"name\": \"ignoreKnownHosts\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"submoduleCheckout\",\n        \"value\": \"CHECKOUT\"\n      },\n      {\n        \"name\": \"teamcitySshKey\",\n        \"value\": \"VCS SSH Key\"\n      },\n      {\n        \"name\": \"url\",\n        \"value\": \"github.com:HariSekhon/TeamCity-CI\"\n      },\n      {\n        \"name\": \"useAlternates\",\n        \"value\": \"true\"\n      },\n      {\n        \"name\": \"username\",\n        \"value\": \"git\"\n      },\n      {\n        \"name\": \"usernameStyle\",\n        \"value\": \"USERID\"\n      }\n    ]\n  },\n  \"vcsRootInstances\": {\n    \"href\": \"/app/rest/vcs-root-instances?locator=vcsRoot:(id:TeamCity)\"\n  }\n}\n"
  },
  {
    "path": "teamcity/teamcity.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-24 17:09:11 +0000 (Tue, 24 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# dipping into interactive library for opening browser to TeamCity to accept EULA\n# XXX: order is important here because there is an interactive library of retry() and a scripting library version of retry() and we want the latter, which must be imported second\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/.bash.d/network.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034\nusage_description=\"\nBoots TeamCity CI cluster with server and agent(s) in Docker, and builds the current repo\n\n- boots TeamCity server and agent in Docker\n- authorizes the agent(s) to begin building\n- waits for you to accept the EULA\n  - prints the TeamCity URL\n  - opens the TeamCity web UI\n\n- creates an administator-level user (\\$TEAMCITY_USER, / \\$TEAMCITY_PASSWORD - defaults to admin / admin)\n  - sets the full name, email, and VCS commit username to Git's user.name and user.email if configured for TeamCity to Git VCS tracking integration\n  - opens the TeamCity web UI login page in browser\n\n- creates a GitHub OAuth connection if credentials are available (\\$TEAMCITY_GITHUB_CLIENT_ID and \\$TEAMCITY_GITHUB_CLIENT_SECRET)\n  - this saves you having to use your own username and password for the GitHub VCS such as the config repo - just click the GitHub icon next to the VCS url to auto-authenticate\n\n- if there is a .teamcity.vcs.json VCS configuration in the current directory, creates the VCS to use as a config sync repo\n  - if this is a private repo, you either need to put the credentials in the file temporarily, or set the password to blank, and edit it after boot\n    - currently must use authentication even if th repo is public:  https://youtrack.jetbrains.com/issue/TW-69183\n  - if GitHub OAuth connection credentials are available, will instead look for .teamcity.auth.vcs.json\n  - you'll need to disable & re-enable the project's Versioned Settings to get the import dialog for your projects before it starts sync'ing\n    - this is another TeamCity limitation:  https://youtrack.jetbrains.com/issue/TW-58754\n\nUsage:\n\n    ${0##*/} [up]\n\n    ${0##*/} down\n\n    ${0##*/} ui     - prints the TeamCity Server URL and automatically opens in browser\n\n\nIdempotent, you can re-run this and continue from any stage\n\n\nThe official docker images from JetBrains are huge so the first pull may take a while\n\n\nSee Also:\n\n    teamcity_api.sh - makes heavy use of this script to handle setup API calls with authentication\n\n\nAdvanced:\n\nTeamCity GitHub OAuth integration - set up your TeamCity OAuth credentials here:\n\n    https://github.com/settings/developers\n\nIf \\$TEAMCITY_GITHUB_CLIENT_ID and \\$TEAMCITY_GITHUB_CLIENT_SECRET are available in the environment it will configure a connection for your GitHub VCS roots authentication\n\n\nIf your GitHub OAuth connection has been created you can use this to authenticate the TeamCity VCS root in the Root project,\nand use that to sync your Project configuration to/from Github under Project's Settings -> Versioned Settings using the VCS referenced from the Root project.\n\nIt's better to keep the TeamCity config VCS in the Root project because when you sync a project and it replaces the VCS json credential it breaks the GitHub sync\nand needs to be re-created. By putting it in the Root project and only enabling VCS sync on the sub-project you avoid this problem.\n\n\nTested on TeamCity 2020.1, 2020.2\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[ up | down | ui ]\"\n\nhelp_usage \"$@\"\n\nexport COMPOSE_PROJECT_NAME=\"bash-tools\"\nexport COMPOSE_FILE=\"$srcdir/../docker-compose/teamcity.yml\"\n\nvcs_config=\".teamcity.vcs.json\"\n# OAuth connection\nvcs_config_auth=\".teamcity.vcs.oauth.json\"\n# SSH key connection\nvcs_config_ssh_auth=\".teamcity.vcs.ssh.json\"\n\nTEAMCITY_SSH_KEY=\"${TEAMCITY_SSH_KEY:-$HOME/.ssh/id_rsa}\"\n\nproject=\"GitHub\"\n\n#teamcity_port=\"$(docker-compose config | sed -n '/teamcity-server:[[:space:]]*$/,$p' | awk '/- published: [[:digit:]]+/{print $3; exit}')\"\n\n# don't take any change this script could run against a real teamcity server for safety\n#export TEAMCITY_URL=\"http://${TEAMCITY_HOST:-localhost}:${TEAMCITY_PORT:-8111}\"\nexport TEAMCITY_URL=\"http://${DOCKER_HOST:-localhost}:8111\"\n\nif ! type docker-compose &>/dev/null; then\n    \"$srcdir/../install/install_docker_compose.sh\"\nfi\n\naction=\"${1:-up}\"\nshift || :\n\nif [ \"$action\" = up ]; then\n    timestamp \"Booting TeamCity cluster:\"\n    # starting agents later they won't be connected in time to become authorized\n    # only start the server, don't wait for the agent to download before triggering the URL to prompt user for initialization so it can progress while agent is downloading\n    #docker-compose up -d teamcity-server \"$@\"\n    docker-compose up -d \"$@\"\n    echo\nelif [ \"$action\" = restart ]; then\n    docker-compose down\n    echo\n    exec \"${BASH_SOURCE[0]}\" up\nelif [ \"$action\" = ui ]; then\n    echo \"TeamCity Server URL:  $TEAMCITY_URL\"\n    open \"$TEAMCITY_URL\"\n    exit 0\nelse\n    docker-compose \"$action\" \"$@\"\n    echo\n    exit 0\nfi\n\n# fails due to 302 redirect to http://localhost:8111/setupAdmin.html\n# / and /setupAdmin.html and /login.html\n#when_url_content \"$TEAMCITY_URL/login.html\" '(?i:teamcity)'\n#when_ports_available 60 \"${TEAMCITY_HOST:-localhost}\" \"${TEAMCITY_PORT:-8111}\"\nwhen_url_content 60 \"$TEAMCITY_URL\" '.*'\necho\n\n# XXX: database.properties is mounted to skip the database step now\nis_setup_in_progress(){\n     # don't let cut off output affect the return code\n     { curl -sSL \"$TEAMCITY_URL\" || : ; } | \\\n       grep -qi -e 'first.*start' \\\n                -e 'database.*setup' \\\n                -e 'TeamCity Maintenance' \\\n                -e 'Setting up'\n}\n\ntimestamp \"TeamCity Server URL:  $TEAMCITY_URL\"\necho\nif is_setup_in_progress; then\n    timestamp \"Opening TeamCity Server URL in web browser to continue, click proceed, accept EULA etc..\"\n    echo\n    open \"$TEAMCITY_URL\"\nfi\n\n# too late, agent won't arrive in the unauthorized list in time to be found and authorized before this script exits, agents must boot in parallel with server not later\n# now download and start the agent(s) while the server is booting\n#docker-compose up -d\n\n# now continue configuring server\n\nmax_secs=300\n\nSECONDS=0\ntimestamp \"waiting for up to $max_secs seconds for user to click proceed through First Start and database setup pages\"\nwhile is_setup_in_progress; do\n    timestamp \"waiting for you to click proceed through First Start & setup pages and then preliminary initialization to finish\"\n    if [ $SECONDS -gt $max_secs ]; then\n        die \"Did not progress past First Start and setup pages within $max_secs seconds\"\n    fi\n    sleep 3\ndone\necho\n\n# second run would break here as this wouldn't come again, must use .* search\n# just to check we are not getting a temporary 404 or something that happens before the EULA comes up\n#when_url_content 60 \"$TEAMCITY_URL\" \"license.*agreement\"\nwhen_url_content 60 \"$TEAMCITY_URL\" \".*\"\necho\n\nSECONDS=0\ntimestamp \"waiting for up to $max_secs seconds for user to accept EULA\"\n# curl gives an error when grep cuts its long EULA agreement short:\n# (23) Failed writing body\nwhile { curl -sSL \"$TEAMCITY_URL\" 2>/dev/null || : ; } |\n      grep -qi 'license.*agreement'; do\n    timestamp \"waiting for you to accept the license agreement\"\n    if [ $SECONDS -gt $max_secs ]; then\n        die \"Did not accept EULA within $max_secs seconds\"\n    fi\n    sleep 3\ndone\necho\n\nSECONDS=0\ntimestamp \"waiting for up to $max_secs seconds for TeamCity to finish initializing\"\n# too transitory to be idempotent\n#while ! curl -sS \"$TEAMCITY_URL\" | grep -q 'TeamCity is starting'; do\n# although hard to miss this log as not a fast scroll, might break idempotence for re-running later if logs are cycled out of buffer\n#while ! docker-compose logs --tail 50 teamcity-server | grep -q 'TeamCity initialized'; do\nwhile ! { docker-compose logs teamcity-server || : ; } |\n      grep -q -e 'Super user authentication token'; do\n              #-e 'TeamCity initialized' # happens just before but checking for the super user token achieves both and protects against race condition\n    timestamp 'waiting for TeamCity server to finish initializing and reveal superuser token in logs'\n    if [ $SECONDS -gt $max_secs ]; then\n        die \"TeamCity server failed to initialize within $max_secs seconds (perhaps you didn't trigger the UI to continue initialization?)\"\n    fi\n    sleep 3\ndone\necho\n\nTEAMCITY_SUPERUSER_TOKEN=\"$(docker-compose logs teamcity-server | grep -E -o 'Super user authentication token: [[:alnum:]]+' | tail -n1 | awk '{print $5}' || :)\"\n\nif [ -z \"$TEAMCITY_SUPERUSER_TOKEN\" ]; then\n    timestamp \"ERROR: Super user token not found in docker logs (maybe premature or late ie. logs were already cycled out of buffer?)\"\n    exit 1\nfi\n\nexport TEAMCITY_SUPERUSER_TOKEN\n\ntimestamp \"TeamCity superuser token: $TEAMCITY_SUPERUSER_TOKEN\"\ntimestamp \"(this must be used with a blank username via basic auth if using the API)\"\necho\n\nteamcity_user=\"${TEAMCITY_USER:-admin}\"\nteamcity_password=\"${TEAMCITY_PASSWORD:-admin}\"\n\nuser_already_exists=0\napi_token=\"\"\n#timestamp \"Checking if teamcity user '$teamcity_user' exists\"\ntimestamp \"Checking if any user already exists\"\nusers=\"$(\"$srcdir/teamcity_api.sh\" /users -sSL --fail | jq -r '.user[].username')\"\n#if grep -Fxq \"$teamcity_user\" <<< \"$users\"; then\n#    timestamp \"teamcity user '$teamcity_user' user already detected, skipping creation\"\nif [ -n \"${users//[[:space:]]/}\" ]; then\n    timestamp \"users already exist, not creating teamcity administrative user '$teamcity_user'\"\n    user_already_exists=1\nelse\n    #timestamp \"Creating teamcity user '$teamcity_user':\"\n    timestamp \"no users exist yet, creating teamcity user '$teamcity_user'\"\n    \"$srcdir/teamcity_api.sh\" /users -sSL --fail \\\n         -d \"{ \\\"username\\\": \\\"$teamcity_user\\\", \\\"password\\\": \\\"$teamcity_password\\\"}\"\n         # Note: Unnecessary use of -X or --request, POST is already inferred.\n         #-X POST \\\n    # no newline returned if error eg.\n    #       Details: jetbrains.buildServer.server.rest.errors.BadRequestException: Cannot create user as user with the same username already exists, caused by: jetbrains.buildServer.users.DuplicateUserAccountException: The specified username 'admin' is already in use by some other user.\n    #       Invalid request. Please check the request URL and data are correct.\n    echo\n    echo\n    git_user=\"$(git config user.name)\"\n    git_email=\"$(git config user.email)\"\n    if [ -n \"$git_user\" ]; then\n        timestamp \"Setting teamcity user $teamcity_user's username to '$git_user'\"\n        \"$srcdir/teamcity_api.sh\" \"/users/$teamcity_user/name\" -X PUT -d \"$git_user\" -H 'Content-Type: text/plain'  -H 'Accept: text/plain'\n        # API echo's username without newline\n        echo\n        timestamp \"Setting teamcity user $teamcity_user's VCS default username to '$git_user'\"\n        \"$srcdir/teamcity_api.sh\" \"/users/admin/properties/plugin:vcs:anyVcs:anyVcsRoot\" -X PUT -d \"$git_user\" -H 'Content-Type: text/plain'  -H 'Accept: text/plain'\n        # API echo's username without newline\n        echo\n    fi\n    if [ -n \"$git_email\" ]; then\n        timestamp \"Setting teamcity user $teamcity_user's email to '$git_email'\"\n        \"$srcdir/teamcity_api.sh\" \"/users/$teamcity_user/email\" -X PUT -d \"$git_email\" -H 'Content-Type: text/plain'  -H 'Accept: text/plain'\n        # prints email without newline\n        echo\n    fi\n    timestamp \"Setting teamcity user '$teamcity_user' as system administrator:\"\n    \"$srcdir/teamcity_api.sh\" \"/users/username:$teamcity_user/roles/SYSTEM_ADMIN/g/\" -sSL --fail -X PUT > /dev/null\n    # no newline returned\n    echo\n    api_token=\"$(\"$srcdir/teamcity_api.sh\" \"/users/$teamcity_user/tokens\" -sSL | \\\n                 jq -r '.token[]' || :)\"\n    # XXX: could create expiring self-deleting token here each time, but would make idempotence tricker\n    # due to timings and also use might want to use it in teamcity_api.sh later\n    if [ -n \"$api_token\" ]; then\n        timestamp \"TeamCity user '$teamcity_user' already has an API token, skipping token creation\"\n        timestamp \"since we cannot get existing token value out of the API, will load TEAMCITY_SUPERUSER_TOKEN to environment to use instead\"\n    else\n        timestamp \"Creating API token for user '$teamcity_user'\"\n        api_token=\"$(\"$srcdir/teamcity_api.sh\" \"/users/$teamcity_user/tokens/mytoken\" -sSL --fail -X POST | jq -r '.value')\"\n        timestamp \"here is your user API token, export this and then you can easily use teamcity_api.sh:\"\n        echo\n        # this takes precedence so disable it and use the user's api token instead\n        unset TEAMCITY_SUPERUSER_TOKEN\n        echo \"export TEAMCITY_URL=$TEAMCITY_URL\"\n        export TEAMCITY_URL=\"$TEAMCITY_URL\"\n        echo \"export TEAMCITY_TOKEN=$api_token\"\n        export TEAMCITY_TOKEN=\"$api_token\"\n    fi\nfi\necho\n\nif [ \"$user_already_exists\" = 0 ]; then\n    timestamp \"Login here with username '$teamcity_user' and password: \\$TEAMCITY_PASSWORD (default: admin):\"\n    echo\n    login_url=\"$TEAMCITY_URL/login.html\"\n    echo \"$login_url\"\n    echo\n    timestamp \"Ppening TeamCity Server URL\"\n    open \"$login_url\"\n    echo\n    echo\nfi\n\ntimestamp \"getting list of expected agents\"\nexpected_agents=\"$(docker-compose config | awk '/^[[:space:]]+AGENT_NAME:/ {print $2}' | sed '/^[[:space:]]*$/d')\"\nnum_expected_agents=\"$(grep -c . <<< \"$expected_agents\" || :)\"\n\nget_connected_agents(){\n    \"$srcdir/teamcity_api.sh\" \"/agents?locator=connected:true,authorized:any\" -sSL --fail |\n    jq -r '.agent[].name'\n}\n\nSECONDS=0\ntimestamp \"waiting for $num_expected_agents expected agent(s) to connect before authorizing them\"\nwhile true; do\n    num_connected_agents=\"$(get_connected_agents | grep -c . || :)\"\n    timestamp \"connected agents: $num_connected_agents\"\n    if [ \"$num_connected_agents\" -ge \"$num_expected_agents\" ]; then\n        timestamp \"$num_connected_agents connected agents >= $num_expected_agents expected agents, continuing\"\n        break\n    fi\n    if [ $SECONDS -gt $max_secs ]; then\n        timestamp \"giving up waiting for connect agents after $max_secs\"\n        break\n    fi\n    sleep 3\ndone\necho\n\ntimestamp \"getting list of unauthorized agents\"\n# using our new teamcity API token, let's agents waiting to be authorized\nunauthorized_agents=\"$(\"$srcdir/teamcity_api.sh\" \"/agents?locator=authorized:false\" -sSL --fail | jq -r '.agent[].name')\"\n\ntimestamp \"authorizing any expected agents that are not currently authorized\"\nif [ -z \"$unauthorized_agents\" ]; then\n    timestamp \"no unauthorized agents found\"\nfi\nfor agent in $unauthorized_agents; do\n    # XXX: recreated agents end up with a digit appended to the name to avoid clash with old stale agent reference\n    #      if the agent disk state isn't lost this shouldn't be needed, but this environment is disposable so allow this\n    #      this is only a local environment so we don't have to worry about rogue agents\n    for expected_agent in $expected_agents; do\n        # grep -f would be easier but don't want to depend on have the GNU version installed and then remapped via func\n        if [[ \"$agent\" =~ ^$expected_agent(-[[:digit:]]+)?$ ]]; then\n            timestamp \"authorizing expected agent '$agent'\"\n            # needs -H 'Accept: text/plain' to override the default -H 'Accept: application/json' from teamcity_api.sh\n            # otherwise gets 403 error and then even switching to -H 'Accept: text/plain' still breaks due to cookie jar behaviour,\n            # so teamcity_api.sh now uses a unique cookie jar per script run and clears the cookie jar first\n            \"$srcdir/teamcity_api.sh\" \"/agents/$agent/authorized\" -X PUT -d true -H 'Accept: text/plain' -H 'Content-Type: text/plain'\n            # no newline returned\n            echo\n            continue 2\n        fi\n    done\n    timestamp \"WARNING: unauthorized agent '$agent' was not expected, not automatically authorizing\"\ndone\necho\n\n# this stops us accumulating huge numbers of agent-[[:digit:]] increments each time\ntimestamp \"deleting old disconnected agent references\"\n# slight race condition here but it's not critical\ndisconnected_agents=\"$(\"$srcdir/teamcity_api.sh\" \"/agents?locator=connected:false\" -sSL --fail | jq -r '.agent[].name')\"\nfor disconnected_agent in $disconnected_agents; do\n    timestamp \"deleting disconnected agent '$disconnected_agent'\"\n    \"$srcdir/teamcity_api.sh\" \"/agents/$disconnected_agent\" -X DELETE\ndone\necho\n\nif [ -f \"$TEAMCITY_SSH_KEY\" ]; then\n    # overwrites key if already exists\n    if \"$srcdir/teamcity_upload_ssh_key.sh\" \"$TEAMCITY_SSH_KEY\" \"VCS SSH Key\"; then\n        if [ -f \"$vcs_config_ssh_auth\" ]; then\n            echo\n            timestamp \"switching VCS config to '$vcs_config_ssh_auth'\"\n            vcs_config=\"$vcs_config_ssh_auth\"\n        fi\n    fi\n    echo\nfi\n\nif [ -n \"${TEAMCITY_GITHUB_CLIENT_ID:-}\" ] && [ -n \"${TEAMCITY_GITHUB_CLIENT_SECRET:-}\" ]; then\n    # detects and skips creation if an OAuth connection named 'GitHub.com' already exists\n    \"$srcdir/teamcity_create_github_oauth_connection.sh\"\n    echo\n    if [ \"$vcs_config\" != \"$vcs_config_ssh_auth\" ]; then\n        if [ -f \"$vcs_config_auth\" ]; then\n            timestamp \"switching VCS config to '$vcs_config_auth'\"\n            vcs_config=\"$vcs_config_auth\"\n            echo\n        fi\n    fi\nfi\n\nif [ -f \"$vcs_config\" ]; then\n    #timestamp \"Now creating primary project '$project'\"\n    # XXX: TeamCity API doesn't yet support creating a project from a saved configuration via the API, see this ticket:\n    #\n    #      https://youtrack.jetbrains.com/issue/TW-43542\n    #\n    # So we create an empty project, then configure a VCS root to GitHub and reconfigure the project to pull from a GitHub repo\n    # TODO: get the project name from the config file\n    \"$srcdir/teamcity_create_project.sh\" \"$project\"\n    echo\n    vcs_id=\"$(jq -r .id < \"$vcs_config\")\"\n    project_id=\"$(jq -r .project.id < \"$vcs_config\")\"\n    if \"$srcdir/teamcity_vcs_roots.sh\" | grep -qi \"^${vcs_id}[[:space:]]\"; then\n        timestamp \"VCS root '$vcs_id' already exists, skipping creation\"\n    else\n        if [ \"$project_id\" != \"_Root\" ]; then\n            timestamp \"Creating VCS container project '$project_id' if not already exists...\"\n            \"$srcdir/teamcity_create_project.sh\" \"$project_id\"\n        fi\n        # XXX: this fails when TeamCity has only just booted, probably due to some initialization timing, but works on second run, so just wait for it to succeed\n        # UPDATE: seems this errors the first time yet still creates it and the second try skips as already exists\n        retry 300 \"$srcdir/teamcity_create_vcs_root.sh\" \"$vcs_config\"\n    fi\n    echo\n    timestamp \"Configuring project Versioned Settings to import all buildTypes and VCS\"\n    \"$srcdir/teamcity_project_set_versioned_settings.sh\" \"$project\"\n    echo\n    echo\n    timestamp \"NOTICE: you need to enable VCS authentication for write access to be able to sync project configs:\"\n    timestamp \"        (even if you have GitHub OAuth connection automated, you still need to click on the GitHub icon to initialize the sign-in)\"\n    echo\n    printf '\\t%s\\n' \"$TEAMCITY_URL/admin/editVcsRoot.html?action=editVcsRoot&vcsRootId=$vcs_id\"\n    echo\n    echo\n    timestamp \"This is a limitation of the TeamCity API (as of Dec 2020), documented here:\"\n    echo\n    printf '\\t%s\\n' \"https://youtrack.jetbrains.com/issue/TW-69183\"\n    echo\n    echo\n    timestamp \"NOTICE: one you've enabled authenticated access to the VCS root you'll have to disable and re-enable the '$project' project's Versioned Settings to get the import dialog for config sync to start working\"\n    echo\n    printf '\\t%s\\n' \"$TEAMCITY_URL/admin/editProject.html?projectId=$project&tab=versionedSettings\"\n    echo\n    echo\n    timestamp \"This is a limitation of the TeamCity API (as of Dec 2020), documented here:\"\n    echo\n    printf '\\t%s\\n' \"https://youtrack.jetbrains.com/issue/TW-58754\"\n    echo\n    echo\nelse\n    timestamp \"no config found: $vcs_config - skipping VCS setup and versioning integration / import\"\nfi\necho\n\n#timestamp \"Optimistically setting any buildTypes descriptions from their GitHub repos (ignoring failures)\"\n#\"$srcdir/teamcity_buildtypes_set_description_from_github.sh\" || :\n\ntimestamp \"Build status icons:\"\necho\nprintf '\\t%s\\n' \"$TEAMCITY_URL/app/rest/builds/<build>/statusIcon.svg\"\necho\ntimestamp \"(requires this setting for each buildType:  General Settings -> 'enable status widget'  to permit unauthenticated status badge access)\"\necho\necho\ntimestamp \"TeamCity is up and ready\"\n"
  },
  {
    "path": "teamcity/teamcity_agents.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 18:53:42 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Agents\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the Teamcity agents and their states via the Teamcity API\n\nOutput format:\n\n<agent_id>  <connected>     <authorized>      <enabled>    <up_to_date>    <name>\n\nSpecify \\$NO_HEADER to omit the header line\n\nSee adjacent teamcity_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n{\nif [ -z \"${NO_HEADER:-}\" ]; then\n    printf 'Agent_ID\\tConnected\\tAuthorized\\tEnabled\\tUp_to_Date\\tName\\n'\nfi\n\"$srcdir/teamcity_api.sh\" /agents |\njq -r '.agent[].id' |\nwhile read -r id; do\n    \"$srcdir/teamcity_api.sh\" \"/agents/id:$id\" |\n    jq -r '[.id, .connected, .authorized, .enabled, .uptodate, .name] | @tsv'\ndone\n} |\ncolumn -t\n"
  },
  {
    "path": "teamcity/teamcity_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 16:38:30 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Teamcity API\n\nRequires \\$TEAMCITY_TOKEN be available in the environment, generation a token here:\n\n    \\$TEAMCITY_URL/profile.html?item=accessTokens\n        or\n    https://\\$TEAMCITY_HOST:\\$TEAMCITY_PORT/profile.html?item=accessTokens\n\nIf using the superuser token, it must instead be specified as \\$TEAMCITY_SUPERUSER_TOKEN and takes precedence\n\n\n\\$TEAMCITY_URL or \\$TEAMCITY_HOST must be set to point to the Teamcity server\n\nIf \\$TEAMCITY_HOST is used, then the following may also be set:\n\n\\$TEAMCITY_PORT - defaults to 8111 and is only used if \\$TEAMCITY_URL is not used\n\\$TEAMCITY_SSL  - defaults to http, any value enables https\n\nIf no Accept or Content-Type headers are passed in the arguments, then sets them to application/json by default because the API returns XML otherwise, and who wants that in the modern age... but this still allows you to override them for the odd endpoint that requires instead sending text/plain (looking at you Agent authorization endpoint)\n\n\nAPI Reference:\n\n    https://www.jetbrains.com/help/teamcity/rest-api.html\n\n    https://www.jetbrains.com/help/teamcity/rest-api-reference.html\n\n\nThe API version used by default is latest, but you can specify an older API version like so:\n\n\\$TEAMCITY_API_VERSION=2018.1\n\nAt time of writing, prior API versions are: 2018.1, 2017.2, 2017.1, 10.0, 9.1, 9.0, 8.1, 8.0. See:\n\n    https://www.jetbrains.com/help/teamcity/rest-api.html#REST+API+Versions\n\n\nSee Also:\n\n    teamcity.sh - boots a TeamCity cluster in Docker and makes heavy use of this script against many API endpoints to configure it\n                  convenient way of getting a TeamCity API to test this script against, outputs the TEAMCITY_URL and TEAMCITY_TOKEN for you\n\n    teamcity_builds.sh - lists builds using this API\n    teamcity_agents.sh - lists agents using this API\n\n\nExamples:\n\n(don't forget to add '\\$help' to the end to find out what attributes endpoints support):\nRemember you can check logs/teamcity-rest.log server log for these API requests\n\n\n\n# Explore API:\n\n    ${0##*/} /server\n\n\n# Explore supported requests and parameters:\n\n    ${0##*/} /application.wadl\n\n\n# Swagger endpoint:\n\n    ${0##*/} /swagger.json\n\n\n# Show Teamcity agents:\n\n    ${0##*/} /agents\n\n\n# Get an agent's details:\n\n    ${0##*/} /agents/id:3\n\n\n# Get list of builds:\n\n    ${0##*/} /builds\n\n\n# Get list of builds filtered by successful status with a specific tag:\n\n    ${0##*/} /builds?locator=status:SUCCESS,tag=dev\n\n\n# Get the build queue:\n\n    ${0##*/} /buildQueue\n\n\n# Get list of projects:\n\n    ${0##*/} /projects\n\n\n# Get details on a specific project:\n\n    ${0##*/} /projects/<id>\n# or\n    ${0##*/} /projects/id:<id>\n\n\n# Get list of revision control repositories roots:\n\n    ${0##*/} /vcs-roots\n\n\n# Get details on one specific revision control repository:\n\n    ${0##*/} /vcs-roots/id:<id>\n\n\n# Get list of cloud profiles (kubernetes is configured here):\n\n    ${0##*/} /cloud/profiles\n\n\n# Check your license details:\n\n    ${0##*/} /server/licensingData\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nurl_path=\"$1\"\nshift || :\n\nurl_path=\"${url_path##/}\"\n\ncurl_api_opts \"$@\"\n\n# don't enforce as hard requirements here, instead try alternation further down and construct from what's available\n#check_env_defined \"TEAMCITY_URL\"\n#check_env_defined \"TEAMCITY_HOST\"\n#check_env_defined \"TEAMCITY_TOKEN\"\n\n# to speed up http basic auth:\n#\n# https://youtrack.jetbrains.com/issue/TW-14209\n#\n# https://youtrack.jetbrains.com/issue/TW-36844#comment=27-752545\n#\n# used to put this in /tmp but it is created world readable by default and when securing it there is a race condition\n# between the 2 curl and chmod lines, whereas $HOME is more likely to be read restricted\n# now additionally this is initialized with restricted permissions before it is ever used to avoid this race condition\ncookie_jar=~/\".teamcity_cookie_jar/.${EUID:-$UID}.$$\"\nmkdir -p \"${cookie_jar%/*}\"  # pre-create the directory\n# XXX: this cookie jar causes 403 to API endpoints such as /agents/id:1/authorized when switching between -H 'Accept: application/json' and -H 'Accept: text/plain' as that endpoint only works with the latter but this cache breaks this\n#rm -f \"$cookie_jar\"\n#touch \"$cookie_jar\"\n: > \"$cookie_jar\"\nchown \"$(whoami)\" \"$cookie_jar\"\nchmod 0600 \"$cookie_jar\"\n\nCURL_OPTS+=(-b \"$cookie_jar\" -c \"$cookie_jar\")\n\nif [ -n \"${TEAMCITY_URL:-}\" ]; then\n    url_base=\"${TEAMCITY_URL%%/}\"\nelse\n    protocol=\"http\"\n    if [ -n \"${TEAMCITY_SSL:-}\" ]; then\n        protocol=\"https\"\n    fi\n    [ -n \"${TEAMCITY_HOST:-}\" ] || usage \"neither \\$TEAMCITY_URL nor \\$TEAMCITY_HOST defined in environment\"\n    host=\"$TEAMCITY_HOST\"\n    port=\"${TEAMCITY_PORT:-8111}\"\n    url_base=\"$protocol://$host:$port\"\nfi\n\n# for superuser account, empty username and system generated password, but curl_auth.sh won't allow that so handle it separately via $TEAMCITY_SUPERUSER_TOKEN further down\nif [ -n \"${TEAMCITY_TOKEN:-}\" ]; then\n    export TOKEN=\"$TEAMCITY_TOKEN\"\nelse\n    # XXX: might have to disable this is configuring CORS, see here:\n    #\n    # https://www.jetbrains.com/help/teamcity/rest-api.html#CORS-support\n    #\n    # for HTTP basic auth, set this to force it\n    url_base+=\"/httpAuth\"\nfi\n\nurl_base+=\"/app/rest\"\n\n# fix to a specific API version, but teamcity only supports one legacy API version so this is more likely to break things over time :-/  (currently 2018.1)\nif [ -n \"${TEAMCITY_API_VERSION:-}\" ]; then\n    #url_base+=\"/2018.1\"\n    url_base+=\"/${TEAMCITY_API_VERSION##/}\"\nfi\n\n# use superuser token override to support teamcity.sh when token has already been created but we cannot get it's key value out of the API, so need to continue using superuser token\nif [ -n \"${TEAMCITY_SUPERUSER_TOKEN:-}\" ]; then\n    # XXX: superuser token can only be used with blank user which cannot be used with curl_auth.sh\n    curl -u \":$TEAMCITY_SUPERUSER_TOKEN\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nelse\n    \"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\nfi\n#chmod 0600 \"$cookie_jar\"\n\n# args: /swagger.json | jq .\n# args: /server | jq .  # get all the API details, takes a moment to query\n# args: /projects | jq .\n# args: /users | jq .  # you might get a 403 Forbidden\n# args: /application.wadl | jq .  # 406\n# args: /agents | jq .\n# args: /agents/id:10 | jq.\n# args: /builds | jq .\n# args: /builds?locator=status:SUCCESS,tag=dev | jq .\n# args: /vcs-roots | jq .\n# run: teamcity_api.sh /vcs-roots | jq -r '.\"vcs-root\"[].id' | while read -r id; do teamcity_api.sh \"/vcs-roots/id:$id\"; break; done | jq .\n# args: /cloud/profiles\n"
  },
  {
    "path": "teamcity/teamcity_builds.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-27 18:55:40 +0100 (Thu, 27 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Build+Requests\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the last 100 Teamcity builds and their results via the Teamcity API\n\nOutput format:\n\n<Number>    <BuildType_ID>    <Build_ID>    <State>    <Status>\n\nSpecify \\$NO_HEADER to omit the header line\n\nSee adjacent teamcity_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n{\nif [ -z \"${NO_HEADER:-}\" ]; then\n    printf 'Num\\tBuildType_ID\\tBuild_ID\\tState\\tStatus\\n'\nfi\n\"$srcdir/teamcity_api.sh\" /builds |\njq -r '.build[] | [.number, .buildTypeId, .id, .state, .status] | @tsv'\n} |\ncolumn -t\n"
  },
  {
    "path": "teamcity/teamcity_buildtype_create.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Build+Configuration+And+Template+Settings\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a TeamCity BuildType (build pipeline) from a local JSON configuration file\n\nUses the adjacent teamcity_api.sh\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<build.json>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbuild_file=\"$1\"\n\nif ! [ -f \"$build_file\" ]; then\n    die \"ERROR: build file '$build_file' does not exit\"\nfi\n\nif ! jq < \"$build_file\" > /dev/null; then\n    die \"ERROR: build file '$build_file' is not valid json\"\nfi\n\n# XXX: technically this could have a different project ID and project name, in which case this won't work, they're assumed to usually be the same - if this is the case you must create the Project by hand, this should error out on the last line with an error if you've created your project name and id to be different\ntimestamp \"determining project from build file\"\nproject=\"$(jq -r '.project.name' < \"$build_file\")\"\n\n\"$srcdir/teamcity_create_project.sh\" \"$project\"\necho\n\ntimestamp \"uploading build '$build_file' to TeamCity\"\n\n# XXX: can't export arrays in Bash :-( - must pass as a string and split inside teamcity_api.sh\n# Update: unfortunately removing --fail causes curl to not error out properly, so other scripts that depend on this would not be notified of the failure, so only do this when debugging\n#export CURL_OPTS=\"-sS\" # this overrides teamcity_api.sh to not include --fail so we can get decent error messages here\n\n# create build type using the given JSON configuration file\n\n# API doesn't output newline, so we insert one ourselves to not mess up terminal output\nset +e\n\"$srcdir/teamcity_api.sh\" \"/buildTypes\" -X POST -d @\"$build_file\"\nexitcode=$?\nset -e\necho\nexit $exitcode\n"
  },
  {
    "path": "teamcity/teamcity_buildtype_set_description_from_github.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-06 22:49:36 +0000 (Sun, 06 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor a given TeamCity buildtype, finds the top GitHub VCS root and sync's the description from the GitHub repo to the TeamCity buildtype\n\nTeamCity buildType ID is case sensitive\n\nSee Also:\n\n    teamcity_buildtypes.sh - lists the build types showing ID, project and name\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<buildType_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nbuildtype=\"$1\"\n\nbuildtypes=\"$(\"$srcdir/teamcity_api.sh\" \"/buildTypes\" | jq -r '.buildType[].id')\"\n\nif ! grep -Fxq \"$buildtype\" <<< \"$buildtypes\"; then\n    die \"TeamCity buildtype with ID '$buildtype' does not exist. Please run teamcity_buildtypes.sh to see the valid list of buildTypes and their IDs\"\nfi\n\nvcs_root_ids=\"$(\"$srcdir/teamcity_api.sh\" \"/buildTypes/$buildtype\" | jq -r '.[\"vcs-root-entries\"][\"vcs-root-entry\"][][\"vcs-root\"][\"id\"]')\"\n\nvcs_root_id=\"\"\nrepo=\"\"\nfor id in $vcs_root_ids; do\n    url=\"$(\"$srcdir/teamcity_api.sh\" \"/vcs-roots/$id\" | jq -r '.properties.property[] | select(.name == \"url\") | .value')\"\n    shopt -s nocasematch\n    if [[ \"$url\" =~ ^https://github.com/ ]]; then\n        vcs_root_id=\"$id\"\n        repo=\"$(perl -pne 's|^https://github.com/||i' <<< \"$url\")\"\n    fi\n    shopt -u nocasematch\ndone\n\nif [ -z \"$vcs_root_id\" ]; then\n    die \"No GitHub.com VCS root url found in any of the attached VCS roots for buildType '$buildtype'\"\nfi\n\ntimestamp \"Sync'ing TeamCity buildtype '$buildtype' description from GitHub repo '$repo'\"\ngithub_description=\"$(\"$srcdir/../github/github_repo_description.sh\" \"$repo\" | cut -d $'\\t' -f2-)\"\n\"$srcdir/teamcity_api.sh\" \"/buildTypes/$buildtype/description\" -X PUT -d \"$github_description\" -H \"Content-Type: text/plain\" -H \"Accept: text/plain\" >/dev/null\ntimestamp \"Description set to '$github_description'\"\n"
  },
  {
    "path": "teamcity/teamcity_buildtypes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-04 17:10:48 +0000 (Fri, 04 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Build+Requests\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists the Teamcity BuildTypes (pipelines) via the Teamcity API\n\nOutput format:\n\n<BuildType_ID>    <Project>    <BuildType_Name>\n\nSpecify \\$NO_HEADER to omit the header line\n\nSee adjacent teamcity_api.sh for authentication details\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\n{\nif [ -z \"${NO_HEADER:-}\" ]; then\n    printf 'BuildType_ID\\tProject\\tBuildType_Name\\n'\nfi\n\"$srcdir/teamcity_api.sh\" /buildTypes |\njq -r '.buildType[] | [.id, .projectId, .name] | @tsv'\n} |\n# the $'' quoting evaluates the tab \\t properly - has to be single not double quotes\ncolumn -t -s $'\\t'\n# POSIX, but above works just fine in bash\n#column -t -s \"$(printf '\\t')\"\n"
  },
  {
    "path": "teamcity/teamcity_buildtypes_set_description_from_github.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-07 17:09:00 +0000 (Mon, 07 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFor each TeamCity buildType in the given project, or all projects if none given, attempt to sync the first VCS GitHub repo description to the buildType description\n\nSee Also:\n\n    teamcity_api.sh - see here for connection and authentication details\n    teamcity_projects.sh - lists projects and their IDs\n    teamcity_buildtype_set_description_from_github.sh - sets a single buildtype's description to match its first GitHub VCS\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nproject=\"${1:-}\"\n\nif [ -n \"$project\" ]; then\n    echo \"$project\"\nelse\n    \"$srcdir/teamcity_api.sh\" /projects |\n    jq -r '.project[].id'\nfi |\nwhile read -r project_id; do\n    timestamp \"getting list of buildtypes in project '$project_id'\"\n    echo\n    \"$srcdir/teamcity_api.sh\" \"/projects/$project_id\" |\n    jq -r '.buildTypes.buildType[].id' |\n    while read -r buildtype_id; do\n        \"$srcdir/teamcity_buildtype_set_description_from_github.sh\" \"$buildtype_id\"\n        echo\n    done\n    echo\ndone\n"
  },
  {
    "path": "teamcity/teamcity_create_github_oauth_connection.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-06 18:03:20 +0000 (Sun, 06 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a TeamCity GitHub OAuth connection configuration in the Root project\n\nRequires \\$TEAMCITY_GITHUB_CLIENT_ID and \\$TEAMCITY_GITHUB_CLIENT_SECRET environment variables to be declared\n\nSet up your GitHub OAuth application here to obtain these credentials:\n\n    https://github.com/settings/developers\n\n\nVCS Root created in the Root project can be seen here:\n\n    \\$TEAMCITY_URL/admin/editProject.html?projectId=_Root&tab=projectVcsRoots\n\n\nCalls the connection GitHub.com as it would if created in the UI for better deduplication\nand also to avoid a prefix of (GitHub.com) from being added.\n\n\nThis is useful for (re)creating projects from scratch using VCS stored configuration\n\n\nIdempotent - if a connection called GitHub.com already exists, skips creation and exits with success code zero.\n\n\nUses the adjacent teamcity_api.sh\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\ncheck_env_defined \"TEAMCITY_GITHUB_CLIENT_ID\"\ncheck_env_defined \"TEAMCITY_GITHUB_CLIENT_SECRET\"\n\n# calling the GitHub OAuth connector this name matches the default you'd set by hand, leading to better deduplication\n# and also it prevents a (GitHub.com) prefix for using any other name\nname=\"GitHub.com\"\n\nif \"$srcdir/teamcity_api.sh\" /projects/_Root |\n   jq -r '.projectFeatures.projectFeature[].properties.property[] | select(.name == \"displayName\") | .value' |\n   grep -Fxqi \"$name\"; then\n    timestamp \"TeamCity GitHub OAuth connection in Root project already exists, skipping...\"\n    exit 0\nfi\n\ntimestamp \"Creating TeamCity GitHub OAuth connection in Root project\"\n\"$srcdir/teamcity_api.sh\" \"/projects/_Root/projectFeatures\" -X POST -d @<(cat <<EOF\n    {\n      \"id\": \"GitHub\",\n      \"type\": \"OAuthProvider\",\n      \"properties\": {\n        \"property\": [\n          {\n            \"name\": \"providerType\",\n            \"value\": \"GitHub\"\n          },\n          {\n            \"name\": \"displayName\",\n            \"value\": \"$name\"\n          },\n          {\n            \"name\": \"gitHubUrl\",\n            \"value\": \"https://github.com/\"\n          },\n          {\n            \"name\": \"defaultTokenScope\",\n            \"value\": \"public_repo,repo,repo:status,write:repo_hook\"\n          },\n          {\n            \"name\": \"clientId\",\n            \"value\": \"$TEAMCITY_GITHUB_CLIENT_ID\"\n          },\n          {\n            \"name\": \"secure:clientSecret\",\n            \"value\": \"$TEAMCITY_GITHUB_CLIENT_SECRET\"\n          }\n        ]\n      }\n    }\nEOF\n)\n# API doesn't output newline\necho\ntimestamp \"TeamCity GitHub OAuth connection created in Root project\"\n"
  },
  {
    "path": "teamcity/teamcity_create_project.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Projects+and+Build+Configuration%2FTemplates+Lists\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a TeamCity project\n\nIdempotent - if the named project already exists, skips creation and returns success exit code zero\n\nUses the adjacent teamcity_api.sh\n\nSee teamcity_api.sh for required connection settings and authentication\n\nIf you need to delete a project you can call\n\n    teamcity_api.sh /projects/NAME -X DELETE\n\n\nUnfortunately you can't yet create a project from a saved configuration via the API, see this ticket:\n\n    https://youtrack.jetbrains.com/issue/TW-43542\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_name>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject=\"$1\"\n\ntimestamp \"checking if project '$project' already exists in TeamCity\"\nproject_names=\"$(\"$srcdir/teamcity_api.sh\" \"/projects\" | jq -r '.project[].name')\"\nproject_ids=\"$(\"$srcdir/teamcity_api.sh\" \"/projects\" | jq -r '.project[].id')\"\nif grep -Fxq \"$project\" <<< \"$project_names\" ||\n   grep -Fxq \"$project\" <<< \"$project_ids\"; then\n    timestamp \"project '$project' already exists in TeamCity\"\nelse\n    # XXX: can't export arrays in Bash :-( - must pass as a string and split inside teamcity_api.sh\n    # Update: unfortunately removing --fail causes curl to not error out properly, so other scripts that depend on this would not be notified of the failure, so only do this when debugging\n    #export CURL_OPTS=\"-sS\" # this overrides teamcity_api.sh to not include --fail so we can get decent error messages here\n    timestamp \"creating project '$project' in TeamCity\"\n    set +e\n    # create new empty project\n    \"$srcdir/teamcity_api.sh\" \"/projects/\" \\\n        -X POST \\\n        -H \"Content-Type: text/plain\" \\\n        -d \"$project\"\n        # can't use this for a simpler response, not valid\n        #-H \"Accept: text/plain\" \\\n    # API doesn't output newline, so we insert one ourselves to not mess up terminal output\n    exitcode=$?\n    set -e\n    echo\n    exit $exitcode\nfi\n"
  },
  {
    "path": "teamcity/teamcity_create_vcs_root.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#VCS+Roots\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates a TeamCity VCS root from a saved configuration file (XML or JSON)\n\nThe config file is the same format as downloaded by the adjacent script:\n\n    teamcity_vcs_roots_download.sh\n\nbut you will need to add the password back in to the file before upload (can be blank and set later in UI)\n\n\nUnfortunately TeamCity doesn't allow creating VCS roots using say the GitHub OAuth connection via the API,\nor at least I can find no reference to how to do this - so you might still need to create those type of VCS roots\nby hand or else switch to using another mechanism like SSH keys\n\n\nUses the adjacent teamcity_api.sh\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<vcs_root_config_file>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nconfig=\"$1\"\nshift || :\n\nif ! [ -s \"$config\" ]; then\n    die \"ERROR: file not found or is empty: $config\"\nfi\n\nopts=()\n\nshopt -s nocasematch\nif [[ \"$config\" =~ \\.xml$ ]]; then\n    # teamcity_api.sh defaults to JSON\n    opts+=(-H \"Accept: application/xml\" -H \"Content-Type: application/xml\")\nelse\n    vcs_id=\"$(jq -r .id < \"$config\")\"\n    if \"$srcdir/teamcity_vcs_roots.sh\" | grep -qi \"^${vcs_id}[[:space:]]\"; then\n        timestamp \"VCS root '$vcs_id' already exists, skipping creation\"\n        exit 0\n    fi\nfi\nshopt -u nocasematch\n\ntimestamp \"creating TeamCity VCS root from '$config'\"\n# create VCS root from given configuration\n\"$srcdir/teamcity_api.sh\" \"/vcs-roots\" \\\n    -X POST \\\n    -d @\"$config\" \\\n    \"${opts[@]:-}\"\necho\n"
  },
  {
    "path": "teamcity/teamcity_export.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-09 15:23:27 +0000 (Wed, 09 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExports TeamCity configs for projects, buildTypes and VCS roots to local directories of the same name as each project\n\nThis mimicks the directory structure of TeamCity's datadir and Versioned Settings VCS export integration\n\nProject IDs can be specified as arguments, otherwise iterates over all discovered projects including the Root project\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id1> <project_id2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nbasedir=\"$PWD\"\n\n\n# shellcheck disable=SC2103\nif [ $# -gt 0 ]; then\n    for project_id in \"$@\"; do\n        echo \"$project_id\"\n    done\nelse\n    \"$srcdir/teamcity_projects.sh\" |\n    awk '{print $1}'\nfi |\n# don't use grep -v it can pipefail\nsed '/^[[:space:]]*$/d' |\nsort -u |\nwhile read -r project_id; do\n    projectdir=\"$basedir/$project_id\"\n    mkdir -p -v \"$projectdir\"\n    timestamp \"Exporting TeamCity project '$project_id' to $projectdir\"\n    cd \"$projectdir\"\n    # printed by the script\n    #timestamp \"Exporting project '$project_id' config\"\n    \"$srcdir/teamcity_export_project_config.sh\" \"$project_id\"\n    mkdir -p -v buildTypes\n    cd buildTypes\n    #timestamp \"Exporting project '$project_id' buildTypes\"\n    # restrict buildType exports to only this project\n    TEAMCITY_BUILDTYPES_PROJECT=\"$project_id\" \"$srcdir/teamcity_export_buildtypes.sh\"\n    mkdir -p -v ../vcsRoots\n    cd ../vcsRoots\n    #timestamp \"Exporting project '$project_id' VCS roots\"\n    TEAMCITY_VCS_ROOTS_PROJECT=\"$project_id\" \"$srcdir/teamcity_export_vcs_roots.sh\"\n    cd ..\n    echo\ndone\n"
  },
  {
    "path": "teamcity/teamcity_export_buildtypes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#Build+Configuration+And+Template+Settings\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExports all TeamCity BuildTypes (build pipelines) to local JSON configuration files for backup/restore / migration purposes, or even just to backport changes to Git for revision control tracking\n\nIf arguments are specified then only downloads those named BuildTypes, otherwise finds and downloads all BuildTypes\n\nIf \\$TEAMCITY_BUILDTYPES_PROJECT is set then filters to only export the buildTypes belonging to that project. If the project doesn't exist no buildTypes will be found to export, but it will not error\n\n\nResets buildNumberCounter to 1 in the JSON output to avoid this counter causing non-functional revision control changes\n\n\nUses the adjacent teamcity_api.sh and jq (installed by 'make')\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<buildtype1> <buildtype2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# prevent silent failures\ntrap 'echo ERROR' EXIT\n\nif [ $# -gt 0 ]; then\n    for build_name in \"$@\"; do\n        echo \"$build_name\"\n    done\nelse\n    \"$srcdir/teamcity_api.sh\" /buildTypes |\n    if [ -n \"${TEAMCITY_BUILDTYPES_PROJECT:-}\" ]; then\n        # this multiplies out the results\n        #jq -r \"select(.buildType[].projectId == \\\"$TEAMCITY_BUILDTYPES_PROJECT\\\")\"\n        jq -r \"{ \\\"buildType\\\": [ .buildType[] | select(.projectId == \\\"$TEAMCITY_BUILDTYPES_PROJECT\\\") ] } | @json\"\n    else\n        cat\n    fi |\n    jq -r '.buildType[].id'\nfi |\nsort -u |\n# grep -v breaks pipe if no input eg. no buildTypes in _Root project\nsed '/^[[:space:]]*$/d' |\nwhile read -r build_id; do\n    filename=\"$build_id.json\"\n    timestamp \"Exporting build '$build_id' to '$filename'\"\n    output=\"$(\"$srcdir/teamcity_api.sh\" \"/buildTypes/$build_id\")\"\n    # using jq just for formatting\n    #jq . > \"$filename\" || :  # some builds get 400 errors, ignore these\n    # reset the buildNumberCounter to 1 every time so that we don't incur pointless revision changes\n    build_number_counter_index=\"$(jq '.settings.property | map(.name == \"buildNumberCounter\") | index(true)' <<< \"$output\")\"\n    jq -r \".settings.property[$build_number_counter_index].value = \\\"1\\\"\" <<< \"$output\" |\n    # normalize the href's as they can be /app/rest or /httpAuth/app/rest depending on how you query it\n    sed 's|/httpAuth/app/rest/|/app/rest/|' |\n    cat > \"$filename\"\ndone\n\ntrap '' EXIT\n"
  },
  {
    "path": "teamcity/teamcity_export_project_config.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#project+Configuration+And+Template+Settings\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExports all TeamCity Projects to local JSON configuration files for backup/restore / migration purposes, or even just to backport changes to Git for revision control tracking\n\nIf arguments are specified then only downloads those named Projects, otherwise finds and downloads all Projects settings.\n\nIf a single project is given, the export filename is called project-settings.json in line with standard TeamCity exports, but if multiple projects are given or the project list is queried from TeamCity then the export file is called <project>.json to differentiate them.\n\n\nUses the adjacent teamcity_api.sh and jq (installed by 'make')\n\nSee teamcity_api.sh for required connection settings and authentication\n\nSee teamcity_projects.sh for the list of projects and their IDs vs Names\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<project_id1> <project_id2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# use this to figure out if we should set filename to <project>.json or leave as the single standard project-config.json - this is important for bulk teamcity_export.sh per project to mimick the official TeamCity exports and Versioned Settings sync directory structure\nexport projects=(\"$@\")\nfilename=\"project-config.json\"\n\nif [ $# -gt 0 ]; then\n    for project_id in \"$@\"; do\n        echo \"$project_id\"\n    done\nelse\n    \"$srcdir/teamcity_api.sh\" /projects |\n    jq -r '.project[] | [.id, .name] | @tsv'\nfi |\n# grep -v breaks pipe if no input, prefer sed\nsed '/^[[:space:]]*$/d' |\nwhile read -r project_id project_name; do\n    # basing the filename off the ID instead of the Name is because it's more suitable for filenames\n    # instead of '<Root>.json', '_Root.json' is safer and easier to use in daily practice\n    if [ ${#projects[@]} -gt 1 ]; then\n        filename=\"$project_id.json\"\n    fi\n    project_name=\"${project_name:-$project_id}\"\n    timestamp \"Exporting project '$project_name' config to '$filename'\"\n    #project_name=\"$(\"$srcdir/../bin/urlencode.sh\" <<< \"$project_name\")\"\n    #\"$srcdir/teamcity_api.sh\" \"/projects/$project_name\" |\n    \"$srcdir/teamcity_api.sh\" \"/projects/$project_id\" |\n    # using jq just for formatting\n    jq |\n    # normalize the href's as they can be /app/rest or /httpAuth/app/rest depending on how you query it\n    sed 's|/httpAuth/app/rest/|/app/rest/|' |\n    cat > \"$filename\"\ndone\n"
  },
  {
    "path": "teamcity/teamcity_export_vcs_roots.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#vcs_root+Configuration+And+Template+Settings\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExports all TeamCity VCS roots to local JSON configuration files for backup/restore / migration purposes, or even just to backport changes to Git for revision control tracking\n\nIf arguments are specified then only downloads those named VCS roots, otherwise finds and downloads all VCS roots\n\nIf \\$TEAMCITY_VCS_ROOTS_PROJECT is set then filters to only export the vcsRoots belonging to that project. If the project doesn't exit the API will return a 404 error\n\n\nUses the adjacent teamcity_api.sh and jq (installed by 'make')\n\nSee teamcity_api.sh for required connection settings and authentication\n\nSee teamcity_vcs_roots.sh for the list of vcs_roots and their IDs vs Names\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<vcs_root_id1> <vcs_root_id2> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# prevent silent failures\ntrap 'echo ERROR' EXIT\n\nif [ $# -gt 0 ]; then\n    for vcs_root_id in \"$@\"; do\n        echo \"$vcs_root_id\"\n    done\nelse\n    url_path=\"/vcs-roots\"\n    if [ -n \"${TEAMCITY_VCS_ROOTS_PROJECT:-}\" ]; then\n        # XXX: this will 404 if the project doesn't exist\n        url_path+=\"?locator=project:$TEAMCITY_VCS_ROOTS_PROJECT\"\n    fi\n    \"$srcdir/teamcity_api.sh\" \"$url_path\" |\n    jq -r '.[\"vcs-root\"][] | [.id, .name] | @tsv'\nfi |\nsort -u |\n# grep -v breaks pipe if no input eg. no vcs roots in _Root project\nsed '/^[[:space:]]*$/d' |\nwhile read -r vcs_root_id vcs_root_name; do\n    # basing the filename off the ID instead of the Name is because it's more suitable for filenames\n    # instead of 'github.com/harisekhon/blah (1)', 'MyProject_GithubComHariSekhonBlah1 ' is safer and easier to use in daily practice\n    filename=\"$vcs_root_id.json\"\n    vcs_root_name=\"${vcs_root_name:-$vcs_root_id}\"\n    timestamp \"Exporting vcs_root '$vcs_root_name' to '$filename'\"\n    \"$srcdir/teamcity_api.sh\" \"/vcs-roots/$vcs_root_id\" |\n    # using jq just for formatting\n    jq |\n    # normalize the href's as they can be /app/rest or /httpAuth/app/rest depending on how you query it\n    sed 's|/httpAuth/app/rest/|/app/rest/|' |\n    cat > \"$filename\"\ndone\n\ntrap '' EXIT\n"
  },
  {
    "path": "teamcity/teamcity_project_set_versioned_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-07 18:13:22 +0000 (Mon, 07 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCreates TeamCity versioning for a given project\n\nRequires a VCS root to already be configured and available - the ID provided as the second arg (defaults to 'TeamCity')\n\nRecommend you create this VCS in the Root project and only enable on sub-projects since VCS sync with credentials omitted breaks its own VCS if you store it within the same project\n\n\nIdempotent - detects if versioning is already configured for the given project and skips creation if so\n\n\nIf you want to force feature replacement of an already existing versionSetting configuration, set TEAMCITY_FORCE_FEATURE_REPLACE=1\n\n\nSee Also:\n\n    teamcity_api.sh - for connection and authentication requirements, used by this script\n    teamcity_projects.sh - lists projects and their IDs\n    teamcity_vcs_roots.sh - lists VCS roots and their IDs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<project_id> [<vcs_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject_id=\"$1\"\nvcs_id=\"${2:-TeamCity}\"\n\nfeature_id=\"$(\"$srcdir/teamcity_api.sh\" \"/projects/$project_id\" |\n              jq -r '.projectFeatures.projectFeature[]? | select(.type == \"versionedSettings\") | .id')\"\nif [ -n \"$feature_id\" ]; then\n    if [ -n \"${TEAMCITY_FORCE_FEATURE_REPLACE:-}\" ]; then\n        timestamp \"deleting all feature ids of type versionedSettings for project '$project_id'\"\n        for id in $feature_id; do\n            \"$srcdir/teamcity_api.sh\" \"/projects/$project_id/projectFeatures/id:$id\" -X DELETE\n        done\n    else\n        timestamp \"TeamCity project '$project_id' is already configured for versionedSettings with id '$feature_id', skipping\"\n        exit 0\n    fi\nfi\n\n# XXX: setting buildSettings to PREFER_VCS makes no different - TeamCity still demands write access to even load the config from VCS\ntimestamp \"Creating version integration settings to VCS '$vcs_id' for project '$project_id'\"\n\"$srcdir/teamcity_api.sh\" \"/projects/$project_id/projectFeatures\" -X POST -d @<(cat <<EOF\n      {\n        \"type\": \"versionedSettings\",\n        \"properties\": {\n          \"property\": [\n            {\n              \"name\": \"buildSettings\",\n              \"value\": \"PREFER_CURRENT\"\n            },\n            {\n              \"name\": \"credentialsStorageType\",\n              \"value\": \"credentialsJSON\"\n            },\n            {\n              \"name\": \"enabled\",\n              \"value\": \"true\"\n            },\n            {\n              \"name\": \"rootId\",\n              \"value\": \"$vcs_id\"\n            },\n            {\n              \"name\": \"showChanges\",\n              \"value\": \"true\"\n            }\n          ]\n        }\n      }\nEOF\n)\n"
  },
  {
    "path": "teamcity/teamcity_project_vcs_versioning.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-07 16:31:41 +0000 (Mon, 07 Dec 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nEnable or disable TeamCity VCS versioning for a project\n\nUseful to stop auto-committing to GitHub when you're testing settings\n\nDefaults to enable since you only want to disable VCS explicitly\n\n\nSee teamcity_api.sh for connection and authentication settings\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<projectID> [<enable|disable>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nproject=\"$1\"\n\ntoggle=\"${2:-enable}\"\n\nif [[ \"$toggle\" =~ ^disabled?$ ]]; then\n    value=\"false\"\nelse\n    value=\"true\"\nfi\n\nfeature_id=\"$(\"$srcdir/teamcity_api.sh\" \"/projects/$project\" | jq -r '.projectFeatures.projectFeature[] | select(.type == \"versionedSettings\") | .id')\"\n\n# XXX: there is a slight race condition between getting and setting property back but the TeamCity API won't allow me to send only the enabled setting back - trying to send this resulted in wiping out the versionedSettings config\n#\"$srcdir/teamcity_api.sh\" \"/projects/$project/projectFeatures/id:$feature_id\" -X PUT -d '{\"type\": \"versionedSettings\", \"properties\": { \"property\": [ { \"name\": \"enabled\", \"value\": \"true\" } ] } }'\n\n# having to get whole property config, mangle and send back instead :'-(\n\nsettings=\"$(\"$srcdir/teamcity_api.sh\" \"/projects/$project/projectFeatures/id:$feature_id\")\"\n\nindex_of_property_enabled=\"$(jq '.properties.property | map(.name == \"enabled\") | index(true)' <<< \"$settings\")\"\n\nnew_settings=\"$(jq \".properties.property[$index_of_property_enabled].value = \\\"$value\\\"\" <<< \"$settings\")\"\n\n\"$srcdir/teamcity_api.sh\" \"/projects/$project/projectFeatures/id:$feature_id\" -X PUT -d \"$new_settings\"\n"
  },
  {
    "path": "teamcity/teamcity_projects.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#project+Configuration+And+Template+Settings\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists TeamCity Projects - useful to find the IDs needed to download specific projects using teamcity_projects_download.sh\n\nOutput Format:\n\n<project_id>    <project_name>\n\nUses the adjacent teamcity_api.sh and jq (installed by 'make')\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n\"$srcdir/teamcity_api.sh\" /projects |\njq -r '.project[] | [.id, .name] | @tsv'\n"
  },
  {
    "path": "teamcity/teamcity_upload_ssh_key.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-01-11 13:14:37 +0000 (Mon, 11 Jan 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nUploads SSH private key to TeamCity via the Web UI\n\nThere's no TeamCity Rest API support for SSH key management at the time of writing so this script posts to the Web UI :'-(\n\n    https://youtrack.jetbrains.com/issue/TW-42311\n\nIf no project is specified as the 3rd arg, uploads to the Root project:\n\n    \\$TEAMCITY_URL/admin/editProject.html?projectId=_Root&tab=ssh-manager#\n\n\nThis is useful for (re)connecting VCS configurations using SSH auth which can be used to load entire teamcity projects\n\n\nThe SSH private key must be in standard PEM format to be accepted by TeamCity (generated via 'ssh-keygen -m PEM') rather than new non-standard OpenSSH format\n\n    https://youtrack.jetbrains.com/issue/TW-53615\n\nIdempotent - if the named SSH key already exists, will replace it and exit with success code zero\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<key_file> [<key_name> <teamcity_project_id>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nssh_private_key=\"$1\"\n\nname=\"${2:-${ssh_private_key##*/}}\"\n\n# defaults to the Root project because this is the best place to use external VCS connections and auth from, to prevent them being reset and broken by import from VCS which by default excludes secrets\nproject_id=\"${3:-_Root}\"\n\nurl_path=\"\"\n\nif [ -n \"${TEAMCITY_URL:-}\" ]; then\n    url_base=\"${TEAMCITY_URL%%/}\"\nelse\n    protocol=\"http\"\n    if [ -n \"${TEAMCITY_SSL:-}\" ]; then\n        protocol=\"https\"\n    fi\n    [ -n \"${TEAMCITY_HOST:-}\" ] || usage \"neither \\$TEAMCITY_URL nor \\$TEAMCITY_HOST defined in environment\"\n    host=\"$TEAMCITY_HOST\"\n    port=\"${TEAMCITY_PORT:-8111}\"\n    url_base=\"$protocol://$host:$port\"\nfi\n\nteamcity_curl_auth(){\n    local url_path=\"$1\"\n    shift || :\n    local curl_opts=(-sS --fail --connect-timeout 5)\n    # use superuser token override to support teamcity.sh when token has already been created but we cannot get it's key value out of the API, so need to continue using superuser token\n    if [ -n \"${TEAMCITY_SUPERUSER_TOKEN:-}\" ]; then\n        # XXX: superuser token can only be used with blank user which cannot be used with curl_auth.sh\n        curl -u \":$TEAMCITY_SUPERUSER_TOKEN\" \"$url_base/$url_path\" \"${curl_opts[@]}\" \"$@\"\n    else\n        \"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${curl_opts[@]}\" \"$@\"\n    fi\n}\n\n\ntimestamp \"Uploading TeamCity SSH key '$ssh_private_key' to project '$project_id' called '$name' on TeamCity server at '$url_base'\"\noutput=\"$(teamcity_curl_auth /admin/sshKeys.html -X POST \\\n                                                 -F \"action=createSshKey\" \\\n                                                 -F \"projectId=$project_id\" \\\n                                                 -F \"fileName=$name\" \\\n                                                 -F \"file:fileToUpload=@$ssh_private_key\"\n)\"\necho\nif grep -i error <<< \"$output\"; then\n    echo\n    timestamp \"ERROR: TeamCity SSH key upload FAILED\"\n    exit 1\nfi\n\ntimestamp \"TeamCity SSH key '$name' created in '$project_id' project\"\n"
  },
  {
    "path": "teamcity/teamcity_vcs_roots.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-11-30 19:06:40 +0000 (Mon, 30 Nov 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.jetbrains.com/help/teamcity/rest-api-reference.html#VCS+Roots\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists TeamCity VCS roots - useful to find the IDs needed to download specific vcs roots using teamcity_vcs_roots_download.sh\n\nOutput Format:\n\n<vcs_root_id>    <vcs_root_name>\n\nUses the adjacent teamcity_api.sh and jq (installed by 'make')\n\nSee teamcity_api.sh for required connection settings and authentication\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# XXX: this key vcs-roots doesn't exist if there are none configured, eg. during teamcity.sh bootstrap, resulting in the old error\n# jq: error (at <stdin>:0): Cannot iterate over null (null)\n#\n#output=\"$(\"$srcdir/teamcity_api.sh\" /vcs-roots)\"\n#\n#count=\"$(jq -r '.count' <<< \"$output\")\"\n#\n#if [ \"$count\" -gt 0 ]; then\n#    jq -r '.[\"vcs-root\"][] | [.id, .name] | @tsv' <<< \"$output\"\n#fi\n\n# using special ? suffix operator now\n\"$srcdir/teamcity_api.sh\" /vcs-roots |\njq -r '.[\"vcs-root\"]? | .[]? | [.id, .name] | @tsv'\n"
  },
  {
    "path": "terraform/terraform_cloud_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-11-12 14:37:10 +0000 (Fri, 12 Nov 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Terraform Cloud API\n\nAuthentication requites the environment variable \\$TERRAFORM_TOKEN to be set\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nSet up a personal access token here:\n\n    https://app.terraform.io/app/settings/tokens\n\n\nAPI Reference:\n\n    https://www.terraform.io/docs/cloud/api/index.html\n\n\nFor convenience, the following tokens in the format ':token' or '{token}' are automatically replaced if the environment variables are available, for easy copy-pasting from API documentation:\n\n:org, :organization, :organization_name         \\$TERRAFORM_ORGANIZATION\n:workspace, :workspace_id                       \\$TERRAFORM_WORKSPACE\n:user, :userid, :user_id                        \\$TERRAFORM_USER_ID, otherwise must make extra call to API to determine\n\n\nExamples:\n\n\n# Get Account Details for the currently authenticated user:\n\n    ${0##*/} /account/details | jq .\n\n\n# Get Account Details for a given user (doesn't contain email like /account/details self-describing endpoint):\n\n    ${0##*/} /users/{userid} | jq .\n\n\n# Get your user ID (export as \\$TERRAFORM_USER_ID for other queries):\n\n    ${0##*/} /account/details | jq .data.id\n\n    export TERRAFORM_USER_ID=\\\"\\$(${0##*/} /account/details | jq .data.id)\\\"\n\n\n# List organizations:\n\n    ${0##*/} /organizations | jq .\n\n\n# List workspaces:\n\n    ${0##*/} /organizations/:organization_name/workspaces | jq .\n\n\n# List an organization's variable sets:\n\n    ${0##*/} /organizations/:organization_name/varsets | jq .\n\n\n# List workspace variables (see terraform_cloud_workspace_set_vars.sh for an easy way to add/update them):\n\n    ${0##*/} /workspaces/:workspace_id/vars | jq .\n\n# See terraform_cloud_*_vars.sh for easier listing/adding/updating/deleting variables in workspaces and variable sets\n\n\n# Lock/Unlock a workspace:\n\n    ${0##*/} /workspaces/:workspace_id/actions/lock -X POST | jq .\n    ${0##*/} /workspaces/:workspace_id/actions/unlock -X POST | jq .\n\n\n# List workspace resources:\n\n    ${0##*/} /workspaces/:workspace_id/resources | jq .\n\n\n# IP Ranges used by Terraform Cloud (see terraform_cloud_ip_ranges.sh for a processed version, one per line):\n\n    ${0##*/} /meta/ip-ranges | jq .\n\n\n# Registry Modules for an org:\n\n    ${0##*/} /organizations/{org}/registry-modules | jq .\n\n\n# Get Agent Pools:\n\n    ${0##*/} /organizations/{organization}/agent-pools | jq .\n\n\n# Get Audit Trails:\n\n    ${0##*/} /organization/audit-trail | jq .\n\n\n# Get Feature Sets:\n\n    ${0##*/} /feature-sets | jq .\n\n\n# Get your user authentication tokens (eg. to check for old tokens programmatically):\n\n    ${0##*} /users/{userid}/authentication-tokens | jq .\n\n\n# Create a new user token:\n\n    ${0##*} /users/{userid}/authentication-tokens -X POST | jq .\n\n# Delete a user token:\n\n    ${0##*} /authentication-tokens/{id} -X DELETE | jq .\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://app.terraform.io/api/v2\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncurl_api_opts \"$@\"\n\ncheck_env_defined TERRAFORM_TOKEN\n\nexport TOKEN=\"$TERRAFORM_TOKEN\"\n\nurl_path=\"${1:-}\"\nshift || :\n\n# stripping url_base for convenience in case copying and pasting from docs\nurl_path=\"${url_path//https:\\/\\/app.terraform.io\\/api\\/v2}\"\nurl_path=\"${url_path##/}\"\n\nif [[ \"$url_path\" =~ [\\{:]org(anization)?(_?name)?\\}? ]]; then\n    if [ -n \"${TERRAFORM_ORGANIZATION:-}\" ]; then\n        url_path=\"${url_path//:organization_name/$TERRAFORM_ORGANIZATION}\"\n        url_path=\"${url_path//:organization/$TERRAFORM_ORGANIZATION}\"\n        url_path=\"${url_path//:org/$TERRAFORM_ORGANIZATION}\"\n        url_path=\"${url_path//\\{organization_name\\}/$TERRAFORM_ORGANIZATION}\"\n        url_path=\"${url_path//\\{organization\\}/$TERRAFORM_ORGANIZATION}\"\n        url_path=\"${url_path//\\{org\\}/$TERRAFORM_ORGANIZATION}\"\n    else\n        die \"organization placeholder found but \\$TERRAFORM_ORGANIZATION is not set\"\n    fi\nfi\n\nif [[ \"$url_path\" =~ [\\{:]workspace(_?id)?\\}? ]]; then\n    if [ -n \"${TERRAFORM_WORKSPACE:-}\" ]; then\n        url_path=\"${url_path//:workspace_id/$TERRAFORM_WORKSPACE}\"\n        url_path=\"${url_path//:workspace/$TERRAFORM_WORKSPACE}\"\n        url_path=\"${url_path//\\{workspace_id\\}/$TERRAFORM_WORKSPACE}\"\n        url_path=\"${url_path//\\{workspace\\}/$TERRAFORM_WORKSPACE}\"\n    else\n        die \"workspace placeholder found but \\$TERRAFORM_WORKSPACE is not set\"\n    fi\nfi\n\nif [[ \"$url_path\" =~ [\\{:]?user(_?id)?\\}? ]]; then\n    if [ -n \"${TERRAFORM_USER_ID:-}\" ]; then\n        url_path=\"${url_path//:user_id/$TERRAFORM_USER_ID}\"\n        url_path=\"${url_path//:userid/$TERRAFORM_USER_ID}\"\n        url_path=\"${url_path//:user/$TERRAFORM_USER_ID}\"\n        url_path=\"${url_path//\\{user\\}/$TERRAFORM_USER_ID}\"\n        url_path=\"${url_path//\\{userid\\}/$TERRAFORM_USER_ID}\"\n        url_path=\"${url_path//\\{user_id\\}/$TERRAFORM_USER_ID}\"\n    else\n        user_id=\"$(\"$srcdir/../bin/curl_auth.sh\" \"${CURL_OPTS[@]}\" \"$url_base/account/details\" | jq -r .data.id)\"\n        url_path=\"${url_path//:user_id/$user_id}\"\n        url_path=\"${url_path//:userid/$user_id}\"\n        url_path=\"${url_path//:user/$user_id}\"\n        url_path=\"${url_path//\\{user\\}/$user_id}\"\n        url_path=\"${url_path//\\{userid\\}/$user_id}\"\n        url_path=\"${url_path//\\{user_id\\}/$user_id}\"\n    fi\nfi\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "terraform/terraform_cloud_ip_ranges.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-20 13:32:36 +0000 (Mon, 20 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/ip-ranges\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nReturns the list of IP ranges that Terraform Cloud may use via the API\n\nCan optionally return just the IP lists for one or more of the following range types:\n\n    api\n    notifications\n    sentinel\n    vcs\n\nFor more details, see:\n\n    https://www.terraform.io/cloud-docs/api-docs/ip-ranges\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<range_type> <range_type> ...]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nfor range_type in \"$@\"; do\n    if ! [[ \"$range_type\" =~ ^(api|notifications|sentinel|vcs)$ ]]; then\n        usage \"invalid range type given, must be one of: api, notifications, sentinel, vcs\"\n    fi\ndone\n\ndata=\"$(curl -sS https://app.terraform.io/api/meta/ip-ranges)\"\n\nif [ -n \"${DEBUG:-}\" ]; then\n    jq . <<< \"$data\" >&2\nfi\n\nif [ $# -gt 0 ]; then\n    for range_type in \"$@\"; do\n        jq -r \".${range_type}[]\" <<< \"$data\"\n    done\nelse\n    jq -r '.[][]' <<< \"$data\"\nfi |\nsort -u\n"
  },
  {
    "path": "terraform/terraform_cloud_organizations.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args:\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/organizations\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Terraform Cloud organization IDs (needed by many adjacent scripts)\n\nOutput:\n\n<id>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\n# TODO: add pagination support\n\"$srcdir/terraform_cloud_api.sh\" \"/organizations\" |\njq_debug_pipe_dump |\njq -r '.data[].id'\n"
  },
  {
    "path": "terraform/terraform_cloud_varset_delete_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: :organization $TERRAFORM_VARSET_ID haritest\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/variable-sets\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes one or more variables in a given Terraform Cloud variable set id\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of variable sets and their IDs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> <varset_id> <variable_name> [<variable_name2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 3 \"$@\"\n\norg=\"$1\"\nvarset_id=\"$2\"\nshift || :\nshift || :\n\nif [ -z \"$varset_id\" ]; then\n    usage \"no terraform varset id given\"\nfi\n\n\"$srcdir/terraform_cloud_varset_vars.sh\" \"$org\" \"$varset_id\" |\nwhile read -r varset_id varset_name id _ _ name _; do\n    for var in \"$@\"; do\n        if [ \"$var\" = \"$name\" ]; then\n            timestamp \"deleting variable '$name' (id '$id') in varset '$varset_name' (id '$varset_id')\"\n            \"$srcdir/terraform_cloud_api.sh\" \"/varsets/$varset_id/relationships/vars/$id\" -X DELETE\n        fi\n    done\ndone\n"
  },
  {
    "path": "terraform/terraform_cloud_varset_set_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: $TERRAFORM_ORGANIZATION $TERRAFORM_VARSET_ID haritest=myvalue\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/variable-sets\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates Terraform Cloud variables for a given variable set from args or stdin\n\nBy default, creates variables as Environment Variables and marks them as Sensitive for safety as the primary use case for this code was easy uploading AWS access key credentials from things like aws_csv_creds.sh\n\nIf you want to create Terraform variables instead:\n\n    export TERRAFORM_VARIABLES=1\n    export TERRAFORM_VARIABLES_HCL=1  # mark the variables as HCL code (implies TERRAFORM_VARIABLES=1)\n\nIf you want to mark the variables as non-sensitive:\n\n    export TERRAFORM_VARIABLES_SENSITIVE=false\n\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of variable sets and their IDs\n\n\nExamples:\n\n    ${0##*/} {varset_id} AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} {varset_id}\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} {varset_id}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> <varset_id> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\norg=\"$1\"\nvarset_id=\"$2\"\nshift || :\nshift || :\n\nif ! [[ \"$varset_id\" =~ ^varset-[[:alnum:]]+$ ]]; then\n    usage \"invalid varset id given: $varset_id - should be in format varset-[[:alnum:]]+\"\nfi\n\nif [ -n \"${TERRAFORM_VARIABLES_HCL:-}\" ]; then\n    TERRAFORM_VARIABLES=1\n    hcl=true\nelse\n    hcl=false\nfi\n\nif [ -n \"${TERRAFORM_VARIABLES:-}\" ]; then\n    category=\"terraform\"\nelse\n    category=\"env\"\nfi\n\nif [ \"${TERRAFORM_VARIABLES_SENSITIVE:-}\" = false ]; then\n    sensitive=false\nelse\n    sensitive=true\nfi\n\nvarsets_env_vars=\"$(\"$srcdir/terraform_cloud_varset_vars.sh\" \"$org\" \"$varset_id\")\"\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    local id\n    # shellcheck disable=SC2154\n    id=\"$(awk \"\\$1 == \\\"$varset_id\\\" && \\$4 == \\\"env\\\" && \\$6 == \\\"$key\\\" {print \\$3}\" <<< \"$varsets_env_vars\")\"\n    varset_name=\"$(awk \"\\$1 == \\\"$varset_id\\\" {print \\$2; exit}\" <<< \"$varsets_env_vars\")\"\n    if [ -n \"$id\" ]; then\n        timestamp \"updating Terraform environment variable '$key' (id: '$id') in variable set '$varset_name' (id '$varset_id')\"\n        # shellcheck disable=SC2154\n        \"$srcdir/terraform_cloud_api.sh\" \"/varsets/$varset_id/relationships/vars/$id\" \\\n            -X PATCH \\\n            -H \"Content-Type: application/vnd.api+json\" \\\n            -d \"{\n                    \\\"data\\\": {\n                        \\\"id\\\": \\\"$id\\\",\n                        \\\"attributes\\\": {\n                            \\\"key\\\": \\\"$key\\\",\n                            \\\"value\\\": \\\"$value\\\",\n                            \\\"category\\\": \\\"$category\\\",\n                            \\\"hcl\\\": $hcl,\n                            \\\"sensitive\\\": $sensitive\n                        },\n                        \\\"type\\\":\\\"vars\\\"\n                    }\n                }\" > /dev/null\n        #echo  # JSON output doesn't end in a newline\n    else\n        timestamp \"adding Terraform environment variable '$key' in variable set '$varset_name' (id '$varset_id')\"\n        \"$srcdir/terraform_cloud_api.sh\" \"/varsets/$varset_id/relationships/vars\" \\\n            -X POST \\\n            -H \"Content-Type: application/vnd.api+json\" \\\n            -d \"{\n                    \\\"data\\\": {\n                        \\\"attributes\\\": {\n                            \\\"key\\\": \\\"$key\\\",\n                            \\\"value\\\": \\\"$value\\\",\n                            \\\"category\\\": \\\"$category\\\",\n                            \\\"hcl\\\": $hcl,\n                            \\\"sensitive\\\": $sensitive\n                        },\n                        \\\"type\\\":\\\"vars\\\"\n                    }\n                }\" |\n                jq_debug_pipe_dump >/dev/null\n        #echo  # JSON output doesn't end in a newline\n    fi\n    echo\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "terraform/terraform_cloud_varset_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args:\n#  args: :organization\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/variable-sets\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Terraform Cloud variables in all variable sets for a given organiztion\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of variable sets and their IDs\n\n\\$TERRAFORM_ORGANIZATION and \\$TERRAFORM_VARSET_ID can be used instead of arguments\n\nOutput:\n\n<varset_id>    <varset_name>    <id>    <type>    <sensitive>    <name>    <value>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<organization> <varset_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\norg=\"${1:-${TERRAFORM_ORGANIZATION:-}}\"\nvarset_id=\"${2:-${TERRAFORM_VARSET_ID:-}}\"\n\nif [ -z \"$org\" ]; then\n    usage \"no terraform organization given and TERRAFORM_ORGANIZATION not set\"\nfi\n\n# TODO: add pagination support\nif [ -n \"$varset_id\" ]; then\n    if ! [[ \"$varset_id\" =~ ^varset-[[:alnum:]]+$ ]]; then\n        usage \"invalid varset id given: $varset_id - should be in format varset-[[:alnum:]]+\"\n    fi\n    variable_sets=\"$(\"$srcdir/terraform_cloud_api.sh\" \"/varsets/$varset_id\" | jq -r '.data | [.id, .attributes.name] | @tsv')\"\nelse\n    variable_sets=\"$(\"$srcdir/terraform_cloud_api.sh\" \"/organizations/$org/varsets\" | jq -r '.data[] | [.id, .attributes.name] | @tsv')\"\nfi\n\nwhile read -r varset_id varset_name; do\n    # TODO: add pagination support\n    \"$srcdir/terraform_cloud_api.sh\" \"/varsets/$varset_id/relationships/vars\"  |\n    jq -r \".data[] | [\\\"$varset_id\\\", \\\"$varset_name\\\", .id, .attributes.category, .attributes.sensitive, .attributes.key, .attributes.value] | @tsv\" |\n    column -t\ndone <<< \"$variable_sets\"\n"
  },
  {
    "path": "terraform/terraform_cloud_varsets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args:\n#  args: :org\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/variable-sets\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Terraform Cloud variable sets for a given organization\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\n\n\\$TERRAFORM_ORGANIZATION can be set instead of providing an argument\n\n\nOutput:\n\n<id>    <name>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<organization>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\norg=\"${1:-${TERRAFORM_ORGANIZATION:-}}\"\n\nif [ -z \"$org\" ]; then\n    usage \"no terraform organization given and TERRAFORM_ORGANIZATION not set\"\nfi\n\n# TODO: add pagination support\n\"$srcdir/terraform_cloud_api.sh\" \"/organizations/$org/varsets\" |\njq -r '.data[] | [.id, .attributes.name] | @tsv'\n"
  },
  {
    "path": "terraform/terraform_cloud_workspace_delete_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: :workspace haritest\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/workspace-variables\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes one or more variables in a given Terraform Cloud workspace\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of workspaces and their IDs\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<workspace_id> <variable_name> [<variable_name2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\n\nworkspace_id=\"$1\"\nshift || :\n\nif [ -z \"$workspace_id\" ]; then\n    usage \"no terraform workspace id given\"\nfi\n\n\"$srcdir/terraform_cloud_workspace_vars.sh\" \"$workspace_id\" |\nwhile read -r id _ _ name _; do\n    for var in \"$@\"; do\n        if [ \"$var\" = \"$name\" ]; then\n            timestamp \"deleting variable '$name' (id '$id') in workspace id '$workspace_id'\"\n            \"$srcdir/terraform_cloud_api.sh\" \"/workspaces/$workspace_id/vars/$id\" -X DELETE\n        fi\n    done\ndone\n"
  },
  {
    "path": "terraform/terraform_cloud_workspace_set_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: :workspace haritest=myvalue\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/workspace-variables\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdds / updates Terraform Cloud workspace variables for a given workspace id from args or stdin\n\nBy default, creates variables as Environment Variables and marks them as Sensitive for safety as the primary use case for this code was easy uploading AWS access key credentials from things like aws_csv_creds.sh\n\nIf you want to create Terraform variables instead:\n\n    export TERRAFORM_VARIABLES=1\n    export TERRAFORM_VARIABLES_HCL=1  # mark the variables as HCL code (implies TERRAFORM_VARIABLES=1)\n\nIf you want to mark the variables as non-sensitive:\n\n    export TERRAFORM_VARIABLES_SENSITIVE=false\n\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of workspaces and their IDs\n\n\nExamples:\n\n    ${0##*/} {workspace_id} AWS_ACCESS_KEY_ID=AKIA...\n\n    echo AWS_ACCESS_KEY_ID=AKIA... | ${0##*/} {workspace_id}\n\n\n    Loads both AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY via stdin:\n\n        aws_csv_creds.sh credentials_exported.csv | ${0##*/} {workspace_id}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<workspace_id> [<key>=<value> <key2>=<value2> ...]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nworkspace_id=\"$1\"\nshift || :\n\nif [ -z \"$workspace_id\" ]; then\n    usage \"no terraform workspace id given\"\nfi\n\nif [ -n \"${TERRAFORM_VARIABLES_HCL:-}\" ]; then\n    TERRAFORM_VARIABLES=1\n    hcl=true\nelse\n    hcl=false\nfi\n\nif [ -n \"${TERRAFORM_VARIABLES:-}\" ]; then\n    category=\"terraform\"\nelse\n    category=\"env\"\nfi\n\nif [ \"${TERRAFORM_VARIABLES_SENSITIVE:-}\" = false ]; then\n    sensitive=false\nelse\n    sensitive=true\nfi\n\nenv_vars=\"$(\"$srcdir/terraform_cloud_workspace_vars.sh\" \"$workspace_id\")\"\n\nadd_env_var(){\n    local env_var=\"$1\"\n    parse_export_key_value \"$env_var\"\n    local id\n    # shellcheck disable=SC2154\n    id=\"$(awk \"\\$4 == \\\"$key\\\" {print \\$1}\" <<< \"$env_vars\")\"\n    if [ -n \"$id\" ]; then\n        timestamp \"updating Terraform environment variable '$key' (id: '$id') in workspace '$workspace_id'\"\n        # shellcheck disable=SC2154\n        \"$srcdir/terraform_cloud_api.sh\" \"/workspaces/$workspace_id/vars/$id\" \\\n            -X PATCH \\\n            -H \"Content-Type: application/vnd.api+json\" \\\n            -d \"{\n                    \\\"data\\\": {\n                        \\\"id\\\": \\\"$id\\\",\n                        \\\"attributes\\\": {\n                            \\\"key\\\": \\\"$key\\\",\n                            \\\"value\\\": \\\"$value\\\",\n                            \\\"category\\\": \\\"$category\\\",\n                            \\\"hcl\\\": $hcl,\n                            \\\"sensitive\\\": $sensitive\n                        },\n                        \\\"type\\\":\\\"vars\\\"\n                    }\n                }\" |\n                jq_debug_pipe_dump >/dev/null\n        #echo  # JSON output doesn't end in a newline\n    else\n        timestamp \"adding Terraform environment variable '$key' in workspace '$workspace_id'\"\n        \"$srcdir/terraform_cloud_api.sh\" \"/workspaces/$workspace_id/vars\" \\\n            -X POST \\\n            -H \"Content-Type: application/vnd.api+json\" \\\n            -d \"{\n                    \\\"data\\\": {\n                        \\\"attributes\\\": {\n                            \\\"key\\\": \\\"$key\\\",\n                            \\\"value\\\": \\\"$value\\\",\n                            \\\"category\\\": \\\"$category\\\",\n                            \\\"hcl\\\": $hcl,\n                            \\\"sensitive\\\": $sensitive\n                        },\n                        \\\"type\\\":\\\"vars\\\"\n                    }\n                }\" |\n                jq_debug_pipe_dump >/dev/null\n        #echo  # JSON output doesn't end in a newline\n    fi\n    echo\n}\n\n\nif [ $# -gt 0 ]; then\n    for arg in \"$@\"; do\n        add_env_var \"$arg\"\n    done\nelse\n    while read -r line; do\n        add_env_var \"$line\"\n    done\nfi\n"
  },
  {
    "path": "terraform/terraform_cloud_workspace_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args:\n#  args: :workspace\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/workspace-variables\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Terraform Cloud workspace variables for a given workspace id\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of workspaces and their IDs\n\n\nOutput:\n\n<id>    <type>    <sensitive>    <name>    <value>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<workspace_id>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nworkspace_id=\"${1:-${TERRAFORM_WORKSPACE:-}}\"\n\nif [ -z \"$workspace_id\" ]; then\n    usage \"no terraform workspace id given and TERRAFORM_WORKSPACE not set\"\nfi\n\n# TODO: add pagination support\n\"$srcdir/terraform_cloud_api.sh\" \"/workspaces/$workspace_id/vars\" |\njq_debug_pipe_dump |\njq -r '.data[] | [.id, .attributes.category, .attributes.sensitive, .attributes.key, .attributes.value] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "terraform/terraform_cloud_workspaces.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args:\n#  args: :org\n#\n#  Author: Hari Sekhon\n#  Date: 2021-12-21 13:30:39 +0000 (Tue, 21 Dec 2021)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://www.terraform.io/cloud-docs/api-docs/workspaces\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists Terraform Cloud workspaces for a given organization\n\nSee terraform_cloud_organizations.sh to get a list of organization IDs\nSee terraform_cloud_varsets.sh to get a list of workspaces and their IDs\n\n\nOutput:\n\n<id>    <name>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<organization>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\norg=\"${1:-${TERRAFORM_ORGANIZATION:-}}\"\n\nif [ -z \"$org\" ]; then\n    usage \"no terraform organization given and TERRAFORM_ORGANIZATION not set\"\nfi\n\n# TODO: add pagination support\n\"$srcdir/terraform_cloud_api.sh\" \"/organizations/$org/workspaces\" |\njq_debug_pipe_dump |\njq -r '.data[] | [.id, .attributes.name] | @tsv'\n"
  },
  {
    "path": "terraform/terraform_gcs_backend_version.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 01:28:12 +0000 (Fri, 16 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDetermines the Terraform state version from the tfstate file in a GCS bucket found in a local given backend.tf\n\nParses backend.tf for the bucket and file path to the tfstate file in GCS\n\nThen curls that GCS file's contents and parses it to get the state version\n\nThis is important so that you know what terraform version to set without accidentally updating the client's terraform state, which can potentially break existing clients and CI/CD which will no longer be able to run using the older client version\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<backend.tf>\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\nbackend_tf_file=\"${1:-$PWD/backend.tf}\"\n\nif ! [ -f \"$backend_tf_file\" ]; then\n    die \"File not found: $backend_tf_file - check you are specifying the right path to the .tf file containing the backend definition\"\nfi\n\nlog \"Parsing $backend_tf_file for bucket and prefix\"\n# TODO: check if backend \"gcs\" vs other clouds and extend this to AWS and Azure backends\nbucket=\"$(grep -m1 '^[[:space:]]*bucket[[:space:]]*=.*' \"$backend_tf_file\" | sed 's/.*=//; s/[[:space:]]//g; s/\"//g' || die \"Failed to parse bucket from $backend_tf_file\")\"\nprefix=\"$(grep -m1 '^[[:space:]]*prefix[[:space:]]*=.*' \"$backend_tf_file\" | sed 's/.*=//; s/[[:space:]]//g; s/\"//g' || :)\" # || die \"Failed to parse bucket from $backend_tf_file\")\"\n\nlog \"Fetching and parsing bucket '$bucket' file '$prefix${prefix+/}default.tfstate'\"\n\"$srcdir/../gcp/gcs_curl_file.sh\" \"$bucket\" \"$prefix${prefix+/}default.tfstate\" |\njq -r '.terraform_version'\n"
  },
  {
    "path": "terraform/terraform_gitlab_download_backend_variable.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-02-16 10:55:55 +0000 (Fri, 16 Feb 2024)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\nbackend_variables=\"\nTERRAFORM_BACKEND\nTF_BACKEND\nTFENV\n\"\n\nbackend_scopes=\"\ndev\ndevelopment\n\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDownloads backend.tf contents from a GitLab CI/CD environment variable to be able to quickly iterate plans locally\n\nFor companies that don't store their backend.tf in Git but instead materialize it inside the CI/CD system from an environment variable\n\nIf variable is not specified defaults to searching for these variables in this order: $(tr '\\n' ' ' <<< \"$backend_variables\")\nIf environment scope is not specified defaults to searching for it in this order: $(tr '\\n' ' ' <<< \"$backend_scopes\")\n\nRequires GitLab CLI to be installed and authenticated to list and fetch the backend environment variable, and should be executed from within the current terraform project folder so it defaults to the right GitLab project repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<variable> <environment_or_scope>]\"\n\nhelp_usage \"$@\"\n\nmax_args 2 \"$@\"\n\nbackend_tf_file=\"$PWD/backend.tf\"\n\nvariable=\"${1:-}\"\nscope=\"${2:-}\"\n\ntimestamp \"Getting GitLab CI/CD variables\"\nvariables=\"$(glab variable list 2>&1)\"\n\nif [ -z \"$variable\" ]; then\n    for possible_variable in $backend_variables; do\n        if grep -q \"^${possible_variable}[[:space:]]\" <<< \"$variables\"; then\n            variable=\"$possible_variable\"\n            break\n        fi\n    done\nfi\nif [ -z \"$variable\" ]; then\n    die \"Failed to find variable with terraform backend in GitLab CI/CD, please specify variable manually\"\nfi\n\nif [ -z \"$scope\" ]; then\n    for possible_scope in $backend_scopes; do\n        if grep -q \"${variable}[[:space:]].*[[:space:]]${possible_scope}$\" <<< \"$variables\"; then\n            scope=\"$possible_scope\"\n            break\n        fi\n    done\nfi\nif [ -z \"$scope\" ]; then\n    die \"GitLab variable '$variable' does not exist in any of the following scopes: $(tr '\\n' ' ' <<< \"$backend_scopes\"). Please specify the scope manually as an argument\"\nfi\n\n# false positive\n# shellcheck disable=SC2016\ntimestamp \"Getting backend config from GitLab variable '$variable' ${scope+scope '$scope'}\"\nbackend_config=\"$(glab variable get \"$variable\" ${scope+-s \"$scope\"} )\"\n\ntimestamp \"Variable contents:\"\necho >&2\necho \"$backend_config\" >&2\necho >&2\nif ! grep -q backend <<< \"$backend_config\"; then\n    die \"ERROR: Backend not found in variable contents\"\nfi\nread -r -p \"Are you sure you want to overwrite '$backend_tf_file' with this content? (y/N) \" answer\ncheck_yes \"$answer\"\necho >&2\ntimestamp \"Writing $backend_tf_file\"\necho \"$backend_config\" > \"$backend_tf_file\"\n"
  },
  {
    "path": "terraform/terraform_import.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all given resource references in ./*.tf code not in Terraform state and imports them assuming the same resource name as Terraform ID in the code\n\nWill do nothing if the resource_type you specify doesn't match anything in the local code eg. 'github_repo' won't match, it must be the terraform type 'github_repository'\n\nThis is a general case importer that will only cover basic use cases such as GitHub repos where the names usually match the terraform IDs\n(except for things like '.github' repo which is not a valid terraform identifier. Those must still be imported manually)\n\nIf \\$TERRAFORM_PLAN is set to any value, gets the resources from the Terraform Plan rather than ./*.tf\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<resource_type> [<dir>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nresource_type=\"$1\"\ndir=\"${2:-.}\"\n\ncd \"$dir\"\n\ntimestamp \"getting terraform state\"\nterraform_state_list=\"$(terraform state list)\"\necho >&2\n\nif [ -n \"${TERRAFORM_PLAN:-}\" ]; then\n    timestamp \"getting terraform plan\"\n    plan=\"$(terraform plan -no-color)\"\n    echo >&2\n\n    timestamp \"getting '$resource_type' from terraform plan output\"\n    grep -E \"^[[:space:]]*resource[[:space:]]+\\\"$resource_type\\\"\" <<< \"$plan\"\nelse\n    timestamp \"getting '$resource_type' from $PWD/*.tf code\"\n    grep -E \"^[[:space:]]*resource[[:space:]]+\\\"$resource_type\\\"\" ./*.tf\nfi |\nawk '{gsub(\"\\\"\", \"\", $3); print $3}' |\nwhile read -r resource; do\n    echo >&2\n    if grep -q \"$resource_type\\\\.$resource$\" <<< \"$terraform_state_list\"; then\n        echo \"'$resource_type.$resource' already in terraform state, skipping...\" >&2\n        continue\n    fi\n    cmd=(terraform import \"$resource_type.$resource\" \"$resource\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_iam_groups.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-24 15:11:14 +0100 (Mon, 24 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_iam_group additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\n#group_arn_mapping=\"$(aws iam list-groups | jq -r '.Groups[] | [.GroupName, .Arn] | @tsv' | column -t)\"\n\nterraform plan -no-color |\nsed -n '/# aws_iam_group\\..* will be created/,/}/ p' |\nawk '/# aws_iam_group/ {print $2};\n     /name/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n2 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r group name; do\n    [ -n \"$name\" ] || continue\n    timestamp \"Importing group: $name\"\n    #arn=\"$(awk \"/^${name}[[:space:]]/{print \\$2}\" <<< \"$group_arn_mapping\")\"\n    #if is_blank \"$arn\"; then\n    #    die \"Failed to determine group ARN\"\n    #fi\n    # shellcheck disable=SC2178\n    cmd=(terraform import \"$group\" \"$name\")\n    # shellcheck disable=SC2128\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_iam_policies.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-24 15:11:14 +0100 (Mon, 24 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_iam_policy additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\npolicy_arn_mapping=\"$(aws iam list-policies | jq -r '.Policies[] | [.PolicyName, .Arn] | @tsv' | column -t)\"\n\nterraform plan -no-color |\nsed -n '/# aws_iam_policy\\..* will be created/,/}/ p' |\nawk '/# aws_iam_policy/ {print $2};\n     /name/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n2 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r policy name; do\n    [ -n \"$name\" ] || continue\n    timestamp \"Importing policy: $name\"\n    arn=\"$(awk \"/^${name}[[:space:]]/{print \\$2}\" <<< \"$policy_arn_mapping\")\"\n    if is_blank \"$arn\"; then\n        die \"Failed to determine policy ARN\"\n    fi\n    cmd=(terraform import \"$policy\" \"$arn\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_iam_users.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-10-24 15:11:14 +0100 (Mon, 24 Oct 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_iam_user additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\nterraform plan -no-color |\nsed -n '/# aws_iam_user\\..* will be created/,/}/ p' |\n#sed -n '/tags = {/,/}/d; p' |\nawk '/# aws_iam_user/ {print $2};\n     /name/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n2 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r user name; do\n    [ -n \"$name\" ] || continue\n    timestamp \"Importing user: $name\"\n    # shellcheck disable=SC2178\n    cmd=(terraform import \"$user\" \"$name\")\n    # shellcheck disable=SC2128\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_sso_account_assignments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-20 16:24:51 +0100 (Tue, 20 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_ssoadmin_account_assignment additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\n#TMP_PLAN=\"$(mktemp)\"\n#TMP_PLAN_JSON=\"$(mktemp)\"\n\n#timestamp \"Getting Terraform Plan\"\n#terraform plan -out \"$TMP_PLAN\"\n#echo >&2\n#\n#timestamp \"Parsing Terraform Plan\"\n## have to parse references, and then double parse to find the values of the object references :-/\n#terraform show -json \"$TMP_PLAN\" |\n#jq -Mr > \"$TMP_PLAN_JSON\"\n\n#jq -Mr <\"$TMP_PLAN_JSON\" '\n#    .configuration.root_module.resources[] |\n#    select(.type == \"aws_ssoadmin_account_assignment\") |\n#    [.address, ] |\n#    @tsv' |\n\nterraform plan -no-color |\n#grep -FA8 '+ resource \"aws_ssoadmin_account_assignment\" ' |\nsed -n '/# aws_ssoadmin_account_assignment\\..* will be created/,/}/ p' |\nawk '/# aws_ssoadmin_account_assignment/ {print $2};\n     /instance_arn|permission_set_arn|principal_id|principal_type|target_id/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n6 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r name instance_arn permission_set_arn principal_id principal_type target_id; do\n    [ -n \"$target_id\" ] || continue\n    timestamp \"Importing $name\"\n    cmd=(terraform import \"$name\" \"$principal_id,$principal_type,$target_id,AWS_ACCOUNT,$permission_set_arn,$instance_arn\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_sso_managed_policy_attachments.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-20 16:24:51 +0100 (Tue, 20 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_ssoadmin_account_assignment additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\n# don't do this because would have to parse references, and then double parse to find the values of the object references :-/\n#terraform show -json plan.zip\n\nterraform plan -no-color |\nsed -n '/# aws_ssoadmin_managed_policy_attachment\\..* will be created/,/}/ p' |\nawk '/# aws_ssoadmin_managed_policy_attachment/ {print $2};\n     /instance_arn|managed_policy_arn|permission_set_arn/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n4 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r name instance_arn managed_policy_arn permission_set_arn; do\n    [ -n \"$permission_set_arn\" ] || continue\n    timestamp \"Importing $name\"\n    cmd=(terraform import \"$name\" \"$managed_policy_arn,$permission_set_arn,$instance_arn\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_sso_permission_set_inline_policies.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-20 16:24:51 +0100 (Tue, 20 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan for aws_ssoadmin_permission_set_inline_policy additions and imports each one into Terraform state\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\n# would have to parse references, and then double parse to find the values of the object references :-/\n#terraform show -json \"$TMP_PLAN\" |\n\nterraform plan -no-color |\nsed -n '/# aws_ssoadmin_permission_set_inline_policy\\..* will be created/,/permission_set_arn/ p' |\nawk '/# aws_ssoadmin_permission_set_inline_policy/ {print $2};\n     /instance_arn|permission_set_arn/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n3 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r name instance_arn permission_set_arn; do\n    [ -n \"$permission_set_arn\" ] || continue\n    timestamp \"Importing $name\"\n    cmd=(terraform import \"$name\" \"$permission_set_arn,$instance_arn\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_aws_sso_permission_sets.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-09-12 23:58:12 +0100 (Mon, 12 Sep 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all AWS SSO permission set references in ./*.tf code not in Terraform state and imports them\n\nDetermines the permission set ARNs via AWS CLI and then runs the terraform import command\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\nCaveat: takes the first AWS SSO instance arn found unless \\$AWS_SSO_INSTANCE_ARN is explicitly specified in the environment\n\n\nRequires Terraform and AWS CLIv2 to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\n# terraform plan -out plan.zip\n#terraform show -json plan.zip | jq -r '.configuration.root_module.resources[] | select(.type == \"aws_ssoadmin_permission_set\") | [.address, .expressions.name.constant_value] | @tsv'\n\n#timestamp \"getting terraform state\"\n#terraform_state_list=\"$(terraform state list)\"\n#echo >&2\n#\n#timestamp \"determining AWS SSO instance arn\"\n## XXX: assumes only 1 instance\ninstance_arn=\"${AWS_SSO_INSTANCE_ARN:-$(aws sso-admin list-instances | jq -r '.Instances[0].InstanceArn')}\"\necho >&2\n\ntimestamp \"getting AWS permission set list\"\npermset_arns=\"$(aws sso-admin list-permission-sets --instance-arn \"$instance_arn\" | jq -r '.PermissionSets[]')\"\necho >&2\n\n# XXX: for Bash 3 portability on Macs, not using associative arrays which aren't available, so building a string map as a workaround\npermset_map=\"\"\n\n# XXX: single pass to generate map is better than iterating all permsets for each permset ie. O(n) + O(p) vs O(n * p)\ntimestamp \"Mapping permset ARNs\"\necho >&2\nfor permset_arn in $permset_arns; do\n    timestamp \"Mapping permset '$permset_arn'\"\n    name=\"$(\n        aws sso-admin describe-permission-set --instance-arn \"$instance_arn\" --permission-set-arn \"$permset_arn\" |\n        jq -r '.PermissionSet.Name'\n    )\"\n    if [ -z \"$name\" ]; then\n        die \"Failed to resolve name for permset arn '$permset_arn'\"\n    fi\n    permset_map+=\"\n$permset_arn $name\n\"\n    timestamp \"Mapped permset '$name'\"\ndone\necho >&2\n\n#timestamp \"getting permission sets from $PWD/*.tf code\"\n#grep -E '^[[:space:]]*resource[[:space:]]+\"aws_ssoadmin_permission_set\"' -- *.tf |\n#awk '{gsub(\"\\\"\", \"\", $3); print $3}' |\n#while read -r permset; do\n#    echo >&2\n#    if grep -q \"aws_ssoadmin_permission_set\\\\.$permset$\" <<< \"$terraform_state_list\"; then\n#        timestamp \"permission set '$permset' already in terraform state, skipping...\"\n#        continue\n#    fi\n#    timestamp \"Permission set '$permset' needs importing\"\n#    timestamp \"Determining permission set name for '$permset'\"\n#    set +o pipefail\n#    name=\"$(\n#        grep -Eh -A 10 '^[[:space:]]*resource[[:space:]]+\"aws_ssoadmin_permission_set\"[[:space:]]+\"'\"$permset\"'\"' -- *.tf |\n#        awk '/^[[:space:]]+name[[:space:]]+=/ {print $3; exit}' |\n#        sed 's/\"//g'\n#    )\"\n#    set -o pipefail\n#    if [ -z \"$name\" ]; then\n#        die \"Failed to determine name for permission set '$permset'\"\n#    fi\n#    timestamp \"Determined permission set name for '$permset' to be '$name'\"\n#    timestamp \"Determining ARN for permission set '$name'\"\n#    # XXX: string map workaround for Bash 3\n#    permset_arn=\"$(awk \"/[[:space:]]${name//\\//\\\\/}$/\"' {print $1}' <<< \"$permset_map\")\"\n#    if [ -z \"$permset_arn\" ]; then\n#        die \"Failed to determine permission set arn for '$name'\"\n#    fi\n#    timestamp \"Importing permission set '$permset'\"\n#    cmd=\"terraform import \\\"aws_ssoadmin_permission_set.$permset\\\" \\\"$permset_arn,$instance_arn\\\"\"\n#    echo \"$cmd\"\n#    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n#        eval \"$cmd\"\n#    fi\n#    echo >&2\n#done\n\nterraform plan -no-color |\nsed -n '/# aws_ssoadmin_permission_set\\..* will be created/,/name/ p' |\nawk '/# aws_ssoadmin_permission_set/ {print $2};\n     /instance_arn|name/ {print $4}' |\nsed 's/^\"//; s/\"$//' |\nxargs -n3 echo |\nsed 's/\\[/[\"/; s/\\]/\"]/' |\nwhile read -r id instance_arn name; do\n    [ -n \"$name\" ] || continue\n    timestamp \"Determining permset arn from name '$name'\"\n    timestamp \"Determining ARN for permission set '$name'\"\n    # XXX: string map workaround for Bash 3\n    permset_arn=\"$(awk \"/[[:space:]]${name//\\//\\\\/}$/\"' {print $1}' <<< \"$permset_map\")\"\n    if [ -z \"$permset_arn\" ]; then\n        die \"Failed to resolve permset arn for '$name'\"\n    fi\n    timestamp \"Importing $name\"\n    cmd=(terraform import \"$id\" \"$permset_arn,$instance_arn\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\n    echo >&2\ndone\n"
  },
  {
    "path": "terraform/terraform_import_foreach.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all given for_each generated resource references in Terraform plan output not in Terraform state and imports them\n\nWill do nothing if the resource_type you specify doesn't match anything in the local code eg. 'github_repo' won't match, it must be the terraform type 'github_repository'\n\nThis is a general case importer that will only cover basic use cases such as GitHub repos where the names usually match the terraform IDs\n(except for things like '.github' repo which is not a valid terraform identifier. Those must still be imported manually)\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<resource_type> [<dir>]\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\nresource_type=\"$1\"\ndir=\"${2:-.}\"\n\ncd \"$dir\"\n\ntimestamp \"getting terraform plan\"\nplan=\"$(terraform plan -no-color)\"\necho >&2\n\ntimestamp \"getting '$resource_type' from terraform plan output\"\ngrep -E \"^[[:space:]]*# $resource_type\\\\..+\\\\[\\\"[^\\\"]+\\\"\\\\] will be created\" <<< \"$plan\" |\nawk '{print $2}' |\nwhile read -r resource_path; do\n    echo >&2\n    # <resource_type>.resource2[resource1] - resource 1 is usually the differentiator, eg. github repo, whereas resource2 is usually what is applied to each one, such as the same branch\n    resource1=\"${resource_path##*[\\\"}\"\n    resource1=\"${resource1%%\\\"]*}\"\n    resource2=\"${resource_path%%[*}\"\n    resource2=\"${resource2##*.}\"\n    cmd=(terraform import \"$resource_path\" \"$resource1:$resource2\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\ndone\n"
  },
  {
    "path": "terraform/terraform_import_github_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all github_repository references in ./*.tf code not in Terraform state and imports them\n\nRequires the github_repository identifiers in *.tf code to match the GitHub repo name, which does not work with repos which have dots in them eg. '.github'. Those rare exceptions must be imported manually.\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform to be installed and configured\n\n\nSee Also:\n\n    github_repos_not_in_terraform.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\ndir=\"${1:-.}\"\n\ncd \"$dir\"\n\ntimestamp \"getting terraform state\"\nterraform_state_list=\"$(terraform state list)\"\necho >&2\n\ntimestamp \"getting github repos from $PWD/*.tf code\"\ngrep -E '^[[:space:]]*resource[[:space:]]+\"github_repository\"' ./*.tf |\nawk '{gsub(\"\\\"\", \"\", $3); print $3}' |\nwhile read -r repo; do\n    echo >&2\n    if grep -q \"github_repository\\\\.$repo$\" <<< \"$terraform_state_list\"; then\n        echo \"repository '$repo' already in terraform state, skipping...\" >&2\n        continue\n    fi\n    cmd=(terraform import \"github_repository.$repo\" \"$repo\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\ndone\n"
  },
  {
    "path": "terraform/terraform_import_github_team.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTerraform imports a given GitHub team into a given Terraform resource, by first querying the GitHub API for the team ID needed to import into Terraform\n\nThe GitHub Organization must be specified as the second arg or found in \\$GITHUB_ORGANIZATION environment variable\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\nRequires Terraform and GitHub CLI to be installed and configured\n\n\nSee Also:\n\n    github_teams_not_in_terraform.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<terraform_state_resource> <github_team> [<github_organization>]\"\n\nhelp_usage \"$@\"\n\nmin_args 2 \"$@\"\nmax_args 3 \"$@\"\n\nresource=\"$1\"\nteam=\"$2\"\norg=\"${3:-${GITHUB_ORGANIZATION:-}}\"\n\nif is_blank \"$org\"; then\n    usage \"Organization not specified\"\nfi\n\ntimestamp \"querying GitHub team '$team' for ID\"\nid=\"$(gh api \"/orgs/$org/teams/$team\" | jq -r 'select(.id) | .id' || :)\"\nif [ -z \"$id\" ]; then\n    die \"ERROR: team '$team' not found in GitHub API\"\nfi\ncmd=(terraform import \"'$resource'\" \"$id\")\ntimestamp \"${cmd[*]}\"\nif [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n    \"${cmd[@]}\"\nfi\n"
  },
  {
    "path": "terraform/terraform_import_github_team_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nParses Terraform Plan output, find all github_team_repository references that would be added, queries the GitHub API to determine if they exist, and if so imports them\n\nTerraform Plan is used because the team_id is also needed for import but is often a terraform reference which only resolves at runtime\n\n\nThe GitHub Organization must be specified as the first arg or found in \\$GITHUB_ORGANIZATION environment variable\n\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\nRequires Terraform and GitHub CLI to be installed and configured. Tested on Terraform 1.1.6\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> [<dir>]\"\n\nhelp_usage \"$@\"\n\norg=\"${1:-${GITHUB_ORGANIZATION:-}}\"\ndir=\"${2:-.}\"\n\nif is_blank \"$org\"; then\n    usage \"Organization not specified\"\nfi\n\ncd \"$dir\"\n\ntimestamp \"getting GitHub org id for org '$org'\"\norg_id=\"$(gh api \"/orgs/$org\" | jq -r .id)\"\ntimestamp \"org id determined to be '$org_id'\"\necho >&2\n\ntimestamp \"getting terraform plan\"\nterraform_plan=\"$(terraform plan -no-color)\"\necho >&2\n\ntimestamp \"parsing github_team_repository references\"\ngrep -E -e '^[[:space:]]+\\+[[:space:]]+resource[[:space:]]+\"github_team_repository\"' \\\n        -e '^[[:space:]]+\\+[[:space:]]+repository[[:space:]]+=' \\\n        -e '^[[:space:]]+\\+[[:space:]]+team_id[[:space:]]+=' \\\n        <<< \"$terraform_plan\" |\nsed 's/.*=//; s/.*resource \"github_team_repository\"//' |\nawk '{gsub(\"\\\"\", \"\", $1); print $1}' |\n# collapses every 3 lines together\nxargs -d '\\n' -n 3 echo |\nwhile read -r team_repo repo team_id; do\n    echo >&2\n    timestamp \"querying repo '$repo'\"\n    if ! gh api \"/repos/$org/$repo\" | jq -e '.id' >/dev/null; then\n        warn \"repo '$repo' not found in GitHub API, skipping...\"\n        continue\n    fi\n    timestamp \"querying team id '$team_id'\"\n    if ! gh api \"/organizations/$org_id/team/$team_id\" | jq -e '.id' >/dev/null; then\n        warn \"team id '$team_id' not found in GitHub API, skipping...\"\n        continue\n    fi\n    cmd=(terraform import \"github_team_repository.$team_repo\" \"$team_id:$repo\")\n    timestamp \"${cmd[*]}\"\n    if [ -z \"${TERRAFORM_PRINT_ONLY:-}\" ]; then\n        \"${cmd[@]}\"\n    fi\ndone\n"
  },
  {
    "path": "terraform/terraform_import_github_teams.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-25 18:14:24 +0000 (Fri, 25 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds all github_team references in ./*.tf code and then queries the GitHub API for their IDs and if they exist imports them to Terraform state\n\nThe GitHub Organization must be specified as the first arg or found in \\$GITHUB_ORGANIZATION environment variable\n\nRequires the github_repository identifiers in *.tf code to match the GitHub team slug in the GitHub API\n\nIf \\$TERRAFORM_PLAN is set to any value, gets the resources from the Terraform Plan rather than ./*.tf\nIf \\$TERRAFORM_PRINT_ONLY is set to any value, prints the commands to stdout to collect so you can check, collect into a text file or pipe to a shell or further manipulate, ignore errors etc.\n\n\nRequires Terraform and GitHub CLI to be installed and configured\n\n\nSee Also:\n\n    github_teams_not_in_terraform.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<organization> [<dir>]\"\n\nhelp_usage \"$@\"\n\norg=\"${1:-${GITHUB_ORGANIZATION:-}}\"\ndir=\"${2:-.}\"\n\nif is_blank \"$org\"; then\n    usage \"Organization not specified\"\nfi\n\ncd \"$dir\"\n\ntimestamp \"getting terraform state\"\nterraform_state_list=\"$(terraform state list)\"\necho >&2\n\nif [ -n \"${TERRAFORM_PLAN:-}\" ]; then\n    timestamp \"getting terraform plan\"\n    plan=\"$(terraform plan -no-color)\"\n    echo >&2\n\n    timestamp \"getting github teams from terraform plan output\"\n    grep -E '^[[:space:]]*resource[[:space:]]+\"github_team\"' <<< \"$plan\"\nelse\n    timestamp \"getting github teams from $PWD/*.tf code\"\n    grep -E '^[[:space:]]*resource[[:space:]]+\"github_team\"' ./*.tf\nfi |\nawk '{gsub(\"\\\"\", \"\", $3); print $3}' |\nwhile read -r team; do\n    echo >&2\n    if grep -q \"github_team\\\\.$team$\" <<< \"$terraform_state_list\"; then\n        echo \"team '$team' already in terraform state, skipping...\" >&2\n        continue\n    fi\n    \"$srcdir/terraform_import_github_team.sh\" \"github_team.$team\" \"$team\" \"$org\"\ndone\n"
  },
  {
    "path": "terraform/terraform_managed_resource_types.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2023-05-06 00:18:38 +0100 (Sat, 06 May 2023)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuick Terraform code parser of the given or current directory tree to list all the resources types found in Terraform *.tf code files\n\nUseful to give you a quick glance of what services you are managing. Usually you're want to run this at the top of your Terraform repo\n\nCaveat: won't return anything from modules outside your current or given directory tree, or any resources created by external referenced modules\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\nfind \"$dir\" -type f -iname '*.tf' -print0 |\nxargs -0 grep -hR '^[[:space:]]*resource' |\nawk '/^[[:space:]]*resource[[:space:]]/{print $2}' |\nsed 's/^\"//; s/\"$//' |\nsort -u\n"
  },
  {
    "path": "terraform/terraform_provider_count_sizes.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2025-03-11 03:28:53 +0800 (Tue, 11 Mar 2025)\n#\n#  https///github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nFinds duplicate Terraform providers and their sizes\n\nUseful to find space wastage caused by using Terragrunt without configuring a unified Terraform Plugin Cache:\n\n    https://developer.hashicorp.com/terraform/cli/config/config-file#provider-plugin-cache\n\n    https://terragrunt.gruntwork.io/docs/features/provider-cache-server/\n\n    https://github.com/gruntwork-io/terragrunt/issues/561\n\n    https://github.com/gruntwork-io/terragrunt/issues/2920\n\nFor example, in a repo checkout for a single project, I had 30 x 600MB AWS provider\n\n    30  597M  hashicorp/aws/5.80.0/darwin_arm64/terraform-provider-aws_v5.80.0_x5\n    7   637M  hashicorp/aws/5.90.1/darwin_arm64/terraform-provider-aws_v5.90.1_x5\n    4   637M  hashicorp/aws/5.90.0/darwin_arm64/terraform-provider-aws_v5.90.0_x5\n    3   599M  hashicorp/aws/5.81.0/darwin_arm64/terraform-provider-aws_v5.81.0_x5\n    2   593M  hashicorp/aws/5.79.0/darwin_arm64/terraform-provider-aws_v5.79.0_x5\n\nOutput format:\n\n    <count>    <provider_size>    <provider>\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<dir>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\ndir=\"${1:-.}\"\n\n#timestamp \"Finding Terraform providers\"\n# slow way of finding duplicates\n#providers=\"$(find \"$dir\" -type f -name 'terraform-provider-*' -exec md5sum {} \\;)\"\nproviders=\"$(find \"$dir\" -type f -name 'terraform-provider-*')\"\n#echo\n\nif [ -z \"$providers\" ]; then\n    die \"ERROR: no Terraform providers found. Did you run this from a Terraform / Terragrunt working directory that has been used?\"\nfi\n\n#timestamp \"Ranking providers by duplication level\"\n#echo\n\nstrip_prefix(){\n    sed '\n        s|.*\\.terraform/providers/||;\n        s|registry.terraform.io/||;\n    '\n}\n\nstrip_prefix <<< \"$providers\" |\nsort |\nuniq -c |\nsort -k1nr |\nwhile read -r count filepath; do\n    echo -n \"$count \"\n    # head -n 1 is more reliable than grep -m 1 on some platforms (macOS BSD)\n    filename=\"$(grep \"$filepath\" <<< \"$providers\" | head -n 1)\"\n    du -h \"$filename\" |\n    awk '{printf $1\" \"}'\n    strip_prefix <<< \"$filename\"\ndone |\ncolumn -t\n"
  },
  {
    "path": "terraform/terraform_registry_url_extract.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2024-12-05 00:26:53 +0700 (Thu, 05 Dec 2024)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nExtracts the Terraform Registry URLs in either tfr:// or https://registry.terraform.io/ format\nfrom a given string, file or standard input\n\nUseful to fast load Terraform Module documentation via editor/IDE hotkeys\n\nSee advanced .vimc in this repo\n\nBased on ../bin/urlextract.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<string_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\narg=\"${1:-}\"\n\nif [ $# -eq 0 ]; then\n    cat\nelif [ -f \"$arg\" ]; then\n    cat \"$arg\"\nelse\n    echo \"$arg\"\nfi |\n# [] break the regex match, even when escaped \\[\\]\ngrep -Eom 1 \\\n     -e 'tfr://[[:alnum:]./?&!$#%@*;:+~_=-]+' \\\n     -e 'https://registry.terraform.io/[[:alnum:]./?&!$#%@*;:+~_=-]*' ||\ndie \"No Terraform Registry URLs found\"\n"
  },
  {
    "path": "terraform/terraform_registry_url_open.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Thu Dec 5 00:39:05 2024 +0700\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nOpens the Terraform Registry URL in either tfr:// or https://registry.terraform.io/ format\n\nURL can be given as a string arg, file or standard input\n\nIf the given arg is a file, then opens the first Terraform URL found in the file\n\nUsed by .vimrc to instantly open a URL on the given line in the editor\n\nVery useful for quickly referencing Terraform documentation for modules defined in Terraform code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<url_or_file_with_url>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\n#if \"$srcdir/terraform_registry_url_extract.sh\" \"$@\" |\n#    sed 's|tfr://registry.terraform.io/|https://registry.terraform.io/modules/|; s|$|/latest|g'\n#    then\n#    timestamp \"Found Terraform Registry URL(s)\"\n#elif \"$srcdir/../bin/urlextract.sh\" \"$@\"; then\n#    timestamp \"Found URL(s)\"\n#fi |\n\n\"$srcdir/terraform_registry_url_extract.sh\" \"$@\" |\n\"$srcdir/terraform_registry_url_to_https.sh\" |\n# head -n1 because grep -m 1 can't be trusted and sometimes outputs more matches on subsequent lines\nhead -n1 |\n\"$srcdir/../bin/urlopen.sh\"\n"
  },
  {
    "path": "terraform/terraform_registry_url_to_https.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: Thu Dec 5 00:39:05 2024 +0700\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nConverts one or more Terraform Registry URLs from tfr:// to https://registry.terraform.io/ format\n\nURLs can be given as a string argument, file or standard input containing URLs\n\nUsed by .vimrc to convert tfr:// URLs found in a file to HTTPS format to present in a menu\n\nVery useful for quickly referencing Terraform documentation for modules defined in Terraform code\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<tfr_url_or_file_with_tfr_urls>]\"\n\nhelp_usage \"$@\"\n\nmax_args 1 \"$@\"\n\narg=\"${1:-}\"\n\nif [ $# -eq 0 ]; then\n    cat\nelif [ -f \"$arg\" ]; then\n    cat \"$arg\"\nelse\n    echo \"$arg\"\nfi |\nsed '\n    /tfr/{\n        s|tfr://registry.terraform.io/|https://registry.terraform.io/modules/|;\n        s|$|/latest|g;\n    }\n'\n"
  },
  {
    "path": "terraform/terraform_resources.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2022-02-28 19:18:26 +0000 (Mon, 28 Feb 2022)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\n\n# sourcing lib.sh results in Terraform errors 'is_verbose: command not found'\n\nusage=\"\nTerraform external program that returns a list of resource ids and attribute for a given resource_type\n\nWorkaround for Terraform Splat expressions not supporting top level resources\n\n    https://github.com/hashicorp/terraform/issues/19931\n\nReturns a JSON output in format 'map[string]=string' where the key is set to the id and the value is set to the name or selected attribute value of the resource\n\nReturns a non-zero error code if the resource_type is not found which will be picked up by Terraform to error out, but a missing attribute will get a null value\n\n\nExample:\n\n    ${0##*/} github_repository\n\nTerraform:\n\n    data \\\"external\\\" \\\"github_repos\\\" {\n        program = [\\\"/path/to/${0##*/}\\\", \\\"github_repository\\\"]\n    }\n\n    resource \\\"github_team_repository\\\" \\\"devops\\\" {\n      permission = \\\"admin\\\"\n      for_each   = data.external.github_repos.result\n      repository = each.key\n      team_id    = github_team.devops.id\n    }\n\n\nRequires Terraform and jq to be installed and configured\n\n\nusage: ${0##*/} <resource_type> [<attribute>]\n\"\n\nif [ $# -lt 1 ] ||\n   [ $# -gt 2 ] ||\n   [[ \"$1\" =~ ^- ]] ||\n   [[ \"${2:-}\" =~ ^- ]]; then\n    echo \"$usage\"\n    exit 3\nfi\n\nresource_type=\"$1\"\nattribute=\"${2:-name}\"\n\n#terraform state list  |\n#grep \"^$resource_type\\\\.\" |\n#awk -F. '{print $2}' |\n#while read -r resource; do\n#    # Terraform state outputs control chars, remove them so grep will work - hard to remove all escape sequences and slow\n#    # we need literal escapes here\n#    # shellcheck disable=SC1117\n#    terraform state show \"$resource_type.$resource\" |\n#    sed \"s,\\x1B\\[[0-9;]*[a-zA-Z],,g\" |\n#    # nested attributes, eg. branches have greater depth - this code is brittle but Terraform doesn't support -json for terraform state show unfortunately\n#    grep -E \"^    ${attribute}[[:space:]]+= \" |\n#    awk -F= '{print $2}' |\n#    sed 's/^\"//;s/\"$//'\n#done |\n\nterraform show -json -no-color |\njq -er \"\n    .values.root_module |\n      [\n        .resources[],\n        .child_modules[].resources[]\n      ] |\n    flatten[] |\n    select(.type == \\\"$resource_type\\\") |\n    select(.values.id) |\n    { (.values.id) : .values.$attribute }\" |\njq -en 'reduce inputs as $in (null; . + $in)'\n"
  },
  {
    "path": "tests/README.md",
    "content": "Tests\n=====\n\nUnlike the `tests/` directory of my other repos, most of the scripts in this repo are actually really used and tested via the CI of most of my other repos that use it as a submodule, while some other scripts aren't easily testable as their require non-trivial infrastructure (eg. Cloudera) or local access keys.\n\n`spotify_uri_to_name.sh` has its alternate modes tested here only because adjacent scripts and repos only use its original design for track conversions.\n"
  },
  {
    "path": "tests/azure_devops_url_conversion.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-12-04 23:05:31 +0000 (Fri, 04 Dec 2020)\n#\n#  https://devops.azure.com/HariSekhon/bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090\n. \"$srcdir/../lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTests the git_to_azure_url function from lib/git.sh\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nsrc[0]='git@devops.azure.com:/HariSekhon/pylib'\ndest[0]='git@devops.azure.com:v3/harisekhon/GitHub/pylib'\n\nsrc[1]='git@devops.azure.com:/harisekhon/nagios-plugin-kafka'\ndest[1]='git@devops.azure.com:v3/harisekhon/GitHub/nagios-plugin-kafka'\n\nsrc[2]='git@devops.azure.com:HariSekhon/DevOps-Bash-tools'\ndest[2]='git@devops.azure.com:v3/harisekhon/GitHub/DevOps-Bash-tools'\n\nsrc[3]='git@devops.azure.com:HariSekhon/DevOps-Golang-tools'\ndest[3]='git@devops.azure.com:v3/harisekhon/GitHub/DevOps-Golang-tools'\n\nsrc[4]='git@devops.azure.com:HariSekhon/Dockerfiles.git'\ndest[4]='git@devops.azure.com:v3/harisekhon/GitHub/Dockerfiles'\n\nsrc[5]='git@devops.azure.com:HariSekhon/HAProxy-configs'\ndest[5]='git@devops.azure.com:v3/harisekhon/GitHub/HAProxy-configs'\n\nsrc[6]='git@devops.azure.com:HariSekhon/Nagios-Plugins'\ndest[6]='git@devops.azure.com:v3/harisekhon/GitHub/Nagios-Plugins'\n\nsrc[7]='git@devops.azure.com:HariSekhon/SQL-scripts'\ndest[7]='git@devops.azure.com:v3/harisekhon/GitHub/SQL-scripts'\n\nsrc[8]='git@devops.azure.com:HariSekhon/Spotify-Playlists.git'\ndest[8]='git@devops.azure.com:v3/harisekhon/GitHub/Spotify-Playlists'\n\nsrc[9]='git@devops.azure.com:HariSekhon/Templates'\ndest[9]='git@devops.azure.com:v3/harisekhon/GitHub/Templates'\n\nsrc[10]='git@devops.azure.com:harisekhon/kubernetes-templates'\ndest[10]='git@devops.azure.com:v3/harisekhon/GitHub/kubernetes-templates'\n\nsrc[11]='git@devops.azure.com:harisekhon/spotify-tools.git'\ndest[11]='git@devops.azure.com:v3/harisekhon/GitHub/spotify-tools'\n\nsrc[12]='git@devops.azure.com:harisekhon/sql-keywords'\ndest[12]='git@devops.azure.com:v3/harisekhon/GitHub/sql-keywords'\n\nsrc[13]='git@devops.azure.com:/HariSekhon/Nagios-Plugin-Kafka'\ndest[13]='git@devops.azure.com:v3/harisekhon/GitHub/Nagios-Plugin-Kafka'\n\nsrc[14]='git@devops.azure.com:/HariSekhon/pylib'\ndest[14]='git@devops.azure.com:v3/harisekhon/GitHub/pylib'\n\nsrc[15]='git@devops.azure.com:HariSekhon/DevOps-Bash-tools'\ndest[15]='git@devops.azure.com:v3/harisekhon/GitHub/DevOps-Bash-tools'\n\nsrc[16]='git@devops.azure.com:HariSekhon/DevOps-Golang-tools'\ndest[16]='git@devops.azure.com:v3/harisekhon/GitHub/DevOps-Golang-tools'\n\nsrc[17]='git@devops.azure.com:HariSekhon/Dockerfiles.git'\ndest[17]='git@devops.azure.com:v3/harisekhon/GitHub/Dockerfiles'\n\nsrc[18]='git@devops.azure.com:HariSekhon/HAProxy-configs'\ndest[18]='git@devops.azure.com:v3/harisekhon/GitHub/HAProxy-configs'\n\nsrc[19]='git@devops.azure.com:HariSekhon/Nagios-Plugins'\ndest[19]='git@devops.azure.com:v3/harisekhon/GitHub/Nagios-Plugins'\n\nsrc[20]='git@devops.azure.com:HariSekhon/SQL-scripts'\ndest[20]='git@devops.azure.com:v3/harisekhon/GitHub/SQL-scripts'\n\nsrc[21]='git@devops.azure.com:HariSekhon/Spotify-Playlists.git'\ndest[21]='git@devops.azure.com:v3/harisekhon/GitHub/Spotify-Playlists'\n\nsrc[22]='git@devops.azure.com:HariSekhon/Spotify-tools.git'\ndest[22]='git@devops.azure.com:v3/harisekhon/GitHub/Spotify-tools'\n\nsrc[23]='git@devops.azure.com:HariSekhon/Templates'\ndest[23]='git@devops.azure.com:v3/harisekhon/GitHub/Templates'\n\nsrc[24]='git@devops.azure.com:harisekhon/kubernetes-templates'\ndest[24]='git@devops.azure.com:v3/harisekhon/GitHub/kubernetes-templates'\n\nsrc[25]='git@devops.azure.com:harisekhon/sql-keywords'\ndest[25]='git@devops.azure.com:v3/harisekhon/GitHub/sql-keywords'\n\nsrc[26]='git@devops.azure.com:HariSekhon/Spotify-Playlists.git'\ndest[26]='git@devops.azure.com:v3/harisekhon/GitHub/Spotify-Playlists'\n\nsrc[27]='ssh://git@devops.azure.com/HariSekhon/DevOps-Perl-tools'\ndest[27]='ssh://git@devops.azure.com/v3/harisekhon/GitHub/DevOps-Perl-tools'\n\nsrc[28]='ssh://git@devops.azure.com/HariSekhon/lib-java'\ndest[28]='ssh://git@devops.azure.com/v3/harisekhon/GitHub/lib-java'\n\nsrc[29]='ssh://git@devops.azure.com/harisekhon/lib'\ndest[29]='ssh://git@devops.azure.com/v3/harisekhon/GitHub/lib'\n\nsrc[30]='ssh://git@devops.azure.com:/HariSekhon/DevOps-Python-tools'\ndest[30]='ssh://git@devops.azure.com:/v3/harisekhon/GitHub/DevOps-Python-tools'\n\nsrc[31]='ssh://git@devops.azure.com/HariSekhon/lib-java'\ndest[31]='ssh://git@devops.azure.com/v3/harisekhon/GitHub/lib-java'\n\nsrc[32]='ssh://git@devops.azure.com:/HariSekhon/DevOps-Python-tools'\ndest[32]='ssh://git@devops.azure.com:/v3/harisekhon/GitHub/DevOps-Python-tools'\n\nsrc[33]='ssh://git@ssh.devops.azure.com/HariSekhon/DevOps-Perl-tools'\ndest[33]='ssh://git@ssh.devops.azure.com/v3/harisekhon/GitHub/DevOps-Perl-tools'\n\nsrc[34]='ssh://git@ssh.devops.azure.com/HariSekhon/lib'\ndest[34]='ssh://git@ssh.devops.azure.com/v3/harisekhon/GitHub/lib'\n\n# expands to the list of indicies in the array, starting at zero - this is easier to work with that ${#src} which is a total\n# that is off by one for index usage and doesn't support sparse arrays for any  missing/disabled test indicies\ntest_numbers=\"${!src[*]}\"\n\nfor i in $test_numbers; do\n    [ -n \"${src[$i]:-}\" ]  || { echo \"code error: src[$i] not defined\";  exit 1; }\n    [ -n \"${dest[$i]:-}\" ] || { echo \"code error: dest[$i] not defined\"; exit 1; }\n    echo \"git_to_azure_url ${src[$i]}\"\n    converted_repo_url=\"$(git_to_azure_url \"${src[$i]}\")\"\n    if [ \"$converted_repo_url\" != \"${dest[$i]}\" ]; then\n        echo \"ERROR: unit test failed\"\n        echo\n        echo \"Expected: ${dest[$i]}\"\n        echo \"Got:      $converted_repo_url\"\n        exit 2\n    fi\n    #echo \"checking URL result '$converted_repo_url' is valid\"\n    #if ! git ls-remote \"$converted_repo_url\"; then\n    #    echo \"ERROR: unit test failed - URL failed git ls-remote test\"\n    #    exit 3\n    #fi\n    echo\ndone\n\necho\necho \"SUCCESS: git_to_azure_url URL conversion tests passed\"\n"
  },
  {
    "path": "tests/test_spotify_uri_to_name.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-07-11 19:22:31 +0100 (Sat, 11 Jul 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# Requires $SPOTIFY_USER, $SPOTIFY_ID and $SPOTIFY_SECRET environment variables to run\n\ncd \"$srcdir/..\"\n\nURIs=\"\nspotify:track:3VLj6KVC1ZQMF36t24hvuL\nhttps://open.spotify.com/track/3VLj6KVC1ZQMF36t24hvuL?si=qFwFApYmQRuJ2Nqo3QyKng\n\nspotify:artist:1J2VVASYAamtQ3Bt8wGgA6\nhttps://open.spotify.com/artist/1J2VVASYAamtQ3Bt8wGgA6?si=wHJ4qxScSCihB8LgNbNjTQ\n\nspotify:album:2PIXzzS8WEzv8Ws92qspEH\nhttps://open.spotify.com/album/2PIXzzS8WEzv8Ws92qspEH?si=J_pcVRWrQc6_zJmH8P5yRg\n\"\n\nfor uri in $URIs; do\n    ./spotify_uri_to_name.sh <<< \"$uri\"\n    echo\n    SPOTIFY_CSV=1 ./spotify_uri_to_name.sh <<< \"$uri\"\n    echo\ndone\n\nSPOTIFY_CSV=1 ./spotify_uri_to_name.sh < \"../playlists/spotify/Rocky\"\necho\n\n./spotify_uri_to_name.sh \"../playlists/spotify/Rocky\"\necho\n\necho \"Spotify URI to name tests SUCCEEDED\"\n"
  },
  {
    "path": "travis/.travis.yml",
    "content": "#  vim:ts=2:sts=2:sw=2:et\n#\n#  Author: Hari Sekhon\n#  Date: 2014-11-29 01:02:47 +0000 (Sat, 29 Nov 2014)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback\n#  to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# ============================================================================ #\n#                               T r a v i s   C I\n# ============================================================================ #\n\n# https://docs.travis-ci.com/user/customizing-the-build/\n\n---\nversion: ~> 1.0\n\n# ==============================================================\n# https://docs.travis-ci.com/user/languages/minimal-and-generic/\nlanguage: bash\n\n# =========================================\n# https://docs.travis-ci.com/user/multi-os/\nos:\n  - linux\n  - osx\n\n# ==============================================\n# https://docs.travis-ci.com/user/reference/osx/\n# macOS 10.15.7 - otherwise defaults to Mac macOS 10.13 with xcode9.4 otherwise - and HomeBrew update takes 50 minutes until the build times out :-/\nosx_image: xcode12.2\n\n# =======================================\n# https://docs.travis-ci.com/user/docker/\nservices:\n  - docker\n\n# ======================================================\n# https://docs.travis-ci.com/user/environment-variables/\nenv:\n  - PYTHONUNBUFFERED=1\n\n# ==============================================\n# https://docs.travis-ci.com/user/notifications/\nnotifications:\n  email: false\n\n# =================================================================================\n# https://docs.travis-ci.com/user/customizing-the-build/#building-specific-branches\n# https://docs.travis-ci.com/user/conditional-builds-stages-jobs\n#branches:\n#  only:\n#    - master\n\n# ========================================\n# https://docs.travis-ci.com/user/caching/\n\nbefore_cache:\n  - rm -f -- \"$HOME/.cache/pip/log/debug.log\"\n\ncache:\n  - pip\n  - directories:\n      - $HOME/.cache\n      - $HOME/.cpan\n      - $HOME/.cpanm\n      - $HOME/.gem\n      - pytools_checks\n\n# ==============================================\n# https://docs.travis-ci.com/user/job-lifecycle/\n\n# avoid package checksum mismatches when installing packages\nbefore_install:\n  - sudo rm -f  -- \"${TRAVIS_ROOT}/etc/apt/apt.conf.d/99-travis-apt-proxy\"\n  - sudo rm -rf -- \"${TRAVIS_ROOT}/var/lib/apt/lists/\"*\n\ninstall:\n  #- travis_retry make\n  - make\n\nscript:\n  #- travis_retry make test\n  - make test\n\nafter_success:\n  # Alpine Github\n  #- curl --header \"Content:Type:application/json\" --data '{\"build\":true}' -X POST https://cloud.docker.com/api/build/v1/source/df816f2a-9407-4f1b-8b51-39615d784e65/trigger/8d9cb826-48df-439c-8c20-1975713064fc/call/\n  # Debian Github\n  #- curl --header \"Content:Type:application/json\" --data '{\"build\":true}' -X POST https://cloud.docker.com/api/build/v1/source/439eff84-50c7-464a-a49e-0ac0bf1a9a43/trigger/0cfb3fe7-2028-494b-a43b-068435e6a2b3/call/\n  # CentOS Github\n  #- curl --header \"Content:Type:application/json\" --data '{\"build\":true}' -X POST https://cloud.docker.com/api/build/v1/source/efba1846-5a9e-470a-92f8-69edc1232ba0/trigger/316d1158-7ffb-49a4-a7bd-8e5456ba2d15/call/\n  # Ubuntu Github\n  #- curl --header \"Content:Type:application/json\" --data '{\"build\":true}' -X POST https://cloud.docker.com/api/build/v1/source/8b3dc094-d4ca-4c92-861e-1e842b5fac42/trigger/abd4dbf0-14bc-454f-9cde-081ec014bc48/call/\n"
  },
  {
    "path": "travis/travis_api.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: /repos\n#  args: /user\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-17 18:35:11 +0100 (Thu, 17 Sep 2020)\n#\n#  https://travis.com/harisekhon/bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/git.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQueries the Travis CI API v3 for travis-ci.org (open source site)\n\nModify url_base from travis-ci.org to travis-ci.com if you need to use this on the pro site\n\nRequires \\$TRAVIS_TOKEN to be available in the environment\n\nCan specify \\$CURL_OPTS for options to pass to curl or provide them as arguments\n\n\nGet an API access token here:\n\n    https://travis-ci.org/account/preferences\n\nor call 'travis token' using the CLI gem as documented here:\n\n    https://developer.travis-ci.org/authentication\n\n\nAPI Reference:\n\n    https://developer.travis-ci.com/\n\n\nSee Also:\n\n    https://developer.travis-ci.org/explore/\n\n\nExamples:\n\n\n# Get the currently authenticated user:\n\n    ${0##*/} /user\n\n\n# Get Organizations the current user has access to:\n\n    ${0##*/} /orgs\n\n\n# Get Repositories:\n\n    ${0##*/} /repos\n\n\n# Get builds:\n\n    ${0##*/} /builds\n\n\n# Get crons (repository slug must be url-encoded replacing slash with %2F):\n\n    ${0##*/} /repo/<user>%2F<repo>/crons\n\n    ${0##*/} /repo/HariSekhon%2FDevOps-Bash-tools/crons\n\n\n# Get jobs:\n\n    ${0##*/} /jobs\n\n\n# Get repository settings:\n\n    ${0##*/} /repo/<user>%2F<repo>/settings\n\n    ${0##*/} /repo/HariSekhon%2FDevOps-Bash-tools/settings\n\n\n# List caches for a repository:\n\n    ${0##*/} /repo/<user>%2F<repo>/caches\n\n    ${0##*/} /repo/HariSekhon%2FDevOps-Bash-tools/caches\n\n\n# List environment variables for a repository:\n\n    ${0##*/} /repo/<user>%2F<repo>/env_vars\n\n    ${0##*/} /repo/HariSekhon%2FDevOps-Bash-tools/env_vars\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"/path [<curl_options>]\"\n\nurl_base=\"https://api.travis-ci.org\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncheck_env_defined TRAVIS_TOKEN\n\nexport TOKEN=\"$TRAVIS_TOKEN\"\n\ncurl_api_opts \"$@\"\n\nurl_path=\"${1:-}\"\nshift || :\n\nurl_path=\"${url_path//https:\\/\\/api.travis-ci.org}\"\nurl_path=\"${url_path##/}\"\n\nexport CURL_AUTH_HEADER=\"Authorization: token\"\n\n\"$srcdir/../bin/curl_auth.sh\" \"$url_base/$url_path\" -H 'Travis-API-Version: 3' \"${CURL_OPTS[@]}\" \"$@\"\n"
  },
  {
    "path": "travis/travis_delete_cron.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 20:47:00 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes a given Travis CI cron by ID\n\nSee travis_repo_crons.sh which outputs the IDs are the first field\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<cron_id>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\ncron_id=\"$1\"\n\nurl_path=\"/cron/$cron_id\"\n\ntimestamp \"Deleting Travis CI cron job '$cron_id'\"\n\"$srcdir/travis_api.sh\" \"$url_path\" -X DELETE\n"
  },
  {
    "path": "travis/travis_foreach_repo.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: bash -c 'echo user={user} name={name} repo={repo}'\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 23:53:57 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nRun a command for each Travis CI repo\n\nAll arguments become the command template\n\nWARNING: do not run any command reading from standard input, otherwise it will consume the repo names and exit after the first iteration\n\nThe command template replaces the following for convenience in each iteration:\n\n{username}, {user}    => your authenticated user / organization\n{name}                => the repo name without the user prefix\n{repo}                => the repo name with the user prefix\n\neg.\n    ${0##*/} echo user={user} name={name} repo={repo}\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<command> <args>\"\n\nhelp_usage \"$@\"\n\nmin_args 1 \"$@\"\n\n\"$srcdir/travis_repos.sh\" |\nwhile read -r repo; do\n    user=\"${repo%%/*}\"\n    name=\"${repo##*/}\"\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo \"# ============================================================================ #\" >&2\n        echo \"# $repo\" >&2\n        echo \"# ============================================================================ #\" >&2\n    fi\n    cmd=(\"$@\")\n    cmd=(\"${cmd[@]//\\{username\\}/$user}\")\n    cmd=(\"${cmd[@]//\\{user\\}/$user}\")\n    cmd=(\"${cmd[@]//\\{repo\\}/$repo}\")\n    cmd=(\"${cmd[@]//\\{name\\}/$name}\")\n    # need eval'ing to able to inline quoted script\n    # shellcheck disable=SC2294\n    eval \"${cmd[@]}\"\n    if [ -z \"${NO_HEADING:-}\" ]; then\n        echo >&2\n    fi\ndone\n"
  },
  {
    "path": "travis/travis_lint.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLints the local .travis.yml using the Travis CI API\n\nFor production CI see instead the 'travis' gem and the adjacent script 'check_travis_yml.sh'\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<.travis.yml>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -eq 1 ]; then\n    travis_yml=\"$1\"\nelif [ -f .travis.yml ]; then\n    travis_yml=.travis.yml\nelse\n    usage\nfi\n\n\"$srcdir/travis_api.sh\" \"/lint\" -X POST -d @\"$travis_yml\"\n"
  },
  {
    "path": "travis/travis_repo_build.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-16 09:51:44 +0100 (Fri, 16 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nTriggers a build for the given Travis CI repo\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\ntimestamp \"Triggering Travis CI build for repo '$repo'\"\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\n\"$srcdir/travis_api.sh\" \"/repo/$repo/requests\" -X POST \"$@\"\n"
  },
  {
    "path": "travis/travis_repo_caches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.travis-ci.org/resource/caches#Caches\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the caches for a given Travis CI repo\n\nOutput Format;\n\n<repo>    <branch>    <size>    <last_modified>    <long_name>\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\n\"$srcdir/travis_api.sh\" \"/repo/$repo/caches\" \"$@\" |\njq -r '.caches[] | [.repo.slug, .branch, .size, .last_modified, .name] | @tsv'\n"
  },
  {
    "path": "travis/travis_repo_create_cron.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 12:15:48 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd a cron job to a given Travis CI repo using the Travis CI API\n\nSince you can only have 1 cronjob per branch per repo, this overwrites any existing cron on that branch\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nArgs:\n\nbranch              - defaults to 'master'\ninterval            - defaults to 'monthly'. Options: daily, weekly, monthly\nrecent_dont_rerun   - boolean, defaults to 'true'. Don't run cron job if a build has occurred in the last 24 hours. Set to 0 or 'false' to disable this, any other value is taken as 'true'\n\nPrints the JSON of the cron it just created showing all the details\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<branch>] [<interval>] [<recent_dont_rerun>] [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nbranch=\"${2:-master}\"\ninterval=\"${3:-monthly}\"\nrecent_dont_rerun=\"${4:-true}\"\nshift || :\nshift || :\nshift || :\nshift || :\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\nrecent_dont_rerun=\"${recent_dont_rerun//[[:space:]]/}\"\ncase \"$recent_dont_rerun\" in\n    0  | false ) recent_dont_rerun=0\n                 ;;\n             * ) recent_dont_rerun=1\n                 ;;\nesac\n\n\"$srcdir/travis_api.sh\" \"/repo/$repo/branch/$branch/cron\" -X POST -d \"cron.interval=$interval&cron.dont_run_if_recent_build_exists=$recent_dont_rerun\" |\njq -r '\"Created cron for repo \\\"\" + .repository.slug + \"\\\" on branch \\\"\" + .branch.name + \"\\\" at interval \\\"\" + .interval +\"\\\"\"' |\ntimestamp \"$(cat)\"\n"
  },
  {
    "path": "travis/travis_repo_crons.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 20:47:00 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all crons for a given Travis CI repo using the Travis CI API\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\n\nOutput Format:\n\n<id>    <branch>    <interval>    <created_at>    <last_run>    <next_run>\n\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\nnext=\"/repo/$repo/crons\"\n\nget_crons(){\n    local url_path=\"$1\"\n    shift || :\n    local output\n    output=\"$(\"$srcdir/travis_api.sh\" \"$url_path\" \"$@\")\"\n    jq -r '.crons[] | [.id, .branch.name, .interval, .created_at, .last_run, .next_run] | @tsv' <<< \"$output\"\n    next=\"$(jq -r '.[\"@pagination\"].next[\"@href\"]' <<< \"$output\")\"\n}\n\n# iterate over all next hrefs to get through all pages of crons\nwhile [ -n \"$next\" ] &&\n      [ \"$next\" != null ]; do\n    get_crons \"$next\" \"$@\"\ndone\n"
  },
  {
    "path": "travis/travis_repo_delete_caches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes all caches for the given Travis CI repo\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nrepo_encoded=\"$(travis_prefix_encode_repo \"$repo\")\"\n\ntimestamp \"Deleting all caches for repo '$repo'\"\n\"$srcdir/travis_api.sh\" \"/repo/$repo_encoded/caches\" -X DELETE \"$@\"\n"
  },
  {
    "path": "travis/travis_repo_delete_crons.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes all cron jobs for the given Travis CI repo\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nif [ -z \"$repo\" ]; then\n    repo=\"$(git_repo)\"\nfi\n\ntimestamp \"Deleting all cron jobs for repo '$repo'\"\n\"$srcdir/travis_repo_crons.sh\" \"$repo\" \"$@\" |\nwhile read -r cron_id rest; do\n    \"$srcdir/travis_delete_cron.sh\" \"$cron_id\" \"$@\"\ndone\n"
  },
  {
    "path": "travis/travis_repo_env_vars.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the environment variables for a given Travis CI repo in JSON format\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\n\"$srcdir/travis_api.sh\" \"/repo/$repo/env_vars\" \"$@\"\n"
  },
  {
    "path": "travis/travis_repo_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: DevOps-Bash-tools\n#  args: HariSekhon/DevOps-Bash-tools\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the settings for a given Travis CI repo in TSV format\n\nOutput format:\n\n<setting_name>      <setting_value>\n\nIf no repo is given, then tries to determine the repo name from the local git remote url\n\nIf the repo doesn't have a user / organization prefix, then queries\nthe Travis CI API for the currently authenticated username first\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<user>/]<repo> [<curl_options>]\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nrepo=\"${1:-}\"\nshift || :\n\nrepo=\"$(travis_prefix_encode_repo \"$repo\")\"\n\n\"$srcdir/travis_api.sh\" \"/repo/$repo/settings\" \"$@\" |\njq -r '.settings[] | [.name, .value] | @tsv' |\ncolumn -t\n"
  },
  {
    "path": "travis/travis_repos.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 20:47:00 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n#. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all Travis CI repos using the Travis CI API\n\nSince the API returns all GitHub repos, this code filters for those marked 'active'\nwhich are the ones which show up in Travis CI and which run CI builds\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<curl_options>]\"\n\nhelp_usage \"$@\"\n\nnext=\"/repos\"\n\nget_repos(){\n    local url_path=\"$1\"\n    shift || :\n    local output\n    output=\"$(\"$srcdir/travis_api.sh\" \"$url_path\" \"$@\")\"\n    jq -r '.repositories[] | select(.active == true) | .slug' <<< \"$output\"\n    next=\"$(jq -r '.[\"@pagination\"].next[\"@href\"]' <<< \"$output\")\"\n}\n\n# iterate over all next hrefs to get through all pages of repos\nwhile [ -n \"$next\" ] &&\n      [ \"$next\" != null ]; do\n    get_repos \"$next\" \"$@\"\ndone\n"
  },
  {
    "path": "travis/travis_repos_caches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://developer.travis-ci.org/resource/caches#Caches\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the caches for all Travis CI repos\n\nOutput Format;\n\n<repo>    <branch>    <size>    <last_modified>    <long_name>\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_caches.sh' '{repo}'\"\n"
  },
  {
    "path": "travis/travis_repos_create_cron.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 12:15:48 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nAdd a cron job to all Travis CI repos using the Travis CI API\n\nSince you can only have 1 cronjob per branch per repo, this overwrites any existing crons\n\nArgs:\n\nbranch              - defaults to 'master'\ninterval            - defaults to 'monthly'. Options: daily, weekly, monthly\nrecent_dont_rerun   - boolean, defaults to 'true'. Don't run cron job if a build has occurred in the last 24 hours. Set to 0 or 'false' to disable this, any other value is taken as 'true'\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"[<branch>] [<interval>] [<recent_dont_rerun>] [<curl_options>]\"\n\nhelp_usage \"$@\"\n\nbranch=\"${1:-master}\"\ninterval=\"${2:-monthly}\"\nrecent_dont_rerun=\"${3:-true}\"\nshift || :\nshift || :\nshift || :\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_create_cron.sh' '{repo}' '$branch' '$interval' '$recent_dont_rerun'\"\n"
  },
  {
    "path": "travis/travis_repos_crons.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-09-30 20:47:00 +0100 (Wed, 30 Sep 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nLists all crons for all Travis CI repos using the Travis CI API\n\nOutput Format:\n\n<repo>    <id>    <branch>    <interval>    <created_at>    <last_run>    <next_run>\n\n\nUses the adjacent travis_api.sh script\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_crons.sh' '{repo}' | perl -pn -e 's/^/{name}\\\\t/'\"\n"
  },
  {
    "path": "travis/travis_repos_delete_caches.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes all caches for all Travis CI repos\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_delete_caches.sh' '{repo}' > /dev/null\"\n"
  },
  {
    "path": "travis/travis_repos_delete_crons.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nDeletes all cron jobs for all Travis CI repos\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_delete_crons.sh' '{repo}'\"\n"
  },
  {
    "path": "travis/travis_repos_settings.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-10-01 00:19:22 +0100 (Thu, 01 Oct 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/travis.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nShows the settings for all Travis CI repos in TSV format\n\nOutput format:\n\n<repo_name>     <setting_name>      <setting_value>\n\nHandy way of checking your settings are consistent across repos.\n\nTo compare each setting side by side, consider this command:\n\n    travis_repos_settings.sh | column -t | sort -k3\n\nUses the adjacent travis_*.sh scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"\"\n\nhelp_usage \"$@\"\n\nexport NO_HEADING=1\n\n\"$srcdir/travis_foreach_repo.sh\" \"'$srcdir/travis_repo_settings.sh' '{repo}' | perl -pn -e 's/^/{name}\\\\t/'\"\n"
  },
  {
    "path": "vagrant/vagrant_hosts.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: vagrant/kubernetes/Vagrantfile\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-15 12:50:08 +0100 (Sat, 15 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nQuick script to parse a given Vagrantfile and emit an /etc/hosts format output, eg:\n\n172.16.0.2 kube1.local kube1\n172.16.0.3 kube2.local kube2\n\nVagrantfile can be given as first arg, otherwise checks for \\$PWD/Vagrantfile or /vagrant/Vagrantfile for convenience\n\nTested on vagrant/kubernetes/Vagrantfile in this repo and used as part of provision scripts\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<Vagrantfile>\"\n\nhelp_usage \"$@\"\n\n#min_args 1 \"$@\"\n\nif [ $# -gt 0 ]; then\n    Vagrantfile=\"$1\"\nelif [ -f Vagrantfile ]; then\n    Vagrantfile=Vagrantfile\nelif [ -f /vagrant/Vagrantfile ]; then\n    # auto-detect when running inside a Vagrant VM\n    Vagrantfile=/vagrant/Vagrantfile\nelse\n    usage \"Vagrantfile not specified and no Vagrantfile found in \\$PWD or /vagrant/\"\nfi\n\nsed 's/#.*//; /^[[:space:]]*$/d' \"$Vagrantfile\" |\ngrep -A 20 '^[[:space:]]*config.vm.define' |\ngrep -e '^[[:space:]]*config.vm.define' \\\n     -e '^[[:space:]]*config.vm.network.*private_network.*ip' |\ngrep -v -e \"^--\" -e \"default_hostname\" |\nsed '\n    s/ do .*//;\n    s/config.vm.[[:alnum:]]*//;\n    s/private_network//g;\n    s/[^[:alnum:][:space:].]//g;\n    s/[[:space:]]ip[[:space:]]//\n' |\nwhile read -r host; do\n    read -r ip\n    echo \"$ip $host.local $host\"\ndone\n"
  },
  {
    "path": "vagrant/vagrant_total_mb.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#  args: vagrant/kubernetes/Vagrantfile\n#\n#  Author: Hari Sekhon\n#  Date: 2020-08-23 23:08:43 +0100 (Sun, 23 Aug 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\n# shellcheck disable=SC1090,SC1091\n. \"$srcdir/lib/utils.sh\"\n\n# shellcheck disable=SC2034,SC2154\nusage_description=\"\nCalculates the total combined RAM in MB allocated to all VMs in one or more Vagrantfiles\n\nAccepts one or more Vagrantfiles as arguments, otherwise tries to read \\$PWD/Vagrantfile or /vagrant/Vagrantfile for convenience\n\nTested on vagrant/kubernetes/Vagrantfile in this repo\n\"\n\n# used by usage() in lib/utils.sh\n# shellcheck disable=SC2034\nusage_args=\"<Vagrantfile>\"\n\nhelp_usage \"$@\"\n\nif [ $# -gt 0 ]; then\n    Vagrantfiles=(\"$@\")\nelif [ -f Vagrantfile ]; then\n    Vagrantfiles=(Vagrantfile)\nelif [ -f /vagrant/Vagrantfile ]; then\n    # auto-detect when running inside a Vagrant VM\n    Vagrantfiles=(/vagrant/Vagrantfile)\nelse\n    usage \"Vagrantfile not specified and no Vagrantfile found in \\$PWD or /vagrant/\"\nfi\n\ngrep -E '^[^#]+\\.memory' \"${Vagrantfiles[@]}\" |\nsed 's/.*=[[:space:]]*//' |\ngrep -E '^[[:digit:]]+(\\.[[:digit:]]+)?$' |\ntr '\\n' '+' |\nsed 's/+$//' |\nbc -l\n"
  },
  {
    "path": "wercker/.werckerignore",
    "content": "**/.md\n"
  },
  {
    "path": "wercker/wercker.yml",
    "content": "#\n#  Author: Hari Sekhon\n#  Date: 2020-02-24 15:41:04 +0000 (Mon, 24 Feb 2020)\n#\n#  vim:ts=2:sts=2:sw=2:et\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# https://devcenter.wercker.com/reference/wercker-yml/\n\nbox: debian\n\nbuild:\n  steps:\n    - script:\n        name: ci bootstrap\n        code: setup/ci_bootstrap.sh\n    - script:\n        name: init\n        code: make init\n    - script:\n        name: build\n        code: make ci\n    - script:\n        name: test\n        code: make test\n"
  },
  {
    "path": "wercker/wercker_api_app.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-02 17:53:39 +0000 (Mon, 02 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Fetch Wercker App details - needed for getting Wercker CI build IDs (eg. for shields.io)\n\n# https://devcenter.wercker.com/development/api/endpoints/applications/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncd \"$srcdir\"\n\nusage(){\n    echo \"${0##*/} <user>/<application_name>\"\n    exit 3\n}\n\nif [ $# -ne 1 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\napplication=\"${1:-}\"\n\nif ! [[ \"$application\" =~ / ]]; then\n    application=\"${GITHUB_USER:-${GIT_USER:-${USER:-}}}/$application\"\nfi\n\ncurl -sS \"https://app.wercker.com/api/v3/applications/$application\"\n"
  },
  {
    "path": "wercker/wercker_api_runs.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-02 17:53:39 +0000 (Mon, 02 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Fetch Wercker App runs by name\n\n# https://devcenter.wercker.com/development/api/endpoints/runs/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncd \"$srcdir\"\n\nusage(){\n    echo \"${0##*/} <user>/<application_name>\"\n    exit 3\n}\n\nif [ $# -ne 1 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\napplication=\"${1:-}\"\n\napplication_id=\"$(\"$srcdir/wercker_app_id.sh\" \"$application\")\"\n\n# pipeline id also works but easier to reuse the app id\n#\n# eg. curl -sS \"https://app.wercker.com/api/v3/runs?pipelineId=5e53ee690783f9080047a6ce\"\n#\ncurl -sS \"https://app.wercker.com/api/v3/runs?applicationId=$application_id\"\n"
  },
  {
    "path": "wercker/wercker_api_workflows.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-02 17:53:39 +0000 (Mon, 02 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# Fetch Wercker App workflows by name\n\n# https://devcenter.wercker.com/development/api/endpoints/workflows/\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\ncd \"$srcdir\"\n\nusage(){\n    echo \"${0##*/} <user>/<application_name>\"\n    exit 3\n}\n\nif [ $# -ne 1 ]; then\n    usage\nfi\n\nfor arg; do\n    case \"$arg\" in\n        -*) usage\n            ;;\n    esac\ndone\n\napplication=\"${1:-}\"\n\napplication_id=\"$(\"$srcdir/wercker_app_id.sh\" \"$application\")\"\n\ncurl -sS \"https://app.wercker.com/api/v3/workflows?applicationId=$application_id\"\n"
  },
  {
    "path": "wercker/wercker_app_id.sh",
    "content": "#!/usr/bin/env bash\n#  vim:ts=4:sts=4:sw=4:et\n#\n#  Author: Hari Sekhon\n#  Date: 2020-03-02 17:53:39 +0000 (Mon, 02 Mar 2020)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\n# fetch Wercker repo details - needed for getting Wercker CI build IDs (eg. for shields.io)\n\nset -euo pipefail\n[ -n \"${DEBUG:-}\" ] && set -x\nsrcdir=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\n\nusage(){\n    echo \"${0##*/} <user>/<application>\"\n    exit 3\n}\n\nif [ $# -ne 1 ]; then\n    usage\nfi\n\n\"$srcdir/wercker_api_app.sh\" \"$@\" |\njq -r '.id'\n"
  },
  {
    "path": "yamllint/config",
    "content": "#  vim:ts=2:sts=2:sw=2:et:filetype=yaml\n#\n#  Author: Hari Sekhon\n#  Date: 2019-10-10 10:32:04 +0100 (Thu, 10 Oct 2019)\n#\n#  https://github.com/HariSekhon/DevOps-Bash-tools\n#\n#  License: see accompanying Hari Sekhon LICENSE file\n#\n#  If you're using my code you're welcome to connect with me on LinkedIn and optionally send me feedback to help improve or steer this or other code I publish\n#\n#  https://www.linkedin.com/in/HariSekhon\n#\n\nextends: default\n\nrules:\n  # don't warn on no --- at start since many yamls don't have this\n  document-start: disable\n\n  # often want to differentiate no space commented out '#code' from '# intentional human comments'\n  comments: disable\n\n  # > 80 char length? Really? I love the 80s but this one should have been left there...\n  line-length: disable\n\n  # too many spaces after colons - meh - sometimes it's nice to align adjacent lines values\n  colons: disable\n"
  }
]