[
  {
    "path": ".gitignore",
    "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n#  Usually these files are written by a python script from a template\n#  before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n/site\n\n# mypy\n.mypy_cache/\n"
  },
  {
    "path": "README.md",
    "content": "<p align=\"center\"><img src=\"images/devops_resources.png\"/></p>\n\n:information_source: &nbsp;This repository is about gathering any useful resources and information regarding DevOps and secondly, provide some roadmap for those who want to practice DevOps.\n\n:pencil: &nbsp;Feel free to add more resources by submitting pull requests\n\n****\n\n<!-- ALL-TOPICS-LIST:START -->\n<!-- prettier-ignore-start -->\n<!-- markdownlint-disable -->\n<center>\n<table>\n  <tr>\n    <td align=\"center\"><a href=\"#devops\"><img src=\"images/devops.png\" width=\"70px;\" height=\"75px;\" alt=\"DevOps\" /><br /><b>DevOps</b></a></td>\n    <td align=\"center\"><a href=\"resources/jenkins/README.md\"><img src=\"images/jenkins.png\" width=\"70px;\" height=\"85px;\" alt=\"Jenkins\"/><br /><b>Jenkins</b></a></td>\n    <td align=\"center\"><a href=\"resources/git.md\"><img src=\"images/git.png\" width=\"75px;\" height=\"75px;\" alt=\"Git\"/><br /><b>Git</b></a></td>\n    <td align=\"center\"><a href=\"resources/ansible.md\"><img src=\"images/ansible.png\" width=\"65px;\" height=\"75px;\" alt=\"Ansible\"/><br /><b>Ansible</b></a></td>\n    <td align=\"center\"><a href=\"resources/network.md\"><img src=\"images/network.png\" width=\"80x;\" height=\"75px;\" alt=\"Network\"/><br /><b>Network</b></a></td>\n    <td align=\"center\"><a href=\"resources/linux.md\"><img src=\"images/linux.png\" width=\"75x;\" height=\"75px;\" alt=\"Linux\"/><br /><b>Linux</b></a></td>\n    <td align=\"center\"><a href=\"resources/terraform.md\"><img src=\"images/terraform.png\" width=\"75px;\" height=\"75px;\" alt=\"Terraform\"/><br /><b>Terraform</b></a></td>\n  </tr>\n  <tr>\n    <td align=\"center\"><a href=\"resources/coding.md\"><img src=\"images/coding.png\" width=\"75px;\" height=\"75px;\" alt=\"coding\"/><br /><b>Coding</b></a></td>\n    <td align=\"center\"><a href=\"resources/python.md\"><img src=\"images/python.png\" width=\"70px;\" height=\"75px;\" alt=\"Python\"/><br /><b>Python</b></a></td>\n    <td align=\"center\"><a href=\"resources/go.md\"><img src=\"images/go.png\" width=\"80px;\" height=\"75px;\" alt=\"Go\"/><br /><b>Go</b></a></td>\n    <td align=\"center\"><a href=\"resources/bash.md\"><img src=\"images/bash.png\" width=\"70px;\" height=\"75px;\" alt=\"Bash\"/><br /><b>Shell Scripting</b></a></td>\n    <td align=\"center\"><a href=\"resources/kubernetes.md\"><img src=\"images/kubernetes.png\" width=\"75px;\" height=\"75px;\" alt=\"kubernetes\"/><br /><b>Kubernetes</b></a></td>\n    <td align=\"center\"><a href=\"resources/victoriametrics.md\"><img src=\"images/victoriametrics.png\" width=\"64px;\" height=\"75px;\" alt=\"VictoriaMetrics\"/><br /><b>VictoriaMetrics</b></a></td>\n    <td align=\"center\"><a href=\"resources/prometheus.md\"><img src=\"images/prometheus.png\" width=\"75px;\" height=\"75px;\" alt=\"Prometheus\"/><br /><b>Prometheus</b></a></td>\n    <td align=\"center\"><a href=\"resources/mongo.md\"><img src=\"images/mongo.png\" width=\"75px;\" height=\"75px;\" alt=\"Mongo\"/><br /><b>Mongo</b></a></td>\n  </tr>\n  <tr>\n    <td align=\"center\"><a href=\"resources/cloud.md\"><img src=\"images/cloud.png\" width=\"90px;\" height=\"85px;\" alt=\"Cloud\"/><br /><b>Cloud</b></a></td>\n    <td align=\"center\"><a href=\"resources/aws.md\"><img src=\"images/aws.png\" width=\"100px;\" height=\"75px;\" alt=\"AWS\"/><br /><b>AWS</b></a></td>\n    <td align=\"center\"><a href=\"resources/azure.md\"><img src=\"images/azure.png\" width=\"80px;\" height=\"75px;\" alt=\"azure\"/><br /><b>Azure</b></a></td>\n    <td align=\"center\"><a href=\"resources/gcp.md\"><img src=\"images/gcp.png\" width=\"75px;\" height=\"75px;\" alt=\"gcp\"/><br /><b>Google Cloud Platform</b></a></td>\n    <td align=\"center\"><a href=\"resources/openstack.md\"><img src=\"images/openstack.png\" width=\"75px;\" height=\"75px;\" alt=\"openstack\"/><br /><b>OpenStack</b></a></td>\n    <td align=\"center\"><a href=\"resources/security.md\"><img src=\"images/security.png\" width=\"75px;\" height=\"75px;\" alt=\"security\"/><br /><b>Security</b></a></td>\n    <td align=\"center\"><a href=\"resources/puppet.md\"><img src=\"images/puppet.png\" width=\"75px;\" height=\"75px;\" alt=\"puppet\"/><br /><b>Puppet</b></a></td>\n  </tr>\n\n  <tr>\n    <td align=\"center\"><a href=\"resources/elk.md\"><img src=\"images/elk.png\" width=\"75px;\" height=\"75px;\" alt=\"ELK\"/><br /><b>ELK</b></a></td>\n    <td align=\"center\"><a href=\"resources/dataops.md\"><img src=\"images/dataops.png\" width=\"75px;\" height=\"75px;\" alt=\"ELK\"/><br /><b>DataOps</b></a></td>\n    <td align=\"center\"><a href=\"resources/openshift.md\"><img src=\"images/openshift.png\" width=\"75px;\" height=\"75px;\" alt=\"OpenShift\"/><br /><b>OpenShift</b></a></td>\n    <td align=\"center\"><a href=\"resources/sql.md\"><img src=\"images/sql.png\" width=\"75px;\" height=\"75px;\" alt=\"sql\"/><br /><b>SQL</b></a></td>\n    <td align=\"center\"><a href=\"resources/containers.md\"><img src=\"images/containers.png\" width=\"75px;\" height=\"75px;\" alt=\"Containers\"/><br /><b>Containers</b></a></td>\n  </tr>\n</table>\n</center>\n<!-- markdownlint-enable -->\n<!-- prettier-ignore-end -->\n<!-- ALL-TOPICS-LIST:END -->\n\n\n* [DevOps Roadmap](#devops-roadmap)\n* [Alternative Roadmaps](#alternative-roadmaps)\n* [Learn About DevOps](#learn-about-devops)\n* [Learn Git](#learn-git)\n* [Programming](#programming)\n* [Operating Systems](#operating-systems)\n* [DevOps Toolchain](#devops-toolchain)\n\n\n<a name=\"devops\"></a>\n## DevOps\n\n### DevOps Roadmap\n\n  * The following path is opinionated. You can find alternative paths in the [alternative roadmaps section](#alternative-roadmaps)\n  * Do NOT attempt to learn everything, especially if you are a complete beginner (the burnout will make you regret you ever heard the word DevOps)\n  * Learn enough to say \"I know what is DevOps and I understand how to practice it\". You can learn everything else by demand or when you feel ready\n\nThe must list\n\nTopic | What to learn | Alternatives | Description\n:------|:------|:------|:------:\nVersion Control | [Git](#Learn-git) | - | Version Control is used everywhere and Git is the leading version control technology\nProgramming | At least one [programming language](#programming) | None. Don't settle for learning only shell scripting  | Anyone who practice DevOps should know programming to some extent\nOperating System | [Linux](#operating-systems) | Other distribution of Linux :D | Operating Systems is another must when it comes to DevOps\nProtocols | DHCP, UDP, TCP, HTTP, SSH | Not as an alternative, you can simply learn later: NFS, NTP, FTP | Learn how these protocols work and what they are used for\nApplications | Web Server, Load Balancer, Cache, Database | | Learn how they work and try out some implementations (e.g. MySQL for databases and httpd for web servers)\n[System Design](https://github.com/bregman-arie/system-design-notebook) | Availability, Reliability, Scalability, Performance | Learn how to design infrastructure and systems\n\nThe second must list :)\nSeriously though, don't try to master everything in this list and start with something that you are interested in. The order has no meaning in this case.\n\nTopic | What to learn | Alternatives | Description\n:------|:------|:------|:------:\nConfiguration Management and IaC | Ansible and Terraform | CM: Chef, Puppet and Salt. IaC: Pulumi and CloudFormation |\nCloud | Azure, AWS, GCP | - |\nContainers | Podman and Kubernetes| Docker instead of Podman. Nomad instead of Kubernetes | \nCI/CD | Jenkins, GitHub Actions | Gitlab CI, Circle CI, Argo CD |\nLogging and Dashboards | Elastic Stack, Report Portal, Grafana | Graylog, Splunk |\nMonitoring | VictoriaMetrics, Prometheus | Nagios, Centreon, Datadog | \n\n### Alternative Roadmaps\n\nThere are other DevOps roadmaps and learning plans out there. If you feel the one above is little bit too much or just want to look at other options, maybe you'll find the following useful\n\nName | Description\n:------|:------:\n[DevOps Roadmap](https://github.com/kamranahmedse/developer-roadmap) | DevOps skills roadmap. What to to learn at each step.\n[DevOps 3 Weeks Plan](https://medium.com/@medineshkatwal/devops-3-week-plan-from-16-oct-2019-to-31st-oct-2019-af486e625f72) | Detailed plan on what to do at each day\n[How to Become a DevOps Engineer in Six Months?](https://spacelift.io/blog/how-to-become-devops-engineer) | Skills you need to become DevOps Engineer\n\n### Learn about DevOps\n\nName | Description\n:------|:------:\n[What is DevOps? - Amazon](https://aws.amazon.com/devops) | Learn what is DevOps by Amazon and which AWS services can help you to practice DevOps\n[What is DevOps? - Red Hat](https://www.redhat.com/en/topics/devops) | Learn what is DevOps by Red Hat, including many interesting articles\n[What is DevOps? - Microsoft](https://docs.microsoft.com/en-us/azure/devops/learn/what-is-devops) | Learn what is DevOps by Microsofy, including Azure DevOps service\n[What is DevOps? - Google](https://cloud.google.com/devops) | Learn what is DevOps by Google\n[Who is DevOps? - Spacelift](https://spacelift.io/blog/who-is-devops-engineer) | Who is DevOps? Is it Worth it to Become a DevOps Engineer?\n\n### Learn Git\n\nMake sure you are comfortable with the following list once you done learning Git basics\n\n  * Commit changes to a project\n  * Submit pull/merge requests\n  * Working with branches\n  * Rebase changes (change order, squash, ...)\n\nName | Description\n:------|:------:\n[Interactive Git Branching Learning](https://learngitbranching.js.org/) | Visual and interactive way to learn Git branching\n[Learn git concepts, not commands](https://dev.to/unseenwizzard/learn-git-concepts-not-commands-4gjc) | Article on Git concepts\n[Codeacademy Learn Git](https://www.codecademy.com/learn/learn-git) | Not Free \n\n### Programming\n\nPracticing DevOps without knowing programming is like eating cornflakes with ketchup - it's just doesn't makes sense (yes, my analogies are the worst).<br>\nHere is a list of some good sites to practice programming in general:\n\nName | Comments\n:------|:------:\n[HackerRank](https://www.hackerrank.com) | Practice Coding skills\n[Exercism](https://exercism.io) | Over 48 languages and 2600 exercises\n[LeetCode](https://leetcode.com) | Practice programming online\n\nA better approach might be to actually write some DevOps related scripts or projects. Start simple. Here are some ideas:\n\n  * Write a script that gets a number of instances and a name of a platform (AWS, local, Azure, ...) as parameters and provisions those instances on the chosen platform\n  * Write a script to parse logs and extract some specific information you are interested in\n  * Write a script that extracts information from some website using web scraping libraries\n\n### Operating Systems\n\nDeep dive into operation system internals. Linux should be your first choice.\nHere is a list of resources that will help you learn Linux.<br>\nAlso, don't waste a second on picking the right distribution. For learning OS concepts and internals, any distribution would work fine.\n\nName | Comments\n:------|:------:\n[Linux Journey](https://linuxjourney.com) | Written guides + exercises + quiz\n[Techmint Linux](https://www.tecmint.com/free-online-linux-learning-guide-for-beginners) | Written articles/lessons\n[Linux Survival](https://linuxsurvival.com/linux-tutorial-introduction) | Interactive guide\n\nOnce you are familiar with the commands and feel comfortable enough to keep deep diving, learn some OS concepts:\n\n  * Memory Management\n  * Processes\n  * Kernel\n  * Virtualization\n  * Storage\n\n#### Linux Master Application\n\nA completely free application for testing your knowledge on Linux\n\n<a href=\"https://play.google.com/store/apps/details?id=com.codingshell.linuxmaster\"><img src=\"images/linux_master.jpeg\"/></a>\n\n### System Design\n\nSystem Design can be a complex topic. For that reason, I've created a separate repository for you to learn it properly.\n\n<p align=\"center\"><a href=\"https://github.com/bregman-arie/system-design-notebook\"><img src=\"images/system_design_notebook.png\"/></a></p>\n\n### Tooling, Apps & Services\n\nName | Description\n:------|:------:\n[Periodic table of DevOps tools](https://xebialabs.com/periodic-table-of-devops-tools) | Table of DevOps tools per category\n[DevOps tool chest](https://xebialabs.com/the-ultimate-devops-tool-chest/#tool-chest-type) | List of DevOps tools\n[DevOps Bookmarks](http://www.devopsbookmarks.com) | NSIA\n[CNCF Cloud Native Interactive Landscape](https://landscape.cncf.io) | CNCF's recommended path through the cloud native landscape\n[DevOpsBookmarks](https://www.devopsbookmarks.org) | To discover tools in the devops landscape.\n\n## Interactive practice and learning sites\n\nLearning by doing proved to be a very effective way to learn new things. If you are at a point where you don't feel comfortable enough to do it by yourself and you look for some guidance, the following sites might assist you\n\nName | Description\n:------|:------:\n[Kodekloud Engineer](https://www.kodekloud-engineer.com/) | perform real tasks by working in a fictional organization\n[QWIKLBAS](https://www.qwiklabs.com) | AWS, GCP, ... lab practice\n\n## Community\n\nAnother great way to learn is to learn from the experience of others. For that, the are some excellent DevOps communities.\n\nName | Comments\n:------|:------:\n[Reddit DevOps](https://www.reddit.com/r/devops) | Reddit DevOps Community \n[Linkedin DevOps](https://www.linkedin.com/groups/2825397) | Linkedin DevOps Community\n[DevOps Bits](https://www.facebook.com/groups/538897960007080) | Facebook DevOps group. Disclosure: I started it :)\n\n## Conferences\n\nName | Comments\n:------|:------:\n[DevOpsDays](https://devopsdays.org) | DevOps Conferences around the world\n[Velocity](https://conferences.oreilly.com/velocity) | Oracle's DevOps conference\n\n## Books\n\nRead book! Practice is important and the best way to learn practical things but reading about cultures, methods, concepts, ... has its own important role in your path towards mastering DevOps and SRE.\n\nName | Comments\n:------|:------:\n[The Phoenix Project](https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/1942788290) | DevOps explained through a story of a fictional company on the brink of a failure\n[The DevOps Handbook](https://www.amazon.com/dp/1942788002) |\n[Google SRE books](https://landing.google.com/sre/books) | Very detailed books on the theory and also practice of SRE\n[Essential Infrastructure as Code](https://www.manning.com/books/essential-infrastructure-as-code) | VBook about automating software infrastructure by capturing desired configurations as a set of scripts.\n\n## Blogs & News\n\nMy top picks:\n\nName | Description\n:------|:------:\n[Codefresh Blog](https://codefresh.io/blog) | Probably best GitOps content out there\n\nOther DevOps & Software Engineering related blogs\n\nName | Description\n:------|:------:\n[Spacelift Blog](https://spacelift.io/blog) | Excellent tech blog on Kubernetes, Ansible, Terraform and Engineering in general\n[opensource.com](https://opensource.com) | open source related articles including DevOps\n[CooperPress](https://cooperpress.com/publications) | various newsletters on different topics\n[afreshcup](https://afreshcup.com) | \"covering Ruby on Rails and whatever else I find interesting in the universe of software\"\n[thisweekindevops.com](https://thisweekindevops.com) | \"DevOps news without the hype\"\n[Uber Engineering](https://eng.uber.com/) | Not only DevOps\n[Azure DevOps Blog](https://devblogs.microsoft.com/devops) | Microsoft DevOps Blog\n[DevOps Dudes](https://medium.com/devops-dudes) | \"Shortening the system development life cycle\"\n[Netflix Tech Blog](https://netflixtechblog.com/tagged/devops) | Not Only DevOps\n[CloudBees Blog](https://www.cloudbees.com/blog) |\n[Red Hat Blog](https://www.redhat.com/en/blog) | Not only DevOps\n[Atlassian DevOps Blog](https://www.atlassian.com/blog/devops) |\n[Facebook Engineering Blog](https://engineering.fb.com) | Not only DevOps\n[DoorDash Engineering Blog](https://doordash.engineering) | Not only DevOps\n[Palark tech blog](https://blog.palark.com/) | Tech blog dedicated to DevOps & SRE\n\n## Articles\n\nName | Description\n:------|:------:\n[Infrastructure as Code, Part One](https://crate.io/a/infrastructure-as-code-part-one) |\n[DevOps Best Practices](https://gist.github.com/jpswade/4135841363e72ece8086146bd7bb5d91) | What is DevOps + list of resources\n[Infrastructure as Code: Examples, Best Practices & Tools](https://spacelift.io/blog/infrastructure-as-code) |\n\n## DevOps Snippets\n\nName | Description\n:------|:------:\n[DevOpsnipp](https://www.devopsnipp.com/) | Sharing DevOps Snippets\n[Gist](https://gist.github.com) | Not DevOps specific\n\n## Other Great Projects\n\nName | Comments\n:------|:------:\n[DevOps Wiki](https://github.com/Leo-G/DevopsWiki) | \"A wiki of Devops Tools, Tutorials and Scripts\"\n[cheat-sheet-pdf](https://github.com/sk3pp3r/cheat-sheet-pdf) | \"Cheat-Sheet Collection for DevOps, Engineers, IT and more\"\n[System Design Primer](https://github.com/donnemartin/system-design-primer) | \"Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.\"\n[awesome-sre](https://github.com/dastergon/awesome-sre) | A curated list of Site Reliability and Production Engineering resources\n\n## Interview Questions\n\nName | Comments\n:------|:------:\n[System Design Primer](https://github.com/donnemartin/system-design-primer) | \"Learn how to design large-scale systems. Prep for the system design interview. Includes Anki flashcards.\"\nkdn251 | [Interviews](https://github.com/kdn251/interviews) | \n[DevOps Exercises](https://github.com/bregman-arie/devops-exercises) | By topic, DevOps related questions and exercises\n\n## Misc\n\nName | Description\n:------|:------:\n[CheatSheets](https://cheatsheet.dennyzhang.com) | CheatSeets for Kubernetes, Jenkins, YAML, Groovy, etc.\n[Codelabs](https://codelabs.transcend.io) | \"Codelabs provide a guided, tutorial, hands-on coding experience\"\n\n## Design & Architecture\n\nName | Description\n:------|:------:\n[The 12 Factor App](https://12factor.net) | must-read in my opinion \n[A distributed Systems Reading List](https://dancres.github.io/Pages/?fbclid=IwAR2gapLU03iaxH4NDotc0LD_GiXOkyByfMT3iwufofUOEfCKEVwmezE3vCw)\n\n## DevOps Tooling\n\n- [ ] CI/CD\n  - [ ] [Jenkins](https://www.jenkins.io/) (Managed & on-premise, Containers supported, Plugins)\n  - [ ] [GitHub Actions](https://docs.github.com/en/actions) (Hosted service)\n  - [ ] [Travis](https://travis-ci.org/) (Hosted service)\n  - [ ] [CircleCI](https://circleci.com/) (Hosted service)\n  - [ ] [GitLab CI](https://docs.gitlab.com/ee/ci/) (Hosted service and on-premise)\n  - [ ] [Bamboo](https://www.atlassian.com/software/bamboo) (Managed service and on-premise)\n  - [ ] [BuildKite](https://buildkite.com) (Hosted)\n  - [ ] [Zuul](https://zuul-ci.org/docs/zuul) (Hosted and on-premise)\n  - [ ] [Drone](https://www.drone.io)\n  - [ ] [Tekton](https://cloud.google.com/tekton)\n  - [ ] [Argo CD](https://argoproj.github.io/argo-cd)\n  - [ ] [werf](https://werf.io/)\n\nNote: \"Provisioning\" tools can be used to perform configuration management to some extent. The same applies for configuration management tools, which can be used for pprovisioning.\n\n- [ ] Provisioning\n  - [ ] Terraform (Open Source)\n  - [ ] CloudFormation (Available only on AWS)\n  - [ ] Pulumi (Open Source)\n\n- [ ] Configuration Management (all Open Source)\n  - [ ] Ansible (Agentless, Mutable Infrastructure, Procedural)\n  - [ ] Puppet (Agent, Mutable Infrastructure, Procedural)\n  - [ ] Chef (Mutable Infrastructure)\n  - [ ] Saltstack (Mutable Infrastructure)\n  - [ ] [Spacelift](https://spacelift.io/) (`$$$`)\n\n- [ ] Images\n  - [ ] [diskimage-builder](https://github.com/openstack/diskimage-builder)\n  - [ ] [Packer](https://www.packer.io)\n\n- [ ] Cloud\n  - [ ] AWS\n  - [ ] Azure\n  - [ ] GCP\n  - [ ] OpenStack\n \n- [ ] Monitoring\n  - [ ] [AppDynamics](https://www.appdynamics.com/product/infrastructure-monitoring)\n  - [ ] [Datadog](https://www.datadoghq.com/)\n  - [ ] [Dynatrace](https://www.dynatrace.com/platform/infrastructure-monitoring/)\n  - [ ] [Monit](https://mmonit.com/monit/)\n  - [ ] [Nagios](https://www.nagios.org/)\n  - [ ] [Centreon](https://www.centreon.com/centreon-editions/centreon-open-source/)\n  - [ ] [New Relic](https://newrelic.com/)\n  - [ ] [Prometheus](https://prometheus.io)\n  - [ ] [Sensu](https://sensu.io/)\n  - [ ] [SolarWinds](https://www.solarwinds.com/server-application-monitor)\n  - [ ] [Statuscake](https://www.statuscake.com) - Website Uptime & Performance\n  - [ ] [VictoriaMetrics](https://victoriametrics.com/) - a fast, cost-effective and scalable monitoring solution and time series database.\n  - [ ] [Zabbix](https://www.zabbix.com/)\n\n- [ ] Collector of Metrics by agent\n  - [ ] [Telegraf](https://www.influxdata.com/time-series-platform/telegraf/)\n\n Monitoring\n\n- [ ] Log Management\n  - [ ] Logstash\n  - [ ] Graylog\n  - [ ] Papertrail\n  - [ ] Fluentd\n  - [ ] [Splunk](https://www.splunk.com)\n  - [ ] [SumoLogic](https://www.sumologic.com)\n\n- [ ] Log aggregation system\n  - [ ] [Loki](https://grafana.com/oss/loki/)\n\n- [ ] Dashboards\n  - [ ] Grafana\n  - [ ] Kibana (Elastic)\n  - [ ] Report Portal (Mostly for Test Automation)\n\n- [ ] Security\n  - [ ] [Vault](https://www.vaultproject.io) - \"Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets...\"\n  - [ ] [Open Policy Agent](https://www.openpolicyagent.org) - \"Flexible, fine-grained control for administrators across the stack\"\n\n- [ ] Code review \n  - [ ] Gerrit - Code Review system\n  - [ ] PullPanda - A collection of Tools such as reminder & analytics\n  - [ ] Review Board\n\n- [ ] Issue Tracking\n  - [ ] Jira\n  - [ ] Bugzilla\n\n- [ ] Code coverage\n  - [ ] Cobertura\n  - [ ] Clover\n  - [ ] JaCoCo\n\n- [ ] Data Processors\n  - [ ] Spark\n  - [ ] Apache Hadoop\n  - [ ] Apache Airflow\n  - [ ] Presto\n\n- [ ] Analytics Engines\n  - [ ] Druid\n  - [ ] Dremio\n  - [ ] Snowflake\n\n- [ ] Operations Data\n  - [ ] Salesforce\n  - [ ] Zuora\n\n- [ ] Containers Orchestration\n  - [ ] Kubernetes\n  - [ ] OpenShift\n  - [ ] Nomad\n\n- [ ] Alerting\n  - [ ] PagerDuty\n  - [ ] ServiceNow\n  - [ ] Slack (Mostly used for IM)\n\n- [ ] Metric Storage\n  - [ ] InfluxDB\n  - [ ] Splunk\n  - [ ] Cloud Storage services and Databases\n\n## More Infra & DevOps projects\n\n<p align=\"center\"><a href=\"https://github.com/bregman-arie/sre-checklist\"><img width=\"500px\" src=\"images/sre_checklist.png\"/></a></p>\n<p align=\"center\"><a href=\"https://github.com/bregman-arie/howtheydevops\"><img src=\"images/how_they_devops.png\"/></a></p>\n<p align=\"center\"><a href=\"https://github.com/bregman-arie/devops-exercises\"><img src=\"images/devops_exercises.png\"/></a></p>\n<p align=\"center\"><a href=\"https://github.com/bregman-arie/infraverse\"><img src=\"images/infraverse.png\"/></a></p>\n\n## Credits\n\nImages and icons attribution can be found [here](credits.md)\n\n## License\n\n[![License: CC BY-NC-ND 3.0](https://img.shields.io/badge/License-CC%20BY--NC--ND%203.0-lightgrey.svg)](https://creativecommons.org/licenses/by-nc-nd/3.0/)\n"
  },
  {
    "path": "credits.md",
    "content": "The image used for \"Dataops\" is taken from [Flaticon.com](Flaticon.com)\n"
  },
  {
    "path": "resources/Grafana/deployment.yml",
    "content": "apiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: grafana\n  namespace: graf   # Create and write the name of your own namespace\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: grafana\n  template:\n    metadata:\n      name: grafana\n      labels:\n        app: grafana\n    spec:\n      containers:\n      - name: grafana\n        image: grafana/grafana:latest\n        ports:\n        - name: grafana\n          containerPort: 3000\n       "
  },
  {
    "path": "resources/Grafana/grafana.md",
    "content": "## Grafana\n\n### Random\n\n* Visualize latency for API endpoints: \n\n```\n1. Use https://github.com/prometheus/blackbox_exporter to point API endpoints\n2. Scrape exporter with Prometheus\n3. Graph from Grafana\n```\n## Setting up Grafana for Prometheus on Kuberentes\n* Grafana is a Visualisation tool that we most commonly use with prometheus Now many of you might have struggled while setting up grafana and Prometheus. To Solve this Problem I have created this deployment file that you can Integrate with Prometheus. If you Don't know how to Setup Prometheus for monitoring in Kuberentes. You can visit prometheus section in this repository, There you will find all the files needed to setup prometheus.\n\nTo access Prometheus you can click on this. [Prometheus](../prometheus.md)\n\n```bash\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  name: grafana\n  namespace: graf  #<-- Here you can add your own namespace.\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: grafana\n  template:\n    metadata:\n      name: grafana\n      labels:\n        app: grafana\n    spec:\n      containers:\n      - name: grafana\n        image: grafana/grafana:latest\n        ports:\n        - name: grafana\n          containerPort: 3000\n```       \nI have included this files, You can access them here [Deployment.yml](./deployment.yml) \n\n\nNote:-\n After this deployment file, You will need to setup dashboard on your own. \nAlso since this is running inside container, so to access grafana dashboard you will\neither have to port-forward or else you can make your own service file of type NodePort.\nfor a example, I will add a service file here that you can reuse with little changes.\nMake sure you create your own namespace and edit your own namespace in this file or create a namespace with this name.\n\n```bash\napiVersion: v1\nkind: Service\nmetadata:\n  name: grafana\n  namespace: graf #--> write your own namespace make sure all files are in same namespace.\nspec:\n  selector: \n    app: grafana\n  type: NodePort  \n  ports:\n    - port: 3000\n      targetPort: 3000\n      nodePort: 32000\n```\nI have included this files, You can access them here [Service.yml](./service.yml) \n\nNote: Make sure you have the basic knowledge of kuberentes. By default, the range of the service NodePorts is 30000-32768. So it means you can choose any range of your own.\nWhen you will run this file, there is no need to port-forward and you can access this on port \ni.e \n``` <your-ip>:nodePort\n```\nThis was how you can make use of grafana."
  },
  {
    "path": "resources/Grafana/service.yml",
    "content": "apiVersion: v1\nkind: Service\nmetadata:\n  name: grafana\n  namespace: graf #create and Write your own namespace \nspec:\n  selector: \n    app: grafana\n  type: NodePort  \n  ports:\n    - port: 3000\n      targetPort: 3000\n      nodePort: 32000"
  },
  {
    "path": "resources/ansible.md",
    "content": "## Ansible\n\n### Learn Ansible\n\nName | Comments\n:------ |:--------:\n[Ansible 101 - Jeff Geerling](https://www.youtube.com/watch?v=goclfp6a2IQ&list=RDCMUCR-DXc1voovS8nhAvccRZhg&index=1) | Comprehensive practical way to learn Ansible\n[What is Ansible? - TechWorld with Nana](https://www.youtube.com/watch?v=1id6ERvfozo) | High-level short overview of Ansible\n[Learning Ansible basics - Red Hat](https://www.redhat.com/en/topics/automation/learning-ansible-tutorial) | Red Hat's guide on how to learn Ansible basics + links to the content itself\n[Introduction to Ansible - 2021](https://medium.com/@bagusays/introduction-to-ansible-82f2bc12cd87) |\n\n### Articles\n\nName | Comments\n:------ |:--------:\n[Writing reliable Ansible Playbooks - 2021](https://dev.to/xlab_si/writing-reliable-ansible-playbooks-295i) | \n[A CI/CD Pipeline Project for a Trunk-Based Development Strategy in a Kubernetes Environment](https://medium.com/swlh/a-ci-cd-pipeline-project-for-a-trunk-based-development-strategy-in-a-kubernetes-environment-c4ffea9700fe) |\n\n### Books\n\nName | Comments\n:------ |:--------:\n[Ansible for DevOps](https://www.amazon.com/Ansible-DevOps-Server-configuration-management/dp/098639341X) | \n[Ansible: From Beginner to Pro](https://www.amazon.com/Ansible-Beginner-Pro-Michael-Heap/dp/1484216601) |\n[Ansible: Up and Running](https://www.amazon.com/Ansible-Automating-Configuration-Management-Deployment/dp/1491979801) |\n\n\n### Cheat Sheet\n\n* Check if list has elements\n\n```\nwhen: my_list | length > 0\n```\n\n* Update all packages\n\n```\n- name: Update system packages\n  package:\n    state: latest\n    name: \"*\"\n```\n\n* Update packages informations and display packages informations\n\n```\n- name: Update packages informations\n  package_facts:\n    manager: \"auto\"\n\n- name: Display all installed packages informations\n  debug:\n    msg: \"{{ ansible_facts.packages }}\"\n\n- name: Display all Chromium package informations\n  debug:\n    msg: \"{{ ansible_facts.packages['chromium'] }}\"\n  when: \"'chromium' in ansible_facts.packages\"\n```\n"
  },
  {
    "path": "resources/argocd.md",
    "content": "# ArgoCD\n\n## Commands\n\n* login: `argocd login <ADDRESS>`\n* version: `argocd version`\n\n### Apps\n\n* create an application:\n\n```\nargocd app create {APP NAME} \\\n--project {PROJECT} \\\n--repo {GIT REPO} \\--path {APP FOLDER} \\\n--dest-namespace {NAMESPACE} \\\n--dest-server {SERVER URL}\n```\n* app list: `argocd app list`\n* app detailed info: `argocd app get <APP NAME>`\n\n* sync app: `argocd app sync <APP NAME>`\n* app history: `argocd app history <APP NAME>`\n* delete app: `argocd app delete <APP NAME>`\n\n* Add cluster: `argocd cluster add CLUSTER`\n* List clusters: `argocd cluster list`\n* Remove cluster: `argocd cluster rm CLUSTER\n"
  },
  {
    "path": "resources/aws.md",
    "content": "# AWS\n\n## Infrastructure\n\nName | Comments\n:------|:------:\n[infrastructure.aws](https://infrastructure.aws) | AWS Infrastructure Overview\n[Regions and Availability Zones](https://aws.amazon.com/about-aws/global-infrastructure/regional-product-services) | See services list per region and AZ\n\n## Tools and Projects\n\nName | Comments\n:------|:------:\n[Cloud Mapper](https://github.com/duo-labs/cloudmapper) | \"analyze your Amazon Web Services (AWS) environments\"\"\n[EC2 Instances Info](https://www.ec2instances.info) | \"Easy Amazon EC2 Instance Comparison\"\n[amazon-redshift-utils](https://github.com/awslabs/amazon-redshift-utils) | \"Contains utilities, scripts and view which are useful in a Redshift environment\"\n[Cloudcraft](https://www.cloudcraft.co) | Automated AWS diagram mapping of your cloud resources.\n[nOps](https://www.nops.io/devops/) | \"for fast-moving DevOps teams helping them ... monitor, analyze, and manage provisioned cloud workloads.\", **Not Free.**\n[EC2Instances.info](https://instances.vantage.sh) | \"Easy Amazon EC2 Instance Comparison\"\n\n## Blog Posts & Articles\n\nName | Comments\n:------|:------:\n[The Hitchhiker’s Guide to Redshift — part 1: With great power come performance issues](https://blog.atomdata.io/the-hitchhikers-guide-to-redshift-part-1-with-great-power-comes-performance-issues-748d293e0b18) | \n\n## IaC\n\nName | Comments\n:------|:------:\n[terraformer](https://github.com/GoogleCloudPlatform/terraformer) | \"A CLI tool that generates tf/json and tfstate files based on existing infrastructure (reverse Terraform).\"\n[terraforming](https://github.com/dtan4/terraforming) | \"Export existing AWS resources to Terraform style (tf, tfstate)\"\n\n## Videos\n\nName | Comments\n:------|:------:\n[Amazon Redshift Tutorial](https://www.youtube.com/watch?v=fc5WPKnbam8) | Edureka, 2018\n\n## Certificates\n\nName | Comments\n:------|:------:\n[DevOpsBit Cloud Practitioner Summary](http://devopsbit.com/aws-cloud-practitioner) | 2020\n\n## Interview Questions\n\nName | Description\n:------|:------:\n[devops-interview-questions repository](https://github.com/bregman-arie/devops-interview-questions/#aws) | Over 100 questions\n[AWS interview questions and answers](https://www.besanttechnologies.com/aws-interview-questions-and-answers) | 250 AWS interview questions\n\n### Best Practices\n\n#### IAM\n\n* Set up MFA\n* Delete root account access keys\n* Create IAM users instead of using root for daily management\n\n### Cheat Sheet\n\n### IAM\n\n* List users: `aws iam list-users`\n\n### EC2\n\n* List instances: `aws ec2 describe-instances`\n* Remove security group rules:\n\n```\naws ec2 revoke-security-group-ingress \\\n    --group-name someHTTPSecurityGroup\n    --protocol tcp \\\n    --port 80 \\\n    --cidr 0.0.0.0/0`\n```\n\n* Add security group rules:\n\n```\naws ec2 authorize-security-group-ingress \\\n    --group-name someHTTPSecurityGroup\n    --protocol tcp \\\n    --port 80 \\\n    --cidr 0.0.0.0/0\n```\n\n### RDS\n\n* Encryption in PostgreSQL: `rds.force_ssl=1 (parameter groups)`\n* Encryption in MySQL: `GRANT USAGE ON *.* TO 'mysqluser'@'%' REQUIRE SSL;`\n\n"
  },
  {
    "path": "resources/azure.md",
    "content": "# Azure\n\n## IaC\n\nName | Comments\n:------|:------:\n[terraformer](https://github.com/GoogleCloudPlatform/terraformer) | \"A CLI tool that generates tf/json and tfstate files based on existing infrastructure (reverse Terraform).\"\n"
  },
  {
    "path": "resources/bash.md",
    "content": "# Bash\n\n## Resources\n\nName | Description\n:------ |:--------:\n[Bash Wiki Hackers](http://wiki.bash-hackers.org/start) | \"hold documentation of any kind about GNU Bash\"\n[Bash Reference Manual](https://tiswww.case.edu/php/chet/bash/bashref.html) | everything there is to know about bash\n[Hacker Rank Linux Shell](https://www.hackerrank.com/domains/shell) | Bash Challenges\n[ExplainShell](https://explainshell.com) | Provides explanation on commands\n\n## Best Practices\n\nSome best practices for you to learn and use in your scripts\n\n### 1. Exit script when a certain command fails\n\nWhen a script written in Bash fails to run a certain command it will keep\nrunning and will execute all other commands mentioned after the command which failed.\nMost of the time we would actually want the opposite to happen.\nIn order to make Bash exit when a specific command fails, use 'set -e' in your script.\n\n```\n#!/usr/bin/env bash\nset -e\n\nls /usr/lib/cowabunga\ndate # It will never run this (unless you have /usr/lib/cowabunga on your system for some reason...)\n```\n\n### 2. Allow some commands to fail\n\nNot every failure is a good reason to exit a script.\nYou will probably want to allows some commands to fail and keep running\nTo achieve that simple append '|| true' to your command\n\n```\n#!/usr/bin/env bash\nset -e\n\ndance || true\nls /tmp #  This command will run even if `dance` failed\n```\n\n### 3. Get used to using curly braces {}\n\nIt's a good habit to use curly braces when\nreferring to variables. Let's see few examples.\n\n`echo \\${foo}bar` if not used you would expand \\$foobar\n`echo ${10}` to expand any positional parameter beyond you'll have to use {}\n`echo ${A[0]}` to expand array elements\n\nThere are additional advantages covered in next sections.\n\n### 4. Undeclared Variables\n\nThe following one-line \"script\" will work just fine\n\n```\nls ${x}\n```\n\nIt's reasonable to think that whoever wrote this amazing script\nwanted x to be defined somewhere and either forgot to do it or the\nuser didn't follow the instructions.\nIn order to avoid strange failures with your script you would\nwant to execute 'set -u' in order to make sure variable are not\nundefined\n\n```\n#!/usr/bin/env bash\nset -u\n\nls ${x}\n```\n\nNow the script will return error trying to use undefined x variable.\n\n### 5. Use ':-' for allowing undeclared variables\n\nSometimes you would want to allow variables to be undeclared.\nIn that case use this syntax to do so: '\\${X:-}'.\nYou can also set default value if variable is undeclared\nlike this: '\\${X:-mario}'\n\n```\n#!/usr/bin/env bash\n\necho \"Nothing more than emptiness ${X:-}\"\necho \"But here, there is ${X:-something}\"\n\nX='anymore'\n\necho \"It doesn't feel empty ${X:-}\"\n```\n\n## Things you should never do\n\n### 1. Don't use backticks (``) for running commands\n\nbackticks were (and still are) used for executing a command and using its\noutput as a command as well. For some reason people are still using backticks\nin scripts although they are deprecated long time ago and were replaced\nby \\$(...) which was introduced by ksh for the first time and now part of\nPOSIX standardization.\n\n```\nx=`date` -> No!\nx=$(date) -> Yes :D\n```\n\n## Cheat Sheet\n\n#### Variables\n\n* Set variable with default value (string): `x=${x:-'some_default'}`\n* Set variable with default value (variable): `y=${y:-$z}`\n* return value of a program: `$?`\n* Check if variable is empty: `if [ -z \"$var\" ]; then`\n* Variable length: `${#string}`\n\n#### Arguments\n\n* Read input: `read -p \"enter a number: \" num`\n* Number of arguments: `$#`\n* Check if an argument was passed\n\n```\nif [ \"$#\" -lt 1 ]; then\n    echo \"Illegal number of parameters\"\nfi\n```\n\n* Check if two arguments were passed\n\n```\nif [ \"$#\" -ne 2 ]; then\n  echo 'Please pass two arguments'\n  exit 1\nfi\n```\n\n* Check if two arguments were passed and both are numbers\n\n```\nre='^[0-9]+$'\nif ! [[ $1 =~ $re && $2 =~ $re ]]; then\n    echo \"Oh no...I need two numbers\"\n    exit 2\nfi\n```\n\n* Check if arguments' strings length is equal\n\n```\nif [ ${#1} -ne ${#2} ]; then\n  echo 'Not equal`\n  exit 1\nfi\n```\n\n#### Files\n\n* check if file exists\n\n```\nFILE=/some/file\nif [ -f \"$FILE\" ]; then\n    echo \"$FILE exists\"\nfi\n```\n\n* check if directory exists\n\n```\nDIR=/some/dir\nif [ -d \"$DIR\"]; then\n    echo \"$DIR\" exists\"\nfi\n```\n\n#### Loops\n\n* Iterate over a string: `for i in $(seq 1 ${#1}); do`\n\n#### Arithmetic Operations\n\n* print the sum of two numbers: `echo $((20+17))`\n* Check factor: `if [ $(($1 % 3)) -eq 0 ]; then`\n\n#### Extract Patterns\n\n* Extract date with sed: `echo $line | sed 's/.*\\[//g;s/].*//g;s/:.*//g'`\n* Extract first field (space separator) with awk: `echo $line | awk '{print $1}'`\n\n#### Dictionary / Hash Table\n\n* Define a dictionary: `declare -A somedict`\n* Print one value based on given key: `echo ${somedict[some_key]}`\n* Print all the keys of a dictionary: `echo ${!somedict[*]}`\n* Check if key exists: `if [[ -v some_dict[$day] ]]; then`\n\n* Update dict based values and generate top 10:\n\n```\nfunction update_dict() {\n    declare -A some_dict\n    while read line; do\n        day=$line\n        if [[ -v some_dict[$day] ]]; then\n            some_dict[$day]=$((some_dict[$day]+1))\n        else\n            some_dict[$day]=1\n        fi\n    done < $FILE\n\n    for day in ${!some_dict[@]}; do echo ${some_dict[$day]} $day; done | sort -rn | head -10\n}\n```\n\n#### Common algorithms\n\n* Hamming distance\n\n```\ndistance=0\nfor i in $(seq 1 ${#1}); do\n  if [ ${1:$i-1:1} != ${2:$i-1:1} ]; then\n    distance=$((distance+1))\n  fi\ndone\necho $distance\n```\n\n#### Text Manipulation\n\n* Take the first letter of every word in a line: `echo $line | sed 's/\\(.\\)[^ ]* */\\1/g'`\n"
  },
  {
    "path": "resources/chaos_engineering.md",
    "content": "# Chaos Engineering\n\n## Platforms and Services\n\n* https://www.gremlin.com\n\n## Resources\n\n* Diagram for tracking Chaos Engineering state: http://coggle.it/diagram/WiKceGDAwgABrmyv/0a2d4968c94723e48e1256e67df51d0f4217027143924b23517832f53c536e62\n"
  },
  {
    "path": "resources/cicd.md",
    "content": "## CI/CD\n\n### Best Practices\n\n* Commit and test often.\n* Testing/Staging environment should be a clone of production environment.\n* Clean up your environments (e.g. your CI/CD pipelines may create a lot of resources. They should also take care of cleaning up everything they create)\n* The CI/CD pipelines should provide the same results when executed locally or remotely\n* Treat CI/CD as another application in your organization. Not as a glue code.\n* On demand environments instead of pre-allocated resources for CI/CD purposes\n* Stages/Steps/Tasks of pipelines should be shared between applications or microservices (don't re-invent common tasks like \"cloning a project\")\n* The CI/CD pipelines should have short execution time in order to provide a short feedback loop\n"
  },
  {
    "path": "resources/coding.md",
    "content": "## Videos\n\nName | Comments\n:------|:------:\n[SOLID design principles](https://www.youtube.com/watch?v=rtmFCcjEgEw) | by Katerina Trajchevska, 2019\n"
  },
  {
    "path": "resources/containers.md",
    "content": "## Containers\n\n### Articles\n\nName | Comments\n:------ |:--------:\n[Docker CheatSheet](https://cheatsheet.dennyzhang.com/cheatsheet-docker-a4) |\n[Everything you need to know about containers](https://medium.com/faun/everything-you-need-to-know-about-containers-7655badb4307) |\n[A container networking overview](https://jvns.ca/blog/2016/12/22/container-networking) |\n[My Docker Cheat Sheet](https://medium.com/statuscode/dockercheatsheet-9730ce03630d) |\n[Docker Networking Deep Dive](http://100daysofdevops.com/21-days-of-docker-day-19-docker-networking-deep-dive/?fbclid=IwAR19KJWwhZjulbn7JNbBYLFxKFf-x0v25TSc-_bOJ6YieUND4A6UZcBSUhA) |\n\n### Projects\n\nName | Comments\n:------ |:--------:\n[container-diff](https://github.com/GoogleContainerTools/container-diff) | \"container-diff is a tool for analyzing and comparing container images\"\n[awesome-docker](https://github.com/veggiemonk/awesome-docker) | \n\n### Books\n\n\nName | Comments\n:------ |:--------:\nDocker Deep Dive |\n\n### Tools\n\nName | Description\n:------|:------:\n[dive](https://github.com/wagoodman/dive) | \"A tool for exploring a docker image, layer content, ...\"\n[trivy](https://github.com/aquasecurity/trivy) | \"A Simple and Comprehensive Vulnerability Scanner for Containers and other Artifacts, Suitable for CI.\"\n\n### Production Best Practices\n\n* Secured communication between daemon and clients using TLS\n\n### Best Practices\n\n#### Dockerfile Best Practcies\n\n* Include only the packages you are going to use. Nothing else.\n* Specify a tag in FROM instruction. Not using a tag means you'll always pull the latest, which changes over time and might result in unexpected result.\n* Do not use environment variables to share secrets\n* Use images from official repositories\n* Keep images small! - you want them only to include what is required for the application to run successfully. Nothing else.\n* If are using the apt package manager, you might use 'no-install-recommends' with `apt-get install` to install only main dependencies (instead of suggested, recommended packages)\n\n#### Security Best Practcies\n\n  * Install only the necessary packages in the container\n  * Don't run containers as root when possible\n  * Don't mount the Docker daemon unix socket into any of the containers\n  * Set volumes and container's filesystem to read only\n  * DO NOT run containers with `--privilged` flag\n\n### Cheatsheet\n\n* Stop and remove all containers: `podman container stop $(docker container ls -aq)`\n* Run container with bash shell: `podman run -ti ubuntu:latest /bin/bash`\n* Check how many containers are running: `podman info`\n* Cleanup everything: `podman system prune -a -f`\n\n#### Images\n\n* List images: `podman image ls`\n* Pull latest ubuntu image: `podman image pull ubuntu:latest`\n* Remove image from the host: `podman rmi IMAGE`\n* Remove all images: `podman rmi -a` or the more extreme version of it: `podman rmi $(podman images -qa) -f`\n* Login to Red Hat registry: `podman login registry.redhat.io`\n* List all available tags for a given image using the CLI: `podman search --list-tags IMAGE_NAME`\n* List all available tags for a given image using an API request: `curl -Ls https://SERVER/v2/.../tags/list | python -m json.tool`\n* Create an archive from the image: `podman save -o some_image.tar IMAGE`\n* Load image from the archive: `podman load -i some_image.tar`\n* List added/modified/deleted files on container's image filesystem: `podman diff IMAGE_NAME`\n* List of mounted files for a running container: `podman inspect -f \"{{range .Mounts}}{{println .Destination}}{{end}}\" CONTAINER_NAME/ID`\n* Commit changes to another image: `podman commit EXISTING_IMAGE NEW_IMAGE`\n\n#### Registries\n\n* List all repositories in a registry: `curl -Ls https://SERVER/v2/_catalog?n=`\n* List all repositories in a registry and convert to JSON: `curl -Ls https://SERVER/v2/_catalog?n=3 > | python -m json.tool`\n\n#### Security\n\n* Secure communication between client and server:\n\n```\n# On CA node\nopenssl genrsa -aes256 -out ca-k.pem 4096\nopenssl req -new -x509 -days 730 -key ca-k.pem -sha256 -out ca.pem\nopenssl genrsa -out daemon-key.pem 4096\nopenssl req -subj \"/CN=daemon.host.address\" -sha256 -new -key daemon-key.pem -out daemon.csr\ncat << EOF >> file.conf\nsubjectAltName = DNS:daemon.host.address,IP:X.X.X.X\nextendedKeyUsage = serverAuth\nEOF\nopenssl x509 -req -days 730 -sha256 -in daemon.csr -CA ca.pem -CAkey ca-k.pem -CAcreateserial -out daemon-cert.pem -extfile file.conf\nopenssl genrsa -out client-key.pem 4096\nopenssl req -subj '/CN=client.address' -new -key client-key.pem -out client.csr\necho \"extendedKeyUsage = clientAuth\" > file.conf\nopenssl x509 -req -days 730 -sha256 -in client.csr -CA ca.pem -CAkey ca-k.pem -CAcreateserial -out client-cert.pem -extfile file.conf\nchmod 0400 ca-k.pem client-key.pem daemon-key.pem && chmod -v 0444 ca.pem client-cert.pem daemon-cert.pem\n# On daemon host put the keys ca.pem, daemon-cert.pem and daemon-key.pem in ~/.docker\n# On client host put the keys ca.pem, client-cert.pem and client-key.pem in ~/.docker\n```\n\n* Enable TLS\n\n```\n# Put it in /etc/docker/daemon.json\n{\n\"hosts\": [\"tcp://node3:2376\"],\n\"tls\": true,\n\"tlsverify\": true,\n\"tlscacert\": \"/home/ubuntu/.docker/ca.pem\",\n\"tlscert\": \"/home/ubuntu/.docker/cert.pem\",\n\"tlskey\": \"/home/ubuntu/.docker/key.pem\"\n}\n```\n\n#### Cleanup\n\n* Remove everything: `podman system reset`\n"
  },
  {
    "path": "resources/dataops.md",
    "content": "# DataOps\n\n## Videos\n\nYear | Video\n:------ |:------:\n2020 | [DataOps: Why is it so hard?](https://youtu.be/XZ7KhOPCiKU?t=956)\n"
  },
  {
    "path": "resources/docker.md",
    "content": "## Docker\n\n### Learn\n\nName | Comments\n:------ |:--------:\n[Play with Docker](https://labs.play-with-docker.com) | \"A simple, interactive and fun playground to learn Docker\"\n\n### Articles\n\nName | Comments\n:------ |:--------:\n[Docker CheatSheet](https://cheatsheet.dennyzhang.com/cheatsheet-docker-a4) |\n[Everything you need to know about containers](https://medium.com/faun/everything-you-need-to-know-about-containers-7655badb4307) |\n[A container networking overview](https://jvns.ca/blog/2016/12/22/container-networking) |\n[My Docker Cheat Sheet](https://medium.com/statuscode/dockercheatsheet-9730ce03630d) |\n[Docker Networking Deep Dive](http://100daysofdevops.com/21-days-of-docker-day-19-docker-networking-deep-dive/?fbclid=IwAR19KJWwhZjulbn7JNbBYLFxKFf-x0v25TSc-_bOJ6YieUND4A6UZcBSUhA) |\n\n### Projects\n\nName | Comments\n:------ |:--------:\n[awesome-docker](https://github.com/veggiemonk/awesome-docker) | \n\n### Cheatsheet\n\n* Stop and remove all containers: `docker container stop $(docker container ls -aq)`\n* Check how many containers are running: `docker info`\n* Check the docker images on your system: `docker images`\n* Cleanup everything: `docker system prune -a -f`\n\n#### Managing Containers\n\n* Launch a container and attach to it: `docker container run -it ubuntu:latest /bin/bash`\n* Run a command in the container: `docker exec -it <container ID/name> <commands>`\n* Attaching to running container: `docker container exec -it <name> bash`\n* List running containers: `docker container ls`\n* List all containers (including stopped): `docker container ls -a`\n* List containers (including stopped): `docker container ls -a`\n* Stop a container: `docker container stop <name>`\n* Remove a container: `docker container rm <name>`\n* Remove all containers: `docker container rm $(docker container ls -aq) -f`\n* Create, start, run command and destroy the container: `docker run --rm -it <image> <command>`\n\n#### Images\n\n* List images: `docker image ls`\n* Pull image: `docker image pull <name>:<tag>`\n* List only images tagged as \"latest\": `docker image ls --filter=reference=\"*:latest\"`\n* List dangling images: `docker image ls --filter dangling=true`\n* Which default registry is used: `docker info`\n* List supported image architectures: `docker manifest inspect golang | grep 'architecture\\|os'`\n* Search all repositories that contain the string 'yay': `docker search yay`\n* Search only for official repositories: `docker search <name> --filter \"is-official=true\"`\n* Build an image: `docker image build -t <name>:<tag> .`\n* List images with the following data: name, tag and size: `docker image ls --format \"{{.Repository}}: {{.Tag}}: {{.Size}}\"`\n* List images with their digests: `docker image ls --digests`\n* Specify architecture when building an image: `docker buildx build --platform linux/arm/v1985 -t some_image:arm-v1985 .`\n* Delete an image: `docker image rm <name or ID>:<tag>`\n* Remove all images: `docker image rm $(docker image ls -q) -f`\n* Tag an image: `docker tag <image name>:<tag> <repo>/<image>:<tag>`\n* Push an image: `docker push <repo>/<image>:<tag>`\n* Save a running container as an image: `docker commit -m \"some commit message\" -a \"author name/username\" <container ID/name> <username>/<image name>:<tag>`\n\n#### Logs\n\n* Print the last 100 lines from container's logs: `docker container logs --tail 100 <container name/ID>`\n* Follow container logs: `docker logs -ft <container ID/name>`\n\n#### Network\n\n* List networks: `docker network ls`\n\n#### Compose\n\n* Deploy compose app: `docker-compose up`\n* Stop and delete a running Compose app: `docker-compose down`\n* Restart a Compose app: `docker-compose restart`\n* List each container in the Compose app: `docker-compose ps`\n* Stop all of the container in Compose app: `docker-compose stop`\n* Delete a stopped Compose app: `docker-compose rm`\n\n### Common Failures\n\n#### Unable to start the Docker service\n\nError:\n\n` Failed to start Docker Application Container Engine.`\n\nFix:\n\n```\nsudo firewall-cmd --permanent --zone=docker --change-interface=docker0\nsudo firewall-cmd --reload\n```\n"
  },
  {
    "path": "resources/elk.md",
    "content": "## ELK articles\n\nName | Comments\n:------|:------:\n[logz.io - complete-guide-elk-stack](https://logz.io/learn/complete-guide-elk-stack) | name says it all\n\n## Logstash articles\n\nName | Comments\n:------|:------:\n[Logstash Grok Guide](https://logz.io/blog/logstash-grok) | Learn Grok, It's important to know\n\n## Data Sets & Patterns\n\nName | Comments\n:------|:------:\n[Kaggle](https://www.kaggle.com) | Free Data Sets\n[Logstash Patterns](https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns) | Logstash Patterns Code\n[Grok Patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns) | Logstash Patterns Code\n[RE used by Grok](https://github.com/kkos/oniguruma/blob/master/doc/RE) |\n\n## Elasticsearch Tools\n\nName | Comments\n:------|:------:\n[elasticsearch-dump](https://github.com/taskrabbit/elasticsearch-dump) | Import and export tools for elasticsearch\n\n## Logstash Tools\n\nName | Comments\n:------|:------:\n[GrokDebug](https://grokdebug.herokuapp.com) | Test Grok Patterns\n\n## Kibana Tools\n\nName | Comments\n:------|:------:\n[Rubban](https://github.com/sherifabdlnaby/rubban) | \"Kibana Automatic Index Pattern Discovery and Other Curating Tasks\"\n\n## Elsaticsearch Cheat Sheet\n\n* Test elasticsearch: `curl <es_host>:9200`\n* List of APIs: `curl <es_host>:9200/_cat`\n* List nodes: `curl <es_host>:9200/_cat/nodes`\n* List nodes with headers: `curl <es_host>:9200/_cat/nodes?v`\n* Check elasticsearch health (also good for checking number of nodes): `curl -X GET \"localhost:9200/_cat/health?v\"`\n* List indexes: `curl 'localhost:9200/_cat/indices?v'`\n* Create a new index called \"customers\" and add a document with the field name\n\n```\ncurl -X PUT \"localhost:9200/gamer/_doc/1?pretty\" -H 'Content-Type: application/json' -d'\n{\n  \"name\": \"John Doe\"\n}\n'\n```\n\n* Add documents from a JSON file: `curl -H \"Content-Type: application/json\" -XPOST \"localhost:9200/customer/_bulk?pretty&refresh\" --data-binary \"@customers.json\"`\n* Get the document with ID 1: `curl -X GET \"localhost:9200/gamer/_doc/1\"`\n* List all indices: `curl \"localhost:9200/_cat/indices?v\"`\n\n* Increase fields limit\n```\nPUT test_index/_settings\n{\n  \"index.mapping.total_fields.limit\": 2000\n}\n```\n\n* Delete index from console UI: `DELETE /<index_name>`\n* Delete all indexes from CLI: `curl -X DELETE 'http://<elasticsearch_address>:9200/_all` # Don't run this! it will delete also the Kibana index and you'll not have default space available\n\n* Create cluster\n\n```\n# On the master node\ndnf install -y elasticsearch\nsystemctl enable elasticsearch\nvi /etc/elasticsearch/elasticsearch.yml\n  set cluster.name\n  set node.name\n  set network.host to [_local_, _site_]\n  set discovery.seed_hosts to master private IP address (or addresses if you are using multiple nodes)\n  set cluster.inital_master_nodes to master node name(s)\n  set node.master to true\n  set node.data to false\n  set node.ingest to true\n  set node.ml to false\n# On second node (and any additional node)\n  set cluster.name\n  set node.name\n  set network.host to [_local_, _site_]\n  set discovery.seed_hosts to master private IP address (or addresses if you are using multiple nodes)\n  set cluster.inital_master_nodes to master node name(s)\n  set node.master to false\n  set node.master to true\n  set node.ingest to false\n  set node.ml to false\n# On all nodes\nsystemctl start elasticsearch\n```\n\n## Logstash Cheat Sheet\n\n* input - filebeat\n\n```\ninput {\n  beats {\n    port => 5044\n  }\n}\n```\n\n* Check if Logstash receiving logs from filebeat:\n\n```\ntcpdump -Xni eth0 port 5140\n```\n\n* Load Parquet files\n\n```\nUse https://github.com/Parquet/parquet-compatibility/blob/master/parquet-compat/src/test/java/parquet/compat/test/ConvertUtils.java\nSpecifically https://github.com/Parquet/parquet-compatibility/blob/master/parquet-compat/src/test/java/parquet/compat/test/ConvertUtils.java#L111\n\nOnce it's convreted to CSV, you process it with\n\ninput {\n    file {\n        path => \"/path/to/your/parquet/as/csv/file\"\n    }\n}\nfilter {\n    csv {\n        columns => [\"col1\", \"col2\"]\n    }\n}\n```\n\n## Kibana Cheat Sheet\n\n* Unlock Indexes (in case of forbbiden error)\n\n```\nPUT .kibana/_settings\n{\n\"index\": {\n\"blocks\": {\n\"read_only_allow_delete\": false\n}\n}\n}\n```\n\n* Unlock specific index\n\n```\nPUT your_index_name/_settings\n{\n\"index\": {\n\"blocks\": {\n\"read_only_allow_delete\": \"false\"\n}\n}\n}\n```\n"
  },
  {
    "path": "resources/gcp.md",
    "content": "# GCP\n\n## IaC\n\nName | Comments\n:------|:------:\n[terraformer](https://github.com/GoogleCloudPlatform/terraformer) | \"A CLI tool that generates tf/json and tfstate files based on existing infrastructure (reverse Terraform).\"\n[validkube](https://validkube.com/) | \"A tool that to verify and validate your yml file.\"\n[validiac](https://www.validiac.com/) | \"A tool that to verify and validate your terraform file.\"\n"
  },
  {
    "path": "resources/git.md",
    "content": "# Git\n\n## Learn Git\n\nName | Comments\n:------|:------:\n[git-scm](https://git-scm.com) | The best place (imho) to learn everything about Git (through reading)\n[Interactive Git Branching Learning](https://learngitbranching.js.org/) | Visual and interactive way to learn Git branching\n[Learn git concepts, not commands](https://dev.to/unseenwizzard/learn-git-concepts-not-commands-4gjc) | Article on Git concepts\n[Codeacademy Learn Git](https://www.codecademy.com/learn/learn-git) | Not Free \n[Git for Computer Scientists](https://eagain.net/articles/git-for-computer-scientists/) |\n[shortcutfoo](https://www.shortcutfoo.com/app/dojos/git) |  interactive learning \n\n## Best Practices\n\n* Use a descriptive commit message\n* Make each commit a logical unit\n* Incorporate others' changes frequently\n* Share your changes frequently\n* Coordinate with your co-workers\n* Don't commit generated files\n\n## CheatSheet\n\n* Clone a repository: `git clone https://github.com/bregman-arie/devops-resources.git`\n* Pull changes from remote repository: `git pull`\n* Pull changes without trying to merge the changes between the local branch and the remote one: `git pull --ff-only`\n\n### Branches\n* Switch to a branch called \"main\": `git checkout main`\n* Create (if doesn't exists) and switch to a branch called `devel`: `git checkout -b devel`\n* List branches: `git branch`\n* Update based on status of remote branches: `git remote prune origin`\n* Delete local branch: `git branch -d some-branch`\n* What are the changes if merging branch y into x: `git merge-tree $(git merge-base x y) x y`\n\n### Staging\n\n* See what the current status in the repository: `git status`\n* Add changes to the staging area: `git add <FILE>` or `git add .` to add everything\n\n### Commits\n* Create a commit: `git commit`\n* List of latest commits: `git log --oneline`\n* Push commits to the remote branch: `git push origin main`\n* Revert to commit X\n\n```\ngit revert --no-commit X..HEAD\ngit commit\n```\n\n### References\n\n* All references in current repository: `find .git/refs/`\n* Update master reference: `git update-ref refs/heads/master <SOME_COMMIT>`\n\n### Tags\n\n* Create a new tag: `git tag -a \"v0.0.1\" -m \"First release...yay`\n"
  },
  {
    "path": "resources/github.md",
    "content": "# GitHub\n\n## Common Issues\n\n* 'can't sync because main is not tracking [PROJECT NAME]'\n\nThis happens the branch is no longer tracking main from upstream. Can be fixed with this command: `git push --set-upstream origin main`.\n\n## Cheat Sheet\n\n### Repository\n\n* Clone repository: `gh clone <REPO NAME>`\n* Sync your fork from source repo: `gh repo sync --source <USERNAME>/<REPONAME>`\n\n### Pull Requests\n\n* Create PR: `gh pr create --title \"Some Pull Request ;)`\n* List PRs: `gh pr list`\n* Change to PR (aka checkout): `gh pr checkout <PR NUMBER>`\n* Update PR: `git push`\n\n### CI\n\n* List builds/workflow runs: `gh run list`\n\n### Rebase\n\n* Rebase current branch commits on top of latest changes in main branch: `git rebase origin/main`\n"
  },
  {
    "path": "resources/gitops.md",
    "content": "https://opengitops.dev\n"
  },
  {
    "path": "resources/go.md",
    "content": "## Go\n\n### Cheat Sheet\n\n* Run your program:\n\n```\ngo run <file path>\n```\n\n* Compile your app package:\n\n```\ngo build <package path>\n```\n\n* Install and run from a remote location:\n\n```\ngo install github.com/bregman-arie/myApp\nbin/myApp\n```\n\n* var definition\n\n```\nvar i int = 42\nOR\ni := 42\n```\n\n* Define several variables\n\n```\nvar (\n     name string = \"hello\"\n     num int = 42\n```\n"
  },
  {
    "path": "resources/groovy.md",
    "content": "### Cheat Sheet\n\n* Type of a variable\n\n`println obj.getClass()`\n"
  },
  {
    "path": "resources/jenkins/README.md",
    "content": "# Jenkins\n\n<div align=\"center\"><img src=\"../../images/jenkins_map.png\"></div><hr/>\n\n## Documentation\n\nTitle | Author | Comments\n:------ |:------|:------:\n[Official Jenkins Doc](https://jenkins.io/user-handbook.pdf) | jenkinsci-docs |  |\n[Job DSL Plugin](https://jenkinsci.github.io/job-dsl-plugin) | jenkinsci | |\n\n## Tutorials & Articles\n\nTitle | Author | Comments\n:------ |:------|:------:\n[Jenkins Tutorial For Beginners](https://devopscube.com/jenkins-2-tutorials-getting-started-guide) | Focusing on installing and configuring Jenkins\n[Top 10 Best Practices for Jenkins Pipeline Plugin](https://www.cloudbees.com/blog/top-10-best-practices-jenkins-pipeline-plugin) | apemberton |  |\n[The Need For Jenkins Pipeline](https://www.cloudbees.com/blog/need-jenkins-pipeline) | Viktor Farcic | Great article. Recommended.\n[Jenkins Pipeline](https://www.cloudbees.com/blog/jenkins-pipeline) | Viktor Farcic | Another great article from Victor.\n\n## Scripts\n\nRepository | Comments\n:------ |:------:\n[pipeline-examples](https://github.com/jenkinsci/pipeline-examples) | The [HTML](https://jenkins.io/doc/pipeline/examples) version |\n[This Repository](https://github.com/bregman-arie/jenkins-goodies/tree/master/scripts) | |\n\n## Presentations\n\nTitle | Author | Comments\n:------ |:------|:------:\n[Jenkins API](https://www.slideshare.net/ArieBregman/jenkins-api) | Arie Bregman |  |\n\n## Blogs\n\nName | Comments\n:------ |:------:\n[Cloudbees Blog](https://www.cloudbees.com/blog) | |\n[Jenkins Community Blog](https://jenkins.io/node/) | |\n\n## Videos\n\nName | Comments\n:------ |:------:\n[Multi Branch Pipeline](https://www.youtube.com/watch?v=11z2x3VYO_I&t=) | For projects with multiple branches |\n[How to create Jenkins Pipeline with an Example](https://www.youtube.com/watch?v=s73nhwYBtzE) | Great for beginners |\n[Codifying the Build and Release Process with a Jenkins Pipeline Shared Library](https://www.youtube.com/watch?v=lzzx59kLW9w) |\n\n## Checklist\n\n- [ ] **Terms & Entities**\n  - [ ] Job\n  - [ ] Build\n  - [ ] Plugin\n  - [ ] Slave\n\n- [ ] **Job**\n  - [ ] Freestyle\n  - [ ] Pipeline\n    - [ ] Declarative \n    - [ ] Scripted\n  - [ ] Checkpoint (cloudbees only at this point)\n\n- [ ] **Slave**\n  - [ ] Agent\n  - [ ] Executor\n  - [ ] Label\n\n### Cheat Sheet\n\n* Set build result to FAILURE/SUCCESS/UNSTABLE: `currentBuild.result = 'FAILURE'`\n"
  },
  {
    "path": "resources/jenkins/jjb/gate.md",
    "content": "The following is the job definition of a gate to test changes to job definitions\n\n- job:\n    name: 'gate-jenkins-jjb'\n    defaults: job_defaults\n    concurrent: true\n    parameters:\n        - string: { name: GERRIT_REFSPEC, default: +refs/heads/master }\n        - string: { name: GERRIT_BRANCH,  default: origin/master }\n        - string: { name: DEBUG,  default: '' }\n    description: |\n        Managed by Jenkins Job Builder. Do not edit via web.\n        <p>\n        Gate against changes to job definitions\n        </p>\n    node: <node_label_CHANGEME>\n    properties:\n        - build-discarder:\n            days-to-keep: 14\n            artifact-days-to-keep: 14\n    scm:\n        - codeng-scm:\n              project: '<project_that_holds_jobs_CHANGEME>\n              branch: \"master\"\n              refspec: \"$GERRIT_REFSPEC\"\n    triggers:\n        - trigger-rhos-qe-jenkins:\n              file_paths:\n                  - compare-type: 'REG_EXP'\n                    pattern: '(jobs/.*|bin/.*|requirements\\.txt)'\n    builders:\n        - shell: |\n            #!/bin/bash\n            set -euo pipefail\n            IFS=$'\\n\\t'\n            [ \"${DEBUG:-0}\" != 0 ] && set -x\n            PIP_LOG=$WORKSPACE/pip.log\n\n            tox -e jobs\n    publishers:\n        - archive:\n            artifacts: '**/*.log'\n\nAlso, add this in tox.ini:\n\n[testenv:jobs]\ncommands =\n    {[testenv]commands}\n    mkdir -p out-xml\n    bash -c \"XDG_CACHE_HOME={toxworkdir}/.jjb_cache jenkins-jobs --ignore-cache test --config-xml -o out-xml --recursive jobs 2>{envname}.log || \\{ >&2 cat {envname}.log; exit 1; \\}\"\n"
  },
  {
    "path": "resources/jenkins/pipeline/README.md",
    "content": "## Pipeline in a nutshell\n\n* A moment of history: Pipeline plugin was developed after the success\n  of Build Flow Plugin and it was first called 'Workflow Plugin'.\n\n* It's considered to be better than freestyle type because it's implemented\n  in code, survives restarts and can be extended easily.\n\n* Pipeline supports two syntaxes, Declarative and Scripted.\n\n* [Here](https://github.com/bregman-arie/jenkins-goodies/blob/master/pipeline/basic_declarative_pipeline) is an example for simple declarative pipeline\n\n    * 'step' is a single task and tells Jenkins what to do.\n    * 'stage' is a collection of steps. Example for stages: 'Test', 'Build', 'Deploy'.\n    * 'node' allocates an executor and workspace for executing the steps in all the stages\n\n* To use Jenkins pipeline, you need Jenkins 2 and Pipeline plugin installed.\n\n* Scripted pipeline is written in Groovy.\n\n* You can create a new pipeline by using Jenkins UI or adding a Jenkinsfile in the relevant repository. Using Jenkinsfile is recommended.\n\n* To declare pipeline with UI -> click on 'new item', enter the name, choose 'Pipeline' from type options and click on 'save'.\n\n* You can use https://<my-jenkins>/pipeline-syntax as built-in documentation for generating simple or complex\n  pipeline snippets.\n\n* There are several types of variables supported in a pipeline. You can find updated reference on global variables\n  here: https://<my-jenkins>/pipeline-syntax/globals\n\n    * env variables are accessible from Groovy code with 'env.<var_name>'.\n    * params are read-only variables exposed for you to use in the pipeline.\n    * currentBuild variables are set in the context of a specific build\n\n* A reference of the available steps can be found [here](https://jenkins.io/doc/pipeline/steps).\n  There is also such reference in your Jenkins instance https://<my-jenkins>/pipeline-syntax/html\n\n* Pipeline examples can be found [here](https://jenkins.io/doc/pipeline/examples)\n"
  },
  {
    "path": "resources/jenkins/pipeline/basic_declarative_pipeline",
    "content": "// Best practice is to actually call this file 'Jenkinsfile'\n// I named it 'basic_pipeline' just to explain its purpose\n\n// Pipeline is written in Pipeline DSL syntax\n\npipeline {\n\n    // Jenkins to allocate workspace and executor\n    agent any\n\n    // Stages are block of code you execute through your build\n    stages {\n\n    // This is the first stage the pipeline will execute\n    // A stage execute specific step of your flow\n        stage('Build') {\n            // Steps to be executed as part of the 'Build' stage\n            steps {\n                sh 'echo \"Building\"' // Execute the 'echo' shell command\n            }\n        }\n\n    // Once previous stage finished successfully, Jenkins will \n    // execute this stage.\n        stage('Test'){\n            steps {\n                sh 'echo \"Testing\"'\n                // To publish test results add this: junit 'reports/**/*.xml'\n            }\n        }\n\n        stage('Deploy') {\n            steps {\n                sh 'echo \"Deploying\"'\n            }\n        }\n    }\n\n} // End of the pipeline \n"
  },
  {
    "path": "resources/jenkins/pipeline/basic_scripted_pipeline",
    "content": "#!/usr/bin/env groovy\n"
  },
  {
    "path": "resources/jenkins/pipeline/tips.md",
    "content": "# Pipeline - Tips\n\n* Use https://<my-jenkins>/pipeline-syntax as built-in documentation for generating simple or complex\n  pipeline snippets.\n\n* Use https://<my-jenkins>/pipeline-syntax/globals to find out what variables you can use in your pipeline.\n    \n* A reference of the available steps can be found [here](https://jenkins.io/doc/pipeline/steps).\n  There is also such reference in your Jenkins instance https://<my-jenkins>/pipeline-syntax/html\n\n* Learn from [examples](https://jenkins.io/doc/pipeline/examples)\n"
  },
  {
    "path": "resources/jenkins/scripts/README.md",
    "content": "## How to run\n\nThere are several ways to run the snippets in this repository. The quickest would be:\n\n    1. Go to https://<Jenkins_URL>/script\n    2. Paste the script\n    3. Click on \"Run\"\n    4. Enjoy :)\n"
  },
  {
    "path": "resources/jenkins/scripts/builds/abort.groovy",
    "content": "Jenkins.instance.getItemByFullName(\"<JOB_NAME>\").getBuildByNumber(<BUILD_NUMBER>).finish(hudson.model.Result.ABORTED, new java.io.IOException(\"Aborting build\")); \n"
  },
  {
    "path": "resources/jenkins/scripts/builds/remove_old.groovy",
    "content": "// Remove old builds that are older than the number of given days\n\ndef removeOldBuilds(buildDirectory, days = 14) {\n\n  def wp = new File(\"${buildDirectory}\")\n  def currentTime = new Date()\n  def backTime = currentTime - days\n\n  wp.list().each { fileName ->\n      folder = new File(\"${buildDirectory}/${fileName}\")\n      if (folder.isDirectory()) {\n          def timeStamp = new Date(folder.lastModified())\n          if (timeStamp.before(backTime)) {\n            folder.delete()\n          }\n      }\n  }\n}\n\nreturn this;\n"
  },
  {
    "path": "resources/jenkins/scripts/builds/remove_old_v2.groovy",
    "content": "// Remove builds older than 5 days\n\nbuilds = Jenkins.instance.getItem('test2').getBuilds()\ndef daysToKeep = 5\n \nbuilds.each{ build ->\n  def runningSince = groovy.time.TimeCategory.minus( new Date(), build.getTime() ).getDays();\n  if (runningSince > daysToKeep){\n    build.delete();\n    println \"Build \" + build + \"was deleted since it was built \" + runningSince + \" days ago\";\n  }\n}  \n"
  },
  {
    "path": "resources/jenkins/scripts/jobs/delete_jobs_with_regex.groovy",
    "content": "def jobs = Jenkins.instance.items.findAll { job -> job.name =~ /the_regex/ }\n    \njobs.each { job ->\n    println job.name\n    //job.delete()\n}\n"
  },
  {
    "path": "resources/jenkins/scripts/jobs/print_all_jobs.groovy",
    "content": "jobs = jenkins.model.Jenkins.instance.getJobNames()\n\nprintln \"Number of jobs: \" + jobs.size()\nprintln \"\\nList of jobs:\\n\" + jobs.join(\"\\n\")\n\n\n/*\n\nAnother version:\n\njobs = jenkins.model.Jenkins.instance.items\n\nprintln \"Number of jobs: \" + jobs.size()\nprintln \"\\nList of jobs:\\n\"\n\njobs.each { job -> println job.name }\n\n*/\n"
  },
  {
    "path": "resources/jenkins/scripts/jobs/print_jobs_with_regex.groovy",
    "content": "jobs = jenkins.model.Jenkins.instance.items.findAll { job -> job.name =~ /network.*10_((?!SRIOV).)*$/ }\n\nprintln \"Number of jobs: \" + jobs.size() + \"\\n\"\n\njobs.each { job -> println job.name }\n"
  },
  {
    "path": "resources/jenkins/scripts/jobs/rename_job.groovy",
    "content": "def job = jenkins.model.Jenkins.instance.getItem('dashboard');\ndef currentName = job.name;\ndef newName = 'test2';\n\njob.renameTo(newName);\n\nprintln \"\\nRenamed \" + currentName + \" to \" + newName;\n"
  },
  {
    "path": "resources/jenkins/scripts/jobs/rename_jobs_with_regex.groovy",
    "content": "def jobs = Jenkins.instance.items.findAll { job -> job.name =~ /the_regex/ }\ndef newNameAddition = 'test2'\n    \njobs.each { job ->\n    currentName = job.name\n    job.renameTo(job.name + newNameAddition);\n    println \"\\nRenamed \" + currentName + \" to \" + currentName + newNameAddition + \"\\n\";\n}\n"
  },
  {
    "path": "resources/jenkins/scripts/nodes/check_all_online.groovy",
    "content": "// Check if all slaves are online. Fail the script if not.\n\nint excode = 0\n\nfor (slave in hudson.model.Hudson.instance.slaves) {\n\n    if (slave.getComputer().isOffline().toString() == \"true\"){\n     println('The Slave ' + slave.name + \" is offline\");\n     excode++;\n    }  \n}\n\nif (excode > 0) {\n    println(\"We have a Slave down! I repeat, we have a slave down!\");\n    return 1;\n}\n"
  },
  {
    "path": "resources/jenkins/scripts/nodes/print_node_groups.groovy",
    "content": "import jenkins.model.Jenkins;\n\ndef list = []\ndef description = ''\n\ndef slave_label_map = [:]\nfor (slave in Jenkins.instance.slaves) {\n    description = slave.getNodeDescription()\n    if (description.contains('<#{') && description.contains('}#>')) {\n        words = slave.getLabelString().split()\n        def labelListForSlave = []\n        words.each() {\n            labelListForSlave.add(it);\n            list.add(it)\n        }\n        slave_label_map.put(slave.name, labelListForSlave)\n    }\n}\nlist.unique()\n"
  },
  {
    "path": "resources/jenkins/scripts/notification/notifier/__init__.py",
    "content": "# Copyright 2018 Arie Bregman\n#\n#    Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n#    not use this file except in compliance with the License. You may obtain\n#    a copy of the License at\n#\n#         http://www.apache.org/licenses/LICENSE-2.0\n#\n#    Unless required by applicable law or agreed to in writing, software\n#    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n#    License for the specific language governing permissions and limitations\n#    under the License.\n"
  },
  {
    "path": "resources/jenkins/scripts/notification/notifier/main.py",
    "content": "# Copyright 2018 Arie Bregman\n#\n#    Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n#    not use this file except in compliance with the License. You may obtain\n#    a copy of the License at\n#\n#         http://www.apache.org/licenses/LICENSE-2.0\n#\n#    Unless required by applicable law or agreed to in writing, software\n#    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n#    License for the specific language governing permissions and limitations\n#    under the License.\nimport json\nimport requests\nimport sys\n\nfrom notifier import parse\n\nmessage = 'Build {}'.format('FAILED')\n\nheaders = {\n    'Content-Type': 'application/json',\n}\n\npayload = {\n    'message': message,\n}\n\nendpoint = ''\n\n\ndef main():\n    \"\"\"Main entry point.\"\"\"\n\n    # Parse given arguments\n    parser = parse.create_parser()\n    args = parser.parse_args()\n\n    r = requests.post(endpoint,\n                      data=json.dumps(payload + args),\n                      headers=headers)\n    print(r.status)\n\n\nif __name__ == '__main__':\n    sys.exit(main())\n"
  },
  {
    "path": "resources/jenkins/scripts/notification/notifier/parse.py",
    "content": "# Copyright 2018 Arie Bregman\n#\n#    Licensed under the Apache License, Version 2.0 (the \"License\"); you may\n#    not use this file except in compliance with the License. You may obtain\n#    a copy of the License at\n#\n#         http://www.apache.org/licenses/LICENSE-2.0\n#\n#    Unless required by applicable law or agreed to in writing, software\n#    distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT\n#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the\n#    License for the specific language governing permissions and limitations\n#    under the License.\n\nimport argparse\n\n\ndef create_parser():\n    \"\"\"Returns argument parser\"\"\"\n\n    parser = argparse.ArgumentParser(add_help=True)\n\n    parser.add_argument('--debug', '-d', action='store_true',\n                        dest=\"debug\", help='Turn on debug')\n\n    parser.add_argument('--yaml-file', '-y', dest='yaml_f', nargs='+',\n                        help='YAML files')\n    return parser\n"
  },
  {
    "path": "resources/jenkins/scripts/plugins/list_all_plugins.groovy",
    "content": "Jenkins.instance.pluginManager.plugins.sort { it.getShortName() }.each{ plugin -> println \"${plugin.getShortName()}:${plugin.getVersion()}\" }\n"
  },
  {
    "path": "resources/jenkins/scripts/tests/count_test_failures.groovy",
    "content": "import jenkins.*\nimport jenkins.model.*\nimport hudson.*\nimport hudson.model.*\n  \ndef jobNameStartsWith = 'DFG-network-neutron'\ndef textToMatch = \"Traceback\"\n\nJenkins.instance.items.findAll { it.displayName.startsWith(jobNameStartsWith) }\n.each { job ->   \n  if(job.builds) {    \n    def build = job.isBuilding() ? job.builds[1] : job.builds[0]\n\n    def log = build.logFile.text\n    def regexPattern = \"(?i).*${textToMatch}.*\"\n    def summary = log =~ /${regexPattern}/\n    println \"${summary.size()}: $job.displayName\"\n  }\n}\nreturn \"\"\n"
  },
  {
    "path": "resources/jinja2.md",
    "content": "### CheatSheet\n\n* Split string to list of items using comma as the separator: `{% set list1 = variable1.split(',') %}`\n"
  },
  {
    "path": "resources/kubernetes.md",
    "content": "## Kubernetes\n\n### Getting started with Kubernetes\n\nName | Comments\n:------ |:--------:\n[kubernetes.io](https://kubernetes.io) | Official Kubernetes site by Google\n[Kubernetes 101](https://medium.com/google-cloud/kubernetes-101-pods-nodes-containers-and-clusters-c1509e409e16) | Great beginner article on Kubernetes fundamental concepts\n[Kubernetes Tutorial for Beginners](https://www.youtube.com/watch?v=X48VuDVv0do&ab_channel=TechWorldwithNana) | Full video of 4 hours on Kubernetes (2020)\n[Learning Path: Kubernetes](https://developer.ibm.com/series/kubernetes-learning-path/) | From basic to advanced Kubernetes learning series\n[Kubernetes 101 - Concepts and Why It Matters](https://www.magalix.com/blog/kubernetes-101-concepts-and-why-it-matters?fbclid=IwAR10FZlZ9Pw5c94tGRlgsCrVZTa1bSV2mbxEP8p4cXZ5T-k4VXF-3OUKkFo) |\n[kubernetes-workshop](https://github.com/eon01/kubernetes-workshop) |\n[Kubernetes Deployment Tutorial](https://devopscube.com/kubernetes-deployment-tutorial) |\n[Katacoda](https://www.katacoda.com/courses/kubernetes) | Learn Kubernetes using Interactive Browser-Based Scenarios\n[Kubernetes Scheduler](https://jvns.ca/blog/2017/07/27/how-does-the-kubernetes-scheduler-work/) | How does the Kubernetes scheduler work?\n\n### Kubernetes - Deep Dive\n\nName | Comments\n:------ |:--------:\n[Kubernetes Networking](https://github.com/nleiva/kubernetes-networking-links) | Kubernetes Networking Resources\n[Liveness and Readiness Probes](https://www.openshift.com/blog/liveness-and-readiness-probes) |\n\n### Kubernetes - Troubleshooting\n\nName | Comments\n:------ |:--------:\n[troubleshoot.sh](https://troubleshoot.sh) | \"A kubectl plugin providing diagnostic tools for Kubernetes applications\"\n[Kubernetes Troubleshooting Visual Guide](https://learnk8s.io/troubleshooting-deployments) |\n\n### Kubernetes - Security\n\nName | Comments\n:------ |:--------:\n[Kubescape](https://github.com/armosec/kubescape) | \"Kubescape is the first tool for testing if Kubernetes is deployed securely as defined in Kubernetes Hardening Guidance by NSA and CISA\"\n[Falco](https://falco.org) | \"Falco...is the de facto Kubernetes threat detection engine\"\n\n### Kubernetes - Misc\n\nName | Comments\n:------ |:--------:\n[confTest](https://www.conftest.dev) | \"Conftest is a utility to help you write tests against structured configuration data\" (Used in the development phase)\n[datree](https://www.datree.io) | \"Prevent Kubernetes Misconfigurations From Reaching Production\" (Used in development phase)\n[gatekeeper](https://github.com/open-policy-agent/gatekeeper) | Used in the production\n[telepresence](https://www.telepresence.io) | \"FAST, LOCAL DEVELOPMENT FOR KUBERNETES AND OPENSHIFT MICROSERVICES\"\n[Kubernetes CheatSheet](https://cheatsheet.dennyzhang.com/cheatsheet-kubernetes-A4) |\n[OperatiorHub.io](https://www.operatorhub.io) | Kubernetes native applications\n[YAML templates](https://cheatsheet.dennyzhang.com/kubernetes-yaml-templates) |\n[Kubesort](https://github.com/AATHITH/kubesort) | \"kubesort helps you sort the results from kubectl get in an easy way\"\n[IngressMonitorController](https://github.com/stakater/IngressMonitorController) | \"A Kubernetes controller to watch ingresses and create liveness alerts for your apps/microservices\"\n\n### Kubernetes - SRE\n\nName | Comments\n:------ |:--------:\n[KubeInvaders](https://github.com/lucky-sideburn/KubeInvaders) | \"Chaos Engineering Tool for Kubernetes and Openshift\"\n\n### Kubernetes - Certificates\n\nName | Comments\n:------ |:--------:\n[CKAD-Practice-Questions](https://github.com/bbachi/CKAD-Practice-Questions) | \"a consolidated list for CKAD practice questions\"\n[CKAD Prep Exam Video](https://www.youtube.com/watch?v=TPXwVmvzlV4&ab_channel=TheFrontOpsGuy) | A video of doing a CKAD prep exam (2020)\n[CKA Complete Prep Exam ](https://github.com/walidshaari/Kubernetes-Certified-Administrator) | CKA prep exam (2021)\n[CKAD Complete Prep Exam ](https://github.com/dgkanatsios/CKAD-exercises) | CKAD prep exam (2021)\n[CKS Complete Prep Exam ](https://github.com/walidshaari/Certified-Kubernetes-Security-Specialist) | CKA prep exam (2021)\n\n### Kubernetes - Labs\n\nName | Comments\n:------ |:--------:\n[Building a Kubernetes 1.23 Cluster with Kubeadm](https://github.com/croguerrero/CKA-Labs/blob/main/resources/KC.md) | \"Labs CKA - Build a Kubernetes cluster 1.23\"\n\n### Best Practices\n\n#### Security Best Practices\n\n* Secure inter-service communication (one way is to use Istio to provide mutual TLS)\n* Isolate different resources into separate namespaces based on some logical groups\n* Use supported container runtime (if you use Docker then drop it because it's deprecated. You might want to CRI-O as an engine and podman for CLI)\n* Test properly changes to the cluster (e.g. consider using Datree to prevent kubernetes misconfigurations)\n* Limit who can do what (by using for example OPA gatekeeper) in the cluster\n* Use NetworkPolicy to apply network security\n* Consider using tools (e.g. Falco) for monitoring threats\n\n### CheatSheet\n\n#### Minikube\n\n* Minikube version: `minikube version`\n* Start cluster: `minikube start`\n* Delete cluster: `minikube delete`\n\n#### Common Kubectl Operations\n\n* Create objects defined in a YAML: kubectl apply -f rs.yaml\n\n#### Service Accounts\n\n* List service accounts: `kubectl get serviceaccounts`\n\n### Cluster\n\n* Cluster version: `kubectl version`\n* Cluster information: `kubectl cluster-info`\n* List nodes: `kubectl get nodes`\n\n### Images\n\n* List all containers images from all namespaces:\n\n```\nkubectl get pods --all-namespaces -o jsonpath=\"{.items[*].spec.containers[*].image}\" |\\\ntr -s '[[:space:]]' '\\n' |\\\nsort |\\\nuniq -c\n```\n\n### Pods\n\n* List of Pods in current namespace: `kubectl get po`\n* List of Pods in all amespaces: `kubectl get po --all-namespaces`\n* Get containers names: `kubectl get po <POD_NAME> -o jsonpath=\"{.spec.containers[*].name}\"`\n\n* Create a Pod from file: `kubectl create -f pod_definition.yaml`\n* Delete a Pod using a YAML definition: `kubectl delete -f pod_definition.yaml`\n* Delete a Pod using the Pod name: `kubectl delete <POD_NAME>`\n* Delete a Pod instantly: `kubectl delete <POD_NAME> --grace-period=0 --force`\n\n* Execute commands inside a container: `kubectl exec -it -c <CONTAINER_NAME> <POD_NAME> ls`\n\n* Display logs of a Pod: `kubectl logs <POD_NAME>`\n* Display logs of a specific container in a Pod: `kubectl logs <POD_NAME> -c <CONTAINER_NAME>`\n\n* Get Pod name based on specific labels\n\n```\nPOD_NAME=$(kubectl get pod \\\n--no-headers \\\n-o=custom-columns=NAME:.metadata.name \\\n-l type=api,service=some-service \\\n| tail -1)\n```\n\n* Get pods sorted by time creation: `kubectl get pods --sort-by='{.metadata.creationTimestamp}'`\n* Run long/endless running container: `kubectl run test --image ubuntu -- sleep 2000000000`\n* Get a shell access to a container: `kubectl exec --stdin --tty some-shell -- /bin/bash`\n\n### User\n\n* Creating a new user\n\n```\nopenssl genrsa -out user.key 2048 # create key\nopenssl req key user.key user.csr -subj \"/CN=user /O=sgroup\" # create csr\nopenssl x509 -req -in user.csr -CA ca.crt -CAkey ca.key -CAcreateseral -out user.crt -days 365\nkubectl config set-credentials myuser --client-certificates=$PWD/user.crt --client-key=$PWD/user.key\nkubectl config set-context myuser-context --cluster=k8s-cluster --user=user\n```\n\n### Service\n\n* Expose a ReplicaSet: `kubectl expose rs REPLICASET_NAME --name=SERVICE_NAME --target-port=PORT --type=NodePort/SOME_OTHER_SERVICE_TYPE`\n\n### Deployment\n\n* Create and delete a deployment in one step:\n```\n$ kubectl create deployment demo --image=cloudnatived/demo:hello\n\n$ kubectl get pods --selector app=demo\n\n$ kubectl delete pods --selector app=demo\n\n$ kubectl delete all --selector app=demo\n```\n### Aliases \n\n* Use kubectl aliases to speed up and reduce typo errors, practice these alaises early at your work and study for the exam. some example aliases:\n```\nalias k='kubectl'\nalias kg='kubectl get'\nalias kgpo='kubectl get pod'\nalias kcpyd='kubectl create pod -o yaml --dry-run=client'\nalias ksysgpo='kubectl --namespace=kube-system get pod'\n\nalias kd='kubectl delete'\nalias kdf='kubectl delete -f'\n## for quick deletes you can add --force --grace-period=0  **Not sure if it is a good idea if you are in a production cluster**\nalias krmgf='kubectl delete --grace-period 0 --force'\nalias kgsvcoyaml='kubectl get service -o=yaml'\nalias kgsvcwn='watch kubectl get service --namespace'\nalias kgsvcslwn='watch kubectl get service --show-labels --namespace'\n\n#example usage of aliases\nkrmgf nginx-8jk71    # kill pod nginx-8jk71 using grace period 0 and force\n\n```\n\n### Monitoring\n\n* Install metrics-server using Minikube: `minikube addons enable metrics-server`\n\n### Secrets\n\n* Install Bitnami Sealed secrets controller\n\n```\nhelm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets\nhelm repo update\nhelm install sealed-secrets-controller sealed-secrets/sealed-secrets\n```\n\n* sealed-secrets binary: `https://github.com/bitnami-labs/sealed-secrets/releases`\n\n* encrypt secret in yaml or json format: `kubeseal -n my-namespace <.db-creds.yml> db-creds.json`\n* apply secret on the cluser: `kubectl apply -f db-creds.json -n my-namespace`\n\n### Kustomize\n\n* Generate customized YAML file(s): `kustomize build name_of_application`\n* Apply generated customized YAML file(s) to the cluster: `kustomize build name_of_application`\n"
  },
  {
    "path": "resources/linux.md",
    "content": "# Linux\n\n<div align=\"center\"><img src=\"../images/linux_map.png\"></div><hr/>\n\n## Learn Linux - Tutorials and Guides\n\nName | Comments\n:------|:------:\n[Linux Journey](https://linuxjourney.com) | Written guides + exercises + quiz\n[Techmint Linux](https://www.tecmint.com/free-online-linux-learning-guide-for-beginners) | Written articles/lessons\n[Linux Survival](https://linuxsurvival.com/linux-tutorial-introduction) | Interactive guide\n[NixCraft](https://www.cyberciti.biz/) | Succinct guides\n[Linux Filesystem Explained](https://www.linux.com/training-tutorials/linux-filesystem-explained) | NSIA\n[Linux SysOps Handbook](https://abarrak.gitbook.io/linux-sysops-handbook) | A study notes book for the common knowledge and tasks of a Linux system admin.\n[LinuxToday](https://www.linuxtoday.com) |  Contributor-driven news resources\n\n\n## Linux Internals - Articles\n\nName | Comments\n:------|:------:\n[How are Unix pipes implemented?](https://toroid.org/unix-pipe-implementation) | \n\n## Linux Internals - Videos\n\nName | Comments\n:------|:------:\n[Understanding fork() system call for new process creation](https://www.youtube.com/watch?v=PwxTbksJ2fo) | \n\n## Learn Linux - Videos\n\nName | Comments\n:------|:------:\n[learnlinux.tv](https://www.learnlinux.tv) | Very practical videos on various topics\n\n## Books\n\nAuthor | Title | Comments\n:------ |:------:|:--------:\nWilliam Shotts | [The Linux Command Line](http://linuxcommand.org/tlcl.php) | Free |\nChristopher Negus | [Linux Bible](https://www.wiley.com/en-us/Linux+Bible%2C+9th+Edition-p-9781118999875) | |\n\n## Sites\n\nName | Comments\n:------|:------:\n[Linux From Scratch](http://www.linuxfromscratch.org) |\n[Operating Systems: Three Easy Pieces](http://pages.cs.wisc.edu/~remzi/OSTEP) |\n\n\n## Videos\n\nPresenter | Video | Length | Level\n:------ |:------|:------:|:--------:\nJoseph Delgadillo | [The Complete Linux Course: Beginner to Power User!](https://www.youtube.com/watch?v=wBp0Rb-ZJak&t=6578s) | 07:23:52 | Beginner |\n\n## Tweak OS\n\nFedora disable alert sound\n\n`dconf write /org/gnome/desktop/sound/event-sounds \"false\"`\n\n## Presentations\n\nAuthor | Presentation | Level | Comments\n:------ |:------|:--------:|:--------:\nArie Bregman | [Linux Networking](https://www.slideshare.net/ArieBregman/linux-networking-113100224) | Beginner | |\nAndrew McNico | [tcpdump](https://www.slideshare.net/j0b1n/tcpdump-hunter?qid=b71dea53-7829-40a3-b82b-4a669383eac6) | Beginner | |\n\n## Cheat Sheet\n\n### Common Operations\n\n* Copy files to a remote host: `rsync -azv dir USER:REMOTE_HOST_ADDR`\n* View file size: `du -sh DIR/FILE`\n\n### Users\n\n* Who is connected: `w`\n\n### Troubleshooting and Debugging\n\n* see list of processes and the CPU and memory the consume: `top`\n* show how long the server is up running: `uptime`\n* List open files: `lsof`\n* List open connections: `lsof -i`\n\n### SElinux\n\nGet SElinux denials\n\n```\nsemodule -DB\n\nsudo ausearch -m avc -m user_avc -m selinux_err -m user_selinux_err -i -ts today\n```\n\n### Communication\n\n* Broadcast a message to everyone: `wall`\n\n### Tmux\n\n* Join a session `tmux a`\n* Attach to existing session: `tmux attach -t <SESSION_NAME>`\n* New tab: `ctrl + B + c`\n* New session called \"bla\": `tmux new -s bla`\n\n#### Virsh\n\n* Destroy: `virsh destroy <vm_name>`\n* Shutdown: `virsh shutdown <vm_name>`\n\n#### Date\n\n* year: `date +”%Y”`\n* month: `date +”%m”`\n* day: `date +”%a”`\n\n#### Fedora\n\n* Reset \"interface text\" value: `gsettings reset org.gnome.desktop.interface font-name`\n* Reset \"monospace text\" value: `gsettings reset org.gnome.desktop.interface monospace-font-name`\n* Reset \"document text\" value: `gsettings reset org.gnome.desktop.interface document-font-name`\n* Reset \"Legacy Window Titles\": `gsettings reset org.gnome.desktop.wm.preferences titlebar-font`\n\n#### Files\n\n* Sort files by size: `ls -l | sort -nk5`\n* Find broken links: `find /some/path -type l -exec test ! -e {} \\; -print`\n\n### LDAP\n\n* LDAP search for a username: `ldapsearch -x -W -D username`\n* LDAP search for a username specifying DC and ldap server: `ldapsearch -x -H ldap://ldap.blabla.com -LLL -b 'dc=DOMAIN,dc=com' \"rhatNickName=abregman\"`\n\n### YAML\n\n* Validate YAML file with Ruby: `ruby -ryaml -e \"p YAML.load(STDIN.read)\" < some_file.yaml`\n* Validate YAML file with Python: `pip install pyyaml; python -c 'import yaml, sys; print(yaml.safe_load(sys.stdin))' < some_file.yaml`\n\n### Misc\n\n* Random string of 7 characters: `mkpasswd -l 8`\n* Generate 8 digit random number: `shuf -i 9999999-99999999 -n 1`\n\n## Checklist\n\nCheck your Linux educational progress with the following list:\n\n- [ ] **File System Hierarchy**\n  - [ ] `/` (aka root)\n  - [ ] `/bin` and `/sbin`\n  - [ ] `/opt`\n  - [ ] `/usr`\n  - [ ] `/var`\n  - [ ] `/home`\n  - [ ] `/etc`\n  - [ ] `/proc`\n  - [ ] `/tmp`\n\n  - [ ] Test\n    - [ ] What is '/'?\n    - [ ] What is stored in '/bin'? what about `/etc`?\n    - [ ] What '/home' is usually used for?\n    - [ ] What type of files can you find in '/var'?\n    - [ ] What is unique about `/tmp`?\n\n- [ ] **Navigation**\n  - [ ] Commands\n    - [ ] `pwd` - where am I?\n    - [ ] `cd` - change directory\n  - [ ] relative vs. absolute paths\n\n  - [ ] Test\n    - [ ] How to check your current path?\n    - [ ] Where `cd ..` will take you?\n    - [ ] What would be the effect of running `cd .`?\n    - [ ] What will happen when running 'cd -'?\n\n- [ ] **Files**\n  - [ ] Types of files\n    - [ ] Regular\n    - [ ] Directory\n    - [ ] Socket\n    - [ ] Block\n    - [ ] Link\n\n  - [ ] Commands\n    - [ ] `ls` - list files and directories\n      - [ ] `-a` for listing hidden files\n      - [ ] `-l` for list formt\n      - [ ] `-t` order by time\n      - [ ] `-F` better distinguish between regular files and directories\n    - [ ] `touch` - creating files (original intention is updating timestamp)\n      - [ ] nice to know: `touch file{1..5}`\n    - [ ] `rm` - remove files\n      - [ ] `-r` for recursive\n      - [ ] `-f` to force removal, no questions asked\n    - [ ] `mkdir` - create directories\n      - [ ] `-p` - for creating multiple nested directories\n    - [ ] `rmdir` - remove directories\n    - [ ] `echo` - display a line of text\n    - [ ] `cat` - concatenate files (common usage: read a file)\n    - [ ] `mv` - move files directories (also rename files and directories)\n    - [ ] `cp` - copy a file\n      - [ ] `-r` for recursive (copy a directory)\n\n  - [ ] Test\n    - [ ] How to list hidden files?\n    - [ ] How to create an empty new file?\n    - [ ] How to remove a directory?\n    - [ ] How to rename a file?\n    - [ ] How to copy an entire directory with all its files?\n\n- [ ] Commands\n    - [ ] `man` - manual for commands\n    - [ ] `which` - get full path for given command\n    - [ ] `whatis` - one-line manual page descriptions\n\n- [ ] **I/O redirection**\n  - [ ] File Descriptor\n    - [ ] stdin 0 (input) <\n    - [ ] stdout 1 (output) >\n    - [ ] stderr 2 (error) 2>\n  - [ ] Append >>\n\n  - [ ] Test\n    - [ ] How to redirect output?\n    - [ ] What would be the result of the following command `blop 2> file`?\n\n- [ ] Text Editor (one is enough)\n  - [ ] vim\n    - [ ] `i` to start typing\n    - [ ] `:wq` to exit (or `shit+zz`)\n  - [ ] nano\n  - [ ] emacs\n  - [ ] atom\n  - [ ] sublime\n  \n  - [ ] Test (mainly for vim)\n    - [ ] How to remove an entire line?\n    - [ ] How to copy 5 lines?\n    - [ ] How to jump to the end of the line\n    - [ ] How to remove one word\n    - [ ] How to jump to the end of the file\n\n- [ ] **Users**\n  - [ ] Commands\n    - [ ] useradd\n    - [ ] usermod\n    - [ ] userdel\n    - [ ] who\n\n  - [ ] Nice to know commands\n    - [ ] lastlog\n\n  - [ ] Test\n    - [ ] how to add a new user?\n    - [ ] should you be using your user or root?\n\n- [ ] **Network**\n  - [ ] netstat\n\n- [ ] **Monitoring and performence commands**\n    - [ ] top\n    - [ ] stat\n\n- [ ] **Processes**\n    - [ ] Running in ackground (&)\n\n- [ ] **Archives**\n  - [ ] what is it good for\n  - [ ] tar\n  - [ ] zip\n  - [ ] commands\n    - [ ] create an archive\n    - [ ] list archive's content\n    - [ ] remove an archive\n\n- [ ] **Storage, Filesystem**\n  - [ ] inode\n"
  },
  {
    "path": "resources/machine_learning.md",
    "content": "# Machine Learning\n\n## Checkbox List\n\n- [ ] **Basic Terms**\n  - [ ] Sample\n  - [ ] Attribute\n\n- [ ] **Data Mining**\n  - [ ] Association Rules\n    - [ ] Apriori\n      - [ ] Support\n      - [ ] Confidence\n      - [ ] Conviction\n      - [ ] Lift\n    - [ ] Eclat\n    - [ ] FP-growth\n\n\n## Videos\n\nName | Description\n:------|:------:\n[Apriori](https://www.youtube.com/watch?v=WGlMlS_Yydk) | Apriori algorithm beginner explanation \n[Bayesian Networks](https://www.youtube.com/watch?v=TuGDMj43ehw) | Bayesian beginner explanation \n\n## Repos\n\nName | Description\n:------|:------:\n[PracticalAI](https://github.com/GokuMohandas/practicalAI) | A practical approach to learning and using machine learning.\n"
  },
  {
    "path": "resources/mongo.md",
    "content": "# MongoDB\n\n## Tutorials\n\nName | Description\n:------|:------:\n[Guru99 MongoDB Tutorial](https://www.guru99.com/what-is-mongodb.html) | MongoDB Tutorial\n[Tutorialspoint Tutorial](https://www.tutorialspoint.com/mongodb) | MongoDB Tutorial\n\n## Authentication and Authorization\n\nName | Description\n:------|:------:\n[Tutorial User Administration Without You Integrating LDAP with MongoDB](https://www.youtube.com/watch?v=ky8OKWPCZXs&ab_channel=MongoDB) | Full explanation on Mongo and LDAP integration\n\n## Cheat Sheet\n\n### Mongo Shell\n\n#### General\n\n* DB stats: `db.stats()`\n* Switch to a DB: `use DB_NAME`\n* Show collections: `show collections`\n* Show all the document in a collection: `db.COLLECTION_NAME.find({})`\n* Count the number of documents in a collection: `db.COLLECTION_NAME.count()`\n* Remove all the documents from a collection: `db.COLLECTION_NAME.remove({})`\n* Remove field from all documents: `db.COLLECTION_NAME.update({}, {$unset: { FIELD_NAME: \"\"}}, {multi:true})` \n* Rename field: ```db.hosts.updateMany( {}, { $rename: { \"CURRENT_FIELD_NAME\": \"NEW_FIELD_NAME\" } } )```\n\n#### Mongoexport \n\n* Export Mongo DB collection to a JSON file: `mongoexport --db=DB_NAME --collection=COLLECTION_NAME --out=data.json`\n* Export specific fields from a Mongo DB collection to a  JSON file: `mongoexport --db=DB_NAME  --fields=FIELD_NAME,FIELD_NAME2 --collection=COLLECTION_NAME --out=data.json`\n\n#### Aggregations\n\n* Count the number of documents: `db.accounts.aggregate([ { \"$count\": \"number of documents\" }])`\n* Show only the sub element 'attachment_uuids': `db.accounts.aggregate( [ {$project: {\"hosts.attachment_uuids\": 1, _id:0}} ] )`\n\n#### Users\n\n* Display users:\n\n```\nuse admin\ndb.getUsers()\n```\n\n* Create administrative user\n\n```\nuse admin\ndb.createUser({user: \"admin\", pwd: \"some_password\", roles: [\"root\"]})\n```\n\n* Authentice: `db.auth(USERNAME, PASSWORD)`\n\n#### Roles\n\n* Display roles\n\n```\nuse admin\ndb.getRoles()\n```\n\n* Drop/Remove role: `db.dropRole(\"NAME_OF_THE_ROLE\")`\n\n#### LDAP\n\n* Mongo Conf LDAP configuration\n\n```\nsecurity:\n  # authorization: \"enabled\"\n  ldap:\n    servers: 'some.ldap.server.com'\n    userToDNMapping:\n         '[\n            {\n               match: \"(.+)\",\n               ldapQuery: \"DC=X,DC=com??sub?(uid={0})\"\n            }\n         ]'\n    authz:\n         queryTemplate: \"{USER}?memberOf?base\"\nsetParameter:\n  authenticationMechanisms: 'PLAIN,SCRAM-SHA-1'\n```\n\n* Create role where the LDAP group \"unicorns\" has permissions to only run 'find' on \"hosts\" collection in 'test' db:\n\n```\ndb.createRole({role:\"cn=unicorns,ou=groups,dc=X,dc=com\", privileges: [{ resource: {db: \"\", collection: \"hosts\"}, actions: [\"find\"]}], roles: [{ role: \"read\", db: \"test\"}]})\n```\n\n* Drop/Remove role from the previous example: `db.dropRole(\"cn=unicorns,ou=groups,dc=X,dc=com\")`\n\n### Mongo Python\n\n* Add new element to an arrray only if it doesn't exists already\n\n```\nmongo_client.update(\n       { \"account_number\": account_num },\n       { \"$addToSet\": { \"hosts\": {'hostname': hostname} } }\n    )\n```\n\n* Add a new element to an array in sub-document (if it doesn't exists already):\n\n```\nmongo_client.update_one(\n       { \"account_number\": account_num, \"hosts.hostname\": hostname },\n       { \"$addToSet\": { \"hosts.$.attachment_uuids\": attachment_uuid } }\n    )\n```\n\n* Add all the elements in an array to an array in document (This is not about adding the array as it is but adding the elements of it):\n\n```\nmongo_client.update_one(\n    {'_id': 1},\n    {'$addToSet': {'some_list': { '$each': [1, 2, 3]} }}, upsert=True)\n```\n\n"
  },
  {
    "path": "resources/monitoring.md",
    "content": "# Monitoring\n\n## Articles\n\nName | Comments\n:------|:------:\n[Stack Overflow: How We Do Monitoring - 2018 Edition](https://nickcraver.com/blog/2018/11/29/stack-overflow-how-we-do-monitoring) |\n\n## CLI Tools\n\nName | Comments\n:------|:------:\n[statuscakectl](https://github.com/omerh/statuscakectl) | Statuscake CLI\n"
  },
  {
    "path": "resources/network.md",
    "content": "# Network\n\n## Docs\n\nName | Comments\n:------|:------:\n[TCP/IP networking referen](http://www.penguintutor.com/linux/basic-network-reference)\n\n## Books\n\nAuthor | Title | Comments\n:------ |:------:|:--------:\nJames Kurose & Keith Ross | Computer Networking: A Top-Down Approach | Great for beginners \nAndrew S. Tanenbaum | Computer Networks | Commonly used by academic networking courses \nRichard Stevens | TCP/IP illustrated | \nRichard Stevens                                              |                   UNIX Network Programming                   |                                                              \nOccupyTheWeb                                                 | Linux Basics for Hackers: Getting Started with Networking, Scripting, and Security in Kali |          Includes a chapter on wireless networking           \nEvi Nemeth & Garth Snyder & Trent R. Hein &  Ben Whaley & Dan Mackin |        UNIX and Linux System Administration Handbook         | A great book on Linux and there is a dedicated chapter for Linux networking \n\n## Videos - Basic Networking Concepts\n\nPresenter | Video | Length | Level\n:------ |:------|:------:|:--------:\nRene Molenaar | [Address Resolution Protocol (ARP) Explained](https://www.youtube.com/watch?v=xTOyZ6TWQdM) | 00:05:13 | Beginner |\nEli the Computer Guy | [The OSI Model Demystified](https://www.youtube.com/watch?v=HEEnLZV2wGI) | 00:18:40 | Beginner |\nPieter De Decker | [Hub, Switch or Router? Network Devices Explained](https://www.youtube.com/watch?v=Ofjsh_E4HFY) | 00:07:39 | Beginner |\nPieter De Decker | [How Network Address Translation Works](https://www.youtube.com/watch?v=QBqPzHEDzvo) | 00:10:04 | Beginner |\nPieter De Decker | [How DHCP Works](https://www.youtube.com/watch?v=RUZohsAxPxQ) | 00:07:55 | Beginner |\nPieter De Decker | [UDP and TCP: Comparison of Transport Protocols](https://www.youtube.com/watch?v=Vdc8TCESIg8) | 00:11:34 | Beginner |\nRicky Tapia | [CSMA/CD](https://www.youtube.com/watch?v=IAKncL67Pp4) | 00:04:01 | Beginner |\n\n\n## Videos - Linux\n\nPresenter | Video | Length | Level\n:------ |:------|:------:|:--------:\nDavid Mahler | [Introduction to Linux Network Namespaces](https://www.youtube.com/watch?v=_WgUwUf1d34) | 00:25:50 | Beginner |\n\n\n## Videos - OpenvSwitch\n\nPresenter | Video | Slides | Length | Level\n:------ |:------|:------:|:--------:|:--------:\nDavid Mahler | [Introduction to Open vSwitch](https://www.youtube.com/watch?v=rYW7kQRyUvA) | | 00:14:35 | Beginner |\nDavid Mahler | [An Introduction to Open vSwitch](https://www.youtube.com/watch?v=_PCRNUB7oNw) | [Slides](http://openvswitch.org/slides/openvswitch.en-2.pdf)| 00:35:49 | First half beginners, second half intermediate level |\n\n\n## Videos - OpenFlow\n\nPresenter | Video | Length | Level | Comments\n:------ |:------|:------:|:--------:|:--------:\nDavid Mahler | [Introduction to OpenFlow](https://www.youtube.com/watch?v=l25Ukkmk6Sk) | 00:15:25 | Beginner | |\nSteven Wallace | [Describes the OpenFlow flow table](https://www.youtube.com/watch?v=-xLQHld3fPI) | 00:03:00 | Beginner | |\nDavid Mahler | [OpenFlow with multiple Flow Tables](https://www.youtube.com/watch?v=TD5wmoD7XOE) | 00:09:30 | Intermediate | |\nDavid Mahler | [OpenFlow flow entries on Open vSwitch (OVS)](https://www.youtube.com/watch?v=FyV4MoQ3T0I) | 00:14:22 | Intermediate | Required Knowledge: Open vSwitch |\n\n## Videos - OpenStack Neutron\n\nPresenter | Video | Length | Level | Comments\n:------ |:------|:------:|:--------:|:--------:\n[Shalu George & Naveen Joy] | [Neutron networking demystified for beginners](https://www.youtube.com/watch?v=c-x90PZFv8k) | 00:39:53 | Beginner | 2017 |\nDavid Mahler | [Introduction to OpenStack Neutron](https://www.youtube.com/watch?v=yqFpyubsYfE) | 00:19:38 | Beginner | |\nAssaf Muller | [Introduction to OpenStack Neutron](https://www.youtube.com/watch?v=IGGgVuZe7UA) | 00:33:20 | Beginner | |\nRossella Sblendido | [I Can't Ping My VM! Learn How to Debug Neutron and Solve Common Problems](https://www.youtube.com/watch?v=aNA8Pvewu2M) | 00:42:16 | Intermediate | Required Knowledge: neutron basics |\nDavid Mahler | [OpenStack Neutron Packet Walkthrough (DVR)](https://www.youtube.com/watch?v=7IXEtUEZslg) | 00:44:56 | Intermediate | Required Knowledge: neutron, openvswitch, namespaces and openflow |\nMohamed Elsakhawy | [Neutron- Where Did My Packets Go?](https://www.youtube.com/watch?v=uKgMp5c6R-4) | 00:39:32 | Beginner | |\n\n## Checklist - Network Newbie\n\n- [ ] **OSI model**\n  - [ ] What layers the OSI model has and what is the purpose of each one of them?\n\n- [ ] **Ethernet**\n  - [ ] What is CSMA/CD and how it works?\n\n- [ ] **MAC address**\n  - [ ] What is it used for? Is it unique?\n  - [ ] How many bits are in a MAC address?\n\n- [ ] **IP address**\n  - [ ] What is it used for? Is it unique worldwide?\n  - [ ] What are the two parts of an IP address?\n  - [ ] How many bits are in an IP address?\n\n- [ ] **Subnets**\n  - [ ] Two hosts are on the same subnet if they have the same '_____'\n  - [ ] What is a netmask?\n  - [ ] What two syntaxes exist for expressing a netmask?\n\n- [ ] **ARP**\n  - [ ] Which layer of the OSI model do ARP belong to?\n  - [ ] What ARP table includes?\n  - [ ] Give an example of how ARP protocol works\n\n- [ ] **IP**\n  - [ ] What it defines?\n  - [ ] What is a routing table? When is it being used?\n\n- [ ] **TCP**\n  - [ ] What are the differences between TCP and UDP? When would you prefer to use TCP over UDP?\n  - [ ] What is a \"three-way handshake\"?\n\n- [ ] **UDP** \n  - [ ] Why UDP considered to be unreliable when compared to TCP?\n  - [ ] Which layer of the OSI model do UDP belong to?\n\n- [ ] **DHCP**\n  - [ ] How DHCP client locates the DHCP server?\n  - [ ] Does the client has to be on the same local network as the server?\n  - [ ] How the exchange between client and the server looks like? what steps are included?\n  - [ ] Is it using TCP or UDP?\n  - [ ] What ports is it using?\n\n- [ ] **Netowrk Devices**\n  - [ ] What is a HUB?\n  - [ ] What is a Switch?\n    - [ ] Forwarding table or (aka forwarding information base)\n  - [ ] What is a Router?\n  - [ ] What are differences between the three devices above?\n\n- [ ] **Delivery schemes**\n  - [ ] What is unicast?\n  - [ ] What is multicast?\n  - [ ] What is broadcast?\n    - [ ] What is the broadcast address?\n    - [ ] What protocols use broadcast?\n    - [ ] What is a broadcast domain?\n\n- [ ] **VLAN**\n  - [ ] What are the advantages of using VLAN?\n  - [ ] What is the range for VLAN IDs?\n  - [ ] What is a trunk port?\n\n- [ ] **Topologies**\n  - [ ] Bus topology\n  - [ ] Ring topology\n  - [ ] Star topology\n  - [ ] Tree topology\n\n- [ ] **IP**\n  - [ ] What layer of OSP model it belongs to?\n  - [ ] IPv4\n  - [ ] IPv6\n    - [ ] Why IPv6 is needed if IPv4 exists?\n\n- [ ] **ICMP**\n  - [ ] What is it used for?\n  - [ ] How do you use it?\n\n- [ ] **NAT**\n  - [ ] What is it and why it's needed?\n  - [ ] What is SNAT? how it is different from NAT?\n  - [ ] What is DNAT? how it is different from SNAT?\n\n## Intermediate\n\n- [ ] **Routing**\n  - [ ] Distance Vector Routing\n    - [ ] \"Count-to-infinity\" problem\n  - [ ] Dijkstra's algorithm\n\n- [ ] **TCP**\n  - [ ] Silly window syndrome\n    - [ ] Nagle's algorithm\n    - [ ] Clark's solution\n\n## Linux Networking\n\n- [ ] **Commands**\n  - [ ] ping\n  - [ ] ip addr\n  - [ ] ip neigh\n  - [ ] ip link\n  - [ ] ip route\n  - [ ] ip netns\n  - [ ] ethtool\n  - [ ] netstat\n  - [ ] traceroute\n  - [ ] tcpdump\n  - [ ] mtr\n\n- [ ] **ARP**\n  - [ ] How to view the contents of the ARP cache?\n  - [ ] How to initiate an ARP request manually?\n\n- [ ] **MTU**\n  - [ ] How to see what is the MTU of a given interface?\n  - [ ] How do you set MTU?\n\n- [ ] **Namespaces**\n  - [ ] How to create & delete namespaces?\n  - [ ] How to enter a specific namespace?\n\n- [ ] **Routing*\n  - [ ] How to display the routing table?\n  - [ ] How to print the IP address of each router that an IP packet traverses along its path to its destination?\n\n- [ ] **ICMP**\n  - [ ] How do you use it?\n\n## Academic Level\n\n- [ ] **Digital Modulation**\n  - [ ] What is Amplitude Shift keying?\n  - [ ] What is Frequency Shift Keying?\n  - [ ] What is Phase Shift keying?\n\n- [ ] **Multiplexing**\n  - [ ] What is it used for and how it works?\n\n- [ ] **Wireless networks**\n  - [ ] The hidden terminal problem\n  - [ ] The exposed terminal problem\n  - [ ] **MACA**\n    - [ ] What is RTS?\n    - [ ] What is CTS?\n\n- [ ] **ALOHA**\n  - [ ] Pure\n  - [ ] Slotted\n\n- [ ] **Noisy-channel coding theorem**\n  - [ ] Shannon capacity/limit\n\n- [ ] **CSMA**\n  - [ ] The Binary Exponential Backoff Algorithm\n\n- [ ] **Stop and Wait**\n\n- [ ] **The adaptive tree walk**\n\n- [ ] **Cryptography**\n  - [ ] Symmetric\n  - [ ] Asymmetric\n  - [ ] Digital Signature\n\n- [ ] **Error Detection**\n  - [ ] Parity Bit\n  - [ ] Checksum\n  - [ ] CRC code\n\n- [ ] Data transmission\n  - [ ] Piggybacking\n\n- [ ] **Error Correction**\n    - [ ] Hamming Code\n    - [ ] Reed Solomon code\n    - [ ] The NASA binary convolutional code\n    - [ ] Low-Density Parity Check code\n\n## Cloud Networking (OpenStack Neutron)\n\n- [ ] **Components**\n  - [ ] API Server\n  - [ ] Plugins\n  - [ ] Agents\n  - [ ] Messaging Queue\n\n- [ ] **Entities**\n  - [ ] Network\n  - [ ] Router\n  - [ ] Port\n  - [ ] Subnet\n  - [ ] Security Group\n\n- [ ] **ML2 Plugin**\n  - [ ] What is it used for?\n  - [ ] Explain type drivers\n  - [ ] Explain mechanism drivers\n\n- [ ] **dnsmasq***\n\n- [ ] **Floating IP**\n\n- [ ] **Provider Network**\n"
  },
  {
    "path": "resources/openshift.md",
    "content": "## OpenShift\n\n### Learn OpenShift\n\nName | Comments\n:------ |:--------:\n[Learn OpenShift](https://learn.openshift.com) | Interactive way to learn OpenShift\n\n### CheatSheet\n\n* Login: `oc login -u my_user -p my_password`\n\n#### Projects\n\n* Highlevel overview of the project: `oc status`\n* Create a new project: `oc new-project my_project`\n* List projects: `oc get projects`\n\n#### Accounts\n\n* The username of the user currently logged in: `oc whoami`\n\n* Add to user \"user1\" the ability to view the project \"wonderland\": `oc adm policy add-role-to-user view user1 -n wonderland`\n* Add a user as admin to the project \"wonderland\": `oc adm policy add-role-to-user admin some_user -n wonderland`\n\n* Get a list of all context whihc have ever been created: `oc config get-contexts`\n* Check what is the current context: `oc whoami --show-context`\n* The OpenShift server currently used: `oc whoami --show-server`\n* Get the list of all the OpenShift clusters you have ever logged into: `oc config get-clusters`\n\n#### Pods\n\n* List pods: `oc get po`\n* List pods with with node info: `oc get po -o wide`\n\n#### Deployments\n\n* List deployments: `oc get deployments`\n\n#### Service Accounts\n\n* List service accounts: `oc get serviceaccounts`\n\n#### Misc\n\n* Login: `oc login --token=<TOKEN> --server=https://<ADDRESS>:<PORT>`\n"
  },
  {
    "path": "resources/opensource.md",
    "content": "## Open Source - Resources\n\nName | Comments\n:------|:------:\n[Benevolent dictator for life](https://en.wikipedia.org/wiki/Benevolent_dictator_for_life) | \n"
  },
  {
    "path": "resources/openstack.md",
    "content": "# OpenStack\n\n## Checklist\n\n- [ ] **Components & Services**\n  - [ ] Neutron (Networking)\n  - [ ] Cinder (Storage)\n  - [ ] Nova (Compute)\n\n- [ ] **Neutron**\n  - [ ] Agents\n  - [ ] Server\n\n## Resources\n\nName | Description\n:------|:------:\n[OpenStack Operator Tools](https://github.com/openstack/osops-tools-contrib) | Tools and scripts for neutron, nova, etc.\n[Infrared](https://github.com/redhat-openstack/infrared) | OpenStack Deployment with Ansible\n\n## Cheatsheet\n\n### Infrared\n\n* Add plugin: `infrared plugin add <plugin_path>`\n* Add plugin when the spec not in root directory: `infrared plugin add <plugin_path> --src-path infrared_plugin`\n* Remove plugin: `infrared plugin remove <plugin_name>`\n\n* Import workspace (= inventory update): `infrared workspace import https://my-jenkins.com/job/blipblop/1/artifact/workspace.tgz`\n* List nodes in inventory/workspace: `infrared workspace node-list`\n\n* If you get error when executing external plugin: \"no such role\" then you need to create a soft link: \n```\nmkdir ~/.infrared/plugins/<plugin_name>/infrared_plugin/roles\nln -s ~/<plugin_name> ~/.infrared/plugins/<plugin_name>/infrared_plugin/roles/<plugin_name>\n```\n\n### Servers\n\n* List openstack instances: `openstack server list`\n"
  },
  {
    "path": "resources/packer.md",
    "content": "## Packer\n\n### Official Resources\n\n* Site: https://www.packer.io\n* Docs: https://www.packer.io/docs\n\n### Why Packer\n\nI recommend reading [this discussion](https://www.reddit.com/r/devops/comments/ko3cwq/why_packer) on Reddit\n\n### Blogs Posts\n\n* Getting started with Packer: https://codingshell.com/packer-getting-started\n"
  },
  {
    "path": "resources/prometheus.md",
    "content": "## Prometheus\n\n### Exporters\n\nName | Comments\n:------|:------:\n[Blackbox Exporter](https://github.com/prometheus/blackbox_exporter) |  \"blackbox probing of endpoints over HTTP, HTTPS, DNS, TCP and ICMP\"\nElastic_Search Exporter | Prometheus exporter for various metrics about ElasticSearch, written in Go. \n\n**How to Setup Prometheus in Kubernetes cluster for Elastic Search ?**\n\nTo Setup Prometheus in Kubernetes so that we can monitor third party applications like elastic search , we will make use of elastic search exporter which was written in go language and it is now maintained by Prometheus Community. Elastic search exporter will run as a side car container alongside Elastic Search.\n\nnow to run as side car container we will just use its image and specify the port:\n\n```\n containers:\n      - name: exporter\n        image: justwatch/elasticsearch_exporter:1.0.2\n        ports:\n        - containerPort: 9108\n        args: ['-es.uri=http://localhost:9200']\n ```       \n\nNow we will also create a config file for Prometheus:\n\n```\napiVersion: v1\ndata:\n  prometheus.yml: |-\n    global:\n      scrape_interval: 15s \n      evaluation_interval: 15s \n    scrape_configs:\n      - job_name: \"prometheus\"\n        static_configs:\n          - targets: [\"localhost:9090\"]\n      - job_name: \"exporter\"\n        static_configs:\n          - targets: [\"elasticsearch:9108\"]\nkind: ConfigMap\nmetadata:\n  name: pr-conf\n  namespace: elk\n```  \n  \n  \nHere make sure you either write your own namespace or create this elk namespace before running.\n\nNow next we will write a deployment file for Prometheus.\n```\n\napiVersion: apps/v1\nkind: Deployment\nmetadata:\n  namespace: elk\n  labels:\n    app: pr\n  name: pr\nspec:\n  replicas: 1\n  selector:\n    matchLabels:\n      app: pr\n  template:\n    metadata:\n      creationTimestamp: null\n      labels:\n        app: pr\n    spec:\n      containers:\n      - image: prom/prometheus\n        name: prometheus\n        ports:\n        - containerPort: 9090\n        volumeMounts:\n        - name: data\n          mountPath: /etc/prometheus\n      volumes: \n      - name: data\n        configMap:\n          name: pr-conf\n          items:\n              - key:  prometheus.yml\n                path: prometheus.yml\n\n        \n```\n\n​        \n​        Now we will create a service file for nodePort , so that we can access it through our browser through our <node-ip>:<node-port>  \n\n```\n apiVersion: v1\nkind: Service\nmetadata:\n  name: prometheus\n  namespace: elk\n  \nspec:\n  selector: \n    app: pr\n  type: NodePort  \n  ports:\n    - port: 9090\n      targetPort: 9090\n      nodePort: 32200\n```\nNow we will be able to access the prometheus on 32200 which is our nodePort  so start all the services one by one and you're good to go and also we can use Grafana for visualisation too along with prometheus.\n\n"
  },
  {
    "path": "resources/python.md",
    "content": "# Python\n\n## Learning Python (Courses)\n\nName | Description\n:------|:------:\n[Introduction To Python Programming](https://www.udemy.com/course/pythonforbeginnersintro) | Free, Udemy\n[Python for Beginner](https://www.udemy.com/course/python-hackcc) | Free, Udemy\n[Learn Python](https://www.youtube.com/watch?v=rfscVS0vtbw&feature=emb_logo) | Free, freeCodeCamp\n[Learn Python from Scratch](https://www.educative.io/courses/learn-python-from-scratch) | Free, educative.io\n\n## Learning Python (Tutorials and Interactive platforms)\n\nName | Description\n:------|:------:\n[HackerRank](https://www.hackerrank.com) | Python challenges/exercises\n[LeetCode](https://leetcode.com) | Python challenges/exercises\n[Exercism](https://exercism.io) | Python challenges/exercises\n[py4e](https://www.py4e.com) | Python lessons and materials\n[W3 Python](https://www.w3schools.com/python) | Python tutorial\n[Mode Python Tutorial](https://mode.com/python-tutorial) | Python Tutorial\n[Khan Academy](https://www.khanacademy.org) | Python courses and lessons\n[Python resources for everybody](https://learnbyexample.github.io/py_resources) | Python written resources\n\n## Modules\n\nName | Description\n:------|:------:\nsubprocess | Spawn new processes (=execute commands).\nagithub | Rest API client for rapid prototyping on any rest API.\n\n## Books\n\nAuthor | Title | Comments\n:------ |:------:|:--------:\nJeff Knupp | Writing Idiomatic Python 3.3 | |\nDane Hillard | [Publishing Python Packages](https://www.manning.com/books/publishing-python-packages) | |\nYong Cui | [Python How-To](https://www.manning.com/books/python-how-to) | |\nPael Anni | [Let's Talk Python](https://www.manning.com/books/lets-talk-python)) | |\n\n## Articles, Tutorials & Blog Posts\n\nAuthor | Article | Level | Comments\n:------ |:------|:--------:|:--------:\nMegha Mohan | [Mutable vs Immutable Objects in Python](https://medium.com/@meghamohan/mutable-and-immutable-side-of-python-c2145cf72747) | Beginner | |\nKenneth Reitz | [The Hitchhiker’s Guide to Python](http://docs.python-guide.org/en/latest) | | |\nKenneth Reitz | [Serialization](https://docs.python-guide.org/scenarios/serialization/) | | |\n\n## Libraries, Frameworks & Tools\n\nName | Description\n:------:|:-------:\n[Natural Language Toolkit](https://www.nltk.org)  | Platform for language processing programming |\n[Flask](http://flask.pocoo.org)  | Web microframework based on Werkzeug, Jinja 2\n[Django](https://www.djangoproject.com)  | Web framework with batteries included\n[Mypy](http://mypy-lang.org) | Static type checker\n[Pandas](https://pandas.pydata.org) | \"open source data analysis and manipulation tool\"\n\n### Cheat Sheet\n\n#### YAML\n\n* Read YAML\n```\nwith open(r'/file/path') as file:\n```\n\n#### Files\n\n* Read remote file(s)\n\n```\nfrom fabric import Connection\n\nwith Connection(host) as conn:\n    with conn.sftp().open(path) as stream:\n    ...\n```\n\n#### Dictionaries\n\n* Define dictionary: `some_dict = {'first_number': 2017, 'second_number': 2022}`\n* Add item to dictionary: `some_dict['third_number'] = 1991`\n* Remove last item: `some_dict.popitem()`\n* Remove item by key: `some_dict.pop(\"third_number\")`\n* Get all keys without values: `some_dict.keys()`\n* Get all values without keys: `some_dict.values()`\n* Access item: `some_dict['first_number']` or `some_dict.get('second_number')`\n* Number of items in the dictionary: `len(some_dict)`\n* Update value of a certain key: `some_dict.update({\"first_number\": 02017})\n\n## Python Checklist\n\n<div align=\"center\"><img src=\"images/python_map.png\"></div><hr/>\n\n## Checklist\n\n- [ ] **Data Types**\n  - [ ] Numbers (int, long, float, complex)\n  - [ ] List\n  - [ ] Dictionary\n  - [ ] String\n  - [ ] Tuple\n\n- [ ] **Mutability**\n  - [ ] What data types are mutable?\n  - [ ] What data types are immutable?\n\n- [ ] **PEP8**\n  - [ ] What is it?\n  - [ ] Give an example of three coding conventions Python developers should always follow\n\n- [ ] **Errors & Exceptions**\n  - [ ] How do you handle exceptions?\n\n- [ ] Iterators\n  - [ ] What 'enumerate' is used for?\n\n- [ ] **List Comprehensions**\n  - [ ] Is it better than for loop? If yes, why?\n  - [ ] How to perform list comprehensions for nested lists?\n\n- [ ] Data serialization\n    [ ] How you do with Python?\n\n- [ ] **Type Annotations**\n\n- [ ] **Dataclass**\n\n- [ ] **What's that?**\n  - [ ] What _ is used for in Python?\n\n- [ ] Meta-programming\n\n- [ ] Descriptors\n\n- [ ] Decorators\n\n [ ] Context Managers\n\n- [ ] Buffering Protocol\n"
  },
  {
    "path": "resources/security.md",
    "content": "## Security\n\n### Books\n\n* [building_secure_and_reliable_systems](https://static.googleusercontent.com/media/sre.google/en//static/pdf/building_secure_and_reliable_systems.pdf)\n"
  },
  {
    "path": "resources/sql.md",
    "content": "# Linux\n\n## Checklist\n\nCheck your Linux educational progress with the following list:\n\n- [ ] **Join**\n\n\n## Interview Questions\n\n* What is the difference between a hash join to merge join?\n\n* How to compare the performance of inner loop a \"when\" statement and analytic\n  function when those are interchangeable?\n"
  },
  {
    "path": "resources/terraform.md",
    "content": "# Terraform\n\n## Terraform Resources\n\nhttps://project-awesome.org/shuaibiyy/awesome-terraform\n[Terraform Script Builder](https://terraparty.com) | export valid Terraform scripts using UI\n\n## Projects\n\nName | Comments\n:------|:------:\n[terraformer](https://github.com/GoogleCloudPlatform/terraformer) | \"A CLI tool that generates tf/json and tfstate files based on existing infrastructure (reverse Terraform).\"\n[terraforming](https://github.com/dtan4/terraforming) | \"Export existing AWS resources to Terraform style (tf, tfstate)\"\n[terrascan](https://github.com/tenable/terrascan) | \"Detect compliance and security violations across Infrastructure as Code to mitigate risk before provisioning cloud native infrastructure\"\n[terraform-kvm](https://github.com/dmacvicar/terraform-provider-libvirt) | \"It's a one Terraform Provider for KVM\"\n\n### Best Practices\n\n#### tfstate\n\n* Don't edit it manually. tfstate was designed to be manipulated by terraform and not by users directly.\n* Store it in secured location (since it can include credentials and sensitive data in general)\n* Backup it regularly so you can roll-back easily when needed \n* Store it in remote shared storage. This is especially needed when working in a team and the state can be updated by any of the team members\n* Enabled versioning if the storage where you store the state file, supports it. Versioning is great for backups and roll-backs in case of an issue.\n\n### Cheat Sheet\n\n* Initialize terraform: `terraform init`\n* See what Terraform will execute if running apply: `terraform plan`\n* Apply Terraform configuration/code: `terraform apply`\n* Cleanup, remove Terraform generated resources: `terraform destroy`\n\n* Dependency graph: `terraform graph`\n\n#### State\n\n* Show state: `terraform show`\n* List resources from the state: `terraform state list`\n* Rename resource: `terraform state mv`\n* Import existing infrastructure into state: `terraform import`\n\n#### Variables\n\n* Pass var: `terraform -var`\n* Pass vars file: `terraform -var-file`\n\n* List all outputs: `terraform output`\n* Get specific output: `terraform output <OUTPUT_VAR>`\n\n#### Data sources\n\n* Get data: `data.<PROVIDER_AND_TYPE>.<NAME>.<ATTRBIUTE>`\n\n#### Backends\n\n* Use variables in a remote backend: `terraform init -backend-config=some_backend_partial_conf.hcl`\n\n#### Workspaces\n\n* Create a new workspace: `terraform workspace new <WORKSPACE_NAME>`\n* Show current workspace: `terraform workspace show`\n\n#### Productivity\n\n* Console: `terraform console`\n"
  },
  {
    "path": "resources/vault.md",
    "content": "## Cheat Sheet\n\n* Start Vault server in dev mode (NOT FOR PRODUCTION): `vault server -dev`\n* Set up vault client (dev mode):\n\n```\nexport VAULT_ADDTR='http://<VAULT_ADDRESS>'\necho <KEY> > unseal.key\nexport VAULT_DEV_ROOT_TOKEN_ID=...\nvault status\n```\n\n* Write a secret using one pair of key, value: `vault kv put secret/hello foo=bar`\n* Write a secret using multiple pairs of key, value: `vault kv put secret/hello foo=bar y=x`\n\n* Get a secret: `vault kv get secret/hello`\n"
  },
  {
    "path": "resources/victoriametrics.md",
    "content": "## VictoriaMetrics\n\n### Official Resources\n\n* Site: https://victoriametrics.com\n* Docs: https://docs.victoriametrics.com\n\n### Why VictoriaMetrics\n\nI recommend to read [case stadies](https://docs.victoriametrics.com/CaseStudies.html) from companies who use VictoriaMetrics in production and [Prominent features](https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html#prominent-features).\n\n### Blogs Posts\n\n* [Getting started with VictoriaMetrics](https://victoriametrics.com/blog/victoriametrics-monitoring/)\n* [How to Choose a Scalable Open Source Time Series Database: The Cost of Scale](https://victoriametrics.com/blog/the-cost-of-scale/)\n* [Benchmarking Prometheus-compatible time series databases](https://victoriametrics.com/blog/remote-write-benchmark/)\n"
  },
  {
    "path": "resources/virtual_machines.md",
    "content": "## Vagrant Resources\n\nName | Description\n:------|:------:\n[Official Docs](https://www.vagrantup.com/intro/index.html) | Multi-container applications\n\n## Vagrant Commands\n\nBe familiar with the following commands:\n\n    vagrant up\n    vagrant ssh\n"
  },
  {
    "path": "resources/virtualization.md",
    "content": "## Virtualization\n\n### Virsh CheatSheet\n\n* List VMs: `virsh list --all`\n* Shutdown VMs: `virsh shutdown <VM_NAME>`\n* Delete VMs: `virsh undefine <VM_NAME>`\n\n* List pools: `virsh pool-list`\n* Delete pool content: `virsh pool-delete <POOL_NAME>`\n* Deactivate pool: `virsh pool-destroy <POOL_NAME>`\n* Delete the pool object: `virsh pool-undefine <POOL_NAME>`\n\n* List networks: `virsh net-list`\n* Delete network: `virsh net-undefine <NETWORK_NAME> && virsh net-destroy <NETWORK_NAME>`\n\n### Vagrant Resources\n\nName | Description\n:------|:------:\n[Official Docs](https://www.vagrantup.com/intro/index.html) | Multi-container applications\n\n#### Vagrant Cheatsheet\n\n* Initialize vagrant file using Fedora image: `vagrant init fedora/33-cloud-base`\n* Bring up the VM: `vagrant up`\n* SSH into the machine: `vagrant ssh`\n* Shutdown: `vagrant halt`\n* Delete the VM: `vagrant destroy`\n* Reload Vagrant configuration: `vagrant reload`\n"
  },
  {
    "path": "roadmaps.md",
    "content": "## Roadmaps\n\n<div align=\"center\"><img src=\"images/linux_map.png\"></div><hr/>\n\n<div align=\"center\"><img src=\"images/python_map.png\"></div><hr/>\n\n<div align=\"center\"><img src=\"images/jenkins_map.png\"></div><hr/>\n\n<div align=\"center\"><img src=\"images/terraform_map.png\"></div><hr/>\n\n"
  }
]