Repository: karpathy/micrograd
Branch: master
Commit: c911406e5ace
Files: 10
Total size: 68.9 KB

Directory structure:
gitextract_jag07i_y/

├── .gitignore
├── LICENSE
├── README.md
├── demo.ipynb
├── micrograd/
│   ├── __init__.py
│   ├── engine.py
│   └── nn.py
├── setup.py
├── test/
│   └── test_engine.py
└── trace_graph.ipynb

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
.ipynb_checkpoints/


================================================
FILE: LICENSE
================================================
The MIT License (MIT) Copyright (c) 2020 Andrej Karpathy

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


================================================
FILE: README.md
================================================

# micrograd

![awww](puppy.jpg)

A tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode autodiff) over a dynamically built DAG and a small neural networks library on top of it with a PyTorch-like API. Both are tiny, with about 100 and 50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. Potentially useful for educational purposes.

### Installation

```bash
pip install micrograd
```

### Example usage

Below is a slightly contrived example showing a number of possible supported operations:

```python
from micrograd.engine import Value

a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}') # prints 24.7041, the outcome of this forward pass
g.backward()
print(f'{a.grad:.4f}') # prints 138.8338, i.e. the numerical value of dg/da
print(f'{b.grad:.4f}') # prints 645.5773, i.e. the numerical value of dg/db
```

### Training a neural net

The notebook `demo.ipynb` provides a full demo of training an 2-layer neural network (MLP) binary classifier. This is achieved by initializing a neural net from `micrograd.nn` module, implementing a simple svm "max-margin" binary classification loss and using SGD for optimization. As shown in the notebook, using a 2-layer neural net with two 16-node hidden layers we achieve the following decision boundary on the moon dataset:

![2d neuron](moon_mlp.png)

### Tracing / visualization

For added convenience, the notebook `trace_graph.ipynb` produces graphviz visualizations. E.g. this one below is of a simple 2D neuron, arrived at by calling `draw_dot` on the code below, and it shows both the data (left number in each node) and the gradient (right number in each node).

```python
from micrograd import nn
n = nn.Neuron(2)
x = [Value(1.0), Value(-2.0)]
y = n(x)
dot = draw_dot(y)
```

![2d neuron](gout.svg)

### Running tests

To run the unit tests you will have to install [PyTorch](https://pytorch.org/), which the tests use as a reference for verifying the correctness of the calculated gradients. Then simply:

```bash
python -m pytest
```

### License

MIT


================================================
FILE: demo.ipynb
================================================
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  MicroGrad demo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from micrograd.engine import Value\n",
    "from micrograd.nn import Neuron, Layer, MLP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(1337)\n",
    "random.seed(1337)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x10f636400>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAEyCAYAAACVsznTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VFX6wPHvmZJJJiGQhNB7C0iVqoCKAiIgAiqCiqKyshYsu7qKrq4/62Jd7IoKCqKAsDakF3UVpUqXEppAaCFAyiSZdn5/ZIgZkkDKJHcm836eJ09y75x775tJ8ubce5rSWiOEEKLkTEYHIIQQoUYSpxBClJIkTiGEKCVJnEIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUrJYnQAZVGzZk3dpEkTo8MQQlQx69atS9VaJ56vXEgmziZNmrB27VqjwxBCVDFKqf0lKSe36kIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUpJEqcQQpSSJE4hhCglSZxhzOPxsmfPSY4ezTQ6FCFCSkiOHBLld+RIJpdf/gl//HGanBw3JpOiRg0bEyf2Y+zYzkaHJ0RQkxpnmLrttq9ITk7D4XDh9Wrcbi+pqdncf/9CFi/ebXR4QgQ1SZxhasOGI7jd3kL7HQ4XX3+93YCIhAgdkjjDVPPmcZhMqtB+q9VEzZp2AyISInRI4gxTH388jISEKKKjrfn7IiJMJCTYGT++u4GRCRH8pHEoTLVsmUBy8v2sW5fC4cMZ7N17ipiYCEaP7kBCgtQ4hTgXSZxhLDbWxuWXN83f3rEjldWrD9G6dU2aNo0zMDIhgpvcqocJrTVvvLGKZs1ep3nz1/nww/V+r0+c+BMXXvg+o0bNpW3bd5g2bUOpr3HqVA4//rifLVuOobUOVOhCBB0Vir/gXbt21TIDfOl8+OF6HnhgIQ6HCwC73cqUKdcwcmQ7du9Oo337d8nOdueXj4y0cOTIQ1SvHlmi8//222GuuGIaWmtcLi/XXdeGTz4ZhlKFG6CECFZKqXVa667nKyc1zjAxZcpv+UkT8rodffxxXq1y375TRESY/cpbLCYOHy75iKLrr/+CU6dyOH06F4fDxeefb2Hu3N8DE7wQQUYSZ5iIiYkoYp8NgDZtEnG5/Pt0mkzQqFH1Ep9///5Tfttut5f33ivfXUFGRi4rVx5gx47Ucp1HiECTxBkmnnnmcuz2P7seRUdbeeKJSwCoV68a06cPJyrKQnS0ldhYG998c6Nf+fOJjCzcznjwYHqZ49206ShNmrzOwIEzuPDC97n99q/kuakIGtKqHiYuuqgBK1fewdSpGzCZFHfe2Zk2bf5cBfXaa9tw4sQjHDmSSb161bDZSverceWVzfnyS/8RRx061C5zvCNGfEFaWnb+9hdfbGPo0NYMG9a6zOcUIlAkcYaRjh3rMGnSVcW+HhVlLXM3pLffHsTPP/9BZqYL0NjtEbz8cv8yRpr33LWg3FyP3LKLoCGJUwRE3brV2LHjPhYs2IXWcNVVLYiPj8Lt9vLDD/vIyHDSs2dDatWKLtH5WraMZ9u245y5O7fZzLRrV6sCvwMhSk4SpwiYGjUiufHG9vnbTqeHPn0+ZvPmY/nj4n/44TY6dapz3nPNmXMDffp8jMPhwun0MHZsZwYNallhsQtRGpI4RYX58MP1bNhwxK9/6JgxX7Fx413nPbZ165rs2/cgu3adID4+ivr1YysyVCFKJSCt6kqpKUqpY0qpLcW8rpRSbyilkpVSm5RSnQu8NkYptcv3MSYQ8YjgsHfvKb+kCcW3tDudnkKt5pGRFtq3ry1JUwSdQHVH+hgovtUBBgItfR/jgHcBlFLxwFNAD6A78JRSSgZJVxEXX9zAb/Ylq9VE9+71/MocOZJJ166TiYp6nujoF5g69bfKDlOIUgtI4tRa/wiknaPIUGCazvMrUEMpVRcYACzRWqdprU8CSzh3AhYhZPjw1tx3X3csFhMREWbatq3FtGnD/cpce+0sNm48iteryc52M378AlavPmRQxEKUTGU946wPHCiwfdC3r7j9hSilxpFXW6VRo0YVE6UIKKUU//53P5588jKys13Ex0cVGru+Zk2K30z0breXn3/+g+7di/w1ECIoVNbIoaJmetDn2F94p9aTtdZdtdZdExMTiyoigpTdbiUhwV7khB9xcf6TiFitJurUiams0IQok8pKnAeBhgW2GwAp59gvwsTUqUOx2/OGesbERNC5c11GjGhrdFhCnFNl3ap/A4xXSs0kryHotNb6sFJqEfBCgQahK4HHKimmkJSSksGmTUepX78a7duXfUhjsBg8uBXr1/+Vn376g5o17Qwe3AqLpWT/z7dvT2XVqoPUqRND//7Ni1xDSYiKEJDEqZT6HOgD1FRKHSSvpdwKoLV+D5gPDAKSAQdwu++1NKXUs8Aa36me0Vqfq5EprC1YsIsRI77AYjHhcnkZN64L//nPAKPDKrekpJokJdUs1TFz527j1lu/RCmFUoo+fRrz9dc3SvIUlUImMg4RXq+mevWJZGY68/dFR1tZuvRWLrqogYGRVT6tNbGx/u9FTIyV22/vxOzZ28jN9XDjje14/fWrsFrN5ziTEP5kIuMqJj09l9xc/87kJpNiz56TBkVkHJfL6zcp85l977+/nqNHszh1KoePP97AhAnLDIpQVHWSOENE9eo24uKi/Pa53d5ST93m9Wpef/1XrrxyOnfc8TUpKRmBDLNSRESYad26pt9tudvtxen05G9nZ7uZO3ebEeGJMCCJM0QopVi48GZq1sxbC91mM/Paa1eWesagv/1tIY8/vpwlS/YwffpGLrzwfb95L0PF/Pk3kZSUgNmsiIqyMHBgi0KNSiVdL0mI0pJnnCHG5fJw4EA6iYl2qlWzlepYr1cTGfmc3zIZ0dFW3n57EGPGdAp0qJUiO9tFZKSFo0ez6NDhXU6fzsXj8WKzWZg370a/5Y+FOJ+SPuOU2ZHOw+vVvPbaL8yevZWEBDsvvtivXDObl5fVaqZZs7IP5z/7/6TW4PGE3j/PM6Ki8sbC16kTw5Yt9zB9+kYcDhdDh7Y29OckqjZJnOfx+OPLePPN1TgcLpSCn376g40b7ypX8jKKyaQYPbo9s2dvxeFwYzIpbDYzgwdXjXkua9WK5qGHehodhggDkjjP47331ua34GoNublu5szZxiOP9KrQ6+7de5I5c7ZhMilGjmxHWlo2S5fuoXp1G6NGtSM6uvCqlSXxwQfX0KBBLAsWJFO/fjVefXUAtWvLEEchSkMS53mc3aFaqcL7Am3z5qP07DmF3Fw3SsG//rUCrzfvsYHFYuKll1ayfv24MiVPi8XEs89ewbPPXlEBkZdOZqaTjIxcateOkY7rIqRIq/p5PPRQz/xlck0mRVSUlVGj2lXoNSdMWEpWlhOXy4vT6cXhcJOT48bp9OBwuDhw4DQff7yhQmOoaM888wMJCS/SrNkbJCW9xYEDp40OSYgSkxrneTz+eG/q1o1h9uytJCba+b//60ODBhU7I3lqqqNQI05BubluUlMdFRpDRVqyZDcvvfQzTqcX8LJ370muv342q1bdaXRoQpSIJM7zUEpxxx0XcscdF1baNUeMaMuWLcfzn62aTAqTSeXPWxkZaaFfv2aVFk+grV2bQk7On6OgPB7Npk3HDIxIiNKRW/Ug9Pe/X8x993X3jRaK5LHHejNwYAtsNjPx8VG8//4QevUK3cmcGzeuQWSk///sunWlgUqEDukAH0BaaxYsSGbbtuO0bl2TwYNbFjl5b7jzeLxcc83n/PDDfsxmE1prFi++JewmKxHBRzrAG+D++xcwdeoGnE4PERFmbrmlA+++e7XRYQUds9nEvHk38fPPBzh5Mpvu3etLlygRUqTGGSD795+ideu3/Z7dRUZa2LLlbpo3jzcwMiFEScm0cpUsLS2biAj/uR8jIsycOBF6E2gIIc5NEmeAJCXVLJQ4rVYTTZrU4Pbbv6J27Vdo0+YtVqzYa1CEoS0lJYMtW4751eiFMIokzgCx2618//0YkpISsFhMtGwZz/LlYxg/fj4zZ27l2LEstm8/wdVXf87vvx83OtyQ8ve/L6JZs9fp2fMjGjeexPbtqUaHJMKcNA4FUNu2tdi+fbzfvm+/3elXS3K7vSxcmEybNrLEcUksWLCLyZPXkZvrITfXQ2amk2uvncW2bfcaHZoIY1LjrGCRkf637xaLiZiYsk3QEY62bDnmN7O71pCcLOv5CWNJ4qxgL77YP3+se0SEmVq17IwcWbFj3auSVq0SCj07btSoukHRCJFHbtUr2LhxXWjatAYLFiRTq1Y0d93VldjY0s3cHs6uuSaJG25oy8yZW7BazZjNirlzb6jw62qtmTjxJ15+eSVer2bcuC5MnNhPZnESgPTjFCFix45UTpzIpl27WpXyj+fjj3/j3nsX5M8XYLdbeeKJS3nssd4Vfm1hHBk5FKL27TvF3LnbUEoxcmRb6tev2JmYQkVSUs1Kvd4XX2zzW4LY4XAxd+42SZwCkMQZVLZuPcbFF3/kWz9d8cwzP7Bu3TgZeWSAmjWjMZnA++e6diQkRBV/gAgr0jgURB59dCmZmU6czrw1wjMynPzrXyuMDissPfXUZcTGRmKxmPKXIH7ppf5GhyWChCTOIHL2BMZer+b48dCdsDiUNWsWx733dkMpfDM4wVdfbTc6LBEkApI4lVJXKaV2KKWSlVITinj9P0qpDb6PnUqpUwVe8xR47ZtAxBOqrr/+gvyuS5DXIHHddRcYGFH4Ono0k1dfXelbvsRDTo6biRN/Zt++U+c/WFR55X7GqZQyA28D/YGDwBql1Dda621nymit/1ag/H1AwenUs7XWncobR1Xw979fzPHjWbz33jqUgr/97SLGjetsdFhh6fDhTCIiLOTk/Nn53mYzk5KSQZMmNQyMTASDQDQOdQeStdZ7AJRSM4GhwLZiyt8IPBWA61Y5JpPixRf78+KL8izNaC1aFG6Q83g0rVtXbuu+CE6BuFWvDxwosH3Qt68QpVRjoCmwvMDuSKXUWqXUr0qpYQGIR4hyi4mJYMGCm4mLiyQiwkxsrI1vvhlFfLy0rIvA1DiLGkpRXK/6UcAcrbWnwL5GWusUpVQzYLlSarPWenehiyg1DhgH0KhR6K63I0JHz54NOXHiEU6ezKFGjUgZNSTyBaLGeRBoWGC7AZBSTNlRwOcFd2itU3yf9wDf4//8s2C5yVrrrlrrromJMrOQqBxKKeLjoyRpCj+BSJxrgJZKqaZKqQjykmOh1nGlVBIQB/xSYF+cUsrm+7om0Ivin40GrcOHM5g7dxtLluzG4/Ge/wAhREgr96261tqtlBoPLALMwBSt9Val1DPAWq31mSR6IzBT+w+ObwO8r5TykpfEJxZsjQ8Fa9Ycom/faUDelGedOtVh+fJbsVrN5zlSCBGqZJKPMsrOdjFjxmYee2wZqal/dlK3261MmjSAO+/sYmB0QoiykEk+KlBOjpvu3T9kz56TfhNBQN5kEAcOpBsUmRCiMkjiLINZs7awd2/hpAl5Nc4ePYrsjSWCTGqqgylTfiMjI5drrkmiWzf5uYmSkcRZBidP5uB2F24EslhMPPJITwYPbmVAVKI0jh/Pon37dzl5MgeXy8Nrr/3C7Nkj5GcnSkQSZxlccUVTv+4pERFmevVqyPz5NxMZKW9pKHj//XWkpWXjcuX9A3Q43Dz44CJJnKJEZHakMujQoTazZl1PnToxREVZ6Nu3Kf/970hJmiEkr6bpf9eQkZFrUDQi1MhfehkNGZLE4cNJRochymjYsCTee28NDkfe0s1RURauu66NwVGJUCE1ThGWLrmkMVOnDqNhw1gSEqK49daO/Oc/VxkdlggR0o+zlLZuPcaGDUdo0qQGvXrJmHkhqhLpx1kBPvroN+67bz4WiwmvVzN6dAfee+9qo8MKW1prPvhgPbNmbSE+3s6zz14u076JSiE1zhLKznYRF/ciubl/Tuxkt1v58cfb6NKlXqXGIvI8//z/eOGF/+FwuFAqbyq4TZvulomGRZmVtMYpzzhL6MSJ7EIz5FgsJg4dyjAoIjFp0i/5gxC0huxsNzNnbin3eXfvTuOtt1bz0UfrSU+XlnZRmNyql1DdujHExtrIznbn73O7vXTsWNvAqMJbUTdLXm/57qBWrTpI377T8Hi8mEwmnn76B3777a8kJNjLdV5RtUiNs4TMZhNLltxCvXrVsFhMREdbmTnzOho3lttCo4wf3z1/cTulIDLSwqhR7cp1znvumU9WloucHA8Oh4sjRzKZNOnXQIQrqhCpcRYjJSWDp5/+gZSUdIYMSeLOOzvTvn1tDh78GxkZTmJiImRyW4M99dRlJCREMXPmVuLjo3jhhSto1iyuXOcsONMVgMvl5fDhzHKdU1Q9kjiLkJaWzYUXvkdaWjZut2b58n3s2XOSiRP7oZQiNtZmdIiCvNnZ77uvB/fd1yNg5xw4sAXTpm3MfyRjt1sZPLhlwM4vqga5VS/Cl1/+TmamC7c773mZw+Fi0qRfCcUeCKJ0Jk26iiFDkrBa8x7HPP10H4YPlxFFwp/UOIvgdnsLJcnyNjqI0BAZaWHWrOuNDkMEOalxFuHqq1thtZpRvkeYdruV0aM7oJQ80xRCSOIsUv36sfz661iuvLI5HTvW5sEHe/D++zJCSAiRR27Vi9GmTSILF442OgwRBJxODx6Pl6goq9GhiCAhNU4hiqG15r775hMd/TzVqv2bwYM/Izu78HIpIvxI4hSiGO+/v44pUzbgdms8Hs3y5Xt56KHFRoclgoAkTiGKsWzZHr8F+XJy3CxfvtfAiESwkMQpRDGaNIkjIsKcv20yKRo1qm5gRCJYSOIUohiPP96bBg2qUa1aBDExEVSvbuPttwcZHZYIAtKqLkQx4uKi2Lz5HhYtSsbp9NC3bzNq1pRZkoQkTiHOyW63ypBLUYjcqgshRCkFJHEqpa5SSu1QSiUrpSYU8fptSqnjSqkNvo+/FHhtjFJql+9jTCDiEUKIilTuW3WllBl4G+gPHATWKKW+0VpvO6voLK31+LOOjQeeAroCGljnO/ZkeeMSQoQfrTW7Fy3i+LZt1GzdmhYDB1bIHBOBeMbZHUjWWu8BUErNBIYCZyfOogwAlmit03zHLgGuAj4PQFxCiDCz6MEHWf/RR3hdLkxWKx3HjGHw228H/DqBuFWvDxwosH3Qt+9s1ymlNiml5iilGpbyWJRS45RSa5VSa48fPx6AsEUgJS9cyLfjxrHk0UfJOHzY6HBEGDr9xx+smzwZV1YWHqcTV1YWG6ZM4eSePQG/ViASZ1H14LMnr/wWaKK17gAsBT4pxbF5O7WerLXuqrXumpiYWOZgReBtmDqV2dddx/oPPuDX117jvY4dyTx61OiwRJhxpKZijojw22eOiMBx4kTArxWIxHkQaFhguwGQUrCA1vqE1vrMOqsfAF1Kemwoyc52sW5dCrt2nQir2eKX/fOfuBx5a/V43W5yT59m47RpBkclwk3N1q0xWf1nsDJZLCS2CXx3skAkzjVAS6VUU6VUBDAK+KZgAaVU3QKb1wC/+75eBFyplIpTSsUBV/r2hZzdu9No3vwNLr/8Ezp2fI+bbpobNrPGu3Ny/LY9bnd+Ig01J09m8913O1mxYi9ut/e85XfvTmPs2K8ZPnwms2dvrYQIw0P2yZP8NHEiSx55hH0//FCiY6x2O7d9/z3xLVuizGbimjfn1uXLiYiJCXh85W4c0lq7lVLjyUt4ZmCK1nqrUuoZYK3W+hvgfqXUNYAbSANu8x2bppR6lrzkC/DMmYaiUHPzzf/l6NGs/GT57bc7+eyzzYwe3cHgyCpe+5tv5rcpU3D7kqU1MpLWw4YZHFXp7diRSs+eU3C7PXi90Lp1TX788bZi5+H844/TdOkymYyMXLxeWLx4D8ePO7j33m6VHHnVknPqFO916EDWsWN4nE7WvP02V0+eTIebbz7vsbXateO+nTsrPMaA9OPUWs/XWrfSWjfXWj/v2/cvX9JEa/2Y1rqt1rqj1vpyrfX2AsdO0Vq38H1MDUQ8Rtix44RfDTMry8XWrccMjKjyDHjtNbrdfTfVGzemdocO3DhvHnU6djQ6rFIbO/YbTp7MJj3dSWamky1bjvHWW6uLLT99+kayslx4fRVTh8PF88//WEnRVl0bp03DkZqKx+kEwOVwsPihhwyOyp8MuQyQpKQE1qxJyU+e0dFW2ratZXBUlcNstXLlK69w5SuvGB1Kuezbd4qCj6Zzctzs2lX8DZDT6Sn0OKYkt/fi3JyZmXhc/hNGB9ujHxlyGSCffXYddepEExtrIyrKwjXXJHHTTe2NDkuUQo8eDYiI+PNPwm630qtXw2LLjxzZjqgoi1/5ceO6FFtelEyLgQMx22z525YgfPSjQrH1t2vXrnrt2rVGh1FITo6b338/TrVqNpo3j6uyq2K6HA48TieRNWoYHUpApaVlc+WV09my5Rher2bs2At5553B5/w5rlp1kEceWcrp0zmMGtWORx7phclUNX/ulWnXggUsuO8+ck+fptWQIQx+5x0skZEVfl2l1DqtddfzlpPEKUpKa83CBx5g7bvvglLU79aNm+bPJ7J61ZncV2vN8eMOIiMtxMbazn+AqFJKmjjlVl2U2Kbp0/ltyhS8bjdel4uUdeuYd9ddRocVULNnb+Xmm//LmDFfsn69jIASRZPGIVFi+374AVdWVv62JzeXAz//bGBEgfXRR+u5//6F+esMLVmyh19+GUv79rUNjkwEmypf4/R6NWvXpvDjj/vJzHQaHU5Ii2/e3P85k1JUb+jfeOLOyWH3kiUkL1yIMzOzkiMsnxdf/NlvcbasLBcffLDewIhEsKrSNU6Xy8PAgTP49deDmM0moqIsrFw5lmbN4owOLST1eOABts6axck9e1AmE8psZsiHH+a/nnPqFB/26EHG4cMopYioVo07V6+mWr16BkZdckWN9AqX0V+idKp0jfPdd9eycuUBsrJcpKfncvy4g9tu+8rosEJWRHQ0d65Zw4g5cxj68cfct3On3zjg7596ilP79uHMyCA3PZ3Mo0dZ8MADBkZcOg891BO7/c9RQna7lb/8pbOBEYlgVaVrnNu2HSc7252/7fVqkpNDckRn0DBHRNBiwIAiX0vduTN/tAeAdrtJS06urNDK7e67uxIVZeGjj34jJsbKU0/1oVOnOkaHJYJQla5xdutWz68GYbGY6NhR/hAqSuNLLsFq/3MVSEtkJA179jQwotK77bZO/O9/t7NgwWguuqiB0eGIIFWlE+ftt1/I8OGtsdnMREdbadYsjqlTh/qVSU5O45lnfuCZZ36Q2mg59XrkEVoMGoTJasUcEUGDnj258uWXjQ5LiIALiw7wKSkZOBwumjatgdn85/+KzZuP0rPnFLKz81pSo6KsrFx5h3Q/KafstDS8Hg/2mjWr7OgpUTVJB/gC6tWrRosW8X5JE+Bf/1pBVpYTj0fj8Wiyspw8+eQKg6KsOqLi44lOTJSkKaqsKt04dD5paTl+s+FoDSdP5hR/gAhrCxcms2BBMnXqRHPPPd2oXr3ix05XNVprMg4dwuNyUaNxY5QpNOtuYZ04b765PWvXpuR3eo6OtsqMRqIQj8fL3/++iPfeW4fT6cFmM/PBB+vZtOluYmIizn8CAYDH5WLW8OHsXbYMlKJWu3bcunQptthYo0MrtdBM9wFy552defLJS6ldO5pataJ54olLGTdO+u2JP7lcHq64YhpvvLEap9MDQG6uh2PHspgzpyQrYIszfn7pJfYuX447Jwd3djZHN21i4YMPGh1WmYRljfOTTzbyn//8gtls4p//vIQjRx42OiQRpKZO3cDatYXXD/R4tAzhLaVDv/6KOzs7f9uTm8uhNWvOcUTwCrvE+emnm7jnnu/yb89vueVLbDYzgwe3MjgyEYx2707zG79+hsmkuPLK5gZEFLoS27Zl99KleHyL+5ms1gpZgbIyhN2t+jvvrPH7Q3A4XLz7rsztKYp20UUNiI72X6zNbreycOHNtGqVYFBUoenSJ54gsU0bImJiiKhWjWr16zPwjTeMDqtMwq7GGRFhLrQvMjLs3oYyObJhA1+MGMGp/fuJb9mSkf/9LzWTkowOq0ING9aae+7pxqRJv2IyKVq2TGDp0luoXTvwS85WdRExMdy5ejUpa9fidbup26UL1qgoo8Mqk7DoAF/QsmV7GDLk8/wx7Ha7lRUrxtC9e/1Ahljl5KanM6lJE3JOnszboRTRtWrx4P79WGxVf6b0zEwnDoeLxES79E8NgD9++on548eTnZZGqyFDGPDaa0Hxe1TSDvBhV9Xq27cZixffwjvvrMFiMfHAAz3o0iU0pj0z0tHNm9HeAis4ao3L4SAtOZlabdsaF1gliYmJkK5HAZK6fTufDhiQv3LlhqlTcWVmMuyTTwyOrOTCLnEC9O7diN69GxkdRkiJio/He9aSrR6nk6j4eIMiEqFq53ff+S3/687OZtucOedMnBkpKSz5xz84uXcvzfr149Inn8RstRZbvqKFZeIUpZfYpg0XjBjBtjlz8OTmYrbZ6DJuHNXq1jU6NBFirFFRmCwWv3/E5ojia/M5p08zuUsXslJT0W43RzZs4MTOnVw/c2ZlhFskSZyixIZOnUrrYcM4sXMntdq3p+XAgUaHJEJQuxtv5MfnnsPhW/TPardzxXPPFVt+z9KlOLOy0O68dokzNVR3Tk6lLBlcFEmcosSUUrQeNszoMESIi4qL4+5Nm/j19ddxHDtGqyFDaHX11cWWD8bGOEmcZeTxeDl6NIv4+CjpziREKdlr1uSKZ58tUdlm/foRERODOzsbr9uN1W4naehQw2qbEKAO8Eqpq5RSO5RSyUqpCUW8/nel1Dal1Cal1DKlVOMCr3mUUht8H98EIp6KtnnzUerXf40WLd6gRo2JfPLJBqNDEqLKssXGMm7dOtrfdBONLrmEno88wvBp0wyNqdz9OJVSZmAn0B84CKwBbtRabytQ5nJgldbaoZS6G+ijtR7pey1Ta12q3sTl6cdZXlprGjT4DykpGfn7oqIsrF39MSFPAAAgAElEQVQ7jgsuSDQkJiFEYFTmRMbdgWSt9R6ttROYCfitT6G1XqG1dvg2fwVCdjGXU6dySE3N8ttnsZjYuPGIQREJISpbIBJnfeBAge2Dvn3FGQssKLAdqZRaq5T6VSlVbMuDUmqcr9za48ePly/icqhePRKLxX/YpteradSoukERCSEqWyASZ1FNXkXe/yulRgNdgYIreDXyVY1vAiYppYqcckZrPVlr3VVr3TUx0bhbYpNJMWPGtdjtVmJjbdjtVkaP7kCvXtKhXpSN1pqjRzPz174SwS8QzcEHgYYFthsAhSYwVEr1A/4JXKa1zj2zX2ud4vu8Ryn1PXAhsDsAcVWYYcNas3XrPWzYcIQGDWLp2lWGbIqiZWU5efjhxfz88wFatkzgjTeuon79P2c8P3DgNH37TuOPP07j9WqefvpyHnust4ERi5IIROOQhbzGob7AIfIah27SWm8tUOZCYA5wldZ6V4H9cYBDa52rlKoJ/AIMLdiwVBQjG4eEKCmtNZdd9jFr1hwiJ8eDxaKoXTuG7dvH549779btA3777TAeT97fod1uZd68G7n88qZGhh62Kq1xSGvtBsYDi4Dfgdla661KqWeUUtf4ir0MxABfnNXtqA2wVim1EVgBTDxf0jTS778f57vvdrJnz0mjQxEh4NixLFavzkuaAG63JiMjl5Ur/2wS2LTpaH7SBHA6PaxefajSYxWlE5Ce21rr+cD8s/b9q8DX/Yo5biUQEqujPffcj/z73//DYjHjcnl4993BjBnTyeiwgtqGqVNZ9vjjuHNyaDtyJAPffNPQiRkqm9ls4uwbOq3zemGcUadONH/8kZ6/bbOZady4RmWFKMoo7GaAL4tdu07wwgv/w+Fwk56eS3a2m7vu+o709NzzHxymkhctYv748WQeOULOqVNsnDaNJY88YnRYlapmTTtDhrTCbs/7Z2GzmWnQIJZevf5sEvjss+uIiYkgNtZGTEwEl13WmBEjLjAqZFFCMlawBPbvP01EhDl/8mPIqzUcOZJJbKzxk68Go+1ffpk/3yLkTczw+9y5XPWf/xgYVeWbOfN6Xn55JT/99AdJSQk89dRl2Gx//tn16tWIHTvGs3r1IeLjo+jduxEmU/CNzQ5WWmuSFywgdft2Ei+4gOYDBlTK2HZJnCXQpk1NXC6P3z6TSdGwYeitB11a2usFpUr9yxgVH583dZj7z382kdXDr6+rxWI6byt5vXrVGDasdSVFVLXMHz+ejZ98gtflwmS10nnsWK56/fUKv27Y3qq73V7efHMVY8Z8xSuvrMxfM7so9evHMn36tURFWbDbrVSvbmPevBuJiqq6z+s8Tidf3nILz9lsPB8VxdIJEyhND4weDzxAZHw85ogIlNmMxW6vlF9oYTxnZia56ennL1hOJ/fsYcOUKbiysvA4nbiyslg3eTKn9u+v8GuHZY1Ta831189myZI9OBwuoqIszJ+/i6VLby32Nunaa9uQlvYoR49mUrdutSIXfatKlj/xBNvmzs2rMbrdrH7zTeJbtKDzX/5SouNjatfmni1b2DhtGi6Hg6RrrqFOx44VHLUwktft5ssxY9g2ezYAzQcM4IY5cypsFiNHairmiAjcvuWGIW9C5OwTJ6jRuPE5jiy/sKxx7tt3isWLd+cvE5yd7Wb16kNs2XLsnMdFRlpo3LhGlU+aALvmz8ednZ2/7XI42DlvXqnOEZ2YSM+HHuKyJ5+UpBkGVr7yCju++gqv243X7WbvsmUse/zxCrte4gUXYLL41/1MVisJlbDyalgmzpwcd6GapcmkZMhbATF160KB55omi4XYBiE7N4uoBPtWrPBvEMzJYd8PP1TY9SJiYhizYgVxzZujzGbiW7ZkzIoVRERHV9g1zwjLW/WWLROoXz+WvXtP4nJ5MZsVNWpE0qFDbaNDCxoDX3+djy6+OO9WXSls1apx6ZNPGh2WCGJxLVpgWrEify0hZTYT17RiR0DV7tCB+5OTK/QaRQm7ddXPOHYsizvv/IYNG47Spk1NPvzwGho0qPqt5KWRfugQyQsWYLJYSBo6lKi4OKNDEkHMceIEH3TrhiM1FYCI6GjuXLMmpO5USjrkMmwTpxAi8FwOB3uXL8fr8dCkT5+Q64JW0sQZlrfqongel4tNn35K+oED1O/RgxYDBpT5XBkpKeSmpxPXvHlYDbUMZ1a7/ZwLr1UVkjhFPq/Hw/T+/UlZswZXdjbWqCh6P/44l/7zn6U6j9aaeX/9KxunTcNktWJPSOD2H3+keiOZs1RUDWHZqi6Ktm/FCg6vW5fXMqo1LoeDH55+Gndu6cbkb509m82ffYYnNxdXZibpBw8y96abKihqISqfJE6RL+fUKb8uSAAo5dfFpCSObNiAK+vPdZm0x8OxLVsCEaIQQUESp8jXsGdPCs6DpiwWaiYlEVmjdNOc1WzdGmvBvnRKEde8yBVRhAhJkjhFvmr16nHLkiXEtWhBREwMjXr14pbFi0s9wUeH0aNp1rcvVrsdW2ws9oQErpsxo4KiFqLySXckUSG01hzdtInc9HTqdOqErVo1o0MKapMnr+P//u97nE4Pt9/eiYkT+2E2m9i//xS33fY127en0q5dLT7+eKjfmkUisKQ7kjCUUkrGp5fQ119v529/W5Q/d8I776wlOjqCRx/tRe/eUzh8OBOPR3P8eBa9e09lx47xYTFfQjCTW3UhDDZr1tb8pAngcLiYOXMLW7Yc4/Tp3Pw1iTweTWqqg507TxgVqvCRGud5pKVl8+abqzh6NItBg1py9dWtjA5JVDHx8VGYzcpv0bbq1SOx261++wA8Hm/+UhzCOFLjPIfTp3Po1Ok9XnjhJ959dy0jR87hjTdWGR2WqGIeeaQXsbE2rFYTJlPeEsGvvnolF1yQSJ8+TfITpd1uZeDAljRtKou5GU0ah87hgw/W8eCDi/xuo6pViyA9/bEKv3Yo2714MUc2biS+eXNaDx9eKWvAhLpDh9KZNm0jTqeH6667gHbtagF5KxV88ME6Nm48Spcudbnjjgsxm6W+U1GkcSgAHA4XHo/Xb9+5ltgQsOzxx1n1xht4nE7MEREkDR3KtZ9+KsnzPOrXj+Wxxy4ptN9iMXH33d0MiCiwvB4PSimUqWok/arxXVSQgQNb+q2BHRVlkUW1zsGRmsovr76KKysLr8uFKyuLHV99xdFNm4wOTVQS7fWScfgwzsxMIG/SmK/GjOH5yEies9mYf999eQsAhjipcZ5Dq1YJLFo0mnvvnU9qqoNBg1ry+utXGR1W0Mo+eRJzRAQepzN/n8lqJfuEtAKHg/RDh5jWty+n9u9Hezxc8vjjeD0ets2Zk7/a6YYpU4hv0YKLHnjA4GjLRxLnefTq1YgNG+4yOoyQUKNJE2zVq+PMyvIbulmnUycDoxKVZc4NN5CWnIz25D3OWvnKK8TUres314HL4WDX/PkhnzjlVl0EjNlq5bbvv6dWu3aYrFZqNG3KrUuXEhUfb3RoohIc2bAhP2kCuLKzMVssfs81VRVZuyogiVMpdZVSaodSKlkpNaGI121KqVm+11cppZoUeO0x3/4dSqmyz5orgkJ8ixbcvWkTTzqdPLBnD/W6nreBUlQR1erX99u2RkVx4dix2KpXxxodjTU6Gnt8PFc8+6xBEQZOuW/VlVJm4G2gP3AQWKOU+kZrva1AsbHASa11C6XUKOBFYKRS6gJgFNAWqAcsVUq10lpL03UVcGLnTubedBNpu3ZRs00brvvsM+KaNTM6LFFBrp0xg2l9+6KUQnu9NOzZk4v+9jc63noru+bPR5lMtBoypEqsXVXufpxKqYuB/9NaD/BtPwagtf53gTKLfGV+UUpZgCNAIjChYNmC5c51TZnkI/i5HA5eb9qUrOPHQWuUyURM3brcv3s3FpvN6PBEBck8epRDq1cTWaMGjXr1CrnuRyXtxxmI76o+cKDA9kHfviLLaK3dwGkgoYTHihB0bMsW3Dk5+Y1E2uslNz2dEzt2GByZqEgxtWuTNGQIjS+5JOSSZmkE4jsrqmfz2dXY4sqU5Ni8Eyg1Tim1Vim19vjx46UMUVQ2W2wsHpfLb5/X5cIWYqseClGUQCTOg0DDAtsNgJTiyvhu1asDaSU8FgCt9WStdVetddfExMQAhC0qUkJSEklDhuTPBG+NjqbtqFHUaNzY4MiCn8fj5emnv6dDh3e5/PJPWL/+sNEhibMEoh/nGqClUqopcIi8xp6zV+b6BhgD/AJcDyzXWmul1DfAZ0qp18hrHGoJrA5ATMJgSimu+/xzNn/+Oam//06tdu1oO3Kk0WGFhL//fREffvhb/hwJl146lY0b76J5c+nWFSzKnTi11m6l1HhgEWAGpmittyqlngHWaq2/AT4CpiulksmraY7yHbtVKTUb2Aa4gXulRT30eFwuTu7ZQ2SNGsTUrp2/X5lMdLj5ZgMjC01Tp27wm1jG6fTw5ZfbefjhngZGJQoKyMghrfV8YP5Z+/5V4OscYEQxxz4PPB+IOETlS9u9m48vu4zc06fxuFx0u/deBrz6qtFhhbSzZz8ymRRWa9VtaAlF8tMQ5TL7+uvJ9E3q4MnNZd3777Nz3jyjwwppEyb0yp+D02xWREdHMHJkO4OjEgXJWHVRLie2b/eb7cadk8PRTZtodfXVBkYV2h55pBf168cyd+42atWK5oknLqVOnRijwxIFSOIU5VK9USNO7NyZv22JjCS+RQsDIwp9SilGj+7A6NEdjA5FFENu1UW5XD9rFpFxcXnjke12Wl19NRdcf73RYQlRoaTGKcqlTqdOPLBnD0c2biQqLo5a7dvLbO+iypPEKcotskYNmlx2WZmP114vRzZuxOVwUKdTJyJ8neaFCFZyqw4sWpTMgAHTGTDgUxYv3m10OGHF43IxfcAApl5yCZ8NGsSbLVtycu9eo8MSBtizbBmv1q3Ls1YrH3TrRvrBg0aHVKywT5yLF+/m2mtnsXjxHhYv3s2wYTMleVaite+9x4Gff8aVlUVuejpZR4/y9e23l+jY/T/+yNIJE1j5yivknDpVwZGKinRq/35mDh1K5pEjeN1uDv/2G9MHBO/0vGF/q/7yyytxONz529nZbl59dSVXXtncwKjCx/EtW3BnZ+dva6/Xr5W+OJumT2feXXflzTIeEcHqt97iro0biZRJRELSwV9+8ZtNSXs8pO3aRc7p00H5Mw37GmdRQnCp+ZBVr1s3rHZ7/rbJYqFOx47nPW7RQw/lrWWjNZ7cXLKOHWPzjBkVGaqoQFEJCYX/8JQK2ufdYZ84H374YqKi/qx4R0VZeOihiw2MKLxceMcdJA0diiUyEmt0NDWaNmXo1KkAHPjlF1a++iqbZszIXyXxjIILgEHelHW5GRmVFnc4ObhqFV+MGMHM4cPZs3RphVyjWd++NOjZE2t0NOaICKx2OwNefRWTJThviss9A7wRAj0D/MKFybz66kpA8fDDFzNggHTgrmzpBw/icjiIa9YMk8XCusmTWfS3v+F1uzFZrdTt3JkxK1ZgMpsBmHPjjez46qu8yZIBi93O2J9/lhU1A+zQ6tV8cvnl+f+oLHY7I2bPptXgwQG/1pmlhDMOHaJ+jx406tUr4Nc4n5LOAC+JUwQdrTUv2O35SREgIiaGaz/7jKQhQ4C8FRTn3XUXu777DltsLIPfeYcWV8ma94H2xciRbJs9229fg4svZuzKlaU+17GtW8k6dozaHTpgT0gIVIgBVdLEGZz1YBHWPE4nHqfTb5/WmuwTJ/K3rVFRDP/kk8oOLeyc/YikuH3norVm3l13sfnTTzFZrWivl9ELF9KwZ+hOkxf2zzhF8LHYbNTp1AlV8PmW1jTq3du4oMJUt3vu8Wu8s9rtXPTgg6U6x54lS9g8YwYuh4Pc06dxZmQwO8SH5UriFEHppu++o0GPHpisVmLq1OGG//5XJg8xQLO+fRkxZw4Ne/akXvfuDPngA9rfdPYCD+eWlpzsN4MWkNdf0xO6c5bLrboISjF16nDHTz8ZHYYAWg4cSMuBA8t8fO0OHQrNXxDXtGl+Q18okhqnEKJCNerdm94TJmC22YiIicGemMior782OqxykVZ1IUS5aa8XrfU5a5GOEydwpKYS17Qp5oiIYsvlpqfz7V//yh8//khM3boM+eAD6l54YUWEXUhJW9WlximEKDOtNUsefZTnIiN5zmbjixtuwJ2bW2RZe0ICNZOSzpk0AWYNH872L78kIyWFw+vW8fFll5GRUuSq4YaRxCmEKLMNU6ey5q238LpcaI+HnfPmsfSRR8p8PndODvt++AFPweSrNXtXrAhAtIEjiVMIUWa7FizwG/7qzs4medGiMp/PZLUWORF2wS5RwUASpxCizKo3bIjJav1zh1JUq1evzOczmc30fuyx/ERpttmIbdiwXK36FUEah4QQZeZITeX9Cy8k+9Qp8DUOjf3lFxIvuKDM59Ras23OHPYuW0aNJk3oPn48ETGVs8qnjFUXQc/lcLD4H//gj59+Ir55cwa+8QaxDRoYHZYopdz0dHZ8+y0ep5MWAwaUq8ZpNEmcIqhprZnevz8Hfv4Zd04OymwmulYtxu/Yga1aNaPDE2FKuiOJoJadlsYf//tf/gxI2uPBmZnJH//7n8GRhTbt9frNKiUqhiROYQh3dnaRY5VVCA/DM9r6Dz/khehoXoiJ4f3Onck8csTokKqsciVOpVS8UmqJUmqX73NcEWU6KaV+UUptVUptUkqNLPDax0qpvUqpDb4PmYU2DGSfPMnUSy/122eyWqlWt265lhkOZwd//ZWFDzyAOycH7fFwdPNmZl17rdFhVVnlrXFOAJZprVsCy3zbZ3MAt2qt2wJXAZOUUjUKvP4PrXUn38eGcsYjQsBPEyeScegQukCNMzoxkb+sWoUlMtLAyELXgZUr/ebJ1G43KWvWGBhR1VbexDkUODOb7CfAsLMLaK13aq13+b5OAY4BieW8rghh6QcOFJqo2BIVRWSNGsUcIc4npk4d//6U+BZAExWivImzttb6MIDvc61zFVZKdQcigIILlz/vu4X/j1LKdo5jxyml1iql1h4/frycYQsjNb/ySqwFVi+0REbSrF8/AyMKfW1vuIF6XboQERODNToaq93OMJkhv8KctzuSUmopUKeIl/4JfKK1rlGg7EmtdaHnnL7X6gLfA2O01r8W2HeEvGQ6GdittX7mfEFLd6TQprVm6YQJ/Praa2itaTloENfPmoU1Ksro0EKa1+Nh1/z5ZJ84QcNevUho2dLokEJOpfTjVErtAPporQ+fSYxa66QiysWSlzT/rbX+ophz9QEe1lpffb7rSuKsGrweD9rrxXzWLaYQRqmsfpzfAGN8X48BCs1OqpSKAL4Epp2dNH3JFpU3qn8YsKWc8YgQYjKbJWmKkFTexDkR6K+U2gX0922jlOqqlPrQV+YG4FLgtiK6Hc1QSm0GNgM1gefKGY8QQlQ4GXIphBA+MuRSCCEqiCROIYQoJUmcosrKPHKEz4YMYVLjxswYOJD0Q4eMDilk5WZkcGLnTr/Z3sOZrKsuqiSPy8XUSy7h1L59eN1u0g8dYkqvXozfvl2GdZbSlpkz+fqOO/JWsFSKkV9+SbO+fY0Oy1BS4xRV0okdO8g8ciR//Lb2eMhOS+Po5s0GRxZa0g8d4us77sCdnY0zMxNnRgazhg8P+5qnJE5RJVkiIwtNW6e9XhmdVEqp27cXXs5Xa07/8YcxAQUJSZyiSopr3pymffvmL/pliYqiYc+eJLZta3BkoSWuadNCE7K4cnLAFN6pI7y/e1FlKaUY9eWX9P33v+l4661c8fzz3PTdd0UuPSuKF9esGVc8+6zfzEva42Fq795khfFkO9IBXogQpr1e0nbvBq2Jb9ECr8eD1+UK+DrkE+PiyD11Kn/bbLPR78UXueiBBwJ6HaOVtAO8tKoLUQbu3Ny8iYKVon63boWfA1YCl8PB9P79ObJhA1prImvUwOGrBTa4+GJu/PZbIqtXB/JmpDq0ejVZx45Rr0uXUq9Eqc9+Xuzx4MnNDcw3EoIkcQpRStlpaXzUsycZKSkAxNavzx0rVxIVV+SMihVm+ZNPcnj9+vzF2TKzs/NfO7RqFd/85S/c8MUXaK357+jR7Pj6a0xmM16Phxu//Zaml19e4mu1v/lmNk6bhtvXmm622Ui65prAfkMhRJ5xClFKSx59lFN79+LMyMCZkcHJPXtYNqGoVWMqVsqaNcWuaOlxOvNXDE1esIAdX3+NKyuL3PR0XFlZzLnhhlJda9Cbb9L93nuJb9GCet27c8uSJdRs3brc30OokhqnEKWUum2bX0uzx+nk+LZtlR5H7Q4dOLR6dbG3zGdux0/t24f2ev1ec5w4gdfjyevUXgImi4X+L71E/5deKl/QVYTUOIXw8Xo87F2xgu1ffUXm0aPFlqvfo4ff6CNLZCT1L7ronOc+tX8/8/76V2aPGMG2OXMCEm/fF16gZlJS/nIZZpsNa0wMEdWqYYuN5ZqPPgKgbpcu/r0JlCKhZcsSJ01RmLSqC0HeEM3p/ftzeN06lMmE1poxy5dTr2vhBlaXw8GMQYM4tGoVkJdIb16woNjO9emHDvFuu3bkpqfndcK32+n34ot0Hz++3HF73W6ObNwIWpPQujX7li/HmZVFkz59qFa3bn65XydNYumjj6LMZuwJCdy6fLksrVGESlk6wyiSOEWgrf/oIxbef7/fUMKE1q0Z//vvRZbXvtEzSiliGzY8Z//Qn158kRVPPonX5crfF127Ng8fOeJ/Tq8XVYEdy10OB9knT1Ktbt0KvU4ok/k4hSiF0/v3Fxp/nXGO2ZSUUtRo3JjqjRqdt1O9x+ks9IyxYBI9tW8f77Zvz7NWKy8nJrJ78eIyfAfnZ7Xbia1fX5JmAMg7KKqc1W++yWsNGvBq3br8+NxzlOSuqn6PHn5LFiuLhXpduuB1u1n8j3/wSp06TGrShM2ffQbA5s8+4+XERF6Ijmb29deTdfw4h9asIXXHjkLXu+D66/2eiVrtdi684w4gr+Y6rV8/jm/bhvZ6caSmMmv4cE7t2xeAd0JUFGlVF1XKpk8/ZemECfm1x58mTsRWvTo97rvvnMe1GjyYix96iJ/+/W+UyUR8ixZc9/nnrPjXv1j7zjv55/v2zjvJOnaMZf/8Z36fxp3ffsvOefOw2Gx4XC5aDxvGtZ9+ml+zS2zThluXLmXxww+Tc+oUbUeO5JLHHwfy+oSmHzjgVyM1WSwcWrOGGk2aBPrtEQEizzhFlTJj0CCSFyzw21eve3fu9DXknI/L4cCZlYW9Zk2UUkxq2pTTZ9X+6nTuzJHffoNi/nas0dEMmTyZ9jfddN7reZxOXqhWDW+B7k3W6GhumjePJn36lChmETjyjFOEJVv16nDWM8czww5Lwmq3E52YmP/c0hYT4/e6MpuJrFEDi81W7DlcWVkcL6ZR6WzmiAiumjQJq92OJTKSiJgYmvXrR+PLLitxzKLySeIUVUqfp54iIiYG5Zut3BodTd8XXijz+a589VUsvm5GJouFyOrVGfTWW8TUq4clKgqTxZKXqAska2t0NLVKMX1dt7vvZsyKFfR/5RWGf/opI//7X5nFKcjJrbqock7u2cOGTz5Bezy0v/lmEtu0Kdf5Dq1Zw+9z52KNjqbz2LFUq1eP3IwMNs+YQc7p0yRecAHz7roLZ0YGXpeLtiNHMnTq1DIlP601rqwsrNHRkjwNIP04hahE7pwcUrdvx1a9OnFNm5bpHEc2buSzQYPIOnYMq93OiDlzaN6/f4AjFeciiVOIEOJxOnmtfn0cqan5+6zR0dyfnExMnTrlPr/2etk0Ywapv/9OrfbtaTdqlNRoiyDzcQoRQtIPHsRVYFo4yHumenTTpnInTq01c0aNYtf8+fmPAfYsWcLQKVPKdd5wJo1DQgQBe82a+StynuFxOks94XBRUn//nV3ffYcrKwvIa/Xf8vnnnNq/v9znDleSOIUIArbYWK58+WWsdnv+bEed//IXarVrV+5z56an57X+F2CyWslNTy/3ucNVuW7VlVLxwCygCbAPuEFrfbKIch7gzILWf2itr/HtbwrMBOKB9cAtWmvn2ccLEQ6633cfjS65hCMbNxLfvDmNevcOyHlrtW+PJTKS3IwM0BplMmGrXp2EVq0Ccv5wVN4a5wRgmda6JbDMt12UbK11J99Hwfn2XwT+4zv+JDC2nPEIEdLqdOpEpzFjApY0ASKio7n9f/+jbufO2GJjqdetG7f/+OM5O/GLcytXq7pSagfQR2t9WClVF/hea51URLlMrXXMWfsUcByoo7V2K6UuBv5Paz3gfNeVVnUhREWorCGXtbXWhwF8n2sVUy5SKbVWKfWrUmqYb18CcEprfeaJ+EGgfnEXUkqN851j7fEwXs9ZCGG88z7jVEotBYrqD/HPUlynkdY6RSnVDFiulNoMFPVkutjqr9Z6MjAZ8mqcpbi2EAGnvV7SkpMBiG/RQua4DDPnTZxa637FvaaUOqqUqlvgVv1YMedI8X3eo5T6HrgQmAvUUEpZfLXOBkBKGb4HISqVMyuL6f37c3TjRiCv8eXWpUuJOGtCEFF1lfff5DfAGN/XY4Cvzy6glIpTStl8X9cEegHbdN7D1RXA9ec6Xohgs+KJJzjy22+4HA5cDgdHNm5kmW9+zZJwZWeTc+pUBUYoKlp5E+dEoL9SahfQ37eNUqqrUupDX5k2wFql1EbyEuVErfWZtVQfBf6ulEom75nnR+WMR4gKl7Jund965p6cHFLO01jp9XjYNGMGU3r35t8xMbxcqxYf9ewpCTRElasfp9b6BNC3iP1rgb/4vl4JtC/m+D1A9/LEIERlO3s9c7PNRu0OHYotr71ePrv6avatWJF/jPZ6ObxuHd/eeScjvviiUuIWgSNPtIUopYLrmUfExJDQsiX9Xnyx2PL7f/yRA6Ylhc0AAAVQSURBVD/9lJ80z/A4nRxYubKiwxUVQCb5EKKUbLGxjFu3Ln8989odO2K2Wostn33yZLGt7rENG1ZUmKICSeIUogxMvlUwS6JBjx6FlgdGKWyxsVzzkTzWD0Vyqy5EBatWrx6jFy2ieuPGmG02EpKSuPr997lv165SLbEhgofUOIWoBA179uRBWSu9ypAapxBClJIkTiGEKCVJnEIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUpJEqcQQpSSJE4hhCilcq1yaRSl1HFgfxkPrwmkBjCcQAvm+II5Ngju+II5NpD4zmistU48X6GQTJzloZRaW5LlP40SzPEFc2wQ3PEFc2wg8ZWW3KoLIUQpSeIUQohSCsfEOdnoAM4jmOML5tgguOML5thA4iuVsHvGKYQQ5RWONU4hhCgXSZxCCFFKVT5xKqVGKKW2KqW8SqliuzMopa5SSu1QSiUrpSZUYnzxSqklSqldvs9xxZTzKKU2+D6+qeCYzvleKKVsSqlZvtdXKaWaVGQ8pYztNqXU8QLv1V8qKzbf9acopY4ppbYU87pSSr3hi3+TUqpzEMXWRyl1usB7969KjK2hUmqFUup339/rA0WUMey9K0RrXaU/gDZAEvA90LWYMmZgN9AMiAA2AhdUUnwvARN8X08AXiymXGYlxXPe9wK4B3jP9/UoYFYQxXYb8JaBv2+XAp2BLcW8PghYACjgImBVEMXWB5hn0PtWF+js+7oasLOIn61h793ZH1W+xqm1/l1rveM8xboDyVrrPVprJzATGFrx0YHvOp/4vv4EGFZJ1y1OSd6LgjHPAfoqpVSQxGYorfWPQNo5igwFpuk8vwI1lFJ1gyQ2w2itD2ut1/u+zgB+B+qfVcyw9+5sVT5xllB94ECB7YMU/qFVlNpa68OQ98sD1CqmXKRSaq1S6lelVEUm15K8F/lltNZu4DSQUIExlSY2gOt8t3JzlFINKyGu0jDyd60kLlZKbVRKLVBKGbJ2se/Rz4XAqrNeCpr3rkosD6yUWgrUKeKlf2qtvy7JKYrYF7B+WueKrxSnaaS1TlFKNQOWK6U2a613ByZCPyV5Lyr0/TqHklz3W+BzrXWuUuou8mrGV1R4ZCVn1HtXEuvJG6udqZQaBHwFtKzMAJRSMcBc4EGtdfrZLxdxiCHvXZVInFrrfuU8xUGgYM2kAZBSznPmO1d8SqmjSqm6WuvDvtuOY8WcI8X3eY9S6nvy/iNXROIsyXtxpsxBpZQFqE7l3AKeNzat9YkCmx8AL1ZCXKVRob9r5VEwUWmt5yul3lFK1dRaV8rkH0opK3lJc4bW+r9FFAma905u1fOsAVoqpZoqpSLIa/Co0JbrAr75//btX6WBIIjj+Hcqbf1TiKWVDyAi0SdIERCsTZEmhU9hY5fOLtYWdhaChUkrVuKhFmotFhaWYnEpdgJHQowL3t4Vvw8sWe4uZBiOuexsArR93gamviGb2ZKZLfh8FdgFnkqK5y+5KMZ8AAxy796XbG5sEz2vFqFXVieXwKHvEO8AX+NWTdXMbG3cqzazbUJ9+Pz9Xf/22QacAc95nvdmXFaf3FW1K5VqAPuEJ9U38AFc+/F14KpwXZOwk/dGWOKnim8FuAFe/HXZj28BfZ83gIywi5wBnZJjmsoFcAy0fL4IXACvwB2wkTBf82I7AR49V0NgM/H9dg68Az9+33WALtD18wacevwZM37pUVFsR4Xc3QKNhLHtEZbdD8C9j2Zdcjc59JdLEZFIWqqLiERS4RQRiaTCKSISSYVTRCSSCqeISCQVThGRSCqcIiKRRqEh9G3ntFKpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 360x360 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# make up a dataset\n",
    "\n",
    "from sklearn.datasets import make_moons, make_blobs\n",
    "X, y = make_moons(n_samples=100, noise=0.1)\n",
    "\n",
    "y = y*2 - 1 # make y be -1 or 1\n",
    "# visualize in 2D\n",
    "plt.figure(figsize=(5,5))\n",
    "plt.scatter(X[:,0], X[:,1], c=y, s=20, cmap='jet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLP of [Layer of [ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2)], Layer of [ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16)], Layer of [LinearNeuron(16)]]\n",
      "number of parameters 337\n"
     ]
    }
   ],
   "source": [
    "# initialize a model \n",
    "model = MLP(2, [16, 16, 1]) # 2-layer neural network\n",
    "print(model)\n",
    "print(\"number of parameters\", len(model.parameters()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Value(data=0.8958441028683222, grad=0) 0.5\n"
     ]
    }
   ],
   "source": [
    "# loss function\n",
    "def loss(batch_size=None):\n",
    "    \n",
    "    # inline DataLoader :)\n",
    "    if batch_size is None:\n",
    "        Xb, yb = X, y\n",
    "    else:\n",
    "        ri = np.random.permutation(X.shape[0])[:batch_size]\n",
    "        Xb, yb = X[ri], y[ri]\n",
    "    inputs = [list(map(Value, xrow)) for xrow in Xb]\n",
    "    \n",
    "    # forward the model to get scores\n",
    "    scores = list(map(model, inputs))\n",
    "    \n",
    "    # svm \"max-margin\" loss\n",
    "    losses = [(1 + -yi*scorei).relu() for yi, scorei in zip(yb, scores)]\n",
    "    data_loss = sum(losses) * (1.0 / len(losses))\n",
    "    # L2 regularization\n",
    "    alpha = 1e-4\n",
    "    reg_loss = alpha * sum((p*p for p in model.parameters()))\n",
    "    total_loss = data_loss + reg_loss\n",
    "    \n",
    "    # also get accuracy\n",
    "    accuracy = [(yi > 0) == (scorei.data > 0) for yi, scorei in zip(yb, scores)]\n",
    "    return total_loss, sum(accuracy) / len(accuracy)\n",
    "\n",
    "total_loss, acc = loss()\n",
    "print(total_loss, acc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0 loss 0.8958441028683222, accuracy 50.0%\n",
      "step 1 loss 1.7235905336972022, accuracy 81.0%\n",
      "step 2 loss 0.7429006313851131, accuracy 77.0%\n",
      "step 3 loss 0.7705641260584198, accuracy 82.0%\n",
      "step 4 loss 0.3692793385976538, accuracy 84.0%\n",
      "step 5 loss 0.313545481918522, accuracy 86.0%\n",
      "step 6 loss 0.2814234349772435, accuracy 89.0%\n",
      "step 7 loss 0.26888733313983904, accuracy 91.0%\n",
      "step 8 loss 0.2567147286057417, accuracy 91.0%\n",
      "step 9 loss 0.2704862551637922, accuracy 91.0%\n",
      "step 10 loss 0.24507023853658053, accuracy 91.0%\n",
      "step 11 loss 0.2509905529791503, accuracy 92.0%\n",
      "step 12 loss 0.21560951851922952, accuracy 91.0%\n",
      "step 13 loss 0.23090378446402726, accuracy 93.0%\n",
      "step 14 loss 0.20152151227899445, accuracy 92.0%\n",
      "step 15 loss 0.22574506279282217, accuracy 93.0%\n",
      "step 16 loss 0.19447987596204114, accuracy 92.0%\n",
      "step 17 loss 0.21089496199246363, accuracy 93.0%\n",
      "step 18 loss 0.159830773563036, accuracy 94.0%\n",
      "step 19 loss 0.1845374874688392, accuracy 93.0%\n",
      "step 20 loss 0.18977522856087634, accuracy 91.0%\n",
      "step 21 loss 0.19072704042579647, accuracy 93.0%\n",
      "step 22 loss 0.11733695088756485, accuracy 97.0%\n",
      "step 23 loss 0.12173524408232454, accuracy 95.0%\n",
      "step 24 loss 0.1261571261277045, accuracy 95.0%\n",
      "step 25 loss 0.16049097780801674, accuracy 95.0%\n",
      "step 26 loss 0.18747197705245805, accuracy 92.0%\n",
      "step 27 loss 0.16741837891059408, accuracy 95.0%\n",
      "step 28 loss 0.09586583491455399, accuracy 97.0%\n",
      "step 29 loss 0.0877878370742091, accuracy 96.0%\n",
      "step 30 loss 0.11731297569011848, accuracy 95.0%\n",
      "step 31 loss 0.09340146460619836, accuracy 97.0%\n",
      "step 32 loss 0.12454454903103446, accuracy 95.0%\n",
      "step 33 loss 0.07984002652777272, accuracy 97.0%\n",
      "step 34 loss 0.07727519232921673, accuracy 97.0%\n",
      "step 35 loss 0.07661250143094483, accuracy 98.0%\n",
      "step 36 loss 0.10610492379198365, accuracy 96.0%\n",
      "step 37 loss 0.09062808429265976, accuracy 99.0%\n",
      "step 38 loss 0.10671887043036932, accuracy 95.0%\n",
      "step 39 loss 0.05225659921975849, accuracy 98.0%\n",
      "step 40 loss 0.06016009895234464, accuracy 100.0%\n",
      "step 41 loss 0.08596724533333942, accuracy 96.0%\n",
      "step 42 loss 0.051121079431796, accuracy 99.0%\n",
      "step 43 loss 0.052401424016428284, accuracy 97.0%\n",
      "step 44 loss 0.045306841790015734, accuracy 100.0%\n",
      "step 45 loss 0.07211073370655095, accuracy 97.0%\n",
      "step 46 loss 0.03334238651310234, accuracy 99.0%\n",
      "step 47 loss 0.03143222795751122, accuracy 100.0%\n",
      "step 48 loss 0.03658536747111507, accuracy 99.0%\n",
      "step 49 loss 0.04829139382390309, accuracy 99.0%\n",
      "step 50 loss 0.09875114765619622, accuracy 96.0%\n",
      "step 51 loss 0.05449063965875453, accuracy 99.0%\n",
      "step 52 loss 0.03392679435708309, accuracy 100.0%\n",
      "step 53 loss 0.05261517263568441, accuracy 97.0%\n",
      "step 54 loss 0.03250295251424923, accuracy 99.0%\n",
      "step 55 loss 0.02888327387207822, accuracy 100.0%\n",
      "step 56 loss 0.04139151104027239, accuracy 98.0%\n",
      "step 57 loss 0.018987407426128502, accuracy 100.0%\n",
      "step 58 loss 0.0252383352388374, accuracy 100.0%\n",
      "step 59 loss 0.02079656521341895, accuracy 100.0%\n",
      "step 60 loss 0.0325971115781023, accuracy 99.0%\n",
      "step 61 loss 0.017863351693480307, accuracy 100.0%\n",
      "step 62 loss 0.023008717832211683, accuracy 100.0%\n",
      "step 63 loss 0.022079325463581503, accuracy 100.0%\n",
      "step 64 loss 0.029432917853529684, accuracy 99.0%\n",
      "step 65 loss 0.01625151464409193, accuracy 100.0%\n",
      "step 66 loss 0.02846853448326446, accuracy 99.0%\n",
      "step 67 loss 0.013994365546208731, accuracy 100.0%\n",
      "step 68 loss 0.015552344843651405, accuracy 100.0%\n",
      "step 69 loss 0.0338911994616017, accuracy 99.0%\n",
      "step 70 loss 0.014229870065926908, accuracy 100.0%\n",
      "step 71 loss 0.013255281583285504, accuracy 100.0%\n",
      "step 72 loss 0.012300277590022063, accuracy 100.0%\n",
      "step 73 loss 0.012676052498355976, accuracy 100.0%\n",
      "step 74 loss 0.020593811955954763, accuracy 100.0%\n",
      "step 75 loss 0.011845398205364453, accuracy 100.0%\n",
      "step 76 loss 0.016012697472883086, accuracy 100.0%\n",
      "step 77 loss 0.025458360239222128, accuracy 100.0%\n",
      "step 78 loss 0.014382930289661911, accuracy 100.0%\n",
      "step 79 loss 0.011698962425817985, accuracy 100.0%\n",
      "step 80 loss 0.012318500800515763, accuracy 100.0%\n",
      "step 81 loss 0.014121117031464233, accuracy 100.0%\n",
      "step 82 loss 0.011664591962446225, accuracy 100.0%\n",
      "step 83 loss 0.011589314549188726, accuracy 100.0%\n",
      "step 84 loss 0.010990299347735226, accuracy 100.0%\n",
      "step 85 loss 0.01098922672069161, accuracy 100.0%\n",
      "step 86 loss 0.010988193757655071, accuracy 100.0%\n",
      "step 87 loss 0.010987200447388707, accuracy 100.0%\n",
      "step 88 loss 0.010986246779084925, accuracy 100.0%\n",
      "step 89 loss 0.010985332742365272, accuracy 100.0%\n",
      "step 90 loss 0.010984458327280174, accuracy 100.0%\n",
      "step 91 loss 0.010983623524308862, accuracy 100.0%\n",
      "step 92 loss 0.010982828324359073, accuracy 100.0%\n",
      "step 93 loss 0.010982072718767003, accuracy 100.0%\n",
      "step 94 loss 0.010981356699297042, accuracy 100.0%\n",
      "step 95 loss 0.010980680258141723, accuracy 100.0%\n",
      "step 96 loss 0.010980043387921506, accuracy 100.0%\n",
      "step 97 loss 0.010979446081684675, accuracy 100.0%\n",
      "step 98 loss 0.010978888332907229, accuracy 100.0%\n",
      "step 99 loss 0.010978370135492717, accuracy 100.0%\n"
     ]
    }
   ],
   "source": [
    "# optimization\n",
    "for k in range(100):\n",
    "    \n",
    "    # forward\n",
    "    total_loss, acc = loss()\n",
    "    \n",
    "    # backward\n",
    "    model.zero_grad()\n",
    "    total_loss.backward()\n",
    "    \n",
    "    # update (sgd)\n",
    "    learning_rate = 1.0 - 0.9*k/100\n",
    "    for p in model.parameters():\n",
    "        p.data -= learning_rate * p.grad\n",
    "    \n",
    "    if k % 1 == 0:\n",
    "        print(f\"step {k} loss {total_loss.data}, accuracy {acc*100}%\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-1.548639298268643, 1.951360701731357)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmMJPl12Pnvi4jMrDvrvo/u6mumZzhDzsUZUpRIkRSHWoK0vBRA7mIh7doY7MKE9/rDMgTIWAMGZHixC68lWDsraSUDlmSCXko0TJkUD/MQRWqaw+FwZrp7+q6u7qqu+8ysPCLe/hF1ZeVRWZVZZ74P0OjKzMiMqKzM34v4/d7v/URVMcYYU3ucoz4AY4wxR8MCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1yjvqAyilLVav/Y0tR30YxhhzYrwzPzWjql3lbHusA0B/Ywt/8vHPH/VhGGPMifHeL/7Le+Vua11AxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1ygKAMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1qioBQET+UESmROStIo9/WEQWReSN9X+/VY39GmOM2T+vSq/zR8DvAP+mxDbfU9VPVWl/xhhjKlSVKwBV/S4wV43XMsYYczgOcwzgJRH5qYj8pYg8UWwjEXlFRK6IyJX5VPIQD88YY2rLYQWA14ERVX0a+FfAnxfbUFVfVdXnVPW5tlj9IR2eMcbUnkMJAKq6pKor6z9/FYiISOdh7NsYY0xhhxIARKRXRGT95xfW9zt7GPs2xhhTWFWygETkT4EPA50iMg78EyACoKq/B3wW+B9EJAskgc+pqlZj38YYY/anKgFAVT+/y+O/Q5gmaowx5piwmcDGGHNKDPXe3dP2FgCMMeYUGOq9S8Po0J6eU62ZwMYYY47Axll/w+gQkaca9vRcuwIwxpgTqpLGHywAGGPMiVSo8Xde/IU9vYYFAGOMOWGq0fiDBQBjjDlRSjX+Ut+9p9eyQWBjjDkhdjb+TvcwjJ4Fwsb/2sKNPb2eXQEYY8wJUF7jv7eBYLsCMMaYY26j8Y//nUsAOY3/9dQipBaBBr50e28l9O0KwBhjjrFdG39gP40/WAAwxphja6+N/2vfaNnT61sAMMaYY+igG3+wAGCMMcfOfhv/nvq9BQEbBDbHxtpyitk7cySXUogDDfF62oZbqWuOHfWhGXNoNoq6beb4rzf+uWmeYeNf57byva8FwN4bf7AAYI6J2XvzjP34AepvrRO0+GCZiWtT9D7WTf8TPUd4dMbsbq+lmIspNsFro/H/0m0BKm/8wQKAOQb8jJ/X+G8KYOr6NPHeZho79l7sypjDsD1Hv1K7N/5UpfEHCwDmiCWX1ph4+xEaFF8hNPCVqVuznC0jAKgqibkkGigN7fU4rg1zmYO1s8umGg6j8QcLAOYIPXhzgqmbs4XP/HdYuL9I6okeYo3RotuszKxy6/t38bMBKIgI/U/30nOhs5qHbQxQpCZP93DlL1ygtMNBNP5gAcAckZXpVabLbPwBNFAe/HSC0Q+MFHw8k8py4zt3cq4kVJUHb0wQrfNoG2otaz/pRJr7b0ywNLEMQLy/mcH39hOtj5T1fFMbqlWNs5jDaPzBAoA5IjN35gjKbPw3LK43ygVf79Zs0W6ksZ88BBEidR6NHQ2ISMHtsmmfa9+4STblb963ML7EykyCJ16+iBtx93S85nSqZjXOYjbq+mykeR5E4w8WAMwR8TPBnp9TpN0GYGUmUXxfKZ97r40D4MU8Lvz8WWJN+V1JM3fmwu6jvGP1mb07T7d1JdW8o2z8q9nwb6hKABCRPwQ+BUyp6pMFHhfgXwK/DCSAX1fV16uxb3MytQ62sPxoOe8qQBwAyT+bF2gdjBd9vbqWGMuPVoo+Hqw37OlsmhvfvcMTn7xIOpHh4c8mWZxYRhzB9ZyCXVLqK8tTqxYAatxujf9eSzEXt3OC18E0/lC9mcB/BLxc4vFPAhfW/70C/Osq7decUG1Dcepa6hBn67ReHKE+Xs/ICwOIK7D+kOM6ROsjDDzdV/T1+i6Xf/aVTWVZnFjm2jduMn9/kSAb4Kd90olM4ScIRBttDKCWFSrFXLjxb6j4X6Wze/eiKlcAqvpdETlTYpPPAP9GVRX4oYi0ikifqk5UY//meEitpFh4sIQGSry/hfp4XdFtHcfh4kdGmb41y+zdBQA6RlrpOt+B4zo0tjcyc3uOTCJDc3cjbcOtJVM6vajHyAuD3Pvb8bKOdfbO3OZVwW7EEbrOdaCBsjgRjglE6jzaR9qI1Fkv6mlXbh3+sOHee0XOQg6j8YfDGwMYAO5vuz2+fl9eABCRVwivEuhraD6UgzOVm7w2xcTbU4CiChNXp+g8287ge/uKDro6rkPPxS56LnblPRZrjDLwnt49HUPHSBst3U3M3J0jnciwNLFCJpl/Vq+BsraSLjn3wPHWg43C8PMDeDGPq391g3QiQ5ANEEd4+LNJvDoPPxNQ1xyl/8leWnrtM3ua7LUOf51bXrZZKQcx2FvMYQWAQi1AwW+fqr4KvArwRHvP3tJEzJFILCSZeGcqNwXTV2bvzNHS10y8RKOoGm43cXWaTDJDrDFK3xM9tA/v74sUqY/Q93hYNmLp0Qq3/vpuTr++uELbYJxsyie1lMp7vjhC72NdNLTXA0JzdyOO6zD24wekltOEF7Fs/q6ZZDZ8D+bXuPWDe5x5frDslFNzvO2vINvekxsKOYzGHw4vAIwD2+dIDwIPD2nfNSe1mmbq3RlW5xLEmqL0XOyioa3+wPY3e2e+4OBp4Cszt+dKBoDJa9M8ujq1ORicWkkzdmWcIBvQOdpe0XG19DRx7oNnePDTCZJLa3hRl67zHfQ+3s3y1ArL0yv5xy3QcbadaENun//c2MJm41+M+sr4TydoHYwXveoxJ8NhVeM8aocVAL4CfEFE/gx4P7Bo/f8HI7GQ5N1v3ybww9mwibkkCw+WGHl+kPYqnZmqKosTy0y9O0M2lS3Zl+6n/aKPBdkgp/HfvN9XHvxsko6zbRU3pC09TbT80oW8+5u6Guk428bs7fmtgWhVzrw4nNf4AyW7i7bLpnyyaZ9IzMYGTqpaafyhemmgfwp8GOgUkXHgnwARAFX9PeCrhCmgNwnTQP/bauzX5Bu78iCvQVZfuf/jB7QOtOA4lSd+PXxrkukbs7tO5HJcoXWw+JdibSW1ntxf6OohILOWPZAZuDN35njw0wlUw2AWiXr0XOqi/Uwbrlf4/WnuadqcHbybYq9hjr9aavyhellAn9/lcQX+QTX2ZYoLsgGJhcJZCKqQmE/S1NG4eZ+f8VlbThGp84g2FK+xs106kWbq3eKzbrfz6iN0nCnejePFvOKvoxzIzNvFyWXGf/IwJ3ilExkmr02X7HIafLqP6zOrm3WGChFHwiBrBehOpMOsw39c2HXqabJLb8nG2b+q8vBnk0zdmEWccNJVY0cDoy8N4+3SdbH8aAWRom1gzrGc+8BIybPhaH2Exs4GVqZXc15QnPDK4SDOpCffye9ygjAYLk4u09pf+Mtc1xzj8V+6wKPrMyxPr+BFXVLL6fWAEL5eXbyO4WcHqn7M5mCVP8HrdDX+YAHgVHFch+buJpanVvJaaNdzqG8N8/Knb8yGhdgC3TwDX5le5dq3blHXHEP9gLbhVtoL5N6L5xTttsnZTmTXYAIw+uIwN753J8zIEUFVaWxvOLCGNLWSLni/BkHeY9m0z+LEEuorLb1NRBuiDL2vf+s5qqxMr5JaTVMfr6Ohrd4Gf0+Ychv/ai7CcpxYADhlRp4b4No3bxFkfAJfEVcQEc6+NLzZOE1emy54FpxeSZNebwRXZxPM3Jrj4kdGc4JAvLd512wYgPrWurImSXkxj8c/doHEfJLUSpq6lljJCWSVqmuJsTKdzbtfHIf6lq2lJ+fuL3DvtfHwPdNwbkP3xc6cuQkiQnN3E5b5fzLtrfE/uIJsR8kCwCkTbYjyxCcvMTc2T2JujVhzlI5tM1ZVlWwqvwHcKfCV5NIas3fn6TrXsXm/G3E5++Iwd344BpqfHeO4Do4nnHn/3lZGamirP9BU1Q19T/Rw83t3ctM/BSJ1Hs09TUA4znHvtXHUV3Tblc70jRmaOhuJ91Wvyd9YBzmdzNDS00zbUNzGEA6BNf4hCwCnkOs5dI12wGj+YyJCtCFSvO7NNuorc/cWcgKAnw3IrGXCyVTpLLHGKE1dTWRTWVLLKeridbQNtR7bTJjmrkbOvDDE+E8ekl1PUW3ubmTk+aHNK6TZewub/frbBb4ydWOmagFgcx3kQEFh8eEyk1enuPTR83hRKz19UKzx32IBoAb1v6cnTBctox7/9mJtqdU01795kyAbhN1LjrDiJGgfaaOtRKXO46ZtME7rQAuZtSyu5+RlG2XXsmiRqQ3lXD2VI5vOXwc5yIbjEBNvTTL0jA0mH4Ty6vrURuMP1asGak6Q9uE2Bt/XjxdzwwZeKJhB5LhCx5m2zdv3Xhsnm/I3A4cGSpANuPM3Y2WNCxwnIkK0PlIw1bS5p2mrFtD25zhStVo/S5MlFre5O1+VfZhc5Rd1y238e+pbTmXjD3YFULM6z7bTcaaNbNrH9RwevvWImVtbk7scz6Gps2GzJo+f9VmdWS34WtlUlrWl1IEO3h6meF8zdc0xkotrW2McAm7EqdqaAKpadA6E+oqfDY5tN9pJtNeibgddh/+4sABQg5JLa2SSGerjdUTqwpm2g0/30TYYZ/ZeWNendTBOS2/TZr94sS4RIEzfLLNUwkkgIlz48CiTV6eYuztPECit/S30PdlTtfLPLb3NxSeUCaxMrxDvO92Nz2Gptdm9e2EBoEZkU1lm7swxfWOWTDq7nt4I7cOtDD87gDhCY0cDjR0NBZ/vRV3qWupILq7lPSYip+bsf4PrOQy8p3fPJanLFYl5YbdbgSAglgVUNdb4l2YBoAYsPlzi9g/HcgYcN9Ib5+4vEGmI0P9Ez66vM/zcADe+c2ez0ByE5ZVHnh/IGSw2ubKpLDO351iZSRBrjtJ1roO65hhtA3HmxxcLPqe5u+mQj/L0scZ/dxYATjk/43NnR+O/nfrK9I0Z+i537zqLtbG9gcc/foFH706TmEsSa47Sc6mLhtaDz98/qdaWU1z/1i0CPwj/Bo9g5vYcoy8NM/DePpZnVvEzfviYhAPNw88N2FyAClnjXx4LAKfcwsOlXev2+Jn1M/oyTuJjTVGGLUWxbGOvP8gtia1h0L37o/s89enLPPHyRWbvzrM8vUq0IbJ5dWD2rxaLuu2XBYBTbG1pjcm3p4qe/W+INkSsC6dMqZUUk9dnWJ1ZJdoYXgE1dzUW3DYIgrDQXQGqsDqXoKmzke4LnVXLLqpltV7XZz8sAJxSmbUM1791Kzy7L0Fcof+ABjpPm52L7awtpViZWmHw6T46t82WLoeqMv7GBNl0lvrWevoe7z6UUhinlc3u3R8LAKdIJplh+vYcycU1/G0Ttorx6jwGn+7b9/q7e7XxJT0O7k+e2ftzXn+Yt9hOsL4MZNtI/mIyjuPQ3NXI8lT+VYD6SmI+7H9Or2ZYmlzm3AfP0NJjg797ZY3//lkAOCVW5xLc+M6dnBLPxUQbI5z/0NlD7Wve6Jc9Loa4u6cgoIGyOpso/KAIq7OJgo330LMDXP/mtkHgIqmf6iv3X3/A5ZcvWknpPbDGvzIWAE4BVeXOD++XXJt3gzjQNtR6aI1/oS/ocdDAEEPcBcq8GpBwglaxiheOW7jRrmuK8cQnLzJze57V2VUcz2XhwWLBcZl0IoOf9staR8FY418N9kk7BdKraTJru1f3hHDSVqmlD6up4Be0e/hQ9l1KMDVG5KkGGhgicfs+Q727Xw2ICPGBOAsPFvPO4B1Xik6gA/CiHr2PdQFdrC2nwtcoQAPl4VuT9D3RszlD2xRWfl2fsJvNGv/CLNn4FNgtzdPxHBzPwY26jH5whFhjeev/VuK4Nv6wdRyRpxo2u6XKGZ8YeqafaEN0s1CcuILjOYx+YKTsbptYU7Tk+sszd+a5+vUbZNaqU3X0NNqe47+Xxv80F3XbL7sCOAVijVEiMa9gjf9oU5Qzzw8iIjS0H86ShaUuzY8LZ/QswQ+/s6crgUjM4/LLF1h8sMTqXJJoQ4T2kVa8aPlfIxFh9APD3Pj2bfyNcYHtNJy8N3V9moGn+/b5251eVtStuiwAnAIi4QpcN797lyAIUxTFCZc5PPv+IRrbD6/vfbd+2eNCk1M4L/7CnoOA4zi0DbXSNrT/zKn6ljqe/NRj3PnhGIsP88tCawALE0sWAHaw2b3VV5UAICIvA/8ScIHfV9Xf3vH4rwP/AniwftfvqOrvV2PfJtTU2cjjn7jA9M1ZkotrNLTV03Wug2jD4fUll7vYxnHwWOuFfQeBanBch6auJpYmVwpmbRVap6CWVavx11SW7NgiOpeEiIs71ILT2VCzmVcVBwARcYHfBT4OjAOvichXVPWdHZv+O1X9QqX7M8XFGqMMHtFZY7mDcgfFzyrf/0qK17+ZJpWEnmGHj36ujuHHCn/Ery3cyAkC3L5D5KmxQw0CbUNxHv5sMu9+xxW6zu9tYtlpVrXGfy1L+m8fQHajmGGG7GIKZ7CFyIXDSYw4bqpxBfACcFNVbwOIyJ8BnwF2BgBzTGRTWcZ/OsHC+CKq0NLbxODT/cSa9jc4vNd+2YPw8ItZEjcVXR87nbwb8Cf/e4KB/9qlfiQ/1+Gzow2bVySXYnEYPYsDRJ4aI/7UJRb//PqBBQE/4zN7b57EXJJ4f3PYDSRhFpA4QutgnPbhVoIgYO7uArP3whXCOs600THSVlNlO4rV9YHws5VYUX7yLZfXXl8mEocH2gJNhbt9srfmYefM+EAJxpfQoRakSms9nCTV+I0HgPvbbo8D7y+w3X8pIj8PvAv8z6p6v8A25oAFfsC1b94MB4zXex4WHy6zMn2Tyy9f2HP64V7Pzurc6s86XpvOkrg5v9n4b9AszH5LOP/3c/e55i/wpdtJPjvaACS4nlrMCQLB1Bjxv3MwQWBjXWU/Gw4AbzTmXec6iNR5tPQ2Ux+vQwPlxnfukJxPbs7oTswnmbu3wIWfP1sTQWC3om6LM8r/+78FrCV9NAsJoMlZxLvUCQWqagQzRSbyAcFcEre/Ost9niTVCACFPok7OzX/A/CnqpoSkf8e+GPgFwu+mMgrwCsAfQ219wc5aPP3F8mm/Ly/UOAHTN2Y3dMCKPtp/DdysSumirucxV3zkTWfWFA4pznxIJu3zw99ovXIgsC9K+G6ypu/xnr//9zYAu/51GObfdHz44sk59dyynmoryTmEiw8XKJtMF6V4zmOyp3g9ce/H5Bc0a21KQAC8K/P4nY3IjuX1CwWNKXEY6dcNQLAOLB9jv8g8HD7Bqo6u+3m/wP882IvpqqvAq8CPNHec3rWGTwmVqZXC84Y1kBZmV4p+3X21y9bnXQ8TftkXp9A17LrX36FInFFIm7ePr/3tSWghec/tnSoQSDwi1cHDbIByYW1zYJw8/cXw6Jzea+hzN9fOLUBYC+zexM3tPAkGIFgfg23K3fMyelrIri/mP9ZUXA6j88s9cNUjYlgrwEXROSsiESBzwFf2b6BiGwfmfw0cLUK+z31kktr3P3b+1z9+g3u/HBss3hYJUqVfo6UmKC03fYvKZA34BvK7/apVjpe5q0pNJEBXyEo3viHh+Gh2+vxbzuOjcHC8DjDBmAzeI2e3ZwwthHkKi1mp0rRWXuBH+RkAxUrLQFhKupptNfSDiXXryjwmHe2FWmKgrvt+Y7gPdGZf7VQIyr+rVU1C3wB+Bphw/5FVX1bRP6piHx6fbN/KCJvi8hPgX8I/Hql+z3tlqdWuP6Nm8yNLZBcXGP+/iLXv32LhYdLFb1ux9m2gl8OcYXuC+VnnhQq7FYoz7/aff6a9tGFteLTn3f+bvMp0j96UDII1LmteUFA6rvzgkDD6FBFQcD1HKRYw65Q17JVn6njTFvBIOC4Tvg3PGX2U9cn1Rot+jFw2vLXqBbXIfJcP96T3TjDcdxzbUQ/MIhbw8tvViXsqepXVfWiqp5T1X+2ft9vqepX1n/+x6r6hKo+raofUdVr1djvaaWq3LvyIOz/3fYJV18ZuzJOEATMjy/y7nduc/WvbjDxziOyOxq4YqINUUZfGt4sD+GsN0qDT/XR1FF4YZPjRDP+3vtrMz7Zuwt5d28Ege99LcgLAtcWbuQFgY3SEUO9d/cVCPxsULRSq7hCcnFt83ZzTxNtw605QcBxhfaRVpqKLEBzUu23qFvyTCMSc7c+Dxtn9Jc7kSJLaooIbmcDkQvteCOtSI0X3qvt3/6Yyq5lySQLF3cLfOXOj+6zPLG8OUC4tpRi5vY8j3/8fFmVJON9LTz16cdZnlpFA6W5u/HETDyS+hJZSk6Rcp0KwXQCLuZf4fTUt/AoucT3vhbwoU+08qXbC4Dw2VHdmiswur/SEXnHXiJubQz+zt6dJ51IU99az9Az/XSebWd+PAxebYOtJYvOnUS7zR95a/pdxq7F+O54ivoRaGxs2xzU7463oi+14E+soPNrUOfiDbQghzj58aSzAHAMFe0mIBysXXywlHtlECjZtSyPrk8z8FR5E8Ec1yHed/KyrMQR3PPt+Dfmwv7/DY4gXQ3odKJgECiVNrkzCIQZQjuCQBVmDW+854sTy3ldWKrKu9++vXW8rhBtiHLpI6MMPt1f1uufNKUSCaS+m7/8/nW+/LsBmSARvl0+rA6u0nOuZ/M1xHXwBltg0Mo97Edtjnwcc17UK7o8oAaFMx9UlfnxwmWGTxtvsAX3cifUe2Gd/sYI3pNdRIqNYTiCM1A62O3sDoKtLofN7iC2uib2Wkl0w/CzA0TrIzkVRYG8onDqK6nlFPd/8jDvNU6D3bLIrtx9l3//fwWkkhCkQFPhvI6GsSRBFZIhTMgCwDF15oUhvJiLs9GXWUa392nNDtkpWFjDvzUPG/n0jiANESTm4j7WEXYFbbxfriCtMdwyzhAPIwhE6iI88clLjDw/SO/jXeFynCX+bAvjS2ixVWhOqHJSiN/6gUe20LBWoGRvzp+69+So1EaLcQLFmqI8+cuPMfS+PrrOd+za/osrdBzSQi9HSRMZMj+ZhGQ27AJS0OU0mSsTaMbH62sm+tIg7rk2nDNxIk/3EHlvb9kzZw8jCIgjtA3G6X+yNxwYKJHGqrr7Ep8nSbnzR16/k0aL5DXoUors1Zm8IKCq+I9WSb8+QfpvH5C9Mx8mDZiiLAAcY47n0HG2nYGneksu+iKO0NjeQNf50x8AsmOLuX3/GwLFnwgnskmdhzfSSuRcO07b3tdA2Fg4JD8INOQFAad7eN/dQQDR+uLzMgDqW+u2rgJPuL1MHqwfdtASv3bwaBVdSOXcl702Q/adaXR+DV1O499dDFOALQgUdTo+WaecOEK0WPaLwJn3D3LhF84W7QJKraa5d+UBb//ldd799i0WHlQ2l+Ao6Uq68AOBFn9snzaCwM4JY9cWbuRNGIs81bCvCWMdZ9uKZgeJA8PPDFTwGxwfG3V9yp05fu1BnCBaKgIo/qOtmevBSppgcjX35CBQSPtk79XG2Nh+WAA4AUSEgad787KDHFfoPNdB22Br0bPctaU1rn79BrN350itpFmZSXD3R2M8fPvRYRx61UljkUDoSPHHKnDQs4aj9RHOrs/LEHdr7KK+tY7HPnbhVKR9lirqtrPxr3Nbw/faEepfGIRSQWCbYLZw9hcKwVTh8hvG0kBPjLbBVsRxePDmBKmVNF7Uo+dSJ90XO0s+b/zNybzaP4GvPLo2vVmB8iRxh+P5Z3oAwoFVc9xIE33tGwdTP2hrXsbK+ryMprLmZawtp5gfWyAIlHh/M43t4cImqsrC+BJz9+ZBoH2kjdaBlkNf9GRvE7ySBRdu9863k70+G5b92M4R3J7GnNvF1ELl1P06Wd/+Gtfa30Jrf/n5zmvLKZYm85cchPBLsTy9QnsFSxseBacxSuSpHjLvTIcLe7De5/9kF1Kg0dRUlmAxhXgO0la370bwoINAOEeg/L/t5LUpJt6eCgdCFaZvzBDvb2HkhUFu//W9sOjfeqO5/GiFue4mRj9Y/uL1ldrv7F7IrRnl9DQhEyvoUmorCLiC092ItG6Ve3C7GvFvzudfBTiCU4NlnstlXUCn1IO3Jrn69RvFa+YQ1qY5iZyOeqI/N0TkhX6i7x8g8uIATnMsZxtVJXNjlvQP7pN9Z5rMm49If2+MYClV5FV3d1RF5HZKLq0x8c5UzpyQwFcWHy7z4M1JVqYTOWWkA19Znl4NJ6Adgmo1/hCeqETe24t3uQunqwGnp5HIe7rxHu/MCWZS5+FeaM9PAY6HKcDBSppgNoGmdiwaUePsCuAESiwkWZpY3lw9KtaYW8VzZWaVqXdnSqcPCjSfkCJYqorOJfEfLoOvOD2NOD1NOI3Fq5cGk6sE48vrKZbr74OvZH4ySfTnhorWitnN9iuBD33C4Uu3F3KuBDZLR7B1JZB5M8EQ1VtTYG5soeDfNvAD5u4tFC4jnQ2YG1vY0xXkfuyr8f9PPk7Cp0PqUMcP6/tsI47gdjfidpeugeQNtuC21+NPrqDZALejARo9MlceoquZMDAoOD2NeI91WtcQFgBOFFXl/usPmL0XNgAi8PCtR/Q/2UPPpa7N7WZuz+XNLN3OcYVzHxg5MemF2euzBBMrm/3+wfwaMr5E5Jm+og25P7ZQOF1UlWAmgduz/+BXqH7QRhAoVT9oiLsAFQcC3VzTttCDxZ930L0/+2n8v/+VDM3vLOOmAjKyBKo4fc14lzr21V0lDRG80a1qqekfPcjLDgsereLHXLxzpz9tejcnowUwACxOLDN3bzFs3BU0CEtDPHz7UU4lSb9E3nN9vI4nP/XYvs/+t3dn7CzeBWz7km9N169kFbBgKZV8EezWAAAdD0lEQVTT+Id3KrqSCa8IithZ/jnnuanK88ILVRI9iNIRhcT7WwoGb3GF+EBz0TLS7SMHV0a6UFG3/Ma/Ie/MP351BTfph39fP1zbIZhYwR+rPHUzWE6H60bkPaD444fTHXbcWQA4QWZuzRa8vFdfw4yPda0D8aKNQPelTrzo/i78dqvcWPBL/rVgc2LVfmTHl4pO/Aomi6f3OS359eDDAxWceKzwY3t0WPWDdmrqaqS5uzHnb7wxV2Tw6T5a+ppzAoTjCvH+ZloquOopZfsEr52fi+upxW2fi9ylQbsysa1yHtsFij9W+VwVTWWLl1DJBlZOAgsAJ4qfKX4mvf2xtqE4saZYTh+nOEKsObrvpQRLfclzG//cL/l+G35VJXN9Bp0of5nKHM0FxgcEpCWGtFQnAMDRBAERYfSDIww9M0BjRwP1rXX0PdHDYx87jxf1OPviMKMfGKZ9pJX2kVZGPzjCmfcPHUgG0H6WBt34XOhaiSuxKszedZqihecGANR5h54WexzZGMAJ0jrYQmIhmde/73gO8W2pbo7rcPEXzzF9Y4bZe2Et+Y6RVrovdO6r33/XL3lqkWqv/xvMJgkelmj8HcHpK3xGGyynCQrN/lRwL7RX/Yt/WOWktxMROs600XEmv1tHRGjpbaalt5lsKkuw4woqCAKyKR8v6lY0DrS/daG3gqbTHC06ZlGNmv5S5+F0N4UTwXaUDvfOn75V1fbDAsAJ0jnazsytOdKJzGYWiLhCQ1s9Lb25uc6u59D7eDe9j+cv07gXlX7J9yt4UKTrB9bP5KNFJ37544tFz/yCiWXcKl4BbNhLEOD2HSJPjVUcBEpJraS5+7f3N9eRjtR5DD07wOpsIswQU0WArvMd9L+nd89BsRqfC2mI4HTUE8wm8xpot0p1rbzHO/HrXPz7S+EYQ52Hd76toiSA08QCwAniei6Pfew8UzdmmBtbxHGEjrNtdJ7bOqtVVZYmlpm+PYef8WkbjNNxtg3X2/uKX0fV+MN6pksR0lpXssKnpvzi2TCluh0qtNUdtLQjCNTnBIGNCWMHFQSCbMD1b90ku61/PZ3IcOv7d8OZwuuNrQJTN2cJfGXofeUvOlPNz4X3ZDfZW3MED8IUX+o9vAsduJ3VKYEhjuCda8cdbQO1WcE7WQA4YdyIS9/lHvou9xR8/P5PHjJ3d35zIlBiPsn0zVke+9j5PS37eJSNP4DT1Yi/lM6/CnAFd6il5BdZGiPobIFFQxxB2osMDldRWERuCcidNbyRIbV91nDkqTHiT+1t1vBu5scXCbKF6+LklVD2lZnbc/Q/2VPW56NUXZ+N328vnwtxhMiFDvR8e04DrWkfXcsi9V7BGd57JSLFB4RrmA0CnyKJhSSz2xp/CL/g6USGqRszZb9OscqNhYp3QfUbf1iv67N9wW8IG/CmKE6Js8NgYY3gfpEMkoiDe0jLYB7lrOHk4lrBbLFixBHShdIld9hL479Z1I3SnwsNlOzYIpkfPSD9w3Ey786SfvMR6b8eI/P6BOnv3yfzzvSpWhPhOLEAcIosTSwX/KJooMyVmVd9EF/y/RDPIfrCAO5IHGnwkMYI7rm2cPJXif7q7PXZot0/kWf7kEMsf3FUQaCuOVYwDbgYDZRIsXLj68eT97l48RfyPhdfui2bn4tipR1y9qtK5o1J/Fvz4UzdZJbg/lK4rnPA+rwAJXi0Svbd2bJ/H1O+qnwbRORlEbkuIjdF5DcKPB4TkX+3/viPRORMNfZrcokjJWrLl24QDupLXgnxHLzRNqIvDRF9cRBvOF7y99Bd1gTIvvGIYHn/tYD24yiCQNtwvOgM6Z3vnzhC21AcL1q4m6WadX120rm1sMjbbmf3gRJMrKB7uKox5ak4AIiIC/wu8EngMvB5Ebm8Y7O/B8yr6nng/wT+eaX7NflaB+IU6ugUV+g8Wzzt7SC/5IdKKNnPq4kMmR9PoMnduzuq6bCDgOu5XPzIKHUtMcQVHNfBjTr0PdlDU2cjSJg6HNaSamH42cKLzhz058KfTeSXeS6l2Oxus2/VuAJ4AbipqrdVNQ38GfCZHdt8Bvjj9Z+/BHxUbBZG1cWaovQ92ZOzcIzjOjS01dNZpO7JqWn8ATIB0rbLIG+gZKswy3SvtgeBjdIR24OA1HfnBYGG0aF9B4H6ljouf+Iilz9xkTMvDuJ6Lo+uTbM6l0AkXJP4yf/iEmffP1xwLsBhfC7Ec/Y2MFvkKsXsXzUCwABwf9vt8fX7Cm6jqllgEeiowr7NDr2Xunjso+fpvthJx9k2zr44xMUPjxZcLvK0NP6aDcJyz389hi7u0sWjoNvqJh2mQvWDNoLA5qzh0bN5s4Y3uuf2I1LvMXblAelEhiAbEGQDNFDm7y+w/KjwRLvD+ly4vU3lVahz1jO/TkjxwpOkGu9oob/gzuu6crYJNxR5RUSuiMiV+VSBVD6zq/p4HYNP9zHy3CDx/sIrQZ2Wxh8g+/YUwcy2gcNdSIkBz4N22EXkliaWc7LCNgS+MnltOu/+fRV122e9J2mI5NbwF8KfmyJhy+RImPY7Esc9ZzN3D0I15gGMA0Pbbg8CD4tsMy4iHhAH5gq9mKq+CrwK8ER7j+V+7WJzRvAeJriUX9StOnV9DpKuZQnm1kqWQc7hhA3K5vM1rCxK1keaY4eSJXSYpSPSyUzRFMrMWu7iKHst+VGNz4U32ILb2YA/HZZrcDoacJqi4TFnfIi4xSf8ZXzIBGFdH5vgtS/VCACvARdE5CzwAPgc8F/t2OYrwK8BfwN8FviWWim+iqRW09z/8QOWpsLL+OauRoaeGaCuuXSZg9PU+ANoMru++kqRDSJOTpaJe6kDZ70UhCYyZH76CF3Lbi4W4o7Ecc+2Frxq2lyYZm4NiTq4vU1IbH9focMKAo1tDYgUjo8NbfWbP+9v4l91PhdS5+EN5RYpFEegyHuraZ/M29PofDLsQnIE90I7ni39uGcVn+6s9+l/AfgacBX4oqq+LSL/VEQ+vb7ZHwAdInIT+F+AvFRRU75s2uf6N26y9Ggl/GYrLE+tcv2bt8iUseTdRpfCdhtdD9ttVLc8zqTBK55G6DlEfm6IyPv6iDzdS/TnR/DWJ4JpoKRfnwjrxW/Wolf8e4v4kyvhOgTLqc2Zs+oHZH48QeZnUwRji/i35kn/YBx/ap/VSjmcSqIN7fXUt9bnp3+6Qv+TPTmvdRCzvtUPCGYT+DOJkuU9yqWqZF6fCBt/JfzbZQP867P404mKX7/WVKUUhKp+Ffjqjvt+a9vPa8CvVmNfBmbvzuEXWvbPD5i5NVu0TMRpJDEPp6uRYDqRX1DsTGs4+F2g/n8wl9xcVD73AcV/ZwZ/I5PKc4i8p5tgNokup7auNBRQJfv2DE57w767jiotIleO/l91ePuvPe5fzeBnId7p8J6fj9ExMLW5zUE0/v70Ktm3t40zKLiPdWwG4d1oMkOwmgmrejaF5b11MRVese2M+YHi357H7apODaFaYbWATqCVmUTBJR81UFZmau8syLvcRfbd9WUjAQTcM624wyUaqUKNyHYb76/vk/nJJLhO4W4mgWA6gVukNHU59ltEbi9eugQvqhYsiLZz1jdU4cw/mSH71nTe1Zl/bRanKYpToqtSA10f2E9uds1JY4TI073hFVuRv9thz+84DSwAnEB1jVEW178YOQTqmoovlH5aiSNEHutEL7SHg4LR4gOHm88ptGBMMUrhq4WNx4LqzFDdaxE5gMyb5Qf8QgXRqlXUbafs+FLhktyB4t9fwrnclf/YxnNvzoWN/7bgoctpMm8+wjvfXnTugNRbc7ZX9o6dQJ3nOpi+NZuX3ieO0HW+dqdXiOuEZ+rlbNsSQ5qjYSmC3dIRgrBMMcnC4ytOe33B+/djo0votW/kBgFIcD21mBMEgqmxzQa8ErvVe9pX+u9a8ZLcupYNxwYerYZdd56DO9CM01qHqoaloQvVtFpJo1EHqfMKXgmorwQLazitB1/x9bSwAHACxZqinH1phLs/Gts8yRKBkecHqSu2Fq7JISJE3ttL9sZsuLZwoOAJFCqj7ApOfxPB2FJ4JaDb72+u+ryCvQSBiuVlgG0MRFdW70na6mAmkd+QO4LEY2ReexhmcK0/HkythhlYI/HiyzgKSNon8kwf6Z89goUdk/6SWTI/mSTybN9mppcpzQLACRXva+apT19mdTaBAo0d9QVn+5oSsgFOcwxpjuF01CNA+kcP8ieTuU6YptjXTHZsMVxrIOLiDreULE1diXKDQDUcxMQ/t7cJ/+5C/qLvrkBATuMPbGZgOb2NYcmHQovFK0hjFIm6RM61h2MzOwPM+mCw897ePR9zLbIAcIKJIzR1NR71YZxI2dvz+NvWDfZvhOsFR57pI3ttZrOqqLTWEXm8c7N7KXKhAy4czjHuFgQea63OgRRq/L//H7J4iSxtLfvL8xfPIfp8P9l35whmVkHB6WjAu9hOulDDDaBKMJPEPd+Of3UmL6vL6W9C1usBBUup4st+HnLF15PMAoCpOZm7CwR3FvLu92/M4bzQT/SFgTBnXTjy+jPbg8CHPuHwpdsLm0Fgq7++chuNf4w4V35niZaZFOIIGV3ZzMCR2N6KsUnMI/Ke/PklhfIXNh8AvN4mRCB7cz7M1oo4uMPxnBncEvPCUhEFsuEkas1auazPwNSUYDZBcGu+yIOK/3AZCM9gj7rx31C8iFx1/m0/83/tD1aJzqYQZXNyXJiBM1m138fpbcpd6W2TbObxuz1NxD44RPQXzxD7+RG8M7mzs52uhsKv4QjumXj+/aag4/EJN+aQZG8ULEG1JX08Fx0pXESu8mKJOSU//pNP/eQaUuAt0JUMwWrxxXb2wh2OhymbO5b7dM/E8wbUi1WNF0eIPNMXLhvqhkXjcAjHZbqtW7Rcdq1kaoqulpgs5IDTUb2UzmorPGu48iCwUdenu66FtF8kQAphd0xj5fNMxHOIPN9PMLUjDTS+tww2pylK9IND6FIKzQQ48VhVFpCvJRYATG2JOOFksUJiXsVnj5oNwq6TqFNy7eL92hkEKpWX6VPnhjn8O6kiVZxkKK6D29eMW2ZZiKKvI4LsMXCYLRYATE1xh+JheuLOLBRPiDzfv++ywrqWJfPONLqwvthMzCPyeGdVJ4lt2B4EqvV6G9xzRTJwehr3VfnUn0ng31tA13ycliju2bbNuj7m6FkAMDXFPRMPi4xNrmz2QUvMI/Lenn13H2igpK88zM1dX8uS+ekjIs/1lax7s18HVZ7b6w1rGvm35sIrATdcjcstsaZ0Mdn1qqmbk73WsgQzSSLP9O65u8ccDAsApqaICJHLXehoG8FKGom6SHO0ou6aYDpRvLLonQWcp05GdVbNBujiGk7Mw3muH13NIA0eTt3eZzprNshp/DcFSuYnk0R/bvhQFt8xpVkAMDVJ6jzcuup8/HU1XXQpymBl75kz6gfhugQPl8FXnI56vHNtB7qUZfb+Ev7NOUC3qp6uXyE57fV4T3ahC2tkb82jiQwS83DPtobr+hb6HZZSxRd895XM21NEn7bZukfNAoAxFZJ6L0xDLBAEnIb8Rls1zK0H8q4+VMMz5O1rDwSPVknPJom+fwDZEbRUNayRn8wgjdF91cAJ5pNh41/gbB3CtRMyr0+ES2eu36eJDNmrM+haFu9MgcFot/QVlc4k0YxvWTtHzAKAMRVyuhvhxlx+AFhflGa7YD5J5q2prW0dwbvchbteU0jn18IyFDt7lPyA7L1FIpe2qr1qKix+phtr+ypIU5TI+3oRzwm7dJZTEHVxSqRv+vcWi6+qBuFksKUCVzLrXVzuUEvepDlpiRUNigC4gmYCCwBHzAKAMRXQQPEnlsMCZhuVQh3AcXAvteeUJta1LJk3HuU2tr6S/dkUQV8jupAKFzov1GgqBJMr+B31YeE6ETI/m8qb16DLKTLvTOE0xcKGfWNBlXqPyNM9BbuRdG33ZUSLkrBM885UTBHBe6qH7JWJ4k+tUhec2T/7CxizT6pK5o1JdDG11agLYQro8/04O85uSy2SEjwoY23hbED2ralwEfUnuja7kXIPCnQ6iT+7lrugymqG9OuTRD8wmDfgLfFY6QlypShFz+LdeB3BaBvBnfnc4j+O4I627Tvl1lSPDcMbs0/BTDIc7Nx+Rq9Ayid4tJr/hBLLGZbN17D//dZ88UFWKNylk/HR+bW8u70zraX77B2BBi9/fxIu1SgFxjm2XjuOe6kjLNkAEHNxL7XjDVu9nuPArgCM2adgerVwd02gBFOrMJibqy8tMZhNlu5vL4eCziXDVYD2wleyt+aItOZOeJP6CJFn+8henw2vZmCrsXcEdziOM9RC5o1J2D4WEHWJ7JLiKiJ4Ay14Awczb8FUxgKAMftVolqoFDijdgeadx9wLZeCMxonuLPj9RwJxyOK9Ovrcprs7Xki59tz7neaY0Sf60e3d1FlA/DCkhaa9sOJbg5bA9SZAH8mgTdojftJVVEXkIi0i8hficiN9f8LThcUEV9E3lj/95VK9mnMceH2FSlr7ApOgTNeibhhuYnWWHiGLYS1d/bRFS7xGJEzbbiPdWxW1pTGCN6TXUQe7yz+zVYIxpdyG/rtryuy9S/ibo4XZG/NQdrPzU4KFP/GXDhwbU6kSq8AfgP4pqr+toj8xvrtf1Rgu6SqvrfCfRlzrDgtMdyReO5Z/XrdnGJVRZ2GCNFn+1F/I2NIyPx0fSDZ1/xgUKiddgTvYpgO6vU1Q4GCau5IK36BRW+AcD/KngJPMJUofCwCwWyy6IQwc7xVGgA+A3x4/ec/Bv4zhQOAMaeSN9qG09NI8GgVVcXtaixrMtb2vPnIe3vRhTX82STiOThtdWTfnduaTetIeKUQgBOP4Z5pzcvrV9X1+QOKNMfC7qa7C4Ub7TrPMnAMUHkA6FHVCQBVnRCR/PXfQnUicgXIAr+tqn9e4X6NOTacxijO6P4rXIoI0laP01YfFpb7wf2twnIbK3MlfSLP9uUFF396ley7c1t9/usBw3u8E6eviWByNW+MwLuw98JuTncDwcRKfkDR472Ggilt1wAgIt8AChXt+M097GdYVR+KyCjwLRH5mareKrK/V4BXAPoaKqsVbsxJE0yvFi8sdze3sJz/aIXsOzP5aai+kn1nBu+ZXqQugn9/ETIBUu/hXmjH7dr7mgfeuXbSs8lwLYVt3V3uxXabzXuC7RoAVPVjxR4TkUci0rd+9t8HTBV5jYfr/98Wkf8MvA8oGABU9VXgVYAn2nuqkC5hzMmhq5myCsuparhoerGMokAJ7i8RebIb72wrqlpRxVOJukRfHMSfWEbn1sJ8/oEWnGar7X+SVToR7CvAr63//GvAX+zcQETaRCS2/nMn8EHgnQr3a8ypJA2RopOycgrLBVo01XODJrZm91ZjdTLxHLyhOJGne4g81mmN/ylQaQD4beDjInID+Pj6bUTkORH5/fVtHgeuiMhPgW8TjgFYADCmAKeroXAAcAT3bGvO7YIpqNvIPiqDmtpS0SCwqs4CHy1w/xXg76///APgPZXsx5haIa5D9Nn+sNBbIrM1qHupI2cVLRHBGWgmeLCUXzkUwBW8ESu3YEqzmcDGHDPSECH6/gF0LYtmg7DeToEuHO98O9lklmAukRsEGiNELncd6AIy5nSwAGDMMSV1Xsm5WuIIkad70EQmHCCOOEhTNK8KqTHFWAAw5oSThghuiYqcxhRj5aCNMaZGWQAwxpgaZQHAGGNqlAUAY4ypURYAjDGmRlkAMMaYGmUBwBhjapQFAGOMqVEWAIwxpkZZADDGmBplAaAGJW7f3/w5mBoDQJPb1/JJALDmby0q/ii5dCjHZow5PBYAasz9yTNAGAQyb4YNffDD7wBhEHis9QIAnx0NV5pa8xf40CfCj4kFAWNOFwsANciCgDEGLADULAsCxhgLADVsL0Hgs6P1FgSMOWUsANS4+5NnuD95Jj8I3L6TEwQgYUHAmFPGAoAB2AwCi39+HVjPDloPApdiG2vLbgWB5z8WNv4WBIw5uSwAmE0bXUI7gwCQFwQACwLGnHAWAEwOCwLG1I6KAoCI/KqIvC0igYg8V2K7l0XkuojcFJHfqGSf5uBZEDCmNlR6BfAW8HeB7xbbQERc4HeBTwKXgc+LyOUK92sOmAUBY06/igKAql5V1eu7bPYCcFNVb6tqGvgz4DOV7Nccju1BIPNmIi8IhBlCuUHgQ59wLAgYc0IcxhjAAHB/2+3x9fvMCbBzrsD2ILCVJprImzD2KLlkgcCYY27XACAi3xCRtwr8K/csXgrcpyX294qIXBGRK/OpZJm7MAepUBCwWcPGnHy7BgBV/ZiqPlng31+UuY9xYGjb7UHgYYn9vaqqz6nqc22x+jJ3YQ6alY4w5vQ5jC6g14ALInJWRKLA54CvHMJ+TZVZEDDmdKk0DfRXRGQceAn4jyLytfX7+0XkqwCqmgW+AHwNuAp8UVXfruywzVHZCALAZhDYGBMAtgWBrau3jSBgjDlevEqerKpfBr5c4P6HwC9vu/1V4KuV7MsYY0x12amZMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1ygKAMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjKgoAIvKrIvK2iAQi8lyJ7e6KyM9E5A0RuVLJPo0xxlSHV+Hz3wL+LvB/l7HtR1R1psL9GWOMqZKKAoCqXgUQkeocjTHGmENzWGMACnxdRH4sIq8c0j6NMcaUsOsVgIh8A+gt8NBvqupflLmfD6rqQxHpBv5KRK6p6neL7O8V4BWAvobmMl/eGGPMXu0aAFT1Y5XuRFUfrv8/JSJfBl4ACgYAVX0VeBXgifYerXTfxhhjCjvwLiARaRSR5o2fgV8iHDw2xhhzhER1/yfZIvIrwL8CuoAF4A1V/YSI9AO/r6q/LCKjwJfXn+IBf6Kq/6zM158GVgHLHgp1Yu/FBnsvtth7scXeCxhR1a5yNqwoABwGEbmiqkXnGNQSey+22Huxxd6LLfZe7I3NBDbGmBplAcAYY2rUSQgArx71ARwj9l5ssfdii70XW+y92INjPwZgjDHmYJyEKwBjjDEH4EQEABH5FyJyTUTeFJEvi0jrUR/TUSm3AutpJSIvi8h1EbkpIr9x1MdzlETkD0VkSkRqel6NiAyJyLdF5Or6d+N/POpjOilORAAA/gp4UlWfAt4F/vERH89R2qjAWnAm9WkmIi7wu8AngcvA50Xk8tEe1ZH6I+Dloz6IYyAL/K+q+jjwIvAPavxzUbYTEQBU9euqml2/+UNg8CiP5yip6lVVvX7Ux3FEXgBuquptVU0DfwZ85oiP6cis19OaO+rjOGqqOqGqr6//vAxcBQaO9qhOhhMRAHb474C/POqDMEdiALi/7fY49kU324jIGeB9wI+O9khOhkoXhKmacqqOishvEl7u/dvDPLbDVqUKrKdRoYUnLI3NACAiTcC/B/4nVV066uM5CY5NANit6qiI/BrwKeCjespzV6tRgfWUGgeGtt0eBB4e0bGYY0REIoSN/79V1f/vqI/npDgRXUAi8jLwj4BPq2riqI/HHJnXgAsiclZEosDngK8c8TGZIybhkoR/AFxV1f/jqI/nJDkRAQD4HaCZcDGZN0Tk9476gI6KiPyKiIwDLwH/UUS+dtTHdFjWEwG+AHyNcKDvi6r69tEe1dERkT8F/ga4JCLjIvL3jvqYjsgHgf8G+MX19uENEfnloz6ok8BmAhtjTI06KVcAxhhjqswCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yN+v8Bo+UrVvPjKW4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# visualize decision boundary\n",
    "\n",
    "h = 0.25\n",
    "x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
    "y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
    "                     np.arange(y_min, y_max, h))\n",
    "Xmesh = np.c_[xx.ravel(), yy.ravel()]\n",
    "inputs = [list(map(Value, xrow)) for xrow in Xmesh]\n",
    "scores = list(map(model, inputs))\n",
    "Z = np.array([s.data > 0 for s in scores])\n",
    "Z = Z.reshape(xx.shape)\n",
    "\n",
    "fig = plt.figure()\n",
    "plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)\n",
    "plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}


================================================
FILE: micrograd/__init__.py
================================================


================================================
FILE: micrograd/engine.py
================================================

class Value:
    """ stores a single scalar value and its gradient """

    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0
        # internal variables used for autograd graph construction
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op # the op that produced this node, for graphviz / debugging / etc

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')

        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward

        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')

        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward

        return out

    def __pow__(self, other):
        assert isinstance(other, (int, float)), "only supporting int/float powers for now"
        out = Value(self.data**other, (self,), f'**{other}')

        def _backward():
            self.grad += (other * self.data**(other-1)) * out.grad
        out._backward = _backward

        return out

    def relu(self):
        out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')

        def _backward():
            self.grad += (out.data > 0) * out.grad
        out._backward = _backward

        return out

    def backward(self):

        # topological order all of the children in the graph
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._prev:
                    build_topo(child)
                topo.append(v)
        build_topo(self)

        # go one variable at a time and apply the chain rule to get its gradient
        self.grad = 1
        for v in reversed(topo):
            v._backward()

    def __neg__(self): # -self
        return self * -1

    def __radd__(self, other): # other + self
        return self + other

    def __sub__(self, other): # self - other
        return self + (-other)

    def __rsub__(self, other): # other - self
        return other + (-self)

    def __rmul__(self, other): # other * self
        return self * other

    def __truediv__(self, other): # self / other
        return self * other**-1

    def __rtruediv__(self, other): # other / self
        return other * self**-1

    def __repr__(self):
        return f"Value(data={self.data}, grad={self.grad})"


================================================
FILE: micrograd/nn.py
================================================
import random
from micrograd.engine import Value

class Module:

    def zero_grad(self):
        for p in self.parameters():
            p.grad = 0

    def parameters(self):
        return []

class Neuron(Module):

    def __init__(self, nin, nonlin=True):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(0)
        self.nonlin = nonlin

    def __call__(self, x):
        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)
        return act.relu() if self.nonlin else act

    def parameters(self):
        return self.w + [self.b]

    def __repr__(self):
        return f"{'ReLU' if self.nonlin else 'Linear'}Neuron({len(self.w)})"

class Layer(Module):

    def __init__(self, nin, nout, **kwargs):
        self.neurons = [Neuron(nin, **kwargs) for _ in range(nout)]

    def __call__(self, x):
        out = [n(x) for n in self.neurons]
        return out[0] if len(out) == 1 else out

    def parameters(self):
        return [p for n in self.neurons for p in n.parameters()]

    def __repr__(self):
        return f"Layer of [{', '.join(str(n) for n in self.neurons)}]"

class MLP(Module):

    def __init__(self, nin, nouts):
        sz = [nin] + nouts
        self.layers = [Layer(sz[i], sz[i+1], nonlin=i!=len(nouts)-1) for i in range(len(nouts))]

    def __call__(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

    def parameters(self):
        return [p for layer in self.layers for p in layer.parameters()]

    def __repr__(self):
        return f"MLP of [{', '.join(str(layer) for layer in self.layers)}]"


================================================
FILE: setup.py
================================================
import setuptools

with open("README.md", "r") as fh:
    long_description = fh.read()

setuptools.setup(
    name="micrograd",
    version="0.1.0",
    author="Andrej Karpathy",
    author_email="andrej.karpathy@gmail.com",
    description="A tiny scalar-valued autograd engine with a small PyTorch-like neural network library on top.",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/karpathy/micrograd",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)


================================================
FILE: test/test_engine.py
================================================
import torch
from micrograd.engine import Value

def test_sanity_check():

    x = Value(-4.0)
    z = 2 * x + 2 + x
    q = z.relu() + z * x
    h = (z * z).relu()
    y = h + q + q * x
    y.backward()
    xmg, ymg = x, y

    x = torch.Tensor([-4.0]).double()
    x.requires_grad = True
    z = 2 * x + 2 + x
    q = z.relu() + z * x
    h = (z * z).relu()
    y = h + q + q * x
    y.backward()
    xpt, ypt = x, y

    # forward pass went well
    assert ymg.data == ypt.data.item()
    # backward pass went well
    assert xmg.grad == xpt.grad.item()

def test_more_ops():

    a = Value(-4.0)
    b = Value(2.0)
    c = a + b
    d = a * b + b**3
    c += c + 1
    c += 1 + c + (-a)
    d += d * 2 + (b + a).relu()
    d += 3 * d + (b - a).relu()
    e = c - d
    f = e**2
    g = f / 2.0
    g += 10.0 / f
    g.backward()
    amg, bmg, gmg = a, b, g

    a = torch.Tensor([-4.0]).double()
    b = torch.Tensor([2.0]).double()
    a.requires_grad = True
    b.requires_grad = True
    c = a + b
    d = a * b + b**3
    c = c + c + 1
    c = c + 1 + c + (-a)
    d = d + d * 2 + (b + a).relu()
    d = d + 3 * d + (b - a).relu()
    e = c - d
    f = e**2
    g = f / 2.0
    g = g + 10.0 / f
    g.backward()
    apt, bpt, gpt = a, b, g

    tol = 1e-6
    # forward pass went well
    assert abs(gmg.data - gpt.data.item()) < tol
    # backward pass went well
    assert abs(amg.grad - apt.grad.item()) < tol
    assert abs(bmg.grad - bpt.grad.item()) < tol


================================================
FILE: trace_graph.ipynb
================================================
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# brew install graphviz\n",
    "# pip install graphviz\n",
    "from graphviz import Digraph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from micrograd.engine import Value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trace(root):\n",
    "    nodes, edges = set(), set()\n",
    "    def build(v):\n",
    "        if v not in nodes:\n",
    "            nodes.add(v)\n",
    "            for child in v._prev:\n",
    "                edges.add((child, v))\n",
    "                build(child)\n",
    "    build(root)\n",
    "    return nodes, edges\n",
    "\n",
    "def draw_dot(root, format='svg', rankdir='LR'):\n",
    "    \"\"\"\n",
    "    format: png | svg | ...\n",
    "    rankdir: TB (top to bottom graph) | LR (left to right)\n",
    "    \"\"\"\n",
    "    assert rankdir in ['LR', 'TB']\n",
    "    nodes, edges = trace(root)\n",
    "    dot = Digraph(format=format, graph_attr={'rankdir': rankdir}) #, node_attr={'rankdir': 'TB'})\n",
    "    \n",
    "    for n in nodes:\n",
    "        dot.node(name=str(id(n)), label = \"{ data %.4f | grad %.4f }\" % (n.data, n.grad), shape='record')\n",
    "        if n._op:\n",
    "            dot.node(name=str(id(n)) + n._op, label=n._op)\n",
    "            dot.edge(str(id(n)) + n._op, str(id(n)))\n",
    "    \n",
    "    for n1, n2 in edges:\n",
    "        dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
    "    \n",
    "    return dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a very simple example\n",
    "x = Value(1.0)\n",
    "y = (x * 2 + 1).relu()\n",
    "y.backward()\n",
    "draw_dot(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a simple 2D neuron\n",
    "import random\n",
    "from micrograd import nn\n",
    "\n",
    "random.seed(1337)\n",
    "n = nn.Neuron(2)\n",
    "x = [Value(1.0), Value(-2.0)]\n",
    "y = n(x)\n",
    "y.backward()\n",
    "\n",
    "dot = draw_dot(y)\n",
    "dot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dot.render('gout')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}