Repository: karpathy/micrograd
Branch: master
Commit: c911406e5ace
Files: 10
Total size: 68.9 KB
Directory structure:
gitextract_jag07i_y/
├── .gitignore
├── LICENSE
├── README.md
├── demo.ipynb
├── micrograd/
│ ├── __init__.py
│ ├── engine.py
│ └── nn.py
├── setup.py
├── test/
│ └── test_engine.py
└── trace_graph.ipynb
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
.ipynb_checkpoints/
================================================
FILE: LICENSE
================================================
The MIT License (MIT) Copyright (c) 2020 Andrej Karpathy
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: README.md
================================================
# micrograd
A tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode autodiff) over a dynamically built DAG and a small neural networks library on top of it with a PyTorch-like API. Both are tiny, with about 100 and 50 lines of code respectively. The DAG only operates over scalar values, so e.g. we chop up each neuron into all of its individual tiny adds and multiplies. However, this is enough to build up entire deep neural nets doing binary classification, as the demo notebook shows. Potentially useful for educational purposes.
### Installation
```bash
pip install micrograd
```
### Example usage
Below is a slightly contrived example showing a number of possible supported operations:
```python
from micrograd.engine import Value
a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
print(f'{g.data:.4f}') # prints 24.7041, the outcome of this forward pass
g.backward()
print(f'{a.grad:.4f}') # prints 138.8338, i.e. the numerical value of dg/da
print(f'{b.grad:.4f}') # prints 645.5773, i.e. the numerical value of dg/db
```
### Training a neural net
The notebook `demo.ipynb` provides a full demo of training an 2-layer neural network (MLP) binary classifier. This is achieved by initializing a neural net from `micrograd.nn` module, implementing a simple svm "max-margin" binary classification loss and using SGD for optimization. As shown in the notebook, using a 2-layer neural net with two 16-node hidden layers we achieve the following decision boundary on the moon dataset:
### Tracing / visualization
For added convenience, the notebook `trace_graph.ipynb` produces graphviz visualizations. E.g. this one below is of a simple 2D neuron, arrived at by calling `draw_dot` on the code below, and it shows both the data (left number in each node) and the gradient (right number in each node).
```python
from micrograd import nn
n = nn.Neuron(2)
x = [Value(1.0), Value(-2.0)]
y = n(x)
dot = draw_dot(y)
```
### Running tests
To run the unit tests you will have to install [PyTorch](https://pytorch.org/), which the tests use as a reference for verifying the correctness of the calculated gradients. Then simply:
```bash
python -m pytest
```
### License
MIT
================================================
FILE: demo.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### MicroGrad demo"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import random\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"from micrograd.engine import Value\n",
"from micrograd.nn import Neuron, Layer, MLP"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"np.random.seed(1337)\n",
"random.seed(1337)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<matplotlib.collections.PathCollection at 0x10f636400>"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAU4AAAEyCAYAAACVsznTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzs3Xd4VFX6wPHvmZJJJiGQhNB7C0iVqoCKAiIgAiqCiqKyshYsu7qKrq4/62Jd7IoKCqKAsDakF3UVpUqXEppAaCFAyiSZdn5/ZIgZkkDKJHcm836eJ09y75x775tJ8ubce5rSWiOEEKLkTEYHIIQQoUYSpxBClJIkTiGEKCVJnEIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUrJYnQAZVGzZk3dpEkTo8MQQlQx69atS9VaJ56vXEgmziZNmrB27VqjwxBCVDFKqf0lKSe36kIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUpJEqcQQpSSJE4hhCglSZxhzOPxsmfPSY4ezTQ6FCFCSkiOHBLld+RIJpdf/gl//HGanBw3JpOiRg0bEyf2Y+zYzkaHJ0RQkxpnmLrttq9ITk7D4XDh9Wrcbi+pqdncf/9CFi/ebXR4QgQ1SZxhasOGI7jd3kL7HQ4XX3+93YCIhAgdkjjDVPPmcZhMqtB+q9VEzZp2AyISInRI4gxTH388jISEKKKjrfn7IiJMJCTYGT++u4GRCRH8pHEoTLVsmUBy8v2sW5fC4cMZ7N17ipiYCEaP7kBCgtQ4hTgXSZxhLDbWxuWXN83f3rEjldWrD9G6dU2aNo0zMDIhgpvcqocJrTVvvLGKZs1ep3nz1/nww/V+r0+c+BMXXvg+o0bNpW3bd5g2bUOpr3HqVA4//rifLVuOobUOVOhCBB0Vir/gXbt21TIDfOl8+OF6HnhgIQ6HCwC73cqUKdcwcmQ7du9Oo337d8nOdueXj4y0cOTIQ1SvHlmi8//222GuuGIaWmtcLi/XXdeGTz4ZhlKFG6CECFZKqXVa667nKyc1zjAxZcpv+UkT8rodffxxXq1y375TRESY/cpbLCYOHy75iKLrr/+CU6dyOH06F4fDxeefb2Hu3N8DE7wQQUYSZ5iIiYkoYp8NgDZtEnG5/Pt0mkzQqFH1Ep9///5Tfttut5f33ivfXUFGRi4rVx5gx47Ucp1HiECTxBkmnnnmcuz2P7seRUdbeeKJSwCoV68a06cPJyrKQnS0ldhYG998c6Nf+fOJjCzcznjwYHqZ49206ShNmrzOwIEzuPDC97n99q/kuakIGtKqHiYuuqgBK1fewdSpGzCZFHfe2Zk2bf5cBfXaa9tw4sQjHDmSSb161bDZSverceWVzfnyS/8RRx061C5zvCNGfEFaWnb+9hdfbGPo0NYMG9a6zOcUIlAkcYaRjh3rMGnSVcW+HhVlLXM3pLffHsTPP/9BZqYL0NjtEbz8cv8yRpr33LWg3FyP3LKLoCGJUwRE3brV2LHjPhYs2IXWcNVVLYiPj8Lt9vLDD/vIyHDSs2dDatWKLtH5WraMZ9u245y5O7fZzLRrV6sCvwMhSk4SpwiYGjUiufHG9vnbTqeHPn0+ZvPmY/nj4n/44TY6dapz3nPNmXMDffp8jMPhwun0MHZsZwYNallhsQtRGpI4RYX58MP1bNhwxK9/6JgxX7Fx413nPbZ165rs2/cgu3adID4+ivr1YysyVCFKJSCt6kqpKUqpY0qpLcW8rpRSbyilkpVSm5RSnQu8NkYptcv3MSYQ8YjgsHfvKb+kCcW3tDudnkKt5pGRFtq3ry1JUwSdQHVH+hgovtUBBgItfR/jgHcBlFLxwFNAD6A78JRSSgZJVxEXX9zAb/Ylq9VE9+71/MocOZJJ166TiYp6nujoF5g69bfKDlOIUgtI4tRa/wiknaPIUGCazvMrUEMpVRcYACzRWqdprU8CSzh3AhYhZPjw1tx3X3csFhMREWbatq3FtGnD/cpce+0sNm48iteryc52M378AlavPmRQxEKUTGU946wPHCiwfdC3r7j9hSilxpFXW6VRo0YVE6UIKKUU//53P5588jKys13Ex0cVGru+Zk2K30z0breXn3/+g+7di/w1ECIoVNbIoaJmetDn2F94p9aTtdZdtdZdExMTiyoigpTdbiUhwV7khB9xcf6TiFitJurUiams0IQok8pKnAeBhgW2GwAp59gvwsTUqUOx2/OGesbERNC5c11GjGhrdFhCnFNl3ap/A4xXSs0kryHotNb6sFJqEfBCgQahK4HHKimmkJSSksGmTUepX78a7duXfUhjsBg8uBXr1/+Vn376g5o17Qwe3AqLpWT/z7dvT2XVqoPUqRND//7Ni1xDSYiKEJDEqZT6HOgD1FRKHSSvpdwKoLV+D5gPDAKSAQdwu++1NKXUs8Aa36me0Vqfq5EprC1YsIsRI77AYjHhcnkZN64L//nPAKPDKrekpJokJdUs1TFz527j1lu/RCmFUoo+fRrz9dc3SvIUlUImMg4RXq+mevWJZGY68/dFR1tZuvRWLrqogYGRVT6tNbGx/u9FTIyV22/vxOzZ28jN9XDjje14/fWrsFrN5ziTEP5kIuMqJj09l9xc/87kJpNiz56TBkVkHJfL6zcp85l977+/nqNHszh1KoePP97AhAnLDIpQVHWSOENE9eo24uKi/Pa53d5ST93m9Wpef/1XrrxyOnfc8TUpKRmBDLNSRESYad26pt9tudvtxen05G9nZ7uZO3ebEeGJMCCJM0QopVi48GZq1sxbC91mM/Paa1eWesagv/1tIY8/vpwlS/YwffpGLrzwfb95L0PF/Pk3kZSUgNmsiIqyMHBgi0KNSiVdL0mI0pJnnCHG5fJw4EA6iYl2qlWzlepYr1cTGfmc3zIZ0dFW3n57EGPGdAp0qJUiO9tFZKSFo0ez6NDhXU6fzsXj8WKzWZg370a/5Y+FOJ+SPuOU2ZHOw+vVvPbaL8yevZWEBDsvvtivXDObl5fVaqZZs7IP5z/7/6TW4PGE3j/PM6Ki8sbC16kTw5Yt9zB9+kYcDhdDh7Y29OckqjZJnOfx+OPLePPN1TgcLpSCn376g40b7ypX8jKKyaQYPbo9s2dvxeFwYzIpbDYzgwdXjXkua9WK5qGHehodhggDkjjP47331ua34GoNublu5szZxiOP9KrQ6+7de5I5c7ZhMilGjmxHWlo2S5fuoXp1G6NGtSM6uvCqlSXxwQfX0KBBLAsWJFO/fjVefXUAtWvLEEchSkMS53mc3aFaqcL7Am3z5qP07DmF3Fw3SsG//rUCrzfvsYHFYuKll1ayfv24MiVPi8XEs89ewbPPXlEBkZdOZqaTjIxcateOkY7rIqRIq/p5PPRQz/xlck0mRVSUlVGj2lXoNSdMWEpWlhOXy4vT6cXhcJOT48bp9OBwuDhw4DQff7yhQmOoaM888wMJCS/SrNkbJCW9xYEDp40OSYgSkxrneTz+eG/q1o1h9uytJCba+b//60ODBhU7I3lqqqNQI05BubluUlMdFRpDRVqyZDcvvfQzTqcX8LJ370muv342q1bdaXRoQpSIJM7zUEpxxx0XcscdF1baNUeMaMuWLcfzn62aTAqTSeXPWxkZaaFfv2aVFk+grV2bQk7On6OgPB7Npk3HDIxIiNKRW/Ug9Pe/X8x993X3jRaK5LHHejNwYAtsNjPx8VG8//4QevUK3cmcGzeuQWSk///sunWlgUqEDukAH0BaaxYsSGbbtuO0bl2TwYNbFjl5b7jzeLxcc83n/PDDfsxmE1prFi++JewmKxHBRzrAG+D++xcwdeoGnE4PERFmbrmlA+++e7XRYQUds9nEvHk38fPPBzh5Mpvu3etLlygRUqTGGSD795+ideu3/Z7dRUZa2LLlbpo3jzcwMiFEScm0cpUsLS2biAj/uR8jIsycOBF6E2gIIc5NEmeAJCXVLJQ4rVYTTZrU4Pbbv6J27Vdo0+YtVqzYa1CEoS0lJYMtW4751eiFMIokzgCx2618//0YkpISsFhMtGwZz/LlYxg/fj4zZ27l2LEstm8/wdVXf87vvx83OtyQ8ve/L6JZs9fp2fMjGjeexPbtqUaHJMKcNA4FUNu2tdi+fbzfvm+/3elXS3K7vSxcmEybNrLEcUksWLCLyZPXkZvrITfXQ2amk2uvncW2bfcaHZoIY1LjrGCRkf637xaLiZiYsk3QEY62bDnmN7O71pCcLOv5CWNJ4qxgL77YP3+se0SEmVq17IwcWbFj3auSVq0SCj07btSoukHRCJFHbtUr2LhxXWjatAYLFiRTq1Y0d93VldjY0s3cHs6uuSaJG25oy8yZW7BazZjNirlzb6jw62qtmTjxJ15+eSVer2bcuC5MnNhPZnESgPTjFCFix45UTpzIpl27WpXyj+fjj3/j3nsX5M8XYLdbeeKJS3nssd4Vfm1hHBk5FKL27TvF3LnbUEoxcmRb6tev2JmYQkVSUs1Kvd4XX2zzW4LY4XAxd+42SZwCkMQZVLZuPcbFF3/kWz9d8cwzP7Bu3TgZeWSAmjWjMZnA++e6diQkRBV/gAgr0jgURB59dCmZmU6czrw1wjMynPzrXyuMDissPfXUZcTGRmKxmPKXIH7ppf5GhyWChCTOIHL2BMZer+b48dCdsDiUNWsWx733dkMpfDM4wVdfbTc6LBEkApI4lVJXKaV2KKWSlVITinj9P0qpDb6PnUqpUwVe8xR47ZtAxBOqrr/+gvyuS5DXIHHddRcYGFH4Ono0k1dfXelbvsRDTo6biRN/Zt++U+c/WFR55X7GqZQyA28D/YGDwBql1Dda621nymit/1ag/H1AwenUs7XWncobR1Xw979fzPHjWbz33jqUgr/97SLGjetsdFhh6fDhTCIiLOTk/Nn53mYzk5KSQZMmNQyMTASDQDQOdQeStdZ7AJRSM4GhwLZiyt8IPBWA61Y5JpPixRf78+KL8izNaC1aFG6Q83g0rVtXbuu+CE6BuFWvDxwosH3Qt68QpVRjoCmwvMDuSKXUWqXUr0qpYQGIR4hyi4mJYMGCm4mLiyQiwkxsrI1vvhlFfLy0rIvA1DiLGkpRXK/6UcAcrbWnwL5GWusUpVQzYLlSarPWenehiyg1DhgH0KhR6K63I0JHz54NOXHiEU6ezKFGjUgZNSTyBaLGeRBoWGC7AZBSTNlRwOcFd2itU3yf9wDf4//8s2C5yVrrrlrrromJMrOQqBxKKeLjoyRpCj+BSJxrgJZKqaZKqQjykmOh1nGlVBIQB/xSYF+cUsrm+7om0Ivin40GrcOHM5g7dxtLluzG4/Ge/wAhREgr96261tqtlBoPLALMwBSt9Val1DPAWq31mSR6IzBT+w+ObwO8r5TykpfEJxZsjQ8Fa9Ycom/faUDelGedOtVh+fJbsVrN5zlSCBGqZJKPMsrOdjFjxmYee2wZqal/dlK3261MmjSAO+/sYmB0QoiykEk+KlBOjpvu3T9kz56TfhNBQN5kEAcOpBsUmRCiMkjiLINZs7awd2/hpAl5Nc4ePYrsjSWCTGqqgylTfiMjI5drrkmiWzf5uYmSkcRZBidP5uB2F24EslhMPPJITwYPbmVAVKI0jh/Pon37dzl5MgeXy8Nrr/3C7Nkj5GcnSkQSZxlccUVTv+4pERFmevVqyPz5NxMZKW9pKHj//XWkpWXjcuX9A3Q43Dz44CJJnKJEZHakMujQoTazZl1PnToxREVZ6Nu3Kf/970hJmiEkr6bpf9eQkZFrUDQi1MhfehkNGZLE4cNJRochymjYsCTee28NDkfe0s1RURauu66NwVGJUCE1ThGWLrmkMVOnDqNhw1gSEqK49daO/Oc/VxkdlggR0o+zlLZuPcaGDUdo0qQGvXrJmHkhqhLpx1kBPvroN+67bz4WiwmvVzN6dAfee+9qo8MKW1prPvhgPbNmbSE+3s6zz14u076JSiE1zhLKznYRF/ciubl/Tuxkt1v58cfb6NKlXqXGIvI8//z/eOGF/+FwuFAqbyq4TZvulomGRZmVtMYpzzhL6MSJ7EIz5FgsJg4dyjAoIjFp0i/5gxC0huxsNzNnbin3eXfvTuOtt1bz0UfrSU+XlnZRmNyql1DdujHExtrIznbn73O7vXTsWNvAqMJbUTdLXm/57qBWrTpI377T8Hi8mEwmnn76B3777a8kJNjLdV5RtUiNs4TMZhNLltxCvXrVsFhMREdbmTnzOho3lttCo4wf3z1/cTulIDLSwqhR7cp1znvumU9WloucHA8Oh4sjRzKZNOnXQIQrqhCpcRYjJSWDp5/+gZSUdIYMSeLOOzvTvn1tDh78GxkZTmJiImRyW4M99dRlJCREMXPmVuLjo3jhhSto1iyuXOcsONMVgMvl5fDhzHKdU1Q9kjiLkJaWzYUXvkdaWjZut2b58n3s2XOSiRP7oZQiNtZmdIiCvNnZ77uvB/fd1yNg5xw4sAXTpm3MfyRjt1sZPLhlwM4vqga5VS/Cl1/+TmamC7c773mZw+Fi0qRfCcUeCKJ0Jk26iiFDkrBa8x7HPP10H4YPlxFFwp/UOIvgdnsLJcnyNjqI0BAZaWHWrOuNDkMEOalxFuHqq1thtZpRvkeYdruV0aM7oJQ80xRCSOIsUv36sfz661iuvLI5HTvW5sEHe/D++zJCSAiRR27Vi9GmTSILF442OgwRBJxODx6Pl6goq9GhiCAhNU4hiqG15r775hMd/TzVqv2bwYM/Izu78HIpIvxI4hSiGO+/v44pUzbgdms8Hs3y5Xt56KHFRoclgoAkTiGKsWzZHr8F+XJy3CxfvtfAiESwkMQpRDGaNIkjIsKcv20yKRo1qm5gRCJYSOIUohiPP96bBg2qUa1aBDExEVSvbuPttwcZHZYIAtKqLkQx4uKi2Lz5HhYtSsbp9NC3bzNq1pRZkoQkTiHOyW63ypBLUYjcqgshRCkFJHEqpa5SSu1QSiUrpSYU8fptSqnjSqkNvo+/FHhtjFJql+9jTCDiEUKIilTuW3WllBl4G+gPHATWKKW+0VpvO6voLK31+LOOjQeeAroCGljnO/ZkeeMSQoQfrTW7Fy3i+LZt1GzdmhYDB1bIHBOBeMbZHUjWWu8BUErNBIYCZyfOogwAlmit03zHLgGuAj4PQFxCiDCz6MEHWf/RR3hdLkxWKx3HjGHw228H/DqBuFWvDxwosH3Qt+9s1ymlNiml5iilGpbyWJRS45RSa5VSa48fPx6AsEUgJS9cyLfjxrHk0UfJOHzY6HBEGDr9xx+smzwZV1YWHqcTV1YWG6ZM4eSePQG/ViASZ1H14LMnr/wWaKK17gAsBT4pxbF5O7WerLXuqrXumpiYWOZgReBtmDqV2dddx/oPPuDX117jvY4dyTx61OiwRJhxpKZijojw22eOiMBx4kTArxWIxHkQaFhguwGQUrCA1vqE1vrMOqsfAF1Kemwoyc52sW5dCrt2nQir2eKX/fOfuBx5a/V43W5yT59m47RpBkclwk3N1q0xWf1nsDJZLCS2CXx3skAkzjVAS6VUU6VUBDAK+KZgAaVU3QKb1wC/+75eBFyplIpTSsUBV/r2hZzdu9No3vwNLr/8Ezp2fI+bbpobNrPGu3Ny/LY9bnd+Ig01J09m8913O1mxYi9ut/e85XfvTmPs2K8ZPnwms2dvrYQIw0P2yZP8NHEiSx55hH0//FCiY6x2O7d9/z3xLVuizGbimjfn1uXLiYiJCXh85W4c0lq7lVLjyUt4ZmCK1nqrUuoZYK3W+hvgfqXUNYAbSANu8x2bppR6lrzkC/DMmYaiUHPzzf/l6NGs/GT57bc7+eyzzYwe3cHgyCpe+5tv5rcpU3D7kqU1MpLWw4YZHFXp7diRSs+eU3C7PXi90Lp1TX788bZi5+H844/TdOkymYyMXLxeWLx4D8ePO7j33m6VHHnVknPqFO916EDWsWN4nE7WvP02V0+eTIebbz7vsbXateO+nTsrPMaA9OPUWs/XWrfSWjfXWj/v2/cvX9JEa/2Y1rqt1rqj1vpyrfX2AsdO0Vq38H1MDUQ8Rtix44RfDTMry8XWrccMjKjyDHjtNbrdfTfVGzemdocO3DhvHnU6djQ6rFIbO/YbTp7MJj3dSWamky1bjvHWW6uLLT99+kayslx4fRVTh8PF88//WEnRVl0bp03DkZqKx+kEwOVwsPihhwyOyp8MuQyQpKQE1qxJyU+e0dFW2ratZXBUlcNstXLlK69w5SuvGB1Kuezbd4qCj6Zzctzs2lX8DZDT6Sn0OKYkt/fi3JyZmXhc/hNGB9ujHxlyGSCffXYddepEExtrIyrKwjXXJHHTTe2NDkuUQo8eDYiI+PNPwm630qtXw2LLjxzZjqgoi1/5ceO6FFtelEyLgQMx22z525YgfPSjQrH1t2vXrnrt2rVGh1FITo6b338/TrVqNpo3j6uyq2K6HA48TieRNWoYHUpApaVlc+WV09my5Rher2bs2At5553B5/w5rlp1kEceWcrp0zmMGtWORx7phclUNX/ulWnXggUsuO8+ck+fptWQIQx+5x0skZEVfl2l1DqtddfzlpPEKUpKa83CBx5g7bvvglLU79aNm+bPJ7J61ZncV2vN8eMOIiMtxMbazn+AqFJKmjjlVl2U2Kbp0/ltyhS8bjdel4uUdeuYd9ddRocVULNnb+Xmm//LmDFfsn69jIASRZPGIVFi+374AVdWVv62JzeXAz//bGBEgfXRR+u5//6F+esMLVmyh19+GUv79rUNjkwEmypf4/R6NWvXpvDjj/vJzHQaHU5Ii2/e3P85k1JUb+jfeOLOyWH3kiUkL1yIMzOzkiMsnxdf/NlvcbasLBcffLDewIhEsKrSNU6Xy8PAgTP49deDmM0moqIsrFw5lmbN4owOLST1eOABts6axck9e1AmE8psZsiHH+a/nnPqFB/26EHG4cMopYioVo07V6+mWr16BkZdckWN9AqX0V+idKp0jfPdd9eycuUBsrJcpKfncvy4g9tu+8rosEJWRHQ0d65Zw4g5cxj68cfct3On3zjg7596ilP79uHMyCA3PZ3Mo0dZ8MADBkZcOg891BO7/c9RQna7lb/8pbOBEYlgVaVrnNu2HSc7252/7fVqkpNDckRn0DBHRNBiwIAiX0vduTN/tAeAdrtJS06urNDK7e67uxIVZeGjj34jJsbKU0/1oVOnOkaHJYJQla5xdutWz68GYbGY6NhR/hAqSuNLLsFq/3MVSEtkJA179jQwotK77bZO/O9/t7NgwWguuqiB0eGIIFWlE+ftt1/I8OGtsdnMREdbadYsjqlTh/qVSU5O45lnfuCZZ36Q2mg59XrkEVoMGoTJasUcEUGDnj258uWXjQ5LiIALiw7wKSkZOBwumjatgdn85/+KzZuP0rPnFLKz81pSo6KsrFx5h3Q/KafstDS8Hg/2mjWr7OgpUTVJB/gC6tWrRosW8X5JE+Bf/1pBVpYTj0fj8Wiyspw8+eQKg6KsOqLi44lOTJSkKaqsKt04dD5paTl+s+FoDSdP5hR/gAhrCxcms2BBMnXqRHPPPd2oXr3ix05XNVprMg4dwuNyUaNxY5QpNOtuYZ04b765PWvXpuR3eo6OtsqMRqIQj8fL3/++iPfeW4fT6cFmM/PBB+vZtOluYmIizn8CAYDH5WLW8OHsXbYMlKJWu3bcunQptthYo0MrtdBM9wFy552defLJS6ldO5pataJ54olLGTdO+u2JP7lcHq64YhpvvLEap9MDQG6uh2PHspgzpyQrYIszfn7pJfYuX447Jwd3djZHN21i4YMPGh1WmYRljfOTTzbyn//8gtls4p//vIQjRx42OiQRpKZO3cDatYXXD/R4tAzhLaVDv/6KOzs7f9uTm8uhNWvOcUTwCrvE+emnm7jnnu/yb89vueVLbDYzgwe3MjgyEYx2707zG79+hsmkuPLK5gZEFLoS27Zl99KleHyL+5ms1gpZgbIyhN2t+jvvrPH7Q3A4XLz7rsztKYp20UUNiI72X6zNbreycOHNtGqVYFBUoenSJ54gsU0bImJiiKhWjWr16zPwjTeMDqtMwq7GGRFhLrQvMjLs3oYyObJhA1+MGMGp/fuJb9mSkf/9LzWTkowOq0ING9aae+7pxqRJv2IyKVq2TGDp0luoXTvwS85WdRExMdy5ejUpa9fidbup26UL1qgoo8Mqk7DoAF/QsmV7GDLk8/wx7Ha7lRUrxtC9e/1Ahljl5KanM6lJE3JOnszboRTRtWrx4P79WGxVf6b0zEwnDoeLxES79E8NgD9++on548eTnZZGqyFDGPDaa0Hxe1TSDvBhV9Xq27cZixffwjvvrMFiMfHAAz3o0iU0pj0z0tHNm9HeAis4ao3L4SAtOZlabdsaF1gliYmJkK5HAZK6fTufDhiQv3LlhqlTcWVmMuyTTwyOrOTCLnEC9O7diN69GxkdRkiJio/He9aSrR6nk6j4eIMiEqFq53ff+S3/687OZtucOedMnBkpKSz5xz84uXcvzfr149Inn8RstRZbvqKFZeIUpZfYpg0XjBjBtjlz8OTmYrbZ6DJuHNXq1jU6NBFirFFRmCwWv3/E5ojia/M5p08zuUsXslJT0W43RzZs4MTOnVw/c2ZlhFskSZyixIZOnUrrYcM4sXMntdq3p+XAgUaHJEJQuxtv5MfnnsPhW/TPardzxXPPFVt+z9KlOLOy0O68dokzNVR3Tk6lLBlcFEmcosSUUrQeNszoMESIi4qL4+5Nm/j19ddxHDtGqyFDaHX11cWWD8bGOEmcZeTxeDl6NIv4+CjpziREKdlr1uSKZ58tUdlm/foRERODOzsbr9uN1W4naehQw2qbEKAO8Eqpq5RSO5RSyUqpCUW8/nel1Dal1Cal1DKlVOMCr3mUUht8H98EIp6KtnnzUerXf40WLd6gRo2JfPLJBqNDEqLKssXGMm7dOtrfdBONLrmEno88wvBp0wyNqdz9OJVSZmAn0B84CKwBbtRabytQ5nJgldbaoZS6G+ijtR7pey1Ta12q3sTl6cdZXlprGjT4DykpGfn7oqIsrF39MSFPAAAgAElEQVQ7jgsuSDQkJiFEYFTmRMbdgWSt9R6ttROYCfitT6G1XqG1dvg2fwVCdjGXU6dySE3N8ttnsZjYuPGIQREJISpbIBJnfeBAge2Dvn3FGQssKLAdqZRaq5T6VSlVbMuDUmqcr9za48ePly/icqhePRKLxX/YpteradSoukERCSEqWyASZ1FNXkXe/yulRgNdgYIreDXyVY1vAiYppYqcckZrPVlr3VVr3TUx0bhbYpNJMWPGtdjtVmJjbdjtVkaP7kCvXtKhXpSN1pqjRzPz174SwS8QzcEHgYYFthsAhSYwVEr1A/4JXKa1zj2zX2ud4vu8Ryn1PXAhsDsAcVWYYcNas3XrPWzYcIQGDWLp2lWGbIqiZWU5efjhxfz88wFatkzgjTeuon79P2c8P3DgNH37TuOPP07j9WqefvpyHnust4ERi5IIROOQhbzGob7AIfIah27SWm8tUOZCYA5wldZ6V4H9cYBDa52rlKoJ/AIMLdiwVBQjG4eEKCmtNZdd9jFr1hwiJ8eDxaKoXTuG7dvH549779btA3777TAeT97fod1uZd68G7n88qZGhh62Kq1xSGvtBsYDi4Dfgdla661KqWeUUtf4ir0MxABfnNXtqA2wVim1EVgBTDxf0jTS778f57vvdrJnz0mjQxEh4NixLFavzkuaAG63JiMjl5Ur/2wS2LTpaH7SBHA6PaxefajSYxWlE5Ce21rr+cD8s/b9q8DX/Yo5biUQEqujPffcj/z73//DYjHjcnl4993BjBnTyeiwgtqGqVNZ9vjjuHNyaDtyJAPffNPQiRkqm9ls4uwbOq3zemGcUadONH/8kZ6/bbOZady4RmWFKMoo7GaAL4tdu07wwgv/w+Fwk56eS3a2m7vu+o709NzzHxymkhctYv748WQeOULOqVNsnDaNJY88YnRYlapmTTtDhrTCbs/7Z2GzmWnQIJZevf5sEvjss+uIiYkgNtZGTEwEl13WmBEjLjAqZFFCMlawBPbvP01EhDl/8mPIqzUcOZJJbKzxk68Go+1ffpk/3yLkTczw+9y5XPWf/xgYVeWbOfN6Xn55JT/99AdJSQk89dRl2Gx//tn16tWIHTvGs3r1IeLjo+jduxEmU/CNzQ5WWmuSFywgdft2Ei+4gOYDBlTK2HZJnCXQpk1NXC6P3z6TSdGwYeitB11a2usFpUr9yxgVH583dZj7z382kdXDr6+rxWI6byt5vXrVGDasdSVFVLXMHz+ejZ98gtflwmS10nnsWK56/fUKv27Y3qq73V7efHMVY8Z8xSuvrMxfM7so9evHMn36tURFWbDbrVSvbmPevBuJiqq6z+s8Tidf3nILz9lsPB8VxdIJEyhND4weDzxAZHw85ogIlNmMxW6vlF9oYTxnZia56ennL1hOJ/fsYcOUKbiysvA4nbiyslg3eTKn9u+v8GuHZY1Ta831189myZI9OBwuoqIszJ+/i6VLby32Nunaa9uQlvYoR49mUrdutSIXfatKlj/xBNvmzs2rMbrdrH7zTeJbtKDzX/5SouNjatfmni1b2DhtGi6Hg6RrrqFOx44VHLUwktft5ssxY9g2ezYAzQcM4IY5cypsFiNHairmiAjcvuWGIW9C5OwTJ6jRuPE5jiy/sKxx7tt3isWLd+cvE5yd7Wb16kNs2XLsnMdFRlpo3LhGlU+aALvmz8ednZ2/7XI42DlvXqnOEZ2YSM+HHuKyJ5+UpBkGVr7yCju++gqv243X7WbvsmUse/zxCrte4gUXYLL41/1MVisJlbDyalgmzpwcd6GapcmkZMhbATF160KB55omi4XYBiE7N4uoBPtWrPBvEMzJYd8PP1TY9SJiYhizYgVxzZujzGbiW7ZkzIoVRERHV9g1zwjLW/WWLROoXz+WvXtP4nJ5MZsVNWpE0qFDbaNDCxoDX3+djy6+OO9WXSls1apx6ZNPGh2WCGJxLVpgWrEify0hZTYT17RiR0DV7tCB+5OTK/QaRQm7ddXPOHYsizvv/IYNG47Spk1NPvzwGho0qPqt5KWRfugQyQsWYLJYSBo6lKi4OKNDEkHMceIEH3TrhiM1FYCI6GjuXLMmpO5USjrkMmwTpxAi8FwOB3uXL8fr8dCkT5+Q64JW0sQZlrfqongel4tNn35K+oED1O/RgxYDBpT5XBkpKeSmpxPXvHlYDbUMZ1a7/ZwLr1UVkjhFPq/Hw/T+/UlZswZXdjbWqCh6P/44l/7zn6U6j9aaeX/9KxunTcNktWJPSOD2H3+keiOZs1RUDWHZqi6Ktm/FCg6vW5fXMqo1LoeDH55+Gndu6cbkb509m82ffYYnNxdXZibpBw8y96abKihqISqfJE6RL+fUKb8uSAAo5dfFpCSObNiAK+vPdZm0x8OxLVsCEaIQQUESp8jXsGdPCs6DpiwWaiYlEVmjdNOc1WzdGmvBvnRKEde8yBVRhAhJkjhFvmr16nHLkiXEtWhBREwMjXr14pbFi0s9wUeH0aNp1rcvVrsdW2ws9oQErpsxo4KiFqLySXckUSG01hzdtInc9HTqdOqErVo1o0MKapMnr+P//u97nE4Pt9/eiYkT+2E2m9i//xS33fY127en0q5dLT7+eKjfmkUisKQ7kjCUUkrGp5fQ119v529/W5Q/d8I776wlOjqCRx/tRe/eUzh8OBOPR3P8eBa9e09lx47xYTFfQjCTW3UhDDZr1tb8pAngcLiYOXMLW7Yc4/Tp3Pw1iTweTWqqg507TxgVqvCRGud5pKVl8+abqzh6NItBg1py9dWtjA5JVDHx8VGYzcpv0bbq1SOx261++wA8Hm/+UhzCOFLjPIfTp3Po1Ok9XnjhJ959dy0jR87hjTdWGR2WqGIeeaQXsbE2rFYTJlPeEsGvvnolF1yQSJ8+TfITpd1uZeDAljRtKou5GU0ah87hgw/W8eCDi/xuo6pViyA9/bEKv3Yo2714MUc2biS+eXNaDx9eKWvAhLpDh9KZNm0jTqeH6667gHbtagF5KxV88ME6Nm48Spcudbnjjgsxm6W+U1GkcSgAHA4XHo/Xb9+5ltgQsOzxx1n1xht4nE7MEREkDR3KtZ9+KsnzPOrXj+Wxxy4ptN9iMXH33d0MiCiwvB4PSimUqWok/arxXVSQgQNb+q2BHRVlkUW1zsGRmsovr76KKysLr8uFKyuLHV99xdFNm4wOTVQS7fWScfgwzsxMIG/SmK/GjOH5yEies9mYf999eQsAhjipcZ5Dq1YJLFo0mnvvnU9qqoNBg1ry+utXGR1W0Mo+eRJzRAQepzN/n8lqJfuEtAKHg/RDh5jWty+n9u9Hezxc8vjjeD0ets2Zk7/a6YYpU4hv0YKLHnjA4GjLRxLnefTq1YgNG+4yOoyQUKNJE2zVq+PMyvIbulmnUycDoxKVZc4NN5CWnIz25D3OWvnKK8TUres314HL4WDX/PkhnzjlVl0EjNlq5bbvv6dWu3aYrFZqNG3KrUuXEhUfb3RoohIc2bAhP2kCuLKzMVssfs81VRVZuyogiVMpdZVSaodSKlkpNaGI121KqVm+11cppZoUeO0x3/4dSqmyz5orgkJ8ixbcvWkTTzqdPLBnD/W6nreBUlQR1erX99u2RkVx4dix2KpXxxodjTU6Gnt8PFc8+6xBEQZOuW/VlVJm4G2gP3AQWKOU+kZrva1AsbHASa11C6XUKOBFYKRS6gJgFNAWqAcsVUq10lpL03UVcGLnTubedBNpu3ZRs00brvvsM+KaNTM6LFFBrp0xg2l9+6KUQnu9NOzZk4v+9jc63noru+bPR5lMtBoypEqsXVXufpxKqYuB/9NaD/BtPwagtf53gTKLfGV+UUpZgCNAIjChYNmC5c51TZnkI/i5HA5eb9qUrOPHQWuUyURM3brcv3s3FpvN6PBEBck8epRDq1cTWaMGjXr1CrnuRyXtxxmI76o+cKDA9kHfviLLaK3dwGkgoYTHihB0bMsW3Dk5+Y1E2uslNz2dEzt2GByZqEgxtWuTNGQIjS+5JOSSZmkE4jsrqmfz2dXY4sqU5Ni8Eyg1Tim1Vim19vjx46UMUVQ2W2wsHpfLb5/X5cIWYqseClGUQCTOg0DDAtsNgJTiyvhu1asDaSU8FgCt9WStdVetddfExMQAhC0qUkJSEklDhuTPBG+NjqbtqFHUaNzY4MiCn8fj5emnv6dDh3e5/PJPWL/+sNEhibMEoh/nGqClUqopcIi8xp6zV+b6BhgD/AJcDyzXWmul1DfAZ0qp18hrHGoJrA5ATMJgSimu+/xzNn/+Oam//06tdu1oO3Kk0WGFhL//fREffvhb/hwJl146lY0b76J5c+nWFSzKnTi11m6l1HhgEWAGpmittyqlngHWaq2/AT4CpiulksmraY7yHbtVKTUb2Aa4gXulRT30eFwuTu7ZQ2SNGsTUrp2/X5lMdLj5ZgMjC01Tp27wm1jG6fTw5ZfbefjhngZGJQoKyMghrfV8YP5Z+/5V4OscYEQxxz4PPB+IOETlS9u9m48vu4zc06fxuFx0u/deBrz6qtFhhbSzZz8ymRRWa9VtaAlF8tMQ5TL7+uvJ9E3q4MnNZd3777Nz3jyjwwppEyb0yp+D02xWREdHMHJkO4OjEgXJWHVRLie2b/eb7cadk8PRTZtodfXVBkYV2h55pBf168cyd+42atWK5oknLqVOnRijwxIFSOIU5VK9USNO7NyZv22JjCS+RQsDIwp9SilGj+7A6NEdjA5FFENu1UW5XD9rFpFxcXnjke12Wl19NRdcf73RYQlRoaTGKcqlTqdOPLBnD0c2biQqLo5a7dvLbO+iypPEKcotskYNmlx2WZmP114vRzZuxOVwUKdTJyJ8neaFCFZyqw4sWpTMgAHTGTDgUxYv3m10OGHF43IxfcAApl5yCZ8NGsSbLVtycu9eo8MSBtizbBmv1q3Ls1YrH3TrRvrBg0aHVKywT5yLF+/m2mtnsXjxHhYv3s2wYTMleVaite+9x4Gff8aVlUVuejpZR4/y9e23l+jY/T/+yNIJE1j5yivknDpVwZGKinRq/35mDh1K5pEjeN1uDv/2G9MHBO/0vGF/q/7yyytxONz529nZbl59dSVXXtncwKjCx/EtW3BnZ+dva6/Xr5W+OJumT2feXXflzTIeEcHqt97iro0biZRJRELSwV9+8ZtNSXs8pO3aRc7p00H5Mw37GmdRQnCp+ZBVr1s3rHZ7/rbJYqFOx47nPW7RQw/lrWWjNZ7cXLKOHWPzjBkVGaqoQFEJCYX/8JQK2ufdYZ84H374YqKi/qx4R0VZeOihiw2MKLxceMcdJA0diiUyEmt0NDWaNmXo1KkAHPjlF1a++iqbZszIXyXxjIILgEHelHW5GRmVFnc4ObhqFV+MGMHM4cPZs3RphVyjWd++NOjZE2t0NOaICKx2OwNefRWTJThviss9A7wRAj0D/MKFybz66kpA8fDDFzNggHTgrmzpBw/icjiIa9YMk8XCusmTWfS3v+F1uzFZrdTt3JkxK1ZgMpsBmHPjjez46qu8yZIBi93O2J9/lhU1A+zQ6tV8cvnl+f+oLHY7I2bPptXgwQG/1pmlhDMOHaJ+jx406tUr4Nc4n5LOAC+JUwQdrTUv2O35SREgIiaGaz/7jKQhQ4C8FRTn3XUXu777DltsLIPfeYcWV8ma94H2xciRbJs9229fg4svZuzKlaU+17GtW8k6dozaHTpgT0gIVIgBVdLEGZz1YBHWPE4nHqfTb5/WmuwTJ/K3rVFRDP/kk8oOLeyc/YikuH3norVm3l13sfnTTzFZrWivl9ELF9KwZ+hOkxf2zzhF8LHYbNTp1AlV8PmW1jTq3du4oMJUt3vu8Wu8s9rtXPTgg6U6x54lS9g8YwYuh4Pc06dxZmQwO8SH5UriFEHppu++o0GPHpisVmLq1OGG//5XJg8xQLO+fRkxZw4Ne/akXvfuDPngA9rfdPYCD+eWlpzsN4MWkNdf0xO6c5bLrboISjF16nDHTz8ZHYYAWg4cSMuBA8t8fO0OHQrNXxDXtGl+Q18okhqnEKJCNerdm94TJmC22YiIicGemMior782OqxykVZ1IUS5aa8XrfU5a5GOEydwpKYS17Qp5oiIYsvlpqfz7V//yh8//khM3boM+eAD6l54YUWEXUhJW9WlximEKDOtNUsefZTnIiN5zmbjixtuwJ2bW2RZe0ICNZOSzpk0AWYNH872L78kIyWFw+vW8fFll5GRUuSq4YaRxCmEKLMNU6ey5q238LpcaI+HnfPmsfSRR8p8PndODvt++AFPweSrNXtXrAhAtIEjiVMIUWa7FizwG/7qzs4medGiMp/PZLUWORF2wS5RwUASpxCizKo3bIjJav1zh1JUq1evzOczmc30fuyx/ERpttmIbdiwXK36FUEah4QQZeZITeX9Cy8k+9Qp8DUOjf3lFxIvuKDM59Ras23OHPYuW0aNJk3oPn48ETGVs8qnjFUXQc/lcLD4H//gj59+Ir55cwa+8QaxDRoYHZYopdz0dHZ8+y0ep5MWAwaUq8ZpNEmcIqhprZnevz8Hfv4Zd04OymwmulYtxu/Yga1aNaPDE2FKuiOJoJadlsYf//tf/gxI2uPBmZnJH//7n8GRhTbt9frNKiUqhiROYQh3dnaRY5VVCA/DM9r6Dz/khehoXoiJ4f3Onck8csTokKqsciVOpVS8UmqJUmqX73NcEWU6KaV+UUptVUptUkqNLPDax0qpvUqpDb4PmYU2DGSfPMnUSy/122eyWqlWt265lhkOZwd//ZWFDzyAOycH7fFwdPNmZl17rdFhVVnlrXFOAJZprVsCy3zbZ3MAt2qt2wJXAZOUUjUKvP4PrXUn38eGcsYjQsBPEyeScegQukCNMzoxkb+sWoUlMtLAyELXgZUr/ebJ1G43KWvWGBhR1VbexDkUODOb7CfAsLMLaK13aq13+b5OAY4BieW8rghh6QcOFJqo2BIVRWSNGsUcIc4npk4d//6U+BZAExWivImzttb6MIDvc61zFVZKdQcigIILlz/vu4X/j1LKdo5jxyml1iql1h4/frycYQsjNb/ySqwFVi+0REbSrF8/AyMKfW1vuIF6XboQERODNToaq93OMJkhv8KctzuSUmopUKeIl/4JfKK1rlGg7EmtdaHnnL7X6gLfA2O01r8W2HeEvGQ6GdittX7mfEFLd6TQprVm6YQJ/Praa2itaTloENfPmoU1Ksro0EKa1+Nh1/z5ZJ84QcNevUho2dLokEJOpfTjVErtAPporQ+fSYxa66QiysWSlzT/rbX+ophz9QEe1lpffb7rSuKsGrweD9rrxXzWLaYQRqmsfpzfAGN8X48BCs1OqpSKAL4Epp2dNH3JFpU3qn8YsKWc8YgQYjKbJWmKkFTexDkR6K+U2gX0922jlOqqlPrQV+YG4FLgtiK6Hc1QSm0GNgM1gefKGY8QQlQ4GXIphBA+MuRSCCEqiCROIYQoJUmcosrKPHKEz4YMYVLjxswYOJD0Q4eMDilk5WZkcGLnTr/Z3sOZrKsuqiSPy8XUSy7h1L59eN1u0g8dYkqvXozfvl2GdZbSlpkz+fqOO/JWsFSKkV9+SbO+fY0Oy1BS4xRV0okdO8g8ciR//Lb2eMhOS+Po5s0GRxZa0g8d4us77sCdnY0zMxNnRgazhg8P+5qnJE5RJVkiIwtNW6e9XhmdVEqp27cXXs5Xa07/8YcxAQUJSZyiSopr3pymffvmL/pliYqiYc+eJLZta3BkoSWuadNCE7K4cnLAFN6pI7y/e1FlKaUY9eWX9P33v+l4661c8fzz3PTdd0UuPSuKF9esGVc8+6zfzEva42Fq795khfFkO9IBXogQpr1e0nbvBq2Jb9ECr8eD1+UK+DrkE+PiyD11Kn/bbLPR78UXueiBBwJ6HaOVtAO8tKoLUQbu3Ny8iYKVon63boWfA1YCl8PB9P79ObJhA1prImvUwOGrBTa4+GJu/PZbIqtXB/JmpDq0ejVZx45Rr0uXUq9Eqc9+Xuzx4MnNDcw3EoIkcQpRStlpaXzUsycZKSkAxNavzx0rVxIVV+SMihVm+ZNPcnj9+vzF2TKzs/NfO7RqFd/85S/c8MUXaK357+jR7Pj6a0xmM16Phxu//Zaml19e4mu1v/lmNk6bhtvXmm622Ui65prAfkMhRJ5xClFKSx59lFN79+LMyMCZkcHJPXtYNqGoVWMqVsqaNcWuaOlxOvNXDE1esIAdX3+NKyuL3PR0XFlZzLnhhlJda9Cbb9L93nuJb9GCet27c8uSJdRs3brc30OokhqnEKWUum2bX0uzx+nk+LZtlR5H7Q4dOLR6dbG3zGdux0/t24f2ev1ec5w4gdfjyevUXgImi4X+L71E/5deKl/QVYTUOIXw8Xo87F2xgu1ffUXm0aPFlqvfo4ff6CNLZCT1L7ronOc+tX8/8/76V2aPGMG2OXMCEm/fF16gZlJS/nIZZpsNa0wMEdWqYYuN5ZqPPgKgbpcu/r0JlCKhZcsSJ01RmLSqC0HeEM3p/ftzeN06lMmE1poxy5dTr2vhBlaXw8GMQYM4tGoVkJdIb16woNjO9emHDvFuu3bkpqfndcK32+n34ot0Hz++3HF73W6ObNwIWpPQujX7li/HmZVFkz59qFa3bn65XydNYumjj6LMZuwJCdy6fLksrVGESlk6wyiSOEWgrf/oIxbef7/fUMKE1q0Z//vvRZbXvtEzSiliGzY8Z//Qn158kRVPPonX5crfF127Ng8fOeJ/Tq8XVYEdy10OB9knT1Ktbt0KvU4ok/k4hSiF0/v3Fxp/nXGO2ZSUUtRo3JjqjRqdt1O9x+ks9IyxYBI9tW8f77Zvz7NWKy8nJrJ78eIyfAfnZ7Xbia1fX5JmAMg7KKqc1W++yWsNGvBq3br8+NxzlOSuqn6PHn5LFiuLhXpduuB1u1n8j3/wSp06TGrShM2ffQbA5s8+4+XERF6Ijmb29deTdfw4h9asIXXHjkLXu+D66/2eiVrtdi684w4gr+Y6rV8/jm/bhvZ6caSmMmv4cE7t2xeAd0JUFGlVF1XKpk8/ZemECfm1x58mTsRWvTo97rvvnMe1GjyYix96iJ/+/W+UyUR8ixZc9/nnrPjXv1j7zjv55/v2zjvJOnaMZf/8Z36fxp3ffsvOefOw2Gx4XC5aDxvGtZ9+ml+zS2zThluXLmXxww+Tc+oUbUeO5JLHHwfy+oSmHzjgVyM1WSwcWrOGGk2aBPrtEQEizzhFlTJj0CCSFyzw21eve3fu9DXknI/L4cCZlYW9Zk2UUkxq2pTTZ9X+6nTuzJHffoNi/nas0dEMmTyZ9jfddN7reZxOXqhWDW+B7k3W6GhumjePJn36lChmETjyjFOEJVv16nDWM8czww5Lwmq3E52YmP/c0hYT4/e6MpuJrFEDi81W7DlcWVkcL6ZR6WzmiAiumjQJq92OJTKSiJgYmvXrR+PLLitxzKLySeIUVUqfp54iIiYG5Zut3BodTd8XXijz+a589VUsvm5GJouFyOrVGfTWW8TUq4clKgqTxZKXqAska2t0NLVKMX1dt7vvZsyKFfR/5RWGf/opI//7X5nFKcjJrbqock7u2cOGTz5Bezy0v/lmEtu0Kdf5Dq1Zw+9z52KNjqbz2LFUq1eP3IwMNs+YQc7p0yRecAHz7roLZ0YGXpeLtiNHMnTq1DIlP601rqwsrNHRkjwNIP04hahE7pwcUrdvx1a9OnFNm5bpHEc2buSzQYPIOnYMq93OiDlzaN6/f4AjFeciiVOIEOJxOnmtfn0cqan5+6zR0dyfnExMnTrlPr/2etk0Ywapv/9OrfbtaTdqlNRoiyDzcQoRQtIPHsRVYFo4yHumenTTpnInTq01c0aNYtf8+fmPAfYsWcLQKVPKdd5wJo1DQgQBe82a+StynuFxOks94XBRUn//nV3ffYcrKwvIa/Xf8vnnnNq/v9znDleSOIUIArbYWK58+WWsdnv+bEed//IXarVrV+5z56an57X+F2CyWslNTy/3ucNVuW7VlVLxwCygCbAPuEFrfbKIch7gzILWf2itr/HtbwrMBOKB9cAtWmvn2ccLEQ6633cfjS65hCMbNxLfvDmNevcOyHlrtW+PJTKS3IwM0BplMmGrXp2EVq0Ccv5wVN4a5wRgmda6JbDMt12UbK11J99Hwfn2XwT+4zv+JDC2nPEIEdLqdOpEpzFjApY0ASKio7n9f/+jbufO2GJjqdetG7f/+OM5O/GLcytXq7pSagfQR2t9WClVF/hea51URLlMrXXMWfsUcByoo7V2K6UuBv5Paz3gfNeVVnUhREWorCGXtbXWhwF8n2sVUy5SKbVWKfWrUmqYb18CcEprfeaJ+EGgfnEXUkqN851j7fEwXs9ZCGG88z7jVEotBYrqD/HPUlynkdY6RSnVDFiulNoMFPVkutjqr9Z6MjAZ8mqcpbi2EAGnvV7SkpMBiG/RQua4DDPnTZxa637FvaaUOqqUqlvgVv1YMedI8X3eo5T6HrgQmAvUUEpZfLXOBkBKGb4HISqVMyuL6f37c3TjRiCv8eXWpUuJOGtCEFF1lfff5DfAGN/XY4Cvzy6glIpTStl8X9cEegHbdN7D1RXA9ec6Xohgs+KJJzjy22+4HA5cDgdHNm5kmW9+zZJwZWeTc+pUBUYoKlp5E+dEoL9SahfQ37eNUqqrUupDX5k2wFql1EbyEuVErfWZtVQfBf6ulEom75nnR+WMR4gKl7Jund965p6cHFLO01jp9XjYNGMGU3r35t8xMbxcqxYf9ewpCTRElasfp9b6BNC3iP1rgb/4vl4JtC/m+D1A9/LEIERlO3s9c7PNRu0OHYotr71ePrv6avatWJF/jPZ6ObxuHd/eeScjvviiUuIWgSNPtIUopYLrmUfExJDQsiX9Xnyx2PL7f/yRA6Ylhc0AAAVQSURBVD/9lJ80z/A4nRxYubKiwxUVQCb5EKKUbLGxjFu3Ln8989odO2K2Wostn33yZLGt7rENG1ZUmKICSeIUogxMvlUwS6JBjx6FlgdGKWyxsVzzkTzWD0Vyqy5EBatWrx6jFy2ieuPGmG02EpKSuPr997lv165SLbEhgofUOIWoBA179uRBWSu9ypAapxBClJIkTiGEKCVJnEIIUUqSOIUQopQkcQohRClJ4hRCiFKSxCmEEKUkiVMIIUpJEqcQQpSSJE4hhCilcq1yaRSl1HFgfxkPrwmkBjCcQAvm+II5Ngju+II5NpD4zmistU48X6GQTJzloZRaW5LlP40SzPEFc2wQ3PEFc2wg8ZWW3KoLIUQpSeIUQohSCsfEOdnoAM4jmOML5tgguOML5thA4iuVsHvGKYQQ5RWONU4hhCgXSZxCCFFKVT5xKqVGKKW2KqW8SqliuzMopa5SSu1QSiUrpSZUYnzxSqklSqldvs9xxZTzKKU2+D6+qeCYzvleKKVsSqlZvtdXKaWaVGQ8pYztNqXU8QLv1V8qKzbf9acopY4ppbYU87pSSr3hi3+TUqpzEMXWRyl1usB7969KjK2hUmqFUup339/rA0WUMey9K0RrXaU/gDZAEvA90LWYMmZgN9AMiAA2AhdUUnwvARN8X08AXiymXGYlxXPe9wK4B3jP9/UoYFYQxXYb8JaBv2+XAp2BLcW8PghYACjgImBVEMXWB5hn0PtWF+js+7oasLOIn61h793ZH1W+xqm1/l1rveM8xboDyVrrPVprJzATGFrx0YHvOp/4vv4EGFZJ1y1OSd6LgjHPAfoqpVSQxGYorfWPQNo5igwFpuk8vwI1lFJ1gyQ2w2itD2ut1/u+zgB+B+qfVcyw9+5sVT5xllB94ECB7YMU/qFVlNpa68OQ98sD1CqmXKRSaq1S6lelVEUm15K8F/lltNZu4DSQUIExlSY2gOt8t3JzlFINKyGu0jDyd60kLlZKbVRKLVBKGbJ2se/Rz4XAqrNeCpr3rkosD6yUWgrUKeKlf2qtvy7JKYrYF7B+WueKrxSnaaS1TlFKNQOWK6U2a613ByZCPyV5Lyr0/TqHklz3W+BzrXWuUuou8mrGV1R4ZCVn1HtXEuvJG6udqZQaBHwFtKzMAJRSMcBc4EGtdfrZLxdxiCHvXZVInFrrfuU8xUGgYM2kAZBSznPmO1d8SqmjSqm6WuvDvtuOY8WcI8X3eY9S6nvy/iNXROIsyXtxpsxBpZQFqE7l3AKeNzat9YkCmx8AL1ZCXKVRob9r5VEwUWmt5yul3lFK1dRaV8rkH0opK3lJc4bW+r9FFAma905u1fOsAVoqpZoqpSLIa/Co0JbrAr75//btX6WBIIjj+Hcqbf1TiKWVDyAi0SdIERCsTZEmhU9hY5fOLtYWdhaChUkrVuKhFmotFhaWYnEpdgJHQowL3t4Vvw8sWe4uZBiOuexsArR93gamviGb2ZKZLfh8FdgFnkqK5y+5KMZ8AAxy796XbG5sEz2vFqFXVieXwKHvEO8AX+NWTdXMbG3cqzazbUJ9+Pz9Xf/22QacAc95nvdmXFaf3FW1K5VqAPuEJ9U38AFc+/F14KpwXZOwk/dGWOKnim8FuAFe/HXZj28BfZ83gIywi5wBnZJjmsoFcAy0fL4IXACvwB2wkTBf82I7AR49V0NgM/H9dg68Az9+33WALtD18wacevwZM37pUVFsR4Xc3QKNhLHtEZbdD8C9j2Zdcjc59JdLEZFIWqqLiERS4RQRiaTCKSISSYVTRCSSCqeISCQVThGRSCqcIiKRRqEh9G3ntFKpAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 360x360 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# make up a dataset\n",
"\n",
"from sklearn.datasets import make_moons, make_blobs\n",
"X, y = make_moons(n_samples=100, noise=0.1)\n",
"\n",
"y = y*2 - 1 # make y be -1 or 1\n",
"# visualize in 2D\n",
"plt.figure(figsize=(5,5))\n",
"plt.scatter(X[:,0], X[:,1], c=y, s=20, cmap='jet')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"MLP of [Layer of [ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2), ReLUNeuron(2)], Layer of [ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16), ReLUNeuron(16)], Layer of [LinearNeuron(16)]]\n",
"number of parameters 337\n"
]
}
],
"source": [
"# initialize a model \n",
"model = MLP(2, [16, 16, 1]) # 2-layer neural network\n",
"print(model)\n",
"print(\"number of parameters\", len(model.parameters()))"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Value(data=0.8958441028683222, grad=0) 0.5\n"
]
}
],
"source": [
"# loss function\n",
"def loss(batch_size=None):\n",
" \n",
" # inline DataLoader :)\n",
" if batch_size is None:\n",
" Xb, yb = X, y\n",
" else:\n",
" ri = np.random.permutation(X.shape[0])[:batch_size]\n",
" Xb, yb = X[ri], y[ri]\n",
" inputs = [list(map(Value, xrow)) for xrow in Xb]\n",
" \n",
" # forward the model to get scores\n",
" scores = list(map(model, inputs))\n",
" \n",
" # svm \"max-margin\" loss\n",
" losses = [(1 + -yi*scorei).relu() for yi, scorei in zip(yb, scores)]\n",
" data_loss = sum(losses) * (1.0 / len(losses))\n",
" # L2 regularization\n",
" alpha = 1e-4\n",
" reg_loss = alpha * sum((p*p for p in model.parameters()))\n",
" total_loss = data_loss + reg_loss\n",
" \n",
" # also get accuracy\n",
" accuracy = [(yi > 0) == (scorei.data > 0) for yi, scorei in zip(yb, scores)]\n",
" return total_loss, sum(accuracy) / len(accuracy)\n",
"\n",
"total_loss, acc = loss()\n",
"print(total_loss, acc)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"step 0 loss 0.8958441028683222, accuracy 50.0%\n",
"step 1 loss 1.7235905336972022, accuracy 81.0%\n",
"step 2 loss 0.7429006313851131, accuracy 77.0%\n",
"step 3 loss 0.7705641260584198, accuracy 82.0%\n",
"step 4 loss 0.3692793385976538, accuracy 84.0%\n",
"step 5 loss 0.313545481918522, accuracy 86.0%\n",
"step 6 loss 0.2814234349772435, accuracy 89.0%\n",
"step 7 loss 0.26888733313983904, accuracy 91.0%\n",
"step 8 loss 0.2567147286057417, accuracy 91.0%\n",
"step 9 loss 0.2704862551637922, accuracy 91.0%\n",
"step 10 loss 0.24507023853658053, accuracy 91.0%\n",
"step 11 loss 0.2509905529791503, accuracy 92.0%\n",
"step 12 loss 0.21560951851922952, accuracy 91.0%\n",
"step 13 loss 0.23090378446402726, accuracy 93.0%\n",
"step 14 loss 0.20152151227899445, accuracy 92.0%\n",
"step 15 loss 0.22574506279282217, accuracy 93.0%\n",
"step 16 loss 0.19447987596204114, accuracy 92.0%\n",
"step 17 loss 0.21089496199246363, accuracy 93.0%\n",
"step 18 loss 0.159830773563036, accuracy 94.0%\n",
"step 19 loss 0.1845374874688392, accuracy 93.0%\n",
"step 20 loss 0.18977522856087634, accuracy 91.0%\n",
"step 21 loss 0.19072704042579647, accuracy 93.0%\n",
"step 22 loss 0.11733695088756485, accuracy 97.0%\n",
"step 23 loss 0.12173524408232454, accuracy 95.0%\n",
"step 24 loss 0.1261571261277045, accuracy 95.0%\n",
"step 25 loss 0.16049097780801674, accuracy 95.0%\n",
"step 26 loss 0.18747197705245805, accuracy 92.0%\n",
"step 27 loss 0.16741837891059408, accuracy 95.0%\n",
"step 28 loss 0.09586583491455399, accuracy 97.0%\n",
"step 29 loss 0.0877878370742091, accuracy 96.0%\n",
"step 30 loss 0.11731297569011848, accuracy 95.0%\n",
"step 31 loss 0.09340146460619836, accuracy 97.0%\n",
"step 32 loss 0.12454454903103446, accuracy 95.0%\n",
"step 33 loss 0.07984002652777272, accuracy 97.0%\n",
"step 34 loss 0.07727519232921673, accuracy 97.0%\n",
"step 35 loss 0.07661250143094483, accuracy 98.0%\n",
"step 36 loss 0.10610492379198365, accuracy 96.0%\n",
"step 37 loss 0.09062808429265976, accuracy 99.0%\n",
"step 38 loss 0.10671887043036932, accuracy 95.0%\n",
"step 39 loss 0.05225659921975849, accuracy 98.0%\n",
"step 40 loss 0.06016009895234464, accuracy 100.0%\n",
"step 41 loss 0.08596724533333942, accuracy 96.0%\n",
"step 42 loss 0.051121079431796, accuracy 99.0%\n",
"step 43 loss 0.052401424016428284, accuracy 97.0%\n",
"step 44 loss 0.045306841790015734, accuracy 100.0%\n",
"step 45 loss 0.07211073370655095, accuracy 97.0%\n",
"step 46 loss 0.03334238651310234, accuracy 99.0%\n",
"step 47 loss 0.03143222795751122, accuracy 100.0%\n",
"step 48 loss 0.03658536747111507, accuracy 99.0%\n",
"step 49 loss 0.04829139382390309, accuracy 99.0%\n",
"step 50 loss 0.09875114765619622, accuracy 96.0%\n",
"step 51 loss 0.05449063965875453, accuracy 99.0%\n",
"step 52 loss 0.03392679435708309, accuracy 100.0%\n",
"step 53 loss 0.05261517263568441, accuracy 97.0%\n",
"step 54 loss 0.03250295251424923, accuracy 99.0%\n",
"step 55 loss 0.02888327387207822, accuracy 100.0%\n",
"step 56 loss 0.04139151104027239, accuracy 98.0%\n",
"step 57 loss 0.018987407426128502, accuracy 100.0%\n",
"step 58 loss 0.0252383352388374, accuracy 100.0%\n",
"step 59 loss 0.02079656521341895, accuracy 100.0%\n",
"step 60 loss 0.0325971115781023, accuracy 99.0%\n",
"step 61 loss 0.017863351693480307, accuracy 100.0%\n",
"step 62 loss 0.023008717832211683, accuracy 100.0%\n",
"step 63 loss 0.022079325463581503, accuracy 100.0%\n",
"step 64 loss 0.029432917853529684, accuracy 99.0%\n",
"step 65 loss 0.01625151464409193, accuracy 100.0%\n",
"step 66 loss 0.02846853448326446, accuracy 99.0%\n",
"step 67 loss 0.013994365546208731, accuracy 100.0%\n",
"step 68 loss 0.015552344843651405, accuracy 100.0%\n",
"step 69 loss 0.0338911994616017, accuracy 99.0%\n",
"step 70 loss 0.014229870065926908, accuracy 100.0%\n",
"step 71 loss 0.013255281583285504, accuracy 100.0%\n",
"step 72 loss 0.012300277590022063, accuracy 100.0%\n",
"step 73 loss 0.012676052498355976, accuracy 100.0%\n",
"step 74 loss 0.020593811955954763, accuracy 100.0%\n",
"step 75 loss 0.011845398205364453, accuracy 100.0%\n",
"step 76 loss 0.016012697472883086, accuracy 100.0%\n",
"step 77 loss 0.025458360239222128, accuracy 100.0%\n",
"step 78 loss 0.014382930289661911, accuracy 100.0%\n",
"step 79 loss 0.011698962425817985, accuracy 100.0%\n",
"step 80 loss 0.012318500800515763, accuracy 100.0%\n",
"step 81 loss 0.014121117031464233, accuracy 100.0%\n",
"step 82 loss 0.011664591962446225, accuracy 100.0%\n",
"step 83 loss 0.011589314549188726, accuracy 100.0%\n",
"step 84 loss 0.010990299347735226, accuracy 100.0%\n",
"step 85 loss 0.01098922672069161, accuracy 100.0%\n",
"step 86 loss 0.010988193757655071, accuracy 100.0%\n",
"step 87 loss 0.010987200447388707, accuracy 100.0%\n",
"step 88 loss 0.010986246779084925, accuracy 100.0%\n",
"step 89 loss 0.010985332742365272, accuracy 100.0%\n",
"step 90 loss 0.010984458327280174, accuracy 100.0%\n",
"step 91 loss 0.010983623524308862, accuracy 100.0%\n",
"step 92 loss 0.010982828324359073, accuracy 100.0%\n",
"step 93 loss 0.010982072718767003, accuracy 100.0%\n",
"step 94 loss 0.010981356699297042, accuracy 100.0%\n",
"step 95 loss 0.010980680258141723, accuracy 100.0%\n",
"step 96 loss 0.010980043387921506, accuracy 100.0%\n",
"step 97 loss 0.010979446081684675, accuracy 100.0%\n",
"step 98 loss 0.010978888332907229, accuracy 100.0%\n",
"step 99 loss 0.010978370135492717, accuracy 100.0%\n"
]
}
],
"source": [
"# optimization\n",
"for k in range(100):\n",
" \n",
" # forward\n",
" total_loss, acc = loss()\n",
" \n",
" # backward\n",
" model.zero_grad()\n",
" total_loss.backward()\n",
" \n",
" # update (sgd)\n",
" learning_rate = 1.0 - 0.9*k/100\n",
" for p in model.parameters():\n",
" p.data -= learning_rate * p.grad\n",
" \n",
" if k % 1 == 0:\n",
" print(f\"step {k} loss {total_loss.data}, accuracy {acc*100}%\")\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(-1.548639298268643, 1.951360701731357)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmMJPl12Pnvi4jMrDvrvo/u6mumZzhDzsUZUpRIkRSHWoK0vBRA7mIh7doY7MKE9/rDMgTIWAMGZHixC68lWDsraSUDlmSCXko0TJkUD/MQRWqaw+FwZrp7+q6u7qqu+8ysPCLe/hF1ZeVRWZVZZ74P0OjKzMiMqKzM34v4/d7v/URVMcYYU3ucoz4AY4wxR8MCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1yjvqAyilLVav/Y0tR30YxhhzYrwzPzWjql3lbHusA0B/Ywt/8vHPH/VhGGPMifHeL/7Le+Vua11AxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1ygKAMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1qioBQET+UESmROStIo9/WEQWReSN9X+/VY39GmOM2T+vSq/zR8DvAP+mxDbfU9VPVWl/xhhjKlSVKwBV/S4wV43XMsYYczgOcwzgJRH5qYj8pYg8UWwjEXlFRK6IyJX5VPIQD88YY2rLYQWA14ERVX0a+FfAnxfbUFVfVdXnVPW5tlj9IR2eMcbUnkMJAKq6pKor6z9/FYiISOdh7NsYY0xhhxIARKRXRGT95xfW9zt7GPs2xhhTWFWygETkT4EPA50iMg78EyACoKq/B3wW+B9EJAskgc+pqlZj38YYY/anKgFAVT+/y+O/Q5gmaowx5piwmcDGGHNKDPXe3dP2FgCMMeYUGOq9S8Po0J6eU62ZwMYYY47Axll/w+gQkaca9vRcuwIwxpgTqpLGHywAGGPMiVSo8Xde/IU9vYYFAGOMOWGq0fiDBQBjjDlRSjX+Ut+9p9eyQWBjjDkhdjb+TvcwjJ4Fwsb/2sKNPb2eXQEYY8wJUF7jv7eBYLsCMMaYY26j8Y//nUsAOY3/9dQipBaBBr50e28l9O0KwBhjjrFdG39gP40/WAAwxphja6+N/2vfaNnT61sAMMaYY+igG3+wAGCMMcfOfhv/nvq9BQEbBDbHxtpyitk7cySXUogDDfF62oZbqWuOHfWhGXNoNoq6beb4rzf+uWmeYeNf57byva8FwN4bf7AAYI6J2XvzjP34AepvrRO0+GCZiWtT9D7WTf8TPUd4dMbsbq+lmIspNsFro/H/0m0BKm/8wQKAOQb8jJ/X+G8KYOr6NPHeZho79l7sypjDsD1Hv1K7N/5UpfEHCwDmiCWX1ph4+xEaFF8hNPCVqVuznC0jAKgqibkkGigN7fU4rg1zmYO1s8umGg6j8QcLAOYIPXhzgqmbs4XP/HdYuL9I6okeYo3RotuszKxy6/t38bMBKIgI/U/30nOhs5qHbQxQpCZP93DlL1ygtMNBNP5gAcAckZXpVabLbPwBNFAe/HSC0Q+MFHw8k8py4zt3cq4kVJUHb0wQrfNoG2otaz/pRJr7b0ywNLEMQLy/mcH39hOtj5T1fFMbqlWNs5jDaPzBAoA5IjN35gjKbPw3LK43ygVf79Zs0W6ksZ88BBEidR6NHQ2ISMHtsmmfa9+4STblb963ML7EykyCJ16+iBtx93S85nSqZjXOYjbq+mykeR5E4w8WAMwR8TPBnp9TpN0GYGUmUXxfKZ97r40D4MU8Lvz8WWJN+V1JM3fmwu6jvGP1mb07T7d1JdW8o2z8q9nwb6hKABCRPwQ+BUyp6pMFHhfgXwK/DCSAX1fV16uxb3MytQ62sPxoOe8qQBwAyT+bF2gdjBd9vbqWGMuPVoo+Hqw37OlsmhvfvcMTn7xIOpHh4c8mWZxYRhzB9ZyCXVLqK8tTqxYAatxujf9eSzEXt3OC18E0/lC9mcB/BLxc4vFPAhfW/70C/Osq7decUG1Dcepa6hBn67ReHKE+Xs/ICwOIK7D+kOM6ROsjDDzdV/T1+i6Xf/aVTWVZnFjm2jduMn9/kSAb4Kd90olM4ScIRBttDKCWFSrFXLjxb6j4X6Wze/eiKlcAqvpdETlTYpPPAP9GVRX4oYi0ikifqk5UY//meEitpFh4sIQGSry/hfp4XdFtHcfh4kdGmb41y+zdBQA6RlrpOt+B4zo0tjcyc3uOTCJDc3cjbcOtJVM6vajHyAuD3Pvb8bKOdfbO3OZVwW7EEbrOdaCBsjgRjglE6jzaR9qI1Fkv6mlXbh3+sOHee0XOQg6j8YfDGwMYAO5vuz2+fl9eABCRVwivEuhraD6UgzOVm7w2xcTbU4CiChNXp+g8287ge/uKDro6rkPPxS56LnblPRZrjDLwnt49HUPHSBst3U3M3J0jnciwNLFCJpl/Vq+BsraSLjn3wPHWg43C8PMDeDGPq391g3QiQ5ANEEd4+LNJvDoPPxNQ1xyl/8leWnrtM3ua7LUOf51bXrZZKQcx2FvMYQWAQi1AwW+fqr4KvArwRHvP3tJEzJFILCSZeGcqNwXTV2bvzNHS10y8RKOoGm43cXWaTDJDrDFK3xM9tA/v74sUqY/Q93hYNmLp0Qq3/vpuTr++uELbYJxsyie1lMp7vjhC72NdNLTXA0JzdyOO6zD24wekltOEF7Fs/q6ZZDZ8D+bXuPWDe5x5frDslFNzvO2vINvekxsKOYzGHw4vAIwD2+dIDwIPD2nfNSe1mmbq3RlW5xLEmqL0XOyioa3+wPY3e2e+4OBp4Cszt+dKBoDJa9M8ujq1ORicWkkzdmWcIBvQOdpe0XG19DRx7oNnePDTCZJLa3hRl67zHfQ+3s3y1ArL0yv5xy3QcbadaENun//c2MJm41+M+sr4TydoHYwXveoxJ8NhVeM8aocVAL4CfEFE/gx4P7Bo/f8HI7GQ5N1v3ybww9mwibkkCw+WGHl+kPYqnZmqKosTy0y9O0M2lS3Zl+6n/aKPBdkgp/HfvN9XHvxsko6zbRU3pC09TbT80oW8+5u6Guk428bs7fmtgWhVzrw4nNf4AyW7i7bLpnyyaZ9IzMYGTqpaafyhemmgfwp8GOgUkXHgnwARAFX9PeCrhCmgNwnTQP/bauzX5Bu78iCvQVZfuf/jB7QOtOA4lSd+PXxrkukbs7tO5HJcoXWw+JdibSW1ntxf6OohILOWPZAZuDN35njw0wlUw2AWiXr0XOqi/Uwbrlf4/WnuadqcHbybYq9hjr9aavyhellAn9/lcQX+QTX2ZYoLsgGJhcJZCKqQmE/S1NG4eZ+f8VlbThGp84g2FK+xs106kWbq3eKzbrfz6iN0nCnejePFvOKvoxzIzNvFyWXGf/IwJ3ilExkmr02X7HIafLqP6zOrm3WGChFHwiBrBehOpMOsw39c2HXqabJLb8nG2b+q8vBnk0zdmEWccNJVY0cDoy8N4+3SdbH8aAWRom1gzrGc+8BIybPhaH2Exs4GVqZXc15QnPDK4SDOpCffye9ygjAYLk4u09pf+Mtc1xzj8V+6wKPrMyxPr+BFXVLL6fWAEL5eXbyO4WcHqn7M5mCVP8HrdDX+YAHgVHFch+buJpanVvJaaNdzqG8N8/Knb8yGhdgC3TwDX5le5dq3blHXHEP9gLbhVtoL5N6L5xTttsnZTmTXYAIw+uIwN753J8zIEUFVaWxvOLCGNLWSLni/BkHeY9m0z+LEEuorLb1NRBuiDL2vf+s5qqxMr5JaTVMfr6Ohrd4Gf0+Ychv/ai7CcpxYADhlRp4b4No3bxFkfAJfEVcQEc6+NLzZOE1emy54FpxeSZNebwRXZxPM3Jrj4kdGc4JAvLd512wYgPrWurImSXkxj8c/doHEfJLUSpq6lljJCWSVqmuJsTKdzbtfHIf6lq2lJ+fuL3DvtfHwPdNwbkP3xc6cuQkiQnN3E5b5fzLtrfE/uIJsR8kCwCkTbYjyxCcvMTc2T2JujVhzlI5tM1ZVlWwqvwHcKfCV5NIas3fn6TrXsXm/G3E5++Iwd344BpqfHeO4Do4nnHn/3lZGamirP9BU1Q19T/Rw83t3ctM/BSJ1Hs09TUA4znHvtXHUV3Tblc70jRmaOhuJ91Wvyd9YBzmdzNDS00zbUNzGEA6BNf4hCwCnkOs5dI12wGj+YyJCtCFSvO7NNuorc/cWcgKAnw3IrGXCyVTpLLHGKE1dTWRTWVLLKeridbQNtR7bTJjmrkbOvDDE+E8ekl1PUW3ubmTk+aHNK6TZewub/frbBb4ydWOmagFgcx3kQEFh8eEyk1enuPTR83hRKz19UKzx32IBoAb1v6cnTBctox7/9mJtqdU01795kyAbhN1LjrDiJGgfaaOtRKXO46ZtME7rQAuZtSyu5+RlG2XXsmiRqQ3lXD2VI5vOXwc5yIbjEBNvTTL0jA0mH4Ty6vrURuMP1asGak6Q9uE2Bt/XjxdzwwZeKJhB5LhCx5m2zdv3Xhsnm/I3A4cGSpANuPM3Y2WNCxwnIkK0PlIw1bS5p2mrFtD25zhStVo/S5MlFre5O1+VfZhc5Rd1y238e+pbTmXjD3YFULM6z7bTcaaNbNrH9RwevvWImVtbk7scz6Gps2GzJo+f9VmdWS34WtlUlrWl1IEO3h6meF8zdc0xkotrW2McAm7EqdqaAKpadA6E+oqfDY5tN9pJtNeibgddh/+4sABQg5JLa2SSGerjdUTqwpm2g0/30TYYZ/ZeWNendTBOS2/TZr94sS4RIEzfLLNUwkkgIlz48CiTV6eYuztPECit/S30PdlTtfLPLb3NxSeUCaxMrxDvO92Nz2Gptdm9e2EBoEZkU1lm7swxfWOWTDq7nt4I7cOtDD87gDhCY0cDjR0NBZ/vRV3qWupILq7lPSYip+bsf4PrOQy8p3fPJanLFYl5YbdbgSAglgVUNdb4l2YBoAYsPlzi9g/HcgYcN9Ib5+4vEGmI0P9Ez66vM/zcADe+c2ez0ByE5ZVHnh/IGSw2ubKpLDO351iZSRBrjtJ1roO65hhtA3HmxxcLPqe5u+mQj/L0scZ/dxYATjk/43NnR+O/nfrK9I0Z+i537zqLtbG9gcc/foFH706TmEsSa47Sc6mLhtaDz98/qdaWU1z/1i0CPwj/Bo9g5vYcoy8NM/DePpZnVvEzfviYhAPNw88N2FyAClnjXx4LAKfcwsOlXev2+Jn1M/oyTuJjTVGGLUWxbGOvP8gtia1h0L37o/s89enLPPHyRWbvzrM8vUq0IbJ5dWD2rxaLuu2XBYBTbG1pjcm3p4qe/W+INkSsC6dMqZUUk9dnWJ1ZJdoYXgE1dzUW3DYIgrDQXQGqsDqXoKmzke4LnVXLLqpltV7XZz8sAJxSmbUM1791Kzy7L0Fcof+ABjpPm52L7awtpViZWmHw6T46t82WLoeqMv7GBNl0lvrWevoe7z6UUhinlc3u3R8LAKdIJplh+vYcycU1/G0Ttorx6jwGn+7b9/q7e7XxJT0O7k+e2ftzXn+Yt9hOsL4MZNtI/mIyjuPQ3NXI8lT+VYD6SmI+7H9Or2ZYmlzm3AfP0NJjg797ZY3//lkAOCVW5xLc+M6dnBLPxUQbI5z/0NlD7Wve6Jc9Loa4u6cgoIGyOpso/KAIq7OJgo330LMDXP/mtkHgIqmf6iv3X3/A5ZcvWknpPbDGvzIWAE4BVeXOD++XXJt3gzjQNtR6aI1/oS/ocdDAEEPcBcq8GpBwglaxiheOW7jRrmuK8cQnLzJze57V2VUcz2XhwWLBcZl0IoOf9staR8FY418N9kk7BdKraTJru1f3hHDSVqmlD6up4Be0e/hQ9l1KMDVG5KkGGhgicfs+Q727Xw2ICPGBOAsPFvPO4B1Xik6gA/CiHr2PdQFdrC2nwtcoQAPl4VuT9D3RszlD2xRWfl2fsJvNGv/CLNn4FNgtzdPxHBzPwY26jH5whFhjeev/VuK4Nv6wdRyRpxo2u6XKGZ8YeqafaEN0s1CcuILjOYx+YKTsbptYU7Tk+sszd+a5+vUbZNaqU3X0NNqe47+Xxv80F3XbL7sCOAVijVEiMa9gjf9oU5Qzzw8iIjS0H86ShaUuzY8LZ/QswQ+/s6crgUjM4/LLF1h8sMTqXJJoQ4T2kVa8aPlfIxFh9APD3Pj2bfyNcYHtNJy8N3V9moGn+/b5251eVtStuiwAnAIi4QpcN797lyAIUxTFCZc5PPv+IRrbD6/vfbd+2eNCk1M4L/7CnoOA4zi0DbXSNrT/zKn6ljqe/NRj3PnhGIsP88tCawALE0sWAHaw2b3VV5UAICIvA/8ScIHfV9Xf3vH4rwP/AniwftfvqOrvV2PfJtTU2cjjn7jA9M1ZkotrNLTV03Wug2jD4fUll7vYxnHwWOuFfQeBanBch6auJpYmVwpmbRVap6CWVavx11SW7NgiOpeEiIs71ILT2VCzmVcVBwARcYHfBT4OjAOvichXVPWdHZv+O1X9QqX7M8XFGqMMHtFZY7mDcgfFzyrf/0qK17+ZJpWEnmGHj36ujuHHCn/Ery3cyAkC3L5D5KmxQw0CbUNxHv5sMu9+xxW6zu9tYtlpVrXGfy1L+m8fQHajmGGG7GIKZ7CFyIXDSYw4bqpxBfACcFNVbwOIyJ8BnwF2BgBzTGRTWcZ/OsHC+CKq0NLbxODT/cSa9jc4vNd+2YPw8ItZEjcVXR87nbwb8Cf/e4KB/9qlfiQ/1+Gzow2bVySXYnEYPYsDRJ4aI/7UJRb//PqBBQE/4zN7b57EXJJ4f3PYDSRhFpA4QutgnPbhVoIgYO7uArP3whXCOs600THSVlNlO4rV9YHws5VYUX7yLZfXXl8mEocH2gJNhbt9srfmYefM+EAJxpfQoRakSms9nCTV+I0HgPvbbo8D7y+w3X8pIj8PvAv8z6p6v8A25oAFfsC1b94MB4zXex4WHy6zMn2Tyy9f2HP64V7Pzurc6s86XpvOkrg5v9n4b9AszH5LOP/3c/e55i/wpdtJPjvaACS4nlrMCQLB1Bjxv3MwQWBjXWU/Gw4AbzTmXec6iNR5tPQ2Ux+vQwPlxnfukJxPbs7oTswnmbu3wIWfP1sTQWC3om6LM8r/+78FrCV9NAsJoMlZxLvUCQWqagQzRSbyAcFcEre/Ost9niTVCACFPok7OzX/A/CnqpoSkf8e+GPgFwu+mMgrwCsAfQ219wc5aPP3F8mm/Ly/UOAHTN2Y3dMCKPtp/DdysSumirucxV3zkTWfWFA4pznxIJu3zw99ovXIgsC9K+G6ypu/xnr//9zYAu/51GObfdHz44sk59dyynmoryTmEiw8XKJtMF6V4zmOyp3g9ce/H5Bc0a21KQAC8K/P4nY3IjuX1CwWNKXEY6dcNQLAOLB9jv8g8HD7Bqo6u+3m/wP882IvpqqvAq8CPNHec3rWGTwmVqZXC84Y1kBZmV4p+3X21y9bnXQ8TftkXp9A17LrX36FInFFIm7ePr/3tSWghec/tnSoQSDwi1cHDbIByYW1zYJw8/cXw6Jzea+hzN9fOLUBYC+zexM3tPAkGIFgfg23K3fMyelrIri/mP9ZUXA6j88s9cNUjYlgrwEXROSsiESBzwFf2b6BiGwfmfw0cLUK+z31kktr3P3b+1z9+g3u/HBss3hYJUqVfo6UmKC03fYvKZA34BvK7/apVjpe5q0pNJEBXyEo3viHh+Gh2+vxbzuOjcHC8DjDBmAzeI2e3ZwwthHkKi1mp0rRWXuBH+RkAxUrLQFhKupptNfSDiXXryjwmHe2FWmKgrvt+Y7gPdGZf7VQIyr+rVU1C3wB+Bphw/5FVX1bRP6piHx6fbN/KCJvi8hPgX8I/Hql+z3tlqdWuP6Nm8yNLZBcXGP+/iLXv32LhYdLFb1ux9m2gl8OcYXuC+VnnhQq7FYoz7/aff6a9tGFteLTn3f+bvMp0j96UDII1LmteUFA6rvzgkDD6FBFQcD1HKRYw65Q17JVn6njTFvBIOC4Tvg3PGX2U9cn1Rot+jFw2vLXqBbXIfJcP96T3TjDcdxzbUQ/MIhbw8tvViXsqepXVfWiqp5T1X+2ft9vqepX1n/+x6r6hKo+raofUdVr1djvaaWq3LvyIOz/3fYJV18ZuzJOEATMjy/y7nduc/WvbjDxziOyOxq4YqINUUZfGt4sD+GsN0qDT/XR1FF4YZPjRDP+3vtrMz7Zuwt5d28Ege99LcgLAtcWbuQFgY3SEUO9d/cVCPxsULRSq7hCcnFt83ZzTxNtw605QcBxhfaRVpqKLEBzUu23qFvyTCMSc7c+Dxtn9Jc7kSJLaooIbmcDkQvteCOtSI0X3qvt3/6Yyq5lySQLF3cLfOXOj+6zPLG8OUC4tpRi5vY8j3/8fFmVJON9LTz16cdZnlpFA6W5u/HETDyS+hJZSk6Rcp0KwXQCLuZf4fTUt/AoucT3vhbwoU+08qXbC4Dw2VHdmiswur/SEXnHXiJubQz+zt6dJ51IU99az9Az/XSebWd+PAxebYOtJYvOnUS7zR95a/pdxq7F+O54ivoRaGxs2xzU7463oi+14E+soPNrUOfiDbQghzj58aSzAHAMFe0mIBysXXywlHtlECjZtSyPrk8z8FR5E8Ec1yHed/KyrMQR3PPt+Dfmwv7/DY4gXQ3odKJgECiVNrkzCIQZQjuCQBVmDW+854sTy3ldWKrKu9++vXW8rhBtiHLpI6MMPt1f1uufNKUSCaS+m7/8/nW+/LsBmSARvl0+rA6u0nOuZ/M1xHXwBltg0Mo97Edtjnwcc17UK7o8oAaFMx9UlfnxwmWGTxtvsAX3cifUe2Gd/sYI3pNdRIqNYTiCM1A62O3sDoKtLofN7iC2uib2Wkl0w/CzA0TrIzkVRYG8onDqK6nlFPd/8jDvNU6D3bLIrtx9l3//fwWkkhCkQFPhvI6GsSRBFZIhTMgCwDF15oUhvJiLs9GXWUa392nNDtkpWFjDvzUPG/n0jiANESTm4j7WEXYFbbxfriCtMdwyzhAPIwhE6iI88clLjDw/SO/jXeFynCX+bAvjS2ixVWhOqHJSiN/6gUe20LBWoGRvzp+69+So1EaLcQLFmqI8+cuPMfS+PrrOd+za/osrdBzSQi9HSRMZMj+ZhGQ27AJS0OU0mSsTaMbH62sm+tIg7rk2nDNxIk/3EHlvb9kzZw8jCIgjtA3G6X+yNxwYKJHGqrr7Ep8nSbnzR16/k0aL5DXoUors1Zm8IKCq+I9WSb8+QfpvH5C9Mx8mDZiiLAAcY47n0HG2nYGneksu+iKO0NjeQNf50x8AsmOLuX3/GwLFnwgnskmdhzfSSuRcO07b3tdA2Fg4JD8INOQFAad7eN/dQQDR+uLzMgDqW+u2rgJPuL1MHqwfdtASv3bwaBVdSOXcl702Q/adaXR+DV1O499dDFOALQgUdTo+WaecOEK0WPaLwJn3D3LhF84W7QJKraa5d+UBb//ldd799i0WHlQ2l+Ao6Uq68AOBFn9snzaCwM4JY9cWbuRNGIs81bCvCWMdZ9uKZgeJA8PPDFTwGxwfG3V9yp05fu1BnCBaKgIo/qOtmevBSppgcjX35CBQSPtk79XG2Nh+WAA4AUSEgad787KDHFfoPNdB22Br0bPctaU1rn79BrN350itpFmZSXD3R2M8fPvRYRx61UljkUDoSPHHKnDQs4aj9RHOrs/LEHdr7KK+tY7HPnbhVKR9lirqtrPxr3Nbw/faEepfGIRSQWCbYLZw9hcKwVTh8hvG0kBPjLbBVsRxePDmBKmVNF7Uo+dSJ90XO0s+b/zNybzaP4GvPLo2vVmB8iRxh+P5Z3oAwoFVc9xIE33tGwdTP2hrXsbK+ryMprLmZawtp5gfWyAIlHh/M43t4cImqsrC+BJz9+ZBoH2kjdaBlkNf9GRvE7ySBRdu9863k70+G5b92M4R3J7GnNvF1ELl1P06Wd/+Gtfa30Jrf/n5zmvLKZYm85cchPBLsTy9QnsFSxseBacxSuSpHjLvTIcLe7De5/9kF1Kg0dRUlmAxhXgO0la370bwoINAOEeg/L/t5LUpJt6eCgdCFaZvzBDvb2HkhUFu//W9sOjfeqO5/GiFue4mRj9Y/uL1ldrv7F7IrRnl9DQhEyvoUmorCLiC092ItG6Ve3C7GvFvzudfBTiCU4NlnstlXUCn1IO3Jrn69RvFa+YQ1qY5iZyOeqI/N0TkhX6i7x8g8uIATnMsZxtVJXNjlvQP7pN9Z5rMm49If2+MYClV5FV3d1RF5HZKLq0x8c5UzpyQwFcWHy7z4M1JVqYTOWWkA19Znl4NJ6Adgmo1/hCeqETe24t3uQunqwGnp5HIe7rxHu/MCWZS5+FeaM9PAY6HKcDBSppgNoGmdiwaUePsCuAESiwkWZpY3lw9KtaYW8VzZWaVqXdnSqcPCjSfkCJYqorOJfEfLoOvOD2NOD1NOI3Fq5cGk6sE48vrKZbr74OvZH4ySfTnhorWitnN9iuBD33C4Uu3F3KuBDZLR7B1JZB5M8EQ1VtTYG5soeDfNvAD5u4tFC4jnQ2YG1vY0xXkfuyr8f9PPk7Cp0PqUMcP6/tsI47gdjfidpeugeQNtuC21+NPrqDZALejARo9MlceoquZMDAoOD2NeI91WtcQFgBOFFXl/usPmL0XNgAi8PCtR/Q/2UPPpa7N7WZuz+XNLN3OcYVzHxg5MemF2euzBBMrm/3+wfwaMr5E5Jm+og25P7ZQOF1UlWAmgduz/+BXqH7QRhAoVT9oiLsAFQcC3VzTttCDxZ930L0/+2n8v/+VDM3vLOOmAjKyBKo4fc14lzr21V0lDRG80a1qqekfPcjLDgsereLHXLxzpz9tejcnowUwACxOLDN3bzFs3BU0CEtDPHz7UU4lSb9E3nN9vI4nP/XYvs/+t3dn7CzeBWz7km9N169kFbBgKZV8EezWAAAdD0lEQVTT+Id3KrqSCa8IithZ/jnnuanK88ILVRI9iNIRhcT7WwoGb3GF+EBz0TLS7SMHV0a6UFG3/Ma/Ie/MP351BTfph39fP1zbIZhYwR+rPHUzWE6H60bkPaD444fTHXbcWQA4QWZuzRa8vFdfw4yPda0D8aKNQPelTrzo/i78dqvcWPBL/rVgc2LVfmTHl4pO/Aomi6f3OS359eDDAxWceKzwY3t0WPWDdmrqaqS5uzHnb7wxV2Tw6T5a+ppzAoTjCvH+ZloquOopZfsEr52fi+upxW2fi9ylQbsysa1yHtsFij9W+VwVTWWLl1DJBlZOAgsAJ4qfKX4mvf2xtqE4saZYTh+nOEKsObrvpQRLfclzG//cL/l+G35VJXN9Bp0of5nKHM0FxgcEpCWGtFQnAMDRBAERYfSDIww9M0BjRwP1rXX0PdHDYx87jxf1OPviMKMfGKZ9pJX2kVZGPzjCmfcPHUgG0H6WBt34XOhaiSuxKszedZqihecGANR5h54WexzZGMAJ0jrYQmIhmde/73gO8W2pbo7rcPEXzzF9Y4bZe2Et+Y6RVrovdO6r33/XL3lqkWqv/xvMJgkelmj8HcHpK3xGGyynCQrN/lRwL7RX/Yt/WOWktxMROs600XEmv1tHRGjpbaalt5lsKkuw4woqCAKyKR8v6lY0DrS/daG3gqbTHC06ZlGNmv5S5+F0N4UTwXaUDvfOn75V1fbDAsAJ0jnazsytOdKJzGYWiLhCQ1s9Lb25uc6u59D7eDe9j+cv07gXlX7J9yt4UKTrB9bP5KNFJ37544tFz/yCiWXcKl4BbNhLEOD2HSJPjVUcBEpJraS5+7f3N9eRjtR5DD07wOpsIswQU0WArvMd9L+nd89BsRqfC2mI4HTUE8wm8xpot0p1rbzHO/HrXPz7S+EYQ52Hd76toiSA08QCwAniei6Pfew8UzdmmBtbxHGEjrNtdJ7bOqtVVZYmlpm+PYef8WkbjNNxtg3X2/uKX0fV+MN6pksR0lpXssKnpvzi2TCluh0qtNUdtLQjCNTnBIGNCWMHFQSCbMD1b90ku61/PZ3IcOv7d8OZwuuNrQJTN2cJfGXofeUvOlPNz4X3ZDfZW3MED8IUX+o9vAsduJ3VKYEhjuCda8cdbQO1WcE7WQA4YdyIS9/lHvou9xR8/P5PHjJ3d35zIlBiPsn0zVke+9j5PS37eJSNP4DT1Yi/lM6/CnAFd6il5BdZGiPobIFFQxxB2osMDldRWERuCcidNbyRIbV91nDkqTHiT+1t1vBu5scXCbKF6+LklVD2lZnbc/Q/2VPW56NUXZ+N328vnwtxhMiFDvR8e04DrWkfXcsi9V7BGd57JSLFB4RrmA0CnyKJhSSz2xp/CL/g6USGqRszZb9OscqNhYp3QfUbf1iv67N9wW8IG/CmKE6Js8NgYY3gfpEMkoiDe0jLYB7lrOHk4lrBbLFixBHShdIld9hL479Z1I3SnwsNlOzYIpkfPSD9w3Ey786SfvMR6b8eI/P6BOnv3yfzzvSpWhPhOLEAcIosTSwX/KJooMyVmVd9EF/y/RDPIfrCAO5IHGnwkMYI7rm2cPJXif7q7PXZot0/kWf7kEMsf3FUQaCuOVYwDbgYDZRIsXLj68eT97l48RfyPhdfui2bn4tipR1y9qtK5o1J/Fvz4UzdZJbg/lK4rnPA+rwAJXi0Svbd2bJ/H1O+qnwbRORlEbkuIjdF5DcKPB4TkX+3/viPRORMNfZrcokjJWrLl24QDupLXgnxHLzRNqIvDRF9cRBvOF7y99Bd1gTIvvGIYHn/tYD24yiCQNtwvOgM6Z3vnzhC21AcL1q4m6WadX120rm1sMjbbmf3gRJMrKB7uKox5ak4AIiIC/wu8EngMvB5Ebm8Y7O/B8yr6nng/wT+eaX7NflaB+IU6ugUV+g8Wzzt7SC/5IdKKNnPq4kMmR9PoMnduzuq6bCDgOu5XPzIKHUtMcQVHNfBjTr0PdlDU2cjSJg6HNaSamH42cKLzhz058KfTeSXeS6l2Oxus2/VuAJ4AbipqrdVNQ38GfCZHdt8Bvjj9Z+/BHxUbBZG1cWaovQ92ZOzcIzjOjS01dNZpO7JqWn8ATIB0rbLIG+gZKswy3SvtgeBjdIR24OA1HfnBYGG0aF9B4H6ljouf+Iilz9xkTMvDuJ6Lo+uTbM6l0AkXJP4yf/iEmffP1xwLsBhfC7Ec/Y2MFvkKsXsXzUCwABwf9vt8fX7Cm6jqllgEeiowr7NDr2Xunjso+fpvthJx9k2zr44xMUPjxZcLvK0NP6aDcJyz389hi7u0sWjoNvqJh2mQvWDNoLA5qzh0bN5s4Y3uuf2I1LvMXblAelEhiAbEGQDNFDm7y+w/KjwRLvD+ly4vU3lVahz1jO/TkjxwpOkGu9oob/gzuu6crYJNxR5RUSuiMiV+VSBVD6zq/p4HYNP9zHy3CDx/sIrQZ2Wxh8g+/YUwcy2gcNdSIkBz4N22EXkliaWc7LCNgS+MnltOu/+fRV122e9J2mI5NbwF8KfmyJhy+RImPY7Esc9ZzN3D0I15gGMA0Pbbg8CD4tsMy4iHhAH5gq9mKq+CrwK8ER7j+V+7WJzRvAeJriUX9StOnV9DpKuZQnm1kqWQc7hhA3K5vM1rCxK1keaY4eSJXSYpSPSyUzRFMrMWu7iKHst+VGNz4U32ILb2YA/HZZrcDoacJqi4TFnfIi4xSf8ZXzIBGFdH5vgtS/VCACvARdE5CzwAPgc8F/t2OYrwK8BfwN8FviWWim+iqRW09z/8QOWpsLL+OauRoaeGaCuuXSZg9PU+ANoMru++kqRDSJOTpaJe6kDZ70UhCYyZH76CF3Lbi4W4o7Ecc+2Frxq2lyYZm4NiTq4vU1IbH9focMKAo1tDYgUjo8NbfWbP+9v4l91PhdS5+EN5RYpFEegyHuraZ/M29PofDLsQnIE90I7ni39uGcVn+6s9+l/AfgacBX4oqq+LSL/VEQ+vb7ZHwAdInIT+F+AvFRRU75s2uf6N26y9Ggl/GYrLE+tcv2bt8iUseTdRpfCdhtdD9ttVLc8zqTBK55G6DlEfm6IyPv6iDzdS/TnR/DWJ4JpoKRfnwjrxW/Wolf8e4v4kyvhOgTLqc2Zs+oHZH48QeZnUwRji/i35kn/YBx/ap/VSjmcSqIN7fXUt9bnp3+6Qv+TPTmvdRCzvtUPCGYT+DOJkuU9yqWqZF6fCBt/JfzbZQP867P404mKX7/WVKUUhKp+Ffjqjvt+a9vPa8CvVmNfBmbvzuEXWvbPD5i5NVu0TMRpJDEPp6uRYDqRX1DsTGs4+F2g/n8wl9xcVD73AcV/ZwZ/I5PKc4i8p5tgNokup7auNBRQJfv2DE57w767jiotIleO/l91ePuvPe5fzeBnId7p8J6fj9ExMLW5zUE0/v70Ktm3t40zKLiPdWwG4d1oMkOwmgmrejaF5b11MRVese2M+YHi357H7apODaFaYbWATqCVmUTBJR81UFZmau8syLvcRfbd9WUjAQTcM624wyUaqUKNyHYb76/vk/nJJLhO4W4mgWA6gVukNHU59ltEbi9eugQvqhYsiLZz1jdU4cw/mSH71nTe1Zl/bRanKYpToqtSA10f2E9uds1JY4TI073hFVuRv9thz+84DSwAnEB1jVEW178YOQTqmoovlH5aiSNEHutEL7SHg4LR4gOHm88ptGBMMUrhq4WNx4LqzFDdaxE5gMyb5Qf8QgXRqlXUbafs+FLhktyB4t9fwrnclf/YxnNvzoWN/7bgoctpMm8+wjvfXnTugNRbc7ZX9o6dQJ3nOpi+NZuX3ieO0HW+dqdXiOuEZ+rlbNsSQ5qjYSmC3dIRgrBMMcnC4ytOe33B+/djo0votW/kBgFIcD21mBMEgqmxzQa8ErvVe9pX+u9a8ZLcupYNxwYerYZdd56DO9CM01qHqoaloQvVtFpJo1EHqfMKXgmorwQLazitB1/x9bSwAHACxZqinH1phLs/Gts8yRKBkecHqSu2Fq7JISJE3ttL9sZsuLZwoOAJFCqj7ApOfxPB2FJ4JaDb72+u+ryCvQSBiuVlgG0MRFdW70na6mAmkd+QO4LEY2ReexhmcK0/HkythhlYI/HiyzgKSNon8kwf6Z89goUdk/6SWTI/mSTybN9mppcpzQLACRXva+apT19mdTaBAo0d9QVn+5oSsgFOcwxpjuF01CNA+kcP8ieTuU6YptjXTHZsMVxrIOLiDreULE1diXKDQDUcxMQ/t7cJ/+5C/qLvrkBATuMPbGZgOb2NYcmHQovFK0hjFIm6RM61h2MzOwPM+mCw897ePR9zLbIAcIKJIzR1NR71YZxI2dvz+NvWDfZvhOsFR57pI3ttZrOqqLTWEXm8c7N7KXKhAy4czjHuFgQea63OgRRq/L//H7J4iSxtLfvL8xfPIfp8P9l35whmVkHB6WjAu9hOulDDDaBKMJPEPd+Of3UmL6vL6W9C1usBBUup4st+HnLF15PMAoCpOZm7CwR3FvLu92/M4bzQT/SFgTBnXTjy+jPbg8CHPuHwpdsLm0Fgq7++chuNf4w4V35niZaZFOIIGV3ZzMCR2N6KsUnMI/Ke/PklhfIXNh8AvN4mRCB7cz7M1oo4uMPxnBncEvPCUhEFsuEkas1auazPwNSUYDZBcGu+yIOK/3AZCM9gj7rx31C8iFx1/m0/83/tD1aJzqYQZXNyXJiBM1m138fpbcpd6W2TbObxuz1NxD44RPQXzxD7+RG8M7mzs52uhsKv4QjumXj+/aag4/EJN+aQZG8ULEG1JX08Fx0pXESu8mKJOSU//pNP/eQaUuAt0JUMwWrxxXb2wh2OhymbO5b7dM/E8wbUi1WNF0eIPNMXLhvqhkXjcAjHZbqtW7Rcdq1kaoqulpgs5IDTUb2UzmorPGu48iCwUdenu66FtF8kQAphd0xj5fNMxHOIPN9PMLUjDTS+tww2pylK9IND6FIKzQQ48VhVFpCvJRYATG2JOOFksUJiXsVnj5oNwq6TqFNy7eL92hkEKpWX6VPnhjn8O6kiVZxkKK6D29eMW2ZZiKKvI4LsMXCYLRYATE1xh+JheuLOLBRPiDzfv++ywrqWJfPONLqwvthMzCPyeGdVJ4lt2B4EqvV6G9xzRTJwehr3VfnUn0ng31tA13ycliju2bbNuj7m6FkAMDXFPRMPi4xNrmz2QUvMI/Lenn13H2igpK88zM1dX8uS+ekjIs/1lax7s18HVZ7b6w1rGvm35sIrATdcjcstsaZ0Mdn1qqmbk73WsgQzSSLP9O65u8ccDAsApqaICJHLXehoG8FKGom6SHO0ou6aYDpRvLLonQWcp05GdVbNBujiGk7Mw3muH13NIA0eTt3eZzprNshp/DcFSuYnk0R/bvhQFt8xpVkAMDVJ6jzcuup8/HU1XXQpymBl75kz6gfhugQPl8FXnI56vHNtB7qUZfb+Ev7NOUC3qp6uXyE57fV4T3ahC2tkb82jiQwS83DPtobr+hb6HZZSxRd895XM21NEn7bZukfNAoAxFZJ6L0xDLBAEnIb8Rls1zK0H8q4+VMMz5O1rDwSPVknPJom+fwDZEbRUNayRn8wgjdF91cAJ5pNh41/gbB3CtRMyr0+ES2eu36eJDNmrM+haFu9MgcFot/QVlc4k0YxvWTtHzAKAMRVyuhvhxlx+AFhflGa7YD5J5q2prW0dwbvchbteU0jn18IyFDt7lPyA7L1FIpe2qr1qKix+phtr+ypIU5TI+3oRzwm7dJZTEHVxSqRv+vcWi6+qBuFksKUCVzLrXVzuUEvepDlpiRUNigC4gmYCCwBHzAKAMRXQQPEnlsMCZhuVQh3AcXAvteeUJta1LJk3HuU2tr6S/dkUQV8jupAKFzov1GgqBJMr+B31YeE6ETI/m8qb16DLKTLvTOE0xcKGfWNBlXqPyNM9BbuRdG33ZUSLkrBM885UTBHBe6qH7JWJ4k+tUhec2T/7CxizT6pK5o1JdDG11agLYQro8/04O85uSy2SEjwoY23hbED2ralwEfUnuja7kXIPCnQ6iT+7lrugymqG9OuTRD8wmDfgLfFY6QlypShFz+LdeB3BaBvBnfnc4j+O4I627Tvl1lSPDcMbs0/BTDIc7Nx+Rq9Ayid4tJr/hBLLGZbN17D//dZ88UFWKNylk/HR+bW8u70zraX77B2BBi9/fxIu1SgFxjm2XjuOe6kjLNkAEHNxL7XjDVu9nuPArgCM2adgerVwd02gBFOrMJibqy8tMZhNlu5vL4eCziXDVYD2wleyt+aItOZOeJP6CJFn+8henw2vZmCrsXcEdziOM9RC5o1J2D4WEHWJ7JLiKiJ4Ay14Awczb8FUxgKAMftVolqoFDijdgeadx9wLZeCMxonuLPj9RwJxyOK9Ovrcprs7Xki59tz7neaY0Sf60e3d1FlA/DCkhaa9sOJbg5bA9SZAH8mgTdojftJVVEXkIi0i8hficiN9f8LThcUEV9E3lj/95VK9mnMceH2FSlr7ApOgTNeibhhuYnWWHiGLYS1d/bRFS7xGJEzbbiPdWxW1pTGCN6TXUQe7yz+zVYIxpdyG/rtryuy9S/ibo4XZG/NQdrPzU4KFP/GXDhwbU6kSq8AfgP4pqr+toj8xvrtf1Rgu6SqvrfCfRlzrDgtMdyReO5Z/XrdnGJVRZ2GCNFn+1F/I2NIyPx0fSDZ1/xgUKiddgTvYpgO6vU1Q4GCau5IK36BRW+AcD/KngJPMJUofCwCwWyy6IQwc7xVGgA+A3x4/ec/Bv4zhQOAMaeSN9qG09NI8GgVVcXtaixrMtb2vPnIe3vRhTX82STiOThtdWTfnduaTetIeKUQgBOP4Z5pzcvrV9X1+QOKNMfC7qa7C4Ub7TrPMnAMUHkA6FHVCQBVnRCR/PXfQnUicgXIAr+tqn9e4X6NOTacxijO6P4rXIoI0laP01YfFpb7wf2twnIbK3MlfSLP9uUFF396ley7c1t9/usBw3u8E6eviWByNW+MwLuw98JuTncDwcRKfkDR472Ggilt1wAgIt8AChXt+M097GdYVR+KyCjwLRH5mareKrK/V4BXAPoaKqsVbsxJE0yvFi8sdze3sJz/aIXsOzP5aai+kn1nBu+ZXqQugn9/ETIBUu/hXmjH7dr7mgfeuXbSs8lwLYVt3V3uxXabzXuC7RoAVPVjxR4TkUci0rd+9t8HTBV5jYfr/98Wkf8MvA8oGABU9VXgVYAn2nuqkC5hzMmhq5myCsuparhoerGMokAJ7i8RebIb72wrqlpRxVOJukRfHMSfWEbn1sJ8/oEWnGar7X+SVToR7CvAr63//GvAX+zcQETaRCS2/nMn8EHgnQr3a8ypJA2RopOycgrLBVo01XODJrZm91ZjdTLxHLyhOJGne4g81mmN/ylQaQD4beDjInID+Pj6bUTkORH5/fVtHgeuiMhPgW8TjgFYADCmAKeroXAAcAT3bGvO7YIpqNvIPiqDmtpS0SCwqs4CHy1w/xXg76///APgPZXsx5haIa5D9Nn+sNBbIrM1qHupI2cVLRHBGWgmeLCUXzkUwBW8ESu3YEqzmcDGHDPSECH6/gF0LYtmg7DeToEuHO98O9lklmAukRsEGiNELncd6AIy5nSwAGDMMSV1Xsm5WuIIkad70EQmHCCOOEhTNK8KqTHFWAAw5oSThghuiYqcxhRj5aCNMaZGWQAwxpgaZQHAGGNqlAUAY4ypURYAjDGmRlkAMMaYGmUBwBhjapQFAGOMqVEWAIwxpkZZADDGmBplAaAGJW7f3/w5mBoDQJPb1/JJALDmby0q/ii5dCjHZow5PBYAasz9yTNAGAQyb4YNffDD7wBhEHis9QIAnx0NV5pa8xf40CfCj4kFAWNOFwsANciCgDEGLADULAsCxhgLADVsL0Hgs6P1FgSMOWUsANS4+5NnuD95Jj8I3L6TEwQgYUHAmFPGAoAB2AwCi39+HVjPDloPApdiG2vLbgWB5z8WNv4WBIw5uSwAmE0bXUI7gwCQFwQACwLGnHAWAEwOCwLG1I6KAoCI/KqIvC0igYg8V2K7l0XkuojcFJHfqGSf5uBZEDCmNlR6BfAW8HeB7xbbQERc4HeBTwKXgc+LyOUK92sOmAUBY06/igKAql5V1eu7bPYCcFNVb6tqGvgz4DOV7Nccju1BIPNmIi8IhBlCuUHgQ59wLAgYc0IcxhjAAHB/2+3x9fvMCbBzrsD2ILCVJprImzD2KLlkgcCYY27XACAi3xCRtwr8K/csXgrcpyX294qIXBGRK/OpZJm7MAepUBCwWcPGnHy7BgBV/ZiqPlng31+UuY9xYGjb7UHgYYn9vaqqz6nqc22x+jJ3YQ6alY4w5vQ5jC6g14ALInJWRKLA54CvHMJ+TZVZEDDmdKk0DfRXRGQceAn4jyLytfX7+0XkqwCqmgW+AHwNuAp8UVXfruywzVHZCALAZhDYGBMAtgWBrau3jSBgjDlevEqerKpfBr5c4P6HwC9vu/1V4KuV7MsYY0x12amZMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yNsgBgjDE1ygKAMcbUKAsAxhhToywAGGNMjbIAYIwxNcoCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjKgoAIvKrIvK2iAQi8lyJ7e6KyM9E5A0RuVLJPo0xxlSHV+Hz3wL+LvB/l7HtR1R1psL9GWOMqZKKAoCqXgUQkeocjTHGmENzWGMACnxdRH4sIq8c0j6NMcaUsOsVgIh8A+gt8NBvqupflLmfD6rqQxHpBv5KRK6p6neL7O8V4BWAvobmMl/eGGPMXu0aAFT1Y5XuRFUfrv8/JSJfBl4ACgYAVX0VeBXgifYerXTfxhhjCjvwLiARaRSR5o2fgV8iHDw2xhhzhER1/yfZIvIrwL8CuoAF4A1V/YSI9AO/r6q/LCKjwJfXn+IBf6Kq/6zM158GVgHLHgp1Yu/FBnsvtth7scXeCxhR1a5yNqwoABwGEbmiqkXnGNQSey+22Huxxd6LLfZe7I3NBDbGmBplAcAYY2rUSQgArx71ARwj9l5ssfdii70XW+y92INjPwZgjDHmYJyEKwBjjDEH4EQEABH5FyJyTUTeFJEvi0jrUR/TUSm3AutpJSIvi8h1EbkpIr9x1MdzlETkD0VkSkRqel6NiAyJyLdF5Or6d+N/POpjOilORAAA/gp4UlWfAt4F/vERH89R2qjAWnAm9WkmIi7wu8AngcvA50Xk8tEe1ZH6I+Dloz6IYyAL/K+q+jjwIvAPavxzUbYTEQBU9euqml2/+UNg8CiP5yip6lVVvX7Ux3FEXgBuquptVU0DfwZ85oiP6cis19OaO+rjOGqqOqGqr6//vAxcBQaO9qhOhhMRAHb474C/POqDMEdiALi/7fY49kU324jIGeB9wI+O9khOhkoXhKmacqqOishvEl7u/dvDPLbDVqUKrKdRoYUnLI3NACAiTcC/B/4nVV066uM5CY5NANit6qiI/BrwKeCjespzV6tRgfWUGgeGtt0eBB4e0bGYY0REIoSN/79V1f/vqI/npDgRXUAi8jLwj4BPq2riqI/HHJnXgAsiclZEosDngK8c8TGZIybhkoR/AFxV1f/jqI/nJDkRAQD4HaCZcDGZN0Tk9476gI6KiPyKiIwDLwH/UUS+dtTHdFjWEwG+AHyNcKDvi6r69tEe1dERkT8F/ga4JCLjIvL3jvqYjsgHgf8G+MX19uENEfnloz6ok8BmAhtjTI06KVcAxhhjqswCgDHG1CgLAMYYU6MsABhjTI2yAGCMMTXKAoAxxtQoCwDGGFOjLAAYY0yN+v8Bo+UrVvPjKW4AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# visualize decision boundary\n",
"\n",
"h = 0.25\n",
"x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1\n",
"y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1\n",
"xx, yy = np.meshgrid(np.arange(x_min, x_max, h),\n",
" np.arange(y_min, y_max, h))\n",
"Xmesh = np.c_[xx.ravel(), yy.ravel()]\n",
"inputs = [list(map(Value, xrow)) for xrow in Xmesh]\n",
"scores = list(map(model, inputs))\n",
"Z = np.array([s.data > 0 for s in scores])\n",
"Z = Z.reshape(xx.shape)\n",
"\n",
"fig = plt.figure()\n",
"plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral, alpha=0.8)\n",
"plt.scatter(X[:, 0], X[:, 1], c=y, s=40, cmap=plt.cm.Spectral)\n",
"plt.xlim(xx.min(), xx.max())\n",
"plt.ylim(yy.min(), yy.max())\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: micrograd/__init__.py
================================================
================================================
FILE: micrograd/engine.py
================================================
class Value:
""" stores a single scalar value and its gradient """
def __init__(self, data, _children=(), _op=''):
self.data = data
self.grad = 0
# internal variables used for autograd graph construction
self._backward = lambda: None
self._prev = set(_children)
self._op = _op # the op that produced this node, for graphviz / debugging / etc
def __add__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data + other.data, (self, other), '+')
def _backward():
self.grad += out.grad
other.grad += out.grad
out._backward = _backward
return out
def __mul__(self, other):
other = other if isinstance(other, Value) else Value(other)
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.grad
other.grad += self.data * out.grad
out._backward = _backward
return out
def __pow__(self, other):
assert isinstance(other, (int, float)), "only supporting int/float powers for now"
out = Value(self.data**other, (self,), f'**{other}')
def _backward():
self.grad += (other * self.data**(other-1)) * out.grad
out._backward = _backward
return out
def relu(self):
out = Value(0 if self.data < 0 else self.data, (self,), 'ReLU')
def _backward():
self.grad += (out.data > 0) * out.grad
out._backward = _backward
return out
def backward(self):
# topological order all of the children in the graph
topo = []
visited = set()
def build_topo(v):
if v not in visited:
visited.add(v)
for child in v._prev:
build_topo(child)
topo.append(v)
build_topo(self)
# go one variable at a time and apply the chain rule to get its gradient
self.grad = 1
for v in reversed(topo):
v._backward()
def __neg__(self): # -self
return self * -1
def __radd__(self, other): # other + self
return self + other
def __sub__(self, other): # self - other
return self + (-other)
def __rsub__(self, other): # other - self
return other + (-self)
def __rmul__(self, other): # other * self
return self * other
def __truediv__(self, other): # self / other
return self * other**-1
def __rtruediv__(self, other): # other / self
return other * self**-1
def __repr__(self):
return f"Value(data={self.data}, grad={self.grad})"
================================================
FILE: micrograd/nn.py
================================================
import random
from micrograd.engine import Value
class Module:
def zero_grad(self):
for p in self.parameters():
p.grad = 0
def parameters(self):
return []
class Neuron(Module):
def __init__(self, nin, nonlin=True):
self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
self.b = Value(0)
self.nonlin = nonlin
def __call__(self, x):
act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)
return act.relu() if self.nonlin else act
def parameters(self):
return self.w + [self.b]
def __repr__(self):
return f"{'ReLU' if self.nonlin else 'Linear'}Neuron({len(self.w)})"
class Layer(Module):
def __init__(self, nin, nout, **kwargs):
self.neurons = [Neuron(nin, **kwargs) for _ in range(nout)]
def __call__(self, x):
out = [n(x) for n in self.neurons]
return out[0] if len(out) == 1 else out
def parameters(self):
return [p for n in self.neurons for p in n.parameters()]
def __repr__(self):
return f"Layer of [{', '.join(str(n) for n in self.neurons)}]"
class MLP(Module):
def __init__(self, nin, nouts):
sz = [nin] + nouts
self.layers = [Layer(sz[i], sz[i+1], nonlin=i!=len(nouts)-1) for i in range(len(nouts))]
def __call__(self, x):
for layer in self.layers:
x = layer(x)
return x
def parameters(self):
return [p for layer in self.layers for p in layer.parameters()]
def __repr__(self):
return f"MLP of [{', '.join(str(layer) for layer in self.layers)}]"
================================================
FILE: setup.py
================================================
import setuptools
with open("README.md", "r") as fh:
long_description = fh.read()
setuptools.setup(
name="micrograd",
version="0.1.0",
author="Andrej Karpathy",
author_email="andrej.karpathy@gmail.com",
description="A tiny scalar-valued autograd engine with a small PyTorch-like neural network library on top.",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/karpathy/micrograd",
packages=setuptools.find_packages(),
classifiers=[
"Programming Language :: Python :: 3",
"License :: OSI Approved :: MIT License",
"Operating System :: OS Independent",
],
python_requires='>=3.6',
)
================================================
FILE: test/test_engine.py
================================================
import torch
from micrograd.engine import Value
def test_sanity_check():
x = Value(-4.0)
z = 2 * x + 2 + x
q = z.relu() + z * x
h = (z * z).relu()
y = h + q + q * x
y.backward()
xmg, ymg = x, y
x = torch.Tensor([-4.0]).double()
x.requires_grad = True
z = 2 * x + 2 + x
q = z.relu() + z * x
h = (z * z).relu()
y = h + q + q * x
y.backward()
xpt, ypt = x, y
# forward pass went well
assert ymg.data == ypt.data.item()
# backward pass went well
assert xmg.grad == xpt.grad.item()
def test_more_ops():
a = Value(-4.0)
b = Value(2.0)
c = a + b
d = a * b + b**3
c += c + 1
c += 1 + c + (-a)
d += d * 2 + (b + a).relu()
d += 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g += 10.0 / f
g.backward()
amg, bmg, gmg = a, b, g
a = torch.Tensor([-4.0]).double()
b = torch.Tensor([2.0]).double()
a.requires_grad = True
b.requires_grad = True
c = a + b
d = a * b + b**3
c = c + c + 1
c = c + 1 + c + (-a)
d = d + d * 2 + (b + a).relu()
d = d + 3 * d + (b - a).relu()
e = c - d
f = e**2
g = f / 2.0
g = g + 10.0 / f
g.backward()
apt, bpt, gpt = a, b, g
tol = 1e-6
# forward pass went well
assert abs(gmg.data - gpt.data.item()) < tol
# backward pass went well
assert abs(amg.grad - apt.grad.item()) < tol
assert abs(bmg.grad - bpt.grad.item()) < tol
================================================
FILE: trace_graph.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# brew install graphviz\n",
"# pip install graphviz\n",
"from graphviz import Digraph"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from micrograd.engine import Value"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def trace(root):\n",
" nodes, edges = set(), set()\n",
" def build(v):\n",
" if v not in nodes:\n",
" nodes.add(v)\n",
" for child in v._prev:\n",
" edges.add((child, v))\n",
" build(child)\n",
" build(root)\n",
" return nodes, edges\n",
"\n",
"def draw_dot(root, format='svg', rankdir='LR'):\n",
" \"\"\"\n",
" format: png | svg | ...\n",
" rankdir: TB (top to bottom graph) | LR (left to right)\n",
" \"\"\"\n",
" assert rankdir in ['LR', 'TB']\n",
" nodes, edges = trace(root)\n",
" dot = Digraph(format=format, graph_attr={'rankdir': rankdir}) #, node_attr={'rankdir': 'TB'})\n",
" \n",
" for n in nodes:\n",
" dot.node(name=str(id(n)), label = \"{ data %.4f | grad %.4f }\" % (n.data, n.grad), shape='record')\n",
" if n._op:\n",
" dot.node(name=str(id(n)) + n._op, label=n._op)\n",
" dot.edge(str(id(n)) + n._op, str(id(n)))\n",
" \n",
" for n1, n2 in edges:\n",
" dot.edge(str(id(n1)), str(id(n2)) + n2._op)\n",
" \n",
" return dot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# a very simple example\n",
"x = Value(1.0)\n",
"y = (x * 2 + 1).relu()\n",
"y.backward()\n",
"draw_dot(y)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# a simple 2D neuron\n",
"import random\n",
"from micrograd import nn\n",
"\n",
"random.seed(1337)\n",
"n = nn.Neuron(2)\n",
"x = [Value(1.0), Value(-2.0)]\n",
"y = n(x)\n",
"y.backward()\n",
"\n",
"dot = draw_dot(y)\n",
"dot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dot.render('gout')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.5"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
gitextract_jag07i_y/ ├── .gitignore ├── LICENSE ├── README.md ├── demo.ipynb ├── micrograd/ │ ├── __init__.py │ ├── engine.py │ └── nn.py ├── setup.py ├── test/ │ └── test_engine.py └── trace_graph.ipynb
SYMBOL INDEX (35 symbols across 3 files)
FILE: micrograd/engine.py
class Value (line 2) | class Value:
method __init__ (line 5) | def __init__(self, data, _children=(), _op=''):
method __add__ (line 13) | def __add__(self, other):
method __mul__ (line 24) | def __mul__(self, other):
method __pow__ (line 35) | def __pow__(self, other):
method relu (line 45) | def relu(self):
method backward (line 54) | def backward(self):
method __neg__ (line 72) | def __neg__(self): # -self
method __radd__ (line 75) | def __radd__(self, other): # other + self
method __sub__ (line 78) | def __sub__(self, other): # self - other
method __rsub__ (line 81) | def __rsub__(self, other): # other - self
method __rmul__ (line 84) | def __rmul__(self, other): # other * self
method __truediv__ (line 87) | def __truediv__(self, other): # self / other
method __rtruediv__ (line 90) | def __rtruediv__(self, other): # other / self
method __repr__ (line 93) | def __repr__(self):
FILE: micrograd/nn.py
class Module (line 4) | class Module:
method zero_grad (line 6) | def zero_grad(self):
method parameters (line 10) | def parameters(self):
class Neuron (line 13) | class Neuron(Module):
method __init__ (line 15) | def __init__(self, nin, nonlin=True):
method __call__ (line 20) | def __call__(self, x):
method parameters (line 24) | def parameters(self):
method __repr__ (line 27) | def __repr__(self):
class Layer (line 30) | class Layer(Module):
method __init__ (line 32) | def __init__(self, nin, nout, **kwargs):
method __call__ (line 35) | def __call__(self, x):
method parameters (line 39) | def parameters(self):
method __repr__ (line 42) | def __repr__(self):
class MLP (line 45) | class MLP(Module):
method __init__ (line 47) | def __init__(self, nin, nouts):
method __call__ (line 51) | def __call__(self, x):
method parameters (line 56) | def parameters(self):
method __repr__ (line 59) | def __repr__(self):
FILE: test/test_engine.py
function test_sanity_check (line 4) | def test_sanity_check():
function test_more_ops (line 28) | def test_more_ops():
Condensed preview — 10 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (73K chars).
[
{
"path": ".gitignore",
"chars": 20,
"preview": ".ipynb_checkpoints/\n"
},
{
"path": "LICENSE",
"chars": 1081,
"preview": "The MIT License (MIT) Copyright (c) 2020 Andrej Karpathy\n\nPermission is hereby granted, free of charge, to any person ob"
},
{
"path": "README.md",
"chars": 2420,
"preview": "\n# micrograd\n\n\n\nA tiny Autograd engine (with a bite! :)). Implements backpropagation (reverse-mode aut"
},
{
"path": "demo.ipynb",
"chars": 57550,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"### MicroGrad demo\"\n ]\n },\n {\n"
},
{
"path": "micrograd/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "micrograd/engine.py",
"chars": 2730,
"preview": "\nclass Value:\n \"\"\" stores a single scalar value and its gradient \"\"\"\n\n def __init__(self, data, _children=(), _op="
},
{
"path": "micrograd/nn.py",
"chars": 1613,
"preview": "import random\nfrom micrograd.engine import Value\n\nclass Module:\n\n def zero_grad(self):\n for p in self.paramete"
},
{
"path": "setup.py",
"chars": 717,
"preview": "import setuptools\n\nwith open(\"README.md\", \"r\") as fh:\n long_description = fh.read()\n\nsetuptools.setup(\n name=\"micr"
},
{
"path": "test/test_engine.py",
"chars": 1470,
"preview": "import torch\nfrom micrograd.engine import Value\n\ndef test_sanity_check():\n\n x = Value(-4.0)\n z = 2 * x + 2 + x\n "
},
{
"path": "trace_graph.ipynb",
"chars": 2934,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": "
}
]
About this extraction
This page contains the full source code of the karpathy/micrograd GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 10 files (68.9 KB), approximately 39.0k tokens, and a symbol index with 35 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.