Showing preview only (3,845K chars total). Download the full file or copy to clipboard to get everything.
Repository: fastai/numerical-linear-algebra
Branch: master
Commit: 694453e105b2
Files: 20
Total size: 3.7 MB
Directory structure:
gitextract_8s6cyyht/
├── .gitignore
├── README.md
├── excel/
│ └── britlit.xlsx
└── nbs/
├── 0. Course Logistics.ipynb
├── 1. Why are we here.ipynb
├── 2. Topic Modeling with NMF and SVD.ipynb
├── 3. Background Removal with Robust PCA.ipynb
├── 4. Compressed Sensing of CT Scans with Robust Regression.ipynb
├── 5. Health Outcomes with Linear Regression.ipynb
├── 6. How to Implement Linear Regression.ipynb
├── 7. PageRank with Eigen Decompositions.ipynb
├── 8. Implementing QR Factorization.ipynb
├── Homework 1.ipynb
├── Homework 2.ipynb
├── Homework 3.ipynb
├── Project_ideas.txt
├── convolution-intro.ipynb
├── graddesc.xlsm
├── gradient-descent-intro.ipynb
└── images/
└── AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9.url
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
nbs/.ipynb_checkpoints
================================================
FILE: README.md
================================================
## Computational Linear Algebra for Coders
This course is focused on the question: **How do we do matrix computations with acceptable speed and acceptable accuracy?**
This course was taught in the [University of San Francisco's Masters of Science in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) program, summer 2017 (for graduate students studying to become data scientists). The course is taught in Python with Jupyter Notebooks, using libraries such as Scikit-Learn and Numpy for most lessons, as well as Numba (a library that compiles Python to C for faster performance) and PyTorch (an alternative to Numpy for the GPU) in a few lessons.
Accompanying the notebooks is a [playlist of lecture videos, available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY). If you are ever confused by a lecture or it goes too quickly, check out the beginning of the next video, where I review concepts from the previous lecture, often explaining things from a new perspective or with different illustrations, and answer questions.
## Getting Help
You can ask questions or share your thoughts and resources using the [**Computational Linear Algebra** category on our fast.ai discussion forums](http://forums.fast.ai/c/lin-alg).
## Table of Contents
The following listing links to the notebooks in this repository, rendered through the [nbviewer](http://nbviewer.jupyter.org) service. Topics Covered:
### [0. Course Logistics](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb) ([Video 1](https://www.youtube.com/watch?v=8iGzBMboA0I&index=1&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
- [My background](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Intro)
- [Teaching Approach](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Teaching)
- [Importance of Technical Writing](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Writing-Assignment)
- [List of Excellent Technical Blogs](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Excellent-Technical-Blogs)
- [Linear Algebra Review Resources](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/0.%20Course%20Logistics.ipynb#Linear-Algebra)
### [1. Why are we here?](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb) ([Video 1](https://www.youtube.com/watch?v=8iGzBMboA0I&index=1&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
We start with a high level overview of some foundational concepts in numerical linear algebra.
- [Matrix and Tensor Products](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Matrix-and-Tensor-Products)
- [Matrix Decompositions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Matrix-Decompositions)
- [Accuracy](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Accuracy)
- [Memory use](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Memory-Use)
- [Speed](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Speed)
- [Parallelization & Vectorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Scalability-/-parallelization)
### [2. Topic Modeling with NMF and SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb) ([Video 2](https://www.youtube.com/watch?v=kgd40iDT8yY&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=2) and [Video 3](https://www.youtube.com/watch?v=C8KEtrWjjyo&index=3&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
We will use the newsgroups dataset to try to identify the topics of different posts. We use a term-document matrix that represents the frequency of the vocabulary in the documents. We factor it using NMF, and then with SVD.
- [Topic Frequency-Inverse Document Frequency (TF-IDF)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#TF-IDF)
- [Singular Value Decomposition (SVD)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Singular-Value-Decomposition-(SVD))
- [Non-negative Matrix Factorization (NMF)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Non-negative-Matrix-Factorization-(NMF))
- [Stochastic Gradient Descent (SGD)](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Gradient-Descent)
- [Intro to PyTorch](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#PyTorch)
- [Truncated SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/2.%20Topic%20Modeling%20with%20NMF%20and%20SVD.ipynb#Truncated-SVD)
### [3. Background Removal with Robust PCA](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb) ([Video 3](https://www.youtube.com/watch?v=C8KEtrWjjyo&index=3&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY), [Video 4](https://www.youtube.com/watch?v=Ys8R2nUTOAk&index=4&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY), and [Video 5](https://www.youtube.com/watch?v=O2x5KPJr5ag&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=5))
Another application of SVD is to identify the people and remove the background of a surveillance video. We will cover robust PCA, which uses randomized SVD. And Randomized SVD uses the LU factorization.
- [Load and View Video Data](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Load-and-view-the-data)
- [SVD](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#SVD)
- [Principal Component Analysis (PCA)](https://github.com/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb)
- [L1 Norm Induces Sparsity](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#L1-norm-induces-sparsity)
- [Robust PCA](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Robust-PCA-(via-Primary-Component-Pursuit))
- [LU factorization](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#LU-Factorization)
- [Stability of LU](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Stability)
- [LU factorization with Pivoting](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#LU-factorization-with-Partial-Pivoting)
- [History of Gaussian Elimination](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#History-of-Gaussian-Elimination)
- [Block Matrix Multiplication](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb#Block-Matrices)
### [4. Compressed Sensing with Robust Regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#4.-Compressed-Sensing-of-CT-Scans-with-Robust-Regression) ([Video 6](https://www.youtube.com/watch?v=YY9_EYNj5TY&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=6) and [Video 7](https://www.youtube.com/watch?v=ZUGkvIM6ehM&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=7))
Compressed sensing is critical to allowing CT scans with lower radiation-- the image can be reconstructed with less data. Here we will learn the technique and apply it to CT images.
- [Broadcasting](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Broadcasting)
- [Sparse matrices](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Sparse-Matrices-(in-Scipy))
- [CT Scans and Compressed Sensing](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Sparse-Matrices-(in-Scipy))
- [L1 and L2 regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/4.%20Compressed%20Sensing%20of%20CT%20Scans%20with%20Robust%20Regression.ipynb#Regresssion)
### [5. Predicting Health Outcomes with Linear Regressions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb) ([Video 8](https://www.youtube.com/watch?v=SjX55V8zDXI&index=8&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
- [Linear regression in sklearn](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Linear-regression-in-Scikit-Learn)
- [Polynomial Features](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Polynomial-Features)
- [Speeding up with Numba](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Speeding-up-feature-generation)
- [Regularization and Noise](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/5.%20Health%20Outcomes%20with%20Linear%20Regression.ipynb#Regularization-and-noise)
### [6. How to Implement Linear Regression](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb)([Video 8](https://www.youtube.com/watch?v=SjX55V8zDXI&index=8&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
- [How did Scikit Learn do it?](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#How-did-sklearn-do-it?)
- [Naive solution](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Naive-Solution)
- [Normal equations and Cholesky factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Normal-Equations-(Cholesky))
- [QR factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#QR-Factorization)
- [SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#SVD)
- [Timing Comparison](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Timing-Comparison)
- [Conditioning & Stability](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Conditioning-&-stability)
- [Full vs Reduced Factorizations](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Full-vs-Reduced-Factorizations)
- [Matrix Inversion is Unstable](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/6.%20How%20to%20Implement%20Linear%20Regression.ipynb#Matrix-Inversion-is-Unstable)
### [7. PageRank with Eigen Decompositions](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb) ([Video 9](https://www.youtube.com/watch?v=AbB-w77yxD0&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY&index=9) and [Video 10](https://www.youtube.com/watch?v=1kw8bpA9QmQ&index=10&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
We have applied SVD to topic modeling, background removal, and linear regression. SVD is intimately connected to the eigen decomposition, so we will now learn how to calculate eigenvalues for a large matrix. We will use DBpedia data, a large dataset of Wikipedia links, because here the principal eigenvector gives the relative importance of different Wikipedia pages (this is the basic idea of Google's PageRank algorithm). We will look at 3 different methods for calculating eigenvectors, of increasing complexity (and increasing usefulness!).
- [SVD](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Motivation)
- [DBpedia Dataset](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#DBpedia)
- [Power Method](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Power-method)
- [QR Algorithm](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#QR-Algorithm)
- [Two-phase approach to finding eigenvalues](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#A-Two-Phase-Approach)
- [Arnoldi Iteration](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/7.%20PageRank%20with%20Eigen%20Decompositions.ipynb#Arnoldi-Iteration)
### [8. Implementing QR Factorization](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb) ([Video 10](https://www.youtube.com/watch?v=1kw8bpA9QmQ&index=10&list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY))
- [Gram-Schmidt](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Gram-Schmidt)
- [Householder](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Householder)
- [Stability Examples](http://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/8.%20Implementing%20QR%20Factorization.ipynb#Ex-9.2:-Classical-vs-Modified-Gram-Schmidt)
<hr>
## Why is this course taught in such a weird order?
This course is structured with a *top-down* teaching method, which is different from how most math courses operate. Typically, in a *bottom-up* approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures. The problems with this are that students often lose motivation, don't have a sense of the "big picture", and don't know what they'll need.
Harvard Professor David Perkins has a book, [Making Learning Whole](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719) in which he uses baseball as an analogy. We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game. Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on.
If you took the fast.ai deep learning course, that is what we used. You can hear more about my teaching philosophy [in this blog post](http://www.fast.ai/2016/10/08/teaching-philosophy/) or [this talk I gave at the San Francisco Machine Learning meetup](https://vimeo.com/214233053).
All that to say, don't worry if you don't understand everything at first! You're not supposed to. We will start using some "black boxes" or matrix decompositions that haven't yet been explained, and then we'll dig into the lower level details later.
To start, focus on what things DO, not what they ARE.
================================================
FILE: nbs/0. Course Logistics.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 0. Course Logistics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Ask Questions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let me know how things are going. This is particularly important since I'm new to MSAN, I don't know everything you've seen/haven't seen."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Intro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**My background and linear algebra love**:\n",
"\n",
"- **Swarthmore College**: linear algebra convinced me to be a math major! (minors in CS & linguistics) I thought linear algebra was beautiful, but theoretical\n",
"- **Duke University**: Math PhD. Took numerical linear algebra. Enjoyed the course, but not my focus\n",
"- **Research Triangle Institute**: first time using linear algebra in practice (healthcare economics, markov chains)\n",
"- **Quant**: first time working with lots of data, decided to become a data scientist\n",
"- **Uber**: data scientist\n",
"- **Hackbright**: taught software engineering. Overhauled ML and collaborative filtering lectures\n",
"- **fast.ai**: co-founded to make deep learning more accessible. Deep Learning involves a TON of linear algebra"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Teaching"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Teaching Approach**\n",
"\n",
"I'll be using a *top-down* teaching method, which is different from how most math courses operate. Typically, in a *bottom-up* approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures. The problems with this are that students often lose motivation, don't have a sense of the \"big picture\", and don't know what they'll need.\n",
"\n",
"If you took the fast.ai deep learning course, that is what we used. You can hear more about my teaching philosophy [in this blog post](http://www.fast.ai/2016/10/08/teaching-philosophy/) or [in this talk](https://vimeo.com/214233053).\n",
"\n",
"Harvard Professor David Perkins has a book, [Making Learning Whole](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719) in which he uses baseball as an analogy. We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game. Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on.\n",
"\n",
"All that to say, don't worry if you don't understand everything at first! You're not supposed to. We will start using some \"black boxes\" or matrix decompositions that haven't yet been explained, and then we'll dig into the lower level details later.\n",
"\n",
"To start, focus on what things DO, not what they ARE."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"People learn by:\n",
"1. **doing** (coding and building)\n",
"2. **explaining** what they've learned (by writing or helping others)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Text Book**\n",
"\n",
"The book [**Numerical Linear Algebra**](https://www.amazon.com/Numerical-Linear-Algebra-Lloyd-Trefethen/dp/0898713617) by Trefethen and Bau is recommended. The MSAN program has a few copies on hand.\n",
"\n",
"A secondary book is [**Numerical Methods**](https://www.amazon.com/Numerical-Methods-Analysis-Implementation-Algorithms/dp/0691151229) by Greenbaum and Chartier."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Basics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Office hours**: 2:00-4:00 on Friday afternoons. Email me if you need to meet at other times.\n",
"\n",
"My contact info: **rachel@fast.ai**\n",
"\n",
"Class Slack: #numerical_lin_alg\n",
"\n",
"Email me if you will need to miss class.\n",
"\n",
"Jupyter Notebooks will be available on Github at: https://github.com/fastai/numerical-linear-algebra Please pull/download before class. **Some parts are removed for you to fill in as you follow along in class**. Be sure to let me know **THIS WEEK** if you are having any problems running the notebooks from your own computer. You may want to make a separate copy, because running Jupyter notebooks causes them to change, which can create github conflicts the next time you pull."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that you have MathJax running (which renders LaTeX, used for math equations) by running the following cell:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ e^{\\theta i} = \\cos(\\theta) + i \\sin(\\theta)$$ "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"check that you can import:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"import sklearn"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Grading Rubric**:\n",
"\n",
"| Assignment | Percent |\n",
"|-------------------|:-------:|\n",
"| Attendance | 10% |\n",
"| Homework | 20% |\n",
"| Writing: proposal | 10% |\n",
"| Writing: draft | 15% |\n",
"| Writing: final | 15% |\n",
"| Final Exam | 30% |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Honor Code** \n",
"\n",
"No cheating nor plagiarism is allowed, please see below for more details."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**On Laptops**\n",
"\n",
"I ask you to be respectful of me and your classmates and to refrain from surfing the web or using social media (facebook, twitter, etc) or messaging programs during class. It is absolutely forbidden to use instant messaging programs, email, etc. during class lectures or quizzes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Syllabus"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Topics Covered:\n",
"\n",
"1\\. Why are we here?\n",
" - Matrix and Tensor Products\n",
" - Matrix Decompositions\n",
" - Accuracy\n",
" - Memory use\n",
" - Speed\n",
" - Parallelization & Vectorization\n",
"\n",
"\n",
"2\\. Topic Modeling with NMF and SVD\n",
" - Topic Frequency-Inverse Document Frequency (TF-IDF)\n",
" - Singular Value Decomposition (SVD)\n",
" - Non-negative Matrix Factorization (NMF)\n",
" - Stochastic Gradient Descent (SGD)\n",
" - Intro to PyTorch\n",
" - Truncated SVD, Randomized SVD\n",
"\n",
"\n",
"3\\. Background Removal with Robust PCA\n",
" - Robust PCA\n",
" - Randomized SVD\n",
" - LU factorization\n",
"\n",
"\n",
"4\\. Compressed Sensing for CT scans with Robust Regression\n",
" - L1 regularization\n",
"\n",
"\n",
"5\\. Predicting Health Outcomes with Linear Regression\n",
" - Linear regression\n",
" - Polynomial Features\n",
" - Speeding up with Numba\n",
" - Regularization and Noise\n",
" - Implementing linear regression 4 ways\n",
"\n",
"\n",
"6\\. PageRank with Eigen Decompositions\n",
" - Power Method\n",
" - QR Algorithm\n",
" - Arnoldi Iteration\n",
"\n",
"\n",
"7\\. QR Factorization\n",
" - Gram-Schmidt\n",
" - Householder\n",
" - Stability"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Writing Assignment"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Writing Assignment:** Writing about technical concepts is a hugely valuable skill. I want you to write a technical blog post related to numerical linear algebra. [A blog is like a resume, only better](http://www.fast.ai/2017/04/06/alternatives/). Technical writing is also important in creating documentation, sharing your work with co-workers, applying to speak at conferences, and practicing for interviews. (You don't actually have to publish it, although I hope you do, and please send me the link if you do.)\n",
"- [List of ideas here](Project_ideas.txt)\n",
"- Always cite sources, use quote marks around quotes. Do this even as you are first gathering sources and taking notes. If you plagiarize parts of someone else's work, you will fail.\n",
"- Can be done in a Jupyter Notebook (Jupyter Notebooks can be turned into blog posts) or a [Kaggle Kernel](https://www.kaggle.com/xenocide/content-based-anime-recommender)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For the proposal, write a brief paragraph about the problem/topic/experiment you plan to research/test and write about. You need to include **4 sources** that you plan to use: these can include Trefethen, other blog posts, papers, or books. Include a sentence about each source, stating what it's in it.\n",
"\n",
"Feel free to ask me if you are wondering if your topic idea is suitable!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Excellent Technical Blogs\n",
"\n",
"Examples of great technical blog posts:\n",
"- [Peter Norvig](http://nbviewer.jupyter.org/url/norvig.com/ipython/ProbabilityParadox.ipynb) (more [here](http://norvig.com/ipython/))\n",
"- [Stephen Merity](https://smerity.com/articles/2017/deepcoder_and_ai_hype.html)\n",
"- [Julia Evans](https://codewords.recurse.com/issues/five/why-do-neural-networks-think-a-panda-is-a-vulture) (more [here](https://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-tcp-stack-in-python/))\n",
"- [Julia Ferraioli](http://blog.juliaferraioli.com/2016/02/exploring-world-using-vision-twilio.html)\n",
"- [Edwin Chen](http://blog.echen.me/2014/10/07/moving-beyond-ctr-better-recommendations-through-human-evaluation/)\n",
"- [Slav Ivanov](https://blog.slavv.com/picking-an-optimizer-for-style-transfer-86e7b8cba84b)\n",
"- [Brad Kenstler](https://hackernoon.com/non-artistic-style-transfer-or-how-to-draw-kanye-using-captain-picards-face-c4a50256b814)\n",
"- find [more on twitter](https://twitter.com/math_rachel)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Deadlines"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"| Assignment | Dates |\n",
"|-------------------|:--------:|\n",
"| Homeworks | TBA |\n",
"| Writing: proposal | 5/30 |\n",
"| Writing: draft | 6/15 |\n",
"| Writing: final | 6/27 |\n",
"| Final Exam | 6/29 |"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Linear Algebra"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will review some linear algebra in class. However, if you find there are concepts you feel rusty on, you may want to review on your own. Here are some resources:\n",
"\n",
"- [3Blue1Brown Essence of Linear Algebra](https://www.youtube.com/playlist?list=PLZHQObOWTQDPD3MizzM2xVFitgF8hE_ab) videos about *geometric intuition* (fantastic! gorgeous!)\n",
"- Lectures 1-6 of Trefethen\n",
"- [Immersive linear algebra](http://immersivemath.com/ila/) free online textbook with interactive graphics\n",
"- [Chapter 2](http://www.deeplearningbook.org/contents/linear_algebra.html) of Ian Goodfellow's Deep Learning Book\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## USF Policies"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Academic Integrity** \n",
"\n",
"USF upholds the standards of honesty and integrity from all members of the academic community. All students are expected to know and adhere to the University’s Honor Code. You can find the full text of the [code online](www.usfca.edu/academic_integrity). The policy covers:\n",
"- Plagiarism: intentionally or unintentionally representing the words or ideas of another person as your own; failure to properly cite references; manufacturing references.\n",
"- Working with another person when independent work is required.\n",
"- Submission of the same paper in more than one course without the specific permission of each instructor.\n",
"- Submitting a paper written (entirely or even a small part) by another person or obtained from the internet.\n",
"- Plagiarism is plagiarism: it does not matter if the source being copied is on the Internet, from a book or textbook, or from quizzes or problem sets written up by other students.\n",
"- The penalties for violation of the policy may include a failing grade on the assignment, a failing grade in the course, and/or a referral to the Academic Integrity Committee."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Students with Disabilities**\n",
"\n",
"If you are a student with a disability or disabling condition, or if you think you may have a disability, please contact USF Student Disability Services (SDS) at 415 422-2613 within the first week of class, or immediately upon onset of disability, to speak with a disability specialist. If you are determined eligible for reasonable accommodations, please meet with your disability specialist so they can arrange to have your accommodation letter sent to me, and we will discuss your needs for this course. For more information, please visit [this website]( http://www.usfca.edu/sds) or call (415) 422-2613."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Behavioral Expectations**\n",
"\n",
"All students are expected to behave in accordance with the [Student Conduct Code and other University policies](https://myusf.usfca.edu/fogcutter). Open discussion and disagreement is encouraged when done respectfully and in the spirit of academic discourse. There are also a variety of behaviors that, while not against a specific University policy, may create disruption in this course. Students whose behavior is disruptive or who fail to comply with the instructor may be dismissed from the class for the remainder of the class period and may need to meet with the instructor or Dean prior to returning to the next class period. If necessary, referrals may also be made to the Student Conduct process for violations of the Student Conduct Code."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Counseling and Psychological Services**\n",
"\n",
"Our diverse staff offers brief individual, couple, and group counseling to student members of our community. CAPS services are confidential and free of charge. Call 415-422-6352 for an initial consultation appointment. Having a crisis at 3 AM? We are still here for you. Telephone consultation through CAPS After Hours is available between the hours of 5:00 PM to 8:30 AM; call the above number and press 2."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Confidentiality, Mandatory Reporting, and Sexual Assault**\n",
"\n",
"As an instructor, one of my responsibilities is to help create a safe learning environment on our campus. I also have a mandatory reporting responsibility related to my role as a faculty member. I am required to share information regarding sexual misconduct or information about a crime that may have occurred on USFs campus with the University. Here are other resources:\n",
"\n",
"- To report any sexual misconduct, students may visit Anna Bartkowski (UC 5th floor) or see many other options by visiting [this website](https://myusf.usfca.edu/title-IX)\n",
"- Students may speak to someone confidentially, or report a sexual assault confidentially by contacting Counseling and Psychological Services at 415-422-6352\n",
"- To find out more about reporting a sexual assault at USF, visit [USF’s Callisto website](https://usfca.callistocampus.org/)\n",
"- For an off-campus resource, contact [San Francisco Women Against Rape](http://www.sfwar.org/about.html) 415-647-7273"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/1. Why are we here.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 1. Why are we here?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note: Future lessons have a lot more code than this one**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Why study Numerical Linear Algebra?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Key Question of this course**: How can we do matrix computations with acceptable speed and acceptable accuracy?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A list of the [Top 10 Algorithms](http://www.cs.fsu.edu/~lacher/courses/COT4401/notes/cise_v2_i1/index.html) of science and engineering during the 20th century includes: the **matrix decompositions** approach to linear algebra. It also includes the QR algorithm, which we'll cover, and Krylov iterative methods which we'll see an example of. (See here for [another take](https://nickhigham.wordpress.com/2016/03/29/the-top-10-algorithms-in-applied-mathematics/))\n",
"\n",
"<img src=\"images/top10.png\" alt=\"\" style=\"width: 50%\"/>\n",
"(source: [Top 10 Algorithms](http://www.cs.fsu.edu/~lacher/courses/COT4401/notes/cise_v2_i1/guest.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are 4 things to keep in mind when choosing or designing an algorithm for matrix computations:\n",
"- Memory Use\n",
"- Speed\n",
"- Accuracy\n",
"- Scalability/Parallelization\n",
"\n",
"Often there will be trade-offs between these categories."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Matrices are everywhere-- anything that can be put in an Excel spreadsheet is a matrix, and language and pictures can be represented as matrices as well. Knowing what options there are for matrix algorithms, and how to navigate compromises, can make enormous differences to your solutions. For instance, an approximate matrix computation can often be thousands of times faster than an exact one.\n",
"\n",
"It's not just about knowing the contents of existing libraries, but knowing how they work too. That's because often you can make variations to an algorithm that aren't supported by your library, giving you the performance or accuracy that you need. In addition, this field is moving very quickly at the moment, particularly in areas related to **deep learning**, **recommendation systems**, **approximate algorithms**, and **graph analytics**, so you'll often find there's recent results that could make big differences in your project, but aren't in your library.\n",
"\n",
"Knowing how the algorithms really work helps to both debug and accelerate your solution. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Matrix Computations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are two key types of matrix computation, which get combined in many different ways. These are:\n",
"- Matrix and tensor products\n",
"- Matrix decompositions\n",
"\n",
"So basically we're going to be combining matrices, and pulling them apart again!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Matrix and Tensor Products"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Matrix-Vector Products:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The matrix below gives the probabilities of moving from 1 health state to another in 1 year. If the current health states for a group are:\n",
"- 85% asymptomatic\n",
"- 10% symptomatic\n",
"- 5% AIDS\n",
"- 0% death\n",
"\n",
"what will be the % in each health state in 1 year?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img src=\"images/markov_health.jpg\" alt=\"floating point\" style=\"width: 80%\"/>(Source: [Concepts of Markov Chains](https://www.youtube.com/watch?v=0Il-y_WLTo4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.765 ],\n",
" [ 0.1525],\n",
" [ 0.0645],\n",
" [ 0.018 ]])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: Use Numpy to compute the answer to the above\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Matrix-Matrix Products"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"<img src=\"images/shop.png\" alt=\"floating point\" style=\"width: 100%\"/>(Source: [Several Simple Real-world Applications of Linear Algebra Tools](https://www.mff.cuni.cz/veda/konference/wds/proc/pdf06/WDS06_106_m8_Ulrychova.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 50. , 49. ],\n",
" [ 58.5, 61. ],\n",
" [ 43.5, 43.5]])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: Use Numpy to compute the answer to the above\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Image Data"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Images can be represented by matrices.\n",
"\n",
"<img src=\"images/digit.gif\" alt=\"digit\" style=\"width: 55%\"/>\n",
" (Source: [Adam Geitgey\n",
"](https://medium.com/@ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Convolution"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"*Convolutions* are the heart of convolutional neural networks (CNNs), a type of deep learning, responsible for the huge advances in image recognitionin the last few years. They are now increasingly being used for speech as well, such as [Facebook AI's results](https://code.facebook.com/posts/1978007565818999/a-novel-approach-to-neural-machine-translation/) for speech translation which are 9x faster than RNNs (the current most popular approach for speech translation).\n",
"\n",
"Computers are now more accurate than people at classifying images.\n",
"\n",
"<img src=\"images/sportspredict.jpeg\" alt=\"ImageNet\" style=\"width: 80%\"/>\n",
" (Source: [Andrej Karpathy](http://karpathy.github.io/2014/07/03/feature-learning-escapades/))\n",
"\n",
"<img src=\"images/InsideImagenet.png\" alt=\"ImageNet\" style=\"width: 80%\"/>\n",
" (Source: [Nvidia](https://blogs.nvidia.com/blog/2014/09/07/imagenet/))\n",
"\n",
"You can think of a convolution as a special kind of matrix product\n",
"\n",
"The 3 images below are all from an excellent blog post written by a fast.ai student on [CNNs from Different Viewpoints](https://medium.com/impactai/cnns-from-different-viewpoints-fab7f52d159c): \n",
"\n",
"A convolution applies a filter to each section of an image:\n",
"<img src=\"images/cnn1.png\" alt=\"CNNs\" style=\"width: 40%\"/>\n",
"\n",
"Neural Network Viewpoint:\n",
"<img src=\"images/cnn2.png\" alt=\"CNNs\" style=\"width: 40%\"/>\n",
"\n",
"Matrix Multiplication Viewpoint:\n",
"<img src=\"images/cnn3.png\" alt=\"CNNs\" style=\"width: 80%\"/>\n",
"\n",
"Let's see how convolutions can be used for *edge detection* in [this notebook](convolution-intro.ipynb)(originally from the [fast.ai Deep Learning Course](http://course.fast.ai/)) "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Matrix Decompositions"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We will be talking about Matrix Decompositions every day of this course, and will cover the below examples in future lessons:"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- **Topic Modeling** (NMF and SVD. SVD uses QR) A group of documents can be represented by a term-document matrix\n",
"<img src=\"images/document_term.png\" alt=\"term-document matrix\" style=\"width: 70%\"/>\n",
" (source: [Introduction to Information Retrieval](http://player.slideplayer.com/15/4528582/#))\n",
"<img src=\"images/nmf_doc.png\" alt=\"NMF on documents\" style=\"width: 70%\"/>\n",
" (source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- **Background removal** (robust PCA, which uses truncated SVD)\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- **Google's PageRank Algorithm** (eigen decomposition)\n",
"\n",
"<img src=\"images/page_rank_graph.png\" alt=\"PageRank\" style=\"width: 70%\"/>\n",
" (source: [What is in PageRank?](http://computationalculture.net/article/what_is_in_pagerank))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- List of other decompositions and some applications [matrix factorization jungle](https://sites.google.com/site/igorcarron2/matrixfactorizations)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Floating Point Arithmetic"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To understand accuracy, we first need to look at **how** computers (which are finite and discrete) store numbers (which are infinite and continuous)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Exercise"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take a moment to look at the function $f$ below. Before you try running it, write on paper what the output would be of $x_1 = f(\\frac{1}{10})$. Now, (still on paper) plug that back into $f$ and calculate $x_2 = f(x_1)$. Keep going for 10 iterations.\n",
"\n",
"This example is taken from page 107 of *Numerical Methods*, by Greenbaum and Chartier."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def f(x):\n",
" if x <= 1/2:\n",
" return 2 * x\n",
" if x > 1/2:\n",
" return 2*x - 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Only after you've written down what you think the answer should be, run the code below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"x = 1/10\n",
"for i in range(80):\n",
" print(x)\n",
" x = f(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What went wrong?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Problem: math is continuous & infinite, but computers are discrete & finite"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Two Limitations of computer representations of numbers:\n",
"1. they can't be arbitrarily large or small\n",
"2. there must be gaps between them\n",
"\n",
"The reason we need to care about accuracy, is because computers can't store infinitely accurate numbers. It's possible to create calculations that give very wrong answers (particularly when repeating an operation many times, since each operation could multiply the error)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How computers store numbers:\n",
"\n",
"<img src=\"images/fpa.png\" alt=\"floating point\" style=\"width: 60%\"/>\n",
"\n",
"The *mantissa* can also be referred to as the *significand*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"IEEE Double precision arithmetic:\n",
"- Numbers can be as large as $1.79 \\times 10^{308}$ and as small as $2.23 \\times 10^{-308}$.\n",
"- The interval $[1,2]$ is represented by discrete subset: \n",
"$$1, \\: 1+2^{-52}, \\: 1+2 \\times 2^{-52},\\: 1+3 \\times 2^{-52},\\: \\ldots, 2$$\n",
"\n",
"- The interval $[2,4]$ is represented:\n",
"$$2, \\: 2+2^{-51}, \\: 2+2 \\times 2^{-51},\\: 2+3 \\times 2^{-51},\\: \\ldots, 4$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Floats and doubles are not equidistant:\n",
"\n",
"<img src=\"images/fltscale-wh.png\" alt=\"floating point\" style=\"width: 100%\"/>\n",
"Source: [What you never wanted to know about floating point but will be forced to find out](http://www.volkerschatz.com/science/float.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Machine Epsilon**\n",
"\n",
"Half the distance between 1 and the next larger number. This can vary by computer. IEEE standards for double precision specify $$ \\varepsilon_{machine} = 2^{-53} \\approx 1.11 \\times 10^{-16}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Two important properties of Floating Point Arithmetic**:\n",
"\n",
"- The difference between a real number $x$ and its closest floating point approximation $fl(x)$ is always smaller than $\\varepsilon_{machine}$ in relative terms. For some $\\varepsilon$, where $\\lvert \\varepsilon \\rvert \\leq \\varepsilon_{machine}$, $$fl(x)=x \\cdot (1 + \\varepsilon)$$\n",
"\n",
"- Where * is any operation ($+, -, \\times, \\div$), and $\\circledast$ is its floating point analogue,\n",
" $$ x \\circledast y = (x * y)(1 + \\varepsilon)$$\n",
"for some $\\varepsilon$, where $\\lvert \\varepsilon \\rvert \\leq \\varepsilon_{machine}$\n",
"That is, every operation of floating point arithmetic is exact up to a relative error of size at most $\\varepsilon_{machine}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### History"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Floating point arithmetic may seem like a clear choice in hindsight, but there have been many, many ways of storing numbers:\n",
"- fixed-point arithmetic\n",
"- logarithmic and semilogarithmic number systems\n",
"- continued-fractions\n",
"- rational numbers\n",
"- possibly infinite strings of rational numbers\n",
"- level-index number systems\n",
"- fixed-slash and floating-slash number systems\n",
"- 2-adic numbers\n",
"\n",
"For references, see [Chapter 1](https://perso.ens-lyon.fr/jean-michel.muller/chapitre1.pdf) (which is free) of the [Handbook of Floating-Point Arithmetic](http://www.springer.com/gp/book/9780817647049). Yes, there is an entire 16 chapter book on floating point!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Timeline History of Floating Point Arithmetic:\n",
"- ~1600 BC: Babylonian radix-60 system was earliest floating-point system (Donald Knuth). Represented the significand of a radix-60 floating-point representation (if ratio of two numbers is a power of 60, represented the same)\n",
"- 1630 Slide rule. Manipulate only significands (radix-10)\n",
"- 1914 Leonardo Torres y Quevedo described an electromechanical implementation of Babbage's Analytical Engine with Floating Point Arithmetic.\n",
"- 1941 First real, modern implementation. Konrad Zuse's Z3 computer. Used radix-2, with 14 bit significand, 7 bit exponents, and 1 sign bit.\n",
"- 1985 IEEE 754-1985 Standard for Binary Floating-Point Arithmetic released. Has increased accuracy, reliability, and portability. [William Kahan](https://people.eecs.berkeley.edu/~wkahan/) played leading role."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"Many different ways of approximating real numbers on computers have been introduced.. And yet, floating-point arithmetic is **by far the most widely used** way of representing real numbers in modern computers. Simulating an infinite, continuous set (the real numbers) with a finite set (the “machine numbers”) is not a straightforward task: **clever compromises must be found between, speed, accuracy, dynamic range, ease of use and implementation, and memory**. It appears that floating-point arithmetic, with adequately chosen parameters (radix, precision, extremal exponents, etc.), is a very good compromise for most numerical applications.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although a radix value of 2 (binary) seems like the pretty clear winner now for computers, a variety of other radix values have been used at various point:\n",
"\n",
"- radix-8 used by early machines PDP-10, Burroughs 570 and 6700\n",
"- radix-16 IBM 360\n",
"- radix-10 financial calculations, pocket calculators, Maple\n",
"- radix-3 Russian SETUN computer (1958). Benefits: minimizes beta x p (symbols x digits), for a fixed largest representable number beta^p - 1. Rounding = truncation\n",
"- radix-2 most common. Reasons: easy to implement. Studies have shown (with implicit leading bit) this gives better worst-case or average accuracy than all other radices."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Conditioning and Stability"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Since we can not represent numbers exactly on a computer (due to the finiteness of our storage, and the gaps between numbers in floating point architecture), it becomes important to know *how small perturbations in the input to a problem impact the output*.\n",
"\n",
"**\"A stable algorithm gives nearly the right answer to nearly the right question.\"** --Trefethen\n",
"\n",
"**Conditioning**: perturbation behavior of a mathematical problem (e.g. least squares)\n",
"\n",
"**Stability**: perturbation behavior of an algorithm used to solve that problem on a computer (e.g. least squares algorithms, householder, back substitution, gaussian elimination)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Example: Eigenvalues of a Matrix"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 1. 1000.]\n",
" [ 0. 1.]]\n",
"[[ 1. 1000. ]\n",
" [ 0.001 1. ]]\n"
]
}
],
"source": [
"import scipy.linalg as la \n",
"\n",
"A = np.array([[1., 1000], [0, 1]])\n",
"B = np.array([[1, 1000], [0.001, 1]])\n",
"\n",
"print(A)\n",
"\n",
"print(B)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"np.set_printoptions(suppress=True, precision=4)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"wA, vrA = la.eig(A)\n",
"wB, vrB = la.eig(B)\n",
"\n",
"wA, wB"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"hidden": true
},
"source": [
"**Reminder: Two properties of Floating Point Arithmetic**\n",
"\n",
"- The difference between a real number $x$ and its closest floating point approximation $fl(x)$ is always smaller than $\\varepsilon_{machine}$ in relative terms.\n",
"\n",
"- Every operation $+, -, \\times, \\div$ of floating point arithmetic is exact up to a relative error of size at most $\\varepsilon_{machine}$ \n",
"\n",
"Examples we'll see:\n",
"- Classical vs Modified Gram-Schmidt accuracy\n",
"- Gram-Schmidt vs. Householder (2 different ways of computing QR factorization), how orthogonal the answer is\n",
"- Condition of a system of equations"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Approximation accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"It's rare that we need to do highly accurate matrix computations at scale. In fact, often we're doing some kind of machine learning, and less accurate approaches can prevent overfitting.\n",
"\n",
"If we accept some decrease in accuracy, then we can often increase speed by orders of magnitude (and/or decrease memory use) by using approximate algorithms. These algorithms typically give a correct answer with some probability. By rerunning the algorithm multiple times you can generally increase that probability multiplicatively!\n",
"\n",
"**Example**: A **bloom filter** allows searching for set membership with 1% false positives, using <10 bits per element. This often represents reductions in memory use of thousands of times. \n",
"\n",
"<img src=\"images/bloom_filter.png\" alt=\"Bloom Filters\" style=\"width: 60%\"/>\n",
"\n",
"The false positives can be easily handled by having a second (exact) stage check all returned items - for rare items this can be very effective. For instance, many web browsers use a bloom filter to create a set of blocked pages (e.g. pages with viruses), since blocked web pages are only a small fraction of the whole web. A false positive can be handled here by taking anything returned by the bloom filter and checking against a web service with the full exact list. (See this [bloom filter tutorial](https://llimllib.github.io/bloomfilter-tutorial/) for more details)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expensive Errors"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*The below examples are from Greenbaum & Chartier.*\n",
"\n",
"European Space Agency spent 10 years and $7 billion on the Ariane 5 Rocket.\n",
"\n",
"What can happen when you try to fit a 64 bit number into a 16 bit space (integer overflow):"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAgICAgICAgICAgICAgICAgICAgICAkICAYHBwcI\nCAgHChALBwgaCQcHGCEYGhERHx8fBwsgIBYeIBwSEx4BBQUFCAcIDAkJDRIMDQwSEhISEhISEhIS\nEhISEhISEhISEhISEhISEhISEhISEhISEhISHhISEhISEhISEhISEv/AABEIAWgB4AMBIgACEQED\nEQH/xAAdAAACAwEBAQEBAAAAAAAAAAACAwABBAUGCAcJ/8QAUhAAAQMCBAMEBQgGBwUFCQEAAQAC\nEQMhBBIxQQVRYQZxgZETIqGx8AcyVcHR1OHxCBRCUqSlFRcYI2KS0zNERXKTJTQ1Q3UWU1RjZISU\nldIk/8QAGAEBAQEBAQAAAAAAAAAAAAAAAAECAwT/xAAiEQEBAQEAAgIDAQEBAQAAAAAAARECEiED\nMRNBUWFxgSL/2gAMAwEAAhEDEQA/APjJRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRR\nRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFE\nEUUUQRRRRBFFFEEUUUQfo7PkP7UOcWjhnrC8fr3Dhprribo/6iu1Vv8AsrXT/wD28O+8r7ZwbAQH\nOBz5b5gJbOostTGDujbkvV+GOfk+Hx8gnaz6J/juG/elY+QPtb9E/wAdw370vueg2ei1Uqcn2Kfh\njWvhFv6P3a4/8I/j+GfekbP0ee2B04P/ADDhf3tfetNsbdFtpMBIssfjhr4AH6OnbL6G/mHCvvaY\nz9G3to4SODW/9R4T9eMX9CaLCbbLTRpQRayv44a/ngP0ae230L/MuE/fEQ/Rm7b/AEJ/MuEffF/R\nqk0zcRy7k1gv9a42Y0/nEf0Yu3Gn9CfzLg/1YxF/Zg7c/QfT/wAT4P8AfV/SCgeVu9aGGPKVGsfz\nY/sv9uvoP+Z8H++qv7L/AG6if6D/AJnwf76v6VCoLoqZt0RMfzUH6L3br6D/AJnwf76r/su9uvoP\n+Z8H++r+lxVBylpj+aX9l3t39B/zPg/31Cf0YO3X0H/MuD/fV/S3NdC56pj+aR/Rj7cD/gf8y4R9\n8VH9GXtwP+CfzLhH1Yxf0me5Ie625jTkrF8X83Xfo09thrwX+ZcJ++IP7N3bT6G/mPCff+tr+jFZ\n0iIjms1VxNo0W/CHi/ne79G/toP+DD/9lwn74qP6OPbP6G/mPCtv/u1/QlzjrB+ClVH3nTVbnxxm\nv58H9Hbtj9D/AMw4X97VH9Hjtj9EfzDhf3tf0Df7ktzGq/ijfi/n+f0ee2H0R/H8L+9Kh+j12wP/\nAAj+P4X96X3+T7ELWztron4oTl8Cj9Hftj9D/wAw4X97Vn9HTtl9D/zDhX3tf0DpNAsNXWHerey1\n/NZvxyL4R/Ps/o6dson+hrf+ocL+9qh+jr2x+h/5hwv72v6CMBM6/OVZNL5T77lZ8IzeX8/P7O/b\nH6H/AJhwv72hd+j12wGvCNLf+IcL+9L+gT2Eam2v1bJD9b3v7IXSfFF8XwJ/Z57YTH9EfzDhf3tT\n+zz2wt/2Rrp/2hwv70vvZl/HW231KvS7ck/FDxfBP9nztfp/RH8fwz70qP6Pva/6I/j+Gfel97vd\nqRdJfon4oni+DXfo/drhrwn+P4Z96Q/1A9rfon+P4Z96X3ZUcDE6z7Ekkyb2JT8UMfDDvkE7WDXh\nP8fw370l/wBRPar6K/juG/eV9zucb/urM+esdwUnxxl8Q/1E9qvov+O4b95Qn5De1P0X/HcO+8r7\nYrOkQdNkh17b7LV+KQfFh+Q/tR9GfxvDvvKE/In2n+jP43h/3lfZxfHfoe9Iq2N99FJ8UHxv/Up2\nm+jf43h/3lUPkV7TfRun/wBZw/7wvsN52AS33m/gp+KI+Pv6mu0v0b/GYD7wh/qd7R/R38XgPvC+\nvq/LlokuGsnvU/HDXyJ/VD2i+j/4vA/66r+qPtD9H/xeC/119ZmIPek1dN/Ba/FFfKB+SftAP9w/\nisF/roD8lfHh/uH8Tg/9dfVJHxus9RxU/FB8uO+S7jo1wP8AE4P/AFks/Jpxsf7if/yML9VZfUFf\nnHislSTf2yl+OD5nd8nPGhc4J3/Ww3+qln5P+MD/AHJ3/VofVUX0pUnbN5pDp0k+e6z+MfNx7C8W\n/wDg3f8AUo/6iF3Yjio1wjv+pR//ALX0RVnmfNIcDuT5qeC4+i8IyS4nd7p74WgM6JWHacz4gtzC\nO/LdaxTO1/FevXllUxmi10dQhYzTn5p9KldY6uukp1Bstk87LdTZEWlKw1DbkZ6LfRpkfgsxYNgN\nrLVk306KmMtfZNAJAcfCdVi9KOm0jx9iaWHnYXgeSWwApjWDa1oPfMrlWoJp3P4hNpg7adVGUzFr\nqgXDqstrc6LbSownXW+nJW6nZCGQY6abIezWm+u/1K3G5hLLJ3vqraZHK6EMBPRKqH1fco4xEJTw\nT4JFA8lKqg622TXbCdUqoLHv0C3IuslV0Tfn71neCeS1VGgg2OqQ/wBXpaFoZnNMRB87eKQ9hkbT\npF4T67p0CTiRb3de/mtzpislUGbKg345poAtOvsRFnLXpaAr5NB9FZEykf2SiE6m+n4oCTtzlL22\ndJnu5c+itzHjUkA6QOSS07g669Fbau0yeunVZvSznTGhw5GTZA8Dl+SJ1SI5TIKRUcTGwH5pEsG9\nvIrNUBlObfSY/BJqDS569y1rLI8kEgaHVSbDqreAbwRtyS4Me7orKKLhoNSUovIm/wCShIBGp6oK\njh175v3JahTxJ1sCDbvQVT15+9XUfy96Q65vskqWrc+AZus9R5EHojeTokv/AAVZKe/2pbzAJgdT\nv4InHRKde/groS8mxFvs5pdRx+OSZUZ1j60stnUdEC3tJ03SjTMfinkeXLZZ3gSbKJS6oEWv36pM\n+1Pe2yQR0hSIz1fFJdZPq9SkvWljLUGvl9azvJ0Wmo3n4pL+ntWVZag5rJWp6i+kra88/wAlnrMv\nqrRiqjvSaoun1R5C3XvWd9vdBWRlqCPNZnla64sstQQSmLK+mKbR6oEC3K+kT3p1Fulvjql4Sk4T\nJmdOYW7D0CdPGdVrrqPIOhTB21+LLZTpRFp2Q4ejsfgLdSYQbBc7WuRMp/vDyWqjSjuQlPolZtdI\nJlOU1tMxeY77JlDyTxSsdYJWNWECkLTbqEdOmLkRBTMvId07Ii02iLahZaAwkG0pu55BLGtz5WVh\nonUwi6IE3kRGl1QfJkBBUbYoc0W840RRCdvbeyqoqJMeKF5I5ePxqihe6LRc6fggBO8hU8k6n7fB\nDczr4qwR5S80zrzVOn8UhwvO3etxIXXqnqPj3JNWTF5PsTXtmZ35qhTtb43RrYQ5tiN5VGjNjNlt\nY0G5VVaon1QDa6eSubUZASQ+8xYWiVrxzyNWwCuW98Sdvi6nk3hrqsm9gLQqdVnS34LBUqHz5IRW\nI6d6s6i5rc5/f4WUa4mO7RYTW/xJjXbzJAidlG76je11vqQVH6cllZUPO6jqvx15rTlWh9WNNR5J\ndSuTp8DVJc/Qb7lLL41+N1WR1DKU7TdC6tFvbKTUqC9ygt7+WoQ1XmPylU8zOk8wkF/cVUQu84SK\njeuuoTajeYHO2sLO8xzKrFUec30Wd7jomvFrd6UVqBYMEW6Jbjp8bonNO53lLImAO6fyVAudPhp+\nPNLcT+SMj2FJcdZWoBcZkT9qXU8EbhZZ3W9qylR5gbykvMpjzBSnn46KRCX8/Kb2SahHNOPn37dE\nl4I2F1vmhFaVnqbp9Qfks1Qws1dKeFlrR5BaHFZ6rd+atVles+IbF99+5aqjfiUl+3VYGGqkVBut\nlQSYWWppGv1JSPqmlRtcLXSpAR8eapogSdk1iy8x9KlF9eXcntAS2d/VMaDYxbT61Mbkw1nLZOps\nQMF4Gy10h4LKjoifBamVNvJJpiO9NidNlzrQkDu+Dv16I2+1A436hFAmMEnRWATqiAhAFRsfWlZe\n/qE/qUpz9xEct0aDB6/gge1EXxpv+yUsVOnX4KKB406aISw5Y5o84B5cjyVVHk9VYpD9B01WY3m2\nu61VHWSWnVaZKFP3bom07W0VwnU6U2JUtXACkIidUdPDR+wTJseiaaUjYRz0S8PxXC7VmToRm3Cz\nW5A4jBF4h0CNF5nieGNMmbTp1ScV2mrekPreq1+gGt0XFcd6d2c+rAsO8LM9ukuORVqb9I5bpNOt\nrI9v2p+KYADdc6sQNVvwZ8pG+m8DkURxTQSOa4r6hHcoKpN4W+ecZveu2MU3VQ4kSuGzEcgjGIPP\nRakZ12P1kKqlaVyXVtyVPTRBGm6LrpFw5ylF0wkmuHQRorkeSJ5Cc481QfEpebb2KMf4IyIH4+NE\nB7kJd5JbqnkqKc/8kl0o3QlOdcX2VlAudzSs/LY/V7UVWpzSS4HvGncqyokhJcic/fZLJVUNTbVL\nnx1TCRbkNUioTJ5bIKdyS536K/eqeb+CIQ7dLfoFbjZJefeqFVHbJBCeW6pNQRpqiUtwhIe2dh4r\nRFo3S3Ka1GR7Gjbe/ks1UaX30WurN40WaobdUtGSoDJ8llqtieZWyoTB71lrqfofWAafzRtBmTdN\np0xrPfKY1m9ljXHnkbGzEJ7Bsgo31stI2hTWotjPsWigw+G6BkaEx3LS3WLxCxauHNCNqodytvJZ\naR56oQ6NkwgIQEEAVE3mEXsUbCkaUDF0tzeQsjkdUOvRVcZ67CIIAS3ag3jktfulJqmfNE0qoCRd\no6DolE2vaE17xmtyQlu096rTO4bFKe0jS60VAgbGk3TQovHK6bTdG5vYoH2uhc7NI6H3J9o8l8oH\nHzSNOnRqOaQ2Kkb96/N28ZeysHxebjmvQ9uOFPo1nPkuY4znN41suHwbhratQ5xmbElPFryb8PxX\n0hzfta2tqtf9JGPWfcLiYuiKbyGi0x4DRZX1tV0nEYtegfxM85CA41pF7nmuC2vG/mocR3LeRHYd\njBz81nOI0PnC5j8QD8X8FRryJnzS8jrnESNYU/WBGq5Irz0TDWkWOmvJDXU/W46j61oZWBHzvjl3\nLhvrRHfCKjiInlNx0/NDXWFfJJFxy5dQtVOs1zZGbrdckVfHfwS6eKyEQTDrEbApIrtmpurFTcLI\nHyL6j2o6b9tCVMGguQVHJZPjt9at52zRv+CCFxHX3pTjvp13VuIHM9Up/wASqI907JThvItaNwm6\nckp8HX2LTJbjYzISyB7ET/GEs76oAfCB1j0RIHiUA1Cl1DZMiUt4SBDuaS4SnvCU4bc1oJc0+aSQ\neQWipa06JTiAs2pShaZWd6bVekPBuVPtqEPHks1R21vBPqGVnduoM9S9llqNMrU9tz3LNWPxsiPq\ntte15up6fbSAsNCq43W+hQLiJG11K48io4hx3vomsxbx5rVQwgEIxhddNVn02LC4jN0ldGjWMwuZ\nTokarXh325rKz7dpkKySk4OoXCeSeFzranSfJRqsqKKtyp0fkrKoBUUR4KZUGYi2qE1SqJUgJFVw\n23uUZfrKzOqydERTxN48kDXRKt5KQ43ke1DTnGxSXkD8Ep1W8JbnjWCtYaKrVsliv6piNPeYVVKg\nNuiy1HAfHRXFcbtpiabaBa4Zi6zB11uvA8JcTXYGkgEwRzBXsO2+Z1FoYyb3PL7F4fBYg06rdocJ\nKW4ldLi2GknlJ9i4VanuOq9bjmy0EEEOkg964FWhdv8AikLcvpXIIPelPGq3PpxMdyQ+jfUaT4qy\nssubnspTI3mFC2/sQVFb0sMa/wAPFFTfrfVLAnT3BDedN1NRqDrwjzX02j67rPGlt9lowzoBsCQZ\ng62MFNbkacEdZ2Em99YhC62ZvO7eaGm4tc12UZb5o5HvRvf6zSOceHLvTRq4e9zmHcsMXOo+tac8\nO3uuXZtQOEi+k2vzWh1aNjGbyCsSuiKk/WhDtUllWCDaHe9R7yCD593NEPdzBug+LqZ+SF5lWIko\nZF9JlWZSnmPj3qiO8bpb0RQlAsoXKyUB2RFPPJKeEZSnPCFpZSqhHjKJz1ne+471SKfqkG/4Jj3X\nsk1HctVmhTzsEio6+qaXQkPI5KGlvO3NZ6oRVCkPcqpdUrNXTKjlnquKU19W4XDe/wBi6GFpwSb6\n27lno1csXWzDVhGqxXHlups0+NkynTmddbJLH6XWyg/SDdYdIW7CjzTqGFIjSE6k2+m1+i0sELNr\nUVRp5bDRNaVRIS31FhTELnDcwsOJqnnCQCT3c1odP0ojW6F+JA6LnsOXdSQdborWytmnL9nvUzCN\nYIOvVZxFjEqOqWi3kgbWvJGu/RZag9qs1Iknfw9iW582F0Slk+xC6fzROjldZ3uMrSJWdusbz006\n6pzpNys9Vp0MStQKcOXW2u3NJeeenjyTqkx3ckioSdSYHNaajHj6jGU3F8FsG06r884lVbmJAaBc\nxvK6/bXGPbVAmGtEj3LxdcuqOJEnzXO9/osev4FixWZk/wDdWVV2AQY0d9aDs/w5+HYxxcf74A5Y\niPsWvEwc0iZd5XWv0SubXpDMfKOuq59VonTp+C7NZgzm9wR7lixtP5sbyo05TqcyCNO5Jq0rrS9u\n41lIqkG6upgaTOfmjbRu7bdLD9L3mPNOz+sJ0dYe5AbacAc9fBPFJskxMR7UNEg5h0MDuunYW7iP\n3mDKOo1SqzPYRpcA+r156og0WOsO7k+oAWtEXvH/ADD4KytMkjWbgK6GYunYHk4/ghZU9U31F9/c\ntNOnmaQ9wAyZgdZPJZabcpIgQ4SElDMFUmRoYkTudPctDrjvt4rnMBEXOuvXl5LW18kfuPHk8Gy1\nrNPw7zoeUeSM9Fm0JvGaP8wTWW1uDorKmCsDcd0FVVIN9FTiTpsgkFaQQS6h+PrVlxQv1QpZ5e3m\nqnyRP5JTzARA1J3KQACjc9Ie66CnG6XU+O9U915SKtZQU9Z6jkVV3ks76t+QUsRVUpNQi91VRw2K\nTVddBdQhZqpn42RVH69yyud7kaSq8WHJZahnojqnVZ33VI+sqVYbkeSdSfcQVhzx4rXhzpaVhynL\nr0KkRK34eoFxw/QAx7loo1TdZ6jcd70kDQo2VgVzaFfSb96dSqjW3mudajYTcH2fYh9OCY8ikio3\naeisg67e1FwNZpcQCdELhEtjU6pnpR5bHXw6JOIqxt1QKe/KkHEg9EvFVLjlv9iyVB3lXDHTpV+X\nrIi52unvWClm31RmuSbnRMMac5J57SVTqgWSpiI6pRrbpi43mr6tkio/uWZtUmUh9SFZDGt9QAEz\nrsEqu+3MTqsr624JCW+vY3t71pLBvqXOq5/F8W+nTLmesR7LarQ+vItZZa7pBBEg2uqSvy7jeOqV\nXmbmbg9673Y2jQexzz61UatJ2XL7UYH0dUkHU37pW3sJGesG7jXpy7lwk/8Apq/T01VskHouZXua\ngi0D2Ls1gYHOdVzvRyXf8seMmV2tYc3Mc5MaQUBaTMjY+9an04zSOnkbIXgB4gasM/HNRqVxjRbM\nExAzD3QufWokECNS65XVLb2G512ScVTAIP7XMrn7dHIcyZt0CtjyQzoYW99NoYIs4uiO9ZcJTdBa\n4Qc5IPPkrOkw2k7K8SmB5a8f4TfuddAaZIzg62vsd1eIF37WHmBMremNIJBcJn9oLHiIY7ONdVqZ\nSd88QYDSTyBsbbocSxsAiPVdlcdTfQwluGaZnkZv+UgJOJbuOfKRB2R0wRIBHRpH7O0KFsw0Wm99\nsuquoN7AG/OaQ8elAaZhzPVynkYCAG9vmvHpGj/FoR3oJAzSPmuDidodyCItsWi7mHOw8wRok9oY\n5oc251kjaHjTuWzgeNZSe2pUotrjK8GnVMMLnMyjT4ssFB2o1Dhmb3j5w80TT6xG5Ae20afOgbJc\n/bU1oefnEazI80p03sIiyIdwQVHcvYunN9MdX2suj7Ep56FCXwJSK1WVqVk4myzVKsdyB9UrPUfJ\nWUFWqefJA6prISqlRJfUTQdSoPD61lq1JVVHSNVnJ6poZUqrK5+3JW8pLz1SUxHPWd71HO/JJqOT\nyTEqPlIqvVvclPciheUio/ZE9yS42Upj61osMSfxWrCtKy0n36rbSeIupWTwy3tTWPj7NkplXxHL\n61ootzDMYtsPxUakMZUn3ptKpGqgpjWItZE9ukLnVaGu1WqjUae8LC1h2UcHzv4KK01nNJtr7fyS\nK3VDkPU9d0BpE2lCfbPVbJht1baDhc+S20qIbEaonj4KrVYnM5hA9ka2Wuo4d/ck1b6jzSEYKl9j\nayU8xAuFoqU728b/AGLNWZIkzCqpUI2KQQbz4ImvAEbe1LfVzWK2gTH2DdBUGvJG0gGNeZ+xLqQC\nBeOqM6S7vSzP2J1SAlVDYnYb7KXrCPCdtix1SWu3hwOnhCrsPScKzzbIG3IRdpAPSGACHG/RN7J0\n8tX9oDLYbFcp76bv09UXRrfcc1ka351lrzXnokmp6pC645sZZAMnUzz1SK4AM8hput7cvqicrDDX\nu1j1rmyOvh6OaoA/OQQKRykZxEEidAs7+nSc/twKlM5mi17/AJpNfDmSHDeI6dF0atOH7erM/hzQ\nVGn1dzMk9NolZbYMZhqP6u5/pD6RroFPITLA0lxzDeViwjMzMzheARF9dO8LsYkNHpG+uGFrhfLm\nuNossGGLWk04PqBoHdE7brn9VfKWSMbqVgI0JBPfe3mmOpwSCNGgR36LeMLmJBgT82XDnqZV1cjR\nVBbmqEtGbZrWiSR/ildNTxc6kYa6JJc0t6t7oT8eBUYCGMZNJvzNT6Pdw3KHDNklskAzd2xOkxom\nsw803P8AVhvqkgwSCYIbN5Ut1cxhaHEWmWASBErQ4eqfrkcpv4oTR9G+Hf3YNPOCTdw5QN10qhFa\nnRZTYWGlRd6QveGl5BJzBp1Fx5K3o4n3WbHcGfQa1znU3sdDC+m8OALmCo1hH70ELIwH+7eASRmp\nmNTsO4x7l2K3D3jC0XkS1zS9j6IJyuY4l7a0/t5fZC1disTSoYsem9GKdem5pe8A5Q9shw/dcuU+\nXJa69fFNjz2EwVR/pQ3IPQevd+U5ZGbLPf7F6bs5hqoq18NhnYOr6XDgvqVSMsOHrCkX3a/Vcrj1\nGizF1G0XmrRzhnpIn1XQTduo18lq4jUwFD0jKA/W2VKAa2q+WPoVJvlDNRCnXVv1+1nMjg17EiQY\nLgO4OgGd0mqY+xdDheMo0vTenw4xGei5lPM8g06hMteI2XIdU5T4mV6uerufx5u5M3+hqVEiq7fq\npUqc0h9RdNcll/1pLn2VOqpD3pqLqvnRKqO6KVKnSEhzupWVxHuskOcre8T7Emo+9u9UwL6lkDn+\nxA959qU52qm4VHuSHOujcbmEs81EA5yW53O6MoHmFrVwk3nml5iAmPbugPTZQfXNDuWqmBFwsNM2\nWqm9Kw1UWRvC2Ug2QOaxU38+SfQdrA0vKjTe11u4omt3Wak6T8arQ1y51WhtO9kxrdBzS6Rk38E5\nl/P6lFGaYIj2qhRgbHlKtolMpOGkaI0zubAvCSQStxbImIukVR3IVhcyOncsznk7krbWB5LI6nGy\nsZJfbkCszn6j3LW+m7u6fGiQ5utrqtMlRvIpDzYxFtVsc0biFndIOjbLTOslSpeIlUSNNtU19KDZ\nLcyIVQuoV5ntTxwUQ6m0w46x7l6So1eF7dUgKkkXiZ59Fz+T6a5+3Ew+IfVqBvzpI+Lr33D8D6MA\n2zR9V1+ccEw9apWaKZiHAz0lfqFOQGzqGmT4LHx+17qqhj7EFuWys03EEgfN2QAkGRbKNDou7BTZ\n8Npteb6IDykzNr2AlXiOp3n1dAeSqqw26i0X8+SzY3Ec1mSdXAls8jrKRVd6QtkHMcobHSw9y34G\nowCDSNTNd7wYOWZIA5whx2GBc6rSHoqTcrm+kqDMQXbABco6eP7JwXDa2IeabGgEAkmpDQ0cjO65\nFKlFepECAJ/dkZhae5dijUdJy5iXk5w506uEZlkpeq+sHgFnpWgiBIyiWj2lRbJhdDDseHekDjAl\nuT2XKR+pPysqBoLC4gS4TLbmwM6BasdT9aWnKyA8R6oHNp5mUp2Im7oED1QGwXTqCO6VZKep6OpY\nN9alXrFjWUqRaDlj1nEeq0jXWFzajWuacurYMHctHrADdPp1wxlSn6MFzwRnD3SJMtOWYJgHzS6d\nCcrMzQZAYf2Q55vnesTferOpZP3XOdLqjPnOdIDbCejb2AXSqY01C52JDajyW5spyEAerksLaBMx\nvBzT/WZl/wCrmmw1qTgaIL4kzM7rDji4uEmk+GBoNP5paP3ifnFa56nVxq85P+rrY4jM2mX06RJc\nKYeXNacuV0ncrCwj1ffMm5mL6hdjhfFqdKk5lbC0sT6sUs393khxcdPnIWegxtZ16eCcfSVg53+z\nyxDcO1o/aOvgpJeb9emc859q4SytiXjCUn02DEEMPpPmerLhmIHqjXTkldouE/qcB2IpVKmZzXUq\nU+pl0vvK5tb0lMw6WHURLd7ETcaLJVrSS7e5J+cZPetzm75b9pOs58bKp9RIc/ZVUclPK9Dj0lR6\nz1HK3EpTwozFVOiS9XUOqXmQ5U8pTzf4hW96BxRSXnpHNAfqRuNo5pLrKoU4mfcgTHtUyjXZZpSC\nEJEJrmwluGiIU9qAjbomPCFxVUg9UuodSmuSaghXR9W0n2aFppVBzXLpVLLTh6lxdRwjr03z+Kex\n50gQLrmtdJB30+xPZUvBt1R1jp0qgOtlppPEa3XNbUBEeS0YYiyxWo6THea0UXHdYqL1oYdLLCxu\na6SBp47Jrmzppyjfms1N3insdKLiOok3IBSKjSIkSTotWaL+GntQPbbrNu5GmOq2PBIyW6rZVA6+\nKyuF9IhAh3TVZ6zJ5z8WT8Q4ja2/NZC8+3xRNA9h0ss1VpCfVudD1lZqxjRbiMztDIhKEJ1U+33p\nNQjcqlZ8Q6A5xk5QTC/N+1GLdUcJJzGY6DkvcdqsQW0Ja4NnnqQvzHG1QSTM6i64/N1+muY73Yem\ncz3R8wXK9eCYlcDsDRb6B75nM4bctQvQmjrddPj+mL9+iiSBHraweUJbzaJ+1Ma+AQTAdYjmmZBp\nFtgVtGR7bCDB6oTTvYmN/JaXUvj7FQpkkjciBFt1LFJw82IcA4gt9b61TAHTmaDsHToeic9kHL5j\n8Uv0c30uo1OsUajiwwwZmH1njl/i5lZano2H+7BOZvr+luCTu3kFqqjK4ta4lsattPTqsFQ6n1r6\njbks+K3puwHBnVGF5xGGosYDHpX3vfK1ouSuXVdD/nEwYzatIGpE3AVOMQLX5/YU2tSgF4Z6Rgpk\nkzES4Akc7qSWX7b8tjLULCCSAHAnQkEj9m26Q6qIiSDEE7E8k7E4osYWS1wqgT6slo2aHbLIMY/0\nXoi2mBmzSG+sfFaZvtqwjazmYkUafpmBgNcNa45GtcC2IsHylcU4bUwzKdR5pllZmdr2OzNP+G+j\nvsTeHcS/VWF9OtUY+q4MfSpmA5gd6wfY2WYvq4irWqspF4IqVi0kZQAMrnNzQNAPJcc6nTvfG8/2\nhwXomVJxNN1SmaZORjw18keo8kdVXCcGzE+lb6T0dRjHPpgtL2lrbuBcNHWXK9M0G8kWMCRMtuiG\nNezMKTnU2v2Bg5IgtJ/dXW89X6rjOpL7M4jxCriKnpKr31HhoYHOn5jQA3XT8Via8AknTcIHzMyU\nD3wt885HO9b9rM6dY6JM6xstFXFg0m0vRsGV5f6QD+8IOgJWOo5JUqn3uNUp5I1nx0RGI6pVRaZo\nSlPRu70p7rSihdM6ygqH2oyEOX42QKLPtS3ck8goCzzQKa0jVDCY4wkucs0oXG/clONoTCSB3pSI\nWgeUxyzvN+iIF5KRUKc53lz5pFR3hf2KkfTdKoIHTVaKbrhcqnV0OxWunVv8QtOcjqsq8imseRqZ\nlc9lVNbWM2i9lHSOnTrRy+xa6FWwM2XJpx1lPpviLws1qOzSrkn61sbVO5XCp4rKrqcQy6LGLHpK\neIHNMdii2xjwK8sziB11sm4PGZnXGvMq4uvU0sW06+WyJtQm4INog/Gq4dGsT0Gi0sqgaOus9RrX\nUa05ZJvySagPwVkdjIGpASK+KJBFhdZkS1oq1gOp0gLJvob3Wc12MPU7zZaG1RGxVRnqucgqu2hP\nriSCICU+8rUoyuE+Cy1hZ1pgE381qcRyHKFy+NcQbSpvJIBLYDbfEK300/POPcZe8va8mA4gDYdA\nvJYmoS6GyS4iANdbBdHizXOc9+gJJhaexGD9JimOdENk33IvHevLL5dNdeo/Qez+G9FhqLC3Kcsu\n8b+a2RrdJNQeJvf2aIm1CNrL2SenLRPpSNghLT0smB5KWdeXRTQYcDqgfGwuLyPJCe+Ty+1GwGE0\nVWdYCL7lKsIB2TnG07rO5wBkp9qTWEnRZ679uqa6oFmq1GA7nu/FEIqDW+/j11Saj/W5iIyknKQd\nZG6Ks761jeRMnZLNINgpvHrVsgvAiXSNB3LJVdLB6xLgYPq2A2PUqqriZ6keASXG31bDuRot7okE\nxoRG9wTdDjcQwvPos9OkRDWucSbgZm22mfNLqO2G+2yUZ35z4rUiahJ/dMCL6gDTwCFrS4gNvYzb\nYFF+sPax7A4hr4zDnBmehSS8j1gYMRa0+Sns2Aqnvn6kqoVHG3xogqOstJgC+58PYhUcVHnRNTAO\nshcZ/BUXEoXk/Wka0t7Z0PmhnaPJG8jcIHdyIoAKnO0AV6JRsfBESo6DslZjJKJzvd9aW5ZtA1Cl\nlyOqdkgn8FEVUd+SVsfjyROdulOcqKft7Ut5GnX2IX1EGbc8lTFF1h3pFZ47rRdG51vrWeu6RooP\noajXad5Gy1U64mN/YvH0scToLrdQrPmZ6Qt6zj19Gv3HuWlj1wsJV0W6jXBKVOXXD48URJ2Kw0ao\nmNlspkac1itylurOCsPcd040Z6o6VGPsWdUVIEBaaDY9afBU1rSRpbWVobh5i8CduUaKtSCD3EWM\nJrXG9yOt1ZaAbZuiMuGkmVnpaFz7TOugWIuPziSdikcVxLKYGYxyXmeJdo2wW0tZ+Cskj09TFspm\n7p9vvS3doqTbFwkneNIX51iuIPeTLzM3HVINSSJmOeq1hj9NbxxjzDXtJGg5rdgcb6TX2EL8ww+a\n2XbQru8DxTqZgyRumJHtcXWDWOcS0Q1xkCNB1X4zxfidRz3FznO9Z0E6ROy7XHuPVK1R7DLabXOa\nI3XDdh2Vh6rXnnB0815/k6125kcXE4/qNfgL1vye4Jr82Iky2WluwJvK8xh+EB1YMqF7KZd86Nph\nfrHB+F08LSbSp/NsZjUkdFr4udY7q8mhjwVPmNbJ1WxS517l6Ncl4c2Vki/xCW/TdZcRWLT3jRRW\nwER71VTEMb0EbrE2qS3Q96y40EgCTHVFaqvEGDT3rFV4iSbLEaQEb+C0UsM06hMFtqHXVCRJmE4s\nGmw0QlsCVqMkuCRWpgplSoEsvF1RirNj7FkrVJ2jqttdqw1aZPco1GdsRb8UDimuYeXigIWolIIQ\ngBNefsSaiUwt5hILvyTnjRJKzFC4wl1jumEoKhvdVATZA42vr9SupzQEhVA1HBAXclTz8dEsu89k\nFud1lA/6lRdCUXLOiybJedBUfslOeoT2Iv5pbnadxQF6VVPirDBmpPck1HaKidksm/JUxbnJT5nV\nFUcNUkn2+5NFVHeSS/nmVPfCW9/VQx+w4djY1v0Wuk+PxWN78tykVsf0BK0z6d2hiIOoXRw9bQyv\nEtx7wtWH4k8C5Pcl6PT3OGrj97fbXwXSo4sDffdfn9DiDjED2wurg8aS4SFNHr3cTAjc6StNCu9+\n9lwsG9hGi6YrtYBchZ8VvTr0mDmmPqlgytPSfauVS4g217afimUsay/rDXzQnbSceR6riUjGccFJ\nrgLktS61RpEAgH2lYMZgg67ip461OnmOPcRqVXxLv+XZcgB0n1j4r01XgpLhqRqY5JGK4Q+ZDTB5\nqY1OnCc5NwoJ5rps4QReCtVLAOkWt3b9VU1lwrnW5aLuUcS0AA6peH4cSBIJttz28UY4e8GS0jkU\no8d2spmnWcQSWkErif0nUZEAwb+xe84xw5r2OLyRkY6bexfneLGcwCeQ2kdV5u+crpKc3j1R7xmu\nBGvKV+v8CxdOphqbmGWkAeIEFfjHDOC1azjkDnQQHBouATqF+08G4ezDYalQYcwYBJPzsxEmVv4v\n8Z6NfU6JDnp9YdfBc7EErs5ml5vCQReT3JRrPGk96pz385HJUaSZEAxCU9vO5SQ683RGrAmfBFXk\nbr4InN6pLqnqxzuqNTT3oVocbpL3aoHP6oc+2quIVUCyvZmkbdFrf1SXWPxooMtSnobwUNRo/O6O\npF+miBz+9FjO9p5WSKjU+q74Cy1jC0FPSKgt8c06o5JJVCnj3/UszjC0PWWtpZZiqNTpBSnut4qz\nKU5qWso6JSn396j3IVNAFLc9E4FKV2hVQlAXQmP1S3i/gsxIS93vS3lMekPbJnZaaxHmySDrKY5p\njn0QPHgqFjdA5MCWUQqoY2SJTamspL3lRAOWesPanvduk1J9bflOyRX7C9oJ1Kp2EYdwJWJ+IMdU\nsF1jEq1z9N1bDs2IMboKbG6QT1SSXnaB8ckym0jn9Szi61sIAgfHituGfl1WSky0mPrWbE4o6clb\n6V2KnFclmn2qN40Tq620ry4eXaCSmDD1TtCm1qe3phxPeVY4g4mWuj6/Ncihg6nL48Vpp4Codvbu\ns+y5HbwvF3AwIJjddXC4yR6+a/kvNUcG9h0M7rcapywZlbhuvV0scGiwvC1UHB4BcF4J9etaCY0l\ndbh2LrgDNpCmD2lEUtI9i2MoUtgPBedwGJcYJA81024jS47lmxXXw9CmCDAA3TMTgaVTSD3rk/rB\nGpkdCjHEWt3AhMVze3tXDYXBVBU/2lZuRmQiQZi87L8OxNYAktudvcv3fjWCwGPaP1yl6SxDXSQQ\ndQR1X4lxzhVKnXqNomoKQeWsbUMvDQdzvuuPetw3s3xipha3pW2P7Q2cOvNfrHZPjbMYyq40y19L\nLmj5pnvvK/H6lJgBFpMAX5XX6l8lOCb+ouOcGoXy8DYAeqDKzxWrHosTSBMjQ8/jRYK1KbQuzUw9\ntDqOXsWTGU40IXZycKowxHvSSwgGAulXjcLBjGB1pLea3ozPqaXlA5yJ1MAQFmFN53ViQ0HmilCG\nGLnwKIUfG8KWpUzHnoqnRNbhndw70L6UQJGi1qFPSnFMqpFQ+PP4Kis7zqkVahTH3NtFmrmbIsKq\nuPckVHSjrk7rM47puNI8pTn3Ctzo96RVfJU8kVUfv1jwSPSK3lLOi1qF138kg5vD2+KZUdHekmr1\n71ki9LlC4/G6VUqckl7xurqmVHxuge6Uqo4aIc6aGOJS3E66qpHPvS3PhSM4t5CCRoUmrWSjVVaa\nHnXuSXlUahMJT6isi6soXvGiU55idEp7lUE93dZIqv5R1VvJWZxnXT45Ii31PYkvfKqoRtslOcFF\nfq1Oo0SSfanUsUzmF5unWE9/NdDDVhAnKFrzcPCR26WLYeSp3EG6Na5x25SuX6Vk8+7609uOA9UN\nF91J1pI6mHLnXn1h7E12Gb+0665dDE5raRumVg6JLxCarqYbB0xBHiui0Mju1K81h8aAYkuXRpVp\nFrdTurKjq18bTpg8tZ1/JZH8aaPmG/gudVoPqTdsRsVzcVg6rf2fV5j4us9X+NzmPQ0OM1HHUW16\njonVOOGwDR3ryDKz2bSOadTqO1IsseVXMe2wPEWvEECd/wAE48VYDlAsB7V5TCvj5wIPVdTDYmmx\nvpCP+ULULXpcPjC9u4OpSW8VsQDJFgvM4zjkyGWvdZ28UiPi6WmvV1uL1dBa3kkNx9YEF89y5LOI\n1HQWAIcZUxFQh5IETACzqya9SHV8TAmAId864HgvL9oOGsmpVZXa91N3rsNtYi+6v0tZjBNQAusR\n03WGrxCmwPE3gAHLNwZ8dVxvW+nSRw61MzmiYOk6L2PyZcafhqzgSPR1WHMDMAg2K8XX40KjyxzZ\ncNCBAudOq/RODUMP+rtgsDwyTNrmySe1tx6B3aB7qm7hsAfVi90dXiJf850ToBqvN/qhu1j2l3RC\n3h1YAOdUPcFrXPXXHEpdlM96aagIHVcOnRdHrG+ZbqRDRC3IH1agBWPE4kyQPNBicWxupnmudXxw\nJtYDRWqaeIVGOuQRP1rfT4hHzrXmy8ljcYS+Y7uSx4jiLySZPjos7SY91W4k0AnM2B1XPp8blxGW\nbrydKo9+uaI5rs4Wg0gRc8uSsS49AcWCAUl+IJ5LM14yxyWfF4sCABtCUns2tiWibwsVXEysD3OJ\n13uEiu6pKkp9Oi+oDzSS6VlDzEHdCKnIpapr3JL952UNQG0yfwS3LUiBzSluMH2IH1iNRPXkkOrH\nuGqIa+8Trus9YCbJb6qF1VrkVb4SnDqpUcluPNDVOeOuiW98hLrk67IRboqYt1UBJ9J5KnpDyQq0\nKd5UzhZDVOhS3udoNVKmtdSqLjNdJdiG85WSox6zvadymrJrpOqgwhdfRZKdhPJMOLLbQO9XV8TX\nU3cj5JdSi7WCp/ST9pSn8QdzM7ptTxCaXSEt9M6FPbjXDZvegfiyT6zbKa1zHpqVED9qStuGpwAT\nGi/P8L8pPDXfOZiGcjlBnvutzPlI4XGuI2/8v8UzXDHuWDW8WhaWOaAIbnMarw9Dt7wtxPr1W6Xc\ny2nQpuN7d4BlImi81Khs0EZW3GqXIc869o1xJ1yjcWspVDZ+efBfnPY7tNh2Gu/GYkhzneo0Bzmw\nT0C9XT7YcKGmLphwscwdrz0Uns9z9PR4drQJvO2ZU99Q6u7vwXFo9suGkwMfSJjkfsW/D8YwNWIx\nlFx2l4aPbotWek92/TeypUGjiegRZ6p1JA5FZa3FcHTicZhgCY/2zTfuWyi+lUgtrUng6FtRuUjz\n1TP9XYJtIshxgjUnVOPFXAZWhnflSyWRDa1MRr/eMMebrrj9qsacBhX4mmadaoMoYwva7MSRJIae\nqhz7ejp497gMzGgAwocXTzAvDTFoCxNxAfhm1akA+hFWowFohxEwL6ryeE7aYdutCqCXaGDN48U1\ncr3YxlHem0J9Grh3bALFQwwqMZUDHEVG5riIkSJ5plagWhmUBs6rUY/8bMbxClQa2W5pMNhXQx9G\npAIPncLhcY4Y6vSs+HtktDzvtcafgvLMFTC4hgxtU0mBv+0Ev9JyIIs1c/ltl+vTrxZn+v0muaDo\nBcQb2m4BXkO2FXKCaNstpOpXl+M9qjTql1CsaoiBIseS8/xDtbXrEB5AGtvtXCx2kes7HPL8R/eG\nd4i0c5XtncaoU3kawL30uvzHhfFWNaajXBrosZg22HVdHFU6tXCDG1c7KecNinq6T+0SFedh1P6/\nXOFcQo1WtcCLibouI8Wy2YQSd9l4bslxMYhj/R0yynSIptn1s1pOmi7n6y9wyCmYzROX5vcusrh9\nenQw1fEEkuIid10v1oRLjfKb7HuC8zUp1s4DS8A7qVX1BkDgT36DuUnpHbq4ymCZN4GnclVqtOJm\nZHKfyXLdUklpY7vi2nvS2PGax0tBELerrVUqUDFr7g/YlVY1DWQlvretdovvCyYwveBDh5rMXR08\ndlI0IldGljCI07xvdceiwMjMGX67rfnZESPBJcMHjOJANN4KxUar6nrAyhfhWGZBKugMgswgJR0K\nR3IEqqlQHYLKJy5vYkOed1nFBjHvOllle95FzKLFVXfs5Y3vdL9O3SIG5VFNrPZePyRHFk9xQfrD\nNjOyz4qvlHqid0XW3PI96zVxrdc6pi6k6EA7bJf6zUOrZC1amNjnNA1SfS2MWukmoQfWaEFQyZHk\nNJUVRrOzEXSnYx4dG19U62u6zYmlPx0VxBjGaEj8kwV2lc51C9yUbWAaErRrVUd1Sw/mlF3wUp9T\nZMDatUNSH1t90D6kyEh5AuUwNfXnVA+Dus9QbrNVL4VWc62WG6X6RpWL9Ye2xCH0w6A6lFmtbhyK\nU9u+aEDX5roKoO35I1pjK0amU2nXadVzyHjqlOquGyYPywflfRS/M+aHMo0rmz6MNV3M+ZRCqevm\nUkqwixoZi3iwe4DlmPkp+sO/eJJ3JJlZ4VtClKc2u7SYtCJtdw/aPeCUgjuUgKGGuqzqZ77p1HiF\nRggVKgGwa9wHsKyhSFRtbxF+73nnL3377ptDjNZhJbVeJEXcXC3R265oVQh09Ng+1+KY3KHAt3Dn\nPM9NdE6l2yeILqDXls5XSRE72XkwFZKzkZnVn09iz5Q+Kt+bjsQwQGtZLXBrRoBIsulg/lX43TFs\nWH8hUpsOncF+dCeimYrftH6i75auNZSM2FBIgO9FJHW9iuZxb5VeLYmm+nUfRh9PIS2kJ/5gSLOX\ngxNvrCKPgKE+9dap2jxbgGl5I2gAeCbV49mblyFpA1nUrhEXnl7lSOj3XYftRg8M4vxeGfWfY0ya\nuWkzb1mn5116UfKPSrTh31nswoe6pBFi4izWhv7K/IQia889t7+HcnUi/f2/buxfyrcPwFF9GpTx\nFUvrF+emGhuUnqJXdq/Lpw7MQ3CYstB2cwT7F85Enr5+5FmPNJ6c7Jr6IZ8ufDyb4bGdIcw/UmM+\nWrhryM9HGNA3LWmPIL50k81XpHaSfjqnoyPpF/yw8JIcc+IMCQz0REnvCw/10cMifQ4qeWVp8R0X\nz96R3ND6V32Jp4x9ED5ZuFer/d4rr6gtZQfLDwiwyYoTqTRB9y+eS92ioPg9eYRJzP8AX0K/5V+C\nuqMzDEmZ9c0sobGkjkumz5ReDPeKYxbQSJDiwtZ3ElfNDX79/t6KjVJ85+Oia1OI+oW9tOFEw3H0\nBGvr69eiOp214bEjH4Yc/XXy4ag5ewKi4HYeX2q7UvMj6gPbDh5H/fsOZv8A7QSl1O1fDx87G4c8\nvXt7F8yZhMwFWYKGPpX/ANrOHuOUYygTE6i3ikDtNw92mMw5B2LwL6X8l835hy9uiheOXsT2ZX0Z\nU49gAf8AvVBt9qgI6X2TKnF8LocTRmJj0rfC6+bweg6z7IV5vjXpa6pj6EPGMJJJxVG3/wA1qB3H\nsJtiqHT+9bovn22qou6dPD7U+1kr9+rcfwoucVh/F/2JFXtNgw2RiqEdHjXu5L8HzfBUHhcIuP27\n/wBr8FnLP1ijMTmkx5qP7UYMf71R6+svxE/HNTMOS15RMftY7T4Mm2Kof5lmr9q8EyxxLCel/cvx\n23L2BQu6DwU8kx+vDtZgjpiWaX19iQ3tdgnEg1wOsGF+UFyolPJbzj9dZ2gwp0xFKOroPtVDj+GJ\ny+mpTt64+tfkR+LR7lPKVfJMfsTcewj1XMN4kOB8kD8YO/u/NfkVOoRYEgdCUfpXbPf/AJinksj9\nTr4hp1t8d6zvqNjv3+CvzP0jt3P8yqdWd+87p6xTyi7X6cMUG8lBjuUGy/MRVf8Avu/zFF+s1B/5\njv8AMU8k1+j1MeRsEv8ApDp7Cvzs4l/77/8AMVbcXVH/AJjv8yu/w2kR0VJipc5TFABWoVbQiorV\nFSVBFUqxdSyCvNQeKuFC1XRYcrzIS3REWaIIorjZVTaSYAkxdSCiVSlQHcR+aIBXQIHVFKp0aKNa\noLNlWZWWoQ1BQRtIQyraqCcoVaoqLilJPJQORIgSVMxVOCtoQQOKkgqnFVTRVlSVVRQFEHBVOlVd\nU8qxKjlZ0QTzTGocgCJypRFW0jMDEgESOfRU+CSQIBNhyChCiaqFRWokRSiitKBJUVFXy70FqFUd\nVaaBIVAoxuhhBahKofWpCCAqw5VlVQqCN0LkaEhQCrVwrcgBRFCmUoGZVWVENFAFnQIaihW0SrhF\nAWqsiMqJpgWhVCMhQN7kAwoNR3ooVhv2prUgOaJqtzbK2676clC+1dyFwOs6Jnn5KvjRNYLyTeZR\nZSiI+IUCaBLQryqyPiFUXRftcIcqIHv8lRCF5wuFaYAqcNLexCQF1cFGR3+SkIUvIpCMKwmr4lqJ\nh+LKR8QmplLIUATMqgHxCGUoqwETh3qAdPYqsgYUREKDxRMA4KwEeX4hWWxspq/9Kyq4R5fj4Cpg\nM/bKqUEKJhG6gHRDAQhIRxqry9/kiEnZWUzJ0Pkpk6HyV0Kgqwm5e/yVZOhTQLhCpMy96rJ0KmgF\nE4N6exXl6exNXCGhQpjx3+SHJ8QrqBKFNy9FeUdfJNNJARBMySiDensU0JVOCa9vRU1vTZXVwuFa\nYW9EOU/BT7Cc5UznmhUWsc9FnPNT0h5oVExdH6Q8/cpnPxCBRMNH6Q8/cqznmhUTE0Wc81C880Ki\nYavMVec8yhUVNFnPMqsx5qlEF5jzUzHmqUQXmPNTMeapRAWc81WY81SiLq8x5q8x5oVENFnPMqsx\nVKIi8x5qZjzVKIurzHmpmPNUoiavMeamY81SiGizHmqzHmqUQ1eY81JVKIavMeavMeZQqICznmfN\nTOeZ81SpFEXHmVA48yhURNFnPM+amY8yhUQ0WY8ypmPMoVENFmPMqZjzKFRDRZjzKmY8yhUQ1eY8\n1Mx5qlENXmPNSTzVKILk81Mx5lUogvMeamY81SiAg88yoXnmUKiGrzHmpKpRDUUUUQRRRRBFFFEE\nUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRR\nRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEH//2Q==\n",
"text/html": [
"\n",
" <iframe\n",
" width=\"400\"\n",
" height=\"300\"\n",
" src=\"https://www.youtube.com/embed/PK_yguLapgA\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.YouTubeVideo at 0x7fb330414cc0>"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.display import YouTubeVideo\n",
"YouTubeVideo(\"PK_yguLapgA\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here is a floating point error that cost Intel $475 million:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[1994 NYTimes article about Intel Pentium Error](http://www.nytimes.com/1994/11/24/business/company-news-flaw-undermines-accuracy-of-pentium-chips.html)\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Resources**: See Lecture 13 of Trefethen & Bau and Chapter 5 of Greenbaum & Chartier for more on Floating Point Arithmetic"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Memory Use"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Sparse vs Dense"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Above we covered how *numbers* are stored, now let's talk about how *matrices* are stored. A key way to save memory (and computation) is not to store all of your matrix. Instead, just store the non-zero elements. This is called **sparse** storage, and it is well suited to sparse matrices, that is, matrices where most elements are zero.\n",
"\n",
"<img src=\"images/sparse.png\" alt=\"floating point\" style=\"width: 50%\"/>\n",
"\n",
"Here is an example of the matrix from a finite element problem, which shows up in engineering (for instance, when modeling the air-flow around a plane). In this example, the non-zero elements are black and the zero elements are white:\n",
"<img src=\"images/Finite_element_sparse_matrix.png\" alt=\"floating point\" style=\"width: 50%\"/>\n",
"[Source](https://commons.wikimedia.org/w/index.php?curid=2245335)\n",
"\n",
"There are also special types of structured matrix, such as diagonal, tri-diagonal, hessenberg, and triangular, which each display particular patterns of sparsity, which can be leveraged to reduce memory and computation.\n",
"\n",
"The opposite of a sparse matrix is a **dense** matrix, along with dense storage, which simply refers to a matrix containing mostly non-zeros, in which every element is stored explicitly. Since sparse matrices are helpful and common, numerical linear algebra focuses on maintaining sparsity through as many operations in a computation as possible."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Speed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Speed differences come from a number of areas, particularly:\n",
"- Computational complexity\n",
"- Vectorization\n",
"- Scaling to multiple cores and nodes\n",
"- Locality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Computational complexity"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are unfamiliar with computational complexity and $\\mathcal{O}$ notation, you can read about it [on Interview Cake](https://www.interviewcake.com/article/java/big-o-notation-time-and-space-complexity) and [practice on Codecademy](https://www.codecademy.com/courses/big-o/0/3). Algorithms are generally expressed in terms of computation complexity with respect to the number of rows and number of columns in the matrix. E.g. you may find an algorithm described as $\\mathcal{O(n^2m)}$."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Vectorization"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Modern CPUs and GPUs can apply an operation to multiple elements at once on a single core. For instance, take the exponent of 4 floats in a vector in a single step. This is called SIMD. You will not be explicitly writing SIMD code (which tends to require assembly language or special C \"intrinsics\"), but instead will use vectorized operations in libraries like numpy, which in turn rely on specially tuned vectorized low level linear algebra APIs (in particular, BLAS, and LAPACK)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Matrix Computation Packages: BLAS and LAPACK "
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[BLAS (Basic Linear Algebra Subprograms)](http://www.netlib.org/blas/): specification for low-level matrix and vector arithmetic operations. These are the standard building blocks for performing basic vector and matrix operations. BLAS originated as a Fortran library in 1979. Examples of BLAS libraries include: AMD Core Math Library (ACML), ATLAS, Intel Math Kernel Library (MKL), and OpenBLAS.\n",
"\n",
"[LAPACK](http://www.netlib.org/lapack/) is written in Fortran, provides routines for solving systems of linear equations, eigenvalue problems, and singular value problems. Matrix factorizations (LU, Cholesky, QR, SVD, Schur). Dense and banded matrices are handled, but not general sparse matrices. Real and complex, single and double precision.\n",
"\n",
"1970s and 1980s: EISPACK (eigenvalue routines) and LINPACK (linear equations and linear least-squares routines) libraries\n",
"\n",
"**LAPACK original goal**: make LINAPCK and EISPACK run efficiently on shared-memory vector and parallel processors and exploit cache on modern cache-based architectures (initially released in 1992). EISPACK and LINPACK ignore multi-layered memory hierarchies and spend too much time moving data around.\n",
"\n",
"LAPACK uses highly optimized block operations implementations (which much be implemented on each machine) LAPACK written so as much of the computation as possible is performed by BLAS."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Locality"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Using slower ways to access data (e.g. over the internet) can be up to a billion times slower than faster ways (e.g. from a register). But there's much less fast storage than slow storage. So once we have data in fast storage, we want to do any computation required at that time, rather than having to load it multiple times each time we need it. In addition, for most types of storage its much faster to access data items that are stored next to each other, so we should try to always use any data stored nearby that we know we'll need soon. These two issues are known as locality."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Speed of different types of memory"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Here are some *numbers everyone should know* (from the legendary [Jeff Dean](http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf)):\n",
"- L1 cache reference 0.5 ns\n",
"- L2 cache reference 7 ns\n",
"- Main memory reference/RAM 100 ns\n",
"- Send 2K bytes over 1 Gbps network 20,000 ns\n",
"- Read 1 MB sequentially from memory 250,000 ns\n",
"- Round trip within same datacenter 500,000 ns\n",
"- Disk seek 10,000,000 ns\n",
"- Read 1 MB sequentially from network 10,000,000 ns\n",
"- Read 1 MB sequentially from disk 30,000,000 ns\n",
"- Send packet CA->Netherlands->CA 150,000,000 ns\n",
"\n",
"And here is an updated, interactive [version](https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html), which includes a timeline of how these numbers have changed.\n",
"\n",
"**Key take-away**: Each successive memory type is (at least) an order of magnitude worse than the one before it. Disk seeks are **very slow**."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This video has a great example of showing several ways you could compute the blur of a photo, with various trade-offs. Don't worry about the C code that appears, just focus on the red and green moving pictures of matrix computation.\n",
"\n",
"Although the video is about a new language called Halide, it is a good illustration the issues it raises are universal. Watch minutes 1-13:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkz\nODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2Nj\nY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQED\nEQH/xAAbAAEAAwEBAQEAAAAAAAAAAAAAAwQFAgEGB//EADsQAAEDAgMECAYCAgICAwEBAAEAAgME\nERIhMQUiMlETFBVBQ1JhkSMzQmJxoSSBU7EGNMHRFpLwVLL/xAAaAQEAAgMBAAAAAAAAAAAAAAAA\nAQQCAwUG/8QAMBEBAAIBAQYFBAEDBQAAAAAAAAECAxEEEhMUUVIhMTKRoQUVIjNBcYHwIyRCYfH/\n2gAMAwEAAhEDEQA/APz9ERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQ\nEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBER\nAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQER\nEBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAR\nEQEREBERAREQEREH1lJs6hfSwufTxEmIEkvOql7M2fZ/8aHIm2+VPQYup09sfyG8lPv2fx6u8q5N\nsltZ8fl52+W+9P5T7qPZmz7N/jQ5kX3yvezNn4v+tDw+c81e37M49W8uSb+P6+H7eajiW6/LDjX7\np91DszZ9nfxocibb5TszZ9m/xocyL75V7ftJx6u5Jv2Zx6t5ck4luvyca/dPuzKvZ1Cymmc2niBE\nRIIedVL2Zs+zP40OZF98qxXYuqT8fyXclZwu6Nh39W8lM5LaebKc192Pyn3/AKM7szZ+L/rQ8PnP\nNOzKCz/40ORNt8q/v4/r4ft5rzfwyceruSjiW6/LHjX7p91HszZ9m/xocyL75XvZmz8X/Wh4fOea\nu79mcerfLyXu/j+vh+3mnEt1+TjX7p91DszZ9n/xocibb5Xy1NSGpL7Pa3DzX2+/hk49XeVfJ7KO\nEVL87NAJtyVjDe27adejs/SP9W8xedY/9cdluz+NHldedluy+NHnZWu04N7ekzJ+lO04LN3pMiPp\nC2b2Xo7/AA9m6/Kr2W69umj0unZbs/jR5XVvtOC/FJpbhHNO04N7ekzJ+kKd7L0OHs3X5VOy3ZfG\njzsoKqlNNhu9rsXLuWj2nBZu9JkR3KntGqjqejwFxwg3uFlS2Sbfk1ZqYIpM0nxb9Js6hfSwufTx\nEmIEkvOql7MoLO/jQ5E23yp6DF1Ontj+Q3kp9+z+PV3lVG2S2s+Py8ffLfen8p91HszZ9m/xocyL\n75XvZmz8X/Wh4fOeau79mcereXJe7+P6+H7eax4luvyw41+6fdQ7M2fZ38aHIm2+U7M2fZv8aHMi\n++VoBry1/Fq7kvHB7cAOPVvJTxLdfk41+6fdl1mzqFlLM5lPEHNiJBDzqsHoIst0d3evqq/F1Oo4\n/kO5L5nuH9K7s1pmJ1dXYLTas6zqj6CK/CNOadDFnujv71J36dyXydl3lWnQR9BFluju706CK/CN\nOakvw5clNLA+JrHPtZ7bhRrEeDKKzMTMfwq9BFY7o7+9Q0tKanFZ7W4eatnQ/kpsa95rX0GixyWm\ntdYbMFIvkiso+y3Z/GjyunZbsvjR52WtvWfxau5JvWZxat5KpxrulymLoyey3Xt00el172W7P40e\nV1rb2L6uH05pvWdxau5Jx7nKYujI7Ldl8aPOygqqU02G72uxcu5bu9ZnFq3ks3bN7w3vodVsx5bW\ntpLTn2fHTHNohs0mzqF9LC59PEXGIEkvOql7M2fZ/wDGhyJtvlT0GLqdPbH8hvJT79n8eruSp2vb\nWfF4u+W+9P5T7qPZlBZv8aHMi++V72Zs/F/1oeHznmr2/ZnHq3y8k38f18P281HEt1+WHGv3T7qH\nZmz7O/jQ5E23ynZmz7N/jQ5kX3yr2/hk49Xck37M49W8uScS3X5ONfun3ZlXs6hZTTOZTxAiIkEP\nOql7M2fZn8aHMi++VYrsXVJ74vku5Kbfwx8ereXJTxLaefyznLfdj8p9/wCil2Zs/F/1oeG/Gea8\n7M2fZ/8AGhyJtvlX9/H9fD9vNeb9pOPV3JRxLdfljxr90+6j2Zs+zf40OZF98r3sygxf9aHhvxnm\nru/ZnHq3lyXu/j+vh+3mnEt1+Uca/dPuodmUGF/8aHIm2+V8tTUpqS+z2tw819vv4ZOPV3l5L5TY\n98U9r92n5VjDe27adXZ+kf6uSa3nX/JRdluz+NHldOy3ZfGjzstbes/i1dyTeszi1HJTxrvScpi6\nMnst17dNHpdOy3Z/GjyutbexfVw+nNN7f4tXck41zlMXRk9luy+NHnZQVVKabDd7XYuS3d6zOLVv\nJZu2b3hvfQ6rZjy2tbSWnPs+OmObRD6OhDepwZD5LfDJU9m2fkNXeGVBQlvVIM2/Jb4hU+Jtn5t1\nd4hVG3nLxF/VJZtmZN1b4ZSzcWjeH/EeaYm2Zm3VviFMTcWreH/IeahgWbZ+Q1d4ZSzbMyGrfDKY\nm2fm3V3iFMTbMzbq3xCoPFBWgdUnyHyXeGVFJtEvpuiEbRiw5hpUtcW9Unzb8l3iFZlxZuY1H1rb\nWNY8Xofo2z4ssWtkjWYmNHeN1+I8PlPNMbrO3jqfpK4uMWo4fP6pcWdmNT9ay0el3K9Id43WbvHU\nfSV50jsXEeHynmubizcxqPrS4xajh8/qmhuV6Q6xus7fOp+kr5wOc24DiAdbHVfQ3FnZjU/WvnDq\nrOz/AMqG2xEbugiIrSgIiICIiD62jr6JlLC11RCCIgCC3QqbtGgs7+VBmTbdXxiKrOy1mddVCdgp\nM66y+z7RoLN/lQZEX3U7RoMX/agth8vqvjETla9Ufb6d0vs+0qEB38qHU23Vjn/kcht/FiyssRFl\nXZqR5+LOmxYq+fi2JtvvmifGaaIY2Ftx3Kv08dhvjuWei3VpWvpWceOmPwrC/wBNHfjGi96ePPfG\nd1nosmxf6aPLfHcu62saWQiNwfZtj6LNRTE6RMGqx1t/lCUtUabHZjXYuarosZiJjSWVbTWdYX+1\nHZ/BjzunajsvgsysqCLDhU6NvM5erQ7Ude/Qx6WUVRXvmjwhjWb17t1VRFMY6x46InPkmNJl30sn\nnd7rlznO4nE/krxFno1ay+2oQ3qcGQ+S3wyp7Ns/Iau8MqChLepwZj5LfEIU+Jtn5t1d4hXHt5y8\nxf1SWbZmQ1b4ZSzcejeH/GeaYm2Zm3VviHkmJuLVvD/kPNYsSzbPyGrvDKWbZmQ1b4ZTE2z826u8\nQpibZmbdW+IeSCGtA6pPkPku8MqazcLMhq3wzyUNa5vVJ8x8l3iFTYm2Zm3VviHksv4Zz6Y/zoWb\ni0bw/wCI80s2z8hq7wymJuLVvD/kPNAWkPzbq7xCoYFm2ZkNW+GUs3Ho3h/xnmmJtmZt1b4hTE3F\nq3h/yHmgWbhfkNXeGV8rsnim07u66+qLm4X5t1d4hXyuySA6a9u7vsrGL0W/s7n0T9s/51aFhZ+m\np+gpYWbpqPoKXFnZjU/WlxZumo+tYvWvbDF3cPkPNLCztNT9BXJe1rhdzRu97/VedJHZ2+zU+Ip0\nlGsOrCzNNR9BWdti14bW0P02V/pI7N32ajxFn7Wc13RYSDkdHXW3FE78K+0zHCl9PQY+p0/H8lvJ\nT79n8ervKq1CB1ODIfJb4ZU9hZ+Q1d4ZVO3nLwN/VLrfszj1b5eS938f18P281xYWZkNW+GeS9sM\neg4f8Z5qGD3fs/j1d5U37M49W+XkubCz8hq7wylhZmQ1b4ZQR12PqlRx/JdyWXvWZxat5LSrgOqT\n5D5LvDKy7CzchqPoW2nk9T9A9F/7Ot7F9XD6c03rP4tTyXlhi7uHyHmlhZ2mp+grJ6F7vWZxajkm\n9i+rh9Oa5sLN01H0FLC/dw+Q80HRxWdxau5L5o6r6Sws7TU/QV82dVZ2f+XP27/j/cREVpzhERAR\nEQEREBERAREQEREBERAREQEREBERAREQEREH01Ltykip4mOklBbEGmzRqpf/AJBRWd8SbMn6AvlE\nVadmpKlOw4pnV9Z/8gorN+JNkR9AXT/+QUHQvLJJelwENuzvXyKJyuNHIYWr/wDIq/Pejzvfd5r3\n/wCRV+W9Hlb6VkotvCx9G/lsPbDUk29Wyscx7mYXtwmze5fSiqYWM+I/Kx0C+GX1TOBv4W7DseLN\nrvR5NeTZMVtPDT+jQ60zFfpH8NtBzXraqMYrvfmT3BUEW/7Xg/7auQxJ+tzZb2lrZJ1ua98XdbRQ\nIrXKYOyPZv5fF2wmdWTBjt4Z3Oi+UgqpaYu6MgYtbhfSP4Hfgr5Q6qptOHHTSK101bsVYxTrjjRc\n7Uqc825+idqVOWbcvtVNFU4dOizx8vdKepq5arB0pBwaWCgRFlEREaQ12tNp1kREUsX2lFLEKSAG\nSO/Qt8Wym6aKz/iR6u8ZfKdGzyj2To2eUeypzssTOurmz9PiZ13n1nTRWZ8SPVvjJ00WL5kfD/m9\nV8n0bPKPZOjZ5R7JykdUfbo7n1fTRWf8SPV3jJ00VmfEj1b4y+U6NnlHsnRs8o9k5SOp9ujufS1s\nsRpJ7SR/JdpLdZvSMs3ebqPEWZ0bPKPZOjZ5R7LKNmiI01dPYf8AZxaI8dWl0jL8TeH/ACeqdIze\n3m6nxFm9Gzyj2To2eUeynl46uhz09rT6Rlm7zdR4i86Rl+JvD/k9Vm9Gzyj2To2eUeycvHU56e1p\nGRlnbzdT4i+eOqv9Gzyj2To2eUey248e4r58/F08NNFBFf6NnlHsnRs8o9lsV1BFf6NnlHsnRs8o\n9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6N\nnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8\no9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6\nNnlHsnRs8o9kFBfTMniwN+KzTzLI6NnlHsnRs8o9luxZuHr4GjZ6eH/Kz/7J08P+Vn/2WN0bPKPZ\nOjZ5R7Lfzk9EaNnp4f8AKz/7J08P+Vn/ANljdGzyj2To2eUeyc5PQ0bD54sDvis08y+YOqv9Gzyj\n2To2eUey0Zc3E08E6KCK/wBGzyj2To2eUey0igiv9Gzyj2To2eUeyCgiv9Gzyj2To2eUeyDsC5A5\nrSn2FVwdK17d9j2saB9dzbL+1mA2NwtB22q5z3PMrbusfltsCDe4FsjfO6JQnZ9WKZ1SYSIW6uJH\np/7CjdSzNpm1JjIhc7CH9xPJSVO0amqwCdzXNYSWtwAAXAGg/AXo2lUiJkQLOiY7E1nRggHnmPVB\n7DsutnDDFA52NuJuYzF7f7Up2FtMMxmkfhte9wuIdrVkAYI5GjAMIJY0mwNwL27jpyUjduVwpXwd\nIDidixFou3IjLlqh4OBsWvMBl6HIPEeEkXJJI/2CFCaCqFYKQwnpzoznlf8A0p5Nt18j8bpWl1wR\n8NuRBJBGWRuT7qHtCfrHTjo2y+YMA7rIJextoa9WdbncW/36Lp+xaqGPpagNij6PpC4nTO1rc7ri\nHa9bBE2KOVuBotYsabjUXuM7HRO1aksLJcErHNLSHt1BN8yM8jmh4I6ihlhaJAWyRFgkDwdQTb/e\nSqq5LtGSWN0ZiiDDGI2tDeAA3y9b/wC1TQEREQIiICIiAiIg9AuQOa0p9hVcHSte3fY9rGgfXc2y\n/tZgNjcLQdtquc9zzK27rH5bbAg3uBbI3zuiUJ2fVimdUmEiFuriR6f+wo3UszaZtSYyIXOwh/cT\nyUlTtGpqsAnc1zWElrcAAFwBoPwF6NpVIiZECzomOxNZ0YIB55j1Qew7LrZwwxQOdjbibmMxe3+1\nKdhbTDMZpH4bXvcLiHa1ZAGCORowDCCWNJsDcC9u46clI3blcKV8HSA4nYsRaLtyIy5aoeDgbFrz\nAZehyDxHhJFySSP9ghQmgqhV9VMLumtfD6Wvf2U8m26+R+N0rS64I+G3IgkgjLI3J91G3aVQ2oNQ\n3oxKb3cIxpa1raWsgm7ErGtjfMGxMfiJc53CG5m/skuxqiKF0ryMAiErT5hu5eh3gue2asyB8pjl\nsXXD2DMOFiMu5cu2vWOYGGRpYGdHhwC1svT7Rn6IeDnsquEcshp3BkV8ZNsrXv8A6KhZSzvgM7Yy\nY23u7uFrX/2PdT1W1aysjDKiQPZiDrYQM8+X5PuvWbSfHS9AyJjWukbI+2jsOgsgrU9PLVTCGBhk\nkdo0alTybOlbFHIxzXhxc0gGxa5ou4H+lHHWzQ1jqqAiKUkkFo4b8uSmdtWZxZeOGzcZIwZOc4WL\niOf/AKQUUREQIiICIiD0C5A5rSn2FVwdK17d9j2saB9dzbL+1mA2NwtB22q5z3PMrbusfltsCDe4\nFsjfO6JQnZ9WKZ1SYSIW6uJHp/7CjdSzNpm1JjIhc7CH9xPJSVO0amqwCdzXNYSWtwAAXAGg/AXo\n2lUiJkQLOiY7E1nRggHnmPVB1T7JrKnD0cQs+MyNxOAuAvex6+1+ruADcWZGi5ZtSrZHFG17MEQc\nGtMbSN4WN8s8uaT7UrKiPo5JbtsBYNA0QcTUFVBUx00sLmzSWwt53Nv9qZ2xdot1pne4/PNcP2lU\nPqW1LujMzSHB+AAgg3v7r2Da9bBF0ccoDLk2LAb31Gmnogk7ErGtjfM1sTHYruceEN1v7KrPSuig\ninD2viluAR3EagqwNsVeMOkMcti7J7BYhwsRl3eiifXSOiZE1kbI2NLQ0Nvrqc75+qCRmx9oPcWt\npnFwtcAi+a7dsLabGlzqR4AF9R/7XjNt18by9kzQ82u7o23JHfpr3XUp/wCQ7RMLGdK27CTiwC5B\nsbaaXCHgiOxK/BG7oL9JewuLi3NQQ0FVPUvp44XOljNnN5Z2/wBqY7ZrjJjdI0uuTcxtOosRppbu\nXNPtSppZHSQFjHu4iGDPO+n5CCR+xqmEF05ZGwRtkve98Wg/K9n2JVQPkjcAZWPDQwfWCSAQfyCo\nhtSo6NzJBHI10Yj3mDQaG47wun7ZrnzumdK3G4tN8DcrG4tllnf3Q8EUuzquGm6xJC5sWW8SO8Aj\n/YUTqaZkAndGRE61ndxve3/+T7KWp2hUVcjX1DmvwZAYQAMgNB+ApTtR+CmjbFGI4MRDDmC52pQV\nqeknqsfQRl/RtxOt3DmpKiglhaJAWyRFgkD2nUE2/wB5LmkraiiLzTvwdIMLt0G45Z9ykl2lJLG6\nMxRBhjEbWhvAAb5et/8AaCmiIiBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQERE\nBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERARE\nQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBE\nRAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQE\nREBERAREQEREBERAREQV+tfZ+0619n7X1NFBAaSAmKMkwt8K6n6vBZ/wo9XeCqk7VETpo50/UIid\nN18d1r7P2vetfZ+19h1eCzPhR6t8FOrQYvlR8P8Ah9VHNx0R9xr2vj+tfZ+0619n7X2HVoLP+FHq\n7wU6vBZnwo9W+CnNx0PuNe18d1r7P2nWvs/a+qraeAUk5EUYIhcRaKywGxR4RuN05Lfiy8SNdFrZ\n88ZomYjRU619n7TrX2ftXOij8jfZDEzCdxvstqwp9a+z9p1r7P2tjoorN3Gajw170UWLgZp/j9VX\n5iOi/wAjPcxutfZ+0619n7Wz0UVnfDZqfDWTRUjakyYnFuG2gusq5YmJlqybNasxWJ11cda+z9p1\nr7P2rvZUWfxX5E/SnZUVh8V+ZH0qeNQ5TL0UutfZ+0619n7XddSNpcGF7nYgdRZVFsraLRrCves0\nndlY619n7TrX2ftV0UsVnrX2fteda+z9r6qip4TSQExRkmFvhXU/VoLP+FHq7wVUnaoidNHOn6hE\nTpuvjutfZ+0619n7X2PVoLM+FHq3wU6tBi+VHw/4fVRzcdEfca9r4/rX2ftOtfZ+19h1aCz/AIUe\nrvBTq8FmfCj1b4Kc3HQ+417Xx3Wvs/ada+z9r6qtp4RSTkRRgiFx+VZYDYmYRuN05Lfiy8SNdFrZ\n88ZomYjRU619n7TrX2ftXOij8jfZOij8jfZbVhT619n7TrX2ftXOij8jfZOij8jfZBT619n7TrX2\nftW3RR4TuN05KvQ0jarHicW4baC6i1orGss6Um87sOOtfZ+0619n7V3sqKzvivyJ+lOyorD4r8yP\npWvjUb+Uy9FLrX2ftOtfZ+1d7KivbpX6X4U7Kiz+K/K/0pxqHKZeil1r7P2nWvs/au9lRWHxX5kf\nSqtdSNpcGFznYgdRZTXLW06Qwvs+Skb0uetfZ+151r7P2vqqKCE0kBMUZJhac4rqfq8Fn/Cj1d4K\n0TtUROmjjz9QiJ03Xx/Wvs/a8619n7X2PV4LM+FHq3wU6tBi+VHw/wCH1Uc3HRH3Gva+P619n7Xn\nWvs/a+x6vBZ/wo9XeCnVoLM+FHq3wU5uOh9xr2vj+tfZ+151r7P2vqq2ngFJORFGCIXW+FZYDYo8\nI3G6clvxZeJGui1s+eM0TMRoqda+z9p1r7P2rnRR+RvsnRR+Rvstqwp9a+z9p1r7P2rnRR+RvsnR\nR+Rvsgp9a+z9p1r7P2rboo8J3G6clXoaVtTjxPLcNtBdRa0VjWWdKTed2HHWvs/ada+z9q72VFZ3\nxX5E/SnZUVh8V+ZH0rXxqN/KZeil1r7P2nWvs/au9lRXt0r9L8KdlRZ/Fflf6U41DlMvRS619n7T\nrX2ftXeyorN+K/O30qrXUjaXBhcXYgdRZTXLW06Qwvs+Skb0vq6FzepwZt+S3xCFPibZ+bdXeIVF\nQY+p09sfyG8lPv2fx6u8q5dvOXkr+qXOJtmZt1b4h5LqrqYmUcxbYPbESCHJv2Zx6t8vJQV+PqdR\nx/IdyURGswisRNofJ9q1+f8AKkz9U7Wr8v5UmXqqaLrblej0fCx9sey2/adbI0tfUvLXDCRfUK23\nhH4WStZvCPwsoiI8mda1r5Q9XjuE/herx3CfwpZNRgBY3IaBdWHILlnA38LpdfhY+2DiX6y5fkx1\nh3FY+yCLzXt3d9lsP4HfgrI2Pe81r6DRc/6hWtaeELGy2m2WNZaFxZ2Y1P1pcWbpqPrK63rP4tXc\nuSb1mcWo5clxXYZm17fBtyPfdZq09s3vDe+h1WYr2H0Q421ftkREW1XfbUJb1ODMfJb4hU+Jtn5t\n1d4hUVBj6nT2x/JbyU+/Z/Hq7yrj285eZv6pc4m2Zm3VviFMTcWreH/Iea637M49W+Xkvd/H9fD9\nvNYsHGJtn5t1d4hTE2zM26t8QrrfwycervKm/ZnHq3y8kFauc3qdRYt+S7xCvn28I/C+ir8fU6jj\n+Q7kvnW8I/Cv7J5S7H0702eoiK26IiIg8dwn8KPZBzmvbQd9lI7hP4XGxr/GtfQaLVm9ErOy/tho\nXFnZjU/WUu2zcxqPrXW9Z/Fq7km9ZnFqOSouw5uMWo4fP6pcWdmNT9a63sX1cPpzTes7i1dyQc4h\nZuY1H1rO2wR8G3I991p71mcWreSzds3vDe+h1stuH1wr7V+qX0dC5vU4M2/Jb4hU+Jtn5t1d4hUV\nBj6nT2x/IbyU+/Z/Hq7yqrbzl4C/qlzibZmbdW+IUxNxat4f8h5rrfszj1b5eS938f18P281iwcY\nm2fm3V3iFMTbMzbq3xCut+0nHq7ypv2Zx6t8vJBWrnN6nUZt+S7xCvn28I/C+ir8fU6i+P5DuS+d\nbwj8K/snpl2Pp3ps9REVt0RERB47hP4UeyLfGvyHfZSO4T+Fxse95rX0Gi1ZvRKzsv7YaFxZ2Y1P\n1pcWbmNR9a63rP4tXck3rM4tRy5Ki7Dm4xajh8/qlxZ2Y1P1rrexfVw+nNN6zuLV3JBzcWbmNR9a\nztsEEw2toe+6096zOLVvJZu2b3hvfQ62W3D64V9q/VL6OhA6nBkPkt8MqezbPyGrvDKgoSOpwZj5\nLfEKnuLP4dXeIVVt5y8Df1SWFmZDVvhlC1pdYhtsP+M80uLM4dW+IVFWPwU0zmOAcIiQRIcioiNZ\nYxEzOjroYrP+FHq7wl70MVmfCj1b4S+Q7Vrs/wCVJn6p2rX5fypMvVWuWv1dDkMvc+oroYhRzkRs\nB6F1vhWWC0bo/CqP2nWvaWvqXlrhhIvqFbbwj8KzhxzSJ1Xdmw2xRMWnV7ZeOG6fwvV47hP4W5bX\nw42ZvHUfSV6C4uyceHynmpGAFjbjuC6GWiy+35u50edx9qI4yH5nU/SVlbI1mvyHddbLzZjvwVjb\nIOc1/TvstObZ74aTvTrqVzUy5a7saebSsLO01P0FLDC3TUfQlxZ2mp+sry4s3TUfWVQX2dtgD4Nu\nR7rLNWltg/J/B77rNV7D6Icbav2yIiLarvtqEDqcGQ+S36Cp7Cz8hq7wyoKEjqcHD8lviEKe4s/N\nurvEK49vOXmb+qSwszIat8Mr2wx6Dh/xnmvLizOHVviFLjFq3h/ynmsWJYWfkNXeGUsLMyGrfDPJ\nLiz+HV3iFLizM26t8QoIK4DqdRkPku8Mr59vCPwvoK4jqdRp8l3iFfPt4R+Ff2Xyl1/p/ps9REVt\n0RERB47hP4Uex9Zr8h3XUjuE/hR7H1m/A77LVm9ErOy/thpWFn5DU/QUsLNyGo+gry4s7TU/WlxZ\numo+sqi7D2wxd3D5DzSws7Ian6Clxi7uHznmlxvaan6yiCws3Iaj6Fm7YteG3I91lo3Fm6aj6ys7\nbBv0NuR77rbh9cK+1fql9JQgdTgyHyW+GVPYWfkNXeGVBQkdTg0+S3xCFPcWfm3V3iFVbecvA39U\nlhZmQ1b4Z5L2wxaDh/xnmvLizOHVviFLjFq3h/yHmsWJYWfkNXeGUsLMyGrfDKXFn8OrvEKXFmcO\nrfEKCCuA6nUZD5LvDK+fbwj8L6CuI6nUafJd4hXz7eEfhX9l8pdf6d6bPbJZEVt0SyWREHjhun8K\nPZFrzX5DuupHcJ/Cj2RrN+B32WrN6JWNl/bDRsLOyGp+gr2ws3Iaj6CvLiztNT9ZS4s3TUfWqLsF\nhi0HD5DzXthZ2Q1P0FeXGLu4fOeaXFnaan6ygWFm5DUfQs7bAsYbcj3WWjcWbpqPrKztsEEw25Hv\nutuH1wr7V+qX01Bj6nT2x/Ib5VPv2fx6u8qzaOvomUsLX1MIIiAII0Km7RoLP/lQZk23VXtS2s+D\nwl8V96fx+Fzfszj1b5eSgr8fU6i+P5DvKou0aCzf5UGRF91Q1lfRPpZmsqICTEQAG6lK0trHgUxX\n3o/H4fJIiLrPRi1m8I/CyVotqIsI3wgmXjuE/hR9Yi84XjqiIg74RLZZwN/C6VVtfShoBmbovev0\nv+Zq7PEr1YJ38DvwVkbHvea19BpZX319KWOAmbosWmqjTF9mNdi5rn7dpesRVv2e8UyRazdGKz+L\nU8k3rM4tRyWT2o/P4Med07Ufl8GPKy5HBu6fN4uqTbN/g3vodbLMViqqjU4bsa3DyVdW8dZrXSXN\nz3i+SbQIiLNpfcUGPqdPbH8hvlU+/Z/Hq7yrNo6+iZSwtdUQgiIAgjQqbtGgs7+TBqfpXJtS2s+D\nzl8V96fx+Fzfszj1b5eSb+P6+H7eap9o0Fm/yYNRfdTtGgxf9qC2Hy+qx3LdPhjwr9vwub+GTj1d\n5UOOzOPVvl5Kn2jQWf8AyYMybbqdoUFm/wAmDUfSm5bocK/b8Ja/H1Oo4/kO8q+dbwj8LXrK+ifS\nzNbUQkmIgAN1KxG1EQaN8K9ssTETq6uwVmtZ1hKii6xF5wnWIvOFadBKii6xF5wnWIvOEEjuE/hc\nbGvea19BpZeGoiwnfCq0tU6mxYWNdi5rDJWbV0huwXimSLS3bOs/i1PJN6zOLVvJZPaj8/gx53Tt\nR+XwY8rKpwbulzeLq1t7F9XD6c03rP4tXclk9qPv8mPSydqPz+DHndODc5vF1au9ZnFq3ks7bN/g\n3vodbKPtR+XwY8rKCqqnVOHExrcPJbMeK1bay059ox3xzWJfYUOPqdPbH8hvlU+/Z/Hq7y8lm0df\nRMpYWvqIQREAQRoVN2hQWd/JgzJtuqnaltZ8Hir4r70/j8Lm/ZnHq3y8k38f18P281T7RoLN/kwa\ni+6naNBi/wCzBw+X1WO5box4V+34XN+z+PV3lQ47M49W+XkqfaNBZ/8AJg1Nt1O0aCzf5MGovupu\nW6fBwr9vwlr8fU6i+P5DvKvnW8I/C16yvon0szWVEJJiIAA1Kw21EWEb4V7ZYmInV1dgrNazrCZF\nF1iLzhOsRecK06CVFF1iLzhOsRecIJHcJ/C42Pe81r6DSy8NRFhO+FVpao02KzGuxc1hkrNq6Q3Y\nLxTJFpbu9Z/FqeSb1mcWo5LJ7Ufn8GPO6dqPy+DHlZVeDd0ubxdWtvYvq4fTmm9vcWp5LJ7Uff5M\nelk7Ufn8GPO6cG5zeLq1t6zeLUclm7ZveG99DrZR9pvy+DHlZQVVUanDdjW4eSzx4rVtrLTn2jHf\nHNYlZRetF3Acytqo/wCOSQukYJ2vd0jWRkDJ13Yc+Vj+Vac5iIr82yZ4KM1M0kLBewY51nO00FvU\nKN2z5m7PjrbsMUjsDQL3vnla3ogqItOm2FVVLYyx8LTI0PDXE3AJsL5d5KnP/Ga0RdJ0tORyDzcm\n17aa5IaMVFqv2DVMYMTmdI6VsbWX1Jc5t/dpUDtlVDdpGhxRmQC5cHboFr39kFFFru2G+CRgqZ2t\nbaQvLGkloaL5DK69m2EYqZ0/T4m9CJGANzvu3B5cQ5oaMdFpz7CqqeCWaeSCNsZLd5xBcRfIZZ8J\nVaKgmlon1TC0xsNnZ5g5WH93y/BQVUVihopa6rbSw4RK69g42uR3KSromwUzJmS47yOieMNrOba9\nuYzQU0REQIiICIiAi9aLuA5lbVR/xySF0jBO17ukayMgZOu7DnysfyiWIivzbJngozUzSQsF7Bjn\nWc7TQW9Qo3bPmbs+OtuwxSOwNAve+eVreiCoi0YtjzSQRTdJGGyxve3UndF7HLVSP2BUxsxSTU7G\n4Q67nnXlpqgykV6p2TUU20YqGR0ZllLQMLrgEm2f9qy3/jtU5pcyemcwXu5ryQABmdO7RBkItY7E\ndE9nWJmtZaRzywYiA0Xy53VOppWR0kFTFI57JbtOJti1wtca+oQVUREQIiICIiAiIgIiICL1ou4D\nmVtVH/HJIXSME7Xu6RrIyBk67sOfKx/KJYiK/NsmeCjNTNJCwXsGOdZztNBb1Cjds+Zuz4627DFI\n7A0C9755Wt6IKiLRi2PNJBFN0kYbLG97dSd0XsctVI/YFTGzFJNTsbhDruedeWmqDKRXqnZNTTbQ\njonYHTSWw4XXGZsrJ2BKyWFkk8fxHlt23Nhha6/s5BkItpuw43NhlbUkwztODc3gd61xfSzT3qEb\nCqer9YfLBHFYHE95GoBHd9wQ0ZaK1TUE1VBLNGWYIc33OgsTf9W/Kip6d9RUsgZYPebDFlmgiRX6\nrZpp6V8hkvLE8MmZbhLgSLHv0N1QQEREQIiICIiArBrqtzi41U5cRYkyG5HJQNF3Acytqf8A47JC\n6Rgna93SNZGQMjd2HPlY/lEszrtV0zZXTve9puC84rH+106vrJZMRqJcV7jC62foApZtkzwUZqZp\nIWC9msc4hztNBb7guWUVTDSMr43hrL2Y5riHYrnIeuV0F+i2dteWKIwVZjjczEz4xAt3/sgH1XJg\n2jDSTwGttkx0kYcb55AX/v8ACihpqqNsL461zC6mfM0NcQQBe7R+cKnbQbQmxupqyQxxU7HXkeW5\nObfCNf8A8ESS0O043OfPtINBLWYnTO3s3ZDLuLXKeOk2vM6SOGvG7O6K7yWvJF8725DuKqVtFVPb\nUydcfUNpCGymQm4dplzGZsfyq1LPtGqqBBHVz9ISXAOldm4D/aDRZQ7QjqoadlcX1bi57Y3EuZYX\naTc6k2OVtPZY76qrDheolyBYLPIGHkPRT1VTXU73U8tQ4ujed8OzB7xi117uYVifZ9VJQxy1FQws\nZAZIWi5JbiF+7m79IhPWUFVEWiWvklmbAKlrTvN3Sbi9+63JUanadaHCN07HYHtfiYwC5GncL2uo\nIKiow9WEuBstmEu7m3va+oFzdW9pUFUxklTWSA1HSNY+MDMXBtfu0HcgzzUTGd83SOEjyS5wNib6\nruernqWMZK/E1l7C1szqTzJ5qXZdD2hXspi/owQSXWvawuoKqA09TLCST0b3Mva17GyCJFs1Owmw\nYGdZcZXwGVrejsMhcgm+WSxkBERECIiArBrqtzi41U5cRYkyG5HJV0RKx12q6Zsrp3ve03BecVj/\nAGunV9ZLJiNRLivcYXWz9AFVXUT5I5Gvic5rwci02IQbFLQ7UnbT9HWYTgDo2mRwLA44R3ZXvZSy\n7D2rLBikrI5GgYQ0zE917aei4otn7Vmgp3QV2Bj2ksHSvGHeAtkLXJIXLmVUVNUwx18nR9CycsNw\nX3sLfgYuahKY022IosJq7vfK2INJu7MuaN45jNhVRtNtGCv7Pjqi1zd+7JTgGVyfZWJqasZJKJtp\nymQvjiYcTiHE5i5vkAqtbU7UoZ3081S/pGEYnB1yDbTFqpQuCirqWeMz15DR0sj7Xfawz3Tre/7X\nNbsuZ1IJZKkOjjgD4mtjAtwkgju4wb5qq7tGnpKet6y58cjjgBcXXOYdcH0/3+VzRNr68y9FVOHQ\nREkGQizO8C3d6IOp9hVVPBLNNJDG2Mlu84guIvkMs+EqtFQTS0T6thaYmGzs8wcsv7vl+Cpa99VD\nUNa6rknDWska4kkZgEHP8rRpXVNXsqSsnrCxkMzXEMiBLrWse69sSDGpKSWsn6CEAyEEhp77C9vy\npqygFNAyRsvSfEdE8WtZ7bXse8Z6rqeSp2btaZ4ka6cOJ6TCDe+dxyOarTVU00ccb3bkfC0Cw9T+\nfVBCiIiBERAREQFYNdVucXGqnLiLEmQ3I5KuiJWOu1XTNldO972m4Lzisf7XTq+slkxGolxXuMLr\nZ+gCqrqJ8kcjXxOc14ORabEINilodqTtp+jrMJwB0bTI4FgccI7sr3spZdh7VlgxSVkcjQMIaZie\n69tPRc0WzdsSxRGCsLI3MxR/GcBbv/ZAPquDBtGCkng67bJjpIg5188gCf77rhQlMabbEUWE1d5H\nytiAJu7MuaN45jNhVYx7UZtU0nXHOmB6Qu6UluTb3P8AXoppaHacbnPqNpBoJazG6V+9mchl3Frv\n/Cmjo9rzOkZDXi7Z3RXe5weSL53tyHcUEU1DWsnjNRXGNrekeeiv8PCLnC3LX+l5U7NqRRFz6+SS\nFsIkjZmR9N2kXy4hpddtodoMqoadtcX1bi97Y3OLmWzaTc6k2Pdp7LHfV1YcAaiXIFgs8gYeQ9PR\nSNisp9rCiqDV1FOyMHC69ml9ichYZ5tKoxtrJse1g6N5jdvFwBs4WAFrW78vwVcrKCqiIbNXySzN\ngFS1pOJu6TcXv3W5KjUbTrQ7o3VDX4HtfiY0C5GncL2uiHMRrNpOFI1wc67pMJFi92pvzOuqmbsc\nCoggfUjHNia0sbiaHg2sTl7qnDWzw1bqpjh07iTjIBzOp/K8hraiH5crhYEC+eG+tuRPNBARYkHu\nRERAiIgIiICsGuq3OLjVTFxFiTIbkclXRErHXqrpmzOnkdI04gXuxWP9ruXaVXN0eKUhzHFwLBhO\nI9+Xfkqi6je+ORr4nOa8aFpsQg2qbY+1JYqcxVbWscAW/FcMGIC17Dvxd3Nex7M2nmxle2zYw0tb\nK42FrhvsCeS7p6LbVU2CSKswN6ICIiUtAbYcvWwPqqpoq6nZORW4ZMLDIxsjrm+gP/4qErNVT1E0\n88UdXK+aOSOMAtDRIHWsXWOunsF66gqY9pwzSVLHVT5nB78Ic1tmh2IetjyGYUUmzdoROc6baLGC\n7WYjK7ezNgMu4tPtkpYtn7VmqHmHaBLm1BZaSRwJc3v0sch6oIRRS1MdO2WpPVJZCKd2G5L3HvH9\nZ8rhZhqamMlnTyjDdtg86aW/QW6afaHWKemZUwirs90bGRtDGi5BcCBxZE6aD+liVlIaYQHH0nSx\ndJcd2ZH/AIUoRyVEkkrJJCHOYGtFwLWAsBb+lI6vqnVDpjM7E54eQNCRplpkpqDZpq6WqqTJgjp2\ngkCxLsxoPwo62jFM92GVr2WY5txZzg5uIGyCIVU7al1Q2V7ZnEuL2mxudV5NUzTsYyV5eGEkX1uT\ncqJEFo7RqzG5jp3ODm4LuNyG8ge4KqiICIiIEREBF60XcBzK2p/+OyQmRona93SNZGQMjd2HPlY/\nlEsRdRvfHIHxOcx40c02IV2bZM0FGamaSFgvZrHOIc7TQW9QuW0dTDSM2hHIGsvZrmuIdiuch65X\nQaVDRbZkgb0NWGxSxbpdKbYcr25Z2BVWSirI4pA+sGKONrTGHuuGu+nl/WiloNl7SljgmpqxsYe0\n4D0jhh3gLZDIkkLmSGoYJTHXzWhpWPaC4glrsNxroLqEpnUW0oWvE1fEGEtbie8m5BIGHK4ILXJ2\nXX1cb6ZlVG6NtS5mGRxxFzQRfTQNHcclHU0M7nyNqq973iRkcZJLgXEXFz3AC/uq1bLtKhqX081Z\nNjjcC7DK4gOt/tSLsOzq2Cogpoau9WcTmR3uwNGIXB58R00us2eKWgEPRTu+NEJCWEgakW/SsTna\nLNmwVL6kuilecBDjjv3569wOvJc7ObXVYqBBUZRwuxiRxO4eK2qISbOhqaqhrZDUubDHE1rhfEXN\nDhkLnQapFFVU+0GUVHU3jlwyMx5MILQ4Fwz7v9KpUwvoKjBHKHgsY4uA3XAgHQ6j8rToBUTUs+1Z\nK6WMxyC4Yy9wABe17ZB39IOJdgV80kks1RAXG73vc46ZnFppkfZRdhvjY500zLGN7mdHvXLWlwue\nRA/YVevkrKWolpZKuV4aT9Zs6+d7et1AK2qAYBUzARizLPO6PTkgv9hySh76eRpijhZK50m6d5t7\nD11Xk2yI4pZ29YIDZWRRFzeIuF8+QsqLa2qa3C2pmAw4LB54eX4XQ2hVhrh1h7sVs3G5FtLE6W9E\nFqbZDomysfK1s8EjWStPC3Fob+neswixsrTNpVbGsDZjZjxILi5uNL87Ks5xc4uOZJuUHiIiIetF\n3Acytqf/AI6+EyNE7Xu6RrIyNHXdhz5EH8rEVg11Y55e6qnLiMJJkNyOSJTzbKmgozUzSRMbezWO\ndZztNBb7guW0dTBSM2hHIGsvZrmuIOK5yHrldRdeqjM2Z08j5GnEC92Kx55ruXaNZMWXlIcxxe0s\nAacR78u/JBfodmbSljgmpqxsYe04LSOGHeAtkMiSRkuZIahglMdfNaGlY9ocSCWuw3HKwv8A6U9D\nRbalgaIKzDFJFul0pthyuByzsCq0lDWxxPD6zejja0xh7smu+nl/WihKSooJ3PkbU17nvEjI4y4k\nhziLi5Olv/KqVc20qOfoZqqcPaQ8jpTk4i+fqtA0e0qfEZ66AsuxvxXF4cQXAAAjuLXI/Zu0a8zx\ndaje01bgWvJBc8Xu7TkO4qRUmdtFmz4p31BMT3XY4O3iTe4vr3XI/Cip4KvazRG2RpFO3R5tZpOZ\n9zmtJtFXMlpKOKpjFUwPMTGtGANzBN+8mx/oLFjnlphLHGQOkGBzsOZHoUQsVbp6LFSsn6SN8TCX\nAatIxWBOdrlWIaLtCjqNp1dQ4FrwHBrASRkCf2FPUbPqKWGSB1a50j6QSFtrtLWk3be+VrLIp62o\npm4YpXBmLEWatJ9RoUCtpnUdXJA5wcWHUd41CgXcsr5pXSyuLnvN3OPeVwgIiIgREQEREBERAXUb\n3xPD43OY9uYc02IXKIJo6qoiLTHPKwtBAwvItfVeisqha1RLk3AN85N5fhQIiUstVUTG808khy4n\nk6aLvtCs/wD659cXzDrzVdEFk7Qq3RhhqHmxJBvvZ656rllbUsDgJn2cwxm5vunUZ6XUCIOo5Hxu\nuw2vqO4+hHeFNLXVMzZWySkiUhzxYZ209lXRARERAiIgIiICIiApzXVbnFxqpi4ixJkNyOSgREp+\nu1XTNlM8jpGnEC84rH+1JLtGsnLAZTdji5pYA04j35d+SqLqOR8Tw+NzmPbo5psQg3aGh2zJA0QV\ngbFLFul0pthyvblnYFVpKCtjieH1e9HG1pjD3GzXfTy/rRZ8dVURFpinlYWggYXkWvqvRW1Qtapm\nybgG+cm8vwg130O0oA8TV8TWEsbie8m5BIGG4uCC13JOytoVUb6ZtUx0bakswyEglzQRfTQNHNY0\ntVUTG808shyO88nTT/ak7Qrf/wCufXF8w680GrFs2thqIKaGsvVnE5jL3YGi4uDz4u5Z08M1CIjF\nM/40XSExkgakW/S4O0at0YYah5sSQ6+9nrnquWVtSwOAmeQ5hjNzfdOoF9LoLuzoamqoa2Q1Lmwx\nxta5uTi5oIyF9ANV7FBVwV7KOjqT0cmGRpfkw3biBcMxp/pZkcj4nXYbX1HcRyI7wrDdp1jak1An\ncJCQctLgWGWmQQaEv/H6+aSSWaeAuJLnPc86ZnFppkfZRdhyRsLppmZxvczo96+FpcM8siB66hUH\nVtU5hY6pmLTe4LzY31QVlUAwColAjFmWed0enJBf7DllD308gMTIWSl0m6d4XsPXVeTbHZFLM3rB\nAbKyKIuZxFwvnyFvyqLa2qa3C2pmAw4LB54eX4XTa+qDXDp3uxW4jci2hBOlkFqbY7omysdK1s8E\njGStI3W4tCD3+uXus0ixIVpm0qtjYwJjuPEguLkuGl+dlVcS5xccyTcoPEREQIiIC6ifIyRronOa\n8HItNiFyuo5HxPD43OY9ujmmxCJbdJsbarmwSQVTWB7QWkSuGAOtYGw77jRGbL2m42bXN3I7ENlc\nS0EXw+wvbRZh2lWkxkVUrejYGNwuLbD+vwFGyqqGY8E8rceTrPIxflBq1NBPNNPEKyeaaOWNrBJo\n7Fob3yIuVK6hr45YYn7SmxumcHlryWts0OxDPM2PosXrlSSSaiW5ABOM92nsuxtCsEmPrUxdiDs3\nk3PNBqxw1UzYcddIKeZ+GCW13lzicj3jMZ58llVb5iyFk0oeGtIaB9O8b3/tSna1XiYWuY3o8WDD\nG0BpOpGWvqq0tRLNHEyR12xAtYOQvdB6yrnZC+FshwPaGkel72v3C/coURARERAiIgIiICIiAiqd\nZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7\nILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZ\nOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp\n1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyen\nsgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09\nk6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoq\nnWZPT2TrMnp7IN3YtXTUksrqoYmOZbDgDsXp6LzblXTVtcJaVmGPBa2HD3n/AMWH9LD6zJ6eydZk\n9PZE6oUREQIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIi\nAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIi\nICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAi\nIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiIC\nIiD/2Q==\n",
"text/html": [
"\n",
" <iframe\n",
" width=\"400\"\n",
" height=\"300\"\n",
" src=\"https://www.youtube.com/embed/3uiEyEKji0M\"\n",
" frameborder=\"0\"\n",
" allowfullscreen\n",
" ></iframe>\n",
" "
],
"text/plain": [
"<IPython.lib.display.YouTubeVideo at 0x7f86d4fade48>"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from IPython.display import YouTubeVideo\n",
"YouTubeVideo(\"3uiEyEKji0M\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Locality is hard. Potential trade-offs:\n",
"- redundant computation to save memory bandwidth\n",
"- sacrificing parallelism to get better reuse"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Temporaries"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The issue of \"temporaries\" occurs when the result of a calculation is stored in a temporary variable in RAM, and then that variable is loaded to do another calculation on it. This is many orders of magnitude slower than simply keeping the data in cache or registers and doing all necessary computations before storing the final result in RAM. This is particularly an issue for us since numpy generally creates temporaries for every single operation or function it does. E.g. $a=b\\cdot c^2+ln(d)$ will create four temporaries (since there are four operations and functions)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Scaling to multiple cores and nodes"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We have a separate section for scalability, but it’s worth noting that this is also important for speed - if we can't scale across all the computing resources we have, we'll be stuck with slower computation."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Scalability / parallelization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Often we'll find that we have more data than we have memory to handle, or time to compute. In such a case we would like to be able to scale our algorithm across [multiple cores](http://www.makeuseof.com/tag/processor-core-makeuseof-explains-2/) (within one computer) or nodes (i.e. multiple computers on a network). We will not be tackling multi-node scaling in this course, although we will look at scaling across multiple cores (called parallelization). In general, scalable algorithms are those where the input can be broken up into smaller pieces, each of which are handled by a different core/computer, and then are put back together at the end."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/2. Topic Modeling with NMF and SVD.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 2. Topic Modeling with NMF and SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Topic modeling is a great way to get started with matrix factorizations. We start with a **term-document matrix**:\n",
"\n",
"<img src=\"images/document_term.png\" alt=\"term-document matrix\" style=\"width: 80%\"/>\n",
"(source: [Introduction to Information Retrieval](http://player.slideplayer.com/15/4528582/#))\n",
"\n",
"We can decompose this into one tall thin matrix times one wide short matrix (possibly with a diagonal matrix in between).\n",
"\n",
"Notice that this representation does not take into account word order or sentence structure. It's an example of a **bag of words** approach."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider the most extreme case - reconstructing the matrix using an outer product of two vectors. Clearly, in most cases we won't be able to reconstruct the matrix exactly. But if we had one vector with the relative frequency of each vocabulary word out of the total word count, and one with the average number of words per document, then that outer product would be as close as we can get.\n",
"\n",
"Now consider increasing that matrices to two columns and two rows. The optimal decomposition would now be to cluster the documents into two groups, each of which has as different a distribution of words as possible to each other, but as similar as possible amongst the documents in the cluster. We will call those two groups \"topics\". And we would cluster the words into two groups, based on those which most frequently appear in each of the topics. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### In today's class"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll take a dataset of documents in several different categories, and find topics (consisting of groups of words) for them. Knowing the actual categories helps us evaluate if the topics we find make sense.\n",
"\n",
"We will try this with two different matrix factorizations: **Singular Value Decomposition (SVD)** and **Non-negative Matrix Factorization (NMF)**"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.datasets import fetch_20newsgroups\n",
"from sklearn import decomposition\n",
"from scipy import linalg\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"np.set_printoptions(suppress=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional Resources"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Data source](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html): Newsgroups are discussion groups on Usenet, which was popular in the 80s and 90s before the web really took off. This dataset includes 18,000 newsgroups posts with 20 topics.\n",
"- [Chris Manning's book chapter](https://nlp.stanford.edu/IR-book/pdf/18lsi.pdf) on matrix factorization and LSI \n",
"- Scikit learn [truncated SVD LSI details](http://scikit-learn.org/stable/modules/decomposition.html#lsa)\n",
"\n",
"### Other Tutorials\n",
"- [Scikit-Learn: Out-of-core classification of text documents](http://scikit-learn.org/stable/auto_examples/applications/plot_out_of_core_classification.html): uses [Reuters-21578](https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection) dataset (Reuters articles labeled with ~100 categories), HashingVectorizer\n",
"- [Text Analysis with Topic Models for the Humanities and Social Sciences](https://de.dariah.eu/tatom/index.html): uses [British and French Literature dataset](https://de.dariah.eu/tatom/datasets.html) of Jane Austen, Charlotte Bronte, Victor Hugo, and more"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scikit Learn comes with a number of built-in datasets, as well as loading utilities to load several standard external datasets. This is a [great resource](http://scikit-learn.org/stable/datasets/), and the datasets include Boston housing prices, face images, patches of forest, diabetes, breast cancer, and more. We will be using the newsgroups dataset.\n",
"\n",
"Newsgroups are discussion groups on Usenet, which was popular in the 80s and 90s before the web really took off. This dataset includes 18,000 newsgroups posts with 20 topics. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']\n",
"remove = ('headers', 'footers', 'quotes')\n",
"newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, remove=remove)\n",
"newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, remove=remove)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((2034,), (2034,))"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"newsgroups_train.filenames.shape, newsgroups_train.target.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's look at some of the data. Can you guess which category these messages are in?"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hi,\n",
"\n",
"I've noticed that if you only save a model (with all your mapping planes\n",
"positioned carefully) to a .3DS file that when you reload it after restarting\n",
"3DS, they are given a default position and orientation. But if you save\n",
"to a .PRJ file their positions/orientation are preserved. Does anyone\n",
"know why this information is not stored in the .3DS file? Nothing is\n",
"explicitly said in the manual about saving texture rules in the .PRJ file. \n",
"I'd like to be able to read the texture rule information, does anyone have \n",
"the format for the .PRJ file?\n",
"\n",
"Is the .CEL file format available from somewhere?\n",
"\n",
"Rych\n",
"\n",
"\n",
"Seems to be, barring evidence to the contrary, that Koresh was simply\n",
"another deranged fanatic who thought it neccessary to take a whole bunch of\n",
"folks with him, children and all, to satisfy his delusional mania. Jim\n",
"Jones, circa 1993.\n",
"\n",
"\n",
"Nope - fruitcakes like Koresh have been demonstrating such evil corruption\n",
"for centuries.\n",
"\n",
" >In article <1993Apr19.020359.26996@sq.sq.com>, msb@sq.sq.com (Mark Brader) \n",
"\n",
"MB> So the\n",
"MB> 1970 figure seems unlikely to actually be anything but a perijove.\n",
"\n",
"JG>Sorry, _perijoves_...I'm not used to talking this language.\n",
"\n",
"Couldn't we just say periapsis or apoapsis?\n",
"\n",
" \n"
]
}
],
"source": [
"print(\"\\n\".join(newsgroups_train.data[:3]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"hint: definition of *perijove* is the point in the orbit of a satellite of Jupiter nearest the planet's center "
]
},
{
"cell_type": "code",
"execution_count": 249,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['comp.graphics', 'talk.religion.misc', 'sci.space'], \n",
" dtype='<U18')"
]
},
"execution_count": 249,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.array(newsgroups_train.target_names)[newsgroups_train.target[:3]]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The target attribute is the integer index of the category."
]
},
{
"cell_type": "code",
"execution_count": 58,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([1, 3, 2, 0, 2, 0, 2, 1, 2, 1])"
]
},
"execution_count": 58,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"newsgroups_train.target[:10]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"num_topics, num_top_words = 6, 8"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Next, scikit learn has a method that will extract all the word counts for us."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer"
]
},
{
"cell_type": "code",
"execution_count": 342,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(2034, 26576)"
]
},
"execution_count": 342,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vectorizer = CountVectorizer(stop_words='english')\n",
"vectors = vectorizer.fit_transform(newsgroups_train.data).todense() # (documents, vocab)\n",
"vectors.shape #, vectors.nnz / vectors.shape[0], row_means.shape"
]
},
{
"cell_type": "code",
"execution_count": 343,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2034 (2034, 26576)\n"
]
}
],
"source": [
"print(len(newsgroups_train.data), vectors.shape)"
]
},
{
"cell_type": "code",
"execution_count": 303,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"vocab = np.array(vectorizer.get_feature_names())"
]
},
{
"cell_type": "code",
"execution_count": 304,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(26576,)"
]
},
"execution_count": 304,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocab.shape"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['cosmonauts', 'cosmos', 'cosponsored', 'cost', 'costa', 'costar',\n",
" 'costing', 'costly', 'costruction', 'costs', 'cosy', 'cote',\n",
" 'couched', 'couldn', 'council', 'councils', 'counsel', 'counselees',\n",
" 'counselor', 'count'], \n",
" dtype='<U80')"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vocab[7000:7020]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Singular Value Decomposition (SVD)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"SVD is not nearly as famous as it should be.\" - Gilbert Strang"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We would clearly expect that the words that appear most frequently in one topic would appear less frequently in the other - otherwise that word wouldn't make a good choice to separate out the two topics. Therefore, we expect the topics to be **orthogonal**.\n",
"\n",
"The SVD algorithm factorizes a matrix into one matrix with **orthogonal columns** and one with **orthogonal rows** (along with a diagonal matrix, which contains the **relative importance** of each factor).\n",
"\n",
"<img src=\"images/svd_fb.png\" alt=\"\" style=\"width: 80%\"/>\n",
"(source: [Facebook Research: Fast Randomized SVD](https://research.fb.com/fast-randomized-svd/))\n",
"\n",
"SVD is an **exact decomposition**, since the matrices it creates are big enough to fully cover the original matrix. SVD is extremely widely used in linear algebra, and specifically in data science, including:\n",
"\n",
"- semantic analysis\n",
"- collaborative filtering/recommendations ([winning entry for Netflix Prize](https://datajobs.com/data-science-repo/Recommender-Systems-%5BNetflix%5D.pdf))\n",
"- calculate Moore-Penrose pseudoinverse\n",
"- data compression\n",
"- principal component analysis (will be covered later in course)"
]
},
{
"cell_type": "code",
"execution_count": 344,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 27.2 s, sys: 812 ms, total: 28 s\n",
"Wall time: 27.9 s\n"
]
}
],
"source": [
"%time U, s, Vh = linalg.svd(vectors, full_matrices=False)"
]
},
{
"cell_type": "code",
"execution_count": 345,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(2034, 2034) (2034,) (2034, 26576)\n"
]
}
],
"source": [
"print(U.shape, s.shape, Vh.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confirm this is a decomposition of the input."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 346,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 346,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: confrim that U, s, Vh is a decomposition of the var Vectors\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Confirm that U, V are orthonormal"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 246,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 246,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: Confirm that U, Vh are orthonormal\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Topics"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What can we say about the singular values s?"
]
},
{
"cell_type": "code",
"execution_count": 96,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGWpJREFUeJzt3X2MHPd93/H3d/bp7ngkRYrHE0NSJhXTD3TcSAkjCVDs\nOpYdMU5qygmg0G1SthUgFFACG00RSDGSpn+wcerGTZzaTZTECOM4llkkqQgjqS1TclwDtWXKkWRR\nFMPTAy1SR/JESXw43tPufvvH/HZv73ize0fePszw8wIOO/ebmd0vZ4+f+e1vfztr7o6IiGRX1O0C\nRESkvRT0IiIZp6AXEck4Bb2ISMYp6EVEMk5BLyKScQp6EZGMU9CLiGScgl5EJOPy3S4AYN26db5l\ny5ZulyEikipPPvnka+4+1Gq7ngj6LVu2cOjQoW6XISKSKmZ2fDHbaehGRCTjFPQiIhmnoBcRyTgF\nvYhIxinoRUQyTkEvIpJxCnoRkYxLddCfOjfJp792lBfGLna7FBGRnpXqoD99fpLPPDbC8bPj3S5F\nRKRnpTroa/T95iIiyVId9GbxrYJeRCRZuoMe63YJIiI9L9VBX6MOvYhIslQHvalDLyLSUqqDvsY1\nSC8ikigTQS8iIskyEfTqz4uIJEt10Gt6pYhIa+kOek2vFBFpKdVBP0tdehGRJIsOejPLmdk/mtlX\nwu9rzexRMzsWbtc0bPugmY2Y2VEzu6sdhceP0657FhHJjqX06D8GHGn4/QHgoLtvAw6G3zGz7cBu\n4F3ATuBzZpZbnnIXpjF6EZFkiwp6M9sE/Czwpw3Nu4B9YXkfcHdD+8PuPuXuLwEjwK3LU+78utpx\nryIi2bLYHv3vA78OVBvaht19NCyfAobD8kbglYbtToS2tlGHXkQkWcugN7OfA864+5NJ23j80dQl\n5a2Z3Wdmh8zs0NjY2FJ2nb2PMOtGQzciIskW06O/A/iwmb0MPAy838z+EjhtZhsAwu2ZsP1JYHPD\n/ptC2xzu/pC773D3HUNDQ1dUvIZuRERaaxn07v6gu29y9y3Eb7I+5u6/BBwA9oTN9gCPhOUDwG4z\nK5nZVmAb8MSyV95YowZvREQS5a9i308C+83sXuA4cA+Aux82s/3Ac0AZuN/dK1dd6QLUoRcRaW1J\nQe/u3wC+EZbPAncmbLcX2HuVtS2hrk49kohI+qT6k7H1a910twwRkZ6W6qDX4I2ISGspD/qYvnhE\nRCRZqoNe0ytFRFpLddCLiEhrqQ56dehFRFpLddDXaIheRCRZqoPewiC9PhkrIpIs3UHf7QJERFIg\n1UFfo6EbEZFkqQ56Ta8UEWkt1UFfox69iEiyVAe9aZReRKSlVAd9jTr0IiLJUh309atXauxGRCRR\nqoNeRERay0TQqz8vIpIs1UGv6ZUiIq2lOujr1KUXEUmU6qDXtW5ERFpLd9B3uwARkRRIddDXaHal\niEiyVAe93owVEWkt1UFfow69iEiyVAe9rnUjItJaqoO+RmP0IiLJUh309WvdaPBGRCRRuoO+2wWI\niKRAqoO+RkM3IiLJ0h306tKLiLSU7qAP1KEXEUmW6qDX9EoRkdZSHfR1GqQXEUmU6qCfnV4pIiJJ\n0h303S5ARCQFUh30NRq5ERFJluqgN12+UkSkpZZBb2Z9ZvaEmT1tZofN7D+H9rVm9qiZHQu3axr2\nedDMRszsqJnd1c5/AICrSy8ikmgxPfop4P3u/qPAzcBOM7sdeAA46O7bgIPhd8xsO7AbeBewE/ic\nmeXaUXytP6+YFxFJ1jLoPXYx/FoIPw7sAvaF9n3A3WF5F/Cwu0+5+0vACHDrslYdaORGRKS1RY3R\nm1nOzJ4CzgCPuvt3gGF3Hw2bnAKGw/JG4JWG3U+Etvn3eZ+ZHTKzQ2NjY1f8DwC9GSsi0syigt7d\nK+5+M7AJuNXMfmTeemeJIyju/pC773D3HUNDQ0vZtU6fjBURaW1Js27c/U3gceKx99NmtgEg3J4J\nm50ENjfstim0tY069CIiyRYz62bIzK4Ly/3AB4HngQPAnrDZHuCRsHwA2G1mJTPbCmwDnljuwuPi\n2nKvIiKZkl/ENhuAfWHmTATsd/evmNn/A/ab2b3AceAeAHc/bGb7geeAMnC/u1faUXz9EggapBcR\nSdQy6N39GeCWBdrPAncm7LMX2HvV1bUQhaRXzouIJEv3J2PDbVVJLyKSKNVBX+/Rd7kOEZFeluqg\nr43Rq0cvIpIsE0GvnBcRSZbqoJ99M1ZJLyKSJBNBX1XOi4gkSnXQa9aNiEhr6Q56jdGLiLSU8qA3\nzDRGLyLSTKqDHuJxeo3Ri4gkS33QGxqjFxFpJvVBH5npk7EiIk2kPujN1KMXEWkmE0GvnBcRSZb6\noI/MNOtGRKSJTAS9Zt2IiCRLfdBr1o2ISHPpD3qN0YuINJX6oI8ijdGLiDST+qCPh266XYWISO9K\nfdDHH5hS0ouIJEl90JsZlWq3qxAR6V2pD/p8ZFSqSnoRkSSpD/pcZJQ1SC8ikij1QZ/PGRUFvYhI\novQHvXr0IiJNZSDoIyoVBb2ISJLUB73G6EVEmkt90Mdj9Jp1IyKSJPVBrx69iEhz6Q96M8oaoxcR\nSZT+oI+Mii5qJiKSKBNBX9XQjYhIokwEvXr0IiLJUh/0kalHLyLSTOqDXp+MFRFprmXQm9lmM3vc\nzJ4zs8Nm9rHQvtbMHjWzY+F2TcM+D5rZiJkdNbO72voPiHStGxGRZhbToy8Dv+bu24HbgfvNbDvw\nAHDQ3bcBB8PvhHW7gXcBO4HPmVmuHcVDPL1SXw4uIpKsZdC7+6i7fy8sXwCOABuBXcC+sNk+4O6w\nvAt42N2n3P0lYAS4dbkLr8np6pUiIk0taYzezLYAtwDfAYbdfTSsOgUMh+WNwCsNu50IbW2RMwW9\niEgziw56MxsE/hr4uLufb1zn7g5L++JWM7vPzA6Z2aGxsbGl7DqHpleKiDS3qKA3swJxyH/R3f8m\nNJ82sw1h/QbgTGg/CWxu2H1TaJvD3R9y9x3uvmNoaOhK6w/TK694dxGRzFvMrBsD/gw44u6fblh1\nANgTlvcAjzS07zazkpltBbYBTyxfyXPlNetGRKSp/CK2uQP4ZeD7ZvZUaPsN4JPAfjO7FzgO3APg\n7ofNbD/wHPGMnfvdvbLslQeR5tGLiDTVMujd/VuAJay+M2GfvcDeq6hr0XIRml4pItJE6j8Zq1k3\nIiLNpT/oo0jXuhERaSIDQY/G6EVEmkh90EeaRy8i0lTqgz6nyxSLiDSV+qDPq0cvItJU6oM+igx3\n1KsXEUmQ+qDPWTzFX716EZGFpT7ooygEvXr0IiILSn3Q5xX0IiJNpT7ocyHoNZdeRGRhqQ/6Qi7+\nJ6hHLyKysNQH/WyPXhelFxFZSOqDXmP0IiLNpT7o6z36ioJeRGQhqQ/62hi93owVEVlY6oM+Vx+6\n0Ri9iMhCUh/0eU2vFBFpKvVBrzF6EZHmUh/0+Zx69CIizaQ/6KPaB6Y0Ri8ispAMBL2GbkREmkl9\n0OtaNyIizaU+6DVGLyLSXPqDXmP0IiJNpT7oNb1SRKS51Ae9hm5ERJpLfdCvKOYBOD8x0+VKRER6\nU+qDfsPqPgBGz012uRIRkd6U+qDP5yIGijnGp8rdLkVEpCelPugBVpTyjE8r6EVEFpKJoB8s5bk4\nVel2GSIiPSkTQb+ipKEbEZEk2Qj6Yp6LCnoRkQVlIuhvWN3HS6+Nd7sMEZGelImg37SmnzfGp7td\nhohIT8pE0A8U85SrznRZ17sREZmvZdCb2efN7IyZPdvQttbMHjWzY+F2TcO6B81sxMyOmtld7Sq8\n0UAxB8AlTbEUEbnMYnr0fw7snNf2AHDQ3bcBB8PvmNl2YDfwrrDP58wst2zVJqhdBmF8WlMsRUTm\naxn07v5N4PV5zbuAfWF5H3B3Q/vD7j7l7i8BI8Cty1Rrov7Qo59Qj15E5DJXOkY/7O6jYfkUMByW\nNwKvNGx3IrRdxszuM7NDZnZobGzsCsuIrSjFQT+uD02JiFzmqt+MdXcHlnyNYHd/yN13uPuOoaGh\nq6phoD50ox69iMh8Vxr0p81sA0C4PRPaTwKbG7bbFNraqv5mrHr0IiKXudKgPwDsCct7gEca2neb\nWcnMtgLbgCeursTWaj36SzMKehGR+fKtNjCzLwHvA9aZ2QngPwGfBPab2b3AceAeAHc/bGb7geeA\nMnC/u7c9fWtj9Jd0GQQRkcu0DHp3/2jCqjsTtt8L7L2aopZqoKDplSIiSTLxyVhNrxQRSZaJoC/m\nI4q5iAsauhERuUwmgh5gw3V9vPqmvjdWRGS+zAT9jWsH+MFZXapYRGS+TAX98dcvdbsMEZGek5mg\nX7+yjzcvzVCpLvlDuiIimZaZoNelikVEFpadoK99aEpz6UVE5shM0K8bLAHwsr47VkRkjswE/R1v\nXUcxF/HY82dabywicg3JTNAPlvLcNLSCF8YudrsUEZGekpmgB9i0ZoAfaIqliMgcmQr6G9cO8Mrr\nE8TfhSIiIpCxoH/nhpVMzFR45sS5bpciItIzMhX0d7x1HQCPH9UbsiIiNZkK+h+6rp9t6wd5+pU3\nu12KiEjPyFTQA7xn2xD/99hrjJ6b6HYpIiI9IXNB/2/v2IIDnzk40u1SRER6QuaCfvPaAX7pthv5\n8nd/wOFX9aasiEjmgh7g4x94G4OlPL/6pX9kqqxr34jItS2TQb9mRZH//os38+LYOJ9+9J+6XY6I\nSFdlMugB7nznMPfs2MQf/8OLfOqrz+tDVCJyzcp3u4B22vuRd2MYn338BSIzPnbnNvK5zJ7bREQW\nlOmgL+Qifufn381MpcofPjbC0VMX+MxHb6GvkOt2aSIiHZP57m0UGb93z4/yGx96B1977jT3feFJ\nXn1Tc+xF5NqR+aAHMDPue+8P818+8m6+/eJZfuq/fYNPffV5LkzOdLs0EZG2uyaCvuZf3nYjj/3a\nP2fnj9zAZx9/gfd96ht84dvHmalUu12aiEjbXFNBD/E16/9g9y0c+JU7+OH1g/zm/36Wu37/mxx4\n+lXGp/TF4iKSPdYL0w537Njhhw4d6vjjujtfP3KG3/n7I7w4Nk4pH/HB7cP84k9s5vabrqegGToi\n0sPM7El339Fqu0zPumnFzPjg9mF+6u1DPPHy6/yfZ0/xyFOv8pVnRukv5Hj/O9Zz201ruW3r9bx1\n/SC5yLpdsojIkl3TPfqFTM5UePz5M3xr5DW+evg0r12cAmBFMcc7NqziHTes5J0bVvHODat4+w0r\nGSxd0+dKEemixfboFfRNuDsn3pjgiZde55kTb3Jk9AJHTp3nwuTsWP5brh/gbcMr2bxmgE1r+sPP\nABvX9LO6v9DF6kUk6zR0swzMjM1rB9i8doBf+PFNQBz+J9+c4PnRCxwZPc+RU+c5dvoi3zr2GhMz\ncy+gtrKU54bVfdywuo/hVX0MryoxvKqP9Sv7WL+qxPqVJdYNlvQBLhFpKwX9EpkZm9YMsGnNAB/Y\nPlxvd3deH5/m5JsTnHhjghNvXOLkGxOcOj/JqXOTHDt9kbGLU1Sql7+CGijmWLuiyPUrilw/WKov\nXzdQZM1AgdX9BVbXbvsLrOovMFjME+k9AxFZBAX9MjEzrh8scf1giX+26boFt6lU45PB6fOTnLkw\nyZnzU5wdn+b18WnOXoyXT5+f5Mjoec6OTzNdTp7fHxmsKOVZ1VdgsJRnsC/PilKelaU8g6V4eUUp\nR38xx0Ahx0AxHy8XQ1sxHy8XZtv6CznMdPIQyRoFfQflImNoZYmhlSVgddNt3Z2JmQpvXJrh3KUZ\nzk3McG5imvMTZc5Pxr9fmCxzcarMxckyF6bitpNvXGJ8qsLFqTLj02WW+hZMfyEO/b58RF8hRzEf\nUchFFPMRxVxEIdyW8hGFnMXt+YhiLkchb5TCtvV9wnIp7Dd/XXGB+y7ma/cfaaaTyDJoW9Cb2U7g\nD4Ac8Kfu/sl2PVYWmVnodefZeF3/Fd2HuzNVrnJpusKl6TIT05WwXGFipjy7XL+N2ybLFSZnqkzO\nVJguV5mpVJmuVJkuV7k0EbdNlyvMVDxeDutqt8spF1l8EsgZxXxugRPM7Imj1OSkVDvB5HNGPjLy\nUbyci4xCFJ9Q4nXxciGsq22Xj4x8LiJnVt82srg9F9W2nV2u7RtFxLeGXi1J17Ql6M0sB3wW+CBw\nAviumR1w9+fa8XiyMDOjr5CjrxC/B9AJ7s5MxeOTwwIngXp7ucpUpcrMAuumytWGk0jtZBOftObc\nRyVenipXuTBZ5uy8k1JtXW3bbk8wm3NCMCMXTiD1E0bOZk8kUUS04Mlj7nJkFk5YETmDXBTF7Qvs\nG5mRiwiPEZGL4ov+1R4zqt3W26i3zVlfb2tYb2G/xvVmRBFz1897jKhejy1ci06Qy6JdPfpbgRF3\nfxHAzB4GdgEK+owzM4r5uMe9otTtauYqV6qUq0656lQqTrk6+3ttXaUan6Ti2/j3crVKOWxfqUIl\n3JarVarulMN2FQ/bVzxuD/dXqT1mw/7lqlOtzt2mvp3X6mu8n2q9tomZJvs2LMe1Veu1Ves1d/uZ\nWJrIZk+SjSeU+SeP2RMSc06U9ZPevO0aT5j5XESh9qquvhy/mitEs68GC+FVXyFsl4+M/mKOFcX4\nfbGBYo7BUp6BUo61A8We+f6LdgX9RuCVht9PALe16bFEFiX+j9vtKrrP3al6PDmg6t5wIvA5J4XL\n22ZPZrX1jfcxZ717OKk1rPfZE1F9fcNjXF4Llz1uZd62s/cZTsBOOAnGJ9XaiXK2zZkqV6h4OOFW\nZk+MM5XZE/pMJT5JzlScmeqVvRpc3V9g/crWvZ33vX2IT/zs9it4Jheva2/Gmtl9wH0AN954Y7fK\nELnmmFkY5tGQyGLVXk3VXv3NVGZf6U3MVBifit/fujhV5tJ0PEHi6RPnuDTd+kKJw6v62l5/u4L+\nJLC54fdNoa3O3R8CHoL4k7FtqkNE5KrFwzxLezn4y22q5Uq0awDpu8A2M9tqZkVgN3CgTY8lIiJN\ntKVH7+5lM/sV4KvE0ys/7+6H2/FYIiLSXNvG6N3974C/a9f9i4jI4vTG3B8REWkbBb2ISMYp6EVE\nMk5BLyKScQp6EZGM64mvEjSzMeD4VdzFOuC1ZSpnuaimxevFunqxJujNunqxJujNupa7pre4+1Cr\njXoi6K+WmR1azPcmdpJqWrxerKsXa4LerKsXa4LerKtbNWnoRkQk4xT0IiIZl5Wgf6jbBSxANS1e\nL9bVizVBb9bVizVBb9bVlZoyMUYvIiLJstKjFxGRBKkOejPbaWZHzWzEzB7o4ONuNrPHzew5Mzts\nZh8L7b9tZifN7Knw86GGfR4MdR41s7vaWNvLZvb98PiHQttaM3vUzI6F2zWdqsvM3t5wPJ4ys/Nm\n9vFuHCsz+7yZnTGzZxvalnxszOzHwzEeMbPP2FV8qWlCTZ8ys+fN7Bkz+1szuy60bzGziYZj9kcd\nrGnJz9dy1tSkri831PSymT0V2jt1rJKyoKt/V5dx91T+EF/++AXgJqAIPA1s79BjbwB+LCyvBP4J\n2A78NvAfF9h+e6ivBGwNdefaVNvLwLp5bf8VeCAsPwD8bqfranjOTgFv6caxAt4L/Bjw7NUcG+AJ\n4HbAgL8HfmaZa/ppIB+Wf7ehpi2N2827n3bXtOTnazlrSqpr3vrfA36rw8cqKQu6+nc1/yfNPfr6\nF5C7+zRQ+wLytnP3UXf/Xli+ABwh/p7cJLuAh919yt1fAkaI6++UXcC+sLwPuLtLdd0JvODuzT4c\n17aa3P2bwOsLPN6ij42ZbQBWufu3Pf7f+RcN+yxLTe7+NXevfQfdt4m/oS1RJ2pqoiPHqVVdofd7\nD/ClZvfRhmOVlAVd/buaL81Bv9AXkDcL27Ywsy3ALcB3QtOvhpfcn294udbJWh34upk9afH38gIM\nu/toWD4FDHehLoi/aazxP2K3jxUs/dhsDMudqu/fEffuaraGoYh/MLP3NNTaiZqW8nx1+ji9Bzjt\n7sca2jp6rOZlQU/9XaU56LvOzAaBvwY+7u7ngf9JPJR0MzBK/FKy037S3W8Gfga438ze27gy9BY6\nPtXK4q+U/DDwv0JTLxyrObp1bJKY2SeAMvDF0DQK3Bie3/8A/JWZrepQOT33fM3zUeZ2Ijp6rBbI\ngrpe+LtKc9C3/ALydjKzAvET+0V3/xsAdz/t7hV3rwJ/wuyQQ8dqdfeT4fYM8LehhtPhpWHtpeuZ\nTtdFfOL5nrufDvV1/VgFSz02J5k7lNKW+szs3wA/B/yrEBSEl/tnw/KTxOO7b+tETVfwfHXkOAGY\nWR74eeDLDfV27FgtlAX02N9VmoO+a19AHsYD/ww44u6fbmjf0LDZR4Da7IADwG4zK5nZVmAb8Rsv\ny13XCjNbWVsmflPv2fD4e8Jme4BHOllXMKfH1e1j1WBJxya8HD9vZreHv4N/3bDPsjCzncCvAx92\n90sN7UNmlgvLN4WaXuxQTUt6vjpRU4MPAM+7e33oo1PHKikL6LW/q+V6V7cbP8CHiN/lfgH4RAcf\n9yeJX4o9AzwVfj4EfAH4fmg/AGxo2OcToc6jLOO76fPquon4Hf2ngcO1YwJcDxwEjgFfB9Z2uK4V\nwFlgdUNbx48V8YlmFJghHgO990qODbCDOOheAP4H4YOHy1jTCPE4bu1v64/Ctr8QntengO8B/6KD\nNS35+VrOmpLqCu1/Dvz7edt26lglZUFX/67m/+iTsSIiGZfmoRsREVkEBb2ISMYp6EVEMk5BLyKS\ncQp6EZGMU9CLiGScgl5EJOMU9CIiGff/AQqkXAuonCReAAAAAElFTkSuQmCC\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7fcadb741128>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(s);"
]
},
{
"cell_type": "code",
"execution_count": 97,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7fcada6c6828>]"
]
},
"execution_count": 97,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD9CAYAAACyYrxEAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl01fWd//HnOzd7gLAkhCQ3FZAogqyJSBUc616xgAqW\nThdnnFZnxrHaX8+vp3bmTO2ZX6edni62ndFWbXvsatlESt23wZ0m7AGRsCgJARK2AIGs798f96LB\nqrmBJN+be1+Pc3LyvZ/7/Sbv3AOv+72f7+f7+Zi7IyIiiSsl6AJERKR3KehFRBKcgl5EJMEp6EVE\nEpyCXkQkwSnoRUQSXMxBb2YhM1tjZiuij+8xs1ozWxv9urbTvnebWbWZbTGzq3ujcBERiU1qN/a9\nE9gMDOrU9iN3/37nncxsHLAAGA8UAc+a2Tnu3n6mxYqISPfFdEZvZmFgFvBQDLvPAR5x92Z33wFU\nA9NOv0QRETkTsXbd3At8Deh4X/sdZrbezH5pZkOibcXArk771ETbREQkAF123ZjZdcA+d680s0s7\nPXU/8B+AR7//ALgl1l9sZrcCtwLk5OSUjR07thtli4hIZWVlg7vnd7VfLH30FwOzoxdbM4FBZvZb\nd//cyR3M7EFgRfRhLVDS6fhwtO0U7v4A8ABAeXm5V1RUxFCKiIicZGZvx7Jfl1037n63u4fdfSSR\ni6zPu/vnzKyw027XAxuj28uBBWaWYWajgFJgVbeqFxGRHtOdUTfv9z0zm0yk62YncBuAu1eZ2UJg\nE9AG3K4RNyIiwbF4mKZYXTciIt1nZpXuXt7VfrozVkQkwSnoRUQSnIJeRCTBKehFRBJcvw763YeO\n85+Pb6bhaHPQpYiIxK1+HfRHm9t4YOV2lq35q/uxREQkql8H/TkFA5kUzmVRRQ3xMExURCQe9eug\nB5hXXsKWvUfYUHs46FJEROJSvw/62ROLSE9NYVFFTdCliIjEpX4f9LnZaVw9fgSPra3lRKtmWhAR\neb9+H/QA88vCNJ5o45lNe4MuRUQk7iRE0F88Jo/C3EwWVar7RkTk/RIi6EMpxryyMC9trafu8PGg\nyxERiSsJEfQA88rCuMPS1RpTLyLSWcIE/VnDcpg2aiiLKnZpTL2ISCcJE/QQuSi7c38TFW8fDLoU\nEZG4kVBBf+2EQrLTQyyq2BV0KSIicSOhgj4nI5VZEwr58/o6mlragi5HRCQuJFTQA8wvL+FYSzuP\nb9gTdCkiInEh4YL+gpFDGDksW903IiJRCRf0ZpEx9W/sOMA7+5uCLkdEJHAJF/QAN0wNYwaLK3VW\nLyKSkEFfNDiLGWPyWLK6lo4OjakXkeQWc9CbWcjM1pjZiujjoWb2jJltjX4f0mnfu82s2sy2mNnV\nvVF4V+aXl1B76DivbtsfxK8XEYkb3TmjvxPY3Onx14Hn3L0UeC76GDMbBywAxgPXAPeZWahnyo3d\nVeMKGJSZyiJ134hIkosp6M0sDMwCHurUPAd4OLr9MDC3U/sj7t7s7juAamBaz5Qbu8y0ELMnF/Hk\nxj0cPt7a179eRCRuxHpGfy/wNaCjU1uBu9dFt/cABdHtYqDzaXRNtO0UZnarmVWYWUV9fX33qo7R\n/LISmts6WLF+d6/8fBGR/qDLoDez64B97l75Yft4ZBaxbl31dPcH3L3c3cvz8/O7c2jMJoZzOadg\nAAu1zKCIJLFYzugvBmab2U7gEeAyM/stsNfMCgGi3/dF968FSjodH4629TkzY35ZCet2HWLr3iNB\nlCAiErgug97d73b3sLuPJHKR9Xl3/xywHLg5utvNwGPR7eXAAjPLMLNRQCmwqscrj9HcKcWEUkyr\nT4lI0jqTcfTfBa40s63AFdHHuHsVsBDYBDwJ3O7uga3anT8wg8vGDmfp6lpa2zu6PkBEJMF0K+jd\n/UV3vy66vd/dL3f3Une/wt0PdNrv2+5+truf6+5P9HTR3TW/LEzD0Wb+d0vvXPQVEYlnCXln7Pt9\nYuxw8gaka0y9iCSlpAj6tFAKcycX89zmfew/2hx0OSIifSopgh4iUyK0dTjL1mpMvYgkl6QJ+nNH\nDGRiOFeLh4tI0kmaoIfIRdk39xyhandj0KWIiPSZpAr62ZOKSU9N0epTIpJUkiroc7PTuGpcAY+t\n201zW2BD+0VE+lRSBT1ELsoeamrl2U37ut5ZRCQBJF3QzxiTR2FupsbUi0jSSLqgD6UYN0wtZuVb\n9ew5fCLockREel3SBT3AvLISOhyWrtFEZyKS+JIy6Efl5XDByCEsrqjRmHoRSXhJGfQQWX1qe8Mx\nVr9zMOhSRER6VdIG/bUTC8lKC7FIq0+JSIJL2qAfkJHKtRMKWbG+jqaWtqDLERHpNUkb9AA3lYc5\n2tzGkxv3BF2KiEivSeqgnzZqKGcNy1b3jYgktKQOejNj3tQwr23fz64DTUGXIyLSK5I66AFuLAtj\nBou1eLiIJKikD/qiwVnMGJPH4soaOjo0pl5EEk/SBz3AvLIwtYeO89r2/UGXIiLS4xT0wNXjRzAw\nM1Xz1ItIQuoy6M0s08xWmdk6M6sys29F2+8xs1ozWxv9urbTMXebWbWZbTGzq3vzD+gJmWkhZk8q\n4omNe2g80Rp0OSIiPSqWM/pm4DJ3nwRMBq4xs+nR537k7pOjX48DmNk4YAEwHrgGuM/MQr1Qe4+a\nX15Cc1sHK9bVBV2KiEiP6jLoPeJo9GFa9OujrlrOAR5x92Z33wFUA9POuNJeNimcS+nwAZqnXkQS\nTkx99GYWMrO1wD7gGXd/I/rUHWa23sx+aWZDom3FQOe0rIm2xTUzY355mDXvHKJ635GgyxER6TEx\nBb27t7v7ZCAMTDOz84H7gdFEunPqgB905xeb2a1mVmFmFfX19d0su3fMnVJMKMVYpDH1IpJAujXq\nxt0PAS8A17j73ugbQAfwIO91z9QCJZ0OC0fb3v+zHnD3cncvz8/PP73qe9jwgZl84tx8lq6upa29\nI+hyRER6RCyjbvLNbHB0Owu4EnjTzAo77XY9sDG6vRxYYGYZZjYKKAVW9WzZvWdeWQn1R5pZuTU+\nPmWIiJyp1Bj2KQQejo6cSQEWuvsKM/uNmU0mcmF2J3AbgLtXmdlCYBPQBtzu7u29Un0vuGzscIbm\npLOooobLxhYEXY6IyBnrMujdfT0w5QPaP/8Rx3wb+PaZlRaM9NQU5k4u5jev7+TAsRaG5qQHXZKI\nyBnRnbEfYH55mNZ257G1f3VpQUSk31HQf4DzCgdxfvEgzVMvIglBQf8hbiovYVNdI1W7DwddiojI\nGVHQf4jZk4pID6XorF5E+j0F/YcYnJ3OleMLeGxtLS1tGlMvIv2Xgv4jzC8Lc7Cplec27w26FBGR\n06ag/wgzS/MZMShTUyKISL+moP8IoRTjhqnFvLhlH/saTwRdjojIaVHQd2FeWZgOh6VrNKZeRPon\nBX0XRucPoPysISyq2IW7Fg8Xkf5HQR+D+eVhttUfY82uQ0GXIiLSbQr6GMyaWERWWkhj6kWkX1LQ\nx2BARiqfnDCCFet2c7yl30zEKSICKOhjNr+shCPNbTxVtSfoUkREukVBH6MLRw2lZGgWCyu0eLiI\n9C8K+hilpBjzppbw6rb97DrQFHQ5IiIxU9B3w41lxZjBktW6KCsi/YeCvhvCQ7K56OxhLK6soaND\nY+pFpH9Q0HfT/LISag4e5/Ud+4MuRUQkJgr6brp6/AgGZqSyWGPqRaSfUNB3U1Z6iE9NLuLxjXUc\nOdEadDkiIl1S0J+G+WVhTrR28Of1dUGXIiLSpS6D3swyzWyVma0zsyoz+1a0faiZPWNmW6Pfh3Q6\n5m4zqzazLWZ2dW/+AUGYXDKYMcMHaJ56EekXYjmjbwYuc/dJwGTgGjObDnwdeM7dS4Hnoo8xs3HA\nAmA8cA1wn5mFeqP4oJgZ88vCVL59kG31R4MuR0TkI3UZ9B5xMs3Sol8OzAEejrY/DMyNbs8BHnH3\nZnffAVQD03q06jhw/dRiQinGYp3Vi0ici6mP3sxCZrYW2Ac84+5vAAXufrKTeg9QEN0uBjrPE1AT\nbUsowwdmcuk5+SxdXUO7xtSLSByLKejdvd3dJwNhYJqZnf++553IWX7MzOxWM6sws4r6+vruHBo3\n5peH2dvYzMqt/bN+EUkO3Rp14+6HgBeI9L3vNbNCgOj3fdHdaoGSToeFo23v/1kPuHu5u5fn5+ef\nTu2Bu2xsAUNz0jWmXkTiWiyjbvLNbHB0Owu4EngTWA7cHN3tZuCx6PZyYIGZZZjZKKAUWNXThceD\n9NQU5kwu4plNeznU1BJ0OSIiHyiWM/pC4AUzWw/8hUgf/Qrgu8CVZrYVuCL6GHevAhYCm4Angdvd\nPWFX65hfVkJLewePrd0ddCkiIh/I4mHB6/Lycq+oqAi6jNM26ycvYQYr7pgZdCkikkTMrNLdy7va\nT3fG9oAFF5SwsbaRHz69hXh44xQR6Sw16AISwd9eeBZVuxv5yfPVNJ5o49+vG0dKigVdlogIoKDv\nEaEU4zs3TGBARioPvbyDo81tfPeGCaSG9IFJRIKnoO8hZsa/zjqPgZlp/OjZtzjW3Ma9CyaTkZpQ\nsz+ISD+kU84eZGbceUUp/37dOJ7YuIcvPlxBU0tb0GWJSJJT0PeCW2aM4ns3TuSV6ga+8ItVNGre\nehEJkIK+l9x0QQk//cxU1tUc4jMPvM7+o81BlyQiSUpB34tmTSzkwS+Us63+KDf9/DXqDh8PuiQR\nSUIK+l526bnD+fUtF7K3sZl597/GzoZjQZckIklGQd8Hpo0ayh++NJ2mljbm//w1tuw5EnRJIpJE\nFPR9ZEI4l4W3fZwUg08/8Bprdx0KuiQRSRIK+j5UWjCQxf94EYMy0/jsg6/z2rb9QZckIklAQd/H\nSoZms+gfP07R4Cxu/tUqntu8N+iSRCTBKegDUDAokz/e9nHGjhjIbb+p5LG1f7Uui4hIj1HQB2Ro\nTjq/++KFTD1rCHf9cS2/f+OdoEsSkQSloA/QwMw0fn3LNC49J59vPLqBn//vtqBLEpEEpKAPWGZa\niJ9/vpzrJhbynSfe5PtPaU57EelZmr0yDqSnpvDjBVMYkJHKf79QzdFmzWkvIj1HQR8nTs5pPzAz\nlQdf2sGRE238142a015EzpyCPo6YGd+4NjKn/Q+ficxp/+PPaE57ETkzOl2MM2bGly+PzGn/ZJXm\ntBeRM6egj1O3zBjF9+ZF5rT//C9Wcfi45rQXkdPTZdCbWYmZvWBmm8ysyszujLbfY2a1ZrY2+nVt\np2PuNrNqM9tiZlf35h+QyG4qL+G//3Yq66Nz2jdoTnsROQ2xnNG3AV9193HAdOB2MxsXfe5H7j45\n+vU4QPS5BcB44BrgPjNTJ/NpunZCZE777Q2ROe13H9Kc9iLSPV0GvbvXufvq6PYRYDNQ/BGHzAEe\ncfdmd98BVAPTeqLYZHVyTvv6xmbm/+w1dmhOexHphm710ZvZSGAK8Ea06Q4zW29mvzSzIdG2YmBX\np8Nq+Og3BonBtFFD+cOt0zne2s78n73Gm3sagy5JRPqJmIPezAYAS4C73L0RuB8YDUwG6oAfdOcX\nm9mtZlZhZhX19fXdOTRpnV+cy8LbphNKgU///HXWvHMw6JJEpB+IKejNLI1IyP/O3ZcCuPted293\n9w7gQd7rnqkFSjodHo62ncLdH3D3cncvz8/PP5O/IamMGR6Z0z43K43PPvQGr25rCLokEYlzsYy6\nMeAXwGZ3/2Gn9sJOu10PbIxuLwcWmFmGmY0CSoFVPVeynJzTPjwki7/71V94dpPmtBeRDxfLGf3F\nwOeBy943lPJ7ZrbBzNYDnwC+AuDuVcBCYBPwJHC7u7f3TvnJq2BQJn+8NTqn/W81p72IfDiLh5kS\ny8vLvaKiIugy+qUjJ1r54sMVrNp5gP8393w+e+FZQZckIn3EzCrdvbyr/XRnbD83MDONh2+ZxifO\nHc6/PrqRHzy9hcNNuotWRN6joE8AmWkhfva5MuZMLuKnz1cz7T+f5c5H1vBKdQMdHcF/YhORYKnr\nJoG4O1W7G1lYsYtla2ppPNFG8eAs5peHmVcWJjwkO+gSRaQHxdp1o6BPUCda23l6014W/mUXr0SH\nYM4Yk8f88hKuGldAZppmpRDp7xT08q5dB5pYsrqGRRU11B46zqDMVOZOKeam8hLOL84NujwROU0K\nevkrHR3Oa9v3s7BiF09s3ENLWwfjCgdxU3mYOZOLGZKTHnSJItINCnr5SIebWlm+rpaFFTVsqD1M\neiiFK8cX8OnyEi4ek0dI69WKxD0FvcRs08kLuGtrOdTUSlFuJvPKwswrK+Fjw3QBVyReKeil25rb\n2nl20z4WVuxi5dZ63OGis4dxU3kJ15w/QhdwReKMgl7OyO5Dx1lSWcOiyhreOdDEwMxUZk8q4tMX\nlDChOJfIFEgiEiQFvfSIjg7njR0HWFSxi8c31nGitYOxIwYyv7yEuZOLGDYgI+gSRZKWgl56XOOJ\nVv60bjcLK2pYt+sQaSHjivMKuKm8hEvOydcFXJE+pqCXXrVlzxEWVexi6ZpaDhxrYcSgTG4sK2Z+\nWQkj83KCLk8kKSjopU+0tHXw/Jt7WVhRw4tb9tHhkWUPvzRzNFeOKwi6PJGEpqCXPre38QRLVtfw\nx7/s4u39TdwwpZhvzh5PblZa0KWJJCRNUyx9rmBQJv986Rie/T9/w52Xl/LYut188t6VvFKt5Q5F\ngqSglx6XFkrhK1eew5J/uojMtBCffegNvvWnKk60aqExkSAo6KXXTC4ZzJ+/PJO/u2gkv3plJ7N+\n8hLraw4FXZZI0lHQS6/KSg9xz+zx/OYfpnGsuZ0b7nuVHz+7ldb2jqBLE0kaCnrpEzNL83nqrku4\nbmIhP3r2Lebd/yrb6o8GXZZIUlDQS5/JzU7j3gVT+J+/ncrbB5qY9ZOXePjVnVruUKSXKeilz82a\nWMjTd13C9NHD+ObyKr7wy1XUHT4edFkiCUtBL4EYPiiTX/3dBXz7+vOpfPsgV/1oJcvW1BIP93WI\nJJoug97MSszsBTPbZGZVZnZntH2omT1jZluj34d0OuZuM6s2sy1mdnVv/gHSf5kZn73wLJ64cybn\nFAzkrj+u5V9+v4aDx1qCLk0kocRyRt8GfNXdxwHTgdvNbBzwdeA5dy8Fnos+JvrcAmA8cA1wn5lp\nInP5UCPzclh428f5v1efy9Ob9nDVvSt5Ycu+oMsSSRhdBr2717n76uj2EWAzUAzMAR6O7vYwMDe6\nPQd4xN2b3X0HUA1M6+nCJbGEUozbPzGGZbdfzNDsdP7+V3/hG49u4FhzW9ClifR73eqjN7ORwBTg\nDaDA3euiT+0BTs5gVQzs6nRYTbRNpEvji3J57F8u5tZLRvOHVe9w7U9eovLtA0GXJdKvxRz0ZjYA\nWALc5e6NnZ/zyBW0bl1FM7NbzazCzCrq6+u7c6gkuMy0EN+49jwe+dJ02juc+T97je89+SYtbbrJ\nSuR0xBT0ZpZGJOR/5+5Lo817zaww+nwhcLJTtRYo6XR4ONp2Cnd/wN3L3b08Pz//dOuXBHbh6GE8\ncedM5peVcN+L25jzP6+wZc+RoMsS6XdiGXVjwC+Aze7+w05PLQdujm7fDDzWqX2BmWWY2SigFFjV\ncyVLMhmYmcZ/zZvIg18op/7ICT7105d5YOU22nWTlUjMYjmjvxj4PHCZma2Nfl0LfBe40sy2AldE\nH+PuVcBCYBPwJHC7u2vaQjkjV44r4Km7LuHSc/P5z8ff5DMPvs6uA01BlyXSL2jhEelX3J0lq2u5\nZ3kV7s43PzWe+eVhIh88RZKLFh6RhGRmzCsL8+RdM5kQzuVrS9bzpV9XUn+kOejSROKWgl76pfCQ\nbH7/xen826zzWLm1nmvuXclTVXuCLkskLinopd9KSTG+OHM0K+6YwYjcTG77TSVfXbiOxhOtQZcm\nElcU9NLvnVMwkEf/+WLuuGwMj66p4ZP3vsRr2/YHXZZI3FDQS0JIT03hq1edy+J/uoj01BQ+8+Dr\n/MeKTZpCQQSNupEE1NTSxncef5PfvP42aSGj7KwhzCzNZ8aYPM4vziWUohE6khhiHXWjoJeEVfn2\nQZ6q2sNLWxvYXBeZtWNwdhoXnT3s3eAvGZodcJUipy/WoE/ti2JEglB21hDKzoosk1B/pJlXtzWw\n8q0GXq6u5/ENkRE6I4dlM6M0j5ml+Xz87GEMykwLsmSRXqEzekk67k71vqO8tLWBl6sbeH37fppa\n2gmlGJPCucwozWdmaR6TSwaTFtJlLIlf6roRiVFLWwer3znIy1sbeKm6gQ01h+hwGJCRyvTRw5hZ\nmseM0jxG5+XoDlyJKwp6kdN0uKmVV7dFQv+lrfXsOhBZuLwoNzPSt1+ax8Vj8hiakx5wpZLsFPQi\nPeTt/cci3TxbG3h1WwONJ9owg/FFg5gxJp9LSvMoGzmEjFStmCl9S0Ev0gva2jtYX3uYl6PBv/qd\ng7R1OJlpKUwbNYyZY/KYeU4e5xYMVDeP9DoFvUgfONrcxhvb9/PS1kg3z7b6YwDkD8xgxpg8ZozJ\nY2ZpHsMHZQZcqSQiDa8U6QMDMlK5/LwCLj8vsmRy3eHj73bzrHyrnkfXRBZXu2DkEG6YGmbWxEIN\n4ZQ+pzN6kV7S0eFs3tPIi1vqWbq6hm31x8hITeGq8SO4cWoxM0vzdZeunBF13YjEEXdnXc1hlq6u\nYfm63RxqamX4wAyun1LMDVPDnDtiYNAlSj+koBeJU81t7bzw5j4WV9by4pZ9tHU45xcP4sapYWZP\nKmLYgIygS5R+QkEv0g80HG1m+drdLFldQ9XuRlJTjEvPHc68smIuG1tAeqruzJUPp6AX6Wfe3NPI\n0tW1PLqmlvojzQzOTmP2pCJunBpmYjhXwzXlryjoRfqptvYOXqpuYOnqWp6u2kNzWwdjhg/ghqnF\nXD+lmMLcrKBLlDihoBdJAIePt/L4hjqWVNZQ8fZBzGDGmDxunBrm6vEjyErX3bjJrMeC3sx+CVwH\n7HP386Nt9wBfAuqju33D3R+PPnc38A9AO/Bld3+qqyIU9CJd29lwjKWra1iyupbaQ8fJSQ9x7YRC\nbiwLM23kUFI0VDPp9GTQXwIcBX79vqA/6u7ff9++44A/ANOAIuBZ4Bx3b/+o36GgF4ldR4ezaucB\nllTW8PiGOo61tBMeksUNU8PcOLWYs4blBF2i9JEeuzPW3Vea2cgYf+8c4BF3bwZ2mFk1kdB/Lcbj\nRaQLKSnG9NHDmD56GN+aM56nqvawpLKWnz6/lZ88t1V34cpfOZMpEO4wsy8AFcBX3f0gUAy83mmf\nmmibiPSC7PRUrp8S5vopYeoOH+fRNbUsqazh7qUbuGd51bt34c4Yk0eqFlFJWqcb9PcD/wF49PsP\ngFu68wPM7FbgVoCPfexjp1mGiJxUmJvFP186hn/6m7NZV3OYJZWRu3D/tG43wwdmMHdKZNTOeYWD\ngi5V+lhMo26iXTcrTvbRf9hz0QuxuPt3os89Bdzj7h/ZdaM+epHe8UF34Y4dMZC5U4qZPamIosEa\nqtmf9ejwyvcHvZkVuntddPsrwIXuvsDMxgO/572Lsc8BpboYKxK8/Ueb+fOGOpatqWX1O4cwg+mj\nhjF3ShHXnF9Ibpb68/ubnhx18wfgUiAP2At8M/p4MpGum53AbZ2C/1+JdOO0AXe5+xNdFaGgF+lb\nOxuO8dja3SxbW8uOhmOkp6ZwxXnDmTu5mEvPHa6pF/oJ3TAlIl06OavmsjW1/GndbvYfa2Fwdhqz\nJhRy/ZRiys4aoqkX4piCXkS6pbW9g5erG1i2ppanqvZworWD8JAs5k4uZu6UYsYMHxB0ifI+CnoR\nOW1Hm9t4umoPj66p5ZXqBjocJhTnMndKMZ+aVMjwgVoaMR4o6EWkR+w7coI/rYtcxN1Qe5gUg4vH\n5HH9lGKuHj+CnAytSBoUBb2I9LjqfUdYtiZyEbfm4HGy0kJcNb6AuVOKmambsvqcgl5Eeo27U/n2\nQR5dU8uK9XUcPt5K3oB0rptYxNwpxUzS/Pl9QkEvIn2ipa2DF7fsY9naWp7dvI+Wtg5G5eVEL+IW\naZK1XqSgF5E+d/h4K09urGPZmt28vmM/7jD1Y4O5fkoxsyYWMTQnPegSE4qCXkQCtfvQcZav282j\nq2vZsvcIqSnG35yTzxXjChiclUZWeojs9FSy00NkpoXITo98ZaWHSA+lqOsnBgp6EYkbm+saWbam\nlsfW7mZP44ku9w+lGNlpoeibQYis9FSy0lLITk99ty07PURWWuq7bw6Rx6FT3kDe3Tct9ZR9EmWR\nlh6bj15E5EydVziI8woH8bVrxlJ78DjHWtpoamnnRGs7TS3tNLW0cbwlsn28NfK4qaX9lLbj0f0a\njjZH93mvraOb56sZqSkMy0lnVH4Oo/MGMDo/h1F5OZydP4CiwVmEEuSN4CQFvYj0mVCK8bFh2T36\nM92d5rYOjr/7JvHeG0BTa6c3i5NvHtG2fUea2d5wjGVrazlyou3dn5eemsLIYdmMzhsQfSPIYXT+\nAEbn5TCkn15jUNCLSL9mZmSmRfr5h5zG8e7O/mMtbK8/xvb6o+xoOMa2+mO8te8Iz27eS1unjwtD\nstMYFQ3+yCeAyPbHhmaTmRa/C7Ur6EUkqZkZeQMyyBuQwbRRQ095rq29g10Hj7Oj4Sjb6yNvADsa\njrLyrXoWV9Z0+hkQHpLFqLzImf/Z+TmR7fwcRgzKDPyagIJeRORDpIZSGJUX6b+/bOypzx1tbmNH\n/TG2R98EtjdEPhFU7DxAU8t7S3BkpYUYmZfD6He7gd57E+irNX0V9CIip2FARioTwrlMCOee0u7u\n7G1sfu8NIPopYGPtYZ7YUHfKheO8ARnMnVzEv103rldrVdCLiPQgM2NEbiYjcjO56Oy8U55raevg\nnQMnu4AinwAK+2A5RwW9iEgfSU9NYczwgYwZPrBPf6+mmhMRSXAKehGRBKegFxFJcAp6EZEEp6AX\nEUlwCnoRkQSnoBcRSXAKehGRBBcXC4+YWT3w9hn8iDygoYfK6e/0WpxKr8d79FqcKhFej7PcPb+r\nneIi6M8h1tCiAAACwklEQVSUmVXEsspKMtBrcSq9Hu/Ra3GqZHo91HUjIpLgFPQiIgkuUYL+gaAL\niCN6LU6l1+M9ei1OlTSvR0L00YuIyIdLlDN6ERH5EP066M3sGjPbYmbVZvb1oOsJkpmVmNkLZrbJ\nzKrM7M6gawqamYXMbI2ZrQi6lqCZ2WAzW2xmb5rZZjP7eNA1BcnMvhL9f7LRzP5gZplB19Sb+m3Q\nm1kI+B/gk8A44DNm1rvrccW3NuCr7j4OmA7cnuSvB8CdwOagi4gTPwaedPexwCSS+HUxs2Lgy0C5\nu58PhIAFwVbVu/pt0APTgGp33+7uLcAjwJyAawqMu9e5++ro9hEi/5GLg60qOGYWBmYBDwVdS9DM\nLBe4BPgFgLu3uPuhYKsKXCqQZWapQDawO+B6elV/DvpiYFenxzUkcbB1ZmYjgSnAG8FWEqh7ga8B\nHUEXEgdGAfXAr6JdWQ+ZWU7QRQXF3WuB7wPvAHXAYXd/Otiqeld/Dnr5AGY2AFgC3OXujUHXEwQz\nuw7Y5+6VQdcSJ1KBqcD97j4FOAYk7TUtMxtC5NP/KKAIyDGzzwVbVe/qz0FfC5R0ehyOtiUtM0sj\nEvK/c/elQdcToIuB2Wa2k0iX3mVm9ttgSwpUDVDj7ic/4S0mEvzJ6gpgh7vXu3srsBS4KOCaelV/\nDvq/AKVmNsrM0olcTFkecE2BMTMj0ge72d1/GHQ9QXL3u9097O4jify7eN7dE/qM7aO4+x5gl5md\nG226HNgUYElBeweYbmbZ0f83l5PgF6dTgy7gdLl7m5n9C/AUkavmv3T3qoDLCtLFwOeBDWa2Ntr2\nDXd/PMCaJH7cAfwuelK0Hfj7gOsJjLu/YWaLgdVERqutIcHvktWdsSIiCa4/d92IiEgMFPQiIglO\nQS8ikuAU9CIiCU5BLyKS4BT0IiIJTkEvIpLgFPQiIgnu/wPmVE6l8LHNJQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7fcadb7d1710>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(s[:10])"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"num_top_words=8\n",
"\n",
"def show_topics(a):\n",
" top_words = lambda t: [vocab[i] for i in np.argsort(t)[:-num_top_words-1:-1]]\n",
" topic_words = ([top_words(t) for t in a])\n",
" return [' '.join(t) for t in topic_words]"
]
},
{
"cell_type": "code",
"execution_count": 347,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['critus ditto propagandist surname galacticentric kindergarten surreal imaginative',\n",
" 'jpeg gif file color quality image jfif format',\n",
" 'graphics edu pub mail 128 3d ray ftp',\n",
" 'jesus god matthew people atheists atheism does graphics',\n",
" 'image data processing analysis software available tools display',\n",
" 'god atheists atheism religious believe religion argument true',\n",
" 'space nasa lunar mars probe moon missions probes',\n",
" 'image probe surface lunar mars probes moon orbit',\n",
" 'argument fallacy conclusion example true ad argumentum premises',\n",
" 'space larson image theory universe physical nasa material']"
]
},
"execution_count": 347,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(Vh[:10])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We get topics that match the kinds of clusters we would expect! This is despite the fact that this is an **unsupervised algorithm** - which is to say, we never actually told the algorithm how our documents are grouped."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will return to SVD in **much more detail** later. For now, the important takeaway is that we have a tool that allows us to exactly factor a matrix into orthogonal columns and orthogonal rows."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Non-negative Matrix Factorization (NMF)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"<img src=\"images/face_pca.png\" alt=\"PCA on faces\" style=\"width: 80%\"/>\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))\n",
"\n",
"A more interpretable approach:\n",
"\n",
"<img src=\"images/face_outputs.png\" alt=\"NMF on Faces\" style=\"width: 80%\"/>\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Idea"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Rather than constraining our factors to be *orthogonal*, another idea would to constrain them to be *non-negative*. NMF is a factorization of a non-negative data set $V$: $$ V = W H$$ into non-negative matrices $W,\\; H$. Often positive factors will be **more easily interpretable** (and this is the reason behind NMF's popularity). \n",
"\n",
"<img src=\"images/face_nmf.png\" alt=\"NMF on faces\" style=\"width: 80%\"/>\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))\n",
"\n",
"Nonnegative matrix factorization (NMF) is a non-exact factorization that factors into one skinny positive matrix and one short positive matrix. NMF is NP-hard and non-unique. There are a number of variations on it, created by adding different constraints. "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Applications of NMF"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- [Face Decompositions](http://scikit-learn.org/stable/auto_examples/decomposition/plot_faces_decomposition.html#sphx-glr-auto-examples-decomposition-plot-faces-decomposition-py)\n",
"- [Collaborative Filtering, eg movie recommendations](http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/)\n",
"- [Audio source separation](https://pdfs.semanticscholar.org/cc88/0b24791349df39c5d9b8c352911a0417df34.pdf)\n",
"- [Chemistry](http://ieeexplore.ieee.org/document/1532909/)\n",
"- [Bioinformatics](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0485-4) and [Gene Expression](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2623306/)\n",
"- Topic Modeling (our problem!)\n",
"\n",
"<img src=\"images/nmf_doc.png\" alt=\"NMF on documents\" style=\"width: 80%\"/>\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**More Reading**:\n",
"\n",
"- [The Why and How of Nonnegative Matrix Factorization](https://arxiv.org/pdf/1401.5226.pdf)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### NMF from sklearn"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"First, we will use [scikit-learn's implementation of NMF](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html):"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"m,n=vectors.shape\n",
"d=5 # num topics"
]
},
{
"cell_type": "code",
"execution_count": 363,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"clf = decomposition.NMF(n_components=d, random_state=1)\n",
"\n",
"W1 = clf.fit_transform(vectors)\n",
"H1 = clf.components_"
]
},
{
"cell_type": "code",
"execution_count": 296,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['jpeg image gif file color images format quality',\n",
" 'edu graphics pub mail 128 ray ftp send',\n",
" 'space launch satellite nasa commercial satellites year market',\n",
" 'jesus matthew prophecy people said messiah david isaiah',\n",
" 'image data available software processing ftp edu analysis',\n",
" 'god atheists atheism religious believe people religion does']"
]
},
"execution_count": 296,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### TF-IDF"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[Topic Frequency-Inverse Document Frequency](http://www.tfidf.com/) (TF-IDF) is a way to normalize term counts by taking into account how often they appear in a document, how long the document is, and how commmon/rare the term is.\n",
"\n",
"TF = (# occurrences of term t in document) / (# of words in documents)\n",
"\n",
"IDF = log(# of documents / # documents with term t in it)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"vectorizer_tfidf = TfidfVectorizer(stop_words='english')\n",
"vectors_tfidf = vectorizer_tfidf.fit_transform(newsgroups_train.data) # (documents, vocab)"
]
},
{
"cell_type": "code",
"execution_count": 263,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"W1 = clf.fit_transform(vectors_tfidf)\n",
"H1 = clf.components_"
]
},
{
"cell_type": "code",
"execution_count": 255,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['don people just think like know say religion',\n",
" 'thanks graphics files image file program windows format',\n",
" 'space nasa launch shuttle orbit lunar moon earth',\n",
" 'ico bobbe tek beauchaine bronx manhattan sank queens',\n",
" 'god jesus bible believe atheism christian does belief',\n",
" 'objective morality values moral subjective science absolute claim']"
]
},
"execution_count": 255,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H1)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7f0e1039a7b8>]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmUHGd5LvDn9Sh2SMBhsTC+XpCdCBxBbC5MBOFwCZvx\nlhOZYwdksyUBdA0xCQEuCAzGAUNsjBcEsibyAsgYy5tsCUu2bMmWJWsZaSRLo3Wk2aRZJM0izb5P\nv/ePrh7V9FR3fdVd1VVd/fzO0VFPdXX1V11Vb331raKqICKieDkt7AQQEZH/GNyJiGKIwZ2IKIYY\n3ImIYojBnYgohhjciYhiiMGdiCiGGNyJiGKIwZ2IKIamhfXFZ511ls6YMSOsryciKkrbt2/vUNXp\nbuuFFtxnzJiBqqqqsL6eiKgoichhk/VYLENEFEMM7kREMcTgTkQUQwzuREQxxOBORBRDDO5ERDHE\n4E5EFENGwV1ErhCRGhGpFZH5Gdb5iIjsFJG9IvKKv8kkonwdOt6LyvrOsJNBBeLaiUlEygAsBHAZ\ngGYA20Rkharus63zRgD3AbhCVY+IyFuDSjAR5eaye9YDABpvvzrklFAhmOTcZwOoVdV6VR0BsBTA\nnLR1bgCwTFWPAICqtvmbTCIi8sIkuJ8LoMn2d7O1zO4dAN4kIutEZLuIfMFpQyIyT0SqRKSqvb09\ntxQTEZErvypUpwF4H4CrAVwO4Ici8o70lVR1saqWq2r59Omu494QEVGOTIJ7C4DzbX+fZy2zawaw\nWlX7VbUDwHoAl/qTRIqqBzbU47UjJ8NOBhE5MAnu2wDMFJELReR0AHMBrEhbZzmAD4nINBH5MwDv\nB7Df36RS1Ny2cj8+dd+msJNBRA5cW8uo6piI3ARgNYAyAA+p6l4RudF6v0JV94vI8wCqASQAPKCq\ne4JMOBERZWY0nruqrgKwKm1ZRdrfdwK407+kERFRrthDlYgohhjciagkdQ+Ohp2EQDG4E1HJeba6\nFZf+1wvY2dQVdlICw+BORCVnY21yjJ29rd0hpyQ4DO5ERDHE4E5EFEMM7kREMcTgTkQUQwzuREQx\nxOBORBRDDO5ERDHE4E5EFEMM7kRUslTDTkFwGNyJqOSIhJ2C4DG4ExHFEIM7kQc1x3oxPDYedjKI\nXDG4Exlq6xnC5feuxw+f4SRjFH0M7kSGeoaS439vP8xJwSn6GNyJiGKIwZ2IKIYY3ImIYojBnYgo\nhhjciYhiyCi4i8gVIlIjIrUiMt/h/Y+ISLeI7LT+3eJ/UomIyNQ0txVEpAzAQgCXAWgGsE1EVqjq\nvrRVN6jqPwSQRiIi8sgk5z4bQK2q1qvqCIClAOYEmywiIsqHSXA/F0CT7e9ma1m6D4pItYg8JyLv\n8iV1RESUE9diGUM7AFygqn0ichWAZwDMTF9JROYBmAcAF1xwgU9fTURE6Uxy7i0Azrf9fZ61bIKq\n9qhqn/V6FYA/EZGz0jekqotVtVxVy6dPn55HsomIKBuT4L4NwEwRuVBETgcwF8AK+woi8jaR5AjJ\nIjLb2m6n34klIiIzrsUyqjomIjcBWA2gDMBDqrpXRG603q8AcB2Ar4rIGIBBAHNV4zzHCRFRtBmV\nuVtFLavSllXYXv8awK/9TRoREeWKPVSJiGKIwZ2ISlacy44Z3Ik8inNAKBUlMD82gzuRuVIICRQX\nDO5ERDHE4E5EFEMM7kREMcTgTkQUQwzuREQxxOBORBRDDO5ERDHE4E5EFEMM7kREMcTgTkQUQwzu\nREQxxOBORBRDDO5ERDHE4E5EFEMM7kREMcTgTkQUQwzuREQxxOBO5BXn2aMiwOBOZEg4y178aHzv\n1AzuRFRySuFGbRTcReQKEakRkVoRmZ9lvb8VkTERuc6/JBIRkVeuwV1EygAsBHAlgFkArheRWRnW\nuwPAC34nkoiIvDHJuc8GUKuq9ao6AmApgDkO630dwFMA2nxMHxER5cAkuJ8LoMn2d7O1bIKInAvg\nUwAW+Zc0IiLKlV8VqvcC+K6qJrKtJCLzRKRKRKra29t9+moiIko3zWCdFgDn2/4+z1pmVw5gqSSr\noM8CcJWIjKnqM/aVVHUxgMUAUF5eHt82SEREITMJ7tsAzBSRC5EM6nMB3GBfQVUvTL0Wkd8CeDY9\nsBMRUeG4BndVHRORmwCsBlAG4CFV3SsiN1rvVwScRiIi8sgk5w5VXQVgVdoyx6Cuqv+cf7KIiCgf\n7KFKRBRDDO5ERDHE4E5EFEMM7kREMcTgTkQUQwzuREQxxOBO5BG7VlMxYHAnMlQC8ztQjDC4ExHF\nEIM7kSEWx1AxYXAn8ojFM/ER5xs2gzsRlRwpgVs0gzsRUQwxuBMRxRCDO5HPrl20Cfetqw07GVTi\nGNyJfLb98En8/PmasJNBJY7BnYgohhjciYhiiMGdiCiGGNyJiGKIwZ2oiC3Z3IgZ81eia2Ak7KRQ\nxDC4ExWxP1QeAQC0dg2FnBKKGgZ3IqIYYnAnIooho+AuIleISI2I1IrIfIf354hItYjsFJEqEfmQ\n/0klioY4jyRI8THNbQURKQOwEMBlAJoBbBORFaq6z7baWgArVFVF5BIAjwO4OIgEE4Ul/uMIUpyY\n5NxnA6hV1XpVHQGwFMAc+wqq2qeqqQzNn4OZGyKiUJkE93MBNNn+braWTSIinxKRAwBWAvhXpw2J\nyDyr2Kaqvb09l/QSEZEB3ypUVfVpVb0YwDUAfpJhncWqWq6q5dOnT/frq4mIKI1JcG8BcL7t7/Os\nZY5UdT2Ai0TkrDzTRkREOTIJ7tsAzBSRC0XkdABzAaywryAifyUiYr1+L4AzAHT6nVgiIjLjGtxV\ndQzATQBWA9gP4HFV3SsiN4rIjdZq1wLYIyI7kWxZ8xlbBSsR0RTdg6O4d81BjCfCCxVxjlKuTSEB\nQFVXAViVtqzC9voOAHf4mzQiirMf/3EfntrRjL8+50xc/q63FfS7pQTatbKHKhGFYnB0DAAwNh7j\n7HOIGNyJiGKIwZ2IKIYY3IkiqLNvGDubusJORkEoO7QHgsGdKILmLNyIaxZuDDsZgRKO1hMoBnei\nCGo+ORh2EqjIMbgT5aClaxDv+fELqG/vCzspRI4Y3Ily8OyuVnQNjGLptib3lSmrOHckChODOxGF\ng0XugWJwJ/KII2tQMWBwJzIkpdBnPU9Do+PoGRoNOxkEBnci8tGVv9yAS259wdNn+BwUDAZ3IvJN\nQ0e/8bp8DgoWgzu5qmvvw6X/9QJau9j2mqhYMLiTq0e2HEH34ChW7T4adlKIyBCDOxGFiq2PgsHg\nTkShYOujYDG4kyteg0TFh8GdKAY4bG5h7ThyEu/7yYvoHohum34GdzIWpaLRsfEEhsfGw05G6Fi0\nkZ9cy/t/tfYQOvtHsP3ICZ9T5B8Gd3IVxfBxwwOVeOcPng87GZSHMM+r1HePJRSj44kQUxIcBncy\nFqVH/60N0c0xFRJbmuTntpX7MfPm58JORiAY3MkVn/yjj7MaUTqj4C4iV4hIjYjUish8h/c/KyLV\nIrJbRDaJyKX+J5WoeI2NJ7B4fR2GRllPkC5ODx8Hj/fiF6trIvFE5RrcRaQMwEIAVwKYBeB6EZmV\ntloDgL9X1b8B8BMAi/1OKIUvAudr0Vq2owU/W3UAC9YeCjspvvq/D1fhY79Yl9Nn4/hEeP3iLfj1\ny7XoHgy/Fc00g3VmA6hV1XoAEJGlAOYA2JdaQVU32dbfAuA8PxNJ4WKLjPwNjIwBAPqHx0JOib9W\n7z0edhIiJVU5G4ViMpNimXMB2OcSa7aWZfIlAPGsoShxzLjnrxR/w46+Ydz94kEkEqWz91FofGCS\nczcmIh9FMrh/KMP78wDMA4ALLrjAz6+mAIWfByl+pfz0890nq7H2QBv+7qK34O/+8i1T3o9CIPRL\nlI6zSc69BcD5tr/Ps5ZNIiKXAHgAwBxV7XTakKouVtVyVS2fPn16LuklCl18QlFhDFqVyIm0Spvo\nhMF4Mgnu2wDMFJELReR0AHMBrLCvICIXAFgG4POqetD/ZFIUlHqFqkkwqmrM3v6+FH9D7nM4XItl\nVHVMRG4CsBpAGYCHVHWviNxovV8B4BYAbwFwn/VYMqaq5cElmwqKWSxjrx3pclweoaf10KT/BBGI\nf76L0nE2KnNX1VUAVqUtq7C9/jKAL/ubNKL4iEJOjswMj40jkQBed3pZ2EnJC3uokrE4VXxR4WQ6\nbyKUyZ3kk/esx1/fkt+4RVG4UhjcyVUU2uxGgckFmzGQ8SeMbjRPc7hzIOfPRmkXGdzJmL1ooXco\n/B54YbFfwFHoZk7khME9B4mE4rndR0umU4ZTrvMHz+wpfEJiIIpFWx19wzjWPRTY9t3uf3G8P0bh\nps/gnoNHth7BVx/ZgceqmtxXjqnOvhHPn3l06xE8sKE+gNSEJ73TSqZrOkqP6+nKb1uDD/z32sC/\nJ714L0odfvwSpX1icM9BW08yl9PeOxxySorL95btxm0r94edjIKpajyBxo7+sJMRuvDzsOFq7x3G\ngWM9Bf9eX4cfKDURePIqiOjkRaLPnnG7rmIzAKDx9qsnlpXKOeMkQpnawKUOc82xXlx+73oAk8+D\nQmDOPQcldI6WtERC8fi2Jk/TsJVy8Pbisw9swdOvJUcxidNvlh4bfrupMYxkAGBwpxyVQi7syR3N\n+M5T1Vi8Pl71BAXnELw31joOP1UwQZeNO92wmk8O4PdbDgf6vXYM7nnI1PJhaHQc//rbbahr7ytw\nishP3QPJ5p4n+71XHk9RCndDF6XwC2Q7zNffvwU/eGZPwZoRM7jnwuVCrWw4gZcOtOHWFXsLlKDC\niELzLgrf1oYTnoqqSpk9VHT1J4N6oVpQM7jnoVRiXalnOr0cZrd17e9vP3wCI2PFFSR3N3fj0/+z\nGT9//oDxZ9za9gd1GfUNj6G2rXBPz6qKDquJsOM+F/g6YnDPQZRiXW1bLz7/YCUnXo649HPm4PFe\nXLtoM362qriahnb0J5v/HjzuPWgWug345x+sxCfufiWQbbd2Te309ZuNjVOWOe4xc+7RF4WM+3/9\ncR82HOpAZUP2ccT9UCpPKumCCEknrHL8fUcL3/650MI6bzINv5yvo92DqDneO2X5QYdldoXOFDK4\n5yCKxRRBlodz4LDJfP2lS/SGWcw6es0r2J1iRaGGoGBwLxIn+0cwY/5KLN85eYbDQj7qlnoc8vOX\nLobb5S3L9+CJDENs5HIuZDpV41JR73YpFrpYisHd0PDYOBa+XDu5AizDSWl6COva+zBj/kq8dOC4\n67r1Vjf2TJ0igrw8nM7JKI2hQcFYsvkw/t+T1ZOW5XLU4xG6c+P01FuoexmHHzD0wIYG3Lm6BmdM\nO823YopUmeCz1UfxsYvPzmkbDLHFJf3C9usRPYzRJhs8jpsT/3PVtodOjWUk41uBYM7d0MDIGABg\ncORUq5RMB8nzwfPjaBfgjLEHJj8u1LtfqMGL+9yfWsLmqSlkplEhJf1vf0JdLtu5Zbk/wzX3D4/5\nsp2oaut1HgY51xspK1QjKpVbV/hXoerHZk7lBoKsUA3Ggpdq8ZUlVQFtPaomH6cwipuXbM6/C/z6\ng+3G14FbmXpUi23aeryN+upcfDl1WaHqGBjcDfldxNzQ0Y9vPbELQH45w0LmBnY1n2paxiJ3c6o6\npSivkJ1r/JTPE0e+58yrhzrQ0jWY30YKxOmaZoVqxNmDaz4TM7yw95in73U7L4LMDIyMJzf+0oG2\ngnxf3Gw41DFl2fef3h1CSvxV6Cayn3uwEh+/a11Bv9MLp1/DaRnL3CMmdZCCKP7I5zGNrVaiQ1Vx\non/qo3z3YOaBoor5HmlcLOPjdw6NFsdwDU6XNMvco8rnIGrfnC/1qQFGCdOyRFNxadecbsnmw7h/\nQ8OU5UHef8P8Lb3vl/MHvvNkNY50DuSdnmyGRsfROzSKtt4h4xnUMv20ppXmyWXhNYU0Cu4icoWI\n1IhIrYjMd3j/YhHZLCLDIvJt/5MZHare7sCb6jrwrlueR09Aw3wWIjfg93es3H3U5y1GwysH2z1/\nxq/gXKgikpzauRvs4qPbjuSwZXOfvGc9/ubWFzD7p2vxtz9dE+h32dmPbyEaP9i5BncRKQOwEMCV\nAGYBuF5EZqWtdgLAvwP4he8pjICFL9dOlJFnOixtPUO44t71aLVV+KgC9645hP6RcexrnTyGiP1i\n9HJ9+9H8ct6SKsyYv9LDJ6bKJ5T42YSu5lj28TyiwORYN50YwMKXa0PNiT+1vRlbPYxR5PWGki2n\nn9rtldVHMWP+SnQN+DCGvs2RE8E+GQBpxzkCBW4mOffZAGpVtV5VRwAsBTDHvoKqtqnqNgCFGYW+\nwO5cXYMDDkHEfgAfr2rCgWO9eKTSuZlZtmt2xa5WzHbJTUycNqp4bvfRiZ6yE7kB6wu+t2w3PnLn\ny1m39YLHtuVRK9a3PwWFPWSufZambIE5Y9d76/8v/W4b7lxdg+aT4bUG+dYTu/Dp/9mcdR37Hno9\nL7JdA6lr6YFXk79nXXvmDlIdfaeKVVQVy3Y0T/RDWbK5EbNueT7te3MLtKk0barryHnU1TDrxEyC\n+7kA7ANMNFvLPBOReSJSJSJV7e3eH2EjQdXxpE4NwD+po49Mroht6x3C1Qs24Gj34JRttBmWA+5q\n7sZXH9mBu16sSX3LpPcf3XoEjT6XX/r9yJ9P5nRkLIFLbn1h4u8gr51M2x4bV/zqpdopy/PJqw2M\nRGvI5nyf7HJlcm58+OenMi9Vh0/im4/vwo+WJyfGuWX5Xt9+S1Xg0PFe3HB/pS8T76TGei+Uglao\nqupiVS1X1fLp06cX8qt9Yz/37Cfi3S8enPI+MDlAPL6tCXtbe3yZR7ElwBxeIqHYcKg9a47n5Zrc\nb857W3uwPofyaQAYcZgBaMb8lfjxH/cBAH60fE8ggWlLfSf6rOKklq7BidcmJlWeh/+0nhfJ8Nro\ns9k+oJP+y7quPXinjkN65sh+7ub6mytOtXQ6ZPVLeOa1lomJvbN+Ntt3RmhsmRYA59v+Ps9aVpJU\nc3zUCvGiPtzZj7LTBOe96c+M1v/Npkb85Nl9qPjce3HFu8/xPXf88JbDeNh2g6s51ou+4VG87+1v\nznmbD21swLwPX4TfZeh9ubmuE1sbTuDkwAje8KfT8K1PvtN4292Do5i7eAvOf/Prsq6XT7+HYuTn\neaEAjnQOYFdTsqOc8aYDvK4Sk24QydffeGynp21kGuivEEyC+zYAM0XkQiSD+lwANwSaqiLhdF5N\n7UEqGdfN97tbuwaxZv9x1+3//Z3rAACNt19ttO0jncnyzmPdybE1gg5Ol9+7HoB5+jK58ffbM753\n/f1bJv3tJbgPW+X6TSecn5Zq2/pw7aJNOPvMM1y35bUz2sfvWoe13/qISTIBAGv2HUdZmeCj73yr\n8WeCZnruX1exaeJ1KgP1eFUTZp1zJt597l8E/v3peofG8PozyozXD6PdfzauwV1Vx0TkJgCrAZQB\neEhV94rIjdb7FSLyNgBVAM4EkBCRbwCYparxn2Ymjb2SNZnLd17Pa+4/ff2ugRF88PaXPKcvbipe\nqZt4HdRAVm5H6vdbDqN7cDRrZ6WUKaNCWn9nOh2yVSw6+bI1Vk++N0ozHs/hLO+pKroGpv5+37GG\nHPa6P34Xf3ndXNZSmSi1c1fVVar6DlX9S1X9qbWsQlUrrNfHVPU8VT1TVd9ovY5lYDdp4tRo5Xzt\nrTr8PqAbazsnp8tg+5+6b6O/iQhBeoCwT1MXVOWq+yQMBtvIMw3ffGwnPvdApfH6QYzBsq3xhOdh\nfgEYnZz5jpmUvZllPhefeUqiVvzGHqousp0Yjm8p8Ms1hwAA1c3dBey44L79oOaUDEptWx9W7GrN\nvtKkYYinXl61bX2Bt/xwa000uTey9/Ngxa5WLHutBa/WdqBveAzDY+6tQX6xusZ1Ha/+qWIzfmRr\nNeL1ZurladVt1VTZfCZuv3KPwVNWUCLTianU3beubtLf2YpaAIfWMhku/EaHHNDYeMJTK4yghd2w\n4xN3v4J/f/S1rOvY0zjkEPS2H3bulJNIqHGOzi0kuccscW3nfupvxXhi8lL7b/DuH63GtYs2Id3g\nqFnQjyrT6yZlzsLkU2iz9YQynGXMGaejvMyoxYu3Fjf2m1cUhthgcHfx6NbJ3aInNYU0CH+v1k4d\nEXDhy3WTWouk/MfSnXj3j1Y7bmdva3fW7/HjXFq2oxkz5q/EYI4dNgohWyBN79B0uLMf333KefTF\ni76/yjFIOn+nS87caCunpAfv5DZObcWtmeielqklntcu2ox//FVhi91M99v41LRt0CSTr6oTE49s\nrk8vpsz/glDDdHjebpTK3GmybLkKzdDJyUS2MVduftqf2XOyWbA2WZx0zOMkBWHKdhH/cHn2jic7\nbMVU4wnFi/uOO27P7XCedpp5sUxCgVcOnho62elKdwr+JmqOF3YoBs+NAlze99rb2HScp5yDqTq+\nNPto+Bl3Bnc36edvzh0iAj7Yqz2OD+8k1bN1SoCL2vgDNtl647rE3EkqXqnDV5ZUeR6aATALMMet\nG+aT25sxNj71ZLA/BUb45w5MZUOn+0ppEqoZbzC+jLQK9fRUFrWmkAzuLtJz6Wp0O3cY5tO/JDl6\nZqdLxaMHrUUy2026o92T57w8zUOUTI3pYh+3ZIJrobv79u09ivtHTtWr5Hpe/MOvNuDhzY05fjrp\n+T3H8qpsdmvamJLKkSey5HDq2ry3wlGXNJxarzDh1B4rgmu9Y47B3aORsQQGrYvT6fE523H7/ZZg\nhzX1S7a21eW3vZhxcLQg3XD/FsyYv9LTE5BJzn3HkZOu67i2hnF9HzjNdqX952O7sm7D5J60p6XH\ntdgpZV9rD3Y6tC55bk9+Qy9nS+ez1ae2nRp0b7mVAWnrcZ54Otu2X3M4Tm6NG/LlNQZHbZgJBncX\nTsUyC6xBo1Y5lJHvbnGu+OwaGMExg5M6iuw/QUffSEHK/9Ntqks+tj9R1eSyZlJ9e59RmfDe1vy7\nY5gFmPDKWq5asAHXLCxsZWtqyF57X4++4TF09A1j9s/WGm3jRP+pgbY+/+DWKe8ni00yFMuo82sv\nJn3OYCPGFcysUI2+4bEE7l1zEGv3nyqnrWw44ZhjdCpnJWf/+dhOLEprgpry3B6zuoXr79/iqcw9\nHwddxpQXydIU0uG0KPTcpLnK2rAAyUng7SN4AkBnhpER01toCWTSHAhOTYT9DJKX3Dq1lVqytUz2\nY7Fk8+GJ8eejVldiMrZMSTucpcJOkZyMI+7cTtrOvmG85fXu46qYyjbqXqXhZBKDI+OeytxTbn56\nDz77/rejsr4Tt63cD8B5JEq7tbaJwzPJdKPZ3dKN7YdPFTlE4XHeVLafN5FQNHT0TV4f5uXfWxs6\n8Y6z32CQCOfFXsvZe4ay9y/JtLV11uiorV1DkZvPmDl3j45222dacj7kTmOzD0Sw7bi9DX+2yQjc\nAs77bluD2rZozYikMMvl7zh8EjfcvwXjickB/JuPnyoX/6NbL1kD2W401y7alHOuz2kSmbBM6vQD\n51xvwrC1461/3OcaLBOqk5pP+t0T+StLqrB8p/kAuNEK7Qzunq3e672pHBDs+OtObn7aufNOwlYJ\n/L1lu3HgWPLRt/nk1CeU1LVqMuVZQ8fkzx863utpyja/9brkxFKefq0Fm+o6p4z46GcmbGh03NuF\nH3KUqKz33iwRwKRZpFSTUwfaiXjLUbsVq2Wd2cmHMncA+M3GRgDJoUScrpFsnOrkConBPQ9ezhkv\nJ3X6E8E91kQgXjxS6dwy5//YZrEBsnfbnkiPwfelN3O77J71rlO2RYk9/cNjuRXpZPL9DDfaMGVr\njvd1lyEfUtx+olvSWvOcHBj1FGirm116ZRtuZ8nmRqP16tv7sr5/9YJXs2/A9nt8/+nd+NojOxxX\nY4VqEQjqINm3OzaewC/X+leunz5aYM/QKJ6oasq6L9naJ6dkW2U8objj+QOmSZz6/Tn22PTipO3p\n5LZn9/uac+8dGjMujzVtux11TkfsRY8dxH66an/W98cNGyks22FWtPKxu17J+n62IZ3TW+5sODR1\n2JFCY4VqHjx1Rshh1ermLvzjr3NvwtZ0YgC/ein7jeE7T1bjaPcQ7vnMpTl/D5D8LTbWdmDFzlbc\ncd0lk95bsaslY+sXE24XuR8OHj+Va3Ma9ydo9oDuV8VcfXtfxqKB7JNV5y/TteFnhuhrf8g8OYud\nl58zn4H7zHuoFibrXnLBfWdTF9559hvwutPNZ1jJxFuxjId1NZl/e8mgFUY26UUwTlK9Ok/2Z8mV\nGCT+q7ZH0PTg7tRpx0nTiQGc/+apUwE++GqD0eej7HBn9h6Yfk9qDgA33F+ZsW9FtkM6dUIRxZb6\nqfUn9ptQtgp5O5OnQFPpcxrY5fo1pv0o0kkEG7CWVLFMe+8wrlm4Ed9+0izY+MlrheqidXWodLig\nsvFa4WP342f3TVm24VA7/uU3WzOW35vwUqRicjMqVtma1NoJkudpvsYT6jgEckplfedEj1E3D285\nPGWawnTps4JlCq6FulHbr4V6DxOM5FPXYpxzL1CZe0nl3FO5C7eB/k15OUjZRnxMt7GuM6cy6h88\n42/P0Zdrsg89m4n9kfyi76/yKznFzcOgUt9+Iv/Mh9skJzdkmdUpfXyd9IrRlE9XnKowt/cmBTIX\nPbhOvuKTy+5ZP/Hay2iTuXZ8O94zFLm8e9Hl3IfHxrFiV2tOg+9MzIrk050zqIk1vvjQ1K7WJtbl\nGIz91tqd+zALQc+aFBbTAPPRX6wLNiE+Sq+ctzc/PXA0Ou3vvci1vqOy4UTkRoUsupz7vWsOYdG6\nOrz+jDJ87OKzPX029cjlZ7kfTVXoNv3FoLkEfpOrFmyYeG0y01EU5Voso6roHza7gad3mAtK0eXc\nj1sVRLube1xnJ0pnD+4n+0cwY/7KKTMtpTR29OOBDfX5JbZEubUXJooq+0BnXiRU8ZtNZvUJn7h7\nfV71Y6aKLrhPswrF7llz0L1TgU19e99EV+KEAk3Wj/uHDJWFn1m8eWJsEfJm/rLoddohMnH7c7n1\nx7h/Q4PXBDStAAAF4ElEQVSn4t5CFFsVXXAvy1Ljoap4bNsR9Drcfa9asAH/bR24REKRasTh9BTW\n1js0MXMOEZHfFrj0P/GDUXAXkStEpEZEakVkvsP7IiILrPerReS9/ic1Kb1MrGtgBOsPtqO+vQ9P\nVDXju09N7vZb394HVcWQrZv9yYGRiQrZ6uZuDI2OJwO+FfH/+aFtQSWfiAj1WSbE8YtrhaqIlAFY\nCOAyAM0AtonIClW1N4y+EsBM69/7ASyy/vddes79PT9+cco6Gw51YO3+49hx5CQWvlyHa97zvya9\nn1BM6tJ/8Q+fn3h91z9din1H85/AgYgok6Ba2tmZtJaZDaBWVesBQESWApgDwB7c5wBYosns8BYR\neaOInKOqvg+Llq1Yxu5Lv6uaeO00v2imZoPf8qGNMRFR2EyKZc4FYO+T22wt87qOL6YVanodIqIi\nVtAKVRGZJyJVIlLV3p5bh5uPvvOtPqeKiKiw5n34osC/w6RYpgXA+ba/z7OWeV0HqroYwGIAKC8v\nz6kn0Qf/6iw03n51Lh8lIioZJjn3bQBmisiFInI6gLkAVqStswLAF6xWMx8A0B1EeTsREZlxzbmr\n6piI3ARgNYAyAA+p6l4RudF6vwLAKgBXAagFMADgX4JLMhERuTEaW0ZVVyEZwO3LKmyvFcC/+Zs0\nIiLKVdH1UCUiIncM7kREMcTgTkQUQwzuREQxxOBORBRDkst0db58sUg7gMM5fvwsAB0+JieqSmE/\nuY/xUAr7CERjP9+uqtPdVgotuOdDRKpUtTzsdAStFPaT+xgPpbCPQHHtJ4tliIhiiMGdiCiGijW4\nLw47AQVSCvvJfYyHUthHoIj2syjL3ImIKLtizbkTEVEWRRfc3SbrjjoRaRSR3SKyU0SqrGVvFpEX\nReSQ9f+bbOt/z9rXGhG53Lb8fdZ2aq3JyUObokpEHhKRNhHZY1vm2z6JyBki8pi1vFJEZhRy/6w0\nOO3jrSLSYh3LnSJyle29YtzH80XkZRHZJyJ7ReQ/rOVxO5aZ9jNWxxOqWjT/kBxyuA7ARQBOB7AL\nwKyw0+VxHxoBnJW27OcA5luv5wO4w3o9y9rHMwBcaO17mfXeVgAfACAAngNwZYj79GEA7wWwJ4h9\nAvA1ABXW67kAHovIPt4K4NsO6xbrPp4D4L3W6zcAOGjtS9yOZab9jNXxLLac+8Rk3ao6AiA1WXex\nmwPgd9br3wG4xrZ8qaoOq2oDkuPlzxaRcwCcqapbNHn2LLF9puBUdT2AE2mL/dwn+7aeBPDxQj+p\nZNjHTIp1H4+q6g7rdS+A/UjOhRy3Y5lpPzMpyv0stuBesIm4A6QA1ojIdhGZZy07W0/NXHUMwNnW\n60z7e671On15lPi5TxOfUdUxAN0A3hJMsj37uohUW8U2qeKKot9HqxjhfwOoRIyPZdp+AjE6nsUW\n3OPgQ6r6HgBXAvg3Efmw/U0rBxCrJkxx3CfLIiSLCN8D4CiAu8JNjj9E5PUAngLwDVXtsb8Xp2Pp\nsJ+xOp7FFtyNJuKOMlVtsf5vA/A0kkVNx61HPFj/t1mrZ9rfFut1+vIo8XOfJj4jItMA/AWAzsBS\nbkhVj6vquKomANyP5LEEingfReRPkAx4j6jqMmtx7I6l037G7XgWW3A3maw7skTkz0XkDanXAD4J\nYA+S+/BFa7UvAlhuvV4BYK5V834hgJkAtlqPyD0i8gGrHO8Lts9EhZ/7ZN/WdQBesnKQoUoFPMun\nkDyWQJHuo5WmBwHsV9W7bW/F6lhm2s+4Hc+C1t768Q/JibgPIlljfXPY6fGY9ouQrHXfBWBvKv1I\nlsWtBXAIwBoAb7Z95mZrX2tgaxEDoBzJk68OwK9hdUgLab8eRfIxdhTJcscv+blPAP4UwBNIVmRt\nBXBRRPbxYQC7AVQjeTGfU+T7+CEki1yqAey0/l0Vw2OZaT9jdTzZQ5WIKIaKrViGiIgMMLgTEcUQ\ngzsRUQwxuBMRxRCDOxFRDDG4ExHFEIM7EVEMMbgTEcXQ/wfAybDICFnzbQAAAABJRU5ErkJggg==\n",
"text/plain": [
"<matplotlib.figure.Figure at 0x7f0e103e4198>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt.plot(clf.components_[0])"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"43.71292605795277"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clf.reconstruction_err_"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### NMF in summary"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Benefits: Fast and easy to use!\n",
"\n",
"Downsides: took years of research and expertise to create"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Notes:\n",
"- For NMF, matrix needs to be at least as tall as it is wide, or we get an error with fit_transform\n",
"- Can use df_min in CountVectorizer to only look at words that were in at least k of the split texts"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### NMF from scratch in numpy, using SGD"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Gradient Descent"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The key idea of standard **gradient descent**:\n",
"1. Randomly choose some weights to start\n",
"2. Loop:\n",
" - Use weights to calculate a prediction\n",
" - Calculate the derivative of the loss\n",
" - Update the weights\n",
"3. Repeat step 2 lots of times. Eventually we end up with some decent weights.\n",
"\n",
"**Key**: We want to decrease our loss and the derivative tells us the direction of **steepest descent**. \n",
"\n",
"Note that *loss*, *error*, and *cost* are all terms used to describe the same thing.\n",
"\n",
"Let's take a look at the [Gradient Descent Intro notebook](gradient-descent-intro.ipynb) (originally from the [fast.ai deep learning course](https://github.com/fastai/courses))."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Stochastic Gradient Descent (SGD)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Stochastic gradient descent** is an incredibly useful optimization method (it is also the heart of deep learning, where it is used for backpropagation).\n",
"\n",
"For *standard* gradient descent, we evaluate the loss using **all** of our data which can be really slow. In *stochastic* gradient descent, we evaluate our loss function on just a sample of our data (sometimes called a *mini-batch*). We would get different loss values on different samples of the data, so this is *why it is stochastic*. It turns out that this is still an effective way to optimize, and it's much more efficient!\n",
"\n",
"We can see how this works in this [excel spreadsheet](graddesc.xlsm) (originally from the [fast.ai deep learning course](https://github.com/fastai/courses)).\n",
"\n",
"**Resources**:\n",
"- [SGD Lecture from Andrew Ng's Coursera ML course](https://www.coursera.org/learn/machine-learning/lecture/DoRHJ/stochastic-gradient-descent)\n",
"- <a href=\"http://wiki.fast.ai/index.php/Stochastic_Gradient_Descent_(SGD)\">fast.ai wiki page on SGD</a>\n",
"- [Gradient Descent For Machine Learning](http://machinelearningmastery.com/gradient-descent-for-machine-learning/) (Jason Brownlee- Machine Learning Mastery)\n",
"- [An overview of gradient descent optimization algorithms](http://sebastianruder.com/optimizing-gradient-descent/)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Applying SGD to NMF"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"hidden": true
},
"source": [
"**Goal**: Decompose $V\\;(m \\times n)$ into $$V \\approx WH$$ where $W\\;(m \\times d)$ and $H\\;(d \\times n)$, $W,\\;H\\;>=\\;0$, and we've minimized the Frobenius norm of $V-WH$.\n",
"\n",
"**Approach**: We will pick random positive $W$ & $H$, and then use SGD to optimize."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**To use SGD, we need to know the gradient of the loss function.**\n",
"\n",
"**Sources**:\n",
"- Optimality and gradients of NMF: http://users.wfu.edu/plemmons/papers/chu_ple.pdf\n",
"- Projected gradients: https://www.csie.ntu.edu.tw/~cjlin/papers/pgradnmf.pdf"
]
},
{
"cell_type": "code",
"execution_count": 272,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"lam=1e3\n",
"lr=1e-2\n",
"m, n = vectors_tfidf.shape"
]
},
{
"cell_type": "code",
"execution_count": 252,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"W1 = clf.fit_transform(vectors)\n",
"H1 = clf.components_"
]
},
{
"cell_type": "code",
"execution_count": 253,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['jpeg image gif file color images format quality',\n",
" 'edu graphics pub mail 128 ray ftp send',\n",
" 'space launch satellite nasa commercial satellites year market',\n",
" 'jesus matthew prophecy people said messiah david isaiah',\n",
" 'image data available software processing ftp edu analysis',\n",
" 'god atheists atheism religious believe people religion does']"
]
},
"execution_count": 253,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H1)"
]
},
{
"cell_type": "code",
"execution_count": 265,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"mu = 1e-6\n",
"def grads(M, W, H):\n",
" R = W@H-M\n",
" return R@H.T + penalty(W, mu)*lam, W.T@R + penalty(H, mu)*lam # dW, dH"
]
},
{
"cell_type": "code",
"execution_count": 266,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def penalty(M, mu):\n",
" return np.where(M>=mu,0, np.min(M - mu, 0))"
]
},
{
"cell_type": "code",
"execution_count": 267,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def upd(M, W, H, lr):\n",
" dW,dH = grads(M,W,H)\n",
" W -= lr*dW; H -= lr*dH"
]
},
{
"cell_type": "code",
"execution_count": 268,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def report(M,W,H): \n",
" print(np.linalg.norm(M-W@H), W.min(), H.min(), (W<0).sum(), (H<0).sum())"
]
},
{
"cell_type": "code",
"execution_count": 348,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"W = np.abs(np.random.normal(scale=0.01, size=(m,d)))\n",
"H = np.abs(np.random.normal(scale=0.01, size=(d,n)))"
]
},
{
"cell_type": "code",
"execution_count": 349,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"44.4395133509 5.67503308167e-07 2.49717354504e-07 0 0\n"
]
}
],
"source": [
"report(vectors_tfidf, W, H)"
]
},
{
"cell_type": "code",
"execution_count": 350,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"upd(vectors_tfidf,W,H,lr)"
]
},
{
"cell_type": "code",
"execution_count": 351,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"44.4194155587 -0.00186845669883 -0.000182969569359 509 788\n"
]
}
],
"source": [
"report(vectors_tfidf, W, H)"
]
},
{
"cell_type": "code",
"execution_count": 352,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"44.4071645597 -0.00145791197281 -0.00012862260312 343 1174\n",
"44.352156176 -0.000549676823494 -9.16363641124e-05 218 4689\n",
"44.3020593384 -0.000284017335617 -0.000130903875061 165 9685\n",
"44.2468609535 -0.000279317810433 -0.000182173029912 169 16735\n",
"44.199218 -0.000290092649623 -0.000198140867356 222 25109\n"
]
}
],
"source": [
"for i in range(50): \n",
" upd(vectors_tfidf,W,H,lr)\n",
" if i % 10 == 0: report(vectors_tfidf,W,H)"
]
},
{
"cell_type": "code",
"execution_count": 281,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['cview file image edu files use directory temp',\n",
" 'moral like time does don software new years',\n",
" 'god jesus bible believe objective exist atheism belief',\n",
" 'thanks graphics program know help looking windows advance',\n",
" 'space nasa launch shuttle orbit station moon lunar',\n",
" 'people don said think ico tek bobbe bronx']"
]
},
"execution_count": 281,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This is painfully slow to train! Lots of parameter fiddling and still slow to train (or explodes)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### PyTorch"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[PyTorch](http://pytorch.org/) is a Python framework for tensors and dynamic neural networks with GPU acceleration. Many of the core contributors work on Facebook's AI team. In many ways, it is similar to Numpy, only with the increased parallelization of using a GPU.\n",
"\n",
"From the [PyTorch documentation](http://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html):\n",
"\n",
"<img src=\"images/what_is_pytorch.png\" alt=\"pytorch\" style=\"width: 80%\"/>\n",
"\n",
"**Further learning**: If you are curious to learn what *dynamic* neural networks are, you may want to watch [this talk](https://www.youtube.com/watch?v=Z15cBAuY7Sc) by Soumith Chintala, Facebook AI researcher and core PyTorch contributor.\n",
"\n",
"If you want to learn more PyTorch, you can try this [tutorial](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) or this [learning by examples](http://pytorch.org/tutorials/beginner/pytorch_with_examples.html)."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Note about GPUs**: If you are not using a GPU, you will need to remove the `.cuda()` from the methods below. GPU usage is not required for this course, but I thought it would be of interest to some of you. To learn how to create an AWS instance with a GPU, you can watch the [fast.ai setup lesson](http://course.fast.ai/lessons/aws.html)."
]
},
{
"cell_type": "code",
"execution_count": 282,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"import torch\n",
"import torch.cuda as tc\n",
"from torch.autograd import Variable"
]
},
{
"cell_type": "code",
"execution_count": 283,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def V(M): return Variable(M, requires_grad=True)"
]
},
{
"cell_type": "code",
"execution_count": 284,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"v=vectors_tfidf.todense()"
]
},
{
"cell_type": "code",
"execution_count": 285,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"t_vectors = torch.Tensor(v.astype(np.float32)).cuda()"
]
},
{
"cell_type": "code",
"execution_count": 286,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"mu = 1e-5"
]
},
{
"cell_type": "code",
"execution_count": 287,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def grads_t(M, W, H):\n",
" R = W.mm(H)-M\n",
" return (R.mm(H.t()) + penalty_t(W, mu)*lam, \n",
" W.t().mm(R) + penalty_t(H, mu)*lam) # dW, dH\n",
"\n",
"def penalty_t(M, mu):\n",
" return (M<mu).type(tc.FloatTensor)*torch.clamp(M - mu, max=0.)\n",
"\n",
"def upd_t(M, W, H, lr):\n",
" dW,dH = grads_t(M,W,H)\n",
" W.sub_(lr*dW); H.sub_(lr*dH)\n",
"\n",
"def report_t(M,W,H): \n",
" print((M-W.mm(H)).norm(2), W.min(), H.min(), (W<0).sum(), (H<0).sum())"
]
},
{
"cell_type": "code",
"execution_count": 354,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"t_W = tc.FloatTensor(m,d)\n",
"t_H = tc.FloatTensor(d,n)\n",
"t_W.normal_(std=0.01).abs_(); \n",
"t_H.normal_(std=0.01).abs_();"
]
},
{
"cell_type": "code",
"execution_count": 355,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"d=6; lam=100; lr=0.05"
]
},
{
"cell_type": "code",
"execution_count": 356,
"metadata": {
"hidden": true,
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"44.392791748046875 -0.0060190423391759396 -0.0004986411076970398 1505 2282\n",
"43.746803283691406 -0.009054708294570446 -0.011047963984310627 2085 23854\n",
"43.702056884765625 -0.008214150555431843 -0.014783496037125587 2295 24432\n",
"43.654273986816406 -0.006195350084453821 -0.006913348101079464 2625 22663\n",
"43.646759033203125 -0.004638500511646271 -0.003197424579411745 2684 23270\n",
"43.645320892333984 -0.005679543130099773 -0.00419645756483078 3242 24199\n",
"43.6449089050293 -0.0041352929547429085 -0.00843129213899374 3282 25030\n",
"43.64469528198242 -0.003943094052374363 -0.00745873199775815 3129 26220\n",
"43.64459991455078 -0.004347225651144981 -0.007400824688374996 3031 26323\n",
"43.64434051513672 -0.0039044099394232035 -0.0067480215802788734 3930 33718\n"
]
}
],
"source": [
"for i in range(1000): \n",
" upd_t(t_vectors,t_W,t_H,lr)\n",
" if i % 100 == 0: \n",
" report_t(t_vectors,t_W,t_H)\n",
" lr *= 0.9"
]
},
{
"cell_type": "code",
"execution_count": 291,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['objective morality values moral subjective science absolute claim',\n",
" 'god jesus bible believe atheism christian does belief',\n",
" 'ico bobbe tek bronx beauchaine manhattan sank queens',\n",
" 'thanks graphics files image file program windows know',\n",
" 'space nasa launch shuttle orbit lunar moon earth',\n",
" 'don people just think like know say religion']"
]
},
"execution_count": 291,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(t_H.cpu().numpy())"
]
},
{
"cell_type": "code",
"execution_count": 292,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"[<matplotlib.lines.Line2D at 0x7fe4173f1d68>]"
]
},
"execution_count": 292,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHrRJREFUeJzt3XmcXGWd7/HPj4RFZVNoGA1ggoJOXEBpEeciKiIEvHe4\njjqCzqiMGlHw6jjz0gguiI46LLJDiJmALBIUwppAQmLIYhbSHUJn7XRn7yS9ZOslne708rt/VFWn\nurq66lT36a6qk+/79corVeecOvU8fep86znPec4pc3dERCRajsh3AUREJHwKdxGRCFK4i4hEkMJd\nRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBI/P1xieffLKPHj06X28vIlKUysvLd7l7Sbbl\n8hbuo0ePpqysLF9vLyJSlMxsS5Dl1C0jIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuI\nSAQp3EWKWOOBDl54Y0e+iyEFKG8XMYnI4P3wyRXMWVfP+95xPGeWHJvv4kgBydpyN7MpZlZvZqv6\nmf8VM6sws5VmtsjMzgm/mCKSzvZ9BwBo6+jOc0mk0ATplnkYGJdh/ibgE+7+AeBXwKQQyiUiIoOQ\ntVvG3eeb2egM8xclPV0CnDb4YomIyGCEfUL1G8BLIa9TRERyFNoJVTP7FLFwvzDDMuOB8QBnnHFG\nWG8tIiIpQmm5m9kHgcnAle6+u7/l3H2Su5e6e2lJSdbbEYuIyAANOtzN7AxgGvCv7r5+8EUSEZHB\nytotY2ZPAJ8ETjazGuAXwJEA7j4R+DlwEnC/mQF0unvpUBVYRESyCzJa5uos878JfDO0EomIyKDp\n9gMiIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuI\nSAQp3EVEIkjhLiISQQp3EZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEK\ndxGRCFK4i4hEUNZwN7MpZlZvZqv6mW9mdreZVZtZhZl9OPxiiohILoK03B8GxmWYfzlwVvzfeOCB\nwRdLREQGI2u4u/t8YE+GRa4EHvGYJcCJZvb2sAooIiK5C6PPfRSwLel5TXyaiIjkybCeUDWz8WZW\nZmZlDQ0Nw/nWIiKHlTDCfTtwetLz0+LT+nD3Se5e6u6lJSUlIby1iIikE0a4Pw98NT5q5gKg0d13\nhrBeEREZoJHZFjCzJ4BPAiebWQ3wC+BIAHefCMwArgCqgVbgmqEqrIiIBJM13N396izzHbgutBKJ\niMig6QpVEZEIUriLiESQwl1EJIIU7iIiEaRwFxGJIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hE\nkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRQpQY2sHlbXN+S6GFDGFu0gB+sLERVx25/x8F0OKmMJd\npABV1bfkuwhS5BTuIiIRpHAXEYkghbuISAQp3EUiwPF8F0EKjMJdpIiZWb6LIAVK4S4iEkGBwt3M\nxplZpZlVm9mENPNPMLMXzOwNM1ttZteEX1QRSeWu7hhJL2u4m9kI4D7gcmAscLWZjU1Z7Dpgjbuf\nA3wSuN3Mjgq5rCLSD0PdM9JbkJb7+UC1u29094PAVODKlGUcOM5iHYDHAnuAzlBLKiIigQUJ91HA\ntqTnNfFpye4F/h7YAawEvu/u3aGUUEREchbWCdXLgBXAO4BzgXvN7PjUhcxsvJmVmVlZQ0NDSG8t\nIiKpgoT7duD0pOenxacluwaY5jHVwCbgvakrcvdJ7l7q7qUlJSUDLbOIiGQRJNyXAWeZ2Zj4SdKr\ngOdTltkKfBrAzE4F3gNsDLOgIiIS3MhsC7h7p5ldD8wERgBT3H21mV0bnz8R+BXwsJmtBAz4sbvv\nGsJyi4hIBlnDHcDdZwAzUqZNTHq8A7g03KKJiMhA6QpVEZEIUriLiESQwl1EJIIU7iIiEaRwFxGJ\nIIW7iEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4iIhGkcBcRiSCFu4hIBCncRUQiSOEu\nIhJBCncRkQhSuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQYHC3czGmVmlmVWb\n2YR+lvmkma0ws9VmNi/cYoqISC5GZlvAzEYA9wGfAWqAZWb2vLuvSVrmROB+YJy7bzWzU4aqwCIi\nkl2Qlvv5QLW7b3T3g8BU4MqUZb4MTHP3rQDuXh9uMUVEJBdBwn0UsC3peU18WrKzgbea2atmVm5m\nX023IjMbb2ZlZlbW0NAwsBKLiEhWYZ1QHQmcB3wWuAz4mZmdnbqQu09y91J3Ly0pKQnprUVEJFXW\nPndgO3B60vPT4tOS1QC73X0/sN/M5gPnAOtDKaWIiOQkSMt9GXCWmY0xs6OAq4DnU5Z5DrjQzEaa\n2ZuBjwJrwy2qiIgElbXl7u6dZnY9MBMYAUxx99Vmdm18/kR3X2tmLwMVQDcw2d1XDWXBRUSkf0G6\nZXD3GcCMlGkTU57fCtwaXtFERGSgdIWqiEgEKdxFRCJI4S4iEkEKdxGRCFK4i4hEkMJdRCSCFO4i\nIhGkcBcRiSCFu4hIBCncRUQiSOEuIhJBCncRkQhSuIuIRJDCXUQkghTuIiI56Op2fvbsKjbv2p/v\nomSkcBcRycGaHU08umQL1/1peb6LkpHCXSQCHM93EQ47ZvkuQWYKd5EiZoWeMJI3CncRkQhSuIuI\nRJDCXUQkghTuIiID4AV+DlvhLiKSg2I5hx0o3M1snJlVmlm1mU3IsNxHzKzTzL4QXhFFRApHobfY\nE7KGu5mNAO4DLgfGAleb2dh+lvtvYFbYhRSR9LxYkiaCCr0FH6Tlfj5Q7e4b3f0gMBW4Ms1y3wOe\nBupDLJ+IBGAUeNLIsAsS7qOAbUnPa+LTepjZKOBzwAOZVmRm482szMzKGhoaci2riIgEFNYJ1TuB\nH7t7d6aF3H2Su5e6e2lJSUlIby0iMvwKvUdsZIBltgOnJz0/LT4tWSkwNX4p9MnAFWbW6e7PhlJK\nEZECUeh97QlBwn0ZcJaZjSEW6lcBX05ewN3HJB6b2cPAiwp2EZH8yRru7t5pZtcDM4ERwBR3X21m\n18bnTxziMoqISI6CtNxx9xnAjJRpaUPd3b8++GKJiBSmQu9rT9AVqiIiA1Dofe8KdxGRASj0FrzC\nXUQkB4XeYk9QuIuIRJDCXUQkghTuIiIRpHAXEYkghbuISAQp3EVEIkjhLiISQQp3EZEIUriLiESQ\nwl1EJIIU7iIiEaRwFxEZAN04TEREhp3CXUQio6G5nXnrG/JdjIKgcBeRyPjSpMV8bcprw/JehX7r\nX4W7iETGxob9+S5CwVC4i4hEkMJdRPJm9Y5GRk+YTvmWvaGu14dhKItGy4iI9OPVytjJz9lr6/Jc\nkuhRuGfxYsUOzr7xJdo6uvJdFJHIKvRWcDEKFO5mNs7MKs2s2swmpJn/FTOrMLOVZrbIzM4Jv6j5\n8buX1nGwq5uG5vZ8F0UkcoZqxIm+LAKEu5mNAO4DLgfGAleb2diUxTYBn3D3DwC/AiaFXVBJr62j\ni5mra/NdDJFBcZTGYQvScj8fqHb3je5+EJgKXJm8gLsvcvfEGZElwGnhFjP/CrUlcPOLa/j2o+Us\n3xruCSmR4WAU+GDxIhYk3EcB25Ke18Sn9ecbwEuDKVQhKfQLFbbtaQWgua0zzyURGYQCbTxlUujZ\nMDLMlZnZp4iF+4X9zB8PjAc444wzwnzrIafDRpHwJQIy7L1rOPbWQj2aTwjSct8OnJ70/LT4tF7M\n7IPAZOBKd9+dbkXuPsndS929tKSkZCDlHXY6bBQZOmHvXYXemh5OQcJ9GXCWmY0xs6OAq4Dnkxcw\nszOAacC/uvv68IuZf4X+LS1SzIbjoqPDTdZuGXfvNLPrgZnACGCKu682s2vj8ycCPwdOAu632Fdn\np7uXDl2xh49aAiJDZ+iGQjrhHxcUl0B97u4+A5iRMm1i0uNvAt8Mt2hyuPv1i2uYvHATm3/32XwX\nRYaYGu7h0xWqAemzN/wmL9yU7yLIENM5raGjcM9CH73C09nVzVenvEbZ5j35LopIRm0dXTS1deTl\nvRXuUnR2NrYxf30DP3hyRb6LIgUq25H2gYNdTJy3ga7uoT0mv+T38/jgTbOG9D36o3AXKQJtHV3s\nbx/chWqPLdnCL19YHVKJwhVWxAY90r5j9np+99I6nn29z6juUNXsPTCk689E4R6QhmoVnsNpk5z/\nX7N53y9mDmodP312FQ/9bXM4BQpJvkajJa7oPhDhu70q3LOwgJ++bz9axuQFG4e4NHK4aor47SUG\n+kVd29jGlt19f1ov2/oOhyHOCveQzFxdx6+nr813MYbNss17OOeXs2g8kJ+TRRItA729xwW/ncMn\nbn2153nQxtjhQOEeUJg9AI0HOtiz/2CIaxx6+1p7l/eu2VU0HuigomZfTusp37KXD908i8bW/H4p\nbN3dyq4W3aM/bC+v2tnns5LJcIVx+ZY9TK/Y2Wf6YPbrQu8VVLhnMRQfvXN+OYsP/+qVIVjz0Ji5\nupZzb36F1zb1HXqY6+H03XOq2NvawfJtA79FcRh5cNGtcyn99ezBr0h61DW1ce1jy7n2sfKcXxv2\n+ZPUI4HPP7CY6/60vOf5YD5CxXJwoHAP6HA6eZdq6cZYqCe30gf9AT+M/54QG/0yesJ0nlsxtKM1\nhtPBzm4gtxEiod84LNcXDGDHLpYsULhnUyTf0kMpU39orp/zYmn1DLXaxjYAbp8VyfvsDalHl2zh\nAzcNbuRQGJ/DQv8oK9wDK5Kv6xBs3d1KfVNbn+np+kcHOkR0uO+PP2n+Bu6bWz2s75lJovaH+5dd\nLvV/tbKebXta+dmzqwrix2k6urrzXYSMQv2xjig6HPe9i26dC5Dxhl2JoM+55T7QQqWRyxfLb2as\nA+C6T707xBIMXKLsh+PnK8Hd2bHvQM9jd6er2xk5In2b8+sPLeOopHmZtn/Qj8ZAmhgrtsW6J6vq\nWwbw6uGjlntAizYc+v2R6voWRk+YzsaGwt64YUm3o+QzlKIw3O1Qy31wdVlX2wzACxU7qG/ue7RV\naLq6nR8/VUFVXTOPLN7CHxYcujncr6ev5d03vpTxlgAHQ2otZ7phWV1TW8YrV/cOYKRb4nzEcFK4\nB/Tz5w5dtp04CfZimqFVufj4LX/Ny0YfjOdWbKdmb+uhCRHprfqn+//G+EfKhu39El+YYX1NPfDq\nBs7/rzmUbynsm6ltaGjhybJtfPfx5X1GXz26eAsQvLsjjBOb6dbxlclL+cGTK2gZ5O0ekp390+H/\nWWl1ywzCYD9c2/YcoKGlnVEnvimcAg2D709dQclxR/O+dxwP5NZ33tzWwdzKhkGX4Y34YXGY3yvL\nt6Yfr19Z28zZpx4b6tHCwc5u/jA/fjVzyAch62qbOe+dbwt3pUMk+U/qDO/5h0zvVRc/39TfEUSx\nHDiq5T4AmbbtL19YzWd+Pw+IXay0dmdTxnWl9hu2tHcy4ekKmvNwm9DGAx2sr2vud36i3g3N7RyR\n6HPPIWFvm1nZ89g9doJs9ITpNDQHv5ioraOL7z6+PPuC/Vi6cTflWw6Nsa9vauPRxZvTLjt3XT2X\n3Tmfp5eHO1xx8sKNPFm2LdCy2/a08tsZwa98Ttfd0NTW0evCosvvWtBr/qzVtfwlYHlSfeCmmVx8\n+6sDeu0RKSmZeNod8EOVbqkwR8Gk69N//y9mcluRjHBSyz2LTC22dK3W5BszXTVpCWt3NuX0S0IP\nLdzE1GXbOOW4o/nhpe/pNe/p8houGXsqJ7zpyMDry8WXHlzc04ebUNvYxrravl9Qib9KLndMbU/q\ngqpramfWmloAVm7fx8XvPTXQOgY7QuFLk5b0ev7tx8p5vZ9W+4b4OZU1O5rgvNi0v1Xv4tzTT+Qt\nR+e+67S0d3LLy+sYccShz1S2LLr+idd7jlQg9gWc6/ZPveVsaoNj/KOxi46+WHp6TuuF2A24Bjpy\npVfL3aGto7vncbbXuYdzM79067AMDZfBdNWsq23i2dd38ONx7xmW80ZquQ9EwA2T2IleeGNHzm/R\nlfLJWrOjif/4yxv86Kk3Ar1++74DjJ4wnWnLa3qm7d1/MOMOkRrsELt3x5L4RUzJ1U58yHPZwY5I\nCrXHlmxhRHyF3QPM652Ngz+BmHobhJb2TloPxuqWaFkmWpI79h3gK5OX8h9/DrYNUj04bwOPLN7S\nqwGQ2Mkbmtv5c5rWc0fKOZl/vHdhxvcImhnr65pZvrX3VcJd3T7gwLx7ThUfv2VuTq/JVtSOru7A\nJy9Tj/4S1ejo6u5Vp32tB7n49lepqut/MMRQ5e4/T1zMxHkbhu0mcAr3LJK38+gJ0we0jsSJonRS\n96VEAKY2UBO3Jq1rSt+FsSE+gifRyq6Kd6/88M9vcMMzK9m0az8f+tUr/HHR5gHUIKY2aez70vjJ\nsFyiIHWfsZTwzKSto4tP3/5q2lsgJMxdV8/Xprw24C8ciB12j/35TMq37OnZyRPrS9xPfd76gZ03\n6ExzmJN49/GPlvGjpyrY2Zj56s4tu1szzg/q0jvm80/3L+o17V03zODfHl7W83zm6to+J2ib2jrS\n3izu96/031XR1e1Upwwb7Hbv1S2zIWnkWeLWxt+f+jofynCbjuS/5k+mVQC9u6VqG9s468aXuCTe\nTQpw7s2vsLFhP4s37iabsMcK9JxEH6Y+e4V7jpKDI1OGJH8RdGRomr6edMi9v72zZ/2pAZX4QKzY\nto87XlnP3HX1veY/Hz86eG5F7P/kQ/8/Ld3K5vhtUQdzQvPBeX1vadyeZrRPe2cXP5lW0efGXKl9\nrLPX1gF976nd0t7Z64jjB1NfZ+m
gitextract_8s6cyyht/
├── .gitignore
├── README.md
├── excel/
│ └── britlit.xlsx
└── nbs/
├── 0. Course Logistics.ipynb
├── 1. Why are we here.ipynb
├── 2. Topic Modeling with NMF and SVD.ipynb
├── 3. Background Removal with Robust PCA.ipynb
├── 4. Compressed Sensing of CT Scans with Robust Regression.ipynb
├── 5. Health Outcomes with Linear Regression.ipynb
├── 6. How to Implement Linear Regression.ipynb
├── 7. PageRank with Eigen Decompositions.ipynb
├── 8. Implementing QR Factorization.ipynb
├── Homework 1.ipynb
├── Homework 2.ipynb
├── Homework 3.ipynb
├── Project_ideas.txt
├── convolution-intro.ipynb
├── graddesc.xlsm
├── gradient-descent-intro.ipynb
└── images/
└── AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9.url
Condensed preview — 20 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (3,935K chars).
[
{
"path": ".gitignore",
"chars": 23,
"preview": "nbs/.ipynb_checkpoints\n"
},
{
"path": "README.md",
"chars": 17083,
"preview": "## Computational Linear Algebra for Coders\n\nThis course is focused on the question: **How do we do matrix computations w"
},
{
"path": "nbs/0. Course Logistics.ipynb",
"chars": 17943,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/1. Why are we here.ipynb",
"chars": 79902,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/2. Topic Modeling with NMF and SVD.ipynb",
"chars": 137422,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/3. Background Removal with Robust PCA.ipynb",
"chars": 2669489,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/4. Compressed Sensing of CT Scans with Robust Regression.ipynb",
"chars": 272340,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/5. Health Outcomes with Linear Regression.ipynb",
"chars": 29267,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/6. How to Implement Linear Regression.ipynb",
"chars": 80835,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/7. PageRank with Eigen Decompositions.ipynb",
"chars": 292797,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/8. Implementing QR Factorization.ipynb",
"chars": 70625,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"You can read an overview of this Nu"
},
{
"path": "nbs/Homework 1.ipynb",
"chars": 2968,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Homework 1\"\n ]\n },\n {\n \"ce"
},
{
"path": "nbs/Homework 2.ipynb",
"chars": 3408,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"## Homework 2\"\n ]\n },\n {\n \"ce"
},
{
"path": "nbs/Homework 3.ipynb",
"chars": 1599,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Homework 3\"\n ]\n },\n {\n \"cel"
},
{
"path": "nbs/Project_ideas.txt",
"chars": 957,
"preview": "Suggestions for Writing Topics:\n\n- Algorithms to research and implement:\n -- Cholesky Factorization\n -- Gauss-Legendre"
},
{
"path": "nbs/convolution-intro.ipynb",
"chars": 120815,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Intro to Convolutions\"\n ]\n },\n"
},
{
"path": "nbs/gradient-descent-intro.ipynb",
"chars": 44326,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {\n \"toc\": \"true\"\n },\n \"source\": [\n \"# Gradient De"
},
{
"path": "nbs/images/AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9.url",
"chars": 102,
"preview": "[InternetShortcut]\r\nURL=https://photos.google.com/photo/AF1QipMKlINdas9JzCH0FUrOj7FyVKhQUAGBSHTLSbK9\r\n"
}
]
// ... and 2 more files (download for full content)
About this extraction
This page contains the full source code of the fastai/numerical-linear-algebra GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 20 files (3.7 MB), approximately 961.5k tokens. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.