![\"\"](\"images/top10.png\")
\n",
"(source: [Top 10 Algorithms](http://www.cs.fsu.edu/~lacher/courses/COT4401/notes/cise_v2_i1/guest.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are 4 things to keep in mind when choosing or designing an algorithm for matrix computations:\n",
"- Memory Use\n",
"- Speed\n",
"- Accuracy\n",
"- Scalability/Parallelization\n",
"\n",
"Often there will be trade-offs between these categories."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Matrices are everywhere-- anything that can be put in an Excel spreadsheet is a matrix, and language and pictures can be represented as matrices as well. Knowing what options there are for matrix algorithms, and how to navigate compromises, can make enormous differences to your solutions. For instance, an approximate matrix computation can often be thousands of times faster than an exact one.\n",
"\n",
"It's not just about knowing the contents of existing libraries, but knowing how they work too. That's because often you can make variations to an algorithm that aren't supported by your library, giving you the performance or accuracy that you need. In addition, this field is moving very quickly at the moment, particularly in areas related to **deep learning**, **recommendation systems**, **approximate algorithms**, and **graph analytics**, so you'll often find there's recent results that could make big differences in your project, but aren't in your library.\n",
"\n",
"Knowing how the algorithms really work helps to both debug and accelerate your solution. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Matrix Computations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There are two key types of matrix computation, which get combined in many different ways. These are:\n",
"- Matrix and tensor products\n",
"- Matrix decompositions\n",
"\n",
"So basically we're going to be combining matrices, and pulling them apart again!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Matrix and Tensor Products"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Matrix-Vector Products:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The matrix below gives the probabilities of moving from 1 health state to another in 1 year. If the current health states for a group are:\n",
"- 85% asymptomatic\n",
"- 10% symptomatic\n",
"- 5% AIDS\n",
"- 0% death\n",
"\n",
"what will be the % in each health state in 1 year?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"floating](\"images/markov_health.jpg\")
(Source: [Concepts of Markov Chains](https://www.youtube.com/watch?v=0Il-y_WLTo4))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"import numpy as np"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.765 ],\n",
" [ 0.1525],\n",
" [ 0.0645],\n",
" [ 0.018 ]])"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: Use Numpy to compute the answer to the above\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Matrix-Matrix Products"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"![\"floating](\"images/shop.png\")
(Source: [Several Simple Real-world Applications of Linear Algebra Tools](https://www.mff.cuni.cz/veda/konference/wds/proc/pdf06/WDS06_106_m8_Ulrychova.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 50. , 49. ],\n",
" [ 58.5, 61. ],\n",
" [ 43.5, 43.5]])"
]
},
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise: Use Numpy to compute the answer to the above\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Image Data"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Images can be represented by matrices.\n",
"\n",
"![\"digit\"](\"images/digit.gif\")
\n",
" (Source: [Adam Geitgey\n",
"](https://medium.com/@ageitgey/machine-learning-is-fun-part-3-deep-learning-and-convolutional-neural-networks-f40359318721))\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Convolution"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"*Convolutions* are the heart of convolutional neural networks (CNNs), a type of deep learning, responsible for the huge advances in image recognitionin the last few years. They are now increasingly being used for speech as well, such as [Facebook AI's results](https://code.facebook.com/posts/1978007565818999/a-novel-approach-to-neural-machine-translation/) for speech translation which are 9x faster than RNNs (the current most popular approach for speech translation).\n",
"\n",
"Computers are now more accurate than people at classifying images.\n",
"\n",
"![\"ImageNet\"](\"images/sportspredict.jpeg\")
\n",
" (Source: [Andrej Karpathy](http://karpathy.github.io/2014/07/03/feature-learning-escapades/))\n",
"\n",
"![\"ImageNet\"](\"images/InsideImagenet.png\")
\n",
" (Source: [Nvidia](https://blogs.nvidia.com/blog/2014/09/07/imagenet/))\n",
"\n",
"You can think of a convolution as a special kind of matrix product\n",
"\n",
"The 3 images below are all from an excellent blog post written by a fast.ai student on [CNNs from Different Viewpoints](https://medium.com/impactai/cnns-from-different-viewpoints-fab7f52d159c): \n",
"\n",
"A convolution applies a filter to each section of an image:\n",
"![\"CNNs\"](\"images/cnn1.png\")
\n",
"\n",
"Neural Network Viewpoint:\n",
"![\"CNNs\"](\"images/cnn2.png\")
\n",
"\n",
"Matrix Multiplication Viewpoint:\n",
"![\"CNNs\"](\"images/cnn3.png\")
\n",
"\n",
"Let's see how convolutions can be used for *edge detection* in [this notebook](convolution-intro.ipynb)(originally from the [fast.ai Deep Learning Course](http://course.fast.ai/)) "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Matrix Decompositions"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We will be talking about Matrix Decompositions every day of this course, and will cover the below examples in future lessons:"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- **Topic Modeling** (NMF and SVD. SVD uses QR) A group of documents can be represented by a term-document matrix\n",
"![\"PageRank\"](\"images/page_rank_graph.png\"){kind=icon}
\n",
" (source: [What is in PageRank?](http://computationalculture.net/article/what_is_in_pagerank))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- List of other decompositions and some applications [matrix factorization jungle](https://sites.google.com/site/igorcarron2/matrixfactorizations)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Floating Point Arithmetic"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To understand accuracy, we first need to look at **how** computers (which are finite and discrete) store numbers (which are infinite and continuous)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Exercise"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Take a moment to look at the function $f$ below. Before you try running it, write on paper what the output would be of $x_1 = f(\\frac{1}{10})$. Now, (still on paper) plug that back into $f$ and calculate $x_2 = f(x_1)$. Keep going for 10 iterations.\n",
"\n",
"This example is taken from page 107 of *Numerical Methods*, by Greenbaum and Chartier."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def f(x):\n",
" if x <= 1/2:\n",
" return 2 * x\n",
" if x > 1/2:\n",
" return 2*x - 1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Only after you've written down what you think the answer should be, run the code below:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"x = 1/10\n",
"for i in range(80):\n",
" print(x)\n",
" x = f(x)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What went wrong?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Problem: math is continuous & infinite, but computers are discrete & finite"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Two Limitations of computer representations of numbers:\n",
"1. they can't be arbitrarily large or small\n",
"2. there must be gaps between them\n",
"\n",
"The reason we need to care about accuracy, is because computers can't store infinitely accurate numbers. It's possible to create calculations that give very wrong answers (particularly when repeating an operation many times, since each operation could multiply the error)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How computers store numbers:\n",
"\n",
"![\"floating](\"images/fpa.png\")
\n",
"\n",
"The *mantissa* can also be referred to as the *significand*."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"IEEE Double precision arithmetic:\n",
"- Numbers can be as large as $1.79 \\times 10^{308}$ and as small as $2.23 \\times 10^{-308}$.\n",
"- The interval $[1,2]$ is represented by discrete subset: \n",
"$$1, \\: 1+2^{-52}, \\: 1+2 \\times 2^{-52},\\: 1+3 \\times 2^{-52},\\: \\ldots, 2$$\n",
"\n",
"- The interval $[2,4]$ is represented:\n",
"$$2, \\: 2+2^{-51}, \\: 2+2 \\times 2^{-51},\\: 2+3 \\times 2^{-51},\\: \\ldots, 4$$\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Floats and doubles are not equidistant:\n",
"\n",
"![\"floating](\"images/fltscale-wh.png\")
\n",
"Source: [What you never wanted to know about floating point but will be forced to find out](http://www.volkerschatz.com/science/float.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Machine Epsilon**\n",
"\n",
"Half the distance between 1 and the next larger number. This can vary by computer. IEEE standards for double precision specify $$ \\varepsilon_{machine} = 2^{-53} \\approx 1.11 \\times 10^{-16}$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Two important properties of Floating Point Arithmetic**:\n",
"\n",
"- The difference between a real number $x$ and its closest floating point approximation $fl(x)$ is always smaller than $\\varepsilon_{machine}$ in relative terms. For some $\\varepsilon$, where $\\lvert \\varepsilon \\rvert \\leq \\varepsilon_{machine}$, $$fl(x)=x \\cdot (1 + \\varepsilon)$$\n",
"\n",
"- Where * is any operation ($+, -, \\times, \\div$), and $\\circledast$ is its floating point analogue,\n",
" $$ x \\circledast y = (x * y)(1 + \\varepsilon)$$\n",
"for some $\\varepsilon$, where $\\lvert \\varepsilon \\rvert \\leq \\varepsilon_{machine}$\n",
"That is, every operation of floating point arithmetic is exact up to a relative error of size at most $\\varepsilon_{machine}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### History"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Floating point arithmetic may seem like a clear choice in hindsight, but there have been many, many ways of storing numbers:\n",
"- fixed-point arithmetic\n",
"- logarithmic and semilogarithmic number systems\n",
"- continued-fractions\n",
"- rational numbers\n",
"- possibly infinite strings of rational numbers\n",
"- level-index number systems\n",
"- fixed-slash and floating-slash number systems\n",
"- 2-adic numbers\n",
"\n",
"For references, see [Chapter 1](https://perso.ens-lyon.fr/jean-michel.muller/chapitre1.pdf) (which is free) of the [Handbook of Floating-Point Arithmetic](http://www.springer.com/gp/book/9780817647049). Yes, there is an entire 16 chapter book on floating point!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Timeline History of Floating Point Arithmetic:\n",
"- ~1600 BC: Babylonian radix-60 system was earliest floating-point system (Donald Knuth). Represented the significand of a radix-60 floating-point representation (if ratio of two numbers is a power of 60, represented the same)\n",
"- 1630 Slide rule. Manipulate only significands (radix-10)\n",
"- 1914 Leonardo Torres y Quevedo described an electromechanical implementation of Babbage's Analytical Engine with Floating Point Arithmetic.\n",
"- 1941 First real, modern implementation. Konrad Zuse's Z3 computer. Used radix-2, with 14 bit significand, 7 bit exponents, and 1 sign bit.\n",
"- 1985 IEEE 754-1985 Standard for Binary Floating-Point Arithmetic released. Has increased accuracy, reliability, and portability. [William Kahan](https://people.eecs.berkeley.edu/~wkahan/) played leading role."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"Many different ways of approximating real numbers on computers have been introduced.. And yet, floating-point arithmetic is **by far the most widely used** way of representing real numbers in modern computers. Simulating an infinite, continuous set (the real numbers) with a finite set (the “machine numbers”) is not a straightforward task: **clever compromises must be found between, speed, accuracy, dynamic range, ease of use and implementation, and memory**. It appears that floating-point arithmetic, with adequately chosen parameters (radix, precision, extremal exponents, etc.), is a very good compromise for most numerical applications.\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Although a radix value of 2 (binary) seems like the pretty clear winner now for computers, a variety of other radix values have been used at various point:\n",
"\n",
"- radix-8 used by early machines PDP-10, Burroughs 570 and 6700\n",
"- radix-16 IBM 360\n",
"- radix-10 financial calculations, pocket calculators, Maple\n",
"- radix-3 Russian SETUN computer (1958). Benefits: minimizes beta x p (symbols x digits), for a fixed largest representable number beta^p - 1. Rounding = truncation\n",
"- radix-2 most common. Reasons: easy to implement. Studies have shown (with implicit leading bit) this gives better worst-case or average accuracy than all other radices."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Conditioning and Stability"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Since we can not represent numbers exactly on a computer (due to the finiteness of our storage, and the gaps between numbers in floating point architecture), it becomes important to know *how small perturbations in the input to a problem impact the output*.\n",
"\n",
"**\"A stable algorithm gives nearly the right answer to nearly the right question.\"** --Trefethen\n",
"\n",
"**Conditioning**: perturbation behavior of a mathematical problem (e.g. least squares)\n",
"\n",
"**Stability**: perturbation behavior of an algorithm used to solve that problem on a computer (e.g. least squares algorithms, householder, back substitution, gaussian elimination)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Example: Eigenvalues of a Matrix"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[ 1. 1000.]\n",
" [ 0. 1.]]\n",
"[[ 1. 1000. ]\n",
" [ 0.001 1. ]]\n"
]
}
],
"source": [
"import scipy.linalg as la \n",
"\n",
"A = np.array([[1., 1000], [0, 1]])\n",
"B = np.array([[1, 1000], [0.001, 1]])\n",
"\n",
"print(A)\n",
"\n",
"print(B)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"np.set_printoptions(suppress=True, precision=4)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"wA, vrA = la.eig(A)\n",
"wB, vrB = la.eig(B)\n",
"\n",
"wA, wB"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"hidden": true
},
"source": [
"**Reminder: Two properties of Floating Point Arithmetic**\n",
"\n",
"- The difference between a real number $x$ and its closest floating point approximation $fl(x)$ is always smaller than $\\varepsilon_{machine}$ in relative terms.\n",
"\n",
"- Every operation $+, -, \\times, \\div$ of floating point arithmetic is exact up to a relative error of size at most $\\varepsilon_{machine}$ \n",
"\n",
"Examples we'll see:\n",
"- Classical vs Modified Gram-Schmidt accuracy\n",
"- Gram-Schmidt vs. Householder (2 different ways of computing QR factorization), how orthogonal the answer is\n",
"- Condition of a system of equations"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Approximation accuracy"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"It's rare that we need to do highly accurate matrix computations at scale. In fact, often we're doing some kind of machine learning, and less accurate approaches can prevent overfitting.\n",
"\n",
"If we accept some decrease in accuracy, then we can often increase speed by orders of magnitude (and/or decrease memory use) by using approximate algorithms. These algorithms typically give a correct answer with some probability. By rerunning the algorithm multiple times you can generally increase that probability multiplicatively!\n",
"\n",
"**Example**: A **bloom filter** allows searching for set membership with 1% false positives, using <10 bits per element. This often represents reductions in memory use of thousands of times. \n",
"\n",
"![\"Bloom](\"images/bloom_filter.png\")
\n",
"\n",
"The false positives can be easily handled by having a second (exact) stage check all returned items - for rare items this can be very effective. For instance, many web browsers use a bloom filter to create a set of blocked pages (e.g. pages with viruses), since blocked web pages are only a small fraction of the whole web. A false positive can be handled here by taking anything returned by the bloom filter and checking against a web service with the full exact list. (See this [bloom filter tutorial](https://llimllib.github.io/bloomfilter-tutorial/) for more details)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expensive Errors"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"*The below examples are from Greenbaum & Chartier.*\n",
"\n",
"European Space Agency spent 10 years and $7 billion on the Ariane 5 Rocket.\n",
"\n",
"What can happen when you try to fit a 64 bit number into a 16 bit space (integer overflow):"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAUDBAgICAgICAgICAgICAgICAgICAgICAkICAYHBwcI\nCAgHChALBwgaCQcHGCEYGhERHx8fBwsgIBYeIBwSEx4BBQUFCAcIDAkJDRIMDQwSEhISEhISEhIS\nEhISEhISEhISEhISEhISEhISEhISEhISEhISHhISEhISEhISEhISEv/AABEIAWgB4AMBIgACEQED\nEQH/xAAdAAACAwEBAQEBAAAAAAAAAAACAwABBAUGCAcJ/8QAUhAAAQMCBAMEBQgGBwUFCQEAAQAC\nEQMhBBIxQQVRYQZxgZETIqGx8AcyVcHR1OHxCBRCUqSlFRcYI2KS0zNERXKTJTQ1Q3UWU1RjZISU\nldIk/8QAGAEBAQEBAQAAAAAAAAAAAAAAAAECAwT/xAAiEQEBAQEAAgIDAQEBAQAAAAAAARECEiED\nMRNBUWFxgSL/2gAMAwEAAhEDEQA/APjJRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRR\nRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFE\nEUUUQRRRRBFFFEEUUUQfo7PkP7UOcWjhnrC8fr3Dhprribo/6iu1Vv8AsrXT/wD28O+8r7ZwbAQH\nOBz5b5gJbOostTGDujbkvV+GOfk+Hx8gnaz6J/juG/elY+QPtb9E/wAdw370vueg2ei1Uqcn2Kfh\njWvhFv6P3a4/8I/j+GfekbP0ee2B04P/ADDhf3tfetNsbdFtpMBIssfjhr4AH6OnbL6G/mHCvvaY\nz9G3to4SODW/9R4T9eMX9CaLCbbLTRpQRayv44a/ngP0ae230L/MuE/fEQ/Rm7b/AEJ/MuEffF/R\nqk0zcRy7k1gv9a42Y0/nEf0Yu3Gn9CfzLg/1YxF/Zg7c/QfT/wAT4P8AfV/SCgeVu9aGGPKVGsfz\nY/sv9uvoP+Z8H++qv7L/AG6if6D/AJnwf76v6VCoLoqZt0RMfzUH6L3br6D/AJnwf76r/su9uvoP\n+Z8H++r+lxVBylpj+aX9l3t39B/zPg/31Cf0YO3X0H/MuD/fV/S3NdC56pj+aR/Rj7cD/gf8y4R9\n8VH9GXtwP+CfzLhH1Yxf0me5Ie625jTkrF8X83Xfo09thrwX+ZcJ++IP7N3bT6G/mPCff+tr+jFZ\n0iIjms1VxNo0W/CHi/ne79G/toP+DD/9lwn74qP6OPbP6G/mPCtv/u1/QlzjrB+ClVH3nTVbnxxm\nv58H9Hbtj9D/AMw4X97VH9Hjtj9EfzDhf3tf0Df7ktzGq/ijfi/n+f0ee2H0R/H8L+9Kh+j12wP/\nAAj+P4X96X3+T7ELWztron4oTl8Cj9Hftj9D/wAw4X97Vn9HTtl9D/zDhX3tf0DpNAsNXWHerey1\n/NZvxyL4R/Ps/o6dson+hrf+ocL+9qh+jr2x+h/5hwv72v6CMBM6/OVZNL5T77lZ8IzeX8/P7O/b\nH6H/AJhwv72hd+j12wGvCNLf+IcL+9L+gT2Eam2v1bJD9b3v7IXSfFF8XwJ/Z57YTH9EfzDhf3tT\n+zz2wt/2Rrp/2hwv70vvZl/HW231KvS7ck/FDxfBP9nztfp/RH8fwz70qP6Pva/6I/j+Gfel97vd\nqRdJfon4oni+DXfo/drhrwn+P4Z96Q/1A9rfon+P4Z96X3ZUcDE6z7Ekkyb2JT8UMfDDvkE7WDXh\nP8fw370l/wBRPar6K/juG/eV9zucb/urM+esdwUnxxl8Q/1E9qvov+O4b95Qn5De1P0X/HcO+8r7\nYrOkQdNkh17b7LV+KQfFh+Q/tR9GfxvDvvKE/In2n+jP43h/3lfZxfHfoe9Iq2N99FJ8UHxv/Up2\nm+jf43h/3lUPkV7TfRun/wBZw/7wvsN52AS33m/gp+KI+Pv6mu0v0b/GYD7wh/qd7R/R38XgPvC+\nvq/LlokuGsnvU/HDXyJ/VD2i+j/4vA/66r+qPtD9H/xeC/119ZmIPek1dN/Ba/FFfKB+SftAP9w/\nisF/roD8lfHh/uH8Tg/9dfVJHxus9RxU/FB8uO+S7jo1wP8AE4P/AFks/Jpxsf7if/yML9VZfUFf\nnHislSTf2yl+OD5nd8nPGhc4J3/Ww3+qln5P+MD/AHJ3/VofVUX0pUnbN5pDp0k+e6z+MfNx7C8W\n/wDg3f8AUo/6iF3Yjio1wjv+pR//ALX0RVnmfNIcDuT5qeC4+i8IyS4nd7p74WgM6JWHacz4gtzC\nO/LdaxTO1/FevXllUxmi10dQhYzTn5p9KldY6uukp1Bstk87LdTZEWlKw1DbkZ6LfRpkfgsxYNgN\nrLVk306KmMtfZNAJAcfCdVi9KOm0jx9iaWHnYXgeSWwApjWDa1oPfMrlWoJp3P4hNpg7adVGUzFr\nqgXDqstrc6LbSownXW+nJW6nZCGQY6abIezWm+u/1K3G5hLLJ3vqraZHK6EMBPRKqH1fco4xEJTw\nT4JFA8lKqg622TXbCdUqoLHv0C3IuslV0Tfn71neCeS1VGgg2OqQ/wBXpaFoZnNMRB87eKQ9hkbT\npF4T67p0CTiRb3de/mtzpislUGbKg345poAtOvsRFnLXpaAr5NB9FZEykf2SiE6m+n4oCTtzlL22\ndJnu5c+itzHjUkA6QOSS07g669Fbau0yeunVZvSznTGhw5GTZA8Dl+SJ1SI5TIKRUcTGwH5pEsG9\nvIrNUBlObfSY/BJqDS569y1rLI8kEgaHVSbDqreAbwRtyS4Me7orKKLhoNSUovIm/wCShIBGp6oK\njh175v3JahTxJ1sCDbvQVT15+9XUfy96Q65vskqWrc+AZus9R5EHojeTokv/AAVZKe/2pbzAJgdT\nv4InHRKde/groS8mxFvs5pdRx+OSZUZ1j60stnUdEC3tJ03SjTMfinkeXLZZ3gSbKJS6oEWv36pM\n+1Pe2yQR0hSIz1fFJdZPq9SkvWljLUGvl9azvJ0Wmo3n4pL+ntWVZag5rJWp6i+kra88/wAlnrMv\nqrRiqjvSaoun1R5C3XvWd9vdBWRlqCPNZnla64sstQQSmLK+mKbR6oEC3K+kT3p1Fulvjql4Sk4T\nJmdOYW7D0CdPGdVrrqPIOhTB21+LLZTpRFp2Q4ejsfgLdSYQbBc7WuRMp/vDyWqjSjuQlPolZtdI\nJlOU1tMxeY77JlDyTxSsdYJWNWECkLTbqEdOmLkRBTMvId07Ii02iLahZaAwkG0pu55BLGtz5WVh\nonUwi6IE3kRGl1QfJkBBUbYoc0W840RRCdvbeyqoqJMeKF5I5ePxqihe6LRc6fggBO8hU8k6n7fB\nDczr4qwR5S80zrzVOn8UhwvO3etxIXXqnqPj3JNWTF5PsTXtmZ35qhTtb43RrYQ5tiN5VGjNjNlt\nY0G5VVaon1QDa6eSubUZASQ+8xYWiVrxzyNWwCuW98Sdvi6nk3hrqsm9gLQqdVnS34LBUqHz5IRW\nI6d6s6i5rc5/f4WUa4mO7RYTW/xJjXbzJAidlG76je11vqQVH6cllZUPO6jqvx15rTlWh9WNNR5J\ndSuTp8DVJc/Qb7lLL41+N1WR1DKU7TdC6tFvbKTUqC9ygt7+WoQ1XmPylU8zOk8wkF/cVUQu84SK\njeuuoTajeYHO2sLO8xzKrFUec30Wd7jomvFrd6UVqBYMEW6Jbjp8bonNO53lLImAO6fyVAudPhp+\nPNLcT+SMj2FJcdZWoBcZkT9qXU8EbhZZ3W9qylR5gbykvMpjzBSnn46KRCX8/Kb2SahHNOPn37dE\nl4I2F1vmhFaVnqbp9Qfks1Qws1dKeFlrR5BaHFZ6rd+atVles+IbF99+5aqjfiUl+3VYGGqkVBut\nlQSYWWppGv1JSPqmlRtcLXSpAR8eapogSdk1iy8x9KlF9eXcntAS2d/VMaDYxbT61Mbkw1nLZOps\nQMF4Gy10h4LKjoifBamVNvJJpiO9NidNlzrQkDu+Dv16I2+1A436hFAmMEnRWATqiAhAFRsfWlZe\n/qE/qUpz9xEct0aDB6/gge1EXxpv+yUsVOnX4KKB406aISw5Y5o84B5cjyVVHk9VYpD9B01WY3m2\nu61VHWSWnVaZKFP3bom07W0VwnU6U2JUtXACkIidUdPDR+wTJseiaaUjYRz0S8PxXC7VmToRm3Cz\nW5A4jBF4h0CNF5nieGNMmbTp1ScV2mrekPreq1+gGt0XFcd6d2c+rAsO8LM9ukuORVqb9I5bpNOt\nrI9v2p+KYADdc6sQNVvwZ8pG+m8DkURxTQSOa4r6hHcoKpN4W+ecZveu2MU3VQ4kSuGzEcgjGIPP\nRakZ12P1kKqlaVyXVtyVPTRBGm6LrpFw5ylF0wkmuHQRorkeSJ5Cc481QfEpebb2KMf4IyIH4+NE\nB7kJd5JbqnkqKc/8kl0o3QlOdcX2VlAudzSs/LY/V7UVWpzSS4HvGncqyokhJcic/fZLJVUNTbVL\nnx1TCRbkNUioTJ5bIKdyS536K/eqeb+CIQ7dLfoFbjZJefeqFVHbJBCeW6pNQRpqiUtwhIe2dh4r\nRFo3S3Ka1GR7Gjbe/ks1UaX30WurN40WaobdUtGSoDJ8llqtieZWyoTB71lrqfofWAafzRtBmTdN\np0xrPfKY1m9ljXHnkbGzEJ7Bsgo31stI2hTWotjPsWigw+G6BkaEx3LS3WLxCxauHNCNqodytvJZ\naR56oQ6NkwgIQEEAVE3mEXsUbCkaUDF0tzeQsjkdUOvRVcZ67CIIAS3ag3jktfulJqmfNE0qoCRd\no6DolE2vaE17xmtyQlu096rTO4bFKe0jS60VAgbGk3TQovHK6bTdG5vYoH2uhc7NI6H3J9o8l8oH\nHzSNOnRqOaQ2Kkb96/N28ZeysHxebjmvQ9uOFPo1nPkuY4znN41suHwbhratQ5xmbElPFryb8PxX\n0hzfta2tqtf9JGPWfcLiYuiKbyGi0x4DRZX1tV0nEYtegfxM85CA41pF7nmuC2vG/mocR3LeRHYd\njBz81nOI0PnC5j8QD8X8FRryJnzS8jrnESNYU/WBGq5Irz0TDWkWOmvJDXU/W46j61oZWBHzvjl3\nLhvrRHfCKjiInlNx0/NDXWFfJJFxy5dQtVOs1zZGbrdckVfHfwS6eKyEQTDrEbApIrtmpurFTcLI\nHyL6j2o6b9tCVMGguQVHJZPjt9at52zRv+CCFxHX3pTjvp13VuIHM9Up/wASqI907JThvItaNwm6\nckp8HX2LTJbjYzISyB7ET/GEs76oAfCB1j0RIHiUA1Cl1DZMiUt4SBDuaS4SnvCU4bc1oJc0+aSQ\neQWipa06JTiAs2pShaZWd6bVekPBuVPtqEPHks1R21vBPqGVnduoM9S9llqNMrU9tz3LNWPxsiPq\ntte15up6fbSAsNCq43W+hQLiJG11K48io4hx3vomsxbx5rVQwgEIxhddNVn02LC4jN0ldGjWMwuZ\nTokarXh325rKz7dpkKySk4OoXCeSeFzranSfJRqsqKKtyp0fkrKoBUUR4KZUGYi2qE1SqJUgJFVw\n23uUZfrKzOqydERTxN48kDXRKt5KQ43ke1DTnGxSXkD8Ep1W8JbnjWCtYaKrVsliv6piNPeYVVKg\nNuiy1HAfHRXFcbtpiabaBa4Zi6zB11uvA8JcTXYGkgEwRzBXsO2+Z1FoYyb3PL7F4fBYg06rdocJ\nKW4ldLi2GknlJ9i4VanuOq9bjmy0EEEOkg964FWhdv8AikLcvpXIIPelPGq3PpxMdyQ+jfUaT4qy\nssubnspTI3mFC2/sQVFb0sMa/wAPFFTfrfVLAnT3BDedN1NRqDrwjzX02j67rPGlt9lowzoBsCQZ\ng62MFNbkacEdZ2Em99YhC62ZvO7eaGm4tc12UZb5o5HvRvf6zSOceHLvTRq4e9zmHcsMXOo+tac8\nO3uuXZtQOEi+k2vzWh1aNjGbyCsSuiKk/WhDtUllWCDaHe9R7yCD593NEPdzBug+LqZ+SF5lWIko\nZF9JlWZSnmPj3qiO8bpb0RQlAsoXKyUB2RFPPJKeEZSnPCFpZSqhHjKJz1ne+471SKfqkG/4Jj3X\nsk1HctVmhTzsEio6+qaXQkPI5KGlvO3NZ6oRVCkPcqpdUrNXTKjlnquKU19W4XDe/wBi6GFpwSb6\n27lno1csXWzDVhGqxXHlups0+NkynTmddbJLH6XWyg/SDdYdIW7CjzTqGFIjSE6k2+m1+i0sELNr\nUVRp5bDRNaVRIS31FhTELnDcwsOJqnnCQCT3c1odP0ojW6F+JA6LnsOXdSQdborWytmnL9nvUzCN\nYIOvVZxFjEqOqWi3kgbWvJGu/RZag9qs1Iknfw9iW582F0Slk+xC6fzROjldZ3uMrSJWdusbz006\n6pzpNys9Vp0MStQKcOXW2u3NJeeenjyTqkx3ckioSdSYHNaajHj6jGU3F8FsG06r884lVbmJAaBc\nxvK6/bXGPbVAmGtEj3LxdcuqOJEnzXO9/osev4FixWZk/wDdWVV2AQY0d9aDs/w5+HYxxcf74A5Y\niPsWvEwc0iZd5XWv0SubXpDMfKOuq59VonTp+C7NZgzm9wR7lixtP5sbyo05TqcyCNO5Jq0rrS9u\n41lIqkG6upgaTOfmjbRu7bdLD9L3mPNOz+sJ0dYe5AbacAc9fBPFJskxMR7UNEg5h0MDuunYW7iP\n3mDKOo1SqzPYRpcA+r156og0WOsO7k+oAWtEXvH/ADD4KytMkjWbgK6GYunYHk4/ghZU9U31F9/c\ntNOnmaQ9wAyZgdZPJZabcpIgQ4SElDMFUmRoYkTudPctDrjvt4rnMBEXOuvXl5LW18kfuPHk8Gy1\nrNPw7zoeUeSM9Fm0JvGaP8wTWW1uDorKmCsDcd0FVVIN9FTiTpsgkFaQQS6h+PrVlxQv1QpZ5e3m\nqnyRP5JTzARA1J3KQACjc9Ie66CnG6XU+O9U915SKtZQU9Z6jkVV3ks76t+QUsRVUpNQi91VRw2K\nTVddBdQhZqpn42RVH69yyud7kaSq8WHJZahnojqnVZ33VI+sqVYbkeSdSfcQVhzx4rXhzpaVhynL\nr0KkRK34eoFxw/QAx7loo1TdZ6jcd70kDQo2VgVzaFfSb96dSqjW3mudajYTcH2fYh9OCY8ikio3\naeisg67e1FwNZpcQCdELhEtjU6pnpR5bHXw6JOIqxt1QKe/KkHEg9EvFVLjlv9iyVB3lXDHTpV+X\nrIi52unvWClm31RmuSbnRMMac5J57SVTqgWSpiI6pRrbpi43mr6tkio/uWZtUmUh9SFZDGt9QAEz\nrsEqu+3MTqsr624JCW+vY3t71pLBvqXOq5/F8W+nTLmesR7LarQ+vItZZa7pBBEg2uqSvy7jeOqV\nXmbmbg9673Y2jQexzz61UatJ2XL7UYH0dUkHU37pW3sJGesG7jXpy7lwk/8Apq/T01VskHouZXua\ngi0D2Ls1gYHOdVzvRyXf8seMmV2tYc3Mc5MaQUBaTMjY+9an04zSOnkbIXgB4gasM/HNRqVxjRbM\nExAzD3QufWokECNS65XVLb2G512ScVTAIP7XMrn7dHIcyZt0CtjyQzoYW99NoYIs4uiO9ZcJTdBa\n4Qc5IPPkrOkw2k7K8SmB5a8f4TfuddAaZIzg62vsd1eIF37WHmBMremNIJBcJn9oLHiIY7ONdVqZ\nSd88QYDSTyBsbbocSxsAiPVdlcdTfQwluGaZnkZv+UgJOJbuOfKRB2R0wRIBHRpH7O0KFsw0Wm99\nsuquoN7AG/OaQ8elAaZhzPVynkYCAG9vmvHpGj/FoR3oJAzSPmuDidodyCItsWi7mHOw8wRok9oY\n5oc251kjaHjTuWzgeNZSe2pUotrjK8GnVMMLnMyjT4ssFB2o1Dhmb3j5w80TT6xG5Ae20afOgbJc\n/bU1oefnEazI80p03sIiyIdwQVHcvYunN9MdX2suj7Ep56FCXwJSK1WVqVk4myzVKsdyB9UrPUfJ\nWUFWqefJA6prISqlRJfUTQdSoPD61lq1JVVHSNVnJ6poZUqrK5+3JW8pLz1SUxHPWd71HO/JJqOT\nyTEqPlIqvVvclPciheUio/ZE9yS42Upj61osMSfxWrCtKy0n36rbSeIupWTwy3tTWPj7NkplXxHL\n61ootzDMYtsPxUakMZUn3ptKpGqgpjWItZE9ukLnVaGu1WqjUae8LC1h2UcHzv4KK01nNJtr7fyS\nK3VDkPU9d0BpE2lCfbPVbJht1baDhc+S20qIbEaonj4KrVYnM5hA9ka2Wuo4d/ck1b6jzSEYKl9j\nayU8xAuFoqU728b/AGLNWZIkzCqpUI2KQQbz4ImvAEbe1LfVzWK2gTH2DdBUGvJG0gGNeZ+xLqQC\nBeOqM6S7vSzP2J1SAlVDYnYb7KXrCPCdtix1SWu3hwOnhCrsPScKzzbIG3IRdpAPSGACHG/RN7J0\n8tX9oDLYbFcp76bv09UXRrfcc1ka351lrzXnokmp6pC645sZZAMnUzz1SK4AM8hput7cvqicrDDX\nu1j1rmyOvh6OaoA/OQQKRykZxEEidAs7+nSc/twKlM5mi17/AJpNfDmSHDeI6dF0atOH7erM/hzQ\nVGn1dzMk9NolZbYMZhqP6u5/pD6RroFPITLA0lxzDeViwjMzMzheARF9dO8LsYkNHpG+uGFrhfLm\nuNossGGLWk04PqBoHdE7brn9VfKWSMbqVgI0JBPfe3mmOpwSCNGgR36LeMLmJBgT82XDnqZV1cjR\nVBbmqEtGbZrWiSR/ildNTxc6kYa6JJc0t6t7oT8eBUYCGMZNJvzNT6Pdw3KHDNklskAzd2xOkxom\nsw803P8AVhvqkgwSCYIbN5Ut1cxhaHEWmWASBErQ4eqfrkcpv4oTR9G+Hf3YNPOCTdw5QN10qhFa\nnRZTYWGlRd6QveGl5BJzBp1Fx5K3o4n3WbHcGfQa1znU3sdDC+m8OALmCo1hH70ELIwH+7eASRmp\nmNTsO4x7l2K3D3jC0XkS1zS9j6IJyuY4l7a0/t5fZC1disTSoYsem9GKdem5pe8A5Q9shw/dcuU+\nXJa69fFNjz2EwVR/pQ3IPQevd+U5ZGbLPf7F6bs5hqoq18NhnYOr6XDgvqVSMsOHrCkX3a/Vcrj1\nGizF1G0XmrRzhnpIn1XQTduo18lq4jUwFD0jKA/W2VKAa2q+WPoVJvlDNRCnXVv1+1nMjg17EiQY\nLgO4OgGd0mqY+xdDheMo0vTenw4xGei5lPM8g06hMteI2XIdU5T4mV6uerufx5u5M3+hqVEiq7fq\npUqc0h9RdNcll/1pLn2VOqpD3pqLqvnRKqO6KVKnSEhzupWVxHuskOcre8T7Emo+9u9UwL6lkDn+\nxA959qU52qm4VHuSHOujcbmEs81EA5yW53O6MoHmFrVwk3nml5iAmPbugPTZQfXNDuWqmBFwsNM2\nWqm9Kw1UWRvC2Ug2QOaxU38+SfQdrA0vKjTe11u4omt3Wak6T8arQ1y51WhtO9kxrdBzS6Rk38E5\nl/P6lFGaYIj2qhRgbHlKtolMpOGkaI0zubAvCSQStxbImIukVR3IVhcyOncsznk7krbWB5LI6nGy\nsZJfbkCszn6j3LW+m7u6fGiQ5utrqtMlRvIpDzYxFtVsc0biFndIOjbLTOslSpeIlUSNNtU19KDZ\nLcyIVQuoV5ntTxwUQ6m0w46x7l6So1eF7dUgKkkXiZ59Fz+T6a5+3Ew+IfVqBvzpI+Lr33D8D6MA\n2zR9V1+ccEw9apWaKZiHAz0lfqFOQGzqGmT4LHx+17qqhj7EFuWys03EEgfN2QAkGRbKNDou7BTZ\n8Npteb6IDykzNr2AlXiOp3n1dAeSqqw26i0X8+SzY3Ec1mSdXAls8jrKRVd6QtkHMcobHSw9y34G\nowCDSNTNd7wYOWZIA5whx2GBc6rSHoqTcrm+kqDMQXbABco6eP7JwXDa2IeabGgEAkmpDQ0cjO65\nFKlFepECAJ/dkZhae5dijUdJy5iXk5w506uEZlkpeq+sHgFnpWgiBIyiWj2lRbJhdDDseHekDjAl\nuT2XKR+pPysqBoLC4gS4TLbmwM6BasdT9aWnKyA8R6oHNp5mUp2Im7oED1QGwXTqCO6VZKep6OpY\nN9alXrFjWUqRaDlj1nEeq0jXWFzajWuacurYMHctHrADdPp1wxlSn6MFzwRnD3SJMtOWYJgHzS6d\nCcrMzQZAYf2Q55vnesTferOpZP3XOdLqjPnOdIDbCejb2AXSqY01C52JDajyW5spyEAerksLaBMx\nvBzT/WZl/wCrmmw1qTgaIL4kzM7rDji4uEmk+GBoNP5paP3ifnFa56nVxq85P+rrY4jM2mX06RJc\nKYeXNacuV0ncrCwj1ffMm5mL6hdjhfFqdKk5lbC0sT6sUs393khxcdPnIWegxtZ16eCcfSVg53+z\nyxDcO1o/aOvgpJeb9emc859q4SytiXjCUn02DEEMPpPmerLhmIHqjXTkldouE/qcB2IpVKmZzXUq\nU+pl0vvK5tb0lMw6WHURLd7ETcaLJVrSS7e5J+cZPetzm75b9pOs58bKp9RIc/ZVUclPK9Dj0lR6\nz1HK3EpTwozFVOiS9XUOqXmQ5U8pTzf4hW96BxRSXnpHNAfqRuNo5pLrKoU4mfcgTHtUyjXZZpSC\nEJEJrmwluGiIU9qAjbomPCFxVUg9UuodSmuSaghXR9W0n2aFppVBzXLpVLLTh6lxdRwjr03z+Kex\n50gQLrmtdJB30+xPZUvBt1R1jp0qgOtlppPEa3XNbUBEeS0YYiyxWo6THea0UXHdYqL1oYdLLCxu\na6SBp47Jrmzppyjfms1N3insdKLiOok3IBSKjSIkSTotWaL+GntQPbbrNu5GmOq2PBIyW6rZVA6+\nKyuF9IhAh3TVZ6zJ5z8WT8Q4ja2/NZC8+3xRNA9h0ss1VpCfVudD1lZqxjRbiMztDIhKEJ1U+33p\nNQjcqlZ8Q6A5xk5QTC/N+1GLdUcJJzGY6DkvcdqsQW0Ja4NnnqQvzHG1QSTM6i64/N1+muY73Yem\ncz3R8wXK9eCYlcDsDRb6B75nM4bctQvQmjrddPj+mL9+iiSBHraweUJbzaJ+1Ma+AQTAdYjmmZBp\nFtgVtGR7bCDB6oTTvYmN/JaXUvj7FQpkkjciBFt1LFJw82IcA4gt9b61TAHTmaDsHToeic9kHL5j\n8Uv0c30uo1OsUajiwwwZmH1njl/i5lZano2H+7BOZvr+luCTu3kFqqjK4ta4lsattPTqsFQ6n1r6\njbks+K3puwHBnVGF5xGGosYDHpX3vfK1ouSuXVdD/nEwYzatIGpE3AVOMQLX5/YU2tSgF4Z6Rgpk\nkzES4Akc7qSWX7b8tjLULCCSAHAnQkEj9m26Q6qIiSDEE7E8k7E4osYWS1wqgT6slo2aHbLIMY/0\nXoi2mBmzSG+sfFaZvtqwjazmYkUafpmBgNcNa45GtcC2IsHylcU4bUwzKdR5pllZmdr2OzNP+G+j\nvsTeHcS/VWF9OtUY+q4MfSpmA5gd6wfY2WYvq4irWqspF4IqVi0kZQAMrnNzQNAPJcc6nTvfG8/2\nhwXomVJxNN1SmaZORjw18keo8kdVXCcGzE+lb6T0dRjHPpgtL2lrbuBcNHWXK9M0G8kWMCRMtuiG\nNezMKTnU2v2Bg5IgtJ/dXW89X6rjOpL7M4jxCriKnpKr31HhoYHOn5jQA3XT8Via8AknTcIHzMyU\nD3wt885HO9b9rM6dY6JM6xstFXFg0m0vRsGV5f6QD+8IOgJWOo5JUqn3uNUp5I1nx0RGI6pVRaZo\nSlPRu70p7rSihdM6ygqH2oyEOX42QKLPtS3ck8goCzzQKa0jVDCY4wkucs0oXG/clONoTCSB3pSI\nWgeUxyzvN+iIF5KRUKc53lz5pFR3hf2KkfTdKoIHTVaKbrhcqnV0OxWunVv8QtOcjqsq8imseRqZ\nlc9lVNbWM2i9lHSOnTrRy+xa6FWwM2XJpx1lPpviLws1qOzSrkn61sbVO5XCp4rKrqcQy6LGLHpK\neIHNMdii2xjwK8sziB11sm4PGZnXGvMq4uvU0sW06+WyJtQm4INog/Gq4dGsT0Gi0sqgaOus9RrX\nUa05ZJvySagPwVkdjIGpASK+KJBFhdZkS1oq1gOp0gLJvob3Wc12MPU7zZaG1RGxVRnqucgqu2hP\nriSCICU+8rUoyuE+Cy1hZ1pgE381qcRyHKFy+NcQbSpvJIBLYDbfEK300/POPcZe8va8mA4gDYdA\nvJYmoS6GyS4iANdbBdHizXOc9+gJJhaexGD9JimOdENk33IvHevLL5dNdeo/Qez+G9FhqLC3Kcsu\n8b+a2RrdJNQeJvf2aIm1CNrL2SenLRPpSNghLT0smB5KWdeXRTQYcDqgfGwuLyPJCe+Ty+1GwGE0\nVWdYCL7lKsIB2TnG07rO5wBkp9qTWEnRZ679uqa6oFmq1GA7nu/FEIqDW+/j11Saj/W5iIyknKQd\nZG6Ks761jeRMnZLNINgpvHrVsgvAiXSNB3LJVdLB6xLgYPq2A2PUqqriZ6keASXG31bDuRot7okE\nxoRG9wTdDjcQwvPos9OkRDWucSbgZm22mfNLqO2G+2yUZ35z4rUiahJ/dMCL6gDTwCFrS4gNvYzb\nYFF+sPax7A4hr4zDnBmehSS8j1gYMRa0+Sns2Aqnvn6kqoVHG3xogqOstJgC+58PYhUcVHnRNTAO\nshcZ/BUXEoXk/Wka0t7Z0PmhnaPJG8jcIHdyIoAKnO0AV6JRsfBESo6DslZjJKJzvd9aW5ZtA1Cl\nlyOqdkgn8FEVUd+SVsfjyROdulOcqKft7Ut5GnX2IX1EGbc8lTFF1h3pFZ47rRdG51vrWeu6RooP\noajXad5Gy1U64mN/YvH0scToLrdQrPmZ6Qt6zj19Gv3HuWlj1wsJV0W6jXBKVOXXD48URJ2Kw0ao\nmNlspkac1itylurOCsPcd040Z6o6VGPsWdUVIEBaaDY9afBU1rSRpbWVobh5i8CduUaKtSCD3EWM\nJrXG9yOt1ZaAbZuiMuGkmVnpaFz7TOugWIuPziSdikcVxLKYGYxyXmeJdo2wW0tZ+Cskj09TFspm\n7p9vvS3doqTbFwkneNIX51iuIPeTLzM3HVINSSJmOeq1hj9NbxxjzDXtJGg5rdgcb6TX2EL8ww+a\n2XbQru8DxTqZgyRumJHtcXWDWOcS0Q1xkCNB1X4zxfidRz3FznO9Z0E6ROy7XHuPVK1R7DLabXOa\nI3XDdh2Vh6rXnnB0815/k6125kcXE4/qNfgL1vye4Jr82Iky2WluwJvK8xh+EB1YMqF7KZd86Nph\nfrHB+F08LSbSp/NsZjUkdFr4udY7q8mhjwVPmNbJ1WxS517l6Ncl4c2Vki/xCW/TdZcRWLT3jRRW\nwER71VTEMb0EbrE2qS3Q96y40EgCTHVFaqvEGDT3rFV4iSbLEaQEb+C0UsM06hMFtqHXVCRJmE4s\nGmw0QlsCVqMkuCRWpgplSoEsvF1RirNj7FkrVJ2jqttdqw1aZPco1GdsRb8UDimuYeXigIWolIIQ\ngBNefsSaiUwt5hILvyTnjRJKzFC4wl1jumEoKhvdVATZA42vr9SupzQEhVA1HBAXclTz8dEsu89k\nFud1lA/6lRdCUXLOiybJedBUfslOeoT2Iv5pbnadxQF6VVPirDBmpPck1HaKidksm/JUxbnJT5nV\nFUcNUkn2+5NFVHeSS/nmVPfCW9/VQx+w4djY1v0Wuk+PxWN78tykVsf0BK0z6d2hiIOoXRw9bQyv\nEtx7wtWH4k8C5Pcl6PT3OGrj97fbXwXSo4sDffdfn9DiDjED2wurg8aS4SFNHr3cTAjc6StNCu9+\n9lwsG9hGi6YrtYBchZ8VvTr0mDmmPqlgytPSfauVS4g217afimUsay/rDXzQnbSceR6riUjGccFJ\nrgLktS61RpEAgH2lYMZgg67ip461OnmOPcRqVXxLv+XZcgB0n1j4r01XgpLhqRqY5JGK4Q+ZDTB5\nqY1OnCc5NwoJ5rps4QReCtVLAOkWt3b9VU1lwrnW5aLuUcS0AA6peH4cSBIJttz28UY4e8GS0jkU\no8d2spmnWcQSWkErif0nUZEAwb+xe84xw5r2OLyRkY6bexfneLGcwCeQ2kdV5u+crpKc3j1R7xmu\nBGvKV+v8CxdOphqbmGWkAeIEFfjHDOC1azjkDnQQHBouATqF+08G4ezDYalQYcwYBJPzsxEmVv4v\n8Z6NfU6JDnp9YdfBc7EErs5ml5vCQReT3JRrPGk96pz385HJUaSZEAxCU9vO5SQ683RGrAmfBFXk\nbr4InN6pLqnqxzuqNTT3oVocbpL3aoHP6oc+2quIVUCyvZmkbdFrf1SXWPxooMtSnobwUNRo/O6O\npF+miBz+9FjO9p5WSKjU+q74Cy1jC0FPSKgt8c06o5JJVCnj3/UszjC0PWWtpZZiqNTpBSnut4qz\nKU5qWso6JSn396j3IVNAFLc9E4FKV2hVQlAXQmP1S3i/gsxIS93vS3lMekPbJnZaaxHmySDrKY5p\njn0QPHgqFjdA5MCWUQqoY2SJTamspL3lRAOWesPanvduk1J9bflOyRX7C9oJ1Kp2EYdwJWJ+IMdU\nsF1jEq1z9N1bDs2IMboKbG6QT1SSXnaB8ckym0jn9Szi61sIAgfHituGfl1WSky0mPrWbE4o6clb\n6V2KnFclmn2qN40Tq620ry4eXaCSmDD1TtCm1qe3phxPeVY4g4mWuj6/Ncihg6nL48Vpp4Codvbu\ns+y5HbwvF3AwIJjddXC4yR6+a/kvNUcG9h0M7rcapywZlbhuvV0scGiwvC1UHB4BcF4J9etaCY0l\ndbh2LrgDNpCmD2lEUtI9i2MoUtgPBedwGJcYJA81024jS47lmxXXw9CmCDAA3TMTgaVTSD3rk/rB\nGpkdCjHEWt3AhMVze3tXDYXBVBU/2lZuRmQiQZi87L8OxNYAktudvcv3fjWCwGPaP1yl6SxDXSQQ\ndQR1X4lxzhVKnXqNomoKQeWsbUMvDQdzvuuPetw3s3xipha3pW2P7Q2cOvNfrHZPjbMYyq40y19L\nLmj5pnvvK/H6lJgBFpMAX5XX6l8lOCb+ouOcGoXy8DYAeqDKzxWrHosTSBMjQ8/jRYK1KbQuzUw9\ntDqOXsWTGU40IXZycKowxHvSSwgGAulXjcLBjGB1pLea3ozPqaXlA5yJ1MAQFmFN53ViQ0HmilCG\nGLnwKIUfG8KWpUzHnoqnRNbhndw70L6UQJGi1qFPSnFMqpFQ+PP4Kis7zqkVahTH3NtFmrmbIsKq\nuPckVHSjrk7rM47puNI8pTn3Ctzo96RVfJU8kVUfv1jwSPSK3lLOi1qF138kg5vD2+KZUdHekmr1\n71ki9LlC4/G6VUqckl7xurqmVHxuge6Uqo4aIc6aGOJS3E66qpHPvS3PhSM4t5CCRoUmrWSjVVaa\nHnXuSXlUahMJT6isi6soXvGiU55idEp7lUE93dZIqv5R1VvJWZxnXT45Ii31PYkvfKqoRtslOcFF\nfq1Oo0SSfanUsUzmF5unWE9/NdDDVhAnKFrzcPCR26WLYeSp3EG6Na5x25SuX6Vk8+7609uOA9UN\nF91J1pI6mHLnXn1h7E12Gb+0665dDE5raRumVg6JLxCarqYbB0xBHiui0Mju1K81h8aAYkuXRpVp\nFrdTurKjq18bTpg8tZ1/JZH8aaPmG/gudVoPqTdsRsVzcVg6rf2fV5j4us9X+NzmPQ0OM1HHUW16\njonVOOGwDR3ryDKz2bSOadTqO1IsseVXMe2wPEWvEECd/wAE48VYDlAsB7V5TCvj5wIPVdTDYmmx\nvpCP+ULULXpcPjC9u4OpSW8VsQDJFgvM4zjkyGWvdZ28UiPi6WmvV1uL1dBa3kkNx9YEF89y5LOI\n1HQWAIcZUxFQh5IETACzqya9SHV8TAmAId864HgvL9oOGsmpVZXa91N3rsNtYi+6v0tZjBNQAusR\n03WGrxCmwPE3gAHLNwZ8dVxvW+nSRw61MzmiYOk6L2PyZcafhqzgSPR1WHMDMAg2K8XX40KjyxzZ\ncNCBAudOq/RODUMP+rtgsDwyTNrmySe1tx6B3aB7qm7hsAfVi90dXiJf850ToBqvN/qhu1j2l3RC\n3h1YAOdUPcFrXPXXHEpdlM96aagIHVcOnRdHrG+ZbqRDRC3IH1agBWPE4kyQPNBicWxupnmudXxw\nJtYDRWqaeIVGOuQRP1rfT4hHzrXmy8ljcYS+Y7uSx4jiLySZPjos7SY91W4k0AnM2B1XPp8blxGW\nbrydKo9+uaI5rs4Wg0gRc8uSsS49AcWCAUl+IJ5LM14yxyWfF4sCABtCUns2tiWibwsVXEysD3OJ\n13uEiu6pKkp9Oi+oDzSS6VlDzEHdCKnIpapr3JL952UNQG0yfwS3LUiBzSluMH2IH1iNRPXkkOrH\nuGqIa+8Trus9YCbJb6qF1VrkVb4SnDqpUcluPNDVOeOuiW98hLrk67IRboqYt1UBJ9J5KnpDyQq0\nKd5UzhZDVOhS3udoNVKmtdSqLjNdJdiG85WSox6zvadymrJrpOqgwhdfRZKdhPJMOLLbQO9XV8TX\nU3cj5JdSi7WCp/ST9pSn8QdzM7ptTxCaXSEt9M6FPbjXDZvegfiyT6zbKa1zHpqVED9qStuGpwAT\nGi/P8L8pPDXfOZiGcjlBnvutzPlI4XGuI2/8v8UzXDHuWDW8WhaWOaAIbnMarw9Dt7wtxPr1W6Xc\ny2nQpuN7d4BlImi81Khs0EZW3GqXIc869o1xJ1yjcWspVDZ+efBfnPY7tNh2Gu/GYkhzneo0Bzmw\nT0C9XT7YcKGmLphwscwdrz0Uns9z9PR4drQJvO2ZU99Q6u7vwXFo9suGkwMfSJjkfsW/D8YwNWIx\nlFx2l4aPbotWek92/TeypUGjiegRZ6p1JA5FZa3FcHTicZhgCY/2zTfuWyi+lUgtrUng6FtRuUjz\n1TP9XYJtIshxgjUnVOPFXAZWhnflSyWRDa1MRr/eMMebrrj9qsacBhX4mmadaoMoYwva7MSRJIae\nqhz7ejp497gMzGgAwocXTzAvDTFoCxNxAfhm1akA+hFWowFohxEwL6ryeE7aYdutCqCXaGDN48U1\ncr3YxlHem0J9Grh3bALFQwwqMZUDHEVG5riIkSJ5plagWhmUBs6rUY/8bMbxClQa2W5pMNhXQx9G\npAIPncLhcY4Y6vSs+HtktDzvtcafgvLMFTC4hgxtU0mBv+0Ev9JyIIs1c/ltl+vTrxZn+v0muaDo\nBcQb2m4BXkO2FXKCaNstpOpXl+M9qjTql1CsaoiBIseS8/xDtbXrEB5AGtvtXCx2kes7HPL8R/eG\nd4i0c5XtncaoU3kawL30uvzHhfFWNaajXBrosZg22HVdHFU6tXCDG1c7KecNinq6T+0SFedh1P6/\nXOFcQo1WtcCLibouI8Wy2YQSd9l4bslxMYhj/R0yynSIptn1s1pOmi7n6y9wyCmYzROX5vcusrh9\nenQw1fEEkuIid10v1oRLjfKb7HuC8zUp1s4DS8A7qVX1BkDgT36DuUnpHbq4ymCZN4GnclVqtOJm\nZHKfyXLdUklpY7vi2nvS2PGax0tBELerrVUqUDFr7g/YlVY1DWQlvretdovvCyYwveBDh5rMXR08\ndlI0IldGljCI07xvdceiwMjMGX67rfnZESPBJcMHjOJANN4KxUar6nrAyhfhWGZBKugMgswgJR0K\nR3IEqqlQHYLKJy5vYkOed1nFBjHvOllle95FzKLFVXfs5Y3vdL9O3SIG5VFNrPZePyRHFk9xQfrD\nNjOyz4qvlHqid0XW3PI96zVxrdc6pi6k6EA7bJf6zUOrZC1amNjnNA1SfS2MWukmoQfWaEFQyZHk\nNJUVRrOzEXSnYx4dG19U62u6zYmlPx0VxBjGaEj8kwV2lc51C9yUbWAaErRrVUd1Sw/mlF3wUp9T\nZMDatUNSH1t90D6kyEh5AuUwNfXnVA+Dus9QbrNVL4VWc62WG6X6RpWL9Ye2xCH0w6A6lFmtbhyK\nU9u+aEDX5roKoO35I1pjK0amU2nXadVzyHjqlOquGyYPywflfRS/M+aHMo0rmz6MNV3M+ZRCqevm\nUkqwixoZi3iwe4DlmPkp+sO/eJJ3JJlZ4VtClKc2u7SYtCJtdw/aPeCUgjuUgKGGuqzqZ77p1HiF\nRggVKgGwa9wHsKyhSFRtbxF+73nnL3377ptDjNZhJbVeJEXcXC3R265oVQh09Ng+1+KY3KHAt3Dn\nPM9NdE6l2yeILqDXls5XSRE72XkwFZKzkZnVn09iz5Q+Kt+bjsQwQGtZLXBrRoBIsulg/lX43TFs\nWH8hUpsOncF+dCeimYrftH6i75auNZSM2FBIgO9FJHW9iuZxb5VeLYmm+nUfRh9PIS2kJ/5gSLOX\ngxNvrCKPgKE+9dap2jxbgGl5I2gAeCbV49mblyFpA1nUrhEXnl7lSOj3XYftRg8M4vxeGfWfY0ya\nuWkzb1mn5116UfKPSrTh31nswoe6pBFi4izWhv7K/IQia889t7+HcnUi/f2/buxfyrcPwFF9GpTx\nFUvrF+emGhuUnqJXdq/Lpw7MQ3CYstB2cwT7F85Enr5+5FmPNJ6c7Jr6IZ8ufDyb4bGdIcw/UmM+\nWrhryM9HGNA3LWmPIL50k81XpHaSfjqnoyPpF/yw8JIcc+IMCQz0REnvCw/10cMifQ4qeWVp8R0X\nz96R3ND6V32Jp4x9ED5ZuFer/d4rr6gtZQfLDwiwyYoTqTRB9y+eS92ioPg9eYRJzP8AX0K/5V+C\nuqMzDEmZ9c0sobGkjkumz5ReDPeKYxbQSJDiwtZ3ElfNDX79/t6KjVJ85+Oia1OI+oW9tOFEw3H0\nBGvr69eiOp214bEjH4Yc/XXy4ag5ewKi4HYeX2q7UvMj6gPbDh5H/fsOZv8A7QSl1O1fDx87G4c8\nvXt7F8yZhMwFWYKGPpX/ANrOHuOUYygTE6i3ikDtNw92mMw5B2LwL6X8l835hy9uiheOXsT2ZX0Z\nU49gAf8AvVBt9qgI6X2TKnF8LocTRmJj0rfC6+bweg6z7IV5vjXpa6pj6EPGMJJJxVG3/wA1qB3H\nsJtiqHT+9bovn22qou6dPD7U+1kr9+rcfwoucVh/F/2JFXtNgw2RiqEdHjXu5L8HzfBUHhcIuP27\n/wBr8FnLP1ijMTmkx5qP7UYMf71R6+svxE/HNTMOS15RMftY7T4Mm2Kof5lmr9q8EyxxLCel/cvx\n23L2BQu6DwU8kx+vDtZgjpiWaX19iQ3tdgnEg1wOsGF+UFyolPJbzj9dZ2gwp0xFKOroPtVDj+GJ\ny+mpTt64+tfkR+LR7lPKVfJMfsTcewj1XMN4kOB8kD8YO/u/NfkVOoRYEgdCUfpXbPf/AJinksj9\nTr4hp1t8d6zvqNjv3+CvzP0jt3P8yqdWd+87p6xTyi7X6cMUG8lBjuUGy/MRVf8Avu/zFF+s1B/5\njv8AMU8k1+j1MeRsEv8ApDp7Cvzs4l/77/8AMVbcXVH/AJjv8yu/w2kR0VJipc5TFABWoVbQiorV\nFSVBFUqxdSyCvNQeKuFC1XRYcrzIS3REWaIIorjZVTaSYAkxdSCiVSlQHcR+aIBXQIHVFKp0aKNa\noLNlWZWWoQ1BQRtIQyraqCcoVaoqLilJPJQORIgSVMxVOCtoQQOKkgqnFVTRVlSVVRQFEHBVOlVd\nU8qxKjlZ0QTzTGocgCJypRFW0jMDEgESOfRU+CSQIBNhyChCiaqFRWokRSiitKBJUVFXy70FqFUd\nVaaBIVAoxuhhBahKofWpCCAqw5VlVQqCN0LkaEhQCrVwrcgBRFCmUoGZVWVENFAFnQIaihW0SrhF\nAWqsiMqJpgWhVCMhQN7kAwoNR3ooVhv2prUgOaJqtzbK2676clC+1dyFwOs6Jnn5KvjRNYLyTeZR\nZSiI+IUCaBLQryqyPiFUXRftcIcqIHv8lRCF5wuFaYAqcNLexCQF1cFGR3+SkIUvIpCMKwmr4lqJ\nh+LKR8QmplLIUATMqgHxCGUoqwETh3qAdPYqsgYUREKDxRMA4KwEeX4hWWxspq/9Kyq4R5fj4Cpg\nM/bKqUEKJhG6gHRDAQhIRxqry9/kiEnZWUzJ0Pkpk6HyV0Kgqwm5e/yVZOhTQLhCpMy96rJ0KmgF\nE4N6exXl6exNXCGhQpjx3+SHJ8QrqBKFNy9FeUdfJNNJARBMySiDensU0JVOCa9vRU1vTZXVwuFa\nYW9EOU/BT7Cc5UznmhUWsc9FnPNT0h5oVExdH6Q8/cpnPxCBRMNH6Q8/cqznmhUTE0Wc81C880Ki\nYavMVec8yhUVNFnPMqsx5qlEF5jzUzHmqUQXmPNTMeapRAWc81WY81SiLq8x5q8x5oVENFnPMqsx\nVKIi8x5qZjzVKIurzHmpmPNUoiavMeamY81SiGizHmqzHmqUQ1eY81JVKIavMeavMeZQqICznmfN\nTOeZ81SpFEXHmVA48yhURNFnPM+amY8yhUQ0WY8ypmPMoVENFmPMqZjzKFRDRZjzKmY8yhUQ1eY8\n1Mx5qlENXmPNSTzVKILk81Mx5lUogvMeamY81SiAg88yoXnmUKiGrzHmpKpRDUUUUQRRRRBFFFEE\nUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRR\nRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEEUUUQRRRRBFFFEH//2Q==\n",
"text/html": [
"\n",
" ![\"floating](\"images/sparse.png\")
\n",
"\n",
"Here is an example of the matrix from a finite element problem, which shows up in engineering (for instance, when modeling the air-flow around a plane). In this example, the non-zero elements are black and the zero elements are white:\n",
"![\"floating](\"images/Finite_element_sparse_matrix.png\")
\n",
"[Source](https://commons.wikimedia.org/w/index.php?curid=2245335)\n",
"\n",
"There are also special types of structured matrix, such as diagonal, tri-diagonal, hessenberg, and triangular, which each display particular patterns of sparsity, which can be leveraged to reduce memory and computation.\n",
"\n",
"The opposite of a sparse matrix is a **dense** matrix, along with dense storage, which simply refers to a matrix containing mostly non-zeros, in which every element is stored explicitly. Since sparse matrices are helpful and common, numerical linear algebra focuses on maintaining sparsity through as many operations in a computation as possible."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Speed"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Speed differences come from a number of areas, particularly:\n",
"- Computational complexity\n",
"- Vectorization\n",
"- Scaling to multiple cores and nodes\n",
"- Locality"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Computational complexity"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you are unfamiliar with computational complexity and $\\mathcal{O}$ notation, you can read about it [on Interview Cake](https://www.interviewcake.com/article/java/big-o-notation-time-and-space-complexity) and [practice on Codecademy](https://www.codecademy.com/courses/big-o/0/3). Algorithms are generally expressed in terms of computation complexity with respect to the number of rows and number of columns in the matrix. E.g. you may find an algorithm described as $\\mathcal{O(n^2m)}$."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Vectorization"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Modern CPUs and GPUs can apply an operation to multiple elements at once on a single core. For instance, take the exponent of 4 floats in a vector in a single step. This is called SIMD. You will not be explicitly writing SIMD code (which tends to require assembly language or special C \"intrinsics\"), but instead will use vectorized operations in libraries like numpy, which in turn rely on specially tuned vectorized low level linear algebra APIs (in particular, BLAS, and LAPACK)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Matrix Computation Packages: BLAS and LAPACK "
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[BLAS (Basic Linear Algebra Subprograms)](http://www.netlib.org/blas/): specification for low-level matrix and vector arithmetic operations. These are the standard building blocks for performing basic vector and matrix operations. BLAS originated as a Fortran library in 1979. Examples of BLAS libraries include: AMD Core Math Library (ACML), ATLAS, Intel Math Kernel Library (MKL), and OpenBLAS.\n",
"\n",
"[LAPACK](http://www.netlib.org/lapack/) is written in Fortran, provides routines for solving systems of linear equations, eigenvalue problems, and singular value problems. Matrix factorizations (LU, Cholesky, QR, SVD, Schur). Dense and banded matrices are handled, but not general sparse matrices. Real and complex, single and double precision.\n",
"\n",
"1970s and 1980s: EISPACK (eigenvalue routines) and LINPACK (linear equations and linear least-squares routines) libraries\n",
"\n",
"**LAPACK original goal**: make LINAPCK and EISPACK run efficiently on shared-memory vector and parallel processors and exploit cache on modern cache-based architectures (initially released in 1992). EISPACK and LINPACK ignore multi-layered memory hierarchies and spend too much time moving data around.\n",
"\n",
"LAPACK uses highly optimized block operations implementations (which much be implemented on each machine) LAPACK written so as much of the computation as possible is performed by BLAS."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Locality"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Using slower ways to access data (e.g. over the internet) can be up to a billion times slower than faster ways (e.g. from a register). But there's much less fast storage than slow storage. So once we have data in fast storage, we want to do any computation required at that time, rather than having to load it multiple times each time we need it. In addition, for most types of storage its much faster to access data items that are stored next to each other, so we should try to always use any data stored nearby that we know we'll need soon. These two issues are known as locality."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Speed of different types of memory"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Here are some *numbers everyone should know* (from the legendary [Jeff Dean](http://static.googleusercontent.com/media/research.google.com/en/us/people/jeff/stanford-295-talk.pdf)):\n",
"- L1 cache reference 0.5 ns\n",
"- L2 cache reference 7 ns\n",
"- Main memory reference/RAM 100 ns\n",
"- Send 2K bytes over 1 Gbps network 20,000 ns\n",
"- Read 1 MB sequentially from memory 250,000 ns\n",
"- Round trip within same datacenter 500,000 ns\n",
"- Disk seek 10,000,000 ns\n",
"- Read 1 MB sequentially from network 10,000,000 ns\n",
"- Read 1 MB sequentially from disk 30,000,000 ns\n",
"- Send packet CA->Netherlands->CA 150,000,000 ns\n",
"\n",
"And here is an updated, interactive [version](https://people.eecs.berkeley.edu/~rcs/research/interactive_latency.html), which includes a timeline of how these numbers have changed.\n",
"\n",
"**Key take-away**: Each successive memory type is (at least) an order of magnitude worse than the one before it. Disk seeks are **very slow**."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This video has a great example of showing several ways you could compute the blur of a photo, with various trade-offs. Don't worry about the C code that appears, just focus on the red and green moving pictures of matrix computation.\n",
"\n",
"Although the video is about a new language called Halide, it is a good illustration the issues it raises are universal. Watch minutes 1-13:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEABALDA4MChAODQ4SERATGCgaGBYWGDEjJR0oOjM9PDkz\nODdASFxOQERXRTc4UG1RV19iZ2hnPk1xeXBkeFxlZ2MBERISGBUYLxoaL2NCOEJjY2NjY2NjY2Nj\nY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY2NjY//AABEIAWgB4AMBIgACEQED\nEQH/xAAbAAEAAwEBAQEAAAAAAAAAAAAAAwQFAgEGB//EADsQAAEDAgMECAYCAgICAwEBAAEAAgME\nERIhMQUiMlETFBVBQ1JhkSMzQmJxoSSBU7EGNMHRFpLwVLL/xAAaAQEAAgMBAAAAAAAAAAAAAAAA\nAQQCAwUG/8QAMBEBAAIBAQYFBAEDBQAAAAAAAAECAxEEEhMUUVIhMTKRoQUVIjNBcYHwIyRCYfH/\n2gAMAwEAAhEDEQA/APz9ERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQ\nEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBER\nAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQER\nEBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAR\nEQEREBERAREQEREH1lJs6hfSwufTxEmIEkvOql7M2fZ/8aHIm2+VPQYup09sfyG8lPv2fx6u8q5N\nsltZ8fl52+W+9P5T7qPZmz7N/jQ5kX3yvezNn4v+tDw+c81e37M49W8uSb+P6+H7eajiW6/LDjX7\np91DszZ9nfxocibb5TszZ9m/xocyL75V7ftJx6u5Jv2Zx6t5ck4luvyca/dPuzKvZ1Cymmc2niBE\nRIIedVL2Zs+zP40OZF98qxXYuqT8fyXclZwu6Nh39W8lM5LaebKc192Pyn3/AKM7szZ+L/rQ8PnP\nNOzKCz/40ORNt8q/v4/r4ft5rzfwyceruSjiW6/LHjX7p91HszZ9m/xocyL75XvZmz8X/Wh4fOea\nu79mcerfLyXu/j+vh+3mnEt1+TjX7p91DszZ9n/xocibb5Xy1NSGpL7Pa3DzX2+/hk49XeVfJ7KO\nEVL87NAJtyVjDe27adejs/SP9W8xedY/9cdluz+NHldedluy+NHnZWu04N7ekzJ+lO04LN3pMiPp\nC2b2Xo7/AA9m6/Kr2W69umj0unZbs/jR5XVvtOC/FJpbhHNO04N7ekzJ+kKd7L0OHs3X5VOy3ZfG\njzsoKqlNNhu9rsXLuWj2nBZu9JkR3KntGqjqejwFxwg3uFlS2Sbfk1ZqYIpM0nxb9Js6hfSwufTx\nEmIEkvOql7MoLO/jQ5E23yp6DF1Ontj+Q3kp9+z+PV3lVG2S2s+Py8ffLfen8p91HszZ9m/xocyL\n75XvZmz8X/Wh4fOeau79mcereXJe7+P6+H7eax4luvyw41+6fdQ7M2fZ38aHIm2+U7M2fZv8aHMi\n++VoBry1/Fq7kvHB7cAOPVvJTxLdfk41+6fdl1mzqFlLM5lPEHNiJBDzqsHoIst0d3evqq/F1Oo4\n/kO5L5nuH9K7s1pmJ1dXYLTas6zqj6CK/CNOadDFnujv71J36dyXydl3lWnQR9BFluju706CK/CN\nOakvw5clNLA+JrHPtZ7bhRrEeDKKzMTMfwq9BFY7o7+9Q0tKanFZ7W4eatnQ/kpsa95rX0GixyWm\ntdYbMFIvkiso+y3Z/GjyunZbsvjR52WtvWfxau5JvWZxat5KpxrulymLoyey3Xt00el172W7P40e\nV1rb2L6uH05pvWdxau5Jx7nKYujI7Ldl8aPOygqqU02G72uxcu5bu9ZnFq3ks3bN7w3vodVsx5bW\ntpLTn2fHTHNohs0mzqF9LC59PEXGIEkvOql7M2fZ/wDGhyJtvlT0GLqdPbH8hvJT79n8eruSp2vb\nWfF4u+W+9P5T7qPZlBZv8aHMi++V72Zs/F/1oeHznmr2/ZnHq3y8k38f18P281HEt1+WHGv3T7qH\nZmz7O/jQ5E23ynZmz7N/jQ5kX3yr2/hk49Xck37M49W8uScS3X5ONfun3ZlXs6hZTTOZTxAiIkEP\nOql7M2fZn8aHMi++VYrsXVJ74vku5Kbfwx8ereXJTxLaefyznLfdj8p9/wCil2Zs/F/1oeG/Gea8\n7M2fZ/8AGhyJtvlX9/H9fD9vNeb9pOPV3JRxLdfljxr90+6j2Zs+zf40OZF98r3sygxf9aHhvxnm\nru/ZnHq3lyXu/j+vh+3mnEt1+Uca/dPuodmUGF/8aHIm2+V8tTUpqS+z2tw819vv4ZOPV3l5L5TY\n98U9r92n5VjDe27adXZ+kf6uSa3nX/JRdluz+NHldOy3ZfGjzstbes/i1dyTeszi1HJTxrvScpi6\nMnst17dNHpdOy3Z/GjyutbexfVw+nNN7f4tXck41zlMXRk9luy+NHnZQVVKabDd7XYuS3d6zOLVv\nJZu2b3hvfQ6rZjy2tbSWnPs+OmObRD6OhDepwZD5LfDJU9m2fkNXeGVBQlvVIM2/Jb4hU+Jtn5t1\nd4hVG3nLxF/VJZtmZN1b4ZSzcWjeH/EeaYm2Zm3VviFMTcWreH/IeahgWbZ+Q1d4ZSzbMyGrfDKY\nm2fm3V3iFMTbMzbq3xCoPFBWgdUnyHyXeGVFJtEvpuiEbRiw5hpUtcW9Unzb8l3iFZlxZuY1H1rb\nWNY8Xofo2z4ssWtkjWYmNHeN1+I8PlPNMbrO3jqfpK4uMWo4fP6pcWdmNT9ay0el3K9Id43WbvHU\nfSV50jsXEeHynmubizcxqPrS4xajh8/qmhuV6Q6xus7fOp+kr5wOc24DiAdbHVfQ3FnZjU/WvnDq\nrOz/AMqG2xEbugiIrSgIiICIiD62jr6JlLC11RCCIgCC3QqbtGgs7+VBmTbdXxiKrOy1mddVCdgp\nM66y+z7RoLN/lQZEX3U7RoMX/agth8vqvjETla9Ufb6d0vs+0qEB38qHU23Vjn/kcht/FiyssRFl\nXZqR5+LOmxYq+fi2JtvvmifGaaIY2Ftx3Kv08dhvjuWei3VpWvpWceOmPwrC/wBNHfjGi96ePPfG\nd1nosmxf6aPLfHcu62saWQiNwfZtj6LNRTE6RMGqx1t/lCUtUabHZjXYuarosZiJjSWVbTWdYX+1\nHZ/BjzunajsvgsysqCLDhU6NvM5erQ7Ude/Qx6WUVRXvmjwhjWb17t1VRFMY6x46InPkmNJl30sn\nnd7rlznO4nE/krxFno1ay+2oQ3qcGQ+S3wyp7Ns/Iau8MqChLepwZj5LfEIU+Jtn5t1d4hXHt5y8\nxf1SWbZmQ1b4ZSzcejeH/GeaYm2Zm3VviHkmJuLVvD/kPNYsSzbPyGrvDKWbZmQ1b4ZTE2z826u8\nQpibZmbdW+IeSCGtA6pPkPku8MqazcLMhq3wzyUNa5vVJ8x8l3iFTYm2Zm3VviHksv4Zz6Y/zoWb\ni0bw/wCI80s2z8hq7wymJuLVvD/kPNAWkPzbq7xCoYFm2ZkNW+GUs3Ho3h/xnmmJtmZt1b4hTE3F\nq3h/yHmgWbhfkNXeGV8rsnim07u66+qLm4X5t1d4hXyuySA6a9u7vsrGL0W/s7n0T9s/51aFhZ+m\np+gpYWbpqPoKXFnZjU/WlxZumo+tYvWvbDF3cPkPNLCztNT9BXJe1rhdzRu97/VedJHZ2+zU+Ip0\nlGsOrCzNNR9BWdti14bW0P02V/pI7N32ajxFn7Wc13RYSDkdHXW3FE78K+0zHCl9PQY+p0/H8lvJ\nT79n8ervKq1CB1ODIfJb4ZU9hZ+Q1d4ZVO3nLwN/VLrfszj1b5eS938f18P281xYWZkNW+GeS9sM\neg4f8Z5qGD3fs/j1d5U37M49W+XkubCz8hq7wylhZmQ1b4ZQR12PqlRx/JdyWXvWZxat5LSrgOqT\n5D5LvDKy7CzchqPoW2nk9T9A9F/7Ot7F9XD6c03rP4tTyXlhi7uHyHmlhZ2mp+grJ6F7vWZxajkm\n9i+rh9Oa5sLN01H0FLC/dw+Q80HRxWdxau5L5o6r6Sws7TU/QV82dVZ2f+XP27/j/cREVpzhERAR\nEQEREBERAREQEREBERAREQEREBERAREQEREH01Ltykip4mOklBbEGmzRqpf/AJBRWd8SbMn6AvlE\nVadmpKlOw4pnV9Z/8gorN+JNkR9AXT/+QUHQvLJJelwENuzvXyKJyuNHIYWr/wDIq/Pejzvfd5r3\n/wCRV+W9Hlb6VkotvCx9G/lsPbDUk29Wyscx7mYXtwmze5fSiqYWM+I/Kx0C+GX1TOBv4W7DseLN\nrvR5NeTZMVtPDT+jQ60zFfpH8NtBzXraqMYrvfmT3BUEW/7Xg/7auQxJ+tzZb2lrZJ1ua98XdbRQ\nIrXKYOyPZv5fF2wmdWTBjt4Z3Oi+UgqpaYu6MgYtbhfSP4Hfgr5Q6qptOHHTSK101bsVYxTrjjRc\n7Uqc825+idqVOWbcvtVNFU4dOizx8vdKepq5arB0pBwaWCgRFlEREaQ12tNp1kREUsX2lFLEKSAG\nSO/Qt8Wym6aKz/iR6u8ZfKdGzyj2To2eUeypzssTOurmz9PiZ13n1nTRWZ8SPVvjJ00WL5kfD/m9\nV8n0bPKPZOjZ5R7JykdUfbo7n1fTRWf8SPV3jJ00VmfEj1b4y+U6NnlHsnRs8o9k5SOp9ujufS1s\nsRpJ7SR/JdpLdZvSMs3ebqPEWZ0bPKPZOjZ5R7LKNmiI01dPYf8AZxaI8dWl0jL8TeH/ACeqdIze\n3m6nxFm9Gzyj2To2eUeynl46uhz09rT6Rlm7zdR4i86Rl+JvD/k9Vm9Gzyj2To2eUeycvHU56e1p\nGRlnbzdT4i+eOqv9Gzyj2To2eUey248e4r58/F08NNFBFf6NnlHsnRs8o9lsV1BFf6NnlHsnRs8o\n9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6N\nnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8\no9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6NnlHsnRs8o9kFBFf6\nNnlHsnRs8o9kFBfTMniwN+KzTzLI6NnlHsnRs8o9luxZuHr4GjZ6eH/Kz/7J08P+Vn/2WN0bPKPZ\nOjZ5R7Lfzk9EaNnp4f8AKz/7J08P+Vn/ANljdGzyj2To2eUeyc5PQ0bD54sDvis08y+YOqv9Gzyj\n2To2eUey0Zc3E08E6KCK/wBGzyj2To2eUey0igiv9Gzyj2To2eUeyCgiv9Gzyj2To2eUeyDsC5A5\nrSn2FVwdK17d9j2saB9dzbL+1mA2NwtB22q5z3PMrbusfltsCDe4FsjfO6JQnZ9WKZ1SYSIW6uJH\np/7CjdSzNpm1JjIhc7CH9xPJSVO0amqwCdzXNYSWtwAAXAGg/AXo2lUiJkQLOiY7E1nRggHnmPVB\n7DsutnDDFA52NuJuYzF7f7Up2FtMMxmkfhte9wuIdrVkAYI5GjAMIJY0mwNwL27jpyUjduVwpXwd\nIDidixFou3IjLlqh4OBsWvMBl6HIPEeEkXJJI/2CFCaCqFYKQwnpzoznlf8A0p5Nt18j8bpWl1wR\n8NuRBJBGWRuT7qHtCfrHTjo2y+YMA7rIJextoa9WdbncW/36Lp+xaqGPpagNij6PpC4nTO1rc7ri\nHa9bBE2KOVuBotYsabjUXuM7HRO1aksLJcErHNLSHt1BN8yM8jmh4I6ihlhaJAWyRFgkDwdQTb/e\nSqq5LtGSWN0ZiiDDGI2tDeAA3y9b/wC1TQEREQIiICIiAiIg9AuQOa0p9hVcHSte3fY9rGgfXc2y\n/tZgNjcLQdtquc9zzK27rH5bbAg3uBbI3zuiUJ2fVimdUmEiFuriR6f+wo3UszaZtSYyIXOwh/cT\nyUlTtGpqsAnc1zWElrcAAFwBoPwF6NpVIiZECzomOxNZ0YIB55j1Qew7LrZwwxQOdjbibmMxe3+1\nKdhbTDMZpH4bXvcLiHa1ZAGCORowDCCWNJsDcC9u46clI3blcKV8HSA4nYsRaLtyIy5aoeDgbFrz\nAZehyDxHhJFySSP9ghQmgqhV9VMLumtfD6Wvf2U8m26+R+N0rS64I+G3IgkgjLI3J91G3aVQ2oNQ\n3oxKb3cIxpa1raWsgm7ErGtjfMGxMfiJc53CG5m/skuxqiKF0ryMAiErT5hu5eh3gue2asyB8pjl\nsXXD2DMOFiMu5cu2vWOYGGRpYGdHhwC1svT7Rn6IeDnsquEcshp3BkV8ZNsrXv8A6KhZSzvgM7Yy\nY23u7uFrX/2PdT1W1aysjDKiQPZiDrYQM8+X5PuvWbSfHS9AyJjWukbI+2jsOgsgrU9PLVTCGBhk\nkdo0alTybOlbFHIxzXhxc0gGxa5ou4H+lHHWzQ1jqqAiKUkkFo4b8uSmdtWZxZeOGzcZIwZOc4WL\niOf/AKQUUREQIiICIiD0C5A5rSn2FVwdK17d9j2saB9dzbL+1mA2NwtB22q5z3PMrbusfltsCDe4\nFsjfO6JQnZ9WKZ1SYSIW6uJHp/7CjdSzNpm1JjIhc7CH9xPJSVO0amqwCdzXNYSWtwAAXAGg/AXo\n2lUiJkQLOiY7E1nRggHnmPVB1T7JrKnD0cQs+MyNxOAuAvex6+1+ruADcWZGi5ZtSrZHFG17MEQc\nGtMbSN4WN8s8uaT7UrKiPo5JbtsBYNA0QcTUFVBUx00sLmzSWwt53Nv9qZ2xdot1pne4/PNcP2lU\nPqW1LujMzSHB+AAgg3v7r2Da9bBF0ccoDLk2LAb31Gmnogk7ErGtjfM1sTHYruceEN1v7KrPSuig\ninD2viluAR3EagqwNsVeMOkMcti7J7BYhwsRl3eiifXSOiZE1kbI2NLQ0Nvrqc75+qCRmx9oPcWt\npnFwtcAi+a7dsLabGlzqR4AF9R/7XjNt18by9kzQ82u7o23JHfpr3XUp/wCQ7RMLGdK27CTiwC5B\nsbaaXCHgiOxK/BG7oL9JewuLi3NQQ0FVPUvp44XOljNnN5Z2/wBqY7ZrjJjdI0uuTcxtOosRppbu\nXNPtSppZHSQFjHu4iGDPO+n5CCR+xqmEF05ZGwRtkve98Wg/K9n2JVQPkjcAZWPDQwfWCSAQfyCo\nhtSo6NzJBHI10Yj3mDQaG47wun7ZrnzumdK3G4tN8DcrG4tllnf3Q8EUuzquGm6xJC5sWW8SO8Aj\n/YUTqaZkAndGRE61ndxve3/+T7KWp2hUVcjX1DmvwZAYQAMgNB+ApTtR+CmjbFGI4MRDDmC52pQV\nqeknqsfQRl/RtxOt3DmpKiglhaJAWyRFgkD2nUE2/wB5LmkraiiLzTvwdIMLt0G45Z9ykl2lJLG6\nMxRBhjEbWhvAAb5et/8AaCmiIiBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQERE\nBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERARE\nQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBE\nRAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQE\nREBERAREQEREBERAREQV+tfZ+0619n7X1NFBAaSAmKMkwt8K6n6vBZ/wo9XeCqk7VETpo50/UIid\nN18d1r7P2vetfZ+19h1eCzPhR6t8FOrQYvlR8P8Ah9VHNx0R9xr2vj+tfZ+0619n7X2HVoLP+FHq\n7wU6vBZnwo9W+CnNx0PuNe18d1r7P2nWvs/a+qraeAUk5EUYIhcRaKywGxR4RuN05Lfiy8SNdFrZ\n88ZomYjRU619n7TrX2ftXOij8jfZDEzCdxvstqwp9a+z9p1r7P2tjoorN3Gajw170UWLgZp/j9VX\n5iOi/wAjPcxutfZ+0619n7Wz0UVnfDZqfDWTRUjakyYnFuG2gusq5YmJlqybNasxWJ11cda+z9p1\nr7P2rvZUWfxX5E/SnZUVh8V+ZH0qeNQ5TL0UutfZ+0619n7XddSNpcGF7nYgdRZVFsraLRrCves0\nndlY619n7TrX2ftV0UsVnrX2fteda+z9r6qip4TSQExRkmFvhXU/VoLP+FHq7wVUnaoidNHOn6hE\nTpuvjutfZ+0619n7X2PVoLM+FHq3wU6tBi+VHw/4fVRzcdEfca9r4/rX2ftOtfZ+19h1aCz/AIUe\nrvBTq8FmfCj1b4Kc3HQ+417Xx3Wvs/ada+z9r6qtp4RSTkRRgiFx+VZYDYmYRuN05Lfiy8SNdFrZ\n88ZomYjRU619n7TrX2ftXOij8jfZOij8jfZbVhT619n7TrX2ftXOij8jfZOij8jfZBT619n7TrX2\nftW3RR4TuN05KvQ0jarHicW4baC6i1orGss6Um87sOOtfZ+0619n7V3sqKzvivyJ+lOyorD4r8yP\npWvjUb+Uy9FLrX2ftOtfZ+1d7KivbpX6X4U7Kiz+K/K/0pxqHKZeil1r7P2nWvs/au9lRWHxX5kf\nSqtdSNpcGFznYgdRZTXLW06Qwvs+Skb0uetfZ+151r7P2vqqKCE0kBMUZJhac4rqfq8Fn/Cj1d4K\n0TtUROmjjz9QiJ03Xx/Wvs/a8619n7X2PV4LM+FHq3wU6tBi+VHw/wCH1Uc3HRH3Gva+P619n7Xn\nWvs/a+x6vBZ/wo9XeCnVoLM+FHq3wU5uOh9xr2vj+tfZ+151r7P2vqq2ngFJORFGCIXW+FZYDYo8\nI3G6clvxZeJGui1s+eM0TMRoqda+z9p1r7P2rnRR+RvsnRR+Rvstqwp9a+z9p1r7P2rnRR+RvsnR\nR+Rvsgp9a+z9p1r7P2rboo8J3G6clXoaVtTjxPLcNtBdRa0VjWWdKTed2HHWvs/ada+z9q72VFZ3\nxX5E/SnZUVh8V+ZH0rXxqN/KZeil1r7P2nWvs/au9lRXt0r9L8KdlRZ/Fflf6U41DlMvRS619n7T\nrX2ftXeyorN+K/O30qrXUjaXBhcXYgdRZTXLW06Qwvs+Skb0vq6FzepwZt+S3xCFPibZ+bdXeIVF\nQY+p09sfyG8lPv2fx6u8q5dvOXkr+qXOJtmZt1b4h5LqrqYmUcxbYPbESCHJv2Zx6t8vJQV+PqdR\nx/IdyURGswisRNofJ9q1+f8AKkz9U7Wr8v5UmXqqaLrblej0fCx9sey2/adbI0tfUvLXDCRfUK23\nhH4WStZvCPwsoiI8mda1r5Q9XjuE/herx3CfwpZNRgBY3IaBdWHILlnA38LpdfhY+2DiX6y5fkx1\nh3FY+yCLzXt3d9lsP4HfgrI2Pe81r6DRc/6hWtaeELGy2m2WNZaFxZ2Y1P1pcWbpqPrK63rP4tXc\nuSb1mcWo5clxXYZm17fBtyPfdZq09s3vDe+h1WYr2H0Q421ftkREW1XfbUJb1ODMfJb4hU+Jtn5t\n1d4hUVBj6nT2x/JbyU+/Z/Hq7yrj285eZv6pc4m2Zm3VviFMTcWreH/Iea637M49W+Xkvd/H9fD9\nvNYsHGJtn5t1d4hTE2zM26t8QrrfwycervKm/ZnHq3y8kFauc3qdRYt+S7xCvn28I/C+ir8fU6jj\n+Q7kvnW8I/Cv7J5S7H0702eoiK26IiIg8dwn8KPZBzmvbQd9lI7hP4XGxr/GtfQaLVm9ErOy/tho\nXFnZjU/WUu2zcxqPrXW9Z/Fq7km9ZnFqOSouw5uMWo4fP6pcWdmNT9a63sX1cPpzTes7i1dyQc4h\nZuY1H1rO2wR8G3I991p71mcWreSzds3vDe+h1stuH1wr7V+qX0dC5vU4M2/Jb4hU+Jtn5t1d4hUV\nBj6nT2x/IbyU+/Z/Hq7yqrbzl4C/qlzibZmbdW+IUxNxat4f8h5rrfszj1b5eS938f18P281iwcY\nm2fm3V3iFMTbMzbq3xCut+0nHq7ypv2Zx6t8vJBWrnN6nUZt+S7xCvn28I/C+ir8fU6i+P5DuS+d\nbwj8K/snpl2Pp3ps9REVt0RERB47hP4UeyLfGvyHfZSO4T+Fxse95rX0Gi1ZvRKzsv7YaFxZ2Y1P\n1pcWbmNR9a63rP4tXck3rM4tRy5Ki7Dm4xajh8/qlxZ2Y1P1rrexfVw+nNN6zuLV3JBzcWbmNR9a\nztsEEw2toe+6096zOLVvJZu2b3hvfQ62W3D64V9q/VL6OhA6nBkPkt8MqezbPyGrvDKgoSOpwZj5\nLfEKnuLP4dXeIVVt5y8Df1SWFmZDVvhlC1pdYhtsP+M80uLM4dW+IVFWPwU0zmOAcIiQRIcioiNZ\nYxEzOjroYrP+FHq7wl70MVmfCj1b4S+Q7Vrs/wCVJn6p2rX5fypMvVWuWv1dDkMvc+oroYhRzkRs\nB6F1vhWWC0bo/CqP2nWvaWvqXlrhhIvqFbbwj8KzhxzSJ1Xdmw2xRMWnV7ZeOG6fwvV47hP4W5bX\nw42ZvHUfSV6C4uyceHynmpGAFjbjuC6GWiy+35u50edx9qI4yH5nU/SVlbI1mvyHddbLzZjvwVjb\nIOc1/TvstObZ74aTvTrqVzUy5a7saebSsLO01P0FLDC3TUfQlxZ2mp+sry4s3TUfWVQX2dtgD4Nu\nR7rLNWltg/J/B77rNV7D6Icbav2yIiLarvtqEDqcGQ+S36Cp7Cz8hq7wyoKEjqcHD8lviEKe4s/N\nurvEK49vOXmb+qSwszIat8Mr2wx6Dh/xnmvLizOHVviFLjFq3h/ynmsWJYWfkNXeGUsLMyGrfDPJ\nLiz+HV3iFLizM26t8QoIK4DqdRkPku8Mr59vCPwvoK4jqdRp8l3iFfPt4R+Ff2Xyl1/p/ps9REVt\n0RERB47hP4Uex9Zr8h3XUjuE/hR7H1m/A77LVm9ErOy/thpWFn5DU/QUsLNyGo+gry4s7TU/WlxZ\numo+sqi7D2wxd3D5DzSws7Ian6Clxi7uHznmlxvaan6yiCws3Iaj6Fm7YteG3I91lo3Fm6aj6ys7\nbBv0NuR77rbh9cK+1fql9JQgdTgyHyW+GVPYWfkNXeGVBQkdTg0+S3xCFPcWfm3V3iFVbecvA39U\nlhZmQ1b4Z5L2wxaDh/xnmvLizOHVviFLjFq3h/yHmsWJYWfkNXeGUsLMyGrfDKXFn8OrvEKXFmcO\nrfEKCCuA6nUZD5LvDK+fbwj8L6CuI6nUafJd4hXz7eEfhX9l8pdf6d6bPbJZEVt0SyWREHjhun8K\nPZFrzX5DuupHcJ/Cj2RrN+B32WrN6JWNl/bDRsLOyGp+gr2ws3Iaj6CvLiztNT9ZS4s3TUfWqLsF\nhi0HD5DzXthZ2Q1P0FeXGLu4fOeaXFnaan6ygWFm5DUfQs7bAsYbcj3WWjcWbpqPrKztsEEw25Hv\nutuH1wr7V+qX01Bj6nT2x/Ib5VPv2fx6u8qzaOvomUsLX1MIIiAII0Km7RoLP/lQZk23VXtS2s+D\nwl8V96fx+Fzfszj1b5eSgr8fU6i+P5DvKou0aCzf5UGRF91Q1lfRPpZmsqICTEQAG6lK0trHgUxX\n3o/H4fJIiLrPRi1m8I/CyVotqIsI3wgmXjuE/hR9Yi84XjqiIg74RLZZwN/C6VVtfShoBmbovev0\nv+Zq7PEr1YJ38DvwVkbHvea19BpZX319KWOAmbosWmqjTF9mNdi5rn7dpesRVv2e8UyRazdGKz+L\nU8k3rM4tRyWT2o/P4Med07Ufl8GPKy5HBu6fN4uqTbN/g3vodbLMViqqjU4bsa3DyVdW8dZrXSXN\nz3i+SbQIiLNpfcUGPqdPbH8hvlU+/Z/Hq7yrNo6+iZSwtdUQgiIAgjQqbtGgs7+TBqfpXJtS2s+D\nzl8V96fx+Fzfszj1b5eSb+P6+H7eap9o0Fm/yYNRfdTtGgxf9qC2Hy+qx3LdPhjwr9vwub+GTj1d\n5UOOzOPVvl5Kn2jQWf8AyYMybbqdoUFm/wAmDUfSm5bocK/b8Ja/H1Oo4/kO8q+dbwj8LXrK+ifS\nzNbUQkmIgAN1KxG1EQaN8K9ssTETq6uwVmtZ1hKii6xF5wnWIvOFadBKii6xF5wnWIvOEEjuE/hc\nbGvea19BpZeGoiwnfCq0tU6mxYWNdi5rDJWbV0huwXimSLS3bOs/i1PJN6zOLVvJZPaj8/gx53Tt\nR+XwY8rKpwbulzeLq1t7F9XD6c03rP4tXclk9qPv8mPSydqPz+DHndODc5vF1au9ZnFq3ks7bN/g\n3vodbKPtR+XwY8rKCqqnVOHExrcPJbMeK1bay059ox3xzWJfYUOPqdPbH8hvlU+/Z/Hq7y8lm0df\nRMpYWvqIQREAQRoVN2hQWd/JgzJtuqnaltZ8Hir4r70/j8Lm/ZnHq3y8k38f18P281T7RoLN/kwa\ni+6naNBi/wCzBw+X1WO5box4V+34XN+z+PV3lQ47M49W+XkqfaNBZ/8AJg1Nt1O0aCzf5MGovupu\nW6fBwr9vwlr8fU6i+P5DvKvnW8I/C16yvon0szWVEJJiIAA1Kw21EWEb4V7ZYmInV1dgrNazrCZF\nF1iLzhOsRecK06CVFF1iLzhOsRecIJHcJ/C42Pe81r6DSy8NRFhO+FVpao02KzGuxc1hkrNq6Q3Y\nLxTJFpbu9Z/FqeSb1mcWo5LJ7Ufn8GPO6dqPy+DHlZVeDd0ubxdWtvYvq4fTmm9vcWp5LJ7Uff5M\nelk7Ufn8GPO6cG5zeLq1t6zeLUclm7ZveG99DrZR9pvy+DHlZQVVUanDdjW4eSzx4rVtrLTn2jHf\nHNYlZRetF3Acytqo/wCOSQukYJ2vd0jWRkDJ13Yc+Vj+Vac5iIr82yZ4KM1M0kLBewY51nO00FvU\nKN2z5m7PjrbsMUjsDQL3vnla3ogqItOm2FVVLYyx8LTI0PDXE3AJsL5d5KnP/Ga0RdJ0tORyDzcm\n17aa5IaMVFqv2DVMYMTmdI6VsbWX1Jc5t/dpUDtlVDdpGhxRmQC5cHboFr39kFFFru2G+CRgqZ2t\nbaQvLGkloaL5DK69m2EYqZ0/T4m9CJGANzvu3B5cQ5oaMdFpz7CqqeCWaeSCNsZLd5xBcRfIZZ8J\nVaKgmlon1TC0xsNnZ5g5WH93y/BQVUVihopa6rbSw4RK69g42uR3KSromwUzJmS47yOieMNrOba9\nuYzQU0REQIiICIiAi9aLuA5lbVR/xySF0jBO17ukayMgZOu7DnysfyiWIivzbJngozUzSQsF7Bjn\nWc7TQW9Qo3bPmbs+OtuwxSOwNAve+eVreiCoi0YtjzSQRTdJGGyxve3UndF7HLVSP2BUxsxSTU7G\n4Q67nnXlpqgykV6p2TUU20YqGR0ZllLQMLrgEm2f9qy3/jtU5pcyemcwXu5ryQABmdO7RBkItY7E\ndE9nWJmtZaRzywYiA0Xy53VOppWR0kFTFI57JbtOJti1wtca+oQVUREQIiICIiAiIgIiICL1ou4D\nmVtVH/HJIXSME7Xu6RrIyBk67sOfKx/KJYiK/NsmeCjNTNJCwXsGOdZztNBb1Cjds+Zuz4627DFI\n7A0C9755Wt6IKiLRi2PNJBFN0kYbLG97dSd0XsctVI/YFTGzFJNTsbhDruedeWmqDKRXqnZNTTbQ\njonYHTSWw4XXGZsrJ2BKyWFkk8fxHlt23Nhha6/s5BkItpuw43NhlbUkwztODc3gd61xfSzT3qEb\nCqer9YfLBHFYHE95GoBHd9wQ0ZaK1TUE1VBLNGWYIc33OgsTf9W/Kip6d9RUsgZYPebDFlmgiRX6\nrZpp6V8hkvLE8MmZbhLgSLHv0N1QQEREQIiICIiArBrqtzi41U5cRYkyG5HJQNF3Acytqf8A47JC\n6Rgna93SNZGQMjd2HPlY/lEszrtV0zZXTve9puC84rH+106vrJZMRqJcV7jC62foApZtkzwUZqZp\nIWC9msc4hztNBb7guWUVTDSMr43hrL2Y5riHYrnIeuV0F+i2dteWKIwVZjjczEz4xAt3/sgH1XJg\n2jDSTwGttkx0kYcb55AX/v8ACihpqqNsL461zC6mfM0NcQQBe7R+cKnbQbQmxupqyQxxU7HXkeW5\nObfCNf8A8ESS0O043OfPtINBLWYnTO3s3ZDLuLXKeOk2vM6SOGvG7O6K7yWvJF8725DuKqVtFVPb\nUydcfUNpCGymQm4dplzGZsfyq1LPtGqqBBHVz9ISXAOldm4D/aDRZQ7QjqoadlcX1bi57Y3EuZYX\naTc6k2OVtPZY76qrDheolyBYLPIGHkPRT1VTXU73U8tQ4ujed8OzB7xi117uYVifZ9VJQxy1FQws\nZAZIWi5JbiF+7m79IhPWUFVEWiWvklmbAKlrTvN3Sbi9+63JUanadaHCN07HYHtfiYwC5GncL2uo\nIKiow9WEuBstmEu7m3va+oFzdW9pUFUxklTWSA1HSNY+MDMXBtfu0HcgzzUTGd83SOEjyS5wNib6\nruernqWMZK/E1l7C1szqTzJ5qXZdD2hXspi/owQSXWvawuoKqA09TLCST0b3Mva17GyCJFs1Owmw\nYGdZcZXwGVrejsMhcgm+WSxkBERECIiArBrqtzi41U5cRYkyG5HJV0RKx12q6Zsrp3ve03BecVj/\nAGunV9ZLJiNRLivcYXWz9AFVXUT5I5Gvic5rwci02IQbFLQ7UnbT9HWYTgDo2mRwLA44R3ZXvZSy\n7D2rLBikrI5GgYQ0zE917aei4otn7Vmgp3QV2Bj2ksHSvGHeAtkLXJIXLmVUVNUwx18nR9CycsNw\nX3sLfgYuahKY022IosJq7vfK2INJu7MuaN45jNhVRtNtGCv7Pjqi1zd+7JTgGVyfZWJqasZJKJtp\nymQvjiYcTiHE5i5vkAqtbU7UoZ3081S/pGEYnB1yDbTFqpQuCirqWeMz15DR0sj7Xfawz3Tre/7X\nNbsuZ1IJZKkOjjgD4mtjAtwkgju4wb5qq7tGnpKet6y58cjjgBcXXOYdcH0/3+VzRNr68y9FVOHQ\nREkGQizO8C3d6IOp9hVVPBLNNJDG2Mlu84guIvkMs+EqtFQTS0T6thaYmGzs8wcsv7vl+Cpa99VD\nUNa6rknDWska4kkZgEHP8rRpXVNXsqSsnrCxkMzXEMiBLrWse69sSDGpKSWsn6CEAyEEhp77C9vy\npqygFNAyRsvSfEdE8WtZ7bXse8Z6rqeSp2btaZ4ka6cOJ6TCDe+dxyOarTVU00ccb3bkfC0Cw9T+\nfVBCiIiBERAREQFYNdVucXGqnLiLEmQ3I5KuiJWOu1XTNldO972m4Lzisf7XTq+slkxGolxXuMLr\nZ+gCqrqJ8kcjXxOc14ORabEINilodqTtp+jrMJwB0bTI4FgccI7sr3spZdh7VlgxSVkcjQMIaZie\n69tPRc0WzdsSxRGCsLI3MxR/GcBbv/ZAPquDBtGCkng67bJjpIg5188gCf77rhQlMabbEUWE1d5H\nytiAJu7MuaN45jNhVYx7UZtU0nXHOmB6Qu6UluTb3P8AXoppaHacbnPqNpBoJazG6V+9mchl3Frv\n/Cmjo9rzOkZDXi7Z3RXe5weSL53tyHcUEU1DWsnjNRXGNrekeeiv8PCLnC3LX+l5U7NqRRFz6+SS\nFsIkjZmR9N2kXy4hpddtodoMqoadtcX1bi97Y3OLmWzaTc6k2Pdp7LHfV1YcAaiXIFgs8gYeQ9PR\nSNisp9rCiqDV1FOyMHC69ml9ichYZ5tKoxtrJse1g6N5jdvFwBs4WAFrW78vwVcrKCqiIbNXySzN\ngFS1pOJu6TcXv3W5KjUbTrQ7o3VDX4HtfiY0C5GncL2uiHMRrNpOFI1wc67pMJFi92pvzOuqmbsc\nCoggfUjHNia0sbiaHg2sTl7qnDWzw1bqpjh07iTjIBzOp/K8hraiH5crhYEC+eG+tuRPNBARYkHu\nRERAiIgIiICsGuq3OLjVTFxFiTIbkclXRErHXqrpmzOnkdI04gXuxWP9ruXaVXN0eKUhzHFwLBhO\nI9+Xfkqi6je+ORr4nOa8aFpsQg2qbY+1JYqcxVbWscAW/FcMGIC17Dvxd3Nex7M2nmxle2zYw0tb\nK42FrhvsCeS7p6LbVU2CSKswN6ICIiUtAbYcvWwPqqpoq6nZORW4ZMLDIxsjrm+gP/4qErNVT1E0\n88UdXK+aOSOMAtDRIHWsXWOunsF66gqY9pwzSVLHVT5nB78Ic1tmh2IetjyGYUUmzdoROc6baLGC\n7WYjK7ezNgMu4tPtkpYtn7VmqHmHaBLm1BZaSRwJc3v0sch6oIRRS1MdO2WpPVJZCKd2G5L3HvH9\nZ8rhZhqamMlnTyjDdtg86aW/QW6afaHWKemZUwirs90bGRtDGi5BcCBxZE6aD+liVlIaYQHH0nSx\ndJcd2ZH/AIUoRyVEkkrJJCHOYGtFwLWAsBb+lI6vqnVDpjM7E54eQNCRplpkpqDZpq6WqqTJgjp2\ngkCxLsxoPwo62jFM92GVr2WY5txZzg5uIGyCIVU7al1Q2V7ZnEuL2mxudV5NUzTsYyV5eGEkX1uT\ncqJEFo7RqzG5jp3ODm4LuNyG8ge4KqiICIiIEREBF60XcBzK2p/+OyQmRona93SNZGQMjd2HPlY/\nlEsRdRvfHIHxOcx40c02IV2bZM0FGamaSFgvZrHOIc7TQW9QuW0dTDSM2hHIGsvZrmuIdiuch65X\nQaVDRbZkgb0NWGxSxbpdKbYcr25Z2BVWSirI4pA+sGKONrTGHuuGu+nl/WiloNl7SljgmpqxsYe0\n4D0jhh3gLZDIkkLmSGoYJTHXzWhpWPaC4glrsNxroLqEpnUW0oWvE1fEGEtbie8m5BIGHK4ILXJ2\nXX1cb6ZlVG6NtS5mGRxxFzQRfTQNHcclHU0M7nyNqq973iRkcZJLgXEXFz3AC/uq1bLtKhqX081Z\nNjjcC7DK4gOt/tSLsOzq2Cogpoau9WcTmR3uwNGIXB58R00us2eKWgEPRTu+NEJCWEgakW/SsTna\nLNmwVL6kuilecBDjjv3569wOvJc7ObXVYqBBUZRwuxiRxO4eK2qISbOhqaqhrZDUubDHE1rhfEXN\nDhkLnQapFFVU+0GUVHU3jlwyMx5MILQ4Fwz7v9KpUwvoKjBHKHgsY4uA3XAgHQ6j8rToBUTUs+1Z\nK6WMxyC4Yy9wABe17ZB39IOJdgV80kks1RAXG73vc46ZnFppkfZRdhvjY500zLGN7mdHvXLWlwue\nRA/YVevkrKWolpZKuV4aT9Zs6+d7et1AK2qAYBUzARizLPO6PTkgv9hySh76eRpijhZK50m6d5t7\nD11Xk2yI4pZ29YIDZWRRFzeIuF8+QsqLa2qa3C2pmAw4LB54eX4XQ2hVhrh1h7sVs3G5FtLE6W9E\nFqbZDomysfK1s8EjWStPC3Fob+neswixsrTNpVbGsDZjZjxILi5uNL87Ks5xc4uOZJuUHiIiIetF\n3Acytqf/AI6+EyNE7Xu6RrIyNHXdhz5EH8rEVg11Y55e6qnLiMJJkNyOSJTzbKmgozUzSRMbezWO\ndZztNBb7guW0dTBSM2hHIGsvZrmuIOK5yHrldRdeqjM2Z08j5GnEC92Kx55ruXaNZMWXlIcxxe0s\nAacR78u/JBfodmbSljgmpqxsYe04LSOGHeAtkMiSRkuZIahglMdfNaGlY9ocSCWuw3HKwv8A6U9D\nRbalgaIKzDFJFul0pthyuByzsCq0lDWxxPD6zejja0xh7smu+nl/WihKSooJ3PkbU17nvEjI4y4k\nhziLi5Olv/KqVc20qOfoZqqcPaQ8jpTk4i+fqtA0e0qfEZ66AsuxvxXF4cQXAAAjuLXI/Zu0a8zx\ndaje01bgWvJBc8Xu7TkO4qRUmdtFmz4p31BMT3XY4O3iTe4vr3XI/Cip4KvazRG2RpFO3R5tZpOZ\n9zmtJtFXMlpKOKpjFUwPMTGtGANzBN+8mx/oLFjnlphLHGQOkGBzsOZHoUQsVbp6LFSsn6SN8TCX\nAatIxWBOdrlWIaLtCjqNp1dQ4FrwHBrASRkCf2FPUbPqKWGSB1a50j6QSFtrtLWk3be+VrLIp62o\npm4YpXBmLEWatJ9RoUCtpnUdXJA5wcWHUd41CgXcsr5pXSyuLnvN3OPeVwgIiIgREQEREBERAXUb\n3xPD43OY9uYc02IXKIJo6qoiLTHPKwtBAwvItfVeisqha1RLk3AN85N5fhQIiUstVUTG808khy4n\nk6aLvtCs/wD659cXzDrzVdEFk7Qq3RhhqHmxJBvvZ656rllbUsDgJn2cwxm5vunUZ6XUCIOo5Hxu\nuw2vqO4+hHeFNLXVMzZWySkiUhzxYZ209lXRARERAiIgIiICIiApzXVbnFxqpi4ixJkNyOSgREp+\nu1XTNlM8jpGnEC84rH+1JLtGsnLAZTdji5pYA04j35d+SqLqOR8Tw+NzmPbo5psQg3aGh2zJA0QV\ngbFLFul0pthyvblnYFVpKCtjieH1e9HG1pjD3GzXfTy/rRZ8dVURFpinlYWggYXkWvqvRW1Qtapm\nybgG+cm8vwg130O0oA8TV8TWEsbie8m5BIGG4uCC13JOytoVUb6ZtUx0bakswyEglzQRfTQNHNY0\ntVUTG808shyO88nTT/ak7Qrf/wCufXF8w680GrFs2thqIKaGsvVnE5jL3YGi4uDz4u5Z08M1CIjF\nM/40XSExkgakW/S4O0at0YYah5sSQ6+9nrnquWVtSwOAmeQ5hjNzfdOoF9LoLuzoamqoa2Q1Lmwx\nxta5uTi5oIyF9ANV7FBVwV7KOjqT0cmGRpfkw3biBcMxp/pZkcj4nXYbX1HcRyI7wrDdp1jak1An\ncJCQctLgWGWmQQaEv/H6+aSSWaeAuJLnPc86ZnFppkfZRdhyRsLppmZxvczo96+FpcM8siB66hUH\nVtU5hY6pmLTe4LzY31QVlUAwColAjFmWed0enJBf7DllD308gMTIWSl0m6d4XsPXVeTbHZFLM3rB\nAbKyKIuZxFwvnyFvyqLa2qa3C2pmAw4LB54eX4XTa+qDXDp3uxW4jci2hBOlkFqbY7omysdK1s8E\njGStI3W4tCD3+uXus0ixIVpm0qtjYwJjuPEguLkuGl+dlVcS5xccyTcoPEREQIiIC6ifIyRronOa\n8HItNiFyuo5HxPD43OY9ujmmxCJbdJsbarmwSQVTWB7QWkSuGAOtYGw77jRGbL2m42bXN3I7ENlc\nS0EXw+wvbRZh2lWkxkVUrejYGNwuLbD+vwFGyqqGY8E8rceTrPIxflBq1NBPNNPEKyeaaOWNrBJo\n7Fob3yIuVK6hr45YYn7SmxumcHlryWts0OxDPM2PosXrlSSSaiW5ABOM92nsuxtCsEmPrUxdiDs3\nk3PNBqxw1UzYcddIKeZ+GCW13lzicj3jMZ58llVb5iyFk0oeGtIaB9O8b3/tSna1XiYWuY3o8WDD\nG0BpOpGWvqq0tRLNHEyR12xAtYOQvdB6yrnZC+FshwPaGkel72v3C/coURARERAiIgIiICIiAiqd\nZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7\nILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZ\nOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp\n1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyen\nsgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09\nk6zJ6eyC2iqdZk9PZOsyensgtoqnWZPT2TrMnp7ILaKp1mT09k6zJ6eyC2iqdZk9PZOsyensgtoq\nnWZPT2TrMnp7IN3YtXTUksrqoYmOZbDgDsXp6LzblXTVtcJaVmGPBa2HD3n/AMWH9LD6zJ6eydZk\n9PZE6oUREQIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIi\nAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIi\nICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAi\nIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiIC\nIiD/2Q==\n",
"text/html": [
"\n",
" ![\"term-document](\"images/document_term.png\")
\n",
"(source: [Introduction to Information Retrieval](http://player.slideplayer.com/15/4528582/#))\n",
"\n",
"We can decompose this into one tall thin matrix times one wide short matrix (possibly with a diagonal matrix in between).\n",
"\n",
"Notice that this representation does not take into account word order or sentence structure. It's an example of a **bag of words** approach."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Consider the most extreme case - reconstructing the matrix using an outer product of two vectors. Clearly, in most cases we won't be able to reconstruct the matrix exactly. But if we had one vector with the relative frequency of each vocabulary word out of the total word count, and one with the average number of words per document, then that outer product would be as close as we can get.\n",
"\n",
"Now consider increasing that matrices to two columns and two rows. The optimal decomposition would now be to cluster the documents into two groups, each of which has as different a distribution of words as possible to each other, but as similar as possible amongst the documents in the cluster. We will call those two groups \"topics\". And we would cluster the words into two groups, based on those which most frequently appear in each of the topics. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### In today's class"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We'll take a dataset of documents in several different categories, and find topics (consisting of groups of words) for them. Knowing the actual categories helps us evaluate if the topics we find make sense.\n",
"\n",
"We will try this with two different matrix factorizations: **Singular Value Decomposition (SVD)** and **Non-negative Matrix Factorization (NMF)**"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.datasets import fetch_20newsgroups\n",
"from sklearn import decomposition\n",
"from scipy import linalg\n",
"import matplotlib.pyplot as plt"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"np.set_printoptions(suppress=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional Resources"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [Data source](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html): Newsgroups are discussion groups on Usenet, which was popular in the 80s and 90s before the web really took off. This dataset includes 18,000 newsgroups posts with 20 topics.\n",
"- [Chris Manning's book chapter](https://nlp.stanford.edu/IR-book/pdf/18lsi.pdf) on matrix factorization and LSI \n",
"- Scikit learn [truncated SVD LSI details](http://scikit-learn.org/stable/modules/decomposition.html#lsa)\n",
"\n",
"### Other Tutorials\n",
"- [Scikit-Learn: Out-of-core classification of text documents](http://scikit-learn.org/stable/auto_examples/applications/plot_out_of_core_classification.html): uses [Reuters-21578](https://archive.ics.uci.edu/ml/datasets/reuters-21578+text+categorization+collection) dataset (Reuters articles labeled with ~100 categories), HashingVectorizer\n",
"- [Text Analysis with Topic Models for the Humanities and Social Sciences](https://de.dariah.eu/tatom/index.html): uses [British and French Literature dataset](https://de.dariah.eu/tatom/datasets.html) of Jane Austen, Charlotte Bronte, Victor Hugo, and more"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Scikit Learn comes with a number of built-in datasets, as well as loading utilities to load several standard external datasets. This is a [great resource](http://scikit-learn.org/stable/datasets/), and the datasets include Boston housing prices, face images, patches of forest, diabetes, breast cancer, and more. We will be using the newsgroups dataset.\n",
"\n",
"Newsgroups are discussion groups on Usenet, which was popular in the 80s and 90s before the web really took off. This dataset includes 18,000 newsgroups posts with 20 topics. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"categories = ['alt.atheism', 'talk.religion.misc', 'comp.graphics', 'sci.space']\n",
"remove = ('headers', 'footers', 'quotes')\n",
"newsgroups_train = fetch_20newsgroups(subset='train', categories=categories, remove=remove)\n",
"newsgroups_test = fetch_20newsgroups(subset='test', categories=categories, remove=remove)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((2034,), (2034,))"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"newsgroups_train.filenames.shape, newsgroups_train.target.shape"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's look at some of the data. Can you guess which category these messages are in?"
]
},
{
"cell_type": "code",
"execution_count": 55,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hi,\n",
"\n",
"I've noticed that if you only save a model (with all your mapping planes\n",
"positioned carefully) to a .3DS file that when you reload it after restarting\n",
"3DS, they are given a default position and orientation. But if you save\n",
"to a .PRJ file their positions/orientation are preserved. Does anyone\n",
"know why this information is not stored in the .3DS file? Nothing is\n",
"explicitly said in the manual about saving texture rules in the .PRJ file. \n",
"I'd like to be able to read the texture rule information, does anyone have \n",
"the format for the .PRJ file?\n",
"\n",
"Is the .CEL file format available from somewhere?\n",
"\n",
"Rych\n",
"\n",
"\n",
"Seems to be, barring evidence to the contrary, that Koresh was simply\n",
"another deranged fanatic who thought it neccessary to take a whole bunch of\n",
"folks with him, children and all, to satisfy his delusional mania. Jim\n",
"Jones, circa 1993.\n",
"\n",
"\n",
"Nope - fruitcakes like Koresh have been demonstrating such evil corruption\n",
"for centuries.\n",
"\n",
" >In article <1993Apr19.020359.26996@sq.sq.com>, msb@sq.sq.com (Mark Brader) \n",
"\n",
"MB> So the\n",
"MB> 1970 figure seems unlikely to actually be anything but a perijove.\n",
"\n",
"JG>Sorry, _perijoves_...I'm not used to talking this language.\n",
"\n",
"Couldn't we just say periapsis or apoapsis?\n",
"\n",
" \n"
]
}
],
"source": [
"print(\"\\n\".join(newsgroups_train.data[:3]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"hint: definition of *perijove* is the point in the orbit of a satellite of Jupiter nearest the planet's center "
]
},
{
"cell_type": "code",
"execution_count": 249,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array(['comp.graphics', 'talk.religion.misc', 'sci.space'], \n",
" dtype='![\"PCA](\"images/face_pca.png\")
\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))\n",
"\n",
"A more interpretable approach:\n",
"\n",
"![\"NMF](\"images/face_outputs.png\")
\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Idea"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Rather than constraining our factors to be *orthogonal*, another idea would to constrain them to be *non-negative*. NMF is a factorization of a non-negative data set $V$: $$ V = W H$$ into non-negative matrices $W,\\; H$. Often positive factors will be **more easily interpretable** (and this is the reason behind NMF's popularity). \n",
"\n",
"![\"NMF](\"images/face_nmf.png\")
\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))\n",
"\n",
"Nonnegative matrix factorization (NMF) is a non-exact factorization that factors into one skinny positive matrix and one short positive matrix. NMF is NP-hard and non-unique. There are a number of variations on it, created by adding different constraints. "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Applications of NMF"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- [Face Decompositions](http://scikit-learn.org/stable/auto_examples/decomposition/plot_faces_decomposition.html#sphx-glr-auto-examples-decomposition-plot-faces-decomposition-py)\n",
"- [Collaborative Filtering, eg movie recommendations](http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/)\n",
"- [Audio source separation](https://pdfs.semanticscholar.org/cc88/0b24791349df39c5d9b8c352911a0417df34.pdf)\n",
"- [Chemistry](http://ieeexplore.ieee.org/document/1532909/)\n",
"- [Bioinformatics](https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-015-0485-4) and [Gene Expression](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2623306/)\n",
"- Topic Modeling (our problem!)\n",
"\n",
"![\"NMF](\"images/nmf_doc.png\")
\n",
"\n",
"(source: [NMF Tutorial](http://perso.telecom-paristech.fr/~essid/teach/NMF_tutorial_ICME-2014.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**More Reading**:\n",
"\n",
"- [The Why and How of Nonnegative Matrix Factorization](https://arxiv.org/pdf/1401.5226.pdf)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### NMF from sklearn"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"First, we will use [scikit-learn's implementation of NMF](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html):"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"m,n=vectors.shape\n",
"d=5 # num topics"
]
},
{
"cell_type": "code",
"execution_count": 363,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"clf = decomposition.NMF(n_components=d, random_state=1)\n",
"\n",
"W1 = clf.fit_transform(vectors)\n",
"H1 = clf.components_"
]
},
{
"cell_type": "code",
"execution_count": 296,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['jpeg image gif file color images format quality',\n",
" 'edu graphics pub mail 128 ray ftp send',\n",
" 'space launch satellite nasa commercial satellites year market',\n",
" 'jesus matthew prophecy people said messiah david isaiah',\n",
" 'image data available software processing ftp edu analysis',\n",
" 'god atheists atheism religious believe people religion does']"
]
},
"execution_count": 296,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H1)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### TF-IDF"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[Topic Frequency-Inverse Document Frequency](http://www.tfidf.com/) (TF-IDF) is a way to normalize term counts by taking into account how often they appear in a document, how long the document is, and how commmon/rare the term is.\n",
"\n",
"TF = (# occurrences of term t in document) / (# of words in documents)\n",
"\n",
"IDF = log(# of documents / # documents with term t in it)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"vectorizer_tfidf = TfidfVectorizer(stop_words='english')\n",
"vectors_tfidf = vectorizer_tfidf.fit_transform(newsgroups_train.data) # (documents, vocab)"
]
},
{
"cell_type": "code",
"execution_count": 263,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"W1 = clf.fit_transform(vectors_tfidf)\n",
"H1 = clf.components_"
]
},
{
"cell_type": "code",
"execution_count": 255,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"['don people just think like know say religion',\n",
" 'thanks graphics files image file program windows format',\n",
" 'space nasa launch shuttle orbit lunar moon earth',\n",
" 'ico bobbe tek beauchaine bronx manhattan sank queens',\n",
" 'god jesus bible believe atheism christian does belief',\n",
" 'objective morality values moral subjective science absolute claim']"
]
},
"execution_count": 255,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(H1)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"[![\"pytorch\"](\"images/what_is_pytorch.png\")
\n",
"\n",
"**Further learning**: If you are curious to learn what *dynamic* neural networks are, you may want to watch [this talk](https://www.youtube.com/watch?v=Z15cBAuY7Sc) by Soumith Chintala, Facebook AI researcher and core PyTorch contributor.\n",
"\n",
"If you want to learn more PyTorch, you can try this [tutorial](http://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) or this [learning by examples](http://pytorch.org/tutorials/beginner/pytorch_with_examples.html)."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Note about GPUs**: If you are not using a GPU, you will need to remove the `.cuda()` from the methods below. GPU usage is not required for this course, but I thought it would be of interest to some of you. To learn how to create an AWS instance with a GPU, you can watch the [fast.ai setup lesson](http://course.fast.ai/lessons/aws.html)."
]
},
{
"cell_type": "code",
"execution_count": 282,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"import torch\n",
"import torch.cuda as tc\n",
"from torch.autograd import Variable"
]
},
{
"cell_type": "code",
"execution_count": 283,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def V(M): return Variable(M, requires_grad=True)"
]
},
{
"cell_type": "code",
"execution_count": 284,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"v=vectors_tfidf.todense()"
]
},
{
"cell_type": "code",
"execution_count": 285,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"t_vectors = torch.Tensor(v.astype(np.float32)).cuda()"
]
},
{
"cell_type": "code",
"execution_count": 286,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"mu = 1e-5"
]
},
{
"cell_type": "code",
"execution_count": 287,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def grads_t(M, W, H):\n",
" R = W.mm(H)-M\n",
" return (R.mm(H.t()) + penalty_t(W, mu)*lam, \n",
" W.t().mm(R) + penalty_t(H, mu)*lam) # dW, dH\n",
"\n",
"def penalty_t(M, mu):\n",
" return (M![\"research](\"images/nimfa.png\")
\n",
"source: [Python Nimfa Documentation](http://nimfa.biolab.si/)\n",
"\n",
"#### Using PyTorch and SGD\n",
"- Took us an hour to implement, didn't have to be NMF experts\n",
"- Parameters were fiddly\n",
"- Not as fast (tried in numpy and was so slow we had to switch to PyTorch)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Truncated SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We saved a lot of time when we calculated NMF by only calculating the subset of columns we were interested in. Is there a way to get this benefit with SVD? Yes there is! It's called truncated SVD. We are just interested in the vectors corresponding to the **largest** singular values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![\"\"](\"images/svd_fb.png\")
\n",
"(source: [Facebook Research: Fast Randomized SVD](https://research.fb.com/fast-randomized-svd/))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Shortcomings of classical algorithms for decomposition:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Matrices are \"stupendously big\"\n",
"- Data are often **missing or inaccurate**. Why spend extra computational resources when imprecision of input limits precision of the output?\n",
"- **Data transfer** now plays a major role in time of algorithms. Techniques the require fewer passes over the data may be substantially faster, even if they require more flops (flops = floating point operations).\n",
"- Important to take advantage of **GPUs**.\n",
"\n",
"(source: [Halko](https://arxiv.org/abs/0909.4061))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Advantages of randomized algorithms:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- inherently stable\n",
"- performance guarantees do not depend on subtle spectral properties\n",
"- needed matrix-vector products can be done in parallel\n",
"\n",
"(source: [Halko](https://arxiv.org/abs/0909.4061))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Randomized SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Reminder: full SVD is **slow**. This is the calculation we did above using Scipy's Linalg SVD:"
]
},
{
"cell_type": "code",
"execution_count": 384,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(2034, 26576)"
]
},
"execution_count": 384,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"vectors.shape"
]
},
{
"cell_type": "code",
"execution_count": 344,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 27.2 s, sys: 812 ms, total: 28 s\n",
"Wall time: 27.9 s\n"
]
}
],
"source": [
"%time U, s, Vh = linalg.svd(vectors, full_matrices=False)"
]
},
{
"cell_type": "code",
"execution_count": 345,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(2034, 2034) (2034,) (2034, 26576)\n"
]
}
],
"source": [
"print(U.shape, s.shape, Vh.shape)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Fortunately, there is a faster way:"
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 144 ms, sys: 8 ms, total: 152 ms\n",
"Wall time: 154 ms\n"
]
}
],
"source": [
"%time u, s, v = decomposition.randomized_svd(vectors, 5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The runtime complexity for SVD is $\\mathcal{O}(\\text{min}(m^2 n,\\; m n^2))$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Question**: How can we speed things up? (without new breakthroughs in SVD research)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Idea**: Let's use a smaller matrix (with smaller $n$)!\n",
"\n",
"Instead of calculating the SVD on our full matrix $A$ which is $m \\times n$, let's use $B = A Q$, which is just $m \\times r$ and $r << n$\n",
"\n",
"We haven't found a better general SVD method, we are just using the method we have on a smaller matrix."
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 144 ms, sys: 8 ms, total: 152 ms\n",
"Wall time: 154 ms\n"
]
}
],
"source": [
"%time u, s, v = decomposition.randomized_svd(vectors, 5)"
]
},
{
"cell_type": "code",
"execution_count": 177,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((2034, 5), (5,), (5, 26576))"
]
},
"execution_count": 177,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u.shape, s.shape, v.shape"
]
},
{
"cell_type": "code",
"execution_count": 178,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['jpeg image edu file graphics images gif data',\n",
" 'jpeg gif file color quality image jfif format',\n",
" 'space jesus launch god people satellite matthew atheists',\n",
" 'jesus god matthew people atheists atheism does graphics',\n",
" 'image data processing analysis software available tools display']"
]
},
"execution_count": 178,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(v)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are some results from [Facebook Research](https://research.fb.com/fast-randomized-svd/):\n",
"\n",
"![\"\"](\"images/randomizedSVDbenchmarks.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Johnson-Lindenstrauss Lemma**: ([from wikipedia](https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma)) a small set of points in a high-dimensional space can be embedded into a space of much lower dimension in such a way that distances between the points are nearly preserved.\n",
"\n",
"It is desirable to be able to reduce dimensionality of data in a way that preserves relevant structure. The Johnson–Lindenstrauss lemma is a classic result of this type."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Implementing our own Randomized SVD"
]
},
{
"cell_type": "code",
"execution_count": 112,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from scipy import linalg"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The method `randomized_range_finder` finds an orthonormal matrix whose range approximates the range of A (step 1 in our algorithm above). To do so, we use the LU and QR factorizations, both of which we will be covering in depth later.\n",
"\n",
"I am using the [scikit-learn.extmath.randomized_svd source code](https://github.com/scikit-learn/scikit-learn/blob/14031f65d144e3966113d3daec836e443c6d7a5b/sklearn/utils/extmath.py) as a guide."
]
},
{
"cell_type": "code",
"execution_count": 182,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# computes an orthonormal matrix whose range approximates the range of A\n",
"# power_iteration_normalizer can be safe_sparse_dot (fast but unstable), LU (imbetween), or QR (slow but most accurate)\n",
"def randomized_range_finder(A, size, n_iter=5):\n",
" Q = np.random.normal(size=(A.shape[1], size))\n",
" \n",
" for i in range(n_iter):\n",
" Q, _ = linalg.lu(A @ Q, permute_l=True)\n",
" Q, _ = linalg.lu(A.T @ Q, permute_l=True)\n",
" \n",
" Q, _ = linalg.qr(A @ Q, mode='economic')\n",
" return Q"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And here's our randomized SVD method:"
]
},
{
"cell_type": "code",
"execution_count": 236,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def randomized_svd(M, n_components, n_oversamples=10, n_iter=4):\n",
" \n",
" n_random = n_components + n_oversamples\n",
" \n",
" Q = randomized_range_finder(M, n_random, n_iter)\n",
" \n",
" # project M to the (k + p) dimensional space using the basis vectors\n",
" B = Q.T @ M\n",
" \n",
" # compute the SVD on the thin matrix: (k + p) wide\n",
" Uhat, s, V = linalg.svd(B, full_matrices=False)\n",
" del B\n",
" U = Q @ Uhat\n",
" \n",
" return U[:, :n_components], s[:n_components], V[:n_components, :]"
]
},
{
"cell_type": "code",
"execution_count": 237,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"u, s, v = randomized_svd(vectors, 5)"
]
},
{
"cell_type": "code",
"execution_count": 238,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 136 ms, sys: 0 ns, total: 136 ms\n",
"Wall time: 137 ms\n"
]
}
],
"source": [
"%time u, s, v = randomized_svd(vectors, 5)"
]
},
{
"cell_type": "code",
"execution_count": 239,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((2034, 5), (5,), (5, 26576))"
]
},
"execution_count": 239,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"u.shape, s.shape, v.shape"
]
},
{
"cell_type": "code",
"execution_count": 247,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['jpeg image edu file graphics images gif data',\n",
" 'edu graphics data space pub mail 128 3d',\n",
" 'space jesus launch god people satellite matthew atheists',\n",
" 'space launch satellite commercial nasa satellites market year',\n",
" 'image data processing analysis software available tools display']"
]
},
"execution_count": 247,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"show_topics(v)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Write a loop to calculate the error of your decomposition as you vary the # of topics. Plot the result"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Answer"
]
},
{
"cell_type": "code",
"execution_count": 248,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Exercise: Write a loop to calculate the error of your decomposition as you vary the # of topics\n"
]
},
{
"cell_type": "code",
"execution_count": 242,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[![\"Robust](\"images/faces_rpca.png\")
\n",
" ([Source: Jean Kossaifi](https://jeankossaifi.github.io/tensorly/rpca.html))\n",
"\n",
" ![\"Robust](\"images/faces_rpca_gross.png\")
\n",
" ([Source: Jean Kossaifi](https://jeankossaifi.github.io/tensorly/rpca.html))\n",
"\n",
"- Latent Semantic Indexing: $L$ captures common words used in all documents while $S$ captures the few key words that best distinguish each document from others\n",
"\n",
"- Ranking and Collaborative Filtering: a small portion of the available rankings could be noisy and even tampered with (see [Netflix RAD - Outlier Detection on Big Data](http://techblog.netflix.com/2015/02/rad-outlier-detection-on-big-data.html) on the official netflix blog)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### L1 norm induces sparsity"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The unit ball $\\lVert x \\rVert_1 = 1$ is a diamond in the L1 norm. It's extrema are the corners:"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
" ![\"L1](\"images/L1vsL2.jpg\")
\n",
" ([Source](https://www.quora.com/Why-is-L1-regularization-supposed-to-lead-to-sparsity-than-L2))\n",
" \n",
" A similar perspective is to look at the *contours* of the loss function:\n",
" ![\"L2](\"images/L1vsL2_2.png\")
\n",
" ([Source](https://www.quora.com/Why-is-L1-regularization-better-than-L2-regularization-provided-that-all-Norms-are-equivalent))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### Optimization Problem"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Robust PCA can be written:\n",
"\n",
"$$ minimize\\; \\lVert L \\rVert_* + \\lambda\\lVert S \\rVert_1 \\\\ subject\\;to\\; L + S = M$$\n",
" \n",
"where:\n",
"\n",
"- $\\lVert \\cdot \\rVert_1$ is the **L1 norm**. Minimizing the [L1 norm](https://medium.com/@shiyan/l1-norm-regularization-and-sparsity-explained-for-dummies-5b0e4be3938a) results in sparse values. For a matrix, the L1 norm is equal to the [maximum absolute column norm](https://math.stackexchange.com/questions/519279/why-is-the-matrix-norm-a-1-maximum-absolute-column-sum-of-the-matrix).\n",
"\n",
"- $ \\lVert \\cdot \\rVert_* $ is the **nuclear norm**, which is the L1 norm of the singular values. Trying to minimize this results in sparse singular values --> low rank.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Implementing an algorithm from a paper"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Source\n",
"We will use the general **primary component pursuit algorithm** from this [Robust PCA paper](https://arxiv.org/pdf/0912.3599.pdf) (Candes, Li, Ma, Wright), in the specific form of **Matrix Completion via the Inexact ALM Method** found in [section 3.1 of this paper](https://arxiv.org/pdf/1009.5055.pdf) (Lin, Chen, Ma). I also referenced the implemenations found [here](https://github.com/shriphani/robust_pcp/blob/master/robust_pcp.py) and [here](https://github.com/dfm/pcp/blob/master/pcp.py)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### The Good Parts\n",
"\n",
"Section 1 of Candes, Li, Ma, Wright is nicely written, and section 5 Algorithms is our key interest. **You don't need to know the math or understand the proofs to implement an algorithm from a paper.** You will need to try different things and comb through resources for useful tidbits. This field has more theoretical researchers and less pragmatic advice. It took months to find what I needed and get this working.\n",
"\n",
"The algorithm shows up on page 29:\n",
"\n",
" ![\"PCP](\"images/pcp_algorithm.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"needed definitions of $\\mathcal{S}$, the Shrinkage operator, and $\\mathcal{D}$, the singular value thresholding operator:\n",
"\n",
" ![\"PCP](\"images/candes.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Section 3.1 of [Chen, Lin, Ma]() contains a faster vartiation of this:\n",
"\n",
" ![\"Inexact](\"images/rpca_inexact.png\")
\n",
"\n",
"And Section 4 has some very helpful implementation details on how many singular values to calculate (as well as how to choose the parameter values):\n",
"\n",
" ![\"SVP](\"images/svp_value.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### If you want to learn more of the theory:\n",
"\n",
"- Convex Optimization by Stephen Boyd (Stanford Prof): \n",
" - [OpenEdX Videos](https://www.youtube.com/playlist?list=PLbBM_dvjud8oFj09MqqYnGSrT6zek42Q0)\n",
" - [Jupyter Notebooks](http://web.stanford.edu/~boyd/papers/cvx_short_course.html)\n",
"- Alternating Direction Method of Multipliers (more [Stephen Boyd](http://stanford.edu/~boyd/admm.html))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Robust PCA (via Primary Component Pursuit)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Methods"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We will use [Facebook's Fast Randomized PCA](https://github.com/facebook/fbpca) library."
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"from scipy import sparse\n",
"from sklearn.utils.extmath import randomized_svd\n",
"import fbpca"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"TOL=1e-9\n",
"MAX_ITERS=3"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def converged(Z, d_norm):\n",
" err = np.linalg.norm(Z, 'fro') / d_norm\n",
" print('error: ', err)\n",
" return err < TOL"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def shrink(M, tau):\n",
" S = np.abs(M) - tau\n",
" return np.sign(M) * np.where(S>0, S, 0)"
]
},
{
"cell_type": "code",
"execution_count": 52,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def _svd(M, rank): return fbpca.pca(M, k=min(rank, np.min(M.shape)), raw=True)"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def norm_op(M): return _svd(M, 1)[1][0]"
]
},
{
"cell_type": "code",
"execution_count": 54,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def svd_reconstruct(M, rank, min_sv):\n",
" u, s, v = _svd(M, rank)\n",
" s -= min_sv\n",
" nnz = (s > 0).sum()\n",
" return u[:,:nnz] @ np.diag(s[:nnz]) @ v[:nnz], nnz"
]
},
{
"cell_type": "code",
"execution_count": 97,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def pcp(X, maxiter=10, k=10): # refactored\n",
" m, n = X.shape\n",
" trans = m"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Results"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"240"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m, n = M.shape\n",
"round(m * .05)"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"rank sv: 1\n",
"error: 0.131637937114\n",
"rank sv: 241\n",
"error: 0.0458515689278\n",
"rank sv: 49\n",
"error: 0.00591314217762\n",
"rank sv: 289\n",
"error: 0.000567221885441\n",
"rank sv: 529\n",
"error: 2.4633786172e-05\n"
]
}
],
"source": [
"L, S, examples = pcp(M, maxiter=5, k=10)"
]
},
{
"cell_type": "code",
"execution_count": 96,
"metadata": {
"hidden": true,
"scrolled": false
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAugAAARiCAYAAAD2oQIUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3UmsZld5vv1VNMblak71javclF2mMzbGEEJA8JdIQCEj\nxIBRMsgkmZB5BpEyQ4kiwzzTZJhESEgRCkySgINxsDHBYNxX35zqq1xl0/kbfDqLay/2vWu9Zex3\n21y/0ar37Gbt5thb777P86x79dVXiyRJkqR5eNuyJyBJkiTp13xAlyRJkmbEB3RJkiRpRnxAlyRJ\nkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbkHW/kztatW/fq\nG7k/SW+8V199dd2y5yC92Tz00EP1/49ve9uvvzt7xzveMTrmMqWUsm7dutHx29/+9tH9nTt3ro5P\nnTpVx08++WQdnzhxYrDOL37xizreuHHj6Hb5+bve9a46vvnmm+v4pptuGl23lFI2bNhQx7/85S/r\n+PTp03X8q1/9qo43bdpUxysrK3V899131/FHPvKRwT4OHTpUxxcuXBjdH8/vlStX6nj37t2jx9Ge\nZ16DV1/99aMPt9tzndK67c8497QMr1+aa7sdrvPOd75zdJ00j5///Od1nM5zKaW89NJLdXz+/Pk6\nfvnll+t47969o3PnObl8+fLo56UMzy/vH849nSvOI43H/r3mmWee4bj7/49+gy5JkiTNyBv6Dbok\nSZqWvtXjuMVvNJP0LS2/0dy1a1cd89viUobfoPJnZ8+eHd0uv9Hmt6Tr16+Pc+Qxrq6u1nE6Pn7O\nb+z5OedRSik/+9nP6pjnJH2TzG9T+RZj6hv0pOdNR8837u28uA6/IU7zSt+At8unb80pfSOdjrX9\nBp3r90hvAjhXXuNSht/m843ItWvXRud19erV0flx3M6D++c4zfd6/AZdkiRJmhEf0CVJkqQZMeIi\nSdKS8TV4+wdua1KMoV2HY26Xf1DJqAYxGtBGXG677bY6ZmyEsQhGBvianxEDal//p4gEt8W5cx5c\nhn+U+LWvfW2wDx7j1q1b65h/ZLh58+Y6Tn+0S238g//mtUoxEe7jlltuqeNt27bV8Z49ewbrHD9+\nvI6PHDlSxzyn6Y9H+fnUfZWiMGmdFLvh/tp99ERcGEvhtnjNeD7abfKPl7mtFDNKkax0/dttpfO+\nCL9BlyRJkmbEB3RJkiRpRoy4SJK0ZIyApJrNUxUkUsSF23rf+95Xx9/73vfqmFEExkT4mr+UUl58\n8cU65qt+zp3zStGOV155ZXSupeRKLFu2bKljxk8YP0hRiYsXLw7+naptpCgDIzE/+clPRufXaquI\nrOH1YAzirrvuquP9+/ePzonxoVJK+cEPflDHP/rRj+qY54HXhuea22Wkpo1wsFIN1+G2uA9eW54D\n1gifquLDGA+vM++Ln/70p3XM83np0qXRddv5siY/58jzln6fUgSr/Rn3kXoGXI/foEuSJEkz4gO6\nJEmSNCM+oEuSJEkzYgZdkqQlY7m4lHllXrot48blNmzYUMfM7rIzZypbN9U5khndlN1lhjnllKeO\ng7gtjnmszGWn/bUdWNkl8ty5c3XMnDRzw5zjE088UcfMW7d5/TNnztRx6ubJc/jAAw/UMa8TMQ9e\nSilHjx4dnTsz5akj5o4dO+r41ltvreP2mjNLfc8999Qxj+mpp56q4//8z/+sY16D++67r47/5E/+\nZLAP/m0Es/i8j3muT58+PbqPnTt31nGb+77//vvLGN5XPFf8/Xj++efrmPcby5aWMjxX+/btG93W\nIvwGXZIkSZoRH9AlSZKkGTHiIknSkp09e3b089RNs+3MybgEX7Wz3NzDDz9cx1Pl4ta0JRAZa+A+\nOBfGMFJUh5GR9jgY1WCsgculEoapvGRbOpDRBC7HTpTsosr1uXzqQtni8XJbLEnJTqDc9wsvvDC6\nv1KmYzxrUldazjeNSxnec4zOcDlec8ZoUrfSdh/vfve765hxm0OHDtUxrzmjKPyc5Rs/+9nPDvbB\nGM2dd95Zx4zRsJQnI0tf/epX65hlNnlu2v3zHkvxmuvxG3RJkiRpRnxAlyRJkmbEiIskSUvGrpCM\nMqysrNQxYyXHjh0brM8uiowicMxIxYc//OE65qt9RhHa2ATn1UZT1jCWwv1NVWuhVLWG0YnUiTLF\ndtpIBX+2devWOv7Qhz40ukyK6kxFTFJXSuK5YuUWdtNMVUZKGcaBOE6Rk9Txdeqa89+pMyxxu+n6\nt11weVyMZKVKMewMy22xYhGr85RSype//OU6/tSnPlXHrPbypS99afTzAwcO1DGjSG10ivvkz3iP\nLcJv0CVJkqQZ8QFdkiRJmhEjLpIkLRljDVNVNdbcfvvtg38zfpAqmLSv5NewSRJjF4zUlJIrhXDf\nSaqw0jbfYWSBDV5S3IXRiRQraefH42XjGR4fq4lwTpwHz097HNu2bRvdbmrkxM8vXLhQx7zOrOjS\n4vqMeqTKP4yopBhLKcPzyzHnnraVYjTtPh599NE6fuaZZ+qY54HHxG3xWtL/+3//b/DvP/zDP6xj\nxsaI9z6Pj3EwNqBqGxC9//3vr2NWgeE9tgi/QZckSZJmxAd0SZIkaUaMuEiStGSsGsHIQIpETGGE\nJFU2SZU7GCVoMZrAcdrHVMxkDaMA7dw5l9T0huenJ2pTSm6GxH0wcsR5fPSjHx39vK1qkiIdqZpJ\nauTERjqsMtMeR7oexGNNzZ6mKvdwf6k5FZvyMNqxa9euOm6jVmwQxQo9H//4x0f3zXvm6NGjdcy4\nC7dZyrAZEquq8Di+/e1v1zErG7Fh0ve///06bq8597Fjx4467q1g1PIbdEmSJGlGfECXJEmSZmRd\nehXyuuxs3bo3bmeSluLVV18d72AhKfriF79Y//+YmuwwMtLGXaZ+dj2MHKQYTCv9LM03Ld/GBLg+\nf5biIL3zJcYaGJ1gFOG+++4b/ZwVQNhUp63i0lPlpKf6Ca8NY1ClDJtTpXgPq7hQiri08ZPUbIhj\nNhriPBhF4VwZHymllNOnT9cxq+fcc889o+s//vjjdcwGX+fPnx+dd/tv3mObNm2qY8ZrUiMuXgNW\ndCllWHWG55H32AsvvND9/0e/QZckSZJmxAd0SZIkaUas4iJJ0pKlaEfSs0yvNg7QI0Vqesap0ksp\nw2hBqjQztf4aRi3ayjSMNXBbe/furWNGVs6ePTu6DzaqaeeRGkcxcsIYRKriwrm3EZdz586NrpMq\nyPRU8Wkrjtxyyy2jP0uREW6LjZxS9KnF7bIKzCOPPFLHrEZz8ODB0c9XV1cH22WMhtVeTp48WceM\n3hw4cKCOGWXiOWQ1mFKG9y6Pg9GbRfgNuiRJkjQjPqBLkiRJM2LERZKkJeMr8RQHYeWNtsoEYxup\nKkuKxfTGD1LMhLGNFHdITY5695EiQD2VWxh9KCU32WFVjlQ1hrGLqWPiv3lOGRnhMXFOXJ6VRdjs\np5RSjh07VsYwWpIiNTy+FCtq55KiRfw8VUKZaqTEn7HBEOfI42CsheeT22ljW4wjsSJMup78PN0L\nbXSK91JqTrUIv0GXJEmSZsQHdEmSJGlGfECXJEmSZsQMuiRJS9aTU2VJuDZjyzws86/MHXOcSiAy\nFz2Vq05zn8qXj+1jqiNqkvLMKRff2xGVnS9fq5RBZhaa5yHNiddsz549g+WY1+Y+eA57/h6htwNr\n2gfvPR5T6jzaditlbp3b4t9c7N+/f3TfzJPz8/ZvNFIJRHb5TN1DU96+/Z1N3WB7fidG93tDa0mS\nJEl6XfiALkmSJM2IERdJkpYslXJjZIBl9trX64wGsMweX8+nGE0qh9jGTVJspKeraYpRTJXc64m7\ncBnOL8UVWj3xg1RecGp+PeUtU1QnlQtk5KPFsoDscNlTQnOqzCLvpXQcnCO7dLK8ZW+3WkZZuP6P\nfvSj0fmytOGOHTviXC9evFjHPI+8Z7gt4vI8P1PdY3uv2xS/QZckSZJmxAd0SZIkaUaMuEiStGTb\nt2+vY76eZ8SAr9dZfaKU3+xquCZFRqY6OyaMyKR1UnWQni6UrZ59pG6jjBj0RHCm9s0xz8FUpY6e\n6EyKy/CaM3bRRjAY6UjdSlOsifcLIzFtbIMVV1JVFp5fxk/S+OrVq4N98PwyxsXlzp8/X8c7d+78\njeNpl2/PP6MsHKdj6rl3W6l60u7du+M6U/wGXZIkSZoRH9AlSZKkGTHiIknSkvGVOCthMMqSGtCU\nMmyyk5oNpc/ZKIav/9toSIp9UIqvLFr1ZWp/abs8V1PVYFJEJsUXUtOhVHllar7EyApjLYyoUBtj\n2rZtWx2z4gm3tXHjxtE5MZ7D426vK+e+YcOG0c8ZX0nXhvcV51fK8Lqx+RKXW1lZGd0f8f7m/qbm\nm34/UsMlaivTcLs8phQ/ux6/QZckSZJmxAd0SZIkaUaMuEiStGRHjx6t49R0iK/d2+YnfN3O+Avj\nC1yfkQi+5t+0adPo8qUMIwucF9dP0RJui+u2UR3OK+2b56enwU+rJ+KQYjDcLqMPUzEGxi3SeWDc\nhdc2ndt2Lrxu/DzFPlIFmvZ68Nh5bXgOGX1hXIpS3KWUUrZs2VLHly5dGh1zHtxHuv5tVCdFenjd\neuJOPAdtVZ377ruvjr/+9a/X8YkTJ0a3dT1+gy5JkiTNiA/okiRJ0owYcZEkacne+9731nGKWrBS\nS1tZgpED/ixFKlIUgVGJttoGX+8zQsBtpagHl2c0YKpJUop3pEgG58tzxXE7X0pRFErzbaM5vB6p\neg6xcs9Pf/rT0f2xaksppayuro4ux+O9du1aHbMSSop2tPNLlYN4TGwQxHuP55n7aJts8dgZReF9\nwv1xTlw+VZOZmnuKyKS4FMd33XXXYB/3339/Hf/rv/7r6LYW4TfokiRJ0oz4gC5JkiTNiBEXSZKW\njDEBNqpJVTh27tw5WJ9RBq7PV/t8PZ+qw1BbCYXrMGbAV/ipKU9PVKKUUjZv3jz6M1Y2YfQhxTN4\nDtisp5RhvIPHmCp0cLs8z6xecvvttw/20VZDWcN4B8fc95133lnHzz//fB2fPXt2sK0zZ87U8d69\ne+v4/Pnzo/vgOeE5SDGoUoYREM6R5709v2t4T6cmUqUM7w3e49x3ut+mqsNQajbEz3kvcO7cH++L\n06dPD7b1jW98o475+9let15+gy5JkiTNiA/okiRJ0oz4gC5JkiTNiBl0SZKWjPlelqdL3Tjb7DZL\nzLVdRtekjpjE7HRbMpH5W2aKU4dRZp65v9Sls90/f5ZK7qX8O7fTlkbkv1l6kPtgF0suz6w58/Lt\nuUrdSplhZzlEdtPctWtXHe/fv7+Od+zYMdgH988cOLPxxPOTynK254rb5X3J9a9cuVLHKafedt0k\n7p/bTeewp+Nni/cM98fflQsXLozuj+Utjx07VsdttpzbvdHSiuQ36JIkSdKM+IAuSZIkzYgRF0mS\nluyDH/xgHTMuwVf4fLXflo3bunVrHbPcXOqImF7HT3X2ZHyB82J8gbGEixcvju6Px9eWxksl/zhH\nxhJ4TIzUcH7tMTG+wqgOoyE8JsZgOL8UVyll2B2TUQiWQEylNVMJTM6jlOF54PnhXLitFF9KJQxL\nydeQuH5PKc52O1yO49SVlvchz0Eq5dnuk+vwGnC7XJ/ngHGX9r46cuTI6LZ4nyzCb9AlSZKkGfEB\nXZIkSZoRIy6SJC0ZK2QwysKIQ29lEsYzGCHp6fJJbdyBcZIUneGrfVbFYPSFUQ3GTUoZxgQYRWBE\ngbEUnhNWQuHyJ06cGOyD1VAYDUqRlfR56m5aSo7k8JxwuzyHjNrce++9dcyqLaUMj3d1dbWOWXkl\ndYlN8Zj2XkixGG43VeVJXV5bKcaVYjFTMayx5UsZnl8ee4oQMXJE3HdbCSlFcnjvL8Jv0CVJkqQZ\n8QFdkiRJmhEjLpIkzUhqxNLboCXFGnoiLmk8tX5aZt++fXXMaADHbMpTyjAiwwobjCz0VKBh85y2\nwQ9jG4yscLupMRJjJjw/bTWaFOngXHisrH6yadOm0XEbu+D+GeNhBIixi1Q9Zyp+wuVSrCXdoymu\n0t5H6Rqmz1MEjMvw3JYyjJkwmsL1GRn6+Mc/XsfPPvtsHT///PN1PFWNhvdJTyRnjN+gS5IkSTPi\nA7okSZI0I0ZcJEmaqRt5PZ7WSTGBFD+Z2neKMqR1eirTlDKsYLJ9+/aF9kGMNLBqRytV5eDnKeZB\nbWwjnUfugxEexk9Y0YfRF56Pdn1GZ1jFpeeaM0rSxl1SfGVRU1VYUvwlzb2NE63hNW/jJ+lnKQ7E\n+5LzYwOqtgER55siWYvwG3RJkiRpRnxAlyRJkmbEiIskSUvGaiKp8gZflU9VcUmv1FNcIcUd2jhH\nT0WZFINIFUCm4io9FT1SBZKpWMGiDZsYqeCY82jX7Wm+w20xUsEx4zltUydGLNK2UtSG52eqoRCP\ng9tNx8cIT4p5tNGQ1HiK22VEheN0L0xFjlJM6eLFi3V8/PjxOuZ551zbRkVpvu1yvfwGXZIkSZoR\nH9AlSZKkGTHiIknSkv3P//xPHbOKB5vTrKys1DEbobQ/Y+UPRmRSXCFFTtqYSIoQpOZGPXGXqRhN\njxR94XanIgapUgjxXKXGOL2Ro54KNJwvIy5nzpwZLJeaKaUIR4ovTVVnSdV6KB1TisS054r3OPfB\nZkOMgDFmwn0zVtJKVWD4u8IKLadOnarj22+/vY7ZAOvkyZODffBapftyEX6DLkmSJM2ID+iSJEnS\njBhxkSRpyc6dO1fHqZEPX9Mz3lBKrjTCxj9sYMMYDeMyXJ6v/0sZxmW4/1QFhK/2Gc/gdnfv3j3Y\nR4pn9MRoUsWSNpqR4iht06Sxz1PMYyomkprepOgMMe7SXo8UWUpVXFKFlRtpQMTjSNVaeirIlJKv\nJ+9j3q9cPsXBepsIMRbD83v48OHRbfF+bRtgra6uds2ll9+gS5IkSTPiA7okSZI0Iz6gS5IkSTNi\nBl2SpCVLed2pzp7ELC3L07EsHDOyPfnlNufODDrzuinDvGnTpjpmSTpmdz/2sY8N1tm6devoPnjs\nqXto0pa54/o9ZSH5eSpPOJXjTnl4ZqxTLr53Hymjn/bdW2Yxda9NpQOnupKmdXnvpnmlv3lIJTTb\nffd0HOXfaPBYWX6R9+SBAwfiPq9evVrH/L1ZhN+gS5IkSTPiA7okSZI0I0ZcJElaMpZiS2UWUxfL\nVk9HTL6O55hxA76mb7ebukTSc889V8eM3bAE3cMPPzxYZ/v27XW8Y8eO0THLQjJGw8hIOr5S8nlc\ntOPj1PIpOtMTM0kRjPY40j5StCRFpHrPAeMuqXNpT/nFqe2mDrWLHlO7fLof0rVJcZVDhw7FfTNu\nw1jLVDRtit+gS5IkSTPiA7okSZI0I0ZcJElasqkIwJrUpbOU3FEzVQ1JEYXeqiGpYyS3xa6kHDO6\n0L7+v3TpUh0zWnD06NE6ZkRmZWVldMwYzJYtWwb7YLWOFPth1CdFIqg9jnQeF43UTF2P9LNF95di\nKS0eY08n0p6Or+2/032V4jLpXmq7wk797lxvvinCxXu1lGE3YFZ7aash9fIbdEmSJGlGfECXJEmS\nZsSIiyRJS5ZiKT3VPUrJr9S5XTYISpVMUqWOdp/ptX+KSKSqL+1xcC6pCQ2P4/z586Pz5f4YiSll\nWBFmz549o5/zfDIaxO3eSHMi6mlO1FPJpN1f2la6l6aiNos2JOL1791HOg8pnpP2MVUthetzf+l3\nhdvi9efyvEdKGVYU4r3bE18b4zfokiRJ0oz4gC5JkiTNiBEXSZKWLEUGel7/t1LViKmIxJqpBkRp\nLmmZdEyskNJucyo2siZFBrhdYhOoUobVN44fP17HjCyw6sz+/fvr+MCBA3XM6Exv3CVdg55KJoz2\nlJKjGj3zoKno0lTDp7H1OWZzqql7j/vsafDEZXqaHJUyjKyk6BTXSY2GOF6/fv1gfR4v93HixInR\n/V2P36BLkiRJM+IDuiRJkjQjRlwkSVoyvhJnzISf9zTMKSVHPRhxYGUKbouRgTY2kapypMovPVGE\n9jhSjCY1U0rVUlKlj1KG5+fll18eXZ/XgDGYQ4cO1THjLvv27RuddynDSjOsFNMTGUoxoVL6ojPU\nUzWmjcGkiEtPlRLGRBjP4Tlvt8vzzns0zZe4j/Y4OJdFz0Oq3NNWjeFyPCes7rIIv0GXJEmSZsQH\ndEmSJGlGjLhIkrRkqcIGK4ukahKlDCuVpCgMX+2niAq1VUMYE+hpVJOqZUxJ0Zt0TD1VX9pzlWIx\n/JwxGEYXTp48Wcerq6t1vLKyMtjH1q1b63jbtm2jY0aIbqQ5FaWmPknPfTG1XNp3T9yF57OUvqoq\nqQoQK6ekObX7TMfRE32Z0tMMaRF+gy5JkiTNiA/okiRJ0oz4gC5JkiTNiBl0SZKWLJUqZEm6qbJ8\nKQeeujSmHHbK/bbbTeukcohpHlPHlObCeaSyfFPZ7Z5zwn2kXDWxlGIppZw7d66OmVu/ePFiHbND\nKbPp7FDK+bXZ7TSvlKtP17ynW+jUXHq2xZx5mw9nRrtnLvycvx8ct7l2/i3H5s2b65jdQNP9lkor\nTuXc07lehN+gS5IkSTPiA7okSZI0I0ZcJElasrYU4Bq+ak9RkvZnqdRhiiVw3amyeilykKIvqSzj\nVMSF22I5valjH5tfb3fMVJJw0VKHbdyE82Ws5aWXXqrjF198sY4ZcbnrrrvqeO/evXW8cePGwT5S\nFIp6Si4Su6aWMoyAsHRkkrqxpmVK6bseKTLCeEzqoNvOJV3bnnM1dS/weqTf20X4DbokSZI0Iz6g\nS5IkSTNixEWSpCXj6/n0Cj7FVUrJ1VPSuOd1/lRXybTv9Do/vfJvl5+qIpPmtSZVwmn1dMdMc+K+\nuY/2erALazomxl04ZofS7du31/GBAwcG+2AVGGLFkhR9StVSnn766cG27rnnnjpuu6WuSdcjXb+p\nqjo9n6fqNewq2l5/npNU7SfFXW6kCkuq/LIIv0GXJEmSZsQHdEmSJGlGjLhIkrRkfL3OeESKqEzh\nK3W+zuc4xUym9pea3lBPJIfbbRvKUJpLigzw81RRo/13anqUpGZG7bnithhfShEZfs6KLGxyxOvX\nrnP27Nk6vvPOO+uYlXQY82D1E859z549g30wYtODc1q0QkorNUbiMXEZnrc2BpPiSJwX78WpijBJ\nipBN3eNT/AZdkiRJmhEf0CVJkqQZMeIiSdKS8bV7T9WHnjhGKcPX66kqRqoa00ZDUrOgnqoo3Bbj\nA1PHkbY7Va1j7PPeRkWMFrVxkrG5p2og7c84l9S8idcpVSlpYxfXrl2r40OHDtUxq8Aw1sK4yq23\n3lrHjLWwMVEpOQ6S4lbpmvGYpuIuqZpR2i7P8y233FLHbeOmFOmhdH/3NOia+hnvq0X4DbokSZI0\nIz6gS5IkSTNixEWSpCVLVSNStZWpJkIpQpDiLr1NWXqauqSoDpfnPqZiAqkKTIrRcPmpijNp/+n4\nUqRiKp7DdRivSJGahMd09erVwc94z3BbXOfSpUt1zEovx44dq+Ndu3bV8ZNPPjnYx+c///k63rJl\ny+g+ehoSca4XL14cLMf7hLEdxnM2bNgwul2OeT3aiBL/vWjjId4jU1Gdnt+PRfgNuiRJkjQjPqBL\nkiRJM2LERZKkGUnNXqb0VH7h63lWuUiNX1KU5Eb2nWI07et//puxCEYqeirITMVoeiqQcB4p4pIa\nI7VS8yauw2PlfKeiIalJT6oaw+2eOXOmjp999tk6PnLkyGAfrPzykY98ZPRz7iM1++H55PKlDKMp\nGzduLGMWrSAzdV/1rJ+uZ4pqteuke2YRfoMuSZIkzYgP6JIkSdKMGHGRJGnJUrSjpzlQuz5fw6dG\nM1wmxSNYfaSVtpvGqRnSVBRhqhLL2DyIcY42qpOa4aRqNPTSSy+NrttGIhjjSE2TUoQnXZsrV64M\n9sGqLCmaxPPAqAX3R1u3bh38+7nnnqtjVpHZvXt3He/bt2/0c1Zh4floIx8pxtUTS0mRk/aap/V7\n7rc0j/aap2vbG1Nr+Q26JEmSNCM+oEuSJEkz4gO6JEmSNCNm0CVJWrKUIe7NoKcsbuqI2ZPpZf65\nXZ+dK9Mc0+cpY1/Kb5bgG5sX5566o3L5njKQ7XLcbsprU29X0VS6MpViTOewlNyFNV1/ji9fvlzH\nvSUQmYFnh1KWZlxZWanj/fv31/Edd9xRx9u2bYvHwXuuJ7udcufMv7c/S3/PkM5b6lw61Un0t8Fv\n0CVJkqQZ8QFdkiRJmhEjLpIkzUh6Vc7P2xKIvV00x5YhLt/OI8Uz+DkjEZxjmseUtL9Uzi51+ezd\ndzon1FMSsJVKV3Kd1D2UNmzYMPg3j5dzZ/yI27p27droMoycsIxkKaW88sorcf9j22X05fjx43X8\nwgsv1PHBgwcH6+/Zs6eOeX7S/ZOuQSr92f67Jw42FWUZW+b14DfokiRJ0oz4gC5JkiTNiBEXSZKW\njNUrUkWQVKWkXSctl5ZJ43Yeabs91WFSXKWVKqakzqA90Z72XDFykqrOpEhEGrfHl6qv9FSH4Xmf\n6uyazmnqAMsoCuMqXL6NuHCOKSrEqE2K3bDr6blz5wbrr1+/vo537txZx+xQyhgOK82kCiu9FVVS\nx9Ceajs3st1F+A26JEmSNCM+oEuSJEkzYsRFkqQl42twxgp6m+ywKkeqZsLtprhLipK06xO3xX1T\nqorRbrMncsBtpcodU42KUkSCx56aBXF/qerH1HyppwEO59TGT1LkJF1DHhMjLoxX8T5q/83t8hzy\n8xQZmqroc/HixTo+f/58HR8+fLiO2QBp69atdcxITIrBTEn3TDLVOIpupGrRb+zrNW9BkiRJ0m+N\nD+iSJEnSjBhxkSRppvgavTdSkWIGaczoQ6r6Ukqu0NETw2GUYKrBS08Mp6daS1q3lGGkg9vi8aWI\nBKMaKRrUStVF0jXjtnje2DSo/Vm65ozFcB6pGVIbcaF2/2PbTdGgqSgJrxX3waZHHJ8+fbqOjx07\nVsc7duyHeA3WAAAgAElEQVSo4+3btw/2sWXLltG58/N0X/U0SSolX8/emNpvbO+G1pIkSZL0uvAB\nXZIkSZoRIy6SJM1IagiUqnuUMowJpLhFirgwtpFiE6UMIwA9VUooNbxpq76k6iupQVCSKr20P0vV\nU7hvVjz57Gc/W8d33HFHHX/rW98a7OPJJ58cnVe6BinWQu31SLEWNiRixIXVT1Jcisfabosxk40b\nN9bxzTffPHocPO+p+VY7Fx5TipwwBsMxq8EcOnRosA9WeGEUhlEfzpHHlO6R3rhLb5Ou39jGDa0l\nSZIk6XXhA7okSZI0I0ZcJElastTgh1I1kPbfPc1+GDlhxKVnHq3UWCk1EeK4rRrCuVCK/fTMqY2M\npGobqRETl/nJT35Sx4yPXL16Nc4lVUxJcY4U5+k57lLyte3ZXxvH4P2QGiCl40vns8Vtpf3xPuFx\n8NpyHm3FGVZ+4T727t1bx6zosmvXrjpm3IXVfXp/V1Jk6Xr8Bl2SJEmaER/QJUmSpBnxAV2SJEma\nETPokiQtGfOzPWUL25J7KUfc010z5dfbeSzaEZE54FTWsdXTnTN1NE3bmTqfqbxh+vyHP/xhHT/2\n2GN13J6bdB5Tyb2ea84sdCnD65Zy58TrwXM4dW14XFw//W1DOtb0dwrttvh70NM9ltLfPJQyPHc8\njhdeeGF0maNHj9Yxu5KuX7++jjdv3jzYB5djVj11pb0ev0GXJEmSZsQHdEmSJGlGjLhIkrRkLCOX\nXu1PxSBS10aOX0sn0FKG0YJF4y7cRyqTV8owCsF1GMlI5QJTbKctT5iON5UF5PqMK/DctueD56o3\nQjQmla1scVuMifSUSey9lps2bRrdbk+8KnU9bfVElpK2ZCcxupPujStXrtTxhQsX6nh1dbWO2UG1\nja6wWynHU/Oa4jfokiRJ0oz4gC5JkiTNiBEXSZKWLEVZ+Gp/6jV/T9fGFJHg/hgFYFWK9md8bX/p\n0qU6TlVjUiwlVeRol0tuJFKxaNdVjlNHy6mKN4yDpP1xfX7OubbnKsVo0nlP53oq7pK6vnJb6b7q\njfb0xH64rZ5l2uhUusdTBIiVXrjdl19+uY4ZgyllWPmFcSDGYhbhN+iSJEnSjPiALkmSJM2IERdJ\nkmYkxV342p6v5kvJsYYU9WCTla1bt9bxF77whTr+/d///cE++Kqer/MPHz5cx//xH/9Rx88999zo\n/KZiLSkukarOMH7A6iWMY7TnKp3fnn23FWHWMPrQSpVYUkSGcSWeq/a8cVupOVFquJQqmbT74BxT\nxClVIErH157DdE57mla1UZY17bVMx8t98D5J0RfeY23jKG6L8Rf+rizCb9AlSZKkGfEBXZIkSZoR\nIy6SJC0ZX8n3NGVJsYBShnGCO++8s44feOCBOv7EJz5Rx2y4cu+999YxYzCllHLx4sU6/s53vlPH\nTzzxRB0fOXKkjhn74PExltBWDempZpIaGHH5VOGkXS5Va+F8U3WQqaohKXpBKWqR5sd4RSnDKAzj\nFpw7zw/PW2qe00ZJuM+e5kbpOk1dj55tUYoATZ3znthY+jxVSJqK0fB36urVq3FeU/wGXZIkSZoR\nH9AlSZKkGTHiIknSkvVEIm7Ee9/73jp+8MEH6/i73/1uHTPuwBjLmTNnBtt67LHH6vjHP/5xHbfR\nizWMH6QYQxvn6amqwnPFfafqNWl+Y/tfw0Y1aZmpCiuMO6TGOKkCTWpy1M6D143zTdGiNL8Ur2mX\nSw20OPee5kTtPtK57rkXaCoaxnmlqjM946lYE5djxGX9+vXXnfsYv0GXJEmSZsQHdEmSJGlG1r1e\nr9VGd7Zu3Ru3M0lL8eqrr667/lKSaO/evfX/j4w19EZDGKNgHCCNadEKMi3uu6chTK907D3z4DG1\nERfOi8ulGEZqEJWqxpQyrPyRoh6pskk6jqmGS6mpD5ehNKdWmi+PL8U+UtWXtipKumcoNdxiPIb7\nbu/1VO0lNVDqqfTSNodK54GxsUcffbT7/49+gy5JkiTNiA/okiRJ0oxYxUWSpCXraYwyha/X0/op\nRsEISIootOvQtWvX6jjFLnqbCLHhToo1cB1ui9U5OG7jDvx3ioP0NENKkYh27otKx9dGktP56Yks\npeNur0faf0/0Kt1X7blKlVsYX+mJ7VBvjCbtu6c5URud4jmZaiTWy2/QJUmSpBnxAV2SJEmaER/Q\nJUmSpBkxgy5J0pKlPDOztzfS8ZGfp1J3KTvbZn25DnPZqSRdT5nF9nPmjqktMbiGmeBU+q83D9xb\nevB665aSjzfluNM1m5pTyvKna54y1lP5bp7fdG25j1S6e6rrasrSp78JSMeUOsy2eq5N799M0KI5\n+evxG3RJkiRpRnxAlyRJkmbEiIskSUvWE8PojbikEnh8BZ9KDU6VB2Tpwpdffnn0c66fyhOmiEL7\nb243RRlSt8rUNbWUvggIl+E80lynOqWm+EmKcKRyllP774n6pIhTitRM4TlJcac01/ZcpfhLTxfU\npN1HzzGmUp69JTM5X/5+3EjZ1FL8Bl2SJEmaFR/QJUmSpBkx4iJJ0pKlCEd6Hd++/ucreVZCYWfO\nnk6J/Lzdd0+FFkYUOA/GNqYiLqkjZppXqvrBddu4Szr2FMlhnCPFINpIRZp7us492uV7Op8SK5uk\nOM9U/GTRijC8/um6tv9OFVNSd1TOY6pyS8JjStVoUuyqxXuc871w4cLC8yrFb9AlSZKkWfEBXZIk\nSZqRdYu+YpEkSZL0+vEbdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmS\nZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECX\nJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlG\nfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmS\nJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQHdEmSJGlGfECXJEmSZsQH\ndEmSJGlG3vEG7+/VN3h/kt5465Y9AenN5qGHHqr/f1y37te/Qu94xztGx29/+9sH63Mdjt/2tvHv\n4c6fP1/Hp06dquMnn3yyjk+cODFY5xe/+EUdb9y4cXS7/Pxd73pXHd988811fNNNN42uW0opt9xy\nSx3/6le/quPTp0+Pfs79bdmypY7vuuuuOv7IRz4y2MehQ4fq+OLFi3XM4+N5u3LlSh3v3r179Dh4\nbVqvvvrrRx9ul9epvZ5reKztMtwul0vr//KXvxxdl/Not/Pzn/+8jt/5zneOrpPm8corr9TxpUuX\nRudRyvD8XrhwoY5ffvnlOt67d28Zw3lwO+19z3OXzgPHvBd4HNeuXRudX/tvzuuZZ57huPv/j36D\nLkmSJM3IG/0NuiRJugH8drL9hpDf2CX8FpHfFvIbxV27dtUxvy0uZfgNKn929uzZ0f2trKzU8Usv\nvVTH69evH51Ha3V1tY7T8fFzfqPNz/nNeinDbzR5HvktOM8Jx1zmRr5BT2860rVJ37iXMvyWN32D\nntZP35q339LzuNI1SN88p7c37Tfoae4J90GcK7/5b//Nb9r5jTjndfXq1dHP07iU4e8Hx2m+1+M3\n6JIkSdKM+IAuSZIkzYgRF0mSloyv4FPcIX1eSl9cYvPmzXXMV/D0s5/9rI7biMttt91Wx4yvcH/8\nQznuo40crGlf/6c4CLfF/XEeXIZ//Pm1r31tsA/+0d+2bdvq+PLly3W8adOm0Xn0RD7an6VjIsZB\n+IeynF/7h5LHjx+v48OHD9dxir6kWNPUH5um40jrpH2ncbuthLEUzp3XjOej3Qf/mJjbSr83KZKV\nYlDtttqf3Qi/QZckSZJmxAd0SZIkaUaMuEiStGSMBqQqHqlaRinD1/P8GV/Jv+9976vj733ve6Pb\nZUyEr/lLKeWFF14YXY7xFY5TtIMRk7bSB2unp0osjOowfpDiHIy7tHPnmLXaed4Y+/nJT34yOr9W\nivTweDlf1m3fv39/HTPuwoojpZTygx/8oI5/+MMfXnce3DfPW4pwlDKsVMPleL/x/PDa8nNGn7id\n1p49e+qY15n3xVNPPTU6D15nrlvK8Nh5nTnHnko46Xez/Rn3sWHDhtHtXo/foEuSJEkz4gO6JEmS\nNCM+oEuSJEkzYgZdkqQlu3TpUh2nMonMB7d5Wf6bJeWY3WVnzlS2jpnsNmPNuaTsLufI42AWeuo4\niMtxzHkwl506V7Zl/Ngl8ty5c6Pb4jnkHJ944ok6Zt66zeufOXPmunPnOXzggQfqmNeJmAcvpZSj\nR4/WMTPezK3zWHkcO3bsqONbb721jttrziz1wYMH65jXlpnw//qv/6pjXvP77ruvjj/3uc8N9nHH\nHXfUMbP4zG7zXJ86dWp0Hzt37qxjlskspZT777+/jtPvFK8/fz+ee+650WV4j5QyzNbznHJbi/Ab\ndEmSJGlGfECXJEmSZsSIiyRJS8aoBaVumm0ZP8YlGEtgubmHH364jhftblnKMNaQStUxUpPK0zFS\n0x4HoxqMNXA57i/Nl3PluJRhNIE/O3HiRB2zi2qK7aQOoy1GSzhHlsM8cuRIHe/atauOn3/++Xgc\nqZQjpdhPOj/t8rznGJ3hOoze9HQPbffx7ne/u4737dtXx4cOHapjXnPGTHgOOL/Pfvazg32wxOid\nd95Zx4zRsJQnI0tf+cpX6phxnjZyxN8JRmwYr1mE36BLkiRJM+IDuiRJkjQjRlwkSVoydoVkdGJl\nZaWO+UqdcYxShlVgGOHga3dGKj784Q/XMV/tM4rSVj/hvFK8grEU7o9jbqfdR+pqyohE6kTJZbiP\nFPMoZXh+WUmF66SoDs/BVFUd4rZ4rli5hd00ec1ZkaWUYRyI43TeON/UubbFOFI6v5S6yqa4SynD\nyAojWTxXjJZcuHChjnkcjFedP39+sI8vf/nLdfzJT36yjvm78ld/9Vejn7OyzOHDh+u4jbgwpsZr\nwN+vRfgNuiRJkjQjPqBLkiRJM2LERZKkJWOsYaoCyZrbbrtt8O/ULIgxgfaV/BrGY1I1mFJyFGKq\n2dCadEztnBipYIOXFNvg8hxPVVjhdl988cXR7bLRTGqyxLlzfqWUsm3btjpOjXF4bfg5Ixy8zpxr\ni+eUc0mVf1LMqL2W6Wece4q7pBhNu49HH320jp955pk65nlI9+7ly5dH9/GpT31qsNynP/3pOmas\niRg54vE9+OCDdcwGVPy9KWXYjIlxpLaJVS+/QZckSZJmxAd0SZIkaUaMuEiStGSsGpHiGVPVSIhx\nh9SQKMVVWAmjlSpxcFs90Qeuywoe7TqcS6omkhoHTWHFE0Z6GMlgxIXz+OhHPzr6ebvvFOngPtI5\n5LGy+U4bqeC5S9ui1PiJ2vgJj4vnLcWU2JSH0Y6dO3eOrltKKcePH69jVuj5xCc+Ucc8Vm736NGj\ndXzx4sU6Pnbs2GAf73nPe+p469atdczz8O1vf7uOGYNhw6Tvf//7JeG54vHyvC/Cb9AlSZKkGfEB\nXZIkSZqRdVMF6l8Hb+jOJC1F33tmSdUXv/jF+v9HvipnHCA1z7nez8ak5XtiMC0ul+ablm+jIWl9\nRhEWje20nzPWkiIujGowcsJGOhy3VVx6Gh2luXMe3C5jUKUMq5xQavZEPO50v7VzSRVzWLGEERlW\ny2G1lbZxD5s0MVp08ODB0fUff/zxOmaDLzYK4vza4+Axbtq0qY4Zr0kRoA0bNtTx2bNnBz/j9WA0\njdGgF154ofv/j36DLkmSJM2ID+iSJEnSjFjFRZKkJVu0GklvxZIkVWShNqLS06gmRTt6li9lGC1g\nTCFFQ7h+Oo62Mg2jKdwWm0UxGsLoBE01oEmxH1Y8SdV22njGGsYrShlGLNK54jlJkZqpyBGryHAf\nvE7pOHjee6sRcbus3PLII4/UMaui3H333XXMpk6MzZRSyunTp+uYcZmTJ0/WMaM3d955Zx2nxkZs\nRtXOnZGa8+fPj65/PX6DLkmSJM2ID+iSJEnSjBhxkSRpyRgfSBEAVt5oq0wwtkGMLKRYTE/llfZn\n3BbjICnu0BOpaddJ1UVSVCPFXdpGMTyPN998cx0zMsKIAvfBihxTx5TiQKwUwtgHryeX5/x27do1\n2Aeb8aQGT7w23B/vN56PtuLNolEmzjc1l2r3weVOnDgx+jnnzuoujOCkhlClDCvKsOETK9Bwfz1N\nnRhXav/NbbVz6eU36JIkSdKM+IAuSZIkzYgP6JIkSdKMmEGXJGnJUqdDOnXqVB23udhU0o7bYh45\nZYi5/FSumlLWOEn7a3+WpDxzysX3Zt6nyiYuivvk+WEWOl1nLs/jYBnIUnJem+eQ90L6e4TejrHc\nB9dn7pzHxHua4za7zRw572PmwFlCkcfH85k6z7b757VJXWV7/v6hLYeZrnnv+W35DbokSZI0Iz6g\nS5IkSTNixEWSpCVLZf34en3nzp113L5eZ8yAMQG+9k8dKlM5xDZuwrjFVPfJ6+1jKl7R062UuAzn\nx4jD1PxS/CWVb0wRnPbznvKWU6UH16RyiC123WRHzJ6IC89VOw/eV+k4uD5jQpwv7++pDrX79u0b\nXf/JJ58c3R/nt3v37tH5lTLsBpvuMUZv+HlbpnNsmXa7PN5UpvF6/AZdkiRJmhEf0CVJkqQZMeIi\nSdKS7dixY/RzvsJPHS2nlkuv87nMjVTxSOuk+ErqQjkVP0lzTFGNVEGkJ4LT6qlYk87z1PrU042T\nsZ22+sn27dvrmMe4YcOGOk5dLLmPbdu2xTly/z3VehjnSGN272wxpsLYDiMqPG7Ob2q7PR1HGalJ\nFWt6qxwxWtR2gO3lN+iSJEnSjPiALkmSJM2IERdJkpaMr84vXbpUx3zNz2XaJjepYUuqCMPX8evX\nr69jRgbaaMiisY2euMuUnohLiv1MVaNJEZkUOelpQNOem57jZWSFkRPGMYjVeUoZRlNYaYTb2rhx\n4+icUlWVNrqUojf8PFWX4TnhfcX7rZ376urq6DqsTJNiO1P3LufL36l0HhhR4XFMVd5hjIf3YhtN\n6uU36JIkSdKM+IAuSZIkzYgRF0mSluzo0aN1zCgDx3zt3jY/4av69Eo9NV/ha/tNmzbVMSMDpQwj\nMtwHYx8cp0hOOqZSclyCcQKun2IwqfFTKcNzx2PnHFNDIm6Xx9rGT4jnkftgZITnk+cgxWvaf2/e\nvHn0c+6P55rHwX209w6jIrxneA4ZyWF8JTV74nGXMjw/jGpdvHhxdO481ynW0v5+8JxyOR5vij6l\npkPtubrvvvvq+Otf/3odX758udwIv0GXJEmSZsQHdEmSJGlGjLhIkrRk733ve+u4jX2sYfWJ9hU+\n/50iDj2VQhhx4biUYUyAr/25LcYPUtUXRgOmqob0VM/g8tzuSy+9VMdtA5s2YrEmRVEoNWhi/KOU\n4TXgdtPcOd+nnnpqdPm2oRArnnC5dOy8nina0cZoUkwpzT3dF1y3vXdZtShFYXqq4kxVGeq5nilO\nlCIuBw4cGGzr/vvvr+N/+Zd/6ZrXFL9BlyRJkmbEB3RJkiRpRoy4SJK0ZGxswkoYqQrH9u3bB+sz\n/sJoQarcwfhAitS0cQ5uN1VM6WkCNNVwiQ1p+DPGa9rqMmNzYoMenttShvEOzp0xldTUieeZkZE2\n7sA5puo53BbneNddd9Xxs88+W8dnz54d7IP/3rNnTx2fO3du9DgYP+K14fVvr0c611w/VdVh9IXH\n18ZoUoOgFL3hvcDP073Xbpd4bRlfYTSIy/BYT58+PdjWN77xjTretWtXHbfXrZffoEuSJEkz4gO6\nJEmSNCM+oEuSJEkzYgZdb1n/8A//UMfM/f3N3/zNMqYjSRHzrxs2bKhjZoKZ722zwexWyP/epbJ+\nqYRhyryXMsygs+MjMdvO4+jJqZeSM8/MjfM42jzz2NzbfXAuKysro+swN8zlDx48WMcse8hjbefF\n/TN3zpwzj4n55VtvvbWO27874L95bXr+VqAtPTm2fCml7Ny5s46ZuWcOnB0/ee+lnHvbpTX9rQJ/\nJ9L9k/5WoL3mqUwn53v+/PnR5bdu3VrHx48fLwmveSrHuQi/QZckSZJmxAd0SZIkaUaMuOgt6+jR\no3XMV3CSNDcPPPBAHadyeOk1fSm/GX8YW44xgbStqVfzjGcwisBYCqMPHHN/jMHwWNv9s9wkMZbA\nGAS3y0hM27kydb7kXDjmuWVUg/NoO4lyH4xO8P9FPJ9p7jwfLEHZzp0YX2nP79g+GM1ozxWjPoyT\npJgS98flOdd23jxGzovXP5XD7LknSxkeF9fnueLnvM7sdMpYU/u7cuTIkevOcRF+gy5JkiTNiA/o\nkiRJ0owYcdFb1t///d/XcXoVKElzwIgDX9WnMSMjpeRKGIwMpGU4TpGaUoZRCP4svc5nNIBxF1ZO\naavBMFKRuk9yf5w7IyD8/NSpU4N97NixY3Qu/P8EIyoXLlwYPQ5GHNooSao0k7bFCMbu3bvr+O67\n767jzZs3D/bB42WMhsfEOfKa8bzzPLMCTCnDqEjqUMt7hvtrK6kkqRJPW+1lTIratNvkv3mdGcPh\neWvPw5oUGSslx4na2FAvv0GXJEmSZsQHdEmSJGlGjLjoLYuvjCVpznoam6QoSrt+am6U1ufnqVJH\nuxzjEhs3bhxdhtq4TPr89ttvr+NUbSU1WeKY8RFus5ThcTEawtgHYwmMc/D/K6k5ULsOz+n+/ftH\n55FiSWwO1P4/jeufPHmyjlmZhBGVtF3Or61+kqrLcJ00p2TqXu9ppsVxut/aqjqM8fDe4HXiffzx\nj3+8jp999tk6fv755+u4jdFwLowj3WjTIr9BlyRJkmbEB3RJkiRpRoy4qJRSymOPPVbHDz744BJn\nIklakyIDUxGXJC3T26iIr/R7K3SsYZQgVZAppZQNGzbUMZvC9JiK5xCPg3GQ1dXVOmZDIW6X0ZCp\niAujIilyxHmkJjsc79y5c7APRoBYmYbxihQHYeSD1Xba6ApjHz0NtIixpKl7h+tz/2mdNoazhtGV\nNn7CyBL3x3PICA/3weUvX75cx20DIl5b7i9Vqbkev0GXJEmSZsQHdEmSJGlGjLj8DvvmN79Zx1/9\n6lfr+N///d+XMR1J+p3FCiLr16+vY752n2oCkxqoMFqS8BU+X8e3cYeeKjCpccxUrIVS3KYnhpMi\nEe3+0vFSOo4Ug+DnpQyrn6SmPqlSDOfLaiSMopQyjFhwLtx3itGkObXHwYgLt8t7I1XP4ZhRkrZx\nD+9X3vucI9dJDZ5o6h5LUSieX1bF4e8mr8dUA8TUGGkRfoMuSZIkzYgP6JIkSdKMGHH5Hca/kP/R\nj35Ux//0T/9Ux3/2Z3/2hs7prYLxoVJK+cxnPrOkmUh6M3j44YfrOFUHYXUOVhMppZSVlZU6ZrUN\njvlqP0UqpqIkKcqSqpSkmElq4jM1F+qtbJP0RG8YGempQNNGKlK1Fu47xU/4OeMcZ8+eHeyD15zR\nkFQVJTUBSsfdbjfFpVIVl97rwWeR1PQqxXnSeW7vK/47NVlilOXMmTN1fNddd9UxozonTpwYrM/4\nC6+bjYokSZKktwAf0CVJkqQZMeLyO4yveY4cOVLHjzzySB0bcbkxDz300ODfRlwkTWF8ITXJmaq2\nkZrbMDLAMaMzqfkOq3a0/+b+WaUiRWQYGeB2du/ePdhHih+kSE2KcKR123/zXLFJEmMbXCbFNqZi\nDGmdqUoja3h8nF8pw/gSce4ccx6cL+fRGzlK0RJWVWEzJI7bBj9cn/dGinrx3mMEh5EfNqAqpZQt\nW7bUMaMoHHMfhw8fHp3vvn376ritIHP69OnRObbH28tv0CVJkqQZ8QFdkiRJmhEf0CVJkqQZMYP+\nO+yBBx6o4/vvv7+O2/JdWtwTTzwx+PfRo0freP/+/W/0dCS9iaSsb8pnlzLMwzJXy9JxKbudctxt\nxpn5YP4slV/k/0vYmZHz+4M/+IPBPpgVTp0reU5SucAbKbnI3HAqBck8evr7gHYuScp+pzz71DbT\nz9LnqXRk73HwvKf7NXX8bI+P+fS0XDrv/PuHlLEvpe940985pC6/Bw8eHOyD+2Q5xvS3AtfjN+iS\nJEnSjPiALkmSJM2IEReVUoYd6u69994lzuSt4dKlS4N/P/XUU3VsxEVSi2XheuIVbZnF1MEz6el0\n2Zaq6+0GOrZ+Kmf3ne98Z7DO9u3b63jnzp2jn/P/VyytxzndSPfGni6o6Vjb/fV0Qe0pv8hlUsfO\nViqtmO6lFBlqpfskxaV4PdL5LGUYU+kpC8noTDqm9jh6us/23MeHDh2q4/Y4OK8UqVmE36BLkiRJ\nM+IDuiRJkjQjRlxUShlWdPnTP/3TJc7kraHtHPbP//zPdfxHf/RHb/R0JM1cig8QX9tz+XadVNkk\nVSnp7cCZYjGUOl9ynCIRpQy7qDJawG7XrKTBqi8cM/rCcTsXnhNWEOE5SRVLUsfOdrkU1eDn6XxO\n6YnRpLmnZabuK+K56tnHVMUb3ovcP6MvlO7vFMFq12l/tiZVikndVC9cuDBYf3V1tY55j7ZxtF5+\ngy5JkiTNiA/okiRJ0owYcVEppZS//Mu/XPYU3vRefPHFOmb1gVKGr7skqdVTdSRVuChl2NSHr9S5\nXcYSehr/TFWDSa/9U+OYVNFjah+cL7EiDGMGKTrDqjGllLJjx4463rNnTx1v3bq1jjdu3FjHqXnT\nVAWQRauG9DZWShgNSfdSitFMVW5JP0v3FZsOsaoJx+2xMlqS4ifpvkyfT1VOmYpYjc2Jv08c83eu\nlOE9w3PCY1+E36BLkiRJM+IDuiRJkjQjRlxUSinlAx/4wLKn8Kb36KOP1vHjjz8++Nm3v/3tN3o6\nkkzokfQAACAASURBVN5EUqWPG2m4w9fwabtJiqL0SrENxiumYjQ9+0yxixQlaKtqsZHc8ePH65jn\njXGFffv21fGBAwfq+LVGF9N1TsfHaE8pw3OVKoX07GPqvkhxkp71eT14DaaaCPVUgUnHNBU5YjSJ\nMZw0D57PFIlpIy6sDsR9HDt2bHR/1+M36JIkSdKM+IAuSZIkzYgRF+m35J577qnjW2+9dfCzL37x\ni3V8+PDhOr799ttf/4lJmj1WP2EFilSdo61SkapqJKxSwW1xHlMNVlJzm9REJjViaiMtqfpGis70\nNP5pIxWp6gj3ff78+To+ceJEHbNh0h133FHHjMG0+2SlmV27dtVxahaVqpFMVVtJFVrS+Un7a6V9\n9jQU4j3GMRtQlTK8BoyicB8p4nIj9xW3le7XFMNK0ZcWr+2mTZviclP8Bl2SJEmaER/QJUmSpBkx\n4iK9BmxO9MADD3Sts3fv3tdpNpLerFIlDFaKSK/pSxnGBNK2UhOYFFFoq6IwfkApcpC2O4VRiJ6G\nND1VX9pzlaJCKSbE83by5Mk6Xl1dreOnn356sI+VlZU63r59ex2zSVKKuCRT1VZSVZW0Tm+FnnSu\nU0QmzYMVb6aiIbz+xPuK1Wx4j7IhVXs+0xxpKkLUI8Vw0u/d9fgNuiRJkjQjPqBLkiRJM+IDuiRJ\nkjQjZtCl1+DOO+9ceJ2UsZP0u4uZVWa32YFxKqfM/GzKvKbSgykT3GZyU0k7zitlk1OZxKljSmXv\n0tz539apbqVcp2cfqQNnKqVYSilnz56t41OnTtXx5cuX65jleHfu3FnH7Eg59XcHqbRi0pNNn8ph\n8zzyHlv07wPaffO4uN103vn3ATyfqZNsO5fUDbSnzOJUdr8nr78Iv0GXJEmSZsQHdEmSJGlGjLhI\nkrRk6dV56nTYxh1S186efaRX8+3yKSKR9sf4QJpr+/o/RRm4XIrwTMVaKEV90jo90Z72HPBnly5d\nquOnnnqqjlmml6UY77777jpmWV5GX0oZ3gMss/laHD9+fPBvXsOtW7fWcYrIpPhIbxfcNGbsi5+z\n9OeNxEd7Yk1p+VaKihlxkSRJkt4CfECXJEmSZsSIiyRJS8ZX+Knr4VSlllQ9hXqqavR2mKRUFYVS\n5ZU2FtBT/SLFDxgzSdVAeveRoj7cVqo+UsrweqZ9vPTSS3V89erVOj5z5kwds/NoWzVs3759o9vt\n6fiaKqS0HVHvueeeOt6yZcvoOpTOVW83zakY19h2OWbEZapSDOfOeybFtnp/P9JyU51Tp/gNuiRJ\nkjQjPqBLkiRJM2LERZKkJUuv1HsiI6UMIwSpeQ+jDynm0duUpafKBauwcB7c7lSznXQe0jlJ56A9\nVyniwPV7oj5ct51TanqTIjLE5lQnT56My3P/586dq+M77rijjnnN169fX8c33XRTHfNY9+zZM9gH\nq8tQT1wqRT56m1Olyj09DbCm7qu0/1QpZqpyS8JtcbwIv0GXJEmSZsQHdEmSJGlGjLhIkrRkfIWf\nKpDwFX5vVYyeyEl6hZ9iLO3+eyIy3Adf+U8dR4o78DhS1ILLt8eR4g/p/KRlpuaezg+b96RIBvfB\nuf/sZz8b7IPNidj06PTp03V8yy231DEbDd166611zGZIXL6UXBklRVZSY63U7KnFY+y5x1L0pb02\n3H+KnKT7raeRUvvvnv1dj9+gS5IkSTPiA7okSZI0I0ZcJElaMsYaqKdZSym5QUyKc/RUyGj3wfgC\nowipWUza1lRMIM23p9lLqsLR7iNFMnoq6fBcpRhMKcNjZyUVbjddc+IybGbU/ozXg+fh0qVLdby6\nulrHx44dq+Ndu3bV8Y9//OPBPj7/+c/X8crKSh33nCueA8ZxLly4MNgHry2XY9xm06ZNo9tN1Vam\nYjQ9zal6mjq10ZXe+6+X36BLkiRJM+IDuiRJkjQjRlwkSVqynsYv1EZBUkwlVQpJEZWphkApZjLV\neGZsH72VMFK1lp6KHqmCyNg+x7bFKAOPmw1+0v5aqaIH1+expooujKuUMoypMBrC+TJew22dOXOm\njp999tk6PnLkyGAfbFT04Q9/uI537tx53X1wzPPDhkmlDM/D5s2b67inWVRqAtWu29N4KDVA6o2r\n8DhSNZtF+A26JEmSNCM+oEuSJEkzYsRF0mv2j//4j3X8F3/xF0ucifTmlBoKpdfubSOW9Bo+xV0Y\ntWCkIsU52vVTMyVut6fxS1sJozeyMDaPtJ02mpPmyM9ThZa2kkpanv9OcQle53Q9GF25fPnyYB9n\nz56t4xT74bY4p9Q8h82MShnGX1566aU63rNnTx2z6dHu3bvrmE2ZGINpz9WiFU9SfGQqxtKz3Z55\n8Ny2x5GiTFZxkSRJkt4CfECXJEmSZsQHdEmSJGlGzKBLes3+/M//fNlTkN7UUl67p3xiKX0dFVOp\nwoT551KGWWqWaeTn3C4/5ziV3ytlmFtO5fCIWd+UX28z6D3lFHvKBaZttssxm9zTrZLHxDm15yOd\nn9QpleMrV66Mfs6seCnD+4/rPP3003XM0owsk7h///46vuOOO+p427Ztg33w/LzyyiujnyfpWNvr\n1FNmcdEs/FSn3d8Gv0GXJEmSZsQHdEmSJGlGjLhIes1SSTJJi+Or8vSav/2dS2UIuX4qxUeMrrSv\n7FO8I0VZ2rjEoniMKQLCObIs5FQ05LVIcZW2bGGKE/WUp0wlEDds2DD4N88P98GYCOM5165dG12G\nkROWUmyX27hx4+i8WHry4sWLdXz8+PE6fuGFF+r44MGDg/VZspHXkOMU1bnREoZjejqX3si2bnS7\nfoMuSZIkzYgP6JIkSdKMGHGRJGnJGC1J3S1T18T2Z6lzZVomxTHaTqKUKmakOaZlWoyQpDGlKMFU\nBZCeKjmp6keKuLRS7IfrM35CKbrSXg/OfaoSyxpGXBiX4fJtp9Se6A3nxbnz/LDr6blz5wbrr1+/\nvo537txZx/v27avj7du3j+6vt5MopQo7aRn6bVdqmeI36JIkSdKM+IAuSZIkzYgRF0mSloyv2lPz\nHGpftadX9Vw/RRRSVKLdR6qYwThAquiU4ie9zXdSDCdVeknxmlKGkY7UtCg1J+K5onYfab6cV091\nD56rtsJKqtDDuad4Dc8bG1Kxaku7La7Pc8jPU2SIsZT2XmXllwsXLtQxGyCtrKzU8ZYtW+p4165d\ndcxqNDdSQWjRxkjpXijltxOF8Rt0SZIkaUZ8QJckSZJmxIiLJEkzlSIubdyEy6WGRHwln6Iv3G67\n7xRf4XZ7ql9MNRHiv9N2UyylZ91ShpEObouxiBSRSBVLpo4jxXv4eYqi8Hq08RP+LF1bxmJ6Ykas\nJtTuo/3ZGp5fntveSig8jzxGRl84PnXqVB2zGdKOHTtGx6WUsnnz5tF58XOeQ/5OpHPV/g6mWFP6\nHb4ev0GXJEmSZsQHdEmSJGlGjLhIkrRkKe5AqTlQ+7NUQSJV2Lj55ptHl596NZ8aHSWMCTA+0MZm\nUvWVVKGlZ9222kZPcyNGWdjU54//+I/r+ODBg3X8rW99a7CPH/7wh6PzSlEknpN0/dpzle4ZNiRi\nxIXVT1IDrI0bNw72wcZFly5dqmOeEzYa4nZ53nsrnqR7NDV4OnPmTB2zAsyLL7442AcbHTH+snv3\n7tF983ciNbBq78OpykE3wm/QJUmSpBnxAV2SJEmaESMukiQtWYoApLhD21SFr+EZAejZLiuWpEot\nLb7OT/tL+07rTu0/VW5JcReeH1YWKWUYB0nVbLg/rv/jH/+4jhkluXLlymAfnFeKr6RoENflcUxd\n856oToolpcoy7fqpQguru6TrNNUEKMWRUnWZFDPheWY0p5RSVldX65j32J49e+qYjY4Yg2GEJ1Xx\naY+j5x69Hr9BlyRJkmbEB3RJkiRpRnxAlyRJkmbEDLokSUvG/GyPtgQiM778Wcq/MuubMr1tHnnR\n0orMDXNd5nhbzPGmjpo9OflUrq+VMt4pP/3EE0/U8WOPPRbnNHUex6RSflPlMDnH9HcAXIfXI3Va\nba8Nj6vN8o+tTynn3i6fupWmkpTt3y2MLd/+7UY6D4cOHarjkydP1vGmTZvqmCUauZ2VlZXBPrgc\nz2MqY3o9foMuSZIkzYgP6JIkSdKMGHGRJGnJUqm6FB9oP09dJVOUoSeu0sZjGFNIMZOeknJTXRZT\nqUPuL0UfUkykjQ9x/RRxSR0xGVdI82j3n0oMpk6paa69HVFTXIqf95zDdu6bN28enQvjJCle1XNP\nl9IXX0qlR1P0ZWq7PL7Lly/X8fnz5+v49OnTdczoC8uTllLK1q1b65glG/m7vQi/QZckSZJmxAd0\nSZIkaUaMuEiSNCMpDpC6NLbSK/We+ACXYVWKUoav9LkPRgNStQ5ud+o4eo49jRmpmKpMwn1yOUZA\n+DnHrGSSYjftOilSk7qKpmvbRle4XIqZ9ERLpqI6/BkjJDyOVBEodSid6iqa9FTFSdWISslRIUZW\n2DGU15nbYvdYxmBKKeXIkSOj2924cePofK/Hb9AlSZKkGfEBXZIkSZoRIy6SJC1Zim0kbUUP/puv\n89OYr+BZfeILX/hCHX/sYx8b7GPDhg11fPTo0To+fPhwHX/zm9+s4+eee25036kBUbscpXhGanLD\nz6caI/Fct+f0evsmRh+m1umppMPzw1hJe254XIwfpegMzwmX6Y2GMNLDc/3KK6+MLp+Or43wpAo2\naZ10rrhMey17mnHx+DhmtGiqoRivLeMv/F1ZhN+gS5IkSTPiA7okSZI0I0ZcJElasqlKGmOmqrjw\ntf2BAwfq+IMf/GAdf+ITn6hjxiPuvffeOmZjmlJKuXjxYh0//PDDdfyDH/ygjlnJgtGHFA1IsZIW\nowypIkdqntNWDUlVXNJ207meioakajYp3pNiTVMNiLgtNlBK1WW4LV6bVA1mao5pu6mKS280pEeq\nJjPVnGrRCNnUtta09y7PHa9HG3/q5TfokiRJ0oz4gC5JkiTNiBEXSZKWLDXZmYqy9Hjve99bxw8+\n+GAdf/e7361jRgYuXbpUx6urq4NtPf7443X85JNP1nGqpJEqhUzFDVJVlVSBhFGNFO3g8bW4XGpI\nkyIYUzEabovHwcorKQKSYjvtPHhcrKrCJjupKk5PvKaU3AApxXZ4HL2xEh5Hak7VE4WaisrweFPV\nmdTUqff3MTXmYgOkRfgNuiRJkjQjPqBLkiRJM7IuFYh/nbyhO5O0FNf/E3lJA7feemv9/2NqOkTt\n53ylzhjFVERibPlUkWPqZ1w/VfFYtAFR+7Opea3paTQzNa8U26B0rFP7SHGSFA3h3Flhp415pAhJ\nqiBDqZLNlBTpSRGn1LSojRylajgcp+vEOA/xHE7NkdI56Wls1K7P/V2+fLmOH3300e7/P/oNuiRJ\nkjQjPqBLkiRJM2IVF0mSlmyqIsiaqUoYjD8wLpEiB3w9zwogKVbQ7oNzYcwgxVJ6Kr2UMmzwkmIN\naX1GJzhuoyHcbtpWijKk69RbQYR6IhW8lu19kSIknG9PNGhqmRQHScfLc81rkKoUtT9L1VZ4HlJc\nitoYTc9143lIlX/S8fXOaxF+gy5JkiTNiA/okiRJ0oz4gC5JkiTNiBl0SZKWLOWRe8vhMVebsurM\nz6a8derY2e6jLTG36HzTMsyzcy4pV83cb0/pv1ZPLrtHu4+e8oapiyXz1lPZ7Z5cf8pYp79TaPex\naJnGdD6nrkfq2pk6e6bsPfHvKlqLZsXTuWr1/A4uwm/QJUmSpBnxAV2SJEmaESMukiQtWSqtl2IQ\nbZSgpys4t9uWHrze56UMy8pdu3ZtdLuMItzIPrg+t5siA/yc202RmFL6IiBchnGJ1L1zal49EQku\nz+6hvftI8aUUu0hxoPY+SlEm3gtcpieiMlXKcdFrk85PG2Pp6eyafr96IzGp9GjbZbaX36BLkiRJ\nM+IDuiRJkjQjRlwkSXqTaV/t85U8IxnszEkpVjD1Oj51fEzxA8ZgGMGYirhQT2WT1G2UsYs2ppG2\nleaVIi6MTUx1XU3L9ZzPKan6SYqTpLjLVBWXNPd0bThmJ1Cej3YfvD491yZFZ3or91Dabqqek7qj\nllLKTTfdNLrdCxcudM2l5TfokiRJ0oz4gC5JkiTNyLqev/yWJEmS9MbwG3RJkiRpRnxAlyRJkmbE\nB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJ\nkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxA\nlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRp\nRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbEB3RJ\nkiRpRnxAlyRJkmbEB3RJkiRpRnxAlyRJkmbkHW/kzp599tlX38j9SXrjHTx4cN2y5yC92Tz00EP1\n/49ve9uvvzt7+9vfXsfveMc7Rj8vpZR169aNjrncq6/++n/B58+fr+NTp07V8U9+8pM6PnHixGAf\nP/vZz+p448aNo8fBz9/1rnfV8c0331zHN9100+i6pZRyyy231PGvfvWrOl5dXa3jX/7yl3W8adOm\nOt6yZUsdHzhwoI4/8pGPDPZx6NChOr5w4cLodnkNrly5Use7du0aPQ5em1KG14DnPV0nrs/jJs5p\najniMXGctNv8+c9/XsfvfOc765hzT/vj/cLz3O7j6tWrdcz78uWXX67jvXv3Xnffly9fruP2XKXz\nyzGv0y9+8Ys6fuWVV+r42rVro/Nr/805Pv3003X87LPPdv//0W/QJUmSpBnxAV2SJEmakTc04iJJ\nkqbxVTvxdXz7Cj+99ie+5uc+ON6xY0cd79y5c7A+Iw579uyp4zNnzozub2VlpY5feumlOl6/fv3o\nvtt/nz17to7T8fFzRk74OaMvpZTyzDPP1HEbFVrDqAbjDjwHUxGXHj0RF17n9hxwXhwTjy/Fa7i/\n9nzweJN0L7X36Jo2atMT1SHGbohzbZdh/ISRJUZWOC/GblJMqD0OnjveGz3RojF+gy5JkiTNiA/o\nkiRJ0owYcZEkackYUUjRgFSRpf0Z1+d2WfGEcQBGDFh5Y/fu3YN93HbbbXXM+Ar3x8jAVORgbH6l\n5IhEqmDDeXB/Fy9erON/+7d/G+yDc9m2bVsdswoIz1WqipOuU7tOquKSlue+t2/fXsesZFJKKceO\nHavjw4cP1zGPL8VPUiWTNm6SjiONeT1ThZR2HyniwnUYS2FkhNfs+PHjcZusLsRtpWubIllcpo2u\n8H5YNLYzxm/QJUmSpBnxAV2SJEmaESMukiQtGWMJfI0+FT+gVK2Dn7///e+v40ceeWR0O4yJ8DV/\nKaW8+OKLdczICWMNjMikaAcbv7RVQ9jciD9jJZbNmzfXMRsbMXKQ4hHt3Dnmvvk5j+mpp54aPaZW\nivSkGMRdd91Vx/v27atjxisYHyqllCeeeKKOf/jDH47uj3Pn+WTjKJ7DqevBufA4eKy8tqycwvGG\nDRtG51rKsDoQrzPnwWvA+TLWxHXb+fLYeX4WjaVMRZxYxWXqeCe3f0NrSZIkSXpd+IAuSZIkzYgP\n6JIkSdKMmEGXJGnJLl26VMepq2T6vJRh5pqZV2Z3T58+XcepbB0z6G3GOpVj5Odcn/lg5tSnStVR\nKrPI/TEnn8pQtvtgZvrcuXN1zJw0Sx1yf8x6M2/d5vXZXTX9fQDX/+AHP1jHq6urZUzb1ZNlBTl3\nZspTR0x2iWX5xvaaM69999131zGvBzPh//3f/13HPM8f+MAH6vhzn/vcYB/vec976phZfN7HvN9O\nnTo1ug8eE69fKaXcd999ZQzvMV4P/n4899xzdcy/A2DpxlKGGX3+HUF7b/TyG3RJkiRpRnxAlyRJ\nkmbEiIskSUt2/vz56y4z1ZmTEQDGElhu7uGHH67jthPpmLaMXCrTx7mwvBzXT5GT9jgY1WDkIJVy\nZCQjlZdsSwcymsCoxokTJ+p4165do8v0lLNs8XhTecqjR4+O7vv555+v4/Z6pI6hlK5Big+1eM8x\ncsLjZYyKcaBU9rI9V+9+97vrmNGQQ4cO1TGPlTETfs6IyWc+85nBPlhilDEarsNSnoylfOUrX6lj\nxnl43O22GLG5//77y43wG3RJkiRpRnxAlyRJkmbEiIv0FsRXrz2vsm8EX19Ovd6VdH0/+MEP6pjx\nA74qZ6zk2LFjg/UvX75cx4wi8LU7IxUPPvhgHW/durWO+d+L9r8dqXskpVjKVLWWhHEZzoXHRKkT\n5FTHR8YaPvShD40uw/+ech9TERMulzqOMs7Dyi27d++uY8ZPuHz7b4553jgPXg+OuUx7rlJkJZ1T\nnpN0/dt7gZGVlZWVOub/V/7v//6vji9cuDC6Px43lymllL/7u7+r409+8pN1zN+VL33pS3XMGBQj\nMYwicX+lDCv38PzwHluE36BLkiRJM+IDuiRJkjQjRlykt6DXK9ZCxlqk3x7GGqYqkKy5/fbbB/9O\nDY24ftvoZg2bJDGiwkjNFP73pidmkiqAlDKMRbCSBueeYhcpgtHGMdiE5sUXXxw9jj179oxul/Ng\nxKE9t9u2bRudb7oGvGYXL16s4zvuuGN0rq10nbldfp6qu7TXj/+eij+N4bWc2sf//u//1vGzzz5b\nx4yptHGSNbyW3MenPvWpwXKf/vSn65gxGq7De5/nk3Gws2fP1jFjZaUMmzFxXozwLMJv0CVJkqQZ\n8QFdkiRJmhEjLpo1vn5Kfwn/Zt6fJJUyrNaSGuCkWMLUOkmKK7SRE2LUI0UfOMf031Bu5+rVq4Of\ncR3GGlLTm7S/qQgGK54w3sNIBiMunMfv/d7vjX7exmhSpCNFclIjp1tuuaWO2yozjE70RFHY4InV\nT3je2gor/BnPG9dndOa+++6rY17bnTt31nF7fx4/fryOWaHn4x//+Oi+GX1iVRVGTrjNUobNkBg/\n4jn59re/XcesvMKGSYzjTOHx8r5ahN+gS5IkSTPiA7okSZI0I0Zc9LpaNDIy9dr2jWCsRdIyMFrQ\nEw1hdY6pdZIUwXitTc7Sf/NTo6J23inSk84Pl0/zbauGpAovqfkOIxybN28eHbdVRnqiN+lzRiK4\nbzbPKWVY7SVdz1T9hNEOauMnKZLD8W233VbHjAAxisIoUXvvsloQt8t1GF/hmMfB/f3t3/7tYB88\np9w/zynjNakRF5dnY6JSSvnud787ug/Gc/76r/96dLtj/AZdkiRJmhEf0CVJkqQZMeKi19WisZbf\nZsTktW43vS41BiPpt43RgvTfmKkKKYv+d4kxiNRcaGr/6fOeMefabpNRhFTlZGr9NTwmRhdKGUZT\nuK1UueXcuXOjyzNS0Vaj4f45R26XMYhUxYXaiAub5qRKIem89TSUKmVYRYbzYgQkVQ1idCVFl1o8\nDlapeeSRR+qYkZG77767jhm1WV1dHWz39OnTdcyIzMmTJ+uYlVvuvPPOOmZjI57DrVu3DvbBc8Lj\nYMOlRfgNuiRJkjQjPqBLkiRJM2LERUv3WiMjqSnDa92HURZJbxS+Ek+RgdRoppRhbIPRiZ7/jvVE\nUUrJcQlWGklVVXojNWkfi8ZaiJGIUobnkTEMVuVI8RpuKzUgav/Nc8K4Da8TryePj/PbtWvXYB9s\n0sN12GyK14b74/3Gfbf/D+2JJqX59jRiav/NyAnPL+f+gQ98oI4ZweE+2t8PxpFY/SY1e+LnqaJL\nWyGH55rbslGRJEmS9BbgA7okSZI0Iz6gS5IkSTNiBl1veimnqP9fyhCmUl6S3ng9v4+nTp2q4zYX\ny9wxs7j8/WeOl58zI81xO6f0Nz5crqdkY9p3+7O0b+6Dn/P4uN12TmlbbanEMSmP3J4bbjd1u0x/\na8DzyVwz89ntv9N55/pT5Q3XTHWPTftg7jzl3FN31FKGOXLex6+88kods4Qij+PKlSt1zPPZXqd0\nj3IfXIf/r2w7n45ts91uyt8vwm/QJUmSpBnxAV2SJEmaESMu0u8QI0DSPDECwFfifG2/c+fOOm5f\nrzNmwHHqHpnK03KZNoLBsnKLdmpOy09FQ6ZKPo7NkdGJFFeYmleKIvTMo/2c6/Sc9574EKMy7T5Z\nFpAdMVPJTY5TbKeUfC9xvrx3L126NDpfLtNGbXh+9u/fP7r+j370o9H5smzl7t27R+dayrCbZypp\nymPl8XEeKT7WHgfnOHV+p/gNuiRJkjQjPqBLkiRJM2LERXqLm6pmIGkeduzYMfo5q1oQIw2lDF/1\np8hKimf0Vpn4bVV+6u1Wmv57laIaqYpHT/WSdt9pnLpC3ki3yP+PvXf7teQoz/+LAMae8/nk04zt\nMWCMwTYEx5GVr6wQZCUXQFASlItIUS59GUXcoUgRd4iLSPkrEBdJFERMooTYYA4KNkY2ZsYzY49n\nPN57DnvPyTbH38VPu/L0Sz+131699t6N/flcvbN2d1V1Va/qmlVPP28mG6fKdmLmyt27d9dY+2Hz\n5s2rtkvr2LlzZ285sX4df3cvaL+7bKXRLUfrPHjwYI3V8ebixYs1VqmXSnhcltdSvHxF26hSFve9\n0XFqybO037S9Q+AXdAAAAACACcECHQAAAABgQiBxAVgDWtu2mXOc88Ks9Y8pCwDWHpVnqBOGbtu3\nEvxcuXLF/q3vfJ0LVB6jW/NxHhkqOXG0XFwUJzPR61Cphcp+WuVmJDKKXrdz25kFlayoBEMT9yhR\n7rRr164aa8IdLWvLli01dtKOlgTISW/0c5WGROnVCnpfaZti2xcWFmqs/aPX6hx6WveuknGXUemM\njr9zaonl6vc2SpOy8As6AAAAAMCEYIEOAAAAADAhkLjA25ahiTRmwb3dPYvbwVhZi6LXPmuS+T2C\nbgAAIABJREFUBABYP1599dUaq5RBY51j4vdat+p1e945pqisQOebrVu31lglA6X4pEBOApBxKdG2\nluKTwijaJ871RefgWI72nUvM5GQfeozWodcU69S+colxVAbh3ETic0X/vW3btt7PtT7ta9c/cTz0\nntF2aR+qa4wbcy0n9pVeu0q1lpeXe69Dj3f3evx+OFlMRn6i52q/xe/HfffdV+N/+Zd/qbFK1obA\nL+gAAAAAABOCBToAAAAAwISYnMRlPWQJ8M7gwoULNXZJQFq4N/3d1qduMc/i4oLDCsA7lw984AM1\ndm4imuAlyh3cNryWpdIQ5xSiEpfotuGS0OhWvzrCOOmLygri3OgSq2ms5zjJiCa5iYlx9Npdu7Tc\nzPytrh2leMmRK+vq1as1fvHFF3uPVyeTUrrPOL0OvXZtl46tXmtLnqlt13HWz9W5Re8R7WftT5XK\nlNJNQqToeDrplLsv4jN4qLORjp/2iV7f4cOHO2WpxOWrX/1q7/lD4Bd0AAAAAIAJwQIdAAAAAGBC\nTE7igqzlN8kmdRhT7jwdRKbC888/X2Pdzvt//+//dY5zb8m7fnfbVe7t9fjvuL3XR+ZtewB4+6BS\nBJWJ6NyhEoN9+/Z1zlcZh7pqONcRlbu47f843zhZjJbrJIAuyVKse+fOnTXWeVDnTe0H5+Ki8pwo\nP9Fr17brcS5Rjco5VDJy5MiRTh0uaY7WoWOmrjpa1vHjx2uskpZSSjl//nyNDx48WGOVjDiJk0vK\nEx1W9G8uqY9z1VGpjfZbXGO0EnD11e2kVq2EWa5cRe8xHRttr17r4uJi5/yvf/3rNdbvZxy3LPyC\nDgAAAAAwIVigAwAAAABMCBboAAAAAAATYnIa9N9mnH4+fj5U471WumMt12nIXMa1iLNAyqD68Jj9\ny1276uK0f5eWlmr8r//6r73te/DBBztlqY7Q1aefZ8YjZmNTnK2Toro6AHj7oxpk1aA77WzMgKg2\nfVqWzu3uXSP93GULLaWrQdf6XJZI1cJntcJu7tSy3HU4+8U4Z6vufPv27TVWO17Vd2vb77rrrhqr\n7aG2L56j7VVdtvaVHqPtOHToUI3jewf6b71ntC3uHSunsY7Pnv3799dY+0qf1Zr9U7XmintvoBR/\nPzjrUCVjuRj/prG+E3Dp0qXeY/S9iLNnz/a2I7ZxHu/y8Qs6AAAAAMCEYIEOAAAAADAhNkzikskY\nGj+fihWgs99z0oe4RemyUup2jrN4yl63lqvbXVrHtm3baqzbWhpr23XbtJSctZbWoRZGzhqrJR9x\nNoZ6/okTJ2qs21V6TXGL6kMf+lDvcUOzf+kWXuyP1pYuAMBHP/rRGjs7PCXOIy5bss6PTjLg5uOI\nyhf02eCeYzoHa30qx9BrLaV7XXqcorIErU+PV/lIfK6oFHJ5ebm3LXq+y8Cpz8Ron6sSErU91Fiv\nQ+vTOnQ8YibRjIwmSqH6Ptc+jDJTtQjUvnLrGCepctk4I3q+jqf2r7ND1HGOUintR3e+jqfe35cv\nX66xjkG8r06fPt1blpP9rAa/oAMAAAAATAgW6AAAAAAAE2LDJC6Zbf64tadbDu7N5PXItDg0s2fL\nkcPJKJx8RONWH+rfdKtFz9c6dOtKt5W07bP0rW61DZWMRJycSO+L22+/vcZ/9md/VmPdNtO30kvJ\nuapkrr01HtpGJC4AEFGJg5vznXtV/LfON+6cTBznQ+dSoZIBfd7oc0U/VzcQdQAppZSTJ0/2lqt1\nq2RAr2/Hjh011n57/fXXO3Xs3bu3ty3OmUSfic7ZLMpxnBuJymtcRkx1Zzl8+HBvW0vpXq/KMPQ4\nJyFVFx7tz5hJVJ+dGqtEJiP1bT33nAucO8fV0XKQc5lI9RztNx1Pt+aLUp0o11ohkz28D35BBwAA\nAACYECzQAQAAAAAmxKQTFUVJRCaJzEbiJDjzbJ97Kzritkj1DWR9g/zYsWM1vv/++2usWzZxu3Oo\nk05mGzWW6d7K1mtS9xRN6vCJT3yixroVeODAgU4dujXo6s7QcibSf+t1kJAIACLOISObBC7r/LJa\nHTFRUQad32699dYau+dF/Py2226rsZNIOqmNzsEqK1DpYyndflCnGZXk6PnaD/pM1MRNGsdztE9v\nueWW3nboMSofUSe0loxmcXGxxk4CpOVu2bKlt5w45k7WouWOeVYO+VsfzvEuus7peDrHPHXrefjh\nh2us6yOVYMU1mI5hJgHiavALOgAAAADAhGCBDgAAAAAwISYtcZkquo2SlZw4skmaxtThkjc89dRT\nNf7Rj35U4zvuuKPGKgeJbyxn5BlDkzrFz935rn9cAgJNTrRz587OOboVNiYBlm5vxeRUum0Yt95W\nKyt7jw194x0AfrvIzIGznD/LfJyR2zhpX8upQxPr7d69e9Vydc52dUT0HJXRvPbaazVWuYuWq04f\nKmtpuZ9kHOic847GMRmVtl2fdypfcQ4penwrMZZKNZyziVsL6PW11gvaRtdXTgKk6Bol3rvu2a5y\nIO0HJ51Rd5+YgMgl/JpVysov6AAAAAAAE4IFOgAAAADAhPitkriM3d5bC8bKB9w1jZFaxLJcoor/\n+Z//qfETTzxRY3V6+fznP99bTild2YZ7q95tZbp+i9IQJydSVNbiJC4LCws11q3LUrrbT3odrr1O\nyqKuOBqX8ptbk31lOTebbJIj55IzNkEUAKw9mjgmOoL0EaUILjGOztNOouKS/cX52M1Xbp539bXm\nscx85eSSbp6O9elxbs5387zKILRvo8RF/+3GwDnF6DEqo4jPLn3O6PkqAXFrCTdOUT6ichm9JicB\ncUmrtD+jXFbPcf2mz2nnyKLEezcjt1L5iia3UimRXlMrAZFbVwyBX9ABAAAAACYEC3QAAAAAgAnx\nWytxeTsyz+tzb4rrNo++Lf/qq6/W+Mknn6zxY4891ntuKX67U7eW3NvW2ibdMsw4nMQ6nExI3Wg0\nbr3dH7fF+nDbgS+88EKNv/GNb3TO+du//dve+jNSJpeIozUeAPDbxXe+850auwQ46iCiCWziv2NC\nmxWcXE7nDif/KMVLDjPyTK3bOZbEtigZWWvWZcbJc5ysRduon2eTSLlnsJM16hiohOPixYudcvV+\nUGmIK9e1Ua8pJirS+8/dP04i5e6riLr1qKTGSUhdAi2VlbTkWfr90HZp0sLz58/XWNcPKp1V159S\nvPyFREUAAAAAAG8DWKADAAAAAEyI3yqJi/J2l7uMRftHt4bUKUC3pVTicuLEiRrr28u69RTPd9IL\nty3ptg/jlqrbGnR1K7rlp/WNTSilkhyN9Zj/+I//6Jz/p3/6pzW+++67a6xv6GeThTjc1uJQSQ0A\nrD8XLlyosbp1OGlI3ObXeVT/pvO2xipxdPKYKJXR+U7lfU62oXPS4uJib7n79+/v1OFkhk5+4iQn\nLQcw/bf2lfaJnh8dWlZwiWniv901uWelc9LR9pXi3X5U6pF5NreS7DlZixtnrds5m6lMpJTuM1n7\nWu83vVa99/Re0vs71rFjx44aO6cZrfuVV16psT6nDx061FtOKd17XNseExpl4Rd0AAAAAIAJwQId\nAAAAAGBCsEAHAAAAAJgQv7UadMijGivNlPWZz3ymxv/0T/9UY9XkuWxhEWenpNor1abp8ZmsYLHc\n1nF99TmNXBan13PWVqdPn+6c/8Mf/rDGH/7wh3vLymQxbVmNueN4XwPgtwuXHdMdU0p3TtQ5X98j\nUo2sew/IzW+l5N6/0Vg1wefOnett38MPP9ypY+fOnb11OGvdTEbTiPub0w27vnJZumNbMjpwjTNj\nE8vVv+kz3N1Lekyr39zfnAbdPadbz2zVp+txWofrB/dOWfx+uOtV9H7V+lyW36NHj/aWU0pXA+/e\nYVgNfkEHAAAAAJgQLNABAAAAACYEEpd3AG7LSbd8ND5w4ECNVQ4St75mkYqsoFtXur3VwmW4c3IO\nPcZtx2XJZJjT7dFov7SwsNBbVsyit4KOh/ZPq+0ZWQxyF4BpopIKN0fovBCzNDtrPkXnAp2/3Xwa\n7eH0by15xwq6za+yFpUJPPXUU51zNKvk3r17ez9XW0jNppnJBF2Kl2S4fnASx9ZzxUlCW5aGfZ+3\nMm2763AWk04G07I6dv3j7tGMBChKTDK2kIquY1zm8tj/mQy17hi9j0+dOlXj+PzW5762JXtfRvgF\nHQAAAABgQrBABwAAAACYEEhc3mHots/ly5drfPvtt9f4c5/7XO+5GeeUjWat5Bxuq07rUIecKP/5\n1re+VeO//uu/rrHbcnQyGh2/lqtOJtMqAEwHnTPclrh+56MUwckBMnOMyxYZcS4eis5XLoupq7uU\nbhZVdaDRzI4qkdEMkSp3cXEp3YycTpKhOLmkO6aUnPwkK5dxuLJcu5x8RI/J3le6Hsi012V8LaXb\n70565WJXbhxLPceNsyvLfR+XlpY6/9ZswJrhNMrRsvALOgAAAADAhGCBDgAAAAAwIZC4vE1pua+s\noJKM3/3d362xmuo715cpkdleG+vcoui2or5Vf/z48RrrFmpEx8ZtObvtOD03brXqv11Sj7G4e8lt\nGQLAcJyDiHPnKKUr+9Atdefipd/ZjJNJPE7R+cY9e7JzRCapnM67ly5d6m2H1qdyg1K6jjDqWrZr\n164au6Q1LkFTlEG4BD/6uYudjDLWofdDxlXNyV2cBCfi2qXuJer8o59HZzNF+9Tdlxl5TktG48ZA\nvyvO2U6P0bbG+0rdhXQ8ZpUH8ws6AAAAAMCEYIEOAAAAADAh2Jd+m5Ixxtdtvr/4i7+osUuMkWUj\nE+PMs2631afbVS6BwT/8wz90znn22WdrfPbs2RofOnSoxi4JhBLfsHdkjxvDWsloAN7pZLb2Izpf\n6Zb80ARmmQRErfY6RxjnhNKSIjjc/KZzs9ah83QpXacYnY9V3qkyxZtvvrnGR44cqbHKimK7M4lx\nFOfQ5SSVsayMU4iTMmXn74ybjaLjodKX1rmZJEvu+6HENZD2T0zA1XdOxtFF75dSuveMSnrOnDnT\nW99q8As6AAAAAMCEYIEOAAAAADAhkLi8TclsE+r2TNyqGVPfRsod1qpu3a7SLThNQLRnz54a/9Ef\n/VHn/H379tX4G9/4Ro3/6q/+qrc+twU4JSnJVMYc4O2Auj7olnom0UwkI3FT+aKTokTZRMb5w8lX\nnDtMVhripDOZuSc6nOi/33zzzd66Verz2muv1VgTJqnc5ZZbbunUoW1Up5m9e/fWOJMsKpMcqpTh\nc7Dr54irU/tQn49OLqX3W5QcqXRH1yJaR8Z1SIn3lXPccec7qY1+J1rOPXq9W7du7a1jNfgFHQAA\nAABgQrBABwAAAACYEEhcYC5MReKwVrIL3e5aWFiosb6d/clPfrLGcetLt7h27txZY7eF57ZaW8ke\n1oOMIwQADMfNV7rl7+aFUrrOFO67qfOSzmmu7phgJSOL0TiTPCfWrXXo9TrJQWbuifOxSwSndehc\nq206d+5cjS9cuFDjY8eOderQeV4TIKl7mkt6lJWfZGRGiuur7Pzt7qWMDEfv43jv6nW4JFs6Zjo2\neo9qHS2JiyPjTKPEe9f1z6yJ/PgFHQAAAABgQrBABwAAAACYECzQAQAAAAAmBBp0eFsxVnfutH+q\nQTx48GCNP//5z9d427ZtvceXUsoHPvCBGh8+fLi3joy+bz1o6TvHahgBoB+nt1UbwNb3zOmR3bzi\ndOo6B0YNumrKXVv0fFeuy8zYOkevyWnIdd51fRCPc+0aqtFfWlrq/O3ixYs1Vt26ZjFVa0bVpmtG\nSte+bBsVd4y7dyLuXlItvTu+1T4dQ6fX1jHXfr98+XKNXYbQ2BbtR9WtO/vF7LtXQ9+NWA1+QQcA\nAAAAmBAs0AEAAAAAJgQSF3jHEbcJ3faek5no9qjGujUXt223bNlSY7WRunbtWm+7stnj1oLWVqTr\nn6nYbAL8tqLff5ex00kM4t+cveEYWUqrXQ6dEzNyl1K6c4xKGVoWk33ntuQVmevQz520R8cg9rm2\nV2UtOuefOnWqxipxufPOO2uskkqVvsT6nQVv5vr0ms6ePds5TiUgO3bs6D1fY22TnqvHtO4r97mT\n86i8Jn4nHBl7Snd9rfvefYeztpm/Ud5MZwEAAAAAwJrAAh0AAAAAYEIgcYF3HFnJiG5R6fahcyDQ\nLc5Yh8u6l3nTW2U00VVhrWlt57nsgQAwHP1u63feyVWizGPoNrrOUU7i0iIjWXFSPZ03W7IE54rh\n2q7zUFbukHHeyEiDYn1OnqNtV7nL9evXa7y4uFjjPXv21Dg6gKkLjLZRpSWKk3Bov7344oudc44e\nPVrj7du3957vZD9arko7W/dYRkKk3w8nr4nProwjkOufrHTKgcQFAAAAAOBtAAt0AAAAAIAJgcQF\nwOC28Nz2XCaJRyndLbXM2+vrLWsBgPXHycTcXBJldC7Zi8ZO+pBNsOK2950EQOdE57wSJTyZpDCu\nLP3cSRpabXflOnlNKxmSk3pknLA0OZUmOYrt07IuXbpUY5XCqMuJjr9KTrSv9u/f36lD3WUyuGda\ny4FIycilnLSolXApI91ysh/33YzX6qSss0pA+QUdAAAAAGBCsEAHAAAAAJgQSFwADJnkBE6ukk2G\npAz9HADePjhZylCZRzxH5w91o1KJg9vajziZQqstfe3Q+tSRI6J16Jyqsj8nXVBpR8Rdo3PSUbQP\n9Zg4T+vfXPIeJ4lwzj0xGdEbb7xR45MnT9ZYXWBuuummGu/cubPGN998c40PHDhQ402bNnXq0LFt\nSXpWcI4lmXNL6V6jc/vJymWUjOTEyWgyiY0iOoYuidRq8As6AAAAAMCEYIEOAAAAADAhkLgADMRt\na+mWVtx2c9vUYxMgjMFtOWelOrod7NwhACCHkzVkt/AdLimLxipjcPNTRNur84Iry81p2ToyMhq9\nprfeesvW7a7Rua3oPKjSFZcwJ57j5EQuAZ6i86wmM4p/07KWl5d74/Pnz9f47NmzNd63b1+Nf/zj\nH3fq+PSnP13jHTt21Fj7J3PvqjONOs6U0u1Hle2oPGfr1q29dczy3HTfqaEuRbE+J3md9XnOL+gA\nAAAAABOCBToAAAAAwIRA4gIwJ7Jvd2cTj6x2/DxlMG47LyZJ0n9r/F//9V81fvzxx+fWLoB3Ou57\nHmUFQyUgKolwErwoBXBzUSbRkauj1UadY/R6XQIbl6go9pWTk7j2OomLcxmJ7XJyED1HP3fjpHKV\n2C6VkGg/qLxG+3NhYaHGx44dq/Err7zSqWPPnj01fuCBB2q8d+/eGqtjjtbh3FKiw462ccuWLTV2\n95WTj7Tc0tz3yCW0cjKYloOQS5qU+W72tm2mswAAAAAAYE1ggQ4AAAAAMCGQuACsM0OlKWvl6JJ5\nYz1uDeu/1bnl3//932uMxAVgOE5a5iQnMZGO2+p3chInqdA6VHoQy3Xb9s7RwyV+icc7dymHa4eW\nE4/R/tV2aT+4REXRScUdr33nrt3JQZwbzdWrVzt1XLhwobcOjXWe1ja6cdJkRqWUcvz48Rpfu3at\nxprcSJMeqfRF61ZHlnhfZcbcPa9mcXHJJA7MfJ/ifeW+U63kXy34BR0AAAAAYEKwQAcAAAAAmBAs\n0AEAAAAAJgQadIB3KE5Lp59HOzL9m9p0PfLII2vRRIB3DO47qJrwlj2h4jIfunKdBlizOsY2qjZa\ny3JaaD3GacBL6eqWnXWtXofLaJqxzGsd5yxls7h+z2Qx1Wtq2V7qOZl3ijRWPbt+Hi0Qtc4rV67U\nWC0fX3755Rpv3769xrfeemuNb7vtthrv2rXLXofeV65/Mnaf8b7K2CAPff9hrTN+8ws6AAAAAMCE\nYIEOAAAAADAhkLgAQNM6StG/7du3r8aPPvro2jQM4B2Cbq876z8lShHc9r6zUHTb/M4qLp7jbAxb\nVpBD0fMzVrAZCU8sKysbcmWtEGUwbjzUYlCPcRlUlc2bN3f+rf2jdahMRMtSyZIeo9aK0UZSM5TG\n+vvKvXz5co3Pnj1b4xMnTtT46NGjnfP3799fY70mZ1WpfeVkKbPIT9yYtTKDryX8gg4AAAAAMCFY\noAMAAAAATAgkLgCQRrf9NDPcWr/NDvB2x2UP1e9cK2uiykzc9vxQp5iWjMbJRJxkxLm7RFwGRufi\n4fqnlb3Rucu465slK6Ser/WpJEfH3GVpbWV21eNaTiwrqBRF5SrqnBOdezLXru1ysibNenrx4sXO\n3/RZoplINUOpOr/o9bksrVlZSiZjqKtjlnthCPyCDgAAAAAwIVigAwAAAABMCCQuADCaoU4IANDF\nbZc7uUNEXTlUZqDn/OxnP1u1XOf6Uop3TNGynLOIk5+0nGIUJ8nRup2UIM5PGXcQJzlyLlexjoxE\nQq/DSS20TdeuXevUoX/TOrTtbmx1nNSpRe+jWJaerzIT/dzJqFryI016tLS0VOPTp0/XeNu2bTVW\nucuePXt6P2/JsxQni8lIuFqOZ5nESqvBL+gAAAAAABOCBToAAAAAwIRA4gIAM9FyYgCA2XEuIy2X\nCidZcbHWodIX5yZSipevuHKdK4qTXZTiXWeGymVcW0vpSjpcEiHnhKLlapx1jXGyFO0T5+6i7W6h\nZaksxvWb9rneC7F+/ZuTbagMyj0jWveuSmxU7qIymIWFhRpv3bq1xip30biUrkRG2bFjR40z95VL\n1hXPd9+7IfCEBQAAAACYECzQAQAAAAAmBBIXAACACeHkA7qlHo/JJFBxkhFNVKPErXknM1EyyX70\n85jYRq9R63fSAud+4uQj8W+ubm2XJvX51Kc+VeM777yzxt/85jc7ZT333HO9dWi5Wp9KQ5wbTewr\nJ6nQZEMqcVE5h5OibNmypVPH9evXa3z58uUaa59ooiEtV++3luOJG3PnFKMSoPPnz9dYJTGnTp3q\n1LF79+7eWMvSNuo1KdlkSJnvymrwCzoAAAAAwIRggQ4AAAAAMCGQuAAAAGww6iCiOLmDSgkibutd\ny1K3DJW4uEQzsdyWE0tfWYq2XSUGpfymjKOvLU4uo8doHbE/nMOG61N1T3n++edrrPKPmERIic4o\nq+FkFC1Jhesfva/c2LakUzoees84FyDF9Wesw8mtnKzF9Y8eExMuLS4u1liv6cCBAzVWCdC+fft6\n26SyG3evzgt+QQcAAAAAmBAs0AEAAAAAJgQLdAAAAACACYEGHQAAYINxOl5nCRg/V72usyfUz1WP\n7HTuEacpd1aHek0uY2dEj4v69L52OAs7vdaWzZ3Ts7tMks8880yN//d//7e3TbFOvSaNnX7efR6z\nmzr9tWqjN23aVGPVZbtMq3FsnAZd0TY660dnv1iKv2cymW+1vtY9rTpyfafg5MmTvcecOXOmxmrL\nqMfE7KR6XCYr7WrwCzoAAAAAwIRggQ4AAAAAMCGQuAAAAGwwTg7i5BnR4s1lbXRWcFkJiKIyBS3X\nne+kM062E+tQ3PW5vlL5R7SBdBaRLkOptkklDq1spYqzR3RyF9fWlq2fs5jMjEFLMqT9o5IOPSdj\nv+lkPhG9RpcZ1kmtWnaWrlyNr169WmPNSrqwsFDjrVu31jhKV3bu3Fljlbs4adBq8As6AAAAAMCE\nYIEOAAAAADAhkLjAmuLeTAcAgP/DOWFk5C6leGmIzrtOIuHkA3v27Okcp84U6hpy+fLlGjs3Geca\nEuUnGfcT11d6rS5baCldSYZri2uHOoA42U2sU689kxnUZcqMY+zkOU4C5KQlznkltkXHXOvW+jKS\noVZGVEfG8SaLtnfLli01VvmSylK0vW+88UaNL1261Cn39OnTNVYpjMZD4Bd0AAAAAIAJwQIdAAAA\nAGBCIHGBNQVZCwDAMJyEQ4lyBSe3cFIP3XZX94nPfvazNX7ooYc6dagc4NVXX63xK6+8UuMnnnii\nxi+99FJv3S2Jg0s241xSXJKbjLSnlLa8Y4WW68gKKn2J5zi3F3eMSlmcdCX+Wx1FMq4xWfmJk+po\nX6scxN1vzmEn214nP3KSoZasycll9Po01rpV5qOSmNhelb/od2UI/IIOAAAAADAhWKADAAAAAEwI\nJC4AAAAbTCtZTB9Z+eDtt99e4/vvv7/GDz/8cI1VHvGhD32oxpqYppSuW8tTTz1V42effbbG6mSh\nsg8nM4myDSc/cElvnLSnlajIJcBRWYQrK0uss69u5xSj6OcxEY9LoKR97WQmzqUkXqtrrytXj9c+\ncLKdUnziIdfvWpbW4RIQlZKTKSkuUZUS263t1e/UtWvXBtW9Ar+gAwAAAABMCBboAAAAAAATAokL\nAADABqPb4y6xjZKVuNxzzz01fuCBB2r89NNP19glHTp//nynrB/+8Ic1fv7552vsJAcZ54wo53GS\nE0XrUKmGk3ZEaYiixzm3DiePaElDnPuKSlH0GG2jk+1ESYW2UftBk+lk3G9akiqX1Mk5rOi9kEko\nFdvuElINvY44Znod2m9OquOu1d3T8Tgd582bN/e2fTX4BR0AAAAAYEKwQAcAAAAAmBDvWs9EMseP\nHydrDcDbnLvuumt1CwoA6HDo0KH6fHSJhtz2fyldeccNN9xQYy3LbdsPdZCJaN3OxcM5dbTqc+c4\ntw5th54bXThcQhsn21C0b1sJfrROJ8PQc/R4rUPjKPNwfadtd9IQrTs75s4Zx8maXN+2xsOtSV1Z\nMUHUCvHeGToeWodrk45NPF/PWV5ervH3v//99BeMX9ABAAAAACYEC3QAAAAAgAmBiwsAAMAG46Qh\nituCL8VLOpzkQOUg6iDScvfQ5CuKygycDMdJDOK13nTTTb3nOycN587hnDrivzPyCifhaSUwGpoY\nx8lr9DqiNMTJPrS9LfeU1eouxV+jGw+VfWTvK70OPU7dXfTaM44u2m+xjRlnmnh+X93xGHdfzQq/\noAMAAAAATAgW6AAAAAAAE4IFOgAAAADAhECDDgAAsMGo9jdj/ZY9X1GdurPZa2mFVQcjPQTtAAAg\nAElEQVSsOmcl016tI+qJNQumHufsIvV8p73P2gi6drnxmEWD7vpHr0/HsjUeQ60OFddXsd2Z63B1\nZN5BiHW0rERXO17JZo9VnH2ny5TaOn8Wu9IIv6ADAAAAAEwIFugAAAAAABMCiQsAAMAGo9vjzlKu\nZZnnLPScPMPhsneW0rWVU2tF/dzJNlwdLdmOs7pT9PpcBtZIS2KzgpOZuLrjdTh5h0OPd/KhFk6+\npHU7O0VnVdlqix6nx7jxdLaMrfa68zNSktgOZ4/pxrA1tq5u7VOV2LTkNi34BR0AAAAAYEKwQAcA\nAAAAmBBIXAAAAH7LiNvruoWvW+o33nhjjd3W/iwuFU5moHVcv369xiqDcHKMUoZnInVlaX1RpuGu\n0cldVM7h+q2V2TWbfXQoQ91PnNNP6zr0b05upf2psWYC1f6I7dPsoy6zq57j5CcZKVE837XL3et6\nL8Ssq3odWi4SFwAAAACAtwEs0AEAAAAAJsS73NvJAAAAAACw/vALOgAAAADAhGCBDgAAAAAwIVig\nAwAAAABMCBboAAAAAAATggU6AAAAAMCEYIEOAAAAADAhWKADAAAAAEwIFugAAAAAABOCBToAAAAA\nwIRggQ4AAAAAMCFYoAMAAAAATAgW6AAAAAAAE4IFOgAAAADAhGCBDgAAAAAwIVigAwAAAABMCBbo\nAAAAAAATggU6AAAAAMCEYIEOAAAAADAhWKADAAAAAEwIFugAAAAAABOCBToAAAAAwIRggQ4AAAAA\nMCFYoAMAAAAATAgW6AAAAAAAE4IFOgAAAADAhGCBDgAAAAAwIVigAwAAAABMCBboAAAAAAATggU6\nAAAAAMCEYIEOAAAAADAhWKADAAAAAEwIFugAAAAAABOCBToAAAAAwIRggQ4AAAAAMCFYoAMAAAAA\nTAgW6AAAAAAAE+I961nZsWPHfr0Sv+td76qf//rXv+49/nd+p/v/h1/96le9x7my9HM9Vz9v4drl\ncO2I1+HK1fNdWdm+yrRd+2Rom7J1ZMZ5nrjxn+V8JVNWq+7MPTq03NjWMdfujo91ZMo9evTo8I4H\neIfz5S9/uX7ZdD5/z3v+7zH97ne/uzcuxc/V8bgVLl26VONz587V+IUXXqjxa6+91jnnZz/7WY23\nbNnSW65+/r73va/GN910U43f+9739p5bSimbNm3q/fz111+v8S9/+csab926tcY7duyo8ZEjR2r8\nsY99rFPWyy+/XOPl5eUa/+IXv6ixjsGVK1dqvH///hrfcMMNNdZxKiU352sdmbk1uyZStK80ds+6\n+Ln2iV6juz5tk94vS0tLtt3Xr1+vsd6Xb775Zo0PHjzYW7ei4xT7Stvu+kFjve633nqrxm+88UZv\n++K/tY0//elPa3z8+PH085Ff0AEAAAAAJgQLdAAAAACACbGuEhcnOVF0W6K1vT5UkuG2RLJSBCeR\nychP4rVm5QRDjs/W4RgqtWidn+mTsXU4WvfP0DoyYz5PudTQvppluzNTd6sPtY5YPwDMh8wcOov8\nTOUuro49e/bUeO/evZ3zVZpy4MCBGp8/f763vu3bt9f42rVrNb7xxhtXbWssN/MMV0mNfq7Sl1JK\nOXbsWI2dnEglDiqJ0D5oSVwyaBv1fPeMiX2gbdRY0TF3ZbWeHU7WojhpiHtGaH+uVv9qdSg6Hiqv\nKaUrTdF7UeU12g793MmE4nVoX2tb4nFZeMICAAAAAEwIFugAAAAAABNiwyQus0gR9Jyhji6O1tbK\nUCnK0HJaZLa4Wm3PyDDGOJaU4rev1qp/nORkrdx63D06T2eazBhkx2nM/er6MI6x26pbD4cegLcz\nGWmAfs+iO4ueo7GWq44nKtVwzhvqWFJKKbfeemuNVb6i9amUQOv4+c9//hvXE9tXipfbOTcbla/o\nMerO8rWvfa1Th7Zl165dNVYXEHWjca44rWfMUFmkfr558+Ya7969u8bqZFJKKWfOnKnxK6+8UmO9\nPidl0jF3cQtXlhvnliQ3swZUWYo+h3TMzp49a+vQ8bx69WqN3dhqfepApMe0pCtjJKcr8As6AAAA\nAMCEYIEOAAAAADAh1lXi4rZashKXjHxhrVwm5iUfaJHph0wyo9Y5YxP5uDqGHp+V6rhtWyXj3NO6\ndzLbyVkXoL5zW3W0zumrQ4+Z9c3w1epujQ1SFoC1QaUBKtVw8oMoDXFJb7Sse+65p8bf+973etuh\n7hO6zV9KKadOneotV9ui1+HmXU38EqU66sTipCzbtm2rsSY2csn3VNIQ266x1q3yHJXL/OQnP+m9\npoiTejip7h133FHjm2++ucYqr1D5UCmlPPvsszX+0Y9+1FufSpa0bi1X+7A1HnqOlqV16NhqrEl8\nVMJTSnesVMajkixth46BtlfHSe+R2F51EdI2DpWltJ7r2t54venyZzoLAAAAAADWBBboAAAAAAAT\nggU6AAAAAMCEWFcNusPZ2WUzpQ21pJslW+UYy6SWPnxMe1tWlRmLwIzeap469XlmK3XnOIuuodfd\nqiPTjnjMmCyxSsZqtEXmPs5+V1xfA8Bw1C5OdbX6PXfa9FK630fVvKoWdmFhobc+fZ+llTnS2TG6\nNmqsmuysVZ2zWdT6VCcf9dMrxHlMtdwXL16sseqk1ZZP61Pdt/ZB1OtrFlRnzaj654985CM1Xlxc\n7L0O1cWX0rUV1LarptxlxNQssar7jmOueu0777yzxjoeqgn/1re+VWMd8w9/+MM1fuyxxzp1vP/9\n76+xavH1PtayXn/99RrrGOg1qX491q/oPabjoe8tvPTSSzXWe0fvkVK6Gv1Dhw71ljUEnqoAAAAA\nABOCBToAAAAAwITYMIlLxgKvJRMYI3dR4ta8s2mcl0QhnjNWyuLakZFCjLVczLTXHe/qjudmzhmb\n5XNouWPlMq5uN2ZjM5cOzfjqrMpabRkrhQJ4p6NSC0crM6duz6vEQbf6v/3tb9c48/2NkhH9t0of\ntC1q0+is7VRqEa9DpRp6TWrlqLIGNzdrW+N1qDRBpRqvvfZajfft29d7TEa2E3ESIj3/1Vdf7a37\nxIkTNY5zucsYqrjnq5MPRfSeU8mJXq+OubMC1jj21d13311jtZh8+eWXa6zXqjITvS9UYvLJT36y\nU8e9995b48OHD9dYvyuaHVclS1/5yldq/MILL9RY5WOldO9x/d7dd999ZRb4BR0AAAAAYEKwQAcA\nAAAAmBCTcHFRdHskboM4uYOTGcziTDFUOjM0+2frb0OzVbZkBRk3k4x8ZBYyGV8dY8dmljoyGUrd\n+fOU1GSkSK3P3fbuLFl7+8qMx7W2LAFgGM8880yNdU7SrXLdQj9z5kznfHVlUSmCbvurHOCBBx6o\n8c6dO2uczezs3FecLCWb+VjnGCed0GtSsnJZRTOU3n///b31uXm6JTHJPAe1r9S5Zf/+/TVW+UnM\nJKpyII1dv+l4aNzKwK5/0zF0z03tEz2+lQVbr0szgOpz5bnnnqvx0tJSb3163XpMKaV86UtfqvEj\njzxSY/2uPP744zVWGdSRI0dqfPr06d76SumOocuCOwR+QQcAAAAAmBAs0AEAAAAAJsSGSVwyMoHW\nVpt7M3moTKB1nCt3bPKdTGKmsXKXocmQ3LnzdOeYJSmTG4+hiYNmIXN+dhtVcUlIMnVk0e3LMeW2\nXHUAYH6orKHlQLLCbbfd1vm3Sxak58dENytcvny5xiq7UElNKV4ukXGE0rlE2xSdMLRcddLQtjvZ\nhZNgxD7Uck+dOtV7HQcOHOgtV9uhEofYtyob0nL1OOeqsry8XOPbb7+9xidPnuzU4WSNWofeC67u\n1rrLrX0yTmPuHol1/OAHP6jx8ePHa6z3pXOa0SRAWofKWEop5dFHH62xurXoOXrva38++OCDNb5w\n4UKNVVZWStcpRtsVpUlZ+AUdAAAAAGBCsEAHAAAAAJgQ6ypxycgVWtIM99bwUPlJNtGMe+t8bOKY\noS4nma2kbF9l36TPMEbukJHglJLrq6HOJLMkQxo75krGdWhom2ZhlqRD2W1RABiGurU46UIraZ07\nx+HmsSg5UfT54Z67zjXMlXP9+vXO3/QclZA4RxC91uzzMZMMSSUu2o6Pf/zjvZ+3nisZSY6ThqjL\niLrMlNKVTrQSzK2g16ruJ9pvcY2gf9N+089VOqNJeXRs9+7dW+PYV2fPnq2xOvQ89NBDvXWrREkT\nPKnkRJNOldJNhrRr164aa9KrJ598ssbqvKIJk1SO00KvNyM57YNf0AEAAAAAJgQLdAAAAACACbGu\nEpeMTKC1hT9UGjKLFCFT7lC5QzahkPt8FilDJmFDRgIyS+KgzJve7visVEdxTgHZhEJD76W1cr/J\n3EvZBFiOjKwl66oDAPMjkwTGSUlKGZewTeseK+dz80pGMlpK163Dne/mq6wEz53j+l0lHJpIR+Po\n4pJJHJdxP9FyNXlOKV23F3e9MZnOCirtUKI8yklyNFanGZUAqRRF2xr7St2CtFx1cVH5isZ6HVrf\nF7/4xU4d2qd6j2mfqrxGJUCKHn/+/PnO355++ukaaz+qtOgLX/hCb7l98As6AAAAAMCEYIEOAAAA\nADAh1lXiknEjyUoR5rXV3pIJzCsBTksmkJEWZB1PsvXP6/jMuGWcdFrbtkPblXETycp2Mm4rrtzW\n9vNQx5Ps1vXQLW7XpqzEaUzdANDFuZE4xkpc9Dvv5qSMxDAel4mdxKQULy1wz5WMu5tKF0rpSlO0\nLOfcoslp9HiVVLTaq23UclVq4SRH+nmUuKjEwkmIXL9lx3zTpk29deg4Odcgla44N6KIc/j57ne/\nW2N1dLnrrrtqrFKSxcXFTrkLCws1VonMuXPnaqzOLYcPH66x3i/adk1GVUpXbqPXsbS0VGaBX9AB\nAAAAACYEC3QAAAAAgAmxrhIXZahjRWSM/KQlwRgqOcm0ryXV0W2mTMKFLJmkUJlzW9umGZnK2AQ2\nrv5MUgZXTiuhhJJxBHDbti03mjFykOy94LYQ3dg4l4EsSFwAxqEuE26+0C30KAXRREfqkjFULpOd\nK11Cocy82XouZCSgQ+crlUSU0u1HlWE4yYjWoTIKl7gp/k3bq3Ib/dyNv7Zv//79nTrOnDnTe44m\nm9J7QSU1Wp/eS9lnl5MvOUcW15+ldPtOEwzpOdr2e++9t8YqwdFy4/fDOcq4ZE/6ufaV1hEdcrSv\ntSwSFQEAAAAAvA1ggQ4AAAAAMCFYoAMAAAAATIgN06ArGc1Z/Pc8tdSZ89fKqnAWS7tMfWPaPk9b\nv1ky4mUYmjE0e+8MzSSbzcDpsuBl6nBjmX0/wGkIne3pLO8NjH3XAOCdjrPJU15//fUaR42t6o4V\n/f6rjldRLbTG8Xvt5iLV2A59LkSLvswzI2OB6ywMW2WprZ9D9eutOV/LddkunT2h014fPHiwc5xa\nBDrNu3u3bZb3+NwzQ+89p3N32VFL6eryVcOuev9bb721xnqPXL16tffz+P3IvFOg52Tupfid1f5p\nae6z8As6AAAAAMCEYIEOAAAAADAh1lXi4izeslstmePG2r2NKWsWGcUYWct6W9vNK3trKbnsb6Xk\nJCRjxqlV/1DLzbUaD7cdG+vTPtXttYzl4jzHFgCG46zcVKKwd+/eGsftdZUGaOzka24eczaypXSl\nCfOa+1ryE8XV4bJ0uoyksayMPM9ZPLba5+ZjZxHprttJZWKdaiWpGTFVnuGeaa2+cveSkzhdvny5\nt73ORrKU7jXecsstvef/+Mc/7m2vymPUhjKOh2bz1LboeOi1KiqDUVrXoW1s9W8LfkEHAAAAAJgQ\nLNABAAAAACbEJFxclFneIM5IRsY6k8xT7rIeEomhbiaZt7tbdYwhW/eY65jF1WaoU1CrDicncfKT\nlpNChjGZS+clwQKAPHv27On93LmzaKbDUrpb/VlZXN/xrfki4zSToTU/ufYOzTCqsoI4z2b6wcXO\nnWOWbJGZbJwqK4qZK3fv3l1jvabNmzev2i6tY+fOnb3llNKVyDiXEkX73WUrjW45Wqc61ajjzaVL\nl2qsUi9tn8p84nNTr9eNp0pZ3H3YkobqOTpu2t4h8As6AAAAAMCEYIEOAAAAADAh1lXiMs9kJkOl\nIZltvlYdjrGSmLWQ1JSSu97MNl82iZBz6HH1ZZJOtMgk2ZnF9WWM/Mhtg8V2ufvKvd2fdT/KbAFn\nkidk+wcA5od+/9UJQ7ftWwl+rly5Yv+2gpsTVR7jXD9Kyc3VmWdGVkbn5mM3JzqJQ6tNGYcWLcu5\n7cyCSlZUgrFp06ZVjy+llF27dtVY5Rkqi1LplJPktPpH+8QlJFK3lSi9WkElH1u2bOn8TcdtYWGh\nxnq9KsPJJBRqXZO210mhnLTH3QuxXL2mOG5Z+AUdAAAAAGBCsEAHAAAAAJgQ6ypxGeoOMYsUIXN+\nNknBUCeUrBxEGdoPs2ypubZnpBNZ95PM2/0ZCUYcj6HuJ3qMbjfp9loW1z+65efewm/dV67tQyVH\nLXcgNx5DHR3m6XgDAJ5XX321xiol0Fi//zH5iW63u61+/dw5VmzdurXGcd7UuU//5iQAGZcSbVMp\n3XnbzSvaJ26+aklRtO8yc7uix+g8G5PcaJ3aVy4xjh7j2hefK1r/tm3bej/X/nVt17bG8dDnhI6N\nOqyoa4wbc42jM5HKcFSqtby83Nt2J7VRWvITvaaM/MS50cS6P/GJT9T4n//5n2us1zQEfkEHAAAA\nAJgQLNABAAAAACbEhklcdHtFtz7085Zh/tAkMsosriFODjBUrhDRNuoWTkZmMsub8I6hiZ+yDG1v\nVibkZB96j5w7d67G+ra7S/wR2+gcEzIOKZHMPT424VJGNjT2XgKAteGDH/xgjd08qAle4nyjW+/6\nNycNcE4hKnGJbhsqE9A6VJ6hjjBOntGSFbgkbW4ec7IblWDExDhRjtJ3vpNOuPlRXTtK6coinBxI\n0fa++OKLvfXpc6yUUi5cuNBbrpal7dKx1TFwz6fYdtc/6tyi1633gksiVUopFy9erLFer46Tk061\nJECKcydyZel1OInL4cOHO3Xcd999Nf7qV79a41kTfPELOgAAAADAhGCBDgAAAAAwITYsUVFmS72V\niCXrbDGmjozLhZMVtOrIbNVkzPezshZXrtvmmcXFw9WRaa8eE9+8zvSplqvbT7pNqNt5f/iHf9gp\ny53vjnHyrFaioqF96sqdp6xpluNnSf4EAKujMgyVn+hcoBKDffv2dc5XmYGer9ICnV9VPuAS9MR5\nwcli3JyoOMlgrHvHjh01dslwtB/c3KrynCg/UamPtl2dbZzURvtZJSNHjhzp1KFt1PlRz9dY61bp\nxEsvvVTj8+fPd+pYXFys8aFDh2qskhEncXLS0Cj/0b+55EZ6jo7B1atXa6zX2nJrc/eik5no561n\ncAa9x/T7qO3VPtT+L6WUr3/96zXeu3dvjXU8hsAv6AAAAAAAE4IFOgAAAADAhGCBDgAAAAAwIdZV\ng57RkDs7u8hQG8KxNoJjNLZZPbKLMzZ5Eaefc6hG22Vji/ps1e45yy2nX9P2ad2qAYt16hhoHdq/\nly5dqvG//du/lT4eeOCBzr/VuipzX2X06FFH58bT6efc9yOrsZvFCrKPeercAcCjc5/TkCvRqlD1\nvqpndrpj996Qsy2M7dL69HzVI2uGSTfXteyUFS0rYyPbmh91Hty+fXuN9XpV763H33XXXTXWZ4e2\nr5TffJ9qBdU2qy5b0fcLbrnllhqrrrmUUvbv319jtTTUtmj/6HU4jXVst9ahfaXHXb58ucbumvR+\njWsS91xztp5KJjtq/LfWr98VXT/oMTt37qzx2bNne9sR65/FzjvCL+gAAAAAABOCBToAAAAAwITY\nsEyiSta2MLOtlamvhW6XzCvrYmyHkyyonKO1HdRH6xi1m9ItGP3cZaFrXavrX90mdFucbvzi9pq2\nRfvEjdPJkydrrNtVimYYLaW7hadbatkt2RWcHWZE+9ptuymZ700L19dZW1AH2UcB5sdHP/rRGjtZ\nihK/p3v27Ok9Tr//bp7PzEOldOULzipR69A5WOtTOUbM7KzXpZIabaPaJrrMlRrH+WlpaanGKs/Q\nZ4xKdVwGTpVHRHmmSkg046fWrf2p9TnJqUotSvG2h5pJ1GVtddLSKGXVtmtfKVq3jqeTV8bnvF6j\nGwNto7ND1Pridet16fku66o+z5eXl2ussqZ4X50+fbrGem842c9q8As6AAAAAMCEYIEOAAAAADAh\nNkziMjSbYvx3Ru4yiywlI7cZmkHRZacsxW9fDs2UGstx0hnnkqPbPC7LWqzbuQAouv2UkYzE7dXM\n2Oq133bbbTX+8z//8xrrNcUMfEPvS1d3S9ai22Wur7SvnQtPVn7i2piRu+DcArD+6Ha+m1da80jW\niaXv+KyMzj0PnNRD5Rx6jGYLvXLlSqcOlSk6Fw+VDGifqARE++3111/v1KFyIHUm0WeltkuvQ9uk\n86lKakrxz2CV/biMmPqM0gyl27Zt69Sh/ajyE/3cjZlen/ZnvA59dqpsxMlw3HOltQ5y2dUdmXVQ\nXEtoP7g1kfab9oNbo0SpjuuTKH/Kwi/oAAAAAAATggU6AAAAAMCE2LBERZlj4haFOz/jcpLZgolk\nXC60rIwDTGxvRsri2qHbKy2HFd3i1DeQdZvnJz/5SY0ffPDBGus1xTeRXSIGJSMH0XbEscw4GOgW\n5+HDh3vPVacWjUvxb6a7+sbKPHQ8dOsrk1irda8PlUU5cGcB2Fjc9z+z/R+PG+rcpHN+S2aqc5G6\ndel8rEl23FwSE9DcfvvtNVbppT5vMglhVFKj0sdSutdx8eLFGuuzQM/Xa1LJh7rMaFyKT6Z36623\n9rbXyZLUQaYlo1lcXKyxSku1LFeuti9KopzERftk6Nou4mQ4GUmntsMlICrFO9U4icvv/d7v1fj4\n8eM1VglWvHe1LJUjzfoc5Rd0AAAAAIAJwQIdAAAAAGBCrKvEJcMsyYUcGUeW1pvwQ2UxLccTR0ZO\n4BL5tOrQcnVbTJ0CnnzyyRo/99xzNdYtON0ajG9ha1vc9qNul+nncWtoKJlkT7qlpden21jxOJdE\nSGOXPKm1TajbcG57Tck4LLQSYGUTj/Qd32Ke308AGMbQ7f9549w2hkpn9POYqEjlMirJdHVom3Te\nbfWPnnPgwIEaaxI7dW7R+VtdX1TWos/WUrrX5Vy5nHOXXod+HpNRqQRIZS2ahNCNmR7fSozVktj0\nXYf7vCWJ1eOcjMolQ1Lc8zuWpah0V8vVMdBznftNrNOtiYbAL+gAAAAAABOCBToAAAAAwISYnMSl\n5X7iXCocmeQL8ZiMzGToG8tZGY1z65jFkcO5nGhZTz31VI2/+c1v1lglIOqKouX0/buvjdkEGCtE\nyYfb+lJUoqJvbuv2n77hHl1bdFuslUxhBde3b775Zm9cSim7d+/uPT+TRGjofR/rcPeVS4DkZDcR\nHF4A5sfVq1drrNIJJxNsJadzW+pu7nHyzDgXZOQHTvqSfRa4ecnNiZm5svWcd32l52s/OBlElLjo\nv1UKqeizy8lzVEaxvLzcOV+fM3q+tjGTnE7j2FaVy+g1OfcUvSaNtT+j3EWfu1qHXpOe49zPlHjv\nZtZEKl/R5Fa6lnBJuSJZeU8LfkEHAAAAAJgQLNABAAAAACbEJCQuWZeKoTKTodtj2XYNTfzQkupk\ntgDdtmbLqcO9Ka7o2/KnT5+usUpf/uRP/qS37vhvt53o3oTW7Sfd0opbRrrl5OrWftCtK014oXHr\nvnLbjNperUPjF154ocZPPPFEp46/+7u/672ObPKOvjZlpVPuGCeXmUW6grsLwDi+853v1NglwFEH\nEU2EEv+tbhvOTcTNBS35h5OfZJKpaR1uDo3HKZlnZaacUnIJ9Nyc745v1acSB9dXrt9V/qFJlUrp\n3g/6HHVrA1d3azz0/svKH/vqa6271K1HXWNcosTokraCc8UppTs++v3Qtqj89fz58zW+4447aqzu\nN2fPnu3U4eS2s8Iv6AAAAAAAE4IFOgAAAADAhNgwiUtW1uIY6qSiZJ0wMjIBd3zWxcWd77aGMkln\n4jm61aJOAVqWSlxOnDjRe3zcUs2Q2QLUrai4LeWcYnQryzmT6NvnWk4raYDbRnXbs7p9rMf/93//\nd6fcz3zmMzX+4Ac/WGMdG23XLJKRodIpR2u7eux3CgD6uXDhQo3VrcPNj61kaPo3nQc1Vomjk8fE\nxDQ636nbR0a2oU5aWu7+/fs7dTgZhZOfaOxcZlpzoPaV9omer/KRTMKk+G8nLcrM03qutq+UrvxE\ncQkNXX0qo4lzuWu7ewZr3fp8U8cZlYnEspz7jV6rfq73kt7fMYmQS0io7dK6X3nlld6yDh482Ftm\nKd17XNsY25KFpyoAAAAAwIRggQ4AAAAAMCFYoAMAAAAATIjJ2Sy29NZDtbQZO8TWOUMzeGY160M1\n7O7cbEZUl13zj//4j2v8j//4jzV2mdJaemTXV6q9chpr1azFTJ7uftDjXF+pPsxp5GK7MveMXqvL\n2Kaa/lJKefbZZ2v80Y9+tLesTNbU7H2susGh91VLF5nJfAsA43BZOp2+u5TunJjJqJyxEYzZMXW+\nc+/fKGqZ99prr/W27/d///c75+zcubO3Djenub5qzZXub1qfPrsyFpEtG+Js5uy+z/UZHOtwz2A9\nx83/ekzLPtH1qbsXXVbRVpZu1YHrcc4K0j03ldjner6zQNSy9HiX5ffo0aO95ZTS1dnHdzmy8As6\nAAAAAMCEYIEOAAAAADAhJiFxGWpnOEtZ7phstrEMs9g3zisDY2yrk8LoFo7aAGl84MCBVc8tJZcF\nTT9XaYmTq0TcdWhZGWst3WqbRe401HIxbqEtLCz01qFbi247T8tqSUncFrAy9N5rfQey3yMAWJ2M\npKIlRdDvYybjo8s2rfNCtIfTv2m73Fyi2/wqY1C7QM1cXUopu3fvrvHevXt7PxYQN80AACAASURB\nVFdbSM2mmZkDI+7Z5eb8WbJ5KxkZjrM2jM8VJ19xz0GtW5/5rczjrn/cParSGXePxOdFxhZS0TWD\n9pVeUxz/aInYR+Y+PnXqVI2jDbTWoW0ZmoF1BZ6qAAAAAAATggU6AAAAAMCEWFeJi5MrzFPykXGE\nccfEf7tt+4wrhmtTxMlEMplWs+U6CciVK1dqfOTIkRp/7nOfq7GTpcT6Xf+6LcPslo+73sz94/qg\nVYfiMte5OtQtITrFPPnkkzX+m7/5m946XGY+J4Np3XsZZ5qhjjWxfgCYHzpnuPmx9f3PPDudK4oj\n+4xRdNtfpSwau7m1lG4WVZ1TNbOjumKo64vKXVwc2+JcWZRM1sx4HRn5ScahpzXPZ55xmWeBHhMl\nLu4ecGsDV4fL+FpKt9+13EzGcFduHA/nWpRpo/uuLC0tdf6t2YD1HtVn+xD4BR0AAAAAYEKwQAcA\nAAAAmBAb5uLSkpk4hiYOytQ3T0nNLNekZN5Ad1ta8XjncqJvI+v24Sc+8Ykaa3IKffNet0djua4f\n9Jq0TVmpjpLpE/d5K6lOJkmPO1+v6fjx4zXWLdSInqNvvLt+0O241lak2/bLbJ3OkgAp8zkADCcz\n98TvrCZQcW4W6n6h80pGDtpqV2aebyX1UZxbl6JuJioz0HZorH1TStcRRl3LVC6zZcuW3rZr37Zk\nEO756Bxh3PEtuUsrAV8fTu7ScqPJSEjVvUSdf3ScWi4qurYYel+6NVHrHnPHOTcaHXNta0xApO5C\nOh6Zselt50xnAQAAAADAmsACHQAAAABgQqyrxCWTkCi71Z7dApq1Ta3jxkpZxshq3FvfrbeiFT1H\nt/M++9nP9p7rtvNKGZ4Yx8kgxkqOht4/rWMy95VuE6sE6OWXX67x3//933fKfe6553qPO3z48KB2\ntFwcdHwyb+47iVS2f8ZIzgDAM4sUTeeGKEdcrSw3T8+SfM85XjlHjlhHZg53ZTliwiV1ijl79myN\nVd6psphbbrmlxup4pv0c2+2ej+76nEuJxjFRkZaVcQrJJBRskel3bZM+K3UMWmOWSbKUaW9cb2j/\nuOR/TiLlpEx6v5TSlbaqpOfMmTOrtrcPfkEHAAAAAJgQLNABAAAAACbEJBIVzcKYhClOJlJKro1j\n5S6Zt6eHJirK9odeu27Pue2x1nbn0OREStaRZRb506x1t87Xt7B1e+zb3/52jffs2VPjxx57rFPW\nvn37avyf//mfNf7Lv/zLGutb4669zp2nlNz943AOApHMfQkAw3Hf7YxcrVWWO845SGkdbk6KxzlJ\nhruO1rzupAxOOuPObTmc6L9VpugcPc6dO1djTZikcheVwcQ2Xrp0qcb6LHDP11me85n1ipKVL7k6\ndTxVzqHHO1cUdZMrpftMVdmIjlPG3aX1/Ri6lnD3sa6VWs49er1bt25dtb4++AUdAAAAAGBCsEAH\nAAAAAJgQk3NxyRwf/zb07d6s68t6bNtnnDQy/Ra3q4ZuP2bGoCU/yXyekeHEc12fZOrObu25e0nR\n7bzFxcUa69vZjz76aI3j1pe+3b1jx44a61vu7k1xjbNv3mf6yrkMtL5PYyRHAOBx32fd8ncSjFK8\nS4ZLFtRyhOo7txQvi1EpQibRUOsZrHXo9WYS3blnTJyPdR5186DKNrRNKne5cOFCjY8dO9apQ13S\ndu3aVWNNkqRyidbzvK99kcyzYaxE0T2jMmOg93G8d7W9LsmWk9ToMzS6qri2u89d22eRxWr/OFe9\n1eAXdAAAAACACcECHQAAAABgQrBABwAAAACYEBtms+g+nyWjYaYsJVvHUB3XLGTsBjP9E8vJWO45\nvVXWcq+VDa7v/EyWzmxWuaGWUi0y9laqIbvttttq/PnPf77G27dv7z2+lFLuvffeGt955529dbg2\nuTjiNKWqhctYUmW0kK22A8BwnN5WbQD1uxyzY2Yse4d+/+MzImPf6ObTjH65VZa2RftH51pnxRdx\nOnc3Bhorqn9eWlrq/O3ixYs1Vt365cuXa3zzzTfXWLXp+s6S67dShttNZ56brTLdeLrMtVkbajeG\nio6B3ofan/E74dqibVfdunvfa5YM9/N4PvILOgAAAADAhGCBDgAAAAAwITZM4jJ0qy0e5z7PSCqy\n2y5D5TJjGZuhVHGyBiWzBTNWDuTkK24LdpY6HJkt2FYd7ji3HdfaptuyZUuNdXtVt+QyVodj2z7U\nkjTi7MmQuwCMw303nfwgyh0ytocZaWhLUjfUhk6lD9k6tCyX5TNe+2rtiGQsJt3xzkovWkpqe5eX\nl2t87dq1Gp88ebLGmon6jjvuqPHBgwdrrNKXWL9aDypD5/mzZ892/q0SELUIduVqm2666abe41tj\n7nB20XqPufsikrGhzmY7VzLP3SHwCzoAAAAAwIRggQ4AAAAAMCEmnUk0m7lyaB2tbQx3XMZNJOs4\nMk+nGcfQbc3M2+DZrJJue3YWmdAYSYbrg+xb6pkMrG6LMb7178p1sh9tu8plnJtALCszBmMyvrba\nCwDDUUcQlcG5bJxRRpdx0lLcPNSas11Z2paMI0xGrtKqz803mQyhrTYOzbTZug6XJVpRuYvGCwsL\nNVbpy+HDhzvn33LLLb1tdBk13Zytn7/44oudc44ePVpjdSpzzxLnhOKypkZcWZk1XGvMM45Ark/G\nSpyz0psIv6ADAAAAAEwIFugAAAAAABNiwyQus7iUjJGDzLJFkXGHyUhiZjG5H5rAKFuukhmPWRxW\nhkoqWjKWjANN5v4ZKxnK9KFuRbtt6VKGvzWedWTIOOZk2qTEbXQ3ti3pDQCsjvsOZaVoGUcp56ri\nHFayTmrOXUbnLieDac2VSiZhm9bdcjLLOOO4Otz8HftKr9c9G1wd6u6lSY7ifKz1X7p0qcYqhdEx\nV+mLSk60HQcOHOjUoQmUlKESWSddibhxdverK6slM3Xo2DgZjJKVL2fv8Qi/oAMAAAAATAgW6AAA\nAAAAE2JdJS7KWNeHoTIKpSXbyEg6XN1Zx5GM3MYdMzbRzFBmqWOo5KT1udt6HSr1mSUhVSbxj5Of\nxG23zNvzQ2VN8XgnORl6H2usTgSxrFbSFAAYhpOluOdblDsobo7R77NKHGaRq2n9maRlbq5Tx5qI\nm/8zDimtOcldo5br+lePadWh16XHqcxEr0/b5CQRMRmRSmE06dHi4mKNNVnQzp07a3zzzTfXWGUt\nenwp3X5wTjwZ+ao7N6LX6CRLGaeXODbav+756O63TGKjiI6hSyK1GvyCDgAAAAAwIVigAwAAAABM\niA2TuIxlqKwhswUTz8mUm5HaZN/0zXw+i4vLUDIJiFp/G+py48qMDHXxyY5HZswzW2Kt7a7s9l4f\nWTnQ0MRMek3umNY2oW71bt26tfd8AMjhZA2zyMfclrzWoeU6h5XWnKbff5dYbagzWavOTLk6P6ms\nIJapc7CbB13SI5WuaNxKjOPkRBnpg/azJjOKf9OylpeXe+Pz58/X+OzZszXet29fjX/84x936vj0\npz9d4x07dtTY3VeK9tubb75ZY3WcKaXbjyrb2bRpU423bNlS4+waxeGSKblER1mZqUt6NKscmV/Q\nAQAAAAAmBAt0AAAAAIAJsWGJijJuJC25g9uicsdktxiGJvUZenyWzJvpmeQAsS3KUJlJSw6UcUIZ\nK8mZl2vNLAmXMm+sZ50QMkkPMnKg1vhntqnd59qOt956q/M33crULcunnnqqxo8//rhtFwAMw81J\nUVbgJCCKfudVEuHm6Sivcc+lzDPRuT61kgiphMPJcxTtg9YaIfPsdH2icgxXXzwnIzPSz51U5/Ll\ny506zpw5U2Odj7VdKq/R/lxYWKjxsWPHavzKK6906tizZ0+NH3jggRrv3bu3xupM48ZM+0OTJ8U2\nZqQsmWdX675y6Bi6e0yvL97rTiLTclxqtmemswAAAAAAYE1ggQ4AAAAAMCE2zMUl+1asMlSyMtT1\nI9vGDK0tv6y7yGqfZ51JxkhLWnW4JATufNe+TDmtczJvS2f7I5MMaa0cdjLbdi6RQut8xZ3vkjjE\nREW6ZakJLZ544okaI3EBGI5LKOQkJy3JYWa+c5IKrUOlB/G4+Le+chU338TjhyZ1c4mOWonUVDai\n8gNtl5arfXj9+vXe+mI7tH/c80rH2c3BKjO8evVqp44LFy701qGxyk+0jW6cdu/e3fn38ePHa6wu\nMprcSJMeqSOMSllcO0rJy3VXGOuQMjRZZDZZmPtOZZN/RfgFHQAAAABgQrBABwAAAACYECzQAQAA\nAAAmxLpq0DO2fi3N89BMlLPYN2balWEeWaSG1J21DsxoqTPnZtuS0Whn9WdjMrC2xmDMWGU0a6Xk\ntO2ZMWjVkbHHzNiptWzWVDO5uLhY40ceeaS3bgDI4SzeWrpzJWPT6r7bTt+tWR3jOU7H7bTQeoye\nG69JtduZZ47OSXodGrds7pztoZYb38XJ4PrdPfucLr81Hztts6tDY9Wz6+dRH651XrlypcaaoVSt\nGbdv317jW265pca33XZbjXft2tWpQ+tXu0h3Xzqy72hl3vca857jvOAXdAAAAACACcECHQAAAABg\nQqyrxGWoTeIs9oRD7fuydbhzZpFUZBha1iz1jZHtZOscaz2YYag95Sx9pdcx1IqxVacrd57j6drl\nsp61sqbqVrFmkvuDP/iDwe0FgH5cVmKNYzZGJy1xForumeis4uI5LjumzhHOArGFk/q4LKpu7nL2\niZGhzyXXb1EG4+QWTsLjJC56fZs3b+7Uof2rdag1o7ZLJUt6zM6dO2usVoqldCUnsf4V1HpSpS+a\n6fTEiRM1Pnr0aOd8tWx0WVAV7Ssng5llveLGbKgN5LzgF3QAAAAAgAnBAh0AAAAAYEJsWCZRJeNe\nEY8bKndRWlsXY+QrszieZLJjuvZmnUnm5dCSlV24cmdxbnFk75kxZblyM2+AZ/82RhbVyiSYPaev\nHdnssZpJdK3fZgd4u+OyhzrJWfzOOScVJ6NzTjFaR5TRZJ6PGTeplhuNk/RonOkfPT72VebaMzKj\nFs5FRuUrOuZOftjK7Krlan1x3FZQiYvKVfT4mCk1c+3aLidr0qynFy9e7Pxt06ZNNd6zZ0+NNUOp\nOr9oe926KbvGyGQMHSt3mfX5yC/oAAAAAAATggU6AAAAAMCE2DAXl3luE8xT7jCvulsMTaY0TxcX\nJ52Zpe4xbXRv52clR5ltqSzzuk820sVnvcrK1AEAw3HyAZUxxEQ1irpyqMxAz1F5haLHOClJKX6u\ndc4tGflJnPMzjmvaXq074+hWindS0bqd5Eivr/XscZKezHPQyWuiw4r+TetwiY70eL0OdWrRJFKl\ndPvBuQg5p6DM+JdSytLSUo0vXbpUY5cASeUuKolxMphYv+L62p3rnIIiWSlUC35BBwAAAACYECzQ\nAQAAAAAmxLpKXDI/+bfeBh+aRMidm5XXjHF0aW35D3WgGfq2fLaNY8/NuNk4Gc0sb1hnGCvnGCqX\nmcWtJ+OY4hIgjZWSZL43WaegsYmVAKCfTLKeOIc6KYyLtQ6VNWgdcQvfyVdcuYqTdrSSITlnGieX\ncBKO2CaVdGhZKn1xTiharsat9Y1LHOX60Lm7aLvj31xfXb16ddV2aJ9HiYvWoTIqRftQ5TXZ57+2\nXeu4fPlyb7ywsFDjLVu21FiT56n0pZRStm3b1tuWHTt21DhzX7lkXfF8J8MaAr+gAwAAAABMCBbo\nAAAAAAATYl0lLkOT50QyW+pjJQpDExK1ynKMkW3Mcox7A9m1Y+zbx5nEGmOdV7KJdcaQuUez9Q0t\nK9MncfswK02ZdzsAYL64752TvpWSc6NwMogbb7yx9/joGuPmdsXNxyp90M9jYhu9Rq3fSQsyzi2t\n61CcZEST+nzqU5+q8Z133lnjb37zm52ynnvuud46nOtM5rkb+8pJKjQhkSYeUjmHk6KoZCSef+XK\nlRrrGGjSOi1Xr7Ul83Bj7iQnKoPReHl5ucanTp3q1OGcX1ROpG3Ua1LGyqWHwC/oAAAAAAATggU6\nAAAAAMCE2LBERRniNv1QWUvGTaRVxxiZgDLL9kamr7IOKxmZiZKVHzm5zCxbQENxYzvUbScyVE6U\n7cNM/UNlX62+zSQFy7gUzeMcAFgddRBR3Pc/Jh1yTkyuLJUGqMRFt/nj99pJEZw8w7m7aNtVYlDK\nb8o4+tqi9bl5SOuI/eGcbVwiJ3VPef7552us8g91S4k4l5yMlKnlGqK4/tH7yo1tSzrlEho5FyDF\n9WcrqZPei3odTorikktFx5nFxcUanzhxosYHDhyosUqA9u3b19smdfdx9+q84Bd0AAAAAIAJwQId\nAAAAAGBCsEAHAAAAAJgQ66pBz2QrnCWT6NCMn9ksn0PJar2HWgQO1Vu32jWGsbrqjF57ntk/Zylr\n6D0zy5ivhc1mqw6nRyf7J8B0cDpeRb/L8T0l1fs6e0KX3TA7FzirPDf3qSZY2+v09vG4qE/va4eb\nH51dX/y307Nrf+oxzzzzTI3/93//t7dNsY6M9XDm85jd1OmvVRu9adOmGqsu22WejWOjf9Pz3TPG\nWT86+8VYln4PMplvtb7WPa06ctXSnzx5sveYV199tcZqy6jHxOyku3fvrnEmK+1q8As6AAAAAMCE\nYIEOAAAAADAhNiyT6DzLGirhaLVjrE1jpk2ZrI3OTsnZIWUlFZkxyEqOdNvPbee5NmWuNcta3Vfz\nPH6MrGWsleM8rS6RxQCsDSpRcHOlzrnR4s1lbXRWcC0JiENlClruUItYrTse46z53PU5eUVLzuf6\n1J2jx6jEoZWtVNFynUwp01ctGY2zmMyUm8kQW0pX0uGsDp39Zkuepeh95dZgTmrVkom5cjVWq8yl\npaUaq0Xj1q1baxylKzt37qyxyl2i5WMWfkEHAAAAAJgQLNABAAAAACbEukpclKFSi7HM4rahOGnJ\nUBeW+O+hb3S3ynXtVWaRTrgyM5lEh0qDZiGzPTtLJtFZ+n1eZOqI4zHWwaavnFYdALA2ZBxAIk4a\nojhpgJMPqHtFKV1nCpXkXL58ucZO7uAkMa1nsJNFZCQ1s0hOMpIMdQBxsptYv167c41RXDbumLHV\nyXOcfMU9j11m1dgWHXMnd8pIhmaRXWbWR9lyday2bNlSY5UvqSxFy33jjTdqfOnSpU65p0+frrFK\nYTQeAr+gAwAAAABMCBboAAAAAAATYsNcXIYmfmn9LbNNNNTJpMXY8zMygaFbQK02td6YH8PQ7bn1\nSEg0i2TISZbmKZ1x52S2arOyknklbJqnHAgAhpOZh6JcwcktnHxBt93VfeKzn/1sjR966KFOHSoH\nOHPmTI1ffvnlGj/xxBM1fumll3rrbjmsuGQzLeePFdTFQ891Tjal+MQ6SqZulT7Espzbi5OfqHxE\n+yrKaPTf6iii52QcVloyESfV0b5WOUjm+mIdbr3j5EcZF7g4lpnkmHp9Gut4aKySmNhelb9o0qMh\n8As6AAAAAMCEYIEOAAAAADAhNszFxdHa2nd/i282u/NX+7x1XEbOMcuW/1DZjyNuwbW2k/rKnaer\njnvbfhYZxFCZSWYrsnVfZZxthtYXzxnjpJPFSYscQ114Yh0AMA6VK8xTXnf77bfX+P7776/xww8/\nXGOVR3zoQx+qsSamKaXr1vLUU0/V+JlnnqmxOlmo44mTmUTZhrsufc675D1uvoprBDfHuXIzc10r\nkZ+rO5PgTz+PiXic3EL72klOnNtKvFbXXpfoKuNAE+tw7kKu3/W6nVQnK2V1uERVSpTROMnRtWvX\nVq2vD35BBwAAAACYECzQAQAAAAAmxIa5uCgZJ4v478zW0FB3j2y7XB3zlHDM0t61YJa6M64Drt9m\nuT73Rnf2vsqUO08p07yI930mmZYytN9a5wPAOJzbhiM7p9xzzz01fuCBB2r89NNP19glHTp//nyn\nrB/+8Ic1fv7552usEgDnQOLm0DiP6L+do4uWqw4iTiYUpSGKczbR63BzXUu2oe3VdqkURc/RNjpH\nnigF0XHTflBpUWZ95Bx24r+dXEbbMVQmGtvuJLLuO5FdF7q+dlKdrAONosfpOG/evNme04Jf0AEA\nAAAAJgQLdAAAAACACfGu9Uww8tOf/rS3silJEZwMw7VrqNNLZKiMJuv0MvQ61irhztAxiOXMIr1Y\njdgO11dD68uO85h7NCvJmuVezJD5rt59991oXwAGcujQofolcomG3PZ/KV3njhtuuKHGzuVEYycN\nyc6VWrdzv3BOHa05bajUR9uh9alcpZScbMM5d2jfat2tOtycr8fo+VqHu6aIc7ZzfTjLs9W5sjip\nj+vbbF8priyV8yixr7TOzHhk1lDRmcitqZaXl2v8/e9/P/185Bd0AAAAAIAJwQIdAAAAAGBCTDpR\n0Sxk3CtadWSS+oyVWsxLqtEqZ0xyIlduLDPzdvhaSSrmyVA5yFinmQxuqy27LZm5poy0p+WkBADz\nw8kH3DFx/nWSDic50O15dbVoyfxUeqF/c64heoyTGMRrVfeLjJOGxuomonGUeTjXkoyUwbnJZJPW\nOZy8ppWU0ck+tL2ZZ35L2uPWEm489B5x7i7xvtLr0OPU3SXjqqPnat2xjW7cnDONa2uU6jhHoFnh\nF3QAAAAAgAnBAh0AAAAAYEKwQAcAAAAAmBDrqkEfa6focBmx5pnlc6j9Xuv4Wewfh9aROce1ybUv\nauwyWrx5Xes8Gatrn6XfnXY8owmcZybZTPbQWbKjklUUYByZ905aOmenW1ZUp575/sdytI5oMTek\nve6ZXUpXz+4sFLVcpwnO9Ee2XW7eVLLvaLn+0bHRtrfGw+myM1lp3XsKsd2ZDOBu3eXqi/psZwWa\nuY8d2eyxirs+l2E2Ms93EkrhF3QAAAAAgEnBAh0AAAAAYEKsq8RlaKbElsWb23Zx20/zJLO13zom\n0w+ZbaUsQ7NYziJxGPN5JmNX65yxUo0xjM3AmmEWadgYedbYdgHAcFxWSmelF7//mWdf5piWfECt\n51SKop872YaroyUFyFjr6ecuA2skI2VxMhNXd7yOjMWkuyYnH2rV7+RLWrezU2xdq2uLSkjUWtGN\nZ/bedRKSobLPrCTXjWFrbFeru5Tu96Mli2nBL+gAAAAAABOCBToAAAAAwISYRCbRWbbKM1INt0Ux\nTxlMRrYxtlznsOK2QVtlzVPukNn2GyqpmUUOlHGgcce3jhvj3NPaXnOslTxnXuMEAOvD0PkiovID\nzczptvazLhWZeVvruH79eo1VKuHkGKUMz0TqytLPVYIRy1L0OaqxSngyji6xTievmOfz2GXEdG4k\n7jrisyuTdVP7U2PNBNqSNem9kekft7bLSkmcNCizRtF7IbrR6JhruUtLS6l2RfgFHQAAAABgQrBA\nBwAAAACYEO9iCxsAAAAAYDrwCzoAAAAAwIRggQ4AAAAAMCFYoAMAAAAATAgW6AAAAAAAE4IFOgAA\nAADAhGCBDgAAAAAwIVigAwAAAABMCBboAAAAAAATggU6AAAAAMCEYIEOAAAAADAhWKADAAAAAEwI\nFugAAAAAABOCBToAAAAAwIRggQ4AAAAAMCFYoAMAAAAATAgW6AAAAAAAE4IFOgAAAADAhGCBDgAA\nAAAwIVigAwAAAABMCBboAAAAAAATggU6AAAAAMCEYIEOAAAAADAhWKADAAAAAEwIFugAAAAAABOC\nBToAAAAAwIRggQ4AAAAAMCFYoAMAAAAATAgW6AAAAAAAE4IFOgAAAADAhGCBDgAAAAAwIVigAwAA\nAABMCBboAAAAAAATggU6AAAAAMCEYIEOAAAAADAhWKADAAAAAEwIFugAAAAAABPiPetZ2bFjx349\n5Ph3vetdnX//+tf9p+txeox+/qtf/cqW63D1OVw7fud3uv8PylyHKytzbsSdk/nctal1vmtXpu06\nTn11roZre6vMsf2bqTtzjw4tN7Z1aLmufa7MbLlHjx4dVjkAlC9/+cv1y6bPjPe85/8e0+9+97t7\n41K63009Pz5/Vrh06VKNz507V+MXXnihxq+99lrnnJ/97Gc13rJlS2+5+vn73ve+Gt900001fu97\n39t7bimlbNq0qffz119/vca//OUva7x169Ya79ixo8ZHjhyp8cc+9rFOWS+//HKNl5eXa/yLX/yi\nxtpvV65cqfH+/ftrfMMNN9RYx6mU3JyvdWTm1jiW8XnZh/aVxtl1gfaJXqO7Pm2T3i9LS0u23dev\nX6+x3pdvvvlmjQ8ePNhbt3L16lV7jLbd9YPGet1vvfVWjd94443e9sV/a/0//elPa3z8+PH085Ff\n0AEAAAAAJgQLdAAAAACACbGuEpeWlGEFtx0Xzx8qE8nKHVwbMzID16asbGOMZKQlRcj0m6JjkLm+\nIe3qo3XMUNmGa7vS2hYcK4tyjO2HPmbZ7szU3epDraP1XQWAYQydQ1tzvjtHZTGujj179tR47969\nnfNVmnLgwIEanz9/vre+7du31/jatWs1vvHGG3uPj2i5mWe4Smr0c5W+lFLKsWPHauzkRCpxUEmE\n9kFL4pJB26jnZ2Wm2kaNFR1zV1br2eFkLYqThrhnhPbnavWvVoeiY6PymlK60hS9F1Veo+3Qz51M\nKF6H9rXeG/G4LDxhAQAAAAAmBAt0AAAAAIAJsWESl1lkHrpd4rZEhso5snIHZYybTJaWC8wKrbZn\n6h8qqYi4do1xv2k5rDjJyVDpU1ZS46QeQ++x1jljXGOy54yRTsUx1q06PWfWLTwA+P/5+c9/XuPM\n3BpdXJxzi8oB1PFE5QDOeUMdS0op5dZbb62xyle0PpUSaB16fUqUK7j5x7nZqHxFj1F3lq997Wud\nOrQtu3btqrG6tagbjc51TjISGboe0M83b95c4927d9dYnUxKKeXMmTM1fuWVV2qs1+eeNzrmLm7h\nynLj7I5vtVFjlaXo80bH7OzZs7YOHU/n9qJjq/WpA5Ee03rujZGcrsAvbQDzVQAAIABJREFU6AAA\nAAAAE4IFOgAAAADAhFhXicvQxDgtBwm3TbRWLhOzvP3cd3yLoQmFsmVl5Atj6xjTplYfZhJuZLYM\nZ7kvZkl61HduPG6MI8xQh5xZys2OzVgZFwD8HyoNUKmGkx9EaYhLeqNl3XPPPTX+3ve+19sOdZ/Q\nbf5SSjl16lRvudoWvQ73zNbEL1Gqo04sTsqybdu2GmtiI+e8ppKG2HaNtW6V56hc5ic/+UnvNUWc\n1MNJde+4444a33zzzTVWeYXKh0op5dlnn63xj370o976VLKkdWu52oet8dBztCytQ8dWY03ioxKe\nUrpjdejQoRqrJEvboWOg7dVx0nsktlddhLSNQ2UprbWEtjdeb7r8mc4CAAAAAIA1gQU6AAAAAMCE\nYIEOAAAAADAh1lWD7shmrszogN35WWu8oXW441v67kwb3fGZNrXqXyv9vGOo5n2WsRlqI5jN7Orq\nyNyvLbtIV/cYO8TWcY5MtrrsvUtWUYBxqF2c6mp1vnLa9FK630fVvKoWdmFhobc+tYtrZY50doyu\njRqrJjtrVedsFrU+1clH/fQKcR5TLffFixdrrDppteXT+lT3rX0Q9fqaBdVZM6r++SMf+UiNFxcX\ne69DdfGldG0Fte2qKXcZMTVLrNo3xjFXvfadd95ZYx0P1YR/61vfqrGO+Yc//OEaP/bYY5063v/+\n99dYtfh6H2tZr7/+eo11DPSaVL8e61f0HtPx0PcWXnrppRrrvaP3SCldjb5q6eM7EFl4qgIAAAAA\nTAgW6AAAAAAAE2LDJC5Dsz/G48ba3q0Qt+adHd9QiUJWfpDph4xcpVWHY57ZTpWhGUa17njuUMnS\nLFk+h5brzlWyEhcnE8lYirbIyE8ybWrJzLBZBJgfKrVwtDJz6va8Shx0q//b3/52jTPf3ygZ0X+r\n9EHbojaNztpOpRbxOlSqodekVo4qa3DPU21rvA6VJqhU47XXXqvxvn37eo/JyHYiTkKk57/66qu9\ndZ84caLGcS53GUMV93x18qGI3nMqOdHr1THXduh1axz76u67766xWky+/PLLNdZrVZmJ3hcqMfnk\nJz/ZqePee++t8eHDh2us3xXNjquSpa985Ss1fuGFF2qs8rFSuve4fu/uu+++Mgv8gg4AAAAAMCFY\noAMAAAAATIhJuLi4rZm4DeK23p0byVApSTzOfZ6RsmQdPTIZJod+HslIHGaRD7lrHyPPyDr3DD0m\ne37mnslIYmbJ8unu46yEx21fDr1fldaYt7YsAWAYzzzzTI31+6tb5bqFfubMmc756sqiUgTd9lc5\nwAMPPFDjnTt31jib2dm5rzhZSsutRdE5xkkn9JoU9+xpPSM0Q+n999/fW59zAWtJTDLPQe0rdW7Z\nv39/jVV+EjOJqhxIY9dvOh4atzJt6990DN26QvtEj3fPjlK616UZQPW58txzz9V4aWmptz69bj2m\nlFK+9KUv1fiRRx6psX5XHn/88RqrDOrIkSM1Pn36dG99pXTH0GXBHQK/oAMAAAAATAgW6AAAAAAA\nE2JdJS5DnVBaW23uzeSMTCDbRlfuUPeKWWQbjqz8ZKjsJ1PfLLINZZZkPRlXnaEOK1lnksz1apuy\nfZW5X137Wp+7rcyh5SotVx0AmB8qa2g5kKxw2223df7tkgXp+THRzQqXL1+uscouVFJTip9jMnJH\nnUu0TdEJQ8tVJw1tu5NdOAlG7EMt99SpU73XceDAgd5ytR0qcYh9q7IhLVePc7LE5eXlGt9+++01\nPnnyZKcO51qjdei94Opurbvc2sdJeDL3SKzjBz/4QY2PHz9eY70vndOMJgHSOlTGUkopjz76aI3V\nrUXP0Xtf+/PBBx+s8YULF2qssrJSuk4x2q4oTcrCL+gAAAAAABOCBToAAAAAwITYMBeXjMtEy6VC\nGSo/cW9kR9xb52NlH0MlMpmtpKyMJvMmfdbRIyPPyNTR6sOhkpOhCYFadQx1dMmSkR8NbdMszJJ0\nKLstCgDDULcWJ11oJc9z5zjc3BUlJ4o+P9xz18n+XDnXr1/v/E3PUQmJcwTRa83O05lkSCpx0XZ8\n/OMf7/289VzJSHKcNERdRtRlppSudMKNh6LXqu4n2m9xjaB/037Tz1U6o0l5dGz37t1b49hXZ8+e\nrbE69Dz00EO9datESRM8qeREk06V0k2GtGvXrhpr0qsnn3yyxuq8ogmTVI7TQq83Izntg1/QAQAA\nAAAmBAt0AAAAAIAJsa4Sl4w0pLWFPyYBjjJLoiJ3TFYOMktbhpaVKTfT9kxio1a73NaeK1eJUonM\nOe7N/YwDTCnDnXjGSkvGkE2A5Rgqa4ll4uICsDZkksC0pHpjksJp3fN07lIyktFSum4dTnrh1glZ\nCZ47x/W7Sjg0kY7G0cXFSY4yiedUEqHlavKcUrpuL+56YzKdFVTaoUR5lJPkaKxOMyoBUimKtjX2\nlboFabnq4qLyFY31OrS+L37xi506tE/1HtM+VXmNSoAUPf78+fOdvz399NM11n5UadEXvvCF3nL7\n4Bd0AAAAAIAJwQIdAAAAAGBCrKvEJeNG0tpeG7r1NtTRZUhbhtSdJfMG+ljXmKHHt7ZNM32VSS6U\nlcHMy8UlK3Ea02+t7eehybSyW9dDt7gzY5Dtg42U/QC8HXBuJI6xEhf9zrs5KSMxjMdl5CPumFK8\ntMA9PzLubipdKKUrTdGynHOLJqfR41VS0WqvtlHLVamFkxzp51HiohILJyHKPHeV2J+bNm3qrUPH\nyY2tOgI5N6KIc/j57ne/W2N1dLnrrrtqrFKSxcXFTrkLCws1VonMuXPnaqzOLYcPH66x3i/adk1G\nVUpXbqPXsbS0VGaBX9ABAAAAACYEC3QAAAAAgAmxYYmKxsgVSsm7Tqz2eSwnkwhmjFtGRLeZMgkX\nsuVmZCYZZpGDOCeVWeQjbgxcUobMOGe3bTPjn71H5pVgKDserg43Ns5lIAsSF4BxqMuEkwzoFnqU\ngvx/7Z1Lrx1H1YaLCVLs48vx3YnjOAlGCkQomEG4CJAQEkJiwE1CwJQhY+b8B34GIySERBASUhyQ\nkCDgKJCLYye+kPjYPj62E8LsG7l4e9FvnVXd+xz3B88zWnvv7qrq6u7q0q6336WJjtQlo1cu0xor\n3djnEvY4Z5FWYjNXR+b56FBJRCnDflQHEScZ0TpURuESN8XftO0qt9Hv3fnX9h0/fnxQx7Vr10b3\nUWmJXgsqqdH69FpqJYjMyJe0vS5BU6xDt9MEQ7qPtv3ZZ5+tsUpwtNx4fzhHGZfsSb/XvtI6okOO\n9rWWRaIiAAAAAID/ApigAwAAAAAsCCboAAAAAAAL4qFp0BWnOWvpw1dls5jdf6e027368F6Nfbbc\nKUyxAntAy5IwQ8a+cYp2f057W3X0ZsHNaO9b5bhrbI7dY4uWphQAtsfZ5CnvvfdejaPGVnXHit7/\nquNVVAutcbyv3TPRWe45dJuYuTIzBrvxRsc3Z2HYKktt/RwuA2dEy3XZLuOxP8Bpr0+ePDnYTi0C\nnebdafenvGPnnj967Tmdu8uOWspQl68adtX7P/744zXWa+T+/fuj38f7I/NOge6TuZbiPZvV3Gfh\nH3QAAAAAgAXBBB0AAAAAYEHsqsQlY783l7llzZHFzMlCOYVsls+dYk4dmexvpfgluVWdp1b9mbqz\n5fbilpKzWVcz282x3wSA1eKs3FSicPTo0RrH5XWVBmjsZG1TxhiVJsyVkI6VM/Z5uzpclk6XkTSW\nlZHnObvHFtouJz/JyGidVCa2S60kNSOmyjPcM63VV+5a0vbqtXv37t3R9jobyVKGx3jq1KnR/V95\n5ZXR9qo8Rm0o4/Wi2Ty1LXo+9FgVlcEorePQNrb6twX/oAMAAAAALAgm6AAAAAAAC2JXJS5zs4f2\nMmXZze0zR+7QWsLLONjMzezY63IyJTtmbzvcNlnZTu9xTHHk6b1eW3U4OUlGfuKykE3JVppxAcpe\ne7st6QL4b+bIkSOj3zt3Fs10WMpwqd9JVuZkRy4l5zSTwTlLlZLLPu3GOpUrtJxlMv3gYufOMSVb\nZCYbp8qKYubKw4cP11iPae/evdu2S+tYX18fLaeUoUTGuZQo2u8uW2l0y9E61alGHW82NzdrrFIv\nbZ/KfKJ0SY/XnU+Vsrjr0Em142963rS9PfAPOgAAAADAgmCCDgAAAACwIBYncVHiUssUOcnY91np\nS6a9GUnMlP3nbpM53swyXzaJkFv2ybRjSpKb3mVQJSuj6b1m3DJYbK+7rpw7QNZtxbXXuQnMvSeQ\ntQCsDr3/1QlDl+1bCX7u3btnf3uAcy/Zs2dPjZ3rR9zHkXlmZOVxbrxxzxsncWi1KePQomU5t50p\nqGRFJRh6Ptz2pZRy6NChGqs8Q2VRKp1ykpxW/2ifuIRE6rYSpVcPUMnH2tra4Dc9bzdu3KixHq/K\ncDIJhVrHpO11Uign7XFxLFePKSZmysI/6AAAAAAAC4IJOgAAAADAgthVictOOZPMkSK0lux6nVCy\nchAl0w9O4pBdGlQybyNPcT/JSDgyb+G33op2yTucY4EuN+kSU1bO4fpH63Zv4beuK3ed9F7f2fOh\nS3iZe2WnHG8AwHP16tUaq5RAYx07YvITlSy4pX4nS9D7d9++fTWOS/M69ulvbtk/41KibWq1S9E+\nceNVS4qifZcZ2xXdRiWDMcmN1ql95RLj6DauffG5ovXv379/9HvtX9d2bWs8H/qc0HOjDivqGuPO\nucbRmUhlOCrV2traGm27k9oo0b1G267HFGVD25Wl5ybW/fzzz9f4F7/4RY31mHrgH3QAAAAAgAXB\nBB0AAAAAYEHsqsRFcctguuzSMszPyCgydWdxcoCMXCGLS94wV+6g9CYLyiaqydSXKTcrE3LXj14j\n7777bo31bfe4FOlwjgmur1pJPDLX+NyES3OSb5B0CODh8swzz9TYjYOa4CWON7r0rr85aYBzClGJ\nS3TbUJmA1qHyDE2Y5OQZLVmBttft7+SO2g6VYMTEOO4ZoPs76YQbH9W1o5ShLMLJgRRt72uvvTZa\nnz7HSinl1q1bo+VqWdouPbd6DtzzKbZd99H+UecWPW69FlwSqVJKuX37do31ePU8uWd+SwKkOHci\nV5Yeh5O4nDlzZlDHpz71qRr//Oc/r/HUBF/8gw4AAAAAsCCYoAMAAAAALIiHlqgos6TeSr7j3FN6\npS9TEvxkpAitOpxjivvevVk+xVXFtXeuU4xre6a9uk188zrTp1quLj+9/vrrNdZluq9+9auDstz+\nbhtdrnKypKxUp1fqM1fWNMWhJ1P/FEkXAPwblWGo/ETHAh3Hjh07NthfZQa6vz4/dHxV+YBL0BPH\nBSeLcWOi4iSDse6DBw/W2LnOaD+4cUzlOVF+olIfbbsm+3FOYXqeVDLy5JNPDupwzmF6njTWulU6\ncfHixRrfvHlzUMfGxkaNH3300RqrZMRJnJw0NMp/9Dcn29F9tK/u379fYz3WllubuxadzES/n/IM\nVvQa0/Ps5Fna/6WU8qtf/arGR48erbGejx74Bx0AAAAAYEEwQQcAAAAAWBBM0AEAAAAAFsRD06Ar\nqhty1katfTI4PdIU7W0vse6M7V1GY99qe8s2aQzVVTlNdtSHa7nOcsvp11RnpnWrBizWr+dA69Bj\n3dzcrPEvf/nLMsa5c+cGn9W6KqPRzujR43XsNPoZy8UpOne3Xa/d05RstQDQj459TkOuRKtC1fuq\nntnpjjPvPMVMotourU/3Vz2yZph0WuGWnbKiZbl3t7KZvXUcPHDgQI31eFXvrdt/7GMfq7E+O7R9\npfzn8/IBqm1WXbai7xecOnWqxqprLqWU48eP11gtDbUt2j96HE5jHdutdWhf6XZ3796tsTsmvV6j\nBt09r5ytp5LJjho/a/16rzi7x/X19Rpfv369xmpzGeufYucd4R90AAAAAIAFwQQdAAAAAGBB7KrE\npdfiLS5Ruf2nLHE5nLXSnKyLcQnPSRZUztFaDhqjtWSkdlO6BKPfuyx0LWmP+02XCd0SpztnLRmN\n9ok7T5cvX66xyl0UzTBaynA5UZfUskuyD3CZymJ7nZWjO88Zu88WGcvNTH2t7ZC7AMzjueeeq7GT\npSjxPj1y5Mjodnr/u3HePU8jKl9wVolah47BWp/KMfRYSxkel0pqtI1qm+gyV2ocx6c7d+7UWOUZ\n+oxRqY7LwKnyiCjPVAmJSiG0bu1Prc9JTlVqUcrw2PU60UyiLmurk5bG55O2XftK0br1fDo5Z3zO\n6zG6c6BtdHaIWl88bj0u3V/7Ss+nPs+3trZqrLKmeF1duXJltCwn+9kO/kEHAAAAAFgQTNABAAAA\nABbEIiQujri0595AV+bKUnqzI2a2cZKGUnLuMr3ZOGO7tH5ditRtdJnHZVmLdTsXAEWXn9xb7a3z\n6uQZ+r0e0+OPP17j733vezXWY4oZ+HqvS3eNtCRVTv6Sce7J9HOrja6sufcHAKwOXc7Xe9a5Q8Wx\nIOvEMrZ9VkbnngdO6qFyDt1Gs4Xeu3dvUMelS5dGy9K6VTKgfaISEO239957b1CHyoHUmUSfldou\nPQ5tk46nKqkpxT+DVfbjXNL0GaUZSvfv3z+oQ/tR5Sf6vTtnenzan/E49NmpshEnw3HPldY8SPsx\nI03OzIPiXEL7wc2JtN+0H5xrUJzTuD6J8qcs/IMOAAAAALAgmKADAAAAACyIRSQqctvEJQq3f8bl\nJLMEE8nICbSsjANMKTnXmUwbdXkl1uGS+ugbyLrM8/e//73Gn/nMZ2qsxxTfRHaSlYxsQ2NtRzwO\nt9zlln3PnDkzuo0mW9C4FP9m+pxz00LPh1v6csvPrWu9VxblwJ0F4OHi7v/W8r8bX3udm3TMz8pM\n1a1Lx2NNsuPGkpiA5oknnqixSi/1eZNJCKOSmtOnTw9+0+PQ5DT6LHCOHir5UJcZjUvxyfRUhqk4\nWZI6yLRkNBsbGzVWaamW5crV9kVJlJO4aJ/0zu0iToaTeb5qO1wColK8U42TuHzuc5+r8Ztvvllj\nlWDFa1fLUjnS1Oco/6ADAAAAACwIJugAAAAAAAtiVyUuGaZIBhxuqaT19nrGYcO1seV44sjICfT7\nbB1ari6LqVPAiy++WOMLFy7UWJcYdYkyvoWdWXLUJR/dJi4NPSDbb5lkT7qk9de//rXGX/rSlwZl\nuaWvjKTKLSvHZUJdhnPLa0rGYaGVACubeGRs+xarvD8BoI/Wc8iNu6vEjX290hn9PiYqUrmMSjIz\nTjMtlxtFj+PEiRM11iR26tyi5arri8pa9NlayvC49Ni1bn0OOomLfn/48OFBHSpl0ViTELpzptu3\nEmO1JDZjx+G+d5LYuJ2TUblkSErr+e2egyrd1XL1HOi+zv0m1unmRz3wDzoAAAAAwIJggg4AAAAA\nsCAWJ3FpuZ84lwpHZkksbpORmcxJZhM/u+1cchr3fcvFRZfntN/Onz9f41//+tc11jeZf/CDH4yW\nEz9n2uhwTjil+KUvRSUq+ua2OgDcvHmzxtG1xSWLcOfZyYc+/PDD0biU4dKkc2Vw7j69132sw52b\njASsxVw3GwD4N/fv36+xSid0qb01Huo96JbU3djjpJNxPM7ID5z0JZsMKfOMc3FWguNko4rur/3g\nZBBR4qKf1f1E0WeXlqXtVRlFfHbpc0b31zb2jvmxrSqX0WNy7il6TBprf0a5iz6DtQ49Jt1Hy3XX\nVbx2M3Mila9ociudS7ikXJGsvKcF/6ADAAAAACwIJugAAAAAAAtiERKXrEtFr8yk1yGl1Ra3T6bt\nLalO7xKgc+poLRO6JTx9W/7q1as1fumll2r8jW98Y7Tu+Fnr0+Uj9ya0oktacclIl5Zc3e4tfnWj\n0bh1XTnZji6Xaazb/+1vf6vxCy+8MKjjJz/5yehxODJL1FnplNsmA4mKAHaH3//+9zV2CXDUQUQT\nocTPKr1zbiJO7tKSfzj5SSaZmksuF6WTTsbXK8NryQEzCfTcOO+2b9WnEoeM5FD7XeUfmlSplOH1\noM9RJ/txdbfOh15/ve5AGYlSKUO3HnWNcYkSo0vaA5wrTinD86P3h7ZFJUQqi33qqadqrO43169f\nH9Th5LZT4R90AAAAAIAFwQQdAAAAAGBBPDSJS1bW4nAykYwMxi3ntfbJtDGb2GiOzCCTdCbuo0st\n6hSgy0Eqcbl48eLo9nFJNUNmCdDJR1r7uMRIGuvb57ps10oa4JZRXRIiXT7W7X/3u98Nyv3Wt75V\n42eeeabGbhlsla4qvUvRrszWb1l3GQAY59atWzXe2tqqsRsfW8nQ9DcdBzVWiaOTx8TENDreqdtH\nRraxsbExWu7x48cHdTgZhZOfaOxkl60xVPtK+0T3V/mIKyuOpy5xVDZp4ti+2r5ShvITxbmRufpU\nRhPHctd29wzWuvX5po4zKhOJZTn3Gz1W/V6vJb2+YxIhl5BQ26V1v/POO6NlnTx5crTMUobXuLYx\ntiULT1UAAAAAgAXBBB0AAAAAYEEwQQcAAAAAWBCLs1ls6a17desZnXprH6erzVg2Zu3wMuW6fbMZ\nUVUnpbaFaqH4s5/9rMYuU1o8Hxn7R9Veqc5M26uaNdWGRbRc3c71mx6308jFdmUsNPVYXcY21a+V\nUspf/vKXGj/33HM1Vn2fs4iaovV2+zvdeVYXmbHsAoB5uIyYTt9dynBMdBmVVSObsRGM2TF1vHPv\n3yhqmfePf/xjtH1f+MIXBvusr6+P1pEZ01rZQxX3m9anz66MRWTLhtg9o5ze3r1rEOtwzwbdx70b\npxrpln2i61N3Lbqsoq1nu+rAdTtnBemy4Cqxz3V/9+6XlqXbuyy/Z8+eHS2nlKHOPr7LkYV/0AEA\nAAAAFgQTdAAAAACABbEIicvcrIe9GUazkoFeyUlWfpCRUfTSyojqsorqEpfGJ06cGC03SkOcvaFb\n+or7P6C19JWpL2OtpUttc20L3XKnLo/GJbQbN26M1pGxUNPlv5aUxC0BK6u89rBZBFgdGUlFS4qg\n92Mm46N7Lui4EO3h9DeXdVnRZX4dx9Qu8Pz584N9Dh8+XOOjR4+Ofq+2kJpNMzMGRlwGTzfmO+lK\ny2ZRychwnLVhfK649rrnYCZrapSMuP5x16g+u9w1Ep8XGVtIRecM2nadx8TzHy0Rx8hcx5cvX65x\nzLqqdWQlRC14qgIAAAAALAgm6AAAAAAAC2JXJS6ZjJ9zl917s4q2MolmXFwy7hWtY3JykIxjTbZc\nJwG5d+9ejZ988skaf/e7362xLj21XHXckppbgsvijjdz/bg+aNWhuCVAV4e6JUQ5z4svvljjH/3o\nR6N1uMx8WnfLxUHJONP0OtZsVycATEfHDLck3rr/M+Orc0VxZJ8xii77q5RFYze2ljLMoqpjqjpj\nqSuGur6o3MXFsS3OlUXJZM2Mx+GeUb0OPXPH3MyzQLeJEhd3Dbi5gavDZXwtZdjvWm4mY7grN54P\n51qUaaO7V+7cuTP4rNmA9RrVZ3sP/IMOAAAAALAgmKADAAAAACyIh+bikk0cpGSM/zPslKRmyjEp\nbhklszwW69NlIt1O30bW5cPnn3++xpqcQt+81+XRWK7iloy0TVmpjuL61C21ZpPqZJL0uP31mC5e\nvFjjtbW10TbFffSNd9cPuhzXWop0y36ZpdNs8iwn9cHFBWB1ZMaeeM9qAhXnZqHuFzquuGdMK8le\n7zjfSuqjZJy/1M1EZQbOFUv7ppShI4y6lqlcRsdwbbv2bUsG4dxanJOK274ld2kl4BvDyV1ast2M\nhFTdS9T5R89Ty0VF5xa916WbE7WuMbedc6PRc65tjQmI1F1Iz0fm3Iy2c9JeAAAAAACwIzBBBwAA\nAABYEA9N4uKWTbJyh163lkw7stvNlbJkjtGV6976br0Vreg+upz3ne98Z3Rft5wXP2dccjISnlU6\nvWS2b+3jritdJlYJkCYw+OlPfzoo68KFCzV+++23a3zmzJlt2+jaEftcz4dbynTLqK1lbVfH3PsA\nAMaZ4nKm93CUI25XlhunpyTfczI458gR68iM4a4sR0y4pE4x169fr7HKO1UWc+rUqRqr45n2c2y3\nS5rkji+T+C8mKtKyMk4hLulQdvzO9Lu2SZ+Veg5a5yyTZElxbY/zDe0f7UcnDc04uuj1UsrQHUgl\nPdeuXRtt43bwDzoAAAAAwIJggg4AAAAAsCAemsRlLr0Jgtwyf2tpJyPbmLLMn3l7ujdRUatut48u\nz2mcXe7MvN2ddQfZ7vtWWXNx7dVY38LW5bGXXnqpxkeOHKnx17/+9UEdx44dq/Fvf/vbGv/whz+s\nsb41rjhXg3g+evs9I33JslPnBuB/BXdvZxLNtMpy2zkHKa3DjUlxOyfJcMfRkiW657OTzrh9Ww4n\n+lllis7R4913362xJkxSuYvKYGIbNzc3a6zPAvd8zbj4RHqfqVn5kqtTz6fKOXR754qibnKlDJ+p\nKhvR85Rxd2ndH71SWHcdq1Sm5dyjx7tv375t6xuDf9ABAAAAABYEE3QAAAAAgAWxqxIXt9TSK4OI\nv2Xe7nVLeNmkNaskI1PJyFdabc04f2RkNK2+cvs4MrKkiGtvZnluyrl05epy3sbGRo317eyvfOUr\nNY5LX/p298GDB2usb7m7N8VdoqIpsibFuQy0JGC7cX8A/C/i7mdd8ncSjFK8S4ZLFuSkAS10ed9J\nADOJhlpuVCoN0OPNJLpzz5g4Hus46sZBlW1om1TucuvWrRq/8cYbgzrUJe3QoUM11iRJrj97HUtK\nyT0bMs/8Fu4ZlTkHeh3Ha1fb65JsOUmNPkOjq4pru/vetT17f7jkXc5Vbzv4Bx0AAAAAYEEwQQcA\nAAAAWBBM0AEAAAAAFsSuatB7Nc8tXVRvxsgpmdl6dVxZMnompyHM6tR3w3Ivk8E1o1nOWiMpvXr2\nVh0ZeyvVkJ0+fbrG3//+92t84MCB0e1LKeXZZ5+t8dNPP23bsl0Zw7QiAAAWx0lEQVSbWtp7pylV\nLVzGkqp1zuZqGAFgHKe3VRtAvZdjdsyMZW/v/R/HWWeVqGTGroxlYtxH26Lt0LHWWfFFnM7d9ZvW\np6j++c6dO4Pfbt++XWPVrd+9e7fGjz32WI1Vm67vLLl+i+3NkLFizL5rpu1ymWuzVtDuHCp6DvR9\nBO3PeE+4tmjbVbeu/Zu51lt1rOKZyD/oAAAAAAALggk6AAAAAMCCeGgSl96ltrid+36O9GXKdln7\nnQxzM5QqzjpqjtXlFDmQk85ksne2yspeM3PqcHIStxzXWqZbW1ursS6v6pJcxuowm/0ts/w85bpy\n9mQAMA93bzr5Qbz/MraHmfFR29HKVqy4sUSlD9k6tCyX5dONPdkxrddi0j2vtB3RUlLbu7W1VeP3\n33+/xpcuXaqxZqJ+6qmnanzy5Mkaq/Ql1q/Wg0rvOH/9+vXBZ5WAqEWwK1fb9Mgjj4xu3zrnDpet\nVq+x7DMpkwXXtaklAcpYOfbAP+gAAAAAAAuCCToAAAAAwIJ4aJlEp2SezGTgzEg4WssYbrs5EpBW\nHauS7WTpzcDm2tQqK5PFtFWu4pZ9M30ypT7XJxq7pVZdYoxv/btynexH2+gyibbaPieTbNYZYEom\nQgAYRx1BVAbnsnFGGZ1bRs/c5xl5TassbUtvfS1ZQm+W8UyG0Fhur0RSaR2HyxKtqNxF4xs3btRY\npS9nzpwZ7H/q1KnRNrqMmm7M1u9fe+21wT5nz56tsTqVKU72o+fDZU3NlpWZw7XOecYRyPXJXInz\nVDko/6ADAAAAACwIJugAAAAAAAtiERKX7DLBTshBpizNZ2QireQyGRnO3CWVXunNTjmsZNrRchaZ\nk3DJbbNKyZCWq0vRblm6lP63xrOODJnzmbkHdfu4jO76uiW9AYDtcfdQVoqWcZRyrirueZV1UnPu\nMi6hkHOcadFyfhmr2yVoKyXnjOPqcON37Cs9XvdscHWou5cmOYrjsda/ublZY5XC6DlX6YtKTrQd\nJ06cGNShCZQcmXmJk65E3Hl216srqyUzdei5cTIYJStfzl7jEf5BBwAAAABYEEzQAQAAAAAWxENL\nVDTX9aFXRpFNbNPrDpJJLrNTEp6dSpI0Z5u5ZWXfts9IX7Jk3E+cZCTzNnlru97ECK3z7yQnc9x2\n1Ikg7t9KmgIAfThZihv/o9xBcWOM3s8qcZgiV9P6nduG4sY6dayJOOlNxiGlNSa5Y9RyXf/qNq06\n9Lh0O5WZ6PFpm5wkIiYjUimMJj3a2NiosSYLWl9fr/Fjjz1WY5W16Pal5K7FjHzVufhE9BidZCnj\n9BLPjfZv5vno5nPu+R/Rc+iSSG0H/6ADAAAAACwIJugAAAAAAAviobm4TGGOQ0tmCSbukyl3jiNL\na59MMptVupG4drSOw/2W2X9KAqI5iYqy0hBHZkmstTQ857xl5UDueN0ynGtvS7qi++hS7759+0bL\nAoAcTtYwRT7mluS1Di3XOay0lvD1/neJ1TLPwViHqzNTro5PKiuIZerzR/dxzh3aVypd0biVGMfJ\niTLSB+1nTWYUf9Oytra2RuObN2/W+Pr16zU+duxYjV955ZVBHd/85jdrfPDgwRq760rRc/bhhx/W\nWB1nShn2o8p29uzZU+O1tbUaZ+coDuc05BIdZWWmLunR1Lkv/6ADAAAAACwIJugAAAAAAAvioUlc\nWslpHtCSArglCiVTR7bOXglHpswWbqmllXzBkWlXpo0tOVDGHSTTphaZ+jLHMSXhUit5xxjxmux1\nGsoki2pJajLL1O57bce//vWvwW/uzfTz58/X+Mc//rFtFwD04cakKCtwEhBF73m9f934FOU1bizK\nPEucdK6VREglHE6eo2gfOBnLWJ1juD5ROYarL+6TkRnp906qc+/evUEd165dq7FKSLRdKq/R/rxx\n40aN33jjjRq/8847gzqOHDlS43PnztX46NGjNVZnGnfOtD80eVJso0pZMsmiFPcMjb85tD53jenx\nxWvdzdtajkvN9kzaCwAAAAAAdgQm6AAAAAAAC2JXJS5K9q1YJeP8scrkPXOcN1pLfr3SEHdMzt2l\nVYdjioSnVf92dWeSAGXb6Lbp3Te2pVdSk71e5iYkGmtr3N/hlq/d8mpcRtfPukz5wgsv1BiJC0A/\nLqGQu2dbksPMeOfuea1DpQcR95tz9HCSk7h9b1I3l+io5UalshGVH2i7tFztww8++GC0vtgO7R/3\nXNHz7BLpqMzw/v37gzrUlUXr0FjlJ9pGd54OHz48+Pzmm2/WWF1kNLmRJj1SRxh9Rrh2lJKX6z5g\nrkNKZp6QuZ+idMXdU9nkXxH+QQcAAAAAWBBM0AEAAAAAFgQTdAAAAACABbGrGnSnhctqnjMa3VXq\n0XttCJWsRqrXFnCOvWCkV9OdrcO1152/rP5szrG3zkFvZtjMvq2sq47e48jWkdHVq0azZbOmZW1s\nbNT4i1/84rZtBwCPs3hr6c6VXptWZ3Wo5WhWx7iP03E7LbRuo/vGY1LtdmZMVB23HofGLZs7Z3uo\n5WqcJWND6TKXOnvCVmZnV7frE9Wz6/dRH651qs2jZihVa8YDBw7U+NSpUzU+ffp0jQ8dOjSoQ+tX\nzb3LzOnQ4269o5WxaezNKr8T8A86AAAAAMCCYIIOAAAAALAgdlXiMsfCbrvfxrZx9n1ZC7yMJGdu\ndss51oG920SmyGKUjDRlip1mL73nYErdehyZ62qKpKY38+0UyVEmO6rLoFbKcKlY7bS+/OUvp9oC\nANuj96mza4vZGJ20xFkouvHbWcXFfVx2TB0jnAViCyf1cVlU3Zjm7BMjvc9B129RBuPkFk7C4yQu\nenx79+4d1KH9q3WoTETbpZIl3WZ9fb3GaqVYyjBDaaz/AWo9qdIXzXT61ltv1fjs2bOD/dWy0WVB\nVbSvnDwrPkPnSGR6bSBXBf+gAwAAAAAsCCboAAAAAAAL4qFlEnW0ZAK9MoXMNnHpYk620rmuKr3y\njLkykblOKL39MEfCE5mTSTRblts/8wZ49rc50puWBMzhXCCy/eky1O302+wA/+247KFOihbvOeek\n4uR5zilG64gyGifJcPsrToITcZIejTP903IAyRx7RmbUwrnIqHxFz7nW59oXJR9artYXz9sDVOKi\nchXdPmZKzRy7tsvJmm7dulXj27dvD37bs2dPjY8cOVJjzVCqzi/aXjdny8pSMhlD58pdpj4f+Qcd\nAAAAAGBBMEEHAAAAAFgQD83Fxb1525JEuKWFXjnJnAREc+uO++yE9CHuk3HxmFL3HMlRNjmV23+n\nXGDmsFNJsrLJk3bqWsow9z4C+F/HyQdUxtByWFJXDpUZ6D4qr1B0GyclKSU3hjtnESc/ic/1zHNe\n26v9k3HIKsU7qWjdTnKkx9eSiTpJj5OyuLZrX0WHFf1N63CJjnR7PQ51atEkUqUM+8G5CDm5U+b8\nl1LKnTt3ary5uVnjK1eu1Hj//v01VrmLSmKcDCbWr7i+dvs6p6BIVgrVgn/QAQAAAAAWBBN0AAAA\nAIAFsasSl1X85f+ATBIhR/Yt3DkJaaYkJ3JLXFOkIRk3Gsdc15iMjGZKWbtdX29fTSnX9bW73ubK\nVXqTb7Xuld7ESgCQI5OsJ96bTgrjYq1DZQ1aR1zCd/IVV67ipB2tZEjOmcbJJZyEI7ZJJR1alkpf\nnBOKlqtxa37jEke5PnTuLtru+Jvrq/v372/bDu3zKHHROlRGpWgfqrzGXQuta1fr0KRHGt+4caPG\na2trNT569GiNVfpSylAioxw8eLDGmevKJeuK+zsZVg/8gw4AAAAAsCCYoAMAAAAALIhdlbhk5AOt\npfLMkvpciUJvQqJWWRky0oC5Uote2cdcKVImsUZvEqjIKmUfjl4Jx5RkSJk+ybSvFO+M5NrhtnHy\nmha4uACsDnc/te7NjBuFk0Fo0jElusa4sT2DSh+0rTGxjR6j1u+kBRnnltZxKE4yokl9vva1r9X4\n6aefrvFvfvObQVkXLlwYrUOPV48j89yNfeUkFZqQSBMPqZzDSVFUMhL3v3fv3mjbH3nkkdFytQ9b\nMg93zp3kRGUwThJz+fLlQR3O+UXlRNpGPSZlSjKkqc9H/kEHAAAAAFgQTNABAAAAABbEQ0tUlCEu\nwWckJ855JSvzyLiW9MpgsglllExfzU1g5L6fImtw/ZZdAnJ1Z9rl2jFXOjNH1jLlnPfKvrIOK5lz\nkJW1TJGmAcD2qIOI4u6tmHTIJdxxZak0QCUuuszfShbYcmIZ28Yl0lGJQSn/KeMYa4vW58Z8rSP2\nh3O2cYmc1D3l1VdfrbHKP9QtJeJccjJSppZriOL6R68rd25b479LaORcgBTXn62kTnot6nE4KYpL\nLhUdZzY2Nmr81ltv1fjEiRM1VgnQsWPHRtuk7j7uWl0V/IMOAAAAALAgmKADAAAAACwIJugAAAAA\nAAtiVzXoTvOUybKY3c7p0bMa617d8ZTMpRkbwt76pmi3p+jO3f6O3qyrLY3dquqbsn/ruszUN6dd\nc98J6NWjZ+tGdw6wOpyOV9F7Ob6npHpfZ0/oshtm72VnlefeQVJNsLbX6e3jdlGfPtYON0Y5u774\n2enZtT91m5dffrnGf/rTn0bbFOvIPPMz38fspk5/rdroPXv21Fh12S7zbDw3+pvu754xzvrR2S/G\nsvQ+yGS+VVrXtOrIVUt/6dKl0W2uXr1aY7Vl1G1idtLDhw/XOJOVdjv4Bx0AAAAAYEEwQQcAAAAA\nWBAPLZPoKsvqlRxMkSL0ZuN0y1ul5OQ9zvZoio1gxjrSbd+SduiyX+t4tyurV3YRmSsZmdM/U66x\nTLsy/ZO1cpzTv1PsKQGgH5UouLFAx9xo8eayNjoruJYExKEyBS239xmsdcdtnDWfOz4nr2hlVnZ9\n6iwNdRuVOLSylSrO8jmTrdrJkkrx14nL2urKzWaIVUmHszp09psteZai15WbgzmpVUsm5srVWK0y\n79y5U2O1aNy3b1+No3RlfX29xip3iZaPWfgHHQAAAABgQTBBBwAAAABYELsqcelllZKYTNbLuJ3i\nJCcZmUmUGPS+uT3FbSWb4bKnjlhmZnluNzJPZvokK2PpddVZJb1Sm3hdzXWwGSunVYeCowvA6sg4\ngEScNERx0gAnH1D3ilKGzhQqybl7926NndzBSWJaz2Ani8iMQ1MkJyrJcBIQdQBxspu4vx67c41R\nXDbumLHVyXNc2520xGVWjW3Rc+7kTq5NczKMx/3d9ZMtV8/V2tpajVW+pLIULfef//xnjTc3Nwfl\nXrlypcYqhdG4B/5BBwAAAABYEEzQAQAAAAAWxCJcXLKSkaxMZbtt5i7HZ44jK6NwZFxcssfklvp6\n+2E35EDZ+udIg1rHkUlI1NvW1m+rTCI1xeFnTt2rktQAwJDMOBTlCk5u4eQLuuyu7hPf/va3a/zZ\nz352UIfKAa5du1bjt99+u8YvvPBCjS9evDhad8thxSWbaTl/PEBdPHRf52RTik+so2TqVulDLMu5\nvTj5icpHtK+ijEY/q6OIc6NxDistmYiT6mhfqxwkc3yxDidNcfKjjMtZPJcZ9zw9Po31fGiskpjY\nXpW/aNKjHvgHHQAAAABgQTBBBwAAAABYEItzcclKA5RsIpftvm9t1+vokV3ynyJlGCMuwc2RH2T3\nzUhZ5sogMufQvZHvyCaO6m1H6/te+cncxD9z5UuZdsxNMAUA/0blCr3OVC2eeOKJGn/605+u8ec/\n//kaqzzik5/8ZI01MU0pQ7eW8+fP1/jll1+usTpZqOOJk5lE2YY7Luew4pIWuX1jHU5+4hLxOFqJ\n/FzdmQR/+n1MxOPkFtrXTnKSSdDUaq9LdJVxoIl1OHch1+963E6qk5WyOnR/dy6jjMZJjt5///1t\n6xuDf9ABAAAAABYEE3QAAAAAgAWxOBeX7P4Z8/25iX9WuczYW0emvTvlsDElqU/me5fYqHU+Mg42\nTu4yJalOr6vKFCnTqmhJdZTeayzb1kyiKgDI4dw2HNl77hOf+ESNz507V+M//OEPNXZJh27evDko\n689//nONX3311RqrBMA5kGSfY/rZObpoueog4mRCURqiOGcTPQ43hrZkG9pebZdKUXQfbaNz5IlS\nED1v2g8qLcrMj5zDTinDY3RyIm3HFJc7bbtLSJVxo2tJhlxfO6lO1oFG0e30PO/du9fu04J/0AEA\nAAAAFgQTdAAAAACABfGR3Vyafv3110crmyJFyOw/paw5y/ZTHF16256VuGSOY27ypoxUqFcO0nI/\nWRWxTNdXvU4q2fM8Ry4zV9bUmyQpW7/u8/GPf3yeBQ3A/yCPPvpovYlcoiG3/F/K0Lnjox/9aI2d\nLEFjJw3JjpVat3O/cE4drTGtV+qj7dD6VK5SipcTOZcTRftW627VkZFk6v5ahzumiJbrHG9c+7LP\nOufK4qQ+rm+zfaW4slTOo8S+0joz5yMzh4rORG5OtbW1VeM//vGP6ecj/6ADAAAAACwIJugAAAAA\nAAvi/1WiIiWzJJNZjo/19TphTGGVZTkyx9Er52gtd7rt5rqfKKuSg8R9e6VJU45jjlym16WmVVav\ns9GUxGEA0I+TD7ht4vjrJB1OcqDL8+pq0XouqPRCf3OuIbqNkxjEY1X3i4yThsbqJqJxlHk415KM\nlMG5yWSS5LVw8honXSnFyz60vZnxvCXtcXMJdz70GnHuLvG60uPQ7dTdJeOqo/tq3bGN7rw5ZxrX\n1ijVcY5AU+EfdAAAAACABcEEHQAAAABgQTBBBwAAAABYELuqQc9kksySseLZKdu6jKVgVps2R8c7\nxRKwt09adWS0eHN1566+OdvPfQdgil7bacd7de5T2u6sOXvbFH+b2y4A+DduDM0+S5xuWVGdeub+\nj+VoHdFirqe9LotlKUM9u7NQ1HKdJjjTH9l2uXFTid+7Y3f9o+dG2946H06XnclK695TiO3OzNsy\n8y53bkrxVqCZ69iRzR6ruONzGWYjq3wnoRT+QQcAAAAAWBRM0AEAAAAAFsSuSlzc0pDSWmrXz27Z\nxdWxSku4XonDlLpXKQfqtdPLSkMybcmUm83euiqpxiqzla5SnuWYsjybOedzZSnIWgBWh8tK6az0\nshbBvdu05ANqPadSFP3eyTZcHS0pQMZaT793GVgjGSmLk5m4uuNxZCwm3TE5+VCrfidf0rqdnWLr\nWF1bVEKi1orufGavXSch6c34Hdvh5onuHLbO7XZ1lzK8P1qymBb8gw4AAAAAsCCYoAMAAAAALIhF\nZBKdslTeu5zv3oqfwhQZRS8ZWYJbBm2VNcd5I5uB05GR1LTkJ5k+mZvlc05fOQlWti29bY/bZDOn\nblffKp13AKCfuQ5kKj/QzJxuaT/rUpEZH7WODz74oMYqlXByjFL6M5G6svR7lWDEshR9jmqsEp6M\no0us08krVumE5TJiOjeSrOtcJuum9qfGmgm0JWvSayPTP25ul5WSOGlQZi6g10J0o9FzruXeuXMn\n1a4I/6ADAAAAACwIJugAAAAAAAviIyxhAwAAAAAsB/5BBwAAAABYEEzQAQAAAAAWBBN0AAAAAIAF\nwQQdAAAAAGBBMEEHAAAAAFgQTNABAAAAABYEE3QAAAAAgAXBBB0AAAAAYEEwQQcAAAAAWBBM0AEA\nAAAAFgQTdAAAAACABcEEHQAAAABgQTBBBwAAAABYEEzQAQAAAAAWBBN0AAAAAIAFwQQdAAAAAGBB\nMEEHAAAAAFgQTNABAAAAABYEE3QAAAAAgAXBBB0AAAAAYEEwQQcAAAAAWBBM0AEAAAAAFgQTdAAA\nAACABfF/L9T72NqAJVEAAAAASUVORK5CYII=\n",
"text/plain": [
"![\"Block](\"images/block_matrix.png\")
\n",
" ([Source](http://avishek.net/blog/?p=804))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"What this looks like:\n",
"\n",
" for i=1 to N\n",
" for j=1 to N\n",
" for k=1 to N\n",
" {read block (i,k) of A}\n",
" {read block (k,j) of B}\n",
" block (i,j) of C += block of A times block of B\n",
" {write block (i,j) of C back to slow memory}\n",
" \n",
"**Question 1**: What is the big-$\\mathcal{O}$ of this?\n",
"\n",
"**Question 2**: How many reads and writes are made?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# End"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/4. Compressed Sensing of CT Scans with Robust Regression.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 4. Compressed Sensing of CT Scans with Robust Regression "
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Broadcasting"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The term **broadcasting** describes how arrays with different shapes are treated during arithmetic operations. The term broadcasting was first used by Numpy, although is now used in other libraries such as [Tensorflow](https://www.tensorflow.org/performance/xla/broadcasting) and Matlab; the rules can vary by library.\n",
"\n",
"From the [Numpy Documentation](https://docs.scipy.org/doc/numpy-1.10.0/user/basics.broadcasting.html):\n",
"\n",
" Broadcasting provides a means of vectorizing array operations so that looping \n",
" occurs in C instead of Python. It does this without making needless copies of data \n",
" and usually leads to efficient algorithm implementations."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The simplest example of broadcasting occurs when multiplying an array by a scalar."
]
},
{
"cell_type": "code",
"execution_count": 718,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 2., 4., 6.])"
]
},
"execution_count": 718,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"a = np.array([1.0, 2.0, 3.0])\n",
"b = 2.0\n",
"a * b"
]
},
{
"cell_type": "code",
"execution_count": 128,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[1 2 3] (3,)\n"
]
}
],
"source": [
"v=np.array([1,2,3])\n",
"print(v, v.shape)"
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(array([[1, 2, 3],\n",
" [2, 4, 6],\n",
" [3, 6, 9]]), (3, 3))"
]
},
"execution_count": 129,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m=np.array([v,v*2,v*3]); m, m.shape"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"n = np.array([m*1, m*5])"
]
},
{
"cell_type": "code",
"execution_count": 134,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[[ 1, 2, 3],\n",
" [ 2, 4, 6],\n",
" [ 3, 6, 9]],\n",
"\n",
" [[ 5, 10, 15],\n",
" [10, 20, 30],\n",
" [15, 30, 45]]])"
]
},
"execution_count": 134,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"n"
]
},
{
"cell_type": "code",
"execution_count": 136,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"((2, 3, 3), (3, 3))"
]
},
"execution_count": 136,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"n.shape, m.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We can use broadcasting to **add** a matrix and an array:"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 2, 4, 6],\n",
" [ 3, 6, 9],\n",
" [ 4, 8, 12]])"
]
},
"execution_count": 48,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m+v"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Notice what happens if we transpose the array:"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(array([[1],\n",
" [2],\n",
" [3]]), (3, 1))"
]
},
"execution_count": 49,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v1=np.expand_dims(v,-1); v1, v1.shape"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 2, 3, 4],\n",
" [ 4, 6, 8],\n",
" [ 6, 9, 12]])"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m+v1"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### General Numpy Broadcasting Rules"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"When operating on two arrays, NumPy compares their shapes element-wise. It starts with the **trailing dimensions**, and works its way forward. Two dimensions are **compatible** when\n",
"\n",
"- they are equal, or\n",
"- one of them is 1"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Arrays do not need to have the same number of dimensions. For example, if you have a $256 \\times 256 \\times 3$ array of RGB values, and you want to scale each color in the image by a different value, you can multiply the image by a one-dimensional array with 3 values. Lining up the sizes of the trailing axes of these arrays according to the broadcast rules, shows that they are compatible:\n",
"\n",
" Image (3d array): 256 x 256 x 3\n",
" Scale (1d array): 3\n",
" Result (3d array): 256 x 256 x 3"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Review"
]
},
{
"cell_type": "code",
"execution_count": 165,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"v = np.array([1,2,3,4])\n",
"m = np.array([v,v*2,v*3])\n",
"A = np.array([5*m, -1*m])"
]
},
{
"cell_type": "code",
"execution_count": 166,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"((4,), (3, 4), (2, 3, 4))"
]
},
"execution_count": 166,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"v.shape, m.shape, A.shape"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Will the following operations work?"
]
},
{
"cell_type": "code",
"execution_count": 159,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[[ 5, 10, 15],\n",
" [10, 20, 30],\n",
" [15, 30, 45]],\n",
"\n",
" [[-1, -2, -3],\n",
" [-2, -4, -6],\n",
" [-3, -6, -9]]])"
]
},
"execution_count": 159,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A"
]
},
{
"cell_type": "code",
"execution_count": 158,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[[ 6, 12, 18],\n",
" [11, 22, 33],\n",
" [16, 32, 48]],\n",
"\n",
" [[ 0, 0, 0],\n",
" [-1, -2, -3],\n",
" [-2, -4, -6]]])"
]
},
"execution_count": 158,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A + v"
]
},
{
"cell_type": "code",
"execution_count": 167,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[[ 5, 10, 15, 20],\n",
" [ 10, 20, 30, 40],\n",
" [ 15, 30, 45, 60]],\n",
"\n",
" [[ -1, -2, -3, -4],\n",
" [ -2, -4, -6, -8],\n",
" [ -3, -6, -9, -12]]])"
]
},
"execution_count": 167,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A"
]
},
{
"cell_type": "code",
"execution_count": 168,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(4, 3, 2)"
]
},
"execution_count": 168,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A.T.shape"
]
},
{
"cell_type": "code",
"execution_count": 169,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[[ 5, -1],\n",
" [ 10, -2],\n",
" [ 15, -3]],\n",
"\n",
" [[ 10, -2],\n",
" [ 20, -4],\n",
" [ 30, -6]],\n",
"\n",
" [[ 15, -3],\n",
" [ 30, -6],\n",
" [ 45, -9]],\n",
"\n",
" [[ 20, -4],\n",
" [ 40, -8],\n",
" [ 60, -12]]])"
]
},
"execution_count": 169,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A.T"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Sparse Matrices (in Scipy)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"A matrix with lots of zeros is called **sparse** (the opposite of sparse is **dense**). For sparse matrices, you can save a lot of memory by only storing the non-zero values.\n",
"\n",
"\n",
"\n",
"Another example of a large, sparse matrix:\n",
"\n",
"\n",
"[Source](https://commons.wikimedia.org/w/index.php?curid=2245335)\n",
"\n",
"There are the most common sparse storage formats:\n",
"- coordinate-wise (scipy calls COO)\n",
"- compressed sparse row (CSR)\n",
"- compressed sparse column (CSC)\n",
"\n",
"Let's walk through [these examples](http://www.mathcs.emory.edu/~cheung/Courses/561/Syllabus/3-C/sparse.html)\n",
"\n",
"There are actually [many more formats](http://www.cs.colostate.edu/~mcrob/toolbox/c++/sparseMatrix/sparse_matrix_compression.html) as well.\n",
"\n",
"A class of matrices (e.g, diagonal) is generally called sparse if the number of non-zero elements is proportional to the number of rows (or columns) instead of being proportional to the product rows x columns.\n",
"\n",
"**Scipy Implementation**\n",
"\n",
"From the [Scipy Sparse Matrix Documentation](https://docs.scipy.org/doc/scipy-0.18.1/reference/sparse.html)\n",
"\n",
"- To construct a matrix efficiently, use either dok_matrix or lil_matrix. The lil_matrix class supports basic slicing and fancy indexing with a similar syntax to NumPy arrays. As illustrated below, the COO format may also be used to efficiently construct matrices\n",
"- To perform manipulations such as multiplication or inversion, first convert the matrix to either CSC or CSR format.\n",
"- All conversions among the CSR, CSC, and COO formats are efficient, linear-time operations."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Today: CT scans"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[\"Can Maths really save your life? Of course it can!!\"](https://plus.maths.org/content/saving-lives-mathematics-tomography) (lovely article)\n",
"\n",
"![\"Computed](\"images/xray.png\")
\n",
"\n",
"(CAT and CT scan refer to the [same procedure](http://blog.cincinnatichildrens.org/radiology/whats-the-difference-between-a-cat-scan-and-a-ct-scan/). CT scan is the more modern term)\n",
"\n",
"This lesson is based off the Scikit-Learn example [Compressive sensing: tomography reconstruction with L1 prior (Lasso)](http://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Our goal today"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Take the readings from a CT scan and construct what the original looks like.\n",
"\n",
"![\"Projections\"](\"images/lesson4.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"For each x-ray (at a particular position and particular angle), we get a single measurement. We need to construct the original picture just from these measurements. Also, we don't want the patient to experience a ton of radiation, so we are gathering less data than the area of the picture.\n",
"\n",
"![\"Projections\"](\"images/data_xray.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### Review"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"In the previous lesson, we used Robust PCA for background removal of a surveillance video. We saw that this could be written as the optimization problem:\n",
"\n",
"$$ minimize\\; \\lVert L \\rVert_* + \\lambda\\lVert S \\rVert_1 \\\\ subject\\;to\\; L + S = M$$\n",
"\n",
"**Question**: Do you remember what is special about the L1 norm?"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Today"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We will see that:\n",
"\n",
"![\"Computed](\"images/sklearn_ct.png\")
\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Resources:\n",
"[Compressed Sensing](https://people.csail.mit.edu/indyk/princeton.pdf)\n",
"\n",
"![\"Computed](\"images/ct_1.png\")
\n",
"\n",
"[Source](https://www.fields.utoronto.ca/programs/scientific/10-11/medimaging/presentations/Plenary_Sidky.pdf)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### Imports"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import numpy as np, matplotlib.pyplot as plt, math\n",
"from scipy import ndimage, sparse"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"np.set_printoptions(suppress=True)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Generate Data"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Intro"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"hidden": true
},
"source": [
"We will use generated data today (not real CT scans). There is some interesting numpy and linear algebra involved in generating the data, and we will return to that later.\n",
"\n",
"Code is from this Scikit-Learn example [Compressive sensing: tomography reconstruction with L1 prior (Lasso)](http://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Generate pictures"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def generate_synthetic_data():\n",
" rs = np.random.RandomState(0)\n",
" n_pts = 36\n",
" x, y = np.ogrid[0:l, 0:l]\n",
" mask_outer = (x - l / 2) ** 2 + (y - l / 2) ** 2 < (l / 2) ** 2\n",
" mx,my = rs.randint(0, l, (2,n_pts))\n",
" mask = np.zeros((l, l))\n",
" mask[mx,my] = 1\n",
" mask = ndimage.gaussian_filter(mask, sigma=l / n_pts)\n",
" res = (mask > mask.mean()) & mask_outer\n",
" return res ^ ndimage.binary_erosion(res)"
]
},
{
"cell_type": "code",
"execution_count": 208,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"l = 128\n",
"data = generate_synthetic_data()"
]
},
{
"cell_type": "code",
"execution_count": 209,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAATsAAAEyCAYAAACF03cPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFMVJREFUeJzt3V/MZHd93/H3p+tgCqjBLtVqY7u1K60SGSsp0QpBkgsU\nQ3EowvTG2qhIm4JkVaINiSJFdrmIeoeUKEouSqoVIViNZWoRUq+QEnA2qdJexLAbIuo/LHbjGK+7\n9hJRJRWpIhy+vZjzmPEwzz7z78ycM7/3S3r0PHPm3/c5M/M9n/M758xJVSFJ++7v7boASdoGm52k\nJtjsJDXBZiepCTY7SU2w2Ulqgs1OUhN6a3ZJ7kpyKckzSe7r63kkaRHpY6fiJMeArwHvAi4DXwJ+\nuqqe3PiTSdICruvpcd8KPFNVfw6Q5NPA3cDcZpfEwzgkreovq+ofHXWjvlZjbwKen7p8uZv2iiT3\nJrmQ5EJPNUhqw3OL3KivZHekqjoLnAWTnaT+9ZXsXgBumbp8czdNknair2b3JeBkktuSvAY4DZzr\n6bkk6Ui9rMZW1ctJ/i3weeAY8MmqeqKP55KkRfSy68nSRThmJ2l1F6vq1FE38ggKSU2w2Ulqgs1O\nUhNsdpKaYLOT1ASbnaQm7OxwMW3GYbsOJdlyJa+26C5Nu65T7TDZSWqCyW5E5qWlw5LRwW23lZxm\na1v0eZf5n6R1mOwkNcFkNwLbTmmLmk5lq9Y2735D/X81biY7SU0w2Q3QquNfkg5nspPUBJPdAIwt\nyfU9pnbwuI7daZNMdpKaYLLbIZOLtD0mO0lNMNntGdOiNJ/JTlITbHaSmmCzk9QEm52kJriBYodm\nd56dnb6Ise2QLO2KyU5SE0x2AzCbxhb9SvN5992GeYl0k3W4+4z6YLKT1AST3Zr6GDMz0UibZ7KT\n1AST3YoOG1dqabxp+n/0hDsaOpOdpCaY7JbQ19bHfbDqFmXno7bFZCepCSs3uyS3JPmjJE8meSLJ\nR7rpNyZ5NMnT3e8bNlfublQVVUWSV34Oc3D9wX1aNT2vrvUjbcs6ye5l4Beq6nbgbcCHk9wO3Aec\nr6qTwPnusiTt1MrNrqquVNWfdn//X+Ap4CbgbuCB7mYPAO9ft8hdmU10UsvGvraykTG7JLcCbwEe\nA45X1ZXuqheB45t4Dklax9pbY5O8Afgd4Oeq6q9n9r2qJHMXBUnuBe5d9/klaRFrJbsk38ek0T1Y\nVZ/tJr+U5ER3/Qng6rz7VtXZqjpVVafWqaEPrr5K3zX7eRjr6uw6W2MD/CbwVFX96tRV54Az3d9n\ngEdWL0+SNiOrdugkPwH8d+B/At/pJv97JuN2DwP/GHgOuKeqvnnEYw1iMbHJQ71aOmxM+20Eh0Ze\nXGQNceUxu6r6H8Bh/+Wdqz6uJPXBw8UY1BJKGoRFDo2c/RLXoX9+PFxMUhNMdhs2lqWc1BqTnaQm\nNJ3sTGHS+sYydmeyk9SEppPdJpZInqRamhh6wjPZSWpC08nuwFFLpGsdZTK0pZe0a32fRH1VJjtJ\nTTDZTZm3RJqeLrVi3mkyx/45MNlJaoLNTlITXI3F1VapD0NbFTbZSWpCM8lumd1HhrAUUr9M84vb\n5Ia7XX62THaSmrC3yW6dpdDQD3vR4g5L9O44vrxl5tlR990Fk52kJuxdsttkGjPhjdeyr9m1bufr\nP9/Y5ofJTlIT9i7ZqW19pDAT/n4w2Ulqwt4kuz6XukP9yhpNbOt1MeGNm8lOUhNsdpKaYLOT1IS9\nGbNTe3Y1dubY3TiZ7CQ1wWYnqQk2O0lNsNlJaoIbKJYwtK+ZlrQ4k52kJqzd7JIcS/LlJJ/rLt+Y\n5NEkT3e/b1i/zIXqIAlVtdSXCo7l+aQW9Pl52kSy+wjw1NTl+4DzVXUSON9dlqSdWqvZJbkZ+BfA\nJ6Ym3w080P39APD+dZ5j6GYT3uyPpKMdfF76XGNaN9n9GvCLwHemph2vqivd3y8Cx+fdMcm9SS4k\nubBmDZJ0pJWbXZL3Aler6uJht6lJa57bnqvqbFWdqqpTq9ZwSF1zk1bfDp539sek159583iXz6/l\nzSa6Pq2z68mPA+9L8h7gtcA/SPLbwEtJTlTVlSQngKubKFSS1rFysquq+6vq5qq6FTgN/GFVfQA4\nB5zpbnYGeGTtKldwVMLaVhq4Vh2S5uvj89LHfnYfA96V5Gngnd1lSdqpDCFhJNlpEds8GsKvdu/X\nLo9s8bVd3lGv14Lz9OIiY/8eQSGpCR4by3a/jNHja/eXr+2wmewkNcFmJ6kJrsZO8dwC4+druF82\nOTRgspPUBJOd9pIJT7NMdpKaYLLTXjPh6YDJTlITTHaSdmabydtkJ6kJJrs5+lzaeLD4bjh2N37r\nvoYmO0lNMNlJG2JqX902krfJTlITTHbXMLu0mXfdUWbv6xJ/txy7a5fJTlITTHYLmLf0X/Tr7E0O\nw2Rqb4/JTlITbHaSmuBq7IpcZdkPDlGMz6obmUx2kppgspNmmNi2zy8CkKQNsdlJaoLNTlITHLOT\ntDPbPGzPZCepCTY7SU2w2UlqgmN2GhyPYFAfTHaSmrBWskvyRuATwB1AAR8ELgH/BbgV+Avgnqr6\nP2tVuacWTTCz9inRbPIrlpa5r9qzbrL7deD3q+qHgB8BngLuA85X1UngfHdZknYqa6SL7wf+DPin\nNfUgSS4B76iqK0lOAP+tqn7wiMdarYgR2sRJWfbhxC597V+1zf22Vv3swHhft01b5/Wauu/Fqjp1\n1O3XSXa3Ad8AfivJl5N8IsnrgeNVdaW7zYvA8Xl3TnJvkgtJLqxRgyQtZJ1mdx3wo8BvVNVbgG8x\ns8raJb65i7+qOltVpxbpyJK0rnWa3WXgclU91l3+DJPm91K3+kr3++p6Je6HqqKqSPLKz6qmH+Pg\ncdWfg3k8+zP9Oiz7c9hjtmab7+OVm11VvQg8n+RgPO5O4EngHHCmm3YGeGStCiVpA9bdqfjfAQ8m\neQ3w58C/ZtJAH07yIeA54J41n2PU+h4wH9t5UMc0P/qs9bDH3IeNT0O18tbYjRaxx1tjt9WEbHab\nf55dzNNWm90q83rZrbGDPFxslQbc0htD/drlgmP6OceyANuEbayheLiYpCYMKtltYgfDA7teGra0\nVJY2pc+EZ7KT1IRBJbt1zC4BdpWsdvW8Y9sqOzRD3DDQ8ms6+79vgslOUhP2JtlJ2j+O2UnSkvY2\n2W17vKPFcZUh8nXQYUx2kpqwt8lOw9TyFkbtlslOUhNsdpKa4Grsmlwd2x3PLqZlmOwkNcFkp53Y\nxOFAm0xxQ/xqpaHUsS9MdpKaYLLTRqxx/uENVyLNZ7KT1IS9TXaOd/Rnk1tB/WqlVxvi/NgXJjtJ\nTdjbZKfNay0tH7XFeBNnLtvkY+raTHaSmrA3yW5oJ9zZlbEdaD/E/dtmLXJC6009pl5tkXm86Lw0\n2UlqwqCS3Tp71buk7M9QE9euOT/6s8h7btk+YbKT1IRBJbsDY1piDnWMbKh1SdeyTKJb9j1tspPU\nBJudpCYMcjVWm+PqrMZgG+9Pk52kJpjsJO3MNtc4THaSmrBWs0vy80meSPJ4koeSvDbJjUkeTfJ0\n9/uGTRWr1SUhCVW11mFO0iYcvA8P3pfbsHKzS3IT8LPAqaq6AzgGnAbuA85X1UngfHdZknZq3dXY\n64C/n+Q64HXA/wbuBh7orn8AeP+az6EdMxVqXQfvnXUS3bppcOVmV1UvAL8CfB24AvxVVX0BOF5V\nV7qbvQgcn3f/JPcmuZDkwqo1SNKi1lmNvYFJirsN+AHg9Uk+MH2bmsSAuVGgqs5W1amqOrVqDVre\nbEobUlIzQY7bbHqbl+S2OUY3a53V2HcCz1bVN6rq28BngR8DXkpyAqD7fXX9MiVpPevsZ/d14G1J\nXgf8P+BO4ALwLeAM8LHu9yPrFnktiyYAjxxY37W+gquPE1b72g7b2L4wd+VmV1WPJfkM8KfAy8CX\ngbPAG4CHk3wIeA64ZxOFStI6MoSxkSQLF7Hq0qTvNDK2Y083fcq+Xaawvl9bvdqu3uvXeN6Li4z9\newSFpCaM4tjYTaSQefcbWxrbpE2f6GbZhN3HGF/fz6Pt2+QaiMlOUhNsdpKaMIrVWO0Pv0x0vMb+\nmpnsJDWh6WS3yZRhYhkWXw/NMtlJasIokt2md5OQNGx9fM5NdpKaMIpkNyaOFUnDZLKT1ITRJbux\nJKex1CkNSZ+fF5OdpCaMLtmNjQlP+2Ls72WTnaQmNJ3strmEmvdV47tcMu66jr7nx1jTR6u28XqZ\n7CQ1YbTJbqzjB7s+GmRs80vDc9gJkZZ5T+3iZD0mO0lNsNlJasJoV2OXNcQzUG1zVXyoq6/zVuvn\nXXctQ3xtWzA7j5c5U+EuXh+TnaQmjD7ZLXr2+FaW9IfNhzH8/6smhTH8by0Y+utgspPUhNEnuwND\nX6r0ZReb8Ldln/4X7Z7JTlIT9ibZjcUmthzu+lAvaYxMdpKaYLLbkqHu5zaPW0G1j0x2kppgsuvZ\nGBKdRy6oBSY7SU04stkl+WSSq0ken5p2Y5JHkzzd/b5h6rr7kzyT5FKSd/dVeMuSvPJTVUsdk3jg\n4H5V9arHWyaZzd5vnXqkvi2S7D4F3DUz7T7gfFWdBM53l0lyO3AaeHN3n48nObaxaiVpRUc2u6r6\nY+CbM5PvBh7o/n4AeP/U9E9X1d9W1bPAM8BbN1TrqMymJkm7teqY3fGqutL9/SJwvPv7JuD5qdtd\n7qZ9jyT3JrmQ5MKKNUjSwtbeGltVlWTpQZqqOgucBVjl/pK0jFWT3UtJTgB0v692018Abpm63c3d\nNA3EvI0Sm+SGCg3Vqs3uHHCm+/sM8MjU9NNJrk9yG3AS+OJ6JUrS+o5cjU3yEPAO4E1JLgO/BHwM\neDjJh4DngHsAquqJJA8DTwIvAx+uqr/rqXZJWliGsKqxj2N22zpyYtnnGWpd0houVtWpo27kERSS\nmmCzk9QEm52kJtjsJDXBr3gaIb+WXVqeyU5SE0x2PZk9ebcJTNotk52kJpjseraJhLfJE2HP1rPu\n40ljYbKT1AST3TUscyjdUeloXqLa1GOvYvoxHVdUC0x2kppgs5PUBFdjr2GZjQuLrgq2sqrorjca\nGpOdpCaY7BawyO4aJpn5nC8aCpOdpCaY7JawyO4aY0ky296peCzzRfvLZCepCSa7FY01qYytXmlT\nTHaSmmCyW9M6h4Ft01AS3VgTscbPZCepCSa7DZlNKENPelJrTHaSmmCy68lQxqIcG5MmTHaSmmCz\nk9QEm52kJtjsJDXBDRTSSB22e5Mbo+Yz2UlqgslOW+WuMKs7akd15+m1mewkNeHIZpfkk0muJnl8\natovJ/lqkq8k+d0kb5y67v4kzyS5lOTdfRUutaKq5qa6JK/60bUtkuw+Bdw1M+1R4I6q+mHga8D9\nAEluB04Db+7u8/EkxzZWrSSt6MhmV1V/DHxzZtoXqurl7uKfADd3f98NfLqq/raqngWeAd66wXq1\npIOl/mHpYFsOnt8UsjqT3Ho2MWb3QeD3ur9vAp6fuu5yN+17JLk3yYUkFzZQgyRd01pbY5N8FHgZ\neHDZ+1bVWeBs9zh+H9KemU2RJpHlueV6s1Zudkl+BngvcGd99539AnDL1M1u7qZJ0k6ttBqb5C7g\nF4H3VdXfTF11Djid5PoktwEngS+uX6bWNTt2N/2zqnmPNTs25/jS6px3m3VkskvyEPAO4E1JLgO/\nxGTr6/XAo92L8SdV9W+q6okkDwNPMlm9/XBV/V1fxUvSojKErw93zG63Vn0PmDo0EBer6tRRN/II\nCklN8NhYmdDUBJOdpCbY7CQ1wWYnqQk2O0lNsNlJaoLNTlITbHaSmmCzk9SEoexU/JfAt7rfY/Am\nxlHrWOqE8dQ6ljphPLWuW+c/WeRGgzg2FiDJhUWObxuCsdQ6ljphPLWOpU4YT63bqtPVWElNsNlJ\nasKQmt3ZXRewhLHUOpY6YTy1jqVOGE+tW6lzMGN2ktSnISU7SeqNzU5SEwbR7JLcleRSkmeS3Lfr\neg4kuSXJHyV5MskTST7STb8xyaNJnu5+37DrWgGSHEvy5SSf6y4Ptc43JvlMkq8meSrJ2wdc6893\nr/3jSR5K8toh1Jrkk0muJnl8atqhdSW5v/t8XUry7gHU+svd6/+VJL+b5I1917rzZpfkGPAfgZ8C\nbgd+Osntu63qFS8Dv1BVtwNvAz7c1XYfcL6qTgLnu8tD8BHgqanLQ63z14Hfr6ofAn6ESc2DqzXJ\nTcDPAqeq6g7gGHCaYdT6KeCumWlz6+res6eBN3f3+Xj3uduWT/G9tT4K3FFVPwx8jclJvPqt9Vqn\nw9vGD/B24PNTl+8H7t91XYfU+gjwLuAScKKbdgK4NIDabmbyBv9J4HPdtCHW+f3As3Qbx6amD7HW\nm4DngRuZHG30OeCfD6VW4Fbg8aPm4exnCvg88PZd1jpz3b8EHuy71p0nO777hjpwuZs2KEluBd4C\nPAYcr6or3VUvAsd3VNa0X2NyLt/vTE0bYp23Ad8Afqtb5f5EktczwFqr6gXgV4CvA1eAv6qqLzDA\nWjuH1TX0z9gHgd/r/u6t1iE0u8FL8gbgd4Cfq6q/nr6uJoufne6/k+S9wNWqunjYbYZQZ+c64EeB\n36iqtzA5JvpVq4FDqbUb87qbSYP+AeD1ST4wfZuh1DprqHXNSvJRJsNFD/b9XENodi8At0xdvrmb\nNghJvo9Jo3uwqj7bTX4pyYnu+hPA1V3V1/lx4H1J/gL4NPCTSX6b4dUJkyX15ap6rLv8GSbNb4i1\nvhN4tqq+UVXfBj4L/BjDrBUOr2uQn7EkPwO8F/hXXXOGHmsdQrP7EnAyyW1JXsNkcPLcjmsCIJNz\nDP4m8FRV/erUVeeAM93fZ5iM5e1MVd1fVTdX1a1M5t8fVtUHGFidAFX1IvB8kh/sJt0JPMkAa2Wy\n+vq2JK/r3gt3MtmYMsRa4fC6zgGnk1yf5DbgJPDFHdT3iiR3MRl2eV9V/c3UVf3VuouB1TkDlO9h\nskXmfwEf3XU9U3X9BJNVga8Af9b9vAf4h0w2BjwN/AFw465rnar5HXx3A8Ug6wT+GXChm6//Fbhh\nwLX+B+CrwOPAfwauH0KtwENMxhG/zSQtf+hadQEf7T5fl4CfGkCtzzAZmzv4XP2nvmv1cDFJTRjC\naqwk9c5mJ6kJNjtJTbDZSWqCzU5SE2x2kppgs5PUhP8PfOp98vHhkc8AAAAASUVORK5CYII=\n",
"text/plain": [
"![\"Cython](\"images/cython_vs_numba.png\")
\n",
"\n",
"Here is a [thorough answer](https://softwareengineering.stackexchange.com/questions/246094/understanding-the-differences-traditional-interpreter-jit-compiler-jit-interp) on the differences between an Ahead Of Time (AOT) compiler, a Just In Time (JIT) compiler, and an interpreter."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Experiments with vectorization and native code"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's first get aquainted with Numba, and then we will return to our problem of polynomial features for regression on the diabates data set."
]
},
{
"cell_type": "code",
"execution_count": 140,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 141,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import math, numpy as np, matplotlib.pyplot as plt\n",
"from pandas_summary import DataFrameSummary\n",
"from scipy import ndimage"
]
},
{
"cell_type": "code",
"execution_count": 142,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from numba import jit, vectorize, guvectorize, cuda, float32, void, float64"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will show the impact of:\n",
"- Avoiding memory allocations and copies (slower than CPU calculations)\n",
"- Better locality\n",
"- Vectorization\n",
"\n",
"If we use numpy on whole arrays at a time, it creates lots of temporaries, and can't use cache. If we use numba looping through an array item at a time, then we don't have to allocate large temporary arrays, and can reuse cached data since we're doing multiple calculations on each array item."
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Untype and Unvectorized\n",
"def proc_python(xx,yy):\n",
" zz = np.zeros(nobs, dtype='float32')\n",
" for j in range(nobs): \n",
" x, y = xx[j], yy[j] \n",
" x = x*2 - ( y * 55 )\n",
" y = x + y*2 \n",
" z = x + y + 99 \n",
" z = z * ( z - .88 ) \n",
" zz[j] = z \n",
" return zz"
]
},
{
"cell_type": "code",
"execution_count": 176,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"nobs = 10000\n",
"x = np.random.randn(nobs).astype('float32')\n",
"y = np.random.randn(nobs).astype('float32')"
]
},
{
"cell_type": "code",
"execution_count": 177,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"49.8 ms ± 1.19 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
"source": [
"%timeit proc_python(x,y) # Untyped and unvectorized"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Numpy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Numpy lets us vectorize this:"
]
},
{
"cell_type": "code",
"execution_count": 146,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# Typed and Vectorized\n",
"def proc_numpy(x,y):\n",
" z = np.zeros(nobs, dtype='float32')\n",
" x = x*2 - ( y * 55 )\n",
" y = x + y*2 \n",
" z = x + y + 99 \n",
" z = z * ( z - .88 ) \n",
" return z"
]
},
{
"cell_type": "code",
"execution_count": 147,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 147,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose( proc_numpy(x,y), proc_python(x,y), atol=1e-4 )"
]
},
{
"cell_type": "code",
"execution_count": 148,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"35.9 µs ± 166 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
]
}
],
"source": [
"%timeit proc_numpy(x,y) # Typed and vectorized"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Numba"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Numba offers several different decorators. We will try two different ones:\n",
"\n",
"- `@jit`: very general\n",
"- `@vectorize`: don't need to write a for loop. useful when operating on vectors of the same size"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"First, we will use Numba's jit (just-in-time) compiler decorator, without explicitly vectorizing. This avoids large memory allocations, so we have better locality:"
]
},
{
"cell_type": "code",
"execution_count": 149,
"metadata": {
"collapsed": true,
"scrolled": false
},
"outputs": [],
"source": [
"@jit()\n",
"def proc_numba(xx,yy,zz):\n",
" for j in range(nobs): \n",
" x, y = xx[j], yy[j] \n",
" x = x*2 - ( y * 55 )\n",
" y = x + y*2 \n",
" z = x + y + 99 \n",
" z = z * ( z - .88 ) \n",
" zz[j] = z \n",
" return zz"
]
},
{
"cell_type": "code",
"execution_count": 150,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 150,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"z = np.zeros(nobs).astype('float32')\n",
"np.allclose( proc_numpy(x,y), proc_numba(x,y,z), atol=1e-4 )"
]
},
{
"cell_type": "code",
"execution_count": 151,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6.4 µs ± 17.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
]
}
],
"source": [
"%timeit proc_numba(x,y,z)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we will use Numba's `vectorize` decorator. Numba's compiler optimizes this in a smarter way than what is possible with plain Python and Numpy."
]
},
{
"cell_type": "code",
"execution_count": 152,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"@vectorize\n",
"def vec_numba(x,y):\n",
" x = x*2 - ( y * 55 )\n",
" y = x + y*2 \n",
" z = x + y + 99 \n",
" return z * ( z - .88 ) "
]
},
{
"cell_type": "code",
"execution_count": 153,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 153,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(vec_numba(x,y), proc_numba(x,y,z), atol=1e-4 )"
]
},
{
"cell_type": "code",
"execution_count": 154,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"5.82 µs ± 14.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
]
}
],
"source": [
"%timeit vec_numba(x,y)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Numba is **amazing**. Look how fast this is!"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Numba polynomial features"
]
},
{
"cell_type": "code",
"execution_count": 155,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"@jit(nopython=True)\n",
"def vec_poly(x, res):\n",
" m,n=x.shape\n",
" feat_idx=0\n",
" for i in range(n):\n",
" v1=x[:,i]\n",
" for k in range(m): res[k,feat_idx] = v1[k]\n",
" feat_idx+=1\n",
" for j in range(i,n):\n",
" for k in range(m): res[k,feat_idx] = v1[k]*x[k,j]\n",
" feat_idx+=1"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Row-Major vs Column-Major Storage"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"From this [blog post by Eli Bendersky](http://eli.thegreenplace.net/2015/memory-layout-of-multi-dimensional-arrays/):\n",
"\n",
"\"The row-major layout of a matrix puts the first row in contiguous memory, then the second row right after it, then the third, and so on. Column-major layout puts the first column in contiguous memory, then the second, etc.... While knowing which layout a particular data set is using is critical for good performance, there's no single answer to the question which layout 'is better' in general.\n",
"\n",
"\"It turns out that matching the way your algorithm works with the data layout can make or break the performance of an application.\n",
"\n",
"\"The short takeaway is: **always traverse the data in the order it was laid out**.\"\n",
"\n",
"**Column-major layout**: Fortran, Matlab, R, and Julia\n",
"\n",
"**Row-major layout**: C, C++, Python, Pascal, Mathematica"
]
},
{
"cell_type": "code",
"execution_count": 156,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"trn = np.asfortranarray(trn)\n",
"test = np.asfortranarray(test)"
]
},
{
"cell_type": "code",
"execution_count": 157,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"m,n=trn.shape\n",
"n_feat = n*(n+1)//2 + n\n",
"trn_feat = np.zeros((m,n_feat), order='F')\n",
"test_feat = np.zeros((len(y_test), n_feat), order='F')"
]
},
{
"cell_type": "code",
"execution_count": 158,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"vec_poly(trn, trn_feat)\n",
"vec_poly(test, test_feat)"
]
},
{
"cell_type": "code",
"execution_count": 159,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"LinearRegression(copy_X=True, fit_intercept=True, n_jobs=1, normalize=False)"
]
},
"execution_count": 159,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"regr.fit(trn_feat, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 160,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(55.74734592292935, 42.836164292252306)"
]
},
"execution_count": 160,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"regr_metrics(y_test, regr.predict(test_feat))"
]
},
{
"cell_type": "code",
"execution_count": 161,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"7.33 µs ± 19.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n"
]
}
],
"source": [
"%timeit vec_poly(trn, trn_feat)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Recall, this was the time from the scikit learn implementation PolynomialFeatures, which was created by experts: "
]
},
{
"cell_type": "code",
"execution_count": 136,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"635 µs ± 9.25 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit poly.fit_transform(trn)"
]
},
{
"cell_type": "code",
"execution_count": 162,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"78.57142857142857"
]
},
"execution_count": 162,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"605/7.7"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"This is a big deal! Numba is **amazing**! With a single line of code, we are getting a 78x speed-up over scikit learn (which was optimized by experts)."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Regularization and noise"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Regularization is a way to reduce over-fitting and create models that better generalize to new data."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### Regularization"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Lasso regression uses an L1 penalty, which pushes towards sparse coefficients. The parameter $\\alpha$ is used to weight the penalty term. Scikit Learn's LassoCV performs cross validation with a number of different values for $\\alpha$.\n",
"\n",
"Watch this [Coursera video on Lasso regression](https://www.coursera.org/learn/machine-learning-data-analysis/lecture/0KIy7/what-is-lasso-regression) for more info."
]
},
{
"cell_type": "code",
"execution_count": 163,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"reg_regr = linear_model.LassoCV(n_alphas=10)"
]
},
{
"cell_type": "code",
"execution_count": 164,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/home/jhoward/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:484: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.\n",
" ConvergenceWarning)\n"
]
},
{
"data": {
"text/plain": [
"LassoCV(alphas=None, copy_X=True, cv=None, eps=0.001, fit_intercept=True,\n",
" max_iter=1000, n_alphas=10, n_jobs=1, normalize=False, positive=False,\n",
" precompute='auto', random_state=None, selection='cyclic', tol=0.0001,\n",
" verbose=False)"
]
},
"execution_count": 164,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reg_regr.fit(trn_feat, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 165,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"0.0098199431661591518"
]
},
"execution_count": 165,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"reg_regr.alpha_"
]
},
{
"cell_type": "code",
"execution_count": 166,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(50.0982471642817, 40.065199085003101)"
]
},
"execution_count": 166,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"regr_metrics(y_test, reg_regr.predict(test_feat))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"### Noise"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Now we will add some noise to the data"
]
},
{
"cell_type": "code",
"execution_count": 167,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"idxs = np.random.randint(0, len(trn), 10)"
]
},
{
"cell_type": "code",
"execution_count": 168,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"y_trn2 = np.copy(y_trn)\n",
"y_trn2[idxs] *= 10 # label noise"
]
},
{
"cell_type": "code",
"execution_count": 169,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(51.1766253181518, 41.415992803872754)"
]
},
"execution_count": 169,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"regr = linear_model.LinearRegression()\n",
"regr.fit(trn, y_trn)\n",
"regr_metrics(y_test, regr.predict(test))"
]
},
{
"cell_type": "code",
"execution_count": 170,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(62.66110319520415, 53.21914420254862)"
]
},
"execution_count": 170,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"regr.fit(trn, y_trn2)\n",
"regr_metrics(y_test, regr.predict(test))"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Huber loss is a loss function that is less sensitive to outliers than squared error loss. It is quadratic for small error values, and linear for large values.\n",
"\n",
" $$L(x)= \n",
"\\begin{cases}\n",
" \\frac{1}{2}x^2, & \\text{for } \\lvert x\\rvert\\leq \\delta \\\\\n",
" \\delta(\\lvert x \\rvert - \\frac{1}{2}\\delta), & \\text{otherwise}\n",
"\\end{cases}$$"
]
},
{
"cell_type": "code",
"execution_count": 171,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(51.24055602541746, 41.670840571376822)"
]
},
"execution_count": 171,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"hregr = linear_model.HuberRegressor()\n",
"hregr.fit(trn, y_trn2)\n",
"regr_metrics(y_test, hregr.predict(test))"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"# End"
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
================================================
FILE: nbs/6. How to Implement Linear Regression.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 6. How to Implement Linear Regression"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"In the previous lesson, we calculated the least squares linear regression for a diabetes dataset, using scikit learn's implementation. Today, we will look at how we could write our own implementation."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn import datasets, linear_model, metrics\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import PolynomialFeatures\n",
"import math, scipy, numpy as np\n",
"from scipy import linalg"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"np.set_printoptions(precision=6)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"data = datasets.load_diabetes()"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"feature_names=['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"trn,test,y_trn,y_test = train_test_split(data.data, data.target, test_size=0.2)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((353, 10), (89, 10))"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trn.shape, test.shape"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true,
"scrolled": true
},
"outputs": [],
"source": [
"def regr_metrics(act, pred):\n",
" return (math.sqrt(metrics.mean_squared_error(act, pred)), \n",
" metrics.mean_absolute_error(act, pred))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### How did sklearn do it?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"How is sklearn doing this? By checking [the source code](https://github.com/scikit-learn/scikit-learn/blob/14031f6/sklearn/linear_model/base.py#L417), you can see that in the dense case, it calls [scipy.linalg.lstqr](https://github.com/scipy/scipy/blob/v0.19.0/scipy/linalg/basic.py#L892-L1058), which is calling a LAPACK method:\n",
"\n",
" Options are ``'gelsd'``, ``'gelsy'``, ``'gelss'``. Default\n",
" (``'gelsd'``) is a good choice. However, ``'gelsy'`` can be slightly\n",
" faster on many problems. ``'gelss'`` was used historically. It is\n",
" generally slow but uses less memory.\n",
"\n",
"- [gelsd](https://software.intel.com/sites/products/documentation/doclib/mkl_sa/11/mkl_lapack_examples/_gelsd.htm): uses SVD and a divide-and-conquer method\n",
"- [gelsy](https://software.intel.com/en-us/node/521113): uses QR factorization\n",
"- [gelss](https://software.intel.com/en-us/node/521114): uses SVD"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"#### Scipy Sparse Least Squares"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We will not get into too much detail about the sparse version of least squares. Here is a bit of info if you are interested: "
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"[Scipy sparse lsqr](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.linalg.lsqr.html#id1) uses an iterative method called [Golub and Kahan bidiagonalization](https://web.stanford.edu/class/cme324/paige-saunders2.pdf)."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"from [scipy sparse lsqr source code](https://github.com/scipy/scipy/blob/v0.14.0/scipy/sparse/linalg/isolve/lsqr.py#L96):\n",
" Preconditioning is another way to reduce the number of iterations. If it is possible to solve a related system ``M*x = b`` efficiently, where M approximates A in some helpful way (e.g. M - A has low rank or its elements are small relative to those of A), LSQR may converge more rapidly on the system ``A*M(inverse)*z = b``, after which x can be recovered by solving M\\*x = z.\n",
" \n",
"If A is symmetric, LSQR should not be used! Alternatives are the symmetric conjugate-gradient method (cg) and/or SYMMLQ. SYMMLQ is an implementation of symmetric cg that applies to any symmetric A and will converge more rapidly than LSQR. If A is positive definite, there are other implementations of symmetric cg that require slightly less work per iteration than SYMMLQ (but will take the same number of iterations)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### linalg.lstqr"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The sklearn implementation handled adding a constant term (since the y-intercept is presumably not 0 for the line we are learning) for us. We will need to do that by hand now:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"trn_int = np.c_[trn, np.ones(trn.shape[0])]\n",
"test_int = np.c_[test, np.ones(test.shape[0])]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since `linalg.lstsq` lets us specify which LAPACK routine we want to use, lets try them all and do some timing comparisons:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"290 µs ± 9.24 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit coef, _,_,_ = linalg.lstsq(trn_int, y_trn, lapack_driver=\"gelsd\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"140 µs ± 91.7 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
]
}
],
"source": [
"%timeit coef, _,_,_ = linalg.lstsq(trn_int, y_trn, lapack_driver=\"gelsy\")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"199 µs ± 228 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit coef, _,_,_ = linalg.lstsq(trn_int, y_trn, lapack_driver=\"gelss\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Naive Solution"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Recall that we want to find $\\hat{x}$ that minimizes: \n",
"$$ \\big\\vert\\big\\vert Ax - b \\big\\vert\\big\\vert_2$$\n",
"\n",
"Another way to think about this is that we are interested in where vector $b$ is closest to the subspace spanned by $A$ (called the *range of* $A$). This is the projection of $b$ onto $A$. Since $b - A\\hat{x}$ must be perpendicular to the subspace spanned by $A$, we see that\n",
"\n",
"$$A^T (b - A\\hat{x}) = 0 $$\n",
"\n",
"(we are using $A^T$ because we want to multiply each column of $A$ by $b - A\\hat{x}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This leads us to the *normal equations*:\n",
"$$ x = (A^TA)^{-1}A^T b $$"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ls_naive(A, b):\n",
" return np.linalg.inv(A.T @ A) @ A.T @ b"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"45.8 µs ± 4.65 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
]
}
],
"source": [
"%timeit coeffs_naive = ls_naive(trn_int, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(57.94102134545707, 48.053565198516438)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coeffs_naive = ls_naive(trn_int, y_trn)\n",
"regr_metrics(y_test, test_int @ coeffs_naive)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Normal Equations (Cholesky)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Normal equations:\n",
"$$ A^TA x = A^T b $$\n",
"\n",
"If $A$ has full rank, the pseudo-inverse $(A^TA)^{-1}A^T$ is a **square, hermitian positive definite** matrix. The standard way of solving such a system is *Cholesky Factorization*, which finds upper-triangular R s.t. $A^TA = R^TR$."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following steps are based on Algorithm 11.1 from Trefethen:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"A = trn_int"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"b = y_trn"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"AtA = A.T @ A\n",
"Atb = A.T @ b"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Warning:** Numpy and Scipy default to different upper/lower for Cholesky"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"R = scipy.linalg.cholesky(AtA)"
]
},
{
"cell_type": "code",
"execution_count": 196,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.9124, 0.1438, 0.1511, 0.3002, 0.2228, 0.188 ,\n",
" -0.051 , 0.1746, 0.22 , 0.2768, -0.2583],\n",
" [ 0. , 0.8832, 0.0507, 0.1826, -0.0251, 0.0928,\n",
" -0.3842, 0.2999, 0.0911, 0.15 , 0.4393],\n",
" [ 0. , 0. , 0.8672, 0.2845, 0.2096, 0.2153,\n",
" -0.2695, 0.3181, 0.3387, 0.2894, -0.005 ],\n",
" [ 0. , 0. , 0. , 0.7678, 0.0762, -0.0077,\n",
" 0.0383, 0.0014, 0.165 , 0.166 , 0.0234],\n",
" [ 0. , 0. , 0. , 0. , 0.8288, 0.7381,\n",
" 0.1145, 0.4067, 0.3494, 0.158 , -0.2826],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0.3735,\n",
" -0.3891, 0.2492, -0.3245, -0.0323, -0.1137],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0. ,\n",
" 0.6406, -0.511 , -0.5234, -0.172 , -0.9392],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0. ,\n",
" 0. , 0.2887, -0.0267, -0.0062, 0.0643],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0. ,\n",
" 0. , 0. , 0.2823, 0.0636, 0.9355],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0. ,\n",
" 0. , 0. , 0. , 0.7238, 0.0202],\n",
" [ 0. , 0. , 0. , 0. , 0. , 0. ,\n",
" 0. , 0. , 0. , 0. , 18.7319]])"
]
},
"execution_count": 196,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.set_printoptions(suppress=True, precision=4)\n",
"R"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"check our factorization:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4.5140158187158533e-16"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.norm(AtA - R.T @ R)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ A^T A x = A^T b $$\n",
"$$ R^T R x = A^T b $$\n",
"$$ R^T w = A^T b $$\n",
"$$ R x = w $$"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"w = scipy.linalg.solve_triangular(R, Atb, lower=False, trans='T')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It's always good to check that our result is what we expect it to be: (in case we entered the wrong params, the function didn't return what we thought, or sometimes the docs are even outdated)"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1.1368683772161603e-13"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.norm(R.T @ w - Atb)"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"coeffs_chol = scipy.linalg.solve_triangular(R, w, lower=False)"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"6.861429794408013e-14"
]
},
"execution_count": 24,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.norm(R @ coeffs_chol - w)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ls_chol(A, b):\n",
" R = scipy.linalg.cholesky(A.T @ A)\n",
" w = scipy.linalg.solve_triangular(R, A.T @ b, trans='T')\n",
" return scipy.linalg.solve_triangular(R, w)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"111 µs ± 272 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n"
]
}
],
"source": [
"%timeit coeffs_chol = ls_chol(trn_int, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(57.9410213454571, 48.053565198516438)"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coeffs_chol = ls_chol(trn_int, y_trn)\n",
"regr_metrics(y_test, test_int @ coeffs_chol)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### QR Factorization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ A x = b $$\n",
"$$ A = Q R $$\n",
"$$ Q R x = b $$\n",
"\n",
"$$ R x = Q^T b $$"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ls_qr(A,b):\n",
" Q, R = scipy.linalg.qr(A, mode='economic')\n",
" return scipy.linalg.solve_triangular(R, Q.T @ b)"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"205 µs ± 264 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit coeffs_qr = ls_qr(trn_int, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(57.94102134545711, 48.053565198516452)"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coeffs_qr = ls_qr(trn_int, y_trn)\n",
"regr_metrics(y_test, test_int @ coeffs_qr)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$ A x = b $$\n",
"\n",
"$$ A = U \\Sigma V $$\n",
"\n",
"$$ \\Sigma V x = U^T b $$\n",
"\n",
"$$ \\Sigma w = U^T b $$\n",
"\n",
"$$ x = V^T w $$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"SVD gives the pseudo-inverse"
]
},
{
"cell_type": "code",
"execution_count": 253,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def ls_svd(A,b):\n",
" m, n = A.shape\n",
" U, sigma, Vh = scipy.linalg.svd(A, full_matrices=False, lapack_driver='gesdd')\n",
" w = (U.T @ b)/ sigma\n",
" return Vh.T @ w"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.11 ms ± 320 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit coeffs_svd = ls_svd(trn_int, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 254,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"266 µs ± 8.49 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)\n"
]
}
],
"source": [
"%timeit coeffs_svd = ls_svd(trn_int, y_trn)"
]
},
{
"cell_type": "code",
"execution_count": 255,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(57.941021345457244, 48.053565198516687)"
]
},
"execution_count": 255,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"coeffs_svd = ls_svd(trn_int, y_trn)\n",
"regr_metrics(y_test, test_int @ coeffs_svd)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Random Sketching Technique for Least Squares Regression"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"[Linear Sketching](http://researcher.watson.ibm.com/researcher/files/us-dpwoodru/journal.pdf) (Woodruff)\n",
"\n",
"1. Sample a r x n random matrix S, r << n\n",
"2. Compute S A and S b\n",
"3. Find exact solution x to regression SA x = Sb"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Timing Comparison"
]
},
{
"cell_type": "code",
"execution_count": 244,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import timeit\n",
"import pandas as pd"
]
},
{
"cell_type": "code",
"execution_count": 245,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def scipylstq(A, b):\n",
" return scipy.linalg.lstsq(A,b)[0]"
]
},
{
"cell_type": "code",
"execution_count": 246,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"row_names = ['Normal Eqns- Naive',\n",
" 'Normal Eqns- Cholesky', \n",
" 'QR Factorization', \n",
" 'SVD', \n",
" 'Scipy lstsq']\n",
"\n",
"name2func = {'Normal Eqns- Naive': 'ls_naive', \n",
" 'Normal Eqns- Cholesky': 'ls_chol', \n",
" 'QR Factorization': 'ls_qr',\n",
" 'SVD': 'ls_svd',\n",
" 'Scipy lstsq': 'scipylstq'}"
]
},
{
"cell_type": "code",
"execution_count": 247,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"m_array = np.array([100, 1000, 10000])\n",
"n_array = np.array([20, 100, 1000])"
]
},
{
"cell_type": "code",
"execution_count": 248,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"index = pd.MultiIndex.from_product([m_array, n_array], names=['# rows', '# cols'])"
]
},
{
"cell_type": "code",
"execution_count": 249,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"pd.options.display.float_format = '{:,.6f}'.format\n",
"df = pd.DataFrame(index=row_names, columns=index)\n",
"df_error = pd.DataFrame(index=row_names, columns=index)"
]
},
{
"cell_type": "code",
"execution_count": 256,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# %%prun\n",
"for m in m_array:\n",
" for n in n_array:\n",
" if m >= n: \n",
" x = np.random.uniform(-10,10,n)\n",
" A = np.random.uniform(-40,40,[m,n]) # removed np.asfortranarray\n",
" b = np.matmul(A, x) + np.random.normal(0,2,m)\n",
" for name in row_names:\n",
" fcn = name2func[name]\n",
" t = timeit.timeit(fcn + '(A,b)', number=5, globals=globals())\n",
" df.set_value(name, (m,n), t)\n",
" coeffs = locals()[fcn](A, b)\n",
" reg_met = regr_metrics(b, A @ coeffs)\n",
" df_error.set_value(name, (m,n), reg_met[0])"
]
},
{
"cell_type": "code",
"execution_count": 257,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
"# rows 100 1000 \\\n",
"# cols 20 100 1000 20 100 1000 \n",
"Normal Eqns- Naive 0.001276 0.003634 NaN 0.000960 0.005172 0.293126 \n",
"Normal Eqns- Cholesky 0.001660 0.003958 NaN 0.001665 0.004007 0.093696 \n",
"QR Factorization 0.002174 0.006486 NaN 0.004235 0.017773 0.213232 \n",
"SVD 0.003880 0.021737 NaN 0.004672 0.026950 1.280490 \n",
"Scipy lstsq 0.004338 0.020198 NaN 0.004320 0.021199 1.083804 \n",
"\n",
"# rows 10000 \n",
"# cols 20 100 1000 \n",
"Normal Eqns- Naive 0.002226 0.021248 1.164655 \n",
"Normal Eqns- Cholesky 0.001928 0.010456 0.399464 \n",
"QR Factorization 0.019229 0.116122 2.208129 \n",
"SVD 0.018138 0.130652 3.433003 \n",
"Scipy lstsq 0.012200 0.088467 2.134780 "
]
},
"execution_count": 257,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "code",
"execution_count": 252,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| # rows | \n", "100 | \n", "1000 | \n", "10000 | \n", "||||||
|---|---|---|---|---|---|---|---|---|---|
| # cols | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "
| Normal Eqns- Naive | \n", "0.001276 | \n", "0.003634 | \n", "NaN | \n", "0.000960 | \n", "0.005172 | \n", "0.293126 | \n", "0.002226 | \n", "0.021248 | \n", "1.164655 | \n", "
| Normal Eqns- Cholesky | \n", "0.001660 | \n", "0.003958 | \n", "NaN | \n", "0.001665 | \n", "0.004007 | \n", "0.093696 | \n", "0.001928 | \n", "0.010456 | \n", "0.399464 | \n", "
| QR Factorization | \n", "0.002174 | \n", "0.006486 | \n", "NaN | \n", "0.004235 | \n", "0.017773 | \n", "0.213232 | \n", "0.019229 | \n", "0.116122 | \n", "2.208129 | \n", "
| SVD | \n", "0.003880 | \n", "0.021737 | \n", "NaN | \n", "0.004672 | \n", "0.026950 | \n", "1.280490 | \n", "0.018138 | \n", "0.130652 | \n", "3.433003 | \n", "
| Scipy lstsq | \n", "0.004338 | \n", "0.020198 | \n", "NaN | \n", "0.004320 | \n", "0.021199 | \n", "1.083804 | \n", "0.012200 | \n", "0.088467 | \n", "2.134780 | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
"# rows 100 1000 \\\n",
"# cols 20 100 1000 20 100 1000 \n",
"Normal Eqns- Naive 1.702742 0.000000 NaN 1.970767 1.904873 0.000000 \n",
"Normal Eqns- Cholesky 1.702742 0.000000 NaN 1.970767 1.904873 0.000000 \n",
"QR Factorization 1.702742 0.000000 NaN 1.970767 1.904873 0.000000 \n",
"SVD 1.702742 0.000000 NaN 1.970767 1.904873 0.000000 \n",
"Scipy lstsq 1.702742 0.000000 NaN 1.970767 1.904873 0.000000 \n",
"\n",
"# rows 10000 \n",
"# cols 20 100 1000 \n",
"Normal Eqns- Naive 1.978383 1.980449 1.884440 \n",
"Normal Eqns- Cholesky 1.978383 1.980449 1.884440 \n",
"QR Factorization 1.978383 1.980449 1.884440 \n",
"SVD 1.978383 1.980449 1.884440 \n",
"Scipy lstsq 1.978383 1.980449 1.884440 "
]
},
"execution_count": 252,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df_error"
]
},
{
"cell_type": "code",
"execution_count": 618,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"store = pd.HDFStore('least_squares_results.h5')"
]
},
{
"cell_type": "code",
"execution_count": 619,
"metadata": {
"collapsed": true
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"C:\\Users\\rache\\Anaconda3\\lib\\site-packages\\IPython\\core\\interactiveshell.py:2881: PerformanceWarning: \n",
"your performance may suffer as PyTables will pickle object types that it cannot\n",
"map directly to c-types [inferred_type->floating,key->block0_values] [items->[(100, 20), (100, 100), (100, 1000), (1000, 20), (1000, 100), (1000, 1000), (5000, 20), (5000, 100), (5000, 1000)]]\n",
"\n",
" exec(code_obj, self.user_global_ns, self.user_ns)\n"
]
}
],
"source": [
"store['df'] = df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Notes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I used the magick %prun to profile my code."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternative: least absolute deviation (L1 regression)\n",
"- Less sensitive to outliers than least squares.\n",
"- No closed form solution, but can solve with linear programming"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Conditioning & stability"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Condition Number"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"*Condition number* is a measure of how small changes to the input cause the output to change.\n",
"\n",
"**Question**: Why do we care about behavior with small changes to the input in numerical linear algebra?"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The *relative condition number* is defined by\n",
"\n",
"$$ \\kappa = \\sup_{\\delta x} \\frac{\\|\\delta f\\|}{\\| f(x) \\|}\\bigg/ \\frac{\\| \\delta x \\|}{\\| x \\|} $$\n",
" \n",
"where $\\delta x$ is infinitesimal\n",
"\n",
"According to Trefethen (pg. 91), a problem is *well-conditioned* if $\\kappa$ is small (e.g. $1$, $10$, $10^2$) and *ill-conditioned* if $\\kappa$ is large (e.g. $10^6$, $10^{16}$)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Conditioning**: perturbation behavior of a mathematical problem (e.g. least squares)\n",
"\n",
"**Stability**: perturbation behavior of an algorithm used to solve that problem on a computer (e.g. least squares algorithms, householder, back substitution, gaussian elimination)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Conditioning example"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The problem of computing eigenvalues of a non-symmetric matrix is often ill-conditioned"
]
},
{
"cell_type": "code",
"execution_count": 178,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"A = [[1, 1000], [0, 1]]\n",
"B = [[1, 1000], [0.001, 1]]"
]
},
{
"cell_type": "code",
"execution_count": 179,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"wA, vrA = scipy.linalg.eig(A)\n",
"wB, vrB = scipy.linalg.eig(B)"
]
},
{
"cell_type": "code",
"execution_count": 180,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(array([ 1.+0.j, 1.+0.j]),\n",
" array([ 2.00000000e+00+0.j, -2.22044605e-16+0.j]))"
]
},
"execution_count": 180,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"wA, wB"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Condition Number of a Matrix"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The product $\\| A\\| \\|A^{-1} \\|$ comes up so often it has its own name: the *condition number* of $A$. Note that normally we talk about the conditioning of problems, not matrices.\n",
"\n",
"The *condition number* of $A$ relates to:\n",
"- computing $b$ given $A$ and $x$ in $Ax = b$\n",
"- computing $x$ given $A$ and $b$ in $Ax = b$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Loose ends from last time"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Full vs Reduced Factorizations"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**SVD**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Diagrams from Trefethen:\n",
"\n",
"| # rows | \n", "100 | \n", "1000 | \n", "10000 | \n", "||||||
|---|---|---|---|---|---|---|---|---|---|
| # cols | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "
| Normal Eqns- Naive | \n", "1.702742 | \n", "0.000000 | \n", "NaN | \n", "1.970767 | \n", "1.904873 | \n", "0.000000 | \n", "1.978383 | \n", "1.980449 | \n", "1.884440 | \n", "
| Normal Eqns- Cholesky | \n", "1.702742 | \n", "0.000000 | \n", "NaN | \n", "1.970767 | \n", "1.904873 | \n", "0.000000 | \n", "1.978383 | \n", "1.980449 | \n", "1.884440 | \n", "
| QR Factorization | \n", "1.702742 | \n", "0.000000 | \n", "NaN | \n", "1.970767 | \n", "1.904873 | \n", "0.000000 | \n", "1.978383 | \n", "1.980449 | \n", "1.884440 | \n", "
| SVD | \n", "1.702742 | \n", "0.000000 | \n", "NaN | \n", "1.970767 | \n", "1.904873 | \n", "0.000000 | \n", "1.978383 | \n", "1.980449 | \n", "1.884440 | \n", "
| Scipy lstsq | \n", "1.702742 | \n", "0.000000 | \n", "NaN | \n", "1.970767 | \n", "1.904873 | \n", "0.000000 | \n", "1.978383 | \n", "1.980449 | \n", "1.884440 | \n", "
![\"\"](\"images/full_svd.JPG\")
\n",
"\n",
"![\"\"](\"images/reduced_svd.JPG\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**QR Factorization exists for ALL matrices**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Just like with SVD, there are full and reduced versions of the QR factorization.\n",
"\n",
"![\"\"](\"images/full_qr.JPG\")
\n",
"\n",
"![\"\"](\"images/reduced_qr.JPG\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Matrix Inversion is Unstable"
]
},
{
"cell_type": "code",
"execution_count": 197,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from scipy.linalg import hilbert"
]
},
{
"cell_type": "code",
"execution_count": 229,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"n = 14\n",
"A = hilbert(n)\n",
"x = np.random.uniform(-10,10,n)\n",
"b = A @ x"
]
},
{
"cell_type": "code",
"execution_count": 230,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"A_inv = np.linalg.inv(A)"
]
},
{
"cell_type": "code",
"execution_count": 233,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"5.0516495470543212"
]
},
"execution_count": 233,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.norm(np.eye(n) - A @ A_inv)"
]
},
{
"cell_type": "code",
"execution_count": 231,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"2.2271635826494112e+17"
]
},
"execution_count": 231,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.cond(A)"
]
},
{
"cell_type": "code",
"execution_count": 232,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1. , 0. , -0.0001, 0.0005, -0.0006, 0.0105, -0.0243,\n",
" 0.1862, -0.6351, 2.2005, -0.8729, 0.8925, -0.0032, -0.0106],\n",
" [ 0. , 1. , -0. , 0. , 0.0035, 0.0097, -0.0408,\n",
" 0.0773, -0.0524, 1.6926, -0.7776, -0.111 , -0.0403, -0.0184],\n",
" [ 0. , 0. , 1. , 0.0002, 0.0017, 0.0127, -0.0273,\n",
" 0. , 0. , 1.4688, -0.5312, 0.2812, 0.0117, 0.0264],\n",
" [ 0. , 0. , -0. , 1.0005, 0.0013, 0.0098, -0.0225,\n",
" 0.1555, -0.0168, 1.1571, -0.9656, -0.0391, 0.018 , -0.0259],\n",
" [-0. , 0. , -0. , 0.0007, 1.0001, 0.0154, 0.011 ,\n",
" -0.2319, 0.5651, -0.2017, 0.2933, -0.6565, 0.2835, -0.0482],\n",
" [ 0. , -0. , 0. , -0.0004, 0.0059, 0.9945, -0.0078,\n",
" -0.0018, -0.0066, 1.1839, -0.9919, 0.2144, -0.1866, 0.0187],\n",
" [-0. , 0. , -0. , 0.0009, -0.002 , 0.0266, 0.974 ,\n",
" -0.146 , 0.1883, -0.2966, 0.4267, -0.8857, 0.2265, -0.0453],\n",
" [ 0. , 0. , -0. , 0.0002, 0.0009, 0.0197, -0.0435,\n",
" 1.1372, -0.0692, 0.7691, -1.233 , 0.1159, -0.1766, -0.0033],\n",
" [ 0. , 0. , -0. , 0.0002, 0. , -0.0018, -0.0136,\n",
" 0.1332, 0.945 , 0.3652, -0.2478, -0.1682, 0.0756, -0.0212],\n",
" [ 0. , -0. , -0. , 0.0003, 0.0038, -0.0007, 0.0318,\n",
" -0.0738, 0.2245, 1.2023, -0.2623, -0.2783, 0.0486, -0.0093],\n",
" [-0. , 0. , -0. , 0.0004, -0.0006, 0.013 , -0.0415,\n",
" 0.0292, -0.0371, 0.169 , 1.0715, -0.09 , 0.1668, -0.0197],\n",
" [ 0. , -0. , 0. , 0. , 0.0016, 0.0062, -0.0504,\n",
" 0.1476, -0.2341, 0.8454, -0.7907, 1.4812, -0.15 , 0.0186],\n",
" [ 0. , -0. , 0. , -0.0001, 0.0022, 0.0034, -0.0296,\n",
" 0.0944, -0.1833, 0.6901, -0.6526, 0.2556, 0.8563, 0.0128],\n",
" [ 0. , 0. , 0. , -0.0001, 0.0018, -0.0041, -0.0057,\n",
" -0.0374, -0.165 , 0.3968, -0.2264, -0.1538, -0.0076, 1.005 ]])"
]
},
"execution_count": 232,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A @ A_inv"
]
},
{
"cell_type": "code",
"execution_count": 237,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"row_names = ['Normal Eqns- Naive',\n",
" 'QR Factorization', \n",
" 'SVD', \n",
" 'Scipy lstsq']\n",
"\n",
"name2func = {'Normal Eqns- Naive': 'ls_naive', \n",
" 'QR Factorization': 'ls_qr',\n",
" 'SVD': 'ls_svd',\n",
" 'Scipy lstsq': 'scipylstq'}"
]
},
{
"cell_type": "code",
"execution_count": 238,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"pd.options.display.float_format = '{:,.9f}'.format\n",
"df = pd.DataFrame(index=row_names, columns=['Time', 'Error'])"
]
},
{
"cell_type": "code",
"execution_count": 239,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"for name in row_names:\n",
" fcn = name2func[name]\n",
" t = timeit.timeit(fcn + '(A,b)', number=5, globals=globals())\n",
" coeffs = locals()[fcn](A, b)\n",
" df.set_value(name, 'Time', t)\n",
" df.set_value(name, 'Error', regr_metrics(b, A @ coeffs)[0])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### SVD is best here!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"DO NOT RERUN"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Time Error\n",
"Normal Eqns- Naive 0.001334339 3.598901966\n",
"QR Factorization 0.002166139 0.000000000\n",
"SVD 0.001556937 0.000000000\n",
"Scipy lstsq 0.001871590 0.000000000"
]
},
"execution_count": 240,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | Time | \n", "Error | \n", "
|---|---|---|
| Normal Eqns- Naive | \n", "0.001334339 | \n", "3.598901966 | \n", "
| QR Factorization | \n", "0.002166139 | \n", "0.000000000 | \n", "
| SVD | \n", "0.001556937 | \n", "0.000000000 | \n", "
| Scipy lstsq | \n", "0.001871590 | \n", "0.000000000 | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Time Error\n",
"Normal Eqns- Naive 0.001334339 3.598901966\n",
"QR Factorization 0.002166139 0.000000000\n",
"SVD 0.001556937 0.000000000\n",
"Scipy lstsq 0.001871590 0.000000000"
]
},
"execution_count": 240,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Another reason not to take inverse**"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Even if $A$ is incredibly sparse, $A^{-1}$ is generally dense. For large matrices, $A^{-1}$ could be so dense as to not fit in memory."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Runtime"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Matrix Inversion: $2n^3$\n",
"\n",
"Matrix Multiplication: $n^3$\n",
"\n",
"Cholesky: $\\frac{1}{3}n^3$\n",
"\n",
"QR, Gram Schmidt: $2mn^2$, $m\\geq n$ (chapter 8 of Trefethen)\n",
"\n",
"QR, Householder: $2mn^2 - \\frac{2}{3}n^3$ (chapter 10 of Trefethen)\n",
"\n",
"Solving a triangular system: $n^2$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Why Cholesky Factorization is Fast:**\n",
"\n",
"| \n", " | Time | \n", "Error | \n", "
|---|---|---|
| Normal Eqns- Naive | \n", "0.001334339 | \n", "3.598901966 | \n", "
| QR Factorization | \n", "0.002166139 | \n", "0.000000000 | \n", "
| SVD | \n", "0.001556937 | \n", "0.000000000 | \n", "
| Scipy lstsq | \n", "0.001871590 | \n", "0.000000000 | \n", "
![\"\"](\"images/cholesky_factorization_speed.png\")
\n",
"(source: [Stanford Convex Optimization: Numerical Linear Algebra Background Slides](http://stanford.edu/class/ee364a/lectures/num-lin-alg.pdf))"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### A Case Where QR is the Best"
]
},
{
"cell_type": "code",
"execution_count": 65,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"m=100\n",
"n=15\n",
"t=np.linspace(0, 1, m)"
]
},
{
"cell_type": "code",
"execution_count": 66,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"# Vandermonde matrix\n",
"A=np.stack([t**i for i in range(n)], 1)"
]
},
{
"cell_type": "code",
"execution_count": 67,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"b=np.exp(np.sin(4*t))\n",
"\n",
"# This will turn out to normalize the solution to be 1\n",
"b /= 2006.787453080206"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"from matplotlib import pyplot as plt\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 69,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"[\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Time Error\n",
"Normal Eqns- Naive 0.001565099 1.357066025\n",
"QR Factorization 0.002632104 0.000000116\n",
"SVD 0.003503785 0.000000116\n",
"Scipy lstsq 0.002763502 0.000000116"
]
},
"execution_count": 76,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"\n",
"The solution for least squares via the normal equations is unstable in general, although stable for problems with small condition numbers."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"### Low-rank"
]
},
{
"cell_type": "code",
"execution_count": 258,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"m = 100\n",
"n = 10\n",
"x = np.random.uniform(-10,10,n)\n",
"A2 = np.random.uniform(-40,40, [m, int(n/2)]) # removed np.asfortranarray\n",
"A = np.hstack([A2, A2])"
]
},
{
"cell_type": "code",
"execution_count": 259,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"((100, 10), (100, 5))"
]
},
"execution_count": 259,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A.shape, A2.shape"
]
},
{
"cell_type": "code",
"execution_count": 260,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"b = A @ x + np.random.normal(0,1,m)"
]
},
{
"cell_type": "code",
"execution_count": 263,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"row_names = ['Normal Eqns- Naive',\n",
" 'QR Factorization', \n",
" 'SVD', \n",
" 'Scipy lstsq']\n",
"\n",
"name2func = {'Normal Eqns- Naive': 'ls_naive', \n",
" 'QR Factorization': 'ls_qr',\n",
" 'SVD': 'ls_svd',\n",
" 'Scipy lstsq': 'scipylstq'}"
]
},
{
"cell_type": "code",
"execution_count": 264,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"pd.options.display.float_format = '{:,.9f}'.format\n",
"df = pd.DataFrame(index=row_names, columns=['Time', 'Error'])"
]
},
{
"cell_type": "code",
"execution_count": 265,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"for name in row_names:\n",
" fcn = name2func[name]\n",
" t = timeit.timeit(fcn + '(A,b)', number=5, globals=globals())\n",
" coeffs = locals()[fcn](A, b)\n",
" df.set_value(name, 'Time', t)\n",
" df.set_value(name, 'Error', regr_metrics(b, A @ coeffs)[0])"
]
},
{
"cell_type": "code",
"execution_count": 266,
"metadata": {
"hidden": true,
"scrolled": false
},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | Time | \n", "Error | \n", "
|---|---|---|
| Normal Eqns- Naive | \n", "0.001565099 | \n", "1.357066025 | \n", "
| QR Factorization | \n", "0.002632104 | \n", "0.000000116 | \n", "
| SVD | \n", "0.003503785 | \n", "0.000000116 | \n", "
| Scipy lstsq | \n", "0.002763502 | \n", "0.000000116 | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
" Time Error\n",
"Normal Eqns- Naive 0.001227640 300.658979382\n",
"QR Factorization 0.002315920 0.876019803\n",
"SVD 0.001745647 1.584746056\n",
"Scipy lstsq 0.002067989 0.804750398"
]
},
"execution_count": 266,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Comparison"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Our results from above:"
]
},
{
"cell_type": "code",
"execution_count": 257,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"| \n", " | Time | \n", "Error | \n", "
|---|---|---|
| Normal Eqns- Naive | \n", "0.001227640 | \n", "300.658979382 | \n", "
| QR Factorization | \n", "0.002315920 | \n", "0.876019803 | \n", "
| SVD | \n", "0.001745647 | \n", "1.584746056 | \n", "
| Scipy lstsq | \n", "0.002067989 | \n", "0.804750398 | \n", "
\n",
"\n",
"\n",
" \n",
"
\n",
"
"
],
"text/plain": [
"# rows 100 1000 \\\n",
"# cols 20 100 1000 20 100 1000 \n",
"Normal Eqns- Naive 0.001276 0.003634 NaN 0.000960 0.005172 0.293126 \n",
"Normal Eqns- Cholesky 0.001660 0.003958 NaN 0.001665 0.004007 0.093696 \n",
"QR Factorization 0.002174 0.006486 NaN 0.004235 0.017773 0.213232 \n",
"SVD 0.003880 0.021737 NaN 0.004672 0.026950 1.280490 \n",
"Scipy lstsq 0.004338 0.020198 NaN 0.004320 0.021199 1.083804 \n",
"\n",
"# rows 10000 \n",
"# cols 20 100 1000 \n",
"Normal Eqns- Naive 0.002226 0.021248 1.164655 \n",
"Normal Eqns- Cholesky 0.001928 0.010456 0.399464 \n",
"QR Factorization 0.019229 0.116122 2.208129 \n",
"SVD 0.018138 0.130652 3.433003 \n",
"Scipy lstsq 0.012200 0.088467 2.134780 "
]
},
"execution_count": 257,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"df"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"From Trefethen (page 84):\n",
"\n",
"Normal equations/Cholesky is fastest when it works. Cholesky can only be used on symmetric, positive definite matrices. Also, normal equations/Cholesky is unstable for matrices with high condition numbers or with low-rank.\n",
"\n",
"Linear regression via QR has been recommended by numerical analysts as the standard method for years. It is natural, elegant, and good for \"daily use\"."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/7. PageRank with Eigen Decompositions.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 7. PageRank with Eigen Decompositions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Two Handy Tricks"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here are two tools that we'll be using today, which are useful in general."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1\\. [Psutil](https://github.com/giampaolo/psutil) is a great way to check on your memory usage. This will be useful here since we are using a larger data set."
]
},
{
"cell_type": "code",
"execution_count": 462,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import psutil"
]
},
{
"cell_type": "code",
"execution_count": 447,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"process = psutil.Process(os.getpid())\n",
"t = process.memory_info()"
]
},
{
"cell_type": "code",
"execution_count": 449,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(19475513344, 17856520192)"
]
},
"execution_count": 449,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"t.vms, t.rss"
]
},
{
"cell_type": "code",
"execution_count": 450,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def mem_usage():\n",
" process = psutil.Process(os.getpid())\n",
" return process.memory_info().rss / psutil.virtual_memory().total"
]
},
{
"cell_type": "code",
"execution_count": 451,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"0.13217061955758594"
]
},
"execution_count": 451,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mem_usage()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2\\. [TQDM](https://github.com/tqdm/tqdm) gives you progress bars."
]
},
{
"cell_type": "code",
"execution_count": 364,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from time import sleep"
]
},
{
"cell_type": "code",
"execution_count": 463,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"45\n"
]
}
],
"source": [
"# Without TQDM\n",
"s = 0\n",
"for i in range(10):\n",
" s += i\n",
" sleep(0.2)\n",
"print(s)"
]
},
{
"cell_type": "code",
"execution_count": 465,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"100%|██████████| 10/10 [00:02<00:00, 4.96it/s]"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"45\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"\n"
]
}
],
"source": [
"# With TQDM\n",
"from tqdm import tqdm\n",
"\n",
"s = 0\n",
"for i in tqdm(range(10)):\n",
" s += i\n",
" sleep(0.2)\n",
"print(s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Motivation"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Review**\n",
"- What is SVD?\n",
"- What are some applications of SVD?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Additional SVD Application"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An interesting use of SVD that I recently came across was as a step in the de-biasing of Word2Vec word embeddings, from [Quantifying and Reducing Stereotypes in Word Embeddings](https://arxiv.org/pdf/1606.06121.pdf)(Bolukbasi, et al).\n",
"\n",
"Word2Vec is a useful library released by Google that represents words as vectors. The similarity of vectors captures semantic meaning, and analogies can be found, such as *Paris:France :: Tokyo: Japan*. \n",
"\n",
"| # rows | \n", "100 | \n", "1000 | \n", "10000 | \n", "||||||
|---|---|---|---|---|---|---|---|---|---|
| # cols | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "20 | \n", "100 | \n", "1000 | \n", "
| Normal Eqns- Naive | \n", "0.001276 | \n", "0.003634 | \n", "NaN | \n", "0.000960 | \n", "0.005172 | \n", "0.293126 | \n", "0.002226 | \n", "0.021248 | \n", "1.164655 | \n", "
| Normal Eqns- Cholesky | \n", "0.001660 | \n", "0.003958 | \n", "NaN | \n", "0.001665 | \n", "0.004007 | \n", "0.093696 | \n", "0.001928 | \n", "0.010456 | \n", "0.399464 | \n", "
| QR Factorization | \n", "0.002174 | \n", "0.006486 | \n", "NaN | \n", "0.004235 | \n", "0.017773 | \n", "0.213232 | \n", "0.019229 | \n", "0.116122 | \n", "2.208129 | \n", "
| SVD | \n", "0.003880 | \n", "0.021737 | \n", "NaN | \n", "0.004672 | \n", "0.026950 | \n", "1.280490 | \n", "0.018138 | \n", "0.130652 | \n", "3.433003 | \n", "
| Scipy lstsq | \n", "0.004338 | \n", "0.020198 | \n", "NaN | \n", "0.004320 | \n", "0.021199 | \n", "1.083804 | \n", "0.012200 | \n", "0.088467 | \n", "2.134780 | \n", "
![\"\"](\"images/word2vec_analogies.png\")
\n",
"(source: [Vector Representations of Words](https://www.tensorflow.org/versions/r0.10/tutorials/word2vec/))\n",
"\n",
"However, these embeddings can implicitly encode bias, such as *father:doctor :: mother:nurse* and *man:computer programmer :: woman:homemaker*.\n",
"\n",
"One approach for de-biasing the space involves using SVD to reduce the dimensionality ([Bolukbasi paper](https://arxiv.org/pdf/1606.06121.pdf)).\n",
"\n",
"You can read more about bias in word embeddings:\n",
"- [How Vector Space Mathematics Reveals the Hidden Sexism in Language](https://www.technologyreview.com/s/602025/how-vector-space-mathematics-reveals-the-hidden-sexism-in-language/)(MIT Tech Review)\n",
"- [ConceptNet: better, less-stereotyped word vectors](https://blog.conceptnet.io/2017/04/24/conceptnet-numberbatch-17-04-better-less-stereotyped-word-vectors/)\n",
"- [Semantics derived automatically from language corpora necessarily contain human biases](https://www.princeton.edu/~aylinc/papers/caliskan-islam_semantics.pdf) (excellent and very interesting paper!)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Ways to think about SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- Data compression\n",
"- SVD trades a large number of features for a smaller set of better features\n",
"- All matrices are diagonal (if you use change of bases on the domain and range)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Perspectives on SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We usually talk about SVD in terms of matrices, $$A = U \\Sigma V^T$$ but we can also think about it in terms of vectors. SVD gives us sets of orthonormal vectors ${v_j}$ and ${u_j}$ such that $$ A v_j = \\sigma_j u_j $$\n",
"\n",
"$\\sigma_j$ are scalars, called singular values\n",
"\n",
"**Q**: Does this remind you of anything?"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Answer"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Relationship between SVD and Eigen Decomposition**: the left-singular vectors of A are the eigenvectors of $AA^T$. The right-singular vectors of A are the eigenvectors of $A^T A$. The non-zero singular values of A are the square roots of the eigenvalues of $A^T A$ (and $A A^T$).\n",
"\n",
"SVD is a generalization of eigen decomposition. Not all matrices have eigen values, but ALL matrices have singular values.\n",
"\n",
"Let's forget SVD for a bit and talk about how to find the eigenvalues of a symmetric positive definite matrix..."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Further resources on SVD"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"- [SVD: image compression and least squares](http://andrew.gibiansky.com/blog/mathematics/cool-linear-algebra-singular-value-decomposition/)\n",
"\n",
"- [Image Compression with SVD](http://nbviewer.jupyter.org/gist/frankcleary/4d2bd178708503b556b0)\n",
"\n",
"- [The Extraordinary SVD](https://sites.math.washington.edu/~morrow/498_13/svd_applied.pdf)\n",
"\n",
"- [A Singularly Valuable Decomposition: The SVD of a Matrix](https://sites.math.washington.edu/~morrow/498_13/svd.pdf)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Today: Eigen Decomposition"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The best classical methods for computing the SVD are variants on methods for computing eigenvalues. In addition to their links to SVD, Eigen decompositions are useful on their own as well. Here are a few practical applications of eigen decomposition:\n",
"- [rapid matrix powers](http://www.onmyphd.com/?p=eigen.decomposition#h2_why)\n",
"- [nth Fibonacci number](http://mathproofs.blogspot.com/2005/04/nth-term-of-fibonacci-sequence.html)\n",
"- Behavior of ODEs\n",
"- Markov Chains (health care economics, Page Rank)\n",
"- [Linear Discriminant Analysis on Iris dataset](http://sebastianraschka.com/Articles/2014_python_lda.html)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check out the 3 Blue 1 Brown videos on [Change of basis](https://www.youtube.com/watch?v=P2LTAUO1TdA) and [Eigenvalues and eigenvectors](https://www.youtube.com/watch?v=PFDu9oVAE-g)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\"Eigenvalues are a way to see into the heart of a matrix... All the difficulties of matrices are swept away\" -Strang "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Vocab**: A **Hermitian** matrix is one that is equal to it's own conjugate transpose. In the case of real-valued matrices (which is all we are considering in this course), **Hermitian** means the same as **Symmetric**.\n",
"\n",
"**Relevant Theorems:**\n",
"- If A is symmetric, then eigenvalues of A are real and $A = Q \\Lambda Q^T$\n",
"- If A is triangular, then its eigenvalues are equal to its diagonal entries"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## DBpedia Dataset"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start with the **Power Method**, which finds one eigenvector. *What good is just one eigenvector?* you may be wondering. This is actually the basis for PageRank (read [The $25,000,000,000 Eigenvector: the Linear Algebra Behind Google](http://www.rose-hulman.edu/~bryan/googleFinalVersionFixed.pdf) for more info)\n",
"\n",
"Instead of trying to rank the importance of all websites on the internet, we are going to use a dataset of Wikipedia links from [DBpedia](http://wiki.dbpedia.org/). DBpedia provides structured Wikipedia data available in 125 languages. \n",
"\n",
"\"*The full DBpedia data set features 38 million labels and abstracts in 125 different languages, 25.2 million links to images and 29.8 million links to external web pages; 80.9 million links to Wikipedia categories, and 41.2 million links to [YAGO](http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/yago/) categories*\" --[about DBpedia](http://wiki.dbpedia.org/about)\n",
"\n",
"Today's lesson is inspired by this [SciKit Learn Example](http://scikit-learn.org/stable/auto_examples/applications/wikipedia_principal_eigenvector.html#sphx-glr-auto-examples-applications-wikipedia-principal-eigenvector-py)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Imports"
]
},
{
"cell_type": "code",
"execution_count": 361,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import os, numpy as np, pickle\n",
"from bz2 import BZ2File\n",
"from datetime import datetime\n",
"from pprint import pprint\n",
"from time import time\n",
"from tqdm import tqdm_notebook\n",
"from scipy import sparse\n",
"\n",
"from sklearn.decomposition import randomized_svd\n",
"from sklearn.externals.joblib import Memory\n",
"from urllib.request import urlopen"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Downloading the data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The data we have are:\n",
"- **redirects**: URLs that redirect to other URLs\n",
"- **links**: which pages link to which other pages"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: this takes a while"
]
},
{
"cell_type": "code",
"execution_count": 367,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"PATH = 'data/dbpedia/'\n",
"URL_BASE = 'http://downloads.dbpedia.org/3.5.1/en/'\n",
"filenames = [\"redirects_en.nt.bz2\", \"page_links_en.nt.bz2\"]\n",
"\n",
"for filename in filenames:\n",
" if not os.path.exists(PATH+filename):\n",
" print(\"Downloading '%s', please wait...\" % filename)\n",
" open(PATH+filename, 'wb').write(urlopen(URL_BASE+filename).read())"
]
},
{
"cell_type": "code",
"execution_count": 368,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"redirects_filename = PATH+filenames[0]\n",
"page_links_filename = PATH+filenames[1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Graph Adjacency Matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will construct a graph **adjacency matrix**, of which pages point to which.\n",
"\n",
"![\"\"](\"images/graph.png\")
\n",
"(source: [PageRank and HyperLink Induced Topic Search](https://www.slideshare.net/priyabrata232/page-rank-and-hyperlink))\n",
"\n",
"![\"\"](\"images/adjaceny_matrix.png\")
\n",
"(source: [PageRank and HyperLink Induced Topic Search](https://www.slideshare.net/priyabrata232/page-rank-and-hyperlink))\n",
"\n",
"The power $A^2$ will give you how many ways there are to get from one page to another in 2 steps. You can see a more detailed example, as applied to airline travel, [worked out in these notes](http://www.utdallas.edu/~jwz120030/Teaching/PastCoursesUMBC/M221HS06/ProjectFiles/Adjacency.pdf)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We want to keep track of which pages point to which pages. We will store this in a square matrix, with a $1$ in position $(r, c)$ indicating that the topic in row $r$ points to the topic in column $c$\n",
"\n",
"You can read [more about graphs here](http://www.geeksforgeeks.org/graph-and-its-representations/)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Data Format"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"One line of the file looks like:\n",
"- `![\"\"](\"images/cincinnati_reds.png\")
"
]
},
{
"cell_type": "code",
"execution_count": 431,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"test_inds = [i for i,x in enumerate(source) if x == 9991050]"
]
},
{
"cell_type": "code",
"execution_count": 466,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"47"
]
},
"execution_count": 466,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(test_inds)"
]
},
{
"cell_type": "code",
"execution_count": 433,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[119076756, 119076757, 119076758, 119076759, 119076760]"
]
},
"execution_count": 433,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_inds[:5]"
]
},
{
"cell_type": "code",
"execution_count": 434,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"test_dests = [destination[i] for i in test_inds]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now, we want to check which page is the source (has index 9991174):"
]
},
{
"cell_type": "code",
"execution_count": 435,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"b'Baseball'\n",
"b'Ohio'\n",
"b'Cincinnati'\n",
"b'Flash_Thompson'\n",
"b'1940'\n",
"b'1938'\n",
"b'Lonny_Frey'\n",
"b'Cincinnati_Reds'\n",
"b'Ernie_Lombardi'\n",
"b'Baseball_card'\n",
"b'James_Wilson'\n",
"b'Trading_card'\n",
"b'Detroit_Tigers'\n",
"b'Baseball_culture'\n",
"b'Frank_McCormick'\n",
"b'Bucky_Walters'\n",
"b'1940_World_Series'\n",
"b'Billy_Werber'\n",
"b'Ival_Goodman'\n",
"b'Harry_Craft'\n",
"b'Paul_Derringer'\n",
"b'Johnny_Vander_Meer'\n",
"b'Cigarette_card'\n",
"b'Eddie_Joost'\n",
"b'Myron_McCormick'\n",
"b'Beckett_Media'\n",
"b'Icarus_affair'\n",
"b'Ephemera'\n",
"b'Sports_card'\n",
"b'James_Turner'\n",
"b'Jimmy_Ripple'\n",
"b'Lewis_Riggs'\n",
"b'The_American_Card_Catalog'\n",
"b'Rookie_card'\n",
"b'Willard_Hershberger'\n",
"b'Elmer_Riddle'\n",
"b'Joseph_Beggs'\n",
"b'Witt_Guise'\n",
"b'Milburn_Shoffner'\n"
]
}
],
"source": [
"for page_name, index in index_map.items():\n",
" if index in test_dests:\n",
" print(page_name)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can see that the items in the list appear in the wikipedia page:\n",
"\n",
"![\"\"](\"images/cincinnati_reds2.png\")
\n",
"(Source: [Wikipedia](https://en.wikipedia.org/wiki/W711-2))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Create Matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Below we create a sparse matrix using Scipy's COO format, and that convert it to CSR.\n",
"\n",
"**Questions**: What are COO and CSR? Why would we create it with COO and then convert it right away?"
]
},
{
"cell_type": "code",
"execution_count": 341,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"X = sparse.coo_matrix((data, (destination,source)), shape=(n,n), dtype=np.float32)\n",
"X = X.tocsr()"
]
},
{
"cell_type": "code",
"execution_count": 347,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"del(data,destination, source)"
]
},
{
"cell_type": "code",
"execution_count": 342,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<119077682x119077682 sparse matrix of type '![\"2](\"images/nonhermitian_eigen.JPG\")
\n",
"(source: Trefethen, Lecture 25)\n",
"\n",
"In the case of a Hermitian matrix, this approach is even faster, since the intermediate step is also Hermitian (and a Hermitian Hessenberg is *tridiagonal*).\n",
"\n",
"![\"2](\"images/hermitian_eigen.JPG\")
\n",
"(source: Trefethen, Lecture 25)\n",
"\n",
"Phase 1 reaches an exact solution in a finite number of steps, whereas Phase 2 theoretically never reaches the exact solution.\n",
"\n",
"We've already done step 2: the QR algorithm. Remember that it would be possible to just use the QR algorithm, but ridiculously slow."
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Arnoldi Iteration"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"We can use the Arnoldi iteration for phase 1 (and the QR algorithm for phase 2)."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Initializations"
]
},
{
"cell_type": "code",
"execution_count": 459,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"n = 5\n",
"A0 = np.random.rand(n,n) #.astype(np.float64)\n",
"A = A0 @ A0.T\n",
"\n",
"np.set_printoptions(precision=5, suppress=True)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Linear Algebra Review: Projections"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"When vector $\\mathbf{b}$ is projected onto a line $\\mathbf{a}$, its projection $\\mathbf{p}$ is the part of $\\mathbf{b}$ along that line $\\mathbf{a}$.\n",
"\n",
"Let's look at interactive graphic (3.4) for [section 3.2.2: Projections](http://immersivemath.com/ila/ch03_dotproduct/ch03.html) of the [Immersive Linear Algebra online book](http://immersivemath.com/ila/index.html).\n",
"\n",
"![\"projection\"](\"images/projection_line.png\")
\n",
"(source: [Immersive Math](http://immersivemath.com/ila/ch03_dotproduct/ch03.html))\n",
"\n",
"And here is what it looks like to project a vector onto a plane:\n",
"\n",
"![\"projection\"](\"images/projection.png\")
\n",
"(source: [The Linear Algebra View of Least-Squares Regression](https://medium.com/@andrew.chamberlain/the-linear-algebra-view-of-least-squares-regression-f67044b7f39b))\n",
"\n",
"When vector $\\mathbf{b}$ is projected onto a line $\\mathbf{a}$, its projection $\\mathbf{p}$ is the part of $\\mathbf{b}$ along that line $\\mathbf{a}$. So $\\mathbf{p}$ is some multiple of $\\mathbf{a}$. Let $\\mathbf{p} = \\hat{x}\\mathbf{a}$ where $\\hat{x}$ is a scalar."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Orthogonality"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**The key to projection is orthogonality:** The line *from* $\\mathbf{b}$ to $\\mathbf{p}$ (which can be written $\\mathbf{b} - \\hat{x}\\mathbf{a}$) is perpendicular to $\\mathbf{a}$.\n",
"\n",
"This means that $$ \\mathbf{a} \\cdot (\\mathbf{b} - \\hat{x}\\mathbf{a}) = 0 $$\n",
"\n",
"and so $$\\hat{x} = \\frac{\\mathbf{a} \\cdot \\mathbf{b}}{\\mathbf{a} \\cdot \\mathbf{a}} $$"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### The Algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Motivation**:\n",
"\n",
"We want orthonormal columns in $Q$ and a Hessenberg $H$ such that $A Q = Q H$.\n",
"\n",
"Thinking about it iteratively, $$ A Q_n = Q_{n+1} H_n $$ where $Q_{n+1}$ is $n\\times n+1$ and $H_n$ is $n+1 \\times n$. This creates a solvable recurrence relation.\n",
"\n",
"![\"arnoldi\"](\"images/arnoldi.jpg\")
\n",
"(source: Trefethen, Lecture 33)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**Pseudo-code for Arnoldi Algorithm**\n",
"\n",
" Start with an arbitrary vector (normalized to have norm 1) for first col of Q\n",
" for n=1,2,3...\n",
" v = A @ nth col of Q\n",
" for j=1,...n\n",
" project v onto q_j, and subtract the projection off of v\n",
" want to capture part of v that isn't already spanned by prev columns of Q\n",
" store coefficients in H\n",
" normalize v, and then make it the (n+1)th column of Q"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Notice that we are multiplying A by the previous vector in Q and removing the components that are not orthogonal to the existing columns of Q.\n",
"\n",
"**Question:** Repeated multiplications of A? Does this remind you of anything?"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true,
"hidden": true
},
"source": [
"#### Answer:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"#Exercise Answer\n",
"The *Power Method* involved iterative multiplications by A as well! "
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### About how the Arnoldi Iteration works"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"- With the Arnoldi Iteration, we are finding an orthonormal basis for the *Krylov subspace*. \n",
"The Krylov matrix $$ K = \\left[b \\; Ab \\; A^2b \\; \\dots \\; A^{n-1}b \\right]$$\n",
"has a QR factorization\n",
"$$K = QR$$\n",
"and that is the same $Q$ that is being found in the Arnoldi Iteration. Note that the Arnoldi Iteration does not explicity calculate $K$ or $R$.\n",
"\n",
"- Intuition: K contains good information about the largest eigenvalues of A, and the QR factorization reveals this information by peeling off one approximate eigenvector at a time.\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"The Arnoldi Iteration is two things:\n",
"1. the basis of many of the iterative algorithms of numerical linear algebra\n",
"2. a technique for finding eigenvalues of nonhermitian matrices\n",
"(Trefethen, page 257)\n",
"\n",
"**How Arnoldi Locates Eigenvalues**\n",
"\n",
"1. Carry out Arnoldi iteration\n",
"2. Periodically calculate the eigenvalues (called *Arnoldi estimates* or *Ritz values*) of the Hessenberg H, using the QR algorithm\n",
"3. Check at whether these values are converging. If they are, they're probably eigenvalues of A."
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Implementation"
]
},
{
"cell_type": "code",
"execution_count": 246,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"# Decompose square matrix A @ Q ~= Q @ H\n",
"def arnoldi(A):\n",
" m, n = A.shape\n",
" assert(n <= m)\n",
" \n",
" # Hessenberg matrix\n",
" H = np.zeros([n+1,n]) #, dtype=np.float64)\n",
" # Orthonormal columns\n",
" Q = np.zeros([m,n+1]) #, dtype=np.float64)\n",
" # 1st col of Q is a random column with unit norm\n",
" b = np.random.rand(m)\n",
" Q[:,0] = b / np.linalg.norm(b)\n",
" for j in range(n):\n",
" v = A @ Q[:,j]\n",
" for i in range(j+1):\n",
" #This comes from the formula for projection of v onto q.\n",
" #Since columns q are orthonormal, q dot q = 1\n",
" H[i,j] = np.dot(Q[:,i], v)\n",
" v = v - (H[i,j] * Q[:,i])\n",
" H[j+1,j] = np.linalg.norm(v)\n",
" Q[:,j+1] = v / H[j+1,j]\n",
" \n",
" # printing this to see convergence, would be slow to use in practice\n",
" print(np.linalg.norm(A @ Q[:,:-1] - Q @ H))\n",
" return Q[:,:-1], H[:-1,:]"
]
},
{
"cell_type": "code",
"execution_count": 86,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"8.59112969133\n",
"4.45398729097\n",
"0.935693639985\n",
"3.36613943339\n",
"0.817740180293\n"
]
}
],
"source": [
"Q, H = arnoldi(A)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Check that H is tri-diagonal:"
]
},
{
"cell_type": "code",
"execution_count": 70,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 5.62746, 4.05085, -0. , 0. , -0. ],\n",
" [ 4.05085, 3.07109, 0.33036, 0. , -0. ],\n",
" [ 0. , 0.33036, 0.98297, 0.11172, -0. ],\n",
" [ 0. , 0. , 0.11172, 0.29777, 0.07923],\n",
" [ 0. , 0. , 0. , 0.07923, 0.06034]])"
]
},
"execution_count": 70,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"H"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Exercise"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Write code to confirm that:\n",
"1. AQ = QH\n",
"2. Q is orthonormal"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### Answer:"
]
},
{
"cell_type": "code",
"execution_count": 71,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 71,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise:\n",
"np.allclose(A @ Q, Q @ H)"
]
},
{
"cell_type": "code",
"execution_count": 456,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 456,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#Exercise:\n",
"np.allclose(np.eye(len(Q)), Q.T @ Q)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"#### General Case:"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"**General Matrix**: Now we can do this on our general matrix A (not symmetric). In this case, we are getting a Hessenberg instead of a Tri-diagonal"
]
},
{
"cell_type": "code",
"execution_count": 74,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.44287067354\n",
"1.06234006889\n",
"0.689291414367\n",
"0.918098818651\n",
"4.7124490411e-16\n"
]
}
],
"source": [
"Q0, H0 = arnoldi(A0)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Check that H is Hessenberg:"
]
},
{
"cell_type": "code",
"execution_count": 76,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1.67416, 0.83233, -0.39284, 0.10833, 0.63853],\n",
" [ 1.64571, 1.16678, -0.54779, 0.50529, 0.28515],\n",
" [ 0. , 0.16654, -0.22314, 0.08577, -0.02334],\n",
" [ 0. , 0. , 0.79017, 0.11732, 0.58978],\n",
" [ 0. , 0. , 0. , 0.43238, -0.07413]])"
]
},
"execution_count": 76,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"H0"
]
},
{
"cell_type": "code",
"execution_count": 77,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 77,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(A0 @ Q0, Q0 @ H0)"
]
},
{
"cell_type": "code",
"execution_count": 78,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"(True, True)"
]
},
"execution_count": 78,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(np.eye(len(Q0)), Q0.T @ Q0), np.allclose(np.eye(len(Q0)), Q0 @ Q0.T)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Putting it all together"
]
},
{
"cell_type": "code",
"execution_count": 215,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"def eigen(A, max_iter=20):\n",
" Q, H = arnoldi(A)\n",
" Ak, QQ = practical_qr(H, max_iter)\n",
" U = Q @ QQ\n",
" D = np.diag(Ak)\n",
" return U, D"
]
},
{
"cell_type": "code",
"execution_count": 213,
"metadata": {
"collapsed": true,
"hidden": true
},
"outputs": [],
"source": [
"n = 10\n",
"A0 = np.random.rand(n,n)\n",
"A = A0 @ A0.T"
]
},
{
"cell_type": "code",
"execution_count": 242,
"metadata": {
"hidden": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"14.897422908\n",
"1.57451192745\n",
"1.4820012435\n",
"0.668164424736\n",
"0.438450319682\n",
"0.674050723258\n",
"1.19470880942\n",
"0.217103444634\n",
"0.105443975462\n",
"3.8162597576e-15\n",
"[ 27.34799 1.22613 1.29671 0.70253 0.49651 0.56779 0.60974\n",
" 0.70123 0.19209 0.04905]\n",
"[ 27.34981 1.85544 1.04793 0.49607 0.44505 0.7106 1.00724\n",
" 0.07293 0.16058 0.04411]\n",
"[ 27.34981 2.01074 0.96045 0.54926 0.61117 0.8972 0.53424\n",
" 0.19564 0.03712 0.04414]\n",
"[ 27.34981 2.04342 0.94444 0.61517 0.89717 0.80888 0.25402\n",
" 0.19737 0.03535 0.04414]\n",
"[ 27.34981 2.04998 0.94362 0.72142 1.04674 0.58643 0.21495\n",
" 0.19735 0.03534 0.04414]\n",
"[ 27.34981 2.05129 0.94496 0.90506 0.95536 0.49632 0.21015\n",
" 0.19732 0.03534 0.04414]\n",
"[ 27.34981 2.05156 0.94657 1.09452 0.79382 0.46723 0.20948\n",
" 0.1973 0.03534 0.04414]\n",
"[ 27.34981 2.05161 0.94863 1.1919 0.70539 0.45628 0.20939\n",
" 0.19728 0.03534 0.04414]\n",
"[ 27.34981 2.05162 0.95178 1.22253 0.67616 0.45174 0.20939\n",
" 0.19727 0.03534 0.04414]\n",
"[ 27.34981 2.05162 0.95697 1.22715 0.66828 0.44981 0.2094\n",
" 0.19725 0.03534 0.04414]\n",
"[ 27.34981 2.05162 0.96563 1.22124 0.66635 0.44899 0.20941\n",
" 0.19724 0.03534 0.04414]\n",
"[ 27.34981 2.05162 0.97969 1.20796 0.66592 0.44864 0.20942\n",
" 0.19723 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.00135 1.18652 0.66585 0.44849 0.20943\n",
" 0.19722 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.03207 1.15586 0.66584 0.44843 0.20943\n",
" 0.19722 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.07082 1.11714 0.66584 0.4484 0.20944\n",
" 0.19721 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.11307 1.07489 0.66585 0.44839 0.20944\n",
" 0.1972 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.15241 1.03556 0.66585 0.44839 0.20945\n",
" 0.1972 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.18401 1.00396 0.66585 0.44839 0.20945\n",
" 0.1972 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.20652 0.98145 0.66585 0.44839 0.20946\n",
" 0.19719 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.22121 0.96676 0.66585 0.44839 0.20946\n",
" 0.19719 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.23026 0.95771 0.66585 0.44839 0.20946\n",
" 0.19719 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.23563 0.95234 0.66585 0.44839 0.20946\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.23876 0.94921 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24056 0.94741 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24158 0.94639 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24216 0.94581 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24249 0.94548 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24268 0.94529 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24278 0.94519 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24284 0.94513 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24288 0.94509 0.66585 0.44839 0.20947\n",
" 0.19718 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.2429 0.94507 0.66585 0.44839 0.20947\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24291 0.94506 0.66585 0.44839 0.20947\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24291 0.94506 0.66585 0.44839 0.20947\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20947\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20947\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20948\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20948\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20948\n",
" 0.19717 0.03534 0.04414]\n",
"[ 27.34981 2.05162 1.24292 0.94505 0.66585 0.44839 0.20948\n",
" 0.19717 0.03534 0.04414]\n"
]
}
],
"source": [
"U, D = eigen(A, 40)"
]
},
{
"cell_type": "code",
"execution_count": 240,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 5.10503, 0.58805, 0.49071, -0.65174, -0.60231, -0.37664,\n",
" -0.13165, 0.0778 , -0.10469, -0.29771])"
]
},
"execution_count": 240,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"D"
]
},
{
"cell_type": "code",
"execution_count": 241,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([ 27.34981, 2.05162, 1.24292, 0.94505, 0.66585, 0.44839,\n",
" 0.20948, 0.19717, 0.04414, 0.03534])"
]
},
"execution_count": 241,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.eigvals(A)"
]
},
{
"cell_type": "code",
"execution_count": 236,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"0.0008321887107978883"
]
},
"execution_count": 236,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.linalg.norm(U @ np.diag(D) @ U.T - A)"
]
},
{
"cell_type": "code",
"execution_count": 237,
"metadata": {
"hidden": true
},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 237,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(U @ np.diag(D) @ U.T, A, atol=1e-3)"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Further Reading"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"Let's find some eigenvalues!\n",
"\n",
"\n",
"from [Nonsymmetric Eigenvalue Problems](https://sites.math.washington.edu/~morrow/498_13/eigenvalues.pdf) chapter:\n",
"\n",
"Note that \"direct\" methods must still iterate, since finding eigenvalues is mathematically equivalent to finding zeros of polynomials, for which no noniterative methods can exist. We call a method direct if experience shows that it (nearly) never fails to converge in a\n",
"fixed number of iterations.\n",
"\n",
"Iterative methods typically provide approximations only to a subset of the eigenvalues and eigenvectors and are usually run only long enough to get a few adequately accurate eigenvalues rather than a large number\n",
"\n",
"our ultimate algorithm: the shifted Hessenberg QR algorithm\n",
"\n",
"More reading:\n",
"- [The Symmetric Eigenproblem and SVD](https://sites.math.washington.edu/~morrow/498_13/eigenvalues2.pdf)\n",
"- [Iterative Methods for Eigenvalue Problems](https://sites.math.washington.edu/~morrow/498_13/eigenvalues3.pdf) Rayleigh-Ritz Method, Lanczos algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"### Coming Up\n",
"\n",
"We will be coding our own QR decomposition (two different ways!) in the future, but first we are going to see another way that the QR decomposition can be used: to calculate linear regression."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## End"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Miscellaneous Notes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Symmetric matrices come up naturally:\n",
"- distance matrices\n",
"- relationship matrices (Facebook or LinkedIn)\n",
"- ODEs\n",
"\n",
"We will look at positive definite matrices, since that guarantees that all the eigenvalues are real."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note: in the confusing language of NLA, the QR algorithm is *direct*, because you are making progress on all columns at once. In other math/CS language, the QR algorithm is *iterative*, because it iteratively converges and never reaches an exact solution."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"structured orthogonalization. In the language of NLA, Arnoldi iteration is considered an *iterative* algorithm, because you could stop part way and have a few columns completed.\n",
"\n",
"a Gram-Schmidt style iteration for transforming a matrix to Hessenberg form"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
},
"widgets": {
"state": {
"2987aec2ec494d2b9b31bdf93ba87cb6": {
"views": [
{
"cell_index": 18
}
]
},
"84bffde89e894158860b175e986f4d61": {
"views": [
{
"cell_index": 14
}
]
}
},
"version": "1.2.0"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/8. Implementing QR Factorization.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can read an overview of this Numerical Linear Algebra course in [this blog post](http://www.fast.ai/2017/07/17/num-lin-alg/). The course was originally taught in the [University of San Francisco MS in Analytics](https://www.usfca.edu/arts-sciences/graduate-programs/analytics) graduate program. Course lecture videos are [available on YouTube](https://www.youtube.com/playlist?list=PLtmWHNX-gukIc92m1K0P6bIOnZb-mg0hY) (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).\n",
"\n",
"You can ask questions about the course on [our fast.ai forums](http://forums.fast.ai/c/lin-alg)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 8. Implementing QR Factorization"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We used QR factorization in computing eigenvalues and to compute least squares regression. It is an important building block in numerical linear algebra.\n",
"\n",
"\"One algorithm in numerical linear algebra is more important than all the others: QR factorization.\" --Trefethen, page 48\n",
"\n",
"Recall that for any matrix $A$, $A = QR$ where $Q$ is orthogonal and $R$ is upper-triangular."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"**Reminder**: The *QR algorithm*, which we looked at in the last lesson, uses the *QR decomposition*, but don't confuse the two."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### In Numpy ###"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"np.set_printoptions(suppress=True, precision=4)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"n = 5\n",
"A = np.random.rand(n,n)\n",
"npQ, npR = np.linalg.qr(A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that Q is orthogonal:"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(True, True)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(np.eye(n), npQ @ npQ.T), np.allclose(np.eye(n), npQ.T @ npQ)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that R is triangular"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.8524, -0.7872, -1.1163, -1.2248, -0.7587],\n",
" [ 0. , -0.9363, -0.2958, -0.7666, -0.632 ],\n",
" [ 0. , 0. , 0.4645, -0.1744, -0.3542],\n",
" [ 0. , 0. , 0. , 0.4328, -0.2567],\n",
" [ 0. , 0. , 0. , 0. , 0.1111]])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"npR"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Gram-Schmidt ##"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Classical Gram-Schmidt (unstable) ###"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For each $j$, calculate a single projection $$v_j = P_ja_j$$ where $P_j$ projects onto the space orthogonal to the span of $q_1,\\ldots,q_{j-1}$."
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def cgs(A):\n",
" m, n = A.shape\n",
" Q = np.zeros([m,n], dtype=np.float64)\n",
" R = np.zeros([n,n], dtype=np.float64)\n",
" for j in range(n):\n",
" v = A[:,j]\n",
" for i in range(j):\n",
" R[i,j] = np.dot(Q[:,i], A[:,j])\n",
" v = v - (R[i,j] * Q[:,i])\n",
" R[j,j] = np.linalg.norm(v)\n",
" Q[:, j] = v / R[j,j]\n",
" return Q, R"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"Q, R = cgs(A)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(A, Q @ R)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check if Q is unitary:"
]
},
{
"cell_type": "code",
"execution_count": 109,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 109,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(np.eye(len(Q)), Q.dot(Q.T))"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(npQ, -Q)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gram-Schmidt should remind you a bit of the Arnoldi Iteration (used to transform a matrix to Hessenberg form) since it is also a structured orthogonalization."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Modified Gram-Schmidt ###"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Classical (unstable) Gram-Schmidt: for each $j$, calculate a single projection $$v_j = P_ja_j$$ where $P_j$ projects onto the space orthogonal to the span of $q_1,\\ldots,q_{j-1}$.\n",
"\n",
"Modified Gram-Schmidt: for each $j$, calculate $j-1$ projections $$P_j = P_{\\perp q_{j-1}\\cdots\\perp q_{2}\\perp q_{1}}$$"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"n = 3\n",
"A = np.random.rand(n,n).astype(np.float64)"
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def mgs(A):\n",
" V = A.copy()\n",
" m, n = A.shape\n",
" Q = np.zeros([m,n], dtype=np.float64)\n",
" R = np.zeros([n,n], dtype=np.float64)\n",
" for i in range(n):\n",
" R[i,i] = np.linalg.norm(V[:,i])\n",
" Q[:,i] = V[:,i] / R[i,i]\n",
" for j in range(i, n):\n",
" R[i,j] = np.dot(Q[:,i],V[:,j])\n",
" V[:,j] = V[:,j] - R[i,j]*Q[:,i]\n",
" return Q, R"
]
},
{
"cell_type": "code",
"execution_count": 34,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"Q, R = mgs(A)"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(np.eye(len(Q)), Q.dot(Q.T.conj()))"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(A, np.matmul(Q,R))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Householder"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Intro"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\\begin{array}{ l | l | c }\n",
"\\hline\n",
"Gram-Schmidt & Triangular\\, Orthogonalization & A R_1 R_2 \\cdots R_n = Q \\\\\n",
"Householder & Orthogonal\\, Triangularization & Q_n \\cdots Q_2 Q_1 A = R \\\\\n",
"\\hline\n",
"\\end{array}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Householder reflections lead to a more nearly orthogonal matrix Q with rounding errors\n",
"\n",
"Gram-Schmidt can be stopped part-way, leaving a reduced QR of 1st n columns of A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Initialization"
]
},
{
"cell_type": "code",
"execution_count": 113,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import numpy as np\n",
"n = 4\n",
"A = np.random.rand(n,n).astype(np.float64)\n",
"\n",
"Q = np.zeros([n,n], dtype=np.float64)\n",
"R = np.zeros([n,n], dtype=np.float64)"
]
},
{
"cell_type": "code",
"execution_count": 114,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.5435, 0.6379, 0.4011, 0.5773],\n",
" [ 0.0054, 0.8049, 0.6804, 0.0821],\n",
" [ 0.2832, 0.2416, 0.8656, 0.8099],\n",
" [ 0.1139, 0.9621, 0.7623, 0.5648]])"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A"
]
},
{
"cell_type": "code",
"execution_count": 115,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from scipy.linalg import block_diag"
]
},
{
"cell_type": "code",
"execution_count": 116,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"np.set_printoptions(5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Algorithm"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I added more computations and more info than needed, since it's illustrative of how the algorithm works. This version returns the *Householder Reflectors* as well."
]
},
{
"cell_type": "code",
"execution_count": 117,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def householder_lots(A):\n",
" m, n = A.shape\n",
" R = np.copy(A)\n",
" V = []\n",
" Fs = []\n",
" for k in range(n):\n",
" v = np.copy(R[k:,k])\n",
" v = np.reshape(v, (n-k, 1))\n",
" v[0] += np.sign(v[0]) * np.linalg.norm(v)\n",
" v /= np.linalg.norm(v)\n",
" R[k:,k:] = R[k:,k:] - 2*np.matmul(v, np.matmul(v.T, R[k:,k:]))\n",
" V.append(v)\n",
" F = np.eye(n-k) - 2 * np.matmul(v, v.T)/np.matmul(v.T, v)\n",
" Fs.append(F)\n",
" return R, V, Fs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that R is upper triangular:"
]
},
{
"cell_type": "code",
"execution_count": 121,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.62337, -0.84873, -0.88817, -0.97516],\n",
" [ 0. , -1.14818, -0.86417, -0.30109],\n",
" [ 0. , 0. , -0.64691, -0.45234],\n",
" [-0. , 0. , 0. , -0.26191]])"
]
},
"execution_count": 121,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"R"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a check, we will calculate $Q^T$ and $R$ using the block matrices $F$. The matrices $F$ are *the householder reflectors*.\n",
"\n",
"Note that this is not a computationally efficient way of working with $Q$. In most cases, you do not actually need $Q$. For instance, if you are using QR to solve least squares, you just need $Q^*b$.\n",
"\n",
"- See page 74 of Trefethen for techniques to implicitly calculate the product $Q^*b$ or $Qx$.\n",
"- See [these lecture notes](http://www.cs.cornell.edu/~bindel/class/cs6210-f09/lec18.pdf) for a different implementation of Householder that calculates Q simultaneously as part of R. "
]
},
{
"cell_type": "code",
"execution_count": 175,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"QT = np.matmul(block_diag(np.eye(3), F[3]), \n",
" np.matmul(block_diag(np.eye(2), F[2]), \n",
" np.matmul(block_diag(np.eye(1), F[1]), F[0])))"
]
},
{
"cell_type": "code",
"execution_count": 178,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.69502, 0.10379, -0.71146],\n",
" [ 0.10379, 0.99364, 0.04356],\n",
" [-0.71146, 0.04356, 0.70138]])"
]
},
"execution_count": 178,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"F[1]"
]
},
{
"cell_type": "code",
"execution_count": 177,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1. , 0. , 0. , 0. ],\n",
" [ 0. , -0.69502, 0.10379, -0.71146],\n",
" [ 0. , 0.10379, 0.99364, 0.04356],\n",
" [ 0. , -0.71146, 0.04356, 0.70138]])"
]
},
"execution_count": 177,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"block_diag(np.eye(1), F[1])"
]
},
{
"cell_type": "code",
"execution_count": 176,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.87185, -0.00861, -0.45431, -0.18279],\n",
" [ 0.08888, -0.69462, 0.12536, -0.70278],\n",
" [-0.46028, 0.10167, 0.88193, -0.00138],\n",
" [-0.14187, -0.71211, 0.00913, 0.68753]])"
]
},
"execution_count": 176,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.matmul(block_diag(np.eye(1), F[1]), F[0])"
]
},
{
"cell_type": "code",
"execution_count": 123,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.87185, -0.00861, -0.45431, -0.18279],\n",
" [ 0.08888, -0.69462, 0.12536, -0.70278],\n",
" [ 0.45817, -0.112 , -0.88171, 0.01136],\n",
" [ 0.14854, 0.71056, -0.02193, -0.68743]])"
]
},
"execution_count": 123,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"QT"
]
},
{
"cell_type": "code",
"execution_count": 124,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"R2 = np.matmul(block_diag(np.eye(3), F[3]), \n",
" np.matmul(block_diag(np.eye(2), F[2]),\n",
" np.matmul(block_diag(np.eye(1), F[1]), \n",
" np.matmul(F[0], A))))"
]
},
{
"cell_type": "code",
"execution_count": 125,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 125,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(A, np.matmul(np.transpose(QT), R2))"
]
},
{
"cell_type": "code",
"execution_count": 126,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 126,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(R, R2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here's a concise version of Householder (although I do create a new R, instead of overwriting A and computing it in place)."
]
},
{
"cell_type": "code",
"execution_count": 179,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def householder(A):\n",
" m, n = A.shape\n",
" R = np.copy(A)\n",
" Q = np.eye(m)\n",
" V = []\n",
" for k in range(n):\n",
" v = np.copy(R[k:,k])\n",
" v = np.reshape(v, (n-k, 1))\n",
" v[0] += np.sign(v[0]) * np.linalg.norm(v)\n",
" v /= np.linalg.norm(v)\n",
" R[k:,k:] = R[k:,k:] - 2 * v @ v.T @ R[k:,k:]\n",
" V.append(v)\n",
" return R, V"
]
},
{
"cell_type": "code",
"execution_count": 157,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"RH, VH = householder(A)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Check that R is diagonal:"
]
},
{
"cell_type": "code",
"execution_count": 129,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[-0.62337, -0.84873, -0.88817, -0.97516],\n",
" [-0. , -1.14818, -0.86417, -0.30109],\n",
" [-0. , -0. , -0.64691, -0.45234],\n",
" [-0. , 0. , 0. , -0.26191]])"
]
},
"execution_count": 129,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"RH"
]
},
{
"cell_type": "code",
"execution_count": 130,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"[array([[ 0.96743],\n",
" [ 0.00445],\n",
" [ 0.2348 ],\n",
" [ 0.09447]]), array([[ 0.9206 ],\n",
" [-0.05637],\n",
" [ 0.38641]]), array([[ 0.99997],\n",
" [-0.00726]]), array([[ 1.]])]"
]
},
"execution_count": 130,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"VH"
]
},
{
"cell_type": "code",
"execution_count": 131,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 131,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.allclose(R, RH)"
]
},
{
"cell_type": "code",
"execution_count": 132,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def implicit_Qx(V,x):\n",
" n = len(x)\n",
" for k in range(n-1,-1,-1):\n",
" x[k:n] -= 2*np.matmul(v[-k], np.matmul(v[-k], x[k:n]))"
]
},
{
"cell_type": "code",
"execution_count": 133,
"metadata": {
"scrolled": true
},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 0.54348, 0.63791, 0.40114, 0.57728],\n",
" [ 0.00537, 0.80485, 0.68037, 0.0821 ],\n",
" [ 0.2832 , 0.24164, 0.86556, 0.80986],\n",
" [ 0.11395, 0.96205, 0.76232, 0.56475]])"
]
},
"execution_count": 133,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"A"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Both classical and modified Gram-Schmidt require $2mn^2$ flops."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Gotchas"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Some things to be careful about:\n",
"- when you've copied values vs. when you have two variables pointing to the same memory location\n",
"- the difference between a vector of length n and a 1 x n matrix (np.matmul deals with them differently)"
]
},
{
"cell_type": "markdown",
"metadata": {
"heading_collapsed": true
},
"source": [
"## Analogy"
]
},
{
"cell_type": "markdown",
"metadata": {
"hidden": true
},
"source": [
"| | $A = QR$ | $A = QHQ^*$ |\n",
"|------------------------------|--------------|-------------|\n",
"| orthogonal structuring | Householder | Householder |\n",
"| structured orthogonalization | Gram-Schmidt | Arnoldi |\n",
"\n",
"**Gram-Schmidt and Arnoldi**: succession of triangular operations, can be stopped part way and first $n$ columns are correct\n",
"\n",
"**Householder**: succession of orthogoal operations. Leads to more nearly orthogonal A in presence of rounding errors.\n",
"\n",
"Note that to compute a Hessenberg reduction $A = QHQ^*$, Householder reflectors are applied to two sides of A, rather than just one."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Examples"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following examples are taken from Lecture 9 of Trefethen and Bau, although translated from MATLAB into Python"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ex: Classical vs Modified Gram-Schmidt ###"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This example is experiment 2 from section 9 of Trefethen. We want to construct a square matrix A with random singular vectors and widely varying singular values spaced by factors of 2 between $2^{-1}$ and $2^{-(n+1)}$"
]
},
{
"cell_type": "code",
"execution_count": 68,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"from matplotlib import rcParams\n",
"%matplotlib inline"
]
},
{
"cell_type": "code",
"execution_count": 183,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"n = 100\n",
"U, X = np.linalg.qr(np.random.randn(n,n)) # set U to a random orthogonal matrix\n",
"V, X = np.linalg.qr(np.random.randn(n,n)) # set V to a random orthogonal matrix\n",
"S = np.diag(np.power(2,np.arange(-1,-(n+1),-1), dtype=float)) # Set S to a diagonal matrix w/ exp \n",
" # values between 2^-1 and 2^-(n+1)"
]
},
{
"cell_type": "code",
"execution_count": 184,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"A = np.matmul(U,np.matmul(S,V))"
]
},
{
"cell_type": "code",
"execution_count": 188,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"QC, RC = cgs(A)\n",
"QM, RM = mgs(A)"
]
},
{
"cell_type": "code",
"execution_count": 189,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAnYAAAJKCAYAAACh7AgvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3XlYlOX6wPHvCwrImopbKZu5kLikmJppqFSWWWpKFqUo\n5VaZ/TJLzf2IuVRkmUsmmmEnyXI7LS6BnsrTUUvrqGkpQpkLouKCO8/vj5GRYWZgBmaF+3Ndc73O\nM+9yz+LMzbNqSimEEEIIIYT783B2AEIIIYQQwjYksRNCCCGEqCAksRNCCCGEqCAksRNCCCGEqCAk\nsRNCCCGEqCDcOrHTNO19TdP+1DTtrKZpRzRNS9Y0zcvZcQkhhBBCOINbJ3bAe0BTpVQg0PLGbZxz\nQxJCCCGEcI4qzg6gPJRSe4vc1YACoJGTwhFCCCGEcCqXq7HTNK2xpmlTNU37j6ZpOZqmndM0bZem\naeM1TfMzsf9rmqadB06gq7FLdnjQQgghhBAuQHO1lSc0TXsDeA5YC/wHuAp0AeKAX4D2SqmLJo6L\nBOKBBUqpv0q6RnBwsAoLC7Nx5EIIIYQQtrdz586TSqlaluzrik2xnwEzlFJ5RcoWaJr2OzAeSETX\nt86AUmqfpmm7geXoEkGzwsLC2LFjhw1DFkIIIYSwD03Tsizd1+WaYpVSO4oldYU+vbGNKuHwqkBj\n20clhBBCCOH6XC6xK0H9G9vjAJqmBWmalqBp2i2aTgvgdeAbp0UohBBCCOFEbpHYaZrmCUwArgEr\nbhQr4CngEHAOWA18CbzgjBiFEEIIIZzNFfvYmZIMdADGKaX2AyilzgKxlp5A07QhwBCAkJAQe8Qo\nhBBCCOFULl9jp2naNOB5YJFSakZZz6OUWqSUilZKRdeqZdHAEiGEEEIIt+LSiZ2maZPR9ZtLAYY5\nNxohhBBCCNfmsondjaRuErAMeEa52oR7QgghhBAuxiX72GmaNhFdUrccGKyUKnBySEIIYbW8vDxO\nnjzJlStXnB2KEMLFeHl5ERwcTFBQkE3P63KJnaZpzwFTgGxgE/CkpmlFdzmulNrojNiEEMJSly5d\n4vjx49SvX59q1apR7HtMCFGJKaW4ePEif/31F97e3vj4+Njs3C6X2AFtb2xD0DXDFrcFkMROCOHS\ncnJyqFWrFr6+vs4ORQjhYjRNw9fXl+DgYHJycmjQoIHNzu1yfeyUUglKKa2EW4yzYxRCiNJcunQJ\nf39/Z4chhHBhAQEBXLp0yabndLnETgghKoJr165RpYorNooIIVxFlSpVuHbtmk3PKYmdEELYifSr\nE0KUxB7fEZLYCSGEEEJUEJLYCSGEEEJUEJLYCSGEEKW4dOkSmqYxbJhrLoLUvn17mjZt6uwwbCo6\nOpqoqChnh+F2JLETQghRJpqmWXw7fPiws8M1aefOncTFxdGwYUN8fHwIDg6mVatWDB8+nF9//dXZ\n4bmsnj17omkav/32m9l9rl+/zm233UZwcLBM0u1AMmRLCCFEmSxfvtzg/r///W8WLVrEkCFD6NSp\nk8FjtWrVcmRoFlm1ahVxcXHUrVuXAQMGEBERwenTp9m/fz9r1qyhWbNmNG/eHAAfHx8uXrwoI51v\nSExMZP369aSkpDBz5kyT+3zzzTf8/fffvPjii3h5eTk4wspLPqFCCCHK5KmnnjK4f+3aNRYtWkSH\nDh2MHjNHKUV+fj5+fn72CLFEr732GoGBgezcuZO6desaPHb9+nVOnTplUGbL1QHcQX5+Pt7e3nh6\neho99vDDD1OnTh2WL19OUlKSyX1SUlIAXRIoHEeaYoUQQjjE119/jaZpfPLJJ7zzzjs0bdoUb29v\n3n33XQDq1q1L9+7dzR73z3/+06D84sWLTJ06lTvuuAMfHx9q1KhBr169LGpCLSgo4ODBgzRr1swo\nqQPw9PQ0qGU01ceuaNnWrVu555578PX1pVatWgwbNoz8/Hyj827atIl27drh4+NDvXr1GD16NLt2\n7ULTNN544w39fgsWLEDTNP7zn/8YncPS/nQ//PADAwYMoFGjRvj6+hIYGEjnzp1Zv3690b79+/fH\nx8eHY8eOMWDAAGrXro2/vz85OTkmz12lShUGDBjA0aNH+eqrr4wez83NZe3atURHR+trPQE++ugj\nevToQYMGDfD29qZ27dr069ePffv2lfp8AIKDg3n44YeNytevX4+maXz22WcG5fn5+UyePJnIyEj9\nZ6R3797s2bPHYL9r164xa9YsoqKi8Pf3JygoiMjISIYMGWJRXK5EauyEEMIdbdsGGRkQEwMdOjg7\nGqvMnDmTvLw8Bg8eTO3atYmIiLD6HJcvXyY2NpadO3cycOBARo4cyalTp/Q1ht9//z0tW7Y0e7yH\nhwfh4eH8/PPPbN++nbZt25rdtzT//e9/SUtL45lnnuGpp55i8+bNLFy4EC8vL+bOnavfb/PmzTz4\n4IPUrl2bcePGERAQwD//+U+2bNlS5muXJC0tjUOHDtG/f39CQkLIyclh6dKl9OzZk1WrVtGnTx+D\n/QsKCujWrRvh4eFMnDiRc+fOUa1aNbPnHzx4MLNnzyYlJcUo2UpNTeXKlStGtXXvvPMO4eHhDB8+\nnFq1anHgwAE++OADNm3axO7duwkJCbHZ87906RJdu3Zl165dJCQk8OKLL5Kbm8vChQtp374927Zt\n0w/OGD9+PLNmzeKxxx7j+eefB+DgwYOsXr3aZvE4jFKq0t3atGmj7O6HH5RKStJthRCVzt69e+13\n8h9+UKpaNaU8PXVbF/meSUlJUYBKSUkx+fhXX32lAFWrVi2Vm5tr9HidOnXUAw88YPa4Tz75RF+W\nlJSkPDw81Lfffmuwb25urqpbt67J8xS3fPlyBShN01TLli3V8OHD1ZIlS1RWVpbRvhcvXlSAGjp0\nqFGZp6en+umnnwz279q1q/L29laXLl3Sl7Vo0UL5+vqq7Oxsfdnly5dVmzZtFKBmzJihL58/f74C\n1LZt24xiadeunWrSpEmpZefPnzc69ty5cyo8PFzdeeedBuWPP/64AlRiYqLRMSW5++67VdWqVVVO\nTo5BeatWrVS1atXUmTNnSo3pp59+Uh4eHuqVV14xKG/Tpo1q1qyZQVnNmjVVjx49jM6xbt06Bai0\ntDR92dSpU5Wnp6faunWrwb45OTmqdu3aBudp1KiRatu2bSnP1j4s+a4AdigLcxxpirWHbdugWzeY\nMEG33bbN2REJISqSjAy4cgWuX9dtMzKcHZFVBg8eTI0aNcp1jo8//pjmzZvTvHlzTp48qb8V1jql\np6eXulTTU089RXp6Or169eLw4cPMnz+fwYMHExYWRt++fY362Jlz7733cueddxqUde3alcuXL/Pn\nn38CkJWVxS+//ELfvn0NFnz38vJi5MiRVj57yxTtt5ifn09ubi6XLl3i3nvvZdeuXVy+fNnomNGj\nR1t1jcTERK5evUpqaqq+bNeuXezatYvHHnuMoKAgkzEppTh79iwnT56kQYMGhIWF8eOPP1p17dJ8\n/PHHtGrVisjISIPPCECXLl3YvHkzBQUFAAQFBZGZmcl///tfm8bgDNIUaw+mvnTdrKlECOHCYmLA\ny0v3/eLlpbvvRho3blyu45VS7N+/n+vXr5c42vb06dOljsaNiYkhJiZGf8709HTee+89Vq1axdWr\nV1mzZk2p8ZhqSq5Zsyag62t2++23k5mZCUCTJk2M9jVVZgtHjx5l/PjxrFu3Tp/QFJWXl0ft2rX1\n9z08PGjYsKFV14iLi+PFF18kJSWFF198EYAlS5YAugS+uB9//JGJEyfy3XffGfVBtOUAmuvXr/P7\n77+jlCrxM5CXl0f16tWZNWsWffv2pV27djRo0IAuXbrQo0cP+vTp43Yjod0rWnfh5l+6QggX16ED\nbN7stn3sfH19TZabWzezeM2butmtxmDAQXHFa4tKomkaTZs2pWnTpgwYMIDIyEjWrl3LyZMnCQ4O\nLvFYUyNCi8ZaFiWtIWrJovHXr1+nW7duZGZm8uKLL9KmTRuCgoLw8PBg4cKFfPbZZ/raqkJVq1al\natWqVsXp7+9PXFwcS5Ys4aeffiIqKooVK1YQERFBTLHfvgMHDhATE0OdOnWYMmUKt99+O35+fmia\nxtChQ43iMcXaz0jbtm1JSkoqMX7Q1eBlZmby9ddfk56ezrfffstHH31Eq1at2Lp1KwEBAaXG5iok\nsbMHN//SFUK4gQ4dKtx3S40aNUw2fx46dMjgfmHN0smTJ+nWrZvNF1L38/OjefPm/Pnnnxw5cqTU\nxM4SYWFhAOzfv9/oMVNlhU3VxV8PpRSZmZml1kTu2LGDffv2kZSUxNixYw0ee++996wJvVSJiYks\nWbKElJQU7r33XnJzcxk1apTR+5KWlsalS5dIS0szGKxSUFBATk6ORQMnLP2MVKlShfDwcHJzc4mN\njbXoeQQGBhIXF0dcXBwAs2fPZsyYMXz88ccMHz7conO4AuljZy8dOsDYsYZfvNu2wYwZ0udOCCFM\naNy4Mb/++isnTpzQl128eJH58+cb7TtgwACysrKYN2+eyXMdP3681OuZmqYDdE2YP/74I97e3mUa\nsWtKWFgYUVFRfPbZZ/p+dwBXrlwxGDlbqLC5etOmTQblS5cutajvX2EtYvEaw59++ol//etfVsdf\nkrvvvpumTZuyYsUKFixYgIeHBwkJCRbH9M4773Du3DmLrtW4cWN2795t0LScn5/PggULjPYdMGAA\nhw4dMvkYGH5GTDVVF/abtLSvpauQGjtHKRxQUdg8u3lzhftrWwghyuP5559n9erVdO3alSFDhnDx\n4kWWLVtmskn1lVdeYfPmzbzwwgt88803xMTE4O/vT3Z2Nhs3bqRmzZpmE7dCPXv2JCwsjIcffpim\nTZvi6enJwYMH+eijj8jNzSUpKcmmTXBvvfUWDz74IO3bt2fYsGEEBATwySef6B8vWsPVsmVL7rnn\nHt555x2uXr1KVFQUO3fuZP369frav5K0aNGCxo0b849//IMzZ87QqFEj9u3bxwcffECLFi346aef\nbPa8QFdrV/iedO/enfr16xvt07NnTyZNmkRcXBwjRozA39+frVu3kp6ebvE0J88//zzr16+na9eu\nPPvss+Tn55OSkmJyMM5rr73Gt99+y/Dhw/nqq6/o3Lkzfn5+ZGdns2HDBurVq8e6desACAkJ4YEH\nHiA6Opp69erx119/sXDhQnx9fenbt2/5XhxHs3T4bEW6OWS6k+KSknRTE4Bum5Tk+BiEEA5j1+lO\nXJSl050UnbakuA8++EDdfvvtqmrVqioiIkK99dZb6ssvvzR53JUrV9Sbb76pWrdurXx9fZWfn59q\n1KiRevrpp9XmzZtLjXfFihVq4MCBKjIyUgUFBakqVaqoOnXqqIceekitWbPGYN+SpjspWlbI3HQl\n33zzjYqOjlbe3t6qTp066sUXX1RbtmxRgHrnnXcM9v3rr79U7969lb+/v/L391c9evRQBw4csHi6\nkz/++EP17t1b1axZU/n6+qp27dqpdevWqVdffVUB6ujRo/p9H3/8ceXt7V3qa2bO8ePHVdWqVRWg\nVq5caXa/DRs2qHbt2ik/Pz9VvXp19eijj6r9+/ebnNrEVJlSute2YcOGqmrVqqphw4YqOTlZrV27\n1mi6E6V008nMnj1b3XnnnapatWr6z8iAAQMMpsqZPHmyuvvuu1VwcLDy8vJSDRo0UE888YT65Zdf\nyvyaWMrW051oqowdO91ZdHS02rFjh2MvKjV2QlQq+/btIzIy0tlhCDeQmprKU089xRdffEGvXr2c\nHY5wMEu+KzRN26mUirbkfNIU6ygyoEIIISq1goICrl27hpeXl77s8uXLJCcn4+3tTefOnZ0Ynago\nJLFzgFmzIK/mJlLPPUP2lWxCfgwhfu9ignJjGdPJfZcFEkIIYbmzZ88SGRlJfHw8jRs3Jicnh08+\n+YQ9e/YwadKkck/aLARIYucQeTU3kTSyJfQLh/AssnaFk5TWknFjlkK3EdI8K4QQlUC1atW4//77\n+fzzzzl27BgATZs2ZdGiRTz77LNOjk5UFJLYOUDquWd0SV3aSoieDzuGQ784UtnFdFmhQgghKgVv\nb2+WLVvm7DBEBSeJnQNk52VDeJYuqds6ETpPhfAMstHAy0dWqBBCCCGETUhi5wAhQSFk7QrX1dR1\nnqrbhqcT0ioTNn8ifeyEEEIIYROS2DlAfMBiktJaQr84CM+A8HRIW0l8p92mlwXaJgMqhBBCCGE9\nSewcICg3lnFzN5F6LpPsPI2QVpnEd9pNUK6J9etkvjshhBBClJEkdg4wZgxALNM5XPrOGRm6pE4G\nVAghhBDCSh7ODkAUExOjq6nz9JQBFUIIIYSwitTYuRpZoUIIIYQQZSSJnY0ZrDKRl01IUAjxATdW\nmRhj4UlkQIUQQgghykCaYm2scJWJrF3hKJRulYmRLcmruansJy0cUDFhgm67bZvtAhZCCDe3dOlS\nNE0jIyOjxDKAzMxMevXqRa1atdA0jYSEBACDf9s7NmGZsLAwYhzcHcmaa7rqeyuJnY3pVpmI060y\n8e0U3bZfnK68rEwNqBBCCBeQkZGBpmlomsbzzz9vcp8TJ07g5eWFpmkO/6EuLiEhgS1btvDqq6+y\nfPlyhg4d6tR4TDl48CCjRo0iKiqKwMBAvLy8uPXWW3nooYeYP38+Fy5ccHaIZXbs2DFGjx5NVFQU\nAQEBBAYG0qhRI/r378/nn3/u7PDKLSMjg8mTJ3PmzBmnxSBNsTZmdpWJPK3sJy0cUCErVAghXJSP\njw8rVqzgzTffxNvb2+Cx5cuXo5SiShXH/eQ8/fTT9O/fHy8vL33Z5cuX+fe//83zzz/P6NGjDfa/\nePEinp6eDovPnKVLlzJs2DCqVKlCXFwcw4YNw9fXl2PHjrF161aef/55Vq9ezTfffOPsUK2WlZXF\nXXfdxdmzZ4mPj2f48OEA/PHHH6Snp5OSkkKfPn2cGuP+/fvRtLL/XmdkZDBlyhQSEhK45ZZbbBiZ\n5SSxs7ESV5koKxlQIYRwcb179+aTTz5hzZo1xMXFGTyWkpLCQw89xObNmx0Wj6enp1Gidvz4cZRS\n1KhRw2h/Hx8fR4Vm1ubNm0lMTCQqKoovv/yS2267zeDxcePGcejQIT799NNSz3Xu3DkCAgLsFWqZ\nzJkzhxMnTrB69WoeffRRo8ePHTvmhKgMFf+jxB1JU6yNxQcs1je/0nWSvlk2PmBx+U7coQOMHWuY\n1G3bBjNmSJ87ISqZ1F9TCUsOw2OKB2HJYaT+murskGjdujUtWrQgJSXFoPy///0ve/bsYdCgQWaP\nXb16NR07dsTPzw9/f386duzImjVrTO77wQcf0LRpU7y9vbn99ttJTk5GKWW0X/H+TwkJCYSGhgIw\nZcoUffNx4ePm+tht2rSJ+++/n1tuuQUfHx9atGjBggULyhWbOWNujLBbuXKlUVJXKCIigrFjxxqU\nFfYL+/nnn3nggQcICgqiRYsWgC7Be/3112nXrh3BwcH62F577TXy8/MNzlPYrL506VLef/99mjRp\ngo+PD82bN2f9+vUA/Prrr3Tv3p3AwEBq1qzJyJEjuXr1qkXP7/fffwegW7duJh+vW7euyfLffvuN\nHj16EBAQQFBQEH379jVKAidPnoymaezdu5dRo0ZRr149fH196datG/v37wfg888/p3Xr1lSrVo2w\nsDAWLVpkdC1zfewseW8TEhKYMmUKAOHh4frP2OTJk0t8XWxNauxszNJVJso9elZWqBCiUkr9NZUh\n64aQf1X3o5yVl8WQdUMAiG8e78zQGDx4MP/3f//HkSNH9InJkiVLqF27Ng8//LDJY95//32ee+45\nmjZtysSJEwFdUtarVy8WLlzIkCFD9PsmJyfz0ksv0bJlS5KSksjPz2fOnDnUrl271NiGDh1Kq1at\neOmll+jdu7e+yS8yMtLsMYsWLWLYsGG0b9+e8ePH4+fnx8aNGxk+fDgHDx5k9uzZNokNdIM6fvrp\nJzp37kyTJk0sOqao7OxsunbtSr9+/Xjsscc4f/48AEeOHGHx4sU89thjPPnkk1SpUoUtW7Ywa9Ys\nfv75Z5NNuvPmzeP06dM888wz+Pj4MHfuXHr37k1aWhrPPvssTzzxBL169WLDhg28++671K5dm9df\nf73UGBs2bAjokqRRo0ZZ1OR55MgRYmJi6N27N7Nnz2b37t0sXLiQs2fPsmHDBqP9Bw4ciL+/P+PG\njSMnJ4c333yTBx54gGnTpjFmzBiGDx/O4MGD+fDDDxk6dCh33HEH99xzT4kxWPreDh06lLNnz/LF\nF1/w9ttvExwcDKBPsh1GKVXpbm3atFHONm7xRoXvCcXAGMVkdFvfE2rc4o2WnSApSSlPT6VAt01K\nsm/AQgir7N271y7nDX07VPedUewW+naoXa5XmvT0dAWo2bNnq5MnTyovLy81ffp0pZRS+fn5Kigo\nSL388stKKaX8/PzUvffeqz/21KlTys/PTzVs2FDl5eXpy/Py8lRERITy9/dXp0+fVkopdfr0aeXr\n66siIyPVhQsX9Pv++eefys/PTwEqPT1dX56SkmJUlpmZqQA1adIko+cBqIEDB+rv//3338rb21s9\n8cQTRvuOHDlSeXh4qIMHD5YpNlPWrl2rADVy5Eijxy5cuKBycnIMbgUFBfrHQ0NDFaA++OADo2Mv\nX76srly5YlT++uuvK0D9+OOP+rLC9/LWW29VZ86c0Zfv3r1bAUrTNLVq1SqD87Ru3VrVrVu3xOdW\n6ODBgyowMFABqkGDBurJJ59Ub7/9ttqxY4fJ/Quf16effmpQPmLECAWo3377TV82adIkBaiHH37Y\n4LV55513FKACAgJUdna2vvzEiRPK29tb9e/f3+iaRT+j1r63hXFkZmZa9JooZdl3BbBDWZjjSFOs\nk5R79KysUCFEpZSdl21VuSPVrFmTRx55hKVLlwK6pq+8vDwGDx5scv+NGzdy4cIFRo4cSWBgoL48\nMDCQkSNHcv78eTZt0k0VtWHDBvLz83nuuefw9fXV71u/fn3i421fU/nZZ59x+fJlEhMTOXnypMGt\nZ8+eFBQU2DS2s2fPAhi8DoUmTpxIrVq1DG65ubkG+9SoUcNkc7eXlxdVq1YF4Nq1a5w+fZqTJ08S\nG6trRfrxxx+NjklISCAoKEh/v0WLFgQGBnLrrbcaDW645557OHbsmL6GsCQRERHs3r2b5557DoAV\nK1bw0ksvER0dTYsWLdi5c6fRMbfeeqtRn82uXbsCN5t2ixo5cqRBTWCnTp0AeOSRR2jQoIG+vFat\nWjRp0sTkOYpy9OfOFqQp1knKPXpWBlQIUSmFBIWQlZdlstwVDBo0iB49evDdd9+xZMkS7rrrLu64\n4w6T+2Zm6gaVNWvWzOixwrJDhw4ZbJs2bWq0r7nzl8e+ffsA9AmQKcePH7dZbIUJXWGCV9TQoUPp\n3r07ALNnzzbZBNmwYUOzo3rff/99FixYwJ49eygoKDB47PTp00b7R0REGJVVr17dIDEqWg6Qm5uL\nv78/58+fN0ryatSooR+dHBYWxnvvvcd7773H0aNH+e6771i+fDnr1q3j4YcfZs+ePQaDW0zFUrNm\nTf01S4u9ML7w8HCTsWdlGf9fKsrRnztbkMTOSUyOnr1Qh4A2XxGWHGZZvztZoUKISmd6t+kGfewA\nfKv6Mr3bdCdGddMDDzzAbbfdxpQpU0hPT2f+/PnODqlM1I2O8R999BH16tUzuY+ppKOsoqKiANi1\na5fRY40aNaJRo0YAfPzxxyaPL1qbVNRbb73Fyy+/zP3338/IkSO59dZb8fLy4siRIyQkJBgleoDZ\nBLGk6WAKX685c+boBxAUSk9PNzkgoV69evTr149+/foRHx/PihUr+PLLL3nqqaesumZ5Yjd1Dncn\niZ2TxAcsJimtpa45NjwDwtPhn19w9n+Pc7Z/bwjP0q1akdaScXM3Aeb/atSTARVCVHiFAyTGbx6v\n/wNwerfpTh84UcjT05MBAwYwY8YMqlWrxhNPPGF238LEaM+ePUYjJffu3WuwT+H2t99+M7uvLRUm\nUsHBwSXW2tkqtvDwcFq3bs13333H/v37yzSAwpTly5cTFhbGV199hYfHzd5XX3/9tU3OX9yAAQOM\nBiO0bNmy1OPat2/PihUrOHLkiF3iKitr39vyzIFnK9LHzkl0o2d3E9oqEw2N0FaZBA5MgGaflr3f\nnaxQIUSlEN88nsOjDlMwqYDDow67TFJXaNiwYUyaNIkFCxaY7DNW6L777sPPz493332Xc+fO6cvP\nnTvHu+++i7+/P/fdd59+32rVqjFv3jyDaTr++usvVqxYYfPnEBcXh7e3N5MmTeLixYtGj+fl5XH5\n8mWbxjZz5kz9tf/++2+T+1hbw+Tp6YmmaQbHXbt2jTfeeMOq81gqIiKC2NhYg1thc2hGRobJ17Kg\noIB169YBrte8ae176+/vD8CpU6ccFmNxbl1jp2laODAXuBu4BiwBxiuljOuWXYyuaTWW6RzWl3lM\n8YBH1oD/8RL73ZmdKuWvJxnjNU1WqBBCOFVISIhFc3fdcsstzJo1i+eee4527drp55FbunQpf/zx\nBwsXLtR34q9evTrTpk1j9OjR3H333QwYMID8/HwWLFhAo0aN+Pnnn236HOrXr8/8+fN55plniIyM\n5OmnnyY0NJScnBx+/fVXVq9ezd69ewkLC7NZbLGxsfppOBo3bky/fv1o06YNvr6+HD9+nK1bt7Jh\nwwbq1atn8YTKffv2ZezYsTz44IP06dOHs2fPsmLFCv2ACkeaM2cO33//PT179qR169YEBQVx7Ngx\nVq1axc6dO+nSpQs9evRweFwlsfa9bd++PQCvvvoq8fHx+Pj4EBUVpW9qdwS3Tew0TfME1gHfAH2B\n2sB64Aww04mhlZmlq1bk1dxE0siW0C+8WJPtbnhKBlQIIdzHiBEjqFevHrNnz9b3zWrZsiVffPEF\nvXr1Mtj35Zdfxt/fn7feeouxY8fSoEEDRo8eTVBQkNmRt+UxaNAgGjduzJw5c1i4cCFnzpwhODiY\nJk2aMG1FiAqlAAAgAElEQVTaNIMJdW0VW0JCAp06dWLu3Lls2rSJtLQ0rly5QnBwMC1btmTevHk8\n/fTT+Pn5WXS+V155BaUUH374IS+++CJ169bl8ccfZ9CgQQ6vHXv99ddJS0tj69atfPPNN5w6dQo/\nPz8iIyN58803ee655wyai12FNe9tx44dmTlzJgsWLODZZ5/l2rVrTJo0yaGJneauHQc1TbsD+B/g\np5S6eKMsAZiklDIe/lJEdHS02rFjh/2DtNL4DwsTthv97jJjIG0l4+buZnrizT4eYclhugQwbaVu\nVO2O4dAvjtBWmRweddj0yWVQhRAOtW/fvhInvxVCCLDsu0LTtJ1KqWhLzueQGjtN0xoDTwH3Aw0B\nH+AgkAYkK6UulOW0xbaF/w7TNC1QKWU8ZtzFWbpqhdVTpcigCiGEEKJScFRT7GDgOWAtkApcBboA\n/wDiNE1rX6TW7Z/A4yWcq4tSKgPYjy45nK5p2jigLvDSjX0CAbdL7Ez1uyvsTxeWfLM/XcDfj3B2\nx4PwW58Sm2z1TA2qkMROCCGEqHAcldh9BsxQSuUVKVugadrvwHggEXjvRvmzwPMlnCsPQCl1TdO0\nnsDbQBZwCvgQXf864xkX3ZSp/nT8c6nuwf69b06VkraS+E67TZ+kcJUKGVQhhBBCVGgOSeyUUuY6\ntH2KLrGLKrLvOeCcmf2Ln/c34MHC+5qmPQdsL2PTrkvSLT1WrD9d1KcEtvmK6pElN9nqySoVQggh\nRKXg7FGx9W9sj5flYE3TWgCHgEvomnbHAwNtE5prMNmfruskzqGRN+rmrC6mmmwNVq2QVSqEEEKI\nCs9p44pvTFcyAd38c2WdXbIfumbYPOAN4Fml1EbbROgaQoJCdKNji06BkhljtC5kYZNt1q5wFEo3\nBcrIluTV3GT6xIUDKiZM0G23bbP/kxFCCCGEXTmzxi4Z6ACMU0rtL8sJlFIT0CWHpdI0bQgwBHST\nZ7oLk0uPmehPZ7LJtl8cqecyDQZj6BUbUDFr6kXy+pqY9NjcOrVCCCGEcDlOqbHTNG0augESi5RS\nMxxxTaXUIqVUtFIqulatWo64pE2YWnps3FxzU6Bk3GyyjZ5/YwqUbNMnLhxQ4ekJXl7ktc+2rsZP\nCCGEEC7H4TV2mqZNBl4HUoBhjr6+uzE1BYoplq5aAYX98S6QOimQ7EvHCfEJ5PSh1dD0suU1fkII\nIYRwOQ5N7G4kdZOAZcAzyl2XvXBBljbZQtEpVCIh/DhZ+yIh7QPdsaWsUyuEEEII1+WwxE7TtIno\nkrrlwGClVEEphwgrWLpqBZjvjwdYVOMnhBBCCNfkqCXFngOmANnAJuBJTTOoCTpe0UazOpqlTbZg\nZgoV0CV6FtT4CSGEEMI1OWrwRNsb2xB0zbDLi93GOygOgZkpVP7Xn8D+gwitsQVNQWiNLYwb8y/z\nkx4LIUQZxcTEEBYW5uwwANeKpSLKyMhA0zSWLl3qstesaJ8BhyR2SqkEpZRWwi3GEXEInfiAxTdr\n57pO0m339eH5u0dxuN/3FHgncbjf90yflCBTnQghLJKfn09ycjKdOnWiRo0aVK1alTp16vDQQw+x\ndOlSrl275uwQHW7dunX06dOH+vXr4+3tjb+/P5GRkSQmJvLtt986O7xy2bp1K4888ghhYWF4e3tT\nu3ZtoqOjGTlyJIcOHXJ2eOWWnJzs0GTUlpy98oRwghL743VAVqgQQljljz/+oEePHhw4cIDY2FjG\njh1LcHAwJ06cYNOmTQwaNIi9e/cya9YsZ4dqZMOGDdh6HN/Fixd58sknWb16NU2aNGHAgAFERERw\n/fp1Dhw4wPr161myZAkrVqzgiSeesOm1HWH+/PmMGDGCiIgIBg4cSIMGDcjJyWHfvn188skndO7c\nmYiICKfF17lzZy5evEjVqlXLfI7k5GTCwsJISEiwXWAOIoldBVe41JjBxMM1dRMPHx5zuPQTFK5Q\nceWKbt67zZsluRNC6F28eJGHH36YQ4cOsWrVKvr06WPw+Kuvvsr27dvZvn27kyIsmZeXl83POXz4\ncFavXs0rr7zCG2+8gYeHYePYnDlz+OKLL/D19S3xPEopLly4gL+/v81jLKtr164xbtw4QkJC+Pnn\nnwkMDDR4/MqVK5w/f95J0el4eHjg4+Pj1BicyWlLignHsHqpsSJmzYLxH2wgbOhFPF6/TtjQi4z/\nYAMu+Ee3EJXGrFmQnm5Ylp6O0/5fLl68mP379/Pyyy8bJXWF2rZty4gRI0o8z3//+18SEhJo3Lgx\nvr6+BAQE0LFjR7744gujff/8808GDx5MaGiovhnw7rvvZtmyZfp9CgoKSE5OpkWLFgQEBBAYGEiT\nJk1ITEzk6tWr+v3M9a/6448/GDRoEPXr18fLy4tbb72VRx99lJ07d5b4PH755ReWLVtGx44dmTlz\nplFSB6BpGn369KF79+76sqL9wubNm8cdd9yBj48Pc+bMsfr1SUhIQNM0cnNzSUhIIDg4mICAAHr1\n6sWxY8cAWLRoEZGRkfj4+NC0aVPWrFlT4vMqdPLkSc6cOUPbtm2NkjrQJco1atQweWxKSgrNmjXD\n29ub0NBQkzW4YWFhxMTEsHv3bmJjY/H396d27dq8/PLLXLt2jUuXLjF69Ghuu+02fHx86Ny5M/v2\n7TM4h7k+dqdPn+bZZ58lODgYPz8/YmJiTL6fmqaRlZXFli1b0DRNfzt8+LBFr5GzSY1dBWf1UmNF\n5NXcRNKnI6BfBtySQdbpGJLSRjBu7iZABlUI4Qxt20JcHKxcCV266JK6wvvO8NlnnwEwZMiQcp3n\niy++4LfffiMuLo7Q0FByc3NZtmwZffr0ITU1lSeffBLQ1Rjdd999HDlyhBEjRtC4cWPy8vL45Zdf\n+Pe//83AgQMBmD59OhMnTqRnz54MGzYMT09PMjMzWbt2LZcvXy6xmW7Hjh1069aNq1evkpiYSFRU\nFKdOnWLLli388MMPtGnTxuyxq1atAiAxMZFisz9YJDk5mdzcXJ599lnq1q1LgwYNrHp9iurevTv1\n69dn6tSp/PHHH8ydO5fevXvTp08fFi1aRGJiIj4+PsydO5e+ffty4MABwsPDS4yvTp06+Pv7s3Xr\nVvbv30+TJk0sel4LFizg+PHjJCYmcsstt/Dxxx/z6quvUr9+faPY//rrL+677z4ef/xx+vbty4YN\nG3jrrbeoUqUKe/bs4eLFi7z22mucPHmSOXPm0KtXL/bt22cyiS509epVHnjgAbZv387TTz9N+/bt\n2bVrF7GxsdSsWdNg3+XLl/PSSy8RHBzM+PE3x3a6zapVSqlKd2vTpo2qLLTJmmIyis5TFCjddjJK\nm6yVemzo26GKgTEK3xO643xPKAbGqNC3Q+0fuBBubu/evXY797ffKhUcrNSECbrtt9/a7VKlqlGj\nhgoMDLTqmHvvvVeFhoYalJ0/f95ovwsXLqjGjRuryMhIfdnu3bsVoGbOnFniNe68806D4yyNpaCg\nQDVr1kx5e3ur3bt3G+1//fr1Es/Xp08fBaiffvrJ6LHc3FyVk5Ojv+Xl5ekfS09PV4CqXr26On78\nuNGxlr4+Sik1cOBABagRI0YYlL/00ksKUA0aNDC4duFr+tprr5X43ArNmTNHAcrT01O1bdtWjRw5\nUn388cfq6NGjRvsWPq969eqpM2fOGMQeHBys2rdvb7B/aGioAtTKlSsNylu3bq00TVOPPPKIKigo\n0Je/8847ClBff/210TVTUlL0ZQsXLlSAmjhxosF53377bQUYfR5DQ0PVvffea9HrUV6WfFcAO5SF\nOY40xVZwJqc2yYzRlZeixPVnt22DGTN0WyGEQ3XpAsOHw7Rpum2XLs6L5ezZswQEBJT7PH5+fvp/\n5+fnk5ubS35+Pl27dmXfvn2cPXsWgKCgIADS09M5ceKE2fMFBQVx5MgRvvvuO6vi2LVrF3v27GHQ\noEG0aNHC6PGSaoUAfZymmikbN25MrVq19DdTtWwDBgygdu3aRuWWvj5FjRo1yuB+p06d9NcoGl+L\nFi0IDAzk999/L/G5FXr55ZdZu3Yt999/P3v37mXu3Lk89dRT1K9fn8TERPLz842OGTRokP69A/D1\n9aV9+/Ymr3nbbbfRr18/g7J77rkHpRQvvPCCQU1o4XMqLfbVq1fj6enJyy+/bFA+fPhwk++VO5Om\n2ArOmqXGijO7/mzkPhlQIYQTpafD/PkwYYJu26WL85K7wMBAzp07V+7znDhxgtdff501a9aYTNjO\nnDlDYGAgoaGhjB8/nhkzZlCvXj1atWpFt27d6NevH23bttXvn5SURK9evejUqRO33norMTEx9OjR\ng759+5Y4YKIwQbjzzjvL9DwKkwRTidbnn3/OlStXALjvvvtMHt+4cWOT5Za+PkUVH5lavXp1AJPN\nrdWrVyc3N1d/Pycnh+vXr+vve3p6GjRF9uzZk549e3L9+nX27t3L5s2beeedd1iyZAlVqlRh4cKF\nJcYCULNmTYNrFjIXn6nHCstNnaeoQ4cOUa9ePaPXyNvbm4iICE6fPl3i8e5EauwqON3UJrsJbZWJ\nhkZoq0zGzTW91FhxJue7S1tJ/P7huqTu+nXdNiPD/k9ECAEY9qmbOlW3jYszHlDhKFFRUZw9e7Zc\nc5cppbj//vtZtmwZAwcO5NNPP+Xrr79m48aN+lqtgoKbq1D+4x//4Pfffyc5OZmGDRuyePFi7rrr\nLl599VX9Ph06dODgwYN89tln9O7dm127dhEfH0+rVq04depU2Z9wKaKiogBdzV9xnTt3JjY2lthY\n89+/pkbKWvv6FPL09DR5DXPlqsi0L23btqVevXr6W9Gkufi5mjdvzqhRo9i+fTtBQUEsW7bMICks\n6ZrWxGdp7JWdJHYV3JgxMD0xlsOjDlMwqYDDow4zPTHWoomHzSaF1RJ0NXWenrptTIy9n4YQ4obt\n228OnADdduVKXbkzPPbYY4BudGxZ/fLLL+zevZvXXnuNWbNmERcXxwMPPEBsbKxRglAoIiKCF154\ngZUrV/L333/TuXNnZs2aZVCb5e/vz2OPPcZ7773Hnj17mDdvHvv27ePDDz80G0thjZmpxMwSha/H\nhx9+aLNkoyyvT3mlpqayceNG/S01NbXUY4KDg2nYsCGXL1/m5MmTdomrrCIiIjh69KhRTerly5dN\n/lFSloEvrkISO2GW2aRwXqiu+XXaNGmGFcLBxowxbnbt0gWnrRLzzDPP0KRJE+bMmWN2yoydO3fy\n/vvvmz1HYS1M8UTof//7n9F0Hnl5eQbTlQD4+PgQGRkJoG9SM5VYtG7dGqDEGruWLVvSrFkzlixZ\nwp49e4weLy1Za9GiBQMGDOD777/ntddeM1mTZm3CZ83rYysdO3bU1y7GxsbSsWNHQNe/b8uWLSaP\n+f3339m7dy/BwcEuN4L00Ucf5fr167z55psG5fPnzzfZbO7v72/Xml17kj52omw6dJAVKoQQ+Pr6\nsn79enr06EGvXr24//77ue+++6hZsyY5OTmkp6fzzTff8Morr5g9R2RkJM2aNWPWrFnk5+fTpEkT\nDhw4wMKFC2nevLnBXGPp6ekMGTKExx57jCZNmuDv78/OnTtZvHgx7dq100+/ERkZSfv27WnXrh23\n3norR48eZdGiRXh5edG/f3+zsWiaRkpKCt26deOuu+7ST3dy5swZtmzZQvfu3XnhhRdKfE0WLFhA\nXl4es2bNYs2aNfTp04eIiAiuXr1Kdna2foqY0qYWKcvrY2/5+fnExMQQFRVF9+7dadSoEUopfvvt\nNz766CMuXbrEvHnzSh1k4miDBg1i0aJFTJ06lczMTDp06MDPP/9MWloaDRs2NFryrn379nz44YdM\nmDCByMhIPDw86Nmzp8EgFlcliZ2wDVmhQohK6/bbb+fnn39m4cKFrFq1iunTp3P+/HmqV6/OnXfe\nSUpKCvHx8WaP9/T05F//+hejR49m2bJlXLhwgaioKJYtW8bu3bsNEpeWLVvSp08fMjIySE1N5fr1\n64SEhDBu3DiDEY8vv/wyX375JXPnziUvL4/atWvTvn17xo4dS8uWLUt8Pm3btmX79u1MmzaNlStX\nsmDBAoKDg7nrrrv0NVclqVatGl988QVr165l6dKlLFu2jJycHKpWrUqDBg3o1KkTixYtoouFI16s\neX3s7ZZbbmHJkiVs2LCBtWvXcvToUS5dukStWrW49957eeGFFyx+Xo7k5eXFxo0beeWVV1i9ejWr\nVq2ibdu2bNy4kdGjRxtNPjx9+nROnTrFvHnzOHPmDEopMjMz3SKx0ypjh8Po6Gi1Y8cOZ4fhlkwu\nURawmKDPPBiz8X7dgApPT10z7dixzg5XCKfZt2+fvnlQCCHMseS7QtO0nUqpaEvO51p1pcLlmV2i\nrH22DKgQQgghnEyaYoVVzC5RFpTJ9M2bpY+dEEII4USS2Amr6FajyLq5GkXnqTdWo9BkQIUQQgjh\nZJLYCauYXY2iVabxzjKgQgghhHAo6WMnrGJ2NYoAE5OTZmTIChVCCCGEA0liJ6xi1RJlMTEyoEII\nIYRwIGmKFVbRzW4fy3QOl75zhw665lfpYycqKaWUWy9NJISwL3tMOSeJnbAvGVAhKqkqVapw7do1\nqlat6uxQhBAu6tq1a1SpYttUTBI74VgyoEJUEj4+PvrVF4QQwpRz587h4+Nj03NKHzvhWDKgQlQS\ntWrVIicnh/z8fLs0twgh3JdSivz8fE6ePEmtWrVsem6psROOVTigorDGTgZUiArKx8eHOnXqcOzY\nMS5fvuzscIQQLsbb25s6derYvMZOEjthF2bXlM2NZYwMqBCVRFBQEEFBQc4OQwhRiUhiJ8rNVBJ3\n24E5/PDRg/BkOIRn6daUTWvJuLmboEOs6YROBlUIIYQQ5SKJnSi3vJqbSBrZUreG7I0kLivtXujy\nuvGasucyTU+VIoMqhBBCiHKTxE6UW+q5Z3RJXbEkjvAMuFTdeE1ZU0wNqpDETgghhLCKJHai3LLz\nsiE8S5fUFUniyIyxbE1ZsGhQRYn99sbY8QkKIYQQbkISO1FuIUEhZO0KN0zifE7Dd+Nu1tyFp+vW\nlO202/RJLFilwlSTr77fHiaWNBNCCCEqGUnsRLnFBywmKa2lYRK3Yh13D/iaI1GZZOdphLTKJOL8\nGbZ9FUrYuTDTNW6lrFJhrsnXbL89IYQQopKRCYpFuQXlxjJu7m5CW2WioRHaKpNx8//Do437cnjU\nYQomFXB41GE6PJhF+le3kLUrHIXS1biNbElezU2mT1w4oGLCBOjW7UaTb8bNJt/o+Tf67WU79PkK\nIVzDrFmQnm5Ylp6uK5d4HMeVnrcrxeIsktiJchszBqYnxhokcdMTjfu96Wrc4nQ1bt9O0W37xenK\nTSk2oCKEION+e5kxhASF2PspCuEU8iNVsrZtIS7u5muUnq6737atxONIlj5vR3yey/seVIj/c0qp\nSndr06aNEo6nTdYUk1F0nqJA6baTUdpkzfQBP/ygVLVqSnl6KlWtmho3OUXhe0IxMEZ3noExCt8T\natzijY59IkKU08yZSn37rWHZt9/qyouXBQff3Lf4fXHzNZkwwfxrY+nr7ah4KiJLnrejPs/leQ9c\n9f8csENZmOM4Pclyxk0SO+cIfTtUn4zReYo+SQt9O9T8QT/8oFRSklI//KBmzlRq3OKNKvTtUKVN\n1lTo26Fq3OKNdvlyFsKerPnxqKyJgjUmTND9mk2YYPpxR/9YlxZPRWXJ83bU59mSWMwl/EOGuN7/\nOUnsJLFzSeMWb7RNjVuRZE8Id2XND1xlTRQsYenr6KiEwpUScVetqSz6ebZHjNZ+Jkwl/K72f04S\nO0nsXJJNatyKNc9KcifcmSvVcLgja2vi7P1j7WrNeOWJx5qEqzw10G++advXzNrnbOr/lyv+n5PE\nThI7t2Iq4evy2AHV5bEDxklg9826pA5026QkZ4cvRJm4Up8ke3BEbVFZko/SfqzLE7epY4cM0d3K\ncj5bKGuSYs1nr7x9RguTO1skUmV5/4om/K76f04SO0ns3IrJJlrv07pb8WbbySlSYyccztZJiqU/\nHvZIjhzVPOdKP5BlqVFydA1Sed+Xko4va02lrWuu7BFjeRV/jkOGOK752hqS2Eli51bMDaowO9BC\n+tgJB7P1j70j+z4VZ+q5+Prqak1sHY8tEwNb16SVdKytExpH1M7aqzbMEQlXWV/v8v4/cqU/Pkoj\niZ0kdm7F3DQoVk+NIsmesCNX6ndjqx80e/VzKspWiYGjf4Rt3cHfEf0pbf2+OuIzX573tbyfCWf+\ngWUtSewksXMrVtfYFScDKtyGO32RmuIqI+Vs0bxX/LnYo0nKGTVf1rB0ugtHJkjl/YyVJSE1td+b\nbyrl52f/RNrWf6S4Ym2bLUhiJ4mdW7Gqj52pqVGSkmRAhZtwVtOHLRJKV/sBsSQRM/fjbK55zpad\nyO31XtsyuTYVY2CgUkFBtmvStOZ1sHWNXXlitFfzvD3Y+w8uV/iDtEIkdoA38AFwCDgHHABeKLZP\nHPAdcB44bOm5JbFzLVaNijX1H0lq7NyKMxKk8v5wuWpfHEsSseIJibnaJ1OJiyXvlbkfvQcftP2P\noT0+O9bUVJYlgXDUKiO2Ot5VVvCwlKs3F9tKRUns/IBpwO3o1rRtBRwH4orscx/QH3hRErtKTvrY\nuRVnNGkW/wGwpnnNXX7QzP3IldY8V1Kzm6us6mDpdco73UVp17dXAmHPUbGWsuV77Yj/M45MuJxd\nY18hEjuTwepq8OaaKO8liZ0wIsmeS3LmF2Rp/cqcXQNnqZJ+0MryHMu7tJIjXkd71XxZEru7N1Va\nwtL30Nr9bJV0ucIcgc7sY+vwxA5oDEwF/gPk3Gg63QWMB/xsdI2qwG/AMyYek8ROGJLmWatY+qVZ\n3i9SZzZpWFKbZQ+OnIvOlp3+rX2vbPk6OqpDfXlqAcszuMCa5+eKNV+WvtdlTfidOZjDHGf/EeiM\nxO6NG8lcKvACMAz4FFDAbqBakX3/eaPc3C3GzDUWAtsBLxOPSWInDMmACquY+mI31YncXMfy8nxh\nW/ojVZ5jzf1wlWeOL0f1nbKUrWuV7LGqg6Vs8ZpZknw4a0SmNc/PEbWF9nyvy5Lw2+P/a3k48w/S\nQs5I7KKBIBPl/7iRrD1fpCwACC7hVtXEed4CfgGCzVxfErtKyuzAi9jtKvQlTWmTUKEvaapL7HbL\nB2NUUqa+sC0tc2R8tqohKW8NQFl+nF2hqdLWyvujV95m4JJicsRn1BGrOhTftzw1sfb4A6m0GlFb\nvofOaA51hT62LtPHDmh+I7FbUI5zJAO/ArVK2EcSu0rK4qlSrJk+pRIz9aVpaVlZOatmyNFToLjK\nHHi2Zs/Z/21Z22OP5K68n0drnp+t+ofa+g+k0v6v2rLW1Vl/VJri6GTPlRK7B28kdlPKePxc4H/m\nkjrAE/AB+gFZN/7tXdp5JbGrOKyZ3NjiCY8rKWfV2Fn7A1CWCVjtyRVGVDqKvUZumqqdK+tr5i5r\n4Zanxq68tVfuUINsy1pKe3B086xLJHY3kq4fgKtAkzIcH3ojKbyEbp66wttXRfZJMNFHr9SaO0ns\nKg5rliOzaomySsbUl5Q9+tiVdn1rR+TZ48ve1jWIjv4BsCdbJTOl1c65w2vmqCZNc/u6wxqwZWXq\nObviKGRH/sHmKonduzcSrbH2uoaV8QwBdgA7QkJCyv0iC9cgNXa24ahRsSUp6xxatu5QbemPrqX7\nuUKtoi2V98fMklpgWyxn5srKOyrWVv1DXbUG2Z3+zzgqQXZ6YoduYmEFLLTH+ct7kxq7ikP62Lkm\na7+YLfmhsWbd0/IqbzwVXXlfb3ernXM1jhwAIcyrNDV2wOQbSd0SQLP1+W1xk8Su4rB0OTJ92Rt1\nlDYZFfpGnUo7KtbV5sWyVfOerb9cy5K8VIZkz9Y1dhW9ds7VVIbPqCM4OkF2WmJXJKlbCnjY8ty2\nvEliJwxUshUqHPWFZGkC4Iq1D2VNXip6bYg9+9gJ4U4qxahYYOKNpO4jV07qlCR24oaZM5UaNznF\nYL67cZNTKsVfro5qQnDGqg7OXh3D1fsvlYcrrGcqRFm4+2fPGRMUP3cjqcsCBgBPFbvdZ4vr2Oom\niZ1Qykz/PDfud2ftF5e9ky5nJTjlTcwcsZi6va4rhDDN3WuLnZHYLTUx7UjRW4YtrmOrmyR2Qinz\nI2rddaRsWfq12SvpcvaXqDNrzaQZVwjX5M616U4fFevqN0nshFLm58Bzpbnt7DG61NYJhD2aQ23B\nGfN0STOuEK7NlefvK4k1iZ0HQlRSIUEhkBkDO4ZD56m6bWaMrnzbNpgxQ7d1orZtIS4O0tN199PT\ndffbtjW9f5cuMHw4TJum23bpYrzP9u2wcuXNx7p00d3fvt12MX7+OfTvbxzbmDFlu4a10tNh/nyY\nMEG3LYzN3sr72lry/gkhysZZ3wsOZ2kGWJFuUmMnlCqhj93kFKWqVVPK01O3dfJoWWtqcZzdr81W\n13XFkbKOIDV2QtiHO38vKCU1dkJYJCg3lnFzdxPaKhMNjdBWmYybu5ug/4TAlStw/bpum5FRruvM\nmmX8l2F6uq7cEpbW4hTW5q1cCVOn6rZFa9LsydY1TdbWVBZl6xpJR3Hm+ydEReeu3wtlYmkGWJFu\nUmMnzDE3BcqDDzqvBskR88GVlz1qmipb7ZWMihVCmIMMnpDETpSNuebZJ0fvUAHVL6o6I/orbbKm\n6ozorwKqXyxzcmZqtn1TAw5MrQnpaoth27OJw107OgshhC1JYieJnSgjc1Og1JxZU3kndjco907s\nrj7+5WOLz13a+piBgUoFBZWexFmzALgjaoHsdY3KVmMnhBDmSGIniZ0oI3NToOhvxcotnfPOVJJi\naZml5ytpP3frMOyucQshhD1Yk9jJ4AkhijA3BQpgsjw7L7vUc5rrFA/GAw4sHYRgzX6F15s48WYc\nrnYpoacAACAASURBVD6NRqXq6CyEELZkaQZYkW5SYyfMMdfHzvfh8SbL64zoX+o5zTVVDhli/xq7\nQtJXTQgh3BdSYydE2ZibAqXRmeF4PzEAwjN0O4Zn4P3EALp4Gc64a2pqk7ZtTdc0ff65YS1er17Q\nu3fp011YOy1GpZmUUwghBJouEaxcoqOj1Y4dO5wdhnAzqb+mMn7zeLLzsgkJCqFD1hdEROWSevIp\nsi8dJ8SnDh1P/os189uwbp2u+bBoEla0+XPWLF3CV7Rs6FDdduHCm2Xp6bqksOiKDaaONbVfYXnR\n65uLRwghhOvSNG2nUiraon0lsROibMZ/uImkkS2hX5yuJi8zBtJW8uSIbDYsbcPw4boaMmcmUdYk\ngUIIIVyTJHalkMRO2EJYchhZu8IhbSVEz9cNqOgXR2irTAacOsy0abrmz6lTnR2pEEIId2ZNYid9\n7IQoo+y8bF1NXfR82DpRtw3PIGtXhPRpE0II4RRVnB2AEO4qJChEV2NXdAoUn9N4fP86K9fdnL5E\n+rQJIYRwFKmxE6KM4gMW65ph+8VB10m6bfo0+g8/TBefbTBjBl18tsn8a0IIIRxGEjshysjk1Cjz\n/0PBsVOEpXXE4/I4wtI68vepyTJQQQghhEPI4AkhbCj111SGfD6IfK7qy3ypyqI+KcQ3j3diZEII\nIdyVDJ4QwknGbx5vkNQB5HOV8ZvHOykiIYQQlYkkdkLYkLm1Y7PysghLDsNjigdhyWGk/prq4MiE\nEEJUBpLYCWFDIUEhJss1NLLyslAosvKyGLJuiCR3QgghbE4SOyFsaHq36fhW9TUo09BQGPZlzb+a\nL82zQgghbE4SOyFsKL55PIt6LiI0KFQ3UjYo1CipK2Su2VYIIYQoK5mgWAgbi28ebzACNiw5jKy8\nLKP9zDXbCiGEEGUlNXZC2Jmp5lnfqr5M7zbdSREJIYSoqCSxE8LOTDXPLuq5SFert023QgXbtjk7\nTCGEEBWATFAshLNs2wbdusGVK+DlBZs3Q4cOzo5KCCGEi5EJioVwBxkZuqTu+nXdNiPD2REJIYRw\nc5LYCeEsMTG6mjpPT902JsbZEQkhhHBzMipWCGfp0EHX/JqRoUvqpBlWCCFEOUliJ4QzdehgnNBt\n2ybJnhBCiDKRxE4IVyIDKoQQQpSD9LETwpXIgAohhBDlIImdEC4ktfFlwkYW4DEJwkYWkNr4srND\nEkII4UYksRPCRaT+msqQ/bPJClIoDbKCFEP2z2bEv0YQlhyGxxQPwpLDSP011dmhCiGEcFHSx04I\nFzF+83jyr+YblOVfzWfBjgUodBOJZ+VlMWTdEACD9WiFEEIIkBo7IVxGdl62yfLCpK5Q/tV8xm8e\n74iQhBBCuBlJ7IRwktRfUw2aWGtUq2HxseaSQCGEEJWbNMUK4QSpv6YyZN0QfdNrVl4WVT2q4uXp\nxZXrV/T7aWhGNXYAIUEhDotVCCGE+5AaOyGcwFR/uqsFVwnwCiA0KBQNjdCgUIZFD8O3qq/Bfr5V\nfZnebbojwxVCCOEmpMZOCCcw15R66uIpTo45aVDWMaQj4zePJzsvm5CgEKZ3m64bOCErVAghhChG\nEjshnCAkKISsvCyT5cXFN483GgGbumoy47+fSnagIiRNY/rfE4l/bLK9whVCCOEmpClWCCeY3m16\nmZtYU39NZcj/kgznu/tfksxvJ4QQwrUTO03T3tc07U9N085qmnZE07RkTdO8LH1cCFcV3zyeRT0X\nGfSnW9RzkUVz043fPJ58rhqU5XNVpkARQgiBppTxiDtXoWnaHUCWUuqCpmnBQBqwRSk12ZLHzYmO\njlY7duywb/BC2InHFA+TI2U1NAomFTghIiGEEPakadpOpVS0Jfu6dI2dUmqvUurCjbsaUAA0svRx\nISoic1Od6Mu3bYMZM3RbM4rPoSfNuEIIUTHYJLHTNK2xpmlTNU37j6ZpOZqmndM0bZemaeM1TfMr\n57lf0zTtPHACaAkkW/O4EBVNif3ztm2Dbt1gwgTd1kRyVziHXlZeFgqlX6ZMkjshhHB/tqqxGwy8\nBBwEpgKvAPuBfwA/aJpWrXBHTdP+qWmaKuEWU/TESqk3lFL+wB3AAuCoNY8LUdGU2D8vIwOuXIHr\n13XbjAyj482tSSt99IQQwv3ZarqTz4AZSqm8ImULNE37HRgPJALv3Sh/Fni+hHPlmSpUSu3TNG03\nsBzoYu3jQlQkpqZAAXRz2nl56ZI6Ly/d/WLMzaEny5QJIYT7s0mNnVJqR7GkrtCnN7ZRRfY9p5Q6\nWcLtqonzFKoKNC7H40JUaKn+hwibFIjHhOuETQok1f+Q0T6l9tETQgjhtuw9eKL+je1xaw/UNC1I\n07QETdNu0XRaAK8D31jyuBCVjb7v3KXjKCDr0vGbfeeKDKgozxx6QgghXJvdpjvRNM0T+DfQFohS\nSu238vhA4HOgNeCFbnDE58CkG9OblPh4SeeW6U5ERRSWHGZyNYtQnzocnnL2ZvPs5s2k+h8yvUyZ\nEEIIl2PNdCf2TOzeRdeXbpxSaoZdLmJdPEOAIQAhISFtsrKMfwCFcGfm5rcDCD0D2UEQkgfTa/Yj\n/vWVDo5OCCFEWTl9HjtN06ahS+oWuUJSB6CUWqSUilZKRdeqVcvZ4Qhhc+b6yGlA1i3olh+7BYao\ntTK1iRBCVFA2T+w0TZuMrq9bCjDM1ucXQphmqu+chmZUh5dfcFmmNhFCiArKpondjaRuErAMeEa5\n8nplQlQwpua3M9c0m52XbdEKFUIIIdyLreaxQ9O0ieiSuuXAYKWULFophIMVn9/O3ICKEJ/aupUp\nigyooEMHR4YqhBDCDmy1pNhzwBQgG9gEPKlp2lNFbvfZ4jpCCOuYndrkaudSV6gQQgjhfmxVY9f2\nxjYEXTNscVuAjTa6lhDCQoW1d0ZTm5yPgKT1Ja5QIYQQwv3YbboTVybz2AmBrm9dRoYuqZNmWCGE\ncFnWTHdisz52Qgg306GDcUInyZ4QQrg1SeyEEDrbtsmACiGEcHP2XitWCOEuMjJkQIUQQrg5SeyE\nEDoxMbqaOk9PGVAhhBBuSppihRA6HTroml+lj50Q/9/e/UfZXdd3Hn++MySQARyLWpXizPgDsF2C\nbh26nbXbhsa2iqW61h9bRmsKOLKCAttjWxggQBgj2kqsrIeOVeCwY6G0Fk2LP3M2VEs47ejSpqtG\nKpuJP9OIMgQmJCH57B/fO+HOnXtn7r1zf9/n45x7bub7/d7v/Q6fE3jx+Xzen4/Utuyxk/S04WG4\n4oqjoW5yxySDNz6PFdcFgzc+zz1mJanFGewkFTW5Y5LRey5g+sk9JGD6yT2M3nNB0XA3uWOSwc2D\nrLhuBYObBw2AktQkBjtJRY1tHWP2yIF5x2aPHGBs69i8Y5M7JhndMsr0zDSJxPTMNKNbRg13ktQE\nBjtJRe2e2V3W8bGtY8wemp13bPbQ7IIAKEmqP4OdpKL6+/rLOl5uAJQk1Z/BThKwcJ7cOaeeQ+/K\n3nnX9K7sZXzdeLaY8aZNsH172QFQklR/BjtJRefJ3f7Pt/P2l72dgb4BgmCgb4CJcycYefxF2Q4V\nV18N69Yx/oL1pQOgJKmhXMdOUsl5cvc+dC+7Lts1/+JNm+btUDHyrWPh3AnGto6xe2Y3/X39jK8b\nZ2TNSON+AUkSYLCTRGXz5CZPO8DYe46w+xnQ/9gRxk87wMiakQVBbnLHpGFPkhrMoVhJZc+Tm9wx\nyejODzLdl0gB032J0Z0fXLC0iUugSFJzGOwkMb5uvKx5cosubZJXUOESKJLUHA7FSjo6RLrU0Omi\nQ7br1mVz71atYvcfPln6OklS3RjsJAEUnSdXqL+vn+mZ6YXH6YOD+44WVPTTxzSPFv28JKl+HIqV\nVLaSQ7ZnXAqrVkFPD6xaxfgZl7oEiiQ1gT12ksq26JDtyb8B27bB2rWMDA/DjlOtipWkBouUUrOf\noeGGhobS1NRUsx9D6nzbtx8NewwPN/tpJKktRcRXU0pD5Vxrj52k+ti+fV5BBVu3Gu4kqc6cYyep\nPrZtm7dDBdu2NfuJJKnjGewk1cfatfMKKli7ttlPJEkdz6FYSfUxPJwNvzrHTpIaxmAnqX6Gh4sH\nOosqJKkuDHaSGsuiCkmqG+fYSWosiyokqW4MdpIay6IKSaobh2IlNZZFFZJUNwY7SY1XrKjCggpJ\nWjaDnaSGmtwxuXAP2cdfZEGFJNWAwU5Sw0zumGR0yyizh2YBmJ6ZZnTLKBx5LSOFBRUGO0mqmMUT\nkhpmbOvY0VA3Z/bQLGMr/96CCkmqAXvsJDXM7pndRY9PP7mHwQ3PZfeTe+g/7hmMn/AwI9hjJ0mV\nMthJapj+vn6mZ6YXHA+C6Sf3AFnIG90yCpDNvbOgQpLK5lCspIYZXzdO78reeceCIJHmHZs9NMul\nn3kXg3e/khUHrmTw7lcy+dfXNvBJJak9GewkNczImhEmzp1goG+AIBjoG1gQ6uY88tRjTPclUsB0\nX2L0X9/H5I7JBj+xJLWXSKn4v1Q72dDQUJqammr2Y0gCBjcPFh2eLWagb4Bdl+2q7wNJUouJiK+m\nlIbKudYeO0lNVWx4tpRSxReSpIzBTlJTFRuefdbqZxW9tr+vP9uhYtOm7F2SNI9VsZKabmTNCCNr\nRo7+XLiQMUDvyl7GX7DeHSokaRH22ElqOcV68SbOnYBvfJ3Bd+5nxVWHGXznfia33sTkjkkGNw+y\n4roVDG4etMBCUldri+KJiFgN7ACel1I6Ie/4bcB5wMG8y9+YUvrcYvezeEJqP5M7Jhm95wJmjxw4\nemwlPURPDwcPP/2vgN6VvUycOzGvB1CS2lknFk9cD5Qqm5tIKZ2Q91o01ElqT2Nbx+aFOoBDHJ4X\n6iC3RdnWsUY+miS1jJYPdhHxCuDVwI3NfhZJzVNJRazVs5K6VU2CXUScFhHXR8QDEbE3IvZFxIMR\nMRYRxy/jvscAHwMuZv5wa76RiPhxRHwj930WhEgdqL+vvy7XSlInqVWP3fnA5cC3yYZN3wvsBG4A\n7s/NkQMgIu6MiLTIa23efd8L/J+U0t+X+N4/BU4Hng28DVgPbKjR7ySphRRb727lipWsomfesV5W\nMr5uvJGPJkkto1bB7q+AU1JKIymlj6SUbkkpvQUYB84ELsi79h3AcxZ5/QNARLwEuIgs3BWVUvpa\nSunfU0pHUkpTZKHuv9Xod5LUQopVyt76+lv5xBlXMTATRIKBmWDijCstnJDUtWoybJkLVcXcBYwB\nZ+Rduw/YV8Ztfwl4LvCtiABYCRwfET8C3lCiFy8BUcGjS2ojhevdAbAGRk7+Ddi2Dd601nXtJHW1\nes9HOyX3vqeKz/4l8KW8n4eB24CXA3sBIuItwOeAx4A1wDXA3VU+q6R2NTy8MNBt356FvbVrDXuS\nukbdgl1E9ABXA08Bn6z08ymlWeDosvMRsTc7nL6bd9m7gFvIevN+ANwBbFrGY0vqBNu3u0OFpK5U\nz+VONpP1sl2TUtq53JullLblL06cO/YrKaWfyq1fd2pK6fqU0qFin4+I0YiYioipvXv3LvdxJLWy\nbduyUHf4cPa+bZs7VEjqCnUJdhGxEbiEbPHgluhBSylNpJSGUkpDz3nOc5r9OJLqae3arKeupwdW\nrWLytAOMbhllemaaRGJ6ZprRLaOGO0kdp+bBLiKuBa4CbiWrapWkxhoezoZfN26ErVsZ+85tzB6a\nnXeJO1RI6kQ1nWOXC3UbgNuBC1M7bEQrqTPlFVTs/kLxnSjcoUJSp6lZj11EXEMW6u4Azk8pHanV\nvSVpOUrtROEOFZI6Ta22FLsYuA7YTbZEyXkR8da816/V4nskqRrj68bpZeW8Y+5QIakT1arH7qzc\nez/ZMOwdBS8nskhqmMIKWICJM650hwpJHS+6cRrc0NBQmpoqtVmGpHY2uWOS0S2j84olelf2MnHu\nBCOPv8hFiyW1nYj4akppqJxr67mOnSQ13NjWsdIVsMPDcMUVR0Pd5I5JBm98HiuuCwZvfJ7Ln0hq\ne/XeUkySGqpUpWvh8ckdk4zecwGzRw4AMP3kHkbvueDo+bGtY+ye2U1/Xz/j68YdtpXUFgx2kjpK\nf18/0zPTRY/nG9s6djTUzZk9coBLP3sp+5/af7TXb24xY8BwJ6nlORQrqaOMrxund2XvvGO9K3sX\nVMCW6tl7ZP8jLmYsqW0Z7CR1lJE1I0ycO8FA3wBBMNA3kBVOFPS2VbqGnYsZS2oHDsVK6jgja0aW\nHDYdXzdetHp29TGreWT/IwuudzFjSe3AHjtJXalUz96HX/NhelccO+/a3hXHupixpLZgj52krlWy\nZ+/Tn2bs0bvZ3Qf9MzD+rN+ycEJSWzDYSVKBkXWXM7Lub+HgQVi1CrZe3uxHkqSyGOwkqdDwMGzd\n6i4VktqOc+wkqRh3qZDUhuyxk6QlLLZLhXPvJLUSe+wkaQmldqlw0WJJrcZgJ0lLKHf/WUlqNoOd\nJC2h1OLELlosqdUY7CRpCYvuP7t9O2zalL1LUpNZPCFJS5grkBjbOsbumd309/Uzvm6ckcdfBOvW\n5a13t9WlUSQ1lcFOkspQdJeKTZuyUHf4cPa+bZvBTlJTORQrSVWaPO0Ag+85wooNMPieI0yedmDp\nD0lSHdljJ0lVmNwxyejODzLblwCY7kuM7vwg7Dg1G6J11wpJTWCwk6QqjG0dY/bQ7Lxjs4dmGbv3\n9xm57jHn3UlqCoOdJFWh1Bp200/uYfCdsLsP+mf2M771Jjjh4YWFF+5YIakOnGMnSVUotYZdANPP\nhBTZ++8d/hTnf/p8pmemSSSmZ6YZ3TJadK/ZyR2TDG4eZMV1KxjcPOh+tJIqZrCTpCoUW9suCFLB\ndYc4zMHDB+cdmz00u2A7sskdk4xuGS0rAEpSKQY7SarCyJoRJs6dYKBvgCAY6BsgLYh1pRUO5Zac\ns+d+tJIq4Bw7SapS4dp2g5sHmZ6ZLuuzhUO57kcrqRbssZOkGik2PLtyxUpW0TPvWC8rs+3I8rgf\nraRaMNhJUo0UG5699fW38okzrmJgJogEAzPBxBlXAswrlDjn1HNK70crSWWKlMqfE9IphoaG0tTU\nVLMfQ1I32b796KLFkyc8zOiW0Xlz6npX9vL2l72dex+6d96yKFBkj1qXSpG6SkR8NaU0VNa1BjtJ\naqxSc/EG+gbYddmuoz/PVcoWBsCJcycMd1IXqSTYORQrSQ1WbqGElbKSKmWwk6QGK7dQotJKWRc4\nlmSwk6QGG183Tu+KY+cd611xbEWVsoUh7l1/9y4XOJZksJOkRhtZM8LE6z/OwHHPJYCB457LxOs/\nvmDeXLHlU3pX9nLOqecsCHG3TN1S82FbewCl9mPxhCS1krzqWYaHmdwxuaAqdmzrWNkLIUNWlFFp\nVW2pwo1ilbsWckj1ZVXsEgx2klrS9u2wbh0cPAirVsHWrTA8vOCyFdetKHv7smz/2qevLTeclarc\nLXY/q3Sl+rIqVpLa0bZtWag7fDh737at6GWl5t4FseDnwgA4e2iWW6ZuWXIuXqkCjWL3s0pXah0G\nO0lqFWvXZj11PT3Z+9q1RS8rNffuoqGL5u16UapXr1g4u/Szl86bT3fS6pPKfmz3s5Vah8FOklrF\n8HA2/LpxY8lhWCi+ddnEuRN89LUfZddluziy4Qi7LtvFQN9A2V/9yP5H5vXiPXbgMVb1rJp3TWGP\n4JyTVp9UdpHFcgoyWq2Yo9WeRwLn2ElS6ysoqChXsQKIYsOzpTxr9bM4YdUJR+finXPqOdz+z7fP\nu9/KFSuJCA4ePnj02Ny8O5i/HVqxz5e7lVoln23EfD93BVEjWTyxBIOdpLZRZkFFKYVVtcUCUilB\ncGTDkUXv9/jBx3lk/yMLPvus1c9i/1P7ywqVhceLhcVyP1tpuCpWdQxL789b7rZwUi0Y7JZgsJPU\nNjZtgquvzgoqenqyYdorrljWLcsNZ+WElEoqdBulsKexVC9esV63xXog8++x2O9dzfIy9VIsuNqj\n2H4Mdksw2ElqG8vssSvHcoYVS/VctZJSQ7aVrAdYGHIrWQ7G4WItV8cEu4i4DTgPOJh3+I0ppc/l\nzj9e8JFjgW+klM5c7L4GO0ltpco5dpWotmenVHhYfczqor2AhcGnkjl/tfxs78resoaj8+X3xBUb\n0q7XcHG1SoXPcns0W0039z52WrB7PKV0SZnX/wtwZ0rpfYtdZ7CT1BEaEPjKUWqeWjk7V1RSkFHO\nZyvREz0cTofLuracnrhKei6LhSsoPrev2kBT7jB5O+wo0u29j10Z7CLiF4D7gf6U0vcXu9ZgJ6nt\nNWCIdrnKDSTVFjAU+2yp+YKlFPbcVVK4Ue7wbDkWC7PFqoGLBZrl/LNopSHkYrq9WKXhwS4iTgPe\nCvw68GLgOODbwN3A5pTSE1Xe9zbgdUAC9gD/C7gxpfRUkWv/DDg5pXTuUvc12Elqe3UoqugElSzx\nMtA3cHSu3WKhslRYK6waXu7yMsWU6lWsdimaSiw37C1n6LTws+W2wXK/t1U1I9i9H7gY+AzwAHAI\nOBt4M/AvwC+mlPbnrr0TeMsitzs7pbQtd+3PA98FfgT8PPAXZEOtVxd8//HA94HfTSl9eqnnNdhJ\nantt0GPXLOUs8VLJMF4lvUXLWV5mOUoFyMIAWGmP5lLf06g1C8vtNe3UIdtmBLsh4KGU0kzB8RuA\nMeDdKaWbc8dOJCtyKGUmpXSoxPecB1yXUjq14Ph6YBPwgmK9eYUMdpI6QovMsWsHy+09Wk5YWM4Q\naSXzAItpRK9iK61ZWOmQbbv07rXMHLuIWEPWY/dnKaWLanC/3wE2ppReUnD8K8BXUkp/VM59DHaS\nOpZhry5qGQAqWT+v2By7SlTbq7jcsFcvhWsEQnXD5lBZYF/OPNBaaKVg9xrgXuD6lNKGKj7/FuBz\nwGPAGuAu4J6U0hV515wOfAM4PaX0UDn3NdhJ6kgOz7aNSoJCuT1+y1lWpR3CXjnDrpXMp1xs15Tl\nbKVXj3DXEsEuInqALwNnAWeklHZWcY/7gDOBlcAPgDuATflDtRHxAeA/pZR+ZYl7jQKjAP39/a+Y\nnm7tBTUlqWIWVHSFUj1Nta5iLXeuYiPWLKxk2LVU0Ue1PZ+VPHe9qnQrCXbH1Pzbn7YZGAaurCbU\nASwV1nLX/EGZ95oAJiDrsavmeSSppa1dm/XUzfXYrV3b7CdSHcyFm3oPA46sGVlwz1f2v7IpaxYW\n+/12z+wu+tyJtGDIdmzrWNXD2ZX0UpZ6pkaqS49dRGwErgImUkrvrPkXLJNDsZI6lnPs1ASNWLOw\nUCWFEo3a07gVeuxqHuwi4lpgA3ArcEFqwRWQDXaSuophTx2okuKHcrdXK3f+YivPsVtR4y++lizU\n3Q5c2IqhTpK6ylxBxdVXZ+/btzf7iaSaGFkzwsS5Ewz0DRAEA30DJYPV+Lpxelf2zjvWu7KXD7/m\nw+y6bBdHNhxh12W7+PBrPlz0uouGLpr3Pbe+/lY+8bpPlPXdjVazOXYRcQ1ZqLsDOD+ldGSJj0iS\n6m3btmzO3eHD2fu2bfbaqWMUmwdY6jpYesi30vmLrRDkCtVqgeKLgZuB3cDVQGGo25NS+uKyv6hG\nHIqV1DVcAkVqe82oij0r995PNgxb6D6gZYKdJHWN4eEszDnHTuoKNQl2KaX1wPpa3EuSVGPDwwsD\nnQUVUkeq5zp2kqRW5PCs1LFqWhUrSWoDxQoqJHUEg50kdZu5HSp6etyhQuowDsVKUrexoELqWAY7\nSepGFlRIHclgJ0myoELqEM6xkyRZUCF1CIOdJMmCCqlDOBQrSbKgQuoQBjtJUsaCCqntGewkScVZ\nUCG1HefYSZKKs6BCajsGO0lScRZUSG3HoVhJUnEWVEhtx2AnSSrNggqprRjsJEnls6BCamnOsZMk\nlc+CCqmlGewkSeWzoEJqaQ7FSpLKZ0GF1NIMdpKkylhQIbUsg50kaXksqJBahnPsJEnLY0GF1DIM\ndpKk5bGgQmoZDsVKkpbHggqpZRjsJEnLV6ygAiyqkBrMYCdJqg+LKqSGc46dJKk+LKqQGs5gJ0mq\nD4sqpIZzKFaSVB8WVUgNZ7CTJNWPu1RIDWWwkyQ1jgUVUl05x06S1DgWVEh1ZbCTJDWOBRVSXTkU\nK0lqHAsqpLoy2EmSGsuCCqluDHaSpOayoEKqGefYSZKay4IKqWYMdpKk5rKgQqoZh2IlSc1lQYVU\nMwY7SVLzWVAh1YTBTpLUeiyokKriHDtJUuuxoEKqisFOktR6LKiQqtLywS4iXhsRX4uIJyLihxHx\n3rxzL4yILRHxSETsiYhNEdHyv5MkaQlzBRUbNzoMK1WgpefYRcSvAxPA7wL3Ab1Af+5cD7AF+Dzw\nRuCngb8FHgVubMbzSpJqyIIKqWItHeyAjcDGlNLW3M+PAf+a+/PpwM8BZ6WUDgDfiYibgA0Y7CSp\n81hQIS2pJsOWEXFaRFwfEQ9ExN6I2BcRD0bEWEQcX+U9jwfOAp4XEd/MDbV+JiJeOHdJwfvcnwcj\n4hnV/zaSpJZkQYW0pFrNRzsfuBz4NnA98F5gJ3ADcH9ErJ67MCLujIi0yGtt7tKfIgtqvw28Gngh\n8EPgUxERuft/GxiPiNW5wHd57rMGO0nqNBZUSEuKlNLybxIxBDyUUpopOH4DMAa8O6V0c+7YicCx\ni9xuJqV0KCL6yObLvSOl9Oe5zz4b2AsMpJR2R8RLgZuAVwA/Bj5ONgx7YkrpiVJfMDQ0lKampqr8\nbSVJTeMcO3WhiPhqSmmonGtrMscupVQqJd1FFuzOyLt2H7CvjHvORMQ0UDJ5ppS+Cbxm7ueIuBj4\np8VCnSSpjVlQIS2q3sUTp+Te91T5+VuASyPiC2Q9dRuBr6aUdgNExJnAw8CTwNlkIfLty3piSVL7\nsKBCmqdua77lliO5GngK+GSVt/kA8Fnga8D3gJOBN+SdfxMwDcwA7ycbtv1iiecZjYipiJjaE06e\nqwAAFMNJREFUu3dvlY8jSWopFlRI89Rkjl3RG0d8BLgEuDKltKkuX1Il59hJUoewx05doOFz7Io8\nwEayUDfRaqFOktRB5naocI6dBNQh2EXEtcBVwK3ARbW+vyRJ81hQIR1V02CXC3UbgNuBC1O9xnkl\nSSrF4Vl1sZoVT0TENWSh7g7g/JTSkVrdW5KksllQoS5Wkx673Ppx1wG7gS8B52WbQxy1p1S1qiRJ\nNTW3Q8Vcj507VKiL1Goo9qzcez/ZMGyh+wCDnSSp/iyoUBer1c4T64H1tbiXJEnLZkGFulS9d56Q\nJKn5LKhQl6jbzhOSJLUMCyrUJQx2kqTON1dQ0dNjQYU6mkOxkqTOZ0GFuoTBTpLUHYoVVIBFFeoo\nBjtJUveyqEIdxjl2kqTuZVGFOozBTpLUvSyqUIdxKFaS1L0sqlCHMdhJkrqbu1SogxjsJEnKZ0GF\n2phz7CRJymdBhdqYwU6SpHwWVKiNORQrSVI+CyrUxgx2kiQVsqBCbcpgJ0nSUiyoUJtwjp0kSUux\noEJtwmAnSdJSLKhQm3AoVpKkpVhQoTZhsJMkqRwWVKgNGOwkSaqGBRVqQc6xkySpGhZUqAUZ7CRJ\nqoYFFWpBDsVKklQNCyrUggx2kiRVy4IKtRiDnSRJtWJBhZrMOXaSJNWKBRVqMoOdJEm1YkGFmsyh\nWEmSasWCCjWZwU6SpFqyoEJNZLCTJKmeLKhQAznHTpKkerKgQg1ksJMkqZ4sqFADORQrSVI9WVCh\nBjLYSZJUbxZUqEEMdpIkNZoFFaoT59hJktRoFlSoTgx2kiQ1mgUVqhOHYiVJajQLKlQnBjtJkprB\nggrVgcFOkqRWYEGFasA5dpIktQILKlQDBjtJklqBBRWqgZYOdhHx/Ij464j4UUQ8EhH3RMQpeeff\nHBFfiYjHI2JXEx9VkqTlmSuo2LjRYVhVrdXn2H2U7BlfCBwGPgZ8Avj13PmfADcDzwUub8YDSpJU\nM8UKKsCiCpWt1YPdi4E/TintA4iITwIfnzuZUvpi7vjrm/N4kiTVmUUVqkBNhmIj4rSIuD4iHoiI\nvRGxLyIejIixiDh+Gbf+EPDGiHhmRJwIvA3YUotnliSpLVhUoQrUao7d+WRDod8GrgfeC+wEbgDu\nj4jVcxdGxJ0RkRZ5rc2771eAZwI/Bh4FTgeurNEzS5LU+iyqUAVqNRT7V8CmlNJM3rFbIuIhYAy4\ngGwuHMA7gEsWudcMQESsAL4EfAo4h2yO3R8A2yLi5SmlQzV6dkmSWpe7VKgCNQl2KaWpEqfuIgt2\nZ+Rduw/YV8ZtTwIGgD9NKT0OEBEfAq4lm3v3zWU8siRJ7cNdKlSmehdPzC1NsqfSD6aUfhQR/wa8\nKyI2kPXYXUpWCbsLICJ6gJW5V0TEcdlH04EaPLskSa3JggqVULd17HKh62rgKeCTVd7mdcCZwHfJ\nwuFvAL+ZUnoyd/5twH7gL4H+3J93lnie0YiYioipvXv3Vvk4kiS1AAsqVEI9e+w2A8PAlSmlomFr\nKSmlrwOvXuT8bcBtZd5rApgAGBoaStU8jyRJLWGuoGKux86CCuXUJdhFxEayAomJlNKmenyHJEld\ny4IKlVDzYBcR1wJXAbcCF9X6/pIkCQsqVFRNg10u1G0AbgcuTCk55ClJUiNYUCFqWDwREdeQhbo7\ngPNTSkdqdW9JkrQECypEjXrsIuJi4DpgN9miwudFRP4le+b2dZUkSXVgQYWo3VDsWbn3frJh2EL3\nAQY7SZLqxYIKUbudJ9YD62txL0mSVCULKrpevXeekCRJzWJBRdep284TkiSpySyo6DoGO0mSOtVc\nQUVPjwUVXcKhWEmSOpUFFV3HYCdJUiezoKKrGOwkSeomFlR0NOfYSZLUTSyo6GgGO0mSuokFFR3N\noVhJkrqJBRUdzWAnSVK3saCiYxnsJEnqdhZUdAzn2EmS1O0sqOgYBjtJkrqdBRUdw6FYSZK6nQUV\nHcNgJ0mSLKjoEAY7SZK0kAUVbck5dpIkaSELKtqSwU6SJC1kQUVbcihWkiQtZEFFWzLYSZKk4iyo\naDsGO0mSVB4LKlqec+wkSVJ5LKhoeQY7SZJUHgsqWp5DsZIkqTwWVLQ8g50kSSpfsYIKsKiiRRjs\nJEnS8lhU0TKcYydJkpbHooqWYbCTJEnLY1FFy3AoVpIkLY9FFS3DYCdJkpbPXSpagsFOkiTVngUV\nTeEcO0mSVHsWVDSFwU6SJNWeBRVN4VCsJEmqPQsqmsJgJ0mS6sOCioYz2EmSpMawoKLunGMnSZIa\nw4KKujPYSZKkxrCgou4cipUkSY1hQUXdGewkSVLjWFBRVwY7SZLUPBZU1JRz7CRJUvNYUFFTBjtJ\nktQ8FlTUVEsHu4h4YURsiYhHImJPRGyKiBV55z8aEd+JiMci4nsRsTkiVjXzmSVJUgXmCio2bnQY\ntgZaNthFRA+wBfgWcDIwBJwDvDfvspuBl6aUngG8LPe6ssGPKkmSlmN4GK64Yn6o274dNm3K3lW2\nVi6eOB34OeCslNIB4DsRcROwAbgRIKX09bzrAzgCnNroB5UkSTVkQUXVatJjFxGnRcT1EfFAROyN\niH0R8WBEjEXE8dXetuB97s+DEfGMvO/+o4h4HPh3sh67zVV+nyRJagUWVFStVkOx5wOXA98Gricb\nLt0J3ADcHxGr5y6MiDsjIi3yWpu7dGfufuMRsToiXpj7DoCjwS6l9P6U0glkvXu3AD+o0e8kSZKa\nwYKKqkVKafk3iRgCHkopzRQcvwEYA96dUro5d+xE4NhFbjeTUjqUu/alwE3AK4AfAx8nG4Y9MaX0\nRJHneBPwrpTS2Ys979DQUJqamir315MkSY3mosVHRcRXU0pD5Vxbkzl2KaVSKekusmB3Rt61+4B9\nZd73m8Br5n6OiIuBfyoW6nJWAqeVc29JktTC3KGiKvUunjgl976nmg9HxJnAw8CTwNlkIfHtuXN9\nwH8F7gFmgDXAVcDnl/fIkiSp5VhQUZa6LXeSW67kauAp4JNV3uZNwDRZcHs/8I6U0hdz5xLwVrLg\nt48s4N0LvLvE84xGxFRETO3du7fKx5EkSU1hQUVZ6tljtxkYBq5MKe2s5gYppavJwmGxc48Br6rg\nXhPABGRz7Kp5HkmS1CRzBRVzPXYWVBRVl2AXERuBS4CJlNKmenyHJEnqInM7VDjHblE1D3YRcS3Z\nXLdbgYtqfX9JktSlLKhYUk2DXS7UbQBuBy5MtVhLRZIkqRgLKhaoWfFERFxDFuruAM5PKR2p1b0l\nSZIWsKBigZr02OXWl7sO2A18CTgvIn8nMPbkVbNKkiQtnwUVC9RqKPas3Hs/2TBsofsAg50kSaod\nCyoWqNXOE+uB9bW4lyRJUtksqJin3jtPSJIkNU6XF1TUbecJSZKkhuvyggqDnSRJ6hxzBRU9PV1Z\nUOFQrCRJ6hxdXlBhsJMkSZ2lWEEFdEVRhcFOkiR1vi4pqnCOnSRJ6nxdUlRhsJMkSZ2vS4oqHIqV\nJEmdr0uKKgx2kiSpO3TBLhUGO0mS1J06sKDCOXaSJKk7dWBBhcFOkiR1pw4sqHAoVpIkdacOLKgw\n2EmSpO7VYQUVBjtJkqQ5bV5Q4Rw7SZKkOW1eUGGwkyRJmtPmBRUOxUqSJM1p84IKg50kSVK+Ni6o\nMNhJkiQtpo0KKpxjJ0mStJg2Kqgw2EmSJC2mjQoqHIqVJElaTBsVVBjsJEmSltImBRUGO0mSpEq1\naEGFc+wkSZIq1aIFFQY7SZKkSrVoQYVDsZIkSZVq0YIKg50kSVI1ihVUNJlDsZIkSR3CYCdJktQh\nDHaSJEkdwmAnSZLUIQx2kiRJHcJgJ0mS1CEMdpIkSR3CYCdJktQhDHaSJEkdwmAnSZLUIQx2kiRJ\nHcJgJ0mS1CEMdpIkSR2iqcEuIt4cEV+JiMcjYleR88dExIcj4scR8WhEfDwijss7f1tEHMx9fu71\n6ob+EpIkSS2i2T12PwFuBsZKnL8SOBtYA5wK/BzwgYJrJlJKJ+S9Ple3p5UkSWphTQ12KaUvppTu\nBKZLXHIh8L6U0vdSSnuBa4H1EdHTqGeUJElqF2UFu4g4LSKuj4gHImJvROyLiAcjYiwijq/Hg0XE\nM4EXAA/mHf4acCIwmHdsJDdU+43c8xxTj+eRJElqdeX22J0PXA58G7geeC+wE7gBuD8iVs9dGBF3\nRkRa5LW2zO88Mff+aN6xRwvO/SlwOvBs4G3AemBDmfeXJEnqKOX2bv0VsCmlNJN37JaIeIhsftwF\nZHPlAN4BXLLIvWYWOZdvX+69D/hh7s/PzD+XUvpa3vVTEbEBuA64uszvkCRJ6hhlBbuU0lSJU3eR\nBbsz8q7dx9OhrGoppUcj4jvAy8l6BwF+PnfvXaU+BsRyv1uSJKkdLbd44pTc+55qPhwRPbnlS1Zm\nP8ZxEXFs3iV/DlwRESdHxHPIiiduSykdzn3+LRHRF5kzgWuAu6v9ZSRJktpZpJSq+2BWmfpl4Czg\njJTSziU+Uuwe64FbCw5Pp5QGc+ePAT5ENn9uBdmQ8CUppf258/cBZ5IFwx8Ad5ANGR8q8l2jwGju\nx9N5uhewnp4N/KgB36PK2C6ty7ZpTbZL67JtWlOt22UgpfScci5cTrD7CNlcuitTSpuqukmHi4ip\nlNJQs59D89kurcu2aU22S+uybVpTM9ulqqHYiNhIFuomDHWSJEmtoeJgFxHXAleRDaFeVOsHkiRJ\nUnUqCna5ULcBuB24MFU7jts9Jpr9ACrKdmldtk1rsl1al23TmprWLmXPsYuIa8jWiLsDWJ9SOlLP\nB5MkSVJlygp2EXEx2QLEu8kW/y0MdXtSSl+s/eNJkiSpXOXuPHFW7r2fbBi20H2AwU6SJKmJyppj\nl1Jan1KKRV5r6/ycbSEiVkTE5RHxzYh4MiK+ExF/EhHHN/vZukFEnBYR10fEAxGxNyL2RcSDETFW\nrA0i4vSIuCcifhIRT0TElyPiV5vx7N0mInoj4uHc/tE3Fzlv2zRQRJwUEX8cEf+W+3fX3oj43xHx\nXwqus10aJCJOiIgrI2JH7t9lP4qI+yNifUREwbW2Sx1ExBURcXfev6t2LXF92e1Qz7xQbo+dynMT\n8B7gb4A/AX429/N/jIhXOS+x7s4HLgY+A0wCh4CzgRuAN0fEL+Ytbv1i4H7gKeADZHsYvwP4fES8\nJqX0pSY8fze5Hii62KZt01gRMQBsA04APg58i2yP7jOBn8m7znZpkIhYAXwW+M9ko2QfAXqB3yFb\nkeJngT/MXWu71M/7gB8DX+PpveqLqqId6pcXUkq+avAC/gPZ3MO/Ljj+brI9bM9r9jN2+gsYAvqK\nHL8h1waX5B37S+Aw8PK8YycA02S7kkSzf59OfZHt+fwU8D9y7XJzwXnbprHt8WXgO8Dzl7jOdmlc\nmwzn/m7cVHB8FfAw8Kjt0pB2eFHen/8V2LXItWW3Q73zwnL3itXTfgcIYHPB8Y8Bs8BbG/5EXSal\nNJVSmily6q7c+xkAua7u3wK2pZQezPv842T7E5/G0/NKVUO5rQg/BnwO+FSR87ZNA0XELwO/BHwg\npfSDiFgZEb1FrrNdGusZuffv5x9MKR0k26bqCbBd6i2l9HA511XRDnXNCwa72jmLLIH/Y/7BlNKT\nwIP4l6uZTsm978m9nwkcC2wvcu0DuXfbqz4uB15KtnNNMbZNY52Te98dEVuA/cATEfGtiMj/j4vt\n0lj/CDwK/EFEvCki+iPipRGxCXgFcG3uOtulNVTaDnXNCwa72jkZ+FFK6UCRc98Dnh0Rqxr8TF0v\n10N0NdnQ3ydzh0/OvX+vyEfmjv1MkXNahoh4IdlamNenlHaVuMy2aazTc+8fA04C3k42V/UgcEdE\n/F7uvO3SQCmln5D1AP2YbIhvGvgG2Rzi304pfSx3qe3SGipth7rmBYsnaqcXKNZIAE/mXXOwMY+j\nnM1k81WuTCntzB2bG2oq1l5PFlyj2rmFbH7Qhxa5xrZprBNz7/uAs3NDfUTEPWRt9b6IuB3bpRke\nJ5vX9RmySfknkQW7T0bE61K2dqzt0hoqbYe65gWDXe3MAj9d4txxedeoQSJiI9mQ30RKaVPeqbl2\nOLbIx2yrOsgN6/0a8MsppUOLXGrbNNb+3PtfzIU6yHqMIuIzwO+S9erZLg0UEWvIwtzlKaVb8o7/\nBVnY+1iuCtN2aQ2VtkNd84JDsbXzfbLu02IN+zNk3a721jVIbl/jq8iWBrio4PTchORiQxRzx4p1\nqasKub8THwLuBX4YES+JiJcAA7lL+nLHnolt02jfzb3/sMi5H+TefwrbpdEuJ/sP/N35B1NKs8Df\nkf3dGcR2aRWVtkNd84LBrnb+ieyf5y/kH4yI44CXA1PNeKhulAt1G8jWf7ow5erI8+wg6wYfLvLx\nX8y92161s5pszbrXAg/lvbblzr819/OF2DaNNjd5+5Qi5+aO/Tu2S6PNhYGeIueOyXu3XVpDpe1Q\n17xgsKudu8jWn7ms4Pg7yMbKJxv+RF0oIq4hC3V3AOenIos85krQtwBrI+JleZ89gSxcPERBtZKW\n5QngTUVe78qd/1zu58/YNg13D9n8urfm/hkDEBHPB14PfCul9G+2S8N9Pfe+Pv9grlf7dcBPANul\nRVTRDnXNC7GwM0PVioiPkM3p+huyYae5laT/AfjVYiFDtRMRFwM3A7vJKmEL/3nvyU04JjcU+I9k\nu1PcBDxG9pdqDfDalNLnG/Xc3SoiBoH/B/zPlNIlecdtmwaKiFHgz4D/C3yCbBHc/w48H/jNlNIX\nctfZLg2S2w3ka2TD4JNk/w05ieyf9yBwcUrpo7lrbZc6iYi38fSUkXeT/d34k9zP0ymlO/Kuragd\n6poXmr2ycye9yLrNf59slekDZGPqHwJOaPazdcMLuI3s/4JKvbYVXP+zwKfJ1ouaBb4CvKrZv0e3\nvMj+A7Vg5wnbpilt8Qay9baeIOvB+wLwStulqW3yYrLpJN8lCwuPAX8PvMF2aVgbbCv3vyeVtkM9\n84I9dpIkSR3COXaSJEkdwmAnSZLUIQx2kiRJHcJgJ0mS1CEMdpIkSR3CYCdJktQhDHaSJEkdwmAn\nSZLUIQx2kiRJHcJgJ0mS1CH+P/joemlAiGYsAAAAAElFTkSuQmCC\n",
"text/plain": [
"![\"Halide\"](\"images/Halide1.gif\")
\n",
" \n",
" b. ![\"Halide\"](\"images/Halide2.gif\")
\n",
" \n",
" c. ![\"Halide\"](\"images/Halide3.gif\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"6\\. Prove that if $A = Q B Q^T$ for some orthnogonal matrix $Q$, the $A$ and $B$ have the same singular values."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"7\\. What is the *stochastic* part of *stochastic gradient descent*?"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda root]",
"language": "python",
"name": "conda-root-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/Homework 2.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Homework 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This homework covers the material in Lessons 3 & 4. It is due **Thurday, June 15**. Please submit your **answers as a PDF**."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1\\. Modify the LU method (without pivoting) to work **in-place**. That is, it should not allocate any new memory for L or U, but instead overwrite A."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def LU(A):\n",
" U = np.copy(A)\n",
" m, n = A.shape\n",
" L = np.eye(n)\n",
" for k in range(n-1):\n",
" for j in range(k+1,n):\n",
" L[j,k] = U[j,k]/U[k,k]\n",
" U[j,k:n] -= L[j,k] * U[k,k:n]\n",
" return L, U"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"2\\. Modify our LU method from class to add pivoting, as described in the lesson. *Hint: the swap method below will be useful* "
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def swap(a,b):\n",
" temp = np.copy(a)\n",
" a[:] = b\n",
" b[:] = temp"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"3\\. For each of the following sets of dimensions, either\n",
" - give the dimensions of the output of an operation on A and B, **or** \n",
" - answer *incompatible* if the dimensions are incompatible according to the [numpy rules of broadcasting](https://docs.scipy.org/doc/numpy-1.10.0/user/basics.broadcasting.html)."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
" a. A (2d array): 3 x 3\n",
" B (1d array): 1\n",
"\n",
" b. A (2d array): 2 x 1\n",
" B (3d array): 6 x 4 x 2\n",
"\n",
" c. A (2d array): 5 x 4\n",
" B (1d array): 4\n",
"\n",
" d. A (3d array): 32 x 64 x 8\n",
" B (3d array): 32 x 1 x 8\n",
"\n",
" e. A (3d array): 64 x 1\n",
" B (3d array): 32 x 1 x 16\n",
"\n",
" f. A (3d array): 32 x 64 x 2\n",
" B (3d array): 32 x 1 x 8"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"4\\. Write how this matrix would be stored in compressed row format:\n",
"\n",
"\\begin{pmatrix}\n",
" 1 & & & & -2 & -3 \\\\\n",
" & 3 & & & & -9 \\\\\n",
" & & & -7 & 4 & \\\\ \n",
" -1 & 2 & & 7 & & \\\\\n",
" -3 & & & 26 & &\n",
" \\end{pmatrix}"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda root]",
"language": "python",
"name": "conda-root-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/Homework 3.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Homework 3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"1\\. Add shifts to the QR algorithm\n",
"\n",
"Instead of factoring $A_k$ as $Q_k R_k$ (the way the pure QR algorithm without shifts does), the shifted QR algorithms:\n",
"\n",
"i. Get the QR factorization $$A_k - s_k I = Q_k R_k$$\n",
"ii. Set $$A_{k+1} = R_k Q_k + s_k I$$\n",
"\n",
"Choose $s_k = A_k(m,m)$, an approximation of an eigenvalue of A. \n",
"\n",
"The idea of adding shifts to speed up convergence shows up in many algorithms in numerical linear algebra (including the power method, inverse iteration, and Rayleigh quotient iteration). "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def practical_qr(A, iters=10):\n",
" # Fill method in\n",
" return Ak, Q"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python [conda root]",
"language": "python",
"name": "conda-root-py"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
================================================
FILE: nbs/Project_ideas.txt
================================================
Suggestions for Writing Topics:
- Algorithms to research and implement:
-- Cholesky Factorization
-- Gauss-Legendre Quadrature
-- Conjugate Gradient
-- Rayleigh Quotient Iteration
-- Power Method with shifts and deflation
-- QR algorithm with Wilkinson shifts
-- Golub-Kahan bidiagonalization with Householder reflectors
-- Divide-and-conquer eigenvalue algorithm (Cuppen)
-- GMRES (generalized minimal residuals)
-- Sparse NMF
-- Monte Carlo Methods
-- Feel free to ask me about additional ideas
- Speed up an algorithm of your choice using one or more of...:
-- The GPU with PyTorch
-- Numba or Cython
-- Parallelization
-- Randomized projections
- Find an interesting paper (related to linear algebra). Summarize it and implement it. Some interesting ideas in:
-- https://arxiv.org/abs/1608.04481
- Look at the Matrix Factorization Jungle for ideas: https://sites.google.com/site/igorcarron2/matrixfactorizations
================================================
FILE: nbs/convolution-intro.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Intro to Convolutions"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set up"
]
},
{
"cell_type": "code",
"execution_count": 84,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import math,sys,os,numpy as np\n",
"from numpy.linalg import norm\n",
"from PIL import Image\n",
"from matplotlib import pyplot as plt, rcParams, rc\n",
"from scipy.ndimage import imread\n",
"from skimage.measure import block_reduce\n",
"import pickle as pickle\n",
"from scipy.ndimage.filters import correlate, convolve\n",
"rc('animation', html='html5')\n",
"rcParams['figure.figsize'] = 3, 6\n",
"%precision 4\n",
"np.set_printoptions(precision=4, linewidth=100)"
]
},
{
"cell_type": "code",
"execution_count": 85,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def plots(ims, interp=False, titles=None):\n",
" ims=np.array(ims)\n",
" mn,mx=ims.min(),ims.max()\n",
" f = plt.figure(figsize=(12,24))\n",
" for i in range(len(ims)):\n",
" sp=f.add_subplot(1, len(ims), i+1)\n",
" if not titles is None: sp.set_title(titles[i], fontsize=18)\n",
" plt.imshow(ims[i], interpolation=None if interp else 'none', vmin=mn,vmax=mx)\n",
"\n",
"def plot(im, interp=False):\n",
" f = plt.figure(figsize=(3,6), frameon=True)\n",
" # plt.show(im)\n",
" plt.imshow(im, interpolation=None if interp else 'none')\n",
"\n",
"plt.gray()\n",
"plt.close()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## MNIST Data"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.datasets import fetch_mldata\n",
"mnist = fetch_mldata('MNIST original')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys(['DESCR', 'COL_NAMES', 'target', 'data'])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mnist.keys()"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((70000, 784), (70000,))"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mnist['data'].shape, mnist['target'].shape"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"((70000, 28, 28), (70000,))"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"images = np.reshape(mnist['data'], (70000, 28, 28))\n",
"labels = mnist['target'].astype(int)\n",
"n=len(images)\n",
"images.shape, labels.shape"
]
},
{
"cell_type": "code",
"execution_count": 53,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"images = images/255"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAM0AAADKCAYAAAAGucTRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAC7pJREFUeJzt3W2MVOUZxvHrhhajpRJfUiSuVJvgB2MEfCF8MJVq2lg0\nWYwRIcaln+BDMdQYIzYopLHREKBFW4mUEiFS1hdUVhNLqBJtE2NckVjUthLjC7guokaWmEiEux/m\nrF23z2Hnnvc5+/8lZGcvh5lnwIsz8+w5z2PuLgDlG9PsAQDthtIAQZQGCKI0QBClAYIoDRBEaYAg\nSgMEURog6DvV/GYzu1rSWkljJW1w9/tGuD+nH6BlubuVcz+r9DQaMxsr6T+Sfippv6RXJc1397dO\n8HsoDVpWuaWp5u3ZDEn73P1ddz8qqVtSZxWPB7SFakpztqQPh3y/P8u+xcwWmlmvmfVW8VxAy6jq\nM0053H29pPUSb89QDNUcaQ5IOmfI9x1ZBhRaNaV5VdIUMzvPzMZJmieppzbDAlpXxW/P3P1rM1ss\naYdKU84b3f3Nmo0MaFEVTzlX9GR8pkELa8SUMzAqURogiNIAQZQGCKI0QBClAYIoDRBEaYAgSgME\nURogiNIAQZQGCKI0QBClAYIoDRBU9zUC0HiXXHJJMl+8eHEy7+rqSuabN29O5g888EAy3717dxmj\na38caYAgSgMEURogiNIAQZQGCKpqNRoze0/SgKRjkr5290tHuD+r0dTQtGnTkvkLL7yQzE899dSa\nPO8XX3yRzM8444yaPH6zlLsaTS2mnH/i7odq8DhAW+DtGRBUbWlc0t/M7DUzW5i6A7sGoGiqfXt2\nubsfMLMfSNppZv9y95eG3oFdA1A0VR1p3P1A9vWgpKdU2ugJKLSKjzRm9j1JY9x9ILv9M0m/qdnI\n8I0ZM9L/Fm3bti2ZT5gwIZnnzZQODAwk86NHjybzvFmymTNnJvO8c9LyHr/VVfP2bKKkp8xs8HH+\n4u5/rcmogBZWzVYb70qaWsOxAG2BKWcgiNIAQZQGCGIntCY45ZRTkvnFF1+czB955JFk3tHRkcyz\nyZn/k/d3nTe7tXLlymTe3d0det5ly5Yl83vvvTeZNws7oQF1QmmAIEoDBFEaIIjSAEGse9YEDz30\nUDKfP39+g0dSkjdrN378+GT+4osvJvNZs2Yl84suuqiicbUqjjRAEKUBgigNEERpgCBKAwQxe1ZH\neav3X3PNNck879ytPHmzWM8880wyX7VqVTL/6KOPkvnrr7+ezD///PNkfuWVVybz6OtqdRxpgCBK\nAwRRGiCI0gBBI5bGzDaa2UEz2zskO93MdprZO9nX0+o7TKB1jHjlppn9WNIRSZvd/cIsWynpM3e/\nz8yWSjrN3e8Y8ckKeuVmvVfvf+6555J53rlqV1xxRTLPOwdsw4YNyfyTTz4pY3T/c+zYsWT+5Zdf\nJvO8cTZr786aXbmZLTP72bC4U9Km7PYmSXNCowPaWKWfaSa6e192+2OVFg4ERoWqf7jp7n6it13Z\nbgLJHQWAdlTpkabfzCZJUvb1YN4d3X29u1860i5pQLuotDQ9khZktxdI2l6b4QCtr5zZs62SZkk6\nU1K/pOWSnpb0mKTJkt6XNNfdh08WpB6rrWfPzj///GS+fPnyZD5v3rxkfuhQerfFvr6+ZH7PPfck\n8yeeeCKZN0ve7Fne/2OPPvpoMr/ppptqNqaImu256e551+BeFRoRUBCcEQAEURogiNIAQZQGCOLK\nzYSTTjopmedd+Th79uxknreXZVdXVzLv7U3vGn/yyScn83Y3efLkZg+hIhxpgCBKAwRRGiCI0gBB\nlAYIYvYsYfr06ck8b5YsT2dnZzLPW68M7YEjDRBEaYAgSgMEURogiNIAQcyeJaxZsyaZ561+nzcb\nNtpmycaMSf8bfPz48QaPpL440gBBlAYIojRAEKUBgirdNWCFmR0wsz3Zr9j5JUAbK2f27GFJf5C0\neVj+O3dPX8rYJq699tpknrcLQN76XT09PTUbUzvLmyXL+3Pbs2dPPYdTN5XuGgCMWtV8prnFzN7I\n3r6xqRNGjUpLs07SjyRNk9QnaXXeHc1soZn1mll61QigzVRUGnfvd/dj7n5c0p8kzTjBfdk1AIVS\nUWkGt9nIXCdpb959gaIZcfZs6K4BZrZfpV0DZpnZNEku6T1Ji+o4xrrJW09s3LhxyfzgwfQ2PHmr\n37e7vPXfVqxYEXqcvL1H77zzzuiQWkKluwb8uQ5jAdoCZwQAQZQGCKI0QBClAYK4cjPgq6++SuZ5\ne2W2i7xZsmXLliXz22+/PZnv378/ma9enf7Z95EjR8oYXevhSAMEURogiNIAQZQGCKI0QBCzZwHt\nfoVm3hWpebNhN954YzLfvn17Mr/++usrG1ib4UgDBFEaIIjSAEGUBgiiNEDQqJ49y9sFIC+fM2dO\nMl+yZEnNxlQLt956azK/6667kvmECROS+ZYtW5J5V1dXZQMrCI40QBClAYIoDRBEaYCgcnYNOMfM\ndpnZW2b2ppktyfLTzWynmb2TfWVpWowK5cyefS3pNnffbWbfl/Same2U9AtJz7v7fWa2VNJSSXfU\nb6i1l7eafV5+1llnJfP7778/mW/cuDGZf/rpp8l85syZyfzmm29O5lOnTk3mHR0dyfyDDz5I5jt2\n7EjmDz74YDIf7crZNaDP3XdntwckvS3pbEmdkjZld9skKT0fCxRM6DONmZ0rabqkVyRNdPfBi+M/\nljSxpiMDWlTZP9w0s/GStkn6lbsfHvoDQHd3M0u+pzGzhZIWVjtQoFWUdaQxs++qVJgt7v5kFvcP\nLoSefU0udMyuASiacmbPTKW1m9929zVD/lOPpAXZ7QWS0lcmAQVjeTNF39zB7HJJf5f0T0mDmyr+\nWqXPNY9JmizpfUlz3f2E2wzmvYVrlhtuuCGZb926tSaP39/fn8wPHz6czKdMmVKT53355ZeT+a5d\nu5L53XffXZPnbXfunj7pcJhydg34h6S8B7sqMiigCDgjAAiiNEAQpQGCKA0QNOLsWU2frMVmz/LO\n0Xr88ceT+WWXXRZ6/LwrQKN/5nnnqnV3dyfzVruStF2UO3vGkQYIojRAEKUBgigNEERpgKBRPXuW\nZ9KkScl80aJFyTxvb8ro7NnatWuT+bp165L5vn37kjkqw+wZUCeUBgiiNEAQpQGCKA0QxOwZkGH2\nDKgTSgMEURogiNIAQdXsGrDCzA6Y2Z7s1+z6DxdovnLWPZskadLQXQNUWux8rqQj7r6q7Cdj9gwt\nrJbrnvVJ6stuD5jZ4K4BwKhUza4BknSLmb1hZhvZ1AmjRdmlGb5rgKR1kn4kaZpKR6LVOb9voZn1\nmllvDcYLNF1ZZwRkuwY8K2nHsEXQB//7uZKedfcLR3gcPtOgZdXsjIC8XQMGt9nIXCdpb3SQQDuq\nZteA+Sq9NXNJ70laNGRntLzH4kiDllXukYYTNoEMJ2wCdUJpgCBKAwRRGiCI0gBBlAYIojRAEKUB\ngigNEERpgKARL0KrsUOS3s9un5l9P1rwelvbD8u9Y0PPPfvWE5v1uvulTXnyJuD1Fgdvz4AgSgME\nNbM065v43M3A6y2Ipn2mAdoVb8+AoIaXxsyuNrN/m9k+M1va6Oevt2w5q4NmtndIdrqZ7TSzd7Kv\nhVnu6gQrsBb2NTe0NGY2VtIfJf1c0gWS5pvZBY0cQwM8LOnqYdlSSc+7+xRJz2ffF8XXkm5z9wsk\nzZT0y+zvtLCvudFHmhmS9rn7u+5+VFK3pM4Gj6Gu3P0lSZ8Nizslbcpub1JpWd9CcPc+d9+d3R6Q\nNLgCa2Ffc6NLc7akD4d8v1+jY4nbiUNW6vlY0sRmDqZehq3AWtjXzERAg3lpurJwU5aJFVi/UbTX\n3OjSHJB0zpDvO7Ks6PoHF1fMvh5s8nhqKluBdZukLe7+ZBYX9jU3ujSvSppiZueZ2ThJ8yT1NHgM\nzdAjaUF2e4Gk7U0cS03lrcCqIr/mRv9wM9v86feSxkra6O6/begA6szMtkqapdJZvv2Slkt6WtJj\nkiardJb3XHcfPlnQlk6wAusrKupr5owAIIaJACCI0gBBlAYIojRAEKUBgigNEERpgCBKAwT9F2fv\ngiVI6GTUAAAAAElFTkSuQmCC\n",
"text/plain": [
""
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import math,sys,os,numpy as np\n",
"from numpy.random import random\n",
"from matplotlib import pyplot as plt, rcParams, animation, rc\n",
"from __future__ import print_function, division\n",
"from ipywidgets import interact, interactive, fixed\n",
"from ipywidgets.widgets import *\n",
"rc('animation', html='html5')\n",
"rcParams['figure.figsize'] = 3, 3\n",
"%precision 4\n",
"np.set_printoptions(precision=4, linewidth=100)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def lin(a,b,x): return a*x+b"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"a=3.\n",
"b=8."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"n=30\n",
"x = random(n)\n",
"y = lin(a,b,x)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 0.3046, 0.918 , 0.7925, 0.8476, 0.2508, 0.3504, 0.8326, 0.6875, 0.4449, 0.4687,\n",
" 0.5901, 0.2757, 0.6629, 0.169 , 0.8677, 0.6612, 0.112 , 0.1669, 0.6226, 0.6174,\n",
" 0.3871, 0.4724, 0.3242, 0.7871, 0.0157, 0.8589, 0.7008, 0.2942, 0.3166, 0.5847])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"x"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([ 8.9138, 10.7541, 10.3775, 10.5428, 8.7525, 9.0511, 10.4977, 10.0626, 9.3347,\n",
" 9.4062, 9.7704, 8.827 , 9.9888, 8.507 , 10.603 , 9.9836, 8.336 , 8.5006,\n",
" 9.8678, 9.8523, 9.1614, 9.4172, 8.9725, 10.3614, 8.0471, 10.5766, 10.1025,\n",
" 8.8827, 8.9497, 9.7542])"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"