[ { "path": "LICENSE", "content": "BSD 2-Clause License\n\nCopyright (c) 2017, thom lake\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice, this\n list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE\nFOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL\nDAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR\nSERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER\nCAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,\nOR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE\nOF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "README.md", "content": "```python\ndef attend(query, context, value=None, score='dot', normalize='softmax',\n context_sizes=None, context_mask=None, return_weight=False\n ):\n \"\"\"Attend to value (or context) by scoring each query and context.\n\n Args\n ----\n query: Variable of size (B, M, D1)\n Batch of M query vectors.\n context: Variable of size (B, N, D2)\n Batch of N context vectors.\n value: Variable of size (B, N, P), default=None\n If given, the output vectors will be weighted\n combinations of the value vectors.\n Otherwise, the context vectors will be used.\n score: str or callable, default='dot'\n If score == 'dot', scores are computed\n as the dot product between context and\n query vectors. This Requires D1 == D2.\n Otherwise, score should be a callable:\n query context score\n (B,M,D1) (B,N,D2) -> (B,M,N)\n normalize: str, default='softmax'\n One of 'softmax', 'sigmoid', or 'identity'.\n Name of function used to map scores to weights.\n context_mask: Tensor of (B, M, N), default=None\n A Tensor used to mask context. Masked\n and unmasked entries should be filled \n appropriately for the normalization function.\n context_sizes: list[int], default=None,\n List giving the size of context for each item\n in the batch and used to compute a context_mask.\n If context_mask or context_sizes are not given,\n context is assumed to have fixed size.\n return_weight: bool, default=False\n If True, return the attention weight Tensor.\n\n Returns\n -------\n output: Variable of size (B, M, P)\n If return_weight is False.\n weight, output: Variable of size (B, M, N), Variable of size (B, M, P)\n If return_weight is True.\n \"\"\"\n```\n\nInstall\n-------\n```bash\npython setup.py install\n```\n\nTest\n----\n```bash\npython -m pytest\n```\nTested with pytorch 1.0.0\n\nAbout\n-----\nAttention is used to focus processing on a particular region of input.\nThe `attend` function provided by this package implements the most\ncommon attention mechanism [[1](#1), [2](#2), [3](#3), [4](#4)], which produces\nan output by taking a weighted combination of value vectors with weights\nfrom a scoring function operating over pairs of query and context vectors.\n\nGiven query vector `q`, context vectors `c_1,...,c_n`, and value vectors\n`v_1,...,v_n` the attention score of `q` with `c_i` is given by\n\n```\n s_i = f(q, c_i)\n```\n\nFrequently `f` takes the form of a dot product between query and context vectors.\n\n```\n s_i = q^T c_i\n```\n\nThe scores are passed through a normalization functions `g` (normally the softmax function).\n\n```\n w_i = g(s_1,...,s_n)_i\n```\n\nFinally, the output is computed as a weighted sum of the value vectors.\n\n```\n z = \\sum_{i=1}^n w_i * v_i\n```\n\nIn many applications [[1](#1), [4](#4), [5](#5)] attention is applied\nto the context vectors themselves, `v_i = c_i`.\n\nSizes\n-----\nThis `attend` function provided by this package accepts\nbatches of size `B` containing\n`M` query vectors of dimension `D1`, \n`N` context vectors of dimension `D2`, \nand optionally `N` value vectors of dimension `P`.\n\nVariable Length\n---------------\nIf the number of context vectors varies within a batch, a context\ncan be ignored by forcing the corresponding weight to be zero.\n\nIn the case of the softmax, this can be achieved by adding negative\ninfinity to the corresponding score before normalization.\nSimilarly, for elementwise normalization functions the weights can\nbe multiplied by an appropriate {0,1} mask after normalization.\n\nTo facilitate the above behavior, a context mask, with entries\nin `{-inf, 0}` or `{0, 1}` depending on the normalization function,\ncan be passed to this function. The masks should have size `(B, M, N)`.\n\nAlternatively, a list can be passed giving the size of the context for\neach item in the batch. Appropriate masks will be created from these lists.\n\nNote that the size of output does not depend on the number of context vectors.\nBecause of this, context positions are truly unaccounted for in the output.\n\nReferences\n----------\n#### [[1]](https://arxiv.org/abs/1409.0473)\n\n @article{bahdanau2014neural,\n title={Neural machine translation by jointly learning to align and translate},\n author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},\n journal={arXiv preprint arXiv:1409.0473},\n year={2014}\n }\n\n#### [[2]](https://arxiv.org/abs/1410.5401)\n @article{graves2014neural,\n title={Neural turing machines},\n author={Graves, Alex and Wayne, Greg and Danihelka, Ivo},\n journal={arXiv preprint arXiv:1410.5401},\n year={2014}\n }\n\n#### [[3]](https://arxiv.org/abs/1503.08895)\n\n @inproceedings{sukhbaatar2015end,\n title={End-to-end memory networks},\n author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others},\n booktitle={Advances in neural information processing systems},\n pages={2440--2448},\n year={2015}\n }\n\n#### [[4]](https://distill.pub/2016/augmented-rnns/)\n\n @article{olah2016attention,\n title={Attention and augmented recurrent neural networks},\n author={Olah, Chris and Carter, Shan},\n journal={Distill},\n volume={1},\n number={9},\n pages={e1},\n year={2016}\n }\n\n#### [[5]](https://arxiv.org/abs/1506.03134)\n\n @inproceedings{vinyals2015pointer,\n title={Pointer networks},\n author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep},\n booktitle={Advances in Neural Information Processing Systems},\n pages={2692--2700},\n year={2015}\n }\n" }, { "path": "attention/__init__.py", "content": "from . attention import attend\n" }, { "path": "attention/attention.py", "content": "from torch import FloatTensor\nfrom torch.autograd import Variable\nfrom torch.nn.functional import sigmoid, softmax\n\n\ndef mask3d(value, sizes):\n \"\"\"Mask entries in value with 0 based on sizes.\n\n Args\n ----\n value: Tensor of size (B, N, D)\n Tensor to be masked. \n sizes: list of int\n List giving the number of valid values for each item\n in the batch. Positions beyond each size will be masked.\n\n Returns\n -------\n value:\n Masked value.\n \"\"\"\n v_mask = 0\n v_unmask = 1\n mask = value.data.new(value.size()).fill_(v_unmask)\n n = mask.size(1)\n for i, size in enumerate(sizes):\n if size < n:\n mask[i,size:,:] = v_mask\n return Variable(mask) * value\n\n\ndef fill_context_mask(mask, sizes, v_mask, v_unmask):\n \"\"\"Fill attention mask inplace for a variable length context.\n\n Args\n ----\n mask: Tensor of size (B, N, D)\n Tensor to fill with mask values. \n sizes: list[int]\n List giving the size of the context for each item in\n the batch. Positions beyond each size will be masked.\n v_mask: float\n Value to use for masked positions.\n v_unmask: float\n Value to use for unmasked positions.\n\n Returns\n -------\n mask:\n Filled with values in {v_mask, v_unmask}\n \"\"\"\n mask.fill_(v_unmask)\n n_context = mask.size(2)\n for i, size in enumerate(sizes):\n if size < n_context:\n mask[i,:,size:] = v_mask\n return mask\n\n\ndef dot(a, b):\n \"\"\"Compute the dot product between pairs of vectors in 3D Variables.\n \n Args\n ----\n a: Variable of size (B, M, D)\n b: Variable of size (B, N, D)\n \n Returns\n -------\n c: Variable of size (B, M, N)\n c[i,j,k] = dot(a[i,j], b[i,k])\n \"\"\"\n return a.bmm(b.transpose(1, 2))\n\n\ndef attend(query, context, value=None, score='dot', normalize='softmax',\n context_sizes=None, context_mask=None, return_weight=False\n ):\n \"\"\"Attend to value (or context) by scoring each query and context.\n\n Args\n ----\n query: Variable of size (B, M, D1)\n Batch of M query vectors.\n context: Variable of size (B, N, D2)\n Batch of N context vectors.\n value: Variable of size (B, N, P), default=None\n If given, the output vectors will be weighted\n combinations of the value vectors.\n Otherwise, the context vectors will be used.\n score: str or callable, default='dot'\n If score == 'dot', scores are computed\n as the dot product between context and\n query vectors. This Requires D1 == D2.\n Otherwise, score should be a callable:\n query context score\n (B,M,D1) (B,N,D2) -> (B,M,N)\n normalize: str, default='softmax'\n One of 'softmax', 'sigmoid', or 'identity'.\n Name of function used to map scores to weights.\n context_mask: Tensor of (B, M, N), default=None\n A Tensor used to mask context. Masked\n and unmasked entries should be filled \n appropriately for the normalization function.\n context_sizes: list[int], default=None,\n List giving the size of context for each item\n in the batch and used to compute a context_mask.\n If context_mask or context_sizes are not given,\n context is assumed to have fixed size.\n return_weight: bool, default=False\n If True, return the attention weight Tensor.\n\n Returns\n -------\n output: Variable of size (B, M, P)\n If return_weight is False.\n weight, output: Variable of size (B, M, N), Variable of size (B, M, P)\n If return_weight is True.\n \n \n About\n -----\n Attention is used to focus processing on a particular region of input.\n This function implements the most common attention mechanism [1, 2, 3],\n which produces an output by taking a weighted combination of value vectors\n with weights from by a scoring function operating over pairs of query and\n context vectors.\n\n Given query vector `q`, context vectors `c_1,...,c_n`, and value vectors\n `v_1,...,v_n` the attention score of `q` with `c_i` is given by\n\n s_i = f(q, c_i)\n\n Frequently, `f` is given by the dot product between query and context vectors.\n\n s_i = q^T c_i\n\n The scores are passed through a normalization functions g.\n This is normally the softmax function.\n\n w_i = g(s_1,...,s_n)_i\n\n Finally, the output is computed as a weighted\n combination of the values with the normalized scores.\n\n z = sum_{i=1}^n w_i * v_i\n\n In many applications [4, 5] the context and value vectors are the same, `v_i = c_i`.\n\n Sizes\n -----\n This function accepts batches of size `B` containing\n `M` query vectors of dimension `D1`,\n `N` context vectors of dimension `D2`, \n and optionally `N` value vectors of dimension `P`.\n\n Variable Length Contexts\n ------------------------ \n If the number of context vectors varies within a batch, a context\n can be ignored by forcing the corresponding weight to be zero.\n\n In the case of the softmax, this can be achieved by adding negative\n infinity to the corresponding score before normalization.\n Similarly, for elementwise normalization functions the weights can\n be multiplied by an appropriate {0,1} mask after normalization.\n\n To facilitate the above behavior, a context mask, with entries\n in `{-inf, 0}` or `{0, 1}` depending on the normalization function,\n can be passed to this function. The masks should have size `(B, M, N)`.\n\n Alternatively, a list can be passed giving the size of the context for\n each item in the batch. Appropriate masks will be created from these lists.\n\n Note that the size of output does not depend on the number of context vectors.\n Because of this, context positions are truly unaccounted for in the output.\n\n References\n ----------\n [1](https://arxiv.org/abs/1410.5401)\n @article{graves2014neural,\n title={Neural turing machines},\n author={Graves, Alex and Wayne, Greg and Danihelka, Ivo},\n journal={arXiv preprint arXiv:1410.5401},\n year={2014}\n }\n\n [2](https://arxiv.org/abs/1503.08895)\n\n @inproceedings{sukhbaatar2015end,\n title={End-to-end memory networks},\n author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others},\n booktitle={Advances in neural information processing systems},\n pages={2440--2448},\n year={2015}\n }\n\n [3](https://distill.pub/2016/augmented-rnns/)\n\n @article{olah2016attention,\n title={Attention and augmented recurrent neural networks},\n author={Olah, Chris and Carter, Shan},\n journal={Distill},\n volume={1},\n number={9},\n pages={e1},\n year={2016}\n }\n\n [4](https://arxiv.org/abs/1409.0473)\n\n @article{bahdanau2014neural,\n title={Neural machine translation by jointly learning to align and translate},\n author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},\n journal={arXiv preprint arXiv:1409.0473},\n year={2014}\n }\n\n [5](https://arxiv.org/abs/1506.03134)\n\n @inproceedings{vinyals2015pointer,\n title={Pointer networks},\n author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep},\n booktitle={Advances in Neural Information Processing Systems},\n pages={2692--2700},\n year={2015}\n }\n \"\"\"\n q, c, v = query, context, value\n if v is None:\n v = c\n\n batch_size_q, n_q, dim_q = q.size()\n batch_size_c, n_c, dim_c = c.size()\n batch_size_v, n_v, dim_v = v.size()\n\n if not (batch_size_q == batch_size_c == batch_size_v):\n msg = 'batch size mismatch (query: {}, context: {}, value: {})'\n raise ValueError(msg.format(q.size(), c.size(), v.size()))\n\n batch_size = batch_size_q\n\n # Compute scores\n if score == 'dot':\n s = dot(q, c)\n elif callable(score):\n s = score(q, c)\n else:\n raise ValueError(f'unknown score function: {score}')\n\n # Normalize scores and mask contexts\n if normalize == 'softmax':\n if context_mask is not None:\n s = context_mask + s\n\n elif context_sizes is not None:\n context_mask = s.data.new(batch_size, n_q, n_c)\n context_mask = fill_context_mask(context_mask,\n sizes=context_sizes,\n v_mask=float('-inf'),\n v_unmask=0\n )\n s = context_mask + s\n\n s_flat = s.view(batch_size * n_q, n_c)\n w_flat = softmax(s_flat, dim=1)\n w = w_flat.view(batch_size, n_q, n_c)\n\n elif normalize == 'sigmoid' or normalize == 'identity':\n w = sigmoid(s) if normalize == 'sigmoid' else s\n if context_mask is not None:\n w = context_mask * w\n elif context_sizes is not None:\n context_mask = s.data.new(batch_size, n_q, n_c)\n context_mask = fill_context_mask(context_mask,\n sizes=context_sizes,\n v_mask=0,\n v_unmask=1\n )\n w = context_mask * w\n\n else:\n raise ValueError(f'unknown normalize function: {normalize}')\n\n # Combine\n z = w.bmm(v)\n if return_weight:\n return w, z\n return z\n" }, { "path": "examples/Pointer-Network-Argmin-Argmax.ipynb", "content": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Pointer Network Attention Demo\\n\",\n \"\\n\",\n \"The below code trains a [pointer network](https://arxiv.org/abs/1506.03134) like architecture that takes as input a sequence of vectors and outputs the vector with the minimum or maximum value along a particular coordinate. In other words, a neural network version of argmax/argmin.\\n\",\n \"\\n\",\n \"### Setup\\n\",\n \"\\n\",\n \"Let $\\\\{\\\\mathbf{c}_i\\\\}_{i=1}^n = (\\\\mathbf{c}_1, \\\\ldots, \\\\mathbf{c}_n)$ be a sequence of $n$ vectors with each $\\\\mathbf{c}_i \\\\in \\\\mathbb{R}^d$. The minimum and maximum target positions, $i_\\\\min$ and $i_\\\\max$, for the sequence are given by\\n\",\n \"\\n\",\n \"$$\\n\",\n \"\\\\begin{align*}\\n\",\n \" i_\\\\min &= \\\\text{argmin}_i \\\\left\\\\{ x_{i, k_\\\\min}\\\\right\\\\} \\\\\\\\\\n\",\n \" i_\\\\max &= \\\\text{argmax}_i \\\\left\\\\{ x_{i, k_\\\\max}\\\\right\\\\} \\\\\\\\\\n\",\n \"\\\\end{align*}\\n\",\n \"$$\\n\",\n \"\\n\",\n \"where $1 \\\\leq k_\\\\min \\\\neq k_\\\\max \\\\leq d$ are a priori chosen coordiantes along which to compute the minimum or maximum.\\n\",\n \"\\n\",\n \"### Model\\n\",\n \"\\n\",\n \"The model has the following form\\n\",\n \"\\n\",\n \"$$\\n\",\n \"\\\\begin{align*}\\n\",\n \" \\\\mathbf{u}_i &= A \\\\mathbf{c}_i & \\\\; i = 1, \\\\ldots, n \\\\\\\\\\n\",\n \" \\\\mathbf{v} &= B \\\\mathbf{q} \\\\\\\\\\n\",\n \" \\\\mathbf{p} &= \\\\text{softmax}_i(\\\\mathbf{v}^T \\\\mathbf{u}_i) \\\\\\\\\\n\",\n \" \\\\mathbf{z} &= \\\\sum_i p_i \\\\mathbf{c}_i\\n\",\n \"\\\\end{align*}\\n\",\n \"$$\\n\",\n \"\\n\",\n \"where $A, B \\\\in \\\\mathbb{R}^{p \\\\times d}$ and $\\\\mathbf{q} \\\\in \\\\{\\\\mathbf{q}_\\\\min, \\\\mathbf{q}_\\\\max\\\\} \\\\subseteq \\\\mathbb{R}^d$ is a query vector indicating whether the model should output the minimum or maximum.\\n\",\n \"\\n\",\n \"The loss is defined as the mean squared error between the output and target vector.\\n\",\n \"\\n\",\n \"$$\\n\",\n \"\\\\begin{align*}\\n\",\n \" l &= \\\\frac{1}{n}\\\\sum_j (z_j - c_{t,j})^2\\n\",\n \"\\\\end{align*}\\n\",\n \"$$\\n\",\n \"\\n\",\n \"where $t$ is either $i_\\\\min$ or $i_\\\\max$.\\n\",\n \"\\n\",\n \"### Details\\n\",\n \"The vectors $\\\\mathbf{q}_\\\\min$ and $\\\\mathbf{q}_\\\\max$ are initialized to random values sampled from $N(\\\\mathbb{0}, \\\\mathbb{I}_d)$ and held constant throughout training. The code below uses 10 dimensional context and query vectors and 7 dimensional hidden representations. The model is optimized over 1,600 training instances using RMSProp with mini batches of size 8. Each training instance contains betwen 5 and 14 context vectors. To better assess generalization validation instances are longer, containing between 15 and 24 context vectors.\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"Seed: 1273\\n\"\n ]\n }\n ],\n \"source\": [\n \"%matplotlib inline\\n\",\n \"import matplotlib.pyplot as plt\\n\",\n \"import numpy as np\\n\",\n \"import seaborn as sns\\n\",\n \"\\n\",\n \"import torch\\n\",\n \"from torch.nn import Linear, Module\\n\",\n \"\\n\",\n \"from attention import attend\\n\",\n \"\\n\",\n \"\\n\",\n \"seed = sum(map(ord, 'les bons mots'))\\n\",\n \"np.random.seed(seed)\\n\",\n \"torch.manual_seed(seed)\\n\",\n \"print(f'Seed: {seed}')\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 2,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"class Data(object):\\n\",\n \" dim = 10\\n\",\n \" min_position, max_position = 3, 7\\n\",\n \" q_min = np.random.normal(0, 1, dim).astype(np.float32)\\n\",\n \" q_max = np.random.normal(0, 1, dim).astype(np.float32)\\n\",\n \" query = np.row_stack([q_min, q_max])\\n\",\n \"\\n\",\n \" @staticmethod\\n\",\n \" def create_minibatches(n, m, min_length, max_length):\\n\",\n \" assert 0 < min_length <= max_length\\n\",\n \"\\n\",\n \" minibatches = []\\n\",\n \" for i in range(n):\\n\",\n \" lengths = np.random.randint(min_length, max_length + 1, m)\\n\",\n \" context = np.zeros((m, lengths.max(), Data.dim), dtype=np.float32)\\n\",\n \" target = np.zeros((m, 2, Data.dim), dtype=np.float32)\\n\",\n \" target_indices = []\\n\",\n \"\\n\",\n \" for j, length in enumerate(lengths):\\n\",\n \" c = np.random.normal(0, 1, (length, Data.dim))\\n\",\n \" k_min = np.argmin(c[:,Data.min_position])\\n\",\n \" k_max = np.argmax(c[:,Data.max_position])\\n\",\n \" target_min = c[k_min]\\n\",\n \" target_max = c[k_max]\\n\",\n \" context[j,:length] = c\\n\",\n \" target[j,0] = target_min\\n\",\n \" target[j,1] = target_max\\n\",\n \" target_indices.append((k_min, k_max))\\n\",\n \"\\n\",\n \" query = torch.from_numpy(np.tile(Data.query, (m, 1, 1)))\\n\",\n \" context = torch.from_numpy(context)\\n\",\n \" target = torch.from_numpy(target)\\n\",\n \" minibatches.append((query, context, target, lengths, target_indices))\\n\",\n \" return minibatches\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 3,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"class PointerNet(Module):\\n\",\n \" def __init__(self, n_hidden):\\n\",\n \" super().__init__()\\n\",\n \" self.n_hidden = n_hidden\\n\",\n \" self.f = Linear(Data.dim, n_hidden)\\n\",\n \" self.g = Linear(Data.dim, n_hidden)\\n\",\n \"\\n\",\n \" def forward(self, q, x, lengths=None, **kwargs):\\n\",\n \" batch_size_q, n_queries, dim_q = q.size()\\n\",\n \" batch_size_x, n_inputs, dim_x = x.size()\\n\",\n \" assert batch_size_q == batch_size_x\\n\",\n \" assert dim_q == dim_x\\n\",\n \" batch_size = batch_size_q\\n\",\n \" dim = dim_q\\n\",\n \"\\n\",\n \" q_flat = q.view(batch_size*n_queries, dim)\\n\",\n \" u_flat = self.f(q_flat)\\n\",\n \" u = u_flat.view(batch_size, n_queries, self.n_hidden)\\n\",\n \"\\n\",\n \" x_flat = x.view(batch_size*n_inputs, dim)\\n\",\n \" v_flat = self.g(x_flat)\\n\",\n \" v = v_flat.view(batch_size, n_inputs, self.n_hidden)\\n\",\n \"\\n\",\n \" return attend(u, v, value=x, context_sizes=lengths, **kwargs)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 4,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"batch_size = 8\\n\",\n \"\\n\",\n \"n_train = 200\\n\",\n \"min_length_train, max_length_train = 5, 14\\n\",\n \"train_batches = Data.create_minibatches(n_train, batch_size, min_length_train, max_length_train)\\n\",\n \"\\n\",\n \"n_valid = 100\\n\",\n \"min_length_valid, max_length_valid = 15, 24\\n\",\n \"valid_batches = Data.create_minibatches(n_valid, batch_size, min_length_valid, max_length_valid)\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 5,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n \"net = PointerNet(7)\\n\",\n \"opt = torch.optim.RMSprop(net.parameters(), lr=0.001)\\n\",\n \"mse = torch.nn.MSELoss()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 6,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"[ 1] 0.647\\n\",\n \"[ 2] 0.264\\n\",\n \"[ 3] 0.162\\n\",\n \"[ 4] 0.120\\n\",\n \"[ 5] 0.096\\n\",\n \"[ 6] 0.081\\n\",\n \"[ 7] 0.070\\n\",\n \"[ 8] 0.063\\n\",\n \"[ 9] 0.057\\n\",\n \"[10] 0.052\\n\"\n ]\n }\n ],\n \"source\": [\n \"epoch = 0\\n\",\n \"max_epochs = 10\\n\",\n \"while epoch < max_epochs:\\n\",\n \" sum_loss = 0\\n\",\n \" for query, context, target, lengths, target_indices in train_batches:\\n\",\n \" net.zero_grad()\\n\",\n \" output = net(query, context, lengths=lengths)\\n\",\n \" loss = mse(output, target)\\n\",\n \" loss.backward()\\n\",\n \" opt.step()\\n\",\n \" sum_loss += loss.item()\\n\",\n \" epoch += 1\\n\",\n \" print('[{:2d}] {:5.3f}'.format(epoch, sum_loss / n_train))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 7,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\": \"stdout\",\n \"output_type\": \"stream\",\n \"text\": [\n \"valid loss: 0.059\\n\",\n \"valid error min: 0.018\\n\",\n \"valid error max: 0.015\\n\"\n ]\n }\n ],\n \"source\": [\n \"sum_loss = 0\\n\",\n \"sum_error_min = 0\\n\",\n \"sum_error_max = 0\\n\",\n \"\\n\",\n \"for query, context, target, lengths, target_indices in valid_batches:\\n\",\n \" with torch.no_grad():\\n\",\n \" weight, output = net(query, context, lengths=lengths, return_weight=True)\\n\",\n \" loss = mse(output, target)\\n\",\n \"\\n\",\n \" sum_loss += loss.item()\\n\",\n \" weight = weight.data.numpy()\\n\",\n \"\\n\",\n \" for i, (i_min_true, i_max_true) in enumerate(target_indices):\\n\",\n \" w_min, w_max = weight[i]\\n\",\n \" i_min_pred = w_min.argmax()\\n\",\n \" i_max_pred = w_max.argmax()\\n\",\n \" sum_error_min += int(i_min_true != i_min_pred)\\n\",\n \" sum_error_max += int(i_max_true != i_max_pred)\\n\",\n \"\\n\",\n \"print('valid loss: {:5.3f}'.format(sum_loss / n_valid))\\n\",\n \"print('valid error min: {:5.3f}'.format(sum_error_min / (n_valid * batch_size)))\\n\",\n \"print('valid error max: {:5.3f}'.format(sum_error_max / (n_valid * batch_size)))\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": 8,\n \"metadata\": {},\n \"outputs\": [\n {\n \"data\": {\n \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAr8AAALICAYAAAB2PpiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8VNX9//HXmSUECCCrQFhV9kWWqLijCIJVoBYtLS6ttfitxa39fbVq/f60/bba2vprrbWKy5dWqYooghsCflFEsRLQKqsoBNlJWEKALLOc3x+ThITJMsncWch9Px8PyMzce88952bymc+ce+65xlqLiIiIiIgbeFJdARERERGRZFHyKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn7lhGOMecIYc5/T64qISP2MMYeNMaekuh4ijWU0z6+kE2NMHtAV6GqtLajy+mfA6UBva21eamonIpL+FEdF6qaeX0lHW4DvVTwxxgwBmqeuOiIiJxzFUZFaKPmVdPQccF2V59cD/6h4YoyZZYz57/LHo40x240xPzfG7DXG7DLG/LCede+ssu5kY8xlxpgvjTH7jTH31LRt1e2rPM8zxvynMeZzY8wRY8wzxpiTjTFvG2OKjDFLjDFtE3KERETqVl8c/ZYx5lNjzCFjzDZjzP1Vln3XGLPZGNO6/PkEY8xuY0zH8ufWGHNa+eNZxpjHy+PeYWPMh8aYzsaYPxljDhhjNhhjhlcpu3LbKts3KkaLNJaSX0lHHwOtjTEDjDFe4LvA83Ws3xloA2QDPwL+WkfS2RnILF/3v4CngGuAkcD5wH81cCzbd4CxQF/gCuBt4B6gA5G/r1sbUJaIiFPqi6NHiCTHJwHfAn5ijJkMYK19CVgBPGqMaQ88A9xorc2vZV9XA78kEvdKy7ddXf58LvBIA+rtdIwWiaLkV9JVRa/FWGADsKOOdQPAr6y1AWvtW8BhoF8d6/7GWhsAXiQSnP9srS2y1q4F1gJDG1DPv1hr91hrdwAfAP+y1n5qrS0F5gHD695cRCRhao2j1tr3rLVfWGvD1trPgReAC6ts+1PgYuA94HVr7Rt17GeetXaVtbaESNwrsdb+w1obAl6iYXHQ6RgtEsWX6gqI1OI5YBnQmyqn6mqxz1obrPL8KJBVx7qh8sfF5T/3VFleXMe2NTl+23jKEhFxUq1x1BhzFvAQMBjIAJoBL1cst9YeNMa8DPyMyBmuujgZB52O0SJR1PMraclau5XIBRuXAa+mqBpHgBZVnndOUT1ERBqsnjj6T2AB0N1a2wZ4AjAVC40xw4AbiPQIP+pgtY6iuCoppuRX0tmPgIuttUdStP/PgMuMMe2MMZ2B21NUDxGRxqotjrYC9ltrS4wxZwLfr1hgjMkkMj74HuCHQLYx5maH6vMZ8H1jjNcYM57qQy1EkkLJr6Qta+3X1trcFFbhOeDfQB6wiMjYNRGRE0YdcfRm4FfGmCIiF5bNqbLsQWC7tfZv5dcvXAP8tzGmjwNVuo3IxcEHgWnAaw6UKdIgusmFiIiIiLiGen5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6RkptcjB8/3i5cuDAVu5YUm7GoEIDHxrVJcU1E0oKpf5X6KaZKU6DPB4lTzPE0JT2/BQUFqditiEiTpJgqIhI7DXsQEREREddQ8isiIiIirqHkV0RERERcI+4L3owx3YF/AJ2BMDDTWvvneMsVkdQIBAJs376dkpKSVFelycjMzKRbt274/f5UV0VEkkBxNHGciKdOzPYQBH5urV1tjGkFrDLGLLbWrnOgbBFJsu3bt9OqVSt69eqFMY5MRuBq1lr27dvH9u3b6d27d6qrIyJJoDiaGE7F07iHPVhrd1lrV5c/LgLWA9nxlisiqVFSUkL79u0VsB1ijKF9+/bqARJxEcXRxHAqnjo65tcY0wsYDvyrhmXTjTG5xpjc/Px8J3crIg5TwHZWIo6nYqpIelMcTQwnjqtjya8xJgt4BbjdWnvo+OXW2pnW2hxrbU7Hjh2d2q2IiCsppoqINI4jya8xxk8k8Z1trX3ViTJFROqyYMECHnrooYTuIysrK6Hli4i4xXvvvcfll1+e6moAzsz2YIBngPXW2kfir5KISP0mTpzIxIkTU10NERE5wTjR83sucC1wsTHms/J/lzlQroi4UF5eHv379+fGG29k8ODBTJs2jSVLlnDuuefSp08fPvnkEwBmzZrFjBkzAPjBD37ArbfeyjnnnMMpp5zC3Llzo8q96667ePzxxyuf33///fzxj3/k8OHDjBkzhhEjRjBkyBDmz58fte3xPRYzZsxg1qxZAKxatYoLL7yQkSNHcumll7Jr1y4nD4eISIPFGkc/+eQTzjnnHIYPH84555zDxo0bAXjkkUe44YYbAPjiiy8YPHgwR48erbaPs846i7Vr11Y+Hz16NKtWraq1zKruv/9+/vCHP1Q+Hzx4MHl5eQA8//zznHnmmQwbNoybbrqJUCjk6LEBB3p+rbXLAY3qlgb79oIRMa87b+LqBNZEavPn3CNs2h90tMw+7XzcltOyznW++uorXn75ZWbOnMkZZ5zBP//5T5YvX86CBQv47W9/y2uvvRa1za5du1i+fDkbNmxg4sSJTJkypdryqVOncvvtt3PzzTcDMGfOHBYuXEhmZibz5s2jdevWFBQUMGrUKCZOnBjTRRWBQIBbbrmF+fPn07FjR1566SXuvfdenn322QYcEZGmL9Z4n525NME1Sb50jqP9+/dn2bJl+Hw+lixZwj333MMrr7zC7bffzujRo5k3bx6/+c1vePLJJ2nRokW18qdOncqcOXN44IEH2LVrFzt37mTkyJEcOnSoxjJjsX79el566SU+/PBD/H4/N998M7Nnz+a6665r9LGqiRPz/IpII8X6geC25L93794MGTIEgEGDBjFmzBiMMQwZMqSyd+B4kydPxuPxMHDgQPbs2RO1fPjw4ezdu5edO3eSn59P27Zt6dGjB4FAgHvuuYdly5bh8XjYsWMHe/bsoXPnzvXWc+PGjaxZs4axY8cCEAqF6NKlS+MbLiLikFjiaGFhIddffz2bNm3CGEMgEADA4/Ewa9Yshg4dyk033cS5554bVf7VV1/N2LFjeeCBB5gzZw5XXXVVnWXG4t1332XVqlWcccYZABQXF9OpU6d4DkONlPyKSK3q61lIlGbNmlU+9ng8lc89Hg/BYM09KFW3sdbWuM6UKVOYO3cuu3fvZurUqQDMnj2b/Px8Vq1ahd/vp1evXlFzSPp8PsLhcOXziuXWWgYNGsSKFSsa0UoRcYN0jqP33XcfF110EfPmzSMvL4/Ro0dXbrNp0yaysrLYuXNnjeVnZ2fTvn17Pv/8c1566SWefPLJesusUFdMvf7663nwwQfjant9lPymkHr9RJJr6tSp/PjHP6agoID3338fiPRSdOrUCb/fz9KlS9m6dWvUdj179mTdunWUlpZSUlLCu+++y3nnnUe/fv3Iz89nxYoVnH322QQCAb788ksGDRqU7KaJiDRYYWEh2dmR+5JVXMdQ8fptt93GsmXLmDFjBnPnzo0aSgaRmPr73/+ewsLCyl7m2sqsqlevXrzxxhsArF69mi1btgAwZswYJk2axB133EGnTp3Yv38/RUVF9OzZ06kmAw7f5EKkPsZaqKVXTiTRBg0aRFFREdnZ2ZXDE6ZNm0Zubi45OTnMnj2b/v37R23XvXt3rr76aoYOHcq0adMYPnw4ABkZGcydO5e77rqL008/nWHDhvHRRx8ltU0iTUGb0kyu+WoYV679gHD+gVRXxzXuvPNO7r77bs4999xqF5bdcccd3HzzzfTt25dnnnmGX/ziF+zduzdq+ylTpvDiiy9y9dVX11tmVd/5znfYv38/w4YN429/+xt9+/YFYODAgfz3f/8348aNY+jQoYwdOzYhFxGb2k4PJlJOTo7Nzc1N+n7TjZt6fm3YEnrvE/YtXkmJL4NuV41mSt7UmLdvCsegJun4Hli/fj0DBgxI2v7copbj6sjFwoqpku7qinVdj7Tm7s8vpENp+fCA5s3I+NGVeE7pnqTaOU9xNLHijafq+ZWEsyWlBGbNI/jG+7QpPcrJRw4SmPUak7YOBHUCi4i4Vt/CDjzw6ZhjiS9AcSllT8wh9PmXqauYNGlKfiWhwnv2Ufan5wiv+Spq2dQtQ/nJhrPwhfU2FBFxm5z8bO7992iygs2iFwZDBP7+GsHlTfOsn6SWLnhzgVSdWg99/iWBF96E0tqnOblgT286lWTxyKDlFGWUOrr/ZEvHIQwiIunokh2n8sNNI/HU1QdnIfjqEuyhw/gmnB/T3NsisVDyK46z4TDBhcsJLfk4atmOVu05qeQwLQPHEt3+hR359eqxPDxkGTtaHkpmVUVEEk439KnCwlV5Q7hya/SMKHN6fYHNmsF31y2D8LExcaElH2MLD+O/+lKM15vM2koT1eSTXwWd5LJHignMfoPwhi1Ry7xnDOYvJ53FSSVHuHvDO9gqV/SeXJLFr1Zfwp8GfcQX7XYns8oiIpIE3rDhR1+ewUW7T6n2eogwz/TNZWnXzWRn9ueaUR0I/GMBlB07axheuYZA0RH810/CNMtIdtWliWnyya8kT3jHXgKzXsPuO1h9gceD79tj8J4zjODiQxS0bEPGbdey6g/3MvjgyZWrtQhlcNfnF/D3PqtZnB09RlhExC3SueOmMUO8moW83Lb2XIbv71ptnVJPkD8P/IhPOxy7kYJ34KmYn0yl7Om5cKS48vXwhi2UPf4iGTd+B9MqNTeOkKbhhEt+3T6uMl3bH1q9jsBLCyFw3N23WrUk4/pJeE7pVu1l0yKTh4a+xw835TBm16mVr3vxcMOmHLoebc1zp35K2KPpIOSY3/72t9xzzz0AHDx4kH/+85/cfPPNjS5v1qxZjBs3jq5dIx/IN954Iz/72c8YOHCgI/UVEbCHj/LLzy7mtKL21V4v8pXy+yHL+KrNvqhtPD27kHHrNAJPvozdX3isrG27KXt0Nv6brsLToW3C694UKY6egMlvukrnb+mJZENhgm+8R+j96DlGTa+uZFw/CdOmVY3bhjyWp/uuZGeLQ0z7ehieKlP0jd/Rl5OLs/jLwI8o9h1LqN16nCXi+KD9+OOPxx20Bw8eXBm0n376aUfqKSIR4X0HCcx8OSrx3Zt5mIeGvs+uFkW1buvp2I6M266h7Km52O17Kl+3+w5S9uhsMn78HTzdu1TbJl07iNKJ4qiS3xopwYqNPXyUwD8WEP7qm6hl3nOG4Zs8BuOr5+IEA29138ju5kXcsu5sMsP+ykXD93flgdWX8PCQD5yuer0UQKHkZ79PaPmZj9xZ5/LJkyezbds2SkpKuO2229i8eTPFxcUMGzaMQYMGEQqF+Prrrxk2bBhjx47l4Ycf5uGHH2bOnDmUlpby7W9/mwceeIC8vDwmTJjAeeedx0cffUR2djbz58/nzTffJDc3l2nTptG8eXNWrFjBhAkT+MMf/kBOTg4vvPACv/3tb7HW8q1vfYvf/e53AGRlZXHbbbfxxhtv0Lx5c+bPn8/JJ59cZ1vSlWKdJFJ4+27KnnoFio5Ue31L1n5+N2QZhc1K6i3DtGpJxs1TCfx9PuGNeccWHD5K2V9fxP+DyXj793a45s5RHE3POOpI8muMGQ/8GfACT1trH3Ki3IbyhA3djrYGILwzchu+HofbxLx9um5TsX7Ct9m2C1tSBqWRf7a0DErKf9bwmt27v9p4LAC8XnxTxuI7a2hM+6ywusNO/u+Id/nPL86vNtl596Mn8evVYwkN+RpzUqtGHefGSLvfTfk2pmM7jN8d31mfffZZ2rVrR3FxMWeccQbvv/8+jz32GJ999hkAeXl5rFmzpvL5okWL2LRpE5988gnWWiZOnMiyZcvo0aMHmzZt4oUXXuCpp57i6quv5pVXXuGaa67hscceqwzSVe3cuZO77rqLVatW0bZtW8aNG8drr73G5MmTOXLkCKNGjeI3v/kNd955J0899RS//OUvk358kqFVWQZty5oD8f09uVKVEVs9Dp8U82bhHRWfDw3YJsm/m1jiVuejrSj76wtRU11+0XY3/2/Q8mpn9OpjMpvh/9F3CMxZSDh37bEFZQECT7+CvfISPL26xlw3KD9mPh+eTu1irseJSHG0ZnF/ihpjvMBfgbHAdmClMWaBtXZdvGU3VIuQn9/lTgCgLHcWAL9jQszbp+s2FesnfpvnYlqvVie1IuMHk/H06FL/ujX4Jusg941YzM/XnF/tFFmbQCaBp18BGnecGyP9fjeRbTJ+8SNMp/Z1r9xEPProo8ybNw+Abdu2sWnTpjrXX7RoEYsWLWL48OEAHD58mE2bNtGjRw969+7NsGHDABg5ciR5eXl1lrVy5UpGjx5Nx44dAZg2bRrLli1j8uTJZGRkcPnll1eWtXjx4niamdZG5ffghk2RD7R4/p7c7neMj3ndY58PDd8mWWKPw9UT3+Wd8nii/yeEPOEG79P4vPi/dxnB1lmE/vdfxxaEwwTnLmpw3cpyZ2G6dqLZ//lBg+tyIlEcrZkTXUhnAl9ZazcDGGNeBCYBSU9+JXU8p3bHf93EuK/APdishF8N+19u3nAWo/J7OFQ7OdG89957LFmyhBUrVtCiRQtGjx5NSUndp0ittdx9993cdNNN1V7Py8ujWbNjd5Dyer0UFxcfv3lUWbXx+/2Vk+17vV6Cwdh7sETc6vXu63nhlH9j47hPhTEG/+UXYtpkEXzt3Wq96xJNcbR2TiS/2cC2Ks+3A2cdv5IxZjowHaBHj8YnNXWNO7NHiin98C+NLlsawefDe/4IfJddwJVv5tS/PpCduRSo53cZtgTfWU5o8QpHqikNV99YskQqLCykbdu2tGjRgg0bNvDxx5Ebpvj9fgKBAH6/n1atWlFUdOximUsvvZT77ruPadOmkZWVxY4dO/D7/bXtAiCqjApnnXUWt912GwUFBbRt25YXXniBW265xdlGxsmJmFrfON7gh58S3NR0e7YlOXyTLuaqC+/kqnrWm7GosNrzusaknzWgOz9dPwq/Te+bXiiOpmccdSL5rel7XFS6b62dCcwEyMnJScz3NY/BdOmYkKJdwWOgWQYmM4PlBUsp9gYp8QYiP32BY8+rPH7kyrcxzWu4L3ucjMfgn3A+nq6dCC7LhZIyx/dxQvK5Y7zv+PHjeeKJJxg6dCj9+vVj1KhRAEyfPp2hQ4cyYsQIZs+ezbnnnsvgwYOZMGECDz/8MOvXr+fss88GIhdUPP/883jruCPUD37wA/7jP/6j8kKNCl26dOHBBx/koosuwlrLZZddxqRJkxLb6AZKRkw1LTJPqJi69VDdp3Qr9GzdJ+nbJEPataV5M3wXnYl30GkN37Ye/+q0jf3NjnJV3hCGZg5pVBmmY9OeKk1xtHamrm7pmAow5mzgfmvtpeXP7waw1j5Y2zY5OTk2Nzd6aixJH42Z7SDWbbIzl/LYuNgvXJPkWr9+PQMGDEh1NZqcWo5rHCeBj1FMjUhk3Ip3m2Q40dtS0fNb8fmQTnVrKMXRxIo3njrRjbQS6GOM6Q3sAKYC33egXBEREYlRY5LAdEwcRRIt7uTXWhs0xswA3iEy1dmz1tq19WwmIiIiIpJ0jgwgtNa+BbzlRFkiIiIiIonijqtnRESkydMpfBGJhSfVFRARERERSRYlvyIiIiLiGhr2IDXS6UOpEOt0Q7FKxHsrKyuLw4cPO16uiIgTFEfTi3p+RURERMQ1lPyKSNq56667ePzxxyuf33///TzwwAOMGTOGESNGMGTIEObPnx+13Xvvvcfll19e+XzGjBnMmjULgFWrVnHhhRcycuRILr30Unbt2gXAo48+ysCBAxk6dChTp05NbMNERJJEcbR2GvYgjonlNMzx924XqcnUqVO5/fbbufnmmwGYM2cOCxcu5I477qB169YUFBQwatQoJk6ciDH139QnEAhwyy23MH/+fDp27MhLL73Evffey7PPPstDDz3Eli1baNasGQcPHkx000QkRhp+Fx/F0dop+RWRtDN8+HD27t3Lzp07yc/Pp23btnTp0oU77riDZcuW4fF42LFjB3v27KFz5871lrdx40bWrFnD2LFjAQiFQnTp0gWAoUOHMm3aNCZPnszkyZMT2i5pGpSUyYlAcbR2Sn5FJC1NmTKFuXPnsnv3bqZOncrs2bPJz89n1apV+P1+evXqRUlJSbVtfD4f4XC48nnFcmstgwYNYsWKFVH7efPNN1m2bBkLFizg17/+NWvXrsXnU2gUkROf4mjNNOZXRNLS1KlTefHFF5k7dy5TpkyhsLCQTp064ff7Wbp0KVu3bo3apmfPnqxbt47S0lIKCwt59913AejXrx/5+fmVQTsQCLB27VrC4TDbtm3joosu4ve//z0HDx50zdXOItL0KY7WLH3TchFJC6k6xTto0CCKiorIzs6mS5cuTJs2jSuuuIKcnByGDRtG//79o7bp3r07V199NUOHDqVPnz4MHz4cgIyMDObOncutt95KYWEhwWCQ22+/nb59+3LNNddQWFiItZY77riDk046KdlNlRTSEAZJBsXR9GKstUnfaU5Ojs3NzU36fiX1Ki54e2xcmxTXRGqzfv16BgwYkOpqNDm1HNf6rzKJgWKqNAVN6fNBcTSx4o2nGvYgIiIiIq6h5FdEREREXCOu5NcY87AxZoMx5nNjzDxjTHoP8hCRmKRiOFRTpuMp4j76u08MJ45rvD2/i4HB1tqhwJfA3XHXSERSKjMzk3379ilwO8Ray759+8jMzEx1VUQkSRRHE8OpeBrXbA/W2kVVnn4MTImrNiKSct26dWP79u3k5+enuipNRmZmJt26dUt1NUQkSRRHE8eJeOrkVGc3AC/VttAYMx2YDtCjRw8HdysiTvL7/fTu3TvV1ZB6KKaKpC/F0fRW77AHY8wSY8yaGv5NqrLOvUAQmF1bOdbamdbaHGttTseOHZ2pvYiISymmiog0Tr09v9baS+paboy5HrgcGGM1uEVERERE0lhcwx6MMeOBu4ALrbVHnamSiIiIiEhixDvbw2NAK2CxMeYzY8wTDtRJRERERCQh4p3t4TSnKiIiIiIikmi6w5uIiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4KZsxJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLxU7HT9+vF24cGEqdl2rGYsKAXhsXJsU10REXMQ4UUi6xVTFUxFJgZjjaUp6fgsKClKxWxGRJkkxVUQkdhr2ICIiIiKuoeRXRERERFwj7jG/xpjuwD+AzkAYmGmt/XO85YpI6nx7wYiY1503cXUCayIiIuIsJy54CwI/t9auNsa0AlYZYxZba9c5ULaIiIiIiGPiHvZgrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnEk+TXG+IkkvrOtta86UaaIiIiIiNOcmO3BAM8A6621j8RfJRERSVexzgSSnbk0wTUREWkcJ3p+zwWuBS42xnxW/u8yB8oVEREREXFU3D2/1trlOHR/ehERERGRRHJinl8RERERqUWsw4XivWlQsvZzolPyKycU3XlMRERE4qHkN4X0DU1EREQkuU645FcJo4iIiIg0lqN3eBMREQFoFixLdRVERGp0wvX8iohI+up2uA03bTyT04r+h9It3fFPnYCn/UmprpaISCUlvyIi4oiRBV356fqzaR7yA2C/3kbZn54j4/pJeE7rkeLaiUTTRdTupGEPQGjDFr7/73f51saPsYcOp7o6IiInFguTtg7gZ2vOr0x8Kx0ppuyJOQRXfJaauomIHMfVPb+26AiB194l/OkGKr77lT60Ad8Vo/GeNRTj0b07RETq4g95mb7xDM7b26v2lcJhgi8vwu4qwDfpIozXm7T6iYgcz5XJr7WW0Mo1BOcvheKS6gtLSgm+/A6hVWvxX30pnk7tU1NJEZE017a0OT9bcx6nFVWPkyHCrDn5NE7fs7n668tXY/cU4L9uEqZl82RWVUSkkuuGPYTzDxB4Yg7BF9+OTnyrsJu3U/aHWQQXr8AGQ0msoYhI+jv1UDt+s2pcVOJ72FfKQ0Pf57nhY/FfPwkyqg+DCG/6hrI/PUd4d0EyqysiUsk1Pb82FCL0Xi7Bdz6EYDBq+e6strQrLiIjVGVZMETw7Q8Ifboe/3fH4+nZNYk1FhFJT+fu6cn0DWeSYasPX9jRopA/DP6A3S0Okw14T++H6dCWsmdfhQOHKtez+w5S9ufn8V97Bd6Bpya59tU1Zu54zTcvcmJzRfIb3rabwJyF2B17oxf6ffguPZdHyvrSpuQI9+1dQXjDlmqr2N0FlD36PN7zRuKbcB4ms1mSai4iADYUxubvx7RrgzmuJ1GSx4bDTP16KJO2DYxa9mm7nfxl4AqKfYFqr3uyO9Hs9mspmzUfu2X7sQWlZQSeeQX7rQvxXnQmxlS/xkJJqYgkSpNOfm1pGcGFywktWwXWRi339O2J76pL8bQ/ifCiQg60aIX/x1MIr15P4LV34UhxlcIg9MEqQl98iX/KuJT1Vii4S1NnS0qxuwoI79yL3bGH8I692F0FEAzi/8l38fbpmeoqupItKSXw/Bs1Jr6vd1/PC6d8jjXRcRbgyqXn4+3u4YbikVy8u0rstBB8433+d+UWIGZhAAAgAElEQVQsnu67kjnfzk1U9cUl9BkpsXAk+TXGjAf+DHiBp621DzlRbjxCG7dEri7eXxi9sEUm/kkX48kZFNXbYIzBO3Ignn69CCxYSjh3bfVtDxYRePoVQsP74588BtOqZQJb0bRpfkVn2GAQjpZgi0uhuAR7tCQynt0CzZthmmdW+0mGP+p9n5J6WwuFh8uT3L2Rnzv3YgsOROpe0zY79oKS36QL7ztI4JlXsceN0w2YEE/3W8myznn1lhHyhHmq30q+yTrIdV8Nx1PlkpML9vSmy9FW2DGHMa2znK6+iFRhrMGWlkFZAFo2x3hcd/lX/MmvMcYL/BUYC2wHVhpjFlhr18VbdmO0Ksvg2q9HEHjv5RqXe0YMwD/p4nqTVpPVgozvf4vQyIE1JtHhTzdQuiEP75mDoZHT9kz9emhM6wXeeD+ubdJVrG2BY+1pzDZNQiCILS4pT26PS3ID0WPY6+TxlCfDzaB55rGfmc3AHwkJ128ZgS3PQK2h8jFE8tKKHj5LI45zIIjdU0B4x97qZ1diEN5Zw9AlSajQpq0E/j4fjla/QPigv5g/Dl7OV232xV6YgXe6bWJniyJuXXcOWcGMykV9ijpQ+v/+gXfEQDAmafHR7XG4qWno7yahnynWQjAEgSA/2XIW/rAXf9hDRtiHP+zBH/aSEfZW+emh5OM/gc8HPi/G74s89lc894PfW/maKX8djyfyORAIcsuWs2kW9pER8lb56SUj5Kv8mWG9lL7/JwAyfnkTpl2bhrWrCTC2huEADSrAmLOB+621l5Y/vxvAWvtgbdvk5OTY3NzGnd6qq7dw1N7u/HDTSFoHMqMXtm0dGa4w4JQat52xKJLcPjYu+k1gS8sIvvMhofdzaxw+ISJJ0Kol3qF98X9nbKpr4iRHuuDjial1CX74KcF570I4XO31zVn7+ePgD9ifWfuXl+zMpZXxtKa43floK/7PF+eTXdza2UqLSMwy7rwBT+cOqa6GU2KOp04Me8gGtlV5vh04K6pGxkwHpgP06NH421zWdfo7uGwVwXXvVnstTJiF3TYxp9cXlG6aCZtqLzs7cylQe4Lde0RbfrzxDHofbtfwiotITMJYdrU4xNasg+RlHWBr1kG2tjzIrKs+SnXV0ooTMbWuzoTmQR8Pr7yM9uEW1V73DOvHgKmX8UwdFx5WdCZUqC1u20klBJ57PeoiYxFJkkCg/nWaICeS35oy7ajuUWvtTGAmRHopHNhvFO95wwmtXof9ZlekYl06cl+X2Wxuvd+R8re0OsAvRyzmsu39uCpvMBnhJn29oJwgQoQ56gtw2F/GEd+xf+d1uxSKy7AlJWzbu56WwQxaBv1p9b4t8QT5JusgW8uT3LysA2xvWUipV3Nr1yfRMbXYF+SPgz/g/346hmbl7xnfhPPwXnK2Y2PGTfNM/Dd+h+Dr70XOrIlI4hnA74/MwR1259lsJz4FtwPdqzzvBux0oNwGMx4P/qvHU/aX2fguGYV39BlsfvMvju4j7LG80WMDH3Xayoh9Xbmp788BeG5d7Pu5duAtjtYpEWJtT9W2NGabZEjW7yah7fd6ysfnZnLfp7dypEqiW+IN1vgV9OKJf618/J9Vevj8IQ8tQn5aBDNoEfTTsvxnViCDmwb/ovyrq43+WfHDVnlc25VpdTKYdq0x2Z34/opxtc4QIKm3pdUBnuj/L6ZvPJPW116Fd2hfx/dhPB78ky7Ge+YQwhu3QChc/0YnmCYRg+LcT1P6TGkQn69ynK7xV4zf9WF83kgCevwyr6dynPBPF008Nk445COjfJyw/7hxwjcMuKM8mfVFpoL0+yIXNpe/ht9f7XV83rS46DmVnEh+VwJ9jDG9gR3AVOD7DpTbKJ6uHWn2Xz+JXMTTSI2ZWWDBkZtjXveHY0Y1uPxki7U9VdvSmG2SIVm/m2S1f31efBd9BbxhCr2lFGaURi376fkj4yq7oZT4plassc4WHUn4zDaeLh3xdOmY0H2kSlOLQY3ZT2P2ma6fKcmyq0VRTOtNvzhqpKnUI+7k11obNMbMAN4hMtXZs9batfVsllDxJL4iIlKdpnQUkabEkcF/1tq3gLecKEvkRKW5iEVERNJf+lz5Iic8JX8iIqmjGCwSGyW/0uTpA0FEREQqKPkVERERSTPquEkc993QWURERERcSz2/DtE3NBEREZH0p+RXaqRkXkRERJoiDXsQEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLngTERGRtKMLryVRlPyKuIA+RERERCI07EFEREREXEPJr4iIiIi4RlzDHowxDwNXAGXA18APrbUHnaiYiNRMQxhEREQaL94xv4uBu621QWPM74C7gbvir5aIuIESeRERSba4kl9r7aIqTz8GpsRXHefF+uE6Y1FhgmsiIiIiIqnm5JjfG4C3HSxPRERERMRR9fb8GmOWAJ1rWHSvtXZ++Tr3AkFgdh3lTAemA/To0aNRlRURkQjFVEkVDVeSE129ya+19pK6lhtjrgcuB8ZYa20d5cwEZgLk5OTUup6IiNRPMVVEpHHine1hPJEL3C601h51pkoiIiIiIokR75jfx4BWwGJjzGfGmCccqJOIiIiISELEO9vDaU5VREREREQk0XSHNxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa8R1wZuIiIjUTTeFEEkv6vkVEREREddQ8isiIiIirmHquCNx4nZqTD6wNcG76QAUJHgf6czt7QcdA7U//dtfYK0dH28hiqlJofa7u/2gY5Du7Y85nqYk+U0GY0yutTYn1fVIFbe3H3QM1H53t99pbj+ear+72w86Bk2p/Rr2ICIiIiKuoeRXRERERFyjKSe/M1NdgRRze/tBx0DtFye5/Xiq/eL2Y9Bk2t9kx/yKiIiIiByvKff8ioiIiIhUo+RXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENXyp2On78eLtw4cJU7FpqMWNRIQCPjWuT4pqIuIpxohDFVGkK9DkkcYo5nqak57egoCAVuxURaZIUU0VEYqdhDyIiIiLiGkp+RURERMQ14k5+jTHdjTFLjTHrjTFrjTG3OVExERERERGnOXHBWxD4ubV2tTGmFbDKGLPYWrvOgbJFRERERBwTd8+vtXaXtXZ1+eMiYD2QHW+5IiIiIiJOc3TMrzGmFzAc+FcNy6YbY3KNMbn5+flO7lZExHUUU0VEGsex5NcYkwW8AtxurT10/HJr7UxrbY61Nqdjx45O7VZExJUUU0VEGseR5NcY4yeS+M621r7qRJkiIiIiIk5zYrYHAzwDrLfWPhJ/lUREREREEsOJnt9zgWuBi40xn5X/u8yBckVEREREHBX3VGfW2uU4dH96ETlxfXvBiJjXnTdxdQJrIiIiUjvd4U1EREREXEPJr4iIiIi4hhN3eBMBYj/trVPeIiIikipKfqXJ01hUERERqaDkV0RERBKqvk6I3kVtuf7r02nF6YR6XIC3f+8k1UzcSMmviIiIpMyZe7vx0w2jyAj7gD0EnpqLnXQR3vNHErmVgIizlPyKiIhI8lm4Ylt/vr952HGvW4Kv/S82/wC+yWMwXl2bL85S8iuEt+7iB6uX0aq0mGCrYXjPOp0r3xgZ07YaIysiJzJdqJsa3rCHH32Zw0W7T6l1ndCHn2L3FeK/7gpMZrMk1k6aOiW/LmaLSwm+tYzQR58y2EZeC768iNAna+jRsQ3fZBWmtoIiIpJQqUj+Wwb83LH2PAYdPLna60ETJuhpRmYoUPlaeMNmyv7yTzJu/A6mbWvH6iDupnMJLmStJfTpekofeprQh5+CPW751p38NvdSvv/16TQLeVNTSRERaXI6FWfxwOqxUYnvYV8ZDw59j7+OmgQntaq2zO7Kp/RPzxHetiuZVZUmTD2/LhPed5Dg3MWEN26pcz0vHq7YNoBRe3swq88qVnfYmaQa1k3TlomIpFZje4v7HezAz9ecT6tg9SEMuzOL+P3QZexqUUR2Znua3X4tZc+8it22+9hKRUcoe+wF/NdcgXdIn7jbIO6mnl+XsMEQwSUrKPvdszUmvvktWrOuY4+o1zuWtuQ/11zAHWvOpV1J82RUVU4Qdn8hoS82Edq0FRsIpro6IpLGztnTk3v/fVFU4ruhdT7/NWIxu1oUVb5mWmeR8dPv4Tk+yQ0ECcyaR3DpJ1h73ClLkQZQz68LhL/eRmDuIuyefdELvV68Y87ij3YAQa+PP3fdS+DVJXCwqNpqZxZ0Z8iBzrzc6wveyd5E2KPAc7ymfuGMLSkl/PU2whvzCG/cgs0/cGyhz8cvWl3IF+1283nb3WxrWQiaoUjE9ay1hBZ9xC3rz45atrxTHk/2/4SgJxy1zGT48V8/meCb7xNa+kmVAiH4+nvYggP4rrwE49XQPGk4Jb9NmD1STPD19wh98kWNyz2ndsc3ZRyek9sTXBS5uM07uA+ePj155a//wYTtffFWOTnQPOTnuq9HcP6eXjzdN5fNrfcnpR2SGjYcxm7bTfjLPEIb87B5OyEc/SEFQDDI6Qe6cPqBLgAcyCjmi7a7y//tobBZSRJrLk1BU/8yWZ+mMMTLBoMEXlxIePW6qGVze33BKz3X1vkl2XgM/itGYzq0JfjKIggf63QJrfh3ZCaI6ydhmmsmCGkYR5JfY8x44M+AF3jaWvuQE+VKtJgCooW5Xf9BYMFSOFIcvbxlc/wTL8KTM6jGCcRNswxmn/YZH3TO48aNOfQp6lBtee/D7fj16rEszt6ELS5V4GlCwvsLIz27X+YR/nIrFDcuaW1b1pwL9vTmgj2RuzTltTzAF+0iyfCGNgVOVlnEGRY81uCzHrzWgy9s8FY+9hAuOBDpZfR6oOKnp/yxxzSNmzFY8Npj7fZagzcceWz3F5a310PLQAZhEyZoLGETJmRsVBLbqiyDsr/NwW7ZXu31gAnxZP9P+PDkrTFXy3f26Zh2bQj8/TUoKat8PfxlHmWPPo//x1PwtGsTV9PFXUy842aMMV7gS2AssB1YCXzPWhv9Va9cTk6Ozc3NjWu/NbFHSyj9w/84Xm46KSjeXe86vrCHkwI1j8/1njkE3xWjMS2rL59R3vP72LhIAKlIso01XLLzVKZuHkqLUEZ0gc0yoDz5jaVuAB2ad45pvZrEuo+q+4l1G4OhffMqVyDX9LdRx5/L/pK9Na9y3IdCtX2kk1AYio7Evr7Hg+nRGXuwKGqYTF3KPEEyWrdtRAUbx3/5hXhHDEza/hrIkYwpUTE1tHINgbc/cLzcWCQ0noRt5CxGMERJ2RF81uCzcZ4+r0yMKxJiD3ga/+ttTKxrkFAYQiGOlhZVJrw+2/jLgEKECXkiiXDIhPGFvWSGq/evFflK+ePgD9h4Uu1fgLMzl1Z+Dh0vvLuAsqfmwoFD1Rdk+KFFZqPrLg3T7M4b0nXe5Zj/4Jzo+T0T+MpauxnAGPMiMAmoNflNGGsb9CF8IupAy0ZtZ05uj3/KODyndm/QdtZYFmd/xcoO27n2q+Gck9+z+gqlZZF/DalbaeN/Rw1qf/l+GrNNY7SjRcL3kWqmUzs8fXvh6dcLz6ndMZnNsNZi8/czc+7NDDnQmUEHOpEZ9tdaRkbYl9S/U1sWqH8lqZEtC6QspiYjngBkOjX6LxSK/HNIsuJWC2r/W20ILx684dqT5x3ND/H7ocvY2/xwo/fh6dzh2EwQ31SZ9qwsEPknEiMn/uqzgW1Vnm8Hzjp+JWPMdGA6QI8e0bMKxKqu0/5ZgQye4spGl90k+Xy82H01b3TfQGjtX2Ft7atmZy6tfFzbGLLQhi0EX1mM3XfQ6ZpKGirylbKm7Z7I2N12uynIPBpZ8HX5PyLvFdOpPf9x80tAZGYRu3UnoYoL47bvrrPHXBrHiZha3zCqsTtO4wZyGlW2SAXPaT045QeTebKe3tmKM5AVant/+nt6ubn4LEblNz6XEHdzIvmtqZs56qPOWjsTmAmRU3QO7FfqYgyeAb3xTR7D/I+ed6xYb//eeP7zhwSXrCD0werKXl9pIjweTO9sXgy/yedtd7Ol1QGsadifq/F5Mad2j5xluOx87JFiwpu2Et6YR+jLvOhTltIoiqkJYEz08AWvJzLW11psOAyhMEXF+yNjg8MefNbgaUqzhnoMeKqOaS7/Z8yx4SKhMIRD5T/D1S5EO573nGH4Jo/B+JyblSHgDfHowI/YkXeIyVsHVrswWyQWTiS/24Gq59K7ASm5I8IRX4AZoxYA8NTYtwD48eLLYt4+2dskVGazhF2IZjL8+C+7AN+4cxs2RvQEcmPl7zM6qNsqX/eeGbuw7oLqGVMf6/um6numMdvErGVzTIaf1xY83PBta2FaNsc7rD/eYf3xWQuHj0Iwcno4kX9rle3XWMBG++DkPFa3j4TzRh//Rm7TGDHvZ9zb1S9c83rA48XEOEb3e8f1SBpL5YVxkbGzkQvFnh77doPqBfEdgwYd56oX7FU89nhiPgZV2bAFW54UVyTEoRBk+BM2NtQamNt7Da9338ALFy1JyD5qk5S4lc7bZNRw/c8JxonkdyXQxxjTG9gBTAW+70C5NWrolC77Kk7TxqDivuHJ2uZEZ3xeaCJtOd7+GH+f5rjbcDZUrO+bqu+ZxmyTLowx0OrYWMbG/N08ffVyx+slNSvxBSnxRW5gEs/xT9Z7Nub9xPl3ezxrIGjCUfPVJvvzIVWxIZIweyOJdJKV+oJJj3WJzBHijfVu+0xprLiTX2tt0BgzA3iHyFRnz1pr6xhZmlzpOv+hCOj9KSIikmyOXOZqrX0LSMK5fBFpSpT8u4N+zyKSTnSHNxERkSagqX3JaGrtSVduPM5KfmvgxjeCnDj0/hSn6L0kIm6k5Nch+hAREZGa6PNBJL0o+RURERGJkb7MnPg0M7SIiIiIuIZ6fkVEpElQj5ykK70304t6fkVERETENdTzK1IDfUsXkdooPoic2NTzKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIamu1BRKLoanYREWmq4kp+jTEPA1cAZcDXwA+ttQedqJg4J5ZEZsaiwiTURERERCS14u35XQzcba0NGmN+B9wN3BV/tURERETcS2fgEieuMb/W2kXW2mD504+BbvFXSUREREQkMZy84O0G4O3aFhpjphtjco0xufn5+Q7uVkTEfRRTRUQap97k1xizxBizpoZ/k6qscy8QBGbXVo61dqa1Nsdam9OxY0dnai8i4lKKqSIijVPvmF9r7SV1LTfGXA9cDoyx1lqnKiYiIiIisdM44djEO9vDeCIXuF1orT3qTJVERERERBIj3jG/jwGtgMXGmM+MMU84UCcRERERkYSIq+fXWnuaUxUREREREUk03d5YRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4I7ExJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcw5eKnY4fP94uXLgwFbsWSakZiwoBeGxcmxTXRNKEcaIQxVSR9KN4n3Qxx9OU9PwWFBSkYrciIk2SYqqISOw07EFEREREXEPJr4iIiIi4hpJfEREREXGNuJNfY0x3Y8xSY8x6Y8xaY8xtTlRMRERERMRpTsz2EAR+bq1dbYxpBawyxiy21q5zoGwREREREcfE3fNrrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnHkDm/GGD+RxHe2tfZVJ8oUERERSTffXjAi5nWzM5cmsCbSWHEnv8YYAzwDrLfWPhJ/lURERBou1qRk3sTVCa6JiKQzJ4Y9nAtcC1xsjPms/N9lDpQrIiIiIuKouHt+rbXLAeNAXUREREREEkp3eBMRERER13DkgjcREXEHjasVkROdkl8RERGX0pcZcSMlvyJJYkvLGP/lJ3Q6cpBQzxy8/Xonbd/6gBMREYlQ8iuSBLboCGVPzeWS7XsACDy5BTvpInwXnpHimolIU6EvuSKxUfIrkmDh/AMEZr6M3Xew2uvB+UuxhYfxXT4a49GEKSIiTYmxqa6B1EbJr0gChbftouypV+Dw0RqXh95biT10GP/UyzA+b5JrJyIiTutQ0oLrNo3g9AOd2XrS64SHXIqni25Bnk6U/IokSGj9ZgJ/nw9lgTrXC69eT6DoKP4fTsZkNktS7URExFEWLtzdm+u+GkGLkB+APvt3UvbIP/BNOA/v6DMwHs0wmw6U/IokQPCTLwjOWQjh6ue9crv25cMeg7ht7SIoOlL5enjTVsr++gIZP56CaZ2V7OpGaci96zV+UMR5+htsnFSNe25T1owbN55Bzr5u0QtDIYJvvE9o7df4vzcBT4e2ju5bGk7Jr6RMUwzu1lpC735M8K0PopZ5x4ziRd9QMIaMW6dFxgHnHzi27Y69lD06G//0KXg6tU9mtUVcSxeJSbzOzO/Gj77MoXUgs8717JbtHHrocWaf+hlLun5d471x9T5LDvW/izjEhsMEX10Snfga8H37EvzfugBMJNp52p9Exi3TMD26VC9jfyFlf/kn4bydyaq2iIg0QsuAn5+uG8Uda8+LSnxLPUHe7LaBI/7qQ9kyw35+tOkM7vriAtqWNk9mdaUKJb8iDrBlAQJ/X0Dow0+rL/B58V83Cd/50b1LJqsFGT/5Lp6Bp1RfcKSYsr+9SGjtVwmssYiINNaQ/Z353coJnLe3V9SyL1sX8IuchTx/2mf84dyr8Aw4JWqdYfu78vuVEzhnT0/QrBBJp2EPJxidoks/9mgJZc+8it2yvfqCzGZk/OhKPKd2r3Vb0ywD/w+vJPjyO4Q++eLYgkCQwLPzsFeNwzfq9ATVXKTp84YN3Y+cxKlF7TjlUDt6H25L6YZnMV074enRmT6F7dmadZAybyjVVZUTgC0t44YvRzJ2Z5+oZUET4uVea3i9xwZs+TxnRZkt8d/4HR578iqu+Xo4zcsvhAPICmZwy/qzOaMgm2f75FKUUZa0dridkl8XUMKcOPbAIcpmvozds6/6gjZZZEy/KqbpbYzXg++746FNFqHFK6oUbgnOeQcKD+Mddw7GaC5gkboYa+h6tFVlontKUTt6Hm5Lhq0+jaA9XIDdXUB49Tp+xVhChNnespCvW+1nc+v9fN1qP9taHiTkUZecHBPesp3AP99i7L7oxHdrywM8PuBffJN1MGqZMYb/7bqZNW338B8bzmJAYadqy0fl96D/wY7M7LcyYXWX6hxJfo0x44E/A17gaWvtQ06U21A2FMJ+sysVu06avoUdYlovXKUXMuZtNm+vZUljPwBqSdbKX461XlC9PenCFpcSeDmSnFZlTm5PxvSrMG1bx1yWMQb/hPMxbVoRfGUx2GPHPPjOh9iDRXjPHAzA3ctviKnMB897tvJxQ983Df7dNMvA07VT/SvLCcUeOhy5OYuFijjQ/2DkC50BsObY48qfkf9DX249VlDVUGDMsacVX+hMlZUa+B3PFh4m/M0uwtt280zeldV61mLlxUPPI23peaQtF+8+FYAyE2Jr1gE2t97P5lb7I/Exid8/Y/0bNPZY7O53MNZYv63yccXvE2ponj32Suirb2per+qXclPDGkn+zt7QWBdrPB1ZkM3E7QOrxWaAMGEW9NjAK73WEPSE6yxjb/Mj/HrYUi7b3perNw+t9qXspEBz7lxzAYGX3sY36WJNe5lgxtr4vtkaY7zAl8BYYDuwEvietXZdbdvk5OTY3NzcuPZbE3ukmNL7/uJ4uSKxMqd0I+OGKzEtar7qd8aiQgAeG9em1jJCX2wi8NzrEAwmpI6JYHpl0+zWaamuxonIkdQgUTE1uHw1wVeXOF6uSFOwq3kRf+v/MZva7Kt1nezMpTw2rk3UGdhuR1rzk/WjOOVwu+iN2rbG/73L8J7Ww+kqN3Uxx1MnLng7E/jKWrvZWlsGvAhMcqBckROKZ0hfMm66utbEN1beIX3I+MnV0Dy+ckQkYl/GUVZ22M6LvT/nt0OXkvHT7+GbeBGe4f3ZnVmU6urJCeidrl9yd87COhPfumxveYj/GrGYV3quIcRxPcYHDhH4n9ewJaUO1FRq4sSwh2xgW5Xn24Gzjl/JGDMdmA7Qo0fjv83UNX41K5DBU1zZ6LJFGst77nC+6/sF9u1f1btudubSysd1vZ+7Dm7NLz6/kI6lLR2pozQtTsTU+q4HGLvjNG4gp1Flp0zL5ni6d8Z074yne2c8PbqQ3TqLbOD8KqtVXIjai4nYI8WEt+/GbttN+JvdhLftihrOJAJAmyz8Uy9jUr9e9fbyVZzpg7qvqQl/s4vAP9/E7t1f+Zr/22M09CGBnEh+a+pmjhpLYa2dCcyEyCk6B/YbJWQsG1vnA9C/3TAANuz/LObtK7ZpjMbsJ9Zt4qlXg5X/Ntfv+6zG1483oN3wysfr91eZ5quW37Ah+cc5kfsZcPJIPMMG4B01FPu6s2/rneU9A1dtGcKYzAscLdtpni6xjxEWZyQjph7MKDkWU3nukL0AACAASURBVNsPo8Zxuab6uF+Mqb7cwhcFkQt5jDVRocRgKuPFgEb8zZoMX/nMDV0w3Ttj2rVp8MWhpmVzvP16Q7/ex6p96NhYYrt9D7b4WC9c2sXuRg2eiXFc7vHHsrahkjbqQbWHjTlmaXWcPQbPqd3xjT4D4/BZOU+PLmT87HqCb39AaFkuniF98Ywc6Og+pDonxvyeDdxvrb20/PndANbaB2vbJp7xaQ2duSBZdxFrzH7SeRaGxtQtGe1Jt99nY9pfMQassfuRJiWlY36b4l0Wk0F/tw2Xrp8piRTLNR7HC3/1DaZzB0xWi0RVqymLOZ460fO7EuhjjOkN7ACmAt93oNwapeubPF3rJSIiciJy4+eqRxe5JUXcya+1NmiMmQG8Q2Sqs2ettWvjrpmIiIiIiMMcmefXWvsW8JYTZTnNjd8cJX5634iIiDRNTkx1JiIiIiJyQtDtjUVSSD3MIiIiyaWeXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa+iCNxERkQbQhaoiJzb1/IqIiIiIa/z/9u48Tq66zPf496mlO/tGNukQkrBvgUCzGTaRJSCyyggqoijRYeLgdkcZx6tXr3d0FjdwmbAMClFGCZsCMQEjmwRpkgAJYQmEJSQknX3v2p77R1Wn9+7qqlNV3XU+79eru0+d9fmdrvqdp37nd86h5beCaD0AAAAoL1p+AQAAEBokvwAAAAgNkl8AAACEBn1+AQBASXGNC/oSkl/0K1SgAACgGEV1ezCzfzezl83sBTO718xGBBUYAAAAELRiW34XSLrB3VNm9gNJN0j6WvFhoT+iVRYAAPR1RbX8uvt8d0/lXi6SNKH4kAAAAIDSCPJuD9dIeririWY208wazKyhsbExwM0CQPhQpwJAYXrs9mBmj0ga38mkb7j7/bl5viEpJWlOV+tx99mSZktSfX29FxQtyoYuDEDfFkSdyuccQBj1mPy6+1ndTTezqyVdIOmD7k5SCwAAgD6rqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqQd7tAQAAAOjTSH4BAAAQGsU+4Q1ATj63jZo1f2sZIgEAAF2h5RcAAAChQfILAACA0CD5BQAAQGhYJR7KZmaNkt4q8WZGS9pQ4m30ZWEvv8Q+oPx9v/wb3H1GsSuhTi0Lyh/u8kvsg75e/rzr04okv+VgZg3uXl/pOCol7OWX2AeUP9zlD1rY9yflD3f5JfZBNZWfbg8AAAAIDZJfAAAAhEY1J7+zKx1AhYW9/BL7gPIjSGHfn5QfYd8HVVP+qu3zCwAAALRXzS2/AAAAQBskvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEIjVomNzpgxw+fNm1eJTSNAs+ZvlSTddM7wCkcC9FsWxEqoU/se6keg7PKuTyvS8rthw4ZKbBYAqhJ1KgDkj24PAAAACA2SXwAAAIRG0cmvme1nZgvNbIWZLTez64MIDAAAAAhaEBe8pSR9xd0Xm9lQSc+Z2QJ3fymAdQMAAACBKbrl193Xuvvi3PB2SSsk1RW7XgAAACBogfb5NbNJkqZJeqaTaTPNrMHMGhobG4PcLACEDnUqABQmsOTXzIZImivpi+6+rf10d5/t7vXuXj9mzJigNgsAoUSdCgCFCST5NbO4sonvHHe/J4h1AgAAAEEr+oI3MzNJt0pa4e4/LD4kAAD6rkseODav+eoGLCxxJAAKEUTL73RJV0k608yW5n7OD2C9AAAAQKCKbvl19ycV0PPpAQAAgFLiCW8AAAAIjSAecgGgQPn2Hbz3wsUljgQAgHAg+QUA9Dl8MUQheN8gH3R7AAAAQGjQ8gsAQIDev25/ffidQ7VnwEPKHH2OIuP2qXRIAFqh5RcAgCC49HdvHKUvrDhZk3aM1KEb3lHip3cq8/o7lY4MQCskvwAAFCmaiei6l0/SJW8f0XbC7iYlfvk7pZesqExgADqg2wMCw4UGADpT7XXDwFRcX152io7cMq7zGdJpJe/4g3zLdkXPOF7ZB6P2X9X+/0T1I/lFQTzjmrxpraKekaeHyqKcROCAAITPqD2D9LUXT9PEnSPajN8VTWhQuqbNuNQf/iLfvE2xi8+URagzgUoh+UWvuLsyL72u1ENP6B/WNkqSEm8+pdh5p8pc8v7doAEAeZu4Y7j+6YXTtU9iUJvxawdu1/enPqYTd/2nPrb8MSmT2Tst/eRi+Zbtin/iAllNvNwhAxB9ftELmZVvK3Hjb5S89R55LvGVJG/crOSvH9D3njtHR28cL3kFgwSAMjhy0zh9a8lZHRLfV4dt0LemPaL1A3docd3Bis+8XBrQtgU4s+w1JX5+l3zHrnKGDCCn6lp+U396Spn1mxQ9fIoih06RDR5Y6ZD6vczq95R68AllXlnV7XyTd4zS1188QyuGr9dvpzyv14ZvLFOEwaMLA4CunPbeJF37ygmKedv2o7+Nfkc/O2yREtH03nHRg/eXzfq4Ejf/Xtq6Y+94f3utEj+9U/FrL1dkzMiyxQ6gCpPf9OKX5I2blVmyQjKTTapT9PADFDl8imz86LwuNMg38ZGqO/nJrN+o1LwnlVn6SufTJbmZot62qfewrWP1nSVn67l93tXvJr+gt4dsLUO0AFBa7q70gqf19y+f1GHaw3Wv6I4Dl8qt46mvyL5jVHv9VUrcfHfbs2Ybtijx0ztV85nLFJm0b0ljB9CiqpLfzPpN8sbNLSPc5atWK7VqtfTgY9LIYblE+ABFDpwoi1dV8QPjm7cpNf8ppZ9dJmU678MQOeIA/eeIaUpGovrn7Us7TZCP21inaRv31V/HvqXfT16m9QN3dLImAOj7PJ1W6u4FSj/zQodpdxywRA9NeEXqpm3FRgxVzayPKXn7fcq89lbLhJ27lfjFXYp/4sOKHnVQCSIH0F4g2Z+ZzZD0E0lRSbe4+/eDWG9vZV7u/rS8Nm9T+qklSj+1RKqJK3Lw/oocdoCih0+RDR9aniD7MN+xS6lHF2X3Tyrd6Tw2ZYLiHzpdkcl1Wjs/26Jbc8lFypz5nv52+/d1zKa2rRcRmU5ZP0knNU7Uwve9rnv3X15UjEG1yns6Le1JyJsS0p6m7HAiKauJSbW1GrN7sHbHktodTSodoRMzEGa+p0nJXz/Q4RiTtLR+ftgiLRqb30MsbGCt4td+RMnfzVOmoVVdmEwpefu98ovPUuzU/Os4AIUpOvk1s6ikn0k6W9JqSc+a2QPu/lKx6+6t6CnTFNlvvNIvva7MS6+3Ob3UQSKpzLKVyixbqZQkmzBONn60ZKbPvXOCpOx1Wy6XrOUaLs8NNZ/aSt49v/ug2nUJyK7U9/54q+GWH2VbXD3T8tpMMtPT7/1Zbq6MXG5SxlwuV6bVOJfr3MkfyS6Tr2RK6edflpqSnU62urGKnX+aIodO7rTrSGTCeP1g6uM6dMsYXfHGVB2ybUyb6TGP6Ow1B+m09yYrkXxQVlujh1f9Lu/wzpv8d5KkT686Lq/5TVLiNw+2JLZ7mqSmhHxPLtlNprpd/qf68N7hRCSl3dGUduWS4d17/6aUTMzfu5/zjS05d8He4d4uEzvn/bKhg/NaBuhJT18mD90yRievnyip5TNYiLzf53fP31v3fe6tExRxk8kUccsNq9Vw9m/ivd9LEcvVkZGW4b1/I9k6q3lc8y3GelE/Zt5Y3eF4siPWpP848km9MqKb40wnLBZV/MrzlRo5TOkFT7dMcCl17yPKrHxbNqxvf8YLqetKpvkYmztWXvvW8ZJnG14kyXLvG5PJvOVvYsv9slEjFL/g9NLHiD7HvH1y1tsVmJ0s6dvufm7u9Q2S5O7/2tUy9fX13tDQUNR28+Gbt+1NhDOvvdVlaya6ZmNGKnbeqYpMPUQWaXuwmJVr+b3pnOGSWh1IXZq2cV99dNVR2n8nF3IEqebrn1Fk7D6VDgPBCeTmgIXWqT0lv2e/e6Cuea2+0LCq1voBO/SDox7TmsHbu52vbsDCvfVjZ1JPP6/U3Plddi9DaVndWNV+5VOVDgPBybs+DaLbQ52k1ud8Vks6sUNEZjMlzZSkiRMnFryx3lyFbyOH6fKNn5HGSbWjozpi8zhN27ivpm3ct8PtadDWxtpdmrv/Mj02fpUyb/9Cervz+eoGLNw73L6bgWdcmSUrlJr3pHzjllKGC4ROEHVqTxfspp5aotRrZWi960dswjjt99nr9LNh3+l2vubGAan749YxR7xP1y9/vwZkuOdv2fGdI7SCSH47y7Q7vKXcfbak2VK2lSKA7fZKUzStxaPXaPHoNZJL++8YoWNzifAB2/fZe4ok9AYPVOyDJ+lLm/9eyWim5/m7YRFT9LjDFTnmEKWfeUGp+X+Vtu0MKNDiZOQdujA0RVKaOuI4aU9Tm77AHbquAH1ApevUMIocNkXxT14oq63peeZWevqSkXnnPSVumStt7xv1Y2hQt4dWEMnvakn7tXo9QdKaANZbOia9NXSL3hq6RfdOeknDErU6dMsY/a+p38tOb+5nq/Z/2w277+03NvuFf22e2om2Yz93zDcki+im5/9Prv9utg9xc99dz73O9uF13XDCj7ILZlr1C8609Af2vcPedp7e7pYhg7J9egfUKvlAcYlvm/VGo4q9f5qi9Ucq8/Iq+bby3PXBamuk2hp9e/H1rZLcpHZFU2qKpjr92nbvhT9s89rdpUQy11+4uf9wLjFuyl0wV0Y2pG/3BUR1iUyeoNilZ7UZN/vF/K5nnnnU1wvcaK7vbq6PrrXuq9vcd7d1n16ztnVfc33Yqi70Tsb1lo0anr13fCT4hpLIfuNV+0/XKL18Zba+Uf77WWrZ1yX/3xShkNjyXmbqDdkBs2y9btlevs3De/t8t59nQG3e8aO6BJH8PivpIDObLOldSVdI+lgA6y2bbTVN+tvY1YqdcFTB67huev4XbzV7bH0Pd6fIiR5ZHbe/sZq4olMPLvt2l725ruBlzUzKJdE2bEiAUQF9X2TfMYrs2/bi1etO6X1dh57Z4IFtjkELNq3Me9nrTjm2V8s0z19OhcSW9zLTpxUUE8Kr6OTX3VNmNkvSn5S91dlt7l7c/awAAACAEgjkPr/u/pCkh4JYV9Cq+QlsAAA043gH5IdHnFUQFRUAAEB5kfwCANDH0DjCPkDpkPyi6lGBAgCAZv0u+SWRAQAAQKH6XfKL8sjnS0brJxgBAAD0ByS/AACgpDhri74kUukAAAAAgHIh+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFRVPJrZv9uZi+b2Qtmdq+ZjQgqMAAAACBoxbb8LpB0pLtPlfSqpBuKDwkAAAAojaKSX3ef7+6p3MtFkiYUHxIAAABQGkH2+b1G0sNdTTSzmWbWYGYNjY2NAW4WAMKHOhUACtNj8mtmj5jZsk5+Lmo1zzckpSTN6Wo97j7b3evdvX7MmDHBRA8AIUWdCgCFifU0g7uf1d10M7ta0gWSPujuHlRgAAAAQNB6TH67Y2YzJH1N0unuviuYkAAAAIDSKLbP702ShkpaYGZLzeyXAcQEAAAAlERRLb/ufmBQgQAAAAClxhPeAAAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaFglnkhsZo2S3irxZkZL2lDibfRlYS+/xD6g/H2//BvcfUaxK6FOLQvKH+7yS+yDvl7+vOvTiiS/5WBmDe5eX+k4KiXs5ZfYB5Q/3OUPWtj3J+UPd/kl9kE1lZ9uDwAAAAgNkl8AAACERjUnv7MrHUCFhb38EvuA8iNIYd+flB9h3wdVU/6q7fMLAAAAtFfNLb8AAABAGyS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQiNWiY3OmDHD582bV4lNA6iwWfO3SpJuOmd4hSPpEyyIlVCnVgc+G0BR8q5PK9Lyu2HDhkpsFgCqEnUqAOSPbg8AAAAIjYp0ewDQt13ywLF5z3vvhYtLGAkAAMEquuXXzPYzs4VmtsLMlpvZ9UEEBgAAAAQtiJbflKSvuPtiMxsq6TkzW+DuLwWwbgAAACAwRbf8uvtad1+cG94uaYWkumLXCwAAAAQt0AvezGySpGmSnulk2kwzazCzhsbGxiA3CwChQ50KAIUJ7II3Mxsiaa6kL7r7tvbT3X22pNmSVF9f70FtFwDCiDq1/+jNBaR1AxaWMBIAUkAtv2YWVzbxnePu9wSxTgAAACBoRbf8mplJulXSCnf/YfEhAQhavi1P3LYMAFDtgmj5nS7pKklnmtnS3M/5AawXAAAACFTRLb/u/qQCej49AAAAUEo83hgAAAChQfILAACA0CD5BQAAQGgEdp9fAOiJ79qjY9au1Jqh+0gaXulwgD5jSLJGx22oU2LkOvHZAEqr6pPf3txcnNs8AaWTeXONErfO1Sd27lZGptToMxU77bhKh4Ve4rZ5wTtg2yj904unaVhygKT7lBx0omIfOk3ZO4kCCFq/S36peIH+J73sNSXv+IOUTEmSInKl7ntUvnW7Yh86XRbhII9wmrbxffrH5dM1INNyOE7/+Rn5th2Kf3SGLBqtYHRAdep3yW81IZFHGKSeXqrU3Qsk7/gE3vTCv8m37lD8ivNkMQ7yCJcz1k7WZ185XtFOLr/JNCxXcvsuxT91kay2pgLRAdWLC94AlIS7KznvSaV+P7/TxLdZZvFLSt58t3xPUxmjAyrIpUvePFyfe+XEThPfZplXVinxs9/Kt+8sY3BA9aPlF0DgPJ1R6u4/Kf3Mi20nmOmRycfoxNUva2hi997RmdfeUuKm36rm2stkw4eWOVqgfMxNn37tWJ295qA24zNyzZ20TB9YN12jd2/bO95Xr1Pip3MUn3m5ImNGljtcBKSQM72cHS4dWn4BBMqbEkredk/HxDcWU/zTF2vewSfoxpMulrU7kPua9Wr66Rxl1m0sY7RA+cTTUX1p+fQOiW/C0vrREU/qnknLddNJF8kmjGsz3TduUeLGOcq8s7ac4QJVi5ZfVAx34qg+vn2nErfMlb/zXtsJgweq5jOXKjKpTlqzVZsGDVPNFz6enfftVgf0zduUuHGOaj5zmSKT68obPFBCg5M1+uqLp+rQbWPajN8RS+g/jnxcr4zYkH1dO0g1112h5K/uV+aVN1vNuEuJn92l+NUXKXrYlDJGjvY4dvV/tPwCCERmw2YlbpzTIfG1UcNV84WPZRPf1uOHZA/ykcMPaLuiXXuU+MX/KP3ia6UOGSiLffYM0reXfLBD4ruhdqe+Pe2RvYlvMxtQq/hnLlOk/oi2K0oklbz1HqWfXVbqkIGqRvILoGiZd9Yq8dM58g1b2oy3urGq+cePKzJ2n06Xs5q44p++RNGTpradkEopeft9Sv11SalCBsois6ZR31l8libsavvgircHb9G3pj2idwdv63Q5i0UVv/J8Rc88sd0KM0r+9iGlHl0k7+ZCUgBdo9sDgKKkV7yh5K/ulxLJNuMjB++v+Kculg2o7XZ5i0YUu/xc2fChSv3pqZYJ7krdvUC+ZYdi553CDf/R76RXvq3kbfdoVGJQm/Erhq/Xfxz5hHbFk10smWVmil9wumz4EKXue1RqleumHnxcvnWHYhefKYuUtx2Li7fQ3wXyiTGzGWb2ipmtNLOvB7FOAH3faWsnK3nr3I6J73GHK/7Zj/SY+DYzM8XOna7Y382Q2j3wIv3I00re9bA8nQ4sbqDU0ktfVvK/fi/tSbQZv2jM2/rXqX/pMfFtLXbqcYpfdaHU7oEX6ScXK3nHH+S5h8cAyI8Ve9rEzKKSXpV0tqTVkp6VdKW7v9TVMvX19d7Q0FDQ9rr79jg4WaMfP/MhSdKQePYU047k1rzXXcwyhch3O8Vsoy8r137uewr4zHnul7ca0Tzc/BnuME+OSYlMk3zvlnO/reW1514PiQ+T1Iv3Zqpjchs980TFzj+ty6e2zZqfXfdN53T+P02/9LqSv36gY0J9yORqvOF/IM3ZxdSp3UktekGpPyxsM25Hcnteyw6JF3rLuipp4d+9p8OoeXWv6tcHLpFb13VA3YCFXX82Vr6t5G33Su3viV0T75AYl1Ihx65qOt6VMq8o9z6r+cLHFBk/uuDl+5i8K48guj2cIGmlu78hSWZ2l6SLJHWZ/JaKqdXBOJWteIYov5anYpcpRN7bKWIbfVm59jOkmnw/6oV8BpqZFLv4LMVOzf9K6M5EDz9A9vdXKHHL3dLOVvcCzt3wv2bm5bIhg7pZAwKTTku72yZaQ5Tnl48UDy1p7TdTluoP+71cVG4fPXCibNbHlLj599LWHS0TEklJ+bckF6uQY1c1He9KmleUe59lMoUv248FkfzWSXqn1evVkk5sP5OZzZQ0U5ImTpxY8Ma66w/kO3er6akbC143gALFovrRIY/pb5vvkh7oefa6AdnWxO7O5Iw/Yoi+/sIZGrdnSJvtqCZebLRVIYg6tad+mGe/e6CuUX1B60ZOJKL4Fefpmvp/0jU9zNp8VkTq/n8z6rBBuuGF0ztcRAcgP0H0+e3se2yHczruPtvd6929fsyYMZ0sAqBfGjlMNZ//O/1tzOpAV/veoB361rEL9MaQTZIkGztKNZ+5TEbyK4k6tV8YNljxay9TtP0ty4q0acAufXvao1oyak2g6wXCIoiW39WS9mv1eoKkynwiBw5Q7Xe/UJFNXzXvA3nPe8eMhT3PVOR2itlGocoRW9Xs59ydC656+PT8tnPeY7mvmaaPP3yqJLX04TVv6f6b++0m/e6CRbmZmvsEt/pO2lk/4UL6/5tJA2uzd2Iowa1Ht9Y06TvH/FnXvHaczrr2c7LBA4PfSIj1dGW9J1NSABdT5f0+n/GX4rZTwOe2pMtc+nTJ7lKyM57Qv019XLWpmH577mO9i6sCx4dCVNMxpVx6vc/yvCi52gSR/D4r6SAzmyzpXUlXSPpYAOvtNYuYVKGD4854oueZcoo5gN952aKCly21fPdBMeUv134uR1my28mvn54NGrB3eHcsv2TEYtVxJ8OmWEq/OOwZnTOKU7zlZvGYFC/+fZT3+7zoz1PvP7clXaYMt+driqX2xtaXjw+FqKZjSrmU69jV3xVdq7l7ysxmSfqTpKik29x9edGRAQAAAAELpGnI3R+S9FAQ6+qvuDE3AABA31cd50URGnzJAAAAxSjvMxEBAACACqLlFwBQUpyx6T32GVA6JL8AAKDP4QsASoXkFwAAoArwhSE/9PkFAABAaNDyC3SCb88AAFQnkl8EhoQRQH9DvQWED8kvAAC9QMIM9G/0+QUAAEBo0PILVFDYW5DCXn4AQPnR8gsAAIDQIPkFAABAaNDtAQAAhBJdr8KpqOTXzP5d0oclJSS9LunT7r4liMAAdI7KGgCAwhXb7WGBpCPdfaqkVyXdUHxIAAAAQGkUlfy6+3x3T+VeLpI0ofiQAAAAgNII8oK3ayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeL3hz97O6m25mV0u6QNIH3d2DCgxA/5LvhXiz5m8tcSQIKy4GBZCPYu/2MEPS1ySd7u67ggkJAAAAKI1i+/zeJGmopAVmttTMfhlATAAAAEBJFNXy6+4HBhUIAAAAUGo83hgAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2rxEPZzKxR0lsl3sxoSRtKvI2+LOzll9gHlL/vl3+Du88odiXUqWVB+cNdfol90NfLn3d9WpHktxzMrMHd6ysdR6WEvfwS+4Dyh7v8QQv7/qT84S6/xD6opvLT7QEAAAChQfILAACA0Kjm5Hd2pQOosLCXX2IfUH4EKez7k/Ij7PugaspftX1+AQAAgPaqueUXAAAAaIPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEKD5BcAAAChQfILAACA0CD5BQAAQGjEKrHRGTNm+Lx58yqxaXRh1vytkqSbzhle4UiAULEgVkKdGl7U3cBeedenFWn53bBhQyU2CwBViToVAPJHtwcAAACERkW6PQBAoS554Ni85rv3wsUljgQA0B8V3fJrZvuZ2UIzW2Fmy83s+iACAwAAAIIWRMtvStJX3H2xmQ2V9JyZLXD3lwJYNwAAABCYopNfd18raW1ueLuZrZBUJ4nkF31CvqfJJU6VA+gbelNv1Q1YWMJIgOoT6AVvZjZJ0jRJzwS5XgAAACAIgV3wZmZDJM2V9EV339bJ9JmSZkrSxIkTg9osAIQSdSqA9rggOD+BJL9mFlc28Z3j7vd0No+7z5Y0W5Lq6+s9iO0CQFhRp6I/ISlDXxLE3R5M0q2SVrj7D4sPCQAAACiNIPr8Tpd0laQzzWxp7uf8ANYLAAAABCqIuz08qYCeTw8AAACUEo83BgAAQGiQ/AIAACA0ArvVGQAAQH/CQ5DCiZZfAAAAhAYtv5C7a8qmNRqS2C1PTpXFeVsAAIDqRJYTcp5KK/n7P+m6Z5dJkhJrFit+7UcU2WdEhSMDEGY8FKEHLp2ybpLqN9Rp7egX5amTZbFopaMC+gWS3xDzXXuUvP0+ZVa+3TJu/SYlfnKnaq65VJFJ+1YwOgBAZ2rSUX3u5RP0/sb9syM2/FWJX76tmk9dLBsyqLLBAf0AyW9IZTZtVfLmu+XrNnacuGOXEj+/S/GPf0jRow8pWQxcaAAA9WFJZwAAHilJREFUvbPPnkH6yrJTNHnHqDbj/Y3VavrRr1VzzSWK1I2rUHRA/8AFbyGUeWutEj++o/PEt1kqpeSv7lfqz8/I3csXHACgU4dsGaPvPXdOh8R3r83blPjpHKWXvlzewIB+huQ3ZNIvvqbEz38r7djVZvyqEeO0cPLRHeZP/fExpe6eL09nyhUiAKCdD645QP/y/Ac0PDmgzfimSKrtjMmUkr9+QMmHnpBnaLgAOkPyGxLurtRjDUrefq+UbFtZRo45VP91/AV68JCTFLviPCnS9m2Rfvp5JW+dK9/TVM6QASD0opmIrnn1OH321eMV87Z187OjV+sfTr5fT0w8ssNy6UeeVvK/76HeBjpBn98Q8ExGqfv+rPSTHfvNRs88UbHzT1PqkW2SpNgJR8lGDMsmyXsSe+fLvLxKiRt/o5prPyIbMbRssQNAWA1N1OpLy6frsK1jO0ybu/8yzZ20TG7S/YdP1wdOmqDU3PlSq7N0meWvK/GTOxW/5hJFxnTRVQKhF8Y7q9DyW+W8KaHkf9/bMfGNmGKXn6v4BafLItZmUvTg/VXzj5+QRg5ru661jWr68R3KvLuu1GEDQKjtv32EvvfcOR0S3z2RpH50xJO6e3I28W0WO2mqaq67Uho6uM38vm6jEj++Q+lXVpUjbKBfoOW3ivm2HUrcMle+ul2yWluj+NUXKXro5C6XjYwfrdrrP6HErffI33mvZcK2HUrc+BvFP3mhoocfUKLIERbc8QPoKL30Zf2fJWepNtP2EL1+wA7955FP6O0hWztdLjK5TrVfvEqJ/763bb2/u0nJ2XfLP3yGoqfXy8w6XR4IC5LfKpVZ26jELXOlzdvaThgxVDWfvUyRfTueRmvPhg1RzXVXKDnnj8osW9kyIZFU8tZ75Jeepdj0aQFHjv6KRDYcwniKtFw840rNe1LpR55WbbvD8/IR6/STw5/S9ppEF0tn2chhqpn1MSV/N0+ZxStardyVemChMmvWK375uf3iSZ6811Aqgbz7zWyGpJ9Iikq6xd2/H8R6e8tTaWVee6sSm9Z3F83Kc07TN0+6sePoQm8n1sk3eN++U6n7Hm3TZ1eSrG6saj57mWx4/n12rbZG8U9drNQDC5V+/Lk28abmLpCv26jIwft3HksPr4/eOD7vONIr3sh73vbKtZ2qkMlIqbSUSslTaSmdzr3O/nhumtKZlr8mKRbTJ946RqlIRinLKBnJKBVJK2nZv6lIRknLKJkbbt7Phfxv8l2m3P/LyAH7yWriZd1mqfnmbcq8t6HNuEL2f7714zdPuqmo7fR36b8uUWb56x3G/6nuVd1xwBKlI/kdJ6wmrvjHL1B637FKPfiY1GqxTMNyJdZtVOzc6Z0ePwKVyWQvsE6mdNa7ByieiaomE1NNJqradFQ1mWh2XG64NhNT4t3/keIxqSamz689UYloSolIOvsTTbcMR9J7p4X++JDJSImkPJHUuasP0oB0TLXpWMvfTLvX6Zialt8s1cSl2hp9bctp2hNNqSmS1p5oUk3RdPZ17ic7LVXW8tuAWkUm15Vu/cXew9XMopJelXS2pNWSnpV0pbu/1NUy9fX13tDQUND2uvsmOCRZo5ufurSg9Va7yOFTFL/qQlltTafTZ83Pnka76ZzhXa4j9cTibFLNfX+BDmr++VpFRo/s7WKBZB/F1KndST21RKm5CwJfL/KTsrRuO+g5Ldy3+6SjbsBC3XTO8E6Pj0dvHK8vvPR+DU53XvcDfZFNGKfaL1/d68XynTGIC95OkLTS3d9w94SkuyRdFMB6EZDo+49R/NOXdpn45it26rGKX3NJ9tsiAKBktsR36zvH/LnHxLcnz+/znr553AK9O3BbzzMDIRFEt4c6Se+0er1a0ontZzKzmZJmStLEiRML3lh3fXt85241PdVJl4KwMin24TN0+dbPSw/2PHvdgIV7h7trYZ901Ej904unaWRiYBBRAihAEHVqT30qz373QF2j+oLWjcLZhHEa9+lL9G8jv9XjvM1n7aQejo8XNSl55x+U6aun7oEyCiL57ayZucN5cXefLWm2lD1FF8B2O4pGFDmk7R0Mlqz/a96LTxv7/oKX6ZXcHlu8/qk2o7vaKceNPWXv8HPrn2y/mk4dW3eGoicfnb2jwwO9D7E7bw7drG8cN18XvX2Yzhtxvpojf/a9x7NxeXN8bSM0SeZW2D7LKfn/ply6eA905dix03u/jVZvqHz327T3TZeiMT2xbkGu726uv24krZRl2gwnIxmlLaPrjv6XbJ/gdOt+wa37C6c6jitzz5m8y1/Ee6bcFxCVo07dWLtLz49cK6mPf55yyvF/Lul2oqbIwZMUPenowPuP28BaxT9zqdINy7MXMLd72FFv5F3+8dOz/XfjseznIx5vNdzqdU3LsEx7+wl7MiUlkm1fJ5NSIiVPpXLjk1K65e1fjvdAKfOK1nHlvcy46VJtPPueyfXjtZq4VJt9bTU1ueGalnmiESmRkJqyfYWVSMqbEtn93Twu9zo7PSGlWu4hXdLyjH2/bPSIvOYtVBB9fk+W9G13Pzf3+gZJcvd/7WqZUvVP60whV6CX66r1Qq5kLeUyzf3Gyhlbb1XbHQX66tXM7OeyqWifX/7PfXs7+cjneo2g9aXyt9dXj0N99ZhaqD5anrzr0yCaKp6VdJCZTZb0rqQrJH0sgPVWTH+o5AEAANB7RSe/7p4ys1mS/qTsrc5uc/flRUeGfqccXxr4YgIAQP/Tl47fgXRSc/eHJD0UxLqC1pd2NgAA6N/KlVeQv5RO33/EC9rgwwAAQP/CsbtvIfkFgJDigAwgjEh+AQBVgdPRAPIRxBPeAAAAgH6Bll8AVY+WOgBAM5JfAACQN75Mor8j+QUAAEDe+vsXIJJfoIL6ewUCAH0JdSryQfJbQXxIAQAAyovkF50iMQcAANWIW50BAAAgNGj5BdABLf8AgGpFyy8AAABCg+QXAAAAoVFU8mtm/25mL5vZC2Z2r5mNCCowAAAAIGjF9vldIOkGd0+Z2Q8k3SDpa8WHhSDl039z1vytZYgEAACgsopq+XX3+e6eyr1cJGlC8SEBAAAApRFkn99rJD3c1UQzm2lmDWbW0NjYGOBmASB8qFMBoDA9Jr9m9oiZLevk56JW83xDUkrSnK7W4+6z3b3e3evHjBkTTPQAEFLUqQBQmB77/Lr7Wd1NN7OrJV0g6YPu7kEFBgAAAAStqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqPOENAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNq8RD2cysUdJbJd7MaEkbSryNvizs5ZfYB5S/75d/g7vPKHYl1KllQfnDXX6JfdDXy593fVqR5LcczKzB3esrHUelhL38EvuA8oe7/EEL+/6k/OEuv8Q+qKby0+0BAAAAoUHyCwAAgNCo5uR3dqUDqLCwl19iH1B+BCns+5PyI+z7oGrKX7V9fgEAAID2qrnlFwAAAGiD5BcAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBoxCqx0RkzZvi8efMqselQmDV/qyTppnOGVzgSAD2wIFZCnQrkj2Nk1cq7Pq1Iy++GDRsqsVkAqErUqQCQP7o9AAAAIDRIfgEAABAaRSe/ZrafmS00sxVmttzMrg8iMAAAACBoQVzwlpL0FXdfbGZDJT1nZgvc/aUA1g0AAAAEpuiWX3df6+6Lc8PbJa2QVFfsegEAAICgBdrn18wmSZom6ZlOps00swYza2hsbAxyswAQOtSpAFCYwJJfMxsiaa6kL7r7tvbT3X22u9e7e/2YMWOC2iwAhBJ1KgAUJpDk18ziyia+c9z9niDWCQAAAAQtiLs9mKRbJa1w9x8WHxIAAABQGkG0/E6XdJWkM81sae7n/ADWCwAAAASq6FudufuTCuj59Pm45IFj85rv3gsXlzgSAAAA9Dc84Q0AAAChEcRDLgAAACou37PDdQMWljgS9GUkvwAAoKTosoi+hG4PAAAACA1afquMb9qqa557WGN3blEqOk3RD5wgi5TtekQAAIA+jeS3imRWv6fEzXN1+PadkqTUg48ps2a94leeJ4vxrwbQf3CaHECpkBFVifSKN5T81f1SItlmfGbJCiW27VDNpy+RDRqQ9/o48KCa8H4GADQj+a0CqUXPK3X3fCnjnU73199R4sY5qrn2I7JRw8scHdC1fJNSicQUQPEiGdOU7aM0ILNdEsfDQvX3BgWS337M3ZWa96TSC57uMG1HfICGJPe0zLtuo5p+cqdqrr1MkQnjyxkmAAAVd+SmcfrUymNVt2u4pN8oseNwxS88QzZsSKVDQ5mR/PZTnkor+bt5yjQsbzvBpHsPna4l7ztQ3131iHzVuy3Ttu9U4qbfKn71RYoeNqW8AQMAUAH77Bmkq1ZO04kb9mszPrP4JTUtX6nYudMVPfVYWTRaoQhRbiS//ZDvblLy9vuUee2tthNiMcWv+rCeWjtWklTz+Y8q+ZsHlXn+lZZ5Ekklb50r/8g5ip10dBmjBoC+p7+fvkXXYpmIPvTOobr4rcM1INNFutOUUOqBhUo/84Jil56l6EH7lzfIPiCMnwGS337Gt2xX4ua75Wsb204YPFA1n7lMkUn7Smu3SpIsHlP8qguVGrFQ6ccaWubNuFK/+5N88zbFZpwiM26F1p+EsaICUP2GJmrk23ZIQwcXfVw6euN4Xb3yWL1v97C85vd1G5X8xf8ofcyhil/4AdmIoUVtH30byW8/klmzXomb75a27mgz3kaPUPzayxUZM7LDMhYxxS86UzZyuFL3Pyq1uiYuveBp+eZtiv/dDFms+NM9JGUAgHzF01EdvmWMjto8Xkdvep8m7Bqupr/+XDZquCKHTlbk4EmKHLS/bGBt3uscvXuwPrlymo7fOKHT6SuHbtSvD1ysaXu+rUtWNUh7mtpMzyx9WU0vva7YOe9X9LT6QI6N6HtIfvuJ9CtvKnn7fVJTos14239f1XzmUtmQQd0uHzvtONmIoUre+Ucpldo7PtOwXMmtOxT/1MW9qmAAAOgVl/bbOVxTN43X1M3v06FbxqjGOyaXvmmr0n9dqvRfl0oRk+1fp+ihkxQ5eLJsv3GySMeH03oypfSfn9F/Pnueajrp4rA91qS7pjyvhe97Q27SrnFH6oorjlHqj48p/eyytjMnktnxf3tRsUvOUvSQSUHtAfQRgSS/ZjZD0k8kRSXd4u7fD2K9yEo/u0zJ/5knZTJtxkeOOkjxj18gq4nntZ7o1INl131UiVvvkXbu3js+89pbStz0m+yt0DjVAwDdqqazXKUui+/Ypcyrb+rzK07U1M3jNTIxsHcryLh81WqlVq2WHn5SGjQg2yJ8yCRFD5ksGzFU6eUrlbrvz/KNW1TTLq3JyPXoviv1P5Nf1M54u8ajoYMVv/J8RU8+Wsm5C+Tvrm8b+/pNSv7X75SeenDuDGp+XSjQ9xWd/JpZVNLPJJ0tabWkZ83sAXd/qdh199aAVEyffu04SVJix4Ol36B7rhtB7q97y7CUu+9u63la9Tkwk2SS5Yab+zdZbpxy4xJJZZa91mHT0VOPU+yiD3T6Dbg7kUl1qvnHjys5+275xi0tRVnbqKYf36HIwdnO/n//zol5ra/1fi5kmbx4J/cv7jCq83sct9VJH7L2o/pB/+dC9vNf3vljXsucsd8FvQvGc+/rTPPfTKthlzz3OpORt54v957/5qYzlTHP/WSUUXbYTXuHm3+K+Uz3dp/FzjxRkfGjC95etUi/+qbS7e8oU0oZlzJpKZ3RV9ecqlgmoqhHFPOIohlrGXZTNDdtz9JfZOvBaESKRrN/Iy3D1jyueXokIkVa6tw+VddVQCnrbV+3Uf7uOsml0zW5x0UyyigSjUvpdNcz7dqjzNKXlVn6slKSNGKotGV7p7O+OmyD/vug5/Tm0M3dbjcyqU41X/qk0k8/r9RDT0i797SZnnnhVTWteEORIw/MvoeqTN7vge2544hL/7D6pDbTLHcwbXMEdSmx5QFZbY3iH50RQKTBMe8ssejNCsxOlvRtdz839/oGSXL3f+1qmfr6em9oaOhqcre6+5Y6JFmjm5+6tKD19iexCz+g6On1XV4QMGt+9oK3m87p+gbevmOXErfMlb+9tiQxAv1Z/PMfVfTgslz1Hci3rWLq1O6knlqi1NwFga8XaNZYu1PPj1qrF0a9p+Uj1unOC55Q5o3Vyry8SplXVsnXb+r1OrfG9+i3U57X4+NXybv4hNUNWNjpMdJ37FLqoceVfuaF/NpU0LPBAzXgu18ox5byrk+D6PZQJ+mdVq9XS+rwNcLMZkqaKUkTJ04seGPdnXrxnbvV9NSNBa+7z4tF9eODH9cz2+6S/tD9rHUDFu4d7uoLQ83+Uc3aeXKXFwYA6LuCqFN7OuV99rsH6hrVF7RuoFM1cUUO3E+RQyYrcshkTRgzUvuZqfU5p+hhUxQ9bIoueeBY7TNlUK6P8HgduXm8hqRqul63maLTp2nseafoiwMH6ItdzNbcQCR18RkYIB0wbZQ+9dpxOnD7PgUVE60U2chaCkEkv51l2h1PSrvPljRbyrZSBLDdcBk0QDXXXKpnls0JbJWJaFo/OvIpfXLlNM149+DA1gug9KhT0V9Y3dhcsjtJkcl1slj+qcfGAbu0cN83tHDfN2RuOmDbKB29ebyO2jReB23bRxFlu/7Z5AmKX3qWInVjA4n59WGb9L+PXaAPrJ2iK944WkNTXBBeTYJIfldLav3YlAmS1gSw3t6rjSt+5flFr+YnS/533vNef+x3JUk/WvIvcnm2669lj0PZHr/Z/oue+z7gJt1w/A8ld/3g2a/KPNtXpvkbhHl22GTKrsb0xRP+nyIHTsze0aHdRanFcnP96qDFemLcm/q3Q34c7Mp7UMh+/vHib7Ya2/Z43/701peO/b+SpB8t/pfcvmzPOry6ftp38o6pvXzL03obhSxTDr3+30RM/7H4hmz/3L39dr1Dv93m198/5b+zF7JkMi39g1sPe+vXuX7DvYytmH0WGU9rjyQtG7lOPz90kaSW/VnS9/mx323VZ7dVv91WfXjb9N2NRmRm8nRGymT0pUc/oqhH9vYHjrm1em25/sIRfemY7/ZyT5Rfrz6DRfxvCpH3dk76fjbZHTq4qO01c3OtHL5RK4dv1NxJyzUoGdfB20brm6f9UjZlQuD3rHeT/rzvG/rr2Ld1xJaxuuGowq/lL9fxoeTvgc72sbUbaD2LWZ/sJx1E8vuspIPMbLKkdyVdIeljAay31ywWU/T4I4tezxNr38x73i/XHyFJemrNWz3M2SJ61EGSpGffXJ3X/F855tC8112oN4ZtCmTf9UYh+/nJNfkv89Xjcv+bd/P/33y5iH2Qb3lab6OQZcqhkP/NotXv9DBni8jkwrva9NV9Vo3WDtqutYOyFxM178+Svs9z76Xeaj7Wvj1ka7fzNftqP3hvfPn4e3q9TLk+G3lvZ2ppzyjuiie1dJ+1ihywX88zd6Fcd+Qo1/GB+jE/RSe/7p4ys1mS/qTsrc5uc/cyXh4MAChEf7gVF/oe3jflwX4unUDu8+vuD0l6KIh19QW84QCgc9SPfRf/GyA/POENFUNF3Xf3QV+NCwD6I+rUvoXkFwgIlVt5sJ8BAMUg+Q0BkgUAAICs3j0bFwAAAOjHaPmtIFpky4P9DAD9D3U3SoXkFwAAoArwhSE/dHsAAABAaNDyCwAoKVqjAPQltPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFR1K3OzOzfJX1YUkLS65I+7e5bgggM6Ay3TALQFeoHAPko9j6/CyTd4O4pM/uBpBskfa34sPofKl0AAIC+r6jk193nt3q5SNJHigsHQH/FF0AAQH8QZJ/fayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeuz24+1ndTTezqyVdIOmD7u5BBQYAAAAErdi7PcxQ9gK30919VzAhAQAAAKVR7N0ebpJUK2mBmUnSInf/fNFRoUv5XFQ0a/7WMkQCAADQ/xR7t4cDgwoEAAAAKDWe8AYAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAELDKvFEYjNrlPRWiTczWtKGEm+jLwt7+SX2AeXv++Xf4O4zil0JdWpZUP5wl19iH/T18uddn1Yk+S0HM2tw9/pKx1EpYS+/xD6g/OEuf9DCvj8pf7jLL7EPqqn8dHsAAABAaJD8AgAAIDSqOfmdXekAKizs5ZfYB5QfQQr7/qT8CPs+qJryV22fXwAAAKC9am75BQAAANog+QUAAEBoVF3ya2YzzOwVM1tpZl+vdDyVYGZvmtmLZrbUzBoqHU+pmdltZrbezJa1GjfKzBaY2Wu5vyMrGWOpdbEPvm1m7+beB0vN7PxKxlhKZrafmS00sxVmttzMrs+ND9X7oBSoU6lTc+NC81miPq3++rSqkl8zi0r6maTzJB0u6UozO7yyUVXMB9z9mGq5J18PbpfU/sbWX5f0qLsfJOnR3Otqdrs67gNJ+lHufXCMuz9U5pjKKSXpK+5+mKSTJP1D7rMftvdBoKhT26BODc9n6XZRn1Z1fVpVya+kEyStdPc33D0h6S5JF1U4JpSYuz8uaVO70RdJ+lVu+FeSLi5rUGXWxT4IDXdf6+6Lc8PbJa2QVKeQvQ9KgDo1hMJep1KfVn99Wm3Jb52kd1q9Xp0bFzYuab6ZPWdmMysdTIWMc/e1UvaDLGlsheOplFlm9kLuNF6/PUXVG2Y2SdI0Sc+I90GxqFOzqFP5LEnUp1XzHqi25Nc6GRfGe7lNd/djlT1V+Q9mdlqlA0JF/ELSAZKOkbRW0n9WNpzSM7MhkuZK+qK7b6t0PFWAOjWLOhXUp1Wk2pLf1ZL2a/V6gqQ1FYqlYtx9Te7vekn3KnvqMmzWmdn7JCn3d32F4yk7d1/n7ml3z0i6WVX+PjCzuLIV9Rx3vyc3OvTvgyJRp4o6NSfUnyXqU0lV9B6otuT3WUkHmdlkM6uRdIWkByocU1mZ2WAzG9o8LOkcScu6X6oqPSDp6tzw1ZLur2AsFdFcSeVcoip+H5iZSbpV0gp3/2GrSaF/HxSJOpU6tVmoP0vUp5Kq6D1QdU94y91+5MeSopJuc/fvVTiksjKzKcq2TEhSTNJvqn0fmNlvJZ0habSkdZK+Jek+Sb+TNFHS25Iud/eqvYChi31whrKn6FzSm5I+19xfq9qY2SmSnpD0oqRMbvQ/K9tPLTTvg1KgTqVOVcjqVOrT6q9Pqy75BQAAALpSbd0eAAAAgC6R/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS9Cx8w+b2afzA1/ysz2bTXtFjM7vHLRAUD/QX2K/ohbnSHUzOwvkr7q7g2VjgUA+jPqU/QXtPyiXzGzSWb2spn9ysxeMLO7zWyQmX3QzJaY2YtmdpuZ1ebm/76ZvZSb9z9y475tZl81s49Iqpc0x8yWmtlAM/uLmdXn5rsyt75lZvaDVjHsMLPvmdnzZrbIzMZVYl8AQDGoTxFWJL/ojw6RNNvdp0raJunLkm6X9FF3P0rZpzD9vZmNUvYxlEfk5v2/rVfi7ndLapD0cXc/xt13N0/Lnbr7gaQzlX2qz/FmdnFu8mBJi9z9aEmPS7q2ZCUFgNKiPkXokPyiP3rH3Z/KDd8p6YOSVrn7q7lxv5J0mrIV+R5Jt5jZpZJ29WIbx0v6i7s3untK0pzcOiUpIemPueHnJE0qtCAAUGHUpwgdkl/0R3l1VM9VsidImivpYknzerEN62Za0ls6y6eVbRkBgP6I+hShQ/KL/miimZ2cG75S0iOSJpnZgblxV0l6zMyGSBru7g9J+qKyp9va2y5paCfjn5F0upmNNrNobjuPBVkIAOgDqE8ROnzDQn+0QtLVZvZfkl6TdL2kRZJ+b2YxSc9K+qWkUZLuN7MByrY8fKmTdd0u6ZdmtltS8wFA7r7WzG6QtDC37EPufn/pigQAFUF9itDhVmfoV8xskqQ/uvuRFQ4FAPo16lOEFd0eAAAAEBq0/AIAACA0aPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhMb/B0Hm1BpCSN+pAAAAAElFTkSuQmCC\\n\",\n \"text/plain\": [\n \"
\"\n ]\n },\n \"metadata\": {\n \"needs_background\": \"light\"\n },\n \"output_type\": \"display_data\"\n }\n ],\n \"source\": [\n \"query, context, target, lengths, target_indices = valid_batches[0]\\n\",\n \"\\n\",\n \"with torch.no_grad():\\n\",\n \" weight, output = net(query, context, lengths=lengths, return_weight=True)\\n\",\n \"\\n\",\n \"context = context.numpy()\\n\",\n \"weight = weight.data.numpy()\\n\",\n \"\\n\",\n \"colors = sns.color_palette('husl', 3)\\n\",\n \"\\n\",\n \"fig, axs = plt.subplots(batch_size, 2, figsize=(10, 10), sharex=True, sharey=True)\\n\",\n \"sns.despine(fig=fig)\\n\",\n \"axs_min, axs_max = axs[:,0], axs[:,1]\\n\",\n \"\\n\",\n \"for i, (i_min, i_max) in enumerate(target_indices):\\n\",\n \" length = lengths[i]\\n\",\n \" w_min, w_max = weight[i]\\n\",\n \" c_min = [context[i,j,Data.min_position] for j in range(length)]\\n\",\n \" c_max = [context[i,j,Data.max_position] for j in range(length)]\\n\",\n \"\\n\",\n \" axs_min[i].axvline(i_min, zorder=1, color=colors[2], label='min value')\\n\",\n \" axs_min[i].bar(np.arange(length) - 0.4, c_min, zorder=2, color=colors[1], lw=0, label='values')\\n\",\n \" axs_min[i].plot(np.arange(length), w_min[:length], zorder=3, color=colors[0], lw=4, label='attention')\\n\",\n \"\\n\",\n \" axs_max[i].axvline(i_max, zorder=1, color=colors[2], label='max value')\\n\",\n \" axs_max[i].bar(np.arange(length) - 0.4, c_max, zorder=2, color=colors[1], lw=0, label='values')\\n\",\n \" axs_max[i].plot(np.arange(length), w_max[:length], zorder=3, color=colors[0], lw=4, label='attention')\\n\",\n \"\\n\",\n \"axs_min[0].set_title('Minimum')\\n\",\n \"axs_max[0].set_title('Maximum')\\n\",\n \"axs_max[0].legend(loc='best') \\n\",\n \"axs_min[0].legend(loc='best')\\n\",\n \"axs_max[0].legend(loc='best')\\n\",\n \"axs_min[-1].set_xlabel('position')\\n\",\n \"axs_max[-1].set_xlabel('position')\\n\",\n \"plt.tight_layout()\"\n ]\n },\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": []\n }\n ],\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"language\": \"python\",\n \"name\": \"python3\"\n },\n \"language_info\": {\n \"codemirror_mode\": {\n \"name\": \"ipython\",\n \"version\": 3\n },\n \"file_extension\": \".py\",\n \"mimetype\": \"text/x-python\",\n \"name\": \"python\",\n \"nbconvert_exporter\": \"python\",\n \"pygments_lexer\": \"ipython3\",\n \"version\": \"3.7.1\"\n }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 1\n}\n" }, { "path": "setup.py", "content": "from distutils.core import setup\n\nsetup(\n name='attention',\n version='0.1.0',\n author='tllake',\n author_email='thom.l.lake@gmail.com',\n packages=['attention'],\n description='An attention function for PyTorch.',\n long_description=open('README.md').read())" }, { "path": "test/test_attention.py", "content": "import numpy as np\nimport pytest\n\nimport torch\nfrom attention import attention\n\n\ndef test_apply_mask_3d():\n batch_size, m, n = 3, 4, 5\n sizes = [4, 3, 2]\n values = torch.randn(batch_size, m, n)\n masked = attention.mask3d(values, sizes=sizes).data\n assert values.size() == masked.size() == (batch_size, m, n)\n for i in range(batch_size):\n for j in range(m):\n for k in range(n):\n if j < sizes[i]:\n assert masked[i,j,k] == values[i,j,k]\n else:\n assert masked[i,j,k] == 0\n\n\n@pytest.mark.parametrize('v_mask, v_unmask', [(0, 1), (float('-inf'), 0)])\ndef test_fill_context_mask(v_mask, v_unmask):\n batch_size, n_q, n_c = 3, 4, 5\n query_sizes = [4, 3, 2]\n context_sizes = [3, 2, 5]\n mask = torch.randn(batch_size, n_q, n_c)\n mask = attention.fill_context_mask(\n mask, sizes=context_sizes,\n v_mask=v_mask, v_unmask=v_unmask)\n\n for i in range(batch_size):\n for j in range(n_q):\n for k in range(n_c):\n if k < context_sizes[i]:\n assert mask[i,j,k] == v_unmask\n else:\n assert mask[i,j,k] == v_mask\n\n\ndef test_dot():\n batch_size, n_q, n_c, d = 31, 18, 15, 22\n q = np.random.normal(0, 1, (batch_size, n_q, d))\n c = np.random.normal(0, 1, (batch_size, n_c, d))\n\n s = attention.dot(torch.from_numpy(q),\n torch.from_numpy(c)\n )\n s = s.data.numpy()\n\n assert s.shape == (batch_size, n_q, n_c)\n\n for i in range(batch_size):\n for j in range(n_q):\n for k in range(n_c):\n assert np.allclose(np.dot(q[i,j], c[i,k]), s[i,j,k])\n\n\n@pytest.mark.parametrize(\n 'batch_size,n_q,n_c,d', [\n (1, 1, 6, 11),\n (20, 1, 10, 3),\n (3, 10, 15, 5)])\ndef test_attention(batch_size, n_q, n_c, d):\n q = np.random.normal(0, 1, (batch_size, n_q, d))\n c = np.random.normal(0, 1, (batch_size, n_c, d))\n\n w_out, z_out = attention.attend(torch.from_numpy(q),\n torch.from_numpy(c),\n return_weight=True\n )\n w_out = w_out.data.numpy()\n z_out = z_out.data.numpy()\n\n assert w_out.shape == (batch_size, n_q, n_c)\n assert z_out.shape == (batch_size, n_q, d)\n\n for i in range(batch_size):\n for j in range(n_q):\n s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)]\n max_s = max(s)\n exp_s = [np.exp(si - max_s) for si in s]\n sum_exp_s = sum(exp_s)\n\n w_ref = [ei / sum_exp_s for ei in exp_s]\n assert np.allclose(w_ref, w_out[i,j])\n\n z_ref = sum(w_ref[k] * c[i,k] for k in range(n_c))\n assert np.allclose(z_ref, z_out[i,j])\n\n\n@pytest.mark.parametrize(\n 'batch_size,n_q,n_c,d,p', [\n (1, 1, 6, 11, 5),\n (20, 1, 10, 3, 14),\n (3, 10, 15, 5, 9)])\ndef test_attention_values(batch_size, n_q, n_c, d, p):\n q = np.random.normal(0, 1, (batch_size, n_q, d))\n c = np.random.normal(0, 1, (batch_size, n_c, d))\n v = np.random.normal(0, 1, (batch_size, n_c, p))\n\n w_out, z_out = attention.attend(torch.from_numpy(q),\n torch.from_numpy(c),\n value=torch.from_numpy(v),\n return_weight=True\n )\n w_out = w_out.data.numpy()\n z_out = z_out.data.numpy()\n\n assert w_out.shape == (batch_size, n_q, n_c)\n assert z_out.shape == (batch_size, n_q, p)\n\n for i in range(batch_size):\n for j in range(n_q):\n s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)]\n max_s = max(s)\n exp_s = [np.exp(si - max_s) for si in s]\n sum_exp_s = sum(exp_s)\n\n w_ref = [ei / sum_exp_s for ei in exp_s]\n assert np.allclose(w_ref, w_out[i,j])\n\n z_ref = sum(w_ref[k] * v[i,k] for k in range(n_c))\n assert np.allclose(z_ref, z_out[i,j])\n\n\n@pytest.mark.parametrize(\n 'batch_size,n_q,n_c,d,context_sizes', [\n (1, 1, 6, 11, [3]),\n (4, 1, 10, 3, [7, 5, 10, 9])])\ndef test_attention_masked(batch_size, n_q, n_c, d, context_sizes):\n q = np.random.normal(0, 1, (batch_size, n_q, d))\n c = np.random.normal(0, 1, (batch_size, n_c, d))\n\n w_out, z_out = attention.attend(torch.from_numpy(q),\n torch.from_numpy(c),\n context_sizes=context_sizes,\n return_weight=True\n )\n w_out = w_out.data.numpy()\n z_out = z_out.data.numpy()\n\n assert w_out.shape == (batch_size, n_q, n_c)\n assert z_out.shape == (batch_size, n_q, d)\n\n w_checked = np.zeros((batch_size, n_q, n_c), dtype=int)\n z_checked = np.zeros((batch_size, n_q, d), dtype=int)\n\n for i in range(batch_size):\n for j in range(n_q):\n n = context_sizes[i] if context_sizes is not None else n_c\n\n s = [np.dot(q[i,j], c[i,k]) for k in range(n)]\n max_s = max(s)\n exp_s = [np.exp(sk - max_s) for sk in s]\n sum_exp_s = sum(exp_s)\n\n w_ref = [ek / sum_exp_s for ek in exp_s]\n for k in range(n_c):\n if k < n:\n assert np.allclose(w_ref[k], w_out[i,j,k])\n w_checked[i,j,k] = 1\n else:\n assert np.allclose(0, w_out[i,j,k])\n w_checked[i,j,k] = 1\n\n z_ref = sum(w_ref[k] * c[i,k] for k in range(n))\n for k in range(d):\n assert np.allclose(z_ref[k], z_out[i,j,k])\n z_checked[i,j,k] = 1\n\n assert np.all(w_checked == 1)\n assert np.all(z_checked == 1)\n" } ]