Full Code of thomlake/pytorch-attention for AI

Repository: thomlake/pytorch-attention
Branch: master
Commit: 666be8bb9ba1
Files: 7
Total size: 76.3 KB

Directory structure:
gitextract_e3e3pn4t/

├── LICENSE
├── README.md
├── attention/
│   ├── __init__.py
│   └── attention.py
├── examples/
│   └── Pointer-Network-Argmin-Argmax.ipynb
├── setup.py
└── test/
    └── test_attention.py

================================================
FILE CONTENTS
================================================

================================================
FILE: LICENSE
================================================
BSD 2-Clause License

Copyright (c) 2017, thom lake
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


================================================
FILE: README.md
================================================
```python
def attend(query, context, value=None, score='dot', normalize='softmax',
           context_sizes=None, context_mask=None, return_weight=False
           ):
    """Attend to value (or context) by scoring each query and context.

    Args
    ----
    query: Variable of size (B, M, D1)
        Batch of M query vectors.
    context: Variable of size (B, N, D2)
        Batch of N context vectors.
    value: Variable of size (B, N, P), default=None
        If given, the output vectors will be weighted
        combinations of the value vectors.
        Otherwise, the context vectors will be used.
    score: str or callable, default='dot'
        If score == 'dot', scores are computed
        as the dot product between context and
        query vectors. This Requires D1 == D2.
        Otherwise, score should be a callable:
             query    context     score
            (B,M,D1) (B,N,D2) -> (B,M,N)
    normalize: str, default='softmax'
        One of 'softmax', 'sigmoid', or 'identity'.
        Name of function used to map scores to weights.
    context_mask: Tensor of (B, M, N), default=None
        A Tensor used to mask context. Masked
        and unmasked entries should be filled 
        appropriately for the normalization function.
    context_sizes: list[int], default=None,
        List giving the size of context for each item
        in the batch and used to compute a context_mask.
        If context_mask or context_sizes are not given,
        context is assumed to have fixed size.
    return_weight: bool, default=False
        If True, return the attention weight Tensor.

    Returns
    -------
    output: Variable of size (B, M, P)
        If return_weight is False.
    weight, output: Variable of size (B, M, N), Variable of size (B, M, P)
        If return_weight is True.
    """
```

Install
-------
```bash
python setup.py install
```

Test
----
```bash
python -m pytest
```
Tested with pytorch 1.0.0

About
-----
Attention is used to focus processing on a particular region of input.
The `attend` function provided by this package implements the most
common attention mechanism [[1](#1), [2](#2), [3](#3), [4](#4)], which produces
an output by taking a weighted combination of value vectors with weights
from a scoring function operating over pairs of query and context vectors.

Given query vector `q`, context vectors `c_1,...,c_n`, and value vectors
`v_1,...,v_n` the attention score of `q` with `c_i` is given by

```
    s_i = f(q, c_i)
```

Frequently `f` takes the form of a dot product between query and context vectors.

```
    s_i = q^T c_i
```

The scores are passed through a normalization functions `g` (normally the softmax function).

```
    w_i = g(s_1,...,s_n)_i
```

Finally, the output is computed as a weighted sum of the value vectors.

```
    z = \sum_{i=1}^n w_i * v_i
```

In many applications [[1](#1), [4](#4), [5](#5)] attention is applied
to the context vectors themselves, `v_i = c_i`.

Sizes
-----
This `attend` function provided by this package accepts
batches of size `B` containing
`M` query vectors of dimension `D1`, 
`N` context vectors of dimension `D2`, 
and optionally `N` value vectors of dimension `P`.

Variable Length
---------------
If the number of context vectors varies within a batch, a context
can be ignored by forcing the corresponding weight to be zero.

In the case of the softmax, this can be achieved by adding negative
infinity to the corresponding score before normalization.
Similarly, for elementwise normalization functions the weights can
be multiplied by an appropriate {0,1} mask after normalization.

To facilitate the above behavior, a context mask, with entries
in `{-inf, 0}` or `{0, 1}` depending on the normalization function,
can be passed to this function. The masks should have size `(B, M, N)`.

Alternatively, a list can be passed giving the size of the context for
each item in the batch. Appropriate masks will be created from these lists.

Note that the size of output does not depend on the number of context vectors.
Because of this, context positions are truly unaccounted for in the output.

References
----------
#### [[1]](https://arxiv.org/abs/1409.0473)

    @article{bahdanau2014neural,
        title={Neural machine translation by jointly learning to align and translate},
        author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
        journal={arXiv preprint arXiv:1409.0473},
        year={2014}
    }

#### [[2]](https://arxiv.org/abs/1410.5401)
    @article{graves2014neural,
      title={Neural turing machines},
      author={Graves, Alex and Wayne, Greg and Danihelka, Ivo},
      journal={arXiv preprint arXiv:1410.5401},
      year={2014}
    }

#### [[3]](https://arxiv.org/abs/1503.08895)

    @inproceedings{sukhbaatar2015end,
        title={End-to-end memory networks},
        author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others},
        booktitle={Advances in neural information processing systems},
        pages={2440--2448},
        year={2015}
    }

#### [[4]](https://distill.pub/2016/augmented-rnns/)

    @article{olah2016attention,
        title={Attention and augmented recurrent neural networks},
        author={Olah, Chris and Carter, Shan},
        journal={Distill},
        volume={1},
        number={9},
        pages={e1},
        year={2016}
    }

#### [[5]](https://arxiv.org/abs/1506.03134)

    @inproceedings{vinyals2015pointer,
        title={Pointer networks},
        author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep},
        booktitle={Advances in Neural Information Processing Systems},
        pages={2692--2700},
        year={2015}
    }


================================================
FILE: attention/__init__.py
================================================
from . attention import attend


================================================
FILE: attention/attention.py
================================================
from torch import FloatTensor
from torch.autograd import Variable
from torch.nn.functional import sigmoid, softmax


def mask3d(value, sizes):
    """Mask entries in value with 0 based on sizes.

    Args
    ----
    value: Tensor of size (B, N, D)
        Tensor to be masked. 
    sizes: list of int
        List giving the number of valid values for each item
        in the batch. Positions beyond each size will be masked.

    Returns
    -------
    value:
        Masked value.
    """
    v_mask = 0
    v_unmask = 1
    mask = value.data.new(value.size()).fill_(v_unmask)
    n = mask.size(1)
    for i, size in enumerate(sizes):
        if size < n:
            mask[i,size:,:] = v_mask
    return Variable(mask) * value


def fill_context_mask(mask, sizes, v_mask, v_unmask):
    """Fill attention mask inplace for a variable length context.

    Args
    ----
    mask: Tensor of size (B, N, D)
        Tensor to fill with mask values. 
    sizes: list[int]
        List giving the size of the context for each item in
        the batch. Positions beyond each size will be masked.
    v_mask: float
        Value to use for masked positions.
    v_unmask: float
        Value to use for unmasked positions.

    Returns
    -------
    mask:
        Filled with values in {v_mask, v_unmask}
    """
    mask.fill_(v_unmask)
    n_context = mask.size(2)
    for i, size in enumerate(sizes):
        if size < n_context:
            mask[i,:,size:] = v_mask
    return mask


def dot(a, b):
    """Compute the dot product between pairs of vectors in 3D Variables.
    
    Args
    ----
    a: Variable of size (B, M, D)
    b: Variable of size (B, N, D)
    
    Returns
    -------
    c: Variable of size (B, M, N)
        c[i,j,k] = dot(a[i,j], b[i,k])
    """
    return a.bmm(b.transpose(1, 2))


def attend(query, context, value=None, score='dot', normalize='softmax',
           context_sizes=None, context_mask=None, return_weight=False
           ):
    """Attend to value (or context) by scoring each query and context.

    Args
    ----
    query: Variable of size (B, M, D1)
        Batch of M query vectors.
    context: Variable of size (B, N, D2)
        Batch of N context vectors.
    value: Variable of size (B, N, P), default=None
        If given, the output vectors will be weighted
        combinations of the value vectors.
        Otherwise, the context vectors will be used.
    score: str or callable, default='dot'
        If score == 'dot', scores are computed
        as the dot product between context and
        query vectors. This Requires D1 == D2.
        Otherwise, score should be a callable:
             query    context     score
            (B,M,D1) (B,N,D2) -> (B,M,N)
    normalize: str, default='softmax'
        One of 'softmax', 'sigmoid', or 'identity'.
        Name of function used to map scores to weights.
    context_mask: Tensor of (B, M, N), default=None
        A Tensor used to mask context. Masked
        and unmasked entries should be filled 
        appropriately for the normalization function.
    context_sizes: list[int], default=None,
        List giving the size of context for each item
        in the batch and used to compute a context_mask.
        If context_mask or context_sizes are not given,
        context is assumed to have fixed size.
    return_weight: bool, default=False
        If True, return the attention weight Tensor.

    Returns
    -------
    output: Variable of size (B, M, P)
        If return_weight is False.
    weight, output: Variable of size (B, M, N), Variable of size (B, M, P)
        If return_weight is True.
        
    
    About
    -----
    Attention is used to focus processing on a particular region of input.
    This function implements the most common attention mechanism [1, 2, 3],
    which produces an output by taking a weighted combination of value vectors
    with weights from by a scoring function operating over pairs of query and
    context vectors.

    Given query vector `q`, context vectors `c_1,...,c_n`, and value vectors
    `v_1,...,v_n` the attention score of `q` with `c_i` is given by

        s_i = f(q, c_i)

    Frequently, `f` is given by the dot product between query and context vectors.

        s_i = q^T c_i

    The scores are passed through a normalization functions g.
    This is normally the softmax function.

        w_i = g(s_1,...,s_n)_i

    Finally, the output is computed as a weighted
    combination of the values with the normalized scores.

        z = sum_{i=1}^n w_i * v_i

    In many applications [4, 5] the context and value vectors are the same, `v_i = c_i`.

    Sizes
    -----
    This function accepts batches of size `B` containing
    `M` query vectors of dimension `D1`,
    `N` context vectors of dimension `D2`, 
    and optionally `N` value vectors of dimension `P`.

    Variable Length Contexts
    ------------------------    
    If the number of context vectors varies within a batch, a context
    can be ignored by forcing the corresponding weight to be zero.

    In the case of the softmax, this can be achieved by adding negative
    infinity to the corresponding score before normalization.
    Similarly, for elementwise normalization functions the weights can
    be multiplied by an appropriate {0,1} mask after normalization.

    To facilitate the above behavior, a context mask, with entries
    in `{-inf, 0}` or `{0, 1}` depending on the normalization function,
    can be passed to this function. The masks should have size `(B, M, N)`.

    Alternatively, a list can be passed giving the size of the context for
    each item in the batch. Appropriate masks will be created from these lists.

    Note that the size of output does not depend on the number of context vectors.
    Because of this, context positions are truly unaccounted for in the output.

    References
    ----------
    [1](https://arxiv.org/abs/1410.5401)
        @article{graves2014neural,
          title={Neural turing machines},
          author={Graves, Alex and Wayne, Greg and Danihelka, Ivo},
          journal={arXiv preprint arXiv:1410.5401},
          year={2014}
        }

    [2](https://arxiv.org/abs/1503.08895)

        @inproceedings{sukhbaatar2015end,
            title={End-to-end memory networks},
            author={Sukhbaatar, Sainbayar and Weston, Jason and Fergus, Rob and others},
            booktitle={Advances in neural information processing systems},
            pages={2440--2448},
            year={2015}
        }

    [3](https://distill.pub/2016/augmented-rnns/)

        @article{olah2016attention,
            title={Attention and augmented recurrent neural networks},
            author={Olah, Chris and Carter, Shan},
            journal={Distill},
            volume={1},
            number={9},
            pages={e1},
            year={2016}
        }

    [4](https://arxiv.org/abs/1409.0473)

        @article{bahdanau2014neural,
            title={Neural machine translation by jointly learning to align and translate},
            author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua},
            journal={arXiv preprint arXiv:1409.0473},
            year={2014}
        }

    [5](https://arxiv.org/abs/1506.03134)

        @inproceedings{vinyals2015pointer,
            title={Pointer networks},
            author={Vinyals, Oriol and Fortunato, Meire and Jaitly, Navdeep},
            booktitle={Advances in Neural Information Processing Systems},
            pages={2692--2700},
            year={2015}
        }
    """
    q, c, v = query, context, value
    if v is None:
        v = c

    batch_size_q, n_q, dim_q = q.size()
    batch_size_c, n_c, dim_c = c.size()
    batch_size_v, n_v, dim_v = v.size()

    if not (batch_size_q == batch_size_c == batch_size_v):
        msg = 'batch size mismatch (query: {}, context: {}, value: {})'
        raise ValueError(msg.format(q.size(), c.size(), v.size()))

    batch_size = batch_size_q

    # Compute scores
    if score == 'dot':
        s = dot(q, c)
    elif callable(score):
        s = score(q, c)
    else:
        raise ValueError(f'unknown score function: {score}')

    # Normalize scores and mask contexts
    if normalize == 'softmax':
        if context_mask is not None:
            s = context_mask + s

        elif context_sizes is not None:
            context_mask = s.data.new(batch_size, n_q, n_c)
            context_mask = fill_context_mask(context_mask,
                                             sizes=context_sizes,
                                             v_mask=float('-inf'),
                                             v_unmask=0
                                             )
            s = context_mask + s

        s_flat = s.view(batch_size * n_q, n_c)
        w_flat = softmax(s_flat, dim=1)
        w = w_flat.view(batch_size, n_q, n_c)

    elif normalize == 'sigmoid' or normalize == 'identity':
        w = sigmoid(s) if normalize == 'sigmoid' else s
        if context_mask is not None:
            w = context_mask * w
        elif context_sizes is not None:
            context_mask = s.data.new(batch_size, n_q, n_c)
            context_mask = fill_context_mask(context_mask,
                                             sizes=context_sizes,
                                             v_mask=0,
                                             v_unmask=1
                                             )
            w = context_mask * w

    else:
        raise ValueError(f'unknown normalize function: {normalize}')

    # Combine
    z = w.bmm(v)
    if return_weight:
        return w, z
    return z


================================================
FILE: examples/Pointer-Network-Argmin-Argmax.ipynb
================================================
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pointer Network Attention Demo\n",
    "\n",
    "The below code trains a [pointer network](https://arxiv.org/abs/1506.03134) like architecture that takes as input a sequence of vectors and outputs the vector with the minimum or maximum value along a particular coordinate. In other words, a neural network version of argmax/argmin.\n",
    "\n",
    "### Setup\n",
    "\n",
    "Let $\\{\\mathbf{c}_i\\}_{i=1}^n = (\\mathbf{c}_1, \\ldots, \\mathbf{c}_n)$ be a sequence of $n$ vectors with each $\\mathbf{c}_i \\in \\mathbb{R}^d$. The minimum and maximum target positions, $i_\\min$ and $i_\\max$, for the sequence are given by\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "    i_\\min &= \\text{argmin}_i \\left\\{ x_{i, k_\\min}\\right\\} \\\\\n",
    "    i_\\max &= \\text{argmax}_i \\left\\{ x_{i, k_\\max}\\right\\} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "where $1 \\leq k_\\min \\neq k_\\max \\leq d$ are a priori chosen coordiantes along which to compute the minimum or maximum.\n",
    "\n",
    "### Model\n",
    "\n",
    "The model has the following form\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "    \\mathbf{u}_i &= A \\mathbf{c}_i & \\; i = 1, \\ldots, n \\\\\n",
    "    \\mathbf{v} &= B \\mathbf{q} \\\\\n",
    "    \\mathbf{p} &= \\text{softmax}_i(\\mathbf{v}^T \\mathbf{u}_i) \\\\\n",
    "    \\mathbf{z} &= \\sum_i p_i \\mathbf{c}_i\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "where $A, B \\in \\mathbb{R}^{p \\times d}$ and $\\mathbf{q} \\in \\{\\mathbf{q}_\\min, \\mathbf{q}_\\max\\} \\subseteq \\mathbb{R}^d$ is a query vector indicating whether the model should output the minimum or maximum.\n",
    "\n",
    "The loss is defined as the mean squared error between the output and target vector.\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "    l &= \\frac{1}{n}\\sum_j (z_j - c_{t,j})^2\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "where $t$ is either $i_\\min$ or $i_\\max$.\n",
    "\n",
    "### Details\n",
    "The vectors $\\mathbf{q}_\\min$ and $\\mathbf{q}_\\max$ are initialized to random values sampled from $N(\\mathbb{0}, \\mathbb{I}_d)$ and held constant throughout training. The code below uses 10 dimensional context and query vectors and 7 dimensional hidden representations. The model is optimized over 1,600 training instances using RMSProp with mini batches of size 8. Each training instance contains betwen 5 and 14 context vectors. To better assess generalization validation instances are longer, containing between 15 and 24 context vectors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Seed: 1273\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "\n",
    "import torch\n",
    "from torch.nn import Linear, Module\n",
    "\n",
    "from attention import attend\n",
    "\n",
    "\n",
    "seed = sum(map(ord, 'les bons mots'))\n",
    "np.random.seed(seed)\n",
    "torch.manual_seed(seed)\n",
    "print(f'Seed: {seed}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Data(object):\n",
    "    dim = 10\n",
    "    min_position, max_position = 3, 7\n",
    "    q_min = np.random.normal(0, 1, dim).astype(np.float32)\n",
    "    q_max = np.random.normal(0, 1, dim).astype(np.float32)\n",
    "    query = np.row_stack([q_min, q_max])\n",
    "\n",
    "    @staticmethod\n",
    "    def create_minibatches(n, m, min_length, max_length):\n",
    "        assert 0 < min_length <= max_length\n",
    "\n",
    "        minibatches = []\n",
    "        for i in range(n):\n",
    "            lengths = np.random.randint(min_length, max_length + 1, m)\n",
    "            context = np.zeros((m, lengths.max(), Data.dim), dtype=np.float32)\n",
    "            target = np.zeros((m, 2, Data.dim), dtype=np.float32)\n",
    "            target_indices = []\n",
    "\n",
    "            for j, length in enumerate(lengths):\n",
    "                c = np.random.normal(0, 1, (length, Data.dim))\n",
    "                k_min = np.argmin(c[:,Data.min_position])\n",
    "                k_max = np.argmax(c[:,Data.max_position])\n",
    "                target_min = c[k_min]\n",
    "                target_max = c[k_max]\n",
    "                context[j,:length] = c\n",
    "                target[j,0] = target_min\n",
    "                target[j,1] = target_max\n",
    "                target_indices.append((k_min, k_max))\n",
    "\n",
    "            query = torch.from_numpy(np.tile(Data.query, (m, 1, 1)))\n",
    "            context = torch.from_numpy(context)\n",
    "            target = torch.from_numpy(target)\n",
    "            minibatches.append((query, context, target, lengths, target_indices))\n",
    "        return minibatches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class PointerNet(Module):\n",
    "    def __init__(self, n_hidden):\n",
    "        super().__init__()\n",
    "        self.n_hidden = n_hidden\n",
    "        self.f = Linear(Data.dim, n_hidden)\n",
    "        self.g = Linear(Data.dim, n_hidden)\n",
    "\n",
    "    def forward(self, q, x, lengths=None, **kwargs):\n",
    "        batch_size_q, n_queries, dim_q = q.size()\n",
    "        batch_size_x, n_inputs, dim_x = x.size()\n",
    "        assert batch_size_q == batch_size_x\n",
    "        assert dim_q == dim_x\n",
    "        batch_size = batch_size_q\n",
    "        dim = dim_q\n",
    "\n",
    "        q_flat = q.view(batch_size*n_queries, dim)\n",
    "        u_flat = self.f(q_flat)\n",
    "        u = u_flat.view(batch_size, n_queries, self.n_hidden)\n",
    "\n",
    "        x_flat = x.view(batch_size*n_inputs, dim)\n",
    "        v_flat = self.g(x_flat)\n",
    "        v = v_flat.view(batch_size, n_inputs, self.n_hidden)\n",
    "\n",
    "        return attend(u, v, value=x, context_sizes=lengths, **kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 8\n",
    "\n",
    "n_train = 200\n",
    "min_length_train, max_length_train = 5, 14\n",
    "train_batches = Data.create_minibatches(n_train, batch_size, min_length_train, max_length_train)\n",
    "\n",
    "n_valid = 100\n",
    "min_length_valid, max_length_valid = 15, 24\n",
    "valid_batches = Data.create_minibatches(n_valid, batch_size, min_length_valid, max_length_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "net = PointerNet(7)\n",
    "opt = torch.optim.RMSprop(net.parameters(), lr=0.001)\n",
    "mse = torch.nn.MSELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1] 0.647\n",
      "[ 2] 0.264\n",
      "[ 3] 0.162\n",
      "[ 4] 0.120\n",
      "[ 5] 0.096\n",
      "[ 6] 0.081\n",
      "[ 7] 0.070\n",
      "[ 8] 0.063\n",
      "[ 9] 0.057\n",
      "[10] 0.052\n"
     ]
    }
   ],
   "source": [
    "epoch = 0\n",
    "max_epochs = 10\n",
    "while epoch < max_epochs:\n",
    "    sum_loss = 0\n",
    "    for query, context, target, lengths, target_indices in train_batches:\n",
    "        net.zero_grad()\n",
    "        output = net(query, context, lengths=lengths)\n",
    "        loss = mse(output, target)\n",
    "        loss.backward()\n",
    "        opt.step()\n",
    "        sum_loss += loss.item()\n",
    "    epoch += 1\n",
    "    print('[{:2d}] {:5.3f}'.format(epoch, sum_loss / n_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "valid loss: 0.059\n",
      "valid error min: 0.018\n",
      "valid error max: 0.015\n"
     ]
    }
   ],
   "source": [
    "sum_loss = 0\n",
    "sum_error_min = 0\n",
    "sum_error_max = 0\n",
    "\n",
    "for query, context, target, lengths, target_indices in valid_batches:\n",
    "    with torch.no_grad():\n",
    "        weight, output = net(query, context, lengths=lengths, return_weight=True)\n",
    "        loss = mse(output, target)\n",
    "\n",
    "    sum_loss += loss.item()\n",
    "    weight = weight.data.numpy()\n",
    "\n",
    "    for i, (i_min_true, i_max_true) in enumerate(target_indices):\n",
    "        w_min, w_max = weight[i]\n",
    "        i_min_pred = w_min.argmax()\n",
    "        i_max_pred = w_max.argmax()\n",
    "        sum_error_min += int(i_min_true != i_min_pred)\n",
    "        sum_error_max += int(i_max_true != i_max_pred)\n",
    "\n",
    "print('valid loss: {:5.3f}'.format(sum_loss / n_valid))\n",
    "print('valid error min: {:5.3f}'.format(sum_error_min / (n_valid * batch_size)))\n",
    "print('valid error max: {:5.3f}'.format(sum_error_max / (n_valid * batch_size)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAr8AAALICAYAAAB2PpiXAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzs3Xl8VNX9//HXmSUECCCrQFhV9kWWqLijCIJVoBYtLS6ttfitxa39fbVq/f60/bba2vprrbWKy5dWqYooghsCflFEsRLQKqsoBNlJWEKALLOc3x+ThITJMsncWch9Px8PyMzce88952bymc+ce+65xlqLiIiIiIgbeFJdARERERGRZFHyKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn7lhGOMecIYc5/T64qISP2MMYeNMaekuh4ijWU0z6+kE2NMHtAV6GqtLajy+mfA6UBva21eamonIpL+FEdF6qaeX0lHW4DvVTwxxgwBmqeuOiIiJxzFUZFaKPmVdPQccF2V59cD/6h4YoyZZYz57/LHo40x240xPzfG7DXG7DLG/LCede+ssu5kY8xlxpgvjTH7jTH31LRt1e2rPM8zxvynMeZzY8wRY8wzxpiTjTFvG2OKjDFLjDFtE3KERETqVl8c/ZYx5lNjzCFjzDZjzP1Vln3XGLPZGNO6/PkEY8xuY0zH8ufWGHNa+eNZxpjHy+PeYWPMh8aYzsaYPxljDhhjNhhjhlcpu3LbKts3KkaLNJaSX0lHHwOtjTEDjDFe4LvA83Ws3xloA2QDPwL+WkfS2RnILF/3v4CngGuAkcD5wH81cCzbd4CxQF/gCuBt4B6gA5G/r1sbUJaIiFPqi6NHiCTHJwHfAn5ijJkMYK19CVgBPGqMaQ88A9xorc2vZV9XA78kEvdKy7ddXf58LvBIA+rtdIwWiaLkV9JVRa/FWGADsKOOdQPAr6y1AWvtW8BhoF8d6/7GWhsAXiQSnP9srS2y1q4F1gJDG1DPv1hr91hrdwAfAP+y1n5qrS0F5gHD695cRCRhao2j1tr3rLVfWGvD1trPgReAC6ts+1PgYuA94HVr7Rt17GeetXaVtbaESNwrsdb+w1obAl6iYXHQ6RgtEsWX6gqI1OI5YBnQmyqn6mqxz1obrPL8KJBVx7qh8sfF5T/3VFleXMe2NTl+23jKEhFxUq1x1BhzFvAQMBjIAJoBL1cst9YeNMa8DPyMyBmuujgZB52O0SJR1PMraclau5XIBRuXAa+mqBpHgBZVnndOUT1ERBqsnjj6T2AB0N1a2wZ4AjAVC40xw4AbiPQIP+pgtY6iuCoppuRX0tmPgIuttUdStP/PgMuMMe2MMZ2B21NUDxGRxqotjrYC9ltrS4wxZwLfr1hgjMkkMj74HuCHQLYx5maH6vMZ8H1jjNcYM57qQy1EkkLJr6Qta+3X1trcFFbhOeDfQB6wiMjYNRGRE0YdcfRm4FfGmCIiF5bNqbLsQWC7tfZv5dcvXAP8tzGmjwNVuo3IxcEHgWnAaw6UKdIgusmFiIiIiLiGen5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6RkptcjB8/3i5cuDAVu5YUm7GoEIDHxrVJcU1E0oKpf5X6KaZKU6DPB4lTzPE0JT2/BQUFqditiEiTpJgqIhI7DXsQEREREddQ8isiIiIirqHkV0RERERcI+4L3owx3YF/AJ2BMDDTWvvneMsVkdQIBAJs376dkpKSVFelycjMzKRbt274/f5UV0VEkkBxNHGciKdOzPYQBH5urV1tjGkFrDLGLLbWrnOgbBFJsu3bt9OqVSt69eqFMY5MRuBq1lr27dvH9u3b6d27d6qrIyJJoDiaGE7F07iHPVhrd1lrV5c/LgLWA9nxlisiqVFSUkL79u0VsB1ijKF9+/bqARJxEcXRxHAqnjo65tcY0wsYDvyrhmXTjTG5xpjc/Px8J3crIg5TwHZWIo6nYqpIelMcTQwnjqtjya8xJgt4BbjdWnvo+OXW2pnW2hxrbU7Hjh2d2q2IiCsppoqINI4jya8xxk8k8Z1trX3ViTJFROqyYMECHnrooYTuIysrK6Hli4i4xXvvvcfll1+e6moAzsz2YIBngPXW2kfir5KISP0mTpzIxIkTU10NERE5wTjR83sucC1wsTHms/J/lzlQroi4UF5eHv379+fGG29k8ODBTJs2jSVLlnDuuefSp08fPvnkEwBmzZrFjBkzAPjBD37ArbfeyjnnnMMpp5zC3Llzo8q96667ePzxxyuf33///fzxj3/k8OHDjBkzhhEjRjBkyBDmz58fte3xPRYzZsxg1qxZAKxatYoLL7yQkSNHcumll7Jr1y4nD4eISIPFGkc/+eQTzjnnHIYPH84555zDxo0bAXjkkUe44YYbAPjiiy8YPHgwR48erbaPs846i7Vr11Y+Hz16NKtWraq1zKruv/9+/vCHP1Q+Hzx4MHl5eQA8//zznHnmmQwbNoybbrqJUCjk6LEBB3p+rbXLAY3qlgb79oIRMa87b+LqBNZEavPn3CNs2h90tMw+7XzcltOyznW++uorXn75ZWbOnMkZZ5zBP//5T5YvX86CBQv47W9/y2uvvRa1za5du1i+fDkbNmxg4sSJTJkypdryqVOncvvtt3PzzTcDMGfOHBYuXEhmZibz5s2jdevWFBQUMGrUKCZOnBjTRRWBQIBbbrmF+fPn07FjR1566SXuvfdenn322QYcEZGmL9Z4n525NME1Sb50jqP9+/dn2bJl+Hw+lixZwj333MMrr7zC7bffzujRo5k3bx6/+c1vePLJJ2nRokW18qdOncqcOXN44IEH2LVrFzt37mTkyJEcOnSoxjJjsX79el566SU+/PBD/H4/N998M7Nnz+a6665r9LGqiRPz/IpII8X6geC25L93794MGTIEgEGDBjFmzBiMMQwZMqSyd+B4kydPxuPxMHDgQPbs2RO1fPjw4ezdu5edO3eSn59P27Zt6dGjB4FAgHvuuYdly5bh8XjYsWMHe/bsoXPnzvXWc+PGjaxZs4axY8cCEAqF6NKlS+MbLiLikFjiaGFhIddffz2bNm3CGEMgEADA4/Ewa9Yshg4dyk033cS5554bVf7VV1/N2LFjeeCBB5gzZw5XXXVVnWXG4t1332XVqlWcccYZABQXF9OpU6d4DkONlPyKSK3q61lIlGbNmlU+9ng8lc89Hg/BYM09KFW3sdbWuM6UKVOYO3cuu3fvZurUqQDMnj2b/Px8Vq1ahd/vp1evXlFzSPp8PsLhcOXziuXWWgYNGsSKFSsa0UoRcYN0jqP33XcfF110EfPmzSMvL4/Ro0dXbrNp0yaysrLYuXNnjeVnZ2fTvn17Pv/8c1566SWefPLJesusUFdMvf7663nwwQfjant9lPymkHr9RJJr6tSp/PjHP6agoID3338fiPRSdOrUCb/fz9KlS9m6dWvUdj179mTdunWUlpZSUlLCu+++y3nnnUe/fv3Iz89nxYoVnH322QQCAb788ksGDRqU7KaJiDRYYWEh2dmR+5JVXMdQ8fptt93GsmXLmDFjBnPnzo0aSgaRmPr73/+ewsLCyl7m2sqsqlevXrzxxhsArF69mi1btgAwZswYJk2axB133EGnTp3Yv38/RUVF9OzZ06kmAw7f5EKkPsZaqKVXTiTRBg0aRFFREdnZ2ZXDE6ZNm0Zubi45OTnMnj2b/v37R23XvXt3rr76aoYOHcq0adMYPnw4ABkZGcydO5e77rqL008/nWHDhvHRRx8ltU0iTUGb0kyu+WoYV679gHD+gVRXxzXuvPNO7r77bs4999xqF5bdcccd3HzzzfTt25dnnnmGX/ziF+zduzdq+ylTpvDiiy9y9dVX11tmVd/5znfYv38/w4YN429/+xt9+/YFYODAgfz3f/8348aNY+jQoYwdOzYhFxGb2k4PJlJOTo7Nzc1N+n7TjZt6fm3YEnrvE/YtXkmJL4NuV41mSt7UmLdvCsegJun4Hli/fj0DBgxI2v7copbj6sjFwoqpku7qinVdj7Tm7s8vpENp+fCA5s3I+NGVeE7pnqTaOU9xNLHijafq+ZWEsyWlBGbNI/jG+7QpPcrJRw4SmPUak7YOBHUCi4i4Vt/CDjzw6ZhjiS9AcSllT8wh9PmXqauYNGlKfiWhwnv2Ufan5wiv+Spq2dQtQ/nJhrPwhfU2FBFxm5z8bO7992iygs2iFwZDBP7+GsHlTfOsn6SWLnhzgVSdWg99/iWBF96E0tqnOblgT286lWTxyKDlFGWUOrr/ZEvHIQwiIunokh2n8sNNI/HU1QdnIfjqEuyhw/gmnB/T3NsisVDyK46z4TDBhcsJLfk4atmOVu05qeQwLQPHEt3+hR359eqxPDxkGTtaHkpmVUVEEk439KnCwlV5Q7hya/SMKHN6fYHNmsF31y2D8LExcaElH2MLD+O/+lKM15vM2koT1eSTXwWd5LJHignMfoPwhi1Ry7xnDOYvJ53FSSVHuHvDO9gqV/SeXJLFr1Zfwp8GfcQX7XYns8oiIpIE3rDhR1+ewUW7T6n2eogwz/TNZWnXzWRn9ueaUR0I/GMBlB07axheuYZA0RH810/CNMtIdtWliWnyya8kT3jHXgKzXsPuO1h9gceD79tj8J4zjODiQxS0bEPGbdey6g/3MvjgyZWrtQhlcNfnF/D3PqtZnB09RlhExC3SueOmMUO8moW83Lb2XIbv71ptnVJPkD8P/IhPOxy7kYJ34KmYn0yl7Om5cKS48vXwhi2UPf4iGTd+B9MqNTeOkKbhhEt+3T6uMl3bH1q9jsBLCyFw3N23WrUk4/pJeE7pVu1l0yKTh4a+xw835TBm16mVr3vxcMOmHLoebc1zp35K2KPpIOSY3/72t9xzzz0AHDx4kH/+85/cfPPNjS5v1qxZjBs3jq5dIx/IN954Iz/72c8YOHCgI/UVEbCHj/LLzy7mtKL21V4v8pXy+yHL+KrNvqhtPD27kHHrNAJPvozdX3isrG27KXt0Nv6brsLToW3C694UKY6egMlvukrnb+mJZENhgm+8R+j96DlGTa+uZFw/CdOmVY3bhjyWp/uuZGeLQ0z7ehieKlP0jd/Rl5OLs/jLwI8o9h1LqN16nCXi+KD9+OOPxx20Bw8eXBm0n376aUfqKSIR4X0HCcx8OSrx3Zt5mIeGvs+uFkW1buvp2I6M266h7Km52O17Kl+3+w5S9uhsMn78HTzdu1TbJl07iNKJ4qiS3xopwYqNPXyUwD8WEP7qm6hl3nOG4Zs8BuOr5+IEA29138ju5kXcsu5sMsP+ykXD93flgdWX8PCQD5yuer0UQKHkZ79PaPmZj9xZ5/LJkyezbds2SkpKuO2229i8eTPFxcUMGzaMQYMGEQqF+Prrrxk2bBhjx47l4Ycf5uGHH2bOnDmUlpby7W9/mwceeIC8vDwmTJjAeeedx0cffUR2djbz58/nzTffJDc3l2nTptG8eXNWrFjBhAkT+MMf/kBOTg4vvPACv/3tb7HW8q1vfYvf/e53AGRlZXHbbbfxxhtv0Lx5c+bPn8/JJ59cZ1vSlWKdJFJ4+27KnnoFio5Ue31L1n5+N2QZhc1K6i3DtGpJxs1TCfx9PuGNeccWHD5K2V9fxP+DyXj793a45s5RHE3POOpI8muMGQ/8GfACT1trH3Ki3IbyhA3djrYGILwzchu+HofbxLx9um5TsX7Ct9m2C1tSBqWRf7a0DErKf9bwmt27v9p4LAC8XnxTxuI7a2hM+6ywusNO/u+Id/nPL86vNtl596Mn8evVYwkN+RpzUqtGHefGSLvfTfk2pmM7jN8d31mfffZZ2rVrR3FxMWeccQbvv/8+jz32GJ999hkAeXl5rFmzpvL5okWL2LRpE5988gnWWiZOnMiyZcvo0aMHmzZt4oUXXuCpp57i6quv5pVXXuGaa67hscceqwzSVe3cuZO77rqLVatW0bZtW8aNG8drr73G5MmTOXLkCKNGjeI3v/kNd955J0899RS//OUvk358kqFVWQZty5oD8f09uVKVEVs9Dp8U82bhHRWfDw3YJsm/m1jiVuejrSj76wtRU11+0XY3/2/Q8mpn9OpjMpvh/9F3CMxZSDh37bEFZQECT7+CvfISPL26xlw3KD9mPh+eTu1irseJSHG0ZnF/ihpjvMBfgbHAdmClMWaBtXZdvGU3VIuQn9/lTgCgLHcWAL9jQszbp+s2FesnfpvnYlqvVie1IuMHk/H06FL/ujX4Jusg941YzM/XnF/tFFmbQCaBp18BGnecGyP9fjeRbTJ+8SNMp/Z1r9xEPProo8ybNw+Abdu2sWnTpjrXX7RoEYsWLWL48OEAHD58mE2bNtGjRw969+7NsGHDABg5ciR5eXl1lrVy5UpGjx5Nx44dAZg2bRrLli1j8uTJZGRkcPnll1eWtXjx4niamdZG5ffghk2RD7R4/p7c7neMj3ndY58PDd8mWWKPw9UT3+Wd8nii/yeEPOEG79P4vPi/dxnB1lmE/vdfxxaEwwTnLmpw3cpyZ2G6dqLZ//lBg+tyIlEcrZkTXUhnAl9ZazcDGGNeBCYBSU9+JXU8p3bHf93EuK/APdishF8N+19u3nAWo/J7OFQ7OdG89957LFmyhBUrVtCiRQtGjx5NSUndp0ittdx9993cdNNN1V7Py8ujWbNjd5Dyer0UFxcfv3lUWbXx+/2Vk+17vV6Cwdh7sETc6vXu63nhlH9j47hPhTEG/+UXYtpkEXzt3Wq96xJNcbR2TiS/2cC2Ks+3A2cdv5IxZjowHaBHj8YnNXWNO7NHiin98C+NLlsawefDe/4IfJddwJVv5tS/PpCduRSo53cZtgTfWU5o8QpHqikNV99YskQqLCykbdu2tGjRgg0bNvDxx5Ebpvj9fgKBAH6/n1atWlFUdOximUsvvZT77ruPadOmkZWVxY4dO/D7/bXtAiCqjApnnXUWt912GwUFBbRt25YXXniBW265xdlGxsmJmFrfON7gh58S3NR0e7YlOXyTLuaqC+/kqnrWm7GosNrzusaknzWgOz9dPwq/Te+bXiiOpmccdSL5rel7XFS6b62dCcwEyMnJScz3NY/BdOmYkKJdwWOgWQYmM4PlBUsp9gYp8QYiP32BY8+rPH7kyrcxzWu4L3ucjMfgn3A+nq6dCC7LhZIyx/dxQvK5Y7zv+PHjeeKJJxg6dCj9+vVj1KhRAEyfPp2hQ4cyYsQIZs+ezbnnnsvgwYOZMGECDz/8MOvXr+fss88GIhdUPP/883jruCPUD37wA/7jP/6j8kKNCl26dOHBBx/koosuwlrLZZddxqRJkxLb6AZKRkw1LTJPqJi69VDdp3Qr9GzdJ+nbJEPataV5M3wXnYl30GkN37Ye/+q0jf3NjnJV3hCGZg5pVBmmY9OeKk1xtHamrm7pmAow5mzgfmvtpeXP7waw1j5Y2zY5OTk2Nzd6aixJH42Z7SDWbbIzl/LYuNgvXJPkWr9+PQMGDEh1NZqcWo5rHCeBj1FMjUhk3Ip3m2Q40dtS0fNb8fmQTnVrKMXRxIo3njrRjbQS6GOM6Q3sAKYC33egXBEREYlRY5LAdEwcRRIt7uTXWhs0xswA3iEy1dmz1tq19WwmIiIiIpJ0jgwgtNa+BbzlRFkiIiIiIonijqtnRESkydMpfBGJhSfVFRARERERSRYlvyIiIiLiGhr2IDXS6UOpEOt0Q7FKxHsrKyuLw4cPO16uiIgTFEfTi3p+RURERMQ1lPyKSNq56667ePzxxyuf33///TzwwAOMGTOGESNGMGTIEObPnx+13Xvvvcfll19e+XzGjBnMmjULgFWrVnHhhRcycuRILr30Unbt2gXAo48+ysCBAxk6dChTp05NbMNERJJEcbR2GvYgjonlNMzx924XqcnUqVO5/fbbufnmmwGYM2cOCxcu5I477qB169YUFBQwatQoJk6ciDH139QnEAhwyy23MH/+fDp27MhLL73Evffey7PPPstDDz3Eli1baNasGQcPHkx000QkRhp+Fx/F0dop+RWRtDN8+HD27t3Lzp07yc/Pp23btnTp0oU77riDZcuW4fF42LFjB3v27KFz5871lrdx40bWrFnD2LFjAQiFQnTp0gWAoUOHMm3aNCZPnszkyZMT2i5pGpSUyYlAcbR2Sn5FJC1NmTKFuXPnsnv3bqZOncrs2bPJz89n1apV+P1+evXqRUlJSbVtfD4f4XC48nnFcmstgwYNYsWKFVH7efPNN1m2bBkLFizg17/+NWvXrsXnU2gUkROf4mjNNOZXRNLS1KlTefHFF5k7dy5TpkyhsLCQTp064ff7Wbp0KVu3bo3apmfPnqxbt47S0lIKCwt59913AejXrx/5+fmVQTsQCLB27VrC4TDbtm3joosu4ve//z0HDx50zdXOItL0KY7WLH3TchFJC6k6xTto0CCKiorIzs6mS5cuTJs2jSuuuIKcnByGDRtG//79o7bp3r07V199NUOHDqVPnz4MHz4cgIyMDObOncutt95KYWEhwWCQ22+/nb59+3LNNddQWFiItZY77riDk046KdlNlRTSEAZJBsXR9GKstUnfaU5Ojs3NzU36fiX1Ki54e2xcmxTXRGqzfv16BgwYkOpqNDm1HNf6rzKJgWKqNAVN6fNBcTSx4o2nGvYgIiIiIq6h5FdEREREXCOu5NcY87AxZoMx5nNjzDxjTHoP8hCRmKRiOFRTpuMp4j76u08MJ45rvD2/i4HB1tqhwJfA3XHXSERSKjMzk3379ilwO8Ray759+8jMzEx1VUQkSRRHE8OpeBrXbA/W2kVVnn4MTImrNiKSct26dWP79u3k5+enuipNRmZmJt26dUt1NUQkSRRHE8eJeOrkVGc3AC/VttAYMx2YDtCjRw8HdysiTvL7/fTu3TvV1ZB6KKaKpC/F0fRW77AHY8wSY8yaGv5NqrLOvUAQmF1bOdbamdbaHGttTseOHZ2pvYiISymmiog0Tr09v9baS+paboy5HrgcGGM1uEVERERE0lhcwx6MMeOBu4ALrbVHnamSiIiIiEhixDvbw2NAK2CxMeYzY8wTDtRJRERERCQh4p3t4TSnKiIiIiIikmi6w5uIiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4KZsxJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLxU7HT9+vF24cGEqdl2rGYsKAXhsXJsU10REXMQ4UUi6xVTFUxFJgZjjaUp6fgsKClKxWxGRJkkxVUQkdhr2ICIiIiKuoeRXRERERFwj7jG/xpjuwD+AzkAYmGmt/XO85YpI6nx7wYiY1503cXUCayIiIuIsJy54CwI/t9auNsa0AlYZYxZba9c5ULaIiIiIiGPiHvZgrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnEk+TXG+IkkvrOtta86UaaIiIiIiNOcmO3BAM8A6621j8RfJRERSVexzgSSnbk0wTUREWkcJ3p+zwWuBS42xnxW/u8yB8oVEREREXFU3D2/1trlOHR/ehERERGRRHJinl8RERERqUWsw4XivWlQsvZzolPyKycU3XlMRERE4qHkN4X0DU1EREQkuU645FcJo4iIiIg0lqN3eBMREQFoFixLdRVERGp0wvX8iohI+up2uA03bTyT04r+h9It3fFPnYCn/UmprpaISCUlvyIi4oiRBV356fqzaR7yA2C/3kbZn54j4/pJeE7rkeLaiUTTRdTupGEPQGjDFr7/73f51saPsYcOp7o6IiInFguTtg7gZ2vOr0x8Kx0ppuyJOQRXfJaauomIHMfVPb+26AiB194l/OkGKr77lT60Ad8Vo/GeNRTj0b07RETq4g95mb7xDM7b26v2lcJhgi8vwu4qwDfpIozXm7T6iYgcz5XJr7WW0Mo1BOcvheKS6gtLSgm+/A6hVWvxX30pnk7tU1NJEZE017a0OT9bcx6nFVWPkyHCrDn5NE7fs7n668tXY/cU4L9uEqZl82RWVUSkkuuGPYTzDxB4Yg7BF9+OTnyrsJu3U/aHWQQXr8AGQ0msoYhI+jv1UDt+s2pcVOJ72FfKQ0Pf57nhY/FfPwkyqg+DCG/6hrI/PUd4d0EyqysiUsk1Pb82FCL0Xi7Bdz6EYDBq+e6strQrLiIjVGVZMETw7Q8Ifboe/3fH4+nZNYk1FhFJT+fu6cn0DWeSYasPX9jRopA/DP6A3S0Okw14T++H6dCWsmdfhQOHKtez+w5S9ufn8V97Bd6Bpya59tU1Zu54zTcvcmJzRfIb3rabwJyF2B17oxf6ffguPZdHyvrSpuQI9+1dQXjDlmqr2N0FlD36PN7zRuKbcB4ms1mSai4iADYUxubvx7RrgzmuJ1GSx4bDTP16KJO2DYxa9mm7nfxl4AqKfYFqr3uyO9Hs9mspmzUfu2X7sQWlZQSeeQX7rQvxXnQmxlS/xkJJqYgkSpNOfm1pGcGFywktWwXWRi339O2J76pL8bQ/ifCiQg60aIX/x1MIr15P4LV34UhxlcIg9MEqQl98iX/KuJT1Vii4S1NnS0qxuwoI79yL3bGH8I692F0FEAzi/8l38fbpmeoqupItKSXw/Bs1Jr6vd1/PC6d8jjXRcRbgyqXn4+3u4YbikVy8u0rstBB8433+d+UWIGZhAAAgAElEQVQsnu67kjnfzk1U9cUl9BkpsXAk+TXGjAf+DHiBp621DzlRbjxCG7dEri7eXxi9sEUm/kkX48kZFNXbYIzBO3Ignn69CCxYSjh3bfVtDxYRePoVQsP74588BtOqZQJb0bRpfkVn2GAQjpZgi0uhuAR7tCQynt0CzZthmmdW+0mGP+p9n5J6WwuFh8uT3L2Rnzv3YgsOROpe0zY79oKS36QL7ztI4JlXsceN0w2YEE/3W8myznn1lhHyhHmq30q+yTrIdV8Nx1PlkpML9vSmy9FW2DGHMa2znK6+iFRhrMGWlkFZAFo2x3hcd/lX/MmvMcYL/BUYC2wHVhpjFlhr18VbdmO0Ksvg2q9HEHjv5RqXe0YMwD/p4nqTVpPVgozvf4vQyIE1JtHhTzdQuiEP75mDoZHT9kz9emhM6wXeeD+ubdJVrG2BY+1pzDZNQiCILS4pT26PS3ID0WPY6+TxlCfDzaB55rGfmc3AHwkJ128ZgS3PQK2h8jFE8tKKHj5LI45zIIjdU0B4x97qZ1diEN5Zw9AlSajQpq0E/j4fjla/QPigv5g/Dl7OV232xV6YgXe6bWJniyJuXXcOWcGMykV9ijpQ+v/+gXfEQDAmafHR7XG4qWno7yahnynWQjAEgSA/2XIW/rAXf9hDRtiHP+zBH/aSEfZW+emh5OM/gc8HPi/G74s89lc894PfW/maKX8djyfyORAIcsuWs2kW9pER8lb56SUj5Kv8mWG9lL7/JwAyfnkTpl2bhrWrCTC2huEADSrAmLOB+621l5Y/vxvAWvtgbdvk5OTY3NzGnd6qq7dw1N7u/HDTSFoHMqMXtm0dGa4w4JQat52xKJLcPjYu+k1gS8sIvvMhofdzaxw+ISJJ0Kol3qF98X9nbKpr4iRHuuDjial1CX74KcF570I4XO31zVn7+ePgD9ifWfuXl+zMpZXxtKa43floK/7PF+eTXdza2UqLSMwy7rwBT+cOqa6GU2KOp04Me8gGtlV5vh04K6pGxkwHpgP06NH421zWdfo7uGwVwXXvVnstTJiF3TYxp9cXlG6aCZtqLzs7cylQe4Lde0RbfrzxDHofbtfwiotITMJYdrU4xNasg+RlHWBr1kG2tjzIrKs+SnXV0ooTMbWuzoTmQR8Pr7yM9uEW1V73DOvHgKmX8UwdFx5WdCZUqC1u20klBJ57PeoiYxFJkkCg/nWaICeS35oy7ajuUWvtTGAmRHopHNhvFO95wwmtXof9ZlekYl06cl+X2Wxuvd+R8re0OsAvRyzmsu39uCpvMBnhJn29oJwgQoQ56gtw2F/GEd+xf+d1uxSKy7AlJWzbu56WwQxaBv1p9b4t8QT5JusgW8uT3LysA2xvWUipV3Nr1yfRMbXYF+SPgz/g/346hmbl7xnfhPPwXnK2Y2PGTfNM/Dd+h+Dr70XOrIlI4hnA74/MwR1259lsJz4FtwPdqzzvBux0oNwGMx4P/qvHU/aX2fguGYV39BlsfvMvju4j7LG80WMDH3Xayoh9Xbmp788BeG5d7Pu5duAtjtYpEWJtT9W2NGabZEjW7yah7fd6ysfnZnLfp7dypEqiW+IN1vgV9OKJf618/J9Vevj8IQ8tQn5aBDNoEfTTsvxnViCDmwb/ovyrq43+WfHDVnlc25VpdTKYdq0x2Z34/opxtc4QIKm3pdUBnuj/L6ZvPJPW116Fd2hfx/dhPB78ky7Ge+YQwhu3QChc/0YnmCYRg+LcT1P6TGkQn69ynK7xV4zf9WF83kgCevwyr6dynPBPF008Nk445COjfJyw/7hxwjcMuKM8mfVFpoL0+yIXNpe/ht9f7XV83rS46DmVnEh+VwJ9jDG9gR3AVOD7DpTbKJ6uHWn2Xz+JXMTTSI2ZWWDBkZtjXveHY0Y1uPxki7U9VdvSmG2SIVm/m2S1f31efBd9BbxhCr2lFGaURi376fkj4yq7oZT4plassc4WHUn4zDaeLh3xdOmY0H2kSlOLQY3ZT2P2ma6fKcmyq0VRTOtNvzhqpKnUI+7k11obNMbMAN4hMtXZs9batfVsllDxJL4iIlKdpnQUkabEkcF/1tq3gLecKEvkRKW5iEVERNJf+lz5Iic8JX8iIqmjGCwSGyW/0uTpA0FEREQqKPkVERERSTPquEkc993QWURERERcSz2/DtE3NBEREZH0p+RXaqRkXkRERJoiDXsQEREREddQ8isiIiIirqHkV0RERERcQ8mviIiIiLiGLngTERGRtKMLryVRlPyKuIA+RERERCI07EFEREREXEPJr4iIiIi4RlzDHowxDwNXAGXA18APrbUHnaiYiNRMQxhEREQaL94xv4uBu621QWPM74C7gbvir5aIuIESeRERSba4kl9r7aIqTz8GpsRXHefF+uE6Y1FhgmsiIiIiIqnm5JjfG4C3HSxPRERERMRR9fb8GmOWAJ1rWHSvtXZ++Tr3AkFgdh3lTAemA/To0aNRlRURkQjFVEkVDVeSE129ya+19pK6lhtjrgcuB8ZYa20d5cwEZgLk5OTUup6IiNRPMVVEpHHine1hPJEL3C601h51pkoiIiIiIokR75jfx4BWwGJjzGfGmCccqJOIiIiISELEO9vDaU5VREREREQk0XSHNxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa8R1wZuIiIjUTTeFEEkv6vkVEREREddQ8isiIiIirmHquCNx4nZqTD6wNcG76QAUJHgf6czt7QcdA7U//dtfYK0dH28hiqlJofa7u/2gY5Du7Y85nqYk+U0GY0yutTYn1fVIFbe3H3QM1H53t99pbj+ear+72w86Bk2p/Rr2ICIiIiKuoeRXRERERFyjKSe/M1NdgRRze/tBx0DtFye5/Xiq/eL2Y9Bk2t9kx/yKiIiIiByvKff8ioiIiIhUo+RXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXUPIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENXyp2On78eLtw4cJU7FpqMWNRIQCPjWuT4pqIuIpxohDFVGkK9DkkcYo5nqak57egoCAVuxURaZIUU0VEYqdhDyIiIiLiGkp+RURERMQ14k5+jTHdjTFLjTHrjTFrjTG3OVExERERERGnOXHBWxD4ubV2tTGmFbDKGLPYWrvOgbJFRERERBwTd8+vtXaXtXZ1+eMiYD2QHW+5IiIiIiJOc3TMrzGmFzAc+FcNy6YbY3KNMbn5+flO7lZExHUUU0VEGsex5NcYkwW8AtxurT10/HJr7UxrbY61Nqdjx45O7VZExJUUU0VEGseR5NcY4yeS+M621r7qRJkiIiIiIk5zYrYHAzwDrLfWPhJ/lUREREREEsOJnt9zgWuBi40xn5X/u8yBckVEREREHBX3VGfW2uU4dH96ETlxfXvBiJjXnTdxdQJrIiIiUjvd4U1EREREXEPJr4iIiIi4hhN3eBMBYj/trVPeIiIikipKfqXJ01hUERERqaDkV0RERBKqvk6I3kVtuf7r02nF6YR6XIC3f+8k1UzcSMmviIiIpMyZe7vx0w2jyAj7gD0EnpqLnXQR3vNHErmVgIizlPyKiIhI8lm4Ylt/vr952HGvW4Kv/S82/wC+yWMwXl2bL85S8iuEt+7iB6uX0aq0mGCrYXjPOp0r3xgZ07YaIysiJzJdqJsa3rCHH32Zw0W7T6l1ndCHn2L3FeK/7gpMZrMk1k6aOiW/LmaLSwm+tYzQR58y2EZeC768iNAna+jRsQ3fZBWmtoIiIpJQqUj+Wwb83LH2PAYdPLna60ETJuhpRmYoUPlaeMNmyv7yTzJu/A6mbWvH6iDupnMJLmStJfTpekofeprQh5+CPW751p38NvdSvv/16TQLeVNTSRERaXI6FWfxwOqxUYnvYV8ZDw59j7+OmgQntaq2zO7Kp/RPzxHetiuZVZUmTD2/LhPed5Dg3MWEN26pcz0vHq7YNoBRe3swq88qVnfYmaQa1k3TlomIpFZje4v7HezAz9ecT6tg9SEMuzOL+P3QZexqUUR2Znua3X4tZc+8it22+9hKRUcoe+wF/NdcgXdIn7jbIO6mnl+XsMEQwSUrKPvdszUmvvktWrOuY4+o1zuWtuQ/11zAHWvOpV1J82RUVU4Qdn8hoS82Edq0FRsIpro6IpLGztnTk3v/fVFU4ruhdT7/NWIxu1oUVb5mWmeR8dPv4Tk+yQ0ECcyaR3DpJ1h73ClLkQZQz68LhL/eRmDuIuyefdELvV68Y87ij3YAQa+PP3fdS+DVJXCwqNpqZxZ0Z8iBzrzc6wveyd5E2KPAc7ymfuGMLSkl/PU2whvzCG/cgs0/cGyhz8cvWl3IF+1283nb3WxrWQiaoUjE9ay1hBZ9xC3rz45atrxTHk/2/4SgJxy1zGT48V8/meCb7xNa+kmVAiH4+nvYggP4rrwE49XQPGk4Jb9NmD1STPD19wh98kWNyz2ndsc3ZRyek9sTXBS5uM07uA+ePj155a//wYTtffFWOTnQPOTnuq9HcP6eXjzdN5fNrfcnpR2SGjYcxm7bTfjLPEIb87B5OyEc/SEFQDDI6Qe6cPqBLgAcyCjmi7a7y//tobBZSRJrLk1BU/8yWZ+mMMTLBoMEXlxIePW6qGVze33BKz3X1vkl2XgM/itGYzq0JfjKIggf63QJrfh3ZCaI6ydhmmsmCGkYR5JfY8x44M+AF3jaWvuQE+VKtJgCooW5Xf9BYMFSOFIcvbxlc/wTL8KTM6jGCcRNswxmn/YZH3TO48aNOfQp6lBtee/D7fj16rEszt6ELS5V4GlCwvsLIz27X+YR/nIrFDcuaW1b1pwL9vTmgj2RuzTltTzAF+0iyfCGNgVOVlnEGRY81uCzHrzWgy9s8FY+9hAuOBDpZfR6oOKnp/yxxzSNmzFY8Npj7fZagzcceWz3F5a310PLQAZhEyZoLGETJmRsVBLbqiyDsr/NwW7ZXu31gAnxZP9P+PDkrTFXy3f26Zh2bQj8/TUoKat8PfxlHmWPPo//x1PwtGsTV9PFXUy842aMMV7gS2AssB1YCXzPWhv9Va9cTk6Ozc3NjWu/NbFHSyj9w/84Xm46KSjeXe86vrCHkwI1j8/1njkE3xWjMS2rL59R3vP72LhIAKlIso01XLLzVKZuHkqLUEZ0gc0yoDz5jaVuAB2ad45pvZrEuo+q+4l1G4OhffMqVyDX9LdRx5/L/pK9Na9y3IdCtX2kk1AYio7Evr7Hg+nRGXuwKGqYTF3KPEEyWrdtRAUbx3/5hXhHDEza/hrIkYwpUTE1tHINgbc/cLzcWCQ0noRt5CxGMERJ2RF81uCzcZ4+r0yMKxJiD3ga/+ttTKxrkFAYQiGOlhZVJrw+2/jLgEKECXkiiXDIhPGFvWSGq/evFflK+ePgD9h4Uu1fgLMzl1Z+Dh0vvLuAsqfmwoFD1Rdk+KFFZqPrLg3T7M4b0nXe5Zj/4Jzo+T0T+MpauxnAGPMiMAmoNflNGGsb9CF8IupAy0ZtZ05uj3/KODyndm/QdtZYFmd/xcoO27n2q+Gck9+z+gqlZZF/DalbaeN/Rw1qf/l+GrNNY7SjRcL3kWqmUzs8fXvh6dcLz6ndMZnNsNZi8/czc+7NDDnQmUEHOpEZ9tdaRkbYl9S/U1sWqH8lqZEtC6QspiYjngBkOjX6LxSK/HNIsuJWC2r/W20ILx684dqT5x3ND/H7ocvY2/xwo/fh6dzh2EwQ31SZ9qwsEPknEiMn/uqzgW1Vnm8Hzjp+JWPMdGA6QI8e0bMKxKqu0/5ZgQye4spGl90k+Xy82H01b3TfQGjtX2Ft7atmZy6tfFzbGLLQhi0EX1mM3XfQ6ZpKGirylbKm7Z7I2N12uynIPBpZ8HX5PyLvFdOpPf9x80tAZGYRu3UnoYoL47bvrrPHXBrHiZha3zCqsTtO4wZyGlW2SAXPaT045QeTebKe3tmKM5AVant/+nt6ubn4LEblNz6XEHdzIvmtqZs56qPOWjsTmAmRU3QO7FfqYgyeAb3xTR7D/I+ed6xYb//eeP7zhwSXrCD0werKXl9pIjweTO9sXgy/yedtd7Ol1QGsadifq/F5Mad2j5xluOx87JFiwpu2Et6YR+jLvOhTltIoiqkJYEz08AWvJzLW11psOAyhMEXF+yNjg8MefNbgaUqzhnoMeKqOaS7/Z8yx4SKhMIRD5T/D1S5EO573nGH4Jo/B+JyblSHgDfHowI/YkXeIyVsHVrswWyQWTiS/24Gq59K7ASm5I8IRX4AZoxYA8NTYtwD48eLLYt4+2dskVGazhF2IZjL8+C+7AN+4cxs2RvQEcmPl7zM6qNsqX/eeGbuw7oLqGVMf6/um6numMdvErGVzTIaf1xY83PBta2FaNsc7rD/eYf3xWQuHj0Iwcno4kX9rle3XWMBG++DkPFa3j4TzRh//Rm7TGDHvZ9zb1S9c83rA48XEOEb3e8f1SBpL5YVxkbGzkQvFnh77doPqBfEdgwYd56oX7FU89nhiPgZV2bAFW54UVyTEoRBk+BM2NtQamNt7Da9338ALFy1JyD5qk5S4lc7bZNRw/c8JxonkdyXQxxjTG9gBTAW+70C5NWrolC77Kk7TxqDivuHJ2uZEZ3xeaCJtOd7+GH+f5rjbcDZUrO+bqu+ZxmyTLowx0OrYWMbG/N08ffVyx+slNSvxBSnxRW5gEs/xT9Z7Nub9xPl3ezxrIGjCUfPVJvvzIVWxIZIweyOJdJKV+oJJj3WJzBHijfVu+0xprLiTX2tt0BgzA3iHyFRnz1pr6xhZmlzpOv+hCOj9KSIikmyOXOZqrX0LSMK5fBFpSpT8u4N+zyKSTnSHNxERkSagqX3JaGrtSVduPM5KfmvgxjeCnDj0/hSn6L0kIm6k5Nch+hAREZGa6PNBJL0o+RURERGJkb7MnPg0M7SIiIiIuIZ6fkVEpElQj5ykK70304t6fkVERETENdTzK1IDfUsXkdooPoic2NTzKyIiIiKuoeRXRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIamu1BRKLoanYREWmq4kp+jTEPA1cAZcDXwA+ttQedqJg4J5ZEZsaiwiTURERERCS14u35XQzcba0NGmN+B9wN3BV/tURERETcS2fgEieuMb/W2kXW2mD504+BbvFXSUREREQkMZy84O0G4O3aFhpjphtjco0xufn5+Q7uVkTEfRRTRUQap97k1xizxBizpoZ/k6qscy8QBGbXVo61dqa1Nsdam9OxY0dnai8i4lKKqSIijVPvmF9r7SV1LTfGXA9cDoyx1lqnKiYiIiIisdM44djEO9vDeCIXuF1orT3qTJVERERERBIj3jG/jwGtgMXGmM+MMU84UCcRERERkYSIq+fXWnuaUxUREREREUk03d5YRERERFxDya+IiIiIuIaSXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIayj5FRERERHXMKm4I7ExJh/YmuDddAAKEryPdOb29oOOgdqf/u0vsNaOj7cQxdSkUPvd3X7QMUj39sccT1OS/CaDMSbXWpuT6nqkitvbDzoGar+72+80tx9Ptd/d7Qcdg6bUfg17EBERERHXUPIrIiIiIq7RlJPfmamuQIq5vf2gY6D2i5PcfjzVfnH7MWgy7W+yY35FRERERI7XlHt+RURERESqUfIrIiIiIq6h5FdEREREXEPJr4iIiIi4hpJfEREREXENJb8iIiIi4hpKfkVERETENZT8ioiIiIhrKPkVEREREddQ8isiIiIirqHkV0RERERcw5eKnY4fP94uXLgwFbsWSakZiwoBeGxcmxTXRNKEcaIQxVSR9KN4n3Qxx9OU9PwWFBSkYrciIk2SYqqISOw07EFEREREXEPJr4iIiIi4hpJfEREREXGNuJNfY0x3Y8xSY8x6Y8xaY8xtTlRMRERERMRpTsz2EAR+bq1dbYxpBawyxiy21q5zoGwREREREcfE3fNrrd1lrV1d/rgIWA9kx1uuiIiIiIjTHB3za4zpBQwH/lXDsunGmFxjTG5+fr6TuxURcR3FVBGRxnEs+TXGZAGvALdbaw8dv9xaO9Nam2OtzenYsaNTuxURcSXFVBGRxnHkDm/GGD+RxHe2tfZVJ8oUERERSTffXjAi5nWzM5cmsCbSWHEnv8YYAzwDrLfWPhJ/lURERBou1qRk3sTVCa6JiKQzJ4Y9nAtcC1xsjPms/N9lDpQrIiIiIuKouHt+rbXLAeNAXUREREREEkp3eBMRERER13DkgjcREXEHjasVkROdkl8RERGX0pcZcSMlvyJJYkvLGP/lJ3Q6cpBQzxy8/Xonbd/6gBMREYlQ8iuSBLboCGVPzeWS7XsACDy5BTvpInwXnpHimolIU6EvuSKxUfIrkmDh/AMEZr6M3Xew2uvB+UuxhYfxXT4a49GEKSIiTYmxqa6B1EbJr0gChbftouypV+Dw0RqXh95biT10GP/UyzA+b5JrJyIiTutQ0oLrNo3g9AOd2XrS64SHXIqni25Bnk6U/IokSGj9ZgJ/nw9lgTrXC69eT6DoKP4fTsZkNktS7URExFEWLtzdm+u+GkGLkB+APvt3UvbIP/BNOA/v6DMwHs0wmw6U/IokQPCTLwjOWQjh6ue9crv25cMeg7ht7SIoOlL5enjTVsr++gIZP56CaZ2V7OpGaci96zV+UMR5+htsnFSNe25T1owbN55Bzr5u0QtDIYJvvE9o7df4vzcBT4e2ju5bGk7Jr6RMUwzu1lpC735M8K0PopZ5x4ziRd9QMIaMW6dFxgHnHzi27Y69lD06G//0KXg6tU9mtUVcSxeJSbzOzO/Gj77MoXUgs8717JbtHHrocWaf+hlLun5d471x9T5LDvW/izjEhsMEX10Snfga8H37EvzfugBMJNp52p9Exi3TMD26VC9jfyFlf/kn4bydyaq2iIg0QsuAn5+uG8Uda8+LSnxLPUHe7LaBI/7qQ9kyw35+tOkM7vriAtqWNk9mdaUKJb8iDrBlAQJ/X0Dow0+rL/B58V83Cd/50b1LJqsFGT/5Lp6Bp1RfcKSYsr+9SGjtVwmssYiINNaQ/Z353coJnLe3V9SyL1sX8IuchTx/2mf84dyr8Aw4JWqdYfu78vuVEzhnT0/QrBBJp2EPJxidoks/9mgJZc+8it2yvfqCzGZk/OhKPKd2r3Vb0ywD/w+vJPjyO4Q++eLYgkCQwLPzsFeNwzfq9ATVXKTp84YN3Y+cxKlF7TjlUDt6H25L6YZnMV074enRmT6F7dmadZAybyjVVZUTgC0t44YvRzJ2Z5+oZUET4uVea3i9xwZs+TxnRZkt8d/4HR578iqu+Xo4zcsvhAPICmZwy/qzOaMgm2f75FKUUZa0dridkl8XUMKcOPbAIcpmvozds6/6gjZZZEy/KqbpbYzXg++746FNFqHFK6oUbgnOeQcKD+Mddw7GaC5gkboYa+h6tFVlontKUTt6Hm5Lhq0+jaA9XIDdXUB49Tp+xVhChNnespCvW+1nc+v9fN1qP9taHiTkUZecHBPesp3AP99i7L7oxHdrywM8PuBffJN1MGqZMYb/7bqZNW338B8bzmJAYadqy0fl96D/wY7M7LcyYXWX6hxJfo0x44E/A17gaWvtQ06U21A2FMJ+sysVu06avoUdYlovXKUXMuZtNm+vZUljPwBqSdbKX461XlC9PenCFpcSeDmSnFZlTm5PxvSrMG1bx1yWMQb/hPMxbVoRfGUx2GPHPPjOh9iDRXjPHAzA3ctviKnMB897tvJxQ983Df7dNMvA07VT/SvLCcUeOhy5OYuFijjQ/2DkC50BsObY48qfkf9DX249VlDVUGDMsacVX+hMlZUa+B3PFh4m/M0uwtt280zeldV61mLlxUPPI23peaQtF+8+FYAyE2Jr1gE2t97P5lb7I/Exid8/Y/0bNPZY7O53MNZYv63yccXvE2ponj32Suirb2per+qXclPDGkn+zt7QWBdrPB1ZkM3E7QOrxWaAMGEW9NjAK73WEPSE6yxjb/Mj/HrYUi7b3perNw+t9qXspEBz7lxzAYGX3sY36WJNe5lgxtr4vtkaY7zAl8BYYDuwEvietXZdbdvk5OTY3NzcuPZbE3ukmNL7/uJ4uSKxMqd0I+OGKzEtar7qd8aiQgAeG9em1jJCX2wi8NzrEAwmpI6JYHpl0+zWaamuxonIkdQgUTE1uHw1wVeXOF6uSFOwq3kRf+v/MZva7Kt1nezMpTw2rk3UGdhuR1rzk/WjOOVwu+iN2rbG/73L8J7Ww+kqN3Uxx1MnLng7E/jKWrvZWlsGvAhMcqBckROKZ0hfMm66utbEN1beIX3I+MnV0Dy+ckQkYl/GUVZ22M6LvT/nt0OXkvHT7+GbeBGe4f3ZnVmU6urJCeidrl9yd87COhPfumxveYj/GrGYV3quIcRxPcYHDhH4n9ewJaUO1FRq4sSwh2xgW5Xn24Gzjl/JGDMdmA7Qo0fjv83UNX41K5DBU1zZ6LJFGst77nC+6/sF9u1f1btudubSysd1vZ+7Dm7NLz6/kI6lLR2pozQtTsTU+q4HGLvjNG4gp1Flp0zL5ni6d8Z074yne2c8PbqQ3TqLbOD8KqtVXIjai4nYI8WEt+/GbttN+JvdhLftihrOJAJAmyz8Uy9jUr9e9fbyVZzpg7qvqQl/s4vAP9/E7t1f+Zr/22M09CGBnEh+a+pmjhpLYa2dCcyEyCk6B/YbJWQsG1vnA9C/3TAANuz/LObtK7ZpjMbsJ9Zt4qlXg5X/Ntfv+6zG1483oN3wysfr91eZ5quW37Ah+cc5kfsZcPJIPMMG4B01FPu6s2/rneU9A1dtGcKYzAscLdtpni6xjxEWZyQjph7MKDkWU3nukL0AACAASURBVNsPo8Zxuab6uF+Mqb7cwhcFkQt5jDVRocRgKuPFgEb8zZoMX/nMDV0w3Ttj2rVp8MWhpmVzvP16Q7/ex6p96NhYYrt9D7b4WC9c2sXuRg2eiXFc7vHHsrahkjbqQbWHjTlmaXWcPQbPqd3xjT4D4/BZOU+PLmT87HqCb39AaFkuniF98Ywc6Og+pDonxvyeDdxvrb20/PndANbaB2vbJp7xaQ2duSBZdxFrzH7SeRaGxtQtGe1Jt99nY9pfMQassfuRJiWlY36b4l0Wk0F/tw2Xrp8piRTLNR7HC3/1DaZzB0xWi0RVqymLOZ460fO7EuhjjOkN7ACmAt93oNwapeubPF3rJSIiciJy4+eqRxe5JUXcya+1NmiMmQG8Q2Sqs2ettWvjrpmIiIiIiMMcmefXWvsW8JYTZTnNjd8cJX5634iIiDRNTkx1JiIiIiJyQtDtjUVSSD3MIiIiyaWeXxERERFxDSW/IiIiIuIaSn5FRERExDWU/IqIiIiIa+iCNxERkQbQhaoiJzb1/IqIiIiIa/z/9u48Tq66zPf496mlO/tGNukQkrBvgUCzGTaRJSCyyggqoijRYeLgdkcZx6tXr3d0FjdwmbAMClFGCZsCMQEjmwRpkgAJYQmEJSQknX3v2p77R1Wn9+7qqlNV3XU+79eru0+d9fmdrvqdp37nd86h5beCaD0AAAAoL1p+AQAAEBokvwAAAAgNkl8AAACEBn1+AQBASXGNC/oSkl/0K1SgAACgGEV1ezCzfzezl83sBTO718xGBBUYAAAAELRiW34XSLrB3VNm9gNJN0j6WvFhoT+iVRYAAPR1RbX8uvt8d0/lXi6SNKH4kAAAAIDSCPJuD9dIeririWY208wazKyhsbExwM0CQPhQpwJAYXrs9mBmj0ga38mkb7j7/bl5viEpJWlOV+tx99mSZktSfX29FxQtyoYuDEDfFkSdyuccQBj1mPy6+1ndTTezqyVdIOmD7k5SCwAAgD6rqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqQd7tAQAAAOjTSH4BAAAQGsU+4Q1ATj63jZo1f2sZIgEAAF2h5RcAAAChQfILAACA0CD5BQAAQGhYJR7KZmaNkt4q8WZGS9pQ4m30ZWEvv8Q+oPx9v/wb3H1GsSuhTi0Lyh/u8kvsg75e/rzr04okv+VgZg3uXl/pOCol7OWX2AeUP9zlD1rY9yflD3f5JfZBNZWfbg8AAAAIDZJfAAAAhEY1J7+zKx1AhYW9/BL7gPIjSGHfn5QfYd8HVVP+qu3zCwAAALRXzS2/AAAAQBskvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEIjVomNzpgxw+fNm1eJTSNAs+ZvlSTddM7wCkcC9FsWxEqoU/se6keg7PKuTyvS8rthw4ZKbBYAqhJ1KgDkj24PAAAACA2SXwAAAIRG0cmvme1nZgvNbIWZLTez64MIDAAAAAhaEBe8pSR9xd0Xm9lQSc+Z2QJ3fymAdQMAAACBKbrl193Xuvvi3PB2SSsk1RW7XgAAACBogfb5NbNJkqZJeqaTaTPNrMHMGhobG4PcLACEDnUqABQmsOTXzIZImivpi+6+rf10d5/t7vXuXj9mzJigNgsAoUSdCgCFCST5NbO4sonvHHe/J4h1AgAAAEEr+oI3MzNJt0pa4e4/LD4kAAD6rkseODav+eoGLCxxJAAKEUTL73RJV0k608yW5n7OD2C9AAAAQKCKbvl19ycV0PPpAQAAgFLiCW8AAAAIjSAecgGgQPn2Hbz3wsUljgQAgHAg+QUA9Dl8MUQheN8gH3R7AAAAQGjQ8gsAQIDev25/ffidQ7VnwEPKHH2OIuP2qXRIAFqh5RcAgCC49HdvHKUvrDhZk3aM1KEb3lHip3cq8/o7lY4MQCskvwAAFCmaiei6l0/SJW8f0XbC7iYlfvk7pZesqExgADqg2wMCw4UGADpT7XXDwFRcX152io7cMq7zGdJpJe/4g3zLdkXPOF7ZB6P2X9X+/0T1I/lFQTzjmrxpraKekaeHyqKcROCAAITPqD2D9LUXT9PEnSPajN8VTWhQuqbNuNQf/iLfvE2xi8+URagzgUoh+UWvuLsyL72u1ENP6B/WNkqSEm8+pdh5p8pc8v7doAEAeZu4Y7j+6YXTtU9iUJvxawdu1/enPqYTd/2nPrb8MSmT2Tst/eRi+Zbtin/iAllNvNwhAxB9ftELmZVvK3Hjb5S89R55LvGVJG/crOSvH9D3njtHR28cL3kFgwSAMjhy0zh9a8lZHRLfV4dt0LemPaL1A3docd3Bis+8XBrQtgU4s+w1JX5+l3zHrnKGDCCn6lp+U396Spn1mxQ9fIoih06RDR5Y6ZD6vczq95R68AllXlnV7XyTd4zS1188QyuGr9dvpzyv14ZvLFOEwaMLA4CunPbeJF37ygmKedv2o7+Nfkc/O2yREtH03nHRg/eXzfq4Ejf/Xtq6Y+94f3utEj+9U/FrL1dkzMiyxQ6gCpPf9OKX5I2blVmyQjKTTapT9PADFDl8imz86LwuNMg38ZGqO/nJrN+o1LwnlVn6SufTJbmZot62qfewrWP1nSVn67l93tXvJr+gt4dsLUO0AFBa7q70gqf19y+f1GHaw3Wv6I4Dl8qt46mvyL5jVHv9VUrcfHfbs2Ybtijx0ztV85nLFJm0b0ljB9CiqpLfzPpN8sbNLSPc5atWK7VqtfTgY9LIYblE+ABFDpwoi1dV8QPjm7cpNf8ppZ9dJmU678MQOeIA/eeIaUpGovrn7Us7TZCP21inaRv31V/HvqXfT16m9QN3dLImAOj7PJ1W6u4FSj/zQodpdxywRA9NeEXqpm3FRgxVzayPKXn7fcq89lbLhJ27lfjFXYp/4sOKHnVQCSIH0F4g2Z+ZzZD0E0lRSbe4+/eDWG9vZV7u/rS8Nm9T+qklSj+1RKqJK3Lw/oocdoCih0+RDR9aniD7MN+xS6lHF2X3Tyrd6Tw2ZYLiHzpdkcl1Wjs/26Jbc8lFypz5nv52+/d1zKa2rRcRmU5ZP0knNU7Uwve9rnv3X15UjEG1yns6Le1JyJsS0p6m7HAiKauJSbW1GrN7sHbHktodTSodoRMzEGa+p0nJXz/Q4RiTtLR+ftgiLRqb30MsbGCt4td+RMnfzVOmoVVdmEwpefu98ovPUuzU/Os4AIUpOvk1s6ikn0k6W9JqSc+a2QPu/lKx6+6t6CnTFNlvvNIvva7MS6+3Ob3UQSKpzLKVyixbqZQkmzBONn60ZKbPvXOCpOx1Wy6XrOUaLs8NNZ/aSt49v/ug2nUJyK7U9/54q+GWH2VbXD3T8tpMMtPT7/1Zbq6MXG5SxlwuV6bVOJfr3MkfyS6Tr2RK6edflpqSnU62urGKnX+aIodO7rTrSGTCeP1g6uM6dMsYXfHGVB2ybUyb6TGP6Ow1B+m09yYrkXxQVlujh1f9Lu/wzpv8d5KkT686Lq/5TVLiNw+2JLZ7mqSmhHxPLtlNprpd/qf68N7hRCSl3dGUduWS4d17/6aUTMzfu5/zjS05d8He4d4uEzvn/bKhg/NaBuhJT18mD90yRievnyip5TNYiLzf53fP31v3fe6tExRxk8kUccsNq9Vw9m/ivd9LEcvVkZGW4b1/I9k6q3lc8y3GelE/Zt5Y3eF4siPWpP848km9MqKb40wnLBZV/MrzlRo5TOkFT7dMcCl17yPKrHxbNqxvf8YLqetKpvkYmztWXvvW8ZJnG14kyXLvG5PJvOVvYsv9slEjFL/g9NLHiD7HvH1y1tsVmJ0s6dvufm7u9Q2S5O7/2tUy9fX13tDQUNR28+Gbt+1NhDOvvdVlaya6ZmNGKnbeqYpMPUQWaXuwmJVr+b3pnOGSWh1IXZq2cV99dNVR2n8nF3IEqebrn1Fk7D6VDgPBCeTmgIXWqT0lv2e/e6Cuea2+0LCq1voBO/SDox7TmsHbu52vbsDCvfVjZ1JPP6/U3Plddi9DaVndWNV+5VOVDgPBybs+DaLbQ52k1ud8Vks6sUNEZjMlzZSkiRMnFryx3lyFbyOH6fKNn5HGSbWjozpi8zhN27ivpm3ct8PtadDWxtpdmrv/Mj02fpUyb/9Cervz+eoGLNw73L6bgWdcmSUrlJr3pHzjllKGC4ROEHVqTxfspp5aotRrZWi960dswjjt99nr9LNh3+l2vubGAan749YxR7xP1y9/vwZkuOdv2fGdI7SCSH47y7Q7vKXcfbak2VK2lSKA7fZKUzStxaPXaPHoNZJL++8YoWNzifAB2/fZe4ok9AYPVOyDJ+lLm/9eyWim5/m7YRFT9LjDFTnmEKWfeUGp+X+Vtu0MKNDiZOQdujA0RVKaOuI4aU9Tm77AHbquAH1ApevUMIocNkXxT14oq63peeZWevqSkXnnPSVumStt7xv1Y2hQt4dWEMnvakn7tXo9QdKaANZbOia9NXSL3hq6RfdOeknDErU6dMsY/a+p38tOb+5nq/Z/2w277+03NvuFf22e2om2Yz93zDcki+im5/9Prv9utg9xc99dz73O9uF13XDCj7ILZlr1C8609Af2vcPedp7e7pYhg7J9egfUKvlAcYlvm/VGo4q9f5qi9Ucq8/Iq+bby3PXBamuk2hp9e/H1rZLcpHZFU2qKpjr92nbvhT9s89rdpUQy11+4uf9wLjFuyl0wV0Y2pG/3BUR1iUyeoNilZ7UZN/vF/K5nnnnU1wvcaK7vbq6PrrXuq9vcd7d1n16ztnVfc33Yqi70Tsb1lo0anr13fCT4hpLIfuNV+0/XKL18Zba+Uf77WWrZ1yX/3xShkNjyXmbqDdkBs2y9btlevs3De/t8t59nQG3e8aO6BJH8PivpIDObLOldSVdI+lgA6y2bbTVN+tvY1YqdcFTB67huev4XbzV7bH0Pd6fIiR5ZHbe/sZq4olMPLvt2l725ruBlzUzKJdE2bEiAUQF9X2TfMYrs2/bi1etO6X1dh57Z4IFtjkELNq3Me9nrTjm2V8s0z19OhcSW9zLTpxUUE8Kr6OTX3VNmNkvSn5S91dlt7l7c/awAAACAEgjkPr/u/pCkh4JYV9Cq+QlsAAA043gH5IdHnFUQFRUAAEB5kfwCANDH0DjCPkDpkPyi6lGBAgCAZv0u+SWRAQAAQKH6XfKL8sjnS0brJxgBAAD0ByS/AACgpDhri74kUukAAAAAgHIh+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFRVPJrZv9uZi+b2Qtmdq+ZjQgqMAAAACBoxbb8LpB0pLtPlfSqpBuKDwkAAAAojaKSX3ef7+6p3MtFkiYUHxIAAABQGkH2+b1G0sNdTTSzmWbWYGYNjY2NAW4WAMKHOhUACtNj8mtmj5jZsk5+Lmo1zzckpSTN6Wo97j7b3evdvX7MmDHBRA8AIUWdCgCFifU0g7uf1d10M7ta0gWSPujuHlRgAAAAQNB6TH67Y2YzJH1N0unuviuYkAAAAIDSKLbP702ShkpaYGZLzeyXAcQEAAAAlERRLb/ufmBQgQAAAAClxhPeAAAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFB8gsAAIDQIPkFAABAaFglnkhsZo2S3irxZkZL2lDibfRlYS+/xD6g/H2//BvcfUaxK6FOLQvKH+7yS+yDvl7+vOvTiiS/5WBmDe5eX+k4KiXs5ZfYB5Q/3OUPWtj3J+UPd/kl9kE1lZ9uDwAAAAgNkl8AAACERjUnv7MrHUCFhb38EvuA8iNIYd+flB9h3wdVU/6q7fMLAAAAtFfNLb8AAABAGyS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQiNWiY3OmDHD582bV4lNA6iwWfO3SpJuOmd4hSPpEyyIlVCnVgc+G0BR8q5PK9Lyu2HDhkpsFgCqEnUqAOSPbg8AAAAIjYp0ewDQt13ywLF5z3vvhYtLGAkAAMEquuXXzPYzs4VmtsLMlpvZ9UEEBgAAAAQtiJbflKSvuPtiMxsq6TkzW+DuLwWwbgAAACAwRbf8uvtad1+cG94uaYWkumLXCwAAAAQt0AvezGySpGmSnulk2kwzazCzhsbGxiA3CwChQ50KAIUJ7II3Mxsiaa6kL7r7tvbT3X22pNmSVF9f70FtFwDCiDq1/+jNBaR1AxaWMBIAUkAtv2YWVzbxnePu9wSxTgAAACBoRbf8mplJulXSCnf/YfEhAQhavi1P3LYMAFDtgmj5nS7pKklnmtnS3M/5AawXAAAACFTRLb/u/qQCej49AAAAUEo83hgAAAChQfILAACA0CD5BQAAQGgEdp9fAOiJ79qjY9au1Jqh+0gaXulwgD5jSLJGx22oU2LkOvHZAEqr6pPf3txcnNs8AaWTeXONErfO1Sd27lZGptToMxU77bhKh4Ve4rZ5wTtg2yj904unaVhygKT7lBx0omIfOk3ZO4kCCFq/S36peIH+J73sNSXv+IOUTEmSInKl7ntUvnW7Yh86XRbhII9wmrbxffrH5dM1INNyOE7/+Rn5th2Kf3SGLBqtYHRAdep3yW81IZFHGKSeXqrU3Qsk7/gE3vTCv8m37lD8ivNkMQ7yCJcz1k7WZ185XtFOLr/JNCxXcvsuxT91kay2pgLRAdWLC94AlIS7KznvSaV+P7/TxLdZZvFLSt58t3xPUxmjAyrIpUvePFyfe+XEThPfZplXVinxs9/Kt+8sY3BA9aPlF0DgPJ1R6u4/Kf3Mi20nmOmRycfoxNUva2hi997RmdfeUuKm36rm2stkw4eWOVqgfMxNn37tWJ295qA24zNyzZ20TB9YN12jd2/bO95Xr1Pip3MUn3m5ImNGljtcBKSQM72cHS4dWn4BBMqbEkredk/HxDcWU/zTF2vewSfoxpMulrU7kPua9Wr66Rxl1m0sY7RA+cTTUX1p+fQOiW/C0vrREU/qnknLddNJF8kmjGsz3TduUeLGOcq8s7ac4QJVi5ZfVAx34qg+vn2nErfMlb/zXtsJgweq5jOXKjKpTlqzVZsGDVPNFz6enfftVgf0zduUuHGOaj5zmSKT68obPFBCg5M1+uqLp+rQbWPajN8RS+g/jnxcr4zYkH1dO0g1112h5K/uV+aVN1vNuEuJn92l+NUXKXrYlDJGjvY4dvV/tPwCCERmw2YlbpzTIfG1UcNV84WPZRPf1uOHZA/ykcMPaLuiXXuU+MX/KP3ia6UOGSiLffYM0reXfLBD4ruhdqe+Pe2RvYlvMxtQq/hnLlOk/oi2K0oklbz1HqWfXVbqkIGqRvILoGiZd9Yq8dM58g1b2oy3urGq+cePKzJ2n06Xs5q44p++RNGTpradkEopeft9Sv11SalCBsois6ZR31l8libsavvgircHb9G3pj2idwdv63Q5i0UVv/J8Rc88sd0KM0r+9iGlHl0k7+ZCUgBdo9sDgKKkV7yh5K/ulxLJNuMjB++v+Kculg2o7XZ5i0YUu/xc2fChSv3pqZYJ7krdvUC+ZYdi553CDf/R76RXvq3kbfdoVGJQm/Erhq/Xfxz5hHbFk10smWVmil9wumz4EKXue1RqleumHnxcvnWHYhefKYuUtx2Li7fQ3wXyiTGzGWb2ipmtNLOvB7FOAH3faWsnK3nr3I6J73GHK/7Zj/SY+DYzM8XOna7Y382Q2j3wIv3I00re9bA8nQ4sbqDU0ktfVvK/fi/tSbQZv2jM2/rXqX/pMfFtLXbqcYpfdaHU7oEX6ScXK3nHH+S5h8cAyI8Ve9rEzKKSXpV0tqTVkp6VdKW7v9TVMvX19d7Q0FDQ9rr79jg4WaMfP/MhSdKQePYU047k1rzXXcwyhch3O8Vsoy8r137uewr4zHnul7ca0Tzc/BnuME+OSYlMk3zvlnO/reW1514PiQ+T1Iv3Zqpjchs980TFzj+ty6e2zZqfXfdN53T+P02/9LqSv36gY0J9yORqvOF/IM3ZxdSp3UktekGpPyxsM25Hcnteyw6JF3rLuipp4d+9p8OoeXWv6tcHLpFb13VA3YCFXX82Vr6t5G33Su3viV0T75AYl1Ihx65qOt6VMq8o9z6r+cLHFBk/uuDl+5i8K48guj2cIGmlu78hSWZ2l6SLJHWZ/JaKqdXBOJWteIYov5anYpcpRN7bKWIbfVm59jOkmnw/6oV8BpqZFLv4LMVOzf9K6M5EDz9A9vdXKHHL3dLOVvcCzt3wv2bm5bIhg7pZAwKTTku72yZaQ5Tnl48UDy1p7TdTluoP+71cVG4fPXCibNbHlLj599LWHS0TEklJ+bckF6uQY1c1He9KmleUe59lMoUv248FkfzWSXqn1evVkk5sP5OZzZQ0U5ImTpxY8Ma66w/kO3er6akbC143gALFovrRIY/pb5vvkh7oefa6AdnWxO7O5Iw/Yoi+/sIZGrdnSJvtqCZebLRVIYg6tad+mGe/e6CuUX1B60ZOJKL4Fefpmvp/0jU9zNp8VkTq/n8z6rBBuuGF0ztcRAcgP0H0+e3se2yHczruPtvd6929fsyYMZ0sAqBfGjlMNZ//O/1tzOpAV/veoB361rEL9MaQTZIkGztKNZ+5TEbyK4k6tV8YNljxay9TtP0ty4q0acAufXvao1oyak2g6wXCIoiW39WS9mv1eoKkynwiBw5Q7Xe/UJFNXzXvA3nPe8eMhT3PVOR2itlGocoRW9Xs59ydC656+PT8tnPeY7mvmaaPP3yqJLX04TVv6f6b++0m/e6CRbmZmvsEt/pO2lk/4UL6/5tJA2uzd2Iowa1Ht9Y06TvH/FnXvHaczrr2c7LBA4PfSIj1dGW9J1NSABdT5f0+n/GX4rZTwOe2pMtc+nTJ7lKyM57Qv019XLWpmH577mO9i6sCx4dCVNMxpVx6vc/yvCi52gSR/D4r6SAzmyzpXUlXSPpYAOvtNYuYVKGD4854oueZcoo5gN952aKCly21fPdBMeUv134uR1my28mvn54NGrB3eHcsv2TEYtVxJ8OmWEq/OOwZnTOKU7zlZvGYFC/+fZT3+7zoz1PvP7clXaYMt+driqX2xtaXjw+FqKZjSrmU69jV3xVdq7l7ysxmSfqTpKik29x9edGRAQAAAAELpGnI3R+S9FAQ6+qvuDE3AABA31cd50URGnzJAAAAxSjvMxEBAACACqLlFwBQUpyx6T32GVA6JL8AAKDP4QsASoXkFwAAoArwhSE/9PkFAABAaNDyC3SCb88AAFQnkl8EhoQRQH9DvQWED8kvAAC9QMIM9G/0+QUAAEBo0PILVFDYW5DCXn4AQPnR8gsAAIDQIPkFAABAaNDtAQAAhBJdr8KpqOTXzP5d0oclJSS9LunT7r4liMAAdI7KGgCAwhXb7WGBpCPdfaqkVyXdUHxIAAAAQGkUlfy6+3x3T+VeLpI0ofiQAAAAgNII8oK3ayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeL3hz97O6m25mV0u6QNIH3d2DCgxA/5LvhXiz5m8tcSQIKy4GBZCPYu/2MEPS1ySd7u67ggkJAAAAKI1i+/zeJGmopAVmttTMfhlATAAAAEBJFNXy6+4HBhUIAAAAUGo83hgAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2rxEPZzKxR0lsl3sxoSRtKvI2+LOzll9gHlL/vl3+Du88odiXUqWVB+cNdfol90NfLn3d9WpHktxzMrMHd6ysdR6WEvfwS+4Dyh7v8QQv7/qT84S6/xD6opvLT7QEAAAChQfILAACA0Kjm5Hd2pQOosLCXX2IfUH4EKez7k/Ij7PugaspftX1+AQAAgPaqueUXAAAAaIPkFwAAAKFB8gsAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAEKD5BcAAAChQfILAACA0CD5BQAAQGjEKrHRGTNm+Lx58yqxaXRh1vytkqSbzhle4UiAULEgVkKdGl7U3cBeedenFWn53bBhQyU2CwBViToVAPJHtwcAAACERkW6PQBAoS554Ni85rv3wsUljgQA0B8V3fJrZvuZ2UIzW2Fmy83s+iACAwAAAIIWRMtvStJX3H2xmQ2V9JyZLXD3lwJYNwAAABCYopNfd18raW1ueLuZrZBUJ4nkF31CvqfJJU6VA+gbelNv1Q1YWMJIgOoT6AVvZjZJ0jRJzwS5XgAAACAIgV3wZmZDJM2V9EV339bJ9JmSZkrSxIkTg9osAIQSdSqA9rggOD+BJL9mFlc28Z3j7vd0No+7z5Y0W5Lq6+s9iO0CQFhRp6I/ISlDXxLE3R5M0q2SVrj7D4sPCQAAACiNIPr8Tpd0laQzzWxp7uf8ANYLAAAABCqIuz08qYCeTw8AAACUEo83BgAAQGiQ/AIAACA0ArvVGQAAQH/CQ5DCiZZfAAAAhAYtv5C7a8qmNRqS2C1PTpXFeVsAAIDqRJYTcp5KK/n7P+m6Z5dJkhJrFit+7UcU2WdEhSMDEGY8FKEHLp2ybpLqN9Rp7egX5amTZbFopaMC+gWS3xDzXXuUvP0+ZVa+3TJu/SYlfnKnaq65VJFJ+1YwOgBAZ2rSUX3u5RP0/sb9syM2/FWJX76tmk9dLBsyqLLBAf0AyW9IZTZtVfLmu+XrNnacuGOXEj+/S/GPf0jRow8pWQxcaAAA9WFJZwAAHilJREFUvbPPnkH6yrJTNHnHqDbj/Y3VavrRr1VzzSWK1I2rUHRA/8AFbyGUeWutEj++o/PEt1kqpeSv7lfqz8/I3csXHACgU4dsGaPvPXdOh8R3r83blPjpHKWXvlzewIB+huQ3ZNIvvqbEz38r7djVZvyqEeO0cPLRHeZP/fExpe6eL09nyhUiAKCdD645QP/y/Ac0PDmgzfimSKrtjMmUkr9+QMmHnpBnaLgAOkPyGxLurtRjDUrefq+UbFtZRo45VP91/AV68JCTFLviPCnS9m2Rfvp5JW+dK9/TVM6QASD0opmIrnn1OH321eMV87Z187OjV+sfTr5fT0w8ssNy6UeeVvK/76HeBjpBn98Q8ExGqfv+rPSTHfvNRs88UbHzT1PqkW2SpNgJR8lGDMsmyXsSe+fLvLxKiRt/o5prPyIbMbRssQNAWA1N1OpLy6frsK1jO0ybu/8yzZ20TG7S/YdP1wdOmqDU3PlSq7N0meWvK/GTOxW/5hJFxnTRVQKhF8Y7q9DyW+W8KaHkf9/bMfGNmGKXn6v4BafLItZmUvTg/VXzj5+QRg5ru661jWr68R3KvLuu1GEDQKjtv32EvvfcOR0S3z2RpH50xJO6e3I28W0WO2mqaq67Uho6uM38vm6jEj++Q+lXVpUjbKBfoOW3ivm2HUrcMle+ul2yWluj+NUXKXro5C6XjYwfrdrrP6HErffI33mvZcK2HUrc+BvFP3mhoocfUKLIERbc8QPoKL30Zf2fJWepNtP2EL1+wA7955FP6O0hWztdLjK5TrVfvEqJ/763bb2/u0nJ2XfLP3yGoqfXy8w6XR4IC5LfKpVZ26jELXOlzdvaThgxVDWfvUyRfTueRmvPhg1RzXVXKDnnj8osW9kyIZFU8tZ75Jeepdj0aQFHjv6KRDYcwniKtFw840rNe1LpR55WbbvD8/IR6/STw5/S9ppEF0tn2chhqpn1MSV/N0+ZxStardyVemChMmvWK375uf3iSZ6811Aqgbz7zWyGpJ9Iikq6xd2/H8R6e8tTaWVee6sSm9Z3F83Kc07TN0+6sePoQm8n1sk3eN++U6n7Hm3TZ1eSrG6saj57mWx4/n12rbZG8U9drNQDC5V+/Lk28abmLpCv26jIwft3HksPr4/eOD7vONIr3sh73vbKtZ2qkMlIqbSUSslTaSmdzr3O/nhumtKZlr8mKRbTJ946RqlIRinLKBnJKBVJK2nZv6lIRknLKJkbbt7Phfxv8l2m3P/LyAH7yWriZd1mqfnmbcq8t6HNuEL2f7714zdPuqmo7fR36b8uUWb56x3G/6nuVd1xwBKlI/kdJ6wmrvjHL1B637FKPfiY1GqxTMNyJdZtVOzc6Z0ePwKVyWQvsE6mdNa7ByieiaomE1NNJqradFQ1mWh2XG64NhNT4t3/keIxqSamz689UYloSolIOvsTTbcMR9J7p4X++JDJSImkPJHUuasP0oB0TLXpWMvfTLvX6Zialt8s1cSl2hp9bctp2hNNqSmS1p5oUk3RdPZ17ic7LVXW8tuAWkUm15Vu/cXew9XMopJelXS2pNWSnpV0pbu/1NUy9fX13tDQUND2uvsmOCRZo5ufurSg9Va7yOFTFL/qQlltTafTZ83Pnka76ZzhXa4j9cTibFLNfX+BDmr++VpFRo/s7WKBZB/F1KndST21RKm5CwJfL/KTsrRuO+g5Ldy3+6SjbsBC3XTO8E6Pj0dvHK8vvPR+DU53XvcDfZFNGKfaL1/d68XynTGIC95OkLTS3d9w94SkuyRdFMB6EZDo+49R/NOXdpn45it26rGKX3NJ9tsiAKBktsR36zvH/LnHxLcnz+/znr553AK9O3BbzzMDIRFEt4c6Se+0er1a0ontZzKzmZJmStLEiRML3lh3fXt85241PdVJl4KwMin24TN0+dbPSw/2PHvdgIV7h7trYZ901Ej904unaWRiYBBRAihAEHVqT30qz373QF2j+oLWjcLZhHEa9+lL9G8jv9XjvM1n7aQejo8XNSl55x+U6aun7oEyCiL57ayZucN5cXefLWm2lD1FF8B2O4pGFDmk7R0Mlqz/a96LTxv7/oKX6ZXcHlu8/qk2o7vaKceNPWXv8HPrn2y/mk4dW3eGoicfnb2jwwO9D7E7bw7drG8cN18XvX2Yzhtxvpojf/a9x7NxeXN8bSM0SeZW2D7LKfn/ply6eA905dix03u/jVZvqHz327T3TZeiMT2xbkGu726uv24krZRl2gwnIxmlLaPrjv6XbJ/gdOt+wa37C6c6jitzz5m8y1/Ee6bcFxCVo07dWLtLz49cK6mPf55yyvF/Lul2oqbIwZMUPenowPuP28BaxT9zqdINy7MXMLd72FFv5F3+8dOz/XfjseznIx5vNdzqdU3LsEx7+wl7MiUlkm1fJ5NSIiVPpXLjk1K65e1fjvdAKfOK1nHlvcy46VJtPPueyfXjtZq4VJt9bTU1ueGalnmiESmRkJqyfYWVSMqbEtn93Twu9zo7PSGlWu4hXdLyjH2/bPSIvOYtVBB9fk+W9G13Pzf3+gZJcvd/7WqZUvVP60whV6CX66r1Qq5kLeUyzf3Gyhlbb1XbHQX66tXM7OeyqWifX/7PfXs7+cjneo2g9aXyt9dXj0N99ZhaqD5anrzr0yCaKp6VdJCZTZb0rqQrJH0sgPVWTH+o5AEAANB7RSe/7p4ys1mS/qTsrc5uc/flRUeGfqccXxr4YgIAQP/Tl47fgXRSc/eHJD0UxLqC1pd2NgAA6N/KlVeQv5RO33/EC9rgwwAAQP/CsbtvIfkFgJDigAwgjEh+AQBVgdPRAPIRxBPeAAAAgH6Bll8AVY+WOgBAM5JfAACQN75Mor8j+QUAAEDe+vsXIJJfoIL6ewUCAH0JdSryQfJbQXxIAQAAyovkF50iMQcAANWIW50BAAAgNGj5BdABLf8AgGpFyy8AAABCg+QXAAAAoVFU8mtm/25mL5vZC2Z2r5mNCCowAAAAIGjF9vldIOkGd0+Z2Q8k3SDpa8WHhSDl039z1vytZYgEAACgsopq+XX3+e6eyr1cJGlC8SEBAAAApRFkn99rJD3c1UQzm2lmDWbW0NjYGOBmASB8qFMBoDA9Jr9m9oiZLevk56JW83xDUkrSnK7W4+6z3b3e3evHjBkTTPQAEFLUqQBQmB77/Lr7Wd1NN7OrJV0g6YPu7kEFBgAAAAStqAvezGyGshe4ne7uu4IJCQAAACiNYvv83iRpqKQFZrbUzH4ZQEwAAABASRTV8uvuBwYVCAAAAFBqPOENAAAAoUHyCwAAgNAg+QUAAEBokPwCAAAgNEh+AQAAEBokvwAAAAgNq8RD2cysUdJbJd7MaEkbSryNvizs5ZfYB5S/75d/g7vPKHYl1KllQfnDXX6JfdDXy593fVqR5LcczKzB3esrHUelhL38EvuA8oe7/EEL+/6k/OEuv8Q+qKby0+0BAAAAoUHyCwAAgNCo5uR3dqUDqLCwl19iH1B+BCns+5PyI+z7oGrKX7V9fgEAAID2qrnlFwAAAGiD5BcAAAChQfILAACA0CD5BQAAQGiQ/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS8AAABCg+QXAAAAoUHyCwAAgNAg+QUAAEBoxCqx0RkzZvi8efMqselQmDV/qyTppnOGVzgSAD2wIFZCnQrkj2Nk1cq7Pq1Iy++GDRsqsVkAqErUqQCQP7o9AAAAIDRIfgEAABAaRSe/ZrafmS00sxVmttzMrg8iMAAAACBoQVzwlpL0FXdfbGZDJT1nZgvc/aUA1g0AAAAEpuiWX3df6+6Lc8PbJa2QVFfsegEAAICgBdrn18wmSZom6ZlOps00swYza2hsbAxyswAQOtSpAFCYwJJfMxsiaa6kL7r7tvbT3X22u9e7e/2YMWOC2iwAhBJ1KgAUJpDk18ziyia+c9z9niDWCQAAAAQtiLs9mKRbJa1w9x8WHxIAAABQGkG0/E6XdJWkM81sae7n/ADWCwAAAASq6FudufuTCuj59Pm45IFj85rv3gsXlzgSAAAA9Dc84Q0AAAChEcRDLgAAACou37PDdQMWljgS9GUkvwAAoKTosoi+hG4PAAAACA1afquMb9qqa557WGN3blEqOk3RD5wgi5TtekQAAIA+jeS3imRWv6fEzXN1+PadkqTUg48ps2a94leeJ4vxrwbQf3CaHECpkBFVifSKN5T81f1SItlmfGbJCiW27VDNpy+RDRqQ9/o48KCa8H4GADQj+a0CqUXPK3X3fCnjnU73199R4sY5qrn2I7JRw8scHdC1fJNSicQUQPEiGdOU7aM0ILNdEsfDQvX3BgWS337M3ZWa96TSC57uMG1HfICGJPe0zLtuo5p+cqdqrr1MkQnjyxkmAAAVd+SmcfrUymNVt2u4pN8oseNwxS88QzZsSKVDQ5mR/PZTnkor+bt5yjQsbzvBpHsPna4l7ztQ3131iHzVuy3Ttu9U4qbfKn71RYoeNqW8AQMAUAH77Bmkq1ZO04kb9mszPrP4JTUtX6nYudMVPfVYWTRaoQhRbiS//ZDvblLy9vuUee2tthNiMcWv+rCeWjtWklTz+Y8q+ZsHlXn+lZZ5Ekklb50r/8g5ip10dBmjBoC+p7+fvkXXYpmIPvTOobr4rcM1INNFutOUUOqBhUo/84Jil56l6EH7lzfIPiCMnwGS337Gt2xX4ua75Wsb204YPFA1n7lMkUn7Smu3SpIsHlP8qguVGrFQ6ccaWubNuFK/+5N88zbFZpwiM26F1p+EsaICUP2GJmrk23ZIQwcXfVw6euN4Xb3yWL1v97C85vd1G5X8xf8ofcyhil/4AdmIoUVtH30byW8/klmzXomb75a27mgz3kaPUPzayxUZM7LDMhYxxS86UzZyuFL3Pyq1uiYuveBp+eZtiv/dDFms+NM9JGUAgHzF01EdvmWMjto8Xkdvep8m7Bqupr/+XDZquCKHTlbk4EmKHLS/bGBt3uscvXuwPrlymo7fOKHT6SuHbtSvD1ysaXu+rUtWNUh7mtpMzyx9WU0vva7YOe9X9LT6QI6N6HtIfvuJ9CtvKnn7fVJTos14239f1XzmUtmQQd0uHzvtONmIoUre+Ucpldo7PtOwXMmtOxT/1MW9qmAAAOgVl/bbOVxTN43X1M3v06FbxqjGOyaXvmmr0n9dqvRfl0oRk+1fp+ihkxQ5eLJsv3GySMeH03oypfSfn9F/Pnueajrp4rA91qS7pjyvhe97Q27SrnFH6oorjlHqj48p/eyytjMnktnxf3tRsUvOUvSQSUHtAfQRgSS/ZjZD0k8kRSXd4u7fD2K9yEo/u0zJ/5knZTJtxkeOOkjxj18gq4nntZ7o1INl131UiVvvkXbu3js+89pbStz0m+yt0DjVAwDdqqazXKUui+/Ypcyrb+rzK07U1M3jNTIxsHcryLh81WqlVq2WHn5SGjQg2yJ8yCRFD5ksGzFU6eUrlbrvz/KNW1TTLq3JyPXoviv1P5Nf1M54u8ajoYMVv/J8RU8+Wsm5C+Tvrm8b+/pNSv7X75SeenDuDGp+XSjQ9xWd/JpZVNLPJJ0tabWkZ83sAXd/qdh199aAVEyffu04SVJix4Ol36B7rhtB7q97y7CUu+9u63la9Tkwk2SS5Yab+zdZbpxy4xJJZZa91mHT0VOPU+yiD3T6Dbg7kUl1qvnHjys5+275xi0tRVnbqKYf36HIwdnO/n//zol5ra/1fi5kmbx4J/cv7jCq83sct9VJH7L2o/pB/+dC9vNf3vljXsucsd8FvQvGc+/rTPPfTKthlzz3OpORt54v957/5qYzlTHP/WSUUXbYTXuHm3+K+Uz3dp/FzjxRkfGjC95etUi/+qbS7e8oU0oZlzJpKZ3RV9ecqlgmoqhHFPOIohlrGXZTNDdtz9JfZOvBaESKRrN/Iy3D1jyueXokIkVa6tw+VddVQCnrbV+3Uf7uOsml0zW5x0UyyigSjUvpdNcz7dqjzNKXlVn6slKSNGKotGV7p7O+OmyD/vug5/Tm0M3dbjcyqU41X/qk0k8/r9RDT0i797SZnnnhVTWteEORIw/MvoeqTN7vge2544hL/7D6pDbTLHcwbXMEdSmx5QFZbY3iH50RQKTBMe8ssejNCsxOlvRtdz839/oGSXL3f+1qmfr6em9oaOhqcre6+5Y6JFmjm5+6tKD19iexCz+g6On1XV4QMGt+9oK3m87p+gbevmOXErfMlb+9tiQxAv1Z/PMfVfTgslz1Hci3rWLq1O6knlqi1NwFga8XaNZYu1PPj1qrF0a9p+Uj1unOC55Q5o3Vyry8SplXVsnXb+r1OrfG9+i3U57X4+NXybv4hNUNWNjpMdJ37FLqoceVfuaF/NpU0LPBAzXgu18ox5byrk+D6PZQJ+mdVq9XS+rwNcLMZkqaKUkTJ04seGPdnXrxnbvV9NSNBa+7z4tF9eODH9cz2+6S/tD9rHUDFu4d7uoLQ83+Uc3aeXKXFwYA6LuCqFN7OuV99rsH6hrVF7RuoFM1cUUO3E+RQyYrcshkTRgzUvuZqfU5p+hhUxQ9bIoueeBY7TNlUK6P8HgduXm8hqRqul63maLTp2nseafoiwMH6ItdzNbcQCR18RkYIB0wbZQ+9dpxOnD7PgUVE60U2chaCkEkv51l2h1PSrvPljRbyrZSBLDdcBk0QDXXXKpnls0JbJWJaFo/OvIpfXLlNM149+DA1gug9KhT0V9Y3dhcsjtJkcl1slj+qcfGAbu0cN83tHDfN2RuOmDbKB29ebyO2jReB23bRxFlu/7Z5AmKX3qWInVjA4n59WGb9L+PXaAPrJ2iK944WkNTXBBeTYJIfldLav3YlAmS1gSw3t6rjSt+5flFr+YnS/533vNef+x3JUk/WvIvcnm2669lj0PZHr/Z/oue+z7gJt1w/A8ld/3g2a/KPNtXpvkbhHl22GTKrsb0xRP+nyIHTsze0aHdRanFcnP96qDFemLcm/q3Q34c7Mp7UMh+/vHib7Ya2/Z43/701peO/b+SpB8t/pfcvmzPOry6ftp38o6pvXzL03obhSxTDr3+30RM/7H4hmz/3L39dr1Dv93m198/5b+zF7JkMi39g1sPe+vXuX7DvYytmH0WGU9rjyQtG7lOPz90kaSW/VnS9/mx323VZ7dVv91WfXjb9N2NRmRm8nRGymT0pUc/oqhH9vYHjrm1em25/sIRfemY7/ZyT5Rfrz6DRfxvCpH3dk76fjbZHTq4qO01c3OtHL5RK4dv1NxJyzUoGdfB20brm6f9UjZlQuD3rHeT/rzvG/rr2Ld1xJaxuuGowq/lL9fxoeTvgc72sbUbaD2LWZ/sJx1E8vuspIPMbLKkdyVdIeljAay31ywWU/T4I4tezxNr38x73i/XHyFJemrNWz3M2SJ61EGSpGffXJ3X/F855tC8112oN4ZtCmTf9UYh+/nJNfkv89Xjcv+bd/P/33y5iH2Qb3lab6OQZcqhkP/NotXv9DBni8jkwrva9NV9Vo3WDtqutYOyFxM178+Svs9z76Xeaj7Wvj1ka7fzNftqP3hvfPn4e3q9TLk+G3lvZ2ppzyjuiie1dJ+1ihywX88zd6Fcd+Qo1/GB+jE/RSe/7p4ys1mS/qTsrc5uc/cyXh4MAChEf7gVF/oe3jflwX4unUDu8+vuD0l6KIh19QW84QCgc9SPfRf/GyA/POENFUNF3Xf3QV+NCwD6I+rUvoXkFwgIlVt5sJ8BAMUg+Q0BkgUAAICs3j0bFwAAAOjHaPmtIFpky4P9DAD9D3U3SoXkFwAAoArwhSE/dHsAAABAaNDyCwAoKVqjAPQltPwCAAAgNEh+AQAAEBokvwAAAAgNkl8AAACEBskvAAAAQoPkFwAAAKFR1K3OzOzfJX1YUkLS65I+7e5bgggM6Ay3TALQFeoHAPko9j6/CyTd4O4pM/uBpBskfa34sPofKl0AAIC+r6jk193nt3q5SNJHigsHQH/FF0AAQH8QZJ/fayQ93NVEM5tpZg1m1tDY2BjgZgEgfKhTAaAwPSa/ZvaImS3r5OeiVvN8Q1JK0pyu1uPus9293t3rx4wZE0z0ABBS1KkAUJgeuz24+1ndTTezqyVdIOmD7u5BBQYAAAAErdi7PcxQ9gK30919VzAhAQAAAKVR7N0ebpJUK2mBmUnSInf/fNFRoUv5XFQ0a/7WMkQCAADQ/xR7t4cDgwoEAAAAKDWe8AYAAIDQIPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhAbJLwAAAELDKvFEYjNrlPRWiTczWtKGEm+jLwt7+SX2AeXv++Xf4O4zil0JdWpZUP5wl19iH/T18uddn1Yk+S0HM2tw9/pKx1EpYS+/xD6g/OEuf9DCvj8pf7jLL7EPqqn8dHsAAABAaJD8AgAAIDSqOfmdXekAKizs5ZfYB5QfQQr7/qT8CPs+qJryV22fXwAAAKC9am75BQAAANog+QUAAEBoVF3ya2YzzOwVM1tpZl+vdDyVYGZvmtmLZrbUzBoqHU+pmdltZrbezJa1GjfKzBaY2Wu5vyMrGWOpdbEPvm1m7+beB0vN7PxKxlhKZrafmS00sxVmttzMrs+ND9X7oBSoU6lTc+NC81miPq3++rSqkl8zi0r6maTzJB0u6UozO7yyUVXMB9z9mGq5J18PbpfU/sbWX5f0qLsfJOnR3Otqdrs67gNJ+lHufXCMuz9U5pjKKSXpK+5+mKSTJP1D7rMftvdBoKhT26BODc9n6XZRn1Z1fVpVya+kEyStdPc33D0h6S5JF1U4JpSYuz8uaVO70RdJ+lVu+FeSLi5rUGXWxT4IDXdf6+6Lc8PbJa2QVKeQvQ9KgDo1hMJep1KfVn99Wm3Jb52kd1q9Xp0bFzYuab6ZPWdmMysdTIWMc/e1UvaDLGlsheOplFlm9kLuNF6/PUXVG2Y2SdI0Sc+I90GxqFOzqFP5LEnUp1XzHqi25Nc6GRfGe7lNd/djlT1V+Q9mdlqlA0JF/ELSAZKOkbRW0n9WNpzSM7MhkuZK+qK7b6t0PFWAOjWLOhXUp1Wk2pLf1ZL2a/V6gqQ1FYqlYtx9Te7vekn3KnvqMmzWmdn7JCn3d32F4yk7d1/n7ml3z0i6WVX+PjCzuLIV9Rx3vyc3OvTvgyJRp4o6NSfUnyXqU0lV9B6otuT3WUkHmdlkM6uRdIWkByocU1mZ2WAzG9o8LOkcScu6X6oqPSDp6tzw1ZLur2AsFdFcSeVcoip+H5iZSbpV0gp3/2GrSaF/HxSJOpU6tVmoP0vUp5Kq6D1QdU94y91+5MeSopJuc/fvVTiksjKzKcq2TEhSTNJvqn0fmNlvJZ0habSkdZK+Jek+Sb+TNFHS25Iud/eqvYChi31whrKn6FzSm5I+19xfq9qY2SmSnpD0oqRMbvQ/K9tPLTTvg1KgTqVOVcjqVOrT6q9Pqy75BQAAALpSbd0eAAAAgC6R/AIAACA0SH4BAAAQGiS/AAAACA2SXwAAAIQGyS9Cx8w+b2afzA1/ysz2bTXtFjM7vHLRAUD/QX2K/ohbnSHUzOwvkr7q7g2VjgUA+jPqU/QXtPyiXzGzSWb2spn9ysxeMLO7zWyQmX3QzJaY2YtmdpuZ1ebm/76ZvZSb9z9y475tZl81s49Iqpc0x8yWmtlAM/uLmdXn5rsyt75lZvaDVjHsMLPvmdnzZrbIzMZVYl8AQDGoTxFWJL/ojw6RNNvdp0raJunLkm6X9FF3P0rZpzD9vZmNUvYxlEfk5v2/rVfi7ndLapD0cXc/xt13N0/Lnbr7gaQzlX2qz/FmdnFu8mBJi9z9aEmPS7q2ZCUFgNKiPkXokPyiP3rH3Z/KDd8p6YOSVrn7q7lxv5J0mrIV+R5Jt5jZpZJ29WIbx0v6i7s3untK0pzcOiUpIemPueHnJE0qtCAAUGHUpwgdkl/0R3l1VM9VsidImivpYknzerEN62Za0ls6y6eVbRkBgP6I+hShQ/KL/miimZ2cG75S0iOSJpnZgblxV0l6zMyGSBru7g9J+qKyp9va2y5paCfjn5F0upmNNrNobjuPBVkIAOgDqE8ROnzDQn+0QtLVZvZfkl6TdL2kRZJ+b2YxSc9K+qWkUZLuN7MByrY8fKmTdd0u6ZdmtltS8wFA7r7WzG6QtDC37EPufn/pigQAFUF9itDhVmfoV8xskqQ/uvuRFQ4FAPo16lOEFd0eAAAAEBq0/AIAACA0aPkFAABAaJD8AgAAIDRIfgEAABAaJL8AAAAIDZJfAAAAhMb/B0Hm1BpCSN+pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 16 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "query, context, target, lengths, target_indices = valid_batches[0]\n",
    "\n",
    "with torch.no_grad():\n",
    "    weight, output = net(query, context, lengths=lengths, return_weight=True)\n",
    "\n",
    "context = context.numpy()\n",
    "weight = weight.data.numpy()\n",
    "\n",
    "colors = sns.color_palette('husl', 3)\n",
    "\n",
    "fig, axs = plt.subplots(batch_size, 2, figsize=(10, 10), sharex=True, sharey=True)\n",
    "sns.despine(fig=fig)\n",
    "axs_min, axs_max = axs[:,0], axs[:,1]\n",
    "\n",
    "for i, (i_min, i_max) in enumerate(target_indices):\n",
    "    length = lengths[i]\n",
    "    w_min, w_max = weight[i]\n",
    "    c_min = [context[i,j,Data.min_position] for j in range(length)]\n",
    "    c_max = [context[i,j,Data.max_position] for j in range(length)]\n",
    "\n",
    "    axs_min[i].axvline(i_min, zorder=1, color=colors[2], label='min value')\n",
    "    axs_min[i].bar(np.arange(length) - 0.4, c_min, zorder=2, color=colors[1], lw=0, label='values')\n",
    "    axs_min[i].plot(np.arange(length), w_min[:length], zorder=3, color=colors[0], lw=4, label='attention')\n",
    "\n",
    "    axs_max[i].axvline(i_max, zorder=1, color=colors[2], label='max value')\n",
    "    axs_max[i].bar(np.arange(length) - 0.4, c_max, zorder=2, color=colors[1], lw=0, label='values')\n",
    "    axs_max[i].plot(np.arange(length), w_max[:length], zorder=3, color=colors[0], lw=4, label='attention')\n",
    "\n",
    "axs_min[0].set_title('Minimum')\n",
    "axs_max[0].set_title('Maximum')\n",
    "axs_max[0].legend(loc='best')    \n",
    "axs_min[0].legend(loc='best')\n",
    "axs_max[0].legend(loc='best')\n",
    "axs_min[-1].set_xlabel('position')\n",
    "axs_max[-1].set_xlabel('position')\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}


================================================
FILE: setup.py
================================================
from distutils.core import setup

setup(
    name='attention',
    version='0.1.0',
    author='tllake',
    author_email='thom.l.lake@gmail.com',
    packages=['attention'],
    description='An attention function for PyTorch.',
    long_description=open('README.md').read())

================================================
FILE: test/test_attention.py
================================================
import numpy as np
import pytest

import torch
from attention import attention


def test_apply_mask_3d():
    batch_size, m, n = 3, 4, 5
    sizes = [4, 3, 2]
    values = torch.randn(batch_size, m, n)
    masked = attention.mask3d(values, sizes=sizes).data
    assert values.size() == masked.size() == (batch_size, m, n)
    for i in range(batch_size):
        for j in range(m):
            for k in range(n):
                if j < sizes[i]:
                    assert masked[i,j,k] == values[i,j,k]
                else:
                    assert masked[i,j,k] == 0


@pytest.mark.parametrize('v_mask, v_unmask', [(0, 1), (float('-inf'), 0)])
def test_fill_context_mask(v_mask, v_unmask):
    batch_size, n_q, n_c = 3, 4, 5
    query_sizes = [4, 3, 2]
    context_sizes = [3, 2, 5]
    mask = torch.randn(batch_size, n_q, n_c)
    mask = attention.fill_context_mask(
        mask, sizes=context_sizes,
        v_mask=v_mask, v_unmask=v_unmask)

    for i in range(batch_size):
        for j in range(n_q):
            for k in range(n_c):
                if k < context_sizes[i]:
                    assert mask[i,j,k] == v_unmask
                else:
                    assert mask[i,j,k] == v_mask


def test_dot():
    batch_size, n_q, n_c, d = 31, 18, 15, 22
    q = np.random.normal(0, 1, (batch_size, n_q, d))
    c = np.random.normal(0, 1, (batch_size, n_c, d))

    s = attention.dot(torch.from_numpy(q),
                      torch.from_numpy(c)
                      )
    s = s.data.numpy()

    assert s.shape == (batch_size, n_q, n_c)

    for i in range(batch_size):
        for j in range(n_q):
            for k in range(n_c):
                assert np.allclose(np.dot(q[i,j], c[i,k]), s[i,j,k])


@pytest.mark.parametrize(
    'batch_size,n_q,n_c,d', [
    (1, 1, 6, 11),
    (20, 1, 10, 3),
    (3, 10, 15, 5)])
def test_attention(batch_size, n_q, n_c, d):
    q = np.random.normal(0, 1, (batch_size, n_q, d))
    c = np.random.normal(0, 1, (batch_size, n_c, d))

    w_out, z_out = attention.attend(torch.from_numpy(q),
                                    torch.from_numpy(c),
                                    return_weight=True
                                    )
    w_out = w_out.data.numpy()
    z_out = z_out.data.numpy()

    assert w_out.shape == (batch_size, n_q, n_c)
    assert z_out.shape == (batch_size, n_q, d)

    for i in range(batch_size):
        for j in range(n_q):
            s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)]
            max_s = max(s)
            exp_s = [np.exp(si - max_s) for si in s]
            sum_exp_s = sum(exp_s)

            w_ref = [ei / sum_exp_s for ei in exp_s]
            assert np.allclose(w_ref, w_out[i,j])

            z_ref = sum(w_ref[k] * c[i,k] for k in range(n_c))
            assert np.allclose(z_ref, z_out[i,j])


@pytest.mark.parametrize(
    'batch_size,n_q,n_c,d,p', [
    (1, 1, 6, 11, 5),
    (20, 1, 10, 3, 14),
    (3, 10, 15, 5, 9)])
def test_attention_values(batch_size, n_q, n_c, d, p):
    q = np.random.normal(0, 1, (batch_size, n_q, d))
    c = np.random.normal(0, 1, (batch_size, n_c, d))
    v = np.random.normal(0, 1, (batch_size, n_c, p))

    w_out, z_out = attention.attend(torch.from_numpy(q),
                                    torch.from_numpy(c),
                                    value=torch.from_numpy(v),
                                    return_weight=True
                                    )
    w_out = w_out.data.numpy()
    z_out = z_out.data.numpy()

    assert w_out.shape == (batch_size, n_q, n_c)
    assert z_out.shape == (batch_size, n_q, p)

    for i in range(batch_size):
        for j in range(n_q):
            s = [np.dot(q[i,j], c[i,k]) for k in range(n_c)]
            max_s = max(s)
            exp_s = [np.exp(si - max_s) for si in s]
            sum_exp_s = sum(exp_s)

            w_ref = [ei / sum_exp_s for ei in exp_s]
            assert np.allclose(w_ref, w_out[i,j])

            z_ref = sum(w_ref[k] * v[i,k] for k in range(n_c))
            assert np.allclose(z_ref, z_out[i,j])


@pytest.mark.parametrize(
    'batch_size,n_q,n_c,d,context_sizes', [
    (1, 1, 6, 11, [3]),
    (4, 1, 10, 3, [7, 5, 10, 9])])
def test_attention_masked(batch_size, n_q, n_c, d, context_sizes):
    q = np.random.normal(0, 1, (batch_size, n_q, d))
    c = np.random.normal(0, 1, (batch_size, n_c, d))

    w_out, z_out = attention.attend(torch.from_numpy(q),
                                    torch.from_numpy(c),
                                    context_sizes=context_sizes,
                                    return_weight=True
                                    )
    w_out = w_out.data.numpy()
    z_out = z_out.data.numpy()

    assert w_out.shape == (batch_size, n_q, n_c)
    assert z_out.shape == (batch_size, n_q, d)

    w_checked = np.zeros((batch_size, n_q, n_c), dtype=int)
    z_checked = np.zeros((batch_size, n_q, d), dtype=int)

    for i in range(batch_size):
        for j in range(n_q):
            n = context_sizes[i] if context_sizes is not None else n_c

            s = [np.dot(q[i,j], c[i,k]) for k in range(n)]
            max_s = max(s)
            exp_s = [np.exp(sk - max_s) for sk in s]
            sum_exp_s = sum(exp_s)

            w_ref = [ek / sum_exp_s for ek in exp_s]
            for k in range(n_c):
                if k < n:
                    assert np.allclose(w_ref[k], w_out[i,j,k])
                    w_checked[i,j,k] = 1
                else:
                    assert np.allclose(0, w_out[i,j,k])
                    w_checked[i,j,k] = 1

            z_ref = sum(w_ref[k] * c[i,k] for k in range(n))
            for k in range(d):
                assert np.allclose(z_ref[k], z_out[i,j,k])
                z_checked[i,j,k] = 1

    assert np.all(w_checked == 1)
    assert np.all(z_checked == 1)