Showing preview only (3,673K chars total). Download the full file or copy to clipboard to get everything.
Repository: PacktPublishing/Transformers-for-Natural-Language-Processing
Branch: main
Commit: 149c3a314e0e
Files: 40
Total size: 35.9 MB
Directory structure:
gitextract_98o3bluv/
├── .other/
│ └── technical_requirements.md
├── Chapter01/
│ ├── Multi_Head_Attention_Sub_Layer.ipynb
│ ├── positional_encoding.ipynb
│ └── text.txt
├── Chapter02/
│ ├── BERT_Fine_Tuning_Sentence_Classification_DR.ipynb
│ ├── in_domain_train.tsv
│ └── out_of_domain_dev.tsv
├── Chapter03/
│ ├── KantaiBERT.ipynb
│ └── kant.txt
├── Chapter04/
│ └── Transformer_tasks.ipynb
├── Chapter05/
│ ├── BLEU.py
│ ├── Trax_Translation.ipynb
│ ├── read.py
│ └── read_clean.py
├── Chapter06/
│ ├── OpenAI_GPT_2.ipynb
│ ├── Training_OpenAI_GPT_2.ipynb
│ ├── gpt-2-train_files/
│ │ ├── accumulate.py
│ │ ├── dset.txt
│ │ ├── encode.py
│ │ ├── load_dataset.py
│ │ ├── memory_saving_gradients.py
│ │ └── train.py
│ └── head_view_bert.ipynb
├── Chapter07/
│ └── Summarizing_Text_with_T5.ipynb
├── Chapter08/
│ ├── Summarizing_Text_V2.ipynb
│ ├── Tokenizer.ipynb
│ ├── Training_OpenAI_GPT_2_CH08.ipynb
│ ├── gpt-2-train_files/
│ │ ├── accumulate.py
│ │ ├── encode.py
│ │ ├── load_dataset.py
│ │ ├── mdset.txt
│ │ ├── memory_saving_gradients.py
│ │ └── train.py
│ └── text.txt
├── Chapter09/
│ └── SRL.ipynb
├── Chapter10/
│ ├── Haystack_QA_Pipeline.ipynb
│ └── QA.ipynb
├── Chapter11/
│ └── SentimentAnalysis.ipynb
├── Chapter12/
│ └── Fake_News.ipynb
└── README.md
================================================
FILE CONTENTS
================================================
================================================
FILE: .other/technical_requirements.md
================================================
**Software and hardware specifications**
========================================
<table>
<thead>
<tr class="header">
<th><strong>Chapter number</strong></th>
<th><strong>Software required<br />
(with version)</strong></th>
<th><strong>Hardware specifications</strong></th>
<th><strong>OS required</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td rowspan="2">1-12</td>
<td>Python 3.8</td>
<td rowspan="2">x86/AMD64 system</td>
<td rowspan="2">Windows,<br />
any Linux distro,<br />
or macOS</td>
</tr>
<tr>
<td>Any modern web browser, <br />such as Firefox, Edge, Safari, <br />or Chrome (recommended) </td>
</tr>
</tbody>
</table>
**\*Note**: The code in this book is in the form of python notebooks, and can be executed in Google Colaboratory. If you wish to execute these on your local machine, the (Python) package requirements are mentioned in the following section.
**Package specifications**
==========================
| **Package required** | **Version** | **Installation command (pip)** |
|---------------------------|-------------------|--------------------------------|
| Transformers | 4.1.1 or higher | `pip install transformer` |
| genism | 3.8.3 or higher | `pip install genism` |
| TensorFlow | 2.4.0 or higher | `pip install tensorflow` |
| NumPy | 1.19.5 or higher | `pip install numpy` |
| SciPy | 1.6.0 or higher | `pip install scipy` |
| pandas | 1.2.0 or higher | `pip install pandas` |
| Matplotlib | 3.3.3 or higher | `pip install matplotlib` |
| scikit-learn | 0.24.0 or higher | `pip install scikit-learn` |
| toposort | 1.6.0 or higher | `pip install toposort` |
| Sentencepiece | 0.1.94 or higher | `pip install sentencepiece` |
| trax | 1.3.7 or higher | `pip install trax` |
| Allennlp | 1.0.0 or higher | `pip install allennlp` |
| Allennlp-models | 1.0.0 or higher | `pip install allennlp-models` |
| Farm-haystack | 0.6.0 or higher | `pip install farm-haystack` |
| Torch | 1.6.0+cu101 or higher | `pip install torch==1.6.0+cu101` |
**\*Note**: This isn’t an exhaustive list of all the packages required to run the codes in this book, but only the essential and most commonly used packages in the book. You will likely encounter some more required packages as you read through the book, which you can install using `pip` or `conda`.
================================================
FILE: Chapter01/Multi_Head_Attention_Sub_Layer.ipynb
================================================
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "Multi-Head Attention Sub-Layer.ipynb",
"provenance": [],
"collapsed_sections": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU",
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"946c90b82f7f46caa25c885668b75eab": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HBoxModel",
"state": {
"_view_name": "HBoxView",
"_dom_classes": [],
"_model_name": "HBoxModel",
"_view_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"_view_count": null,
"_view_module_version": "1.5.0",
"box_style": "",
"layout": "IPY_MODEL_4191af78535e4da8bb797690eff84e00",
"_model_module": "@jupyter-widgets/controls",
"children": [
"IPY_MODEL_9ce3d57b96b64da0b15e3f3626bacb30",
"IPY_MODEL_f8da2c91156342a69d9b262f4f993aa4"
]
}
},
"4191af78535e4da8bb797690eff84e00": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_view_name": "LayoutView",
"grid_template_rows": null,
"right": null,
"justify_content": null,
"_view_module": "@jupyter-widgets/base",
"overflow": null,
"_model_module_version": "1.2.0",
"_view_count": null,
"flex_flow": null,
"width": null,
"min_width": null,
"border": null,
"align_items": null,
"bottom": null,
"_model_module": "@jupyter-widgets/base",
"top": null,
"grid_column": null,
"overflow_y": null,
"overflow_x": null,
"grid_auto_flow": null,
"grid_area": null,
"grid_template_columns": null,
"flex": null,
"_model_name": "LayoutModel",
"justify_items": null,
"grid_row": null,
"max_height": null,
"align_content": null,
"visibility": null,
"align_self": null,
"height": null,
"min_height": null,
"padding": null,
"grid_auto_rows": null,
"grid_gap": null,
"max_width": null,
"order": null,
"_view_module_version": "1.2.0",
"grid_template_areas": null,
"object_position": null,
"object_fit": null,
"grid_auto_columns": null,
"margin": null,
"display": null,
"left": null
}
},
"9ce3d57b96b64da0b15e3f3626bacb30": {
"model_module": "@jupyter-widgets/controls",
"model_name": "FloatProgressModel",
"state": {
"_view_name": "ProgressView",
"style": "IPY_MODEL_97370923218945c5b80ab468751ac8a7",
"_dom_classes": [],
"description": "Downloading: 100%",
"_model_name": "FloatProgressModel",
"bar_style": "success",
"max": 230,
"_view_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"value": 230,
"_view_count": null,
"_view_module_version": "1.5.0",
"orientation": "horizontal",
"min": 0,
"description_tooltip": null,
"_model_module": "@jupyter-widgets/controls",
"layout": "IPY_MODEL_0ba4a91f472e4c41ba80ab4025288446"
}
},
"f8da2c91156342a69d9b262f4f993aa4": {
"model_module": "@jupyter-widgets/controls",
"model_name": "HTMLModel",
"state": {
"_view_name": "HTMLView",
"style": "IPY_MODEL_15aa4b6f8f784c74804107be249126b9",
"_dom_classes": [],
"description": "",
"_model_name": "HTMLModel",
"placeholder": "",
"_view_module": "@jupyter-widgets/controls",
"_model_module_version": "1.5.0",
"value": " 230/230 [00:01<00:00, 185B/s]",
"_view_count": null,
"_view_module_version": "1.5.0",
"description_tooltip": null,
"_model_module": "@jupyter-widgets/controls",
"layout": "IPY_MODEL_edea457617ed4792aeeb65292019ceb4"
}
},
"97370923218945c5b80ab468751ac8a7": {
"model_module": "@jupyter-widgets/controls",
"model_name": "ProgressStyleModel",
"state": {
"_view_name": "StyleView",
"_model_name": "ProgressStyleModel",
"description_width": "initial",
"_view_module": "@jupyter-widgets/base",
"_model_module_version": "1.5.0",
"_view_count": null,
"_view_module_version": "1.2.0",
"bar_color": null,
"_model_module": "@jupyter-widgets/controls"
}
},
"0ba4a91f472e4c41ba80ab4025288446": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_view_name": "LayoutView",
"grid_template_rows": null,
"right": null,
"justify_content": null,
"_view_module": "@jupyter-widgets/base",
"overflow": null,
"_model_module_version": "1.2.0",
"_view_count": null,
"flex_flow": null,
"width": null,
"min_width": null,
"border": null,
"align_items": null,
"bottom": null,
"_model_module": "@jupyter-widgets/base",
"top": null,
"grid_column": null,
"overflow_y": null,
"overflow_x": null,
"grid_auto_flow": null,
"grid_area": null,
"grid_template_columns": null,
"flex": null,
"_model_name": "LayoutModel",
"justify_items": null,
"grid_row": null,
"max_height": null,
"align_content": null,
"visibility": null,
"align_self": null,
"height": null,
"min_height": null,
"padding": null,
"grid_auto_rows": null,
"grid_gap": null,
"max_width": null,
"order": null,
"_view_module_version": "1.2.0",
"grid_template_areas": null,
"object_position": null,
"object_fit": null,
"grid_auto_columns": null,
"margin": null,
"display": null,
"left": null
}
},
"15aa4b6f8f784c74804107be249126b9": {
"model_module": "@jupyter-widgets/controls",
"model_name": "DescriptionStyleModel",
"state": {
"_view_name": "StyleView",
"_model_name": "DescriptionStyleModel",
"description_width": "",
"_view_module": "@jupyter-widgets/base",
"_model_module_version": "1.5.0",
"_view_count": null,
"_view_module_version": "1.2.0",
"_model_module": "@jupyter-widgets/controls"
}
},
"edea457617ed4792aeeb65292019ceb4": {
"model_module": "@jupyter-widgets/base",
"model_name": "LayoutModel",
"state": {
"_view_name": "LayoutView",
"grid_template_rows": null,
"right": null,
"justify_content": null,
"_view_module": "@jupyter-widgets/base",
"overflow": null,
"_model_module_version": "1.2.0",
"_view_count": null,
"flex_flow": null,
"width": null,
"min_width": null,
"border": null,
"align_items": null,
"bottom": null,
"_model_module": "@jupyter-widgets/base",
"top": null,
"grid_column": null,
"overflow_y": null,
"overflow_x": null,
"grid_auto_flow": null,
"grid_area": null,
"grid_template_columns": null,
"flex": null,
"_model_name": "LayoutModel",
"justify_items": null,
"grid_row": null,
"max_height": null,
"align_content": null,
"visibility": null,
"align_self": null,
"height": null,
"min_height": null,
"padding": null,
"grid_auto_rows": null,
"grid_gap": null,
"max_width": null,
"order": null,
"_view_module_version": "1.2.0",
"grid_template_areas": null,
"object_position": null,
"object_fit": null,
"grid_auto_columns": null,
"margin": null,
"display": null,
"left": null
}
}
}
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "aXACkAtfNpG0",
"colab_type": "text"
},
"source": [
"# The Attention Mechanism\n",
"Copyright 2020, Denis Rothman, MIT License. Denis Rothman rewrote the reference notebook entirely in basic Python with no frameworks. Three more steps were added, and a Hugging Face transformer example was added. The original images were taken out, redesigned by Denis Rothman for educational purposes, and inserted in the book descriptions of the multi-attention sub-layer.\n",
"\n",
"[The Reference Colaboratory Notebook was written by Manuel Romero](https://colab.research.google.com/drive/1rPk3ohrmVclqhH7uQ7qys4oznDdAhpzF)\n",
"\n",
"[A Medium article was written by Raimi Karim](https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "veRoFjFRNXwJ",
"colab_type": "code",
"colab": {}
},
"source": [
"import numpy as np\n",
"from scipy.special import softmax"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "JLe9lWCJNogW",
"colab_type": "code",
"outputId": "733e039b-343e-4161-9919-19b3a1ec130f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 90
}
},
"source": [
"print(\"Step 1: Input : 3 inputs, d_model=4\")\n",
"x =np.array([[1.0, 0.0, 1.0, 0.0], # Input 1\n",
" [0.0, 2.0, 0.0, 2.0], # Input 2\n",
" [1.0, 1.0, 1.0, 1.0]]) # Input 3\n",
"print(x)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 1: Input : 3 inputs, d_model=4\n",
"[[1. 0. 1. 0.]\n",
" [0. 2. 0. 2.]\n",
" [1. 1. 1. 1.]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "JZImwtHPN91V",
"colab_type": "code",
"outputId": "07706940-e200-4956-b957-fe9681139d0d",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 126
}
},
"source": [
"print(\"Step 2: weights 3 dimensions x d_model=4\")\n",
"print(\"w_query\")\n",
"w_query =np.array([[1, 0, 1],\n",
" [1, 0, 0],\n",
" [0, 0, 1],\n",
" [0, 1, 1]])\n",
"print(w_query)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 2: weights 3 dimensions x d_model=4\n",
"w_query\n",
"[[1 0 1]\n",
" [1 0 0]\n",
" [0 0 1]\n",
" [0 1 1]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "7kRBS7MUOFgV",
"colab_type": "code",
"outputId": "8b0bcc03-88b1-4e8d-a483-dacc91ffa9ee",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 108
}
},
"source": [
"print(\"w_key\")\n",
"w_key =np.array([[0, 0, 1],\n",
" [1, 1, 0],\n",
" [0, 1, 0],\n",
" [1, 1, 0]])\n",
"print(w_key)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"w_key\n",
"[[0 0 1]\n",
" [1 1 0]\n",
" [0 1 0]\n",
" [1 1 0]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Napm2VtkOIEN",
"colab_type": "code",
"outputId": "7331eb08-64d5-4a36-eeef-0a0a556f130b",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 108
}
},
"source": [
"print(\"w_value\")\n",
"w_value = np.array([[0, 2, 0],\n",
" [0, 3, 0],\n",
" [1, 0, 3],\n",
" [1, 1, 0]])\n",
"print(w_value)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"w_value\n",
"[[0 2 0]\n",
" [0 3 0]\n",
" [1 0 3]\n",
" [1 1 0]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "JqapIgfDOQ7d",
"colab_type": "code",
"outputId": "fd610d7a-968a-47e6-d614-40ad03c1d172",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 108
}
},
"source": [
"print(\"Step 3: Matrix multiplication to obtain Q,K,V\")\n",
"\n",
"print(\"Queries: x * w_query\")\n",
"Q=np.matmul(x,w_query)\n",
"print(Q)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 3: Matrix multiplication to obtain Q,K,V\n",
"Queries: x * w_query\n",
"[[1. 0. 2.]\n",
" [2. 2. 2.]\n",
" [2. 1. 3.]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "NmfMln1Wmv73",
"colab_type": "code",
"outputId": "065b63ba-7584-4302-97cd-d5e1765470ed",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 108
}
},
"source": [
"print(\"Step 3: Matrix multiplication to obtain Q,K,V\")\n",
"\n",
"print(\"Keys: x * w_key\")\n",
"K=np.matmul(x,w_key)\n",
"print(K)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 3: Matrix multiplication to obtain Q,K,V\n",
"Keys: x * w_key\n",
"[[0. 1. 1.]\n",
" [4. 4. 0.]\n",
" [2. 3. 1.]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "v3Asv-8mOWkN",
"colab_type": "code",
"outputId": "2ec71310-0486-46f4-d9f5-d12a1a6ad0e6",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 90
}
},
"source": [
"print(\"Values: x * w_value\")\n",
"V=np.matmul(x,w_value)\n",
"print(V)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Values: x * w_value\n",
"[[1. 2. 3.]\n",
" [2. 8. 0.]\n",
" [2. 6. 3.]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "gfgRAHUuOp5c",
"colab_type": "code",
"outputId": "ad02f055-11e0-4b9a-eb15-b66e4846c95e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 90
}
},
"source": [
"print(\"Step 4: Scaled Attention Scores\")\n",
"k_d=1 #square root of k_d=3 rounded down to 1 for this example\n",
"attention_scores = (Q @ K.transpose())/k_d\n",
"print(attention_scores)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 4: Scaled Attention Scores\n",
"[[ 2. 4. 4.]\n",
" [ 4. 16. 12.]\n",
" [ 4. 12. 10.]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "hg2t6KuNOjzM",
"colab_type": "code",
"outputId": "c0610f91-cd1d-4b0f-b5ce-f6445481186a",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 90
}
},
"source": [
"print(\"Step 5: Scaled softmax attention_scores for each vector\")\n",
"attention_scores[0]=softmax(attention_scores[0])\n",
"attention_scores[1]=softmax(attention_scores[1])\n",
"attention_scores[2]=softmax(attention_scores[2])\n",
"print(attention_scores[0])\n",
"print(attention_scores[1])\n",
"print(attention_scores[2])"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 5: Scaled softmax attention_scores for each vector\n",
"[0.06337894 0.46831053 0.46831053]\n",
"[6.03366485e-06 9.82007865e-01 1.79861014e-02]\n",
"[2.95387223e-04 8.80536902e-01 1.19167711e-01]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "R4Es7A7NOvjD",
"colab_type": "code",
"outputId": "b86060fe-1292-47c5-93f6-ddeeca1bfb62",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 199
}
},
"source": [
"print(\"Step 6: attention value obtained by score1/k_d * V\")\n",
"print(V[0])\n",
"print(V[1])\n",
"print(V[2])\n",
"print(\"Attention 1\")\n",
"attention1=attention_scores[0].reshape(-1,1)\n",
"attention1=attention_scores[0][0]*V[0]\n",
"print(attention1)\n",
"\n",
"print(\"Attention 2\")\n",
"attention2=attention_scores[0][1]*V[1]\n",
"print(attention2)\n",
"\n",
"print(\"Attention 3\")\n",
"attention3=attention_scores[0][2]*V[2]\n",
"print(attention3)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 6: attention value obtained by score1/k_d * V\n",
"[1. 2. 3.]\n",
"[2. 8. 0.]\n",
"[2. 6. 3.]\n",
"Attention 1\n",
"[0.06337894 0.12675788 0.19013681]\n",
"Attention 2\n",
"[0.93662106 3.74648425 0. ]\n",
"Attention 3\n",
"[0.93662106 2.80986319 1.40493159]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "uBDKhaCvOzXj",
"colab_type": "code",
"outputId": "138901d8-0aa9-4db9-b8b1-76ad557e6688",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 54
}
},
"source": [
"print(\"Step 7: summed the results to create the first line of the output matrix\")\n",
"attention_input1=attention1+attention2+attention3\n",
"print(attention_input1)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 7: summed the results to create the first line of the output matrix\n",
"[1.93662106 6.68310531 1.59506841]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "iEjgRcqHO4ik",
"colab_type": "code",
"outputId": "675a154b-a305-4c0c-e314-353541abfd3e",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 635
}
},
"source": [
"print(\"Step 8: Step 1 to 7 for inputs 1 to 3\")\n",
"#We assume we have 3 results with learned weights (they were not trained in this example)\n",
"#We assume we are implementing the original Transformer paper. We will have 3 results of 64 dimensions each\n",
"attention_head1=np.random.random((3, 64))\n",
"print(attention_head1)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 8: Step 1 to 7 for inputs 1 to 3\n",
"[[0.05750794 0.25966685 0.80912647 0.00841755 0.53786959 0.05089332\n",
" 0.17938191 0.91091697 0.20593063 0.27634727 0.33869867 0.25488968\n",
" 0.88673807 0.56544205 0.69075114 0.56069125 0.92579273 0.46042461\n",
" 0.78471374 0.93064241 0.99626239 0.13662306 0.72892312 0.52327088\n",
" 0.90128711 0.28245531 0.05630861 0.55857421 0.50998676 0.59709355\n",
" 0.40038745 0.70580749 0.18971837 0.78544634 0.35815199 0.57527984\n",
" 0.38283035 0.94917395 0.25450774 0.85725663 0.27262613 0.5720429\n",
" 0.38092713 0.34721503 0.38857267 0.50218029 0.74035216 0.37789311\n",
" 0.12812721 0.42074447 0.39534834 0.4927362 0.65353466 0.86485487\n",
" 0.22989766 0.87239043 0.64613354 0.89034403 0.29338559 0.1671029\n",
" 0.1675619 0.70683457 0.03683821 0.37657364]\n",
" [0.08308343 0.01529261 0.34000535 0.48559272 0.25036425 0.98195061\n",
" 0.72015388 0.03838282 0.18674587 0.33203929 0.82965726 0.6962791\n",
" 0.49038184 0.97126469 0.25373185 0.18486967 0.38352481 0.68254099\n",
" 0.01014604 0.51217341 0.17219508 0.14178547 0.74892979 0.12190071\n",
" 0.0090985 0.09704158 0.70447804 0.21374912 0.72523093 0.89713875\n",
" 0.28817021 0.56472583 0.59136866 0.7711216 0.78839121 0.03607145\n",
" 0.33438564 0.99970048 0.80579864 0.79923327 0.57124039 0.64183951\n",
" 0.11464931 0.703289 0.64033748 0.5799896 0.14488077 0.90946673\n",
" 0.4189947 0.99825172 0.28607413 0.6801013 0.16240732 0.25219133\n",
" 0.30470031 0.30292756 0.15999459 0.52230381 0.82012623 0.33586634\n",
" 0.25613996 0.60354742 0.26006038 0.23281006]\n",
" [0.37977727 0.7429604 0.38837932 0.18434243 0.84440271 0.53955069\n",
" 0.40121556 0.83114666 0.48845808 0.58768546 0.4097926 0.29445373\n",
" 0.22750019 0.9520429 0.99964437 0.57829693 0.32369595 0.60769326\n",
" 0.76116892 0.14857116 0.07462658 0.01199289 0.37147371 0.80177111\n",
" 0.60845313 0.33410248 0.06017335 0.447363 0.31500924 0.95988807\n",
" 0.41506716 0.33740287 0.38991258 0.23478571 0.57808465 0.48520973\n",
" 0.48241035 0.35030686 0.90598744 0.1296871 0.57966373 0.98736092\n",
" 0.43859306 0.5358377 0.25181342 0.0195783 0.51178364 0.26981021\n",
" 0.04674047 0.97762416 0.72747288 0.75616534 0.68105477 0.06914679\n",
" 0.14054312 0.42816012 0.66792325 0.03168237 0.68685758 0.43487164\n",
" 0.08064005 0.23444144 0.60360253 0.21423994]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "QI50dkZ1O630",
"colab_type": "code",
"outputId": "7d467842-f837-4e41-e099-534549b6fc05",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 54
}
},
"source": [
"print(\"Step 9: We assume we have trained the 8 heads of the attention sub-layer\")\n",
"z0h1=np.random.random((3, 64))\n",
"z1h2=np.random.random((3, 64))\n",
"z2h3=np.random.random((3, 64))\n",
"z3h4=np.random.random((3, 64))\n",
"z4h5=np.random.random((3, 64))\n",
"z5h6=np.random.random((3, 64))\n",
"z6h7=np.random.random((3, 64))\n",
"z7h8=np.random.random((3, 64))\n",
"print(\"shape of one head\",z0h1.shape,\"dimension of 8 heads\",64*8)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 9: We assume we have trained the 8 heads of the attention sub-layer\n",
"shape of one head (3, 64) dimension of 8 heads 512\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "3n87LE92_Puf",
"colab_type": "code",
"outputId": "55d00415-ebea-43a6-b4c5-ff13e02c3052",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 90
}
},
"source": [
"print(\"Step 10: Concatenation of heads 1 to 8 to obtain the original 8x64=512 output dimension of the model\")\n",
"output_attention=np.hstack((z0h1,z1h2,z2h3,z3h4,z4h5,z5h6,z6h7,z7h8))\n",
"print(output_attention)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"Step 10: Concantenation of heads 1 to 8 to obtain the original 8x64=512 ouput dimension of the model\n",
"[[0.46950893 0.88546586 0.47615937 ... 0.08285802 0.16577096 0.61094461]\n",
" [0.31638247 0.24246402 0.30390966 ... 0.42283366 0.62127905 0.64414042]\n",
" [0.1922683 0.7017995 0.60116595 ... 0.20012387 0.16264044 0.93645276]]\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PJLl4Jf3fPLh",
"colab_type": "text"
},
"source": [
"And now with Hugging Face in one line!"
]
},
{
"cell_type": "code",
"metadata": {
"id": "CZIRvcRmfTPb",
"colab_type": "code",
"colab": {}
},
"source": [
"#@title Transformer Installation\n",
"!pip -qq install transformers"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "cNwLYc-SfXdF",
"colab_type": "code",
"outputId": "d1314cc6-74d6-45cf-b8d6-0a903e58ac60",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 85,
"referenced_widgets": [
"946c90b82f7f46caa25c885668b75eab",
"4191af78535e4da8bb797690eff84e00",
"9ce3d57b96b64da0b15e3f3626bacb30",
"f8da2c91156342a69d9b262f4f993aa4",
"97370923218945c5b80ab468751ac8a7",
"0ba4a91f472e4c41ba80ab4025288446",
"15aa4b6f8f784c74804107be249126b9",
"edea457617ed4792aeeb65292019ceb4"
]
}
},
"source": [
"#@title Retrieve pipeline of modules and choose English to French translation\n",
"from transformers import pipeline\n",
"translator = pipeline(\"translation_en_to_fr\")\n",
"#One line of code!\n",
"print(translator(\"It is easy to translate languages with transformers\", max_length=40))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "display_data",
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "946c90b82f7f46caa25c885668b75eab",
"version_minor": 0,
"version_major": 2
},
"text/plain": [
"HBox(children=(FloatProgress(value=0.0, description='Downloading', max=230.0, style=ProgressStyle(description_…"
]
},
"metadata": {
"tags": []
}
},
{
"output_type": "stream",
"text": [
"\n",
"[{'translation_text': 'Il est facile de traduire des langues avec des transformateurs.'}]\n"
],
"name": "stdout"
}
]
}
]
}
================================================
FILE: Chapter01/positional_encoding.ipynb
================================================
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "positional_encoding.ipynb",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "7fjcTlyE3WvR"
},
"source": [
"#A Positional Encoding Example\n",
"Copyright 2021 Denis Rothman, MIT License\n",
"\n",
"Reference 1 for Positional Encoding:\n",
"Attention is All You Need paper, page 6,Google Brain and Google Research\n",
"\n",
"\n",
"Reference 2 for word embedding:\n",
"https://www.geeksforgeeks.org/python-word-embedding-using-word2vec/\n",
"Reference 3 for cosine similarity:\n",
"SciKit Learn cosine similarity documentation\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "JKJ8Saf6vR9b",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "654ce4ae-0115-46d4-a186-ee2581f1ee4f"
},
"source": [
"!pip install gensim==3.8.3\n",
"import torch\n",
"import nltk\n",
"nltk.download('punkt')"
],
"execution_count": 6,
"outputs": [
{
"output_type": "stream",
"text": [
"Requirement already satisfied: gensim==3.8.3 in /usr/local/lib/python3.7/dist-packages (3.8.3)\n",
"Requirement already satisfied: scipy>=0.18.1 in /usr/local/lib/python3.7/dist-packages (from gensim==3.8.3) (1.4.1)\n",
"Requirement already satisfied: smart-open>=1.8.1 in /usr/local/lib/python3.7/dist-packages (from gensim==3.8.3) (4.2.0)\n",
"Requirement already satisfied: numpy>=1.11.3 in /usr/local/lib/python3.7/dist-packages (from gensim==3.8.3) (1.19.5)\n",
"Requirement already satisfied: six>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from gensim==3.8.3) (1.15.0)\n",
"[nltk_data] Downloading package punkt to /root/nltk_data...\n",
"[nltk_data] Package punkt is already up-to-date!\n"
],
"name": "stdout"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"True"
]
},
"metadata": {
"tags": []
},
"execution_count": 6
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "PGXgeOyS5qBP"
},
"source": [
"# upload to the text.txt file to Google Colaboratory using the file manager "
]
},
{
"cell_type": "code",
"metadata": {
"id": "7o7EeDUUu0Sh"
},
"source": [
"import math\n",
"import numpy as np\n",
"from nltk.tokenize import sent_tokenize, word_tokenize \n",
"import gensim \n",
"from gensim.models import Word2Vec \n",
"import numpy as np\n",
"from sklearn.metrics.pairwise import cosine_similarity\n",
"import matplotlib.pyplot as plt\n",
"import warnings \n",
"warnings.filterwarnings(action = 'ignore') \n",
"\n",
"\n",
"dprint=0 # prints outputs if set to 1, default=0\n",
"\n",
"#‘text.txt’ file \n",
"sample = open(\"text.txt\", \"r\") \n",
"s = sample.read() \n",
"\n",
"# processing escape characters \n",
"f = s.replace(\"\\n\", \" \") \n",
"\n",
"data = [] \n",
"\n",
"# sentence parsing \n",
"for i in sent_tokenize(f): \n",
"\ttemp = [] \n",
"\t# tokenize the sentence into words \n",
"\tfor j in word_tokenize(i): \n",
"\t\ttemp.append(j.lower()) \n",
"\tdata.append(temp) \n",
"\n",
"# Creating Skip Gram model \n",
"#model2 = gensim.models.Word2Vec(data, min_count = 1, size = 512,window = 5, sg = 1) \n",
"#model = Word2Vec(sentences=common_texts, vector_size=100, window=5, min_count=1, workers=4)\n",
"model2 = gensim.models.Word2Vec(data, min_count = 1, size = 512,window = 5, sg = 1)\n",
"\n",
"# 1-The 2-black 3-cat 4-sat 5-on 6-the 7-couch 8-and 9-the 10-brown 11-dog 12-slept 13-on 14-the 15-rug.\n",
"word1='black'\n",
"word2='brown'\n",
"pos1=2\n",
"pos2=10\n",
"a=model2[word1]\n",
"b=model2[word2]\n",
"\n",
"if(dprint==1):\n",
" print(a)\n",
"\n",
"# compute cosine similarity\n",
"dot = np.dot(a, b)\n",
"norma = np.linalg.norm(a)\n",
"normb = np.linalg.norm(b)\n",
"cos = dot / (norma * normb)\n",
"\n",
"aa = a.reshape(1,512) \n",
"ba = b.reshape(1,512)\n",
"cos_lib = cosine_similarity(aa, ba)"
],
"execution_count": 7,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "xlTeXmatz7bP"
},
"source": [
"A Positional Encoding example using one line of basic Python using a few lines of code for the sine and cosine functions.\n",
"I added a Pytorch method inspired by Pytorch.org to explore these methods.\n",
"The main idea to keep in mind is that we are looking to add small values to the word embedding output so that the positions are taken into account. This means that as long as the cosine similarity, for example, displayed at the end of the notebook, shows the positions have been taken into account, the method can apply. Depending on the Transformer model, this method can be fine-tuned as well as using other methods. "
]
},
{
"cell_type": "code",
"metadata": {
"id": "EmBUq9MzxQxz"
},
"source": [
"pe1=aa.copy()\n",
"pe2=aa.copy()\n",
"pe3=aa.copy()\n",
"paa=aa.copy()\n",
"pba=ba.copy()\n",
"d_model=512\n",
"max_print=d_model\n",
"max_length=20\n",
"\n",
"for i in range(0, max_print,2):\n",
" pe1[0][i] = math.sin(pos1 / (10000 ** ((2 * i)/d_model)))\n",
" paa[0][i] = (paa[0][i]*math.sqrt(d_model))+ pe1[0][i]\n",
" pe1[0][i+1] = math.cos(pos1 / (10000 ** ((2 * i)/d_model)))\n",
" paa[0][i+1] = (paa[0][i+1]*math.sqrt(d_model))+pe1[0][i+1]\n",
" if dprint==1:\n",
" print(i,pe1[0][i],i+1,pe1[0][i+1])\n",
" print(i,paa[0][i],i+1,paa[0][i+1])\n",
" print(\"\\n\")\n",
"\n",
"#print(pe1)\n",
"# A method in Pytorch using torch.exp and math.log :\n",
"max_len=max_length \n",
"pe = torch.zeros(max_len, d_model)\n",
"position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)\n",
"div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
"pe[:, 0::2] = torch.sin(position * div_term)\n",
"pe[:, 1::2] = torch.cos(position * div_term)\n",
"#print(pe[:, 0::2])"
],
"execution_count": 8,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "pgrXed2FwHDC",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "54dcdded-8470-47fc-999e-67294ee67dd2"
},
"source": [
"\n",
"for i in range(0, max_print,2):\n",
" pe2[0][i] = math.sin(pos2 / (10000 ** ((2 * i)/d_model)))\n",
" pba[0][i] = (pba[0][i]*math.sqrt(d_model))+ pe2[0][i]\n",
" \n",
" pe2[0][i+1] = math.cos(pos2 / (10000 ** ((2 * i)/d_model)))\n",
" pba[0][i+1] = (pba[0][i+1]*math.sqrt(d_model))+ pe2[0][i+1]\n",
" \n",
" if dprint==1:\n",
" print(i,pe2[0][i],i+1,pe2[0][i+1])\n",
" print(i,paa[0][i],i+1,paa[0][i+1])\n",
" print(\"\\n\")\n",
"\n",
"print(word1,word2)\n",
"cos_lib = cosine_similarity(aa, ba)\n",
"print(cos_lib,\"word similarity\")\n",
"cos_lib = cosine_similarity(pe1, pe2)\n",
"print(cos_lib,\"positional similarity\")\n",
"cos_lib = cosine_similarity(paa, pba)\n",
"print(cos_lib,\"positional encoding similarity\")\n",
"\n",
"if dprint==1:\n",
" print(word1)\n",
" print(\"embedding\")\n",
" print(aa)\n",
" print(\"positional encoding\")\n",
" print(pe1)\n",
" print(\"encoded embedding\")\n",
" print(paa)\n",
"\n",
" print(word2)\n",
" print(\"embedding\")\n",
" print(ba)\n",
" print(\"positional encoding\")\n",
" print(pe2)\n",
" print(\"encoded embedding\")\n",
" print(pba)\n",
"\n"
],
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"text": [
"black brown\n",
"[[0.9998703]] word similarity\n",
"[[0.8600013]] positional similarity\n",
"[[0.96135795]] positional encoding similarity\n"
],
"name": "stdout"
}
]
}
]
}
================================================
FILE: Chapter01/text.txt
================================================
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too wet.The dog sat on the couch near the rug.
================================================
FILE: Chapter02/BERT_Fine_Tuning_Sentence_Classification_DR.ipynb
================================================
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "BERT_Fine_Tuning_Sentence_Classification_DR.ipynb",
"provenance": [],
"collapsed_sections": [],
"toc_visible": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
},
"accelerator": "GPU"
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "jNKaJz5j_ylj"
},
"source": [
"# BERT Fine-Tuning Sentence Classification\n",
"Copyright 2020 Denis Rothman. The text cells were taken out and replaced by titles of each cell withing the cell. The titles of the cells refer to the titles of the sections of the book. The descriptions of the cells have been rewritten for educational purposes.\n",
"\n",
"Contributer: George Mihaila\n",
"\n",
"[Reference Notebook by Chris McCormick and Nick Ryan](https://colab.research.google.com/drive/1pTuQhug6Dhl9XalKB0zUGf4FIdYFlpcX)\n",
"\n",
"[Reference Article by Chris McCormick and Nick Ryan](https://mccormickml.com/2019/07/22/BERT-fine-tuning/)"
]
},
{
"cell_type": "code",
"metadata": {
"id": "DEfSbAA4QHas",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "d8560cba-271c-451a-cc53-a088af8ce80e"
},
"source": [
"#@title Activating the GPU\n",
"# Main menu->Runtime->Change Runtime Type\n",
"import tensorflow as tf\n",
"device_name = tf.test.gpu_device_name()\n",
"if device_name != '/device:GPU:0':\n",
" raise SystemError('GPU device not found')\n",
"print('Found GPU at: {}'.format(device_name))"
],
"execution_count": 26,
"outputs": [
{
"output_type": "stream",
"text": [
"Found GPU at: /device:GPU:0\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "0NmMdkZO8R6q"
},
"source": [
"#@title Installing the Hugging Face PyTorch Interface for Bert\n",
"# !pip install pytorch-pretrained-bert pytorch-nlp\n",
"!pip install -q transformers"
],
"execution_count": 27,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Ok002ceNB8E7"
},
"source": [
"#@title Importing the modules\n",
"import torch\n",
"from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler\n",
"from keras.preprocessing.sequence import pad_sequences\n",
"from sklearn.model_selection import train_test_split\n",
"from transformers import BertTokenizer, BertConfig\n",
"from transformers import AdamW, BertForSequenceClassification, get_linear_schedule_with_warmup\n",
"from tqdm import tqdm, trange\n",
"import pandas as pd\n",
"import io\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"% matplotlib inline"
],
"execution_count": 28,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "oYsV4H8fCpZ-",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"outputId": "5a2b9ada-305c-4c38-f5c1-9daf64758ff9"
},
"source": [
"#@title Specifying CUDA as the device for Torch\n",
"device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
"n_gpu = torch.cuda.device_count()\n",
"torch.cuda.get_device_name(0)"
],
"execution_count": 29,
"outputs": [
{
"output_type": "execute_result",
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'Tesla P4'"
]
},
"metadata": {
"tags": []
},
"execution_count": 29
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "JpfK9OOJy1OY"
},
"source": [
"@article{warstadt2018neural,\n",
" title={Neural Network Acceptability Judgments},\n",
" author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R},\n",
" journal={arXiv preprint arXiv:1805.12471},\n",
" year={2018}\n",
"}\n"
]
},
{
"cell_type": "code",
"metadata": {
"id": "_UkeC7SG2krJ",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "1d576ac9-fb08-4715-bb7b-dd0b55983bac"
},
"source": [
"#@title Loading the Dataset\n",
"#source of dataset : https://nyu-mll.github.io/CoLA/\n",
"df = pd.read_csv(\"in_domain_train.tsv\", delimiter='\\t', header=None, names=['sentence_source', 'label', 'label_notes', 'sentence'])\n",
"df.shape"
],
"execution_count": 30,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"(8551, 4)"
]
},
"metadata": {
"tags": []
},
"execution_count": 30
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "AQfTaYDo42zu",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 376
},
"outputId": "abf38bbc-0bc2-42e8-9709-b25614e6f3d3"
},
"source": [
"df.sample(10)"
],
"execution_count": 31,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>sentence_source</th>\n",
" <th>label</th>\n",
" <th>label_notes</th>\n",
" <th>sentence</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>1171</th>\n",
" <td>r-67</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>john is as tall as that man .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5605</th>\n",
" <td>c_13</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>i expect soon to see the results .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>6605</th>\n",
" <td>g_81</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>john hummed , and mary sang , at equal volumes .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>3537</th>\n",
" <td>ks08</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>they can smile .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>8483</th>\n",
" <td>ad03</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>alison ran</td>\n",
" </tr>\n",
" <tr>\n",
" <th>4709</th>\n",
" <td>ks08</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>the news was dealt with carefully .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7690</th>\n",
" <td>sks13</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>i sent money to mary .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>7515</th>\n",
" <td>sks13</td>\n",
" <td>0</td>\n",
" <td>*</td>\n",
" <td>mary wrote a letter to himself last year .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>2443</th>\n",
" <td>l-93</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>a flowering plant is on the windowsill .</td>\n",
" </tr>\n",
" <tr>\n",
" <th>5680</th>\n",
" <td>c_13</td>\n",
" <td>1</td>\n",
" <td>NaN</td>\n",
" <td>the canadian bought himself a barbecue .</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" sentence_source ... sentence\n",
"1171 r-67 ... john is as tall as that man .\n",
"5605 c_13 ... i expect soon to see the results .\n",
"6605 g_81 ... john hummed , and mary sang , at equal volumes .\n",
"3537 ks08 ... they can smile .\n",
"8483 ad03 ... alison ran\n",
"4709 ks08 ... the news was dealt with carefully .\n",
"7690 sks13 ... i sent money to mary .\n",
"7515 sks13 ... mary wrote a letter to himself last year .\n",
"2443 l-93 ... a flowering plant is on the windowsill .\n",
"5680 c_13 ... the canadian bought himself a barbecue .\n",
"\n",
"[10 rows x 4 columns]"
]
},
"metadata": {
"tags": []
},
"execution_count": 31
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "GuE5BqICAne2"
},
"source": [
"#@ Creating sentence, label lists and adding Bert tokens\n",
"sentences = df.sentence.values\n",
"\n",
"# Adding CLS and SEP tokens at the beginning and end of each sentence for BERT\n",
"sentences = [\"[CLS] \" + sentence + \" [SEP]\" for sentence in sentences]\n",
"labels = df.label.values"
],
"execution_count": 32,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "Z474sSC6oe7A",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "59ff8f5c-64ab-46a5-b74b-6fb24ded5cb8"
},
"source": [
"#@title Activating the BERT Tokenizer\n",
"tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)\n",
"tokenized_texts = [tokenizer.tokenize(sent) for sent in sentences]\n",
"print (\"Tokenize the first sentence:\")\n",
"print (tokenized_texts[0])"
],
"execution_count": 33,
"outputs": [
{
"output_type": "stream",
"text": [
"Tokenize the first sentence:\n",
"['[CLS]', 'our', 'friends', 'wo', 'n', \"'\", 't', 'buy', 'this', 'analysis', ',', 'let', 'alone', 'the', 'next', 'one', 'we', 'propose', '.', '[SEP]']\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "Cp9BPRd1tMIo"
},
"source": [
"#@title Processing the data\n",
"# Set the maximum sequence length. The longest sequence in our training set is 47, but we'll leave room on the end anyway. \n",
"# In the original paper, the authors used a length of 512.\n",
"MAX_LEN = 128\n",
"\n",
"# Use the BERT tokenizer to convert the tokens to their index numbers in the BERT vocabulary\n",
"input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]\n",
"\n",
"# Pad our input tokens\n",
"input_ids = pad_sequences(input_ids, maxlen=MAX_LEN, dtype=\"long\", truncating=\"post\", padding=\"post\")"
],
"execution_count": 34,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "cDoC24LeEv3N"
},
"source": [
"#@title Create attention masks\n",
"attention_masks = []\n",
"\n",
"# Create a mask of 1s for each token followed by 0s for padding\n",
"for seq in input_ids:\n",
" seq_mask = [float(i>0) for i in seq]\n",
" attention_masks.append(seq_mask)"
],
"execution_count": 35,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "aFbE-UHvsb7-"
},
"source": [
"#@title Splitting data into train and validation sets\n",
"# Use train_test_split to split our data into train and validation sets for training\n",
"\n",
"train_inputs, validation_inputs, train_labels, validation_labels = train_test_split(input_ids, labels, \n",
" random_state=2018, test_size=0.1)\n",
"train_masks, validation_masks, _, _ = train_test_split(attention_masks, input_ids,\n",
" random_state=2018, test_size=0.1)"
],
"execution_count": 36,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "jw5K2A5Ko1RF"
},
"source": [
"#@title Converting all the data into torch tensors\n",
"# Torch tensors are the required datatype for our model\n",
"\n",
"train_inputs = torch.tensor(train_inputs)\n",
"validation_inputs = torch.tensor(validation_inputs)\n",
"train_labels = torch.tensor(train_labels)\n",
"validation_labels = torch.tensor(validation_labels)\n",
"train_masks = torch.tensor(train_masks)\n",
"validation_masks = torch.tensor(validation_masks)"
],
"execution_count": 37,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GEgLpFVlo1Z-"
},
"source": [
"#@title Selecting a Batch Size and Creating and Iterator\n",
"# Select a batch size for training. For fine-tuning BERT on a specific task, the authors recommend a batch size of 16 or 32\n",
"batch_size = 32\n",
"\n",
"# Create an iterator of our data with torch DataLoader. This helps save on memory during training because, unlike a for loop, \n",
"# with an iterator the entire dataset does not need to be loaded into memory\n",
"\n",
"train_data = TensorDataset(train_inputs, train_masks, train_labels)\n",
"train_sampler = RandomSampler(train_data)\n",
"train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=batch_size)\n",
"\n",
"validation_data = TensorDataset(validation_inputs, validation_masks, validation_labels)\n",
"validation_sampler = SequentialSampler(validation_data)\n",
"validation_dataloader = DataLoader(validation_data, sampler=validation_sampler, batch_size=batch_size)\n"
],
"execution_count": 38,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "JzX6dkOHCv9F",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "7729a80b-bc93-41be-f999-063c77f5e2a4"
},
"source": [
"#@title Bert Configuration\n",
"# Initializing a BERT bert-base-uncased style configuration\n",
"#@title Transformer Installation\n",
"try:\n",
" import transformers\n",
"except:\n",
" print(\"Installing transformers\")\n",
" !pip -qq install transformers\n",
" \n",
"from transformers import BertModel, BertConfig\n",
"configuration = BertConfig()\n",
"\n",
"# Initializing a model from the bert-base-uncased style configuration\n",
"model = BertModel(configuration)\n",
"\n",
"# Accessing the model configuration\n",
"configuration = model.config\n",
"print(configuration)"
],
"execution_count": 39,
"outputs": [
{
"output_type": "stream",
"text": [
"BertConfig {\n",
" \"attention_probs_dropout_prob\": 0.1,\n",
" \"gradient_checkpointing\": false,\n",
" \"hidden_act\": \"gelu\",\n",
" \"hidden_dropout_prob\": 0.1,\n",
" \"hidden_size\": 768,\n",
" \"initializer_range\": 0.02,\n",
" \"intermediate_size\": 3072,\n",
" \"layer_norm_eps\": 1e-12,\n",
" \"max_position_embeddings\": 512,\n",
" \"model_type\": \"bert\",\n",
" \"num_attention_heads\": 12,\n",
" \"num_hidden_layers\": 12,\n",
" \"pad_token_id\": 0,\n",
" \"type_vocab_size\": 2,\n",
" \"vocab_size\": 30522\n",
"}\n",
"\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "0z3-ZV0k2qk8",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "852674d6-31a6-40e5-eed1-5548bdb03584"
},
"source": [
"#@title Loading the Hugging Face Bert Uncased Base Model \n",
"model = BertForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=2)\n",
"model.cuda()"
],
"execution_count": 40,
"outputs": [
{
"output_type": "stream",
"text": [
"Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']\n",
"- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
"- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
"Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
],
"name": "stderr"
},
{
"output_type": "execute_result",
"data": {
"text/plain": [
"BertForSequenceClassification(\n",
" (bert): BertModel(\n",
" (embeddings): BertEmbeddings(\n",
" (word_embeddings): Embedding(30522, 768, padding_idx=0)\n",
" (position_embeddings): Embedding(512, 768)\n",
" (token_type_embeddings): Embedding(2, 768)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (encoder): BertEncoder(\n",
" (layer): ModuleList(\n",
" (0): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (1): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (2): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (3): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (4): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (5): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (6): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (7): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (8): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (9): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (10): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (11): BertLayer(\n",
" (attention): BertAttention(\n",
" (self): BertSelfAttention(\n",
" (query): Linear(in_features=768, out_features=768, bias=True)\n",
" (key): Linear(in_features=768, out_features=768, bias=True)\n",
" (value): Linear(in_features=768, out_features=768, bias=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" (output): BertSelfOutput(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" (intermediate): BertIntermediate(\n",
" (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
" )\n",
" (output): BertOutput(\n",
" (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
" (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" )\n",
" )\n",
" )\n",
" )\n",
" (pooler): BertPooler(\n",
" (dense): Linear(in_features=768, out_features=768, bias=True)\n",
" (activation): Tanh()\n",
" )\n",
" )\n",
" (dropout): Dropout(p=0.1, inplace=False)\n",
" (classifier): Linear(in_features=768, out_features=2, bias=True)\n",
")"
]
},
"metadata": {
"tags": []
},
"execution_count": 40
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cJO7qtU_SsDy"
},
"source": [
"##@title Optimizer Grouped Parameters\n",
"#This code is taken from:\n",
"# https://github.com/huggingface/transformers/blob/5bfcd0485ece086ebcbed2d008813037968a9e58/examples/run_glue.py#L102\n",
"\n",
"# Don't apply weight decay to any parameters whose names include these tokens.\n",
"# (Here, the BERT doesn't have `gamma` or `beta` parameters, only `bias` terms)\n",
"param_optimizer = list(model.named_parameters())\n",
"no_decay = ['bias', 'LayerNorm.weight']\n",
"# Separate the `weight` parameters from the `bias` parameters. \n",
"# - For the `weight` parameters, this specifies a 'weight_decay_rate' of 0.01. \n",
"# - For the `bias` parameters, the 'weight_decay_rate' is 0.0. \n",
"optimizer_grouped_parameters = [\n",
" # Filter for all parameters which *don't* include 'bias', 'gamma', 'beta'.\n",
" {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],\n",
" 'weight_decay_rate': 0.1},\n",
" \n",
" # Filter for parameters which *do* include those.\n",
" {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],\n",
" 'weight_decay_rate': 0.0}\n",
"]\n",
"# Note - `optimizer_grouped_parameters` only includes the parameter values, not \n",
"# the names."
],
"execution_count": 41,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "GLs72DuMODJO"
},
"source": [
"#@title The Hyperparemeters for the Training Loop \n",
"# optimizer = BertAdam(optimizer_grouped_parameters,\n",
"# lr=2e-5,\n",
"# warmup=.1)\n",
"\n",
"# Number of training epochs (authors recommend between 2 and 4)\n",
"epochs = 4\n",
"\n",
"optimizer = AdamW(optimizer_grouped_parameters,\n",
" lr = 2e-5, # args.learning_rate - default is 5e-5, our notebook had 2e-5\n",
" eps = 1e-8 # args.adam_epsilon - default is 1e-8.\n",
" )\n",
"# Total number of training steps is number of batches * number of epochs.\n",
"# `train_dataloader` contains batched data so `len(train_dataloader)` gives \n",
"# us the number of batches.\n",
"total_steps = len(train_dataloader) * epochs\n",
"\n",
"# Create the learning rate scheduler.\n",
"scheduler = get_linear_schedule_with_warmup(optimizer, \n",
" num_warmup_steps = 0, # Default value in run_glue.py\n",
" num_training_steps = total_steps)"
],
"execution_count": 42,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "9cQNvaZ9bnyy"
},
"source": [
"#Creating the Accuracy Measurement Function\n",
"# Function to calculate the accuracy of our predictions vs labels\n",
"def flat_accuracy(preds, labels):\n",
" pred_flat = np.argmax(preds, axis=1).flatten()\n",
" labels_flat = labels.flatten()\n",
" return np.sum(pred_flat == labels_flat) / len(labels_flat)"
],
"execution_count": 43,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "6J-FYdx6nFE_",
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputId": "791a450e-3e14-435f-bdd8-77f54e520e99"
},
"source": [
"#@title The Training Loop\n",
"t = [] \n",
"\n",
"# Store our loss and accuracy for plotting\n",
"train_loss_set = []\n",
"\n",
"# trange is a tqdm wrapper around the normal python range\n",
"for _ in trange(epochs, desc=\"Epoch\"):\n",
" \n",
" \n",
" # Training\n",
" \n",
" # Set our model to training mode (as opposed to evaluation mode)\n",
" model.train()\n",
" \n",
" # Tracking variables\n",
" tr_loss = 0\n",
" nb_tr_examples, nb_tr_steps = 0, 0\n",
" \n",
" # Train the data for one epoch\n",
" for step, batch in enumerate(train_dataloader):\n",
" # Add batch to GPU\n",
" batch = tuple(t.to(device) for t in batch)\n",
" # Unpack the inputs from our dataloader\n",
" b_input_ids, b_input_mask, b_labels = batch\n",
" # Clear out the gradients (by default they accumulate)\n",
" optimizer.zero_grad()\n",
" # Forward pass\n",
" outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)\n",
" loss = outputs['loss']\n",
" train_loss_set.append(loss.item()) \n",
" # Backward pass\n",
" loss.backward()\n",
" # Update parameters and take a step using the computed gradient\n",
" optimizer.step()\n",
"\n",
" # Update the learning rate.\n",
" scheduler.step()\n",
" \n",
" \n",
" # Update tracking variables\n",
" tr_loss += loss.item()\n",
" nb_tr_examples += b_input_ids.size(0)\n",
" nb_tr_steps += 1\n",
"\n",
" print(\"Train loss: {}\".format(tr_loss/nb_tr_steps))\n",
" \n",
" \n",
" # Validation\n",
"\n",
" # Put model in evaluation mode to evaluate loss on the validation set\n",
" model.eval()\n",
"\n",
" # Tracking variables \n",
" eval_loss, eval_accuracy = 0, 0\n",
" nb_eval_steps, nb_eval_examples = 0, 0\n",
"\n",
" # Evaluate data for one epoch\n",
" for batch in validation_dataloader:\n",
" # Add batch to GPU\n",
" batch = tuple(t.to(device) for t in batch)\n",
" # Unpack the inputs from our dataloader\n",
" b_input_ids, b_input_mask, b_labels = batch\n",
" # Telling the model not to compute or store gradients, saving memory and speeding up validation\n",
" with torch.no_grad():\n",
" # Forward pass, calculate logit predictions\n",
" logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)\n",
" \n",
" # Move logits and labels to CPU\n",
" logits = logits['logits'].detach().cpu().numpy()\n",
" label_ids = b_labels.to('cpu').numpy()\n",
"\n",
" tmp_eval_accuracy = flat_accuracy(logits, label_ids)\n",
" \n",
" eval_accuracy += tmp_eval_accuracy\n",
" nb_eval_steps += 1\n",
"\n",
" print(\"Validation Accuracy: {}\".format(eval_accuracy/nb_eval_steps))"
],
"execution_count": 44,
"outputs": [
{
"output_type": "stream",
"text": [
"\rEpoch: 0%| | 0/4 [00:00<?, ?it/s]"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"Train loss: 0.4848619277793837\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"\rEpoch: 25%|██▌ | 1/4 [03:06<09:20, 186.97s/it]"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"Validation Accuracy: 0.8182870370370371\n",
"Train loss: 0.2855956747942446\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"\rEpoch: 50%|█████ | 2/4 [06:15<06:14, 187.34s/it]"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"Validation Accuracy: 0.8283179012345678\n",
"Train loss: 0.17187516562857075\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"\rEpoch: 75%|███████▌ | 3/4 [09:23<03:07, 187.60s/it]"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"Validation Accuracy: 0.82445987654321\n",
"Train loss: 0.10979274041270566\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"Epoch: 100%|██████████| 4/4 [12:31<00:00, 187.93s/it]"
],
"name": "stderr"
},
{
"output_type": "stream",
"text": [
"Validation Accuracy: 0.8217592592592593\n"
],
"name": "stdout"
},
{
"output_type": "stream",
"text": [
"\n"
],
"name": "stderr"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "68xreA9JAmG5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 513
},
"outputId": "c7a845b2-9633-406b-bb6b-9d45a966c02f"
},
"source": [
"#@title Training Evaluation\n",
"plt.figure(figsize=(15,8))\n",
"plt.title(\"Training loss\")\n",
"plt.xlabel(\"Batch\")\n",
"plt.ylabel(\"Loss\")\n",
"plt.plot(train_loss_set)\n",
"plt.show()"
],
"execution_count": 45,
"outputs": [
{
"output_type": "display_data",
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA3wAAAHwCAYAAAD9+W2oAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOy9eZwkR33m/URWdfdcutBIIJDECCNzYxsLMGBsMGALH+B38euFlzWLP2Ds3QXv2l7bY3stMNh+sbENthcfnMbmXhuDQCBOHUgIoRGgG0kjaaSZkWY0M9LM9ExPd1dlxv6RFZkRkRGZkVWZXUc/389Hqq6qyMjIqurpeOr5HUJKCUIIIYQQQgghs0c07gUQQgghhBBCCGkHCj5CCCGEEEIImVEo+AghhBBCCCFkRqHgI4QQQgghhJAZhYKPEEIIIYQQQmYUCj5CCCGEEEIImVEo+AghhKwLhBBfEEL856bH1lzDC4QQe5qelxBCCPHRHfcCCCGEEB9CiGPa3U0AVgDEg/u/KqX8SOhcUsqXtjGWEEIImWQo+AghhEwsUsot6mchxC4Ar5dSfsUeJ4ToSin7a7k2QgghZBpgSCchhJCpQ4VGCiF+VwixD8AHhRCnCSE+J4Q4IIR4ePDz2doxlwshXj/4+bVCiKuEEH8xGHuPEOKlQ449TwhxpRBiUQjxFSHEu4UQHw68jicNznVYCHGLEOJl2nM/LYS4dTDvXiHE/xw8vnVwbYeFEA8JIb4uhODfc0IIIU74B4IQQsi08igAjwDwWABvQPo37YOD++cCOAHgf5cc/2wAtwPYCuDPAbxfCCGGGPtRAN8CcDqAtwD4pZDFCyHmAHwWwJcAnAngTQA+IoR4wmDI+5GGrZ4E4KkAvjZ4/LcA7AFwBoBHAvh9ADLknIQQQtYfFHyEEEKmlQTAm6WUK1LKE1LKQ1LKf5NSLkkpFwH8CYAfLzn+Xinle6WUMYAPATgLqYAKHiuEOBfAMwFcJKVclVJeBeDiwPX/CIAtAN4+OPZrAD4H4FWD53sAniyEOFlK+bCU8tva42cBeKyUsiel/LqUkoKPEEKIEwo+Qggh08oBKeWyuiOE2CSE+EchxL1CiKMArgRwqhCi4zl+n/pBSrk0+HFLzbGPBvCQ9hgA7A5c/6MB7JZSJtpj9wJ4zODnVwD4aQD3CiGuEEI8Z/D4OwDsBPAlIcTdQojtgecjhBCyDqHgI4QQMq3YrtZvAXgCgGdLKU8G8GODx31hmk3wAIBHCCE2aY+dE3js/QDOsfLvzgWwFwCklNdJKV+ONNzz0wA+OXh8UUr5W1LKxwF4GYDfFEK8aMTrIIQQMqNQ8BFCCJkVTkKat3dYCPEIAG9u+4RSynsB7ADwFiHE/MCF+7nAw68FsATgd4QQc0KIFwyO/fhgrlcLIU6RUvYAHEUawgohxM8KIR4/yCE8grRNReI+BSGEkPUOBR8hhJBZ4V0ANgI4COCbAC5do/O+GsBzABwC8McAPoG0X2ApUspVpALvpUjX/HcAXiOl/N5gyC8B2DUIT/21wXkA4HwAXwFwDMA1AP5OSnlZY1dDCCFkphDM8yaEEEKaQwjxCQDfk1K27jASQgghVdDhI4QQQkZACPFMIcT3CSEiIcSFAF6ONOeOEEIIGTvdcS+AEEIImXIeBeBTSPvw7QHwX6SU3xnvkgghhJAUhnQSQgghhBBCyIzCkE5CCCGEEEIImVEo+AghhBBCCCFkRpm6HL6tW7fKbdu2jXsZhBBCCCGEEDIWrr/++oNSyjNCxk6d4Nu2bRt27Ngx7mUQQgghhBBCyFgQQtwbOpYhnYQQQgghhBAyo1DwEUIIIYQQQsiMQsFHCCGEEEIIITMKBR8hhBBCCCGEzCgUfIQQQgghhBAyo1DwEUIIIYQQQsiM0qrgE0JcKIS4XQixUwix3fH8O4UQ3x38d4cQ4nCb6yGEEEIIIYSQ9URrffiEEB0A7wbwEgB7AFwnhLhYSnmrGiOl/A1t/JsA/FBb6yGEEEIIIYSQ9UabDt+zAOyUUt4tpVwF8HEALy8Z/yoAH2txPYQQQgghhBCyrmhT8D0GwG7t/p7BYwWEEI8FcB6Ar3mef4MQYocQYseBAwcaXyghhBBCCCGEzCKTUrTllQD+VUoZu56UUr5HSnmBlPKCM844Y42XRgghhBBCCCHTSZuCby+Ac7T7Zw8ec/FKMJyTEEIIIYQQQhqlTcF3HYDzhRDnCSHmkYq6i+1BQognAjgNwDUtroUQQgghhBBC1h2tCT4pZR/AGwF8EcBtAD4ppbxFCPFWIcTLtKGvBPBxKaVsay2EEEIIIYQQsh5prS0DAEgpPw/g89ZjF1n339LmGgghhBBCCCFkvTIpRVsIIYQQQgghhDQMBR8hhBBCCCGEzCgUfGSq+Im/uBwXvuvKcS+DEEIIIYSQqaDVHL71zOJyD/PdCAvdzriXMlPcffD4uJdACCGEEELI1ECHryWe9pYv4Rf+np0mCCGEEEIIIeODgq9Fbtp7ZNxLIIQQQgghhKxjKPgIIYQQQgghZEah4COEEEIIIYSQGYWCjxBCCCGEEEJmFAq+CeU79z2MbdsvwX2Hlsa9FEIIIYQQQsiUQsE3ofyf6/cAAK6488CYV0IIIYQQQgiZVij4JoQ4kfj37+xBnEgAQCQGT0g5vkURQgghhBBCphoKvgnh/+zYjd/4xA340Dd2AQAikSq+hHqPEEIIIYQQMiQUfBPCsZU+AGD3w2nOXi74mlN8/ThBP04am48QQgghhBAy2VDwjYl9R5bxor+8HHfuXwQAbJzvAABOrMbGuCYdvh/+46/gWX/61eYmJIQQQgghhEw0FHxj4jPf3Yu7DhzHe79+NwBg00DwLQ0En3L4ZIMO35ETPTx0fLWx+QghhBBCCCGTDQXfmNg1aLew3EtDLBe6A4evlwq+gd4Lrtny/qvuwbbtlxQcQkIIIYQQQsj6hYJvDGz/txvxsW/dBwC47YGjAJBV5zyROXzp2NAcvn+84i4AwNHlXpNLXTM+/M17sffwiXEvgxBCCCGEkJmCgm8MfPGWfQCAR2yex10HjuHEaox+kjp9S6tp8Za6VTrVMOUMThPLvRj/69M347M33D/upawZ9x8+gW3bL8GNew6PeymEEEIIIWSGoeAbA2eftgkveMIZ+M2XfD8SCSwu99CPU8mmcviEyuFDmOJTuX4C7So+VU20SZS7GQ9RoebH33EZfvmD32p6Sa1zxR0HAAAfvfa+Ma+EEEIIIYTMMt1xL2AWqSq0kkiJbiTQGcRtxlKir0I6h8zhU1qpyTYONg8uLuNH334ZHnv6Jtz54DHsevvPNDLvKCu+99AS7h3kQ04TLb5NhBBCCCGEZNDha4GqzXwiUwcvz9NDJviW7By+QNdLCb1+i53aDy/1sBonuPPBYwCaqyCq5gm91mF562dvxZ9ccmur56jLNIbgEkIIIYSQ6YGCrwWqXDYpJSKh5eklMmuIfsJqyxCcw6ccvhZFk31dxxuqCKqW3Lbp9YGr78F7v35Py2chhBBCCCFkcqDga4EqzRUnEpEQmqiTWg5fmiNXN4dPibFh8uBCGdSVyTi8VN7T7/p7H8ZyL0AUKsG3jsIcQ99XQgghhBBCRoGCrwWqNvOJlIj0HL4kz+FLZOrSiWxs4DkH4+IWVZPt8B1e8reAuO/QEl7x99/ARZ+5uXJe9Xq1mX84uTCmkxBCCCGEtAcFXwtU6RYp05BNoefwxbl9ttyPM/cvNE8uWYM8OHspR074BZ/qB3jz3qPB865HuUcIIYQQQkibUPC1QJVTlQxy+JTDJ7UqnUDq+ImajdfVsDaLttjOZZnDVwd1jU0VgSGEEEIIIYSkUPC1QGUOnzRz+NK2DLnDl0jN9Qpuy7AGOXzW1A+X5PBlbSUC5lVj1pPeW0/XSgghhBBCxgcFXwtUOnwJzKItienMSSmzXLzQnLysSucQSuLSm/fhaW/5YmWBFXvuspBO1QA+xLUbZe3TDtsyEEIIIYSQNqHga4HqHD7VliG9r1fpTO/nuXhxHCaCRnH4/vTzt2FxuY99R5Yr161TVqWzTuN4Ne/6k3uEEEIIIYS0CwVfC1S5WiqkU+XwJVIaQi1OZO1G6mrUMIIv1GWypy7L4ctDOgMcPnU7Q4rvTR/7Dj70jV3jXgYhhBBCCFnnUPC1QJXmSiQQRVoOXyLR06p06iGdem7f9/YdxSd37PbMWe7wNVEQxa4A+nCJ4MurjFbPm+crTpfiW+7FeOtnb80qkup89ob78eaLb/EeO11XSgghhBBCppXuuBcwi1TlomUhnZnDZwo1I6RTe/zCd30dAPCLF5zjmDO99eX8hTh/VSPsKVbjxD0QeXe5EGGTrEFIZxvtKv71+j34wNX3AAAu+rknDzUHU/gIIYQQQkib0OFriNd+8Ft42+duBRDSlkEVbUE2vqfl6sVSQmmpXizx4OIytm2/JHt+pe8vrpJ4NFhZaGio6LDDM8MKsoQLzTZ7CPZ8L8wIrPbTOddjsRlCCCGEEDIdUPA1xOW3H8D7r0rdniqrKk5U0RZVpVMi1tsyaDl8cSKx+6El4/hjy318466DeM7//1UsrfaN5/oeYdNEuwZb15TNWastwxo4fL3A4jd1UO9RNEypzTGJxN0PLeH4Sr96ICGEEEIImQko+FqgOodPmjl8UqJntGXIxUQvTrBlYc44/thKH3/2he/hgSPLuGP/scLcLkLaO7zun67DX335jtJ1h5wrJVzxrUVbhl6/eYcvF3zDz7HWbRme/+eX4dXvu3ZtT0oIIYQQQsYGBV8LVOfwmSGdUprtF2KtaqfLRVtc7mdKwT6XL62urL2DGMx198Hj+Juv3ukdZy8lJEoyRMTVbTI/DL2SfMNhUa9HNIriGwPf3X143EsghBBCCCFrBAVfw+jhmN4xg6ItelsGPRQzkVJz+IrzHV3u5UVRAsMsQ9s7VK277L5JeJhmSOuGUSkrMDMs6rUeJqSz7hV/b99R9Fu4BkIIIYQQMttQ8DXM0eWeIcJchUjSHD6ROWtxIg1BJg2HLymIuGPLfS2M0Hb4hq/SWYVdgKUsTLSOW7cmIZ0t5PDJBkI6Q7hj/yIufNfX8c6v+MNtCSGEEEIIcUHB1zAHj60aYscliuSgD59y+KQE+npIZ5KHC/YdjuHicj8Ti7aO87ZlKBFTtl751Lf34Ef+9KsFsWpPUVZVUz0VouGytgw1NNlyz1+p1EWrIZ0tJ+LtP7oMgKGYhBBCCCGkPhR8DXPo2Ioh0FzOmgrpVM5Q6vBZIZ2D4/qxLDp8K31vSKdPhJXl8NmKb/unbsK+o8uFMMhCDl/JlDIL6azRlqGG4nviH16KS258IHj8agtFW7KQziEsPnWpYoo78f3av1yPy29/cNzLIIQQQgghJVDwNYAusg4eWzWEi0vEpIIvr9KZSGk4fImUmSPXT5KCsFpc7mXVHYtFW3w5fPUFT1XOXlmYaBamGXDarGhLrdUBX71tf/DYNhw+FdLZadnhU6Jwktr9SSlx6S378NoPXjfupRBCCCGEkBIo+BpAb+p98NiKIdBsUSSlRCLTypiG4NPGJUl+nDOkc6WfiYB+HCbCauXwaeGk9tp1yhy5ejl80rgNPq7G2FYcvjXK4RNaNddJYZLWQgghhBBC/FDwNYAuuhaXe3jgyInsvu1wqY1yRwitSicKIZ1qnCukM83hS3/uWSdoIodPhWHa4aHFkM4SwYdwEadG1BURdQRiO43X09uykE7fGtXjIeZgU3qyrqAunauxmQghhBBCSJt0x72AWUAXfH/xJbOSoi209GbdRg5fLDHfjbDaT9KQTt3h05RWJxI4pgs+y7nyhnQOIXhsh68Y4uk/tk6Y5rB9+HzDL715H57zuNNxyqa8YX07RVuq2zL0E4m5TjOSbdT2FU26ck2KR0IIIYQQ0h50+BqgrMebLcCyMMBIZM6QCulc6ETZ/SyHL85z+D7y+mfjyWednObwqZBOSxD6XLdh2jIUw1HN58uqdNYRcWrN1Q3ry9cDAPcfPoFf+/D1eOPHvm083kYfPnX9ZSGdK02EkjYU0tmkRGugywchhBBCCFkDKPgaoKwgii1i1F27aEucpA5fej8XE3GSi78NcxG2LHRxfCVGNHjnenFihHX6nLyyxuvCcqiycNJKhy8gpDOkSmegG1hoMu84vxJYux9aMh5vw+FTU3ZKFN+Kp32EWnmI95cVbamxNhdN9jkc1W0khBBCCCFrAwVfA/T6/s2v7ZK5QjqTJBUkc8rh00ReL0k0J0lg80IHx1fzoi29uFjh00Wdzb4aWdWHL6TxumvIdbseMnrKZTmDVQ6fdd/lMPpCDZXga7KgplqvLZh1mnAWm1pzsyGd6W3LBUoJIYQQQsiIUPA1gF04RUcJvm3bL8GbP3Oz0axbOXyx5fDdvn8R+4+upM9pRVs6kcCm+S6Or+Q5fP04sZq2j57Dp0RTpcNXomWkdavz//7DNfj5d1+tna9kcMn5y8JUbRGmRPlc1NxHXmri3cdKr0FnceSQTrpyhBBCCCHrjVYFnxDiQiHE7UKInUKI7Z4xvyiEuFUIcYsQ4qNtrqctysSULlI+dM29mUgRIq/uKKVEL84F30WfuQW3PXAUANDT2jKkDl8Xx1fjTNDYIZ3eKp1l4sjzeJxUNV4vc/hqVOmU1fPp40LOb6Octm5DBVSA/LUuDemsyOErcwebpkmHL3M3m5uSEEIIIYS0QGtVOoUQHQDvBvASAHsAXCeEuFhKeas25nwAvwfgeVLKh4UQZ7a1njYpyw+LE2n0gJOaSOhkOXypuJrvFPV3bAu++Q6WVvrZRtsO6Yy9OXzhTpOaocrhK228Hnw2Pd8vbFy+nvBzqPeo22DTPHX+0pDOBoq2NNeWoaGJGp6LEEIIIYS0R5sO37MA7JRS3i2lXAXwcQAvt8b8CoB3SykfBgAp5YMtrqc1ygRfIiVOrMba/fQ2DelMf1ZtGea6xbejFydGcZBNC10saYVA+klinH8Yh89GTVE4puCwFY+9ee8RvOXiWzSHL/x8VWNHcfjUazTnENXDEhTS2fcUbRlCMI3clqHBkE7qPUIIIYSQ6aDNPnyPAbBbu78HwLOtMd8PAEKIqwF0ALxFSnlpi2tqhbKm3nECLPX62X2jaIsW0qm3ZTCPzx2+TgRsnu9AylxI9GJpOHG+Vgnl+W6+tdev0vmq93wTiyt9/Pj3nwEgTBiEt2UoX1/Z+dR71GRIpzJNRYkH10RbBuUgjuqqNdlKIaRgDSGEEEIIGT/jbrzeBXA+gBcAOBvAlUKIp0kpD+uDhBBvAPAGADj33HPXeo2V9CtCOo+vaA5fkm+Us6ItiUQ/SbIcPnNuaWyuNy2kb9nSqhJ8iXH+4XL43Jv2YkgnrPvVgivEhQus2RIkOLPqkdbjKrSy22DRljhAqDYS0qn68I04T5PN0n2vMyGEEEIImSzaDOncC+Ac7f7Zg8d09gC4WErZk1LeA+AOpALQQEr5HinlBVLKC84444zWFjwsvRIx5QvptHP4+lqVTnPuJK/SKQS2LHQAAMeWU9ewH0vDYfT12yvrw+ejyuFzOmwqlFPl5dUK6axw+CrWZ2ApkTb68CnxXrZqb0hnjfM0lsPX0DzNT0YIIYQQQtqiTcF3HYDzhRDnCSHmAbwSwMXWmE8jdfcghNiKNMTz7hbX1Aq9EhcnTiSWVt0hnWLw6veTBFLCWbRFylysqbYMAHB0uZeeO06MgizDhHSWrd1ci32/eIx6SC0prEpnmDi053LXoSkXvE26XEmuVL1jmgjpVIy6dtlohwgqPkIIIYSQaaA1wSel7AN4I4AvArgNwCellLcIId4qhHjZYNgXARwSQtwK4DIAvy2lPNTWmtpCCS5XOlMsZRZ+GQlobRlyh0+F/bkcPiBv+xBFApuV4DuRisie5fD5jCxdvNnCITyHr/z5dO7Bc8M0eq84pnD+kpDO4uNhlUDroNZTpqWb6MPXWEhng1efVyhtbEpCCCGEENICrebwSSk/D+Dz1mMXaT9LAL85+G9qWR0Iro1znUzcKZIkF3zdTpQJko6Ww3fw2CoA4NRNc875laCMBLBpENKp+sr1EzOHzyeadHGWSCCkdkllW4YSgRYHhDsqQqt02pO5BKdPfCWh56hBnDmTJTl8HgWujgkRTE2tudm2DHT4CCGEEEKmgVYbr68XlODaONcpPKeHdM5FIg/pjNL/AODeQ8cBAI89fbNzfuUApjl8pkbvxYkhKnz99volDp8Pu/G6fVjZPHkIZfV51DxVUae24HSd3yd4M1HZZGuCANfQt546LSWaqq5Z55xVqJnKKpSSyeEDV92D5739a+NeBiGEEELGAAVfA6iQyw0uwaeFdHZ0wac5fLsOLQEAtp2+yTl/Twvp3DTfKTxn9OHzhnTqLmDlJTnnCiraMpACSY2cuXwac+yhYyu45q48wrdQtKVE8NkypE4RmVDUS1r2evpyKuuso05Pw9J5RjvcnIsG31Tx1s/dir2HT4x7GYQQQggZAxR8DaActo3zRcGXJDBCOvUWC7ngOw4hgHMe4RN8ucOncvj051b7WrhmQNGWUKenyuFzncpu2h4U0ukRY//xPd/Eq977TW1uW3D6z+97vEmdEhLSWRViGuKQSet2WBp1+NiXgRBCCCFkKqDga4DSkE4pcWIQ0hkJrS2DEIhUMQ4JnHXyBmzoFo/X54+EyHL48ucsh8+zqR+mLUNVDh9QFJjqXlzHlcqKn5iDdz54bDCHO+TTJW7rNm8fhTD30uPw1ZBvw6w5Scx2IIOTNgYNPkIIIYSQ6YCCrwGUMHIJviSROJ41SZcw2jIIkRXtOOcRm+DrCb6ahXSmrRu6UW6rpA5fvbYMtggRnsohVVU6AWD7p27E4aVV77EhwiYrqOJ53pd/5w7p9M3RlE+mnT9RQrTM4WsupLPOQRddfDOedNGlZnXW8FOWcs/B4/jKbfsbmo0QQgghhLRJq1U61wsqpHODI6Qz1qp0rvaTLO9LiayOEOhLia1bFrIQT5sspDMSEEJgw1wHx1b62XPq+YVuFNR4PVQ3VDVeB4BP7tiDjXMd/NHLnzqY3DxfUNEWlffnE2vqdgSHzzfHKPgqf+rOny+nUq09qEqndRvCx7+1e3B+ic7gC4KmQjpf+BeXZz8zopMQQgghZLKhw9cA/awtQ/Hl1EM6V+PEcPjS2/SHuY7wCj49pBMwi8P0kzykc8NcxxvSmZQ4fN7rCgwDdY3KiraEHJ8JJ58b5haPruvI2x0I59gmQxF9c+rLqhKgIQyj0/LeffWF/jDnIYQQQgghkwkFXwP04gRCAPOOHDy9D1+siTPluqgwzm4nKoR0qs10L1EiUQm+fGAvTrDST7LHldDqW9ZSvyS0z7dnLzh8AQJQCYy4Kk5Tn7ci3y9vcD5KSKe6bU71qPUU2kUY5/UI8MB1fOueh3DFHQ8O5gpfmyoGox/DvDtCCCGEkPUHBV8D9GKJuSjCXFSUTrGUONHLi2cs90y3LsTh6/VNkag7fGlbhjyHME4krrzjAB7/B1/ATXuO5OvQBZ8VZuhzaYpFW9zj9MOlJc5ChE0eslhe4MR+1tVy0NUqwjxHc8SesNWgkM7Ahfzd5Tvx3q/fk847xOoNt7Gphn6EEEIIIWRqoOBrgH6cYK4j0O04BF8iDRGy3E/FnxJZnUzwRZU5fEpP6g5fX8vh2zAQfJfffgAAcO09eQ870+EL2/jbAkEXb7q21cMn1Yg6bRnUoMqWCgF9ACsbr7eQw1c4V8B6UOFqKnwCthJRcf6GYON1QgghhJDJhoKvAXpxgm4nQsdRZjORpuBbGbh9StwprdSNokz8FeeXWVVPAEb7hsXlPpZWY0QCmO9GiKVEZ7AMfa8/TON12+HTBVfXV1LUc2wZoW6g/bQrpNPfHk4JrOYEUNYuovA6FcfY+MJUi+cYbm3q+vX52SydEEIIIWT9QcHXAL1EYq4jMOd0+MyNtgrpVOGZ6nauIyCsd0Md14sTw/3TQzrvPngc/3DFXZlDGCcyG6sLIl2TFNsyuK/Lbryuz+FyM9M1mzl8IQKryn3z5fi55s4b23vOUbmacEJcTG8RnUD1NWrRlbL3vQlYtGW6aPILD0IIIYRMBxR8DdDrJ5jrRJl400kSaWz6l3tmSKcSZ92yHL44QRTpgq/4tqn+fImU2Vh9g2/k8A3r8GniQ79W17LzHnXV58ny6ypCOgtFW5whneVz1FF8/3zNLlxz1yHv8962DNpJqltNlC9I19xD7dVZtIVoUO8RQggh6w/24WuAfiLR7QjMddxtGRKH4LNDOuc65SGd+nMLjgbv890IUSTQH4R/AmaooS4sQr/lL+bw5T/r16rncclsbPjOMmu74C3aYt4q6uTwDdOW4aLP3AIA2PX2n3Gvy1elM6BQSh7GWr4Gw+ErH2qgPi7JEO97HWjwTRfUe4QQQsj6gw5fA/zZK56OL/6PH3M6fL04MTb9J6yQTiVa5jqRNzxuNU6MuTc42j8owZjIXBzqYkJ3Ge1Nn6/wRrFKZ7XDp4bUKTaS57P5nneHh7rGe3v5VTxfxm0PHHU+HgeISH/RFnVT4fANncNXdHlZpJNMWkjng4vL+N4+9+8XIYQQQpqBgq8B5rsRNs13nW0ZlnvxoJCKyO4DeZVLJYy6UXnjdf2phUFI56/++OPwrPMeAQCY6wp0ojSHTxV30UXXMLlctmjTD3Nda9mx+Ryux8vz/aRHEK46eh64WjXocwyz3b1h92Hn4/3YHdNpNl53z1nVezCfaziHLnf4jNmCjyezyaR9At79tZ34tX+5ftzLIIQQQmYaCr4GcVXpXO4lSJK0Rx6ArEm6sFw4Xw4gMAjp1J5TYuvkDXM49xGbjONjmYeL+sTCjXuOBPVkKzh82v2Op2iL79hsDqcrN7j1TSYLP6TncAk+r8MXJrBc+F4qX56i7tr5hG+Vq5nNNbTDp46vn7tZ6zys2jJVTJjBh6XV2OhTSgghhJDmoeBrEFflyuVejETKrLKmncOnBEraeN09by9OjBy+7iB/LhIiE5LzSvAlSVZds5e4hcev/sv1+Psr7sru+/bspTl8kZ7DV8QndMoKraLTHMcAACAASURBVPgLnLiFlWu8eqwQphoYQunCV2kzr9Lpd/iqXMsqz2XUypqmszvSVE4o96aLYT7/bSIxeSKUEEIImTUo+BrE1ZZhuZcMBF80uJ8Kvo4VdtntRF63ZDVOjOeUsJSQ2Dg/EHxd1ZYBWBk4X8vaN+f2Zv+6XQ9lP/s27WU5fLq4dS27qniKTrYJrehZV7YxzAq/VObw+eeomtsm9oRlSscYm9CQzsQQj+VjdYT1hUK6Lu6s1zuTJq4SKZlbSgghhLQMBV+DOEM6+zHipOjw2VUUXRU+Ff04b6YO5O5aP87n7UYCnSh15Xp9aZwLKLp1x1f6lddTVnhFv1aXUPXn8Pkf87dUcDtpw81RH1/4q8rhK4R0BhRK8VX4LIwLXKMNG6+TqUBm/yOEEEJIS1DwNYjt8C10Iyz3YkgJbF5IO2AcGwityHL4XO6gol8I6RTZ4yqkM5FAN4oQS4nVOBV6qsl7+ry5qTq2Up030y80Xs/nKFuv63xljycVgk496ivIYoypcAmH2VtW5fCVC9HRHEc5rEMnivO30XidMZ1kFBjSSQghhLQPBV+D2EVXNs53sNJLEEuJM7YsAAAOLK4AAJRBpsRE1+EOKlZjaTReV25gL5HYNAjpjJN0TOxx+GJrU6UcvivvOICHllad57VronjbMjiO7dsnzNbh3935BF2VINTH+Iu2qFszr/HFf3UFPn/TA955y+bMQl7LQjor+vBV7XVHFWlmPuFIU5EZYNI+A1Iy0JgQQghpGwq+BpmzRNvGuU5WtOWkDV1smIsywWc3WS9zzHpxYrRs6A7EVq+fO3z9RKIjUoGx6szhK4Z0rvYTvOYD38Luh044zxsXHD7PtTqW7g3pdIi60Cqd5Tl85q0dZZrn+OWP7XzwGHY+eAy//+83FebTwzir3MrSxuvekM7yuV1zDVdhtN3tNA2+6WLS5FXq8E3WmgghhJBZg4KvQZwOXz9tyxAJgdM3L+DAsVTw2Xlv5Tl8ZuN1VaWzn0hsyBy+JHP4lOC77PYDeN7bvwagmId2bKVfCNksnLfQh8/t8Lly3Gwn74EjJ7Bt+yX49+/sKYzNWyaY7lv+vDq/f61qDq/QtG4B4IY9aX+9s07ZWLp+n2hTbSEK69IFX1V7ijpFW8qHGrhy+NoWf2TymbSPgJTM4COEEELahoKvQey2DJvmc4evEwFbt8yjNwh1tFswuFo6PPmskwGkffj08coN7Cemw9eNBBIpsdrPhdzew6l7Z2/2V/pJthYftnjS9aG+XlfPPfvYew4cBwB85Nr7CmPVvPoSe1o8aYgbVjkmcwDz528cCL6TBvmVxpoChFKew2efqnjsci/Gr/zzDuw6eNxYR9Vmd1j3w+7zmM411FRB5yHTwTjF1Vdu3W9EHQDM4SOEEELWAgq+BrHz8DbNdbMqnZEQOH2QxwcU3UDb4fvjn38q/vQ/PA1A2pbBcPi0Kp1K8MVJ2py9n0hDLCkcDzkbl5vHWA6ftl3U1+vurWc+porWLC4Xq4O68utW42LBmbJ9oR3SWTxHcY5b7z8KADg4cF11dHHrm1MJ3ZCQzqt3HsSXb92PP/rsLcaYeiGd4TtjkRVtKTqlZP0yrvDJb9/3MF7/zzvwJ5fcVljPMM7z5268H9u2X4LDnvxjQgghhORQ8DWI7dJtnO9kffiiSOD0zfPZc1FFDt8zzj0tK8hijz/zpFQ4nnnyBmycz8VfJASSxHT4FK6NnsuZ0yk4fNpdXYC6BJ9dtEWNP7rc865Nn6anXYO0xrmoLNriEIRLq6nboPIqdfSQzsrCK9bT+l27V19k9cer7sM3mmAzG69T8q13xvUJOLKU/t7vfnjJeFxKDLWo9155NwBg16GlipGEEEIIKcaykaHp2jl8WdGWNIRTd/jsSDjbHex2hCHy9J9f9KQz8fevfgZe/ORH4o79iwByhy+WHofPsdl3CUMdJQhv3nsEV+886G3LEOLwqTFKZOnkYiw/Rg83dQnCwhzZedNbO9SwrB3E4kofJ1bjrIm9vl7fsVJKr2A2XDXLnRQCuPTmfbjs9ge9cxtzee+Us1Z9+BjROV1MmuaXQ5aRUb979r+5hBBCCClCwdcgXSssc9PA4YsTiY4Q2LqlzOEzj42EMFw0ow2CEHjp084anCN9C/tK8HkcPpc2sfNpbJTo+dm/vQoA8P/80GO09ZSHdNpiqMxNzIu25I+5cvjKFE+ow5f+LCGEMNZ0YHEF556+KZ/PEHzF+Q4srnjDMg2HT+X5ZWMEfu3D1zvHuhjWlROWkxh2tiHO0/iMpFXGJPi8PTblcGGm6vfKDo0nhBBCSBGGdDaICrVUpH344iyk8xFaSGcxh8+834mE0boh8mxs8sbrqahMq3Q63CyHajl03J//stCNHCGdmsNXEdJpP+YLiwTcbRlWnUVbvFM4XULjecfYJJE4ddMcAODBxWVzvQ6XTueO/ccK87nuJ9baC45YxV53WEdGnSakRcR6RkqJy29/0FtNddYYd1uGwsdfDqdB1RdCdPgIIYSQaij4GuRJZ52MHf/rxdn9TfMdLPdjJIOiLcqNAxwhnZbD1xECepSnb1+zYW7QhH1Q2CV1+HLnTm2IXE6Rq4CK4qQN3fAcvpJwSUVZC4jEIdYMh8/hABaoEIWGwze4jaXMRLidW1hVpVOF0urrc583dVyVgLXfxuqiLaPl8I3ax2/WufTmfXjtB6/DB7+xa9xLWRPG9RmQ2Rce5m+AhBxqTXT4CCGEkHAY0tkwW7U8vY1zHfRiCSHScMuNniIsQNHhiyJzM2M3aldsGDh8P/HEMxEN2jL0YomzTtmAB44sY8uG9C12OWxlOXybF7oFkaa7ILpADWnLUOrwoZij1+sXc89GKdqiy6V0HoE4ljh5Q/r62a+FfumuYqZ3PriI0zbNoROJgsjUBWCcSDz37V/FwWOpm1psCO+9pHQdI27Q2YevnAeOpM7u7ofWR/GPcX0CMsHneHwY17FPwUcIIYQEQ4evRVRT9NV+AiHgrboJFHP4ulEUFNK5Ya6Dq373hfjzX3h6HtLZT/D887fil5+3TcshKx67Gvtz+DbPFx0+3XXTQ6mcjdfr5PBlIZ35GFdbhlpFWzzn0MfGMm9rsWIJvqqQzrsePI7Hn7kFgChvvC6RiT0AWO4l1tDyzW5SsQ4f6qPTftEWbriniXG1ZSj7GmaYLzXK/l0jhBBCiAkFX4tsmssFXkeITFyo+zp2LkoUmZtpn8MHAGeftgkL3U7mNq3GCea7EbqDEE/A7e6UOXxbHCGdseHw1Wu8brdp0MmqcGrLMUM61W21w1cnhy9OZOa6Fh2+cmdsqdfHyRvmBqG25vP6PfvYxULoqHO5hbUWz1KFo/H6GPyd4yv+sOFJYL3p1XHro6LDPVwSn/o3Z9zXQwghhEwDFHwtsmHOdPT0kE5hvfJz3WIOnx6uFAW8U2r8ci/GXCdCJ4qyjZErz852tXQ2znWKDp92v6rxelEs+s/l2rS5qnSWfZtf1cjc7GcnszUqEd6L7fXqgq84X5ykrqsQwOGlHu4/fKKwFtd6jlkC6NhyH/uOmAVjdOSQDp2z8bp1/G0PHMXdB45hFMr00g27D+MH/uhL2Ku9NmS8jC+Hz/NFzJAhnUnJF1mEEEIIMaHgaxFD8EWmw1cI6bQUXaFKZ4AVoQTfiV6M+Y7t8BXzXcocvrmOKIigvibCqhqv2yGRISGd+ubNFHzV4VuuPEDXOfSfTYfPDG/V1+/aVMZJgm4kICDwhZv34blv/1phLYDpWgKpwNO55u5DeM0HrnUvuuR6Qkkc1634g3+/CX9+6e3Bc/3LNbtw894jxmNlH8t9R5fRTyQeOuavBkvWG3bRluFEqPr3gXqPEEIIqYZFW1pEVdAE0iqbZg6fObbraMtQdt+FEoVSAvPdCNEgp09KiUTr06coc/i6UbEtgy7aqtsy5D8nsqotQ1HQ6SGgISGdVQ6f69E4kdl7shqXhHR6HMwoEs7qqfoSbGd10RHieKhEEOnXXMcJcTZet45f6SdY6Zf3YtT5w8/cEjwWqH5PyNoz7rYMNlIOt6L835PJuh5CCCFkEqHga5Gu5tp1hCiEeJpjKwRfgMOnzzHXibJNf5xIJFKiGwno0qJM8M11I8NlA2yHryqkMzGeL8vhy9oy6M6YIxQxpA+fEmfOXCFrbCxl9p4Ucvik+2dFnKSvp6toiZkvWB7SCRTFpm8ddXAVbbHnSmS7vfkyId/eKUhdxhXSObgt/F5iuEIyWQ4fP1yEEEJIJRR8LdLp6Dl4Agtd3fEzdz62cKh63oVeyXO+G2Wb/X4iEQ8cPp2qkE5biOgOn+5IVuXwJVJWtGVQ4/Tj9efV5q5MNJohnWUvl57Dt9B1Cz57/TbxoNG9c35tvH3dzmqpJe/D0Dl8yN1e11zq/uj7Zf8Lnb+v3JVPCuN6J8rbMtQnq9I50qoIIYSQ9QFz+FpEd9wiYbpBVRGancgs2tIJKdqizTk3yOEDUqEmZdFFLBMa853U4dNdPj2nT5/L3Xg9/1lKdyXPPJSz+G29kUOXoPB8Ya7svP7iEPbPyqWb70RYsUM6S5wxtaZOJCqL6YQ4aKtxUlrUYhSqqnyOWqa/TFgnjvd1UhlXu4K1ZnyX6Q+1HmZNdPgIIYSQcCj4WqRMsFXl5HWEWbQlJIdPHzPfEVlIaRzL1JGy1ElZH765ToReLLG0mo/RwzSrGq/rTdtTh68oLu3qm9IQecXcs9K2DFpxGn1Oe4705/RcqpDNfDcyGr0DpjPnEgP9JEFnULTFHlfmqrmQ0p/jOGofvbI+fm07b/n0k7srX2ddGcaew+cLtR5WcI/7egghhJBpgIKvRfQcvrohmlEkjBDNsCqdkfGzCrvsJ0mWw6ez0isL6YzQ6yc4oQm+vsfhS5Lipk3Xd7GnSmdedqEYnmUKruKcPrKefna+WmKOUfMrwWeLX91h9BWl6QzaMpSOC9zI+vL4ht3OhjRel7Jd0ecK1SXjZXxtGcofH3ZddPgIIYSQaij4WqRTU7CVzREm+PKfu508JDRO8iqdOmXFQua6aQ7f0mpeZKRnOHzFxuv6xl53+GTiFkN2yJ8uPmKHUCnb29lzFPLVLIdPzd8ZhHSWN14vni8eOHz6+6JCXvVTh4hUAAWH0bWOOqhVuUJZs/uOx5okpJ0GWVvG/VaIQluG0XLx+NkihBBCqmHRlhYxc/jcY/7hP/0wbr3/iPtJpKGdMYpizYUuPuY6AlLmOXyJLLZ+CMnhM0I6NYdvzqjSmc5jCDajLYPH4ctEmnkf8IR01ija4nKz9J8LDl9JlU7XeeNEpnmZ2mOrcYKN6HirjZaxEscA5gqP1w0PVSgH2cxFtESwlCNvmMs+lWzLMHmMK1fRd1YznLv+l2IM6SSEEEKqocPXImYOn3szc+FTH4Xf/MkneOdQuqpO4/X05ygL8Yw9VTpL2zJ0IiTSbCPQ01SQq/G6r7Klr0pn5sY5BJ3L4Stty2DNWRA31h21nm4knBVJK6t0Dgq+6HtUVzPoULHjE9/S83MohnC155bVG+bF5R7+9qt3llZZ9ZG/r7UPXdccW+lj2/ZL8Mkduxufe9whncUcvsHtiPMSQgghxA8FX4vojlpIWwUXqnBLUJVOvQ9fJKwqncUcPp/ImO9GmBuc8OiJXva43ofP1ZbBdPhM8VbWh8+Vx2M6fMWfbMrCQ+1DJXIBGgmB+W7H4fBVhHRK6QjpTAqrDNVJPc/r02zRFvM5ier1vf0L38NffvkOfP6mB5zPl1bpzKqrcldeh31HTgAA/uGKu8a8ktE4vLSKN3/mZqz040z8F/vw8UsBQgghpG0o+FqkG+DwVaEERV2Hr9uJtBy+xF2l0yP4FjoR5gaC7p6Dx7PHzaItWkinLDp8dtEVV5XOMhfO5bDVabxeWqXTFdJpCS5bsNoox1R/V1Qenq/aaBm+92LYcEj1cTH7+BVDOqusleMDh9fnBts5Wcb81u0kM1lrVG+e+9m9h09g2/ZL8I27DtaeeS2F1Tu+eDs+dM29+Pdv780eK+TwjRj2y3BhQgghpBoKvhbpGFU6h5tDVeqMAibQ2zh0NYevFw9y+Aohne62DHPdCPODJvF/8aXbs8e9jdcHYsmozGkJttIcPpdDKIvjyvd2pii8++BxvPuynYU51Ei9aMtCJ8Jq31+lUwmlXpzgos/cjAcXl3PBp72kq06Hb8SQziH3s3mVTm0ua0wSENKZibYhFpJXTJ3cTfmwznubiHK9hx27HgIAfOxb9UM+1zLnTX1BVHbGUT8aE/zRIoQQQiYGCr4WsRuvD4Ny6ToBx0eGw1ddpdPn2py6cS4L6ezFEi958iMBmJU3XY3Xfa0MEunu1Zfn8KX4QjrViNI+fA6n4B1fvD0LQ7X70VUWbdHXP3jq8tsP4J+vuRd/+Ombsx5+rpBORScSwSGd3rYMVihqXcqrdFYXbRllQz1qyf31Sl5h1f3CuQryhDLuHL7C4yOGdPKjRWaNOJH49Y99Bzfv9RdzI4SQulDwtYiraMvl//MFeN9rLgieI8py+KoFnx1CqkRbWqVTZmGaCpfIeMUzzsY//fKzsmMB4OQNc9k82bm05/cfXcG7vnKHt61Ckrhz+Gyh5z1emrcuzGp/5jnuO7RkOoZwhXT6q3TarR6UULZFuF20pSNEcLGTpkM6Xcern/Nwz+r5q8Iyy76HkJnrym15HSp7dGaKsP7cY38nvEVbhlsZ80PJrHH0RA8X33A/rhs4+YQQ0gStCj4hxIVCiNuFEDuFENsdz79WCHFACPHdwX+vb3M9a42rD9+2rZvx4oFjFjZHert5oVM5Vnf45uwcvkEbAR2XyPjvLzof556+yRCH6tz63mrOEqDv+sqdhitmh3S6cvhsoadv3VyN1+u0ZVDcc/A4fuwdl+GG3YeN+TLBJ9x9+NTzQuRzqo14dmxHGJvz3OFLn4+i8A2p7Q5ma9V/rrG3VblShrNpzy1rCADPwJC2DNySD4dXZDve2+A511AgZYVaUNKWQd3S4SMEwOi/E4QQ4qK1PnxCiA6AdwN4CYA9AK4TQlwspbzVGvoJKeUb21rHODEE35BJfMoZ27xQ/Vb5cvj6cRq6Z/fhc4V0qinmNQdv03zx3C7H0VXcBMhDOruRMFxCaQs9T0hnSANv9Zy9hvsPnyiOhRnSOdeNClUy1WZ6Loqyn9UlK3HWEcLIzVztm+vsCOF9TWxCHL46f/91Fy873lG0pTqkszqc1kfeE5E7lzrkIZ3u59VnbhqdU/tfjcK/ATWZwpeAkFJG/Z0ghBAXbTp8zwKwU0p5t5RyFcDHAby8xfNNHCGN16tY7qXFRDY7RJeNUaUziswcvsAqnUqY6iGdm+eL7mLX0SfCF76o+vAtdM1jss2ao7hH7BA65Tl8Zthl4RzGg/n4TlTu8HU7ohAKqUS4XbTFbsvQeA5fLYcvxdWWQY8IrJpylE0HS+63g/odHaI14tg2kTL7HbKqdFrPDzHz8IsiZAIZ/XeCEEKKtCn4HgNALyO3Z/CYzSuEEDcKIf5VCHFOi+tZc4wcviGLtpwYCL5NDtFVdr5uR2SOXj+RiGVYHz41QncDNzncxU4k8C+vexZ+4YfPzh5zRG0CAN700e+gn0hsmDOvwQ7D1P+8JVZIaNn8xpzWGJcIlcjzEVUOn+12qnN2NdGmQul62rF6mflCDl8kgtsyuEI6i3/wwzcAeWEP/eiiGK7cVMh8bF1Cci8nhUnaW+VVOt2LqirqUsZaXmf2BYP2z07R4Utvh/2MTNL7RkgT8DNNCGmDcRdt+SyAbVLKpwP4MoAPuQYJId4ghNghhNhx4MCBNV3gKOi96kLLvz/ncadjw1x+nNoIBYV0Gjl8InP0UoevGIbpasug8vzmKxy+SADPP/8MnHnSQvaYL3zx1geOInYIPtu58zl8mehwzj54Lts4ukMz7bF6SOdC19GWYfD8fDfS3In0OVX5sxuZIZ25w6fmjpzVSV24wmvtQ4cTXUWHL7sfUKUzH+um9HPtcV0niQnsypB9ieAP6Rze4RuXI+at0jlioufkfrIIGQ5GRhBC2qBNwbcXgO7YnT14LENKeUhKuTK4+z4AP+yaSEr5HinlBVLKC84444xWFtsGriqdVXzsDT+C773tpYXHQwSfXpSlE0WZo/fuy3ZitZ8UHD7XhlENmdPCL10On6shfFlOUS9Ost5+9nhX+X7d8AoJcfEVbXEKPj2HT7irdGYhnVoOnxI3KqQzioShGFTz9tzh8xdjsXGN84nXxeUeXvmea3DfoSXvfC4XyC4+E9aHb/jNh7RuSTOo75GmqS2DwhbYIeHaZYz7eghpHH6mCSEt0Kbguw7A+UKI84QQ8wBeCeBifYAQ4izt7ssA3NbietacJnL4FC6XzcbM4cv78O249+H0MUfenY0SA3oO3xZHhVC1cVtazZ2xsvDFflwMKX3Wn3wV39192Aw7dOTz2cLQhU8UuvSW4fB13Dl8elhmVrFTXUuSO3z6FfX6xbYMrnYULlzhtfb1qtfpK7ftxzfvfgh/+eXbK+d1vbb6/FVhsqNsqNXnYZIdvkmkSmSr39FRRPhaIiC8gs71ZU8VZgVffrbIbDLslyCEEOKiNcEnpewDeCOALyIVcp+UUt4ihHirEOJlg2G/LoS4RQhxA4BfB/DattYzDvTKnMNW6VS4KmXadI2QzqjQd88WXC4yh0/P4XOcWzl7x1Z62WN2pUsAePrZpwBIi5K4XM5/uvoe4w+b2r/pQimP+gpx+AJCOrXHOyLtV5hIcyOpQkrnOnkOn7pmFaYZiYqQzo7wFmOxcQk+e+1qcxsU0ueq0lkYVL2lCHntvcdac5Awql4vV0GepuZum0IOnxK3NebQ3fBpyA8lpA78d5MQ0gattWUAACnl5wF83nrsIu3n3wPwe22uYVKwe+DVJagPnxHSKQpVOUPCSoUjh89VMCYXfP3sMTsnsBsJ/NRTHoUb9xxJQ0o7xfMfX42N3Z760dWOIKTxelBIp5RGpU0VarraT7BxcK15lU5XDl9+rLsPX4qvUM98pxhCGhL6mRfBCO/DlkgJKSUeXurl15E9F+6Q+N2m6mO4Ka9HVQhzNJLDt3ZvRsiZcocvfF1Gaxe6IGTGGDGtlRBCnIy7aMtM8siTFwqPDVulUzFM0Rbb0avn8OUfjW5U/JiocYvLueCzXSoh8jWlDl9xnuMrfbM65+CvnStsqzSks47DJ3MHzxZ89nHK/VPXA2h9+KyQTjuHz+fq2u0p7HP71q67kul5/C+InsP3yR278Yy3fRm3PbBojAnZZA/jwNjr5aa8HlV9uNTv1NQ4fKL4ZYW9nloOX9+R4EvIjMCiLYSQNmjV4VuvXPHbLyy0A3BonVqE9eHTfxYFR6+Ow6cXbXEdp8YdNxw+S/AhD3l0FY0BUofPbLKe3hp9+ALCCn1hML4cNb1KpzIe9XOq5+e0PnwKXfBFQzh8C3MRFlfMx1acRVvc99XL6Ot7CJiFWa688yAA4I79i4Pn0jF6aKsP6Xths/OUHm6se5KZJFFaFdI1jSGdmeCzHx/c1rmWnvZLPTnvGiHNMEoYPSGE+KDgawG7/QAwekin3qrBh+6gzXWiER0+PTzUP26hm19robWAyK87rdJZ/Ljt3L+Iee1cec+9YtiWLhzOOmUDHjiynI+R5vEKV6sIuy2DyzHJq3TmRVvU318V0tmNhLGDLRRt8Tp8xc9Hr+8OPdVR6wtpvK2LAvWzLRClrBYAVeG0orCFN+dPbyd34zKBXRmC35OJD+l0hGoXx9S3+Fz5vYTMCqP8fhNCiA+GdK4RoW0ZfIT08dMdpa7T4at+u119+FxiVT32rlf+IB5/5hYAjpBObVyZw3fdrocLj8cO10/98Nev/EF85r89zxjvK9riEht2W4bMDXOcs6uFdKpbvWiLUaWz0IfP7/DZrMbFnoi+PnyRY7026i1LpL+qYyLDt/+h/QTt+V3nXUte+Z5r8N4r7x7fAoYgD+l0v3DDVLa0jx0bdluGIUI6DcFHF4TMGJP8BRkhZHqh4FsjRm3LEHQOK6TTzr1zFU0pzOFoy6ALF+U0KtHyyJM34A3PfxyAomiJRC46e54qnTZZDp/RlsG8/dHHb8WZJ28wjvMXbSmew+/w5WPU+buRKIjJrC1DxwzpLPbha8nhE+b9MqTm8GX9BLMSntWbiyzEtqp/g+vY7HZ8G5hv3v0Q/uTz09Xtpeobft+XG5OM73M2TL6SEdI5PS8BIYQQMjYY0rlGjBrSGYIuMIQQ6HRshy8khy+9nfM4fBc+5VH4uR94NM46ZWP2mBKSK71i0ZaqHD6bTLg5i7YMBIvjtbTHKFx5bnruWpqHNxjrzOHTHb70B9V+IhLCyGHLHT5kc7twFm1x5PC5HDl1Xv08LpSoS6TU3D7TOZKOcxRRInd4h28IrbiuqXK98jDb+u/JOASSMH72FW0JX5jp8BEyW0xDKHwIN+89gsefucWZ4kIIWXvo8K0Ray34gGLOXojgygVfPlavNrlxvosXPemR5rwDcahX7AQGIZ0VVTpt1J84vaWftG5dl+HLNXOGdEqZCRi9tYIuMpNM8Amtgbg5TzeK3ILPqqZpM3yVzvRWTVtWtEU/Jsvhy9w6iTv3L2YhnfceOo4/u/R7ntdqMI/nXCFtGaZ727L2VImf3OFrfu62qBKvdfa2fcPhq3c9uw4ex7btl+BzN95f6zhC1ppp1nv7jizjZ//2Kvzhp28e91IIIQPo8K0Rw+bwfeq/PrfgnHnPIcoFXsgalDDVXTR9XtcUc4MHVRGVbiTQTySEyEMeV/pJoRG8C2fRFitc01UoRGpiRsfn8JlVOos5bnlIZ+QNoYsiOKt0ZsLUm8NX/MbT5fC5w1FlUA5XVolT6jl8uVB4yTuvzB57/Yd24M4Hj+EXLzgH523d7JyvFw+/+5im0MNJOJ2etgAAIABJREFUoPL9HcEBWMu3whCX1pcV+Xqk/nQQRn5vzTXdcv9RAMAlNz6An336o2seTUj7zMIXZUdO9AAA3919eMwrIYQoKPjWiGENvmece1rw2Ko2DGFVOh0FWqLy59V59h9NBd+ZJy3g/iPLEMjF4mo/LIfPJdwKIZ0Oo9AnynTRJERemVIv2qKuz8gbVM93hNdRsXMkV608PJ/DtyHQ4XO5MVK6exX60Kt0OnsSIg/XdBe4Qem5yt7RTLRPwc5lkjRp1VpGcfjGgRACiUw/3962DDUuxggvrvkarEGgBSEjMUt9+Pj7RsjkwJDONWLUKp0h2I6SnesWVqWz+JguXFz/gKt8v/uPnMCWhS42qSbxIl9TLw7N4RuIGSmztdhhX65ZfKFhusjJxao0HL6s6qVVKEY9lzuMlqCLzNc4D+lUz9dw+FyCz/EHP5H52kOLtqgXzGEiGnO4ZtPfj7rkYbYTvHOZwB1J/nq5X7dh8t6yY4dc06hkv7ueKp11MEKvh/xsTfJHkqxv+NkkhLQBBd8asRY5fFWCKiyHz+/gAe7rUEVb9h1ZxtYt85lAjITQKkqGCU49Z0w5aPYG17VGv8OX39dDN82iLcUcPiU4I6G3F7AFX2QI5Lzxej63C1cOn9203XUt6WO5s1Me0plXHlUhsL4cPaH9XHh+cOt1+Eo+17MQmjROfO9v7ooNM+eaxnS6fjSHZL9b4dP2HeHeoeRf+fBTSSaT/Oue6f2MTvPaCZlVGNK5RqyF4Ks6R5XLaB/+0dc/G485baPhHLpO0dFy+J657bSseIsQ5jlDBKfelqHbEViNiyFsahoVogkg+ytpiyR9c5jltWmP+9oyJIlM20oIkblbtubpFPrwmZtX/dp//6efiG/cdQiX335g6Cqd6vpCyvLrYZx2lc7COQI+mv0hcvimsX3AJFAllEd5XdfqnXje27+GvYdPAEg/XlXitc4GMRkhh0/PbSVkrXngyAncc+A4nvv4rZVjZ+Ez6sq3J4SMBzp8a8RahHRWnaOqD58tGJ/7+K147OmbraItxTn0Fg5btyzkhV9gFX9xnP8Xfvhs477uKKnrKYZ0po9f/bs/gbe+/CkA/HlNsSZUIt3hM0I683Nmxw3OL4TInJRiSKcwrk9VD1Sj9Nfq/EeehJM3zAFw9+ELqdKZrb1GmKVepbM6TcojCDFiH74Z2LisJXkOT0VI5xCv61q9F0rsKQo9IK31qNtdB4/jxj3lhR5Mh6+25Ks5npDm+Kl3Xon/733Xlo4ZppARIYRUQcG3RqxF43VfkZDs+YpF+J6ODMFXfF537rZuWciKoAghjDW5HD5dLAJ6RU6Zjc//AKqQznTMo0/diB8851TjOHsDqG8O85xArS2D0NoyaMfGUqYFXYRZ3VJHF4tA7oKpabqWu6nmV83rdeo4fHkOX/F5fVw6h+bwOQ7Qi7qUhnR6NtalnyhuXIaiyuGzfx9qzj7UmkbFey0wPyN//dU78Tv/emPpXKNU6Rz1OEJG4ajVushF9tnkN2WEkAah4Fsj1iSk0yGo3v+fL8h+rgqp9OVj6al3LneqzOHTT+l6DeY79jf+A/cqkYWcP1fhB7sJecHhMwRf7vBlRVs67rYMSSIRDfL78pw52+FTV5lS6MOnXXxHiGyNoQ6fT/DpeY5VpIKuKGj1c5Tn4eXvh5OSj5TvdSPlVL1avs960NxjeiuyCrueoi3q+ZV+jBXH74KOWcG33joY0kkmHeY+E0LagIJvjfD1ZGubFz3pkTjrlA0Aqoum+Faou3TzjvwzXdicsrGbCQghzOt2CU57Pgngk9ftxjfvfijr22cXTXEJR19ekzOHTwuL9LVliGUa0hlF+vnt6zaLtmTtDbLnNWc0EtkTrtewTtGWkCqdeqhcnsPnGAfN4fPONlwO3zSVF5+kJVYVMhl3Dp/UXOYQ9Fxbn+BT0yVJ9XUZbRlqXlF++kl6xwnRmZ5/N31M89oJmVUo+NaIqnDLNlECqcrh87mQunBxFRzRG6pvmu9C3dUbrwPuHD5b/CRS4nf+7UbjvPamUJ8lEu4x+nzZ2CwnUBpFW3xtGdIqncK7wU7DQfP7/ZK2DJ1IZALI1YDe3YeviJTSW0TGPLY4xtmEXoa5HnU2+Prc6Rom96//JGZ0VTmj487he8+Vd+Nn//aqWsfknwHfK57/jlV9XkZz+CbxHSckZ4L/uawNf90ImRwo+NaIMRl8AHLhMWwOn75Jcgk+vQH5hvmOEdKpp+i5RO98xwxv1P/YdS3Bl7sExaqhdnN2he5MGW0ZXIJP7+81qNKph3QWcvg6wihC0bNcMDP3Me/n5xLe9rHp+dwCzXet9jg1RmTX7XYMy8iKtgyRwxfSPoK4MJ1i97NDOnwNvBl7D5/A3oeXap7X/bjtnieyut2E/lkc9mr4mSSTisxu+SElhDQHBd8aMa6QTiAXGMM6fDrzjvwzvfrnpjlN8AmrSqeraEvXfEzfxHYHajHbFEKFdObj9VYL9vGAWV3SrKiZh3S62jKoKp16H75Khy+r0qnEpDZWK9rStQrVdCPhyeFzi8DYUzXUHqduy0I6gYocv8G19Idx+OB+3Ug5Mt/xeZ4fd0hnvXn0EGpvDl92v3pm/Xe6tsOnjqt3GCFrxigO/qQwzWsnZFah4Fsj1qJoi/fcgQ5fyBKrHL5N851sHgGzSqfr/POeKp3pvOn42x5YxKFjK3lIp6NNRF7IxJy/56rSOXD4VI6hms5oyyBThy9ty+B21OzrydxEV0hnicO3Ya6D1TgpzO8u2qK1ZSgRYVIbr19TYZzUc/z888WeHL6QxuvTwOGlVeyp6Vq1RYXe837Wg+Zu4D1Jaiq+sqF2nmdYSGfx+FAYYkamhSn655MQMgVQ8K0R48zhU+eu7MMX4EK6Co7o826cNx0+W/T88vO2Vc6XjR8c+4Gr78GFf/11QHOrFOquz4Vz9uGDzNouGI9LiRv3HMa27Zfgjv2LWbinLz+wEwljA90r6cOX5vCl2K0oNsylrqke1nnD7sN4yTuvhI3UNsSl+2JtA63egr6r9YMxt2OawWNDOXwjOFFrhVrZ52/ahx/9s8vGuhZF1cs1yuvZRJiYRL3NqC7i7H9hcodPfVaqr093+IapVJqed3I/k2R9M03Frggh0wMF3xohxvhKKyFX5TKGSFJn0RbN4ds438mEmoD5jXqnI/Dmn3sKPvy6Z2eP2Q7fJ67bnf2sO2EHFleMJuLZmgttGcy/ks4+fDJ93M5tTCTw2RvuBwDcvPcoOpFAJ4JWJKXo8Okb6LI+fKk4VCGdtsOXvgZ6L75P7NgNF4nMcw1Lq3SqW5k3u3ZV2jR6D3qKupSdq+wzMxWhSRO4uCoxMlIxnAYuVw/RDD2nb7j+OQWUOCyfbpTG63YIOCHjICj/eoo/pdO8dkJmFQq+NWKsDl+kbpvI4St3+DbNdfOQTqvxekdz/hS22/W/L9upzWu3bJCFEEK7aEuxD18xh08iFU2dTAgPxlp/hItVOs25Cw6fEnwq19AQfLoQdDt8eh5fWZGLONscu8eocfot4Hbp0pDOdnL4fMVuJolJXFpVlc5xt2UAZC2dnMh8++dvy5BfU5WIC+k/6UN9+TGBOp+sI8o+f7P02WRVXEImBwq+NaJKbLV6bit00UfIP86upuH6tdkhnZHlcgGmKxQS0qmQslhJtNiWwe/w5W0epOHw6W0Z9MOzHD7PBryrhWmm5zLbMugOX6Q1Xrcdvo1ZSGd1Upa+IS7N4dNeD/W29h1JX/o1uaZTh8RDJIxloUljklUh7s+4Nlfbtl+Cv/ryHc7n8tfNTe5m1z9vIzl8Sb33VEIr2lIM6jTWlfbhK5/PdPiCl4HBAggZO+XRGZnFN7XMkmglZFag4Fsj1vKLrlM3zRn3Q0M6QzSpS6DpLl0q+NKfhSjmsaWP54+VCT67uEkiixtGO4fP/kMTu0I6YTt8A8Fn7TRVlc7MPbSet1/PLKQzOz6/NiOks1C0ZRDSaVTqdP/FlDK/ptKwIG0jrc7mav0gUXwNdZTr6Wu8XvaRGndIZ8h5x5nL9TdfvdP9RNXrVuEAto2s6fDJkpBOV9/MWn34htwVcz86PUgp8b6v343DS6vjXkpjlH3+8pDO6YffrxAyOXTHvYD1wlqFdH5j+09g87z5trpCKbPnIpFtoEJCOl05fIbDp7dlgDBEZC748sfsHD4dd0inOcZuGl6ew5e7gUurcSa89Bw+/WjVON4XmtjVQjrnOiJz6NRGfMtC7obqDp/9OmchnYEOn1qHrzeeuka15qpKmlmVToe1oj4bo/R8G58waWbMWiOz26qQzmHmHv2Kpaz3uumudDGk0/yiRsrqkM2RGq9b5yWTz7fvO4w/vuQ2XHvPQ3jvay4Y93IaIeTfU35GCSFNQodvjVirtgyPPnUjTvE4fOrvx9Yt89lzuhAdti2DTicSmcAoVOl0OI1llUNtJ0wXJ4q88mZKQfDFeg5f/thltz+IZ257hPF4sShL+tpljpq1vkhz7eY7USYu1bhTNmmvcyTwuxc+EU99zMl45nmPwA0X/SRe/KQzAeRhsiE5fHrBjLJ9cb6BLt80VIZ0KofPc7LSXJSSedeCSQ3pDC3K4n1+cDucCK99iPv8dRy+kvPa4tYOq3YxSkgnc4qmD/Vv+JGl3phX0hxh0Qftr4MQsn6g4FsjJqHxeiIlPvy6Z+Nzb3p+9pxePyTM4Svm8NkoY05Yc7qcxrLcxmIOn/SuMRNBlknmcviu3nkQB4+t4j8842zj8SQxN5sdkTuU0iomYYvR+W6EOJFY7sW471Daz+2Ujbnw7kQCTzrrZHzuTc/HloUuTtk0l13fxvl6RVt8fQF19JYIVdU8y/rwKbHryxcMCk0a08YlRGiOY2lV66oqyx7UlsM79+ikRVjqxXRmPTSLTxm3ISGdiRHSSWadSPv7NSuEFG2Z5qudobeKkJmBIZ3rACUs+onEj3//GcZzacXIVGiEfPldlnNnny8ahEQq8lxC+/xu3Dl8JpmQ9oR06vfVWnYNBNkPnnOqsV57Iy609dv9wTLXdHA/FcI9/PrHvoMv3bofQFHw2ajzbRi8pnrRlrJwvrxNhHOIsa6qvmZpjp9/Q6WEni+Hr0x0jlJNsglCRMlahk1df+9DePrZpwb02UtvfcN84cshNHK9sp5rm4ZKl39+zM9r+XyjtGXIjxvqMDIGopIvpKaVsn+bZqEPH9syEDJ50OFrmd/+qSeMewneoiSAKUSGzeErzplbfPr8uYArhnm6sNeThnSWF20phnTm99Whx1f6AIAtC93BedLHY8u5UEVb1Lz6y9cRpkhSQliJPQA4aUO3MN6+HqBeW4bVfu7wuRqp28dX9TXTX6+ykM5RWgCM60//JG2Ybt57BK/4+2vwl1+6o/K1zESMLwxyzA6fRD2hJXWHz/6dHtzmrmW1exiP4PDlAnOCPhykFCHcX8hNMyHh+NPMLFwDIbMGBV/L/LcXPh673v4zY12DElWusDxdcJ11yobKuUIcPiXs0pDO4rlMh88v+Oz19pOkkPNnN1K2NZA+hzr/4kofkcirY+bFXMzzRcLsUadv1F0hnTYdh7tpks6n1rESULRlpR9nm4Wy3nj6t8RlRTCMkM6Soi3eHL6StebCpb2//l+46QFcvfNga/M3xb4jywCAO/YvBufoeV2xwe24Gq/rffXCxiP7DBSEYiGks3pjbxQrqnk93IdOH75/n6eZqjD79HZ6rzdrw8KUWUImBgq+dYD6g+mq6qhE0G//1BPwkdc/u3KuMIcvb7/g7MPnaNXgwv6j2ItlIQTU7sNnbwpcOXzHV/rYPN/N1pG9PlYOnxHSmZi6xS6E46o2WpWrqI7N+vDpDl9hdMpKPzFaJUgpsf3fbsR1ux7CnoeXCnNX5kTJ8rYMWUVQ3y484JvqNr+Z/y8f+TZe/b5rnc+FVcJrekVu1HvWiUT1uip08kghnU1V6awxjYTf7c2vIXeSqzb2+mex7mswijtKxkMeZTHedTRJeQ7f9H9Gp3jphMwszOFbBygt4gzp1Aqp2G0QXISMyeaEu2iLy/VzYW/m4iTBnO3wWWOLx+iCL709ttLH5gUt3NKTw9cRIn/ttGIpQO7oqUcW5hyCD8Vr18mPDW/LsNJPtMqZCRZX+vj4dbvx8et2AwBuePNP4pSNc0YrifJcv/xJl6jLHT732sodvuI51pKQDZO9NillK5Uc1evYEdWCr+r1GqktQwNvxTBhlL5CM5ncq+Hw6WHa9ddi3pLJJ7LC52eB8h6q088subGEzAp0+NYBr3zmuQCAp519SuG5MsE1LJ1O7py5whp1IVQW0mmHEfZjWViv2pznIZ22K5gUxh5f6WOT1iPPV6UyiszNhj715kFlTb0tg42uG1y1adT5VA6fUbTF8/dytZ9kwjORwNJKbDyv8hPVKyItoVpcQ75Q17CsSudIRVu8Q1ol5LT28ttaa9brMgqo0pkJ5YrnhyraUvsQxxz5Z+vA4go+9I1dlef0iX/7WnVx6EN//s8v/R7ef9U9NdYePJRMCGKdOXz5mOm9YLV0hnQSMjnQ4VsHvPCJZ3rzCEuKZNbiY7/yIzhtc1qVMsvhE+Y/+PrjijLBWRBvicScJazyxutuB8Hl8C0u9/HoUzdqj7uL2kRCGAUD9I3mRqu5fVVuY3nRlvTY1X6Cuw8cw9Hlvtfl0XP4AODICbM3ld2IXsryb8allKXNqNWxZU3evXNn89Y+tBGC+vBZ9xMp0SnUgh0d9Tp2oqhyXVUb23G3ZdA/W2/86Ldx7T0P4XmPPx2PP/MkzznzL0uKDp95LaoPX5nTqrvNDy/18LbP3YrX/eh5YWu3zksmn3WXw1fxhc80MM1rJ2RWocO3zlE5caP+LX3O952OJz7qZADuZuuA5vAFtmVwhXTaAtHO4SsP6UzHrvQTbJ53h3Tqm4qopA/fpszhS++7chsrc/gGtxu0xus/8ZdX4OfffXVhrGKllxjXZAs+e7Nwx/5FHDq+6p1Pf7ViKXHVnQdxdDmfs5k+fGMK6QwZU/IFQZPkIZ0hDp80bn2Mqy2D3iT98KAZds/jAKfn1ISd4zl9Xep1KltmWeTztu2X4LUf/Jb3+VHEMhkPsxjSWf5vwPRfp/ryVLTw5RkhZDgo+NY5bYR0dvWiLZrq6TpCOjudGg5fLAshoHYOn+1EuYq2AMBmR0infWwkzIIu+nIywTf44+xqSK9fp8utUJtclf+3qm+aPX/zV/qJsWm3BZ/Kb1Kvx50PHsPX7/RXsZQyv/5jy3285gPX4tPf2Zs9n+cL+kI6vVMXeqytNbI6JbLg9LS1qcwEXxQF5PCZtzYj5fDVP6Q4h/algu0ou0ik/3mpjdHnKXuN4iQpDQW//PYD3udmwT1Zr7T1Zcw4KO3D53HDp4kpXjohMwtDOtc5rlDDkeccuHYCZshoVrTF8ZgLu05IXBrSOTimLKRTO9Qo2qKFDOmHR1qVUTuk03b4fCGdX/wfP+ZtG5CFdA7E4uJyzzlOZ6UfG8L0qCX4eoMXbZjNQloQJnURFZU5fKUNhFPKcgjbJKzxunm/rU1lHtJZLSqrNnyjbASbyeHLb/Mc2rLPgYS38qDMxwBhYrafSHQ7orQtiZ/Z2YredeAY7ntoCS98wpnjXsqaMEN6r/yLsux2ei94msUqIbMKBd86J28v0Ny/0HquXlXRlnptGYohnXbRFltc6Pk+usO3SQvpVI9f9JlbjGOjyAzpNB2+7uDx9L6vaMsTHnUSnvAof24TkBZtOf/MLfjOfYe159ysDkSZwnb48pC4+u+nKhqjv+6Z4PPMN9EOn+PEV+88iO87Ywse5ek56SlGOjIqDDESImAzFOYADsfo70YWFqnNVf450DfrHkfV+sKmTBQniWrPUv/NytYxAxvSD159D758635c+/svHvdSWsX+MmAWqPp9qRoz6ah/+1m0hZDJgYJvndPNBF9zc+q5eq6QzuDG63Z4ZiyLbRmsoi0hOXwAsEUL6XQ3RTdDOl/13m/irgPHs+c2zpshnC6Hr+qPXZKJAOBHHnc6PvXtPdlzq333ZnZFq9IJAIdth2+gLOq8neolU6+VLiirmrxXhfKlt+Ny+Iq8+n3XYuuWeez4Xy9Jx9ifl9ZCOtP3pROJShexagmjvJ6NOHzaXCH7OSmldxNr66+QHDvl8NmEOMm5vpzi3fSAOJEzFeboIyTMd9qoKqQFTPd3EtO8dkJmFebwrXN8YsfmV55/Hl773G1BY3URGVUUbSnL4bM3cHFS3Xg9kcDrf/Q8vOpZ52THKPTz6iGdvpdAL9qiiz0gD8PMc/jK+/C5yDadA8F3fDVvsXAsa6+Q041EGtKpC74lsyBL1qOsxl9cNbSfCT6HwzfExlIXBlVcdefBwrWMis/lPHhsVRtjPtd60ZaAxuuhbRvSn9d+a6WfMuQb/NThcws5O9QzRJCl/w4UT7zcjx2jrfPN0FY0SabbBQol+7e9Jfd9HISI12l+b6d57YTMKhR865xMnFWM+4OfeTLe8rKnBM2pwi7jRDrdPOFw/VzYbksvSQrf7OtFW5RA3LKhi9/76ScBMJs0G0VbHCGdNlEkvKXhVSsFtTkfxuFTlycgcN7WzcZzrny++W6ElZ4Z0qmqJCpcoi0U9VrpIkK9B8MIoXyTX37s8ZU+/tP7r8XrPrSj9jnKz189xh7SloBSH8NOVB3SWSVK9DXWfVuauDqXmx7i9KbnNwdK6/GQHD5XLi8AnFgNEHwztBFN213M0AV5yFt3zM61huTwTbNPpj6XjOgkZHKg4FvndFoI6VQiLpFmo3QlrPQ/AuV9+Mz7fVeVThXSCa0whtbw3azSmR9nFG3xhnQKb1EZW+A5BZ/zyBz1mkcCOH3LvPHc0eWiw7fQjQaFVfwhnf0hQjqzYxOVw5c/VhUmV7oJCzQb1Xt0x77FqiXWotjku7iSgsPX0qZSvY4dUe3wVQpC7fm6QrzpkM7IUbTFfp3LirbYzl7iEJM2sSek80SvWvDNUlsGKadZEoSjR2/MCiE5fNPMDFwCITMHBd86p50qnbrDVyzQYjym/fzl3/gxY56nPPpk436au+MO6UykzDbrabGVfA32WMDdlqFwHcLfmD5rwzCKwzc4WAiB0zZZgu9E0eFb6HYGRVtKQjoD+pgVFyKNY42QzipxUvrc8G5jI1indW0YbVHYWkin9tmsDNmsmMvMsawp+BrYiuk5Rq62DPaS0gq3yI5xYTt7ZS01YlkM7QaA5V54zN8sbEglZkMchDJLbmZVVVtgut/bWXqvCJkVKPjWOZnD1+AWKHf43EVbfA3JddftD376Sfjjn3+qMW8v9vffklILkRT5OYwqndqxes6dT/RGVh9BHXW8+sPm6sNX5fHp67UF41FHSOfCXOTI4fMVbQl/P9XIfmw6fHqxDe+xZaF8SfUYfQFNbxHs+VxiriBOWqvSOXD4AnL4qkLX9Pe2bmuCZh0+d/BZ4RRSOj+PruN9xZd0vDl8A4ev7IuWsuv/m6/eic/ecL9/wISR/ps3+xvrWXT4Sq/Fcr2nkmleOyEzCgXfOqeNkM5OJxdD+r7M1ZbBl8/3jMeehg1zpoha7sUFh0+tP0mkEdKZO3/a+fW16O0iPDtEIfw5fKpZumKoKp3K9XGMc7kV850IN+09gjv3H8sesx0+V6XNUHJ3UBpzDUv2TXXFOOV+Nb15tUWDS0QURGFbOXxaSOeo1zlSSOdIZ07R3Tr10TXEWyGks9hYXZ9HX1hIZdd+XHT6gTyk01VAKT+n/7P2V1++A2/62He8x04adt/QWWUWc/hKq3Rmt9N7vdO8dp37D5/AL73/2qAeuYRMOhR865yyHLqh59TCKfX5XQ6fTuQYq7PcSzBnPa7G9bQS5Xp1TWN+R3ipfV7jOiJ/Bc+FrErn4L5jA1pF/icx7D1YmItw14HjuH1/nutm5/r1hqnSORgbWyGdIeKnNDRJcwrLyPu6NUshP88Z07k2IZ397LNZrwqn+/l8QN2m9k1smo18PKsPpv0zkL6/2WfBk+tXq2iLdDt8qmiL221X53SvcWm1mDM76cjsf7ONnd85C4Tk8E3z5WaRElPeiO+mvUfw9TsPYtfBpXEvhZCRoeBb5+Qhnc2ht2UQDifN9zdA38S5hOhKL3Y2Xk97m+XVK/UcPh2f4PNRVrRFuQhq02o7fkB40ZbQv4llG1mFCmGtF9KZjlViMXNZAsIbQ6ozVm1ckkQGjSueuyr00cQV/ugSJ23Q08Jlq9syhF9XW45kENL9GXcVZpEOpSWtMYD+WagI6Swp2rLB8bvoOqfO/YdPeI+ZVNaPw5cySyGdIV+8TPPlTvPaCZlVKPjWOZmgaXDjqBdtcT3uC6HsVAi+5X4xpBNIhWI/zisBRsItony5gz6UmHSRCb7B/XnHunzhoAq1XjXKFRaqo5/DF7YWDyme0mOVKCl3+PTXJESaVAmYLKSz5jah2imzQjoDcvhac/ji/LWtEtKVAlkvqjOOHD6H42IUbXG0XnA5a65CLyH5WlU5fHYouLl29+/HnodTwbd1y4L/xBNG+rrO/tY6JK9z2ii7klmQ8bPyuXRFJhAyrVDwrXNU4/NGHb7BnLZgqArp1CvvuYRWz9GWAQDmOhF6sRnSKRxhnb7qoD46kVlIRmfToI+fusSh2jKocYO1lOUeAWZjad+mVrl0dd5PdQ15H770vk9M6K9dE6FJoU6gTZXYKYg5Zw6fJQpbc/hy97SOg+d8XhtQu2hLA7/peghulH1f5BZ/arwrd84I7xzcxgGb+1Twuap0Vufw+UI69w4cvkefusF77KQh5Wy5Xj7UJc6IhgBQkcM35L+Hk4T6XE53QCegPn3T/F4QomhV8AkhLhRC3C6E2CmE2F4y7hVCCCmEuKDN9ZAibbZlsB2VaESHD4AzlKvbEehbIZ2u87gKyJTlqOeHAAAgAElEQVQRCYGN86awmu9E+K8v+D485/tOB5BvRlwCLLTxulrK217+VJy0wS0wAeCI1qrBt6nVnaS6ZG0ZrFsb87r85wnJx9LPU3fFdfvZhTh8bVXpzKqnypDKp+UD6vRJLM5da3jpHFLKShc7Haht2j1rsfvvlRZt8YV0huTweUog7h04fGdMkcOXSHf101lj3eXwZbfTfL3TvPYc3xdEhEwjrQk+IUQHwLsBvBTAkwG8SgjxZMe4kwD8dwDXtrUW4kfPt2t6TvsPdObwVRwH+IXonC+kU6vSqaYpCL6KojA2kRDYPG8KsDNOWsDvXPjEXJAOzrlp3iH4Kr7fVOtV437+hx6Dm97yU9joce/03nw+h2+YPnzFoi2D+55JfIK9MG92W76YYcMoKx0+u8ee0+FD5RjF/qPLmYtUF+Wexokcqbdh+nw+YjxtGYoC3SfkANPhq1pXiNubeEI6Twwq25bm8Hnm3Xd0GUB1GPYkIbFenIf8d2dWKHf4pl9lzMrnMneXZ+SCyLqmTYfvWQB2SinvllKuAvg4gJc7xr0NwJ8BWG5xLcRDG334OoNwK/sPdNaWwdf3LsDhcz3ejSL047whuRKL9mmMVgyBgs8WcnYkmbpC4RhbtXc886TUTbA3qHpTeJ0jhuArd/iGoWe5gz73yMjhK/umOjA0KRlyg1MpnKynQ/rw+TaVR5d7ePFfXYF/vOLuWmtUuIq2eD8flQ5g/nNc05Js4rdcF2XqEvTPSiGHT+Yndrl6+sp8OXY6/SRx5vIqMV6WC+trAK+OnaqNnZxqTVCbaXprqghz+KaXLKRzer4/cTID2puQjDYF32MA7Nbu7xk8liGEeAaAc6SUl5RNJIR4gxBihxBix4EDB5pf6Tom1K2pg9qL2XtnnxBzz+Fx+FyCr5MWbclCOkV1SGdIKGskijl89pzqD4IASsMxXbzzP/4g3vELT8f5jzzJeHzTvHuerOUCyh0+tWn9xQvOxg+cfUrlOtSsdh++VY941F/Hsj+EZT3PXOPqhmxVhTPaz7q0UWgO37/u2IPF/8vee4fJcpTn4m91z+zu2ROUdQTKCBEESAgEJhkTBJhgY8AEy9cZ29jmcp2vudcGg3+2MTa6mGvwJRgMXLjGGNuAAQE2VkDhIImkgCSUUTjSOTo57M5Md/3+6K6qr6q+6q6esDuz2+/z7DOzPd1VX4eZ/t5+v7A0sEh3E/RJywu3WI+LJo3Zmyt8Y5H4yhepv8u5BHqDHP/nktvQH0hvdc6J5Yq2xPThy3KJLhPSGaO+mp5u9vLlvqpuOzuQ64TxrbeQTvNwZHb3d7bDUQ1CvxctWswiVq1oixAiAXAhgN+pW1dK+QEp5XlSyvOOO+64yRu3jhCjdDWFUvjcG3RaE9JJEQq55J7sd9ME/VxqAqCc0MqiLTEKX8IofMEm7cDmha63rApHLs7h1eed7C0/YkOXWRt44slH6vcLgTylfmZyxE48chFnHL+p2ggYx2LghHTuOthj17cVvvo7YR0nUbyy6T21vmiL/TmrCEYofHku8fGr7ireD3nn7w9MDl/I7KV+hu/dv69Z0Zas3h67UMro4JwgKSU+cvkdeMeXbsJHrrjDWj9ctAXe+xjyX/ThCzdej1GdXSwPhs99XS3k+dpxrKvgXhtrATH7Mst7u1aKtpjTNMtno0WLApMkfPcCoN7sSeUyhc0AHg/gYiHEnQCeBuBzbeGWlcUkc/iatmWgCDdD95ebPnyych7RlPAJ4RVHcU1XDpeA8BS+YfOBQoTvY7/0VPzo404AYBevoe0a6HEQIi5XUZ0nt+DLzgPLAIAjF2176HGNUvhq5o9VAl3UhXS6xIoN6XS3Yda57NaduGPnwUa2uRjkJKQzwPh+61Pfxov/+jIrV5MD3Twmr4mJnBwJNFRXXeOZlDiwXDQvP9TLvPW50Ci3sictaFPZliGrLtoyTEGM5UH9ttMGifoCQGsBMdfErKFa4Jv9HZ1ldZJC/16sjd1psc7RLAatGa4GcKYQ4nQURO91AC5QH0op9wI4Vv0vhLgYwO9KKa+ZoE0tHMQQn2HH9BS+QEjn2378cdi6ZZ5d1wUXytVJBPpMSKefw+fbWIWin59DGp11VJggp/ANiyMW+XG2LHTx6BM246Ibtluka9NCR6tx/cy4CwJGba2CJnyOwrdjf0H4tm5ewJ5DhoTEqsKxoVjDFmOoT19zFD42h69eBbzitp2YSxPMd5LhFT5VtIUofO61te2OXQCM2hRC06It4/ZVDJGXJodP0iJE4W2oMdL53FUMQygUPn8WddyqHOZQjuAsKnxyfUR0rhnyQLHW2zKsFejfi1W2o0WLcWBihE9KORBCvBHAlwGkAD4spbxBCPF2ANdIKT83qblbxMMUbRkfTJVOe3moaMvPPeM03y6G2BVj8yGdgyzXDr0a3iV1VtGWqD58/jpVx8lT+Gpn4BFS+ABTkIKSl82E8A3y3OQVRip8utG6k8O380Ax5vFb5nHzA/v1+k378MUWIWl6DQ5btIWeencdjj8dXB5g80JnpKqIdlsGfhDTR7J6LLtoS4zCR8MoR/+mc0+9XcJGUbQP8Oe3CB5sJ7i28ToT2k0L4wRtDzjTKodvllBcj2vfFV2Le1h13tZCoZBZenASgzW2Oy3WKSap8EFK+UUAX3SWvSWw7nMmaUsLHpMI6UwDIZ0KUUVbAiuF+/AZRzoU0hnT58+2kyF8zi7Rf7d4IZ21U7CoJHylozsg8tYmUlhmkJveXEKIqP1UxMkN6dyxfxmb5ztem4jYHL7YYixq/qbXYOOiLYwC5a3DjHloOcPifIrDvUyP8boPXIljNs3jvRc8KcpWlWuX52FCovsR1hJkovBFVGW1wyhrV68fjzik6hqXMkwlVbhmlWFubmNtHz7muu5HJIOGrlcV0jlLjqqUayvMMYQZOiXRiAo7nuEdn2HTWczyuWjRQmHVira0mA5MomgLp8JZcw6proXG7iRFlU5FHExIp5vDR8aPrNLpwv3hl2TOrVsW7PmG1PiOrCB8KqR14Ch8CgNStCVa4Sv9ZC+k88Ayjts8z+RCmvdVt0FOCQKAWx88gDtJTtyk+vC5zrtR+MIKJTfmwd6g7Mco9D5ddfsufOG790fb2iNkOhT6qIl37X6Z91EhnY6SNipMWKTU13hBZNWFZ69P1T9XFXTXC31GkeWS/X1QYbOVIZ3Oq4IK6RxkEp//zn2NG9qvBtaLE7oWctpcVKvQ6jqeXayVS3MtqK0tWihMVOFrMf1QxGecN9W6tLEYGhQkfGwOX4JBnmviYtoyOHbRoi2BkFHLBoYUhm7UQgBv+JEzcPWdu3D5rQ/pZcNgS2VIZ6G2UWKyad6sP8hISCdE1H6qsUzj9eJ15/5lHLtp3juf1rmJYHyu837+hZcAAO58x0uL1Yb0DhqHdGpyTk2U7DoUh3oZFudSJGJ4R0YpspkMNyF3j38I1OaokM5AGOWw4Ig8JXXugw4JWpjHH0ctt3P4wvNnpcInnPOhjnHVtqaHoL2SInzb7tiFbXfswqHeAK99yinhgaYAVAmapYbxjbEGve3KkE7vzezBFA6b7euybcvQYi2hVfjWObTzPsYftLEofKGQzkAfvqJoi+3Qj9yHj5nLdcbpvwvdFL95/qP0/8Pe6tzefxQqh4+W46cKX98K6Yys0lnuhMkzK5bvPLCMYzfP+aGxY67SOXzRlmaET7ftIGcmtA7FweUBNs53SoIxnK3qfLnExprbyaUMwSY5TRW+0b/oVK1LdM9Nvy0KXd88Kafkk7yHTYSrDvMgl0gSXz9X/f+qC2Lwny07PfxUwaJpRkyT+rWAtbh7MyAgj4S1snvc71aLFrOKlvCtc+gcvjGOWZc3FvPQLxRqGurDNyDtCGIUvojilaydfg5fxZxDMr6Nc3yPPcCEdLpFWxQyGtKJuCqdeUDh23t4gCMXfcIXXaVTvdZcXHVK3bDbeepdRHOoKoVPQAztXNOQzhCh00przSR2W4ZmxUbGo/CZ0Ekd0inDvyE0P69K4YsN6ZRSIhXCUw/6SuGL2w0LbmXUWXDIpfO6VrGWCC3NeQ1hLZAMtX+zre+Ra292T0WLFhot4VvnmEwO3+iEr8nYaZnDp5w03eC9qg/fkCpjSOFTq9I5hs3hq1L4VF/AUNGWfp7re1MiREOFTxG+YvnyIMNCJ2WIs3lf7bjI2nWKz2tNZNFECQPMflpFWwJ5fhQqhy8RtpLVBANybGmfRAo1LB2fP3Zm2bhz+Jb6We3+qUsvFNLJz++rvfb2dpuNKsKXSz5KgKqo4W39dQZZ7h3HWSjeYpTW6bd1FMQQn9t3HMCfffF7U38s1FVb96gKmG2iO8u2U7R8r8VaQkv41jl0W4Yx/kLXKnwjPPfjCEy3rNLpOtJ+WwY6Tv2lzzmVHuGr2GZYYrspIqTTVvhoDp99HNQxOH6z3eeQwvThs6t0Lg9yzHcTJjQ2NqSzfh13X5qgTtzywzWL1yr7OUdfVekUoijasjTIvHXq0LeKthTLQt+DzCI+/ud0v2OOnbVPFd/zA8sDPOaPLsK7vnpz5XjUCVKHsqoPH20QHgovlU5uY9Vu5VKyBZVo64ug7Yx60mMqnc6CwxobMj3riDkXF9+8Ax+49HarX+g0Qj0QXOt9+GbYdAvrJWy6xfpAS/jWOXQK3xh/0LjCKhSjKHxdJqSzkxR9+JQyEQrppASwSUjnTz31ZL3MC+l0nNxx6KVnPWwLfuGZp7GfdXVbBmPIJhrSmdtKiyLIbgVRCjekU/WK6w1yzHcSTwW22zKE90N9VKeWxKopH7/yTrzsf18WvZ2rDChCW9mHjyGRSuETpcJ3qDcM4SsVPtI+JAQapsnto9V4Pas/djLw3sW+w4Wz/Jlr7w2uc/HND7LEqro9B90PXm6UsPc1NJ4slUSuGIS2K2gJ/9kS04Nv2pUiihkydSjE7N6skd+Y381ZDukMRTHMGmb3DLRo4aMlfOscqsLjpoXxFWytU/hiirY0GdsUbbHX8dsJkJDOiFBHtc6fveIJeNerzwHgKw9/9epz8MxHHoPTjt3ozTnsXiaJwFt/7HHsZ6oPn6Xw0ZDOLNd3KdqHb+uWsMKnhtJtGXKT0zTXSSqJc2UJ/Mino7GE748+ewOuv3ef/r9plU43xxNg8vyYEM+lfo7FkvBBFopfU1D1VJ+6wAVCbeCOjZ3DV4SYfurqu00fuhL/8b0H8C/fuschZmEb676W1961Cz//katx64MHirEsm3yFnc7JTSudf+w+fLwNankihEfKBnn99capJ8uMYjsTOXyMWrkW0YR8TztRV1+NSSh8//CNu/HHn7thOMPGjCk/DfFYJ9+xFusDbVuGdY7XnHcSDvUG+Jmnnzq2Mevy40Z56Me3ZRDIckn6rMF6VaDEJYZ00n5+KpTSdSjOPeUofOL1T9P/02EnUZK622EUvkDj9UQYG2jYpwvTeL0kfFJqwjffSStDOqugDlVt4/VmdUc0mjZeV/PYOXzVYx7qDQAAG+eL45BLiUP9QWNbMyaHj8752g9c6dnJ2aeWdZIijHmQS3z62h/gv3/mOuw51Mev/sgZer1f+ug1AIDv/vELybYxiiC/zs4DPccO0w4gz8NOXqhoi6XowVY+Q9eMWu4K/b/xiW/igX1LlfbT7ekay5zCNwMOXlsy3sCQ39lAtQo9nFr5B/98HQDgj3+cf1i4kpiV81CH9jvWYi2hVfjWOTppgtf/8CMw3wlXhmw8Zk285NhDOssqncphTHVIZ5ioxBQzsdo4RFYzHYfCVwVO4VucN+eun+UkRwxIU56oUvg5fEb1mO8kHnG1i7aEbY11moctkDFs43V6YjxS6GyjwjcX5zooBb6hQjrVuG6uGgAc6A1w9Z27jd1kv0IhnSa0N9dEbHcgfym2aIvKKawuvmL/rw5lJmXwfEsQtTdkl6fwVRM+lU+p8IXr7tehmQ0Ll3oVOot5mo2xGlgLuV4utt3+ED77bTukOGb3ZsUxb1Klc5ZhUh1mO6Zz1h4ktGhRhVbhazF21DX7HkX54kIxu4kd0pkEQjo5AlcFmrum1q8jJ5TrTiJ/QVXppOF7Jx+1iE3zHcx1kjKHzzjFHW03cNRilyUFap8sha+vFL7EU1PskM4wlONdm8M3pHc9bEhnE4Xv4LKr8A0X0kmb26s5Q3mfdkinP5aURaGiw/3inKnzHbykHWIVQt316jqpErYDGypGo/Lu3DEs8gc3hy9kQ/FahHQG1onYB2oHH9I5/S6eUSun39ZYvPYDVwEAXv7EE83CiN2blfBWUT42inkoMQOXYBCzbDuF2o1pDxVu0SIGrcLXYuyIUc8A4MjFcJhhaMwuox520qJoS+Y4vq4Da7VMaBDSSeevIyfU2Z3E080uo/CdfPQirn/bi/CorZsKAqDmF7Yy+Y3/eT4ufM053pjqXqZz+GhIZ7cmpLNS4bPHD2HYPnz1ThOv8NnnPqAClqAKH0Rx41dhnk1gwlu5kE573axO4ZPSqtaquH/oIYZVDbOp4RWgCl8uw0530aOPU/js/XT78O1b6uPbP9hjjaWOTdVPTF0BGdcOmq9qBgmPPy1YiwofhxgSJ703043KhxJRa0031hpBWlt702K9oiV8LcaOGPXsr1/3RHz+jc+KHvO4sq0Ad/NX+UxuUQ6/LUMzAjZcSCf5ZwIK3xzJ4Xv9s07Hw44w1Te7aVL04SMKUocok900iTo3bkine9y4oi37l/pY6ttKibrp11fprDWJxdAKX0VIqlv0Uit8c6lWlUYJ6bSKtpQYOIyP5mdKhtTm0oRND3JJHnQECJ+l8NWHktE1DiwP8Ddf+36hTLrrg+TwyXD1UQm+d5/73s7vA173/qvwE++9nA1xjbmOWVv0UwizTKnZC4TwzYLCx+zKmkTMqZiZKp3lZVtdtEWp0Cth0GQwCyHRMeB+L1YLH7z0djz1T/99tc1oMcNoCV+LsSOmqfnLn3giTj56MXrMU48p1j2w7KsrnVSgl+XYfbDIZUoCOXxM+l8l6PamX2H1NraK2Gw+F497+BZvGVV2/vBlZ+HKNz9ff6aK1yjXnFbppGGedZBU4eskTPEbvy3DL330GvzpF77njGO/hjBsSGfzoi0+MaoL6TxUktjFeZXDN1xbBtPygskt9BS/GoUPQLcjynFNO5KwwhcHrin5Oy+6CX/1lVvwhevuZ3P4zLY0ZJYJ6WQUPvpdlrD3VUqJG+8vKrIOrONRvFZdx5XONHM01MONDXMmF3YmHFb9/ZoFY4dHzO7NmtoZtU+TN2NimGXbKczv1urv0Z9+8Xt4cP/yapuBe/ccxuW37lxtM1oMgZbwrVO86fln4hHHbZzI2G7ftnHgL3/yHLzxuY/ED51+jPdZJ0kgJfDb//idYv5AH77mCt8QOXy0Smej2Xx88vVPwz//+jOsZVxIp7ExQT8jOWLCKEFu3lgV7By+1CPw3PnduX8ZOw/YN6PYQgpN1RTl4NYWbXE+58IB/fBD+391HObShCh8zUI6XQKn/tdNyxuGdOZS6tDmpX6u8zlDX7tYQsCtdWCp2NceU9iEhnTSPD2uLYN7qm64by9e/NeXWSvRdajJVAGtzVdE9fXGqZjq4cZCN/XWm2YMW81x1tBk/6bBMa8C/b6EwOW7zhpkkxvOFGPWHiSsBJ7/rovx0x/attpmtBgCbdGWdYrffsGj8NsveNSKzPU3F5zLOowcPv2Gp+OhA/5TrGM3zeN3X/Rodptu6hKS4tVVAZoWi6GKiUucQhilx6CLIxa7eNIpR1nL5jvhZzTdVGCQ0ZBOoVXNJo1wiz58ZUhnl2m8zqTwuTlYQHwhhTri5tknCxtqQzqD84QVPndMXfk1EV7j9dhcVXpcsrw+pLOuaAtkcR4TAfzNf96qF4cetNAhqg4Zp5jSI8aRY/Wd4vbLjCE9NerG+/Y569jHad+SKS7EKXxV37Oqq4ILM9YKX5cqfNPv4XGtLtYiolqJ6PM6aWtGg37IU/VQYspJawzWyjWpdmOt7M84sMS0sWkxG4gifEKIjQAOSylzIcSjADwGwJeklHwd8BYtCF529sOj133KaUezy6uISuoUcgkrfNFmeHMq4lRHMqy2DJPow1cRl1q0p6AhnYAiN6EKihzsPnx+SKeVw6fUNiY3LY90wpo617mUSCHqQzoDZK6q5ow7pgkhNGM2Demk1wwN6VTnwlX4qA2c8ychkQiBTpKgR6q1hkKprVw5Zrylfobbdxwk/Rzt0EoAJdn1x6YOrBqbK0rjzusStiKHz6xz8/b9+v2AJFbGFG2pDun019FqtqXwTb+Hp22cflNHQszuzZoqVkXq1sJpXQukFYA+GWtkb1qsc8SGdF4KYEEIcSKArwD4GQB/PymjWrRwUVWkwVX41LrDNgzn1k+T+n52gNN4vdFscVD7dsQGv8JpNxXoZ3bRloQQFSCO9EoJq/G6q2Q9+oTNZt3ytWi87ahjejyfQFA0fSqfE5JZBU+RUqGUdB2XFDrkizaxV20ZDpchnYM8XKjEnte2XTrE090PlyBy49EKrApRVTqZ8f7wX6/HS95zGR7cvxRcRwjm/FIiSUI63fMp4Tde98M+7QcG9+45rN8PMiakszKms+IjZSM5J6rY0IauuR3OgoPHkde1iCb5btN+KITzAK4K074vVWBans4kuHtYixaziljCJ6SUhwC8EsD7pJSvBvC4yZnVooWNKrLmEhL1b1UfvhjYIZ2xRVv49+PE31xwLlvhdC5NCsJX/p8IoY+BSzKqUOTwmSqdSlWc7yT41K88Db91vgkFpk/WPUefIQBc+Ca37NJbduC0P/iCF/pHx60NBfXIHKfwhckWYBcJKbaT6DMhhlWwQzT945S5IZ0RjdcT0mNRIagoS/atxjV37gIA7F/ycxOrdk9CkrYMhsi6KilV72z12SCX9n7vKgswAWMO6dR5pWYtVZjHKJyzQaLWghIUh/o9nIHTZaFptdxZw6ydjxCm8Vy05LPFsIgmfEKIpwP4aQBfKJelFeu3aDFWVJG1jhPmqBzfqj58TedUDmZ90RYS0jmh55svO/vhOOUYv8JpNy2KtuTk8aoJuVN2Nwzp7CZWoZgfesQx3vEutvGPjXHyDThVjjumX7lxOwDg2rt2eZ8ZIll9LjxixZEFZx2frCiFT2iFjypObv4dh8whLF6VzsqiLf54alnqKtuBU1unGOrPKoqAcNeylG5Ip3rvk2ZX4fNCOp3tdh4ghC+jx8MorsH9qNjJXNqvAHBoOUMi7KIt054LBqwf9YHuXiiMm6swO42goeEhjHpep+F6WCshndPYImOabGkxW4glfL8J4M0A/kVKeYMQ4hEA/nNyZrVoYaOKrLlKh8plig15i5mzk8aF4tg5fI2mC+JxD9+CN7/4MbXrddMEfVIcRxB7NN2LUvjskM651PR8C4HL4dP/OkVLvPlozlpMiKQK6azhWlUhnb/36e/gtD/4gl/YxSMrZhsV1kiPQ0zBGb/BOPSY3BgW4eMKqUiwCl8ozLE/IMeXzQlUc3G28+/pdoC9X/55kcSJZU20QkIBYNdBU7iJkmqj0oYv5MqCGMzDgkO9DBvnOk67jun3qrgHKmsRdQ+M6DqTIho/83fb8Mr3XT7yOFQRD2HUa69pEaxJIBS6PWuQzLvVxixEH7SYTkQVbZFSXgLgEgAQQiQAdkop3zRJw1rMNs456Qj8yKOPX5G5XMUp1IevcUgn05ahDhPoSIEvvOmHo9brdop+hOZmKywFBojLqSj68NGQzuqtZBnO5zoqXNEWNqSTPsGXYaXKG3fooi0Cn772Hm07N7Y7RiIEhFb4zDpVJFjP64Ro1s1p96Pzx5NSllU67QMVcvR6Vg5c2E5OWaAPCjgF1xSeISGdzDF095kjbHS7hw7wIZ1qlVCBmjQR1QUxdGEZs+xQb4DF+dRu1zEDPpUJp15dOyYNun9ZLtHlYosmrMRc9v3x9h2L+x4ON3Ym5aqXX5+FByYxmMbv2BTw+RYzitgqnZ8E8AYAGYCrAWwRQvy1lPIvJ2lci9nFZ5kcs0nBJSTKFxw5pJPwyJhm8sUkvh0rhbm0qNr4/ktvA1CQT7XPTRqv59LuP9etaAVRjO3nYKnlgP3UnSMl1DnIcllLrrWSVHMX9tS7irYDeuyKKp2iXJ+Ok2X1d18rhy/3CZFLGutz+Ap7qpRBin6NFKqmUKGq0vosTD5zGRvSKb1rwT3F6hpSoMeE2q9DOgOX5FyaVDvT2oEzKx0sFT763ZiFp+jT1BR6pRA6L9J5nVaoa6zy+hpxJ6ZB4ZsCE8aCabyuZuG3qcV0Ijak8ywp5T4APwHgSwBOR1Gps0WLVUfH8f7GVaVTDKXwTT6HLwTVgP4T2+4u5hd+lc4Yi1QO31xa9OCragUBGBIUUsdoqGBd0Rb1vtppl8GxKPz2AD5Z8FRAt0WCJsrFsZRSWgQkRuFzq3T6RVsc4ibt9V1IWRRtcecOOQJ9psqlNV7pzvQqiKEom867W9Ic0dB5kcQ2cx26OXz+9aPAEeDQd7mbcnaSeRgV8tByofDRIWfBYdW7MAO2joK6B0YAT+SnGVVWjkrkp4Hwhb7ns4ppuqymyZYWs4VYwtcVQnRREL7Plf332suuxVTA5SPj6sNHVb1OXZyhMzew8gpft+MonRBesZnYHL7eINdN3ucYwvfSsx+m3xcVOrkqncpxMeBDOiX7PmSwKbxR8xPkfKwIEnVCPIUvkKimQjql5AlqFeiYtEF96Gl/bhEcbryCuPsKHz9/vyakUy3rM2qlDLx3x8qlIbauzbmkam8BTuFznfWjFrueXe6xczHXSaMqIFIbD/YGWJzr2NfFDNzd3GO6mrjq9odw03a/ou44YF1ngWucC9WdRqgrLK6Z/HBzRNSRmjjWiursVheeBrQKX4thEUv43g/gTgAbAVwqhDgVwJSVOSUAACAASURBVGR+3Vu0aIjlgX2HUwrOfMdO9qhS+D79hqfjl551enD92JDOSeTwxcIlZioMEWhG+FQO33zZl4xT+N57wZPwOy+w2zPE9OHjFDErRyfzSaILU7Sl+sbnFW1hbt6uze6YVFFKyjw2ug9DVel05hg4RMvOWeNJmBDCC9UMK3xkvAo7dUgns5JgbJFkXUX6Ab7SqaQbgW/L4J7O4zcvWHap9YDw92y+k9SoJ2ocovD1Mmycc3P4pt+pmqbKlK/7wFX40XdfNpGxrQdGgZ3N3QtsyhFTWGiUHL7VxrQT71lGS/haDIsowielfI+U8kQp5UtkgbsAPHfCtrVoEQWVb6agiNrmBTtFtYrsPOW0o/GqJ53kjGPex4Z0Ciukc2XBETOTw1f+36AtgyLMtUVboEI6neWM41IX0ukSKG5mTqVh7XI+V/PQKdwhwn34imMnpW2jS9Y4+Apf+PPCvmqFTxVtGSaHr1Lh0+G0ZKUax9QU5jGkjnM4/SflTEinY/9xm+eL8ZiQztADmLqQTo4kHVweYHG+44R0zo5TNUu2DgM3x5dfx36dWjRpyzDDIZ3wvu+ziWm8rqbh9LaYTUQRPiHEEUKIC4UQ15R/70Kh9rVosepYKitKKijCt2VD11peR9rcsE1a5t7NEwzBbry+spTPJXxKlQLIDatBWwYV0hkq2kJ7StFQRTOOtF6BiBy+CNUi1ODbX8+Zh2U79r9+0RaijIrCgRmlSiclScG2DA5B9EyWYHP4oghfhQPWH1Tl8HEhndIi3+a9fwxdx6muaAsAHL1xrrDLalOh7AmFdCbVVToZG43CR4u2BIeYGkxTSOdKIUj4KnpIThPi2jLYr00xDYRPk9bVN2UkTON1NS3RB9NiR4t4xIZ0fhjAfgCvKf/2AfjIpIxq0aIJ3FBN9fR/y0K3cj0XLiG0QjqHyeGL2mJ84KqVnnvKUXjp2Q/Dn7/qCdE25VJi3+E+5sv651wOXzF+qR5Csg67dF6BavUHiM2JC49lr8crZ67iRqHmv+K2nfjSdfeTnDMV0onGffi0KpUIvmhLpcLnj59LyZ7HqJDOCnO5ap7K4RHwGZ9VmZO2ZWCqtZpw2gIcYXMdiI3zxfWX5UyVzsCFPNdJosgaPVYHl8scPhrSCeCbd+/GvXsO1w+2StDO6Bp0vELVYYPf+SlUYqpQZeaoqth0hHTWP7ibBdCQ9WnBFPB5ANNjR4t4xLZrOUNK+Sry/9uEEN+ehEEtWjTFa847Gdffuxf/cPUPAACi5CdbNtiXd11Upkcch2jLQOdY8bYMjhInIDDXSfDeC55EbKo3aqmX4eo7d+EV554IgA8VpZCycDK8HC9GTakt2uJ8zpnL5fAVYY5umKAzjy4qElbo1L8XfHAbAODtL38cgLJoCwQwRA6fWr2TCGRMvzq3tUMWcHbpMu64DN+WQVrryXKsB/YtkUbnviNZEH2zL5r8ucRfVju4ygZ3uw3dTmmXT4DDVTqr2zK4jqiUslD45lOrSmkuJV75visAAHe+46U11q8OmipBB5YHAIBN86vdpa0euTT9OCkBCqn65uHSdHuhbpscDqMqt3WRDysBsw+rb8somEbrpyWEu7iG10YV1vWCWIXvsBBCN1YTQjwTwPQ++myxrjDXSfCm55+p/zc5fF1nzeofJ/fTkdsyrHJIJzd9jEX37V3CoV6G88/aWo7Lb+XmPPl9+HzPhSMlec3n/vq+klSVm+ZtVzFfSBUUKIoB5VJaRURi7FXrKDLiql0ukRrUKHISPrnlxlGwQzrD0MRKAm/57PV4xju+hi/f8ID+nKtgSkM6NYFlQjoN0bLJrh5K+ssW5wqFb2ARvuI19H0s1OgIZ7p87WU5Brksq3TS9aofUkwDmjp+j3/rl/H4t355QtaMF3TfLIUvmMM3W4pSZci696YZpuF61QR89U0ZCaOG104CU0P4VtuAFo0R+6jvDQA+JoQ4ovx/N4Cfm4xJLVo0ByU7JqQzvmgLUN23rxNdtCVqtYnAz+Hz12li3w+dfjQ7LoeqkM46hY8StxhnRQ1nKWH8mtZ/psdfWOHjesgBRuFT6tdCN8FSP2+Uw9dJ7ZDOUFVLLqeRQrVl8JYHFT6ynAup1esZYrh975K1jmDGl7DPsSZ/3DF0uD9X8dN1ZDYowseEdIau47qQTlO+v3g9tFzk/7o5fNSU6+/di3NOPjI86Coh1qleHmT44nX3T9yecSJI+AI7q875lPjCQajf5BinfVh1LOY3adJwH2rNKkbtiTgJTMs1Pi3Es0U8Yqt0fkdKeQ6AswGcLaU8F8DzJmpZixYNQFUo5be5RVvquI7rRNIwziSW8K1iiIOvxPm2NLFvg8rhCxVtKcfiiBTA3/RjG6+rrTh7eYWPITLOIuUs2gqfHe7oK37Fq0iK6yOXReN1VcG0iSLZSRJL7VKbeqSTGM7eVCUf0hjVh6/CTiuk05mXqwpqqZWSJ/jFZ9Jar1jHnpsr+qOuP1vxLN6HQjrn0iQqXE7NdbBXhDlWVem88f4p7UCkSXT1NXjhV27Bb33qOytg0PhATyHdu2BIZ+SxaILlQYZXvO9yfPPu3WMb0/TerFhpRLVyKhxxTcCnwJYR0Cp8YUyJGS0aoFEwv5SS3vl+G8C7x2tOixbDoUMVvoQv2tIUw6h109aHb1ikidDOSUjhU+MrwsKF6bnLuafPFgFznB0+h89e1x0jtEz3+KP2uPlzgbBUAViN1xe6CfYejmvLYEI6i8qa0nGGqtoycA5TLiU6bEN63pbaENFyGVUC60JvAZXDZ+ZWtrrEk5LBMCmE1zBahXTaDwSK1xDhE0LU9HC0Xw/1lMLXCVbpHNTkQK4WYhS+Z77ja1NdeCYEW+Gr/p4DRIkZoxN6y/YD+Nbde/CWz14/vkFLVD6UcF6bYhpCOteKwqcwTeRmmmxpMVuIzeHj0GZrtpgaUHUr3Iev+pJ1Vby6qp7sGKsY0+m2T2B72EXegmmOVDCHr3w11S+duZgn1TyRCBMwDhyx4PbLXcYrfGF1jdqmqnRKKdHPZEOFr3jtpAJ5bvrN6dDUChs4riERUvh4W6LbMujG634+JiBYkqaWZAyRpfbm5kMAfPhvKKSzz1XpDNy5aKuIwBrWuorwbZizB4zJG1ttyAinehbJHuAUZCLLQ9fvNCoxHNS3tjKHb0R1bBqu11k5H3WI+Y6tNKZF4ZsWO1rEYxTC157tFlODLvEAFVdpGtL58CMW8FvnPwpPOe0oAHxhCJdEupimHD6+9H3cWDRnMdyWoXg1RIpX+OqcZ66xdpW9bg4ct94HL70db/zkt+ztGCXSq9LJhC0CKocPOodP9SiMq9JZKnw6pNNeXpVHGGzLwFXpDJzcXl3j9fKnvF9TjIYrcGNUynDRlmvv2o0f7DpczqXs8J4OMFU6uaIthoC7+Nmnn4qTj16sdEToR7k0Ib1pkthtGSjpntI7Hfc9WCuwrgXyvm5XJ5FrNYnDW90rcrQJp+F6GFWlnDZMU2jqtJgyLXa0iEcl4RNC7BdC7GP+9gN4+ArZ2KJFLag6p4jO5oblx4UQ+G/nn4nTj91YjOn4lB/7xafiot98du0YqwVXiePCS2N/ozuWwledw+cqVu5cdRUPqwhYVd8+t6E5xZ9/6XvedlxTd9eeUJsGIQqSkUuJQZ5joRuv8NlFW/xqoV6z9xrCJyV/nYXym6yQzgo7VUgnp7YBkmnLYKsRaj+b5DXaY9nLFueK7++AOc8c4Xv7yx9fkLYI9UTZoDhuJxFOSCe9Zqc1pJP/3q0FWH34UP19oOuP81jokPUxMn41ZnVhIfu1KaapaMusX5x6N1bXDAvTQOiB6TomLeJQSfiklJullFuYv81Syulv5tNiXSO20Iq3XXlXdp3qZz/qOJx45IaR7ZoUYnL4Yp9U0pxIN1TURVjhMwTCXZeCq+Kpc8MqcsnssC+eqCl8/fs79frVffj4/4Uo/qQstjEKXwS5UYTPKdoClESp/P8zv/YMb0yOa0jZtPF6jcJXLqNKIEd8GQ5oVb10zx0Hc1795Z7Cp9syxDdeV5VUQ6DHSJH3Yjy7PJB1zU4n3yPHee25XsFcvdBy53WciFHx46GKthSWfucHe/CZa++x1hiVK01XH77ZxjR+xabg9AKYHuLZIh6jhHS2aDH1eNerz8HWLfONtlFEL7b33rTAC+lkaEHsT3RUDp8K6QwUbeFCzji1hK/Syf9Px7Odd9ZEjf/yd9tw5W0P1doTUqeKhwCFQpdlspHCp1bpqhw+JxRVOWiKsLukxIVEqPE6P3+vJodPLRnoHD4+tNVvWSE1cculIfNVx8Rti2CN5Sl8qi2Df32ElHSVZxmen7yXhnimibAeENnX5HQyvjUiorCoCteuWn8SoXeTUMyUmS9/7+X4nU/zFVSHnXUacvgUxnU67th5EK99/5U4sDwYz4CRMMR1eo7ptBCtKTGjRQO0hK/FmsarnnwSHv/wI+pXJFB+34zxPV+J46u2xI1FCV+oQkYJ7fS7qg0TcmYVWnFCGwETPhUiBnSZHdJZv2MPHex59rjOXCinzyrakuda4etHyD+KCHXSpAzptPdFzdlJ7TYX6nMXUvIhjTFVOrnzrzbrk2PPFa/h2jLQ88TlSIbmclehoa4KXON1tUroYUzROiM4vR/SWS5IE/vrMrAqlobHmwasRb/LqlRLlwdDOv11x4VJhHRGPZQY0qMO5fKuJEyVzvHYcv29e7Htjl24Z/ehsYzHF6Zi1pvCsOmpySecEjNaxKMlfC1aOFDO5GpW3BwGXccJbmK921g+pVVPg851sVyFPIXbMhSv37p7N375Y9dYn//H9x7AP5GQJjd8KkR43M+aPNSuyuHz8unKf4u2DGabRgqfDukUVqETALoIDGCUVJcQeuOVIZ0X/+5zcOFrzgnui0JvUNeHr1hKyavr5ErGFiltAhcK7XW34dahYynMd1IIwTdeD4Z0ChFdECOXNqGnqiGdcxocaA5ub8O1BHopy8B7ClnzeZ7LqIczHIbdrgrVYc+KLA2HaRCkx60+x4SL1+HjV96JD3/9DgDAr3/imzjjf3yxdptpDE2dFgF3WpTGFvFoCV+LqcYJWxbGprTF/jwpojdsDuBqwQ3p5AhryBl2t+3UqHoAacsQcPTdHKP3XXyb9XkuJX7po9dYy7TKVG7L+VpqHlVSn0wRhao2EKGcPlq0pZ9JzHf9HL4DywM2f8YQusQKI1Tj6yqeTHEcPoevIDanHbsRz3n08WaeqBy+8IGixLCqeTpdRy3JclMApTqkU23rLvfH76YC3SSxcxrLt+E+fDXOtGO/LqjjVOnsWQrfFHjQDKYx3GxcsEI6aSuNwPp1+Ywfv+ounH/hJUPZMs6QTvObObYhPYw353A4jFsZG4yB8H3huvvxxevuBwB86frtUdvUPUhYDUwL0ZoOK1o0QVt4pcVU47L//tyRf2ybCnVq/Rnje34fPrZoC7/tXCfB4b4hUK7ix8EoXsUr58TT5UoVM5/7cJtc8wpX8UoJXxOnlysSE5pPQhVsEaXaVHzuKnyHegM8/q1fxq8++xF480sea41hQjoFU7QFJKSTIXwhha889nPknIeKNfQjG69Tp4oL3/QaqoOG5dKQTtaMcpyACij97bqdBJ1UWNeEyeHjxxcQUeoJAMjc7HOS2N/3/oDOGR5vNTGN4WbjQiiHL+zsVl979+05jPv3LA1lyyQUvmHCnscx9kpBnYdxWaIrQY8wYhGCPsRGI847bkwBnwcwRaGlLaLRKnwtphrdNLGc2lEQ+wOl1IN01kI6neIqbNGWCsJH0aRgjQm3cckTrOXzzhycY9KPKNqixjvUMwn8TZ6YWzmDNUVbaEVMIYR2/twqnbsP9QEAn/vOfUF7O0mCTNr95ihRckNy1ecc1Kq0Mqtru7p87cbrYejG68xYLlEFhgvpNNv6xNrdbi5NkCbCIazF+6ocvsqQTvKe7lOaCOv70mcqg04bxh02N02wwjgDy7n1gyGdzPVbh0m0ZVCoGlET+SFJxjQ8oDDnYzzHbhwKH31A1WSbacO0/B5NS2hpi3i0hK/FOkBx5479fVLO5Gr21BsGbnEVVuELbOu2dOgEKnNa45evbl85PZezfKGbOJ/7Yw4I6QBCVTqL14PLROEb8iZY1fS8mEuaNh0A+gNH4SvtVeGQ3MMJ5YB1U1G2H7DVC90LjlX4fJuLHD6h57vk956DM47b6DkCyu7atgzlKw3p9HruMaofdUlji7aYfXBGktI7h900QTdNkOXF2Dfct7c+pBPVTqFLttU+pWVRHoWqfMZpAT32aw3WPln7x+9rXZVO93vXBOMMkVQWVBZtGZHIT0OVznH3RRxLWPUQCt80PlSZFlumSfVsEYeW8LVY81gvIZ1uziFf8IT/kXbVt6gcPmFXleTCIemc8x07pHOJhJAqKOc6lBdIx6MK37C3nsxx5rlQRp3TWaHwLQ8ya7k1pg7pTLw57EqRcQqflPY1feoxG7E41/FsTzXhqz46ao6q/n8hhY+eJ9OWITyXGoIfy143TQTSRGCQ57jklh146Xu+jh+UVfpC381EVPfho9f/d+/da4rAJMI6qPSYTSuhWsshVXaPTYO6/nyhI5IP4eyrhyp1358maEIgZjmk05yP8Sp8o+ybhGx8TI3aOj2YhvMLYLoOSosotISvxbpB7O+kDumcNcbngFMmggqfR/jic/j0zZghS/TVVfh27F/2xuyrip8VN3hO4Rv+6b3ZToiCnLmVHNV+CmF62rk5fFUKnxuyaRO+4um1Ijcu2JBW+ApXkgi4p1tx9n5NHz5FenWunPRVDQnm/IISOEL8o/rw2cvp9hTdMqRz31IRMrvvcPEaUt+LtgxV8xv8wkeuJkVbbIWvx+QNThvqyMMsE8KxV+mUsgxBXu1jUk9cpPPaFJPoG9gU484vHUeVzmHOv6y7sFYB00L4psOKFk3QEr4W6w6/+iOPqPw80Q7+bBM+rthA6F7hVoiMIbtqDa6fntt6AAAWHIXvQY7wlcRJOS3VVToHhpgOefehznw3KcIHqcMkpSSKb5XCp5bb+0jtVWGy1nmRxT66IYVmfn9ZLqXXcyMVYcJdF9Kp9sHqw+cWaJF+bz5JVJM8rw7pfP2zTseLH39CMLenePrub9cpQzrVwwu1L3UhnTfet4/93J1DF20Rdg4fDW+dBgeagyEGvH2jmi2lxMeuvBMPHfC/p5OG9fthPYDhdypWUVptX7lOibTWaWis+kpUPXBZKYy7aIsmfCOMQasKNwW33bfu3o09h3ojWDQ+W1YD00I8W8SjJXwt1hXufMdL8eYXP7ZynVSH8K2ERZNDk1AkT+GLyOFT4EI66b3gwPIAV9y205uDU/jc0J2qxusHexk2znfKZebzvWUBlRhQJauTCotcqLmsHL7yM1fhW+6XCh+Th6dVpNRv5aBCJdNExDdTlz7hSRPBhqMC9Tlo6hj0KgqVcCGXtKcgbSDPtYfopElZUCV+fKBQ3vpZrsdUxz9YUKlc/pL3XIav3viAXtzPcuw93PccfuUcp0k4h28aHGgOdXlSo6pZNz+wH2/57A34zU99e6RxhoH18IgsDyt81QzDNAKPxyRylELX/zjmTYUfQbBaGJa0hmCKtgw/nkRzwl8l8P30h7bh41feNbQ9w2L1VeoCU2JGiwaYKOETQvyoEOJmIcStQog/YD5/gxDiOiHEt4UQXxdCnDVJe1qsTxg/Lu4XSqyRkE6+nDh/DFyiEtV0viKHz3VoLvjgNm9zNqQzU8qe/Urx/QcOYM+hHnqDHJtKwqecpL2H+jjn7V+pt13bad53EoE8lxYJpDl8EHZPPcA4IqqlBRvSWR4LFdLpEr5BVhA+esjVpccXrZFe/dVECI9oqXl7lsLnj2cUvlzvs1+8pj6ks6oReDcVZcsEqdfn9suFIuG5Y2Po8qRf2Vse2K/ff+TyO/Did1/q2UbzJ+mYlCRPb+P14jVk3ah+v1I59zR4gDIuhB4eBclQzbEI5Y5WYRKnPaaYybDz6jY5U3C9jjv3bRwKXxGR0GyEKuJ6uJ9haeDnoU8aU8DnAUyP0tgiHhMjfEKIFMB7AbwYwFkAfoohdJ+UUj5BSvlEAO8EcOGk7GmxftE0MpMW6ZhlNArp7DQnfGqNjHHiuWlcR+TA8sBbR+WRVTXxfuvnbsAr33cFAGDjfKG0qaH3Hh7eOe2mCe7bu4Tn/tXFelkuzX7SY9JNBTqJ0NXjVAEatmiLU4WTVpyTUCqirVqpojkBgc9Tn9OSrFKYYizhtgxZbgoZDDLjVHlqISSjIEqSkycrSXonSawwVL6xu7cZOkmCfmbCSZWNboEihVBrhR37l3H/viVvXnXMEiGsEO7ZyOGrVj1GDblSu70aP4N243WDuhy+XEq85z++j7sfOsSOt+qET49dP3jT+dX1Ow2KtMl9G89442vL0HSbMNEMRSVMGpM8vzsPLOOnP3RVVBj3NFxnLZphkgrfUwHcKqW8XUrZA/APAF5OV5BS0kSLjWgfGrSYIGJ/7JXYNYt879Lfey5ecNZWAHzuUegQuAqfq27+3c+dh/f/zJOtZcJRoey8G38O9wbRYwip6sNX56DdvvMgAJCQTkUE2NWjoMJYdx4weRm0yTk9Ip0kKStI1it8tPE6AC9kNMv9kE61H+HG60xIp5cXV0C1kgD880LJYBXJySWvHKjNqSqoXv/Xa8/BC8trUe27cXjtcQpnjFf4Bnmux1RFfULiOz0sdi+9Ys5+luMxJ2zGm55/JgDS9N5R+CjosTiwPMA/Xv2DqQirqvOpx2Vik5/BcTmBtGZQ3e8KYL4n9+w+jAu/egt+8aNXO59Xb1815jhBFfHwOopkNJtffSemIefU7MN4YK6rURgf/1Cpehu9qTPU6AS0KXSO5gTn/PvL78Tltz6ET2y7e3KTtFg1TJLwnQjgB+T/e8plFoQQvyGEuA2FwvemCdrTYo3jhWdtxTEb57zlomEfvpc84WH4w5c+li3AMe045ZhFnLBlAUAzhc9VplyH+vmP3YoXPe4Ea5k6rlwfPs5Zcs2hhTGAQjVTCt/AIQ8h6JDOIRw6F1wrilxKrSZRQpYmhcKnyNuyVvj8a8Y0XvdDOqUs/k+TxCIcyhaOZEnpO+GJsBW+oq8dyvnCVTop+TQhnX6BFki/pHkRIlUgJ86U2t9Tjl7U6xYhndAbcAVmeGWwCOnUhK8mh49ety6xBoriOokQ+lzoIjCBHErAPgfvvOgm/P5nvouv37qTXXdF4RxvF6MSlmFI7bjCCYPFWQK/4m7I5uGeG2pnHPRr7tyFp/3Zf2D/UnU0wCT86iZEoW6dWx/cj38nearq+p2GYhrub0FTXHfPXqttj8ntHt6m0EOlum3oq16u9m8FNQq3SNokoPYn5iHPFFxmLRpi1Yu2SCnfK6U8A8B/B/CH3DpCiF8RQlwjhLhmx44dK2tgi5nBB372PFz7Ry8YeZxHHLcJr//h6kqe04wXP74gZuedenT0Nl2nSEtMhVKj8BWvdY6G+7lL+DpJ4rV4qLvBu4RvlPA79xgAhaOhltJD0k2FVSilUuEL5P0BJVHKJdKkOOZuD0hud6T0z4+r8NFDneX8cgC4+s5dtdsoO7gwT1q0RSt8eiBh7bsgPfK4AjDcA4oipNMofAOdwxdqy8CHZSqTlvoZhDAKtroG00QEnRzaq1EV57l716HA2iuHkFqqMDbHv0GoQ+j79583P4i7HjoYPU7I9tDXu/ZYkN+od375Zmzft4QbApVc62wYBfQBSXAdaa8bwvkXXorXf+wa/b8ifNMQgjyKHndgeYBXvO9yfPbb9+plKgx+lFMyzPkMhU1X5StPGpM8vWp/Yr7y0/BgoUUzTJLw3QvgZPL/SeWyEP4BwE9wH0gpPyClPE9Ked5xxx03RhNbtFh7eMYjj8Wd73gpznr4Fu+z0BNJl6gEqyAyoD2SQk+wNy90vBuE6+B3UqEd8JieboAJ6VT7NUo4U4ersJmRKp2OwtdNE62eHe7Z7RqsMRzClzlFYTIpSWVYu2AQ9zSXtopQSISwFFS6laUokuV5LvHzH7FD31yb6ZxsWwat2Pm5nKqvIVBW6ST74+Xwga8q20kLFVWtr9aJqafUZ6qOLvVzCGHUVk34RJzCd9zmeQB8waGVRl3lyZHbMpSvjUI6A07gL3zkavzIX17cYBxih/U+pPBVHwsu17Su1yg31cjqSoTyNewMboj9amKUkMfeIMcgl1af1bFU6ZS+PXXjhT42DxhWUOFbAQVXf+cj7v+rf5W1aIpJEr6rAZwphDhdCDEH4HUAPkdXEEKcSf59KYDvT9CeFusU6rerfSAVJnIu4YvJhVMj2dU57WW/+MzTccrRi2zbgN4gtxx3SqCUk10XIkYVvu/eswdfvmF7veEBcA7gIDf5csKxle5TVbU23YdPhXQ6oYZ5LpGmdiuQVIV0BvIw/aItNjmm5ySk8IXIMV9B0yffubQd6dwh6YLM3S1z5EKOkpQFQXNV1k5aqL7K1oHO4QspfGT/2JDODIkwTe6XdUhn+Kk23e9jNhUh49NA+OrCmEd1RtX2TXKZR3ngEuq9Rx9ShQmd3pD9nFY0VTamicDyIMP/9283BsI7uYctAQMiEaN8mfPabDLu93gYSCmxPMjwvfv34cF9S0OOUb4OQQu4/G31HRxlz6T07am7XM3vlT8Wt3ySGNf5HRemIY+5KT6x7S7ccN/e1TZj1dCZ1MBSyoEQ4o0AvgwgBfBhKeUNQoi3A7hGSvk5AG8UQpwPoA9gN4Cfm5Q9LdYvHnPCFnzp+u3YumV+tU1ZdbzgrK341Wc/Au+/9HZrudt4vUlIpxuimMKE7j3siAW88Kyt+H/fuNsjb70sRydNtMpCc+Jche+BgOOxiRRt+fG/ubzW5ipwvQez3JBSSrLcHD6VM8TdBNXhIgY88QAAIABJREFUMVU6bQI2yI3CVxx3qclhqH2BG4DohnRSp8BWzvh16iCZ9WmeYE4UwEwTBXN8jMKn1vdnKAhfgn5myHMnKYq2uG0ZQlU6KRG0i9AUr0v9HJvmDflWIZqdJAle8/T6VuNPBeHTeWnxxH0YrFTRFrobTRW+Oiag1dDcKOydJME/XXsPPvT1OyAB/NHLznK24cdJGh0Rx8yAwm2tM2RLA/WdGLVoy9s+fyP+/oo7ARTFvG750xc3HsNcm83n57YZV5VO99AU5yN8PkPEdZi+juPCJHlWs5DOydkxKfzPf7keQNGPeT1iYoQPAKSUXwTwRWfZW8j7/zbJ+Vu0AIA3Pu+ReNaZx+DJDXLa1io6aYI3v+SxHuFzi43EhHTqoi2MspRrp79o+dDPpF+lc5CjmwiompiFs28TvkxK3L7jAJ73rktYGxZVW4Zaa+vhkl5AKXzFe0qyuqlAmhqFb7lU+LiboAnp5Iu2ZJIWhimWV4d0+gqsX7SFzu/nsrl21EFK6RXdoSQwkzS8k1H4UlHm8PEOb6HwyfIcOIQvM3OHQjq5SqqU6KrjuDzIACGQludakcJEhN0+aqs6ZjsiypZPGsYZ5REsfCL9Kq9V4zfBKOGEQVU6wqZaB5w8mFAPIahCH1vg6uYH9uPtn78RH/mFp2Bxrrn7FFKM2HkbHkr1mz1qpVRF9gC+knIMRlHAOLJo+vCN8kBBer+n9QpfNdFcSbXNVOmc3JymaEtU2ZaJ2dFiMlj1oi0tWkwaaSJaslcDL6Qz5veeyRlR96K/vOhmvaybJuhluUcwlMJn1hM6ZE8TiVzi1gcPBE3Y0FV9+Ea/+XSZONYsNzl8tsKXWEVmlMLHtlHQOUOqaIudW5bnRtFzc/j4xuuAS0+qirYMMn55E+dcSmbfpLnlD5wWCEDhoAyImkIt9p+0F9eDS7q7ZUinIq1a4XMIS8KE3fYHfg7fcj+HgJ3Dlwi7YI4LepzUuZwOha98beiMxn5V1Gox5FBhlCqdtqrHX7Ph8NXqz+mDKBrSaSofhrehePvnb8S2O3bhG3fs8jeIQEjhtkNY7ddYqPM0JEcDAGzfO1wIpwtT1KT59cCpavreMeLPvLt5LIH0thuB0A4LUxV7gpM0GHsGIzrXPVrC16JFC8ylvAMdAy6U8ONX3aXHUWMvD3JsWejg3a99IoBS4UsFzn9sEWbaSRM/pFNKXQGTg1Imx3HzcXsPKju4oi2dxK7SudRXRNUft64P38AilTbh42/uftGWVNg5kpwqVWzJqyh1oMUurLHKRVy+nICAWqz78GlHyR+rP/Bz+NKkaNXhNV53DoDQr2a524cPKHItE1Klc3mQ6/eha54eP/V+18Eeu+5Koq5f26iqhA7vamDTuBQ+i/xFhCHHhrdKOGpRxe9cFQlsQoJj7OTmGvYh1iiku+rhWhPUqc8x29JrwJyzEW1iHjTF2OKut5p5dCtRtCXmge8shnSud0w0pLNFixazAb9oS0xIZwEaQvTtu/fgX0k5bSFMqORSPyueqpcb9gY5OkmCD/3ceQCAS27ZoZ10k8MHHPJ6ahmoqpjjuPfwOXxmZKtPXip0fhlg2jJw4VRG4WNCOsvPFenw2zL440np35CTRDQu2tJI4WNsoarfgeWBN64QJpx0Lk0AQZ05e6xcFiphN03whBOPwFNOKxT5ovE6DenM9dgUnMLXY6p09rMinNEofBm7rWUbc1zH4XPdv/cwTtiyMDx5qJGCmrY28Mc35zEWoxA+ai613VL4arbVxM57oGDG1VEEVlVb7nvmL1P7FxUBUQFGLPc+bHoklb00hLspBiNsa9miXoe4HKpCOkchO7n0z3K82h2+nlYaE+3D1+A7v5I9CFuMB63C16JFi8IhJ4hxaHQIEbkB/Ze/24Z/vOYesw4MmVzu5yXhK7brZ7lFsoocPpvwZVJWEj41duyNt6oUeyiHT0V6UvWokzhVOvsVIZ2yUJQUqVsi+6PaHbgqkwr/ZAmkZIq2CLdoC90H3omrc87pTb9orO6TNLXk3j2HrXX13Lpoi7Bs5kI6+7lENxX4/H99Ft7yY0UBDZXDZwib6ZtHcc7JR5Q2m+WhUFYBU0BneZDra4IjXomwr+9Bg7YQISz1M3zjjl14+p9/Df94zQ+ittm/1McHLr3NztMsX0OnMby8maMWl89Tjj0CX7AVvmaqZZ0DTom66qtYfI/C43Ij6YcZQxZuCT/w8M9r47HL11FCOsdFYOrU52ob4G1LW/8MbROYh1Y19tW2AVlJziPU3JObosnYsxbSOWpu61pAq/C1aNECXS+HL17ho8QhEcK6qe4+1Ne9y5b6hZqinOXlgZ2zpdQcgPR0yyUO94x6RKEqZQLxN580EcFiJVzjdTuk05974BE+f9xMSiQCOGqxKOm/f3mAbiqKQjaymMMQPjN+aDwJXuGzHD1LLWEXW0TmYUcs4H4nf6dLKqhKGWq8bv4/5+QjcWh5gNt3Fg22hTAOVlEF0zhKnOPVH/g5fLotQ+mkD7TCYg7Aey94En74UccWc5Jt+1aVTvsatXL4FOGDj7lOYm07inqi8Ni3XKSP2zfu2I3XPuWU2m3e/vkb8elr78Ejj9+E5z1mK4B6pzrk4ESHdEatZTvFo+Xw8YQv1K7BtsF+9Vcwn9OKj7plT409Cmr/hiX80nnVy6X/vumhjKkAWodRyKJti/3abFufTKmHVqO47LSqsEJ90Ra1Lb/daqhck+QtauiYBxrT0h4iFrNm7yTQKnwtWrTwFb6YkE6maItLFO/dc1iPvTQoQjrVOr0stxQ3qvDlhPiFcvjmUlNKP/a3vErhYxuvB4q2hHL42LYM5RiqhxsAHLepIMEqNy7UeJ1zoPPcr7KYJvEKSZ5L3P3QIa10/NWrz8GZWzd767rXBKfK0XmefMpRZQN4o4Io57qbFi5EyIGCLAiaG1rcVW0Zyg04he3Jpx6FLQtdbznXeL00TB/fXk0OXzdxCJ92qr1Vo2GHKMYNtPdw0SeuRwrRBI9lzfL4kM7yTc1PgR16x1W7jJuQ2mWFWwYeWDizFPPXKHxUqa5zALmP9SU1rMIrbXvMYum9j7023vGlm3D9vXv18RslrHYcDzSAiKqpFTBkkT5okd6yxuOCCc0c4hrQg1V9PgHoFIqJhnSWc8WEdM4YfxrlYdRaQUv4WrRoMVSVTo7wuY7Q9r1L6HbKUMZ+bil8vQFTpTOzHbcsD4d0FqX+i/exN0GuMIsC33g9NwVByF0wSURRpbO0t+9UF6VQCp4ieQC06qmUM60yETWRKmIUEv4Nuapoi72txFe/9wCe966LsePAkt5vbt+p4pnnflsNCfumf9JRGyy7hCD5i6lS+Hj7JGhbBrJf5TF2y+fTc2HPSYu2UGWIrANSpTPLSQ9EeOg6Cp8isGNzumLJl35njNSOcWCbkI2xhMCUaI+1zVaI9i318adfuBHLgzgSQa/1UIhjXV5iXUgnrdJJQ6NjQzpz8jBjGGjCwTw8cd/HXmL/55Lb8Mr3XUFy+EYhfENvaqHuYUTltsz+j6NoC2RYqQtvwhNN015mFIOaYSXaMqxljOlZxkyjDels0aIFk8PXIG+H3IDcrd76Y2fh5gf2A6BFW4y6QklFJ0lwIFfFT8zYhwOEb66TNrIT4PP06PwurD58NKRTiDI81FUk/XEzWSh4R20kCt9mo/Dl0hAdxbtEOUeoaIvrcPpFWwI7KYuWAoNcatWI5hdS0GMlwTwhdf4/+ehFj4gp0tVJ3Bw+/0m7q/gWNhTHWM3dz3LPVpdkKoQUPuEofG4PRIq51CZ8uVYa/HWHQeww7pP3mDDHqj58TYyr+4pxSgwAXPiVW/D3V9yJU4/ZGDWdpfAxBMh9z9kQCmOlJMLk8NXZwzy8GTWkM/TAg+4jsyxq7PJ1FMI3tqIt2oTmtnCN58fRlsF9QBUzXoh85yPs36iYJN9rEqI6LuJ50fX3Y8uGLp5xxrFjGS+EVuFrFb4WLVrAD2eMy+Erww6Jj0A3e+8FT8KZWzd7VTrDIZ1C93NTjkeVwjeXGgoR2yC4SuHjcvhUOCZgk6wksXMOqxQGlSvUTRMctViEHlKFb5D7RVsgRBke6dtZNM529ssp2hIs1Q8TFtgbFOukQYXPXBPfvWcPvnX3Hm8sur8nHbXBcoQFjOqoxuKejAuhFD4/pLNTqr7KSR9k0s9fpCSTLA8RviKHjynawqg23Y59XHX+15gcvfgQNVttiwlzbFrMhZ+xXs0KKXDqOxyr8IXy9kJzcctDvMBU76Q5fP73qG4yrfAN3ZbBtscsp/s+xMBEPR9XHuUoGKWaLUeKx9F4PZdc4/Xq8aTzamwc74OfGJg+fJObVA0dc/8flxlv+L/fxAUf3DaewSowyoOQtYJW4WvRooXnRMf4M1yICXUOF7qFU61z+Po5FrqmAXffCemkIZKZVvjCbRnmOom2IaQCuqjO4eNCOo1TaDVeLwt/6GqiFTkmtCjLsZvmy0I2C+X6NqmkCl+ShEM63d2g+YRqXA5SSu2A04qXHBGm5Ovfv/eg9/lbPnuD9T8X0kl7ENKQTt/xKsgcG9KZS6Pw5bnnbNP/QiGdLsG0FL6qkM6Awjcu3yGa7mm1Tfjb1aheLqJDOvWccesBdssRdVxj88Ji+vDVFW0J75s6byY8OK85iJNR+Ix6JQP7OyzqVM4YDLgQhREwzGgcWdQP1kYQIKVkiFvENq4tdLuVDK8099uVm6sKs0af2iqdrcLXokUL+E/00gZPsO2iLWb5hm7RFN1T+MpfnV7mhHSmwhRtIc7L4T5fpbObJtruqtYNFClD6vT8TEhnVZXOlBBUY68/Lm27oAq30JBO+rkgxM+teKqQS79oS5IIx2GuV/g04RP1OXwx2LzQ9dQ2dW3MqRw+ZZ9TlENKiX7mN17vEmJW2Fyj8AVCOqlzLSA0uaeN1znVxg3pHHsOXyTUbGrfrZDOUJXOhkTQ3775elbfyvI1VBXXt8u8t9Rqya9jbcvYYttoPh+QfFtOMa2aS33fhxT4LDvtBzT++WxSoESQsbNc4sb79uEvLrqpcZGT8bVlUK/Nx+NUtbE0XgdXpbOW8ultue1WI0pwsgpf/NizlkvYhnS2hK9FixaAl3wXV6VTPcGX3jIAWJhThK9YptoymD580iJZc2mii5/E9OGb6yTa7lAlTxccqVPgCM6AVMSkxCJJeIWPJWiENB5bFm45riR+uSxaDijSQckHDelcHmT40nX369Li9UVb+H2UEuhlxbHShC8tyKt/POJuD497+Bb87586t7CbLBcCOkS3IFiismhLL+PbMgDF/gM2AVewSSZV+Pi2DLbCl3ktMSi6aWI9Gab9wMbRADl2CLchcijHjWLUxuuZM2cM7FzJMgTNmfBrNz2Ay2/dWbmtTYBA3ocUPvt7GBpbwiZ/VfsWetgCjBDSSdgMJcKc2Nj06lK2ZVLiNe+/En978W3Rv4sKseQ81pZhRtPnngvpHOE7Vyh80ltWtw273pDnaBSYhxMTJHzOXJXrzhh/ahW+lvC1aNECwKZ5O7p7eCfPLF/oFIRPhQYulWoKddBdhc+EdBav37rbzx0z25rw0FCvPheVVToZgpPnRgWgWyZCIE1J0RbtRPrj5kTBO/3YjThhywIWSvVTotjXjkM6RPleHdsLv3oLfu0T38QVtz0ULtpCSEhVDpTO4SuPdVGAxl83lvCdfdKR+LFzHl7YbV04Qh8P1YfPhNb5hvWz3CsepI4LzQPz1GcnjFRhEAzpFFaVTn3MWcJn922sCpuVUuKK23Y2cshi1zSOmArprCf3dRUra+dU5KYuh48qc0xIJz1+B5cH+MW/vwY//SE/Z8cu2sIz2lBIn3mQ4NtE/7eL3fD7oJcx84wrD4i2hwAASRXvCpuqQMNaTVXShnaNaf+CRKnBtjlzXY1iXfGQxl0W+V0I/L+yIZ0qh29yc5gw7ijKNzlDJoBW4WsJX4sWLQAcv3ne+j8mpFOtEerDt2HODulUuWx2PzvaeD3ReVcxjtVcx4R0jiOHr8t8NsilDkGlN0E3h6+qt1eWm+PyG899JD73X5+p/1ehjIoU0/DRJBHaIdleNkXfvncJmZTefqSOMxAMfZPShEeWr50kpPDFsX56qSQO+VKEWPfhYxxvoCAw/YH08ijV/7T/nHtpJoH5e6EqnTDEvyjaYh97im6aBEMW3XN96fd34oIPbsPfXnKbN04I7nE43Muwb6nvraen1SGd4THMcn7OWNKiyFVtDh/446POBV32L9+6F4DJ77Xn44mdrfBV21DflsFexhFoPWaFwjdUqKITcm0rfNJbr2mBErX2IDN9PZsS1HESWmDUkE6zrSm0M6pd/FxBWwLsW19nq8AhVoJkxvC9WRPMZs3eSaAlfC1atMDxZRERhagqnYxDR502lcNHi38kwlb4qIOvmmwDcU/jVF4YABzux2Xzc4VZzGfxOXxJgrItg01QQ2Fgik8tdFMcv3nBCs0r2lMk1hwCZUhnOZ5SS5cGmVUERkGZXmUHYEInARPyqMJTXcQqfDbJs3P4lOPeIeeKs0+WCp8X0skofG64cUxIJ51OCPOgQcrq8OW5Dt94HfB9PTXKRy6/Uy/73v378LcXhwmgO8b5F16Cs//4K8H1OUtD35RwW4bg8FHbV43nhs4CtsJ364MHABSqcNU4dngnXad6n2LaMpg5wBJobl6FUYiHqyhSWzm1sckcRUGkYoNelptj37Cx3rhCOqXz2gRcfpwp/DO8fcW4AeIWANcigtq2khxCffcnqvA51YAr150xAtWGdLaEr0WLFjBqnEJU43XdlsH8kC4NjNKmnuJTJz5N7AAxSjQ6KSmCEvHjTMNDY0M6q3L4KIk65+TCIR2QqpDUck/hq6jemOXSU0yV3Xle9J9zFT4IVbSl+He+PJYHlwflfvghncX8YTsAABJelc5ORJXOKljtKhzyp3Iylb3KLO9Je4jwOTl8xRxhwkcvrqLyp10ESK1P91dNGVL4qBNM37vOotp8x/5lrUi++K8vw19cdJM3roZzHO7dc5hfzckdqwtHLOwLLY9U+CL9I7pa5hxnwP590O85ghVoTRDTskCHMwYfdPgPQurUJ+7TUaq00k1yGb6WhiVLaogeKUTUlMCNTT0agrTqTZlLJBuBaNNxQ6G+tba42+nPV55ETHJOE0lQ7wCsxr6PgrYtQ0v4WrRoUeIzv/Z0vPTshwGILdpSvFIni5bCX+jaRVuAgigJYZM8815o9YmO+YpzT2TnT4Tx72OLE1Tm8OmCKcBvPOeMwg6Sw2e1ZVBVOh1FjW3LIKV3PNUhkCgcNJW7RtszJOSpvbJ7x/7l4n9HqXRDuNR2LjGUkF4OX0jhqyLH3L4ANvmjIZxFvqUJUaWO5eaFDnIp0c+k34dPKXxEwXWJmR1San92qLwu3JBOur/q2HE+zpxbtIVc31XO4/17DzufVROROujcmvJ/2zHnxwg577G5LGr7OrXfqg7LFHDiFFJeCedtDJE/ClPZkl9PCUSucqj2jCV3FTYOQ4ykc84sha/xaMz45esohG/sRVuGOk72GACNXBjeJslcPdxwN2/f733uX0+j2zMsVkKpilH4Zo0/tTl8LeFr0WJd4rxTj/KWPfnUo3H6MRsBxDZeL8DdgIQA5jt2Hz6gCIWk3IKSwbk0wSDLked2+ewNcynrjAthGN84CB9tfk4LTnBFPVThD1pNFOAdQSn9ypK0iS5V+LyQznL8Q8vF/j2oCJ/bRqM00thRLLdCZtOiSqbblqGTCLZdxVwn5rYfbosgBPCMM47R9tG2DFICpx2ziK/+1rNxytGLyHK+LQNtn2Dms+cPCHwAzHGjOWG0Sqeyzd0PhW4nsRwF+t491/T/e3cfdj7zhgYQr1joUCvyoKBujHD4YzPCV5vDR4az+/AVrxlDmKsqYLrv4/bVn8v63CGE3hzMZnxIZ15pRxXoJq7CN2pIZ7F+sUE/M1EJTUM6x1a0xXltZAMj8elQ2hGocaHw8cRN4fPfuQ8veveluOj6+/U29NUbc2hrhgDz3R839IOliJ/+Uc7FakCd62F7aK4FtI3XW7RYh/jkLz8Nh3oDPPHtX7WWm6f68WN99Mq7vGULnVQ7HW5IJ1W7rKItSYJcQocBKix2UyufTUH1qgPi+/BVFSLRLREAnXOXZTSHzydZOmRQKQhcHz42pLN4LUIZpafwKVKi/JEDZcjqg/uWLVsVFGFUZE45Nt0kwRIUsUsgJQnpLF8TwSt8bsXMGLj5dB/82fOwfd9S+b/99H6uk+DMrZvRTRP0shyD3G+8rq6P2JBO11E5WB43t10AJcL0vLvopoIPSQQflqpwj0f4JFJmhmjCpxU+FdJZrw6FQzrj5oz2/SmJYgo40VzKQYUyEmpEbpO0gAk1yhtXzCiX1dUI+RYrvq2xcMlmXR++Jq69IBVxe1muf2/6DRupjy2HTyt8I4xB3o8lpBP11/SN9+8DANy242C5jSKazlj6elo50mPU/UnOUj7kidH4Zovvaf8h5mH2WkVL+Fq0WIcoKlz6X39FHFQ4ZhWqfjdpTmDXK9pi1us4bRkAuyIjACzOpUgE4FM6c1uKrdLJVaPU8xOlR4ejSdOry91frkonFzaS5UyhERJylRGiYyqCFn9qXJW798D+JctWBXW+1HHgFD73+PZJjzzuuHBFbDi4qh59v3G+gzOO21T+b4d0qhvvXJpowu4Svm6MwhewBSAKnxPSSQlziNAr28KEL6zw3bPHJ3wcGod0NlL4+OXN2zLUrEerdJKxWYUvtx9I2Hb5c7vj14V0hhpic6HEw4Rlqv0bxtd1cxEtwkfXG0Lho5dub5Czxz4G41L4TKrm8MSY2mIUvuHBKnzO/+p46d/XwLlYzcbrE+3D12DomQvpLA1ex3yvJXwtWqxXpInAa847CT/xRJMj96vPPgNZLvEzTz81YoTwL+cCIXlzbtEW8otLHXzToN0mfBvmOuU29h1GtS4AmjRer1L4FNuye4i5jdeV+Wkq0HdCOkOl3F3upMxYLu322jKUIZ3K6VGEb4dS+JwBFeFTSphytFKnsb2E1HmStA/faFU6ffIUgjo6VF3pdgT2HeiXc7oqaqnw0Ry+iiqd7vxG4TPLaJXOYg77vFIUbRmK8yqEcFQZe106hxvSOaqP5uWoSf8zF8EcvkhPbfQqncJbVqXwhXIB6e4NHdLJKIBSmvxc7hjyLVZ84hgLT+GrCykdcuweCensr3aVziGG45q2q+thNLLD5PA5C1TRMPWboPfD2TK0fJIwffimg/DNXkhn8RrXY3Btos3ha9FinUIIgXf+5Dl4xiOP1cs2zKX4nRc+GvOd0RS+BarwpbZDbrVlYJSWgRvSWSp8LmjRFlcVDKGK8CkFTMBWJ2gzdPqqFD4pJXEo/XFz6Yd0qlGUcqWOkSCk0grpLJWq/YEqnYpgK7KsDmHXyuFL2By+NFClcz66Sid5H1D79P+asJjj2q1Q+LQqmVGFzx646jo8VBI+6igKuFU6bUJPQXtI0ld3TMA4YotzKe7Zfcj6rI6I1MGkNfmOb9NqnLFzqkMeS+KLbajC5xO+yvYl5Csc4HsVxW/87ayxORIhybXDEa6K4zQOnzukFptwyPhJqPM9StEWLkLh/33jbkgpcag3iM8JHEkJtYYAMB4imkvfIF/hM7+JhQ2h79DKK3zqWp2kslbXz9Jad7b4HgnpXGVDVhEt4WvRosVQCP1uPmrrJlzw1FPMekJo4pEIpy1Daod7AkB/YN9JNsylrNMpIPRN0CV8773gSaxtVX34FCmjpJT24XND/zpJEe7nOm7Lg0wTDTWG+1RR3XSWSoVvXit80HNwIZ2urQpK4VvqZ7jxvn14yXsu8/a32xG6KihgEz6+SmfcndEtZsO9d5FLczwrCV/ikwa/aEt4zoPLdogrUITNdhjCx1nbLQvXKGehOoevWLB1ywJ2HexZn1X1RYyBS2gsha8hsYtvy1CsV/dAPNSYXh1irq1FrcLHECAgfLx0yGawaEs5f8bPwaHKsR5V4XN/NyxS23hke+x+ZghfxiUVV4A7fm/+5+vwhevux1lv+TJ+5ePXxo1jJLDG4BrPj6ctQ32VTvU9d9vI+ERRjTm8PcNiJRS+mBlmjO+Z37OoGqRrE21IZ4sWY8bbX/44L6RrPeEV556E1//wI6xl3TRBP8uQJrZi0GUc717GKXwM4RPGwXe3edHjtrK2VeXw0dA+TqkyFTQLKKegbzmRwJ/82424fcdBfPKXn1Yu8xulq/0xCp8b0qn68AUIXyCHb6mf49+++wO9vEv2t1sWbXEbr4cUvm6kwmeHVJrlvqYpCHGhCp/QBNktFMO1hqhSnNxP1LhWqKGr8DmhuhTKHuU3U/XZz+FTNhfnLUaFi/bdiDJK/q3EqCGdsQqTG6qooI4np/BxY4fGCa1jLXe2c1fTBU0GubWMy4k0c4X3fzjlihDfLFy0ZZg56Lq9QY6N88X7cRVt2Xu4CLn+2k0PRtrjk7ZYcGRKfe9GITuS2d497uqcqPtEmADFK2HjghajJ6rwxU+ykvs+DqiHGes4orNV+Fq0GDd+9umn4c0veexqmzFxhBQcrmG3IjSqPL+CrfAVr65at6EbassQDukMkYLqHD5f4aNjubl8KVNkJs8l7tuzpNsnAHyVTvXvkpfDZz5PhdBE44BD+FylcoMq2tLPLCem44R0AqQP36Ba4XPVtp988kk462FbvPUoywqFd6r/pSz6j1HV01L4nFYQXFXVqhu2l8PHFG2BcEKJK0I6FeFTDif1n0NFW9KkaH9Bc1HD5CGSVHk5fJQs8NtQ3/1hRyzgo7/4VG95Fcx6dSGdlNCZ5bo1ACHJbt9Kez6eIDchgiF+o44XrfZa5PCFQ/eqDtOoVTp7WR7MBzXnuMngZOyBqdI5aEj4Qgpp02IuppqO842RAAAgAElEQVRpo81K+A8FRhvPbOtuH8rh8/qXekSRWroy0Dl8E4zprEpN8FeemBkTgfq+recqnS3ha9GixVAI/WxyhG+eFCWxFD7i0Ksb2tLALsBSFdKplnuEL0DsYhuv0/V0Xp2Z2Fp/OTP25rIgVPSmnOem+qZrhyIkpg+fmUuFdA6yHMuD3DquvsKncvgyKw/HantR9uFbdhqvp4nwisAAwJxDthLBky0aIuO2ZbDXK3yEc//kq7js+zutHD4Fl2Ry56vqHHpVOnUOH90PXuFzzytgHkgoh5OGyIWcR6XM7l/u68+MasE7jnVwqx7SzWKKttDquOPuw0enp8fHhEX7nzcK6aRT1RC6OkJoK3zVx6HqODWMlCzGI+8HeR7uw8cQnvqxzbp90krGzYeuQ0jha8oxpPPaaFuGTOn+h0OMZ8b1W6+7+6VJgVu0xVlvNUI61ddwJapj0mvvqtsfwh/+63X+OmNgfCvZ1iKL/T1bw2gJX4sWLYZC6IdzniMPHaPwUfLD5VItORU3F+c6/FyEgCwP8qgf8hiFrwintKYpXss36jMV9uOGiS0PMq9Bt0tYH3HsJnRTgW/evRsADelUcwlNHBQpPHJDN7gfNIePOgSKUHdI43Olcqg+fLFVOkNEyzpWVj6dvV6h8JnwMJrDZ/bLbcswfEhnNxU42OPbMghC+tyiLW6zeoDm8JnxfSdQ6u0lgP1LA+8zjyQG94SH7vcYofC5zhRXRKV6rtJBqlmPjsbl8FESOKhsvE7GbKrwOTb7YyuFj6rx/vbWmBWHaTgiQ0jZQDqkVmKpn+H7D+wfSj1S28x3krJKZ/F/Y4VvxDBgY09z0mpsUGOU/+cy+MCkkU3gvn/2Ai+HLxAiPAwpHxcmmsOnQ1XNsktu2YFPbLvbt2OIhx4uml5Xo0DZ2yp8LVq0aNEQod9NTuHThM+t0smEdC47bRlCOXxFAZhS4ctybJyrT0muKtqiPkuc1hHKLtoyAeBz+GSp8Fl5S0wO34a5FI8/8QhcfutOAFxbBlOlUzVd30IIn3s85pXCN8gtJ4SSGgGh7SvsNg3Z2Rw+h2ypQjIuQpU53VWFEJaDZfrwmTXnnJBO7nxVFYOhHy3OdXBomWu8Xry6hE9tSvdbKdM6pLMih0/9m5Y5fAcYwudWQYzPk3OIUoAQUVCHLCG5s01DOuv8Izo/xy+oalTVv4weC7ugSb3BJhSNH18NR0M6c6L4cPaMvWgLed/PcyvPO5fA//jn6/CC/3Up9hzq+xvUQNmjrlf1f2OFL0AQmxO+8rXRVmpb+1q32lcMMR61yd3ePTzqWvXaMrhEcYT9GxbGlgnOyuxXb5Dz39cxTMdVhZ0U2iqdLeFr0aLFkAhVu+JDOgsFKkmEp8IomEImTkhnl2/LoMIegeKmRJu9h8AVAVFQip2ATajc/nuJQxjcMLFl5waZ577CBwBPPe1o7WDMuUVbyhC8XEqtxG0k+xfK4VvuZ5YT0yHjKoVPF21Rzk0qmLYRftGWlBBsCjuk0/rAWc92nNSUVQofpzxW3bApGdw4l+rcQGteZ2xP4bNUZ7doi01eaJ8zrfAlRe4lzbs0IaEO4QvvioXcccTodqExbFVz+JDOJutZIa96mfm8OoePH1NKENt5GxRxC3UNUPtMv6tcThc3Jj9eeLuYbW7fcRC/+alvW/ZddftDAMx1M4wrPF/+DqjjPI62DIBfFKsOVUS6flv71frOjMAPivY57vfPUfhKwqt/RiS/nnmwsHKExRDhCc7hzAWYB4N+OProhoxDJYyfS53b9cv4WsLXokWLscKttAjYCh/9waUOvnK43cbri3Mp+yNNq2n2BrkmPVWoyv+y2zLY8xSvwnrtsIQPUQofADzy+E36vcnho/tW9PlTTts82T+32qgK6Tzcs8NJTX+/gugMMtMz0ArpZJQ0LoePgx3+Ktj35QJnuzKks+MragocQa/M4SPvF+c7mvC5+Wx0HJfQ2/MXC3WVQHJe79l9GGe95SJcd8/ecg5jX9G3zFaTXDuAeIfYdcTskEd7kLseOohnvuNr2L5vSS9LhMlLaqzw1RZtMbhvj5lTKzSxffisMGj7s1TbzhvvKnyhz91z4hJp2x52qHK8EZhMzUd1RHvngWU84a1fZj+bc/pGxoR0XnHrTn1dh5S8pg3cq67POrgqbVO1N2gT/O+b+7/bNiREXFdD4ctrrvFxgPt9MYSPt2cUrKjCp4u2rNiUU4eW8LVo0WI4NAjp1EVbEptMUaKhfHsuh4/7kX7iyUdaIZ3cvC6qcvhU4ROVP6fgFm1xQwJ7tGhLXih89EaW5XzewOYFE6I5R6qYqrlUtUd1o6JkyN2PblqEZS4NMssh6BDVEkJY6mltWwaHuIcK4VCmRPkZR6Do7V2tS1tzLDiknbOrOqTTVvgOqrYMjMRnFD7bXiucVxENovCpj+/edQj9TOLOhw4W62iFL0EubUeVIz+N4Dhi1PF1faaPXXkX7t1zGJ/99n16mSAPMWKdrNgy5tShv+j67Xo7zmHXOXwMfwj23pOmomvIcm4u6/NyS5pXKVFDRio+G+YsViuGZh/VA6SQbVfc9hD2O1V71ZoqtFsd5zqidtP2fbjgQ9twZakujovwxeSYBqGucR2WGn4Q0GhYyeTiOQuUQi2lxE3b9+HBfcuowgqmoK2swkeOVG8QekgzuiErmcNnirasX8bXEr4WLVoMhdDPpqvS0GVeHz6m8boqrPCorYUCttBNPML0z7/+DPz8M07TpCHLZVSjcI5AvOCsrXjnq87GyUcvFvslbCfXz+Er0NFtGWzHftnJo5NSsoR184LJOfT68AnokE5D+KjC5w+40Emw1LfDSd2G97RoRV3jdY/whXL46HurSqe7nrB8BK5oi0v4+KItvg3cZ0UOX6nw0XYBUETPJdn+wGp65SzkudS9DVVvxMM6bLR8glzm8HFl913/JtbdUevlzDgx4WbF9eSrZHsP9/HLH7sGO/b7jm1slU413DPOOAbb9y3hhvv2FdszYYXD9OGTqA/p9BVU95gUr/uW+tY2eu2aENPQfOMCHU59R0MzcKdDba9+I9RxrnOoVZ7pASbXlcKtglwHGXjfZFtW4RuF8KE+pJOGHP/ouy/DNXftLuet/45NGhwZG/sczO+LCuf1frvGYIZdzXqyx1Kdq/VL91rC16JFiyERelLG5dLNh4q20H5oOqSzcKD/4lVn44a3vchT3ADgMSdsLoqIkJ/vGIWPIxDHbprHa55ysr6BCdik1A35U/utq3Rmdkjn8iDzwtg4gkYJn9eHD0KHdKqxlAJZzO2Pt2Euxaev+QG+cccuvUwrfCWJpQVxVLGZRPAN6bkqndwZTwIkzz1nRQ4hJSF+SCfdR4Av2lJZpZN8tHG+o5WQQkFR25djezl8/njquKgn/4NcapsU4VOtH9Qp75TFdtxKrYDv1IQKlbhwQxar+vBx4Wb0mqbb/sM37sZXb3wA77/kNm/O2JBOhROP3AAA2F+SKrW9ncMX25bBLJfShFvXEa3Q52q5XTm12mGnn7nXxlARneU2XG9JCbvicNUcVde/W2SoX+NEDxxiGAoBbdrA3S4q1GxbfY1DPTCgeaHDkwJO4XMPj/meOtsyY6009Pd6JRQ+MkdfP4DgH6KMAiv/t3z/R/96Pf7vVXeNPrgDdZtexwJfS/hatGgxHEK/m1wu3RwJ6aQ/uJQcujl8C90UG+cLUuT+SHNOOkfmXLgKEh1b3dDcPm0uUXAJg1u0hcvh48IhN80Twpc6ffhE4eRKaRyeOoVvvpNi39IA9+4x1f+syqOwC+L0srwgcYJX+FylNnSjtNVQ/7jp/2E7ElwfPvf8NC7aQq7KLRs62HfYkA91XSq71LFJ9TFnFL5ymXIWcim1vQdK9fBQ384TLI6BtJ9eK/JT4a1Vhc25/fesHKnQNtaxNg9N6DSKANHwYmMzeQJSAbWaybOzt+cUvtqiLU7eVl0pdU2IQ4ew/Hw/UfiKIh7Wx0F73FzSUap0cjnOOXkgob6jIXJTdf0bwqcIXLUyp85HX6s4AYVvlJDORlv6xMZqnzGKwie5hyOOwhdoG+JtV742uQ62713C7//Td7yiZLHQIZ0TVMK4fpahHL5xKI126kPx/pJbdmAbeWg5LmTW7/P6RH0d8xYtWrRoAF7hK5alTljgglWIpHhVN0Sq2Lk/0ikhRgoxIZ2uggQYf3br5gW88twT8fPPPI3tLafIhFH4bMJXVGcsQjqpU5fnkq2CaeXwOQpfIop+hf3M9OuaJ7Zz+8rdxxQ5EaXdS307pNNtTaCQJsIjfKkb66rmDdjgNV63IzrZtgwLHYfwNezDR6c8YkOXED6JDd2iaqfbViOpVPiKhQOi8HUdhe+wUwlUK3wc0fEUPvO+qqKicYL9deqcWKAM6SwPJXXmVCjfplJt3n2why0burrwTAw00XUKq6itKelwi2Jw47jvpSQPZRybfv0T1+KoxTlTpbNG4dt3mFZOlZDqcmM2o4s6qQCp9zKkwldsNNdJdH9IOp66LpUKH5qj6vJXhZ1Utcm6kE7lzGuFL5TDN0pIZ8NjZcJzUdrkV34dBlzvPHd3Qw8kwmHT8fO/7fM34EvXb8dzHn08XvKEh8VvqG1QczfetPEcFL3Aw4DxhHSa9/TYZxMo36l+e9cz4WsVvhYtWgyF0O8mq/CRoiT0B5euq4iUIiVzTI8+BUNQmoV0UoI571TGTBKBC1/7RJx90pFOHz6bXBplqiR8ZdGWNBH65pg5TkV8SKeO6UQiBDIp9VPnhRqFb/fBnreMVqJ0Fb7+INdE1CWQrznvZE+VVGO4oOslNuOzoPoAmnWLV6vvnRvSySp84Rs2/eyIDV3sXx5gkOXIpTn3btEd7uGBO7/OicqMwqcI30MHe9h9sKf3LU25HD6ekFBHskqNcVUPW+FznVG13KBoy+CHdBqFr4OlfoZnv/M/8a/furfcZ7VtNdRo6jgaFdJWj4oxfacbzvp0H9R4rnqo8MXrtuMT2+6uJMTUJjuHr1qh4fpZmvGGV/i4SARKausUoKqiE67CVxeK6YZyhgjiaApfs2Olrx8wNo3AMviQSHu8zCGboWmHCa9MnN+SpghV+p0E6IOpXiDEeBx20N/Dgf5tmEwxFzXmOuZ7LeFr0aLFcAjl9rAKX5dW6SSKDlX4FOErHR7qGLlOjvrfDek85ehFvOa8k4I2U4VPEwBmP7i8NGODUsXKHL7yhthNE632uOoOd5OhChotrqJmSIStFNkKn//T7aoGdFy3AiBQOINuHtvJR2/A777wUXjLy87ynNxglU4Ci++5hM/7vySbTKNzBY7wqevrwz9/Ht70vEfaY5L3R5aN6vctDZBLaVVhLcZ2irY4RXkK24r/lOOcSZPDp/IDP7ntbpz7J1+1c/hy6ZF++qpg5coQ59wvLqG2952+YKgV+YAWbbFDOgsCtHGug92Heti/PMDOA8vWHHVV7TTRdRQ+9Ur3q7IPH7HLV/iEfs/aUL6GHEU+h4+GdHIE1Lx3r8OmAoSUEu/7zyJPknswJSH1daceeIVc3pgcPkPkqg11+/VNpi1Do00p47Nsox8NA2dYAE0UPn60JqTHbfHSFHUPNf7yyzfhdR+4cqixzSTWC4D6cN9RwBVtkVIG+2mOAlOlc/xjzwrakM4WLVoMhdAPpxuWBxCFT9htGSgBc9syxCh2dk8/gUt//7mV61PbXIWPgtro5u6Fcvi6qcDew361u1zyIZ3UdrWvx26a058lorgJ6j585HhU9aKjUKQmKaMxaZXOXmaa1SsSs9jt4I3PO7OwwRkroiuD04fPB3UZlONCi1i45ILbz8XS5uc9Ziue95itQVuOWCwI397D/ZLwBRQ+0grDhXrooBznAanSqYq1KKhzrnIvubYM1SGddvEfWtvD7Y9lHUfHZs4vo20ZuJBOwIQ75o5jWRfaqT51FUQ1To9R+DheYV8b9ie6SmfA5TcKKre9IWj7lvrYNN/BgeWBVcSD20U6l1vUqKnru+2OXfjw5XcA4HP4KKnVCl9gkqpvvgqdpyHIVVDKnlo/qPANmu3xKNzADQnOGFIwDLjr2bXT5PC527o2wrIxBuq3uK43opQS7/mPW/HKJ52oK0fTuUKH4J7dh6387WHA9R3sRx6TYWAXdDLztCGdk0Gr8LVo0WIohH42OSXILtpSHdK5TAhU7VxU4WsY0mkUPm7ccEinSxh65Q2xkyaskhOq0kmhSMQpx2wEAOw51CsVPqMU1RVt4dCx2jIIzwE0Cl+Y/Oo5RagtAz1WZLmryoLPWeMcYDqGq65snAs/p6S2HLGBEL5ceudbhdS6hIXCPJU3hM0t2qKgDm1atmXgctKqQpWoI/j/s/fm8ZIc1ZnoF5lVd+td3a1dQhLaEItYxC6BNYhVtjH+eQWPzTO2x9iMAW8PD9jjARtj+w14Gc/zBs/2Ay94mcFmNQLMYptFBiFk0I5QS2qpW0uv995aMmP+iDwRJ05EZGZV3dt9u8lPv9atqsyMjMzKzIoT33e+U+c2eXQwxqv+5AvuQ7FubDCaqTDHDnCMV6m1lTvKHKqmcZ22x13tnxncmOOKBXxhq6kcPsBdS6nTR6s3BQSHV8fYWn3vpdZ2w9jp9k1bBMM34WiXM2RRhk+765IG2EnTlprHHKkAYvlvMdjAkEk6Y8+WmSSdEwYG9ruM3DOzxBixgEl+j2nJcWKiZoIOSXl4Cl9/8Cjecd2t+PF3/5vYZ/reMZ9PzjxLyHMPMFVIEPTOHvHF7nnzmzdz0wE605Yu4OvQocO0mOC5yc1D+HiCyz+JBRuMQklnal9eWYYWLp18MCMlfl67kcCF9kU/GJLh44NCye40ySFp+bnVjO6eh5erem7OgGGhwbQlBnveFaLnkPZL7Xl19FS4bjw4dq9rUviCBukcNbmrygHo0nzIIMf6wgM+rfn3bZa/6srzAQB7q1nxaA5f1bcRY/goiD7C8sEANyDr5QoafgBH45q6HL6RV94jPsDUGrjroaO488GjbF3Z6zCI4ZJO/jmvv2YNbkRQ1sjw0UBKmrZoOi63vRtQp9vh69G6ir+J4Oiwvo4c/5zMkkwOX3T1YFdBEDThgDRVPuZN1z6mak4HN0zqtLfJ4SM0sUl0noeFcRYudLye6aSmLX6QNtnJsrEU3TPiWpgWMfluwPBR2ZCGoMyyhRMcW07y8Ia2acJTfncuGItvZyTKs0VKkl0F0pLOtWD4eJvHiuH75g33uoCvQ4cOxwB8dtPL4WOslSzLwAO41EN6UpdOvu/5iPQ0tp5k9mhJ4NIp6muV7AesZXzmAr5HViKSznqG74M/dRUuPX2L95kN5ICorLRNLTqCKbxeHxw3lWXgoAFNEzMrZ2TrGD7p0gk4SSexydTe8y87DW9+6WPxumsuju4HcCzzuHBlFpxpi2T4zPIYwzepS2coIXMDTMkQBPl+kYFtxiSdfHti+LQGY/iq9WrYL29/1V9XK6/qMwsmCK4On9/oPY8sew6y0qfDBKzpOOuRo6Pg2Dj4/rYuOoavbuDOtwnu7RlGu3yC4/xdm6r2wvsjtYc2kk5CW0nnb3zY5H+VZSLgO4YMX+DyugYMX0rGKe+dlOQ4JemcRGFqfwOb8iqtYsT/HlwgnNhpwwRGG0g5N+DuX9n0GsR7Xq6eezaGz7i1gG2y4bf4377xMP7zX3xp3QvBHw90AV+HDh2mQtuCzABjSsrSDxC4wyMry9DL1MQmIW0knbH8wZjsyZd00r58FoxyMijnpp+o1VWU8Ry+GCjgI2kVDxz47H1sUHbZmVvxgsee7n3Wyx2LGStJIdlKP+hRYt14n728Pc/sRkg6xfaO4WtgP8XiOobPl3SafMiDy0PfpZP19QefeR7OqwbdMdi8m7K0A09ZloHgTFsyz2zHLKuuhYDhc6hl+Ng+5CA+GIhp/y/gTIBk24eZjPOQyD+NybticJJOOrP+oDnm0snbHIwLXPnrn8Dr33sDa9NniBTMZENTnTgX7Mb7CABbK4ZPaz/M+8o9B/GNh44ihlDSGV0tCX7t++7DxLrqYDIlNbCvY/ikXLStaUupgT0Pr1QMdvicmDTgm0WG6a5fen6ysgxTBto1MZKHlKmQ3D5W4qEJrsRL/TZ0LUvlQ5NLp5xkmgYyVxhIM3xr4tLJzgUPtteB4LPtN0k6P3vnw/iHL9+HldF09RI3MrqAr0OHDlNhEim8m90MBzYEzvDJH7vUNp7xSQtJJw8iKQCIyZ547Eb9coGf+dvvmReDBMNnLb7LZkkngSSuW+Z7doBLAwS/ZmG8vSXhkNq3tebijKYtPh5h+OQpT+0z5cwZMnz+BxRoN0k6Ny/4jF4dw5eSdBaam7bUX3/eJIItvaEDCerhIOBzAwrNvjezrPorBzKc4SvC9e1qbCBWxxKaJsPrWSk3gULrD8aFvXZLDa9moelrW4bPH0g59iPcPlaHz+assc9idfhUi76k6/C511tsDp//+c/9zZfxW9fd5rZhC6Ur7ixFp/nEFF1ra8UlSEnnqNQ4vDrC737stihrwoOpcWlknbEJmMGEkk6frZ445Ku2oz5OuHmyReqPex0EMYlrPvi+ExMLdWibw0cBlvw9ayPpnJWUspJrdmTDcTzgW4uLNi7p1FM7mbbZV9NPcaqEzsmALuDr0KHDVODPzf/ykktr16XAYlzq5APXBXxFMOhISjr5PqaUdMYMCaKSTvgBAQ38ieGTg0L6zSp0muH7h9dcid/5/id5n33gp67ER17/HFaWoaz62+zSKQO+nJm2xBg+V4cv8w82AqXinK5fszC5eQ3DV/8ztHvLvPdeHmOqL3O9DIv9vMrh00EOX1P/eN/GRWkHADEGBOBMF81Sh4PeuoLOvktniuHTjQyfk2W5z0gSCbiBzI/9mTOFKMu1MG3xGcS6IvF8Wezr8CSdcAxl04A2zTq4z10On/YC6cG49Grg8Zbk/TbLwNqTqjOGT7JxqV3UMSsy4CsKjd/48C347x+9FR+6aW+wPs+vHFWTGrGSLxNLOmdg+Nx1Z154hdenPO9+jUd/MoHDlqdoYPimknTSs6RlwEcTimb/8f6HfZotSIkdl71GWkwsTYqUk/G6mLa0ZPjoUu8knR06dOhQgQYrl5+zHS994lm16/IaRKkHLo2DVscF5mry6zh4W1NLOmMMnxfwCYavGqKShHMwCl1FAV7EN+3S+fizt+HbLz/T++yxZ27DmdsXkavKsaz6AeJ1+FLtLfQlw+cKustlvB0a49UFbCnlpS8CDQPlFGgg0STp3L3ZD/g2zde4dIqmNi/0cGRQoNTOETYlRY66dLIcPjLP6SdOkqtHl4G7qwIsL66GnfPq8CWYQB1tI94m/zRTiuXYmSWfvHU/659mjp3w1ms2bTF/ZXH01HhJKX9Zk2OnYfgMxdc0yEwN0vjHxKLLoLLU2mNgebckez9p5MG3nmMDeSuC1WGR9KQMsWbfcvJkVJZYrupzrkTqdPJraVRNagTHiuYC7kG72uUCThqkSUlyLCiYFDrxOuXS2VSHL1brsgltGT6b28y+S76b9HWxFjl8bgKEMCycCsBfd7Z9AaLwunWnXR/TFleHryHgs1Liky/g6+rwdejQYSrYOmpZ3K6fI2c1iFIBHz2IV0cl5iTD14KVaWKKAF/SSQxfbPY6VoeP9kXvaQZ21Uo6EzkXZfOPTLSvVJah+vHjAVtsFh4I2S8avCkVBoOAG6T3WGBIkD3OEt9z7FyZ7etZWvpBbZLi7to8AcMn3vczhaIsUWptv+8219Kvfefj8cChVXtNjUrH8KWuM5fDpwLpZZscPs/Vky153w33WldOKRWNIZZfpBCWNti1eQ5POHs7Pn7zPiPpDBi+cPAXA5ey8v2mBuf9PBP5iv5ypeIuj5lCMPKmmnqyLRkU8b7kdB2znCczWI6X0qBt/Paih9YK0Rw+tJdN1o2FZT/HhcZ8Lx148WtpbBm+WMA3KcNnnj3jcnLnSPq+bVmPNTFt4a/T7aWMilLvJ+mPzeFrOJexHD4vYE2cT61nz+GTkzxa63V16Ywxl1qvjYxXoq1LJ63XSTo7dOjQoQINynq5apRJUOAxKnRywM3LMki2LsXKeAHfhJJOYvhig5mY+Yhj+vwgicpIBMYO7IejRSwa7UNZApMUXg8DPlcOYz4m6ZzApdNIOsMV/FIOaYZPvm+bwyclnfUMny+7zXOFcaGhtQlY5/KslTz4+592Ll53zcWOmS5cXkmMAQHcgIXOJbdft46tNezcqIwHQa/9S2dmUpY6mP3WGrjuqw/gGW/9GAbjImq2wl06aZ+HVsfYvuTkjda0RbAcbYc99DW6/cfX61cBsQ0MxYr9LPNn1zXl8IWmLalJBAm+Wa6UlYfyz8vS9fn2fYdxw54DXp/89iYcDLKOzkVy+MpSR59Dsf3UDeoDlUHp1AWxASwPPqg0w5qYtmgnDZ30VElJcsEnQqYcg3uMMV8g2qN9BSy6WLHtZAiHrOmZQiyHzw+M4tsZhnrGgE+U3DMBu/+Z7dOaSDrda57Dty4MX9VkXR1L2j+wPsYxxxtdwNehQ4epQA/oXpY1Bnw0ECmESyeHzeEbl60MWAA/EJxc0kkMX/jDFWX43E4BuB/kpGlL6QYPbV06OUwumPsR56YrqXzFxb4fDFEQrJSyJTDyTAXlGFz5hnjwBlQBeWS3SdOWYL34+aHvLXWKJmL4xHfVyzI7Y54rhf/03Avw/MtOS2wbk3S6SYEmRpLGWrGZ/JTMMcXwpQb1GqEEWUPjl//h33H/oVU8cHBgB4fewFK5+6soNVZHBYbj0hrbaK2DsgxS2pkCLZYuoKmgiL7vWGAKmGeFHKArZf7JJts6B3rBb2aC31J7FdmMa2e13jVv/5QneZWDxFID9x1YwXlv+ABuvOcAJgGf4LCSTrAC1wyxw6kb00vmf1SUVoB2LEUAACAASURBVNUgg5g//NQdeM/n7vY+G47L6LNlUtMWHmhOGha4UhlVW961MHuQ4QX54gSP2TPb3yj+dhJGjVQuTVLBWJmfJgk0rTMrKSUNmzwmvpT32mz7AlIunetTlsFKOhs4PlrvdX/1JbzrM19f834cT3QBX4cOHaYCZzyaAhoaiIzKOkmn+Tscx1w64+1OLOn0TFvSDF88h8//ayWdo4Rpi3Y/km1dOmUffJdOJgWbkOHLlM/w0TFYho9JPwmBpLPFIcTOWwq2zAH1IbF+wPDVuHRSC650hrIDqEwBP/OCS/D0C3bGt5URPVgdvjJ06QSAF7EyGFobQyI6T6NIAFeXw+cHiOlBnWyj1Nzt0Q2J+XqZ4oXRXf297VXpilKHLp0xi/YYaI9hDl8i4KvOnzSHIfTyTBStN2UZMqWSEjz7PjVQZB/3Mirx4OeLlTrd59ClE/inW0xA+OciaGqCz/A5uWVbFq2OXQwknWyySR7bWz94M+4/tOp9ZgyzwufoYNQ+4KNnnsvhm27wbhk+LumcMg5ISjpFe/SbFjJ8ssHJ+9OW4aPg2pd0Np8DjelzHAmS1R+Na9pbA8ljtFapXptgMthXScdW3zit99k7H8ab3//Vte/IcUQX8HXo0GEq2AKxWQZVPUlkHSgCL8uQNm2ZjK0DhGlLg/kHrU+7qS3L4DlP+sEQZ48Al8Mn989zgpoY0BiUUihK90NYVyieIGWbfe7S2SOzCm0Zi55g+Hg/Lzl9Cx5zxlb7PhVk8plfvkbI8PnvpTlBSqb6xHO2e+/buHRSS3mmLMPXFHRHTVts7qlj+PjM+2VnbsVrrr7Q1kzkxeljdfXqzCBGLQa2sRw+rd2stWZBDL+uTcDk2qD6e9tYEfJDq0LSSW51LRm+0KUzvv6cDfji7ffzLM7wRdaVsisrB4T83H2SVwyfzHnyGT8fQRFs7ULrVrc2a9h36TR/izKemxnrT91gWCn/PuKuyG1Yk9VxGZUsx5yMU6DjmJbhkxMOTQFSG6QCJt8cSNtJmtClU06y+IFRG6iW3wOlCPDfkzpWkvdxdtMWf3+DomDLGoLgKRBj+KDjtXFnRVsZ7nrkD24UrGvAp5R6kVLqFqXU7UqpN0SW/7RS6qtKqRuVUh9TSj1qPfvToUOHtQMvRO1kh4mAz1pSl41lGQBgvrWk06GdaYvrC/U1NpiJyRSdaUvF8FU/yIMEw1eUzqkxFczUIc/Mj3jRcG45pByL9quYS6fWIVsp3ToBExB/6LVX4bSt83bd2FHwQQYPqoIcvqAOnx9Apc7ROacs4ZZfeZF9v1STwxfUSmQGIU0D89jiPss9HUXc8yh4oBpYJuAzy2LMRCgVc+/bMHw6wkJpuEG9BuIMX+ZLOim427405z5b8SWdbU0LaK3QtCW+Pp3TVBDcr8w+bPu6+m5mknS61+Y7C2WjdcxCWHhds74039v8XHoMX/V3WMSLPE+aw6eUr7YY10g6YxiOyyBfcVJQ/+g+mZQIkjlja114nbfAX49jwUdkvYMrI9y535koTdqHpmAmyvC1CPjWovC6vCd9lYJYdw0CcX5fcEln6hTddO9BfOWegzPtq6nbs57DjYx1C/iUUjmA3wPwYgCXAfh+pdRlYrUvAbhCa/0EAH8D4DfWqz8dOnRYW9BAupdn1ur8tc+7KLruxadtBgBc85jTklI/Hq/xGkRAWh7oM3zNj7NcKSshtAxf5NeFBx8uiBBMn1Lo5yqacwGYHzBqehqGL1MKR4djK7+Lma5ISEMTCkIVfEmorb+XK7Fe2E/H4MT3yX8g/UBZfof+dq6+oPkeXvH0c+M7gM9uLkbcRu0+yGAHLoh0ks52DJ88hjxTGJclHjoyAOBLTPNKHqh1NQBSrp2opLNmlryu8DpvJ8jh09yB0w36+ABWQXmSS8fwmRy+5WHhmSbwv03jH2lW42SS8Q1lICCPtZ/7pi0a5vhi3588n+ncR/e5b9ri2CSewychJyO0dt9dm1ubH4/n0pnRpFF8hBvrTW3AB3/SZlw4SWfbcWzKlKgt6FgdwzfZAFpeP3w+btoYwwvyBKtH4Ix87B4j/OGn7sCvfvBrwedNsPdlQ4mLJkln8tlQrkEOn2h8NE4H22sRFvE2eRmjVFD8rb/7GXzb//jMVPuyks6GkyTvr1g5kxMV68nwPQ3A7VrrO7XWQwB/CeClfAWt9Se01svV288COHsd+9OhQ4c1hJXkZQrzvRx3ve1a/MhVF0TXfdTOTfjqm1+I733qOcn2VE3wlhyCsAVtJJ1KKcvwWZfOSJ5CVNIZaa+XZTaHT/ZZazCGr7Fr0T6sjkr8j0/cDqC5fAEAnLFtEf/fK5/K2qj+Zn5ZBslWSmaMgzM40eXs9DUlxHPQ9TPXy3DzW16EX3jxY2rXf9K5RtpZx5ba/lV/e5nCsNpPE8maGrj3MuP0SflOp29dsMtyFogUpZ/DxwctRWKwwd96Lp2JUZ1GPVOjGcXH90+mJ0Al36wcObdVLp0HK3bPLPf71lrSaYPO+u2CHD5xPL1cCXZU22OQEryA8bPnWXwuGD5qS5phtM3hK7XbSeqyeu8X9uDho8OgP/0Iw5cyRYl1p+7r4PUWAXMN1Ll0xhBz6ZwEtJ8ep50ngJRL+gzfdH1KuXTy9vg1JwMOvs2R1TH7fAKGz7bdFPCZ3xP+vPIYyhqGb1ZTG3nv1pVPWQsijAfz9D2bPOXZ267bF0dZanufmvX8A9t7cGXtO3OcsJ4B31kA9rD391SfpfAqAB9ax/506NBhDUE/im3liktzPS+ou/bxZ3jLeZDV2qXTC/jauXRSYNi2Dh9EcOQHpsq5dIrzUOjZcvjkNm3P89WXnhptayFSuJ1LPntZnEWhH3Yj6QyX+/b/qNoL+yU/4gOfhX7emGP37lc9HZ/6uatr1wnyLPP2DF9qsZGFajxQBXxnbHcBHzk+Aub7zlgAGBssyUuND9DGEfv50BkvlsOnWQFvN+jjdvacIStLHTB8B5ZZwCeYvuaBXXWNixy+pKSzJ01b/BWDsgww36dCeoBu+57YJx8kk6RTaymVC3MC+TZee+ybi103t+87gp//2xvxur+6IehrLIcv5tBJ+5Gol3RGcvgmkHQCaQfgtqDrZ2qXTmrHTlzwYG26KCMliSwT11PsHottMwnjmDJukiC2V05GNO1TT9ifGOQ9P1hnhs/P4eNy8LVo3UfqefaO627Fk9/yUTxYKTjkvu874BsbncjYEKYtSqkfAHAFgN9MLP8xpdT1Sqnr9+/fH1ulQ4cOxxgUMJ2yaW7ibb/8Sy/Ab33fE73P+Mx0UIcvMQapk3T+/WuejR+96nx/H5mys/WW4YuM8nhQJ6V+fDzUz7PasgzWpXOqgM/v9zTF2wlKKU8W6fIQw3y0EPXHwAcZMuDy++C/n9R6e9N8D+fuXKpdx9VMNO97mcvhaw744stNAekSDxwaYGkux9aFvl2WK3gDat+0JWSp5DHzADBm8hIzaInV4aMTPmYyYr5txqSmpXZF1rcvEsPnZrhlwNY0yJaS36ZB21zu+gGE56TfU4Gkk6SyvC8xxqou95HA8y5tkXo0MHyBaYtrMzYJcrQqBv9IxRzwdvsRl05ider6Tag1bYEf8I0Kl9/Jg8q6/KtZA75A0jllxBe7ZyZt6+hgjCt+5aP459sfDNo3L3nAwV4HpU8cJMvcFrRqE8NHOeUpVrLetGW2QInn0QH+M0m2PG0+JYfv0lkyw6n1qMMXfy599KsPAIAN+OTzaM8jyzhZsJ4B370AuH7r7OozD0qpawC8EcC3a60HsYa01n+otb5Ca33F7t2716WzHTp0mAzfdvmZeNO1j8Hrn3/xxNtuW+rXll5oX4fPQQ7KnnD2djxDWPBnStn1LMOXsJ62csiA4XPrmIAvJenUweBnEvDZ1Wnq+HEoxMs68PPcSwSVckAfLGdDgRgL6vrgf7YeTmwu4DQvfIavftvUchM0Gknn6VsXvO+f5IGAGcgpLumM1uFLMwdBkJNYPwgC4e6DonRBjCfpZMdXao3Dq2NkCti8YHI+H1kOJZ2O6UMtaDEPKE1f4+u7HL54+70s84M5XTF8SrAekcsnNYkQunQqT9JJAVyqzzII4vl+sVuTl6yR++eGVFLSGXtO/Md3fg5/9q93RY9FgnJOeT/pMuTmVHWum22UEnWwks6cGL7JAoOgNl4R3hdtcdu+I3jwyBC//uGbo214DB/bbxCUJVi9SfrjAtgG05aK4fNlnPH9c5T2Gp4+EKNNqYVYHnKsT9NCMnxN6oCZ9iWkwhL0tchb4+M371v7zhwnpO3OZscXAFyklDofJtD7PgAv5ysopZ4E4A8AvEhrffKc1Q4dvgmQZyqZszdte4S2s8x8sEVsRWo5vacBDZmgpAIPGhTKHD6PVewp+6MYuHRq59I5TR2+fYedlGSagBFwgVemIBg+85fnPaYYPvqBVIkcPq8sQ02+o9x2PX7UJQvr1eFrOIep/MN+rjAuSuw7tIpTt85733+eZS6Hryo5Qkv5oDEl5/IGPLp5fcPw+Z+VWtvzPi51lEkg5lEpVGUZxtg837MTCQeWYwyfGP0l4CYEiNGJz6QTejaHD976hLleZp1vze7p+JQv6axh+OrkZ8SWv/uzd2PX5nm7XR3Dl0dy+Ox9EVnfOrpm/rHS8RHoe/uTf7kLgDEkOjJwOWIA8OnbHsSnb3sQP/jM86LHxiElnaaItbn+OcNXV1dv7UxbpnTprP5GJb8TNsYnQmJNeFLNSD6Z61OC1ZugO1ai2mjaUlR9iO+zLofPLG9ZKqShDcC/ZmQwOWu+IN8fQCkQ5vW6lGVI5FHbWpiIL//kLftxcGVk5e8nMtaN4dNajwG8BsBHAHwNwHu11v+ulHqzUurbq9V+E8BmAH+tlLpBKfX369WfDh06bGzwgbQcdNCip563A2//nsuj2/CacXY7MRwzkk5fzpjKn5GBXiyY4Rbm0jSGu3ROw9A9cMgJHmaVWfGyDLw/fDa/l2cJhk/bbaIBH/t9dOdqpu5ODckw5pliZRmaJJ3xz3tVmQBi+PyAj5u0aK/A+agoPaMU/pfg5eZEBnihC2U8CHQD2zLKsHGTnkJrDIsSc73MBgdk2rJ5vhf0tUkmVrLrg/pYt92cYL3kMc73smhZBtNVzgiE7cc+k+YuxmjHvCYZl9ZNpi01ks7IhUPXHLkN837xZxs1e88jxhhioS9VAmFfGiWdrD+ldoG/F/AlJKRAOHE1KehYyQ150rBAi+vXz+GbDDxv1bURD/7qGD6fYUuzXnVIybQliO31paPx/cf6OIus07JgEUlnKe6jtZiw8xm+0j1z1sW0pTo28bmrT1qtxw7ycWdtxbAo8Y2Hjq59h44D1pPhg9b6gwA+KD77Jfb6mvXcf4cOHU4cyJw1DgrcvuspZ+M7nxw3812oset3+1CBC90oMeNq5YFCyplyE5V9LplEZZp4bd8hxvBNOet+yWlbAACvfu6jvcEkHQPPJ2pi+LLM1W3zlrPT52qxhe3IgfFVF+1qcwgTQXa/l2etJZ0p9Ks8wIeODLFz87zXDjdpKcrSy+EbFxr9av/pgM+99hnB6m8b0xa4827Yg/Dk2wBYuXIEvK9k2rJtsc8CNtd+HeiQWpu2BC6d/vL5XhYwMkpVkk5u0d/StEVeiyTp9Lcz7EKqz/LevvPBI8nc5fsOrOBrew8BcMGTb2xk2rr09C3BhNTSXA+AY1tjgVmTpDMTDB8xSjzgW61h+Nq4HdfBTnIJxrf19oJlKrz7IsFulRpHhmMvvxZwz2yfPXfLvXIHXvCR7vP0kk7zN2YSxuECvvh+UsFQU/3LNpDBNpf+au33Q2vg6w8exXk7l6bOL/dNc9zr9Sm8bv7KS4i6HrveSBWzHmqU44F1Dfg6dOjQoS34QEXOMseCLaCFGUpU0ulkjoCrERhsylgR/pfvks/Wy0DSyKmqgG+KaGPHpjncd9AEfdPm8G3f1Mddb7sWgGMzADcYC3L4aurwZUrhrS97PJ5y7g68+f1ftcv5YKptN9/+PZfjxY87o3nFCeGuE/PXlGWo8qNadi4MGo0T6/KwwOZ532mWm+mYHD4XNI9KjX6mMEQ6PyTF8CVNWxAv3m4HtmVYqoAfk8mDM8wzL8kxLjXme6aeppQ+1Q3Y/+X2B+0gUNZ7a1+HTwfL/XxGDVX912TaEoNci+ddEgx7ka7DJxm+991wn30t23rW2z5uX8si84D5Lt7/n6/EOTuWPEOIax5zGk7bOo/3fO5u+9nRSA2wusFnJiSdmjN8RTuGr1XZkxoEOXwTDpa1/VvdA6ysSqqtD3xlL37h776C6990TXTij993aRnn5AzfJMdG+20r6fSdQeMMJUcTs94GYeH1tKTzC3c9jF//8M341Zc9Dq94+qOm21+K4av2NYtRWbCvxPOMfvPovHoGS5H790TGhnDp7NChQwdP0pkYdMhP53oZfu6Fl+Cjr39Oq/Vz5SSdWgN/+WPPwJ/8X0+r7ZcMIlLOoH3RZ16WYZqA7V2vfCoee6aRqU4TMAJ+X/lAKJbDlymFmJqLfiAzpbBtsY8fvvL86HLebuznkR/Bo3YuYXGumZGdHMrbF5d0TuOUCpjJh0OV5HHTfB4w0c6kherwVZLOcRmWIKjJ4fMYvhLR9VOmLbyNWCDEJy3Kah1pRrRloW/dK3nfUgHGJ27Zh5f/8efwh5+607RTXTsTM3xigJVnyjsGYvjkgL9uEOYxImK9OMPn/sVQx7DX1Z6kiSsecCgFPO6sbdi21PcCqGdccEpgmLIs8vmAOLPJ2+YBW1G6HD4e5KXq/gEmjzDZfnKJA103vWlz+MT142qZqiSj9sChVRwZjLEsAuRxGV5jvD+8PX7NBbLphAx0kkCA1mxk+GxZhvg+m4yJZolNpOzRc3bVfqh5+74jAIAv7zkw/f68c+7ff2vNqqUknZLh45MA9ll1klB8XcDXoUOHDQEeFMkBlmMowiHHT159IS6qpIsSMUbwqeefAsCUk3jGBTuxIyHNooGc1PjzAf+cyIHjy8tyNpfO07Yu4CVVrcJpndf4XueZfJMCSD+HL+HSWf1NHQPvmk2Aj/SXNy1NMNYKrg5gdXxZZgcO004W93uZzXHbJBi+TDlXxMClsyyDQa8MxlI5RKn8Nq0TZRmoDXbNcVCX80xZNzy6Bij4WZzLrFGR6UPVftCawf4qx5QGflbOK45BYq5Hs+bw/lL/TB85o0HHoPwBYUvVV0zSKa8FsrRP9blfc73WXVcy4A+2ZXcoHTtHnOGrCfigRA5fXNJZx/Btmp9N+EX9sxNrE2be2cOje6YsrfogdeicLS5KjQ99Za+ZHKmCq1SQ5LN97rUMytaC4bOsWescPrZ/8L6ljiVcd1JI1l3m8E2bv5hCKocPWHtZp8tP9D+n53nsmdsXBlMnOrqAr0OHDhsCij2NAoZP+cFX6zbF+0wp/NwLLsFHXvccXLA7LuUM2hA/CHyQxiWdxJbRDLlhDCp2bEqGjgLKSWvWEVJspIp89riztuHSWODcEDC1lXTyRbOa0CT3IWS3uWAw65Aav/Q
gitextract_98o3bluv/ ├── .other/ │ └── technical_requirements.md ├── Chapter01/ │ ├── Multi_Head_Attention_Sub_Layer.ipynb │ ├── positional_encoding.ipynb │ └── text.txt ├── Chapter02/ │ ├── BERT_Fine_Tuning_Sentence_Classification_DR.ipynb │ ├── in_domain_train.tsv │ └── out_of_domain_dev.tsv ├── Chapter03/ │ ├── KantaiBERT.ipynb │ └── kant.txt ├── Chapter04/ │ └── Transformer_tasks.ipynb ├── Chapter05/ │ ├── BLEU.py │ ├── Trax_Translation.ipynb │ ├── read.py │ └── read_clean.py ├── Chapter06/ │ ├── OpenAI_GPT_2.ipynb │ ├── Training_OpenAI_GPT_2.ipynb │ ├── gpt-2-train_files/ │ │ ├── accumulate.py │ │ ├── dset.txt │ │ ├── encode.py │ │ ├── load_dataset.py │ │ ├── memory_saving_gradients.py │ │ └── train.py │ └── head_view_bert.ipynb ├── Chapter07/ │ └── Summarizing_Text_with_T5.ipynb ├── Chapter08/ │ ├── Summarizing_Text_V2.ipynb │ ├── Tokenizer.ipynb │ ├── Training_OpenAI_GPT_2_CH08.ipynb │ ├── gpt-2-train_files/ │ │ ├── accumulate.py │ │ ├── encode.py │ │ ├── load_dataset.py │ │ ├── mdset.txt │ │ ├── memory_saving_gradients.py │ │ └── train.py │ └── text.txt ├── Chapter09/ │ └── SRL.ipynb ├── Chapter10/ │ ├── Haystack_QA_Pipeline.ipynb │ └── QA.ipynb ├── Chapter11/ │ └── SentimentAnalysis.ipynb ├── Chapter12/ │ └── Fake_News.ipynb └── README.md
SYMBOL INDEX (63 symbols across 12 files)
FILE: Chapter05/read.py
function load_doc (line 12) | def load_doc(filename):
function to_sentences (line 22) | def to_sentences(doc):
function sentence_lengths (line 26) | def sentence_lengths(sentences):
function clean_lines (line 34) | def clean_lines(lines):
FILE: Chapter05/read_clean.py
function load_clean_sentences (line 14) | def load_clean_sentences(filename):
function save_clean_sentences (line 18) | def save_clean_sentences(sentences, filename):
function to_vocab (line 23) | def to_vocab(lines):
function trim_vocab (line 31) | def trim_vocab(vocab, min_occurance):
function update_dataset (line 36) | def update_dataset(lines, vocab):
FILE: Chapter06/gpt-2-train_files/accumulate.py
class AccumulatingOptimizer (line 9) | class AccumulatingOptimizer(object):
method __init__ (line 10) | def __init__(self, opt, var_list):
method reset (line 18) | def reset(self):
method compute_gradients (line 25) | def compute_gradients(self, loss):
method apply_gradients (line 33) | def apply_gradients(self):
FILE: Chapter06/gpt-2-train_files/encode.py
function main (line 21) | def main():
FILE: Chapter06/gpt-2-train_files/load_dataset.py
function load_dataset (line 8) | def load_dataset(enc, path, combine, encoding=None):
function binary_search (line 46) | def binary_search(f, lo, hi):
class Sampler (line 58) | class Sampler(object):
method __init__ (line 64) | def __init__(self, chunks, seed=None):
method sample (line 72) | def sample(self, length):
FILE: Chapter06/gpt-2-train_files/memory_saving_gradients.py
function gradients_speed (line 23) | def gradients_speed(ys, xs, grad_ys=None, **kwargs):
function gradients_memory (line 26) | def gradients_memory(ys, xs, grad_ys=None, **kwargs):
function gradients_collection (line 29) | def gradients_collection(ys, xs, grad_ys=None, **kwargs):
function gradients (line 32) | def gradients(ys, xs, grad_ys=None, checkpoints='collection', **kwargs):
function tf_toposort (line 302) | def tf_toposort(ts, within_ops=None):
function fast_backward_ops (line 320) | def fast_backward_ops(within_ops, seed_ops, stop_at_ts):
function capture_ops (line 326) | def capture_ops():
function _to_op (line 342) | def _to_op(tensor_or_op):
function _to_ops (line 347) | def _to_ops(iterable):
function _is_iterable (line 352) | def _is_iterable(o):
function debug_print (line 360) | def debug_print(s, *args):
function format_ops (line 372) | def format_ops(ops, sort_outputs=True):
function my_add_control_inputs (line 384) | def my_add_control_inputs(wait_to_do_ops, inputs_to_do_before):
FILE: Chapter06/gpt-2-train_files/train.py
function maketree (line 56) | def maketree(path):
function randomize (line 63) | def randomize(context, hparams, p):
function main (line 72) | def main():
FILE: Chapter08/gpt-2-train_files/accumulate.py
class AccumulatingOptimizer (line 9) | class AccumulatingOptimizer(object):
method __init__ (line 10) | def __init__(self, opt, var_list):
method reset (line 18) | def reset(self):
method compute_gradients (line 25) | def compute_gradients(self, loss):
method apply_gradients (line 33) | def apply_gradients(self):
FILE: Chapter08/gpt-2-train_files/encode.py
function main (line 21) | def main():
FILE: Chapter08/gpt-2-train_files/load_dataset.py
function load_dataset (line 8) | def load_dataset(enc, path, combine, encoding=None):
function binary_search (line 46) | def binary_search(f, lo, hi):
class Sampler (line 58) | class Sampler(object):
method __init__ (line 64) | def __init__(self, chunks, seed=None):
method sample (line 72) | def sample(self, length):
FILE: Chapter08/gpt-2-train_files/memory_saving_gradients.py
function gradients_speed (line 23) | def gradients_speed(ys, xs, grad_ys=None, **kwargs):
function gradients_memory (line 26) | def gradients_memory(ys, xs, grad_ys=None, **kwargs):
function gradients_collection (line 29) | def gradients_collection(ys, xs, grad_ys=None, **kwargs):
function gradients (line 32) | def gradients(ys, xs, grad_ys=None, checkpoints='collection', **kwargs):
function tf_toposort (line 302) | def tf_toposort(ts, within_ops=None):
function fast_backward_ops (line 320) | def fast_backward_ops(within_ops, seed_ops, stop_at_ts):
function capture_ops (line 326) | def capture_ops():
function _to_op (line 342) | def _to_op(tensor_or_op):
function _to_ops (line 347) | def _to_ops(iterable):
function _is_iterable (line 352) | def _is_iterable(o):
function debug_print (line 360) | def debug_print(s, *args):
function format_ops (line 372) | def format_ops(ops, sort_outputs=True):
function my_add_control_inputs (line 384) | def my_add_control_inputs(wait_to_do_ops, inputs_to_do_before):
FILE: Chapter08/gpt-2-train_files/train.py
function maketree (line 56) | def maketree(path):
function randomize (line 63) | def randomize(context, hparams, p):
function main (line 72) | def main():
Condensed preview — 40 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (3,835K chars).
[
{
"path": ".other/technical_requirements.md",
"chars": 2651,
"preview": "**Software and hardware specifications**\n========================================\n\n<table>\n<thead>\n<tr class=\"header\">\n<"
},
{
"path": "Chapter01/Multi_Head_Attention_Sub_Layer.ipynb",
"chars": 28993,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Multi-Head Attention Sub-Layer.i"
},
{
"path": "Chapter01/positional_encoding.ipynb",
"chars": 9712,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"positional_encoding.ipynb\",\n "
},
{
"path": "Chapter01/text.txt",
"chars": 24151,
"preview": "The black cat sat on the couch and the brown dog slept on the rug.The cat did not cross the street because it was too we"
},
{
"path": "Chapter02/BERT_Fine_Tuning_Sentence_Classification_DR.ipynb",
"chars": 183184,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"BERT_Fine_Tuning_Sentence_Classi"
},
{
"path": "Chapter02/in_domain_train.tsv",
"chars": 438705,
"preview": "gj04\t1\t\tour friends wo n't buy this analysis , let alone the next one we propose .\ngj04\t1\t\tone more pseudo generalizatio"
},
{
"path": "Chapter02/out_of_domain_dev.tsv",
"chars": 28504,
"preview": "clc95\t1\t\tsomebody just left - guess who .\nclc95\t1\t\tthey claimed they had settled on something , but it was n't clear wha"
},
{
"path": "Chapter03/KantaiBERT.ipynb",
"chars": 278901,
"preview": "{\"nbformat\":4,\"nbformat_minor\":0,\"metadata\":{\"accelerator\":\"GPU\",\"colab\":{\"name\":\"KantaiBERT_2.ipynb\",\"provenance\":[],\"c"
},
{
"path": "Chapter04/Transformer_tasks.ipynb",
"chars": 175969,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Transformer tasks.ipynb\",\n "
},
{
"path": "Chapter05/BLEU.py",
"chars": 1985,
"preview": "#BLEU : Bilingual Evaluation Understudy Score\n#Copyright 2020, MIT License BLEU Examples\n#REF PAPER: Kishore Papineni, e"
},
{
"path": "Chapter05/Trax_Translation.ipynb",
"chars": 4094,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Trax_Translation.ipynb\",\n \""
},
{
"path": "Chapter05/read.py",
"chars": 2555,
"preview": "#Pre-Processing datasets for Machine Translation\n#Copyright 2020, Denis Rothman, MIT License\n#Denis Rothman modified the"
},
{
"path": "Chapter05/read_clean.py",
"chars": 2355,
"preview": "#Pre-Processing datasets for Machine Translation\n#Copyright 2020, Denis Rothman, MIT License\n#Denis Rothman modified the"
},
{
"path": "Chapter06/OpenAI_GPT_2.ipynb",
"chars": 25264,
"preview": "{\"nbformat\":4,\"nbformat_minor\":0,\"metadata\":{\"colab\":{\"name\":\"OpenAI_GPT_2_KS.ipynb\",\"provenance\":[],\"collapsed_sections"
},
{
"path": "Chapter06/Training_OpenAI_GPT_2.ipynb",
"chars": 12880,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Training OpenAI GPT-2.ipynb\",\n "
},
{
"path": "Chapter06/gpt-2-train_files/accumulate.py",
"chars": 1464,
"preview": "import argparse\nimport json\nimport os\nimport numpy as np\nimport tensorflow as tf\nimport time\n\n\nclass AccumulatingOptimiz"
},
{
"path": "Chapter06/gpt-2-train_files/encode.py",
"chars": 1384,
"preview": "#!/usr/bin/env python3\n# Usage:\n# PYTHONPATH=src ./encode.py <file|directory|glob> /path/to/output.npz\n# PYTHONPATH=sr"
},
{
"path": "Chapter06/gpt-2-train_files/load_dataset.py",
"chars": 2637,
"preview": "import glob\nimport numpy as np\nimport os\nimport tensorflow as tf\nimport tqdm\n\n\ndef load_dataset(enc, path, combine, enco"
},
{
"path": "Chapter06/gpt-2-train_files/memory_saving_gradients.py",
"chars": 17312,
"preview": "from toposort import toposort\nimport contextlib\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.contrib.gra"
},
{
"path": "Chapter06/gpt-2-train_files/train.py",
"chars": 13236,
"preview": "#!/usr/bin/env python3\n# Usage:\n# PYTHONPATH=src ./train --dataset <file|directory|glob>\n\nimport argparse\nimport json\ni"
},
{
"path": "Chapter06/head_view_bert.ipynb",
"chars": 1091596,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"head_view_bert.ipynb\",\n \"pr"
},
{
"path": "Chapter07/Summarizing_Text_with_T5.ipynb",
"chars": 296801,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Summarizing_Text_with_T5.ipynb\","
},
{
"path": "Chapter08/Summarizing_Text_V2.ipynb",
"chars": 267738,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Summarizing Text V2.ipynb\",\n "
},
{
"path": "Chapter08/Tokenizer.ipynb",
"chars": 10765,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Tokenizer.ipynb\",\n \"provena"
},
{
"path": "Chapter08/Training_OpenAI_GPT_2_CH08.ipynb",
"chars": 123255,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"Training OpenAI GPT-2-CH08.ipynb"
},
{
"path": "Chapter08/gpt-2-train_files/accumulate.py",
"chars": 1464,
"preview": "import argparse\nimport json\nimport os\nimport numpy as np\nimport tensorflow as tf\nimport time\n\n\nclass AccumulatingOptimiz"
},
{
"path": "Chapter08/gpt-2-train_files/encode.py",
"chars": 1384,
"preview": "#!/usr/bin/env python3\n# Usage:\n# PYTHONPATH=src ./encode.py <file|directory|glob> /path/to/output.npz\n# PYTHONPATH=sr"
},
{
"path": "Chapter08/gpt-2-train_files/load_dataset.py",
"chars": 2637,
"preview": "import glob\nimport numpy as np\nimport os\nimport tensorflow as tf\nimport tqdm\n\n\ndef load_dataset(enc, path, combine, enco"
},
{
"path": "Chapter08/gpt-2-train_files/mdset.txt",
"chars": 88968,
"preview": "This suggests that, as amoeboid cells are less contractile, while mesenchymal\ncells are more contractile, and there may "
},
{
"path": "Chapter08/gpt-2-train_files/memory_saving_gradients.py",
"chars": 17312,
"preview": "from toposort import toposort\nimport contextlib\nimport numpy as np\nimport tensorflow as tf\nimport tensorflow.contrib.gra"
},
{
"path": "Chapter08/gpt-2-train_files/train.py",
"chars": 13236,
"preview": "#!/usr/bin/env python3\n# Usage:\n# PYTHONPATH=src ./train --dataset <file|directory|glob>\n\nimport argparse\nimport json\ni"
},
{
"path": "Chapter09/SRL.ipynb",
"chars": 275415,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"colab\": {\n \"name\": \"SRL.ipynb\",\n \"provenance\": "
},
{
"path": "Chapter10/Haystack_QA_Pipeline.ipynb",
"chars": 119701,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"l"
},
{
"path": "Chapter10/QA.ipynb",
"chars": 63895,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"l"
},
{
"path": "Chapter11/SentimentAnalysis.ipynb",
"chars": 14692,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"l"
},
{
"path": "Chapter12/Fake_News.ipynb",
"chars": 6489,
"preview": "{\n \"nbformat\": 4,\n \"nbformat_minor\": 0,\n \"metadata\": {\n \"kernelspec\": {\n \"display_name\": \"Python 3\",\n \"l"
},
{
"path": "README.md",
"chars": 5673,
"preview": "\n\n---\n\n## Join Our Newsletters 📬\n\n### DataPro \n*The future of AI is unfolding. Don’t fall behind.*\n\n<p><a href=\"https:/"
}
]
// ... and 3 more files (download for full content)
About this extraction
This page contains the full source code of the PacktPublishing/Transformers-for-Natural-Language-Processing GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 40 files (35.9 MB), approximately 916.3k tokens, and a symbol index with 63 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.